International Journal of Applied Engineering Research ISSN 0973-4562 Volume 11, Number 2 (2016) pp 829-833 © Research India Publications. http://www.ripublication.com
A Development of Virtual Reality Game utilizing Kinect, Oculus Rift and Smartphone DongIk Lee, KiYeol Baek, JiHyun Lee Student, Department of Multimedia Engineering, Andong National University,
[email protected],
[email protected],
[email protected] Hankyu Lim(corresponding author), Professor, Department of Multimedia Engineering Andong National University
[email protected]
The KOS, a VR game developed in this work, is controlled with a combination of a Kinect (a motion tracking input device) and an Oculus Rift (a representative HMD product). It also uses a smartphone to offer more diverse input methods. The KOS was implemented using C++ and DirectX 11. The
AbstractVideo games such as arcade games, computer games, and smart device games have attracted the attention of many people, regardless of age, gender, and social status. Video games are controlled via various types of input devices that have emerged over time. In recent years, virtual reality (VR) games that require VR-specific devices like a head-mounted display (HMD) have become popular. The Oculus Rift is one of the most widely used VR HMDs. This paper describes the design and implementation of a VR game that is operated using three different types of device. It uses a Kinect, an Oculus Rift, and a smartphone as input device in order to increase user convenience and cost efficiency. The developed VR game provides a greater degree of user control, thus giving a competitive edge over existing games employing only one or two input devices.
KOS manipulates the gun aiming based on the player ’ s movements tracked by the Kinect and utilizes smartphone's touch, swipe and drag features to play the game. The smartphone was chosen as a supplementary input device because of its high penetration rates. For dedicated input devices, additional costs are necessary to purchase third party hardware, but the majority of people nowadays already own a smartphone. Today's smartphone incorporates various sensors and advanced features that can be exploited to feed useful inputs to the VR content including video games [1-5]. The function spec and flow-chart of KOS and server are introduced. Also the map design is showed. The VR game is implemented by using three different types of input device, which allows players to make diverse kinds of motions to operate the game.
Keywords: Head-mounted Display, Oculus Rift, Kinect, Smartphone, Virtual Reality Game
Introduction Game providers and gamers are watching with great interest the next-generation input devices. For example, the Oculus Rift is an upcoming HMD for virtual reality games. The growing popularity of VR has brought about a steady increase in VR content along with continuous innovations in VR devices. In particular, HMD technology has attracted significant interest from industry. Virtual Reality games that require VR-specific devices like a head-mounted display have become popular. One of the most widely known HMDs is the Oculus Rift, a relatively cheap and easy-to-use VR HMD developed by Oculus VR. It is expected that VR content combined with an HMD like the Oculus Rift will be used in a wide variety of industrial sectors, such as video gaming, film and media, education, sightseeing, the healthcare and medical field, sports, and advertising. However, the majority of VR games that have been in development over the past few years rely on traditional controllers like a keyboard, a mouse, and a gamepad. The expectations of fully immersive VR cannot be met by such devices, and a natural user interface using gestures and spoken commands is required to enable users to control and interact with objects in the virtual world. Nevertheless, there are still some types of inputs that need to be explicitly given via an input device, which makes the use of wired and wireless input devices inevitable.
Related Research 1) Kinect The KOS game was implemented using Kinect v1. The Kinect SDK provides skeletal tracking, the capability to extract and track the skeleton image of one or two people moving within Kinect's field of view. A Kinect skeleton is made up of 20 body joints (head, hands, feet, hip center, etc.). Among the Kinect skeleton data (20 joints coordinates in 3D space), the KOS game uses those corresponding to the player's right hand in order to manipulate the gun aiming. Microsoft DirectX 11, a collection of application programming interfaces (APIs) used to implement the KOS, is based on a left-handed coordinate system, whereas the Kinect uses a right-handed coordinate system. To obtain the correct joint coordinates for DirectX 11, the joint location information from the Kinect is modified by multiplying -1 to the z-coordinate.
829
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 11, Number 2 (2016) pp 829-833 © Research India Publications. http://www.ripublication.com PC using the tap, swipe, drag, and vibration features of the smartphone. There exist some other commercial devices for VR games, including the Razer Hydra (a motion and orientation detection game controller), the Myo armband (a wearable gesture control and motion control device), Virtuix Omni (an omnidirectional treadmill), the Emotiv Epoc (a neuro-signal acquisition and processing headset), and Oculus Touch (a hand controller system that understands hand gestures) [5].
Fig. 1. Coordinates System[2]
Function and Implementation 1) Function Analysis The developed KOS game can be divided into three parts: the KOS client implementing the VR game, the smartphone as an input device, and the server responsible for the communication between the KOS client and the smartphone. The communication between the server and the client is performed using Input/Output Completion Ports (IOCP). In the smartphone, the Channel Selector is used to process asynchronous sockets. It is possible that the KOS is implemented using the Transmission Control Protocol/Internet Protocol (TCP/IP) instead of IOCP. The KOS program includes features such as head tracking and display capabilities relying on the Oculus Rift, player motion sensing capabilities relying on the Kinect, gun aiming manipulations, and conversion of the smartphone packets sent by the server to an input data format. Figure 3 shows the data flow diagram of the KOS game.
2) Oculus Rift The KOS was developed using the Oculus Rift DK2, the latest development kit for the Oculus Rift. The Oculus Rift requires graphic rendering for two different viewpoints of the left and right eyes. In addition, fish eye rendering is made for the left and right screens at a significant computational cost. The use of Oculus Rift thus increases the time of graphic rendering, which is a big obstacle in improving graphic quality and gaming performance. This is why the implemented KOS performs only lighting calculations with regard to the diffuse texture and bump mapping, without introducing shadow and blur effects. Like the Kinect, the Oculus Rift is based on a right-handed coordinate system. Hence, Oculus Rift's head tracking data passed to DirectX 11 that uses a left-handed coordinate system need to be corrected to compensate the differences [6].
Fig. 2. Oculus Rift Coordinate System[3]
3) Input Devices The KOS game presented in this paper uses the smartphone as input device. The smartphone has a number of built-in sensors, such as gyroscopes, accelerometers, proximity sensors, ambient light sensors, and Hall sensors, that are useful in capturing player gestures and movements. The use of smartphone also gives an advantage that an application that communicates with a PC can be built using tap, swipe, and drag gestures. That is, it is possible to implement an input device with various features for VR games without the need for manufacturing or purchasing dedicated input devices. This paper develops a game application that communicates with a
Fig. 3. DFD Structure Chart of KOS
2) Flow Chart Figure 4 below is a flow chart of the KOS and the Server. The Server was implemented as an IOCP Server and was designed
830
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 11, Number 2 (2016) pp 829-833 © Research India Publications. http://www.ripublication.com to wait for clients’ access when the program has started and process Packets on receipt of them according to packet sources. Packets coming from the PC are processed by the Server only and Packets coming from Smartphones are also processed directly by the Server sometimes. Since the Packets are for game manipulation in most cases, they are to be transmitted to the KOS. When started, the KOS initializes the System, attempts access to Clients, and processes Packets coming from the Server to manipulate the game. The KOS is configured to make the game progress in accordance with determined logics and store the scores and time in the DB when the game is over.
Fig. 5. Map Design on 3Ds Max
Fig. 4. Flow Chart of KOS and Server
Fig. 6. Coordinate and Rotation Value by Object
3) Map Design The map was designed by arranging the objects in 3Ds Max as shown in Figure 5. However, to utilize DirectX11 Instancing, designed objects should be directly arranged in games. As shown in Figure 6, information on the Position (XYZ) and Rotation(Pitch(x), Yaw(y), Roll(z)) of objects was recorded by object. Then, the objects were rearranged in the GameTool as shown in Figure 7, extracted in the form of scenes, and used in the implementation of InGame. Fig. 7. Object Scene in GameTool
4) Aiming Function utilizing Kinect The KOS moves the gun aiming point along the x and y axes of the screen based on the x and y coordinates of the player's hand detected by the Kinect. To shoot, the KOS computes the 3D ray direction of the current x and y coordinates of the hand on the screen using the camera's view and projection matrix. The computed ray direction becomes the ballistic trajectory,
831
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 11, Number 2 (2016) pp 829-833 © Research India Publications. http://www.ripublication.com and the effects of its encounters with virtual objects (monsters) are checked to harm the monsters [7]. 5) View of Left and Right eyes of Oculus Rift, Projection Matrix Calculation The KOS uses Oculus Rift's head tracking information and the camera's view and projection matrix in order to compute the view and projection matrix for the left and right eyes [8].
Fig. 20. 3D UI Object With Collision OBB(Different angle)
Fig. 8. Rendering of KOS with Oculus Rift
6) UI Event Processing In the developed VR game, the Kinect and Oculus Rift devices enable players to control and interact with objects in a virtual world, but it is not possible to use mouses or keyboards to play the game. The developed game has user interfaces (UIs) on a three-dimensional (3D) plane in the 3D game world and renders a 3D scene to a texture. Button UI events are related to harming colliding objects (monsters). When the smartphone is tapped over the Button UI, ScreenRay is created to determine whether any monster is hit by the gunshot. Figure 9 and Figure 10 presents a screenshot of the Ranking and Start Button UIs of the developed VR game. Note that there are colliding objects shown in the screenshot. To make them easily recognizable, the colliding objects are marked with a red line. Figure 11 is another screenshot of the Ranking and Start Button UIs that are viewed at a different angle.
Fig. 31. Play KOS with Kinect, Oculus Rift, Smartphone
Conclusion The implemented VR game uses three different types of input device, which allows players to make diverse kinds of motions to operate the game. It also allows more various input methods and additional game controls, maximizing the utilization of the features provided by each device. The Kinect and Oculus Rift devices enable players to control and interact with objects in a virtual world, but it is not possible to use mouses or keyboards to play the VR game. The Kinect provides fairly accurate motion sensing, but it may be difficult to use it in a limited space due to the need for a practical tracking area. With the Oculus Rift, the wearer's movement range can be further restricted, which makes the use of Kinect less effective. Apart from the Kinect v1 and the smartphone, other types of input devices that are suitable for use in VR content are currently under development and continue to grow in number and diversity. One can use all of the three input devices at the same time but it is also possible to use them selectively i.e., only one or two devices best suited to the characteristics of a game are used. The developed game has user interfaces on a three-dimensional plane in the 3D game world and renders a 3D scene to a texture. The developed VR game provides a greater degree of user control, thus giving a competitive edge over existing games employing only one or two input devices.
Fig. 9. 3D UI Object With Collision OBB
832
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 11, Number 2 (2016) pp 829-833 © Research India Publications. http://www.ripublication.com Acknowledgement This work was supported by a grant from 2015 Seoul Accord project(R0613-15-1148) of MISP(Ministry of Science, ICT and Future Planning) and IITP(Institute for Information and Communication Technology Promotion). .
References [1]
[2]
[3]
[4]
[5]
[6] [7] [8]
Junghyun Kim, Hankyu Lim, “Game Implementation based on Event utilizing gameUnity 3D”, 2015 Proceedings of KIPS Fall Conference, pp. 977-979, 2015. Dongik Lee, Hankyu Lim, “Virtual Reality Contents using the OculusLift and Kinect”, Proceedings of the MCSI, pp. 102-105, 2015. Hangchan Jung, Hankyu Lim, “Kids Game Storytelling”, 2015 Proceeding of KMMS Fall Conference, pp. 416-418, 2015. Daegeun Kwon, Hankyu Lim, “Action Simulation Game Design using the Leap Motion Controller”, Proceeding of ICMEM 2014, pp. 381-391, 2014. Dongik Lee, Hankyu Lim, “Game Development utilizing Several Input Devices”, Advanced Science and Technology Letters, vol. 113, pp. 134-139, 2015. Coordinate Systems (Direct3D 9), https://msdn.microsoft.com/kokr/library/windows/desktop/bb204853(v=vs.85).aspx Oculus Rift Coordinate System, https://developer.oculus.com/documentation/
833
