3D Interaction techniques using gestures recognition

3D Interaction techniques using gestures recognition in virtual environment A.Messaci1,2 1 2

N.Zenati1

A.Bellarbi1

M.Belhocine1

Centre de Développement des Technologies Avancées CDTA Alger, Algerie. Département des Mathématiques et de l'Informatique Université Larbi Ben M'hidi Oum-El-Bouaghi. Algerie 1{amessaci, nzenati, abellarbi,mbelhocine}@cdta.dz

Abstract In virtual and augmented reality (VR/AR), the interaction is used to designate a set of rules and techniques that allows the user to perform interaction tasks in a virtual/augmented environment. The use of hand gestures is the most natural way to perform an interaction, for this reason gestures recognition has a part in the word of interaction. In this context, the aim is to study the different interaction techniques that are used to interact with 3D objects in the case of virtual/augmented reality and the different methods for vision based hand gesture recognition. In this paper we develop a 3D interaction technique "selection/manipulation" using gestures recognition in virtual environment, the choice of the interaction technique and the method of gesture recognition are explained. This implemented system provides more flexible, natural and intuitive interaction possibilities, and also offers an economic and practical way of interaction. Keywords: Virtual reality, augmented reality, 3D interaction, gesture recognition.

I. INTRODUCTION To exploit virtual and augmented reality, the concept of interaction was introduced. However, experiments in this field have shown that the use of 2D interfaces in virtual environments was not effective at all from the point of view of the interaction that is what led to the 3D interaction. The interaction plays a very important role in the effective use of a computer and its different applications. The interaction can be defined as a language of communication between man and machine. This language is the set of actions/reactions loop and computer through sensor and motor interfaces and interaction techniques. The fundamental tasks of 3D interaction are: (i) Navigation, it includes all view point movements of the users in virtual environments. (ii) Selection, it is the task of acquisition or designation of a target. (iii) Manipulation, it represents the process that allows to change the properties of an object or a set of objects. Virtual world and application control allow users to run a command to achieve a specific goal or

objective. These3D interactions techniques are often performed by tools or devices brought by hand, which need a special learning because it is not natural, In this paper, we are interested in the selection and the manipulation of virtual objects in a virtual environment using a vision based hand gesture recognition technique. Interacting by hand is the most natural and intuitive way for a person to communicate with its environment, it would be interesting to change the computer systems to take into account this form of interaction.

II. RELATED WORK Much research has examined hand-gesture-based interaction with objects in virtual environment. Mechanical data gloves such as the CyberGlove were used to obtain information on hand motions [1, 2]. In this approach, users had to wear burdensome, uncomfortable data gloves that limited the natural movements of their hands. To overcome this drawback, vision-based approaches are driven by gesture recognition, using a video camera or webcam instead of a wearable device. One such approach is DesIRe [3], which enables users to create, move, rotate, scale, and select objects in VEs by using predefined hand gestures by means of an infrared LED. Dardas and Alhaj [1] implemented a technique that uses hand gestures to generate commands for interacting with objects in games, combining Bag-of-Features and Support Vector Machine (SVM). Some searchers have used colored markers placed on hands [4,5], the fudicial markers have used as an interaction tools [6]. More recently, recognition-based approaches using Microsoft Kinect [7] have attracted greater interest than techniques that employ general 2D cameras (such as webcams, video cameras, or digital cameras) to obtain a sequence of hand motions because of Kinect’s superior real-time image-processing features, which enable greater accuracy and enhanced performance [8]. Slambekova [9] developed a more natural technique for controlling objects in a virtual world, using Kinect to combine gaze direction with hand gestures. Several university research groups have used Kinect to acquire information on hand motions. The MXR research lab at the University of Southern California has uploaded a video that

2) Tracking gesture a)

3D gesture recognition

Our tests have been realized in a simple virtual environment containing a cube, sphere and cylinder, using Unity3D (see Figure 3).

There are two gesture recognition approaches: based vision and based sensors [14, 15]. The choice between these approaches depends on various parameters. To distinguish between the two approaches, hardware tool is one of the most important criteria. In fact , if the material used is known before the choice will be guided by the material because it is obvious to opt for sensors based gesture recognition techniques if we use data gloves or opt for techniques based vision if the capture tool is a camera[16].We are interested in techniques of gesture recognition based vision because our system is based on Kinect for gesture recognition. These techniques of gesture recognition are divided into two representative families: recognition of 3D model-based gestures [17,18] and gesture recognition based appearance. These approaches consist of several representations[19].In our approach we need some precision to the gestures recognition, the Kinect gives us depth information which leads us to favor gestures based on 3D models recognition techniques. we choose geometrical 3D model this approach contain essential information of the skeleton. b)

3D Gestures modelisation

Using Nimble SDK system developed by 3Gear provides 3D hand tracking for 2 hands, specifically for desktop use. The 3Gear system runs as a server application. Developers wanting to develop their own software based on the interaction captured by 3Gear should let their software connect to the locally running 3Gear hand tracking server. The 3Gear server sends messages to connected client, most of these messages are regarding the detection of hand poses. These messages contain the global orientation of each detected hand, as well as the positions of all the joints per hand. we consider a 3Dskeleton model for each hand (left/right).The hand’s joints are considered as local landmarks. Using Nimble SDK library we get the 3D position and orientation of each joint’s. The wrist’s joint is also positioned relatively to the Kinect. The used 3D skeleton model simplify the 3D hand reconstruction or reproduction on the worker side, it also gives a robust gestures tracking and recognition.

IV. IMPLEMENTATION AND RESULTS A. Virtual environment The virtual environment are generated by Unity3D, Unity3D is a cross-platform game creation system developed by Unity Technologies, including a game engine and integrated development environment (IDE). It is used to develop video games for web sites, desktop platforms, consoles, and mobile devices. First announced only for Mac OS, at Apple’s Worldwide Developers Conference in 2005, it has since been extended to target more than fifteen platforms. B. Results

1) The Virtual environment

Fig .3. Virtual environment

2) Hands gesture Users hands gesture are reproducing using Nimble SDK (see Figure 4)

Fig .4. Hands gestures

3) Object selection If the hand approaches the volume surrounding the object, the object change color; the color blue was associated for the selection phase (see Figure 5).

Fig .5. selection phases

4) Manipulation

interaction tasks in a virtual or augmented environment. In this paper, we have listed 3D interaction and gesture recognition techniques, advantages and drawbacks of each. Then, we have chosen both adequate 3D interaction and gesture recognition techniques according to a specific application in order to develop an interactive virtual reality application, in which we have used the "virtual hand" as 3D interaction techniques (selection/manipulation) and the vision based 3D Geometric Model technique for gesture recognition. In addition, we have integrated the user view point using an HMD as a display device, in order to give more degrees of freedom to the user. As the first achieved results shown, this developed system allows the user to be more flexible when he performs a set of tasks. In addition, the chosen techniques reduces errors due to bad detection and the brightness variation and offers an economic and practical way of interaction.

As soon as we manipulate the object the hand change color (visual indication) in this phases the position of the object is changed (see Figure 6).

VI. REFERENCES [1]

Fig .6. Manipulation of object

5) Navigation We define two gestures (i) moving forward, if an object is far (see Figure 7.a), and (ii) moving backward(see Figure 7.b)

Fig .7. (a) Move Forward (b)Move Backward

V. CONCLUSION The concept of interaction is now used to refer to a set of rules and techniques that allows the user to perform

N. H. Dardas and M. Alhaj, “Hand Gesture Interaction with a 3D Virtual Environment”, The Research Bulletin of Jordan ACM, vol. 2, no. 3, (2011), pp. 86-94. [2] S. S. Rautaray and A. Agrawal, “Real Time Hand Gesture Recognition Systems for Dynamic Applications”, International Journal of UbiComp (IJU), vol. 3, no. 1, (2012), pp. 21-31. [3] M. A. Kavakli, “Designing in virtual reality (DesIRe): a gesturebased interface”, Proceedings of the 2nd international conference on Digital interactive media in entertainment and arts, (2007), pp. 131136. [4] A. Bellarbi, S., Benbelkacem, N., Zenati-Henda, M. Belhocine. Hand Gesture Interaction Using Color-Based Method For Tabletop Interfaces. IEEE 7th International Symposium on Intelligent Signal Processing (WISP), 2011. [5] Wang, R. Y., & Popović, J. (2009). Real-time hand-tracking with a color glove.ACM Transactions on Graphics (TOG), 28(3), 63. [6] A.Bellarbi, H.Belghit, S.Bwenbelkacem, N.Zenati, and M.Belhocine. "Hand gesture recognition using contour based method for tabletop surfaces." In Networking, Sensing and Control (ICNSC), 2013 10th IEEE International Conference on, pp. 832-836, 2013. [7] Microsoft Corporation, Kinect for Windows & Kinect for Windows SDK Beta 1 Programming Guide Version, http://www.microsoft.com/en-us/kinectforwindows/, Kinect for Windows SDK, http://msdn.microsoft.com/enus/library/hh855347.aspx (accessed on March 2013). [8] J. O. Kim, C. Park, J. S. Jeong, N. Baek and K. H. Yoo, “A Gesture Based Camera Controlling Method in the 3D Virtual Space”, International Journal of Smart Home, vol. 6, no. 4, (2012), pp. 117126. [9] T. Phan, “Using Kinect and OpenNI to Embody an Avatar in Second Life”, http://www.youtube.com/watch?v=ehTvtkybubM (http://projects.ict.usc.edu/mxr/) (accessed on March 2013) [10] D. Slambekova, “Gaze and Gesture Based Object Interaction in Virtual World”, Master thesis, Rochester Institute of Technology. Rochester, New York, (2011). [11] J.O.Kim, M.Kim, K.H.Yoo"Real-Time Hand Gesture-Based Interaction with Objects in 3D Virtual Environments", communication of the International Journal of Multimedia and Ubiquitous Engineering Vol.8, No.6 (2013), pp.339-348 http://dx.doi.org/10.14257/ijmue.2013.8.6.33. [12] Melgar, E. R. and C. C. Diez. Arduino and Kinect Projects: Design, Build, Blow Their Minds, Apress.2012

[13] Martin, J. Reconnaissance de gestes en vision par ordinateur, Institut National Polytechnique de Grenoble-INPG.2000 [14] Knecht, Martin, Georg Tanzmeister, Christoph Traxler, and Michael Wimmer. "Interactive BRDF estimation for mixed-reality applications." (2012). [15] Ouramdane, N., Otmane, S., Davesne, F., &Mallem, M. FOLLOWME: a new 3D interaction technique based on virtual guides and granularity of interaction. In Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications.(2006, June). [16] Ouramdane, N., Davesne, F., Otmane, S., &Mallem, M. 3D interaction technique to enhance telemanipulation tasks using virtual environment. In IEEE/RSJ International Conference on Intelligent Robots and Systems, (pp. 5201-5207). 2006. [17] Roth, S. D. Ray casting for modeling solids. Computer graphics and image processing, 18(2), 109-144. 1982. [18] Quek, F. K. Toward a vision-based hand gesture interface. Virtual Reality Software and Technology Conference. 1999 [19] Rautaray, S. and A. Agrawal. "Vision based hand gesture recognition for human computer interaction: a survey." Artificial Intelligence Review: 1-54. 2012