Chun-yu CHENG, Meng-lin SHENG, Zong-min YU, Wen-xuan ZHANG, An-qi LI, Kai-yu WANG*
Application Development of 3D Gesture Recognition and Tracking Based on the Intel Real Sense Technology Combing with Unity3D and WPF Abstract: Intel real sense technology makes the man-machine interaction easier. In this paper, the Intel real sense technology and the F200 camera are adopted. API and some functions in Intel SDK are used. In addition, they are tentatively combined with WPF application framework and Unity3D on traditional PC. A 3D gesture recognition and tracking application is realized with C # programming language. This paper is a meaningful attempt to combine Intel real sense technology with some other platforms except Intel SDK. It has a guiding significance for the popularization and flexible application of the Intel real sense technology. Keywords: Intel real sense technology; 3D gesture recognition; windows presentation foundation (WPF)
1 Introduction 3D gesture recognition and tracking is an important application of the real sense technology [1,2]. Current research on 3D gesture recognition focuses on algorithm level [3-6]. However, the research on the perfect fusion of the real sense technology and various experimental platforms on traditional PC is rare. This kind of fusion can make the application development easier. And it can make the man-machine interaction more friendly. WPF is the next generation of display system that Microsoft promotes vigorously. As a subset of the.net framework, it’s suitable for most.net developers to learn quickly. It has universal meanings [7-8]. Unity platform is a comprehensive development tool to construct three-dimensional models. It has a large user group in creating 3D animation, making three-dimensional models, developing games and other fields [9-11]. At the same time, Intel real sense technology has good support to Unity. The interconnection between Unity3D software and Visual Studio platform is perfect.
*Corresponding author: Kai-yu WANG, College of Electronic Science and Technology, Dalian University of Technology, Dalian, China, E-mail:
[email protected] Chun-yu CHENG, Meng-lin SHENG, Zong-min YU, Wen-xuan ZHANG, An-qi LI, College of Electronic Science and Technology, Dalian University of Technology, Dalian, China
Unauthenticated Download Date | 12/31/17 6:01 PM
Application Development of 3D Gesture Recognition
631
Based on the above analysis, this paper combines the WPF application framework of Microsoft and the Unity development platform with Intel real sense technology respectively. A new real sense application is finally realized based on the gesture recognition and tracking module. There are two visualization projects realized: 1) a 3D gesture recognition system based on WPF framework which can make different responses to different gestures; 2) a simple 3D visual gesture tracking system based on Unity3D platform; 3) A final system by which users can use their hands to control the 3D models to hit balls. It’s a perfect fusion of 1) and 2).
2 The design idea and the overall design framework Figure 1 shows the overall design diagram and Figure 2 shows the structure of the system. First, a new project integrated Intel real sense SDK examples is created. The SDK API layer is invoked to realize a simple gesture recognition application without the constraint of the samples from the SDK itself; Secondly, a 3D scene and threedimensional objects based on the Unity development platform are created; Finally, Intel real sense technology is connect with the objects to realize movement control using gestures. The Ping-Pong interactive applications is realized: Users can control the two rackets in the scene. The ball bounces back and forth in the scene. Once it touches something, it will go to the opposite. If one of the rackets cannot catch the ball, the other will get a point. Unity project creation WPF project creation
Connect VS with unity
Add the SDK library reference
Create objects and scenes
Create the user interface
Configure the physical properties
Create utility classes
Write the script files Import real sense SDK reference
WPF, unity3D and Intel real sense technology integration
Figure 1. The overall design diagram.
Unauthenticated Download Date | 12/31/17 6:01 PM
632
Application Development of 3D Gesture Recognition
Application
C# Interface
Unity Interface
SDK Interfaces SDK Core Module Management Pipeline Execution Interoperability
I/O Module
Algorithm Module
Figure 2. The structure of the system
3 Software development environment 3.1 Intel F200 Real Sense Camera The basic principle of Intel F200 real sense camera is structure light. The structure light from an infrared launcher will be accepted by the infrared sensor after the reflection from the object. Because the distance between the infrared equipment and the surface of the object is different, the position and shape of the “structured light” which the receiver catches changes. According to them, the space information of the object can be calculated [12].
3.2 Maintaining the Integrity of the Specifications 3.2.1 Intel real sense SDK Intel real sense SDK is an Intel official development suite. It contains a variety of functions such as hand tracking and recognition, facial analysis, speech recognition, augmented reality, background segmentation and so on. Intel Real Sense SDK is the only bridge from hardware layer to software layer. All the interface functions in this paper are from the SDK. 3.2.2 Microsoft visual studio 2015 Visual Studio is the most popular integrated development environment of Windows applications. The source code in the project routine from Intel Real Sense SDK can be changed with the Microsoft Visual Studio development platform to obtain the data flow needed. The design based on Windows Presentation Foundation (WPF) is also based on Microsoft Visual Studio development platform.
Unauthenticated Download Date | 12/31/17 6:01 PM
Application Development of 3D Gesture Recognition
633
3.2.3 Unity3D Unity3D developed by Unity Technologies is a multi-platform integrated game development tool. It allows players to create such as 3D video games, visual architectures, real-time 3D animations and other types of interactive content easily. It is a fully integrated professional game engine. It supports different programming languages.
4 The design and implementation of 3D gesture recognition application based on WPF and Intel real sense technology Most of the SDK C # example applications are based on the Windows Forms user interface model, this article explores another user interface developing method with WPF. It is focused on the fusion of Intel real sense technology and the WPF application framework. Creating utility classes is the most important process. 4.1 The Intel Real Sense SDK API Initialization In Intel real sense SDK framework, all cameras and data flow are managed by PXCMCaptureManager. The interface classes PXCMSenseManager and PXCMCaptureManager are used in this paper. Figure 3 is a chart of invoking SDK and the camera to initialize the environment. Initializes the multimodal pipeline Create SenseManager instances
Activate function modules in the corresponding pipelines
Using QueryHand model to create and return the Hand model module instances
Create gestures configuration module instances And open the corresponding function
Figure 3. The chart of invoking SDK and the camera to initialize the environment
Unauthenticated Download Date | 12/31/17 6:01 PM
634
Application Development of 3D Gesture Recognition
4.2 Add the SDK Reference There are two kinds of dynamic link library (DLL) needed when an Intel real sense application is created with C# language: –– libpxcclr. Cs. DLL - managed C # interface DLL –– libpxccpp2c. DLL - unmanaged C + + P/Invoke DLL In Intel real sense SDK working reference manual, there are two methods given to add the required Intel real sense SDK library (DLL) supports for the project. One kind is creating reference for the necessary external DLL of the projects. It is usually finished during the installation of the Intel real sense SDK within the scope of the system. The other kind is adding the required DLL to the project and creating local reference. This article uses the first one. The advantage of this approach is that the project will be able to update to the latest version of the DLL when the SDK upgrades to the new version. The disadvantage is that if the SDK installation path changes or the built target platforms is different, reference may need to be modified.
4.3 Create the User Interface This application can demonstrate video flow and gesture recognition from the color camera. It uses three kinds of WPF control: –– Image control used for hosting video –– When users wave hands in front of the camera, the “refueling” label can be displayed on the screen –– The Stack Panel container containing other control
4.4 Create a Utility Class First, a ConvertBitmap.Cs utility class is created. It is used to implement bitmap conversion of color video stream. This step can realized by right-clicking on the project name in the solution explorer, selecting “add item”, selecting “class” and clicking OK. Then gesture detection, obtaining color images from the cameras and so on are realized in the MainWindow. Xaml. Cs class.
5 The design and Implementation of the gesture tracking application based on Unity3D and Intel real sense technology It is very important to give laws of physics to objects in the game. It can enhance the playability and Authenticity of the game. Similar to the development of 3D
Unauthenticated Download Date | 12/31/17 6:01 PM
Application Development of 3D Gesture Recognition
635
gesture recognition application with WPF framework, the development process of gesture tracking application based on Unity platform can also be understood as a basic development process of the three dimensional scene application based on Unity platform. “Write the script file” is the most critical process for the entire application. It is where the Intel real sense technology and the Unity can perfectly fuse.
5.1 Create 3D Objects In a Table tennis game, some essential game objects like the racket, Table tennis, the spring back wall, the console Table and so on are needed. In this article, however, it is not entirely in accordance with the Table tennis’s performance in real life to create the object. It is simplified and only the racket, balls, and the wall are kept. This is because the emphasis of Unity3D development lies in how to make use of Intel real sense SDK API to design a new and interesting game.
Figure 4. The Unity 3D scene view
5.2 The Perfect Fusion of Visual Studio, Unity3D and Intel Real Sense Technology It is the most important to edit the object script file of the “racket”. The Intel real sense camera captures the hand data. The data is expressed in detail in 3D coordinates and passed to the Y direction of the velocity vector of the “racket”. That means the relevant interface classes in Intel real sense SDK and the camera need to be initialized, and object instances need to be created. Then this article creates a HandModel interface class instance, use the instance to invoke a member function, and output the hand model data to the objects of the HandData interface class. By invoking the HandData member function, hand palm data can be extracted and used. Above is the basic process of the core module of the Unity and Intel real sense technology fusion.
Unauthenticated Download Date | 12/31/17 6:01 PM
636
Application Development of 3D Gesture Recognition
5.2.1 The fusion of Visual Studio and Unity3D The project codes are based on the C # programming language. The default crossingplatform IDE Unity provides is MonoDevelop. However, cooperative working of the Unity and Microsoft Visual Studio is more welcome. In order to do this, Edit>Preferences needs to be selected from the drop-down menu. After opening the Unity Preferences dialog, the left External Tools need to be select. After clicking MonoDevelop in the ComboBox, and the Browse button should be used to find the Visual Studio.Exe file. 5.2.2 Intel real sense SDK with Unity Unity 3D supports third-party plug-ins. It creates such a plug-in for the Intel real sense SDK. For integrating Intel real sense SDK, SDK DLL file must be added to the project. The external package and function libraries are imported into Unity by looking for UnityToolkit.Unitypackage file in the Intel real sense SDK installation directory. Only the plug-in and the plug-ins management directory need to be selected in the pop-up dialog window. If it is based on 64-bit computer game development, DLL files need to be replaced manually. However, in the latest version of Intel real sense SDK, only the x86_64 plug-in management needs to be imported.
5.3 Figures and Tables Final system effect is shown in Figure 5. Users can control two rackets by gestures to make the ball go to the opposite after touching the racket in the scene. If the person miss the ball, the other get one point. The success of the application development of 3D gesture recognition and tracking confirms the success of the collaboration and fusion between the Intel real sense technology and traditional PC programs.
Figure 5. Final system effect
Unauthenticated Download Date | 12/31/17 6:01 PM
Application Development of 3D Gesture Recognition
637
6 Conclusion This paper is based on the Intel real sense technology and the F200 camera. Intel real sense SDK development platform is combined with WPF application framework and Unity3D successfully. C# programming language is used to realize the 3D gesture recognition and tracking applications “Table Tennis Interactive”. From the invoking of the SDK API to the connection between the application layer, every detail is practiced. Hope the attempt to combine Intel real sense technology with some other development platforms can bring inspiration to the readers. Acknowledgment: This work is supported by the National Natural Science Foundation of China (No. 61003175), “the Fundamental Research Funds for the Central Universities” of China and the Liaoning Provincial Natural Science Foundation of China (No. 201601523).
References [1] H.B. Li, L.J. Ding, Y. Wu, and G.Y. Ran, “Static three-dimensional gesture recognition based on kinect skeleton data,” Computer Applications and Software, Sep. 2015, pp. 161-165. [2] L. Zhang, “3D Hand Gesture Recognition,” North China university of technology, 2013. [3] Y. Wang and Q.Z. Zhang, “Gesture Recognition based on kinect depth information,” Journal of Beijing Information Science & Technology University (Natural Science Edition), Jan. 2013, pp. 22-26. [4] R. Deng, L.L. Zhou, and R.D. Ying, “Gesture extraction and recognition research based on kinect depth data,” Application Research of Computers, Apr. 2013, pp. 1263-1265. [5] H.L. Zhang, “The research and development of real-time vision- based gesture recognition system,” Anhui University, 2013. [6] Gupta L, Ma S, “Gesture-based interaction and communication: automated classification of hand gesture contours,” IEEE Transactions on Systems Man & Cybernetics Part C, Jan. 2001, pp. 114-120. [7] Y. Lv, “The research of the gesture interaction system based on cameras,” Qingdao University, 2009. [8] T. Bian and W.M. Guo, “Development and research of force sensing touch gestures in tennis game scenarios,” Beijing University of Post Telecommunication Software Engineering, 2011. [9] F.F. Zeng, Y. Su, and J. Chen, “A new interactive input technology, 3D gesture recognition,” Journal of East China Shipbuilding Institute, Jun. 2000, pp. 38-44. [10] C.Q. Zhang and X. Song, “A real-time gesture recognition based applications development framewok,” Microcomputer and its application, Sep. 2011, pp. 9-11. [11] S.S. Guo, “Combination gesture recognition and application based on Kinect,” Liaoning University, 2015. [12] Y. Zhou, K. Liu, J. Gao, K.E. Barner and F. Kiamilev, “High-speed Structured Light Scanning System and 3D Gestural Point Cloud Recognition,” Conference on Information Sciences and Systems, 2013, pp. 1-6.
Unauthenticated Download Date | 12/31/17 6:01 PM
Unauthenticated Download Date | 12/31/17 6:01 PM
