Design and Implementation of a serious game based on Kinect Liu Qingtang1, WangYang2,Wu Linjing3, Huangjingxiu4 , WuPeng5 (School of Educational Information Technology, Central China Normal University, Hubei Wuhan 430079, China)
[email protected], 1284526718 @ qq.com
Abstract—Educational game is a certain computer games that can develop the user's knowledge, skills, intelligence, emotion, attitude, values. Kinect can listen to the voice of the people and observe people's action through the skeleton tracking technology and speech recognition technology which make it easier to communicate with computer. By analyzing the the key technology of the existing education games, we can find that the application on Kinect is rarely. This paper makes full use of the advantages of Kinect and the characteristics of the education game to design a serious game model based on Kinect.The working principle of Kinect and the definition of serious game is introduced. A case according to the educational game model based on Kinect is
and movements which is based on the most natural way to interact with the computer . The drive of PC on Kinect and the Microsoft Kinect for Windows SDK of programming are open source which could be used on C# ,C++ and VB, as well as other development language such as Java to attract more developers to develop on Kinect. However, at present, the development of small educational game is mostly based on Flash, Dreamweaver and other software in this type, or it is made into the phone APP. These educational games' interaction between computer and people is not natural enough.
implemented at last.
Keywords-educational game; Kinect motion-sensor technology; the skeleton tracking technology
I.
INTRODUCTION
With the popularity of computer in people's daily life, the traditional way of human-computer interaction based on the keyboard and mouse can not satisfy people's needs any more, more intelligent and friendly way of human-computer interaction is urgently needed now. In early 2008, Bill Gates, the former boss of Microsoft, presented the concept of Natural User Interface(NUI) which will bring new experience for people. Natural user interface refers to a user interface which is effectively and remains invisible as the user continuously learns increasingly complex interactions[1]. It is based on the users' most natural way of operation, such as actions, gestures, language and so on to operate the computer and get rid of the bondage of the mouse and keyboard at the same time.[2] Kinect is launched as a peripheral device when Microsoft released Xbox 360 in 2010. It operates with a natural user interface through language, gestures
While the application of Kinect motion-sensor technology is a lot now, for example, González-Ortega , Dí az-Pernas have studied a Kinect-based system for cognitive rehabilitation exercises monitoring which contributes to health care[3],Ke-Yu Lee and Jhih-Sian Jheng researched into the fabrication of the kinect remote-controlled cars and planned the motion interaction courses in Taiwan[4].In Greece, Mike van Diest and Stegenga proposed exergames, a kind of game used Kinect for capturing and analyzing whole body movement patterns[5].There are a lot of similar study now, but they are not aimed at education. It is obvious that the application in the field of educational game remain to be developed. The application of Kinect in serious game has its unique advantages though there are a lot of motion sensors.It can recognize human body movement to make learning become happy and rousing. What's more, the learning process can deepen students knowledge in the process of serious game, raise study efficiency and make the students love learning. Firstly the related technologies and principle of Kinectare introduced in the article, then it presents the design model of
serious game based on the motion-sensor deviceKinect and the process of design and implementation. At last, we end the article with the discussion and the conclusion of the study. II.
METHODS
space coordinates of bones and the color image coordinates. At last, the 20 joints' coordinates of the tracked object are determined and they can define a match human skeleton diagram which will be output on the screen[10].
A. The working principle of the motion-sensor device Kinect
Kinect empowers developers and businesses to create innovative natural human computing solutions across a variety of industries[6]. The skeleton tracking technology and NUI coordinate transformation are the main working principle of Kinect. Skeletal tracking technology is the core of the device which can recognize 20 key points of the body accurately and track them in real time[ 7 ].There are two depth cameras and one RGB camera in the device. The operation process is as follows to reach the transformation from depth image to the skeleton one. Firstly, a moving object in the vision should be recognized, focused and extracted from the background. Following that, infrared sensor will detect the depth image of the human body and classify different parts of the body through the pixel distribution in each layer and the characteristic values. Then given the overlapping parts of the human body, the processing system will analyze from the front, side and reverse side and identify key points according to all pixels and their information. Finally, the system can generate a skeleton diagram according to the 20 key points.[8]
Fig. 1. Space coordinate transformation of Kinect
B. Serious game The definition includes two parts, one is the game, the other is serious. As for game, there are many debates about it on the journals and conference. Different researchers in different fields and cultural backgrounds may hold different opinions about it , and it is not easy for them to understand each other. But if we sit together, we will find it is acceptable. Wittgenstein has said that is almost impossible to define game for it is changing with time, but we will get it if we see
NUI API is the most important part of Microsoft Kinect for Windows SDK which can be used to process the data flow of color image and the depth image as well as the skeletal tracking, it can also be used to control and manage the device of Kinect[ 9 ]. NUI coordinate transformation includes the space coordinates of depth image, bones and color image. The transformation among the coordinate system of them is shown in figure 1. The function MapDepthToSkeletonPoint, MapSkeletonPointToDepth and MapDepthToColorImagePoint as well as MapSkeletonPointToColor can be used to achieve the coordinates mapping among the depth image coordinates,
it [11]. It is not easy for every one to understand why the educational game can be called serious game. The original intention is to make it acceptable by the traditional education workers to combine it with education. The increasing application of serious game in daily teaching confirmed its success. As mentioned above, different scholars may hold different ideas about the definition of serious game. But the definition of serious game is nothing more than an idea, it is the outside force which promotes the combination of learners and knowledge. Kalapanidas proposes that serious game is aimed at motivating and educating the players in immersive environment to help them towards knowledge[ 12 ]. He
emphasized the function of building environment. Crookall
the reflection of the personality of the player. The players'
thinks that game is a tool for communication and education
common features such as their common hobbies and
which transforms complex and serious knowledge into a
concerns should be taken into account during the design of
acceptable one by inspiring the interest of the learners[13].It
the game plot. A popular game must be designed according
is obvious that Crookall lays more emphasis on the
to the players' common characteristics. Of course, it is wrong
psychological motivation function[14].
to pursue the attractiveness of the game story but ignore the
C. The designing model of serious game based on the motion-sensor device Kinect
main purpose of serious game, education. The depth of knowledge in each level is different, and it needs to be designed according to the gamers' knowledge background. It
Serious game is aimed at education. A reasonable serious
is necessary to do a pre-test before designing the game story .
game design needs to follow its own design model. This study proposes a serious game design model based on Kinect
3)
motion-sensor technology on the basis of the previous
suitable
for
physical
The best recognition distance of Kinect is 2-3 meters[17]
research[15]. The figure 2 shows the design model to provide a reference for the designers of serious game [16].
Movement interaction operation control
which should be taken into account during movement interaction design of serious game based on Kinect to take advantage
of
the
Kinect
motion-sensor
technology.
Therefore, the design of the movement interaction needs to be as easy as possible. For example, left arm waving can be set as a click of the left button of mouse. Right arm waving can be set as a click of the right button of mouse. But during the algorithm design, it is necessary to take the misoperation and
the
simplicity
of
behavior
identification
into
consideration.
4) Fig. 2. Serious game design model based on Kinect motion-sensor technology
1)
A game target agrees with educational goal
The game motivation which can be stimulate and maintain The main reason why game can let the player ecstatic is
The goal of serious game needs to accord with
its stimulative and maintainable motivation. It can keep a
educational objective, for its main aim is directing the player
player in a continuous excited state for the desire of
to knowledge. It is required that educational goal be clear
achieving the goals. The maintenance of the motivation in
and reasonable which will be helpful to game designers to
the player mainly depends on the combination of internal and
confirm the educational target. It is necessary to make the
external factors. The educational function of the serious
player understand the goal at their first entrance to the game
game is the internal maintenance of motivation to help the
and know what kind of knowledge and skill can they acquire
player learn something they are interested in. The beautiful
through the game. The player can choose the appropriate
interface, shocking sound, absorbing barriers and increasing
serious game for themselves. At the same time, the goal in
grades in the game are external incentives. The combination
the serious game should be clear and be shown to the game
of internal and external factors of the game has great effect
player to let them understand how to finish the task of the
on the maintenance of motivation undoubtedly which
game and what can they acquire after finishing it.
contributes to the player's learning in return. The main
2)
Reasonable game story Reasonable game story is mainly manifested in two
aspects: one is consistency of the game target and the other is
motivation in the game includes scoring, drawing a lottery and getting the equipment, etc. These are like constantly temptation presented to players to stimulate learners' motivation constantly and help them keep a good condition
to meet challenges.
5)
The game rules abide by the players The success of a educational game requires reasonable
and widely accepted rules. Only in the same rules, can it make sense. No rules, no accomplishment. The rule is more important in realizing fair competition in such a virtual
shapes' color to attract their attention. Players get points by crashing into corresponding shapes within the allotted time. If successful, there is the sound of praise. This educational game is aimed at helping the players deepen the cognition of the colors and shapes . B. The design phase
network environment. Of course, the rules must be
After the decision of educational target during analysis
reasonable and acceptable to most people. There must be
phase. Design phase mainly includes the game story design,
penalties to those who do not follow the rules of the game
rules design, motion interaction design and motivation
such as deduction of points and not refusal of passing the
inspiring mechanism design. Game designers need to follow
barrier in order to strengthen the awareness of the rules. Only
the guidance of teaching design ideas according to the
when all the players abide by the rules of the game, can the
educational game model based on the Kinect motion-sensor
game be meaningful and proceed smoothly[18].
technologies.[ 19 ]The goal of this serious game is to
In addition to meeting the points mentioned above, a successful serious game based on Kinect motion-sensor technology should design the game goal, game story, interactive actions, and game motivation according to the
strengthen the cognition of colors and shapes of the players and let them get rid of the sub-health habits at the computer all day. At the same time, it can encourage the players to do some exercises and improve their learning efficiency.
teaching design idea. At the same time, a successful serious
Tom wants to buy a book he likes, but he doesn't have
game should regard the rules of the game as a common
enough money, he need to gain some gold by his own
constraints of players and the Kinect motion-sensor
wisdom such as contacting the corresponding color of shapes
technology as technical support. III.
THE PROCESS OF DESIGN AND IMPLEMENTATION
A. The analysis phase The main task of analysis phase is to choose teaching content and teaching organization form based on the idea of teaching design. The main purpose is to establish the teaching goal according to the characteristics of the teaching object. This stage mainly includes requirement analysis, students and the teaching analysis. Before the educational game design and development, it is necessary to determine its users and their needs. The purpose of demand analysis is to determine the educational game's applicable objects and understand the needs of the users according to the literature review, network multimedia course at home and abroad. The main applicable objects of the game are primary school students in grade two or three. The students of this age are mainly thinking in images, active, tending to bright colors and playing games, interested in role playing and others' praise. According to these characteristics of them, the bright green is set as the background, bright yellow is used as
Fig. 3. Case screenshot
of with body. There will be the game rules on the screen before the game. After entering, the player will play the role of Tom and collide the corresponding shapes according to the prompt. If successful, it will automatically reward score accordingly until the time is over. There will be accompany during the game to enhance the enjoyment. The rules of the game is easy to understand when the plot is determined. During the game, it is not allowed to go out of the road, if he did, the game ended. The player can use the speech control "Restart" to start a new game but the score will be reset. The stay time of the shape in each level is
different, the farther back, the faster the speed, the shorter
based on the motion-sensor device Kinect. It is hoped that
time the shape stay[20].
the serious game can be integrated into the daily courses of
The applicable object of the educational game is students of Grade two or three. Therefore, its interactive action is simple and easy to understand and master. Jump represents on the scooter, the voice "Start" will control the start of the game. The body tilt will control the direction of the scooter through the avatar, Tom. Leaning forward will speed up and leaning back will slow down the scooter as shown in figure 3 named case screenshot. The game will end automatically if time over.
the students, but it will be better to have a post-test after the session of the game. It can collect the information of the game's effects such as the interactivity and the immersion as well as the willingness of replay. These will contribute to the improvement of the game. At the same time, it is promising to collect and store the course and results of the learners to analyze and evaluate their learning skills which will be significant to guide them learning effectively. B. Conclusion and future work Just like the announce of Microsoft Kinect, ''You are the
The game motivation mechanism is simple and clear. Only by colliding the corresponding shape with his own body accurately within the allotted time, can the players get points which can exchange for gold to achieve Tom's dream.
controller!'' Kinect has changed the traditional way of human-computer interaction and has deepen into the sports, healthcare and entertainment industries. Though there are some applications of Kinect motion-sensor technology in
C. The phase of development
education, compared with other industries, it is far from
The development phase is the implementation of
enough. We hope this research can encourage a wider
educational game after completing the design. The main
adoption of serious game based on Kinect and inspire related
software
researchers to achieve greater breakthrough.
includes
Kinect
for
Windows
SDK,VisualStudio2010, 3D MAX, PhotoShop, etc. C# is ACKNOWLEDGMENT
used as the development language. Microsoft Speech SDK is invoked to implement the speech control. 3D MAX and
We would like to thank the anonymous reviewers for
PhotoShop are used for the background design. VisualStudio
their comments which contribute to the study. This work was
is the development platform.
funded by the National Science and Technology support Program, Key Technology and Terminal development of
D. The phase of evaluation Evaluation is an important element to improve serious game in the process of game design. The evaluation content mainly includes educational goal and the applicable objects, etc. The determination of educational goal is according to the
low-cost
experiential
rural
information
service
(2015BAK03B03) and National Science and Technology plan project, Key Technology Research and Demonstration of Tujia music culture digital protection and display (2012BAD35B02).
demand analysis. And the game interface should be designed on the basis of users' characters[21]. The game story is easy and full of fun which will make players learn through play.
REFERENCES
At the same time, the game is in accord with the active qualities of the children and can train them mentally and physically. IV.
DISCUSSION AND CONCLUSION
[1]
[2]
A. Discussion of study on serious game based on Kinect This study mainly focused on the design and implementation of a serious game based on Kinect, it is designed on the basis of the designing model of serious game
[3]
Boulos M N K, Blanchard B J, Walker C, et al. Web GIS in practice X: a Microsoft Kinect natural user interface for Google Earth navigation[J]. International journal of health geographics, 2011, 10(1): 45. Galna B, Barry G, Jackson D, et al. Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson's disease[J]. Gait & posture, 2014, 39(4): 1062-1068. González-Ortega D, Díaz-Pernas F J, Martínez-Zarzuela M, et al. A Kinect-based system for cognitive rehabilitation exercises monitoring[J]. Computer methods and programs in biomedicine, 2014, 113(2): 620-631.
[4]
[5]
[6] [7] [8] [9] [10]
[11] [12]
[13] [14]
Lee K Y, Jheng J S, Chen S C, et al. Fabrication of the Kinect Remote-controlled Cars and Planning of the Motion Interaction Courses[J]. Procedia-Social and Behavioral Sciences, 2015, 174: 3102-3107. Mike van Diest, Jan Stegenga, Heinrich J. W02rtche, et al. Suitability of Kinect for measuring whole body movement patterns during exergaming[J]. Journal of Biomechanics, 2014, 47:2925–2932. Zhang Z. Microsoft kinect sensor and its effect[J]. MultiMedia, IEEE, 2012, 19(2): 4-10. Boutsika E. Kinect in Education: A Proposal for Children with Autism[J]. Procedia Computer Science, 2014, 27: 123-129. Khoshelham K. Accuracy analysis of kinect depth data[C]//ISPRS workshop laser scanning. 2011, 38(5): W12. Smisek J, Jancosek M, Pajdla T. 3D with Kinect[M]//Consumer Depth Cameras for Computer Vision. Springer London, 2013: 3-25. Xia L, Chen C C, Aggarwal J K. Human detection using depth information by kinect[C]//Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on. IEEE, 2011: 15-22. Pondy L R. Leadership is a language game[J]. Readings in managerial psychology, 1989: 224-233. Kalapanidas E, Fernandez-Aranda F, Jimenez-Murcia S, et al. PlayMancer: games for health with accessibility in mind[J]. Communications & Strategies, 2009 (73): 105. Crookall D. Serious games, debriefing, and simulation/gaming as a discipline[J]. Simulation & Gaming, 2010, 41(6): 898-920. Boendermaker W J, Prins P J M, Wiers R W. Cognitive Bias Modification for adolescents with substance use problems–Can serious games help?[J]. Journal of behavior therapy and experimental psychiatry, 2015.
[15] Guardia D L, Gentile M, Grande V D, et al. A Game based Learning Model for Entrepreneurship Education ☆ [J]. Procedia - Social and Behavioral Sciences, 2014, 141:195–199. [16] Carvalho M B, Bellotti F, Berta R, et al. A case study on Service-Oriented Architecture for Serious Games[J]. Entertainment Computing, 2015, 6: 1-10. [17] Tian Y, Meng X, Tao D, et al. Upper limb motion tracking with the integration of IMU and Kinect[J]. Neurocomputing, 2015, 159: 207-218. [18] Schulze J, Martin R, Finger A, et al. Design, implementation and test of a serious online game for exploring complex relationships of sustainable land management and human well-being[J]. Environmental Modelling & Software, 2015, 65: 58-66. [19] Liarokapis F, Debattista K, Vourvopoulos A, et al. Comparing interaction techniques for serious games through brain–computer interfaces: A user perception evaluation study[J]. Entertainment Computing, 2014, 5(4): 391-399. [20] Carvalho M B, Bellotti F, Berta R, et al. An activity theory-based model for serious games analysis and conceptual design[J]. Computers & Education, 2015. [21] Alaribe I. Design a Serious Game to Teach Teenagers with Intellectual Disabilities How to Use Public Transportation[J]. Procedia-Social and Behavioral Sciences, 2015, 176: 840-845.
