Design of Human-Computer Interaction Control System Based on Hand-Gesture Recognition Wang Zhi-heng,Cao Jiang-tao
Liu Jin-guo,Zhao Zi-qi
School of Information and Control Engineering Liaoning Shihua University Liaoning FuShun, China E-mail:
[email protected]
State Key Laboratory of Robotics Shenyang Institute of Automation Chinese Academy of Science Liaoning Shenyang,China operator controls the space detection robot in real-time through Human-Computer Interaction technology to ensure that the detection robot can successfully complete the task. HumanComputer Interaction technology has been widely concerned by the researchers and developed rapidly, there are a variety of mature Human-Computer Interaction have appeared. Speech recognition and gesture recognition are the two main direction of the Human-Computer Interaction development. Speech recognition system which was exploited by Department of Engineering, University of Cambridge widely used in speech recognition, speech synthesis ,character recognition and other fields[4]. CyberGloveII-18sensor data glove was exploited by CyberGlove Systems company of USA[5]. The action of hands and fingers is accurately converted into digital real-time data via measures up to 22 joint angles by data glove sensors to achieve the purpose of gesture recognition through virtual system.
Abstract—A gesture recognition based Human-Computer Interaction control system is developed via LabVIEW in this paper. Furthermore, to solve the existing problems of lower precision and poor real-time ability in gesture recognition algorithm, an improved PSO-SVM classification algorithm of hand-gesture recognition is proposed. Firstly, the gesture sample data is collected by using five bending sensor of data glove, then, in order to improve the recognized precision, the data collected is preprocessed and the optimize the SVM kernel parameter value is found by using the improved PSO algorithm. Finally, the recognized hand-gesture is divided into numbered 1-11 control status values, and sent to the sample robot controller by wireless transmission module and achieving the realization of sampling robot motion control. The simulation results illustrate the effectiveness of designed method on Keywords—Human-Computer Interaction control system; sample robot; gesture recognition; improved PSO-SVM algorithm
As the gesture recognition compared to voice recognition has the advantages of a rich amount of information and easy to achieve, etc, it is widely used and an important part in HumanComputer Interaction system[6].In this paper, we used the Woodpecker bionic pecking sampling robot principle machine (hereinafter referred to as sampling robot) of State Key Laboratory of Robotics in Shenyang Institute of Automation Chinese Academy of Science for the control platform to validate the experimental algorithm. The main research contents were as follows: First of all, the operator gestures were collected in real time, and then used the improved PSOSVM classification algorithm in the upper computer to classify the collected data, so that achieved the operator's gesture recognition and matching. Finally, according to the result of gestures to match an order in the Human-Computer Interaction system and issued instructions to complete the real-time control of the sampling robot. System flow chart is shown in Figure 1.
. I.
INTRODUCTION
With the researchers probed the extraterrestrial planets deeply in the recent years, study on surface sample of planets had been became the important part to detect the planet. Currently, the extraterrestrial planets detected and sampled by researchers were mainly concentrated on the moon and on Mars, And the space exploration robot was used to analysis the resources and sample soil of the planets. The former Soviet Union's Luna 16-month Milano[1] was the first one achieved to sample the Mare Fecunditatis and sent back to earth's detectors automatically by human. NASA had launched "courage" and "opportunity" Mars patrol detector to Mars that completed the task of collecting and grinding rocks in 2003[2]. The woodpecker bionic pecking based sampling robot principle machine was invented by State Key Laboratory of Robotics in Shenyang Institute of Automation Chinese Academy of Science, which used the bionics and near-perfect biological institutions, applied the bio-shock absorption characteristics of the woodpecker into sampling system[3]. A new bionic - based rock sampling technique was proposed through studied the mechanism of woodpecker fast impact without concussion.
Human - Computer Interaction System Send the collected data
Operator
Wireless transmission module
Processing instructions
Send control instructions
USB wireless transmission module
Sampling robot
Data processing of MATLAB node
There are some problems just like the complex and unknown space operating environment when study the sample of extraterrestrial planet surface. While human operators have better judgment and insight than space detection robots, so the
978-1-5386-2901-7/17/$31.00 ©2017 IEEE
Data gloves
Upper monitor
Fig. 1. Composition block diagram of the Human-Computer Interaction System
143
II.
it satisfied the design requirements. The structure of the gesture recognition system is shown in Fig.3.
SYSTEM OVERALL DESIGN
A. Data glove In order to enable the sampling robot to operate stability according to the human mind, at the same time to facilitate the operator through the Human-Computer Interaction system monitoring function and feel the sampling robot movement of the surrounding environment and state, this paper used data glove as the input of the Human-Computer Interaction system.
The establishment of gesture prediction model
Match gesture classification
Collect the bending sensor data
Operate the sampling robot Build gesture templates
Match the target gesture
Data normalization
According to the structural design of the sampling robot, 11 kinds of gestures were defined to control the motion state of each joint of the sampling robot. Specific gesture corresponding to the control state shown in Figure 2. Experiments shown that the use of data glove, not only in the input of the operator's gestures for real-time data collection, but also increased the flexibility and freedom of the operator to control the sampling robot.
The improved PSO algorithm is used to optimize the kernel function parameters
Obtain the corresponding motion control classification number value
Gesture training set template
Reasonable SVM Model
SVM Classification Forecasting Model
Acquire sensor realtime gesture values
Gesture data is classified accurately
Fig. 3. Structure diagram of gesture recognition
III.
DESIGN OF GESTURE RECOGNITION ALGORITHM
A. Improved PSO algorithm PSO (Particle Swarm Optimization) was developed by J. Kennedy and R.C. Eberhart in 1995[10]. A simulation derived from a simplified social model, The main idea was to imitate the behavior of birds foraging, through the study found that birds in the flight process would often suddenly change the direction of flight, scatter and collect. Their behavior was unknown, but the overall total consistent, each individual also maintained the most appropriate distance. Each member of the group continually changed the search pattern by learning from itself and the experience of other members. Assuming that the search space was D dimensional , the total number of particles was N. The position of the i-th particle was expressed as Pi=(Pi1,Pi2,…PiD ); The i-th particle flight position change rate (ie "speed") was expressed as Vi=(Vi1,Vi2,...ViD); The optimal position of the ith particle in the flight history (ie, the individual optimal value of the particle) was expressed as pBesti=(Pi1,Pi2,…PiD); The optimal position of the current population for all particles in flight the global optimal value is expressed as gBest (The value is the optimal value for all pBesti). The position and velocity of each particle in the population were changed according to equation (1).
Fig. 2. The schematic view of 11 kinds gesture control state
B. Gesture recognition design In order to ensure the real-time and accuracy of the control system, and could accurately identify the gesture in the process of controlling the sampling robot. The SVM classification algorithm was used in the gesture recognition process. The classification algorithm was widely used for data classification[7-9]. Its main idea was to establish a classification hyperplane as a decision surface, so that the isolation edge between the positive and negative examples was maximized. The SVM theory is based on statistical learning theory, more precisely, it is the approximation of structural risk minimization. However, there are some unavoidable defects in the classification algorithm, and there are still some difficulties in solving the multi-classification problem. Aiming at the problems of traditional SVM classification, this paper presented a SVM classification method based on improved PSO algorithm. The improved PSO algorithm was applied to the system of gesture recognition by optimizing the kernel function of SVM, the parameters of the optimized parameters were introduced into SVM prediction function. After training on the obtained sample data with the test set prediction template to derive the classification model. The completed classification model had a good recognition ability in gesture real-time recognition, and achieved a certain recognition rate,
(1)
Where c1 and c2 were accelerating factors and two positive real numbers known as cognitive learning rate and social learning rate, usually the value of c1 = c2 = 2, rand () was a random number independent of [0,1]. Because the basic particle swarm algorithm has some shortcomings, it was easy to fall into the local optimal at the end of the iteration, and the global search performance could not get the global optimal solution quickly, therefore, on the basis of PSO added an inertia weight ω[11-12], which used to coordinate the local and global search capabilities of the algorithm. The standard particle swarm algorithm is shown in equation (2).
144
Griewank fitness change curve
0
10
improve PSO PSO
(2)
Z =Zstart
Zstart Zend uT Tmax
Fitness log value
The value of ω was linearly decreasing the inertia weight formula.
漏3漐 -2
Tmax and T represent the maximum number of iterations and the current number of iterations; ωstar and ωend represented the initial iteration weight and the termination iteration weight, respectively, ωstart=0.9, ωend=0.4. For the acceleration coefficient was too large or too small, it was not conducive to particle swarm optimization[12], under normal circumstances the relationship between c1 and c2 was c1 + c2> 4. When c1 = 0 the particles did not have cognitive ability, but social experience, the convergence speed became faster and easy to fall into the local optimal. When c2 = 0 the particles did not have social experience, algorithm accuracy was reduced. In the standard PSO algorithm, under normal circumstances c1 = c2 = 2. However, a large number of experiments show that the algorithm in the early search convergence speed became faster and easy to fall into the local optimal, resulting in precocious phenomenon, so the acceleration coefficient to be limited to the range of values, the algorithm was enough convergence. The choice of acceleration coefficient is related to the global optimization ability of the algorithm. Taking into account the acceleration coefficient of the strategy that c1 from large to small, c2 from small to large. The strategy formula as (4)shown[13].
c1 c2
-1
10
10
0
10
20
30
40 50 60 Iteration times
70
80
90
100
Ackly fitness curve improve PSO PSO
0.6
Fitness log value
10
0.5
10
0.4
10
0.3
10
0
10
20
30
40 50 60 Iteration times
70
80
90
100
Rastrigin fitness curve
3
10
Fitness log value
improve PSO PSO
T c1i Tmax T (c2 f c2i ) c2i Tmax
2
10
(c1 f c1i )
1
10
c1i,c1f,c2i and c2f are constants, the optimal range of c1 and c2 was found by simulating the values. c1 known by the experience in the range [2.5,0.5] interval, c2 was in the range of [0.5,2.5], through the study of acceleration coefficient in the range of adjustment found, when c1 = 2.5, c2 = 1 (ie c1f = 1, c2f = 2.5, c2i = 1, c2f = 2.5) had good convergence accuracy and optimization ability.
0
10
20
30
40 50 60 Iteration times
70
80
90
100
Shpere fitness curve
Fitness log value
improve PSO PSO
In order to verified the ability to improve the PSO algorithm to overcome premature convergence. Aiming at the selection of four test functions, the iterative optimization process of improved PSO algorithm and standard PSO algorithm was given respectively. The population size was 40, the dimension is 10, Tmax=100,other parameters in the PSO algorithm were defined as: c1=c2=2,ω=1.2. In the improved PSO algorithm, c1 and c2 were obtained by formula(4), ω was calculated by the linear decreasing formula(3), the results shown as below:
0
10
0
10
20
30
40 50 60 Iteration times
70
80
90
100
Fig.4. The curve of four kinds test function corresponding to the fitness function
From the two particle swarm algorithm in the figure, we can see the optimization curve of four test functions, the fitness value is in a state of decline in the whole iterative process of the improved PSO algorithm. It is shown that the improved PSO algorithm can effectively improve the premature
145
convergence of PSO algorithm which can overcome the shortcomings of PSO algorithm into local optimal. B. Gesture recognition implementation of Improved PSOSVM classification algorithm First, the gestures of the operator were collected, and the activity value of each gesture could be uniquely determined. The same gesture collected 20 times, so that each gesture would have 20 different sample description. Sample collection contained a total of 11 kinds of gestures samples, each sample consisted of five data members, thus constituting a characteristic of the sample of 220 × 5 matrix. In the original data in the training collection and test collection was extracted, and each sample data was labeled category label. Wherein 120 samples belonged to the first category (category label is 1), 21-40 samples belonged to the second category (category label is 2)and so on, every 20 samples for a class. Twenty samples of each category were divided into two groups, the first 15 data was used as training set (train_data), and the last five data was used as test set (test_data). The training and test set of data were normalized . The SVM kernel function chosen in this paper was the radial basis function (RBF), and the kernel function is shown in equation (5). K ( xi x j )
Fig.5. The upper monitor Interface of Human-Computer Interaction System
B. Experiment and analysis Based on the above design content of the experiment, first set up Human-Computer Interaction system experimental platform. The interactive platform of Human-Computer Interaction system was composed of three parts: sampling robot, PC and data glove.
2
exp( J xi , x j ), J ! 0
(5)
In the case of training the data with the kernel function, the penalty factor c and the radial basis parameter g, which had a great influence on the classification accuracy of the SVM algorithm. In this paper, the traditional PSO algorithm and the improved PSO algorithm are used to determine the parameters. The penalty parameter c = 1.5 and the radial basis parameter g = 0.7 are obtained by the PSO algorithm . After the parameters are optimized by the improved PSO algorithm, the parameter c = 0.1 and the radial basis parameter g = 0.01. IV.
Fig.6. Experimental platform of Human-Computer Interaction System
In order to verify that the improved PSO-SVM classification algorithm is more accurate than the traditional PSO-SVM classification algorithm in gesture recognition and matching process, this paper uses these two methods to process the collected gesture data respectively in the experiment process. Test for all 11 kinds of gestures test match, by the parameter optimization understood through PSO optimization algorithm to obtain parameter values c = 1.5, g =0.7,improved PSO optimization algorithm to obtain parameter values c = 0.1, g = 0.01. The two sets of parameters are introduced into the SVM classification algorithm, gesture matching and running in the upper computer interface. The gesture classification results of the two algorithms are shown in TABLE I. It is proved that the gesture recognition technique based on the improved PSO-SVM algorithm to the HumanComputer Interaction system can not only improve the control performance of the sampling robot, but also improve the Human-Computer Interaction performance of the sampling robot system, which makes the operator have a significant increase in the degree of freedom and accuracy of the control robot.
EXPERIMENT AND ANALYSIS
A. HCI upper computer system design Because of the gesture data classification, completed the establishment of the gesture template, in order to meet the real-time matching gesture, we need to design upper computer system for data processing. The upper computer interface consisted of three parts, as shown in Figure 5.
TABLE I. Gesture number
control commands
1
GESTURE RECOGNITION RATE OF PSO-SVM ALGORITHM AND IMPROVED PSO-SVM ALGORITHM Testing times
Advance
20
Correctly identify (times)
Unrecognized
Correct rate (%)
PSO-SVM algorithm
Improved PSO-SVM algorithm
PSO-SVM algorithm
Improved PSO-SVM algorithm
PSO-SVM algorithm
Improved PSO-SVM algorithm
12
20
8
0
60
100
146
2
Back
20
14
19
6
1
70
95
3
Left
20
17
20
3
0
85
100
4
Right
20
17
20
3
0
85
100
5
20
13
17
7
3
65
85
20
14
18
6
2
70
90
7
Turntable left Turntable right Head bow
20
16
18
4
2
80
90
8
Head up
20
17
17
3
3
85
85
6
9
Head stretch
20
13
20
7
0
65
100
10
Head back
20
11
20
9
0
55
100
11
Stop
20
12
20
8
0
60
100
[7]
V. DISCUSSION AND CONCLUSIONS This paper designed a Human-Computer Interaction system based on gesture recognition, which built a gesture template by collecting and normalized the operator's gesture data. In order to made the gesture template classified accurately, this paper proposes an improved PSO-SVM classification algorithm. Overcome the traditional PSO algorithm was easy to fall into the local optimal problem. Through the Human-Computer Interaction system, the correct recognition and matching gesture data were used to control the sampling robot in real- time. It could be seen from the experimental results that the improved PSO-SVM classification algorithm to the correct rate of recognition had been significantly improved, made the whole set of Human-Computer Interaction system in the sampling robot real-time control more convenient and efficient, thus it reduced the number of misuse effectively and improved the control accuracy.
[8]
[9]
[10]
[11]
[12]
[13]
ACKNOWLEDGMENT This work was funded in part by State Key Laboratory of Robotics Shenyang Institute of Automation Chinese Academy of Science and Liaoning Shihua University. The authors thank all the experts of State Key Laboratory of Robotics Shenyang Institute of Automation Chinese Academy of Science and Liaoning Shihua University. REFERENCES [1]
[2] [3]
[4]
[5]
[6]
Zheng Wei, Xu Hou Ze, Zhong Min, Liu Cheng Shu, Yuan Mei Juan. Progress in international lunar exploration programs[J]. Progress In Geophysics, 2012, 27(6): 2296-2307 Sima Hangren. The Popularity of Mars Exploration -- a glance at the famous Mars probe[J]. Space Exploration, 2009, (4):45-47 Sun Hongtao. Research on the Bionic Peck Institutions and Structures Based on Woodpecker[D]. Shenyang: Northeastern Univer-sity, 2015:16-17. K. Kumar, A. Jain, R. K. Aggarwal. A Hindi speech recognition system for connected words using HTK[J]. Int. J. Computational Systems Engineering, 2012, 1(1):25-32. Gao Longqin, Wang Aimin, Huang Weiyi, Dai Jinqiao. Interactive Model of Human Hand Based on 18-Sensor Data-Glove[J]. Chinese Journal Of Sensors and Actuators, 2007, 20(3):523-527 Liu Jinguo, Luo Yifan, Ju Zhaojie. An Interactive Astronaut-Robot System with Gesture Control[J]. Computational intelligence and neuroscience, 2016:1-11.
147
Hu Mingqing , Chen Yiqiang , James Tin-Yau Kwok. Building sparse multiple-kernel SVM classifiers[J]. IEEE Transactions on Neural Networks , 2009,20(5): 827-839. Mattia Marconcini, Gustavo Camps-Valls, Lorenzo Bruzzone. A composite semis-upervised SVM for classification of hyperspectr-al images[J]. IEEE Geoscience and Remote Sensing Letters, 200-9, 6(2): 234-238. You Changhuai , Lee Kong Aik , Li Haizhou . An SVM kernel with GMM-super vector based on the Bhattacharyya distance for speaker recognition[J]. IEEE Signal Processing Letters, 2009, 16(1): 49-52. Kennedy. J, Eberhart R. Particle swarm optimization[J]. Proceedi-ngs of the IEEE International Conference on Neural Networks, 1995:1942-1948. Y. Shi, R. Eberhart. A Modified Particle Swarm Optimizer [C]. Proceedings of the IEEE World Congress on Computation Intelligence, 1998:69-73. Y. Shi, R. Eberhart. Empirical Study of Particle Swarm Optimization[C], Proceedings of the IEEE Congress on Evolutionary Computation, 1999:1945-1950. A. Ratnaweera, S.K. Halgamuge, H.C. Watson. Self-organizing hierarchical particle swarm optimizer with time-varying accelerat-eon coefficients[J]. IEEE Transaction on Evolutionary Computati-on, 2004,8(3):240-255.
