Electromyography-based gesture recognition: Fuzzy classification evaluation Aleksandar Gogić, Nadica Miljković, and Đorđe Đurđević
Abstract—We designed electromyography–based system for online recognition of hand gestures. The system encompasses of dual-channel electromyography recordings, preprocessing techniques, and Fuzzy logic classification. Online performance of the system was evaluated on five able-bodied subjects. Forearm flexor and extensor muscles were measured in each subject. The obtained classification accuracies were compared to the commercial Myo Armband operation that uses 8 channel electromyography recordings from forearm muscles. The presented results showed that averaged classification accuracy of designed system was 94.6% and the commercial system had averaged classification accuracy of 82.8% in all subjects. These results indicate the potential application of the system in various human-computer interface applications. Index Terms — fuzzy logic, online electromyography, hand gesture recognition.
classification,
I. INTRODUCTION GESTURE recognition can be performed non-invasively by application of diverse sensing technology, and can be applied in various fields of human-computer interface (HCI). The HCI application is ranging from rehabilitation medicine to commercial applications for control of electronic devices (such as control systems for playing video games). Many HCI applications encompass manipulation of physical devices such as computer mouse, keyboards, and joysticks. Hands-free input techniques based on variety of sensing modalities have been explored. For example, one of the hands-free interactive systems is speech-recognition [1, 2]. Also the computer vision techniques allow machine to recognize faces, eye, and gesture motions [3]. However, these approaches have limitations, such as sensitivity to environmental factors such as noise and light. In this paper we designed hand gesture system, analyzed classification accuracy of gesture recognition, and tested its online performance. We hypothesized that arm muscle activity during specific grasp performance can be used in order to manipulate computer interface. Alongside HCI applications, hand gesture recognition has been thoroughly explored for design of neural prosthesis for amputees [4]. These systems record muscle contractions generated from residual limb of amputees and decode those signals by applying various gesture recognition strategies in order to provide adequate neural prosthesis control [4]. Aleksandar Gogić, dipl. eng. is master student at the University of Belgrade - School of Electrical Engineering, Bulevar kralja Aleksandra 73, 11000 Belgrade, Serbia (e-mail:
[email protected]). ass. prof. Nadica Miljković, PhD is with the University of Belgrade School of Electrical Engineering, Bulevar kralja Aleksandra 73, 11000 Belgrade, Serbia (e-mail:
[email protected]). ass. prof. Đorđe Đurđević, PhD is with the University of Belgrade School of Electrical Engineering, Bulevar kralja Aleksandra 73, 11000 Belgrade, Serbia (e-mail:
[email protected]).
Surface electromyography (sEMG) based gesture recognition systems have been tested during offline operation in process of learning characteristics of electromyography signal [5-7]. To test system's performance in a real-life environment, online training methods were introduced [8]. We applied Fuzzy logic classifier in order to recognize a defined set of hand gestures. Fuzzy logic is a form of manyvalued logic in which the truth values of variables can range between completely true and completely false [9]. Fuzzy logic includes 0 and 1 as extreme cases of truth, but also incorporates various states of truth in-between so that, for example, the results of a comparison between two things could be not “cold” or “hot”, but “0.33 of coldness” [10]. Operation of the presented system was evaluated in 5 able-bodied subjects during online gesture performance, and compared to commercial Myo Armband device [11]. Myo Armband is one of the commercially available hand gesture recognition devices designed for diverse applications that range from controlling characters in computer games to controlling devices in real time. Myo Armband is an 8 channel EMG sensor with 8 nineaxis inertial measurement units (IMU). One IMU unit contains three-axis gyroscope, three-axis accelerometer, and three-axis magnetometer [12], but their application is not relevant to the research presented in this paper, since we used only sEMG recordings. Myo Armband communicates with the controlled device via Bluetooth LE wireless protocol. It can distinguish 5 different hand gestures, and it can follow arm motion as well as rotation of forearm and hand rising. Users have feedback of performed gesture in terms of vibration of Myo Armband device [12].
II. METHODOLOGY A. Signal acquisition and preprocessing We tested system in five healthy subjects with no known history of neuro-muscular disorders. All subjects signed Informed Consent approved by the Local Ethics Committee. TABLE I CHARACTERISTICS OF A SUBJECTS SAMPLE
Characteristics
ID1
ID2
ID3
ID4
ID5
Age [years] Height [cm] Weight [kg] Dominant arm [R-right, L-left] Sex [M-Male, F-Female]
23 198 75 R
23 180 72 R
30 176 73 L
40 185 96 L
23 186 76 R
M
M
F
M
M
Proceedings of 3rd International Conference on Electrical, Electronic and Computing Engineering IcETRAN 2016, Zlatibor, Serbia, June 13 – 16, 2016, ISBN 978-86-7466-618-0
pp. MEI1.6.1-4
General data on healthy subject's characteristics is given in Table I. We recorded sEMG signals from two muscles. Surface Ag/AgCl electrodes KendalTM (Coviden, Massachusetts, USA) were placed on forearm muscles flexor digitorum profundus, and extensor digitorum, according to SENIAM (Surface Electro-Myography for Non-Invasive Assessment of Muscles) protocol [12]. The electrode locations on subject's forearm are presented in Fig. 1. For acquisition of sEMG signals, we used Biovision (Biovision, Wehrheim, Germany) preamplifiers with gain of 5000 and ADC (analog-to-digital converter) integrated with NI ELVIS II platform (National Instruments Inc., Austin, USA). NI ELVIS II is an integrated suite of 12 instruments with 16 single ended analog input channels, ADC resolution of 16 bits and maximum sample rate of 1.25 GHz. Sampling rate was set to 1 kHz. Custom made software designed in LabVIEW version 14.0 (National Instruments Inc., Austin, USA) was used for acquisition, signal processing and classification.
Membership functions used in this project were trapeziums. Points of trapeziums were specified by average of collected data during calibration process. We had two input variables in Fuzzy system: sEMG envelopes from two forearm muscles. For each input variable, there were two membership functions, and therefore we could distinguish four different hand gestures: relaxation, grasp, wave out, and fingers spread gestures (Fig. 2). We chose these particular gestures in order to directly compare the presented system with the commercially available device Myo Armband. The device can be trained with the software provided by the manufacturer to recognize these gestures. Fuzzy system outputs are integer values for each of four different membership functions. These numbers were calculated in Center of Area (CoA) defuzzification method. Defuzzification is the process of converting the degrees of membership of output variables into crisp numerical values.
Fig. 2. Membership function for Fuzzy logic classification. Top panel presents membership functions for first input variable. Red trapezium is membership function for “relaxation” gesture and blue one is for “grasp” gesture. Bottom panel presents membership functions for second input variable. Red trapezium is for “fingers spread” gesture and blue one is for “wave out” gesture. Abbreviation CH1 is for input variable (envelope) from channel 1, and abbreviation CH2 is for input variable (envelope) from channel 2.
Fig. 1.Location of two pairs of bipolar electrodes. Abbreviations are ref reference electrode, CH1 - electrodes location on flexor muscle (posterior view), and CH2 - electrodes location on extensor muscle (anterior view).
sEMG signals were filtered with notch filter (50 Hz) in order to reduce power noise, and with 3rd order band-pass Butterworth filter with cutoff frequencies at 10 Hz and 450 Hz. Root Mean Square (RMS) window width of 50 ms was applied on sEMG signal. We used 3rd order polynomial Savitzky-Golay filter with window length of 500 ms in order to perform signal smoothing and to calculate sEMG envelope. B. Fuzzy Logic and Classification Application of fuzzy logic in our system characterizes various sub-ranges of continuous variables. For sEMG envelopes we used separate membership functions. Each function maps the same envelope value to true value in the 0-1 range. These true values were used to determinate performed gesture.
Fuzzy controller first calculates the area under the scaled membership functions, and within the range of the output variable. The fuzzy logic controller then uses built-in equations to calculate the geometric center of this area. After defuzzification process we recorded output numerical values. Relative to those values, we distinguished different hand gestures: relaxation, grasp, wave out, and fingers spread gestures. C. System evaluation At the beginning of system evaluation, subjects were instructed to perform four specific hand gestures (tasks) in order to calibrate the membership functions of the fuzzy logic classifier. After calibration, the subjects were asked to perform 16 randomly ordered gestures (four repetitions for each task) in a session. All subjects performed three sessions. Classification accuracy was assessed by visual inspection: investigator compared performed gesture with recognized gesture during online operation, and expressed it in percents. For additional confirmation of classification accuracy, recognized gestures notification on computer screen output, and subject's performance were made with software
Screencast-O-Matic© by recording videos with 25 frames per second. All false positives and false negatives were registered. Classification accuracies were averaged for all sessions and results are presented with standard deviations. The proposed fuzzy logic classifier performance was compared with the performance of the commercial device Myo Armband. The device diagnostics software tool, supplied by the device manufacturer, provides the user with real-time visual feedback about the currently recognized gesture. The same previously described approach of accuracy assessing by visual inspection was used with Myo Armband as well.
Averaged classification accuracy in all subjects for designed system is 94.6%, and averaged classification accuracy for commercial Myo Armband system is 82.8% in all subjects. Averaged False Positive registrations of hand gestures are presented in Fig. 5. The averaged False Positive registrations of hand gestures for subjects ID1-5 are 18.8%, 32.2%, 27.4% 52.4%, and 31.3%, respectively.
III. RESULTS Fig. 3. shows processing steps applied on sEMG signals recorded from forearm flexor muscles in a healthy subject (ID1, during grasp task). Raw sEMG signal recorded from forearm flexor muscles is presented, filtered sEMG signal, envelope of sEMG signal after application of RMS filter sEMG signal, and envelope of sEMG signal after application of Savitzky-Golay filter are presented in Fig. 3 a), b), c), and d), respectively.
Fig. 3. Processing steps applied on forearm flexor digitorum profundus during grasp gesture in subject ID1: a) raw sEMG signal, b) filtered sEMG signal, c) sEMG envelope after application of RMS filter, and d) sEMG envelope after application of Savitzky-Golay filter.
In Fig. 4. averaged classification accuracies of designed system versus averaged classification accuracies of commercial Myo Armband system are presented. The standard deviations are presented in Fig 4. for classification accuracies for each subject through all three sessions. The averaged classification accuracies of designed system for subjects ID1-5 are 100%, 95.8%, 81.2%, 95.8%, and 100%, respectively. The averaged classification accuracies of Myo Armband system for subjects ID1-5 are 66.7%, 93.8%, 87.5%, 93.8%, and 82.8%, respectively.
Fig. 4. Averaged classification accuracies expresed in percents for all subjects during hand gesture recognition using designed Fuzzy system and, commercial Myo Armband system, with standard deviations.
Fig. 5. Averaged registered False Positive gestures expressed in percents for all subjects during hand gesture recognition using designed Fuzzy system, with standard deviations.
IV. DISCUSSION AND CONCLUSION The designed system had higher classification accuracies in 4 out of 5 cases, compared to commercial Myo Armband system (Fig. 4) for considered gestures. One of the problems that we have encountered during the evaluation of the designed system was registration of False Positive gestures presented in Fig. 5. We realized that False Positive gesture recognitions appeared in similar pattern: during the transition between “wave out”, and “fingers spread” gestures and in certain subjects occurred during “grasp” gesture, where also was registered “fingers spread” gesture. Possible solution, in order to decrease the number of False Positive gesture registrations, would be to remove “relaxation” indicator, so as further improvement of Fuzzy logic membership functions, and rules. As for Myo Armband device, we noticed that classification accuracy increases if subject performs “relaxation” gesture between two tasks. The second issue that we encountered during evaluation was response time of the designed system. Namely, the system response is delayed in relation to performed gesture for 50 ms due to RMS filter, plus 500 ms due to SavitzkyGolay filter. Also, additional pause of up to 500 ms was introduced in system response time in order to decrease False Positive gesture registrations. In Fig. 6. registered “wave out” gesture is presented (frame 369 up to frame 397). After that, from frame 397 up to frame 400 the False Positive registration of “relaxation” gesture appeared. From frame 400 up to frame 419 no gesture was registered, which corresponded to delay of designed system. Afterward, there was registration of “grasp” gesture. These results were taken from video record of first session of evaluation of designed system performed by subject ID1. Frame rate of video record was 25 frames per second (fps), which means that one frame duration is 40 ms. Converting frame scale from Fig. 6. to time scale, we can see that False Positive registration time was 120 ms, and delay time due to filters and pause time of the system was
760 ms. The longest pause of system response was during transition between “wave out” and “grasp” gesture, since the transition between these two movements was timeconsuming due to their diversity (extreme extension to extreme flexion transition).
[3]
[4]
[5] [6]
[7]
[8]
Fig. 6. Sketch of appearances of False Positive gesture registration and pause time of the designed system presented in frames.
Limitations of this study are: 1) the Fuzzy and Myo Armband were tested during separate sessions, and future evaluation should include its parallel evaluation, 2) system performance was tested in 5 subjects, and in future it should be tested in larger sample of able-bodied subjects, and 3) our system requires electrode placement by an expert (according to SENIAM protocol [13]) and future studies should include system evaluation of more robust application with electrode re-positioning. Future work, additionally, should include reduction of False Positives, and time delay reduction in relation to performed gesture. Also it would be interesting to assess the applicability, and effectiveness of the developed system in interactive virtual reality environments, where the subject communicates his intentions through body and arm gestures. In particular, it would be of special interest to develop a virtual reality system to help amputees learn manipulating prosthetic devices.
ACKNOWLEDGMENTS Authors would like to thank to all volunteers for participation in this study. Also, Authors would like to thank prof. Mirjana B. Popović from University of Belgrade - School of Electrical Engineering for her kind support in this study. Ass. prof. Nadica Miljković would like to acknowledge that the work on this project was partly supported by the Ministry of education, science, and technological development, Republic of Serbia, grant No. 175016. The work of ass. prof. Đorđe Đurđević was partially supported by projects TR32039 and TR32047 by the Ministry of education, science and technological development, Republic of Serbia. REFERENCES [1] [2]
V. Lakshmipathy, C. Schmandt, and N. Marmasse,”TalkBack: a conversational answering machine”, UIST ’03, 41-50, 2003. K. Lyons, C. Skeels, T. Starner, C. M. Snoeck, B. A. Wong & D. Ashbrook, “Augmenting conversations using dual-purpose speech”, UIST '04, 237-246, 2004.
[9] [10]
[11] [12] [13]
K.R. Wheeler, M.H. Chang & K.H. Knuth, “ Gesture-Based Control and EMG Decomposition”, IEEE Trans. on Systems, Man, and Cybernetics, 36(4), 2006. C. Cipriani, C. Antfolk, M. Controzzi, G. Lundborg, B. Rosen, M.C. Carozza, “Online myoelectric control of dexterous hand prosthesis by transradial amputees,” IEEE Transactions of Neural Systems and Rehabilitation, 19(3):260- 270, 2011. B. Hudgins, P. Parker, R.N. Scott, “A new strategy for multifunction myoelectric contro”l. IEEE Tran Biomed Eng, 40(1):82-94, 1993. K. Englehart, B. Hudgins,” A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans Biomed Eng, 50(7):848-854, 2003. K. Englehart, B. Hudgins, P.A. Parker, A wavelet-based continuous classification scheme for multifunction myoelectric control. IEEE Trans Biomed Eng, 48(3):302-311, 2001. D. Nishikawa., W. Yu, H. Yokoi, & Y. Kakazu, EMG prosthetic hand controller discriminating ten motions using real-time learning method. In Intelligent Robots and Systems, 1999. IROS'99. Proceedings.,1999 IEEE/RSJ International Conference on (Vol. 3, pp. 1592-1597). IEEE, 1999. V. Novak, I. Perfilieva and J. Mockor , “Mathematical Principles of Fuzzy Logic”, Kluwer Academic Publishers, 1999. Turajlić S., "Osnovni principi Fazi upravljanja procesima (Basic principles of Fuzzy process control)" Belgrade, 2014 (materials for Industrial Process Control course at University of Belgrade - School of Electrical Engineering, in Serbian, available online: http://automatika.etf.rs/images/FAJLOVI_srpski/predmeti/izborni_ku rsevi_os/upravljanje_procesima/OS4UIP/materijali/fazi_final.pdf). [online] https://www.myo.com/, retrieved on April 2016. [online] https://www.developer.thalmic.com/myo/techspecs, retrieved on April 2016. H.J. Hermens, B. Freriks, C. Disselhorst-Klug, G. Rau., “Development of recommendations for SEMG sensors and sensor placement procedures,” Journal of Electromyography and Kinesiology, 10:361- 374, 2000.
