16th IEEE International Conference on Robot & Human Interactive Communication
WP-28
August 26 ~ 29, 2007 / Jeju, Korea
Development and Performance Evaluation of a Neural Signal-based Assistive Computer Interface Changmok Choi1, Hyonyoung Han2,Chunwoo Kim3, and Jung Kim4 1,2,4
School of Mechanical, Aerospace & Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Korea, e-mail: (1axlguitar, 2hhn98, 4jungkim)@kaist.ac.kr 3 School of Mechanical and Aerospace Engineering, Seoul National University, Seoul,Korea, e-mail:
[email protected]
Abstract—This paper presents the development and performance evaluation of a human-computer interface that enables a limb-disabled person to access a computer via neural signals. For this purpose, electromyogram (EMG) signals were extracted from four muscles on the lower arm, and signal statistics (namely the mean and variance) were used for a filtering process. Six patterns were then classified through the application of a supervised multilayer neural network trained by a backpropagation algorithm. To extract the user’s intentions, such as cursor movements and a clicking, the authors applied the neural network in the classification of the six patterns. In addition, an on-screen keyboard was developed so that letters of the Roman and Korean alphabets could be keyed into the computer. Finally, to confirm the appropriation of the developed computer interface, the authors applied Fitts’ law in an experimental study to evaluate the performance of the computer interface. The experimental results show that the computer interface had an index of performance, or bandwidth, of 1.299. However, although the developed EMG-based human-computer interface had a lower index of performance than a mouse, it provides an alternative means of computer access for those with a disabled upper limb.
I.
INTRODUCTION
Many individuals have suffered motor disabilities as a result of spinal cord injuries or amputations. The Ministry of Health and Welfare in South Korea estimated in 2005 that in Korea alone approximately one million people suffered this type of disability, and that the number has been steadily increasing since 1995. These people have great difficulty using a computer, which is a fundamental tool in modern life. Many researchers have developed alternative interfaces to enable people with an upper limb disability to access computers. Recently, the use of neural signals has been attracting attention with respect to extracting data on the user’s intention because these signals can provide information related to body motion faster than other means (such as kinematic and dynamic interfaces). Notably, a variety of methods has been developed for the execution of a user’s intention: these methods are based on the central nervous system and the peripheral nervous system. At the level of the central nervous system, signals from brain activities have the potential for revealing human thoughts. The electroencephalogram (EEG) is a noninvasive monitoring method of recording brain activities on the scalp [1]. However, signals acquired via this method represent the massed activities of many cortical neurons; they also provide a low spatial resolution and a low signal-to-noise ratio (SNR). Invasive monitoring methods, on the other hand,
978-1-4244-1635-6/07/$25.00
2007 IEEE.
capture the activities of individual cortical neurons in the brain [2]. However, many fundamental neurobiological questions and technical difficulties need to be solved [3], and extensive training is generally required for interface methods based on brain activities [4]. Despite these challenges, researches in this area have the potential to help people with severe motor disabilities (such as loss of skeletal muscle control from below the shoulders). Signals at the level of the peripheral nervous system, which result in body motion, can be detected by an electroneurogram (ENG) [5] and an electromyogram (EMG) [6]. However, ENG-based interfaces have limitations with respect to the SNR, dimensions, and drifts: that is, damage to the neural tissue [7] and continued differential motion of the electrode within the fascicle cause a reduction in the SNR and a gradual drift in the recorded nerve fiber population [8]. EMG signals, on the other hand, can be measured more conveniently and safely than other neural signals. Furthermore, this noninvasive monitoring method produces a good SNR. Hence, an EMG-based human-computer interface (HCI) is the most practical with current technology. This paper presents an EMG-based computer interface so that people with upper limb disabilities, such as hand amputees or quadriplegics (with a C7 or C8 functional level), can access a computer without standard interfacing devices (such as a mouse or a keyboard). Using the developed computer interface, a user can move a cursor, click a button, or spell words on the computer through the alternative means of wrist movement. Additionally, to confirm the appropriation of the developed computer interface, an experimental study was conducted to evaluate the performance of the interface using Fitts’ law. Although some researchers have presented similar computer interface schemes [9, 10], they focused on the implementation of the interface and offered no quantitative evaluation of the performance of their computer interfaces. II. FRAMEWORK OF AN EMG-BASED COMPUTER INTERFACE
Fig. 1. Block diagram of an EMG-based computer interface
944
A. System Design Fig. 1 shows a block diagram of the developed computer interface. Low-pass filtering process was implemented, because the measured EMG signals were contaminated by noise. From the filtered signals, different body motions were discriminated by using an artificial neural network. Furthermore, from those results, the user’s intention was estimated in mapping discriminated motions of the user’s wrist. Thus, clicks and cursor movements could be made (horizontally and vertically) on the computer from the user’s intention. The interface system was developed with Microsoft Visual C++ 6.0. B. Myoelectric Site Selection The ability to use a computer via alternative access depends on how well the user can control the computer in a natural and intuitive manner. For this criterion, four different wrist movements (namely radial deviation, ulnar deviation, wrist extension, and wrist flexion) were chosen by which the user’s intention could be expressed, and these movements were mapped to cursor movement commands (namely left, right, up, and down). The user can intuitively control the cursor through these movements because the direction of the wrist movement corresponds to the direction of the cursor movement. For the clicking of a mouse button and stop commands, the motion of finger extension and resting were selected, respectively. Then, for each muscle contraction, four muscles were selected by palpation: the flexor carpi ulnaris (FCU), the extensor carpi radialis (ECR), the extensor carpi ulnaris (ECU), and the abductor pollicis longus (APL) [11]. Because quadriplegic patients (with a C7 and C8 functional level) can still weakly activate these muscles [12], the selection of the muscles as myoelectric commands is plausible. Fig. 2 shows where the electrodes for observing the EMG signals are located on the skin of the lower arm C. Data Collection and Signal Processing To collect EMG signals, four active surface electrodes (DE-2.1, Delsys) and a data acquisition board (PCI 6034e, National InstrumentTM) were used. Each signal was sampled at 1 kHz and amplified 1000 times. It is well established that the EMG signal can be modeled as a zero mean Gaussian process [13, 14]. Thus, the following equation is used to easily estimate the variance of the signal for feature extraction and low-pass filtering:
N
σˆ 2 =
∑ (M
−M)
2
i
(1) N −1 where M i is the magnitude of the ith signal, N is the number i =1
of EMG data sets in a window, and M is the mean of the magnitude of the N signal data, respectively. The function form of variance is analogous to a moving average filter except for a square term and a denominator. A squaring process increases the degree of the difference between the activation state and the inactivation state of the muscle. Therefore, the use of variance to extract the EMG feature is more effective than using only a moving average. Since the function of variance is similar to the function of a moving average filter, the cut-off frequency of the low-pass filtering used here can be defined in relation to a moving average filter as follows:
fc =
fs 2N
(2)
where fs is the sampling frequency. Whenever a large number of data sets are used, the effectiveness of the low-pass filtering is increased substantially. However, this increase introduces a delay in the estimation of the user’s intention. Hence, there is a tradeoff between the real-time signaling and the effectiveness of low-pass filtering. Currently, there is no standard related to the issue of perceivable delay, though Englehart et al. [15], Ajiboye et al. [16], and Soares et al. [17] have defined a perceivable delay as 300 ms, 100 ms, and 100 ms, respectively. In our signal processing, the number of EMG data sets in a window for low-pass filtering was defined as 100. Thus, the process not only provides effective low-pass filtering (fc = 5 Hz) but also prevents any significant delay. D. Pattern Recognition Fig. 3 depicts the structure of a supervised multilayer neural network that can be used to recognize the user’s intention. For the training stage, a backpropagation algorithm was used with the following neural network training parameters: • learning rate: 1.2 • momentum: 0.8.
Fig. 3. Architecture for the pattern classification Fig. 2. Myoelectric sites for the extraction of EMG signals
945
TABLE I Target vectors to classify user's intention Class of movement Stop Left Right Up Down Click
Desired network’s response 1 0 0 0 0 0
0 1 0 0 0 0
0 0 1 0 0 0
0 0 0 1 0 0
0 0 0 0 1 0
0 0 0 0 0 1
Table I shows the target vectors that are used to discriminate classes of intentions from the EMG signals. A maximum selector is located at the end of the network for the purpose of selecting the most activated neuron at the output layer. The user’s intention can therefore be identified by matching the most activated neuron to a class of movement. E. On-screen Keyboard Fig. 4 depicts the developed on-screen keyboard that enables a user with an upper limb disability to enter Roman and Korean letters on a computer. The size of this interface is 6.2 cm × 10.6 cm, and each button on the interface is a 1.5 cm × 1.2 cm rectangle. There are 15 buttons on the interface, and they represent the letters ‘a’ to ‘z’, ‘SPACE’ (Æ), ‘ERASE’ (Å), and so on. This interface is inspired by the interface system of the Samsung mobile phone [18]. Movement of the cursor is restricted discretely only on each button so that the user can easily use the interface. III. PERFORMANCE EVALUATION For the performance evaluation, a Fitts’ law test [19] was designed. Fitts’ law is a quantitative model for evaluating the effectiveness of a computer pointing device (see the review in [20]). This evaluation was conducted in two sessions: one session was for the use of the developed computer interface and one was for the use of a mouse. For this evaluation, five subjects with intact limbs (5 males, 26.4 years of age) volunteered. For the experimental test, a testbed was designed shown in Fig. 5. In the task, the subjects were asked to point and to click on a target (a dark rectangle) by moving a cursor, and time (movement time, MT) was measured taken to complete
Fig. 5. Snapshot of the testbed
the task. The difficulty of the task depends on the width of the target (W) and the distance (D) between the cursor and the target. To mathematically express the difficulty, an index of difficulty (ID) was used [21, 22], ID is expressed in “bits” as follows: ⎛D ⎞ (3) ID = log 2 ⎜ + 1⎟ ⎝W ⎠ Thus, the task becomes more difficult as D increases and W decreases. In this experiment, three different widths (W = 30, 70, 110 pixels) and three different distances (D = 150, 300, 450 pixels) were selected in line with Pino et al. [23], who evaluated the performance of a commercial assistive pointing device called BrainfingersTM (Brain Actuated Technologies [24]), which is based on Fitts’ law. According to Fitts’ law, the MT and the ID have the following linear relation: (4) MT = a + b ⋅ ID In this form, a reciprocal number of b is in “bits/s” and is called the index of performance (IP) or bandwidth. The IP represents how quickly the pointing and clicking can be done with a computer pointing device. The position of the target in this experiment was randomly assigned for each session so that a user could not expect it. The cursor was positioned on the right side or the left side of a target in accordance with the ID of each session. At the beginning of the experiment, each subject was instructed to click a dummy target and then to click nine targets with different IDs. The duration of the pointing and clicking at each session was measured, and this process was repeated as 20 times per subject IV. EXPERIMENTAL RESULTS A. Pattern Recognition Test and Analysis
Fig. 4. Word interface
As shown in Fig. 6, the EMG signals were measured from the muscles (FCU, ECR, ECU, and APL) in the lower arm of the subjects as the subjects executed different wrist motions during a 20 s period. Low-pass filtering was then applied to these signals to reduce the effect of noise, and a supervised multilayer neural network to classify the user’s intended motion. Fig. 7 shows to extent to which each neuron is activated by the changing wrist movements, as designed by the target vectors (in Table I). On the basis of these results, the maximum selector picks out the highly activated neuron at
946
TABLE II Success rate of the proposed classification method for the discrimination of subjects' intentions S1
Fig. 6. EMG signals from four muscles (FCU, ECR, ECU, and APL)
the output layer. Fig. 8 shows the results of the pattern classification: the red solid line denotes the intended motion of a subject and the green crosses show the recognized results in numerical values (0-5). Table II shows the success rate of the pattern classification: all the patterns are classified above 96 %. As shown in Fig. 8, the misclassifications mostly occur during the transition phase of the wrist motion. B. Performance Evaluation Test and Analysis Table III summarizes the performance data of the experimental test for the five subjects; each movement time is averaged from the 20 experimental data sets for each subject. Fig. 9 shows that the experimental data of the MT and the ID for a subject (S1) have a linear relationship in accordance with Fitts’ law. Fig. 9(a) shows the results of the mouse and
Fig. 7. Activation level of neurons at the output layer
S2
S3
S4
S5
Overall
Stop (%)
97.45
97.68
99.47
98.08
97.18
97.97
Left (%)
97.28
90.47
99.47
96.76
97.76
96.95
Right (%)
98.47
98.55
96.27
94.73
94.49
96.50
Up (%)
99.48
94.22
99.94
96.12
91.28
96.21
Down (%)
99.69
95.94
99.53
99.83
93.35
97.67
Click (%)
99.47
92.33
96.75
99.74
97.18
97.10
Fig. 9(b) shows the results of the developed computer interface. From these results, the IP was calculated in Table IV; the overall IP of our interface was 1.299 bits/s, whereas the overall IP of the mouse was 7.733 bits/s. In the literature, the IP of a commercial assistive computer pointing device that is based on the results of EMG and EEG is 0.386 bits/s [23]. Thus, when viewed solely in terms of the IP values, our computer interface system cannot compete with a mouse but is comparable to the commercial assistive computer pointing device. V.
DISCUSSION
Several commercial assistive computer pointing devices have been developed but the usability of these devices has seldom been evaluated. Exceptionally, BrainfingersTM (Brain Actuated Technologies), which is based on EMG and EEG, was evaluated by employing a Fitts’ law [23]. However, its efficiency in terms of the IP is approximately 20 times less than that of a mouse. In contrast, the IP of our interface is three times greater than that of this commercial assistive computer pointing device, which means that our interface can help reduce the gap in efficiency between a mouse and an assistive pointing device. In spite of this positive development, our interface is still inferior to a mouse in terms of efficiency. The lack of efficiency is due to the constant velocity of the cursor, which can be problematic if the cursor is far from the target. In addition, the cursor movement is restricted to only four directions (in horizontal and vertical movements). Hence, a more intelligent technique is required to produce cursor movement with an adjustable velocity and a wider scope of direction from EMG signals. Furthermore, all subjects reported fatigue in their lower arm during the experimental test because of the strong wrist motions required to discriminate classes of motions within the noise effect. This fatigue implies that our computer interface is implausible for long-use applications. The EMG signal measurement techniques should therefore be further developed to capture small body motions. This aspect is of particular importance in the development of an HCI for limb-disabled persons because the signals of disabled people are weaker than the signals of people with intact limbs. VI. CONCLUSION
Fig. 8 Results of pattern classification by the maximum selector
This paper reports an EMG-based computer interface that
947
TABLE III Experimental results of the Fitts’ law test for the efficiency of a mouse and pointing devices based on an EMG interface D (pixels)
W (pixels)
Movement time (MT)
ID (bits) 1.2410 1.8981
Subject 1
Subject 2
Subject 3
Subject 4
Subject 5
EMG
mouse
EMG
mouse
EMG
mouse
EMG
mouse
EMG
mouse
1920.5 2770.3
715.9 773.2
1520.8 2443.3
762.7 857.4
1925.6 2693.5
695.8 737.7
1729.0 2466.2
724.9 859.6
1664.5 2480.5
728.2 787.9
150 300
110 110
450
110
2.3479
3437.1
830.0
3175.2
910.5
3251.5
818.4
3479.9
915.8
3578.3
898.2
150
70
1.6521
2073.0
750.4
1680.1
831.4
1881.8
774.1
1717.0
870.5
1762.6
920.1
300
70
2.4021
2825.9
848.3
2536.2
914.2
2557.5
827.6
2614.4
960.0
2656.6
939.6
450
70
2.8931
3763.1
864.1
3186.5
954.2
3353.9
908.2
3338.1
971.0
3630.9
932.9
150
30
2.5850
2694.6
837.7
2366.0
982.2
2007.4
920.8
2721.4
1096.5
2407.1
923.5
300
30
3.4594
3626.5
925.5
3421.5
1081.4
2624.7
957.3
3087.1
1078.0
3255.8
1012.6
450
30
4.0000
3928.0
1015.8
4008.5
1151.0
4221.9
1063.9
3813.3
1215.0
4276.2
1157.0
Fig. 9. Relation between the movement time (MT) and the index of difficulty (ID) from the experiment on a subject (S1): (a) the developed computer interface and (b) a mouse. The dotted line in (a) represents the graph in (b), which is depicted for comparison. The gentle slope of the line illustrates the high IP value.
provides a scheme for controlling a cursor and entering text on a computer. To extract the user’s intention, EMG signals were acquired from four muscle sites in the lower arm; the signals are produced by wrist movements. After processing the signals with low-pass filtering, six classes of wrist movements were discriminated by employing a multilayer neural network trained by a backpropagation algorithm. To enable users to enter Roman and Korean letters on the computer, the on-screen keyboard was designed, which resembles the keypad of a mobile phone. Finally, a Fitts’ law test was conducted to evaluate the efficiency of our computer interface as used by five volunteers. The results were compared with the performance of a mouse and another commercial assistive computer pointing device. In terms of TABLE IV Experimental results of efficiency of pointing devices from Fitts’ law test using an EMG-based interface and a mouse. S1 EMG mouse
IP (bits/s) R2 IP (bits/s) R2
S2
S3
S4
S5
the IP values, our EMG-based computer interface system cannot compete with the mouse. However, this EMG-based HCI provides an alternative means for people with upper limb disabilities to access a computer without standard computer interface devices (such as a mouse or keyboard). A usability test was conducted for the cursor control method but not for the developed on-screen keyboard. In the literature, Cheng et al. [4] designed and evaluated a virtual telephone keypad based on EEG signals. Thus, the evaluation approach can be used with our on-screen keyboard so that the efficiency can then be quantitatively compared with other interfaces. Moreover, a possible extension of our interface methodology can be used to control various platforms, such as bionic robot systems for people with limb disabilities (for example disabilities involving exoskeletons and limb prostheses) and teleoperated robotic systems that can carry out the tasks of humans in hazardous environments.
Overall
ACKNOWLEDGMENTS This work was supported by the Korea Science and Engineering Foundation(KOSEF) grant funded by the Korea government(MOST) (No. R01-2007-000-11659-0).
1.340
1.145
1.503
1.376
1.129
1.299
0.795
0.863
0.554
0.736
0.752
0.740
9.600
7.238
7.674
6.343
7.811
7.733
0.979
0.975
0.940
0.868
0.824
0.917
948
REFERENCES [1]
[2]
[3] [4]
[5]
[6]
[7]
[8]
[9] [10]
[11] [12] [13] [14] [15] [16]
[17]
[18] [19]
[20] [21]
[22]
J. D. Millan, F. Renkens, J. Mourino, and W. Gerstner, "Noninvasive brain-actuated control of a mobile robot by human EEG," IEEE Transactions on Biomedical Engineering, vol. 51, pp. 1026-1033, 2004. J. Wessberg, C. R. Stambaugh, J. D. Kralik, P. D. Beck, M. Laubach, J. K. Chapin, J. Kim, J. Biggs, M. A. Srinivasan, and M. A. L. Nicolelis, "Real-time prediction of hand trajectory by ensembles of cortical neurons in primates," Nature, vol. 408, pp. 361-365, 2000. M. A. L. Nicolelis, "Actions from thoughts," Nature, vol. 409, pp. 403-407, 2001. M. Cheng, X. R. Gao, S. G. Gao, and D. F. Xu, "Design and implementation of a brain-computer interface with high transfer rates," Ieee Transactions on Biomedical Engineering, vol. 49, pp. 1181-1186, 2002. E. Cavallaro, S. Micera, P. Dario, W. Jensen, and T. Sinkjaer, "On the intersubject generalization ability in extracting kinematic information from afferent nervous signals," IEEE Transactions on Biomedical Engineering, vol. 50, pp. 1063-1073, 2003. J. U. Chu, I. Moon, and M. S. Mun, "A real-time EMG pattern recognition system based on linear-nonlinear feature projection for a multifunction myoelectric hand," IEEE Transactions on Biomedical Engineering, vol. 53, pp. 2232-2239, 2006. S. Bossi, S. Micera, A. Menciassi, L. Beccai, K. P. Hoffmann, K. P. Koch, and P. Dario, "On the Actuation of Thin Film Longitudinal Intrafascicular Electrodes," in Proceedings in The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006, pp. 383-388. S. M. Lawrence, G. S. Dhillon, W. Jensen, K. Yoshida, and K. W. Horch, "Acute peripheral nerve recording characteristics of polymer-based longitudinal intrafascicular electrodes," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 12, pp. 345-348, 2004. H. Jeong and J. H. Choi, "An EMG-controlled Graphic Interface Considering Wearability," in Proceedings of the Human-Computer Interaction INTERACT' 03, 2003, pp. 958-961. O. Fukuda, T. Tsuji, and M. Kaneko, "An EMG controlled pointing device using a neural network," in Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, vol. 4, 1999, pp. 63-68. A. O. Perotto, anatomical guide for the electromyographer, 3rd ed. IL: Charles C. Thomas, 2005. L. Weiss, J. K. Silver, and J. Weiss, Easy EMG: Butterworth-Heinemann, 2004. E. Shwedyk, R. Balasubramanian, and R. N. Scott, "A Nonstationary Model for the Electromyogram," IEEE Transactions on Biomedical Engineering, vol. 24, pp. 417-424, 1977. L. Dharashwar, "Cross Talk in Myoelectric Control Systems," vol. M.Sc.E. Fredericton, Canada: University of New Brunswick, 1967. K. Englehart and B. Hudgins, "A robust, real-time control scheme for multifunction myoelectric control," IEEE Transactions on Biomedical Engineering, vol. 50, pp. 848-854, 2003. A. B. Ajiboye and R. F. Weir, "A heuristic fuzzy logic approach to EMG pattern recognition for multifunctional prosthesis control," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 13, pp. 280-291, 2005. A. Soares, A. Andrade, E. Lamounier, and R. Carrijo, "The development of a virtual Myoelectric prosthesis controlled by an EMG pattern recognition system based on neural networks," Journal of Intelligent Information Systems, vol. 21, pp. 127-141, 2003. http://www.samsung.com. P. M. Fitts, "The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement (Reprinted from Journal Experimental-Psychology, Vol 47, Pg 381-391, 1954)," Journal of Experimental Psychology-General, vol. 121, pp. 262-269, 1992. I. S. MacKenzie, "Movement time prediction in human-computer interfaces," in Proceedings of Graphics Interface 1992, pp. 140-150. J. Accot and S. Zhai, "Beyond Fitts' law: models for trajectory-based HCI tasks," in Proceedings of the SIGCHI conference on Human factors in computing systems. Atlanta, Georgia, U.S., 1997, pp. 295-302. I. S. MacKenzie, "Fitts' law as a performance model in human-computer interaction," Doctoral dissertation. Toronto, Ontario, Canada: University of Toronto, 1991.
[23] A. Pino, E. Kalogeros, E. Salemis, and G. Kouroupetroglou, "Brain Computer Interface Cursor Measures for Motion-impaired and Able-bodied Users," in Proceddings of HCI International 2003: The 10th International conference on Human-Computer Interaction,. Crete, Greece, 2003, pp. 1462-1466. [24] www.brainfingers.com.
949
