Classification of tetraplegics through automatic ... - Health Advance

Medical Engineering & Physics 21 (1999) 313–327 www.elsevier.com/locate/medengphy

Classification of tetraplegics through automatic movement evaluation Radmila Maksimovic a

a,*

, Mirjana Popovic

b

Faculty of Electrical Engineering, 11000 Belgrade, Yugoslavia Institute for Medical Research, 11000 Belgrade, Yugoslavia

b

Received 16 October 1998; received in revised form 1 June 1999; accepted 7 July 1999

Abstract The general problem of classification of functional movements in humans with spinal cord injuries requires the following questions to be answered: what are the essential kinematic parameters that we have to observe during the movement? Is it possible to estimate preserved motor skills based on kinematics? Which computational method for identification is suited to geometric feature analysis? To answer these questions we have developed the methodology which has two phases: (1) recordings of a series of specified arm movements; and (2) custom made software for graphical presentation of arm movements and the design of wavelet and neural networks for movement classification. The proposed protocol is automated and both graphical presentation and neural networks allow easy interpretation of the instrumented assessment to accomplish automatic classification of arm movements in tetraplegics. The protocol was evaluated on 16 spinal cord injury (SCI) patients and seven healthy control subjects for three different arm movements. The classification rate yielded results in the range 46–100% for movement trials that were tested. The application of neural networks for classification of arm movements is completed with results using different neural networks: backpropagation, radial basis, recurent (Elman), self-organizing and Learning Vector Quantization (LVQ).  1999 IPEM. Published by Elsevier Science Ltd. All rights reserved. Keywords: Joint angles; Trajectories; Wavelet network; Neural networks; Classification; Medical engineering

1. Introduction For natural arm movements, a placement of the wrist is the result of different combinations of joint rotations as well as different combinations of muscle activities [1]. Four rotations: shoulder flexion/extension and abduction/adduction, elbow flexion/extension and humeral rotation determine reaching. Another three degrees of freedom (pronation/supination, internal/external wrist rotation and wrist flexion/extension) determine the orientation of the hand, but not the reach itself. Despite the different strategies, all those degrees of freedom are in ample use in normal functional reaching movements. Spinal cord injury (SCI) at cervical levels, called tetraplegia, leaves many humans totally reliant on other people or devices for even the simplest tasks that are

* Corresponding author. Tel.: +381-11-3192-175. E-mail address: [email protected] (R. Maksimovic)

normally taken for granted. Neurologists define as a rule functional status [2]: a subject who has suffered an SCI at a C4 cervical level generally has control of his/her scapula but has lost control of the entire arm; a C5 lesion leaves a human with limited shoulder control and elbow flexion, but the control of elbow extensor, wrist and hand are lost. In humans with a C5 lesion, sensations are lost from below the forearm. Humans with C6 level damage have minimal wrist extension, along with some sensation extending distally to the hand. In subjects with an injury at the C7 level, some control of fingers along with some sensation in the hand is intact. In practice, each subject shows a different profile of lost functions and sensations, and recovers in his/her way. To facilitate communication among professionals who treat these patients, a classification based on the segmental enervation of individual muscles of the upper limb was developed at an International Conference in Edinburgh, Scotland, in 1978 [3]. This classification was later modified to its present form of nine groups in 1984 [4]. International Classification takes into account the motor groups that are functioning.

1350-4533/99/$ - see front matter.  1999 IPEM. Published by Elsevier Science Ltd. All rights reserved. PII: S 1 3 5 0 - 4 5 3 3 ( 9 9 ) 0 0 0 5 6 - 9

314

R. Maksimovic, M. Popovic / Medical Engineering & Physics 21 (1999) 313–327

In the past decade, the classification system for neurological level and motor level recommended by ASIA, [5] has become widely accepted. The greatest potential for improvement of quality of life lies in rehabilitation and maximal restoration of upper extremity function. Design of an assistive system for humans with tetraplegia [6,7:162–90] requires detailed biomechanical considerations of individual properties such as: (1) range of voluntary movements; (2) range of passive movements; (3) response to external activation of muscles that are paralysed due to SCI; (4) intellectual ability to use an assistive system and finally, (5) one’s knowledge about reaching abilities of the user. A careful evaluation of motor functions is needed for effective candidate selection. Both Zancolli and Moberg [8,9] made significant contributions to the patient classification due to motor abilities. Zancolli classified patients on the basis of residual motor function present in the upper extremity and Moberg emphasized the importance of evaluation of sensation, in addition to motor function. Manual muscle testing of all upper extremity muscles is important because successful hand function depends on precise placement of the hand as well as functioning of the hand. Also, the range of motion at all upper extremity joints has to be assessed. Joints must have sufficient motion to allow for functional use of the extremity with an assistive device. This study searches for new approaches to classify reaching abilities of SCI patients based on Artificial Neural Networks (ANN). The approximations of general continuous functions by nonlinear networks is useful for system modeling and identification [10]. Such approximation methods can be used in black-box identification of nonlinear systems. Artificial Neural Networks have been established as a general approximation tool for fitting nonlinear models from input/output data. The characterizing topology of ANNs is strongly dependent on the problem to be solved. Starting with an initially chosen type of network, the interconnectivity between the neural units is a matter of training. ANNs store information about the specific problem-solving strategy for which they are designed. Therefore, the training or learning strategies are important for practical applications [11,12]. To combine the advantages of ANNs with appropriate feature extraction methods [13–15] we designed a procedure consisting of two parts: (a) for approximating joint angular displacements in reaching movements as arbitrary nonlinear functions we have explored Wavelet Networks (WN), [16] and (b) another ANN has then been used for classification of reaching abilities in SCI patients. This procedure follows recordings of a series of reaching movements in tetraplegics and allows an optimal automatic adjustment of the input parameters dependent upon the correct classifications. The signals from healthy subjects are low frequency

with deterministic components, but the signals recorded from SCI patients had high frequency components which was one of the differences from healthy subjects. Therefore, the application of WN for signal parametrization seemed appropriate (in order to adapt to all the components of the signals). There is a possibility for use of different mother wavelet functions but the first derivative of the Gaussian function by its shape most closely matches the shape of the signals, so we decided to use that function as the mother wavelet. Also, it is a fact that ANNs use sigmoidal functions as activation functions which do not have infinite support. In our study we were dealing with finite length signals, so we needed activation functions with finite support, but not out of range. The application of neural network classification of arm movements is completed by comparing the results of different neural networks: backpropagation, radial basis, recurrent (Elman), competitive and self-organizing. Backpropagation (sigmoid/linear) networks overcome the problems associated with the perceptron and linear networks. Radial basis networks tend to have much more neurons than a comparable network with tansigmoidal or logsigmoidal neurons in the hidden layer, while Elman networks are an extension of the two-layer sigmiod/linear architecture. Self-organizing maps differ from conventional competitive learning in terms of which neurons get their weights updated [17–19].

2. Methods 2.1. Subjects The study included 23 subjects: 16 SCI patients— eight at the Miami Project to Cure Paralysis, Miami, Florida and eight at the Rehabilitation Institute ‘Dr. Miroslav Zotovic’ in Belgrade, Yugoslavia; and seven healthy subjects. Patients were recruited among the local population and all of them have SCI at cervical levels (Table 1). They are able to sit in the wheelchair, they have moderate tonic spasticity, and occasional but controlled phasic spasms. SCI resulted in reduced or lost grasping function and limited control of the wrist, elbow, and shoulder joints. 2.2. Apparatus Arm movements were recorded using a custom made monitoring/recording system based on a 68HC11 microcontroller and flexible goniometers (Penny and Giles, UK Blackwood, UK), (Fig. 1). Apparatus was developed for clinical evaluation of the patient and it can be easily used by a non-professional person after a short period of training. Penny & Giles’ flexible goniometers are easy to use and provide reproducible recordings. These transducers have low power consumption, and


315

Table 1 Patient demographics Rehabilitation center

Belgrade

Miami

Patient

M.Zˇ. D.S. S.T. D.V. D.M. Cˇ.S. R.D. B.Cˇ. A.T. R.P. L.C. R.B. A.P. P.F. J.R. J.B.

Age

3 48 42 18 19 29 60 58 – – – – 26 – 28 –

Gerond

Male Male Male Male Male Male Female Male Male Male Male Male Male Female Male Male

Level of injury

C6/7 C6/7 C7 C5 C4/5/6 C5/6 C4/5/6/7 C3/4 C5/6 C3/4 C3/4/5/6/7 C5/6 C6/7 C4/5 C5/6 C6/7

Dominant hand

Before

After

Right Right Right Right Right Right Right Right Right Right Right Right Left Right Right –

Right Right Right Right Left Right Right Right Left Right Left Left Right Right Right Right

Years after injury

Use of assistive device

12 1 1 2 3 1 – 1 4 7 4 12 2 12 11 12

No No No Yes No No No Yes No No No Yes Yes Yes No No

2.3. Recordings

Fig. 1. Custom made monitoring/recording system [20,21] based on a 68HC11 microcontroller and flexible goniometers (Penny & Giles, Blackwood, UK). Sensors are placed around joints. Portable recording unit is linked with PC laptop via RS-232 connection.

they have a non-compensated bridge configuration. They typically draw 5 mA at 1 V supply, and the output resolution is then 10 µV/deg. Penny & Giles are dual axis transducers which are reasonably accurate if there is no rotation along the ZZ (longitudinal) direction [20,21]. VCR (Sony 990CCD) recordings were taken simultaneously for later review and analysis of unexpected changes. DaDiSP software (DSP Co, Cambridge, MA) was used for data acquisition and analysis, and data reduction. Data were processed within the custom designed software using MatLab 4.2.c.1 software (Mathworks Co.). The complete data analysis was carried out on an IBM PC compatible computer.

All study subjects, after signing the approved informed consent, were asked to reach different points in their working space. Reaching tasks were performed without assistance or assistive devices. Movements were selfpaced, every trial was repeated twice. The dominant arm was evaluated. The following hand movements have been studied: (1) up/down proximal to the body on the lateral side; (2) left/right above the level of the shoulder; and (3) internal/external rotation of the upper arm (humerus). These were designated as set I, II and III, respectively. Patients were asked to follow the evaluators hand and to complete the task as closely as possible. Each movement pattern was repeated after 10 s, and the sampling frequency was 100 Hz. Four relevant joint angles were recorded: (a)—shoulder flexion/extension, (b)—elbow flexion/extension, (g)—shoulder abduction/adduction and (d)—humeral rotation. Arm movements are performed recorded joint angles are presented in Figs. 2 and 3. 2.4. Data analysis Depending on the class of arm movements, a set of dominant joint angles has been chosen to describe the specific movement: (b and g) for the first set of movements, (a,b,g) for the second and (d) for the third. Transformation from recorded internal (a,b,g and d) to external (x,y,z) coordinates was carried out using the equations (x,y,z)=f(a,b,g,d,lFlU), (see Appendix A), where lF is forearm length and lU is upper arm length. Shoulder position (0,0,0) was taken as a reference point.

316


easy interpretation of the instrumented assessment. Different graphical presentations clearly show domains where a subject cannot drive his/her hand. An expert system based software gives the review of different kinematic parameters: minimal and maximal values of all recorded joint angles, minimal and maximal distances between shoulder and hand, minimal and maximal reaching points in the patients’ work space, velocity and movement duration. Performance of patients is presented in comparison with the healthy subjects. The whole procedure for movement visualization is automatic, and numeric values are easily accessible. Also, it is possible to repeat the same type of movements, compare them with previous movements, and follow improvements, if there are any. Fig. 2. Performed arm movements: (A) up/down proximal to the body, set I; (B) left/right above the shoulder, set II; and (C) internal/external forearm rotation, set III. Shoulder position (0,0,0) is a reference point.

2.4.2. Neural network design The concept of WN for parametrization combines aspects of the wavelet transformation (wavelet coefficients) for purposes of feature extraction and selection with the characteristic decision capabilities of neural network approaches (Fig. 4). During the training phase, the WN not only learns adequate decision functions and arbitrary complex decision regions defined by the weight coefficients, but also looks for those parts of the parameter values that are suitable for a reliable categorization of the input signals. A WN can be described as an expanded perceptron with wavelons as preprocessing units for feature extraction; the combination of translation (t), dilatation (d) and wavelet function (y) lying on the same line is called a wavelon. As proposed in [15] we have used the following form of WN: g(x)⫽

冘

wiy[Di(x⫺ti)]⫹g

(1)

where wi are weights, y is the wavelet function, Di are diagonal dilatation matrices D=diag(d), x is the input vector, ti are translation vectors and an additional parameter g is introduced for dealing with non-zero mean Fig. 3. Recorded joint angles: (a)—shoulder flexion/extension; (b)— elbow flexion/extension; (g)—shoulder abduction/adduction; (d)— humeral rotation.

Data analysis was performed using: (1) graphical presentation of arm movements and (2) wavelet and artificial neural networks for movements classification. 2.4.1. Graphical presentation Customized MatLab programs allow simple review and simulation of the recorded movements. A therapist, after loading all the recordings for the patients, has only to follow the menu. The procedure includes joint angles input for the specific patient, and processing of recorded sets is done automatically. There is an option to review all sets and choose the movements to be presented. Graphical presentation in 3D and numeric values allows

Fig. 4. WN structure for approximation of original signal x, where g is defined by Eq. (1). Wavelet function y applies to translated (t) and scaled (d) signal.


317

functions of finite domains. WN uses a stochastic gradient type algorithm, which is very similar to the backpropagation algorithm for neural network learning [22]. The learning process is based on the minimization of the deviation between the original signal and its approximation. All sets for all study subjects are presented to WN and wavelet coefficients (w; s, s=1/d; t and g) are obtained. These coefficients are jointly fitted from data. Wavelet coefficients obtained were then used as an input for training of a feedforward neural network. Corresponding output was the set of the following codes: 0— none, 0.25—poor, 0.5—good, 0.75—very good and 1— excellent for the performance of the related set of joint angles. Healthy subjects have code 1 for all recorded angles for all arm movements. Codes for the quality of the reaching movement performance include expert’s knowledge. These input/output pairs grade elbow and shoulder angles with regard to reaching abilities. Custom made wavelet network design and neural network design were performed using computer software MatLab 4.2.c.1. Combined use of both WN and ANNs is illustrated in Fig. 5. Five different neural networks were used for the classification: (1) backpropagation; (2) radial basis network; (3) recurrent (Elman) network; (4) self-organizing network; and (5) Learning Vector Quantization (LVQ) (see Appendix B). Fig. 6 shows: backpropagation network, radial basis and Elman networks. Fig. 7 shows: self-organizing and LVQ competitive networks. 3. Results Using relatively simple instrumentation and software running on a PC compatible computer we can objectively determine reaching abilities in humans with paralysed arm muscles. In this paper we present results obtained using both graphical presentation (joint angles and trajectories) and wavelet and neural networks.

Fig. 6. Three different network topologies in MatLab enviroment: (a) Backpropagation network: activation function A1 is tansig and A2 is purelin; (b) radial basis network: activation function A1 is radbas and A2 is purelin; (c) Elman network: activation function A1 is recurrent tansig and A2 is purelin. R×Q—input matrix; S1 and S2—number of neurons correspond to the number of input signals; w1 and w2— weights; b1 and b2—bias columns in the first and second layers; and N1 and N2—weighted sums.

3.1. Joint angles The basic evaluation program displays the time functions of joint angles, i.e., there is no loss of information.

Fig. 5. The design of Artificial Neural Networks: wavelet network (WN) was used for signal parametrization and NN for movement classification. The inputs to the WN are recorded and preprocessed joint angles. The outputs of WN are wavelet coefficients for signal identification: w, s=1/d, t and g. Those are transferred to the feedforward ANN which generates codes describing motor skill in reaching movement.

For the chosen set of movements (I, II and III) we show recorded angles: (b) and (g) for the first set of movemets; (a), (b) and (g) for the second set of movements; and (d) for the third set of movements, Fig. 8. The levels of injury of patients are: C5/6, C6/7 and C5/6 [Fig. 8(A)– (C), respectively]. Joint rotations for one healthy subject are shown in the top left corner. All movement trials from all patients in the form of a set of joint angles are processed identically. Preliminary study has shown that it is very important to follow characteristics of joint angles such as, shape, amplitude, range and duration. The simplest way to compare the characteristic joint angles in injured patients and healthy subjects is visualization. A noteworthy finding is that shape and amplitude of joint angles in some patients match closely or

318


Fig. 7. Two different network topologies in MatLab environment: (a) self-organizing network: the competitive transfer function accepts a net input vector for a layer and returns neurons outputs of 0 for all neurons except for the neuron that received the highest net input. (b) LVQ network: activation function A1 is competitive transfer function and A2 is purelin. R×Q—input matrix; S—number of neurons; W—weights; b—bias column; n—weighted sum; and C—competitive function.

are very similar to shape and amplitude of joint angles recorded in healthy subject [Fig. 8(A[b])]. Also, joint angles of the patient in Fig. 8(A[c]) have satisfactory shape and amplitude values. The third patient in Fig. 8(A[d]) has the satisfactory shoulder abduction/adduction (g), but elbow flexion/extension (b) only has the satisfactory amplitude range. Performing the second set of movements, the patients did not have the satisfactory shoulder abduction/adduction (g), while shoulder flexion/extension (a) is preserved, except for the patient in Fig. 8(B[b]). Elbow flexion/extension (b) in all presented patients has almost constant amplitude range. Finally, amplitude values and the range of internal/external forearm rotation (d) are the same [Fig. 8(C[c])], or even better [Fig. 8(C[b] and [d])], than amplitude values and range for the same movement for healthy subject. Note that the duration of the movement is much longer for the patients (see Fig. 8). These three figures show large differences in the pattern of joint rotations among the individuals with the same level of injury. Video recordings of these patients not only confirmed that patients did not perform movements equally successfully, but that they were using different strategies. We can conclude that visual evaluation of these time functions would not yield accurate results for classification. We noticed the variations in amplitude and delay between the recordings, which would be very difficult to express numerically using visual evaluation. Fig. 9 presents the best (left) and the worst (right) patterns of joint rotations in patients with different levels of injury.

Fig. 8. Recorded joint angles: (A) (b,g) for set I of movements for (a) healthy subject (0.45 s); and three SCI C5/6 patients: (b) A.T. (0.64 s); (c) R.B.(0.49 s); and (d) C.S.(0.83 s); B) (a,b,g) for set II of movements for (a) healthy subject (0.8 s), and three SCI C6/7 patients: (b) J.B. (1.1 s); (c) M.Z. (0.9 s); and (d) A.P. (0.9 s); (C) (d) for set III of movements for (a) healthy subject (0.4 s), and three SCI patients: (b) J.R. (0.1 s); (c) C.S. (0.23 s); and (d) A.T. (0.36 s). Notation: a— shoulder flexion/extension; b—elbow flexion/extension; g—shoulder abduction/adduction; d—humeral rotation. Shoulder position (0,0,0) is a reference point. Time course is normalized. Numbers in parenthesis indicate movement duration.


319

Fig. 9. The worst and the best movement trials for SCI patients. Top panel: set I; middle panel: set II; and bottom panel: set III.

3.2. 3D hand trajectories The next step is graphical presentation, including 3D hand trajectories (Fig. 10), corresponding to angles from Fig. 8. Hand trajectories are plotted in 3D space. As mentioned before, seven healthy subjects were also tested under the same protocol. They were the control group for the comparison of joint angles and hand trajectories. Fig. 10(A[a]), (B[a]) and (C[a]) are the 3D presentations of hand trajectories for a healthy subject, for chosen characteristic movements, up/down proximal to the body (set I); left/right above the shoulder (set II); and internal/external forearm rotation (set III). From Fig. 10(A[c]) we notice that the trajectory of the second patient is very similar to the trajectory of the healthy subject. The movement of the third patient [Fig. 10(A[d])], is the worst performed, according to the shape of the hand trajectory and z-values (shoulder plane). The whole movement was performed below the shoulder. The best movement performed in the horizontal plane above the shoulder is in Fig. 10(B[d]), with a small range of shoulder flexion/extension (a) compared with healthy subject. The worst movement performed is the movement of the first patient [Fig. 10(B[b])], who just picked up his hand, but did not move it left/right. Inconsistently performed movements can be noticed from Fig. 10(C) for the set III movements. Fig. 11 shows the best (left) and the worst (right) patient trials which correspond to the angles from Fig. 9. All these hand trajectories are trajectories in patients with different levels of injury. Elbow trajectories are also avaliable. On the basis of these presentations, we concluded that 3D hand trajectories presentation is better than angle presentation

Fig. 10. 3D hand trajectories corresponding to sets of angles in Fig. 8 for: (A) (a) healthy subject, and three SCI C5/6 patients: (b) A.T.; (c) R.B.; and (d) C.S.; (B) (a) healthy subject, and three SCI C6/7 patients: (b) J.R.; (c) M.Z.; and (d) A.P.; C) (a) healthy subject, and three SCI C5/6 patients: (b) J.R.; (c) C.S.; and (d) A.T.

but does not lead to automatic objective determination of reaching abilities in tetraplegic humans. 3.3. Wavelet network (WN) The success of any pattern classification system depends almost entirely on the choice of features used to represent the continuous time waveform. We attempted to recognize joint angles using the wavelet network technique. Although these signals mainly con-

320


Fig. 11. 3D presentation of hand trajectories corresponding to sets of movements shown in Fig. 9.

tain deterministic components, they also have a random component. Initially, parameters for the WN have been chosen: coefficients (w, s, t and g), number of neurons (N) and velocity of convergence for the stochastic gradient algorithm [16]. All sets of recorded and processed data were presented to the wavelet network and all signal approximations obtained. From previously described movements, up/down proximal to the body was chosen. It is described by two angles [shoulder abduction/adduction (g) and elbow flexion/extension (b)]. Results obtained from the WN for healthy subject (A) and patient (B) are shown in Fig. 12. WN input is always only one angle. Fig. 12 also shows, besides original and approximated signals, scaled and translated wavelets. Summation of these wavelets gives the approximation of the original signal. Fig. 12 also shows error defined as the difference between original and approximated signals with respect to the number of iterations. In most cases errors were less than 0.0001 after about 200 iterations. Results obtained after convergence of the net parameters for elbow rotation (b) for healthy subject (M.P.) and SCI patient (J.R.) are listed in Table 2. From Table 2 we cannot interpret the significance of the difference between wavelet coefficients for healthy subject and SCI patient, and their comparison is not feasible. This table is one example of signal parametrization. All sets of data of all subjects are processed identically. 3.4. Artificial neural networks (ANN) Approximated joint angles, obtained as WN outputs, were graded with codes [0; 0,25; 0.5; 0.75 and 1]. Training of ANNs was performed on the set of movement trials (10 from patients and 10 from healthy subjects) for arm movements described in Methods. Sets of wavelet coefficients at input and coded quality of the reaching performance at output were used for network training. Each network was trained with the specific set of arm movements, separately. The physiotherapist provided the

Fig. 12. Signals generated by wavelet network. Top panels: elbow joint rotation: original signal and signal approximated by wavelet network; Middle panels: wavelets; and bottom panels: error defined as a difference between original and approximated signal with respect to number of iterations for (A) healthy subject and (B) SCI patient.

expertise and ‘supervised’ learning to score the output. One example of signal approximations presented to ANN for five subjects that are coded with numbers 0; 0.25; 0.5; 0.75 and 1 is presented in Fig. 13. When a new set of signals (a set which was not used for network training) was presented to the network, output codes were estimated closely to ones from our expertise. ANNs that were the most satisfactory have been chosen and tested with recorded signals that were not presented to the network before. 3.5. Classification procedure Testing of the chosen neural networks was performed on the set of movements that were not used for network training. Tests were done particularly for each movement, with wavelet coefficients previously obtained by WN as inputs. Cross-validation was not used. Since the outputs from backpropagation, radial basis and recurrent


321

Table 2 Wavelet coefficients (w,s,t,g) for elbow extension/flexion in healthy subject A and patient B from Fig. 10 (angle b). N is the number of neurons N w s=1/d t g

A B A B A B A B

1

2

3

4

5

6

7

1.58 0.15 0.11 0.12 0.13 0.14

⫺0.35 ⫺0.25 0.10 0.12 0.38 ⫺0.09

⫺0.64 0.64 0.10 0.16 0.40 0.18

0.01 0.28 0.50 0.45 0.50 0.40

0.72 ⫺0.13 0.10 0.23 0.60 0.76

0.33 0.56 0.10 0.07 0.70 1.05

⫺1.80 0.23 0.17 0.06 0.84 1.04 1.43 1.69

Fig. 13. Training set of five patients’ signals presented to ANNs by their wavelet coefficients coded with: (a) 1; (b) 0.75; (c) 0.5; (d) 0.25 and (e) 0 from the top to the bottom.

(Elman) networks are continuous, those were quantified with five levels [0; 0.25; 0.5; 0.75 and 1] using the rule: ‘Round to the closest level’. Table 3 presents results for the first set of movements, for 10 patients and one healthy subject. The results presented were obtained using three different neural networks: (1) backpropagation (ANN1); (2) radial basis (ANN2); and (3) recurrent (Elman) network. The recurrent (Elman) network is the best network for the classification of the first set of movements—91%, then radial basis—64%, and finally backpropagation—55%. The same networks were used for the second set of movements, and tested for 14 patients and two healthy subjects. The best results for the second set of movements were obtained with the radial basis network— 73%, then recurrent (Elman)—54%, and finally backpro-

pagation—46%. Results obtained for the second set of movements are not presented in this paper. Table 4 presents results for the third set of movements for six patients and seven healthy subjects. The third set of movements has only one relevant angle, and besides backpropagation and radial basis networks it can be classified with self-organizing (Table 5) and LVQ (Table 4) competitive networks. Results obtained with selforganizing and LVQ networks are the best obtained— 100%, 92% respectively for self-organizing and LVQ networks. Analysing all recorded sets of signals we found that ANNs outputs for some subjects differ for two or three codes from expert codes. The noteworthy finding is that the radial basis network output for the second and the third set of movements is either correct or belongs to the very next level suggested by expert. We can conclude that the best results are obtained with a selforganizing network for the third set of movements. Also, we found that this network with three neurons in the hidden layer gave better results for tested signals than the same network with two or five neurons in the hidden layer. The number of neurons in the hidden layer (equal to the number of suggested groups) is given to the network in advance, i.e. it does not ‘decide’ about their number. Movements of healthy subjects for all sets of movements are correctly classified (except for N.S.—set III movements with LVQ, backpropagation and radial basis networks). Slightly worse classification than self-organizing network was achieved with the LVQ competitive network. The LVQ network was applied only for the third set of movements, because its outputs are different groups. For the third set of movements only one angle is important; the code for that angle is then in the same time-group. The worst classification results are obtained with the backpropagation (perceptron) network, i.e., number of successfully classified movements is 55, 46 and 70%, for the first, the second and the third set of movements. Slightly better results (but still worse than competitive) were obtained with the radial basis network. It is worth mentioning that the number of relevant joint angles for

322


Table 3 Compared codes at the output of backpropagation (ANN1), radial basis (ANN2) and recurrent (Elman-ANN3) network are rounded to five levels (0;0.25;0.5;0.75;1) and compared with expert codes for set I movements in six tested patients and seven healthy subjects. C/I designates correct/incorrect classification for specific network

Subject

M.Z. D.S. C.S. R.D. R.P. L.C. R.B. A.P. J.R. J.B. N.S.

Codes for set I movements ANN1 ANN1 C/I output rounded

ANN2 output

ANN2 rounded

C/I

ANN3 output

ANN3 rounded

C/I

Expert

1 0.53 0.97 0 0.7 0.73 0.77 0.31 0.30 0.53 0.41 0.27 1 0.92 0 0.83 0.81 0.23 1 0.89 0.93 0.87

1 1 0.97 1 0.45 0.55 0.98 1 0.7 0.97 0.45 0.11 1 1 0.81 0.86 0.77 0.85 1 0.89 1 1

1 1 1 1 0.5 0.5 1 1 0.75 1 0.5 0 1 1 1 0.75 0.75 0.75 1 1 1 1

C C I I C C C I I I C C C C C I I I C C C C

1 1 0.73 0 0.45 0.55 1 0.54 1 0.71 0.35 0 1 1 0.9 0.55 0.25 0 1 0.92 1 0.82

1 0.5 0.75 0 0.5 0.5 1 0.5 1 0.75 0.5 0 1 1 1 0.5 0.25 0 1 1 1 1

C C C C C C C I I C C C C C C C C C C C C C

1 1 0.75 0 0.5 0.5 0.75 0.25 0.25 0.75 0.5 0 1 1 1 0.5 0.25 0 1 1 1 1

1 0.5 1 0 0.75 0.75 0.75 0.25 0.25 0.5 0.5 0.25 1 1 0 0.75 0.75 0.25 1 1 1 1

C I I C I I C C C I C I C C I I I I C C C C

Table 4 Compared codes at the output of propogation (ANN1), radial basis (ANN2) and LVQ (ANN4) network are rounded to five levels (0,0.25,0.5,0.75,1) and compared with expert codes for set III movements in ten tested patients and one healthy subject (N.S.). C/I designates correct/incorrect classification for specific network

Subject

Codes for set III movements ANN1 ANN1 C/I output rounded

ANN2 output

ANN2 rounded

C/I

ANN4 output

ANN4 rounded

C/I

Expert

L.C. A.T. M.Z. Cˇ.S. R.D. J.R. MP I MP II MP III PM I PM II PM III N.S.

0 1 0.73 0.54 0 1 0.978 1 0.943 0.86 1 1 0

0.47 1 1 0.7 0.52 0.95 0.976 0.946 0.92 0.73 0.83 0.85 0.54

0.5 1 1 0.75 0.5 1 1 1 1 0.75 1 1 0.47

C C C I C C C C C I C C I

0.5 1 1 1 0.5 0.95 1 1 1 1 1 1 0.5

0.5 1 1 1 0.5 1 1 1 1 1 1 1 0.5

C C C C C C C C C C C C I

0.5 1 1 1 0.5 1 1 1 1 1 1 1 1

0.5 1 0.75 0.5 0 1 1 1 1 1 1 1 0

I C I I I C C C C C C C I

the first set of movements is two, and for the second is three; that renders classification more difficult; besides, there was an insufficient number of signals for network training. The Elman (recurrent) network could not be applied for the third set of movements, because its output for tested movements was a constant; besides, the network has a recurrent connection in the first (sigmoidal)

layer and it could not adjust its weights and biases for small differencies of input vectors. Correct classification was used as a criterion for the quality of the method. The classification rate yielded 46– 100% for movement trials that were tested (Fig. 14).


Table 5 Self-organizing networks (ANN5) ‘learn’ to recognize groups of similar input vectors. Output of the ‘winner’ neuron is 1, and of neighbours is 0.5. Obtained results for set III of movements are classified into two groups: 0.5—the second neuron, and 1—neurons 1 and 3. Network correctly classified all ‘unknown’ inputs Subject

ANN5 output

L.C.

(1,1) (2,1) (3,1) (2,1) (3,1) (2,1) (3,1) (1,1) (2,1) (1,1) (2,1) (3,1) (2,1) (3,1) (1,1) (2,1) (1,1) (2,1) (1,1) (2,1) (1,1) (2,1) (1,1) (2,1) (1,1) (2,1) (1,1) (2,1)

A.T. M.Z. C.S. R.D.

J.B. MP I MP II MP III PM I PM II PM III N.S.

0.5 1 0.5 0,5 1 0.5 1 1 0.5 0.5 1 0.5 0.5 1 1 0.5 1 0.5 1 0.5 1 0.5 1 0.5 1 0.5 1 0.5

Winner

Expert

2

0.5

3

1

3

1

1

1

2

0.5

3

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

Fig. 14. Success of classification for all ANNs and all tested movement trials.

4. Conclusion In this paper we presented: (1) apparatus for collecting kinematic data; (2) software for data acquisition and

323

presentation to the therapist; and (3) a methodology developed for arm movement classification in humans with spinal cord injury. Apparatus was developed for clinical evaluation of the patient and it can be easily used by non-professionals after a short period of training. Software runs on a PC platform and uses a customized MatLab program. A therapist after loading the recordings has only to follow the menu, and he/she will obtain a complete set of zones that are of interest for Activities of Daily Living (ADL), but are out of patients reach. Also, there is a possibility to list all recorded signals off-line and exchange among clinical centers which have the same software. A noteworthy finding is that the typical neurological classification (level of injury), which is basically the key diagnosis, often does not correlate with the functional capabilities of humans after injury. The same level of injury results in significantly different motor skillls. The purpose of this study was to introduce the Artificial Neural Network approach for application in this specific medical classification. Realization of our approach does not follow any of the classical methods, but takes advantages of different concepts of Artificial Neural Networks. The pattern classifier is the major component of this novel approach to objective assessment of reaching abilities. Due to the level of injury, it is reasonable to expect a large variation in the pattern of joint rotations between individuals. A suitable classifier must be trainable to accommodate the expected individual differences and as well to adopt to slow variations in feature values. An ANN was chosen as the classifier for this application. The categorizing topology of an ANN is strongly dependent on the problem to be solved. Starting with an initially chosen type of network, the interconnectivity between the neural units is a matter of training [23]. Advantages of the proposed method for classification are: (1) non-invasive, easily implemented in clinical conditions; (2) software is user friendly; (3) WN and ANNs can be easily implemented on different platforms which support MatLab; (4) recorded and approximated signals are close; and (5) small number of neurons for WN and ANN. Also, the method developed can be used for determination of different movement strategies in healthy subjects, for example in sportsmen and women. Disadvantages of the classification approach are: (1) we cannot be sure, that we can always get good results for each patient; and (2) we cannot exactly specify the number of iterations for each movement. Both problems are related to general problems with neural networks and their training. The first problem includes a large variation in the pattern of joint rotations among individuals. If the network ‘did not see’ a specific input, it will not recognize that input, and output codes will be very poor. The problem can be solved with large number of signals

324


which makes it possible for the network to learn input/output mapping. The second problem is related to the wavelet network. It takes into account the automation of the whole proposed procedure. Namely, WN learns quickly or slowly depending on the pattern of joint rotations. In addition to other initial net’s parameters we have to specify the number of training iterations. The problem can be solved by setting the adaptive error between original and approximated signal by WN [24]. There are several factors that must be mentioned: (1) number of evaluated patients was insufficient; (2) many lesions are incomplete; and (3) with the increase of the number of signals which describe the specific movement, the classification rate decreases. The final result of this analysis allows the investigator to complete his/her diagnostic procedure and tentatively discuss with the subjects what are the prospects of application of assistive systems for enhanced reaching. Results obtained indicate which network can be used for the classification of arm movements in humans with spinal cord injury. Based on this classification, we can conclude that the competitive networks are the best, but only in the case when the specific movement performed is described with only one representative angle. On the contrary, information about angles are lost, and we can only determine different groups of patients according to the functionality of movements. Also, more signals in the training set will give better results, bearing in mind the variability of the signals independent of level of the injury. Our approach and results can be regarded as an encouraging method for the classification of reaching abilities in humans with spinal cord injury and can be used for prescription of an adequate assistive system, which eventually may improve reaching abilities. From our work we concluded that: for three characteristic, different arm movements, previously described, it is very important to follow geometric features, such as, minimal and maximal values of x,y,z hand coordinates, minimal and maximal values of relevant angles for the specific movement, closest and the most distant position of handto-shoulder, and also the percentage of the whole elbow and shoulder flexion and extension. It is not possible to estimate preserved motor skills on the basis of kinematics. Objective classification of arm movements was unmanageable using only kinematic parameters. A combination of wavelet and classical neural networks has appeared as a suitable method for geometric features analysis.

written in MatLab, and Ms Tatjana Coric for useful suggestions. We would also like to thank all patients and staff at the Rehabilitation center and Dr. Miroslav Zotovic who helped us to record signals. The work in this project is funded by the Ministry of Science and Technology of Serbia, Belgrade.

Appendix A

Transformation matrix for the calculation of the hand position {xH, yH, zH} in the mobile co-ordinate system in reference to the shoulder can be obtained from the following four steps:

A.1. Step 1

Rotation of the co-ordinate system Oxyz with unit →→ → vectors {i, j, k} into the co-ordinate system with unit →→→ vectors {a1,a2,a3} around the z axes for angle a is expressed by the transformation equation: →→ → i=a1cosa−a2sina →→ → j=a1sina+a2cosa →→ k=a3

A.2. Step 2 Acknowledgements We would like to thank Mrs. Aleksandra Vuckovic for providing us with the Wavelet Network Algorithm

Rotation of ‘a’ co-ordinate system into the co-ordinate system ‘b’ around the a2 axes for angle g is expressed by the transformation equation:


→ → → a1=b1cosg−b3sing → → a2=b2 → → → a3=b1sing+b3cosg

(sina sind−cosa sing cosd)

325

|

−(cosa sind+sina sing cosd) cosg cosd

Hand co-ordinates with reference to the elbow are: xE⫽⫺lFcosb, yE⫽lFsinb, zE⫽0 and, with reference to the shoulder: xH⫽lUcosa cosg⫺lFcosb cosa cosg ⫺lFsinb(cosa sing sind⫹sina cosd) yH⫽lUsina cosg⫺lFcosb sina cosg ⫺lFsinb(sina sing sind⫺cosa cosd) zH⫽lUsing⫺lFcosb sing⫹lFsinb sind cosg. A.3. Step 3 Rotation of ‘b’ co-ordinate system around the b1 axes for angle d is expressed by the transformation equations: →→ b1=i1 →→ → b2=j1cosd−k1sind →→ → b3=j1sind+k1cosd

Transformation matrix is then: cosa cosg −(cosa sing sind+sina cosd)

T⫽ sina cosg −(sina sing sind−cosa cosd) sing

Appendix B Different network topologies with powerful learning strategies to solve nonlinear problems have been developed. An overview is given in [14]. For our application, we refer to the WN and MultiLayer Perceptron (MLP) and self-organizing networks. In this section, a brief summary of the most important properties is given. In our implementation, the first derivative of the Gaussian function has been chosen as the basis wavelet ψ. An MLP is organized in different layers, each containing units called neurons. These units are interconnected by directed and weighted links. A neuron’s output is processed to all the neurons of the next layer, and its outputs come from all units of the previous layer, i.e. subsequent layers are fully connected. Fig. 6(a) is a general function approximator. This network can approximate any function with a finite number of discontinuities, with enough neurons in a hidden layer. Activation functions A1 and A2 are:

A.4. Step 4

|

Notation: a—shoulder flexion/extension; b—elbow flexion/extension; g—shoulder abduction/adduction; d— humeral rotation; lU—upper arm length; lF—forearm length.

sind cosg

A1tansig(W1∗p⫹b1) and A2⫽purelin(W2∗a1⫹b2), where W1 and W2 are the weights matrices, p is the input vector, b1 and b2 are the bias columns and a1 is the output from the previous layer. This network was used in the methodology developed as a classifier of extracted features from WN, for the first and second set of movements. For the third set of movements, instead of a ‘tansig’ activation function

326


‘purelin’ was used, a network without nonlinearity which could learn small differencies between samples of signals. Networks are sensitive to the number of neurons in hidden layers. A small number of neurons results in underfitting, and large in overfitting, when the fitting function oscillates between points. As the optimum number of neurons in the first sigmoidal layer appeared S1=6, and in the output linear layer S2=2;3;1 for the first, the second and the third set of movements, respectively. The use of a nonlinear transfer function often causes an ANN to get trapped in a local minimum. Accepting one local minimum can be good or bad depending on how close it is to the global minimum. So, backpropagation will not always find correct weights for the optimal solution. A further problem is created when the network overfits the data in the training set and does not generalize well to new data outside the training set. It can be prevented by using stopping (the fitting process early e.g. by passing a validation set to the training algorithm). The learning algorithm used was Levenberg–Marquardt optimization [17] with an adaptive learning rate, which is a more sophisticated method than backpropagation. The sumsquared error goal was 0.02. The Nguyen–Widrow rule [18] was used for getting initial conditions, which are created with the MatLab function ‘initff’. We obtained better results using radial basis networks. A radial basis neuron receives, as net input, the vector distance between its weight vector w and the input vector R×Q, multiplied by the bias b. Activation functions A1 and A2 are: A1⫽radbas(dist(W1,p)⫹b1) and A2⫽purelin(W2∗a1⫹b2), with the same notation as above. In our implementation, the optimum number of hidden layers is S1=8 and number of output layers was S2=1,2 or 3 for the first, the second and the third set of movements, respectively. The sum-squared error goal was 0.01. The same learning algorithm with ‘radbas’ neurons and the other same conditions was used. The Elman network is two-layered with feedback in the first layer. This recurrent connection allows the Elman network to both detect and generate time-varying patterns. The Elman network has tansigmoidal neurons in its hidden (recurrent) layer and linear neurons in its output layer. Activation functions A1 and A2 are: A1⫽tansig(W1∗[p;a1]⫹b1) and A2⫽purelin(W2∗a1⫹b2), with the same notation as above and a1—recurrent input. Delay in recurrent connection stores values from the previous steps, so they can be used in the current step. The

Elman network can learn to generate temporal and spatial shapes. The optimum number of recurrent hidden layers was S1=6;4 and S2=2;3 for the first and the second set of movements, respectively. We could not apply the Elman network for the third set of movements because its outputs were constant. All other conditions are the same as for the backpropagation network. Self-organizing networks (they belong to competitive networks) can learn to detect regularities and correlations in their input and accordingly adapt their future responses to that input. Thus self-organizing networks learn both the distribution (as do competitive layers) and topology of the input vectors they are trained on. They allocate more neurons to recognize parts of the input space where many input vectors occur and allocate fewer neurons to parts of input space where few input vectors occur. The activation function A is: A⫽compet(⫺dist(W,p) ⫹b), with the same notation as above. The competitive transfer function accepts input vectors. Its outputs are zero for all neurons, except for the neuron whose output is 1. We could use it successfully only for the third set of movements (d—the only relevant angle). The optimum number for hidden layers was 3, and for output was 1. Distance was calculated using the Manhatten [19] rule, and learning rate was 1. Initial weights were obtained using the MatLab function ‘initsm’. LVQ is a method for training competitive layers in a supervised manner. LVQ networks classify input vectors into target classes by using a competitive layer to find subclasses of input vectors, and then combining them into the target classes. Unlike perceptrons, LVQ networks can classify any set of input vectors, not just linearly separable vectors. The only requirement is that the competitive layer must have enough neurons, and each class must be assigned enough competitive neurons. To ensure that each class is assigned an appropriate amount of competitive neurons, it is important that the target vectors used to initialize the LVQ network have the same distributions of targets as the training data. If this is done, target classes with more vectors will be the union of more subclasses [22]. In our study, they were used only for the third set of movements where only one angle (d) was important. The optimum number for hidden layers was 3, and for output was 1. Learning rate was 0.05. The weights, initialized using random initial values using MatLab functions for initialization, are updated in each iteration until the net has settled down to a minimum. These networks appeared as most suitable for the classification of arm movements in tetraplegics.


References [12] [1] Gowitzke BA, Milner M. Scientific basis of human movement. Baltimore: Williams and Wilkins, 1988. [2] Nathan RH. FNS of the paralyzed upper limb in quadriplegia: fitting the technology to the lesion level. Proc. 1st Int. FES Symp. Sendai, 1995:88–95. [3] Charles L, McDowell CL, Moberg EA, Smith AG. International conference on surgical rehabilitation of the upper limb in tetraplegia. J Hand Surg 1979;4:387–90. [4] McDowell CL, Moberg EA, House JH. The Second International Conference on Surgical Rehabilitation of the Upper Limb in Traumatic Quadriplegia. J Hand Surg 1986;11A:604–8. [5] American Spinal Cord Injury Association. Standards for the neurological classification of spinal injury patients. Chicago: ASIA, 1992. [6] Gellman H. The hand and upper limb in tetraplegia. Curr Orthop 1991;5:233–8. [7] Stein RB, Peckham PH, Popovic DB. Neural prosthesis: replacing motor function after disease or disability. Oxford: Oxford University Press, 1992. [8] Moberg E. Surgical treatment for absent single-hand grip and elbow extension in quadriplegia. J Bone Joint Surg 1975;57A(2):196–206. [9] Zancolli E. Surgery for the quadriplegic hand with active, strong wrist extension preserved. Clin Orthop Rel Res 1975 Oct, 112:101–12. [10] Narendra KS, Pthasarathy K. Identification and control of dynamical systems using neural networks. IEEE Trans Neural Net 1990;1:4–27. [11] Sirnivasan S, Gander RE, Wood HC. A movement pattern gener-

[13]

[14] [15] [16] [17] [18] [19] [20]

[21] [22] [23]

[24]

327

ator model using artificial neural networks. IEEE Trans Biomed Eng 1992;39(7):716–22. Schalkoff R. Pattern recognition: statistical, structural and neural approaches. New York: John Wiley and Sons, 1992. Kawato M, Furukawa K, Suzuki R. A hierarchial neural-network model for control and learning of voluntary movement. Biol Cyber 1987;57:169–85. Lippmann RP. An introduction to computing with neural nets. IEEE ASSP Mag 1987:4–22. Zhang Q, Benveniste A. Wavelet networks. IEEE Trans NN 1992;3(6):889–98. Dickhaus H, Heinrich H. Classifying biosignals with wavelet networks. IEEE Engng Med Biol 1996, 103–11. Masters T. Signal and image processing with neural networks. New York: John Wiley and Sons, 1994. Poggio T, Girosi F. Networks for approximation and learning. Proc IEEE 1992;78(9):91–106. Hornik K. Multilayered feedforward networks are universal approximators. IEEE Trans NN 1989;2:355–66. Tepavac D, Popovic M, Kostov A, Nikolic Z, Popovic D. A portable system for assesment of functional electrical stimulation assisted locomotion. Proc. III Int Symp on FES, Sendai 1992:130–4. Popovic M. A portable 8 channel gait kinematics recording. Proc 16th Ann Int Conf IEEE Eng Med Bio Soc 1992, 1646–7. Wasserman PD. Neural computing theory and practice. New York: ANZA Research Inc, 1989. Akay YM, Akay M, Welkowitz W, Kostis J. Noninvasive detection of coronary artery disease using wavelet based fuzzy neural networks. IEEE EMB Mag 1994;13(5):761–4. Kalayci T, Ozdamar O. Wavelet preprocessing for automated neural network detection of EEG spikes. IEEE EMB Mag 1995;14:160–6.