3. computer (gestures 9-12) It starts with the opening of a laptop. After that some random text is entered via the keyboard. Then some random mouse movements.
Using FSR based muscule activity monitoring to recognize manipulative arm gestures Georg Ogris, Matthias Kreil, Paul Lukowicz Embedded Systems Lab (ESL), University of Passau {georg.ogris,matthias.kreil,paul.lukowicz}@uni-passau.de
Abstract We present an experiment that investigates the usefulness of muscle monitoring information from arm mounted force sensitive resistors (FSR) for activity recognition. The paper is motivated by previous work that has demonstrated the feasibility of using FSRs for muscle activity monitoring (on leg muscles) and presented some initial signals related to distinct arm activities. We systematically investigate the performance of an FSR system on 16 different manipulative gestures and 2 subjects. The aim is to test the limits of the system, compare them to established sensing modalities (3D acceleration and gyro), and establish the value of combining FSR with other sensing modalities. We also present a hardware setup that addresses key problems that were identified in previous work: large variations in the attachment force and sensor placement accuracy issues. For all classifiers the overall accuracy of the FSR system is in the middle between the accelerometer (between 5% and 10% better) and the gyro (between 2% and 11% worse). Adding FSRs to another sensor improves the accuracy by 1% to 29%.
1
Introduction
Arm and hand actions are a key component of user activity. As a consequence much research in wearable context recognition has gone into tracking and recognizing such action. The vast majority of this research is based on motion sensors placed at different arm and hand locations (e.g. [7, 2]). This has both advantages and disadvantages. On the positive side, motion sensors, in particular accelerometers, are cheap, small and low power. At the same time a significant amount of activity information is contained in motion patterns. On the negative side, motion sensors tend to contain a mixture of motion and orientation information that is difficult to separate with simple sensor setups. In addition they provide no information about the palm and finger activity. Alternative approaches that address the above disadvantages include video tracking of the arms [5] and sensor
gloves. Our group has also investigated the use of ultrasonic beacons to track hand positions [6]. General Idea This paper investigates an additional source of information about arms and hand actions: the analysis of arm muscle activity with force sensitive resistors (FSRs). FSRs are thin piezoelectric plates (see figure 2) that change their electrical resistance as mechanical force is applied to their surface. They are cheap and can be easily integrated in garments. It has also been shown ([4]) that such mechanical pressure sensors can be implemented directly into textiles. The idea behind our work is to use such sensors in an elastic sleeve mounted on the forearm. It is motivated by the fact that palm and finger motions are driven by muscles in the forearm. As those muscles contract they change their shape which in turn results in mechanical pressure being applied to the sensors in the elastic sleeve. Paper Contributions In previous work [3] we have demonstrated the general feasibility of using FSRs to monitor leg muscle activity. In [1], we have also shown that different arm actions such as holding a heavy object or making a fist produce distinct FSR signals. This paper goes beyond such simple signal examples to investigate using FSRs for actual recognition of a set of non trivial manipulative gestures. The aim is not to demonstrate the ability to reliably recognize a specific, application relevant activity set. Instead we want to lay the groundwork for other researchers to set up a FSR based activity recognition system. To this end we present the following specific contributions: 1. We systematically investigate the performance of an FSR system with different classifiers on a set of 320 manipulative gestures from 16 different classes performed by 2 subjects. The classes have been chosen to test the limits of the system rather than be recognizable with high accuracy. Thus, the set contains some very subtle gestures. 2. We compare the FSR performance to established sensing modalities (3D acceleration and gyro). To cap-
Figure 1. Contestant wearing the FSR sensor system, a
Figure 2. Schematic of current to voltage converter, small
MTx glove, and a mobile computer, that serves as a power source.
left: addon boards for the Tmote Sky, small right: FSRs
ture hand rather than only arm motions (as do the FSR through muscle monitoring) the additional sensors are mounted on the back of the hand rather than on the wrist. 1 3. We investigate the benefit of combining FSRs with additional sensing modalities by testing different combinations of the three sensor types (acc, gyro, FSR). 4. We describe in detail a hardware setup that addresses key problems that were identified in previous work: large variations in the attachment force, and sensor placement accuracy issues.
2
The Sensor System
Sensor Placement and Attachment In [3] our group has described the effect of sensor displacement on muscle activity monitoring. It was shown that sensor displacement of just 1 cm can lead to false or no signal. However we have also demonstrated that this problem can be overcome by covering a larger area with the FSRs. This can either be accomplished by a matrix of sensors around the point of interest (as discussed in [3]) or by using large FSRs. In this paper we combine both approaches. Both the lower part of the forearm (right behind the wrist) and the upper part of the forearm (right under the ellbow) are covered with a ring of 4 46 x 46 mm FSRs. The attachment of the sensors is another crucial question. FSRs require a moderate counterforce. On the other hand, the system should be wearable, i.e., easy to put on and off and not too tight. As a consequence we opted for a three layer design: a thin inner layer (a thin stocking) on which the FSRs - the second layer - are fixed, and a third outer layer that is tight but stretchable. 1 Wrist mounting is actually more common in application since attaching devices to the hand is mostly considered burdensome (requires at least a glove). However we know from previous work that wrist mounted motion sensors are not good in recognizing gesture primarily defined by hand motions, so that hand mounted sensors offer a more challenging comparison.
We considered an ordinary bicycler’s sleeve to be the right choice. That way we end up with a sleeve that can be put on and off easily. Sensor System Even with the above attachment concept there are significant, user dependent variations in the FSR counter forces. When measuring the resistance of the FSR by means of a voltage divider as done in previous work, see e.g. [1, 3], the initial counter force must be adapted for each user due to the nonlinear relationship of resistance and output voltage. To overcome this problem the new system is based on current measurement instead of voltage measurement. That requires a current/voltage converter (Figure 1) but results in a linear relationship between resistance and output voltage with a much better dynamic range. Thus we get usable FSR signals despite the variations in the counter force. A simple standard motion (e.g. bending the arm) performed after the sensor have been put on can be used to calibrate the system to a specific user. The overall sensor system (Figure 2) comprises a tmote sky from moteiv with addon boards mainly featuring an ADG708 to multiplex the 8 FSR channels and the current/voltage converter. Future work An important next step will be to investigate in more detail and possibly automate the calibration of the system to different user. An auto calibration procedure could also compensate changes in signal baseline caused by slipping sensors.
3
The Experiment
The experiment design was not driven by a specific realistic application. Instead we aimed to have an activity that contained a reasonable variety of realistic gestures that would test the limits of what can be achieved with FSR muscle activity monitoring. In particular we wanted to have a mixture of bold and subtle gestures, gestures performed with different force and a different amount of ’gripping’.
1 2 3 4 5 6 7 8
open the pen write on whiteboard close the pen erase scroll screen down point on screen scroll screen up press buttons on projector remote control
9 10 11 12 13 14 15 16
classifier HMM C4.5 kNN
open notebook type a few words use the mouse close the notebook open bootle pour water in a glas close bottle drink
Table 1. Set of gestures At the same time we wanted to have a situation where the gestures are performed in a natural way as part of an every day activity rather than make the subjects perform a set of artificial gestures. Furthermore we wanted a scenario where the technical and organizational effort involved in the experiment is kept to a minimum needed to collect a solid, representative data set. Experimental Scenario As a consequence of the above considerations we have opted for a scenario of giving a talk at a seminar. Table 1 lists the gestures performed. They can be organized in 4 groups of actions consisting of 4 different subroutines. 1. whiteboard (gestures 1-4) First the test person opens a pen, then he starts to write some random words on the blackboard. After that he closes the pen, which is similar to the opening movement. Finally the words are cleared with an eraser, which is usually done in a periodic circulate movement. 2. beamer screen (gestures 5-8) This gesture set contains first the pushing down of the screen for, e.g., a beamer. Then the test person points on the screen with a finger to show some things. After that the screen is lifted up again, which is similar to the first movement. The last action is using the remote control to switch off the beamer by pressing keys. 3. computer (gestures 9-12) It starts with the opening of a laptop. After that some random text is entered via the keyboard. Then some random mouse movements are done. Next the laptop is closed again, which is a similar movement to the opening of the laptop. 4. drinking (gestures 13 - 16) A bottle of water is first opened by unscrewing the lid. Water is then poured into a glass and the bottle is closed again. Finally the test person takes a sip of water from the glass. Experimental Setup We mounted our FSR sleeve on the right forearm. Furthermore the subjects wore a glove with a MTx sensor from Xsens. The MTx sensor is mounted on the back of the hand rather then on the wrist to prevent its housing from influencing the FSRs. Note that this provides significantly more information on palm actions then
acc 83 76 81
gyr 72 57 65
fsr 73 62 76
acc+gyr 91 82 90
fsr+acc 84 79 86
fsr+gyr 81 68 84
Table 2. Classification results in % the more typical (and much less obtrusive) wrist based setup and to a degree neutralizes the inherent advantage of FSR. The FSRs were sampled at a rate of 20 Hz, the MTx at 50 Hz. In total we recorded 2 subjects, with 10 datasets each and one instance of every activity listed above per dataset (total of 320 gestures). The experiment environment was a meeting room at the university with a whiteboard, a beamer screen and a meeting table. Recognition Method For comparison, three classifiers have been tested. The tree based C4.5 classifier and the instance based k-Nearest-Neighbor (kNN) are used in a sliding window approach: In a time window of fixed size, a set of features is computed using the raw sensor data. Then the sliding window is moved by an offset which determines the overlap with the last window. We used mean and variance as features, with windowsize 30 and stepsize 15. After that a majority decision was applied to the raw classification results. This yields a filtered decision for the particular gesture and constitutes the final result of the frame based classification. We used the YALE implementation of these classifiers. In addition to this frame based approach a Hidden Markov Models (HMM) based classifier is tested as well. For each manipulative gesture in our experiment a separate HMM is trained. During testing a single gesture is aligned with the most likely model. We used the HMM implementation in the Bayes Net Toolbox for Matlab for our experiments.
4
Results and Discussion
Results Due to the small data set we evaluated the data in a cross validation scheme. The classification results for the three classifiers are summarized in Table 2, a per class recognition rate for different sensor modalities for the kNN classification is given in Table 3, for the HMM classifier in Table 4. The following are the main observations: a. The recognition is far from perfect for all combinations of sensors and all classifiers. This was to be expected, as the gesture set was chosen to test the limits of the recognition rather than to be fully recognizable. Typically most errors occur for do/undo gesture pairs. Such pairs of classes are: C1,3 , C5,7 , C9,12 , C13,15 .
sensor
P
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
acc gyr fsr
81 65 76
80 58 54
99 100 93
58 10 55
100 99 100
73 80 85
75 73 90
79 64 74
100 93 100
61 11 45
99 46 87
93 90 88
63 31 34
69 51 59
93 94 82
55 46 80
100 99 98
acc+gyr fsr+acc fsr+gyr
90 86 84
91 71 66
100 100 96
63 61 65
100 100 100
94 85 89
92 100 95
81 78 78
100 100 100
80 77 74
97 96 92
98 97 91
83 62 48
89 76 80
100 99 92
82 87 84
100 100 98
0.7 0.6
0.4 0.3
V
ADC
[V]
0.5
0.2 0.1
Table 3. Recognition results per class for the kNN classi-
0
fication for different sensor modalities.
0.6
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
acc gyr fsr
83 72 73
89 43 63
95 93 79
60 35 51
98 96 100
73 72 74
99 86 81
79 76 76
100 79 99
58 32 53
99 87 51
98 94 70
67 56 68
64 68 74
90 88 76
66 59 63
95 87 88
acc+gyr fsr+acc fsr+gyr
91 84 81
87 74 56
96 98 88
69 55 58
100 100 100
92 83 81
99 96 91
88 77 81
100 100 100
74 74 66
98 76 73
100 83 86
85 74 73
95 82 76
100 93 93
74 80 77
97 94 91
Table 4. Recognition results per class for the HMM classification for different sensor modalities.
b. For all classifiers the overall accuracy of the FSR system is in the middle between the accelerometer (which is between 5% and 10% better) and the gyro (which is between 2% and 11% worse). This is also not surprising, since the accelerometers are mounted on the hand rather than the wrist. Thus, just like FSRs, they provide information not just on motions, but also on grasping which causes vibration on the back of the hand. At the same time the FSRs have less clear signals due to placement issues and what is more lack some motion information. On the other hand gyros lack the grasping information (reaction to the vibrations is minimal). c. Adding FSRs to other sensors always leads to an improvement (between a minimal 1% for HMM and acceleration and 19% for kNN and gyro). This is clear for gyros. For accelerometers it indicates that there is indeed some information that the FSRs have and an accelerometer - even when hand mounted - does not have. This is further confirmed by the fact that even in a single sensor case there are gestures for which the FSR is better than accelerometers.
Lessons Learned The main lesson of this work is that FSR based muscle monitoring is indeed useful for the recognition of activities involving hand actions. While inferior to accelerometers mounted on the hand for the overall accuracy on the 16 gestures, FSRs perform well for many individual gestures. On a few they are even better then the accelerometer, which confirms that there is some grasping related information that even hand mounted motion sensors can not detect. Note, that in our experiment FSRs rely on much less obtrusive arm mounting which makes them preferable for many applications.
0.4 0.3
V
P
ADC
[V]
0.5
sensor
0.2 0.1 0
0
2.5
5
7.5
0
1
2
0
1
2
3
time [s]
Figure 3. Signal samples for the upper 4 FSR channels, left column: 2 samples from class drink, middle: open notebook, right: close notebook.
Limitations Although the gesture set in the experiment has been chosen to be diverse, it cannot be claimed to be representative in any systematic way. In addition, even for the same gesture set the performance for a given sensing modality depends on fine tuning of features, window sizes, etc. which were not explored systematically in this work. Thus the lessons and conclusions discussed above must be taken as indicative rather than proven beyond doubt.
References [1] O. Amft, H. Junker, P. Lukowicz, G. Tr¨oster, and C. Schuster. Sensing muscle activities with body-worn sensors. In BSN 2006: Proceedings of the International Workshop on Wearable andImplantable Body Sensor Networks, April 2006. [2] L. Bao and S. Intille. Activity recognition from userannotated acceleration data. In Proc Pervasive Computing, 2004. [3] P. Lukowicz, F. Hanser, C. Szubski, and W. Schobersberger. Detecting and interpreting muscle activity with wearable force sensors. In Pervasive 2006, pages 101–116, May 2006. [4] J. Meyer, P. Lukowicz, and G. Tr¨oster. Textile pressure sensor for muscle activity and motion detection. In ISWC 2006: Proceedings of the 10th IEEE International Symposium on Wearable Computers, October 2006. [5] T. Starner, B. Schiele, and A. Pentland. Visual contextual awareness in wearable computing. In IEEE Intl. Symp. on Wearable Computers, pages 50–57, Pittsburgh, PA, 1998. [6] T. Stiefmeier, G. Ogris, H. Junker, P. Lukowicz, and G. Tr¨oster. Combining motion sensors and ultrasonic hands tracking for continuous activity recognition in a maintenance scenario. In 10th IEEE International Symposium on Wearable Computers (ISWC), October 11-14 2006. [7] J. Ward, P. Lukowicz, G. Tr¨oster, and T. Starner. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans. Pattern Analysis and Machine Intelligence, 28(10):1553–1567, October 2006.
