A Wearable Multi-Sensor System for Mobile Acquisition of Emotion-Related Physiological Data Christian Peter 1, Eric Ebert 1, 2, Helmut Beikirch 2 1
2
Fraunhofer Institute for Computer Graphics Rostock, Joachim Jungius Str. 11, 18059 Rostock, Germany
[email protected]
University of Rostock, Faculty of Computer Science and Electrical Engineering, Albert-Einstein-Str. 2, 18059 Rostock, Germany
[email protected]
Abstract. Interest in emotion detection is increasing significantly. For research and development in the field of Affective Computing and emotion-aware interaction techniques, reliable and robust technology is needed for detecting emotional signs in users under everyday conditions. In this paper, a novel wearable system for measuring emotion-related physiological parameters is presented. Currently heart rate, skin conductivity, and skin temperature are taken; further sensors can easily be added. The system is very easy to use, robust, and suitable for mobile and long-time logging of data. It has an open architecture and can easily be integrated into other systems or applications. The system is designed for use in emotion research as well as in everyday affective applications.
1 Introduction Affective Computing and Intelligent Interaction are key technologies to enable computers to observe, understand, and exhibit emotion. Researchers and engineers around the world work on ways to incorporate and exploit emotion in systems and applications, computer scientists investigate the role emotions play in HumanMachine Interaction, and psychologists, sociologists and anthropologists examine potential effects of affective systems on people and their relations to each other, machines, and media (cf. [1, 2, 3, 4, 5]). They all rely heavily on reliable and robust equipment to detect signs of emotions in the human in focus. There are a number of commercial systems available for measuring emotionrelated peripheral physiological parameters like skin conductivity, skin temperature, or heart rate. However, those systems have been developed for medical or psychological studies, which usually take place in fixed lab environments with a wired subject sitting fairly motionless in front of a display, occasionally hitting buttons on a keyboard or computer mouse. For studies on emotion in everyday human-computer interaction in natural settings at home or at work, in the office, the train, or on the aeroplane, the systems available proof to be unsuitable.
The system introduced in this paper has been designed for emotion researchers who want to examine emotional aspects of life outside the lab, for software developers who want to make use of emotion information in their systems without bothering about physiology, measurement artefacts, or filter chains, and of course also for psychologists who want to perform their studies in a natural setting without the irritating and distracting effects of wires on a subject. The system has been designed for use by lay-persons in an everyday environment, mobile or stationary, and without putting restrictions on the user’s behaviour. It gives the researcher and programmer the most possible freedom in handling the data, providing the measurements conveniently in engineering units. Developed with the researcher and application developer in mind, the device is fitted with robust and reliable error handling and diagnosis mechanisms, guaranteeing sensible data continuously being available along with reliability information. The system is small, light-weight, functions wirelessly, transmits data immediately and is also able to store data locally. It operates as long as 140 hours with one battery pack. It has an open architecture and sends out the data in an open format, allowing software developers to easily incorporate the device into their systems as emotion sensing input source. In the next section, requirements are defined for developing sensing devices for emotion-related physiological data to be used in HCI research and affect processing applications. A concept of a system meeting the identified requirements is developed in the subsequent section, followed by the description of a system implementing it. A discussion and outlook concludes the paper.
2 Requirements Currently available sensor systems such as Thought Technologies’ Procomp family, Mindmedia’s Nexus device, or BodyMedia’s SenseWear system are widely used in emotion research besides traditional medical devices like electrocardiographs (ECG), electroencephalographs (EEG), and electromyographs (EMG) for collecting emotionrelated physiological data. Except the SenseWear system, they all use traditional electrodes as sensor elements, which are attached to the subjects with tape or Velcro fastener, the wires being directly connected to the data collecting device. This not only irritates and distracts the subject from the task, but also hinders free and natural movements (confer e.g. [6]). The collected data are either stored locally on the device, or transmitted directly to the processing computer. In either case, the data can only be accessed, viewed and analysed using the manufacturer’s software and are not available instantly to other applications. If at all, data can be “exported” after the session for off-line analysis with third-party products. Affective applications, however, need direct and immediate access to the data to allow for continuous adaptation of the system to an ever-changing user state. There should be no need to use proprietary software. Also, there should always be sensible data available, freeing the programmer from caring about lost connections, transmission errors, badly fitted electrodes, and other technical side aspects. For the same reason, data should be made available to processing applications in engineering
units, avoiding inclusion of sensor-specific formulae into applications and eliminating the risk of conversion errors. The device needs to be very robust. In research experiments, subjects should not need to pay attention to wires or cables while performing tasks. In everyday applications, users won’t accept frail hard- or software. Basic sensors should be integrated within the system. There should be no need to fumble about with additional components in order to get sensible data out of the device. At the current state of the art, heart rate and skin conductivity are strong physiological indicators of emotion. Easy usage and a fixed position of the electrodes are prerequisite for reliable acquisition of physiological data. Difficult handling of loose sensor elements as is usual with currently available systems leads to differently attached electrodes, at different positions, with different pressure, for every individual. This is critical, as tests performed by us proofed that e.g. skin resistance taken at the hand differs by several hundred kilo ohms and sometimes even Mega ohms at positions just 1 cm apart. Also, different electrode pressure causes different temperature flows between electrodes and skin, influencing the value and speed of change of the measured temperature. Hence, usual methods to attach sensor elements with tape or Velcro lead to uncontrolled, different conditions for each subject of a study. Concerning the number and type of sensors used, it should be possible to easily add further sensors of any manufacturer to the system. An open protocol to allow third-party sensors feeding their data into the system is hence considered a desired feature. Finally, a small form factor, light weight, and operation over a longer period of time are essential for mobile real-world applications. They increase the willingness of subjects of non-lab and long-time studies to wear the apparatus and are important for the acceptance of everyday applications and system. Operation over a longer period of time is prerequisite for longitudinal research studies, which are very rare in the field of emotion research due to the lack of appropriate equipment. A device that can be used mobile over several days or even weeks will hence open new possibilities for emotion research and will yield new insights into everyday emotional response. Those requirements are main features identified necessary for a system to be used as emotion-related input device for affective applications and emotion-aware interaction techniques. As mentioned above, existing systems show different severe limitations each. A concept for new devices meeting the identified requirements is worked out in the following section.
3 Concept For a system to meet the criteria identified above, a distributed architecture with wirelessly connected components seems to be most appropriate. A sensor unit should be placed close to the body of the user, ideally hiding any wires and being comfortable to wear. A base unit takes care of receiving, validating, and storing the data locally, as well as of making them available immediately to processing applications. The base unit should be freely positionable within a sensible range of the sensor unit to allow, for instance, the experimenter to monitor the captured data
immediately and out of sight of the subject, or for the office worker to go for a cup of coffee without caring about the experiment he is involved in. Of course, the system should be able to cope with the base unit losing contact with the sensor unit, or multiple sensor units communicating with one or many base units at the same time. 3.1 Sensor Unit For acquisition of peripheral physiological data, sensor elements need to be in direct contact with the subject’s skin. To avoid long wires leading from the electrodes to the data processing electronics, the sensor unit should be close to the actual measuring point. To allow for an easy usage and hence increased acceptance of the system, all the electronics and wires should be invisibly integrated in a device or clothing usually worn by people, for instance a glove, wristband, brassiere, jewellery, or a headband. As has been shown in other studies (e.g. [7]), those integrated devices are quickly accepted and users soon forget about being monitored. The main task of the sensor unit is to capture the data and to perform basic preevaluations on them, like detecting measurement errors or sensor failures. If errors occur, appropriate action could be taken, like re-adjustment of the sensing elements, calibration of circuitry, or notification of the base unit. Before the data can be transmitted, they have to be wrapped in an appropriate transmission protocol, and a checksum has to be generated.
Figure 1: Block diagram of the system
3.2 Base Unit The base unit is to receive the data from one or many sensor units. Received data get a time stamp to allow assignment of the data to application events and to make it possible to correlate them with other data. After validating the checksum, data will be evaluated for sensibility, based on stored information on typical values and development characteristics of the data and possibly under consideration of other sensor’s results. For instance, when all sensors don’t send sensible data, it is likely that the sensor unit is not attached properly to the user. When just one sensor reports unusual data, there might be a problem with that sensor, or a single sensing element might not be properly attached. In case of a sensor error, the sensor unit could, for
instance, be reset. In case a bad attachment of the sensor unit is assumed, the user could be notified and asked for assistance. Based on the evaluation result, output data will be prepared in a next step. Since it is desirable to provide data continuously, missing or bad data have to be treated, for instance by filling gaps with likely data. Where alterations have been performed on the data, this has to be made known to the processing applications. Prepared like this, the data can be sent out continuously to a processing host. For applications with no permanent link to a processing host, the possibility to store the data locally has to be provided.
4 Implementation A sensor system has been developed [8] at Fraunhofer IGD Rostock which follows the concept above and offers additional, convenient features. Apart from the required measurements of skin conductivity and heart rate, skin temperature and ambient air temperature are measured. The base unit features a display to visualise the current state of the system, or data. Basic user input is possible by buttons, allowing, for instance, setting user events or toggling between visualizations. As a special feature for applications and mobile devices, the base unit can generate events, such as detection of certain physiological states, which are made available through an optical coupler to the outside world. Communication between sensor unit and base unit is done wirelessly using an ISM (Industry, Scientific, and Medical) band transmitter. With the currently used transmitter, the range is 10-30 meters indoors and up to 300 meters outdoors. The current version of the system uses a glove as garment hosting the sensor unit. 4.1 Sensor Unit All sensors except for the heart rate are integrated in a glove. This allows for short and hidden wires from the sensing elements for skin temperature and skin conductivity to the circuit board, which is also accommodated by the glove. The used heart rate receiver from Polar is designed for being mounted near the hand and is also integrated in the sensor unit’s electronics. As heart rate sensor, a conventional chest belt is used. The skin conductivity sensor is implemented two-fold. This helps to increase the robustness of the system significantly. Data evaluation checks can be performed more reliably based on those duplicate data. The skin temperature is taken at two different positions as well but integrated in one sensor, leading to higher accuracy and higher resolution. Also in the sensor unit, the ambient air temperature near the device is measured. This is another important factor for calculating sensor status and data reliability, since skin temperature and skin conductivity vary with changing environmental temperature. Skin temperature as well as skin conductivity are sampled 20 times per second each. Heart rate data are sent out by the heart rate sensor immediately after a beat has
been detected. The collected data are immediately digitized and assessed for sensor failure as described in the data validation section below. Based on the evaluation results, output data are prepared in a next step. In case of bad or temporarily no data from a sensor, previous data are used to estimate the likely current value. In this case, a flag in the transmission protocol is set accordingly. Wrapped into the protocol and fitted with a CRC check sum, the data are sent out permanently by the integrated ISM-band transmitter. 4.2 Base Unit The base unit receives the data transmitted from the sensor unit. Thanks to the wireless connection, multiple sensor units can be used, provided they have different identifiers. Immediately after they have been received the data get a time stamp. After a positive check sum evaluation, data are validated as described in the data validation section below. The validation results are stored along with the data and their time stamp on a local exchangeable memory card. If desired, they can also be sent out permanently to a host computer using a serial connection like traditional RS232, USB, or Bluetooth. Also in the base unit, the environmental temperature is taken again. This is an additional indicator on environmental changes the person might be exposed to and complements the according measurement of the sensor unit (while the sun might shine on the hands of the user, resulting in a higher local temperature near the hand, the room might still be chilly). All data are made available in engineering units. The temperature is represented in degree Celsius with a resolution of 0.01°C. The skin resistance comes in kilo ohms with a resolution of 300 kilo ohms. The heart rate comes in beats per minute with a resolution of 1 bpm. Transmission speed to a processing host is for each of the sensors 5 validated values per second.
Figure 2: Prototype of glove with sensor unit, and base unit (right).
4.3 Data Validation Data are validated throughout the system at two levels: a low-level evaluation is performed on the data in the sensor unit, and a higher level validation is carried out in the base unit with respect to usefulness of the data. Validation is based on the SEVA (Self Validation) standard [9], [10], [11]. Low Level Validation The sensor unit collects data from several sensors. In the configuration described in this paper there are two skin resistance sensors, one skin temperature sensor, one ambient air temperature sensor, and one heart rate sensor. Except for the air temperature sensor, all sensors need direct connection to the user’s body. As the user moves about sensing elements might become temporarily detached. Those cases are handled at the data level by the sensor unit. If no data are received from a sensing element, a “best estimate” is sent instead. A flag in the transmission protocol is set in this case, indicating that the data are estimates. If no data have been received from a sensor for a longer period of time (e.g. for 10 samples), a sensor status flag is set accordingly in the transmission protocol. High Level Validation Based on the data and sensor status received from the base unit and on the check sum result, the status of each sensor is analysed by the base unit continuously and validated SEVA data are generated. The following data are produced: • Validated measurement value (VMV): this corresponds to the actually measured value as long as no problem occurred and the data are in a sensible range. In case of problems, i.e. estimated data have been received or the sensor status being blind, further considerations on the most likely value are made. In case data move out of a predefined range or develop unusual (i.e. heart rate over 200 bpm, jump of skin temperature by 2 °C within 0.2 seconds), a VMV value will be estimated based on previously received data and stored measurand characteristics. • Validated uncertainty (VU): the VU denotes the likely error of the VMV. It can be seen as an “uncertainty band” enclosing the VMV. • Measurement value status (MVS): a discrete parameter indicating the reliability of the VMV. According to the SEVA standard, possible values are: clear (valid measurement), blurred (estimated data after known errors have occurred), dazzled (estimated data with uncertain cause of problems), and blind (estimated data of increasing uncertainty). • Device status (DS): the device status indicates how operational a sensor is. The SEVA standard proposes 6 states: good (no problems), testing (sensor in self-test), suspect (malfunction possible but not yet verified), impaired (malfunction with low impact), bad (serious malfunction diagnosed), and critical (sensor not operational). • Detailed diagnostics (DD): in case of an error, precise information on the error are provided, e.g. no data, wrong CRC, data out of range. Those SEVA data are stored in the base unit and communicated to processing applications, along with the data.
5 Discussion and Outlook A sensor system has been described for nearly unobtrusively collecting emotionrelated physiological data. The system is suited for mobile use over a longer period of time, storing the data on an exchangeable memory card or transmitting them immediately to a processing host. A robust data evaluation mechanism allows for continuous data availability and provides information on the reliability of the data and the sensor status. Processing applications can therefore make use of the data without the need of sensor-specific knowledge and without caring about technical side aspects such as badly fitted electrodes or transmission errors. An open protocol is used to communicate sensor data from sensor units to the base unit which allows integration of third-party products complying to the protocol. Data are made available to processing applications via an open protocol as well. The system is hence best suited as input device for any application processing emotionrelated physiological parameters. Fixed positions and constant pressure of sensing elements eliminate uncontrolled sensing conditions in emotion studies. Further developments of the system include a heart rate sensor without a chest belt, a respiration sensor, and a sensor for blood volume pressure. The communication protocol will be developed further to allow multiple sensor units and multiple base units operating close to each other at the same time. Also, more powerful ISM band transmitters will be evaluated.
References 1 2
3 4 5 6
7 8
9 10 11
Picard, R.W.: Affective Computing. M.I.T. Press, Cambridge, MA (1997). Arafa Y., Botelho L.M., Bullock A., Figueiredo P., Gebhard P., Höök K., Mamdani E. H., Paiva A., Petta P., Sengers P., Vala M.: Affective Interactions for in Real-time Applications: the SAFIRA Project (2004). Interacting with Computers. Volume 14, Issue 2, (2002). Interacting with Computers. Volume 16, Issue 4, (2004). Reeves B., Nass C.: The Media Equation. Center for the Study of Language and Information (1996). Haag A., Goronzy S., Schaich P., Williams J.: Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System. In: André et al (Eds.): Affective Dialogue Systems. Springer-Verlag Berlin, Heidelberg, New York (2004). Picard, R., Scheirer, J.: The Galvactivator: A Glove that Senses and Communicates Skin Conductivity. 9th Intl. Conference on Human-Computer Interaction, New Orleans (2001) Ebert, E.: Weiterentwicklung und Verifikation eines Messsystem zur Erfassung humanphysiologischer Sensordaten unter Berücksichtigung des SEVA-Konzepts. MsC Thesis, University of Rostock (2005). BS 7986: Industrial process measurement and control – Data quality metrics. Available from BSI Customer Services email:
[email protected] (2004). Henry M.P.: Self-Validating Sensors – Towards Standards and Products. Automatione e Strumentazione (2001). Zhou, F., Archer, N., Bowles, J., Clarke, D., Henry, M., Peter, C.: A general hardware platform for sensor validation. IEE Colloquium on intelligent and self-validating sensors, Oxford (1999).
