Interactive Emotional Content Communications ... - Semantic Scholar

D. K. Kim et al.: Interactive Emotional Content Communications System using Portable Wireless Biofeedback Device

1929

Interactive Emotional Content Communications System using Portable Wireless Biofeedback Device Dong Keun Kim, Jonghwa Kim, Eui Chul Lee, Mincheol Whang, and Yongjoo Cho Abstract — In this paper, we implemented an interactive emotional content communication system using a portable wireless biofeedback device to support convenient emotion recognition and immersive emotional content representation for users. The newly designed system consists of the portable wireless biofeedback device and a novel emotional content rendering system. The former performs the acquisition and transmission of three different physiological signals (photoplethysmography, skin temperature, and galvanic skin response) to the remote emotional content rendering system via Bluetooth links in real time. The latter displays video content concurrently manipulated using the feedback of the user’s emotional state. The results of effectiveness of the system indicated that the response time of the emotional content communication system was nearly instant, the changes of between emotional contents and emotional states base on physiological signals was corresponded. The user’s concentration was increased by watching the measuredemotion-based rendered visual stimuli. In the near future, the users of this proposed system will be able to create further substantial user-oriented content based on emotional changes. 1 Index Terms —Emotion, Emotional Content System, Portable Wireless Biofeedback Device.

I. INTRODUCTION Emotion communication technologies have become one of the most creative and attractive ways of improving affective interactions in the field of human computer interaction (HCI). Furthermore, interactive emotion communication can significantly improve the usage of an affective system and provide users with all the benefits of 1

This research was supported by the Ministry of Culture, Sports and Tourism (MCST) and the Korea Culture Content Agency (KOCCA) in the Culture Technology (CT) Research and Development Program 2011. Dong Keun Kim is with the Division of Digital Media Technology and Department of Emotion Engineering, Graduate School, Sangmyung University, Seoul, Korea (e-mail: [email protected]). Jonghwa Kim is with the Department of Emotion Engineering, Graduate School, Sangmyung University, Seoul, Korea (e-mail: [email protected]) Eui Chul Lee is with the Division of Fusion and Convergence of Mathematical Sciences, National Institute for Mathematical Sciences, Daejeon, Korea (e-mail: [email protected]). Mincheol Whang is with the Division of Digital Media Technology, Department of Emotion Engineering, Graduate School, and Culture Technology Institute, Sangmyung University, Seoul, Korea (e-mail: [email protected]). Yongjoo Cho is with the Division of Digital Media Technology, Sangmyung University, Seoul, Korea (e-mail: [email protected]). Contributed Paper Manuscript received 10/15/11 Current version published 12/27/11 Electronic version published 12/27/11.

natural interaction. Practically, understanding and responding to a user’s emotion provides considerable advantages in various attention-aware applications such as entertainment and game applications, making it possible for the system to react in a friendly manner based on the user’s needs [1]. Recently, many studies on implementing emotion communication systems to enhance emotional experiences have been proposed based on interesting efforts to recognize a human’s emotional state from audiovisual signals such as speech, facial expressions, and gesticulation [2–5,21-22]. The ability to precisely identify emotions based on facial expressions and speech recognition could lead to the development of very flexible user interfaces. However, this requires sophisticated data pre-processing steps such as for image regions, motion detection, and lexical analysis that involve the use of machine learning techniques for feature extraction. Moreover, these approaches can pose difficulties because the audiovisual data used for emotion estimation can be controlled and distorted by an individual user to generate a “fake emotion.” This is because such data are an indirect representation of a user’s emotion [6]. To overcome these obstacles, physiological signals can be used to recognize a user’s emotion. Physiological data require less processing power and are difficult to obscure. However, the biofeedback system used for the physiological data recognition systems is usually too bulky to be worn and carried around. Thus, users are often constrained to stay at a fixed location to use the physiological recognition system. Therefore, more handy and stable emotion recognition methods need to be developed using physiological signals. On the other hands, emotional contents are typically considered as external factors to represent the user’s emotion states. For example, some avatar characters in virtual reality applications can be controlled as emotion contents in terms of current user’s emotion states [7]. For affective interaction services, immersive emotional content representations also need to be resolved. Therefore, in order to have satisfactions with these requirements, designing emotion content rendering system enable to create emotion contents by controlling audial and visual information according to stimulates of human emotions can be a challenging issue. In this study, we implemented an interactive emotional content communication system. The purpose of the proposed system is to support convenient emotion recognition using portable wireless biofeedback devices in

0098 3063/11/$20.00 © 2011 IEEE

1930

IEEE Transactions on Consumer Electronics, Vol. 57, No. 4, November 2011

an unconstrained environment in real time and then displaying immersive emotional content corresponding to the user’s emotional states. In order to demonstrate the feasibility of the system for real-time operations, we tested the processing latency. The changes of between emotional contents and the emotional states detected by the physiological signal data as well as the concentration states in the gaze data were evaluated to determine the effectiveness of the system. This paper is organized as follows. In section II, both a brief overview of emotion recognition using physiological signals and a background of the emotion model are provided. The implemented aspects of the interactive emotional content communication system are discussed in section III. The simulated and experimental schemes and their results are respectively described in sections IV and V. Finally, the conclusions are summarized in section VI.

II. OVERVIEW OF THE PROPOSED SYSTEM This section presents the overview of the proposed system. In detail, we explain the emotion recognitions method using physiological signals, the adopted emotion model, and the workflow of the proposed system, as follows. A. Emotion Recognition using Physiological Signals As previously mentioned, real-time emotion recognition is a great challenge for the current methodology because of the high demands for robustness and comfort. Meanwhile, in the recent years, methods for emotion detection using physiological signals have been extensively investigated and have provided encouraging results where the affective states are directly related to changes in bodily signals [6,9]. In particular, autonomic nervous system (ANS) activity is considered to be a major component of an emotion response because the physiological signals based on ANS activity are very descriptive and easy to measure [6,8–9]. Furthermore, this makes it is possible to obtain reliable representations of true emotion that cannot be actively controlled. Therefore, we are working on recognizing the essential stages of a user’s emotions based on the physiological signals generated by the automatic nervous system, including PPG (photoplethysmography), SKT (skin temperature), and GSR (galvanic skin response), which are then mapped to the corresponding emotions. PPG is a simple and non-invasive method of taking measurements of the cardiac synchronous changes in the blood volume. Pulse waves are caused by periodic pulsations in the blood volume and are measured by the changes in optical absorption that they induce [10–13]. PPG sensors are often placed on the finger, earlobe, and toe to measure changes in the optical absorption in the capillary vessels. Changes in the amplitudes of PPG signals are related to the level of tension in a human. SKT data are used to measure the thermal responses of human skin. SKT depends on the complex relationship between blood perfusion in the skin

layers, heat exchange with the environment, and the central warmer regions of the skin. A peripheral response, like a change in skin temperature, can be used as an indicator of emotion. According to previous research, anger induces a large increase in skin temperature, whereas fear and sadness induce lower variations [10–13]. GSR is an indicator of the autonomic activity of physiological arousal. GSR data can be obtained by measuring the electrical conductance of the skin between two points, which varies with the moisture level of the skin. Therefore, GSR is a measured value of conductance or resistance obtained by using two electrode sensors [10–13]. Major magnitude changes in GSR signals are related to emotional excitement and dynamic activity in humans. B. Emotion Model Emotion is generally determined using either discrete or dimensional approaches. The discrete emotion model includes six basic emotions: anger, disgust, fear, joy, sadness, and surprise [6]. On the other hand, the dimensional approach is used to find independent dimensions for categorizing emotions. In two-dimensional models, the emotional cues are identified by registering their values on two orthogonal dimensions that code the degree of pleasure and the degree of arousal. According to previous researches, the physiological parameters used for emotion recognition are not correlated with certain emotional states, but are instead related to the underlying dimensions [6,14]. Therefore, the two-dimensional model is used to represent the current user’s emotional states in terms of continuous physiological changes and the measurement of affective states. We can also recognize an increase or decrease in excitement or depression because the two-dimensional model can demonstrate variations in the magnitudes of the emotional states. Therefore, among the many emotion models, we adopted Russell’s emotion model, where the two dimensions are represented by a vertical arousal axis and a horizontal valence axis, as shown in Fig. 1 [15].

Fig. 1. Diagram of two-dimensional emotion model and four targeted emotional states.


C. The System Workflow In this section, we try to explain the workflow of the proposed system. In essence, the framework of the proposed system can be used to interpret the user’s constantly changing emotions based on his or her physiological signals and to present the newly manipulated emotional contents, which correspond to the user’s emotional states. The system operation and workflow are illustrated as a sequence diagram in Fig. 2.

1931

sensors, the portable wireless biofeedback device, and the emotional content rendering system. In this section, we describe each component.

Fig. 3. Overall architecture of implemented interactive emotional content communication system.

A. Hardware - Physiological Sensors

Fig. 2. Operation flows of interactive emotion content communication system.

The possible scenarios for the interactive emotion communication and its responses are described as follows: (1) While the user is staring at the emotional content rendering system with the designed portable wireless biofeedback device, physiological data from the user are acquired through the PPG, SKT, and GSR sensors. (2) The acquired data are transmitted to the designed portable wireless biofeedback device and processed for signal sampling. (3) The sampled signal data are placed in a packetized format and transmitted to the remote emotional content rendering system via Bluetooth links in real time. (4) After processing the signal data, the estimated emotion results, with programmable code values, are generated. (5) Based on the results of the current user’s emotional state, the colors displayed by the emotional content rendering system are changed. (6) The immersive emotional content affected by emotions are rendered on the emotional content rendering system, and, consequently, the user can be stimulated as a result of continuous staring at the emotional content.

III. IMPLEMENTATION OF THE PROPOSED SYSTEM The overall architecture of the implemented interactive emotional content communication system is depicted in Fig. 3. The architecture of the designed system is composed of three major components: the user, with the designed physiological

Fig. 4. Three designed physiological sensors: PPG, SKT, and GSR.

Visual stimuli for emotion induction are produced by the watching of the emotional content rendering system. Physiological data are measured using four physiological electrode sensors: a PPG sensor, SKT sensor, and two GSR sensors. All these sensors are mounted on small bands worn around the fingers, which facilitate the measurement of data and protect the sensors from ambient light, as shown in Fig. 4. The PPG sensor is made of reflective object sensors. The SKT sensor is a glass-type negative temperature coefficient thermistor with a 10-kΩ electric resistance property. B. Hardware - Portable Wireless Biofeedback Device F Fig. 5 depicts the designed portable wireless biofeedback device. The size of the prototype device is 250 × 50 × 150 mm3. This device supports the capture of physiological signals using the PPG, SKT, and GSR sensors, as well as the sampling and transmission of these signals to the emotional content rendering system via Bluetooth connections. The data sampling and transmitting process was performed on the Bluetooth and Arduino modules shown in Fig. 5(a) and (b), respectively. The Bluetooth module, which acted as a slave node, has a 2.4-GHz frequency band and 2.5mW of transmitting power as a Class 2 device. The available transmission speed rates range from 1 Mb/s to 3 Mb/s.

1932


Fig. 5. Prototype of portable wireless biofeedback device.

As shown in Fig. 6, in the signal sampling step, PPG, SKT, and GSR data are extracted using a 200-Hz sampling rate. Because of the use of three different signals, we use dynamic memory allocations with a circle queue memory structure to improve memory usage at the buffer controlling step. Next, the obtained physiological signals are transmitted by the Bluetooth link to the remote emotional content rendering system.

Fig. 6. Diagram of system for physiological signal sampling and transmission in portable wireless biofeedback device.

For the wireless transmissions, the device generates a packetized format with serialized PPG, GSR, and SKT signals as digitized data using the interleaving method, as shown in Fig. 7. This packetized format consists of PPG, GSR, SKT, and time stamp data. Two hundred packets of data are processed to transfer per second. The data transmission rate of the Bluetooth module was 150 kb/s.

Fig. 7. Diagram of physiological signal sampling and transmission in portable wireless biofeedback device.

C. Software – Emotion Estimation Once the physiological signals are transmitted, it is necessary to extract features from the signals for the emotion estimation, which can then be applied to the emotional contents. In order to analyze the physiological signal processing for emotion recognition, a time-dependent parameter (TDP) method is adopted as a reference, as studied earlier [16]. The TDP is determined from the normalization and running averages of the physiological signals to find the tonic and phasic responses according to emotion. Therefore, in the signal processing step, the stimulus state of each physiological signal is normalized from the neutral state. To minimize any individual differences in physiological readings, a neutral band is adopted as a reference, as studied earlier. This neutral band is used to normalize the physiological data [17]. After this signal processing, the sliding window method is adopted for data normalization to reduce the possibility of signal noise in the PPG, SKT, and GSR signals. This sliding window method requires less memory and is thus suitable for emotion classification system applications that have memory constraints [18]. Therefore, the PPG frequency, amplitude, GSR average, and SKT average parameters are generated with every 1-s unit. The unit size of the sliding window is 2 s, which contains 400 data packets. The time interval is 0.5 s, which contains 100 data packets. The sliding window keeps moving to the next 400 packets of data. In the emotion estimating step, emotion results as physiological parameters are classified, and programmable emotion status codes are generated. The emotion status codes are scaled from 1 to 9 in terms of the different emotion states given in Table I. These are associated with the aforementioned quadrants of emotion in Russell’s emotion model. Therefore, the model of these nine emotions will be employed in this system for emotion recognizing purposes. As shown in Fig. 1, the circular order of these nine emotions (including Neutral) in Russell’s model is Pleasure, Excitement, Arousal, Distress, Displeasure, Depression, Sleepiness, and Relaxation (contentment). For example, the emotion status code “0X01” represents the “Excitement” state. D. Software - Emotion Content Rendering System The emotion content rendering system provides various modules built on top of the three-dimensional graphics library for the Microsoft Windows platform to support desktop processing. The content rendering module contains predefined rules that describe how to change the video properties of three-dimensional environments. When a user’s emotion is estimated and the pre-defined rule set according to the emotion state is found, the rule set is used for the visual rendering component. The pre-defined rules specify the next steps when a specific emotion state is detected. For example, when the emotion state is “Pleasure,” the emotion state code is “0X05” and the pre-defined color is “Red.” For rendering video content, the visual rendering module is built on top of the open source three-dimensional graphics library, Open Scene Graph, designed for desktop platforms.

D. K. Kim et al.: Interactive Emotional Content Communications System using Portable Wireless Biofeedback Device TABLE I THE EMOTION STATE CODES AND CORREDPONDED VIDEO COLOR DESCRIPTIONS. Emotion State Codes(Hex Format)

Emotion States

0X01

Excitement

RGB(254, 126, 0)

0X02

Distress

RGB(231, 221, 136)

Video Controls (Color Descriptions)

0X03

Depression

RGB(80, 91, 99)

0X04

Contentment

RGB(179, 98, 18)

0X05

Pleasure

RGB(230, 0, 0)

0X06

Arousal

RGB(255, 240, 25)

0X07

Misery

RGB(25, 59, 108)

0X08

Sleepiness

RGB(165, 193, 215)

0X09

Neutral

RGB(148, 191, 38)

1933

system, user-dependent calibration (gazing at the four corner positions of a display) is required in order to define the pupil’s movable area, as shown in Fig. 9.

IV. EXPERIMENTS In the first experiment, the processing latency of the portable wireless device was assessed to determine the feasibility of using it for real-time operations. The changes of between the emotional content and the emotional states detected by the physiological signal data were evaluated to determine the effectiveness of the proposed system. For a reliable test, they experienced 10 s of stimulation and 30 s of rest to induce a neutral emotional state. The distance between the participants and the emotional content rendering system was 1 m. The participant stared for 3 min at video clips displayed on the emotional content rendering system. Then, the color property was changed, based on the user’s emotional state, and physiological signals were measured with the designed portable wireless biofeedback device. The tiled display system used 18 monitors (LCD monitor 23 in), which each had a 2048 × 1152 screen resolution, as shown in Fig. 8.

Fig. 9. Concentration in gaze position experiment using gaze tracking camera device.

The gaze position (xm, ym) can be calculated by mapping the center of the pupil over the four calibrated pupil centers ((x1, y1), (x2, y2), (x3, y3), and (x4, y4)) on the display coordinate system based on a geometric transform with 8 degrees of freedom [20]. In Fig. 9, w and h represent the horizontal and vertical pixel resolutions, respectively.

Fig. 10. Conceptual diagram of mapping from a pupil’s movable area to the display coordinate system [19].

Fig. 8. Emotional content rendering system with tiled display configuration using portable wireless biofeedback device.

In the second experiment, the concentration difference from the gaze position of the displayed emotional contents was assessed to determine immersion of emotional contents for effectiveness to users. Comparative verifications were performed between the two groups (given the rendered visual content and the original one, respectively). To measure the amount of concentration, our previously developed gaze tracking method was adopted [19]. To use the gaze tracking

For a comparative measurement of the concentrations, each group included 5 subjects. Although the physiological signals were not required from one group, they wore the physiological sensors in order to test fairly with the other group. The experiments involved the watching of a 3 min video clip. In our analyses, the gaze-based concentration of each group is estimated by calculating the gaze point deviations of five subjects for one video frame. To obtain the concentration rate for one video frame, the center of the five subjects’ gaze positions (xc, yc) should first be calculated using the following equation:

( xc , y c ) 

1 5 i i  ( xm , y m ) n i 1

(1)

Then, the variation in the five subjects’ gaze positions from the above calculated center is obtained by calculating the

1934


Euclidean distances in the following equation:

Var 

1 5 ( xc  xmi ) 2  ( yc  ymi ) 2  c  n i 1

(2)

In equation (2), Var represents how distributed the gaze positions are from the center gaze position. Here, normalization term c is defined as half of the diagonal screen pixel length, which is the potential maximum value of Var. In our system, because the horizontal and vertical resolutions of the tiled display used were 6144 and 2304 pixels, respectively, c was defined as 3281. Consequently, Var could be normalized to the range of 0 to 1. That is, 0 means that all the subjects gazed at exactly the same position in the same video frame. Assuming that equivalent visual stimuli causing high concentration were given to the subjects of one group, they gazed at a similar position, which is regarded as a region of interest. Consequently, the metric for estimating the concentration rate of a group (Con) is defined as the reciprocal of Var in equation (3).

Con 

1 Var

emotional content stimuli. The PPG frequency and GSR increased from 20 s to 27 s, and a similar response was occurred at 95 s. The physiological pattern and emotion state of the former, and the latter were corresponded to the “Pleasure”, and “Excitement”, respectively. The “Excitement” response persisted for approximately 15 s. Therefore, the video clip content in the emotional content rendering system was masked with RGB (254, 126, 0) because these reactions were classified as an “Excitement”.

(3)

V. EXPERIMENTAL RESULTS A. Processing Delay Table II lists the refresh time and latency analyzed by the emotion recognition software. The latency is the turnaround time between the measuring of the physiological signals and the emotion recognition. The refresh time is the time delay between subsequent emotion recognitions. The mean latency was 0.02 ms, and the mean refresh time was 4 s (Table II). According to the results, the emotion recognition software could recognize the emotion 15 times per min. We found a physiological detection with no in-between pauses longer than 50 ms to be a good compromise for signal processing. The experimental results showed that the method can meet the requirements for monitoring the emotions of the target population in real time. TABLE II THE EMOTION STATUS CODE DESCRIPTION Subject

Latency (ms)

Refresh Time (s)

Numbers of Refresh

A B C D E Mean

0.0190 0.0235 0.0150 0.0460 0.0380 0.0283

Mean=4.0, SD=0.0003 Mean=4.0, SD=0.0004 Mean=4.0, SD=0.0006 Mean=4.0, SD=0.0005 Mean=4.0, SD=0.0006 M=4.0

15 15 15 15 15 15

B. Emotional Content Communications According to the test results, as shown in Fig. 11, the physiological signal pattern of the participant was recognizable during watching the video clip due to the

Fig. 11. Emotional contents and the corresponded emotional states detected by the physiological signals.

C. Concentration Differences As mentioned in section IV, comparative verifications of the two groups (the one shown the rendered visual content and the original one) were performed in terms of the concentration rate (Con). Table III lists the average concentration rates (Con) of the two groups: the rendered visual content group and the no rendered visual stimuli group. TABLE III THE COMPARISON CONCENTRATION RATES ON THE BASIS OF EYE TRACKING METHOD Group No rendered visual stimuli Rendered visual stimuli

Var

Con

0.076(±0.039)

13.147

0.039(±0.021)

25.037

As given in Table III, the group shown the rendered visual stimuli showed a greater concentration rate. This difference was statistically very significant, which was verified by a t-test


(p value = 5.26 × 10-17). In other words, the subjects in this group generally gazed at similar positions compared with the group shown the original visual stimuli.

1935

However, emotion has a very complicated structure and is exhibited differently in different people. In addition, physiological variables frequently differ from one person to another and are very sensitive to day-to-day variations, as well as to the context of the emotion induction. For these reasons, in order to achieve interactive emotion communication services, multimodal emotion recognition is proposed as an appropriate subject for further study. In the near future, users of this proposed system will be able to create substantial useroriented content based on changes in emotion. REFERENCES [1] [2] [3]

[4] [5] [6] [7] Fig. 12. Examples of not rendered (top) and rendered (bottom) scenes and their gaze diffusions based on gaze positions (center of red circle: average gaze position; radius of red circle: gaze diffusion by xvar).

For example, the gaze position diffusion rates of the two groups are visualized, as shown in Fig. 12. By comparing the gaze diffusions (sizes of the circles), we also confirmed that the group shown the rendered visual stimuli showed a smaller circle, which corresponded to a higher concentration rate. This showed that our proposed emotion feedback system operated adequately and appropriately rendered the visual stimuli in terms of inducing human concentration. VI. CONCLUSION An interactive emotional content communication system is one of the most advanced modern technology paradigms for human-computer interaction services in ubiquitous computing environments. In order to increase popular usages of emotional interaction services, handy emotion recognitions, interactive emotion responses, and immersive emotional contents need to be supported. Therefore, in this study, we proposed a novel interactive emotional content communication system using a portable wireless biofeedback device to support convenient emotion recognition and immersive emotional content representation for users. In the experiment, the response time of the emotion communication was very quick, the emotional interaction also agreed with the user’s emotions, and users could concentrate on emotional contents.

[8] [9] [10] [11]

[12] [13]

[14] [15] [16]

[17]

[18]

R. Cowie, E. D. Cowie, N. Tspapatsoulis, G. Votsis, S. Kollians, W. Fellenz, and J. G. Taylor, “Emotion Recognition in Human Computer Interaction,” IEEE Signal Proc. Mag., vol. 18, no. 1, pp. 32-80, Aug, 2002. G. Castellano, L. Kessous, G. Caridakis, “Emotion Recognition through Multiple Modalities: Face, Body, Gesture, Speech,” Lect. Notes Comput. Sci., vol. 4868, pp. 92-103, 2008. C. Y. Chang, 1. S. Tsai, C. J. Wang, and P. C. Chung, “Emotion Recognition with Consideration of Facial Expression and Physiological Signals,” in Proc. IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology, pp.278-283, 2009. Y. Wang and L. Guan, “Recognizing Human Emotional State from Audiovisual Signals,” IEEE Trans. on Multimedia, vol. 10, no. 4, June. 2008. Z. Zeng, J. Tu, B. M. Pianfetti, Jr., and T. S. Huang, “Audio–Visual Affective Expression Recognition through Multistream Fused HMM,” IEEE Trans. on Multimedia, vol. 10, no. 4, June. 2008. K. Jonghwa and E. Ande, “Emotion Recognition Based on Physiological Changes in Music Listening,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, pp. 2067-2083, 2008. R. A. Calix, S. A. Mallepudi, B. Chen, and G. M. Knapp, “Emotion Recognition in Text for 3-D Facial Expression Rendering,” IEEE Trans. on Multimedia, vol. 12, no. 6, pp. 544-551, Oct, 2010. C. Collet, E. V. Maury, G. Delhomme, and A. Dittnar, “Autonomic Nervous System Response Patterns Specificity to Basic Emotions,” J. Auton. Nerv. Syst., vol. 62, pp. 45-57, 1997. S. D. Kreibig, “Autonomic Nervous System Activity in Emotion: A Review,” Biol. Psychol., vol. 84, no. 3, pp. 394-421, 2010 D. W. Ryoo, Y. S. Kim, and J. W. Lee, “Wearable Systems for Service Based on Physiological Signals,” in Proc. IEEE Engineering in Medicine and Biology 27th Annual Conference, pp.2437-2440, 2005 S. Jerritta, M. Murugappan, R. Nagarajan, and K. Wan, “Physiological Signals Based Human Emotion Recognition: A Review,” in Proc. IEEE 7th International Colloquium on Signal Processing and its Applications (CSPA), pp. 410-415, 2011 C. Y. Chang, J. Y. Zheng, and C. J. Wang, “Based on Support Vector Regression for Emotion Recognition using Physiological Signals,” in Proc. International Joint Conference on Neural Networks, pp. 1-7, 2010. J. H. Kim, M. C. Whang, J. Y. Kim, and J. C. Woo, The study on emotion recognition by time dependent parameters of autonomic nervous response, Korean J. Sci. Emo. & Sens., vol. 11, pp. 637-644, 2008. C. Peter and A. Herbon, “Emotion representation and physiology assignments in digital systems,” Interact. Comput., vol. 18, pp.139-170, 2008. J. A. Russell, “A circumplex model of affect,” J. Pers. Soc. Psychol., vol.39, no. 6, pp. 1161-1178, Mar. 1980. J. H. Kim, M. C. Whang, and J. C. Woo, “The Research on Adaptive Process for Emotion Recognition by using Time-Dependent Parameters of Autonomic Nervous Response,” Lect. Notes Comput. Sci., vol. 5610, pp. 77-84, 2009. M. C. Whang, J. S. Lim, K. R Park, Y. J Cho, and W. Boucsein, “Are Computers Capable of Understanding our emotional states?” in Proc. the 7th International Conference on Engineering Psychology and Cognitive Ergonomics, vol. 4562, pp. 204-211, 2007. F. Honig, D J. Wagner, A. Batliner, and E. Noth, “Classification of user states with physiological signals: On-line Generic features vs. Specialized feature sets,” in Proc. 17th European Signal Processing Conference 2009 Glasgow, Scotland, pp. 2357-2361, 2009.

1936


[19] E. C. Lee, K. R. Park, M. C. Whang, and J. Park, “Robust Gaze Tracking Method for Stereoscopic Virtual Reality Systems.” Lect. Notes Comput. Sci., vol. 4552, pp. 700-709, 2007. [20] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed., Prentice-Hall: New Jersey, 2002. [21] I. Bacivarov, P. Corcoran, M. Ionita, “Smart Cameras: 2D Affine Models for Determining Subject Facial Expressions,” IEEE Trans. Consumer Electron, vol. 56, no. 2, pp. 289-297, 2010 [22] K. H. An, M. J. Chung, “Cognitive face analysis system for future interactive TV,” IEEE Trans. Consumer Electron, vol. 55, no. 4, pp. 2271-2279, 2009 BIOGRAPHIES Dong Keun Kim received a B.S. degree in Information of Telecommunication from Sangmyung University in 2001, an M.S. degree in Medical Information Systems from Yonsei University in 2003, and a Ph.D. in Biomedical Engineering associated with Tele-communication Systems from Yonsei University in 2008. Currently he serves as assistant professor at the Division of Digital Media Technology and Department of Emotion Engineering, Graduate School, Sangmyung University, Seoul, Korea. His research interests include digital signal multimedia processing, and Human Computer Interaction (HCI). Jonghwa Kim received an M.S. degree in Computer Science from Graduate School, Sangmyung University, Seoul, Korea, in 2009. He has been a Ph.D. candidate in the Department of Emotion Engineering, Graduate School, Sangmyung University, Seoul, Korea since 2010. His research interests include Human Computer Interaction, and emotion engineering.

Eui Chul Lee received a B.S. degree in Software from Sangmyung University, Seoul, South Korea in 2005. He received M.S. and Ph.D. degrees in Computer Science from Sangmyung University in 2007 and 2010, respectively. He has been a Researcher in Sciences at the National Institute for Mathematical Sciences (NIMS), Daejeon, since March 2010. His research interests include computer vision, image processing, pattern recognition, ergonomics, brain-computer interfaces (BCIs), and human-computer interfaces (HCIs). Mincheol Whang received the M.S. and Ph.D. degrees in Biomedical Engineering from the Georgia Institute of Technology, Atlanta, Georgia, USA, in 1990 and 1994, respectively. He has been a Professor in the Division of Digital Media Technology and Department of Emotion Engineering, Graduate School, Sangmyung University, Seoul, Korea since March 1998. His research interests include human-computer interaction, emotion engineering, human factors, and bioengineering. Yongjoo Cho is currently an assistant professor at the Division of Digital Media, College of Software, Sangmyung University, Korea. He earned his B.S. in Computer Science at the University of Illinois at UrbanaChampaign and his M.S. and Ph.D. degrees at the University of Illinois at Chicago. He is currently the director of the Interactive Computing and Entertainment Laboratory at Sangmyung University. His research interests include Virtual Reality, computer supported cooperative work, and virtual learning environments.