29th International Design 2009 Engineering Technical Conferences (IDETC) & Computers and Proceedings of the ASME International Design Engineering Technical Conferences & Information in Engineering Conference (CIE) Computers and Information in Engineering Conference ASME IDETC/CIE IDETC09 2009 30-September 2009,Diego, San Diego, USA USA August 30August - September 2, 2009,2,San California,
DETC2009-86542 A PILOT STUDY TO ASSESS DESIGNER’S MENTAL STRESS USING EYE GAZE SYSTEM AND ELECTROENCEPHALOGRAM Harshad Petkar and Shivangi Dande Institute for Information Systems Engineering Faculty of Engineering and Computer Science Concordia University 1455 de Maisonneuve Blvd. West Montreal, Quebec, Canada H3G 1M8 Yong Zeng Institute for Information Systems Engineering Faculty of Engineering and Computer Science Concordia University 1455 de Maisonneuve Blvd. West Montreal, Quebec, Canada H3G 1M8 E-mail:
[email protected] ABSTRACT The study of mental stress is of great importance to design in that it enhances our understanding of designer’s cognitive model during the creative design process, among others. As the first step of this effort, this paper focuses on the assessment of mental stress based on the analysis of electroencephalogram (EEG) signals and the eye related data. The stress stimuli used is the computer based Stroop test with six difficulty levels. By using different parameters such as EEG power bands, and other eye behavior data, human mental stresses were assessed. Results indicate a strong correlation between the recorded physiological signals and the emotional state of the designers. This study provides a baseline for the further analysis of designer’s and users mental stresses during design-related tasks. 1 BACKGROUND Over the last few decades, various design methodologies have been proposed to assist designers in generating quality designs in an effective manner [1-6]. There is no doubt that the existing methodologies have been greatly influencing the industrial product design process. However, a big challenge is still faced in applying those design methodologies, which lies in two contrasting facts. On the one hand, design is a creative act, which is rooted in the flexibility and freedom for exploring various avenues to achieve design goals [7-10]. On the other hand, any design methodology implies a set of well structured logical steps for solving a design problem. This contradiction between flexibility and structure and between freedom and
Rajeev Yadav Department of Electrical and Computer Engineering Faculty of Engineering and Computer Science Concordia University 1455 de Maisonneuve Blvd. West Montreal, Quebec, Canada H3G 1M8 Thanh An Nguyen Institute for Information Systems Engineering Faculty of Engineering and Computer Science Concordia University 1455 de Maisonneuve Blvd. West Montreal, Quebec, Canada H3G 1M8
logic is made even more complicated by an intrinsic nature of design: design solutions must pass an evaluation defined by design knowledge that is interdependently and recursively determined by the design solutions to be evaluated [11]. To develop an effective, structured and logical design methodology that can accommodate flexibility and freedom, it is important to quantify the designer’s cognitive processes, particularly the designer’s mental stress. According to the Yerkes-Dodson law [12], there is an inverted U-shaped curve correlation between performance and mental stress. An optimal level of arousal addresses the best performance for a given task whereas performance will decrease when levels of arousal become either too low or too high. Their research indicated that under the pressure of tight schedule, complex tasks, and/or other intensive tasks, people can be stressed out; as a result, their performance may degrade or even fail. Modest levels of supraoptimal stress can be counteracted by the performer by increased effort, or resource mobilization, or straining [13-16]. A design methodology, as a means to manage design tasks, can increase or decrease the designer’s mental stress. Thus a design methodology may bring in the mental relaxation by helping designers release mental workload while it may also lead to designer’s frustration if it is against the designer’s conventional way of thinking and working [17]. Therefore, a good design methodology must be able to enhance the designer’s performance by keeping the designer’s mental stress within an optimal range. For this purpose, it is
1
Copyright © 2009 by ASME
essential to model the relationship between designer’s mental stress and performance. As the first step of this modeling, the designer’s mental stress and performance must be quantified. The quantification of designer’s mental stress was conducted by analyzing the linguistic information collected from verbal protocol analysis [18]. In the literature, most of the reported work is focused on understanding general cognitive processes in design [19]. By focusing on the increase of the germane cognitive load, Kirschner aims at further improving instructions by making designers take advantage of otherwise unused working memory during learning [20]. Dong attempts to quantify coherent thinking by using a latent semantic analysis in a conversation mode, and this measurement also reveals patterns of interrelations between an individual’s ideas and the group’s ideas [21]. Stempfle and Badke-Schaub have investigated the cognitive processes of design teams during the design process by studying three laboratory teams solving a complex design problem [22]. Fuchs-Frohnhofen, et al. have analyzed the user’s mental models in the work setting and have generated variants of human-machine interfaces matching the user’s mental models by using a methodology incorporating the taxonomy of mental models [22]. The work reported in this paper is the first step in our effort to quantify designer’s mental stresses during the conceptual design using eye signals and electroencephalography (EEG) signals. It should be noted that although the experimental task in this paper is independent of design tasks and applies to any other human cognitive activities, the results from the experiments are used as the baseline for our on-going analysis of experimental data collected from conceptual design tasks. The rest of this paper is organized as follows: Section 2 gives an introduction to mental stress and corresponding physiological signals. Section 3 presents the methodology. Data processing and data analysis is given in Section 4 and 5, respectively. Section 6 presents the result and finally Section 7 concludes the paper. 2
ASSESSMENT OF MENTAL STRESS
2.1 Mental stress Mental stress is a normal physical response to both internal and external events that make people feel fatigued or threatened. Mental stress affects every day human life and also affects human work performance. Though we cannot measure mental activities directly, we can turn to neural responses [23]. The autonomic nervous system is a closed-loop automatic control system to balance action and re-action process within our body shows in Figure 1. Stress causes disturbances in the equilibrium of this control system. An elevated and prolonged stress level is a silent killer and cannot be noticed without the help of advance neurophysiologic monitoring tools.
Figure 1 Autonomic nervous system [24] Many researchers assessed mental stress by using heart rate variability (HRV), body language, but the bioelectrical activity of the brain and eye signals are more closely to the information processing during mental stress[24]. Hence, these signals will be recorded for analysis to assess mental stress in the present work. 2.2 Electroencephalogram (EEG) data EEG records electrical activities of the brain using electrodes placed over the scalp [25]. The electrodes are placed over the scalp using 10-20 electrode placement system commonly referred as montage. The montage is specific to the study or to an experiment. There are four following different types of EEG waves [26]. Delta waves lie within the range of 0 to 3 Hz, with variable amplitude. Delta waves are primarily associated with deep sleep [26]. Theta waves lie within the range of 3 to 7 Hz, with an amplitude usually greater than 20µV. Theta wave is associated with emotional stress in adults [26]. Alpha waves lie within 8-13 Hz with 30-50 µV amplitude. Beta waves lie within 13-30 Hz and usually have a low voltage between 5-30 µV. Beta waves are usually associated with active thinking, active attention, and problem solving [26]. EEG signals were recorded from four different scalp locations (Fz, Cz, Pz, Oz). Fz location is for emotional control, Cz location is for sensory and motor functions, Pz is for perception and differentiation, Oz is for visual areas [27]. 2.3 Eye parameters Stress induced in an individual shows significant changes in eye parameters. An average range of pupil diameters, which is 2-8mm [28], increases under stress [29, 30]. Differences in pupil diameter can be associated with differences in mental workload [31]. Other factors affected by mental stress include PERCLOS, blinking rate and inter-blink interval. PERCLOS is
2
Copyright © 2009 by ASME
the proportion of time during which the subject droops [31]. PERCLOS detects fatigue induced by the lack of attention. It increases if a person feels sleepy or drowsy. Blinking rate is related to cognitive state and is useful to measure stress [32]. The reduced blinks show that increase in visual demands on the task [33, 34]. Increase in blinking rate is also an indicator of fatigue [34, 35]. If a subject feels drowsy, then the blinking duration increases, which happens especially when the time on task increases [36]. Inter-blink interval (IBI) refers to the time interval between two blinks. As the difficulty level increases the IBI also increases [37]. In our experiment, FaceLab 4.5 eye tracking system is used to capture various eye parameters. The system is configured according to the physiological characteristics of eyes and head of the subject using precision mode. The tracking quality and the gaze quality are kept at the highest level throughout the experiment. The tracking is frame to frame in real time, and the frame rate is 60 frames per second. The log files generated are converted into text format. The output data comprises of all the eye and head movements. In addition to FaceLab, electro-occuography (EOG), a technique used to capture eye related movements, was also recorded. 3
METHODOLOGY
3.1 Stroop test One’s mental stress depends on one’s knowledge and experience related to the problem as well as many other cognitive parameters. Given the same design problem, different designers will have different brain activities which correspond to different EEG wave patterns and eye behavior. Therefore, it is indispensable to define a baseline in quantifying the mental stresses from different designers. In our experiment, Stroop test is used to achieve this goal and this is the focus of this paper. The experimental result on design problems will be presented in a separate paper. Stroop test, a color naming task, is a classical paradigm in neurophysiologic assessment of mental fitness [38]. The Stroop test is a demonstration of interference in the reaction of the task. In our experiment, the Stroop test is designed as a computer game in which a subject is presented a color name, referred as stimulus word. The stimulus word is displayed in a color which is the same as or different from that it refers to. The subject has to select the answer corresponding to the color of the word. For example, given a GREEN word in BLUE color, the subject has to select the word BLUE in the answer list. Our Stroop test contains six colors: RED, BLUE, YELLOW, PURPLE, GREEN and BLACK, and six difficulty levels as listed below: 1) Difficulty level 1 (DL1): a stimulus word is displayed in the color referred by itself and each word in the answer list is displayed in the color referred by itself. For example, the stimulus word ‘RED’ is written in red color and the word ‘RED’ in response list is also written in red color.
2) Difficulty level 2 (DL2): a stimulus word is displayed in the color referred by itself and each word in the answer list is displayed in black color. For example, the stimulus word ‘RED’ is written in red color and the word ‘RED’ in response list is written in black color. 3) Difficulty level 3 (DL3): a stimulus word is displayed in a color different from that it refers to and each word in the answer list is displayed in black color. For example, the stimulus word ‘RED’ is written in blue color and the word ‘RED’ in response list is written in black color. 4) Difficulty level 4 (DL4): a stimulus word is displayed in a color different from that it refers to and each word in the answer list is displayed in the color referred by itself. For example, the stimulus word ‘RED’ is written in blue color and the word ‘RED’ in response list is written in red color. 5) Difficulty level 5 (DL5): a stimulus word is displayed in the color referred by itself and each word in the answer list is displayed in a different color that it refers to. For example, the stimulus word ‘RED’ is written in red color and the word ‘RED’ in response list is written in yellow. 6) Difficulty level 6 (DL6): a stimulus word is displayed in a color different from that it refers to and each word in the answer list is displayed in a different color that it refers to. For example, the stimulus word ‘RED’ is written in green color and the word ‘RED’ in response list is written in yellow. Figure 2 shows an example of the Stroop test. A stimulus word, as shown in Figure 2a), is presented to the subject in 1 second. The subject has to read the word aloud and then choose the color that the word is printed in from the provided answer list as shown in Figure 2b). The maximum response time is 2 seconds. This process is referred as one Stroop task. The time between the two Stroop tasks should be from 1 to 2 seconds [39]. In our experiment, the time between two tasks is set at 2 seconds. Each subject has to complete four sets of tasks. Each set contains 30 tasks with different combinations of difficult levels. In total, each subject has to complete 120 tasks.
(a) (b) Figure 2 Stroop test: (a) stimuli, (b) responses to stimuli
3
Copyright © 2009 by ASME
3.2 Experiment setup and procedure The Stroop test was conducted with four subjects from different ethnic backgrounds, different genders (2 males and 2 females), age ranging from 22 to 24 years, and with or without glasses. English is the native or foreign language for the subjects. External stimuli such as room lighting and environmental temperature were kept constant.
As is shown in the Figure 4, the experiment protocol includes data processing, data analysis and modeling. Data processing filters the noises from raw EEG data, eye data and segments it with respect to Stroop data whereas data analysis determines the subject’s mental stress by calculating various eye and EEG parameters. Finally, the correlation between subject’s performance and mental stress is presented.
Equipment setup The eye signals were recorded during the experiment by using Face Lab 4.5. The equipment is calibrated according to the standards in the manual. Precision mode was selected for the accurate recording of pupil diameter. The gaze quality level was kept at 3 (highest quality level) and the frame rate is 59 - 60 frames per second (ideal frame rate). Infrared (IR) position was adjusted according to its location in the experiment set up. The time window for PERCLOS were kept at 120 seconds and the threshold at 70%, respectively. The time window used to calculate blink statistics, also called blink filter window size, was set at 60 seconds. Figure 4 Experiment protocol 4
Figure 3 10-20 Electrode placement top view and profile view The EEG signals were recorded during the Stroop test by using four electrodes from positions Fz, Cz, Pz and Oz. All electrodes were referred to lthe inked earlobe. The EOG was recorded below the left eye and was linked with the right earlobe. All electrodes were placed on the scalp according to the international ten-twenty system of electrode placement [40] as shown in Figure 3. The sampling frequency is 200 Hz. EEG and EOG were amplified with 200 Hz frequency and 10 second time constant. Experiment procedure Before starting the experiment, subjects are free to play a few trials to get acquainted with the Stroop test. In general, it took the subject around eight to ten minutes in the trial stage. Eye tracker and EEG equipments are set up and calibrated according to physiological characteristics of the subjects before the trial stage. During the experiment, the subject were asked to sit in front of the computer at ease for the first 15 minutes, which is called pre Stroop test stage, followed by the Stroop test lasted for another 10 minutes, and finally the subjects were again asked to sit in the relaxed state for an extra 10 minutes, which is called post Stroop test stage.
DATA PROCESSING
4.1 Eye data filtering Face Lab 4.5 is susceptible to data loss during recording of eye movement and head movement. This data loss cause major problem during analysis. Two main reasons behind data loss are: eye blink interference and excess head movement. The artifact gaps due to blinking and excess head movement were filled by linear interpolation. The interpolated data is then segmented in Matlab 7.0 according to the difficulty level. The features extracted from the eye data include pupil diameter, blinking frequency, blinking duration, and PERCLOS. The standard deviation and mean value of the features are computed based on the Equation (1) and (2), respectively. ∑
(1) ∑
(2)
where is the sample size, is the sample mean.
is the sample observation and
4.2 EEG data filtering EEG scalp is susceptible to various non-physiological signals that interfere with the recording electrodes. These interfering noises cause a major problem in reliable analysis of the EEG data. Also, the subject’s own physiological signals may interfere with his or her own EEG. Some common interference that were observed in our EEG recording were due to: power line noise (60 Hz), electrode loose leads, eye blink interference, motion/movement artifacts, and muscular activity. These noises were removed using various digital filters. In particular, the power line noise is removed by the Notch function
4
Copyright © 2009 by ASME
in Grasslab. Other noises are removed by Butterworth filter designed from Matlab’s built-in function butter. 5 4
Amplitude
3 2 1 0 -1 -2 0
2
4 6 Time(Seconds)
8
10
(a) 4 3
5
DATA ANALYSIS
5.1 PERCLOS analysis The PERCLOS value is the percentage of time spent while drooping with respect to a fixed time window of 120 seconds. PERCLOS or the eye closure duration reflects the alertness level of the subject [31]. The threshold kept for PERCLOS is 70%, which signifies that the eyelid covers the pupil over an extended period of time. When the PERCLOS measure reaches predetermined level of proportion of time of 120 seconds, the eye is at least 70% closed, this indicates fatigue or drowsiness in an individual [31]. PERCLOS data is processed by taking running average with respect to time in MATLAB from the eye data extracted from the Face Lab 4.5.
Amplitude
2 1 0 -1 -2 0
2
4 6 Time(Seconds)
8
10
(b) Figure 5 (a) Raw data, (b) Filtered data Figure 5 shows a 10 second raw data and the corresponding filtered data for Fz channel recording of two consecutive Stroop tasks, respectively. 4.3 Data segmentation The filtered eye and EEG data is then segmented according to the Stroop tasks. In the Stroop test, we have four profiles and each profile has six difficulty levels. Each profile has 30 screens. Each difficulty level in each profile has 5 screens. So we have 20 screens for each difficulty levels. Time duration for the Stroop test is ten minutes. Therefore, the EEG data is segmented into 5 second segments. Figure 6 shows two segments segmented from the EEG data shown in Figure 5b).
Figure 6 Segmentation of EEG data corresponding to the Stroop test
Figure 7 Pre Stroop test, Stroop test and post Stroop test PERCLOS data In the context of PERCLOS following parameters are taken into consideration: ∑ d C /W (3) where is the time window, is the episodes, is the duration and is the PERCLOS value. During the rest segment, which is the pre Stroop test and the post Stroop test, the PERCLOS value was maximum whereas during the Stroop test, the PERCLOS value was the least as depicted in Figure 7. Visual demand leads to cognitive activity that is reflected in the PERCLOS value. Also it was observed that, during difficulty levels 1, 2 and 3 the PERCLOS value was larger than that at difficulty levels 4, 5 and 6. Since a high difficulty level task induces more visual demands in subject, it makes him/her more alert as compared to a low difficulty level task. The relation between PERCLOS and the difficulty levels is shown in Figure 18. 5.2 Blinking frequency analysis Blinking frequency or blinking rate is the number of times the subject blinks in a minute. One of the factors affecting blinking rate is coping up demand of cognitive work load [36, 41].
5
Copyright © 2009 by ASME
Stroop test, as illustrated in Figure 9, indicates that the subject was more attentive during the test. The relation between mean blinking duration and mental workload induced by the difficulty level is shown in Figure 19.
5
5.4
Pupil diameter analysis
4
7
2
1
0
x 10
6
3
RawPupil Diameter
Average number of blinks
6
1
2
3
4
5
Difficulty Level
5 4 3 2
0 0
Figure 8 Average no of blinks during Stroop test
5.3 Blinking duration analysis Blink duration is the time interval between the time of blink initiation and the time at which the lowest point is reached by the eyelid during a blink. Blink closing duration is supposed to increase with the increased time on task, and difficulty level [41]. Long closure duration of a blink is more than 200 milliseconds. However, blinks with closure duration more than 300 milliseconds are manually removed [41].
Pupil Fluctuations
1
6
Figure 8 shows the average number of blinks for each difficulty level. The number of blinks decreases with the increment in the difficulty levels as the more difficult the level is, the higher visual demand is required, resulting in decrease in blinking frequency.
-3
500 1000 1500 2000 Number of Values of Pupil diameter
2500
Figure 10 Raw data of pupil diameter of pre Stroop test, Stroop test and post Stroop test Dilation of human pupil is controlled by sympathetic and parasympathetic divisions of the Autonomic nervous system (ANS) which controls the muscles that contract and dilate the pupil [42]. Pupil diameter is a significant mental stress indication parameter. Pupil diameter increases with mental stress due to mental workload [24, 43]. Figure 10 shows the raw data of the pupil diameter.
Figure 11 Mean value of the pupil diameter of pre Stroop test, Stroop test and post Stroop test Figure 9 Moving average of blinking duration during pre Stroop test, Stroop test and post Stroop test During the Stroop test, the duration of blinks observed was in the ranges of 220-240 milliseconds. For the pre Stroop test and post Stroop test the range was between 130 and 240 milliseconds. The less variation in blinking duration during
There were significant changes in the mean value of pupil diameter during entire experiment. Figure 11 and Figure 12 shows the mean value and standard deviation of the pupil diameter during the Stroop test.
6
Copyright © 2009 by ASME
x 10
-4
techniques to do spectral analysis which is shown in Equation (4) [46]. / ∑ (4) where x(n) is the EEG data, N is the total number of samples. The power spectrum, P(k), can then be obtained with through Pwelch and Pburg methods for estimating PSD (Power spectrum density) [46]. Figure 14 shows the PSD value for EEG data in Figure 5(b).
6 5.5 5 4.5
1
4
PSD for Channel 1 Fz 0.8
3.5
Amplitude
Standard Deviation Pupil Diameter
6.5
3 0
2
4 6 Time in minutes
8
5.5 Inter-blink interval The parameter included in EOG analysis is inter-blink interval. The blinking rate and time differences of two blinks obtained from EOG are shown in Figure 13. There are four blinks and three IBI in the figure. Inter-blink interval refers to the time difference between two blinks. 4 3
0 0
5
10
15 20 Frequency
25
30
35
Figure 14 PSD for EEG data in Figure 5(b) 5.7 Performance from Stroop test The performance is calculated by average response time combined with the percentage of incorrectness. With the increase in the difficulty level, subjects tended to slow down and make more mistakes. The performance is calculated using Equation (5). 2 (5) where β is the incorrectness rate, is the average response time over one Stroop segment, andDifficulty T is the maximum response time. Levels Figure 15 illustrates a subject’s performance at each difficulty level. It can be observed from the figure that the performance of the subject decreases as the difficulty level increases.
Time differnce between two blinks (IBI)
1.5
2 1
Performace
Amplitude
0.4 0.2
10
Figure 12 Standard deviation of pupil diameter during Stroop test The maximum standard deviation was observed at a high difficulty level. This indicates that the major cognitive activity took place at high difficulty levels was high. The relation between mean pupil diameter and the difficulty level is shown in Figure 17.
0.6
0 -1 -2 0
2
4 6 Time(Seconds)
8
1
0.5
10
Figure 13 Inter-blink interval 5.6 Power spectral analysis In the power spectral analysis from EEG data, the signals are converted from time domain to frequency domain. Spectral analysis divides the original signal into its frequency components, which can be efficiently conducted by using the fast Fourier transform (FFT) [44, 45]. Through the spectral analysis, we can separately study the four bands of EEG waves with their specific frequencies. Fourier transform is one of the
0
DL 1
DL 2
DL 3 DL 4 Difficulty Levels
DL 5
DL 6
Figure 15 Performance of a subject 6
RESULTS
6.1 Correlations A strong correlation was observed between PERCLOS, blinking frequency and performance, which indicates that the
7
Copyright © 2009 by ASME
increased mental stress due to high mental workload results in a decrease in the performance. Table 1 shows the correlations between different eye parameters and performance. Eye parameters IBI and Performance
Correlations -0.89878
Blinking frequency and Performance 0.78129 Blinking duration and Performance 0.647223 PERCLOS and Performance 0.866698 Table 1 Correlations between eye parameters and performance 6.2
Relationship between Mean values of eye parameters and difficulty level
Blinking frequency The blinking frequency initially increases with the increase in the difficulty level. It reaches a maximum at the difficulty level 3, but as the complexity of the task increases further there is sudden fall in blinking frequency due to the high stress level. Figure 16 illustrates a sudden fall in blinking frequency from difficulty level 3 to 6 due to the high visual demand. The performance also reduces during this period.
Figure 17 Mean pupil diameter and mental workload according to difficulty level PERCLOS Figure 18 shows that PERCLOS value decreases with increase in the mental work load. The subjects were more attentive at high difficulty levels. Relaxing state of mind reflects in high PERCLOS value.
Figure 16 Blinking frequency and mental workload at difficulty levels Pupil diameter Figure 17 shows the mean value of pupil diameter at different difficulty levels and it can be observed that there was a sudden rise in the mean pupil diameter at difficulty levels 4,5 and 6, due to the stress induced during the incongruent segments.
Figure 18 PERCLOS with respect to difficulty level Blinking duration The blinking duration, as can be seen in Figure 19, increases after difficulty level 3. High cognitive workload results in longer eye closure which results in high blink duration.
8
Copyright © 2009 by ASME
Oz, respectively. As mentioned in Section 2, different channels and different bands are associated with different functions of the brain. The relation between these channels and their corresponding brain activities is as follows: • Fz channel deals with emotions. • Cz deals with sensory and motor function • Pz deals with percpetion and differentiation • Oz deals with vision. In the following, we will anlayse our data from two aspects. First is the subject’s attentive activities and the second is the subject’s emotional state.. The emotional state will be analyzed through the change of Theta band in channel Fz. The vision activity will be analyzed through the Alpha band in channel Oz. The attentive activities will investigated through Beta band in channel Cz, Pz and Oz. 4 Fz Cz Pz Oz
Figure 19 Blinking duration with respect to difficulty level 3 Theta Power
6.3 Relationship between IBI and difficulty level If a subject gets more attentive on a task, then the subject’s IBI tends to increase [33, 47]. Figure 20 shows average inter-blink interval for all difficulty levels of the Stroop test. It is observed that as difficulty level increases the inter-blink interval also increases, indicating a rise in the subject’s concentration. The inter-blink-interval is the lowest at difficulty level 1. The IBI reaches its peak at difficulty level 5 and then declines significantly. The IBI at difficulty level 6 is much lower than that at difficulty level 5. This decrease in IBI together with more demanding task (the task in difficulty level 6 is harder than that in difficulty level 5) signifies that the subject gets weary or fatigued after prolonged attention.
2
1
0
DL 1
DL 2
DL 3 DL 4 Difficulty Levels
DL 5
DL 6
(a) 25
Fz Cz Pz Oz
Alpha Power
20
7
IBI(Inter blink interval) 6
15 10
Seconds
5 5
4 0
3
DL 1
DL 2
2
DL 5
DL 6
(b) 10
1 0
DL 3 DL 4 Difficulty Levels
DL 1
DL 2
DL 3
DL 4
DL 5
DL 6
Fz Cz Pz Oz
8 Beta Power
Difficulty Levels
Figure 20 Inter-Blink Interval 6.4
Relationship between spectral power and difficulty levels We have conducted analysis on the four bands of the power (Delta, Theta, Alpha, and Beta) for the four channels (Fz, Cz, Pz and Oz) in terms of difficulty levels. No significant changes have been found in the delta power. Figure 21 shows the Theta power, Alpha Power and Beta Power respectively. In these three figures, the dark blue, light blue, yellow and brown bars represents channels Fz, Cz, Pz and
6 4 2 0
DL 1
DL 2
DL 3 DL 4 Diffilculty Levels
DL 5
DL 6
(c) Figure 21 EEG Power bands (a) Theta Power, (b) Alpha Power and (c) Beta Power
9
Copyright © 2009 by ASME
Theta power in channel Fz drops significantly from difficulty level 5 to 6. Considering our observation in the IBI analysis, this shows that a lower theta power is associated with the state of mental fatigue. In Figure 21(b) the amplitude of the power in channel Oz increases as the difficulty level increases from 1 to 4; however, it drops at level 5. In Figure 21 (c), statiscally it can be observed that the amplitude of the power in channel Cz, Pz, and Oz rises as the difficulty level increases from 1 to 5 but again it drops at level 6. The drop at this level may be due to the mental fatigue. The general trend shown in Figure 21 indicates that the subject’s mental stress is positively related to the difficulty level of the task. However, when the task becomes too difficult for the subject, the subject will tend to get into the state of fatigue and that lowers the measured mental stress. 7 CONCLUDING REMARKS The work presented in this paper is still in a preliminary stage. We have not achieved any definite conclusion about designers’ mental stresses and performance. However, this investigation has made the following contributions for further research in understanding designers’ brain activities during the creative design process: 1) developed a system of methods to study subject’s cognitive activities using eye tracker and EEG system, 2) explored the relationship between different channels in the brain, various eye parameters and the cognitive tasks. This work is the first step in our efforts to understand brain activities during the design process. Experiments and related data analysis on design problem solving are on-going. We are also integrating EEG, EyeGaze, HRV, video and linguistic data to understand desingners’ cognitive process.
[8].
[9]. [10].
[11]. [12].
[13].
[14]. [15].
[16].
[17].
[18].
REFERENCES [19]. [1]. [2].
[3]. [4]. [5].
[6].
[7].
Pahl, G. and W. Beitz, "Engineering Design: A Systematic Approach". 1988: Spinger-Verlag. Altshuller, G.S., "Creativity as an Exact Science: the Theory of the Solutions of Inventive Problems". Studies in Cyberbetics, ed. F.H. George. 1984: Gordon and Breach Science Publishers. Suh, N., "The Principles of Design". 1990: Oxford University Press. Gero, J.S., ed. "Artificial Intelligence in Design'00". 2000, Kluwer Academic. Zeng, Y., "Axiomatic theory of design modeling". Transaction of SDPS: Journal of Integrated Design and Process Science, 2002. 6(3): p. 1-28. Maxwell, T.T., A. Ertas, and M.M. Tanik, "Harnessing complexity in design". Transaction of SDPS: Journal of Integrated Design and Process Science, 2002. 6(3): p. 63-74. Dorst, K. and N. Cross, "Creativity in the design process: co-evolution of problem-solution". Design Studies, 2001. 22(5): p. 425-437.
[20]. [21].
[22].
[23].
[24].
[25]. [26].
10
Akin, O. and C. Akin, "On the process of creativity in puzzles, inventions, and designs." Automation in Construction, 1998. 7: p. 123-128. Dasgupta, "Creativity in Innovation and Design". 1994. Wharton, J.G. Cognition and creativity: creativity, imagination and the objectivist epistemology. Series "Cognition and creativity: creativity, imagination and the objectivist epistemology" 1999 [cited; Available from: www.axiomatic.net/cc.pdf. Zeng, Y. and G.D. Cheng, "On the logic of design". Design Studies, 1991. 12(3): p. 137-141. Yerkes, R.M. and Dodson, "The relation of strength of stimulus to rapidity of habit-formation". Comparative Neurology and Psychology, 1908. 18: p. 459-482. Dyregrov, "Mental mobilization processes in critical incident stress situations ". International Journal of Emergency Mental Health, 2000. 2(2): p. 73-81. Gaillard, A.W.K., "Comparing the concepts of mental load and stress", in Ergonomics,36,991-1005. 1993. Gaillard, A.W.K., & Wientjes, C. J. E., "Mental load and work stress as two types of energy mobilization", in Work & Stress,8, 141-152. 1994. Bourne, L.E. and R.A. Yaroush, "Stress and cognition: A cognitive psychological perspective ". 2003, University of Colorado. Zeng, Y., "Environment-based formulation of design problem". Transaction of SDPS: Journal of Integrated Design and Process Science, 2004. 8(4): p. 45-63. Zhu, S., S.J. Yao, and Y. Zeng. "A novel approach to quantifying designer's mental stress in the conceptual design process ". in ASME DETC/CIE. 2007. Las Vegas, Nevada, USA. Tovey, M., S. Porter, and R. Newman, "Sketching, concept development and automotive design". Design Studies, 2003. 24: p. 135-153. Kirschner, "Cognitive load theory", in Learn. Instruct. 12(special issue): 1–154. 2002. Dong, A., "Quantifying coherent thinking in design: a computational linguistics approach", in Design Computing and Cognition'04, 521-540 2004. Stempfle, J. and P. Badke-Schaub, "Thinking in design teams - an analysis of team communication". Design Studies, 2002. 23(5): p. 473-496. Raja, P. and M. Rizzo, "Neuroergonomics- The brain at work", ed. A. Kirlik. 2007, New York: Oxford Unievrsuy Press. 1-430. Atsushi UETAKE, A.M. "Assessment of mental fatigue during VDT task using event-related potential". in IEEE International Workshop on Robot and Human Interactive Coomunication. 2000. Japan. S.Tyner, F., J. R.Knott, and J. W.Brem Mayer, "Fundamentals of EEG Technology". 1983. Shaker, M.M., "EEG waves classifiers using wavelet transform and fourier transform". International
Copyright © 2009 by ASME
[27]. [28]. [29]. [30].
[31].
[32].
[33].
[34].
[35].
[36]. [37].
[38].
[39].
[40].
[41].
Journal of Biological, Biomedical and Medical Sciences, 2006. Teplan, M., "Fundamentals of EEG Mesurement". Mesurement science review, 2002. 2. Mark S. Sanders and E.J. McCormick, "Human factors in engineering and design". 1988. Jing Zhai and A. Barreto, "Stress Recognition Using Non-invasive Technology". 2006. Zhai, J. and A. Barreto. "Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables". in Engineering in Medicine and Biology Society, 2006. EMBS '06. 28th Annual International Conference of the IEEE. 2006. Dinges, D.F. and R. Grace, "PERCLOS: A valid psycho physiological measure of alertness as assessed by psychomotor vigilance". 1998, National Technical Information Service. John A. Stern, L.C.W., Robert Goldstein, "The endogenous eyeblink". The Society for Psycophysiological Resesrch, 1984. 21(1). Fogatry, C. and J. A.Stern, "Eye movements and blinks:their relationship to higher cognitive process". International Journal of Psychophysiology, 1989. 8. Miguel Ángel Recarte, E.P., Ángela Conchillo, Luis Miguel Nunes, "Mental workload and visual impairment: Differences between pupil, blink, and subjective rating". The Spanish Journal of Psychology, 2008. 11 p. 374-385. Stern, J.A., B. D, and D. Schroeder, "Blink Rate: A Possible Measure of Fatigue". Human Factors: The Journal of the Human Factors and Ergonomics Society, , 1994. 36. Stern, J.A., D.Boyer, and D.J. Schroeder, "Blink Rate As a Measure of Fatigue". 1994. Ryu, K. and R. Myung*, "Evaluation of mental workload with combined measure based on physiological indices during a dual task of tracking and mental arithmetic". International Journal of Industrial Ergonomics, 2005. 35(11): p. 991-1009. Stroop, J.R., "Studies of interference in serial verbal reaction". Journal of Experimental Psychology, 1935. 18: p. 643-662. Mario Liotti, M.G.W., Ricardo Perez III, Helen S. Mayberg, "An ERP study of the tempoal course of the Stroop color-word interference effect". Neuropsychologia, 2000. 38: p. 701-711. "The International 10-20 System of Electrode placement", in http://www.brainmaster.com/generalinfo/electrodeuse/ eegbands/1020/1020.html. John A. Stern, D.J.B., David J. Schroeder, R. M. Touchstone, N. Stoliarov and "Blinks, saccades and fixation pauses during vigilance task performance: 2: gender and time of day", in Office of Aviation Medicine Washington, D.C. 20591. 1996.
[42].
[43].
[44].
[45].
[46].
[47].
11
Siegle G J and S.S. R., "Sympathetic and parasympathetic innervation of pupillary dilation during sustained processing". International journal of psychophysiology, 2004. 52: p. 77-86 G. Calcagnini, F.C., S . Lino, S . Cerutti. "Pupil diameter variability in humans". in Annual EMBS International Conference, . 2000. Chicago IL. Muthuswamy, J. and N.V. Thakor, "Spectral analysis methods for neurological signal". Jornal of Neuroscience Methods, 1998: p. 1-14. Akin1, M., "Comparison of wavelet transform and FFT methods in the analysis of EEG signals". Journal of Medical Systems, 2002. 26(3). Thakor, N.V. and S. Tang, "Advances in qunatitative electroencephalogram analysis methods". Annu.Rev.Biomed.Eng, 2004. 6: p. 453-495. Stern, J.A., D. Boyer, and D.J. Schroeder, "Blink rate as a measure of fatigue". 1994.
Copyright © 2009 by ASME
