Mental Stress Grading Based on fNIRS Signals Fares Al-shargie,1 Tong Boon Tang,1 Masashi Kiguchi2
Abstract— In this study, we propose functional near infrared spectroscopy (fNIRS) to objectively grade different levels of mental stress. The levels of stress were set based on the difficulty of arithmetic task, time pressure and negative feedback about peer performance. We examined the proposed approach on twelve human subjects using the Montreal Imaging Stress Task. The experiment results revealed a reduction in cortical activation at prefrontal cortex when stressed, and the differences in hemodynamic response between control condition and under stress were significant for arithmetic difficulty level one, two and three, respectively, (p = 0.0023, 0.0001 and 0.0004). The experiment results thus support the suggestion of fNIRS in grading mental stress. Keywords—Stress, fNIRS, neuroimaging
I. INTRODUCTION Conventionally, questionnaires are used as a tool to assess mental stress. However, such method is subjective [1]. Stress has been reported to activate the hypothalamuspituitary-adrenocortical axis (HPA axis) and sympathetic nervous system (SNS) causing an increase in cortisol secretion in the adrenal cortex. The level of cortisol is widely accepted as a biomarker of stress [2]. Besides cortisol, stress can be detected from human biosignals [3, 4]. Researchers have found a relationship between salivary cortisol levels and physiological variable changes such as heart rate variability (HRV), electrodermal response (EDR) and blood pressure (BP) [4, 5]. Stress causes a decrease in the high frequency components of heart beat interval and an increase in the low frequency components, respectively. Skin conductivity on the other hand varies with the changes in skin moisture level revealing the changes in SNS. It has been reported to increase during a stressful task and can be acquired using a galvanic skin response (GSR) sensor [6]. Changes in autonomous nervous system (ANS) can also be represented by electroencephalography (EEG) signals [7]. EEG is one of the most studied non-invasive neuroimaging modality that measures the electric potential of cortical activation. EEG has the advantages of temporal resolution, ease of use, and low set-up cost. Its signal components are categorized by frequency bands; Delta (0.5-4 Hz), Theta (48 Hz), Alpha (8-13 Hz) and Beta (14-30 Hz). Each of the frequency band can be used as an indicator of one’s mental state. However, EEG has some limitations as it has poor *Research supported by Ministry of Education, Malaysia under the HiCOE grant to Centre for Intelligent Signal and Imaging Research, UTP. 1 Fares. Al-shargie and Tong Boon Tang are with the Centre for Intelligent Signal and Imaging Research, Universiti Teknologi PETRONAS, Malaysia.
[email protected] 2 Masashi Kiguchi is with Hitachi. Ltd., Research& Development Group, 350-0395,Japan.
978-1-4577-0220-4/16/$31.00 ©2016 IEEE
spatial resolution and its signals are susceptive to noise. To overcome these limitations, we proposed a new neuroimaging modality to objectively grade the levels of mental stress. Functional Near-infrared Spectroscopy (fNIRS) is a noninvasive brain imaging technology based on hemodynamic responses to cortical activation [8]. It uses near-infrared light in the wavelength range 650-900 nm and estimates the changes in oxygenated and deoxygenated hemoglobin concentrations (O2Hb and HHb) using modified BeerLambert law [9]. fNIRS has several advantages over other neuro-imaging modalities. Compared to EEG, it has better spatial resolution and less affected by noise [10]. Compared to functional magnetic resonance imaging (fMRI) and positron emission topography (PET), fNIRS is portable, cheaper and does not confine the subjects to lying position. In this study, we aim to grade mental stress by measuring the hemodynamic response at the prefrontal cortex (PFC). PFC is the brain region responsible for regulating thoughts, actions and emotions. It is also the most sensitive area in the brain to detrimental effects of stress exposure [11]. According to previous fMRI studies [12, 13], solving arithmetic tasks under time pressure induces mental stress on the PFC. We developed three levels of arithmetic task difficulty to induce stress on experiment participants. Besides the task difficulty, time pressure and negative feedback of peer performance were used as stressors in this study. To the best of our knowledge, this is the first study to quantify the levels of stress based on hemodynamic responses to cortical activation. II. METHODOLOGY A. Subjects Twelve male healthy, right-handed adults (aged 22 ± 4) participated in this study. All participants were informed prior to the experiment and gave written consent, in accordance with the Declaration of Helsinki and ethics approval granted by local ethics review committee at Universiti Teknologi PETRONAS. None of the participants had a history of psychiatric, neurological illness or psychotropic drug use. To avoid any environmental stress, all participants were seated in a comfortable chair in a room with good air condition. B. Stress stimuli The experiment was developed based on the Montreal Imaging Stress Task (MIST) [13]. In this study, the arithmetic task was defined at three levels of difficulty, where each level corresponded to one level of stress. The task at level one (L1) involved 3-one digit integer (ranging from 0 to 9) and used the operands of + or – (example 9+1-
5140
6). At level two (L2), the task involved 3 integers (ranging from 0 to 99) with at least 2 two-digit integers using the operands of +, –, and × (example 12×3-30). At level three (L3), the task involved 4 integer numbers (ranging from 0 to 99) and the operands include +, –, ×, / and ÷ (example 799/3+35). Besides the task difficulty, time pressure and negative feedback about peer performance were implemented to induce stress on the participants. Participants were first trained at each level of task difficulty and the average time for each individual in answering the questions was recorded. This recorded time was then reduced by 10% and used as time pressure on the participants. Moreover, feedback of answering the questions (“correct”, “incorrect” or “timeout”) and performance indicators (one for the participant’s performance and one for the averaged peer performance fixed at 90% accuracy) were displayed on the computer monitor to further induce stress in experiment participants. C. Experiment procedure Participants were instructed to avoid any head and body movements and deep breathing during fNIRS measurements. The experiment was performed in four successive sessions. In first session, a brief introduction was given to each participant to be familiar with the proposed tasks. In second session, the participants were trained for five minutes at each level of difficulty in the mental arithmetic (MA) task to estimate average time taken to answer each question. In third session (i.e. control session), the fNIRS cap was attached to the participant’s head and fNIRS signals were recorded for a total duration of 15 minutes while solving arithmetic problems at three levels of difficulty without any time limit per question. After fNIRS recording, a questionnaire was filled by the participants as self-reporting about task loading according to NASA-TLX rating scale [14]. The fourth session (i.e. stress phase) was similarly as the control phase where the fNIRS was recorded for 15 minutes but under stress conditions (time limit and negative feedback). Similar questionnaire about task loading was again completed. The entire recording duration for each participant is ~ 1 hour.
Fig. 1 gives an overview of the experiment protocol and the block design. Each block consists of 40 seconds of mental arithmetic task and 30 seconds of rest. During the 40 seconds task, participants were shown mental arithmetic tasks on a computer screen and had to solve them as fast as they could (i.e. the control session) or within a given amount of time (i.e. the stress session). During the 30 s rest, participants needed to focus on a fixation cross with black background to sustain their attention to the monitor display. D. Functional near infrared spectroscopy Relative concentrations of oxygenated and deoxygenated hemoglobin were recorded with 16 optodes (8 sources and 8 detectors) of fNIRS system OT-R40 (Hitachi Medical, Japan) using two infrared wavelengths (695 and 830 nm). The sampling frequency was set to 10 Hz. The distance between pairs of source and detector probes was set to 3.0 cm. Channel (Ch) was defined as the measurement area between a pair of source-detector optodes. A total of 27 channels were measured in this study and PFC was the region of interest. Fig. 2 shows the full configuration of measurement channels. E. fNIRS analysis The fNIRS signals were transformed to concentration changes of oxygenated, deoxygenated and total hemoglobin using modified Beer-Lambert law [9]. In order to reduce the noise and motion artifacts, fNIRS signals passed through several pre-processing steps using the plug-in-based analysis software Platform for Optical Topography Analysis Tool (developed by Hitachi, CRL; run on MATLAB)[15]. The process involved removing the motion artifacts [16], filtering the signal in the range of 0.012 to 0.8 Hz using a 5th order Butterworth filter, baseline correction, and moving average. In baseline correction, we defined a period from 5s prior to task condition to the end of each MA task as an analysis block. Linear regression by least mean square method was applied to remove any dc drift in fNIRS recordings [16] before we averaged all the blocks and performed statistical analysis on the changes of O2Hb.
Figure 1. Experiment protocol of mental stress study. The labels L1, L2 and L3 represent the levels of mental arithmetic task and MA stands for mental arithmetic. Six recordings were performed in this experiment; three for control condition and three for stress condition. In each recording, there were four blocks. In each block, mental arithmetic task was allocated for 40 s followed by 30 s rest.
5141
reporting about task loading questionnaires, NASA-TLX rating scales showed no significant differences among the three mental stress levels. Admittedly the sample size is limited, the experiment results suggested that subjective assessment using questionnaire might not be sensitive enough as a tool for quantifying mental stress levels. Right
Left
Figure 2. Measurement setup: (a) channel configuration and (b) probe holder worn on the PFC area. Eight sources and eight detectors used in the measurements corresponding to 27-channels over the PFC area.
III. RESULT AND DISCUSSION The post-processed fNIRS measurements from the twelve subjects were subsequently averaged channel-bychannel for each difficulty level and experiment condition (control/stress). The results obtained from level one (i.e. control phase) demonstrated significant increase in O2Hb due to MA task, as compared to baseline. Under stress condition, however, there was a significant reduction in this change in O2Hb. This revealed that stress might have impaired the PFC and resulted in reduced cortical activities. Fig. 3(a) and (b) illustrate the topographical map of O2Hb under control and stress conditions, respectively. The channel numbers were marked on the topography maps to indicate their particular locations in the PFC area. Red colour indicates high concentration level and blue colour indicates less concentration level of O2Hb. Similar results were obtained at level two and level three of arithmetic task difficulty where a significant decline was observed in the concentration change of oxygenated hemoglobin under stress condition. Fig. 4 and Fig. 5 illustrate the topographical map of O2Hb under control and stress conditions, at level two and level three, respectively. In addition, we studied, at each level of mental stress, the difference in hemodynamic response to MA task under the control and the stress conditions, and their correlation with mental stress level using two-sample t-test analysis. We found a significant decrease in oxygenated hemoglobin concentration from control condition to different stress conditions with mean p-value of 0.0023, 0.0001 and 0.0004 for level one, level two and level three, respectively. In addition, there was also a significant reduction in hemodynamic response as the level of stress increased from level one to higher levels, with p < 0.01. We further examined the performance of each participant in answering the arithmetic questions (accuracy score), and how they were related to the hemodynamic response. Fig. 6 shows the relationship between the accuracy score by each participant (twelve participants in total) and the mean change in oxygenated hemoglobin concentration, at level 1 difficulty. We found a good agreement in their performance when their cortical activation was affected by stress (R2 = 0.86963). When cross-checked with the results of self-
(a)
(b) Figure 3. Topographical map of oxygenated hemoglobin, (a) under control condition and (b) under stress condition, at level 1 for average of 12 subjects. Red colour indicates high concentration levels of oxygenated hemoglobin and blue colour indicates less activation or distortion.
Right
Left
(a)
(b) Figure 4. Topographical map of oxygenated hemoglobin, (a) under control condition and (b) under stress condition, at level 2 for average of 12 subjects. Red colour indicates high concentration levels of oxygenated hemoglobin and blue colour indicates less activation.
5142
Furthermore, we found the hemodynamic response to mental stress were highly localized. Based on the repeated measurements (three measurements for each level of mental stress), we observed a high reduction in oxygenated hemoglobin concentration on the right PFC in all the three levels of mental stress. To confirm the right dominant of PFC to mental stress, we calculated the laterality index at the three levels of stress, where LIS = (right-left) / (right+left). LIS 0 indicates the dominance of left PFC. The results showed right PFC dominant with mean LIS value of -0.0821 and 0.1428 at level one and level two of mental stress, respectively, with p < 0.05. But, at level three, the result was not significant, perhaps due to the excessive level of stress. Right
IV. CONCLUSION In this study, we investigated if fNIRS signals could be used in grading mental stress. Using time pressure and negative feedback about peer performance as stressors, the experiment results showed that hemodynamic response at the PFC was significantly different among the levels of arithmetic difficulty/stress. Using laterality index, we found right PFC as the brain region more sensitive to mental stress. The results from questionnaire approach (self-reporting about task load) indicated that the engagement of participants reduced when stressed, but were not significant to discriminate different levels of stress. In short, our study supports the suggestion of using fNIRS to objectively grade the levels of mental stress.
Left
REFERENCES
(a)
(b) Figure 5. Topographical map of oxygenated hemoglobin, (a) under control condition and (b) under stress condition, at level 3 for average of 12 subjects. Red colour indicates high concentration levels of oxygenated hemoglobin and blue colour indicates less activation.
Figure 6. The graph shows the experiment result where the mean change in oxygenated hemoglobin concentration is positively correlated with performance (accuracy score).
[1] T.-K. Liu, Y.-P. Chen, Z.-Y. Hou, C.-C. Wang, and J.-H. Chou, "Noninvasive evaluation of mental stress using by a refined rough set technique based on biomedical signals," Artificial intelligence in medicine, vol. 61, pp. 97-103, 2014. [2] D. H. Hellhammer, S. Wüst, and B. M. Kudielka, "Salivary cortisol as a biomarker in stress research," Psychoneuroendocrinology, vol. 34, pp. 163-171, 2009. [3] A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion recognition using bio-sensors: First steps towards an automatic system," in ADS, 2004, pp. 36-48. [4] J. Healey and R. W. Picard, "Detecting stress during real-world driving tasks using physiological sensors," Intelligent Transportation Systems, IEEE Transactions on, vol. 6, pp. 156-166, 2005. [5] H. Ashton, R. Savage, J. W. Thompson, and D. Watson, "A method for measuring human behavioural and physiological responses at different stress levels in a driving simulator," British journal of pharmacology, vol. 45, pp. 532-545, 1972. [6] C. L. Lisetti and F. Nasoz, "Using noninvasive wearable computers to recognize human emotions from physiological signals," EURASIP journal on applied signal processing, vol. 2004, pp. 1672-1687, 2004. [7] F. Al-shargie, T. Tang, N. Badruddin, and M. Kiguchi, "Mental Stress Quantification Using EEG Signals," in Int. Conf. for Innovation in Biomedical Engineering and Life Sciences, 2016, pp. 15-19. [8] S. Cutini, S. B. Moroa, and S. Biscontib, "Functional near infrared optical imaging in cognitive neuroscience: an introductory review, Journal of Near Infrared Spectroscopy, vol. 20, pp. 75-92," 2012. [9] A. Sassaroli and S. Fantini, "Comment on the modified Beer? Lambert law for scattering media," Physics in Medicine and Biology, vol. 49, p. N255, 2004. [10] Y. Hoshi, "Towards the next generation of near-infrared spectroscopy," Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, vol. 369, pp. 4425-4439, 2011. [11] A. F. Arnsten, "Stress signalling pathways that impair prefrontal cortex structure and function," Nature Reviews Neuroscience, vol. 10, pp. 410-422, 2009. [12] C. D'Aguiar, "What stress does to your brain: a review of neuroimaging studies," Canadian Journal of Psychiatry, vol. 54, p. 6, 2009. [13] K. Dedovic, R. Renwick, N. K. Mahani, V. Engert, S. J. Lupien, and J. C. Pruessner, "The Montreal Imaging Stress Task: using functional imaging to investigate the effects of perceiving and processing psychosocial stress in the human brain," Journal of Psychiatry and Neuroscience, vol. 30, p. 319, 2005. [14] S. G. Hart and L. E. Staveland, "Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research," Advances in psychology, vol. 52, pp. 139-183, 1988. [15] S. Sutoko, H. Sato, A. Maki, M. Kiguchi, Y. Hirabayashi, H. Atsumori, et al., "Tutorial on platform for optical topography analysis tools," Neurophotonics, vol. 3, pp. 010801-010801, 2016. [16] H. Sato, N. Tanaka, M. Uchida, Y. Hirabayashi, M. Kanai, T. Ashida, et al., "Wavelet analysis for detecting body-movement artifacts in optical topography signals," Neuroimage, vol. 33, pp. 580-587, 2006.
5143
