Use of head-worn sensors to detect lapses in vigilance through the measurement of PERCLOS and cerebral blood flow velocity Lindsey K. McIntire*a, R. Andy McKinleyb, Chuck Goodyeara, and John P. McIntirec a
Infoscitex, Inc.; 711 Human Performance Wing, Warfighter Interface Division, Applied Neuroscience Branch; c 711th Human Performance Wing, Warfighter Interface Division, Battlespace Visualization Branch b
th
ABSTRACT The purpose of this study is to determine the ability of an eye-tracker to detect changes in vigilance performance compared to the common method of using cerebral blood flow velocities (CBFV). Sixteen subjects completed this study. Each participant performed a 40-minute vigilance task while wearing an eye-tracker and a transcranial doppler (TCD) on each of four separate days. The results indicate that percentage of eye closure (PERCLOS) measured by the eye-tracker increased as vigilance performance declined and right CBFV as measured by the TCD decreased as performance declined. The results indicate that PERCLOS (left eye r=-.72 right eye r=-.67) more strongly correlated with changes in performance when compared to CBFV (r=.54). We conclude that PERCLOS, as measured by a head-worn eye tracking system, may serve as a compelling alternative (or supplemental) indicator of impending or concurrent performance declines in operational settings where sustained attention or vigilance is required. Such head-worn or perhaps even offbody oculometric sensor systems could potentially overcome some of the practical disadvantages inherent with TCD data collection for operational purposes. If portability and discomfort challenges with TCD can be overcome, both TCD and eye tracking might be advantageously combined for even greater performance monitoring than can be offered by any single device. Keywords: PERCLOS, Vigilance, Eye-tracker, Sustained Attention, transcranial Doppler
1. INTRODUCTION In today’s society, many jobs require sustained attention and vigilance for long periods of time. Air traffic controllers, cyber operators, unmanned aerial systems operators, Transportation Security Administration (TSA) inspectors, and satellite imagery analysts may occasionally encounter lapses in attention due to the monotonous and sometimes boring nature of these positions. However, mistakes in these types of environments can have devastating consequences. Currently, there is no tool that can reliably measure operator performance in real-time in these environments, and lapses are typically only noticed after a mistake is made. Therefore, it is important to find a tool that can monitor operator *Corresponding author: Lindsey McIntire, 2510 Fifth Street Building 20840, WPAFB, OH 45433, 937-938-3609,
[email protected]
vigilance in an operational environment so that costly mistakes or attentional lapses can be reduced and overall system performance can be improved. Laboratory vigilance tasks have been well-studied and show that operator performance on vigilance tasks degrades over time [6, 18, and 7]. This phenomenon of decreased performance over time is known as the “vigilance decrement” [18]. A common way to attempt to physiologically monitor this decrement is with cerebral blood flow velocity. Specifically, the vigilance decrement has been correlated with decreased blood flow velocity in the right hemisphere [6, 18, and 7]. The assumption is that reduced blood flow velocity is indicative of decreased metabolic activity and thus decreased cognitive activity in the right prefrontal cortex, which has been associated with executive functions involving attention, planning future action, and decision-making [6, 18, and 7]. Blood flow velocity can be successfully monitored by a device called a transcranial Doppler (TCD). TCD is a non-invasive technique to monitor cerebral blood flow velocities in the middle, anterior, and posterior intracranial arteries by using ultrasound signals [18]. However, there are several possible drawbacks to using TCD to monitor decreases in vigilance. First, the TCD is not portable outside of a medical or laboratory setting. Second, it is not a strictly passive device that could be considered for an operational setting due to its susceptibility to vibration and movement. Third, the device is also bulky and uncomfortable to wear for long periods of time. Wearers commonly complain of headache from the tightness of the device on their head, pressed against the skull, yet this is necessary to get a good signal. Fourth, the TCD is not able to track cerebral blood flow of all ages and races because the temporal window closes and hardens as a person ages and this process occurs faster in certain races [10]. Therefore, it would be of particular interest if another physiological monitoring device is found that is more portable and comfortable yet less sensitive to vibration or movement, which can be applied to a wide segment of the general population in terms of age/ethnicity, and which can effectively and passively monitor or detect operator vigilance in an operational setting. It is worth mentioning, however, that one technology need not necessarily replace the other. It may be worthwhile to combine the different measuring methods, in order to improve the signal-to-noise ratio and improve overall system performance in vigilance settings. In searching for an alternative or supplemental physiological monitoring device that addresses some of the downsides of the TCD, we investigated the use of a wearable eye tracking device to monitor operator vigilance. Eye-trackers are very minimally invasive (can be mounted off-body or in portable spectacles), are less susceptible to movement and vibrations, and do not have difficulty tracking people based on age or race; but eye color can contribute to tracking difficulty, depending on the type of tracking system used [15]. Previous research has observed oculomotor changes while subjects performed a visual attention task [17]. Specifically, previous work has consistently found that the eye metric of percentage of eye closure (PERCLOS) may be a good indicator of arousal levels [1, 3, 8, and 14]. PERCLOS is the percentage of time the eyes are closed during a given span of time. An eye-tracker measures PERCLOS, for instance, by examining the proportion of the pupil that is covered by the upper eyelid in a 1 minute time window when more than 80% of the pupil is occluded by the eyelid [2]. Specifically, research on time-on-task fatigue has shown PERCLOS to be a useful indicator for performance declines [2]. Also, studies have shown that PERCLOS changes in response to changes in cognitive workload [9 and 13]. Since eye trackers are easier to deploy in operational settings than TCD devices, we believe that finding oculometrics that can correlate with decreases in vigilance performance can bring us one step closer to providing real-time biofeedback monitoring of operator vigilance during real-world tasks. It is a reasonable hypothesis that PERCLOS could be an appropriate metric for monitoring vigilance because it has provided robust results in past related research. Specifically, Dinges and Grace [2] found a high degree of correlation between PERCLOS and performance lapses on a test of sustained attention, and Mallis, Maislin, Powell, Konowal, and Dinges [12] found that feedback from the PERCLOS metric improved alertness and driving performance. In fact, the Federal Highway Administration and the
National Highway Traffic Safety Administration consider PERCLOS to be among the most promising known real-time measures of alertness for vehicular applications [2]. Therefore, an operational monitoring device such as a head-worn or perhaps off-body eyetracker to measure PERCLOS (or similar oculometrics) could allow users to be warned about impending lapses in attention and/or to detect concurrent lapses, with the ultimate goal of improving operator safety and performance on tasks involving sustained vigilance.
2. METHODS 2.1. Subjects The study was approved in advance by the Wright-Site IRB at Wright-Patterson AFB. Each participant read and signed an informed consent before participating. A total of 19 volunteer subjects (16 men, 3 women) completed this study. Subjects were civilian and active-duty military ages 19-41 years. Subjects received $10/hour for compensation for their time and travel. Subjects were required to have normal utilization of both arms and legs. Subjects were excluded if they required eyeglasses for vision correction because the eye-tracker used in this study could not be worn with eyeglasses. However, subjects wearing contact lenses for vision correction were permitted to participate. 2.2.
Equipment
Each subject was required to wear the Eye-Com (Reno, NV) alertness monitoring device during the vigilance task which was repeated across four test sessions. The device consisted of two infrared (IR)-sensitive cameras and a linear array of IR-illuminating light emitting diodes (LEDs) mounted on a set of eyeglass frames (one camera below each eye). The wavelength of the light emitted by the LEDs was 840 nm. The cameras were angled upward toward the eyes and extracted real-time pupil diameter, eye-lid movement, and eye-ball movement. The software recorded a variety of measurements, but in this work we were primarily interested in the percentage of time the eyes are closed (PERCLOS). Eye closure was defined by measuring the proportion of the pupil that is occluded by the upper eyelid. PERCLOS is then the proportional amount of time when 80% of the pupil is occluded by the eyelid in a 1 minute time frame. The sampling frequency of this device’s data recording was 30 frames per second. The Sonara/tek (Conshohocken, PA) transcranial Doppler (TCD) unit was used to measure blood flow velocities in the middle cerebral arteries of each participant. It was designed to measure blood flow velocities and other hemodynamic parameters in a non-invasive manner within intracranial and peripheral blood vessels. The Sonara system included an integrated 15” touch screen color LCD display, integrated PC board, and hard disk for data management and display. A 4MHz ultrasound probe frequency was used for this experiment. The ultrasound probes were attached to a helmet and placed on the left and right side of the head over the participant’s temporal window to monitor both the left and right hemispheres. The update rate for this system was 1.1013 Hz. Subjects performed a 40-min vigilance task as described by Funke, Warm, Matthews, Riley, Finomore, Funke, et al. [5]. The task was an air traffic control display where the participant monitored four jet fighters on a circular display divided into four quadrants. Each quadrant contained one triangular jet icon. The jets were presented randomly facing clockwise or counterclockwise around the circular flight path. The subjects were required to respond to cases in which two of the jets were on a collision path (i.e. one jet was oriented in the opposite direction of the rest of the jets. When presented with a case (termed a critical signal), the subjects were to indicate this by pressing the space bar. The variables recorded were percent hits and reaction times that were averaged within four sequential 10 minute blocks because the critical
signal event rate was randomized for every 10 minute period. Twelve of the 300 signals presented in each 10 minute period were critical signals. Real-world vigilance tasks, by their nature, involve long periods of sustained attention while looking for critical signals of which the signal rates are potentially unknown but usually very rare. This makes vigilance exceedingly difficult to study within an operational setting. To counter this difficulty, laboratory vigilance tasks are designed to be very sensitive and devoid of common external distractions like noise, light levels, and other people. The laboratory was isolated from any noise and subjects were required to wear ear plugs to minimize risk of distractions external to the task. Light levels were maintained to be consistent throughout the experiment and glare from the lights onto the task screen was minimized to the extent possible. In this study, a half wall was used to isolate the participant from the experimenter. The experimenters were able to observe the participant but the participant was not able to see the experimenters. While these measures do impact the realism of the laboratory environment, it is a necessary process to be able to facilitate lapses in sustained attention so that they can be experimentally studied. 2.3.
Procedure
No study specific procedures were performed without a written, signed informed consent document. Once completed, subjects received a verbal briefing and PowerPoint presentation that described the vigilance task followed by two 5minute practice sessions. After the practice sessions, the participant was fully instrumented with the TCD and eye-tracker and required to complete the 40-min vigilance task. Afterwards, subjects were released back to their normal duties. For each of the next 3 data collection sessions, the subjects received a full instrumentation and completed the 40-min task. Each data session occurred on a separate day. During the task, subjects’ blood flow velocities within the middle cerebral arteries in each hemisphere were monitored with the TCD while the eye-tracker measured each eye’s oculometrics (as described in the equipment section). 2.4.
Statistical Analysis
Upon completion of testing, eye tracker metrics, blood flow velocities, and vigilance task metrics were averaged in 10min increments (10, 20, 30, 40 min). Univariate repeated-measures analyses of variance (ANOVAs) were used to compare days (1 – 4) and times (10, 20, 30, 40 mins) for each of the variables. Unless otherwise stated, statistical significance was based on alpha=.05. Some of the subjects elected not to complete all 4 data collection sessions (days) and some of the TCD data on various days was discovered to be too noisy (signal less than 20 mL/min) to be usable due to poor signal strength. As a result, it was decided that a participant’s data was not included in the analysis unless there were at least three sessions (days) of usable data. This changed our final number of subjects to 16. Proc Mixed in SAS was used to perform the repeated-measures ANOVAs. This procedure uses maximum likelihood to estimate covariances within a subject and then uses these covariance estimates to estimate coefficients for fixed effects. Least squares means (LSMeans) are means adjusted for missing data and were used since there were some instances of the TCD losing the signal once data collection was underway. Specifically, 9% of the TCD data, 2% of the PERCLOS data, and 0% of the performance data had to be estimated using this method. Next, each individual session (day) for each participant was categorized as either being a “decrement” or “no decrement” day, depending on the percent hits performance change over time within a session (the categorization scheme is described in the next paragraph). This dichotomization was conducted because there was very little consistency within subjects as to whether they tended to have decrements or not on any given day. Indeed it appears to be essentially random and unpredictable. There was an n-size of 13 in the ‘decrement” group and 17 in the “no decrement” group. In
the past, most vigilance researchers chose to throw out the data that did not show a performance decrement, but we thought it may be interesting to include this data as a comparison. Also, to our knowledge, we are also the first vigilance study to use a repeated measures design, and therefore are testing the question of whether some individuals consistently demonstrate decrements, or rarely get them, and to what extent this may vary within individuals. Until now, the implicit assumption in the research community appears to have been that specific individuals are either prone to decrements or prone to avoid them, with subsequent data analyses centered around this (untested) assumption. Participant’s performance was categorized by using linear best-fit slopes of each participant’s percentage of critical signal “hits” calculated across time epochs. Those sessions with a negative slope (i.e. performance decreased over time on a single day), was categorized as a “decrement day” for that participant. Positive slopes or zero + 0.1 slopes (i.e. performance was approximately constant, or increased [positive slope]) were considered a “no decrement” day. For each participant, variables were averaged across days at each time point, separately for decrement and no decrement days. Pearson partial correlations were conducted, controlling for subject effects. These correlations were used to test the relationship between PERCLOS and task performance, and also to test RCBFV versus tasks performance. Separate correlations were computed for sessions classified as either demonstrating a “decrement” or “no decrement” in performance for a particular day of testing.
3. RESULTS Results are segregated into the two types of analysis. First, we present the results for the day and time ANOVAs. Next, we present the results for the correlations of PERCLOS in relation to percent hits and right cerebral blood flow velocity. ANOVAs were used to test for day and time effects for each of the dependent variables. Table 1 shows the variables with the repeated measures ANOVA results with factors day (1, 2, 3, 4) and time (10, 20, 30, 40 minute) and their interaction. Significant F-tests have p-value cells grayed. All dependent variables used data from 16 subjects. Table 1. ANOVA Results for Day and Time (significant values grayed). Dependent Variable DF Percent Hits 3 Left Blood Flow Velocity 3 Right Blood Flow Velocity 3 Left PERCLOS 3 Right PERCLOS 3
DFe 45.0 33.8 39.3 44.2 44.4
Day F
p 1.25 0.64 0.33 3.05 1.61
DF 0.3036 0.5961 0.8037 0.0385 0.1995
3 3 3 3 3
DFe 45.0 38.6 48.8 45.6 45.1
Time F
p 8.66 12.27 17.46 4.39 3.59
DF 0.0001 0.0001 0.0001 0.0086 0.0208
9 9 9 9 9
Day*Time DFe F 134.9 0.57 137 1.06 127.5 2.93 133.4 2.13 132.9 1.56
p 0.8170 0.3952 0.0035 0.0309 0.1333
The Day of data collection had a significant effect only on left PERCLOS. Left PERCLOS was shorter on day 1 than the rest of the days. The biggest difference in PERCLOS between day 1 and any of the other days was 10%, and most were less than 5%. Therefore, even though the effect of Day on left PERCLOS was statistically significant we can offer no clear interpretation for this finding and believe it is not a large enough effect to be meaningful. A possible interpretation could be that there is an initial arousal boost on day 1 due to novelty, which is lost on subsequent days. Time-on-task had significant effects on percent hits, right and left blood flow velocities, and left and right PERCLOS. There was a main effect of Time on percent hits (Figure 1). The LSMean averaged across days for the first 10 minutes of the task was 86% (SEM = 4) and 75% (SEM = 4) for the final 10 minutes. A significant main effect was found for left blood flow velocity. Left cerebral blood flow velocity decreased as time-on-task increased (Figure 1). The LSMeans
across the four days for the first 10 minutes of the task was 47 mL/min (SEM = 2) and 45 mL/min (SEM = 2) for the final 10 minutes of the task. There was also a significant main effect for right blood flow velocity, which was found to decrease with time on task (Figure 1). The LSMean across the four days for the first 10 minutes of the experiment was 46 mL/min (SEM = 2) and 44 mL/min (SEM = 2) for the last 10 minutes of the experiment. Left and right eye PERCLOS increased significantly as a function of time on task (Figure 1). The LSMeans for the first 10 minutes of the task was 6% (SEM = 2) for the left eye and 5% (SEM = 2) for the right eye. During the last 10 minutes of the task the LSMeans were 9% (SEM = 2) for the left eye and 8% (SEM = 2) for the right eye. A
B
80
4 3 2 1
3 2 4 1
75
4 3
3 2 4 1
2 1
70 10
20
1 4 3
46
1 4 3
1
1
4 3
2
Right Blood Flow Velocity (mL/min)
85
C 48
48
Left Blood Flow Velocity (mL/min)
Percent Hits
90
4 3
44
2 2
2
30
40
40
1 4 2
44
3 4 2
1
1
4 3 2
4 3 2
30
40
42
42
30
3 1
46
10
Time (min)
20
10
20
Time (min)
Time (min)
D
E
Left
Right
15
PERCLOS
3
4 3
12 4 3
9 6 3
3 4 2 1
10
4 2
2 1
1
20
30
2 3 1
40
3 4 3 4 2 1
2 1
10
20
4
4 2 1
2 1 3
30
40
Time (min)
Figure 1. (a) Percent Hits Performance by Time-on-Task and Day; (b,c) Blood Flow by Time and Day; and (d,e) PERCLOS by Time and Day. We also found an interaction effect for Day and Time that was statistically significant for right blood flow velocity and left PERCLOS. We can offer no definitive interpretation of these findings, although it appears that they result from changes in right blood flow velocity and left PERCLOS between the 20-minute and 30-minute time interval for the task on day three. Between these time points on day three PERCLOS increases above averages for other days at this same time while right cerebral blood flow velocity decreases below averages compared to other days. While no subjective questionnaires were given to verify our interpretation, we believe by day three subjects are very bored and tired of this task and our results reflect this boredom. Metrics are back to the normal averages compared to days one and two by day four because subjects are excited it is their last day. Pearson partial correlations controlling for subjects were performed (separately for decrement and no decrement days) to relate percent hits with PERCLOS (Table 2). Correlations were also performed to relate right and left cerebral blood flow velocity with percent hits and PERCLOS (Tables 2 and 4). Figures 2 and 3 display the significant partial correlations. If either the decrement or no decrement group were significant, the corresponding correlation is displayed for comparison.
Table 2. Percent Hits Correlated with Other Variables (significant values grayed).
Percent Hits Decrement No Decrement r p r p 0.29 0.0676 0.05 0.7178 0.54 0.0003 -0.09 0.5137 -0.72 0.0001 -0.56 0.0001 -0.67 0.0001 -0.53 0.0001
Variable Correlated With Left Blood Flow Velocity Right Blood Flow Velocity Left PERCLOS Right PERCLOS
Right blood flow velocity and left and right PERCLOS were correlated with percent hits performance on the vigilance task in the decrement group (Figures 2 and 3). The right blood flow velocity of the decrement group positively correlated with percent hits (Figure 2) suggesting that as performance (as measured by percent hits) decreased, right cerebral blood flow velocity decreased, as expected per previous results [6, 18, and 7]. Notably, right cerebral blood flow did not significantly correlate with performance in the no decrement group. PERCLOS in the left and right eye negatively correlated with percent hits performance in both groups (Figure 3). For the decrement group, the correlation between performance and PERCLOS was particularly strong in both eyes, relative to right cerebral blood flow velocity (-.72 and .67 for the left and right eye PERCLOS, respectively, versus .54 for the right blood flow velocity). For the no decrement group, the correlation between performance and PERCLOS was less strong but still large in magnitude and statistically significant (i.e., correlations with PERCLOS over .5 for both eyes); we see in Figure 3 that vigilance performance began to decline before returning to baseline levels during the last 10 minutes of the task. Left (r=-.56, p=.0001) and right (r=.53, p=.0001) PERCLOS also appear to follow this trend in reverse (with a return to baseline in the last epoch) as seen in Figure 3. The evidence suggests that the decrease in percent hits is coupled with an increase in PERCLOS. It is important to note that as percent hits values returned to baseline levels, PERCLOS was observed to decrease as well. Using Soper’s [16] method of testing significant differences between two correlation strengths, we confirmed that the relationship between [Left PERCLOS and performance] is stronger than the relationship between [rCBFV and performance]: z=3.41, p
