NON-INTRUSIVE REAL TIME HUMAN FATIGUE. MODELLING AND MONITORING. Peilin Lan, Qiang Jiâ, Carl G. Looney. Department of Computer Science, ...
NON-INTRUSIVE REAL TIME HUMAN FATIGUE MODELLING AND MONITORING Peilin Lan, Qiang Ji∗ , Carl G. Looney Department of Computer Science, University of Nevada at Reno, NV 89557 ∗
Department of ECSE, Rensselaer Polytechnic Institute, Troy, NY 12180
Abstract In this paper, we introduce a probabilistic model based on the Dynamic Bayesian Networks (DBNs) for dynamically modelling and detecting human fatigue. We first present a static fatigue model that captures the static relationships between fatigue, significant factors that cause fatigue, and various visual cues that are typically resulted from fatigue. The static fatigue model allows to spatially integrate fatigue evidences from different sources. It, however, fails to capture the dynamic aspect of fatigue. Fatigue is a cognitive state that is developed over time. To account for the temporal aspect of human fatigue, the static fatigue model is extended based on the DBNs. The dynamic fatigue model allows to integrate fatigue evidences not only spatially but also temporally, therefore leading to a more robust fatigue modelling and inference. The evaluation of the fatigue model with simulated data reveals the satisfactory performance of the proposed fatigue model. The dynamic fatigue model is then integrated with our computer vision module to perform non-intrusive real-time fatigue monitoring and detection.
1
Introduction
Fatigue has been widely accepted as a significant factor in a variety of commercial transportation accidents [1]. The relationship between drivers’ fatigue and the causes of accidents in aviation, railroad, highway and marine, has been established in many large scale research reports [1] [11]. Therefore, human fatigue monitoring and prevention is essential to improving the commercial transportation safety. Since several decades ago, much research has been conducted on human fatigue prevention and numerous fatigue monitoring systems have been developed [14] [10] [3] [7] [15]. But most of these systems have not been proven effective for real world applications. The main reason for their failures is that these systems often use limited information (often just from one source) to evaluate human fatigue and, furthermore, fatigue in these systems is not evaluated dynamically. Therefore, as noted by recent studies [4] [7], much more effort is still needed to develop improved fatigue detection and monitoring systems, that can simultaneously use information from multiple sources, and that systematically integrate these information sources over time. In our previous study [8], based on the modern research achievements about fatigue [11] [2] [15] [14] [12] and our successful computer vision studies on non-invasive human fatigue [7] [6], we introduced a static probabilistic framework via Bayesian Networks (BNs) for modelling and inferring fatigue. This model systematically represents and integrates the available visual and contextual information from different sources to infer and detect fatigue. Our previous system represents a significant improvement over the existing fatigue monitoring system in that it simultaneously uses information from multiple sources. Further, it systematically represents the fatigue causing factors and
the fatigue observations at different levels of abstraction in a probabilistic framework. This leads to a robust and consistent fatigue characterization. The previous system, however, fails to capture the dynamic aspect of fatigue. Fatigue develops over time. It is the persistent presence of certain visual cues over time that indicate the onset/presence of the fatigue. A fatigue model that can efficiently capture the temporal evolution of fatigue will be more effective in fatigue monitoring and detection. In this paper, we extend the static fatigue model into a dynamic model via DBNs to combine different information sources not only spatially, but also temporally for monitoring and inferring human fatigue. Also, we will discuss the interface we developed for integrating the computer vision system with the fatigue inference engine for real time non-intrusive fatigue monitoring and detection.
2
Fatigue Modelling with DBNs
In our previous study [8], we identified the significant factors that cause human fatigue, and various visual cues that are typically symptoms (observations) of fatigue. We then constructed a static Bayesian Network (SBN) to casually relate these factors as shown in Fig. 1. The target node in the middle of the network represents fatigue variable. It is what we want to infer. The nodes above the target node represent various factors that could lead to fatigue. They are collectively referred to as the contextual information. Typical fatigue-causing factors include sleep quality, time of day, and physical fitness. These factors, along with other fatigue-inducing factors, are systematically represented by the contextual nodes above the fatigue node. The nodes below the target node represent various visual observations from the output of the computer vision sys-
Random_noise
Temperature
Light
Heat
Sleep_Enviornment
Sleep_time
Weather
Noise
Humidity
Napping
Anxiety
Sleep_state
Time_zone
Sleep_disorders
Work_enviornment
Sleep_quality
Physical_condition
Time
Workload
Circadian
Work_type
Work_condition
Fatigue
Facial_Exp
YawnFreq
Facial_Muscle
PERCLOS
Eye_movement
Eyelid_movement
AECS
Head_movement
Gaze
Fixation_dis
Head_tilt_freq
Fixation_saccade_ratio
Figure 1: A static Bayesian Network for Modelling Human Fatigue tem. They are collectively referred to as the observation nodes. The observational nodes represent various visual cues that typically reflect one’s level of vigilance. However, as pointed by some recent studies [15], fatigue has an accumulative property and fatigue gradually develops over time. For example, a driver’s fatigue level at the beginning of driving may be low, but it will gradually increase with time. This fact indicates that, in addition to sleep, circadian and some other environment factors, the fatigue status in the previous time is also a significant factor of current fatigue status. Furthermore, for fatigue detection, it is the persistent presence of certain visual behaviors over time instead of the presence of the behavior at a particular instance that leads to the detection of fatigue. So, it is important for a fatigue model to account for the temporal aspect of fatigue and to integrate fatigue evidences over time. Obviously, the static model fails to capture these dynamic aspects. As an extension of the traditional static Bayesian Networks, DBNs describes a system that is dynamically evolving over time and enables the user to monitor and update the system as time proceeds. It even predicts the behavior of the system over time [9]. Therefore, a fatigue model based on the DBNs is naturally the best option to model and predict fatigue over time. According to the DBNs, time is discrete and the activities within each time interval can be modelled statically using a hierarchical Bayesian Network. The dynamic relationships between the static structures in different time intervals can be modelled by a Hidden Markov Model (HMM), whereby the nodes at some time T are influenced only by the nodes at the same time or at the previous time point T − 1. Based on general DBNs principles [5] [9] [13] and the above considerations, a DBNs model for modelling human fatigue is constructed as shown
in Figure 2. The basic idea of this model is that some hidden nodes (including the fatigue node) at the previous time slice are connected to the corresponding nodes at current time. The previous nodes, therefore, provide a diagnostic support for the corresponding variables at present time. Thus, fatigue at current time is inferred from fatigue at the previous time, along with current observations. These changes allow to perform fatigue estimation over time by integrating information over time. It also affords to predict fatigue over time by the temporal causality of DBNs.
The DBNs are implemented by keeping in memory two slices at any one time, representing previous time interval and current time interval respectively. The slice at the previous time interval provides diagnostic support for current slice. The two slices are such programmed that they rotate as old slices are dropped and new slices are used as time progresses. Specifically, at the start time slice, fatigue is inferred from the static fatigue BNs model in Figure 1. Starting from the second time slice, the static fatigue model is expanded dynamically with additional temporal links that connect the intermediate nodes including the fatigue node at previous time slice to the corresponding nodes at current time slice. Fatigue inference is then performed on the expanded static fatigue model. This repeats with different probabilities for the previous nodes that are connected to current model. All the CPTs in the model is time-invariant. Part of the CPTs and prior probabilities in the model are adopted from the previous SBNs model and the transitional probabilities are specified subjectively. Theoretically, the transitional probabilities can also vary as a function of time elapsing between two consecutive slices.
Con_Fatigue
Fatique_t_1
Facial_Exp_t_1
Eye_movement_t_1
Con_Fatigue
Head_movement_t_1
Fatigue_t YawnFreq
Facial_muscle
PERCLOS
Eyelid_movement
AECS
Gaze
Fixation_dis
Head_tilt_freq
Facial_Exp_t
Fixation_saccade_ratio
YawnFreq
Time Slice t-1
Facial_muscle
PERCLOS
Eye_movement_t
Eyelid_movement
AECS
Head_movement_t
Gaze
Fixation_dis
Head_tilt_freq
Fixation_saccade_ratio
Time Slice t
Figure 2: Dynamic Bayesian Networks Model for Monitoring Human Fatigue
3
Experiment Results and Discussion
After parameterizing the dynamic fatigue models, we can start the inference upon the arrival of new evidences. We developed a computer program to infer the fatigue level at every time slice. In the program, evidences are automatically collected from the computer vision module, which updates the evidence file periodically. The typical inference results from both the static and dynamic fatigue models are plotted in Figures 3 and 4. Figure 3 plots the fatigue index for both models over time, with the observation of high PERCLOS value at different time slices. It is clear that the two models respond to evidences differently. In general, the fatigue curve for dynamic fatigue model changes gradually upon the arrival of an evidence or disappearance of an evidence. On the other hand, the fatigue curve for the static model changes drastically with evidences arrival or disappearance, as represented by sharp turns. The curves also show that the presence of high PERCLOS at one time slice (slice 2) does not cause a significant fatigue change for the dynamic model while it can lead to a sudden change for the static model. However, if high PERCLOS value is observed continuously (e.g. from time 11 to 20), this will cause a gradual increase of fatigue for the dynamic model. The static model stays unchanged over this period. As soon as disappearance of the PERCLOS evidence, the
static model drops immediately while the dynamic model gradually decreases. Therefore, the dynamic fatigue model is more compatible with fatigue development. And, the dynamic model is more tolerant to external signal disturbance. The fatigue index from the dynamic fatigue model begins to change only after persistent and continuous observations of certain visual behavior. We can reach the similar conclusion from Figure 4, where multiple fatigue parameters may be instantiated simultaneously. In summary, the dynamic fatigue model can more accurately characterize and monitor human fatigue.
4
Interfacing with the Vision System
To perform real-time human fatigue monitoring, the computer vision module [7] [6] and the fatigue inference module must be combined via an interface program such that the output of the vision system can be used by the inference module to update its belief in fatigue in real time. Such interface was developed in Visual Basic 6.0 environment. Figure 5 shows the appearance of the interface. The interface program periodically (every 1 second or even shorter period) examines the output (evidences) of the vision module and detects any evidence change. If a change is detected, the interface program instantiates the corresponding observations nodes in the fusion module, which
0.9
0.85 PERCLOS evidence (hihg)
PERCLOS evidence (high)
Fatigue Level /1.0000
0.8
0.75
0.7 DBN model 0.65
0.6 SBN Model
0.55
0
5
10
15
20
25
30
35
Time Slices
Figure 3: Fatigue level changes over time for both the static and dynamic fatigue models with observation of high PERCLOS value at time slice 1 and between time slice 12 and 20.
1 PERCLOS(high)+YawnFreq(high)
PERCLOS(high)+YawnFreq(high)
0.9
0.8 DBN model
Fatigue Level /1.0000
0.7 no evidence
0.6
SBN Model
0.5
0.4 PERCLOS(normal)+YawnFreq(normal) 0.3
0.2 PERCLOS(normal)+YawnFreq(normal)
0.1
0
0
10
20
30 Time Slices
40
50
60
Figure 4: Fatigue level changes over time for both the static and dynamic fatigue models with observation of multiple evidences at different time slices. The evidence from PERCLOS (high) and YawnFreq (high) are observed at time slice 2 and between 22 and 29, and the evidences from PERCLOS(normal) and YawnFreq (normal) are observed at time slice 3 and between time slice 30 and 36
then activates its inference engine, and obtain the new fatigue level. The interface program plots the inference result (fatigue level) in real-time in a separate window. The output fatigue index varies over time as the visual cues change. When the fatigue index exceeds a preset threshold, the color of the screen becomes red and begins to flash continuously. This is accompanied with a warning sound to alert the person. The program also allows to manually instantiate the evidences for debugging purpose.
Acknowledgment This project is supported by a grant from the US Air Force Office of Scientific Research.
References [1] National Transportation Safety Board. Evaluation of us department of transportation efforts in the 1990s to address operator fatigue. Safety Report NTSB/SR-99/01, National Transportation Safety Board, May 1999. PB99-91 7002 Notation 7155. [2] E. L. Co, K. B. Gregory, J. M. Johnson, and M. R. Rosekind. Crew factors in flight operations xi: A survey of fatigue factors in regional airline operations. Technical Report NASA/TM-1999-208799, National Aeronautics and Space Administration, Ames Research Center Moffett Field, California 94035, October 1999. [3] C. A. Czeisler. Lecture on fatigue. In Fatigue Symposium Proceedings, page 59. National Transportation Safety Board and NASA Ames Research Center, November 1995. [4] L. Hartley, T. Horberry, N. Mabbott, and G. P. Krueger. Review of fatigue detection and prediction technologies. Technical Report ISBN 0 642 54469 7, National Road Transport Commission, Level 5/326 William Street MELBOURNE VIC 3000, September 2000. www.nrtc.gov.au. [5] F. V. Jensen. Bayesian Networks and Decision Graphs, Statistics for Engineering and Information Science. Springer, 2001. [6] Qiang Ji and Xiaojie Yang. Real time visual cues extraction for monitoring driver vigilance. In International Workshop on Computer Vision Systems, Vancouver, Canada, July 2001. [7] Qiang Ji and Xiaojie Yang. A computer vision system for real time driver alertness monitoring. Real Time Imaging, 2002. in Press. [8] Peilin Lan, Qiang Ji, and Carl Looney. Information fusion with bayesian belief networks for monitoring human fatigue. In Proceedings of the 5th International Conference on Information Fusion (FUSION 2002), Annapolis, Maryland, July 2002. International Society of Information Fusion.
Figure 5: Interface for DBN model
5
Conclusions
Through the research presented in this paper, we present a probabilistic model based on the DBNs for modelling and detecting human fatigue over time. The fatigue model integrates the relevant contextual information and the available visual cues over time to infer and predict human fatigue. The simultaneous use of multiple visual cues, along with the relevant contextual information, and the integration of these information over time, lead to a robust, consistent, and accurate fatigue characterization. In addition, the fatigue inference engine has been integrated with our computer vision system to allow real time non-intrusive human fatigue monitoring and detection. Study involving human subjects is currently underway to scientifically validate our fatigue model against the established fatigue measures such as EEG and psychomotor vigilance task (PVT) performance lapses.
[9] V. Mihaljlovic and M. Petkovi. Dynamic bayesian networks: A state of the art, computer science department. Technical report, Computer Science Department, University of Twente, The Netherlands, 2001. [10] M. R. Rosekind. Physiological considerations of fatigue. In Fatigue Symposium Proceedings, page 17. National Transportation Safety Board and NASA Ames Research Center, November 1995. [11] M. R. Rosekind, E. L. Co, Gregory, K. B., and D. L. Miller. Crew factors in flight operations xiii: A survey of fatigue factors in corporate/executive aviation operations. Technical Report NASA/TM2000-209610, Ames Research Center Moffett Field at National Aeronautics and Space Administration, California 94035, September 2000. [12] M. R. Rosekind, K. B. Gregory, E. L. Co, D. L. Miller, and D. F. Dinges. Crew factors in flight operations xii: A survey of sleep quantity and quality in on-board crew rest facilities. Technical Report NASA/TM-2000-209611, National Aeronautics and Space Administration, Ames Research Center Moffett Field, California 94035, September 2000. [13] R. Schafer and T. Weyrath. Assessing temporally variable user properties with dynamic baysian networks. In User Modelling: Proceeding of the Sixth International Conference, pages 377–388, 1997. [14] P. Sherry. Fatigue countermeasures in the railroad industry-past and current developments. Technical report, Counselling Psychology Program, Inter-modal Transportation Institute, University of Denver, June 2000. [15] C. D. Wylie, T. Shultz, J. C. Miller, M. M. Mitler, and R. R. Mackie. Commercial motor vehicle driver fatigue and alertness study: Technical summary. Technical Report FHWA-MC-97-001, Trucking Research Institute, American Trucking Associations Foundation, Transportation Development Center, Safety and Security Transport Canada, November 1996. TC report number: TP 12876E.
