Driver State Monitor from DELPHI Proposal for Demonstration at IEEE CVPR 2005, San Diego, CA N. Edenborough, R. Hammoud, A. Harbach, A. Ingold, B. Kisaˇcanin, P. Malawey, T. Newman, G. Scharenbroch, S. Skiver, M. Smith, A. Wilhelm, G. Witt∗, E. Yoder, and H. Zhang
Abstract We present an automotive-grade, real-time, vision-based Driver State Monitor. Upon detecting and tracking the driver’s facial features, the system analyzes eye-closures and head pose to infer his/her fatigue or distraction. This information is used to warn the driver and to modulate the actions of other safety systems. The purpose of this monitor is to increase road safety by preventing drivers from falling asleep or from being overly distracted, and to improve the effectiveness of other safety systems.
1. Introduction Advanced automotive safety systems, such as Driver State Monitor (DSM), are aimed at further reducing fatalities, injuries, and damage in road accidents, below the rates achieved by use of restraining belts and air bags [8, 9]. While the mandated installation of frontal and side airbags has significantly improved the road safety, even as recently as in 2000 and 2001, motor vehicle crashes were still the leading cause of death in children and people ages 4–34 in the US [10, 11]. While published statistics vary, the general consensus is that drivers falling asleep cause too many of the total number of accidents [4, 5]. It is estimated that in the early 1990s around 1550 fatalities annually resulted from drowsiness and fatigue related accidents [6, 7]. While restraining belts and airbags help only in accident mitigation, advanced safety systems will have an active role in accident prevention. In particular, the purpose of the DSM is to prevent drivers from falling asleep or being too distracted with non-driving tasks, and to provide inputs to other safety systems, thereby improving their effectiveness. Detection of fatigue and distraction in drivers is a complex task, combining expertise in computer vision, human factors, automotive systems, mechanical, optical, electrical, and software engineering. In the following we provide only a rudimentary description of the requirements imposed by human factors and the automotive environment and briefly explain our answer to this challenge — ∗ The team is with Delphi Electronics & Safety. For more information about DSM, e-mail to Gerald Witt (
[email protected]).
0-7695-2372-2/05/$20.00 (c) 2005 IEEE
Figure 1: Features of the Driver State Monitor are demonstrated on the “Portable DSM,” consisting of camera, IR illuminator, embedded processing unit, and the display
an automotive-grade, real-time, vision-based, Driver State Monitor (Fig. 1).
2. System Considerations Next, we discuss several system-level requirements on DSM.
2.1. DSM is Automotive-Grade Ensuring that the system works in automotive conditions was the most difficult challenge we had. Some of the automotive requirements addressed were: wide range of illumination conditions and operating temperatures, coverage of the 95th percentile ellipsoid of driver head positions, available camera locations, system cost, heat dissipation, EMI, allowable levels of IR irradiation, and subject variability. These requirements influence every level of the design, including the use of near-IR illumination and sensors as well as the image resolution of 640×480 pixels. We also had to find an alternative to general-purpose processors in a form
of media processors, which combine high-end DSP cores with on-chip video peripherals.
2.2. Real-Time Information Processing One of the system requirements imposed by the human factors research into measures of drowsiness and distraction [1, 3] is the video refresh rate of at least 30 frames per second. This frame rate defines our real-time performance: the system has to process every frame in less than 33 ms.
2.3. Vision-Based Application We considered drowsiness and distraction measures other than the ones based on computer vision, such as driving performance and heart and breathing rates. While these other modalities carry important information, here we concentrate on the computer vision approach, because it offers the most direct indication of early onset of sleepiness and distraction and is seen as a great platform to be shared with other vision-based driver assistance applications in the future.
Figure 2: Video output from the DSM system showing the driver, the eye tracking region, and the fatigue and distraction levels
References
We now describe the demonstration format.
[1] D. F. Dinges, M. M. Mallis, G. Maislin, and J. W. Powell, “Evaluation of techniques for ocular measurement as an index of fatigue and the basis for alertness management,” DOT HS 808-762, 1998.
3.1. Demonstration Format
[2] N. Edenborough et al., “Driver Drowsiness Monitor from DELPHI,” in CD-ROM Proceedings of IEEE CVPR, 2004.
While we often demonstrate DSM in concept and test vehicles, the best format for this particular occasion seemed to be the “Portable DSM,” shown in Fig. 1. We use it to show the most interesting features of our system:
[3] R. J. Fairbanks, S. E. Fahey, and W. W. Wierwille, “Research on vehicle-based driver status/performance monitoring,” DOT HS 808-299, 1995.
• Hands-free operation – For a significant majority of people, the system will detect, track, and analyze their eyes without the need for profile creation or any other user intervention.
[5] W. D. Jones, “Building safer cars,” IEEE Spectrum, Vol. 39, pp. 82–85, Jan. 2002.
3. DSM Demonstration
• Natural behavior – The system allows the user to move and behave naturally. • Fatigue detection – A drowsiness measure based on a 1 minute sliding window is being constantly updated. • Distraction detection – A distraction measure based head pose analysis is displayed.
3.2. Optional Video Output DSM as a product will not have a display, but for our own development, experiments, and demonstrations we found it extremely useful to display the video from the camera and overlay it with relevant information. We show a typical “customer view” in Fig. 2, with relevant outputs regarding the driver fatigue and distraction.
[4] W. D. Jones, “Keeping cars from crashing,” IEEE Spectrum, Vol. 38, pp. 40–45, Sept. 2001.
[6] R. R. Knipling and J. S. Wang, “Crashes and Fatalities Related to Driver Drowsiness/Fatigue,” NHTSA Research Notes, 1994. [7] R. R. Knipling and J. S. Wang, “Revised Estimates of the US Drowsy Driver Crash Problem Size Based on General Estimate System Case Reviews,” 39th Annual Proceedings AAAM, 1995. [8] W. K. Kosiak, W. W. Fultz, G. J. Witt, “Interior Sensing for Automotive Occupant Safety,” SAE 2002-21-0031, 2002. [9] S. N. Rohr, R. C. Lind, R. J. Myers, W. A. Bauson, W. K. Kosiak, and H. Yen, “An Integrated Approach to Automotive Safety Systems,” SAE 2000-01-0346, 2000. [10] R. Subramanian, “Motor Vehicle Traffic Crashes as a Leading Cause of Death in the US, 2000,” DOT HS 809-661, 2003. [11] R. Subramanian, “Motor Vehicle Traffic Crashes as a Leading Cause of Death in the US, 2001,” DOT HS 809-695, 2003.
