Driver Drowsiness Monitor from DELPHI Demonstration at IEEE CVPR 2004, Washington, DC N. Edenborough, R. Hammoud, A. Harbach, A. Ingold, B. Kisaˇcanin, P. Malawey, T. Newman, G. Scharenbroch, S. Skiver, M. Smith, A. Wilhelm, G. Witt∗, E. Yoder, and H. Zhang
Abstract We present an automotive-grade, real-time, vision-based Driver Drowsiness Monitor. Upon detecting and tracking the driver’s eyes, the system analyzes eye-closures to infer his/her drowsiness. This information is used to warn the driver and to modulate the actions of other safety systems. The purpose of this monitor is to increase road safety by preventing drivers from falling asleep and to improve the effectiveness of other safety systems.
1. Introduction Advanced automotive safety systems, such as Driver Drowsiness Monitor (DDM), are aimed at further reducing fatalities, injuries, and damages in road accidents, below the rates achieved by use of restraining belts and air bags [7, 8]. While the mandated installation of frontal and side airbags has significantly improved the road safety, even as recently as in 2000 and 2001, motor vehicle crashes were still the leading cause of death in children and people ages 4–34 in the US [9, 10]. While published statistics vary, the general consensus is that drivers falling asleep cause too many of the total number of accidents [3, 4]. It is estimated that in the early 1990’s around 1550 fatalities annually resulted from drowsiness and fatigue related accidents [5, 6]. While restraining belts and airbags help only in accident mitigation, advanced safety systems will have an active role in accident prevention. In particular, the purpose of DDM is to prevent drivers from falling asleep and to provide inputs to other safety systems, thereby improving their effectiveness. Detection of drowsiness in drivers is a complex task, combining expertise in computer vision, human factors, automotive systems, mechanical, optical, electrical, and software engineering. In the following we provide only a rudimentary description of the requirements imposed by human factors and the automotive environment and briefly explain our answer to this challenge — an automotive-grade, realtime, vision-based, Driver Drowsiness Monitor (Figure 1). ∗ The
team is with Delphi Electronics & Safety. For more information about DDM, e-mail to Gerald Witt (
[email protected]).
0-7695-2158-4/04 $20.00 (C) 2004 IEEE
Figure 1: Features of the Driver Drowsiness Monitor are demonstrated on the “DDM Kiosk,” consisting of camera, IR illuminator, embedded processing unit, and the display.
2. System Considerations 2.1. DDM is Automotive-Grade Ensuring that the system works in automotive conditions was the most difficult challenge we had. Some of the automotive requirements addressed were: wide range of illumination conditions and operating temperatures, coverage of the 95th percentile ellipsoid of driver head positions, available camera locations, system cost, heat dissipation, EMI, and allowable levels of IR irradiation. These requirements influence every level of the design, including the use of near-IR illumination and sensors as well as the image resolution of 640×480 pixels. We also had to find an alternative to general-purpose processors in a form of media processors, which combine high-end DSP cores with on-chip video peripherals.
2.2. Real-Time Information Processing To detect the early onset of sleepiness, we use a drowsiness measure based on eye-closures [1, 2]. One of the system requirements imposed by this choice is the video refresh
rate of at least 30 frames per second. This frame rate defines our real-time performance: the system has to process every frame in less than 33ms.
2.3. Vision-Based Application We considered fatigue and drowsiness measures other than the ones based on eye-closures, such as driving performance and heart and breathing rates. The computer vision approach that measures drowsiness based on eye-closures offered the most direct indication of early onset of sleepiness and was seen as a great platform to be shared with other vision-based safety and security applications in the future.
3. DDM Demonstration 3.1. Demonstration Format While we often demonstrate DDM in concept and test vehicles, the best format for this particular occasion seemed to be the “DDM Kiosk,” shown in Figure 1. We use it to show the most interesting features of our system:
Figure 2: Video output from the system showing the driver, the eye tracking region, the value of the drowsiness measure, and the corresponding drowsiness level.
References
• Hands-free operation – For a significant majority of people, the system will detect, track, and analyze their eyes without the need for profile creation or any other user intervention.
[1] D. F. Dinges, M. M. Mallis, G. Maislin, and J. W. Powell, “Evaluation of techniques for ocular measurement as an index of fatigue and the basis for alertness management,” DOT HS 808-762, 1998.
• Natural behavior – The system allows the user to move and behave naturally.
[2] R. J. Fairbanks, S. E. Fahey, and W. W. Wierwille, “Research on vehicle-based driver status/performance monitoring,” DOT HS 808-299, 1995.
• Short-term drowsiness measure – An “Extended Eye Closure Warning” is issued if the user closes his/her eyes for longer than 2 seconds. • Long-term drowsiness measure – A drowsiness measure based on a 1 minute sliding window is being constantly updated.
3.2. Optional Video Output DDM as a product will not have a display, but for our own development, experiments, and demonstrations we found it extremely useful to display the video from the camera and overlay it with relevant information. While the “diagnostic view” contains much more information, we use the “customer view” for demonstrations. As shown in Figure 2, this view shows the tracking windows (green rectangles) and the long-term drowsiness measure (number and color bars at the bottom). The “Extended Eye Closure Warning” is displayed by changing the color of tracking windows from green to purple.
0-7695-2158-4/04 $20.00 (C) 2004 IEEE
[3] W. D. Jones, “Keeping cars from crashing,” IEEE Spectrum, Vol. 38, pp. 40 - 45, Sept. 2001. [4] W. D. Jones, “Building safer cars,” IEEE Spectrum, Vol. 39, pp. 82 - 85, Jan. 2002. [5] R. R. Knipling and J. S. Wang, “Crashes and Fatalities Related to Driver Drowsiness/Fatigue,” NHTSA Research Notes, 1994. [6] R. R. Knipling and J. S. Wang, “Revised Estimates of the US Drowsy Driver Crash Problem Size Based on General Estimate System Case Reviews,” 39th Annual Proceedings AAAM, 1995. [7] W. K. Kosiak, W. W. Fultz, G. J. Witt, “Interior Sensing for Automotive Occupant Safety,” SAE 2002-21-0031, 2002. [8] S. N. Rohr, R. C. Lind, R. J. Myers, W. A. Bauson, W. K. Kosiak, and H. Yen, “An Integrated Approach to Automotive Safety Systems,” SAE 2000-01-0346, 2000. [9] R. Subramanian, “Motor Vehicle Traffic Crashes as a Leading Cause of Death in the US, 2000,” DOT HS 809-661, 2003. [10] R. Subramanian, “Motor Vehicle Traffic Crashes as a Leading Cause of Death in the US, 2001,” DOT HS 809-695, 2003.
