Driver Modeling for Detection and Assessment of ... - IEEE Xplore

John H.L. Hansen, Carlos Busso, Yang Zheng, and Amardeep Sathyanarayana

V

ehicle technologies have advanced significantly over the past 20 years, especially with respect to novel in-vehicle systems for route navigation, information access, infotainment, and connected vehicle advancements for vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) connectivity and communications. While there is great interest in migrating to fully automated, self-driving vehicles, factors such as technology performance, cost barriers, public safety, insurance issues, legal implications, and government regulations suggest it is more likely that the first step in the progression will be multifunctional vehicles. Today, embedded controllers as well as a variety of sensors and high-performance computing in present-day cars allow for a smooth transition from complete human control toward semisupervised or assisted control, then to fully automated vehicles. Next-generation vehicles will need to be image licensed by ingram publishing, sky—©graphic stock

Driver Modeling for Detection and Assessment of Distraction Examples from the UTDrive Test Bed Digital Object Identifier 10.1109/MSP.2017.2699039 Date of publication: 11 July 2017

130

IEEE Signal Processing Magazine

|

July 2017

|

1053-5888/17©2017IEEE

The need for driver modeling research

more active in assessing driver awareness, vehicle capabilities, and traffic and environmental settings, plus how these factors come together to determine a collaborative safe and effective driver–vehicle engagement for vehicle operation. This article reviews a range of issues pertaining to driver modeling for the detection and assessment of distraction. Examples from the UTDrive project are used whenever possible, along with a comparison to existing research programs. The areas ad dressed include 1) understanding driver behavior and distraction, 2) maneuver recognition and distraction analysis, 3) glance behavior and visual tracking, and 4) mobile platform advancements for in-vehicle data collection and human– machine interface. This article highlights challenges in achieving effective modeling, detection, and assessment of driver distraction using both UTDrive instrumented vehicle data and naturalistic driving data.

Over the past few years, there has been a significant effort in establishing smart cars that have the capability to achieve self/autonomous driving for their passengers or the passive operator. While great strides in autonomous driving will continue, greater research and understanding is needed regarding driver modeling as we transition from full-driver control to various levels of assistive-through-fully automated vehicles. The ability for smart cars to seamlessly move back and forth between com pletely automated, semiautomated, semiassistive, and unassisted cars remains a major challenge. In this article, we consider an overview of the recent advancements in driver modeling to as sess driver status, including the detection and assessment of driver distraction when the vehicle is operated in a user-controlled scenario. The recent large-scale data collection undertaken by the United States Transportation Research Board—Strategic Highway Research Pro gram [1], [91] will provide an enormous (+2 Petabytes) data set of naturalistic data. The ability for researchers to mine this corpus to develop better models of driver status will offer new insights into nextgeneration smart vehicles, which have the capability of migrating between being completely user controlled to fully autonomous. An extensive amount of research and development is currently being conducted by many laboratories in the United States, Japan, Germany, Sweden, South Korea, and other countries; it is therefore not possible to provide exhaustive coverage of all significant advancements. Instead, the goal here is to provide a representative look at the topic of driver modeling, focusing on how advancing technologies impact driver distraction. This includes a range of signal processing technologies related to controller area network (CAN) bus analysis, image/video processing, speech/ audio for human–machine interaction, and other advancements leading to current and future intelligent assistance in the vehicle.


|

July 2017

|

131

deterioration is directly attributed to the driver (distraction, A recent IEEE Signal Processing Magazine special issue, inattention), the vehicle (condition, familiarity), or the sur“Smart Vehicle Technologies: Signal Processing on the Move” rounding environment (traffic, weather). Both driver distraction [2], considered a range of topics for smart vehicles advanceand driver inattention are frequently occurring events in a car. ments that included driver behavior modeling using on-road Driver inattention is defined as insufficient or no attendriving data [3], driver status monitoring systems [4], smart tion given to activities critical for safe driving. Inattention can driver monitoring [5], conversational in-vehicle dialog systems either be a voluntary or involuntary diversion of attention by [6], active noise control in cars [7], and coordinated autonothe driver [15]. Driver distraction has been formally defined mous vehicles [8]. In this article, we provide several compleas “[a]nything that delays the recognition of information necmentary highlights to these excellent overview articles. Several essary to safely maintain the lateral and experiments/data sets have collected inforlongitudinal control of the vehicle (primary mation on driver behavior analysis [9]–[11]. The ability for smart driving task) due to some event, activity, For the sake of illustration, the UTDrive cars to seamlessly move object or person, within or outside the vehinaturalistic driving data set [12] has been back and forth between cle (agent) that compels or tends to induce conducted by the Center for Robust Speech completely automated, the driver’s shifting attention away from the Systems (CRSS)-UTDrive since 2006, with fundamental driving task (mechanism) by the interest of understanding driver behavior semiautomated, compromising the driver’s auditory, bioand distraction from multichannel sensor data semiassistive, and mechanical, cognitive or visual faculties or (see Figure 1) [13]. Here, we focus on current unassisted cars remains combinations thereof (type)” [16]. Without advancements, past efforts, and directions a major challenge. these formal definitions, cross-study comfor future research. Examples stemming parisons cannot be made and statistics can from the UTDrive project are highlighted vary drastically, leading to incorrect observations [15], [16]. as examples, as well as efforts from the Virginia Tech TransIt is important to note that driver distractions are generally portation Institute, the University of Michigan Transportation caused by a competing trigger activity that may lead to driver Research Institute, the University of California, San Diego, inattention, which in turn degrades driving performance. Alterplus studies conducted in Europe, Japan, and South Korea [14]. natively, other forms of driver inattention might not necessarily be due to a trigger or competing activity, making inattention Understanding driver behavior difficult to detect and even harder to control. By identifying and driving distraction some of the causes of driver distraction, it is possible to isolate Driver activities performed within the vehicle can be broadly scenarios when the cause of distraction can be controlled. classified into primary tasks that are essential for operating and Most secondary tasks are not distracting and do not require directing the course of a vehicle in a given environment and secthe complete attention of the driver. However, while executondary tasks that are not essential or related to the primary task ing a complex task such as driving, most the driver’s attenof driving. Secondary tasks divert drivers’ primary attention tion is toward a safe drive, and performing a secondary task of driving and degrade their driving performance. The means sharing limited available human cognitive resources. Some important characteristics related to secondary tasks that distract the driver include the duration of the activity, the freFront View quency of the activity, the attention required to execute the activity (attention demand), the ease of returning to the primaDriver Microphone Array Close-Talk ry task of driving, the location and time at which the activity is Microphone executed, and the individual driver’s comfort in executing the Cameras task and in performing multiple tasks. Since visual modality has been well studied, it has been established that diversion of the driver’s visual focus away from the task of driving for more GPS than 1.5 s distracts the driver [17]. Optical The driver follows road rules and maintains his or her lane Distance as well as an acceptable gap between the car and surroundSensor ing vehicles, all while achieving good reaction time to changes such as traffic signs and taillights [18]. From the vehicle conGas/Brake CAN-Bus Data Acquisition Unit trol side, the driver’s primary physical contacts are the steering Pedal Pressure OBD II (16 Channel: Two Video, Sensors wheel, the gas and brake pedals, the seat, and the ego vehicle Six Audio, CAN-Bus; (i.e., the targeted, controlling vehicle itself) speed as reference. All Synchronized) Any secondary task that distracts the driver has a direct influence on body movements that manifest in control of the vehicle. Hence, a change in driving performance can be evaluated FIGURE 1. The UTDrive experiment test bed: synchronized multichannel by analyzing these signals. Each driver has a comfortable way measurements. GPS: global positioning system. OBD: on-board diagnostic. 132


|

July 2017

|

in which he or she interacts with the vehicle, and analyzing these signals can help build a driver behavior and characteristic model.

Maneuver recognition and distraction analysis

errors and make valuable data from large naturalistic driving corpora more accessible. To prevent these errors from propagating, an automatic maneuver activity detection (MAD) tool (that also detects boundaries) using filter-bank analysis of vehicle dynamic signals is proposed. Using a minimal set of generic vehicle dynamic sensor information, such a MAD tool can match human transcription to an accuracy of up to 99% [24], [25]. Making this tool freely available will offer researchers opportunities to better explore naturalistic driving data.

The ability to continuously evaluate driving performance will be necessary in next-generation smart vehicles, to develop advanced driver-specific active/passive safety systems. One typical approach is to identify careless and risky driving events through analyzing The ability to continuously abrupt variations in vehicle dynamics inforevaluate driving mation. These variations are best captured Maneuver recognition performance will be when evaluated against similar driving patDriving maneuvers, influenced by the drivnecessary in nextterns or maneuvers. This has been predomier’s choice and traffic/road conditions, are nantly adopted in current-day active safety generation smart vehicles, important in understanding variations in systems [19]–[21]. These event detection driving performance and to help rebuild the to develop advanced systems provide an insight into the current intended route. Maneuvers are the basic driver-specific active/ driving conditions of the driver. In addition, units in building up a driving session. passive safety systems. every driver has his or her own unique style While processing massive quantities of of driving. Along with weather and traffic, naturalistic driving data, it is critical to anathe driver’s driving experience, vehicle handling ability, and lyze at a micro level. Understanding how these maneuvers are mental and physical state all influence the way a maneuver is performed can provide information on how the driver controls executed. Figure 2 depicts a system in which the driver is the vehicle and how driving performance varies over time, identified based on his or her driving characteristics; the drivwhich is essential in driver assistance and safety systems. er’s maneuvers are recognized, variations in them are identiSimilar to speech, where phonemes form words, it has been fied, and the driving is thus classified. The driver identification established [26], [27] that the smallest meaningful units of a subsystem reduces the variability for individual drivers, which driving pattern are termed drivemes. Drivemes form maneucan be achieved from face/speech recognition and other vers, and maneuver sequences form a navigation route. This inputs. Next, the driving performance is evaluated by identifyflow is depicted in Figure 3. Therefore, tracking the variation ing maneuvers and detecting their variations against regular of these drivemes can improve the efficiency of active safety (normal execution) patterns. Finally, every driving instance systems not only in providing safety to the driver, but also in (i.e., in terms of processing frames) is classified into neutral predicting drivers’ actions. (normal driving) or distracted driving. This section is focused The definition of driving maneuvers may be considerably on the maneuver recognition, variation detection, and driving wide, depending on the underlying application [28]. Sevclassification subsystems for the distraction analysis. eral existing studies have employed maneuver recognition With the pending availability of a massive free-style natufor vehicle trajectory prediction [29], intersection assistance ralistic driving data corpus (i.e., Strategic Highway Research [30], and lane-change intent recognition on the highway [31]. Program 2 and New Energy and Industrial Technology DevelBased on recent advancements, a study [32] that considered opment [22], [23]), the development of automatic tools to orgadriving maneuvers primarily classified into eight categonize, prune, and cluster drivercentric-based events for driver ries—straight, stop, left turn, right turn, left-lane change, modeling is a growing research topic. Rather than using simulated or fixed test track data, it is important to analyze Driving Performance on-road, real-traffic naturalistic driving Evaluation data for all possible driving variations in different maneuvers. Driver Maneuver Variation Driving Human transcription of these masIdentification Recognition Detection Classification sive corpora is not only a tedious task, but also subjective and prone to errors. Pruning the Search Space Driver These human transcription errors can Characteristics, Context and Distraction potentially hinder the development of Context, and Distraction algorithms for advanced safety systems Distraction and lead to performance degradations. Therefore, an automatic, effective, and computationally efficient tool is needed FIGURE 2. How the driver-dependent, maneuver-based distraction detection system identifies and to help mitigate human transcription evaluates each driving session. IEEE Signal Processing Magazine

|

July 2017

|

133

opportunities for low-cost, low-level maneuver recognition right-lane change, left road curve, and right road curve— for long-term modeling of driver behavior. showed promise. The method of recognition in the literature employed various statistical modeling and machine-learning classification algoDistraction analysis rithms, such as Bayesian models [33], finite-state machines Distraction, in general, affects the attention span of a person; and fuzzy logic [34], hidden Markov models (HMMs) [35], within the vehicular space, it manifests in the driver’s vehicuand decision trees [36]. HMMs have proven to be benefilar controls. Traditionally, distraction has been assessed from cial in predicting driver actions within the the driver’s perspective in terms of either first 2 s of an action sequence [37]. In our stress, eye movements, or cognitive workRather than using previous study, a similar HMM frameload [41]. Physiological measurements simulated or fixed test work was employed in both a top-down as such as heart rate variability and skin contrack data, it is important well as bottom-up approach to find the ductance (e.g., electroencephalogram, best integrated architecture for modeling electromyogram) have proven to be useful to analyze on-road, realdriving behavior and recognizing maneuin detecting the stress levels in drivers traffic naturalistic driving vers and routes [38]. Important features [42]. Studies have also considered body data for all possible include steering wheel angle, speed, and movement sensors to detect drivers’ patdriving variations in brake signals from vehicle CAN bus data, terns for assessment of driver distracdifferent maneuvers. or acceleration and gyroscope readings tions [43]. Though high accuracy has been from a smart portable device [25], [39]. achieve d from a research perspective, Recognition and prediction of lane-change maneuvers have these vision and body sensors are intrusive and unsuitable for been proposed together, suggesting a double-layered HMM naturalistic driving scenarios. Using such sensors can potenframework in the consideration of both maneuver execution tially serve as a baseline when compared with nonintrusive and route information [40]. Thus far, the accuracy of obtained sensors for performance. maneuver recognition ranges between 70–90% and offers Since driver actions and intentions manifest into vehicle movement, vehicle dynamic signals such as steering wheel movements, gas and brake pedal pressure, and vehicle speed Speech Route could potentially contain hidden or embedded information on Recognition Recognition the current status of the driver. Using vehicle dynamic signals, Sentences Routes driving is classified based on the maneuver execution characteristics of a particular driver. The classification could be a Phrases Segments binary classification (neutral versus distracted) [46] or a trend in the variations (safe, moderate, or risky) [32]. Words Maneuvers The assessment of driving distraction underlies two hypotheses. First, good, safe, or convenient driving behavior should Phonemes Drivemes be reflected by stable, steady vehicular dynamic performance. (a) (b) Second, the actions of an experienced driver should meet the characteristics of good driving behavior most of the time, FIGURE 3. A comparison of the structural flow of building blocks between with bad driving occurring as a limited number of events (a) speech recognition and (b) route recognition. [44]. Based on these hypotheses, the good driving events should be clustered in the vehicle dynamical feature space, whereas bad driving events will become more random anomFeature Space for Maneuver X alies or outliers. Figure 4 depicts a typical feature space for an imaginary Driving Behavior maneuver type, X. The green squares, which are clustered together around the centroid of class X, represent the normal Normal (Safe) execution trend for this maneuver. The deviations from the normal execution pattern are reflected in the feature space Moderate of this maneuver as yellow or red squares. These abnormal instances of the maneuver are still recognized as type X by the Risky classifier, but the intraclass separation suggests that they can be marked as outliers. Euclidean distance, cosine distance, and Outlier Mahalanobis distance have been used to detect outliers. Identifying such outliers helps in the evaluation of driving pattern variations and driving performance [45]. Figure 5 illustrates FIGURE 4. An example of the feature space for maneuver X showing the gradient of event variations (classified as safe, moderate, variations in driving performance quantified as normal, moderate, and and risky) along the driving route. risky maneuver actions. 134


|

July 2017

|

electronic devices, categorizing the main sources of distraction Due to the highly dynamic nature of driving and the surinto three categories: visual, cognitive, and manual. It will not rounding environment, drivers generally do not stay in one state be long before the automotive industry and infotainment sysfor long and often toggle between models/states. A microanaltems shift away from visual interaction with the driver and move ysis of individual driving patterns is performed by segmenttoward audio/speech-based interactions with the driver. Thereing the drive into small frames (a few seconds or a few meters fore, it is of great interest to understand the actual influence traveled), which can be scaled to a macro level for preventing of in-vehicle speech on the driver. There has been some preor correcting any unsafe activities. Such a microanalysis has liminary work done in this area to understand the influence provided an insight into how secondary tasks are executed and of in-vehicular speech and audio on drivpotentially influence drivers. Most secondary ing [48]. While some in-vehicle conversatasks can be grouped into three sequential Due to the highly dynamic tions might aid driving, categories such as events [46]. In the anticipation/preparatory nature of driving and the involved, competitive, and argumentative phase, during the start of a task, most drivers surrounding environment, speech can adversely influence the driver are distracted. This is justified as they divert and cause driver distraction. more attention toward the task, assess the drivers generally do not surroundings, and get ready to perform the stay in one state for long task. The second event is the task execution Tracking glance behavior and often toggle between phase, during which the drivers fall into a and visual attention models/states. comfort zone of multitasking. Finally, in the An important aspect in monitoring driver third task, the recovery or postcompletion distraction is to evaluate the visual attention phase, drivers generally reassess their surroundings after secof the driver. There are three main areas that can benefit from ondary task completion. The duration of each of these phases tracking the drivers’ visual attention: assessing the primary is based on the individual driver’s comfort and confidence level driving task, detecting secondary tasks, and supporting [47]; the effect of multitasking is variable on different drivers. advanced user–computer interfaces. As the automotive industry further advances in developing advanced driver-assistance systems (ADASs), such drivercenThe role of visual attention tric adaptive systems will help in personalizing the vehicle by Understanding where the visual focus lies is a key step to triggering the ADAS only when drivers are impacted or when determine driver performance during the primary driving task they show tendencies of such impact. [49]–[51]. A driver should scan the route environment before The National Highway Traffic Safety Administration (NHTSA) conducting a driving maneuver. This action includes checking released visual driver distraction guidelines [17] for in-vehicle the mirrors, looking at the vehicles in front of the driver, and

32.987

33.004

Safe Moderate Risky

33.002 33

32.986 32.985

32.998

32.994 32.992

32.984 32.983

Latitude

Latitude

32.996

32.982 32.981

Longitude (a)

–96.756

–96.758

32.978

–96.76

32.984

–96.762

32.979 –96.764

32.986 –96.766

32.98

–96.768

32.988

–96.74

–96.742

–96.744

–96.746

–96.748

–96.75

–96.752

–96.754

–96.756

–96.758

32.99

–96.754

Safe Moderate Risky

Longitude (b)

FIGURE 5. (a) and (b) This chart shows maneuver variations along two different driving routes. The routes were selected around the University of Texas at Dallas campus, with a mixture of residential and business areas. (a) and (b) The green, yellow, and red colors indicate driving safety levels along the two routes. IEEE Signal Processing Magazine

|

July 2017

|

135

Cognitive Perception Evaluation

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Visual Perception Evaluation Normal Radio Pictures

GPS Operating GPS Following Conversation

Phone Operating Phone Talking

FIGURE 6. An overview of the visual–cognitive space proposed to study driver distraction. Perceptual evaluations are used to determine the perceived level of distractions. Each secondary task affects the driver’s concentration differently.

identifying pedestrian actions. Primary driving tasks such as visual scanning, turning, and switching lanes all require mirror-checking actions [52]–[54]. Failing to accomplish these tasks decreases the drivers’ situational awareness, increasing the chances of accidents [55], [56]. An increase in visual demand due to secondary tasks affects the control of the vehicle, the detection of critical traffic events, and the detection of hazard events [57]. As a result, studies have used features describing eye-off-the-road, head pose, gaze range, and eyelid movements to detect distractions [58]–[65]. Objective measures capturing the duration and frequency of glance behaviors can provide important information to provide warnings to distracted drivers. Visual attention signals temporal deviations from the primary driving task to complete secondary tasks such as ad justing the radio, operating a cell phone, or looking at other passengers. All of these secondary tasks induce visual, cognitive, auditory, and manual distractions. A perceptual evaluation has been conducted to assess the perceived level of cognitive and visual distractions in 10-s videos of drivers who are engaged in different secondary tasks [65], [66], in which the advantages and limitation of using perceptual evaluations to assess driver distractions is discussed. Figure 6 shows that many common secondary tasks induce a high level of visual distractions. For example, operating a cell phone, the radio, or a navigation system increases the perceived level of visual distractions [67]. When a driver fails to glance at traffic, it can also signal cognitive distractions; the driver is looking but not seeing because he or she is daydreaming or thinking about something else [68], [69]. These types of distractions are very difficult to detect with noninvasive sensors [70]. Tracking 136

glance behaviors provides an important tool to address this problem. For all of these reasons, a robust ADAS should be able to detect mirror-checking actions and glance behaviors to prevent hazard situations [71]. The automobile industry is developing new advanced interfaces that do not induce manual or visual distractions. These interfaces are generally implemented using automatic speech-recognition systems. ADASs need to provide essential information to the driver in an effective manner. With more information available to the driver, it is also important that the information is presented without causing significant distractions. By tracking the visual attention of the driver and environment, the ADAS can clarify ambiguities by providing a situated dialog system (e.g., commands such as “What is the address of this building?” while glancing toward a specific building). In an example of such a system [72], the visual saliency of the scene and crowdsourced statistics on how people describe objects were used as prior information to improve the identification of points of interest (POIs). While the visual saliency of the scene did not depend on driver glance behaviors, we expect improved performance by modeling the visual attention of the drivers [73], [78].

Tracking visual attention Tracking eye movement can be an accurate measurement to identify the exact location of the gaze of the driver. However, robustly measuring gaze in a driving environment is challenging due to changes in illuminations in the vehicle and changes in the head poses of the drivers. As a result, most of the studies have approximated gaze with head poses. Zhang et al. [74] argued that even though eye gaze is a better indicator, head pose alone can provide good cues about driver intentions. However, there are differences between head pose and the driver’s gaze that need to be considered [75]–[78]. The driver moves his or her head and eyes to glance at a target object, where the eye–head relationship depends on factors such as the underlying driving task, the type of road, and the driver. Studies have investigated the relation between head motion and gaze on naturalistic recordings [78]. We placed multiple markers on the windshield, side windows, speedometer panel, radio, and gear. The recordings protocol is repeated while driving and when the car was parked. We proposed regression models where the dependent variables were the position and rotation of the head, and the independent variables were the three-dimensional positions of the POIs. While driving, the R2 of the model was about 0.73 for the horizontal direction, but lower than 0.20 for other directions. Motivated by these results, the analysis is extended to incorporate a probabilistic model relying on Gaussian process regression [79]. Instead of deriving the exact location of the POI, the framework creates a salient visual map describing the driver’s visual attention, which is mapped into the route scene (see Figure 7). The 95% confidence region of the models included about 89% of the POIs. This approach provides a suitable tool for situated dialog systems and safety systems that are aware of the driver glance behavior.


|

July 2017

|

An alternative approach to monitoring visual attention is to directly recognize primary or secondary driving tasks that require visual demand. An example of a primary driving task is the detection of mirror-checking actions. We presented an accurate random undersampling boost classifier to recognize mirror-checking actions [71]. The classifier was trained with multimodal features automatically extracted from the driver and road cameras and from the CAN bus signal using naturalistic recording on the UTDrive platform. The task was to recognize each time the driver looked at a given mirror. Figure 8(a) shows an example of a participant looking at the rear mirror. The F-score of the classifier was 91.4%, which is very high given that mirror-checking actions are infrequent events, making this classification problem highly unbalanced. Figure 8(b) shows the performance for different routes, under both normal conditions (during which the driver is not engaged in secondary tasks) and task conditions (during which the driver is engaged in secondary tasks). The classifier showed consistent performance across both normal and task conditions. An example of a secondary task is the detection of activities not related to the driving task requiring visual activities. We trained binary classifiers using a support vector machine, which detect particular secondary activities [60]. For tasks such as looking at pictures (which simulates the task of looking at billboards, sign boards, and shops) and operating a radio and a GPS, the accuracy was

about 80%. The perceptual evaluation showed that these tasks induce high visual demand.

Portable platform advancements With the rapid growth of smartphone capabilities, including entertainment and management of daily activities, individuals are increasingly using smartphones while driving. However, operating a vehicle is a complicated and skilled task requiring multimodal (especially visual) attention and focus. While drivers can multitask comfortably, using a smartphone may become a distraction and contribute to increased risk. Alternatively, the proper use of smart devices could be a source of reduced driving distraction while executing secondary tasks. One feasible approach to achieve this is to interact with these platforms using speech-based interfaces, reducing the visual and cognitive load [6]. Studies have also shown that drivers can achieve better and safer driving performance while using speech interactive systems to operate in-vehicle systems compared to hand-operated interfaces [80]. A more advanced reason for introducing the smartphone is its potential ability to be integrated with intelligent telematics services. Smartphone-based on-board sensing in the vehicle can capture various sources of information, including traffic (other vehicle and pedestrian movements), vehicle (diagnostics), environment (road and weather), and driver behavior in formation [81]. It would be beneficial to connect this platform

(a)

(a)

F-Score

1 0.8 0.6 0.4 0.2

G PS

R

ad io O pe ra G tin PS g Fo Ph llo on w in e g O pe r a Ph tin on g e Ta lk in g Pi ct ur C es on ve rs at io n

0

Normal

(b)

Secondary Task (b)

FIGURE 7. This visual saliency map was created with the probabilistic model and shows (a) the estimation of confidence regions for different distances from the car and (b) an aggregation of the results projected on the road camera. The saliency map characterizes driver visual attention.

FIGURE 8. The detection of mirror checking using multimodal features: (a) a participant looking at the rear mirror and (b) the F-score of the classifier for both normal and task conditions [71].


|

July 2017

|

137

and GPS within the device provide accurate estimates of vehicle dynamics. Mobile-UTDrive has been further developed to take advantage of capabilities such as speech recognition and on-screen map navigation. Figure 9 displays a screenshot of the Mobile-UTDrive app running on a tablet. Using this approach has resulted in studies to detect maneuvers and design driving safety systems that combine in-vehicle speech and video analysis with driving performance evaluation [86], [87]. Freely distributing this platform will offer researchers the opportunity to customize their data collection scenarios while maintaining current goals for naturalistic driving data advancements. In previous studies, it has been shown In-vehicle data collection platform: how vehicle dynamic signals can potentially Mobile-UTDrive app Freely distributing replace the information extracted from a Smartphones contain a variety of useful this platform will CAN bus and thereby extend the use of sensors, including cameras, microphones, offer researchers the maneuver recognition and monitoring algoinertial measurement units (IMU), and opportunity to customize rithms to any vehicle that uses the app [32], GPS. These multichannel signals make the [45]. However, the orientation and relative smartphone a potentially leveraged plattheir data collection movement of the smartphone inside the vehiform for in-vehicle data sensing and moniscenarios while cle yields the main challenge for platform toring, and can be employed for driving maintaining current goals deployment. A recent study [88] proposed a distraction analysis. The use of smart porfor naturalistic driving solution of converting the smartphone-refertable devices in vehicles creates the possidata advancements. enced IMU readings into vehicle-referenced bility to record useful data and helps accelerations, which allows free positioning develop a better understanding of driving of smartphones for the in-vehicle dynamics sensing. In this behavior. This option allows a wider range of naturalistic proposed framework, the raw smartphone IMU readings are driving study opportunities for drivers operating their own first processed through a geometry coordinate transformation vehicles [82]–[84]. to rotate/reorient the smartphone-referenced accelerations into The UTDrive mobile app (Mobile-UTDrive) has been a vehicle-referenced coordinate system. Next, a regression developed with the goal of improving driver/passenger safemodel is established to map the relationship between IMU ty while simultaneously maintaining the ability to establish and GPS data, and therefore provide an adaptive filtering promonitoring techniques that can be used on mobile devices in cess to decouple the smartphone’s relative movement in the various vehicles [85]. Mobile-UTDrive has been primarily vehicle. This serves as a preprocess module and therefore proused and developed as a multimodal data acquisition platform vides the basis for further applications using the smartphone that collects driver, vehicle, and environmental information data (see Figure 10). describing the comprehensive driving scenario. The modalities captured by Mobile-UTDrive are audio, video, GPS, and IMU sensor signals. The app runs on any Android-based smart Voice-based human–machine interaction portable device and uses the front and rear cameras to record The smartphone in the vehicle provides easy access to speech naturalistic driving video as well as in-vehicle audio. The IMU recording and processing, and offers potential integration with

with intelligent transportation systems or V2V or V2I communication, share the information, and realize a wider Internet of Vehicles. However, the challenges of smartphone platform use in the vehicle come from the deployment difficulty, measurement accuracy, and system reliability. In this section, we first discuss the deployment of smartphones as an in-vehicle data collection platform, utilizing its hardware resources. Additionally, we explore the implementation of the voice-based human–machine interface, and its capabilities in applications for vehicle/driver telematics.

(a)

(b)

FIGURE 9. The mobile-UTDrive app display showing (a) the original and (b) the updated version [85]. 138


|

July 2017

|

next generation of autonomous driving vehicles, it is expected the infotainment system, which becomes a good platform for that the vehicle will automatically drive for the human. Passenthe development of a voice-based human–machine interface. gers may inquire about the trip or change the previously selectDrivers would not necessarily be required to perform tasks such ed route through the dialog system, and the vehicle should be as setting map navigation, changing the radio station, adjusting able to understand how it is associated with navigation tasks volume, adjusting the air-conditioning, and controlling the winand provide the necessary responses. dows via hands-on operation, but could employ voice comRecent studies [89], [90] consider natural language processmands instead. The typical manual-entry or tactile-based ing (NLP) for the navigation-oriented human–vehicle dialog. engagement primarily uses various combinations of keypads, The NLP framework is based on a recurrent neural network keyboards, point-and-click techniques, touch-screen displays, and long short-term memory architecture, and contains senor other interface mechanisms. These traditional interfaces, tence-level sentiment analysis and word/phrase-level context which often require the driver to take his or her eyes off the extraction. As shown in Figure 11, the sentiment analysis idenroad, tend to be cumbersome in environments where the speed tifies whether a sentence is navigation related. If the sentence of interactions and dangers of distraction pose significant is navigation related, the next stage is to extract useful context issues, and therefore fall short in providing simple and intuitive by recognizing the word/phrase labels. The extracted informaoperation. In contrast, voice-based interaction would keep the tion will be ready to submit for response or path planning. The driver’s eyes focused on the road and his or her hands on the accuracy of NLP was 70–98%, depending on the accuracy of wheel. Results have shown that hands-on operation could the speech recognition results. potentially be a greater cause of major irregularities in driving performance, despite the latency and recognition error imposed by the speech recognition system [45], [87]. The development of speech recognition compatibility and natural language understanding Driving Assessment for dialog interaction in the car offers 1) Context Identification avenues for lowering driver distraction. 2) Outlier Detection Smartphone Therefore, natural voice-based engage3) Grade and Score Preprocessing Data ments between driver and vehicle offer 1) Noise Reduction the potential to meet an ever-growing IMU GPS 2) Coordinate demand for creating a comfortable, safe, Transformation Hands-Off Interaction and convenient driving experience. 3) Axis Alignment Among the voice-based human– 1) Experiment Protocol 2) Performance vehicle interfaces, the navigation diaEvaluation logue system is the one with the highest demand in recent years. Navigation dialogs may happen in some situations while a user may be driving, on-the- FIGURE 10. The smartphone data processing modules. go, or in other environments where having a hands-free interface provides NLP Tasks critical advantages. The desired intel• Step 1: Sentiment Analysis (Binary Classification) ligent navigation system is about more (a) “Every month, I eat some chocolate.” → False, Nonnavigation than searching locations on the map; (b) “Is there a Japanese restaurant around here?” → True, Navigation it would have the capability of acting • Step 2: Context Extraction (Word/Phrase Tagging) as an assistant, talk with humans in a s natural manner, and guide and drive for the human when needed. Therefore, it QUE POI ARE should have the ability to speak natuaround here is VBZ there EX a DT Japanese restaurant rally and understand natural spoken IN RB JJ NN language. For example, when a driver is trying to find a destination, he or she NN: Noun ARE: Area (Search Range) Phrase may either speak out a POI, specify the Word VB: Verb QUE: Question Labels Labels JJ : Adjective exact address, or spell the name and ACT: Action RB: Adverb ...... number of a street. The navigation sys...... tem should automatically understand what was said without having to ask the FIGURE 11. An example of the processing for NLP tasks. For the sentiment analysis of two sentences, driver to choose the style of his or her (a) is not navigation related, while (b) is navigation related. Therefore, (b) is further processed with spoken language. Furthermore, in the context extraction, and useful information (such as POI and search area) is labeled. IEEE Signal Processing Magazine

|

July 2017

|

139

Discussion and conclusions Vehicle technologies have advanced significantly in terms of improved transportation, comfort, and safety, and will continue to evolve as we move forward into the next generation of transportation systems and infrastructure. From this article, as well as the broad coverage from recent articles in the November 2016 issue of IEEE Signal Processing Magazine, it is clear that new technologies are migrating into novel invehicle systems for route navigation, information access, infotainment, and connected vehicle advancements for V2V and V2I connectivity and communications. Vehicle driving autonomy is also evolving, and further research advancements are needed to better understand the interplay between the driver, the vehicle, and the route/environment. With the motivation of contributing to improved intelligent driver–vehicle systems that incorporate human-specific characteristics, the CRSSUTDrive Lab has focused their research on naturalistic driving studies, with the interest of understanding driver behavior and distraction from multichannel sensor data. Any secondary driver task activity in the vehicle can be a source of driving distraction and therefore impact driving performance. Re garding this, one typical approach is to first extract the driving context in terms of microlevel components (e.g., maneuvers), and then evaluate risky events and variations against similar driving patterns in the vehicle dynamics domain. An alternative approach is to directly monitor drivers’ physical or glance behavior and assess their cognitive and visual attention. Previous studies have shown precise results in the detection of driving distraction, driving performance analysis, and visual attention tracking. To take advantage of the fast-growing smartphone applications market and integrate telematics services, recent activities have resulted in a mobile platform that contributes to in-vehicle naturalistic driving studies and voicebased human–machine interfaces. These studies, if combined, would be able to provide a comprehensive understanding of the driver’s state and driving performance, establish a comfortable driving experience with a humancentric assistant in the vehicle, and contribute intelligent transportation information sharing via V2V/V2I connectivity. While there is great interest in migrating to fully automated, self-driving vehicles, next-generation vehicles will need to be more active in assessing driver awareness, vehicle capabilities, traffic/environmental settings, and how these factors come together to determine a collaborative, safe, and effective driver–vehicle engagement for vehicle operation. Greater interdisciplinary research that addresses multifunctional vehicles to support smooth transitions from complete human control toward semisupervised/assisted control and even fully automated scenarios are needed. In the end, while signal processing technical advancements can enhance vehicles and the comfort, enjoyment, and capabilities of drivers, to be successful these efforts must first do no harm and ensure improved safety.

Authors John H.L. Hansen ([email protected]) received his B.S.E.E. degree from Rutgers University, New Jersey, and his 140

M.S. and Ph.D. degrees in electrical engineering from the Georgia Institute of Technology, respectively. He serves as associate dean for research at the University of Texas at Dallas, where he founded the Center for Robust Speech Systems–UTDrive Lab and the Robust Audio and Speech Processing Lab. He has served as a technical program chair for the IEEE International Conference on Acoustics, Speech, and Signal Processing 2010, the general chair for the International Speech Communication Association (ISCA) INTERSPEECH 2002, and as an organizer/contributor for the Biennial Workshop on Digital Signal Processing for In-Vehicle Systems and Safety (2003–2015), as well as the corresponding edited textbook series on driver modeling and safety. He is an ISCA fellow, IEEE Fellow, and an Acoustical Society of America’s 25-Year Award recipient. He has coauthored more than 650 papers in the fields of digital signal processing, speech processing, machine learning, and driver distraction modeling/detection. Carlos Busso ([email protected]) received his B.S. in electrical engineering from the University of Chile in 2000, his engineering degree in electrical engineering from the University of Chile in 2003, his M.S. degree in electrical engineering from the University of Chile in 2003, and his Ph.D. degree in electrical engineering from the University of Southern California in 2008. Currently, he is an associate professor at the University of Texas at Dallas, where he oversees the Multi modal Signal Processing Lab. His current research includes the broad areas of in-vehicle modeling of driver behavior, affective computing, and multimodal human–machine interfaces. He is a recipient of the National Science Foundation CAREER Award, the International Conference on Multimodal Interaction TenYear Technical Impact Award, and the Hewlett Packard Best Paper Award at the IEEE International Conference on Multimedia and Expo 2011 (with J. Jain). He is an associate editor of IEEE/ ACM Transactions on Audio, Speech, and Language. He is a Senior Member of the IEEE. Yang Zheng ([email protected]) received his B.S. and M.S. degrees in automotive engineering from Nanjing University of Aeronautics and Astronautics, China, and Tongji University, Shanghai, China, in 2010 and 2013, respectively. He is currently a Ph.D. student in electrical engineering at the University of Texas at Dallas, where he works in the Center for Robust Speech Systems–UTDrive Lab. His research interests include driver behavior analysis, human–machine interface, and intelligent vehicle advancements. He has authored/ coauthored three journal articles and 15 conference papers; he also holds two patents. He has served as a reviewer for the IEEE Intelligent Transportation Systems Society and the IEEE Vehicular Technology Society. He is a Student Member of the IEEE. Amardeep Sathyanarayana ([email protected]) received his B.E. degree from Visvesvaraya Technological University, India, in 2004 and his M.S. degree in electrical engineering and his Ph.D. degree in signal processing from the University of Texas at Dallas in 2008 and 2013, respectively. He is currently with Uhnder Inc., furthering next-generation


|

July 2017

|

active safety systems. Previously, he worked for Texas In struments’ Kilby Labs, AT&T Shannon Research Labs, Ford, and Bosch. His research interests include active safety systems and statistical signal processing. He has authored more than 30 book chapters, journal papers, and conference papers, and he has filed ten patents. He has served as a reviewer for various journals and conferences, including Journal of Selected Topics in Signal Processing and the IEEE International Conference on Intelligent Transportation Systems.

References

[1] J. F. Antin (2011). Design of the in-vehicle driving behavior and crash risk study: In support of the SHRP 2 naturalistic driving study. U.S. Transportation Res. Board. Washington, D.C. Rep. S2-S05-RR-1. [Online]. Available: http://apps .trb.org/cmsfeed/TRBNetProjectDisplay.asp?ProjectID=2127 [2] J. H. L. Hansen, K. Takeda, S. M. Naik, M. M. Trivedi, G. U. Schmidt, and Y. J. Chen, “Signal processing for smart vehicle technologies,” IEEE Signal Process. Mag., vol. 33, no. 6, pp. 12–13, Nov. 2016. [3] C. Miyajima and K. Takeda, “Driver-behavior modeling using on-road driving data: a new application for behavior signal processing,” IEEE Signal Process. Mag., vol. 33, no. 6, pp. 14–21, Nov. 2016. [4] Y. Choi, S. I. Han, S.-H. Kong, and H. Ko, “Driver status monitoring systems for smart vehicles using physiological sensors,” IEEE Signal Process. Mag., vol. 33, no. 6, pp. 22–34, Nov. 2016. [5] A. S. Aghaei, B. Donmez, C. C. Liu, D. He, G. Liu, K. N. Plataniotis, H.-Y. W. Chen, and Z. Sojoudi, “Smart driver monitoring: When signal processing meets human factors,” IEEE Signal Process. Mag., vol. 33, no. 6, pp. 35–48, Nov. 2016. [6] F. Weng, P. Angkititrakul, E. E. Shriberg, L. Heck, S. Peters, and J. H. L. Hansen, “Conversational in-vehicle dialog systems,” IEEE Signal Process. Mag., vol. 33, no. 6, pp. 49–60, Nov. 2016. [7] P. N. Samarasinghe, W. Zhang, and T. D. Abhayapala, “Recent advances in active noise control inside automobile cabins,” IEEE Signal Process. Mag., vol. 33, no. 6, pp. 61–73, Nov. 2016.

[21] D. A. Johnson and M.M. Trivedi, “Driving style recognition using a smartphone as a sensor platform,” in Proc. 14th Int. IEEE Conf. Intelligent Transportation Systems, Washington, D.C., 2011, pp. 1609–1615. [22] K. Takeda, J. H. L. Hansen, H. Erdogan, and H. Abut, In-Vehicle Corpus and Signal Processing for Driver Behavior. New York: Springer, 2009. [23] K. Takeda, J. H. L. Hansen, P. Boyraz, L. Malta, C. Miyajima, and H. Abut, “An international large-scale vehicle corpora for research on driver behavior on the road,” IEEE Trans. Intell. Transp. Syst., vol. 12, no. 4, pp. 1609–1623, Dec. 2011. [24] A. Sathyanarayana, S. O. Sadjadi, and J. H. L. Hansen, “Automatic maneuver boundary detection system for naturalistic driving massive corpora,” SAE Int. J. Passeng. Cars—Electron. Electr. Syst., vol. 7, no. 1, pp. 149–156, Apr. 2014. [25] A. Sathyanarayana, S. O. Sadjadi, and J. H. L. Hansen, “Leveraging sensor information from portable devices towards automatic driving maneuver recognition,” in Proc. 15th Int. IEEE Conf. Intelligent Transportation Systems, Anchorage, AK, 2012, pp. 660–665. [26] K. Torkkola, S. Venkatesan, and H. Liu, “Sensor sequence modeling for driving,” in Proc. 18th Int. Florida Artificial Intelligence Research Society Conf., Clearwater Beach, FL, 2005, pp. 721–727. [27] A. Sathyanarayana, P. Boyraz, and J. H. L. Hansen, “Driver behavior analysis and route recognition by hidden Markov models,” in Proc. IEEE Int. Conf. Vehicular Electronics Safety, Columbus, OH, 2008, pp. 276–281. [28] A. D’Agostino, A. Saidi, G. Scouarnec, and L. Chen, “Learning-based driving events classification,” in Proc. 16th Int. IEEE Annu. Conf. Intelligent Transportation Systems, 2013, pp. 1778–1783. [29] A. Houenou, P. Bonnifait, V. Cherfaoui, and W. Yao, “Vehicle trajectory prediction based on motion model and maneuver recognition,” in Proc. 2013 IEEE/RSJ Int. Conf. Intelligent Robots Systems, Tokyo, Japan, 2013, pp. 4363–4369. [30] Q. Tran and J. Firl, “Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression,” in Proc. 2014 IEEE Intelligent Vehicles Symp., Dearborn, MI, 2014, pp. 918–923. [31] J. Nilsson, J. Fredriksson, and E. Coelingh, “Rule-based highway maneuver intention recognition,” in Proc. 2015 IEEE 18th Int. Conf. Intelligent Transportation Systems, Las Palmas, Spain, 2015, pp. 950–955. [32] A. Sathyanarayana, O. Sadjadi, and J. H. L. Hansen, “Automatic driving maneuver recognition and analysis using cost effective portable devices,” SAE Int. J. Passeng. Cars—Electron. Electr. Syst., vol. 6, no. 2, 2013, pp. 467–477.

[8] R. Hult, G. R. Campos, E. Steinmetz, L. Hammarstrand, P. Falcone, and H. Wymeersch, “Coordination of cooperative autonomous vehicles,” IEEE Signal Process. Mag., vol. 33, no. 6, pp. 74–84, Nov. 2016.

[33] A. Gerdes, “Automatic maneuver recognition in the automobile: The fusion of uncertain sensor values using Bayesian models,” in Proc. 3rd Int. Workshop Intelligent Transportation (WIT 2006), Hamburg, DE, 2006, pp. 129–133.

[9] N. AbuAli and H. Abou-zeid (2016, Nov.). Driver behavior modeling: Developments and future directions. Int. J. Veh. Technol. [Online]. Available: http://dx.doi.org/10.1155/2016/6952791

[34] T. Hülnhagen, I. Dengler, A. Tamke, T. Dang, and G. Breuel, “Maneuver recognition using probabilistic finite-state machines and fuzzy logic,” in Proc. IEEE Intelligent Vehicles Symp. (IV), San Diego, CA, 2010, pp. 65–70.

[10] S. Schneegass, B. Pfleging, N. Broy, F. Heinrich, and A. Schmidt, “A data set of real world driving to assess driver workload,” in Proc. 5th Int. Conf. Automotive User Interfaces and Interactive Vehicular Applications, 2013, pp. 150–157.

[35] D. Mitrovic, “Reliable method for driving events recognition,” IEEE Trans. Intell. Transp. Syst., vol. 6, no. 2, pp. 198–205, June 2005.

[11] S. G. Klauer, T. A. Dingus, V. L. Neale, J. D. Sudweeks, and D. J. Ramsey, “The impact of driver inattention on near-crash/crash risk: an analysis using the 100car naturalistic driving study data,” Nat. Highway Traffic Safety Admin., Washington, DC, Rep. DOT HS 810 594, 2006. [12] UTDrive Corpus Classical. [Online]. Available: http://www.utdallas.edu/ research/utdrive/corpus.html [13] J. H. L. Hansen, P. Boyraz, K. Takeda, and H. Abut, Digital Signal Processing for In-Vehicle Systems and Safety. New York: Springer, 2012. [14] G. Schmidt, H. Abut, K. Takeda, and J. H. L. Hansen, Smart Mobile In-Vehicle Systems—Next Generation Advancements. New York: Springer Publishing, 2014. [15] M. A. Regan, C. Hallett, and C. P. Gordon, “Driver distraction and driver inattention: definition, relationship and taxonomy,” Accident Anal. Prev., vol. 43, no. 5, pp. 1771–1781, Sept. 2011. [16] M. Pettitt, G. Burnett, and A. Stevens (2005), “Defining driver distraction,” in Proc. 12th World Congr. Intelligent Transport Systems, San Francisco, CA, Nov. 2005, pp. 1–12. [17] Department of Transportation, “Visual-manual NHTSA driver distraction guidelines for in-vehicle electronic devices,” Nat. Highway Traffic Safety Admin., Washington, DC, Tech. Rep. Docket No. NHTSA-2010-0053, 2012. [18] K. Young, M. Regan, and M. Hammer, “Driver distraction: a review of the literature,” Accident Res. Centre, Monash Univ., Melbourne, Victoria, Tech. Rep. No. 206, 2003.

[36] Y. Zheng, A. Sathyanarayana, and J. H. L. Hansen, “Threshold based decisiontree for automatic driving maneuver recognition using CAN-Bus signal,” in Proc. IEEE Intelligent Transportation Systems Society Conf., Qingdao, China, 2014, pp. 2834–2839. [37] A. Pentland and A. Liu, “Modeling and prediction of human behavior,” Neural Comput., vol. 11, no. 1, pp. 229–242, 1999. [38] A. Sathyanarayana, “Multi-modal signal processing in-vehicular systems for driver distraction identification and driver behavior modeling,” Master’s thesis, Dept. Electrical Engineering, Univ. Texas at Dallas, 2008. [39] Y. Zheng, A. Sathyanarayana, and J. H. L. Hansen, “Non-uniform time window processing of in-vehicle signals for maneuver recognition and route recovery,” presented at the SAE World Congr., Detroit, MI, 2015. [40] Y. Zheng and J. H. L. Hansen, “Lane-change detection from steering signal using spectral segmentation and learning-based classification,” IEEE Trans. Intell. Transp. Syst., to be published. [41] T. B. Hughes, H.-S. Kim, J. H. DiBiase, and H. F. Silverman, “Performance of an HMM speech recognizer using a real-time tracking microphone array as input,” IEEE Trans. Speech Audio Process., vol. 7, no. 3, pp. 346–349, May 1999. [42] J. A. Healey and R. W. Picard, “Detecting stress during real-world driving tasks using physiological sensors,” IEEE Trans. Intell. Transp. Syst., vol. 6, no. 2, pp. 156–166, June 2005. [43] A. Sathyanarayana, S. Nageswaren, H. Ghasemzadeh, R. Jafari, and J. H. L. Hansen, “Body sensor networks for driver distraction identification,” in Proc. IEEE Int. Conf. Vehicular Electronics Safety, Columbus, OH, pp. 120–125, Sept. 2008.

[19] S. Boonmee and P. Tangamchit, “Portable reckless driving detection system,” in Proc. 6th Int. Conf. Electrical Engineering/Electronics Computer Telecommunications and Information Technology, Bangkok, Thailand, 2009, pp. 412–415.

[44] Y. Zheng and J. H. L. Hansen, “Unsupervised driving performance assessment using free-positioned smartphones in vehicles,” in Proc. IEEE Int. Conf. Intelligent Transportation Systems Conf., Rio de Janeiro, Brazil, 2016, pp. 1598–1603.

[20] J. Dai, J. Teng, X. Bai, Z. Shen, and D. Xuan, “Mobile phone based drunk driving detection,” in Proc. 4th Int. Conf. Pervasive Computing Technologies for Healthcare (PervasiveHealth), Munich, Germany, 2010, pp. 1–8.

[45] S. O. Sadjadi, A. Sathyanarayana, and J. H. L. Hansen, “Sensing variations in driving patterns using cost effective portable devices,” presented at the Future Active Safety Technology Towards Zero Traffic Accidents Conf., Nagoya, Japan, Sept. 2013.


|

July 2017

|

141

[46] A. Sathyanarayana, P. Boyraz, and J. H. L. Hansen, “Effects of multi-tasking on drivability through CAN-Bus analysis,” in Smart Mobile In-Vehicle Systems— Next Generation Advancements, G. Schmidt, H. Abut, K. Takeda, and J. Hansen, Eds. New York: Springer, 2014, pp. 169–182. [47] A. Sathyanarayana and J. H. L. Hansen, “Impact of secondary tasks on individual drivers: Not all drivers are created equally,” SAE Int. J. Passeng. Cars— Electron. Electr. Syst., vol. 5, no. 2, pp. 414–420, Sept. 2012 [48] N. Shokouhi, A. Sathyanarayana, O. Sadjadi, J. H. L. Hansen, “Overlappedspeech detection with applications to driver assessment for in-vehicle active safety systems,” in Proc. IEEE Int. Conf. Acoustics Speech Signal Processing, Vancouver, Canada, May 2013, pp. 2834–2838. [49] J.-C. Chien, J.-D. Lee, and L.-C. Liu, “A fuzzy rules-based driver assistance system,” Math. Probl. Eng., vol. 2015, pp. 1–14, Mar. 2015. [50] L. Fletcher and A. Zelinsky, “Driver inattention detection based on eye gazeroad event correlation,” Int. J. Robot Res., vol. 28, no. 6, pp. 774–801, June 2009. [51] Q. Ji and X. Yang, “Real-time eye, gaze, and face pose tracking for monitoring driver vigilance,” Real-Time Imaging, vol. 8, no. 5, pp. 357–377, Oct. 2002. [52] K. K. Ahlstrom and A. Kircher, “A gaze-based driver distraction warning system and its effect on visual behavior,” IEEE Trans. Intell. Transp. Syst., vol. 14, no. 2, pp. 965–973, June 2013. [53] A. Georgeon, M. J. Henning, T. Bellet, and A. Mille, “Creating cognitive models from activity analysis: a knowledge engineering approach to car driver modeling,” in Proc. Int. Conf. Cognitive Modeling (ICCM 2007), Ann Arbor, MI, July 2007, p. 26. [54] E. C. B. Olsen, “Modeling slow lead vehicle lane changing,” Ph.D. dissertation, Industrial and Systems Engineering, Virginia Polytechnic Inst. State Univ., Blacksburg, VA, 2003. [55] K. A. Brookhuis and D. de Waard, “Assessment of drivers’ workload: Performance and subjective and physiological indices,” in Stress, Workload, and Fatigue: Human Factors in Transportation, P. Hancock and P. Desmond, Eds. Mahwah, NJ: Lawrence Erlbaum Associates, 2000, pp. 321–333. [56] M. Recarte and L. Nunes, “Mental workload while driving: Effects on visual search, discrimination, and decision making,” J. Exp. Psychol. Appl., vol. 9, no. 2, pp. 119–137, June 2003. [57] Y. Liang and J. D. Lee, “Combining cognitive and visual distraction: Less than the sum of its parts,” Accident Anal. Prev., vol. 42, no. 3, pp. 881–890, May 2010. [58] A. Azman, Q. Meng, and E. Edirisinghe, “Non-intrusive physiological measurement for driver cognitive distraction detection: Eye and mouth movements,” in Proc. Int. Conf. Advanced Computer Theory Engineering (ICACTE 2010), Chengdu, China, Aug. 2010, p. V3-595. [59] M. Kutila, M. Jokela, G. Markkula, and M. R. Rue, “Driver distraction detection with a camera vision system,” in Proc. IEEE Int. Conf. Image Processing (ICIP 2007), San Antonio, TX, Sept. 2007, pp. 201–204. [60] N. Li and C. Busso, “Predicting perceived visual and cognitive distractions of drivers with multimodal features,” IEEE Trans. Intell. Transp. Syst., vol. 16, no. 1, pp. 51–65, Feb. 2015. [61] Y. Liang, M. L. Reyes, and J. D. Lee, “Real-time detection of driver cognitive distraction using support vector machines,” IEEE Trans. Intell. Transp. Syst., vol. 8, no. 2, pp. 340–350, June 2007. [62] M. C. Su, C. Y. Hsiung, and D. Y. Huang, “A simple approach to implementing a system for monitoring driver inattention,” in Proc. IEEE Int. Conf. Systems Man and Cybernetics (SMC-2006), Taipei, Taiwan, Oct. 2006, pp. 429–433. [63] F. Tango and M. Botta, “Real-time detection system of driver distraction using machine learning,” IEEE Trans. Intell. Transp. Syst., vol. 14, no. 2, pp. 894–905, June 2013. [64] Q. Wu. “An overview of driving distraction measure methods,” in Proc. IEEE 10th Int. Conf. Computer-Aided Industrial Design and Conceptual Design (CAID CD 2009), Wenzhou, China, Nov. 2009, pp. 2391–2394. [65] N. Li and C. Busso. “Analysis of facial features of drivers under cognitive and visual distractions,” in IEEE Int. Conf. Multimedia and Expo (ICME 2013), San Jose, CA, July 2013, pp. 1–6. [66] N. Li and C. Busso, “Using perceptual evaluation to quantify cognitive and visual driver distractions,” in Smart Mobile In-Vehicle Systems—Next Generation Advancements, G. Schmidt, H. Abut, K. Takeda, and J. H. L. Hansen, Eds. New York: Springer, 2014, pp. 183–207. [67] N. Li, J. J. Jain, and C. Busso, “Modeling of driver behavior in real world scenarios using multiple noninvasive sensors,” IEEE Trans. Multimedia, vol. 15, no. 5, pp. 1213–1225, Aug. 2013. [68] J. Harbluk, Y. Noy, P. Trbovich, and M. Eizenman, “An on-road assessment of cognitive distraction: Impacts on drivers’ visual behavior and braking performance,” Accident Anal. Prev., vol. 39, no. 2, pp. 372–379, Mar. 2007. [69] J. L. Harbluk, M. Eisenman, and Y. I. Noy, “The impact of cognitive distraction on driver visual behaviour and vehicle control,” Transport Canada, Ottawa, Ontario, Tech. Rep. TP# 13889 E, 2002.

142

[70] M. H. Kutila, M. Jokela, T. Makinen, J. Viitanen, G. Markkula, and T. W. Victor, “Driver cognitive distraction detection: Feature estimation and implementation,” P. I. Mech. Eng. D-J Aut., vol. 221, no. 9, pp. 1027–1040, Sept. 2007. [71] N. Li and C. Busso, “Detecting drivers’ mirror-checking actions and its application to maneuver and secondary task recognition,” IEEE Trans. Intell. Transp. Syst., vol. 17, no. 4, pp. 980–992, Apr. 2016. [72] T. Misu, “Visual saliency and crowdsourcing-based priors for an in-car situated dialog system,” in Proc. Int. Conf. Multimodal Interaction (ICMI 2015), Seattle, WA, Nov. 2015, pp. 75–82. [73] S. Jha and C. Busso, “Analysis of head pose as an indicator of drivers’ visual attention,” in Proc. 7th Biennial Workshop Digital Signal Processing for In-Vehicle Systems and Safety, Berkeley, CA, Oct. 2015, p. 15. [74] H. Zhang, M. Smith, and R. Dufour. (2008). A final report of safety vehicles using adaptive interface technology (Phase II: Task 7C): Visual distraction. Delphi Electronics and Safety. Kokomo, IN. [Online]. Available: https:// www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja& uact=8&ved=0ahUKEwiRrp7li_PTAhVB5iYKHTMRCLUQFggiMAA&url= h t t p s % 3 A % 2 F % 2 F w w w .v o l p e . d o t . g o v % 2 F s i t e s % 2 F v o l p e . d o t .gov%2Ffiles%2Fdocs%2FSAVE-IT%2520-%2520Visual%2520Distraction. doc&usg=AFQjCNEDi2l1SmiLCsSCqAOirWn8hOuWrw [75] A. Tawari and M. M. Trivedi, “Robust and continuous estimation of driver gaze zone by dynamic analysis of multiple face videos,” in Proc. 2014 IEEE Intelligent Vehicles Symp., Dearborn, MI, June 2014, pp. 344–349. [76] M.-C. Chuang, R. Bala, E. Bernal, P. Paul, A. Burry, “Estimating gaze direction of vehicle drivers using a smartphone camera,” in Proc. 2014 IEEE Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Columbus, OH, 2014, pp. 165–170. [77] W. H. Zangemeister and L. Stark, “Types of gaze movement: Variable interactions of eye and head movements,” Experimental Neurology, vol. 77, no. 3, pp. 563–577, Sept. 1982. [78] S. Jha and C. Busso, “Analyzing the relationship between head pose and gaze to model driver visual attention,” in Proc. IEEE Int. Conf. Intelligent Transportation Systems (ITSC 2016), Rio de Janeiro, Brazil, Nov. 2016, pp. 2157–2162. [79] S. Jha and C. Busso, “Probabilistic estimation of the driver’s gaze from head orientation and position,” submitted for publication. [80] C. Carter and R. Graham, “Experimental comparison of manual and voice controls for the operation of in-vehicle systems,” Ergonomics New Millennium, vol. 44, no. 20, pp. 283–286, July 2000. [81] J. Engelbrecht, M. J. Booysen, G. J. van Rooyen, and F. J. Bruwer, “Survey of smartphone-based sensing in vehicles for intelligent transportation system applications,” IET Intell. Transp. Syst., vol. 9, no. 10, pp. 924–935, Nov. 2015. [82] H. Eren, S. Makinist, E. Akin, and A. Yilmaz, “Estimating driving behavior by a smartphone,” in Proc. IEEE Intelligent Vehicles Symp. (IV), Alcala de Henares, Spain, June 2012, pp. 234–239. [83] M. V. Ly, S. Martin, and M. M. Trivedi, “Driver classification and driving style recognition using inertial sensors,” in Proc. IEEE Intelligent Vehicles Symp. (IV), Gold Coast, Australia, June 2013, pp. 1040–1045. [84] P. Phondeenana, N. Noomwong, S. Chantranuwathana, and R. Thitipatanapong, “Driving maneuver detection system based on GPS data,” presented at the Future Active Safety Technology Towards Zero Traffic Accidents Conf., Nagoya, Japan, Sept. 2013. [85] Y. Zheng and J. H. L. Hansen, “Mobile-UTDrive: An android portable device platform for in-vehicle driving data collection and display,” presented at the Future Active Safety Technology Towards Zero Traffic Accidents Conf., Gothenburg, Sweden, Sept. 2015. [86] Y. Zheng, X. Shi, A. Sathyanarayana, N. Shokouhi, and J. H. L. Hansen, “In-vehicle speech recognition and tutorial keywords spotting for novice drivers’ performance evaluation,” in Proc. IEEE Intelligent Vehicles Symp. (IV), June–July 2015, pp. 168–173. [87] Y. Zheng, N. Shokouhi, A. Sathyanarayana, N. B. Thomsen, and J. H. L. Hansen, “Towards developing a distraction-reduced hands-off interactive driving experience using portable smart devices,” presented at the SAE World Congr., Detroit, MI, Apr. 2016. [88] Y. Zheng and J. H. L. Hansen, “Free-positioned smartphone sensing for vehicle dynamics estimation,” presented at the SAE World Congr., Detroit, MI, April 2017. [89] Y. Zheng, Y. Liu, and J. H. L. Hansen, “Navigation-orientated natural spoken language understanding for intelligent vehicle dialogue,” in Proc. IEEE Intelligent Vehicles Symp. (IV), Redondo Beach, CA, June 2017. [90] Y. Zheng, Y. Liu, and J. H. L. Hansen, “Intent detection and context extraction for navigation dialogue language processing,” submitted for publication. [91] Transportation Research Board. [Online]. Available: http://www.trb.org/ StrategicHighwayResearchProgram2SHRP2/Blank2.aspx


SP

|

July 2017

|