Brain Injury, March 2005; 19(3): 177–188
ORIGINAL ARTICLE
Predictive validity of driving-simulator assessments following traumatic brain injury: a preliminary study
HENRY L. LEW1–3, JOHN H. POOLE1–3, EUN HA LEE1–3, DAVID L. JAFFE4, HSIU-CHEN HUANG1–3 & EDWARD BRODD1 1
Physical Medicine & Rehabilitation Service, VA Palo Alto Health Care System, Palo Alto, CA 94304 USA, Defense and Veterans Brain Injury Center, Palo Alto, CA, USA, 3Division of Physical Medicine & Rehabilitation, Stanford University School of Medicine, Stanford, CA, USA, 4Rehabilitation Research and Development Center, VA Palo Alto Health Care System, Palo Alto, CA, USA 2
Abstract Objective: To evaluate whether driving simulator and road test evaluations can predict long-term driving performance, we conducted a prospective study on 11 patients with moderate to severe traumatic brain injury. Sixteen healthy subjects were also tested to provide normative values on the simulator at baseline. Method: At their initial evaluation (time-1), subjects’ driving skills were measured during a 30-minute simulator trial using an automated 12-measure Simulator Performance Index (SPI), while a trained observer also rated their performance using a Driving Performance Inventory (DPI). In addition, patients were evaluated on the road by a certified driving evaluator. Ten months later (time-2), family members observed patients driving for at least 3 hours over 4 weeks and rated their driving performance using the DPI. Results: At time-1, patients were significantly impaired on automated SPI measures of driving skill, including: speed and steering control, accidents, and vigilance to a divided-attention task. These simulator indices significantly predicted the following aspects of observed driving performance at time-2: handling of automobile controls, regulation of vehicle speed and direction, higher-order judgment and self-control, as well as a trend-level association with car accidents. Automated measures of simulator skill (SPI) were more sensitive and accurate than observational measures of simulator skill (DPI) in predicting actual driving performance. To our surprise, the road test results at time-1 showed no significant relation to driving performance at time-2. Conclusion: Simulator-based assessment of patients with brain injuries can provide ecologically valid measures that, in some cases, may be more sensitive than a traditional road test as predictors of long-term driving performance in the community. Keywords: Closed head injuries, cognition, computer simulation, driving behavior, ecological validity, predictive value of tests, risk assessment, safety standards, virtual reality systems
Introduction For most individuals, the ability to drive is an essential component for independent living. While patients with traumatic brain injury (TBI) have various degrees of cognitive impairments that interfere with their driving performance [1, 2], 40–80% of them eventually resume driving [3–6], often irrespective of professional recommendations. Thus, it is very important to assess patients’ readiness for driving. The present study was conducted to provide preliminary information on the predictive validity of
driving simulator evaluations for patients with TBI. Over the last 35 years, studies on the utility of driving simulators have progressed through several levels of validation. At the simplest level, numerous studies have examined whether simulator performance is sensitive to differences between various groups of drivers (discriminant validity). Such studies have shown that simulators readily detect and characterize important differences between healthy drivers and those with acquired central nervous system disorders [7–11], specific visual deficits [12, 13], advancing age [7, 8, 14, 15], different
Correspondence: John H Poole, Physical Medicine & Rehabilitation Service (140/117), VA Palo Alto Health Care System, 3801 Miranda Avenue, Palo Alto, CA, 94304, USA. E-mail:
[email protected], Tel: 650-493-5000, ext. 65504 ISSN 0269–9052 print/ISSN 1362–301X online # 2005 Taylor & Francis Ltd DOI: 10.1080/02699050400017171
178
H. L. Lew et al.
amounts of prior driving experience [16], or sleep deprivation [17]. Another phase of validation has been to test whether simulator-based assessments agree with the most common method of driver evaluation, a road test conducted by a trained evaluator (convergent validity). The results of such studies have been less consistent. Seven studies examined the relationship between simulator and on-road performance of healthy individuals [16, 18–23]. Correlations between simulator and on-road evaluations were moderate to high in five of these studies and low in two studies. Likewise, six other studies have examined patients with acquired brain injuries and compared their simulator skills to the results of a concurrent road test. Five found moderate to high agreement between simulator and on-road evaluations [9, 24–27]. The sixth study found that reaction time on a driving simulator had little relation to on-road skills; however, no automated measures of simulator performance other than reaction time were examined [28]. Almost all published studies have either focused on simulator performance alone or compared simulator performance to a relatively brief road test. A much more demanding level of validation is to test whether simulator skills are related to actual driving performance over an extended period of time in the community (ecological validity). We have found only three such studies. One examined the relation of simulator and in-car evaluations to the prior 5-year history of citations and accidents of 304 taxi drivers; significant correlations among these measures were few in number and small in magnitude [18]. Another retrospective study found no relation between the simulator performance and prior accident history of 107 subjects with or without visual deficits [29]. The only prospective study examined 19 patients with neuropsychological deficits and found no correlation between their simulator performance and subsequent traffic citations or accidents [27]. However, lanetracking was the only simulator measure used in that study. One methodological limitation of the three studies described above is that the only criteria of driver safety were accidents and traffic citations, which showed only small differences between the groups of subjects being compared. Although accidents and citations are important to consider, they may be inadequate alone as criteria of driver safety, for at least four reasons: (1) Not all unsafe behaviours result in accidents or citations during the time frame evaluated. (2) Not all violations and accidents are observed or reported. (3) External factors that are known to impact accident and citation rates (city of residence, regional climatic differences, local law enforcement practices, etc.) may partly overshadow individual differences in driving skill.
(4) In principle, rare critical events are less sensitive measures than are dimensional rating scales of performance measured during the same time period. No prior validation study has used dimensional ratings of driving performance in addition to critical events as measures of driving performance in the community. While the reviewed studies indicate that simulators can measure and describe current driving skills, this conclusion may be insufficient to justify the additional time and expense required for driving simulation. The critical question is whether simulators can predict who will drive safely or unsafely in the community, prior to their taking the potentially risky step of getting behind the wheel of a car. Thus, the most stringent level of validation extends the question of ecological validity into the future by testing whether simulator-based assessments can predict future driving performance in the community (predictive validity). Such prospective studies are difficult to complete, not only because of the time and effort that follow-up studies require, but also because of ethical constraints against allowing unsafe drivers to continue driving simply to see if they have undesirable outcomes, such as accidents [30]. Predicting driver safety becomes an even more daunting task, if the drivers’ skills are undergoing dynamic changes, as is the case for patients recovering from TBI. The present pilot study was conducted to provide a preliminary test of whether simulator-based assessment can predict patients’ driving performance in the community over an extended period of time, in a manner that complements the traditional road test. To maintain ethical standards, patients’ simulator evaluations were always conducted in parallel with our centre’s standard driver evaluation, which passed or failed drivers and provided recommendations independent of this study. To evaluate longterm outcome, we not only counted critical events such as accidents and violations, but also obtained observational ratings of driving performance over at least a one-month period, to provide potentially more sensitive measures of driver safety.
Methods
Subjects Eleven patients with TBI were recruited from referrals to the driver evaluation program at our local Health Care System. Inclusion criteria for patients were: (1) history of moderate to severe TBI within the past two years, (2) previous driving experience with a valid licence, (3) muscle power in all four extremities of at least 4/5 on neurological screening, and (4) corrected vision >20/40 Snellen, without visual agnosia. Since the goal of this study
Validity of driving-simulator assessments following TBI was to predict differences in driver competency, we selected subjects at various stages of recovery from TBI and across a wide age range, so that diverse driving skills were represented. The time post-TBI ranged from 2 to 25 months (mean SD ¼ 8 2). Sixteen healthy comparison subjects, similar in age to the TBI group, were recruited from hospital staff, subjects’ acquaintances, and contacts in the community. Except for the first criterion, the same inclusion criteria were applied to the control group. The patients’ age ranged from 18 to 58 years (29 12), and the healthy group’s age ranged from 22 to 58 years (36 11). Males constituted 82% of the TBI group and 75% of the healthy group. The age distribution and gender-ratio of the groups did not differ significantly ( p > 0.10, by t-test and chi-squared test respectively). Healthy control subjects only completed the simulator portion of the study, to provide reference values for performance in the programmed driving scenarios. Our local Institutional Review Board approved this protocol, and informed consent was obtained from each participant.
Procedure This study was conducted in two phases (see Figure 1), which are described in detail in the following sections. At time-1 of data collection, patients with TBI completed the baseline evaluation on the driving simulator as well as the in-car road test. The simulator automatically measured 12 driving parameters in a Simulator Performance Index (SPI, described below), which were the primary predictor variables used in this study. Also, an observational Driver Performance Inventory (DPI, described below) was used to rate driving skills both on the simulator and during the road test.
Approximately 9 months later, we telephoned collateral observers who traveled most frequently with each patient, to instruct them regarding their in-car assessment of patients’ driving skills. A month later, at time-2 of data collection, we again called these observers to conduct the observational DPI, which was the primary outcome measure of this study. At this time, the informants and patients also provided data on any driving infractions by the patient. To avoid biasing the outcome of this predictive study, the observers at time-2 completed their driver ratings without knowledge of subjects’ automated or observational simulator measures at time-1. Likewise the two observers at time-1 completed their evaluations without knowledge of each other’s ratings. Healthy comparison subjects completed only the driving simulator evaluation at time-1, to provide normative data on the SPI and DPI.
Driving simulator This study used the Systems Technology Incorporated (STIÕ version 8.16) interactive driving simulator, which consists of a PC with 21-inch (53 cm) color monitor, two speakers, table-mounted steering wheel, accelerator and brake pedals. The STI software provides a series of programmable visual scenes and sounds (engine acceleration and braking sounds). Three road courses were programmed at increasing levels of difficulty: (1) a low-speed, 2-lane road within the hospital grounds, (2) a medium-speed 2-lane road in a residential area, and (3) a higher-speed 4-lane expressway. The first and simplest simulated course resembles the 2-lane perimeter road within the medical centre, which includes stop signs, occasional cars approaching from the opposite direction, and pedestrian crossings. It is 10 000 feet in length (3.0 km), has
Time-1
Time-2
(baseline)
(month 10)
Simulator Evaluation – TBI group and normative group · Automated measures (SPI) · Observational data (DPI)
179
Driver Ratings completed – TBI group · Observational data (DPI) · Report of infractions
Road Test – TBI group · Observational data (DPI) Analyses: · Reliability of SPI and DPI (internal consistency) · Simulator scores of TBI group compared to normative values
Analyses: · Correlate the above two outcome measures with all three measures at Time-1 (simulator and road test).
Figure 1. Timeline of study. SPI ¼ Simulator Performance Index. DPI ¼ Driver Performance Inventory. See Methods section for description of all measures.
180
H. L. Lew et al.
a speed limit of 15 mph (24 kph), and requires approximately 10 minutes to complete. The second, intermediate course was designed to resemble the 2-lane road in a nearby residential area, which includes more cars, pedestrians, and stop signs, with traffic in the driver’s lane as well as oncoming traffic. It is 16 000 feet long (4.9 km), has a speed limit of 25 mph (40 kph), and requires approximately 10 minutes to complete. The third, most difficult simulator course resembles the driving environment of a nearby 4-lane expressway, with high-density traffic, sharper curves, and traffic signals replacing the stop signs of previous courses. Subjects were told that they must overtake slower automobiles, which requires judgments regarding oncoming traffic and careful maneuvers around other vehicles. This course is 31 000 feet in length (9.4 km), with speed limits of 45–55 mph (72–89 kph), and requires approximately 10 minutes to complete. Across all three courses, the total time on the simulator is 30–35 minutes. A divided-attention task was integrated into the simulator program for each road course. Subjects were instructed to watch for a signal in the left or right corners of the simulated rear-view mirror and to immediately press the horn button whenever this signal appeared. There were a total of eleven signals during the simulator trial, each signal having a fixed stimulus duration of 5 seconds, with variable inter-stimulus intervals of approximately 1.5 to 3 minutes. Before starting the simulator evaluation, all subjects were given a 5-minute practice session to become familiar with the operational procedures and feel of the device. Subjects were given a brief description of each road scenario before it began. They were then given instructions to follow road regulations, maintain safe speed, watch for pedestrians and other drivers, pass slow vehicles, and respond to the divided-attention signal. All subjects completed all three simulated roadways. Twelve automated performance measures were collected from the simulator in two categories: speed control and direction control. The Speed-Control measures were: speed (percent of time exceeding the posted limit), speed variability (SD), acceleration variability (SD), speed jerk (SD of throttle speed), and red-light violations (N). The Direction-Control measures were: lane position error on straight roads (mean), lane position variability on straight roads (SD), lane position error on curved roads (mean), lane position variability on curved roads (SD), steering jerk (SD of steering wheel speed), collisions (N), and deviations off-road (N). A composite index of driving performance on the simulator was computed from the mean z-scores of all 12 items. This Simulator Performance Index (SPI) showed very high internal-consistency reliability in assessing
subjects’ overall skill on the simulator (Cronbach’s alpha ¼ 0.9). In addition, separate Speed-Control and Direction-Control indices were computed, consisting of the mean z-score of items in each category (alpha ¼ 0.9 and 0.8, respectively). To define a failing grade on the simulator, we used an SPI cutoff of Z < 2.0 (two standard deviations below the mean of the reference group).
Driver ratings Subjects’ driving performance was evaluated with the Driving Performance Inventory (DPI, see Appendix), developed for the driver safety program at our facility. This instrument uses a 3-point scale (2 ¼ Good, safe; 1 ¼ Fair, needs improvement; 0 ¼ Poor, unsafe) to rate 14 items in four broad domains: Handling of Controls (steering wheel control, throttle-brake coordination); Regulation of Trajectory (speed, lane tracking, brake reaction time); Basic Maneuvers (lane changes, execution of turns, merging into traffic, obedience to traffic signs and signals, following distance); Higher-order Skills (safety judgments when passing and yielding right of way, speed and correctness of decisions, emotional stability/self-control); with a final item rating the need for intervention by the observer/examiner. The DPI was completed by three different observers in the following contexts: (1) by a trained research assistant who observed and rated all simulator trials at time-1 (HCH), (2) by the driving program manager who conducted the concurrent road test on each TBI subject at time-1 (EB), and (3) by a parent or sibling of each TBI subject who provided at least 3 hours of in-car observation during the follow-up period (at time-2). We used the mean of all DPI items as an index of overall driving skill (possible range ¼ 0 to 2). The DPI demonstrated high internal-consistency reliability among the items composing the scale (Cronbach’s alpha ¼ 0.9). Subscale scores were computed for Handling of Controls, Regulation of Trajectory, Basic Maneuvers, and Higher-order Skills, consisting of the mean of items in each category (‘observer intervention’ was included in all four subscales, assigning 1/4 of its weight to each). Reliability of three DPI subscales was moderate to high: Regulation of Trajectory (alpha ¼ 0.7), Basic Maneuvers (0.8), and Higher-order Skills (0.9). Reliability for the Handling of Controls subscale was lower (alpha ¼ 0.5), but acceptable for the purpose of this preliminary study.
Road test At time-1, patients were given a standard road test of approximately 1 hour, conducted by our facility’s
Validity of driving-simulator assessments following TBI driving program manager (EB) in a vehicle equipped with passenger-side brakes for emergency control. The order of the road test and simulator evaluations was randomized. The road test was conducted in three contexts that corresponded to the three simulator courses. The first course, within the hospital facility on a 15 mph (24 kph) two-lane perimeter roadway with minimal traffic, provided a basic evaluation of the patient’s ability to maneuver a vehicle. The second course consisted of a residential area adjacent to the facility, with a 25 mph (40 kph) speed limit and higher vehicular and pedestrian traffic. The third course was on a nearby four-lane expressway with speed limits of 45–55 mph (72– 88 kph). Subjects were given a failing grade on the road test if their overall score was too low on any course (mean DPI < 1.5) or if, in the evaluator’s judgment, they displayed even a single incident of reckless driving (any DPI item ¼ 0). Subjects were only permitted to advance to the next road course if they passed the preceding one. Six subjects passed the road test and four failed the test. One other subject completed the simulator evaluation but was judged unsafe to drive, based on clinical evaluations. This subject was not permitted to take the road test and was assigned a failing grade on the road test.
Follow-up evaluation Upon entry into this study, patients were asked to select a family member to serve as a collateral informant. Eight to 10 months after each subject completed the simulator and road test, their collateral informant was contacted by telephone and oriented to the driving evaluation. Approximately one month later (10 months 1 month after the simulator trial and road test), the collateral informants were again contacted to complete a structured interview focusing on the subject’s driving performance (time-2 evaluation). Each informant had accompanied and observed subjects as they drove for a minimum of 3 hours, spread over at least 1 month (median total observation ¼ 4 hours). During the interview, the 14 items of the DPI were described to the informant, who was asked to rate subjects on the same three-point scale that was used during the simulator and road test evaluations at time-1. An overall pass/fail grade was assigned using the same criteria as the road test at time-1 (fail if mean DPI < 1.5 or any DPI item ¼ 0). In addition, subjects and informants were asked to report any citations, accidents, or licence revocations that the subject had incurred since time-1. Prior to the interview, all informants were reminded that their information was solely for research purposes and would not be relayed to authorities such as the Department of Motor Vehicles. All informants
181
affirmed that they understood this and provided detail-rich examples of their family member’s positive and negative driving habits. Despite the fact that some patients did not pass the road test at time-1, were advised not to drive, and in some cases had their licences revoked, we discovered that all 11 patients chose to drive after their hospital discharge. For patients with revoked licences, this typically consisted of drives in low-traffic conditions, accompanied by family members or friends, to practice driving before seeking re-instatement of their licences.
Data analysis As commonly occurs on measures with a minimum possible score (zero), several performance parameters and observational measures were positively skewed in the TBI and control groups (skewness > 1.0). Prior to all analyses, these were normalized using logarithmic or inverse transformations, allowing parametric statistical tests. On this basis, we report subjects’ simulator scores as normalized z-scores, using the mean and standard deviation of the healthy group as reference values. Where meaningful, we also report the median and inter-quartile range of untransformed variables as descriptive statistics. For the correlations among driving assessments, we considered one-tailed significance tests the most appropriate estimates of type-II error. This is because only a single alternative to the null hypothesis is rational, i.e. that poorer assessed driving skills are associated with worse, not better driving performance—as has been verified in the prior literature. To control type-I error, our primary statistical tests were conducted using three potential predictor variables (the mean Simulator Performance Index, and the mean Driving Performance Inventory index for the simulator and road test). Post hoc subscale and item-wise comparisons were only considered if these global indices attained significance ( p < 0.05). The significance of subscale and item-wise tests were adjusted (¼padj) using Hochberg’s step-up correction for multiple significance tests [31]. However, to allow consideration of possible type-II errors in this small initial study, we also note any trends that approach significance ( p < 0.10, unadjusted). All data analyses were conducted utilizing SPSS version 10.1. Results
Preliminary analysis: comparison of TBI and control subjects’ simulator performance At time-1, patients with TBI performed significantly more poorly than the healthy reference subjects on
182
H. L. Lew et al.
Figure 2. Distribution of Simulator Performance Index scores in the Healthy Reference Group and Subjects with TBI. The Simulator Performance Index is a composite of 12 performance measures on the driving simulator. The index is a normalized z-score in standard deviation units (see Methods). N ¼ 16 healthy reference subjects and 11 patients with TBI.
two global measures of simulator performance: the automated Simulator Performance Index (SPI; t ¼ 3.83, df ¼ 25, p ¼ 0.001), and the observational Driving Performance Inventory (DPI; t ¼ 5.36, p < 0.001). On both indices, patients averaged more than 4 SD below normative values. Figure 2 shows the automated Simulator Performance Index of patients in relation to the healthy reference group. While 55% of patients failed the simulator trial (SPI 2.0), all reference subjects passed. Post hoc examination of automated simulator measures found that the TBI group was significantly impaired on both simulator subscales: Speed-Control and Direction-Control, including eight of the twelve measures composing these subscales (see Table I). Likewise, observational ratings of the TBI group’s simulator performance were significantly poorer on all four subscales of the DPI: Handling of Controls, Regulation of Trajectory, Basic Maneuvers, and Higher Order Skills (all t > 3.15, all padj < 0.01). The TBI group also missed significantly more signals on the divided-attention task (t ¼ 5.25, padj < 0.001), with a non-significant trend for slower reaction times on correct responses (t ¼ 2.01, padj ¼ 0.06). Table II lists violation counts and performance on the divided attention task for each subject group. On average, the TBI group made approximately five times as many traffic violations on the simulator
as the reference group (28 vs 5.5 errors, t ¼ 4.20, p ¼ 0.002). To consider possible mediating variables and artifacts, we analyzed whether subjects’ age, months from time-1 to time-2, or the total hours observed at follow-up were correlated with the SPI measures, DPI ratings on the simulator and at follow-up, passing of the road test, or dividedattention performance. None of these relationships even approached significance (all unadjusted p > 0.10, by Pearson correlations).
Comparison of simulator skills to performance at follow-up Table III compares driving performance at follow-up with simulator performance at baseline, in terms of both automated and observational simulator skills (discussed here and in the following paragraph respectively). In terms of automated measures, patients’ SPI at time-1 significantly correlated with their observational DPI ten months later (r ¼ 0.66, n ¼ 11, p ¼ 0.01). Post hoc analyses found that both automated simulator subscales contributed: SpeedControl and Direction-Control (both r 0.64, padj ¼ 0.02). In analyzing the relation between failing the simulator trial at time-1 (ZSPI < 2.0) and failing the driving evaluation at time-2 (DPI < 1.5 or any DPI item ¼ 0), the simulator trial showed 100%
Validity of driving-simulator assessments following TBI
183
Table I. Automated measures of performance by patients on the driving simulator at time-1. Driving simulator measure
Z-score (mean SD)
Significance ( p)
Overall Simulator Performance Index (SPI) Speed Control speed (time over posted limit) speed variability acceleration variability speed jerk red-light violations
R4.61 4.70 R2.71 0.56 R1.92 1.23 R3.57 2.96 R1.81 1.43 0.98 2.18 0.18 1.18
0.001 0.001 0.001 0.50). Figure 3 shows the relationship of subjects’ driving performance at time-2 to their simulator and road test performance at time-1. Regardless of whether they passed or failed the road test at time-1, all subjects reported driving during the follow-up period. The subject who passed the road test but had the lowest SPI score on the simulator had a rollover car accident midway through the follow-up period that resulted in licence revocation; this subject chose to discontinue driving for the remainder of the followup period.
Discussion
Comparison of road test to simulator skills and performance at follow-up Most subjects in the TBI and reference groups reported that they found operating the driving simulator more difficult than driving a car. This was corroborated in the TBI group by significantly lower DPI scores on the simulator than the road test (t ¼ 2.58, df ¼ 9, p ¼ 0.03), which mainly reflected lower ratings on four DPI items: speed, lane tracking, following distance, and safety judgment. On these items, almost all subjects received the maximum rating of 2 (‘safe’) during the road test, while their median score for these items on the simulator was 1 (‘needs improvement’). Subjects’ DPI score during the road test at time-1 was not significantly correlated with their DPI score on the simulator (r ¼ 0.26, p > 0.10) nor with their automated Simulator Performance Index at time-1 (r ¼ 0.11, p > 0.50). Inspection of the data suggested that this mainly reflected extremely discrepant scores for two subjects: one did well on the simulator (Z ¼ þ0.5) but failed the road test with a very low score; the other subject had the poorest performance on the simulator (Z ¼ 13.2) but passed the road test with a very high score.
There were three primary findings in this study: (1) Patients with TBI performed significantly more poorly on a driving simulator than healthy individuals. (2) Patients’ driving simulator measures correlated significantly with their long-term driving performance in the community. (3) Evaluations with a driving simulator provided unique information on patients’ driving skills that was not redundant with the results of a standard road test. Compared to the normative sample, TBI patients’ largest deficits were in several measures of speed regulation, steering control, and in the ability to follow traffic regulations. Patients had more difficulty controlling the speed of the virtual vehicle, as indicated by higher mean speed and greater variability in speed and acceleration. They had greater difficulty controlling the direction of the virtual vehicle, as indicated by greater variability in lane tracking on straight roadways and a higher index of jerky steering. These problems were reflected in patients’ having greater difficulty following traffic regulations on the simulator, which included more speed-limit violations, leaving the roadway more often, and having more collisions. Overall, the patients with TBI committed about five times as many driving violations on the simulator as
Validity of driving-simulator assessments following TBI
Follow-up Driver Rating (DPI mean)
2.00
P
F
185
P
F
P
1.75
F P 1.50
F* P F
1.25
P R-Square = 0.44 1.00 −12
−10
−8
−6
−4
−2
0
2
Simulator Performance Index (Z) Figure 3. Relationship of Driving Skills at time-2 to Simulator and Road Test Performance at time-1. This scatter-plot shows the relationship of simulator performance at baseline to driver ratings at the 10-month follow-up (r ¼ 0.66, p ¼ 0.01) for 11 patients with TBI. Also included are pass/fail grades on the road test at baseline (P, F), which were not significantly related to simulator performance or to driver ratings at follow-up. One subject (F*) was judged unsafe to drive based on clinical evaluation of safety, was not allowed to take the road test, and was assigned a failing grade on the road test.
the healthy comparison group. In addition, the TBI group performed significantly more poorly on the secondary divided-attention task while driving the simulator, responding to less than half as many signals as normative subjects. These findings suggest that patients with TBI are more likely to have a diminished capacity to control the speed and direction of a car, as well as greater difficulty with divided attention and multitasking while driving. We tested the ability of simulator and road-test ratings of driving skills to predict driving performance during the subsequent 10-month period. The follow-up evaluations consisted of structured observer ratings by a family member, who accompanied each subject while driving for a minimum of three hours over at least a 1-month period. Automated measures of simulator performance showed moderately strong ability to predict driving skills at the 10-month follow-up (82% predictive efficiency overall). This included a non-significant tendency for more frequent car accidents by subjects who performed poorly on the simulator. Interestingly, the automated simulator measures did somewhat better than observer-rated simulator performance in accounting for the details of patients’ driving skills at follow-up. This likely reflects the standardized nature of automated data collection, as well as the capacity of the simulator to measure subtle aspects of performance that may be more
difficult for an observer to assess reliably, such as fluctuations in acceleration and lane tracking, smoothness of steering, and the capacity to divide attention among different tasks. Previous studies have found that subjects’ dividedattention skills are related to their driving simulator performance [14, 21, 32, 33]. The present study extends this finding by showing that dividedattention skills on a stimulator can also predict actual driving performance in the community. In the real world, drivers must constantly react to multiple competing demands (vehicle operation, activities within the car, events outside the car), which test their ability to prioritize and allocate attention safely and efficiently. Prior research has shown that driver distraction reduces performance during simulated driving [34], and also that it is a significant contributor to automotive accidents worldwide [35, 36]. Thus, our next series of studies will examine factors that influence drivers’ susceptibility to distraction, including the efficacy of cognitive training and driver training as a part of the rehabilitation process. As reviewed previously, some studies have found simulator and road-test evaluations to be correlated, while others have not. The present study did not find these two methods of driver evaluation to be correlated. Some individuals did quite well on the road test but not on the simulator, and vice versa.
186
H. L. Lew et al.
As discussed below, there are several possible reasons for this discrepancy, including differences in difficulty, the range of driving situations, and the attitude of drivers toward each type of evaluation. Task difficulty appears to be an important difference between simulator and road-test evaluations. Several prior studies have found that subjects tend to perform more poorly on driving simulators than in road tests [16, 20–22]. In our study also, patients received significantly lower ratings of their driving skills on the simulator than during the road test. They had greater difficulty with speed regulation, lane tracking, vehicle following distance, and showed poorer judgment when making decisions on the simulator. Most patients and control subjects reported that they found the driving simulator more difficult than driving a car. They related this to inherent features of the simulation (e.g. greater difficulty estimating distances and speed on a two-dimensional video screen), as well as to challenges that were intentionally programmed into the driving scenarios (e.g. sudden incursions of pedestrians and vehicles in front of the virtual vehicle, without warning). Simulators have the potential of presenting a wider range of driving situations. Road tests are typically designed to avoid or minimize conditions that could physically threaten the driver and evaluator’s safety, while this factor does not limit the driving scenarios on a simulator. This was evident in the present study, in that one subject was judged by clinicians as unsafe to attempt the road test, and several had their road test discontinued by the certified driving evaluator. In contrast, all subjects were able to complete the entire simulator trial. Thus, a well-designed simulator trial may allow one to assess driving skills in a wider range of standardized scenarios (including adverse conditions) that are difficult for a road test to consistently and safely capture. Furthermore, it appears that drivers often approach a simulated drive with different attitude than a road test. We observed several of our subjects engaging in reckless maneuvers on the simulator that they did not show during the road test. Indeed, most drivers probably assume a more cautious approach during a road test than in their daily driving, due to the visible presence of the evaluator in the seat next to them. In contrast, the observational ratings on our simulator were carried out by an unobtrusive observer who was out of sight, behind the driver. This arrangement may facilitate a more relaxed attitude during simulator trials that allows some drivers’ natural responses to emerge. How should these findings be applied in clinical practice? The present study and prior research indicate that a driving simulator test may be more sensitive in detecting some drivers’ vulnerabilities
than a road test. Therefore, we recommend using both these approaches to evaluate patients with brain injuries. When the two methods agree regarding a patient’s driving skills, then the recommendation to drive or not to drive can be made with greater confidence. When the two evaluations do not agree, we feel that the wisest approach is to recommend that the patient participate in additional driver training and practice, and be re-tested before resuming driving independently. The following limitations of the present study should be addressed to enhance future research: (1) The small number of subjects in this initial study limited the power to detect potentially important relationships (such as the possible relation between simulator performance and subsequent accidents). Also, the small sample required that we analyze patients with TBI as a homogeneous group. A larger sample will allow classification of patients with TBI by the severity of their injury, site of lesion, or time since injury, to determine the impact of these factors on different aspects of driving performance. Furthermore, a larger sample is needed to determine the relationship among the various findings in the present study, such as the possible mediating role of divided-attention in other driving skills. (2) The present study only included healthy controls to obtain reference values for the simulator measures; these subjects were not given a road-test or the 10-month follow-up. Ideally, control subjects should also complete all evaluations, so that more complete comparisons can be made across the full range of driving ability. In addition, to optimize group comparisons, it would be desirable to study TBI and control groups that are more closely matched by age than the present sample’s difference of 7 years (a non-significant difference). Nonetheless, it is unlikely that this difference introduced any important artifacts into our findings, since subjects’ age, in our sample, was unrelated to their driving performance on any measure. (3) The present study did not include neuropsychological or psychophysiological measures of subjects’ perceptual, motor, and cognitive skills. Adding such measures would allow greater understanding of the neurocognitive processes underlying driving skill and safety [9, 10, 32]. Moreover, such studies are important to evaluate the incremental validity of simulator-based assessments—that is, whether driving simulator measures can provide unique and ecologically valid information beyond what is provided by commonly available neuropsychological tests. The present study indicates that comprehensive measures of driving-simulator performance do have sufficient predictive power to warrant larger indepth studies.
Validity of driving-simulator assessments following TBI
Acknowledgments We are grateful to Marie N Dahdah BS, Rose Marie Salerno RN, Jill Storms OTR/L, Jean Gurga MA, OTR/L, Elaine Date MD, and Deborah Warden MD for their invaluable input during the manuscript preparation and revision. Appendix
Driving Performance Inventory (DPI) . This observational inventory has 14 items, each of which is rated on a 3-point scale:
2 ¼ Good, safe 1 ¼ Fair, needs improvement 0 ¼ Poor, unsafe . Items:
A. Handling of vehicle controls (1) steering wheel control, (2) throttle-brake coordination B. Regulation of vehicle trajectory (3) speed, (4) lane tracking, (5) brake reaction time C. Basic maneuvers (6) lane changes, (7) execution of L and R turns, (8) merging into traffic, (9) obedience to traffic signs and signals, (10) following distance D. Higher-order skills (11) safety judgments (passing, yielding right of way, etc.), (12) correctness and speed of decisions, (13) emotional stability and self-control, (14) need for intervention by the observer. The DPI was developed at the Palo Alto VA (2001) as a collaboration of the Driver Assessment Clinic and the Driving Performance Research project.
References 1. Brouwer WH, Withaar FK, Tant MLM, van Zomeren AH. Attention and driving in traumatic brain injury: A question of coping with time-pressure. J Head Trauma Rehabil 2002;17:1– 15. 2. Galski T, Bruno RL, Ehle HT. Driving after cerebral damage: A model with implications for evaluation. Am J Occup Ther 1992;46:324–32. 3. Fisk G, Schneider J, Novack T. Driving following traumatic brain injury: Prevalence, exposure, advice and evaluations. Brain Injury 1998;12:683–95. 4. Oliver JH, Ponsford JL, Curran CA. Outcome following traumatic brain injury: A comparison between 2 and 5 years after injury. Brain Injury 1996;10:841–8. 5. Katz RT, Golden RS, Butter J, Tepper D, Rothke S, Holmes J, Saghal V. Driving safety after brain damage: Follow-up of twenty-two patients with matched controls. Arch Phys Med Rehabil 1990;71:133–7.
187
6. Priddy DA, Johnson P, Lam CS. Driving after a severe head injury. Brain Injury 1990;4:267–72. 7. Gianutsos R. Driving advisement with the Elemental Driving Simulator (EDS): When less suffices. Behavior Research Methods, Instruments, & Computers 1994;26:183–6. 8. Liu L, Miyazaki M, Watson B. Norms and validity of the DriVR: A virtual reality driving assessment for persons with head injuries. Cyberpsychology & Behavior 1999;2:53–67. 9. Lundqvist A, Gerdle B, Roennberg J. Neuropsychological aspects of driving after a stroke—In the simulator and on the road. Applied Cognitive Psychology 2000;14:135–50. 10. Szlyk JP, Myers L, Zhang YX, Wetzel L, Shapiro R. Development and assessment of a neuropsychological battery to aid in predicting driving performance. Journal of Rehabilitation Research & Development 2002a;39:483–95. 11. Zesiewicz TA, Cimino CR, Malek AR, Gardner N, Leaverton PL, Dunne PB, Hauser RA. Driving safety in Parkinson’s disease. Neurology 2002;59:1787–8. 12. Netz J, Wolbers T. Hemianopsia and driving. Europa Medicophysica 2001;37:275–8. 13. Szlyk JP, Taglia DP, Paliga J, Edward DP, Wilensky JT. Driving performance in patients with mild to moderate glaucomatous clinical vision changes. Journal of Rehabilitation Research & Development 2002b;39:467–81. 14. Brouwer, WH, Waterink W, Van Wolffelaar PC, Rothengatter T. Divided attention in experienced young and older drivers: Lane tracking and visual analysis in a dynamic driving simulator. Human Factors 1991;33: 573–82. 15. Lee HC, Drake V, Cameron D. Identification of appropriate assessment criteria to measure older adults’ driving performance in simulated driving. Australian Occupational Therapy Journal 2002;49:138–45. 16. Blaauw GJ. Driving experience and task demands in simulator and instrumented car: A validation study. Human Factors 1982;24:473–86. 17. Roge J, Pebayle T, Hannachi SE, Muzet A. Effect of sleep deprivation and driving duration on the useful visual field in younger and older subjects during simulator driving. Vision Research 2003;43:1465–72. 18. Edwards DS, Hahn CP, Fleishman EA. Evaluation of laboratory methods for the study of driver behavior: Relations between simulator and street performance. Journal of Applied Psychology 1977;62:559–66. 19. Volkerts ER, Van Laar MW, Van Willigenburg AP, Plomp TA, Maes RAA. A comparative study of on-the-road and simulated driving performance after nocturnal treatment with lormetazepam 1 mg and oxazepam 50 mg. Human Psychopharmacology 1992;7:297–309. 20. Toernros J. Driving behaviour in a real and a simulated road tunnel—a validation study. Accident Analysis & Prevention 1998;30:497–503. 21. Reed MP, Green PA. Comparison of driving performance on-road and in a low-cost simulator using a concurrent telephone dialing task. Ergonomics 1999;42:1015–37. 22. Blana E, Golias J. Differences between vehicle lateral displacement on the road and in a fixed-base simulator. Human Factors 2002;44:303–13. 23. Freund B, Gravenstein S, Ferris R, Shaheen E. Evaluating driving performance of cognitively impaired and healthy older adults: A pilot study comparing on-road testing and driving simulation. Journal of the American Geriatrics Society 2002;50:1309–10. 24. Galski T, Bruno RL, Ehle HT. Driving after cerebral damage: A model with implications for evaluation. American Journal of Occupational Therapy 1992;46:324–32. 25. Galski T, Bruno RL, Ehle HT. Prediction of behind-thewheel driving performance in patients with cerebral brain
188
26.
27.
28.
29.
30.
H. L. Lew et al.
damage: A discriminant function analysis. American Journal of Occupational Therapy 1993;47:391–6. Huchler S, Jorg A, Golz D, Kust J, Netz J, Homberg V, Karbe H. Evaluation of driving simulation. Europa Medicophysica 2001;37:283–9. Keller M, Kesselring J, Hiltbrunner B. Fitness to drive with neurological disabilities. Neurorehabilitation and Neural Repair 2003;17:168–75. Nouri FM, Tinson DJ. A comparison of a driving simulator and a road test in the assessment of driving ability after a stroke. Clinical Rehabilitation 1988;2: 99–104. Szlyk JP, Seiple W, Viana M. Relative effects of age and compromised vision on driving performance. Human Factors 1995;37:430–36. Bylsma FW. Simulators for assessing driving skills in demented patients. Alzheimer Disease & Associated Disorders 1997;11 (Suppl. 1):17–20.
31. Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988;75:800–2. 32. Korteling JE, Kaptein NA. Neuropsychological driving fitness tests for brain-damaged subjects. Arch Phys Med Rehabil 1996;77:138–46. 33. Lengenfelder J, Schultheis MT, Al-Shibabi T, Mourant R, Deluca J. Divided attention and driving: A pilot study using virtual reality technology. J Head Trauma Rehabil 2002;17:26–37. 34. Lansdown TC, Brook-Carter N, Kersloot T. Distraction from multiple in-vehicle secondary tasks: Vehicle performance and mental workload implications. Ergonomics 2004;47:91–104. 35. Stutts JC, Reinfurt DW, Rodgman EA. The role of driver distraction in crashes: An analysis of 1995–1999 Crashworthiness Data System Data. Annu Proc Assoc Adv Automot Med 2001;45:287–301. 36. Lam LT. Distractions and the risk of car crash injury: The effect of drivers’ age. J Safety Res 2002;33:411–9.
