Development of an Operational Setting Specific Field ... - SAGE Journals

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006

2572

DEVELOPMENT OF AN OPERATIONAL SETTING SPECIFIC FIELD COGNITIVE ASSESSMENT PROCEDURE Wayne C. Harris1, Dennis L. Reeves2, Timothy F. Elsmore3 and Peter A. Hancock1 The assessment of dynamic cognitive abilities during actual field operations by professional personnel represents a significant challenge to the behavioral scientist. Here, we present a short-duration, portable cognitive assessment instrument which seeks to address such issues. Any such instrument with poor face validity is unlikely to receive the required attention and cooperation of operational personnel. Also, if supervisors see little relationship between the assessment procedure and the actual operational tasks they are unlikely to use outcome results anyway. This work summarizes our customization of an established cognitive assessment battery, which is composed of operationally relevant tasks for field assessment. Our present results suggest equivalent user acceptance and comparable assessment capacity as the longer duration, currently established battery. 1

University of Central Florida, 2 Clinvest, Inc., 3 Activity Research Services

INTRODUCTION Technology now performs tasks that in the past required human operators to accomplish. Technology further provides decision support to human operators but system performance is still largely contingent upon the operator’s ability to manage the system. The frequency of actual operator-machine interactions may have decreased, but cognitive demands are heightened as such periodic decisions assume ever greater importance, while additional supervisory responsibility also results in the need for additional cognitive effort (Sheridan, 1987). Sufficient cognitive ability for jobs is determined during the selection procedure and demonstrated and enhanced during training, but cognitive capacity is itself dynamic. External and internal conditions affect it and conditions that serve to decrease cognitive capacity are frequent in settings where cognitive deficits are often most costly. Individuals, and their supervisors, must continually evaluate cognitive capacity and adjust cognitive demands appropriately, replace impaired personnel from critical assignments and mitigate stress effects with interventions such as nutritional support and flexible work scheduling. However, any such intervention requires an accurate measure of

current cognitive capacity, and especially in field conditions such an assessment represents a formidable challenge. REVIEW There have been several systematic attempts to generate these flexible and robust test capacities. For example, computer versions of classic tests of cognitive functioning are available in the Walter Reed Performance Assessment Battery (PAB) (Englund., et al, 1987). PAB task are classified by cognitive abilities which represent elemental skills that underlie real-world tasks (Thorne, Genser, Sing, & Hegge, 1985). The Automated Neuropsychological Assessment Metric (ANAM) (Reeves, et al., 1991) organized and presented PAB tasks in a user-friendly manner. The Automated Readiness Evaluation System (ARES) developed by Elsmore and Reeves (in press) is an extension of this battery and presents ANAM tasks on handheld computers. The ARES has proven to be a most useful method of assessing cognitive functioning in operational settings (Harris, & Hancock, 2003; Harris & Hancock, & Morgan, 2005), and component test profiles have been constructed for field settings that provide a profile of the cognitive

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006

2573

capacities deemed to be important in those settings (Harris, Hancock, & Harris, 2005; Harris, & Hancock, 2003; Harris, Hancock, & Morgan, 2005). The ARES assesses cognitive capacities using a series of generic tasks and abstract symbols which are derived presently to represent recognizable elements of common military tasks, although in principle any such abstract representations could be developed. Observations across a series of tests may indicate that that cognitive capacity is declining, but changes can also indicate decreases in motivation. Performance during operational conditions is typically accompanied by high levels of motivation, and motivation should also be high during assessments to accurately estimate operational cognitive functioning. The more irrelevant an assessment activity appears the less effort an individual is likely to expend when their cognitive resources become depleted and as a consequence, the less accurately will assessments predict actual operational effectiveness. In the present version, care was taken to address this specific issue of motivation to perform versus actual capacity change.

present for supervisors to justify operational changes. Battery development began with the Automated Readiness Evaluation System (ARES), a cognitive assessment tool that includes tasks that are relevant to military performance and are sensitive to fluctuations in demand associated with infantry operations (Harris, Hancock & Morgan, 2005).

DEVELOPMENT OF AN OPERATIONS TESTING RELEVANT COGNITIVE BATTERY

The standard ARES includes a test that assesses Simple Response Time. A simple asterisk appears on the screen and the participant’s task is to press a designated button as rapidly as possible. The ARES-I includes a task with similar cognitive demands, and adds two tasks in which responses require stimulus classification. Figure 1 presents the revised Simple Reaction Time display in the ARESI. The participant presses a key when a stimulus (a white silhouette) appears. The timing of their appearances is identical to the ARES Simple Reaction Time task. Decisions are frequently not limited to just detecting when an object appears. They require discriminating between objects and responding appropriately. Two ARES-I tasks were reconfigured that required such stimulus classification. In these tasks, white and green silhouettes appear in windows. In the first two-stimuli task, participants respond (by pressing the button labeled ‘5’) only to white figures.. This task corresponds to a shoot/no shoot decision. In the second two-stimulus task, the participant presses the right key when a white

To disambiguate the potential confound between actual decrement and motivation level, a battery of established basic cognitive functions tests was assembled and then modified by changing the appearance of each task so as to be relevant to the current job performance in the field. We anticipated that this manipulation would sustain operational levels of motivation and that assessments would therefore more accurately reflect the person’s current ability to perform cognitively demanding in the field. In addition to the motivational value of the new task configurations it was postulated that assessments with little relationship to current assignments would be unlikely to be used by supervisory personnel. Cancellations and delays are usually costly, and an obvious relationship between an assessment and operational activities needs to be

The ARES-I (ARES Infantry Version) In ARES-I (Infantry Version), tasks were developed to assess Simple Reaction Time and Logical Relations and Spatial Processing. These basic cognitive performance capacities have been shown to be sensitive to infantry operational conditions (Harris, Hancock, & Morgan, 2005). Tests were modified to retain the psychological demands of the ANAM/ARES tasks, while presenting those demands in a task structure relevant to infantry personnel. Three specific ANAM tasks were modified. 1. Simple Reaction Time

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006

silhouette appears, and they are required to press the left key when the silhouette is green. Thus, participants classify all stimuli.

2574

3. Spatial Processing To evaluate spatial processing, the ARES figure rotation task uses 4x4 grids. Two grids are presented, one of which is rotated 900 and the participant’s task is to indicate whether the figures are the same or different. In the ARES-I, a map with three dots representing military units appears (see Figure 3). The map disappears for 2 seconds after which it reappears rotated 900, 1800 or 2700. The participant’s task is to indicate whether the unit locations are the same or different than their location on the first map. The map remains constant for all trials.

Figure 1. ARES-I Reaction Time display. 2. Logical Relations The standard ARES abstract reasoning task is based on a task introduced by Baddeley (1968) in which two symbols appear on the screen (# &). A sentence above this display describes the relationship between the symbols such as “& is first.” The participant’s task is to indicate whether the statement is true (left button) or false (right button). In the ARES-I task two military vehicle map symbols move across the screen from left to right and below them a statement describing the relationship of the vehicles (Figure 2).

Figure 2. ARES-I Logical Relations display.

Figure 3. Map rotation display. ARES-I EVALUATION Initial ARES-I evaluation consisted of comparison of ARES-I results with previous ARES data and comments from users. Evaluations were conducted using police officers who completed the ARES-I four times during a day of weapons qualification, and Marine instructors who completed the ARES-I battery at the end of each workday for 30 days. Evaluation of ARES-I included comparison of data with previous ARES field data and user comments. Response time changes during the first four assessments were consistent with previous ARES data. Accuracy was greater than 90% during the first trial and remained at that level during all four trials. Consistent with previous examinations of ANAM and ARES during initial testing

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006

(Benedetto, Harris, & Goernert, 1995; Harris, Hancock, & Morgan, 2005), response time improved between trials one and four, with the rate of improvement decreasing with the number of trials and performance stabilizing on trial four. Comparison of ARES-I response times with comparable data from Special Forces soldiers (Harris, Hancock, & Morgan, 2005) indicated that SRT mean response times were longer on the ARES-I. The longer response time may reflect smaller target ARES-I stimuli size, more complex background and moving stimuli location. Differences may also reflect the characteristics of the sample. Group means were Special Forces (220ms), Marine Instructors (255ms), and Police Officers (274ms). The Convoy task response time and the relative time difference between Simple Response Time and Abstract Reasoning similarly suggest that the present interface changes did not significantly change the cognitive demands of the task. ARES-I response time differences in the three Reaction Time tasks increased with task complexity. Consistent with previous findings that performance improves during trials when the tasks are being learned and increases with fatigue, within trial performance of police officers improved during the first trial. Within trial improvement ceased as the number of trials increased and within trial performance deteriorated during the fourth trial late in the day. The results are consistent with ANAM within trial data and most probably represents fatigue effects late in the day. (Harris, Hancock, & Harris, 2005) The critical role of motivation was evident from the decrease in the number of Police Officers returning for assessments as the day progressed and the sporadic data from 3 of the 6 Marine volunteers. Police Officers indicated the tasks were similar to challenging games and appeared to be motivated. They were interested in their scores, particularly when testing occurred in group settings. Comments from the Marines indicated that assessment tasks were relevant to their work. Those who completed the battery indicated that it was challenging and even enjoyable at times but that the 15-minute completion time was a problem. Workdays often end between 2200 and 0200 hours thus adding 15

2575

minutes to the workday required a major commitment. Negative comments were aimed primarily at the map task. The task required both spatial rotation and spatial memory, and its relatively high task difficulty was indicated by longer response times and error rates 3-4 times greater than other ARES-I tasks. Average Marine cognitive performance remained stable during 30 days of assessment. Day-to-day changes appear to reflect cognitive ability variability but no decrement indicative of loss of motivation was found. DISCUSSION Difficulty predicting real-world performance from cognitive assessments may be at least partially due to low motivation during assessment. To increase such motivation, an assessment battery was modified to make tasks operationally relevant. Initial evaluation of the ARES-I suggest that the interface change produced an assessment tool with user acceptance and performance data consistent with previous ARES research.

ACKNOWLEDGEMENTS This work was supported by the Department of Defense Multi- Disciplinary University Research Initiative (MURI) program, P.A. Hancock, Principal Investigator, administered by the Army Research Office under Grant DAAD19-01-1-0621. The views expressed in this work are those of the authors and do not necessarily reflect official Army policy. The authors wish to thank Dr. Sherry Tove, Dr. Elmar Schmeisser, and Dr. Mike Drillings for providing administrative and technical direction for the Grant. REFERENCES

Benedetto, J., Harris, W.C., & Goernert, P.N. (1995). Assessing gender differences and norm data on a cognitive performance measure. Proceedings of the Human Factors and Ergonomics Society, 39, 968.

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006

Baddeley, A. D. (1968). A three minute reasoning tested based on grammatical transformation. Psychonomic Science, 10 (10), 341-347. Elsmore, T.F., & Reeves, D.L. (2006). ANAMTM for palm OS handheld computers: The ARES test system for Palm OS handheld computers. Neuropsychology Review, in press. Englund, C.E., Reeves, D.L., Shingledecker, C.A., Thorne, D.R., Wilson, K.P., & Hegge, F.W. (1987).Unified Tri-service cognitive performance assessment battery (UTC-PAB). I. Design and specification of the battery. Report No. 87-10, Naval Health Research Center, San Diego, CA. Harris, W.C., & Hancock, P.A. (2003). Field assessment of cognitive performance under stress, Proceedings of the Human Factors and Ergonomics Society, 47, 1953-1957. Harris, W.C., Hancock, P.A., & Harris, S.C. (2005.). Information processing changes

2576

following extended stress. Military Psychology, 17 (2), 115-128. Harris, W.C., Hancock, P.A., & Morgan, C.A. (2005). Cognitive change following Special Forces training, Proceedings of the Human Factors & Ergonomics Society, 49, 1776-79. Reeves, D.L., Winter, K.P, LaCour, S.J., Raynsfor, M., Vogel, K., & Grissett, J.D. (1991). The UTC-PAB/AGARD stress battery: User’s manual and system documentation. (Special Report 91-3). Pensacola, FL: Navy Aerospace Medical Research Laboratory. Sheridan, T.B. (1987). Supervisory control. In G. Salvendy (Ed.), Handbook of Human Factors (pp.1243-1268). New York: Wiley. Thorne, D.R., Genser, S.G., Sing, H.C., & Hegge, F.W. (1985). The Walter Reed Performance Assessment Battery. Neurobehavioral Toxicity and Teratology, 7, 415-418. .