Ergonomics Society Annual Meeting Proceedings of ...

Proceedings of the Human Factors and Ergonomics Society Annual Meeting http://pro.sagepub.com/

Individual Pilot Differences Related to Situation Awareness Cheryl A. Bolstad Proceedings of the Human Factors and Ergonomics Society Annual Meeting 1991 35: 52 DOI: 10.1518/107118191786755896 The online version of this article can be found at: http://pro.sagepub.com/content/35/2/52

Published by: http://www.sagepublications.com

On behalf of:

Human Factors and Ergonomics Society

Additional services and information for Proceedings of the Human Factors and Ergonomics Society Annual Meeting can be found at: Email Alerts: http://pro.sagepub.com/cgi/alerts Subscriptions: http://pro.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://pro.sagepub.com/content/35/2/52.refs.html

>> Version of Record - Sep 1, 1991 What is This?

Downloaded from pro.sagepub.com by Cheryl Bolstad on April 23, 2012

PROCEEDINGS of the HUMAN FACTORS SOCIETY 35th ANNUAL MEETING-I991

INDIVIDUAL PILOT DIFFERENCES RELATED TO SITUATION AWARENESS Cheryl A. Bolstad Northrop Aircraft Division Hawthorne, California A pilot's ability to maintain a high level of situation awareness (SA) has been widely recognized as an important component of mission success and survivability in the air combat arena. The need for SA enhancement has led to the creation of a test battery designed to measure pilot attributes that are thought to correlate to SA. By correlating SA performance measures with results from the selected attribute tests, it will be possible to develop highly focused instructional methods that will improve performance in the attributes, thus enhancing SA. empirical study is described in which pilot performance on the attribute battery and an objective, quantitative measure of SA were obtained and correlated.

INTRODUCTION A pilot's ability to maintain a high level of situation awareness (SA) has been widely recognized as an important component of mission success and survivability in the air combat arena. The sophistication and complexity of modern, highperformance tactical aircraft require today's pilots to integrate far more information than their predecessors. For this reason, methods need to be developed to enhance pilot SA and thus improve the pilot's ability to function effectively in a highly complex and dynamic environment.

Attribute Test B a r n A literature search was conducted to identify tests measuring attributes which correlate to SA. Six general areas arose from the literature review: spatial abilities, attention, memory, perceptual abilities, cognition, and personality factors (Bolstad, 1990). From the literature in these areas and discussions with professionals directly involved in military testing, hundreds of tests were found that have a possibility of measuring SA correlates.

The first objective toward reaching the goal of developing methods to enhance SA is to define it. Situation awareness is "the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future". Previous studies have shown large differences i n individual performance on generating and maintaining SA, indicating that SA may be trainable (Endsley, 1988).

Selection of specific tests to include in the battery was based on several criteria. First, tests should tap one of the six areas mentioned above. Since the goal of the battery was to aid in the development of training methods designed to enhance SA, tests that measure potentially trainable attributes were selected. Preference was given to tests that are in current military selection batteries, since information on reliability and validity for military populations is readily available. Tests that were easy to develop, administer and have the potential to be administered in a computerized format were given a high priority for selection. Over 100 tests were reviewed, and 18 were selected for inclusion in the battery. The final battery used two methods for administration: papedpencil and computerized format (Table 1-on next page). Many of the tests included in the SA attribute battery were replications of tests used in the current military batteries, such as the Basic Attributes Test (BAT) used by the Air Force (Carretta, 1987). Several additional tests were included, consisting mostly of modifications of standardized tests or psychological paradigms.

This paper describes the compilation and evaluation of a preliminary test battery that is being used to investigate individual attributes thought to be associated with high SA. The attributes may account for at least some individual differences in pilot SA. If correlations between specific attribute measures and SA performance can be identified, and if,the attributes are trainable, then instructional methods can be developed to improve performance in these attributes, thus enhancing SA. The battery reported here is preliminary, and results from this study will aid in the refinement of the battery as well as providing preliminary insights into individual differences in pilot SA. The first section of this paper describes the attribute test battery that was created. In the second section, an

52 Downloaded from pro.sagepub.com by Cheryl Bolstad on April 23, 2012

PROCEEDINGS of the HUMAN FACTORS SOCIETY 35th ANNUAL MEETING-1991

Summaries of the selected tests are presented below:

orientation (Ekstrorn, French and Harmon, 1976; Guilford and Lacey, 1947). A maze task was included to measure graphical orientation.

Spatial Abilities. Spatial ability attribute tests were selected to measure an individual’s ability to mentally visualize and manipulate objects spatially. Four tests were selected. The Revised Minnesota Form Board Test (Form BB) was selected to measure visualization (Lickert and Quasha, 1969). The Cube Comparison Test was used to measure mental rotation, and an Object Identification Test was selected and modified to measure aerial

Attentiom. This test measured attention sharing. It was modified based upon earlier experiments by North and Gopher (1976) and Damos (1978). Subjects were required to use a joystick to track a ball moving randomly across a computer screen, while simultaneously performing a compensatory tracking task.

TABLE 1 Selected Tests and Their Predictor Variables ATTRl BUTES MEASURED

TEST NAME

PREDICTOR VARIABLES-

Revised Minnesota Form Board Test

Visualization

Number Correct

Cube ComparisonTask

Mental Rotation

Number Correct

Object Identification

Aerial Orientation

Number Correct

Maze Task

Graphical Orientation

Test Time

Time Sharing

Attention Sharing

Average RT, Average Tracking Error, Average Tracking Level

Immediate/Delayed Memory

Memory

Average RT, Total Errors

Perceptual Speed

Digit Memory


Encoding Speed

Verbal Classification


PerceptualVigilance

PerceptualVigilance

Average RT

Raven’s Advanced Progressive Matrices Pattern Recognition Analytical Reasoning

GRE Analytic Subtest

Pilots Decisional Attribute Questionnaire Risk Taking

Number Correct Number Correct Highest Scale

InternalTiming

Rate Estimation

Average Absolute Error

This I Believe Test

Cognitive Complexity

Level Obtained

Abstract Orientation Scale

Cognitive Complexity

Level Obtained

Aviator Locus of Control

Locus of Control

Highest Scale

GEFT (Group Embedded Figures Test)

Field Independence

Number Correct

Dot Estimation

Compulsiveness/ Decisiveness

Number Correct, Average RT

Biographical

Background Information

Age, Flight Hours, Flight Years

NOTE: Average response times (RT) based upon correct trials only. Computerized Test


PROCEEDINGS of the HUMAN FACTORS SOCIETY 35th A N N U A L MEETING4991

Jvlemory. This test was used to measure short-term memory capacity. The test was based upon the immediate/delayed memory test developed for use in the BAT (Carretta, 1987), and consisted of two subtests. During both subtests a series of numbers is flashed on a computer screen. In the first subtest, the subject is asked to remember one number occurring prior to the number on the screen. In the second subtest, the subject is asked to remember two numbers prior to the number displayed on their screen.

aDh ical. A biographical survey was developed to address the possibility that individual factors, such as pilot experience or age, may influence pilot SA scores. Situation Awareness Measurement SA was measured using the Situation Awareness Tech niq ue (SAGAT) G Io b al Asse ss me nt developed by Northrop (Endsley, 1987). SAGAT is an objective, quantitative comparison between what a pilot perceives is happening during a mission and what is actually occurring. In the SAGAT test, a pilot flew a man-in-the-loop simulation with a specified scenario and aircraft sub-systems. At a random point in time the simulation was stopped and the pilot’s display screen was blanked. This screen had various aircraft displays and an out-the-window view. During the stops, the pilot was asked a series of randomly selected questions designed to measure his SA knowledge. The questions were asked via a computer interface, and the pilots answered using a mouse to input his answers. The duration of the stop was typically around two minutes. At the end of the two minutes, the simulation was restarted. The subjects’ answers were then compared to what was actually occurring in the simulation at the time of the stop. See Endsley (1988) for a complete description of SAGAT.

...

P e r c e p u A b m . The tests compiled to measure perceptual abilities quantified perceptual vigilance, perceptual speed, encoding speed and pattern recognition. The perceptual speed and encoding speed tasks were adapted from both their psychological paradigms and a version that appears in the BAT (Sperling, 1960; Posner and Mitchell, 1967; Carretta, 1987; Carretta, 1989). The perceptual vigilance task is included in the Jensen, Adrion and Maresh Judgement Evaluation Test (JAMJET) (Jensen, Adrion, Maresh and Weinert, 1987). The pattern recognition test is the Raven’s Advanced Progressive Matrices Test (Raven, Court and Raven, 1985). Coanition. Three tests were selected to measure the cognitive abilities of deductive and inductive reasoning, acquisition of information, information processing, and concept formation. The first test measured analytical reasoning. The second was a risk taking test called Pilot Decisional Attribute Questionnaire. The third test was a rate estimation task that measured internal timing. It is being considered for inclusion in the BAT (Carretta, 1989).

METHOD

Subiects Twenty-one subjects completed the study, which occurred over a two-year period. The subjects were all Northrop employees who had military tactical flying experience. Average age was 44 years. The subjects had logged an average of 3600 military flight hours over an average of 17 years of flying time.

Personalitv Factors. Certain measures of personality were hypothesized to have some relationship to SA, specifically cognitive complexity and field independence, which have been related to problem solving and workload management. Five tests were selected. Two tests measured cognitive complexity: This I Believe Test and the Abstract Orientation Scale (Harvey, 1966; O’Conner, 1971). The other two tests measured locus of control and field independence. Locus of control was measured by the Aviator Locus of Control test which is a modification of the Rotter Internal-External Control Scale included in the JAMJET. Field independence was measured by the Group Embedded Figures Test (GEFT) (Oltman, Raskin and Witkin, 1971). A fifth test designed to measure impulsiveness/decisiveness, called Dot Estimation, was also given (Carretta, 1989).

Procedures The SAGAT measure used for this study was situation awareness of other aircraft and their locations. This represented only a small percentage of the possible SAGAT measures. However, for our purposes, it had the most data available due to the random sampling of the queries and therefore was selected as the dependent variable. The administration of the attribute battery typically took each subject three test sessions of approximately two to three hours each to complete. Tests were given in the same


PROCEEDINGS of the HUMAN FACTORS SOCIETY 35th ANNUAL MEETING-I991

CONCLUSIONS

order for each subject. Once testing was completed, the SAGAT scores were correlated with performance on the SA test battery using a Pearson pairwise correlation matrix. Since missing values existed in the matrix, a pairwise correlation was selected over a listwise correlation. The attribute tests were scored for number correct, response times, and errors (Table 1). Because some attribute tests consisted of more than one subtest, a total of 43 predictor variables were used to determine the correlates of the SA measure.

The sample size was small in this preliminary evaluation. For this reason, the battery will be given to a larger population in the near future. The current evaluation will be used primarily to 1) drop out tests from the battery that have very little or no correlation with SA, and to 2) guide selection of additional tests in areas that correlate with SA. Since tests of perceptual abilities and memory showed a high correlation with the SA measure, tests will be added to the battery to measure different aspects of these attributes.

RESULTS Several tests were dropped from the analysis because no variability existed among the subjects’ performances. These tests included the risk taking task, the Aviator Locus of Control Test, and both measures of cognitive complexity. Fourteen tests remained in the analysis. Correlation coefficients were computed for each predictor variable and the SAGAT measure. The preliminary results from this study indicated that the top five predictors had correlation coefficients (r) between .39 to .72. These correlates were from five different tests (Table 2). As can be seen in the table, the highest correlation was tracking level from the attention sharing task. This was the average level of difficulty that each subject obtained while tracking an object. Difficulty ranged in value from 1 (easiest) to 10 (hardest). The average tracking level during this test was 4.30. These results are preliminary; analysis is continuing.

It is hoped that the complete battery will be fully computerized to aid in both analysis and administration. Future evaluations will lead to the development of instructional methods for enhancing SA and could help to identify the most promising areas for research and development in SA training. Because of the need to develop instructional methods for training the cognitive aspects of flying, this attribute battery holds considerable promise for advancing the field of aircrew training .

TABLE 2

Carretta, T. R. (1989). Personal Communication, Brooks AFB, TX: Manpower and Personnel Division, Air Force Human Resources Laboratory.

REFERENCES Bolstad, C. A. (1990). Evaluation of Pilot Attribute Measures (NOR 90-01). Hawthorne, CA: Northrop Aircraft Division.

TODFive Correlates of Situation Awareness PREDICTOR VARIABLES

r

Attention Sharing Tracking Level

.717

Immediate/ Delayed Memory Total Errors (2-back test)

.547

Encoding Speed Average RT (CategoricalTest)

.547

Carretta, T. R. (1987). Basic Attributes Test (BAT) System: Development of an Automated Test Battery for Pilot Selection (AFHRL-TR-87-9). Brooks AFB, TX: Manpower and Personnel Division, Air Force Human Resources Laboratory. Damos, D. L. (1978). Residual Attention as a Predicto r of Pilot Performance. Human Factors, 20,435-440.

Dot Estimation Number Correct

.415

Ekstrom, R. B., French, J. W., and Harmon, H. H. (1976). Cube Comparison Test ( In The Manual for Kit of Factor Referenced Cognitive Tests). Princeton, NJ: Educational Testing Service.

GEFT (Group Embedded Figures Test) Number Correct

.385


PROCEEDINGS of the HUMAN FACTORS SOCIETY 35th ANNUAL MEETING4991

Endsley, M. R. (1 988). The Functioning and Evaluation of Pilot Situation Awareness (NOR 88-30). Hawthorne, CA: Northrop Aircraft Division.

North, R. A., and Gopher, D. (1976). Measures of Attention as Predictors of Flight Performance. Human Factors, 18, 1-14.

Endsley, M. R. (1987). SAGAT: A Methodology for Measurement of Situation Awareness (NOR 8783). Hawthorne, CA: Northrop Aircraft Division.

O’Conner, J. (1971). Developmental Changes In Abstractness and Moral Reasoning. Unpublished doctoral dissertation, George Peabody College for Teachers, Maryland.

Guilford, J. P. and Lacey, J. 1. (1947). Printed Classification Tests. (AAF Aviation Psychology Program Research Report No5). Washington D.C: U.S. Government Printing Office.

Oltman, P. K., Raskin, E., and Witkin, H. A. (1971). Group Embedded Figures Test (GEFT). Palo Alto, CA: Consulting Psychologists Press. Posner, M.I.,and Mitchell, R. F. (1967). Chronometric Analyses of Classification. Psychological Review, 74, 392-409.

Harvey, 0. J. (1966). System Structure, Flexibility, and Creativity. In 0. J. Harvey (Ed.), Experience, Stnrcture and Adaptability (pp 3965). New Yo&: Springer.

Raven, J. C., Court, J. H., and Raven, J. (1985). Raven’s Advanced Progressive Matrices. London: H. K. Lewis and Co. Ltd.

Jensen, R.S., Adrion, J., Maresh, J., and Weinert, C. (1987). Pilot Judgement Skills Test Baftery (Air Force/ SBIR Phase 1 Final Report). Brooks AFB, TX: USAF School of Aerospace Medicine.

Sperling, G. (1960). The Information Available in Brief Presentations. Psychological Monographs: General and Applied, 74, 1-29.

Lickert, R. and Quasha, W . H. (1969). Revised Minnesota Paper Form Board Test. New York: The Psychological Corporation.
