Theor. Issues in Ergon. Sci., 2003, vol. 4, nos. 1–2, 5–20
Neuroergonomics: research and practice Raja Parasuraman* Cognitive Science Laboratory, Catholic University of America, Washington DC 20064, USA Keywords: Brain and behaviour; cognitive science; neuroscience; ergonomics; human factors. This article describes the characteristics and scope of neuroergonomics, defined as the study of brain and behaviour at work. Neuroergonomics focuses on investigations of the neural bases of mental functions and physical performance in relation to technology, work, leisure, transportation, health care and other settings in the real world. The two major goals of neuroergonomics are to use knowledge of brain function and human performance to design technologies and work environments for safer and more efficient operation, and to advance understanding of brain function underlying real-world human performance. The conceptual, theoretical and philosophical issues at the core of neuroergonomics— that lie at the confluence of cognitive science, neuroscience, and ergonomics—are discussed. Adaptive human–machine systems are then described as an illustration of neuroergonomic research. Several other examples of neuroergonomic research and practice are also described.
1. Introduction 1.1. Neuroergonomics: definitions and scope ‘Neuroergonomics’ is the study of brain and behaviour at work. As the name implies, this emerging area comprises two disciplines that are themselves interdisciplinary, neuroscience and ergonomics. Neuroscience is the study of brain structure and function. Ergonomics (also known as human factors) examines human use of technology at work and in other real-world settings. As the intersection of these two fields, neuroergonomics is concerned both with the brain and with humans at work, but more precisely with their dynamic interaction—with human brains at work (Parasuraman 1998a). Neuroergonomics focuses on investigations of the neural bases of such perceptual and cognitive functions as seeing, hearing, attending, remembering, deciding and planning in relation to technologies and settings in the real world. Because the human brain interacts with the world via a physical body, neuroergonomics is also concerned with the neural basis of physical performance—grasping, moving or lifting objects and one’s limbs. The real-world environments that neuroergonomics deals with are many and diverse. They include, for example: working with computers and various other machines at home, in the workplace, or when engaged in leisure activities, using consumer products and operating vehicles such as aircraft, cars, trains and ships.
* e-mail:
[email protected] Theoretical Issues in Ergonomics Science ISSN 1463–922X print/ISSN 1464–536X online # 2003 Taylor & Francis Ltd http://www.tandf.co.uk/journals DOI 10.1080/14639220210199753
6
R. Parasuraman
1.2. Why neuroergonomics? To answer this query, I hope to show that neuroergonomics provides added value, beyond that available from ‘traditional’ neuroscience and ‘conventional’ ergonomics. The guiding principle of neuroergonomics is that understanding how the brain carries out the complex tasks of everyday life—and not just the simple, artificial tasks of the research laboratory—can provide important benefits for both ergonomics research and practice. An understanding of brain function can lead to the development and refinement of theory in ergonomics, which in turn will promote new, far-reaching types of research. For example, knowledge of how the brain processes visual, auditory and tactile information can provide important guidelines and constraints for theories of information presentation and task design. The basic premise is that the neuroergonomic approach allows the researcher to ask different questions and develop new explanatory frameworks about humans and work than an approach based solely on the measurement of the overt performance or subjective perceptions of the human operator. The added value that neuroergonomics provides is likely to be even greater for work settings such as modern automated systems (Parasuraman and Riley 1997) where measures of overt behaviour can be difficult to obtain (Kramer and Weber 2000). At the practical level, neuroergonomics can lead to the design of more efficient and safer working conditions. These outcomes can provide substantial economic benefits for many. The potential beneficiaries include the developers of technologies, the owners of the systems in which the technologies are used and the work is carried out, and when the systems are widely used by the general public, such as with transportation or health care, society at large. In sum, the basic enterprise of human factors/ergonomics can be considerably enhanced in a fundamental way if we also consider the brain that mediates and makes possible human performance in the real world. Some may see the gaps between brain function, human behaviour, work, technology and society as forbiddingly large. Yet, a rapid expansion of knowledge is bridging the gulfs between these levels of analysis. In pursuing the links between brain function and the world of technology and work, neuroergonomics has two major goals: (1) to use existing and emerging knowledge of human performance and brain function to design technologies and work environments for safer and more efficient operation; and (2) to advance understanding of brain function in relation to human performance in real-world tasks. An example may help to better illustrate the value of this approach. Consider a new traffic monitoring system that is to be installed in the cockpit of a commercial aircraft. The system portrays to the pilot other aircraft that are in the immediate vicinity, showing their speed, altitude, flight path, etc., using colour-coded symbols on a computer display. The design of this system could be informed by various types of neuroergonomic work, both basic and applied. For example, designers may wish to know what features of the symbols (e.g. shape, intensity, motion, etc.) serve to best attract the pilot’s attention to a potential ‘intruder’ in the immediate airspace. At the same time, there may be a concern that the presentation of traffic information, while helping the pilot monitor the immediate airspace, may increase the pilot’s overall mental workload, thereby degrading the performance of the primary flight task. Although subjective or performance measures could be used to evaluate this possibility, a neuroergonomic approach can provide more sensitive evaluation of any impact on flight performance. It may also lead the researcher to ask new and poten-
Neuroergonomics
7
tially more profitable questions about attention allocation than before. Measures of brain function that reflect visual attention and oculomotor control can help determine the impact of the new display on the pilot’s visual scanning and attentional performance. Finally, neuroergonomic evaluation of the manual and physical demands involved in interacting with the information panels and controls of the new traffic monitoring system would also be required for this system to be used effectively and safely by pilots. Several other examples of neuroergonomic research and practice are discussed in a later section of this article. More specifically, three topic areas relevant to this example—assessment of cognitive workload, attention and oculomotor control, and manual performance—are each discussed from a neuroergonomic perspective in other papers in the two special issues on neuroergonomics in this journal, by Just et al. (2003), Kramer and McCarley (2003) and Karwowski, Siemionow and Gielo-Perczak (2003), respectively. 2. Some conceptual, theoretical and philosophical issues in neuroergonomics 2.1. Neuroscience and ergonomics The constituent disciplines of neuroergonomics—neuroscience and ergonomics (or human factors)—are both 20th-century, post-World War II fields. The spectacular rise of neuroscience towards the latter half of that century, and the smaller but no less important growth in ergonomics, can both be linked to technological progress, particularly in digital computers, initiated by engineers and physicists. The brain imaging technologies that have revolutionized modern neuroscience (e.g. functional magnetic resonance imaging) and the sophisticated automated systems that have stimulated much human factors work (e.g. the aircraft flight management system) were both made possible by these engineering developments. Nevertheless, the two fields have developed independently. Traditionally, ergonomics has not paid much attention to neuroscience or to the results of studies concerning brain mechanisms underlying human perceptual, cognitive, affective and motor processes. Many psychologists and most current researchers in ergonomics/human factors ignore the startling discoveries of modern neuroscience. At the same time, neuroscience and its more recent off-shoot, cognitive neuroscience, has only been partially concerned with whether its findings bear any relation to human functioning in real (as opposed to laboratory) settings. The exception concerns applications to the diagnosis and treatment of individuals with neurological and psychiatric disorders. To paraphrase the philosopher Bunge (1980), until recently (i.e. the period up to 1990), psychology (and ergonomics) has been ‘brainless’, whereas neuroscience has been ‘mindless’. Neuroergonomics is a response to this twin disregard. 2.2. Brain, mind and technology The relative neglect by ergonomics of human brain function is understandable given that this discipline had its roots in a psychology of the 1940s that was firmly in the behaviourist camp. More recently, the rise of cognitive psychology in the 1960s influenced human factors, but for the most part neuroscience continued to be ignored, a state of affairs consistent with a functionalist approach to the philosophy of mind (Dennett 1991). Such an approach implies that the characteristics of neural structure and functioning are largely irrelevant to the development of theories of mental functioning. Cognitive psychology (and cognitive science) also went through a functionalist period in the 1970s and 1980s, mainly due to the influence of research-
8
R. Parasuraman
ers from artificial intelligence and computer science. Their rallying cry was that the mind is software. The task of cognitive science, in their view, was to determine the characteristics and rules of the software, irrespective of the hardware that implements the software. Accordingly, the rules of mind could as easily be studied in a computer as they could in a human. The actual hardware, the structures and mechanisms of the brain, were deemed unimportant. In contrast, the recently developed field of cognitive neuroscience proposes that neural structure and function constrain and, in some cases, determine theories of human mental processes (Gazzaniga 2000). The influence of neuroscience has carried over not only to the other cognitive sciences (Pinker 1997), but also to philosophy (Churchland 2002). If neuroscience has freed cognitive science from rigid functionalism, then ergonomics may serve to liberate it from a disembodied existence devoid of context and provide it an anchor in the real world. Even though researchers are aware of the importance of ecological validity, modern cognitive psychology (with a few exceptions) tends to study mental processes in isolation, apart from the artifacts and technologies of the world that require the use of those processes. However, it may be more useful to see technology, particularly computers, as representing an extension of human cognitive capability, a view that emerged as psychology escaped from the grip of behaviourism in the late 1940s (Craik 1947). A modern version of this doctrine is the view espoused by the new field of cognitive engineering that humans and intelligent computer systems constitute ‘joint cognitive systems’ (Roth et al. 1987, Hutchins 1995). Furthermore, much human behaviour is situated and context dependent. Context is often defined and even driven by technological change. How humans design, interact with and use technology—the essence of ergonomics— should, therefore, also be central to cognitive science. The idea that cognition should be considered in relation to action in the world has many antecedents. Piaget’s (1952) work on cognitive development in the infant and its dependence on exploration of the environment anticipated the concept of situated or embodied cognition. In a recent philosophical work, Clark (1997) provides a modern statement of this thesis. Going beyond the old distinction between mind and matter, Clark examines the characteristics of an embodied mind that is shaped by and helps shape action in a physical world. If cognitive science should, therefore, study mind not in isolation but in interaction with the physical world, then it is a natural second step to ask how to design artifacts in the world that best facilitate that interaction. This is the domain of ergonomics. Neuroergonomics goes one, critical, step further. It postulates that the human brain that implements cognition and is itself shaped by the physical environment must also be examined in order to understand fully the inter-relationships of cognition, action and the world of artifacts. Currently, a coherent body of concepts and empirical evidence that constitutes neuroergonomics theory does not exist. Of course, broad theories in the human sciences are also sparse, whether in ergonomics (Hancock and Chignell 1995) or in neuroscience (Albright et al. 2001). What one finds are small-scale theories that could be integrated into a macrotheory but which would still apply only to a specific domain of human functioning. For example, neural theories of attention are becoming increasingly well specified, both at the macroscopic level of large-scale neural networks (Posner and Dehaene 1994, Parasuraman 1998b), at the level of neuronal function (Sarter et al. 2001), and, in the near future, at the more fundamental level of individual genes and their protein products (Fosella et al. 2002, Parasuraman et al. 2002). At the same time, psychological theories of attention have informed human
Neuroergonomics
9
factors research and design (Wickens and Hollands 2000). Difficult though the task may be, one can envisage amalgamation of these respective theories into a neuroergonomic theory of attention. Integration across a broader range of functional domains, however, is as yet premature. 3. Neuroergonomics, psychophysiology and ergonomics 3.1. Brain imaging A core feature of neuroergonomics is an interest in brain mechanisms in relation to human performance at work. To this end, researchers may make use of physiological measures that reflect, more or less directly, aspects of brain function. The most direct of such measures are those derived from the brain itself, as in the electroencephalogram (EEG), which represents the summated electrical activity of populations of neuronal cells as recorded on the scalp, magnetoencephalogaphy (MEG), which consists of the associated magnetic flux that is recorded at the surface of the head, and event-related brain potentials (ERPs) and magnetic fields, which constitute the brain’s specific response to sensory, cognitive and motor events. In addition to these electromagnetic measurements, measures of the brain’s metabolic and vascular responses, e.g. positron emission tomography (PET) and functional magnetic resonance imaging (fMRI), which can be linked to neuronal activity, also provide a noninvasive window on human brain function. Currently, electromagnetic measures such as ERPs provide the best temporal resolution (1 ms or better) for evaluating neural activity in the human brain and metabolic measures such as fMRI have the best spatial resolution (1 cm or better). No single technique combines both high temporal and spatial resolution. Furthermore, some of these techniques (e.g. fMRI) are expensive and restrict participant movement, which makes them difficult to use for neuroergonomic studies. However, new imaging technologies are being developed, such as near infra-red spectroscopy and other forms of optical imaging, that promise to provide high temporal and spatial resolution (Gratton and Fabiani 2001). These techniques have the additional advantage of being more portable and less expensive than fMRI, which will, therefore, add them to the catalogue of available methods that are appropriate for neuroergonomic research. (For a review of brain imaging techniques and their application to psychology, see Cabeza and Kingstone 2001.) These brain imaging technologies can also be complemented by stimulation techniques, such as transcranial magnetic stimulation (TMS), which can be applied to produce transient, reversible disruption of local neuronal function (Pascual-Leone et al. 1999). This allows the investigator to examine the behavioural consequences of TMS applied to specific brain regions (at least of superficial brain structures), thereby permitting the testing of hypotheses linking brain activation to human performance. 3.2. Neuroergonomics and psychophysiology Physiological measures recorded from the body, e.g. heart rate, skin conductance, urinary catecholamines, blood pressure, etc., have also been used in psychological studies and more recently in ergonomic research. The field of psychophysiology is concerned with the use of all such measures in relation to psychological functioning. Moreover, in recent years a ‘Psychophysiology in Ergonomics’ (PIE) interest group has been formed specifically to examine the use of psychophysiological measures in ergonomics (e.g. Boucsein and Backs 2000). One might ask, therefore, why propose
10
R. Parasuraman
neuroergonomics given that this group is already established? After all, the two fields share one goal—the design of safe and efficient technologies and systems for human work. Nevertheless, the two represent distinct endeavours for the following reasons. First, whereas psychophysiology focuses on physiological measures and their psychological correlates, neuroergonomics is more directly concerned with brain function and not with physiological measures per se. The physiological measure is primarily of interest to the extent that it provides an index of neural activity related to the perceptual, cognitive, affective or motor functioning of the human. The measure is used principally to evaluate a theory of brain function or to see how brain function is linked to human performance or use of technology. These characteristics apply to the brain imaging methods discussed previously, so that even if one were to re-label and include fRMI or optical imaging as psychophysiological measures, the distinction between neuroergonomics and psychophysiology would remain. Secondly, if the physiological measure is more ‘distant’ from brain function, as with measures such as heart rate and skin conductance that are typically used in psychophysiological research, then the link to neuroergonomics is less direct or relevant. Thirdly, psychophysiology also has a major focus on autonomic nervous system (ANS) measures in relation to somatic factors, emotion and stress. Although these topics are also of interest to neuroergonomics, especially as they are linked to brain function, a dominant theme in neuroergonomics is neural activity in interaction with the ANS and in relation to human performance. According to this view, stress and emotion are certainly valid topics for neuroergonomics research, but only if they are grounded in theories of brain function. Another major reason for distinguishing neuroergonomics from psychophysiology is that one can easily envisage a neuroergonomic study in which no physiological measure at all is used as a dependent variable, which would rule out a classification of the work as falling within the realm of psychophysiology. Yet, such a study could clearly qualify as a neuroergonomic investigation. Consider the following example. Suppose that as a result of the manipulation of some factor, performance on a target discrimination task (e.g. detection of an ‘intruder’ aircraft in the cockpit traffic monitoring example discussed previously) in which location cues are provided prior to the target yields the following results: reaction time (RT) to the target when preceded by an invalid location cue is disproportionately increased while that to a valid cue is not. This might happen, for example, if the cue is derived from the output of an automated detection system, which is not perfectly reliable (Hitchcock et al. 2003). In simple laboratory tasks using such a cueing procedure, there is good evidence linking this performance pattern to a basic attentional operation and to activation of a specific distributed network of cortical and sub-cortical regions—on the basis of previous research using non-invasive brain imaging in humans, invasive recordings in animals and performance data from individuals who have suffered damage to these brain regions (Posner and Dehaene 1994). One could then conduct a study using the same cueing procedure and performance measures as a behavioural ‘assay’ of the activation of the neural network in relation to performance of a more complex task in which the same basic cognitive operation is used. (See Parasuraman et al. (2002) for a discussion of the logic of behavioural assays of brain function as used in a quite different area of study—the genetic contribution to individual differences in cognition.) If the characteristic performance pattern was observed—a disproportionate increase in RT following an invalid location cue, with a normal decrease in RT following a valid cue, then one could argue
Neuroergonomics
11
that the distributed cortical/sub-cortical network of brain regions is likely to have been involved in task performance. This would then enable the researcher to link the full body of neuroscience work on this aspect of attentional function to performance on the complex intruder detection task. Thus, even though no physiological index was used and although the same performance measure (RT) was used as in a traditional ergonomic analysis, the type of question asked and the explanatory framework can be quite different in the neuroergonomic approach. Finally, a neuroergonomic study could also involve a computational analysis of brain or cognitive function underlying performance of a complex task. So long as the analysis was theoretically driven and linked to brain function, the study would qualify as neuroergonomic, even though no physiological index was used. Several computational models of human performance have been developed for use in human factors (Pew and Mavor 1998). Of these, models that can be linked, in principle, to brain function, such as neural network (connectionist) models (O’Reilly and Munakata 2000) would be of relevance to neuroergonomics. In summary, neuroergonomics and psychophysiology in ergonomics share a common goal of seeking the design of safe and efficient human–machine systems. Nevertheless, there are several reasons why neuroergonomics and psychophysiology can be differentiated. Consequently, the two can be considered complementary, overlapping approaches. 4. Adaptive systems: an avenue for neuroergonomic research and practice 4.1. Adaptive automation The goal of neuroergonomics—designing safe, efficient and usable systems—requires synergy between human and machine, but sometimes this goal is not met. Automated systems provide a good example of the mismatch that can occur between human and machine. A common form of automation is the provision of computer support to the human operator. Alternatively, automation may refer to allocation of a function previously carried out by humans to a computer. These and other forms of automation have yielded several benefits, in terms of improved capacity, efficiency and safety (Parasuraman and Mouloua 1996). At the same time, problems in human–automation interaction have also been well documented, both in the laboratory and the field (Wiener and Curry 1980, Bainbridge 1983, Rasmussen 1986, Sheridan 1992, Billings 1997, Parasuraman and Riley 1997, Sarter et al. 1997, Lewis 1998, Satchell 1998, Parasuraman et al. 2000). In some instances, these performance costs may be severe enough to outweigh the benefits of automation. As a result, there has been considerable interest in developing alternative approaches to the design and implementation of automation. Adaptive human– machine systems, or adaptive automation, represents one such alternative approach. In adaptive systems, the ‘division of labour’ between human and machine agents is not fixed and pre-specified during the design and development of the system, but dynamic and variable during the operation of the system. Computer aiding of the human operator and task allocation between the operator and computer systems are flexible and context-dependent (Hancock et al. 1985, Rouse 1988, Parasuraman et al. 1992, Scerbo 1996, 2001). In contrast, in static automation, provision of computer aiding is pre-determined at the design stage, and task allocation is fixed. One of the problems that has been observed when human operators work with automated systems is that, although automation is often introduced in an attempt to reduce operator mental workload, it may not do so, or may even increase workload at
12
R. Parasuraman
times (Wiener 1988, Parasuraman and Riley 1997). This suggests the need for linking the provision of automated support to the level of operator workload or performance. Automated support is needed only when the operator’s mental workload is relatively high, so that the operator is likely to benefit by the freeing up of cognitive resources that automation provides. At other times, automated support need not be given, or, if operator workload is too low and there is a danger of the operator becoming disengaged from the system, a task previously carried out by automation can be re-allocated to the operator. Parasuraman et al. (1996) showed that such temporary re-allocation of an automated task to human control promotes better monitoring of the automation and improved fault detection when anomalies occur. Either of these adaptive strategies—automated support when workload is high, or task re-allocation when it is low—have been found to benefit performance when their implementation is closely matched to operator workload (Parasuraman et al. 1999). However, accomplishing such workload-matched adaptation is dependent on the ability to measure the operator’s mental workload in real time. Although mental workload can be measured using a number of different techniques (Byrne and Parasuraman 1996), measures of brain function offer particular benefits for use in adaptive systems (Scerbo et al. 2001, Parasuraman and Caggiano 2002). First, measures of cognitive-related brain activity, unlike most behavioural measures (with the exception of continuous motor tasks) can be obtained continuously. In many systems where the operator is placed in a supervisory role over automated subsystems, very few overt responses (e.g. button presses or cursor movements) may be made even as the operator is engaged in considerable cognitive activity. In such a situation, a behavioural measure (e.g. RT or cursor movement error) provides a relatively impoverished sample of the mental activity of the operator. Physiological measures, on the other hand, may be recorded continuously without respect to overt responses and may provide a measure of the covert activities of the human operator. Moreover, if the measures can be directly linked to brain activity, as neuroergonomic measures should, then sensitive, real-time monitoring of cognitive workload may be possible. In other words, brain measures have a higher bandwidth than behavioural or performance measures. As discussed previously, therefore, automated systems in which little overt behaviour can be recorded provide a particularly compelling case for the neuroergonomic approach (see also Kramer and Weber 2000). Secondly, in some instances, measures of brain function may provide more information when coupled with behavioural measures than behavioural measures alone. For example, changes in RT may reflect contributions of both central processing (working memory) and response-related processing to workload. However, when coupled with the amplitude and latency of the P300 component of the ERP, such changes may be more precisely localized to central processing stages than to response-related processing (Donchin et al. 1986, Wickens 1990, Kramer and Weber 2000). In addition, measures of brain function can indicate not only when an operator is overloaded, drowsy, or fatigued, but also which brain networks and circuits may be affected. In short, neuroergonomic measures offer new avenues for adaptive interventions aimed at enhancing system performance. 4.2. Brain work and mental work What measures of brain activity can be used to assess mental workload? There are a number of potential candidates. All stem from the notion of mental workload as
Neuroergonomics
13
reflecting how hard one’s mind is working at any given moment. Accordingly, mental workload can be associated with brain work. Various candidate measures for brain work have been proposed over the years, all of which owe their existence to the brilliant early investigations of Sherrington, who studied the regulation of the blood supply of the brain (Roy and Sherrington 1890). Sherrington demonstrated that there is a close coupling between the electrical activity of neuronal cells, the energy demands of the associated cellular processes and regional blood flow in the brain. His pioneering investigations suggested that, if mental activity results in increased neuronal response in localized regions of the brain, then in principle it should be possible to measure mental workload by assessing regional cerebral metabolism and blood flow. It took several years before sensitive techniques were developed for measuring regional brain blood flow in humans. An early development was the invention of the Xenon-133 (Xe-133) method for assessing regional cortical changes in brain blood flow and glucose metabolism. Injection in human patient volunteers of the radioactively tagged Xenon gas, which passes freely across the blood–brain barrier, showed that performance of various mental tasks led to increased metabolic activity in specific cortical regions. However, the Xe-133 technique was too invasive to be used routinely in normal human participants. The development of PET paved the way for less invasive measurement of regional cerebral metabolism and blood flow. Regional cerebral glucose metabolism can be non-invasively determined using PET and radioactively labelled glucose (18-fluoro-deoxyglucose), while regional cerebral blood flow may be assessed with PET and radioactively-labelled oxygen (O-15) in water. A disadvantage of PET is the need for ionizing radiation, which, although safe when used within exposure limits, prevents frequent use in studies with normal human participants. The recent development of fMRI has overcome this limitation. fMRI provides non-invasive, high-resolution assessments of regional cerebral blood flow that can be repeatedly obtained in the same participant. There is now an emerging literature indicating that different PET and fMRI measures of brain activity, as well as electromagnetic measures such as EEG and ERPs, provide sensitive indexes of moment-to-moment variations in mental workload (Parasuraman and Caggiano 2002). The potential use of these and other physiological measures for real-time assessment of mental workload in adaptive human– machine systems has also been documented (Parasuraman 1990, Kramer et al. 1996, Scerbo et al. 2001). Prinzel et al. (2000) have also specifically demonstrated the feasibility of an adaptive system based on EEG measures (see also the article by Scerbo et al. (2003) in the second special issue on neuroergonomics in this journal). Technical developments will determine the further utility of measures of brain function in neuroergonomic research on mental workload and adaptive systems. For example, PET and fMRI are currently expensive to acquire and to operate, cumbersome, unduly restrictive of participant movement, and are not portable. These characteristics limit their routine use in neuroergonomic studies. Despite the limitations, Peres et al. (2000) recently reported an fMRI study examining changes in regional cerebral blood flow in pilots performing a flight control task. Although it has much poorer spatial resolution than PET or fMRI, transcranial Doppler sonography (TCD) has high temporal resolution, thus allowing for continuous, real-time monitoring of cerebral blood flow, which facilitates neuroergonomic applications (see the TCD study by Hitchcock et al. (2003) in this special issue). The newer optical imaging technologies that are currently being developed will also be cheaper and
14
R. Parasuraman
more portable (Gratton and Fabiani 2001) and will lead to more ergonomic applications. 5. Other examples of neuroergonomics research and practice The explosive growth of neuroscience has provided a wealth of other opportunities for research in neuroergonomics. A number of different investigations that fall within the boundaries of neuroergonomics as defined in this article have been carried out. The research conducted to date can be characterized as broad, diverse and not yet forming a coherent, established body of work. However, this situation is likely to change as the field develops and more neuroergonomic studies are conducted in the future. Some examples are briefly described here. 5.1. Human error A potentially fertile area for neuroergonomic research is in the analysis and possible prediction of human error. Cognitive scientists and human factors analysts have proposed many different approaches to the classification, description and explanation of human error in complex human–machine systems (Norman and Shallice 1986, Reason 1990, Senders and Moray 1991). Analysis of brain activity associated with errors can help refine these taxonomies, particularly in leading to testable hypotheses concerning the elicitation of error. It is reasonable to hypothesize that, whenever an individual commits an error, a particular neural mechanism is activated. A specific ERP component associated with errors has been identified, the error-related negativity (ERN). The ERN, which has a fronto-central distribution over the scalp, reaches a peak 100–150 ms after the onset of the erroneous response (as revealed by measures of electromyographic activity), and is smaller or absent following a correct response (Gerhring et al. 1993). The ERN amplitude is related to perceived accuracy, or the extent to which participants are aware of their errors (Scheffers and Coles 2000). Importantly, the ERN seems to reflect central mechanisms and is relatively independent of output modality. Holroyd et al. (1998) found that errors made in a choice reaction time task in which either the hand or the foot was used to respond led to nearly identical ERN. The relevance of ERN to neuroergonomic research and applications is straightforward. The ERN allows identification, prediction and perhaps prevention of operator errors in real time. For example, the ERN could be used to identify the human operator tendency to either commit, recognize or correct an error. This could potentially be detected covertly by on-line measurement of ERN, prior to the actual occurrence of the error, given that the ERN could be reliably measured on a single trial. Theoretically, a system could be activated by an ERN detector in order to either take control of the situation (for example in those cases where time to act is an issue), or notifying the operator about the error he/she committed, even providing an adaptive interface which selectively presents the critical sub-systems or function. Such a system would have the advantage of keeping the operator still in control of the entire system, while providing an anchor for troubleshooting when the error actually occurs (and having the possibility, for the system, to correct it by itself if needed). 5.2. Learning and skill acquisition Learning represents an area that may be particularly well suited to neuroergonomic studies. As people acquire a skill, the associated brain functional changes that occur
Neuroergonomics
15
can now be monitored concurrently to address various theoretical and practical issues in a more direct way than is possible using only behavioural (performance) measures. For example, procedural learning is generally thought to progress through a series of stages, with the final stage representing full acquisition and the development of a highly-skilled, almost ‘automatic’ stage (Newell and Rosenbloom 1980). The characteristics of these stages can be better understood through functional brain imaging studies. Understanding the brain changes that accompany stages of learning could lead to the development of better training procedures. Conversely, the atrophy of skills—something that has been attributed to particular forms of high-level automation (Wiener 1988)—could also be monitored and assessed. A study by Krebs et al. (1998) provides a good recent example of a neuroergonomic analysis of learning of a complex task, in a novel integration of robotic technology with brain imaging. PET was used to monitor brain activity associated with participants learning to operate a tele-robotic arm. Krebs et al. found different patterns of brain activation early and late in learning of this motor task. They concluded that motor learning of this type involves at least two stages, an early cortico-striatal network and a later cortical-cerebellar loop. The Krebs et al. study is important for being one of the first studies to integrate functional brain imaging and an important area in ergonomics research—telerobotic operation. Nevertheless, because this study used PET, it has the disadvantage that learning could only be studied over a relatively short period. In contrast to PET, which can only be used infrequently on the same participant because of the use of ionizing radiation, fMRI can be administered repeatedly at frequent intervals, making it well suited to studies of learning. Using fMRI to examine acquisition of skilled motor skills, Karni et al. (1998) also found evidence for multiple stages of learning, characterized by different time scales, gains and patterns of activity in primary motor cortex. An important feature of this study was that experience-related brain changes were measured over an impressively long period of training, 3 weeks. In principle, fMRI studies could be conducted for even longer periods of time, which may be required in studying the acquisition (and retention) of skill in more complex tasks representative of modern human–machine systems. 5.3. Other applications Applications can complement the examples of neuroergonomic research that have been discussed so far. Such development-oriented work in neuroergonomics is in its early phases. Although it is premature at this point in time to speak of ‘neuroergonomic practice’, practical applications are around the corner and are likely to grow in number as the field matures. Some examples are briefly discussed here. As discussed previously, EEG and ERPs provide continuous measures of brain electrical activity that can be used to index mental workload for potential use in adaptive systems. A modification of this procedure is to use brain potentials to directly control external devices by the physically handicapped (Farwell and Donchin 1988), including individuals with little or no motor function (Pfurtscheller et al. 2000). As EEG technology continues to become cheaper, smaller, portable and robust applications such as these will increase. Applying knowledge of brain and cognitive architectures could be used to produce ‘neural chips’ (as opposed to traditional VLSI architectures) and intelligent user interfaces with exceptionally fast computing systems. These could then be used to develop ‘neuroergonomic aids’ for the physically incapacitated.
16
R. Parasuraman
Such applications are likely to be influenced by further basic research on the neural control of motor function. A recent example of ground-breaking research is that of Reger et al. (2000), who described a hybrid sensory-motor system involving the brain of a lamprey coupled to a small mobile robot. The brain–robot coupling formed a closed-loop sensory-motor system whose primary research purpose was to provide a platform for validating computational models of neural function. However, such a system or closely-related ones could also be used to test hypotheses concerning perception–action loops in human performance (Hancock and Chignell 1995). Understanding the mechanisms of spatial navigation, both in real and virtual environments, is another example of an area of neuroergonomic research where applications may flourish. Early data from animal studies and from brain damaged individuals have long pointed to the role of the hippocampal formation in the ability to navigate through space, suggesting that the hippocampus provides a spatiallyreferenced ‘cognitive map’ (O’Keefe and Nadel 1978). More recently, functional brain imaging studies have also implicated this brain region in the recall of wellestablished topographical maps. Maguire et al. (1996) studied London taxicab drivers, who typically train for as many as 3 years to acquire the route knowledge of London streets to pass their stringent licensing examinations. PET revealed that the right hippocampus was critically involved in the recall of specific routes around London in this expert group. In subsequent studies by this group using a virtual reality (VR) environment, right hippocampal activation was also associated with navigating accurately between locations in a complex town (Maguire et al. 1998, Pine et al. 2002). These neuroergonomic studies have important implications for further understanding of the mechanisms of spatial navigation and its acquisition in other expert groups such as pilots and controllers. Further studies in virtual environments could also lead to the development of enhanced VR systems. These examples by no means exhaust the range of possibilities. Rather, the types of practical application are limited only by the imagination of researchers and practitioners in the field.
6. Conclusions Neuroergonomics represents a deliberate merger of neuroscience and ergonomics with the goal of advancing understanding of brain function underlying human performance of complex, real-world tasks. A second major goal is to use existing and emerging knowledge of human performance and brain function to design technologies and work environments for safer and more efficient operation. More progress has been made on the first goal than on the second, but both neuroergonomic research and practice should flourish in the future, as the value of the approach is appreciated. The basic enterprise of ergonomics—how humans design, interact with and use technology—can be considerably enriched if we also consider the human brain that makes such activities possible. References Albright, T. D., Jessell, T. M., Kandel, E. R. and Posner M. I. 2001, Progress in the neural sciences in the century after Cajal (and the mysteries that remain), Annals of the New York Academy of Sciences, 929, 11–40. Bainbridge, L. 1983, Ironies of automation, Automatica, 19, 775–779.
Neuroergonomics
17
Billings, C. E. 1997, Aviation Automation: The Search for a Human-Centered Approach (Mahwah, NJ: Erlbaum). Boucsein, W. and Backs, R. W. 2000, Engineering psychophysiology as a discipline: historical and theoretical aspects, in R. W. Backs and W. Boucsein (eds), Engineering Psychophysiology: Issues and Applications (Mahwah, NJ: Erlbaum), 3–30. Bunge, M. 1980, From mindless neuroscience to brainless psychology to neuropsychology, Paper presented at the Annual Winter Conference for Brain Research, January, Keystone, CO. Byrne, E. A. and Parasuraman, R. 1996, Psychophysiology and adaptive automation, Biological Psychology, 42, 249–268. Cabeza, R. and Kingstone, A. 2001, Handbook of Functional Neuroimaging of Cognition (Cambridge, MA: MIT Press). Churchland, P. S. 2002, Self-representation in nervous systems, Science, 296, 308–310. Clark, A. 1997, Being There: Putting Brain, Body, and the World Together Again (Cambridge, MA: MIT Press). Craik, K. 1947, Theory of the human operator in control systems, British Journal of Psychology, 38, 56–61. Dennett, D. C. 1991, Consciousness Explained (Boston: Little Brown). Donchin, E., Kramer, A. F. and Wickens, C. 1986, Applications of brain event-related potentials to problems in engineering psychology, in M. G. H. Coles, E. Donchin and S. Porges (eds), Psychophysiology: Systems, Processes and Applications (Middletown, NJ: Till and Till), 702–718. Farwell, L. A. and Donchin, E. 1988, Talking off the top of your head: toward a mental prosthesis utilizing event-related potentials, Electroencephalography and Clinical Neurophysiology, 70, 510–523. Fosella, J., Posner, M. I., Fan, J., Swanson, J. M. and Pfaff, D. W. 2002, Attentional phenotypes for the analysis of higher mental function, The Scientific World Journal, 2, 217–233. Gazzaniga, M. S. 2000, The New Cognitive Neurosciences (Cambridge, MA: MIT Press). Gehring, W.J., Goss, B., Coles, M. G. H., Meyer, D. E. and Donchin, E. 1993, A neural system for error detection and compensation, Psychological Science, 4, 385–390. Gratton, G. and Fabiani, M. 2001, Sheddding light on brain function: the event-related optical signal, Trends in Cognitive Sciences, 5, 357–363. Hancock, P. A. and Chignell, M. H. 1995, On human factors, in J. Flach, P. Hancock, J. Caird and K. Vicente (eds), Global Perspectives on the Ecology of Human–Machine Systems (Mahwah, NJ: Erlbaum), 14–53. Hancock, P. A., Chignell, M. H. and Lowenthal, A. 1985, An adaptive human–machine system, Proceedings of the IEEE Conference on Systems, Man and Cybernetics, 15, 627– 629. Hitchcock, E. M., Warm, J. S., Matthews, G., Dember, W. N., Shear, P. K., Tripp, L. D., Mayleben, D. W., Rosa, R. R. and Parasuraman, R. 2003, Automation cueing modulates cerebral blood flow and vigilance in a simulated air traffic control task, Theoretical Issues in Ergonomics Science, 4, 89–112. Holroyd, C. B., Dien, J. and Coles, M. G. H. 1998, Error-related scalp potentials elicited by hand and foot movements: evidence for an output-independent error-processing system in humans, Neuroscience Letters, 242, 65–68. Hutchins, E. 1995, Cognition in the Wild (Cambridge, MA: MIT Press). Just, M. A., Carpenter, P. A. and Miyake, A. 2003, Neuroindices of cognitive workload: neuroimaging, pupillometric and event-related potential studies of brain work, Theoretical Issues in Ergonomics Science, 4, 56–88. Karni, A., Meyer, G., Rey-Hopolito, C., Jezzard, P., Adams, M. M., Turner, R. and Ungerleider, L. G. 1998, The acquisition of skilled motor performance: fast and slow experience-driven changes in primary motor cortex, Proceedings of the National Academy of Sciences (USA), 95, 861–868. Karwowski, W., Siemionow, W. and Gielo-Perczak, K. 2003, Physical neuroergonomics: the human brain in control of physical work activities. Theoretical Issues in Ergonomics Science, 4, 175–199.
18
R. Parasuraman
Kramer, A. F. and McCarley, J. S. 2003, Oculomotor behavior as a reflection of attention and memory processes: neural mechanisms and applications to human factors, Theoretical Issues in Ergonomics Science, 4, 21–55. Kramer, A. F. and Weber, T. 2000, Applications of psychophysiology to human factors, in J. T. Cacioppo, L. G. Tassinary and G. G. Berntson (eds), Handbook of Psychophysiology, 2nd edn (New York: Cambridge University Press), 794–814. Kramer, A. F., Trejo, L. J. and Humphrey, D. G. 1996, Psychophysiological measures of workload: potential applications to adaptively automated systems, in R. Parasuraman and M. Mouloua (eds), Automation and Human Performance: Theory and Applications (Mahwah, NJ: Lawrence Erlbaum Associates), 137–162 Krebs, H., Brashers-Krug, T., Rauch, S., Savage, C. R., Hogan, N., Rubin, R. H., Fischman, A. J. and Alpert, N. M. 1998, Robot-aided functional imaging: application to a motor learning study, Human Brain Mapping, 6, 59–72. Lewis, M. 1998, Designing for human-agent interaction, Artificial Intelligence Magazine, Summer (Washington DC: American Association for Artificial Intelligence), 67–78. Maguire, E. A., Burgess, N., Donnett, J. G., Frackowiak, R. S., Frith, C. D. and O’Keefe, J. 1998, Knowing where and getting there: a human navigation network, Science, 280, 921–924. Maguire, E. A., Frackowiak, R. S. J. and Frith, C. D. 1996, Recalling routes around London: activation of the right hippocampus in taxi drivers, Journal of Neuroscience, 17, 7103–7110. Newell, A. and Rosenbloom, P. S. 1980, Mechanisms of skill acquisition and the law of practice, in J. R. Anderson (ed.), Cognitive Skills and their Acquisition (Hillsdale, NJ: Erlbaum) 1–55. Norman, D. A. and Shallice, T. 1986, Attention to action: willed and automatic control of behavior, in R. J. Davidson, G. E. Schwartz and D. Shapiro (eds), Consciousness and Self-Regulation (New York: Plenum Press), 1-18. O’Keefe, J. and Nadel, L. 1978, The Hippocampus as a Cognitive Map (New York: Plenum). O’Reilly, R. C. and Munakata, Y. 2000, Computational Explorations in Cognitive Neuroscience (Cambridge, MA: MIT Press). Parasuraman, R. 1990, Event-related brain potentials and human factors research, in J. W. Rohrbaugh, R. Parasuraman and R. Johnson (eds), Event-related Brain Potentials: Basic and Applied Issues (New York: Oxford University Press), 279–300. Parasuraman, R. 1998a, Neuroergonomics: The Study of Brain and Behavior at Work, http://arts-sciences.cua.edu/csl/neuroerg.htm. Parasuraman, R. 1998b, The Attentive Brain (Cambridge, MA: MIT Press). Parasuraman, R. and Caggiano, D. 2002, Mental workload, in V. S. Ramachandran (ed.), Encyclopedia of the Human Brain Volume 3 (San Diego: Academic Press), 17–27. Parasuraman, R. and Mouloua, M. 1996, Automation and Human Performance: Theory and Applications (Mahwah NJ: Erlbaum). Parasuraman, R. and Riley, V. A. 1997, Humans and automation: use, misuse, disuse, abuse, Human Factors, 39, 230–253. Parasuraman, R., Bahri, T., Deaton, J., Morrison, J. and Barnes, M. 1992, Theory and design of adaptive automation in aviation systems, Progress Report No. NAWCADWAR-92033-60 (Warminster, PA: Naval Air Warfare Center). Parasuraman, R., Greenwood, P. M. and Sunderland, T. 2002, The apolipoprotein E gene, attention, and brain function, Neuropsychology, 16, 254–274. Parasuraman, R., Mouloua, M. and Hilburn, B. 1999, Adaptive aiding and adaptive task allocation enhance human–machine interaction, in M. Scerbo and M. Mouloua (eds), Automation Technology and Human Performance: Current Research and Trends (Mahwah, NJ: Erlbaum), 119–123. Parasuraman, R., Mouloua, M. and Molloy, R. 1996, Effects of adaptive task allocation on monitoring of automated systems, Human Factors, 38, 665–679. Parasuraman, R., Sheridan, T. B. and Wickens, C. D. 2000, A model for types and levels of human interaction with automation, IEEE Transactions on Systems, Man, and Cybernetics, 30, 286–297. Pascual-Leone, A., Bartres-Faz, D. and Keenan, J. P. 1999, Transcranial magnetic stimulation: studying the brain-behavior relationship by induction of ‘virtual lesions’,
Neuroergonomics
19
Philosophical Transactions of the Royal Society of London, Biological Sciences, 354, 1129–1238. Pe¤ re' s, M., Van De Moortele, P. F. and Pierard, C. 2000, Functional magnetic resonance imaging of mental strategy in a simulated aviation performance task, Aviation, Space and Environmental Medicine, 71, 1218–1231. Pew, R. and Mavor, A. 1998, Modeling Human and Organizational Behaviour: Application to Military Simulations (Washington D.C.: National Academy Press). Pfurtscheller, G., Guger, C., Muller, G., Krausz, G. and Neurper, C. 2000, Brain oscillations control hand orthosis in a tetraplegic, Neuroscience Letters, 292, 211–214. Piaget, J. 1952, The Origins of Intelligence in Children (New York: Longman). Pine, D. S., Grun, J., Maguire, E. A., Burgess, N., Zarahn, E., Koda, V., Fyer, A., Szezko, P. R. and Bilder, R. M. 2002, Neurodevelopmental aspects of spatial navigation: a virtual reality fMRI study, Neuroimage, 15, 396–406. Pinker, S. 1997, How the Mind Works (New York: Norton). Posner, M. I. and Dehaene, S. 1994, Attentional networks, Trends in Neuroscience, 17, 75– 79. Prinzel, L. J., Freeman, F. G., Scerbo, M. W., Mikulka, P. J. and Pope, A. T. 2000, A closed-loop system for examining psychophysiological measures for adaptive automation, International Journal of Aviation Psychology, 10, 393–410. Rasmussen, J. 1986, Information Processing and Human–Machine Interaction (Amsterdam: North-Holland). Reason, J. 1990, Human Error (New York: Oxford University Press). Reger, B. D., Fleming, K. M., Sanguineti, V., Alford, S. and Musaa-Ivaldi, F. A. 2000, Connecting brains to robots: an artificial body for studying the computational properties of neural tissues, Artificial Life, 6, 307–324. Roth, E. M., Bennett, K. B. and Woods, D. D. 1987, Human interaction with an ‘intelligent’ machine, International Journal of Man–Machine Studies, 27, 479–525. Rouse, W. B. 1988, Adaptive aiding for human/computer control, Human Factors, 30, 431– 438. Roy, C. S. and Sherrington, C. S. 1890, On the regulation of the blood supply of the brain, Journal of Physiology (London), 11, 85–108. Sarter, M., Givens, B. and Bruno, J. P. 2001, The cognitive neuroscience of sustained attention: where top-down meets bottom-up, Brain Research Reviews, 35, 146–160. Sarter, N., Woods, D. D. and Billings, C. E. 1997, Automation surprises, in G. Salvendy (ed.), Handbook of Human Factors and Ergonomics, 2nd edn (New York: Wiley), 1926– 1943. Satchell, P. 1998, Innovation and Automation (Aldershot, UK: Ashgate). Scerbo, M. W. 1996, Theoretical perspectives on adaptive automation, in R. Parasuraman and M. Mouloua (eds), Automation and Human Performance: Theory and Applications (Mahwah, NJ: Erlbaum), 37–63. Scerbo, M. W. 2001, Adaptive automation, in W. Karwowski (ed.), International Encyclopedia of Ergonomics and Human Factors (London: Taylor and Francis), 1077– 1079. Scerbo, M. W., Freeman, F. G., Mikulka, P. J., Parasuraman, R., Di Nocera, F. and Prinzel, L. J. 2001, The efficacy of psychophysiological measures for implementing adaptive technology, NASA TP-2001-211018 (Hampton, VA: NASA Langley Research Center). Scerbo, M. W., Freeman, F. G. and Mikulka, P. J. 2003, A brain-based system for adaptive automation, Theoretical Issues in Ergonomics Science, 4, 200–219. Scheffers, M. K. and Coles, M. G. H. 2000, Performance monitoring in a confusing world: error-related brain activity, judgements of response accuracy, and types of errors, Journal of Experimental Psychology: Human Perception and Performance, 26, 141–151. Senders, J. W. and Moray, N. 1991, Human Error: Cause, Prediction, Reduction (Hillsdale, NJ: Erlbaum). Sheridan, T. B. 1992, Telerobotics, Automation, and Supervisory Control (Cambridge, MA: MIT Press). Wickens, C. D. 1990, Applications of event-related potential research to problems in human factors, in J. W. Rohrbaugh, R. Parasuraman, and R. Johnson (eds), Event-related
20
R. Parasuraman
Brain Potentials: Basic and Applied Issues (New York: Oxford University Press), 301– 309. Wickens, C. D. and Hollands, J. G. 2000, Engineering Psychology and Human Performance (New York, NY: Longman). Wiener, E. L. 1988, Cockpit automation, in E. L. Wiener and D. C. Nagel (eds), Human Factors in Aviation (San Diego, CA: Academic Press), 433–461. Wiener, E. L. and Curry, R. E. 1980. Flight-deck automation: promises and problems, Ergonomics, 23, 995–1011.
About the author Raja Parasuraman, PhD, is Director of the Cognitive Science Laboratory and Professor of Psychology at The Catholic University of America in Washington, DC. He received a BSc (1st Class Honours) in Electrical Engineering from Imperial College, University of London, UK (1972), an MSc in Applied Psychology (1973) and a PhD in Psychology from the University of Aston, Birmingham, UK (1976). He has carried out research on attention, automation, air traffic control, ageing and Alzheimer’s disease, event-related brain potentials, functional brain imaging, signal detection, vigilance and workload. He is a Fellow of the American Association for the Advancement of Science (1994), the American Psychological Association (1991) and the Human Factors and Ergonomics Society (1994). He is also currently serving as Chair of the Committee on Human Factors of the National Research Council, National Academy of Sciences.
