Extending Ecological Interface Design to Auditory Displays - CiteSeerX

Sanderson, P. M., Anderson, J. & Watson, M. (2000). Extending Ecological Interface Design to Auditory Displays. Proceedings of the 2000 Annual Conference of the Computer-Human Interaction Special Interest Group (CHISIG) of the Ergonomics Society of Australia (OzCHI2000). CSIRO: Sydney. Pp 259-266

Extending Ecological Interface Design to Auditory Displays Penelope Sanderson, Janet Anderson, and Marcus Watson Swinburne Computer-Human Interaction Laboratory (SCHIL) Swinburne University of Technology {penny}{janet}{marcus}@it.swin.edu.au

Abstract We have proposed an extension of Ecological Interface Design (EID) to encompass the design of auditory displays. Our analysis shows that EID can succeed in this task if all stages of Cognitive Work Analysis (CWA) are employed and if an extra “attentional mapping” stage is added, which maps the attentional need of the human operators. A review of auditory perception and attention suggests that the critical design problem is how to facilitate the movement of sound into focal awareness when the system is abnormal and out of focal awareness when the system is normal. Current knowledge of the shift in auditory attention is limited but developing in the sonification community.

1

Introduction

The design of effective auditory displays that support the human operators of complex systems has received substantially less attention in the human factors literature than the design of visual displays. For example, the HCI Handbook only allocates one chapter out of 62 to auditory displays. (Helander et al. 1997). Auditory displays are most often used as an adjunct to visual displays in environments where there is a heavy load on operator’s cognitive resources. (Woods, 1995). The challenges involved in designing auditory displays in such environments are substantial because of the complex interactions that arise between tasks being processed in different modalities (Wickens & Hollands, 2000). For example, questions that arise are which tasks are best suited to the visual and auditory modalities, what kind of data sets might be better suited to visual or auditory display, and how can task sequencing and attentional load affect design decisions. In addition, the perceptual processing of audition and vision differ fundamentally and there is no evidence that guidelines developed for the design of visual displays can be simply transferred to auditory displays. Ecological Interface Design (EID) presents guidelines for the development of visual displays of which a key

component is the mapping of real world properties to the interface. (Dinadis & Vicente, 1996, Burns & Vicente, 1996). However, mapping of data relationships in the auditory domain is not as intuitive or as obvious as it is in the visual domain. Even basic questions remain; how to identify the auditory dimension that best represents a data dimension, how to capture changes in data by a direction change in the auditory dimension, and how to determine the scaling factor that should be used (Walker, Kramer & Lane, 2000). This paper aims to apply the principles of EID to the design of auditory displays. EID has, until now, emphasised the design of visual interfaces and consideration of what visual form the data should take (Vicente & Rasmussen, 1990). The presentation of data to other sensors (auditory, vestibular, haptic and olfactory) has not been addressed. The relative neglect of other sensory modalities in EID is puzzling in view of the emphasis in EID on representing the world in a way that matches human perceptual processes (Vicente, 1999). Humans have evolved to process information in all modalities because it is adaptive to apprehend the world in multiple ways. In this paper we focus on auditory displays but the argument presented applies equally to other modalities. In applying EID to the design of auditory displays, we address three questions: 1. How can the principles of EID clarify when to present information visually or auditorily? 2. Is EID an adequate theoretical framework for guiding the design of auditory displays, or does it need to be extended? 3. Do we have the necessary knowledge about auditory processes to guide the design of auditory displays?

2

Why use auditory displays?

There would be few complex control systems that do not use auditory support somewhere in the design. Control systems still rely heavily on visual displays to convey information, limiting most auditory displays to the role of directing attention, usually in the form of alarms. There are three areas where auditory displays should be considered and may offer an advantage over visual displays, either alone or in combination with visual displays.

2.1

Vigilance tasks

Vigilance tasks are characterised by extended periods of monitoring requiring high levels of operator vigilance. The incidence of abnormal events is usually low but if missed can lead to the evolution of a major incident (Woods, 1995). Low cognitive load tasks such as monitoring in power stations, aircraft cockpits and medical critical care environments, all use an extensive range of auditory displays (Stanton, 1994). The majority of these systems are auditory alarms designed to direct the operator’s attention to visual monitors.

2.2

High cognitive load tasks

High cognitive load tasks are characterised by abnormal states of the system where the need to process large amounts of information increases the workload of the operator. Auditory displays provide a means of accessing an additional information channel when the information load is high. Examples of auditory systems in high cognitive load environments include the use of auditory spatial location information for threat assessment in fighter cockpits and the use of voice loops in the NASA control centre (Patterson, WattsPerotti & Woods, 1999). In both these cases the auditory modality provides a way to distribute attentional resources across modalities. Both these systems use aspects of attention to help direct cognitive processing. In the first example the three-dimensional location and type of the sound is designed to direct the pilot’s focal attention to the type and location of a threat. As the threat diminishes so does the perceived auditory input, which allows the auditory signal to be monitored preattentively. NASA voice loops are used to maintain team awareness of space shuttle operations where events salient to the operator redirect selective attention to the appropriate voice loop. Both of these systems allow the operator to conduct concurrent visual tasks.

2.3

Constraints on visual presentation

There are some data sets that are hard to display visually. For example, very large data sets are difficult to display visually and may be better suited to auditory display. Barrass and Kramer (1999) described the advantages of displaying seismic data sets auditorily. Representing the data in sound and then speeding up the resulting audification allowed small changes in the data spread over many hours to be perceived in a few minutes. Other examples of datasets that might lend themselves to auditory displays are when events are fleeting and capturing them in visual display is difficult, or when events cluster together and distinguishing them visually is difficult (Barrass & Kramer, 1999).

2.4

Perceptual shifts are difficult

In some situations, shifting attention from the visual domain is not possible, and providing information in the auditory domain lets the operator continue to focus on the visual environment. For example, rescue pilots use radio beacons to locate a target while visually monitoring altitude, air speed and external environment.

In most cases auditory displays are added during the evolution of a system, without fully modelling how they can be integrated with existing visual displays. This has led to many poor designs that in some cases hinder the system controller (Watson et al 1999, Woods 1995). There are no rules as to how and when auditory displays should be used. In addition there are many unresolved questions relating to the usability and usefulness of auditory displays, including people’s acceptance of auditory displays (Barrass and Kramer, 1999). The evaluation of these aspects of auditory displays has been relatively neglected (Kramer, 1994). Many control room observations have noted the use of informal auditory cues in managing complex systems. Some arise from mechanical processes that create sound as an artefact. For example, anaesthetists have relied on the audification of bellows in the operating room (OR) to detect respiration problems in patients. A similar use has been noted in operators in nuclear power stations who have associated the rate of change in rod clicks with the state of the reactor (Vicente, 1999). However, no guidelines have been produced about how to design such artefacts into systems. In the next section we discuss approaches to designing sound to be informative. We outline the work of two communities that, together, may lead to a framework for designing auditory displays.

3

Frameworks to guide interface design

In this section we discuss the work of the sonification and EID communities and show how, if put together, the two may lead to a much more robust framework for designing auditory display. The sonification community brings to this task a finegrained understanding of auditory perception and experience in representing data relations and meanings in sound. The EID community has developed a principled basis for interface design that can be extended to auditory displays. We discuss these in turn, and then highlight remaining issues.

3.1

Sonification and auditory perception

Recently the sonification community has investigated the principles for using different properties of sound such as location, duration, timbre, density and rhythm in order to develop guidelines for sonification (Kramer et al, 1999). “Sonification is the transformation of data into perceived relations in acoustic signals for the purposes of facilitating communication or interpretation.” (Kramer et al, 1999, p.3). This has been based on recent advances in sound technology that have allowed the properties of large data sets or of dynamic data streams to be mapped to sound sources (Kramer, 1994; Ballas, 1994). Sonification lets scientists perceive regularities in data that are imperceptible in visual representations (Barrass & Kramer, 1999) and has allowed complex data to be presented simultaneously. It is a natural way of presenting temporal data, and it removes the need to direct visual attention to a stimulus, removing constraints of position and posture. While it is possible to identify a number of examples of successful sonifications, it is clear that it has not yet

been possible to develop guidelines for designing successful sonifications (Kramer et al. 1999). Many sonifications have been developed with reference to theories of auditory perception and psychophysics, whereas analysis of the environment, the nature of the data and the goals of the application have been relatively neglected (Kramer et al. 1999). One of the difficulties of this approach is that it may result in poorly designed displays that are invasive, hard to understand or subject to phenomena such as masking (Watson et al, 2000). EID has been proposed as one way to avoid these problems (Watson et al, 1999).

3.2

Why EID for auditory displays?

EID has been successfully used for designing visual displays to support problem solving during abnormal system states in complex environments. EID aims to support the human operator’s reasoning at different levels of abstraction, so that an interface maps physical functions and measurement to higher order processes and rationale. Using displays designed by this approach an operator will receive a higher level of support for dealing with both normal and unanticipated system states (Vicente, 1999). EID has been successfully applied to complex systems such as power plant and process system control, aviation and more recently medical environments (Dinadis andVicente 1996; Lind, 1994; Rasmussen 1994; Burns & Vicente, 1996; Sanderson, 1998, Miller & Sanderson, 2000). This is achieved through the combined techniques of abstraction hierarchy (Rasmussen, 1986), activity analysis and semantic mapping as seen in Table 1 (Rasmussen, 1994). None of the fundamental principles of EID limits the design of ecological interfaces to visual displays. However, EID has been commonly applied only to visual displays. The studies cited above implicitly assume that the monitoring task is usually visual and takes place in focal attention. It is notable that the most comprehensive treatment of the ecological approach to human-machine systems (Flach, Hancock, Caird & Vicente, 1995) does not include a chapter on auditory interfaces. Gaver (1993) has used ecological principles in his work on auditory icons and earcons. This approach has also been applied to the sonification of real time data (Gaver, Smith & O’Shea, 1991, Mynatt, 1997). The ecological approach used by Gaver et al (1991) in the Arkola simulation of a bottling plant and by Mynatt (1997) in a marine power plant did not use a full EID analysis but instead focussed on how to represent physical functions acoustically. A full EID approach identifies higher order properties that should be displayed. Overall, then, there has not yet been a full integration of the principles of EID with an analysis of what the respective roles of visual and auditory modalities might be. We will examine how we might extend EID to embody principles by which modality decisions can be made during interface design.

4

EID and auditory interface design

EID is an approach to visual display design that stems out of cognitive work analysis (CWA:

Rasmussen, Pejtersen, & Goodstein, 1994; Vicente, 1999). CWA, in turn, is a cognitive engineering approach to identifying requirements for the interfaces of complex real time systems. EID uses some of the phases of CWA; work domain analysis (WDA), control task analysis (CTA) and for actual design adds a semantic mapping step. These three phases provide information for designing auditory displays that in some cases is quite distinct from the advice for visual displays. Moreover, in this section we argue that if EID is to be useful for designing interfaces that include both auditory and visual elements, some of the other phases of CWA are needed. Specifically, in addition to the semantic mapping phase an attentional mapping phase appears to be needed. Table 1 shows the phases of CWA, a definition of each phase and how the framework might be extended to provide guidance for designing auditory displays. The first phase, WDA, starts to define what information should be represented in a display, and results in no distinct information that might indicate either the need for, or form of, an auditory display. One possible exception is that important physical processes may be amenable to audification—the transformation and presentation of their inherent sound. CTA—the phase that indicates what needs to be done in the work domain—can help to indicate the attentional profile that might be maintained across multiple work functions or control tasks. This is especially so if a Temporal Coordination Control Task Analysis (Sanderson & Naikar, 2000) has been performed that shows the constraints on the temporal sequencing of control tasks. The next three CWA phases are usually not discussed in great detail in treatments of EID, but with auditory interface design they may need to be dealt with more directly. Strategies Analysis (SA) is complicated by the possibility of having auditory interfaces, since the range of strategies available to human controllers will be extended and the degrees of design freedom that the interface designer has to resolve in choosing a design are greatly increased. Social-Organisational Analysis (SOA) will become more critical, given the obligatory nature of auditory displays. SOA will need to be performed to indicate where an auditory display may aid coordination and where it may introduce unwanted noise and distraction. Worker Competencies Analysis (WCA) suggests that the level of cognitive control can be manipulated not only with different visual interfaces, but also with different auditory interfaces. Proponents of EID support moving the level of cognitive control from the knowledge-based to the rule-based level, or from the rule-based to the skill-based level wherever possible and appropriate (Vicente & Rasmussen, 1990; 1992). At lower levels of cognitive control tasks are carried out more quickly, more effectively and with less effort compared to higher levels of control. Because higher levels of cognitive control are more effortful and error prone people often prefer to work at the skillbased level. (Vicente, 1999). Sonification may be a highly effective way of achieving this outside the visual modality.

Table 1. Extending EID to design of auditory displays to work alongside visual displays. Shaded cells indicate phases usually associated with EID for visual displays. CWA phase

Description

Issues for auditory displays

Work Domain Analysis (WDA) • Functional purpose • Priorities and values • General function • Physical function • Physical form Control Task Analysis • Temporal coordination control task analysis (TC-CTA) • Control task analysis (CTA)

Provides information about why the system or work domain exists, the flow of information or value through it, its functions, and the physical processes and objects underlying its functions.

Helps to identify work domain characteristics and relations that need to be displayed in any interface. For example, physical properties of work domain may indicate candidates for audification. Information is necessary but insufficient for interface design at this point.

Provides information about what needs to be done in the work domain, by whom, when, and how information about activity might be transmitted. Also gives information about temporal relations between tasks

Strategy analysis (SA)

Provides information about different ways, if more than one way exists, in which the control tasks can be carried out. Provides information about how work is shared across multiple actors in a complex organisation and how multiple actors coordinate efforts Provides information about the form of cognitive control needed for a task, distinguishing skill- rule- and knowledgebased behavior.

In helping to identify a temporal profile of ongoing tasks, and possible competition between tasks, CTA leads analysts to knowledge about an appropriate attentional profile across tasks. This leads to conjectures about which tasks are best displayed visually, and which auditorily. Range of strategies available to human controllers may be extended by considering the possibilities of auditory displays in an interface. Indicates where auditory display might help or hinder coordination between actors, given the obligatory nature of most auditory displays.

Social organisational analysis (SOA)

Worker competencies analysis (WCA)

Semantic mapping (SM)

Attentional mapping (AM)

Provides information about criteria for choosing interface elements so that goalrelevant task invariants are mapped onto key perceptual properties of the interface’s behavior. Provides information about whether and when a control task should be supported in focal or non-focal attention.

The final two steps relate to ways of mapping display requirements onto perceptual forms. Much of the work in the sonification community is dealing with how to map data relations onto sound relations in a meaningful way, so is deeply immersed in auditory semantic mapping using principles of auditory perception to guide the process. An example will be given in the next section. Finally, we argue that a further mapping step is needed—attentional mapping. Attentional mapping provides information about whether and when a control task should be supported in focal or non-focal attention. For visual displays this concern has been dealt with in the context of semantic mapping. However, when display possibilities extend to auditory displays, we argue that designers require guidelines for how an auditory display should control attention alongside other interface elements. This guidance should be based in a knowledge of auditory attention. Next we discuss semantic and attentional mapping in the context of auditory display design.

5

Auditory semantic mapping

One of the most succinct yet complete descriptions of the semantic mapping process has been provided by Hansen (1995) who provides guidelines for the nesting of information in display geometries. He works with the following seven heuristics (paraphrased):

Indicates intrinsic or training-based characteristics of workers that might point to the effectiveness of auditory elements in interface displays. Auditory display and especially sonification may help move cognitive control towards SBB. Gives designers a framework for judging the information-carrying potential of dimensions of an auditory stimulus, based in a knowledge of auditory perception. Gives designers requirements for how an auditory display should control attention alongside other interface elements, based in a knowledge of auditory attention.

1. Goal achievement as figural goodness 2. Work domain constraints as visual containers 3. Process dynamics as figural changes 4. Functional relations as visual connections 5. Pictorial symbols to represent components 6. Alphanumerical output where needed 7. Time as visual perspective. Hansen (1995) integrates these elements into a sentence that, with our paraphrased heuristics, would read as follows: Integrate (1) goal achievement as figural goodness with (3) process dynamics as figural changes and put them on top of (2) work domain constraints as visual containers and let (4) functional relations as visual connections decide the placement of (2) work domain constraints as visual containers without spoiling the integration of (1) goal achievement as figural goodness with (3) process dynamics as figural changes and making a natural relation/transition between (2) work domain constraints as visual containers and (5) pictorial symbols to represent components, and show (3) process dynamics as figural changes by (7) time as visual perspective with (6) alphanumerical output where needed in addition.

How useful might these heurisitcs be when applied to the design of displays in the auditory modality? It is not clear whether these heuristics can be directly transferred to the auditory domain, or whether the perceptual processes underlying audition and vision

demand different approaches to semantic mapping. An example of applying these heuristics to auditory display might shed some light on this issue. Buttigieg and Sanderson (1991) used a temperature control system to examine the effects of display designs on human performance. Two input streams of water with similar flowrates joined to form a single output stream. The output temperature was the average of the two input streams: ( I1 + I2 ) / 2 = O. Subjects monitored a visual display showing the output for I1, I2 and O. A bar graph and shape display were used in which the output lay between the two inputs so that the output’s height fell exactly half way between the heights of the two inputs (Figure 1). Under equality conditions, the shape display produced an emergent feature of a straight line. With the bar graph display, a straight line could be imagined across the tops of the three bars. When the system strayed from equality then the line became broken (see Figure 1). The display was therefore an example of Hansen’s (1995) heuristics 1, 3, and 4. Can something similar be done with an auditory display? A first attempt is shown in Figure 2 which illustrates the possibilities but also the pitfalls. Each ear hears a sonification that consists of the quantities I1 and [O-I2]/2 in the left ear and I2 and [O-I1]/2 in the right ear. Different sound qualities could be used to distinguish the two sounds in each ear. When all is normal the two quantities in each ear yield the same value, and therefore the same sound—and a single sound is heard in each ear. When there is an abnormality, the two quantities in each ear start to separate and two sounds are heard in each ear. If the individual sounds are not too complex, such separations should produce an “auditory imperative when “beats” (throbbing dissonance) are heard as the two sounds separate. However three of many problems are (1) the sounds across ears will usually be dissonant even when all is normal, (2) with different sound qualities for the two parameters in each ear, the beats phenomenon is less likely to arise and (3) configurality is lost because there are no clear correspondents for individual elements I1, I2, and O. Looking now at Hansen’s (1995) heuristics, goal achievement should be represented by figural goodness—in the above case by acoustic simplicity. Work domain constraints should be represented as visual containers—a spatial concept harder to represent in sound. Process dynamics are represented through the movement of particular acoustic parameters, and the relationships between them represent functional relations. The representation of time is particularly suited to an auditory display because sound is the representation of temporal relationships. An auditory display might have other advantages over a visual display in completing such a monitoring task. Buttigieg & Sanderson’s (1991) tasks were only carried out for short period of time so the problem of vigilance was not examined. Over longer periods of time subjects using an auditory display could be expected to perform better than those using a visual display as discussed in the background. Also subjects can conduct the monitoring task “preattentively” while

undertaking other visual tasks. This has been observed with other sonifications such as pulse oximetry in the operating room (Watson et al. 1999).

6

Auditory attention

What difficulties might arise when trying to use EID to guide the design of auditory displays? The differences between vision and audition might suggest suggest challenging areas. Audition and vision differ in two fundamental ways (Wickens & Hollands, 2000). First, because sound can be perceived from any direction there is no equivalent in the auditory modality to visual scanning as an indication of attention. The auditory modality functions to detect stimuli in the environment that may or may not be in the visual field. The sound draws the attention of the visual system to a stimulus, increasing the organism’s ability to detect and avoid threats (Pashler, 1999). Second, sound is transient. It cannot be fixed in the present or returned to in a similar way to visual stimuli. These observations about the differences between vision and audition suggest that an understanding of the differences between auditory and visual attentional processes—and of the interaction between them—are crucial in designing effective auditory displays. The consequences of an incomplete understanding of these processes are ineffective displays. One example is conventional auditory alarms. Auditory alarms rely on the ability of the auditory modality to draw attention to something that is outside focal awareness. However, intrusive sound has been shown to degrade performance at times of high cognitive load, just when the operator needs to marshal all resources and maximise performance (Woods, 1995). Anecdotal evidence of this effect is contained in reports of people silencing alarms because they are distracting. An extensive cognitive psychological literature has also shown that irrelevant sound substantially reduces performance on visual tasks (Jones, 1999). Appropriately designed continuous auditory displays reduce the need for alarms and minimise their intrusive effect, but they also pose the problem of how to use the obligatory processing of sound while minimising the negative effect of the auditory stimulus on performance. The challenge of designing effective auditory displays can therefore be conceptualised as the challenge of facilitating the movement of the sound in and out of focal awareness as appropriate in relation to system status. Table 2 illustrates the problem. Description starts with the system in normal state, and the auditory display being attended to in focal awareness. After a while, as other activities intervene, the auditory display will move outside focal awareness (see arrow 1). If the system then becomes abnormal, either of two things can happen. In the ideal case, the change in the auditory display representing abnormality will be sufficient to bring the auditory display back into focal awareness (see arrow 2). From there, the system will either move back into a normal state and the cycle will start again (arrow 4b) or the sound will drift back outside focal awareness (arrow 4a). Two undesirable situations are possible. First, the

change in the auditory display representing abnormality may not initially be sufficient to bring the auditory display back into focal awareness (see arrow 3a). It is only with additional visual support or additional auditory labelling that the sound comes back into focal awareness (arrow 3b). Second, once in focal awareness when the system state is abnormal, the auditory display may not drift out of focal awareness (arrow 4a) and may disrupt performance on other tasks. We need to identify what factors facilitate shifts in auditory attention. It is appropriate to consider what experimental evidence could help in identifying these factors. First, it is essential to note that our knowledge about auditory attention is much less extensive than our knowledge about visual attention (see Pashler, 1999). In addition, attention research in the visual domain has developed from a rich knowledge of visual perception. However, knowledge of auditory perception is based largely on music and speech perception, to the extent that it has been claimed that there is no comprehensive theory of auditory perception beyond these two areas (Hirsh, 1996). The work by Bregman on auditory scene analysis represents a significant step towards the development of a theory of auditory perception. This work has examined the perceptual organization of sound and seeks to understand the basis on which sounds are organized into streams. (Bregman, 1990). The separation of the different research communities

addressing questions fundamental to the design of auditory displays poses a particular challenge in this area. If the problems of designing auditory displays are to be addressed communication between these communities is essential. Second, a review of research on auditory attention revealed that there has been some extension of the concepts used in visual attention to the auditory modality. For example, auditory attention research has investigated selective attention, divided attention, and cuing (Pashler, 1999). However, we have already identified that the crucial issues in designing auditory displays revolve around the shifting of attention, and the interaction between audition and vision. Cross modal displays have often been used when there is a concern that displaying information in two channels in the same modality might lead to confusion or increased processing time (Miller, 1991, Parkes & Coleman, 1990). However, the conditions under which cross modal displays are better than time-sharing displays and the mechanisms that contribute to cross modal displays being more efficient are not clear (Wickens & Hollands, 2000). Bimodal studies of attention are rare and suffer from the problem of diverse methodologies, reducing the generalisability of the results (Pashler, 1999).

Figure 1. Visual and auditory EID displays. The Bar and Shape are from Buttigieg and Sanderson (1991) whereas the Audio represents an equivalent display using frequency change over time.

Shape (S)

left ear

Audio (A)

right ear

Equality

Bar graph (B)

I1

O

I2

I1

I2

I1

Shape (S)

[O – I2]x2

left ear

I2

[O – I1]x2

Audio (A) right ear

Inequality

Bar graph (B)

O

I1

O

I2

I1

O

I2

I1

[O – I2]x2

I2

[O – I1]x2

System State

Sound Inside Focal Awareness

Sound Outside Focal Awareness 1

Normal

Abnormal

Appropriate if attending to the display does not divert resources from critical tasks. Sound must shift out of focal awareness if cognitive resources are needed on another task Appropriate when attention is drawn to critical 4b system state. Must drift out of awareness once action taken and resources are required.

Appropriate if system state is inside limits.

3a

2 3b

Appropriate only after action has been taken and resources are directed to resolve abnormality

4a Table 2. Mapping of attention to a continuous auditory display when system state moves between normal and abnormal. Studies of involuntary shifts in attention have found that once attention is directed to information processing in one modality, further information presented in another modality is often unattended and is not used in task performance. This has been found in studies of visual tasks (Massaro & Warner, 1977), haptic tasks (Heller, 1992) and auditory tasks (Ward, 1994). Most studies of the causes of involuntary attention shifts have investigated the visual modality (Pashler, 1999). Characteristics of visual stimuli that capture attention include abrupt onset (Yantis & Hillstrom, 1994), uniqueness (Folk & Annett, 1994) novelty (Lorch et al., 1984) or high emotional reactivity (Martin et al., 1991). However, our knowledge of the characteristics of sound stimuli that grab attention is limited. Some work from the sonification community is pertinent here. Mitsopopoulos and Edwards (1997) have worked towards a hierarchical organization for presenting auditory streams of information. Using two types of structures, one across the auditory stream (or an instant of time), and the other within each stream (event over time), they have produced a methodology for evaluating the suitability of auditory designs such as earcons. Mitsopopoulos and Edwards (1997) methodology has so far only been applied to low level widgets and has not incorporated concepts such as the auditory equivalence of a visual glance in their design. Barrass (1996) developed TaDa (Task-oriented Data-sensitive method); which integrates task analysis, a database of sound examples, a rule-based design aid and interactive sound design tools. However both methodologies do not define what should be displayed using sound. In summary, there are many basic questions about auditory perception and auditory attention that remain unanswered. First, apart from the sonification community, investigation of the dimensions of sound and perceptual and attentional processes has been neglected. There is little knowledge about such attributes of sound as timbre, density and rhythm, and it is unclear how information processing across modalities can be maximised. For example, when does redundant information improve perception, and when does crossmodal interference occur? (Kramer et al, 1999). However, there is some indication in the cognitive psychology literature that an understanding of the interaction between modality-specific attention and higher cognitive processes may be essential, but this is

only starting to be investigated. Such an understanding is likely to be crucial to designing effective multi-modal displays. In the meantime specific information to guide the design of auditory displays might be better obtained from the sonification community.

7

Application of EID to other modalities

As with auditory displays the use of other modalities; vestibular, haptic and olfactory will require knowledge about when a control task should be supported in focal or non-focal attention and how many channels are required to convey the environment to the operator. Currently we know less about vestibular, haptic and olfactory attention and processing of information from these senses; however we can expect developments in these areas to require some attention in the near future. Humans have evolved using multi-modalities to cope with the natural environment and therefore we should examine all these modalities when designing interfaces for complex systems.

8

Conclusions

Human responsiveness is likely to remain dominated by the visual domain but we should not neglect other modalities when designing new systems. A well designed system that supports the operator using multiple modalities is likely to support lower levels of cognitive control which are quicker, more effective and reduced effort for the human operator. However, applying EID to any perceptual modality requires some theoretical extension to address the design issues that arise from the perceptual and attentional differences between modalities. Further research is required in the area of the modality of attention so EID can encompass the human operator’s sensory as well as cognitive abilities.

9

Acknowledgments

The authors would like to acknowledge the contributions of Dr Greg Kramer to our understanding of sonification. Thanks also to two anonymous reviewers

10 References [1]

[2]

Ballas, J.A. (1994). Delivery of information through sound. In Kramer, G. (1994) (Ed.) Auditory display. SFI studies in the sciences of complexity, Proc Vol. XVIII, Addison-Wesley. Barrass, S (1996) TaDa! Demonstrations of Auditory Information Design. Proceedings of the International Conference on Auditory Displays (ICAD96). Palo Alto, California.

[3] [4] [5]

[6]

[7]

[8]

[9]

[10] [11]

[12]

[13]

[14] [15] [16]

[17]

[18]

[19] [20]

[21]

[22]

[23]

[24]

[25]

Barrass, S., & Kramer, G. (1999). Using sonification. Multimedia Systems, 7, 23-31. Bregman, A.S. (1990) Auditory scene analysis: The perceptual organization of sound. Cambridge Mass.:MIT Press. Burns, C.M. & Vicente, K.J. (1996). An experimental evaluation of functional displays in process supervision and control. Cognitive engineering laboratory, University of Toronto, Toronto CEL 9609. Buttigieg, M. A., & Sanderson, P. M. (1991). Emergent features in visual display design for two types of failure detection tasks. Human Factors, 33, 631-651. Dinadis, N. & Vicente, K.J. (1996). Ecological interfaces for complex industrial plants. Cognitive engineering laboratory, University of Toronto, Toronto CEL 94-06. Flach, J., Hancock, P., Caird, J., & Vicente, K. (1995). Global perspectives on the ecology of human-machine systems (Volume 1). Hillsdale, NJ: LEA. Folk, C.L. & Annett, S. (1994). Do locally defined feature discontinuities capture attention? Perception and Psychophysics, 56, 277-288. Gaver, W. (1993). How in the world do we hear? Explorations in ecological acoustics. Ecological Psychology, 5, 283-313. Gaver, W.W., Smith, R.B. & O’Shea, T. (1991) Effective sounds in complex systems: The Arkola simulation. Proceedings of CHI ’91., ACM Press. Hansen, J.P. (1995). Representation of system invariants by optical invariants in configural displays for process control. In P. Hancock, J. Flach, J. Caird, & K. Vicente (Eds.), Local applications of the ecological approach to human-machine systems (Volume 2). Hillsdale, NJ: LEA Helander, M.G., Landauer, T.K. & Prabhu P.V. (1997). Handbook of Human-Computer Interaction. Elsevier Science B.V.1000 AM Amsterdam, The Netherlands. Heller, M.A. 1992 Haptic dominance in form perception: Vision versus proprioception. Perception, 21, 655-660. Hirsh, I.J.(1996). Auditory psychophysics and perception. Annual Review of Psychology, 47, 461-485. Jones. D. (1999) The cognitive psychology of auditory distraction: the 1997 BPS Broadbent lecture. British Journal of Psychology, 90, 167-188. Kramer, G. (1994). Some organizing principles for representing data with sound. In Kramer, G. (Ed.) Auditory display. SFI studies in the sciences of complexity, Proc Vol. XVIII, Addison-Wesley. Kramer, G., Walker, B., Bonebright, T., Cook, P., Flower, J., Miner, N., Neuhoff, J., Bargar, R., Barrass, S., Berger, J., Evreinov, G., Fitch, W.T., Grohn, M., Handel, S., Kaper, H., Levkowitz, H., Lodha, S., Shinn-Cunningham, B., Simoni, & M. Tipei, S. (1999) The sonficiation report: Status of the field and research agenda. Report prepared for the National Science Foundation by members of the International Community for Auditory Display. Santa Fe NM: ICAD. Lind, M. (1994) Modeling goals and functions of complex industrial plants. Applied Artificial intelligence, 8, 259-283. Lorch, E.P., Anderson, D.R. & Well, A.D. (1984). Effects of irrelevant information on speeded classification tasks: Interference is reduced by habituation. Journal of Experimental Psychology: Human Perception and Performance, 10, 850-864. Martin, M. Williams, R.M. & Clark, D.M. (1991). Does anxiety lead to selective processing of threat-related stimuli. Behavior Research and Therapy, 29, 147-160. Massaro, D.W. & Warner, D.S. (1977) Dividing attention between auditory and viusal perception. Perception and Psychophysics, 21, 569-574. Miller, A. & Sanderson, P. (2000). Modelling “deranged” physiological systems for ICU information system design. Proceedings of the joint 14th Triennial Congress of the International Ergonomics Association/44th Annual Meeting of the Human Factors and Ergonomics Society (HFES/IEA 2000). San Diego, CA. 30 July – 4 August, 2000. Miller, J. (1991) Channel interaction and the redundant targets effect in bimodal divided attention. Journal of Experimental Psychology: Human Performance and Perception, 17, 160-169. Mitsopoulos, E.N. & Edwards A.D.N. (1997) Auditory Scene Analysis as the Basis for Designing Auditory Widgets.

[26]

[27]

[28] [29]

[30]

[31]

[32] [33]

[34]

[35]

[36] [37]

[38]

[39]

[40]

[41]

[42]

[43]

[44] [45]

[46]

[47]

Proceedings of the International Conference on Auditory Displays (ICAD97). Palo Alto, California. Mynatt, E.D. (1997) Auditory icons. In Marti A. Hearst (Ed.) Dissonance on audio interfaces. IEEE Intelligent Systems, 12, 1016. Parkes, A.M. & Coleman, N. 1990. Route guidance systems: A comparison of methods of presenting directional information to the driver. In E.J. Lovesey (Ed.) Contemporary ergonomics. 1990, 480-485. London: Taylor and Francis. Pashler, H.E. (1999). The psychology of attention. Cambridge Mass.: MIT Press. Patterson, E.S., Watts-Perotti, J.& Woods, D.D. (1999). Voice loops as coordination aids in space shuttle mission control. Computer Supported Cooperative Work, 8, 353-371. Rasmussen, J. (1994). Taxonomy for work analysis. In Salvendy, G. & Karwowski, W. (Eds.) Design of work and development of personnel in advanced manufacturing. New York: John Wiley and Sons, pp. 41-77. Rasmussen, J. (1986) Information processing and human-machine interaction: An approach to cognitive engineering. North Holland, New York. Rasmussen, J., Pejtersen, A. & Goodstein, L. P. (1994). Cognitive systems engineering. New York: Wiley Interscience. Rollins, R.A. and Hendricks, R. 1980. Processing of words presented simultaneously to eye and ear. Journal of Experimental Psychology: Human Performance and Perception, 6, 99-109. Sanderson, P. & Naikar, N. (2000). Temporal coordination control task analysis for analysing human-system integration. Proceedings of joint 14th Triennial Congress of the International Ergonomics Association/44th Annual Meeting of the human Factors and Ergonomics Society (HFES/IEA 2000) San Diego, CA. Sanderson, P.M. (1998). Cognitive work analysis and the analysis, design, and evaluation of human-computer interactive systems. Proceedings of the Australian/New Zealand Conference on Computer-Human Interaction (OzCHI98), Adelaide, Australia, Dec. 1-4. Stanton, N. A. (1994). Human factors in alarm design. London: Taylor & Francis. Vicente, K. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. New Jersey: Lawrence Erlbaum. Vicente, K. & Rasmussen, J. (1990). The ecology of humanmachine systems II: Mediating “direct perception” in complex work domains. Ecological Psychology, 2, 207-249. Vicente, K., & Rasmussen, J. (1992). Ecological interface design: Theoretical foundations. IEEE Transactions on Systems, Man, and Cybernetics, SMC-22, 589-606. Ward, L.M. 1994. Supramodal and modality specific mechanisms for stimulus–driven shifts of auditory and visual attention. Canadian Journal of Experimental Psychology, 48, 242-259. Walker, B.N., Kramer, G., & Lane, D.M. (2000). Psychophysical scaling of sonification Mappings. Georgia Institute of Technology Atlanta, Georgia USA (ICAD 2000). Watson, M., Russell, W. J., & Sanderson, P. (1999). Ecological interface design for anesthesia monitoring. Proceedings of the Australian/New Zealand conference on Computer-Human Interaction (OzCHI99). Wagga Wagga: Charles Sturt University. Pp78-84. Watson, M., Russell, W. J., & Sanderson, P. (2000). Alarm noise and end-user tailoring : The case for continuous auditory displays. Proceedings of the 5th International Conference On Human Interaction With Complex Systems (HICS2000). UrbannaChampaign: University of Illionis. Pg75-79. Wickens, C.D. & Hollands, J.G. (2000) Engineering psychology and human performance. New Jersey: Prentice Hall. Woods, D. (1991). The cognitive engineering of problem representations. In J. Alty & G. Weir (Eds.), Human-computer interaction and complex systems. London: Academic Press. Woods, D. D. (1995). The alarm problem and directed attention in dynamic fault management. Ergonomics, 38, 2371-2394. Yantis, S. & Hillstrom, A.P. (1994). Stimulus-driven attentional capture: Evidence from equiluminant visual objects. Journal of Experimental Psychology: Human Perception and Performance, 20, 95-107.