A Comprehensive Framework for Auditory Display: Comments on Barrass, ICAD 1994 STEPHEN BARRASS University of Canberra
In ‘A Perceptual Framework for the Auditory Display of Scientific Data’ I described the first perceptually scaled sound space designed specifically for sonification. I modeled this sound space, and its underlying theory, on the use of perceptual colour spaces in scientific visualization. As I went on to apply the sound space in mappings of satellite data I introduced methods of data characterization and user-centered task analysis into my design framework. In trials I realized that satellite images allow you to see global information across millions of data values, whereas it was impossible to play all the data at once as sounds. This lead me to explore perceptual streaming as a means for perceiving similarity and difference in masses of sounded data. In work on sonifications for virtual reality applications I recognized the need to consider the semiotic linkage of the sound with the application domain, and the need to also link the sound with the interaction metaphor. The work described in this paper laid the foundation for the ongoing development of a comprehensive framework for auditory display that takes into consideration the perceptual organization of the sounds, the characteristics of the data, the gamut of the display device, the user’s tasks, the semiotic linkage to the application domain, and the affordances for interaction. Categories and Subject Descriptors: [General Literature—General]—Conference proceedings General Terms: Design, Experimentation, Measurement, Theory Additional Key Words and Phrases: Sonification, auditory display, data mapping
1.
HISTORICAL CONTEXT
In the early 20th century Munsell catalogued visible colours by 3 properties commonly known as hue, lightness and saturation. The catalog could be arranged as a 3D space by placing the ten main hues in a circle with those that mixed to grey across from each other. By the middle of the century, the color space had been psychophysically scaled to produce a perceptually uniform 3D continuum. This color space has proven useful for scientific visualization where the scaling preserves relations between data values mapped into color. Analysts have developed specialized data to color mappings for different tasks, such as a “rainbow” scheme to show categories and a “color opponent” scheme to show correlation. In 1992 I joined a visualization group working on an interface for mapping satellite data through a perceptual color space to produce color images for mining exploration and environmental monitoring purposes [Robertson et al. 1994]. I wondered if there was a perceptual sound space that could be used to map the data into sounds in a similar manner. In background research I found there was a long history of psychoacoustic scaling dating back more than a hundred years, but the closest thing to a perceptual color
Authors’ address: Sonic Communications Research Group, School of Creative Communication, University of Canberra, ACT Australia 2601; email:
[email protected]. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or
[email protected]. c 2005 ACM 1544-3558/05/1000-0403 $5.00 ACM Transactions on Applied Perception, Vol. 2, No. 4, October 2005, Pages 403–406.
404
•
S. Barrass
space was a 3D chart for comparing the timbre of pipe organs. I also found that perceptual entanglement of data dimensions had been identified as a major problem in sonification at the inaugural conference on auditory display just that year. 2.
RESEARCH PROCESS
From my experience with color mappings, I envisaged that a perceptually scaled sound space could be used map independent data dimensions to independent auditory dimensions, and to map category, order, and magnitude of differences in the data to category, order and magnitude of differences in sounds. I soon found the step from color space to sound space was not as straightforward as I had thought. Color spaces span all visible colors with just three orthogonal axes. Sound, however, has temporal, as well as spectral components, and there is no low dimensional or standard set of axes that span all sound perception. I sidestepped this stumbling block by generalizing the sound space as an ‘information space’ instead, and modeling the perceptual organization on color space. I selected eight equally different categorical timbres from Grey’s multidimensional scaling of musical instruments, and scaled the brightness of each with the Van-Noorden streaming effect. The biggest challenge was to interpolate a continuous space from the grid of sample points in timbre, pitch and brightness. It took patience and perseverance to tune the many parameters of a thin plate spline algorithm to coerce 8 different timbre segments with disjoint pitch ranges into a polar-cylindrical continuum. The resulting sound gamut was more complicated than a color gamut, with much more variation in the brightness profiles of sound timbres than occurs in the saturation profiles of color hues. The pitch ranges for each timbre did not meet at a common low and high point because the instruments had different pitch ranges. Although it was bumpy, this was the first example of a scaled 3D sound space organized specifically for mapping data into sound. The 3D visualization of the sound gamut provided a new way to think about data to sound mappings in terms of geometric relations that represented both data and perceptual relations. It also provided a way to design mappings to take advantage of the gamut of the auditory display [Barrass 1994]. 3.
BODY OF WORK
I mapped satellite data through the perceptual sound space into sounds that were overlaid on color imagery of the same data. The user could mouse-click the image to hear a data point at a location. In the next experiment I augmented an animated visualization of chemicals in a river system with sonified rainfall data. Through these practical experiments I realized that the mapping of spatial and temporal relations in the data was just as important as the mapping of the data values. In a satellite image the information comes from visual structures arising from spatial relations between millions of data values. If the auditory display was going to provide more than a single point of information at a time I needed to find a way to enable the perception of structures arising from masses of sonified data. At ICAD’92 Williams proposed that emergent auditory figure/ground effects could be produced for sonification purposes using principles of Auditory Scene Analysis [Williams 1994]. Based on this idea, I experimented with sequential streaming to segregate two categories in a data set of 10,000 sixdimensional soil samples played over 90 seconds [Barrass 2000]. I experimented with the Van Noorden effect to group 9 continuous variables into higher-level ‘continuous emergent alarms’ for monitoring a turbine system [Albers et al. 1997]. I explored simultaneous streaming to display correlations between up to six ratio variables in Oil and Gas well-logs [Barrass and Zehner 2000]. Some people responded that my sonifications were ‘too noisy’. Since the sounds weren’t loud or acoustically noise-like I wondered whether this might be more to do with ‘mental noise’ and the distraction of attention from a task. Working with the hypothesis that ‘useful sounds’ would not be considered ‘noise’, I introduced methods of user-centered design into my process, through user scenarios, task analysis and the case-based lookup of potential designs from a database of 200 stories about everyday listening ACM Transactions on Applied Perception, Vol. 2, No. 4, October 2005.
A Comprehensive Framework for Auditory Display: Comments on Barrass
•
405
experiences [Barrass 1996]. These stories provided insights into the function of sounds in everyday activities, and also provided metaphors that could help the listener interpret an auditory display from everyday experiences. I applied this approach and noted a marked increase in engagement with an interactive sonification at an Oil and Gas trade-show when I altered the sound schema from a synthesised tone to a Geiger-counter like ‘clicking’. The Geiger-counter schema also seemed to reduce the amount of time it took na¨ıve users to learn to manipulate the 6 degree of freedom stylus to probe the data, and provided a context for interpreting the sounds in terms of the geological application domain [Barrass and Zehner 2000]. I developed the concept of an integrated schema for the sound and the interaction in a ‘wind-sock’ sonification which could be moved freely around inside a car to hear air-flow data, and a ‘paper-jet-plane’ metaphor for tracking contours of equal flow [Eckel 1998]. Recently I have been interested in ways of moving sonification from the laboratory into actual use. At ICAD’03 I introduced design patterns into the field of auditory display in order to capture repeated examples of good sonification in a manner that could be understood and reused by designers from other disciplines [Barrass 2004]. As part of ICAD’04, I organized the world’s first concert of sonifications, Listening to the Mind Listening, at the Sydney Opera House. This concert brought 10 sonifications of brain activity to an audience of 350 people intrigued to hear what the human brain could sound like. The project generated thirty sonifications of the same data set, along with explicit descriptions of the mapping decisions made by each composer, and ninety reviews of these sonifications by a committee of thirty experts. The event has been a bridge between the auditory display and sonic arts communities that is leading to developments in the aesthetics of sonification [Barrass et al. 2006]. 4.
RELATIONS TO THE FIELD OF AUDITORY DISPLAY
Joseph and Lodha developed an Audio Transfer Function modeled on colour transfer functions as a means for designing sonification mappings [Joseph and Lodha 2002]. Walker and Kramer psychoacoustically scaled one-dimensional data to sound mappings, and found that metaphor had a significant effect on the interpretation of an auditory graph [Walker and Kramer 1996, 2000]. Mitsopoulos and Edwards reiterated data characterization and perceptual alignment as principles for designing Auditory Widgets [Mitsopoulos and Edwards 1998]. Anderson and Sanderson investigated perceptual streaming as a technique for organizing an auditory display in their experiment on how best to distribute 6 data variables between 3 streams in a monitoring task [Anderson and Sanderson 2004]. Rivenez et al. studied the effects on attention of one, two or three streams in an auditory display [Rivenez et al. 2002]. Saue added interaction design into my task and data driven design process in his ‘walking’ interface to the sonification of a 3D geological data-set [Saue 2000]. Daude and Nigay re-configured the task and data driven process as a data pipeline modeled on a pipeline for information visualization [Daude and Nigay 2003]. Nesbitt extended the task and data approach from auditory display to multi-sensory visual, haptic and audio displays in his MS-taxonomy [Nesbitt 2004]. Frauenberg et al. proposed design patterns for interaction with auditory displays [Frauenberger et al. 2005]. 5.
FUTURE WORK
I am currently analysing the mappings submitted to the Listening to the Mind Listening concert for design patterns that can be captured and communicated to other designers. In future I would like to use this corpus to inform experiments on the perception of higher-level patterns and structures, with a view to making sonification a useful tool for understanding complex multivariate temporal data that is difficult to understand with existing tools. The reviews of these sonifications contained many aesthetic observations that provide a basis for developing the aesthetics of sonification in the future. ACM Transactions on Applied Perception, Vol. 2, No. 4, October 2005.
406
6.
•
S. Barrass
CONCLUDING THOUGHTS
This paper is the only attempt I know of to produce a perceptually scaled sound space for sonification mappings. It introduced perceptual scaling and visualization theory for aligning perceptual structure with data structure to the field of auditory display. What now seems most important is the foundation it laid for a comprehensive framework for designing auditory displays that takes into account user tasks, data characteristics, device gamuts, semiotic schema, interaction metaphors, and the perceptual organization of higher levels of information in an auditory display. REFERENCES ALBERS, M., BARRASS, S., BREWSTER, S., AND MYNATT, B. 1997. Dissonance on audio interfaces. IEEE Expert. September. ANDERSON, J. AND SANDERSON, P. 2004. Designing sonification for effective attentional control in complex work domains. In Proceedings of the Human Factors and Ergonomics Society. New Orleans. September. BARRASS, S. 1994. A perceptual framework for the auditory display of scientific data. In Proceedings of the International Conference on Auditory Display (ICAD 1994), Santa Fe. BARRASS, S. 1996. EarBenders: Using stories about listening to design auditory interfaces. In Proceedings of the First AsiaPacific Conference on Human Computer Interaction. Singapore. BARRASS, S. 1998. Auditory Information Design. Ph.D. Thesis. Australian National University . BARRASS, S. 2000. Some golden rules for designing auditory displays. In The Csound Book: Perspectives in Software Synthesis, Sound Design, Signal Processing and Programming, ISBN 0-262-52261, R. Boulanger, Ed. MIT Press, March. BARRASS, S. AND ZEHNER, B. 2000. Responsive sonification of well-logs. In Proceedings of the International Conference on Auditory Display (ICAD 2000). Atlanta, April. BARRASS, S. 2004. Patterns for designing functional sounds. International Symposium on Sound Design, Paris, October. BARRASS, S. WHITELAW, M., AND POTARD, G. 2006. Creativity and Practice-led Research. Media International Australia incorporating Culture and Policy, No. 118, February 2006, L. Green, and B. Haseman Eds. The University of Queensland, ISSN 1329-878X. BREGMAN, A. 1990. Auditory Scene Analysis. MIT Press, Cambridge. DAUDø, S. AND NIGAY, L. 2003. Design process for auditory interfaces. In Proceedings of the International Conference on Auditory Display (ICAD 2003). Boston, July. ECKEL, G. 1998. A spatial auditory display for the CyberStage. In Proceedings of the International Conference on Audio Display (ICAD 1998). Glasgow, November. FRAUENBERGER, C., PUTZ, C., HOELDRICH, R., AND STOCKMAN, T. 2005. Interaction patterns for auditory user interfaces. In Proceedings of the International Conference on Auditory Display (ICAD 2005). Limerick, July. JOSEPH, A. AND LODHA, S. K. 2002. MUSART: Musical audio transfer function real-time tool kit. In Proceedings of the International Conference on Auditory Display (ICAD 2002). Kyoto, July. MITSOPOULOS, E. N. AND EDWARDS, A. D. N. 1998. A principled methodology for the specification and design of non-visual widgets. In Proceedings of the International Conference on Auditory Display (ICAD 1998). Glasgow, November. NESBITT, K. V. 2004. MS-Taxonomy: A conceptual framework for designing multi-sensory displays. In Proceedings of the International Conference on Information Visualization (IV 2004). London, July. RIVENEZ, M., DRAKE, C., GUILLAUME, A., AND SEBASTIEN, D. 2002. Listening to environmental scenes in real time. In Proceedings of the International Conference on Auditory Display (ICAD 2002). Kyoto, July. ROBERTSON, P. K., HUTCHINS, M., STEVENSON, D., BARRASS, S., GUNN, C., AND SMITH, D. 1994. Mapping data into Color Gamuts: Using interaction to increase usability and reduce complexity. Computers & Graphics, 18, 5, 653–665. SAUE, S. 2000. A model for interaction in exploratory sonification displays. In Proceedings of the International Conference on Auditory Display (ICAD 2000). Atlanta, April. WALKER, B. N. AND KRAMER, G. 1996. Mappings and metaphors in auditory displays: An experimental assessment. In Proceedings of the International Conference on Auditory Display (ICAD 1996). Palo Alto, CA, November. WALKER, B. N., KRAMER, G., AND LANE, D. M. 2000. Psychophysical scaling of sonification mappings. In Proceedings of the International Conference on Auditory Display (ICAD 2000). Atlanta, July. WILLIAMS, S. 1994. Perceptual principles in sound grouping. In Auditory Display: Sonification, Audification and Auditory Interfaces, G. Kramer, Ed. SFI Studies in the Sciences of Complexity, Proeedings vol. XVIII. Addison-Wesley, Reading, MA. Received March 2005; revised June 2005; accepted July 2005 ACM Transactions on Applied Perception, Vol. 2, No. 4, October 2005.
