(electronic) system and audience, involving various modes of communication. ..... AN EXAMPLE. In electronic arts, many interaction-modalities can be used.
Exploring Novel Ways of Interaction in Musical Performance Bert Bongers Rijksakademie van Beeldende Kunsten Sarphatistraat 470, 1018 GW Amsterdam, The Netherlands phone: +31 20 5270300 fax: +3 1 20 5270301 e-maiI bertbon@xs4alM
ABSTRACT
There are many interactions possible between performer, (electronic) system and audience, involving various modes of communication. In this paper, a concise overview and theoretic fiamework based on research mainly carried out in the HCI field (Human-Computer Interaction) is described, illustrated with a research project in musical performance and interactive technology. The approach described focuses on the physical interaction between people and systems, rather then the interactive behaviour as a result of machine cognition. Keywords
Interaction, HCI, musical performance, touch, feedback, audience participation
tactile
INTRODUCTION
Real interaction is a living two-way process of giving, receiving and giving back. In a traditional performance set up the audience is passive, the performer active. The increasing use of “audience participation” in a traditional concert setting acknowledges the need but does not address the issue in any depth - typically the situation created is one of “reaction” not “interaction”. It would be interesting to create a situation where the audience and performer meet, each influencing the other, as if conversing, while maintaining the quality of the performance at a high level.
articulate the control, or feed-forwardto actively guide the user. Feed forward can be generated by the system to reveal information about its internal state. In this paper, interaction is grouped in three categories: pet-former- system (e.g. a musician playing an instrument), System (e.g. installation art), and performer system - audience. The categories will be illustrated by projects where the author was involved in as an instrument builder. Performer
- System
The diagram below in Figure 1 illustrates the classic human-machine interaction loop. (By personal definition the machine is square and the human is round.) The system, or ‘machine’ in the diagram, is defined very wide. It can consist of several linked elements or devices, as is often the case with computers through networks and protocols like MIDI (Musical Instrument Digital htetiace, the language through which synthesisers, computers etc. can communicate). The ‘system’ can also refer to a musical instrument. Human \
Interactbn \
Machine /
INTERACTION
Interaction between a human and a system is a two way process: control and feedback. The interaction takes place through an interface (or instrument) which translates real world actions into signals in the virtual domain of the system. These are usually electric signals, often digital as in the case of a computer. The system is controlled by the user, and the system gives feedback to help the user to permission to make digital or hard copies of all or part of this WW~for personal or classroom use is granted without fee prwided that copies are not made or distributed for profit or conmercial advantage and that copies bear this notice and the full citation 011the first Page. ‘I’0 COPY otherwise, to republish, to post on servers or to redistribute to lists. requires prior specific permission and/or a fee. Creativity & Cognition 99 Loughborough UK Copyright ACM 1999 l-581 13-07%3/99/10...$5.00
Fimre
I: the Human-Machine Interaction
An interaction-‘loop’ may start when the user wants to activate the system. The system is controlled by a user
76
through its inputs, it processes the information, and displays a result. For instance, when the user pressesa key on a computer keyboard, a character is displayed on the screen. Or, in the case of an electronic musical instrument, a sound is displayed through the loudspeakers after a key is pressed. The human perceives the information from the system, processesit and controls again, Note that in some casesonly parts of the loop can occur, for instance when the cognition is left out on one side or the other this part rather reacts than interacts. Many interactive systems in new media arts are in fact reactive systems. Ideally, interaction between a human and a system should be mutually influential. Sensors are the sense organs of a machine. Through its sensing inputs, a machine can communicate with its environment and therefore be controlled. A sensor converts any physical energy (from the outside world) into electricity (into the machine world). There are sensors available for all things perceivable by human beings, and more. For instance, kinetic energy (movement), light, sound, but also properties unperceivable for human beings can be sensed such as electromagnetic fields and ultrasonic sound. Machine output takes place through actuators. Actuators are the opposite of sensors, i.e., they convert electrical energy into energy forms perceivable by human beings. For instance, a loudspeaker converts electricity in changes in air pressure perceivable by the human ear, a video display shows images perceivable by the eye, motors or vibrating piezo elements may addressthe senseof touch. The interaction usually takes place by means of an interface (instrument). Following the definitions of the diagram, the interface is part of the system or machine and consists of the sensorsand actuators.
are: vision, smell, taste and touch. Within each sensory modality other modalities can be distinguished, for instance the visual modality can be used for reading text, reading a musical score or watching a movie. Human output modalities are mainly involving the motor system; muscles are used to move things around but are also used for finer tasks such as handwriting and speech. A special case is bio-electricity, such as electromyographic signals (EMG) as measured on the skin which are related to muscle activity, and electroencephalographic signals (EEG) related to brain activity (e.g. alpha waves). These signals can be can be measuredby electrodes and used to control a system, a good description can be found in a paper by Zimmerman et al, [19]. This is used for instance by Atau Tanaka of Sensorbandfor musical performance [9], [16]. Some activities or interactions can be multimodal [ 131, in fact, most activities in everyday life involve multiple modalities. For instance, eating may involve a combination of tusk (the four tastes of sweet, bitter, sour or salt), smell and touch (texture, form, softness). The sense of touch is actually a multimodal stimulus, as it involves two systems: tactile (through different types of mechanoreceptors in the skin, which is also sensitive to pain, temperature etc.) as well as proprioceptive (through mechanoreceptorsin the muscles, tendons and joints) [ 151, [17]. Our &aesthetic sense, which is the awareness d movement / tension, and position / orientation of limbs and parts of our body, largely depends on the information from the proprioceptors. The combination of these senses is often referred to as tuctuul perception [ 111. Virtuality
In addition to the diagram in Figure 2, it is relevant to emphasise the distinction between active and passive system feedback.For instance, when pressing a button on an electronic device, one may feel and hear the mechanical ‘click’ regardless of the state of the device. The device can even be turned off and still the feedbackis perceived. This key click can be called passive or inherent feedback, i.e. not generatedby the system. In the case of a synthesiser, the sounds produced by the system as a result of the user action are active, i.e. generated by the system (controlled by the performer). Synthesised experiences can also be called virtual, as opposed to real (mechanical, inherent or passive) experiences. In the project described towards the end of this paper, the emphasis is on novel forms of active feedback. An electronic system (often a computer) interface generates things that, through its output devices, can be perceived by humans. These things can be called ‘virtual’, because they are not really there but can be perceived. They can be described as experiences. For example, one can listen to sounds generated by a machine (synthesised, or sampled and played back). Another example are computer generated images. In order to synthesise a more convincing experience, multiple senses of the human could be addressed. These multimodal interfaces [3] enable users to hear, see and feel
Modalities
Several interaction modalities, or communication channels, can be distinguished. Modalities are closely related to perception and motor control: the eyes are important for the visual modality, the haptic modality involves the movement control of for instance the human hand, guided by vision and proprioception. Listening to a CD involves the hearing modality. Other examples of sensory modalities Human \
lnteraction \
Machine /
77
the virtual world and influence it in many ways including for instance speech.. The importance of the sense of touch in traditional musical instruments, as well as in tools used for other art forms, is well known. Due to the de-coupling of control and feedback in electronic systems, this form of information about the process is often lost. Ultimately, the ‘feel’ has to be synthesised artificially, just as the sounds and images are synthesised by such a system. This is described in other
In the Netherlands, Michel Waisvisz, director of STEIM (Studio for Electra-Instrumental Music) devised instruments like The Hands and The Web (as pictured) to explore the possibilities of new instrument forms for improved gestural and sensitive control over the sounds produced.[lO]
papersPI, VI. Electronic
Musical
Instruments
The communication between a musician and his or her instrument is often intimate and precise. This has been the case with traditional (acoustic) instruments, and developers of electronic instruments are aware of that. An early example is the Theremin, developed by Leon Theremin in the 192Os,that changed pitch and volume of the sound by the subtle movements of hands waving in the air near the instrument -without touching it- through its antennas. [6] The American synthesiser builder and pioneer Robert Moog worked with extra sensors on the keyboard to enable the player to have more influence on the sound. [12]. Others would experiment with entirely ne instrument forms, for instance Laetitia Sonami from Oakland (CA) uses a glove fitted with a variety of sensors to enhance control (see the picture below).
Fimrre 5: The Web
Another example is the English composer and performer Walter Fabeck, who plays his Chromasone, and Sensorband [4], [14], a trio of musicians all playing on novel instruments. System
- Audience
In the case of an installation work (or a CDROM or web site based work), one could say that the artist communicates to the audience displaced in time. An extreme example of this is a CD, where the perfomrer instead of playing notes directly to the listener records it first. The audience then listens to it at another time. The artist’s actions, intentions and ideas are built in the system, therefore the artist is left out of the interaction diagram in Figure 6.
Fimrre 3: The Ladv’s Glove
Interaction /
Audience/
F&w-e 6: the interaction between audience and installation
FiPure 4: A uair of Hands
78
Interaction between the work and the audience can take place in several ways or modalities. Usually a viewer pushes buttons or controls a mouse to select images on a screen,or the presenceof a person in a room may influence parameters of an installation. The level of interactivity should challenge and engage the audience, but in practice ranges from straight-forward reactive to contisingly overinteractive. Performer
- System
AN
EXAMPLE
In electronic arts, many interaction-modalities can be used playing an instrument and involving the audience, some of which are relatively new and unexplored.
- Audience
In (musical) performance, there can be two active parties: the performer(s) and the audience. Apart Corn the direct interaction between the parties, performer and audience can communicate with each other through the system. The system may facilitate new interaction channels. Therefore, two kinds of interaction with the system can be distinguished: . the interaction between the performer and the system . the interaction between the audience and the work performed through the system In other words, the performer communicates to the audience through the system. For instance, a musician playing an instrument, and the output of the instrument (the sound) goes to the audience. The audience can (and often does) participate by (even subtle and non-verbal) communication directly to the performer(s), which may influence the performance. The diagram in Figure 7 shows the possible communications, both the interaction through the system as well as direct interaction (the large arrow below in the diagram).
F&we 8: The Chromasone The project described here is undertaken in conjunction with performer, composer, and inventor of the Chromasone instrument Walter Fabeck, and visual artist and composer
Audience
79
Performer
Interaction \
\
System
Interaction /
Audience /
Interaction F&w-e 9: The target modalities Yolande Harris from England. The Chromasone is an electronic musical instrument, a gestura] controller based on a pair of datagloves and a keyboard-like template. Figure 8 (below) shows the instrument, and it is further described in an article [ 181and on the web site [7]. The project aims to explore the application of the sense of touch as a means of addressing the audience as well as the exploration of active audience participation. In the diagram above some potentially interesting interaction modalities are suggested, the bold arrows indicate the ones subject to exploration by this project. This will be further described in the next paragraphs. Touching
the
feeling sound without hearing it. One such device is the Aura Interactor Cushion, which delivers tangible feedback in the back of the person leaning against it in the chair. Members of the audience can be touched individually, the audience can be ‘locally shivered’. A picture of the device is shown is Figure 10.
Audience
The level of multimodality (or amount of senses of the audience addressed) generally depends on the kind of performance. Slightly simplified, we can state that the experience of performance of a Dvorak string quartet is largely by ear, and somewhat by sight. A dance performance can be equally about sight and sound. The experience at a techno club is largely by sound and visuals, but definitely involves the sense of touch where the music can be felt owing to the massive power of the PA (Public Address) systems used. The Austrian duo Granular Synthesis explores this in perfomances with subharmonic sounds. In addition to public addressing the audience through sound, Personal Addressing (of each member of the audience) is explored as well as the decoupling of the addressing of the senses. Various technologies (enabling technologies from Virtual Reality industry [I], [8]) are now becoming available to produce sensations of touch, i.e.
F&w-e IO: the Interaction Chair urototwe Initial experiments with this tangibie feedback system are promising, further research to explore this new and powerful form of communication is taking place in this project. It is
80
3. A.J. Bongers, J.H. Eggen,, D.V. Keyson and S.C. Pauws, Multimodal Znteraction Styles. HCI Letters l(l), 1998, pp 3-5 4. Bert Bongers, An Interview with Sensorband. Computer Music Journal 22(l), 1998, pp. 13-24 On line at http://mitpress.mit.edu/joumals/COMJ/samplearticle.html. 5. A. J. Bongers, Tactual display of sound properties in electronic musical instruments. Displays Journal 18(3), 1998 pp. 129-133 6. Joel Chadabe, Electric Sound, the Past and Promise of Electronic Music. Prentice Hall, 1997. 7. Web site Walter Fabeck describing the Chormasone; http://www.chrom.demon.co.ukl 8. Roy S. Kalawsky, The Science of Virtual Reality and Virtual Environments. Addison-Wesley, 9. R. Benjamin Knapp and Hugh S. Lusted, A Bioelectric Controller for Computer Music Applications. Computer Music Journal 14(l) 1990. pp. 42-47 lO.Volker Krefeld., The Hand in The Web: An Interview with Michel Waisvisz. Computer Music Journal, Vol. 14, No. 2, pp. 28-33, 1990. 11. J. M. Loomis and S. J. Leederman, Tactual Perception. Handbook of Perception and Human Performance, Chp. 3 1. 1986. 12.Robert A. Moog and Thomas L. Rhea, Evolution of the Keyboard Interface: The Bosendorfer 290 SE Recording Piano and The Moog Multiply-Touch-Sensitive Keyboards. Computer Music Journal 14(l) 1990. pp. 52-60 13. Lambert Schomaker, Stefan Mtinch and K. Harhmg, eds., A Taxonomy of Multimodal Interaction in the Human Znformation Processing System. Report of the ESPRIT project 8579: MIAMI, 1995. 14. Sensorband Web site: http://www.sensorband.com 15.C. E. Sherrick and R. W. Cholewiak, Cutaneous Sensitivity, in: Handbook of Perception and Human Performance, Chp. 12. 1986 16.Atau Tanaka., Musical Technical Issues in Using Interactive Znstrument Technology with Application to the BioMuse. ICMC Proceedings 1993, Tokyo, pp. 124-126, 1993. 17.R. T. Verrillo, Vibration Sensing in Humans. Music Perception, Vol. 9, No. 3, Spring 1992. pp. 281-302 18.Rob Young, meets Walter Fabeck, Inventor of the Chromasone. The Wire, issue 166, December 1997, PP 69 19.Thomas G. Zimmerman, Joshua R. Smith, Joseph A. Paradiso, David Allport and Neil Gershenfeld, Applying Electric Field Sensing to Human-Computer Interfaces. In: Proceedings of the CHI’95 conference, Denver, Colorado, USA, 1995. pp. 280-287.
a challenge for the composer to use this power in a sensible and creative way. A live electronic musical instrument like the Chromasone is particularly well suited to address this added modality, as it is based on gestural, spatial control. Audience
participation
To provide the audience with ways to directly interact with the system, their chairs will be equipped with sensors. Their actions will be displayed visually by a data-projector, and the images are interpreted by the performer. This way of audience participation, including the translation of the images to musical performance parameters is under exploration in the project. Figure 11 shows a screen image of the moving graphic score, as projected in such a way that they can be seen by both the audience and the performer. Each audience members’ actions will be individually acknowledged on a part of the screen, as instant feedbackby means of a kind of graph which could look like this: IEm
FiPure 1I: exam&e of a visual score and feedback items CONCLUSION
The use of electronic media offers many possibilites for new interactions, both in regarding modalities as well as activating parties previously playing a passive role. Further studies in the project outlined in this paper are needed to investigate how an audience experiences the interaction, as well as experimenting with new technologies. REFERENCES
1. Steve Aukstakalnis and David Blatner. Silicon Mirage, The Art and Science of Vritual Reality. Peachpit Press, Inc., Berkeley, CA, 1992 2. Bert Bongers, The Use of Active Tactile and Force Feedback in Timbre Controlling Musical Instruments. In: Proceedings of the International Computer Music Conference (ICMC) in Arhus, Denmark, 1994. pp 171174
81
