paper
global string a musical instrument for hybrid space Atau Tanaka, Bert Bongers
[email protected],
[email protected]
Abstract Creative and technical considerations in building a musical instrument to traverse acoustical space and network space. Conceiving a musical instrument for heterogeneous space and democratic use requires implementation of diverse modes and techniques satisfying needs of tactile local presence, and tangible telepresence. The result is an artistic project destined for multi-site gallery installation and performance. It is a musical instrument that exists in the mixed realities of acoustical space and network space.
transmitted to the other endpoint of the string, an identical steel cable. Striking the string on one side makes it vibrate on the other end.
Keywords: Network music, sensor instrument, gallery installation Project URL: http://www.sensorband.com/atau/globalstring/ Year the Work was created: 1998 -2000 Project Partners: GMD, WDR, The Daniel Langlois Foundation, V2, Ars Electronica Center
Introduction 1.1 Overview Global String is a multi-site network music installation. The idea was to create a musical string (like the string of a violin or guitar) that spans a large geographical distance. The installation consists of a steel cable (12mm diameter, 15m length) connected to a virtual string on the network. The
Figure 1. (see above) Global String at V2/AEC
real part of the string stretches from the floor diagonally up to the ceiling of the space. Vibration sensors translate the physical vibrations to network data. These vibrations are
177
Sound is synthesized in real time by a software algorithm implementing a physical model of a string of unreal proportions. Audio, video, and sensor data is streamed between the sites, providing a live connection between the players. The system made up of these components conceptually comprises a single vibrating string. It is a musical instrument where the network is its resonating body. This paper discusses conceptual and technical issues confronted during development and realization of the project.
information is derived from a piezo transducer mounted on an aluminum block attached to the string. This signal is amplified, then enters the audio analog-digital converter (ADC) of the computer (sampling rate = 44,100hz, 16 bits). This signal is utilized as an audio rate control signal and for excitation of the audio synthesis engine. Low frequency data is obtained from Hall effect sensors [4], that translate magnetic field perturbations into control voltages. Two small neodymium magnets are attached to the string, mounted in the on-string aluminum block, facing Hall effect detectors mounted in orthogonal positions on a housing around the string. Two sensors are used, one for each degree of freedom (DOF) of the string - movements on horizontal and vertical axes. In the mechanical construction room was left for doubling this system, but in practice one sensor for each DOF proved to be sufficient. These voltages enter a sensor interface, and capture movements of the string from 0 Hz (DC) to a software-limited maximum (250Hz). From the 12-bit value, the most pertinent 7-bits of data are extracted. 2.2 Actuator
Figure 2. Technical overview
1.2 Hybrid space Global String exploits network infrastructure to connect two (or more) parallel physical acoustical spaces. As the vibrations of the string traverse network space, one can think of Global String as a stringed instrument that is made up of heterogeneous materials, or mediums. Change of medium for signals typically requires transduction from one medium to the other. The bridge of a violin is the transducer through which the string vibrations pass to resonate the wooden body. A speaker coil and cone are the transducer that permits audio signals in an electrical cable to become acoustical sound pressure waves. In Global String, a series of sensors and encoders transduce vibrations from the physical part of the string to the network part of the string and back.
An actuator translates electrical signals into perceivable physical events. The Global String uses an electromagnet to generate tactual cues to the player touching the string, creating a relationship to actions performed at the remote end. The large electromagnet is excited in pulse mode with a high DC voltage. This pulls the string by an iron plate mounted on an aluminum block, generating a palpable pulse in the string, resulting in a haptic sense of telepresence. 2.3 Audio synthesis Following the sensors, the transduction process is completed by data encoding. The sum total of actions as detected by the multi-sensor set is treated as excitation and parametrization of a real time physical model synthesis algorithm [5] to generate the sound of the string. The model is a mathematical formula simulating a vibrating string of the total proportions a length of the geographical distance between the two galleries.
The use of the Internet in this case is as a transmission medium and not as a storage or information system. It is a resonating chamber, and not a canvas [1 ]. The instrument cannot be accessed from the web. It is not a means to connect network surfers with people in the gallery. It is a single musical monochord [2, 3] that traverses physical space and network space, and in doing so, connects people in remote locations.
Methods The Global String as an instrument uses several sensor and actuator systems to interface multiple the player and the computer. Following the classical control loop model as described in [4], several stages can be discerned and are described here. The choice of audio synthesis technique and network protocol is outlined. 2.1 Sensing Global String implements two sensor systems, one detecting rapid action on the string and the other following slower gross movement. These reflect two distinct modes of player interaction with the string. The igh frequency
178
Figure 3. Sensor housing
2.4 Videoconference Control data from the piezo sensor is multiplexed with local audio and video. The information enters a standard H.323 codec (encoder/decoder) [6]. The H.323 standard was chosen as it is designed for bi-directional audio/visual transmission over IP (Internet Protocol). H.323 is an integrated solution that assures synchronous sound and image transfer. The video image received from the remote site is projected and serves to visualize the far end of the string. Received audio data is treated in two ways - 1. to be processed for peak detection analysis, and 2. to enter the local audio amplification system.
2.5 Display Audio signal visualization is projected by a second video projector alongside the videoconference image. The analysis consists of a real time amplitude envelope and FFT frequency spectrum of the sound from the two sites. In addition to audio data, network conditions are visualized as readout from the Unix traceroute command [7], mapping the packet trajectory and time from the local site to the remote site
Results The hybrid nature of the instrument posed the dual problem of creating a sense of remote presence while preserving a sense of local materiality [8]. Remote information is coherent only if aural and visual channels are synchronized. Local information is successfully parsed by the player only if there is a tight causal connection and immediacy in physical presence and action. The 15m steel cable establishes a monumental physical presence for the instrument, but is only one part of the total string. The sound of the Global String is not the acoustical sound of the steel cable, but the sound of the physical model. The challenge was to create an immediacy where this reality/representation dichotomy is rendered musically coherent. 3.1 Server-side vs. local synthesis The original concept implied that the physical model synthesis should be one model running in network server space. This however posed problems of local immediacy. Server side synthesis suffers from time latency retrieving and returning data from and to each site. Meanwhile the player needs an immediacy to keep haptic coherence with the physical string he beholds. Server side synthesis was abandoned in favor of a dual local synthesis in a closed loop dynamic. This afforded the immediacy - the string reacted instantly upon touching the steel cable, and yet provided the remote sense, as audio and sensor data streamed in from the remote site. Synthesis at each site was realized based on local sensor data. High frequency transients from the piezo sensor became the excitation signal feeding the physical model synthesis algorithm. Data from the Hall effect sensors were mapped to parameters describing the virtual physical makeup of the synthesis. The data packet traceroute data between sites was considered for use in scaling the length parameter of the string synthesis [9]. Piezo data from the remote site undergoes peak detection analysis and activates the local actuator. A strike on the remote end causes the local end to vibrate. 3.2 Telepresence Visual contact with the remote player complements the audio connection. While most commercial audio/video streaming solutions place priority on the visual channel, the Global String has the opposite priority. As a musical project, sound quality is more crucial than image quality [10]. It was possible to imagine, then, the choice of unequal systems such as a simple webcam for images and high quality audio streamer. Separate webcam and audio streaming combinations were considered and rejected, as they do not synchronize the audio and visual channels - an fundamental quality for remote performance. The need for audio/video synchronization was overriding to capture and convey physical and musical gesture. We have found in practice that low bandwidth connections to the internet (< 256kbps) tend to experience more significant data bottlenecks that risk to cause dropouts and failure of Quality of Service (QOS) [11]. The ideal system, whether standalone or PC based, is one that accepts external camera input, and accepts external stereo line level audio.
Discussion Global String is not a utilitarian device, but a musical instrument that encompasses the technology described above. As the technology is not brought to the fore, the apparent simplicity could be misconstrued in presentation. The naïve visitor could perceive the steel cable to be the total extent of the string. The physical model audio synthesis is natural sounding enough that it could be misperceived by the untrained ear as simply being the sound of the big cable. If quality of the projected image of the remote site is sufficiently high, it could easily be misconstrued to be a documentary videotape of the string at another time or another exposition. The challenge arose then, without resorting to demonstrative techniques such as auditioning the sound of the piezo pickup alone, or labeling the remote screen as such, how to give an intuitive sense to the casual exhibition visitor of the whole instrument. 4.1 The sound of one string… The issue arose that the choice of local synthesis defeated the concept of a single string. Generating sounds separately at each site risked to break the "one string" concept that is the fundamental basis of the project. The solution proposed here maintains the single instrument dynamic while providing the local haptic response. The analogy can be made of an electric guitar whose sound is amplified by the mixing of more than one pickup along the length of the guitar. The distance of each pickup relative to the bridge determines the harmonic content in sound of that pickup. Pickups closer to the bridge tend to be brighter in sound. Different pitches also create varying nodal vibration patterns that create a location dependent frequency response for the pickups. The use of two sets of multi-sensors, one at each end, to synthesize the sound of the string is akin to this multi-pickup approach. The physical modeling algorithms running at each site utilize the same parameters as a point of departure, so would generate identical sounds. The sounds diverge, as input data exciting the model and sculpting the parameters differ. The traceroute data too is not identical - the number of hops and packet delay time from one side to the other is not at all the same as for the opposite direction. This is consistent with the theory that a vibrating string can be modeled as a bi-directional delay line [5, 12]. The stereo audio capability of the videoconference system is utilized as two discrete channels - one to transmit the sound of the remote synthesis, and the other to transmit the sound of the raw remote piezo, to be used locally for analysis and trigger of the actuator. In such a configuration, all local and remote performance data is combined into a single synchronous bidirectional H.323 connection. 4.2 See you, hear me With the choice of a proper videoconferencing system, the work of adapting it to musical use begins. External video input to the system is useful for the connection of a video camera with adjustable zoom lens. The installation of the string is such that the camera will be as far as 10m from the performer it is aimed at. The camera orientation and projection screen position are of utmost importance to facilitate musical communication and the sense of continuity of the string as one entity. The data transduction from physical to network space can be successful only if the mediation in physical space is correctly implemented. In the case of Global String, this entails arranging the physical orientation of camera and monitor to faithfully recreate the sense of playing along one string. The orientation should preserve eye-eye contact between the remote players [13, 14], and allow the
179
local player to envision the remote end of the string as a natural extension of the local end. The camera and projection screen should coincide in position. They are placed at eye level beneath the high point of the string, aimed back towards the base of the string. The performer then plays looking towards the high part of the string - in effect, looking at the other site of the string. If the camera/screen position at the two sites is consistent, this gives a visual continuity based on the arch of the string. The performer sees the local end of the string go up, and sees onscreen the other end of the string coming down, creating a visual coherence between real and virtual space. The eye to eye contact creates the communicative immediacy across the network distance. 4.3 Machine states Whereas in most videoconference applications the goal is to provide a seamless remote communication where the network becomes a transparent intermediary, this is not the case with Global String. Inasmuch as the network, and therefore the string, becomes the conduit of musical communication between remote locations, it is not meant to be completely transparent. If audio quality from the remote side was not ideal, this can be said to reflect the distant nature of the far end. The goal is to create an awareness of the network as an visceral entity equal to the imposing structure of the physical string. The FFT projection being the sum total of sounds of both ends serves to visualize the shared action nature of the instrument. Although the performer will see the projection react as he hits the string, this is not the only movement in the image - it is also moving in response to sound from the other side. This gives evidence that the local performer's actions are contributing only part of the total sound of the string. The traceroute readout establishes visual evidence of the varying organic nature of the network, as data packets take different paths and different transmission times from one side to the other, representing string length that is continuously varying. 4.4 Feedback The actuator circuit provides physical manifestation but is not configured to give simple tactile feedback. Instead the actuator circuit functions to give one player a tactile sensation of what the remote player is doing. As for tactile feedback, this takes place naturally, as the physical part of the instrument is a massive steel cable. Feedback loops do exist in the instrument, as sounding on one end causes vibration on the other end. What keeps the actuated vibration from again causing the opposite end to vibrate, resulting in an infinite loop? The separation of the low and high frequency vibration sensing circuits helps to avoid the problem of infinite feedback. It is the high frequency channels of the remote end that are analyzed to cause pulsing of the actuator on the local end. The resulting movement of the string is as if the string had been tugged from the other end - a low frequency effect. The result of the actuator is then picked up by the low frequency sensors, but not by the high frequency sensors. This avoids the actuator on the remote end to be re-triggered.
180
Conclusion Through this amalgamation of diverse technologies, the objective was to create a musical instrument. As an instrument, it should be musically expressive in the hands of the player [4] be they a non-musician or virtuoso [15]. With Global String, we sought to span this gap with the notion that a rich musical instrument should respond to any of these situations or contexts in a musical way. The goal, then was to create an instrument that was accessible without being a demo or game, and at the same time challenging for the experienced performer. It should respond to a wide range of human input and still sound musical. It should also have the possibility to respond to non-human input, such as network traffic data, to generate sound and physical action. The choice to exhibit Global String as an installation was partly to address these issues. The instrument is in public space, inviting the casual visitor to try it. A simple touch will produce sound. The more inquisitive visitor might find other ways of hitting the string, see that it responds accordingly. The visitor may have the luck to have someone on the other side to play with. If not, the traceroute and FFT data may give an indication to the astute viewer that the string exists beyond the local physical space. The audio synthesis model has parametric input for external data, causing the string to make an ambient sound - inviting the visitor to approach the string. The ambient sound from the remote site gives the player a sound from afar in the absence of a remote partner. This continuous nature of an installation is well suited for the temporally imprecise nature of the network [16]. The string is always there, vibrating, checking traceroutes. At specific moments during the exhibition, performances can be organized. Suddenly this timeless public space becomes a stage, an event space. Having a remote partner is no longer a question of chance, but a rendezvous for performance. In the hands of the artists who created the instrument, its sonic depth becomes apparent. These are the virtuosi of the instrument who are able to exploit the expressiveness of the instrument. The original motivation of the Global String was to create a musical instrument that spanned a wide geographical distance. The project brought together considerations of instrument building, interface design, and network architecture. The design is polyvalent and multimodal in nature. The instrument exists across time and across multiple mixed spaces.
Notes Global String was conceived in collaboration with Kasper Toeplitz. It was developed and realized with the participation of Theo Borsboom, Frederic Voisin, Helmut Schaefer. The project was awarded 2nd prize at Cyberstar98: Shared Visions, organized by the GMD and WDR. It was produced with the financial assistance of the Daniel Langlois Foundation for Art, Science and Technology. It was premiered at the Dutch Electronic Arts Festival (DEAF), 2000, between V2 in Rotterdam and Ars Electronica Center (AEC) in Linz. It has also been presented during the Science + Fiction festival organized in 2001 by C3, held at Trafo in Budapest, connecting to AEC.
References [1]
A. Broeckman, personal communication, 2001.
[2]
H. Helmholtz, On the Sensations of Tone. Dover Pubns, New York, 1954.
[3]
J. R. Pierce, The Science of Musical Sound. W H Freeman & Co., New York, 1992.
[4]
A. J. Bongers, "Physical Interfaces in the Electronic Arts Interaction Theory and Interfacing Techniques for RealTime Performance," Trends in Gestural Control of Music. M. Wanderley, M. Battier (eds.), IRCAM, Paris, 2000.
[5]
J. O. Smith, "Physical modeling using digital waveguides," Computer Music Journal, 16(4), 1992.
[6]
Recommendation H.323 (11/00) - Packet-Based Multimedia Communications Systems http://www.itu.int/itudoc/itu-t/rec/h/h323.html
[7]
V. Jacobson, Traceroute OpenBSD 2.6 Man pages. Berkeley Software Design, Berkeley, 1993.
[8]
A. J. Bongers, "An Interview with Sensorband", Computer Music Journal, 22(1), 1998.
[9]
C. Chafe, Ping, http://wwwccrma.stanford.edu/~cc/sfmoma/topLevel.html
[10] B. Letort, Musiques Plurielles. Ed. Balland, Paris, 1998. [11] W. Woszczyk et al., "Networking Audio and Music Using Internet2 and Next-Generation Internet Capabilities," AESW P—1001 Technology Report TC-NAS 98/1, 1998. [12] C. Sullivan, "Extending the Karplus-Strong Algorithm to
Synthesize Electric Guitar Timbres with Distortion and Feedback," Computer Music Journal, 14(3), 1990. [13] A. Tanaka, “Network Audio Performance and
Installation.” Proc. Intnl Computer Music Conf ICMA, San Francisco, 1999. [14] D. Konstantas, et al., "The Distributed Musical Rehearsal Environment," IEEE MultiMedia, 6(3), 1999. [15] A. Tanaka, "Musical Performance Practice on Sensor-
based Instruments," Trends in Gestural Control of Music. M. Wanderley, M. Battier (eds.), IRCAM, Paris, 2000. [16] A. Tanaka, "Relier l'espace matériel et l'espace
immatériel: les projets de musique en réseau," Dossiers de l'audiovisuel, No. 96, INA, Bry-sur-Marne, 2001.
181
