implementing a temporal phenomenology in an interactive music system

Real times: implementing a temporal phenomenology in an interactive music system Jonathan Impett Faculty of Music, University of Cambridge [email protected] Abstract This paper examines the issue of temporal decision-making in the context of an interactive music system. Conventional views of the time-frames of musical activity, composers' rationalisations, models of jazz improvisation, and the temporal dimensions of other interactive systems are discussed. Central to the study is the question of the distribution in time of the various factors which contribute to a particular musical phenomenon. It is proposed to model this "space of mediation" as a dynamical system, in a development of the arguments of Prigogine and Boon (1998). The architecture of an interactive system currently being developed is based on an adaptive complex of such systems, its activity in time a function of their aggregation and modulation.

1. Introduction Time would appear to be the single common domain in which all music happens and by which it might be understood. As in any field, the mode of our understanding of musical time is governed by the nature and purpose of our questioning. This paper is concerned with the management of time in an interactive music system, by which is here understood a system which instantiates a work through the interaction of musician, instrument / interface / environment, and some computational embodiment of the composition. "Real time" is an epithet often applied to such systems, and begs the question of just how "real" or instantaneous the temporal dimensions of their behaviour can be - the extent to which it can be situated entirely in the present moment. In the context of such a universal, unifying parameter, what is at first remarkable about these various modes is their apparent incommensurability even though the various taxonomies that have been proposed can be reduced to a space of limited dimensions. The shape of this space will be shown to be the very quality of musical activity that must be challenged in order to realise the potential of the mode of music making being proposed. There remains, nevertheless, a sort of quantum effect of rhythm: that the more precisely we try to define the dynamics determining the happening or non-happening of a certain musical event at a certain moment, the less we can be sure that it will actually happen. Perhaps this is a function of the circular, mutually-defining relationship of music and time to which Kramer (1988) draws attention. The construction of an approach to this issue in the context of an interactive music system is the subject of this paper.

2. Usual times 2.1 Conventional time spaces Most discussions of time in music divide fairly precisely along the culturally defined fault-lines of musical practice:

composition->work->performancereception Note that in this - admittedly gross taxonomy, the work stands as an impenetrable monolith between composition and performance or reception, through the medium of which temporal decision-making is passed unidirectionally and absolutely. The path between performance and reception is one-way, except in as far as feedback mechanisms are incorporated for explanatory completeness. Whilst by a meta-theoretical stretch of musicological imagination, we could create "higher" levels for work dealing with compositional practice or the evolution of style or genre, in truth such investigation is rarely concerned with the phenomenological detail of the activities thus engendered. Rather, the opposite is the case; most accounts of the business of composition - even within our own conventional culture - must lead us to question the validity of these divisions of activity. Psychologically-motivated discussions of time naturally focus on processes close to the musical present (for a review see Clarke, 1999). A very relevant exception is the work of Cohen and Katz, who examine the degree to which a musician's relationship to the various time-frames of his music is itself culturally constrained (Cohen and Katz, 1997). Composers tend to be vague as to the rationalisation of such a fundamental issue. Discussion usually centres on the potentially conflicting forces of continuity in experiential time and discontinuity in its formal structuring. Examples are Stravinsky's "ontological" and "psychological" times, the former governed by similarity, the latter by contrast (Stravinsky, 1942, p.31); Stockhausen's "experiential" and "structured" times (Stockhausen, 1955); the "inside-" and "outside-time" formalisms of Xenakis (Xenakis, 1992); and Saxton's distinction between "narrative" and "geometric" modes of composition (Saxton, 1998). Attempts by composers to expand their formalization of time by exploiting the physical continuity between frequency and duration (Stockhausen, 1957) have led to suggestions that this

artificially flattens two perceptually orthogonal axes (Clarke, 1989). Suggestions by eminent writers that this continuity might hold a key to the understanding of the unfolding of musical structure (Prigogine and Boon, 1998) should not be ignored, however. The multifaceted nature of musical time is usually resolved by presenting it in terms of the activities performed upon it: time as measured (chronometric time), time as structured, and time as experienced. In his work on tempo and proportion, Epstein (1995) points to motoric and neuronal periodicity as the phenomena in which composed or formal time, the flow of time as it is experienced, and the measuring of time are grounded. Such periodicity enables both the generation and cognition of motion: "Motion is basically understood by using time as its index. Thus the common way of discussing motion in terms of temporal units…. The reverse of that correlation is equally true: time is only experienced, and thus understood, through motion. Motion is thus the quintessential property of time." Such a dynamic understanding will prove central to the approach discussed below. 2.2 Improvising time Jazz improvisation offers an example of realtime temporal structuring within the which the norms are sufficiently well agreed for generalizable analysis and modelling. We will look briefly at three studies, to distinguish them not by the timing detail they generate, but by the way in which the factors contributing to the occurrence of a certain event at a certain moment are themselves distributed through time. Johnson-Laird's computational model (Johnson-Laird, 1991) sets out from the premise that "The essential psychological feature of musical improvisation … is that the musicians themselves do not have conscious access to the processes underlying their production of music." (p.292) His architecture is cast in a hard-wired division between improvised surface and its underlying compositional structure, and assumes an absolute, non-evolving distinction between right and wrong. A context-free grammar first generates a rhythmic phrase, and then an appropriate set of pitches. It is difficult to see how such a precisely-delineated activity could learn the skill, play with the rules, or generate innovation in style or genre. Pressing's earlier model sought explicitly to address these issues: "Any theory of improvisation must explain three things: how people improvise; how people learn improvisational skill; and the origin of novel behaviour." (Pressing , 1988; p.152) A wider range of parameters contributes to the generation of patterns or "event clusters". These may include shortand long-term goals, stylistic constraints and the activity of other musicians, represented with variable strengths in three kinds of array: object, feature and process. The generative equation for each event cluster thus has a number of different "aspects", with

a high degree of redundancy. The decision to produce a new cluster is based a combination of long-term factors, such as stylistic norms and ongoing processes, and an evaluation of the implications of the present cluster. In effect, the model is expressed as a highly nonlinear filter, with components from multiple time scales. This permits the incorporation and dynamic management of factors on scales beyond the present phrase, or even the present performance, and the model can thus display both learning and the generation of novelty. That the identification and tuning of these parameters is highly subjective is in itself no disadvantage, but there is no mechanism for this to take place in a real-time context, on the basis of the actions of a performer; by that stage of development, the system would be atrophied in a purely algorithmic state. Whilst theoretically possible, extending the set of factors contributing to each cluster would entail an explosion of representations; this would bring problems of tractability and the need to hard-wire solutions to questions of commensurability. Pressing points to the inherently non-unique nature of any theory of musical generation (Pressing, 1998). Sarath's (1996) taxonomy of musical times is based on a more narrowly-defined, conventional set of activities - moment-to-moment improvisation, "extemporised composition", and composition - but these are examined in a wider range of contexts. From this framework he develops a theory of times which permits a more phenomenological approach to the understanding of a particular event. Improvisation and composition are polarised as inner-directed and expanding conceptions, spontaneous and discontinuous processes. To these he adds a "retensive-protensive temporality" (derived from Husserl,1964) in which awareness is projected both backwards and forwards in time from a moving present. This category is extended to be involved in every form of activity - but it is then difficult to imagine truly spontaneous improvisation, or a work that does not reflect to some extent the local temporal dynamics of the process of its composition. It does give a sense of the temporal mode in which a musical event might be said to exist. I would suggest that the "multi-layered temporality" which Sarath ascribes to composition should also be taken into account in an interactive context, in trying to build structural or stylistic sense on the basis of improvisation. 2.3 How real is real? Most interactive music systems work in event time; by and large, they react to notes by generating notes (Rowe, 1993; Winkler, 1998). The implicit assumption is that the system latency will be within the bounds of the musically-perceptual immediate. Microstructural, sample-to-sample decisions can then be taken pre-emptively, or devolved to other software or devices. MIDI is the standard protocol for such operation. That timing information of a much smaller

grain is vital to both the perception and performance of music is clear (Clarke, 1998), but any system that would operate in real time on such a scale must find contextual solutions to two related issues: 1) To maintain sample conversionrate accuracy, all systems use some sort of buffering (i.e. the actual calculation is performed ahead of time, and often, for reasons of efficiency, on a vector of a number of samples). 2) The perceived onset of a musical event might not coincide precisely with the beginning of the activity that produces it. This difference may be of the order of a percussionists arm-gesture, or of the fine detail of the way in which a violin bow sets the string in motion. Both of these latter examples, in their execution and their reception, are highly culturallyconditioned. It seems that the more closely we look at determining the moment-to-moment activity, the wider our vision must be. In practice, the temporal distribution of decision-making in live electronic music is usually conditioned ultimately by pragmatic considerations, as in Emmerson's local/field distinction, or the late works of Nono (Emmerson, 1994; Vidolin, 1998). In working on a larger temporal scale - that of generating musical structures - we are constrained by the unidirectionality of the musical present. To take a conventional example, Marpurg's instructions for composing a fugue involve a succession of calculations forwards and backwards in time, that would seem to be impossible to implement in a rigorously real-time environment (Marpurg, 1750-60). Simple interactive systems can avoid such problems by restricting themselves to the direct mapping or algorithmic transformation of input, or by using the input to pilot generative algorithms or stored material. Proposals for the real-time generation of richer musical structure include the further processing of system-generated material by listening agents (Rowe, 1993), a taxonomy of structures based on their "sightedness" or degree of temporal situatedness (Impett, 1996), and a hybrid, multiple-representation system for dynamic, contextual planning (Camurri & Leman, 1997). Two problems limit the generalizability of such systems: the rigid distinction between composition and performance (programmer and user) modes, and the need for multiple modes of representation and decision-making.

3. The redistribution of time 3.1 When do we decide? As I have tried to show elsewhere (Impett, 1998), with reference to Lyotard's notion of telegraphy (inscription at a distance; Lyotard, 1991), the essential quality of the new situation with which technology presents us is that the conventional distribution of the constituent elements of musical activity is no longer necessary or sustainable. Even within the canon of Western music, the compositional procedures of the icons of rationality belie the notion

that the structuring of time might take place independently of performance and reception: Palestrina (the very embodiment of rationalised horizontality) extemporising compositions at the lute; Beethoven's cycle of improvisation, transcription and reworking; Stravinsky's composing environment of piano and cut-and-paste; or Boulez' unending cycles of provisional performance and recomposition. Sarath's description of the activity of composition as a "discontinuous process of creation and iteration" (Sarath, 1996; 2) seems apposite. But does this not equally well sum up the coupled evolution of interpretation and physical dynamics (Shove and Repp, 1995), the evolution of an improviser's style and material (Radano, 1993; chap. 5), or the mechanisms of interaction amongst musicians (Smith, 1998)? Kramer characterises musical time by its degree of nonlinearity (Kramer, 1988). He suggests that composers should take advantage of the "immediacy of electronic improvisation" (p.79) to bring the act of creation into the musical present. His concern is with the metaphysical status of time in music; the issue here is more prosaic: given the infinitude of factors contributing to this nonlinearity, when do we decide what to do? 3.2 A homogeneous space of mediation Sarath takes issue with Nettl's description of improvisation and composition as "rapid and slow" instances of the same compositional activity (Nettl, 1974; 6). In reviewing the same practices more recently, Nettl observes the significance of "the identification of principles that are characteristic in the macrocosm and microcosm of performances, and at points between these extremes." (Nettl, 1998; 391). We need, then, to find a model that allows us to work with the points both between and beyond these extremes. The various processes which contribute to a particular musical phenomenon each have their own temporal dynamics, from the evolution of style to faster-than-consciousness learned reflexes. The determining itself, however, happens in their situated interaction - the very process of mediation (Adorno's "Vermittlung"; Paddison, 1993), we might dare to suggest - constantly, hierarchically and in parallel generating transformed dynamics for further interaction. In this state, such elements are not best understood as uniquely-defined fragments of musical material, but as dynamic, adaptive sets of bahavioural tendencies and potentials - "inventions", to borrow Dreyfus's Bach-derived term. The process of their interaction and mediation is available to us from the outside only at points of state change: in the transition from idle "play" to a mental construct or to the key to an as-yet-undefinable Idea; from inner ear to paper or instrument; in the symbolic combining or transforming of ideas. The list is endless. There is no priority or unique path through the space of all possible state-changes, even though we might identify

attractors on the part of certain practices or individuals. Each musical experience, seen from this point of view, is the result of a unique trajectory. How much of this trajectory it has in common with other phenomena is a function of the nature of the embodiment of the "work". The formative mediations may not all result in state-changes. What we can observe is a sort of Poincaré section of this activity; snapshots at specific times and locations in the space. Can we cleanly divide up the entire mediational space, or even that of possible statechanges into those relating to composition, performance or reception? In the general case, that would seem to be a disadvantage; an artificial construct creating barriers in the dynamics of mediation, in the very space within which our goal is to increase the flexibility of distribution. 3.3 Working with the bi-directional dynamics of musical time Dynamical systems theory provides the tools for working with such state spaces. Boon and Prigogine have examined the possibility of analysing musical works and characterising styles in terms of the dynamics of their constituent time-series (Boon and Decroly, 1995; Prigogine and Boon, 1998). In his earlier work, Boon constructs phase space portraits of the temporal dynamics of individual works, and seeks to define their dimensionality and complexity. This approach is then considered in the light of Prigogine's theories of dissipative systems. The musical work becomes an open system - one requiring constant energy input - creating order against the inevitable flow of increasing entropy. Perhaps in this idea we can find a tool for dealing with the paradoxical nature of musical time described by so many writers. Once again the "work" is misappropriated to build a barrier, however. "Il n'y a pas de continuité dans l'univers musicale: la musique nâit du silence et retourne au silence." (Prigogine and Boon, 1998). Where then is this external energy, this flow of entropy? Prigogine would perhaps better embrace Serres' notion of a "sea of noise": "Background noise is the ground of our perception … No life without heat, no matter, neither; no warmth without heat, no logos without noise, neither. Noise is the basic element of the software of all our logic, or it is to the logos what matter used to be to form. Noise is the background of information, the material of that form." (Serres, 1995; p.7). In other words, to take advantage of Prigogine's physical metaphor, our dynamical system must incorporate dimensions and parameters beyond both the immediate temporal scope of the particular work and its apparent sphere of determination - it must embody both the cultural and physical environments of the activity itself.

4. Parallelism and dynamical hierarchies in an interactive system

Characteristic of all the many modes of mediation which jointly determine musical phenomena are their parallel and hierarchical nature. Many temporal levels may be simultaneously instrumental, and within these are formed hierarchies which may be "natural" - such as the subsumption of low-level muscular kinetics within the individual - or cultural - as is the case with hierarchical tonal structures. Furthermore, these groupings themselves change over various time-scales. In the system currently under development, each potentially event-generating process is expressed as a separate dynamical system. This multi-agent architecture is implemented in the Swarm simulation environment (Minar et al., 1996), using the Objective C language. The run-time typing of this environment allows the system to dynamically reconfigure itself, such that individual micro-behaviours ("primitives") can constantly change their membership of multiple aggregate systems ("inventions", after Dreyfus, 1996). Pre-composed and real-time generated behaviours are both expressed in the same form and can aggregate arbitrarily. Aspects of the performance environment may constitute parameters of particular behaviours, or may be dynamically coupled. A system of flags - self-evolved or preordained - allows the small-scale behaviours to identify potential inventions within which they might become active (following Holland, 1995). The disposition of any primitive or invention to adaptation is expressed as a dynamic inertia value (after Hofstadter, 1995). Each primitive and invention has a tendency to periodicity - a period before which it would be unwilling or ill-behaved if it were expected to repeat, and after which it may be more or less anxious to do so. An action at a given moment in time is thus in effect the product of the modulation of many dynamical systems, all changing and interacting with their own internal dynamics. Behavioural regularities emerge as attractors on various levels of the system, not necessarily evident in the series of events generated. No planning or scheduling is necessary, as the "future" is both implicit and constantly updated in the situated dynamics of the system as a whole.

5. Conclusion The folding in of the musical subject to become his own material seems to entail a wider, more flexible view of the temporal processes that impinge on the musical present. In the computational environment of an interactive music system, these elements have to be taken into account and reconciled explicitly. A complex of adaptive dynamical systems allows them to be expressed and interact without the restrictions of conventional boundaries of activity. Cognitive, cultural, technological and environmental constraints can be embodied in the same terms. It is hoped to implement the system described here in such a way that its development, learning, composition and performance can be seen as variably-distributed instantiations of the same central, musical activity.

References Boon, Jean-Pierre and Decroly, Olivier(1995). "Dynamical systems theory for music dynamics." Chaos 5, 3: 501-8. Camurri, Antonio and Leman, Marc (1997). "AIbased music signal applications - a hybrid approach." In C. Roads, S. Travis Pope, A. Picciali and G. de Poli (eds.) Musical Signal Processing. Lisse: Swets and Zeitlinger. Clarke, Eric (1989). "Mind the gap: formal structure and psychological processes in music." Contemporary Music Review 3, 1:1-13. Clarke, Eric (1999). "Rhythm and Timing in Music." In D. Deutsch (ed.) The Psychology of Music (2nd. ed.). San Diego: Academic Press. Cohen, Dalia, and Katz, Ruth (1997). "Attitudes to the time axis and cognitive constraints: the case of Arabic vocal folk music." In I. Deliège and J. Sloboda (eds.) Perception and Cognition of Music. Hove: Psychology Press. Dreyfus, Laurence (1996). Bach and the Patterns of Invention. Cambridge Mass.: Harvard University Press. Emmerson, Simon (1994). "'Local/field: towards a topology of live electroacoustic music." In S. Brandorff (ed.) Proceedings of the 1994 International Computer Music Conference. San Francisco: ICMA. Epstein, David (1995). Shaping Time: music, the brain, and performance. New York: Schirmer. Hofstadter, Douglas (1995). Fluid Concepts and Creative Analogies. New York: Basic Books. Holland, John (1995). Hidden Order: How Adaptation Builds Complexity. Reading Mass.: Addison-Wesley. Husserl, Edmund (!964). The Phenomenology of Internal Time Consciousness. Trans. J. Churchill. Bloomington: Indiana University Press. Impett, Jonathan (1996). "Projection and interactivity of musical structures in Mirror-Rite. " Organised Sound 1, 3: 203-11. Impett, Jonathan (1998). "The identification and transposition of authentic instruments" musical practice and technology." Leonardo Music Journal 8: 21-26. Johnson-Laird, P.J. (1991). "Jazz Improvisation: A Theory at the Computational Level." In I. Cross, R. West and P. Howell (eds.) Representing Musical Structure. London: Academic Press. Kramer, Jonathan (1988). The Time of Music. New York: Schirmer. Lyotard, J.-F. (1991). The Inhuman. Trans. G. Bennington and R. Bowlby. Cambridge: Polity Press. Marpurg. G (1750-60). Handbuch bey der Generalbasse und der Composition. Berlin: Hände und Spener. Minar, N., Burckhart, R., Langton, C. and Askenasi, V. (1996). The Swarm Simulation System: A Toolkit for Building Multi-Agent Simulations. Available at: http://www.santafe.edu/projects/swarm. Nettl, Bruno (1974). "Thoughts on improvisation: a comparative approach." Musical Quarterly 60: 1-19.

Nettl, Bruno (1998)."Taqsim Nahawand revisited: the musicianship of Jihad Racy." In B.Nettl and M.Russell (eds.) In the Course of Performance: Studies in the World of Musical Improvisation, Chicago: University of Chicago Press. Paddison, Max (1993). Adorno's Aesthetics of Music. Cambridge: Cambridge University Press. Pressing, Jeff (1988). "Improvisation: methods and models." In J.A.Sloboda (ed.) Generative Processes in Music. Oxford: Oxford University Press. Pressing, Jeff (1998). "Psychological Constraints on Improvisation". In B.Nettl and M.Russell (eds.) In the Course of Performance: Studies in the World of Musical Improvisation, Chicago: University of Chicago Press. Prigogine, Ilya and Jean-Pierre Boon (1998). "Le temps dans la forme musicale." In E.Darbellay (ed.)Le temps et la forme. Geneva: Editions de l'Université. Radano, Ronald M. (1993). New Musical Configurations: Anthony Braxton's Cultural Critique. Chicago: University of Chicago Press. Rowe, Robert (1993). Interactive Music Systems: Machine Listening and Composition. Cambridge Mass.: MIT Press. Sarath, Ed (1996). "A new look at improvisation". Journal of Music Theory 40, 1: 1-38. Saxton, Robert (1998). "The process of composition from detection to confection."In W.Thomas (ed.) Composition- performance - reception: studies in the creative process. Aldershot: Ashgate. Serres, Michel (1995). Genesis. Trans. G. James and J. Nielson. Ann Arbor: University of Michegan Press. Shove, Patrick and Repp. Bruno H.(1995). "Musical motion and performance: theoretical and empirical issues". In J.Rink (ed.) The Practice of Performance: Studies in Musical Interpretation. Cambridge: Cambridge UniversityPress. Smith, Chris (1998). "A sense of the possible: Miles Davis and the semiotics of improvised performance". In B.Nettl and M.Russell (eds.) In the Course of Performance: Studies in the World of Musical Improvisation, Chicago: University of Chicago Press. Stockhausen, Karlheinz (1955). "Die Vielvalt schiesst zusammen: zu einem Stück erlebte Zeit: einem Stück Musik". Die Reihe 2. Stravinsky, Igor (1942). Poetics of Music. Cambridge Mass.: Harvard University Press. Vidolin, Alvise (1998). Musical interpretation and signal processing. In C. Roads, S. Travis Pope, A. Picciali and G. de Poli (eds.) Musical Signal Processing. Lisse: Swets and Zeitlinger. Winkler, Tod (1998). Composing Interactive Music: Techniques and Ideas using Max. Cambridge Mass.: MIT Press. Xenakis, Iannis (1992). Formalized Music: Thought and Mathematics in Composition. Stuyvesant NY: Pendragon.