Composing with Algorithms: Two Novel Generative Composition Tools Iain Foxwell
Don Knox
School of Engineering and Built Environment Glasgow Caledonian University UK ++ 44 (0)141 331 8436
School of Engineering and Built Environment Glasgow Caledonian University UK ++ 44 (0)141 331 8436
[email protected]
[email protected] a very personal experience for most musicians and the idea that a musical device can create something like a melody or chord progression without any input required could seem almost like µFKHDWLQJ¶ &ODUHQFH %DUORZ GHVFULEHG KLV RZQ WKRXJKWV RQ composing rather than relying on mathematics to do it for you ³HYHU\SLHFH,KDYHFRPSRVHGLQUHFHQW\HDUVLVDUHVXOWRIPXVLFDO inspiration. It is not a result of a desire to fumbles with formulaV´ [2]. The use of algorithms could potentially remove composers from this personal process [3]. The idea that algorithm use in music removes creativity from composition has been debated over the years [4,5], along with the concept that using technical and complex tools which require logical or mathematical thought that is distinct and counter to the required mindset for composing music. Thus it is important the composer remains in control of the creative process, their intuition and judgment when writing music is maintained [6], and any technological compositional aid should support these processes rather than seek to replace them.
ABSTRACT Most composers may at some point struggle with the creative process. Breaking frHH IURP ZULWHU¶V EORFN DQG WKH FRPSRVHU¶V own limitations can be achieved through the use of algorithmic approaches to music creation. This project is aimed at development of two algorithmic compositional tools, which can be used within the Digital Audio Workstation Ableton with Max for Live software. The first employs 2nd order Markov chains to aid composers in creation of unique midi melodies. The second approach uses Cellular Automata to select loop points in digital audio files, shuffling audio segments in time to create new patterns. Results suggest these systems have the potential to inspire composers, allow them to augment their compositions, and in turn help them overcome creative problems.
Categories and Subject Descriptors H.5.5 [I nformation I nterfaces And Presentation]: Sound and Music Computing.
The introduction of technology in music composition has given the modern composer a wealth of tools using algorithms, which could potentially be used to break free of their own creative confines. Previously algorithmic composition (or generative music) has been the preserve of the more experimental musicians due to the limited technology and the complexity of applying it to composition. Now modern technology is allowing the possibilities of algorithmic composition to be available to musicians through Virtual Studio technology and Digital Audio Workstations. Though many modern tools do not fully recognise the importance of merging the logical world they are created from and the creative minds that will be using them. The wealth of algorithms used in modern music shows the potential for creativity, though mapping rules to music must also produce a listenable result. Computers are now the driving force behind the creation of most music in modern times. Whether this is in the form of recording or composing the argument that computers influence modern music techniques is strong. Using computers to compose music can and has changed how musicians compose their music, with the modern composer sitting down with their computers alongside the audio software to write music [7]. The computer itself is becoming an instrument and therefore the programs used will dictate how music can be created [8]. Though this has been considered a problem in modern music composition many musicians see the use of technology to aid their composition process as a chance to overcome the traditional patterns of music conception [9].
General Terms Algorithms
Keywords Composition, Algorithms, Markov, Cellular Automata.
1. I NTRODUCTI ON One of the most valuable tools composers have is their own creativity. There are times when the lack of creativity, or what is UHIHUUHGWRDVZULWHU¶VEORFNFDQFDXVHSUREOHPVIRUDFRPSRVHU Although writers block has many causes it can come down to simple things like the time of day chosen to begin writing, or simply being tired [1]. A key issue is the inspiration, and the creation of new ideas. The use of an algorithmic compositional tool during these tough creative moments can potentially help unlock the mind to overcome this problem. Creating music can be Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. AM '12, September 26 - 28 2012, Corfu, Greece Copyright 2012 ACM 978-1-4503-1569-2/12/09 $15.00.
An important area to understand in music composition is the structure and the interactions within a musical framework. Understanding this can inform the development of a compositional aid. The structure of music considers many things
76
creating melodies, for example the Otomata project [17]. The overall appeal of Cellular Automata with composers is due to its relatively simple rules which can produce multiple and complex interactions. As the automata can generate repeating patterns this makes it appealing and versatile in its uses and allows the composer to explore these new emerging patterns within the musical format [18]. The iterative process within music is generally enjoyed by the human ear [7] therefore the repeating patterns can create enjoyable musical variations. These repeating patterns can also work well with rhythms, due to the cyclic nature of the patterns.
such as tempo, pitch, melody, rhythm, harmony and other musical factors. From the algorithmic point of view researchers have likened the structure of a song to small subunits coming together to help the user build a more complex overall idea [7]. Analysis of music using Hidden Markov Model algorithms has further supported the idea that music can be broken into sub structures [10]. Music can also be understood as a sequence of sonic events, which move from one time to another. In order to understand PXVLFZHWDNHFXHVRUµHDUPDUNV¶IRUIXUWKHUVWUXFWXUDOJXLGDQFH beyond the repetition of the structure itself [11]. Humans listen for repetition in order to understand the music, this providing the listener with a reference point so the music is coherent [7]. Composers form patterns of rhythm, harmony, melody and LQWHUYDO8VLQJDSDWWHUQIURPRQHFRPSRVHU¶VVRQJDQGXVHLWLQ DQRWKHU¶VZRXOGUHVXOWLQDQREYLRXVLQFRQVLVWHQF\[11]. Another key element in musical patterns is entropy. Entropy in musical melodies should be sufficiently low in order to establish a pattern, but also sufficiently high to incorporate enough complexity in order to make the music sophisticated enough for the listener to perceive it as interesting [12]. The ability of algorithms in music to create patterns is well known, though enabling them to incorporate things like entropy or recognise the nuances within these patterns and match them off to another pattern is not achieved so easily. The complexity of the algorithm must therefore be considered from the viewpoint of whether it can deal with these factors.
Transferring the numerical data within an algorithm to a musical function is known as mapping. This is done to create the link between the algorithm and the musical functions like pitch, timbre, tempo or many other possibilities [19]. A single rule could therefore be attached to these musical parameters [20]. More complex rules can be attached to many variable or parameters within the music. This in itself gives the algorithm a lot of power to help create music if properly mapped. The types of algorithms used within the project will be applied through mapping these to specific musical functions based on effectiveness and ability to create novel ideas. Ultimately the mapping is a choice of the creator of the algorithm. Any musical function can potentially be mapped to, therefore the important question is whether the mapped data helps control the sound in a way that it creates a listenable, musical experience which composers can then use. For example looking at the potential mapping of a cellular automata algorithm to something like the timbre of an instrument may not be very effective in helping a composer create new music based around their own ideas. Keeping the mapping within the musical form like melody, rhythm, can help achieve a more structured listenable result.
Melody is central to composition in many musical genres [11]. Once a melody is created, the rest of the composition follows. Melodic continuity is change within a continuous process, and listeners expect changes in melodic continuity. [13]. In a similar vein to pattern matching, building this capability into an algorithm can be achieved through the ability of the algorithm to understand the data fed into it. Algorithms designed to control these functions must consider carefully the level of randomness to be allowed from the output if the device is to be useful and produce coherent and interesting music. Musical structure and form can be manipulated by an algorithm and creating relationships within melodies and rhythms is key to the music to sounding natural to the listener. Making sure an algorithmic tool has the ability to recognise the structural hierarchy of the melody and the song in question is therefore very important to the ultimate usefulness of the technology [14].
The composer seeking inspiration requires a tool that helps them produce coherent and creatively inspiring music, whilst remaining in control of the creative process. This must be implemented in a way that allows the composer to gain creative input from technology, helps them augment their creative efforts without compromising their style, and support the creation of musical structure and new melodies. A key point is that the system must help composers augment their existing musical efforts, the algorithm working together with the composer.
Selecting algorithms to achieve a useful compositional aid depends to a large degree on how natural or interesting the output music sounds. Algorithms based on grammars are known to produce music akin to Jazz [15], while genetic algorithms have WKH SRVVLELOLW\ WR VRXQG µH[SHULPHQWDO¶ [16]. Algorithms that output a more natural and coherent sound could be seen with Markov Models - conditional probability systems with future events depending on past events [7]. Markov chains can be applied to melody creation and involves using past inputs (note in the melody) to dictate subsequent notes. Using higher order models creates more complex output and melodies sound more natural [11]. Markov Models are limited in that they are fixed order, (the rules are set before the model is generated) and can only act on what has been input to the system [15]. However they have the ability to quickly create a new melody around a melody that is already present within a composition. Cellular Automata are based on evolutionary rules and behaviours to help create complex results from minimal input. Applying Cellular Automata to music can yield very interesting results, particularly when
2. M ETHODS 2.1 I nterviews Semi structured interviews were conducted with ten electronic dance music composers in order to inform the design of an algorithmic composition aids. These artists were contacted through the live electronic night called Tronic hosted in Glasgow. Of the artists selected seven of them were currently signed to a record label while the remaining three participants had a minimum of 5 years electronic composing experience. The aim was to gather information on tools they use to aid composition; problems they encounter when composing (i.e. creativity, inspiration); their attitudes to algorithms in music and balancing creativity with technology; the types of tools they would like to help them compose. Synthesisers were identified as the main composition tool, but that some means of manipulating digital audio clips would be equally desirable. The majority of the participants did not use algorithmic compositional aids within their composing processes. Though all
77
artists used differing software plug ins and instruments to create their sound the specific use of algorithms to create an intended outcome was rarely considered or ever put into practice. One area picked up on by an interviewee was that many artists prefer to rely on their own creativity ³ZK\ZRXOG\RXXVHLWLQWKHILUVWSODFHLI\RXZDQWHGLWWRFUHate LWDOOIRU\RX´ Many of the musicians interviewed spoke of specific problems they encountered when composing, which they would like help to overcome. This ranged from the creation of melodies, rhythms and compositional structures. One interviewee spoke of what they would like to help them in composition ³7U\LQJWRJHQHUDWHDQHZPHORGLFOLQHRUIRUFLQJDOLQH\RXKDYH LQWRDVSHFLILFNH\´ Other comments regarding the preferred functions of the proposed tool were noted ³,W FRXOG EH VRPHWKLQJ WKDW LV QRW massively fancy or overly complex, its just does it work? Something that can produce new elements to your track without a lot of effort, make life easier in UHJDUGVWRFUHDWLQJ\RXUWUDFN´
Figure 1. M idi Note Routing and pairing prior to analysis
³, ZRXOG VHH LW DV D JUHDW WRRO HYHQ MXVW IRU VXJJHVWLRQV IRUWKDt VSDUNWRKHOSWKLQJVDORQJ´
The analysis section then receives the notes and merges them into pairs in to feed the 2nd order analysis. A key section of the system is the pitch generate patch (Figure 2). This employs the 2nd order Markov Chain to generate new notes based on the captured notes, and these are then routed to the replacenotes.api object. The midi data is taken in through the coll object, and from here new notes can be created and output. The random object in the patch selects the new notes randomly from the possibilities created by the algorithm.
Key issues identified in the interviews included: The desire to have the ability to manipulate digital audio clips to simplify creation of complex rhythm parts; Generation of new melodic lines, or manipulating those already written; 6HOHFWLQJ WKH µEHVW ELWV¶SURGXFHGE\DQDOJRULWKPDQGXVLQJWKHVHDVDEDVLVIRUWKHP to work on; Something that sparks the imagination; A software tool that can be integrated into their existing compositional software; Simplicity, and the ability to easily produce musical output that can be used in a song .
2.2 Tool design Max/MSP was utilised for development of two algorithmic composition aids. This allows seamless integration with a popular Digital Audio Workstation platform (Ableton Live) via the Max for Live environment. Thus allowing the composer to use the devices alongside their usual composition tools in Ableton.
2.2.1 Melody manipulation: Mozart v Markov This first device aims to support composing and melody creation in MIDI using synthesisers and software synthesis plugins. The Mozart v Markov device incorporates a 2nd order Markov algorithm [21] and was designed to work with melodies within an Ableton MIDI clip. A 2nd order Markov Chain was used which operates on the preceding 3 notes in an existing melody line input by the composer, and outputs the next note in the melody. Thus the patch builds on the composer's original idea, and keeps the melody within the correct key to maintain structure. The algorithm data flow comprises: capture notes, analyse notes, generate new notes, then replace notes within the Ableton MIDI clip with these new notes. The first section of the patch captures the selected notes from the Ableton piano roll. Data is routed through the jitter object and then unpacked for analysis (See Figure 1). Figure 2. Pitch Generate patch utilising the 2nd order M arkov Chain
78
states. If any cell encounters a wall, it triggers a pitched sound whose frequency is determined by the x/y position of collision, and the cell reverses its direction. If a cell encounters another cell on its way, it turns itself clockwise.
Figure 3. shows the final result in a screenshot of the Ableton piano roll window. The notes selected up to bar 113 have been processed and replaced with new notes. The composition has now been augmented by the plugin in this section, although all the timing, velocity and note positions have been kept to maintain the integrity of the overall composition.
As this algorithm was developed for melody manipulation the rules were mapped to the live.grid object to manipulate cell movement only. This in turn affects the audio passing through the live.grid, reshuffling the audio segments. The cellular automata rules are applied to the grid cells and used to create repeating patterns from segments of the audio clip being processed. The composer varies this by removing or adding new cell coordinates. The patch takes the automata data relating to cell movements and converts this to data usable by live.grid (Figure 5).
Figure 3. Note replacement by the M ax v M ozart plugin Using the device within the Ableton environment allows the composer to select problem areas of their melodies and quickly create new ideas and variations. The Markov algorithm also allows the composer to retain melodic continuity and leave reference points intact, which can be important in order to create coherent music [7].
Figure 5. Cell M ovement data converted to live.grid data The chucker~ object takes a specified amount of audio data and stores this in an internal buffer. The audio is divided into equal sections allowing the re-ordering of the audio. Optional arguments for the amount of time it can hold in milliseconds can be changed. The patch also has the potential to be used on sampled audio chord progressions, which can be automatically rearranged to new and unique sequences. Rhythm and drum loops also work well. The restructuring of the sequence of audio clips automatically creates new patterns and rhythms, from which the composer can select the ones they like best and utilise in their compositions.
2.2.2 Audio/structure manipulation: Audio Automata The audio automata tool manipulates digital audio in the Ableton environment, in contrast to the MIDI manipulation preformed by Mozart v Markov. This is achieved by using the Max for Live live.grid (Figure 4) and chucker~ objects in Max. The grid affects the audio passing through it, and manipulation of the cells has the effect of reshuffling the time ordering of audio segments.
3. RESULTS In order to evaluate the prototypes, data was gathered from 11 electronic composers who had access to Max for Live within Ableton. This took the form of a questionnaire, completed by participants who used the devices within their normal composing environment. The questionnaire posed several questions around the usefulness of the developed plugins, requiring responses on Likert scales rating the usefulness of the software (1 being least helpful and 5 being most helpful). The questions aimed to assess the usefulness of the software as regards creativity, inspiration, helping you think outside the box, writing melodies, and workflow (i.e. was it intuitive to use?). See Figures 6 and 7.
Figure 4. Live.grid object The audio automata patch was developed using a patch originally developed within the Max/MSP community and based on Batuhan %R]NXUW¶VUXOHV[17]. Each alive cell has 4 states: Up, right, down, and left. At each cycle, the cells move themselves in the direction of their internal
79
³$IWHU XVLQJ LW D IHZ WLPHV , HYHQWXDOO\ JRW XVHG WR KRZ WKH device worked and from there I could create some interesting PXVLF´
4. DI SCUSSI ON Although limited to a small number of participants, the test results show the two differing algorithmic compositional tools were well received by the electronic composers who used them. The Mozart v Markov device demonstrates that an algorithmic tool can help with creativity and inspiration and that it can be used within composition to spark these new ideas. Applying Markov Chains to melody creation is an effective use of this technology. Comments from the participants suggest that using these tools can IDFLOLWDWH µPXVLFDO GLVFRYHU\¶ >@. The results clearly show the midi Mozart v Markov tool was more useful in practice and could be used quite effectively when needed. With over 80% of participants saying it had helped them with their creativity. Some negatives the Mozart v Markov device were picked up on from the survey. Most notably from a user commenting on how they prefer to make their own music and they found it hard to get it the way they wanted. This chimes with some previous research suggesting composers are loath to surrender any control over the creative process. This could be also be explained by any bias the user had towards the use of algorithmic device [5]. In this respect the device could possibly be improved to allow more control on behalf of the composer in terms of controlling the output of the V\VWHP$XGLR$XWRPDWD¶VUHVXOWVDUHQRWDVFRQVLVWHQWO\ positive as Mozart v Markov, though overall the comments and ratings ZHUHJHQHUDOO\SRVLWLYH$XGLR$XWRPDWD¶VSURSHQVLW\WRRXWSXWD less coherent more random sound was cast in a negative light. This could in future be remedied by utilising different mapping techniques to more strictly constrain the control of musical features by the algorithm..
Figure 6. Responses to usefulness questions ± M ozart v M arkov. The number of responses is indicated on the Y axis.
5. CONCLUSI ON This project aimed to build an algorithmic compositional aid, which can help composers augment their musical output while effectively aiding them with creativity and writers block. The research suggests use of algorithms to aid composers can be effective as long as certain criteria are met, namely the incorporation of musical form, structure, and an ability to output natural-sounding, coherent music. Once these factors are met algorithmic aids can add to the ever-increasing compositional tools available to the modern composer. In the study of algorithmic music composition the use of more complex and nondeterministic algorithms would be the next logical step. Algorithms that use artificial intelligence learning techniques, or genetic algorithms have the ability to create more complex data, which in turn can be used to create a richer musical output. Another area along these lines would be to look into creating a compositional aid, which can control multiple elements within musical form and structure allowing the composer more control over their output.
Figure 7. Responses to usefulness questions ± Audio Automata. The number of responses is indicated on the Y axis. Qualitative data was gathered also from all the artists. Selected comments illustrate how the Markov v Mozart device helped aid the composers in creativity and idea generation: ³,WZDVLQWHUHVWLQJWRVHHKRZLWFRXOGDOWHUDPHORG\DQGDLGWKH composition process by throwing out new ideas to play around ZLWKIXUWKHULQWKHFRPSRVLWLRQ´ ³JDYHPHMXPSLQJRIISRLQWVWRZRUNLQQHZLGHDV´ The Audio Automata device had positive benefits for composers, though some users found the random nature of the device a drawback: ³WKHZD\WKHSOXJLQZRUNVLWRIIHUVWKHXVHUYDULDELOLW\ZKLFKFDQ open up different routes in the composition rather than it being a WHGLRXVRQHURXWHFRPSRVLWLRQ´
REFERENCES [1] Flaherty, A.W. 2005. The Midnight Disease. The Drive To Write, Writer's Block, and The Creative Brain, Houghton Mifflin Harcourt.
³,W FDQ FUHDWH VRPH LQWHUHVWLQJ HIIHFWV EXW LW LV DOVR YHU\ UDQGRP sounding.´
80
[13] Meyer, L. 1956, Emotion and Meaning in Music, The University of Chicago Press.
[2] Roads, C. 1989. The Music Machine: Selected Readings from Computer Music Journal, Cambridge, MA, USA: The MIT Press.
[14] Cope, D. 1991, Computers and Musical Style, A-R Editions, Inc, Madison, Wisconsin.
[3] Bilton, C. 2007. Management and creativity: From creative industries to creative management. Wiley-Blackwell.
[15] Nierhaus, G. 2009, Algorithmic Composition:. Paradigms of automated music generation. Springer, New York.
[4] Muscutt, K. & Cope, D. 2007, Composing with Algorithms: An Interview with David Cope, Computer Music Journal, vol. 31, no. 3, 10-22.
[16] Audio Damage, Axon .Retrieved October 13th 2011 from Audio Damage: http://www.audiodamage.com/instruments/product.php?pid= AD026
[5] Tingen, P. Autechre. Retrieved December 10th, 2011 from Sound on Sound: http://www.soundonsound.com/sos/apr04/articles/autechre.ht m
[17] Bozkurt, B. Otomata. Retrieved December 28th, 2011 from Earslap:: http://www.earslap.com/projectslab/otomata
[6] Roads, C. 1985, Composers and the Computer, William Kaufmann, Inc, USA.
[18] Serquera, J. & Eduardo, R.M. 2010. Cellular Automata Sound Synthesis with an Extended Version of the Multitype Voter Model. In Proceedings of the 128th Audio Engineering Society Convention 2010, 19-24.
[7] Miranda, E.R. 2001, Composing Music With Computers, Focal Press, Oxford. [8] Zaft, G. 1996. Computer as tool and technique. Retrieved May 11th 2012 from University of Arizona: http://www.zaft.org/gordon/mse596s/music.html
[19] Diaz-Jerez, G. 2000, Using Mathematical Models in Music Composition. Doctoral Thesis, Manhattan School of Music. [20] Anders, T. 2003, Composing Music by Composing Rules: Computer Aided Composition employing Constraint Logic Programming. Doctoral Thesis, Queens University, Belfast.
[9] Becker, B. and Eckel, G. The Use of Technology in Contemporary Music. Retrieved May 11th 2012 from IRCAM: http://iem.at/~eckel/publications/becker94e.html
[21] Algorithmic Composer. Retrieved 29th September, 2011 from Algorithmic Composer: http://www.algorithmiccomposer.com
[10] Aucoutrier, J-J. and Sandler, M. 2001. Segmentation of Musical Signals Using Hidden Markov Models. In Proceedings of the Audio Engineering Society 110th Convention, May 2001.
[22] Lamb, R. 2011. Is Electronic Music 'Real Music? Retrieved 5th March 2012 from Discovery News: http://news.discovery.com/tech/is-electronic-music-realmusic-110725.html
[11] Cope, D. 2000, The Algorithmic Composer, A-R Editions, Madison, Wisconsin. [12] Olson, H. & Belar, H. 1961, Aid To Music Composition Employing a Random Probability System. Journal of the Acoustical Society of America. Volume 33, Issue 9, 11631170.
81
