Emotion-driven Automatic Music Arrangement Jeffrey MAK Ka-Hing1, Clifford CHOY Sze-Tsan2 *, Clifford SO Kwok-Fung3, Henry MA Chi-Fai4 Multimedia Innovation Centre, School of Design, The Hong Kong Polytechnic University {trjeffrey1, mccliff2, mchenry4}@polyu.edu.hk,
[email protected]
1.
Introduction
Music has a powerful role in invoking emotions, and high quality orchestral music is increasingly being used in video games and interactive applications. This project aims at developing a music arrangement engine for games and interactive applications, in which orchestral music can be automatically arranged from musical materials, rules and parameters provided by composers, subject to the emotional requirements in a game or an interactive application. Through one single graphical user interface, composer can input musical materials while game designer can specify game states, their transitions, and their associated emotions. Based on this information, the music arrangement engine will automatically generate appropriate music with desired emotional qualities. Section 2 describes the system architecture of our engine and explains how this architecture allows adaptation to different musical styles and composition preferences of different composers. Section 3 describes how game designers and composers define their musical requirements on a single platform and our engine does the rest. Section 4 reports the current status.
2.
System Architecture and The Impact
One of the challenges in developing this system is how it can adapt various musical styles as well as new composers’ personal composition preferences. To achieve this, we propose a four-layered architecture for our engine as illustrated in Fig 1.
Process layer states how a given musical tune is arranged to give a complete piece of music. A set of operators are defined in this layer. Each one makes a particular arrangement decision. For example, an operator called Tonality decides the mode and the key of the music. An operator called Harmonization decides the harmonic progression of the accompaniment. An operator called Rhythm Generator decides the rhythmic pattern of the accompaniment. They connect to each other to form an arrangement process. This architecture separates musical data, arrangement rules and arrangement process into four different layers. As a result, new emotions can be supported by inserting new arrangement rules and music materials to the Arrangement Rules layer. These rules define what musical qualities like rhythmic patterns and tempo should be used in the music to express the new emotions. On the other hand, different composers may have different ways to convey a certain emotion. Their ways are defined as different settings of the operators at the Arrangement Process level. To support new kind of musical styles such as Jazz music, new operators can be developed to the Arrangement Process layer to make Jazz arrangement. However, new arrangement rules in this case may need to be added to support the new way of chord usage.
3.
The system introduces two novel diagrams named, auditory scene diagram and fragment diagram. In brief, auditory scene diagram is a non-linear cue sheet. Game and interactive application designer defines the emotion and the expected duration of the music cue at a certain event. Fragment diagram lays out the blue-print of a music cue. Composer defines the static structure of music cues with it. Our engine arranges musical themes (given by the composer) for the events based on these diagrams.
4.
Fig 1. A 4-layer architecture of the arrangement engine The lowest layer defines the most fundamental musical concepts such as note, dynamics, chord, phrase, meter and instrument. This set of concepts is sufficient to represent a music score. Above this is the Arrangement Concept layer, which defines intuitive concepts music composers used when they write or analyze music. These are high level qualities that music composers interested in, including music motif, melodic range and pitch contour, to name a few. In the third layer, arrangement rules are defined to specify how we obtain different emotions of music by varying these high level qualities. Some arrangement rules are, however, not emotion specific. They are applied to ensure the craftsmanship of the generated music. This layer also contains a repository of musical materials including rhythmic patterns and pitch contours. They are used to construct the accompaniments based on different emotions. Arrangement
Production Pipeline
Current Status
Arrangement rules used in the current version are extracted from previous music literatures [Juslin and Sloboda 2000] on music and emotions. According to their findings, there are 8 basic classes of emotions [Hevner 1936] that can be aroused through music by manipulating music qualities like rhythm and tempo. The current version of our engine gives satisfactory results in four of them. These emotions include Dignified, Tragic, Serene and Joy. At the Arrangement Process layer, it has a default process which arranges a 2 phrase monophonic musical themes into a 4 part pieces. Musical themes are imported to our engine in MusicXML format.
5.
References
JUSLIN, P. N., and SLOBODA, J. A. 2000. Music and Emotion – Theory and Research. Oxford. HEVNER, K. 1936. Experimental studies of the elements of expression in music. American Journal of Psychology, 48, 621-30. * Correspondence to: Clifford CHOY Sze-Tsan. This work is supported by grant ITS/131/03 from the Innovation and Technology Commission, The Hong Kong SAR Government.
