Interactive Advertising Jingles: Using Music Generation for Sound Branding*1 Gilbert Beyer, Max Meier LMU Munich
Programming and Software Engineering Oettingenstr. 67, 80538 Munich, Germany
[email protected] [email protected]
Abstract. Music in advertising is an intensely used social engineering technique and a commonplace to today’s customers. From an advertising perspective, the benefit of music is that it is a strong vehicle to convey a memorable message to the target group. Mainly used as jingle in radio and television commercials or as background component in shopping environments, sounds were put in the context of interaction in the internet and digital outdoor media. Yet, there have been only very limited attempts to control music within the interactive experience. This paper describes an approach to make advertising jingles and sounds be controlled by the interaction, utilizing a novel technique of music generation based on soft constraints. The paper starts with a vision on the future development of sound branding. That followed, we present our concept of interactive advertising jingles and describe the design and techniques of our prototype system for music generation. We close with implications of our present results on further work. Keywords: music generation, sound branding, sonic mnemonics, city soundscapes, advertising displays, algorithmic composition, soft constraints.
1
A Vision of Future Sound Branding
The use of music within advertising media began in the 20th century with jingles in radio and television commercials as well as music playing statically in the background at the point-of-sale or sales events. With the emergence of the internet, sounds used in the branding context became attached to interactive elements. Sound in such dynamic and interactive media either appear as sound-logo (a short auditory message which is the acoustic pendant to a visual logo, and thus is often presented at the beginning or the end of a commercial), as advertising jingle (tunes that are often played along with lyrics, to convey an advertising slogan) or as background music [15]. In the case of interactive media and product design, acoustic signals or so-called *
This work has been partially sponsored by the EC project REFLECT, IST-2007-215893.
sound objects are connected to certain events like a mouse click on a graphical element or to activities like the closing of a car door [3]. In the last years, a trend to interactive advertising media can be identified. Casual games in the web constitute an example, while in the out-of-home domain most often interactive plays, i.e. invitations to less structured activities, also imply creative or participatory elements [1]. Such media often focus on manipulating visual objects that are usually constituent parts of the brand identity. Acoustic events do not appear at all or play only a secondary role by supplementing the visual interaction or statically playing in the background. Nevertheless, the identity of many brands is defined by both a visual and acoustic appearance. On the other hand, beyond the advertising context interactive music systems have become increasingly popular: with social music games like Guitar Hero popular songs can be re-played together, and easy-to-use musical interfaces like Yamaha’s Tenori-On give everyone the possibility for musical expression, even without having any musical knowledge. We estimate that these trends will combine in the future, producing new advertising media that will enable customers not only to play with, but also manipulate and shape brand melodies by means of interactive control mechanisms. Generally this may include as well sound logos, jingles and background songs. The vehicles for such applications can be interactive displays in shopping malls and outdoor environments, the internet, mobile devices or gaming consoles. In this work, we present an approach which brings together interactive advertising and interactive music systems with a high focus on integrating a given acoustic brand identity in a suitable way.
2
Related Work
In recent years, many concepts and products making use of music in interactive advertising systems emerged. In the following we present related work focusing on sound branding in general, the combination of interaction and music composition, technical solutions for music generation and for dealing with dynamically changing preferences. Of special interest to our work are theoretical works on sound branding and interactive sound branding. There exist many articles on specific sound branding issues in classical and digital media, but they do not cover the field of usercontrollable brand music. No work so far focused on how to control the brand music itself within the interactive experience, while the same is often done with visual elements of the brand identity. For a general survey on the topic of sound branding we refer to [3] and [15]. A good overview on algorithmic composition is provided by [4] and [13]. Examples for interactive music composition and generation systems are Electroplankton [8] or Cyber Composer [7]. To our knowledge, there is currently no work describing the combination of music generation and interactive advertisements. For interacting with music, out of a great many of possibilities, the interactive system Light Tracer, that invites users to creative activities in physical space, at the same time transforming these interactions into music, is relevant for the interaction concept of our system [11].
Our approach for generating music is based on a reasoning-technique called soft constraints which allows dealing with soft and concurrent problems in an easy way. Bistarelli et. Al. [2] introduced a very general and abstract theory of soft constraints based on semirings. Building on this work, in [6] monoidal soft constraints were introduced, a soft-constraint formalism particularly well-suited to multi-criteria optimization problems with dynamically changing user preferences. Soft constraints have successfully been applied to problems such as optimizing software-defined radios [18] or orchestrating services [17]. We introduced a soft-constraint based system for music therapy in [5], giving us basic proof of concept in composing music with this technique.
3
Interactive Advertising Jingles
The functions of music in advertising are manifold. Sounds are used to gain or hold the attention of the listener [9,10], to influence the mood of consumers, to structure the time of an ad or to persuade consumers by using rhetorical elements like rhythm, repetition, narrative, identification or location [14]. The benefits are a more effective information reception, memorization and an enhanced user experience by the use of multisensoric branding [15], as well as the fact that the acoustic sensory channel is harder to ignore by the audience. But brand melodies are also subject to specific requirements. For example, the characteristics of an effective sound logo can be listed as distinctiveness, memorability, flexibility, conciseness and brand fit [15]. These are strong constraints an interactive advertisement has to conform to. To achieve this, we make use of a novel technique that allows generating music in real-time with respect to certain preferences that express ‘how the music should sound’. Several preferences are derived from user interaction, for example ‘high and fast notes’. These are combined with additional preferences expressing that the resulting melodies should comply with a brand’s distinct acoustic identity. Since a certain amount of control over the music is assigned to the user, it is inherently not possible to exactly play a given melody note by note. Nevertheless, it is possible to generate melodies which are similar to it by using note pitches as well as tonal and rhythmic patterns appearing in the brand’s distinct melody. This way, melodies can be generated considering both interactivity and brand recognition.
4
Prototype for Interactive Advertising Jingles
In order to test our approach, we designed a sample scenario in the out-of-home domain and set up a prototype consisting of a sensor framework to collect information about the user, a wall of luminous plasma displays showing a graphical application adapting to the users’ movements and the resulting music, and a software framework that realizes the music generation. The scenario and components of our prototype are described in the following:
4.1
Sample Scenario
As an example scenario, we developed an interactive advertising installation for an imaginary soft drink using an ‘underwater-theme’ (see Figure 1). Our application consists of a display and an audio system installed at a preferably quiet public place. When no one is standing in front of the display, only a simple background (a seafloor) can be seen and no music can be heard. As soon as someone enters the interaction zone, his silhouette appears on the screen like in an abstract mirror image, visualized by small water bubbles ascending from the person’s shape.
Fig. 1. Prototype for a Music Generating Advertisement
Depending on the passer-by’s movements, notes are played and visualized with bigger colored bubbles such that each note corresponds to one bubble. This way, an initial implicit interaction is performed with visual and acoustic feedback to the user’s movements. When someone gets attracted and starts to explicitly interact with the system, he can realize the connection between his movements and the notes he hears: When the movements become faster, the notes will also play faster – not moving at all leads to silence. Moving the upper parts of the body (e.g. arms) plays higher notes and, vice-versa, moving lower parts (e.g. legs) leads to lower notes. This is also being reflected in the visualization on the display: the note bubbles ascend from a position corresponding to their note pitch. The resulting melodies do not only fit to the person’s movements; they are furthermore being generated in a way that they comply with the company’s brand melody. After a certain time, additional background music is played and accompanies the user’s melodies. The background music’s notes are also visualized by colored bubbles ascending from a bottle sticking in the sand. This way, the product becomes involved in an unobtrusive, yet very relevant way. 4.2
Interaction and Visualization
We see a vision-based sensing framework to be the most convenient technology in our advertising scenario, as it allows collecting information about the passerby’s movements and gestures, and not least supports implicit interaction. For testing reasons, the sensing is currently realized using marker-based techniques in our first
experiments (Touchless SDK [16]). We attach two colored markers to a user’s hands for analyzing his movements and derive two parameters: the total amount of movement (controlling the amount of notes played) and the average position of the markers (controlling pitch). This concept can easily be extended to more markers. Our long-term goal is to use markerless body tracking involving the detection of individual body parts. The visualization of passers-by and the generated music is realized using a so-called particle system. Being widely used in computer graphics, particle systems are capable of simulating ‘fuzzy’ objects with a large number of individual particles (e.g. fire, smoke or water). In our application, we make use of the Mercury Particle Engine [12] which is directly based on Microsoft’s computer gaming framework XNA and allows developing applications for e.g. Windows or the Xbox 360 gaming console. We decided to use a particle system for visualizing persons because we wanted to create a very abstract and fuzzy representation. The user should feel like controlling his portrait rather than playing another character as in a computer game. In our sample scenario, we use water bubbles for visualizing a person’s shape, but many other alternatives based on particles are imaginable. Not only natural phenomena can be simulated; it is of course also possible to create unrealistic ‘freaked out’ effects.
Fig. 2. Abstract Visualization of a Person and Note Visualization
Every particle emerges from a so-called emitter which can be a single point as well as a complex geometric object. In our application, we use a dynamically changing polygon based on a passer-by’s contour as an emitter for the small water bubbles. The bigger colored water bubbles representing playing notes are emitted from a single point which is moved according to the passer-by’s horizontal position and the note’s pitch such that high pitches emerge from higher positions and vice-versa. When a particle has been emitted, several parameters are modified during its lifetime: the water bubbles become bigger and fade out over time, and they are furthermore being accelerated upwards in order to let them ascend to the water surface.
4.3
Music Generation
For generating music in our system, we use a novel approach for algorithmic composition based on soft constraints [5]. With this technique, music can be interactively generated in real-time by defining certain ‘preferences’ which express ‘how the music should sound’. As an example, a typical preference for a single instrument is ‘fast notes with a high pitch’. Besides preferences for single instruments, it is also possible to coordinate multiple instruments with additional preferences. These global preferences typically involve harmonic or rhythmic relations between several instruments, e.g. ‘play together in harmonic intervals and in a similar rhythm’. All preferences can also be generated dynamically while playing, for example based on user interaction: this way, music can be composed interactively in real-time by continually defining preferences which reflect ‘how well the music matches the user interaction’. In our application, we derive two parameters from a user’s movements: the total amount of movement (corresponding to the rate of played notes) and the average vertical position of all movements (corresponding to pitch). Based on these two parameters, preferences are generated reflecting the desired speed and pitch. For example, when the user is moving fast in the upper areas of his body (e.g. mostly with his arms), the music should also be fast and have a rather high pitch. Vice-versa, when the user is moving slowly and rather in the lower areas (e.g. with his legs), the music should be slow with a low pitch. In our application, the music should fit the user interaction on the one hand, but we also want it to fit to a given sound brand on the other hand. This is realized with an additional preference reflecting ‘how well the music matches a jingle’s distinctive melody’. This preference is generated based on a timed transition model representing the jingle’s note pitches and rhythmic patterns as well as transitions between notes (e.g. ‘C is often followed by E or another C’). To sum it up, we have preferences based on user interaction as well as preferences reflecting the similarity to a jingle and - in most cases - these preferences will be concurrent among each other. Soft constraints are very appropriate for dealing with such problems and allow accommodating several concurrent preferences in an easy yet expressive way. When the preferences have been stated, a soft constraint solver can be employed for computing the best possible notes with respect to all preferences: the notes should fit the user interaction and a given distinct brand melody. Furthermore, it is also possible to coordinate several instruments among each other with additional global preferences (see Figure 3). In our case, we define a global constraint which maximizes the amount of musical harmony between the interactive instrument and the background music. We use a soft constraint solver which was originally prototyped in Maude [6] and that we later implemented in an improved version in C#, making it possible to use it in a soft real-time environment.
Fig. 3. Concurrent Preferences
With our approach, it is possible to generate music which strongly resembles a given melody but also fits to body movements as described above. For more detailed information about composing music with soft constraints, we refer to [5]; we are also about to publish a general approach for interactively composing music similar to given melodies.
5
Prospects and Future Work
In this paper we have shown an approach of future sound branding applications, where advertising jingles and sounds are controlled by the interaction. To make sure that the resulting music complies with both the requirements of interactivity and brand recognition, we made use of a novel technique of music generation based on soft constraints. First tests with our prototype with large advertising displays showed that composing music with soft constraint solving works quite well, giving the user the impression of control over the music while producing quite recognizable sound. The next step is to enhance the sensor framework of the prototype with markerless body tracking and, that followed, to conduct user studies on different types of interactive advertising music (sound logos, jingles, background songs) on their recognition value. A further goal is to investigate if functions of music, as described for classical forms of music in advertising, also exist in interactive advertisements.
References 1. 2. 3.
Adamowsky, N.: Homo Ludens – Wahle enterprise: Zur Verbindung von Spiel, Technik und den Künsten. In: Poser, S., Zachmann, K. (eds.), Peter Lang Verlag, Frankfurt am Main (2003) Bistarelli, S., Montanari, U., Rossi, F.: Semiring-based constraint satisfaction and optimization. Journal of the ACM, vol. 44(2), pp. 201–236 (1997) Bronner, K., Hirt, R.: Audio Branding – Entwicklung, Anwendung, Wirkung akustischer Identitäten in Werbung, Medien und Gesellschaft. Nomos Verlagsg., Baden-Baden (2009)
4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.
18.
Essl, K.: Algorithmic composition. In: Collins, N., d’Escrivan, J. (eds.) Cambridge Companion to Electronic Music. Cambridge University Press, Cambridge (2007) Hölzl, M., Denker, G., Meier, M., Wirsing, M.: Constraint-Muse: A Soft-Constraint Based System for Music Therapy. In: Proc. of Third International Conference on Algebra and Coalgebra in Computer Science (CALCO’09), pp. 423–432. Springer, Udine (2009) Hölzl, M., Meier, M., Wirsing, M.: Which soft constraints do you prefer? In: Proc. of Workshop on Rewriting Logic and its Applications (WRLA 2008). Budapest (2008) Ip, H., Law, K. Kwong, B.: Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation. In: Proc. of 11th International Multimedia Modelling Conference (MMM'05), pp.46–52, Melbourne (2005) Iwai, T., Indies Zero and Nintendo: Electroplankton. Music Game for Nintendo DS. Released in (2005) Kellaris, J., Cox, A., Cox, D.: The Effect of Background Music on Ad Processing: A Contingency Explanation. Journal of Marketing, vol. 57 (4), pages 114–125 (1993) Kroeber-Riel, W., Esch, F.-R.: Strategie und Technik der Werbung. Verhaltenswissensch. Ansätze. Diller, H., Köhler, R. (eds.). Vol. 6, Kohlhammer, (2004) Light Tracer project website (2010), http://lighttracer.darcy.co.nz/ Mercury Particle Engine website (2010), http://mpe.codeplex.com/ Nierhaus, G.: Algorithmic Composition - Paradigms of Automated Music Generation. Springer, Heidelberg (2008) Scott, L.: Understanding Jingles and Needledrop: A Rhetorical Approach to Music in Advertising. Journal of Consumer Research: An Interdisciplinary Quarterly, vol. 17 (2), pages 223–36 (1990) Steiner, P.: Sound Branding – Grundlagen der akustischen Markenführung. Gabler, Wiesbaden (2009) Touchless SDK website (2010), http://www.codeplex.com/touchless Wirsing, M., Clark, A., Gilmore, S., Hölzl, M., Knapp, A., Koch, N., Schroeder, A.: Semantic-Based Development of Service-Oriented Systems. In: Najm, E., Pradat-Peyre, J.-F., Donzeau-Gouge, V.V. (eds.) FORTE 2006. LNCS, vol. 4229, pp. 24–45. Springer, Heidelberg (2006) Wirsing, M., Denker, G., Talcott, C., Poggio, A., Briesemeister, L.: A Rewriting Logic Framework for Soft Constraints. In: Proc. of Workshop on Rewriting Logic and its Applications (WRLA 2006). Vienna (2006)
