A Proposed Framework for Visualising Music Mood ...

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13)

A Proposed Framework for Visualising Music Mood using Texture Image Adzira Husain, Mohd Fairuz Shiratuddin, Kok Wai Wong School of Engineering & Information Technology Murdoch University Perth, Australia {a.husain, f.shiratuddin, k.wong}@murdoch.edu.au Abstract — Fairly recently, extensive use of digital music has led to an increase of songs in online music applications and personal music libraries. In large libraries, songs which are not listened to regularly will go unnoticed. There are many ways of browsing and discovering new music. In the field of Music Information Retrieval (MIR), some type of visual form such as colour, avatar, mood picture and album cover to visualise music, have been introduced. In this paper, we propose a framework for visualising music mood using texture images to assist listeners in discovering new songs and artists that might not have been discovered otherwise using conventional means. Keywordsvisualisation

music;

I.

mood;

texture;

visual

elements;

INTRODUCTION

Today, through online music stores and services such as iTunes [1], Grooveshark [2], Spotify [3], and Pandora [4], music consumers can access numerous digital music collections that may offer millions of songs. Various methods have been designed by the music service providers to allow users to easily browse, filter, navigate, and search their online music collections according to selected musical metadata attributes such as artist, musical genre, mood, release year, song name, and tempo [5]. Beside selecting from a list of artist or musical genre, some listeners prefer to search using the mood category that suit their mood at a particular moment. It is well known that music can convey emotion and modulate mood, and sometimes listeners select music by retrieving it through the mood categories such as happy, anger, sad, fear etc. [6]. The traditional way of browsing a music collection is by going through a textual list of songs, and the search methods may not be sufficient to maintain an overview of the collection [7]. A promising alternative for representing musical metadata is through information visualisation (Infovis) [5]. Infovis is a multidisciplinary research field that bridges many areas including Human Computer Interaction (HCI), computer graphics, and cognitive psychology. In this field, images and structures of data are generated to support the exploration, analysis and decision making [8]. There are also a number of research on music visualisation using visual variables such as colour, position, size, shape, and value but there is no research focusing explicitly on textures [5]. In this paper we propose a framework for visualising music mood using texture images to assist listeners in

discovering new songs and artists that might have not been discovered otherwise using conventional means. II.

RELATED WORKS

The most common way to visualise songs and albums in commercial music recommendation applications is to use the album covers [9]. To convey the emotional message of the music, the album covers may utilise colours, fonts, and symbols that are typical for the musical genre of the album. Colours are often used to visualise music collection. There are several approaches of visualising artist or track similarities using colour, and one of them is the Music Rainbow [10]. Music Rainbow is a simple interface for identifying artists where different colours are used to encode different types of music (Figure 1). Each of the eight rings of the rainbow colours corresponds to a label. The outermost red ring represents the most frequent music listened to in the collection while the innermost purple ring represents the least frequent.

Figure 1. The Music Rainbow interface [10].

Other approaches to visualise music is by using representative avatars (Figure 2). New music recommendations were generated based on three userselected avatar parts (head, body and background). Each of the avatar parts represents a certain musical genre. The Avatar prototype application was evaluated by 40 participants and considered to be entertaining, easy to use, and innovative [11] . A software prototype developed by [12] lets users interact with a collection of mood pictures to receive new music recommendations from the associated genres. The prototype features seven different built-in mood pictures with the associated music (Figure 3). The pictures were designed to match up with potential daily activities and situations of the target user group.

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13) III.

Figure 2. The Avatar prototype application [11].

Figure 3. A sample of different mood pictures [12].

In visualising music mood, there is one music recommendation player called Smoodi [13]. Smoodi provides an easy method to recommend music and generate play list by touching and dragging. It has three different views: Mood Square, Cover Flow and Mood Cloud (Figure 4). Users can check their musical preferences by mood and generate playlists by touching different cells or selecting a music clip or mood tags.

OVERVIEW OF THE PROBLEM

Fairly recently, extensive use of digital music has led to an increase in the capacity of online music application and personal music libraries. Music listeners are looking for new ways to search songs in their music libraries or online music collections not only by title, artist or other keywords, but also by similarity, genre, and mood [15]. In large libraries, songs which are not listened to regularly will go unnoticed. A study by [16] on 5000 iPod users found that only 23% of songs were played 80% of the time and 64% of songs were never played at all. Sometimes, users become lost and confused as to which song to listen to at a particular moment because there are too many songs in their library, and this can lead to information anxiety. Information anxiety is a level of inaccessibility that results from not knowing exactly where or what to look for from a vast amount of information or data. This particular problem can be addressed by providing different vantage points and organisational methods when looking at the same information [17]. Using the same motivation, this paper aims to provide different style of indexing music in the mood category by using texture image. A “keyword-based” search is a fairly common way to find music by artist or title, with a quite good search performance. However, sometimes users cannot simply remember all of the artists’ names or songs’ titles [13]. Therefore, a mood-based search provides a more natural way of searching. Music mood has been identified as an important category used by listeners in organising and seeking music [15][18]. There are a few approaches of browsing and discovering new music. In the field of Music Information Retrieval (MIR), some type of visual form such as colour, avatar, mood picture and album cover to visualise music, have been introduced. There are also a number of research on music visualisation using visual variables such as colour, position, size, shape, and value, but there is no research focusing explicitly on textures [5]. Research shows that texture is an associative visual variable [19], but there is still no prove that texture can be associated with mood. Besides using other visual forms and variables, we propose the use of texture image to visualise music in the mood category and to prove that texture can be used as visual variables in music visualisation. In this paper, the proposed framework presented also aims at supporting listeners in discovering new songs and artists that they might have not discovered otherwise. IV.

Figure 4. The different mood pictures used in Smoodi [13].

There is also another system of managing music collection by mood called Affecticon [14]. Affecticon groups pieces of music that convey similar emotions and labels each group with a corresponding icon.

VISUAL TEXTURE

A texture is defined as the look and feel of any surface [20]. The appearance of a texture may well relate to the perception of a particular material from which the image is derived from such as grassland, plastic, water or wood [21]. Textures can be divided into two categories, namely tactile and visual textures. Tactile texture refers to the tangible feel of a surface whereas visual texture is the illusion of having a physical texture. Visual texture refers to the visual impression that a texture produces to a human observer, such

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13) as colour, intensity and orientation of an image found in digital images [22]. Research shows that people tend to give a more positive emotional response such as cheerful and elegant to smooth textures [23]. This response led to the idea of smooth texture could be mapped to the “happy” musical genre such as dance and pop, and rough texture to a more “aggressive” genre such as rock [9]. Research in the product design field has also suggested that surfaces of their packaging would enhance consumers’ emotional engagement with their product [24]. The effect of creative texture design of a product can arouse users' perception of its image [25]. Therefore, the abovementioned research suggest that a texture has some kind of emotional connection to human feelings.

V.

THE PROPOSED FRAMEWORK

The proposed framework is based on the visual abstraction phase in a visualisation reference model by [26]. It comprises of four main phases that are needed in mapping the raw data and transforming them into visual form. In this proposed framework there will be two main components of variables that will be used in visualising music mood which include texture characteristics and music mood (Figure 5).

Figure 5. The components of the proposed framework.

A. Texture Characteristic Humans see images as a whole rather than in parts. However, images can be broken down into visual elements that have certain meaning and ability to express emotion such as colour, line, shape, surface and value. Colours can have an impact to human emotions. A number of research have been conducted on the psychology of colours and found that certain colour portray certain mood. For example the colour green evokes positive emotions because it reminds people about nature [27][28]. Line is one of the most basic artistic elements that can be used for different purposes. A line can also express emotion and character based on how heavy or light, scratchy or smooth, angular or curvy the line is [29]. Different types of lines express different emotive qualities. For example, thin lines can be experienced by some as unstable and vulnerable. [29]. It is also fragile and seems easy to break or knock over [30]. Thin lines suggest frailty and convey an elegant quality.

On the other hand, thick lines appear as difficult to break. They suggest strength and give emphasis to nearby elements. Thick lines are bold and able to make a statement [30]. Shapes are another way to emphasise mood [30]. There are two types of shape; organic and geometric. Geometric shapes have angles and straight lines and have an industrialised feel. Organic shapes are curvy, free-form and have a more natural feel to them. Research shows that people prefer circles over angular shapes, as they associate angular lines with anger and agitation. Soft, curvilinear shape may appear warm and welcoming, whereas a sharp, angular shape may appear cold and threatening [20]. In visual texture, the perception of a surface of an object such as flat, hard, smooth, soft or rough can also influence human emotion. People tend to give a more positive emotional response such as cheerful and elegant to smooth textures [22]. Smooth texture is associated with pleasant characters such as polite, continuous, and unproblematic whereas, rough textures is often associated with unpleasant

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13) characters such as dangerous, impolite, interrupted and problematic [31]. Value or brightness refers to the lightness or darkness of an item. It is used to convey emotion or direct the eye to a focal point. Bright colours, for example yellow and blue are associated with positive emotions such as happy and strong. Dark color such as black and grey, are associated with negative emotions such as sad and angry [32] [33]. These elements have been identified to have certain meaning and ability to express emotion. This indicates that texture image is capable as a new method to select songs in the music mood category. B. Mood Model In general, emotional models are divided into two: categorical and dimensional. In the categorical model, emotions are classified into basic elements such as anger, fear, sadness, happiness and disgust. In the dimensional model, emotions are presented by two or three axes. Russell proposes two-dimensional model, and the most widely used dimensional emotional scale is the Russell’s circumplex model [13]. This model suggests that emotions are distributed in a two-dimensional circular space; containing arousal and

valence dimensions. The emotional states can be represented at any level of valence and arousal, or at a neutral level of one or both of these factors. The adaptation of this model is essential for a mood-based music recommendation as it allows for the categorisation of mood and emotion [13]. In this paper, moods from every quarter of Russell’s model which are happy, sad, angry and calm are included in the proposed framework. C. Establishing the Proposed Framework In order to finalise the proposed framework, three stages of research have to be undertaken (Figure 6). In first stage, texture characteristics will be mapped with music mood. Suitable texture images (Figure 7) will then be produced or reproduced in the visualisation stage and used to visualise the music mood. In this research, texture images were obtained from the standard Brodatz texture album which has been widely used in the fields of texture analysis and visual perception [34]. The images will be evaluated by users in the assessment and feedback stage, and then validated and finalised for the final framework.

Stage 1

Stage 2

Stage 3

Variables

Visualisation

(refer to Figure 5)

(refer to Figure 7)

Assessment & Feedback

Texture Characteristic

Image

Participants

Music Mood

Figure 6. Stages to establish the proposed framework.

VI.

CONCLUSION AND FUTURE WORK

We present our proposed framework for visualising music mood using texture images as an alternative method to visualise music mood. Visual elements such as colour, line, shape, surface and value have been identified to have certain meaning and ability to express emotion. This may lead to the fact that texture image is capable as a new method to select song in the music mood category and can be used as visual variables in music visualisation. Once this framework is established, we will continue our work on developing an online music collection sample application. The application will then be tested on participants. They will then answer a set of questionnaires

and provide feedback on how well the images represent certain music moods. With the establishment of this framework, it is expected that it will help the digital music service developer to design a recognisable visualisations using texture image for music collection in music mood category. While to the wider community, this research will provide an alternative method to browse digital music collection and assist listeners in discovering new songs and artists that might not have been discovered otherwise using conventional means.


Figure 7. Samples of texture images from Brodatz album [34].


REFERENCES [1] [2] [3] [4] [5]

[6]

[7] [8] [9] [10]

[11] [12] [13] [14] [15] [16] [17] [18]

(24 July 2013). iTunes. Available: http://www.apple.com/au/itunes/ (24 July). Grooveshark. Available: http://grooveshark.com/ (July 2013). Spotify. Available: http://www.spotify.com (24 July 2013). Pandora. Available: http://www.pandora.com J. Holm, "Visualizing Music Collections Based on Metadata: Concepts, User Studies and Design Implications," Tampereen teknillinen yliopisto. Julkaisu-Tampere University of Technology. Publication; 1068, 2012. Y. Feng, Y. Zhuang, and Y. Pan, "Music information retrieval by detecting mood via computational media aesthetics," in Web Intelligence, 2003. WI 2003. Proceedings. IEEE/WIC International Conference on, 2003, pp. 235-241. M. Torrens, P. Hertzog, and J. L. Arcos, "Visualizing and exploring personal music libraries," in Proc. ISMIR, 2004, pp. 421-424. C. Forsell, "Evaluation in Information Visualization: Heuristic Evaluation," in Information Visualisation (IV), 2012 16th International Conference on, 2012, pp. 136-142. J. Holm and H. Siirtola, "A Comparison of Methods for Visualizing Musical Genres," in Information Visualisation (IV), 2012 16th International Conference on, 2012, pp. 636-645. E. Pampalk and M. Goto, "MusicRainbow: A New User Interface to Discover Artists Using Audio-based Similarity and Web-based Labeling," in 7th International Conference on Music Information Retrieval, Victoria, Canada, 2006. J. Holm, H. Siirtola, and L. Laaksonen, "Associating Avatars with Musical Genres," in Information Visualisation (IV), 2010 14th International Conference, 2010, pp. 186-193. A. Lehtiniemi and J. Holm, "Using Animated Mood Pictures in Music Recommendation," in Information Visualisation (IV), 2012 16th International Conference on, 2012, pp. 143-150. S. Lee, J. H. Kim, S. M. Kim, and W. Y. Yoo, "Smoodi: Mood-based music recommendation player," in Multimedia and Expo (ICME), 2011 IEEE International Conference on, 2011, pp. 1-4. M.-J. Yoo and I.-K. Lee, "Affecticon: Emotion-Based Icons for Music Retrieval," Computer Graphics and Applications, IEEE, vol. 31, pp. 89-95, 2011. G. Schatter and P. Kramer, "Mood-based selection of music collections by speech," in Consumer Electronics (ISCE), 2011 IEEE 15th International Symposium on, 2011, pp. 249-253. O. Celma, Music recommendation and discovery: Springer, 2010. R. S. Wurman, L. Leifer, D. Sume, and K. Whitehouse, Information anxiety 2 vol. 6000: Que Indianapolis, IN, 2001. X. Hu and J. S. Downie, "Exploring mood metadata: Relationships with genre, artist and usage metadata," in Proceedings of the International Conference on Music Information Retrieval, 2007.

[19] M. Carpendale, "Considering visual variables as a basis for information visualisation," Computer Science TR# 2001-693, vol. 16, 2003. [20] R. Poulin, The Language of Graphic Design: An Illustrated Handbook for Understanding Fundamental Design Principles: Rockport Publishers, 2011. [21] M. S. Landy and N. Graham, "Visual perception of texture," The visual neurosciences, vol. 2, pp. 1106-1118, 2004. [22] D. Zhou, "Texture analysis and synthesis using a generic MarkovGibbs image model," ResearchSpace@ Auckland, 2006. [23] H. Zuo, T. Hope, M. Jones, and P. Castle, "Sensory interaction with materials," in Design and emotion, 2004, pp. 223-227. [24] J. Sedgwick, B. Henson, and C. Barnes, "Sensual surfaces: engaging consumers through surface textures," in Proceedings of the 2003 International Conference on Designing pleasurable products and interfaces, 2003, pp. 148-149. [25] K. Wang, "Research of the affective responses to product's texture based on the Kansei evaluation," in Computational Intelligence and Design, 2009. ISCID'09. Second International Symposium on, 2009, pp. 352-355. [26] S. Card, J. Mackinlay, and B. Shneiderman, Readings in information visualization: using vision to think: Morgan Kaufmann, 1999. [27] N. Kaya and H. H. Epps, "Relationship between color and emotion: a study of college students," College student journal, vol. 38, pp. 396405, 2004. [28] P. Valdez and A. Mehrabian, "Effects of color on emotions," Journal of Experimental Psychology: General, vol. 123, p. 394, 1994. [29] D. Eftaiha. (2010, 22 January 2013). 6 Elements of Design for Striking Photographs. Available: http://photo.tutsplus.com/tutorials/photography-fundamentals/6elements-of-design-for-striking-photographs/ [30] S. Bradley. (2010, 20 Mac 2013). The Meaning of Shapes: Developing A Visual Grammar. Available: http://www.vanseodesign.com/web-design/visual-grammar-shapes/ [31] J. Hurtienne, C. Stößel, and K. Weber, "Sad is heavy and happy is light: population stereotypes of tangible object attributes," in Proceedings of the 3rd International Conference on Tangible and Embedded Interaction, 2009, pp. 61-68. [32] C. J. Boyatzis and R. Varghese, "Children's emotional associations with colors," The Journal of genetic psychology, vol. 155, pp. 77-85, 1994. [33] M. Hemphill, "A note on adults' color–emotion associations," The Journal of genetic psychology, vol. 157, pp. 275-280, 1996. [34] P. Brodatz, Textures: a photographic album for artists and designers vol. 66: Dover New York, 1966.