Intelligent Music Systems in Music Therapy

Intelligent Music Systems in Music Therapy Research Plan Petri Toiviainen University of Jyväskylä Department of Music PL 35(M) 40014 University of Jyväskylä Finland email: [email protected]

Table of contents 1. Background .....................................................................................................2 1.1. Background and significance ...............................................................2 1.2. Previous activity ...................................................................................3 2. Objectives and methods ......................................................................................6 3. Results .................................................................................................................7 4. References ...........................................................................................................8

Intelligent Music Systems in Music Therapy / Research plan

2

1. Background 1.1. Background and significance Intelligent Music Systems refer to computer systems that attempt to model various activities related to music perception, learning, and production (see, e.g., Miranda 2000). To this end, a variety of modeling methods are used, including rulebased Artificial Intelligence, neural networks, dynamic systems, and probabilistic reasoning. From the point of view of the present proposal, two classes of these systems are of interest: automatic music analysis systems and interactive music systems. Automatic music analysis systems apply algorithmic techniques for analyzing, for instance, the melodic, harmonic and/or rhythmic structure of the piece of music under examination. Interactive music systems change their behavior in response to the actions of musical performer. An example of the latter is a beat tracker, that is, a system that synchronizes to the pulse of a musical performance in real time. The clinical populations of music therapy are individuals who have mental retardation, learning disabilities, hearing impairments, visual impairments, orthopedic impairments, communication disorders or impairments, autism, mental or behavioral disorders or severe emotional disturbances (Schmidt Peters, 2000). The strength of music therapy lies in that a meaningful therapeutic process is possible even when the client has severe limitations in expressing him/herself verbally. Improvisation in its many forms and ways of use is one of the most important tools of music therapy. It has been used with numerous client populations in order to help a client to achieve a state of well-being. After Bruscia (1987) the goals of improvisation can be educational, recreational or therapeutic in nature: ”Educational goals are concerned with helping the client acquire knowledge or skills in music or another related discipline. Recreational goals are concerned with improving the client’s use of leisure time. Therapeutic goals are concerned with helping the client gain insight about him/herself, work through feelings, problems, and symptoms, make basic changes in his/her personality, and develop more effective methods of adaptation. Of course, these goal areas overlap frequently.” Theoretically, music therapeutic improvisation can be examined from two main points of view. First, it can be studied as a psychical process, in which case the main interest in the improvisation analysis is in the observation of the client’s mental state (emotions, associations, mental images, and memories), as reflected by the improvisations. In this case, improvisation is thus studied from the point of view of the symbolic meanings represented by it. Second, music therapeutic improvisation can be studied as a physical process, in which case the main interest in the analysis is in the bodily aspects, such as coarse and fine motor actions and their development. Within these two perspectives, several studies can be found that support the meaning of improvisational approach when promoting pscyhological and physical well-being (e.g. Aldridge, 1996; Erkkilä, 1997; Scmeijsters & Van den Hurk, 1993; Aldrige, 2000). The studies clearly show that improvisation reflects both the psychological and physiological state of the client and can thus be used as a controlled tool of therapy by the experienced therapist.


3

Clinicians and researchers of music therapy apply various methods for the analysis of improvisation to evaluate the current state of the client and make treatment plans of both short and long term. Among various models proposed for the analysis of improvisation (Nordoff, Robbins, & Fraknoi, 1980; Bergstrøm-Nielsen, 1991; Erkkilä, 2000), the IAP (Improvisation Assessment Profiles) model by Bruscia (1987) is probably the best known. The IAP model attempts to enhance the therapist’s understanding of the client through an analysis of certain elements of improvisations performed by the client. These elements include the rhythmic, melodic, harmonic, tonal, and textural structure; the analyst rates each element according to its degree of integration, variability, tension, congruence, salience, and autonomy. This results in numerical profiles describing the quality of the improvisation, which enhance the therapist’s understanding of the client. A common problem with all the current analysis methods is that improvisation, due to its complex and multi-layered nature, is difficult and tedious to analyze in detail. Furthermore, traditional music analysis methods can not be applied because of the free character of the improvisations. Consequently, the analyses are often based on a relatively general impression on the improvisation, possibly neglecting some of the micro-structure present in it. A great portion of the components included in, for instance, the IAP model, could be analyzed automatically with computational algorithms, especially when the improvisations are performed with a MIDI instrument. This would help the therapy process by making the analysis stage more objective and less time-consuming. A notable portion of music therapy clients consists of persons whose mental capacities would allow versatile self-expression but whose limited motor abilities make this expression difficult. Typical representatives of such clients are persons with bodily of neurological dysfunctions caused by birth defects, illnesses, or injuries. To enable to musical expression of those clients, sensor technology (e.g., ultra-sound beams) has been used to translate physical gestures into musical output. The use of such systems in music therapy is, however, not well developed: typically, the sensor data are simply mapped onto some musical parameter such as pitch height. More intelligent behavior could be obtained by applying above-mentioned feature extraction methods to the sensor data. An example of that would be a beat tracker connected to motion data taken from a video signal. This kind of a device would enable, for instance, a quadriplegic patient to produce a rhythmic output or conduct a virtual orchestra with head movements. 1.2. Previous activity A research group concentrating on cognitive musicology has been working at the Department of Music of the University of Jyväskylä since 1990. The group also includes a graduate research school. Launched under the Research Program of Cognitive Science of the Academy of Finland, this research project was entitled “Perception, learning, and production of music as a cognitive process”. The group developed new scientific co-operation projects whose funding was continued by the Academy of Finland during the periods 1993-96 (“Physicalism, connectionism, and representation”) and 1996-99 (“Tacit knowledge in complex mind-environment systems”). In 1999, a research school (Pythagoras Graduate School of Sound and Music Research) was established. Coordinated by the Department of Music, the school includes the following co-operative partners: Helsinki University of


4

Technology (Departments of Acoustics and Multimedia), Helsinki University (Cognitive Brain Research Unit), and Sibelius Academy. The research on cognitive musicology at the University of Jyväskylä has used psychological research and computational modeling as the main research paradigms. With these paradigms, various musical activities have been studied. This research has yielded a number of computational algorithms for the melodic, harmonic, rhythmic, and tonal analysis of music that are applicable to the proposed research either as such or as modified. These algorithms are based mainly on the following methods: statistical analysis, neural networks, complex dynamical systems, and probabilistic reasoning. In most studies conducted, the models have been validated by comparing their output with psychological data. Below, the studies most relevant to the proposed research are discussed. Rhythm. Studies on rhythm perception have concentrated on pulse finding and beat tracking. The process of pulse finding has been modeled using a resonance dynamics scheme applied to a bank of oscillators (Toiviainen, 1997; Toiviainen & Snyder, 2000, 2003). This model listens to a musical stimulus and outputs dynamically a set of resonance values, each of which represents the current salience of a possible pulse sensation. Comparisons of the output of the resonance model with tapping data obtained from human listeners have shown that the model reliably predicts the perceived pulse. In addition, it provides a measure of the perceived beat clarity and its change over time. Beat tracking refers to the process of synchronizing to a musical performance that is not metronomic. The process of beat tracking has been modeled with adaptive oscillators (Toiviainen, 1998). Based on the pulse-finding and beat-tracking models described above, the applicant has developed a real-time interactive computer system that finds the basic pulse of a musical performance, follows the tempo changes in it, and produces an accompaniment synchronized with the performance (Toiviainen, 2001). Melody. A number of studies on melody perception have been conducted at the Department of Music. Järvinen and Toiviainen (2000) applied statistical analysis methods to improvised jazz melodies to investigate the effect of musical meter on the use of tones. Melodic expectations were studied by combining psychological experiments, statistical analysis, and neural network modeling (Krumhansl, Louhivuori, Toiviainen, Järvinen, & Eerola, 1999; Krumhansl, Toivanen, Eerola, Toiviainen, Järvinen, & Louhivuori, 2000). These studies showed that certain statistical distributions (e.g., pitch class distributions) derived from the melodies were reliable predictors for listeners' melodic expectations. Categorization of melodies has been studied by applying various automatic feature extraction methods to musical stimuli, and comparing thus obtained representations with similarity ratings (Eerola, Louhivuori, Järvinen, & Toiviainen, 2001). Frequency-based properties of the melodies were found to be relatively accurate predictors for the perceived similarities. Harmony. The P.I. has developed algorithms for computational analysis of harmony. These methods are based on probabilistic reasoning, more specifically, Hidden Markov Models and Bayesian filters. Until now, the output of these algorithms has not been systematically compared with human judgments. The algorithms have been, however, implemented as a real-time interactive music system, the Intelligent Jazz Accompanist (IJA; Toiviainen, 2001; see also http://www.cc.jyu.fi/~ptoiviai/ija/ija.html). The IJA utilizes the pulse-finding and beat-tracking algorithms described earlier to find and follow the beat of a jazz improvisation, and utilizes probabilistic reasoning to find the underlying harmony of


5

the improvisation as well as the current location in the musical structure. After having recognized the improvisation, it joins in with the accompaniment of the piece the musician is playing. Tonality. The perception of tonality has been modeled using dynamical systems and neural networks (Krumhansl & Toiviainen, 2000, 2003). Starting from musical input, the model provides dynamical predictions of both the locus and the degree of clarity of tonality. The model's output has been compared with psychological data obtained from probe tone experiments. It was found that the output of the model correlates significantly with the psychological data, in terms of both the locus and the clarity of tonality. Music therapy. The Master Program of music therapy of the University of Jyväskylä, the only of its kind in Finland, was launched in 1997. The Department of Music also has Finland’s only professor in Music Therapy, Jaakko Erkkilä. Improvisation as a tool in clinical work has been an important line of music therapy research at the department. The topic has been dealt at first theoretically (Erkkilä, 1997; Lehtonen, 1987, 1988), mainly from a psychodynamic angle. As the product of the theoretical studies, a new line of research, where the theoretical knowledge has been applied when solving the problems of improvisation analysis has arisen (Erkkilä, 2000). Furthermore, the possibilities of music technology have been utilized in clinical practice to enable the musical expression of those with severe motor limitations. Preliminary studies on computer-based feature extraction from music therapy improvisations have been carried out in spring 2002, showing promising results (Erkkilä, 2002). In terms of the present proposal, the five most important publications of the research team are: • Eerola, T., Järvinen, T., Louhivuori, J., & Toiviainen, P. 2001. Statistical features and perceived similarity of folk melodies. Music Perception, 18(3), 275-296. • Erkkilä, J. 2000. A proposition for the didactics of music therapy improvisation. Nordic Journal of Music Therapy, 9(1), 13-25. • Toiviainen, P. 2001. Real-time recognition of improvisations with adaptive oscillators and a recursive Bayesian classifier. Journal of New Music Research, 30(2), 137-148. • Toiviainen, P. 1998. An interactive MIDI accompanist. Computer Music Journal, 22(4), 63-75. • Toiviainen, P. & Krumhansl, C. L. (2003). Measuring and modeling realtime responses to music: the dynamics of tonality induction. Perception, 32(6), 741-766.

2. Objectives and methods The objectives of the project are twofold. The first objective is to develop automatic music analysis systems that can be used, among others, in analyzing improvisations produced in clinical music therapy. The second objective is to develop interactive music systems that facilitate clinical music therapy work with persons having limited motor abilities. Although seemingly different, these two goals share similar prerequisites in that they both rely on automatic extraction of musical features.


6

The main difference is that interactive music systems must be implementable in real time, whereas this is not required for automatic music analysis systems. We suppose that suitably chosen features extracted from a musical performance (e.g. clinical improvisation) can be used to predict assessments given, and thus psychic meanings attained, by therapists. Furthermore, we assume that these methods could be developed into computational analysis tools that would help render clinical music therapy work more effective. Finally, we suppose that interactive music systems based on intelligent musical feature extraction would be more rewarding and efficient than the present ones from the point of view of music therapy clients. To carry out automatic extraction of musical features, current knowledge about musicology, psychoacoustics, and the perception of melody, harmony, rhythm, and tonality will be applied. The methods will be based on • statistical analysis; • information theory (entropy-based variables); • neural networks; • dynamic systems (adaptive oscillators, dynamic key-finding models), • Monte Carlo simulation (Hidden Markov Models). The connection between the extracted musical features and the perceived qualities of improvisations will be studied using psychological tests. To this end, a large set of improvisations will be subjected to automatic feature extraction to obtain a feature vector describing each improvisation. For the same set of improvisations, experienced music therapists and musicologists will provide Improvisation Assessment Profiles. A mapping from the musical features to the IAPs will be carried out, for instance, with supervised artificial neural networks using the pairs of feature vectors and IAPs as training data. This yields a system that maps a musical (MIDI) input onto IAPs (see Figure 1.).

Fig. 1. A schematic representation of the method for obtaining a system for automated analysis of improvisations.

Interactive music systems will be developed by applying methods of pattern recognition to various sensor data, such as a video stream, and integrating these methods with existing methods for real-time musical feature extraction, such as the beat-tracking and harmonic analysis systems already developed by the research team (Toiviainen, 1998, 2001). The development of the systems will be carried out in collaboration with clinical music therapists, and their suitability will be assessed by interviewing both therapists and clients. The testing and evaluation of the applications developed will be carried out in collaboration with the Eino Roiha Institute.


7

Time schedule and intermediate objectives for the research year quarter

2003 I

II

III

2004 IV

I

II

III

2005 IV

I

II

III

IV

Collection of improvisations Subjective judgments of improvisations by music therapists using assessment profiles Development of algorithms for musical feature extraction Development of algorithms for using sensor data to control musical output in interactive music systems Study of the correspondence between extracted musical features and subjective judgments of improvisations Development of a system for automatic generation of improvisation assessment profiles Testing of the systems with music therapy clients in laboratory environment Clinical testing of algorithms with various client groups Development of end products (computer programs for the analysis of improvisations, interactive music systems for music therapy use)

Ethical questions All the clinical music therapy work that will part of the research will be conducted obeying the ethical principles established in music therapy and using qualified music therapists. The psychological experiments will be based on voluntariness and anonymity.

3. Results From a theoretical point of view, the proposed research will widen our knowledge about how various musical features of musical improvisations relate to perceived qualities. Furthermore, it will enhance our understanding of musical improvisation as a part of music therapy work. Finally, it will provide new insights into human-computer interaction in musical domain.


8

From a practical point of view, the proposed research will provide software for the analysis of improvisations that can be used by music therapists, musicologists, and music educators. Furthermore, the music analysis tools can be employed in various music-related “welfare technology” innovations, such as intelligent search agents. Finally, the research will provide new tools for interactive control of music. Besides in music therapy, these tools can be applied in music education and performance. Results of the proposed research will be presented at meetings related to cognitive musicology and music technology, such as, the Society for Music Perception and Cognition (SMPC) conference, the European Society for the Cognitive Sciences of Music (ESCOM) conference, and International Computer Music Conference (ICMC), as well as at meetings related to music therapy, such as European and World Congresses of Music Therapy. Manuscripts reporting the results of the proposed research would be appropriate for journals such as Music Perception, Journal of New Music Research, Computer Music Journal, Psychology of Music, Journal of Intelligent Systems, Nordic Journal of Music Therapy, British Journal of Music Therapy, Music Therapy Perspectives, Journal of Music Therapy, and for electronic portals such as Music Therapy World. The proposed research might generate inventions (analysis software, interactive music systems), that could further be exploited commercially. Should this be the case, the innovations will be protected by applying for either patents or utility models, and the commercializing of them commenced in collaboration with Jyväskylä Science Park Ltd. (Jyväskylän teknologiakeskus). The author's rights will be possessed by the researcher(s) behind the innovation. Any patent applications will be reported to the Academy of Finland.

4. References Aldridge, D. (1996). Music therapy research and practice in medicine : from out of the silence. London: J. Kingsley. Aldridge, G. (2000). The implications of melodic expression for Music Therapy with a breast cancer patient. In D. Aldridge (Ed.), Music Therapy in palliative care: more new voices (pp. 135-153). London: Jessicsa Kingsley. Bruscia, K. E. (1987). Improvisation Assessment Profiles - The Bruscia Model. In K. Bruscia (Ed.), Improvisational Models of Music Therapy (pp. 401-487). Springfield, Illinois, U.S.A: Charles C Thomas Publisher. Eerola, T. & North, A. C. (2000). Expectancy-Based Model of Melodic Complexity. In Woods, C., Luck, G.B., Brochard, R., O'Neill, S. A., & Sloboda, J. A. (Eds.), Proceedings of the Sixth International Conference on Music Perception and Cognition. Keele: Keele University. Eerola, T., Järvinen, T., Louhivuori, J., & Toiviainen, P. (2001). Statistical features and perceived similarity of folk melodies. Music Perception, 18(3), 275-296. Erkkilä, J. (1997). From the Unconscious to the Conscious - Musical Improvisation and Drawings as Tools in the Music Therapy of Children. Nordic Journal of Music Therapy, 6(2), 112-120. Erkkilä, J. (2002). The relationship between psychodynamic meanings and musical features in clinical improvisation. Presentation at the Finnish-Hungarian Conference on Musicology, Budapest, April 23, 2002.


9

Järvinen, T. & Toiviainen, P. (2000). The effect of metre on the use of tones in jazz improvisation. Musicae Scientiae, IV/1, 55-74. Krumhansl, C. L., & Toiviainen, P. (2000). Dynamics of tonality induction: A new method and a new model. In C. Woods, G. Luck, R. Brochard, F. Seddon, & J. A. Sloboda (Eds.), Proceedings of the Sixth International Conference on Music Perception and Cognition. Keele: Keele University. Krumhansl, C. L., Louhivuori, J., Toiviainen, P., Järvinen, T., & Eerola, T. (1999). Melodic expectation in Finnish folk hymns: Convergence of statistical, behavioral, and computational approaches. Music Perception, 17(2), 151-196. Krumhansl, C.L., Toivanen, P., Eerola, T., Toiviainen, P., Järvinen, T., & Louhivuori, J. (2000). Cross-cultural music cognition: Cognitive methodology applied to North Sami yoiks. Cognition, 75, 1-46. Lehtonen, K. (1987). Creativity, the Symbolic Process and Object Relationships. Creative Child and Adult Quarterly, 12, 259-270. Lehtonen, K. (1988). Is there correspondence between the structures of music and the psyche? Psychiatria Fennica, 19, 51-61. Scmidt Peters, J. (2000). Music Therapy - An Introduction. Springfield. Illinois. U.S.A: Charles C Thomas Publisher, Ltd. Smeijsters, H., & Hurk, J. v. d. (1993). Research in Practice on the Music Therapeutic Treatment of a Client with Symptoms of Anorexia Nervosa. In M. Heal & T. Wigram (Eds.), Music Therapy in Health and Education (pp. 235-263 (NB In contents: 255-263)). London: Jessica Kingsley Publishers Ltd. Toiviainen, P. (1995). Modeling the target-note technique of bebop-style jazz improvisation: an artificial neural network approach. Music Perception, 12(4), 399-413. Toiviainen, P. (1998). An interactive MIDI accompanist. Computer Music Journal, 22(4), 63-75. Toiviainen, P. (2001). Real-time recognition of improvisations with adaptive oscillators and a recursive Bayesian classifier. Journal of New Music Research, 30(2), 137-148. Toiviainen, P. & Krumhansl, C. L. (2003). Measuring and modeling real-time responses to music: the dynamics of tonality induction. Perception, 32(6), 741-766. Toiviainen, P., & Snyder, J. (2000). The time-course of pulse sensation: Dynamics of beat induction. In C. Woods, G. Luck, R. Brochard, F. Seddon, & J. A. Sloboda (Eds.), Proceedings of the Sixth International Conference on Music Perception and Cognition. Keele: Keele University. Toiviainen, P. & Snyder, J. S. (2003). Tapping to Bach: Resonance-based modeling of pulse. Music Perception, 21(1), 43-80. Toiviainen. P. (1997). Modelling the perception of metre with competing subharmonic oscillators. In A. Gabrielsson (Ed.), Proceedings of the Third Triennial ESCOM Conference. Uppsala: Uppsala University, 511-516.