Musical parameters and images of motion

Musical parameters and images of motion

Zohar Eitan Department of Musicology, Tel Aviv University, Israel [email protected]

Roni Y. Granot Department of Music, The Hebrew University of Jerusalem, Israel [email protected]

In: R. Parncutt, A. Kessler & F. Zimmer (Eds.) Proceedings of the Conference on Interdisciplinary Musicology (CIM04) Graz/Austria, 15-18 April, 2004 http://gewi.uni-graz.at/~cim04/

Background in music theory. Music theorists and aestheticians have long suggested that musical gestures are isomorphic with expressive motion (e.g., Kurth, 1991; Scruton, 1997). The ramifications of this hypothesis can be observed in attempts to map changes in diverse musical parameters onto curves representing spatial motion (see Shove & Repp, 1995), in analyses of musical processes as sonic embodiments of “image schemas” (e.g., Zbikowski, 2002; Spitzer, 2003), or the proposal that analogies of musical and bodily gestures supply the “iconic” component in music semiotics (e.g., Hatten 2002; Lidov, 1999). Background in music psychology. Various empirical and theoretical studies suggest that the relationship between music and motion is fundamental to music processing. For instance, Sundberg and Verillo (1980) showed that performers' tempo fluctuations, such as the final retard, match the velocity profiles of natural human locomotion. In the realm of perception, Gabrielsson (1973) reported that similarity ratings of rhythmic figures are affected by the motion qualities listeners ascribe to these figures. One mechanism assumed to mediate between music and motion is motor images evoked by matching spatiotemporal auditory stimuli (e.g., Todd, 1999). These studies notwithstanding, little attention has been given to the effect musical parameters other than rhythm have on the motional implications of music, and little has been done by way of mapping musical gestures onto listeners' visuo-motor images. Aims. We explore how different musical parameters affect listeners’ mental images of bodily motion by examining the effect of intensifications and abatements (in pitch, loudness, or IOI) on listeners' associated motion parameters. Method o

Subjects: 77 college students, 37 of whom were musically trained.

o Task: Subjects visualized an animated human character of their choice. They were then presented with brief melodic figures, for each of which they were asked to specify their character’s imagined motion, including motion type (e.g., walking, jumping), vertical direction, change of distance from spectator, direction on the horizontal axis, change of pace, energy level, and presence of external force . o Experimental materials: Brief melodic figures were derived from pairs of stimuli, each pair including an increase versus a decrease in a specific musical parameter, (e.g., crescendo vs. diminuendo), with other parameters held constant. Manipulated parameters were: dynamics, pitch direction, pitch intervals, attack rate (IOI), motivic pace, and articulation. Results o All musical parameters affected motion imagery in several dimensions. For instance, pitch contour affected imagined motion along all three spatial axes (not only up-down), as well as velocity and energy. o Many musical-spatial analogies are asymmetrical: a musical change in one direction often evokes a significantly stronger spatial analogy than its opposite. For instance, crescendi are strongly associated with increasing velocity. In contrast, diminuendi are not related to decreasing velocity but rather to spatial descent. o In general, musical abatements are associated with spatial descents, while intensifications are associated with increasing velocity, rather then ascent. Conclusions o

Musical parameters affect motion imagery in demonstrable, consistent ways.

o Listeners’ music-induced imagery suggests that hitherto:

1

musical “space” is much more complex than assumed

CIM04 - Conference on Interdisciplinary Musicology - Proceedings

o

Correspondence of intensity profiles generates one-to-many musical-motional associations.

o

Listeners may utilize diverse, conflicting strategies when mapping sound onto bodily motion.

o

Opposite directions in musical parameters are not experienced as polar opposites. Rather, each stresses different kinetic and spatial associations.

musicians), as are speeding or ascending in the domain of human motion.

Of the ideas associating music with nonauditory domains, the notion that music depicts analogs of physical motion is perhaps the oldest and the most influential (Cohen, 2001). The experience of music as motion has been considered pertinent to musical understanding (e.g., Scruton, 1997), and was proposed as an important link between music and emotion by aestheticians (e.g., Hanslick, 1891; Kivy, 1980; Langer, 1953), semioticians (e.g., Lidov, 1999), and music theorists (e.g., Kurth, 1991; Spitzer, 2003; Zbikowsky, 2002).

Converging evidence from a variety of disciplines suggests that comparable intensity levels or intensity contours may indeed associate different perceptual and conceptual domains. The empirical support for a crossdimensional notion of intensity includes perception and psychophysics experiments using different research paradigms, among them cross-dimensional matching (Stevens, 1975), and cross-dimensional interference in speeded classification, (Marks, 1978; Melara and Marks 1990a, 1990b, 1990c; see Marks, 2000 for a survey of other works). Additional corroboration for both cross-modal and intramodal intensity analogies is supplied by studies of adult-infant communication (e.g., Papousek, 1996; Stern, 1985). These indicate that infants and parents often connect through mutual imitation of crossmodal intensity contours, involving auditory dimensions such as pitch inflection and dynamics, as well as touch and motion.

Associations of specific Parameters. >From the listener’s perspective, the notion of musical-kinetic analogy implies that changes in specific musical parameters (such as pitch height, loudness, or tempo) would be associated with corresponding changes in specific dimensions of motion (such as spatial verticality, distance, or speed). Some such analogies may easily suggest themselves. Temporal features, such as tempo or attack rate (IOI), are associated with the speed or velocity of physical motion, and several empirical studies suggest that such correspondence of expressive timing and patterns of bodily motion indeed affects listeners’ perception and aesthetic response (Repp, 1992; Juslin, Friberg, and Bresin, 2002). Pitch is often related to spatial verticality, and this association, whether innate or learned, seems to affect both visual and auditory perception (e.g., Melara and O’brien, 1987; Wagner et al., 1981). Pitch is also associated with distance (as in the Doppler Effect), and Loudness - with both distance and the level of energy which activates motion.

Also relevant to the present study are findings suggesting that intensity changes in one auditory dimension may create an illusion of corresponding changes in another. Such intra-modal relationships may produce “crossing-over” of the kinetic associations of one musical parameter (e.g., pitch and verticality) to another (e.g., dynamics). Cross-dimensional interference was found for loudness and pitch, reciprocally (Neuhoff and McBeath 1996; Neuhoff, McBeath and Wanzie 1999); for pitch accent and loudness (Tekman, 1997) dynamic (loudness) accent and duration (ibid.), and pitch and tempo (Collier & Hubbard, 2001). The notion of an intensity or tension “curve” which plots the dynamic progression of a musical segment or composition, determined by the combined ebb and flow various musical dimensions, has been independently suggested by several music theorists throughout the 20th century, most notably

Intensity-based mappings. Beyond associations of specific parameters, musical and motional gestures may also be related through isomorphic intensity curves. A pitch rise, a crescendo, and an accelerando are commonly considered intensifying (at least by

2


Kurth (see 1991), as well as (more recently), by Berry (1976), Meyer (1989), and Rink (1999). Diverse concrete mappings of the contours of musical intensity into physical motion have been proposed by early 20th century musicians (E.g., Truslit, Becking, Sievers; see Shove and Repp, 1995), movement theorists (e.g. Jaques-Dalcroze, 1921), and film theorists and practitioners, such as Eisenstein (see Cook, 1998).

Hypotheses In this paper, we examine the hypothesis that musical parameters are associated with spatio-kinetic parameters by investigating whether changes in musical parameters evoke corresponding changes in listeners’ spatial and kinetic imagery. The paper reports the results of a first in a series of experiments investigating this hypothesis, and discusses some of the ramifications for music theory and analysis.

While discussing this extensive body of work is beyond the scope of this article, few recent empirically oriented models of intensity contours in music should be mentioned. Todd (see 1999 for a summary) has proposed a model relating intensity contours in music to human motion perception. He suggests that two sensory mechanisms, the vestibular and the audio-visuo-motor, translate kinetic information into auditory stimuli. This translation, which applies to two kinds of motion, gestural (continuous expressive motion) and locomotive (associated with tempo in metrical music), is mediated through the integrated intensity profile generated by various psychoacoustical parameters (tempo, dynamics, articulation, timbre, and vibrato) throughout a piece or section.

Associations of specific Parameters As suggested above, we may hypothesize that participants will associate

The proposed relationship between intensity curves and motional gestures is echoed in Manfred Clynes’ notion of "essentic form" archetypical icons of basic human emotions (Clynes and Nettheim, 1983). These contours may be depicted aurally through pitch and loudness, as well as through other modes, such as touch, motion, and visual curvature. Clynes demonstrated that the emotions associated with essentic shapes may be correctly identified cross-culturally when these shapes are presented through sound, vision or haptic pressure. Regardless of his theory of “sentics” itself (which has not been not universally accepted), Clynes’ results exhibit the ability to associate intensity contours cross-modally. Cross-modal relationships of motion, emotion, and musical features are also exemplified by a number of more recent studies which demonstrated that the perceived “musical tension" seems to correlate with the tension conveyed by human motion (e.g., Krumhansl and Schenk,1997).

o

IOI changes and speed changes

o

Changes in pitch with changes in spatial verticality

o

Pitch changes with changes in distance (pitch rise would imply approaching motion, and pitch fall – motion away, as in the Doppler effect)

o

Loudness changes with changes in distance

o

Loudness changes with changes in ”energy”

Intensity “crossovers” As mentioned, music and motion may be related through cross-dimensional intensity both directly and indirectly. Directly, similarity of intensity contours may affect the mapping of music into motion: musical processes perceived as intensifying would map into intensifying motions, and musical abatements – into abating motions. Indirectly, as an intensity contour in one musical dimension may be associated with that of another (e.g., a diminuendo and a pitch fall), and thus draw on the latter’s kinetic associations (e.g., a diminuendo might be perceived as “falling”). With regard to the present study, such relationships would suggest the following hypotheses: 1. Correlation of intensity vectors in music and motion. Intensifying changes in musical stimuli would evoke corresponding images of kinetic intensifications, while

3


musical abatements would be associated with motions implying decreasing intensity.

(7) whether this force supports, opposes or sidetracks it

A subsidiary hypothesis follows:

(8) the character’s “energy” level for each motion (on a 1-7 scale).

2. One-to-many mapping. Each musical parameter may map onto several motional features presenting analogous intensity profiles.

Subjects heard stimuli in groups of 5-8 people, in a single session of 45 minutes. Stimuli were presented, to each group, in a different random ordering.

We also hypothesize that associations between music and motion are directionally symmetrical:

Experimental materials: The musical stimuli (see Example 1) consisted of pairs of brief melodic figures. One member of each pair presented an “increase” in a specific musical parameter, while the other presented a “decrease” (e.g., crescendo vs. diminuendo, accelerando vs. ritardando). Other parameters were held constant for each pair. Parameters investigated were dynamics, pitch direction, pitch intervals, attack rate (IOI), motivic pace, and articulation. To minimize the effects of tonal implications, all stimuli were tonally ambiguous. Stimuli were created through Sibelius 1.2 music software, using the software’s Grand Piano sound, recorded into an audio CD using 2 identical tracks, and presented to subjects through loudspeakers. Figure 1 presents some of the stimuli used (BPM=160).

3. Symmetry of associative space. Other things being equal, diametrically opposed musical processes would evoke diametrically opposed kinetic processes . In experimental terms: a listener who associates a musical stimulus m (e.g., a crescendo) with a kinetic quality k (e.g., a spatial ascent) would associate the inverse stimulus –m (e.g., diminuendo) with the opposite kinetic quality –k (e.g., descent).

Method Subjects: 77 college students, 32 males, 45 females (mean age=24.9), 37 of which (“musicians”) had at least 7 years of musical training. Task: Subjects were first asked to visualize an animated (“cartoon”) human character of their choice. They then heard brief melodic figures, and for each figure had to visualize their character moving in an imaginary animated film shot, the melody serving as its “soundtrack.” For each melodic figure, subjects specified their character’s imagined motion in several ways: (1) motion type jumping, sliding)

(e.g.,

walking,

Statistical Methods. Answers to ordered multiple choice items were coded as (–1) or (1) (for opposing answers), and (0) for a neutral answer (e.g., in the verticality item, “ascending” is +1, “descending” is -1, and “level” is 0). Item 8 (level of energy) was analyzed as continuous variables. Most of the statistical analyses are based on differences between the coded answers to paired musical motives representing an intensifying figure (e.g., no. 1, a crescendo) and an abating figure (e.g., no. 2, a diminuendo). Paired comparisons for each question were carried out using the Wilcoxon test on the differences in the coded answers to the two members of each pair of motives. A Mann-Whitney test indicated whether differences between musicians and nonmusicians were significant.

running,

(2) vertical direction (ascending, descending, or level) (3) change of (virtual) distance from spectator: approaching, moving away, or neither (4) direction on the horizontal axis: motion to the right (relative to the spectator), left, or neither

For questions with opposing answers, chisquared tests were used for each motive to test the hypothesis that non-neutral responses favored one of the two directions (e.g., right vs. left motion). In addition, in

(5) change of the character’s pace (6) whether an external force (excluding gravity) interferes with the imagined motion

4


each pair of motives the sign test was used to test whether such tendencies were stronger in the intensifying than in the abating motive. The false discovery rate procedure (Benjamini and Hochberg 1995) was used to account for multiple testing.

Results Results are presented in Tables 1 & 2. Table 1 specifies the motion parameters significantly associated with each musical parameter. The most intriguing finding of this experiment, the asymmetry of imagined musical spaces, is presented is Table 2. These findings are discussed in the conclusions section below.

Conclusions Musical parameters affect motion imagery strongly and diversely. All musical parameters investigated significantly affected parameters of motion imagery, and all parameters of imagined motion were affected by music parameters. Relationships of specific musical and motional parameters. The present experiments corroborate several of the hypotheses concerning particular musical– motional relationships. Aspects of timing (IOI and motivic pace) were strongly related to speed; aspects of pitch contour were related to verticality; and aspects of loudness to distance and energy. In contrast, the expected Doppler-like relationship of pitch and distance was reversed: pitch rise was associated with increasing distance from the observer, and pitch fall - with decreasing distance. However, the overall picture emerging from this study is that imagined musical space is much more complex than that implied by entrenched analogies: deep-rooted associations, such as that of verticality and pitch contour, are intricately and asymmetrically structured (see below) and less established relationships (such as that of dynamics and verticality) prove to be as important as established ones. Intensity direction often matches musical and motional parameters. Overall, intensifications in musical parameters are indeed associated with intensifying motion,

while musical abatements are related to abating or passive motions. Intensity correlations, as predicted, seem to generate one-to-many mappings, in which predictable musical-motional associations “spill over” to less obvious ones. For instance, pitch rises are not related to spatial rise alone, but also to increase in velocity, though there is no IOI change in the musical stimuli. Crescendo also suggests, in addition to incoming motion, increasing velocity, though again IOI are isochronous; and diminuendo is associated not only with motion away from the listener, but also with descent, though there is no pitch change in the musical stimulus. Multiple mapping strategies? Some significant musical-motional mappings, however, cannot be easily explained in terms of intensification relationships alone: pitch tends to be associated with horizontal motion, i.e., ascent to the right and descent to the left; pitch rises move away from the listener, rather than towards him. Decreasing pitch intervals are also related to increasing distance; and musical acceleration (an increase, or intensification, in temporal density) is related to descent (associated with decreasing tension and effort). These mappings suggest that listeners do not necessarily use an all-encompassing mapping strategy (such as intensification analogy) for all stimuli. Rather, different parameters (or even specific stimuli) may suggest different mapping strategies. Some mappings may be based on intensity isomorphism, or on other analogies between visual and auditory “space” For instance, mapping pitch intervals into perceived visual perspective (where distant objects seem “smaller”) may account for the association of decreasing intervals with moving away. Other kinds of mapping may be concrete and ecological: the structure of piano keyboard may have affected associations of pitch height with lateral motion (a hypothesis supported by the more decisive results of musicians in this respect); and the association of an accelerating repeated pitch with descent may also be iconic and particular, such as imagining a ball bouncing downhill, as suggested by some of our participants in their free commentaries.

5


TABLE 1. Motion Features Associated with Musical Parameters. * indicates p