Language and Cognitive Processes Mnemonic effect ...

This article was downloaded by: [psyswc] On: 9 May 2011 Access details: Access Details: [subscription number 937411186] Publisher Psychology Press Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 3741 Mortimer Street, London W1T 3JH, UK

Language and Cognitive Processes

Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t713683153

Mnemonic effect of iconic gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall? Wing Chee Soa; Colin Sim Chen-Huia; Julie Low Wei-Shana a Department of Psychology, National University of Singapore, Singapore First published on: 09 May 2011

To cite this Article So, Wing Chee , Sim Chen-Hui, Colin and Low Wei-Shan, Julie(2011) 'Mnemonic effect of iconic

gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall?', Language and Cognitive Processes,, First published on: 09 May 2011 (iFirst) To link to this Article: DOI: 10.1080/01690965.2011.573220 URL: http://dx.doi.org/10.1080/01690965.2011.573220

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

LANGUAGE AND COGNITIVE PROCESSES 0000, 00 (00), 117

Mnemonic effect of iconic gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall? Wing Chee So, Colin Sim Chen-Hui, and Julie Low Wei-Shan

Downloaded By: [psyswc] At: 08:21 9 May 2011

Department of Psychology, National University of Singapore, Singapore

Abundant research has shown that encoding meaningful gesture, such as an iconic gesture, enhances memory. This paper asked whether gesture needs to carry meaning to improve memory recall by comparing the mnemonic effect of meaningful (i.e., iconic gestures) and nonmeaningful gestures (i.e., beat gestures). Beat gestures involve simple motoric movement produced along with the rhythm of the speech (e.g., hand with open palm flips outwards). Although beat gesture does not carry any semantic meaning, it serves a metacognitive function by marking the parts of speech that the speaker would like to emphasise. It also activates prosodic processing of the accompanying speech. We also asked whether the mnemonic effect of both types of gestures was found in both adults and children. In both experiments adults and 45-year-old children watched 3 different videos, each consisted of a list of 10 words (5 words for children), in 3 conditions (words accompanied by iconic gestures, words accompanied by beat gestures, and words not accompanied by gestures) and were asked to recall the words without moving their hands. Not surprisingly, encoding iconic gesture improved memory in both adults and children*they recalled more words when encoding them with iconic gestures than when encoding words alone. Interestingly, encoding beat gestures aided recall in adults, suggesting that both meaningful and nonmeaningful gestures could strengthen their memory. More importantly, adults recalled comparable

Correspondence should be addressed to Wing Chee So, National University of Singapore, BLK AS4 9 Arts LinkSingapore, Singapore. E-mail: [email protected] This work was supported by National University of Singapore Provost Research Funding Grant No. R-581-000-074-133 to W.C. So. We thank Joseph Tee Can Shou for preparing the stimuli for data collection. # 2011 Psychology Press, an imprint of the Taylor & Francis Group, an Informa business http://www.psypress.com/lcp

DOI: 10.1080/01690965.2011.573220

2

SO ET AL.

number of words when encoding them with iconic gestures and beat gestures. However, such mnemonic effect of beat gesture was not found in children, suggesting that young children might not be sensitive to the meta-cognitive aspects of beat gesture.


Keywords: Gesture; Memory; Semantic meaning.

When we talk, we gesture. Abundant research has shown that gesture and speech form an integrated system during the process of speech production (Cassell, McNeill, & McCullough, 1999; Clark, 1996; Kelly, Barr, Church, & Lynch, 1999; Kendon, 2004; Kita & Ozyurek, 2003; McNeill, 1992, 2005).1 Recently, there is growing evidence showing that gesture is closely integrated with speech during the process of speech comprehension (Bernadis, Salillas, & Caramelli, 2008; Holle & Gunter, 2007; Kelly, Kravitz, & Hopkins, 2004; Kelly, Manning, & Rodak, 2008; Kelly, Ozyurek, & Maris, 2010; Ozyurek, Willems, Kita, & Hagoort, 2007; Yap, So, Yap, Tan, & Teoh, 2011). The tight integration between speech and gesture has an impact on speech processing. Listeners are sensitive to the semantic content conveyed in speech as well as that in gesture when comprehending a message (Alibali, Flevares, & GoldinMeadow, 1997; Goldin-Meadow & Sandhofer, 1999; Kelly et al., 2004; So & Lim, in press). Moreover, listeners’ comprehension improves when speech is accompanied by gesture, especially when gesture and speech convey the same meaning (Beattie & Shovelton, 1999, 2002; Goldin-Meadow, 2003; Rogers, 1978; Valenzeno, Alibali, & Klatzky, 2003). Not only that gesture improves comprehension, it also aids memory. There is abundant evidence showing that encoding gestures is beneficial for memory recall. Riseborough (1981) asked adult participants to watch videos in which the narrator recited a list of verbs with or without gestures. The findings showed that participants recalled more words that were accompanied by gestures than those that were not. Likewise, Thompson (1995) found that adults recalled more words that were accompanied by gestures than those that were not. Goldin-Meadow, Kim, and Singer (1999) also found that children tended to remember the information conveyed by the teachers’ gestures. Cook, Mitchell, and Goldin-Meadow (2008) further showed that 8-year-old children who were instructed in both speech and gestures, as compared to being instructed with speech only, had better retention of mathematics strategies after a month. Finally, Tellier (2008) found that 5-year-old children who encoded words with gestures and reproduced them recalled more words than those who encoded words alone. 1 Other researchers argue that gesture and speech are independent systems (Krauss, 1998; Krauss, Morrel-Samuels, & Colasante, 1991).


GESTURE AND MEMORY

3

However, gestures in the aforementioned studies are iconic gestures or concrete deictic gestures. All of them carry semantic meanings or have concrete contents. It is not surprising that exposure to these gestures that are semantically redundant with accompanying speech could enhance memory. According to Clark and Paivio’s (1991) dual coding theory, learning is reinforced when encoding information in both verbal and nonverbal modalities. Nonverbal modality, such as gesture, can supplement verbal modality and leave a stronger trace of information in memory. Thus, viewing iconic gesture can provide a more elaborated memory trace in one’s mind and hence facilitate memory recall (Cohen, 1981, 1989; Cohen & Bean, 1983; Cohen & Stewart, 1982; Nilsoon & Craik, 1990; Woodall & Folger, 1985). The question of interest is whether gesture needs to carry meaning to improve recall. In order to address this question, we have to compare the mnemonic effect of meaningful gesture to nonmeaningful gesture. However, we have little knowledge of whether encoding nonmeaningful gesture plays a role in recall. Related to this line of research, Cohen and Otterbein (1992) instructed adults to watch two videos in which a narrator told two stories with or without gestures (pantomimes and nonpantomimes) in Experiment 1. Pantomimes were primarily enactments that contained clear semantic meaning even in a speech-less context. Nonpantomimic gestures were hand/ arm movements and were not meaningful in a speech-less context. Half the subjects viewed pantomimes and another half viewed nonpantomimes. In general, adults recalled more sentences when they encoded speech with gestures (regardless of the type of gestures) than when they did not although the difference was greater in pantomimes than nonpantomimes. In addition, there was no significant difference in the number of sentences recalled between pantomimes and nonpantomimes across adults. Cohen and Otterbein then concluded that encoding pantomimic gesture might not be necessarily better than encoding non pantomimic gesture in terms of memory recall. Rather, they showed about the same level of mnemonic effect. However, they might enhance memory in different mechanisms. Encoding iconic gesture leaves a stronger memory trace in the mental representation while encoding noniconic gesture attracts attention. Yet, the conclusion drawn by Cohen and Otterbein’s study should be interpreted with some caution. It is uncertain whether the noniconic gestures in their study were truly meaningless. Unlike pantomimes, nonpantomimic gestures are difficult to be interpreted in a speech-less context. Take an example of a nonpantomimic gesture from their study: right hand with flat palm moved from shoulder to waist. Cohen and Otterbein considered it as a nonpantomimic gesture as it does not have concrete semantic content in a speech-less context. Yet, when this gesture was produced along with the sentence ‘‘he rolled down the rear window’’, it could refer to the downward


4

SO ET AL.

movement of the character. Hence, in a speech context, it is possible that listeners identify the meaning of co-speech gesture when they process it together with the accompanying speech. As a result, the nonpantomimic gestures examined in their study were possibly meaningful to listeners. It may explain why encoding nonpantomimic gesture and pantomimic gesture had a similar level of mnemonic effect. In order to investigate whether gesture needs to carry meaning to support memory recall, we should use gesture that is meaningless in a speech context. Beat gesture, a type of nonrepresentational gesture, is a good candidate. It is a mere hand movement along with the rhythm of the accompanying speech (McNeill, 1992, 2005). Beat gestures can take various forms of hand shapes and movements. One of the most typical forms of beat gesture is a hand with open palm flips outwards (McNeill, 1992). Although beat gestures do not provide a second encoding format to the listeners, researchers speculated that beat gestures might enhance memory recall (Nicoladis, 2007; Nicoladis, Mayberry, & Genesee, 1999; Ravizza, 2003) in two ways. First, it serves metacognitive function by bringing listeners’ attention to certain highlighted parts of accompanying speech (Ekman, 1999; McNeill, 1992). Thus, it aids listeners’ comprehension of speech. Second, previous finding showed that viewing beat gestures could enhance auditory processing of speech (as shown in greater activities in auditory cortex) in listeners (Hubbard, Wilson, Callan, & Dapretto, 2009). Krahmer and Swerts (2007) reported that listeners perceived the words accompanied by beat gestures more prominent than those not accompanied by beat gestures. Taking them together, encoding words with beat gestures could lead to a greater amount of processing of words and, consequently, better memory recall of information. However, Feyereisen (2006) proposed that encoding beat gesture might not enhance memory recall. In one of his experiments, he examined mnemonic effects of encoding two types of gestures*meaningful gestures and nonmeaningful gestures*in adults. Meaningful gestures involved illustrative and symbolic gestures (e.g., right index pointing downwards and drawing a circle), while nonmeaningful gestures involved beat gestures (e.g., bilateral beat gesture movement by joined thumb and index finger), and indefinite gestures2 (e.g., right fist closing and forearm rightwards stretching). The participants were presented with either the sentences accompanied by gestures (of these sentences, half of them were meaningful gestures and another half were nonmeaningful gestures) or the sentences not accompanied by gestures. Both sets of the sentences were the same. After that, they took a recall task followed by a recognition task. Both recall and 2 Indefinite gesture refers to gestures in which referents could not be identified (Hadar & Pinchas-Zamir, 2004).


GESTURE AND MEMORY

5

recognition tasks showed that the participants recalled more sentences that were accompanied by gestures than those that were not. More important, meaningful gestures yielded greater mnemonic effect than nonmeaningful gestures. As a result, Feyereisen concluded that meaningfulness of gesture could influence mnemonic effect. Yet, the conclusion drawn by Feyereisen should be interpreted with caution for two main reasons. First, he classified both indefinite and beat gesture in the same category*nonrepresentational gesture. Of all nonrepresentational gestures, 11 of them were beat gestures and the rest were indefinite gestures. However, Feyereisen did not analyse the mnemonic effect of these two types of gestures separately. Thus, it is still unclear whether encoding beat gesture could facilitate recall. Second, Feyereisen suggested that ‘‘beat gestures have the same form regardless of speech content’’ (p. 188; also see McNeill, 1992, 2005). However, all the 11 beat gestures examined in his study had different form of hand shapes and direction of movements. Thus, one might contend whether these gestures were beat gestures or other types of gestures. In order to examine the role of meaningfulness in mnemonic effect, we examined beat gesture only and picked its most typical form, i.e., a hand with open palm flips outwards (McNeill, 1992). We aimed to examine whether encoding beat gesture could enhance memory recall and whether the mnemonic effect of beat gesture is comparable to that of iconic gesture. We expected that both types of gesture could have the same mnemonic effect on memory recall in adults. However, they might improve memory through distinctive mechanisms. While iconic gesture provides a second encoding format to listeners (Paivio, 1986), beat gesture might emphasise a verbal message (Ekman, 1999; McNeill, 1992). Emphasis of the accompanying speech could heighten attention in listeners and, hence, aiding memory recall. Second, we aimed to investigate whether the mnemonic effects of iconic gestures and beat gestures could be generalised to young children. Thompson, Driscoll, and Markson (1998) examined whether encoding meaningful gesture is beneficial for recalling sentences in 9-year-old children and adults. They found that meaningful gesture facilitated memory recall in both populations. The question of interest was whether encoding iconic gesture enhances memory in younger children as well (i.e., 45-year-olds). Same as adults, children are capable of detecting meanings conveyed by iconic gestures (Kelly & Church, 1997). However, in order to benefit from encoding iconic gestures, a listener not only has to interpret those gestures but also integrate them to the co-occurring speech. Previous findings showed that the processing of multimodal information in working memory is rather limited in young children as compared to adults (Gathercole, Pickering, Ambridge, & Wearing, 2004; Hulme, 1984; Luciana & Nelson, 1998). As


6

SO ET AL.

children get older, they are more capable of incorporating multiple cues (Thompson & Massaro, 1986). Thus, we aimed to investigate any possible developmental differences in the effect of encoding iconic gesture on memory recall. Having established that encoding iconic gesture also facilitates memory recall in young children, we further asked whether encoding beat gesture would be beneficial for their memory. We expected that beat gesture might not help young children’s memory recall. Unlike adults, young children might particularly need the assistance of meaningful gestures, especially those conveying the same meaning with accompanying speech, in order to strengthen the retention of information. Goldin-Meadow (2003) suggests that gesture can help children or learners secure a message conveyed in speech when it conveys the same message. In addition, although beat gesture serves meta-cognitive function (i.e., emphasises speech and tuning listeners’ attention), McNeill (1992) suggested that gestures that reflect meta-cognitive aspects are the latest to develop in childhood. Hence, young children might not be able to make use of cues provided by the beat gestures while processing speech. Altogether, in the present study, we asked whether encoding beat gesture and iconic gesture enhances memory recall in both adults and 45-year-old children. In Experiment 1, we presented adults with three sets of words* words accompanied by iconic gestures, words accompanied by beat gestures, and words not accompanied by gestures. We found that adults recalled more words accompanied by gestures (both iconic gestures and beat gestures) than words not accompanied by gestures. In Experiment 2, we adopted a similar paradigm and asked whether viewing iconic gesture and beat gesture could enhance memory recall in young children.

EXPERIMENT 1 Method Participants Our adult participants were 30 undergraduates (15 males and 15 females) at the National University of Singapore. They participated in the study and received course credits. They were all native English speakers with normal or corrected-to-normal vision.

Stimuli There were 3 lists of verbs (see Appendix 1), 10 in each list. All verbs had one to two syllables. We videotaped a narrator who recited each list of verbs

GESTURE AND MEMORY

7

three times, once reciting the verbs and produced iconic gestures, once reciting the verbs and produced beat gestures simultaneously, and a third time reciting the verbs without gesture (the narrator put his hands down; see Figure 1 for an example of verb ‘‘stack’’). Thus, each list of verb had an equal chance to be accompanied by gestures (iconic gestures or beat gestures) and not accompanied by gestures. Each iconic gesture lasted for 3 seconds and was semantically related to the respective word. In order to ensure that all the iconic gestures were


Iconic gesture condition

Beat gesture condition

No gesture condition

Figure 1. Example of the word ‘‘Stack’’ accompanied by the gestures (iconic gesture and beat gesture) and not accompanied by the gesture.


8

SO ET AL.

meaningful, we asked a separate group of 45 English-speaking undergraduates from the National University of Singapore to interpret these gestures (see Yap et al., 2011). They were presented with 80 silent videotaped gestures, each lasting for 3 seconds on a computer screen in a speechless context. They were then given 7 seconds to write a single word that best described the meaning of a gesture. Since speech was not available, the participants derived meaning from gestures according to their physical forms and movements. We determined that a gesture had a common interpretation if its meaning was agreed upon by 70% of the participants (see also Goh, Sua´ rez, Yap, & Tan, 2009). Our findings showed that 40 of the gestures had consistency rates of 70% or above. Reliability was assessed by having a second coder analysed all the responses and calculated the consistency rate of each gesture. The inter-rater reliability was 0.946. Of the 40 meaningful gestures, we chose 30 of them that described actions of entities and paired them with respective words. The beat gesture was one of the most typical beat gestures speakers often produce when talking, i.e., a narrator flips his right hand with open palm outward once (McNeill, 1992). Like iconic gesture, each beat gesture lasted for 3 seconds.

Procedures The adults were presented with three different lists of verbs in three conditions: (1) iconic gesture, (2) beat gesture, and (3) no gesture. The order of three conditions was fully counterbalanced. They were tested individually in 3 conditions for approximately 15 minutes. In each condition, the adults watched a video on a computer screen and put on headphones to listen to audio inputs. After presentation of words, they were asked to recall as many words as possible to the experimenter. There was no restriction on the recalling time and order of words recalled. Two practice trials were given to the adults before the experiment started. The adults were holding a ball throughout the experiment in order to prevent them from reproducing gestures while watching the videos and recalling the words. And they were not allowed to reproduce the words verbally when watching the videos. After each condition, they were required to complete a math task in order to prevent retroactive interference of the words from the previous conditions.

Results For each adult participant, we counted the number of words correctly recalled and then calculated the proportion of words recalled in each condition. The proportion was calculated as the number of words recalled

GESTURE AND MEMORY

9

1 0.9 0.8 0.7 0.6 Iconic gesture Beat gesture

0.5 0.4

No gesture

0.3 0.2 0.1 0 Adults


Figure 2.

Children

Mean proportion of words recalled by adults and young children in all conditions.

correctly divided by total number of words. Figure 2 (the left panels) shows the proportion of words recalled by the adults in all conditions. The proportion of words recalled was different across the three conditions, F(2, 56) 7.87, p .001, h2 .21. There was no effect of gender, F(1, 28) 2.09, p ns, and no interaction, F(2, 56) 1.41, p ns. Bonferroni-adjusted pairwise comparisons showed that the adults recalled a higher proportion of words in the iconic gesture condition than in the no gesture condition, p .002. Interestingly, they also recalled a higher proportion of words in the beat gesture condition than in the no gesture condition, p .009. There was no difference in the proportion of words recalled between the iconic gesture and beat gesture conditions, p ns. To summarise, encoding words with iconic gestures and beat gestures enhances memory recall in adults. In Experiment 2, we look at whether encoding words with gestures facilitates memory recall in young children.

EXPERIMENT 2 Method Participants Thirty-six children, aged 45 years, participated in this study. There were an equal number of males and females in each age group. The children were native speakers of English and recruited from various childcare centres and kindergartens. Approval was obtained from the parents and principals of the centres. All the children had normal or corrected-tonormal vision.

10

SO ET AL.


Stimuli We presented 5 instead of 10 words to the children in order to accommodate their mnemonic span (Dempster, 1981). Altogether, there were three lists of verbs, with five verbs in each list (see Appendix 1). Same as Experiment 1, each list of verb had an equal chance to be accompanied by gestures (iconic gestures or beat gestures) and not accompanied by gestures. The videos presented to the children were the same as those presented to the adults. In order to ensure that all the iconic gestures were meaningful to the children, we asked a separate group of 40 children aged 45-years-old from various kindergartens and childcare centres to interpret these gestures. They were presented with 15 iconic gestures that would be tested in Experiment 2, each lasting for 3 seconds on a computer screen in a speechless context. After presentation of each gesture, they were asked to tell the meaning to the experimenter. The meaning of all gestures was agreed upon by 70% of the children. The inter-rater reliability was 0.912.

Procedures The procedures were similar to those in Experiment 1. The children were presented with three different lists of verbs in three conditions: (1) iconic gesture, (2) beat gesture, and (3) no gesture. The order of three conditions was fully counterbalanced. After presentation of each video, the children were asked to recall as many words as possible. There was no restriction in terms of the word order and amount of time taken to recall the words. It took approximately 15 minutes to finish the experiment for each child. Instead of working on a math question after each condition, the children were presented with a cartoon clip of 2 minutes after each condition to prevent interference from the previous task. Like the adults, the children were holding a stuffed toy throughout all the tasks to prevent them from reproducing the gestures and they were also told not to reproduce the words verbally. Two practice trials were given at the beginning of experiment. After the experiment, the experimenter asked the children to describe the meaning of words to ensure that they had acquired those words. All the children were familiar with the words presented on the videos.

Results The analyses were the same as in Experiment 1. For each child, we counted the number of words correctly recalled and then calculated the proportion of words recalled in each condition. Figure 2 (right panels) shows the proportion of words recalled by the children in all conditions. The proportion of words recalled was different across the conditions,

GESTURE AND MEMORY

11

F(2, 68) 20.16, p B.001, h2 .37. Bonferroni pair-wise comparisons showed that they recalled a higher proportion of words in the iconic gesture condition than in the beat gesture condition, pB.001 and in the control condition. There was no significant difference between the beat gestures and control conditions, p ns. There was a significant effect of gender, F(1, 34) 6.84, p.013, h2 .17. Male recalled a higher proportion of words than female. There was no interaction between gender and condition, F(2, 68) 1.18, p ns.


Comparison of Experiments 1 and 2 We compared the proportion of words recalled in the adults and children. Not surprisingly, the adults recalled a higher proportion of words than the children, F(1, 64) 192.69, p B.001, h2 .75. The proportion of words recalled was different across conditions, F(2, 128) 22.11, p B.001, h2 .26. There was a significant interaction between condition and group, F(2, 128) 6.91, p .001, h2 .10. All the participants recalled a higher proportion of words in the iconic gesture condition than in the control condition. However, the adults recalled a higher proportion of words in the beat gesture condition than in the control condition but the children recalled comparable proportion of words in these two conditions. In order to check if such developmental difference was due to the subset of words presented to the children, we analysed those five words separately in the adults and looked at the proportion of words they recalled in the beat gesture and control conditions. We found that adults still recalled a higher proportion of words in the beat gesture condition (M 0.79, SD0.08) than in the control condition (M0.68, SD0.07), t(29) 2.89, p B.01.

DISCUSSION Our study explored whether encoding iconic gesture and beat gesture enhanced memory recall in adults and young children. Our findings showed that the adults had a better memory recall than the children. As expected, both the adults and young children recalled more words when they encoded words with iconic gestures than when encoding words alone, suggesting that encoding iconic gestures supported memory recall. Interestingly, encoding beat gestures also enhanced memory recall but such facilitating effect was found only in the adults. The adults recalled more words when they encoded words with beat gestures than when encoding words alone. The children recalled comparable number of words when they encoded words alone and those with beat gestures.


12

SO ET AL.

The children memorised fewer words than the adults in all conditions because of their smaller mnemonic span (Cowan, Sault, Winterword, & Shrek, 1991) and constraints in working memory (Hulme, 1984; Luciana & Nelson, 1998). This is not surprising as children have a much less developed memory capacity, which allows them to hold less information as compared to adults. However, same as the adults, the children recalled more words when they encoded iconic gestures than when they did not. This finding is consistent with previous literature where meaningful gestures were shown to have a beneficial effect on memorisation in adults and older children (e.g., Tellier, 2005, 2008; Thompson, Droscoll, & Markson, 1998; Wagner, Nusbaum, & Goldin-Meadow, 2004; Woodall & Folger, 1985). Our finding extends this line of research by suggesting such beneficial effect of iconic gesture is found in 45-year-old children as well, even though they might have smaller working memory capacity to incorporate iconic gesture with cooccurring speech (Gathercole et al., 2004; Hulme, 1984; Luciana & Nelson, 1998). Further research should be done in order to examine whether encoding iconic gesture strengthens 3-year-old children’s memory. Iconic gestures convey the same meaning as the accompanying words and provide an additional format of information to both adult and children listeners. Encoding iconic gestures, thus, reinforces the processing of respective words, thereby making stronger memory traces as compared to encoding words alone (Cohen & Bean, 1983; Clark & Paivio, 1991; Nilsoon & Craik, 1990). Previous research also showed that meaningful gesture facilitates learning of second language. Quinn-Allen (1995) reported that native English speakers who were taught French learnt vocabularies better when they viewed and produced emblems than when they did not. Kelly, McDevitt, and Esch (2009) further showed that viewing iconic gestures that had congruent meaning with co-occurring words resulted in better memory than viewing iconic gestures that had incongruent meaning. In fact, gestures that are not semantically meaningful could also enhance memory. Our findings showed that encoding words with beat gestures facilitates recall. However, the mnemonic effect of beat gesture was found in the adults (but not the children). While the mechanism of such effect is not yet documented, we suggest beat gesture could aid memory in two ways. First, beat gesture serves meta-cognitive function by marking the word or concept the speaker would like to emphasise (Feyereisen, van de Wiele, & Dubois, 1988; McNeill, 1992). Conceptual salience, thus, leads to improvement of memory recall (Hunt & Lamb, 2001; Hunt & McDaniel, 1993; Schmidt, 1991). Another possibility is beat gesture changes acoustic properties of speech by modulating activity of auditory cortex during speech comprehension. Hubbard et al. (2009) found that encoding speech with beat


GESTURE AND MEMORY

13

gestures resulted in a significant increase in activity in planum temporale, an area that processes meaningful prosodic inputs (Callan et al., 2006; Meyer, Steinhauer, Alter, Friederici, & von Cramon, 2004; Saito, Ishii, Yagi, Tatsumi, Mizusawa, 2006). Krahmer and Swerts (2007) also found that viewing beat gesture influences perception of speech prosody and increases perceived prominence of the accompanying word. More importantly, increase in the processing of speech prosody might enhance intelligibility and comprehension of speech (Humphries, Love, Swinney, & Hickok, 2005) that, in turn, aids memory. In order to explore these two possibilities in future research, we should present listeners regular and continuous spoken dialogue where some words are accompanied by beats and some do not and examine whether listeners remember words with beats than those without. For example, a listener is presented with a video in which a speaker flips his right hand outward while uttering ‘‘the world’’ in this sentence, ‘‘Today, the world is changing rapidly’’. We expect the listener to remember the words ‘‘the world’’ better than the other words as they are more conceptually and perceptually salient. Then why was the mnemonic effect of beat gestures not found in the children? More research is needed to find an adequate explanation for this. One possibility is the children in our study were not sensitive to the beat gestures that reflect meta-cognitive aspects. Same as beat gestures, pitch accents, head nods, and eyebrow movements could signal that a word is important (Krahmer & Swerts, 2007). Previous findings showed that young children do not interpret subtle prosodic cues as effectively as adults (Ackerman, 1983; Vogel & Raimy, 2002). Krahmer and Swerts (2005) also reported that 78-year-old children made less effective use of audiovisual cues such as eyebrow, facial expression, and intonation to form metacognitive judgments about certainty and uncertainty than adults. Thus, the young children in our study might not be able to make use of the beat gestures to perceive prominence of the words. In fact, McNeill (1992) suggested that meta-cognitive functions of gestures (e.g., to emphasise speech) do not develop until the age of seven. However, further research should investigate whether viewing beats changes prosodic processing in young children. To summarise, encoding gesture in general enhances memory recall in both adults and children. Both iconic gestures and beat gestures could support recall in adults, although the underlying mechanisms might be different from each other. On the other hand, iconic gestures but not beat gestures improve recall in young children. Our research is the first study showing that encoding beat gestures with words enhances recall of words.

14

SO ET AL.

Further research should extend this finding by looking at whether encoding beat gestures in natural discourse enhance retrieval of information. Manuscript received 10 July 2010 Revised manuscript received 13 March 2011 First published online XXX


REFERENCES Ackerman, B. (1983). Form and function in children’s understanding of ironic utterances. Journal of Experimental Child Psychology, 35, 487508. Alibali, M. W., Flevares, L. M., & Goldin-Meadow, S. (1997). Assessing knowledge conveyed in gesture: Do teachers have the upper hand? Journal of Educational Psychology, 89, 183193. Beattie, G., & Shovelton, H. (1999). Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language & Social Psychology, 18, 438462. Beattie, G., & Shovelton, H. (2002). An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. British Journal of Psychology, 93, 179 192. Bernadis, P., Salillas, E., & Caramelli, N. (2008). Behavioral and neurophysiological evidence of semantic interaction between iconic gestures and words. Cognitive Neuropsychology, 25, 1114 1128. Callan, D. E., Tsytsarev, V., Hanakawa, T., Callan, A. M., Katsuhara, M., Fukuyama, H., et al. (2006). Song and speech: Brain regions involved with perception and covert production. Neuroimage, 31, 13271342. Cassell, J., McNeill, D., & McCullough, K.-E. (1999). Speech gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics & Cognition, 7, 134. Clark, H. H. (1996). Using language. Cambridge, UK: Cambridge University Press. Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3(3), 149210. Cohen, R. L. (1981). On the generality of some memory laws. Scandinavian Journal of Psychology, 22, 267282. Cohen, R. L. (1989). Memory for action events: The power of enactment. Educational Psychological Review, 1, 5780. Cohen, R. L., & Bean, G. (1983). Memory in educable mentally retarded adults: Deficits in subject or experimenter? Intelligence, 7, 287298. Cohen, R. L., & Otterbein, N. (1992). The mnemonic effect of speech gestures: Iconic and noniconic gestures compared. European Journal of Cognitive Psychology, 4(2), 113139. Cohen, R. L., & Stewart, M. (1982). How to avoid developmental effects in free recall. Scandinavian Journal of Psychology, 23, 916. Cook, S. W., Mitchell, Z., & Goldin-Meadow, S. (2008). Gesturing makes learning last. Cognition, 106(2), 10471058. Cowan, N. J., Sault, S., Winterword, C., & Shrek, M. (1991). Enhancement of 4-year-old children’s memory span for phonologically similar and dissimilar word lists. Journal of Experimental Child Psychology, 51, 3052. Dempster, F. N. (1981). Memory span: Sources of individual and developmental differences. Psychological Bulletin, 89, 63100.


GESTURE AND MEMORY

15

Ekman, P. (1999). Emotional and conversational nonverbal signals. In L. S. Messing & R. Campbell (Eds.), Gesture, speech, and sign (pp. 4555). Oxford, UK: Oxford University Press. Feyereisen, P. (2006). Further investigations on the mnemonic effect of gestures: Their meaning matters. European Journal of Cognitive Psychology, 18, 185205. Feyereisen, P., van de Wiele, M., & Dubois, F. (1988). The meaning of gestures: What can be understood without speech. European Bulletin of Cognitive Psychology, 8, 325. Gathercole, S. E., Pickering, S. J., Ambridge, B., & Wearing, H. (2004). The structure of working memory from 4 to 15 years of age. Developmental Psychology, 40, 177190. Goh, W. D., Sua´ rez, L., Yap, M. J., & Tan, S. H. (2009). Distributional analyses in auditory lexical decision: Neighbourhood density and word frequency effects. Psychonomic Bulletin & Review, 16, 882887. Goldin-Meadow, S. (2003). Hearing gesture. Cambridge, MA: Belknap Press of Harvard University Press. Goldin-Meadow, S., Kim, S., & Singer, M. (1999). What the teacher’s hands tell the student’s mind about math? Journal of Educational Psychology, 91, 720730. Goldin-Meadow, S., & Sandhofer, C. M. (1999). Gesture conveys substantive information about a child’s thoughts to ordinary listeners. Developmental Science, 2, 6774. Hadar, U., & Pinchas-Zamir, L. (2004). The semantic specificity of gesture. Journal of Language and Social Psychology, 23, 204214. Holle, H., & Gunter, T. C. (2007). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience, 19, 11751192. Hubbard, A. L., Wilson, S., Callan, D. E., & Dapretto, M. (2009). Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception. Journal of Human Brain Mapping, 30, 10281037. Hulme, C. (1984). Developmental differences in the effects of acoustic similarity on memory span. Developmental Psychology, 4, 650652. Humphries, C., Love, T., Swinney, D., & Hickok, G. (2005). Response of anterior temporal cortex to syntactic and prosodic manipulations during sentence processing. Human Brain Mapping, 26, 128138. Hunt, R. R., & Lamb, C. A. (2001). What causes the isolation effect? Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 13591366. Hunt, R. R., & McDaniel, M. A. (1993). The enigma of organization and distinctiveness. Journal of Memory and Language, 32, 421445. Kelly, S. D., Barr, D. J., Church, R. B., & Lynch, K. (1999). Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language, 40, 577592. Kelly, S. D., & Church, R. B. (1997). Can children detect conceptual information conveyed through other children’s nonverbal behaviors? Cognition and Instruction, 15, 107134. Kelly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89, 253260. Kelly, S. D., Manning, S., & Rodak, S. (2008). Gesture gives a hand to language and learning: Perspectives from cognitive neuroscience, developmental psychology and education. Language and Linguistics Compass, 2, 120. Kelly, S. D., McDevitt, T., & Esch, M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and Cognitive Processes, 24(2), 313334. Kelly, S. D., Ozyurek, A., & Maris, E. (2010). Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science, 21(2), 260267. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge, UK: Cambridge University Press.


16

SO ET AL.

Kita, S., & Ozyurek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 1632. Krahmer, E., & Swerts, M. (2005). How children and adults produce and perceive uncertainty in audiovisual speech. Language and Speech, 48(1), 2953. Krahmer, E., & Swerts, M. (2007). Effects of visual beat gestures on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 57, 396 414. Luciana, M., & Nelson, C. A. (1998). The functional emergence of prefrontally-guided working memory systems in four-to-eight year-old children. Neuropsychologia, 36, 273293. McNeill, D. (1992). Images, inside and out. In Hand and mind: What gestures reveal about thought (pp. 1135). Chicago, IL: University of Chicago Press. McNeill, D. (2005). Gesture and thought. Chicago, IL: University of Chicago Press. Meyer, M., Steinhauer, K., Alter, K., Friederici, A. D., & von Cramon, D. Y. (2004). Brain activity varies with modulation of dynamic pitch variance in sentence melody. Brain Language, 89, 277 289. Nicoladis, E. (2007). The effect of bilingualism on the use of manual gestures. Applied Psycholinguistics, 28, 441454. Nicoladis, E., Mayberry, R., & Genesee, F. (1999). Gesture and early bilingual development. Developmental Psychology, 35, 514526. Nilsoon, L. G., & Craik, F. I. M. (1990). Addictive and interactive effects in memory for subjectperformed tasks. European Journal of Cognitive Psychology, 2, 305324. Ozyurek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19, 605616. Paivio, A. (1986). Mental representations: A dual coding approach. Oxford, UK: Oxford University Press. Quinn-Allen, L. (1995). The effects of emblematic gestures on the development and access of mental representations of French expressions. The Modern Language Journal, 79, 521529. Ravizza, S. (2003). Movement and lexical access: Do noniconic gestures aid in retrieval? Psychonomic Bulletin & Review, 10(3), 610615. Riseborough, M. G. (1981). Physiolographic gestures as decoding facilitators: Three experiments exploring a neglected facet of communication. Journal of Nonverbal Behavior, 5, 172183. Rogers, W. T. (1978). The contribution of kinesic illustrators toward the comprehension of verbal behavior within utterances. Human Communication Research, 5, 5462. Saito, Y., Ishii, K., Yagi, K., Tatsumi, I. F., & Mizusawa, H. (2006). Cerebral networks for spontaneous and synchronized singing and speaking. Neuroreport, 17, 18931897. Schmidt, S. R. (1991). Can we have a distinctive theory of memory? Memory and Cognition, 19, 523542. So, W. C., & Lim, J. Y. (in press). Point to an entity and say ‘‘What is it?’’ Gesture as a potential cue to identify a referent. Applied Psycholinguistics. Tellier, M. (2005, January). L’utilisation des gestes en classe de langue: comment e´ valuer leur effet sur la me´ morisation du lexique? In M. Billie`res, P. Gaillard, & N. Spanghero-Gaillard (Eds.) Actes du Colloque International de Didactique Cognitive, DidCog 2005. Proceedings on CD-Rom, Toulouse. Tellier, M. (2008). The effect of gestures on second language memorization by young children. Gesture, 8(2), 219235. Thompson, L. A. (1995). Encoding and memory for visible speech and gestures: A comparison between young and older adults. Psychology and Aging, 10, 215228. Thompson, L. A., Driscoll, D., & Markson, L. (1998). Memory for visual-spoken language in children and adults. Journal of Nonverbal Behavior, 22, 167187.

GESTURE AND MEMORY

17

Thompson, L. A., & Massaro, D. W. (1986). Evaluation and integration of speech and pointing gestures during referential understanding. Journal of Experimental Child Psychology, 42, 144 168. Valenzeno, L., Alibali, M., & Klatzky, R. (2003). Teacher’s gestures facilitate students’ learning: A lesson in symmetry. Contemporary Educational Psychology, 28, 187204. Vogel, I., & Raimy, E. (2002). The acquisition of compound vs. phrasal stress: The role of prosodic constituents. Journal of Child Language, 29, 225250. Wagner, S., Nusbaum, H., & Goldin-Meadow, S. (2004). Probing the mental representation of gesture: Is handwaving spatial? Journal of Memory and Language, 50, 395407. Woodall, W. G., & Folger, J. P. (1985). Nonverbal cue context and episodic memory: On the availability and endurance of nonverbal behaviors as retrieval cues. Communication Monographs, 52, 319333. Yap, D. F., So, W. C., Yap, M., Tan, Y. Q., & Teoh, S. (2011). Iconic gestures prime words. Cognitive Science, 35, 171183.


APPENDIX 1 Word lists presented to adults and children Adults

Children

List 1

List 2

List 3

List 1

List 2

List 3

Come Think Fly Comb Stir Pray Stack Beg Hammer Cry

Write Strum Cut Walk Swim Throw Turn Eat Open Push

Cycle Run Read Carry Inject Climb Bounce Brush Knock Listen

Read Brush Throw Come Cycle

Cut Think Clap Comb Walk

Look Swim Pray Fly Strum