perceptual simulation of space, speed and time in language

2 PERCEPTUAL SIMULATION OF SPACE, SPEED AND TIME IN LANGUAGE Laura J. Speed, Gabriella Vigliocco

Introduction How do we map words and sentences to their corresponding concepts in order to comprehend and communicate effectively? Embodied theories of language propose that understanding meaning in language requires mental simulation in the brain’s modality-specific systems, the same systems involved in perceiving and acting in the world (e.g. Barsalou, 1999; Stanfield & Zwaan, 2001; Glenberg & Kaschak, 2002). This mental simulation grounds language in human experience and moves away from a perspective in which linguistic meaning is solely symbolic and abstract (e.g. Landauer & Dumais, 1997; Lund & Burgess, 1996). Embodied theorists argue that amodal theories of meaning are missing the vital link between meaning in language and experience in the world. In others words, it is unclear how meaning is understood if language is composed of only arbitrary symbols with no link to referents in the world (Harnad, 1990). Instead of transducing experiential information into abstract symbols, it is thought that the experience itself is, in a way, recreated in the brain’s sensory and motor systems (Barsalou, 1999a). In this chapter we review experimental evidence for mental simulation in the brain’s perceptual systems for three types of language comprehension: language that describes space, speed of motion and time. There is now evidence for the mental simulation of many fairly concrete dimensions of experience during language comprehension, such as what an object looks like (e.g. Stanfield & Zwaan, 2001, Zwaan, Stanfield & Yaxley, 2002, Zwaan, Madden, Yaxley & Aveyard, 2004). For example readers represent the fact that an eagle would be viewed with its wings outstretched if it were flying in the sky compared to sitting in its nest (Zwaan et al., 2002): responses to a picture of a flying eagle were faster after reading the sentence ‘The ranger saw the eagle in the sky’ compared to ‘The ranger saw the eagle in the nest.’ However there also exist less tangible dimensions that afford little physical interaction, or are more dynamic, yet

6241-1269-PI-002.indd 29

10/3/2015 4:38:15 PM

30

Laura J. Speed, Gabriella Vigliocco

are still critical properties of perceptual events. Prime examples of such domains are space, speed and time and here we discuss whether and to what extent these features can be embodied during language comprehension. Is it the case, for example that to understand a sentence such as ‘The suitcase was sat next to the wardrobe,’ we construct a spatial simulation involving processes used to perceive a real world spatial configuration of a suitcase next to a wardrobe? We consider space, speed and time as more abstract than perceptual qualities such as ‘red’ or ‘round’, because we cannot really interact with them or directly perceive them. Still, they define properties of events taking place outside the human mind and therefore are clearly distinct from other abstract domains such as emotion or mental states, which characterise internal states/events. Space, speed and time themselves differ in abstractness. Time is by far the most abstract, as it cannot be directly experienced through the senses. Klein (2009) sums up Newton’s perception of the elusiveness of time: ‘Real time is, so to speak, unaffected and unaffectable by anything. In fact, it is not even related to anything “external”; in particular, it is not related to any observer’ (p.6). Speed is somewhere in the middle of the three domains, being composed of both space and time. Space is the most concrete, and although it may not exist as a separate concrete entity, we can point to it in the world and draw it on a map. However, it could be argued that we do not directly perceive space, speed or time, but instead infer them via other referents. We experience time by roughly inferring its passing from events that have occurred, or it can be measured relatively using the motion of a clock, but it does not exist without such indicators. Speed needs the motion of an agent or object and space needs land, buildings or people to be defined. Despite their relative abstractness, these domains are integral to our everyday lives and therefore important in communication. Time and space in fact make up the venue of all human experience: everything happens in some place at some time. The ‘past’, ‘present’ and ‘future’ are not directly observable but are central to our experience (Nunez & Cooperrider, 2013). For example our entire day is organized around time: we decide what time we get out of bed, we track how long the work commute is and we know when our first meeting is and how long it will last. Time is always on our mind and thus always present in our topics of conversation. Space is similarly crucial: we need to remember where we placed the car keys, how far it is to walk to a desired location or from what platform our train leaves. Moreover, space is often described in language to direct other people to locations and objects and thus is very often mentioned in discourse (e.g. ‘Pass me the book on the third shelf next to the bookend ’). Speed may be less salient throughout our interactions than space and time but still important, especially when living in such a fast-paced world where time is precious: we may get frustrated because our train is moving into the station slowly, if we are late we may need to run to work quickly and we often monitor the speed of vehicles in order to stay safe, such as when crossing the road. Since space and time are crucial dimensions of our experience, they are likely to be important dimensions of mental simulations during comprehension. Speed may

6241-1269-PI-002.indd 30

10/3/2015 4:38:16 PM

Perceptual simulation

31

be less crucial (i.e. we’re not always monitoring the speed of our own movement or others’), but still can be a salient aspect of events and thus is also likely to be simulated. In this chapter we will describe how each of the three dimensions are described and represented in language, and provide evidence that comprehension of these dimensions in language recruits the perceptual systems. Here we review evidence for an embodied account of the comprehension of language about speed, space and time using work from our own laboratory as well as the work of others. We focus on a number of dominant experimental paradigms within the embodied literature. For example, early work investigated reading times to narratives varying on dimensions such as space and time (e.g. Zwaan, 1996) finding that reading times to objects or events further away in space and time were longer than those closer. Another very common way to test whether language comprehension recruits the perceptual systems during comprehension has been to combine perceptual stimuli or a perceptual task with the presentation of words or sentences. The rationale is that if both the perception of the stimuli and language comprehension recruit similar systems, then their combination should affect performance in one domain. This experimental paradigm has been utilized many times to demonstrate mental simulation of a number of aspects of meaning (e.g. direction of motion, Meteyard, Zokaei, Bahrami & Vigliocco, 2008). Other experimental approaches include the ‘visual word paradigm’ in which participants comprehend language whilst viewing corresponding visual scenes with their eye movements being recorded. Eye movements around a visual scene during language comprehension can indicate mental simulation (such as spatial or motion simulations) in a manner more natural than one that requires explicit linguistic judgments from a participant. Below we present a summary of research evidence for the embodiment of space, speed and time, discussing how the perceptual systems are used in mental simulation during comprehension.

Space, speed, and time in language Space Space in language Within languages there are three main types of spatial reference systems used to describe objects in space (Jansen, Haun & Levinson, 2012): (1) intrinsic/objectoriented frames, in which locations are described in terms of a referent object, e.g. at the back of the house, (2) relative/egocentric frames, in which locations are described from the observers point of view, e.g. front, back, left, right, and absolute frames, in which locations are described in terms of fixed arbitrary points, e.g. North, South, East and West, and (3) egocentric reference frames, which provide a clear example of how space can be embodied because objects are described in terms of the body and its position and motion in space (Wilson & Foglia, 2011).

6241-1269-PI-002.indd 31

AuQ27

10/3/2015 4:38:16 PM

32

Laura J. Speed, Gabriella Vigliocco

This could lead to the prediction that comprehension of egocentric spatial descriptions involves simulations with the body more so than the other reference frames. This is in line with the distinction that Myachykov, Scheepers, Fischer and Kessler (2014) make between representations constrained by the physical world (tropic features) and those constrained by the body (embodied features). Simulations of object-centred descriptions, may instead rely more on perceptual simulations of the objects described (i.e. unrelated to the body of the comprehender). Different languages vary in terms of the frame of reference predominantly used and the one that is adopted in a particular language has been shown to affect the way the environment is encoded in memory and reasoned about (Levinson, Kita, Haun & Rasch, 2002; Majid, Bowerman, Kita, Haun & Levinson, 2004). These differences are reflected in speakers’ navigational choices. Thus, for example in Levinson (1997), speakers of a relative frame language (Dutch) and speakers of an absolute frame language (Tzeltal) viewed an object being moved along a path. The participants were then rotated 180 degrees and asked to choose the path they had just viewed within a maze. Speakers of Tzeltal chose the path that preserved the absolute frame, but speakers of Dutch chose the path that preserved the relative frame. In describing environments, either route or survey descriptions can be used. In route descriptions the environment is described in terms of an observer’s perspective, thus the description of objects and relations change dynamically with the observer’s movement. Survey descriptions instead take a bird’s eye perspective, from above the environment, with spatial landmarks fixed. Thus, comprehending route descriptions is more likely to involve mental simulation involving the body than comprehending survey descriptions. This was suggested in Brunye, Mahoney and Taylor (2010) who found that listening to fast footsteps sped up reading time compared to slow footsteps for route descriptions but not survey descriptions. Describing space in sign languages is particularly interesting because here (signing) space is used to describe (physical) space (Emmorey, Tversky & Taylor, 2000). In all sign languages, space can be used in a ‘topographic’ manner to map the position of objects and people in real world space. In these constructions, often reference to entities is achieved via the use of ‘classifiers’. Classifiers encode physical attributes and spatial properties of their real world referents, such as shape, orientation and movement. But signing space can also be used in a way that bears less of a relation to real positions or orientations in physical space. For example in a sentence meaning ‘He gave her the book,’ the verb give moves from a location in signing space assigned to he towards a location assigned to her. Here, space is used in a ‘referential’ manner, without reference to actual spatial configurations. If we mentally simulate the described referents and events during comprehension, then producing signs in a manner consistent with these simulations will lead to greater overlap between the language comprehension system and the perceptual system.

AuQ28

Simulating space When comprehending discourse, readers construct a situation model, or mental representation, of the events being described, which has been shown to contain

6241-1269-PI-002.indd 32

10/3/2015 4:38:16 PM

Perceptual simulation

33

spatial information. When reading narratives, readers imagine themselves within the story by adopting the perspective of the described protagonist (Avraamides, 2008; Zwaan, 1996). Objects that are described as being near to a protagonist can be accessed more easily by the reader than objects that are described as far from the protagonist (Glenberg, Meyer & Lindem, 1987; Morrow, Greenspan & Bower, 1987). For example responses to a target noun were faster when the object had been described as spatially associated with a protagonist compared to when it was spatially dissociated (he put the last flower in his buttonhole vs. he put the last flower in the vase). The representation of spatial location during narrative comprehension is thought to be automatic and not reliant on specific tasks or the salience of space to the coherence of the narrative storyline (Levine & Klin, 2001). Moreover, it has been suggested that spatial information has a ‘privileged status’ (Levine & Klin, 2001, p. 333) in memory, being accessible even several sentences after its mention. Spatial information is likely an integral component of an event representation: it combines salient objects appropriately and maintains overall discourse coherence. The perceptual simulations built from spatial descriptions are thought to reflect the typical relations between objects (Coventry, Lynott, Cangelosi, Monrouxe, Joyce & Richardson, 2010). It is now well accepted that language drives attention to locations in the world (e.g. Altmann & Kamide, 2004). This supports the proposal that meaning in language is grounded in our spatial representations. It has been shown that spatial representations are activated during single word comprehension (see Kaup Chapter x in this volume for further discussion). For example Dudschig, Lachmair, de la Vega, De Filippis and Kaup (2012) centrally presented participants with nouns that did not explicitly convey spatial information in their meaning but whose referents are typically found to be high or low in the environment (e.g. cloud vs. shoe). Four seconds after presentation of a word, a visual target (a filled white box) was presented above or below a central fixation point and participants had to detect its presence. Target detection was significantly faster when the location of the target matched the location of the referent’s typical location (e.g. target presented above fixation after the word cloud ). Thus, although the word is irrelevant to the task, its spatial meaning affects attention on the vertical axis. Here the spatial features of the words’ referents facilitated target detection, but elsewhere (Estes, Verges & Barsalou, 2008) they have hindered target identification. Estes, Verges and Barsalou (2008) also presented nouns denoting objects with typical locations in the centre of the screen followed by a target above or below fixation, but instead of simply detecting the target, participants were required to identify whether it was the letter X or O. Now performance on the task was worse when the referent of a word’s typical location matched the location of the target. The words oriented spatial attention, as before, but now the perceptual simulations generated for the words’ referents interfered with the identification of the target letter, because the simulated object features did not match the target objects. Thus activation of spatial features from single words may assist or hinder a subsequent perceptual task depending upon whether the task requires detection or identification. Conversely, manipulating whether spatial attention is directed to the upper or lower region of space can affect how a word with spatial features is processed.

6241-1269-PI-002.indd 33

10/3/2015 4:38:16 PM

34

Laura J. Speed, Gabriella Vigliocco

Zwaan and Yaxley (2003) found that judgments about word pairs were faster when they were presented in a spatially congruent configuration (e.g. attic above basement) compared to a spatially incongruent configuration (e.g. basement above attic). Features of motion can also be part of the perceptual simulations occurring during comprehension of spatial language. It has been shown that when mapping a spatial expression to a visual scene, looks to the visual scene reflect motion characteristics of the described spatial configuration. For example when viewing an image of a cereal box over a breakfast bowl and hearing a sentence such as ‘The box is above the bowl,’ participants spend more time looking at an area of the scene consistent with the direction in which the cereal will fall from the box (Coventry et al., 2010). Thus a motion simulation developed out of a spatial simulation. Evidence from sign language suggests that comprehending spatial language recruits similar systems to those used in real spatial cognition. The use of (signing) space to describe space in a topographic manner can facilitate communication when compared with spatial descriptions in spoken languages. For example in one study comparing spatial descriptions in American Sign Language (ASL) and English, it was found that ASL signers were significantly faster at describing a spatial environment than English speakers (Emmorey et al., 2000). It has also been shown that processing of topographic sentences in British Sign Language (BSL) recruits areas of the brain, such as posterior middle temporal cortices bilaterally and left inferior and superior parietal lobules to a greater extent than non-topographic sentences (for a discussion see MacSweeney et al., 2008) which suggests that processing these sentences in sign language is carried out by the same systems that process visuo-spatial information in non-linguistic stimuli. Thus, mental simulations of space underlie comprehension of spatial words, sentences and longer discourse, and these simulations adhere to spatial properties existent in the world.

AuQ29

AuQ30

Speed Speed in language Although likely to be used less often in directing attention and guiding the behaviour of conversation partners than space, speed of motion is important in characterising events. For example moving at a fast or slow speed is an evolutionarily important factor in terms of survival: one may need to move quickly to escape a predator or catch prey, or conversely move slowly to remain hidden from a predator or prey. Thus speed is used in language in a number of ways. In English, speed can be encoded within motion verbs: compare run versus walk, or less frequent verbs such as amble and dash. Speed can also be inferred from other types of action verbs, for example verbs describing actions with the hand: compare smack with stroke. Action verbs can also be modified by adverbs to make them appear fast or slow e.g. The man quickly/slowly went to the shop or The man briskly/sluggishly went to the shop. While there are some words (such as the adverbs

6241-1269-PI-002.indd 34

10/3/2015 4:38:16 PM

Perceptual simulation

35

quick, fast, slow, quickly and slowly) whose meaning appears to be solely related to speed, most words that indicate the speed of an object/event, also reflect other semantic features of the event, such as other types of manner of motion or the mood of the agent in motion (i.e. sluggishly also implies that the agent is perhaps tired or miserable). Thus, speed tends to be linguistically encoded as just one among multiple features of events, of which the more general meaning is a motion or action taking place, such as an agent going to a particular destination (e.g. John dashed to the building). That this event took place at a specific speed does not alter the fundamental meaning. Speed is a more fine-grained description of the motion event and thus may only be necessary in specific situations (such as describing a car crash or running late). Talmy (1975) makes a distinction between manner of motion and path of motion as encoded in languages. Speed describes manner of motion. Languages differ in terms of how they describe both manner and path of motion. For example in English (and other languages such as Chinese), manner of motion is encoded in the main verb (e.g. run) and path of motion is described using satellites (e.g. to the shop). However, other languages instead encode path of motion in the main verb (e.g. Greek vgeno, ‘exit’) and use a manner modifier (e.g. Greek trehontas, ‘running’) (Skovods & Papafragou, 2010). The different ways that speed is encoded crosslinguistically (i.e. in the main verb or in a manner modifier) as well as within a language (e.g. speed verbs vs. adverbs) could have implications for how the meaning of speed is retrieved and simulated (for example at what point in sentence comprehension speed simulation occurs).

Simulating speed Due to the fact that speed is not the only aspect of meaning encoded in speed words and that the more general meaning can still be comprehended without speed information, it is debatable whether or not, and under what circumstances, speed would be simulated. Since, like real-world perception, embodied simulations are constrained by factors such as attentional capacity, it is unclear how much information is contained in them and how schematic they are (Sanford, 2008). As with space, speed of motion is also represented in the simulations built during narrative comprehension. Fecica and O’Neill (2010) investigated the mental simulation of speed in short narratives with children. The children listened to narratives describing the journey of a young boy to his aunt’s house one sentence at a time by pressing a mouse button. Critically, the duration of the journey was manipulated by introducing the character as taking his journey either by walking or by car. Processing time of sentences describing scenery on the journey was measured (mouse button response) and children were found to take significantly longer understanding the sentences when the character was described as walking, compared to driving. Additionally, when a psychological factor of the character was manipulated, processing times were found to be longer when they were described as being less eager to take the journey (e.g. going to the dentist as

6241-1269-PI-002.indd 35

10/3/2015 4:38:16 PM

36

Laura J. Speed, Gabriella Vigliocco

opposed to buying ice cream). Thus children (and comprehenders in general) are able to use expectations about the duration of events based on information regarding the method of transport or the character’s motivation to influence the speed of mental simulation of an event. A more subtle measurement of the online mental simulation of speed has been demonstrated using eye-tracking (Speed & Vigliocco, 2014; Lindsay, Scheepers & Kamide, 2013). We presented participants with spoken sentences describing fast and slow motion of agent to an object (e.g. The lion ambled/dashed to the balloon), whilst they viewed a visual scene containing an agent and a target destination. In one condition, the visual scene also contained a distractor destination. When the scene was unambiguous (i.e. only one possible target destination), the total time spent looking towards the target destination was longer for sentences describing slow motion, compared to sentences describing fast motion. Thus, listeners simulated the meaning of the sentences in a way that was consistent with the kinematics of real-world motion; slow events take longer to unfold in the real world so the simulation was slower and hence the target destination was looked at for longer. The advantage of this method is that the measure of simulation (dwell time on target objects) could be taken whilst participants simply listened to sentences, instead of using a measure from an artificial response to the sentence that is less natural (e.g. Does this sentence make sense?). Based on the rationale that if language recruits perception then their combination should affect processing, we tested the mental simulation of speed in comprehension of both single speed verbs and sentences describing speeded actions (Speed, 2014). In one set of studies, participants completed a lexical decision task (deciding whether a presented item was a real word or not) after perceiving a perceptual speed stimulus for three seconds presented either visually (a set of lines in perspective moving quickly or slowly towards the edge of the screen, see Figure 2.1.) or auditorily (the sound of fast and slow footsteps). Results showed that response time to words was different when the speed of perceptual stimuli matched the speed of the word compared to when they did not match. Interestingly, this effect was only found when both the words and the perceptual stimuli were presented in the same modality (i.e. spoken words and the sound of footsteps, visual words and visual footsteps), which could reflect an effect of selective attention. In another study we presented the sound of fast and slow footsteps at the same time as participants read sentences describing fast and slow actions performed with the whole body (e.g. Daniel rambled through the forest) and fast and slow actions performed with the hands (e.g. The man yanked the door open). Results showed that accuracy differed when sentence speed and footstep speed matched compared to when they did not match (i.e. an interaction), for both sentence types. However, the nature of this interaction differed between sentences describing actions with the whole body and sentences describing actions with the hands. For whole body sentences accuracy was lower when the speed of footsteps sounds matched the speed described in the sentence, reflecting an interference effect. Conversely, for hand action sentences accuracy was higher when the speed

6241-1269-PI-002.indd 36

10/3/2015 4:38:16 PM

Perceptual simulation

37

word

Example of moving visual speed stimulus used in lexical decision task with speed verbs.

FIGURE 2.1

of footsteps sounds matched the speed described in the sentences, reflecting a facilitation effect. This suggests that interference between sentence speed and footstep speed occurs when there is a match in terms of both speed as well as action type (full-body actions produce the sound of footsteps in the real world), i.e. when there is large overlap between the sentence meaning and the perceptual stimulus. On the other hand, facilitation occurs when there is only partial overlap with sentence and footsteps (i.e. only a match in speed and not action specifics). This partial overlap of simulation and perception provides a head start in processing the sentence. Following evidence that perceptual speed affects comprehension of speed words, we tested the bidirectional relationship (that speed words conversely affect speed perception) using a speed discrimination task (Speed, Bruno & Vigliocco (submitted)). We investigated whether listening to words that describe speed of motion (e.g. dash, amble) affects performance on a speed discrimination task. In each run of the experiment, participants were presented with circles containing sinusoidal gratings. The circles remained static but the gratings moved at a fixed speed (3, 5 or 8Hz). On each trial, participants had to decide whether each grating was moving faster or slower than a standard grating that they had been exposed to. During the task, participants passively listened to spoken fast and slow verbs. This speed discrimination task provides two dependent measures: speed discrimination threshold (the smallest difference in speed that a participant can reliably detect) and point of subjective equality (the perceived speed of the standard grating). This method allowed us to assess the effect of speed words on perceptual sensitivity (speed discrimination threshold) and decision criteria (perceived speed) (Morgan, Dillenburger, Raphael & Solomon, 2012). Point of subjective equality was found to be lower after listening to fast verbs compared to slow verbs. That is,

6241-1269-PI-002.indd 37

10/3/2015 4:38:16 PM

38

Laura J. Speed, Gabriella Vigliocco

the standard grating was perceived as moving more slowly when listening to fast words compared to slow words. However, this effect was found only at 3Hz, when speed discrimination was the most difficult. No effect of word speed was found in measures of the speed discrimination threshold. This suggests that interactions between semantic and perceptual processes occurred at stages of perceptual bias, not at levels of sensitivity, and only when perception was difficult. That the effect was only observed during a condition that was particularly difficult is in line with perception research showing that perceptual processes are more susceptible to top-down influence the more ambiguous a signal is (e.g. Ma, Zhou, Ross, Foxe & Parra, 2009). Simulation of speed therefore underlies comprehension of words and sentences that describe speed of motion. These simulations have been observed using fairly naturalistic language comprehension situations, suggesting they are not a result of explicit mental imagery, however, they do not seem to engage low-level perception mechanisms. These speed simulations appear to contain specific details about the agent involved in the motion event and are bidirectional, with speed in language affecting speed perception and vice versa.

Fa

500ms

“dash...run...”

ste

ro

rs

Trial 1

low

er

350ms

th

an

sta

nd

ar

500ms Comparison: variable speed

Trial 2

d?

350ms

Comparison: variable speed

Trial 35

350ms

Comparison: variable speed

Example of one run of speed discrimination task (Speed et al., (submitted)). After observing a grating moving at a standard speed, participants are presented with subsequent gratings and have to decide whether they are moving faster or slower than the standard. At the same time, participants listen to spoken speed verbs.

FIGURE 2.2

6241-1269-PI-002.indd 38

10/3/2015 4:38:16 PM

Perceptual simulation

39

Time Time in language Time is extremely important to our everyday actions, including mental actions, and therefore it is central to cognition. Because of its importance, it is a necessary component of language, being automatically encoded in tense morphemes on verbs that place all descriptions of events in the past, present or future (Zwaan, Madden & Stanfield, 2001) as well as being reflected in the structure of language itself, with the grammatical ordering of events in a sentence often mirroring the order of events in reality (Haiman, 1980). Similar to how language can provide us with spatial representations, language also supports ‘time travel’, allowing us to move from present to past and future. In language we can describe the past, the present and the future as well as other temporal details such as event duration or sequences. Zwaan et al., (2001) describe the representation of time in comprehension as the comprehender’s attempt to form a ‘flow of events comparable to normal perceptual experience’ (p.2) from language. In comparison to speed and space, comprehension of time is more difficult to explain in terms of perception with the senses. So how can it be embodied in our perceptual systems during language comprehension? One way is through metaphor (Lakoff & Johnson, 1980; Boroditsky, 2000). Time can be grounded in the perception of space and motion. These metaphors are extremely prevalent in our everyday talk of time. Examples of time described in metaphors of motion include: ‘The afternoon raced by’; ‘Ski season is approaching’; and ‘She has a bright future ahead of her’ (Nunez & Cooperrider, 2013). Spatial constructions of time include describing a period of time as ‘a tiny fraction of a second’ (Nunez & Cooperrider, 2013) or as having a long or short duration. Time is also perceived as flowing in a certain direction (Cai, Connell & Holler, 2013), for example we may point backwards and say ‘long ago’, with the metaphorical aspect encoded separately in the gesture modality. There is also evidence that we consider time in terms of a left-right mental timeline with the past located to our left and the future to the right (e.g. Santiago, Lupianez, Perez & Funes, 2007), although there is no evidence that this conceptualization is encoded linguistically (Radden, 2004). Based on the prevalence of spatial metaphors used to describe time, we can predict that mental simulations underlying comprehension of temporal language would be comparable to those underlying spatial language. Furthermore, the different predictions for egocentric versus object-centred spatial descriptions, in terms of more or less simulation related to the body, would also hold for temporal descriptions that relate to the self (e.g. Ski season is approaching) and temporal descriptions related to other agents or objects (e.g. She has a bright future ahead of her). The way that time is mapped spatially also differs across language and cultures (Lai & Boroditsky, 2013). For example Mandarin speakers are less likely to take an ego-moving perspective than are English speakers (e.g. We are approaching the deadline), and instead use spatio-temporal phrases of a different direction such

6241-1269-PI-002.indd 39

AuQ31

10/3/2015 4:38:16 PM

40

Laura J. Speed, Gabriella Vigliocco

as the down month (the next month). This has implications for the types of spatial simulations underling comprehension of temporal language across cultures. However, language use can influence and change the type of spatial metaphors used to think about time: Mandarin-English bilinguals were more likely to use an ego-moving perspective than monolingual Mandarin speakers (Lai & Boroditsky, 2013). In sign language, space is similarly adapted to discuss time. In ASL, there are three different mappings of temporal information onto a spatial timeline (Emmorey, 2002). The deictic timeline runs from front to back, typically with the present time represented close to the signer and then future events portrayed away or in front of the body and past events portrayed as towards or behind the body. The left-right mental mapping of time mentioned above is also reflected in sign space, with the past to the left and the future to the right (although this is not reflected in any spoken languages). ASL also uses a temporal sign space that moves diagonally across the signing space, known as the anaphoric timeline. Typically the temporal meaning is determined within the discourse.

AuQ32

Simulating time Temporal information is used in discourse comprehension in the construction of a situation model of the described event (Zwaan, 1996; Anderson, Garrod & Sanford, 1983). Zwaan (1996) investigated how descriptions of time are understood in narratives by manipulating the chronological distance between two narrated events. Sentence reading time was found to be longer for greater narrative time shifts (e.g. an hour later versus a moment later). Moreover, information from a previous event was more difficult to access when it was followed by a shift in time compared to when it was not. This suggests that the representations of the two events in memory are more strongly connected when they are not separated by a time shift. Similar results were found with simple phrases and event duration: Coll-Florit & Gennari (2011) found that durative events, such as ‘The doorman was covering the sign,’ took longer to process than non-durative events, such as ‘The doorman covered the sign.’ The duration of mental simulations thus reflects temporal properties of the described real-world events. Anderson, Matlock, Fausey and Spivey (2008) similarly found that manipulating simple morphological information could change the duration and pattern of a simulated event. Participants were instructed to place a character in the appropriate place in a scene according to the sentence. They placed the character closer to the beginning of a to-be-used path and had longer mouse movement durations in completing the task when they heard a sentence with a past progressive (e.g. ‘Tom was jogging to the woods and then stretched when he got there’) than a simple past tense (e.g. ‘Tom jogged to the woods and then stretched when he got there’). Thus, grammatical aspect influenced how the event was simulated; with a past progressive the event was seen as on going in comparison to a simple past tense where the event was seen as completed.

6241-1269-PI-002.indd 40

AuQ33

10/3/2015 4:38:16 PM

Perceptual simulation

41

Similar studies have investigated how our understanding of time can be metaphorically grounded in our perception of space. Time can be conceptualized along a left-right axis, with the past placed on the left and the future on the right, or along a front-to-back axis with the future ahead and the past behind. It was found that responding to words with temporal connotations was facilitated if the words were presented congruently with either of the two spatial mappings of time (Torralbo, Santiago & Lupianez, 2006). For example responses to the word ayer (the Spanish word for yesterday) were faster when the word was presented behind a silhouette image of a head (facing left or right) than when it was presented in front. Similar results have been found for the comprehension of sentences, where sentences describing the past (e.g. Yesterday, Hanna repaired the bike) were responded to faster with the left hand than right, and vice versa for sentences about the future (e.g. The boss will sign the application tomorrow morning) (Ulrich & Maienborn, 2010). The mental timeline can also be influenced by the direction of reading/writing in a language. In Arabic, a language with a rightto-left writing system, time is conceptualized as moving from right-to-left, but for Hebrew, in which writing occurs right-to-left but calculating occurs left-toright, there is no preference (Tversky, Kugelmass & Winter, 1991). Such an effect can also be temporarily induced: the mental timeline in Dutch speakers reversed direction when they were trained to read from right-to-left (Casasanto & Bottini, 2010). However, the extent to which spatial groundings are recruited in comprehension of language about time may be dependent on context such as salience of attention to the temporal and spatial dimensions. Torralbo et al. (2006) found that the front-back mapping of time did not affect responses when they were made with left and right key presses, instead the left-right mapping of time dominated, and Ulrich and Maienborn (2010) found that the congruency effect between the left-right mental timeline and the response hand disappeared when the task did not focus on the temporal dimension (i.e. deciding whether the sentence made sense instead of whether the sentence described the future or the past). Further, although comprehension of time can be affected by space, time cannot conversely affect perception of space (Casasanto & Boroditsky, 2008). When reproducing the duration or spatial configuration of a set of lines or dots, irrelevant spatial information affected duration reproductions, but irrelevant duration information did not affect spatial reproductions. Thus time is ‘asymmetrically dependent’ on space (Casasanto & Boroditsky, 2008, p. 581). In sum, evidence shows that when comprehending discourse, comprehension of events with a longer or ongoing duration takes longer, reflecting real-world temporal properties. Further, comprehension of temporal language involves spatial simulations, which can be affected by spatial-temporal experience in the world (e.g. reading direction). However, spatial simulations of time are less robust than simulations in other domains, as they appear to be dependent on salience of temporal information, and do not lead to bidirectional effects between time and space.

6241-1269-PI-002.indd 41

AuQ34

10/3/2015 4:38:16 PM

42

Laura J. Speed, Gabriella Vigliocco

Summary and implications In sum, evidence exists for the mental simulation of space, speed and time in the comprehension of both word and sentences. Importantly, the relationship between language and perception appears to be bidirectional (language affects perception and vice versa). Dominant measures of the simulation process include reading time in narratives, response time to combinations of words and perceptual stimuli, and eye movement patterns during comprehension. It should be noted however, that without a fine-grained temporal measure, such as reading time on single words during sentence comprehension, it is not clear at which point during sentence comprehension the mental simulation occurs. For example taking a measure that encompasses the entire process of sentence comprehension (such as sentence sensibility judgments) does not show whether it is a simulation only for a single verb, an incremental building of a simulation beginning at the verb, a sentence wrap-up effect or a combination of effects. This chapter addressed the embodiment of language about space, speed and time. These three domains may less obviously be grounded in our perceptual processes for two reasons. First, the three domains are difficult to describe in terms of perceptual features without reference to other objects or events. Time is particularly inscrutable, the definition of which is of philosophical debate (Klein, 2009). Thus it appears difficult to explain the comprehension of these domains as being linked with perceptual experience. The second reason pertains particularly to the domain of speed. Speed could often be viewed as a fine-grained detail of mental simulation that may not be necessary in ordinary comprehension of motion events, although can become salient when speed is vital to a situation. The evidence reviewed in this chapter has shown that space, speed and time are in fact dimensions that are included in mental simulations, or situation models, built during discourse comprehension. Further, the perception of space and speed can affect the comprehension of language about space and speed. Moreover, this relationship is bidirectional, with language also affecting perception. This is evidence that language comprehension and perceptual processes overlap, supporting an embodied theory. The comprehension of time however, is often grounded in perceptual processes of more concrete domains, such as space. For example spatial configurations in an experiment can affect the comprehension of language describing time. However, it seems likely that the grounding of time in space is less robust and automatic in comparison to grounding for space and speed language because time-space congruency effects are very susceptible to task effects and attentional demands. Further, the relationship between time and space is not bidirectional: although space may influence the comprehension of time, language about time does not affect perception of space. The results presented above also suggest certain factors that have implications for the nature of simulations carried out during simulation. Firstly, there is a suggestion that simulations differ across languages due to the way that domains are described (e.g. temporal descriptions in Mandarin vs. English) as well as the

6241-1269-PI-002.indd 42

10/3/2015 4:38:16 PM

Perceptual simulation

43

modality of language presentation (e.g. efficiency of spatial comprehension in sign languages). Results also raise the importance of contextual factors in comprehension: here we have described how the context of a visual scene and the difficulty of a perceptual task can affect the nature of the mental simulation. Future research needs to further specify the types of constraints that affect the mental simulations carried out during comprehension and what this means for embodied theories of comprehension.

References Altmann, G.T.M. & Kamide, Y. (2004). Now you see it, now you don’t: Mediating the mapping between language and visual world. In J. Henderson & F. Ferreira (Eds.). The interface of language, vision, and action: Eye movements and the visual world (pp. 347–386), New York: Psychology Press. Anderson, A., Garrod, S. C. & Sanford, A. J. (1983). The accessibility of pronominal antecedents as a function of episode shifts in narrative texts. Quarterly Journal of Experimental Psychology, 35A, 427–440. Avraamides, M. N. (2008). Spatial updating of environments described in texts. Cognitive Psychology, 45, 402–431. Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4 ), 577–609. Boroditsky, L. (2000). Metaphoric structuring: Understanding time through spatial metaphors. Cognition, 75(1), 1–28. Cai, Z. G., Connell, L. & Holler, J. (2013). Time does not flow without language: Spatial distance affects temporal duration regardless of movement or direction. Psychonomic Bulletin & Review, 20, 973–980. Casasanto, D. & Boroditsky, K. (2008). Time in the mind: Using space to think about time. Cognition, 106, 579–593. Coventry, K. R., Lynott, D., Cangelosi, A., Monrouxe, L. V., Joyce, D. & Richardson, D.C. (2010). Spatial language, visual attention and perceptual simulation. Brain and Language, 112, 202–213. Denis, M., & Cocude, M. (1989). Scanning visual images generated from verbal descriptions. European Journal of Cognitive Psychology, 1, 293–307. Dudschig, C., Lachmair, M., de la Vega, I., De Filippis, M. & Kaup, B. (2012). From top to bottom: Spatial attention shifts caused by linguistic stimuli. Cognitive Processing, 13, 151–154. Estes, Z., Verges, M. & Barsalou, L. W. (2008). Head up, foot down. Object words orient attention to the objects’ typical location. Psychological Science, 19(2), 93–97. Fecica, A. M. & O’Neill, D. K. (2010). A step at a time: Preliterate children’s simulation of narrative movement during story comprehension. Cognition, 116(3), 368–81. Glenberg, A. M. & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9(3), 558–65. Glenberg, A. M., Meyer, M. & Lindem, K. (1987). Mental models contribute to foregrounding during text comprehension. Journal of Memory and Language, 26, 69–83. Haiman, J. (1980). The iconicity of grammar. Language 56, 515–40. Jansen, G., Haun, D.B.M. & Levinson, S. C. (2012). Tracking down abstract linguistic meaning: Neural correlates of spatial frame of reference ambiguities in language, PLoS ONE, 7(2), e30657.

6241-1269-PI-002.indd 43

AuQ35

10/3/2015 4:38:16 PM

44

Laura J. Speed, Gabriella Vigliocco

Klein, W. (2009). Concepts of time. In W. Klein & P. Li (Eds.), The expression of time (pp. 5–38). Berlin: Mouton de Gruyter. Landauer, T. K. & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104, 211–140. Levine, W. H. & Klin, C. M. (2001). Tracking of spatial information in narratives. Memory and Cognition, 29(2), 327–335. Levinson, S. C. (1997). From outer to inner space: Linguistic categories and non-linguistic thinking. In T. Nuyts & E. Pederson (Eds.), Language and conceptualization, (pp. 13–45), Cambridge, UK: Cambridge University Press. Levinson S. C., Kita S., Haun D.B.M. & Rasch B. H. (2002). Returning the tables: Language affects spatial reasoning. Cognition, 84, 155–188. Lindsay, S., Scheepers, C. & Kamide, Y. (2013). To dash or to dawdle: Verb-associated speed of motion influences eye movements during spoken sentence comprehension. PLoS One. 8, 6, e67187. Lund, K. & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instrumentation, and Computers, 28, 203–208. Ma, W. J., Zhou, X., Ross, L. A, Foxe, J. J. & Parra, L. C. (2009). Lip-reading aids word recognition most in moderate noise: A Bayesian explanation using high dimensional feature space. PloS One, 4(3), e4638. Majid, A., Bowerman, M., Kita, S., Haun, D.B.M., & Levinson, S. C. (2004) Can language restructure cognition? The case for space. Trends in Cognitive Sciences, 8(3), 108–114. Meteyard, L., Zokaei,N., Bahrami, B. & Vigliocco, G. (2008). Visual motion interferes with lexical decision on motion words. Current Biology, 18(17), 732–733. Morrow, D. G., Greenspan, S. L. & Bower, G. H. (1987). Accessibility and situation models in narrative comprehension. Journal of Memory and Language, 26, 165–187. Myachykov, A., Scheepers, C., Fischer, M. H. & Kessler, K. (2014). TEST: A tropic, embodied, and situated theory of cognition. Topics in Cognitive Sciences, 6(3), 442–460. Nunez, R. & Cooperrider, K. (2013). The tangle of space and time in human cognition. Trends in Cognitive Sciences, 17(5), 220–229. Sanford, A. J. (2008). Defining embodiment in understanding. In M. De Vega, A. M. Glenberg & A. Graesser (Eds.), Symbols and embodiment: Debates in meaning and cognition (pp. 181–194). Oxford, England: Oxford University Press. Speed, L. J. (2014). Simulating speed in language: Contributions from vision, action and audition. Unpublished PhD thesis, UCL. Speed, L. J. & Vigliocco, G. (2014). Eye movements reveal the dynamic simulation of speed in language. Cognitive Science, 38(2), 367–382. Stanfield, R. A. & Zwaan, R. A. (2001). The effect of implied orientation derived from verbal context on picture recognition. Psychological Science, 12, 153–156. Torralbo, A., Santiago, J. & Lupianez, J. (2006). Flexible conceptual projection of time onto spatial frames of reference. Cognitive Science, 30, 745–757. Tversky, B., Kugelmass, S. & Winter, A. (1991). Cross-cultural and developmental trends in graphic productions. Cognitive Psychology, 23, 515–557. Ulrich, R. & Maienborn, C. (2010). Left-right coding of past and future in language: The mental timeline during sentence processing. Cognition, 117, 126–138. Wilson, R. A. & Foglia, L. (2010). Embodied Cognition. In E. N. Zalta (Ed.) The Stanford Encyclopedia of Philosophy (Fall 2011 Edition). Stanford, CA: Stanford University. Zhang, Z., Jia, L. & Ren, W. (2014). Time changes with feeling of speed : An embodied perspective. Frontiers in Neurorobotics, 8(14), 2012–2015.

6241-1269-PI-002.indd 44

AuQ36

10/3/2015 4:38:16 PM

Perceptual simulation

45

Zwaan, R. A. (1996). Processing narrative time shifts. Journal of Experimental Psychology: Learning, Memory and Cognition, 22, 1196–1207. Zwaan, R., Madden, C. J., Yaxley, R. H. & Aveyard, M. E. (2004). Moving words: dynamic representations in language comprehension. Cognitive Science, 28 (4), 611–619. Zwaan, R. A., Stanfield, R. A. & Yaxley, R. H. (2002). Language comprehenders mentally represent the shapes of objects. Psychological Science, 13(2), 168–71. Zwaan R. A., Madden C. J. & Stanfield R. A. (2001). Time in narrative comprehension. In D. H. Schram & G. J. Steen (Eds.), Psychology and Sociology of Literature (pp. 71–86). Amsterdam: John Benjamins. Zwaan, R. A. & Radvanksy, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123(2), 162–185. Zwaan, R. A., & Yaxley, R. H. (2003). Spatial iconicity affects semantic-relatedness judgments. Psychonomic Bulletin & Review, 10, 954–958.

6241-1269-PI-002.indd 45

AuQ37

10/3/2015 4:38:16 PM