Differentiation of Visual Action Words - Springer Link

J Psycholinguist Res (2014) 43:1–11 DOI 10.1007/s10936-012-9235-1

Do You ‘See’ What I ‘See’? Differentiation of Visual Action Words Joël Dickinson · Laura Cirelli · Frank Szeligo

Published online: 22 January 2013 © Springer Science+Business Media New York 2013

Abstract Dickinson and Szeligo (Can J Exp Psychol 62(4):211–222, 2008) found that processing time for simple visual stimuli was affected by the visual action participants had been instructed to perform on these stimuli (e.g., see, distinguish). It was concluded that these effects reflected the differences in the durations of these various visual actions, and the results were compared to participants’ subjective ratings of word meaning but it was also possible that word characteristics like length might have influenced response times. The present study takes advantage of word length differences between French and English visual action words in order to address this issue. The goals of the present study were to provide evidence that (1) the processing time differences previously found were due to differences in the cognitive actions represented by these words (and not due to characteristics to the words themselves), and (2) that individuals subjectively differentiate visual action words in such a way that allows for predictable differences in behaviour. Participants differentiated 14 French visual action words along two dimensions. Four of these words were then used in the instructions for a size-discrimination task. Processing time depended on the visual action word in the instruction to the task and differed in a predictable manner according to word meaning but not word length. Keywords

Visual action words · Processing time · Instructions

Introduction In all languages, a great selection of words exist to describe visual actions. For example, if someone wished to describe devoting mental resources towards visually processing a presented stimulus they could choose a number of the following words/phrases to describe J. Dickinson (B) · L. Cirelli Department of Psychology, Laurentian University, Sudbury P3E 2C6, ON, Canada e-mail: [email protected] Frank Szeligo Department of Psychology (Retired), University of New Brunswick, Fredericton, Canada

123

2

J Psycholinguist Res (2014) 43:1–11

that action: see, notice, observe, attend to, be aware of (to name a few). Cacciari and Levorato (2000) suggested that the choice to describe a visual action with one of these words instead of another reflects that these actions differ along some defining characteristic of perception. Therefore, by understanding how the words used to represent these actions are differentiated from one another, we can better understand the perceptual properties of the actions themselves (Miller 1991; Cacciari and Levorato 2000). A number of studies have now explored how individuals differentiate visual action words from one another (e.g. Cacciari and Levorato 2000; Dickinson and Szeligo 2008; Pasanen 1978; Schwanenfluegel et al. 1994; Shabanova 2000). Although these studies provide us with a greater understanding of how we categorize these words, they do not provide insight into how these differentiations relate to behaviour. The unique contribution of Dickinson and Szeligo (2008) is in their development of a behavioural measure for these words. Specifically, by instructing participants to perform the visual actions represented by these words and measuring the time that it takes for them to do so, the relationship between participants’ understanding of the words and the impact of these words on processing time and accuracy in simple visual tasks could be assessed. With this measure, Dickinson and Szeligo (2008) showed that the time required to visually process stimuli (both words and images) significantly differed in terms of the visual action that they were asked to perform on these stimuli. For example, when participants were asked to respond with a mouse click immediately after they ‘see’ that an image was presented, they responded faster (mean RT = 325 ms) than if they were asked to ‘perceive’ it (mean RT = 369 ms) or ‘become conscious’ of it (mean RT = 382 ms). Such results were found in various task types (responding to a word, responding to an image, and a go-nogo task) suggesting that the effect of these words on cognitive performance was consistent (Dickinson and Szeligo 2008). Dickinson and Szeligo (2008) compared the behavioural results to subjective ratings of word meaning, and so participants were asked to rate the similarity in meaning of 14 visual action words in the The Mental Operations: Rating of Sameness Scale (Dickinson and Szeligo 2008, adapted from Cacciari and Levorato 2000). Using multidimensional scaling (MDS) analysis, it was found that 78 % of the variance between the subjectively rated word meanings could be explained by a one dimension solution. The word ratings along this one dimension followed the same ordinal pattern as the processing times found in the above mentioned experiments and so it was concluded that individuals had some meta-cognitive knowledge of how these words differed in terms of the resulting perceptual processes. That is, participants appear to have differentiated the words along a characteristic that has a salient affect on perceptual processing. There were however issues with Dickinson and Szeligo (2008) conclusions, which are listed below and will be addressed in the current paper. The Possible Effect of Word Characteristics on Processing Time Although the processing time effects found were consistent with differences in perceived meaning (as rated by participants), there were also other word characteristics (both objective and subjectively rated) that correlated with response time. Specifically, words eliciting longer response times were longer in length, led to greater certainty that they had been performed correctly, were more difficult to perform, and were understood, by the participants, to take longer to perform (see Dickinson and Szeligo 2008 for a more detailed description of these relationships). This raised the question of whether the results of the behavioural measure truly reflected a measurable difference in the cognitive process being performed or if results were a byproduct

123


3

of the mental action word’s characteristics. Dickinson and Szeligo (2008) argued that since participants are responding to stimuli that they are asked to perform certain visual actions on, and not to the words representing these visual actions themselves, word characteristics should not affect processing speed. However, this requires experimental verification. Previous research on words and their impact on behaviour tends to evaluate participant’s responses to the actual word itself. For example, Pulvermüller et al. (2001) found that, when asked to respond only to actual words but not to pronounceable pseudowords (lexical decision task), participants responded much faster to face-related verbs (e.g. smile) than arm- (e.g. throw) or leg-related (e.g. run) verbs. Pulvermüller et al. (2001) concluded that these response time differences reflect a difference in word meaning retrieval speed. Such retrieval speeds can be greatly affected by characteristics of the word itself like frequency, familiarity and length. Dickinson and Szeligo (2008) claim that their methodology is not measuring the rate of word meaning retrieval but the actual processing time of the visual action represented by the word. Demonstrating that word characteristics like frequency of use and word length (which affect the speed of word meaning retrieval) are not associated with the processing time differences would support this claim. The Ability of MDS Ratings to Predict Behavioural Differences Dickinson and Szeligo (2008) concluded that because the multidimensional rating results followed the same ordinal pattern as the response time differences found, the aspect of word meaning that participants considered when differentiating words along this dimension might account for the time differences. If this were true, then multidimensional scaling results (which inform us on how participants are differentiating these multiple words) could be used to accurately predict the direction at which the visual actions represented by these words vary in terms of processing time. On the other hand, the processing time results found by Dickinson and Szeligo (2008) might simply be an artifact of the four words chosen for their specific task. If this relationship exists within and generalizes across languages, then Dickinson and Szeligo (2008) interpretation can be supported. The Present Study The present study uses Dickinson and Szeligo (2008) multidimensional scaling and behavioural measure with French visual action words in order to confront two main goals: (1) to support the idea that Dickinson and Szeligo (2008) methodology is measuring the performance of the cognitive actions represented by the visual action words and not merely participants‘ responses to the words themselves and (2) to demonstrate that the way participants discriminate between these words in terms of meaning can be used to predict measurable components (specifically processing time) of these actions. Because of the nature of the English visual action words used by Dickinson and Szeligo (2008), a direct investigation on the impact of word length on processing time was impossible. However, since the French counterparts of some of these words varied in word length from their English counterparts, the processing time of French vision words that differed in word length but not multidimensional rating could be taken advantage of. If it were found that the actions represented by French visual action words differing in rating but not word length did differ in processing time, then the first goal of the present study could be supported. Also, if such processing time differences were found in French, this would demonstrate that previous results were not isolated behavioural artifacts to English visual action words. More importantly, if multidimensional scaling of the French words can lead to directional

123

4


predictions in how processing time is affected, the second goal of the present study could be supported. The first experiment of the present study gathered subjective ratings of French visual action word meaning and assessed them using Multidimensional scaling. This was a necessary step before conducting Dickinson and Szeligo (2008) behavioural measure of these words in French since these ratings could be used to help select words for the behavioural measure that differ in either meaning (subjective rating) or word length. By being able to select these words based on meaning and word length, it could be determined which of these factors acts as a predictor for processing time differences. This was the goal of the second experiment. If word meaning and not length predicted processing time differences, then the validity of Dickinson and Szeligo (2008) measure would be supported.

Experiment 1: French Multidimensional Scaling Methods Participants Participants were 45 Francophone undergraduate students recruited from Laurentian University. The average age of participants was M = 20.21 (S = 1.64). Materials and Procedure The entire procedure was conducted in French. English translations of instructions are for the convenience of the reader. A French translation of the ‘Mental Operations: Ratings of Sameness Scale’ (Dickinson and Szeligo 2008) was used. The scale was used to present words/phrases that were selected because they can be used to describe a visual stimulus and they represent mental actions or states (see Table 1). The participants were asked to rate how similar each of the words listed were to each other in terms of meaning (which resulted in a total of 91 word pairs to be rated). For example, the word-pair ‘discerner’ – ‘se render compte de’ (discern– be aware of) was presented and participants were asked to indicate how similar the words in each pair are in terms of their meaning by circling the appropriate number. Ordering/pairings of words were done in a manner consistent with Davison (1983). A rating of ‘1’ signified ‘très différents’ (‘highly dissimilar’) and a rating of ‘7’ signified ‘très semblables’ (‘highly similar’). Since similarity ratings were obtained from the study, data was re-coded (e.g. 1 = 7, 2 = 6) in order to obtain a dissimilarity or distance score. The MDS weights created were then used for comparison with specific word characteristics including word length, the French language written frequency score of Baudot (1992), and the French language written frequency score of New et al. (2004). Results Mean distances were entered into a symmetry matrix in order to perform a multidimensional scaling analysis using SPSS. One, two, and three dimensional solutions were tested. The amount of variance accounted for (R square) by the first dimension was 0.68; R square for a two dimensional solution was 0.87. An F test revealed that the R square change (0.20) was significant F(1,11) = 16.89, p < 0.05. The R square change (.04) from the two dimension

123


5

Table 1 The 14 words/phrases from the ‘Mental Operations: Ratings of Sameness Scale’ (French and English) (Dickinson and Szeligo 2008) with MDS values for English and French 2 dimensional solutions, with words and values sorted by FMDS1 values FRENCH

ENGLISH

French MDS 1

French MDS 2

Distinguer

Distinguish

−1.66

−0.29

0.64

Regarder

View

−1.51

1.21

−1.23

Voir

See

−1.32

0.99

−0.88

Identifier

Identify

−0.64

−0.76

0.36

Discerner

Discern

−0.57

−1.59

1.50

Percevoir

Perceive

−0.23

0.13

Reconnaître

Recognize

−0.03

Prêter attention à

Attend to

0.02

Remarquer

Notice

0.1

0.52

0.03

Détecter

Detect

0.36

−0.39

0.27

Se rendre compte de

Be aware of

0.54

0.12

–0.20

Être conscient de

Be conscious of

1.21

0.17

0.39

Être au courant de

Be cognizant of

1.27

0.86

–0.05

Ressentir

Sense

2.45

−1.54

1.40

−0.6 1.16

English MDS

0.19 0.15 –2.58

solution to a three dimension solution was not significant, therefore a two dimensional solution was selected. The two dimensional solution can be seen in Fig. 1 (the specific coordinates of the French words along each dimension can be seen in Table 1). To evaluate if these words were being differentiated in a way similar to the English differentiations, correlations between the coordinates of the French visual action words on each dimension of the solution and the coordinates of the corresponding English words from the single dimension solution found by Dickinson and Szeligo (2008) were calculated. For the first French dimension 1, the value was a non-significant pearson r of .30. For the second French dimension, the correlation was r = −.85, p < .05. The word ratings for the first and second French dimensions were not related to word length or to either frequency measure. Discussion Comparison to English Dimension The fact that Dickinson and Szeligo (2008) found a one-dimensional solution and the present experiment found a two-dimensional solution for visual action word meaning is not terribly surprising. In both cases a significance test was used to evaluate if the next dimension accounted for a significant amount of additional variability. As with stepwise multiple regressions, this procedure has been known to be somewhat unstable (e.g Thompson 1995). Also, despite the fact that only one dimension of word meaning was found in the English solution, the English and French vision words still seem to be differentiated in a similar manner (the second French dimension had a correlation of r = −.85 with the English dimension). Such similar word meaning differentiations suggest that when these words are differentiated in terms of meaning, it is truly the abstract action represented by the words (which should be

123

6


Fig. 1 French MDS solution

universal) that is being considered. The English MDS should therefore be repeated in future studies to determine if increasing statistical power might lead to results more similar to those found in the present study (specifically, finding two significant dimensions).

Effect of Frequency on Word Meaning The result of most interest is that word length and frequency do not relate to the word ratings for either of the two French dimensions found. This suggests that these word characteristics do not contribute to participants’ subjective differentiation of these words. Therefore, the meaning of these words is not dependent on these word characteristics. Since words are merely symbols which represent concepts, the physical characteristics of the word (word length, frequency, etc) should not affect one’s understanding of the concept represented by the word. However, this possibility could not be ruled out from the data presented in Dickinson and Szeligo (2008). Since word length and frequency do not relate to how these French vision words are differentiated in terms of meaning, it was possible to select words for the behavioural measure that differ in either meaning or word length. By doing so, it could be demonstrated that Dickinson and Szeligo (2008) behavioural measure is indeed tapping into the measurement of the visual process and not just word meaning retrieval (which is affected by word length and frequency). This would therefore increase the validity of the measure, and is explored in Experiment 2.

123


7

Experiment 2: French Behavioural Measure Methods Participants Forty undergraduate students attending Laurentian University who had French as a first language participated in the experiment. Mean age for the participants was 21.35 (S = 6.07). Materials E-Prime (1.0) (Schneider et al. 2002) was used to present instructions and trials as well as to record accuracy and to measure response times in a size discrimination task. Stimuli The French visual action words were selected for Experiment 2 based on their word length and dimensional ratings. The words selected were ‘discerner’, ‘ressentir’, ‘voir’, and ‘regarder’. While ‘discerner’, ‘ressentir’ and ‘regarder’ are similar in word length, ‘discerner’ was rated differently than ‘ressentir’ on only the first French dimension, and different from ‘regarder’ on only the second French dimension. Also, while ‘regarder’ and ‘voir’ differ in word length, they have similar ratings on both dimensions. These words were embedded in a common instruction creating four instructional conditions. The stimuli that the participants were instructed to respond to were two green triangles presented in two conditions: (1) Same (2.5 cm × 2.5 cm × 2.5 cm vs. 2.5 cm × 2.5 cm × 2.5 cm) and (2) Different (2.5 cm × 2.5 cm × 2.5 cm vs. 2.75 cm × 2.75 cm × 2.75 cm). The triangles were displayed in the centre of the 42.4 cm (17 inch) monitor, were 18.75 cm apart, and were aligned from the mid-point. The 2.5 cm stimulus was always displayed on the left. Each of the four instructions were presented 12 times for each stimulus (same, different) for a total of 96 trials. Procedure Participants were familiarized with the conditions by first performing 4 practice trials that were similar to the experimental trials. These practice trials were followed by experimental instructions displayed on the monitor for the participants until they clicked the mouse to continue. The 96 experimental trials then began, within which one of the four instructions was randomly paired with one of the two (same/different) stimuli conditions. Trials consisted of the same basic format: the trial instruction was displayed for 2 s, followed by a blank screen which was displayed for a random period of time (averaging 500 ms) until stimulus onset. The stimuli were then displayed until the participant responded. Response times were measured from stimulus onset to key press (see Fig. 2 for the sequence of events in a typical trial). Participants used their dominant hand to respond. Each trial instruction consisted of the statement: “Répondez aussitôt que vous ____ que les triangles sont de la même taille ou de tailles différentes.” (“Respond immediately after you ____ that the triangles are the same or different.”) Contained in the space was one of four mental action words: ‘regarder’, ‘voir’, ‘discerner’ or ‘ressentir’.

123

8


Fig. 2 Visual representation of trial procedure

Table 2 Response time per instruction from current study and (Dickinson and Szeligo 2008) French

Response time

Instruction

Mean

Voir

1141

English

Response time

SD

Instruction

Mean

SD

396

See

1410

639

Regarder

1149

387

Recognize

1453

611

Discerner

1166

394

Are conscious

1468

680

Ressentir

1239

452

Distinguish

1554

731

Results Processing Time For all variables, repeated measures ANOVA were conducted with an alpha of .05. Processing time data was averaged across trials prior to analysis. An instructional main effect was found for processing time F(3,117) = 4.85, p < .05, η2 = .11. Post-hoc tests indicate ‘voir’ – ‘ressentir’: F(1,39) = 10.28, p < .05, η2 = .21, ‘regarder’ – ‘ressentir’: F(1,39) = 6.89, p < .05, η2 = .15, and ‘discerner’ – ‘ressentir’: F(1,39) = 5.24, p < .05, η2 = .19 to be significantly different. The instructional means are displayed in Table 2 along with the response times from Dickinson and Szeligo (2008).

123


9

Accuracy No Instruction main effect was found for proportion of hits F(3,117) = 1.89, p > .05 or correct rejections F(3,117) = 1.06, p > .05. Mean proportion of hits across all instructions was .75, while mean proportion of correct rejections across all instructions was .74. Same Versus Different Trials No significant differences for response time or accuracy were found between same and different trials. Signal detection analysis was also performed, however no significant differences were found on either sensitivity or criterion measures. Discussion The results of this experiment suggest that word characteristics such as length and frequency do not contribute to the processing time differences found. The words ‘voir’ and ‘regarder’ (which differ in word length) were rated almost identically on both dimensions and did not differ in terms of the processing times elicited. As well, the processing times for the ‘regarder’ and ‘discerner’ conditions significantly differed from the processing times in the ‘ressentir’ condition despite the fact that all three words are highly similar in word length. Together, these results suggest that word length and frequency do not relate or contribute to the processing time differences found. Instead, the ratings of word meaning along the first dimension appear to account for the processing time differences. ‘Ressentir’, which led to significantly longer processing times than the other three instructions differed from these other three instructions along the first dimension but not from ‘discerner’ along the second. As well, ‘discerner’ differed greatly in dimensional rating from ‘regarder’ and ‘voir’ on the second dimension but not the first and these three instructional conditions did not lead to significantly different processing times. Together, these findings suggest that it is the factor(s) that lead individuals to differentiate these words along the first dimension of word meaning and not the second that also account for differences is processing time. So, in simpler terms, the results suggest that the cognitive characteristic of the action ‘ressentir’ which differs from the other three words is associated with the increased processing time when the action represented by this word is performed.

General Discussion Since word length and frequency are not related to the ratings of word meaning, the relationship found between dimensional rating and processing speed can be attributed to the meanings of the words and not to the characteristics of the words (which commonly affect word meaning retrieval time). This supports (Dickinson and Szeligo 2008) claim that their processing time measure allows for quantitative measurement of the cognitive actions represented by visual action words and not merely the effects of word meaning retrieval. These findings support the first goal of the present study, and further validate Dickinson and Szeligo (2008) behavioural measure. The second goal of the study (to demonstrate dimensional ratings of word meaning can be used to predict measurable components of these visual actions) was also supported. Significant processing time differences were found, and are associated with the dimensional ratings

123

10


along the first dimension of word meaning. The fact that these processing time differences were found in a different language using different words suggest that the processing time effects found by Dickinson and Szeligo (2008) were not simply an artifact of the particular English words that had been chosen for their task. Theoretical Implications Based on associative models of verb meaning, results of this study support the hypothesis that the meanings of the visual action words is directly reflective of the visual action represented by these words. This associative model of verb meaning (Pulvermüller 2005) suggests that increasingly strengthened connections form between a verb and the corresponding action performed or observed directly before, during or after hearing that verb. Human neuroimaging evidence has supported this model by demonstrating that processing an action word relating to the face, arms or legs activated the area of the motor strip containing the topographical representation of the related body part (Pulvermüller et al. 2001; Hauk and Pulvermuller 2004). This model corresponds to Barsalou (1999) perceptual symbols theory which suggests that all perceptual input relating to a concept contribute to its meaning. For example, the action of kicking (which commonly occurs before, during or after hearing the word ‘kick) contributes to our understanding of the concept represented by the word ‘kick’. If we extend the theories of Pulvermüller (2005) and Barsalou (1999) to abstract concepts like the vision verb ‘see’, our findings suggest that it is possible that the introspective perceptual experience that we come to associate to the word ‘see’ may come to define our understanding of the concept of ‘seeing’. So when participants are asked to differentiate between the meaning of these visual action words, they may be drawing on the introspective information that represents their understanding of these actions and so distinctions between words are based on distinctions between perceptual characteristics that can differ among these words. This is supported by the finding that visual action words are differentiated in a similar way in both English and French. Future Directions The results of this study suggest that these vision verbs are learned in a similar way within and across language, supporting the associative models of verb learning (Pulvermüller 2005; Barsalou 1999). Therefore, the next logical step would be to evaluate cortical activity using ERP technology to compare activity when reading the different vision words within the instructions (therefore targeting differences in word meaning retrieval for action words with no verifiable product) and when this action is performed on stimuli following the instructions (therefore targeting differences in resultant processing when performing these actions). The previous (Dickinson and Szeligo 2008) MDS showed only a weak relationship with the present MDS dimension that was relevant to response time. Future studies might also investigate the characteristics of these visual actions that might vary from one word to the next and how these differences contribute to the ratings of word meaning found. Specifically, Dickinson and Szeligo (2008) hypothesized that one of these factors may be level of processing; the cognitive actions represented by each visual action word may differ in terms of how deeply information is processed when these actions are performed. Future research might seek to investigate this hypothesis further.

123


11

Conclusion The goals of the present study were both met. We have provide evidence that (1) the processing time differences that occurred in Dickinson and Szeligo (2008) are due to differences in the cognitive actions represented by these words (and not due to characteristics to the words themselves), and (2) individuals subjectively differentiate visual action words in such a way that allows for predictable differences in behaviour. The consistent findings of how people differentiate words both within and between languages suggests that we may be learning visual action words through associative learning in a similar fashion to more concrete action verbs (e.g. Pulvermüller 2005; Barsalou 1999).

References Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577–660. Baudot, J. (1992). Fréquences d’utilisation des mots en français écrit contemporain. Montreal: Les presses de l’Université de Montréal. Cacciari, C., & Levorato, M. C. (2000). The semantic structure of vision verbs: A psycholinguistic investigation of Italian. European Journal of Cognitive Psychology, 12(1), 87–106. Davison, M. L. (1983). Multidimensional scaling. New York: Wiley. Dickinson, J., & Szeligo, F. (2008). Impact of mental operation instructions. Canadian Journal of Experimental Psychology, 62(4), 211–222. Hauk, O., & Pulvermuller, F. (2004). Effects of word length and frequency on the human event-related potential. Clinical Neurophysiology, 115, 1090–1103. Miller, G. A. (1991). Lexical echoes of perceptual structure. In R. Lockhead & J. R. Pomerantz (Eds.), (pp. 249–261). Washington, DC, US: American Psychological Association. New, B., Pallier, C., Brysbaert, M., & Ferrand, L. (2004). Lexique 2: A new French lexical database. Behavior Research Methods, Instruments & Computers, 36, 516–524. Pasanen, M. L. (1978). Finnish and English verbs of vision. Further contrastive papers. Jyvaskyla Contrastive Studies, (6), 1–28. Pulvermüller, F. (2005). Brain mechanisms linking language and action. Nature Reviews Neuroscience. Special Issue: Focus on Pain, 6(7), 576–582. Pulvermüller, F., Härle, M., & Hummel, F. (2001). Walking or talking?: Behavioral and neurophysiological correlates of action verb processing. Brain and Language, 78(2), 143–168. Shabanova, T. (2000). Semantic and pragmatic interface in English verbs of vision. http://fccl.ksu.ru/winter. 2000/paper5.pdf. Schneider, W., Eschman, A., & Zuccolotto, A. (2002). E-Prime (Version 1.0). [Computer software and manual]. Pittsburgh, P.A.: Psychology Software Tools Inc. Schwanenfluegel, P., Fabricius, W. V., Noyes, C. R., & Bigler, K. D. (1994). The organization of mental verbs and folk theories of knowing. Journal of Memory and Language, 33, 376–395. Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4), 525–534.

123