vs Self-paced.
Tilburg University
Max Planck Institute for Psycholinguistics
Radboud University
Complementary routes for conceptual processing? Modality, RTs, and ERPs Supervisors
Pablo Bernabeu Roel Willems (Radboud U & MPI) Max Louwerse (Tilburg U)
ERP experiment on conceptual processing • Main study: Is conceptual processing both symbolic and embodied? Modality-switching paradigm adapted for both effects (Pecher et al., 2003; Hald et al., 2011; Collins et al., 2003)
• Sub-study for stimulus preparation: modality-exclusivity norms: Conceptual modalities, and sound-symbolism (Lynott and Connell, 2009, 2013; Louwerse & Connell, 2011)
Welcome to the experiment
Red
Room
+
Catchy
Melody
+
Modality-switching Controlled transitions across trials: prime + target compounds EVENT = transition = modality shift or non-shift Randomize compounds TASK: Property verification = Property … Concept [verify] *216 Can the property be applied to the concept? E.g.: Solid Ideal [Response] / Tasteless Moment [Response] E.g.: Red Room [Response] / Catchy Melody [Response]
Modality-switching Controlled transitions across trials: prime + target compounds EVENT = transition = modality shift or non-shift Randomize compounds TASK: Property verification = Property … Concept [verify] *216 Can the property be applied to the concept? E.g.: Solid Ideal [Response] / Tasteless Moment [Response] VISUAL
VISUAL
Ind. Var. condition = non-shift
Interpretation of the processing cost • CLASSICAL: conceptual processing is (partly) modality-specific / embodied / situated / substrated in sensory and motor cortex • RE-ANALYSIS: Louwerse and Connell (2011) More to it: heuristic versus detailed processing Where it was coming from
Distributional semantics DOG
pet walk cute
GRAB
GRAB
PARADOX
irony text surprise
Latent Semantic Analysis
Embodied cognition PARADOX
DOG
take force coffee
= EXPERIENCE = (http:// lsa.colorado.edu)
EMOTIONAL MOTOR SENSORY
Kousta et al (2011) *
YES/NO
Willems et al (2011) Hald et al (2011) Simmons et al (2007)
CORPUS: Populations (Louwerse & Zwaan, 2009) RT: animals, body, geogra. (Lund & Burgess, 1996) * BRAIN: Voxel-level neural activation predicted by distributional statistics (Mitchell et al, 2008)
‘Green’ Causality question
Symbol-grounding problem + Occam…
(Hickok, 2014; Willems & Francken, 2012; Mahon & Caramazza, 2008)
Arguments (straw men?) on each side…
Got’cha, symbolic processing
Got’cha, sensorimotor correlates
What was first in comprehension?
Sensorimotor for sure?
Distributional - embodied interplay (Simmons et al, 2008; Willems & Francken, 2011; Andrews et al, 2014) [+ about everyone now]
Time, (goal), (need), imageability, (mode). Gender, (expertise), (literacy).
Modality-exclusivity norms (Lynott & Connell, 2009) 423 adjectives 5 sensory modalities:
Louwerse & Connell (2011) Google corpus = 2-, 3-, 4-, 5-grams 423*423 matrix of co-occurrence values Varimax rotation: Principal Components Factor analysis:
Visuohaptic, olfactogustatory, auditory = 3 modalities Correlation w/ human associations word-modality (r = .74). H+V: Any object that can be touched can also be seen; G+O: Any object that can be tasted can also be smelt.
2 conceptual systems: linguistic, embodied (Louwerse & Connell, 2011) CRUCIAL: Why DS? DS and ES factor loadings from corpus used to predict switching RTs DS shifts + RTs ES shifts - RTs (R2 = .70)
Two systems: speed vs. precision (cf. Good-enough comprehension: Ferreira & Patson, 2007)
Just how do we know the modalities? Solid Ideal [Response] / Tasteless Moment [Response] VISUAL ?
VISUAL ?
Ind. Var. condition = non-shift ! Fundamental
?
Previous modality references Behavioural experiments: rated • Pecher et al. (2003); Van Dantzig et al. (2008); Lynott and Connell (2009, 2013) ERP experiments: mixed or unrated • Hald et al. (2011) borrows from various previous ratings • Collins et al. (2011) creates new relations, not rated
This study • New modality norms in Dutch Stimuli adapted from English (Lynott & Connell, 2009, 2013) 411 concepts, 336 properties N = 42 As before: ‘To what extent do you experience these properties/concepts via the senses of hearing, touch and vision?’ 1 – 5 Vector modality exclusivity (= perceptual strength) Vectors for each modality Dominant modality
Modality exclusivity norms 411 concepts and 336 properties, randomly selected modality-wise - most adapted into Dutch from Lynott and Connell (2009, 2013) - Interrater reliability and interitem consistency: α > .75 - all patterns replicating (also Louwerse & Connell, 2011)
cat
word
conc
Inflected
Main
aankondiging
de
a
0.480
4.285
0.714 2.428
+ Standard Devs. (3)
conc
aantekening
de
v
0.222
1.571
2.857 3.285
+ Length (2)
conc
aanvraag
de
v
0.319
2.000
0.777 2.444
prop
aards
aardse
v
0.142
1.333
1.333 2.000
prop
absorberend
absorberende
v
0.268
1.333
1.000 2.222
conc
academie
v
0.359
2.333
1.111 3.666
de
Exclusivity Auditory Haptic Visual
+ up to 20 variables:
De/Het
+ Frequency (3) + Distinctiveness (2) + Concreteness + Age of acquisition
conc
achtergrond
de
v
0.509
0.666
1.444 3.555
conc
administratie
de
v
0.388
1.888
0.888 3.222
conc
advies
het
a
0.408
4.555
1.333 2.000
conc
afbeelding
de
v
0.578
0.888
1.222 5.000
conc
afdeling
de
v
0.571
1.333
0.333 3.000
+ % Known
Modality exclusivity norms 411 concepts and 336 properties, randomly selected modality-wise - most adapted into Dutch from Lynott and Connell (2009, 2013) - Interrater reliability and interitem consistency: α > .75 - all patterns replicating (also Louwerse & Connell, 2011) Principal Components Analysis (English norms re-analyzed w/ 3 modalities) 3 PCs resulting for both Dutch and Eng. These are plotted into x and y Points = dominant modality of each word (a=audit, h=haptic, v=visual)
Modality exclusivity norms 411 concepts and 336 properties, randomly selected modality-wise - most adapted into Dutch from Lynott and Connell (2009, 2013) - Interrater reliability and interitem consistency: α > .75 - all patterns replicating (also Louwerse & Connell, 2011) Principal Components Analysis (English norms re-analyzed w/ 3 modalities) 3 PCs resulting for both Dutch and Eng. These are plotted into x and y Points = dominant modality of each word (a=audit, h=haptic, v=visual) Density = dispersion/consistence within each dominant modality
See below:
CONCEPTS
Notice different number of items within each dominant modality VISUAL monopolizes = experimental & naturalistic data (conversation) (Connell & Lynott, 2012; Wu, Gan, Huang, Zhou, Qian, 2015; Olofsson & Gottfried, 2015; San Roque et al., 2015) Not just out of visual presentation then
Notice different number of items within each dominant modality VISUAL monopolises = experimental & naturalistic data (conversation) (Connell & Lynott, 2012; Wu, Gan, Huang, Zhou, Qian, 2015; Olofsson & Gottfried, 2015; San Roque et al., 2015) Not just out of visual presentation then
More analyses…
Which bouba, which kiki ?
Sound symbolism? Yes, if Auditory scores best predictor of lexical properties
Sound symbolism: Auditory scores best predictor of lexical properties…
… or else contrary to main predictor
Main experiment:
Main experiment: additional condition • Louwerse and Connell had found fast responses unaffected by shift • Linguistic vs Embodied systems ?
• Current design caters to such a possibility • Previously…
ERP precedents Hald et al (2011): frontal, broad, reversed ‘N400’
Modality-match / Modality-mismatch
E
ERP precedents Hald et al (2011): frontal, broad, reversed ‘N400’
Caveats (also Collins et al., 2011) Modality-match / Modality-mismatch
E
* Modality based on three sources
* Lack of control across conditions: measure at noun, w/out semantic distance control (consider task)
* No distinction for diff. processes ?
ERP precedents
Current design
Hald et al (2011): front, broad, reversed ‘N400’
Listening in to separate processes + Group:
Modality-match / Modality-mismatch
E
E L
Fast vs Self-paced
