Semantic Spacesbased on Free Association that Predict MemoryPerformance MarkSteyvers StanfordUniversity
RichardMShiffrin . DouglasL. Nelson IndianaUniversity
Universit
oySouth f Florida
Submitted toJEPGeneral
Abstract Manymemory modelsrepresentaspectsow f ordssuchasmeaningb yvectorsofeature values,suchthatwordswithsimilarmeaningsarep lacedinsimilarregionsothe f semantic spacewhosedimensionsaredefinedbythevectorpo sitions.Methodsforconstructingsuch spacesincludethosebasedonscalingsimilarityra tingsforpairsowords, f andthosebased ontheanalysisofco-occurrencestatistics ofwor dsincontexts(Landauer&Dumais, 1997).WeutilizedaWordAssociationSpace(WAS), basedonascalingoalarge f data baseofreewordassociations:Wordswithsimilar associativestructureswereplacedin similarregionsotfhehighdimensionalsemanticsp ace.IncomparisontoLSAandother measuresbasedonassociativestrength,weshowedt hatthesimilaritystructureinWASis wellsuitedtopredictsimilarityratingsinrecogn itionmemory,percentagecorrect responsesincuedrecallandintrusionratesinfre erecall. WesuggestthattheWAS approachiuseful as andimportantnewtoolinthe workshopotheorists f studyingsemantic effects in episodicmemory. Anincreasinglycommonassumptionoftheories ofmemoryisthatthemeaningofawordcanbe representedbyavectorwhichplacesawordasa pointinamultidimensionalsemanticspace(Bower, 1967;Landauer&Dumais,1997;Lund&Burgess, 1996;Morton,1970;Norman,&Rumelhart,1970; Osgood,Suci,&Tannenbaum,1957;Underwood, 1969;Wickens,1972).Themainrequirementof spacesisthatwordsthataresimilarinmeaningar representedbysimilarvectors.Representingwords vectorsinamultidimensionalspaceallowssimple geometricoperationssuchastheEuclidiandistance ortheanglebetweenthevectorstocomputethe semantic(dis)similaritybetweenarbitrarypairsor groupsofwords.Thisrepresentationmakesit possibletomakepredictionsaboutperformancein psychologicaltaskswherethesemanticdistance betweenpairsogr roupsow f ordsiasssumedtoplay raole. Onerecentframeworkforplacingwordsina multidimensionalspaceisLatentSemanticAnalysis orLSA(Derweester,Dumais,Furnas,Landauer,& Harshman, 1990; Landauer & Dumais, 1997; Landauer,Foltz,&Laham,1998).Themain assumptionisthatsimilarwordsoccurinsimilar contextswithcontextdefinedaasnyconnectedset textfromacorpussuchasanencyclopedia,or
samplesoftextsfromtextbooks.Forexample,a textbookwithaparagraphabout “cats”mightalso mention“dogs”,“fur”,“pets”etc.Thisknowledge canbeusedtoassumethat“cats”and“dogs”are relatedinmeaning.However,somewordsareclearly relatedinmeaningsuchas“cats”and“felines”but theymightneveroccursimultaneouslyinthesame context. Suchwordsarerelatedprimarilythrough indirectlinksbecausetheysharesimilarcontexts. Thetechniqueofsingularvaluedecomposition (SVD)canbeappliedtothematrixofword-context co-occurrencestatistics.Inthisprocedure,thedi and indirect relationships between words and contextsinthematrixareanalyzedwithsimple matrix-algebraicoperationsandtheresultisahig dimensionalspaceinwhichwordsthatappearin similarcontextsareplacedinsimilarregionsotf space.LandauerandDumais(1997)appliedtheLSA approachtoover60,000wordsappearinginover 30,000contextsofalargeencyclopedia.More recently,LSAwasappliedtoover90,000words appearinginover37,000contextsoreading f materi thatanEnglishreadermightbe xposedtofrom3 st gradeupto1 yearocf ollegefromvarioussources suchastextbooks,novels,andnewspaperarticles. TheSVDmethodplacedthesewordsinahigh dimensionalspacewiththenumberofdimensions
such e as
of
1
rect
h he
al rd
text(ofwhichthereisavirtuallyunlimitedamoun as opposedtoratings providedbpyarticipants. Inthisresearch,wewillintroducearelatedbut newmethodforcreatingpsychologicalspacesthati basedonananalysisofalargefreeassociation database collected by Nelson, McEvoy, and Schreiber(1999)containingnormsforfirstassocia forover5000words.Thismethodplacesover5000 wordsinapsychologicalspacethatwewillcall WordAssociationSpace(WAS).Webelievesucha constructwillbeveryusefulinthemodelingof episodicmemoryphenomena.Atpresentitisnot clearhowwellepisodicmemoryperformancein recognitionorrecallispredictedbyLSAorHAL. Wedoknowthatwordassociationsplay animportant roleinepisodicmemorysinceihas t beenshowntha theassociativestructureowords f playsacentral inrecall(e.g.Bousfield,1953;Cramer,1968;Dees 1959a,b,1965;Jenkins,Mink,& Russell,1958),cue recall(e.g.Nelson,Schreiber,&McEvoy,1992)and priming(e.g.Canas,1990;seealsoNeely,1991),a recognition(e.g.Nelson,Zhang,&McKinney, submitted).Forexample,Deese(1959a,b)foundthat theinter-itemassociativestrengthofthewordsin studylistcanpredictthenumberow f ordsrecalled thenumberofintrusions,andthefrequencywith whichcertainwordsintrude.Basedontheclassic experimentsbyJamesDeese(1959b),Roedigerand McDermott(1995)revivedinterestintheparadigm thatisnowknownasthefalsememoryparadigm (e.g.Brainerd,&Reyna,1998;Brainerd,Reyna,& Mojardin, 1999; Payne, Elie, Blackwell, & Neuschatz,1996;Schacter,Verfaellie,&Pradere, 1996;Tussing&Green,1997;Shiffrin,Huber,& Marinelli, 1995). In the typical false memory experiment,participantsstudywordsthatareall semanticallyrelatedtoanon-studiedcriticalword Inasubsequentrecognitiontest,thecriticalword typicallyleadtohigher a falsealarmratethanth unrelated foils (and sometimes quite high in comparisontothatforstudiedwords).Infree a re test,participantsfalselyintrudethecriticalwor ratehigherthanunrelatedwords(andsometimesat ratesapproachingthoseforstudiedwords).These studiesshowthatepisodicmemorycanbestrongly influencedbsyemanticsimilarity. Inthepresentresearch,wewillcomparethe performanceofLSAwithWASinthreeepisodic memorytasks:recognitionmemory,freerecalland cuedrecall 2.Itwasexpectedthatthesimilarity structureinWASwouldbewellsuitedtopredict varioussemanticsimilarityeffectsintheseepisod memorytasks.Tofurtherourunderstandingofthe similaritystructureofWAS,weperformedseveral analyses.First,wecomparedthepredictionsoWAS f andLSAforthestrengthothe f associatesobtained
chosen between 200 and 400. The LSA representationhasbeensuccessfullyappliedto multiplechoicevocabularytests,domainknowledge tests and content evaluation (see Landauer & Dumais,1997;Landauer etal.1998). Anotherframeworkthathasbeenusedtoplace wordsinahighdimensionalsemanticspaceisthe Hyperspace Analogue to Language (Burgess, Livesay,&Lund,1998;Lund&Burgess,1996;see Burgess&Lund,2000foranoverview).TheHAL model develops high dimensional vector representationsforwordsbasedonaco-occurrence analysisolfargesamplesow f rittentext.For70,0 00 words,theco-occurrencestatisticswerecalculated in a10wordwindowthatwasslidoverthetextfroma corpusofover320millionwords(gatheredfrom Usenet newsgroups). For each word, the cooccurrencestatisticswerecalculatedforthe70,00 0 wordsappearingbeforeandafterthatwordinthe1 0 wordwindow.Theresulting140,000valuesforeach wordwerethefeaturevaluesforthewordsinthe HALrepresentation.Becausetherepresentationis basedonthecontextinwhichwordsappear,theHAL vectorrepresentationis also referred to as a contextualspace:wordsthatappearinsimilar contextsarerepresentedbysimilarvectors.TheHA L and LSA approach are similar in one major assumption:similarwordsoccurinsimilarcontexts . InbothHALandLSA,theplacementofwordsina highdimensionalsemanticspaceisbasedonan analysisoftheco-occurrencestatisticsofwordsi n theircontexts.InLSA,acontextisdefinedbya relativelylargesegmentoftextwhereasinHAL,th e contextis definedbwindow ay of 10words. LSAandHALarebothcorpusbasedmethods that contrast sharply with older methods of constructingsemanticspacesthatdependonhuman 1 judgmentssuchaspairwisesimilarityratings In . thismethod,participantsratethesemanticsimilar ity forpairsow f ords.Then,thosesimilarityratings are subjectedtomultidimensionalscalinganalysesto derivevectorrepresentationsinwhichsimilarvect ors representwordssimilarinmeaning(Caramazza, Hersch,&Torgerson,1976;Rips,Shoben,&Smith, 1973;Schwartz&Humphreys,1973).Whilethis methodisstraightforwardandhasledtointerestin g applications(e.g.Caramazzaeal; t Romney,Brewer, &Batchelder,1993),itisclearly impracticalfor large numberow f ordsasthenumberorfatingsthatmust becollectedgoesupquadraticallywiththenumber of stimuli.OnegreatadvantageoL f SAandHALover approachesdependingonpairwisesimilarityratings isthatalmostanynumberofwordscanbpelacedin a semantic/contextualspace.Thisispossiblebecause thesemethodsrelyuniquelyonsamplesofwritten
2
t)
s
tes
t role e, d nd
a ,
. atfor call daat
ic
in
thefreeassociationtask.BecauseWASisexplicitl basedonthefreeassociationnorms,itwasexpecte thattherewouldbeacloserelationshipbetween distanceinthepsychologicalspaceandassociative strengthinfreeassociation.Inanotheranalysis, issue of whether WAS captures semantic or associativerelationships(orboth)inpredicting similarityratingsiasddressed.Itwillbeargued isdifficulttomakeacategoricaldistinctionbetw purelyassociativeandpurelysemanticrelationship Finally,inthelastanalysisweanalyzetheabilit WAStopredictasymmetriesinthedirectionof association.Becausefreeassociationnormscontain manyexamplesow f ordsthatarestronglyassociated inonebutnottheotherdirection,predictingthes asymmetrieswithamodelsuchasWASpresentsan interesting challenge given that the similarity between twowords is definedtobseymmetric.
overlap in the distribution of free association responsesforasmallsetofwordsandarguedthat theseanalysescouldbeusedtolearnaboutthe mentalrepresentationofwords.Inthispaper,we capitalized onDeese’sideasoutilizing f thepatternof intercorrelationsinthefreeassociationnormsby placingalargenumberofwordassociationsina semanticspaceandthenusedthemtopredict semanticsimilarityeffectsinmemory.Insteadof factoranalyses,weusedthetechniquesofsingular value decomposition (SVD) and metric multidimensionalscalinganalysesthatareclosely relatedtofactoranalysis(e.g.,seeJohnson& Wichern,1998)butthatcanbeappliedtolarge volumes of data. TheSVDmethodhasbeensuccessfully appliedin LSA(e.g.Landauer&Dumais,1997)touncoverthe patterns of intercorrelations of co-occurrence statisticsforwordsappearingincontexts.Inthis research,theSVDmethodwasappliedonalarge databaseoffreeassociationnormscollectedby Nelsonetal.(1999)containingnormsoffirst associatesforover5000words.Theresultisthat the wordsareplacedinahighdimensionalsemantic space.BecauseoftheSVDmethod,thesimilarity betweenwordsinthispsychological spaceisbest expressedbythecosineothe f anglebetweenthetw o correspondingvectors(seeDerweestereat l.1990). Wealsoperformedadditionalanalysesbyapplying metricmultidimensionalscaling(MDS)methods(see Torgeson,1952;Schiffman,Reynolds &Young, 1981)onthenormssuchthattheEuclidiandistance
y d
the
thatit een s. yof
e
WordAssociationSpaces Deese(1962,1965)assertedthatfreeassociations arenottheresultofhaphazardprocessesandthat theyarisefromanunderlyingregularityinpreexisting associative connections. He laid the frameworkforstudyingthemeaningoflinguistic forms that can be derived by analyzing the correspondencesbetweendistributionsofresponses tofreeassociationstimuli:"Themostimportant propertyofassociationsistheirstructure -their patternsofintercorrelations"(Deese,1965,p.1). Deese(1962,1965)appliedfactoranalysestothe
RESPONSES
CUES
!"
# %
% #
$ () ( * & ' " (
345
678
9:5
EF G HIJK LM NO N PQ RS T U VMW UXY V RPZ
5 : ;:< => =
@4
8
AB=
+, ,, - + ,, , . / 0 1 2-
B
6
=C :
?
>=
7?
6 7 =
7?
4 ; > 94 3 : D
Nelson,McEvoy,&Schreiber(1999)
Figure 1.Illustrationothe f creationoWord f Association largedatabaseofreeassociationnorms,wordsare similar associativerelationships areplacedisnim
Spaces(WAS).Byscalingthewordassociationsoaf placedinahighdimensionalsemanticspace.Words ilar regions of thespace.
3
with
Table1. Overview of Methodsfor Quantifying Associative/Seman
Method
Basedon:
Associations
S
ticsimilarity
(Dis)similarity Measure
(1)
forward : plusbackwardstrength
S(2)forward : plusbackwardplus twostepassociativestrengths WAS
LSA
Vocabulary Size
S(1)
5000+
S(2)
5000+
svdoSf
(1)
cosineoangle f
2500
svdoSf
(2)
cosineoangle f
2500,5000+
metricmdsof T (see text)
Euclidiandistance
svdoword-context f matrix oencyclopedia f
cosineoangle f
svdoword-context f matrix otasa f documents cosineo
betweentwowordsintheresultingpsychological spacecapturesthedissimilaritybetweentwowords. Wewillrefertothegeneralmethodofplacingthe wordsinaspaceaw s ellasthespaceitselfaW s or AssociationSpace(WAS).Foranoverviewofallthe differentmethodstoquantifyassociative/semantic similarity itnhis research,seeTable1. ByapplyingscalingmethodssuchasSVDand metricMDSonthenorms,wehopetouncoverthe latentinformationavailableinthefreeassociatio normsthatisnotdirectlyavailablebyinvestigati simplemeasuresforassociativestrengthsbasedon thedirectandindirectassociativestrengthsthrou shortchainsofassociates(e.g.,Nelson&Zhang, 2000).ThebasicapproachisillustratedinFigure Thefreeassociationnormswererepresentedin matrixformwiththerowsrepresentingthecuesand thecolumnsrepresentingtheresponses.Theentries inthematrixarefilledbysomemeasureof associativestrengthbetweencuesandresponses.By applyingscalingmethodsonthematrix,wordsare placedinahighdimensionalspacesuchthatwords withsimilarassociativepatternsareplacedinsim regions of thespace. Intotal,morethan6000peopleparticipatedinthe collectionofthefreeassociationdatabaseoN f els etal.(1999).Anaverageof149(SD=15) participants were eachpresented with100-120 Englishwords.Thesewordsservedascues(e.g. “cat”)forwhichparticipantshadtowritedownthe firstwordthatcametomind(e.g.“dog”).Foreach cuetheproportionofsubjectsthatelicitedthe responsetothecuewascalculated(e.g.60% respondedwith“dog”,15%with“pet”,10%with “tiger”,etc).
angle f
60,000+ 90,000+
Scaling bySingularValueDecomposition WewillfirstexplainhowSVDisappliedtothe normsandthencontinuetothemetricMDSmethod. ThemethodofSVDcanbeappliedtoanymatrix containing some measure of strength or cooccurrencebetweentwowords.Althoughmany differentwayshavebeenproposedtocalculatean indexofassociativestrengthbetweentwowords (e.g.,Marshall&Cofer,1963;Nelson&Zhang, 2000),wewillrestrictourselvestotwosimple measuresoassociative f strength.LetA represent the ij proportionosfubjectsthatgavetheresponse to j the cuei.ThesimplestmeasurewouldbetotakeA ij itself.Inthenorms,theassociativestrengthsA are ij often highly asymmetric where the associative strengthinonedirectionistrongwhileiis twea kor zerointheotherdirection.EventhoughSVDcanbe easilyappliedtoasymmetricmatrices,theresults are moreinterpretablewhenitisappliedtosymmetric matrices3.Inalatersectioninthepaper,wewill show that symmetrizing the norms does not necessarilymeanthatasymmetriesintheword associationscannotbepredicted.Therefore,inour firstmeasurefor associativestrength wteake:
d
n ng gh 1.
ilar
on
Sij(1) = Aij + A ji S(1)ijisequivalenttoaddingforwardstrengthto backward strength. This measure is ofcourse (1) (1) symmetricsothatS ji.Thismeasureis ij=S indexedby(1)becauseitbasedononlythe direct associationbetweeniandjandinvolvesonlyone associativestepgoingfromtioj.Inthenormsof
4
Nelsonetal.(1998),subjectswereonlyallowedto givethefirstresponsethatcametomind.Theseco nd strongestresponseinonesubjects’mind mightbe elicitedbyanothersubjectorimight t notbeelic ited atallifthefirstresponseisastrongassociate. (1) Therefore,theS measuremightbeunderestimating the associative strength between two words especiallyincaseswherethemeasureizsero(Nels on etal.,1998).Inthesecondmeasureforassociativ e strength,wetake:
associative strengths can uncover the latent (1) relationshipsbetweenwords.IntheSVDofS , wordsthatarenotdirectassociatesoeach f other can stillberepresentedbysimilarvectorsiftheir (2) associatesarerelated.IntheSVDofS words , that not directly associated or indirectly associated throughoneintermediateassociate,canstillbe representedbysimilarvectorsithe f associatesof the associatesofthewordsarerelated.Inotherwords , thewholepatternofdirectandindirectcorrelatio ns betweenassociationsistakenintoaccountwhen placing words in thesemanticspace. Animportantvariable(whichwewillcallk)is thenumberofdimensionsothe f space.Onecanthin k ofkasthenumberofeaturevaluesforthewords. Wevaried kbetween10and500.Thenumberof dimensionswilldeterminehowmuchtheinformation 4 ofthefreeassociationdatabaseicsompressed With . toofewdimensions,thesimilaritystructureofthe resultingvectorsdoesnotcaptureenoughdetailof theoriginalassociativestructureinthedatabase. Withtoomanydimensionsorthenumberof dimensionsapproachingthenumberofcues,the informationinthenormsisnotcompressedenough sothatwemightexpectthatthesimilaritystructu re ofthevectorsdoesnotcaptureenoughothe f indir ect relationshipsintheassociationsbetweenwords.In theanalysesopredicting f performanceinvariety a of tasks(recognition,freeandcuedrecall),wewill showthatalthoughtheoptimalvalueokdf ependso n thespecifictask,intermediatevaluesofkbetween 200and500areappropriatefor this method.
Sij( 2 ) = Sij(1) + ∑ S ik(1) S kj(1) k
Thisequalstheforwardplusbackwardplus mediatedstrengththroughotherassociates. Notethat thismeasureinvolvesthedirectstrengthbetweeni andjaswellastheindirectstrengthbysumming overallpathsfromtioktoj,theproductofthe symmetricassociativestrengthsbetween and i kan , d kandj.Theseindirectassociativestrengthsinvol ve thetwostepprobabilitiesogoing f from to and ij vice versaandhencetheindex(2).Researchhasshown thattheindirectassociative strengthsplayarolein cuedrecall(Nelson,Bennet,&Leibert,1997;Nelso n &Zhang,2000)andrecognition(Nelson,Zhang,& McKinney,submitted).Forexample,Nelson& Zhang(2000)foundthatincludingtheindirect associativestrengthsinameasureforassociative strength significantly increases the explained varianceitnheextra-listcuedrecalltask. We applied SVD separately on these two measuresoaf ssociativestrength.Theresultoef ac h SVDistheplacementofwordsinhigh a dimensional space,sothatwordsthathavesimilarassociative structuresarerepresentedbysimilarvectors.Beca use oftheSVDmethod,andbasedonworkinLSA(see Derweestereat l.,1990),asuitablemeasureforth e similaritybetweentwowordsisthecosineofthe anglebetweentwowordvectors.Let
ScalingbyMetric-MDS AninterestingcomparisonforthetwoWAS spacesbasedonSVDwouldbetoconstructametric spaceinwhichthedistancebetweentwowords,i.e. , theirdissimilarity,canbemeasuredbytheEuclidi an distancebetweentheirvectors.MetricMDSisa classicmethodforplacingstimuliin space a such that theEuclidiandistancebetweenpointsinthespace approximates the Euclidian distances in the dissimilaritymatrix(seeTorgeson,1952;Schiffman , Reynolds&Young,1981).Inordertoapplymetric MDS,estimatesareneededforthedistancebetween anytwowords.Infact,allnon-diagonalentriesin the matrixhavetobefilledwithsomeestimateforthe distancebetweenwordssincenomissingvaluesare 5 allowedinthemethod This . raisestheproblemhow toestimatethedistancebetweeniandjwhenthe (1) 6 associativestrength ameasured s S by zero . ij is Inoursolutionotfhisproblem,wewere inspired bynetworkmodelsforproximitydata(e.g.Cooke, Durso,&Schvaneveldt,1986;Klauer,&Carroll, 1995). In these network models, dissimilarity betweentwostimuliiscalculatedbytheshortestp ath
v X i represent
thevectorinWASforwordiThe . similaritybetwee words and i is calculated j by:
n
v v Xi ⋅ X j similarity (i, j ) = cos(α ) = v v Xi X j where
v X isthelengthofthevectorand
v v X i ⋅ X j representstheinnerproductbetweenvectors and i jTwo . wordsthataresimilarinmeaningotr havesimilarassociativestructuresareexpectedto havehighsimilarityasdefinedbythecosineotfh anglebetweenthetwowordvectors.TheSVDofthe
hat e
5
betweentwonodesinagraph.Inthisresearch, canusethewordassociationnormsasdefininga graph:twowordsarelinkedbyanedgeitfheyhave nonzeroassociativestrengths.Wewillusethe symmetricS (1)associativestrengthsbecauseinthe (1) graphdefinedbyS itis ,possibletoreachanyword fromanyotherwordinthegraph(infact,the maximumnumberofstepsbetweenanypairofwords isfour).Thedistancebetweentwowordswillbe definedasthenegativelogarithmoftheproductof theassociativestrengthsalongtheshortestpathi (1) networkdefinedbyS .Thisisequivalenttothe (negative)sumofthelogsothe f associativestren along theshortestpath:
[
we
SinceDeese’s(1959b)classicstudyonintrusions infreerecall,many studieshaveshownthatmemory errorsareinpartbasedonsemanticoverlapbetwee n theresponseandthecontentsofmemory.Inthis research, we introduced WAS as a way of quantifyingthesemanticsimilaritybetweenwords thatmighthelpinpredictingthesememoryerrors. Thedatafromarecognitionexperiment,Deese’s originalfreerecallexperimentandacuedrecall experimentweretakenasabasisfortestingvariou s modelsthatcapturesemanticsimilarity.Wetested threeWASbasedmeasuresforsemanticsimilarity. (1) ThefirsttwowerebasedontheSVDofS the , one stepsymmetricassociativestrengths,andontheSV D ofS (2)the , oneplusthetwostepassociativestrengths involvingindirectassociativestrengths.Inthese two semanticspaces(aswellasinLSA)thecosineotf he anglebetweentwowordsexpressesthesimilarity betweentwowords.ThelastWASmeasurewas basedonmetric-MDSoftheshortestpathassociativ e strengths.Inthisspace,theEuclidiandistance betweentwowordvectorsistakenasameasurefor thedissimilaritybetweentwowords. TheseWAS scalingsolutionswerecontrastedwiththe(unscale d) (1) associativestrengthsS andS (2)thatweretakenas controlcomparisons.WealsotestedtwoLSAbased measures,onewasbasedona corpusofan encyclopediaandanotheroncorpus a calledtasath at includedreadingmaterialthatanEnglishreader rd st mightbeexposedtofrom3 gradeupto1 yearof college.Thedifferentmethodsthatarecomparedar e listediT n able1.
nthe gths
Tij = − log (S ik(1) S kl(1) ⋅ ⋅ ⋅ S qj(1) ) = − log S ik(1) + log S kl(1) + ... + log S qj(1)
]
Here,theshortestpathbetweenwordsand i ijs from to ikto through l otherwordstoqand finallyj. Withthisdistancemeasure,wordpairswithweakor longassociativepathsareassignedlargedistances whereaswordpairswithshortosr trongassociative pathsareassignedsmalldistances.ThedistancesT werecalculatedforallwordpairs intheword associationdatabase.Then,thesedistanceswere scaledbymetric-MDS.Theresultisthatthewords areplacedinamultidimensionalspaceandthe dissimilarityordistancebetweentwowordsis expressedbytheEuclidiandistancebetweenthetwo corresponding wordvectors:
2 distance(i, j ) = ∑ (X ik − X jk ) k
Predicting Semantic Similarity EffectsinMemory
ij
1/ 2
Becauseofcomputationalconstraints,iw t asnot possibletoapplymetric-MDStothefullmatrixT containingthedistancesforallwordpairs.Instea wechose2500wordsfromtheoriginal5018words inthewordassociationdatabase.Thewordsinthis smallersetincludedwordsappearinginvarious experimentslistedinthenextsectionandincluded selectionorandomly f chosenwordsfromtheorigina set.TheSVDprocedurewasappliedonboththe smallermatrixof2500wordsaswellasthesetof 5018words. AswiththeSVDprocedure,thechoiceofthe numberofdimensionsinthespaceisimportant. Havingtoofewdimensionsotroomanydimensions might lead to suboptimal performance when predictingperformanceinvariousmemorytasks.As withtheSVDscalingprocedure,thenumberof dimensions was variedbetween 10and500.
RecognitionMemory:SemanticSimilarity Ratings Inastudybythefirsttwoauthors(Steyvers& Shiffrin,submitted:Experiment1),89subjects studied144wordsthatcontained18semantic categoriesof 5wordseach.Basedonastudyby BrainerdandReyna(1998),subjectsgavetworating foreachof100testitems.Inonerating,theywer instructedtojudgewhethertheitemwasoldornew and were told to judge semantically similar distractorsas“new”.Inanotherrating,theywere instructedtorate(onasixpointscale)how semanticallysimilartheitemwastothestudied items.Wefocusedonthesemanticsimilarityrating forthenewitemsfromthisstudy.Foreachsubject the72newtestitemswererandomlyselectedfroma largerpoolof144words.Anaverage (SD=4.87)subjectsratedthesemanticsimilarityfo eachofthe144wordsthatmightappearasnew
d,
a l
6
s e
s , of44 r
7 wordsinthetestlist The . semanticsimilarity ratings aretheoreticallyinterestingbecausetheycanbeu totestmodelsosemantic f similarity.Subjectsmer havetorememberhowsimilartheitemwastothe studieditemswithoutbeingforcedtogiveold-new judgmentsthatmightbemoreinfluencedbyvarious strategicretrievalfactors(suchaw s ordfrequency previous retrievals). Manymemorymodelsassumethatarecognition memoryjudgmentiproduced s bycalculatingthe globalfamiliarityinvolvingthesummedsimilarity betweenthetestitemandtheepisodictracesin
memory(e.g.MinervaII,Hintzman1984,1988; SAM,Gillund&Shiffrin,1984).Morerecently, ShiffrinandSteyvers(1997,1998)andMcClelland & Chappell (1998) have proposed recognition memorymodelsthatproducerecognitionjudgments withBayesiandecisionprocesses.McClelland& Chappell(1998)proposedthatthebestmatch(i.e., maximumsimilarity)betweenthetestitemandthe episodictracesinmemoryformsthebasisforthe recognitionjudgment.Shiffrin&Steyvers(1998) showedthatintheBayesianframework,amaximum similarityprocessproducedresultsverysimilarto
sed ely
or
Recognition SemanticSimilarityRatings
FreeRecall P( - intrusion)
0.8
0.9
0.7
0.8
0.6
0.7 0.6
0.5 R
a
0.5
0.4
0.4 0.3 0.3 0.2
0.2
0.1
0.1
0.0 0
100
200
300
400
500
0.0
2500
0
100
200
300
400
500
2500
500
2500
SimilarityRatings
CuedRecall P(correct) 0.6 0.4 0.5 0.3
R
0.4 0.3
0.2 0.2 0.1 0.1 0.0
0.0 0
100
200 300 400 Dimensionality WAS SVD - S WAS SVD - S WAS MDS -
(1) (2)
500
0
2500
100
LSA encyclopedia LSA tasa -
200 300 400 Dimensionality S(1) S(2)
Figure 2.Correlationsodf ifferentmeasuresosfemanticsimilarityfordiffer entdimensionalities.Dataare takenfromrecognitionmemory,cuedrecall,freere call,andsimilarityratingsexperiments.Seetext details.
7
for
procedureandthesewerebeneficialinpredictingt he ratings from this recognition memory experiment. Themetric-MDSsolutionshowsquiteadifferent patternofresultsthantheSVDsolution.Thebest correlationwasobtainedwith20-40dimensions whichismuchlowerthanthenumberodf imensions typicallyneededintheSVDsolutionsoeither f WAS orLSA.AlthoughthebestcorrelationformetricMDSwas0.6asopposedto0.7fortheSVDbased solutions, it is interestingthat relativelygood performancecanbeachievedinsemanticspacesthat areoflowdimensionality.Althoughspecifyingwhy thiseffectoccursios utsidethescopeothis f pap er,it couldberelatedtotheestimatesinvolvingthe shortest associative path between words. As describedintheprevioussection,inordertoappl y metric-MDS,estimateswereneededforthedistances between allwordpairsinthevocabulary.The shortestassociativepathdistancewasproposedto meetthisrequirement;estimateswereevengenerate d forwordpairsthatwerenotassociateddirectlyor evenindirectlythroughachainoftwoassociates. In SVD,nosuchestimatesarerequiredandthoseentri es wereleftatzero.Itispossiblethen,thatthefi llingin processoall f wordpairdissimilaritiesbythesho rtest associativepathdistanceshelpedintheglobal in thesemanticspace. placementof allwords OfthetwocorperainLSA,thetasacorpusledto muchbetterperformancethantheencyclopedia corpus.Thisdifferenceinsotsurprisingsincethe tasa corpusincludesmaterialthatreflectsmuchmore closelythereadingmaterialanEnglishreaderis exposedtowhichinturnmightleadtosemantic spacesthataremorepsychologicallyplausiblein termsofpredictingsemanticsimilarityeffectsin recognitionmemory.ComparingWAStoLSA,it becomesclearthatWASleadstomuchhigher correlations than LSA. We will leave the interpretation othis f finding for thediscussion.
summedsimilarityprocess.Inthisresearch,ourai m isnottotestthesemodelsspecificallybuttouse and simplify the underlying mechanisms to predict semanticsimilarity ratings. BasedontheglobalfamiliarityandBayesian recognitionmemorymodels,thesemanticsimilarity ratingsintherecognitionmemoryexperimentshould havebeencorrelatedwiththe sumor maximumof thesimilaritybetweenthetestitemandallstudy words. To facilitate comparisons, all expected negativecorrelationswereconvertedtopositive correlations. Becausetheresultswereverysimilarfor thesumandmaximumcalculations,wewilllistonly theresults for themaximum calculation. ThetopleftpanelofFigure2showsthe correlations between maximum similarity and numberofdimensions(10-500)forthethreeWAS andtwoLSAbasedmeasures.FortheSVDbased semantic spaces, increasing the number of dimensionsineitherWASorLSAincreasesthe correlation generally up to around 200-300 dimensions.ForWAS,anadditionaldatapointwas plottedfor2500dimensionswhichisthemaximum number of dimensions given that the matrix containedonly2500words.Althoughaspacewas availableforall5018wordsinthefreeassociatio n database,thevocabularywasrestrictedto2500 wordsinordertoabletocompareittothemetric MDSsolutions(seeprevioussection).Thisdatapoi nt for 2500 dimensions was included because it representsthecasewherenoneoftheindirect relationshipsinwordassociationmatrixareexploi ted and as such, no dimensionality reduction is performed.Ascanbeobserved,thecorrelationis lowerfor2500dimensionindicatingthatsome dimensionalityreductionisneededtopredictthe semanticsimilarityratings.Also,theSVDbasedon S(2)ledtobettercorrelationsthantheSVDbasedon S(1)This . impliesthataddingtheindirectassociation s inameasureforassociativestrengthhelpsin predictingrecognitionmemoryperformance.Thetwo horizontallinesintheplotindicatethecorrelati on (1) whentheassociativestrengthsS andS (2)areusedas ameasureforsemanticsimilarity.Thecorrelation is (2) higherforS thanS (1)whichagainimpliesthatin recognition memory, the indirect associative strengths help in predicting performance. (2) Interestingly,theSVDscalingofS gavehigher (2) correlations than associative strengths S (2) themselves.EventhoughS includestheforward, backwardandalltwostepassociativestrengths, applyingtheSVDandreducingtheredundanciesin (2) thematrixoSf helpedtoincreasethecorrelation.In otherwords,theindirectrelationshipsandpattern os f correlationsthatgobeyondthoseofthetwostep associativestrengthswereutilizedbytheSVD
PredictingExtralistCuedRecall In extra-list cued recall experiments, after studyingalistow f ords,subjectsarepresentedwi cuesthatcanbuesedtoretrievewordsfrom thest list.Thecuesthemselvesarenovelwordsthat notpresentedduringstudy,andtypicallyeachword isassociativelyand/orsemanticallyrelatedtoone thestudiedwords.Thedegreetowhichacueis successfulinretrievingaparticulartargetwordi measureofinterestbecausethismightberelatedt theassociative/semanticoverlapbetweencuesand theirtargets.Researchinthisparadigm(e.g.,Nel &Schreiber,1992;Nelson,Schreiber,&McEvoy, 1992;Nelson,McKinney,Gee,&Janczura,1998; Nelson & Zhang, 2000) has shown that the associativestrengthbetweencueandtargetisone
8
th udy were of sa o son
importantpredictorforthepercentageofcorrectly recalledtargets.Therefore,weexpectthattheWAS similaritybetweencuesandtargetsarecorrelated withthepercentagesofcorrectrecallinthese experiments.Weusedadatabasecontainingthe percentagesofcorrectrecallfor1115cue-target 8 pairsfrom over29extralistcuedrecallexperiments fromDougNelson’slaboratory(Nelson,2000; Nelson & Zhang,2000). Thecorrelationsbetweenthevariousmeasuresfor semanticsimilarityandtheobservedpercentage correctrecallratesareshowninthebottomleftp anel ofFigure2Overall, . theresultsarevery similar tothe results obtained for the recognition memory (2) experiment.TheWASspacebasedonS ledto (1) betterperformancethantheWASspacebasedonS . (2) Also,theassociativestrengthsS leadstobetter (1) performancethentheS associativestrengths.These findingsare consistentwithfindingsbyNelson& Zhang(2000)thatshowthattheindirectrelationsh ips inwordassociationnormscanhelpinpredicting cuedrecallperformance.Interestingly,theplotal so (2) showsthattheWASspacebasedonS does (2) somewhatbetterthantheassociativestrengthsS it wasbasedon.Thisadvantageimplies thatapplying dimensionalityreductiontomakegreateruseofthe indirectassociativeconnectionshelpedinpredicti ng cuedrecall.Finally,aswiththerecognitionresul ts, theWASspacecorrelatesbetterwithcuedrecallth an LSA. PredictingIntrusion Rates inFreeRecall InaclassicstudybyDeese(1959b),thegoalwas topredicttheintrusionratesow f ordsinfreerec Fiftyparticipantsstudiedthe12strongestassocia toeachof36criticallureswhilethecriticallur themselveswerenotstudied.Inafreerecalltest, somecriticallures(e.g.“sleep”)werefalselyrec about40%ofthetimewhileothercriticallures(e “butterfly”)wereneverfalselyrecalled.Deesewas abletopredicttheintrusionratesforthecritica onthebasisotfheaverageassociativestrengthfr thestudiedassociatestothecriticalluresand obtainedacorrelationofR=0.80.BecauseDeese couldpredictintrusionrateswithwordassociation norms,theWASvectorspacederivedfromthe associationnormsshouldalsopredictthem.Critica itemswithhighaveragesimilarity(orlowaverage distance)tothelistwordsinthesemanticspace shouldbemorelikelytoappearaisntrusionsinfr recall.Theaveragesimilarity(averagedistance)w computedbetweeneachcriticallurevectorandlist wordvectors,andthecorrelationswerecomputed betweenthesesimilaritiesandobservedintrusion rates.
ThetoprightpanelinFigure2showstheresults. Thepatternofresultsisquitedifferentthan the patternoresults f foreitherrecognitionocued r r ecall. (1) Thebestcorrelationof0.82wasobtainedwithS , the sumofbackward and forward associative strength.Thisresultisverysimilartothecorrel ation of0.80Deeseobtainedwithhiswordassociation norms. Interestingly, the plot shows that any manipulationthatincludestheindirectassociation s leadstoworseperformancethanusingthedirect (2) associationsonly.TheWASspacebasedonS now (1) doesworsethantheWASspacebasedonS a, nd eitherspacecorrelatesmorepoorlythanwhenusing (1) theassociativestrengths S andS (2)themselves. Thesefindingsimplythatdirectassociative strengthsarethebestpredictorsoifntrusionrate isn freerecall. Oneexplanationforthisfindingirs elated toimplicitassociativeresponses(IAR’s).Underwoo d (1969)hasarguedthatduringstudy,thewords associatedwiththestudywordsarethoughtofand mightbestoredinmemoryaasnimplicitassociativ e response.InDeese’sstudy,itislikelythatIAR’s weregeneratedbecausethecriticallureswereall stronglyassociatedtothelistwords.Therefore, duringrecall,thewordsthatwereactuallypresent ed andwordsthatwerethoughtofduringstudymightb e confusedleadinginsomecasestodramaticintrusio n rates. Because free associations measure what responsesarethoughtofgivenspecificcues,the directassociativestrengthscanbearguedtobego od predictorsofthestrengthofimplicitassociative responses andsubsequentintrusion rates.
all. tes es
Similaritiesanddifferencesbetween WASandfree associationnorms Because WAS places words in a multidimensionalspacebasedonthepatternointe f correlationsinfreeassociation,thesimilarities differencesofWASandthefreeassociationnorms needstobedetermined.Intheprevioussection,it wasestablishedthatinrecognitionandcuedrecall WASleadstosomewhatbettercorrelationswith observedresultsthantheassociativestrengthsit basedon.Inthissection,thesimilaritystructure WASisinvestigatedandcomparedtothefree associationnorms.First,weinvestigatethedegree whichtheneighborhoodsimilaritystructureinWAS canbue sedtopredicttheorderofresponsestreng infreeassociation.Then,weaddresstheissueof whatkindofrelationshipWAScaptures:semantic, associativeorboth.Finally,weassesswhether asymmetriesinthedirectionofassociationcanbe predictedbytheneighborhoodsimilaritystructure WAS.
alled .g. lures l om
l
ee as
9
rand
, was of to ths
in
PredictingtheOutputOrderoFree f Association Norms BecausethewordvectorsinWASarebased explicitlyonthefreeassociationnorms,itisof interesttocheckwhethertheoutputorderof responses(intermsofassociativestrength)canbe predictedbyWAS.Tosimplifytheanalysis,the resultsareonlypresentedforWASbasedonS tookthisspacebecauseiperformed t wellinallth memorytasksandbecausewehadasolution availableforall5000+wordsappearinginthefree associationnorms(themetricspacewaslimitedto vocabulary o2500 f words).
rankofthesimilarityoaspecific f cue-responsep air wascomputedbyrankingthesimilarityamongthe similaritiesofthespecificcuetoallotherpossi ble nd responses.Forexample,theword‘crib’isthe2 closestneighborto‘baby’inWAS,so‘crib’hasa rankof2forthecue‘baby’.Inthisexample,WAS hasput‘baby’and‘crib’closertogetherthanmigh t beexpectedonthebasisofreeassociationnorms. Averagedacrossthewordsinthecorpus,Table2 givesforeachothe f firsttenrankedresponsesin free association(thecolumns)themedianrankinWAS. Themedianwasusedtoavoidexcessiveskewingof theaveragebyafewhighranks.Anadditional variablethatwastabulatedinTable2isk,the number of dimensions of WAS. TherearethreetrendstobediscernedinTable2. First,itcanbeobservedthatfor400dimensions, the strongestresponsestothecuesinfreeassociation normsarepredominantlytheclosestneighborstoth e cues inWAS.Second,responsesthathavehigher ranksinfreeassociationhaveonaveragehigher ranksinWAS.However,theoutputranksinWAS areinmanycasesfarhigherthantheoutputranks in freeassociation.Forexample,with400dimensions, thethirdrankedresponseinfreeassociationhasa medianrankof10inWAS.Third,forsmaller dimensionalities,thedifferencebetweentheoutput order in freeassociation andWASbecomes larger. Tosummarize,givenasufficientlylargenumber of dimensions, the strongest response in free associationisrepresented(inmostcases)asthe closestneighborinWAS. Theothercloseneighbors inWASarenotnecessarilyassociatesinfree association(atleastnotdirectassociates). Wea lso analyzedthecorrespondencebetweenthesimilaritie s intheLSAspace(Landauer&Dumais,1997)based onthetasacorpuswiththeorderofoutputinfree association. AscanbeobservedinTable2the , ra nk oftheresponsestrengthothe f freeassociationno rms clearlyhasaneffectontheorderingosimilariti f esin LSA:strongassociatesarecloserneighborsinLSA than weak associates. However, the overall correspondencebetweenpredictedoutputranksin LSAandranksinthenormsisweak.Theoverall weakercorrespondencebetweenthenormsand similaritiesfortheLSAapproachthantheWAS approachhighlightsoneobviousdifferencebetween thetwoapproaches.BecauseWASibsasedexplicitly onfreeassociationnorms,iitsexpectedandshown herethatwordsthatarestrongassociatesareplac ed closetogetherinWASwhereasinLSA,wordsare placedinthesemanticspaceinawaymore independentfrom thenorms. Togetabetterideaofthekindsofneighbors words haveiW n AS,in Table3we , listthefirstfi ve
(2)
We . ree
a
Table 2 Median . Rank othe f Output-order in WAS andLSA of Response WordstoGiven Cuesfor the10 StrongestResponses in theFreeAssociation Norms. Rank oResponse f iF n reeAssociation k
1
2
3
4
5
6
7
8
9 10
WordAssociation Space(WAS) 10
86 187 213 249 279 291 318 348 334 337
50
13
36 49 62 82 98 106 113 125 132
100
6
17 26 36 43 62 65 73 78 85
200
3
8 15 20 28 39 40 48 56 58
300
2
6 12 16 21 31 35 38 43 49
400
1
5 10 14 19 27 32 35 38 44 LatentSemanticAnalysis(LSA)
10
733 798 846 845 922 897 903 920 939 955
50
231 313 371 422 475 494 526 510 559 583
100
115 193 256 307 359 384 411 413 451 463
200
63 125 185 225 285 319 347 339 389 395
300
46
99 159 197 254 294 321 324 375 374
400
37
90 149 185 239 278 310 308 349 366
theseanalyses,WASwasbasedotnhesvdof Note:In S(2) and LSA wasbasedotnhetasa corpus
Wetookthe10strongestresponsestoeachothe f cuesinthefreeassociationnormsandrankedthem accordingtoassociativestrengths.Forexample,th response‘crib’isthe8thstrongestassociateto‘ inthefreeassociationnorms,so‘crib’has rank a forthecue‘baby’.UsingthevectorsfromWAS,the
e baby’ of8
10
Table3. TheFiveNearest Neighborsin WAS for theFirst 40 (1954)Norms.
Cue
1
cuesin theRussell& Jenkins
Neighbor 3
2
5
4
afraid
scare(1)[7]
fright(4)[14]
fear(2)[1]
scared[2]
ghost(5)[106]
anger
mad(1)[1]
Angry
rage(5)[4]
enrage
fury[21]
baby
child(1)[2]
crib(8)[13]
infant(6)[7]
cradle
diaper(13)
bath
clean(2)[1]
soap(7)[3]
water(3)[2]
dirty[7]
suds[49]
beautiful
pretty(1)[2]
ugly(2)[1]
cute[39]
girl(4)
flowers[10]
bed
sleep(1)[1]
tired(11)[13]
nap
rest[5]
doze
bible
god(1)[1]
church(3)[3]
religion(4)[4]
Jesus(5)[8]
book(2)[2]
bitter
sweet(1)[1]
sour(2)[2]
Candy
lemon(5)[7]
chocolate[4]
Black
white(1)[1]
Bleach
color(3)[7]
dark(2)[2]
minority
blossom
flower(1)[1] petals[46]
Rose(5)[7]
tulip
daisy
blue
color(5)[4]
red(3)[2]
Jeans
crayon
pants
boy
girl(1)[1]
Guy
Man(4)[2]
woman
nephew[54]
bread
butter(1)[1] toast[19]
rye[26]
loaf(3)[16]
margarine
butter
bread(1)[1]
rye
peanut
margarine(2)[34]
butterfly
bug(15)[10] insect(6)[2]
fly(4)[5]
roach[76]
beetle
toast(6)[18]
cabbage
green(4)[7]
food(10)[4]
vegetable(2)[3] salad(12)[5]
vegetables
carpet
floor(2)[2]
tile(15)
rug(1)[1]
ceiling
sweep[14]
chair
table(1)[1]
seat(4)[4]
sit(2)[2]
couch(3)[20]
recliner
cheese
cracker(2)
cheddar(6)[23]
Swiss(7)[19]
macaroni[39]
pizza
child
baby(1)[1]
kid(2)[7]
adult(3)[3]
young(8)[6]
parent(6)[11]
citizen
person(1)[3] country(3)[5]
people[7]
flag[12]
American(2)[4]
city
town(1)[1]
state(2)[3]
country(9)[4]
New York(4)
Florida
cold
hot(1)[1]
ice(2)[5]
warm(6)[3]
chill
pepsi
comfort
chair(3)[1]
Table
seat
couch(2)[26]
sleep[7]
command
tell(4)[7]
army(5)[2]
rules
navy[17]
ask[22]
cottage
house(1)[1]
home(4)[4]
cheese(2)[3]
cheddar
Swiss
dark
light(1)[1]
Bulb
night(2)[2]
lamp
day
faucet
pool[53]
splash
medical[83]
stethoscope[21]
nap
tired[92]
deep
water(3)[3]
ocean(2)[6]
doctor
nurse(1)[1]
physician(5)[15] surgeon(6)
dream
sleep(1)[1]
fantasy(4)[19]
bed[7]
eagle
bird(1)[1]
Chirp
bluejay
earth
planet(2)[8] mars[14]
Jupiter[97]
Venus[50]
eating
food(1)[1]
eat[30]
hungry(3)[4]
restaurant[75] meal[30]
foot
shoe(1)[1]
sock[16]
fruit
orange(2)[3] apple(1)[1]
nest(10)[5]
sparrow[30] Uranus
toe(2)[3]
sneaker
leg(5)[4]
juice(9)[12]
citrus[35]
tangerine[55] pretty(4)[6]
girl
boy(1)[1]
guy(6)
man[9]
woman(3)[2]
green
grass(1)[1]
lawn[41]
cucumber
vegetable[76] spinach[76]
hammer
nail(1)[1]
tool(2)[7]
wrench
screwdriver
pliers[21]
hand
finger(1)[2] arm(3)[3]
foot(2)[1]
leg(13)[11]
glove(4)[4]
hard soft(1)[1] easy(3)[3] Note: numbers in parentheses and squarebracketsi al.(1998)andRussell& Jenkins (1954)respectivel
difficult[19] difficulty simple ndicateranksof responsesin norms of Nelson et y.
neighborsinWAS(using400dimensions)to40
11
cue words taken from RussellandJenkins (1954). Forallneighborslistedinthetable,iftheywere associatesin thefreeassociationnormsoN f elsonet al.,thenthecorrespondingrankinthenormsigsi inparenthesesandthosefromRussellandJenkinsa showninbrackets.Thecomparisonbetweenthese twodatabasesisinterestingbecauseRusselland Jenkinsallowedparticipantstogenerateasmany responsestheywantedforeachcuewhilethenorms ofNelsonetal.containfirstresponsesonly.We suspectedthatsomecloseneighborsinWASarenot directassociatesintheNelsonetal.normsbutth theywould havebeenvalidassociatesipf articipants hadbeenallowedtogivemorethanoneassociation percue.InTable4,welistthepercentagesof neighborsinWASofthe100cuesothe f Russelland Jenkinsnorms(only40wereshowninTable3t)hat arevalid/invalidassociatesaccordingtothenorms Nelson eal. tand/or thenorms of RussellandJenki Table4. Percentages of Responses of WASmodel that areVali in Russell & Jenkins (1954) andNelsoneal. t(1999
ven re
at
of ns.
d/Invalid Norms )
1
2
Neighbor 3
4
5
validiN n elsoneal. t
96
73
61
45
33
validiJnenkins etal.
96
83
79
69
64
validienither Nelson eal. t or Jenkins etal.
99
86
82
73
66
InvalidinNelson eal. tbut validiJnenkins etal.
3
Validity
throughachainoafssociates.Forexample,thepai rs ‘blue-pants’‘butter-rye’, , ‘comfort-table’arecl ose neighborsinWASbutarenotdirectlyassociated witheachother. ItislikelythatbecauseWASis sensitivetoindirectrelationshipsinthenorms,t hese wordpairswereputclosetogetherinWASbecause oftheindirectassociativelinksthroughthewords ‘jeans’,‘bread’and‘chair’respectively. Inasimilar way,‘cottage’and‘cheddar’arecloseneighborsin WASbecausecottageisrelated(inonemeaningof theword)to‘cheese’,whichisanassociateof ‘cheddar’. Tosummarize,weshowedthattheoutputorder ofwordsinfreeassociationnorms ispreservedto somedegreeinWAS:firstassociatesinthenorms arelikelytobecloseneighborsinWAS.However, therearesomeinterestingdifferencesbetweenthe similaritystructureofWASandtheassociative strengthsotfhewordsinthenorms.Wordsthatare notdirectlyassociatedcanbecloseneighborsin WASwhenthewordsareindirectlyassociatively relatedthroughachainofassociates.Also,althou gh theyappeartobeexceptions,somewordsthatare directlyassociatedinthenormsarenotclose neighborsinWAS.Becauseofthesedifferences, WASisnotanexceptionallygoodmodelforthetask ofpredictingfreeassociationdata.However,itis importanttorealizethatWASwasnotdevelopedas a model offreeassociation(e.g.Nelson&McEvoy, Dennis,2000)butratherasamodel basedon free association.
13
21
Thelastrowshowsthatathirdofthe5 neighborsinWASarenotassociatesaccordingtoth normsofNelsonetal.butthatareassociates accordingtothenormsofRussellandJenkins. Therefore,somecloseneighborsinWASarevalid associatesdependingonwhatnormsareconsulted. However,somecloseneighborsinWASarenot associatesaccordingtoeithernorms.Forexample, nd ‘angry’itshe2 neighborof‘anger’inWAS.These wordsareobviouslyrelatedbywordform,butthey donottoappearasassociates in freeassociation becauseassociationsotfhesamewordformtendto beeditedoutbyparticipants.However,becausethe wordshavesimilarassociativestructures,WASputs them closetogether in thevector space. Also,somecloseneighborsinWASarenotdirect associatesoef achotherbutareindirectlyassocia
28
Semantic/AssociativeSimilarityRelations Intheprimingliterature,severalauthorshave triedtomakeadistinctionbetweensemanticand associativewordrelationsinordertoteaseapart differentsourcesofpriming(e.g.Burgess&Lund, 2000;Chiarello,Burgess,Richards&Pollock,1990; Shelton&Martin,1992).BurgessandLund(2000) have argued that the word association norms confoundmanytypesoword f relations,amongthem, semanticandassociativewordrelations.Chiarello al.(1990)give“music”and“art”asexamplesof wordsthataresemanticallyrelatedbecausethewor areratedtobemembersofthesamesemantic category(Battig&Montague,1969).However,they claimthesewordsarenotassociativelyrelated becausetheyarenotdirectassociatesoef achothe (accordingtothevariousnormdatabasesthatthey used).Thewords“bread”and“mold”weregivenas examplesow f ordsthatarenotsemanticallyrelated becausetheyarenotratedtobm e embersothe f sam semanticcategorybutonlyassociativelyrelated (because“bread”ias nassociateo“fmold”).Finall “cat”and“dog”weregivenasexamplesofwords thatareboth semantically andassociatively relate
33
th
closest e
tasks se
ted
12
et ds
r
e y, d.
between WordPairs with Differe Table5. AverageSimilarity
ntRelations: Semantic,Associative,andSemantic+
Associative k
Relation Random
S #Pairs
* ij
200 .000(.000)
10 0.34
50 (0.277)
0.075
200
400
(0.178)
0.024
(0.064)
0.017 (0.048)
0.215 (0.321)
Wordpairs from Chiarelloeal. t(1990) Semanticonly
33 .000(.000)
0.73
(0.255)
0.457
(0.315)
0.268
(0.297)
Associativeonly
43 .169(.153)
0.902
(0.127)
0.83
(0.178)
0.712
(0.262)
0.666 (0.289)
0.962
(0.053)
0.926
(0.097)
0.879
(0.180)
0.829 (0.209)
Semantic+Associative
44 .290(.198)
Wordpairs from Shelton andMartin (1992) Semanticonly Semantic+Associative
26 .000(.000) 35 .367(.250)
Note:standard deviations given between parentheses *Sij =averageforwardand backward associativestrength
0.724
(0.235)
0.448
(0.311)
0.245
(0.291)
0.166 (0.281)
0.926
(0.088)
0.929
(0.155)
0.874
(0.204)
0.836 (0.227)
A = (
+A ij
ji2/)
aresemanticallyonlyandassociativelyonlyrelate Forhigherdimensionalities,thisdifferencebecome larger as WAS becomes more sensitive in representing more of the direct associative relationships. Toconclude,itisdifficulttodistinguishbetween puresemanticandpureassociativerelationships. Whatsomeresearcherspreviouslyhaveconsideredto bepuresemanticwordrelations,werewordpairsth wererelatedinmeaningbutthatwerenotdirectly associated witheachother.Thisdoesnotmean, however,thatthesewordswerenotassociatively relatedbecausetheinformationinfreeassociation norms goes beyond that of direct associative strengths.Infact,thesimilaritystructureofWAS turnsouttobesensitivetothesimilaritiesthat arguedbysomeresearcherstobepurelysemantic. Wewillillustratethispointwiththefollowing experiment.
Weagreethattheresponsesinfreeassociation normscanberelatedtothecuesinmanydifferent ways, but it seems very hard and perhaps counterproductivetoclassifyresponsesaspurely 9 semanticorpurelyassociative F . orexample,word pairsmightnotbedirectlybutindirectlyassociat ed throughachainofassociates.Thequestionthen becomes,howmuchsemanticinformationdothefree association norms contain beyond the direct associations?BecauseWASissensitivetothe indirectassociativerelationshipsbetweenwords,w e tookthevariousexamplesofwordpairsgivenby Chiarelloeal. t(1990)andSheltonandMartin(199 2) andcomputedtheWASsimilaritiesbetweenthese wordsfordifferentdimensionalitiesasshownin Table5. InTable5the , interestingcomparisonibs etween thesimilaritiesforthesemanticonlyrelatedword pairs10(aslistedbyChiarelloetal.,1990)and200 randomwordpairs.Therandomwordpairswere selected to have zero forward and backward associativestrength. Itcanbeobservedthatthesemanticonlyrelated wordpairshavehighersimilarityinWASthanthe randomwordpairs.Therefore,eventhoughChiarello etal.(1990)havetriedtocreatewordpairsthat were onlysemanticallyrelated,WAScandistinguish betweenwordpairsthatwerejudgedtobeonly semanticallyrelatedandrandomwordpairsbothof which havingnodirectassociationsbetweenthem. ThisdiscriminationispossiblebecauseWASis sensitivetoindirectassociativerelationshipsbet ween words. The table also shows that for low dimensionalities,thereisnotasmuchdifference betweenthesimilarityow f ordpairsthatsupposedl y
Predictingsimilarityratings TofurtherassessthedegreetowhichWASis capable of capturing semantic similarity, the similaritystructure inWAS was compared to semanticsimilarityratingscollectedbyRomneyet (1993).Basedonthetriadsmethod,theyconstructe perceivedsimilaritymatricesfordifferentsetsof wordsthatfellinvarioussemanticcategories.Of thesecategories,wetookthe10matricesforthe1 categoriesofbirds,clothing,fish,fruits,furnit sports,tools,vegetables,vehicles,andweapons. Between23and28subjectsassessedthesimilarity memberswithineachcategory.Becausesubjects assessedthesimilarityformemberswithincategori (e.g.fordifferentbirds),wethoughtiw t ouldbe
13
d. s
at
were
al. d 21 0 ure, of es a
havearguedthatasymmetriesinjudgmentsarenot necessarilyduetoasymmetriesintheunderlying similarityrelationships.Withfreeassociationnor iftheassociationstrengthA->BisstrongerthanB >A,asimpleexplanationcouldbethatwordBhas higherwordfrequency,ismoresalientand/or,its representationismoreavailablethanwordA.To checkwhetherwordfrequencycanbuesedtoexplain asymmetriesinthefreeassociationnormsoN f elson etal.,wetookthe500strongestasymmetric associativewordpairsandassignedthemoreoften producedresponsetoB,andthelessoftenproduced responsetoA(inotherwords:A->Bisstrongertha B->A). InFigure3a,theKuceraandFrancis(1967) wordfrequencyofAisplottedagainsttheword frequencyoB. f Thediagonallinerepresentstheca wherethewordfrequenciesoA fandBareequal.It canbeseenthatmostofthesewordassociationsfa abovethediagonal(463outthe500,o9r 2.6%),so wordfrequencycanindeedbeusedtoexplainthe directionality oword f associations. Krumhansl(1978)hasproposedthatasymmetries canalsoarisebecauseodifferences f inthedensit exemplarsinpartsofthepsychologicalspace.For example,ifA->BisstrongerthanB->A,thiscould explainedbecauseBisplacedinadenseregionof thespacewhereasAisplacedimore na sparseregi ofthespace.Insamplingmodels,thesedifferences densitymightleadtodifferentresults.Inthepre example,samplingBtakingAasareferencewould beamorelikelyeventthansamplingAtakingBas referencewhenAhasfewcloseneighborsotherthan B,andBhasmanycloseneighborsotherthanAWe .
verychallengingtasktocapturethewithincategor y similaritystructure(asjudgedbysubjects)with methods such aWAS s oLSA. r The correlation was calculated between all observedsimilaritiesandpredictedsimilaritiesus ing thedifferentmethodsoW f AS,associativestrengths andLSA.Theresultfordifferentdimensionalities is shown in Figure 2, bottom right panel. All correlations were relativelylowwith the best correlationof0.36obtainedforWASbasedonthe SVDofS (1).Interestingly,thesemanticspacesof WASledtohighercorrelationsthanLSA.This difference provides further evidence that the similaritystructureinwordassociationnormsand the similaritystructureextractedbyWASfromtheword associationnormsdoescontainmanyaspectsof semantic similarity. Therefore, the similarity structureinwordassociationnormsoW r AScannot besimplycategorizedas ‘associative’,norcanone concludethatthereino ssemanticstructure.
PredictingAsymmetries inWordAssociation Norms Inthefreeassociationnorms,alargenumberof wordassociationsareasymmetric.Forexample,the cue“fib”isstronglyassociatedwith“lie”,butno viceversa.InWAS,thedistancebetweenwordA andBisbydefinitionequivalenttothedistance betweenwordsBandAThis . characteristicraisest questionofhowweareabletoaccountforthe asymmetries in free associations with WAS. Nosofsky(1991),andAshandEbenholtz(1962)
10000
t
he
1000
(a)
(b)
Rank(A,B)
FrequencyB
1000
100
10
1
100
10
1 1
10
100
1000
10000
1
FrequencyA
10
100
1000
Rank(B,A)
Figure 3.In(a),theKucera&Franciswordfrequency ipsl ottedf orwordsA versus words Bof wordpairs that arestronglyassociatedinthedirectionA>Bbutnotviceversa.In(b),theplotshowsthat forwordpairsthat arestronglyassociatedfromA->Bbutnotvicevers a,Bisoftencloser a neighbortoAthanAis toBinWAS whenmeasuredbyoutputrank(A,B):thenumberonf e ighborsinterveningbetweenAandBwhenAistaken
14
ms, -
n
se ll
yof be on in vious a
similarityratingsinrecognitionmemory,percentag e correctinextracuedrecallandintrusionratesin free recall.Inallthesememorytasks,WASwasabetter predictorforperformancethanLSA.Thissuggestst o usthatWASformsuseful a representationalbasisf or memorymodelsthataredesignedtostoreand retrievewords as vectors of featurevalues. Manymemorymodelsassumethatthesemantic aspectsowords f canbreepresentedbycollections of featuresabstractlyrepresentedbyvectors(e.g.Ei ch, 1982;Murdock,1982;Pike,1984;Hintzman,1984, 1988;McClelland&Chappell,1998;Shiffrin& Steyvers,1997,1998).However,inmostmemory modeling, the vectors representing words are arbitrarilychosenandarenotbasedonoderived r by someanalysisotfhemeaningoaf ctualwordsinour language.Weexpectthatmemorymodelsbasedon thesesemanticvectorsfromWASwillbeusefulfor makingpredictionsabouttheeffectsofvarying semantic similarityinmemoryexperimentsfor individualwords. ForthefirsttwoversionsoWAS f thatwerebased onsingularvaluedecomposition,thenumberof dimensionsotfhesemanticspacethatledtothebe st fitswithobservedperformancevariedbetween200 and500dimensions.Asimilarrangeinthenumber ofdimensionshasshowntobeeffectiveintheLSA approach(Landauer&Dumais,1997).Interestingly, the third version of WAS based on metric multidimensionalscalingachievednearlythesame performancebutwithonly20-40dimensions.This suggeststhatthesemanticvectorsinmemorymodels mightinvolverelativelyfewfeaturevaluestocapt ure thesemanticsimilaritystructure forafewthousand words. WeproposethatWASisanapproachthat augments other existing methods available for placingwordsinapsychologicalspace.Itdiffers fromtheLSAandHALapproachesinseveralways. LSAandHALareautomaticmethodsand donot requireanyextensivedatacollectionofratingsor freeassociations.WithLSAandHAL,tensof thousandsofwordscanbeplacedinthespace, whereasinWAS,thenumberow f ordsthatcanbe placeddependsonthenumberow f ordsthatcanbe normed.It ookNelsonetal.(1999)morethana decade to collect the norms, highlighting the enormoushumanoverheadofthemethod.Even thoughaworkingvocabularyoabout f 5000wordsin WASismuchsmallerthanthe70,000wordlong vocabulariesoL f SAandHAL,webelieveiis tlarge enoughforthepurposeom f odelingperformancein varietyofmemoryexperiments.Anadvantageof LSAandHAListhattheseapproacheshavethe potentialtomodelthelearningprocessthata languagelearnergoesthrough.Forexample,by
checkedwhetherthismechanismcanexplainthe directionality of the 500 asymmetric word associations.Insteadofcalculatingdensityinloc al regionsoWAS, f wecalculatedtwonumbersforeach of the 500 asymmetric word pairs. First, we calculatedtherankofthesimilarityofA-Bamong thesimilarityofAtoallotherwords(thiswillb e denotedbyrank(B,A))andamongthesimilarityoB f to all other words (this will be denoted by nd rank(A,B)).Forexample,ifBisthe2 neighborof rd AandAisthe3 neighboroB f then , rank(B,A)=2 andrank(A,B)=3.InFigure3b,rank(A,B)ips lotted againstrank(B,A). Thediagonalindicatesthecase s wherethenumberofneighborsseparatingAandBis thesameforAandBMost . ofthecases(486outof 500,or97.2%)lieabovethediagonal,meaningthat inWAS,BisclosertoAthanAistoBinthesens e ofnumberofneighborsseparatingAandB.This meansthatKrumhansl’s(1978)ideaodifferences f i n densitycanbeappliedtoWAStoexplainthe directionality oword f associations.
Discussion Ithasbeenproposedthatvariousaspectsowords f canbreepresentedbyseparatecollectionsofeatu f that codefortemporal,spatial,frequency,modality, orthographic,acoustic,andassociativeaspectsof words(Anisfeld&Knapp,1968;Bower,1967; Herriot,1974;Underwood,1969;Wickens,1972).In this research, we have focused on the associative/semanticaspects of words. Bystatistical a analysisolafargedatabaseofr f associationnorms,theWordAssociationSpace (WAS)wasdeveloped.Inthisspace,wordsthathave similarassociativestructuresareplacedinsimila regionsofthespace.InthefirstversionofWAS, singularvaluedecompositionwasappliedonthe directassociationsbetweenwordstoplacethese wordsinahighdimensionalsemanticspace.Inthe secondversionofWAS,thesametechniquewas appliedonthedirectandindirectassociations betweenwords.InthethirdversionofWAS,metric multidimensionalscalingwasappliedonmeasures fortheassociativestrengthrelatedtotheshortes associativepathbetweenwords(similartothe approachinCookeeal., t 1986andKlauer&Carroll 1995). Becausethefreeassociationnormshavebeenan integral part in predicting episodic memory phenomena(e.g.Cramer,1968;Deese,1965;Nelson, Schreiber,&McEvoy,1992),itwasassumedthata semanticspacebasedonfreeassociationnorms wouldbeanespeciallyusefulconstructtomodel memoryphenomena. WecomparedWASwithLSA inpredictingtheresultsofseveralmemorytasks:
res the
ee
r
t ,
15
feedingtheLSAorHALmodelsuccessivelylarger chunksoftext,theeffectthatlearninghasonthe similaritystructuresow f ordsinLSAorHALcanbe simulated.InWAS,itisinprinciplepossibleto modelalanguagelearningprocessbycollectingfre associationnormsforparticipantsadt ifferentsta ofthelearningprocess.Inpracticehowever,such approachwouldnoteasilybeaccomplished.To conclude,wethinkthattheWAS,LSA,andHAL approachestocreatingsemanticspacesareallusef fortheoreticalandempiricalresearchandthatthe usefulnessofaparticularspacewilldependonthe particular task iis applied t to.
3.Theresultoaf pplyingSVDtoanasymmetric wordassociationmatrixwouldbetwovectorspaces: thesewouldcapturethecommonalitiesamongthe cues(rows)andresponses(columns)respectively. Analysesshowedthatneitherotfhesevectorspaces capturesthevarianceinthememorytasksreported in thispaperasmuchavector as spacederivedaby t an SVDon symmetrized a wordassociation matrix. 4.Thenumberofdimensionsthatcanbe extractedisconstrainedbyvariouscomputational aspects.Wewereabletoextractonlythefirst500 dimensionsinthethreescalingsolutionsofWAS basedon5018wordsand400dimensionsinthe scalingsolutionbasedonall5018wordsinthefre e association database. 5.Variousalternativescalingproceduressuchas nonmetricMDScanhandlemissingvaluessothat notalldistancesinthematrixwouldhavetobe estimated.However,nonmetricMDScannotbe appliedtomatriceswiththousandsofrowsand columnssowewereforcedtowiththemetricMDS procedurethatworkswithlargematrices butthat does notallow for missing values. (1) In 6.fact,thematrixS has99.5%zeroentries respectivelysoestimatingthedistancesforthese zero entries is nontrivial. 7.Theratingwereconvertedtoz-scoresby subtractingthemeanratingfromeachratingandth en dividing by the standard deviation. This was performedforeachsubjectseparatelyandtheresul ts wereaveragedoversubjects.Thisprocedureremoved someoftheidiosyncratictendenciesofsubjectsto useonly partof therating. 8. Theoriginaldatabasehas2272cuetargetpairs. Weaveragedthepercentagecorrectresultsover identicalcue-targetpairsthatwereusedindiffer ent experiments.This gave1115uniquepairs. 9.Sinceresponsesinwordassociationtasksare bydefinitionallassociativelyrelatedtothecue, itis notclearhowitispossibletoseparatetherespon ses as semantically andassociatively related. 10.Somewordpairsinthesemanticonly conditionsthatwerenotdirectlyassociatedaccord ing tovariousdatabasesofreeassociationnormswere actuallydirectlyassociatedusingtheNelsonetal . (1998)database.Thesewordpairswereexcluded from theanalysis.
e ges an
ul
Author Note TheWordAssociationSpace(WAS)vectorscan bedownloadedinMatlabformatfrom(wwwaddress willbegiveninfuturedrafts).Wewouldliketo thankTomLandauerandDarrellLahamforkindly providing us with the LSA vectors for the encyclopediaandtasacorpus.Also,wewouldliket o acknowledgeJoshTenenbaum’shelpinfiguringout ways to applymetric multidimensional scaling techniques on the word association norms. Correspondence concerning this article can be addressedtoMarkSteyvers,PsychologyDepartment, StanfordUniversity,Stanford,CA94305-2130.Emailmay bseentto
[email protected].
Notes 1.ThismethodwasdevelopedbyOsgood,Suci, andTannenbaum(1957).Wordsareratedonset a of bipolarratingscales.Thebipolarratingscalesar e semanticscalesdefinedbypairsopf olaradjective s (e.g.“good-bad”,“altruistic-egotistic”,“hot-cold ”). Eachwordthatonewantstoplaceinthesemantic spaceisjudgedonthesescales.Ifnumbersare assignedfromlowtohighforthelefttorightwor dof abipolarpair,thentheword“dictator”forexampl e, mightbejudgedhighonthe“good-bad”,highonthe “altruistic-egotistic”andneutralonthe“hot-cold ” scale.Foreachword,theratingsaveragedovera largenumberofsubjectsdefinethecoordinatesof the wordinthesemantic space.Becausesemantically similarwordsarelikelytoreceivesimilarratings , theyarelikelytobelocatedinsimilarregionsof the semanticspace.Theadvantageofthesemantic differentialmethodisitssimplicityandintuitive appeal.Theprobleminherenttothisapproachitsh e arbitrarinessinchoosingthesetofsemanticscale ass wellas thenumber of such scales. 2.BecausewewereunabletoobtaintheHAL vectors, we could not include HAL in the comparisons in this paper.
References Anisfeld,M.,&Knapp,M.(1968).Association, synonymity,anddirectionalityinfalse recognition. Journal of ExperimentalPsychology,77 171-179. , Battig,W.F.,&Montague,W.E.(1969).Category normsforverbalitemsin56categories:Areplicat ionand
16
extensionoftheConnecticutcategorynorms. Journalof ExperimentalPsychology Monograph,80(3) 1-46. , Brainerd,C.J.,&Reyna,V.F.(1998).Whenthingst hat wereneverexperiencedareeasierto“remember”th an , thingsthatwere. PsychologicalScience,9 484-489. Brainerd,C.J.,Reyna,V.F.,&Mojardin,A.H.(1999 ). , Conjointrecognition. PsychologicalReview,106 160-179. Bousfield,W.A.(1953).Theoccurrenceocf lusterin g intherecallofrandomlyarrangedassociates. Journalof GeneralPsychology,49 229-240. , Bower,G.H.(1967).Amulticomponenttheoryofthe memorytrace.InK.W.Spence&J.T.Spence(Eds.), The psychologyolfearningandmotivation Vol , 1.NewYork: Academic Press. Burgess,C.,Livesay,K.,andLund,K.(1998). Explorationsincontextspace:Words,sentences,di scourse. , Discourse Processes,25 211-257. Burgess,C.,&Lund,K.(2000).Thedynamicsof meaninginmemory.InE.DietrichandA.B.Markman (Eds.), Cognitive dynamics: conceptual and representational change in humans and machines . Lawrence Erlbaum. Chiarello,C.,Burgess,C.,Richards,L.,&Pollock A. , (1990).Semanticandassociativepriminginthecer ebral hemispheres: Some words do, some words don’t, …sometimes,someplaces. BrainandLanguage,38 7, 5104. Canas,J.J.(1990).Associativestrengtheffectsi nthe lexical decision task. The Quarterly Journal of ExperimentalPsychology,42,121-145. Caramazza,A.,Hersch,H.,&Torgerson,W.S.(1976) . Subjectivestructuresandoperationsinsemanticme mory. Journalofverballearningandverbalbehavior,15 1, 03117. Cooke,N.M.,Durso,F.T.,&Schvaneveldt,R.W. (1986).Recallandmeasuresofmemoryorganization. JournalofExperimentalPsychology:Learning,Memor y, andCognition,12 538-549. , Cramer,P(.1968). WordAssociation N . Y:Academic Press. Deese,J(. 1959a).Influenceofinter-itemassociat ive strengthuponimmediatefreerecall. PsychologicalReports, 5,305-312. Deese,J.(1959b).Onthepredictionoof ccurrences of particularverbalintrusionsinimmediaterecall. Journalof ExperimentalPsychology,58 17-22. , Deese,J.(1962).Onthestructureofassociative meaning. PsychologicalReview,69 161-175. , Deese,J.(1965). Thestructureofassociationsin languageandthought Baltimore, . MD:TheJohnsHopkins Press. Derweester,S.,Dumais,S.T.,Furnas,G.W.,Landaue r, T.K.,&Harshman,R.(1990).Indexingbylatentsem antic analysis. JournalotfheAmericanSocietyforInformation Science,41 391-407. , Eich,J.M.(1982).Acompositeholographic associat ive , recallmodel. PsychologicalReview,89 627-661. Gillund,G.,&Shiffrin,R.M.(1984).Aretrievalm odel forbothrecognitionandrecall. PsychologicalReview,91 , 1-67. Herriot,P.(1974). Attributesofmemory .London: Methuen.
Hintzman,D.L.(1984).Minerva2sa:imulationmode l of human memory. Behavior Research Methods, Instruments,andComputers, 16,96-101. Hintzman,D.L.(1988).Judgmentsoffrequencyand recognitionmemoryinamultiple-tracememorymodel . 528-551. , PsychologicalReview, 95 Jenkins,J.J.,Mink,W.D.,&Russell,W.A.(1958). Associativeclusteringasafunctionofverbalasso ciation , strength. PsychologicalReports,4 127-136. Johnson,R.A.,&Wichern,D.W.(1998). Applied multivariate statisticalanalysis New . Jersey,Prentice Hall. Krumhansl,C.L.(1978).Concerningtheapplicabilit y of geometric models to similarity data: The interrelationshipbetweensimilarityandspatialde nsity. , 463. PsychologicalReview,85 445, Klauer,K.C.,&Carroll,J.D.(1995).Networkmodel s forscalingproximitydata.In,R.D.Luce,M.D’Zmu ra,D. Hoffman,G.J.Iverson,&A.K.Romney(eds.), Geometric representationsofperceptualphenomena M . ahwah,New Jersey:Lawrence Erlbaum Associates. Kucera,H.,&Francis,W.N.(1967). Computational analysisopf resent-dayAmericanEnglish Providence, . RI: BrownUniversity Press. Landauer,T.K.,&Dumais,S.T.(1997).Asolutiont o Plato’sproblem:TheLatentSemanticAnalysistheor yof acquisition,induction,andrepresentationofknowl edge. PsychologicalReview,104 211-240. , Landauer,T.K.,Foltz,P.,&Laham,D(. 1998).An introduction to latent semantic analysis. Discourse Processes,25 259-284. , Lund,K.,&Burgess,C.(1996).Producinghighdimensionalsemanticspacesfromlexicalco-occurre nce. BehaviorResearchMethods,Instruments,andCompute rs, 28,203-208. Marshall,G.R.,&Cofer,C.N.(1963).Associative indicesasmeasuresofwordrelatedness:asummary and comparisonoten f methods. Journalof VerbalLearning and VerbalBehavior,1 408-421. , McClelland,J.L.,&Chappell,M(.1998).Familiarit y breedsdifferentiation:Asubjective-likelihoodapp roachto the effects of experience in recognition memory. PsychologicalReview,105 724-760. , Morton,J.A.(1970).Afunctionalmodelformemory. N . ew InD.A.Norman(Ed.), Modelsofhumanmemory York:Academic Press. Murdock,B.B.(1982).Atheoryforthestorageand retrievalofitemandassociativeinformation. Psychological Review,89 609-626. , Neely,J.H.(1991).Semanticprimingeffectsinvis ual wordrecognition: saelectivereview of currentfin dingsand theories.InD.Besner&G.W.Humphreys(Eds.),Bas ic processesinreading:Visualwordrecognition (pp.264336).Hillsdale,NJ:Lawrence Erlbaum Associates. Nelson, D.L. (2000). The cued recall database. http://www.usf.edu/~nelson/CuedRecallDatabase. Nelson,D.L.,Bennett,D.J.,&Leibert,T.W.(1997) . Onestepisnotenough:makingbetteruseoaf ssoci ation , normstopredictcuedrecall. Memory&Cognition,25 785-706. Nelson,D.L.,McEvoy,C.L.,&Dennis,S.(2000), Whatisfreeassociationandwhatdoesitmeasure? Memory & Cognition,28 887-899. ,
17
Schwartz,R.M.,&Humphreys,M.S.(1973).Similarit y judgmentsandfreerecallofunrelatedwords. Journalof ExperimentalPsychology 101, , 10-15. Shelton,J.R.,&Martin,R.C.(1992).Howsemantic is automaticsemanticpriming? JournalofExperimental Psychology:Learning,Memory,and,Cognition,18 1191, 1210. Schiffman,S.S.,Reynolds,M.L.,&Young,F.W. (1981). Introductiontomultidimensionalscaling;theory, methods,andapplications .NewYork,NY,Academic Press. Shiffrin,R.M.,Huber,D.E.,&Marinelli,K(. 1995) . Effectsofcategorylengthandstrengthonfamiliar ityin recognition. Journal of Experimental Psychology: Learning,Memory,andCognition,21 267-287. , Shiffrin,R.M.,&Steyvers,M(. 1997).Amodelfor recognitionmemory:REM—retrievingeffectivelyfrom memory. Psychonomic Bulletin& Review,4 145-166. , Shiffrin, R. M., & Steyvers, M. (1998). The effectivenessorfetrievalfrommemory.InM.Oaksf ord& (pp. . 73-95), N.Chater (Eds.). Rationalmodels of cognition Oxford,England:OxfordUniversity Press. Steyvers,M.,&Shiffrin,R.R.(tobesubmitted). Modelingsemanticandorthographicsimilarityeffec tson memory for individualwords. Torgeson,W.S.(1952).Multidimensionalscaling:I. Theory andmethod. Psychometrica,17 401-419. , Underwood,B.J.(1965).False recognition produced by implicit verbal responses. Journal of Experimental Psychology,70 122-129. , Underwood, B.J. (1969). Attributes of memory, PsychologicalReview,76 559-573. , Wickens, D.D. (1972). Characteristics of word encoding.InA.W.Melton&E.Martin(Eds.), Coding processesinhumanmemory .Washington,D.C.:V.H. Winston,pp.191-215.
Nelson,D.L.,McEvoy,C.L.,&Schreiber,T.A.(1999 ). TheUniversityofSouthFloridawordassociation,r hyme, and word fragment norms. http://www.usf.edu/FreeAssociation. Nelson,D.L.,McKinney,V.M.,Gee,N.R.,& Janczura , G.A.(1998).Interpretingtheinfluenceofimplicit ly activated memories on recall and recognition. , PsychologicalReview,105 299-324. Nelson, D.L., & Schreiber, T.A. (1992). Word concreteness and word structure as independent determinantsorfecall. JournaloM f emoryandLanguage, 31,237-260. Nelson,D.L.,Schreiber,T.A.,&McEvoy,C.L.(1992 ). Processing implicit and explicit representations. , PsychologicalReview,99 322-348. Nelson,D.L.,Xu,J.(1995).Effectsoimplicit f me mory onexplicitrecall:Setsizeandwordfrequencyeff ects. PsychologicalResearch,57 203-214. , Nelson,D.L.,&Zhang,N.(2000).Thetiesthatbin d whatisknowntotherecallow f hatisnew. Psychonomic Bulletin& Review,7 XXX-XXX. , Nelson, D.L., Zhang, N., & McKinney, V.M. (submitted).Thetiesthatbindwhatisknowntoth e recognitionowhat f isnew. Norman,D.A.,&Rumelhart,D.E.(1970).A system fo r perceptionandmemory.InD.A.Norman(Ed.), Modelsof humanmemory New . York:Academic Press. Osgood,C.E.,Suci,G.J.,&Tannenbaum,P.H.(1957) . .Urbana:Universityof Themeasurementofmeaning IllinoisPress. Palermo,D.S.,& Jenkins,J.J.(1964). Wordassociation norms grade school through college . Minneapolis: University oMinnesota f Press. Payne,D.G.,Elie,C.J.,Blackwell,J.M.,&Neuscha tz, J.S.(1996).Memoryillusions:Recalling,andrecol lecting eventsthatneveroccurred. JournalofMemoryand Language,35 261-285. , Pike,R.(1984).Comparisonoconvolution f andmatr ix distributedmemorysystemsforassociativerecalla nd recognition. PsychologicalReview,91 281-293. , Rips,L.J.,Shoben,E.J.,&Smith,E.E.(1973). Semantic distance and the verification of semantic relations. Journalofverballearningandverbalbehavior, 12,1-20. Roediger,H.L.,&McDermott,K.B.(1995).Creating falsememories:rememberingwordsnotpresentedon lists. JournalofExperimentalPsychology:Learning,Memor y, andCognition,21 803-814. , Romney,A.K.,Brewer,D.D.,&Batchelder,W.H. (1993).Predictingclusteringfromsemanticstructu re. PsychologicalScience, 4,28-34. Russell,W.A.,&Jenkins,J.J.(1954). Thecomplete Minnesotanormsforresponsesto100wordsfromthe Kent-Rosanoffwordassociationtest T . ech.Rep.No.11, ContractNS-ONR-66216,OfficeofNavalResearchand University oMinnesota. f Schacter,D.L.,Verfaellie,M.,&Pradere,D(.1996 ). Theneuropsychologyomemory f illusions:falsereca lland recognitioninamnesicpatients. JournalofMemory& Language,35 319-334. ,
18
