Semantic Spaces based on Free Association that ... - Semantic Scholar

1 downloads 8 Views 249KB Size Report
Black white(1)[1] Bleach color(3)[7] dark(2)[2] blossom flower(1)[1] petals[46] ... carpet floor(2)[2] tile(15) rug(1)[1] ceiling chair table(1)[1] seat(4)[4] sit(2)[2] ... beetle chocolate[4] minority daisy pants suds[49] flowers[10] doze book(2)[2]. 5.
Semantic Spacesbased on Free Association that Predict MemoryPerformance MarkSteyvers StanfordUniversity

RichardMShiffrin . DouglasL. Nelson IndianaUniversity

Universit

oySouth f Florida

Submitted toJEPGeneral

Abstract Manymemory modelsrepresentaspectsow f ordssuchasmeaningb yvectorsofeature values,suchthatwordswithsimilarmeaningsarep lacedinsimilarregionsothe f semantic spacewhosedimensionsaredefinedbythevectorpo sitions.Methodsforconstructingsuch spacesincludethosebasedonscalingsimilarityra tingsforpairsowords, f andthosebased ontheanalysisofco-occurrencestatistics ofwor dsincontexts(Landauer&Dumais, 1997).WeutilizedaWordAssociationSpace(WAS), basedonascalingoalarge f data baseofreewordassociations:Wordswithsimilar associativestructureswereplacedin similarregionsotfhehighdimensionalsemanticsp ace.IncomparisontoLSAandother measuresbasedonassociativestrength,weshowedt hatthesimilaritystructureinWASis wellsuitedtopredictsimilarityratingsinrecogn itionmemory,percentagecorrect responsesincuedrecallandintrusionratesinfre erecall. WesuggestthattheWAS approachiuseful as andimportantnewtoolinthe workshopotheorists f studyingsemantic effects in episodicmemory. Anincreasinglycommonassumptionoftheories ofmemoryisthatthemeaningofawordcanbe representedbyavectorwhichplacesawordasa pointinamultidimensionalsemanticspace(Bower, 1967;Landauer&Dumais,1997;Lund&Burgess, 1996;Morton,1970;Norman,&Rumelhart,1970; Osgood,Suci,&Tannenbaum,1957;Underwood, 1969;Wickens,1972).Themainrequirementof spacesisthatwordsthataresimilarinmeaningar representedbysimilarvectors.Representingwords vectorsinamultidimensionalspaceallowssimple geometricoperationssuchastheEuclidiandistance ortheanglebetweenthevectorstocomputethe semantic(dis)similaritybetweenarbitrarypairsor groupsofwords.Thisrepresentationmakesit possibletomakepredictionsaboutperformancein psychologicaltaskswherethesemanticdistance betweenpairsogr roupsow f ordsiasssumedtoplay raole. Onerecentframeworkforplacingwordsina multidimensionalspaceisLatentSemanticAnalysis orLSA(Derweester,Dumais,Furnas,Landauer,& Harshman, 1990; Landauer & Dumais, 1997; Landauer,Foltz,&Laham,1998).Themain assumptionisthatsimilarwordsoccurinsimilar contextswithcontextdefinedaasnyconnectedset textfromacorpussuchasanencyclopedia,or

samplesoftextsfromtextbooks.Forexample,a textbookwithaparagraphabout “cats”mightalso mention“dogs”,“fur”,“pets”etc.Thisknowledge canbeusedtoassumethat“cats”and“dogs”are relatedinmeaning.However,somewordsareclearly relatedinmeaningsuchas“cats”and“felines”but theymightneveroccursimultaneouslyinthesame context. Suchwordsarerelatedprimarilythrough indirectlinksbecausetheysharesimilarcontexts. Thetechniqueofsingularvaluedecomposition (SVD)canbeappliedtothematrixofword-context co-occurrencestatistics.Inthisprocedure,thedi and indirect relationships between words and contextsinthematrixareanalyzedwithsimple matrix-algebraicoperationsandtheresultisahig dimensionalspaceinwhichwordsthatappearin similarcontextsareplacedinsimilarregionsotf space.LandauerandDumais(1997)appliedtheLSA approachtoover60,000wordsappearinginover 30,000contextsofalargeencyclopedia.More recently,LSAwasappliedtoover90,000words appearinginover37,000contextsoreading f materi thatanEnglishreadermightbe xposedtofrom3 st gradeupto1 yearocf ollegefromvarioussources suchastextbooks,novels,andnewspaperarticles. TheSVDmethodplacedthesewordsinahigh dimensionalspacewiththenumberofdimensions

such e as

of

1

rect

h he

al rd

text(ofwhichthereisavirtuallyunlimitedamoun as opposedtoratings providedbpyarticipants. Inthisresearch,wewillintroducearelatedbut newmethodforcreatingpsychologicalspacesthati basedonananalysisofalargefreeassociation database collected by Nelson, McEvoy, and Schreiber(1999)containingnormsforfirstassocia forover5000words.Thismethodplacesover5000 wordsinapsychologicalspacethatwewillcall WordAssociationSpace(WAS).Webelievesucha constructwillbeveryusefulinthemodelingof episodicmemoryphenomena.Atpresentitisnot clearhowwellepisodicmemoryperformancein recognitionorrecallispredictedbyLSAorHAL. Wedoknowthatwordassociationsplay animportant roleinepisodicmemorysinceihas t beenshowntha theassociativestructureowords f playsacentral inrecall(e.g.Bousfield,1953;Cramer,1968;Dees 1959a,b,1965;Jenkins,Mink,& Russell,1958),cue recall(e.g.Nelson,Schreiber,&McEvoy,1992)and priming(e.g.Canas,1990;seealsoNeely,1991),a recognition(e.g.Nelson,Zhang,&McKinney, submitted).Forexample,Deese(1959a,b)foundthat theinter-itemassociativestrengthofthewordsin studylistcanpredictthenumberow f ordsrecalled thenumberofintrusions,andthefrequencywith whichcertainwordsintrude.Basedontheclassic experimentsbyJamesDeese(1959b),Roedigerand McDermott(1995)revivedinterestintheparadigm thatisnowknownasthefalsememoryparadigm (e.g.Brainerd,&Reyna,1998;Brainerd,Reyna,& Mojardin, 1999; Payne, Elie, Blackwell, & Neuschatz,1996;Schacter,Verfaellie,&Pradere, 1996;Tussing&Green,1997;Shiffrin,Huber,& Marinelli, 1995). In the typical false memory experiment,participantsstudywordsthatareall semanticallyrelatedtoanon-studiedcriticalword Inasubsequentrecognitiontest,thecriticalword typicallyleadtohigher a falsealarmratethanth unrelated foils (and sometimes quite high in comparisontothatforstudiedwords).Infree a re test,participantsfalselyintrudethecriticalwor ratehigherthanunrelatedwords(andsometimesat ratesapproachingthoseforstudiedwords).These studiesshowthatepisodicmemorycanbestrongly influencedbsyemanticsimilarity. Inthepresentresearch,wewillcomparethe performanceofLSAwithWASinthreeepisodic memorytasks:recognitionmemory,freerecalland cuedrecall 2.Itwasexpectedthatthesimilarity structureinWASwouldbewellsuitedtopredict varioussemanticsimilarityeffectsintheseepisod memorytasks.Tofurtherourunderstandingofthe similaritystructureofWAS,weperformedseveral analyses.First,wecomparedthepredictionsoWAS f andLSAforthestrengthothe f associatesobtained

chosen between 200 and 400. The LSA representationhasbeensuccessfullyappliedto multiplechoicevocabularytests,domainknowledge tests and content evaluation (see Landauer & Dumais,1997;Landauer etal.1998). Anotherframeworkthathasbeenusedtoplace wordsinahighdimensionalsemanticspaceisthe Hyperspace Analogue to Language (Burgess, Livesay,&Lund,1998;Lund&Burgess,1996;see Burgess&Lund,2000foranoverview).TheHAL model develops high dimensional vector representationsforwordsbasedonaco-occurrence analysisolfargesamplesow f rittentext.For70,0 00 words,theco-occurrencestatisticswerecalculated in a10wordwindowthatwasslidoverthetextfroma corpusofover320millionwords(gatheredfrom Usenet newsgroups). For each word, the cooccurrencestatisticswerecalculatedforthe70,00 0 wordsappearingbeforeandafterthatwordinthe1 0 wordwindow.Theresulting140,000valuesforeach wordwerethefeaturevaluesforthewordsinthe HALrepresentation.Becausetherepresentationis basedonthecontextinwhichwordsappear,theHAL vectorrepresentationis also referred to as a contextualspace:wordsthatappearinsimilar contextsarerepresentedbysimilarvectors.TheHA L and LSA approach are similar in one major assumption:similarwordsoccurinsimilarcontexts . InbothHALandLSA,theplacementofwordsina highdimensionalsemanticspaceisbasedonan analysisoftheco-occurrencestatisticsofwordsi n theircontexts.InLSA,acontextisdefinedbya relativelylargesegmentoftextwhereasinHAL,th e contextis definedbwindow ay of 10words. LSAandHALarebothcorpusbasedmethods that contrast sharply with older methods of constructingsemanticspacesthatdependonhuman 1 judgmentssuchaspairwisesimilarityratings In . thismethod,participantsratethesemanticsimilar ity forpairsow f ords.Then,thosesimilarityratings are subjectedtomultidimensionalscalinganalysesto derivevectorrepresentationsinwhichsimilarvect ors representwordssimilarinmeaning(Caramazza, Hersch,&Torgerson,1976;Rips,Shoben,&Smith, 1973;Schwartz&Humphreys,1973).Whilethis methodisstraightforwardandhasledtointerestin g applications(e.g.Caramazzaeal; t Romney,Brewer, &Batchelder,1993),itisclearly impracticalfor large numberow f ordsasthenumberorfatingsthatmust becollectedgoesupquadraticallywiththenumber of stimuli.OnegreatadvantageoL f SAandHALover approachesdependingonpairwisesimilarityratings isthatalmostanynumberofwordscanbpelacedin a semantic/contextualspace.Thisispossiblebecause thesemethodsrelyuniquelyonsamplesofwritten

2

t)

s

tes

t role e, d nd

a ,

. atfor call daat

ic

in

thefreeassociationtask.BecauseWASisexplicitl basedonthefreeassociationnorms,itwasexpecte thattherewouldbeacloserelationshipbetween distanceinthepsychologicalspaceandassociative strengthinfreeassociation.Inanotheranalysis, issue of whether WAS captures semantic or associativerelationships(orboth)inpredicting similarityratingsiasddressed.Itwillbeargued isdifficulttomakeacategoricaldistinctionbetw purelyassociativeandpurelysemanticrelationship Finally,inthelastanalysisweanalyzetheabilit WAStopredictasymmetriesinthedirectionof association.Becausefreeassociationnormscontain manyexamplesow f ordsthatarestronglyassociated inonebutnottheotherdirection,predictingthes asymmetrieswithamodelsuchasWASpresentsan interesting challenge given that the similarity between twowords is definedtobseymmetric.

overlap in the distribution of free association responsesforasmallsetofwordsandarguedthat theseanalysescouldbeusedtolearnaboutthe mentalrepresentationofwords.Inthispaper,we capitalized onDeese’sideasoutilizing f thepatternof intercorrelationsinthefreeassociationnormsby placingalargenumberofwordassociationsina semanticspaceandthenusedthemtopredict semanticsimilarityeffectsinmemory.Insteadof factoranalyses,weusedthetechniquesofsingular value decomposition (SVD) and metric multidimensionalscalinganalysesthatareclosely relatedtofactoranalysis(e.g.,seeJohnson& Wichern,1998)butthatcanbeappliedtolarge volumes of data. TheSVDmethodhasbeensuccessfully appliedin LSA(e.g.Landauer&Dumais,1997)touncoverthe patterns of intercorrelations of co-occurrence statisticsforwordsappearingincontexts.Inthis research,theSVDmethodwasappliedonalarge databaseoffreeassociationnormscollectedby Nelsonetal.(1999)containingnormsoffirst associatesforover5000words.Theresultisthat the wordsareplacedinahighdimensionalsemantic space.BecauseoftheSVDmethod,thesimilarity betweenwordsinthispsychological spaceisbest expressedbythecosineothe f anglebetweenthetw o correspondingvectors(seeDerweestereat l.1990). Wealsoperformedadditionalanalysesbyapplying metricmultidimensionalscaling(MDS)methods(see Torgeson,1952;Schiffman,Reynolds &Young, 1981)onthenormssuchthattheEuclidiandistance

y d

the

thatit een s. yof

e

WordAssociationSpaces Deese(1962,1965)assertedthatfreeassociations arenottheresultofhaphazardprocessesandthat theyarisefromanunderlyingregularityinpreexisting associative connections. He laid the frameworkforstudyingthemeaningoflinguistic forms that can be derived by analyzing the correspondencesbetweendistributionsofresponses tofreeassociationstimuli:"Themostimportant propertyofassociationsistheirstructure -their patternsofintercorrelations"(Deese,1965,p.1). Deese(1962,1965)appliedfactoranalysestothe

RESPONSES 

CUES

  



 

      



   

 

 

    

 

  !"

   

 

 #    %    

   %  #

   

 

$ () ( *  & ' "   (

345

   

678

9:5

EF G HIJK LM NO N PQ RS T U VMW UXY V RPZ

5 : ;:< => =

@4

8

AB=

+, ,, - + ,, , . / 0 1 2-

B

6

=C :

?

>=

7?

6 7 =

7?

4 ; > 94 3 : D

Nelson,McEvoy,&Schreiber(1999)

Figure 1.Illustrationothe f creationoWord f Association largedatabaseofreeassociationnorms,wordsare similar associativerelationships areplacedisnim

Spaces(WAS).Byscalingthewordassociationsoaf placedinahighdimensionalsemanticspace.Words ilar regions of thespace.

3

with

Table1. Overview of Methodsfor Quantifying Associative/Seman

Method

Basedon:

Associations

S

ticsimilarity

(Dis)similarity Measure

(1)

forward : plusbackwardstrength

S(2)forward : plusbackwardplus twostepassociativestrengths WAS

LSA

Vocabulary Size

S(1)

5000+

S(2)

5000+

svdoSf

(1)

cosineoangle f

2500

svdoSf

(2)

cosineoangle f

2500,5000+

metricmdsof T (see text)

Euclidiandistance

svdoword-context f matrix oencyclopedia f

cosineoangle f

svdoword-context f matrix otasa f documents cosineo

betweentwowordsintheresultingpsychological spacecapturesthedissimilaritybetweentwowords. Wewillrefertothegeneralmethodofplacingthe wordsinaspaceaw s ellasthespaceitselfaW s or AssociationSpace(WAS).Foranoverviewofallthe differentmethodstoquantifyassociative/semantic similarity itnhis research,seeTable1. ByapplyingscalingmethodssuchasSVDand metricMDSonthenorms,wehopetouncoverthe latentinformationavailableinthefreeassociatio normsthatisnotdirectlyavailablebyinvestigati simplemeasuresforassociativestrengthsbasedon thedirectandindirectassociativestrengthsthrou shortchainsofassociates(e.g.,Nelson&Zhang, 2000).ThebasicapproachisillustratedinFigure Thefreeassociationnormswererepresentedin matrixformwiththerowsrepresentingthecuesand thecolumnsrepresentingtheresponses.Theentries inthematrixarefilledbysomemeasureof associativestrengthbetweencuesandresponses.By applyingscalingmethodsonthematrix,wordsare placedinahighdimensionalspacesuchthatwords withsimilarassociativepatternsareplacedinsim regions of thespace. Intotal,morethan6000peopleparticipatedinthe collectionofthefreeassociationdatabaseoN f els etal.(1999).Anaverageof149(SD=15) participants were eachpresented with100-120 Englishwords.Thesewordsservedascues(e.g. “cat”)forwhichparticipantshadtowritedownthe firstwordthatcametomind(e.g.“dog”).Foreach cuetheproportionofsubjectsthatelicitedthe responsetothecuewascalculated(e.g.60% respondedwith“dog”,15%with“pet”,10%with “tiger”,etc).

angle f

60,000+ 90,000+

Scaling bySingularValueDecomposition WewillfirstexplainhowSVDisappliedtothe normsandthencontinuetothemetricMDSmethod. ThemethodofSVDcanbeappliedtoanymatrix containing some measure of strength or cooccurrencebetweentwowords.Althoughmany differentwayshavebeenproposedtocalculatean indexofassociativestrengthbetweentwowords (e.g.,Marshall&Cofer,1963;Nelson&Zhang, 2000),wewillrestrictourselvestotwosimple measuresoassociative f strength.LetA represent the ij proportionosfubjectsthatgavetheresponse to j the cuei.ThesimplestmeasurewouldbetotakeA ij itself.Inthenorms,theassociativestrengthsA are ij often highly asymmetric where the associative strengthinonedirectionistrongwhileiis twea kor zerointheotherdirection.EventhoughSVDcanbe easilyappliedtoasymmetricmatrices,theresults are moreinterpretablewhenitisappliedtosymmetric matrices3.Inalatersectioninthepaper,wewill show that symmetrizing the norms does not necessarilymeanthatasymmetriesintheword associationscannotbepredicted.Therefore,inour firstmeasurefor associativestrength wteake:

d

n ng gh 1.

ilar

on

Sij(1) = Aij + A ji S(1)ijisequivalenttoaddingforwardstrengthto backward strength. This measure is ofcourse (1) (1) symmetricsothatS ji.Thismeasureis ij=S indexedby(1)becauseitbasedononlythe direct associationbetweeniandjandinvolvesonlyone associativestepgoingfromtioj.Inthenormsof

4

Nelsonetal.(1998),subjectswereonlyallowedto givethefirstresponsethatcametomind.Theseco nd strongestresponseinonesubjects’mind mightbe elicitedbyanothersubjectorimight t notbeelic ited atallifthefirstresponseisastrongassociate. (1) Therefore,theS measuremightbeunderestimating the associative strength between two words especiallyincaseswherethemeasureizsero(Nels on etal.,1998).Inthesecondmeasureforassociativ e strength,wetake:

associative strengths can uncover the latent (1) relationshipsbetweenwords.IntheSVDofS , wordsthatarenotdirectassociatesoeach f other can stillberepresentedbysimilarvectorsiftheir (2) associatesarerelated.IntheSVDofS words , that not directly associated or indirectly associated throughoneintermediateassociate,canstillbe representedbysimilarvectorsithe f associatesof the associatesofthewordsarerelated.Inotherwords , thewholepatternofdirectandindirectcorrelatio ns betweenassociationsistakenintoaccountwhen placing words in thesemanticspace. Animportantvariable(whichwewillcallk)is thenumberofdimensionsothe f space.Onecanthin k ofkasthenumberofeaturevaluesforthewords. Wevaried kbetween10and500.Thenumberof dimensionswilldeterminehowmuchtheinformation 4 ofthefreeassociationdatabaseicsompressed With . toofewdimensions,thesimilaritystructureofthe resultingvectorsdoesnotcaptureenoughdetailof theoriginalassociativestructureinthedatabase. Withtoomanydimensionsorthenumberof dimensionsapproachingthenumberofcues,the informationinthenormsisnotcompressedenough sothatwemightexpectthatthesimilaritystructu re ofthevectorsdoesnotcaptureenoughothe f indir ect relationshipsintheassociationsbetweenwords.In theanalysesopredicting f performanceinvariety a of tasks(recognition,freeandcuedrecall),wewill showthatalthoughtheoptimalvalueokdf ependso n thespecifictask,intermediatevaluesofkbetween 200and500areappropriatefor this method.

Sij( 2 ) = Sij(1) + ∑ S ik(1) S kj(1) k

Thisequalstheforwardplusbackwardplus mediatedstrengththroughotherassociates. Notethat thismeasureinvolvesthedirectstrengthbetweeni andjaswellastheindirectstrengthbysumming overallpathsfromtioktoj,theproductofthe symmetricassociativestrengthsbetween and i kan , d kandj.Theseindirectassociativestrengthsinvol ve thetwostepprobabilitiesogoing f from to and ij vice versaandhencetheindex(2).Researchhasshown thattheindirectassociative strengthsplayarolein cuedrecall(Nelson,Bennet,&Leibert,1997;Nelso n &Zhang,2000)andrecognition(Nelson,Zhang,& McKinney,submitted).Forexample,Nelson& Zhang(2000)foundthatincludingtheindirect associativestrengthsinameasureforassociative strength significantly increases the explained varianceitnheextra-listcuedrecalltask. We applied SVD separately on these two measuresoaf ssociativestrength.Theresultoef ac h SVDistheplacementofwordsinhigh a dimensional space,sothatwordsthathavesimilarassociative structuresarerepresentedbysimilarvectors.Beca use oftheSVDmethod,andbasedonworkinLSA(see Derweestereat l.,1990),asuitablemeasureforth e similaritybetweentwowordsisthecosineofthe anglebetweentwowordvectors.Let

ScalingbyMetric-MDS AninterestingcomparisonforthetwoWAS spacesbasedonSVDwouldbetoconstructametric spaceinwhichthedistancebetweentwowords,i.e. , theirdissimilarity,canbemeasuredbytheEuclidi an distancebetweentheirvectors.MetricMDSisa classicmethodforplacingstimuliin space a such that theEuclidiandistancebetweenpointsinthespace approximates the Euclidian distances in the dissimilaritymatrix(seeTorgeson,1952;Schiffman , Reynolds&Young,1981).Inordertoapplymetric MDS,estimatesareneededforthedistancebetween anytwowords.Infact,allnon-diagonalentriesin the matrixhavetobefilledwithsomeestimateforthe distancebetweenwordssincenomissingvaluesare 5 allowedinthemethod This . raisestheproblemhow toestimatethedistancebetweeniandjwhenthe (1) 6 associativestrength ameasured s S by zero . ij is Inoursolutionotfhisproblem,wewere inspired bynetworkmodelsforproximitydata(e.g.Cooke, Durso,&Schvaneveldt,1986;Klauer,&Carroll, 1995). In these network models, dissimilarity betweentwostimuliiscalculatedbytheshortestp ath

v X i represent

thevectorinWASforwordiThe . similaritybetwee words and i is calculated j by:

n

v v Xi ⋅ X j similarity (i, j ) = cos(α ) = v v Xi X j where

v X isthelengthofthevectorand

v v X i ⋅ X j representstheinnerproductbetweenvectors and i jTwo . wordsthataresimilarinmeaningotr havesimilarassociativestructuresareexpectedto havehighsimilarityasdefinedbythecosineotfh anglebetweenthetwowordvectors.TheSVDofthe

hat e

5

betweentwonodesinagraph.Inthisresearch, canusethewordassociationnormsasdefininga graph:twowordsarelinkedbyanedgeitfheyhave nonzeroassociativestrengths.Wewillusethe symmetricS (1)associativestrengthsbecauseinthe (1) graphdefinedbyS itis ,possibletoreachanyword fromanyotherwordinthegraph(infact,the maximumnumberofstepsbetweenanypairofwords isfour).Thedistancebetweentwowordswillbe definedasthenegativelogarithmoftheproductof theassociativestrengthsalongtheshortestpathi (1) networkdefinedbyS .Thisisequivalenttothe (negative)sumofthelogsothe f associativestren along theshortestpath:

[

we

SinceDeese’s(1959b)classicstudyonintrusions infreerecall,many studieshaveshownthatmemory errorsareinpartbasedonsemanticoverlapbetwee n theresponseandthecontentsofmemory.Inthis research, we introduced WAS as a way of quantifyingthesemanticsimilaritybetweenwords thatmighthelpinpredictingthesememoryerrors. Thedatafromarecognitionexperiment,Deese’s originalfreerecallexperimentandacuedrecall experimentweretakenasabasisfortestingvariou s modelsthatcapturesemanticsimilarity.Wetested threeWASbasedmeasuresforsemanticsimilarity. (1) ThefirsttwowerebasedontheSVDofS the , one stepsymmetricassociativestrengths,andontheSV D ofS (2)the , oneplusthetwostepassociativestrengths involvingindirectassociativestrengths.Inthese two semanticspaces(aswellasinLSA)thecosineotf he anglebetweentwowordsexpressesthesimilarity betweentwowords.ThelastWASmeasurewas basedonmetric-MDSoftheshortestpathassociativ e strengths.Inthisspace,theEuclidiandistance betweentwowordvectorsistakenasameasurefor thedissimilaritybetweentwowords. TheseWAS scalingsolutionswerecontrastedwiththe(unscale d) (1) associativestrengthsS andS (2)thatweretakenas controlcomparisons.WealsotestedtwoLSAbased measures,onewasbasedona corpusofan encyclopediaandanotheroncorpus a calledtasath at includedreadingmaterialthatanEnglishreader rd st mightbeexposedtofrom3 gradeupto1 yearof college.Thedifferentmethodsthatarecomparedar e listediT n able1.

nthe gths

Tij = − log (S ik(1) S kl(1) ⋅ ⋅ ⋅ S qj(1) ) = − log S ik(1) + log S kl(1) + ... + log S qj(1)

]

Here,theshortestpathbetweenwordsand i ijs from to ikto through l otherwordstoqand finallyj. Withthisdistancemeasure,wordpairswithweakor longassociativepathsareassignedlargedistances whereaswordpairswithshortosr trongassociative pathsareassignedsmalldistances.ThedistancesT werecalculatedforallwordpairs intheword associationdatabase.Then,thesedistanceswere scaledbymetric-MDS.Theresultisthatthewords areplacedinamultidimensionalspaceandthe dissimilarityordistancebetweentwowordsis expressedbytheEuclidiandistancebetweenthetwo corresponding wordvectors:

 2 distance(i, j ) =  ∑ (X ik − X jk )   k

Predicting Semantic Similarity EffectsinMemory

ij

1/ 2

Becauseofcomputationalconstraints,iw t asnot possibletoapplymetric-MDStothefullmatrixT containingthedistancesforallwordpairs.Instea wechose2500wordsfromtheoriginal5018words inthewordassociationdatabase.Thewordsinthis smallersetincludedwordsappearinginvarious experimentslistedinthenextsectionandincluded selectionorandomly f chosenwordsfromtheorigina set.TheSVDprocedurewasappliedonboththe smallermatrixof2500wordsaswellasthesetof 5018words. AswiththeSVDprocedure,thechoiceofthe numberofdimensionsinthespaceisimportant. Havingtoofewdimensionsotroomanydimensions might lead to suboptimal performance when predictingperformanceinvariousmemorytasks.As withtheSVDscalingprocedure,thenumberof dimensions was variedbetween 10and500.

RecognitionMemory:SemanticSimilarity Ratings Inastudybythefirsttwoauthors(Steyvers& Shiffrin,submitted:Experiment1),89subjects studied144wordsthatcontained18semantic categoriesof 5wordseach.Basedonastudyby BrainerdandReyna(1998),subjectsgavetworating foreachof100testitems.Inonerating,theywer instructedtojudgewhethertheitemwasoldornew and were told to judge semantically similar distractorsas“new”.Inanotherrating,theywere instructedtorate(onasixpointscale)how semanticallysimilartheitemwastothestudied items.Wefocusedonthesemanticsimilarityrating forthenewitemsfromthisstudy.Foreachsubject the72newtestitemswererandomlyselectedfroma largerpoolof144words.Anaverage (SD=4.87)subjectsratedthesemanticsimilarityfo eachofthe144wordsthatmightappearasnew

d,

a l

6

s e

s , of44 r

7 wordsinthetestlist The . semanticsimilarity ratings aretheoreticallyinterestingbecausetheycanbeu totestmodelsosemantic f similarity.Subjectsmer havetorememberhowsimilartheitemwastothe studieditemswithoutbeingforcedtogiveold-new judgmentsthatmightbemoreinfluencedbyvarious strategicretrievalfactors(suchaw s ordfrequency previous retrievals). Manymemorymodelsassumethatarecognition memoryjudgmentiproduced s bycalculatingthe globalfamiliarityinvolvingthesummedsimilarity betweenthetestitemandtheepisodictracesin

memory(e.g.MinervaII,Hintzman1984,1988; SAM,Gillund&Shiffrin,1984).Morerecently, ShiffrinandSteyvers(1997,1998)andMcClelland & Chappell (1998) have proposed recognition memorymodelsthatproducerecognitionjudgments withBayesiandecisionprocesses.McClelland& Chappell(1998)proposedthatthebestmatch(i.e., maximumsimilarity)betweenthetestitemandthe episodictracesinmemoryformsthebasisforthe recognitionjudgment.Shiffrin&Steyvers(1998) showedthatintheBayesianframework,amaximum similarityprocessproducedresultsverysimilarto

sed ely

or

Recognition SemanticSimilarityRatings

FreeRecall P( - intrusion)

0.8

0.9

0.7

0.8

0.6

0.7 0.6

0.5 R

a

0.5

0.4

0.4 0.3 0.3 0.2

0.2

0.1

0.1

0.0 0

100

200

300

400

500

0.0

2500

0

100

200

300

400

500

2500

500

2500

SimilarityRatings

CuedRecall P(correct) 0.6 0.4 0.5 0.3

R

0.4 0.3

0.2 0.2 0.1 0.1 0.0

0.0 0

100

200 300 400 Dimensionality WAS SVD - S WAS SVD - S WAS MDS -

(1) (2)

500

0

2500

100

LSA encyclopedia LSA tasa -

200 300 400 Dimensionality S(1) S(2)

Figure 2.Correlationsodf ifferentmeasuresosfemanticsimilarityfordiffer entdimensionalities.Dataare takenfromrecognitionmemory,cuedrecall,freere call,andsimilarityratingsexperiments.Seetext details.

7

for

procedureandthesewerebeneficialinpredictingt he ratings from this recognition memory experiment. Themetric-MDSsolutionshowsquiteadifferent patternofresultsthantheSVDsolution.Thebest correlationwasobtainedwith20-40dimensions whichismuchlowerthanthenumberodf imensions typicallyneededintheSVDsolutionsoeither f WAS orLSA.AlthoughthebestcorrelationformetricMDSwas0.6asopposedto0.7fortheSVDbased solutions, it is interestingthat relativelygood performancecanbeachievedinsemanticspacesthat areoflowdimensionality.Althoughspecifyingwhy thiseffectoccursios utsidethescopeothis f pap er,it couldberelatedtotheestimatesinvolvingthe shortest associative path between words. As describedintheprevioussection,inordertoappl y metric-MDS,estimateswereneededforthedistances between allwordpairsinthevocabulary.The shortestassociativepathdistancewasproposedto meetthisrequirement;estimateswereevengenerate d forwordpairsthatwerenotassociateddirectlyor evenindirectlythroughachainoftwoassociates. In SVD,nosuchestimatesarerequiredandthoseentri es wereleftatzero.Itispossiblethen,thatthefi llingin processoall f wordpairdissimilaritiesbythesho rtest associativepathdistanceshelpedintheglobal in thesemanticspace. placementof allwords OfthetwocorperainLSA,thetasacorpusledto muchbetterperformancethantheencyclopedia corpus.Thisdifferenceinsotsurprisingsincethe tasa corpusincludesmaterialthatreflectsmuchmore closelythereadingmaterialanEnglishreaderis exposedtowhichinturnmightleadtosemantic spacesthataremorepsychologicallyplausiblein termsofpredictingsemanticsimilarityeffectsin recognitionmemory.ComparingWAStoLSA,it becomesclearthatWASleadstomuchhigher correlations than LSA. We will leave the interpretation othis f finding for thediscussion.

summedsimilarityprocess.Inthisresearch,ourai m isnottotestthesemodelsspecificallybuttouse and simplify the underlying mechanisms to predict semanticsimilarity ratings. BasedontheglobalfamiliarityandBayesian recognitionmemorymodels,thesemanticsimilarity ratingsintherecognitionmemoryexperimentshould havebeencorrelatedwiththe sumor maximumof thesimilaritybetweenthetestitemandallstudy words. To facilitate comparisons, all expected negativecorrelationswereconvertedtopositive correlations. Becausetheresultswereverysimilarfor thesumandmaximumcalculations,wewilllistonly theresults for themaximum calculation. ThetopleftpanelofFigure2showsthe correlations between maximum similarity and numberofdimensions(10-500)forthethreeWAS andtwoLSAbasedmeasures.FortheSVDbased semantic spaces, increasing the number of dimensionsineitherWASorLSAincreasesthe correlation generally up to around 200-300 dimensions.ForWAS,anadditionaldatapointwas plottedfor2500dimensionswhichisthemaximum number of dimensions given that the matrix containedonly2500words.Althoughaspacewas availableforall5018wordsinthefreeassociatio n database,thevocabularywasrestrictedto2500 wordsinordertoabletocompareittothemetric MDSsolutions(seeprevioussection).Thisdatapoi nt for 2500 dimensions was included because it representsthecasewherenoneoftheindirect relationshipsinwordassociationmatrixareexploi ted and as such, no dimensionality reduction is performed.Ascanbeobserved,thecorrelationis lowerfor2500dimensionindicatingthatsome dimensionalityreductionisneededtopredictthe semanticsimilarityratings.Also,theSVDbasedon S(2)ledtobettercorrelationsthantheSVDbasedon S(1)This . impliesthataddingtheindirectassociation s inameasureforassociativestrengthhelpsin predictingrecognitionmemoryperformance.Thetwo horizontallinesintheplotindicatethecorrelati on (1) whentheassociativestrengthsS andS (2)areusedas ameasureforsemanticsimilarity.Thecorrelation is (2) higherforS thanS (1)whichagainimpliesthatin recognition memory, the indirect associative strengths help in predicting performance. (2) Interestingly,theSVDscalingofS gavehigher (2) correlations than associative strengths S (2) themselves.EventhoughS includestheforward, backwardandalltwostepassociativestrengths, applyingtheSVDandreducingtheredundanciesin (2) thematrixoSf helpedtoincreasethecorrelation.In otherwords,theindirectrelationshipsandpattern os f correlationsthatgobeyondthoseofthetwostep associativestrengthswereutilizedbytheSVD

PredictingExtralistCuedRecall In extra-list cued recall experiments, after studyingalistow f ords,subjectsarepresentedwi cuesthatcanbuesedtoretrievewordsfrom thest list.Thecuesthemselvesarenovelwordsthat notpresentedduringstudy,andtypicallyeachword isassociativelyand/orsemanticallyrelatedtoone thestudiedwords.Thedegreetowhichacueis successfulinretrievingaparticulartargetwordi measureofinterestbecausethismightberelatedt theassociative/semanticoverlapbetweencuesand theirtargets.Researchinthisparadigm(e.g.,Nel &Schreiber,1992;Nelson,Schreiber,&McEvoy, 1992;Nelson,McKinney,Gee,&Janczura,1998; Nelson & Zhang, 2000) has shown that the associativestrengthbetweencueandtargetisone

8

th udy were of sa o son

importantpredictorforthepercentageofcorrectly recalledtargets.Therefore,weexpectthattheWAS similaritybetweencuesandtargetsarecorrelated withthepercentagesofcorrectrecallinthese experiments.Weusedadatabasecontainingthe percentagesofcorrectrecallfor1115cue-target 8 pairsfrom over29extralistcuedrecallexperiments fromDougNelson’slaboratory(Nelson,2000; Nelson & Zhang,2000). Thecorrelationsbetweenthevariousmeasuresfor semanticsimilarityandtheobservedpercentage correctrecallratesareshowninthebottomleftp anel ofFigure2Overall, . theresultsarevery similar tothe results obtained for the recognition memory (2) experiment.TheWASspacebasedonS ledto (1) betterperformancethantheWASspacebasedonS . (2) Also,theassociativestrengthsS leadstobetter (1) performancethentheS associativestrengths.These findingsare consistentwithfindingsbyNelson& Zhang(2000)thatshowthattheindirectrelationsh ips inwordassociationnormscanhelpinpredicting cuedrecallperformance.Interestingly,theplotal so (2) showsthattheWASspacebasedonS does (2) somewhatbetterthantheassociativestrengthsS it wasbasedon.Thisadvantageimplies thatapplying dimensionalityreductiontomakegreateruseofthe indirectassociativeconnectionshelpedinpredicti ng cuedrecall.Finally,aswiththerecognitionresul ts, theWASspacecorrelatesbetterwithcuedrecallth an LSA. PredictingIntrusion Rates inFreeRecall InaclassicstudybyDeese(1959b),thegoalwas topredicttheintrusionratesow f ordsinfreerec Fiftyparticipantsstudiedthe12strongestassocia toeachof36criticallureswhilethecriticallur themselveswerenotstudied.Inafreerecalltest, somecriticallures(e.g.“sleep”)werefalselyrec about40%ofthetimewhileothercriticallures(e “butterfly”)wereneverfalselyrecalled.Deesewas abletopredicttheintrusionratesforthecritica onthebasisotfheaverageassociativestrengthfr thestudiedassociatestothecriticalluresand obtainedacorrelationofR=0.80.BecauseDeese couldpredictintrusionrateswithwordassociation norms,theWASvectorspacederivedfromthe associationnormsshouldalsopredictthem.Critica itemswithhighaveragesimilarity(orlowaverage distance)tothelistwordsinthesemanticspace shouldbemorelikelytoappearaisntrusionsinfr recall.Theaveragesimilarity(averagedistance)w computedbetweeneachcriticallurevectorandlist wordvectors,andthecorrelationswerecomputed betweenthesesimilaritiesandobservedintrusion rates.

ThetoprightpanelinFigure2showstheresults. Thepatternofresultsisquitedifferentthan the patternoresults f foreitherrecognitionocued r r ecall. (1) Thebestcorrelationof0.82wasobtainedwithS , the sumofbackward and forward associative strength.Thisresultisverysimilartothecorrel ation of0.80Deeseobtainedwithhiswordassociation norms. Interestingly, the plot shows that any manipulationthatincludestheindirectassociation s leadstoworseperformancethanusingthedirect (2) associationsonly.TheWASspacebasedonS now (1) doesworsethantheWASspacebasedonS a, nd eitherspacecorrelatesmorepoorlythanwhenusing (1) theassociativestrengths S andS (2)themselves. Thesefindingsimplythatdirectassociative strengthsarethebestpredictorsoifntrusionrate isn freerecall. Oneexplanationforthisfindingirs elated toimplicitassociativeresponses(IAR’s).Underwoo d (1969)hasarguedthatduringstudy,thewords associatedwiththestudywordsarethoughtofand mightbestoredinmemoryaasnimplicitassociativ e response.InDeese’sstudy,itislikelythatIAR’s weregeneratedbecausethecriticallureswereall stronglyassociatedtothelistwords.Therefore, duringrecall,thewordsthatwereactuallypresent ed andwordsthatwerethoughtofduringstudymightb e confusedleadinginsomecasestodramaticintrusio n rates. Because free associations measure what responsesarethoughtofgivenspecificcues,the directassociativestrengthscanbearguedtobego od predictorsofthestrengthofimplicitassociative responses andsubsequentintrusion rates.

all. tes es

Similaritiesanddifferencesbetween WASandfree associationnorms Because WAS places words in a multidimensionalspacebasedonthepatternointe f correlationsinfreeassociation,thesimilarities differencesofWASandthefreeassociationnorms needstobedetermined.Intheprevioussection,it wasestablishedthatinrecognitionandcuedrecall WASleadstosomewhatbettercorrelationswith observedresultsthantheassociativestrengthsit basedon.Inthissection,thesimilaritystructure WASisinvestigatedandcomparedtothefree associationnorms.First,weinvestigatethedegree whichtheneighborhoodsimilaritystructureinWAS canbue sedtopredicttheorderofresponsestreng infreeassociation.Then,weaddresstheissueof whatkindofrelationshipWAScaptures:semantic, associativeorboth.Finally,weassesswhether asymmetriesinthedirectionofassociationcanbe predictedbytheneighborhoodsimilaritystructure WAS.

alled .g. lures l om

l

ee as

9

rand

, was of to ths

in

PredictingtheOutputOrderoFree f Association Norms BecausethewordvectorsinWASarebased explicitlyonthefreeassociationnorms,itisof interesttocheckwhethertheoutputorderof responses(intermsofassociativestrength)canbe predictedbyWAS.Tosimplifytheanalysis,the resultsareonlypresentedforWASbasedonS tookthisspacebecauseiperformed t wellinallth memorytasksandbecausewehadasolution availableforall5000+wordsappearinginthefree associationnorms(themetricspacewaslimitedto vocabulary o2500 f words).

rankofthesimilarityoaspecific f cue-responsep air wascomputedbyrankingthesimilarityamongthe similaritiesofthespecificcuetoallotherpossi ble nd responses.Forexample,theword‘crib’isthe2 closestneighborto‘baby’inWAS,so‘crib’hasa rankof2forthecue‘baby’.Inthisexample,WAS hasput‘baby’and‘crib’closertogetherthanmigh t beexpectedonthebasisofreeassociationnorms. Averagedacrossthewordsinthecorpus,Table2 givesforeachothe f firsttenrankedresponsesin free association(thecolumns)themedianrankinWAS. Themedianwasusedtoavoidexcessiveskewingof theaveragebyafewhighranks.Anadditional variablethatwastabulatedinTable2isk,the number of dimensions of WAS. TherearethreetrendstobediscernedinTable2. First,itcanbeobservedthatfor400dimensions, the strongestresponsestothecuesinfreeassociation normsarepredominantlytheclosestneighborstoth e cues inWAS.Second,responsesthathavehigher ranksinfreeassociationhaveonaveragehigher ranksinWAS.However,theoutputranksinWAS areinmanycasesfarhigherthantheoutputranks in freeassociation.Forexample,with400dimensions, thethirdrankedresponseinfreeassociationhasa medianrankof10inWAS.Third,forsmaller dimensionalities,thedifferencebetweentheoutput order in freeassociation andWASbecomes larger. Tosummarize,givenasufficientlylargenumber of dimensions, the strongest response in free associationisrepresented(inmostcases)asthe closestneighborinWAS. Theothercloseneighbors inWASarenotnecessarilyassociatesinfree association(atleastnotdirectassociates). Wea lso analyzedthecorrespondencebetweenthesimilaritie s intheLSAspace(Landauer&Dumais,1997)based onthetasacorpuswiththeorderofoutputinfree association. AscanbeobservedinTable2the , ra nk oftheresponsestrengthothe f freeassociationno rms clearlyhasaneffectontheorderingosimilariti f esin LSA:strongassociatesarecloserneighborsinLSA than weak associates. However, the overall correspondencebetweenpredictedoutputranksin LSAandranksinthenormsisweak.Theoverall weakercorrespondencebetweenthenormsand similaritiesfortheLSAapproachthantheWAS approachhighlightsoneobviousdifferencebetween thetwoapproaches.BecauseWASibsasedexplicitly onfreeassociationnorms,iitsexpectedandshown herethatwordsthatarestrongassociatesareplac ed closetogetherinWASwhereasinLSA,wordsare placedinthesemanticspaceinawaymore independentfrom thenorms. Togetabetterideaofthekindsofneighbors words haveiW n AS,in Table3we , listthefirstfi ve

(2)

We . ree

a

Table 2 Median . Rank othe f Output-order in WAS andLSA of Response WordstoGiven Cuesfor the10 StrongestResponses in theFreeAssociation Norms. Rank oResponse f iF n reeAssociation k

1

2

3

4

5

6

7

8

9 10

WordAssociation Space(WAS) 10

86 187 213 249 279 291 318 348 334 337

50

13

36 49 62 82 98 106 113 125 132

100

6

17 26 36 43 62 65 73 78 85

200

3

8 15 20 28 39 40 48 56 58

300

2

6 12 16 21 31 35 38 43 49

400

1

5 10 14 19 27 32 35 38 44 LatentSemanticAnalysis(LSA)

10

733 798 846 845 922 897 903 920 939 955

50

231 313 371 422 475 494 526 510 559 583

100

115 193 256 307 359 384 411 413 451 463

200

63 125 185 225 285 319 347 339 389 395

300

46

99 159 197 254 294 321 324 375 374

400

37

90 149 185 239 278 310 308 349 366

theseanalyses,WASwasbasedotnhesvdof Note:In S(2) and LSA wasbasedotnhetasa corpus

Wetookthe10strongestresponsestoeachothe f cuesinthefreeassociationnormsandrankedthem accordingtoassociativestrengths.Forexample,th response‘crib’isthe8thstrongestassociateto‘ inthefreeassociationnorms,so‘crib’has rank a forthecue‘baby’.UsingthevectorsfromWAS,the

e baby’ of8

10

Table3. TheFiveNearest Neighborsin WAS for theFirst 40 (1954)Norms.

Cue

1

cuesin theRussell& Jenkins

Neighbor 3

2

5

4

afraid

scare(1)[7]

fright(4)[14]

fear(2)[1]

scared[2]

ghost(5)[106]

anger

mad(1)[1]

Angry

rage(5)[4]

enrage

fury[21]

baby

child(1)[2]

crib(8)[13]

infant(6)[7]

cradle

diaper(13)

bath

clean(2)[1]

soap(7)[3]

water(3)[2]

dirty[7]

suds[49]

beautiful

pretty(1)[2]

ugly(2)[1]

cute[39]

girl(4)

flowers[10]

bed

sleep(1)[1]

tired(11)[13]

nap

rest[5]

doze

bible

god(1)[1]

church(3)[3]

religion(4)[4]

Jesus(5)[8]

book(2)[2]

bitter

sweet(1)[1]

sour(2)[2]

Candy

lemon(5)[7]

chocolate[4]

Black

white(1)[1]

Bleach

color(3)[7]

dark(2)[2]

minority

blossom

flower(1)[1] petals[46]

Rose(5)[7]

tulip

daisy

blue

color(5)[4]

red(3)[2]

Jeans

crayon

pants

boy

girl(1)[1]

Guy

Man(4)[2]

woman

nephew[54]

bread

butter(1)[1] toast[19]

rye[26]

loaf(3)[16]

margarine

butter

bread(1)[1]

rye

peanut

margarine(2)[34]

butterfly

bug(15)[10] insect(6)[2]

fly(4)[5]

roach[76]

beetle

toast(6)[18]

cabbage

green(4)[7]

food(10)[4]

vegetable(2)[3] salad(12)[5]

vegetables

carpet

floor(2)[2]

tile(15)

rug(1)[1]

ceiling

sweep[14]

chair

table(1)[1]

seat(4)[4]

sit(2)[2]

couch(3)[20]

recliner

cheese

cracker(2)

cheddar(6)[23]

Swiss(7)[19]

macaroni[39]

pizza

child

baby(1)[1]

kid(2)[7]

adult(3)[3]

young(8)[6]

parent(6)[11]

citizen

person(1)[3] country(3)[5]

people[7]

flag[12]

American(2)[4]

city

town(1)[1]

state(2)[3]

country(9)[4]

New York(4)

Florida

cold

hot(1)[1]

ice(2)[5]

warm(6)[3]

chill

pepsi

comfort

chair(3)[1]

Table

seat

couch(2)[26]

sleep[7]

command

tell(4)[7]

army(5)[2]

rules

navy[17]

ask[22]

cottage

house(1)[1]

home(4)[4]

cheese(2)[3]

cheddar

Swiss

dark

light(1)[1]

Bulb

night(2)[2]

lamp

day

faucet

pool[53]

splash

medical[83]

stethoscope[21]

nap

tired[92]

deep

water(3)[3]

ocean(2)[6]

doctor

nurse(1)[1]

physician(5)[15] surgeon(6)

dream

sleep(1)[1]

fantasy(4)[19]

bed[7]

eagle

bird(1)[1]

Chirp

bluejay

earth

planet(2)[8] mars[14]

Jupiter[97]

Venus[50]

eating

food(1)[1]

eat[30]

hungry(3)[4]

restaurant[75] meal[30]

foot

shoe(1)[1]

sock[16]

fruit

orange(2)[3] apple(1)[1]

nest(10)[5]

sparrow[30] Uranus

toe(2)[3]

sneaker

leg(5)[4]

juice(9)[12]

citrus[35]

tangerine[55] pretty(4)[6]

girl

boy(1)[1]

guy(6)

man[9]

woman(3)[2]

green

grass(1)[1]

lawn[41]

cucumber

vegetable[76] spinach[76]

hammer

nail(1)[1]

tool(2)[7]

wrench

screwdriver

pliers[21]

hand

finger(1)[2] arm(3)[3]

foot(2)[1]

leg(13)[11]

glove(4)[4]

hard soft(1)[1] easy(3)[3] Note: numbers in parentheses and squarebracketsi al.(1998)andRussell& Jenkins (1954)respectivel

difficult[19] difficulty simple ndicateranksof responsesin norms of Nelson et y.

neighborsinWAS(using400dimensions)to40

11

cue words taken from RussellandJenkins (1954). Forallneighborslistedinthetable,iftheywere associatesin thefreeassociationnormsoN f elsonet al.,thenthecorrespondingrankinthenormsigsi inparenthesesandthosefromRussellandJenkinsa showninbrackets.Thecomparisonbetweenthese twodatabasesisinterestingbecauseRusselland Jenkinsallowedparticipantstogenerateasmany responsestheywantedforeachcuewhilethenorms ofNelsonetal.containfirstresponsesonly.We suspectedthatsomecloseneighborsinWASarenot directassociatesintheNelsonetal.normsbutth theywould havebeenvalidassociatesipf articipants hadbeenallowedtogivemorethanoneassociation percue.InTable4,welistthepercentagesof neighborsinWASofthe100cuesothe f Russelland Jenkinsnorms(only40wereshowninTable3t)hat arevalid/invalidassociatesaccordingtothenorms Nelson eal. tand/or thenorms of RussellandJenki Table4. Percentages of Responses of WASmodel that areVali in Russell & Jenkins (1954) andNelsoneal. t(1999

ven re

at

of ns.

d/Invalid Norms )

1

2

Neighbor 3

4

5

validiN n elsoneal. t

96

73

61

45

33

validiJnenkins etal.

96

83

79

69

64

validienither Nelson eal. t or Jenkins etal.

99

86

82

73

66

InvalidinNelson eal. tbut validiJnenkins etal.

3

Validity

throughachainoafssociates.Forexample,thepai rs ‘blue-pants’‘butter-rye’, , ‘comfort-table’arecl ose neighborsinWASbutarenotdirectlyassociated witheachother. ItislikelythatbecauseWASis sensitivetoindirectrelationshipsinthenorms,t hese wordpairswereputclosetogetherinWASbecause oftheindirectassociativelinksthroughthewords ‘jeans’,‘bread’and‘chair’respectively. Inasimilar way,‘cottage’and‘cheddar’arecloseneighborsin WASbecausecottageisrelated(inonemeaningof theword)to‘cheese’,whichisanassociateof ‘cheddar’. Tosummarize,weshowedthattheoutputorder ofwordsinfreeassociationnorms ispreservedto somedegreeinWAS:firstassociatesinthenorms arelikelytobecloseneighborsinWAS.However, therearesomeinterestingdifferencesbetweenthe similaritystructureofWASandtheassociative strengthsotfhewordsinthenorms.Wordsthatare notdirectlyassociatedcanbecloseneighborsin WASwhenthewordsareindirectlyassociatively relatedthroughachainofassociates.Also,althou gh theyappeartobeexceptions,somewordsthatare directlyassociatedinthenormsarenotclose neighborsinWAS.Becauseofthesedifferences, WASisnotanexceptionallygoodmodelforthetask ofpredictingfreeassociationdata.However,itis importanttorealizethatWASwasnotdevelopedas a model offreeassociation(e.g.Nelson&McEvoy, Dennis,2000)butratherasamodel basedon free association.

13

21

Thelastrowshowsthatathirdofthe5 neighborsinWASarenotassociatesaccordingtoth normsofNelsonetal.butthatareassociates accordingtothenormsofRussellandJenkins. Therefore,somecloseneighborsinWASarevalid associatesdependingonwhatnormsareconsulted. However,somecloseneighborsinWASarenot associatesaccordingtoeithernorms.Forexample, nd ‘angry’itshe2 neighborof‘anger’inWAS.These wordsareobviouslyrelatedbywordform,butthey donottoappearasassociates in freeassociation becauseassociationsotfhesamewordformtendto beeditedoutbyparticipants.However,becausethe wordshavesimilarassociativestructures,WASputs them closetogether in thevector space. Also,somecloseneighborsinWASarenotdirect associatesoef achotherbutareindirectlyassocia

28

Semantic/AssociativeSimilarityRelations Intheprimingliterature,severalauthorshave triedtomakeadistinctionbetweensemanticand associativewordrelationsinordertoteaseapart differentsourcesofpriming(e.g.Burgess&Lund, 2000;Chiarello,Burgess,Richards&Pollock,1990; Shelton&Martin,1992).BurgessandLund(2000) have argued that the word association norms confoundmanytypesoword f relations,amongthem, semanticandassociativewordrelations.Chiarello al.(1990)give“music”and“art”asexamplesof wordsthataresemanticallyrelatedbecausethewor areratedtobemembersofthesamesemantic category(Battig&Montague,1969).However,they claimthesewordsarenotassociativelyrelated becausetheyarenotdirectassociatesoef achothe (accordingtothevariousnormdatabasesthatthey used).Thewords“bread”and“mold”weregivenas examplesow f ordsthatarenotsemanticallyrelated becausetheyarenotratedtobm e embersothe f sam semanticcategorybutonlyassociativelyrelated (because“bread”ias nassociateo“fmold”).Finall “cat”and“dog”weregivenasexamplesofwords thatareboth semantically andassociatively relate

33

th

closest e

tasks se

ted

12

et ds

r

e y, d.

between WordPairs with Differe Table5. AverageSimilarity

ntRelations: Semantic,Associative,andSemantic+

Associative k

Relation Random

S #Pairs

* ij

200 .000(.000)

10 0.34

50 (0.277)

0.075

200

400

(0.178)

0.024

(0.064)

0.017 (0.048)

0.215 (0.321)

Wordpairs from Chiarelloeal. t(1990) Semanticonly

33 .000(.000)

0.73

(0.255)

0.457

(0.315)

0.268

(0.297)

Associativeonly

43 .169(.153)

0.902

(0.127)

0.83

(0.178)

0.712

(0.262)

0.666 (0.289)

0.962

(0.053)

0.926

(0.097)

0.879

(0.180)

0.829 (0.209)

Semantic+Associative

44 .290(.198)

Wordpairs from Shelton andMartin (1992) Semanticonly Semantic+Associative

26 .000(.000) 35 .367(.250)

Note:standard deviations given between parentheses *Sij =averageforwardand backward associativestrength

0.724

(0.235)

0.448

(0.311)

0.245

(0.291)

0.166 (0.281)

0.926

(0.088)

0.929

(0.155)

0.874

(0.204)

0.836 (0.227)

A = (

+A ij

ji2/)

aresemanticallyonlyandassociativelyonlyrelate Forhigherdimensionalities,thisdifferencebecome larger as WAS becomes more sensitive in representing more of the direct associative relationships. Toconclude,itisdifficulttodistinguishbetween puresemanticandpureassociativerelationships. Whatsomeresearcherspreviouslyhaveconsideredto bepuresemanticwordrelations,werewordpairsth wererelatedinmeaningbutthatwerenotdirectly associated witheachother.Thisdoesnotmean, however,thatthesewordswerenotassociatively relatedbecausetheinformationinfreeassociation norms goes beyond that of direct associative strengths.Infact,thesimilaritystructureofWAS turnsouttobesensitivetothesimilaritiesthat arguedbysomeresearcherstobepurelysemantic. Wewillillustratethispointwiththefollowing experiment.

Weagreethattheresponsesinfreeassociation normscanberelatedtothecuesinmanydifferent ways, but it seems very hard and perhaps counterproductivetoclassifyresponsesaspurely 9 semanticorpurelyassociative F . orexample,word pairsmightnotbedirectlybutindirectlyassociat ed throughachainofassociates.Thequestionthen becomes,howmuchsemanticinformationdothefree association norms contain beyond the direct associations?BecauseWASissensitivetothe indirectassociativerelationshipsbetweenwords,w e tookthevariousexamplesofwordpairsgivenby Chiarelloeal. t(1990)andSheltonandMartin(199 2) andcomputedtheWASsimilaritiesbetweenthese wordsfordifferentdimensionalitiesasshownin Table5. InTable5the , interestingcomparisonibs etween thesimilaritiesforthesemanticonlyrelatedword pairs10(aslistedbyChiarelloetal.,1990)and200 randomwordpairs.Therandomwordpairswere selected to have zero forward and backward associativestrength. Itcanbeobservedthatthesemanticonlyrelated wordpairshavehighersimilarityinWASthanthe randomwordpairs.Therefore,eventhoughChiarello etal.(1990)havetriedtocreatewordpairsthat were onlysemanticallyrelated,WAScandistinguish betweenwordpairsthatwerejudgedtobeonly semanticallyrelatedandrandomwordpairsbothof which havingnodirectassociationsbetweenthem. ThisdiscriminationispossiblebecauseWASis sensitivetoindirectassociativerelationshipsbet ween words. The table also shows that for low dimensionalities,thereisnotasmuchdifference betweenthesimilarityow f ordpairsthatsupposedl y

Predictingsimilarityratings TofurtherassessthedegreetowhichWASis capable of capturing semantic similarity, the similaritystructure inWAS was compared to semanticsimilarityratingscollectedbyRomneyet (1993).Basedonthetriadsmethod,theyconstructe perceivedsimilaritymatricesfordifferentsetsof wordsthatfellinvarioussemanticcategories.Of thesecategories,wetookthe10matricesforthe1 categoriesofbirds,clothing,fish,fruits,furnit sports,tools,vegetables,vehicles,andweapons. Between23and28subjectsassessedthesimilarity memberswithineachcategory.Becausesubjects assessedthesimilarityformemberswithincategori (e.g.fordifferentbirds),wethoughtiw t ouldbe

13

d. s

at

were

al. d 21 0 ure, of es a

havearguedthatasymmetriesinjudgmentsarenot necessarilyduetoasymmetriesintheunderlying similarityrelationships.Withfreeassociationnor iftheassociationstrengthA->BisstrongerthanB >A,asimpleexplanationcouldbethatwordBhas higherwordfrequency,ismoresalientand/or,its representationismoreavailablethanwordA.To checkwhetherwordfrequencycanbuesedtoexplain asymmetriesinthefreeassociationnormsoN f elson etal.,wetookthe500strongestasymmetric associativewordpairsandassignedthemoreoften producedresponsetoB,andthelessoftenproduced responsetoA(inotherwords:A->Bisstrongertha B->A). InFigure3a,theKuceraandFrancis(1967) wordfrequencyofAisplottedagainsttheword frequencyoB. f Thediagonallinerepresentstheca wherethewordfrequenciesoA fandBareequal.It canbeseenthatmostofthesewordassociationsfa abovethediagonal(463outthe500,o9r 2.6%),so wordfrequencycanindeedbeusedtoexplainthe directionality oword f associations. Krumhansl(1978)hasproposedthatasymmetries canalsoarisebecauseodifferences f inthedensit exemplarsinpartsofthepsychologicalspace.For example,ifA->BisstrongerthanB->A,thiscould explainedbecauseBisplacedinadenseregionof thespacewhereasAisplacedimore na sparseregi ofthespace.Insamplingmodels,thesedifferences densitymightleadtodifferentresults.Inthepre example,samplingBtakingAasareferencewould beamorelikelyeventthansamplingAtakingBas referencewhenAhasfewcloseneighborsotherthan B,andBhasmanycloseneighborsotherthanAWe .

verychallengingtasktocapturethewithincategor y similaritystructure(asjudgedbysubjects)with methods such aWAS s oLSA. r The correlation was calculated between all observedsimilaritiesandpredictedsimilaritiesus ing thedifferentmethodsoW f AS,associativestrengths andLSA.Theresultfordifferentdimensionalities is shown in Figure 2, bottom right panel. All correlations were relativelylowwith the best correlationof0.36obtainedforWASbasedonthe SVDofS (1).Interestingly,thesemanticspacesof WASledtohighercorrelationsthanLSA.This difference provides further evidence that the similaritystructureinwordassociationnormsand the similaritystructureextractedbyWASfromtheword associationnormsdoescontainmanyaspectsof semantic similarity. Therefore, the similarity structureinwordassociationnormsoW r AScannot besimplycategorizedas ‘associative’,norcanone concludethatthereino ssemanticstructure.

PredictingAsymmetries inWordAssociation Norms Inthefreeassociationnorms,alargenumberof wordassociationsareasymmetric.Forexample,the cue“fib”isstronglyassociatedwith“lie”,butno viceversa.InWAS,thedistancebetweenwordA andBisbydefinitionequivalenttothedistance betweenwordsBandAThis . characteristicraisest questionofhowweareabletoaccountforthe asymmetries in free associations with WAS. Nosofsky(1991),andAshandEbenholtz(1962)

10000

t

he

1000

(a)

(b)

Rank(A,B)

FrequencyB

1000

100

10

1

100

10

1 1

10

100

1000

10000

1

FrequencyA

10

100

1000

Rank(B,A)

Figure 3.In(a),theKucera&Franciswordfrequency ipsl ottedf orwordsA versus words Bof wordpairs that arestronglyassociatedinthedirectionA>Bbutnotviceversa.In(b),theplotshowsthat forwordpairsthat arestronglyassociatedfromA->Bbutnotvicevers a,Bisoftencloser a neighbortoAthanAis toBinWAS whenmeasuredbyoutputrank(A,B):thenumberonf e ighborsinterveningbetweenAandBwhenAistaken

14

ms, -

n

se ll

yof be on in vious a

similarityratingsinrecognitionmemory,percentag e correctinextracuedrecallandintrusionratesin free recall.Inallthesememorytasks,WASwasabetter predictorforperformancethanLSA.Thissuggestst o usthatWASformsuseful a representationalbasisf or memorymodelsthataredesignedtostoreand retrievewords as vectors of featurevalues. Manymemorymodelsassumethatthesemantic aspectsowords f canbreepresentedbycollections of featuresabstractlyrepresentedbyvectors(e.g.Ei ch, 1982;Murdock,1982;Pike,1984;Hintzman,1984, 1988;McClelland&Chappell,1998;Shiffrin& Steyvers,1997,1998).However,inmostmemory modeling, the vectors representing words are arbitrarilychosenandarenotbasedonoderived r by someanalysisotfhemeaningoaf ctualwordsinour language.Weexpectthatmemorymodelsbasedon thesesemanticvectorsfromWASwillbeusefulfor makingpredictionsabouttheeffectsofvarying semantic similarityinmemoryexperimentsfor individualwords. ForthefirsttwoversionsoWAS f thatwerebased onsingularvaluedecomposition,thenumberof dimensionsotfhesemanticspacethatledtothebe st fitswithobservedperformancevariedbetween200 and500dimensions.Asimilarrangeinthenumber ofdimensionshasshowntobeeffectiveintheLSA approach(Landauer&Dumais,1997).Interestingly, the third version of WAS based on metric multidimensionalscalingachievednearlythesame performancebutwithonly20-40dimensions.This suggeststhatthesemanticvectorsinmemorymodels mightinvolverelativelyfewfeaturevaluestocapt ure thesemanticsimilaritystructure forafewthousand words. WeproposethatWASisanapproachthat augments other existing methods available for placingwordsinapsychologicalspace.Itdiffers fromtheLSAandHALapproachesinseveralways. LSAandHALareautomaticmethodsand donot requireanyextensivedatacollectionofratingsor freeassociations.WithLSAandHAL,tensof thousandsofwordscanbeplacedinthespace, whereasinWAS,thenumberow f ordsthatcanbe placeddependsonthenumberow f ordsthatcanbe normed.It ookNelsonetal.(1999)morethana decade to collect the norms, highlighting the enormoushumanoverheadofthemethod.Even thoughaworkingvocabularyoabout f 5000wordsin WASismuchsmallerthanthe70,000wordlong vocabulariesoL f SAandHAL,webelieveiis tlarge enoughforthepurposeom f odelingperformancein varietyofmemoryexperiments.Anadvantageof LSAandHAListhattheseapproacheshavethe potentialtomodelthelearningprocessthata languagelearnergoesthrough.Forexample,by

checkedwhetherthismechanismcanexplainthe directionality of the 500 asymmetric word associations.Insteadofcalculatingdensityinloc al regionsoWAS, f wecalculatedtwonumbersforeach of the 500 asymmetric word pairs. First, we calculatedtherankofthesimilarityofA-Bamong thesimilarityofAtoallotherwords(thiswillb e denotedbyrank(B,A))andamongthesimilarityoB f to all other words (this will be denoted by nd rank(A,B)).Forexample,ifBisthe2 neighborof rd AandAisthe3 neighboroB f then , rank(B,A)=2 andrank(A,B)=3.InFigure3b,rank(A,B)ips lotted againstrank(B,A). Thediagonalindicatesthecase s wherethenumberofneighborsseparatingAandBis thesameforAandBMost . ofthecases(486outof 500,or97.2%)lieabovethediagonal,meaningthat inWAS,BisclosertoAthanAistoBinthesens e ofnumberofneighborsseparatingAandB.This meansthatKrumhansl’s(1978)ideaodifferences f i n densitycanbeappliedtoWAStoexplainthe directionality oword f associations.

Discussion Ithasbeenproposedthatvariousaspectsowords f canbreepresentedbyseparatecollectionsofeatu f that codefortemporal,spatial,frequency,modality, orthographic,acoustic,andassociativeaspectsof words(Anisfeld&Knapp,1968;Bower,1967; Herriot,1974;Underwood,1969;Wickens,1972).In this research, we have focused on the associative/semanticaspects of words. Bystatistical a analysisolafargedatabaseofr f associationnorms,theWordAssociationSpace (WAS)wasdeveloped.Inthisspace,wordsthathave similarassociativestructuresareplacedinsimila regionsofthespace.InthefirstversionofWAS, singularvaluedecompositionwasappliedonthe directassociationsbetweenwordstoplacethese wordsinahighdimensionalsemanticspace.Inthe secondversionofWAS,thesametechniquewas appliedonthedirectandindirectassociations betweenwords.InthethirdversionofWAS,metric multidimensionalscalingwasappliedonmeasures fortheassociativestrengthrelatedtotheshortes associativepathbetweenwords(similartothe approachinCookeeal., t 1986andKlauer&Carroll 1995). Becausethefreeassociationnormshavebeenan integral part in predicting episodic memory phenomena(e.g.Cramer,1968;Deese,1965;Nelson, Schreiber,&McEvoy,1992),itwasassumedthata semanticspacebasedonfreeassociationnorms wouldbeanespeciallyusefulconstructtomodel memoryphenomena. WecomparedWASwithLSA inpredictingtheresultsofseveralmemorytasks:

res the

ee

r

t ,

15

feedingtheLSAorHALmodelsuccessivelylarger chunksoftext,theeffectthatlearninghasonthe similaritystructuresow f ordsinLSAorHALcanbe simulated.InWAS,itisinprinciplepossibleto modelalanguagelearningprocessbycollectingfre associationnormsforparticipantsadt ifferentsta ofthelearningprocess.Inpracticehowever,such approachwouldnoteasilybeaccomplished.To conclude,wethinkthattheWAS,LSA,andHAL approachestocreatingsemanticspacesareallusef fortheoreticalandempiricalresearchandthatthe usefulnessofaparticularspacewilldependonthe particular task iis applied t to.

3.Theresultoaf pplyingSVDtoanasymmetric wordassociationmatrixwouldbetwovectorspaces: thesewouldcapturethecommonalitiesamongthe cues(rows)andresponses(columns)respectively. Analysesshowedthatneitherotfhesevectorspaces capturesthevarianceinthememorytasksreported in thispaperasmuchavector as spacederivedaby t an SVDon symmetrized a wordassociation matrix. 4.Thenumberofdimensionsthatcanbe extractedisconstrainedbyvariouscomputational aspects.Wewereabletoextractonlythefirst500 dimensionsinthethreescalingsolutionsofWAS basedon5018wordsand400dimensionsinthe scalingsolutionbasedonall5018wordsinthefre e association database. 5.Variousalternativescalingproceduressuchas nonmetricMDScanhandlemissingvaluessothat notalldistancesinthematrixwouldhavetobe estimated.However,nonmetricMDScannotbe appliedtomatriceswiththousandsofrowsand columnssowewereforcedtowiththemetricMDS procedurethatworkswithlargematrices butthat does notallow for missing values. (1) In 6.fact,thematrixS has99.5%zeroentries respectivelysoestimatingthedistancesforthese zero entries is nontrivial. 7.Theratingwereconvertedtoz-scoresby subtractingthemeanratingfromeachratingandth en dividing by the standard deviation. This was performedforeachsubjectseparatelyandtheresul ts wereaveragedoversubjects.Thisprocedureremoved someoftheidiosyncratictendenciesofsubjectsto useonly partof therating. 8. Theoriginaldatabasehas2272cuetargetpairs. Weaveragedthepercentagecorrectresultsover identicalcue-targetpairsthatwereusedindiffer ent experiments.This gave1115uniquepairs. 9.Sinceresponsesinwordassociationtasksare bydefinitionallassociativelyrelatedtothecue, itis notclearhowitispossibletoseparatetherespon ses as semantically andassociatively related. 10.Somewordpairsinthesemanticonly conditionsthatwerenotdirectlyassociatedaccord ing tovariousdatabasesofreeassociationnormswere actuallydirectlyassociatedusingtheNelsonetal . (1998)database.Thesewordpairswereexcluded from theanalysis.

e ges an

ul

Author Note TheWordAssociationSpace(WAS)vectorscan bedownloadedinMatlabformatfrom(wwwaddress willbegiveninfuturedrafts).Wewouldliketo thankTomLandauerandDarrellLahamforkindly providing us with the LSA vectors for the encyclopediaandtasacorpus.Also,wewouldliket o acknowledgeJoshTenenbaum’shelpinfiguringout ways to applymetric multidimensional scaling techniques on the word association norms. Correspondence concerning this article can be addressedtoMarkSteyvers,PsychologyDepartment, StanfordUniversity,Stanford,CA94305-2130.Emailmay bseentto [email protected].

Notes 1.ThismethodwasdevelopedbyOsgood,Suci, andTannenbaum(1957).Wordsareratedonset a of bipolarratingscales.Thebipolarratingscalesar e semanticscalesdefinedbypairsopf olaradjective s (e.g.“good-bad”,“altruistic-egotistic”,“hot-cold ”). Eachwordthatonewantstoplaceinthesemantic spaceisjudgedonthesescales.Ifnumbersare assignedfromlowtohighforthelefttorightwor dof abipolarpair,thentheword“dictator”forexampl e, mightbejudgedhighonthe“good-bad”,highonthe “altruistic-egotistic”andneutralonthe“hot-cold ” scale.Foreachword,theratingsaveragedovera largenumberofsubjectsdefinethecoordinatesof the wordinthesemantic space.Becausesemantically similarwordsarelikelytoreceivesimilarratings , theyarelikelytobelocatedinsimilarregionsof the semanticspace.Theadvantageofthesemantic differentialmethodisitssimplicityandintuitive appeal.Theprobleminherenttothisapproachitsh e arbitrarinessinchoosingthesetofsemanticscale ass wellas thenumber of such scales. 2.BecausewewereunabletoobtaintheHAL vectors, we could not include HAL in the comparisons in this paper.

References Anisfeld,M.,&Knapp,M.(1968).Association, synonymity,anddirectionalityinfalse recognition. Journal of ExperimentalPsychology,77 171-179. , Battig,W.F.,&Montague,W.E.(1969).Category normsforverbalitemsin56categories:Areplicat ionand

16

extensionoftheConnecticutcategorynorms. Journalof ExperimentalPsychology Monograph,80(3) 1-46. , Brainerd,C.J.,&Reyna,V.F.(1998).Whenthingst hat wereneverexperiencedareeasierto“remember”th an , thingsthatwere. PsychologicalScience,9 484-489. Brainerd,C.J.,Reyna,V.F.,&Mojardin,A.H.(1999 ). , Conjointrecognition. PsychologicalReview,106 160-179. Bousfield,W.A.(1953).Theoccurrenceocf lusterin g intherecallofrandomlyarrangedassociates. Journalof GeneralPsychology,49 229-240. , Bower,G.H.(1967).Amulticomponenttheoryofthe memorytrace.InK.W.Spence&J.T.Spence(Eds.), The psychologyolfearningandmotivation Vol , 1.NewYork: Academic Press. Burgess,C.,Livesay,K.,andLund,K.(1998). Explorationsincontextspace:Words,sentences,di scourse. , Discourse Processes,25 211-257. Burgess,C.,&Lund,K.(2000).Thedynamicsof meaninginmemory.InE.DietrichandA.B.Markman (Eds.), Cognitive dynamics: conceptual and representational change in humans and machines . Lawrence Erlbaum. Chiarello,C.,Burgess,C.,Richards,L.,&Pollock A. , (1990).Semanticandassociativepriminginthecer ebral hemispheres: Some words do, some words don’t, …sometimes,someplaces. BrainandLanguage,38 7, 5104. Canas,J.J.(1990).Associativestrengtheffectsi nthe lexical decision task. The Quarterly Journal of ExperimentalPsychology,42,121-145. Caramazza,A.,Hersch,H.,&Torgerson,W.S.(1976) . Subjectivestructuresandoperationsinsemanticme mory. Journalofverballearningandverbalbehavior,15 1, 03117. Cooke,N.M.,Durso,F.T.,&Schvaneveldt,R.W. (1986).Recallandmeasuresofmemoryorganization. JournalofExperimentalPsychology:Learning,Memor y, andCognition,12 538-549. , Cramer,P(.1968). WordAssociation N . Y:Academic Press. Deese,J(. 1959a).Influenceofinter-itemassociat ive strengthuponimmediatefreerecall. PsychologicalReports, 5,305-312. Deese,J.(1959b).Onthepredictionoof ccurrences of particularverbalintrusionsinimmediaterecall. Journalof ExperimentalPsychology,58 17-22. , Deese,J.(1962).Onthestructureofassociative meaning. PsychologicalReview,69 161-175. , Deese,J.(1965). Thestructureofassociationsin languageandthought Baltimore, . MD:TheJohnsHopkins Press. Derweester,S.,Dumais,S.T.,Furnas,G.W.,Landaue r, T.K.,&Harshman,R.(1990).Indexingbylatentsem antic analysis. JournalotfheAmericanSocietyforInformation Science,41 391-407. , Eich,J.M.(1982).Acompositeholographic associat ive , recallmodel. PsychologicalReview,89 627-661. Gillund,G.,&Shiffrin,R.M.(1984).Aretrievalm odel forbothrecognitionandrecall. PsychologicalReview,91 , 1-67. Herriot,P.(1974). Attributesofmemory .London: Methuen.

Hintzman,D.L.(1984).Minerva2sa:imulationmode l of human memory. Behavior Research Methods, Instruments,andComputers, 16,96-101. Hintzman,D.L.(1988).Judgmentsoffrequencyand recognitionmemoryinamultiple-tracememorymodel . 528-551. , PsychologicalReview, 95 Jenkins,J.J.,Mink,W.D.,&Russell,W.A.(1958). Associativeclusteringasafunctionofverbalasso ciation , strength. PsychologicalReports,4 127-136. Johnson,R.A.,&Wichern,D.W.(1998). Applied multivariate statisticalanalysis New . Jersey,Prentice Hall. Krumhansl,C.L.(1978).Concerningtheapplicabilit y of geometric models to similarity data: The interrelationshipbetweensimilarityandspatialde nsity. , 463. PsychologicalReview,85 445, Klauer,K.C.,&Carroll,J.D.(1995).Networkmodel s forscalingproximitydata.In,R.D.Luce,M.D’Zmu ra,D. Hoffman,G.J.Iverson,&A.K.Romney(eds.), Geometric representationsofperceptualphenomena M . ahwah,New Jersey:Lawrence Erlbaum Associates. Kucera,H.,&Francis,W.N.(1967). Computational analysisopf resent-dayAmericanEnglish Providence, . RI: BrownUniversity Press. Landauer,T.K.,&Dumais,S.T.(1997).Asolutiont o Plato’sproblem:TheLatentSemanticAnalysistheor yof acquisition,induction,andrepresentationofknowl edge. PsychologicalReview,104 211-240. , Landauer,T.K.,Foltz,P.,&Laham,D(. 1998).An introduction to latent semantic analysis. Discourse Processes,25 259-284. , Lund,K.,&Burgess,C.(1996).Producinghighdimensionalsemanticspacesfromlexicalco-occurre nce. BehaviorResearchMethods,Instruments,andCompute rs, 28,203-208. Marshall,G.R.,&Cofer,C.N.(1963).Associative indicesasmeasuresofwordrelatedness:asummary and comparisonoten f methods. Journalof VerbalLearning and VerbalBehavior,1 408-421. , McClelland,J.L.,&Chappell,M(.1998).Familiarit y breedsdifferentiation:Asubjective-likelihoodapp roachto the effects of experience in recognition memory. PsychologicalReview,105 724-760. , Morton,J.A.(1970).Afunctionalmodelformemory. N . ew InD.A.Norman(Ed.), Modelsofhumanmemory York:Academic Press. Murdock,B.B.(1982).Atheoryforthestorageand retrievalofitemandassociativeinformation. Psychological Review,89 609-626. , Neely,J.H.(1991).Semanticprimingeffectsinvis ual wordrecognition: saelectivereview of currentfin dingsand theories.InD.Besner&G.W.Humphreys(Eds.),Bas ic processesinreading:Visualwordrecognition (pp.264336).Hillsdale,NJ:Lawrence Erlbaum Associates. Nelson, D.L. (2000). The cued recall database. http://www.usf.edu/~nelson/CuedRecallDatabase. Nelson,D.L.,Bennett,D.J.,&Leibert,T.W.(1997) . Onestepisnotenough:makingbetteruseoaf ssoci ation , normstopredictcuedrecall. Memory&Cognition,25 785-706. Nelson,D.L.,McEvoy,C.L.,&Dennis,S.(2000), Whatisfreeassociationandwhatdoesitmeasure? Memory & Cognition,28 887-899. ,

17

Schwartz,R.M.,&Humphreys,M.S.(1973).Similarit y judgmentsandfreerecallofunrelatedwords. Journalof ExperimentalPsychology 101, , 10-15. Shelton,J.R.,&Martin,R.C.(1992).Howsemantic is automaticsemanticpriming? JournalofExperimental Psychology:Learning,Memory,and,Cognition,18 1191, 1210. Schiffman,S.S.,Reynolds,M.L.,&Young,F.W. (1981). Introductiontomultidimensionalscaling;theory, methods,andapplications .NewYork,NY,Academic Press. Shiffrin,R.M.,Huber,D.E.,&Marinelli,K(. 1995) . Effectsofcategorylengthandstrengthonfamiliar ityin recognition. Journal of Experimental Psychology: Learning,Memory,andCognition,21 267-287. , Shiffrin,R.M.,&Steyvers,M(. 1997).Amodelfor recognitionmemory:REM—retrievingeffectivelyfrom memory. Psychonomic Bulletin& Review,4 145-166. , Shiffrin, R. M., & Steyvers, M. (1998). The effectivenessorfetrievalfrommemory.InM.Oaksf ord& (pp. . 73-95), N.Chater (Eds.). Rationalmodels of cognition Oxford,England:OxfordUniversity Press. Steyvers,M.,&Shiffrin,R.R.(tobesubmitted). Modelingsemanticandorthographicsimilarityeffec tson memory for individualwords. Torgeson,W.S.(1952).Multidimensionalscaling:I. Theory andmethod. Psychometrica,17 401-419. , Underwood,B.J.(1965).False recognition produced by implicit verbal responses. Journal of Experimental Psychology,70 122-129. , Underwood, B.J. (1969). Attributes of memory, PsychologicalReview,76 559-573. , Wickens, D.D. (1972). Characteristics of word encoding.InA.W.Melton&E.Martin(Eds.), Coding processesinhumanmemory .Washington,D.C.:V.H. Winston,pp.191-215.

Nelson,D.L.,McEvoy,C.L.,&Schreiber,T.A.(1999 ). TheUniversityofSouthFloridawordassociation,r hyme, and word fragment norms. http://www.usf.edu/FreeAssociation. Nelson,D.L.,McKinney,V.M.,Gee,N.R.,& Janczura , G.A.(1998).Interpretingtheinfluenceofimplicit ly activated memories on recall and recognition. , PsychologicalReview,105 299-324. Nelson, D.L., & Schreiber, T.A. (1992). Word concreteness and word structure as independent determinantsorfecall. JournaloM f emoryandLanguage, 31,237-260. Nelson,D.L.,Schreiber,T.A.,&McEvoy,C.L.(1992 ). Processing implicit and explicit representations. , PsychologicalReview,99 322-348. Nelson,D.L.,Xu,J.(1995).Effectsoimplicit f me mory onexplicitrecall:Setsizeandwordfrequencyeff ects. PsychologicalResearch,57 203-214. , Nelson,D.L.,&Zhang,N.(2000).Thetiesthatbin d whatisknowntotherecallow f hatisnew. Psychonomic Bulletin& Review,7 XXX-XXX. , Nelson, D.L., Zhang, N., & McKinney, V.M. (submitted).Thetiesthatbindwhatisknowntoth e recognitionowhat f isnew. Norman,D.A.,&Rumelhart,D.E.(1970).A system fo r perceptionandmemory.InD.A.Norman(Ed.), Modelsof humanmemory New . York:Academic Press. Osgood,C.E.,Suci,G.J.,&Tannenbaum,P.H.(1957) . .Urbana:Universityof Themeasurementofmeaning IllinoisPress. Palermo,D.S.,& Jenkins,J.J.(1964). Wordassociation norms grade school through college . Minneapolis: University oMinnesota f Press. Payne,D.G.,Elie,C.J.,Blackwell,J.M.,&Neuscha tz, J.S.(1996).Memoryillusions:Recalling,andrecol lecting eventsthatneveroccurred. JournalofMemoryand Language,35 261-285. , Pike,R.(1984).Comparisonoconvolution f andmatr ix distributedmemorysystemsforassociativerecalla nd recognition. PsychologicalReview,91 281-293. , Rips,L.J.,Shoben,E.J.,&Smith,E.E.(1973). Semantic distance and the verification of semantic relations. Journalofverballearningandverbalbehavior, 12,1-20. Roediger,H.L.,&McDermott,K.B.(1995).Creating falsememories:rememberingwordsnotpresentedon lists. JournalofExperimentalPsychology:Learning,Memor y, andCognition,21 803-814. , Romney,A.K.,Brewer,D.D.,&Batchelder,W.H. (1993).Predictingclusteringfromsemanticstructu re. PsychologicalScience, 4,28-34. Russell,W.A.,&Jenkins,J.J.(1954). Thecomplete Minnesotanormsforresponsesto100wordsfromthe Kent-Rosanoffwordassociationtest T . ech.Rep.No.11, ContractNS-ONR-66216,OfficeofNavalResearchand University oMinnesota. f Schacter,D.L.,Verfaellie,M.,&Pradere,D(.1996 ). Theneuropsychologyomemory f illusions:falsereca lland recognitioninamnesicpatients. JournalofMemory& Language,35 319-334. ,

18