What happens when we relearn part of what we previously knew

Psychological Research (2001) 65: 202±215

Ó Springer-Verlag 2001

ORIGINAL ARTICLE

Paul W. B. Atkins

What happens when we relearn part of what we previously knew? Predictions and constraints for models of long-term memory

Received: 29 April 1999 / Accepted: 26 July 1999

Abstract Part-set relearning studies examine whether relearning a subset of previously learned items impairs or improves memory for other items in memory that are not relearned. Atkins and Murre have examined part-set relearning using multi-layer networks that learn by optimizing performance on a complete set of items. For this paper, four computer models that learn each item additively and separately were tested using the part-set relearning procedure (Hebbian network, CHARM, MINERVA 2, and SAM). Optimization models predict that part-set relearning should improve memory for items not relearned, while additive models make the opposite prediction. This distinction parallels the relative ability of these models to account for interference phenomena. Part-set relearning provides another source of evidence for choosing between optimization and additive models of long-term memory. A new study suggests that the predictions of the additive models are broadly supported.

Introduction Consider what happens when a person relearns part of a previously learned but now forgotten foreign-language vocabulary. Does the person's memory for the unrelearned vocabulary improve, remain unchanged, or get worse? Most people intuitively believe that part-set relearning should improve memory for other material learned at the same time or in the same context (Atkins, 1996). As an illustration, Meara (1999, p.128) describes the related phenomenon of the ``Boulogne Ferry E€ect'':

P.W.B. Atkins Australian Graduate School of Management, The University of New South Wales and University of Sydney, Sydney, NSW, 2052, Australia Tel.: +61 2 9931 9247; e-mail: [email protected]

You get o€ the ferry and walk down the street in France, and you are bombarded with half-forgotten French words from sign-posts, from newspapers, from TV sets, and from snatches of conversation overheard in passing. Somehow, this bombardment reactivates words that you once knew, and a whole raft of French words, many of which have lain dormant for years, suddenly begins to ¯oat through your mind. Perhaps one reason why recovery of unrelearned items seems intuitively likely is because of the widespread acceptance of theories of memory that are based upon the notion of spreading activation (e.g. J. Anderson, 1983; Bower, 1996). Instances of the associative nature of memory abound in everyday life. For example, we see a photo of ourselves as a child, or hear a song that we used to enjoy, and long forgotten memories come ¯ooding back. Intuitively it seems likely that partset relearning should produce similar associative memory processes. Although more part-set cuing than part-set relearning, Meara's ``Boulogne Ferry E€ect'' suggests that relearning a subset of a foreign language vocabulary might remind us of the meaning of other items learned at a similar time or in a similar context. The experiment reported in this paper suggests that this particular type of associative recovery of foreign language vocabulary does not occur. Another reason why this intuition seems plausible is because part-set relearning is what we do whenever we relearn any large knowledge domain. For example, when you relearn a language you learned at school, you relearn the vocabulary in small chunks. It seems highly counter-intuitive to suggest that the more relearning that occurs, the more dicult it will be to relearn other material from the same domain. Despite these intuitions, little is known about the mechanisms that underlie relearning, and no controlled studies have examined the e€ects of part-set relearning. Previous studies have focussed on the e€ects of relearning on information that is relearned. The most

203

robust ®nding from this work is the savings e€ect, whereby information that a person once knew but can no longer remember is relearned more quickly than new information can be learned. Ebbinghaus (1885/1964) proposed a simple threshold model of the savings phenomenon that has remained unchallenged to this day. He suggested that new trace strength is added to residual trace strength from old learning, so that old items require fewer study trials to reach the threshold for recall. Since virtually any model of memory can explain the savings e€ect using a simple mechanism like that proposed by Ebbinghaus, an alternative to savings is required if we are to distinguish between models of relearning. The models and experiments presented in this paper shift the focus to examining the e€ects of relearning on memory for information that is not relearned. The ®rst section of the paper demonstrates that computer models of memory can be divided into two broad classes: (a) those that predict facilitation of unrelearned items and (b) those that predict impairment of unrelearned items following part-set relearning. These predictions re¯ect di€erent general approaches to learning. The former models optimize memory performance on all available items, while the latter use simple addition of trace strength. The second part of this paper reviews related studies of people and presents two new studies of partset relearning that support the latter class of model. To anticipate, this paper makes four main points. First, many existing computer models of memory make the counter-intuitive prediction that part-set relearning will result in impairment rather than facilitation of memory for unrelearned information. Second, this prediction is a natural consequence of the fact that these models have been designed to account for retroactive interference. Third, the empirical evidence supports most of the predictions of the models tested in this paper. Finally, studies of part-set relearning and related phenomena (part-set cuing and the list-strength e€ect) provide useful constraints that should be used in combination with studies of the savings e€ect to inform the development of contemporary models of relearning. The part-set relearning procedure for testing computer models of memory Part-set relearning in computer models involves three steps. First, all training items are learned either to a criterion or for a speci®ed number of trials. Second, forgetting occurs usually by perturbing the stored representations of the learned items, although the method of perturbation depends upon the individual representational assumptions made by each model. Finally, some of the originally learned items are relearned. Let the relearned set be called set R and the unrelearned set be called set U (See right branch of Fig. 1). These three steps were conducted for all the simulations discussed in this paper.

Fig. 1 Similarities and di€erences between the procedures for testing retroactive interference and recovery during part-set relearning

Models where part-set relearning facilitates memory for unrelearned items Four computer models have previously been tested using the part-set relearning procedure. All four models predicted that memory for unrelearned items should improve following part-set relearning. Hinton and Sejnowski (1986) showed recovery of unrelearned items in a semantic feature network that used the Boltzmann machine learning algorithm. Hinton and Plaut (1987) employed a extended variant of back-propagation learning (Rumelhart, Hinton & Williams, 1986) with ``fast'' and ``slow'' weights and also found a robust recovery e€ect. Plaut (1992, 1996) demonstrated recovery in a network designed to model reading processes (Hinton & Shallice, 1991) that used the back-propagation through time learning algorithm. Atkins and Murre (1998) demonstrated recovery in a standard feed-forward, back-propagation network designed to model foreign-language vocabulary acquisition called the VOCAB network (Fig. 2). Recovery occurs in these models because additional learning of related or unrelated items in multi-layer, gradient descent networks results in previously acquired weights being partially reinstated (Atkins & Murre, 1998; Hinton & Plaut, 1987). In brief, this process depends upon optimization of performance on the entire training set during original learning (see Atkins & Murre, 1998, for a detailed description). During relearning, the networks are more likely to return to this optimal solution for the entire training set than they are to move to some other solution that optimizes performance solely on the subset of items that are relearned. One can visualize this process by imagining two irregularly shaped balls on a beach with di€erent shaped holes in the sand separated by large spaces. With original learning of the entire set, the balls move until a hole

204

of the e€ect of part-set relearning rather than the magnitude is of primary interest here.

Model 1: Hebbian neural network model The Hebbian learning rule (Equation 1, after Hebb, 1949) has been widely used to model human memory (see, for example, Churchland & Sejnowski, 1992; Montague & Sejnowski, 1994): wij…t‡1† ˆ wij…t† ‡ Lai aj

…1†

where wij…t‡1† is the weight between input unit i and output unit j at time t+1, L is a learning parameter which was set to 1 for this investigation, ai is the activation of input unit i, and aj is the activation of output unit j. Method

Fig. 2 Recovery in the VOCAB network (20 runs). When the VOCAB network relearns a portion of the originally learned set of training patterns, performance on both sets R and U improves (from Atkins & Murre, 1998). Pattern error was de®ned as the sum of squares of the di€erence between the actual response and the target response across all output units and across all items in the set, divided by the number of patterns so as to eliminate the e€ects of set size (Rumelhart et al., 1986, p. 323)

is found that ®ts both balls comfortably (corresponding to a global minimum in the error space). Perturbation moves the balls out of the hole but, since there are relatively large distances between holes, relearning is likely to move the balls back into the same hole suitable for both balls rather than into another hole which ®ts one ball (set R) but not another (set U). Connectionist networks based on gradient descent learning form a class of models that support the intuitive prediction that part-set relearning should improve memory for unrelearned information. Models where part-set relearning impairs unrelearned items To assess the generality of this prediction, four other computer models of memory were tested using the partset relearning procedure: (1) a Hebbian network (after Hebb, 1949), (2) CHARM/TODAM (Eich, 1982, 1985; Lewandowsky & Murdock, 1989; Murdock, 1993), (3) MINERVA 2 (Hintzman, 1986, 1988), and (4) SAM (Raaijmakers & Shi€rin, 1981). These models have all been successful in providing an account for other ®ndings in the experimental memory research literature. For all the simulations presented below, the absolute magnitude of the errors should not be compared between models. The models are suciently di€erent that it is impossible to directly equate parameters. The direction

A network with 20 input units and 20 output units was used to test the e€ects of the part-set relearning procedure. One hundred simulations were run with 40 di€erent, randomly selected input and output patterns for each simulation. In each pattern, each element had an equal probability of being set to either )1 or 1. Therefore, although the expected mean correlation across all possible pairs of patterns was zero, individual pairs of patterns were not orthogonal. The random patterns were then normalized so that the vectors were of unit length in order to retain comparability between the error terms under di€erent conditions. The use of orthogonal patterns in models that process distributed representations, such as the Hebbian model, MINERVA, and CHARM, would not have been justi®ed since much of the functionality of the models (e.g., accounting for retroactive interference) arises from their distributed nature. The network was ®rst trained for a single epoch on the entire training set of 40 patterns. The network weights were then perturbed by the addition of random noise from a Gaussian distribution with mean 0 and variance 1.0. This was the procedure used by Hinton and Sejnowski (1986) to test their Boltzmann machine network. With Hebbian learning, the forgetting produced by this method is approximately equivalent to training on other random patterns, the approach used by Atkins and Murre (1998) and Hinton and Plaut (1987).

Results and discussion Recovery did not occur in the Hebbian network when the input and output patterns were randomly related (Fig. 3a). During relearning of set R, performance on the 20 items in set U was impaired. The more relearning that occurred for set R, the worse the performance on set U became. When a pattern is presented to a Hebbian network, the changes to the weights correspond to simply adding the outer product of the input and output vectors to the weight matrix. As each pattern is presented, the weights necessary to produce that pattern are simply overlaid on the existing weights. In other words, relearning part of a set of items interferes with performance on the remaining items. Any network without a hidden layer, including those that use a learning algorithm such as the delta rule, will show a similar pattern of behavior (Atkins & Murre,

205

Fig. 3a±d Part-set relearning in the additive models caused the error on the unrelearned items (set U) to increase. OL = original learning, P = perturbation phase, RL1±4 = Relearning epochs 1 to 4. a Hebbian learning. b CHARM. c MINERVA 2. d SAM: Pattern error was the mean number of times that an item was not successfully recovered

1998). In addition, the Matrix model (Humphreys, Bain, & Pike, 1989) is formally equivalent to the Hebbian model and could be expected to make the same prediction.

Model 2: CHARM CHARM (Eich, 1982, 1985) is a non-connectionist model that uses shared internal representations. The CHARM and TODAM models (Lewandowsky & Murdock, 1989; Murdock, 1993) are almost identical so

that, even though only CHARM was simulated for this paper, the same trends should be observed in TODAM. As with the Hebbian model, items in CHARM are represented by distributed vectors of features. CHARM encodes paired associates by convolving the vector representing the stimulus A with the vector representation the response B (see Eich, 1985). The convolved vector representing the pair of items in then added to a single shared memory trace T. Correlating the probe item A with T and extracting the result retrieves an approximation of B. Method The CHARM model was tested using similar parameters to those used in the Hebbian network. Each stimulus and response vector had 20 features. As before, the runs consisted of (a) a single learning epoch on all 40 items, (b) perturbation of T by random

206 Gaussian noise (mean 0.0, variance 2.0), and (c) four epochs of relearning half of the items. Again, 100 runs of each simulation were conducted with di€erent randomly associated stimulus and response vectors for each run. Following Eich (1985), each feature was allowed to adopt any real value between )1.0 and 1.0. The stimulus, target response, and actual response vectors were normalized to be of unit length to allow comparison of error under di€erent conditions.

Results and discussion The e€ect of part-set relearning in the CHARM model (Fig. 3b) follows exactly the same pattern as that observed in the Hebbian network because there are broad similarities between the models. Whereas the Hebbian network adds the outer product of the input and output patterns to the memory trace, the CHARM model adds the convolution of the two items. However, in both cases, the addition of new memories, or the strengthening of some of the old memories, degrades other memories. Additional simulations were also conducted using 200 features to represent memory traces instead of 20. Learning was greatly enhanced with more features, but the basic ®nding of impairment of unrelearned patterns was the same. The complete CHARM model assumes an additional stage in which the degraded response is compared with the target responses. Eich (1982, p. 636) added this stage to allow the network to produce valid responses on the basis of the degraded productions of the model. However, even if this stage is included in the model, part-set relearning increases the rate of intrusions from set R, thereby degrading performance on set U.

Model 3: MINERVA 2 The MINERVA 2 model (Hintzman, 1986, 1988) is di€erent from the Hebbian network and the TODAM and CHARM models in that memories are stored in separate traces instead of in a single, shared memory trace. However, MINERVA 2 can still model interference e€ects because the retrieval process is a function of all the items in memory. Each paired-associate in MINERVA 2 is represented by a single vector containing features representing the context, stimulus, and response. When a vector is presented for learning, each feature is copied into a new memory trace with probability L. For this paper, L was set to 1 so that exact copies of the studied items were stored in memory. For cued recall, a probe containing the context and stimulus features is ®rst presented to the model. The similarity (M) between the probe (P) and each memory trace (T) is calculated using Eq. (2), n X M…i† ˆ …1=NR † Pj T …i; j† …2† jˆ1

where Pj is the jth feature of the probe, T(i, j) is the jth feature of trace i, NR is the number of features for which

either Pj or T(i, j) is non-zero, and n is the number of features in each trace. That is, the similarity between the probe and each vector in memory is simply the dot product of the two vectors divided by the number of relevant features. The activation A(i) of the trace i is calculated by cubing Mi. Cubing M(i) focusses the retrieval process on items in memory that are very similar to the probe. The response trace C, or echo, as Hintzman refers to it, is calculated using Eq. (3), m X C…j† ˆ A…i†T …i; j† …3† iˆ1

where C(j) is the activation of feature j in the echo and m is the number of traces in memory. Note that echo content is calculated for all the features in the memory trace. This has the e€ect of ®lling in the response features missing from the probe, thereby allowing the model to perform cued recall. The MINERVA 2 model has a more elegant way of dealing with the problem of degraded responses than the procedure outlined above for the CHARM model. Hintzman (1986) proposed that the echo vector could be repeatedly fed back into the system as a secondary probe, thereby causing the model to converge upon an echo that is more similar to one of the items in memory. Method Context features were not used for the simulations of MINERVA 2. Memory traces were represented by a vector with 20 elements for the stimulus and 20 elements for the response. For each simulation, 100 runs were conducted with di€erent item vectors. As for the Hebbian simulation, each feature in each item was assigned a value of 1 or )1 randomly. Forgetting was simulated by randomly zeroing features in the memory traces with probability F. For these simulations, F was set to 0.1 unless otherwise stated. Simulations consisted of a single learning epoch with the entire set, perturbation, and then four epochs of relearning of set R. For each test trial, the probe item consisted of a vector in which the stimulus features matched one of the studied items and the response features were set to zero. The echo was then fed back into the model twice to clarify the response. The input, output, and response vectors were normalized to be of unit length to allow comparison of error under di€erent conditions.

Results and discussion As Fig. 3c demonstrates, part-set relearning in MINERVA 2 also impairs performance on the unrelearned items. In MINERVA 2, the e€ect of part-set relearning cannot arise from a direct degrading of the memory traces representing the unrelearned items because all the memory traces are stored separately. However, signi®cant interference still occurs at retrieval. At the end of relearning, each item from set R was represented ®ve times in memory, whereas the items from set U were only represented once. Echo content was calculated by summing across all the memory traces, scaled by their activations from the probe. When a probe from set U was provided, the memory trace matching that probe still

207

received the most activation. However, items from set R were also activated to the extent that they were similar to the probe. Because these items were represented in memory more often, the contribution from the target item was drowned out by the contributions from the similar items in set R. Consequently, the error on set U increased. MINERVA 2 can also be implemented such that repeated presentations of the same pair strengthen the existing trace for that pair rather than causing the formation of new traces. However, this modi®cation does not alter the e€ects of part-set relearning, since the similarity value returned by Eq. (2) remains the same. Note that impairment of set U occurred despite the fact that the response vectors were cleaned up by repeated presentation to the model. Thus, even though a reasonably accurate version of the memory trace for each unrelearned item still existed in memory, its strength as an attractor during resonance was outweighed by the presence of multiple memory traces for each of the items in set R.

Model 4: Search of associative memory (SAM) The SAM model (Gillund & Shi€rin, 1984; Raaijmakers & Shi€rin, 1981; Shi€rin & Raaijmakers, 1992) uses local rather than distributed representations. In this respect it provides a useful comparison against the distributed systems presented above. In SAM, items are represented in memory as separate ``images.'' During learning, the strengths of association between either context or word cues, and the images in memory, are incremented. These associative strengths determine the likelihood that a particular image will be recalled given a particular set of cues. SAM has already been used to model the inhibitory e€ect of part-list cuing in free recall (Raaijmakers & Shi€rin, 1981). However, paired-associate learning and recall, where each response has a unique cue, is of primary interest here. Therefore, a simpli®ed version of SAM was used without the assumption that retrieved items are fed back as cues for further retrieval (see Appendix). The associative strengths acquired during learning in SAM are simply a function of the number of study presentations and three parameters: a, c, and d, where a is the increment for the associations between the context and the image in memory, c is the increment for the associations between the stimulus word and the image, and d is the residual associative strength between a given cue word and the other images with which it was not paired during learning. Thus, if t represents the number of presentations of the pair during study, then after learning, S…CT ; WiS † ˆ at S…WiT ; WiS † ˆ ct S…WiT ; WjS † ˆ dt

where S…CT ; WiS † denotes the associative strength between the context cue (CT) and an image in memory (WiS), and S(WiT, WiS) denotes the associative strength between the stimulus word cue (WiT) and an image. Method In the simulations presented below, a = 0.3, c = 0.3 and d = 0.01. Forty items were learned originally and 20 items were relearned following perturbation. The baseline runs presented below are based upon four epochs of original learning and four epochs of relearning. It was necessary to use four rather than one trial of original learning for the SAM model to avoid the unrelearned items reaching ceiling for the error during relearning. Following Mensink and Raaijmakers (1988; see also Shi€rin, Murnane, Gronlund, & Roth, 1989), it was assumed that a pair was represented by a single image rather than by separate images for the stimulus and response words. In order to test SAM using the part-set relearning procedure, it was necessary to implement forgetting in the model. Mensink and Raaijmakers (1988, 1989) presented a complex procedure to simulate forgetting due to contextual ¯uctuation. Initially, this procedure was implemented for this paper. However, similar results were obtained if forgetting was simulated by decay of the associative strengths between the context and the images in memory. This method was used by Raaijmakers (1992) to simulate delayed part-list cuing e€ects in free recall. For the simulations presented below, forgetting was simulated by multiplying all the associative strengths from the context cues by 0.1 following learning. In SAM, retrieval of an item occurs in two steps: sampling and ``recovery.''1. Items are sampled with a probability determined by the available cues as speci®ed by Eq. (4), S…CT ; WiS †S…WkT ; WiS † PS …WiS jCT ; WkT † ˆ Pn jˆ1 S…CT ; WjS †S…WkT ; WjS †

…4†

where PS …WiS jCT ; WkT † is the probability that the stored image WiS will be sampled given the current context (CT ) and a word cue (WkT ), and n is the number of images in memory. In essence, Eq. (4) speci®es that the likelihood of successfully sampling an item in memory is a function of the strengths from the available cues to the image relative to the sum of the strengths of association of those same cues to all the other images in memory. By contrast, recovery of the image from memory is purely a function of the associative strengths from the available cues to the image. Other items in memory do not a€ect the probability that an item will be recovered once it has been sampled (Raaijmakers & Shi€rin, 1981).

Results and discussion Part-set relearning in SAM also impaired memory performance on the remaining items (Fig. 3d). The impairment of the unrelearned items occurred because the associations between the context and the relearned items were strengthened during relearning. This increased the size of the denominator in Eq. (4), thereby decreasing the probability of successfully sampling the unrelearned items.2 The probability of recovering an unrelearned 1

When discussing SAM, ``recovery'' is used to describe the second phase of retrieval. Recovery in this sense is unrelated to recovery of unrelearned items during part-set relearning.

2

Shi€rin, Ratcli€, and Clark (1990) presented a variant of SAM where stronger items were more di€erentiated from other items in memory. It would appear from Shi€rin et al. (1990 Fig. 1) that a positive list strength e€ect, and therefore the same e€ect of impairment of unrelearned items, would occur even with di€erentiation.

208 Fig. 4 Changes in the amount of error for Set U following relearning of Set R when 10% or 90% of the items were relearned. All di€erences between the 10 % and 90% conditions were signi®cant (p < .001). The ratio rule has no variance and is described in the text

item, once it had been successfully sampled, was unaffected by part-set relearning. To summarize, all four models predict that part-set relearning should impair memory for other items originally learned but not relearned. The models make the same prediction despite the fact that they rely upon utterly di€erent processes for learning, storage, and retrieval. However, despite their di€erences, relearning (and learning) in all of these models depends upon the simple addition of item strength or additional representations. For this reason, these models will be referred to as the additive models throughout the rest of this paper. In additive models, strengthened items interfere with unstrengthened items during either storage or retrieval. The following sections consider whether the models make the same predictions under the in¯uence of other factors that a€ect the extent of impairment. These factors could easily be manipulated in experimental tests of part-set relearning with people and thus form the basis for further experimentation. In the results presented below, the results from the VOCAB network (Atkins & Murre, 1998) have been included to facilitate comparison with the additive models. The VOCAB network was a standard feed-forward back-propagation network with 104 input units, 40 hidden units, and 104 output units. Only the direction of changes, rather than the magnitude of the errors, should be compared because of parameter di€erences such as number of output units and number of learning and relearning trials.

Factors a€ecting the extent of impairment Variations in the proportion of items relearned For each model, runs were conducted varying the proportion of items in set R. Five levels of proportion were assessed: 10%, 30%, 50%, 70%, and 90% relearned. To facilitate comparison between the models, Fig. 4 reports only the results when 10% and 90% were relearned. In all the additive models the error on set U increased more when more items were relearned. In the Hebbian and CHARM models, this occurred because relearning more of the items caused more interference with the stored representations of the unrelearned items in the shared memory trace. In the MINERVA 2 model it occurred because there were more items in memory that had been strengthened and were therefore more likely to compete for retrieval with the unrelearned items. In the SAM model it occurred because more items were strongly associated to the current context cue, and therefore the relative strength of the unrelearned items was reduced. These results were caused by an increased number of patterns (i.e., types) rather than an increased number of pattern presentations (i.e., tokens) during relearning. The models were also tested using a variable number of relearning epochs to control for the total number of pattern presentations. For example, when only 10% of the items were relearned, they were presented for ®ve times as many epochs as when 50% of the items were relearned. The overall trends were basically the same as those

209 Fig. 5 Changes in the amount of error for Set U following relearning of Set R when items were learned to a low or high level originally. All di€erences between levels of original learning were signi®cant ( p < .001) with the exception of the Hebbian Network. The ratio rule has no variance

presented in Fig. 4. Type rather than token frequency of the relearned items was critical during part-set relearning. The results for the additive models contrasted starkly with those produced by the VOCAB network. In that model, Atkins and Murre (1998) found that the more items that were relearned, the more error for the unrelearned items decreased (Fig. 4). In multi-layer, gradient-descent networks, when more items are relearned, these networks move back more closely towards the weights established during original learning (Atkins & Murre, 1998). Variations in the level of original learning The models were also tested using di€erent levels of original learning. Each model was tested with 1 or 4 epochs of original learning.3 Again, all the additive models produced similar results when tested for relearning after di€erent amounts of original learning (Fig. 5), but the VOCAB model behaved di€erently. In the Hebbian, CHARM, and MINERVA 2 models, increasing the level of original learning increased the resistance of all the items in memory to interference from the addition of random noise. That is, the error after perturbation was lower if the model had received 3

In the baseline runs, SAM was tested with four trials of original learning to avoid ¯oor and ceiling e€ects. Learning in the SAM model is linear, so with 16 trials of original learning, the associative strengths are four times as large as they are with 4 trials of original learning. For maximum comparability with the other models, SAM was tested with 4, 8, 12, and 16 epochs of original learning.

more trials of original learning. The increased resistance to interference shown by all the models matches the behaviour of human participants. McGeoch and Irion (1952) noted that the level of original learning and the degree of interference exerted by an interpolated list were inversely related in a range of experiments. Additional original learning also generally increased the resistance of the set-U items to interference from the set-R items during part-set relearning. That is, the error for set U generally did not increase as much following relearning if the patterns had been better learned originally (although this pattern was not observed for the Hebbian network). In the SAM model, items did not directly interfere with one another, but the model behaved in the same way as CHARM and MINERVA 2 because sampling probability was a function of the realtive strengths of the items in memory. With more extensive original learning, each relearning trial of a setR item increased its sampling probability and decreased the sampling probability of the other items, less than was the case with fewer trials of original learning.

Discussion The four additive models all predict that part-set relearning should impair memory for unrelearned information despite di€ering substantially in the operations that they use for the encoding, storage, and retrieval of items. In addition to the basic prediction, all the models predict that relearning a greater proportion of the items should produce more impairment of the unrelearned

210

items, and that stronger representations of the unrelearned items would be more resistant to interference from perturbation. All the models except the Hebbian model displayed a smaller increase in error for set U with more original learning. These results were not strongly parameter dependent. Indeed, they can be seen as an inevitable consequence of models with limited capacity storage or retrieval systems, a fundamental aspect of the Hebbian, CHARM, and MINERVA 2 models that allows them to explain interference e€ects. In the SAM model, part-set relearning impaired unrelearned items because of the fundamental assumption of sampling based on relative strength Eq. (4). The results of the simulations can be summarized by a simple relative strength rule sometimes referred to as the ratio rule (Bjork & Bjork, 1992; Brown, 1968; Luce, 1963, Rundus, 1973; Wixted, Ghadisha, & Vera, 1997), where the probability that an item will be retrieved is a function of its strength relative to the other items in memory. For example, assume that ten items are studied and that each item is represented locally in memory by a certain strength value. Each study trial adds a constant amount of strength to the item. The probability of retrieval is assumed to be a function of the strength of the item relative to the sum of strengths of all the items in memory. Given these assumptions, the ratio rule produces the results shown in Figs. 4 and 5. These results parallel the results for the additive models. This is not to suggest that the ratio rule replaces the models presented above. The models incorporate features that are necessary to account for other memory phenomena. However, the ratio rule illustrates the three fundamental shared features of the additive models presented above: (a) representations of items in memory have a certain strength, (b) studying an item increases its strength additively, and (c) increasing the strength of some items decreases the relative strength of other items. The results for the additive models contrast starkly with those produced by multi-layer connectionist networks (see the results for the VOCAB network in Figs. 4 & 5). During original learning, these networks adjust their weights to a point on the error surface where performance on the combined sets R and U is optimal. That is, the point where minima in the error surfaces for sets R and U approximately coincide. In e€ect, the optimization process during original learning moves the weights in a multi-layer network to a point where the entire list is treated as a singular entity. Random noise or training on other patterns still interferes with memory for the list items, but further learning of items within the list does not interfere with memory for other items from the list.4 This is a fundamental di€erence between opti4

In practice, there is a slight increase in error for set U with additional training (Fig. 2). However, in our simulations, error for set U never approached anything like the level present after perturbation.

mization models of memory and additive models of memory. This distinction between the limited capacity additive models and the optimization models in terms of part-set relearning closely parallels the distinction drawn previously between those models that are subject to catastrophic interference and those that are not. All gradient-descent models with a hidden layer are prone to catastrophic interference (French, 1992, 1994; Hetherington & Seidenberg, 1989; Lewandowsky, 1991; McCloskey & Cohen, 1989; Murre, 1992a, 1992b; Ratcli€, 1990). Given enough time, these networks adjust their weights so that performance on the current set of training patterns is optimal. Any previously learned patterns are likely to be completely ``forgotten'' during this process. Thus, for example, if item A is presented 100 times and then item B is presented 100 times, learning of B obliterates most if not all memory for A. For the network to learn and retain both A and B, presentation of the two patterns must be interleaved (McClelland, McNaughton, & O'Reilly, 1995). If people were subject to this level of interference, we would never be able to acquire the level of stored knowledge that we obviously possess. The optimization process that produces catastrophic interference in multi-layer, gradientdescent models is the same process that causes those models to predict improvement in memory for unrelearned items following part-set relearning (see Atkins & Murre, 1998). This similarity is not surprising. It we ignore the aspect of an extended retention interval, the part-set relearning procedure and the procedure normally used to study retroactive interference di€er in only the ®rst phase (Fig. 1). However, this di€erence is crucial in terms of the behavior of a multi-layer, connectionist network. If all the patterns are learned originally, the network is forced to a point that optimizes performance on the entire training set. Unless perturbation is suciently large to completely reset the network, during relearning it is ``easier'' for the network to move back to this optimal solution than it is to move to another entirely di€erent solution. Thus, performance on set U is improved. By contrast, when only a subset (set U) is learned originally, during learning of set R it is most likely that the network will move to a solution that optimizes performance for set R by moving away from the solution for set U. Therefore, set U su€ers catastrophic interference. In terms of the beach ball analogy discussed above, the retroactive interference procedure corresponds to ®rst ®nding a hole somewhere on the beach which ®ts the ®rst-learned set, then moving to another hole that ®ts the second-learned set. It is extremely unlikely that this second hole will ®t both balls. Correspondingly, it is unlikely that the solution arrived at after training on set R will be adequate for set U. In the additive models, it does not matter whether A and B are presented in blocked or interleaved order; either way, the models ultimately represent the items

211

with the same relative strengths. The additive process that results in gradual interference in the additive models also results in impairment of unrelearned items in the part-set relearning procedure. In addition to the models tested in this paper, many other computer models of memory also rely upon some system for incrementing individual item trace strength during learning (e.g. Humphreys, Bain, & Pike, 1989; Shi€rin & Steyvers, 1997). In part, this is an inevitable consequence of the fact that trace-strength based models provide a parsimonious explanation of interference phenomena. The results presented above suggest that any model that can explain interference phenomena should also predict that part-set relearning will result in impairment of unrelearned items.

Experiment: Part-set relearning in people It is not possible to directly translate the part-set relearning procedure for the computer models into experiments with people because repeated testing on the same items causes improvements in performance even without feedback or explicit study (e.g., Wheeler & Roediger, 1992). To avoid this problem, a between-items design was used to test part-set relearning in people. Participants ®rst learned the entire corpus (sets R & U combined). Some time later, they were tested for their memory for set R, then relearned those items, then were tested for their memory for set U. Sets R and U were equated such that participants were expected to perform equally on the sets without the intervention of relearning set R. Therefore, the primary comparison of interest was between memory of set R prior to relearning and set U following relearning. A pilot experiment involving relearning of a French vocabulary learned at school an average of nine years ago indicated a marginal e€ect of impairment of the unrelearned items F(1,14) = 3.72, p = 0.07. However, interpretation of the pilot experiment was complicated by the fact that the participants had learned French at school. The experiment reported here was designed to provide a more powerful test of the e€ects of part-set relearning on recall of the unrelearned items. More participants were tested and participants had not previously learned the language, so control of the levels of original learning for individual items was possible. Assignment of items to the relearned and unrelearned sets was controlled for each participant on the basis of how dicult each item was for the participant to learn, thereby reducing random variation in recall performance on the two sets. Method A total of 29 ®rst-year psychology participants were recruited. None of the participants knew Finnish prior to the experiment. In the ®rst session, participants learned two sets of 16 Finnish-

to-English translations selected from a pool of 64 translations. Any Finnish word with an unusual character was altered such that it could be produced with a standard American keyboard (e.g., ``aÈ'' converted to ``a''). Each participant learned di€erent sets of translations randomly selected from the pool. All stimuli were presented in white text on a black background on a computer screen. Procedure Session 1: Learning. Participants were informed that all study and testing would be self-paced and that they would only be required to translate from Finnish to English. They were ®rst familiarized with the drop-out learning procedure using three practice Finnish-English translation pairs. If the participants correctly translated a word, they were informed that they were correct. If the participants responded with an incorrect translation or omission, the item and the correct translation were displayed for 5 s. Correctly translated words were dropped from the training for that epoch. Words that were not correctly translated were retained for a subsequent presentation. In this way, within an epoch of training, each word was repeatedly presented until it was correctly translated once. The entire set was then presented again until the participants reached criterion. The order of presentation was randomized. For the main experiment each participant learned two lists of 16 translations, one at each level of learning. The order of the level of learning manipulation was counterbalanced across participants. Fifteen participants were ®rst trained to a criterion of one errorfree epoch (HIGH LEVEL OF LEARNING), whereas the other 14 participants were ®rst trained to the criterion of two correct responses to each word (LOW LEVEL OF LEARNING). To equate expectations, participants were told that their memory for the translations would be tested in Session 2 but that they should not rehearse the translations during the retention interval. All participants reported that they conformed to these instructions. Session 2: Relearning. Twenty participants returned after 14 days for the second session. Five participants returned 1 day early and 4 returned one day late. Analyses revealed that these participants did not behave di€erently, so only aggregate results are reported below. Participants were ®rst tested on 8 of the originally learned translation pairs (set R) selected so as to equate the number of trials taken during original learning. They then relearned those 8 items and ®nally were tested on the complete list of 16 translations from that set (i.e., sets R and U combined). This procedure was then repeated for the other set of 16 translations. The order of presentation was counterbalanced across participants so that approximately half received the ®rst-learned-list ®rst in Session 2.

Results During original learning, participants took an average of 4.6 trials (SD = 1.5) per translation to reach criterion in the low level of learning condition and 11.9 trials (SD = 7.7) in the high level of learning condition. This di€erence was reliable, t(28) = 5.4, p < 0.001, and substantial. For the analysis of recall performance, any responses that were clear spelling errors were recoded as correct. Table 1 presents a summary of the main comparison of interest. In addition, it shows that set R was relearned e€ectively, with recall levels approaching ceiling (8 correct translations) following relearning.

212

Contrary to the predictions of the additive models, level of original learning did not a€ect the di€erence between recall on sets R and U, F(1,28) = 0.44, g2 = 0.016, although there was a clear main e€ect of level of learning on both sets, F(1,28) = 44.3, p < 0.001, g2 = 0.613, indicating that the learning manipulation was successful. The main e€ect of interest was the di€erence between recall for the relearned and unrelearned sets. Relearning set R signi®cantly impaired recall for set U, F(1,28) = 7.9, p < 0.01, g2 = 0.220. The experiment presented above involved a comparison of set R prior to relearning, with set U following relearning. One potential problem with this design is that time-dependent factors such as fatigue and motivation might have a€ected the result. For this experiment, participants learned and relearned two lists of translations. If time-dependent e€ects were implicated in the results, recall performance for the ®rst list set R should have been greater than recall performance for the second list set R. This was not the case, First List Set R M = 2.96, SD = 2.28; Second List Set R M = 2.62, SD = 1.61; F(1,28) = 0.37, ns, g2 = 0.013, suggesting that the impairment of set U relative to set R was not due to time-dependent e€ects.

General discussion In the experiment, relearning part of a previously learned vocabulary impaired performance on the translations that were not relearned. Therefore, the predictions of the additive models but not the optimization models were supported by these experiments. However, the additive models also predicted that translations that were better learned originally should be more resistant to interference from the relearned translations. There was no evidence to support this prediction. The comparison of the two set-R lists suggested that time-dependent e€ects such as fatigue or motivation did not play a role in this experiment. Another way to check for time-dependent e€ects in future part-set relearning experiments would be to include a control group that learned di€erent but equally dicult materials instead of set R. These equivalent materials could not be translations, because the additive models would predict that

Table 1 Recall out of 8 items as a function of list and level of original learning. N = 29 in all conditions List

Relearned (Set R) prior to relearning Unrelearned (Set U) Relearned (Set R) following relearning

Level of learning in Session 1 High M (SD)

Low M (SD)

All M (SD)

3.83 (2.0)

1.76 (1.4)

2.79 (2.0)

3.24 (1.9) 7.86 (0.4)

1.45 (1.4) 7.83 (0.4)

2.34 (1.9) 7.85 (0.4)

any related learning would interfere with memory for set U. Although no previous studies have directly examined part-set relearning, many other studies have examined the e€ects of increasing the relative strength of a subset of items in memory. Providing some of the items from a previously learned word list as cues usually impairs memory for the remaining items. This counter-intuitive ®nding is known as the part-list cueing e€ect (Nickerson, 1984). Similarly, Ratcli€, Clark and Shi€rin (1990) demonstrated that more-often-studied items impair free and cued recall (but not recognition) of less-oftenstudied items (see also Tulving & Hastie, 1972). M. Anderson, Bjork and Bjork (1994) showed that practicing retrieval of a subset of items impairs memory for unpracticed items associated with the same cue. Although they explicitly rule out a simple ratio-rule interpretation of their results for other reasons, the basic ®nding of impairment of unpracticed exemplars remains. Studies that have examined the e€ects of retrieval order have indicated that ®rst-retrieved items can interfere with retrieval of subsequent items (Roediger & Schmidt, 1980). Finally, Wixted et al. (1997) found that increasing the relative strength of a subset of items increased free recall latency for unstrengthened items in the same list. Although these studies di€er in detail, they have all shown that stronger items impair memory for weaker items. This competitive process parallels the competitive processes instantiated in the additive models. What implications do the previous studies and the new study presented above have for real world vocabulary relearning? As noted in the introduction, it is counterintuitive to suggest that relearning some information interferes with relearning other information from the same domain. However, this is what the results indicate. Meara's statement regarding the ``Boulogne Ferry E€ect'' implies that relearning a subset of a foreign language vocabulary should remind us of the meaning of other items learned at a similar time or in a similar context. This intuitively appealing interpretation seems not to be the case. This does not rule out the possibility of other associative memory processes playing a role in naturalistic language relearning. Clearly, there is much more to learning to use a language than simply learning lists of unrelated vocabulary. Additional aspects of language learning such as acquiring grammar or learning standard phrases might lead to a subjective experience like that described by Meara. However, it is important to note that, at the level of individual vocabulary items at least, relearning some translations does not appear to produce spontaneous retrieval of other translations learned at the same time and in the same context. In addition, it seems likely that relearning of a foreign language vocabulary in naturalistic settings will be accompanied by consolidation of material, so that it is relatively resistant to interference from strengthening of a subset of the items. This idea remains to be tested. One

213

might argue, as McClelland, et al. (1995) have done, that two-layer, gradient-descent networks are best seen as simulating long-term memory consolidation rather than shorter term memory processes. If this were the case, then the only way to de®nitively rule out the predictions of multi-layer, gradient-descent networks would be to conduct experiments where the unrelearned items were tested a long time after part-set relearning, so that consolidation could have occurred. However, on the basis of the evidence to date, it must be concluded that part-set relearning strengthens a subset of the items in memory at the expense of memory for other unstrengthened items in memory learned in the same context. The simulations described above and by Atkins and Murre (1998) demonstrate that current computer models of memory can be grouped into two broad classes according to their performance during part-set relearning. Optimization models display recovery, whereas additive models make the opposite prediction. In some additive models, interference with unstrengthened items occurs in storage, and in others it occurs during retrieval. In either case, part-set relearning impairs rather than enhances memory for the remaining items. This is a fundamental attribute of these additive models. The same processes that produce this prediction also allow these models to simulate gradual, non-catastrophic interference phenomena widely observed in experimental studies of human memory. The models presented in this paper commence the process of developing a more general and contemporary theory of relearning. Such a model should be constrained by evidence from studies that consider not only the e€ects of relearning on relearned information, but also its e€ects on unrelearned information.

Appendix: SAM for paired-associate learning and cued recall Since the SAM model was only tested using paired-associate recall and not free recall or recognition, the version of SAM that was implemented for this paper was simpler than the general model (Raaijmakers & Shi€rin, 1981) in the following ways: 1. No short-term bu€er was used. Raaijmakers and Shi€rin (1981b) present evidence that presentation of a new stimulus of a paired associate stops any rehearsal of the previous pair. 2. The only cues that were used were the stimulus words and the context. If an image was successfully recovered it was not used as a cue for further retrieval attempts, as is necessary in free recall. 3. Parameter Kmax was irrelevant (see Figure A1). In the general SAM model, Kmax limits the total number of search attempts that are made during list recall. This is necessary to terminate the search process during free recall, but it is not necessary for paired-associate recall. The only stopping criterion that is required for paired associate recall is Lmax, which limits the number of search attempts that are made with a given stimulusplus-context cue. In the simulations in this paper, Lmax was set to 3. 4. On successful retrieval of an item, its strength in memory was not incremented. This addition, while useful to explain retrievalbased learning e€ects, would not have a€ected the results presented in the body of this paper.

Fig. A1 Retrieval in the SAM model as implemented in this paper for paired-associate cued recall. This was a simpli®cation of the procedure used by Mensink and Raaijmakers (1988) 5. In Mensink and Raaijmakers (1988), a constant (Z) was added to the denominator of the sampling probability equation to simulate the e€ect of extra-experimental associations. If the constant was included, it produced a larger decrement in performance following context decay and a€ected the magnitude of the impairment produced by part-set relearning. However, since the same trends occurred in the results, irrespective of whether this additional parameter was included, it was decided to use the originally speci®ed sampling probability equation (Raaijmakers & Shi€rin, 1981) without the extra parameter. In addition to these simpli®cations, other features of the model implemented for this paper were as follows: 6. It was assumed that a pair of items was represented as a single image (Mensink & Raaijmakers, 1988; Shi€rin et al., 1989; Raaijmakers & Shi€rin, 1981). 7. Delay was simulated by decay of the associative strengths from the context cue (Raaijmakers, 1992, personal communication).

References Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press. Anderson, M. C., Bjork, R. A., & Bjork, E. L. (1994). Remembering can cause forgetting: Retrieval dynamics in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 1063±1087. Atkins, P. W. B. (1996). Models of memory and relearning. Unpublished doctoral dissertation, Cambridge University, U.K.

214 Atkins, P. W. B., & Murre, J. M. (1998). Recovery of unrehearsed items in connectionist models. Connection Science, 10, 99±119. Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus ¯uctuation. In A. F. Healy, S. M. Kosslyn, & R. M. Shi€rin (Eds.), From learning processes to cognitive processes: Essays in honour of William K. Estes (Vol. 2, pp. 35±67). Hillsdale, NJ: Erlbaum. Brower, G. H. (1996). Reactivating a reactivation theory of implicit memory. Consciousness and Cognition, 5, 27±72. Brown, J. (1968). Reciprocal facilitation and impairment of free recall. Psychonomic Science, 10, 41±42. Churchland, P. S., & Sejnowski, T. J. (1992). The computational brain. MIT Press, Cambridge, MA. Ebbinghaus, H. (1964). Memory: A contribution to experimental psychology (H. A. Ruger & C. E. Bussenius, Trans.). New York: Dover. (Original work published 1885) Eich, J. M. (1982). A composite holographic associative recall model. Psychological Review, 89, 627±661. Eich, J. M. (1985). Levels of processing, encoding speci®city, elaboration, and CHARM. Psychological Review, 92, 1±38. French, R. M. (1992). Semi-distributed representations and catastrophic forgetting in connectionist networks. Connection Science, 4, 365±377. French, R. M. (1994). Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference. Proceedings of the 16th Annual Cognitive Science Conference. Gillund, G., & Shi€rin, R. M. (1984). A retrieval model for both recognition and recall. Psychological Review, 91, 1±67. Hebb, D. O. (1949). The organization of behavior. New York: Wiley. Hetherington, P. A., & Seidenberg, M. S. (1989). Is there ``catastrophic interference'' in connectionist networks? Proceedings of the 11th Annual Conference of the Cognitive Science Society, 26± 33. Hinton, G. E., & Plaut, D. C. (1987). Using fast weights to deblur old memories. Proceedings of the Ninth Annual Conference of the Cognitive Science Society, 177±186. Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart, J. L. McClelland, & the PDP Research Group, Parallel Distributed Processing, Vol. 1: Foundations (pp. 282±317). Cambridge, MA: MIT Press. Hinton, G. E., & Shallice, T. (1991). Lesioning an attractor network: Investigations of acquired dyslexia. Psychological Review, 98, 74±95. Hintzman, D. L. (1986). ``Schema Abstraction'' in a multiple-trace memory model. Psychological Review, 93, 411±428. Hintzman, D. L. (1988). Judgements of frequency and recognition memory in a multiple-trace memory model. Psychological Review, 95, 528±551. Humphreys, M. S., Bain, J. D., & Pike, R. (1989). Di€erent ways to cue a coherent memory system: A theory of episodic, semantic, and procedural tasks. Psychological Review, 96, 208±233. Humphreys, M. S., Pike, R., Bain, J. D., & Tehan, G. (1989). Global matching: A comparison of the SAM, Minerva II, Matrix and TODAM models. Journal of Mathematical Psychology, 33, 36±67. Lewandowsky, S. (1991). Gradual unlearning and catastrophic interference: A comparison of distributed architectures. In W. E. Hockley & S. Lewandowsky, (Eds.), Relating theory and data: Essays on human memory in honor of Bennet B. Murdock, (pp. 445±476). Hillsdale, NJ:Erlbaum. Lewandowsky, S., & Murdock, B. B. (1989). Memory for serial order. Psychological Review, 96, 25±57. Luce, R. D. (1963). Detection and recognition. In R. D. Luce, R. R. Bush, & E. Galanter (Eds.), Handbook of mathematical psychology (Vol. 1, pp. 103±189). New York: Wiley. McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995). Why there are complementary learning systems in the Hippocampus and Neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, 419±457.

McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. In G. H. Bower (Ed.), The psychology of learning and motivation. (Vol. 24, pp. 109±164). New York: Academic Press. McGeoch, J. A., & Irion, A. L. (1952). The psychology of human learning. New York: Longmans. Meara, P. (1999). Self-organisation in bilingual lexicons. In P. Broeder and J. Murre (Eds), Language in Thought and Development (pp. 127±144), Tuebingen: Narr. Mensink, G. M., & Raaijmakers, J. G. W. (1988). A model for interference and forgetting. Psychological Review, 95, 434±455. Mensink, G. M., & Raaijmakers, J. G. W. (1989). A model for contextual ¯uctuation. Journal of Mathematical Psychology, 33, 172±186. Montague, P. R., & Sejnowski, T. J. (1994). The predictive brain: Temporal coincidence and temporal order in synaptic learning mechanisms. Learning & Memory, 1, 1±33. Murdock, B. B. (1993). TODAM 2: A model for the storage and retrieval of item, associative, and serial-order information. Psychological Review, 100, 183±203. Murre, J. M. J. (1992a). The e€ects of pattern presentation on interference in backpropagation networks. Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society, 54±59. Murre, J. M. J. (1992b). Categorization and learning in modular neural networks. Hemel Hempstead: Harvester Wheatsheaf. Copublished in Hillsdale, NJ: Erlbaum. Nickerson, R. S. (1984). Retrieval inhibition from part-set cuing: A persisting enigma in memory research. Memory & Cognition, 12, 531±552. Plaut, D. C. (1992). Relearning after damage in coonnectionist networks: Implications for patient rehabilitation. The proceedings of the 14th Annual Cognitive Science Society Conference, 372±377. Plaut, D. C. (1996). Relearning after damage in connectionist networks: Toward a theory of rehabilitation. Brain and Language, 52, 25±82. Raaijmakers, J. G. W. (1992). The story of the two-store model of memory: Past criticisms, current status, and future directions. In D. E. Meyer & S. Kornblum (Eds.), Attention & Performance XIV (pp. 467±488). Cambridge, MA: Bradford. Raaijmakers, J. G. W., & Shi€rin, R. M. (1981). Search of associative memory. Psychological Review, 88, 93±134. Ratcli€, R. (1990). Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97, 285±308. Ratcli€, R., Clark, S. E., & Shri€rin, R. M. (1990). List-strength e€ect: I. Data and discussion. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 163±178. Roediger, H. L., III., & Schmidt, S. R. (1980). Output interference in the recall of categorized and paired-associate lists. Journal of Experimental Psychology: Human Learning and Memory, 6, 91± 105. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. In D. E. Rumelhart, J. L. McClelland, & the PDP Research Group, Parallel distributed processing. Vol. 1: Foundations (pp. 319± 362). Cambridge, MA: MIT Press. Rundus, D. (1973). Negative e€ects of using list items as recall cues. Journal of Verbal Learning and Verbal Behavior, 12, 43±50. Shi€rin, R. M., Murnane, K., Gronlund, S., & Roth, M. (1989). On units of storage and retrieval. In C. Izawa (Ed.), Current issues in cognitive processes: The Tulane Floweree symposium on cognition (pp. 25±68). Hillsdale, NJ: Erlbaum. Shi€rin, R. M., & Raaijmakers, J. G. W. (1992). The SAM retrieval model: A retrospective and prospective. In A. F. Healy, S. M. Kosslyn, & R. M. Shri€rin (Eds.), From learning processes to cognitive processes: Essays in honour of William K. Estes (Vol. 2, pp. 69±86). Hillsdale, NJ: Erlbaum Shi€rin, R. M., Ratcli€, R., & Clark, S. E. (1990). List-strength e€ect: II. Theoretical mechanisms. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 179±195.

215 Shi€rin, R. M., & Steyvers, M. (1997). A model for recognition memory: REM-retrieving e€ectively from memory. Psychonomic Bulletin and Review, 4, 145±166. Siegel, S., & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences. McGraw Hill: New York. Tulving, E., & Hastie, R. (1972). Inhibition e€ects of intralist repetition. Journal of Experimental Psychology, 92, 297±304.

Wheeler, M. A., & Roediger, H. L., III. (1992). Disparate e€ects of repeated testing: Reconciling Ballard's (1913) and Bartlett's (1932) results. Psychological Science, 3, 240±245. Wixted, J. T., Ghadisha, H., & Vera, R. (1997). Recall latency following pure- and mixed-strength lists: A direct test of the relative strength model of free recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(3), 523±538.