benefits of computational modeling for cognitive ... - CiteSeerX

BENEFITS OF COMPUTATIONAL MODELING FOR COGNITIVE NEUROSCIENCE STUDIES OF VERBAL WORKING MEMORY D. E. Meyer, D. E. Kieras, S. T. Mueller, T. L. Seymour University of Michigan Sponsored by the U.S. Office of Naval Research

Introduction Cognitive Neuroscience has two goals: (1) to identify and characterize the mental processes whereby cognitive and perceptual-motor tasks are performed; (2) to discover how the nervous system implements these processes. For these goals to be achieved, the algorithmic operations of each process must be specified in detail. Without such specifications, attempts to discover the functional roles of particular brain mechanisms are doomed to fail (Marr, 1982). Thus we have been formulating algorithmic computational models of performance in important cognitive tasks. One such case is the verbal serial memory-span task. This task interests cognitive neuroscientists because data from it may help analyze mental processes and brain mechanisms of verbal working memory (Baddeley, 1986). For example, D'Esposito and Postle (1999) claim that memory span primarily manifests working-memory (WM) "storage" and is "unconfounded by executive control processes." Some evidence appears to support this claim. Patients with damaged left inferior parietal cortex have abnormally low verbal memory spans (Vallar & Papagno, 1995).

In contrast, patients with damaged dorsolateral prefrontal cortex (DLPFC) have seemingly normal verbal memory spans, even though DLPFC is a putative site of executive control for verbal WM (Jonides et al., 1997). This pattern might lead one to propose the simple mind-brain model of WM shown below, in which parietal cortex mediates storage for the serial memory-span task, ventrolateral prefrontal cortex mediates rehearsal for delayed-response tasks, and DLPFC mediates executive control for N-back memory tasks, whereas DLPFC and executive control play no N-Back Memory-Span Tasks role in the memory-span task (D'Esposito & Postle, 1999). Memory Tasks Nevertheless, insights from our computational modeling reveal that this simple mind-brain model is incorrect. Contrary to it, we have found that verbal memory span requires not only storage but also elaborate executive control processes for coding, maintaining, and reproducing serial-order information. This finding raises fundamental questions about where and how such control is implemented in the brain.

CONTROL

STORAGE

REHEARSAL

Delayed Response Tasks Simple Mind-Brain Model (D’Esposito & Postle, 1999)

The EPIC Architecture Our discoveries are based on modeling with the Executive-Process Interactive Control (EPIC) architecture shown here. EPIC is a theoretical system that implements algorithmic information processing in an interactive brain-like manner. Like

the brain, EPIC has distinct modules for perceptual, cognitive, and motor operations. EPIC's cognitive processor controls perception and action through production rules that "fire" in parallel like neural networks do. Also, EPIC's WM contains stimulus, response, and control codes, enabling performance of various WM tasks to be modeled realistically. Schematic diagram of the EPIC architecture (Meyer & Kieras, 1999).

Functional Properties of EPIC's Working-Memory Stores • EPIC has distinct WM stores for each perceptual and motor modality. • Modality-specific codes for multiple items may be stored in WM. • There is no size-based capacity limit in any WM store. • Each code in WM decays through a stochastic all-or-none process. • Overt and covert speech (phonological) codes are stored in auditory WM. • Speech codes are represented with respect to phonological similarity. • The serial order of speech codes is represented by chain tags. • Object files are stored in visual WM. • Object files represent conjoint perceptual features of objects. • Task goals and status notes for control are stored in a separate part of WM. • Tasks are performed with sets of production rules that sequentially manipulate WM's contents to comply with task instructions and achieve task goals. • The cognitive processor uses WM's contents to test production-rule conditions. • Production-rule actions periodically update the contents of WM.

EPIC Model of Verbal WM in the Memory-Span Task Using the realistic components of EPIC, we have formulated precise computational models of WM that emulate a phonological-loop mechanism in detail (cf. Baddeley & Hitch, 1974; Schweickert & Boruff, 1986; Waugh & Norman, 1965). Our models account quantitatively for performance of representative WM tasks (Kieras, Meyer, Mueller, & Seymour, 1999). For example, one prototypical case with which we have dealt especially is the serial memory-span task. In what follows, a generic version of this task is considered more fully, and empirical results from it are fit with simulated outputs from one of our EPIC models. To achieve this fit, the present model includes sets of production rules that implement a performance strategy with three complementary functions: storage, rehearsal, and recall. We discovered that to implement these functions, they must be coordinated by a highly elaborate executive control process (see Appendix 1). Without such control, it is impossible to perform the memory-span task properly. This realization, and the empirical success of our model, therefore have important implications for cognitive neuroscience studies of verbal WM.

Generic Serial Memory-Span Task The memory-span task modeled here has the design below: • On each trial, 3 to 9 words are presented auditorily at a constant moderate rate. • After the final word of a trial, participants hear a signal (BEEP) that prompts them to recall the presented word sequence in its original serial order. • Ample time is allowed for recall, after which a new trial starts. • Word sequences are constructed randomly from a small pool of words. • No word is used more than once per trial, but words occur repeatedly across trials. ...“river...ticket...crayon” “river…”

…BEEP “river…ticket...”

...“river,ticket,crayon”

Memory-Span Performance Strategy • Consistent with EPIC, the WM storage and perceptual-motor processes for the memory-span task are assumed to involve specific memory modalities and effectors. • Auditorily perceived stimuli are held in a phonological WM buffer. • Items haphazardly decay in an all-or-none fashion from this buffer, so subvocal articulation is used to reactivate the auditory perceptual processor, yielding fresh (covert) copies of the verbal information in the phonological buffer. • Subvocal articulatory and auditory perceptual processors serve as components of a programmable strategic phonological loop. • Operation of the phonological loop is coordinated by an executive control process whose production rules use available auditory and articulatory mechanisms for storage, rehearsal, and recall of word sequences (Appendix 1). Auditory Processor

Phonological Working Memory

Executive Control Processes

Vocal Motor Processor

Rehearsal Process in the EPIC Model • Phonological codes for words are maintained in WM through rehearsal. • Rehearsal is a cyclic process coordinated by executive control. • The rehearsal process is implemented by a set of production rules that embody procedural knowledge for performing the memory-span task. • Rehearsal production rules cause specific words to be articulated, and these rules achieve various subgoals required to complete performance. • The rehearsal process keeps track of two chains of items in phonological (auditory) WM: the rehearsal-chain and the add-chain. • Maintenance of WM involves updating and manipulating the contents of these item chains (see Appendix 2). • Computational modeling allows us to characterize both the phonological loop and the executive control processes more accurately and thoroughly than has been possible before with informal theorizing (cf. Baddeley et al., 1975).

Key Insights about the Rehearsal Process • The detailed executive, storage, and retrieval processes for rehearsal are highly elaborate. The “simple” serial memory-span task is actually quite complex! • When a new stimulus item is perceived and stored, tags are generated that place a link to this item at the end of the add-chain (see below). • Concurrently and asynchronously with the construction of the add-chain, the rehearsal-chain is continuously cycled and rebuilt through covert articulation. A new rehearsal-chain is built by articulating the current rehearsal-chain followed by the current add-chain. For more details, see Appendix 2. • A new add-chain is created after a new rehearsal-chain has been built. • Multiple copies of words may exist in WM, but only the most recent copy is used for rehearsal. Old copies disappear through stochastic all-or-none decay. • Articulatory suppression requires activity of the vocal motor processor and thus precludes subvocal rehearsal. During articulatory suppression, items remain in (and haphazardly decay from) a stimulus chain similar to the add-chain.

Rehearsal-Chain and Add-Chain Construction Rehearsal-Chain

Add-Chain

River3

Cabin1

Ticket2

Pillow1

Rehearsal Cycle 2

River4 Rehearsal Cycle 3

Ticket3 Cabin2 Pillow2

Dagger1 Crayon1

During a single rehearsal cycle, EPIC generates a new rehearsal-chain by first rehearsing all items in the current rehearsal-chain. Next, EPIC completes the new rehearsal-chain by rehearsing the items in the current add-chain. The current add-chain contains copies of new stimulus words that have been received from the auditory-perceptual processor during subvocalization of the prior rehearsal-chain. Subscripts denote the successive copies of each word that are being used in these chains. For instance, during Rehearsal Cycle 3, “River” has been heard and/or rehearsed four times, while “Dagger” has only been heard once.

Recall Process in the EPIC Model • On each trial, after the recall cue has been perceived and the current rehearsal cycle has been completed, EPIC attempts to overtly recall the items in the rehearsal-chain. • These items are transferred individually and serially from phonological WM to the vocal motor processor. • Items are prone to haphazard decay, so recall errors occur when an item decays from WM before it can be recalled. • An item may decay during either rehearsal or recall. • Under the current task strategy, the executive control process terminates performance on a trial when it attempts to recall or rehearse an item that has already decayed from WM and become unavailable. • After a to-be-recalled item has decayed, the current executive process does not attempt to guess. However, guessing may be implemented in the model, which would not change the basic conclusions that we have reached from it here. • For more details about the recall process, see Appendix 3.

Other Parsimonious Assumptions Of The Model • The serial order of items is represented by supplementary tags that form implicit linked-list structures in the rehearsal-chains and add-chains. • Phonologically similar items have shorter decay times. • No inherent fixed limit exists on the number of items stored in phonological WM. • Limitations in phonological WM capacity stem from time-based decay. • Distinct codes are used for items from external (overt) and internal (covert) sources. • The decay of stored items from WM is an all-or-none process. • Individual stored items have stochastically independent decay times. • Decay times have a log-normal distribution with two parameters: M, the median, and s, the “spread”. • The values of M and s are affected by the stored items’ phonological similarity and source (external presentation or internal articulation).

Applications of the EPIC Working Memory Model We have applied EPIC’s verbal WM model to account for the quantitative results of several representative studies that used the generic serial memory-span task. These studies provide informative data on the effects of articulatory suppression, phonological similarity, list length, and word duration, making them good benchmarks against which to test our model. The figures below focus on studies by Baddeley et al. (1975) and Longoni et al. (1993): Experimental design and results of Baddeley, Thomson & Buchanan (1975) • Lists of English words (with British participants). • 2 (Word Duration) x 5 (List Length) design. • Results: Memory Span decreases as word duration increases. Experimental design and results of Longoni, Richardson & Aiello (1993) • Lists of 4 Italian words (with Italian participants). • 2 (Rehearsal/Suppression) x 2 (Word Duration) x 2 (Similarity) design. • Results: Memory Span decreases as word duration increases. Word Duration Effect independent of Similarity.

Simulation of Results from Baddeley et al. (1975) 100

Short Words

75 Observed

50

Predicted 25

0

Percent Recall

Percent Recall

100

Long Words

75 Observed

50

Predicted 25

0 4

5

6

List Length

7

8

4

5

6

7

8

List Length

Observed and predicted results for the study of Baddeley et al. (1975, Exp. 1). Dark bars represent observed percentages of trials on which serial recall was perfectly correct for sequences of short-duration and long-duration words as a function of list length (i.e., the number of words per sequence). White bars adjacent to the dark bars represent corresponding predicted percentages of trials on which serial recall was perfectly correct under the present EPIC model of verbal WM. The good fit of the model supports its plausible assumptions about the executive, storage, and rehearsal processes in the generic memory-span task.

Simulation of Results from Longoni et al. (1993) 100

100

Phonologically distinct words

75 Obs. Distinct Short Pred. Distinct Short

50

Obs. Distinct Long Pred. Distinct Long

25

0

Percent Recall

Percent Recall

75

Phonologically similar words Obs. Similar Short Pred. Similar Short

50

Obs. Similar Long Pred. Similar Long

25

0 Present

Rehearsal

Absent

Present

Absent

Rehearsal

Observed and predicted results for the study of Longoni et al. (1993, Exp. 1). Dark bars represent observed percentages of trials on which serial recall was perfectly correct as a function of word duration (short vs. long) and articulatory suppression (rehearsal absent) versus nonsuppression (rehearsal present). White bars adjacent to the dark bars represent corresponding predicted percentages of correct-recall trials under the present EPIC model of verbal WM. The good fit of the model again supports its plausible assumptions about the executive, storage and rehearsal processes in the generic memory-span task.

Parameter Values in Simulation for Longoni et al. (1993) Item Source external

internal

Phonological Relationships

M (ms)

s

similar

6625

0.2

distinct

7400

0.2

similar

4875

0.5

distinct

5500

0.5

Note: M is the median of the lognormal decay-time distribution for items in auditory WM; s is the distribution's spread parameter. The external source corresponds to overt auditory stimulation, and the internal source corresponds to covert vocal rehearsal. Although there are several other parameters in the EPIC architecture that can change, only the mean and variance of the decay distribution were adjusted to account for these data. All other parameters were held constant at predetermined values in the simulations for Longoni et al. (1993) and Baddeley et al. (1975). These constraints make the model’s good fit to the data even more compelling.

Simulation of Results from Other Memory-Span Studies

Proportion Correct (Observed)

Using our EPIC model of verbal WM, Observed vs Predicted Memory we have simulated results from more studies with the serial memory-span task, including 1 Baddeley (1966, Exps. 1-3; 1968, Exps. 1-3) 0.8 and Longoni et al. (1993, Exps. 1-2). 0.6 Together, they reported 20 distinct observed 0.4 memory spans as a function of word 0.2 duration and phonological similarity 0 (see graph on right). Our model accurately 0 0.2 0.4 0.6 0.8 1 Proportion Correct (Predicted) predicts this large data set (R2 = 0.90) on the basis of a few parameter values corresponding to a priori variations of word duration and phonological similarity across experiments. The model's goodness-of-fit confirms that its assumptions about WM executive control, storage, and rehearsal should be taken seriously.

Conclusions • A precise computational model of verbal WM based on the EPIC architecture accounts accurately for quantitative data from the serial memory-span task. • The model implements a realistic phonological-loop mechanism that is specified with substantially more detail than in previous theories (e.g. Baddeley, 1986). • Rigorous formulation of the model requires not only WM storage and rehearsal but also elaborate executive control processes. • Executive control processes are needed to coordinate phonological coding, itemchain construction, subvocal rehearsal, and final recall in the memory-span task. • Auditory WM stores two types of phonological code: (1) "overt" codes from external speech input; (2) "covert" codes from internal subvocal rehearsal. • Covert phonological codes decay more quickly than do overt phonological codes. • Phonological similarity decreases the useful lifetimes of these codes. • Despite differential decay rates, phonological codes for individual words in WM persist about 5 to 10 sec on average, which is substantially longer than some theorists have claimed (cf. Baddeley, 1986).

Implications for Cognitive Neuroscience Executive control processes in the verbal memory-span task are much more elaborate than has been claimed previously by some cognitive neuroscientists (e.g., D'Esposito & Postle, 1999). The memory-span task is not a "pure storage" task. Nor does it seem likely that any other WM tasks merely involve "pure storage". Instead, rigorous algorithmic computational modeling reveals that most, if not all, WM tasks probably require some significant executive control. This is because storage of items in WM must be maintained to preclude loss of information through decay, and such maintenance entails systematic iterative rehearsal, which in turn must be coordinated carefully. The algorithmic operations for this coordination are necessarily and logically complex, regardless of the "hardware" on which they happen to be implemented (cf. Marr, 1982). Thus, although hypotheses about "pure storage" tasks have pervaded cognitive neuroscience studies of WM during the past decade (for a review, see Smith et al., 1998), these hypotheses henceforth should be abandoned.

Concomitantly, the simple mind-brain model shown before (Introduction) will have to be modified and elaborated. Also, results from past cognitiveneuroscience studies with the memory-span task and other WM procedures will have to be reinterpreted. For example, it has been claimed by some researchers that damage to DLPFC does not affect verbal memory span significantly, whereas damage to left inferior parietal cortex does (D'Esposito & Postle, 1999; Vallar & Papagno, 1995). Assuming that executive control is localized only in DLPFC, these researchers have inferred that the memory-span task requires little, if any, executive control. However, our present work demonstrates compellingly that this inference is false. Given available data and theory, one must conclude instead that either: (1) the brain implements major executive control processes elsewhere than just in DLPFC; and/or (2) previous studies of patients with DLPFC damage have been misleadingly insensitive to reductions of memory span caused by deficient DLPFC executive control. Future progress of Cognitive Neuroscience may therefore occur through combining more rigorous computational modeling of WM task performance with a further careful search for additional brain sites that implement important executive control processes.

Trial Start

Appendix 1: Executive Control Processes in EPIC Model of Verbal WM

Wait for external stimulus Tag as new

Wait for covert input If first external item, tag as stimulus chain start

Tag as new If last external item, wait for recall signal

If add chain empty, tag as add chain start

If start item, tag as new rehearsal chain start

Wait for recall signal and rehearsal complete Start recall process Wait for recall complete

Terminate all threads

Trial done

Wait for rehearsal complete

If item in add chain, start rehearsal process

Flowchart of the executive control processes and overall task strategy used by the EPIC model of verbal WM for performing the serial memory-span task. The task strategy includes concurrent processes for rehearsalchain and add-chain construction, subvocal rehearsal, and final recall.

Rehearsal Start

Very first item?

Appendix 2: Rehearsal Process in EPIC Model of Verbal WM

no

yes Subvocalize external code Subvocalize first item in rehearsal chain

More in rehearsal chain?

no

yes Subvocalize next item

no

More in add chain? yes

Subvocalize next add chain item

Rehearsal Done If required auditory item is missing at any point, clean up and exit

Flowchart of the rehearsal process used by the present EPIC model of verbal WM for performing the serial memory-span task. Represented here are the operations performed during one cycle of rehearsal. These operations include, when need be, subvocalizing the first stimulus item on a trial, subvocalizing each item of the current rehearsal-chain in auditory WM, and then subvocalizing each item in the current add-chain. On trials with articulatory suppression, no rehearsal process is needed.

Appendix 3: Final Recall Start - Rehearsal Used

Say first item in rehearsal chain

More in rehearsal chain?

no

yes

Recall Process in EPIC Model of Verbal WM

Recall Start - No Rehearsal

Say first item in stimulus chain

More in stimulus chain?

no

yes

Say next item

Say next item

Recall Done

Recall Done

If required auditory item is missing at any point, clean up and exit

Flowchart of the final recall process used by the EPIC model of verbal WM for performing the serial memory-span task. The top panel shows steps in recall after prior rehearsal has occurred on a trial. The bottom panel shows steps in recall if there has been no prior rehearsal (e.g., if articulatory suppression has precluded rehearsal).

References Baddeley, A. D. (1966). Short-term memory for word sequences as a function of acoustic, semantic, and formal similarity. Quarterly Journal of Experimental Psychology, 18, 362-365. Baddeley, A. D. (1968). How does acoustic similarity influence short-term memory?. Quarterly Journal of Experimental Psychology, 20, 249-264. Baddeley, A. D. (1986). Working memory. Oxford, UK: Oxford University Press. Baddeley, A. D., Thomson, N., & Buchanan, M. (1975). Word length and the structure of short-term memory. Journal of Verbal Learning and Verbal Behavior, 14, 575-589. D'Esposito, M., & Postle, B. R. (1999). Neural correlates of component processes of working memory: Evidence from neuropsychological and pharmacological studies. In S. Monsell & J. Driver (Eds.), Control of cognitive processes: Attention and performance XVIII. Cambridge, MA: M.I.T. Press. Jonides, J., et al. (1997). Verbal working memory load affects regional brain activation as measured by PET. Journal of Cognitive Neuroscience, 9, 462-475. Kieras, D.E., & Meyer, D. E., Mueller, S. T., Seymour, T. L. (1999). Insights into working memory from the perspective of The EPIC Architecture for modeling skilled perceptual-motor and cognitive human performance, in A. Miyaki & P. Shah (Eds.), Models of working memory. New York: Cambridge University Press. Longoni, A. M., Richardson, A. T. E., & Aiello, A. (1993). Articulatory rehearsal and phonological storage in working memory. Memory & Cognition, 21, 11-22. Marr, D. (1982). Vision. New York: Freeman. Meyer, D. E., & Kieras, D. E. (1999). Precis to a practical unified theory of cognition and action: Some lessons from computational modeling of human multiple-task performance. In D. Gopher & A. Koriat (Eds.), Attention and performance XVII (pp. 15-88). Cambridge, MA: M.I.T. Press. Schweickert, R., & Boruff, B. (1986). Short-term memory capacity: Magic number or magic spell? Journal of Experimental Psychology: Learning, Memory, and Cognition, 12, 419-425. Smith, E. E., Jonides, J., Marshuetz, C., & Koeppe, R. A. (1998). Components of verbal working memory: Evidence from neuroimaging. Proceedings of The National Academy of Sciences, 95, 876-882. Standing, L., Bond, B., Smith, P., & Isely, C. (1980). Is the immediate memory span determined by subvocalization rate? British Journal of Psychology, 71, 525-539. Vallar, G., & Papagno, C. (1995). Neuropsychological impairments of short-term memory. In A. D. Baddeley, B. A. Wilson, & F. N. Watts (Eds.), Handbook of memory disorders (pp. 135-165). New York: John Wiley.