advantage for items presented as spoken words and pictures simultaneously and ... explanations that incorporate at least two modalities. Mayer's theory builds ...
Bimodal Running head: BIMODAL EFFECTS
Bimodal Format Effects in Working Memory Paula Goolkasian and Paul W. Foos University of North Carolina, Charlotte
1
Bimodal
2
Abstract The present work combines presentation formats to test whether bimodal conditions offer any advantages or disadvantages relative to single formats in working memory performance.
A dual task that included recall of 3 or 6
items while verifying the accuracy of math sentences was used in two experiments.
When comparisons were made
between single and dual format conditions, there was an advantage for items presented as spoken words and pictures simultaneously and individually.
In our second experiment,
dual format conditions had incongruent information and spoken words were found to interfere with recall of long sequences of both pictures and printed words. The findings suggest that when dual format items are the same, there are some performance advantages when spoken words are combined with either pictures or printed words. When the dual formats are displaying different items, however, spoken words are a more powerful distractor than either pictures or printed words, and verbal and visual short-term stores can demonstrate similar susceptibility to distractor interference.
Bimodal
3
Bimodal Format Effects in Working Memory Presentation format effects are well established in the literature and have implications for the ways in which we present information in educational, employment, and other settings.
The early work concentrated on either picture-word
differences (Goolkasian & Park, 1980; Kroll & Corrigan, 1981; Pellegrino, Rosinski, Chiesi, & Siegel, 1977; Smith & Magee, 1980) or auditory-visual differences (Greene, 1985; Greene, Elliott, & Smith, 1988; Penny, 1989).
Our recent work
(Goolkasian & Foos, 2002) examined working memory for material presented in different modalities (auditory vs. visual) and different formats in the same modality (picture vs. printed word) with a dual task that measured the cognitive demands of processing and storage.
Results showed that participants
recalled and recognized more concrete nouns presented as spoken words or as pictures than printed words. Reaction time (RT) and accuracy on the processing component of the dual task were maintained at a stable rate in spite of variation in presentation format/modality. The present work builds on this previous effort by combining formats/modalities and testing whether bimodal format conditions offer any advantages relative to single formats in working memory performance due to differences in selective attention to items across presentation conditions.
The
Bimodal
4
rationale for this attention hypothesis comes from presentation format effects that are found in the instructional literature (Chandler & Sweller, 1992; Mayer & Sims, 1994; Mayer, 2001; Moreno & Mayer 2002) and from current conceptualizations of working memory (Baddeley & Hitch, 1974; Baddeley & Logie, 1999; Nairne, 1990). Mayer and his colleagues (Mayer & Sims, 1994; Mayer, 2001) developed a theory to explain why learning is facilitated by explanations that incorporate at least two modalities.
Mayer’s
theory builds on Paivio’s dual coding theory (1971, 1975, 1978) and the multicomponent system of working memory developed by Baddeley (Baddeley & Hitch, 1974; Baddeley & Logie, 1999).
In
Mayer’s theory (2001), the visual-spatial sketchpad and the phonological loop systems of working memory are developed into two separate but interconnected channels (visual/pictorial and auditory/verbal).
When items appear as pictures or as spoken
words, they are processed directly within one of these channels. Because printed words involve visual as well as verbal components, they are processed in a complex manner that is split between the two channels.
That is, although initially
represented in the visual channel, the processing of the printed word would transfer the information to the verbal channel for further processing.
Bimodal presentation formats (visual and
auditory) are hypothesized to be better for learning than single
Bimodal
5
modal formats, because they take advantage of the two parallel channels.
Processing similar or the same information through
the two channels strengthens the representation held in memory. Recently, Moreno and Mayer (2002) investigated the addition of on-screen text to narrated instructional materials, and found an advantage to comprehension of the materials compared to the narration condition alone. However, simultaneous presentation of dual formats (animation and text) in the visual modality did not result in better comprehension relative to the single format condition.
Benefits were evident only when the visual formats
were presented sequentially.
These findings were taken as
support for Mayer’s cognitive theory because the presentation of two visual formats was believed to overload the visual channel and cause participants to split their attention between two representations rather than using processing resources for building connections between visual and auditory channels. finding is similar to the split attention effect reported by
This
Chandler and Sweller (1992) when text added to a graph or a diagram was found to interfere with learning. Nairne’s (1990) feature model of immediate memory emphasizes similarities between formats and modalities. Forgetting happens primarily because of interference due to contiguous traces in working memory, and the more similarity among the features of these traces, the poorer the recall.
Bimodal
6
Distinctive features do not interfere and traces with distinctive features are, thus, better remembered.
Feature
similarities can arise from presentation format conditions or from internal processes that result from identification or categorization of item information.
The most interesting
prediction of the feature model from our perspective is that participants differentially allocate their attention resources across stimulus features with distinctive features receiving more attention. Early work by Posner, Nissen and Klein (1976) provided some evidence for differential attention effects by modality.
They
found that visual signals are less alerting than stimuli in other modalities, and that auditory signals had an advantage because of an automatic alerting effect. Others have shown differences in selective attention to visual and auditory stimuli (Melara & O’Brien 1987; Patching & Quinlan, 2002; Turatto, Benso, Galfano, & Umilta, 2002).
Auditory input of
words often results in better recall than visual input of the same words (the long-term modality effect, Gardiner, Gardiner, & Gregg, 1983; Greene & Crowder, 1986).
Such prior work clearly
suggests greater attention to auditory presentations and better memory for items presented in that mode. Our first comparisons look at predictions derived from Mayer (2001) and Nairne (1990).
According to Mayer’s
Bimodal
7
predictions, the bimodal conditions (i.e., picture and spoken word, printed and spoken word) should provide some benefit relative to the picture and printed word condition since this latter condition involves dual formats in only the visual mode. The predictions from the feature model (Nairne, 1990) are that forgetting results from interference from subsequently occurring items and item similarity is an important factor.
The
probability of a feature being overwritten is higher with increased similarity to subsequent events. Generalizing from this theory, picture and spoken word combinations should produce the most benefit because they are the most dissimilar. In our second experiment, we tested for differences in selective attention to items across presentation format conditions by using the dual format stimuli to present different items and measuring interference from incongruent target and distractor information.
Although Nairne’s model (1990) was
developed to explain relations among items and does not make specific predictions about target-distractor similarity, we tested whether the model could be extended to this situation. Support would be shown if the amount of interference depended upon the similarity between the combined formats/modalities. Those that have similar features are expected to produce more interference.
Bimodal
8
Our dual task, modeled on a complex operations span task (La Pointe and Engle, 1990), was used in our previous work (Goolkasian & Foos, 2002).
Our processing component was a
sentence verification task involving arithmetic facts and it included articulation of the stimulus sentences.
Articulation
has been shown (Goolkasian & Foos, 2002) to eliminate the sound advantage due to echoic memory.
Performance was recorded by
measuring RT and errors to the arithmetic sentences. The storage component was an immediate recall task that required participants to store either 3 or 6 items. Experiment 1 Experiment 1 was conducted to test for advantages of dual over single format items.
The presentation formats used in
previous work (Goolkasian & Foos, 2002) were combined into three dual format conditions (picture and printed word, picture and spoken word, and spoken word and printed word) that included the same item represented in two different presentation formats to test for redundancy benefits on working memory.
Method Participants.
The participants were 53 men and women
students from the University of North Carolina at Charlotte. They were volunteers who participated in the experiment to obtain extra credit points toward their psychology class grade.
Bimodal
9
Twenty-six students participated in Experiment 1, 27 in Experiment 2.
Stimulus materials. The processing task required participants to verify with a key press the accuracy of a series of math sentences. Each sentence contained two components—either multiplication or division of two integers, and the addition or subtraction of an integer from that result. The numbers used in both components were integers between 1 and 10. Correct and incorrect examples of the sentences used in the processing task are presented respectively: 1 = 2?”.
“Is (8 X 2) + 4 = 20?”, “Is (2/1) +
Numbers rather than words were used for the sentence
verification task because Campbell’s (1994) work showed a decline in performance when words were used for a math task. Also, to reduce math anxiety, only one integer above 5 was used for the first component of each sentence. The storage task involved remembering items presented in one of the following single (picture, printed word, spoken word) or dual formats (printed word/picture, printed word/spoken word, and spoken word/picture). The Appendix presents the 54 items used in the task. Each of the items appeared once in each of the single and dual presentation format/modality conditions.
Figure
1 presents an example of a printed word and picture stimulus. The stimuli for the storage task are concrete nouns that were chosen from a pool of several hundred clip art photos based on
Bimodal
10
ratings by an independent group of 40 students drawn from the same population as the participants.
An item was chosen for the
study if at least 90% of the raters agreed on a name for the picture.
Each picture was imported into Adobe Photoshop and its
size was adjusted to approximately 4 X 4 cm for use in the study.
When the items appeared as printed words, they were
upper case characters printed with a Geneva font in a character size of 24 cpi. The spoken words were sound files created by a human female voice and converted into a Macintosh system sound file.
The picture and printed word stimuli were created by
combining the picture and printed word versions of each item with the picture above the printed word.
In the bimodal
conditions, the sound files were played as the picture or the printed word was displayed. The stimuli were displayed in the center of an Apple 15” flat screen monitor.
Stimulus presentation and data collection
were controlled by SuperLab running on a PowerPC G4.
Spoken
words were presented with stereo speakers.
Procedure.
Participants were run individually in sessions
of around 30 min.
They were given 5 practice trials to
familiarize themselves with the task.
They were asked to read
out loud the math sentence as soon as it appeared and then to verify its accuracy while simultaneously remembering a number of items (for a total of 3 or 6 on each trial) presented in one of
Bimodal the 6 presentation format conditions.
11
The math sentence
appeared first and remained on the screen until the participant made a yes/no response using their right or left index finger. A dual or single format item from the storage task followed the participant’s response.
After 1 s, a second math sentence
appeared followed by the second item from the storage task. This continued until a string of three question marks cued recall.
The participants called out as many of the items as
they could remember in any order and their spoken responses were recorded by a research assistant. Participants had 36 trials in which they were asked to recall the items presented.
The number of items to-be-
remembered (3 or 6 items) were combined with the 6 presentation formats (picture, printed word, spoken word, picture and printed word, picture and spoken word, spoken word and printed word). Each of the 12 experimental conditions was replicated 3 times. Use of an item within each of the three single and dual format conditions was counterbalanced across replications.
Within each
replication, the order of the 12 experimental conditions was randomized. There were 54 math sentences developed for the sentence verification task. The sentences were used 3 times—once in each replication of the 12 experimental conditions.
As much as
possible, there were an equal number of multiplication and
Bimodal division, and subtraction and addition facts.
12
The pairing of
the items used in the recall task with the math sentences were random with the constraint that within each of the 12 experimental conditions there were an equal number of correct and incorrect math sentences. In all other respects our working memory task was the same as used in our previous research (Goolkasian & Foos, 2002).
The
computer recorded the RTs and error rates from the sentence verification task.
Participants were encouraged to recall as
many of the items as they could.
Recall was scored as the
proportion of presented items correctly remembered.
Results For both experiments, means were computed from the correct RTs to the sentence verification task and from the proportion of items correctly remembered from each participant across the 3 replications within each of the experimental conditions. recorded were the proportions of incorrect RT responses.
Also Data
were treated separately with a 6 X 2 repeated measures analysis of variance (ANOVA) to test for presentation format and number of items to-be-remembered. The F tests that are reported include the Geisser-Greenhouse correction to protect against possible violation of the homogeneity assumption. Figure 2 presents the means with 95% confidence intervals. The analysis of the recall data (presented in the top panel of
Bimodal
13
Figure 2) showed strong effects of both presentation format, F (5,125) = 18.61, p
