H. Mandl, J.R. Levin (Editors). 0 Elsevier Science ..... 1975; Pepper, 1981). Transformation ..... In J.F. Le Ny, & W. .... Essays in Honor of Allan Paivio. Hillsdale ...
Knowledge Acquisition from Text and Pictures H. Mandl, J.R. Levin (Editors) 0 Elsevier Science Publishers B.V. (North-Holland), 1989
3
1 Problems in Knowledge Acquisition from Text and Pictures
Sylvie Molitor, Steffen-Peter Ballstaedt & Heinz Mandl Deutsches Institut fur Femstudien an der Universitiit Tiibingen Federal Republic of Germany
Introduction Texts and pictures can be regarded as media because they visualize parts of reality which are not immediately present or accessible in one's direct experience (Bruner & Olson, 1973). By means of language symbols and syntactic rules, an expository text objectives the author's knowledge concerning an area of reality (Ballstaedt, Mandl, Schnotz & Tergan, 1981; Britton & Black, 1985). In representational pictures, one can differentiate two kinds of pictures: (1) an illustration objectives an optical arrangement on a surface area, which corresponds to an area of reality in the relevant aspects of perception (Gibson, 1982). Examples of this kind are photos, sketches, and caricatures. (2) On the other hand, a visualization illustrates structures and relationships of the reality, which cannot be perceived under normal visual conditions. Such examples are hierarchies, flow charts, and organizational schemata. Since text and pictures are nowadays treated in parallel as two different media to convey knowledge, one mostly forgets that - in the evolution of cognition - writing developed from simple sketches (FSldes-Papp, 1966; Klix, 1980). The way from the original pictograms to the phonetic alphabet passes through several intermediate phases like, for example, the Egyptian hieroglyphics, where illustrations and texts still coincide. This connection between text and pictures finds structural expression in the organization of the brain: the reading centre in the gyrus angularis is situated between the visual fields of the cortex and the sensory language centre (Geschwind, 1972).
4
S.Molitor, S.-P. Ballstaedt and H. Mandl
For a long period of time, pictures were the primary medium for illiterate people to which religious and political contents could be conveyed by means of illustrations only. However, the spreading of wall paintings and illuminations was kept within narrow bounds. It was only after the invention of wood engraving at the end of the 14th century and subsequently of other kinds of printing techniques (copperplate engraving, etching), that pictures could be used to convey knowledge on a larger scale (e.g., the illustrated broadsheet, which was very popular among the people).
Figure 1: A technical drawing from one of Leonardo's note-books showing a
screw thread with a pressure-operated ball-bearing. The construction is explained to the left of the diagram. The reason why Leonardo wrote in mirror writing has not been clarified definitely. (Adapted from Heydenreich, Dibner & Reti, "Leonardo the Inventor". Stuttgartflurich: Belser, 1981, p. 179).
During the Renaissance pictures became a medium for the emerging sciences. Leonardo da Vinci (1452 - 1519) is considered as a precursor of scientific dia-grams. His note-books contained a wealth of drawings with handwritten comments. They involved every field of science of the time, such as anatomy, botany, architecture, cartography, optics, geometry, and mechanics, etc. Figure 1 shows one of Leonardo's technical drawings. This kind of
Knowledge Acquisition
5
documentation, by means of a representation and commentary, corresponds to the notion of the Renaissance - trust your eyes more than the scholastic knowledge in books. Johann Amos Comenius linked for the first time an expressly pedagogical intention to the application of pictures in his Latin text-book, "Orbis sensualium pictus" (1658). In this piece of work, approximately 4000 words are introduced, each of them grouped together according to an area of reality (i-e., an area of knowledge, and illustrated by a wood carving). Figure 2 shows an opened double page of the text-book that had a wide distribution in its time. Comenius not only regarded the pictures as a mnemonic aid, but he also substantiated their use in the bounds of a holistic pedagogy. According to his sensualistic theory, man primarily acquires knowledge about the world by his senses within coherent scenarios, not just about isolated objects. Thus, the text-book correspondingly offered Latin vocabulary within a coherent text along with a summarizing picture.
Figure 2: Two pages taken from Comenius' Latin text-book. The concepts presented in text and picture belong to the knowledge area "phases of the moon". The numbers are used as a reference between parts of the text and the picture.
6
S. Molitor. S.-P. Ballstaedt and H. Mandl
Since the time of Comenius' "picture-book", graphicness has remained a central concept in pedagogy and didactics. Illustrations have become an indispensable component in teaching material and other expository texts. One was convinced about their usefulness in the acquisition of knowledge. Only since an empirical instructional psychology (and subsequently, media psychology) emerged, learning with pictures has begun to be investigated systematically. Since this time, the problem seems to have become increasingly complex, that is, mainly since theories and methods of cognitive psychology have been considered, which consider the reader (or the observer) to be involved in an active and constructive interaction with the text and the pictures. The state of the art reached in these efforts is presented in the following. We parse the problem area under three aspects: 1. How is knowledge acquired from texts and/or pictures represented in memory? 2. How are texts and pictures related to one another in the process of knowledge acquisition? 3. How can text-picture combinations be designed so as to promote effective learning? Although the largest part of text-picture investigations concerns didactic texts with illustrations, many of the considerations and findings are also valid for other text-picture combinations, as for example, slide-shows, audiovisual shows, film, and video.
Representation In the peripheral stages of processing - during information decoding - texts and pictures are indeed subject to different processes, but the differences between reading and viewing are not as great as one may at first presume. The figural, as well as the graphemic, analysis includes special saccadic eye movements and processes of pattern recognition (Kolers, 1973, 1977). These common features still refer to the development of writing from pictorial representations. Although the decoding processes in reading and viewing have been well researched, little is still known in cognitive psychology about the higher stages of processing. It has been strongly argued as to whether text and pictures are processed and stored in different memory systems and then in a different format, or whether there exists only a unique memory system where all knowledge is stored in one format independently of its origin from text or picture.
KnowledgeAcquisition
7
Before the most general approaches are described, the terminology must be explained. There are two levels of representation which often are not accurately distinguished in the literature: the Mental representation characterizes the representation of reality in our brain. In everyday language this refers to the different forms of knowledge - they are the traces of past processes of knowledge acquisition, which can be reactivated. In contrast, the term knowledge representation is reserved for a formalized description of this knowledge (e.g., as a network, a list of propositions, schemata, etc). Dual and Amodal Representation, There is an approach by Paivio (1971, 1977) which has been repeatedly picked up in media research and which assumes two processing systems which function independently, yet interact with each other: a verbal system specialized in processing linguistic information, and an imaginal system specialized for processing visual information. Paivio is primarily not interested in examining problems of representation, but he investigates the knowledge processing within and between these two systems. Thus, a text is predominantly processed and stored in the verbal system - only the concrete information of the text is visualized and thus also transmitted into the imaginal system. Pictures, on the other hand, are primarily processed and stored in the imaginal system, yet at the same time they enter the verbal system as a partially verbalized "copy". Therefore, dual coding is possible in principle for texts as well as for pictures in actuality, however, mainly for pictures, which are doubly stored. This does not mean that one should think of a picture gallery kept in mind, but one should imagine processes of perception in the imaginal system, which can be used to construe visual representations. By means of this approach, Paivio (1983) can interpret numerous experimental findings, based almost exclusively on the use of nouns and pictures of concrete objects as learning materials. This is how he explains, for example, the often found picture-superiorityeffect: when learning from texts and pictures occurs, pictures can always be retrieved from both memory systems. This approach of a central modalityspecific processing of texts and pictures is schematically represented in Figure 3. Both the approach of dual coding and the interpretation of Paivio's experiments have been violently attacked. Above all, representatives of research in Artificial Intelligence maintain that all of our knowledge is stored in a unique memory system in a propositional format independently of whether it was decoded 'as linguistic or visual information. The proposition stands beyond the linguistic and visual realm, being an amodal, abstract representation of knowledge.
S.Molitor, S.-P. Ballstaedt and H.Mandl
8
1
111
I1
central processing
peripheral processing
I
A
T
P
T
P
T
P
Figure 3: Three possible approaches for processing texts (T) and pictures (P). They all assume modality-specific peripheral processing. They differ, however, in their assumptions on central processing: (I) assumes two modality-specific processing systems, 011) assumes a uniform processing system, and 01) assumes a two-stage processing system (adapted from Farah, 1987). This single-code approach is portrayed in Figure 3 (111). Pylyshyn (1981) is the most radical proponent of this approach. Other authors represent more moderate positions, which assume an imaginal system at a central level of processing, but in the end all knowledge is also propositional for them (Kieras, 1978 Denis, 1982; Kosslyn, 1981). A compromise of this kind (11) is schematized between the other two approaches in Figure 3. "Pictorial knowledge" as a network of propositions is basically conceivable, but then the hardly deniable existence of imagery and dreams causes difficulties. Pylyshyn considers images as an epiphenomenon of processing, a cognitive luxury equally controlled by propositions. But this assumption also presupposes the existence of traces of perception, from which images can be construed. There is no direct way for the proposition (PINK, BIRD) to lead one to the image of a flamingo without
Knowledge Acquisition
9
processes analogous to perception taking place. Since the main proponents of amodal, propositional knowledge representation all come from the camp of Artificial Intelligence, one may suspect that the phenomenon of imagery is burdensome to them because it does not comply with the computational approach and causes difficulties for simulation, whilst propositional networks are relatively easy to program. A number of research programs are also on the test stand in the debate over mental representations, otherwise the pungency of the argument would be hard to understand. There are a series of other approaches which compare word and picture processing (Nelson, 1979; Durso & Johnson, 1980; Snodgrass, 1984; Kolers & Brison, 1984). The central issue concerning all of them, however, is: are there different modality-specific processing systems, or is there only one unique amodal system in the end? We should briefly elucidate the implications brought on by these positions for media research. If the assumption of a possible dual coding in two different knowledge systems is true, then texts and pictures are subject to different processes. It is then also conceivable and probably the case that inter-individual differences as well as the presentation form in the one medium or in the other play an essential role with respect to learning. If, on the other hand, the hypothesis of a uniform amodal processing system is correct, then the mode of presentation should play a lesser role for learning, since it would only affect peripheral processing. If propositions are the common format for knowledge representation, then pictures and texts must be describable and comparable as propositional lists or networks. Chase and Clark (1972) have proceeded in this direction by comparing verification tasks with sentences and simple graphic representations, both of which are recoded into propositions in their model. Stone et.al. (1981) applied these methods consequently to text and picture research by using a uniform knowledge representation taken from Frederiksen (1975) to describe texts and pictures. In one experiment, subjects had to assemble the model of a loading cart according to directions presented to them either by text, by a series of line drawings or by both media (Stone & Glock, 1981). In order to make the contents of the text and of the pictures redundant, in a long process of adjustment a list of propositions was set up out of which the instructional text as well as the contents of the pictures could be generated. The findings of various process and outcome variables indicated a clear advantage for the combined presentation of text and pictures: this presentation resulted in significant,by fewer assembly errors than with the presentation of one medium alone. Although the text and the pictures
10
S. Molitor, S. -P. Ballstaedt and H. Mandl
were assumed to have identical contents, the two media when used individually led to different results. For three of the outcome variables the text led to worse results than the pictures, but for one of the measures the text was significantly better. From this it is quite obvious that to design text and pictures according to a list of propositions does not affect all of the important differences between the two media. In a task requiring especially the comprehension of spatial relations, pictures have an advantage over texts and this advantage may not be eliminated by propositional equivalence. Apart from that, it is doubtful that a structured list of propositions may at all describe more sophisticated pictures in a definite and unambiguous way. Mental Models. Mental models seem to offer a means of mediation between the different forms of knowledge (Johnson-Laird, 1983; Gentner & Stevens, 1983). A mental model is the representation of a limited area of reality in a format which permits the internal simulation of external processes, so that conclusions can be drawn and predictions made. Mental models regularly serve to solve problems and are hence relative to a specific task. Till now, mental models of simple physical systems have been examined (e.g., doorbells, watercycles, and steam-driven engines). But let us explain the construct with the help of a practical example: in order for me to find my bearings in a foreign city, I successively have to "draw a picture", that is, to construct an internal overview or topological map on the basis of external clues. Of course, I will use a city-plan as a medium to build a mental model. I can then "note" the short-cuts or detours before I decide to go a certain way (cf. mental maps by Downs & Stea, 1982). In mental models, reality is represented in an analogous, predominantly imaginative form (Steiner, 1988). Mental models are usually built by means of concrete experiences from the respective areas of reality. They can, however, also be conveyed by media, in which case the illustrative quality of pictures added to a text probably plays a central role. A suitable illustration can stimulate and facilitate the construction of a mental model (Seel, 1986). At the moment, mental models are in vogue in cognitive psychology. They represent a pleasant rehabilitation of imagery by reminding us of the functions images originally had for problem solving. Since mental models even prevail for animals at a prelinguistic stage (Roitblat, Bever & Terrace, 1984), they should only secondarily be subject to linguistic registration, i.e., to partial verbalization. Hence, in Figure 3, the existence of mental models excludes the approach of a unique propositional processing system (111), but it does not make a decision between the other two approaches (I and 11). A few critical comments are, however, necessary. First of all, the mental model construct is still very fuzzy and
11
Knowledge Acquisition
used in a different sense by each author. Indeed, in the mental-model approach imaginative, linguistic, declarative, and procedural knowledge are somehow compounded, but up to now it has remained unclear as to how this knowledge is holistically integrated into the mental model. Mental models are usually represented by labeled flow-charts or other schemata, as they also appear in numerous scientific publications (e.g. De Kleer & Brown, 1983). Whether certain visualizations influence the construction of a mental model, and in what way, remains an interesting issue for research. Empirical pieces of evidence can be referred to by each of the approaches mentioned, but the question as to how knowledge can be stored in the brain and which knowledge representation is the most adequate still remains an experimentally undecidable problem (Wippich, 1984). Mental representations can never be directly observed, but can only be inferred from processes in which they are activated. Seel & Strittmatter (1984)carried out an experiment in view of the various approaches on mental representations. They were mainly interested in the effects that pictures have (i.e., visual images), on the learning process with respect to the different forms of presentation and reproduction. We cannot review the extensive investigation here, but would like to present the interesting design of the study and a few results. In order to understand the experimental design,the reader should refer to Table 1.
Table 1: The experimental design by Seel & Strittmatter (1984,p. 37). Explanation in the text. Group
Presentation
Recoding into
Reproduction
Text Picture Text Picture
Text Picture Picture Text Picture Text
~
1 2
6
Picture Text Picture Text Picture & Text Picture & Text
7 8
Picture Text
3 4 5
--
__
Picture Text
12
S.Molitor, S.-P. Ballstaedt and H. Mandl
The authors of the experiment presented their subjects either a written path description, a stylized map, or both media. When only one medium was being employed, instructions were given to transfer the information into the other medium - i.e., they had to either draw a map from the text or write a path description from the map (groups 1, 2, 3, 4). The performance measure was a reproduction of either a path description (in brief text), or a drawing (in brief picture) - i.e., the medium of reproduction was either congruent or incongruent with the first medium of coding. Two groups ( 5 , 6) were given the text as well as the picture and were likewise tested by means of drawing or by verbal description. The two so-called control groups (7, 8) only got the text or the picture without any transformational instructions. A series of hypotheses can be tested by this experimental design and various dependent variables can be examined in the phases of recoding and reproduction. Unfortunately, the findings by Seel and Strittmatter are not as consistent or as interpretable as the authors suggest; hence, a replication of the investigation would be desirable. A main finding, however, was that - as expected - congruency between the encoding and reproduction media leads to better performance (cf. Levin, 1988). On the other hand, it was unexpectedly shown that active recoding according to the instructions was not more effective than the presentation of both media - a result that contradicts previous findings, however. In general, pictures improve information reproduction. The authors also took this well-known picture-superiority-effect from other investigations as evidence that there is no common amodal processing system for text and pictures. This is also supported by the asymmetry in the recodings between text and picture. In general, the subjects could more easily recode text into a picture than vice versa. The empirical evidence referred to by Seel and Strittmatter for an imaginal processing system could also be explained by propositionalists, although that would be a little less convenient. The problem of mental representations certainly cannot be solved by the methods of cognitive psychologists alone. It is therefore distressing that knowledge from other disciplines is not much taken into consideration. This mainly applies to the neuropsychological findings concerning the architecture and functional division of the brain (Hebb, 1968; Farah, 1985, 1988). In like manner, evolutionary-biological considerations about the phylogenesis of the cognitive apparatus are ignored, especially those concerning the function of visual knowledge- i.e., imagery (Hewes, 1978; Klix, 1980). Furthermore, the results from the cognitive development of children should be evaluated with respect to
Knowledge Acquisition
13
the issue of different mental representations (Piaget & Inhelder, 1966; Bruner, Olver & Greenfield, 1966). Throughout the last few years, experiments which could be of interest for cognitive psychology are carried out within the framework of Artificial Intelligence research to simulate analogous processes (e.g., Habel, 1987).
Processing of text and pictures Within the bounds of a cognitive-psychological research paradigm, knowledge acquisition from text and pictures has been interpreted as an interactive process between learners and the teaching material. Corresponding to their prior knowledge and their cognitive strategies, learners build up a specific knowledge structure based on the information obtained from texts and pictures. The first attempts to specify the processes of interaction between learner and teaching material can be found in the works of Salomon (Salomon & Snow, 1968; Salomon, 1970). Salomon describes his approach as "cognitive-functionalistic". His focus is on the learner's ability to decipher the symbol system of educational films and other visual media, and on the internal processes which are released by such presentations. With his concept of "Supplantation" he shows how, by the external simulation of cognitive processes through media, the corresponding internal processes can be stimulated or intensified. This concept is particularly important when certain cognitive abilities are lacking, e.g. the ability to extract an essential detail from a complex picture. According to Salomon, this ability can be simulated with the help of external media- e.g., by the filmic zoom technique and thereby compensated (Salomon, 1979). In this manner, the corresponding cognitive ability is acquired by means of internalization. The concept of supplantation has been productively applied, along with more recent approaches in cognitive psychology (e.g., Janl3en-Holldiek, 1984). The interaction between learner and educational material as dealt with in the supplantation approach still depicts a rather coarse way of analyzing learning processes. How the relationship between text and picture has an effect on processing has not yet been taken into account in this approach. How to envisage an integrative processing of texts and pictures, and by what means this can be influenced, still remains unclear today. Investigations can be regarded as a first step in this direction, dealing with aspects of this problem: the influence of pictures on the processing of verbal material and the influence of language on the processing of visual material. Investigations such as these will be briefly described in the following sections. After that, a few considerations will be added about an
14
S. Molitor. S. 9.Ballstaedt and H. Mandl
aspect of text-picture relationships, which appear to be important for an integrative processing of both sources of information: the aspect of their complementarity. Influence of Pictures on Language Processing. Most of the investigations carried out on the influence of pictures on language processing are only concerned with single sentences or sentence variants, and not with whole texts. As an example, experiments undertaken by a group of psycholinguists at Bochum will be cited, wherein the ability to retain pronominal sentences is tested with and without pictures (Bock & HiJrmann, 1974; Bock & Milz, 1977). The experiments included sentences like those in (1) and (2), which were offered as captioas for line drawings: (1) - The man washed his car. (2) - He washed it. The line drawings schematically represented the statement in the respective sentences. The picture belonging to these sentences was a man with a bucket standing in front of a car. The control group was given the sentences without the picture. Pictures did not seem to play a role in the retention of the noun sentences, whereas they considerably improved the retention of the pronoun sentences. The retention of noun sentences was improved by picture only when pictures and text were presented sequentially (rather than simultaneously). The authors explain the effect that the pictures have on the pronominal sentences by the "semantic gradient" existing between the sentence and the illustration. In contrast to the noun sentences, the pronominal sentences contain "semantic gaps", which are filled by the picture. Only by looking at the picture one knows who is meant by "he" and what is meant by "it". The semantic discrepancy between sentence and picture is a stronger incentive to examine the picture more closely than the noun sentence, which contains the same information as the picture. The improved recall of the noun sentences when sentence and picture were presented sequentially, shows, however, that discrepancy in the content alone is not sufficient to explain the picture effects. The discrepancy hypothesis and the sequencing effect draw one's attention to important aspects of picture-text interaction (provided that one does not lose sight of the fact that the above-mentioned example is dealing with very simple cases). In a longer and more complex text with its wealth of cross-references for coherence and usually more than one picture, such a one-to-one relation of semantic gaps in the text and individual elements in the picture may become increasingly difficult and, hence, the semantic discrepancy is hard to determine more precisely.
Knowledge Acquisition
15
Not only do the number and kind of pictures accompanying the text have to be taken into account when trying to assess the scope of such investigations, but also the question of whether the pictures are absolutely necessary to fully comprehend the text, or whether they are simply useful. A rather sophisticated example for a case where the picture is an absolute precondition for text comprehension and hence retention, is supplied in the classic experiment by Bransford and Johnson (1972). The subjects were presented with the following text: "If the balloons popped, the sound would not be able to carry since everything would be too far away from the correct floor. A closed window would also prevent the sound from carrying since most buildings tend to be well insulated. Since the whole operation depends on a steady flow of electricity, a break in the middle of the wire would also cause problems. Of course the fellow could shout, but the human voice is not loud enough to carry that far. An additional problem is that a string could break on the instrument. Then there could be no accompaniment to the message. It is clear that the best situation would involve less distance. Then there would be fewer potential problems. With face to face contact, the least number of things could go wrong." After having read this, the subjects were asked to judge the extent to which they had understood the text and then to recall it. Three experimental groups were formed, whereby one group was given the text, and the other two received the text with a picture which either partially (Figure 4a) or fully (Figure 4b) supplied the appropriate context for the text.
uln
0
0 0 0
Figure 4: Pictures presented in the experiment by Bransford and Johnson (1972): (a) the partial context, (b) the appropriate context. Explanation in the text.
16
S. Molitor. S:P. Ballstaedt and H. Mandl
The result was that the group which was shown the appropriate picture (complete context) before (and not after) the text, understood and recalled. In the investigations mentioned above, the effect of pictures on the processing of texts is revealed under various aspects. Where pictures provide the required context for comprehension, they serve to activate the schemata relevant to comprehension. In this function, pictures may enhance comprehension and retention by influencing the type and depth of semantic processing (Sherman, 1976; Brody & Legenza, 1981; Bock & Hormann, 1974). Another effect relies on the sequencing of text and picture: the so-called sequencing effect is due to the fact that the information offered in the picture influences the analytical strategy used in processing the following text. Pictures then function like headings- i.e., they take on a function which Bock has characterized as "analysis criterion" (Bock, 1980, 1981). What he means is that the content of the picture determines how selectively the information in the following text is processed. Even when the text is comprehensible without a picture, pictures can support the comprehension process in many ways. An attempt to roughly categorize the influence pictures can have on text processing may be seen in the approach of Levin (1981), which is based on different picture functions. Levin differentiates between four main functions of pictures, each of which corresponds roughly to certain processes of knowledge acquisition. ReDresentation function: the picture overlaps the contents of a text - with reference to the text, the illustration repeats certain contents. In such a case the picture can be used as a redundant source of information to constantly check text comprehension by offering a, as it were, second opportunity for learning (Willows, 1980). Organization function: the picture provides an organized, coherent, reductive macrostructure of the text content. Visualizations, in which relations between key concepts of a text are depicted, offer more capacity for other aspects of text processing since the main relations of the text are quickly at hand in the picture. Intemretation function: the picture illustrates text contents which is difficult to understand - (e.g., by analogies, visual metaphors, etc.) The experiment by Bransford and Johnson (1972) can be viewed as an extreme example of this kind of function.The text is rendered more comprehensible by providing a concrete example (cf. Nahinsky & Oeschger. 1975; Pepper, 1981). Transformation function: the picture offers a mnemonically useful form of recoding, being a kind of visual mnemonic. This is the function pictures often have in the "keyword" or "hook" technique for learning vocabulary in foreign language instruction. In these examples, however, it is not a matter of recalling
Knowledge Acquisition
17
text contents, but single words, mnemonic pictures have been applied to larger text segments as well (Levin, 1982). The decoration function can also be added to this list. It solely serves to "beautify" the text and is not primarily of cognitive importance. In a metaanalysis of available studies, Levin, Anglin and Carney (1987) found that the first four types of pictures influence retention and comprehension in differing degrees, yet always in a positive way. Decorational pictures produced no positive effect, however, or even a slightly negative one. The Influence of Language on Picture Processing. Till now, there has been little research conducted on the influence of language on picture processing. The reason for this may be that -as in the aforementioned examples - with lengthier texts and numerous pictures it becomes increasingly difficult to assess the semantic overlappings or discrepancies between the two sources of information. Available studies on this subject therefore deal in a rather general manner with the influence of language on picture processing. The experimental material usually consists of single words or sentences and simple line drawings (Zimmer, 1983). Two kinds of investigations can be distinguished: on the one hand those that examine the influence of specific semantic or syntactic features of the text on the depth of picture analysis; on the other hand, studies where the text provides the necessary context for picture comprehension. The studies by Jorg (1978), and Jorg and Hormann (1978) can be regarded as examples for investigations, the results of which are apparently transferable to whole texts even though they used individual sentences only. According to these investigations, the semantic specificity of linguistic temis determines the accuracy of picture analysis. Before the presentation of each of the following sentences, the subjects were shown a picture representing a plaice, a breadknife, a chanterelle and a piece of
Figure 5: Example of a picture used by Jorg and Hormann (1978) to illustrate the nonspecific sentence, "The fish is next to the knife" and the specific sentence, "The plaice is next to the breadknife". Explanation in the text.
18
S. Molitor, S.-P. Ballstaedt and H. Mandl
The following two sentences are the examples: (a) The fish is next to the knife (general terms). (b) The plaice is next to the bread-knife. Five further variants of this picture were also presented, in which the objects differed in the degree of characteristic details so that in the end the specific terms were no longer applicable. Each subject received eight pictures of this kind, and each of the pictures preceded a general or a specific sentence wherein two of the four objects were named. The actual test consisted of a non-verbal task of recognition(i.e., the subjects were asked to' select the picture version of the objects, which they had previously seen from different picture versions available of that object). It appeared that the exactness in recognition depended on the degree of generality of the nouns in the preceding sentences. The specific nouns made exact recognition easier. This was not only the case for the two objects mentioned in the sentence, but also for the two objects in the same picture, that were not mentioned in the sentence. An explanation of this result might be that the specificity of linguistic terms has an effect on the general attitude of the recipient towards analyzing a picture in more or less detail, and hence it influences the kind of processing. Similar control of picture processing has been noted by Homby (1974) with reference to the syntactic structure of sentences (see also Engelkamp & Kumnacker, 1978). According to his findings, the order in which details of a picture are analyzed can be influenced by the syntactic sentence structure, at least for short picture presentations. There are also cases where a complete textual explanation is required to understand a picture: we are all familiar with picture puzzles or so-called "droodles", where one cannot see anything definite unless one gets a verbal clue, as in Figure 6. An experiment regarded as an analogue to the above-mentioned study by Bransford and Johnson (1972), was carried out by Bower, Karlin and Dueck (1975). They investigated the recall performance of pictures with and without a textual explanation:
Knowledge Acquisition
19
Figure 6: A skier at the top of a ski-jumping platform before jumping. Example of a "droodle", or a picture which is comprehensible only with additional verbal information (adapted from R. Price, "Der kleine Psychologe". Zurich: Diogenes, 1975, p. 105) The pictures used were picture puzzles, which were difficult to decipher without knowing the theme. The task of the subjects was to draw the pictures from memory. As was expected, recall performance was much better when the pictures had been accompanied by text. This can be explained by the fact that the subjects were not able to get the "meaning" of the pictures without seeing the text. The text served as a necessary context to activate the relevant schemata, in order for subjects to recognize the contents of the pictures and thus be able to reproduce them. These examples bring up one of the important questions in media psychology: how necessary are texts to avoid too superficial processing of a picture? At least with reference to films and television, pupils are of the opinion that these are "easy" media to learn from with more ease than from reading. This opinion, however, results in information being processed less intensively and pupils' learning being correspondingly worse (Salomon, 1984). Furthermore, the few empirical investigations of this topic have shown that television indeed is first and foremost used for entertainment, whereas education and information are predominantly ascribed to the print media (Katz, Blumler, & Gurevitch, 1974). Of course, the attitude towards a medium influences the extent to which the information offered is analyzed and hence the depth of processing.
20
S. Molitor, S:P. Ballstaedt and H. Mandl
According to an investigation carried out by Rigney (1978), depth of processing can also depend on the reading strategy suggested by the text-picture sequence. Rigney differentiates between "detached' and "embedded" strategies, or how much the text and picture design forces to take notice of both sources of information in order to fully understand the teaching material. In a detached strategy, the learner is explicitly referred from one medium to the other, either by means of corresponding instructions or by explicit clues in the text and the picture. In an embedded strategy, the text-picture combination is designed in such a way that to understand the entire meaning both sources of information have to be used almost automatically( i.e., without explicit cross-references) in order to understand the meaning. Furthermore, depth of processing also depends on the impact of schemadirected top-down processes in picture processing. According to Friedman (1979), top down processes predominate when, for instance, all of the elements in a picture are in conformity with a frame. In this case, a picture is first automatically encoded on the basis of a few noticeable features( i.e., a corresponding "frame" activates certain expectations with respect to the details in the picture). If all of the details "fit" well together, then they will hardly be looked at more closely, and will be processed only superficially. A discordant element in the picture, or between the picture and text, is, in contrast, able to increase the depth of processing. With certain types of pictures, certain cognitive processes are required for their comprehension. Corresponding investigations suggest that there are specific differences in cognitive prerequisites between the social classes and cultures with respect to their "visual literacy" (e.g., Hudson, 1980; for a critique of this position, see Cassidy & Knowlton, 1983). If we summarize the results discussed so far, we find a few basic functions, which a text, as well as a picture may assume with respect to each other: (1) Each medium can serve as an elaborative context for the other medium, and thus contribute to its comprehension. (2) Each medium can also help structure the other medium by repeating the essential content in a reduced form (e.g., a picture which illustrates the most important spatial relations described in the text, or a text describing the major stages of the migration by nations marked in a map of Europe). (3) With respect to sequential presentation, the medium processed in the first place can elicit an attitude, which determines the processing of the other medium. It can direct one's attention to certain aspects in the other medium, thereby generally influencing the depth of processing.
Knowledge Acquisition
21
Integrative Processing of Comdementarv Text-Picture Sequences. As the studies referred to above have shown, integrative processing is not necessarily attainable only by a semantic discrepancy between the text and the picture, because the learning effect of a text-picture combination also rests on the analytical strategy induced by the first medium presented. Due to sequencing effects, teaching material is not automatically perceived and processed in every aspect of its existing features, but the first mentioned information (whether in the text or in the picture) becomes a criterion for analysis, determining how selectively and intensively the following information will be processed. These sequencing effects also occur in a single medium (e.g., Isenhour, 1975; Bock, 1978; Schnotz, 1985). How should text-picture sequences be designed to stimulate integrative processing? According to the level of knowledge attained today,it is plausible to assume that interactive processing is most likely when the information in the text and the information in the picture are related in a way, which we will characterize in the following as "complementary". The term "complementarity" does not explicitly occur in the literature of available research: instead, the term "redundancy" appears, which characterizes the overlapping contents between the information in the picture and the text ( i.e., all the details of content presented in both media). Apart from the term "redundancy", the term "discrepancy" was also mentioned, referring to the content differences between the media. The term "complementarity" as defined here can only be referred to if both sources of information must be employed in order to reveal the entire meaning of the text-picture combination( i.e., textual information has gaps, which have to be filled by the information contained in the picture, and vice versa). The overlapping area between the information in the picture and text - the so-called area of redundancy - provides the necessary references between the two modalities. This kind of dovetailing between the textual and pictorial information determines to what extent complementarity will be perceived and processed by the learner. Accordingly, all differences between the text and picture, that have a necessary function for the construction of the entire significance of the teaching material may be considered as complementary. It is quite obvious that it is not merely a question of differences in contents: for example, one must take into account that it is not immaterial which modalities are employed in the formation of a concept. Complementarity of media must therefore be added to complementarity of contents. The learning objective is to provide a holistic impression of a concept - including term and visualization. Thus, complementarity of the media exists in the sense of dual-coding - even when
22
S.Molitor, S.-P. Ballstaedt and H. Mandl
complete redundancy of content between text and picture prevails - by the combination of two modalities in processing. It further depends on what functions formal characteristics of the text (or the picture) have for the reader within the comprehension process and whether these functions facilitate integrative processing from both sources of information. Here, it is a question of structural complementarity of the teaching material. Complementarity of structure refers to the fact that certain signals of text organization(e.g., headings, advance or post organizers, summaries) are taken over by pictures or visualizations (cf. Jones, Friedman, Tinzman 8z Cox, 1984). This is also the kind of complementarity we are dealing with whenever a picture or sentence is used in the sense of a title to activate a schema that is necessary to understand the complementary medium. Figure 7 is an example of how a specific content may be distributed between the two media - text and picture.
TOTAL INFORMATION
TEXT
INFORPIATION
PICTURE
INFORMATION
Figure 7: The white areas form the complementary area and the hatched areas show the redundancy area of the respective medium. The dotted areas include those parts of a text or pictures that are neither redundant nor complementary,
KnowledgeAcquisition
23
but which would be available as textual or pictorial embellishments for the content. We characterize them as "abundant". Further explanations are in the text. To design complementary text-picture relationships for specific didactic purposes, however, proves to be more difficult, than was the case with the most simple teaching material of the aforementioned experiments. Taking (as an examp1e)the dance used by bees as a form of communication, we have tried to construct a complementary relationship between the text and the picture (Figure 8). Proceeding from the text as the central medium, we can first of all read a description about the moves in the dance. The verbal description appears longwinded, but the diagram opposite illustrates it. Indeed, the information in the text and in the picture may be regarded as redundant with reference to the main statements, but complementarity of the media is present: the concept "a dance used by bees as a form of communication" is conveyed by verbal and visual means.
Figure 8: Example of complementarity between text and picture. Explanation in the text. (Adapted from A. Kuhn, Stuttgart: Thieme, 1969, 17th edition, p. 236/7).
24
S.Molitor, S. -P. Ballstaedt and H. Mandl
Bees convey information about feeding places in their communication dances. In this dance the insect moves in a straight line and vibrates the rear end of its body at the same time. At the end of the line, it dances in a semi-circle to either the right or the left, and then back to the starting point, ending up in the dancing line again. Contact bees feel the bee and note the information conveyed in the dance. The direction of the feeding place is indicated by the deviation of the dancing line from the vertical, of which the sun serves as a point of reference. If the direction of the feeding place is at, e.g., 60 degrees to the left of the sun at that point in time, - then the bee dances 60 degrees to the left of the vertical. The distance is conveyed by the speed of the dance: the further the distance, the slower the dance is. In contrast to human dancing, the choreography is determined by the bees' instinct. The following pieces of information are complementary because they occur only in one medium: the illustration shows how the contact bees extract the information (they touch the abdomen of the dancing bee with their feelers). The text following the description of the choreography exemplifies which features of the dance convey what message about the food source. An additional schema for the picture clarifies the spatial transposition. The sentence referring to the bee's dance being of instinctive origin can be interpreted as abundant information in this context. In this simple example, the text and the picture together are necessary to comprehend the dance. The result of the reviewed investigations, however, was that, even when there is successful complementarity between the text and picture, this does not guarantee that both sources of information will be integratively processed. The task facing future research is to trace the design features, learning strategies, and the conditions of the learning environment which could assist the learner in both perceiving and using the complementarity of teaching material. One basic condition for integrative processing is to refer explicitly from one medium to the other - usually from the text as the leading medium to the illustration. In this case, in the text as well as in the illustration the corresponding concepts should be labeled with the same terms. The illustration should be placed in such a way that an interactive decoding of the two media is not disturbed by turning the pages or by searching.
Knowledge Acquisirion
25
Text-picture Design From the beginning of investigations of pictures in text, an effort was directed toward the optimal design of school-books and textbooks.,At that time, it was a matter of technical effect-oriented research, seeking principles for media design to produce successful learning. The problems of knowledge processing and of mental representation were excluded. After cognitive psychology has yielded research results with respect to these questions, we now possess some basic notions applicable by practitioners to ground their decisions scientifically. At the same time, application-oriented research has begun. It relies on basic notions of cognitive psychology and deduces relevant variables and hypotheses from it. We will now sketch this development. Results from Effect-oriented Research. The first investigations were exclusively concerned with the effect of pictures on the retention and comprehension of information in texts. Since they only measured the effect of various forms of presentation on, learning, they explicitly or implicitly assumed a behavioristic approach. Accordingly, the merit of effect-oriented research is of differentiating on the part of the "stimulus"i.e., - mainly picture description). Various typologies of pictures have been developed, which take the technical design into consideration (e.g., photography, drawing, graphics), as well as the type of representation (e.g., illustration, visualization - Issing, 1983). The experiments typically compare two experimental groups, "text alone" vs. "text plus picture", in which mostly children are presented simple narrative texts to learn. An indicator of successful learning is the verbal reproduction of factual knowledge (Levin & Lesgold, 1978). Levie and Lentz (1982) have meta-analyzed fifty-five of these kinds of experiments with questionings of practical relevance: how does redundance between text and picture (i.e., the repetition of the contents of one medium within the other medium) affect learning success? The text was the leading medium, and recall of verbal information alone was tested. Two kinds of verbal information may be distinguished within a text-picture combination: the information that only occurs in the text, and that which occurs in the text and in the picture. The meta-analysis on the effects of pictures resulted in positive learning from pictures only for redundant information in texts and pictures. Pictures had no influence on the recall of information presented in the text alone. This result is interesting, yet crude, since the relationship between the nonredundant information in the text and the picture can take on various forms. An extensive and sophisticated series of investigations has been camed out by Dwyer and his co-workers (for a summary, see Dwyer, 1978). Although these
26
S. Molitor, S. -P. Ballstaedt and H. Mandl
investigations still belong to effect-oriented research and the search for technological knowledge, they go beyond a simple S-R schema. They take the prior knowledge of the learner into consideration and employ not only verbal, but also visual tests of retention. In the series of studies, a programmed textbook on the "Anatomy and Function of the Heart" was used as the experimental material. The text was equipped with various picture variants, which differed in their degree of reality (simple line-drawings of the human heart, shaded line-drawings, photos of a heart model, photos of anatomic sections of real human hearts), and in colour (blackwhite or colour). In addition to the normal verbal comprehension test, drawing tests and identification tests were also employed in these studies (e.g., making a sketch of the heart, or labeling a shaded drawing). From the results it is worth noting that the illustrations - especially the shaded line-drawings - were helpful especially for students with low prior knowledge. This result was particularly striking in the drawing test, which may be explained by the fact that the shaded drawing, in opposition to the more realistic photo, presents the relevant information for concept formation in a more concise and easily perceivable form. This may have been because in the drawing test the main emphasis was on spatial information, which is better conveyed by pictures than by texts. On the other hand, in the verbal comprehension test information that was less well expressed in the picture was orally tested for, so that the description in the text alone had the same effect. As an important result of this series of investigations it should be noted, that the didactic effectiveness (or ineffectiveness) of pictures depends not only on the design of the pictures, but also on features of the learner, such as prior knowledge. This leads us to an approach, which was expressly concerned with the interaction between features of the media and characteristics of the learner, and which at one time had appeared as a promising possibility to put media research on a firm foundation after it had been so disappointing: the so-called AptitudeTreatment Interaction (ATI) or Trait-Treatment Interaction (?TI) research. This approach assumes that the learning success of a person depends on the interaction between the cognitive and motivational features of the individual on the one hand, and the variables of the respective medium on the other hand (Snow & Salomon, 1968). The following questions are typically posed in the framework of the "TIapproach: what media (e.g. slides, silent movie, or text) or attributes of the media (e.g. colour, movement, or sequencing) produce positive learning results for learners with what features (age, prior knowledge, and available strategies) with respect to which tasks? (see Clark, 1975). There are unfortunately no studies in
Knowledge Acquisition
27
which the media "text", "picture", and "text-picture combinations" have all been compared - only a few comparisons have been made between words, pictures, and word-picture combinations. In this context, however, the search for verbalizers and visualizers deserves to be mentioned on the part of learner characteristics (i.e., persons who learn easier with verbal or pictorial aids). These types of learners or cognitive styles can be postulated on the basis of,for example, the theory of dual-coding, where both a verbal and imaginal system are presumed (Paivio, 1971). However, one could not consistently trace both types of learner, since the learners employ different learning strategies according to task orientation, prior knowledge, and other learning conditions, and use information differently (Gustafson, 1976). This example helps to explain why the yield of replicable interaction effects turned out to be quite poor: the approach only registers statistical interactions, but it does not take the actual processing into account, which includes properties of the media as well as learning strategies (Heidt, 1976; Di Vesta, 1975; Clark, 1975). The first attempt to try to overcome the gap between independent and dependent variables on the basis of cognitive processes was made by Salomon (1970). In summary, it can be concluded that the effect-oriented research is generally unsatisfactory, revealing a few serious methodological and theoretical deficiencies (Stone et al., 1981; Brody, 1981; Schallert, 1980). Indeed, various types of texts and pictures have been examined, but both forms of media have often been described either poorly or not at all with respect to their important cognitive aspects (for example, their complexity, organization, or sequencing). Worse still, systematic control was lacking with regard to the contents and formal relationships between text and picture. Learning success is assessed by favouring mainly verbal information. Special learning effects of pictures cannot be demonstrated that way (the experiments by Dwyer are an exception). The experiments are usually formulated into ad hoc questions, and are not grounded in a cognitive processing theory. The results of applied research are therefore only helpful in practice to a limited degree, although much effort has been invested. The media designer is confronted with a patchwork of findings, which hardly provide recommendations for the respective design problem. This conclusion shows again that the rapid search for technological solutions in psychology is mostly condemned to failure. Research Based on Cognitive Psvchology. More proceeds for media design can be expected from approaches which build on the theories and findings of cognitive psychology. Of course, precise remedies should not be expected concerning all domains of knowledge, types of texts and pictures, learning
28
S. Molitor, S.-P. Ballstaedt and H. Mandl
objectives and learners. But basic research can equip the practitioner with the background knowledge required, making it possible for himiher to come to decisions more reasonably. Until now, problems of instructional design have been decided upon mainly according to technical, aesthetic, or purely intuitive points of view. In the future, didactic decisions should be made according to cognitive and learning psychological conditions. The design of text-picture combinations presents a complex problem, as far as the text, the picture, and their relationship to each other are concerned. The numerous investigations on text comprehension throughout the last decade have been repeatedly revised with respect to text design and practical cues (Kolers, Wrolstad, & Bouma, 1979; Ballstaedt et al., 1981; Groeben, 1982; Jonassen, 1982; Duffy & Waller, 1985). In contrast, the more recent approaches to picture comprehension have not been taken into account by picture designers (Hagen, 1980; Hannemann, 1983; D'Ydewalle & Van Rensbergen, this volume). This is the case, for instance, with the numerous findings from eye movement research in picture processing (Issing et al., 1985). Instructions for picture design usually refer exclusively to gestalt-psychological principles. There are only a few instructions for the design of text-picture combinations, referring to basic notions of cognitive psychology. Hence, Bock suggests that the media practitioner take semantic discrepancy between the text and picture into account, as well as the effects of sequencing to influence processing. Habermann (1985) provides results for audiovisual media concerning specific processes of knowledge acquisition by seeing and hearing, and pleads for a complementary design of text and picture. Cognitive research on text-picture design is still in its initial stages. As examples of the progress that is being made, however, we would like to present the approaches by Levin and the Strittmatter group. Levin has developed a functional-cognitive approach (Levin, 1981, 1982). He distinguishes between four main functions of a picture with reference to the text, which we listed earlier on: the representational, organizational, interpretational, and transformational functions. Various processing strategies are assigned to the four cognitive functions. For example: a strategy for retaining the macro-structure of a text can be supported by the organization function of the picture. Levin works with children and simple pieces of prose; his considerations, however, are transferable to expository texts for adult learners. Concerning questions of design, this approach means that it is not the kind of picture or certain features of pictures which are crucial for learning, but their didactic function which either supports or hinders them. Hence, the design of a picture should be considered in connection with the functional perspective.
Knowledge Acquisition
29
The work done by Strittmatter and Seel also constitutes a promising approach: they try to elucidate the pedagogical term "Anschauung" (contemplation) in terms of the concept of mental models (Strittmatter & Seel, 1984; Dorr, Seel, & Strittmatter, 1986). Their main concern is to analyze the relationship between the presentation of information by media and its representation in mental models. According to their working hypothesis, different media influence mental representations, each in a specific way (Seel, 1986). In a previous section, we described an experiment performed by this research group, which was supposed to examine (somewhat ambitiously) "the acquisition of geographical knowledge". This experiment basically points in the direction of applied cognitive research examining the construction of mental models in ecologically valid learning situations controlled by illustrated teaching material(e.g., from biology, physics, and geography. In consideration of the cognitive psychological background, the picture designer should not be merely an assistant, but a co-worker with the author of the text.
Authors Note A German version of this contribution will be published in: J. Groebel & P. Winterhoff-Spurk (Eds.), Empirische Medienpsychologie. MilnchenTWeinheim: Psychologie Verlags Union, Beltz, (in press).
References Ballstaedt, S.-P., Mandl, H., Molitor, S., Schnotz, W., & Tergan, S . - 0 . (1986). Text und Bild beim Aufbau eines mentalen Modells. Eine empirische Untersuchung. Fernstudium aktuell, 8 (314), 12-13. Bock, M. (1978).Uberschriftspezifische Selektionsprozesse bei der Textverarbeitung. Archiv fur Psychologie, 131, 77-93. Bock, M. (1980). Some Effects of Titles on Building and Recalling Text Structures. Discourse Processes,.3, 301-311. Bock, M. (1981).Der EinfluR von Uberschriften auf die (Re)-Produktion von Textstrukturen.Grund1agenstudien aus Kybernetik und Geistenswissenschaf, 22, 51-65.
30
S. Molitor, S.-P. Ballstaedt and H. Mandl
Bock, M. (1983). Zur Reprasentation bildlicher und sprachlicher Informationen im Langzeitgedlchtnis - Strukturen und Prozesse. In L.J. Issing, & J. Hannemann (Eds.), Lemen mit Bildem. AV-Forschung, 25, 61-94. Bock, M., & Hdnnann, H. (1974). Der Einfld3 von Bildem auf das Behalten von Sltzen. Psychologische Forschung, 36, 343-357. Bock, M., & Milz, B. (1977). Pictorial Context and the Recall of Pronoun Sentences. Psychological Research, 39,203-220. Bower, G.H., Karlin, M. B., & Dueck, A. (1975). Comprehension and Memory for Pictures. Memory and Cognition, 3, 216-220. Bransford, J.D., & Johnson, M.K.(1972). Contextual Prerequisites for Understanding: Some Investigations for Comprehension and Recall. Journal of Verbal Learning and Verbal Behavior, 11,717-726. Britton, B.K., & Black, J.B. (1985). Understanding Expository Text. A Theoretical and Practical Handbook for Analysing Explanatory Text. Hillsdale, N.J.: Erlbaum.. Brody, P.J. (1982). Affecting Instructional Textbooks Through Pictures. In D.M. Jonassen (Ed.), The Technology of Text. Principles f o r Structuring, Designing, and Displaying Text Englewood Cliffs, N.J.: Educational Technology Publications . Brody, P.J., & Legenza, A. (1981). Can Pictorial Attributes Serve Mathemagenic Functions? Educational Communication and Technology Journal, 29, 93100. Bruner, J.S., & Olson, D.R. (1973). Learning Through Experience and Learning Through Media. Prospects, 1, 20-38. Bruner, J.S., Olver, R.R., & Greenfield, P.M. (1966). Studies in Cognitive Growth. New York: Wiley. Cassidy, M.F., & Knowlton, J.Q. (1983). Visual Literacy: A Failed Metaphor? Educational Communication and Technology Journal, 31,67-90. Chase, W.G., & Clark, H.H. (1972). Mental Operations in the Comparison of Sentences and Pictures. In L.W. Gregg (Ed.), Cognition in Learning and Memory . New York: Academic Press. Clark, R.E. (1975). Constructing a Taxonomy of Media Attributes for Research Purposes. AV Communication Review, 23, 197-215. Comenius, J.A. (0.J.). Lateinisches Textbuch. Faksimile der Niimberger Edition 1658.0snabriick/Belm: Zeller. De Kleer, J. & Brown, J.S. (1983). Assumptions and ambiguities in mechanistic mental models. In D. Gentner & A.L. Stevens (Eds.), Mental Models. Hillsdale: Erlbaum. Denis, M. (1982). Images and Semantic Representation. In J.F. Le Ny, & W. Kintsch (Eds.), Language and Comprehension Amsterdam: North Holland. Di Vesta, F.J. (1975). Trait-Treatment Interactions, Cognitive Processes, and Research on Communication Media. Audiovisual Communication Review, 23, 185-197. Ddrr, G., Seel, N.M., & Strittmatter, P. (1986). Mentale Modelle: Alter Wein in neuen Schlauchen? Mediendidaktische Anmerkungen. Unterrichtswissenschaft, 2, 168-189.
Knowledge Acquisition
31
Downs, R.M., & Stea, D. (1982). Kognitive Karten. Die Welt in unseren Kopfen. New York: Harper & Row. Duffy, Th.M., & Waller, R. (1985). Designing Usable Texts. London: Academic Press. Durso, F.T., & Johnson, M.K. (1980). The Effects of Orienting Tasks on Recognition, Recall, and Modality Confusion of Pictures and Words. Journal of Verbal Learning and Verbal Behavior, 19, 416-429. Dwyer, F.M. (1978). Strategies for Improving Visual Learning. A Handbook for the Effective Selection, Design and Use of Visualized Materials. State College, Pennsylvania: Learning Services. Engelkamp, J., & Krumnacker, H. (1978). The Effect of Cleft Sentence Structures on Attention. Psychological Research, 40,27-36. Farah, M.J. (1985). Psychophysical Evidence for a Shared Representational Medium for Mental Images and Percepts. Journal of Experimental Psychology, 114,91-103. Farah, M.J. (1988). Knowledge from Text and Pictures: A Neuropsychological Perspective. In H. Mandl & J.R. Levin (Eds.), Knowledge Acquisition from Text and Pictures. Amsterdam: North Holland. Frederiksen, C.M. (1975). Representing Logical and Semantic Structure of Knowledge Acquired from Discourse. Cognitive Psychology, 7, 37 1-458. Friedman, A. (1979). Framing Pictures: The role of Knowledge in Automatized Encoding and Memory for Gist. Journal of Experimental Psychology, 108, 316-355. Foldes-Papp, K. (1966). Vom Felsbild zum Alphabet. Stuttgart: Belser. Gentner, D., & Stevens, A.L. (1983). Mental Models. Hillsdale, N.J.: Erlbaum. Geschwind, N. (1972). Language and the Brain. ScientSfic American, 226, 76-83. Gibson, J.J. (1982). Wahrnehmung und Umwelt. Miinchen: Urban & Schwarzenberg. Groeben, N. (1982). Leserpsychologie: Textverstbdnis - Textverstlndlichkeit. Miinster: Aschendorff. Gustafson, J.-E. (1976). Verbal and Figural Aptitudes in Relation to Instrumental Methods. Goteborg: Acta Universitatis Gothoburgensis. Habel, C. (1987). Darstellung raumlichen Wissens. In G. Rahmstorf (Ed.), Wissensreprasentation in Expertensystemen. IBM. Habermann, P. (1985). Informationsverarbeitung beim Fernsehen. A V Forschung, 26, 5-46. Hagen, M. (1980). The Perception of Pictures. (Vol. 2.) New York: Academic Press. Hannemann, J. (1983). Uber das Lesen von Bildern. Wahmehmungs- und gedlchtnispsychologische Aspekte der Bilderfassung. In L.J. Issing, & J. Hannemann (Eds.), Lemen mit Bildem. AV-Forschung, 25,40-60. Hebb, D.O. (1968). Concerning Imagery. Psychological Review, 75,466-477. Heidt, E.U. (1976). Medien und Lernprozesse. Das Problem einer Medienklassifikation im Zusammenhang didaktischer Modelle und lernpsychologischer Forschung. WeinheimBasel: Beltz.
32
S.JMolitor.S.-P. Ballstaedt and H.Mandl
Hewes, G.W. (1978). Visual Learning, Thinking, and Communication in Human Biosocial Evolution, In B.S. Randhawa, & W.E. Coffman (Eds)., Visual Learning, Thinking, and Communication New York: Academic Press. Heydenreich, L.H., Dibner, B. & Reti, L. (1981). Leonard0 der Erfinder. StuttgartEurich. Belser. Hornby, P.A. (1974). Surface Structure and Presupposition. Journal of Verbal Learning and Verbal Behavior, 13, 530-538. Hudson, W. (1960). Pictorial Depth Perception in Subcultural Groups in Africa. Journal of Social Psychology, 52, 183-208. Isenhour, J.P. (1975). The Effects of Context and Order in Film Editing. Audiovisual Communication Review, 23, 69-80. Issing, L.J. (1983). Bilder als didaktische Medien. In L.J. Issing, & J. Hannemann (Eds.), Lemen mit Bildem. AV-Forschung, 25, 9-39. Issing, L.J., Beyland, U., Haack ,J.M., & Mickasch, H.D. (1985). Blickbewegungsforschung und Bildverarbeitung. Berlin: Arbeitsbericht der Arbeitsgruppe Medienpsychologiehledienpiidagogik. Janl3en-Holldiek, I. (1984). Bilder und Vorstellungsbilder im Fremdsprachenuntemcht. Unterrichtswissenschaft,12.48-67. Jorg, S. (1 978). Der Einfld sprachlicher Bezeichnungen auf das Wiedererkennen von Bildem. Wien: Huber. Jorg, S., & HBrmann, H. (1978). The Influence of General and Specific Verbal Labels on the Recognition of Labeled and Unlabeled Parts of Pictures. Journal of Verbal Learning and Verbal Behavior, 17, 445454. Johnson-Laird, P.N. (1983). Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge: University Press. Jonassen, D.M. (1982). The Technology of Text: Principles for Structuring, Designing, and Displaying Text. Englewood Cliffs, N.J.: Educational Technology Publications. Jones, B.F., Friedman, L.B., Tinzman, M., & Cox, B.E. (1984). Considerate Graphics Texts. In B.F. Jones et al. (Eds.), Content-Driven Comprehension Instruction and Assessment: A Model for Army Training Literature. Technical Report. Alexandria, VA: Army Research Institute. Katz, E., Blumler, J.G., & Gurevitch, M. (1974). Utilization of Mass Communication by the Individual. In J.G. Blumler & E. Katz (Eds.), The Uses of Mass Communication: Current Perspectives on Gratifications Research Beverly Hills, CA: Sage. Kieras, D. (1978). Beyond Pictures and Words: Alternative InformationProcessing Models for Imagery Effects in Verbal Memory. Psychological Bulletin, 85, 532-554. De Kleer, J., Brown, J.S. (1983). Assumptions and Ambiguities in Mechanistic Mental Models. In D. Gentner, & A.L. Stevens (Eds.), Mental Models Hillsdale, N.J.: Erlbaum. Klix, F. (1 980). Erwachendes Denken. Eine Entwicklungsgeschichte der menschlichen Intelligenz. Berlin: VEB Deutscher Verlag der Wissenschaften.
Knowledge Acquisition
33
Kolers, P.A. (1973). Some Modes of Representation. In P. Pliner, L. Krames, & T. Alloway (Eds.), Communication and Affect: Language and Thought. New York/London. Kolers, P.A. (1977). Reading Pictures and Reading Text. In D. Perkins, & B. Leondar (Eds.), The Arts and Cognition . Baltimore: John Hopkins University Press. Kolers, P.A., & Brison, S.J. (1984). Commentary: On Pictures, Words, and their Mental Representation. Journal of Verbal Learning and Verbal Behavior, 23, 105-113. Kolers, P.A., Wrolstad, M.E., & Bouma, H. (1979). Processing of Visible Language. 2 Vol. New York/London: Plenum Press. Kosslyn, S.M. (1981). The Medium and the Message in Mental Imagery: A Theory. Psychological Review, 88,46-66. Kiihn, A. (1969). GrundriB der allgemeinen Zoologie. Stuttgari: Thieme. Levie, W.H., & Lentz, R. (1982). Effects of Text Illustrations: A Review of Research. Educational Communication and Technology Journal, 30, 195232. Levin, J.R. (1981). On Functions of Pictures in Prose. In F.J. Pirozzolo & M.C. Wittrock (Eds.), Neuropsychological and Cognitive Processes in Reading . New York: Academic Press. Levin, J.R. (1982). Pictures as Prose Learning Devices. In A. Flammer & W. Kintsch (Eds.), Discourse Processing Amsterdam: North-Holland Publishing Company. Levin, J.R., & Lesgold, A.M. (1978). On Pictures in Prose. Education, Communication and Technology, 26, 233-243. Levin, J.R., Anglin, G.J., & Carney, R.N. (1987). On Empirically Validating Functions of Pictures in Prose. In D.M. Willows & H.A. Houghton (Eds.), The Psychology of Illustration. Vol. I : Basic Research . New York/Berlin: Springer-Verlag . Nahinsky, I.D., & Oeschger, D.E. (1975). The Influence of Specific Stimulus Information on the Concept Learning Process. Journal of Experimental Psychology: Human Learning and Memory, I , 660-670. Nelson, D.L. (1970). Remembering Pictures and Words: Appearance, Significance, and Name. In L.S. Cermak & FM Craik (Eds.), Levels of Processing in Human Memory . Hillsdale, N.J.: Erlbaum. Paivio, A. (1971). Imagery and Verbal Processes. New York: Holt, Rinehart and Winston. Paivio, A. (1977). Images, Propositions, and Knowledge. In J.M. Nicholas (Ed.), Images, Perception, and Knowledge Dordrecht: D. Reidel Publishing Company. Paivio, A. (1983). The Empirical Case for Dual Coding. In J.C. Yuille (Ed.), Imagery, Memory, and Cognition. Essays in Honor of Allan Paivio Hillsdale, N.J.: Erlbaum. Pepper, J. (1981). Following Students' Suggestions for Rewriting a Computer Programming Textbook. American Educational Research Journal, 18, 259269.
34
S. Molitor, S.-P. Ballstaedt and H. Mandl
Piaget, J., & Inhelder, B. (1966). L'image mentale chez l'enfant: Etude sur le dkveloppement de reprksentations imagkes. Paris: Presses Universitaires de France. Price, R. (1975). Der Kleine Psycho1oge.Ziirich: Diogenes. Pylyshyn, Z.W. (1981). The Imagery Debate: Analogue Media versus Tacit Knowledge. Psychological Review, 88, 16-45 Rigney, J.W. (1987). Learning Strategies: A Theoretical Perspectives. In H.F. O'Neill (Ed.), Learning Strategies New York: Academic Press. Roitblat, H.L., Bever, T.G., & Terrace, H.S. (1984). Animal Cognition. Hillsdale, N.J.: Erlbaum. Salomon, G. (1970). What Does it Do to Johnny? Viewpoints. Bulletin of the School of Education. Indiana Universiry, 5 , 33-62. Salomon, G. (1979): Interaction of Media, Cognition and Learning. San Francisco: Jossey-Bass. Salomon, G. (1984). Television is "easy" and Print is "though": The Differential Investment of Mental Effort in Learning as a Function of Perceptions and Attribution. Journal of Educational Psychology, 76, 647-658. Salomon, G., & Snow, R.E. (1968). The Specification of Film Attributes for Psychological and Educational Research Purposes. Audiovisual Communication Review, 3, 225-244. Schallert, D.L. (1980). The Role of Illustrations in Reading Comprehension. In R.J. Spiro, B.C. Bruce, & W.F. Brewer (Eds.), Theoretical Issues in Reading Comprehension . Hillsdale, N.J: Erlbaum. Schnotz, W. (1985). Selectivity in Drawing Inferences. In G. Rickheit & H. Strohner (Eds.), Inferences in Text Processing . North-Holland: Elsevier Science Publishers B.V. Seel, N.M. (1986). Wissenserwerb durch Medien und "mentale Modelle". Unterrichtswissenschaft,14 (4), 384-401. Seel, N.M., & Strittmatter, P. (1984). Strategien m m Erwerb geographischen Wissens und bildhafter rtiumlicher Vorstellungen. Unterrichtswissenschaft, 12 ( l ) ,32-47. Sherman, J.L. (1976). Contextual Information and Prose Comprehension. Journal of Reading Behavior, 8, 369-379. Snodgrass, J.G. (1984). Concepts and their Surface Representations. Journal of Verbal Learning and Verbal Behavior, 23, 3-22. Snow, R.E., & Salomon, G. (1968). Aptitudes and Instructional Media. Audiovisual Communication Review, 16, 341-356. Steiner, G. (1988). Analoge Reprasentation. In H. Mandl, & H. Spada (Eds.), Wissenspsychologie . Miinchefleinheim: Psychologie Verlags Union. Stone, D.E., & Glock, M.D. (1981). How do Young Adults Read Directions with and without Pictures? Journal of Educational Psychology, 73,4 19-426. Stone, D.E., Pine, C.K., Bieger, G.R., & Glock, M.D. (1981). Methodological Issues in Research on Reading Text with Illustrations. Department of Education. College of Agriculture and Life Sciences. Cornell University: Technical Report No. 2.
Knowledge Acquisition
35
Strittmatter, P., & Seel, N.M. (1984). Exteme und interne Medien. Konzepte der Medienforschung. Unterrichtswissenschaft, I2,2-17 Tack, W.H. (1988). Ziele und Methoden der Wissensreprasentation. In H. Mandl & H. Spada (Eds.), Wissenspsychologie Munchen/Weinheim: Psychologie Verlags Union. Willows, D.M. (1980). Reading Comprehension of Illustrated and Non-Illustrated Aspects of Text. Paper Presented at the annual meeting of the American Educational Research Association, Boston. Willows, D.M., & Houghton, H.A. (1987). The Psychology of Illustration. Vol. 1: Basic Research. New York/Berlin: Springer Willows, D.M., & Houghton, H.A. (Eds.)(1987). The Psychology of Illustration. Vol. 2: Instructional Issues. New YorkDerlin: Springer Wippich, W. (1984). Lehrbuch der angewandten Gedachtnispsychologie. Bd. 1. Stuttgart: Kohlhammer. Zimmer, H.D. (1983). Sprache und Bildwahmehmung. Die Reprasentation sprachlicher und visueller Informationen und deren Interaktion in der Wahrnehmung. FrankfurtNain: Haag & Herchen.
