Jl. of Educational Multimedia and Hypermedia (2004) 13(2), 163-183
Constructing Knowledge from Dialog in an Intelligent Tutoring System: Interactive Learning, Vicarious Learning, and Pedagogical Agents
SCOTTY CRAIG, DAVID M. DRISCOLL, AND BARRY GHOLSON University of Memphis USA
[email protected] [email protected]
College students either interacted directly with an intelligent tutoring system, called AutoTutor , by contributing to mixed initiative dialog, or they simply observed, as vicarious learners, previously recorded interactive sessions. The mean pretest to posttest effect size (Cohen’s d) across two studies was 1.86 in the interactive conditions and 1.12 in standard vicarious conditions. In Experiment 1, redundant onscreen printed text produced an effect size of 0.43, but the difference was not significant. In addition, the image of a talking head presenting AutoTutor ’s contributions to the dialog while displaying facial expressions, gestures, and gaze did not produce learning gains beyond those produced by the voice alone. In Experiment 2, the effect size was 0.71 when interactive tutoring was contrasted with the standard vicarious condition, but only 0.38 when compared to a collaborative vicarious condition.
Recent advances in educational technology, particularly computerbased courses (Anderson, Corbett, Koedinger, & Pelletier, 1995; Mayer, 2001), and distance learning (Barker & Dickson, 1996; Bourdeau & Bates, 1997; Moore & Kearsley, 1996; Roblyer & Edwards, 2000; Renwick, 1996; Spector, 2001), have created situations where learners are more and more likely to find themselves trying to gain knowledge in settings in which they
164
Craig, Driscoll, and Gholson
are observers (Cox, Mckendree, Tobin, Lee, & Mayes, 1999; Fox Tree, 1999; Schober & Clark, 1989), rather than active participants. These advances have created a need for further empirical understanding of the conditions that promote learning among relatively isolated observers (e.g., Lee, Dineen, & McKendree, 1998; McKendree, Stenning, Mayes, Lee, & Cox, 1998). Little is currently known about how much is acquired by observers when compared to active participants in multimedia educational environments that are designed to promote learning (Mayer, 2001; Sweller, 1999; Wittrock, 1990). To address this issue, the present research was designed, in part, to contrast the relative learning gains of observers (i.e., vicarious learners) when compared to active participants in the learning process (Bandura, 1977; Lee et al., 1998; McKendree et al., 1998). Historically, the term vicarious learning was frequently used synonymously with observational learning, social learning, or modeling (Bandura, 1962; Rosenthal & Zimmerman, 1978). According to this perspective, by simply observing activities carried out by others, learners can master those activities without overt practice or direct incentives (Rosenthal & Zimmerman, 1978, p. xi). In two experiments, learners either interacted directly with an intelligent tutoring system (ITS), called AutoTutor (Graesser, Wiemer-Hastings, Wiemer-Hastings, Kreuz, & TRG, 1999), or they simply observed an interactive sessions. Experiment 2 also included a collaborative vicarious-learning condition. CONSTRUCTIVISM AND INTERACTIVE LEARNING According to constructivism, learners actively create meaning and knowledge by interacting with people and other objects. Rather than simply delivering information, learning environments should stimulate the learner to actively construct knowledge and provide feedback on the constructions. In the context of an ITS, knowledge construction is viewed as a sense-making activity in which the learner attempts to build a coherent representation of the tutorial contents and integrate it with existing knowledge (Graesser et al., 1999; Wittrock, 1974, 1990). Research supporting the epistemological stance of constructivist approaches to learning (Biggs, 1996; Bransford, Goldman, & Vye, 1991; Brown, 1988; Chi, deLeeuw, Chiu, & LaVancher, 1994; Derry, 1996; Mayer, 1997; Moshman, 1982;Ti Palincsar & Brown, 1984; Papert, 1980; Piaget, 1952; Pressley & Wharton-McDonald, 1997; Rogoff, 1990; VanLehn, Jones, & Chi, 1992; Vygotsky, 1978) has a long history in psychology and education. In fact, the research base for construc-
Constructing Knowledge from Dialog in an Intelligent Tutoring System
165
tivism is so compelling that it has shaped the standards for curriculum and instruction in the USA during the last decade, for example, the Standards for the English Language Arts (National Council of Teachers of English [NCTE], 1996), the Curriculum and Evaluation Standards for School Mathematics (National Council of Teachers of Mathematics [NCTM], 1991), and the National Science Education Standards (National Research Council [NRC], 1996). Moreno, Mayer, Spires, and Lester (2001) recently reported research supporting a basic assumption of constructivism, the need for active participation in the learning process. They used an ITS in which an animated pedagogical agent, Herman the bug, teaches students causal relations between the structures and functions of plants. Herman guides learners through a visually-rich, design-a-plant interface, explaining which roots, stems, and leaves survive best in eight different environments (Lester, Converse, Kahler, Barlow, Stone & Bhogal, 1997; Lester, Stone, & Stelling, 1999; Lester, Voerman, Towns, & Callaway, 1999). In a typical example scenario (Moreno et al. 2001, p. 181), Herman first elaborated on features of the environment pictured on the monitor (e.g., sunlight, rainfall, water table), emphasizing how they might potentially affect various plant parts. He then asks the learner to click on one of eight possible plant stems, choosing one that would thrive in that environment. Herman then comments on the choice, saying whether it is correct or incorrect. If the choice is incorrect, he provides corrective feedback by explaining why it is a bad choice, and asks the student to choose another plant stem. If the second choice is wrong, Herman tells them which stem to choose. Navigating through the eight environments requires between about 24 and 28 minutes (Moreno et al., 2001, p. 201). To explore the effects of interactivity, Moreno et al., (2001, Experiment 3), compared two conditions. In one, students actively interacted by designing a plant for each of the eight environments while listening to Herman’s explanations. In the comparison group, students listened to the same eight explanations from Herman, but did not design a plant in any of the environments. Students who interacted with the environments remembered more of the information that was presented by Herman and solved more transfer problems than those who just listened to the explanations. Moreno et al. concluded that interacting with the environments led learners to process the materials to a deeper level (Anderson & Pearson, 1984; Bloom, 1956; Elliott, McGregor, & Gable, 1999). They also pointed out that because the interactive group received corrective feedback while the comparison group did not, they could not rule out the possibility that this difference contributed to their findings (Moreno et al., p. 200).
166
Craig, Driscoll, and Gholson
TUTORING AND AUTOTUTOR There is substantial evidence that one-on-one human tutoring is a powerful method of promoting knowledge construction. When compared to classroom instruction, the gains from tutoring are generally in the range of 0.4 to 2.0 standard deviation units, depending on the expertise of the tutor (Bloom, 1984; Cohen, Kulik, & Kulik, 1982; Corbett, 2001). Cohen et al. performed a meta-analysis on a large sample of studies that compared human tutoring to various classroom controls. Most of the tutors in the studies were untrained in tutoring skills and had only moderate domain knowledge. They were peer tutors, cross-age tutors, or paraprofessionals; rarely accomplished professionals. Still, the average learning gain was 0.4 standard deviation units when compared to various control conditions, such as reading a text or engaging in standard classroom activities. This 0.4 gain translates into about half a letter grade. Bloom (1984) has reviewed evidence showing that, when compared to classroom controls, accomplished human tutors produce gains of about 2.0 standard deviation units, or about two letter grades. AutoTutor , like the design-a-plant environment, includes an animated pedagogical agent, but differs from it in at least one important respect: in tutorial sessions involving AutoTutor , nearly all the learning that takes place results from dialog between the learner and the ITS. This is in contrast to the design-a-plant environment (Lester et al., 1997, 1999; Moreno et al. 2001), in which the learner listens to Darwinian-type explanations and chooses pictorial information to construct plant parts that meet various environmental constraints. AutoTutor implements the tutoring strategies of peer tutors and paraprofessionals (Graesser & Person, 1994). These strategies mostly involve filling in missing pieces of information in expected answers and attempting to fix any bugs and misconceptions that are detected (Graesser & Person, 1994; Graesser, Person, & Magliano, 1995). AutoTutor’s approach to tutoring was inspired by the constructivist approach to learning that was described in the previous section (Aleven & Koedinger, 2002; Chi et al., 1994; Piaget, 1952, 1970; VanLehn et al., 1992; Vygotsky, 1978). AutoTutor was created by the Tutoring Research Group (TRG) at the University of Memphis and has been described previously, so the depiction here is brief (e.g., Craig, Gholson, Ventura, Graesser, & TRG, 2000; Graesser et al., 1999; Graesser, Person, Harter, & TRG, 2001; Graesser, Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, Person, & TRG 2000; Rajan, Craig, Gholson, Person, Graesser, & TRG, 2001). The architecture of AutoTutor used in the present research includes seven modules: (a) curriculum script, (b) language extraction, (c) speech act classifier, (d) latent
Constructing Knowledge from Dialog in an Intelligent Tutoring System
167
semantic analysis (LSA) (Landauer & Dumas, 1997), (e) topic selection, (f) dialog management, and (g) a talking head. In the present research, it tutored college students on 12 computer literacy topics concerned with hardware. AutoTutor was designed, however, as a generic authoring tool, and one recent web-based version tutors students on Newtonian physics (VanLehn & Graesser, 2002). AutoTutor serves as a conversational partner with the student. Its contributions to the tutorial dialog are designed to elicit lengthy explanations and promote deep reasoning. This is done through a variety of dialog moves: questions, assertions, hints, prompts, pumps, and immediate back-channel feedback (Person, Graesser, Kreuz, Pomeroy, & TRG, 2001). The talking head (when used) displays facial expressions, gestures, and gaze, while delivering dialog moves with synthesized speech that includes inflection and intonation (see Graesser et al., 2000). In some of the research described below, the talking head was removed from the monitor, but the same voice was used to deliver dialog moves via the same speech engine (Microsoft, 1998). In other conditions, onscreen printed text was added to the voice as redundant onscreen printed text. Students always used a keyboard and dialog box for their contributions to the dialog. Each of the 12 topics began with AutoTutor presenting a brief information delivery. This was followed by a question for the learner to answer. For each topic’s question, the TRG constructed an ideal answer. This answer was then decomposed into a set of key concepts that are called “good aspects.” Using LSA, AutoTutor is able to assess the learner’s progress by comparing their contributions to the contents of the good aspects. It builds upon the student’s responses by assuring that each good aspect is covered on each topic. Once a good aspect is covered, AutoTutor moves on to another one until all good aspects for a given topic are covered. AutoTutor then presents a brief summary, before moving on to the next topic. When both the learner and AutoTutor are considered, it usually takes between ten and 30 dialog turns to fully address each topic’s question (Graesser et al., 2000). In previous research using AutoTutor, there was always a dialog between the ITS and the learner. Some have suggested that such direct interactions may be critical to understanding the dialog’s contents (Anderson et al., 1995; Fox Tree, 1999). Schober and Clark (1989), for example, have argued that over-hearers of dialog are less successful than active participants because they fail to establish “common ground.” The latter refers to information shared by dialog participants that allows for mutual understanding. McKendree et al. (1998) have suggested that, at least under some conditions, over-hearers
168
Craig, Driscoll, and Gholson
are able to establish common ground and are as capable of knowledge construction as active participants in the dialog (Fox Tree, 1999; McKendree, Good, & Lee, 2001). No published studies were located, however, that directly assessed vicarious learning gains from observing other students’ interactions in an ITS environment. That is, little is known regarding how much is learned by an observer of a tutorial dialog relative to the gains achieved by an active participant in the dialog (Craig et al., 2000; Driscoll, Craig, Gholson, Ventura, Hu, Graesser, & TRG, in press; Fox Tree, 1999; Schober & Clark, 1989). In addition to the research on interactivity (Moreno et al., 2001) that was previously described, Moreno et al. (2001) also investigated the relative contributions of image and voice to the learning process. Neither the image of Herman, who did not display facial expressions or make eye contact (Experiment 4), nor the image of a facially-expressive drama actor, who made continuous eye contact (Experiment 5), contributed anything beyond voiceonly conditions with respect to either retention or transfer. Moreno et al. (2001) concluded that spoken narration was critical, but “that the agent’s visual presence did not provide any cognitive or motivational advantage for students’ learning” (p. 209). Experiment 1 was designed, in part, to explore the relative contributions of image and voice to the learning process in during tutorial dialog. EXPERIMENT 1 This study was designed primarily to compare learning gains obtained in an ITS under standard interactive conditions with gains obtained by vicarious learners. We also included two other manipulations. One involved the talking head (image). The talking head was included in one condition, but removed in another. The other manipulation involved the presence versus the absence of onscreen printed text along with spoken narration. There is some evidence that redundant spoken and printed text promotes learning and recall when no competing visual information needs to be processed (Lewandowski & Kobus, 1993; Moreno & Mayer, 2002; Penney, 1989). There is other evidence, however, for what Chandler and Sweller (1991; Moreno & Mayer, 2002) call the “redundancy effect.” This effect refers to situations in which “eliminating redundant material results in better performance than when the redundant material is included” (Kalyuga, Chandler, & Sweller, 1998, p. 2). For present purposes, the most relevant finding is that computerbased materials consisting of concurrent diagrams and auditory verbal information enhance performance when compared to computer-based materials
Constructing Knowledge from Dialog in an Intelligent Tutoring System
169
consisting of concurrent diagrams along with redundant onscreen printed text and auditory verbal information (Craig, Gholson, & Driscoll, 2002; Moreno &Mayer, 2002; Kalyuga, Ayres, Chandler, & Sweller, 2003; Kalyuga, Chandler, & Sweller, 1999). This negative effect on learning is explained in term of split attention (Sweller, 1988). According to cognitive load theory (Chandler & Sweller, 1991; Sweller, 1999), when learners split their visual attention between visually presented text and graphics, it can overload working memory capacity. Thus, it was deemed possible that the presence of the talking head along with redundant onscreen printed text might impede learning, instead of facilitating it. METHOD Participants and Design A total of 120 students drawn from an introductory psychology at The University of Memphis were tested. An additional 33 participants were replaced, because they exceeded a domain-knowledge criterion on a pretest (see Materials and Procedures). The criterion was adopted because (a) some students in the pool have completed a college-level computer literacy course that this version of AutoTutor was designed to support, and (b) because previous research in multimedia environments has shown that learning gains are greatest among those with low domain knowledge (Mayer, 1997, 2001). The basic design was a 2 (instruction type: interactive vs. vicarious) x 2 (talking head: present vs. absent) x 2 (printed text: present vs. absent) factorial, with 15 participants assigned to each of the 8 cells. Materials and Procedures In the interactive condition, there was direct interactive dialog between the student and AutoTutor. The learner used the dialog box and keyboard to respond to AutoTutor’s various questions, assertions, hints, prompts, and pumps (Rajan et al., 2001). All contents (visual and auditory) of each interactive session were captured as an AVI file using Camtasia (Techsmith, 2001). This included all contents presented by AutoTutor, whether as spoken narration, or spoken narration along with redundant onscreen printed text, along with student contributions to the dialog presented in the dialog box. Each captured session was then presented to a yoked participant in the
170
Craig, Driscoll, and Gholson
vicarious-tutoring conditions, who simply watched and listened to the session. Due to the yoking procedure, the first four students tested were randomly assigned to the four interactive conditions; after that random assignment was to all eight groups. The experiment included both paper-and-pencil and computerized materials. The paper-and-pencil materials consisted of two multiple-choice tests. The two tests, used as pretest and posttest, each comprised 24 four-foil questions, two on each of the 12 topics covered in the session. The data of any participant who exceeded a score of nine on the pretest (chance = six) were replaced. The order in which the tests were administered, as pretest or posttest, was counterbalanced across the first 14 subjects in each cell, with order selected randomly for the 15th. Immediately after informed consent was obtained, the pretest was administered. It was followed by the computerized session, and then the posttest. Three Pentium computer systems with head phones were used for program presentation. The computerized materials were produced using AutoTutor and three other computer applications. Microsoft Agent (1998) was used to provide spoken narration and onscreen printed text in all conditions. As noted earlier, Camtasia 3.0 software (Techsmith, 2001) was used to capture interactive sessions for presentation to yoked participants in the vicarious-tutoring conditions. In all conditions, each of the 12 topics was introduced by a brief information delivery. This was followed by a question for the learner. The tutorial contents consisted of AutoTutor’s 12 computer hardware topics, three of which were accompanied by a relevant picture. The duration of each student’s interactive session with AutoTutor varied, depending on the nature of the dialogue. The computerized sessions usually range from 30 to 40 minutes in duration, with the entire session lasting about one hour. Conditions In the talking-head-present condition, AutoTutor interacted with the learner through the talking head, which was located on the left side of the monitor. The talking head spoke the contents of AutoTutor’s part of the dialog. The talking head included a synthesized voice with intonation and inflection, along with facial expressions, gestures, and gaze. In the talkinghead-absent condition, AutoTutor spoke the tutorial contents using the same synthesized voice, but the talking head was removed from the monitor throughout. After the information delivery, presentation of the question, and
Constructing Knowledge from Dialog in an Intelligent Tutoring System
171
the learner’s first contribution on each topic, the content spoken by AutoTutor on each dialog turn was selected by LSA from the curriculum script in all conditions. In the printed-text-present condition, the spoken narration was accompanied by redundant onscreen printed text. The printed text was placed inside a cartoon-like bubble located directly above the position of the talking head. The printed text was presented one sentence at a time, as it was being spoken by AutoTutor. In the printed-text-absent condition, only AutoTutor’s spoken narration was provided, with no bubble located on the monitor. A screen shot of the interface, containing the talking head, the bubble with text, the question for a topic, and a picture is presented in Figure 1. The lower right section of the interface contains the dialog box for the learner’s typed contributions.
Figure 1. A screen shot of the interface containing the talking head, the bubble with text, the question for the topic, and a picture for the topic
172
Craig, Driscoll, and Gholson
RESULTS AND DISCUSSION A 2 (instruction type: interactive vs. vicarious) x 2 (talking head: present vs. absent) x 2 (printed text: present vs. absent) ANOVA performed on the pretest data yielded no significant effects. A 2 (instruction type: interactive vs. vicarious) x 2 (talking head: present vs. absent) x 2 (printed text: present vs. absent) x 2 (test: pretest vs. posttest) ANOVA performed on the multiple-choice data yielded significant effects of test, F(1, 112) = 138.05, p < .001, and an interaction between test and instruction type, F(1,112) = 3.94, p < .05. Analyses performed on each instruction type taken separately revealed significant differences between pretest and posttest for both the interactive, F(1,56) = 96.37, p < .001, and vicarious condition, F(1,56) = 46.68, p < .001. The mean, standard deviation, and effect size (Cohen’s d) for the differences between pretest and posttest for each instruction type, talking-head, and printed-text condition are presented in Table 2. Contrasting the interactive tutoring condition with the vicarious condition revealed a significant effect, F(1,111) = 4.28, p < .05, in favor of interactive tutoring. The overall mean difference between pretest and posttest in the interactive condition was 4.39 and the mean gain in the vicarious condition was 3.12. The pretest-to-posttest effect size, computed using Cohen’s d statistic with the pooled standard deviation for each condition, for the interactive session type was 1.73, and for the vicarious session was 1.19. This latter score, while smaller than the effect size in the interactive condition, is more than one standard deviation, which is usually considered a full letter grade (Cohen et al., 1982; Graesser & Person, 1994). Table 1 Means (M), Standard Deviations (SD), and Effect Sizes (ES) for the Differences Between Pretests and Posttests for each Session-Type, TalkingHead, and Printed-Text Condition Text Condition
Printed Text No Printed Text
Interactive Session Type: Talking Head Condition Present Absent
M 4.87 3.47
SD 2.78 2.23
ES 1.75 1.56
M 5.00 4.20
SD 2.73 2.57
ES 1.83 1.63
Vicarious Session Type: Talking Head Condition Present Absent
Printed Text No Printed Text
M 3.00 3.73
SD 2.16 2.72
ES 1.39 1.37
M 3.07 2.67
SD 2.80 2.25
ES 1.10 1.18
Constructing Knowledge from Dialog in an Intelligent Tutoring System
173
The presence of an onscreen talking head displaying facial expressions, gestures, and gaze, had no effect when compared to spoken narration alone. Collapsing over other variables, the mean learning gain for talking-head present was 3.76 and the mean for talking-head absent was 3.73. This is consistent with findings discussed earlier (Lester et al., 1997; Moreno et al., 2001). The presence of onscreen printed text along with the spoken narration also failed to significantly affect performance, although the difference in favor of adding printed text was substantial in the interactive condition (M = 4.93 vs. M = 3.83), yielding an effect size of .43 (Cohen’s d). While not significant, this difference in favor of redundant onscreen printed text is consistent with results obtained previously under very different experimental conditions (Colquhoun, 1975; Lewandowski & Kobus, 1989; Montali & Lewandowski, 1996; Moreno & Mayer, 2002; Penney, 1989).
EXPERIMENT 2 The primary purpose of Experiment 2 was to see if the effect size (d = 1.19) obtained in the vicarious condition in the Experiment would replicate. The second purpose reflects a growing interest in computer-supported collaborative learning (Blaye, Light, Joiner, & Sheldon, 1991; Hooper, Chanchai, & Williams, 1993; Jehng, & Chan, 1998). The research we located involved learners working at terminals while interacting with each other and the computerized environment, but we failed to locate any research in which there was collaboration between vicarious learners while observing interactive tutoring sessions in an ITS. If results from the vicarious-tutoring condition of Experiment 1 hold, and the usual gains from collaborative learning are obtained (Hooper et al., 1993; Johnson & Johnson, 1986; Slavin, 1983) in a vicarious condition, collaboration vicarious learning might produce gains that approach those usually obtained from interactive tutoring (Bloom, 1984; Cohen et al., 1982; Graesser & Person, 1994; Graesser, VanLehn, Rose, Jordan, & Harter, 2001; Person et al., 2001). METHOD Design and Participants The participants were 110 students drawn from the same pool as in Experiments 1 and 2. An additional 40 exceeded the domain knowledge criterion
174
Craig, Driscoll, and Gholson
described earlier (Experiment 1) on the pretest. The design was a simple one way with three groups. Twenty-eight students were assigned to an interactive tutoring session, 28 were assigned to an individual vicarious-tutoring condition, and 27 pairs of students were assigned to a collaborative vicarious-tutoring condition. In the collaborative condition, the data were replaced if either of the pair exceeded the domain knowledge criterion. The three conditions will be referred to as the interactive, vicarious, and collaborative conditions, respectively. The first two students were assigned to the interactive condition, after that assignment was random to all three conditions. Materials and Procedures The two multiple-choice tests described earlier were used to assess learning gains. Their use as pretest and posttest was counterbalanced across participants in each condition. After informed consent was obtained, students were administered the pretest, which was immediately followed by the computerized program, and the posttest. Instructions to participants differed somewhat across the three conditions. In the interactive condition, it was emphasized that AutoTutor only responded to the learner’s typed contributions after the return key was hit, so it was possible to stop the program to think at any time during the session. In the vicarious condition, learners were told that anytime they wanted to stop and think they could stop the session by a mouse-click anywhere on the interface, and that another click would continue the session. In the collaborative condition, each of the pair of students used their own mouse. They were told that, to facilitate their collaboration, they could stop and restart the session at any time by mouse-clicks on the interface. AutoTutor’s 12 computer literacy topics were covered in each participant’s session. In the interactive sessions, the students interacted directly with AutoTutor. The talking head delivered AutoTutor’s contributions to the dialog. Learners responded using the keyboard and dialog box. All contents (visual and auditory) of each interactive session were captured as an AVI file and presented to a yoked participant in the vicarious condition and to a pair of yoked participants in the collaborative condition. RESULTS AND DISCUSSION As in Experiment 1, in both the interactive-tutoring and vicarious-tutoring conditions each participant’s pretest and posttest score was computed
Constructing Knowledge from Dialog in an Intelligent Tutoring System
175
and entered for purposes of analysis. In the collaborative condition each pair of collaborators were treated as a single unit. That is, the pair’s pretest scores were both computed, the mean determined, and that score was entered into the analysis for the pretest score. A similar procedure was followed for the posttest scores. A 3 (instruction type: interactive tutoring, individual vicarious, collaborative vicarious) way ANOVA performed on the pretest data yielded no significant effects. A 3 (instruction type: interactive tutoring, individual vicarious, collaborative vicarious) x 2 (test: pretest, posttest) ANOVA, performed on the multiple-choice data, yielded an effect of test, F(1,80) = 133.8, p < .001, and an interaction between test and instruction type, F(2,80) = 3.72, p < .05. The mean change from pretest to posttest, along with the pooled standard deviation, and effect size (Cohen’s d) for each instruction-type condition are presented in Table 2. Tukey HSD tests revealed that learners in the interactive tutoring condition showed significantly greater improvements from pretest to posttest (M = 5.07), than those in the vicarious condition (M = 2.82). Those in the collaborative condition showed intermediate gains (M = 3.87) and did not differ significantly from the other two conditions. The effect size, when learning gains in the interactive condition were compared to those in the vicarious condition was .71, and when the interactive was compared to the collaborative condition, the effect size was .38. The effect size contrasting the collaborative with the vicarious condition was .33. Table 2 Means (M), Standard Deviations (SD), and Effect Sizes (ES) for the Differences Between Pretests and Posttests for Each Instruction Type Instruction Type
Interactive Tutoring Vicarious Tutoring Collaborative
Pretest-Posttest Change M 5.07 2.82 3.87
SD 2.46 2.90 1.78
ES 2.06 0.97 2.17
The mean gains from pretest to posttest in the interactive condition (5.07) were similar to those obtained in that condition in Experiment 1 (4.39). The effect sizes were also reasonably comparable, with 2.05 in Experiment 2 and 1.73 in Experiment 1. The mean gain from pretest to posttest in the vicarious condition (2.82) of Experiment 2 was similar to the gain obtained in the same condition in Experiment 1 (3.12), the effect size from pretest to posttest for the vicarious condition in Experiment 2 (.97) was also comparable to that obtained in Experiment 1 (1.19).
176
Craig, Driscoll, and Gholson
SUMMARY AND CONCLUSIONS Learning gains in the interactive condition were substantial, with a mean effect size from pretest to posttest of 1.77 (Cohen’s d) across the five interactive conditions included in Experiments 1 and 2 (see Tables 1 and 2). This gain appears reasonable, given tutoring sessions of little more than half an hour and learners with no investment in the outcome of the tutoring. In fact though, they were still answering correctly only a little more than half of the 24 the questions on the posttest (chance is 6). Consistent with previous findings involving ITSs (Andre, Rist, & Muller, 1999; Atkinson, 2002, Experiment 1; Moreno et al., 2001; Lester et al., 1997, 1999), Experiment 1 showed that while a talking head displaying facial expressions, gestures, and gaze during dialog does not produce a split attention effect and concomitant decrements in performance, it also does not enhance performance when compared to a condition that includes only spoken narration. Research investigating ITSs that include embodied pedagogical agents is still in its relative infancy (Andre et al., 1999; Lester et al., 1997, 1999; Moreno et al., 2001), and it is possible that the latter finding will change as further research is reported that focuses on how to more effectively employ animated pedagogical agents in ITS environments. We are aware that not everyone shares our optimistic assessment: Erickson (1997), for example, has suggested that the agent metaphor may be more trouble than it is worth (p. 91). Simultaneous presentation of spoken narration and redundant onscreen printed text facilitates performance when compared to either spoken narration or onscreen printed text in a variety of tasks (e.g., Colquhoun, 1975; Lewandowski & Kobus, 1989; Montali & Lewandowski, 1996; Moreno & Mayer, 2002; Penney, 1989), but we failed to obtain a significant effect of redundancy in Experiment 1. As Moreno and Mayer (2002), Chandler & Sweller (1991), Mayer (2001), and Sweller (1999) pointed out, further research is clearly needed to determine the conditions where adding redundant onscreen printed text to spoken narration facilitates performance and the conditions under which it produces the opposite effect (Craig, et al., 2002; Chandler & Sweller, 1991; Kalyuga, Chandler, & Sweller, 1999; Mayer, 1997, 2001; Sweller, 1999). In the vicarious-tutoring condition the effect size averaged 1.20 (Cohen’s d) across the five conditions of Experiments 1 and 2. (see Tables 1 and 2), which is about two thirds of the gains obtained from interactive tutoring. This is a reasonable gain, given that learners were simply asked to watch the capture session, with no further intervention. An important goal
Constructing Knowledge from Dialog in an Intelligent Tutoring System
177
for future research, then, is to investigate conditions designed to promote learning activities among observers that bring their gains to the same level as those obtained among active participants in the tutorial process. The collaborative condition in Experiment 2 may be a small step in that direction. When interactive tutoring was compared to the collaborative condition, the effect size was only .38, in contrast to the .71 effect size that was obtained when interactive tutoring was compared to the vicarious condition. In some respects, this was surprising, because there was little collaboration. The mean number of collaborative activities across the 27 pairs assigned to that condition was 2.91. The collaborative condition in Experiment 2 was exploratory, and in future research it will include the kinds of structured activities that have proven effective in peer tutoring and cooperative learning (Cohen, 1994; King, Staffieri, & Adelgais, 1998; Webb & Farivar, 1994). Although the learning gains in the interactive conditions (Tables 1 and 2) were 1.77 standard deviation units, as indicated previously, the learners were still only answering only about half of the 24 the questions on the posttest correctly. This may be due, at least in part, to the tutoring strategies implemented in AutoTutor. As was noted earlier, current versions implement the relatively unsophisticated tutoring strategies of untrained peer tutors and paraprofessionals. These human tutors mostly just fill in missing pieces of information in expected answers, while attempting to fix any bugs and misconceptions that are detected. The TRG is currently attempting to improve the results of AutoTutor ’s performance in two ways. One is by implementing more sophisticated Socratic tutoring strategies, modeling-scaffoldingfading, and other intelligent pedagogical techniques (Collins, Brown, & Newman, 1989; Lajoie & Lesgold, 1989; Rogoff, 1990). Another is by monitoring the learner’s various affective states during tutoring and tailoring AutoTutor’s ongoing contributions to the dialog to those states (Mandler, 1975, 1984, 1999; Picard, 1997; Stein & Levine, 1991; Stein, Sheldrick & Broaders, 1999) as well as to where students are in the learning cycle (Bruner, 1964; Bruner, Olver, & Greenfield, 1966; Gagne, 1977; Kort, Reilly, & Picard, 2001). References Aleven, V., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science, 26, 147-180. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4, 167-202.
178
Craig, Driscoll, and Gholson
Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading comprehension. In P.D. Pearson, R. Barr, M.L. Kamil, & P. Mosenthal (Eds.), Handbook of reading research (Vol. 1) (pp. 255-291). White Plains, NY: Longman. Andre, E., Rist, T., & Muller, J. (1999). Employing AI methods to control the behavior of animated interface agents. Applied Artificial Intelligence, 13, 415-448. Atkinson, R. K. (2002). Optimizing learning from examples using animated agents. Journal of Educational Psychology, 94, 416-427. Bandura, A. (1962). Social learning through imitation. In M.R. Jones (Ed.), Nebraska symposium on motivation, (pp. 211-269). Lincoln, NB: University of Nebraska Press. Bandura, A. (1977). Social learning theory. Englewood Cliffs, NJ: Prentice Hall. Barker, B. O., & Dickson, M. W. (1996). Distance learning technologies in K-12 schools: Past, present, and future practice. Techtrends, 41, 19-22. Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 347-364. Blaye, A., Light, P., Joiner, R., & Sheldon, S. (1991). Collaboration as a facilitator of planning and problem solving on a computer based task. British Journal of Psychology, 9, 471-483. Bloom, (1956). Taxonomy of education objectives: The classification of educational goals. Handbook I: Cognitive domain. New York: McKay. Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4-16. Bourdeau, J., & Bates, A. (1997). Instructional design for distance learning. In R. D. Tennyson, S. Dijsktra, N. Steel, & F. Schott (Eds.), Instructional design: International perspectives (pp. 369-397). Hillsdale, NJ: Lawrence Erlbaum. Bransford, J.D., Goldman, S.R., & Vye, N.J. (1991). Making a difference in people’s ability to think: Reflections on a decade of work and some hopes for the future. In R.J. Sternberg & L. Okagaki (Eds.), Influences on children (pp. 147-180). Hillsdale, NJ: Lawrence Erlbaum. Brown, A.L. (1988). Motivation to learn and understand: On taking charge of one’s own learning. Cognition and Instruction, 5, 311-321. Bruner, J.S. (1964). The course of cognitive growth. American Psychologist, 19, 1-15. Bruner, J.S., Olver, R.R., & Greenfield, P.M. (1966). Studies in cognitive growth. New York: Wiley. Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293-332. Chi, M. T. H., de Leeuw, N., Chiu, M., & La Vancher, C. (1994). Eliciting selfexplanations improves understanding. Cognitive Science, 18, 439-447. Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Education Research, 64, 1-35.
Constructing Knowledge from Dialog in an Intelligent Tutoring System
179
Cohen, P.A., Kulik, J.A., & Kulik, C.C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237-248. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L.B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Lawrence Erlbaum. Colquhoun, W.P. (1975). Evaluation of auditory, visual, and dual-mode displays for prolonged sonar monitoring in repeated sessions. Human Factors, 17, 425-437. Corbett, A.T. (2001). Cognitive computer tutors: Solving the two-sigma problem. User Modeling: Proceedings of the Eighth International Conference (pp. 137-147). Cox, R., McKendree, J., Tobin, R., Lee, J., & Mayes, T (1999). Vicarious learning from dialogue and discourse. Instructional Science, 27, 431-458. Craig, S. D., Gholson, B., & Driscoll, D. (2002). Animated pedagogical agents in multimedia educational environments: Effects of agent properties, picture features, and redundancy. Journal of Educational Psychology, 94, 428-434. Craig, S.D., Gholson, B., Ventura, M., Graesser, A.C., & Tutoring Research Group (2000). Overhearing dialogues and monologues in virtual tutoring sessions: Effects on questioning and vicarious learning. International Journal of Artificial Intelligence in Education, 11, 242-253. Derry, S. J. (1996). Cognitive schema theory in the Constructivist debate. Educational Psychologist, 31, 163-174. Driscoll, D.M., Craig, S.D., Gholson, B., Ventura, M., Hu, X., Graesser, A.C., & Tutoring Research Group (2003). Vicarious learning: Effects of overhearing dialogue and monologue-like discourse in a virtual tutoring session. Journal of Educational Computing Research, 29, 431-450. Elliott, A.J., McGregor, H.A. & Gable, S. (1999). Achievement goals, study strategies, and exam performance: A mediational analysis. Journal of Educational Psychology, 91, 549-563. Erickson, T. (1997). Designing agents as if people mattered. In J. Bradshaw (Ed.) Software agents, (pp. 79- 96). Menlo Park, CA: AAAI Press. Fox Tree, J.E. (1999). Listening in on monologues and dialogues. Discourse Processes, 27, 35-53. Gagne, R.M. (1977). The conditions of learning (3rd ed.). New York: Holt, Reinhart, & Winston. Graesser, A.C. & Person, N. (1994). Question asking during tutoring. American Educational Research Journal, 31, 104-137. Graesser, A.C., Person, N., Harter, D., & Tutoring Research Group (2001). Teaching tactics and dialogue in AutoTutor. International Journal of Artificial Intelligence in Education, 12, 257-279. Graesser, A.C., Person, N.K., & Magliano, J.P. (1995). Collaborative dialogue patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychology, 9, 359-387.
180
Craig, Driscoll, and Gholson
Graesser, A.C., VanLehn, K., Rose, C., Jordan, P., & Harter, D. (2001). Intelligent tutoring systems with conversational dialogue. AI Magazine, 22, 39-51. Graesser, A.C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N., & Tutoring Research Group. (2000). Using latent semantic analysis to evaluate the contributions of students in AutoTutor. Interactive Learning Environments, 8, 129-148. Graesser, A.C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & Tutoring Research Group (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51. Hooper, S., Chanchai, T., Williams, M.D. (1993). The effects of cooperative learning and learner control on high- and average-ability students. Educational Technology Research and Development 41, 5-18. Jehng, J.C. & Chan T.W. (1998). Designing computer support for collaborative visual learning in the domain of computer programming. Computers in Human Behavior, 14, 429-448. Johnson, R.T., & Johnson, D.W. (1986). Action research: Cooperative learning in the science classroom. Science and Children, 24, 31-32. Kalyga, S., Ayers, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38, 23-31. Kalyuga, S., Chandler, P., & Sweller, J. (1998). Levels of expertise and instructional design. Human Factors, 40, 1-17. Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13, 351-371. King, A., Staffieri, A., & Adelgais, A. (1998). Mutual peer tutoring: Effects of structuring tutorial interaction to scaffold peer learning. Journal of Educational Psychology, 90, 134-152 Kort, B., Reilly, R., & Picard, R. (2001). An affective model of interplay between emotions and learning: Reengineering educational pedagogy building a learning companion. In T. Okamoto, R. Hartley, Kinshuk, & J. P. Klus (Eds.), Proceedings of IEEE International Conference on Advanced Learning Technology: Issues, Achievements and Challenges (pp.43-48). Madison, Wisconsin: IEEE Computer Society. Lajoie, S. P., & Lesgold, A. (1989). Apprenticeship training in the workplace: A computer coached practice environment as a new form of apprenticeship. Machine-Mediated Learning, 3, 7-28. Landauer, T.K., & Dumas, S.T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211-240. Lee, J., Dineen, F., & McKendree, J. (1998). Supporting student discussions: It isn’t just talk. Education and Information Technologies, 3, 17-29. Lester, L.C., Converse, S.A., Kahler, S.F., Barlow, S.T., Stone, B.A., & Bhogal, R. (1997). The persona effect: Affective impact on animated pedagogical agents. Proceedings of CHI’97 (pp. 359-366).
Constructing Knowledge from Dialog in an Intelligent Tutoring System
181
Lester, L.C., Stone, B.A., & Stelling, G.D. (1999). Lifelike pedagogical agents for mixed-initiative problem solving in constructivist learning environments. User Modeling and User-Adapted Interaction, 9, 1-44. Lester, L.C., Voerman, J.L., Towns, S.G., & Callaway, C.B. (1999). Deictic believability: Coordinating gesture, locomotion, and speech in lifelike pedagogical agents. Applied Artificial Intelligence, 13, 383-414. Lewandowski, L.J., & Kobus, D.A. (1989). Bimodal information processing in sonar performance. Human Performance, 2, 73-84. Lewandowski, L.J., & Kobus, D.A. (1993). The effects of redundancy in bimodal word processing. Human Performance, 6, 229-239. Mandler, G. (1975). Mind and emotion. New York: Wiley. Mandler, G. (1984). Mind and body: Psychology of emotion and stress. New York: Norton. Mandler, G. (1999). Emotion. In B.M. Bly & D.E. Rumelhart (Eds.), Cognitive science. Handbook of perception and cognition (2nd ed.), (pp. 367-384). San Diego, CA: Academic Press. Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? Educational psychologist, 32, 1-19. Mayer, RE. (2001). Multimedia learning. Cambridge, UK: Cambridge University Press. McKendree, J., Good, J., & Lee, J. (2001, August). Effects of dialogue characteristics on performance of overhearers. Cognitive Science Society Meetings, Edinburgh, Scotland. McKendree, J., Stenning, K., Mayes, T., Lee, J., & Cox, R. (1998). Why observing a dialogue may benefit learning. Journal of Computer Assisted Learning, 14, 110-119. Microsoft Agent (1998). Microsoft Corporation http://www.microsoft.com/ msagent/ Montali, J., & Lewandowski, L, (1996). Bimodal reading: Benefits of a talking computer for average and less skilled readers. Journal of Learning Disabilities, 29, 271-279. Moore, M.G., & Kearsley, G. (1996). Distance education: A systems view. Albany, NY: Wadsworth. Moreno, R., & Mayer, R. (2002). Verbal redundancy in multimedia learning: When reading helps listening. Journal of Educational Psychology, 94, 156-163. Moreno, R., Mayer, R., Spires, H., & Lester, J., (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19,177-213. Moshman, D. (1982). Exogenous, endogenous, and dialectical constructivism. Developmental Review, 2, 371-384. National Council of Teachers of English (NCTE), (1996). Standards for the English Language Arts. Urbana, IL: NCTE.
182
Craig, Driscoll, and Gholson
National Council of Teachers of Mathematics (NCTM), (1991). Professional Standards for Teaching Mathematics. Reston, VA: NCTM. National Research Council (NRC), (1996). National Science Education Standards. Washington, DC: National Academy Press. Palincsar, A.S., & Brown, A. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition & Instruction, 1, 117-175. Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York: Basic Books. Penney, C.G. (1989). Modality effects and the structure of short-term verbal memory. Memory and Cognition, 17, 257-285. Person, N.K., Graesser, A.C., Kreuz, R.J., Pomeroy, V., & Tutoring Research Group (2001). Simulating human tutor dialog moves in AutoTutor. International Journal of Artificial Intelligence in Education, 12, 23-39. Piaget, J. (1952). The origins of intelligence in children. New York: International University Press. Piaget, J. (1970). Genetic epistemology. New York: Columbia University Press. Picard, R. W. (1997). Affective computing. Cambridge, MA: MIT Press. Pressley, M., & Wharton-McDonald, R. (1997). Skilled comprehension and its development through instruction. School Psychology Review, 26, 448-466. Rajan, S., Craig, S. D., Gholson, B., Person, N. K., Graesser, A. C., & Tutoring Research Group. (2001). AutoTutor: Incorporating back-channel feedback and other human-like conversational behaviors into an intelligent tutoring system. International Journal of Speech Technology, 4, 117-126. Renwick, W. (1996). Informative technology and the future of post secondary education. Paris: OECD. Roblyer, M.D., & Edwards, J. (2000). Integrating educational technology into teaching (2nd ed.). Upper Saddle River, NJ: Merrill. Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social context. New York: Oxford University Press. Rosenthal, R.L., & Zimmerman, B.J. (1978). Social learning and cognition. New York: Academic Press. Schober, M.F., & Clark, H.H. (1989). Understanding by addressees and overhearers. Cognitive Psychology, 21, 211-232. Slavin, R. (1983). When does cooperative learning increase student achievement? Psychological Bulletin, 94, 429-445. Spector, J. M. (2001). Tools and principles for the design of collaborative learning environments for complex domains. Journal of Structural Learning & Intelligent Systems, 14, 483-510. Stein, N.L., & Levine, L.J. (1991). Making sense out of emotion. In W. Kessen, A. Ortony, & F. Craik (Eds.), Memories, thoughts, and emotions: Essays in honor of George Mandler, (pp. 295-322). Hillsdale, NJ: Lawrence Erlbaum. Stein, N.L., Sheldrick, R.A., & Broaders, S.C. (1999). Predicting psychological well-being from beliefs and goal-appraisal processes during experience of emotional events. In S.R. Goldman, A.C. Graesser, & P. van den
Constructing Knowledge from Dialog in an Intelligent Tutoring System
183
Brock (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 279-301). Mahwah, NJ: Lawrence Erlbaum. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285. Sweller, J. (1999). Instructional design in technical areas. Camberwell, Victoria, Australia: Australian Council for Educational Research. TechSmith (2001). Camtasia (Version 3.0) [Computer software]. East Lansing, Michigan: TechSmith Corporation. VanLehn, K., & Graesser, A.C. (2002). Why2 report: Evaluation of why/atlas, why/autotutor, and accomplished human tutors on learning gains for qualitative physics problems and explanations. Unpublished report prepared by the University of Pittsburgh CIRCLE group and the University of Memphis Tutoring Research Group. VanLehn, K., Jones, R.M., & Chi, M.T.C. (1992). A model of the self-explanation effect. The Journal of the Learning Sciences, 2, 1-59. Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press. Webb, N.M., & Farivar, S. (1994). Promoting helping behavior in cooperative small groups in middle school mathematics. American Educational Research Journal, 31, 369-395. Wittrock, M.C. (1974). Learning as a generative process. Educational Psychologist, 1, 87-95. Wittrock, M.C. (1990). Generative processes of comprehension. Educational Psychologist, 24, 345-376.
