the structure of the agent's animated thought into their own thinking. .... students incorporate these representations to organize their own domain knowledge, with.
1 ANIMATIONS OF THOUGHT: INTERACTIVITY IN THE TEACHABLE AGENT PARADIGM Daniel L. Schwartz, Kristen P. Blair Gautam Biswas, Krittaya Leelawong Stanford University Vanderbilt University Joan Davis University of Washington, Seattle To appear in: Learning with Animation: Research and Implications for Design. R. Lowe & W. Schnotz (Eds). UK: Cambridge University Press.
--- NOT THE FINAL VERSION --Animations are abstractions. They highlight select features through spatial structures and their transformations. Cartoons, for example, exaggerate distinctive features and eliminate others. Because animations are abstractions, they do not need to resemble their referents. An animated speedometer does not resemble a car’s speed; it covaries with it. Animations can even depict inherently non-spatial phenomenon, for example, changes in political power. In cases like these, animations reflect an expert’s constructs and theories about a phenomenon. They become animations of thought. Animations of thought may be important for instruction, because the goal of learning is not simply to gather factual information about a phenomenon, but also to learn how experts represent and reason about that phenomenon. We have taken animations of thought to an extreme. In most animations, one needs to guess at the thinking processes behind the animation. In contrast, we have made the infusion of thought explicit. We have created software environments called Teachable Agents (TAs). Students teach a computer agent, and then they see how it animates its thought based on what it was taught. Each TA comes with simple tools that students use to build important representations such as concept maps. Students teach their agents by building representations populated with domain specific facts and relations. For instance,
2 they can make a concept map about a pond’s oxygen cycle; they might teach that bacteria decrease oxygen and that oxygen increases fish. Each TA also has a reasoning engine suited to the structure of the visualization. For example, with the concept map representation, the agent can reason about chains of causality to connect bacteria with fish. The TA uses the reasoning engine to answer questions and solve problems based on what the student taught, and the TA demonstrates its reasoning through an animation. For example, the TA might highlight successive links in the concept map to animate a chain of inference that an increase in bacteria decreases fish. Depending on the conclusion the agent reaches, students can revise its knowledge base (and their own). We propose that interactive opportunities for students to co-mingle their ideas with the TAs animations of thought, plus opportunities to share initiative with the TA in the learning process, can greatly enhance learning from animations. TAs enhance the pedagogical value of animations, because they permit a range of useful interactions like asking questions and receiving answers. The TAs’ animations are often quite minimal and do not create a sense of smooth motion or perceived realism. We are more focused on animating thoughts than animating embodied thinkers, and it is not obvious what a perceptually realistic animation of thought would look like. In fact, TA animations might better be called “dynamic graphics,” because they typically involve discrete changes within formal representations such as graphs and concept maps. Even so, we think the addition of interactivity to these minimal animations can be quite valuable for learning. Our focus in this chapter is how interactivity can help capitalize on animations that “make thinking visible.”
3 The interactive aspect of TAs that we find particularly valuable is an agent’s ability to take independent actions that depend on how it has been taught. Once an agent has been taught, it is no longer under the immediate control of the student, but its performance still reflects what it has been taught. An analogy comes from parents who watch their child perform. The child’s performance depends on the parents, yet it is still independent of them, and this can be very motivating and increase the parent’s own productive agency (Schwartz, 1999). At the same time, we think there are important benefits for learning. We propose two dimensions of interactivity that are particularly beneficial for learning – incorporation and initiative. Incorporation captures the degree to which participants reflect one another in their ideas and actions. People can adopt one another’s words and gestures (Bernieri & Rosenthal, 1991; Brennan, 1996), and they can co-create ideas that neither would have had alone (Schwartz, 1995). Seeing another person reflect oneself can be motivating. Bailenson and Yee (in press), for example, demonstrated that people are tacitly drawn to a virtual agent that subtly echoes their movements. We further propose that people learn best when an interaction co-mingles ideas. This helps learners see their ideas reflected back to them as shaped by another person’s thoughts, and this facilitates their ability to learn. Co-mingling involves the cumulative integration of ideas, and this is different from simply paying attention or reacting. A chess program, for instance, is reactive. Though the moves of the program are contingent upon the player, the chess program hides its underlying reasoning to maintain the competition. This reduces the co-mingling of ideas. A novice, who cannot infer the program’s underlying strategy, may learn less than if there were a shared representation of the strategy. In
4 contrast, TAs’ animations of thought require students to co-mingle their ideas into the visual structures and reasoning processes of the agent, which should improve learning. The second dimension of interactivity – initiative – captures the degree to which each participant can take independent actions. Cassell (2004) demonstrated a storytelling system where a young child and an embodied conversational agent take turns creating a narrative, and this improves children’s linguistic skills. Being completely in charge or completely passive is not ideal. Barron (2003) showed that even when correct ideas are generated within a group, if the group is characterized by an imbalance of initiative, the group may not capitalize on these ideas and the individuals do not learn very well. In this case, the imbalance of initiative interfered with the co-mingling of ideas. A characteristic of initiative is that it involves choice. One makes a choice and receives feedback on that choice. Seeing other people’s initiative helps overcome the inertia of one’s own choices; it offers alternatives for action. It also generates “projective” feedback by seeing the consequences of another person’s initiative. Coaches, for example, may learn a great deal by watching their players take the initiative to adjust a play during a game. Figure 1 portrays an interaction space comprised of incorporation and initiative, where each dimension is characterized by the relative contribution of self and other. We have populated the space with some possible examples, described from the perspective of the self. The circle reflects our proposal that interactions closer to the center are exceptionally valuable for learning, especially for novices. For example, experts who have well-structured knowledge and a wealth of prior experiences can learn by quietly watching lectures and demonstrations in their domain. Novices do not have equal
5 knowledge. Opportunities to co-mingle ideas through animations plus share initiative may be particularly useful for early learning. The current chapter explores this proposal.
A Joint Interaction Space Relevant to Learning Other
Read a Story
Read an Interactive Book
Listen to a Lecture
Write a Book Review
Co-author a Book
Your Unauthorized Biography
Write a Story
Ask a Rhetorical Question
Be Plagiarized
Incorporation (ideas) Comingled
Self
Self
Shared
Other
Initiative (actions) Figure 1. Two generic dimensions of interactivity relevant to novice learning. Each dimension captures the degree to which the learner and another person or technology engages the interaction with their thoughts and actions. The italicized examples characterize activities from the perspective of the self. By hypothesis, interactions in the center lead to superior learning for novices. We begin by briefly reviewing the literature on learning by teaching, and introduce a teachable agent. We then describe evidence relevant to the interaction space. In each case, the TA and the student co-mingle their ideas – the student adds the “facts of the matter” while the agent incorporates the knowledge representations and built in animations to reason about those facts. We think this is one of the great strengths of the TA paradigm, so in the first pair of studies, we show that students do indeed incorporate the structure of the agent’s animated thought into their own thinking. This shows the
6 value of co-mingling ideas. We also show that these benefits depend on the agent’s initiative to answer the students’ questions and animate its thought. Given this baseline of evidence, we explore two forms of agent initiative. We show that both forms of initiative improve student learning, when student initiative is mixed in. We first explore the value of agent initiative in response to the environment (rather than in direct response to the student). Many characterizations of interaction use face-to-face exchange as the gold standard, but initiative that extends beyond the immediate context of exchange may be very valuable for learning. In one study, the agent publicly answers quiz questions in front of other students, and this leads to superior revision of knowledge. In the other study, the agent independently interacts with a game environment. We show that students who have an opportunity to see their agent take initiative in this environment learn more than students who perform on their own in the exact same environment. In the final set of studies, we explore the agent’s initiative in interaction with the student. So, instead of the agent providing feedback through a secondary context, the agent directly initiates interactions with the student about its learning. In the initial study, we describe a constructivist agent who makes explicit its interpretations of what it thinks the student is trying to teach. This independence of interpretation turns out to help students develop a more precise understanding themselves. In the final study, we describe a shared-initiative system where the agent can spontaneously initiate interaction about its learning. We use this system to help students learn to monitor and improve their own learning behaviors. Over a month later, these students demonstrate a superior ability to learn new content compared to students who use a system that owns the initiative and
7 teaches the student. All told, we will describe three separate TAs and demonstrate that students learn from the animations of the agents’ thought to help structure their own thinking, and that this learning depends on co-mingling and shared initiative. 1.0 Learning by Teaching Recently, one of us received a fortune cookie that stated, “Those who teach learn.” TAs let us explore and capitalize on this common wisdom. Our goal is not to make agents that learn automatically, for example, by inducing user preferences for web searches. Instead, we make agents that students must teach explicitly. Our research and development approach has been to consider why people might learn by teaching and to capitalize on the TA metaphor to recruit similar benefits for the learner. We typically design different agents that instantiate one or more of the possible benefits of teaching, and we do not yet have agents that embody all the possibilities, though each TA makes its thinking visible through animations. Our goal has not been to show that teaching an agent is the same as, or better than, teaching real pupils. We assume that TAs that make their thinking visible are good for particular learning outcomes, not all outcomes, and part of our goal is to identify what those outcomes are. Ideally, others can extend the benefits we identify to their own animation efforts, which may or may not use agents. Research in reciprocal teaching (e.g., Palinscar & Brown, 1984), peer-assisted tutoring (e.g., Willis & Crowder, 1974; Cohen, Kulik, & Kulik, 1982), programming (e.g. Kafai & Harel, 1999, Papert, 1980: Taylor,1980 ), small-group interaction (e.g., Webb, 1983), self-explanation (Chi et al., 1994, Ploetzner et al., 1999), classroom teaching (Sherin, 2002), and computerized learning companions (e.g., Chan & Chou, 1997) all suggest the potential of learning by teaching. More directly, Bargh and Shul (1980)
8 showed that people learn more when their grade depends on their pupils’ performance instead of their own. There are many reasons to suppose that teaching improves the learning of the teacher. Teachers organize their knowledge to anticipate the needs of their students, and they can receive feedback about the quality of that organization. Responsibility and public performance may also lead to a more in-depth effort at understanding. For example, we asked students in a college course to study an article to prepare either to teach the class or to take a test (Biswas, Schwartz, Bransford, & TAGV, 2001). Those students who prepared to teach studied longer, weighed the data and arguments, and developed more generative knowledge. For example, 87.5% of the teaching students could construct graphs of the data compared to 25% of the test students. One student who prepared for the test said in frustration, “I assumed I would just have to recognize the conclusions of the paper.” These college students were sophisticated learners to begin with, so the mere prospect of teaching elicited better learning behaviors. However, for less sophisticated students, it may be useful to help them shape their teaching interactions, and this is one reason why TAs can be valuable. Appropriate computerized environments with the right visual interfaces and interaction models can shape the form and content of teaching (e.g., Repenning & Sumner, 1995). TAs can help structure knowledge, teaching, and feedback using visual representations frequently found in specific knowledge domains. This helps students incorporate these representations to organize their own domain knowledge, with the added benefit of avoiding the overhead of asking students to learn to program so they can instruct the computer (Smith, Cypher, & Spohrer, 1997). Agents also help skirt a
9 potential problem of peer-teaching where students are put at risk if the person teaching them, or whom they are teaching, is not very good. 2.0 An Example of a Teachable Agent One strong benefit of the TA paradigm is that it capitalizes on the well-defined teaching schema that includes instruction, assessment, and remediation. This pre-existing schema can help organize otherwise complex student interactions with the computer, much as people’s well-defined schemas for spatial organizations have helped to create the desktop metaphor for computer operating systems. Premack (2003) has argued that only humans explicitly teach in the sense of monitoring the response of the pupil. With the TA metaphor, students bring to bear a host of prior ideas about teaching that can fill in otherwise impoverished graphical representations to engage and guide complex, productive interactions. Our first example of a TA, shown in Figure 2, is named Betty. Students teach Betty by creating a network of qualitative causal relationships that can include increase, decrease, depends-on, is-a, and has-a semantics. Students teach Betty a concept by using the panel at the right of the figure to add a node (e.g., algae, oxygen). They can then specify links between nodes. They label the links using their own terms (e.g., “exhale”) and use pull-down menus to specify the qualitative relation (e.g., algae increases oxygen). At any point, students can ask Betty a question using a floating panel (shown in the figure) that appears when students click on “Ask.” Students can ask questions like “If what happens to ?” Betty answers the question using the map. This constitutes the animation of thought – she highlights nodes and links
10 successively as she reasons through chains of events in the map. Changes in color help to indicate which nodes are increasing and decreasing.
Figure 2. The Teachable Agent named Betty. Betty uses a simple reasoning engine that adopts generic graph traversing algorithms (e.g., depth first and breadth first search) coupled with qualitative inference schemes that can reason through causal and hierarchical chains (see Biswas et al., 2004). Based on what she has been taught, Figure 2 shows the results of a graphical animation and a text-based response that Betty offers for the question, “If fish increase, what happens to dissolved oxygen?” Betty has inferred that fish are a type of animal. She then reasons that animals breathe oxygen, which reduces oxygen. She also reasons that fish eat plants, and because plants increase oxygen, this will further reduce the total amount of oxygen. Students can ask Betty to explain her answer, and through a multi-
11 step animation of the different paths through her concept map, Betty reveals her chains of reasoning in more detail. Betty also unfolds the specific elements of her inference through spoken dialog and a text window. Betty’s responses help students reflect on the consequences of what they have taught her thus far, and if they want, they can improve her knowledge base. During our early stages of development, we examined Betty’s usability. We asked 20 undergraduates to each teach Betty to be a consultant that could help people think about factors that affect getting a job (e.g., studying). At various points, we projected a student’s Betty and asked a question (e.g., “If studying increases what will happen to the chances of a good job”). Students were exceptionally attentive to the front-of-the-class “tests” and spontaneously discussed Betty’s answers and asked to see her reasoning unfolded. The activity was apparently quite motivating. We had not implemented a “save” function at this point, yet 65% of the students continued to work on their Betty for an hour after class, until they finally had to vacate the lab. Importantly, the students had little trouble learning how to teach and generate questions for Betty. This only took about 5 minutes. From there, students were quick to learn, for instance, that there can be competing pathways of causality. This study helped reveal several aspects of TAs that differentiate them from learning by programming, or by creating simulations and models. First, TAs are less expressive and less complete than general programming languages. The limited expressiveness makes it easy for students to get started compared to learning the complexity of a programming language, and it focuses their attention on a specific type of thinking. However, there are thoughts an agent cannot represent or are awkward to
12 represent. Therefore, we do not see TAs as the only means of instruction, but rather as way to help novices abstract important structures. Another difference from programming is that Betty does not “crash” if she is taught incorrect relations. Betty always reasons “logically,” even if the premises she was taught are incorrect, and this can help students identify gaps in knowledge. A third difference is that Betty does not hide her underlying simulation model in code, but instead, she manifests her knowledge through animations of thought. So, rather than trying to infer the rules regulating a simulation, students can focus on the chains of inference that a particular knowledge organization supports. 3.0 Co-Mingling Animations with Students’ Ideas One of the benefits of learning by teaching is the need to structure knowledge in a compact and communicable format. This requires a level of abstraction that may help the teacher develop pertinent explanatory structures for the domain. For example, many people find that preparing a conference presentation helps them decide which concepts deserve the “high level” status of introductory framing. As we mentioned, TAs have been designed to provide visual structures that help organize thinking. The studies in the current section show that students incorporate these representations into their own thinking when they co-mingle ideas with the agent’s visible thought, and this incorporation depends on whether the agent can animate its thinking. 3.1 Adopting the Structure of the Agent’s Thoughts We conducted a study with 16 undergraduates to determine if students incorporate Betty’s knowledge structure into their own. The study complements research on the positive benefits of concept mapping (Chmeilewski, & Dansereau, 1998; Kinchin & Hay, 2000; Novak, 1998; Stoyanov & Kommers, 1999). However, Betty differs from most
13 concept-mapping activities, because she enforces semantic relationships between nodes that students might otherwise violate in paper and pencil activities, and she shows the implications of those relationships by answering questions. Students read a four-page passage on metabolism. The eight students in the Summary condition wrote a summary of cell metabolism. We started them by suggesting they write about things like the relation between anaerobic ATP resynthesis and lactic acid. For the eight students in the Betty condition, we asked them to teach Betty about cell metabolism. They were shown how to teach Betty using the ATP – lactic acid example and how to ask Betty questions. We videotaped the sessions and asked the participants to think aloud. Students could work for 40 minutes. All Betty students worked to the cutoff point. The Summary students worked 32 minutes on average. One notable difference between the conditions was that every Betty student, compared to only one Summary student, realized that they had been thinking in terms of correlation rather than causation. For example, one student realized he did not know whether mitochondria increase ATP resynthesis or vice versa, and another student realized that there must be causality in both directions, though on different time scales. Another difference was that 75% of the Betty students realized they did not have a strong grasp of how much things changed. For example, one student, when teaching Betty that oxygen increases ATP resynthesis and lactic acid inhibits it, had to consider whether these two factors cancelled each other out. This type of quantitative reasoning is what one would expect by asking students to design a simulation, though in this case, students had to design a qualitative simulation and make decisions about big and little changes instead of numerical estimates.
14 As a simple learning assessment, we gave students a sheet with five terms: mitochondria, lactic acid, oxygen, aerobic fitness, and ATP resynthesis. For each term, we asked students to “list relations to other entities or processes in cellular metabolism.” Students in the Summary condition tended to assert single relations; for example, “mitochondria increase ATP resynthesis.” The Betty students tended to assert chains of two or more relations; for example, “mitochondria with glycogen or free fatty acid increase ATP resynthesis.” On average, the Betty students produced 3.75 chains of two or more relations throughout their lists, which is significantly greater than the students in the Summary condition, who only averaged 1.0 chain of multiple relations. Overall, the students who taught Betty exhibited several markers that Betty’s animations of thought had influenced their own knowledge. While teaching Betty, the students became aware that they had not sufficiently differentiated between causation and correlation when reading the passage. In contrast, writing a summary draws attention to topic sentences and paragraphing. Teaching Betty also influenced how students structured their own knowledge. When asked to list relations, the Betty students tended to list complex causal pathways compared to the Summary students. These results make sense because developing chains of causal relations is exactly what Betty requires of students, and the results demonstrate that co-mingling thoughts with Betty can shape student domain knowledge. 3.2 The Benefit of Animated Thought While the preceding study shows that students adopt Betty’s representations when they co-mingle ideas with her, it does not isolate the value of animating thought per se. Simply asking students to draw a concept map might have worked as well. Therefore, we
15 conducted a second study to see if Betty’s ability to animate her thoughts made a difference. We asked 5th-grade students to teach Betty about river ecosystems across three one-hour sessions. Students had online resources to help them learn the relevant content to teach Betty (see Biswas et al., 2004). The two conditions of most relevance controlled whether students could see Betty animate her thought. Some students could ask Betty questions and activate her animations, whereas other students simply made Betty’s concept map without ever seeing it animate in response to questions. Thus, both conditions co-mingled their ideas with Betty, with the difference being whether the system enabled the shared initiative of students asking questions and Betty drawing visible inferences. 20
No Animation
Number of Causal Links
Animation 15
10
5
0
Day 1
Day 2
Day 3
Figure 3. Students who see Betty animate her reasoning add more causal links. Figure 3 shows that students in the Animation condition generated maps that included increasingly more causal links over the days. This makes sense because this is what gets animated in Betty’s reasoning when she answers questions. Evidently,
16 animations of thought make a difference. This is a useful finding, because Tversky, Morrison, and Betrancourt (2002) found that static diagrams can be as effective as animations. However, their work did not examine the interactive potential of animations, which in the current study was more useful than static drawings. 4.0 The Agent’s Independent Performance Supports Learning In the preceding two studies, we showed that student thoughts and representations were influenced by Betty’s animation of chaining through her map. We argued that this incorporation depends on the students’ chances to co-mingle their ideas with the agent’s representations plus see the agent animate its reasoning with the co-mingled representation. However, these studies did not sufficiently isolate the value of the agent’s initiative. For example, one might argue that the critical learning ingredient was simply the opportunity to see an animation, not the independent performance of the agent over a co-mingled representation. The remaining studies more directly target the importance of the TA’s independent performance. One-way Interaction s1
S
Tell
O
o1
S Listen O
Face-to-Face Exchange
Initiative in Secondary Context s1
s2(o1(s1))) s1
S Tutor o1(s1)
O
S
Coach
o1(s1)
O
Perform
C
c1(o1(s1))
Figure 4. Some models of initiative and incorporation. S = self, O = other. Arrows indicate initiative. Letters on arcs indicate the incorporation of ideas. Incorporation is recursive so that ideas can be transformed across multiple interactions. Italicized entries offer exemplars of each interaction model. Figure 4 provides a schematic notation for modeling initiative and incorporation. The left panel shows examples where a single participant predominates. The other panels
17 suggest two ways to enhance the initiative of an agent to improve student learning. In the middle panel, the agent introduces a topic to the student, and the student responds in turn. In the next section, we describe studies relevant to this form of interactive exchange. In the current section, we address the model in the third panel. Here, the TA takes initiative in a secondary context where it performs based on how it was taught. The value of this model can be motivated by dissertation defenses in Sweden. In Sweden, the doctoral candidate does not answer questions directly. Rather, there is an advocate who answers questions based on what was written in the thesis. The advocate’s ability to defend the thesis depends on the candidate, but the advocate has the initiative to decide what to say. We assume this situation leads to much more careful thinking and writing by the candidate. When people teach somebody else who has to perform, they cannot count on their own reactive abilities and “situational smarts” to generate answers on the fly or sidestep challenges as they arise. They need to formalize their knowledge in a clear and unambiguous way. Moreover, by seeing how the other person performs, they receive projective feedback about their knowledge as incorporated into the other person. Our proposal is that we can garner these types of benefits with TAs when they perform in a secondary context. 4.1 Initiative in Social Contexts The agents that we design can be extended (for example, to animate temporal cycles), and they can be combined with other applications. In the following study, we combined Betty with a quiz system to give her additional initiative. The quiz system asks Betty a question and generates feedback on her answer. This permits the agent to demonstrate initiative on the quiz while providing important feedback to the student. We
18 additionally developed the quiz system so it could display and test several agents simultaneously. This work arose from our initial user study where we noticed that the students were quite attentive to other people’s agents when they were shown at the front of the class. We thought that one component of this attentiveness was attributable to the public performance of the agent. Ideally, this public behavior can increase student motivation to have their agent perform well, plus it offers an opportunity for students to see other agents’ maps.
Figure 5. Front of class quiz system for showing agents perform. Figure 5 shows the front-of-the-class (FOC) testing environment that we can project on a large screen. Each panel shows one of the agents created by the students. The classroom teacher can ask a question of all the agents simultaneously. A hidden expert map determines the correct answer to a given question. The results are tabulated and indicated by color coding (red = incorrect; green = correct; yellow = correct for wrong reason). The classroom teacher can zoom in on any Betty to see why she gave the answer she did, and then compare it to another agent.
19 If we shed the TA metaphor, one way to think of the FOC system is that students are communicating their knowledge by creating executable models, and then the models get tested. So, rather than the student answering a small subset of questions, the student needs to create a model that can answer any legitimate question in the domain. Combined with the opportunity to see other students’ agents, we thought the FOC might have some benefits for learning. In particular, we believed that it would help students think about the structure of the domain rather than the answers to a specific set of questions. We conducted a study with 18 college students who had completed a course on the transfer of learning. We wanted to see if the agent’s initiative in answering quiz questions would lead students to improve their agent. We also wanted to see if there were additional benefits when the agents were shown performing publicly at the front of the class. Each student received a Betty that already included a set of nodes relevant to transfer (e.g., “expertise,” “positive transfer,” “analogy”). Their task was to link the nodes so Betty could answer questions about transfer (e.g., “If abstraction increases, what happens to negative transfer?”). After students taught their agent to their satisfaction, they were separated into two conditions. In both conditions, the agents took a three question quiz about nodes that were directly linked (as opposed to nodes that had a number of mediating nodes and links between them), and the system indicated if their answers were right or wrong. In the FOC condition, students “submitted” their agents to the FOC system, and they saw nine agents simultaneously answer the questions. The teacher also zoomed in on a few different agents. In the Solitary condition, the agents took the same quiz and students received feedback on their agent’s performance, but there was no
20 public display of agents. Afterwards, both conditions had a chance to revise their TA, and they resubmitted their agents after revision. We expected the independent performance on the quiz to lead students from both conditions to improve their TA given a second chance. However, we thought the students in the FOC condition would make more extensive revisions. To find out, we compared the students’ initial maps and their revised maps once they had left. Our assessment of the two sets of maps was different from what the students experienced. We reconfigured the question asking system to ask every answerable question based on the hidden expert map. The system took one node, for example, “abstraction,” and asked each agent what would happen to each of the other nodes if abstraction increased or decreased. It compared the agents’ answers with the expert map. It then took another node and did the same thing. In other words, the software automatically asked the complete population of questions of each agent, rather than sampling a select few hoping to infer what the student or agent knew about the domain as a whole. The results of this exhaustive evaluation indicated that the revised maps from the students in both conditions improved compared to their initial maps. Evidently, seeing their agents perform on the three quiz questions lead all the students to improve their maps considerably. Notably, the FOC students’ revised maps were superior to the Solitary students (75% versus 63% correct). Of particular note, the FOC students exhibited their greatest relative gain on complex questions that involved multiple links. Figure 6 shows the improvement broken out by the number of links involved in the question. Students in the Solitary condition tended to remediate links associated with the
21 specific quiz questions, whereas the FOC students tended to fix the larger structure of their maps. Gain Scores after Feedback and Revision Front-of-Class
Increase in Correct Answers
25%
Solitary 20%
15%
10%
5%
0%
One
Two
Three-Four
Seven-Eight
Causal Links Needed to Answer Question
Figure 6. Changes to the quality of Betty maps when assessed by asking all possible questions. Students in both conditions improved their maps after seeing Betty answer a few quiz questions. Students who saw their agents at the front of the class improved the overall structure of their map more. These results are based on a small sample, and they require further research. At this point, we know that seeing Betty answer quiz questions led all students to improve their maps, but for the FOC condition, we cannot separate the benefit of seeing the agent’s public initiative from the benefit of seeing other agents at the front of the class. In the meantime, the positive results have encouraged us to examine further ways to use social performance contexts to improve student learning. For example, students can each teach their agent a portion of a domain. Students can then “jigsaw” (Aronson, 1978) their TAs so they merge their knowledge maps into a Team Betty (Sears & Schwartz, 2004). One recent application, developed with Paula Wellings and Elliot Castillo, builds on the FOC system to design a new model for homework. In the Triple-A Game Show students teach their agent and customize its look. Their agents then participate in a game show with other students’ agents on-line (see Figure 7). Students can log on from home
22 or school. The game host asks agents to answer questions and show their thinking. The application also includes a chat environment so students can discuss and cheer (or jeer) for an agent’s performance. Our hope is that students will find this socially rich environment, plus their agent’s independent performance, both engaging and educative, and it will prepare them for their lessons in school the next day or week.
It decreases, Bob. Figure 7. A customized agent performs in an on-line game show with other agents. 4.2 Initiative in Behavioral Environments For the preceding studies, we have argued that it is valuable for learning if students get to see their own knowledge co-mingled with an agent who also takes initiative by answering questions independently through its animation. However, these studies did not sufficiently isolate the value of the independent behavior of the agent. So far, one might argue that it was not the independent performance of the agent that made a difference. Rather, it was simply the opportunity for students to work with representations, ask questions, and receive corrective feedback. For example, if students were asked to answer questions themselves after making a concept map, and then they
23 received corrective feedback (without the presence of an agent), they might have generated concepts maps of equal quality. A second concern with the studies so far is that we have evaluated the quality of the students’ maps, and this may not reveal how much students actually learned. A person’s ability to perform in a supportive context with an animation does not mean that the person has learned sufficiently well to perform when the supportive environment is no longer available. The next study addresses both of these concerns. To conduct the study, we used a TA called Good Moby. Good Moby reasons over situations rather than verbal questions. This TA environment is more complicated than Betty, because it includes situations, representations for teaching, Good Moby who uses the representations to reason about the situations, and a competing agent called Evil Moby. The Moby environment was designed to help students learn science through a process of hypothesis induction and testing. Like the logic visualization tools of Barwise and Etchemendy (1994), Moby uses a visual interface to help students reason about the relations between logical claims and material circumstances. Moby is not so much a data visualization tool (Gordin & Pea, 1995), as it is a hypothesis visualization tool – it helps visualize thought. We wanted to see if hypothesis visualization coupled with the TA paradigm could improve scientific reasoning. Scientific reasoning is notoriously problematic (e.g., Kuhn, 1995), so scaffolds that not only support reasoning but also improve performance once the scaffolds disappear would be a positive outcome. Thus, in this study we emphasize reasoning, though our ultimate goal is to use Moby to teach science content.
24
Overlay Water
Overlay Sun
Figure 8. Two examples of students overlaying a factor to induce the conditions that cause an outcome. To introduce the Moby environment, Figure 8 uses a simple example where the presence or absence of a flower is the outcome. There are four factors, shown in the right of each panel, that regulate the appearance of a flower: water, sun, shade, and fire. Students click on each of the factors to overlay them on the grid. Ideally, this visual interface can help students induce that for this situation, sun is necessary and sufficient for flowers (this example is illustrative not empirically true). The combinations of factors and their outcome can be expressed as a logical rule. The rules that regulate an outcome can include up to two factors, negations, conjunctions, disjunctions, necessary, sufficient, or necessary and sufficient. The rules can become quite difficult, for example, “fire or shade is sufficient but not necessary for the absence of a flower.” To find multi-factor rules, students can overlay up to two factors at a time. In Figure 9, checkerboard squares indicate the co-presence the two selected factors. After students believe they have induced a rule, they can play a prediction game against Evil Moby. In Figure 9, the student and Evil Moby are playing a game where they have to predict the presence and/or absence of the flowers. Students lay down the
25 factor(s) they hypothesize are responsible, and this guides their choice of locations to predict flowers. Students toggle whether they are predicting presence or absence of flowers, and then they click on a square to make a prediction. The game responds by indicating accuracy with smiling and frowning faces and by updating the scoreboard at the bottom. After each student prediction, Evil Moby makes a prediction. In Figure 9, Evil Moby has the wrong rule, and most of his predictions are incorrect. (The student also has a wrong rule.)
Figure 9. Students play against Evil Moby to see who can better predict outcomes. Students overlay the factors to guide their predictions. In this example, the student believes Water and Shade are involved and overlays both factors. Once students play against Evil Moby and satisfy themselves they have induced the correct rule, they can teach Good Moby. It is in the teaching phase where the students’ visual understanding of a rule becomes co-mingled with Good Moby’s formal representations. Figure 10 shows there are two general representations that students use
26 to teach Good Moby a specific rule. In the top of the figure, the students co-mingle their visual rule with Moby’s sentential representation. Using pull down menus, students choose factor(s), their combination, and the qualifier that governs their relation to the flowers (Necessary, Sufficient, Necessary and Sufficient). In the bottom of the figure, the students teach using Good Moby’s matrix representation. After choosing the factor(s), the students fill the cells with (A)lways, (S)ometimes, (N)ever, indicating whether flowers will appear in these cells. If students are inconsistent between the two representations, Good Moby asks them to try again.
Figure 10. Students teach Good Moby using sentential and tabular formats. After students teach Good Moby, he animates his thought. In this case, the animation consists of laying down factors and making predictions in a game against Evil Moby. The game uses the same interface that the students used when playing against Evil Moby. So, the animation is not of Good Moby’s thoughts per se, but rather the choices based on his thoughts, and the initiative involves Good Moby’s responses to situational circumstances. Students watch Good Moby and Evil Moby play, and if Good Moby loses
27 consistently, students need to re-teach their agent, and if necessary, return to the induction phase to develop a new hypothesis. In the present study, we were specifically interested whether the independent performance of Good Moby through his animated choices would have a significant influence on high school student learning. To test this, we separated 94 students into four conditions. In the Control condition, students never used the software and simply took a logic posttest. For the other three conditions, students played the game for about 90 minutes, and they progressed through a series of increasingly difficult rules. They had to beat Evil Moby twice in a row to progress to the next rule level. In the Teach condition, students completed the full cycle (i.e., play themselves, teach Good Moby, watch Good Moby play). Moby had to beat Evil Moby twice to progress to the next level. In the Represent condition, students played the game themselves, and after beating Evil Moby twice, they used the pull-down menus to describe their hypothesis and moved to the next rule level. In other words, they co-mingled their visual understanding with the formal representations exactly like the Teach condition, but the resulting representations were only used to formalize their hypotheses. They were not used to teach Good Moby, and there was no independent performance of an agent. Finally, in the Explain condition, students also played the game themselves. After beating Evil Moby, a text window appeared asking them to explain the rule they used. They did not see the sentential or matrix representations and simply had to find a way to express the rule in words. Thus, these students neither co-mingled their knowledge with an agent nor did they see any agent initiative. If the initiative of the agent to perform independently is a valuable aspect of animations of thought, then we should expect the students in the Teach condition to do
28 the best on the posttest, even though students in the Represent and Explain conditions would have to induce, use, and formulate rules too. The students in all three software conditions found the program engaging and worked steadily. They reached about the same rule level in the same amount of time with no significant differences. A few days later, students took an 18 question posttest with three classes of questions that used one- and two-factor problems of various logical rules. Induction questions asked students to infer a rule given combinations of factors and outcomes. Implication questions provided a rule and students had to draw logical implications. Implication problems require hypothetico-deductive reasoning to predict outcomes based on a rule, and included, for example, a version of the notoriously difficult selection task (Wason & Johnson-Laird, 1970) in which people need to choose an empirical test to check if a rule is being followed. (Mary claims cockroaches need either warm or damp weather. Where would you check to see if Mary is right? The correct answer is cold and dry places.) Finally, translation problems asked students to translate between sentential and tabular expressions of a logical rule.
29 Figure 11. Students in the Teach condition were significantly more accurate on all question types. Figure 11 shows the average percent of correct answers for each question type. The Teach condition led to significantly better performance across all three types of measures compared to each of the other conditions, whereas none of the other conditions were significantly different from one another. There was no condition by question type interaction (though the translation problems were harder overall). The Teach condition showed a consistent advantage across all of the problems. The fact that the Teach students did better than the students who played without teaching is an important result. It shows that seeing a TA perform with feedback is more valuable than just doing the performance oneself with feedback. Moreover, it showed that agent initiative is important for cashing in the value of co-mingled representations. Students in the Represent condition co-mingled their ideas with the same representations as the Teach students did. Even so, the Represent students did not reap the benefits of this co-mingling; the Express students, who never saw these representations, did about the same as the Represent students. The learning benefits of co-mingling depended on having an agent that had the initiative to perform independently. Evidently, there is some advantage to seeing the implications of one’s co-mingled representations when they are animated independently of oneself. These results suggest one possible learning benefit from programming simulations in general, and not just TAs. Simulations require a full specification of one’s knowledge. The “run” of the simulation then provides independent performance and feedback that is not prone to subtle, situation-specific reasoning that can yield successful
30 results but for the wrong reasons. To our knowledge, there has not been a direct test that compares the benefit of a simulation versus an otherwise equivalent activity of formalizing one’s knowledge, so the current results may have some value beyond the specific demonstration of the value of TA’s. 5.0 Agent Initiative in Interaction with the Student Thus far, TA initiative has been largely reactive, because it has been in direct response to the action of the student or the environment. Even so, for novices, the relatively simple teaching appears to yield complex, choice-filled behavior in the TA because the performance setting provides complexity. Simon (1996), for instance, postulates that the complicated behavior of the ant is actually based on a few simple rules that operate in a complex environment. Even so, the TA is still reactive. In this regard, our findings so far should generalize to non-agent environments that include co-mingling and shared initiative. In the next two examples, we capitalize more fully on the TA metaphor. In these examples, the agent sometimes initiates interactions about its learning so that the student has to make a decision about what to teach next. We demonstrate that there are specific learning benefits that can be targeted when agents adopt a face-to-face, exchange model of interaction. 5.1 Initiative in a Constructivist Agent For our first example, we describe a TA that animates its interpretations of what it “thinks” the student has taught. Agent interpretation differentiates teaching TAs from computer programming and other types of interactive environments. When programming, there is minimal belief that the program could have multiple interpretations of the code. However, an appropriately designed TA can exhibit different
31 possible interpretations. By presenting the different interpretations, the TA initiates an interaction where the student then chooses how to respond to help the agent. We call agents that generate multiple possible interpretations, constructivist agents, because they show that they are trying to build a representation of what they have been told. Our example is Milo. We created Milo to help students learn descriptive statistics, because we thought a constructivist TA could help students appreciate the importance of specificity in mathematics (Blair & Schwartz 2004). Usually, students evaluate the quality of a mathematical expression by whether it computes the right answer. However, Milo is designed to help students evaluate the quality of an expression based on its communicative clarity, an important component of mathematical understanding (Greeno & Hall, 1997). Moreover, by showing different possible interpretations, Milo alerts students to a lack of precision in their teaching (and their own thought). Students teach Milo (an elephant) using mathematical expressions. Students see a statistical distribution, and their task is to send Milo an expression that will help him approximately reproduce the distribution (students know that Milo cannot “see” the original distribution). For example, many students initially communicate a distribution by sending the range formula (Largest – Smallest) and the parameter it yields (e.g., 6). Once Milo receives the expression, he uses a genetic algorithm to generate a set of distributions consistent with the expression and selects three to show the student. The left side of Figure 12 shows that given a range of 6, Milo offers three possible interpretations that vary in shape, sample size, and mean. Students can compare Milo’s interpretations with the original distribution (shown on the right). These three distributions constitute the
32 “animation of thought” for Milo. Clearly, these distributions are not animations, but they do represent several different legitimate spatial transformations of a given referent (which would be difficult to do with a single, flowing animation).
Figure 12. Milo shows his different interpretations of what type of distribution fits a range of 6. The right side of the figure indicates how close Milo’s distributions are to the original. Given the lack of fit between what the student intended and what Milo thought, the student can revise. For example, the student might also communicate the mean. With more precise information, Milo’s interpretations become closer to the original distribution. Students continue until they are satisfied with Milo’s interpretation. The process continues with a new distribution, and Milo can play a guessing game with another agent (not described here). To evaluate whether Milo helped students learn about the value of specificity in mathematical expressions, 39 high-school calculus students worked with Milo for about
33 an hour. We wanted to know if the constructivist quality of Milo’s initiative would help students learn about issues of ambiguity and specificity in mathematical models. To evaluate the learning benefits of Milo, students answered two non-statistical questions, before and after using Milo. Students exhibited significant gains in their appreciation that multiple parameters reduce the ambiguity of a mathematical model. One question showed a picture of a 2x1 rectangle and asked students to create a code, such that a friend could guess what shape it was. A common pretest answer stated, “2 x 1.” A common posttest answer was, “4 sides, L = 2, W = 1, angles = 90o.” The second question asked why they might want to use multiple formulas to characterize a phenomenon. At pretest, a common answer was, “because one formula might be inaccurate, so you would want the other.” They were thinking in terms of error rather than ambiguity. At post-test, students were more likely to give answers that addressed multiple dimensions. For example, one student wrote, “…because each formula can describe a different aspect of the phenomenon that characterizes it.” They also noted that, “It limits the possible phenomena, so you don’t get a different phenomenon that fits the same formula.” Teaching Milo in statistics improved the students’ abilities to recognize that it is important for mathematical models to be communicatively precise – a recognition recognition that generalized to non-statistical problems. More generally, we think that developing an awareness of the thinking that underlies a visual or non-visual representation can be an important benefit of constructivist agents. In this case, students realized that their own thoughts were incomplete. A particularly direct application of a constructivist agent is for teaching teachers. It is a common teaching trap to assume that
34 novices will have the same interpretation that the teacher does. A TA that helps teachers appreciate the risk of misinterpretation, perhaps by exhibiting common misconceptions, could be quite useful. 5.2 A Mixed-Initiative Agent For the final example, we describe a version of Betty that has been modified to increase her initiative. As with Milo, the new Betty initiates an exchange about how she is being taught. Unlike Milo, who always initiates at the same time in the teaching interaction, Betty initiates an exchange at strategic points. In exit interviews from our previous Betty studies, the students often emphasized that they would have liked Betty to be more active and exhibit characteristics of a good student during the teaching phase. Several students suggested that Betty should be more interactive (in the sense of taking more initiative). For example, one student wanted Betty to “react to what she was being taught, and take the initiative and ask more questions on her own.” Our assumption is that the initiative demonstrated by agents that can interrupt willfully and with relevancy is a powerful instrument for learning. Tutors, for example, gain a deeper understanding when they have interactions with a tutee that include answering tutee questions, explaining materials, and discovering misconceptions (Chi et al., 2001; Graesser, Person, & Magliano, 1995; Uresti, 2000). For young children, an agent that initiates a discussion about its learning can be especially valuable. For example, if an agent models how it monitors its own thinking by asking questions, this may help students learn to perform this type of metacognition as well (Lin & Lehman, 1999; Palinscar & Brown, 1984). Shimoda, White, and Frederiksen (2002), for example, developed metacognitive coaches that students can modify to
35 provide hints for managing their scientific investigations, and Baylor (2002) found that agents can improve pre-service teachers’ metacognitive awareness. Table 1. Examples of Betty Taking Initiative. Student State of System Samples of Betty’s Dialog Action Student has not asked “I still do not feel prepared to take a quiz. I don't Tell Betty Betty a question since understand enough about the causal relationships in the river. Please ask me some causal questions to take a the last quiz. to see if I understand. Mr. Davis can help you learn quiz more about being a good teacher.”
Add a link to Betty’s concept map
Betty’s answers have changed since the last quiz First causal link in the session First causal path with two or more links
“What you have taught me has changed my thinking. I had some questions right on the quiz, but now I think I would answer them wrong.” “Hey! Let me see if I understand this.” (Betty reasons with link, and explains her reasoning) “OK. I think I know how this works.” (Betty reasons with path, and explains her reasoning)
To address these issues, we implemented a Betty system that enhanced her initiative on issues of metacognition (Biswas, Schwartz, Leelawong, Vye, & TAG-V, in press). She incorporated the metacognitive strategies of monitoring, assessing, goal setting, seeking assistance, and reflecting on feedback. Under specific conditions that comprise the current student action, Betty’s knowledge state, and the history of prior interactions, Betty can spontaneously offer a metacognitive strategy or concern. For example, as students build Betty’s concept map, she can occasionally start animating her map to draw inferences. She then remarks (right or wrong) that the answer she is deriving does not seem to make sense. These spontaneous prompts help students reflect on what they are teaching, and hopefully like a good teacher, check on their tutee’s learning progress (and their own). Sometimes Betty directly tells the students to ask her a question to ensure that she can reason correctly with the current concept map (students never have to respond to Betty, though they usually do). In the mixed-initiative system, Betty also
36 sometimes refuses to take a quiz, because she feels that she has not been taught enough or received enough practice answering questions. Betty also raises her quiz results indicating her pleasure or disappointment at her level of improvement compared to the prior quiz. To demonstrate how mixed-initiative Betty functions, Table 1 provides a subset of a few student actions, the system state, and the actions Betty initiates. !
Mixed-initiative Betty comes with a number of other assets including on-line
resources for learning about river ecology plus quiz sets. A new asset, to complement mixed-initiative Betty, is a mentor agent named, Mr. Davis. Mr. Davis helps to complete the teaching narrative, because he administers and grades Betty’s quizzes. Mr. Davis also provides a vehicle for delivering additional meta-cognitive tips. Mr. Davis is not mixedinitiative. Students have to ask Mr. Davis for help. He provides tips on being a more effective teacher and learner, but he does not give factual answers. For example, he might suggest which of the on-line resources is helpful for a particular concept, and he provides information on how to be a good teacher (e.g., “test Betty and examine her answers closely”) and how to be a good learner (e.g., “set goals”). To evaluate the benefits of the shared initiative environment, we conducted a study with fifth graders. The students worked for six 45-minute sessions over a period of three weeks to create concept maps about the oxygen cycle, food chain, and waste in river ecosystems. The study included three conditions. As described above, there was the Mixed-Initiative Teaching condition in which the student and agent shared initiative over Betty’s learning. Students were told they had to teach Betty so she could pass a test to become a member of a school science club. In the second condition, students also prepared Betty for the club test. In this Teaching condition, Betty was similar to the prior
37 studies where she could take a quiz and answer questions with her map. She did not initiate interactions, and in this condition, Mr. Davis did not provide tips on teaching and learning. Instead, he provided feedback to Betty on each quiz question. Finally, in the Being-Taught condition, there was no cover story of teaching an agent and Betty was not present in this environment. Mr. Davis directed the students to construct concept maps to demonstrate their learning. They were told to use the quiz questions as a guide for what to learn. Students could ask Mr. Davis if their concept map was correct based on a quiz question. Mr. Davis would tell the student the correct answer and provide directive feedback for how to correct the map. Thus, the Being-Taught condition modeled standard computer-based instruction, where the computer has the initiative and teaches and tests the student. After students completed the six-sessions, we asked them to draw their concept maps from memory. We analyzed the quality of the maps by comparing them against an expert map. (None of the students saw the expert map). By this posttest, the three conditions looked about the same. This result was not a surprise, because the students had worked for a long time developing their maps in each condition. This was not where we expected to find the difference between the conditions. Our hypothesis was that the Mixed-Initiative condition would show its benefits later on a transfer task. In particular, we thought the metacognitive emphasis would prepare students to transfer to learn about a new, related topic (Bransford & Schwartz, 1999). Because the Mixed-Initiative students had exposure and encouragement to use metacognitive strategies, we thought they might apply these strategies when learning new
38 content. In other words, we thought that leveraging Betty’s mixed-initiative capacity to target metacognitive outcomes would influence students’ future abilities to learn. A month after the completion of the instructional intervention, students were asked to construct a concept map and answer questions about the land-based nitrogen cycle. Students had not been taught about the nitrogen cycle, so they would have to learn from resources during the transfer phase. Students completed this “learning at transfer” task in a modified Teaching environment. There were on-line resources, a quiz feature, and the mentor agent only said whether the answers to the quiz were right or wrong. There was no directive feedback, no shared initiative, and no metacognitive guidance. Table 2 shows that students who were originally in the Mixed-Initiative condition learned significantly more about the nitrogen cycle as reflected in their concept maps. Our assumption is that students in the Teaching and Being-Taught conditions had received correct answers during their initial learning of river ecology without much interactive exchange with Betty, and therefore, they had not learned how to use resources to support their learning or how to monitor their understanding. In contrast, in the MixedInitiative condition Betty modeled meta-cognitive strategies. In the Mixed-Initiative system, initiation and incorporation were balanced between the agent and student, and this led to significant gains in abilities to learn, even several weeks later. Table 2: Quality of Student Maps when Learning about Nitrogen Cycle at Transfer. Mixed-Initiative Teaching Being Taught Student Maps Included: M (SD) M (SD) M (SD) a Expert Concepts 6.1 (0.6) 5.2 (0.5) 4.1 (0.6) ab Expert Causal Links 1.1 (0.3) 0.1 (0.3) 0.2 (0.3) Note: a Significantly greater than Being Taught; b Significantly greater than Teaching.
39 6.0 Conclusions Teachable agents have been designed to provide a certain kind of animation that helps form student learning. Each agent relies on a specific visual formalism and an underlying artificial intelligence engine that can reason about specific types of relations. Betty, for example, makes her qualitative reasoning visible through a dynamic, directed graph that models causal relations. The fact that TAs represent knowledge structures rather than the referent domain is a departure from many simulations and animations. Animations often show the behavior of a physical system, for example, the behavior of pulleys. TAs, however, simulate the behavior of a person’s thoughts about a system. This is important because the goal of learning is often to simulate an expert’s reasoning processes about a domain, not the domain itself. Learning brute empirical facts is important, but learning to think with the expert’s organization of those facts is equally important. Therefore, we have structured the agents to animate particular formalisms of thought that help students structure their own thinking. In this article we have demonstrated five important qualities of TAs. One quality, mentioned above, is the visualizations and animations that help students to abstract useful ways to think about a domain. The second quality is that they draw upon the well-known schema of teaching. This helps students quickly engage and benefit from otherwise complex computer interaction models. Our TAs are particularly thin in human-like appearance and behavior, but it never ceases to surprise us how readily children (and adults) are willing to adopt and are motivated by the fiction of teaching another person (Reeves & Nass, 1996).
40
Figure 13. Summarized studies placed in the interaction space of incorporation by initiative. In each study that made a comparison along the initiative dimension, students who received the treatment closer to shared initiative learned more. Also, in each study that made a comparison along the incorporation dimension, students who received the treatment closer to co-mingling learned more when there was a shared initiative.
41 The third and fourth qualities emphasize the interactivity of the agents. We described a joint interactive space characterized by how much the student and the agent incorporate and reflect one another’s ideas and by who initiates the actions. In the studies reported here, we demonstrated that for early conceptual learning, the ideal balance involves the co-mingling of ideas plus shared initiative in actions. Figure 13 provides a summary of the studies and the relative positions of the various conditions in the interactive space. For the quality of incorporation, by design the agents support the co-mingling of representations in an interactive context. Students provide domain specific content, while the agent provides general representations and reasoning. The first study, summarized in the upper-left corner of the figure, demonstrated that this co-mingling helped students learn the representational structures and reasoning methods of the agent. For the fourth quality of shared initiative, the TAs permit different levels of initiative on the part of the student and the agent. Figure 13 shows how the remainder of the studies manipulated the balance of initiative, and how in each case, the version closer to shared initiative led to superior learning. For example, in the upper-right corner, we compared student performance when the agent could initiate a response to a question versus not, and we found that the animated reasoning of the agent led to superior outcomes. The benefits of balanced initiative were not simply the result of increased interaction with the learning content. For example, in the second row of the figure, students in all conditions interacted with the systems the same amount. However, when students saw the independent performance of the agent in a secondary context, they did better than when they only worked with the agent or when they only worked in the
42 secondary context themselves. The lower-right corner of the figure also indicates the importance of balanced initiative by showing what happens when the system takes too much initiative. Students in the Being-Taught condition were directed by the coach, and as a result, they were not as prepared to learn as students who worked with an agent that initiated a discussion about the quality of its knowledge. This reflects our belief that systems that provide students complete initiative (e.g., drawing packages) and systems that own all the initiative (e.g., drill and practice) are not ideal for early learning. In all our studies, balancing agency between student and system led to better learning. The fifth and final quality, which we tried to demonstrate rather than explain, is that the teachable agent paradigm provides a felicitous and generative metaphor for creating learning environments and for recruiting learning behaviors and attitudes (e.g., willingness to revise). This can be especially useful for those who design instruction. In our work, we have not tried to isolate or over-emphasize a single learning behavior, like self-explanation or visualization. Instead, we try to make environments that recruit multiple powerful learning resources. The TA paradigm is new, and we have yet to combine our different techniques into a single instance. We are working in that direction, for example, by making a videogame where students teach their agents to solve challenges. Many people are attracted to the prospect of educational videogames for their motivational value. Motivation is very important, but videogames also provide a natural way to incorporate many forms of learning interactions that can include probing domain simulations, meeting other knowledgeable characters, and assessing one’s knowledge (e.g., to pass levels). Moreover, videogames can garner the potential benefits of embodied partners that can elicit affective responses and use non-verbal communication
43 forms (e.g, pointing, Rickel & Johnson, 1998; Tepper, Kopp, & Cassell, 2004) and social schemas useful for learning (e.g., responsibility). Animations of thought and their thinkers, when viewed from an interactive perspective, offer an exciting range of new possibilities for improving learning.
7.0 Acknowledgments The work was supported by NSF Grants…. 8.0 References Aronson, E. (1978). The Jigsaw Classroom. Oxford, UK: Sage. Bailenson, J., & Yee, N. (in press). Digital Chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological Science. Bargh, J.A., & Schul, Y. (1980). On the cognitive benefits of teaching. Journal of Educational Psychology, 72, 593-604. Barron, B.J. (2003). When smart groups fail. Journal of the Learning Sciences,12, 307359. Barwise, J., & Etchemendy, J. (1994). Hyperproof. Stanford, CA: Center for the Study of Language and Information. Baylor, A. L. (2002). Expanding preservice teachers’ metacognitive awareness of instructional planning through pedagogical agents. Educational Technology Research & Design, 50(2), 5-22. Bernieri, F., & Rosenthal, R. (1991). Interpersonal coordination: Behavior matching and interactional synchrony. In R. S. Feldman & B. Rime (Eds.), Fundamentals of nonverbal behavior (pp. 401-432). Cambridge, UK: Cambridge University Press. Biswas, G., Leelawong, K., Belynne, K., Viswanath, K., Vye, N. J., Schwartz, D. L., & Davis, J. (2004). Incorporating Self Regulated Learning Techniques into Learning by Teaching Environments. 26th Annual Meeting of the Cognitive Science Society. Chicago, Illinois, pp. 120-125. Biswas, G., Schwartz, D. L., Bransford, J. D., & TAG-V. (2001). Technology support for complex problem solving: From SAD Environments to AI. In K. Forbus & P. Feltovich (Eds.), Smart machines in education (pp. 71-98). Menlo Park, CA: AAAI/MIT Press.
44 Biswas, G., Schwartz, D. L., Leelawong, K., Vye, N., & TAG-V. (in press). Learning by teaching: A new paradigm for educational software. Applied Artificial Intelligence, 19(3). Blair, K. P., and Schwartz, D. L. (2004) Milo and J-Mole: Computers as constructivist teachable agents. In Y. Kafai et al. (Eds.) Embracing Diversity in the Learning Sciences: The proceedings of the Sixth International Conference of the Learning Sciences (p. 588). Mahwah, NJ: Erlbaum Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of Research in Education , 24, 61-101. Washington DC: American Educational Research Association. Brennan, S. E. (1966). Lexical entrainment in spontaneous dialog. Proceedings, 1996 International Symposium on Spoken Dialogue, 41-44. Cassell, J., (2004). Towards a model of technology and literacy development: Story listening systems. Journal of Applied Developmental Psychology, 25, 75-105. Chan, T.-W., & Chou, C.-Y. (1997). Exploring the Design of Computer Supports for Reciprocal Tutoring. International Journal of Artificial Intelligence in Education, 8, 1-29. Chi, M. T. H., De Leeuw, N., Mei-Hung, C., & Levancher, C. (1994). Eliciting self explanations. Cognitive Science, 18, 439-477. Chi, M.T.H., Siler, S. A., Jeong, H., Yamauchi, T., & Hausmann, R. G. (2001). Learning from human tutoring. Cognitive Science, 25, 471-533. Chmeilewski, T., & Dansereau, D. (1998). Enhancing the recall of text: Knowledge mapping training promotes implicit transfer. Journal of Educational Psychology, 90, 407-413. Cohen, P. A., Kulik, J. A., & Kulik, C.-L. C. (1982). Educational outcomes of peer tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237-248. Gordin, D. N., & Pea, R. D. (1995). Prospects for scientific visualization as an educational technology. Journal of the Learning Sciences, 4, 249-280. Graesser, A.C., Person, N., & Magliano, J. (1995). Collaborative dialog patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychologist, 1995. 9, 359-387. Greeno, J. G., & Hall, R. P. (1997). Practicing representation learning with and about representational forms. Phi Delta Kappan, 78, 361-367. Kafai, Y., & Harel, I. (1999). Learning through design and teaching: Exploring social and collaborative aspects of constructionism. In I. Harel & S. Papert (Eds.), Constructionism. Norwood, NJ: Ablex. Kinchin, I.M. and D.B. Hay. (2000). How a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development. Educational Research, 42(1), 43-57.
45 Kuhn, D. (1995). Scientific thinking and knowledge acquisition. Monographs of the Society for Research in Child Development, 60(4), 152-157. Lin, X.D., & Lehman, J. (1999).! Supporting learning of variable control in a computer based biology environment: effects of prompting college students to reflect on their own thinking. Journal of Research In Science Teaching, 36, 837-858. Novak, J.D. (1996). Concept Mapping as a tool for improving science teaching and learning. In D.F. Treagust, R. Duit, and B.J. Fraser, (Eds.) Improving Teaching and Learning in Science and Mathematics (pp. 32-43). London: Teachers College Press. Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension monitoring activities. Cognition and Instruction, 1, 117-175. Papert, S. (1980). Mindstorms: Computers, children, and powerful ideas. NY: Basic. Pletzner, R., Dillenbourg, P., Preier, M., & Traum, D. (1999). Learning by explainin to oneself and others. In P. Dillenbourg (Ed.), Collaborative learning: Cognitive and computational approaches (pp. 103-121). Amsterdam: Pergamon. Premack, D. (2003). Is language the key to human intelligence. Science, 303, 318-320. Reeves, B. and Nass C. (1966). The Media Equation: How people treat computers, televisons and new media like real people and places. Cambridge, UK: Cambridge University Press. Repenning, A., & Sumner, T. (1995). Agentsheets: A medium for creating domain oriented visual languages. Computer, 28, 17-25. Rickel, J. and Johnson, W.L., (1998). STEVE: A Pedagogical Agent for Virtual Reality (video). In Proceedings of the Second International Conference on Autonomous Agents, Minneapolis/St. Paul: ACM Press. Schwartz, D. L. (1995). The emergence of abstraction in dyad problem solving. Journal of the Learning Sciences. Schwartz, D. L. (1999). The productive agency that drives collaborative learning. In P. Dillenbourg (Ed.), Collaborative learning: Cognitive and computational approaches (pp. 197-218). New York: Elsevier Science. Sears, D., & Schwartz, D. L. (April, 2004). Risks and Rewards of Jigsawing Teachable Agents. Paper presentation at AERA. San Diego. Sherin, M. (2002). When teaching becomes learning. Cognition & Instruction, 20, 119150. Shimoda, T. A., White, B., Frederisken, J. (2002). Student goal orientation in learning inquiry skills with modifiable software advisors. Science Education, 86, 244-263. Simon, H. A. (1996). Sciences of the Artificial, 3rd Edition. Cambridge, MA: MIT Press. Smith, D.C., Cypher, A., & Spohrer, J. (1997). Programming agents without a programming language. In J. M. Bradshaw (Ed.) Software Agents (pp. 165-190). Menlo Park, CA: AAAI/MIT Press.
46 Stoyanov, S., & Kommers, P. (1999). Agent-Support for Problem Solving Through Concept-Mapping. Journal of Interactive Learning Research, 10, 401-42. Taylor, R. (Ed.). (1980). The computer in school: Tutor, tool, tutee. NY: Teachers College Press. Tepper, P., Kopp, S., Cassell, J. (2004). Content in Context: Generating Language and Iconic Gesture without a Gestionary. Proceedings of the Workshop on Balanced Perception and Action in ECAs at AAMAS. Tversky, B., Morrison, J. B., & Betrancourt, M. (2002). Animation: Can it facilitate? International Journal of Human-Computer Studies, 57, 247-262. Uretsi, J. A. R. (2000). Should I teach my computer peer? Some issues in teaching a learning companion. In G. Gautheir, C. Frasson, & K. VanLehn (Eds.). Intelligent Tutoring Systems (pp. 103-112). Berlin: Springer-Verlag. Wason, P. C., & Johnson-Laird, P. N. (1970). A conflict between selecting and evaluating information in an inferential task. British Journal of Psychology, 61, 509515. Webb, N. M. (1983). Predicting learning from student interaction: Defining the interaction variables. Educational Psychologist, 18, 33-41. Willis, J., & Crowder, J. (1974). Does tutoring enhance the tutor’s academic learning? Psychology in the Schools, 11, 68-70.
