A Case Study of Instruction from Experts - Old City Publishing

© 2009 Old City Publishing, Inc. Published by license under the OCP Science imprint, a member of the Old City Publishing Group

Tech., Inst., Cognition and Learning, Vol. 7, pp. 103–120 Reprints available directly from the publisher Photocopying permitted by license only

A Case Study of Instruction from Experts: Why does Cognitive Task Analysis Make a Difference?1 David F. Feldon1,* and Kirk Stowe2

2

1 University of Virginia University of South Carolina

Experts are frequently called upon to serve as instructors in their disciplines. However, studies indicate that their unaided explanations contain significant inaccuracies and omissions that negatively impact the effectiveness of instruction. Cognitive task analysis (CTA) is an effective tool for eliciting, analyzing, and representing expert knowledge in a more accurate and complete manner. CTA-based instruction is consistently found to be more effective than unaided instructional explanations provided by experts. However, prior studies of the efficacy of CTA-based instruction have included a confound between the source of content (CTA) and the instructional design decisions made by designers. The current study utilizes content analysis of the instruction from the CTA and non-CTA conditions of an ongoing experment to assess the relative contributions of instructional content and instructional design to the overall effectiveness of CTA-based instruction. Results indicate that CTA’s ability to elicit more specificity in instructionally relevant information accounts for significantly more variance than the instructional design decisions related to representing instructional content in procedural or conceptual formats. Keywords: expertise; cognitive task analysis; instructional design.

The work reported in this paper is supported in part by a grant of the National Science Foundation (NSF-0653160) to the authors. The views expressed are those of the authors and do not necessarily represent the views of the supporting funding agency. 1

Corresponding author: [email protected]

*

103

103-120 pp TICL_003_Feldon.indd 103

11/23/2009 3:19:59 PM

104

Feldon and Stowe

Introduction Experts’ roles in education and training are multi-faceted. They are called upon to identify which domain competencies should be expected of students and to model effective problem-solving practices. Often, in university and professional settings, they are also expected to train learners through various instructional formats—most notably, in seminars or course lectures. However, previous research indicates that experts are frequently inaccurate when describing the processes they use to solve problems in their domains (see Feldon [2007] for a review). During self-reports of their problem-solving, they frequently omit essential steps and decision points in their procedures that contribute directly to the solutions they obtain (e.g., Cooke & Breedin, 1994; Chao & Salvendy, 1994; Feldon, in press). There are even documented cases where experts have knowingly fabricated problem-solving procedures to teach their students due to an inability to articulate how they actually perform their problem solving in practice (e.g., Johnson, 1983). To enhance the quality of content used for instruction, many instructional designers rely on cognitive task analysis (CTA) techniques to elicit more complete and accurate explanations from their subject matter experts (Schraagen, Chipman, & Shalin, 2000). Studies of CTA’s effectiveness as a basis for instruction consistently report very high effect sizes for learning outcomes when compared with traditional instruction (Clark, Feldon, van Merriënboer, Yates, & Early, 2008; Lee, 2004). Unfortunately, it is not yet clear exactly why CTAbased instruction is as effective as it is. In each study, the treatment condition has differed from comparison conditions because CTA provided the instructional content in the treatment condition. However, the instructional conditions also differ as a function of their instructional design (ID) in either or both of the following ways: (1) ID decisions regarding the sequencing of the instructional content and (2) the balance of content represented as procedural action steps and decision rules versus more conceptual explanations of target domain knowledge. The analyses presented here examine both the content and instructional design differences in one such CTA study. Exploratory content analysis identifies the differences in included content, its alignment with the assessment task, and the discrepant representation of procedural and conceptual explanations. The findings and their implications are discussed in relation to their apparent effects on learners’ skill acquisition based on the presented knowledge of experts.


11/23/2009 3:19:59 PM

CTA-Based Instruction

105

(In) Accuracy of Experts’ Self-Report Human cognitive architecture allows individuals to develop highly evolved efficiency mechanisms for processing information that is encountered frequently. Extensive prior knowledge of a domain allows experts to attend selectively to salient information in the environment while disregarding irrelevant stimuli (Barrett, Tugade, & Engle, 2004). This reductive approach optimizes the allocation of working memory resources to limit the processing of extraneous information. Further, the well-developed schemas characteristic of experts’ prior knowledge functionally expand their working memory capacity (Ericsson & Kintsch, 1995). This allows experts to attend to a much larger quantity of simultaneous information in working memory than non-experts. In addition, procedures which are rehearsed and performed frequently require decreasing levels of mental effort and conscious monitoring (Anderson, 1982; Moors & De Houwer, 2006). As a result, those problem-solving procedures that are the most central to an individual’s expertise are the least likely to occupy limited working memory resources. Ironically, the mechanisms that enhance performance also limit the ability of experts to fully articulate the strategies and procedures that they use to solve problems in their respective domains (Blessing & Anderson, 1996; Cooke, 1992; Feldon, 2007). Although experts recall specific events and problem states with a high level of accuracy (Gobet & Simon, 1998), self-reports of the mechanisms and strategies experts use to advance from one problem state to the next as they solve problems in their domains are highly inaccurate. Evidence suggests that the selective attention of experts occurs preconsciously, so they are unlikely to be able to directly articulate the basis for disregarding or responding to specific cues (Masunaga & Horn, 2000; Reingold, Charness, Schulteus, & Stampe, 2001). Likewise, well-developed schemas can have a reductive effect on explanations, where experts will articulate fewer elements of a scenario than those who are less proficient (Rikers, Schmidt, & Boshuizen, 2000) and be less likely to recognize and report salient elements that are atypical of their past experiences or theoretical preconceptions (Besnard, 2000; Nisbett & Wilson, 1977; Wigboldus, Dijksterhuis, & van Knippenberg, 2003; Wilson & Nisbett, 1978). Further, experts typically engage in “step-skipping behavior” (Koedinger & Anderson, 1990, p. 511), in which necessary steps for solving problems are omitted from the verbalization of their problem-solving processes due to the highly automated nature of their procedures. For example, Feldon (in press) found that expert experimental psychologists omitted or incorrectly reported an average of 75% of the steps they took to design


11/23/2009 3:19:59 PM

106

Feldon and Stowe

experiments and analyze the resulting data. Further, errors in their self-reports were significantly more likely to occur when explaining their reasoning processes that changed problem states than when recalling specific cues within individual problem states. Similarly, Chao and Salvendy (1994) reported that expert computer programmers were unable to accurately articulate more than 53% of the steps they took during debugging tasks. Even when procedures are articulated, they may not reflect the actual steps taken during the problem solving process. Cooke and Breedin (1994) asked expert mechanical physicists to predict the trajectories of various objects and explain their reasoning. When the authors then attempted to replicate the predictions using the explanations provided, the outcomes were completely uncorrelated with the original trajectory estimates. Likewise, Dunbar’s (2000) studies of in situ scientific problem-solving during laboratory meetings of eight prestigious microbiology research groups found that despite the successful generation of new findings and insights during the meetings, participants in the conversations were unable to accurately recall the processes by which the conclusions were reached.

Cognitive Task Analysis as a Means for Improving Accurate Information from Experts CTA is a collection of knowledge elicitation and analysis tools designed to enhance the quality of applied representations of expert knowledge. Although the efficacy and reliability of specific techniques and individual practitioners vary (Yates & Feldon, 2008), empirical studies have suggested that CTA can enhance the completeness of information obtained between 12% (Chao & Salvendy, 1994) and 43% (Crandall & Getchell-Reiter, 1993; see also Clark & Estes, 1996) when compared with unguided knowledge elicitation (i.e., think aloud and free recall). This difference has been replicated in instructional contexts, as well. In one study, instruction provided by three expert surgeons to residents in a surgical training program was recorded and analyzed. The surgeon-instructors then participated in CTA to identify the critical steps of the procedure they taught. The content of their instruction was subsequently compared to the procedure derived from the CTA. It was found that of the 26 steps in the surgical procedure, individual instructors articulated only 46%–61% of the CTA-obtained procedural steps during their teaching (Sullivan, Ortega, Wasserberg, Kaufman, Nyquist, & Clark, 2008). The benefits of CTA also translate successfully into instructional outcomes (Feldon, 2007). A 2004 meta-analysis found that effect sizes favoring CTA-


11/23/2009 3:19:59 PM


107

based instruction ranged from d = 0.91 to d = 2.25 with a mean of d = 1.72 across a variety of domains (Lee, 2004; see also Clark et al., 2008). These findings have been replicated in a number of domains, including radar troubleshooting (Schaafstal et al., 2000), spreadsheet use (Merrill, 2002), and medicine (Sullivan, Brown, Peyre, Salim, Martin, Towfigh, & Grunwald, 2007)2. Such results are not surprising, given the consistent finding that gaps in instructional content induce higher levels of cognitive load in learners (Chandler & Sweller, 1991; Kirschner, Sweller, & Clark, 2006; Sweller, Chandler, Tierney, & Cooper, 1990; Tuovinen & Sweller, 1999). With inherently challenging material, the additional (extraneous) cognitive load imposed by attempts to fill knowledge gaps while learning and during subsequent problem solving results in poorer acquisition and performance (Kirschner et al., 2006; van Merriënboer & Sweller, 2005). However, the assumption that this phenomenon offers the primary or only explanation for CTA’s instructional effectiveness has not been empirically tested.

Important Questions for Instructional Applications of CTA Despite the consistent success of CTA-based instructional methods, little is known about their underlying causal mechanisms. Most studies that compare a CTA-based instructional condition to alternative conditions have multiple uncontrolled variables that limit our ability to identify the causal mechanisms that generate the effect. Although the instructional objective and assessment mechanisms are the same across conditions, and the means of generating instructional content are controlled (CTA-derived content vs. unguided self-report/expert explanation), both the sequencing and representation of instructional content frequently differ as well. Merrill (2001) describes these elements as “micro decisions wherein a designer must select from all the knowledge that is available those specific components of knowledge and their sequence that will comprise the instructional materials” (p. 294). In the case of CTA, the selection and sequencing of knowledge pertains to its representation as well. Crandall, Klein, and Hoffman (2006) suggest that cognitive task analysis requires three independent elements—knowledge elicitation, analysis, and knowledge representation—because CTA’s utility depends on the The cited examples were not included in the Lee (2004) meta-analysis, but all report significant, positive effects of their CTA-based conditions on student outcomes. 2


11/23/2009 3:20:00 PM

108

Feldon and Stowe

interpretation of the elicited knowledge and its subsequent presentation in a useable form. Thus, designing instruction from CTA entails decision making about whether the content is most effectively presented in procedural or conceptual formats (Yates & Feldon, 2008). In some studies (e.g., Sullivan et al., 2007; Velmahos, Toutouzas, Sillin, Chan, Clark, Theodorou, & Maupin, 2007), the control/comparison condition entails an expert providing instruction of his own design, phrasing, and/or sequencing. In contrast, the treatment conditions present instruction that was assembled and planned by a formally trained instructional designer. Thus, resulting data analyses contrast not only CTA- with non-CTA-generated content, but also the designs of trained designers with those of subject matter experts who may lack formal ID training. In other studies, ID models vary concurrently with the source of subject matter (e.g., Merrill, 2002; Schaafstal et al., 2000). While these studies compare holistic ID models that do or do not entail CTA (for prominent examples that include CTA, see Clark et al., 2008, p. 587), they do not permit examination of the discrete contributions of the ID systems and the sources of instructional content, respectively. Procedural vs. Declarative Representations in Instruction Presenting instructional content in procedural or conceptual form can have significant impacts on the efficacy of training. Traditionally, procedural instruction is considered to be effective for learning algorithmic skills or skills applicable to solving problems in well-defined domains (Rogers, Maurer, Salas, & Fisk, 1997). In contrast, the traditional approach for training individuals to solve illstructured, open-ended (i.e., having multiple viable solution paths), and transfer problems is to teach conceptual knowledge to foster understanding of the principled, deep structure of the domain and productive analogies for applying knowledge across domains (Brown, Kane, & Long, 1989; Clark & Blake, 1997; Schwartz & Martin, 2004). It is often assumed that teaching procedural knowledge for the purpose of preparing learners to perform adaptively will lead to inappropriate rigidity based on automaticity or overly routinized behavior (e.g., Ericsson, 1998, 2004; Jonassen, 1997). However, a number of recent studies suggest that procedural instruction may be more effective for transfer than previously thought (e.g., Paas, 1992; Kalyuga, Chandler, Tuovinen, & Sweller, 2001; van Gog, Paas, & van Merriënboer, 2006). These studies have compared the effectiveness of traditional, conceptually-oriented instruction (i.e., explanation of theory accompanied by examples of completed problems) with a worked-example approach (procedural demon-


11/23/2009 3:20:00 PM


109

strations showing how to perform each necessary step in sequence). When participants solve practice problems and perform subsequent transfer tasks, instruction that provides the necessary procedural steps (i.e., worked examples) is significantly more effective on both measures. Further, evidence suggests that procedural instruction can further facilitate transfer by training learners to differentiate between deep and surface problem features using metacognitive and general strategic skills (Bransford, Brown, & Cocking, 1999; Lehman, Lampert, & Nisbett, 1988). Sternberg, Ferrari, Clinkenbeard, and Grigorenko (1996) and Halpern (1998) found that when learners received explicit training on metacognitive strategies such as planning and monitoring incremental progress toward a goal, they were more successful solving problems in a variety of domains.

The Current Study The analysis reported here takes a first step toward identifying the independent contributions of micro design features (sensu, Merrill 2001) and the source of content as “active ingredients” in CTA-based instruction (Clark et al., 2008, p. 589). Using the instructional materials from a larger, ongoing study of CTAbased training (see Feldon, Stowe, & Showman, 2009), a fine-grained content analysis examines the independent contributions of CTA as a source of instructional content and procedural/conceptual knowledge representations as a difference in instructional design3. Exploratory statistical analyses were used to assess the ability of these characteristics to explain the variance in the performance outcomes across instructional conditions.

Description of the Larger Study The larger study involved students from an introductory biology course in cellular, molecular, and genetic biology for undergraduates at a public, Tier-I university in the Southeast during the Spring semester of 2008. The final sample consisted of 252 participants (n = 133 in control condition; n = 119 in treatment condition). Students attended three hours of lecture on core content and participated in weekly, three-hour laboratory sections taught by graduate teaching assis Sequencing of instructional content was matched across conditions to eliminate it as a source of variance in the design. 3


11/23/2009 3:20:00 PM

110

Feldon and Stowe

tants (TAs). In addition to the lab activities themselves, students were required to log into a course-linked web site and watch brief instructional videos prior to attending class each week. These videos contained 5–10 minute lectures on how to perform the scientific process: making observations, formulating research questions, generating hypotheses, designing experiments and analyzing data. When students attended their assigned lab sections, they watched the week’s video again as a group and discussed it with the graduate student teaching the lab. Of the 13 laboratory sections, 7 were able to access videos of an expert cell biologist who has won multiple awards for university teaching delivering his best lectures on the facets of performing the scientific process in biology (the control condition). The remaining 6 sections likewise viewed videos of this instructor speaking on the same topics (the treatment condition), except instead, in these he was presenting from an instructional script derived from CTA interviews with three experimental biologists who are also recognized as experts in the field. The conditions for this study were double-blind. Neither the TAs nor their students knew that they were participating in the study and were unaware that multiple versions of the videos existed. Laboratory sections were randomly assigned to condition, either control or treatment, with the constraint that all sections taught by the same TA were in the same condition to prevent any diffusion of the treatment effect. Students and TAs were required to login using unique identifiers to view their respective videos, and server logs tracked the viewing behaviors of the participants. In order to encourage students to view the videos, course points were given for watching the weekly videos prior to attending class. A reminder email was sent each week to those students who neglected to view the assigned video outside of class. TAs incorporated the information from each week’s video into their day’s lesson to ensure integration of the videos into the overall course experience. In total, students viewed 10 videos over the course of the semester. Eight were part of the study, and two were condition-neutral to preserve video-watching as a regular part of participants’ weekly routines. We found significant differences favoring the treatment (CTA) condition in both course grades across lectures (76.03 [SD = 14.11] vs. 71.05 [SD = 22.23]; F = 5.085, p = .025) and performance on a written lab report from a Drosophila (Mendelian) genetics experiment4 (see Table 1). Course grades were determined

The laboratory assignment constrained performance on several aspects of the report, because the hypotheses methods, and data display formats were prescribed. Therefore, differences in performance were specifically hypothesized to manifest in the Discussion section of the reports. As seen in Table 1, three of four rubric items and the overall subscale scores significantly favored the treatment. 4


11/23/2009 3:20:00 PM


111

by performance on four exams (three 100-point midterms and a 200-point final), and the lab reports were evaluated using the Universal Rubric for Lab Reports (hereafter referred to as rubric) previously validated on similar assignments (Timmerman, 2008; Timmerman, Johnson, & Payne, 2007).

Table 1 Effect of Cta-based instruction on the discussion sections of lABORATORY REPORTS Universal Lab Report Rubric Criteria Discussion: Alternative explanations Alternative explanations are considered and clearly eliminated by data in a persuasive discussion. Discussion: Limitations of design Limitations of the data and/or experimental design and corresponding implications discussed. Discussion: Implications of research Paper gives a clear indication of the implications and direction of the research in the future. Discussion: Total Score

Treatment Mean (SD)

Control Mean (SD)

FStatistic

P-value (1-tailed)

.4296 (.5197)

.2801 (.4355)

6.171

.007**

.6996 (.6337)

.5350 (.5684)

4.703

.016*

.3130 (.51365)

.2124 (.42598)

3.463

.032*

2.3393 (1.4929)

1.7822 (1.3704)

9.501

.001**

** = significant at p 