Educ Psychol Rev DOI 10.1007/s10648-015-9341-3 I N T E RV I E W
An Interview with Richard E. Clark Daniel H. Robinson 1 & Robert A. Bligh 1
# Springer Science+Business Media New York 2015
Richard (Dick) Clark was born in Howell, Michigan (near Detroit), on September 15, 1940. He attended Western Michigan University where he received his bachelor’s degree in political science and history in 1962. He received his master’s degree in mass communication from the new Annenberg School of Mass Communications at the University of Pennsylvania in 1964 while he worked as a television director and journalist for television stations in Philadelphia and New York. In 1965, he moved to Kalamazoo, Michigan, for 2 years as head of broadcasting for Western Michigan University. In 1967, he left to complete his Ed.D. in educational technology with a minor in educational psychology at Indiana University. After finishing his doctorate in 1970, he was hired by Stanford University as head of technology for the Stanford Center for R&D in Teaching and as an adjunct faculty member in the School of Education. In 1975, he moved to Syracuse University as department chair in educational technology with a joint faculty appointment in psychology. Three years and too many snowstorms later, he left the cold Northeast to take a position as professor of educational psychology and technology in what is now the Rossier School of Education at the University of Southern California (USC) where he stayed until he retired in 2012. Dr. Clark is the author of over 200 journal articles, book chapters, and books. He is best known for his critique of the research on learning from media and his claim that “media are mere vehicles that do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition.” He has received the Thomas F. Gilbert Distinguished Professional Achievement Award and a Presidential Citation for Intellectual Leadership from International Society for Performance Improvement, the SITE Foundation Excellence in Research Award, the American Society for Training and Development research study of the year award for his work on performance incentives, the Thalheimer Neon Elephant Award for bridging the gap between science and practice, the USC Faculty Lifetime Achievement Award, the Socrates Award for excellence in teaching from the graduate students at USC, and the Outstanding Civilian Service Award from the US Army for his work in distance learning. Clark is an elected Fellow of the American Psychological Association (Division 15, Educational Psychology), the American Educational Research Association,
* Daniel H. Robinson
[email protected] 1
Learning Sciences, The University of Texas at Austin, G2100, Austin, TX 78713, USA
Educ Psychol Rev
and the Association of Applied Psychology and is a Founding Fellow of the Association for Psychological Science.
Tell Us How You Became Interested in Educational Psychology and Educational Science My interest in educational psychology and science evolved slowly over a number of years. I graduated from a small, rural high school and was the first in my family to attend college. I finished a bachelor’s and master’s in history, political science, and mass communication— fields that required no interest in science or psychology. I enjoyed my work in television and journalism and that started my interest in educational technology. I worked as a television cameraman and director on the Philadelphia ABC TV evening news, kids’ programs, fashion shows, and sport remotes and even produced a rooftop and window box gardening program in New York called “The Compleat Gardener” that had the largest audience of any public TV program in the state. I chose to go back to Western Michigan University when they recruited me because I needed the intellectual stimulation, and frankly, my wife and I were starting a family. I was working 15 to 18 h a day in television and had no life outside of work. I had the opportunity to teach as well as head Western’s new and large university technology operation that included a number of television production studios for classes, a film unit, and a large FM radio station. They had wired many of their lecture halls, classrooms, and dormitories with cable TV and audio and so had made a serious investment in an area I thought was important. My decision to return to college for a doctorate in 1967 was based in large part on a belief that television could revolutionize education, plus the realization that I needed to know much more about education in general. I also needed more than a master’s degree to make any contributions to education. In my work as a television director and journalist during the early part of the educational television movement, I saw kids and adults who seemed to be much more motivated to learn from television than other sources. I considered myself to be an artist at that point in my career, and I confess I did not know or care much about science. I was offered a scholarship and eventually a fellowship by Indiana University based on my GRE and Miller’s analogy scores and past work in television. Indiana at that time had the largest and most successful university program in instructional technology. They also had a diverse and very active educational psychology faculty. The faculty who managed our fellowship program made an effort to invite visiting faculty from top programs around the country to teach doctoral level seminars. For example, Richard Snow came from Stanford for a couple of summers and Gavriel Salomon came as a new assistant professor after finishing his Ph.D. at Stanford. I soon noticed obvious differences between claims about learning and instruction made by faculty teaching in psychology and those in educational technology. When I asked about the differences in seminars, my questions were welcomed by people in psychology but caused problems in some of my technology seminars. I used what I learned from a literature review I developed for a psychology course to critique claims made in a technology course and was punished with a failing grade on an important paper. The grade was apparently punishment for my conclusion in the paper so I later published a version of the paper (and sent the publication to the instructor who had failed me). Yet that experience turned out to be a very positive experience because it energized me to learn more about psychology and to be more open to the possibility that science might be the best way to filter the many conflicting claims being made about education.
Educ Psychol Rev
Seminars at Indiana with Gavriel (Gabi) Salomon and Richard (Dick) Snow had an important influence on me. Salomon’s first 3-hour seminar started at 2 pm on Tuesday and sometimes ended late in the evening with all of us agreeing to meet the following day to continue an argument or discussion. He was an amazing teacher and had a huge impact on me. He would often start a discussion by making a dramatic counterintuitive statement about some topic in instruction or media and then encourage disagreement. Students in the course got recognition for disagreeing when we could marshal evidence to support our point of view. When we all eventually seemed to find agreement about a topic, Gabi would often go away and write an article on it and then bring a draft to class for us to tear apart. Dick Snow did something similar and since he had started working with Lee Cronbach on the 1977 book Aptitudes and Instructional Methods, he pointed us to fascinating studies and encouraged us to tackle literature reviews.
So You Became Focused on Research. How Did That Play out over the Next Few Years? I was likely perceived as a pain by many of my educational technology instructors by insisting that they offer evidence for their claims, and when they did, I read the studies they mentioned and came back to them with critiques. To say I was a bit confrontational would have been accurate and maybe an understatement. I apparently also ruffled some feathers at the administrative level at Indiana because I learned later that the head of the program told our faculty that no recommendations were be sent to prospective employers on my behalf without passing over his desk for the addition of his view about me. It did not help that I was leading an attempt to change the rules that governed doctoral program requirements at the university level. I started out in a Ph.D. program but transferred to an Ed.D. track when I learned that I would be required to take 2 years of a foreign language and pass a State Department test in that language. I fought to have alternatives such as computer programming substituted for the requirement and finally realized that the university had many language instructors who were there to support the requirement and so were not about to change the rule. I simply chose a degree that would allow me to avoid the requirement but otherwise take the same courses. In retrospect, I made the best choice since my interests have always been both research and practice. The students in my NDEA fellowship program included people like Tom Schwen, Howard Levie, and Harold Stolovitch—all of whom went on to solid careers. We were given an old military Quonset hut built in the 1940s during World War II, so we had offices despite the fact that the building was falling apart. Many of my fellow students (and a number of educational psychology faculty) were behaviorists. These students managed to train the many mice and rats that inhabited the building so that they would disappear whenever we had guests, especially when federal representatives were visiting to assess our fellowship program. Despite the impressive achievement with the mice, I could not fully appreciate behaviorism and so joined three other doctoral students who were more interested in the new “cognitive” psychology. We had plenty of financial support so we had time for constant dialogue. Title VII of the National Defense Education Act provided the funding for our fellowship program. All fellowship students were given healthy stipends, a research budget, a part-time secretary, and some travel funds for AERA and APA conventions. I had paid for all of my university education up to this point with many part- and full-time jobs and loans that were too often a
Educ Psychol Rev
distraction from my studies, so receiving a salary and money for research and travel expenses allowed me to focus entirely on my courses and research. My wife and I were also very lucky to find and rent a huge summer lakefront home with a dock on the lake and a boat—and only a 20-min drive from the campus. The owner was ill and wanted someone to take care of the house and rented it to us very cheaply. So all through the spring, summer, and into the fall times, we held weekend get-togethers and invited fellow students and faculty, including faculty who were visiting. That house and those parties allowed me to meet and get to know many very active researchers and graduate students from all around the university and from other universities in the USA and Europe. My wife was a student in comparative literature so that stretched the backgrounds of the people who came to the house. I remember a view expressed by one of the editors of Science a few years ago that the British overrepresentation in international science prizes may be due in part to their common room tradition. British graduate students were expected to visit a common room for tea and snacks in the afternoon where they would meet a diverse group of other university faculty and students. He claimed that expecting those students to describe their interests and their work to people active in different fields gave them unique insights about connections as well as experience explaining complex ideas while avoiding technical terms. Our discussions at weekend social gatherings gave me some important insights about “proxy variables” and “active ingredients” in complex research. And while I continued to be interested in technology, I started to realize that what it carried was more important than the fact that it was used in instruction. For my dissertation, I changed my focus from technology to the aptitude-treatment research paradigm that Gabi Salomon, Dick Snow, and Lee Cronbach were developing. I became interested in what leads people to become more flexible and be open to change their minds. I realize now that my dissertation was all part of my interest in the capacity of science to produce counterintuitive research findings and a curiosity about how to help people be open to views that were more compatible with solid research findings but very different from those that they strongly believed. My dissertation study was successful but I chose not to pursue the area and instead reverted to technology after graduation. I experienced a number of problems with subjects in the studies that led up to my dissertation. It was the late 1960s and too many students were experimenting with drugs and that led to some unusual behavior during experiments. I had to dump one entire round of data and redo my dissertation study twice because of students who acted out during the data collection phases. I think that experience and a few others made me less inclined toward experimental work. The effort required to mount and conduct an experiment seemed like a huge waste of time to me. I got much more satisfaction out of literature reviews and the use of data collected by colleagues. I finished my degree in 1970 and felt I had a great experience largely because of the effort invested by faculty who managed the fellowship program and the financial support from Title VII. After working to support myself since high school, I had at last been given the freedom to think and work only for courses and my research.
So You Finished Your Degree at Indiana and Decided Not to Go Back to the Television Industry and Ended up at Stanford? I wanted to work in a university. It seemed to me that was the setting where I’d be free to focus on whatever interested me provided it was relevant to a graduate program and hopefully to granting agencies. I was still interested in technology but I wanted to study it more than to
Educ Psychol Rev
work as a director or producer on the creative end. I was very lucky that Stanford was completing a new building for a research and development center in teaching and needed someone to manage a very new and innovative technology center that was intended to support educational research. They were also offering an adjunct faculty appointment with the job. Dick Snow asked me to apply for the job and I was happy when I was chosen. Yet it was not the easiest environment for me to do what I intended. Most of the Stanford faculty I encountered thought technology was something that belonged somewhere else. Dick Snow used an overhead projector in his classes and he told hilarious stories about the jibes and insults he suffered when he pushed it on a dolly from his office to a classroom. My first encounter with Nate Gage was typical. Soon after I arrived, he called and asked me to come to his office. I knew about his research on teaching and his editorial work with the annual reviews and so was prepared for a substantive chat about my research plans and how they might connect with his. When I walked in the door, he said something like “You are Clark, right? The new technology guy?” and then, “That clock on the far wall is broken, can you please either get it fixed or replace it?” and went back to what he was typing. I smiled, took the clock, and left. Nate eventually became a strong supporter and was a wonderful human being. But I had an uphill slog to change people’s expectations about what anyone interested in technology had to offer. I worked at Stanford for 5 years, learned a huge amount, and made some friends and contacts that have been helpful throughout my career. David Berliner and Rich Shavelson were students at that time and both were interested in using technology for their research. I also developed a thicker skin in order to handle the unsolicited “feedback” I received whenever I published anything. Colleagues would show up at my door whenever something I wrote appeared in a journal to let me know how it could have been improved or to suggest that perhaps it should not have been published in the first place. Most junior faculty have to learn to handle very critical feedback from journal consultants—my challenge was to deal with people who sought me out to make comments that were not always easy to view as helpful. I had expected challenges from people like Cronbach (some people viewed him as a brilliant but dismissive person) only to find that he was invariably positive and helpful. Lee was also an amazing mentor and I learned a great deal from him about the best ways to support graduate students. For example, he often asked for a draft of a seminar paper in advance of taking his courses. He read them carefully and handed them back with suggestions on the first day of class. In some cases he suggested that students were not yet ready for his seminar and made recommendations on how they could prepare. He met with each student who was accepted and coached them through a number of revisions of a paper so effectively that many of those papers were published. He listened carefully to my ideas for study designs and then told me how the data would likely fall (he was invariably accurate), and his questions helped me make improvements without him doing the work for me. During my time at Stanford, I also became head of the ERIC Clearinghouse on Information and Technology. It was an opportunity to fund the writing of think pieces by people who were doing great work in technology, and it encouraged me to think about how to access technology research in the form of grant reports and other unpublished sources. The experience helped prepare me for the literature reviews I developed over the next few years. My strongest impression of Stanford at that time was that Stanford faculty were all committed to research and very demanding of themselves and others. It also seemed to be a very democratic environment. All hires and major policy changes were discussed and decided by the entire faculty. There were many informal “interest groups” of faculty and graduate students that met each week to design and conduct studies in specific areas. Attending those meetings was an education in itself.
Educ Psychol Rev
What Motivated You to Leave Stanford for Syracuse? I finally realized that I wanted to teach full time and so both wanted and needed a tenure-track academic appointment. I was offered various jobs with the government and with commercial companies. The argument made by those recruiters was compelling. They asked me why I should take such a minimal salary and work such long hours when I could most likely have a bigger impact and make a lot more money working in a company. I was also offered a job to manage the grant program in technology for the National Library of Medicine. Stanford policy made it nearly impossible for a staff member with an adjunct teaching appointment to move to a tenure-track appointment. I was recruited by Syracuse in 1975 and the dean there offered me a tenure-track associate professor position to head a department whose graduate programs were ranked second or third in the nation. I had the opportunity to work with a terrific group of faculty colleagues to help redesign their educational technology Ph.D. program. I was also offered a joint appointment in psychology since Syracuse did not have an educational psychology program. That appointment gave me an opportunity to learn more about areas like psychotherapy, motor learning, and animal studies. I liked Syracuse’s focus on both research and the application of research in practical settings. But the most attractive draw was David Krathwohl who has always been my image of the ideal dean. As a teaching department chair in a private university, I expected that my meetings with David would focus on enrollments and grants. Yet when I met with David in his office, his first question was most often “What are you writing?” or “What research are you doing now?” I realized eventually that he modeled the way he thought it best for me to interact with the faculty in my department. He was fair, totally supportive, and while he was clear about his goals for the school, he expected faculty to decide what worked for each department. My work at this time was the beginning of my analysis of the research on the use of different media to teach. I conducted a number of research reviews for the ERIC Clearinghouse. Dick Snow and I collaborated on a review of various designs that had promise for instructional technology and I was collaborating with Gabi Salmon on a Review of Educational Research article on media research. Gabi was in Israel at this point and he was reading drafts and sending revisions when he was on leave from the war in Lebanon. We were exchanging airmail letters that in the days before the Internet often reached 20 single-spaced typed pages. We did not always agree and finding common ground was a challenge with the time delay between letters. I found that we agreed about design issues and so co-authored an article with that focus (Salomon & Clark, 1977). I also wrote some articles on the design of graduate programs as an offshoot of my work on our department Ph.D. program (Clark 1978). In 1978, I was promoted to full professor.
But You Moved on After Three Years? I’m not one of those people who enjoy winter sports and I simply could not handle the Syracuse winter snows. It started snowing in mid-October and one year it was May before it melted. During those 3 years, they had an average annual snowfall of 270 in. The temperature fell to 20° below zero on a number of days. I had to ski to work occasionally because my diesel car would not start and the snow was too deep to walk. When, in 1977, the University of Southern California (USC) contacted me about a tenure-track position in educational technology in Los Angeles, I thought about it for about 2 s before throwing my hat in the ring. That move was the last one for
Educ Psychol Rev
me since the philosophy and policies at USC fit me perfectly. I was no longer managing anybody but myself (though I eventually took my turn as department chair of educational psychology). At this point in my career, I was convinced that there was almost no effective and ongoing connection between research and practice in education or technology and that became my passion. The School of Education at USC was in a growth period, and hiring more researchfocused faculty, so in 1978, I joined a number of other newbies from different universities including Myron Dembo, Dennis Hocevar, Robert Rueda, and Harry O’Neil. I spent the first year or so designing the seminars I was going to teach. The most difficult experience for me was handling the expectations of a majority of the graduate students. Most USC education students were working and only part time in the program. Our courses began at 4 pm to accommodate teachers who finished their day at 3:30 pm and students in jobs who negotiated to leave early for a class. Many were exhausted when they arrived and seemed to feel that I should not expect that they finish assigned reading in advance of class. I also brought with me the grading standards I’d learned at Stanford and too often violated student expectations about evaluation. I worked hard to design classes that would be interesting enough and clarify evaluations in order to reduce their complaints about being overworked. Gabi Salomon made a suggestion that helped solve the problem. He recommended that I give out a questionnaire at the conclusion of every class meeting. The form he shared asked three open-ended questions: first, what students thought was interesting, next what could have been improved, and finally exactly how they would improve it. I told them I would not read the answer to what needed improvement unless they gave advice about how. The form also had four or five Likert-type items asking about various aspects of the course. The forms were anonymous and I urged people not to sign them and to print if they thought I’d recognize their handwriting. I started every class with a summary of what students wrote about their previous class, emphasizing what changes were suggested and both why as well as the recommendations for how. Then, I told students what I would change and what I would not and why (using research evidence as a reason where possible). The impact on the students was amazing. I tried to take their advice if I thought it might help even if I did not think it was better than what I was doing. When I did not take advice, I was clear about why and cited studies so they could check my reasoning. The entire climate of my classes changed. Since my courses focused on learning, instruction, motivation to learn, individual differences, and the design of studies in those areas, classroom dialogue about the way the class was taught and assessed served as a concrete and engaging vehicle for elaborating course content. I also learned some very valuable ways to improve my teaching. I was talking too fast and too often, not checking for understanding often enough and needed to do more advanced work to ensure they understood the formats and expectations of my tests, practice problem-solving exercises, and papers. Students also helped me understand what examples were most engaging and what worked to motivate them to do advanced work. I had been teaching for 15 years in one place or the other, but my first decade at USC was where I learned how to teach graduate students—mostly from the students themselves.
Sounds Like You Feel Like You Learned as Much as the Students. Was It the First Few Years at USC When You Published the Reviews on Learning from Media? My students were always my best teachers. As a student, I was impressed with Gabi Salomon’s focus on designing studies and conducting literature reviews in areas that our
Educ Psychol Rev
courses covered. I tried to do the same in my courses and gave drafts to students to review and make suggestions. In order to prepare the reviews, I decided to take my entire summer “vacations” to do research and write. I set out at first to read all of the published comparison studies on learning from various media. My goal was to accept a challenge from Richard Snow at Stanford to create a taxonomy of the ways that different media influence the learning of different students for various subject matters. It was an ambitious goal but I started with the belief I might at least get it started. The more I read the first summer, the more I realized that solidly designed studies were showing no learning benefits for any medium, students, or subject matter. I felt stuck about how to describe what it was about instruction that did influence learning if carefully controlled studies comparing teachers, books, radio, or television produced similar learning gains. The insight came as a result of trying to understand why Jim and Chen Lin Kulik’s metaanalyses of media studies demonstrated such strong effects for newer media (an effect size of about .7). I pulled many of the studies they cited that compared teacher-taught courses with those presented by newer media. I quickly noticed that when teachers designed the new media version of a course and also taught the same course in a classroom, there were no learning differences between the two contexts. I pulled all of the studies the Kuliks listed and conducted my own meta-analysis and found zero effect size when teachers designed and presented the treatments compared (Clark 1985; Clark & Salomon, 1986). I also noticed inflated outcomes due to a number of studies that had been included two or more times in their analyses (a published version and one or two versions of the technical report on the same study). The insight about the “same teacher” effect was one of those “aha” moments that keep most of us slogging through false starts and dead ends. I realized that the active ingredient in treatments that produce learning was the instructional method being used by a teacher, and that method was seldom controlled even in well-designed studies. It was an insight that I should have taken from Lee Cronbach and Dick Snow’s extraordinary book, Aptitudes and Instructional Methods, published in 1977. I’d been involved in the drama that surrounded the draft of the book when I was at Stanford. They had reanalyzed the data from a huge number of instructional studies and too often reached different conclusions from those that occurred to the study authors. A number of people whose studies had been reanalyzed traveled to Stanford and turned up at Lee’s door to complain and argue. Yet, their book presented media as if it were a “method” and so did not clear up the confusion. To say I was excited would be a serious understatement. I began to make a list of the typical design problems in media studies, along with the factors that were most often confounded, as well as look at publication bias on the part of journal editors (e.g., avoiding well-designed studies with no significant differences). I started drafting the Review of Educational Research (RER) article titled Reconsidering Research on Learning from Media (Clark 1983). I argued that all learning differences in studies that compared different instructional media were due to novelty effects that wore off quickly and/or study designs where there were differences in the information provided between two or more versions of treatmentinfluenced learning test results. I argued that it was likely impossible to find any medium (or characteristic of a medium) that was not “replaceable” by another medium or characteristic of a medium to produce the same learning result. The article acknowledged that media had significant impact on student access to instruction and its cost and suggested that these were the outcomes that should be studied and that we should stop conducting media comparison studies “unless a new theory was offered.”
Educ Psychol Rev
What Kinds of Reactions Did You Receive from the Article? I did not anticipate the many strong reactions to the review. The week after it was published, Derek Bok, then president of Harvard, was quoted by the Washington Post using the analogy from my article about media being similar to grocery trucks in that they delivered food to stores and so made food available more or less efficiently but were not responsible for people’s nutrition. He was arguing that investing in computers for Harvard students would not aid their learning though he agreed they needed to be familiar with new technologies. I thought wrongly that his mention of the argument was only the first of many positive reviews. In fact, Bok’s supportive view was the only one I received for many months. Instead, I started to receive a number of very critical letters from media people—one anonymous writer accused me of “taking food from my kids’ mouths with your crazy ideas.” He apparently felt that his job as a media director for a large K-12 school district committed to instructional television was at risk. Most letters argued with my analogy. One typical reaction in this vein was the suggestion that some groceries had to be delivered in refrigerated trucks and so perhaps some media made differences in learning. The head of training for one of the major airlines called me at home to argue that the training of pilots would be “impossible” without computer simulation. He became very angry when I suggested that pilot training had been successful before computers were available, and though pre-computer training was more expensive and difficult (and in some cases more dangerous), it had definitely succeeded without computers. I acknowledged that computer simulations had made pilot training so much more safe and inexpensive but he called me a “publicity seeker” and hung up on me. A very capable and visible colleague from psychology who was heavily invested in the “media literacy” movement accused me in a very public forum of “seeking attention,” “engaging in self-promotion,” and distorting the evidence presented in the article. He also refused to “waste time” describing the distorted evidence. RER received only one rejoinder article whose author, Bob Kozma, simply argued that it was impossible to separate media from instructional methods, and he then described a number of comparison studies of the type I’d tried to discourage. After being accused of self-promotion, I noticed an increasing number of colleagues who suggested that being against something that was new and popular was a great way to get people’s attention. Those same colleagues were not inclined to discuss my suggestion that the educational sciences may more effectively indicate what does not work than what does. The RER article is still heavily cited as well as frequently discussed and has been the subject of many debates over the years. Support for the conclusion of the article has grown over the years but it seems that each new technology restarts the discussion. An average of three different graduate programs each year have asked students to “debate” by comparing my review with Bob Kozma’s rejoinder. The most recent debate happened as we were planning this interview. Many students who participated have stayed in touch with me over the years. In my view, none of the reactions to the review or any of the studies that have been conducted since have changed my conclusion. In Clark (1994a, b), I published a list of all of the challenges to the article and what I thought about the evidence for each alternative view. A number of colleagues have published their views about the RER article (Clark 1994b). I also published a book on the arguments that has attracted readers through two editions (Clark 2012). My strong sense over the years is that too many of the people who have written about the RER review had not read it carefully or at least not thought very deeply about it. And while most of those who contacted me had only thought about the grocery truck analogy, their reactions made me appreciate both the power and the danger of a good analogy. I also had to
Educ Psychol Rev
overcome some disappointment about the strength of the resistance to the evidence by people who should have known better—and how difficult it is to change entrenched beliefs. We continue to waste huge amounts of scarce education resources on the expectation that the use of a new technology will solve learning problems. For my part, the RER article was only an extension of what attracted me to research in the first place.
Media Was Not the Only Area You Questioned in Reviews There were a number of other areas that interested me—all seem to me to be examples of the power of science to produce counterintuitive challenges to closely held beliefs. For example, there is an abiding belief that if we give students a choice in how they will learn, they will be more motivated and successful as learners. In Clark (1982), I published a review in the Educational Psychologist titled Antagonism Between Achievement and Enjoyment in ATI Studies that offered evidence to the contrary for many students. There is considerable evidence that when students can choose more than one way to learn, a significant number expect that the way they’ve chosen will require less effort than the way they rejected. As a consequence they seem to invest less effort and so achieve less. Another topic involved the negative consequences of matching certain students to unhelpful instructional treatments. In the aptitudetreatment interaction research Cronbach and Snow described, there were a sizeable number of disordinal interactions. In many of those studies, common instructional treatments produced significantly less learning for students who would be expected to learn regardless of what method was used to teach them. Others seemed to know less after instruction than before they started. Dick Snow coined the phrase “mathemathantics” (instruction that kills learning) to describe the latter case. In Clark (1989), I offered some hypotheses about the conditions that lead to this result for students at different general ability and prior knowledge levels. For example, I found evidence that most students seem not to learn effectively when they attempt to use unstructured and unguided (often called “discovery” or “constructivist”) instruction. Most dramatic in this literature was the finding that some treatments appear to leave students knowing less after instruction than they did when they started. Nearly everyone in education seems to believe that when instruction fails, no damage is done beyond the failure to learn. I found over 70 studies conducted with different age levels and curricula where student posttest scores were significantly lower than their pretest scores. It seemed reasonable to conclude that in some cases, students can become more ignorant when we offer them unhelpful instruction. More recently I’ve been concerned about the fascination with computer games and animated figures used in children’s computer instruction. I published a number of reviews of studies comparing the use of computer games with other ways to teach in Clark (2007; 2008a, b). I had failed to find any evidence of motivational or learning benefit from games except in questionable studies conducted and reported by companies that sold games. Even though the various branches of the military had spent large sums of money on computer games for training, their evaluations all indicated a lack of learning benefits. I also collaborated with one of my Ph.D. students, Sandy Choi, on a number of reviews of so-called pedagogical agents— the animated figures that appear on the screen in children’s computer-based instructional programs to give advice and direct attention to key information (Choi and Clark 2007; Clark and Choi 2007). We drew on work by people like Richard Mayer and his students and reported that the best evidence was that these agents appeared to be distracting when compared with lower technology devices such as arrows or voice prompts. Although some children reported
Educ Psychol Rev
liking the agents, learning was consistently higher when they were absent. Jeanne Farrington and I published a few articles on what we called “snake oil” products and strategies that sold well but did not work (Farrington and Clark 2000).
It Sounds Like You Have Done Several Reviews Where You Have Uncovered Snake Oil Treatments, or What Is Also Known as Educational Quackery. But Lately You’ve Been More Interested in What Does Make a Difference in Instruction? I’ve always been interested in translating research to help improve both learning and motivation to learn (Clark et al. 2006). The problem for me was that a review of the instructional research would invariably indicate another counterintuitive finding. Bob Heinich, a faculty member at Indiana and former editor for Doubleday, sent me a note after the 1983 article and challenged me to spend effort on a more positive view of what does aid learning. As a result of Bob’s urging and my interest in the design of instruction, I branched out more. I became interested in the research on expertise and began to see that much of the knowledge used by experts was the “automatic” type described by Schneider and Shiffrin’s 1977 analysis, and that had been one of the main features of John Anderson’s research for his ACT-R theory. The early stages of all instructional design require an analysis of the kind of expertise students will learn. Most instructional designers are taught to perform “task analysis,” and the interview of experts is a key part of that process. Yet nearly everyone who uses task analysis interview techniques with experts notices that the experts provide only incomplete descriptions of how to perform complex tasks. I was influenced by the work by Susan Chipman, Gary Klein, and others on Cognitive Task Analysis—a strategy for identifying the non-conscious knowledge used by experts. I had developed my own approach to it and had started to do research in the area soon after moving to USC (Clark and Estes 1996). Over the next 20 years, I invested an increasing amount of effort in extending cognitive task analysis research and practice (Clark 2014). I also became more interested in instructional design models. In most of the instructional studies I was reviewing, the instructional methods that appeared to be most successful were things like worked examples, demonstrations, practice, and speedy corrective feedback. Yet the most common instructional method used in STEM and other curricula in K-12 and higher education seems to be giving students problems or challenges and expecting them to discover or construct their own learning. This finding led to my recent work with John Sweller and Paul Kirschner. John, Paul, and I started our collaboration as a result of an informal chat over coffee at the 2003 American Educational Research Association meeting in Atlanta. I mentioned that during my talks at the convention and other locations, people were rudely interrupting and shouting slogans advocating constructivist and discovery learning. Some had called me “an instructivist” as if it were an insult. John and Paul reported exactly the same experience and it’s fair to report that we were upset that people with different views felt they had to be rude to be heard. We also agreed that they were wrong and decided that maybe it was time to take a more careful look at the evidence for discovery and constructivist learning and write something together. It was not lost on us that Paul’s position in Europe and John’s in Australia permitted us to represent the research literature for a number of continents. It was a very productive collaboration that resulted in our 2006 Educational Psychologist article with what one critic called a “take no prisoners title”—Why Minimally Guided Learning Does Not
Educ Psychol Rev
Work: an Analysis of the Failure of Discovery Learning, Problem-Based Learning, Experiential Learning and Inquiry-Based Learning (Kirschner et al. 2006). The editors told us that the article caused more reaction than they’d experienced for many years, and it is still one of the most heavily cited articles on instruction. EP published a number of interesting rejoinders and in 2007 we responded. Most of the authors of the rejoinders made the point that some approach they favored provided scaffolding or guidance. We realized that the only issue that we had failed to communicate effectively in the original article was our claim that the evidence best supported the claim that more fully guided instruction will almost always result in more learning than less fully guided instruction. The exceptions are a minority of students with very advanced general ability and prior knowledge. Our collaboration has continued and in the past few years we’ve published seven articles together. A few were focused on teaching mathematics since resistance to guided instruction is very strong in that community. We also accepted a challenge from the editors of The American Educator (published by the American Federation of Teachers) to communicate with teachers and school administrators about why and how to implement fully guided instruction. The result was an article for teachers and school administrators called, A Case for Fully Guided Instruction (Clark et al. 2012). We received very helpful and welcome editorial advice from the journals of Lisa Hansel since none of us were experienced in writing for K-12 teachers. As a result of that article, Paul, John, and I continue to receive a large number of emails and letters from parents, school board members, and administrators asking questions and making suggestions. Since we worked to produce an article that avoided jargon and research issues, it has been more accessible to more people.
How Did You End up Working with a Number of European Organizations and Colleagues? I decided that I did not know enough about how to do evidence-based work so wanted to get experience in operating organizations but not in K-12 or university settings. I decided to work in business and government organizations thinking that I’d encounter less resistance and easier acceptance of positive results. My early work with North American organizations had taught me that I would not be allowed to publish what we learned for political and competitive reasons. Colleagues in business suggested I try Europe where organizations might be more inclined to allow data to be published. In 1989, I competed for a contract to revise the training for the Irish electrical utility and won the contract. The Irish Electricity Supply Board (ESB) was eager to base their future training on research evidence and was willing to invest in solid assessment as they progressed. Unlike most North American organizations, they were also willing to let me publish some of the data collected during my work. In 1990, I took a full-year sabbatical leave (and extended it for another year with a leave of absence) and moved to Ireland. It was also the early days of the European Union (in 1990, the presidency of the union was in Ireland), and Germany was being reunified. I was curious to see how it would all develop firsthand. Managers and staff at the ESB were very eager to implement the training system I had developed at USC. I was lucky that they knew more than me about how to make a new system work in an organization so I could concentrate on training people to use the new system and evaluating its impact. The engineers who ran the utility were careful about measuring results and had established solid baselines on their training including both how much time was required to learn and the job performance that resulted and so both costs and
Educ Psychol Rev
benefits. The new training system introduced that year was able to reduce training time by 50 % and save many person years, and it produced performance above required levels. In preparation for the work in Ireland, I developed an evidence-based instructional design system that attempted to assess not only conscious but also non-conscious knowledge. Consistently successful Irish engineering experts could not describe enough of the ways they performed tasks to allow their approach to be used as the basis for training. I felt that most of the experts I was interviewing wanted to cooperate but simply were not aware of how they made complex decisions (they often responded that the “answer just came to me”). They could easily report what they could watch themselves doing but were very vague about cognitive operations such as decisions that could not be directly observed. Given the limits of working memory, it seemed plausible that the “missing” knowledge was automated and not available to conscious inspection and reporting. This issue seemed ripe for more work since most education is based on an assumption that we are providing what students need to apply what they have learned. If some of what they learn were incomplete, application would be impossible. A key part of the ESB work required me to develop and present workshops to training designers, trainers, and managers on how to implement the new design system. The workshops were successful so I decided to branch out and offer public workshops to other government agencies and businesses in Ireland and in the rest of Europe. Through those workshops I had the opportunity to travel and work with a number of terrific people. One of them was Dr. George Woods, a physicist who was head of training for the European Patent Office (EPO) in Munich Germany and The Hague in the Netherlands. George invited me to implement the training design system for the EPO in Munich to see if it were possible to train patent examiners more quickly and effectively. In 1978, a number of European Union countries had agreed to set up a “one stop shop” for patents in all of the member nations, and so it was necessary to train patent application examiners in a new system. After 12 years of operation, the EPO (and its engineers) had kept very careful records of their training and examiner productivity and so there was a solid baseline to evaluate any changes. We conducted CTA interviews with the most successful patent examiners and developed training for parts of the job. As a result of the evidence from the initial training, we realized it was possible to reduce the time required for new patent examiner training from 2 years to 6 months. At the same time, the evidence indicated we could increase the performance of newly trained examiners so that it would exceed the average of experienced examiners. While I was in Ireland, I agreed to teach a couple of seminars at Dublin City University and to give talks at various times for colleagues at universities in Ireland, Belgium, the Netherlands, and Germany. Those talks led to research collaborations and my participation in discussions about the development of a European Educational Research Association. Yet the benefit of the time I spent with European organizations was an education for me in the challenges that must be handled to work in business and government organizations staffed by people from diverse cultures.
You Also Directed a Research Center. How Did That Happen? The experience in Europe and a number of changes in the School of Education led me to collaborate with Allen Munro to set up a research and development center. We called it the Center for Cognitive Technology. Allen’s focus was artificial intelligence and he had
Educ Psychol Rev
developed amazing, interactive computer simulations of complex systems and processes, so his work complemented mine. The center gave both of us a better platform for our work with graduate students on research and development grants and contracts with government, business organizations, and other research centers. I moved into rented offices off campus and with the move also managed to create some distance from university politics. Long before the center was in place, I had encouraged weekly meetings of an informal “research group” of graduate students and faculty on campus who had similar interests in expertise development, learning, and motivation. I modeled the group after the interest groups I had experienced at Stanford. Students were encouraged to tackle reviews in areas that interested them and bring their ideas to the group for discussion. A number of dissertations had resulted from those discussions over time, and the group was very popular in a graduate program where most students were part time. The center allowed us to go beyond discussions and actually fund studies. I had the opportunity to collaborate on challenging grants with some very bright, focused, and hardworking doctoral students such as John Bertrand, Mimi Bong, Sunhee Choi, Andy Dean, Sean Early, David Feldon, Keith Howard, Hiromi Masunaga, and Ken Yates. We also unexpectedly attracted support from other research centers when we advertised our interest in developing literature reviews in various areas. I owe a huge debt to Ken Yates who joined us as a colleague at the center and made important contributions to our programs and research. Some of our relatively small contracts turned out to be very important for many of our students. We were tasked with presenting the results of our reviews to research center clients so our students not only had added motivation to conduct careful reviews but also got invaluable experience making presentations and handling challenging questions. In only a couple of years, we managed to secure research grants and contracts with NSF, DARPA, ONR, and the Army and Navy both funded ongoing training and development grants. Allen specialized in computer-based authoring tools for instructional simulations and assessment. Ken Yates and I conducted instructional research and developed both evidence-based instructional models and presented workshops on training design and assessment. The center also provided an opportunity to extend my work on cognitive task analysis (CTA) with Ken Yates. Ken left a very successful career in the entertainment industry to do research and teach. He and I designed and conducted a number of CTA studies, most of them in the Keck medical school at USC. We realized that the very impressive results of past CTA research were largely ignored because of disagreements about how to measure expertise. The most important part of our CTA system required that we conduct a structured interview with at least three consistently successful experts (where success indicators met reasonable measurement criteria). Yet we found that in many fields, people disagreed about the indicators and so dismissed our findings. The only field we could find where we received no pushback about expertise was healthcare. Jeanine Chalabian MD, one of my Ph.D. students, was a surgeon and another student, Maura Sullivan RN, managed a surgical education center for the department of surgery. Fred Maupin, an Ed.D. student, and I designed a careful study of the use of CTA to teach a surgical procedure in 2002. We randomly divided the third and fourth year medical students into two groups. One group was taught the procedure by the top surgeons on the faculty. Maura Sullivan RN and a Ph.D. student in educational psychology and one of the staff MDs (not a surgeon) taught the other half of the students using training based on the CTA results taken from interviews with the surgeons who were also teaching the control group. Whenever any of the students from the study performed the procedure in the hospital, someone who did not know how they were trained evaluated them using the same structured protocol. Since mistakes made in hospitals must be recorded, we knew how many average injuries new
Educ Psychol Rev
MDs caused with this procedure. The surgeons on the faculty perceived the results as spectacular because the CTA-trained students made no mistakes that hurt patients and performed the procedure more quickly and learned more conceptual knowledge about it (Velmahos et al. 2004). I eventually learned that another source of their acceptance of the approach is that with CTA, they believed they’d be able to spend less time teaching (after participating in the CTA interviews) and more time doing surgery. Ken Yates and I went on to conduct a number of studies in surgery, and to acknowledge their impact, we were both awarded courtesy faculty appointments in surgery by a faculty member who very seldom acknowledged anyone outside of medicine or surgery. The most important finding in our research was that top-performing surgeons could only describe about 30 % of the decisions they routinely made during a surgery. We conducted similar studies with other professions such as engineers, software developers, and salespeople. Our studies matched others conducted by colleagues at Carnegie Mellon. Experts left out so many critical decisions that we came to refer to it as the “70 % principle.” They all seemed to remember everything they could watch themselves doing, but all were vague about what they did with their minds. My view is that the missing information is due to the fact that when we make the same decision over time, the decision becomes automated to free up working memory space to handle more novel problems. This automation process and the non-conscious knowledge stored for use are a huge benefit to the expert but a major problem for novice students when experts teach them. Students get information about nearly 100 % of the actions they must perform in a new skill but only an average of 30 % of the important decisions they must make to be successful. This may be one of the reasons for performance errors in hospitals and in many other fields. When the results of the CTA were combined with training designs based on the fully guided models, the learning benefits were about 40 % higher than non-CTA instruction. I’ve published a number of articles and reviews of this research with the students and colleagues who contributed to the work (Clark et al. 2010). I expected that universities, government organizations, and businesses would be rushing to use the system. The greatest interest has been in the military. The army funded many of the early CTA studies—but I’m not aware of its consistent use anywhere in the military. The Office of Science and Technology Policy (OSTP) in the White House has reviewed the studies and has tried to convince a number of federal agencies to use CTA— without any major success as of yet.
Why Aren’t More People Using CTA? I’m guessing that it is too new, requires funds and the acceptance of changes to implement, and there are not many people trained to conduct the CTA interviews. There are a number of businesses that offer CTA commercially, but for proprietary and competitive reasons, all of them refuse to describe either the system they are using or their results. CTA also adds expense at the front end of the design of training. I did not expect that universities would welcome it since most faculty seem to think that they have total recall of how they’ve done what they are training students to do. When I’ve made presentations on CTA to faculty from different fields, I’m almost always asked for studies in the specialty of the faculty member, and when none are available, I see the interest disappear. And since we don’t evaluate student performance after they leave a class in most professional fields, we have no benchmark for assessing improvements. Faculty have resisted its use in the few universities I’ve monitored as they tried to
Educ Psychol Rev
encourage its use. We conducted a CTA for a university that graduated a very large number of students only to learn that according to top experts, their curriculum ignored the most important skills those students needed in their jobs when they graduated. And even though the program administrators agreed that those skills were both critical and absent, they were not inclined to change their program (because of certification requirements) or adopt CTA. I did, however, expect more interest from businesses where a large number of baby boomers are retiring and taking away the knowledge they developed while working. In business, these kinds of decisions are (or should be) a cost and benefit issue, and while evidence from attempts to implement it in Europe have established a very favorable ratio, it has not led to any largescale implementation. I’ve come to think that scaling the use of evidence-based strategies is the biggest challenge we face. I’ve noticed more interest recently. The new “Deans for Impact” organization has a Science of Learning initiative that was designed in large part by Dan Willingham at Virginia and looks very impressive. Yet when I looked at the organization membership and asked colleagues about the impact of the initiative on their teacher education programs, the answers did not make me optimistic.
What Do You Think Will Solve the Problem? This is the question that has my attention at the moment. I intend to invest my time now in trying to understand how to overcome the resistance to evidence-based solutions to important problems. I also want to continue to urge the use of CTA in education and training. It seems to me that we spend entirely too much of our limited time and funds continually learning to rediscover the decisions needed to solve complex problems that have already been solved by experts. We expect that faculty in professional schools will know what students need to know and the best way to teach them when the evidence does not support their expectations. We also expect that solid evidence against a popular approach will lead decision-makers to search for more robust alternatives. Neither of those expectations is supported with action. So I think we need to spend much more time on ways to change educational policy and procedures when science clearly indicates they are inaccurate, wasteful, and occasionally harmful.
References Choi, S., & Clark, R. E. (2007). Cognitive and affective benefits of animated pedagogical agents for learning English as a second language. Journal of Educational Computing Research, 33(2), 455–480. Clark, R. E. (1978). Doctoral research training in educational technology. Educational Communication and Technology Journal, 26(2), 165–173. Clark, R. E. (1982). Antagonism between achievement and enjoyment in ATI studies. Educational Psychologist, 17(2), 92–101. Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459. Clark, R. E. (1985). Evidence for confounding in computer-based instruction studies: analyzing the metaanalyses. Educational Communication and Technology Journal, 33(4), 249–262. Clark, R. E. (1989). When teaching kills learning: research on mathemathantics. In H. N. Mandl, N. Bennett, E. de Corte, & H. F. Freidrich, (Eds.), Learning and instruction. European research in an international context. Volume II (pp. 1–22). London, UK: Pergammon Press, Ltd. Clark, R. E. (1994a). Media will never influence learning. Educational Technology Research and Development, 42(3), 21–29. Clark, R. E. (1994b). Media and method. Educational Technology Research and Development, 42(4), 7–10.
Educ Psychol Rev Clark, R. E. (2007). Learning from serious games? Arguments, evidence and research suggestions. Educational Technology, 56–59. Clark, R. E. (2008). What evidence would change your mind about the learning benefit of serious games? Educational Technology, 56–57. Clark, R. E. (2008b). Resistance to change: unconscious knowledge and the challenge of unlearning. In D. C. Berliner & H. Kupermintz (Eds.), Changing institutions, environments and people (pp. 75–94). New York: Rutledge. Clark, R. E. (2012). Learning from media: arguments, analysis and evidence (2nd ed.). Greenwich: Information Age Publishers. Clark, R. E. (2014). Cognitive task analysis for expert-based instruction in healthcare. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology, 4th Edition (pp. 541–551). ISBN 978-1-4614-3184-8. Clark, R. E., & Choi, S. (2007). The questionable benefits of pedagogical agents. Journal of Educational Computing Research, 36(4), 379–381. Clark, R. E., & Estes, F. (1996). Cognitive task analysis. International Journal of Educational Research, 25(5), 403–417. Clark, R. E., & Salomon, G. (1986). Media in teaching. In M. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 464–478). New York: Macmillan. Clark, R. E., Howard, K., & Early, S. (2006). Motivational challenges experienced in highly complex learning environments. In J. Elen & R. E. Clark (Eds.), Handling complexity in learning environments: Research and theory (pp. 27–43). Oxford: Elsevier Science Ltd. Clark, R. E., Yates, K., Early, S., & Moulton, K. (2010). An analysis of the failure of electronic media and discovery-based learning: evidence for the performance benefits of guided training methods. In K. H. Silber & R. Foshay (Eds.), Handbook of training and improving workplace performance, Volume I: Instructional design and training delivery (pp. 263–297). New York, NY: Wiley. Clark, R. E., Kirschner, P. A., & Sweller, J. (2012). A case for fully guided instruction. American Educator, 36(1), 6–11. Farrington, J., & Clark, R. E. (2000). Snake oil, science and performance products. Performance Improvement, 39(2), 12–16. Kirschner, P., Sweller, J., & Clark, R. E. (2006). Why minimally guided learning does not work: an analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psychologist, 41(2), 75–86. Salomon, G., & Clark, R. E. (Winter, 1977). Re-examining the methodology of research on media and technology in education. Review of Educational Research, 47(1), 99–120. Velmahos, G. C., Toutouzas, K. G., Sillin, L. F., Chan, L., Clark, R. E., Theodorou, D., & Maupin, F. (2004). Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. The American Journal of Surgery, 18, 114–119.
