Evaluating Quality in Computer-Enhanced Learning Materials in Engineering Education Authors: Joseph G. Tront, Virginia Tech, Blacksburg, VA 24051 USA
[email protected] Brandon Muramatsu, UC Berkeley, Berkeley, CA 94720
[email protected] Flora McMartin, MERLOT, Berkeley, CA 94804
[email protected]
Abstract – Computer-enhanced learning materials can enhance the engineering learning environment in a number of different ways. However, faculty members are generally hesitant to use materials that have not undergone a proper review process to evaluate the quality of the materials. The purpose of this paper is to discuss a set of quality criteria and methods useful in determining the utility of technologies and materials in helping attain specific course goals. The paper spotlights different types of resources, including digital libraries and assessment research available for faculty to use to locate and select helpful computer-based materials, as well as how to evaluate those materials in relation to their specific course goals. Illustrations will be given of how materials judged to be of high-quality, based on the criteria presented, have been used to improve student learning. Examples are taken from general engineering disciplines and will be relevant and of interest to engineers from all disciplines. Index Terms – educational digital library, courseware, adoption, STEM
INTRODUCTION Instructional technology enables new modes of learning—via courseware, course Web sites, collaborative communication, and in ways we have only begun to imagine. Educators around the country and around the world are developing technologybased materials to support their courses. The potential benefits of instructional technology are well known and include: addressing multiple learning styles, promoting active learning, scaffolding learning, interactivity, geography-independent collaborative learning, enhancing visualization, and extending the reach of a typical course. Because anyone and everyone can be a publisher on the Internet, educators and students are experiencing an overload of information coupled with materials of widely varying quality. Across most disciplines, there is a high activation barrier that must be crossed by faculty who wish to use digital learning resources in their courses. These materials are costly to develop, in terms of time and money. There is little re-use or incentive for re-use, which results in duplicated development. Adding educational resources to a digital library collection makes those resources easier to locate but also tends to highlight additional challenges in selecting the app ropriate resource. The contents of digital library collections vary widely and often do not contain a means for faculty to learn how best to make use of them (though various support structures are under development by participants in the National STEM Education Digital Library program). Many of the digital learning resources do not include instructor’s guides, training for faculty, or assessment information on the impact of the materials on student learning. Moreover, the introduction of new instructional methods into the classroom may be hampered by technical issues (e.g., lack of equipment or bandwidth), the need to re-design course materials, and inadequate guidance in aligning learning resources with teaching goals. These barriers are heightened by the time constraints that faculty face as they learn to use new materials [1,2,3,4]. The challenge, then, is to develop a process to allow faculty to quickly select quality materials and to tailor them to fit the learning environment that exists on their campus. Through our research we have gained an understanding of the in-person and computer-mediated social supports necessary to ensure broad use and wide-scale acceptance of the National STEM Education Digital Library (NSDL) by science, technology, engineering, and mathematics (STEM) educators. We have strived to answer the questions: 1. What are the characteristics of learning resources that foster and encourage their adaptation and adoption by others? 2. What kinds of training and support services should educational digital libraries offer to best support faculty in adapting or adopting digital learning resources? July 21–25, 2003, Valencia, Spain. 1
International Conference on Engineering Education 1
BACKGROUND •
As a precursor to the NSDL program, NEEDS—A Digital Library for Engineering Education (see www.needs.org) was developed to support the technological innovation underway in the engineering education reform community. NEEDS is the distributed architecture developed by Synthesis: A National Engineering Education Coalition to enable new pedagogical models based on Internet-mediated learning environments. Since 1994, NEEDS has made continuously available a catalog of courseware and other digital learning resources being developed nationally and internationally to enable educators and learners to search, access, comment about and rate, and download materials to support their learning process. NEEDS is in the process of receiving a major facelift and upgrade to be completed in time for the American Society for Engineering Education (ASEE) Annual Conference in June 2002 (see Figure 1).
FIGURE 1. UPDATED NEEDS INTERFACE
MERLOT MERLOT is a continually growing cooperative with over 6,800 high quality online resources to improve learning and teaching cooperation within higher education. MERLOT Institutional Partners (currently consisting of 22 systems and institutions of higher education representing over 1,400 campuses, 350,000 faculty and over 8 million students) connect systems, consortia and institutions of higher education, professional organizations of academic disciplines, and individual members together to form a community that strives to enrich the teaching and learning experience (see taste.merlot.org). Each learning material record in MERLOT can also have links to sample student assignments for using the materials, peer reviews of the materials and comments by members of the MERLOT community. These supplementary materials provide users with the pedagogical context for choosing, evaluating, and integrating the online materials into teaching and learning. The thriving and growing MERLOT community contains over 7,000 individual users and contributors (about 1,000 of whom are students). MERLOT also provides professional development training for faculty, faculty development personnel, and academic technology staff in participating institutions that are highlighted by the Annual MERLOT International Conference (see taste.merlot.org/conference/). Providing Quality Materials: The Premier Award and Peer Review of Learning Resources A discussion of quality materials often evokes strong reactions on the part of participants. It is important to recognize that quality means different things to different people and different things to the same person for different applications. Research has shown that faculty users of a digital library such as NEEDS want a range of “quality” in the overall collection, but that they want to “find the right thing” at the right time [1]. In response to user requests, contents of the NEEDS and MERLOT engineering collection range from materials under development to fully reviewed, award winning digital learning resources. NEEDS and MERLOT have instituted a number of mechanisms to ensure that the resources in our collections are of high quality and are potentially useful to engineering educators and students. As described above the My Workspace, user comments and ratings, and pedagogical descriptors in the resource’s descriptive metadata [5,6] are just some of the back-end services offered by our educational digital libraries. Participants in this project are leaders in ensuring quality of the resources in the NEEDS and MERLOT engineering collections. At the highest level of quality and review, NEEDS focused on the development of the Premier Awardfor Excellence in Engineering Courseware. In partnership with John Wiley & Sons, Inc. and other industrial sponsors, the Premier Award was developed to recognize outstanding non-commercial courseware designed to enhance engineering education. Now in its seventh year, the Premier Award has recognized 13 outstanding courseware packages and widely disseminated them to engineering educators (see www.needs.org/premier/). NEEDS worked with numerous experts including students, engineering educators, instructional designers, cognitive scientists, and learning theory experts to develop evaluation criteria that balances a resource’s design with its use (see Table 1) [7,8].
July 21–25, 2003, Valencia, Spain. 2
International Conference on Engineering Education 2
TABLE 1. EVALUATION CRITERIA FOR ENGINEERING EDUCATION COURSEWARE Instructional Design: • Interactivity • Cognition/conceptual change • Content • Multimedia use • Instructional use/adaptability
Software Design: • Engagement • User interface and navigation • Interactivity • Multimedia use • Technical reliability
Engineering Content: • Accuracy of content • Organization of content • Consistency with learning objectives
While the Premier Award is intended to recognize a small number of exceptional courseware, NEEDS and MERLOT are jointly implementing the MERLOT process of peer review, using the evaluation criteria described above, to review their entire collections for the engineering community. This peer review process is just beginning with less than 3% of the learning objects currently reviewed. Improvements to this peer review process to increase the qualified pool of reviewers are being sought. The peer review and Premier Award processes give us insight into the characteristics of quality materials. These processes enable NEEDS and MERLOT to provide users with high-quality collections. Coupled with support structures (e.g., user comments and ratings, pedagogical metadata, etc.) they provide quality collections from which faculty and students may obtain quality learning objects. SETTING THE STAGE FIGURE 2. MODEL FOR SUCCESSFUL ADOPTION/ADAPTATION In working with faculty developers and users of digital learning resources in the coalitions, we developed a simple model to illustrate the factors we believe are important in the successful adaptation or adoption of Desired Learning digital learning resources. This insight was further Outcomes enhanced by our work with the Institute on Learning Technology (ILT) located at the National Institute for Science Education (NISE – University of Wisconsin, Madison). During 1999-2000 the ILT researched effective STEM practices that are made possible through the use of Values & computer technology. Fellows from across the country Environment CharacterSuccessful produced vignettes and extensive case studies (see of the istics of the Adoption/ Resource www.wcer.wisc.edu/nise/cl1/ilt/) providing STEM Educator Adaptation instructors with rich contextual examples of appropriate and effective uses of learning technologies—from e-mail to high performance computer applications—at a variety of educational institutions. The combination of this Community research and our NSF Engineering education coalition Support experience led to the development of the model (see Figure 2) for successful adaptation and adoption that we have been using. Each of the contributing elements of the model is generally present in every potential adaptation/adoption situation. We believe training and support services are required to enable faculty to identify, enhance, and bind all four contributing elements together to facilitate a successful process. The Target Audience: Adapters and Adopters A recent survey conducted by SUCCEED shows that overall nearly 45% of the coalition faculty surveyed use some form of technology-mediated learning in their courses; even if the technology used is only at a very fundamental level such as email and email lists to facilitate better communications with their students. Few of these faculty reported that they are innovators who have created content, reporting they lack the time and resources to develop significant materials. In a 1999 survey of potential users of the Web in engineering education, NEEDS found that 45% of the respondents reported using the Web to learn more about teaching while 65% to 75% reported either collecting course materials for students or using it as an instructional resource for students. These surveys support other studies [2,3] showing that faculty are eager to learn how to effectively use technology, and also illustrate the fact that mechanisms are needed to help facilitate this learning. We find it useful to classify the target audience into three basic learner/educator categories:
July 21–25, 2003, Valencia, Spain. 3
International Conference on Engineering Education 3
• • •
Developer – designs courseware with defined learning objectives to meet a particular need, developing most if not all materials from scratch; Adopter – uses courseware developed by others as-is, with little or no modification or additional material; and Adapter – uses courseware developed by others in its developed form with moderate modification and additional materials.
We have found that each group has slightly different motivations for using or developing digital learning resources, and often has different access to support and technology in general. We also find that the majority of educators do not fall into the developer category; hence our work focuses on the adapter and adopter. Learning Outcomes ILT research shows that successful learning environments are those in which faculty have aligned learning tools and activities with their goals for promoting learning. This requires that a faculty member become a “Jack, or Jill, of All Trades,” solving educational problems by combining different learning strategies (such as the collaborative learning method) with instructional technology, student-based inquiry, and relevant assessment techniques. We use the simple Course FIGURE 3. COURSE CONSTRUCTION MODEL Construction Model in Figure 3 to Establish Course Goals illustrate a process through which Computer Based faculty can create courses that align Learning Activity activities and resources with Identify Student Learning Outcomes learning outcomes. When we use this model with faculty, we make it Search for Resources meaningful by providing a set of In Digital Libraries Design Learning Environment resources that we use to help guide faculty in identifying course goals and learning outcomes. When we Review Resources for Design Learning Process Design Learning Activities Applicability work with faculty to select digital (e.g., individual or (e.g., computer or other collaborative) non-computer) resources from educational digital libraries, we focus on the shaded Select Resource And boxes on the right side of the Continue Designing Assess Student Learning figure. In order to assist faculty in Activity designing activities that enhance student learning, we help them to become more cognizant of the overall construction process and then focus in on the issues involved in each of the blocks related to a Computer-Based Learning Activity. A key aspect of the search portion of the model is the fact that the material selected for incorporation into the curriculum comes from high-quality collections such as NEEDS and MERLOT and must be assessed for its contribution to improvement in student learning in much the way that any other learning process or activity is assessed. Given the considerable time and funds that faculty anticipate investing in this type of curriculum reform, they are not likely to do so unless the resources have convincing evidence of improved student learning. This emphasis on evidence-based resources has been well received in the workshops we have presented thus far [¡Error! Marcador no definido.,¡Error! Marcador no definido.,¡Error! Marcador no definido.]. Adaptability and Adoptability Criteria for Digital Learning Resources Digital learning resources come in all shapes and sizes. They are often designed to meet the specific needs of the developer, and often have a multitude of other uses. With such variability, how does one determine the “quality” of the resource? And what does “quality” really mean? While a “quality” resource may have high production value, provide interactive exercises to understand a difficult concept and provide support for multiple learning styles, if an educator cannot use it in his or her classroom then the “quality” resource is useless. Our work with the NEEDS/MERLOT-Engineering peer review process and the Premier Award has given us a solid understanding of the criteria for quality courseware as a whole. Quality courseware is made up of a number of factors; adaptability and adoptability are only two considerations in a courseware’s overall evaluation. These review processes tend to take a high level overview of adaptability and adoptability, to facilitate a reasonable peer review procedure. For example the Premier Award criteria includes the statement: “To be useful and long lasting in the broad context of engineering education July 21–25, 2003, Valencia, Spain. 4
International Conference on Engineering Education 4
courseware must adapt to the needs of varied curricula, because curricula rarely adapt to the courseware.” This is important because we have found that faculty members are generally reluctant to take entire courseware modules and adapt them into a local curriculum without modification [9]. Adaptability or adoptability can be accomplished through conceptual or physical modularity. In the former, modules designed as a laboratory guide can offer a virtual lab experience for those who do not have access to the actual hardware. In the latter the author might make available the individual elements that comprise a courseware module allowing the adapter and adopter to distill the material into a new format. Through the previous example, our existing processes begin to demonstrate some of the characteristics of adaptable and adoptable material. However, because recent work has shown that there are a number of additional factors to consider and more detail is required, additional research is needed. A Community Approach Although we have made strides in forming communities of engineering educators, we are often faced with this question, “How do I use this at my school?” The frequency with which we get this question demonstrates that a better understanding of the overall support environment, the community in which the resources and educator exist, is needed. The environment includes the technical support and training in the development and use of digital learning resources, as well as the nature of the academic environment itself. To set the stage for our work, we pilot tested a workshop titled “Evaluating, Selecting, and Using Computer Enhanced Learning Technologies and Courseware in Engineering Education” at the 2000 and 2001 Frontiers in Engineering Education (FIE) conferences, as well as the recent Joint Coalitions Conference in March 2002. The workshop participants were primarily faculty who had taught for over six years, and were predominantly in electrical and mechanical engineering. These workshops clarified our thinking about the important characteristics of adaptable and adoptable materials, as well as how we might strengthen our support for users using resources available through educational digital libraries. Participants especially appreciated the opportunity to practice selecting materials in NEEDS and learning more about methods for finding resources and criteria judging particular courseware. The workshop approach is important because the literature on faculty and instructor’s adaptation of innovation in higher education shows that the process is most successful when faculty members have the opportunity to learn from respected colleagues who have tried the strategies about which the potential adapters are already curious [10,11,12,13,14]. This finding was confirmed and expanded upon through the case study research conducted by the ILT in 1999-2000. In fact, 68% of the respondents to the recent SUCCEED survey indicated that they would be willing to invest the time to attend half-day, fullday or multiple-day workshops on the use of educational technology in engineering learning environments. There is a strong desire for community assistance—a need to better understand what others are doing. Anecdotally, colleagues from SUCCEED partner institutions, former Synthesis member universities, other active NSF Engineering Education Coalition schools and colleagues at non-coalition affiliated universities, report that many faculty across the country are eager to participate in the use of digital teaching/learning materials. These faculty members also indicate that they have not been able to find a good mechanism to identify materials, nor have they found a consistent process to adapt materials to their campus environment. Through our existing workshop we have anecdotal evidence that faculty face an additional challenge, they have varied access to qualified instructional designers, multimedia developers and other technical support on their campuses, which further hampers their efforts to use digital learning resources in their classes. Another factor that weighs on whether or not faculty members choose to adopt and adapt digital resources in their teaching is the culture of the educational institution where they work. Research has shown that, because of the reward structure, faculty at research and even at some comprehensive universities are reluctant to adapt and adopt resources developed at another institution (“not invented here” syndrome). Over the past twelve years, a national movement led by Boyer [15] to promote and reward the scholarship of teaching has generated modest gains for faculty engaging in teachingrelated research. Recent evidence suggests that nationwide, retention, tenure, and promotion criteria are being modified to provide more weight to educational research in the discipline. The processes proposed in our workshop help faculty define or refine their learning goals, design a learning environment enhanced with digital resources, assess the effectiveness of that environment, and finally improve on its design. SUMMARY AND IMPACT Understanding the nature of how faculty, as one of the key audiences of the National STEM Education Digital Library, adapt and adopt digital learning resources in their classes is vital to the long term success of educational digital libraries. The research underway here, helps to a broad snapshot and needs analysis to better inform collections and educational digital libraries as they are developed. We are examining all facets of the adaptation and adoption model and working to develop a process that binds these entities together into a reproducible and broadly applicable procedure. The outcomes will be tested July 21–25, 2003, Valencia, Spain. 5
International Conference on Engineering Education 5
tools, workshops and an online Best Practices Guide that provides insight into the in-person and computer-mediated social supports necessary to ensure broad use and wide scale acceptance of the STEM education digital libraries. REFERENCES [1]
McMartin, F. “Preliminary findings from Science, Mathematics, Engineering, and Technology Education Digital Library Use Study Focus Groups, URL: http://www.smete/info/survey/user_study_dl.html, 1999.
[2]
Inman, E. & L. Mayes. “Educational Technology: A Survey of Faculty Use and Need,” Journal of Staff, Program, & Organization Development. 16(1), Summer, 1998.
[3]
Tabor, L. Faculty Development for Instructional Technology: A Priority for the New Millennium,” Journal of Staff, Program, & Organization Development. 15(4), 1998.
[4]
Chen, J. C., M. Ellis, J. Lockhart, S. Hamoush, C. Brawner, J. Tront, “Technology in Engineering Education: What do the Faculty Know and Want?”, Proceedings of the 1999 ASEE Annual Conference, Charlotte, NC, June 1999.
[5]
Instructional Management System. (1998). The IMS Metadata Specification. Retrieved March 21, 1998, from http://www.imsproject.org/md_overview.html
[6]
Institute for Electronics and Electrical Engineers. (2002). Draft Learning Object Metadata Standard, Retrieved 2002, from http://ltsc.ieee.org/wg12/LOM_WD6_4.pdf
[7]
Muramatsu, B., Eibeck, P. A., Stern, J. L., and Agogino, A. M., Effective Processes to Give Engineering Educators Easy Access to Quality-Reviewed Electronic Courseware, Invited Presentation, NSF Engineering Education Innovators Conference, Washington, D.C., April 8, 1997.
[8]
Synthesis Coalition, Quality Workshop, Synthesis Coalition: San Luis Obispo, CA: May 1995.
[9]
Fitzgerald, M. A. (2000). The Gateway to Educational Materials: An Evaluation Study. Technical Report submitted to the U.S. Department of Education. Retrieved 2002 from http://www.geminfo.org/Evaluation/index.html
[10] Millar, S. B. “Full scale implementation: The interactive “whole story,” Project Impact: Disseminating Innovation in Undergraduate Education, Arlington, VA: NSF, 1995. [11] Kozma, R. B. “A grounded theory of instructional innovation in higher education.” Journal of Higher Education. 56:3, 300-319, 1985. [12] Hutchinson, J. H., & Huberman, M. “Knowledge dissemination and Use in Science and Mathematics Education: A Literature Review.” NSF report CB 2649X-00-0, Arlington, VA: NSF, 1993 [13] Rogers, E. M. Diffusion of Innovation, 4th edition. New York: Free Press, 1995. [14] Foertsch, J. A., Millar, S. B., Squire, L. L., & Gunter, R. L. “Persuading Professors: A Study of the Dissemination of Educational Reform in Research Institutions.” Madison: University of Wisconsin, LEAD Center. 1997. [15] Boyer, E. (1990). Scholarship Reconsidered: Priorities fo the Professoriate. Carnegie Foundation, NJ.
July 21–25, 2003, Valencia, Spain. 6
International Conference on Engineering Education 6
