Session T1H
Automating Instructional Design with eCAD Timothy J. Ellis1, William Hafner 2 and Frank Mitropoulos 3 Abstract - Colleges and universities are increasingly offering at least part and, in many instances their entire curriculum via online learning modalities. Despite this trend, there is inadequate support for the professors responsible for restructuring the courses they have refined over a career in the classroom for delivery via the Web. Teachers who are expert in their subject area and masters of their craft when in a classroom find themselves in the uncomfortable position of having to relearn how to teach in a new environment with little or no support. The process of planning and developing a course for delivery in an online environment is, in many significant aspects, analogous to the processes required to develop a software system. Both situations require the developer to manage resources through a series of steps with the goal of designing a product that effectively utilizes the computer to solve a problem. The procedures and tools used in software engineering to manage software system development, therefore, offer promise for developing online courses. Computer Aided Software Engineering (CASE) tools are of special interest by virtue of the support afforded the development process through computerization. This paper offers an architectural overview of a knowledge-based, course engineering system: eCAD (electronic Course Analysis & Design). The requirements for the system, the manner in which those criteria were developed and validated, and system design will be detailed. A working prototype will also be presented. THE PROBLEM Colleges and universities are increasingly offering at least part and, in many instances their entire curriculum via online learning modalities. [1][2]. Despite this trend, the process of designing and developing courses for online delivery remains problematic [3][4]. The course instructor – the individual traditionally responsible for planning, designing and developing college courses – is often in the very uncomfortable position of having to make critical pedagogical decisions with incomplete information and inadequate preparation. A great deal of attention has been focused on the effective distribution and management of educational content for delivery in an online modality. Guidelines regarding the features and functionality of course management systems are
being refined [5, 6], and great attention is being devoted to ensuring the reusability of learning objects across course management systems [7, 8]. There is, however, little to help the instructor plan, design and develop online educational programs. Despite suggestions and general guidelines [9, 10] and narrowly focused, partial design guidelines [11], the educator is often left to develop her or his own answers to two vital questions: which online tools will most effectively help the student learn, and how to use those tools to actually improve in the online learning environment. Teaching is not easy, regardless of the environment. Students differ widely in how they learn and what is effective in facilitating that process [12]. The difficulty is compounded at the post-secondary-level where teachers are primarily experts in the subject matter, not in educational processes and learning theory. The characteristics inherent in an online environment further compound the problem. Teachers, by-and-large, teach in the same manner in which they were taught [13]. Since few current teachers have experienced online learning as students, most are confronted with working in an environment for which they have no model. Furthermore, many of the aids available to the teacher in the classroom are not available online. The non-verbal cues that an experienced teacher can read to gauge student understanding are absent. The immediacy of the question-response-follow-up cycle is largely lost in the online environment. It is quite difficult to foster and maintain the peer-to-peer support system of a community of learners when that community is “virtual”. Teaching online does require a reevaluation of the pedagogical approach [14, 15]. INSTRUCTIONAL DESIGN Instructional design can be defined as: “The development of instruction for specified goals and objectives in which (1) the organized sequential selection of components is made on the basis of information, data, and theoretical principles at every stage and (2) the product is tested in real-world situations both during development and at the end of the development process.” [16, p. 408] There are a number of different theories supporting and models for instructional design [17, 18, 19, 20]. Dick and Carey [21] developed a very inclusive, nine-step model of the design process. This model, amplified with examples from a graduate-level multimedia systems courses, is detailed in the following paragraphs.
1 Timothy J. Ellis, Associate Professor, Nova Southeastern University, Graduate School of Computer and Information Sciences, 3301 College Ave, Fort Lauderdale, FL 33314,
[email protected] 2 William Hafner, Associate Professor, Nova Southeastern University, Graduate School of Computer and Information Sciences, 3301 College Ave., Fort Lauderdale, FL 33314,
[email protected] 3 Frank Mitropoulos, Assistant Professor, Nova Southeastern University, Graduate School of Computer and Information Sciences, 3301 College Ave., Fort Lauderdale, FL 33314,
[email protected]
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference T1H-1
Session T1H Step 1: Identify the instructional goals. The goals for instruction are, basically, the role the course plays in the curriculum. Examples of goals for a graduate-level, information systems course in multimedia would include: 1) be an informed consumer of media-enhanced business products; 2) understand the set of technologies that enable computer-based, multimedia products; and 3) be aware of the TABLE I BLOOM’S LEVELS OF COGNITIVE ACTIVITY AND ASSOCIATED BEHAVIORS Level of Cognitive Sample Student Behavior Activity Knowledge
list, define, tell, describe, identify, show, label, collect, examine, tabulate, quote, name
Comprehension
summarize, describe, interpret, contrast, predict, associate, distinguish, estimate, differentiate, discuss, extend
Application
demonstrate, calculate, complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment, discover
Analysis
separate, order, explain, connect, classify, arrange, divide, compare, select, explain
Synthesis
combine, integrate, modify, rearrange, substitute, plan, design, invent, compose, formulate, prepare, generalize, rewrite
Evaluation
assess, decide, rank, grade, test, measure, recommend, convince, select, judge, explain, discriminate, support, conclude, compare
systems course in multimedia is that the student understands the basics of cost-benefit analysis. Step 4: Develop learning outcomes. The learning outcomes must clearly specify what the student should be able to do, the conditions under which the student should produce the desired behavior, and how well the student must be able to perform it if she or he is to attain the goals specified for the course. In addition to focusing on the specific behavior, each learning outcome must also reflect the level of cognitive activity expected of the student. Although educational researchers have developed many different taxonomies for categorizing level of cognitive activity [19, 20], Bloom’s Taxonomy [17] remains a widely accepted model for developing learning outcomes. Bloom identified a sequence of six levels of cognitive activity and associated specific types of student behavior for each. Table 1 summarizes those levels and a representative sample of associated behaviors. The process of developing learning outcomes entails three steps: identify the specific topic or subject matter to be learned, decide what level of cognitive activity is necessary for satisfactory mastery of that topic, and identify what student behavior is necessary to demonstrate that cognitive activity. For example, in the multimedia systems course, the technological impediments to the distribution of mediaenhanced products is a core topic area, students are commonly expected to progress to at least the analysis level of cognition, and typically students demonstrate that mastery by developing and distributing, within the class setting, a media-enhanced product. The learning outcome describing this expectation might be: Plan, develop, document, and distribute as a Website a professional-grade multimedia product that can be used to educate, sell, or inform. Step 5: Design measures of learner performance. Performance measures are essential for determining if the students are in fact meeting the learning outcomes specified for the course. Measures may take the form of test questions, research papers, discussion forum participation, or projectbased exercises, to name just a few examples. The assignment to measure the learning outcome described above in Step 4 might be to work on a development team to create and distribute a media-enhanced product. Step 6: Determine instructional strategy. Once the learning outcomes have been identified for a course, the instructor must determine the instructional strategy by specifying which pedagogical tools should be used to facilitate the students’ attainment of each specific outcome [18, 22]. For online-delivered courses, there are actually two components to this step. First, a decision must be made regarding the timedependency of the course: asynchronous, synchronous, or hybrid. Second, the instructor must select from among the online tools available the one or ones that would be most appropriate for the specific learning outcome. Step 7: Select or develop instructional materials. During this step, the course as designed is built or customized. Videos are filmed, compressed, and placed on Web sites, threaded discussion forums are created, chat rooms established, class
processes and challenges inherent in developing mediaenhanced products. Step 2: Instructional analysis. Instructional analysis entails identifying the skills that are requisite in order to perform each of the goals. For example, to meet the first goal identified for the multimedia course – to be an informed consumer of media-enhanced business products – one must be able to: 1) perform a basic cost-benefit analysis; 2) develop a rubric to rate the strengths and weakness of media-enhanced products; and 3) evaluate the strengths and weakness of media-enhanced products to meet specific business needs. Step 3: Expected learner characteristics. Every instructional process assumes that participants will possess certain competencies. Some, such as the ability to read and write, are implied for all college-level courses. Other, less obvious, more specific competencies also apply and must be specified as part of the instructional design process. For example, one of the assumed competencies for the information 0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference T1H-2
Session T1H notes converted to Web-deliverable format, and the entire package is assembled into a seamless whole ready for student use. This stage is the point in the development process where most online educational tools such as Learning Space, WebCT, and Blackboard start [23]. Step 8: Formative evaluation. Prior to implementing the course, the instructional materials and strategies selected for the course are evaluated to ensure they are, in fact, efficacious in helping students attain the learning outcomes for which they were designed. Formative evaluations typically range in scope from tests involving single students through tests of small groups to tests in full classes. The goal of formative evaluation is to highlight specific instructional materials or strategies that are not effective and to make point-by-point changes in the course design prior to deployment. Step 9: Summative evaluation. The final step in instructional design is to determine if the course as a whole meets the instructional goals identified at the start of the process. Summative evaluations may be implemented as limited pilot studies prior to full deployment of the course, or may be conducted after the course has been offered for several iterations. Although the instructional design model has been well established as a method for producing courses, there are some inherent weaknesses in this approach. Since the process is quite lengthy and demanding, few faculty members have either the time or the background to direct instructional design. Leadership of the process usually falls to a professional instructional designer, with the course instructor assuming something of the consumer role. Concerns such as lack of experience, academic freedom, and ownership of intellectual property cause many instructors to be very reluctant to participate in instructional design efforts and, by extension, to modify their courses for delivery via an online modality. The instructional design process, furthermore, follows what is essentially a linear developmental model, rather like the traditional Waterfall model for software design [24]. This model connotes that a course is a “finished product” at the time of deployment, creating something of a static course design. Since there are always changes in student profile, online capabilities, and the discipline being taught, courses must be inherently quite dynamic. The instructional design model offers no formal procedures for incorporating this environmental-change data into the course design. LEARNING FROM SOFTWARE ENGINEERING The process of designing courses, especially those to be delivered via an online modality, is parallel to the process necessary to develop a computer product. The field of software engineering has developed many process models for the development of software which offer greater flexibility in creating products capable of being adapted to changing needs. Models such as the prototyping and the WINWIN Spiral [25] enrich the design process by actively engaging the prospective end users during the analysis phase. These software engineering models do not, however, adequately address the
dynamic nature of software development in which products are never truly “completed”. The evolutionary model does offer a dynamic development process. It enables developers to produce acceptable product versions that meet customer minimum needs but continuously evolve toward the “ideal” customer solution. The evolutionary model combines elements of the linear sequential model (applied repetitively) with the iterative philosophy of prototyping. The model applies analysis, design, code and test activities to produce incremental product releases as time progresses. Each linear sequence produces a more fully functional deliverable “increment” of the software [26]. The evolutionary model delivers software in small but usable pieces, called “increments.” In general, each increment builds on those that have already been delivered. Since each iteration of a college course builds upon the experiences of previous offerings, the evolutionary process model appears to offer a more robust development process than the essentially linear instructional design model. Instructors can borrow more than just a more robust design model from the software engineering field. One of the greatest strengths inherent in the structure of the software development lifecycle (SDL) model is the capacity to support the development process through computerization with Computer Aided Software Engineering (CASE) tools. CASE tools automate a significant amount of the software development process. A similar automation to the instructional design process offers the promise of placing a significant portion, if not total control of the course design process back in the hands of the course instructor, thereby addressing one of the major challenges in online learning. Borrowing from the software engineering field offers promise to college professor tasked with developing or translating courses for delivery in an online modality. The four-step, iterative design process inherent in the evolutionary development model, supported by computer aided design software potentially gives the teacher the systemic support necessary to effectively direct the course planning and design process. The balance of this paper will describe the processes followed in developing eCAD, a knowledge-based, automated course engineering system based upon an evolutionary perspective of course development. eCAD A group of seven professors with a minimum of five years experience teaching in online environments formed an Expert Panel to develop the requirements for the eCAD system. The backgrounds of the members of the Expert Panel included experience in software development, human computer interaction, information systems, and educational technology. The requirements identified by this panel are summarized in Figure 1.
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference T1H-3
Session T1H
1) User-friendly, no computer expertise necessary 2) Guidance through instructional design process a) Build learning outcomes b) Select appropriate online tool(s) to attain learning outcomes c) Build assignments and evaluation criteria d) Build course schedule, pacing and sequence of activities e) Build course evaluation instrument 3) Build and dynamically update value history a) Components i) Learning outcomes ii) Online pedagogical resources iii) Assignments b) Updated from i) Student evaluations ii) Instructor evaluations iii) Grades 4) Manage course evaluation a) Course-specific evaluation of i) Learning outcomes ii) Pedagogical resources iii) Assignments b) Data used to update value history i) Learning outcomes ii) Pedagogical resources iii) Assignments 5) “Hooks” into existing course management systems (WebCT, Blackboard, etc.) FIGURE 1 ECAD REQUIREMENTS SPECIFICATION
DESIGN The requirements for eCAD were translated into a design by the primary researchers, two of whom each had in excess of 20 years experience in software development and the other with over 10 years experience in educational technology. The system flow diagram for the product is displayed in Figure 2.
Course Requirement Brainstormer
Learning Outcome Generator
Resource Selector
Assignment Deveoper
stepping her or him through the first three steps of the instructional design process: identifying instructional goals, conducting instructional analysis, and identifying expected learner characteristics. Learning Outcome Generator: This module accepts the following inputs from the course instructor: specific topics or subject matter to be learned, the level of learning desired for each topic, and the student behavior that would indicate mastery at the desired level. The system provides input in the form of suggested action verbs appropriate for the desired level of learning for each topic. This module processes the inputs into well-formed learning outcomes, and outputs those learning outcomes to the Assignment Generator, Course Evaluation Generator, and Scratchpad modules. Assignment Generator: This module accepts as inputs well formed learning outcomes from the Learning Outcome Generator module. It also accepts inputs from the system in the form of suggested types of online pedagogical resources and specific assignments appropriate for a given learning outcome, based upon data from the Value History Database. Based upon these inputs, this module guides the instructor through the process of identifying an assignment type – based on online pedagogical tool selected – and specific assignment(s) for each learning outcome. The assignments developed in this module are output to the Schedule Generator, Course Evaluation Generator and Scratchpad modules. Schedule Generator: This module accepts input from the Assignment Generator module in the form of a list of course assignments, plus input from the instructor for term start and end dates, assignment sequence, and days per assignment. The module guides the instructor through the process of planning
Schedule Generator
Course Evaluation Generator
Assignment Generator
"Scratchpad"
Learning Outcome Repository
Online Resource Repository
Syllabus Warehouse
Assignment Repository Course Management System
Value History Database
FIGURE 2 ECAD SYSTEM FLOW
The knowledge-based capability inherent in the system is provided by a Value History Database module. Four ‘generator’ modules corresponding to the four major steps in each cycle within the evolutionary design process complete the system. The functional descriptions of the modules are detailed in the following paragraphs. Course Requirement Brainstormer: This module helps the instructor set the foundation for the course being designed by
the scheduling, flow, and pacing of the assignments, and outputs a course schedule to the Course Evaluation Generator and Scratchpad modules. Course Evaluation Generator: This module accepts inputs from the Learning Outcome Generator (learning outcomes) and the Assignment Generator (assignment types, based on online resource, and specific assignments). Based upon these inputs, it develops a course-specific evaluation that
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference T1H-4
Session T1H measures attainment of learning outcomes and effectiveness of assignments, from both the student and instructor perspectives. The evaluation is output to the Scratchpad module. Scratchpad: This module accepts inputs from the previously described “generator” modules and serves as a review and editing site for the instructor. The instructor can review and change any of the course elements through this module. Output from this module goes to the Course Management System (WebCT, Blackboard, etc) in use at the school. Value History Database: The final module of the eCAD system is a database of the course evaluations. This module accepts as input from the Course Management System both student and instructor assessments of the degree to which the learning outcomes were met and effectiveness of the assignments in promoting attainment of those outcomes. The data from this database serves as input to the Learning Outcome and Assignment Generator modules. SUMMARY AND FUTURE RESEARCH Courses are, without question, being successfully delivered via online modalities [27, 28]. Courses in Engineering and other technical curricula are likewise being effectively developed and delivered in both on-campus and online environments [29, 30, 31]. There are certainly a large number of process models such as the instructional design model that present potential as a solution to the problem of developing courses for an online environment [19]. This paper explored the potential of adapting the SDL model widely used in software engineering as a particularly promising process model. There is an intuitive appeal to adapting a tool developed by the computer industry to plan a course that will be delivered predominately via computer-mediated communication. Developing an effective college-level course is indeed complex and challenging. When the need to adapt the course to an emerging, poorly defined and imperfectly understood delivery system such as an online learning environment is added to the developmental burden, the difficulty becomes exponentially greater. The structure and support inherent in a formalized development process supported by a computerized development system offers great promise as means of addressing this difficulty. The eCAD system described in this paper offers promise as a tool to facilitate development of courses for delivery via an online medium. REFERENCES
[4]
P. Goodyear, G. Salmon, and J. M. Spector. “Competences for online teaching: a special report.” Educational Technology Research and Development, 49(1), pp. 69-72, 2001.
[5]
Learning Technology Standards Committee of the IEEE Computer Society, IEEE P1484.1/D8, 2001-04-06 Draft Standard for Learning Technology – Learning Technology System Architecture. IEEE Standards Department: Piscataway, NJ, 2001.
[6]
AICC CMI Subcommittee, Computer-Managed Instruction, Version 2.0. Aviation Industry CBT Committee, 1998.
[7]
A.E. Saddik, S. Fisher, & R. Steinmetz, “Reusable multimedia content in Web-based learning systems.” IEEE Multimedia, July-September, pp. 30-38, 2001.
[8]
Advanced Distributed Learning, Shareable Content Object Reference Model (SCORM) Version 1.2: The SCORM Overview. Advanced Distributed Learning, 2001.\
[9]
R. Carlson and D.R. Everett. Taking instruction online? Mid-South Instructional Technology Conference. Murfreesboro, TN, 2000.
[10] J. Levin, S.R. Levin, and G. Waddoups. “Multiplicity in learning and teaching: A framework for developing innovative online education.” Journal of Research on Computing in Education, 32(2), pp. 256-269, 1999. [11] T. Janicki & J.O. Liegle, “Development and evaluation of a framework for creating Web-based learning modules: A pedagogical and systems perspective.” Journal of Asynchronous Learning Networks, 5(1), 2001. [12] G. Currie. “Learning theory and the design of training in a health authority.” Health Manpower Management, 21(2), pp. 13-19, 1995. [13] J. I. Goodlad. Teachers for our nation's schools. San Francisco, CA:Jossey-Bass, 1990. [14] A. Carr-Chellman, and P. Duchastel. “The ideal online course.” British Journal of Educational Technology, 31(3), pp. 229-241, 2000. [15] L. P. Dringus. “Towards active online learning: a dramatic shift in perspective for learners.” The Internet and Higher Education, 2(4), pp. 189-195, 2000. [16] M. E. Gredler, Learning and Instruction: Theory into Practice, Fourth Ed. Merrill Prentice-Hall: Columbus, OH, 2001. [17] B.S. Bloom, M.D. Engelhart, E.J. Furst, W.H. Hill, and D.R. Krathwohl. (Eds.) Taxonomy of educational objectives, the classification of educational goals, Handbook I: cognitive domain. New York:Longmans, 1956. [18] N.E. Gronlund. How to Write and Use Instructional Objectives. Upper Saddle River, New Jersey:Prentice Hall, 2000. [19] R.M. Gagne, L.J. Briggs, and W.W. Wagner. Principles of Instructional Design. New York: Harcourt Brace Jovanovich College Publishers, 1992. [20] D.H. Jonassen, M. Tessmer, and W.H. Hannum. Task Analysis Methods for Instructional Design. London: Lawrence Erlbaum Associates, 1999. [21] W. Dick & L. Carey, The Systematic Design of Instruction, 3 Ed. Scott, Foresman & Co. 1990.
[1]
M. Arnone. “San Diego company quietly provides online courses to nearly 1,000.” The Chronicle of Higher Education, 48(6) p. A32, 2001.
[22] T.J. Ellis. “Translating a college course for delivery over the World Wide Web.” Journal of Instruction Delivery Systems, 13(3), pp. 14-18, 1999.
[2]
T. Mayor. “E-Learning: Does it make the grade?” CIO Magazine, Jan. 15, 2001.
[23] S. Fredrickson. “Untangling a tangled Web: An overview of Web-based instruction programs.” T.H.E. Journal, 26(11), pp. 67-77, 1999.
[3]
L.Y. Muilenburg, and Z.L. Berge. “Barriers to distance education: A factor-analytic study.” The American Journal of Distance Education, 15(2), pp. 7-22, 2001.
[24] R. S. Pressman. Software Engineering: A Practitioner's Approach, Fourth Ed. New York:McGraw-Hill, 1997. [25] B. Boehm, Spiral Development: Experience, Principles, and Refinements. CarnegieMellon Software Engineering Institute: Pittsburgh, PA. 2000.
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference T1H-5
Session T1H [26] J. McDermid, & P. Rook, “Software Development Process Models,” in Software Engineer’s Reference Book, CRC Press, pp. 15/26–15/28, 1993.
[29] T. L. Marchioro and R. H. Landau. “Web-based education in computational science and engineering.” IEEE Computational Science and Engineering, 4(2), pp. 19-26, 1997.
[27] K. Barker and R. McCartney. “A structured web-based learning environment for a laboratory course.” Proceedings of the 8th Annual TBEEC Conference on Learning with Technology, Long Beach, CA, 1996.
[30] J.X. Chen, C.J. Dede, X. Fu, and Y. Yang. “Distributed interactive learning environment.” Proceedings of the 3rd International Workshop on Distributed Interactive Simulation and Real-Time Applications, 1998.
[28] K. Sanders. “Dragonweb: Courseware for compilers.” The Journal of Computing in Small Colleges. 14 (4), pp. 102-109, 1999.
[31] M.S. Cohen and T.J. Ellis. “Teaching technology in an online, distance education environment.” Proceedings: Frontiers in Education Conference, 2001 Reno (pp. T1F-1 – T1F-6). Piscataway, NJ: IEEE, 2001.
0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference T1H-6
