Online Course Development Using the ADDIE Model of Instruction Design: The Need to Establish Validity in the Analysis Phase
Shane Moulton and Jane Strickland Idaho State University USA
[email protected] [email protected] Al Strickland Idaho State University (retired) USA
[email protected] Jerry White Federal Bureau of Investigation (retired) USA
[email protected] Lauralee Zimmerly Idaho State University USA
[email protected] Abstract: The goal of this study was to facilitate quality course development in higher education, by providing needed structure to the ADDIE instructional design model. This was the first step in creating a more structured model for the instructional designer to follow. The Delphi Technique was used to establish the validity of the 14 tasks identified by the authors in the Analyze phase of the ADDIE model. A panel of Instructional Design Experts (IDEs) was assembled for Part 1 of the study, followed by Part 2, which codified the process by using an actual instruction design research study with a panel of Subject Matter Experts (SMEs) and IDEs. This process was verified with greater than 90% agreement among the panel members.
Introduction While the primary focus of this research is related to the development and use of instruments for establishing the validity for the various tasks during the analyze phase of the ADDIE model, it applies to many other Instructional Design (ID) models, as well. Instructional design has been used for four decades (Branson, 1975) in PerformanceBased Training (PBT) and Criterion Referenced Instruction (CRI), but with the proliferation of distance learning courses, many institutions are discovering that without appropriate documented instructional design their mission is compromised. Allen and Seaman (2007) asserted that online education in some form of web-based format is without question fastest growing segment of higher education in the United States more than two thirds of higher educational institutions offering online courses and, many of these institutions offer complete online degree programs As early as 1995, Cantelon stated, "... most of higher education will take place off-campus through technological methods of delivery (p. 5). Nagel (2008), in a recent analysis stated that by 2014 more than 18.5 million students in higher education might be enrolled in online degree programs. Stratford (2009) pointed to universities rushing to
- 2046 -
expand their current classroom courses to online offerings, while neglecting the costs associated with such transformation. At the same time online enrollments are increasing, most colleges and universities are facing unprecedented pressures to cut costs. State funding for higher education is being dramatically slashed and university endowments have decreased (Stratford, 2009). Since more than 75% of the costs of an institution are in employment, and as these personnel become older, the institution is faced with three alternatives: (1) increase the student population, (2) increase the price paid by the students (tuition), or (3) remove the older faculty. Increases in online education generally remove geographical restrictions on student recruitment, but at the same time online courses open the door for price and quality shopping among that market. Naively, many institutions have sought to move to distance education as a cost cutting and increased market share measure (Neely & Tucker, 2010). It is not a simple matter to convert a face-to face course to an online presence. In an online class, the instructor role morphs from a source of knowledge to that of a facilitator of student learning. No longer can the instructor look into the eyes of the learner and see the despair of confusion and quickly adapt during the instruction. Simply moving a face-to face course to an online mode does not provide the instructional design time necessary to allow for adapting to learner variances; instead, the online course becomes the one-shoe-fits-all model. Hawkes and Coldeway (2002) envisioned the perfect storm emerging as this change in course delivery from face-to face instruction to online instruction demands more than the format conversion because online courses would call for different pedagogical strategies for engaging students compared to traditional classroom environment. Berge (1998) and Schifter (2000), both characterized online pedagogy as a movement from a teacher centered environment to electronic setting with greater student-centeredness than ever before. Harasim (2000) summarized the visions of Berge and Schifter by labeling the role of the instructor as changing from a provider of information to a facilitator of learning. Neely and Tucker (2010) point out that the unbundling of faculty roles in pursuing distance education is a cost that many higher education institutions have overlooked. The true cost of developing, launching, maintaining, and facilitating may be far greater than most higher education administrators have envisioned. In addition, many faculty have never been trained in the development of materials for distance education. Several institutions, such as Kaplan University, University of Phoenix, Harvard University, Rensselaer Polytechnic Institute, Carnegie Mellon University, Stanford University, have teams of experienced instructional designers to assist faculty in the development of online curriculum (West, Waddoups, & Graham, 2006: Shea, 2007: Xu & Morris, 2007; Hixon, 2008). The faculty are viewed as Subject Matter Experts (SMEs) and are not expected to be Instructional Design Experts (IDEs). On the other hand, IDEs may lack the background in learning theory or in the details associated with instructional design models. In a review of the research (Merrienboer, 1997; Kruse, 2004) for validation of the steps employed in executing an instructional design model, there is a virtual wasteland. The goal of this research, then, is to begin the examination of the detailed structure of one instructional design process, the ADDIE model, and validate the tasks in the foundational first phase of that model.
The ADDIE Model The ADDIE model was first conceived by Gagné in 1967 and placed in a more formal format by Briggs (1970) while working on several military training-based projects. The label, ADDIE, did not appear in the published literature until 1975 (Branson, Rayner, Cox, Furman, King, & Hannum, 1975). The ADDIE model, still utilized in some variation by most United States Military training operations and several intelligent agencies, was intended to have formative evaluation during every phase as indicated in Figure 1. Summative evaluation is expected to be the concluding step. Formative evaluation is used to determine the validity of the tasks required in each phase of the model and establishes the reliability of assessment instruments under development. Summative evaluation’s role is in outcome evaluations related to field-testing of the final product. If the formative evaluation has been thorough, any modifications suggested by the summative evaluation should become minimal (Strickland, 2005).
- 2047 -
While the ADDIE Model has been presented in other research studies (Nagel, 2009) as the distinctive five phases depicted in Figure 1, the corresponding detail for each is seldom completely documented. No doubt, part of this less than complete attention to detail is in the time-consuming process for validation by SMEs and IDEs.
Figure 1. ADDIE Model The literature indicates a lack of consistency for the number of tasks within the Analyze phase and/or the type of tasks. Therefore, the authors’ current research involves establishing a consistency of tasks (see Table 1) performed in the Analyze phase of the ADDIE Model. Guidelines and templates are provided for all 14 of these tasks (see the ISD website for detail). Both face and content validity should be addressed in this phase of the ADDIE model. In Table 1, Delphi judges are needed as indicated by the placement subject matter experts (SME) and Instructional Design Experts (IDE) in the columns labeled Face Validity and Content Validity.
- 2048 -
Table 1 ADDIE Analysis Phase Required Tasks and Types of Delphi Judges Needed Task Task A01 Task A02 Task A03 Task A04 Task A05 Task A06 Task A07 Task A08 Task A09 Task A10 Task A11 Task A12 Task A13 Task A14
Description Rationale Goal Objectives Concept Map Learning Influence Document Expected Learning Outcome Document Learning Hierarchy Document Learner Characteristics Document Target Audience Document Learner Constraints Document Pedagogical Considerations Document Learner Constraints Document Delivery Options Document Analysis Timeline Document
Face Validity SME SME SME SME SME SME SME SME SME SME SME SME SME IDE
Content Validity SME SME SME n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a
A graphical depiction of these 14 tasks is provided in Figure 2.
Figure 2. 14 Tasks in Analyze Phase of ADDIE Model
The Study The goal of this research is to create a consistent set of tasks within the Analysis phase of the ADDIE model so that instructional designers would be supported in documenting their work on this initial, critical phase of the ADDIE process. Similar documentation and instruments have been developed for the remaining phases: Design, Develop, Implement, and Evaluate; however, this detail is not included in the current paper. There are two parts to this study: 1) the selection by a panel of Instructional Design Experts (IDEs) to establish the face validity of the 14 tasks (see Figure 2) of the Analyze phase; and, 2) the verification that the 14 tasks through five Delphi survey instruments which yielded face and content validity data.
- 2049 -
The Delphi Technique Helmer and Dalkey created the Delphi technique as part of the Rand Corporation’s solution to addressing a specific military problem during the Cold War period (as cited in Helmer, 1983). The process is achieved through judges (experts) responding to a series of survey questionnaires in a Likert scale format. If consensus is reached in the first round, the process is halted. If consensus is not reached, feedback is provided to the judges related to the areas of disagreement and the same survey instrument is administered for a second round of survey responses (i.e., the Delphi is usually an iterative undertaking). Again if consensus is not reached, the process is continued with feedback given to the judges and the survey instrument is administered for a third round. At this point, the survey may be edited based on feedback from the judges. The process continues until consensus is reached, or until it is obvious consensus cannot be achieved. Delphi experiments tend to produce convergence of opinion – not just toward the mean, but also toward the true value (Helmer, 1983). The Delphi technique is based on the judges’ feedback, the anonymity of the experts, and statistical analysis. In a Delphi study, the judges do not interact with one another, and their responses are kept anonymous with only the group results given to the judges in the form of measures of central tendency (i.e., mean, median, and standard deviation). The experts on the panel are given the opportunity to reconsider their responses after receiving the group feedback and before proceeding to the next round. In 1972, Dalkey related the value of the Delphi Process in the following: When faced with an issue where the best information obtainable is the judgment of knowledgeable individuals, and where the most knowledgeable group reports a wide diversity of answers, the old rule that two heads are better than one, or more practically, several heads are better than one, turns out to be well founded. (p. 4) Verification of Process: Part 1 The first part of the verification of the 14 tasks (see Figure 2) of the Analyze phase involved the selection of a panel of Instructional Design Experts (IDEs). The IDE judges selected were two from government intelligence agencies or military training programs using the ADDIE model for designing training materials and one judge from higher education with expertise in designing educational and instructional courses. These selected judges have a range of experience from 10 to 20 years. The judges were presented with a Delphi survey instrument consisting of 30 questions targeted at establishing the face validity of the 14 tasks of the Analyze phase. Verification of Online Course: Part 2 The second part of this study focused on the use of the 14 tasks (see Figure 2) of the Analyze phase through an actual content-laden instructional design project (an online course in Dance Education). While the establishment of face validity by the IDEs is important, without Subject Matter Experts’ (SMEs) verification of Tasks A01 through A13 (see Figure 2) the instructional designer has no basis for strengthening the claim of validity of the documents. This study asserted that no less that three SMEs should be used for the Delphi. A number of research studies have used a similar number of judges (Rowe, Wright, & Bolger, 1991; Keeney, Hasson, & McKenna, 2001; Nehiley, 2003; Windle, 2004) and have produced positive results. Tasks A01, A02, and A03 required the establishment of both face and content validity; thus, a different content related survey (Delphi 01) was used. The remaining tasks (A04 through A13) employed Delphi surveys (Delphi 02, Delphi 03, and Delphi 04 that centered only on face validity. It was determined that the timeline (A14) for the Analyze phase was best judged by a panel of IDEs; therefore, a separate Delphi survey (Delphi 05) was created to establish face validity. While the primary focus of this research is related to the development and use of instruments for establishing the validity of the instructional designer’s creation of the various tasks needed during the Analyze phase of the ADDIE model, the impact of having certified tasks through face and content validity by external SMEs sources cannot be understated.
- 2050 -
Procedures The procedures for Part 1, the verification of the tasks (see Figure 2) of the Analyze phase involved the selection of a panel of Instructional Design Experts (IDEs). The potential IDEs were contacted via email to gauge their willingness to participate as judges and to assure that their anonymity would be maintained at all times. Next, the authors created the 30-item, four-choice Likert scale instrument for the judges to review the diagram of the Analyze phase (see Figure 2) and the accompanying guidelines and templates that documented the 14 tasks. The Delphi process was continued until consensus among the judges was reached. The data was collected and measures of central tendency computed, with a statistical analysis of the data and finally inter-rater reliability determined for the panel. The procedures for Part 2, the verification of the tasks (see Figure 2) in creating and implementing an online course involved the selection of a panel of Subject Matter Experts (SMEs) in dance education and Instructional Design Experts (IDEs). Potential SME and IDE professionals were contacted via email. The work of completing the 14 tasks and gaining face and content validity were accomplished in five steps using Delphi Survey instruments 01 through 05. The Delphi process was continued until consensus was reached for each survey instrument. The final step was to collect the data and compute measures of central tendency, then perform a statistical analysis to establish inter-rater reliability for the panel of SMEs and IDEs.
Inter-rater Reliability The establishment of both validity and reliability are critical elements in research-based, instructional, and assessment-based work (Howard, Osterlind, Dogan, & Tirre, 2007)). A valid measure should ensure the research, content, or assessment is an accurate reflection of what is observed. Reliability, on the other hand, only examines consistency of what is observed (Colton, Gao, & Kolen, 1996; Wang, Kolen, & Harris, 1996). The results of the Delphi technique are best reported in the form of inter-rater reliability statistics (Morgan, LamMcCulloch, Herold-McIlory, Tarhis, 2007, Gallagher, 2009). The validity of the current study lies in the ability of the judging panel to reach consensus that they have observed the same event, the whole event, and not simply the individual survey questions. The computation of the inter-rater reliability allows the researcher to infer that, given a different study with the same process, the panel of experts would produce a similar set of results. If using a pair of judges (i.e., two judges), some form of a Pearson Correlation could be used, but with three, or more, judges it is best to use an Interclass Correlation Coefficient. The Interclass Correlation Coefficient statistical method (i.e., Single Measure Interclass Correlation) was utilized to compute the inter-rater reliability of the panel used in Part 1 and Part 2 of this study.
Delphi Survey Data Survey and Data: Part 1 The survey data from Part 1, the verification of the 14 tasks (see Figure 2) of the Analyze phase, was generated by a panel of Instructional Design Experts (IDEs). The survey instrument consisted of 30 items with a four-point Likert scale force response method (1= Strongly Disagree to 4 = Strongly Agree). Analysis of this data is presented in Table 2. Table 2 Delphi IDE Data: Part 1 Face Validity Survey Delphi
Mean
Median
3.56
3.87
- 2051 -
SD .36
It should be noted the median is larger than mean indicting the graph of the data is skewed to the left, this is what Gordon and Helmer (1964) were referring to when the suggestion was made to use the median value instead of the mean. The Interclass Correlation Coefficient statistical method was computed for all 30 items on the survey instrument. In addition to the statistics displayed in Table 2, the mean inter-rater reliability was calculated at 0.977 to determine the agreement level among the expert judges (i.e., 97.7%). Survey and Data: Part 2 The second part of the data analysis was related to the instruction of a dance education course. The purpose of this analysis was the verification of the 14 tasks of the Analyze phase (see Figure 2). This data was gleaned from the interaction of a panel of Subject Matter Experts (SMEs) (Zimmerly, 2010) using the Delphi Technique. The survey instruments used a four-point Likert scale force response method (1= Strongly Disagree to 4 = Strongly Agree). Analyses of this data are presented in Table 3. Table 3 Delphi Data: Part 2 SME/IDE Face Validity and Content Validity Survey Delphi 1 (Content Validity, SME) Delphi 2 (Face Validity, SME) Delphi 3 (Face Validity, SME) Delphi 4 (Face Validity, SME) Delphi 5 (Face Validity, IDE)
# of Items 22 20 20 10 10
Mean 3.25 3.92 4.0 4.0 4.0
Median 3.85 3.97 4.0 4.0 4.0
SD .65 .23 0.0 0.0 0.0
Delphi 1 Survey was aimed at establishing the content validity for Analysis Task 01, Task 02, and Task 03. The survey instruments, Delphi 02, Delphi 03, Delphi 04, and Delphi 05, established the face validity of the remaining tasks within the Analyze phase. The Interclass Correlation Coefficient statistical method was computed for all 22 items on the Delphi 01 survey instrument, and the mean inter-rater reliability was 0.821. The inter-rater reliability for each of the four remaining survey instruments and there values are reported as follows: Delphi 02 = 0.947; Delphi 03 = 0.999; Delphi 04 = 0.999; and, Delphi 05 = 0.999.
Conclusion The details of the ADDIE instructional model have often been left to the discretion of the instructional designer. As a result, the project is many times compromised. It was the purpose of this research to provide the necessary detail to the Analyze phase of the ADDIE instructional design model to avoid this situation. The data displayed in Table 2 indicates the verification of these 14 tasks with 97.7% agreement by a panel of experts independent of the creation process. Equally encouraging are the results from Zimmerly (2010) when putting the 14 tasks in the Analyze phase of the ADDIE Model into an instructional design project. The content validity (Delphi 1) reported by the panel of experts was at the 82.1% level of agreement. In such a diverse area as the fundamentals of dance, under which this course falls, these results are welcome. The face validity reported by the experts for the four remaining Delphi surveys were at 94.7%, or higher. For a panel of SMEs who were new to the instructional design process and relatively unfamiliar with the Delphi technique may be a positive signal for a wide range of future course developments within higher education.
References Allen I.E., & Seaman, J. (2007). Online Nation. Five Years of Growth in Online Learning. Needham, MA: Sloan Consortium.
- 2052 -
Berge, Z.L. (1998). Barriers to online teaching in post-secondary institutions: Can policy changes fix it? Online Journal of Distance Education Administration, 1(2). Retrieved July 8, 2010 from http://www.westga.edu/~distance/berge12.html Branson, R., Rayner, G., Cox, J., Furman, J., King, F., & Hannum, W. (1975). Interservice procedures for instructional systems development. (5 vols.) (TRADOC Pam 350-30 NAVEDTRA 106A). Ft. Monroe, VA: U.S. Army Training and Doctrine Command. (NTIS No. ADA 019 486 through ADA 019 490). Branson, R. (1975). Interservice procedures for instructional systems development: Executive summary and model. Tallahassee, FL: Center for Educational Technology, Florida State University. (National Technical Information Service, 5285 Port Royal Rd., Springfield, VA 22161. Document Nos. AD-A019 486 to AD-A019490). Cantelon, J. (1995). The evolution and advantages of distance education. In M.H. Rossman & M.E. Rossman (Eds.), Facilitating distance education: New Directions for Adult and Continuing Education, no. 67. San Francisco: Jossey-Bass. Colton, D.A. Gao, X., & Kolen, M. J. (1996). Assessing the reliability of performance level scores using bootstrapping. KCl Research Report 97-3. Iowa City, IA: ACT, Inc. Dalkey, N. (1972). The Delphi method: An experimental study of group opinion. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds.), Studies in the Quality of Life: Delphi and Decision-Making. Lexington, MA: Lexington Books. Delbech, A.,Van de Ven, A., & Guftafson, D., (1975). Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Greenview, IL: Scott Foresman. Gallagher C. (2009). Inter-rater reliability of STOPP. Age Ageing 38, 603-606. doi:10.109 Gordon,T.J. & Helmer O.. (1964). Report on a Long Range Forecasting Study. RAND Paper P-2982. Santa Monica, CA, USA: RAND Corporation Harasim, L. (2000). Shift happens: Online education as a new paradigm in learning. The Internet and Higher Education 3(1–2), 41–61 Hawkes, M., & Coldeway, D. (2002). An analysis of team vs. faculty-based online course development: Implications for instructional design. The Quarterly Review of Distance Education, 3(4), 431-441. Helmer, O. (1983). Looking Forward: A Guide to Future Research. Beverly Hills, CA: Sage. Hixon, E. (2008). Team-based Online Course Development: A Case Study of Collaboration ModelsOnline. Online Journal of Distance Learning Administration, 11(4), Retrieved August 01, 2010 from http://www.westga.edu/~distance/ojdla/winter114/hixon114.html Howard H., Osterlind, S., Dogan, E., & Tirre, W. (2007). The performance effects of word locator cues on the NAEP reading assessment. Practical Assessment, Research, and Evaluation, 12(13), 71-86. Keeney, S., Hasson, F., & McKenna, H. (2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195-200. Kruse, K. (2004). Intro to the ADDIE model. Retrieved from http://www.e-learningguru.com/articles/art2_1.htm Merrienboer, J. (1997). Training Complex Cognitive Skills: A Four-Component Instructional Design Model for Technical Training. Englewood Cliffs, NJ: Educational Technology Publications.
- 2053 -
Morgan, P., Lam-McCulloch, J., Herold-McIlory, J., Tarhis, J. (2007). Simulation performance checklist generation using the Delphi technique. Canadian journal of Anesthesia, 54(12), 992-997. Nagel, D. (2009). Most college students to take classes online by 2014. Campus Technology Online. Retrieved from http://campustechnology.com. Neely, P. & Tucker, J. (2010). Unbundling faculty roles in online distance education programs. The International Review of Research in Open and Distance Learning, 11(2). Retrieved from http://www.irrodl.org/index.php/irrodl Nehiley, J. (2003). How to Conduct a Delphi Study. Retrieved from http://waterplanning.org.au/planning-tools-howto/how-to-conduct-a-delphi-inquiry Rowe, G., Wright, G., & Bolger, F. (1991) Delphi: A revaluation of research and Theory. Technology Forecasting and Social Change, 39(3), 235-251. Schifter, C. (2000). Faculty participation in Asynchronous Learning Networks: A case study of motivating and inhibiting factors. Journal of Asynchronous Learning Networks, 4(1), 15–22. Shea, P. (2007). Bridges and barriers to teaching online college courses: A study of experienced online faculty in thirty-six colleges. Journal of Asynchronous Learning Networks, 11(2), 73-128. Stratford, M. (2009). University set to slash budgets. The Cornell Daily Sun. Retrieved from https://cornellsun.com/section/news/ Strickland, A. (2005). ADDIE. Retrieved from http://ed.isu.edu/addie/evaluate/evaluate.html Wang, T., Kolen, M., & Harris, D. (1996). Conditional standard errors, reliability, and decision consistency of performance levels using polytomous IRT. ACT Research Report 97-3. Iowa City, IA: ACT, Inc. West, R., Waddoups, G., & Graham, C. (2006). Understanding the experience of instructors as they adopt a course management system. Educational Technology Research and Development, 55(1), 1-26. Windle, P. (2004). Delphi technique: Assessing component needs. Journal of PeriAnesthesia Nursing, 19(1), 46-47. Xu, H., & Morris, L. (2007). Collaborative course development for online courses. Innovative Higher Education, 32(3), 35-47. Zimmerly, L. (2010) Dancing Online: Can the Students Still Follow the Steps? A Rationale for Online Delivery Methods in Dance. Manuscript in preparation.
- 2054 -
