A Theory-Based Integrated Design Process for ... - CiteSeerX

A Theory-Based Integrated Design Process for Development and Evaluation of Web-Based Supplemental Learning Environments

Chang S. Nam

Dissertation submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Industrial and Systems Engineering Tonya L. Smith-Jackson, Ph.D., Chair Robert J. Beaton, Ph.D. Brian M. Kleiner, Ph.D. Wayne A. Scales, Ph.D. Robert C. Williges, Ph.D. December 2, 2003 Blacksburg, Virginia Keywords: Human-Computer Interaction, Usability Evaluation, Web-Based Distance and Distributed Learning (WD2 L), Instructional Design, e-Learning © 2003, Chang S. Nam

A Theory-Based Integrated Design Process for Development and Evaluation of Web-Based Supplemental Learning Environments

Chang S. Nam

ABSTRACT As an increasingly powerful, interactive, and dynamic medium for sharing information, the World Wide Web (Web) has found many educational applications. Because of educational benefits for both learners and educators, Web-based courses and programs have increasingly been developed and implemented by many academic institutions, organizations, and companies worldwide (Aggarwal & Brento, 2000). This study, however, asserts that many of the developmental approaches lack three important considerations to be used for implementing learning applications based on the Web: (1) integration of the human-computer interface design with instructional design, (2) development of the evaluation framework to improve the overall quality of Web-based learning support environments, and (3) development of an individualistic approach to the Web hosting of courses. This study explored the three Web-based learning environment weaknesses while developing a user-centered, Web-based learning support environment for Global Positioning System (GPS) education: Web-based distance and distributed learning (WD2L) environment. Research goals of the study were all concerned with the improvement of the design process and usability of the WD2L environment based on a theory-based Integrated Design Process (IDP) proposed in the study. Results indicated that the proposed IDP was effective in that the study showed (1) the WD2L environment’s equivalence to traditional supplemental learning, especially as a Web-based supplemental learning program and (2) users’ positive perceptions of WD2L environment resources. The study also confirmed that for an e-learning environment to be successful, various aspects of the learning environment as a Web-based supplemental learning program should be considered such as application domain knowledge (i.e., target subject field), conceptual learning theory, instructional design, human-computer interface design, and evaluation about the overall quality of the learning environment.

ACKNOWLEGEMENTS I am very grateful for having an exceptional doctoral committee. Without their guidance and support, this dissertation would not have taken its current form, nor would it have come to completion.

First and foremost, I must thank my advisor, Dr. Tonya Smith-Jackson, for

providing me with a great research opportunity and allowing me to work at my own pace with little supervision. She has gone beyond what an advisor should do, and I am extremely grateful for all her help. I owe a special note of gratitude to Dr. Robert Williges for his good advice and support throughout my graduate studies.

I would also like to thank Dr. Wayne Scales for being

extremely involved throughout my entire dissertation process. My deepest appreciation goes to Dr. Brain Kleiner for his valuable assistance and guidance. Dr. Robert Beaton’s insightful comments were crucial for editing the drafts into the final dissertation. I also offer my endless gratitude to Dr. Glenda Scales for her support and encouragement. Finally, I would like to acknowledge and thank my family; in particular, I would like to thank my parents, my wife, Jin, and my loving son, Andrew, for their love and unconditional support.

iii

TABLE OF CONTENTS ABSTRACT ..........................................................................................................................................................ii ACKNOWLEGEMENTS......................................................................................................................................iii LISTS OF TABLES ............................................................................................................................................viii LISTS OF FIGURES.............................................................................................................................................. x CHAPTER I. INTRODUCTION ............................................................................................................................ 1 Research Goals .................................................................................................................................................. 4 Outline of the Study........................................................................................................................................... 6 Language Conventions....................................................................................................................................... 8 CHAPTER II. LITERATURE REVIEW............................................................................................................... 10 Introduction ..................................................................................................................................................... 10 Overview of GPS Education ............................................................................................................................ 11 History of Learning Theories in Instructional Designs and Models.................................................................... 16 The Conceptual Learning Theories for Learning Environments .................................................................... 17 Behaviorist Approach to Learning ............................................................................................................................19 Cognitive Approach to Theory .................................................................................................................................21 Constructivist Approach to Learning ........................................................................................................................23

Instructional Design Principles and Models.................................................................................................. 25 Objectivist Instructional Systems Design Models (OISDMs) .....................................................................................25 Constructivist Instructional Systems Design Models (CISDMs).................................................................................27 Mixed Approach to Instructional Designs .................................................................................................................29

Implications for the Current Problems.......................................................................................................... 30 Human-Computer Interface Design for Distance Learning Environments.......................................................... 31 Challenges of Interface Design for Distance Learning Environments............................................................ 32 The Cognitive Perspective........................................................................................................................................32 The Communicative Perspective...............................................................................................................................33

Approaches to Interface Design: Wallace & Anderson’s (1993) Classification.............................................. 34 The craft approach ...................................................................................................................................................34 The enhanced software engineering approach ...........................................................................................................34 The technologist approach........................................................................................................................................35 The cognitive approach ............................................................................................................................................35

Implications for the Current Problems.......................................................................................................... 37 Evaluation of Web-based supplemental learning environments ......................................................................... 39 Models of Formative Evaluation on Distance Education............................................................................... 40 Van Slyke, Kittner, & Belanger’s (1998) Model........................................................................................................40 Marshall & Shriver’s (1994) Five-Level Model.........................................................................................................41 Kirkpatrick’s (1994) Four-Level Model ....................................................................................................................42 Dick & Carey’s (1996) Model ..................................................................................................................................44

iv

Usability Evaluation Approaches................................................................................................................. 45 The User-Based Evaluation Approach ......................................................................................................................45 The Expert-Based Evaluation Approach ...................................................................................................................48

Implications for the Current Problems.......................................................................................................... 50 Summary......................................................................................................................................................... 51 CHAPTER III. DEVELOPMENT OF WD2L ENVIRONMENT ........................................................................... 53 Description of the Integrated Design Process (IDP) .......................................................................................... 55 Phase 1: Needs Analysis.............................................................................................................................. 58 Process 1. Requirements Specification......................................................................................................................58 Process 2. Features & Components Identification......................................................................................................68 Process 3. Design Goals Setting................................................................................................................................72

Phase 2: Conceptual Design......................................................................................................................... 76 Process 1. Design Scenarios Development ................................................................................................................77 Process 2. Information Design..................................................................................................................................81 Process 3. Structure Design ......................................................................................................................................84 Process 4. Page Design.............................................................................................................................................88

Phase 3: Development ................................................................................................................................. 91 Process 1. Low-Fidelity Prototyping .........................................................................................................................91 Process 2. Design Walk-Through..............................................................................................................................94 Process 3. High-Fidelity Prototyping ........................................................................................................................97

Phase 4: Formative Evaluation..................................................................................................................... 99 Description of the Final WD2L Environment Prototype .................................................................................. 100 Overview of the WD2L Environment Prototype ......................................................................................... 100 Navigation Overview on the WD2L Environment Prototype ....................................................................... 105 Detailed Description of the WD2L Environment Prototype ......................................................................... 105 Summary....................................................................................................................................................... 128 CHAPTER IV. EVALUATION METHODS AND RESULTS............................................................................ 129 Description of the Formative Evaluation Framework ...................................................................................... 130 Expert Review (1st) Process............................................................................................................................ 133 Method ..................................................................................................................................................... 134 Participants............................................................................................................................................................134 Equipment/Apparatus.............................................................................................................................................135 Procedures.............................................................................................................................................................137

Results ...................................................................................................................................................... 138 Design Changes and Discussion ................................................................................................................ 141 One-to-One Evaluation Process...................................................................................................................... 146 Evaluation 1.............................................................................................................................................. 146 Method..................................................................................................................................................................146 Results...................................................................................................................................................................152

v

Design Changes and Discussion .............................................................................................................................155

Evaluation 2.............................................................................................................................................. 167 Method..................................................................................................................................................................167 Results...................................................................................................................................................................168 Design Changes and Discussion .............................................................................................................................171

Small Group Evaluations Process ................................................................................................................... 173 Method ..................................................................................................................................................... 173 Participants............................................................................................................................................................173 Experimental Materials and Benchmark Tasks ........................................................................................................173 Evaluation Criteria.................................................................................................................................................173

Results ...................................................................................................................................................... 176 Design Changes and Discussion ................................................................................................................ 177 Expert Review (2nd) Process ......................................................................................................................... 179 Method ..................................................................................................................................................... 179 Participants............................................................................................................................................................179 Equipment/Apparatus.............................................................................................................................................180 Procedures.............................................................................................................................................................180

Results ...................................................................................................................................................... 181 Design Changes and Discussion ................................................................................................................ 184 Field Experiment ........................................................................................................................................... 185 Research Questions ................................................................................................................................... 185 Method ..................................................................................................................................................... 186 Participants............................................................................................................................................................186 General Experimental Design .................................................................................................................................186 Independent Variables............................................................................................................................................186 Dependent Variables ..............................................................................................................................................187 Experimental Materials ..........................................................................................................................................187 Procedure ..............................................................................................................................................................190

Results ...................................................................................................................................................... 191 Discussion................................................................................................................................................. 193 Summary .................................................................................................................................................. 194 CHAPTER V. GENERAL DISCUSSION AND CONCLUSIONS...................................................................... 195 Discussion ..................................................................................................................................................... 195 Recommendations.......................................................................................................................................... 199 Limitations of the Study................................................................................................................................. 205 Areas of Future Research ............................................................................................................................... 206 Conclusions................................................................................................................................................... 208 REFERENCES .................................................................................................................................................. 209 Appendix 1: A Summary of Semi-Structured Interviews ..................................................................................... 218

vi

Appendix 2: A Summary of GPS Survey I .......................................................................................................... 224 Appendix 3: A Summary of GPS Survey II ......................................................................................................... 228 Appendix 4: Requirements Specification Document............................................................................................ 230 Appendix 5: The Content Outline Document....................................................................................................... 234 Appendix 6. A Summary of User Interface Review ............................................................................................. 235 Appendix 7. A Summary of Instructional Design Review.................................................................................... 243 Appendix 8. A Summary of Content Review....................................................................................................... 249 Appendix 9. Task Instructions ............................................................................................................................ 258 Appendix 10. IRB Protocol................................................................................................................................ 259 Appendix 11. Benchmark Tasks ......................................................................................................................... 261 Appendix 12. Data Collection Sheet ................................................................................................................... 267 Appendix 13. Sample Questions of Instruction Evaluation .................................................................................. 267 Appendix 14. Sample Questions of the QUISTM .................................................................................................. 268 Appendix 15. Scoring Rubrics for Transfer Test.................................................................................................. 269 Appendix 16. Sample Questions of Web-based Learning Questionnaire .............................................................. 270 VITA ................................................................................................................................................................. 272

vii

LISTS OF TABLES Table 2-1. Challenging Situations in Development of GPS Learning Support Environment.................................... 12 Table 2-2. Importance of Learning Domains in GPS Education ............................................................................. 13 Table 2-3. Examples of Required Skills to Master GPS Fundamentals................................................................... 14 Table 2-4. Comparisons of Three Views of Learning............................................................................................. 18 Table 2-5. Cognitive Learning Principles and Possible ID Applications................................................................. 22 Table 2-6. Typical Objectivist Instructional Design Approach ............................................................................... 26 Table 2-7. Comparison of Three Instructional Design Approaches......................................................................... 30 Table 2-8. Four Categories of Methods in the Cognitive Approach........................................................................ 36 Table 2-9. Examples of Web Design Guidelines and Main Categories ................................................................... 38 Table 2-10. Precedent and Outcome Variables in Van Slyke et al. (1998) Model ................................................... 41 Table 2-11. Variables to Consider in Marshall & Shriver’s (1994) Five-level Model.............................................. 42 Table 2-12. Summary of Kirkpatrick’s (1994) Four-level Model ........................................................................... 43 Table 2-13. Examples of User-based Usability Evaluation Approach..................................................................... 46 Table 2-14. Sample Measures of Performance and Preference ............................................................................... 46 Table 2-15. Examples of Expert-based Usability Evaluation Approach.................................................................. 48 Table 2-16. Ten Usability Heuristics by Nielsen (1993)......................................................................................... 49 Table 3-1. Descriptions of the UCD Principles Applied in the IDP ........................................................................ 55 Table 3-2. A Summary of Methods Used in the Requirements Specification Process ............................................. 60 Table 3-3. Sample Interview Questions and Responses ......................................................................................... 60 Table 3-4. Description of Student User Characteristics.......................................................................................... 62 Table 3-5. Student Users’ Motivation Levels......................................................................................................... 63 Table 3-6. Potential User Tasks on the WD2L Environment................................................................................... 64 Table 3-7. Examples of Interface Design Requirements that Support User Tasks ................................................... 65 Table 3-8. Examples of Features and Components Associated with WBI............................................................... 68 Table 3-9. Functions of Online Learning Tools ..................................................................................................... 70 Table 3-10. Key Features and Components of WD2L Environment........................................................................ 71 Table 3-11. Design Goals for WD2L Environment Development ........................................................................... 74 Table 3-12. Instructional Design Principles and Applications ................................................................................ 75 Table 3-13. Key User Tasks on the WD2L Environment........................................................................................ 79 Table 3-14. An Example of Theory-Based Design of Learning Content ................................................................. 81 Table 3-15. An Example of the Content Outline Document ................................................................................... 83 Table 3-16. 23 Screens of the Final Low-Fi Prototype ........................................................................................... 93 Table 3-17. A Summary of the Initial User Feedback ............................................................................................ 95 Table 3-18. Menu Structure of WD2L Environment Prototype............................................................................. 102

viii

Table 3-19. Navigation Tools and Descriptions................................................................................................... 105 Table 3-20. Available Functions on the Search Box ............................................................................................ 107 Table 3-21. Available Functions on the Chapter 5 Page....................................................................................... 110 Table 3-22. Available Functions on the Prelaboratory Page ................................................................................. 118 Table 3-23. Entry Fields on the Assignments-Submit Page.................................................................................. 120 Table 3-24. Available Functions on the Assignments-Submit Page...................................................................... 120 Table 3-25. Entry Fields on the Send Email Page ................................................................................................ 125 Table 3-26. Available Functions on the Discussion Board Page........................................................................... 125 Table 3-27. Entry Fields on the Discussion Board – Write page........................................................................... 127 Table 4-1. Examples of Expert Comments/Recommendations on User Interface System...................................... 138 Table 4-2. A Rating Summary of User Interface System...................................................................................... 138 Table 4-3. Examples of Expert Comments on Instructional Design...................................................................... 139 Table 4-4. Examples of Expert Comments/Recommendations on Learning Units................................................. 140 Table 4-5. A Summary of Rating on Learning Units............................................................................................ 140 Table 4-6. Benchmark Tasks and Main Testing Objective ................................................................................... 147 Table 4-7. Usability Metrics and Measurements.................................................................................................. 148 Table 4-8. Usability Specifications and Target Levels ......................................................................................... 149 Table 4-9. A Summary of Usability Specifications: Evaluation 1......................................................................... 152 Table 4-10. A Summary of Usability Specifications: Evaluation 2....................................................................... 168 Table 4-11. Usability Metrics for the Small Group Evaluation............................................................................. 174 Table 4-12. Usability Specifications used for the Small Group Evaluation ........................................................... 174 Table 4-13. A Summary of Usability Specifications: Small Group Evaluation ..................................................... 176 Table 4-14. Examples of Expert Comments/Recommendations on User Interface System .................................... 181 Table 4-15. A Rating Summary of User Interface System.................................................................................... 181 Table 4-16. Examples of Expert Comments on Instructional Design.................................................................... 182 Table 4-17. Examples of Expert Comments/Recommendations on Learning Units............................................... 183 Table 4-18. A Summary of Rating on Learning Units.......................................................................................... 183 Table 4-19. An Example of Scoring Rubrics for Transfer Test............................................................................. 189 Table 4-20 Means and Standard Deviations for Recall and Transfer of Knowledge .............................................. 191 Table 4-21. Frequency Table: Objectives, Content, Interactivity, and Navigation................................................ 192 Table 4-22. Frequency Table: Text, Graphics, Sound, and Interface .................................................................... 193 Table 5-1. Reigeluth’s (1999) Three Criteria for Evaluating Design Process Models............................................ 201 Table 5-2. Check Points for Design Processes ..................................................................................................... 202 Table 5-3. General Design Guidelines for Web-Based Supplemental Learning Environments .............................. 203 Table 5-4. Design Guidelines for Web-Based Supplemental Learning Components ............................................. 204

ix

LISTS OF FIGURES Figure 2-1. Relationship between Learning Theories and Instructional Design Models .......................................... 25 Figure 2-2. An Example of Constructivist Model for Instructional Design ............................................................. 28 Figure 3-1. Integrated Design Process (IDP) Proposed for the WD2L Environment Development .......................... 54 Figure 3-2. An Example of the Design Process Template ...................................................................................... 57 Figure 3-3. Design Process Template: Requirements Specification Process............................................................ 59 Figure 3-4. Design Process Template: Features and Components Identification Process......................................... 69 Figure 3-5. Design Process Template: Design Goals Setting.................................................................................. 73 Figure 3-6. 3A Transformation Model................................................................................................................... 76 Figure 3-7. Design Process Template: Design Scenarios Development Process...................................................... 78 Figure 3-8. An Example of the Design Scenario.................................................................................................... 80 Figure 3-9. Design Process Template: Information Design Process........................................................................ 82 Figure 3-10: Design Process Template: Structure Design Process .......................................................................... 85 Figure 3-11. An Example of Key User Paths......................................................................................................... 86 Figure 3-12. An Example of the Hierarchical Structure ......................................................................................... 87 Figure 3-13. An Example of the Storage Structure................................................................................................. 87 Figure 3-14. Design Process Template: Page Design Process................................................................................. 89 Figure 3-15. An Example of the Wireframe........................................................................................................... 90 Figure 3-16. Design Process Template: Low-Fidelity Prototyping Process............................................................. 92 Figure 3-17. Design Process Template: Design Walk-Through Process ................................................................. 96 Figure 3-18. Design Process Template: High-Fidelity Prototype Process ............................................................... 98 Figure 3-19. Basic Structure of the WD2L Environment ...................................................................................... 100 Figure 3-20. Header Section of the WD2L Environment ...................................................................................... 101 Figure 3-21. Menu Bar Section of the WD2L Environment.................................................................................. 101 Figure 3-22. Navigation Path Section of the WD2L Environment......................................................................... 102 Figure 3-23. Left-Hand Side Navigation Section of the WD2L Environment........................................................ 103 Figure 3-24. Content Display and Footer Navigation Sections ............................................................................. 104 Figure 3-25. Log-in (Home Page) Screen ............................................................................................................ 106 Figure 3-26. Announcements Screen................................................................................................................... 108 Figure 3-27. Course Information Screen.............................................................................................................. 108 Figure 3-28. Staff Information Screen................................................................................................................. 109 Figure 3-29. Classroom Guide Screen................................................................................................................. 110 Figure 3-30. An Example of the Main Screen for the Chapter 5 Page................................................................... 111 Figure 3-31. An Example of Learning Unit Screen.............................................................................................. 112 Figure 3-32. An Example of the Practice Session Screen ..................................................................................... 113 Figure 3-33. Concept Map Screen....................................................................................................................... 114

x

Figure 3-34. GPS Resources Screen.................................................................................................................... 115 Figure 3-35. GPS Glossary Screen ...................................................................................................................... 116 Figure 3-36. Prelaboratory Screen....................................................................................................................... 117 Figure 3-37. Assignments Screen........................................................................................................................ 119 Figure 3-38. Quiz Review Main Screen............................................................................................................... 121 Figure 3-39. Quiz Review: Question Screen........................................................................................................ 122 Figure 3-40. Quiz Review: Feedback Screen ....................................................................................................... 123 Figure 3-41. Send Email Screen.......................................................................................................................... 124 Figure 3-42. Discussion Board: Main Screen....................................................................................................... 126 Figure 3-43. Discussion Board – Write Screen .................................................................................................... 127 Figure 4-1. A Formative Evaluation Framework Proposed for Evaluating the WD2L Environment....................... 131 Figure 4-2. Design Changes to the Home page .................................................................................................... 141 Figure 4-3. Design Change to the GPS Glossary Page ......................................................................................... 142 Figure 4-4. Design Changes to the Send Email Page ........................................................................................... 143 Figure 4-5. An Example of Graphic Design Changes........................................................................................... 143 Figure 4-6. Design Change to the Chapter 5 Page................................................................................................ 144 Figure 4-7. Design Change to the Discussion Board............................................................................................ 145 Figure 4-8. Menu System in the Second Version of the Prototype........................................................................ 155 Figure 4-9. Menu System in the Third Version of the Prototype .......................................................................... 156 Figure 4-10. The Assignments Page in the Second Version of the Prototype ........................................................ 157 Figure 4-11. Design Changes to the Assignments Page........................................................................................ 158 Figure 4-12. Design Changes to the Send Email Page.......................................................................................... 160 Figure 4-13. Design Changes to the Chapter 5 Page: Animation .......................................................................... 161 Figure 4-14. Design Changes to the Chapter 5 Page: Tidal Effects Link............................................................... 162 Figure 4-15. Design Changes to the Practice 1 Page ............................................................................................ 163 Figure 4-16. Design Changes to the Quiz Review Page ....................................................................................... 164 Figure 4-17. Design Change to the Prelaboratory 5 Page ..................................................................................... 165 Figure 4-18. Design Change to the Chapter 5 Page: Solar Effects Link................................................................ 171 Figure 4-19. Design Change to the Prelaboratory 5 Page: Add Save Function...................................................... 172 Figure 4-20. Design Change made to the Discussion Board: Group Selection Options ......................................... 177 Figure 4-21. Design Change made to the Discussion Board: Group Selection Options ......................................... 178 Figure 4-22. General Experimental Design.......................................................................................................... 186 Figure 5-1. Revised Integrated Design Process (IDP) for the WD2L Environment Development .......................... 199

xi

1

CHAPTER I INTRODUCTION As an increasingly powerful, interactive, and dynamic medium for delivering information, the World Wide Web (Web) in combination with information technology (e.g., network technology such as LAN, WAN or Internet), has found many applications. One popular application has been for educational use, such as Web-based, distance, distributed or online learning. The use of the Web as an educational tool has provided learners with a wider range of new and interesting learning experiences, not possible in traditional in-class education (Khan, 1997).

For example, learning environments created on the Web are independent of time and

space. The students and teacher are not necessarily in the same place interacting at the same time. Web-based learning allows learners who are geographically dispersed to study at their own pace in their own time. In addition, Web-based learning helps computer-supported collaborative learning (CSCL), in which distant learners collaborate on a learning activity with a shared outcome with the teacher and other learners through various communication media, such as email, a discussion board, or computer conferencing activity (Abrami & Bures, 1996; Winer, Chomienne, & Vázquez-Abad, 2000).

The Web also allows access to a variety of rich

information and knowledge resources distributed globally. For example, engineering students can access a number of the web sites that provide information about modern technology and share that information with other learners who have the same interests.

According to

McCormack & Jones (1997), Web-based learning also offers an environment for educators, in which they can handle the whole process of classroom management more efficiently. The educator can automatically track the students’ attendance, monitor their academic progress, and manage their grades. Because of these educational benefits for both learners and educators, Web-based courses and programs have increasingly been developed and implemented by many academic institutions, organizations, and companies worldwide (Aggarwal & Brento, 2000). For example, a number of traditional universities in the United States offer Web-based courses, including Harvard University (www.extension.harvard.edu), Duke University (www.fuqua.duke.edu), and New York University (www.scps.nyu.edu/online/index.jsp/). Lucent Technology’s Center for

Chang S. Nam

Chapter I. Introduction

2

Excellence in Distance Learning (www.lucent.com/cedl/) is one example of corporate training through distance learning. These Web-based learning environments have been developed mainly by instructional designers, using traditional instructional design models such as Dick & Carey’s Instructional Systems Design (1996), Spiro, Feltovich, Jacobson, & Coulson’s Cognitive Flexibility Theory (1991), and Jonassen’s Constructivist Learning Environment (1999). This study, however, asserts that many of these approaches lack three important considerations to be used for implementing learning applications based on the Web: (1) integration of the humancomputer interface design with instructional design, (2) development of the evaluation framework to improve the overall quality of Web-based learning support environments, and (3) development of an individualistic approach to the Web hosting of courses. First, little attention has been paid to design issues of the human-computer interface system, which is one of the most important components in the Web-based learning environment (Plass, 1998). Learners must be able to easily focus on learning materials without having to make an effort to figure out how to access them (Lohr, 2000). An effective user interface can fulfill that requirement. However, many instructional design principles and models proposed for the development of Web-based learning environments have not explicitly addressed usability issues of the human-computer interface system, which are critical factors to the success of Webbased instruction (Boiling & Sousa, 1993; Henke, 1997). The main focus of these instructional design models has been on the effectiveness of instructional interventions. In effect, there exist many interface design guidelines available for instructional designers, but these guidelines were intended for software environments rather than learning environments (Jones, Farquhar, & Surry, 1995).

Therefore, for a successful implementation of Web-based learning environments, a

design approach that integrates design issues of the human-computer interface system with those of the instructional system is required. Second, the rapid growth of Web-based learning applications has generated a need for methods to systematically collect continuous feedback from users to revise or improve learning environments. Unfortunately, few attempts have been made to develop formative evaluation frameworks for Web-based learning environments whose foci are both the instructional system and user interface system. There are many formative evaluation models of distance education

Chang S. Nam


3

(e.g., Dick & Carey’s (1996) model; Kirkpatrick’s (1994) four-level model; Marshall & Shriver’s (1994) five-level model; Van Slyke, Kittner, & Belanger’s (1998) model). However, few approaches take problems of the user interface design into account in their evaluation processes. A number of evaluation frameworks that can be used to evaluate the user interfaces have also been proposed (e.g., Nielsen, 1993; Rubin, 1994). These models are intended for software environments rather than for Web-based learning environments in which user interface systems should be developed to support users’ learning activities.

Therefore, Web-based

learning environments require an evaluation framework in which evaluation processes, methods, and criteria are provided to identify problem areas or to draw inferences about the overall quality of both the instructional system and user interface system, which are two main components of Web-based learning environments. Finally, there are few individualistic approaches to the Web hosting of courses. Webbased course materials are mainly hosted on central servers provided by institutions who maintain control over those materials. In the study on desktop distance education, Norman (2000) provided the pros and cons of the institutional versus personal approaches to the Web hosting of courses. For example, institutional hosting is appropriate for Web-based course materials of large standardized courses that need institutional support, such as Introduction to Psychology, Chemistry 101, and Algebra 101. However, desktop hosting or personal Web server hosting located in the faculty member’s office and laboratory can be the ideal solution when an individual course (e.g., Introduction to GPS Theory and Design to be developed in this study) is developed and faculty desire ownership of resources. Many development approaches to distance learning have been proposed for the institutional hosting of the materials, which may not be appropriate for personal hosting of Web course materials. Selecting which Web-hosting approach to take may affect how the Web-based learning environment is developed with regard to analysis, design decisions, and design steps to take. Unlike centralized, institutional hosting approaches, for example, personal hosting of materials should be different in design decisions about the human-computer interface, in which the instructor has the control of all aspects of the materials including navigation (Norman, 2000). In addition, personal hosting of materials will require a smaller range of the analysis and design processes than institutional hosting approaches.

Chang S. Nam


4

Research Goals This study explored the three Web-based learning environment weaknesses listed before while developing a user-centered, Web-based learning support environment for Global Positioning System (GPS) education: Web-based distance and distributed learning (WD2 L) environment. There are three research goals addressed in this study and these goals are all concerned with the improvement of the design process and usability of the WD2L environment. First, this study offered a systematic approach to the development of a user-centered, WD2L environment for supporting engineering courses. The implementation of Web-based learning environments involves the consideration of various aspects of the learning environment, such as application domain knowledge (i.e., target subject field), conceptual learning theory, instructional design, human-computer interface design, and evaluation about the overall quality of the learning environment. Instruction and user interface systems, two main components that make up a WD2L environment, should also be systematically considered together to ensure the success of the Web-based learning environment. These demands make the development of Webbased learning environments an increasingly challenging endeavor.

Unfortunately, few

frameworks for developing the WD2L environment to support engineering education are available, which systematically consider various aspects of the learning environment. To help simplify the challenges while improving the design process of the WD2L environment, this study proposed a new framework, the Integrated Design Process (IDP), which integrates humancomputer interface design with instructional design.

This study also provided the Design

Process Template to effectively implement design processes. Various aspects of the learning environment that should be considered are constantly changing due to changes in technology, course structure, and users’ needs. The Design Process Template can help consider those factors when implementing each design process. Second, based on the proposed framework, the WD2L environment was developed to support learning activities in an engineering field (i.e., GPS education), in which engineering students, including distant learners, can easily pull together many fundamental concepts and effectively gain hands-on experience with the GPS technology at their own pace in their own

Chang S. Nam


5

time. This document described the effort involved in developing the WD2L environment within the context of Global Positioning System (GPS) education in the Bradley Department of Electrical and Computer Engineering (ECE) at Virginia Polytechnic Institute and State University (VT). Finally, this study evaluated the design process model by assessing the overall quality of the WD2L environment prototype developed using the proposed Integrated Design Process. The effectiveness of the WD2L environment, which was modified several times to develop a usable learning environment through a proposed formative evaluation framework, was evaluated by assessing students’ 1) learning performance and 2) evaluation about the quality of resources implemented in the WD2L environment.

Chang S. Nam


6

Outline of the Study The dissertation is organized as follows. Chapter II reviewed relevant literature to guide the design, development, evaluation, and design changes of the WD2L environment that supports GPS education. First, the GPS course (i.e., Introduction to GPS Theory and Design) was reviewed to identify situations that could be challenging across five dimensions: context, delivery mode, time frame, content, and audience characteristics.

These challenging aspects also served as a basis of drawing practical

implications from the literature reviewed.

For example, the reviewed literature provided

practical implications for the current research problems and will be described at the end of the review sections.

Second, in order to understand how people learn and to incorporate that

knowledge when designing instructional systems, three theoretical positions to learning were outlined.

These conceptual learning theories included behaviorism, cognitivism, and

constructivism in relation to learning.

Commonly used instructional design models and

principles were reviewed to find an effective way of designing the instructional system for the GPS course. Third, human-computer interface design issues were discussed, which should be considered when developing the human-computer interface system of WD2L environments. Various approaches to user interface design necessary for designing a usable interface that is also learning-centered were reviewed. Fourth, several useful approaches and models that can be used to identify problem areas or to draw inferences about the overall quality of WD2L environments were discussed. In addition, three main approaches to the use of design guidelines as a tool to improve usability were reviewed: design guidelines supporting tools, user-centered guidelines categorization, and a layered model of usability. Finally, a general discussion on the design, development, evaluation, and design changes of the WD2L environment was included. Chapter III provided an overview of the development process of the WD2L environment prototype developed using the proposed Integrated Design Process (IDP). First, the proposed IDP was described, which consists of four design phases: needs analysis, conceptual design, development, and formative evaluation. Second, workflow steps of the IDP were outlined, describing processes of building the prototype in more detail.

Third, the resulting WD2L

Chang S. Nam


7

environment prototype was overviewed, describing the final user interface system and instructional system prototype. Finally, a general discussion on the development of the WD2L environment prototyping was included. Chapter IV summarized evaluation processes and design changes made according to evaluation results. First, the proposed formative evaluation framework was briefly described, consisting of the expert review (1st), one-to-one evaluation, small group evaluation, and expert review (2nd). Second, each evaluation process was summarized, along with the obtained results and design changes made to the WD2L environment prototype. Finally, a general discussion on the formative evaluation of the WD2L environment prototype was summarized. Chapter V provided a general discussion of the derived results. The discussion of the findings was divided into four sections. (1) Discussion on relationships between the design process framework and effectiveness of the WD2L environment; (2) Recommendations for applications of and modifications to the proposed framework and WD2L environment developed in the study; (3) Limitations of the study; (4) Suggestions for further research in this area.

Chang S. Nam


8

Language Conventions Throughout the dissertation, several terms have been used in specific ways: User and learner: These two terms are used almost interchangeably to refer to any individual who is involved in the use of the WD2L environment. For example, students learning GPS fundamentals through the WD2L environment are learners from the instructional design perspective, but they are also users from the human-computer interaction point of view who are using the interface system to access to learning materials. System and learning environment: The term system is used in two ways. First, it is used to refer to the instructional system, which includes all instructional design activities, such as the design of learning materials and instructional interventions (e.g., assessments and instructional strategies). Second, the term system is also used to refer to the user interface system, which includes all interface components. That is, the instruction and user interface system refer to the two main systems that make up the Web-based distance learning environment. Web-based Instruction (WBI) and Web-based education: These two terms are used interchangeably. This study followed Khan’s (1997) definition of WBI as “a hypertext-based instructional program which utilizes the attributes and resources of the World Wide Web to create a meaningful learning environment where learning is fostered and supported” (p. 6). Collaborative learning (CL): Collaborative learning is defined as learning performed as a group for a common purpose (Smith & McGregor, 1992). CL is based on the premise that computer systems can support and facilitate the group process as well as group dynamics in ways that are not achievable in traditional face-to-face education, but they are not designed to replace face-to-face/traditional learning environments. collaborative learning (CSCL).

It is also referred to as computer-supported

Chang S. Nam


9

Distance learning (DL1): As Perraton states (1988, in Schlosser & Anderson, 1994), distance learning is “an educational process in which a significant proportion of the teaching is conducted by some removed in space and/or time from the learner” (p.34). The U.S. Department of Education’s Office of Educational Research and Improvement (in Bruder, 1989) defines distance education as “the application of telecommunications and electronic devices which enable students and learners to receive instruction that originates from some distance location” (p. 30). Distributed learning (DL2): The concept of distributed learning refers to different kinds of distribution: across time, space, and individuals (Winer et al., 2000). Saltzberg & Polysin (1995) define distributed learning as “an instructional model that allows instructor, students, and content to be located in different, non-centralized locations so that instruction and learning occur independent of time and place” (p. 11). Web-based distance and distributed learning (WD2L) environment:

The WD2L

environment refers to an integrated learning environment developed to support both distance (DL1) and distributed learning (DL2). For engineering students including distant learners, for example, the WD2L environment allows them to easily pull together many fundamental concepts and effectively gain hands-on experience with contemporary technology at their own pace in their own time.

10

CHAPTER II LITERATURE REVIEW Introduction This chapter reviews relevant literature that guided the design, development, and evaluation of the WD2 L environment prototype for supporting GPS education. First, as a part of obtaining application domain knowledge, a GPS course (i.e., Introduction to GPS Theory and Design) provided by the Bradley Department of Electrical and Computer Engineering at Virginia Polytechnic Institute and State University (VT) was reviewed. The GPS course was reviewed to identify developmental situations that could be challenging across five dimensions: context, delivery mode, time frame, content, and audience characteristics. These challenging situations served as a basis from which to draw practical implications from the literature. For example, several implications that the reviewed literature provides for the current research problems were summarized at the end of review sections. Second, in order to understand how people learn and to incorporate that knowledge when designing instructional systems, three theoretical positions to learning were outlined.

These conceptual learning theories included Behaviorism,

Cognitivism, and Constructivism in relation to learning (Driscoll, 2000).

Commonly used

instructional design models and principles were reviewed to find an effective way (or ways) of designing the instructional system for the GPS course. Third, design issues of the humancomputer interface were discussed, which should be considered when developing the humancomputer interface system of WD2 L environments. Various approaches to user interface design necessary to design a usable interface that is also learning-centered were reviewed. Finally, a general discussion on the design, development, evaluation, and design changes of the WD2L environment was included.

Chang S. Nam

Chapter II. Literature Review

11

Overview of GPS Education The Global Positioning System (GPS) is a satellite-based navigation system developed by the U.S. Department of Defense, in which 24 satellites orbiting the Earth provide coded satellite signals that can be processed into the position, velocity and time by a GPS receiver. The GPS was originally intended for military applications. Since the military released GPS technology for public use, however, an explosion of commercial GPS products has flooded the market. Because of such a rapid growth in applications for the GPS and its technology, the demand for engineers educated in GPS fundamentals and techniques has increased as well. To keep up with the trend in GPS technology, several engineering departments, including Electrical and Computer, Aerospace, and Civil engineering, have recommended that their students take GPS related courses. At Virginia Polytechnic Institute and State University (VT), the Bradley Department of Electrical and Computer Engineering (ECE) currently offers a course, Introduction to GPS Theory and Design, consisting of three hours of lecture as well as three hours of laboratory work each week. The overall objectives of this course are to provide students with the fundamental theory and concepts of the GPS and to demonstrate these concepts through laboratory exercises and a final design project. There is the educational demand for a new learning environment to effectively support the course, while meeting the societal demand on engineers educated in GPS fundamentals. However, there are also developmental situations that could be challenging, and thus these issues should be addressed before the development of the new GPS learning support environment. In this section, these issues are discussed across five dimensions: the context, delivery mode, time frame, content, and audience characteristics.

Table 2-1 shows the five dimensions and

corresponding challenging issues. Context: Rapid growth in GPS applications has driven the demand for engineering students to be educated in GPS fundamentals and techniques. To meet this demand, the GPS course needs to be redesigned so that anyone who wants to learn the GPS fundamentals can take advantage of the course easily. Consequently, a new learning environment must make this newly

Chang S. Nam


12

designed instruction possible to deliver on-campus students as well as learners who are geographically dispersed (e.g., those at VT Northern Virginia campus or practicing engineers working in the corporate sector as well as government laboratories). Table 2-1. Challenging Situations in Development of GPS Learning Support Environment Dimension Context Delivery Mode Time Frame

Content

Audience

Challenging Issues • Societal demand on engineering students educated in GPS fundamentals • Development of a new GPS learning support environment (LE) • Redesign of the course relevant for the new LE • Delivery of the course independent of geographic location • Supplemental mode to existing instruction methods • Learning experiences independent of time • At own space in own time • Interdisciplinary subject area • Various types of learning outcomes • Implementation of laboratory exercises • Support of collaboration learning (e.g., group discussion and design project) • Diverse educational backgrounds • Geographically dispersed learners

Delivery Mode: Currently, the GPS course is provided in the traditional classroom and laboratory in which instructor and students must be in the same place and same time. Making the course available to distant learners generates a need for a delivery method independent of geographic locations. As a supplement to existing teaching methods (i.e., face-to-face learning), the course needs to be redesigned so that on-campus students can review GPS fundamentals whenever they want. The course involves a wide spectrum of engineering areas that are not easily understood as well as engineering practices that will be accomplished through laboratory exercises, which are one of the most important learning activities in GPS education. Thus, providing students with a learning opportunity that they can use anytime will help them master GPS fundamentals more effectively. Time Frame: As with the current GPS course, the classroom instruction requires both instructor and students to get together at the same time. On the other hand, the new GPS learning support environment should provide students with the learning opportunity independent of time so that they can study the course at their own time in their own space.

Chang S. Nam Content:


13

The GPS course involves many fields of engineering. To master GPS

fundamentals, for example, students must utilize knowledge in a wide spectrum of engineering areas, including satellite orbit theory, radio-wave propagation, communication system theory, signal processing, probability and statistics, and numerical analysis techniques. GPS education also involves various types of learning outcomes. Table 2-2 shows the importance of the different categories of learning that may occur in the GPS course, which was analyzed through Gagne’s (1985) four domains of learning. Table 2-2. Importance of Learning Domains in GPS Education Importance 1 2 3 4

Learning Domains • Intellectual skills • Verbal information • Attitudes • Psychomotor skills

As shown in Table 2-2, intellectual skills are ranked as the most important learning domain for the GPS course, followed by verbal information, attitudes, and psychomotor skills. Intellectual skills refer to the learner’s interaction with “the environment by using symbols” (p. 47). There are three types of intellectual skills, such as discriminations, concepts, and principles (rules). The reasons for intellectual skills being placed on the most important learning domain will become clear when considering the skills required for mastering GPS fundamentals, as shown in Table 2-3. As shown in Table 2-3, a variety of skills are required, but intellectual skills such as discriminations, concepts, and rules, are always required when studying fundamental theories and concepts of GPS, practicing them through laboratory exercises and applying them to real problem situations. For example, when choosing adequate theories, concepts, and equations to apply, a learner must be able to tell if the particular two theories, concepts, and/or equations are alike or different in one or two respects (Discriminations). In addition, the learner must be able to categorize the situations wherein that theory, concept, or a certain equation is used, and to understand what it means (Concepts). To understand and actually use the algorithms for the computations, the learner needs to identify relationships between the algorithms (Rules).

Chang S. Nam


14

Table 2-3. Examples of Required Skills to Master GPS Fundamentals GPS Fundamental To choose adequate theory, concepts, and equations To understand the algorithms for the computation To identify the differences among various GPS concepts To perform laboratory exercise

To conduct a design project

Required skill • Intellectual (Discriminations & Concepts) • Verbal information • Attitude • Intellectual (Concepts & Rules) • Intellectual (Discriminations, Rules, & Concepts) • Verbal information • Attitude • Intellectual (Discriminations, Rules, & Concepts) • Verbal information • Attitude • Intellectual (Discriminations, Rules, & Concepts) • Attitude

The next important domain of learning is verbal information. Verbal information allows a learner to declare or state something about the GPS fundamentals. This involves the ability of the learner to orally state the important things in the GPS fundamentals. For example, learners must be able to state which concepts or equations are required for a given situation and be able to tell why they are required when performing laboratory exercises as well as when conducting a design project. Attitudes were ranked as the third learning domain. This involves choosing to do something or valuing something. In other words, attitudes are tendencies to make particular decisions under particular circumstances. Performing laboratory exercises and conducting a design project require learners to make such decisions. One of the most important learning activities in GPS education is to train students to actively integrate many fundamental GPS concepts by gaining hands on experience with modern technology. Laboratory exercises in the GPS course are important because those exercises allow students to reinforce knowledge obtained during the lectures as well as to pursue their own interests and design problems. Again, it becomes important to implement laboratory exercises that involve the use of GPS equipment in the WD2 L environment (e.g., a GPS receiver to get real GPS data and MATLAB scripts to test them). As with laboratory activities in GPS education, a design project is also a very important learning activity. The design project involves learning in which students collaborate on a project with the computer-supported tools such as the GPS receiver and MATLAB scripts.

Chang S. Nam


15

Audience Characteristics: Students taking the GPS course have diverse educational backgrounds. In general, 30-35 students attend the course (the department limits enrollment to 35 students). The course is composed of roughly 75% electrical engineering students and 25% aerospace engineering students. Of the total number of students, roughly 30% are graduate students and 70% are undergraduate students. When considering distant learners (e.g., students on other campuses and practicing engineers), differences in educational level and demographics increase. Students also have diverse educational backgrounds as they have taken different courses and studied different fields.

Therefore, some of the students (e.g., electrical and

computer engineering students) may already know a certain topic, but others (e.g., civil and aerospace engineering students) may not be versed in that area. This issue leads to the challenge of designing learning content that is relevant for students with diverse educational backgrounds.

Chang S. Nam


16

History of Learning Theories in Instructional Designs and Models The overview of the GPS course in the previous section reveals that the development of a new GPS learning support environment involves considering various developmental situations. Given the characteristics of the course and audience, for example, an effective design of the instructional system is required. For an instructional system to be effective, it is important to understand how people learn and to incorporate that knowledge when developing the instructional system in terms of the learning content and instructional interventions. Which theoretical position to take significantly affects the design of the instructional system.

As

Bannan & Milheim (1997) pointed out, “the deliberate selection of a particular theoretical position provides the necessary basis for corresponding instructional methods” (p. 382). To effectively design the learning content and instructional interventions such as instructional strategies for the GPS course, this study viewed three main theoretical explanations of how people learn: Behaviorism, Cognitivism, and Constructivism. This section first compared the three underlying philosophical views of knowledge and learning across eight dimensions, aimed at identifying theoretical position(s) to learning theories relevant for designing the learning content of the GPS course.

Then, commonly used

instructional design models and principles were reviewed, which have been affected by diverse approaches to learning. The purpose was to identify an instructional design model that can be used for designing the overall instructional interventions such as instructional strategies and assessment for the GPS course. Finally, implications for the current problems were discussed. A potential theoretical position(s) to learning theory that can be used for designing the learning content was presented.

In addition, an instructional design approach that is relevant for

designing instructional interventions for the GPS course was presented.

Chang S. Nam


17

The Conceptual Learning Theories for Learning Environments To identify a particular learning theory or theories that can be used for designing the learning content of the GPS course, this study reviewed three main learning theories: Behaviorism, Cognitivism, and Constructivism.

These conceptual learning theories were

compared across eight dimensions. In a review of learning theories from an instructional design perspective, Ertmer & Newby (1993) used seven dimensions to compare critical features of the theories. These dimensions included 1) How does learning occur? 2) Which factors influence learning? 3) What is the role of memory? 4) How does transfer occur? 5) What types of learning are best explained by the theory? 6) What basic assumptions/principles of this theory are relevant to instructional design? and 7) How should instruction be structured to facilitate learning? To identify challenges that the designer may face when he/she applies each of the three learning theories to an engineering curriculum, this study added one more dimension: 8) What challenges are there when applied to the engineering curriculum? This dimension will be helpful to choose a particular theoretical position to learning theory that is relevant for the GPS course, which is an engineering course. Table 2-4 briefly summarizes the comparison of the three views of learning from an instructional design perspective.

Chang S. Nam


18

Table 2-4. Comparisons of Three Views of Learning Dimension How does learning occur? Which factors influence learning?

What is the role of memory?

Behaviorism • When a proper response is demonstrated after an environmental stimulus • Learner & environmental factor • Arrangement of stimuli & consequences within the environment • Not addressed by behaviorist • Forgetting: nonuse of a response over time

How does transfer occur?

• Generalization of identical or similar features

What types of learning are best explained by the theory? What basic assumptions/princ iples of this theory are relevant to instructional design?

• Discriminations • Generalizations • Associations • Chaining • An emphasis on producing observable & measurable outcomes in students • Pre-assessment of students to determine where instruction should begin • Emphasis on mastering early steps before progressing to more complex levels of performance • Use of reinforcement to impact performance • Use of cues, shaping & practice to ensure a strong stimulus-response association • Structured around the presentation of the target stimulus & the provision of opportunities for the learner to practice making the proper response • Use cues to prompt the response & reinforcement to strengthen correct response • Cannot adequately explain the acquisition of higher skills or those that require a greater depth of processing (e.g., problem solving, inference generating, and critical thinking)

How should instruction be structured to facilitate learning?

What challenges are there when applying to engineering curriculum?

Cognitivism • Through mental activity that entails internal coding & structuring information • Ways to attend, code, transform, rehearse, store & retrieve information • Learners’ thoughts, beliefs, attitudes, & values • Storing information in an organized, meaningful manner • Forgetting: inability to retrieve information from memory • When understanding how to apply knowledge in different contexts • Complex forms of learning (e.g., reasoning, problem-solving, information processing)

Constructivism • Creation of meaning from experience

• Emphasis on the active involvement of the learner • Use of hierarchical analyses to identify & illustrate prerequisite relationships • Emphasis on structuring, organizing, & sequencing information to facilitate optimal processing • Creation of learning environments that allow & encourage students to make connections with previously learned material • Based on a student’s existing mental structures, or schema • Organize information in such a manner that learners are able to connect new information with existing knowledge in some meaningful way

• An emphasis on learner control & the capability of the learner to manipulate information • Need for information to be presented in a variety of different ways • Supporting the use of problem solving skills that allow going beyond the information given • Assessment focused on transfer of knowledge & skills • Show how to construct knowledge • Promote collaboration with others for multiple perspectives

• Learner learns a way to accomplish a task, but it may not be the best way, or suited to the learner or the situation

• In a situation where conformity is essential, divergent thinking and action may cause problems

• Learner & environmental factor

• Always under construction as a cumulative history of interactions • Facilitated by authentic tasks anchored in contexts • Most effective for the stage of advanced knowledge acquisition

Chang S. Nam


19

Behaviorist Approach to Learning

What is Behaviorism? Behaviorist approaches to understanding learning maintain that learning occurs when a proper response is demonstrated by the learner who is presented with a specific environmental stimulus. Therefore, behaviorism places an emphasis on overt behaviors that can be observed and measured quantitatively (Good & Brophy, 1990). Behaviorists think of both learner characteristics and environmental factors as important elements for learning to occur. The most critical factor is the arrangement of stimuli - response - reinforcement (Driscoll, 2000). The role of memory in learning is not a concern in Behaviorism, in which forgetting means that a response has not been demonstrated over time. Behaviorists believe that transfer of learning is facilitated in situations involving identical or similar features to which the learner can generalize learning experiences (Ertmer & Newby, 1993).

Contributions of Behaviorism to Instruction: Behaviorist approaches to learning are said to be effective for learning types in which the learner is required to recall facts, to define and illustrate concepts, to apply explanation, and to automatically perform a specified procedure. Ertmer & Newby (1993) provide five assumptions of Behaviorism that are directly relevant to designing instruction, including possible instructional design applications. First, “an emphasis on producing observable and measurable outcomes in students” (p. 56) is assumed. Identifying behavioral objectives, performing task analyses, and conducting criterion-referenced assessments are based on this assumption.

Second, “pre-assessment of students to determine where

instruction should begin” is assumed (p. 56). Learner analysis is derived from this idea. Third, “emphasis on mastering early steps before progressing to more complex levels of performance” is also assumed (p. 56). Sequential presentation and mastery of learning are based on this assumption. Fourth, “use of reinforcement to impact performance” (p. 56). Tangible rewards and informative feedback are based on this assumption. Finally, “use of cues, shaping and practice to ensure a strong stimulus-response association” (p. 56) rounds out the five assumptions. Simple to complex sequencing of practice and use of prompts are instructional strategies based on this assumption of behaviorism.

After establishing the basic

assumptions/principles that are relevant to instructional design, behaviorism stresses designing

Chang S. Nam


20

and developing a learning environment in which a proper response is demonstrated by the learner who is presented with a specific environmental stimulus. Challenges in Engineering Curriculum:

Some challenges are expected when

behaviorist approaches to learning are applied to an engineering curriculum. For example, behaviorists state that learning takes place through the arrangement of stimulus-responsereinforcement (Ertmer & Newby, 1993). This also means that learning should not occur if there is no stimulus for the correct response. However, human beings do not initiate all behaviors that have been reinforced. They also tend to demonstrate new behaviors without any reinforcement (Driscoll, 2000). Behaviorism also minimizes the role of thought and the mind by totally ignoring the possibility of mental processes occurring in the mind. However, one of the main pedagogical goals that the engineering curriculum seeks is to train students to actively pull together many fundamental engineering concepts by gaining hands on experience with modern technology. In order to do that, students are required to think creatively, analyze, and solve the problems in different ways. Behaviorists (e.g., Perkins, 1992) ignore human thinking as “an invalid folk theory” (p. 59). Behaviorism considers learning content as a conditioned response to stimuli (Driscoll, 2000). However, this view may lead to detrimental fragmentation of the engineering curriculum. The connections between ideas in the engineering curriculum are important factors because understanding the whole is key to fully understanding the parts.

Chang S. Nam


21

Cognitive Approach to Theory

What is Cognitivism? Cognitivism is a cognitive learning approach to understanding learning, which views the human learner as an information processor (Driscoll, 2000; Ertmer & Newby, 1993).

Good & Brophy (1990) state that learning involves “the acquisition or

reorganization of the cognitive structures through which humans process and store information” (p. 187). Ertmer & Newby (1993) describe knowledge acquisition as “a mental activity that entails internal coding and structuring by the learner” (p. 58). The factors that influence learning include the ones that affect the learner’s mental activities, such as his/her attention to environmental stimuli and information retrieval. To guide learning, therefore, various methods such as instructional explanations, demonstrations, illustrative examples, and corrective feedback are considered. Unlike Behaviorism, cognitive approaches accept the role of memory in the learning process, and view forgetting as the result of an inability to retrieve information from memory due to interference or memory loss. To help the learner link new information to existing knowledge, therefore, cognitive approaches provide various techniques, such as analogies, hierarchical relationships, matrices, and advance organizers.

Cognitive theorists determine transfer of

learning in terms of the degree to which information is stored in memory (Driscoll, 200). Thus, it is said that transfer of learning has taken place when a learner understands how to apply knowledge in new situations (Ertmer & Newby, 1993).

As mentioned earlier, cognitive

approaches focus on learners’ mental structures. Therefore, complex forms of learning, such as reasoning, problem solving, and information processing, are considered appropriate types of learning.

Contributions of Cognitive Approaches to Instruction:

Ertmer & Newby (1993)

provide basic assumptions/principles of cognitive approaches that are related to instructional design. Table 2-5 shows relationships between these principles and possible instructional design (ID) applications.

Chang S. Nam


22

Cognitive approaches maintain that learners’ existing mental structures be a point in which instruction begins. Ertmer & Newby (1993) suggest that instruction allow learners to easily connect new information with existing knowledge in a meaningful way. As described earlier, various cognitive strategies, such as analogies, framing, outlining, mnemonics, concept mapping, and advance organizers, are recommended for use in designing instruction. From a cognitivism perspective, one of the main foci in instructional design is to provide learners with efficient processing strategies through which they receive, organize, and retrieve knowledge in a meaningful way. Table 2-5. Cognitive Learning Principles and Possible ID Applications Principles • Active involvement of the learner • Use of hierarchical analyses • Structuring, organizing, & sequencing information • Link new information to existing knowledge

Possible Applications • Learner control • Metacognitive training (e.g., self-planning, monitoring, & revising techniques) • Cognitive task analysis • Outlining, summarizes, synthesizers, advance organizers • Recall & Recognition • Relevant examples • Analogies

Challenges in Engineering Curriculum:

Cognitive approaches view learning as

involving the acquisition or reorganization of the cognitive structure (i.e., mental model or schema) through which individuals process and store information. The learner learns a way to accomplish a task, although it may not be the best way, or be suited to the learner or the situation. For example, the learner may have a poor knowledge structure because of memory loss, limited information or misinformation, or interference. According to cognitive approaches, one way of providing instruction to the learners is to break a task down into smaller steps or chunks and use that information to develop instruction. However, this can oversimplify the natural complexity of the problem and consequently learners may not be able to apply the learned knowledge to a real-life situation. Spiro et al., (1991) attribute the failure of many instructional systems to unrealistically simplified and well-structured knowledge of structure. Therefore, to prepare the students to solve problems in the real world environment, Honebein, Duffy, & Fishman (1993) recommend using an authentic activity, or an activity close to the environment in which the learning will actually be used.

Chang S. Nam


23

Constructivist Approach to Learning

What is Constructivism? According to constructivist approaches to learning, human beings construct their own perspective of the world through their experiences and schema (Schuman, 1996). The focus of constructivism is on preparing the learner to solve a problem in new, ambiguous situations. Behaviorism and cognitivism both support the practice of analyzing a task and breaking it down into manageable chunks, establishing objectives, and measuring performance based on those objectives. Constructivism, on the other hand, promotes a more open-ended learning experience in which methods and results of learning are not easily measured and may not be the same for each learner (Land & Hannafin, 1996). According to constructivism, learning occurs through creation of meaning from experience (Bednar, Cunningham, Duffy, & Perry, 1995). Learning is affected by both learner and environmental factors. Constructivists also argue that situation is important to produce knowledge (Brown, Collins, & Duguid, 1989; Jonassen, 1991). Constructivism is applied when designing instruction to ensure that learners use prior knowledge ‘more flexibly’ rather than just recall it (Spiro et al., 1991). As Ertmer & Newby (1993) state, therefore, “memory is always under construction as a cumulative history of interaction” because concepts keep evolving with “each new use as new situations, negotiations, and activities recast it in a different, more densely textured form” (p. 63). Transfer of learning is facilitated by providing authentic tasks to the learner. Thus, constructivism emphasizes the flexible use of preexisting knowledge, instead of the recall of prepackaged schemas (Spiro et al., 1991). Contributions of Constructivism to Instruction: The constructivist view maintains that learning content and its context should be considered in order to determine specific types of learning that are best supported by a learning theory (Bendar et al., 1995). In their ‘Continuum of Knowledge Acquisition Model’, however, Jonassen, McAleese, & Duffy (1993) propose that constructivist-learning environments are most effective for advanced knowledge acquisition, while initial knowledge acquisition is better served by instructional techniques that are based upon traditional instructional design models.

Chang S. Nam


24

The main goal of instruction is to design a learning environment that allows the learner to elaborate and interpret information.

Ertmer & Newby (1993) suggest five assumptions of

constructivism that are directly relevant to designing instruction and their application to instructional design. The first assumption is “an emphasis on the identification of the context in which the skills will be learned and subsequently applied” (p. 65). Anchoring learning in meaningful contexts is one example of this assumption. Second, “an emphasis on learner control and the capability of the learner to manipulate information. Learners are required to actively use what is learned” is another assumption. The third assumption is a “need for information to be presented in a variety of different ways” (p. 65). Instruction is designed, in which the learners are allowed to revisit content at different times, in rearranged contexts, for different purposes. Fourth, “supporting the use of problem solving skills that allow going beyond the information given” exists as the fourth assumption (p. 65).

An example of this assumption includes

presenting alternative ways of representing problems. The fifth and final assumption is an “assessment focused on transfer of knowledge and skills” (p. 65). Based on this assumption, instruction is designed to present new problems and situations that differ from the conditions of the initial instruction. Challenges in Engineering Curriculum:

Like Behaviorism and Cognitivism, some

challenges, when applied to an engineering curriculum, are expected.

For example,

constructivism basically leaves the students to find out how to do something for themselves. However, we may want to engage the learners in constructing a new concept by showing them HOW TO DO it, and explaining it when necessary. We may want the students to find out an engineering concept for themselves, but we can also facilitate knowledge construction by showing how to make that discovery. Constructivists assume that there is no information or knowledge outside the human mind. Let’s assume that an instructor explains how GPS works, giving several examples. According to constructivism, the instructor’s lecture (information) is not transformed into real knowledge (construction of knowledge) until it is absorbed by the mind of the learner. If we follow this assumption, however, there is no way in which the instructor can ensure that the subjective sense-making process taking place in the mind of the students is similar to his or her own.

Chang S. Nam


25

Instructional Design Principles and Models As a part of an overall effort to incorporate fundamental elements of the instructional design (ID) process and principles, a number of instructional design models have been proposed. According to their underlying philosophical views of knowledge and learning, design models can be classified as the three main categories outlined by Moallem (2001): Objectivist Instructional Systems Design Models (OISDMs); Constructivist Instructional Systems Design Models (CISDMs); and Mixed approach to Instructional Design (MID).

Figure 2-1 represents the

relationship between learning theories and instructional design models. OISDMs have been influenced by Behaviorism and Cognitivism, while CISDMs are based on Cognitivism and Constructivism. On the other hand, MID employs all three learning theories.

MID

Behaviorism

Constructivism

Cognitivism

OISDMs

CISDMs

Figure 2-1. Relationship between Learning Theories and Instructional Design Models

Objectivist Instructional Systems Design Models (OISDMs)

The underlying philosophical view of traditional models is an objectivist perspective that knowledge and truth exist independently from the human mind. According to Moallem (2001), objectivist design models put their emphases on “the conditions which bear on the instructional system (such as content, the learner, and the instructional setting) in preparation for achieving the intended learning outcomes” (p. 115).

Chang S. Nam


26

Objectivist design models include Dick & Carey’s Instructional Systems Design (1996) and Gagne, Wager, & Briggs’ Principles of Instructional Design (1992), each of which is based on behaviorism and cognitive science (Saettler, 1990).

Behaviorism has contributed to

traditional models by providing relationships between learning conditions and learning outcomes (Saettler, 1990). In objectivist design models, behavioral objectives are developed as a means to measure learning success. To develop those behavioral objectives, a learning task is broken down into specific measurable tasks.

On the other hand, cognitive approaches influenced

objectivist instructional models by emphasizing the use of advance organizers, mnemonic devices, metaphors, and learners’ schemas as an organized knowledge structure (Driscoll, 2000). Table 2-6 shows the design components that objectivist design models have in common, even if there are some differences among OISDMs (Schiffman, 1995). As Goldstein & Ford (2002) point out, these models take an objectivist view of the nature of knowledge in which emphasis is on “the specification of instructional objectives, precisely controlled learning experiences to achieve these objectives, criteria for performance, and evaluative information” (pp. 22-23). Table 2-6. Typical Objectivist Instructional Design Approach Phase I II III IV V VI VII VIII IX

Design Component Conduct Needs Assessments Establish Overall Goal Conduct Task Analysis Specify Objectives Develop Assessment Strategies Select Media Produce Materials Conduct Formative Evaluation Conduct Summative Evaluation

According to Dick & Carey (1996), an objectivist view to instructional design has contributed to a successful learning outcome because it focuses on “what the learner is to know or be able to do when the instruction is completed,” and provides a prescription about “the relationship between the instructional strategy and the desired learning outcomes,” and it is “an empirical and replicable process” in which instruction is designed for use on as many occasions as possible (p. 11).

Chang S. Nam


27

However, there are also some problems with objectivist approaches to instructional design (Moallem, 2001; Reigeluth, 1996). For example, Reigeluth (1996) notes that objectivist approaches have grouped learners into standardized categories, thereby promoting conformity and compliance. He argues that this is not what the business world wants. Today, organizations want their members to develop their own unique potentials and creativity, which can lead to initiative, diversity and flexibility. Furthermore, the objectivist traditional design models have not explicitly addressed the design issues of user interface in the design process. Several studies on instructional design maintain that a critical factor to the success of instruction is the consideration of usability issues on user interfaces in the design process (Boling & Sousa, 1993; Henke, 1997; Lohr, 2000).

Constructivist Instructional Systems Design Models (CISDMs)

The objectivist design models stress a predetermined outcome, as well as an intervention in the learning process that can map a predetermined concept of reality into the learners’ mind. However, learning outcomes are not always predictable so that learning should be facilitated by instruction, not controlled (Jonassen, 1991).

Instructional design models that take a

constructivist view include Spiro’s Cognitive Flexibility Theory (1992), Jonassen’s Constructivist Learning Environment (1999), Hannafin, Land, & Oliver’s Open Learning Environment (1999), Savery & Duffy’s problem-based learning (1995), Schank & Cleary’s goalbased scenarios (1995), and Cognition & Technology Group’s microworlds, anchored instruction (1992). All the models are referred to collectively as ‘Constructivism’ (Jonassen & RohrerMurphy, 1999). Constructivist instructional design models have their roots in cognitive science and social psychology (Saettler, 1990). Constructivists believe that the learner actively constructs knowledge and truth, and focuses on collaboration, learner autonomy, reflexivity and active engagement (Duffy & Jonassen, 1992). According to this view, learners construct their own knowledge by actively participating in the learning process. In his ‘Constructivist Learning Environment’ model, for example, Jonassen (1999) provides a number of design principles for implementing

Chang S. Nam


28

constructivist concepts such as cooperative learning, project-based or problem-based learning, reciprocal learning, etc. These design guidelines include: 1. Create real world environments that employ the context in which learning is relevant. 2. Focus on realistic approaches to solving real-world problems. 3. The instructor is a coach and analyzer of the strategies used to solve these problems. 4. Stress conceptual interelatedness, providing multiple representations or perspectives on the content. 5. Instructional goals and objectives should be negotiated and not imposed. 6. Evaluation should serve as a self-analysis tool. 7. Provide tools and environments that help learners interpret the multiple perspectives of the world. 8. Learning should be internally controlled and mediated by the learner (pp. 11-12).

Figure 2-2 presents the design components that CISDMs have in common, even if there are some differences among them (Moallem, 2001)

Identify learning domain

Map multiple paths through cases

Identify complex cases/problems

Identify learning elements

Provide tools for learner controlled path

Provide tools to help learner decide what to do next

Figure 2-2. An Example of Constructivist Model for Instructional Design Given those design procedures, it seems that constructivist design models also do not consider interface design issues, even if “designing the interface between a learning environment and the learner is an increasingly critical part of the instructional development process” (Lohr, 2000, p. 161).

Chang S. Nam


29

Mixed Approach to Instructional Designs

Unlike objectivist and constructivist design models, the mixed approach to instructional design proposes that an instructional design model reflect all learning theories according to instructional design situations.

For example, Schwier (1995) points out that different

instructional design situations, such as different learners and learning environments, may require different learning theories and thus different instructional design models. Davidson (1998) found that, in practice, a mix of old (objective) and new (constructive) instruction/learning design is increasingly being used. In their ‘Continuum of Knowledge Acquisition Model,’ Jonassen et al. (1993) note that the initial knowledge acquisition is better served by instructional techniques that are based upon traditional instructional design models whereas constructivistlearning environments are most effective for advanced knowledge acquisition. One example of a mixed approach to instructional design is Moallem’s (2001) study on the process of designing and developing a Web-based course. To maximize learners’ learning experiences, who have diverse backgrounds, experiences, and learning styles, Moallem (2001) determined the best instructional design model by matching objectivist design models with constructivist models, based on students’ prior knowledge, experiences, and types of learning outcomes. Based on the Jonassen et al. (1993) study, for instance, constructivist models have been applied when students had more advanced knowledge of the content and the learning outcomes were primarily problem solving and applications of multiple principles. Objectivist design models, on the other hand, were found to be appropriate when students had very little directly transferable prior knowledge and the learning outcomes were focused on learning new concepts and principles. Moallem’s (2001) mixed approach shows a useful way of incorporating various learning theories in an instructional design model; but this approach also did not address the issues involved in human-computer interface design, or the overall effectiveness of a Web-based learning environment.

Chang S. Nam


30

Implications for the Current Problems This section reviewed the three underlying philosophical views of knowledge and learning, as well as commonly used instructional design models and principles that have been affected by diverse approaches to learning.

Table 2-7 shows the comparison of the three

instructional design approaches. Table 2-7. Comparison of Three Instructional Design Approaches Dimension Learning theory involved Design rationale Usability on user interface Evaluation framework

MID • Behaviorism • Cognitivism • Constructivism • Mixed approach • Not explicit • Not explicit

OISDMs • Behaviorism • Cognitivism

CISDMs • Cognitivism • Constructivism

• Objectivist • Not explicit • Yes, but only on instruction

• Constructivist • Not explicit • Not explicit

Given common learning activities (e.g., problem solving, inference generating, critical thinking, and laboratory activities) and types of learning domains (e.g., intellectual skills and verbal information) in the GPS course, this study proposes that many principles provided by the cognitive learning theory would be suited for redesigning the learning content of the course. For example, providing efficient processing strategies through which students receive, organize, and retrieve knowledge in a meaningful way will facilitate learning activities. For

instructional

strategies with which the GPS course is delivered through the Web-based learning environment, this study recommends Objectivist Instructional Design Approaches, which combine Cognitivism and Behaviorism.

For example, Behaviorism provides relationships between

learning condition and learning outcomes, and such relationships can inform the instructional designer of how the instruction should be designed to achieve successful learning outcomes. To effectively deliver the instruction, on the other hand, cognitive approaches provides various instructional methods, such as the use of advance organizers, mnemonic devices, metaphors, and learners’ schemas as an organized knowledge structure. This study also suggests employing constructivist approaches for effective instructional strategies. For example, the constructivist approach that instruction should promote collaboration with other learners and/or instructors provides a ground for the implementation of an email system or group discussion board system for such educational purposes.

Chang S. Nam


31

Human-Computer Interface Design for Distance Learning Environments The previous section gave an overview of a range of theoretical positions to learning and instructional design methods and provided practical implications that can be used to effectively design the instructional system. For a Web-based supplemental learning environment to be successful, it is also important to effectively facilitate interactions of learners with the learning environment. A human-computer interface that is easy to use and is intuitive to anyone can fulfill that requirement.

It is clear that an effective user interface in Web-based learning

environments is one of the most important parts in that it determines how easily learners can focus on learning materials without having to make an effort to figure out how to access them (Lohr, 2000). Unfortunately, little attention has been paid to design issues of the humancomputer interface in the development of Web-based supplemental learning environments (Plass, 1998). Furthermore, few attempts have made to design a ‘usable’ interface that is also ‘learningcentered.’ Although an email system or discussion board system provides a usable interface, for example, the system does not have a learning-centered interface unless it supports the learner writing a mathematical equation, which requires using special characteristics. The purpose of this section, therefore, was to review issues involved in the design of the human-computer interface that should be considered when developing Web-based supplemental learning environments. First, it addressed some of the barriers or incompatibilities that challenge designers when trying to design a usable interface that is also learning-centered for Web-based supplemental learning environments, from the two theoretical perspectives: cognitive and communicative perspective. Second, several different approaches to user interface design were reviewed, which can help design usable, learning-centered interfaces. implications for the current development were discussed.

Finally, practical

Chang S. Nam


32

Challenges of Interface Design for Distance Learning Environments What makes an effective and intuitive interface design difficult for Web-based supplemental learning environments? This study asserts that an answer to this question will also be able to provide the path leading to the design of the user interface such that it is easy to use and intuitive to anyone. Some of the barriers or incompatibilities that challenge the design of a usable interface that is also learning-centered are addressed from the cognitive and communicative perspective.

The Cognitive Perspective In his study on human-computer interaction, Streitz (1987) proposed the concept of “Cognitive Compatibility,” defined as the degree to which a conceptual model of the user matches to that of a given system. He maintains that An interactive computer system is the more user-oriented the less discrepancies do exist between the relevant knowledge representations (models) on either side of the interface: already before the interaction: between S(f) and U(f) and then during the interaction: between S(f) and U(S(f)), with S(f) = the system’s realization of a given functionality f, e.g., to write, to draw, to store U(f) = the user’s mental model of the functionality f U(S(f) = the user’s mental model of the system’s realization of a given functionality f (p. 77).

Given the concept of cognitive compatibility, a user has an adequate U(S(f)) that is compatible with the actual structure of S(f) only when an interface designer takes the user’s general mental model about a function f into account and implements it when designing the user interface for distance learning environments. This aspect challenges the interface designer, especially when developing Web-based supplemental learning environments, in which learners have diverse learning styles and educational backgrounds. Diverse user characteristics make it difficult for the designer to match the user’s conceptual model with that of a given system. Interface design for groupware systems in Web-based supplemental learning environments, such as video conferencing systems, chat rooms, or interactive discussion boards, also challenge designers.

A fundamental difference between single-user and multi-user

interfaces highlights this challenge. With single-user interfaces, for example, users can easily

Chang S. Nam


33

interpret any display changes resulting from their actions. As Ellis, Gibbs, & Rein (1993) stated, “in fact much effort goes toward increasing the system’s responsiveness” when designing singleuser interfaces (p. 50).

With group interfaces, however, users may have difficulty in

understanding sudden display changes that result from others’ actions.

When designing

interfaces for groupware systems, the designer is required to consider each user’s mental model about a function (e.g., display change). Another challenge related to group user interfaces is an individual’s changing role within a group. An example of this challenge includes the ‘shared whiteboard’ designed for supporting group activities. A student can be an author at one moment and then an editor when revising a group project paper within the shared whiteboard. Therefore, multi-user interfaces should be designed to support not only several people working together, but also their different and shifting roles.

The Communicative Perspective

The design team that develops Web-based learning environments primarily consist of interface designer(s) and instructional designer(s). It is important for both design groups to communicate with each other, while sharing their knowledge to effectively develop the environment. The development of a Web-based supplemental learning environment involves the consideration of the various aspects of the environment, including application domain knowledge (i.e., target subject field), the conceptual learning theory, instructional design guidelines, and human-computer interface design. Good communication among design team members facilitates the design process (Shepherd & Martz, 2001). Unfortunately, communication between both types of designers, even within the design team, has been limited in the instructional design area. For example, instructional designers view instructional design as the most important component of Web-based learning environments, and consequently put little emphasis on usability of user interfaces. On the other hand, as user interface designers might have little knowledge about instructional design, they tend to focus on the user interface design, ignoring the problems of instructional design.

Chang S. Nam


34

Approaches to Interface Design: Wallace & Anderson’s (1993) Classification There are a number of design approaches to the user interface, each of which has its own strengths and weaknesses. To review the current user interface design practice, this study borrowed Wallace & Anderson’s (1993) classification, which proposes four different types of approaches to the development of user interfaces: the craft approach, enhanced software engineering approach, technologist approach, and cognitive approach.

Table 2-7 briefly

summarizes each of the four approaches to user interface design.

The craft approach

In the craft approach, interface design is described as a craft activity in which the skill and experience of the interface designer or human factors expert play an important role in the design activity (Dayton, 1991; Wroblewski, 1990). The designer (or human factors expert) is regarded as a craftsman or even an artist with talent. The advocates of this approach tend to believe that the development of a structured methodology for interface design is impossible as design projects are unique (Laurel, 1990; Norman, 1987; Rubinstein & Hersh, 1984). For successful design, this approach relies on the designer’s creativity, heuristics, and development through prototyping.

The enhanced software engineering approach

The enhanced software engineering approach claims that formal HCI methods, such as user characteristics and task analysis, should be introduced into the development life-cycle to support the design process (Damodaran, Ip, & Beck, 1988; Shneiderman, 1993; Sutcliffe, 1988, 1989; Waterworth, 1992). This approach attempts to overcome the short-comings of structured software engineering methods, which tend to focus on a system’s functionality and ignores issues involved in human-computer interaction and user interface design. Since the advocates of this approach view system analysts as the users of methodologies, any interface design methods must be designed to meet the needs of system analysts or developers.

Chang S. Nam


35

The technologist approach

The technologist approach claims that designers produce poor quality interfaces because they have to spend more time in performing time-consuming tasks, such as programming an interface, than in doing design activity during development (Buxton & Lamb, 1983: Cockton, 1988; Wasserman, 1985).

To allow designers to concentrate on design, the technologist

approach attempts to provide automated development tools, such as the User Interface Management System (UIMS). According to Buxton & Lamb (1983), the objective of the technologist approach is “to free the applications programmer from low-level details so as to be able to concentrate on higher application-specific aspects of the user interface” (p. 35). As an effort to identify user requirements, this technology-centered approach stresses the use of rapid prototyping tools such as HyperCard, Visual Basic, Astound, VRML and Multimedia Toolkit in interface design.

The cognitive approach

The cognitive approach applies psychological knowledge, such as theories of information processing and problem solving to the problem of interface design (Barnard, 1991; Card, Moran, & Newell, 1983; Kieras & Polson, 1985; Landauer, 1991). This most theoretical approach to interface design is characterized by an attempt to build precise and accurate users’ cognitive models that represent the interaction of users with computers. This approach provides numerous techniques and methods to use, which can be categorized into four main methods: cognitive metric methods, grammar models, knowledge methods, and user modeling methods (Wallace & Anderson, 1993). Each of these methods is briefly summarized in Table 2-8.

Chang S. Nam


36

Table 2-8. Four Categories of Methods in the Cognitive Approach Goal

Focus

Examples

Cognitive metric methods To quantify the properties of users

To measure the users’ performance time and memory load for a given task • Keystroke-level model • Unit-task-level model

Grammar models To provide a grammatical notations for the description of mental models Dialogue specification and evaluation side of interface design • Grammar model

Knowledge methods To describe users’ mental models

User modeling methods To identify users’ cognitive models

To make explicit the mental processes of the user when performing tasks • Cognitive complexity theory (CCT) • Knowledge analysis of task (KAT)

To develop more sophisticated models of cognition • Programmable user models (PUM) • Approximate Modeling of Cognitive Activity

The goal of cognitive metric methods such as keystroke-level and unit-task-level model is to quantify the properties of users. These methods measure the users’ performance time and memory load for a given task. On the other hand, grammar models provide a grammatical notation for the description of mental models. These models focus on the dialogue specification and evaluation side of interface design. The example includes grammar models. Knowledge methods are similar to grammar models in that their goal is to describe users’ mental models in order to make explicit the mental processes of the user when performing tasks. Examples include cognitive complexity (CCT) and knowledge analysis of task (KAT). Finally, the goal of user modeling methods such as programmable user models (PUM), an approximate modeling of cognitive activity, is to identify users’ cognitive models in order to develop more sophisticated models of cognition.

Chang S. Nam


37

Implications for the Current Problems This section discussed some of the challenges in designing a usable interface that is also learning-centered for Web-based supplemental learning environments. Some of the examples included diverse learning styles and educational backgrounds of the learners. These issues apply to the current development of Web-based supplemental learning environments for the GPS course. This course is composed of roughly 75% electrical engineering students and 25% aerospace engineering students. Of the total number of students, roughly 30% are graduate students and 70% are undergraduate students. The design of multi-user interfaces is another challenge to the interface designers. These challenges require the development of Web-based supplemental learning environments to follow a user-centered design process that takes human factors into account. Gould & Lewis (1985) provide three principles of a user-centered design: 1) an early focus on users and tasks, 2) empirical measurement of product usage, and 3) iterative design whereby a product is designed, modified, and tested repeatedly. Rubin (1994) suggests several techniques, methods, and practices that can be used for the user-centered design. Some of the examples include participatory design, focus group research, surveys, design walkthroughs, expert evaluations, and usability testing. The various approaches to interface design were also reviewed. This study asserts that, in order to design user interfaces such that they are easy to use and intuitive to anyone, it is important to have design skills as well as some knowledge of psychology, methodologies and prototyping. Since all four approaches are fundamental to successful design, therefore, they should be applied as they relate to interface design for Web-based supplemental learning environments. For successful interface designs, this study also recommends identifying and applying interface design guidelines and principles when developing Web-based supplemental learning environments. The ways of designing the Web directly affect how users behave on the Web. For example, several studies show that Web pages with poor usability can have negative effects on users’ interaction with the Web (Borges, Morales, & Rodriguez, 1998; Rizzuti & Dickinson, 2000).

To help design user interfaces on the Web, many interface design principles and

Chang S. Nam


38

guidelines have been proposed. Table 2-9 shows a group of the sources of the Web design guidelines and their main categories. Table 2-9. Examples of Web Design Guidelines and Main Categories Guideline IBM Web Design Guidelines Yale Style manual Ameritech Web Page User Interface Standards and Design Guideline Sun Microsystems Guide to Web Style Guidelines for Multimedia on the Web Web Design Guidelines of The Library of Congress

Source http://www-3.ibm.com/ ibm/easy/eou_ext.nsf/ Publish/572 http://info.med.yale.edu/ caim/manual/ http://web.archive.org/web/ 19990224210202/ www.ameritech.com/corporate/ testtown/library/standard/ web_guidelines/index.html http://web.archive.org/ web/19990508134652/ http://www.sun.com/ styleguide/ http://www.useit.com/ alertbox/9512.html

Main Category Structure, navigation, text, visual layout & elements, media

http://memory.loc.gov/ammem/ ndlpedu/educators/workshop/ thelook/guide.html

Graphic Design (balance, contrast, elements of design, & simplicity) Interface Design (accessibility, clarity, consistency, & hierarchy)

Interface, site, page, graphics, multimedia & animation General Principles and Style in Designing, Organizing Information, Navigation, Search, Text Formatting, Graphics, User Inputs Links, Page Length, Graphics, Image Maps, Navigation, Security, Quality, Netiquette, Content, Selling, Language Animation, Audio, Video, Response Time

Chang S. Nam


39

Evaluation of Web-based supplemental learning environments The growing number of Web-based learning applications has generated a need for methods to systematically evaluate these learning environments. Unfortunately, few attempts have been made to develop evaluation frameworks for Web-based supplemental learning environments whose foci are both the instructional system and user interface system. The evaluation activity can take place either formatively or summatively. The objective of formative evaluation is to collect continuous feedback from users to revise or improve the distance-learning environment as needed (Dick & Carey, 1996). On the other hand, summative evaluation is a range of activities to determine the effectiveness of the program or product at the end of a time period. Summative evaluation focuses on the outcome, such as performance and properties. This study focused on formative evaluation of Web-based supplemental learning environments, because the evaluation in Web-based learning environments is a continuing process throughout the development lifecycle (Belanger & Jordan, 2000). The purpose of this section was to overview several useful approaches that can be used to identify problem areas or to draw inferences about the overall quality of Web-based supplemental learning environments. First, it reviewed the models proposed for formative evaluation about distance learning environments. Second, usability evaluation methods of the user interface were reviewed from two perspectives: the user- and expert-based approach. Finally, practical implications for the current problems were provided.

Chang S. Nam


40

Models of Formative Evaluation on Distance Education Formative evaluation is the process in which a variety of empirical data are obtained to revise the Web-based learning environment, which will make it more efficient and effective (Dick & Carey, 1996). As useful approaches to formative evaluation, Belanger & Jordan (2000) compare three models: Van Slyke, Kittner, & Belanger’s (1998) model; Marshall & Shriver’s (1994) five-level model; Kirkpatrick’s (1994) four-level model.

In addition to these three

models, this section also overviews Dick & Carey’s (1996) model, which has widely been used for the evaluation of the instructional design.

Van Slyke, Kittner, & Belanger’s (1998) Model

The model proposed by Van Slyke et al. (1998) takes into account ‘determinants of success’ as precedent variables and ‘evaluation criteria’ as outcome variables, as shown in Table 2-10.

Determinants of success include four different categories of variables: institutional,

learner, course, and distance learning characteristics. Institutional characteristics are related to the organization, such as its objectives, delivery mechanisms, and support structure. objectives and skills.

Learner characteristics are related to the learner’s

Course characteristics are also taken into account in terms of

characteristics of the course itself, evaluation methods, and hands-on components. Finally, distance-learning characteristics include the consideration of technology used and environment of the learner. This model stresses the importance of evaluation criteria as outcome variables. These variables include four different units of analysis: the learner, instructor, institution, and society. Evaluation criteria at the learner level include learners’ increased technology awareness and skills and active interaction with the teacher. As evaluation criteria for the institution, there are several outcome variables, such as lower costs of providing education, increased reach to potential learners, increased productivity among instructors and the sharing of instructional

Chang S. Nam


41

resources with other institutions. Finally, at the societal level, a more professional workforce, increased quality of life and increased access to education serve as outcome variables. Table 2-10. Precedent and Outcome Variables in Van Slyke et al. (1998) Model Category

Determinants of Success (Precedent Variables)

Institutional Characteristics Learner Characteristics Course Characteristics Distance Learning Characteristics Learner

Evaluation Criteria

Institution

(Outcome Variables)

Instructor Society

Variables • Objectives of the institution • Delivery mechanism • Support structure • Objectives of the learner • Skills • Characteristics of the course itself • Evaluation methods • Hands-on components • Extensive use of technology • Environment of the learner • Increased technology awareness and skills • High quality of interaction with the teacher • Better access to instructor • Lower costs of providing education • Increased reach to potential learners • Increased productivity among instructors • Sharing of instructional resources with other institutions • High quality of interaction with the teacher • More professional workforce • Increased quality of life • Increased access to education

Marshall & Shriver’s (1994) Five-Level Model

Marshall & Shriver’s (1994) five-level model is the performance-based model, in which the learner is required to demonstrate knowledge and skills learned. As seen in Table 2-11, this model provides five main categories of the variables that need to be considered. Self: This category concerns feedback on an instructor’s performance. To identify problem areas in instruction, the instructor needs to ask him/herself about what went well, what went wrong, or what needs to be improved. In addition to that, learners and colleagues can also provide the instructor with his/her performance feedback.

Chang S. Nam


42

Table 2-11. Variables to Consider in Marshall & Shriver’s (1994) Five-level Model Category Self Course Materials Course Curriculum Course Modules Learning Transfer

Related Variable • Feedback from self, learners, & colleagues • Level of difficulty • Overall effectives of the course content • Evaluation of multiple courses • Comparison across the curriculum • Effectiveness of structure & order • Feedback about the transfer of learning

Course materials: Evaluation in this category measures the overall effectiveness of the course content in terms of the level of difficulty. Course curriculum: The design of the courses is required to meet learning objectives and outcomes. Therefore, the course curriculum can be evaluated across multiple courses or other curricula. Course modules:

The instructional strategy determines an effective instruction.

Therefore, instructional strategies can be evaluated in terms of the effectiveness of structure and order. Learning transfer: This category is related to feedback information and to whether learned knowledge and skills can be applied to the workplace or real situations.

Kirkpatrick’s (1994) Four-Level Model

The evaluation model developed by Kirkpatrick (1994) consists of four levels, in which the next level’s evaluation is based on information from each prior level. The levels include the reactions, learning, transfer, and business results, as shown in Table 2-12.

Chang S. Nam


43

Table 2-12. Summary of Kirkpatrick’s (1994) Four-level Model Level Level I: Reactions

Variable • Learners’ reaction to instruction (e.g., satisfaction, perception of value)

Level II: Learning

• Amount of knowledge learned • Transfer of learners’ behavior

Level III: Transfer Level IV: Business Results

• Financial impact • Overall success

Recommended Method • Survey • Focus group • Interview • Criterion-referenced test • Pretest-posttest design • Observation • Survey • Interview • Longitudinal methods

Level I (Reactions): This level measures learners’ affective reactions to instruction. For example, learners’ satisfaction with a course can be evaluated. Recommended methods at this level include a survey, focus group, and/or interview. Information from this level serves as a base for evaluation at level II. Level II (Learning): The evaluation at this level measures the amount of knowledge that learners have acquired.

Assessment is done with criterion-referenced tests, such as the

objectives of the course. Pretest-posttest designs can be used to differentiate between what learners already knew prior to instruction and what they actually learned due to instruction. Level III (Transfer): Evaluation at this level attempts to assess whether what learners learned from instruction has been used in real situations through various methods, such as an observation, survey, and interview. Level IV (Business Results): These measures take place at the organizational level, which is the most difficult to measure.

Assessment focuses on the overall success of the

instructional program in terms of the financial impact of the program, including monetary impact, efficiency, and motivation.

Chang S. Nam


44

Dick & Carey’s (1996) Model

As a formative evaluation process, Dick & Carey (1996) proposed four different methodologies: (1) subject matter expert review, (2) one-to-one evaluation, (3) small group evaluation, and 4) field trial. Since the focus of this study is on formative evaluation, however, the first three methods will be reviewed. Subject Matter Expert Review: Before the whole system under development is tested with users, a dry run should be conducted. In order for a system to be successful, we must discover overlooked areas or problems and review them. The subject matter experts (SMEs) who exhibit the highest level of expertise in the current topic area fill that requirement. The subject matter experts can also perform the final revision of the system as a whole. One-to-One Evaluation: To identify and remove prominent errors in the system, two or more participants, representing a full range of abilities, go through all aspects of the Web-based learning system with an evaluator. Since a Web-based supplemental learning environment provides various tools supporting an instructor, such as a course management system (e.g., WebCT or Blackboard), this stage can also evaluate these systems with the instructor. Learners are also asked to evaluate the system in terms of screen design, information structure, menu structure, and interface. The evaluator observes the problems that caused user confusion. Small-group evaluation:

Web-based learning systems tend to involve several

communication tools to facilitate collaborative learning among the learners. These tools include asynchronous tools, such as e-mail and web forum, and synchronous tools, such as videoconferencing and chat meeting.

A group of participants representative of the target

population evaluate group learning activities (e.g., group discussion) and multi-user interface system (e.g., Discussion Board).

Chang S. Nam


45

Usability Evaluation Approaches Attempts to judge the value of user interface systems have resulted in a number of usability evaluation methods. These methods have been classified in several ways (Home, 1998; Rosson & Carroll, 2002). For example, Rosson & Carroll (2002) distinguish between two different types of evaluation methods: analytic and empirical. Analytic methods are concerned with identifying the characteristics that influence the user’s performance. These methods include claims analysis, usability inspection, and model-based analysis. Empirical methods, such as controlled experiments, think-aloud experiments, and field studies, investigate how users actually use the system in real situations or in experimental environments.

The usability

methods can also be categorized into three different types of approaches: usability testing, inspection, and inquiry approach (Home, 1998). These methods and techniques used in usability evaluation will be overviewed from the two approaches: the user- and expert-based approach. What distinguishes the two approaches is who evaluates a value of the user interface system.

The User-Based Evaluation Approach

In the user-based usability evaluation approach, the main players evaluating a system or its prototype are representative users. The objective of this approach is to see how the user interface supports the users in performing their tasks as they interact with the interface system. There are many methods that can be categorized into three types of user-based evaluation approaches. Table 2-13 shows the three approaches to user-based evaluation and examples of the methods.

Chang S. Nam


46

Table 2-13. Examples of User-based Usability Evaluation Approach Approach Performance-Based Approach Co-Operative Approach

Think Aloud Approach

Example Method • Performance Measurement Method (PMM) • Coaching Method • Co-discovery Learning • Shadowing Method • Teaching Method • Question-asking Protocol • Retrospective Testing • Thinking Aloud Protocol

The Performance-Based Approach: In the performance-based approach, quantifiable data and qualitative information are measured to see if the user interface system meets predefined usability goals, such as effectiveness and efficiency. One typical method of this approach is the Performance Measurement Method (PMM). The purpose of this method is to obtain quantitative data about the users’ performance when they perform predefined tasks (Nielsen, 1993; Rubin, 1994). The PMM can also be used to obtain subjective, qualitative data about the users’ preferences and needs. As the types of measures that can be collected during the test, Rubin (1994) lists a group of performance and preference data. Table 2-14 shows sample measures of performance and preference data. Table 2-14. Sample Measures of Performance and Preference Performance measure • Time to complete each task • Number and percentage of tasks completed correctly • Number and percentage of tasks completed incorrectly • Time needed to recover from error(s) • Time required to access information in online help • Count of all incorrect selections (errors) • Count of incorrect menu choices • Count of “negative comments or mannerisms” • Count of visits to the index

Preference measure • Usefulness of the product • How well product matched expectations • Appropriateness of product functions to user’s tasks • Ease of use overall • Ease of learning overall • Ease of accessibility • Usefulness of the index, table of contents, help, graphics, and so on

The most important benefit of this method is that a quantitative value obtained can easily be compared to established usability goals or criteria. On the other hand, the major difficulty with this method is to establish face validity. environment (i.e., ecological validity).

Another limitation is an unnatural testing

Chang S. Nam


47

The Co-Operative Approach: Unlike other user-based approaches to usability evaluation methods, wherein usually one user participates in a usability test, this approach allows a group of the participants to work together. The main purpose of this approach is to obtain the information needs of users by facilitating interaction between people participating in the usability test. The co-operative approach includes the coaching method, co-discovery learning, shadowing method, and teaching method (Nielsen, 1993). The shadowing method is similar to the coaching method in that the test user has an expert user who will help the user of the computer. However, this method is different in that the shadowing method can be used when it is not appropriate for the test user to think aloud or talk to the tester while working on the tasks. Vora & Helander (1995) describe the teaching method as an alternative to the concurrent think-aloud method for usability testing.

This method

involves both novice users and users who already have some expertise in accomplishing tasks using the system.

The Think-Aloud Approach: This approach is a very efficient way of identifying what users are thinking when they perform some specified tasks (Berry & Broadbent, 1990; Nielsen, 1993). Examples of this approach include the question-asking protocol, retrospective testing, and think-aloud protocol. During usability testing, the experimenter can prompt users by asking direct questions about the system (question-asking protocol), ask them what they were thinking after the task (retrospective testing), or let them verbalize their thoughts while performing the task (think-aloud protocol). The most important benefit of this approach is that a lot of qualitative data can be collected from just a small number of users. However, this approach may influence the user’s behavior in a think-aloud session. For example, Berry & Broadbent (1990) reported that the verbalization could lead to an increase in performance from the test users.

Chang S. Nam


48

The Expert-Based Evaluation Approach

In the expert-based usability evaluation approach, a usability expert (e.g., software developer, experienced user or usability specialist) reviews a system to identify usability-related aspects of the user interface. This approach is also known as Usability Inspection Method (UIM). Table 2-15. Examples of Expert-based Usability Evaluation Approach Approach Walkthrough Approach Inspection Approach

The Walkthrough Approach:

Example Method • Pluralistic walkthrough • Cognitive walkthrough • Feature Inspection • Heuristic Evaluation • Perspective-based Inspection

In this approach, a group of evaluators evaluate the

usability of the user interface system by performing a set of tasks.

This approach is

multidisciplinary collaboration among human factors engineers, software developers, representative users, and even people from marketing. Example methods include the cognitive walkthrough and pluralistic walkthrough method (Bias, 1994; Polson, Lewis, Rieman, & Wharton, 1992). The goal of the pluralistic walkthrough method is to identify a list of potential problems and possible solutions by using a diverse range of skills and perspectives (Bias, 1994). Polson et al. (1992) describe that the cognitive walkthrough method is an efficient way of modeling action sequences that users would perform when they are using an interface for the first time. The Inspection Approach: In this approach, usability experts such as human factors engineers examine the feature set of a product (i.e., Feature Inspection Method), or identify potential usability problems based on a design guideline or general principles (i.e., Heuristic Evaluation Method).

More recently, Zhang, Basil, & Shneiderman (1998) proposed a

perspective-based usability inspection technique, in which usability issues are divided into a group of different perspectives and each inspection session focuses on one usability issue.

Chang S. Nam


49

The perspective-based inspection method was inspired by the results of empirical studies showing that the usability inspection methods such as heuristic evaluation, cognitive walkthrough, and formal usability inspection detect a lower percentage of usability problems by each inspector (Desurvire, 1994; Jeffries, Miller, Wharton, & Uyeda, 1991, in Zhang, Basil, & Shneiderman, 1998). In their empirical study of perspective-based usability inspection, Zhang et al. (1998) showed that the average success rate (percentage of usability problems detected) of the perspective inspection method was higher than heuristic evaluation. According to Mayhew (1999), the heuristic evaluation is defined as a method in which “usability experts review a design based on their knowledge of human cognition and general user interface design guidelines” (p. 246). Table 2-16 shows Nielsen’s (1993) ten typical sets of heuristics. Table 2-16. Ten Usability Heuristics by Nielsen (1993) Heuristic Visibility of system status

Description The system should always keep users informed about what is going on, through appropriate feedback within reasonable time. Match between system and the The system should speak the users' language, with words, phrases and concepts real world familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order. User control and freedom Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. The system should support undo and redo. Consistency and standards Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions. Help users recognize, diagnose, Error messages should be expressed in plain language (no codes), precisely and recover from errors indicate the problem, and constructively suggest a solution. Error prevention Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Recognition rather than recall Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate. Flexibility and efficiency of use Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions. Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility. Help and documentation Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large. (Source: http://www.useit.com/papers/heuristic/heuristic_list.html)

Chang S. Nam


50

Implications for the Current Problems This section overviewed evaluation approaches that can be used for determining the effectiveness of the instructional system and usability of the user interface system. There are many formative evaluation models for Web-based education. However, few approaches take the problems of the user interface design into account in their evaluation process. A number of the evaluation frameworks that can be used to evaluate the user interfaces have also been proposed. However, these models were intended for software environments rather than for learning environments such as Web-based learning that requires considering how effectively user interface system supports users’ learning activities. Thus, an evaluation framework is required for Web-based supplemental learning environments, in which the evaluation process, methods, and criteria are provided to systematically evaluate both the instruction and user interface system. As the evaluation process, Dick & Carey’s (1996) evaluation approach may be the best candidate for the evaluation of the WD2L environment. This approach allows different types of evaluators (e.g., experts, individual, and group of evaluators) to evaluate various aspects of the environment (e.g., individual and group learning activities).

Chang S. Nam


51

Summary This chapter gave an overview of the relevant literature that guided the design, development, and evaluation of the WD2L environment that supports GPS education. First, several developmental issues were found through the review of the GPS course, which should be addressed before the development of the WD2L environment in terms of the context, delivery mode, time frame, content, and audience characteristics. One of the developmental issues was to effectively design the learning content and the overall instructional interventions, which make up the instructional system. Since it is important to understand how people learn and to incorporate that knowledge when designing the instructional system, this study examined the three main theoretical explanations of how people learn (i.e., Behaviorism, Cognitivism, and Constructivism) across eight dimensions. Given common learning activities (e.g., problem solving, inference generating, critical thinking, and laboratory activities) and types of learning domains (e.g., intellectual skills and verbal information) in the GPS course, this study proposed that many principles provided by the cognitive learning theory would be suited for redesigning the learning content of the course. For instructional strategies with which the GPS course is delivered through the Web-based learning environment, this study recommended Objectivist Instructional Design Approaches, which combine Cognitivism and Behaviorism. This study also suggested employing constructivist approaches for effective instructional strategies. Some of the barriers were also discussed, which challenge the design of a usable interface that is also learning-centered for Web-based supplemental learning environments.

These

challenging barriers required the development of Web-based supplemental learning environments to follow a user-centered design process, which takes human factors into account. The various approaches to interface design were discussed. This study asserts that in order to design user interfaces such that they are easy to use and intuitive to anyone, it is important to have design skills, some knowledge of psychology, methodologies and prototyping. Since all four approaches are fundamental to successful design, therefore, they should be applied as they relate to interface design for Web-based supplemental learning environments. For successful

Chang S. Nam


52

interface designs, this study also recommended identifying and applying interface design guidelines and principles when developing Web-based supplemental learning environments. Various evaluation approaches were overviewed, which can be used for the evaluation of the effectiveness of the instructional system and the usability of the user interface system. There are many formative evaluation models on Web-based education. However, few approaches take the problems of the user interface design into account in their evaluation process. A number of the evaluation frameworks that can used to evaluate the user interfaces have also been proposed. However, these models were intended for software environments rather than for learning environments such as Web-based learning that requires considering how effectively user interface system supports learners’ learning activity. Thus, the evaluation framework is required for Web-based supplemental learning environments, in which the evaluation process, methods, and criteria are provided to systematically evaluate both the instruction and user interface system. For the evaluation process, Dick & Carey’s (1996) evaluation approach would be suited for the evaluation of the WD2L environment. It is because this approach allows several different types of the evaluators to evaluate the environment (i.e., expert, individual, and group of evaluators). This approach seems most promising. Based on the information identified in this chapter, Chapter III provided an overview of the development process, the resulting WD2L environment prototype, and additional relevant details in the process of designing and developing the WD2L environment to support GPS education.

53

CHAPTER III DEVELOPMENT OF WD2L ENVIRONMENT Based on the available literature reviewed in Chapter II, this study asserts that for a WD2L environment to be successful, various aspects of the learning environment should be considered, such as application domain knowledge, conceptual learning theory, instructional design, human-computer interface design, and evaluation plan about the overall quality of the learning environment. Unfortunately, few frameworks are available for the WD2L environment development to support engineering education. Moreover, they rarely take those factors into account in their design process. As a new development process framework, this study proposed the Integrated Design Process (IDP) (see Figure 3-1) along with the Design Process Template (see Figure 3-2), which together will help address various factors involved in the development of the WD2L environment. This chapter provided an overview of the development process of the WD2L environment prototype developed using the proposed IDP. First, workflow steps of the IDP were outlined, which described processes of building the WD2L environment prototype in more detail. Second, an overview of the resulting WD2L environment prototype was given, describing the final user interface system and instructional system prototype. Design changes made to the prototype as a result of a formative evaluation were summarized in Chapter IV. Finally, a general discussion on the development of the WD2L environment prototype was included.

Design Goals Setting

Requirements Specification

Features & Components Identification

Needs Analysis

Meet

Page Design

Structure Design

Information Design

User Interface System

Design Scenarios Development

Conceptual Design

High-fidelity Prototype

Design Walk-through

Low-fidelity Prototype

Development

Chapter III. Development of WD2L Environment

Figure 3-1. Integrated Design Process (IDP) Proposed for the WD2L Environment Development

Chang S. Nam

Small Group Evaluation

2nd Expert Review

One-to-one Evaluation

1st Expert Review

Formative Evaluation

Instr uct i o n Syst em

54


Chang S. Nam

55

Description of the Integrated Design Process (IDP) As seen in Figure 3-1, the Integrated Design Process (IDP) consists of four design phases - needs analysis, conceptual design, development, and formative evaluation – each of which has its own design processes. The proposed IDP considers two main systems of the WD2L environment, the instruction and user interface system, starting from the early Needs Analysis phase to the Formative Evaluation phase. The IDP is also proposed for the personal hosting of Web materials of the course. To ensure that a final WD2L environment fulfills the functionality and intended users’ needs, the IDP adopts IBM’s (2003) six user-centered design (UCD) principles. The UCD principles offer designers a number of critical advantages by enabling them to “develop ease-touse products, satisfy customers, decrease expenditures on technical support and training, advertise ease-of-use successes, and ultimately increase market shares” (p.13). However, it is also true that these UCD principles are not widely practiced even if there are many potential benefits (Adler & Winograd, 1992; Greenbaum & Kyng, 1991). Table 3-1 summarizes UCD principles and gives a brief description of how each principle was applied in the IDP. The application of UCD principles in the IDP was described in more detail in the following sections. Table 3-1. Descriptions of the UCD Principles Applied in the IDP UCD Principle I. Set Business Goals II. Understand Users III. Assess Competitiveness IV. Design User Experience V. Evaluate Designs VI. Manage by Continual User Observation

Description • Determining the target application, intended users, and primary competition • Understanding the intended users in terms of their tasks, skill levels, etc. • Evaluating the overall quality of the WD2L environment based on - Experts’ recommendations - Usability criteria • Collecting users’ feedback early and often and applying it consistently • Applying learning theory for instructional system • Designing learning activities supported by user interface • Performing formative evaluation plan • Iterating design solutions • Involving the users in the design process

Chang S. Nam


56

Principle I (Set business goals): The IDP begins with an early determination of business goals such as the target application, intended users, and primary competition of the WD2L environment to be developed. These goals are determined in the Needs Analysis phase, but they are applied throughout all development processes. For example, one of the main goals is to provide engineering students (intended users) with a time and spatial independent self-paced (competition) supplemental learning environment (target application) on the Web. Principle II (Understand users): The proposed IDP provides design steps for a better understanding of who the intended users are, what types of tasks they perform, and what their needs are. This user profile is summarized in the Requirements Specification Document (see Appendix 4 for details) and is frequently consulted through the development process. Principle III (Assess competitiveness): The overall quality of the WD2L environment prototype is systematically evaluated by two agents: experts’ recommendations and predetermined usability criteria. First, the WD2L environment prototype developed will be assessed by subject matter experts (SMEs) who exhibit the highest level of expertise in three main areas: instructional design, user interface design, and GPS content. The WD2L environment prototype is also evaluated by a group of representative users against usability criteria such as effectiveness, efficiency, and satisfaction (ISO 9241-11, 1998). Principle IV (Design user experience): To effectively design the total user experience, the IDP provides design steps to understand how people learn and to incorporate that knowledge. The IDP also allows collecting users’ feedback as early and as often as possible and applying it consistently when revising the WD2L environment prototype.

The IDP supports the

development of the WD2L environment in such a way that the user interface system effectively supports the users’ learning activities. Principle V (Evaluate designs): The IDP provides a formative evaluation framework whose foci are both the instructional system and the user interface system. The IDP also provides evaluation processes, instruments to be used, and evaluation criteria to systematically evaluate and revise both systems of the WD2L environment prototype.

Chang S. Nam


Principle VI (Manage by continual user observation):

57

The IDP supports user

involvement at any stage of the WD2L environment prototype development by providing users’ feedback that can be used for the design refinement, modification, and iteration. This study also offers the Design Process Template to help implement each step of the design process, as shown in Figure 3-2. There are two main reasons for providing the Design Process Template. First, the template is intended to provide factors that should be considered in each design process, such as process objectives, inputs, design steps, outputs, methods and tools. Another reason is that information and developmental factors needing to be considered are not constant because of changes in technology, course structure, and users’ needs, for example. Although it is not intended to be exhaustive, the Design Process Template will be able to help address such issues when developing the WD2L environment prototype. The design processes in each phase are described in more detail in the following sections. DESIGN PROCESS TEMPLATE Phase

The name of the phase in the model

Process Description

The description of the process briefly defines the process.

Process Objectives

The objectives of the process are statements that define the direction that needs to be taken in the process.

Inputs

The inputs to a process represent the elements, materials, resources, or information needed to complete the design steps in the process.

Design Steps

The design steps are those actions or sub-processes that together produce the outputs of the process. For some processes, arrows are omitted due to the non-sequential nature of the steps.

Process

The name of the process in the phase

User Interface System Instruction System

Outputs Methods/ Tools

The outputs represent the end result of the process: the product, deliverable, information or resource that is produced. The methods/tools are collections of resources used to accomplish process objectives.

Figure 3-2. An Example of the Design Process Template

Chang S. Nam


58

Phase 1: Needs Analysis This first phase, Needs Analysis, was concerned with gathering, analyzing, and summarizing information necessary to build the Web-based distance and distributed learning (WD2L) environment prototype. The Needs Analysis phase consisted of three design processes: Requirements Specification, Features and Components Identification, and Design Goals Setting. This phase was conducted to identify factors that may influence the development of the WD2L environment prototype, including immediate needs, requirements, and new design concepts for the development of the instruction and interface system prototype. The Needs Analysis phase also determined high levels of design goals that would guide the entire design process. It is universally understood in the design process that specific goals should be established early in the development process to drive all design decisions. In the words of Mayhew (1999): Establishing usability goals on a development project serves two primary purposes. First, specific, articulated usability goals help to focus user interface design efforts during the design process by giving designers something concrete to aim for and something concrete to assess their design ideas against as they generate and consider them……..The second purpose of usability goals is to serve as acceptance criteria during usability evaluation, especially towards the end of the design process (pp. 123-124).

Process 1. Requirements Specification

The Requirements Specification process provided various design activities involved in capturing abstract, high-level development goals as well as more specific requirements necessary to develop the WD2L environment prototype.

Figure 3-3 represents the Design Process

Template for the Requirements Specification process. The main objective of the process was to specify user- and system-related requirements while developing a full understanding of the target user group and its tasks. Eight design steps performed in this process were described in more detail. 1. Gathering Information: This step collected basic information necessary to produce user-related requirements using various methods/tools. The information obtained in this step was analyzed into more specific user- and system-related requirements. Table 3-2 summarizes methods, purposes, and subject(s) administered in the Gathering Information step.

• Semi-structured interview • Observation

Methods/ Tools

Identifying User Tasks

Figure 3-3. Design Process Template: Requirements Specification Process

• Survey • Literature review

• Requirements specification document

Instruction System

Describing User Group

Outputs

Design Steps

Gathering Information Reviewing Current Course

Specifying Instructional Needs

Identifying Interface Design Requirements

Describing RMA Requirements

• Interview results • Survey results • Literature review

Inputs

Specifying Technology Capabilities

• Understand the target user group • Specify user- and system-related requirements

Process Objectives


The Requirements Specification process describes various design activities involved in capturing abstract, high-level development goals as well as more specific requirements necessary to develop the WD2L environment prototype.

1. Requirements Specification

Process Description

Process

59

I. Needs Analysis


Phase

Chang S. Nam


Chang S. Nam

60

Table 3-2. A Summary of Methods Used in the Requirements Specification Process Method Semi-structured Interview I Semi-structured Interview II GPS Survey I GPS Survey II Observation Literature Review

Purpose • To collect teacher’s opinions on: - Effective GPS course components - Learning environment to effectively support it • To ascertain GPS learners’ needs • To identify GPS learners’ desired components of Web-based learning systems • To observe students’ learning activities • To review established studies on Web-based instruction & existing distance learning systems

Participants Professor 1 at VT (Fall 2001) Professor 2 at North Carolina A&T (Fall 2001) Students (Fall 2001) Students (Fall 2002) Students (Fall 2001 & 2002) -

First, to obtain GPS professors’ opinions on an effective GPS course components or characteristics and a learning environment to effectively support it, semi-structured interview questions were developed and provided to two GPS professors teaching at VT (professor 1) and North Carolina A&T State University (professor 2). These interview questions, which were slightly different from each other due to differences in the courses, were sent to and received from the two GPS professors via email. Appendix 1 provides a full list of the semi-structured interview questions and a summary. Table 3-3 shows sample questions and responses. Table 3-3. Sample Interview Questions and Responses Sample Question To effectively fit into an accredited engineering curriculum, what features must any courses in GPS emphasize? To master GPS fundamentals, what knowledge do you think a student must utilize? To facilitate a student’s learning, what do you think a GPS course must provide?

Response • Fundamental engineering science and design criteria (professor 1) • Design of GPS equipment such as receivers (professor 2) • Ability to solve real world problems (professor 2) • Wide spectrum of engineering areas (professor 1) • Physics (professor 2) • Satellite Communications (professor 2) • Well-equipped laboratory (professor 1 & 2) • Exercise the use of MATLAB in GPS applications (professor 2) • Practicality of GPS with other navigation techniques (professor 2)

Both respondents suggested that to effectively fit into an accredited engineering curriculum, a GPS course must emphasize fundamental engineering science and design criteria (professor 1) and a student’s ability to solve real world problems (professor 2). Students must have a basic knowledge in a wide spectrum of engineering areas from physics to satellite communications theory. To facilitate a student’s learning, both respondents suggested a wellequipped laboratory.

Chang S. Nam


61

Second, to ascertain the needs of GPS learners on the course and the WD2L environment, two different surveys were administered to two different groups of previous students who took the course in Fall 2001 and Fall 2002. The GPS Survey I (see Appendix 2 for questions and summary) was designed to ask students’ opinions about the course who took the course in Fall 2001.

This online survey consisted of three parts: lecture, teaching laboratory, and

demographics. Another survey, GPS Survey II (see Appendix 3 for questions and summary), was administered to students who took the course in Fall 2002 to identify students’ desired system components necessary to develop the WD2L environment prototype. Third, the author attended and observed the class several times, while having informal conversations with students. Finally, this study also consulted previous studies on Web-based instruction and existing distance learning systems. To see the overall site structure and its navigational flow, for example, this study reviewed two distance-learning delivery systems: the Blackboard system at VT (www.learn.vt.edu) and the eCollegeSM system at North Carolina A&T State University’s Online Virtual campus (www.ncatonline.org). 2. Describing the User Group: By analyzing information obtained in the previous step, target user classes were identified by their work context roles in this step. The description of the user group was made primarily based on students’ characteristics who took the class in Fall 2002, which were obtained through the GPS Survey II. Four different types of users and their activities were identified as follows: •

Students: Engineering students in the Electrical and Computer Engineering and other engineering departments (e.g., Aerospace and Civil Engineering), who will take advantage of the WD2L environment.

•

Instructor: This user class serves as a resource to the students. The instructor manages the Web-based learning environment by providing informative feedback on student work and discussion and evaluating students.


Chang S. Nam •

62

Administrator: This user class is a highly trained individual with a specific knowledge of computers and Web-based learning environments.

•

Graduate Teaching Assistant: This user class would be composed of users who assist the instructor in teaching the class. Of the four potential user groups identified, this study focused only on the student user

group and its activities. Table 3-4 summarizes student users’ characteristics. The age of 29 student users ranged from 21 to 30 years (M = 22.3, SD = 2.5); Seventeen percent of users are female and eighty-three percent of users are males. Seventy-five percent of users are from the electrical engineering department, while twenty-five percent are from aerospace engineering department. Thirty percent are graduate students and 70% are undergraduate students. Students used their personal computer (PC) with monitor size of 15 (24.1%) or 17 inches (65.5%) on a daily basis (100%), connected by Ethernet (69%) or 56K (13.8%). Most of the users were web savvy who purchase goods online at least once a month and almost half of them have an elearning experience. Users’ preferred browsers were Microsoft Internet ExplorerTM 5.0 and higher and NetscapeTM 6.0 and higher. Table 3-4. Description of Student User Characteristics Matrix Age Gender Department Education level Computer type Online frequency Connection speed

Online habits Browser type Monitor size E-learning experience

Description • 21 ~ 30 years (M = 22.3, SD = 2.5) • Female (17 %) • Male (83 %) • Electrical engineering (75%) • Aerospace engineering (25%) • Graduate students (30%) • Undergraduate students (70%) • Personal computer (PC, 100%) • Daily (100%) • Ethernet (69.0%) • 56K (13.8%) • Cable modem (10.3%) • T1 (6.9%) • Frequently visited sites: News, VT homepage, etc. • Online purchase: at least once a month (e.g., amazon.com, ebay.com, etc.) • Web savvy • Microsoft Internet ExplorerTM 5.0 and higher • NetscapeTM 6.0 and higher • 15” (24.1%) • 17” (65.5%) • 19” (10.3%) • Yes (41.4%) • No (58.6%)


Chang S. Nam

63

This step also analyzed users’ motivation levels using Keller’s (1987) Attention, Relevance, Confidence, and Satisfaction (ARCS) framework. This analysis was based on class observations, informal conversations with the students and an interview with the GPS professor at VT. Table 3-5 shows a summary of the students’ motivation levels. Table 3-5. Student Users’ Motivation Levels ACRS

Level

Attention

Initially low

Relevance

Moderate to high

Confidence

Variable

Satisfaction

Positive

Description • GPS course is thought to be difficult. • GPS course is the requirement course. • Students consider the course an opportunity for training them as highly skilled engineers. • Students who have already taken GPS-related courses are quite confident. • Students who have not taken GPS-related courses are concerned about their ability to complete the course • Students think of the GPS course as one of the most useful courses.

First, the level of Attention was identified as being initially low. The GPS course had a reputation for having many engineering theories and complex mathematical equations. Furthermore, students were likely to be in the class because they had to take it as a requirement course. Second, the level of Relevance was from moderate to high. The goal of the course was to provide students with the fundamental theory and concepts of the Global Positioning System and to demonstrate these concepts through laboratory exercises and a final design project. Therefore, students considered this course an opportunity for training them as highly-skilled engineers. Third, the level of Confidence was identified as being variable. The primary electrical engineering concepts involved in this class were: (1) Radiowave propagation and (2) Communication system theory. Satellite orbit dynamics was the third important aspect of this course. Students who already took those courses expressed their confidence in their ability to take this course. On the other hand, students who did not take one of these courses were concerned about their ability to complete this course.

Chang S. Nam


64

Finally, the level of Satisfaction was positive. Students thought of this course as one of the most valuable courses because it provides many fundamental concepts about GPS technology as well as allowing many experiences with modern technology. 3. Identifying User Tasks: This step identified possible user activities (or tasks) on the web-based distance learning system. To identify user tasks on the WD2L Environment, this step used various methods including GPS Survey II, observation, and overview of the two distance systems (i.e., Blackboard and eCollegeSM systems). Table 3-6 shows examples of user tasks, and a full list can be found in Appendix 4. Table 3-6. Potential User Tasks on the WD2L Environment • Check Announcement • See Instructor’s Background • See Information on GTA • Check Course Description • See Course Objective • Check Grades

• Upload homework • See GPS resource materials • See Homework Results • See Quiz Results • See Exam Results • Go to VT homepage, Library, & Hokie Spa

4. Specifying Technology Capabilities: This step specified technology capabilities that are needed to successfully support potential users.

As a minimum browser, for example,

Netscape (4.0 or above) and Internet Explorer (5.5 or above) shall be required. A minimum connection speed of 56k or above shall be required.

Appendix 4 provides a full list of

technology capabilities specified in this step. 5. Identifying Interface Design Requirements: This step described how users would interact with the WD2L environment, primarily based on user tasks identified in the previous step.

User interface systems in the Blackboard system and eCollegeSM system were also

consulted. Table 3-7 shows how interface design requirements were specified to support users’ activities on the WD2L environment. For example, the WD2L environment shall provide the structure of menus that allows the user to select menus anywhere in order to support various activities. Users also wanted to check

Chang S. Nam


announcements posted by either the professor or teaching assistant.

65 Therefore, the WD2L

environment shall allow the user to select the day that announcements are provided. Table 3-7. Examples of Interface Design Requirements that Support User Tasks Desired Activity Various activities Go to VT homepage, Library, & Hokie Spa Check Announcement Find definitions of GPS term Perform laboratory activities

Interface Design Requirement Shall provide the structure of menus that allows the user to select them anywhere Shall allow the user to access external links, including the VT homepage, library, Hokie Spa Shall allow the user to select the day that announcements are provided Shall provide the user with the capability to find the definition of GPS terminology Shall provide the capacity for the user to practice MATLAB scripts

Users wanted to go directly to the VT homepage, Library, and Hokie Spa websites, which is not implemented in Blackboard and eCollegeSM systems. The WD2L environment shall allow the user to access external links including the VT homepage and library without having to leave the WD2L environment. SM

eCollege

Another design requirement that was missing in Blackboard and

systems was the support for laboratory activities. The WD2L environment shall

provide the capacity for users to practice MATLAB scripts, one of the most important laboratory activities. Appendix 4 provides a full list of user interface design requirements. 6. Reviewing Current Course:

To address developmental situations before the

development of the new learning environment for the course, these aspects were discussed across five dimensions: context, delivery mode, time frame, content, and audience characteristics. The overview of the current course was described in Chapter I. 7. Specifying Instructional Design Requirements: This step defined instructional design requirements that WD2L environment must meet in order to support users’ learning activities effectively. Main sources used in this step included results of the literature review and user tasks.

As discussed in Chapter II, for example, this study proposed that many principles

provided by cognitive learning theory would be suited for redesigning the learning content of the course. An example of an instructional design requirement is that the WD2L environment shall organize the learning content of the course in such a way that the user is able to connect new information with existing knowledge in a meaningful way.

Chang S. Nam


66

On the other hand, this study sought to employ nine events of instruction suggested by Gagne et al. (1992) as a Web-based instructional strategy given common learning activities (e.g., problem solving, inference generating, critical thinking, and laboratory activities) and types of learning domains (e.g., intellectual skills and verbal information). Appendix 4 provides a full list of instructional design requirements. 8. Describing Reliability, Maintainability, and Availability Requirements:

These

requirements addressed, at the very highest level, the needs associated with the WD2L environment’s availability and maintenance. The WD2L environment shall: •

Be available 24 hours per day, 7 days per week to support the GPS learning on the Web.

•

Be maintainable by a Webmaster.

As a result of performing design steps described above, the Requirements Specification process led to the development of the Requirements Specification Document (see Appendix 4). This document is a summary of user- and system-related requirements, which should be considered for the design of instruction and user interface system, such as user interface, technology, and instructional design requirements. The Requirements Specification Document is intended to serve three purposes for the WD2L environment development: •

Provide a shared understanding of development goals and user requirements of the WD2L environment

•

Identify immediate needs and new design concepts necessary to develop the instruction and interface system

•

Describe all possible factors that may influence the implementation of the WD2L environment

Chang S. Nam


67

The Requirements Specification Document also provides development goals for an effective WD2L environment: •

Provide learners with a time and spatial independent self-paced learning environment on the Web

•

Provide learners with a unique opportunity to pull together many fundamental engineering concepts

•

Provide learners with early and regular opportunities to test their understanding

•

Provide communication channels for increasing and maintaining student-to-student and student-to-teacher interaction outside of the lectures

•

Provide various opportunities for learners to allow gaining hands on experience with modern GPS technology

•

Provide learners with a variety of resources, available to the entire learning community

•

Provide user interfaces to effectively support learning activities on the Web

Chang S. Nam


68

Process 2. Features & Components Identification

The main objective of this process was to identify key features and corresponding components that constituted an effective WD2L environment. Figure 3-4 shows the Design Process Template for the Features & Components Identification process. A well-designed web-based instruction (WBI) program should provide various features conducive to learning, instructions, and the use of the WD2L environment (Khan, 1997). To identify such features and corresponding components, this study used three sources as input to the process: 1) Khan’s (1997) list of WBI features and components, 2) Oliver’s (2003) online learning tools categorized by functions they serve, and 3) a requirements specification document. Design steps conducted in this process are described below. 1. Reviewing Requirements Specification Document: Various functions that students wanted Web-based learning environments to have were reviewed, which were specified in the Requirements Specification process. 2. Reviewing Khan’s (1997) List: The second source was Khan’s (1997) list of 28 features and corresponding components that a well-designed WBI program should provide. Table 3-8 shows some examples of features and components associated with WBI learning environments provided by Khan. Based on Khan’s (1997) list, features and their contributing components that constitute the WD2L environment were identified to support tasks that users will perform on the WD2L environment. Table 3-8. Examples of Features and Components Associated with WBI Features Interactive

Multimedia Distributed

Components • Internet tools • Hyperlinks • Instructional design • Browsers • Web-based conferencing • Internet tools • Databases • Hyperlinks

Features Electronic Publishing

Collaborative Learning Online Evaluation

Components • E-mail • Authoring tools • Listserv • Internet tools • Instructional design • Practice • Online quiz • Databases

Figure 3-4. Design Process Template: Features and Components Identification Process

• Khan’s (1997) and Oliver’s (2003) list

Methods/ Tools

Reviewing Oliver’s (2003) List

• List of key features and conducive components categorized by the functions served

Instruction System

Determining Key Features & Components

Outputs

Design Steps

Reviewing Requirements Specification

• Khan’s (1997) list of WBI features and components • Requirements specification • Oliver’s (2003) list of online tools categorized by functions to be served

Inputs

Reviewing Khan’s (1997) List

• Identify key features conducive to learning and instruction • Specify system components

Process Objectives


This process describes design activities to identify features and components necessary to implement the WD2L environment.

2. Features & Components Identification

Process Description

Process

69

I. Needs Analysis


Phase

Chang S. Nam

Chang S. Nam


70

3. Reviewing Oliver’s (2003) List: Another source used was Oliver’s (2003) online learning tools categorized by functions they serve, such as information processing and scaffolding functions. For example, search engines and site maps can be used to support a user’s information seeking process (IP1). On the other hand, electronic file boxes can serve the function of collecting information (IP5). Table 3-9 lists Oliver’s (2003) online learning tools sorted according to functions they serve. Table 3-9. Functions of Online Learning Tools Dimensions Information Processing Functions

Functions IP1: Seeking Information IP2: Collecting Information IP3: Organizing Information

Empower students to Generic Examples Locate or confirm the presence of online Search engines, indices, information related to a search topic site maps Accumulate relevant information in Downloading, saving, accessible formats or locations electronic file boxes Chunk two or more external information Constructing tables, units with similar characteristics into a charts, diagrams category IP4: Integrating Merge their own personal ideas or Annotating information Information concepts with external online information encountered IP5: Generating Create a new conception based on Collaborative web Information information collected and organized editing IP6: Manipulating Test predictions and hypotheses; revise Simulated events and Information personal theories via trial and error processes IP7: Communicating Discuss course content, share Bulletin boards, chat, Information information, or critique others’ work instant messages IP8: Accessing Receive help to access information in Text to speech Information alternative formats conversion, captioning Scaffolding S1: Procedural Follow guidelines, utilize a system or Software tutorial, Functions software online lab manual S2: Conceptual Gain perspective on content to be studied, Course goals, content recall pertinent prior knowledge objectives S3: Strategic Set goals and timelines for projects, Task manager, chart problems or processes maker S4: Metacognitive Monitor progress in course; select Practice quizzes, grade appropriate study strategies books (Source:http://www.learn.vt.edu/courses/1/OnlineLearningTools/content/_19642_1/tools2.html)

4. Determining Key Features & Components: Using those three sources, key features and corresponding components that will be implemented in the WD2L environment prototype have been determined. Each component was intended to support information processing and scaffolding functions categorized by Oliver (2003). Table 3-10 shows a summary of 14 key features, their relationship to the WD2L environment, contributing components, and functions to be served.


Chang S. Nam

71

Table 3-10. Key Features and Components of WD2L Environment Feature Interactive

Multimedia

Open System Online Search Device, Distance, & Time Independent Globally Accessible Electronic Publishing

Distributed Multiple Expertise Learner Controlled

Collaborative Learning Formal & Information Environments

Online Evaluation

Virtual Cultures

Relationship to WD2L Environment • Allow interactions with students, instructors, and Web resources via various communication channels • Provide interactive feedback on students’ performance • Support students’ various learning styles using a variety of multimedia

• Allow students to move outside the WD2L environment • Provide search engine to find relevant information • Allow the use by anyone at anytime using any computer platform • Allow the rapid access to resources available on the Web • Allow posting students’ opinions and uploading their work • Allow downloading and printing the materials from the WD2L environment and any other Web sources • Take advantage of sources available on the Web that are provided by GPS experts • Put students in control so they have a choice of content, study time, and pace

• Create a medium of collaboration, conversation, discussion, exchange, and communication of ideas • Provide the course syllabus, announcements, assignments, and related resources • Allow submitting assignments and engaging in discussion

• Allow taking online quiz and performing practices • Provide informative feedback • Provide an opportunity for students to engage in a wide range of discussion topics with their peers and GPS experts outside

Component Send Email Discussion Board Practice Sessions Quiz Prelaboratory Concept Map Text to Speech Advanced Organizers Captioning Links to VT Hypermedia Search Engine

Function IP4, IP7 IP4, IP7 IP4, IP7 IP6, S4 IP6, S4 S2 IP8 S2 IP8 IP1 IP1 IP1

All components

All functions

Hypermedia Internet Connection Send Email Discussion Board Assignments Document Sharing GPS Resources

IP1 IP1 IP4, IP7 IP4, IP7 IP2 IP5 IP1, IP2, IP4

GPS Glossary

IP1, IP2, IP4

Discussion Board GPS Resources Guides Help Search Tips Document Sharing GPS Resources GPS Glossary Discussion Board (By Group)

IP4, IP7 IP1, IP2, IP4 S1 S1 S1 IP5 IP1, IP2, IP4 IP1, IP2, IP4 IP4, IP7

Course Objectives Announcements Course Information Staff Information Assignments Document Sharing GPS Resources GPS Glossary Practice Sessions Quiz Prelaboratory Send Email

S2 IP1 IP1 IP1 IP2 IP5 IP1, IP2, IP4 IP1, IP2, IP4 IP6, S4 IP6, S4 IP6, S4 IP4, IP7

Discussion Board

IP4, IP7

Chang S. Nam


72

Process 3. Design Goals Setting The Design Goals Setting process described the determination of design goals and principles that drove all design decisions throughout the development. These design goals will also serve as evaluation criteria for usability testing in the Formative Evaluation Phase. The main objective of the process was to specify design goals and principles for both the instruction and user interface system. Figure 3-5 represents the Design Process Template for the Design Goals Setting process. As inputs to the process, the information obtained in previous processes was used, such as the requirements specification document as well as features and their components determined for the WD2L environment. With those process inputs, several design steps were performed. 1. Defining Design Goals of User Interface System: As design goals for the interface system, this study chose ISO 9241-11 (1998) usability goals: effectiveness, efficiency, and satisfaction.

Effectiveness is defined as the “accuracy and completeness with which users

achieve specified tasks” (p. 5). Efficiency is defined as the “resources expended in relation to the accuracy and completeness with which users achieve goals” (p. 5). As a subjective measure, satisfaction concerns the “comfort and acceptability of use” (p. 5). 2. Defining Design Goals of Instructional system: As design goals for the instructional system, this study followed Dick and Carey’s (1996) evaluation criteria: clarity of instruction and impact on learner. Clarity is a design goal to make sure if the message or what is being presented is clear to individual target learners. Impact is intended to increase an individual learner’s attitude.

Figure 3-5. Design Process Template: Design Goals Setting

• ISO 9241-11 Standard • Norman’s (1990) Four Design Principles • Instructional Design Principles from Learning Theories

Identifying Design Principles of Instruction System

Methods/ Tools

Defining Design Goals of Instruction System

• Design goals for the user interface and instructional system • Design principles for the user interface and instructional system

Instruction System

Outputs

Design Steps

Identifying Design Principles of User Interface System

• User interface requirements • Instructional system requirements • Features and their components determined for the WD2L environment

Inputs

Defining Design Goals of User Interface System

• Determine design goals for the instruction and user interface system • Identify design principles for the instruction and user interface system

Process Objectives


The design goals setting process describes the determination of the design goals that will drive all design decisions throughout the development.

3. Design Goals Setting

Process Description

Process

73

I. Needs Analysis


Phase

Chang S. Nam


Chang S. Nam

74

Table 3-11 shows design goals that will govern all design decisions throughout the development of the WD2L environment. Table 3-11. Design Goals for WD2L Environment Development System User Interface System Instructional system

Design Goal • Effectiveness • Efficiency • Satisfaction • Clarity • Impact

Description • To increase the accuracy and completeness in specified tasks • To reduce the resources expended • To ensure users’ comfort and acceptability of use • To make learning materials clear • To increase users’ attitude

3. Identifying Design Principles of User Interface System: The primary goal of the user interface was to design the interface so the user can quickly and easily complete tasks by allowing simple, natural interactions with the WD2L environment. For principles of the user interface design, this study employed Norman's (1990) four principles of good design: visibility; good conceptual model; good mapping; and feedback. The Visibility principle indicates that the use of a device should be as visible as possible to a user by clearly indicating the state of the device, functionality, and the alternatives of action. That is, good visibility allows the user to effectively interact with the device (Norman, 1990).

A good conceptual model refers to

consistency in the presentation of user operations and results, which in turn allows the user to predict the relationships between his/her actions and subsequent results (i.e., good mapping principle). That is, these two principles refer to the user’s belief that clicking this button will always display the page. Finally, the feedback principle refers to informative feedback that the users receive on their actions. 4. Identifying Design Principles of Instructional system: For the instructional system, this study applied instructional design principles and guidelines provided by both Behaviorism and Cognitivism to learning. Table 3-12 details instructional design principles and guidelines drawn from the two learning theories, which were reviewed in Chapter II. In summary, the Design Goals Setting process identified main design goals and principles for the development of the user interface and instructional system. These design goals and principles were sought to make sure of 1) the development of usable as well as learner-centered

Chang S. Nam


75

user interface and instructional system, and 2) natural interactions of users with the WD2L environment. Table 3-12. Instructional Design Principles and Applications Learning Theory Behaviorism

Cognitivism

Instructional Design Principle • Emphasis on producing observable and measurable outcomes in students • Pre-assessment of students to determine where instruction should begin • Emphasis on mastering early steps before progressing to more complex levels of performance • Use of reinforcement to impact performance • Use of cues, shaping and practice to ensure a strong stimulus-response association • Active involvement of the learner • Use of hierarchical analyses • Structure, organize, and sequence information • Link new information to existing knowledge

Possible Application • Identifying behavioral objectives, performing task analyses, and conducting criterion-referenced assessments • Learner analysis • Sequential presentation and mastery of learning • Tangible rewards and informative feedback • Simple to complex sequencing of practice and use of prompts • Learner control • Meta-cognitive training (e.g., self-planning, monitoring, and revising techniques) • Cognitive task analysis • Outlines, summarizes, synthesizers, advance organizers • Recall and recognition • Relevant examples • Analogies


Chang S. Nam

76

Phase 2: Conceptual Design The Conceptual Design phase was concerned with an explicit construction of ideas or concepts about what a WD2L environment is, what it can do, and how it is intended to be used. This phase consisted of four design processes for translating user requirements into a conceptual user interface and instructional design: design scenarios development, information design, structure design, and page design. To transform existing materials into new, effective materials redesigned for the WD2L environment, the 3A Transformation Model proposed in this study was applied throughout the Conceptual Design phase. The 3A Transformation Model, as represented in Figure 3-6, was also used throughout the Development and Evaluation phase.

Current State

Transformation

Future State

Audit

Apply

Assess

Instruction System

• Existing learning materials • Instructional strategies


• Structure design • Page design • Information design

• Learning theories • Instructional design theories • Instructional strategies for WBI • Heuristic evaluation • Initial User Evaluation • Interface design approaches • Formative evaluation • Design for usability • Design guidelines

Figure 3-6. 3A Transformation Model First, existing materials such as learning content, instructional strategies, and site structures of existing distance learning systems were inspected to examine their problems and advantages. Second, these audited results were transformed into newly designed materials for both the instruction and interface system. To address the problems of existing materials and retain their advantages, transformation was done through the application of various methods, such as theories (e.g., learning theories), concepts (e.g., design for usability), and frameworks (e.g., instructional strategies for Web-based instruction). Finally, these new materials were

Chang S. Nam


77

assessed mainly by the heuristic evaluation, initial user evaluation, and formative evaluation plan. The output of the Conceptual Design phase was an outline of the user interface and instructional system prototype, which were further developed during the Development phase. Each of the conceptual design processes was described in more detail in the following sections.

Process 1. Design Scenarios Development

The Design Scenarios Development process described a set of design steps for developing design scenarios that reflect users’ key tasks. Figure 3-7 shows the Design Process Template for the Design Scenarios Development process. The main objective of the process was to create design scenarios that represented key user tasks on the WD2L environment so that they can be used for the conceptual design of the user interface and instructional system prototype. These design scenarios were developed to reveal as much detail as possible about users’ learning activities as well as relevant user interface objects to support their behaviors on the WD2L environment. As process inputs, this study used information obtained in the previous processes, including the requirements specification document and WD2L environment features and components. The design steps performed were as follows: 1. Selecting Key User Tasks: Possible tasks that users would perform on the WD2L environment were reviewed, and then several different key user tasks were selected. Since the WD2L environment was intended to support users’ learning activities, key user tasks were selected as tasks involving learning activities (e.g., doing a practice or taking a quiz) that were supported by user interface systems (e.g., providing feedback or sending the quiz). Table 3-13 lists seven key user tasks identified and brief descriptions.

• Stories of usage instances • User profile

Methods/ Tools

Figure 3-7. Design Process Template: Design Scenarios Development Process

• Key user tasks • Design scenarios, representing key user tasks

Instruction System

Selecting Key User Tasks

Outputs

Design Steps

Creating Design Scenarios

• Requirements specification document • WD2L environment features and components determined

Inputs

Specifying User Interface Objects

• Create design scenarios that represent different key user tasks • Understand user tasks

Process Objectives


The Design Scenarios Development process describes a set of design steps to develop the design scenarios, representing key user tasks.

1. Design Scenarios Development

Process Description

Process

78

II. Conceptual Design


Phase

Chang S. Nam


Chang S. Nam

79

Table 3-13. Key User Tasks on the WD2L Environment Key Task Search Upload Study Practice Email Discussion Assessment

Description Finding information on the WD2L Environment Uploading assignments on the Web Studying learning contents (e.g., Chapter 5) Practicing what has been learned Sending an email Participating in discussion Taking a quiz

2. Specifying User Interface Objects: User interface objects that users may use when performing key user tasks were identified. To specify user interface objects, this step relied on system components determined for the WD2L environment and identified from existing learning delivery systems (i.e., Blackboard and eCollegeSM system). Examples included menus, submenus, buttons, hyperlinks, and text boxes. 3. Creating Design Scenarios: Based on key user tasks and user interface objects specified, design scenarios were created. Figure 3-8 presents an example of design scenarios, which shows a set of user activities to study learning content (i.e., Chapter 5). These scenarios evolved through iteration into detailed narrations that describe how tasks are performed in terms of specific physical user actions (e.g., click or select) on specific user interface objects (e.g., links or menus) with specific attributes specified (e.g., hyperlinks). Design scenarios as outputs of the process included several user interface objects, such as buttons, menus, dialog boxes, and navigation through the learning environment. 2

Design

scenarios can help make sure that the WD L environment had main functions necessary to allow interactions among learners and between learners and the instructor.


Chang S. Nam

User role - learner

Sub-task

80

Object attribute

After getting into the GPS Theory & Design Website, John who is taking the GPS (ECE4164) course checks out the Announcements, and finds out a new announcement where a quiz about corrections to Keplerian orbits for precise positioning (Chapter 5) has been posted by the instructor. He selects the Ch. 5 in the Lecture Notes sub-menu of the Classroom menu. At the top of the page, objectives of chapter 5 are provided, describing what students will learn and what kinds of achievement they will make after completing this chapter. He also reviews the “Table of Contents” where each topic is hyperlinked to the corresponding learning unit. He clicks the Introduction link, and study it. To make sure that he has a full understanding of the basic knowledge of Chapter 5, he clicks the Practice 1 link where it allows practicing what has been learned and getting feedback on his performance. object

Physical action

Figure 3-8. An Example of the Design Scenario

object

Chang S. Nam


81

Process 2. Information Design

The Information Design process described the conceptual design of information content for the instructional system and user interface system. Figure 3-9 represents the Design Process Template for the Information Design process. The main objective of the process was to identify required content for the user interface and instructional system and conceptually outline them. In the Information Design process, four design steps were performed to achieve the process objective. 1. Auditing Existing Learning Content: Existing learning content that can be incorporated into the user interface and instructional system were reviewed to set aside unnecessary content while identifying those that should be provided on the WD2L environment. Examples of existing materials included the textbook, quiz questions, homework assignment, prelaboratory materials, and design project materials. 2. Outlining Interface Content: In this design step, content related to the user interface system were identified and outlined, such as the help document and search tips.

Design

scenarios developed in the previous process were also used to outline interface-related content. 3. Outlining Learning Content: To outline learning content, this study applied learning theories as well as their instructional design principles (see Table 3-11). Table 3-14 shows an example of how learning content of the instructional system were conceptually designed to meet user requirements by applying instructional design principles drawn from cognitivism to learning. Table 3-14. An Example of Theory-Based Design of Learning Content Requirement Provide efficient information processing strategies to support complex GPS learning

Design Principle • Emphasis on structuring, organizing, and sequencing information to facilitate optimal processing

Learning Theory Cognitivism

Learning Content • Concept map • Think for a while • Interactive practice sessions

• Theory-based design of the learning content • Integration of information content

Methods/ Tools

Outlining Learning Content

Figure 3-9. Design Process Template: Information Design Process

• Theoretically designed learning content • Content Outline Document

Instruction System

Outputs

Design Steps

Integrating Information Content

Inputs

Auditing Existing Information Content

• Requirements specification • Learning theories • Instructional design principles • Design scenarios Outlining Interface Content

• Identify the contents required • Design conceptual content

Process Objectives


This information design process describes the elements to conceptually design information contents.

2. Information Design

Process Description

Process

82



Phase

Chang S. Nam


Chang S. Nam

83

For example, one of user requirements related to the instructional system (described in the Requirements Specification document) was that since the GPS course involves complex forms of learning, the instructional system should provide users with efficient information processing strategies through which they receive, organize, and retrieve knowledge in a meaningful way. Cognitivism recommends providing several different ways in which learners can connect new information with existing knowledge. By employing this design principle, the “Think for a while!” section was conceptually designed, in which learners think back for a while what they have learned in previous chapters, and how their prior knowledge is related to current topics. Another example can be interactive practice sessions in which learners can practice what they have learned and receive immediate feedback on their performance. 4. Integrating Information Content: Information content identified for the user interface and instructional system were integrated, resulting in the Content Outline Document as an output of the process. The Content Outline Document described a list of the content identified for key user tasks in terms of page titles, page elements to be included, and brief descriptions. Table 315 shows an example of the Content Outline Document (see Appendix 5 for a full description). Table 3-15. An Example of the Content Outline Document Component Search engine

Page Title Search

GPS Glossary

Glossary

Chapter

Chapter5

Elements to be Included • Search box • Advanced search • Search tips • GPS terms • Usage tips • Objectives • Think for a while • Practice sessions • Graphic images

Description Text box to type search word(s) Advanced search option Help for search GPS terms listed alphabetically Short description on how to use Objectives of Chapter 5 Think back what has learned Interactive practice sessions Examples in a graphic image form

Chang S. Nam


84

Process 3. Structure Design The Structure Design process described the main structure of the WD2L environment. Figure 3-10 represents the Design Process Template for the Structure Design process. The main objective of the process was to specify the presentation and storage structure of the WD2L environment. The structure of information in a Web site is important in that well-structured information allow users to effectively perform necessary tasks or access the required information. A Web site is a combination of the presentation and storage structure, which should be as similar as possible to easily maintain the site (McCormack & Jones, 1997). The presentation structure is the user’s mental model of the WD2L environment structure. On the other hand, the storage structure (often called folders) is the actual structure of the WD2L environment (e.g., the hierarchy of files and directories), demonstrating how the material, such as Web pages, files, and other data, is actually stored on the Web server. As inputs for this process, the requirements specification document, design scenarios, and content outline document were used. These inputs were resulted in the structure of the WD2L environment by performing following three design steps. 1. Defining Key User Paths: To identify most common and plausible key user paths through the site, design scenarios detailed in the previous process were used. Figure 3-11 shows an example of a key user path identified.

• Match of the presentation structure with the storage structure • Identification of key user paths in design scenarios

Methods/ Tools

Figure 3-10: Design Process Template: Structure Design Process

• Presentation structure • Storage structure • Key user paths

Specifying Presentation Structure

Outputs

Instruction System

Defining Key User Paths

Specifying Storage Structure

• Requirements specification • Design scenarios • Information design

Inputs

Design Steps

• Define the presentation and storage structure, which are as similar as possible • Define key user paths

Process Objectives


This structure design process describes the key elements necessary to design the structure of the WD2L environment.

3. Structure Design

Process Description

Process

85



Phase

Chang S. Nam

Chang S. Nam


86

After getting into the GPS Theory & Design Website, John who is taking the GPS (ECE4164) course checks out the Announcements in the Information menu, and finds out a new announcement where a quiz about corrections to Keplerian orbits for precise positioning (Chapter 5) has been posted by the instructor. He selects the Ch. 5 in the Lecture Notes sub-menu of the Classroom menu. At the top of the page, objectives of chapter 5 are provided, describing what students will learn and what kinds of achievement they will make after completing this chapter. He also reviews the “Table of Contents” where each topic is hyperlinked to the corresponding learning unit. He clicks the Introduction link, and study it. To make sure that he has a full understanding of the basic knowledge of Chapter 5, he clicks the Practice 1 link where it allows practicing what has been learned and getting feedback on his performance.

Figure 3-11. An Example of Key User Paths 2. Specifying Presentation Structure: There are three different types of the presentation structures: hierarchical, sequential, and hypermedia (McCormack & Jones, 1997). Figure 3-12 shows an example of the hierarchical structure. The WD2L environment basically used a hierarchical structure to organize all of the information, which has a group of menus (e.g., Information, Classroom, Laboratory, Assessments, and Communication) at one level, each of which (e.g., Information) provides access to other groups of sub-menu items (e.g., Announcements, Course Information, and Staff Information). The sequential structure allows users to move through a number of pages, one after another. Some examples included pages in Chapter 5 or GPS Resources. On the other hand, the hypermedia presentation structure takes full advantage of the Web’s hypermedia characteristics, allowing users to follow links that may lead to other pages. Some of the pages on the WD2L environment will need to be designed to support multiple presentation structures.

Chang S. Nam


Information

Announcements Course Information Staff Information Gradebook

Classroom

Classroom Guide Lecture Notes

Homepage

87

Chapter 1 : Chapter 5 : Chapter 11

Document Sharing Resource Materials

Concept Map GPS Resources GPS Glossary

Laboratory Assessments Communication

Figure 3-12. An Example of the Hierarchical Structure 3. Defining Storage Structure: For easy maintenance of the WD2L environment, the same file types should be stored together (McCormack & Jones, 1997). Figure 3-13 provides a representation of the storage structure for the WD2L environment. The home page was titled as index.html and other files were stored in folders created for each section of the site.

For

example, announcements (info_annou.html) and Staff Information (staff_info.html) files are stored in the “Information” directory. Information gps.vt.edu

Information

Announcements (info_annou.html) Course Information (course_info.html) Staff Information (staff_info.html) Gradebook (gradebook.html)

Classroom

Classroom Guide (class_main.html)

Classroom

Lecture Notes

Home page Index.html

Chapter 1 : Chapter 5 : Chapter 11

Practice 1 (cha5practice1.html) Practice 2 (cha5practice2.html) Practice 3 (cha5practice3.html) Practice 4 (cha5practice4.html) Practice 5 (cha5practice5.html)

Document Sharing (document.html) Resource Materials

Concept Map (conceptmap.html) GPS Resources (resource.html) GPS Glossary (glossary.html)

Laboratory Assessments Communication

Figure 3-13. An Example of the Storage Structure

Chang S. Nam


88

Process 4. Page Design

The Page Design process described the determination of content layouts or schematics of main pages, displaying rough navigation and the layout of elements that need to appear on a page. Figure 3-14 represents the Design Process Template for the Page Design process. The main objective of the process was to specify the content layout and navigational organization of a few key pages. For the Page Design process, this study adapted the Wireframing process, provided by Koto & Cotler (2002) for the Web redesign (see pages 103-104 for more details). To determine content layouts of a page, all page content identified in the previous process were reviewed. More specifically, the Content Outline Document was reviewed. 1. Reviewing User Paths: To determine navigation of a page, key user paths identified in the previous process were reviewed. The review of the key user paths helped establish a smooth connection between the pages. 2. Wireframing Content: A simple wireframe first was developed, in which general content and navigation were placed in a representative format for main pages. Figure 3-15 shows an example of the wireframe.

• Hand-drawn screen pictures • Visio or some other simple drawing packages (e.g., PowerPoint, Word, Excel, etc.)

Methods/ Tools

Figure 3-14. Design Process Template: Page Design Process

• Wireframes for main pages

Outputs

Instruction System

Wireframing Content

Inputs

Reviewing User Paths

• Requirements specification • Design scenarios • Information content • Presentation structure

Design Steps

• Specify the content layout of main pages • Specify navigational organization of a few key pages

Process Objectives


This page design process describes the elements necessary to determine contents layouts or schematics of main pages.

4. Page Design

Process Description

Process

89



Phase

Chang S. Nam

Chang S. Nam

Chapter III. Development of WD2L Environment Section A

Logo

90

Site Title Log out

Links to VT Main navigation Navigational history Page title

Section B

Section C

Sub-navigation Contents

Section D

Footer navigation

Figure 3-15. An Example of the Wireframe The wireframe was also used to maintain consistency of the page layout through all pages. For example, Section A will be used for the site title and main menu system. On the other hand, Section B is intended for the sub-navigation, in which the same main menus are placed. They move down as the page is scrolled down. Section C is a place in which all content is displayed, while Section D is a place for the footer navigation. Based on simple wireframes, all major page elements, such as content, links, graphics, functional elements, and messages, were presented in more detail during the Development phase.

Chang S. Nam


91

Phase 3: Development The Development phase was concerned with the construction of a high-fidelity (hi-fi) prototype of the WD2L environment, based on results of the initial user evaluation on a lowfidelity (low-fi) prototype. This phase consisted of three design processes for translating the conceptual user interface and instructional design into the high-fi prototype of the WD2L environment: low-fidelity prototyping, design walk-through, and high-fidelity prototyping. The output of this phase was the high-fi prototype of the WD2L environment, which was evaluated by a group of users in the Formative Evaluation phase. Each of design processes in the Development phase was described in more detail in the following sections.

Process 1. Low-Fidelity Prototyping

The Low-Fidelity Prototyping process described the development of the low-fi prototype of the WD2L environment. Figure 3-16 represents the Design Process Template for the LowFidelity Prototyping process. The main goal of the process was to build a rough interface and instructional system by integrating design ideas developed in previous processes. To do this, all outputs produced in previous design processes were used, including the requirements specification document, content outline document, and structure of the WD2L environment. Design steps performed in this process are as follows: 1. Reviewing Requirements Specification Document:

The first step in the low-fi

prototyping process was to have an understanding of the user’s needs by looking at the Requirements Specification Document. 2. Reviewing Features & Components: Key features and their corresponding component (Table 3-4) were reviewed. For example, the “Discussion Board” function is one of the example components that must be implemented in the prototype to ensure features of the WD2L environment, such as “interactive,” “electronic publishing,” and “virtual culture.”

• Visio or some other simple drawing packages (e.g., PowerPoint, Word, Excel, etc.) • Web design tools (e.g., Dreamweaver, Fireworks, etc.)

Methods/ Tools

Figure 3-16. Design Process Template: Low-Fidelity Prototyping Process

• Low-fidelity prototype

Instruction System

Integrating Structure

Outputs

Design Steps

Integrating Information Content

Integrating Page Elements

Developing Low-fi Prototype

Inputs

Reviewing Features & Components

• Requirements specification document • Instructional and user interface design goals • Design scenarios • Information, structure, and page design


• Develop rough interface and instructional system, which present main features • Integrate design ideas identified

Process Objectives


This low-fidelity prototyping process describes the elements necessary to quickly develop the low-fidelity prototype of the WD2L environment.

1. Low-fidelity prototyping

Process Description

Process

92

III. Development


Phase

Chang S. Nam

Chang S. Nam


3. Integrating Information Content:

93

Information content was reviewed and

implemented. For example, for an interactive feedback screen for the practice 1, informative feedback content has been used. 4. Integrating Structure:

In the previous phase, two types of structures were

conceptually developed for the WD2L environment: the presentation and storage structure. In the low-fi prototyping process, the presentation structure was mainly used in which hierarchical, sequential, and hypermedia structures were implemented. 5. Integrating Page Elements: Page elements identified in the previous process were integrated for the low-fi prototype. The wireframe developed for the main pages (e.g., Figure 315) was used through the process to maintain the consistency of the page layout. 6. Creating Low-fi Prototype: Working from the original design ideas identified in previous phases, the low-fi prototype of the WD2L environment was initially drafted by hand on paper onto 11 x 14 photocopied mockups of a web browser window, which are similar to a typical 800 x 600 display screen. Since many changes were necessary to the paper-based prototype, the next iterations of the prototype were generated on the computer, mainly using Dreamweaver, a web design software application. It became easier to revise the content of the screens using the Dreamweaver software, rather than with the paper-based prototype. After many changes, the final low-fi prototype consisted of 23 screens was developed. Table 3-16 represents the resulting 23 screens. Table 3-16. 23 Screens of the Final Low-Fi Prototype Home page Help Announcements Course Information Staff Information Classroom Guide

GPS Glossary Lab Guide Function References MATLAB 101 Prelaboratory 5 Laboratory 5

Chapter 5 Practice 1 Feedback Screen Document Sharing Concept Map GPS Resources

Assessments Guide Assignment Quiz Send Email Discussion Board. GPS Glossary

Chang S. Nam


94

Process 2. Design Walk-Through

The Design Walk-Through process was concerned with soliciting initial feedback from users by having them walk through the low-fi prototype of the WD2L environment. Figure 3-17 represents the Design Process Template for the Design Walk-Through process. The goals of the process were 1) to confirm that the proposed design of the WD2L environment (i.e., the low-fi prototype) is consistent with target users’ expectations and skill levels, and 2) to use initial feedback to revise the low-fi prototype early in the design process before the full functionality is implemented. As inputs for the initial user evaluation process, the requirements specification document and low-fi prototype were used. The following design steps were performed: 1. Walking through the Prototype: There are several methods that can be used to obtain the user’s initial feedback on the low-fi product prototype, such as participatory design, focus group, or formal usability testing. However, this study obtained users’ initial feedback on the prototype in an informal manner for two reasons. The first reason was to obtain user feedback and use them to revise the prototype as quickly as possible. Another reason was that there was the formative evaluation plan in the IDP, which allows a full investigation about the overall quality of the prototype. The evaluator (i.e., author) showed two student users (i.e., one human factors student and one electrical and computer engineering student) the low-fi prototype running on the computer and asked them to provide their feedback on the low-fi prototype in terms of the flow of the processes, the overall look and feel, navigational issues, terminology used, and even colors and graphics. 2. Collecting User Feedback: The users’ feedback on the low-fi prototype was collected to apply them when developing the high-fi prototype in the next process. Table 3-17 shows a summary of the main feedback on the low-fi prototype that participants provided.

Chang S. Nam


95

Table 3-17. A Summary of the Initial User Feedback Dimension Menu structure Left menu Homepage Learning contents Search function Color scheme

User Response • Match the menu structure with the task structure (user 1 & 2) • Should be accessible from anywhere (user 1 & 2) • Needs Log-in function (user 1) • Make it simple (user 2) • More explanations (user 1 & 2) • More examples (user 1 & 2) • Not in a separate page, but in main page accessible from anywhere (user 1 & 2) • Use more colors rather than a white & black (user 1)

• Walk-through method

Methods/ Tools

Figure 3-17. Design Process Template: Design Walk-Through Process

• User’s initial feedback on the low-fi prototype

Instruction System

Outputs

Design Steps

Collecting User Feedback

• Requirements specification document • Low-fi prototype

Inputs

Walking through Prototype

• Obtain initial feedback on the low-fi prototype in terms of consistency with the target user's expectations and skill level • Gain the user’s perspective on desired interactions and information/features

Process Objectives


This walk-through process describes the key elements necessary to quickly test the low-fidelity prototype

2. Design Walk-through

Process Description

Process

96

III. Development


Phase

Chang S. Nam

Chang S. Nam


97

Process 3. High-Fidelity Prototyping The High-Fidelity Prototype process described the development of the high-fi WD2L environment prototype, in which full functionality is completed. Figure 3-18 represents the Design Process Template for the High-Fidelity Prototype process. The main objective of the process was to develop the interface and instructional system with all functionalities. 1. Reviewing Requirements Specification Document: To have an understanding of the user’s needs, application domain, and target audience, a Requirements Specification Document was reviewed. 2. Reviewing Results of Design Walk-through: The data obtained through the design walk-through were reviewed to apply them to the development of the hi-fi prototype. 3. Iterating Low-Ii Prototype:

With information identified in previous processes,

including requirements specification document, the low-fi prototype was iterated several times. Iterating the low-fi prototype involved the use of various design tools, such as HTML, PHP, and JavaScript. For example, developing the interactive feedback function mainly required using PHP and JavaScript.

• Web design tools (e.g., Dreamweaver, HTML) • Script language (e.g., Javascript)

Methods/ Tools

Figure 3-18. Design Process Template: High-Fidelity Prototype Process

• High-fidelity prototype

Instruction System

Outputs

Design Steps

Reviewing Results of Design Walk-through

Iterating Low-fi Prototype

• Requirements specification • Low-fidelity prototype

Inputs


• Develop the interface and instructional system with all functionalities • Build computer-based prototype

Process Objectives


This high-fidelity prototype process describes the key elements necessary to develop a high-fidelity prototype

3. High-fidelity prototyping

Process Description

Process

98

III. Development


Phase

Chang S. Nam

Chang S. Nam


99

Phase 4: Formative Evaluation The primary focus of the formative evaluation phase was to identify current weaknesses in the whole WD2L environment prototype so that it can be revised. This study proposed a formative evaluation framework, which provided evaluation process, instruments to be used, and evaluation criteria to systematically evaluate both the instructional system and the user interface system. The formative evaluation process was described in more detail in Chapter IV.


Chang S. Nam

100

Description of the Final WD2L Environment Prototype The high-fi prototype went through formative evaluation processes, which were detailed in Chapter IV. This section described the final WD2L environment prototype, which was iterated six times in response to users’ feedback.

First, a general structure of the final WD2L

environment prototype was overviewed. Second, it summarized navigational tools available on the WD2L environment prototype. Finally, the WD2L environment prototype was detailed, while describing main screens. Overview of the WD2L Environment Prototype The WD2L environment is a Web-based distance and distributed learning platform that offers students including distant learners a set of tools, functions, and features to support GPS learning. Anyone signed up for the “Introduction to GPS Theory and Design” course can login the WD2L environment by using a 5.0 or greater version of the Web browser such as Microsoft Internet Explorer and Netscape Communicator.

Navigation Path Section

Header Navigation Section Menu bar Section

Left-hand Navigation Section

Content Display Section

Footer Navigation Section

Figure 3-19. Basic Structure of the WD2L Environment

Chang S. Nam


101

Figure 3-19 represents the structure of the final WD2L environment prototype, including header, menu bar, navigation path, left-hand side navigation, content display, and footer navigation sections. Header Section: The course title and GPS image appear at the top of the screen. As shown in Figure 3-20, the header section also contains the “Links to VT” pull-down menu, which allows the user to access VT, Hokie Spa, Webmail, Blackboard, and libraries home pages. The search box with an advanced search function is also available, which provides three search options such as the whole site, GPS resources, GPS glossary. Finally, four links are provided, which allows the user to access the advanced search box (Advanced), visit the help page (Help), see the site structure (Site Map), and end a session (Log Out).

Figure 3-20. Header Section of the WD2L Environment

Menu Bar Section: This section contains menus arranged horizontally, as shown in Figure 3-21.

Figure 3-21. Menu Bar Section of the WD2L Environment


Chang S. Nam

102

Some of the main menu options (e.g., INFORMATION menu) are associated with another pull-down menus (e.g., Announcements, Course Information, and Staff Information) that appear when the user makes a selection with a mouse. Table 3-18 represents the menu structure of the WD2L Environment. Table 3-18. Menu Structure of WD2L Environment Prototype Main Menu HOME INFORMATION CLASSROOM

LABORATORY

ASSIGNMENTS COMMUNICATION

2nd Level Menu

Announcements Course Information Staff Information Classroom Guide Lecture Notes Document Sharing Resource Materials Lab Guide MATLAB Exercise Prelaboratory Lab Exercises Assignments Guide Assignments Quiz Send Email Discussion Board Chat Room

3rd Level Menu

Chapter 1 ~ 11 Concept Map, GPS Resources, & GPS Glossary Function Reference & MATLAB 101 Prelaboratory 1 ~ 11 Laboratory 1 ~ 11

Navigation Path Section: When the user arrives at a page, hypertext links appear in the navigation path section (Figure 3-22) to show which paths the user has taken. The navigation path also allows the user to quickly return to a previous page that led to the current page.

Figure 3-22. Navigation Path Section of the WD2L Environment Left-hand side Navigation Section: The left-hand side navigation section (Figure 3-23) provides the second (e.g., Classroom Guide and Lecture Notes) and third level menu items (e.g., Chapter 1~11) associated with the main menu (e.g., CLASSROOM menu) that led to the current page (e.g., Chapter 5 page). The menu bar and left-hand side navigation are redundant. That is,

Chang S. Nam


103

everything in the menu bar is also placed in the left-hand side navigation section. These menu options in the left-hand side navigation section moves up or down as the user scrolls up or down the page. This function can help the user quickly access another page without having to click on a menu option in the menu bar or in the footer navigation section.

Figure 3-23. Left-Hand Side Navigation Section of the WD2L Environment Content Display Section: As shown in Figure 3-24, the content display section shows specific content (e.g., Chapter 5) of the course when accessed through one of the navigational tools described in Table 3-13. The content display section is also organized into three areas: page title, tab, and content area. The page title area shows the title of the current page. In the tab area, various function icons are placed, such as Single-page (show all the content in one page), Multi-page (show content in several pages), Print (print the content), and Save icons (save the content into the computer). Footer Navigation Section: Links in the footer navigation section (see Figure 3-24) provide access to the same menu options as shown in the menu bar, including access to the Bradley Department of Electrical and Computer Engineering.

The purpose of the Footer

Navigation is to provide the user with quick, easy access to menu options at the bottom of each page.

Chang S. Nam


104

Page Title Area Tab Area

Content Area

Figure 3-24. Content Display and Footer Navigation Sections Footer Navigation Section

Chang S. Nam


105

Navigation Overview on the WD2L Environment Prototype The Final WD2L environment prototype provides several ways to move from one area or page to the next. Only the material in the content section changes when moving to a new area or page. The header, menu bar, navigation path, left-hand side navigation, and footer navigation sections are always available for quick access to those navigation features. Table 3-19 details each navigation tool available in the WD2L environment. Table 3-19. Navigation Tools and Descriptions Navigation Tool Menu Bar Navigation Path Left-Hand Side Navigation Links Footer Navigation Tabs Other Navigational Tools

Description Click on a menu item to navigate to a page within the WD2L Environment. Menu options lead to another menu items (i.e., 2nd and/or 3rd level). Click on one of the hypertext links that appear in the navigation path to access that page. The navigation path appears under the menu bar to allow users to quickly return to the previous page that led to the current page. Click on a menu to navigate to an area. The left-hand side navigation moves up and down as the user scrolls up and down a page. Click on a hypertext link to access another Web page within the WD2L Environment. Links can also open Web sites outside of the WD2L Environment. Click on a link to navigate to a page within the WD2L environment. Click on a tab in a page to access specific content within the page. Site utilities. They provide access to additional tools that can help the user find content on the WD2L environment. Tools include a search box, site map, and help.

Detailed Description of the WD2L Environment Prototype This section described the main screens of the final WD2L environment prototype in more detail. Log-in Page: Figure 3-25 shows the login screen, which also serves as the Home page. In this screen, very few options are available, because the user has not yet logged in. In other words, the user can check out what options including menu items are available, but cannot see actual pages. Without logging in, however, the WD2L environment allows the user to see Help and Site Map

Chang S. Nam


106

pages, which will help the user identify how to use the site as well as its structure, and to navigate to several VT sites, such as the library, Hokie Spa, and Webmail pages.

Figure 3-25. Log-in (Home Page) Screen


Chang S. Nam

107

The user logs in by entering his/her VT PID and password. Any field receiving a mouse click will be active (showing a flashing mouse cursor). The user can also move the active cursor from the user name field to the password field by pressing the “tab” or “return” keys. To submit the information in the fields, the user must click the “Login” button. If the PID and password entered by the user are recognized by the system, the “Announcements” Screen will appear as shown in Figure 3-28. If the user enters a PID that is not recognized by the system, a dialogue box with an “okay” button will appear reading, “The user name entered does not appear in the database.” If a valid PID is entered, but the password is not correct, a dialogue box with an “okay” button will appear which reads, “The password you entered doesn’t match your user name.” In either case, clicking the “okay” button will result in the dialogue box disappearing, and the user returning to the main logon screen. Search Box: Users have three options to search information on the WD2L environment: Whole site, GPS Resources, and GPS Glossary. Table 3-20 describes the available functions on the Search box. Table 3-20. Available Functions on the Search Box To… Search information Complete a broad search

Click… Text box, enter the keyword, then click the Go button Advanced search option

Announcements Page: Once the user has successfully logged on, he/she immediately enters the Announcements page (see Figure 3-26) in which the instructor or teaching assistant (GTA) posted timely messages for students, such as a new quiz or assignment. This page is intended to keep up to date on what is happening in the course. When the instructor or teaching assistant posts a new announcement, the announcement immediately appears under the left-hand side navigation section and stays for 24 hours. The users can see the new announcement and check out by clicking it in any page. They can click on

Chang S. Nam


108

the tabs to view the announcements for a specific time period (e.g., View today, View last 7 days, and View all). The default is the View Today tab.

Figure 3-26. Announcements Screen Course Information Page: As shown in Figure 3-27, the Course Information page contains important course information and is the best place to find the course syllabus. Users can click on the tabs to view the course information about the course overview or course assignments. The default is the Course Overview page.

Figure 3-27. Course Information Screen

Chang S. Nam


109

Staff Information Page: By clicking the Staff Information menu on the menu bar or on the left-hand side navigation section, users can view staff information such as instructor name, email address, office location, and office hours. Figure 3-28 shows the Staff Information screen.

Figure 3-28. Staff Information Screen

Classroom Guide Page: The Classroom Guide page (see Figure 3-29) offers a list of menu options and their submenu items available on the Classroom Menu, which also provides a brief description of them. Menu options and their sub-menu items are hyperlinked to their pages so that the user can go to any page when clicking on the link.

Chang S. Nam


110

Figure 3-29. Classroom Guide Screen Chapter 5 Page: The Chapter 5 page exemplifies a page format used to design learning content. The Chapter 5 page consists of three main screens: main, sub-section, and practice session screen. Table 3-21 presents functions available to users on the Chapter 5 page. Table 3-21. Available Functions on the Chapter 5 Page To… See all sub-sections in one page See sub-sections in separate pages Print Chapter 5 page Save Chapter 5 page into the computer/diskette Move to the next page Move to the previous page Move up to the top of the page See the definition (or page) of a term Listen to voice recordings of Chapter 5

Click… Single-page link/icon Multi-page link/icon Print link/icon Save link/icon Next button Previous button Top link Term links Sound icon

Chang S. Nam


111

Main Page: As shown in Figure 3-30, the main page for each Chapter consists of two main parts: chapter objectives and table of contents. The Objectives section describes main learning objectives of chapter 5. These objectives help students identify what they will be able to do upon completion of the chapter. The Table of Contents section lists all topics (called learning units) that each chapter covers. Each topic is hyperlinked to the corresponding unit so that students can go directly to the unit that they want to study. For example, Chapter 5 consists of five learning units.

Figure 3-30. An Example of the Main Screen for the Chapter 5 Page

Chang S. Nam


112

Learning Unit Page: Figure 3-31 shows an example of the learning unit page (i.e., Introduction unit). The main part of a learning unit is the “Think for a while” section, designed to allow students reflect on what they have learned in the previous chapter(s) and how their prior knowledge can be related to the current topics. In previous chapters, especially Chapter 4, for example, orbital mechanics have been examined, assuming that only two masses - the much more massive Earth and the satellite – exist and that the Earth could be described as a spherical mass of uniform density. However, at the level of precision required for GPS this analysis fails in several different areas.

With such knowledge, students will be able to master learning

materials as they study.

Figure 3-31. An Example of Learning Unit Screen Practice Session Page: At the end of the learning units, practice sessions are provided in which students can practice what they have learned. The intended purpose was to explicitly teach problem-solving strategies because students cannot use appropriate strategies merely by looking at the materials. Various types of questions were used for practice pages, such as a true and false, multiple choice, short essay, and step-by-step procedure.

Chang S. Nam


113

Figure 3-32 shows an example of true and false questions. Practice sessions also provide students with informative feedback. According to Cognitive learning theory, students use their errors as a source of information about their mental models.

Figure 3-32. An Example of the Practice Session Screen

Chang S. Nam


114

Concept Map Page: The Concept Map page (see Figure 3-33) demonstrates what sub-topics each topic contains and how each chapter of the course and sub-topics are related to each other. Basically, the concept map consists of two parts: map on the whole chapters and map on each chapter. For example, by clicking a sub-topic in a chapter, students can see how the sub-topic is related to other chapter(s) or other sub-topics in the chapter. This Concept Map page is intended to help students review what they already know again and link new material with what is currently known as a way of providing mental scaffolding for new material.

Figure 3-33. Concept Map Screen

Chang S. Nam


115

GPS Resources Page: The GPS Resources page (see Figure 3-34) offers links to material outside the course that will be beneficial to the learning experience.

Figure 3-34. GPS Resources Screen

Chang S. Nam


116

GPS Glossary Page: A glossary of GPS terms are provided in the GPS Glossary page, which can be found by using the list of links or the search box. For example, selecting the first letter of the word from the list takes the user to the appropriate section of the glossary.

Figure 3-35. GPS Glossary Screen

Chang S. Nam


117

Prelaboratory Page: To help work ahead on the laboratory, the Prelaboratory page provides students with an opportunity to modify MATLAB codes required in each laboratory. As shown in Figure 3-36, the format of the Prelaboratory 5 page is similar to the code shown in the MATLAB software. For example, the color of variables is black, and explanations were made to be green, just like the colors in the MATLAB program. Second, variables, functions, or even GPS glossary were hyperlinked.

Figure 3-36. Prelaboratory Screen

Chang S. Nam


Table 3-22 shows functions available in the Prelaboratory page. Table 3-22. Available Functions on the Prelaboratory Page To… View explanations See correct answers Upload the file into the system Move up to the top of the page See the definition (or page) of a term

Click… Check out button Show codes button Submit button Top link Term links

118

Chang S. Nam


119

Assignments Page: The Assignments page (see Figure 3-37) lists due dates and descriptions for class assignments posted by the instructor. When clicking the title of an assignment, the user is always asked whether to open the file or save it to the computer by a dialogue box with “Open,” “Save,” “Cancel,” and “More Info” buttons.

Figure 3-37. Assignments Screen When submitting an assignment, the user is required to click the “Submit” button that takes him/her to the Assignments-Submit page in which the “Name” and “E-mail” sections are automatically filled out. Table 3-23 details the entry fields on the Assignments-Submit page.

Chang S. Nam


120

Table 3-23. Entry Fields on the Assignments-Submit Page Select Students Name: Email: Subject: Message: Add Attachments Attachments

Name Email address Enter the subject of the mail (e.g., assignment title) Type the message Click here to add attachments

Table 3-24 presents the functions available to users on the Assignments-Submit page. Table 3-24. Available Functions on the Assignments-Submit Page To… Open a file to be attached Deliver the completed form Erase information that is currently in the form Go back to the Assignments page

Click… Browse Submit Reset List

Quiz Page: In the Quiz page, users can take a quiz developed by the instructor in which their answers can be submitted for grading and results can be recorded in the Gradebook page. The quiz page also provides a place in which users can review the quiz they took, while receiving informative feedback. Taking a quiz: Click the title of a quiz that the instructor set up for a specified date and time. The quiz will be timed, which is described in the Description section. An example of a quiz (i.e., Quiz 1) is shown in Figure 3-38.

Reviewing the quiz: After finishing a quiz, students will be asked to review the quiz while receiving informative feedback. Feedback will show correct answers and comments from the instructor (Figure 3-40).

Chang S. Nam


Figure 3-38. Quiz Review Main Screen

121

Chang S. Nam


Figure 3-39. Quiz Review: Question Screen

122

Chang S. Nam


Figure 3-40. Quiz Review: Feedback Screen

123

Chang S. Nam


124

Send Email Page: The Send Email page allows users to send email to fellow classmates, instructor, teaching assistant or other group members.

Figure 3-41. Send Email Screen The following functions are available from the Send Email page. Table 3-25 details entry fields on the Send Email page. •

Member All: Sends email to all users taking the course.

•

Group: Send email to a single group or select groups.

•

Individual: Sends email to a single user or select users.


Chang S. Nam

125

Table 3-25. Entry Fields on the Send Email Page Select Students Subject: To: Recipient: Message: Special Characters: Add Attachments Attach ments

Enter the subject of the email Select the user type who will receive the email message Select a single group or individual email address Enter the email message Click a special character, if necessary, when typing the message Click here to add attachments

Discussion Board: The Discussion Board page (see Figure 3-42) is another communication tool. It creates an online place to have a lively debate of relevant issues on course material. That is, a question can be freely asked or an opinion can be stated in the Discussion Board. Other students and the instructor can now respond by adding a new thread. This feature was designed for asynchronous use, meaning students do not have to be available at the same time to have a conversation. Table 3-26 details the functions available on the Discussion Board page. Table 3-26. Available Functions on the Discussion Board Page To… Write a new message Delete the message Modify the message Reply to the message See all threads Go to the group discussion board Read messages selected together See a message Find the message See the content of the message ahead

Click… Write icon Delete icon Modify icon Reply icon List icon Drop-down list and select one of the group options Read title Thread link Drop-down list, select one of the options, and type the word Place the mouse on the thread

Chang S. Nam


126

Figure 3-42. Discussion Board: Main Screen

Creating New Message: The Discussion Board – Write page (see Figure 3-43) is used to post a new message. A message may also be added as a response to a post by clicking the Reply icon. Table 3-26 details the entry fields on this page.


Chang S. Nam

Figure 3-43. Discussion Board – Write Screen

Table 3-27. Entry Fields on the Discussion Board – Write page Message Information Name: Email: Password: Subject: Message:

Special Characters: Option Attachment

Enter the poster’s name Enter the poster’s email address Type the poster’s password to delete or edit the message Enter the subject of the thread Enter a message. Select a text type for the message from the following options: • Text: Displays text as written • HTML: Displays text as coded using HTML tags Click a special character, if necessary, when typing the message Enter the file path or click Browse to locate the file

127

Chang S. Nam


128

Summary The previous sections described the development process of the WD2L environment prototype by following the proposed IDP, which consisted of the needs analysis, conceptual design, development, and formative evaluation phase. First, the Needs Analysis consisting of the requirements specification, features and components identification, design guidelines review, and usability goals setting process, was conducted to identify all possible factors that influence the development of the WD2L environments.

These factors included immediate needs,

requirements, and new design concepts for the development of the instruction and interface system. The Needs Analysis phase also determined design goals that guided the entire design process. Second, the Conceptual Design phase was conducted, in which ideas or concepts about what a WD2L environment is, what it can do, and how it is intended to be used were explicitly developed. This phase consisted of four design processes for translating user requirements into a conceptual user interface and instructional design: the design scenarios development, information design, structure design, and page design. Third, the hi-fi prototype of the WD2L environment was built in the Development phase, based on the initial results of the user evaluation on the low-fi prototype.

In the low-fi

prototyping process, the paper-based prototype of the WD2L environment was developed with information identified in the previous processes. The low-fi prototype was also evaluated to assess whether the proposed design of the WD2L environment is consistent with target users’ expectations and skill level early in the design process. Finally, the final WD2L environment prototype was described.

The prototype was

iterated six times in response to users’ feedback obtained through the proposed formative evaluation framework proposed in the study. Chapter IV described the formative evaluation framework and design changes made according to evaluation results in more detail.

129

CHAPTER IV EVALUATION METHODS AND RESULTS The principal purpose of a formative evaluation is to collect continuous feedback from users to improve the WD2L environment prototype (Dick & Carey, 1996). As described in Chapter II, several models and frameworks have been proposed for evaluating distance learning systems and user interface systems. Unfortunately, few attempts have been made to develop a formative evaluation framework for assessing the WD2L environment prototype whose foci are both the instructional system and user interface system.

This study proposed a formative

evaluation framework for assessing the WD2L environment prototype, which provided evaluation processes, instruments, and evaluation criteria to systematically evaluate and revise the instructional system and user interface system. The proposed formative evaluation was also used as a portion of the design process evaluation, used to develop the WD2L environment prototype. Formative evaluation approaches have proven “valuable for identifying ways to improve theories and models” (Reigeluth, 1999, p. 637).

They have been widely used to

improve existing instructional-design theories and models (e.g., Shon, 1996), instructional systems development (ISD) models (e.g., Farmer, 1989, in Reigeluth, 1999), and educational systems design (ESD) models (e.g., Khan, 1994). This chapter summarized formative evaluation processes and design changes made according to results of the evaluation with users. First, the proposed formative evaluation framework was briefly described, consisting of the Expert Review (1st), One-to-One Evaluation, Small Group Evaluation, and Expert Review (2nd) processes. Second, each formative evaluation process was summarized, along with results obtained from the evaluation and design changes made to the WD2L environment prototype. Third, a Field Experiment was described, which was included as improvement of the Integrated Design Process (IDP). This evaluation process was also used as a portion of the evaluation of the design process used to develop the WD2L environment prototype.

Finally, a general discussion on the evaluation of the WD2L

environment prototype was included.

Chang S. Nam

Chapter IV. Evaluation Methods and Results

130

Description of the Formative Evaluation Framework As a formative evaluation process, this study borrowed and modified the first three steps of Dick & Carey’s (1996) evaluation approach, Expert Review, One-to-One Evaluation, and Small Group Evaluation, because the fourth step, Field Trial, is more of a summative evaluation step. Instead, this study suggested conducting the Expert Review (2nd) process in the fourth step again, in which experts finally reviewed the WD2L environment prototype. Figure 4-1 illustrates the formative evaluation framework proposed in the study. The Expert Review (1st) Process: The purpose of the Expert Review process was to perform a dry run of the WD2L environment prototype before testing with representative users. Subject matter experts (SMEs) who exhibited the highest level of expertise in three main areas, the instructional design, user interface design, and GPS content, discovered overlooked areas or problems.

The WD2L environment prototype was revised according to recommendations

provided by the SMEs. Design principles and guidelines, as well as the designer’s knowledge and skills, were also used to change the design of the WD2L environment prototype. The One-to-One Evaluation Process: In this process, representative users identified and removed more prominent errors in the WD2L environment prototype.

The One-to-One

Evaluation process focused on individual learning activities and single-user interfaces.

As

evaluation criteria for determining the overall quality of the instructional system, clarity of instruction and impact of instruction defined by Dick & Carey (1996) were employed. The overall quality of the user interface system was determined by ISO 9241-11 (1998) usability criteria, effectiveness, efficiency, and satisfaction. The Small Group Evaluation Process: This process was performed with a group of participants representative of the target population. The focus of the Small Group Evaluation Process was on evaluation of group learning activities (e.g., group discussion) and multi-user interfaces (e.g., discussion board).

The Small Group Evaluation process used the same

evaluation criteria as used in the One-to-One Evaluation process.

Instruction System

• Group learning activities • Evaluation data • User comments • Principles & guidelines • Designer’s knowledge & skills

• Evaluation data • User comments • Principles & guidelines • Designer’s knowledge & skills

• Multi-user interface

-

• Effectiveness • Efficiency • Satisfaction

• Individual learning activities

• Effectiveness • Efficiency • Satisfaction • Clarity of Instruction • Impact of Instruction • Single-user interface

Small Group Evaluation Remove prominent errors • Benchmark tasks • Usability specifications

131

Figure 4-1. A Formative Evaluation Framework Proposed for Evaluating the WD2L Environment

• Learning content • Instructional interventions • Experts’ recommendations • Principles & guidelines • Designer’s knowledge & skills

• Overall (Single- & multi-user interface)

Instruction System User Interface System

Design Change Strategy

Main Focus

Evaluation Criteria

Expert’s design change recommendations


Expert Review (1st) A dry run • Review forms • Expert’s expertise

Process Purpose Instrument

One-to-One Evaluation Remove prominent errors • Benchmark tasks • Usability specifications


Chang S. Nam

• Learning content • Instructional interventions • Expert’s recommendations • Principles & guidelines • Designer’s knowledge & skills

• Overall (Single- & multi-user interface)

Expert’s design change recommendations

Expert Review (2nd) Re-evaluate the whole prototype • Review forms • Expert’s expertise

Chang S. Nam


132

The Expert Review (2nd) Process: In this process, the WD2L environment prototype was finally reviewed by SMEs to identify overlooked areas and problems again. The main reason that this study employed Dick & Carey’s (1996) evaluation approach was that this approach allows different types of evaluators to assess the WD2L environment prototype (i.e., experts, individual users, and group of users). Another reason, more importantly, was that this approach makes it possible to evaluate various aspects of the WD2L environment prototype. As shown in Figure 4-1, for example, the One-to-One Evaluation process can be conducted mainly to evaluate individual learning activities in the instructional system and singleuser interfaces in the user interface system. On the other hand, the Small Group Evaluation process can be performed to evaluate group learning activities (e.g., group discussion) and multiuser interfaces in each system.

Chang S. Nam


133

Expert Review (1st) Process The Expert Review (1st) process was the first evaluation process in the proposed formative evaluation framework, in which subject matter experts (SMEs) reviewed the first version of the prototype and suggested design recommendations to improve it. In general, the most common way of testing usability of a system prototype is to select several major components and test them with a group of representative users because of limited resources, such as money, development time, and human resources. Following Dick & Carey’s study (1996), however, this study asserts that in order for a system prototype to be evaluated with users, one first must discover overlooked areas or problems of the prototype and revise them. The SMEs who exhibit a high level of expertise in three main areas, instructional design, user interface design, and GPS content, can fill that requirement. These three SMEs reviewed the WD2L environment prototype and suggested design recommendations to improve it two times: before (Expert Review (1st) process) and after (Expert Review (2nd) process) usability testing with representative users. st

This section

summarized the Expert Review (1 ) process, in terms of evaluation methods, obtained results, and design changes made to the first version of the WD2L environment prototype.

Chang S. Nam


134

Method Participants

A 34-year-old Ph.D. candidate in the area of instructional design served as an instructional design expert. The participant worked as an instructional designer for three years in the design, development, implementation, evaluation, and management of online learning environment. While Associate Director of Instructional Design at North Carolina Agricultural and Technical State University, he was instrumental in the establishment of the Office of Instructional Design (OID). His responsibilities included providing instructional design and development expertise and support to faculty in the creation of e-Learning courses.

The

participant also worked with North Carolina Agricultural and Technical State University and Winston-Salem State University in the creation of their virtual campus interactive CD-ROMs. In both projects, the purpose was to increase the enrollment of incoming students and to inform the community about the university’s availability of online instruction. As a GPS content expert, a 27-year-old Master candidate participated in the expert review process. He worked as a research assistant for two years in the GPS laboratory. He was also a teaching assistant for the GPS course during Fall of 2001 and Fall of 2002. As an interface design expert, a 32-year-old human factors Ph.D. student in the area of user interface design participated to provide design suggestions to improve the user interface system. He worked as a human factors engineer for five years in the area of the interface design in vehicle systems. He also completed all the major human factors courses related to user interface design, such as usability engineering and system design courses.

Chang S. Nam


135

Equipment/Apparatus The three SMEs reviewed the first version of the WD2L environment prototype (i.e., high-fidelity prototype) running on a Pentium computer, equipped with a standard keyboard, mouse, and 17” super VGA monitor. Internet Explorer (IE) 6.0 running in Windows 2000 was used as a Web browser. To review and suggest their recommendations to improve the WD2L environment prototype, the SMEs were asked primarily to utilize their expertise in their specialties. In addition, to help the SMEs review important aspects of the WD2L environment prototype, this study developed three types of expert review forms: User Interface Review Form, Instructional Design Review Form, and Content Review Form. User Interface Review Form:

This form provided three sections to help the user

interface expert assess important aspects of the interface system: demographics, evaluation on components of the user interface system, and overall quality of the user interface system. The User Interface Review form can be found in Appendix 6. The “Demographics” section (Part I) collected basic information about the user interface expert. The second section, “Evaluation on Components of the User Interface System” (Part II), allowed the expert to provide design comments/recommendations on 15 main components of the user interface system. For the Overall Quality of the User Interface System section (Part III), this study borrowed and used Reeves & Harmon’s (1993) “User Interface Rating Tool for Interactive Multimedia,” which provided 10 dimensions to evaluate the user interface system: ease of use; navigation; cognitive load; mapping; screen design; knowledge space compatibility; information presentation; media integration; aesthetics; overall functionality. Instructional Design Review Form:

To assist the instructional design expert in

reviewing important aspects of the instructional design, the “Instructional Design Review Form” was developed as shown in Appendix 7. This form consisted of three parts: demographics, review of the instructional design, and overall quality of the instructional design.

Chang S. Nam


136

The “Demographics” section (Part I) collected basic information about the instructional design expert. The second section, “Review of the Instructional Design” (Part II), provided Gagne’s nine events of instruction with which the instructional design expert assessed important aspects of the instructional design. Since Gagne’s nine events of instruction were the foundation for designing the instructional system, in particular, instructional strategy, the value of the instructional design was determined by evaluating how well components of the instructional strategy were implemented.

A brief description on how each instructional event was

implemented in the WD2L environment prototype was also provided. The “Overall Quality of the Instructional Design” (Part III) section provided three statements, in which the expert is required to indicate how much to agree or disagree with them. These statements were designed to cover three important aspects of the instructional design, such as the design for target audience (e.g., Instructional design is appropriately developed for the intended audience.), match to learning objectives (e.g., Instructional design is closely related to learning objectives stated.), and clear to be self-instructional (e.g., Instructional design is clear enough to be self-instructional.). Content Review Form: The Content Review Form was designed for the GPS expert to evaluate four main learning units as shown in Appendix 8. These learning units included the content unit (Chapter 5 - Corrections to Keplerian Orbits: Precise Positioning), practice unit (4 practice sessions), quiz review unit (Quiz 1 review), and laboratory exercise unit (Prelaboratory 5). This form consisted of two parts: demographics, and evaluation on learning units. The “Demographics” section (Part I) collected basic information about the GPS content expert. The part II (Evaluation on Learning Units) provided several statements to review important aspects of learning units, such as a difficulty level for target audience, content accuracy, and match to learning objectives.

Chang S. Nam


137

Procedures

Three SMEs were given written instructions for the task by asking them to review and provide design comments or recommendations that would help revise the WD2L environment prototype, in terms of the GPS learning content, instructional design, and user interface design. Appendix 9 provides the task instruction given to the SMEs. To help the SMEs assess the first version of the WD2L environment prototype, two types of supporting materials were provided: a user profile and review forms.

The user profile

specified in the Requirement Specification Document was given to help the SMEs have a better understanding of the target user group.

The expert review forms developed in the study were

also provided to each expert. It took about two hours for each expert to complete the evaluation of the WD2L environment prototype. At the end of the review, the SMEs were debriefed.

Chang S. Nam


138

Results User Interface Expert Review Evaluation on Components of User Interface System: The interface expert reviewed components of the user interface system and provided design recommendations for the modification of the user interface system.

Table 4-1 shows examples of comments and

recommendations that the user interface expert provided on the user interface system. Problems and design recommendations to improve them (provided by the user interface expert) were addressed more fully in Appendix 6. Table 4-1. Examples of Expert Comments/Recommendations on User Interface System Components Homepage

Comments/Recommendations • Use different color for main menus • Make the log-in box clear

Components Concept Map

Search box

• Links to VT and search options are good features • Make the text box of the search engine larger • Put the Prev and Next button at the top of the page • Single and multi-page feature is good

GPS Glossary

Comments/Recommendations • Highlight box for when mouse is on the box to show selection • Standardize the capital letters of your titles • Make GPS terms link each other

Discussion Board

• Put the caption for the Read function

Chapter 5

Overall Quality of User Interface System: Table 4-2 shows the overall quality of the user interface system evaluated by the interface expert. The Navigation (6.0), Mapping (6.0), Knowledge Space Compatibility (6.0) dimensions were rated highly, while the screen design (3.0) and aesthetics (3.0) dimensions received low points. Table 4-2. A Rating Summary of User Interface System Dimension Ease of use Navigation Cognitive load Mapping Screen design

Rating (6.0) 5.0 6.0 5.0 6.0 3.0

Dimension Knowledge space Information presentation Media integration Aesthetics Overall functionality

Rating (6.0) 6.0 5.0 5.0 3.0 5.0

Chang S. Nam


139

Instructional Design Expert Review Instructional Design: The instructional design expert evaluated how well components of the instructional strategy was implemented and provided design recommendations for the modification of the instructional system.

Table 4-3 shows examples of design

comments/recommendations that the expert provided on the instructional design. Problems and recommendations made by the expert were fully described in Appendix 7. Table 4-3. Examples of Expert Comments on Instructional Design Instructional Event 3. Stimulating recall of prior learning

Comments/Recommendations • Make your feedback screens give either a positive or negative response • Use cues, shaping and practice to ensure a strong stimulus-response association: The accuracy of the response is based on the appropriate cues, which are accompanied by proper instruction. • Determine which cues can elicit the desired response • Arrange a practice situation where target stimuli and prompts are paired within a natural performance setting • Allow correct responses within the teaching environment and provide reinforcement and feedback.

Overall Quality of the Instructional Design: The overall quality of the instructional design was good. The instructional design provided high points to the three dimensions (i.e., Design for Target Audience (6.0), Match to Learning Objectives (4.0), and Clear to be SelfInstructional (5.0)).

Chang S. Nam


140

Content Expert Review The GPS content expert reviewed learning units and provided design recommendations for the modification. Table 4-4 shows examples of comments and recommendations that the GPS content expert provided. Problems and design recommendations to improve them were addressed more fully in Appendix 8. Table 4-4. Examples of Expert Comments/Recommendations on Learning Units Unit Content Unit

Comments/Recommendations • You mention in the last sentence of the paragraph that the continental tectonic plates have densities differing from that of the oceanic crust. I think it could be better to explain it because a sphere is just a shape and it is not related to the mass densities. • There’s no explanation for J2 effects. It could be better to put the definition somewhere: J2 effects describing the ellipsoidal shape of the Earth consist of harmonic and secular effects on the GPS orbits.

Practice Unit

• Explain effects causing perturbations on GPS satellite orbits in Section 2 of Chapter 5 (i.e., NonKeplerian Effects) in more detail. • Equations and notations for the ephemeredes are different from the textbook. It’s not recommended since they will be used throughout the textbook.

Table 4-5 represents the overall quality of learning units evaluated by the GPS content expert. Learning units received relatively high scores, ranging from 4.8 (Practice Unit – practice 4) to 5.4 (Quiz Review Unit – Quiz 1 Review). Ratings on learning units were fully provided in Appendix 8. Table 4-5. A Summary of Rating on Learning Units Learning Unit Content Unit - Chapter 5 Practice Unit – Practice 1 Practice Unit – Practice 2 Practice Unit – Practice 3

Rating (6.0) 5.3 5.3 5.0 5.0

Learning Unit Practice Unit – Practice 4 Practice Unit – Practice 5 Quiz Review Unit – Quiz 1 Review Laboratory Exercise Unit - Prelaboratory 5

Rating (6.0) 4.8 5.0 5.4 5.0

Chang S. Nam


141

Design Changes and Discussion This section describes primary design changes made to the first version of the prototype. These changes were made in response to design change recommendations suggested by the three SMEs, but some of the recommendations were not implemented if they were not relevant as the researcher judged. In the words of Dick & Carey (1996): The designer is not obligated to use the suggestions of these specialists. There may be some recommendations that the designer may want to consider after data from learners have been collected and summarized. At least the designer is sensitized to potential problems before learners become involved in the formative evaluation process (pp. 285-286).

To describe design changes, this section used the term “page” to refer to a system component (e.g., Assignments page) and the “screen” term to refer to sub-pages consisting of that system component (e.g., Assignments page consists of the Main screen and Submit screen). This convention is applied throughout this study. Home page:

Major changes made to the Home page involved the layout of page

components. Figure 4-2 represents main changes made to the Home page. Previous Version (1.0)

Figure 4-2. Design Changes to the Home page

New Version (2.0)

Chang S. Nam


142

First, the Login box located on the left-hand side navigation section of the page was redesigned by changing the title (i.e., LOG IN → Login Here) and colors. As the user interface expert suggested, students will be more comfortable with the Login box similar to the one that they are currently using in other systems (e.g., Blackboard). Second, to make the menu item in the menu bar more distinct, the color scheme used for the background was also changed (i.e., red → blue). Finally, the Home page included a brief description on how to use the “GPS Theory and Design” site. The instructional design expert suggested providing an additional attentionguiding device to what is coming up in the lesson. By following it, this change was made to give users a clear idea of where to find the structure of the site, including the structure of the entire course. GPS Glossary Page: Figure 4-3 shows the main change made to the GPS Glossary page, in which a GPS term and other terms used to describe it linked each other. For example, clicking the term “Doppler Shift” that was used to describe the term “Almanac” displays the definition of the “Doppler Shift.” Previous Version (1.0)

Figure 4-3. Design Change to the GPS Glossary Page

New Version (2.0)

Chang S. Nam


143

Send Email Page: Figure 4-4 represents the Send Email page before and after a design change was made. The change was to eliminate the Configuration field, which was designed to provide the user with more options for editing messages, such as the font color and size. However, the user interface expert suggested not including the Configuration field, because users tend not to use such options to edit their email messages. Previous Version (1.0)

New Version (2.0)

Figure 4-4. Design Changes to the Send Email Page Chapter 5 Page: Several design changes were made to the Chapter 5 page, based on design change recommendations provided by the three SMEs. The first change was to redesign five graphic figures used to explain the main concept of Chapter 5 (i.e., corrections to Keplerian orbits). Figure 4-5 shows an example of graphic images before and after the change was made. Previous Version (1.0)

Figure 4-5. An Example of Graphic Design Changes

New Version (2.0)

Chang S. Nam


144

Graphic images were redesigned to improve visual appearance (aesthetics). For example, the graphic image shown in Figure 4-5 was redesigned to increase perceivable quality of the image, in which careful consideration was given to colors (e.g., add pink color on the plane) and font choices (e.g., use Arial font). As indicated in the interface expert’s evaluation on the overall quality of the user interface system, screen design and aesthetics dimensions received lower points, compared to other dimensions. Graphic images were also redesigned to effectively communicate what needs to be communicated (communications). For example, to illustrate an apparent movement of an inertially fixed satellite at R between the transmission time (tT) and the reception time (tR) in an inertial reference frame, which is the main purpose of the graphic image shown in Figure 4-5, more attention was paid to information design such as the inclusion of more information (e.g., the angle of rotation, δθj) and correct use of special characters (e.g., change from ω to ωE). Improvement of visual appearance and good information design help learners have better understand the subject matter and visualize their understanding (Tufte, 1990). Another change made was to further explain effects causing perturbations on GPS satellite orbits in Section 2 of Chapter 5 (i.e., Non-Keplerian Effects), which was not included in the first version of the prototype (Figure 4-6). New Version (2.0)

Figure 4-6. Design Change to the Chapter 5 Page

Chang S. Nam


145

Discussion Board Page: Figure 4-7 represents the Discussion Board page before and after a design change was made. The change was to add text captioning for the Read function: “The Read displays all messages that you have checked in a single page.” Following the interface design expert’s recommendation, a text alternative was provided to improve Web accessibility and give users information about how to use the Read function. Previous Version (1.0)

New Version (2.0)

Figure 4-7. Design Change to the Discussion Board Other Design Changes: The Practice page was redesigned to put a list of links at the bottom of the page, which was placed only at the top of the page. Another change was to put the links that lead to the previous section and the next section of Chapter 5. Feedback screens were also changed in a way that they can provide users with more clear, informative feedback for each quiz question. More specifically, a graphic-based feedback was included in the Feedback screen that only had text-based feedback. Design Recommendations That were not Addressed: A design recommendation such as the structure of behavioral objectives was held off for this prototype even though the instructional designer suggested it. Instead, the structure of behavioral objectives stated in the Chapter 5 page was decided to change by following four basic dimensions of stating behavioral objectives: Audience, Behavior, Condition, and Degree (Saettler, 1990). However, the “degree” dimension was not included at the current stage of design changes because this study will not evaluate the user’s behavioral performance on learning units. Because of that, this study suggested that stating behavioral objectives at the top of Chapter 5 page using Audience (i.e., the student), Behavior (i.e., to identify, know, perform, and explain), and Condition (e.g., after having completed Chapter 5) dimensions will be able to inform students of what they will accomplish.

Chang S. Nam


146

One-to-One Evaluation Process In the previous evaluation process, the Expert Review (1st) process, the WD2L environment prototype (version 1.0) was revised (version 2.0) in response to design change recommendations suggested by the three SMEs. The main purpose of the Expert Review (1st) process was to address overlooked areas and problems identified in both the interface and instructional system. In the One-to-One Evaluation process, two evaluation sessions (Evaluation 1 and 2) were conducted with representative users to identify and remove more prominent errors in the second version of the WD2L environment prototype. While the Expert Review (1st) process heavily relied on the SMEs’ expertise in their specialty, the One-to-One Evaluation process assessed the second version of the WD2L environment prototype using evaluation criteria (e.g., effectiveness, efficiency, and satisfaction for the user interface system). In particular, this process focused on individual learning activities and single-user interfaces used to support them.

Evaluation 1 Method

Participants: For the first one-to-one evaluation session, five participants were recruited from a pool of the students who took the GPS course in Fall of 2002. The participants were given monetary compensation for their participation. There were 5 males (M = 22.4 years, SD = 1.14).

All participants classified their computer skill level as somewhere between an

intermediate and an experienced level. Experimental Materials and Benchmark Tasks: The participants evaluated the WD2L environment prototype, using a Pentium computer, equipped with a standard keyboard, mouse, and 17” super VGA monitor. Internet Explorer (IE) 6.0 running in Windows XP was used as a Web browser.

Chang S. Nam


147

To evaluate main functions of the interface system and the instructional system, this study developed eight “benchmark” tasks representing users’ most common tasks on the WD2L environment. For the interface system, for example, this study developed four benchmark tasks, which were searching information, uploading assignments, finding GPS resources, and sending email. Another four different benchmark tasks were developed for the instructional system, which were studying Chapter 5, performing practice sessions, reviewing the quiz, and performing prelaboratory activities. Table 4-6 shows these eight benchmark tasks developed for both systems and their main testing objective.

These tasks were addressed more fully in

Appendix 11. Table 4-6. Benchmark Tasks and Main Testing Objective System tested User Interface System

Task Benchmark #1 Benchmark #2 Benchmark #3 Benchmark #6

Instruction System

Benchmark #4 Benchmark #5 Benchmark #7 Benchmark #8

Characteristic Searching Information Uploading Assignments Finding GPS resources Sending Email Studying Chapter 5 Performing Practice Sessions Reviewing Quiz Performing Prelaboratory 5

Testing Objective To verify that searching information on the “GPS Theory & Design” website is clear To assess if the mechanism of uploading assignments is clear To test if it is clear that GPS links and related sites would be easy to access and find To evaluate that the way to communicate via an email is clear and usable To verify that the learning content is clear to learn To test if practice sessions are easy to perform, while providing informative feedback To assess if the system supports the user’s activities of reviewing the quiz To verify that the mechanism for completing the existing MATLAB templates is usable and clear

Evaluation Criteria: As evaluation criteria for determining the overall quality of the instructional system, this study used both clarity and impact of instruction, as defined by Dick & Carey (1996). The overall quality of the user interface system was determined in terms of the effectiveness, efficiency, and user satisfaction.

Table 4-7 shows a list of metrics and

corresponding performance and subjective measurements. To measure user satisfaction with user interfaces, this study employed the Questionnaire for User Interface Satisfaction questions (QUISTM 7.0), answered with 0 to 9 ratings (see Appendix 14 for sample questions). The QUISTM developed by a multi-disciplinary team of

Chang S. Nam


148

researchers in the Human/Computer Interaction Lab (HCIL) at the University of Maryland is the most extensive and most thoroughly validated questionnaire for determining subjective interface usability (Slaughter, Norman, & Shneiderman, 1995). The concept of satisfaction consisted of five categories. These were initial satisfaction, screen, terminology and system information, learning, and system capabilities. Table 4-7. Usability Metrics and Measurements System

Metric Effectiveness

Measurement Percent of tasks completed Number of features used Time to complete a task

Efficiency

Numbers of errors Frequency of the Help use

User Interface System Satisfaction

Instructional system

Clarity Impact

Initial satisfaction Satisfaction with the screen Satisfaction with terminology and system information Satisfaction with system capabilities Satisfaction with multimedia Rating scale for clarity of instruction Rating scale for impact of instruction

Definition Ratio of completed tasks to total tasks Number of action sequences used to complete a task Time from first button press to pressing a terminating or confirmation button (e.g., O.K) Numbers of errors made by users Number of times a user used the Help to complete a task Rating scales for initial satisfaction Rating scales for satisfaction with the screen Rating scales for satisfaction with terminology and system information Rating scales for satisfaction with system capabilities Rating scales for satisfaction with multimedia Degree of clarity of instruction expressed by users Degree of impact of instruction expressed by users

Chang S. Nam


Usability Specifications Table:

149

To obtain quantitative results (i.e., usability

measurements) from participants who perform benchmark tasks, this study employed the usability specifications table (Hix & Hartson, 1997).

Table 4-8 shows the usability

specifications table used in the study and target levels. Table 4-8. Usability Specifications and Target Levels Usability Attribute

Measuring Instrument Benchmark Task #1 (Searching Information)

Initial Performance

Benchmark Task #2 (Uploading Assignments)

Benchmark Task #3 (Finding GPS Resources)

Benchmark Task #6 (Sending Email)

Clarity & Impact of Instruction

Satisfaction

Benchmark Task #4 (Studying Ch. 5) Benchmark Task #5 (Performing Practice) Benchmark Task #7 (Reviewing Quiz) Benchmark Task #8 (Performing Prelaboratory)

QUIS 7.0

Data Collection Sheet

Value to be Measured Number of features Time on task Number of errors Frequency of the Help use Number of features Time on task Number of errors Frequency of the Help use Number of features Time on task Number of errors Frequency of the Help use Number of features Time on task Number of errors Frequency of the Help use Clarity of instruction Impact of instruction Clarity of instruction Impact of instruction Clarity of instruction Impact of instruction Clarity of instruction Impact of instruction Initial satisfaction Satisfaction with the screen Satisfaction with terminology and system information Satisfaction with system capabilities Satisfaction with multimedia Number of negative remarks

Target Level 4 15 0 ≤1 8 40 0 ≤1 4 30 0 ≤1 7 50 0 ≤1 5.10 5.10 5.10 5.10 5.10 5.10 5.10 5.10 8.10 8.10 8.10

Observed Results

8.10 8.10 Negative ≤ 5

In Table 4-8, the first column, the “Usability Attribute” refers to what is to be measured. For the user interface system, the “initial performance” (i.e., the first time the user performed the task) will be reported. On the other hand, the clarity and impact of instruction will be identified for the instructional system. In addition, the satisfaction attribute will be identified for both

Chang S. Nam


150

systems. The “Measuring Instrument” column refers to the specific benchmark task through which usability attributes are measured. Four benchmark tasks (i.e., benchmark task #1, #2, #3, & #4) were designed for the initial performance attribute. On the other hand, the clarity and impact of instruction will be reported through benchmark tasks #4, #5, #7, and #8.

The

satisfaction attribute will be identified through the QUISTM questionnaire. Column three represents the “Value to be Measured.” For the user interface system, four usability measurements will be used: number of features, time on task, number of errors, and frequency of help use.

The fourth column indicates the “Target Level” representing the

performance goal. In the case of the number of features measurement and time on task, target levels were derived by measuring the fastest steps to complete a benchmark task and times to finish it by the expert user (i.e., the researcher of this study). For example, the fastest steps to complete the benchmark task #3 (i.e., Finding GPS resources) are 4 steps (i.e., choose the GPS Resource menu → select the Index → scroll down the page → click GPS resource link). It took the expert user about 30 seconds to finish the task. No mistakes should be allowed. Users will be allowed to consult the Help page less than one time. Target levels of clarity and impact of instruction measurements were set as 85% from a perfect score (i.e., 5.1 out of 6.0). Ninety percent of a perfect score in the QUISTM (i.e., 8.1 out of 9.0) was determined as target levels of satisfaction measurements. On the other hand, target levels of number of positive/negative remarks were decided a little more arbitrarily, but were intended to be rigorous enough to catch major usability problems, in which the number of negative remarks should be less than and equal to 5. Column five represents results obtained from the participants. This study did not use the “current level” (generally the level at which users can use a similar program already available), which is generally used in the usability specifications table. It is because there were no similar WD2L environments available for comparison. Procedure: After the informed consent (Appendix 10) was acquired, participants were given written instructions for the task (Appendix 9) and asked to review the Site Map page of the WD2L environment to familiarize with the prototype. Then, the participants performed eight

Chang S. Nam


151

benchmark tasks (Appendix 11) representing users’ most common tasks on the WD2L environment. These tasks were presented to the participants in a random order. Before doing that, the participants were asked to think aloud throughout the whole session and talk about what they are doing, why they are doing it, and what they expect to happen when they perform an action. That is, they were asked to indicate both positive (good) and negative (bad) aspects of how they perform tasks. These observations were recorded on a “Data Collection Sheet” (see Appendix 12) and mapped to corresponding tasks for further analysis. After benchmark tasks #4, #5, #7, and #8, evaluation of instruction questionnaires (see Appendix 13 for sample questions) were administered to identify participants’ evaluation on the clarity and impact of instruction, respectively.

At the end of the evaluation, participants

completed the Questionnaire for User Interface Satisfaction (QUISTM). Sample questions can be found in Appendix 14.

Chang S. Nam


152

Results To investigate the overall quality of the WD2L environment prototype (version 2.0) from users’ perspective, several measures were employed.

Table 4-9 shows these usability

specifications measured across 5 different participants. Table 4-9. A Summary of Usability Specifications: Evaluation 1 Usability Attribute

Measuring Instrument

Value to be Measured Number of features Time on task Benchmark Task #1 (Searching Information) Number of errors Frequency of the Help use Number of features Benchmark Task #2 Time on task (Uploading Assignments) Number of errors Initial Frequency of the Help use Performance Number of features Benchmark Task #3 Time on task (Finding GPS Resources) Number of errors Frequency of the Help use Number of features Benchmark Task #6 Time on task (Sending Email) Number of errors Frequency of the Help use Benchmark Task #4 Clarity of instruction (Studying Ch. 5) Impact of instruction Benchmark Task #5 Clarity of instruction Clarity & (Performing Practice) Impact of Impact of instruction Instruction Benchmark Task #7 Clarity of instruction (Reviewing Quiz) Impact of instruction Clarity of instruction Benchmark Task #8 (Performing Prelaboratory) Impact of instruction Initial satisfaction Satisfaction with the screen QUIS 7.0 Satisfaction with terminology and Satisfaction system information Satisfaction with system capabilities Satisfaction with multimedia Data Collection Sheet Number of positive/negative remarks (* Observed Results: Mean values across 5 participants)

Target Level 4 15 0 ≤1 8 40 0 ≤1 4 30 0 ≤1 7 50 0 ≤1 5.10 5.10 5.10 5.10 5.10 5.10 5.10 5.10 8.10 8.10 8.10

Observed Results* 5.20 21 0.60 0 9.60 47.8 0.60 0.40 5 47.00 0.40 0.60 6.20 63.4 0 0.40 5.47 5.07 4.76 4.63 4.60 5.07 4.97 4.40 7.53 8.10 6.48

8.10 8.10 Negative ≤ 5

7.96 8.07 Negative = 7

Chang S. Nam


153

Effectiveness of the User Interface System To investigate the effectiveness of the user interface system in the WD2L environment prototype, this study employed two measures: percent of tasks completed and number of features used. Percent of tasks completed: The percent of tasks completed measure was computed as the ratio of completed tasks to total tasks (n = 8). This measurement reflects the overall task performance. Results showed that participants completed all 4 benchmark tasks successfully (i.e., 100% of success). Number of features used: The number of features used measure refers to the number of navigational features that participants used to perform a task on the WD2L environment prototype. Examples include click embedded links, press buttons, and type texts. Results showed that when performing tasks, participants used more features than target levels except for benchmark task # 6. To perform benchmark task #1 (i.e., find an information on the WD2L environment), a mean of 5.20 steps (SD = 1.10) was used (target level was 4 steps). Uploading an assignment on the Web required a mean of 9.60 features (SD = 1.14), while finding GPS resources required a mean of 5 steps (SD = 1.58) by participants. On the other hand, participants used fewer features to complete the sending en email task (M = 6.20; SD = 0.84). Efficiency of the User Interface System The efficiency of the user interface system was determined through three metrics: time on task, number of errors, and frequency of help use. Time on task: As shown in Table 4-9, the participants spent a longer time to get tasks done, compared to the target level of task times. The mean values to complete benchmark tasks

Chang S. Nam


154

#1, #2, #3, and #6 were 21.00 (SD = 4.85), 47.80 (SD = 3.70), 47.00 (SD = 2.83), and 63.40 (SD = 5.94) seconds, respectively. Number of errors: Results showed that participants did not make many mistakes while performing tasks, but levels of errors were not matched to the target levels. Mean numbers of errors were 0.6, 0.6, 0.4, and 0.0 for tasks #1, #2, #3, and #6, respectively. Frequency of the Help use: While performing task #1, all participants did not visit the Help page. On the other hand, participants consulted the Help page less than one time to finish task #2 (M = 0.40 times), #3 (M = 0.6 times), and #6 (M = 0.4 times), respectively. Clarity of the Instructional system Participants rated the degree of clarity of the Chapter 5 learning material higher (M = 5.47, SD = 0.63) than other learning materials such as practice sessions (M = 4.76, SD = 1.29), quiz review (M = 4.60, SD = 1.73), and prelaboratory (M = 4.97, SD = 1.20), respectively. Impact of Instructional system The impact of instruction was determined by measuring the degree of impact of instruction expressed by participants. Results showed that participants rated the degree of impact of the Chapter 5 learning material higher (M = 5.07, SD = 0.59) than other learning materials, such as practice sessions (M = 4.63, SD = 1.32), quiz review (M = 5.07, SD = 1.22), and prelaboratory (M = 4.40, SD = 1.31), respectively. Satisfaction of WD2L Environment The level of satisfaction was determined through five categories: initial satisfaction, screen, terminology and system information, learning, and system capabilities. Mean values were 7.53 (SD = 0.86), 8.10 (SD = 0.85), 6.48 (SD = 2.34), 7.96 (SD = 1.17), and 8.07 (SD = 1.16), respectively.

Chang S. Nam


155

Design Changes and Discussion

Results of the first One-to-One Evaluation session indicated that some changes were necessary to the second version of the WD2L environment prototype. This section summarized main design changes made to solve usability problems identified in the user interface and instructional system. Menu bar

Figure 4-8. Menu System in the Second Version of the Prototype Problems being fixed: Menu slip Results from the evaluation showed one of the menu usability problems identified by Brewster & Crease (1999), menu slip. Sometimes, users slipped into an adjacent menu item, which caused them to make an error and take a longer time. This happened partly because users often started to move the mouse to the location of the next action before the mouse button was released. Solutions: Ease of navigation To make navigation easy and avoid the mouse slipping off a menu by mistake, a new menu system was constructed. The menu bar displays a row of buttons, which display a drop down menu when clicked on. Additionally, menus have one or more sub menus (which may in turn have sub menus). Unlike the previous menu system that used colors, the new menu system

Chang S. Nam


156

change its appearance when a button is clicked on making it look depressed. The active button no longer changes in appearance even when the mouse moves from one button to another.

Figure 4-9. Menu System in the Third Version of the Prototype

Search box:

Problems being fixed: Results of the evaluation confirmed the disadvantage of a drop-down menu used for search options that people cannot see all options in the menu right away. Some of users did not notice that they first had to choose a search option (e.g., Whole site, GPS Resources, or GPS Glossary) corresponding to information they want to find. For example, users had to select the “Whole site” option to find pages containing the term ‘GPS’ before clicking the GO button. When users finally clicked the GO button, however, they realized that they should have chosen that search option.

Chang S. Nam


157

Solutions: Selection of search options in the search box After considering the role of search options, it was decided to make two changes to the search box. First, the “Choose an Option” item set to be default was eliminated while making the “Whole site” option default. Another change was to make the “GPS Resources” option and the “GPS Glossary” option default when users opened each pages, respectively. In this way, users do not have to choose a search option to find information.

Assignments Page: Layout

Figure 4-10. The Assignments Page in the Second Version of the Prototype

Chang S. Nam


158

Problems being fixed: Longer time to submit & few download options As shown in Figure 4-10, each assignment link in the page has its own submit button, but rather it took users a longer time to find certain assignment. Sometimes, they clicked the wrong submit button. Another usability problem identified was that to submit an assignment, some of the users went to the Send Email page rather than visiting the Assignments page. This problem showed the need of more explicit instructions about how to upload assignments. Solutions: Three major changes were made to the Assignments page consisting of the Main screen and Submit screen. As shown in Figure 4-11, these design changes include 1) the number of the Submit button and its position, 2) the use of file download options, and 3) the default setting in the Submit screen.

Figure 4-11. Design Changes to the Assignments Page

Chang S. Nam


159

First, in the second version, only one Submit button was placed at the bottom of the page rather than placing it for all assignments. To submit any assignment, the user clicks on this Submit button, which leads to the Submit page. Another change was the use of file download options for downloading an assignment. This change was made in a way that file download options such as the ‘Open’, ‘Save’, and ‘Cancel’ are provided as the user clicks an assignment link. Finally, the second version of the Submit page was changed, in which when the user opens the Submit screen, “Name” and “E-mail” fields are automatically filled out with his/her name and email address stored in the database system.

Send Email Page: Listing of individual email addresses Problems being fixed: When sending an email, the user had to type the email address. It required the user to spend some time to do it. Another problem identified was that users did not notice the “Special Characters” section quickly, which was intended to use special characters when typing mathematical equations. Solutions: As shown in Figure 4-12, two design changes were made to the Send Email page: list of email addresses of classmates in the drop down menu and instructions about how to use Send Email page. First, a list of email addresses of classmates was placed in the drop down menu so that users do not have to type the address, but to select it. Another change was to put instructions for using features provided in the Send Email page (e.g., the “To” and “Special Characters” section).

Chang S. Nam


Figure 4-12. Design Changes to the Send Email Page

160

Chang S. Nam


161

Chapter 5: Several changes were made to the Chapter 5 page to improve the clarity and impact of the instruction. First, it was made to link terms in the content to other terms. For example, when clicking the term, GPS, it will take to the GPS Glossary page in which the definition of it is shown. Second, underlines used to emphasize contents were eliminated to avoid confusion with links. Third, the mathematical expression has been changed, for example, 0.005 to 5 x 10-3. Fourth, as shown in Figure 4-13, animation was used to explain the effect of the equatorial bulge on orbital orientation more clearly. Many studies have shown that learning conditions using animation plus text result in more efficient learning, compared to text only and text plus graphics (e.g., Baek & Layne, 1988; Rieber & Boyce, 1990; Mayton, 1991).

After (version 3.0):

Figure 4-13. Design Changes to the Chapter 5 Page: Animation

Chang S. Nam


162

Finally, the Chapter 5 page was also changed to provide students with more detailed explanation on certain concept. As shown in Figure 4-14, for example, a pop-up screen describing tidal effects in more detail is displayed when the user clicks the Tidal Effects link.

Figure 4-14. Design Changes to the Chapter 5 Page: Tidal Effects Link

Chang S. Nam


163

Practice Sessions: Figure 4-15 shows an example of design changes made to the Practice 1 page. In the section version of the page, the user had to select an answer and click the Feedback button in order to see feedback on his/her choice. For an effective practice session, the Practice 1 page was redesigned in a way that the user can select an answer, which results in displaying the feedback screen directly.

Before (version 2.0):

Figure 4-15. Design Changes to the Practice 1 Page


Chang S. Nam


164

Quiz Review Page: As shown in Figure 4-16, the Quiz Review page was redesigned to better support the user’s quiz reviewing activities. First, the user does not have to click the Feedback button in order to see feedback in the third version. Selecting an answer directly displays the Feedback screen. Second, unlike in the second version in which one page contained all quiz questions, there is only one quiz question in a page in the third version. Finally, feedback screens were also redesigned to provide graphic-based feedback.

Before (version 2.0):

Figure 4-16. Design Changes to the Quiz Review Page


Chang S. Nam


165

Prelaboaroty 5 Page: Format of MATLAB Code, Submit Button, & Links Major changes to the Prelaboratory 5 page included the format of the MATLAB code, submit button, and links. First, the format of the MATLAB code was changed by making the code format similar to the code shown in the MATLAB software. For example, the color of variables is black and explanations were made to be green, just like colors in the MATLAB software program. Second, variables, functions, or even GPS glossaries were made into links. Finally, the Submit button was placed at the bottom of the page to allow students to save the MATLAB code they filled out into corresponding Laboratory pages. Therefore, students do not have to bring a hard or soft copy of the code completed to the laboratory session. When arriving at the laboratory, students just open the code on the Web by entering their password. After (version 3.0):

Figure 4-17. Design Change to the Prelaboratory 5 Page

Chang S. Nam


166

Other Design Changes Some of users went to the Send Email page rather than visiting the Assignments page. This problem shows more explicit instructions about how to upload assignments. To clearly indicate how to upload assignments, more explicit instructions have been placed in both the Help and the Site Map pages. GPS Resources Page: There are two ways to find a GPS Web site or GPS link in the GPS Resources page, scrolling down the page or clicking GPS Resources index shown at the top of the page. However, it took users a longer time to find a GPS link by scrolling down the page, as they had to inspect every links with the title and a brief description. Changes to the GPS Resources page included the alphabetic listing of GPS indices and links under each GPS index.

For example, the first approach was to list GPS indices

alphabetically. GPS links listed under each GPS index were also sorted alphabetically so that the users can find them more easily and quickly.

Chang S. Nam


167

Evaluation 2 Another One-to-One Evaluation session was conducted as most of the evaluation criteria were not fully met in the first One-to-One Evaluation session. In the second evaluation session, four new participants evaluated the third version of the WD2L environment prototype, which was revised in response to evaluation results with representative users.

Method

Participants: A new pool of four participants participated in the second session of the One-to-One Evaluation. There were 3 males and 1 female (M = 23.0 years, SD = 0.82). Participants were given monetary compensation for their participation.

Most participants

classified their computer skill level as somewhere between an intermediate and an experienced level. Experimental Materials and Benchmark Tasks:

Evaluation 2 used the same

experimental materials and benchmark tasks (see Table 4-6) used in the Evaluation 1. That is, participants evaluated the third version of the WD2L environment prototype by performing 8 benchmark tasks used in the first One-to-One Evaluation session. Procedure: The same experimental procedure used in the Evaluation 1 was also used in the Evaluation 2.

Chang S. Nam


168

Results The overall quality of the WD2L environment prototype was evaluated through several measures.

Table 4-10 shows these usability specifications measured across four different

participants. Table 4-10. A Summary of Usability Specifications: Evaluation 2 Usability Attribute


Value to be Measured Number of features Time on task Benchmark Task #1 (Searching Information) Number of errors Frequency of the Help use Number of features Benchmark Task #2 Time on task (Uploading Assignments) Number of errors Initial Frequency of the Help use Performance Number of features Benchmark Task #3 Time on task (Finding GPS Resources) Number of errors Frequency of the Help use Number of features Benchmark Task #6 Time on task (Sending Email) Number of errors Frequency of the Help use Benchmark Task #4 Clarity of instruction (Studying Ch. 5) Impact of instruction Benchmark Task #5 Clarity of instruction Clarity & (Performing Practice) Impact of Impact of instruction Instruction Benchmark Task #7 Clarity of instruction (Reviewing Quiz) Impact of instruction Clarity of instruction Benchmark Task #8 (Performing Prelaboratory) Impact of instruction Initial satisfaction Satisfaction with the screen QUIS 7.0 Satisfaction with terminology and Satisfaction system information Satisfaction with system capabilities Satisfaction with multimedia Data Collection Sheet Number of positive/negative remarks (* Observed Results: Mean values across four participants)

Target Level 4 15 0 ≤1 8 40 0 ≤1 4 30 0 ≤1 7 50 0 ≤1 5.10 5.10 5.10 5.10 5.10 5.10 5.10 5.10 8.10 8.10 8.10

Observed Results* 4.50 14.50 0.25 0.00 8.00 36.25 0 0.25 4.00 29.50 0.00 0.75 5.50 48.00 0.00 0.25 5.50 5.25 5.20 5.17 5.60 5.58 5.17 5.25 8.13 8.19 8.13

8.10 8.10 Negative ≤ 5

8.15 8.17 Negative = 3

Chang S. Nam


169

Effectiveness of the User Interface System To investigate the effectiveness of the user interface system in the WD2L environment prototype, the Evaluation 2 session also used two measures: percent of tasks completed and number of features used. Percent of tasks completed: The percent of tasks completed was computed as the ratio of completed tasks to total tasks (n = 8). This measurement reflects the overall task performance. Results showed that participants completed all 4 benchmark tasks successfully (i.e., 100% of success). Number of features used:

The number of features used refers to the number of

navigational features that participants used to perform a task on the WD2L environment prototype. These features included click embedded links, press buttons, and type texts. Results showed that when performing tasks, participants used less than or equal to target levels. To perform benchmark task #1 (i.e., find an information on the WD2L environment), a mean of 4.50 steps (SD = 1.00) was used (target level was 4 steps). Uploading an assignment on the Web required a mean of 8.00 features (SD = 0.00), while finding GPS resources required a mean of 4 steps (SD = 0.82) by participants. On the other hand, participants used fewer features to complete the sending en email task (M = 5.50, SD = 0.58).

Efficiency of the User Interface System The efficiency of the user interface system was determined through three metrics: time on task, number of errors, and frequency of help use. Time on task: As shown in Table 4-10, participants spent a shorter time to get tasks done, compared to the target level of task times. The mean values to complete benchmark tasks

Chang S. Nam


170

#1, #2, #3, and #6 were 14.50 (SD = 1.29), 36.25 (SD = 3.10), 29.50 (SD = 2.65), and 48.00 (SD = 4.24) seconds, respectively. Number of errors: Results showed that participants did not make mistakes to perform tasks except for the task of finding information (i.e., Benchmark task #1, mean number of error = 0.25). Frequency of the Help use: While performing task #1, participants did not visit the Help page. On the other hand, participants saw the Help page less than one time to finish the task #2 (M = 0.25 times), #3 (M = 0.75 times), and #6 (M = 0.25 times), respectively. Clarity of Instructional system The degree of clarity of instruction was rated higher than target levels.

Chapter 5

learning material received a mean of 5.50 (SD = 0.51), while practice sessions, quiz review, and prelaboratory got mean values of 5.20 (SD = 0.77), 5.60 (SD = 0.60), and 5.17 (SD = 0.58), respectively. Impact of Instructional system The impact of instruction was determined by measuring the degree of impact of instruction expressed by participants. Chapter 5 learning material received a mean of 5.25 (SD = 0.75), while practice sessions, quiz review, and prelaboratory got mean values of 5.17 (SD = 0.72), 5.58 (SD = 0.67), and 5.25 (SD = 0.62), respectively. Satisfaction of WD2L Environment The level of satisfaction was determined through five categories: initial satisfaction, screen, terminology and system information, learning, and system capabilities. Mean values were 8.13 (SD = 0.61), 8.19 (SD = 0.75), 8.13 (SD = 0.68), 8.15 (SD = 0.59), and 8.17 (SD = 0.72), respectively.

Chang S. Nam


171

Design Changes and Discussion

Results of the second One-to-One Evaluation session showed that almost evaluation criteria were met. However, some changes were still necessary to the third version of the WD2L environment prototype, given design comments provided by participants. This section described main design changes made to the third WD2L environment prototype. Chapter 5 Page: Users considered the Tidal Effects link useful resource that helped them understand tidal effects.

Users also suggested implementing Solar and Relativistic effects.

Figure 4-18 shows an example of implementing the link to show Solar Effects in more detail.

Figure 4-18. Design Change to the Chapter 5 Page: Solar Effects Link

Chang S. Nam


172

Laboratory Exercise Unit: Another main change made to the third version of the WD2L environment prototype was to provide more functions that support prelaboratory exercise activities. As shown in Figure 4-19, for example, a design change was made to add the Save button and a brief description of how to use it.

Figure 4-19. Design Change to the Prelaboratory 5 Page: Add Save Function After completing MATLAB codes while consulting feedback screens, the user clicks the Save to My Computer button to save completed MATLAB codes into his/her computer. Most users wanted this function because they must run their completed MATLAB codes in the MATLAB software program to make sure that there is no error. Clicking on the Save to My Computer button displays a new page showing completed MATLAB codes to allow users to review them once again. If no mistakes, users click on the Save to My Computer button to save the page to their computer.

Chang S. Nam


173

Small Group Evaluations Process The main purpose of the Small Group Evaluation was to assess if the user interface system (i.e., multi-user interfaces) supported group learning activities (e.g., group discussion). In the Small Group Evaluation session, a group of participants representative of the target population evaluated the Discussion Board system to test whether it effectively supported group discussions.

Method Participants For the Small Group Evaluation process, 3 participants were recruited from a pool of the students taking the GPS course.

Participants were given monetary compensation for their

participation. There were 3 males (M = 22.7, SD = 0.58).

Experimental Materials and Benchmark Tasks To evaluate main functions of the Discussion Board system, this study developed one group benchmark task, in which three participants were asked to exchange their opinions about a proposed topic, “Future of GPS Applications,” through group discussion. The group benchmark task was described in Appendix 11.

Evaluation Criteria The overall quality of the Discussion Board was determined in terms of the effectiveness, efficiency, and user satisfaction.

Table 4-11 shows a list of metrics and corresponding

performance and subjective measurements.

Chang S. Nam


174

Table 4-11. Usability Metrics for the Small Group Evaluation Metric Effectiveness Efficiency

Satisfaction

Measurement Number of features used* Numbers of errors* Frequency of the Help use* Initial satisfaction Satisfaction with the screen Satisfaction with terminology and system information Satisfaction with system capabilities

Definition Number of action sequences used to complete a task Numbers of errors made by users Number of times a user used the Help to complete a task Rating scales for initial satisfaction Rating scales for satisfaction with the screen Rating scales for satisfaction with terminology and system information Rating scales for satisfaction with system capabilities

(* These measurements were measured when posting the first message.)

To measure user satisfaction with user interfaces, this study employed the Questionnaire for User Interface Satisfaction questions (QUISTM 7.0), answered with 0 to 9 ratings. The QUISTM developed by a multi-disciplinary team of researchers in the Human/Computer Interaction Lab (HCIL) at the University of Maryland is the most extensive and most thoroughly validated questionnaire for determining subjective interface usability (Slaughter et al., 1995). The concept of satisfaction consisted of five categories. These were initial satisfaction, screen, terminology and system information, learning, and system capabilities. Usability Specifications Table:

To record quantitative results (i.e., usability

measurements) from participants who perform the benchmark task, this study employed the usability specifications table (Hix & Hartson, 1997).

Table 4-12 shows the usability

specifications table used in the study and target levels. Table 4-12. Usability Specifications used for the Small Group Evaluation Usability Attribute Initial Performance

Satisfaction

Measuring Instrument Group Benchmark Task

QUIS 7.0

Data Collection Sheet

Value to be Measured Number of features Number of errors Frequency of the Help use Initial satisfaction Satisfaction with the screen Satisfaction with terminology and system information Satisfaction with system capabilities Number of positive/negative remarks

Target Level 5 0 ≤1 8.10 8.10 8.10

Observed Results

8.10 Negative ≤ 5

As the “Usability Attribute,” referring to what is to be measured, the “initial performance” (i.e., the first time the user performed the task) will be reported. In addition, the

Chang S. Nam


satisfaction attribute will be identified.

175

The “Measuring Instrument” column refers to the

specific benchmark task through which usability attributes are measured. One group benchmark task was designed for the initial performance attribute. identified through the QUIS questionnaire.

The satisfaction attribute will be

Column three represents the “Value to be

Measured.” Three usability measurements will be used: number of features, number of errors, and frequency of help use. The fourth column indicates the “Target Level” representing the performance goal.

In the case of the number of features measurement, target level was

determined by measuring the fastest steps to post the first message. For example, fastest steps to post a message are 5 steps (i.e., click the Write icon → type the password → type the Title → Type the Message → click the Submit button). No mistakes should be allowed, and it will be allowed for users to consult the Help page less than one time. As target levels of satisfaction measurements, 90% of a perfect score in the QUIS (i.e., 8.1 out of 9.0) was determined. On the other hand, target levels of number of negative remarks were decided a little more arbitrarily, but were intended to be rigorous enough to catch major usability problems, in which the number of negative remarks should be less than and equal to 5. Column five represents results obtained from participants. Procedure: After informed consent (Appendix 10) was acquired, participants were given written instructions for the task (Appendix 9).

Then, participants performed one group

benchmark task (Appendix 11). One participant played a role of the group leader and managed the discussion session among group members and also expressed his own opinions. Two other participants were group members, who were asked to provide their own opinions at least three times about the proposed topic, “future of GPS application.” Participants also were asked to think aloud throughout the whole session and talk about what they are doing, why they are doing it, and what they expect to happen when they perform an action. That is, they were asked to indicate both the positive (good) and negative (bad) aspects of how they perform the task. These observations were recorded on a “Data Collection Sheet” (see Appendix 12) and mapped for further analysis. At the end of the evaluation, participants completed the Questionnaire for User Interface Satisfaction (QUISTM). Sample questions can be found in Appendix 14.

Chang S. Nam


176

Results To investigate the overall quality of the Discussion Board in the WD2L environment prototype, several measures were employed. Table 4-13 shows these usability specifications measured across 3 participants. Table 4-13. A Summary of Usability Specifications: Small Group Evaluation Usability Attribute Initial Performance


Value to be Measured Number of features** Group Benchmark Task Number of errors** Frequency of the Help use** Initial satisfaction Satisfaction with the screen QUIS 7.0 Satisfaction Satisfaction with terminology and system information Satisfaction with system capabilities Data Collection Sheet Number of positive/negative remarks * Observed Results: Averaged values across three participants. ** These measurements were measured when posting the first message.

Number of features used:

Target Level 5 0 ≤1 8.10 8.10 8.10

Observed Results* 5 0 0 8.28 8.30 8.37

8.10 Negative ≤ 5

8.40 2

The number of features used refers to the number of

navigational features that participants used to post the first message.

Results showed that

participants used a mean of 5 steps (SD = 0.00). Number of errors: Results showed that participants did not make mistakes while using the Discussion Board. Frequency of the Help use: While performing the task, participants did not visit the Help page. Satisfaction: The level of satisfaction was determined through four categories: initial satisfaction, screen, terminology and system information, and system capabilities. Mean values were 8.28 (SD = 0.46), 8.25 (SD = 0.45), 8.39 (SD = 0.50), and 8.40 (SD = 0.51), respectively.

Chang S. Nam


177

Design Changes and Discussion Results of the Small Group Evaluation process showed that evaluation criteria were met. However, some changes were still necessary to the WD2L environment prototype, given comments provided by participants. This section described two main design changes made to the Discussion Board system of the WD2L environment prototype. Group Selection Options: A design change was made to provide more options for the Discussion Board. For example, each group can select group options (e.g., group 1 and group 2), which displays the Discussion Board page only for that group. Figure 4-20 shows the design change made to the Discussion Board page with group selection options in a pull-down menu.

Figure 4-20. Design Change made to the Discussion Board: Group Selection Options

Chang S. Nam


178

Pre-display Content Function: Another design change made to the Discussion Board page was to add the pre-display function. As shown in Figure 4-21, this function allows user to see a message content without having to click it. Moving the mouse on a message automatically shows users its content.

Figure 4-21. Design Change made to the Discussion Board: Group Selection Options

Chang S. Nam


179

Expert Review (2nd) Process The WD2L environment prototype went through evaluation processes four times. In this process, subject matter experts (SMEs) finally reviewed the prototype to identify and revise overlooked areas or problems. Method Participants A 30-year-old Ph.D. candidate in the area of instructional design served as an instructional design expert. She worked in VT Faculty Development Institute (FDI) as a training assistant in summer 2001 and helped faculty members to develop their online courses. She has also been working as a designer and grader in Instructional Technology online Master Program (ITMA) at VT since 2001. As an independent designer, she has designed and developed two Web sites to teach Mathematics and Chinese language. As a GPS content expert, a 27-year-old Master candidate participated in the expert review process. He worked as a research assistant for two years in the GPS laboratory. He was also a teaching assistant for the GPS course during Fall of 2001 and Fall of 2002. As an interface design expert, a 32-year-old human factors Ph.D. student in the area of user interface design participated to provide design suggestions to improve the user interface system. He worked as a human factors engineer for five years in the area of the interface design in vehicle systems. He also completed all the major human factors courses related to user interface design, such as usability engineering and system design courses.

Chang S. Nam


180

Equipment/Apparatus The three SMEs reviewed the fifth version of the WD2L environment prototype running on a Pentium computer, equipped with a standard keyboard, mouse, and 17” super VGA monitor. Internet Explorer (IE) 6.0 running in Windows 2000 was used as a Web browser. To review and suggest their design change recommendations to improve the WD2L environment prototype, the SMEs were asked primarily to utilize their expertise in their specialties. In addition, to help the SMEs review important aspects of the WD2L environment prototype, this study provided three types of the expert review forms: User Interface Review Form, Instructional Design Review Form, and Content Review Form.

Procedures

This evaluation process employed the same evaluation procedure as used in the first session of the Expert Review process. Three SMEs were given written instructions for the task by asking them to review and provide design comments or recommendations that would help revise the WD2L environment prototype, in terms of the GPS learning content, instructional design, and user interface design. Appendix 9 provides the task instruction given to the SMEs. To help the SMEs assess the first version of the WD2L environment prototype, two types of supporting materials were also provided: a user profile and review forms. The user profile specified in the Requirement Specification Document was given to help the SMEs have a better understanding on the target user group. The expert review forms developed in the study were also provided to each expert. It took about two hours for each expert to complete the evaluation of the WD2L environment prototype. At the end of the review, the SMEs were debriefed.

Chang S. Nam


181

Results User Interface Exert Review Evaluation on Components of User Interface System: The interface expert reviewed components of the user interface system, and provided design recommendations for the modification of the user interface system.

Table 4-14 shows examples of comments and

recommendations that the user interface expert provided on the user interface system. Problems and design recommendations to improve them provided by the user interface expert are addressed more fully in Appendix 6. Table 4-14. Examples of Expert Comments/Recommendations on User Interface System Components Homepage

Search box

Comments/Recommendations • The homepage has good color scheme. Menus are good. • Put more information on the homepage to describe briefly what this site is all about. • Search result for Whole Site is well designed. • Result for glossary should be indexed with numbers. Otherwise, overall the search results are quite well presented with the correct size fonts and color scheme.

Components GPS Resources

Comments/Recommendations • Well laid out, results for search should be done in this manner.

Practice Sessions

• The Practice Link should not be moving, as it is too distracting and does not appear to be a link. Rather you can have a blinking icon by the stationary link to capture attention.

Overall Quality of User Interface System: Table 4-15 shows the overall quality of the user interface system evaluated by the interface expert. Most of dimensions were rated highly. Table 4-15. A Rating Summary of User Interface System Dimension Ease of use Navigation Cognitive load Mapping Screen design

Rating (6.0) 6.0 6.0 5.0 5.0 5.0

Dimension Knowledge space Information presentation Media integration Aesthetics Overall functionality

Rating (6.0) 5.0 6.0 6.0 6.0 6.0

Chang S. Nam


182

Instructional Design Expert Review Instructional Design: The instructional design expert evaluated how well components of the

instructional strategy was

implemented, and provided

various comments and

recommendations for the modification of the instructional system. Table 4-16 shows examples of design comments/recommendations that the expert provided on the instructional design. Problems and recommendations made by the expert are fully described in Appendix 7. Table 4-16. Examples of Expert Comments on Instructional Design Instructional Event 3. Stimulating recall of prior learning

Comments/Recommendations • Good design of “Think for a while” • Detailed informative feedback. Very helpful. And practice parts have different formats, e.g. multiple choice, filling the blanks, and image. The diversity of the practice will help to motivate learners and keep them in the track. Very nice design.

Overall Quality of the Instructional Design: The overall quality of the instructional design was good (M = 6.0, SD = 0.0). The instructional design provided high points to the three dimensions (i.e., Design for Target Audience (6.0), Match to Learning Objectives (6.0), and Clear to be Self-Instructional (6.0)).

Chang S. Nam


183

Content Expert Review The GPS content expert reviewed learning units and provided design recommendations for the modification. Table 4-17 shows examples of comments and recommendations that the GPS content expert provided. Problems and design recommendations to improve them are addressed more fully in Appendix 8. Table 4-17. Examples of Expert Comments/Recommendations on Learning Units Unit Content Unit Practice Unit

Comments/Recommendations • Links for GPS terminology help a lot. • Some graphs are still not exactly as the same as the textbook. • The feedback button and the prompt feedback are very helpful.

Table 4-18 represents the overall quality of learning units evaluated by the GPS content expert. Learning units received relatively high scores, ranging from 4.8 (Practice Unit – practice 4) to 5.4 (Quiz Review Unit – Quiz 1 Review). Ratings on learning units were provided in Appendix 8. Table 4-18. A Summary of Rating on Learning Units Learning Unit Content Unit - Chapter 5 Practice Unit – Practice 1 Practice Unit – Practice 2 Practice Unit – Practice 3

Rating (6.0) 5.3 4.8 5.3 4.5

Learning Unit Practice Unit – Practice 4 Practice Unit – Practice 5 Quiz Review Unit – Quiz 1 Review Laboratory Exercise Unit - Prelaboratory 5

Rating (6.0) 5.0 5.5 5.3 6.0

Chang S. Nam


184

Design Changes and Discussion This section describes primary design changes made to the fifth version of the prototype. These changes were made in response to design change recommendations suggested by the three SMEs. Practice Unit: The Practice 1 page was changed to include a practice question that makes learners think of a big picture of what they have learned. For example, the question, “Please type 5 key words or sentences that best characterize Section 1 of Chapter 5.”, was included in the Practice 1 page. Prelaboratory Page: To provide learners with more help when completing MATLAB codes, the Prelaboratory 5 page implemented the caption for MATLAB variables. For example, when the user moves the mouse on the Mcorr variable, it shows the caption saying, “Mcorr is the anomaly correction item.” Concept Map: The Concept Map page implemented in previous versions of the prototypes was static in that it did not allow users to manipulate GPS concepts. A new type of the Concept Map page was designed, in which learners are asked to move GPS concepts to relevant positions. The Concept Map page was redesigned to allow learners themselves to organize the Concept Map.

Chang S. Nam


185

Field Experiment This section summarized the Field Experiment, which assessed the effectiveness of the WD2L environment as a Web-based GPS supplemental learning program.

The WD2L

environment prototype was evaluated as a way of ascertaining the quality of the design process used in the present study. The Field Experiment was also conducted to identify how users evaluated the quality of resources implemented in WD2L environment prototype. Research Questions The study sought to answer the following research questions concerning the effectiveness of the Web-based GPS supplemental learning program developed in the present study. 1. Are there any differences in students’ learning performance between the Web-based GPS supplemental learning program and traditional supplemental learning? 2. How do users evaluate the quality of resources implemented in the Web-based GPS supplemental learning program? The experiment was designed to acquire data and conduct analyses to support the following hypothesis. H0: There will be no significant mean differences between performance scores of transfer for Web-based supplemental and traditional groups.

Chang S. Nam


186

Method Participants

Twenty students who took the GPS course in Fall of 2003 volunteered in the study. There were 4 female and 16 male participants (M = 23.13 years, SD = 2.9).

General Experimental Design

The experimental design for the present study is presented in Figure 4-22. As shown in Figure 4-22, there was 1, 2 level between-subjects condition: Web-supplemental and traditional conditions. Participants in each condition were pre-tested and post-tested on their recall and assessed on their transfer of knowledge. Participants in the Web-supplemental condition also assessed the quality of WD2L environment resources. R

OPretest

Lecture

OPosttest

XTradition

OTransfer

R

OPretest

Lecture

OPosttest

XWeb

OTransfer

where, R: Randomization; O: Measurement of dependent variable; X: Independent variable

Figure 4-22. General Experimental Design This study employed Campbell & Stanley’s (1966) true experimental design in that the study included a purposively created control group (participants in the “traditional” condition), common measured outcome (learning performance), and random assignment (participants were randomly assigned into each condition).

Independent Variables

There was one independent variable manipulated in the study: supplemental learning type. The supplemental learning type condition was manipulated as a 2 level condition: Web-

Chang S. Nam


187

supplemental and traditional conditions. In the “Web-supplemental” condition, participants were told to study the learning content using the Web-based supplemental learning program developed in the study. In the “Traditional” condition, participants were instructed to study the learning content in their traditional ways such as reading textbook, asking the instructor, or going to the library.

Dependent Variables

Recall Test: Participants were measured on the quantity of knowledge they were able to recall in the study. Participants were assessed on initial knowledge prior to (pretest) and recall following (posttest) the lecture. The test was a short, essay type question developed by the professor who teaches the GPS course, based on studies of Mayer & Chandler (2001) and McFeeters (2003). It took the participants approximately 10 minutes to complete the pre-test and posttest. Transfer Test: Participants were assessed on the quantity of knowledge they were able to integrate and apply to open-ended GPS questions in the study. The test consisted of three short, essay types of questions developed by the professor who teaches the GPS course, based on studies of Mayer & Chandler (2001) and McFeeters (2003). These questions and scoring rubrics were shown in Appendix 15. It took the participants approximately 15 minutes to complete. Quality of WD2L environment resources: The participants in the “Web-supplemental” condition” assessed the quality of resources implemented in the Web-based GPS supplemental learning program on a scale of 1 to 6 or 1 to 9.

Experimental Materials

Instruments for this experimental research included a recall test, a transfer test, and a modification of the evaluation on the Web-based learning program questionnaire originally developed by Gregor Kennedy (Felix, 1998).

Chang S. Nam


188

Recall Test: A recall assessment should be used in order to gauge how much of the presented material the learner can remember (Mayer, 2001). Participants were assessed on their initial knowledge prior to and recall following the lecture. The test question was “What physical effects (including the most important one) produce perturbations on satellite orbits predicted by the basic Kepler orbit theory?” Accuracy was based on the occurrence of acceptable ideas in the participant’s responses. To compute a score for a participant, initial knowledge and recall were measured by the participant’s ability to remember the following idea units in their pre- and posttest responses (Mayer 2001; McFeeters, 2003). 1. Non-sphericity of the Earth 2. Tidal forces 3. Solar radiation 4. Relativistic effects Performance was expressed as the number of idea units reported divided by the total possible (four). The recall test took the participants approximately 10 minutes to complete. Transfer Test: Transfer test questions were developed, based on studies of Mayer & Chandler (2001) and McFeeters (2003). The test sought to measure “meaningful understanding in which participants are required to use the presented information in ways beyond what was presented” (Mayer & Chandler, 2001, p. 393). The transfer test contained the following three questions. 1. Explain what a harmonic correction to a GPS satellite ephemeride is. 2. List and briefly describe at least two gravitational effects that perturb GPS satellite orbits. 3. List and briefly describe at least two non-gravitational effects that perturb GPS satellite orbits. The teaching assistant graded each question by using a separate rubric (Appendix 15). Table 4-19 shows an example of scoring rubrics for transfer test. In order for a participant’s response to be considered accurate, each rubric included specific ideas from each question that should have been included in the participant’s response. The rubric contained four acceptable

Chang S. Nam


189

ideas per question. Each acceptable idea was given a point value. The most specific acceptable idea was given the highest points (3 points). Less specific acceptable ideas were given a lower score (2 points). Table 4-19. An Example of Scoring Rubrics for Transfer Test Question 2: List and briefly describe at least two gravitational effects that perturb GPS satellite orbits. Answer A Answer B Answer C Answer D 3 points 2 points 1 points 1 points “The non-sphericity of the “J2 and Tidal effects. As “Tidal effects – it puts a “The equatorial bulge Earth causes the the Earth spins on its axis, gravitational pull on the causes the satellite to gravitational pull by the forces cause a bulge at the earth and also causes a speed up.” Earth to vary latitude. equator of the Earth. pull on the satellite.” Additionally, the Sun and Therefore the equatorial Moon’s gravitation pull phase is significantly draws the satellite closer to larger that the polar those celestial bodies, just planes. Tidal effects are like tides.” caused by the gravitational forces between the Sun, Moon and Earth.”

Vague answers were given the lowest score (1 point). If an answer was considered unacceptable it was given a score of zero. Students received credit for an answer if they expressed either of four categories of ideas provided in the rubrics regardless of writing style or use of terminology. Each participant’s transfer performance is expressed as the number of acceptable answers generated divided by a total of 9. The transfer of knowledge test took approximately 15 minutes to complete. Web-based Learning Program Questionnaire: To investigate how users evaluated the quality of resources implemented in the Web-based GPS supplemental learning program, this study modified and used Felix’s (1998) questionnaire for evaluation of Web-based learning program, originally developed by Gregor Kennedy (see Appendix 16 for sample questions). The questionnaire included 8 dimensions: objectives/directions, content/structure, interactivity, navigation, text, sound, graphics, and interface.

Chang S. Nam


190

Procedure

The class participants were asked to take a short, essay type of the test (pretest). To learn Chapter 5, all participants took the class through a traditional classroom instruction for three separate days. Right after the class, the short essay-type test was repeated (posttest). Participants who were randomly assigned to the “Web-supplemental” condition (10 students) were instructed to use the WD2L environment as a GPS supplemental learning program to further study. They were told to visit the site at least once a day for 30 minutes. The other half of participants (10 students) who were randomly assigned to the “Traditional” condition was told to study Chapter 5 further using their normal method (e.g., reading books or asking instructor). After 5 days, all participants took the transfer of knowledge test. The “Websupplemental” group completed the “Evaluation of Web-based GPS supplemental learning program” questionnaire.

Chang S. Nam


191

Results Descriptive Analysis: The data collected included pre- and posttest scores on a one question essay test, a three essay question transfer of knowledge test, and a 47-item learner preference questionnaire. Table 4-20 shows mean scores of Web-supplemental and traditional groups on pretest, posttest, and transfer of knowledge. Table 4-20 Means and Standard Deviations for Recall and Transfer of Knowledge Web-supplemental Group Traditional Group

Pretest M = 0.70 SD = 0.37 M = 0.65 SD = 0.29

Posttest M = 0.95 SD = 0.15 M = 0.95 SD = 0.16

Transfer M = 0.57 SD = 0.26 M = 0.63 SD = 0.25

Mean pretest scores were 0.70 (SD = 0.37) for the Web-supplemental group and 0.65 (SD = 0.29) for the traditional group. The mean score of posttest for the Web-supplemental group was 0.65 (SD = 0.15) and 0.95 (SD = 0.16) for the traditional group. Mean transfer scores were 0.57 (SD = 0.26) for the Web-supplemental group and 0.63 (SD = 0.25) for the traditional group. Validity Test: A t-test was performed to show that there was no significant differences in participants’ initial knowledge between the Web-supplemental group and traditional group. The result showed no significant differences in pretest scores between the Web-supplemental group and traditional group: t (18) = -0.34, p = .741. On the other hand, significant differences were found between pretest and posttest scores for both groups (t (9) = 2.37, p = 0.042 for the Websupplemental group; t (9) = 2.45, p = 0.037 for the traditional group). This result indicates a significant increase in scores after the lecture. However, gain scores (difference between pretest and posttest scores) between the two groups were equal after the exposure to the lecture (t (18) = 0.31, p = 0.761). Transfer of knowledge between the two groups (H0): To test the hypothesis that there will be no significant differences in students’ learning performance between the Web-based GPS supplemental learning program and traditional supplemental learning, a t-test on knowledge of transfer was conducted. At an alpha level of 0.05, the hypothesis was accepted, t (18) = 0.59, p =

Chang S. Nam


192

.563, and it was concluded that there were no significant differences in students’ learning performance between the Web-based GPS supplemental learning group and traditional supplemental learning group. Learner Preference: Students were asked to indicate their preference in which GPS learning materials might be used on the Web. Almost all the participants considered that the best way to use Web materials was as an addition to face-to-face teaching used in their own time (6 out of 8 responses). Participants in the Web-supplemental group were asked to evaluate various aspects of the programs they used for GPS learning and to rate them on a scale from 1 to 5 or 1 to 9. Table 421 and 4-22 show the frequency for each category. Responses were favorable, ranging from 71% to 92% agreeing that the objectives were clear, the content was logical, the program was interactive and the navigation was easy. Table 4-21. Frequency Table: Objectives, Content, Interactivity, and Navigation Strongly disagree Disagree Slightly disagree Slightly agree Agree Strongly agree Total

Objectives / Directions Frequency Percent 1 4.2 1 4.2 5 20.8 6 25.0 8 33.3 3 12.5 24 100.0

Content / Structure Frequency Percent 0 0.0 0 0.0 7 10.9 14 21.9 21 32.8 22 34.4 64 100.0

Interactivity Frequency Percent 1 3.1 1 3.1 7 21.9 8 25.0 8 25.0 7 21.9 32 100.0

Navigation Frequency Percent 0 0.0 2 4.2 2 4.2 13 27.1 19 39.6 12 25.0 48 100.0

Some 56% to 100% rated the quality of the text, graphics, and interface as 6 or above on a scale of 1 to 9 (The first dimension in the text category needs to be reversed since lower ratings represent readability). On the other hand, more than 60% of the participants did not consider voice recordings of learning material useful to their GPS learning (rated as 5 or below on a scale of 1 to 9).

Chang S. Nam


193

Table 4-22. Frequency Table: Text, Graphics, Sound, and Interface Low 3 4 5 6 7 8 High 9 Total

Text Frequency Percent 0 0.0 2 12.5 5 31.3 1 6.3 2 12.5 5 31.3 1 6.3 16 100.0

Graphics Frequency Percent 0 0.0 0 0.0 0 0.0 2 6.3 4 12.5 17 53.1 9 28.1 32 100.0

Sound Frequency Percent 6 42.9 0 0.0 3 21.4 2 14.3 3 21.4 0 0.0 0 0.0 14 100.0

Interface Frequency Percent 0 0.0 0 0.0 0 0.0 2 5.0 9 22.5 17 42.5 12 30.0 40 100.0

Discussion The hypothesis that attempted to analyze any differences in students’ learning performance between the Web-supplemental learning group and traditional supplemental learning group was analyzed by performing the t-test on the transfer of knowledge measure. The analysis revealed that although there was a slight difference in mean testing scores between the Web-supplemental group (M = 0.57) and traditional group (M = 0.63), it was not significantly different.

The most important conclusion to be drawn from this result is that Web-based

supplemental learning programs could work as well as traditional supplemental learning in terms of transfer of knowledge. This finding is consistent with a line of studies found that there is no significant difference in delivery methods (see Russell, 1999). The significance of this result was discussed further in the General Discussion and Conclusions section. Another finding related to the usefulness of the WD2L environment resources was that all responses were favorable. This is important from users’ point of view because the study showed that the participants wanted to use Web materials as an addition to face-to-face teaching used in their own time. It is very important for Web resources implemented in the WD2L environment to effectively support users’ learning on their own (Lohr, 2000; Plass, 1998).

Chang S. Nam


194

Summary This chapter summarized formative evaluation processes and design changes made according to results of the evaluation with users. First, the proposed formative evaluation framework was briefly described, consisting of the Expert Review (1st), One-to-One Evaluation, Small Group Evaluation, and Expert Review (2nd) processes. Second, each formative evaluation process was summarized, along with results obtained from the evaluation and design changes made to the WD2L environment prototype. In Expert Review (1st) and Expert Review (2nd) processes, the SMEs who exhibited a high level of expertise in three main areas, instructional design, user interface design, and GPS content, reviewed the WD2L environment prototype and suggested design recommendations to improve it two times. In the One-to-One Evaluation process, two evaluation sessions (Evaluation 1 and 2) were conducted with representative users to identify and remove more prominent errors in the second version of the WD2L environment prototype. While the Expert Review (1st) process heavily relied on the SMEs’ expertise in their specialty, the One-to-One Evaluation process assessed the second version of the WD2L environment prototype using evaluation criteria (e.g., effectiveness, efficiency, and satisfaction for the user interface system). In particular, this process focused on individual learning activities and single-user interfaces used to support them. In the Small Group Evaluation session, a group of participants representative of the target population evaluated the Discussion Board system to test whether it effectively supported group discussions. Finally, the Field Experiment assessed the effectiveness of the WD2L environment as a Web-based GPS supplemental learning program.

The WD2L environment prototype was

evaluated as a way of ascertaining the quality of the design process used in the present study. The Field Experiment was also conducted to identify how users evaluated the quality of resources implemented in WD2L environment prototype.

Chang S. Nam

Chapter V. General Discussion and Conclusions

195

CHAPTER V GENERAL DISCUSSION AND CONCLUSIONS The discussion of the findings is divided into four sections:

(1) discussion on

relationships between the Integrated Design Process and effectiveness of the WD2L environment, (2) recommendations for applications of and modifications to the proposed Integrated Design Process and Design Process Template, (3) limitations of the study, and (4) suggestions for further research in this area.

Discussion This study described an effort to develop a theory-based Integrated Design Process (IDP) in order to improve the design process and usability of the WD2L environment as a Web-based supplemental learning program. As was expected, the proposed Integrated Design Process used to develop the WD2L environment was effective in that the study showed (1) the WD2L environment’s (as a Web-based GPS supplemental learning program) equivalence to traditional supplemental learning, and (2) users’ positive perceptions of WD2L environment resources. Equivalence to traditional supplemental learning: To investigate whether there were significant differences in students’ learning performance between the Web-supplemental group and traditional group, a t-test on knowledge transfer was conducted. Mean transfer scores obtained in the study were 0.57 (SD = 0.26) for the Web-supplemental group and 0.63 (SD = 0.25) for the traditional group. The analysis revealed that mean testing scores between the Websupplemental group and traditional group were not significantly different. Therefore, the WD2L environment as a Web-based GPS supplemental learning program could work as well as traditional supplemental learning. The Web-supplemental group showed a low mean transfer score (57% correctness), but it showed a similar mean score of the GPS class when they were tested on that unit (i.e., Chapter 5), which is one of the most difficult units (Wayne A. Scales, personal communication, September, 27, 2003).

The mean transfer score of the Web-

supplemental group (M = 0.57, SD = 0.26) was also lower than that of the traditional group (M =

Chang S. Nam


196

0.62, SD = 0.25). The observation that the Web-supplemental group showed a low mean transfer score, which was also lower than that of the traditional group, can be explained by two reasons. First, the amount of time that the Web-supplemental group spent using the WD2L environment was not long enough to show the main advantage of the Web-based supplemental learning environment: learners could re-study learning materials whenever they chose. Human memory decays as time elapses (Moray, 1986, in Wickens, 1992). The Web-based supplemental learning environment, which provided learners with opportunities for practice, over-learning, and elaborate rehearsal, should have decreased the rate of forgetting more effectively than learners’ traditional learning supplementation (e.g., reading a textbook and class notes) over time. Second, the WD2L environment still needs more practice sessions and informative feedback, which can facilitate students’ learning. This finding is consistent with a line of studies that found no significant difference in delivery methods.

These studies were referred to as the “The No-Significant-Difference

Phenomenon” (Russell, 1999). A lack of a significant difference between the Web-supplemental group and traditional group provides good evidence that the WD2L environment as a Web-based supplemental learning program does not discernibly create any disadvantage for the students who use it (Andrew, 2003). This finding is important because this study demonstrated that the WD2L environment may support students’ post-study activities just as well as traditional supplemental learning.

From the instructors’ point of view, the Web-based supplemental learning

environment’s equivalence to traditional supplemental learning means that there are more channels with which they can support students’ learning activities (Chadwick, 1999). This finding is also important to students because they can be confident that using the WD2L environment as a Web-based GPS supplemental learning program effectively supports their various learning activities. It is also true that there are many researchers who discredit studies referred to as media comparison studies (e.g., Lockee, Burton, & Cross, 1999; Russell, 1999). They argue that measuring the impact of media on learning is futile comparison studies. For example, Lockee et al. (1999) maintain that media comparison studies are badly flawed in that there is a lack of randomization in the sample selection, there is an assumption that grades actually measure

Chang S. Nam


197

student achievement, and there is no assumption of homogeneity of groups. However, the present study is not a media comparison study and does not exhibit any of these threats to internal validity. This study did not compare face-to-face/campus-based learning and distancelearning programs as mentioned in Lockee et al.’s study (1999), but compared Web-based supplementation to students’ traditional supplementation activities. Furthermore, this study used on-campus students only and compared students who were randomly assigned to one of two conditions.

As a validity check, tests were performed to ensure that the groups were

homogeneous. This study showed that there were no significant differences in participants’ initial knowledge between the Web-supplemental group and traditional group. Given these results, it would be safe to say that the present study has homogeneity and therefore internal validity. Positive perceptions on WD2L environment resources: Participants who used the WD2L environment as a Web-based GPS supplemental learning program expressed an overall positive attitude toward Web resources implemented in the environment. This finding is important from users’ perspectives, because the participants, as the study revealed, wanted to use Web materials as an addition to face-to-face teaching. To effectively support users’ learning, Web resources implemented in the WD2L environment should be easy and intuitive to use (Lohr, 2000; Plass, 1998). Given the two main findings, the Integrated Design Process was an effective framework to develop the Web-based GPS supplemental learning program. From the user interface design point of view, the main reason is that the Integrated Design Process supports usability principles by combining human-computer interface design with instructional design. In other words, user interfaces in the WD2L environment were developed to support students’ learning activities. Unfortunately, few email systems or discussion board systems provide user interfaces that can fully support engineering students’ learning activities. For example, one of the most important learning activities for engineering students was to use special characters for mathematical equations. To write the equation, α + 2δ = β, engineering students had to type the equation in texts on many existing email or discussion board systems as alpha + 2*delta = beta. On the other hand, current email and discussion Board systems on the WD2L environment supported such

Chang S. Nam


learning activities by allowing students to use special characters.

198

This was implemented,

because the Integrated Design Process dictated that user interfaces in the WD2L environment should be developed to support students’ learning activities. This is further supported in users’ positive attitudes toward user interfaces implemented in the WD2L environment, as well as improvement in the overall quality of the user interface system. Evaluation of the overall quality of the user interface system revealed that the mean rating score across 10 dimensions of the User Interface Rating Tool for Interactive Multimedia questionnaire increased from 4.9 (out of 6.0) in the first Expert Review process to 5.6 in the second Expert Review process. From a cognitive perspective, the use of symbols or special characters allows effective communication between learners, and also facilitates information processing (Driscoll, 2000; Spiro et al., 1991). Another possible explanation, from the instructional design point of view, is that the Integrated Design Process supported theory-based design for the instructional system. In order to provide an effective design of learning contents, while meeting user requirements, the Integrated Design Process supported applying learning theories as well as their instructional design principles. For example, one of the user requirements related to the instructional system was that since the GPS course involves complex forms of learning, the instructional system should provide learners with efficient information processing strategies through which they receive, organize, and retrieve knowledge in a meaningful way. The principle of Cognitivism recommended providing several different ways in which learners can connect new information with existing knowledge. By employing this design principle, for example, the “Think for a while!” section was designed. In this section, learners could think back to what they learned in previous chapters and how their prior knowledge was related to current topics. This is further supported in the early meeting of evaluation criteria for the instructional system (i.e., clarity and impact of instruction), as well as improvement in the overall quality of the instructional system evaluated by instructional design experts. Clarity and impact of instruction criteria were met in the second usability testing session. The mean rating score across three dimensions increased from 5.0 (out of 6.0) in the first Expert Review process to 6.0 in the second Expert Review process. This clearly suggests that the theory-based design of the instructional system may play an important role in developing effective learning content.

Chang S. Nam


199

Recommendations The following recommendations are provided for applications of and modifications to the proposed Integrated Design Process and Design Process Template. Design guidelines for the development of Web-based supplemental learning environments are also provided. Revised Integrated Design Process:

To improve the design process of a WD2L

environment development, based on this study, the researcher recommends a revised Integrated Design Process (Figure 5-1).

As compared with the proposed design process, the revised

framework differs in three main respects:

Conceptual Design

Low-fidelity Prototype

Design Walk-through

One-to-one Evaluation

Small Group Evaluation

Structure Design High-fidelity Prototype Page Design


Meet

2nd Expert Review

Field Experiment

tion Syst em

Information Design


Formative Evaluation

1st Expert Review

Design Scenarios Development

Requirements Specification

Development

Instr uc

Needs Analysis


Figure 5-1. Revised Integrated Design Process (IDP) for the WD2L Environment Development • Contingency plan: The proposed Integrated Design Process is only looping at the One-to-One or Small Group Evaluation Process until it meets criteria, whereas the revised framework would add feedback loops.

The revised Integrated Design Process provides

designers with a contingency plan (CP) for situations in which iterations in the One-to-One or

Chang S. Nam


200

Small Group Evaluation Process get worse instead of better (CP – A) and the 2nd expert review is worse than the first review (CP – B). When iterations in the One-to-One or Small Group Evaluation Process get worse, designers can go back to the 1st Expert Review Process in which subject matter experts (SMEs) re-evaluate what was wrong with the system prototype, including overlooked areas. If the 2nd expert review is worse than the first review, on the other hand, designers can go back to the One-to-One and/or Small Group Evaluation Process (CP – B1) or the 1st Expert Review Process (CP – B2). • Role of the 2nd expert review: Since design changes were also made after the 2nd Expert Review Process, the expert review is not just a verification of the previous results as it was in the proposed Integrated Design Process. A user profile and design review forms were provided to help the SMEs assess the WD2L environment prototype in the proposed Integrated Design Process. The revised Integrated Design Process should include an instruction to experts to consider the results of uses that were derived from One-to-One and Small Group Evaluation processes. • Formative evaluation criteria:

This study proposed a formative evaluation

2

framework for assessing the WD L environment prototype, which provided evaluation processes, instruments, and evaluation criteria to systematically evaluate and revise the instructional system and user interface system. Formative evaluation approaches have been widely used to improve instructional systems development (ISD) models (e.g., Farmer, 1989, in Reigeluth, 1999) and educational systems design (ESD) models (e.g., Khan, 1994). In the current study, the proposed formative evaluation framework was used as a portion of the evaluation of the design process employed to develop the WD2L environment prototype. The study relied on two evaluation measurements: learning performance and users’ perceptions on WD2L environment resources. For those who wish to use a formative evaluation approach to improve design theories or design process models, the study recommends using three evaluation criteria suggested by Reigeluth (1999). Table 5-1 provides definitions of the three criteria and measurements.

Chang S. Nam


201

Table 5-1. Reigeluth’s (1999) Three Criteria for Evaluating Design Process Models Criteria Effectiveness

Efficiency

Appeal

Definitions Degree to which the application of the model attained the goal in a given situation Bang (effectiveness) for the buck (cost, either in money or time, or some other cost, or a combination of costs) How enjoyable the resulting designs for all people associated with them

Measurements • Learning performance • Breadth of contexts (or situations) that attains the goal • Human time, effort, and energy required • Cost of further resources needed (e.g., materials, equipments, etc.) • Feedback from teachers, students, support personnel, and administrators

The two evaluation measurements employed in the present study, students’ learning performance and their perceptions on WD2L environment resources, pertain to the effectiveness and appeal criteria from Reigeluth (1999), respectively. For an intensive evaluation, this study recommends employing all three evaluation criteria and measurements. Design Process Template: This study offered the Design Process Template to help implement each step of the design process, as shown in Figure 3-2. There were two main reasons for providing the Design Process Template.

First, information and developmental

factors that needed to be considered are not constant because of changes in technology, course structure, and users’ needs, for example. Another reason was for the template to provide factors that should be considered in each design process, such as process objectives, inputs, design steps, outputs, methods and tools. Although it is not intended to be exhaustive, the Design Process Template will be able to help address such issues when developing the WD2L environment prototype. For an effective use of the Design Process Template, the researcher recommends one more dimension to consider: check points representing factors or issues that should be checked for a successful implementation of the process. Table 5-2 summarized checkpoints for each design process.

Chang S. Nam


202

Table 5-2. Check Points for Design Processes Phase Needs Analysis

Process Requirements Specification



Design

Design Scenarios Development Information Design Structure Design Page Design

Development

Low-fidelity Prototype Design Walk-through High-fidelity Prototype

Check Points • An early focus on users and their tasks • Understanding possible interactions of users with the WD2L environment • Concerned with technological innovation • WD2L environment features determined should meet user’s needs. • Corresponding components should support users’ performance on the WD2L environment. • Determination of features and their components should take advantage of the current technology available • Design goals and principles should be able to make sure - Usable as well as learner-centered user interface and instructional system - Interactions of users with the WD2L environment • Identification of key user tasks • Detailed design scenarios • Extensive audit of information contents • Theory-based design • Key user paths • Match of the presentation structure with the storage structure • Content layout displaying navigation and the layout of the elements • Determination of navigational organization • Integration of design ideas • Development of the low-fi prototype reflecting users’ needs • Make sure that the proposed design of the WD2L environment is consistent with the target user's expectations and skill level early in the design process • Make sure that all functionalities were implemented as users wanted

Chang S. Nam


203

Design Guidelines for Web-based Supplemental Learning Environments: Although many interface design principles and guidelines have been proposed, few design guidelines exist for the development of Web-based supplemental learning environments. The following guidelines are based on the results of the present study. The guidelines are not intended to be exhaustive; they should be used in combination of existing user interface design and instructional design guidelines. Table 5-3 shows general design guidelines for the development of Web-based supplemental learning environments Table 5-3. General Design Guidelines for Web-Based Supplemental Learning Environments Guideline Acquire application domain knowledge

Identifying theoretical position(s) to learning theories relevant for designing the learning content Identify relevant instructional design model Focus on the design of a ‘usable’ interface that is also ‘learning-centered’ Evaluate both the instructional system and user interface system Identify user activities Determine key features and components of the WD2L environment

Description Familiarize yourself with the domain in which the WD2L system will be operating and identify what the system is supposed to be doing in that domain. However, you do not have to be a “domain expert.” There is convincing empirical evidence supporting that domain knowledge is essential to software design and implementation It is important to understand how people learn and to incorporate that knowledge when developing the instructional system in terms of the learning content and instructional interventions. Which theoretical position to take significantly affects the design of the instructional system. Review commonly used instructional design models and principles and identify an instructional design model that can be used for designing the overall instructional interventions such as instructional strategies. For a Web-based supplemental learning environment to be successful, it is important to effectively facilitate interactions of learners with the learning environment. A human-computer interface that is easy to use and is intuitive to anyone can fulfill that requirement. Always consider how effectively user interface system supports users’ learning activities. Try to identify problem areas and draw inferences about the overall quality of Web-based supplemental learning environments by collecting continuous feedback from users. User tasks identified help assess how users would interact with the environment and know how interface design requirements should be specified to support users’ activities on the web-based supplemental learning environment. A well-designed web-based instruction (WBI) program should provide various features conducive to learning, instructions, and the use of the Web-based supplemental learning environment. Identify such features and corresponding components that constituted an effective WD2L environment.

Chang S. Nam


204

In addition to general design guidelines, more specific guidelines for the design of learning content, practice sessions, and prelaboratory activities are also recommended. These are critical components that constitute an effective WD2L environment. Table 5-4. Design Guidelines for Web-Based Supplemental Learning Components Component Learning Content

Guideline Use animation and graphic images for efficient learning Offer external resources Provide concept map Allow accessing more detailed information Provide Think for a while section

Practice Sessions

Use various types of questions

Provide informative feedback Pre-laboratory Activities

Provide opportunities for gaining hands-on experience

Description Many studies have shown that learning conditions using animation plus text result in more efficient learning, compared to text only and text plus graphics. Providing links to material outside the course will be beneficial to the learning experience. The Concept Map is intended to help students review what they already know again and link new material with what is currently known as a way of providing mental scaffolding for new material. Learning content should provide students with a place where they can see more detailed explanation on certain concept. For example, a pop-up screen describing tidal effects in more detail is displayed when the user clicks it. The “Think for a while” section, is designed to allow students to reflect on what they have learned in the previous chapter(s) and how their prior knowledge can be related to the current topics. Practice sessions are provided in which students can practice what they have learned. The intended purpose is to explicitly teach problem-solving strategies because students cannot use appropriate strategies merely by looking at the materials. Various types of questions were used for practice pages, such as a true and false, multiple choice, short essay, and step-by-step procedure. Practice sessions should provide students with informative feedback. According to Cognitive learning theory, students use their errors as a source of information about their mental models. One of the most important learning activities in engineering education is to train students to actively integrate many fundamental concepts by gaining hands on experience with modern technology. Make sure students can have hands-on experience with modern technology at their own pace in their own time.

Chang S. Nam


205

Limitations of the Study The present study contains five major limitations that may hinder the researcher’s ability to generalize results. WD2L Environment Prototype: In the study, the WD2L environment prototype was developed focusing only on one chapter (i.e., chapter 5) of one GPS course for the student user group. Replication of the findings using a fully developed WD2L environment for other user groups (e.g., the instructor and system administrator) and subject matter domains is needed before strong conclusions are warranted. Traditional Activities for Supplemental Learning:

The present study identified

students’ traditional activities for supplemental learning - reading a book, questioning the instructor, and discussing with classmates – in an informal way (e.g., through conversation with a teaching assistant and students). Had the study identified more information about students’ traditional activities for supplemental learning, subjective ratings of the traditional to the Webbased supplemental learning could have been compared. Summative Evaluation:

The evaluation activity takes place either formatively or

summatively (Rubin, 1994). This study focused only on the formative evaluation of the Webbased supplemental learning environment, because the evaluation in Web-based learning environments is a continuing process throughout the development lifecycle (Belanger & Jordan, 2000). A summative evaluation is also needed to fully investigate the effectiveness of the program. External Validity: It is often more valid to evaluate learning and instructional design using action-research methods even during the formative evaluation stage. The external validity of this study could have been enhanced by implementing portions of the prototype in the actual learning environment and, in parallel, conducting formative evaluations. Given the time-cycle of the actual course used in this study, it was difficult to synchronize the research and classroom schedules to apply an action-research approach.

Chang S. Nam


206

Domain-Specific Framework: Given that usability engineering and instructional design are both emerging specialty areas, the integrated framework is constrained by the knowledge domain. Thus, it is expected that the framework that has emerged from this study will require updating in the future on the basis of new theories and empirical evidence relevant to usability and instructional design.

Areas of Future Research WD2L Environment across Disciplines: The WD2L environment prototype developed in this study focused only on GPS content and it was an example of a certain type of instructional content (i.e., engineering) that could be generalized. Future research should be conducted to investigate how WD2L environments should be developed in order to effectively support students’ supplemental learning activities across disciplines. Applications in m-Learning: Andrew (1999) maintains that the future of the Web-based learning (WBL) lies in “m-learning (mobile learning) where the WBL material is presented on a handheld device,” such as a Personal Digital Assistant (PDA) and a mobile phone (p. 486). Further research can be carried out to investigate how Web-based supplemental learning materials should or could be presented on handheld devices. WD2L Environment for Different Learning Styles: Further research that investigates whether there are any differences in learners’ performance with different learning styles in different learning environments would provide a firm foundation for an successful development of the WD2L environment. The research can be conducted to identify which types of learning styles can have benefits most from different types of supplemental learning environments. When experiencing a learning environment at variance with their learning styles, students tend to reject that learning situation (Kolb, 1976). Workload of Web-Supplemental Learning Students: Future studies should focus on the workload of students who use a WD2L environment as their supplemental learning program, to

Chang S. Nam


207

investigate whether it is more difficult for a Web-supplemental learning group to use tools (i.e., Web resources) than for a traditional group to just use their own methods (e.g., reading a textbook or class notes). Effective Use of Design Guidelines: For an effective design of user interfaces, designers and developers have used design guidelines as a method to improve interface usability (Vanderdonckt, 1999). It is clear that recommendations found in design guidelines provide user interface designers with beneficial and valuable knowledge (Lowgren & Lauren, 1993). Unfortunately, there are also indications that these guidelines have problems to be used in practice. For example, many design guidelines, available in the form of documents, are still hard to access, interpret, and contextualize (de Souza & Bevan, 1990). Various attempts can be made to address these issues. First, Web-based or computer-based design guidelines supporting tools can be developed to assist designers in effectively using a set of guidelines. The computer-based supporting tools have a lot of advantages over paper-based guidelines (Vanderdonckt, 1999). Examples include an immediate update and provision of a variety of supporting functions.

In addition, there has been an effort to identify and organize a set of design

guidelines that match users’ (i.e., interface designers) perceptions of relationships between guidelines (e.g., Nam, Kim, Smith-Jackson, & Nussbaum, 2003). In their study on the use of design guidelines of the cell phone interface, Nam et al. (2003) found that a set of design guidelines organized based on closer fit to users’ mental model may lead to a better use of guidelines in terms of ease-of-access and understanding. Usually, guidelines are drawn and organized by research based upon the organizer’s experience, but the resulting organization may not be the same as users’ perceptions of relationships between guidelines. Future studies can be conducted to develop advanced methods, which can provide designers with design guidelines that match their perceptions of relationships between guidelines.

Chang S. Nam


208

Conclusions The first goal of this study was to offer a systematic approach to the development of a user-centered, WD2L environment for supporting engineering courses. The implementation of Web-based learning environments involved the consideration of various aspects of the learning environment, such as application domain knowledge (i.e., target subject field), conceptual learning theory, instructional design, human-computer interface design, and evaluation about the overall quality of the learning environment.

The Integrated Design Process (IDP), which

integrated the human-computer interface design with instructional design, helped simplify developmental challenges while improving the design process of the WD2L environment. The second goal of the study was to develop the WD2L environment based on the proposed design process framework to support learning activities in an engineering field. The study showed that through the WD2L environment engineering students can easily pull together many fundamental concepts and effectively gain hands-on experience with the GPS technology at their own pace in their own time. Finally, this study evaluated the design process model by assessing the overall quality of the WD2L environment prototype developed using the proposed IDP.

It was

2

confirmed that the WD L environment as a Web-based GPS supplemental learning program could work as well as traditional supplemental learning by having positive impacts on students’ learning and perceptions of WD2L environment resources. The Integrated Design Process and Design Process Template can be generalized to courses in other engineering disciplines that need a Web-based supplemental learning environment to support students’ various learning activities. It can be further noted that there are implications for usability studies for educational software.

Since concerns for usability have not been truly addressed when designing and

developing educational software, more usability studies should be conducted (Levi & Conrad, 2000; Pavlik, 2000). Learners in the WD2L environment must be able to easily focus on learning materials without having to make an effort to figure out how to access them (Lohr, 2000). The findings of the study confirmed that the user interface system that supports students’ learning activities can fulfill that requirement.

209

REFERENCES Abrami, P. C., & Bures, E. M. (1996). Computer-supported collaborative learning and distance education. The American Journal of Distance Education, 10(2), 37-42. Adler, P. S., & Winograd, T. A. (1992). Usability: Turning Technologies into Tools. New York: Oxford University Press. Aggarwal, A., & Brento, R. (2000). Web-based education. In A. Aggarwal (Ed.), Web-based learning and teaching technologies: Opportunities and challenges (pp. 2-16). Hershey, PA: Idea Group Publishing. Andrew, M. (2003). Should we be using Web-based learning to supplement face-to-face teaching of undergraduates? In Proceedings of the 6th International Conference on ComputerBased Learning in Science, (478-488). Cyprus. Baek, Y., & Layne, B. (1988). Color, graphics and animation in a computer assisted learning tutorial lesson. Journal of Computer Based Instruction, 15, 131-135. Bannan, B., & Milheim, W. D. (1997). Existing Web-based instruction courses and their design. In B. H. Khan (1997) (Ed.), Web-based instruction (pp. 381-387). Englewood Cliffs, NJ: Educational Technology Publications. Barnard, P. (1991). Bridging between basic theories and the artifacts of human-computer interaction. In J. M. Carroll (Ed.), Designing interaction: Psychology at the humancomputer interface (pp. 103-127). Cambridge University Press. Bednar, A. K., Cunningham, D., Duffy, T. M., & Perry, J. P. (1995). Theory into practice: How do we link? In G. J. Anglin (Ed.), Instructional technology: Past, present, and future. (2nd ed., pp. 100-111), Englewood, CO: Libraries Limited, Inc. Belanger, F., & Jordan, D. H. (2000). Evaluation and implementation of distance learning: technologies, tools and techniques. Hershey, PA: Idea Group Publishing. Berry, D. C., & Broadbent, D. E. (1990). The role of instruction and verbalization in improving performance on complex search tasks. Behaviour & Information Technology, 9, 175-190. Bias, R. (1994). The pluralistic usability walkthrough: Coordinated Empathies, In J. Nielsen & R. Mack (Eds.), Usability inspection methods (pp. 63-76), John Wiley. Boling, E., & Sousa, G. A. (1993). Interface design issues in the future of business training. Business Horizon, November-December, 50-54.

Chang S. Nam

References

210

Borges, J. A., Morales, I., & Rodriguez, N. J. (1998). Guidelines for designing usable world wide web pages. Conference companion of the computer-human interaction conference, (277278). Vancouver, BC, Canada: Association for Computing and Machinery. Brewster, S. A. & Crease, M. G. (1999) Correcting menu usability problems with sound. Behaviour and Information Technology, 18, 165-177. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18, 32-42. Bruder, I. (1989). Distance learning: What’s holding back this boundless delivery system? Electronic Learning, 8, 30-35. Buxton, W., Lamb, M. R., Sherman, D., & Smith, K. C. (1983). Towards a comprehensive user interface management system. Computer Graphics, 17, 35- 42. Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental design for research. Chicago: Rand McNally. Card, S. K., Moran, T. P., & Newell, A. (1983). The psychology of human-computer interaction. Lawrence Erlbaum Associates. Chadwick, S. A. (1999). Teaching virtually via the Web: Comparing student performance and attitudes about communication in lecture, virtual Web-based, and Web-supplemented courses. The Electronic Journal of Communication, 9, 1-13. Cockton, G. (1988). Generative transition networks: A new communications control abstraction. In D. M. Jones, & R. Winder. (Eds). People and computers IV (pp. 509-525), Cambridge University Press. Cognition and Technology Group at Vanderbilt, (1992). The Jasper series as an example of anchored instruction: Theory, program description, and assessment data. Educational Psychologist, 27, 291-315. Damodaran, L., Ip, K., & Beck, M. (1988). Integrating human factors principles into structured design methodology: A case study in the U. K. civil service. In H. J. Bullinger, (Ed.), Information technology for organizational systems (pp. 235-241). Elsevier. Davidson, K. (1998). Education in the internet--linking theory to reality. Retrieved October 3, 2002, from http://www.oise.on.ca/~kdavidson/cons.html Dayton, T. (1991). Cultivated eclecticism as the normative approach to design. In J. Karat (Ed.), Taking software design seriously. Academic Press (pp. 21-44).

Chang S. Nam

References

211

de Souza, F., & Bevan, N. (1990). The use of guidelines in menu interface design: Evaluation of a draft standard. In D. Draper, D. Gilmore, G. Cockton, & B. Shackel (Eds). Humancomputer interaction-interact ’90 (pp. 435-440). North-Holland. Desurvire, H. W. (1994). Faster, chapter!! Are usability inspection methods as effective as empirical testing? In J. Nielsen & R. L. mack (Eds.), Usability inspection methods (pp. 173-202). New York, NY: John Wiley & Sons. Dick, W., & Carey, L. (1996). The systematic design of instruction. New York, NY: Harper Collins Publication. Driscoll, M. P. (2000). Psychology of learning for instruction (2nd Ed.). Needham Heights, Massachusetts: Allyn & Bacon. Duffy, T. M., & Jonassen, D. H. (1992). Constructivism and the technology of instruction: A conversation. Hillsdale, NJ: Lawrence Erlbaum Associates. Ellis, C. A., Gibbs, S. J., & Rein, G. L. (1993). Groupware: Some issues and experiences. In R. M. Baecker (Ed.), Readings in groupware and computer-supported cooperative work: Assisting human-human collaboration (pp. 9-28). San Mateo, CA: Morgan Kaufman Publishers. Ertmer, P. A., & Newby, T. J. (1993). Behaviorism, cognitivism, constructivism: Comparing critical features from an instructional design perspective. Performance Improvement Quarterly, 6, 50-72. Felix, U. (1998). Evaluation of Web-based language learning program. Retrieved September 5, 2003, from http://www.arts.monash.edu.au/lc/sill/evalqst.htm Gagne, R. M. (1985). The conditions of learning (4th ed.). New York, NY: Holt, Rinehart & Winston. Gagn'e, R. M., Briggs, L. J., & Walter, W. W. (1992). Principles of Instructional Design (4th edition). Harcourt, Brace, Jovanovich, New York, USA. Goldstein, I. L., & Ford, J. K. (2002). Training in organizations. (5th Ed.). Wadsworth. Good, T. L., & Brophy, J. E. (1990). Educational psychology: A realistic approach. (4th ed.). White Plains, NY: Longman. Gould, J. D., & Lewis, C. (1985). Designing for usability: Key principles and what designers think. Communications of the ACM, 2, 300-311. Greenbaum, J., & Kyng, M. (1991). Design at Work: Cooperative Design of Computer Systems. Hillsdale, NJ: Lawrence Erlbaum Associates.

Chang S. Nam

References

212

Hannafin, M., Land, S., & Oliver, K. (1999). Open learning environments: Foundations, methods, and models. In C. Reigeluth (Ed.), Instructional Design Theories and Models (pp. 115-140). Mahwah, NJ: Lawrence Erlbaum Associates. Henke, H. A. (1997). Evaluating Web-Based Instruction Design. Retrieved September 5, 2001, from http://scis.nova.edu/~henkeh/story1.htm Hix, D., & Hartson, H. R. (1993). Developing user interfaces: Ensuring usability through product process. Wiley Professional Computing. Home, J. (1998). The usability methods toolbox. Retrieved September 15, 2002, from http://jthom.best.vwh.net/usability/usahome.htm Honebein, P. C., Duffy, T. M., & Fishman, B. J. (1993). Constructivism and the design of learning environments: Context and authentic activities for learning. In T. M. Duffy, J. Lowyck, & D. H. Jonassen (Eds.), Designing environment for constructive learning (pp. 87-108). Heidelburg, FRG: Springer-Verlag. IBM (2003). User-Centered Design principles. Retrieved March 5, 2002, 3.ibm.com/ibm/easy/eou_ext.nsf/Publish/13

from

http://www-

International Organization for Standardization (1998). ISO 9241-11. Ergonomic requirements for office work with visual display terminals (VTDs) – Part 11: Guidance on Usability. Jeffries, R., Miller, J. R., Wharton, C., & Uyeda, K. M. (1991). User interface evaluation in the real world: A comparison of four methods. In Proceedings of ACM CHI ’90 Conference, (261-266). Jonnassen, D. H. (1991). Objectivist vs. constructivist: Do we need a new philosophical paradigm? Educational Technology Research and Development, 39, 5-14. Jonassen, D. H. (1999). Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory (Vol. II, pp. 215-239). Mahwah, NJ: Lawrence Erlbaum Associates. Jonassen, D. H., McAleese, T. M. R., & Duffy, T. M. (1993). A Manifesto for a constructivist approach to technology in higher education. In T. M. Duffy, J. Lowyck, & D. H. Jonassen (Eds.), The design of constructivistic learning environments: Implications for instructional design and the use of technology. Heidelburg, FRG: Springer-Verlag. Jonassen, D. H., & Rohrer-Murphy, L. (1999). Activity theory as a framework for design constructivist learning environment. Educational Technology Research and Development, 47, 61-79. Jones, M. G., Farquhar, J. D., & Surry, D. W. (1995). Using metacognitive theories to design user interface for computer-based learning. Educational Technology, July-August, 12-22.

Chang S. Nam

References

213

Keller, J. M. (1987). The systematic process of motivational design. Performance & Instruction, 29, 1-8. Khan, B. H. (1994). Post facto formative research on the educational systems design (ESD) process. Unpublished dissertation, Indiana University Graduate School, Bloomington, IN. Khan, B. H. (1997) (Editor). Web-based instruction. Educational Technology Publications. Englewood Cliffs: NJ. Kieras, D. E., & Polson, P. G. (1985). An approach to the formal analysis of user complexity. International Journal of Man-Machine Studies, 22, 365-394. Kirkpatrick, D. L. (1994). Education training programs: The four levels. San Francisco: BerrettKohler. Kolb, D. A. (1976). Learning style inventory: Technical manual. Massachusetts, Institute for Development Research. Koto, K., & Cotler, E. (2002). Web Redesign: Workflow That Works. New Riders. Land, S. M., & Hannafin, M. (1996). A conceptual framework for the development of theoriesin-action with open-ended learning environments. Educational Technology Research and Development, 44, 37-53. Landauer, T. K. (1991). Let’s get real: A position paper on the role of cognitive psychology in the design of humanity useful and usable systems. In J. M. Carroll (Ed.), Designing interaction: Psychology at the human-computer interface (pp. 60-73). Cambridge University Press. Laurel, B. (ed)(1990). The art of human-computer interface. Addison-Wesley. Levi, M. D., & Conrad, F. G. (2000). Usability testing of World Wide Web. Retrieved March 5, 2003, from http://stats.bls.gov/ore/htm_papers/st960150.htm Lockee, B. B., Burton, J. K., & Cross, L. H. (1999). No comparison: Distance education finds a new use for “no significant difference.” Educational Technology Research & Development, 7, 33-42. Lohr, L. L. (2000). Designing the instructional interface. Computers in Human Behavior, 16, 161-182. Lowgren, J., & Lauren, U. (1993). Supporting the use of guidelines sand style guides in professional user interface design. Interacting With Computers, 5(4), 385-396. Nam, C. S., Kim, H. N., Smith-Jackson, T. L., & Nussbaum, M. A. (2003). Development of a guidelines tool for mobile phone interfaces. In Proceedings of the Human Factors and

Chang S. Nam

References

214

Ergonomics Society’s 47th Annual Meeting, (pp. 792-796). Santa Monica, CA: Human Factors and Ergonomics Society. Marshall, V., & Schriver, R. (1994). Using evaluation to improve performance. Technical and Skills Training, January, 6-9. Mayer, R. E. (2001). Multimedia learning. New York, Cambridge University Press. Mayer, R. E., & Chandler, P. (2001). When learning is just a click away: Does simple user interaction foster deeper understanding of multimedia messages? Journal of Educational Psychology, 93, 390-397. Mayhew, D. J. (1999). The usability engineering lifecycle: A practitioner's handbook for user interface design. San Francisco, CA: Morgan Kaufmann Publishers, Inc. Mayton, G. B (1991). Learning dynamic processes from animated visual s in microcomputer based instruction. In Proceedings of the Annual Meeting of the Association for Educational Communications and Technology, (13).. McCormack, C., & Jones, D. (1997). Building a Web-based education system. Wiley Computer Publishing. New York: NY McFeeters, F. E. (2003). The effects of individualism Vs. collectivism on learner’s recall, transfer and attitudes toward collaboration and individualized learning. Unpublished dissertation, Virginia Polytechnic Institute and State University, Blacksburg, VA. Moallem, M. (2001). Applying constructivist and objectivist learning theories in the design of a Web-based course: Implications for practice. Educational Technology & Society, 4, 113125. Nielsen, J. (1993). Usability engineering. New York, NY: Academic Press. Norman, D. A. (1987). Design principles of human-computer interfaces. In R. M. Baeker & W. A. S. Buxton (Eds.), Readings in human-computer interaction: A multidisciplinary approach. Morgan Kaufman (pp. 492-501). Norman, K. L. (2000). Desktop distance education: Personal hosting of Web courses. In A. Aggarwal (Ed.), Web-based learning and teaching technologies: Opportunities and challenges (pp. 117-134). Hershey, PA: Idea Group Publishing. Oliver, K. (2003). Effective Uses of Online Course Tools. Retrieved April , http://www.edtech.vt.edu/edtech/id/ocs/

2003,

from

Pavlik, P. (2000). Collaboration, Sharing and Society – Teaching, Learning and Technical Considerations from An Analysis of WebCT, BSCW, and Blackboard. Retrieved September 25, 2002, from,

Chang S. Nam

References

215

http://members.fortunecity.com/pgp5/Collaboration.Learning.and.Society.htm Perkins, D. (1992). Smart schools: Better thinking and learning for every child. NY, NY: The Free Press. Plass, J. L. (1998). Design and evaluation of the user interface of foreign language multimedia software: A cognitive approach. Language Learning & Technology, 2, 35-45. Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 741-773. Reeves, T. C., & Harmon, S. W. (1993). User Interface Rating Tool for Interactive Multimedia. Retrieved March 5, 2001, from http://mime1.marc.gatech.edu/MM_Tools/UIRF.html Reigeluth, C. M. (1996). A new paradigm of ISD? Educational Technology, 36, 13-20. Reigeluth, C. M., & Frick, T. W. (1999). Formative research: A methodology for creating and improving design theories. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory (Vol. 2, pp. 633-651). Hillsdale, NJ: Lawrence Erlbaum Associates. Rieber, L. & Boyce, M. (1990). The effects of computer animation on adult learning and retrieval tasks. Journal of Computer Based Instruction, 17, 46-52. Rizzuti, K., & Dickinson, J. (2000). Satisfying the experienced on-line shopper: Global eshopping survey. A. T. Kearney Consulting Firm. Retrieved November 5, 2002, from www.atkearney.com Rosson, M. B., & Carroll, J. M. (2002). Usability Engineering Scenario-Based Development of Human-Computer Interaction. Morgan Kaufmann Publishers Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. New York, NY: John Wiley & Sons. Rubinstein, R., & Hersh, H. (1984). The human factor: Designing computer systems for people. Digital Press. Russell, T. L. (1999). The no significant difference phenomenon. Chapel Hill, NC: Office of Instructional Telecommunications, North Carolina State University. Saettler, P. (1990). The evolution of American educational technology. Englewood, CO: Libraries Unlimited, Inc. Saltzberg, S., & Polyson, S. (1995). Distributed learning on the Web. Syllabus, 9, 10-12.

Chang S. Nam

References

216

Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist framework. Educational Technology, 35, 31-38. Schank, R. C. & Cleary, C., (1995). Engines for Education. Lawrence Erlbaum Associates, Publishers, Hillsdale, New Jersey. Schiffman, S. S. (1995). Instructional systems design: Five views of the field. In G. J. Anglin Ed.), Instructional technology: Past, present and future. (2nd ed., pp. 102-116). Englewood, CO: Libraries Unlimited, Inc. Schlosser, C. A., & Anderson, M. L. (1994). Distance education: Review of the literature. ashington, DC: Association for Educational Communications and Technology. Schuman, L. (1996). Perspectives on instruction. Retrieved March 25, 2002, http://edweb.sdsu.edu/courses/edtec540/Perspectives/Perspectives.html

from

Schwier, R. A. (1995). Issues in emerging interactive technologies. In G. J. Anglin (Ed.). Instructional Technology: Past, present, and future (2nd Ed., pp. 119-127), Englewood, CO: Libraries Unlimited. Shepherd, M. M., & Martz, W. B. (2001). Group Processes in Conflict: he Impact of Communication Channels on Consensus Building. In Proceedings of the 34th Hawaii International Conference on System Sciences, (1-6). Maui, Hawaii. Shneiderman, B. (1993). Designing the user interface: Strategies for effective human-computer interaction (2nd ed.). Reading, MA: Addison-Wesley. Shon, M. (1996). Formative research on an instructional theory for the design of computer-based simulations for teaching causal principles. Unpublished dissertation, Indiana University Graduate School, Bloomington, IN. Slaughter, L., Norman, K., Shneiderman, B. (1995), Assessing users’ subjective satisfaction with the information system for youth services (ISYS). In Proceedings of Third Annual MidAtlantic Human Facotrs Conference, (164-170). Blacksburg, VA. Smith, B. L., & MacGregor, J. T. (1992). What is collaborative learning? In A. S. Goodsell (Ed.), Collaborative learning: A sourcebook for higher education (pp. 9-22). University Park, PA: Pennsylvania State University, National Center on Post Secondary Teaching Learning and Assessment (NCTLA). Spiro, R. J., & Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1991). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31, 24-33. Streitz, N. A. (1987). Cognitive compatibility as a central issue in human-computer interaction: Theoretical framework and empirical findings. In G. Salvendy (Ed.),

Chang S. Nam

References

217

Cognitive Engineering in the Design of Human-Computer Interaction and Expert Systems: Proceedings of the Second International Conference on Human-Computer interaction (Volume II, pp. 75-82), Honolulu, Hawaii, Amsterdam: Elsevier Science. Sutcliffe, A. (1988). Some experiences in integrating specification of human-computer interaction with a structured system development method. In D. M. Jones, & R. Winder. (Eds.), People and computers IV (pp. 146-160). Cambridge University Press. Sutcliffe, A. (1989). Task analysis, system analysis and design: Symbiosis or synthesis. Interacting with computers, 1, 6-12. Fufte, E. R. (1990). Envisioning information. Cheshire, Connecticut : Graphics Press. Vanderdonckt, J. (1999). Development milestones towards a tool for working with guidelines. Interacting with Computers, 12, 81-118. Van Slyke, Craig; Kittner, Marcy, and, Belanger, France (1998). Identifying Candidates for Distance Education: A Telecommuting Perspective, In Proceedings of the America's Conference on Information Systems, (666-668). Vora, P., & Helander, M. (1995). A teaching method as an alternative to the concurrent thinkaloud method for usablity testing, In Y. Anzai, K. Ogawa & H. Mori (Eds.), Symbiosis of human and artifact (pp.375-380). Wallace, M. D., & Andersen, T. J. (1993). Approaches to interface design. Interacting with computers, 5, 259-278. Wasserman, A. I. (1985). Extending state transition diagrams for the specification of human-computer interaction. IEEE transaction software engineering, 11, 699-713. Waterworth, J. A. (1992). Multimedia interaction with computers: Human factors issues. Chichester, England: Ellis Horwood. Wickens, C. D. (1992). Engineering Psychology and Human Performance (2nd ed.). NY: Harper Collins. Winer, L. R., Chomienne, M., & Vázquez-Abad, J. (2000). A distributed collaborative science learning laboratory on the Internet. The American Journal of Distance Education, 14, 47-62. Wroblewski, D. A. (1991). The construction of human-computer interfaces considered as a craft. In J. Karat (Ed.). Taking software design seriously (pp. 1-9), Academic Press. Zhang, Z., Basil, V., Shneiderman, B. (1998). An empirical study of perspective-based usability inspection. In Proceedings of the Human factors and Ergonomics Society 42nd Annual Meeting, (1346-1350). Santa Monica, CA: Human Factors Society.

Chang S. Nam

Appendices

218

Appendix 1: A Summary of Semi-Structured Interviews Part A: A Summary of Semi-Structured Interview I Interviewee: Department: University: Date:

Wayne A. Scale, Ph.D Bradley Department of Electrical and Computer Engineering Virginia Polytechnic Institute and State University Fall 2001

1. Would you briefly describe your GPS course? The Bradley Department of Electrical and Computer Engineering at VA Tech currently offers a one semester, advanced undergraduate/beginning graduate, four credit hour, capstone design course, Introduction to GPS Theory and Design. The course has three hours of lecture each week as well as three hours of laboratory work. 2. Are there any prerequisites to take the course? ECE 3106 (Radiowave propogation), AOE 4134 (Astrodynamics) or consent. The primary electrical engineering concepts involved in this class are 1) radiowave propogation (ECE 3106), and 2) communication system theory (ECE 3614). Satellite orbit dynamics is the third important aspect of this course. These three core components of the class will be taught from the ground up to accommodate students with diverse backgrounds. Experience with MatLab will also be useful. 3. What are the course objectives? To provide the students with fundamental theory and concepts of the Global Positioning System and to demonstrate these concepts through laboratory exercises and a final design project. 4. How many students are usually taking your course? Is there any department limits to enrollments? And who are they? Currently, the course is attended by 30-35 students (the department limits enrollment to 35 students). Considering that the course is new and the applications of GPS will continue to rapidly grow, this number is expected to increase significantly over the next few years and additional sections will have to be added. The course is composed of roughly 75% electrical engineering students and 25% aerospace engineering students. Of the total number of students, roughly 30% are graduate students and 70% are undergraduate students. 5. What topics do you cover in your course? Lecture topics for the first semester course include • • • • • • • • •

Coordinate and Time Systems Satellite Orbit Theory Notions of Range and PseudoRange Navigation Solution and Solution Techniques Fundamentals of GPS Signal Theory Navigation Solution Error Analysis Concepts and Theory of Dilution of Precision DOP Atmospheric and Propagation Effects on GPS signals GPS Antennas and Communication System Concepts

Chang S. Nam

Appendices

219

6. What kinds of problems with the lab do you think you have? Lecture material must be designed carefully to incorporate significant engineering science for an accredited course. Even though there are a number of excellent reference books on GPS, currently no student-oriented textbooks exist. Most universities currently rely on an internal set of lecture notes supplemented with reference books. It is expected in the near future that GPS textbooks will begin to appear on the market. 7. What do you think is the recent trend in GPS education? Because of the rapid growth in applications for the Global Positioning System GPS over the next five to ten years, the demand for engineers educated in GPS fundamentals and techniques will grow rapidly as well. To insure that their graduates stay abreast of this powerful technology, engineering colleges and departments must meet the challenge to develop GPS courses that will integrate into their curricula. 8. To effectively fit into an accredited engineering curriculum, what features do you think any courses in GPS must emphasize? To fit effectively into an accredited engineering curriculum, a course in GPS must emphasize both fundamental engineering science and as well as engineering design criteria. 9. What is your grading policy? The course grade will be determined form laboratory assignments, a comprehensive final exam and a course project as follows: Laboratory Assignments (50%), Final Exam (25%), and Semester Project (25%). 10. Would you describe a teaching laboratory? And what is the main objective to provide students with the teaching laboratory? A teaching laboratory is an essential for GPS education. We also believe that our teaching laboratory is the strong point of our GPS course. Our teaching laboratory at VA Tech is currently equipped with 7 laboratory stations. Each lab station consists of a Mitel GPS receiver development kit installed in a Gateway Personal Computer. (This same hardware setup is used by a number of research laboratories across the country.) Real-time GPS data is currently taken from a GPS antenna installed on the roof of the Electrical Engineering Building (Whittemore Hall) which is connected to each lab station. We have just ordered a Spirent GSS 4500 12 Channel GPS signal simulator that will be also be used to provide data for the class starting Fall 2001 semester. Laboratory assignments involve the student taking real-time GPS data. The Mitel development kits provide the GPS data in the universal RINEX format. The students then develop MATLAB software to calculate satellite positions, velocity, navigation solutions, navigation solution errors, DOP, ect., from the RINEX data using theory learned in class. They then compare their results to the results calculated by the MITEL receiver using its internal software. MATLAB scripts are also developed to simulate fundamental GPS signal concepts including pseudorandom noise PRN codes as well as the Spread Spectrum and Code Division Multiple Access CDMA signal techniques used by GPS. 11. What would say is a unique opportunity that any GPS course provides to students? Because of the interdisciplinary nature of GPS, an excellent advanced undergraduate/beginning graduate two-course sequence can be developed that will provide students with a unique opportunity to pull together many fundamental concepts they have learned over four years of engineering education as well as gain hands on experience with an important modern technology. 12. To master GPS fundamentals, what knowledge do you think a student must utilize? To master GPS fundamentals, a student must utilize knowledge in a wide spectrum of engineering areas including, satellite orbit theory, radio-wave propagation, communication system theory, signal processing, probability and statistics, and numerical analysis techniques.

Chang S. Nam

Appendices

220

13. Did you have any difficulty teaching your GPS course? If so, what was that? And do you have an opinion on why students had such problems? It is not a trivial task to teach such a broad spectrum of topics to engineering students. Also, most students will typically not have exposure to all of the topics that need to be covered. An emphasis on fundamentals must therefore be stressed in developing a first course. Students with diverse backgrounds will benefit from this approach. 14. To facilitate a student’s learning, what do you think a GPS course must provide? Engineers must also have hands on experience with new technology. Therefore, a well-equipped laboratory is crucial for any university GPS course. The laboratory will allow students to reinforce theory that they have learned during the lectures as well as pursue their own interests and design problems. This laboratory must at a minimum have a way of providing a GPS signal with an antenna system and/or a GPS signal simulator (preferably) and GPS receivers that will allow students to collect and analyze GPS data. The first of a two-course sequence should emphasize more fundamental engineering science concepts of GPS. This is a great benefit to the students because GPS is a perfect example of a real world engineering system that was developed using a broad spectrum of engineering concepts. A second follow-on advanced course will allow students to learn more specific techniques of signal processing and/or aircraft/spacecraft navigation issues.

Chang S. Nam

Appendices

221

Part B: A Summary of Semi-Structured Interview II Interviewee: Department: University: Date:

Dewayne R. Brown, Ph.D Electronics and Computer Technology North Carolina A&T State University Fall 2001

1. Would you briefly describe your GPS course? The name of the course is Wireless Geo-location Systems [ECT 665]. This course will provide practical knowledge of the operation of position location systems such as the Global Positioning System. This course will include integrated practical examples, in-depth case studies and guidelines for building GPS systems. The course will also discuss alternative implementation techniques for position location systems such as the inertial navigation systems. 2. Are there any prerequisites to take the course? Yes, the students must take ECT 350, which is our Communications Systems Course. This course investigates the fundamental concepts of electronic communications systems. Topics include Amplitude Modulation (AM), Frequency Modulation (FM), Phase Modulation (PM), digital modulation schemes, principles of power spectra and time domain analysis. 3. What are the course objectives? How does GPS works? Real World Applications of GPS Building GPS and INS (Inertial Navigation Systems) Protocol 4. How many students are usually taking your course? The course was offered two times so far. The course was offered for the first time during Spring 2001. There were 24 students enrolled. The course was offered again during Summer 2001. There were 5 students enrolled. 5. Is there any departmental limits to enrollments? And who are they? There were no departmental limits to enrollment, since the course was delivered on-line. However, the interim chairperson did not want the course offered every term. 6. What topics do you cover in your course? • • • • • • • • •

Electronic Navigation Introduction to Global Positioning System Basic Concepts of GPS Mechanics of GPS Differential GPS Inertial Navigation In-Depth Studies Guidelines for building GPS and INS Systems Integrating GPS & INS Systems

7. What knowledge/topics/concepts do you consider most difficult for the students to learn? Kepler’s Laws of Planetary Motion & Space Mechanics.

Chang S. Nam

Appendices

222

8. Then, what kinds of other instructional methods do you think are necessary to help the students easily learn? • • •

CD-ROMs Test Item Files Power Point Transparencies

9. If you were to teach the students your GPS course via a distance-learning system, what functions/features would you want the system to have? • •

Threaded Discussions, so that students can share information with each other Computer-oriented GPS labs

Part a) was done successfully in our distance-learning course. Part b) will take many more years of research. 10. Would you describe a teaching laboratory? And what is the main objective to provide students with the teaching laboratory? I would describe a teaching laboratory as one in which the students receive instant feedback on their work in any GPS learning environment. The main objective is that students learn in-depth concepts by performing hands-on experiments and instant feedback response on errors. 11. What kinds of problems with the lab do you think you have? Purchasing new equipment & discovering new ways to help students to do GPS labs at remote sites. 12. What do you think is the recent trend in GPS education? • •

how to effectively deliver a GPS course on-line how to effectively incorporate GPS laboratories into a field as young as GPS

13. To effectively fit into an accredited engineering curriculum, what features do you think any courses in GPS must emphasize? • •

student must learn how to design and build GPS equipment, such as receivers, and simulators student must be able to explain how GPS can be used to solve real world problems

14. What would say is a unique opportunity that any GPS course provides to students? Students who took my GPS got summer interns and full-time employment at Rockwell Collins working on GPSrelated work. I have a position myself as a NASA scientist at Kennedy Space Center this summer. I will be doing GPS-related research working with Advanced Range Technologies. 15. To master GPS fundamentals, what knowledge do you think a student must utilize? • • • •

Newton-Raphson Calculations First and Second Semester Physics Courses Introduction Satellite Communications Course Computer Programming in C

Chang S. Nam

Appendices

223

16. Did you have any difficulty teaching your GPS course? If so, what was that? And do you have an opinion on why students had such problems? Yes, our GPS lab is not fully furnished with all the equipment we wanted, for example a GPS Simulator. We have just recently received our GPS receivers. The next time the course is offered will be during Fall 2002. The students will implement some hands-on simulations, programming and data calculations. Our course so far focused on the theory and mathematical computations. The students had problems with the mathematical calculations due to their poor math backgrounds. 17. To facilitate a student’s learning, what do you think a GPS course must provide? • • • •

background theory on GPS and satellites adequate GPS equipment to do labs (GPS receivers, GPS simulators, Antennas) exercise the use of Matlab in GPS applications cover the practicality of GPS with other navigations techniques such as Inertial

18. How would you summarize the success or failure of the GPS course you offered? Because GPS applications are taking off, more and more students are interested in taking the course. The course will be even larger in the Fall of 2002. It is my personal challenge to do my role in helping to equip the lab, design more challenging lab experiments and assignments. I consider the course to be a great success so far. The only failure with the course was that the chairman would not let us offer it as much as we wanted to.

Chang S. Nam

Appendices

224

Appendix 2: A Summary of GPS Survey I Date: Fall 2001 Number of respondents: 12

[Demographics] – Number in parenthesis indicates the number of respondents D1. When you took the GPS course, Introduction to GPS Theory and Design (ECE4164), You were: ♦ Undergraduate (11) ♦ Graduate (1) D2. Gender:

♦ Male (11)

♦ Female (1)

D3. What is your major? ♦ Electrical engineering (10) ♦ Aerospace engineering (1) ♦ Other (1) D4. How would you classify your computer skill level? ♦ Novice (0) ♦ Intermediate (3) ♦ Highly experienced (9)

[Lecture] Question L1. The content of the GPS course, Introduction to GPS Theory and Design (ECE 4164), was easy to learn. L2. A supplemental tool such as a Web-based distance-learning tool would help me to effectively master the course. L3. The required textbook, Global Positioning System Theory and Design by P.M. Kinter, was helpful for understanding the subject matter. L4. The GPS course, Introduction to GPS Theory and Design (ECE 4164), provided me with the fundamental theory and concepts of the Global Positioning System. L5. The semester project was helpful to further my knowledge on GPS. TL1. The teaching laboratory was helpful to exercise the fundamental theory and concepts of the Global Positioning System that I had learned. TL2. The laboratory assignments were helpful to further my knowledge on GPS. TL3. The use of MATLAB templates was easy to use. TL4. The use of GPS Builder to get GPS data was easy to use. (Strongly disagree (1), Undecided (2), Agree (3), Strongly agree (4), Not applicable)

Average 1.8 1.2 1.6 1.3 1.4 2.3 1.8 1.8 1.2

L6. What prerequisites do you think will be helpful to take GPS course? • • • • • • • • •

Signals and system, electromagnetics Calculus theory, little bit of communications theory, MATLAB experience, and technical writing skills Vector calculus, electromagnetics Fields I & II, intro to comm.., a dynamics course or some other vector intensive course, and possibly Satcomm Those listed already seemed fine. Some knowledge of matlab would be helpful. Possibly, some of the fields and communications classes were unnecessary Some sort of numerical methods would be helpful Some Matlab training. Teaching Mathematica to engineers is a waste of time Nothing extra although Satcomm was good for helping in learning the orbital dynamics material Multivariable calculus, differential equations, Physics 2305 & 2306, A course designed to teach the fundamentals of MATLAB

Chang S. Nam • •

Appendices

225

Intro to communications, A MATLAB programming course and satellite communications would help as well, but would not be necessary Just some math courses. The typical engineering courses were plenty helpful

L7. Which of the following lecture topics were difficult to understand? Please designate a rank order on a scale from 1 to 13, where 1 represents "Most difficult" and 13 represents "Most Easy". E.g., Kepler Orbit Theory 1 = 4, Dilution of Precision = 7, etc. Lecture Topic Introduction to GPS and the Plessey GPS Receiver GPS Coordinate and Time Systems and Transformations Kepler Orbit Theory 1: Coarse Satellite Positions Kepler Orbit Theory 2: Precise Satellite Positions GPS Signal Structure, Range and Pseudorange Point Positioning using Pseudorange Measurements GPS navigation Errors Dilution of Precision Further Improving the Navigation Solution Fundamental GPS Signal Techniques Other GPS Communication System concepts The Velocity Solution Differential GPS

Mean Rank 10.8 7.3 8.2 5.2 6.6 8.9 7.3 5.6 8.3 6.8 8.1 7.9 6.6

L8. If you selected the most difficult topic to understand above, why do you think it is? • • • • • • • • •

Maybe it was just for me but the concept of differential GPS was hard to understand Too many equations and variables that I don’t fully understand. I don’t understand the purpose of each variable in the equation It was only covered briefly, and in a manner that was unnecessary for the course curriculum Having a set of overheads (class notes) online would be helpful to refer to Was not explained very well in textbook There was just a lot to it. It all made sense in the end None. It was all taught. Because I don’t have a background in orbital dynamics or gravitational motion. Development of the matrices and coding them into MATLAB was, for me the most difficult part of the course

L9. Then, what kinds of other instructional methods do you think are necessary to help easily learn? • •

The lab exercises were crucial to the understanding of the course material. Background information on the various concepts that are indirectly used in the class

Chang S. Nam

Appendices

226

[Teaching Laboratory] Question TL1. The teaching laboratory was helpful to exercise the fundamental theory and concepts of the Global Positioning System that I had learned. TL2. The laboratory assignments were helpful to further my knowledge on GPS. TL3. The use of MATLAB templates was easy to use. TL4. The use of GPS Builder to get GPS data was easy to use. (Strongly disagree (1), Undecided (2), Agree (3), Strongly agree (4), Not applicable)

Average 1.3 1.4 2.3 1.8

TL5. What kinds of skills, abilities and/or knowledge do you think are necessary to master the GPS course? • • • • • • • • • •

Calculus theory, little bit of communications theory, MATLAB experience, and technical writing skills Extensive programming knowledge in MATLAB, general comprehension of all coordinate systems MATLAB and a broad background Basic mats as well as matlab Solid programming and matlab skills as well as a decent understanding of communication topics A basic knowledge of MATLAB must be attained Good programming and general interest in the subject Use of the MATLAB templates greatly improves the learning curve. Development of unnecessary code is eliminated. Familiarity with MATLAB is necessary. I had VERY basic understanding of MATLAB and struggled with much of the coding Some understanding of MATLAB and programming Some knowledge of MATLAB

TL6. What do you think could help the student to efficiently exercise the teaching laboratory? • • • • • • • • •

MATLAB learning class before the actual labs start MATLAB programming, effective writing skills Clearly defined objectives. Sometimes it was confusing as to whether you had done all of the lab and had all the required info. Write your own code rather than just completing those Have a TA in the lab with the students Don’t give use so much of the code Some of the labs require a lot of sitting around while collecting data. The TA should set up the PC’s to start taking data early to minimize waiting More comments in the MATLAB code and consistent use of variables between the text book and the MATLAB programs The challenge was understanding the algorithms employed

TL7. What do you think could help the student to efficiently use MATLAB templates? • • • • •

More detailed instruction sheets More teaching assistants. Teaching assistant was helpful in explaining concepts, but he was the only TA for the course Trial and error there, learn from other classes’ projects Try to make them more general. Or explain how you are setting up the matrices better. Often times the user was fighting the matlab syntax and not the algorithm Better comments on the code

TL8. What do you think could help the student to efficiently use GPS builder? •

It’s not difficult to use

Chang S. Nam • • • • • • •

Appendices

227

TA help with GPS builder was adequate Repeatedly use, possibly a mini manual A new windows interface. Also a web interface could be helpful and very possible. It would definitely make data collection on the final project easier Tool is easy to use Have a TA explain how to use it Reading of the manual. It is easy to read and the GUI is very descriptive A windows interface and faster computers

TL9. Which of the following labs were difficult to exercise? Please designate a rank order on a scale from 1 to 13, where 1 represents "Most difficult" and 10 represents "Most Easy". E.g., GPS Observation = 3, Ranging Errors = 7, etc. Laboratory Topic Lab 1: Introduction to Principles of the Global Positioning System Lab 2: Preliminaries: GPS Receivers and the Newton-Raphson Method Lab 3: Reference Frames, Coordinates, and Transformations Lab 4: Keplerian Orbits and Satellite Locations: Coarse Positioning Lab 5: Corrections to Keplerian Orbits: Precise Positioning Lab 6: GPS Observations Lab 7: The Navigation Solution Lab 8: Ranging Errors Lab 9: Dilution of Precision Lab 10: Improvements to the Navigation Solution

Mean Rank 8.2 5.4 6.5 6.3 6.0 5.3 6.3 4.8 5.0 6.0

TL10. If you selected the most difficult topic to exercise above, why do you think it is? • • • • • • • • •

Although it was fundamental concept for understanding the GPS, Newton-Raphson method took me longer than any other labs and was difficult to master the theory The harder the concepts were to understand in class, the harder the labs were because I did not have good understanding of the material This lab involved all of the topics learned up to this point to arrive at an exact method Better write up for what the student should expect when their MATLAB code is finally correct. A lab supplement to the book could help. Pre-programmed templates were confusing at times The biggest problems were with knowing what code to fill in. You had to be psychic to know what to enter sometimes It was just a lot of code to pull together Bringing each of the elements together into the Navigation solution was difficult because there is a lot of information to keep track of. The step by step approach used by Dr. Scales greatly assisted in this and made it bearable. Once again difficulty came in understanding the algorithm

TL11.Then, what kinds of other instructional methods do you think are necessary to help easily exercise? • • • • •

Supplementary references More TA help. More professor office hours. Website help Write crucial parts of code yourself Without the TA in the lab, it would have been impossible If we had to write more of the code ourselves we would have understood what each program did and what was outputted.

Chang S. Nam

Appendices

228

Appendix 3: A Summary of GPS Survey II Date: Fall 2002 Number of respondents: 29 We are going to develop a Web-based distance and distributed learning (WD2L) environment as a new learning environment to support GPS education in both on- and off-campus. As a supplement to existing teaching methods, this learning environment will allow you to study the GPS course, including laboratory exercises, at your own time in your own space on the Web. The objective of the survey is to ascertain the needs of the GPS learners necessary for implementing the WD2L environment. The survey may take no more than 30 minutes to complete. Please provide your experiences and opinions, which will help us develop the WD2L environment to support GPS education. Your response to this survey will be kept confidential, and individual data is not available in public. Thanks for your response, in advance. L Suppose that you want to study the GPS course, Introduction to GPS Theory and Design (ECE 4164), via a Web-based learning system before you actually attend the class and lab session every week. 1. What functions would you want the learning environment to have? (e.g., listserv, online quiz, lab activities, etc.) • • • • • • • • • •

Online quiz Lab activities Online chat Feedback on homework and exams Online submission (assignments and lab reports) Supplemental materials related to GPS Announcements Frequently Asked Questions Practice exercises Discussion board

• • • • •

Information on other students Links to GPS sites Detailed chapters Online lecture notes MATLAB template hints

• • • •

Access GPS data from lab More visual materials supporting the study Pre-lab tutorial GPS glossary

2. What methods do you think could help the student to efficiently exercise the teaching laboratory on the Web? E.g., more practice. • • • •

Feedback on lab exercises Sound/video clips to explain unique exercises MATLAB tutorials More MATLAB examples

• • •

Prelab help Visual aids Have the GPS builder run online

3. What do you think could help the student to efficiently use MATLAB templates? In other words, what kinds of help do you want to receive through the WD2L environment? • • • • •

MATLAB tutorials Clear comments on MATLAB scripts (e.g., more descriptions of what code is doing) An outline of the algorithm used A table mapping variable name to physical quantities Basic matrices functions and roles used in GPS MATLAB scripts

Chang S. Nam

Appendices

4. Please provide ALL possible tasks that you want to do on the WD2L environment. E.g., see announcement, upload homework, Go to VT homepage, etc. • Check Announcement • See Instructor’s Background • See Information on GTA • Check Course Description • See Course Objective • Check Grades • See Honor code • Take Quiz • See Homework description • See Project description • See Exam description • Perform Lab Activities • Receive Feedback • Send E-mail • Use Discussion board • Use FAQs

• Upload homework • Download homework • See Homework Results • See Quiz Results • See Exam Results • Go to VT homepage, Library, & Hokie Spa • Use Search Engines • See Resource materials • Use Help Menu • See Study Guide • Study Lecture Materials • Download lecture materials • Find the definition of the GPS Glossary • Past projects • Past exams

5. Suppose that you are looking at Chapter 5 on the Web. Which type of Web page format do you want? • Multi-pages: Don’t have to scroll down the page, but click the “NEXT” button ( 1 ) • One lengthy page containing all information: Must scroll down to see the whole page ( 1 ) • Both of them (27)

[Demographics] Age Gender Department Education level Computer type Online frequency Connection speed Online habits Browser type Monitor size E-learning experience

• 21 ~ 30 years (average 22.3) • Female (17 %) • Male (83 %) • Electrical engineering (75%) • Aerospace engineering (25%) • Graduate students (30%) • Undergraduate students (70%) • Personal computer (PC, 100%) • Daily (100%) • Ethernet (69.0%) • 56K (13.8%) • Cable modem (10.3%) • T1 (6.9%) • Frequently visited sites: News, VT homepage, etc. • Online purchase: at least once a month (e.g., amazon.com, ebay.com, etc.) • Web savvy • Microsoft Internet ExplorerTM 5.0 and higher • NetscapeTM 6.0 and higher • 15” (24.1%) • 17” (65.5%) • 19” (10.3%) • Yes (41.4%) • No (58.6%)

229

Chang S. Nam

Appendices

230

Appendix 4: Requirements Specification Document 1. Purpose of Requirements Specification Document • • •

Provide a shared understanding of the goals and requirements for the WD2L environment Identify immediate needs and new design concepts necessary to develop the instruction and interface system Describe all possible factors that may influence the implementation of the WD2L environment

2. Development Goals of WD2L Environment • • • • • • •

Provide learners with a time and spatial independent self-paced learning environment on the Web Provide learners with a unique opportunity to pull together many fundamental engineering concepts Provide learners with early and regular opportunities to test their understanding Provide communication channels for increasing and maintaining student-student and student-to-teacher interaction outside of the lectures Provide various opportunities for learners to allow gaining hands on experience with an important modern GPS technology Provide learners with a variety of resources, making them available to the entire learning community Provide user interfaces to effectively support learning activities on the Web

3. User Profile 3-1. User Classes Target user classes have been identified by their work context roles. •

Students: Students in the Electrical and Computer Engineering department, who will take advantage of the WD2L environment. They will easily pull together many fundamental concepts and effectively gain hands-on experience with the GPS technology at their own pace in their own time.

•

Instructor: This user class serves as a resource to the students. The instructor manages the Web-based learning, providing immediate feedback on student work and discussion, and evaluating students.

•

Administrator: This user class is a highly trained individual with a specific knowledge of computers and Web-based learning environments.

•

Graduate Teaching Assistant: This user class would be composed of users who assist instructor in teaching the class.

Chang S. Nam

Appendices

231

3-2. User Characteristics Four different types of users and their activities were identified in the user roles section above. However, this project focuses on only student users and their activities. Matrix Age Gender Department Education level Computer type Online frequency

Description • From 21 to 30 years old (average 22.3 years old) • Female (17 %) • Male (83 %) • Electrical engineering (75%) • Aerospace engineering (25%) • Graduate students (30% ) • Undergraduate students (70%) • Personal computer (PC) • Daily • Ethernet (69.0%) • 56K (13.8%) • Cable modem (10.3%) • T1 (6.9%) • Frequently visited sites: News, VT homepage, etc. • Online purchase: at least once a month (e.g., amazon.com, ebay.com, etc.) • Web savvy • Internet Explorer 6.0 • 15” (24.1%) • 17” (65.5%) • 19” (10.3%) • Yes (41.4%) • No (58.6%)

Connection speed Online habits Browser type Monitor size E-learning experience

3-3. User Motivation Student users’ motivation level has been analyzed through Keller’s (1987) ARCS framework. ACRS

Level

Attention

Initially low

Relevance

Moderate to high

Confidence

Variable

Satisfaction Positive

Description The GPS course (ECE 4164) has a reputation for having many engineering theories and complex mathematical equations. Furthermore, students are likely to be there because they have to take it as a requirement course. The goal of the course is to provide the students with the fundamental theory and concepts of the Global Positioning System and to demonstrate these concepts through laboratory exercises and a final design project. Therefore, students consider this course an opportunity for training them as highly skilled engineers. The primary electrical engineering concepts involved in this class are 1) Radiowave propagation (ECE 3106) and 2) Communication system theory (ECE 3614). Satellite orbit dynamics is the third important aspect of this course. Therefore, students who have already taken all those courses will be quite confident in their ability to take this course. On the other hand, students who have not taken one of these courses will concern about their ability to complete this course. Students think of this course one of the most useful courses in ECE department because it provides many fundamental concepts about GPS technology as well as allows having experiences with an important modern technology.

Chang S. Nam

Appendices

232

3.4. User tasks that should be supported Possible Activities (or tasks) that can be performed by web-based distance learning users. • • • • • • • • • • • • • • • •

Check Announcement See Instructor’s Background See Information on GTA Check Course Description See Course Objective Check Grades See Honor code Take Quiz See Homework description See Project description See Exam description Perform Lab Activities Receive Feedback Send E-mail Use Discussion board Use FAQs

• • • • • • • • • • • • • • •

Upload homework Download homework See Homework Results See Quiz Results See Exam Results Go to VT homepage, Library, & Hokie Spa Use Search Engines See Resource materials Use Help Menu See Study Guide Study Lecture Materials Download lecture materials Find the definition of the GPS Glossary Past projects Past exams

4. Look and Feel Requirements 4-1. User Interface The user interface requirements specify how the users will interact with the WD2L environment. The WD2L environment shall • • • • • • • • • • • • • • • • • •

Provide the structure of menus that allows the user to select them anywhere Ask the user to enter her/his VT ID and password to log in Allow the user to access external links, including VT homepage, library, Hokie Spa, etc Provide the capacity to search information on site, GPS glossary, and discussion topic in the discussion board Allow the user to select the day that announcements are provided Provide the user with information on course and staff involved Provide the user with the capability to find the definition of GPS terminology Allow the user to access GPS resources Provide the user with an interface to upload and download information Provide the capacity for user to practice MATLAB scripts Allow the user to upload assignments, and to see the results Provide a graphic user interface to support the user’s group activities Provide a form-based interface to send a short message to other users Allow the user to ask a question, discuss a topic, and share information with other users Provide concept map to the users, which demonstrates how each chapter of the course and sub-topics are related to each other Allow the users to perform pre-laboratory activities Allow the users to assess what they have learned, and to provide informative feedback on their performance Support the users’ information sharing with other users by providing various communication channels.

Chang S. Nam

Appendices

233

4-2. Development Architecture The WD2L environment architecture requirements describe the top-level requirements that the WD2L environment must meet, in order to fulfill its operational mission. • •

WD2L environment shall have a server application. WD2L environment shall have a client application.

5. Instructional Design Requirements The instruction requirements define the instructional design requirements that the WD2L environment must meet, in order to effectively teach the course. WD2L environment shall: • • • • • • • • • • • • •

Organize the learning content of the course in such a way that user is able to connect new information with existing knowledge in a meaningful way Design the course by emphasizing the active involvement of the user Provide the instruction that allows to effectively attend, code, transform, rehearse, store, and retrieve information Use nine events of instruction as a Web-based instructional strategy, suggested by Gagne, Briggs, and Wager (1992) To gain attention To inform user of learning objective To stimulate recall of prior learning To present stimuli with distinctive features To guide learning To elicit performance To provide informative feedback To assess performance To enhance retention and learning transfer

6. Reliability, Maintainability, and Availability requirements These requirements address, at the very highest level, the needs associated with the WD2L environment’s availability and maintenance. • •

WD2L environment shall be available 24 hours per day, 7 days per week to support the GPS learning on the Web. WD2L environment shall be maintainable by a Web master.

7. Technological requirements The technology requirements specify the technology capabilities that need to successfully support potential users. • •

As a minimum browser, Netscape (4.0 or above) and Internet Explorer (5.5 or above) shall be required. A minimum connection speed of 56k or above shall be required (connection type: modem, cable, DSL, or Ethernet, etc.) • Software requirements: - Word 97 or higher - Excel 97 or higher - PowerPoint 97 or higher - MATLAB program • Windows 98 or higher for PC, OS 8.x or higher for Macintosh shall be required.

Chang S. Nam

Appendices

234

Appendix 5: The Content Outline Document Component

Page Title

Search engine

Search

GPS Glossary

Glossary

Chapter

Chapter5

Send Email

Mailing

Discussion Board

Discussion

Practice sessions

Practice

Quiz Review Prelaboratory

Quiz review Prelaboratory

Concept Map Assignments

Concept Map Assignments

Document Sharing GPS Resources Announcements

Document Resources Announcements

Course Information

Course

Staff Information

Staff

Homepage

Homepage

Elements to be Included • Search box • Advanced search • Search tips • GPS terms • Usage tips • Objectives • Think for a while • Practice sessions • Graphic images • Recipient • Configuration • Attachments • Write • Read • Delete • Modify • Reply • Practice questions • Feedback • Feedback • MATLAB codes • Feedback • Concept map • Links • Submit page • Links • External links • View today • View last 7 • View all • Course overview • Course assignment • Photo • Profile • Course title • Log-in box • GPS image • Menu bar

Description Text box to type search word(s) Advanced search option Help for search GPS terms listed alphabetically Short description on how to use Objectives of Chapter 5 Think back what has learned Interactive practice sessions Examples in a graphic image form All member, group, or individual Font size, color, & alignment File attachment Write a message Real all messages selected Delete a message Modify one’s own message Reply to a message Different types of questions Informative feedback Informative feedback MATLAB program codes Feedback page Concept map linked each chapter Assignment links that can be downloaded Submit page to send assignments Document links that can be downloaded A list of GPS websites Announcement(s) posted today All announcements posted last 7 days All announcements General information on the course Information on course assignments Photo images of instructor and GTA Personal information Class title (GPS Theory & Design) Log in box with PID and password GPS images Menu items

Chang S. Nam

Appendices

235

Appendix 6. A Summary of User Interface Review Part I. Demographics • Age: [ 32 ] years • Gender: male [ V ]

female [ ]

• Highest Education (Please check): [ ] Master Candidate

[ V ] Ph.D. Candidate

[ ] Postgraduate

• Have you had any experience with a Web-based distance learning environment? [ ] No [ V ] If yes, briefly describe: I have used Element K, a self-paced web-based learning environment if that counts. I have tried some online lecture with streaming web-audio. But gave up half way due to low bandwidth. Not very good experience with a real time web lecture. • Please briefly describe your background as a user interface design expert? (e.g., any experience associated with the user interface design & evaluation, human factors courses taken, etc.) I have worked as a HF engineer for five years before studying at VT and was involved in interface design in vehicle system, biomechanical studies, and system design projects. I have been in the HF program for 2 years and completed all the major HF courses including Usability Engineering, Human Information Processing, System Design, Training System, Display Design and so on. I was involved in critical incident analysis of cell phone focus group evaluation providing design guidelines for phone interface.

Chang S. Nam

Appendices

236

Part II: Evaluation on Components of the User Interface System Instruction: Please review each component of the user interface system, and provide comments or recommendations that will help revise the user interface system. [1] Homepage Comments/Recommendations in the 1st Review Session: Main image is blurry, should be sharp as user might think something is wrong with the site or own computer. Main image is not active and occupy such premier area, which is a waste of space and defeating the purpose of the Home Page. It has no informational value at all! The dropdown menu is good and the redundant links below is also good. Should have some vertical gap between the link combo, search input etc to create two functional groups for easier reading. The red color with yellow letters for mouse-on menu followed by light purple with yellow letters somehow is not contrasting enough. The use of red color also does not match the overall blue color scheme (Title, Status bar). Comments/Recommendations in the 2nd Review Session: 1) The homepage has good color scheme. Menus are good. 2) Should have some basic information on the homepage to describe briefly what this site is all about. [2] Search box Comments/Recommendations in the 1st Review Session: Links to VT and Search Option are good features! Search box size should be larger to accommodate more text. There is quite a lot of space on the right when my browser is maximized for my monitor (17 in at 1024X768). Should resize to window size and should design site to optimize to user display setting. Style sheets can be used for such purposes. Comments/Recommendations in the 2nd Review Session: Search result for Whole Site is well designed. The result for resources should have titles to classify the different groupings in addition to the blank space to allow easier scanning. For example, Magazine, Universities, etc. Result for glossary should be indexed with numbers. Otherwise, overall the search results are quite well presented with the correct size fonts and color scheme. [3] Announcements Comments/Recommendations in the 1st Review Session: Three links are OK (View today, View last 7 days, View all). Comments/Recommendations in the 2nd Review Session: Fonts and layout are OK. Very clear.

Chang S. Nam

Appendices

237

[4] Course Information Comments/Recommendations in the 1st Review Session: Too much vertical white space between headings and main text. The running side bar is good, but it is a bit jerky when scrolling up and down. Still, good enough since user would stop to click on the links. Again too much white space on my right as the text is not stretch across screen. Comments/Recommendations in the 2nd Review Session: Disable Course Overview link when already in Course Overview page. Make it visible only when in the Course Assignment page. Likewise for the Course Assignment link. 2. Layout and content is good.

[5] Staff Information Comments/Recommendations in the 1st Review Session: Good as it is. No comments.

Comments/Recommendations in the 2nd Review Session: The instructor and GTA offices are not stated. Should have this information here as well.

[6] Chapter 5 Comments/Recommendations in the 1st Review Session: Prev and next should be at the top as well as at the bottom. Single page, Multi-page links are good. Line spacing in main text should be 1.5 lines instead of single. Comments/Recommendations in the 2nd Review Session: Overall is good, but I cannot activate the single-page and multi-page option. The use of red emphasis, bold, and different font sizes are good.

[7] 5 Practice Sessions Comments/Recommendations in the 1st Review Session: It is quite tedious to click on the feedback button for each question. Use one feedback button for all questions to see the explanation for each correctly answer question. Comments/Recommendations in the 2nd Review Session: The Practice Link should not be moving, as it is too distracting and does not appear to be a link. Rather you can have a blinking icon by the stationary link to capture attention. Practice sessions are generally good but you need to check spelling and the punctuation. Some of the symbol did not turn up well such as “earth’s” appears as “earthi _s”.

Chang S. Nam

Appendices

238

[8] Concept Map Comments/Recommendations in the 1st Review Session: Highlight box for when mouse is on the box to show selection. Use light color for highlight. The arrow lines should be more solid rather than dotted lines as it is here. Standardize the capital letters of your titles. Time Standards. Elevation. GPS Environment etc. Comments/Recommendations in the 2nd Review Session: I don’t find the concept map particularly useful since it only links to the main section similar to Content in Chapter 5 Introduction, and there is no additional information on the map for the rest of the subsection under the main headings. A pop-up menu giving brief notes on the subheadings might be useful. [9] GPS Resources Comments/Recommendations in the 1st Review Session: This page is good. Comments/Recommendations in the 2nd Review Session: Well laid out, results for search should be done in this manner. [10] GPS Glossary Comments/Recommendations in the 1st Review Session: Provide links to related topics within each definition. E.g. On-The_Fly. This is a form of Ambiguity Resolution (AR) which ….. Pretty good layout and organization. Good font size and font type. Comments/Recommendations in the 2nd Review Session: This page is good. [11] Prelaboratory 5 Comments/Recommendations in the 1st Review Session: Try to guide the user with step-by-step instruction and perhaps have a list of functions at the top with links to the codes at the bottom like your glossary page. Comments/Recommendations in the 2nd Review Session: This page is OK. [12] Assignments Comments/Recommendations in the 1st Review Session: Why is the table number starting from 4 to 1? I expect 1 to 4 unless there is a good reason for this. The submit buttons there are kind of confusing to me. I would expect to have download buttons instead and only one Submit button at the bottom since each time I press Submit it is a generic form. I have to fill in the blanks and there

Chang S. Nam

Appendices

239

is no functional purpose that the buttons are in each row. The Browse button is not raised like the rest, in the Submit form. Instead of List, use Assignment List. Name and address should be auto filled since it part of the log on page. Should still display the input box so that user can change it in case there are more than one person. Comments/Recommendations in the 2nd Review Session: 1. The index numbers are still reversed from 4 to 1 down, instead of Homework 1 to 4. [13] Quiz 1 Review Comments/Recommendations in the 1st Review Session: O.K with the interface in this page. Comments/Recommendations in the 2nd Review Session: Feedback dialog does not show properly, probably due to the use of symbols. Need to check the feedback for different computers. [14] Send Email Comments/Recommendations in the 1st Review Session: This page is too complicated for the purpose of sending a message! What is the meaning of HTML Tag: USE/NOT? Keep it simple. Allow choosing of email address for people in the course. Remove the configuration unless it is essential that user have to change the different font size and type etc. The symbols are useful if they are course specific for formulas and so on. But should be useful symbols and not too many as it is harder to use when list is longer. I find the yellow Send button strange. Use default color. Bold the border if you want to highlight it. Comments/Recommendations in the 2nd Review Session: O.K. [15] Discussion Board Comments/Recommendations in the 1st Review Session: Again, I do not know why the numbering is not ascending in the list. The green color letters for the headings are difficult to read. Use bold black letters. What does the read check box do? Users would know whether they have read a particular message. The change in color after clicking is sufficient to let users know they have click on the link before.

Comments/Recommendations in the 2nd Review Session: Use Bold Black Titles instead of the green. Green and Grey are not contrasting The rollover effect for the message is quite neat.

Chang S. Nam

Appendices

240

Part III: Overall Quality of the User Interface System Based on User Interface Rating Tool for Interactive Multimedia © 1993 Thomas C. Reeves, Ph.D. & Stephen W. Harmon, Ed.D. Instruction: Please respond to each of the following statements by circling the appropriate number to indicate how much you agree or disagree with the statements. 1. Ease of Use “Ease of Use” is concerned with the perceived facility with which a user interacts with an interactive multimedia program.

Difficult

Easy

2. Navigation “Navigation” is concerned with the perceived ability to move through the contents of an interactive program in an intentional manner.

Difficult

Easy

3. Cognitive Load In terms of “cognitive load,” the user interface can seem unmanageable (i.e., confusing) at one end of the continuum and easily manageable (i.e., intuitive) at the other end.

Unmanageable

Manageable

4. Mapping “Mapping” refers to the program’s ability to track and graphically represent to the user his or her path through the program.

None

Powerful

5. Screen Design “Screen Design” is a particularly complex dimension of interactive programs that can easily be broken down into many sub-dimensions related to text, icons, graphics, color, and other visual aspects of interactive programs.

Violates Principles

Follows Principles

Chang S. Nam

Appendices

241

6. Knowledge Space Compatibility “Knowledge space” refers to the network of concepts and relationships that compose the mental schema a user possesses about a given phenomena, topic or process.

Incompatible

Compatible

7. Information Presentation The “Information Presentation” dimension is concerned with whether the information contained in the knowledge space of an interactive program is presented in an understandable form.

Obtuse

Clear

8. Media Integration The most important aspect of the media integration dimension refers to how well an interactive program combines different media to produce an effective whole.

Uncoordinated

Coordinated

9. Aesthetics “Aesthetics” refers to the artistic aspects of interactive programs in the sense of possessing beauty or elegance.

Displeased

Pleased

10. Overall Functionality “Overall Functionality” is an aspect of interactive multimedia programs related to the perceived utility of the program.

Dysfunctional

Functional

♦ Please, if any, provide comments or recommendations for the overall user interface system. Comments/Recommendations: Classroom content should have more interactive elements with hyperlinks, animation, and pictures since this is the most important part of the whole site! This is however, usually the weakest link in most web-based instruction where people just use the text element of textbooks and forget about the graphical processing capabilities of computer. Unfortunately the multimedia content is perhaps the most expensive to do requiring more thinking. There should have more contextual links between resources, and glossary to the main classroom content. For example, linking more in-depth discussion on subtopic to other websites or university.

Chang S. Nam

Ease of Use 1st Session 2nd Session

Appendices A Summary of the Overall Quality of the User Interface System 1- Difficult 2 3 4 5 6 - Easy 5 6

Navigation 1st Session 2nd Session Cognitive load 1st Session 2nd Session Mapping 1st Session 2nd Session Screen Design 1st Session 2nd Session Knowledge Space Compatibility 1st Session 2nd Session Information Presentation 1st Session 2nd Session Media Integration 1st Session 2nd Session Aesthetics 1st Session 2nd Session Overall Functionality 1st Session 2nd Session

242

1- Difficult

2

3

4

5

6 - Easy 6 6

1- Unmanageable

2

3

4

5 5 5

6 - Manageable

1- None

2

3

4

5

6 – Powerful 6

5 1- Violates Principle

2

3 3

4

5

6 – Follow Principles

5 1- Incompatible

2

3

4

5

6 - Compatible 6

5 1- Obtuse

2

3

4

5 5

6 – Clear 6

1- Uncoordinated

2

3

4

5 5

6 - Coordinated 6

1- Displeased

2

3 3

4

5

6 – Pleased 6

1- Disfunctional

2

3

4

5 5

6 - Functional 6

Chang S. Nam

Appendices

243

Appendix 7. A Summary of Instructional Design Review Part I. Demographics • Age: [ ] years 1st Session 34

2nd Session 30

• Gender: male [ ] 1st Session male

female [ ]

2nd Session female

• Highest Education (Please check): [ ] Master Candidate 1st Session Ph.D. Candidate

[ V ] Ph.D. Candidate

[ ] Postgraduate

2nd Session Ph.D. Candidate

• Have you had any experience with a Web-based distance learning environment? [ ] No [ V ] If yes, briefly describe: 1st Expert: Online learning and class development 2nd Expert: As a student, I took couple of online courses. As a designer, I helped several teachers design and develop their online courses. • Please briefly describe your background as an instructional design expert? (e.g., any experience associated with the instructional design (ID), ID courses taken, etc.) 1st Expert: I am adept in the design, development, implementation, evaluation, and management of online learning environments. While Associate Director of Instructional Design at North Carolina Agricultural and Technical State University, Forrest was instrumental in the establishment of the Office of Instructional Design (OID). My responsibilities included providing instructional design and development expertise and support to faculty in the creation of eLearning courses. I have worked with North Carolina Agricultural and Technical State University and Winston-Salem State University in the creation of their virtual campus interactive CDROM’s. In both projects the purpose was to increase the enrollment of incoming students and to inform the community about the university’s availability of online instruction. 2nd Expert: ID courses: Instructional design; Applied ID theories; ID on Distance Education; Trends and Issues in ID Instructional Design Experience: I worked in VT Faculty Development Institute as a training assistant in summer 2001 and helped the faculty members to develop their online courses; I have been working as a designer and grader in Instructional Technology online Master Program (ITMA) at VT since 2001; As an independent designer, I designed and developed couple of web sites to teach math and Chinese language.

Chang S. Nam

Appendices

244

Part II: Review of the Instructional Design Instructional Event 1: Gaining attention Description: • • •

Post announcements for students to inform of when and what to study, and to provide updated site information Before starting a new chapter, provide a brief summary of what have learned in the previous chapter, e.g., “Think for a while” Allow the students to see a new announcement anytime and anywhere

Comments/Recommendation in the 1st Review Session: You layout compliments the design of the course. This is good. It is intuitive and user friendly. However, according to Driscoll (2000) Gagne’s 1st event allows for mediated instruction. This is a form of attention guiding device to what is coming up in the lesson (course). I see you have a classroom button on the bar, but I think it would be a more specific guide for students if you titled that button “Classroom Guide.” The way it is label now does not give me a clear idea of where to find the structure of the entire course. Comments/Recommendation in the 2nd Review Session: Nice objective, especially it is put on the first page of each chapter so that the students could not skip it. I was worried about that. Instructional Event 2: Informing learner of lesson objective Description: At the top of the screen, state in simple terms what the student will have accomplished as they learn, e.g., list 4 main objectives in Chapter 5 • Examine the correcting terms in the GPS navigation message required to compile each satellite's position with errors the order of a few meters, • Compare the relative contribution of different physical perturbations, • Investigate the algorithm for computing corrections, the magnitude of the corrections, and the period of validity of the corrections, and • Study the consequences of the radio signal traveling at the speed of light. Comments/Recommendation in the 1st Review Session: Make sure your objectives are written in the proper struture: Behavioral objectives must be included in all behaviorist-learning environments. Objectives should be "specified, quantifiable, terminal behaviors" (Saettler, 1990). Broadly speaking, behavioral objectives have been normally summed up using the mnemonic device ABCD. Example: After having completed the unit the student will be able to answer correctly 90% of the questions on the posttest. A: Audience - the student; B: Behavior - answer correctly; C: Condition - after having completed the unit, on a posttest; D: Degree - 90% correct Comments/Recommendation in the 2nd Review Session: Nice objective, especially it is put on the first page of each chapter so that the students could not skip it. I was worried about that.

Chang S. Nam

Appendices

245

Instructional Event 3: Stimulating recall of prior learning Description: • • •

Concept map demonstrating what sub-topics each topic has and how each chapter of the course and subtopics are related to each other “Think for while” designed to allow students to think back for a while about what they have learned in the previous chapters(s) and how their prior knowledge is related to the current topics Feedback messages also serve for stimulating recall of prior learning (1) Practice sessions; (2) Quiz 1Review; (3) Prelaboratory 5

Comments/Recommendation in the 1st Review Session: Make your feedback screens give either a positive or negative response. Use of cues, shaping and practice to ensure a strong stimulus-response association The accuracy of the response is based on the appropriate cues, which are accompanied by proper instruction. The prescription usually follows a cue/reinforcement relationship (Ertmer & Newby, 1993). Based on the literature, instruction should be structured, where the teacher has done the following: Determined which cues can elicit the desired response Arranged practice situation where target stimuli and prompts are paired within a natural performance setting Allow for correct responses within the teaching environment and provide reinforcement and feedback. Comments/Recommendation in the 2nd Review Session: Nice concept map, but I do not know how I can reach this map, if I am not provided with the direct link to it. I tried to go to classroom guide and click the concept map, but it did not work. Good design of “Think for a while” Detailed informative feedback. Very helpful. And the practice parts have different formats, e.g. multiple choice, filling the blanks, and image. The diversity of the practice will help to motivate the learners and keep them in the track. Very nice design. Instructional Event 4: Presenting stimuli with distinctive features Description: • • •

Allow to use search engine to find GPS glossary Use graphic image to effectively explain a concept/theory/terminology, e.g., Figure 5.2 GPS resources listed by a topic

Comments/Recommendation in the 1st Review Session: Very effective use of the glossary Comments/Recommendation in the 2nd Review Session: Nice use of the graphic image. I tried to click the audio link to listen to the instruction, but I got the error message: “The data that the plugin requested, did not download successfully”. I am not sure what the problem is. It would be nice if there were some kind of “FAQ” to help solve the problem. When using the audio or video links, the designers need to be especially careful with the function of the links. Sometimes the learners will feel very frustrated if they cannot successfully open the links.

Chang S. Nam

Appendices

246

Instructional Event 5: Guiding learning Description: • • •

Provide instructions on the “Classroom Guide” section to guide learning of the content Provide an indication of performing practice sessions Indicate when to use the search engine, external GPS resources, or GPS glossary

Comments/Recommendation in the 1st Review Session: See comments from event #1 Comments/Recommendation in the 2nd Review Session: I would not suggest using animated buttons for online course. To indicate the practice sessions, I would use different color or font size or put it in a more obvious position. Nice job in offering the learning guidance. I also think the SITE MAP did a good job in achieving this instructional event. Instructional Event 6: Eliciting performance Description: • • • •

Ask the students to perform Practice sessions; (2) Quiz 1 Review Ask the students to send e-mails to the instructor or GTA if they have any unanswered questions Ask the students to post their opinions and questions on the Discussion Board

Comments/Recommendation in the 1st Review Session: Good as it is. No comments. Comments/Recommendation in the 2nd Review Session: Good job on practice session and prelaboraory 5. But I would not have feedback immediately follow each question item in the quiz. If it is a quiz, the students were not supposed to view the feedback before they submit it. I would put a submission button at the end of the quiz and give out feedback after the students submit the quiz. Instructional Event 7: Providing informative feedback Description: • • •

Provide feedback during practice sessions Give information about correct and incorrect responses to quiz questions Provide comments on questions posted on the Discussion Board or through e-mail

Comments/Recommendation in the 1st Review Session: The responses you give for the feedback on the quiz review (incorrect/correct) should be consistent through out the course.

Chang S. Nam

Appendices

247

Comments/Recommendation in the 2nd Review Session: Nice job. Instructional Event 8: Assessing performance. Description: • •

Ask the students to perform practice sessions Ask the students to take a quiz

Comments/Recommendation in the 1st Review Session: Is this quiz given throughout the course or unit? There should be a consistent button or marker for where the quiz will be for every unit/chapter/module etc. The way this looks, it may be a final for the entire course. Comments/Recommendation in the 2nd Review Session: Nice practice sessions. Please refer to the former parts for detailed comments on practice and quiz.

Instructional Event 9: Enhancing retention and learning transfer. Description: • •

Post new information and topics on the “Announcements” page Ask the students to post questions on the Discussion Board

Comments/Recommendation in the 1st Review Session: Good as it is. No comments. Comments/Recommendation in the 2nd Review Session: Good job. But how to motivate the learners to participate in the discussion and group work is a big concern for online instruction. It may be a question for both the instructors and the designers

Chang S. Nam

Appendices

248

Part III: Overall Quality of the Instructional Design Instruction: Please respond to each of the following statements by circling the appropriate number to indicate how much you agree or disagree with the statements. 1. Instructional design is appropriately developed for the intended audience.

Strongly Disagree

Strongly agree

2. Instructional design is closely related to learning objectives stated.

Strongly Disagree

Strongly agree

3. Instructional design is clear enough to be self-instructional.

Strongly Disagree

Q1 Q2 Q3

Strongly agree

A Summary of the Overall Quality of the Instructional Design 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 6 2nd Session 6 1st Session 2nd Session st

1 Session 2nd Session

4 6 5 6

Comments/Recommendation in the 1st Review Session: This layout is typical of most application service providers that do online courses (Blackboard, eCollege). The concern I have with this design is it can be hard to find your way around in some courses. Your design is appropriate but in my experience, hard sciences such as engineering and computer science are hard to absorb when not face-to-face. At least that has been the argument I have received from some professors. The simple answer that I have for effective online instruction is consistency, consistency, and more consistency! Don’t let the technology dictate your instructional instinct. It should always be the other way around. Comments/Recommendation in the 2nd Review Session: The whole page is clean and looks nice, but the font size is kind of small or light. The web page has very nice navigation buttons, so it is easy for the learners to browse through the pages. There is some animated navigation button. In one hand, it could gain the learners’ attention, but in the other hand, it is distracting sometimes. I would not suggest using it in an online course. The same format is used throughout the whole website, which will help the students to easily navigate the sites and have the feeling of unity. Nice work on the navigation buttons on both the top and the bottom of the body page. The web site is simple and clean, and no fancy stuff. I used the modem pool connection to browse the whole site, and the speed is acceptable. The print version of each chapter is very helpful for online course since the students may want to print out the lectures.

Chang S. Nam

Appendices

249

Appendix 8. A Summary of Content Review Part I. Demographics • Age: [ 27 ] years • Gender: male [ V ]

female [ ]

• Highest Education (Please check): [ V ] Master Candidate

[ ] Ph.D. Candidate

[ ] Postgraduate

• Have you had any experience with a Web-based distance learning environment? [ V ] No [ ] If yes, briefly describe:

• Please briefly describe your background as a GPS content expert? (e.g., any experience associated with GPS education, GPS-related courses taken, etc.) Participated in the GPS-GSM-SMS switching gateway project Course: GPS Theory and Design Graduate research assistant for the GPS lab

Chang S. Nam

Appendices

250

Part II. Evaluation of Learning Units Content Unit: Chapter 5 Instructions: Please review Chapter 5 web pages at your own pace. For each of 4 learning content dimensions listed below, assess the learning contents of Chapter 5 by circling the appropriate number to indicate how much you agree or disagree with the statements. Please MAKE SURE that you provided any comments or recommendations that may help revise the learning contents of Chapter 5. 1. The difficulty level is appropriate, given the target audience.

Strongly Disagree

Strongly agree

2. The learning content of Chapter 5 is accurate.

Strongly Disagree

Strongly agree

3. The learning content of Chapter 5 is up-to-date.

Strongly Disagree

Strongly agree

4. The learning content of Chapter 5 is closely related to the objectives stated.

Strongly Disagree

Strongly agree

Q1

A Summary of Evaluation of Learning Units: Chapter 5 page 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 5 2nd Session 5

Q2

1st Session 2nd Session

5 5

Q3


5 5

Q4


6 6

Chang S. Nam

Appendices

251

Practice Unit: Practice Sessions Instructions: There are 5 interactive practice sessions implemented in Chapter 5. Please go to each of them, and review them at your own pace. For each of 4 practice session dimensions listed below, assess each practice session by circling the appropriate number to indicate how much you agree or disagree with the statements. Please provide any comments or recommendations that may help revise practice sessions. [Area 1 - Practice 1: Basic Assumptions] 1. The difficulty level is appropriate.

Strongly Disagree

Strongly agree

2. Practice sessions are accurate.

Strongly Disagree

Strongly agree

3. Practice sessions are up-to-date.

Strongly Disagree

Strongly agree

4. Practice sessions provide informative feedback.

Strongly Disagree

Strongly agree

Q1 Q2 Q3 Q4

A Summary of Evaluation of Learning Units: Practice 1 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 4 2nd Session 4 1st Session 2nd Session 1st Session 2nd Session 1st Session 2nd Session

6 5 5 4 6 6

Chang S. Nam

Appendices

252

[Area 2 - Practice 2: Non-Keplerian Effects] 1. The difficulty level is appropriate.

Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree

Q1

A Summary of Evaluation of Learning Units: Practice 2 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 3 2nd Session 4

Q2


Q3


Q4

6 6 5 6

st


6 5

Chang S. Nam

Appendices

253

[Area 3 - Practice 3: Harmonic Corrections to the Ephemerides] 1. The difficulty level is appropriate.

Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree

Q1 Q2

A Summary of Evaluation of Learning Units: Practice 3 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 5 2nd Session 4 1st Session 2nd Session

5 4

Q3


5 5

Q4


5 5

Chang S. Nam

Appendices

254

[Area 4 - Practice 4: Effect of Corrections on Satellite Position] 1. The difficulty level is appropriate.

Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree

Q1 Q2 Q3 Q4

A Summary of Evaluation of Learning Units: Practice 4 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 3 2nd Session 4 1st Session 2nd Session 1st Session 2nd Session st


5 4 5 6 6 6

Chang S. Nam

Appendices

255

[Area 5 - Practice 5: Corrections for Motion during Propagation] 1. The difficulty level is appropriate.

Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree


Strongly Disagree

Strongly agree

Q1

A Summary of Evaluation of Learning Units: Practice 5 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 5 2nd Session 5

Q2


5 5

Q3


5

Q4

st


6 5 6

Chang S. Nam

Appendices

256

Quiz Review Unit: Quiz 1 Review Instructions: Please review the content of Quiz at your own pace. For each of 5 dimensions listed below, rate the quiz session by circling the appropriate number to indicate how much you agree or disagree with the statements. Please provide any comments or recommendations that may help revise the quiz. 1. The difficulty level is appropriate.

Strongly Disagree

Strongly agree

2. The quiz questions are accurate.

Strongly Disagree

Strongly agree

3. The quiz questions are up-to-date.

Strongly Disagree

Strongly agree

4. The quiz provides informative feedback.

Strongly Disagree

Strongly agree

5. The quiz is designed to adequately test student’s learning.

Strongly Disagree

Strongly agree

Q1

A Summary of Evaluation of Learning Units: Quiz 1 Review 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 6 2nd Session 5 1st Session 2nd Session

5

Q3


5 5

Q4


5 5

Q5


5

Q2

6

6

Chang S. Nam

Appendices

257

Laboratory Exercise Unit: Prelaboratory 5 Instructions: Please review the MATLAB Script. Assess informative feedback provided by the MATLAB Script page by circling the appropriate number to indicate how much you agree or disagree with the statement. Please also provide any comments or recommendations that may help revise the MATLAB script. 1. The MATLAB script session provides informative feedback.

Strongly Disagree

Strongly agree

Q1

A Summary of Evaluation of Learning Units: Prelaboratory 5 1- Strongly Disagree 2 3 4 5 6 - Strongly Agree 1st Session 5 2nd Session 6

Chang S. Nam

Appendices

258

Appendix 9. Task Instructions [1] Task Instruction for Instructional Design Expert Thank you for agreeing to participate in this experiment. As an instructional design expert, you will be asked to: 1. 2. 3.

Review the instructional design in the instructional system of the WD2L environment, Identify overlooked areas or problems in need of improvement, and Provide comments or recommendations that will help revise the instructional design.

Since Gagne’s nine events of instruction as the instructional strategy was the foundation for developing the materials, the value of the instructional design will be determined by evaluating how well components of the instructional strategy are implemented. [2] Task Instructions for Content Expert Thank you for agreeing to participate in this experiment. As a GPS content expert, you will be asked to: 1. 2. 3.

Review following 4 types of learning units in the instructional system of the WD2L environment, Identify overlooked areas or problems in need of improvement, and Provide comments or recommendations that will help revise the learning units.

The learning contents that you will evaluate include: • • • •

Content Unit: Chapter 5 (Corrections to Keplerian Orbits: Precise Positioning) Practice Unit: 4 Practice Sessions in Chapter 5 Quiz Review Unit: Quiz 1 Review Laboratory Exercise Unit: Prelaboratory 5 [3] Task Instruction for User Interface Expert

Thank you for agreeing to participate in this experiment. As a user interface design expert, you will be asked to: 1. 2. 3.

Review the user interface system in the WD2L environment, Identify overlooked areas or problems in need of improvement, and Provide comments or recommendations that will help revise the user interface system.

For all SMEs To help you critique all of the important aspects of the instructional design, the following materials are provided: • •

USER PROFILE DOCUMENT: Description of user classes, characteristics, and motivation. INSTRUCTIONAL DESIGN REVIEW FORM: Specific descriptions on each instructional strategy.

First, log in the WD2L environment by using cnam as VT PID and andrew as password. Second, review each component of the instructional design, and provide comments or recommendations that will help revise the instruction design. Finally, evaluate the overall quality of the instructional design by circling the appropriate number to indicate how much you agree or disagree with the statements in the form. Please MAKE SURE that you have provided your comments or recommendations that will help revise the instructional design. If you have any questions while evaluating the instruction design, please send email to the principal investigator, Chang S. Nam ([email protected]), or call him at 231-9065.

Chang S. Nam

Appendices

259

Appendix 10. IRB Protocol Informed Consent for Participant of Investigative Project Virginia Polytechnic Institute and State University Title of Project: A Web-based Distance and Distributed Learning Environment Principal Investigator: Chang S. Nam THE PURPOSE OF THIS RESEARCH You are invited to participate in the evaluation of the Web-based Distance and Distributed Learning (WD2L) environment. The WD2L environment will support learning activities in the GPS education, in which engineering students, including distant learners, can easily pull together many fundamental concepts and effectively gain handson experience with the GPS technology at their own pace in their own time. The purpose of this study is to evaluate and improve the WD2L environment, making it a valuable and useful learning environment for engineering students. PROCEDURES The procedures for this experiment are addressed as follows. If you have any questions related to any of the procedures or purpose of this experiment after reading this document, please feel free to ask them. If you wish to be a participant in this experiment after reading this form, sign your name on the designated line of the following page. In this project, you will participate in one of four studies. Study 1 and 4 are expert review sessions where participants (called ‘Subject Matter Experts’) discover overlooked areas or problems of the WD2L environment prototype. To help experts critique all of the important aspects of the WD2L environment, checklist types of supporting tools will be provided. Second, in the one-to-one evaluation session, participants will be asked to complete the benchmark tasks for laboratory usability testing. Before doing that, the participants will be asked to think aloud throughout the whole session, talking about what they are doing, why they are doing it, and what they expect to happen when they perform an action. Third, in the small group evaluation session, a group of participants evaluate group learning activities and multi-user interfaces in each system, performing the benchmark tasks related to group learning activities. The entire session will be video- and audiotaped for later examination. At the end of each evaluation session lasting about 1.5 hours, a demographic questionnaire will be administered. The participant will then be debriefed and released. RISKS Participation in this study does not place you at more than minimal risk of harm. BENEFITS OF THIS RESEARCH You will be compensated for your participation, and you will be given information to contact the principal investigator to get information about the outcomes of the study. You will also benefit from knowing that you have participated in worthwhile research that has immediate and positive applications. EXTENT OF CONFIDENTIALITY/ANONYMITY The information gained in this study will be kept strictly confidential. At no time will the researcher release any data identified with you to anyone other than personnel working on the project without your written consent. You will be identified only by a 3-digit study code. Data will be stored securely and will be made available only in the context of research publications and discussion. No reference will be made in oral or written reports that could link you to the data nor will you ever be identified as a participant in the study. This study will use digitized portions of the videos of experimental sessions. However, faces will be blurred or obscured so that you cannot be identified. However, your voice may be used to present specific issues requiring design attention. If you feel uncomfortable with the display of your voice to other groups, please indicate on the next page.

Chang S. Nam

Appendices

260

COMPENSATION You will be compensated at the rate of $10 per hour for participation in this study. FREEDOM TO WITHDRAW You are free to withdraw from this study at any time without penalty. APPROVAL OF THIS RESEARCH This research has been approved, as required, by the Institutional Review Board for Research Involving Human Subjects at Virginia Polytechnic Institute and State University, and by the Department of Industrial and Systems Engineering. PARTICIPANT’S RESPONSIBILITIES It is very important that you keep the activities and information discussed confidential, since others will be participating in this study. QUESTIONS If you have questions, or do not understand information on this form, please feel free to ask them now. PARTICPANT’S PERMISSION I have read and understand the Informed Consent and conditions of this study. I have had all questions answered. I hereby acknowledge the above and give my voluntary consent for participation in this study. If I participate, I may withdraw at any time without penalty. _____________________________________ ____________________ Signature Date PARTICIPANT’S PERMISSION TO USE EXCERPTS FROM VIDEOTAPED SESSIONS (FACE WILL BE OBSCURED OR BLOCKED) I have read and understand the manner in which videos will be used for subsequent presentation of information related to this study. I understand that my face will not be identifiable because it will be obscured or blocked. I grant permission to researchers to present this information as necessary in the manner described on this form. _____________________________________ ____________________ Signature Date I do not grant permission to researchers to present this information as necessary in the manner described on this form. _____________________________________ ____________________ Signature Date CONTACT If you have any questions at any time about the study or the procedures, you may contact the principal investigator, Chang S. Nam at 231-9065 or [email protected] (562 Whittemore). If you feel you have not been treated according to the descriptions in this form, or your rights as a participant have been violated during the course of this study, you may contact Dr. David Moore, Chair of the Institutional Review Board Research Division at 231-4991.

Chang S. Nam

Appendices

261

Appendix 11. Benchmark Tasks [Benchmark Tasks for the One-to-One Evaluation] Benchmark Task #1 Task Statement: Find out pages that contain all words, “global positioning systems.” Objective: This benchmark task is designed to verify that searching information on the “GPS Theory & Design” website is clear. Steps involved: • Select the “Site” option in the pull-down menu • Click the text field • Type the keyword(s), “global positioning systems”, in the text field • Click the “GO” button Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The “Search Results” screen displaying a list of documents with words “global positioning systems” Possible errors: • Fail to use the pull-down menu • Fail to type correct keyword(s) in the text field

• Selects the wrong search option • Fail to click the “GO” button

Type of function to be supported: • Information Processing Functions: Seeking & Collecting Benchmark Task #2 Task Statement: Assume that you have already done your homework #1, and saved it in the following directory, C:\Documents and Settings\cnam\My Documents\Homework\HW1. Put this homework on the Web so that the instructor can get it. Objective: This benchmark task is designed to show if the mechanism of uploading assignments on the Web is clear. Steps involved: • Select the “Assignments” menu item from the “Assessments” menu • Click the “Add file” button • Click the “Browse” button • Select the file in the directory • Click the “Add” button • Click the “Send” button Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The “Confirmation” screen, indicating that the file has successfully been saved on the Web Possible errors: • Fail to select the “Assignments” menu item from the “Assessments” menu • Fail to click the “Browse” button • Fail to click the “Add” button

• Fail to click the “Add file” button • Fail to select the file • Fail to click the “Send” button

Chang S. Nam

Appendices

262

Type of function to be supported: • Information Processing Functions: Collecting Benchmark Task #3 Task Statement: Find out a Website or any GPS link providing information about GPS product developers & retailers. Objective: This benchmark task verifies if it is clear that GPS links and related sites would be available in the “GPS Resources” page, and that it is easy to access and find. Steps involved: • Move the mouse to the “Classroom” menu • Move the mouse to the “Resource Materials” menu item • Select the “GPS Resources” menu item • Click the “GPS product developers & retailers” link in the “GPS Resources” page Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The “Search Results” screen displaying a list of GPS links and related sites Possible errors: • Fail to go to the “Classroom” menu • Fail to select the “GPS Resources” item

• Fail to move the mouse to “Resource Materials” item • Fail to click the “GPS product developers & retailers” link

Type of function to be supported: • Information Processing Functions: Seeking, Collecting & Integrating Benchmark Task #4 Task Statement: Study Chapter 5: Corrections to Keplerian Orbits: Precise positioning at your own pace. However, skip practice sessions in the “Chapter 5” page. Tell the observer when you are done. Objective: This benchmark task verifies that the learning content is clear to learn. Steps involved: • Move the mouse to the “Classroom” menu • Move the mouse to the “Lecture Notes” menu item • Select the “Chapter 5 ” menu item • Press the “Page Down” button or scroll down the page Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: User indicates verbally that the task is done. Possible errors: • Fail to go to the “Classroom” menu • Fail to select the “Chapter 5” menu item • Fail to move the mouse to the “Lecture Notes” menu item Type of function to be supported: • Scaffolding Functions: Procedural, Conceptual, & Metacognitive

Chang S. Nam

Appendices

263

Benchmark Task #5 Task Statement: Assume that you want to practice what you have learned in the “Chapter 5” page. Perform the practice 1: Basic assumptions. Objective: This benchmark task is designed to show that practice sessions are easy to perform. Steps involved: • Click the “Let’s Practice: Basic Assumptions” image • Select a correct answer to a question • Read feedback on the response • Click the “Return to Ch.5 Corrections to Keplerian Orbits” link, or • Click the “Let’s Practice: Basic Assumptions” image • Select a correct answer to a question • Read feedback on the response • Click the “Back” button on the browser menu Starting point: The “Chapter 5: Corrections to Keplerian Orbits: Precise positioning” screen Ending point: The “Chapter 5: Corrections to Keplerian Orbits: Precise positioning” screen Possible errors: • Fails to click the “Let’s Practice: Basic Assumptions” image • Fails to click the “Return to Ch.5 Corrections to Keplerian Orbits” link, or click the “Back” button Type of function to be supported: • Information Processing Functions: Manipulating

• Scaffolding Functions: Metacognitive

Benchmark Task #6 Task Statement: Ask one of your classmates, CS, if he can meet you at 10:00am in the library via an email. His email address is [email protected]. Objective: To verify that the way to communicate via an email is clear and usable. Steps involved: • Move the mouse to the “Communication” menu • Select the “Send Email” menu item from the menu • Click the “New message” button • Type the content of the mail • Click the “Send” button Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The message screen displaying confirmation. Possible errors: • Fail to choose the “Communication” menu • Fail to click the “New message” button

• Fail to select the “Send Email” menu item from the menu • Fail to click the “Send” button

Type of function to be supported: • Information Processing Functions: Integrating & Communicating

Chang S. Nam

Appendices Benchmark Task #7

Task Statement: You are asked to take the quiz by the instructor. Answer all quiz questions. Objective: This benchmark task is designed to show if the interface clearly indicates how to take the quiz. Steps involved: • Move the mouse to the “Assessments” menu • Select the “Quiz” menu item from the “Assessments” menu • Select correct answers to quiz questions • Click the “Submit” button Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The confirmation screen displaying the message that the quiz has been sent Possible errors: • Fail to move the mouse to the “Assessments” menu • Fail to click the “Submit” button • Fail to select the “Quiz” menu item from the “Assessments” menu Type of function to be supported: • Scaffolding Functions: Metacognitive Benchmark Task #8 Task Statement: You are asked to complete the existing MATLAB templates before arriving in the laboratory. Complete the “findsat” MATLAB template. Objective: This benchmark task verifies that the mechanism for completing the existing MATLAB templates is usable and clear. Steps involved: • Move the mouse to the “Laboratory” menu • Move the mouse to the “MATLAB Exercises” menu item from the “Laboratory” menu • Select the “MATLAB Script” menu item • Fill out text fields in the “MATLAB Script” Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The MATLAB script screen Possible errors: • Fail to move to the “Laboratory” menu • Fail to move to the “MATLAB Exercises” menu item Type of function to be supported: • Information Processing Functions: Manipulating

• Fail to select the “MATLAB Script” menu item

• Scaffolding Functions: Metacognitive

264

Chang S. Nam

Appendices

265

[Benchmark Tasks for the Small Group Evaluation] Group Benchmark Task (Group Leader) Task Statement: Assume that you and your group members must decide your group’s opinion about a proposed topic, “Future of GPS applications”, through group discussion. As a group leader, manage the discussion among group members and also express your own opinion by using all functions provided by the Discussion Board. Once each group member provided her/his own opinions at least four times, then summarize discussion contents, and ask group members which one should be your group’s opinion. Objective: This benchmark task is designed to verify that the way to communicate with group members in the Discussion Board is clear and usable. Steps involved: • Move the mouse to the “Communication” menu • Select the “Discussion Board” menu item from the menu • Click the “New message” button • Type the message • Click the “Submit” button Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The main screen displaying a list of discussion topics Possible errors: • Fail to choose the “Communication” menu • Fail to select the “Discussion Board” menu item from the menu

• Fail to click the “New message” button • Fail to click the “Submit” button

Type of function to be supported: • Information Processing Functions: Integrating & Communicating Group Benchmark Task (Group member 1) Task Statement: Assume that you and your group members must decide your group’s opinion about a proposed topic, “Future of GPS applications”, through group discussion. You group leader will manage the discussion among group members. Using all functions provided by the Discussion Board, decide your group’s opinion about the topic, “Future of GPS applications.” Objective: This benchmark task is designed to verify that the way to communicate with group members in the Discussion Board is clear and usable. Steps involved: • Move the mouse to the “Communication” menu • Select the “Discussion Board” menu item from the menu • Click the “New message” button • Type the message • Click the “Submit” button Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The main screen displaying a list of discussion topics

Chang S. Nam Possible errors: • Fail to choose the “Communication” menu • Fail to click the “New message” button

Appendices

266

• Fail to select the “Discussion Board” menu item from the menu • Fail to click the “Submit” button

Type of function to be supported: • Information Processing Functions: Integrating & Communicating Group Benchmark Task (Group member 2) Task Statement: Assume that you and your group members must decide your group’s opinion about a proposed topic, “Future of GPS applications”, through group discussion. You group leader will manage the discussion among group members. Using all functions provided by the Discussion Board, decide your group’s opinion about the topic, “Future of GPS applications.” Objective: This benchmark task is designed to verify that the way to communicate with group members in the Discussion Board is clear and usable. Steps involved: • Move the mouse to the “Communication” menu • Select the “Discussion Board” menu item from the menu • Click the “New message” button • Type the message • Click the “Submit” button Starting point: The screen with statement(s) given by either the instructor or GTA today Ending point: The main screen displaying a list of discussion topics Possible errors: • Fail to choose the “Communication” menu • Fail to click the “New message” button

• Fail to select the “Discussion Board” menu item from the menu • Fail to click the “Submit” button

Type of function to be supported: • Information Processing Functions: Integrating & Communicating

Chang S. Nam

Appendices

267

Appendix 12. Data Collection Sheet TASK NAME: Critical Incident Description

Critical Incident Evaluation Sheet Participant ID:___________ Date:_______ ___ Time to Perform Task:_____________ No. of errors:___________ Evaluators Comments

1. 2. 3. 4. 5. 6.

Appendix 13. Sample Questions of Instruction Evaluation 1. Overall, how appropriate was the vocabulary level of the content to you?

Very Inappropriate

Very Appropriate

2. Overall, how appropriate was the complexity level of the content to you?

Very Inappropriate

Very Appropriate

3. Overall, how appropriately was the content elaborated?

Very Inappropriate

Very Appropriate

4. How would you rate the appropriateness of examples that the content provided?

Very Inappropriate

Very Appropriate

Chang S. Nam

Appendices

268

Appendix 14. Sample Questions of the QUISTM Copyright © 1989, 1993, 1998 University of Maryland. All Rights Reserved. PART 3: Overall User Reactions Please circle the numbers which most appropriately reflect your impressions about using this computer system. Not Applicable = NA. 3. Overall reactions to the system: 3.2 3.3

terrible wonderful 1 2 3 4 5 6 7 8 9

NA

frustrating satisfying 1 2 3 4 5 6 7 8 9

NA

dull stimulating 1 2 3 4 5 6 7 8 9

3.4 3.5

3.6

NA

difficult easy 1 2 3 4 5 6 7 8 9

NA

inadequate adequate power power 1 2 3 4 5 6 7 8 9

NA

rigid flexible 1 2 3 4 5 6 7 8 9

NA

PART 4: Screen 4.1 Characters on the computer screen 4.2 Highlighting on the screen 4.3 4.4

Screen layouts were helpful Sequence of screens

hard to read easy to read 1 2 3 4 5 6 7 8 9

NA

unhelpful helpful 1 2 3 4 5 6 7 8 9

NA

never always 1 2 3 4 5 6 7 8 9

NA

confusing clear 1 2 3 4 5 6 7 8 9

NA

Please write your comments about the screens here: _________________________________________________________________________________________ _________________________________________________________________________________________ _____________________________________________________________________________________

Chang S. Nam

Appendices

269

Appendix 15. Scoring Rubrics for Transfer Test Question 1: Explain what a harmonic correction to a GPS Satellite ephemeride is. Answer A Answer B Answer C 3 points 2 points 1 points “A correction that “Non-gravitational “A harmonic correction accounts for the twice-per- correction. Corrects for takes into accounts small orbit speeding up/slowing time differences in nondistances that the satellite down that a satellite gravitational perturbations is projected to make off encounters.” between the satellite and courses.” receiver.”

Answer D 1 points “A harmonic correction involves finding the right harmonic in order to acquire the GPS satellite ephemeride.”

Question 2: List and briefly describe at least two gravitational effects that perturb GPS satellite orbits. Answer A Answer B Answer C Answer D 3 points 2 points 1 points 1 points “The non-sphericity of the “J2 and Tidal effects. As “Tidal effects – it puts a “The equatorial bulge Earth causes the the Earth spins on its axis, gravitational pull on the causes the satellite to gravitational pull by the forces cause a bulge at the earth and also causes a speed up.” Earth to vary latitude. equator of the Earth. pull on the satellite.” Additionally, the Sun and Therefore the equatorial Moon’s gravitation pull phase is significantly draws the satellite closer to larger that the polar those celestial bodies, just planes. Tidal effects are like tides.” caused by the gravitational forces between the Sun, Moon and Earth.” Question 3: List and briefly describe at least two non-gravitational effects that perturb GPS satellite orbits. Answer A Answer B Answer C Answer D 3 points 2 points 1 points 1 points “1. Solar radiation “Solar radiation pressure “Position of the satellite as “Solar radiation/energy pressure – The force forces the satellite compared to the Earth and perturb the satellite’s caused by solar radiation earthward. This can vary how it changes.” orbit.” hitting the satellite causes greatly with solar flares.” the orbit to sway from the ideal path. Very small but builds up over time. 2. Relativistic effects – These are caused by the fact that the receiver is moving when it transmit the sate, and the fact that the Earth is moving when the data was sent and when the data is received.”

Chang S. Nam

Appendices

270

Appendix 16. Sample Questions of Web-based Learning Questionnaire Adapted with permission from CAL Evaluation by Gregor Kennedy, Biomedical Multimedia Unit, University of Melbourne. Name:_____________

Age:___________ Sex: ___________

Approximately how many hours have you spent to further study Chapter 5? ( ) less than 5 ( ) 5-10 ( ) 10-20 ( ) 20-30 ( ) 30+ Which of the following instructional methods do you prefer? (1) in addition to face-to-face teaching with textbook and lecture notes (2) in addition to face-to-face teaching with the Web-based supplemental learning program (used on my own) (3) in addition to face-to-face teaching with the Web-based supplemental learning program (used in class)

A) Introductory Objectives and Directions 1) From the start it was clear what I was going to do in the program.

Strongly Disagree

Strongly agree

2) From the start it was clear what the aims of the program were.

Strongly Disagree

Strongly agree

3) I knew enough about the content area to get the most out of this program.

Strongly Disagree

Strongly agree

B) Content and Structure 1) Generally the content was clear and logically organized.

Strongly Disagree

Strongly agree

Chang S. Nam

Appendices

271

2) Important information or key concepts were easy to identify

Strongly Disagree

Strongly agree

3) The information presented in the program was relevant to my learning goals.

Strongly Disagree

Strongly agree

4) The flow of information from step to step was clear.

Strongly Disagree

Strongly agree

5) Generally there was just the right amount of information on each screen

Strongly Disagree

Strongly agree

6) I thought the content was pitched at an appropriate level.

Strongly Disagree

Strongly agree

7) The web material reinforced what I had learned elsewhere.

Strongly Disagree

Strongly agree

8) I found the web material useful for revision

Strongly Disagree

Strongly agree

Chang S. Nam

VITA

272

VITA Chang S. Nam

EDUCATION: 2000 - 2003

Ph.D., Industrial & Systems Engineering; Human Factors Option - Virginia Polytechnic Institute and State University. Blacksburg, VA.

1998 - 2000

M.S., Industrial Engineering; Human Factors Option - State University of New York at Buffalo. Amherst, NY.

1994 - 1996

M.A., Management School; Organizational Behavior & Human Resource Management Option - Sogang University. Seoul, Korea.

1987 - 1994

B.S., Industrial Engineering - Sungkyunkwan University. Seoul, Korea.

HONORS and AWARDS: 2002 - 2003

United Parcel Service (UPS) Fellowship Award, Virginia Polytechnic Institute and State University

SELECTED EXPERIENCE: Aug. 2003 - Present

Aug. 2003 - Present

Aug. 2003 - Present

Human-Computer Interaction/Assessment and Cognitive Ergonomics Laboratory, Virginia Polytechnic Institute and State University Instructor • Design and teach a supplemental course on experimental designs and statistical analyses for human factors research using the SAS system during Fall of 2003 for human factors graduate students Human-Computer Interaction/Assessment and Cognitive Ergonomics Laboratory, Virginia Polytechnic Institute and State University Research Assistant • Conduct research on the design of web-based supplementary course materials to aid students in understanding of complex Global Positioning System (GPS) concepts • Apply Human Factors and Instructional Design principles to develop GPS concepts for educational outreach Locomotion Research Laboratory, Virginia Polytechnic Institute and State University Research Assistant • Conduct research on age-related visual perception characteristics associated with daytime and nighttime driving conditions • Apply auditory warning signal principles to explore the design attributes and parameters of in-vehicle visual and auditory displays

Chang S. Nam June 2002 - Present

June 2002 - June 2003

Jan. 2002 - May 2002

Sep. 2001 - Dec. 2001

Jan. 2001 - May 2001 Jan. 2001 - May 2001

Jan. 2000 - June 2000

Aug. 1994 - Aug. 1995

Jan. 1994 - July. 1994

VITA

273

Human-Computer Interaction/Assessment and Cognitive Ergonomics Laboratory, Virginia Polytechnic Institute and State University Systems Administrator • Assist with day-to-day laboratory operations Human-Computer Interaction/Assessment and Cognitive Ergonomics Laboratory, Virginia Polytechnic Institute and State University Graduate Research Assistant • Conducted usability research on the software user interface and hardware of existing cell phones • Applied Human Factors principles to the usability inspection, testing, and evaluation of cell phone user interfaces • Supervised project team as a team manager • Prepared several proposals for and conducting research and product development projects Industrial and Systems Engineering Department, Virginia Polytechnic Institute and State University Graduate Teaching Assistant: Human Computer Systems and Principles of Industrial Hygiene Industrial and Systems Engineering Department, Virginia Polytechnic Institute and State University Graduate Teaching Assistant: Work Measurement and Methods Engineering • Taught various basic industrial engineering tools, work analysis, data acquisition and application, performance evaluation and appraisal, and work measurement procedures Industrial and Systems Engineering Department, Virginia Polytechnic Institute and State University Graduate Teaching Assistant: Human Information Processing Human-Computer Interaction Laboratory, Virginia Polytechnic Institute and State University Graduate Research Assistant • Developed a health care web site for older adults • Conducted usability evaluation Industrial Engineering Department, State University of New York at Buffalo Graduate Research Assistant • Conducted research activities including the development of a simulation model, data collection, and data analysis for a manufacturing process Management School, Sogang University Research Assistant • Conducted several business process revolution (BPR) and management innovation projects • Designed surveys, conducted interviews, performed data collection and analysis, and prepared reports Management School, Sogang University Research Assistant: The Institute for Labor and Management (ILM)

Chang S. Nam

VITA

274

PROFESSIONAL EXPERIENCE: 1996 – 1998

1993

Dong-Ah Securities, Co. Ltd., Research Department, Seoul, Korea Analyst: • Performed business analysis for Steel, Shipbuilding, and Automobile business sectors • Published investment reports for both domestic and foreign investors LG Group, Seoul, Korea Student Intern

GRADUATE CURRICULUM: Human Factors Engineering • Models of Human-Computer Interaction • Usability Engineering • Human Information Processing • Human Computer Interaction • Human Factors System Design I & II • Human Factors Research Design I & II • Human Factors in Visual Display Systems • Integrated Systems Design • Human Audition and Auditory Display Design • Training System Design • Human Factors Organizational Found • Human Physical Capabilities • Occupational Safety & Hazard Control • Applied Multivariate Analysis • Work Physiology • Human Factors Research Methodology • Special Topics in Human Factors

Management Systems Engineering, Operations Research, & Manufacturing Systems Engineering • Theories of Manpower Development • Database Management System • Seminar on Organizational Behavior • Business Research Method • Theory of Organizational Development • Linear Programming • Simulation and Stochastic Models

RESEARCH GRANT: “A Web-based Distance and Distributed Learning Environment to Support the Global Positioning System (GPS) Education,” Tonya L. Smith-Jackson, T. L, Ph.D., Wayne A. Scales, Ph.D., & Chang S. Nam, Research Assistant, The ASPIRES (A Support Program for Innovative Research Strategies) Grant Program From Virginia Polytechnic Institute and State University ($65,000; January, 2003).

PUBLICATIONS (Peer Reviewed Journals and Proceedings): Thomas, C., Nam, C. S., & Smith-Jackson, T. L. (2003). The significance of behavior type on Web information retrieval and academic success. Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition (Session 2793; Section 612). American Society for Engineering Education.

Chang S. Nam

VITA

275

Nam, C. S., Kim, H. N., Smith-Jackson, T. L., & Nussbaum, M. A. (2003). Development of a guidelines tool for mobile phone interfaces. Proceedings of the Human Factors and Ergonomics Society’s 47th Annual Meeting (pp. 792-796). Santa Monica, CA: Human Factors and Ergonomics Society. Nam, C. S., & Bisantz, A. M. (2002). A lens model analysis of confidence judgments: Beyond calibration measures. Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 506-510). Santa Monica, CA: Human Factors and Ergonomics Society.

TECHNICAL REPORTS: Smith-Jackson, T. L., Williges, R. C., kwak, J. Capra, M., Durak, T., Nam, C. S., & Ryu, Y. S. (2001). User requirements specification for a prototype healthcare information Web site and an online assessment tool: final version. Technical Report ACE/HCIL-01-01. Assessment and Cognitive Ergonomics Laboratory and Human-Computer Interaction Laboratory, Virginia Tech Nam, C. S. (2000). The effects of the task characteristics on judgmental expressions of confidence. Unpublished Master Thesis (Master of Science). State University of New York at Buffalo, Amherst, NY. Nam, C. S. (1996). A study on the effects of organization members’ exchange quality and problem solving style on individual innovation behavior. Unpublished Master Thesis (Master of Art). Sogang University, Seoul, Korea.

COURSES AND MINICOURSES TAUGHT: • Work Measurement and Methods Engineering • Conducting Human Factors Research Using the SAS System (Supplemental Course)

MEMBERSHIP: • •

Human Factors and Ergonomics Society (HFES) National Chapter

Virginia Tech Student Chapter of the Human Factors and Ergonomics Society

A Theory-Based Integrated Design Process for ... - CiteSeerX

A Theory-Based Integrated Design Process for ... - CiteSeerX

Suggest Documents

A Theory-Based Integrated Design Process for ... - CiteSeerX

Integrated Business-Process-Driven Design for Service ... - CiteSeerX

Integrated Business-Process-Driven Design for Service ... - CiteSeerX

Economical Design Process for Integrated Single ...

The Integrated Design Process - iiSBE

A Theory-Based Integrated Design Process for ... - Virginia Tech

Serendipity: integrated environment support for process ... - CiteSeerX

lessons learned in integrated product and process design ... - CiteSeerX

and Process-Variation Aware Design of Integrated Circuits - CiteSeerX

vcm process design - CiteSeerX

Sustainable Integrated Process Design and Control ...

A Contextual Approach for Process-Integrated Tools

Computer-Aided Integrated Design of Chemical Process for Diels ...

Integrated design and process analysis of a blow ...

PRODUCT ARCHITECTURE DESIGN PROCESS FOR ... - CiteSeerX

Process Design and Economics for Biochemical ... - CiteSeerX

An Integrated Conceptual Design Process for Modeling Interactions ...

Integrated Design and Optimization Models for the Six Sigma Process

AN INTEGRATED TRIM AND STRUCTURAL DESIGN PROCESS FOR ...

THE DESIGN SCIENCE RESEARCH PROCESS: A ... - CiteSeerX

A Business Process Design Language - CiteSeerX

Integrated Product and Process Design Environment Tool for ...

Integrated Process and Product Design Optimization: a Cosmetic ...

The Integrated Process Model For