Developing High-Quality Educational Software - Semantic Scholar

Educational Methodologies

Developing High-Quality Educational Software Lynn A. Johnson, Ph.D.; Titus K.L. Schleyer, D.M.D., Ph.D. Abstract: The development of effective educational software requires a systematic process executed by a skilled development team. This article describes the core skills required of the development team members for the six phases of successful educational software development. During analysis, the foundation of product development is laid including defining the audience and program goals, determining hardware and software constraints, identifying content resources, and developing management tools. The design phase creates the specifications that describe the user interface, the sequence of events, and the details of the content to be displayed. During development, the pieces of the educational program are assembled. Graphics and other media are created, video and audio scripts written and recorded, the program code created, and support documentation produced. Extensive testing by the development team (alpha testing) and with students (beta testing) is conducted. Carefully planned implementation is most likely to result in a flawless delivery of the educational software and maintenance ensures up-to-date content and software. Due to the importance of the sixth phase, evaluation, we have written a companion article on it that follows this one. The development of a CD-ROM product is described including the development team, a detailed description of the development phases, and the lessons learned from the project. Dr. Johnson is Associate Professor and Director, Office of Dental Informatics, School of Dentistry, University of Michigan; Dr. Schleyer is Associate Professor and Director, Center for Dental Informatics, School of Dental Medicine, University of Pittsburgh. Direct correspondence and requests for reprints to Dr. Lynn Johnson, School of Dentistry, Office of Dental Informatics, B322D, University of Michigan, 1011 North University Avenue, Ann Arbor, MI 48109-1078; 734-615-7388 phone; 734-615-7135 fax; [email protected]. Key words: computer-based assessment, computer-assisted instruction, dentistry, dental education, educational software, evaluation, software development Submitted for publication 5/5/03; accepted 8/28/03

C

ourse management tools and educational software are two distinct types of software used in teaching and learning. Course management tools such as Blackboard, WebCT, and TopClass help faculty to manage and organize course resources such as syllabi, assignments, manuals, and links to other website, as well as assist in communicating with the students.1-3 These software tools help to increase students’ access to course resources. The primary goal of educational software, on the other hand, is to teach and/or assess students. These products include tutorials, hypermedia, drill and practice applications, simulations, games, and assessments. They can be delivered through course management systems or separately through DVDs, CD-ROMs, or websites. Dental faculty members frequently use administrative software and course management tools with great success. Most dental schools and/or their associated university usually provide the hardware, software, and personnel resources to support these programs. The development of educational software,

November 2003

■

Journal of Dental Education

however, requires a diverse skill set that faculty rarely have the time to develop and few dental schools can afford to support. To promote the creation of highquality educational software, media development professionals such as instructional designers and web specialists, multimedia artists, and software programmers should be key members of the project team who work in conjunction with faculty.4 An instructional designer usually has earned a Master’s in instructional design or has specialized training in educational psychology and learning technologies. A multimedia artist usually has an art degree with specialized training in image processing, digital video, and/or animation. The programmer may have a degree in computer science. Frequently, the programmer is also the instructional designer with specialized training in databases and programming languages, such as HTML or Java. An experienced programmer will be more effective and productive with software tools than a novice. While a program developed with junior technical personnel may satisfy the requirements of the local institution, it often

1209

is inadequate for general and widespread distribution. This article describes a proven process used by experienced educational software developers to create high-quality products. It is intended for dental faculty and administrators who are interested in gaining a better understanding of the software development process that will solve an instructional need in dentistry. High-quality software can have many definitions. Some consider programs that are free of programming errors to be of adequate quality for students. Others would say that software must only accomplish the intended teaching and learning goals to be of high quality. Is software that teaches, but is difficult to use, high quality? If photographs are used to demonstrate a concept because a specialist in 3-D animations is not available, is the software of lesser quality? Is a program with rich professional graphics of higher quality than software with a simple interface, but depth content? The debate can easily continue without resolution. Table 1 summarizes eight categories of criteria that indicate high-quality educational software as defined in the ANSI/ADA Specification 1001 Guidelines for the Design of Educational Software.5,6 These guidelines are sponsored by the American Dental Association’s Standards Committee for Dental Informatics that is accredited by the American National Standards Institute. They also form the core judging criteria for the American Dental Education Association’s Educational Software in Dentistry Competition.7

The Educational Software Development Team The most critical element of the development process is the team. The size and nature of the team will vary based on the project budget, schedule, audience, and nature of the project. Some dental institutions have created their own software development team, while others use central university personnel. In either case, certain core skills are required for any successful technology project. Table 2 lists these core skills. One clear sign of a successful team is that its members share the same project vision. Team members must know what they are trying to accomplish and what the finished product will look like.

1210

Steps for Success There are numerous benefits of having a development process in place: it gives the team a common focus, facilitates budgeting and progress reporting, maintains professional standards, increases the speed of development, and makes for a smoother transition during personnel changes.8 A six-step process for software development is described in the following sections of this article. This approach blends educational best practices with the practicalities of software development. The end goal is to develop innovative, educationally sound curricula. Table 3 provides a summary of the development process, based on the work of Alessi and Trollip and McCarthy.4,9

Analysis Analysis is the most important activity in the entire development process. During this phase the development team lays the foundation for the project. The better the planning, the better the quality of the final product, and the more you will be assured of attaining your educational goals. Throughout this phase, give careful consideration to whether the computer is really the best medium to deliver this instruction. Define project audience and goals. This first step sounds very simple, but it is essential. The team should immediately answer the following questions and constantly refer back to them during the entire design and development process: What students will use this material? What should students be able to do/know after completing the lesson? Also, clarify the students’ background knowledge. Determine constraints. Constraints include the operating system of the computers that will deliver the program (Macintosh or Windows), the speed of a modem connection (phone lines, broadband, or local area network), style requirements, and the method of delivery (CD-ROM, World Wide Web, or interactive television). For web applications, decide which browsers to target. With multimedia applications, video requires special considerations that need to be carefully planned. Confirm delivery medium. Because educational software is more expensive to develop than other forms of instruction, confirm that the instruction requires the unique capabilities of the computer and cannot be made available in another medium such

Journal of Dental Education ■ Volume 67, Number 11

Table 1. Summary of characteristics of high-quality educational software Pedagogical Issues • Program provides learning capabilities not achieved with other media. • Instructional methodologies match the content and audience needs. • Practice reinforces learned content. • Individual lessons are short and spread out over a number of sessions. • Lessons are customizable to the learner when possible, e.g., pretests customize the material to the student. • Students progress at their own pace. • Help and directions are always available. • Controls are built in for playing movies, animations, and audio. • The interface is easy to use and guards against data loss. • Novelty is used prudently. • Interactions are frequent, varied, and have an instructional purpose. • Interactions require the learner to apply information. • New learning is applied repeatedly over time. Subject Matter • Goals and objectives are stated and focused on throughout the program. • Information is accurate, well organized, appropriately sequenced, and matches the level of the intended audience. • Prerequisites are stated and built upon. • References are provided. Language and Grammar • Text is understandable. • Reading level is appropriate for the audience and content. • Technical terms and cultural bias are avoided. • Glossary contains definitions and abbreviations. • Correct spelling, grammar, and punctuation are used. • Consistent writing style is used. • Space enhances learning. Surface Features (program characteristics that are visible to the student) • Screen displays are uncluttered, aesthetic, and draw attention to important information. • Media (text, video, audio, and animations) support the learning tasks. • Student is protected from irreversible errors (e.g., ending a test early). • Menus are clear. • Incorrect choices are changed without penalty. • Bookmarks show completed sections. • Input is effortless. • Ending is clearly indicated. Questions, Answers, and Feedback • Questions reflect the lesson objectives. • Questions are interspersed throughout the lesson. • Answering questions is simple and easy. • Answers are clearly marked. • Learners do not have to answer a question correctly before proceeding. • Correct answers are confirmed. • Feedback differentiates between inappropriate entries and incorrect responses. • Feedback is informative, corrective, clear, and response-specific. Invisible Functions (functions that operate behind the scenes) • Identifying information is removed when data is used for purposes other than tracking performance. • Data collection can be turned off. • Data is accessible only to authorized personnel. • Data is stored continuously. Offline Materials • Are well written and follow traditional conventions. • Include equipment needs, installation procedures, troubleshooting information, and technical support resources. • Include suggestions for curriculum integration. Evaluation • Formative evaluation identifies design flaws during development. • Summative evaluation assesses the merit of completed program.

November 2003

■


1211

Table 2. Core skills of an educational software development team Project Manager • Defines and completes the project in a timely manner and within budget • Communicates project requirements to the entire team • Often serves as the QA (quality assurance) tester Instructional Designer • Understands usability issues, target audience, interface protocols, educational psychology • Conducts the content analysis and determines instructional strategies • Designs smooth program navigation • Works with programmer on program and database requirements • Works with multimedia artist on interface design • Writes video/audio scripts Faculty Member (Content Expert) • Provides dental subject matter including patient visuals, audio, and video • Writes the dental subject matter for the program Multimedia Artist • Creates the overall “look and feel” • Creates visual and graphical designs • Creates and ensures consistent use of graphical conventions • Works with the programmer to prepare graphic files, video, animations, 3D animations, etc. • Selects the proper encoding for optimal audio/video playback Programmer • Designs the database model and program structure • Organizes the information, data, and files • Writes HTML, JavaScript, C++, etc. code to correctly display pages and incorporate all program elements • Documents the integration and function of the program with other programs • Selects the hardware and software configuration Evaluation Specialist • Designs and conducts beta testing (formative) evaluation • Designs and conducts summative evaluation

Table 3. Overview of educational software development phases and deliverables Analysis • Technical and content specifications; budget; schedule; list of content resources • Estimate the development time and multiply that estimate by three. Adjust cost estimates accordingly. Design • Project specifications including flowcharts, storyboards, concept maps, etc. Possibly include a prototype. • Detailed content; lists of required media; program and possible database design • As soon as possible, the dental expert should review a content outline, three potential screen designs, and a flow chart. Not only is this tangible evidence of work accomplished, but it helps to visualize the final product in order to provide valuable feedback. Development • Completed program; testing protocol; documentation • Outsource specialized tasks such as video productions and 3-D animations. • Plan for extensive alpha and beta testing. Implementation • Hardware and software in place; successful product use • Take environmental operating characteristics into account (such as faculty/student computer literacy and access to computing resources). • Start planning during analysis. Maintenance • Maintenance plan • Provide adequate resources for maintenance (such support personnel and hardware/software). • Start planning during analysis. Summative Evaluation • Summative evaluation protocol and report • Involve evaluation specialist during analysis so that baseline information can be gathered if necessary and valid study design can be developed.

1212


as a videotape, a book, or a live patient. Adapting instruction to the learner, the ability to repeat instruction numerous times, self-assessment with immediate and informative feedback, multimedia examples of authentic patients, and convenient access to instruction are a few reasons why the computer might be the appropriate delivery medium for instruction. Determine availability of content resources. Determine the availability of textbooks, other educational software, slides, patient records, videotapes, actual equipment, and other faculty who can serve as consultants and reviewers. Discover what existing content can be used and what content needs to be located or created. A common practice in dental educational software is to use still images of patients to illustrate a disease or treatment. Prior to using visual information of a patient, the project manager should determine if permission has been obtained from the patient and the clinician who recorded it. Lacking the proper consent or the “right” image, a skilled digital medical illustrator, using Photoshop or another imaging editing tool, can frequently create the tissue or treatment option you require.10 If clinical images are created, a disclaimer stating that some images are artist’s renditions of actual patients should be included. Develop management tools. To complete the project on time and within budget and to ensure it meets the project’s goals, the project manager should develop a budget and a specifications document. Depending upon the project, specialized documents such as a timeline or database of content resources may be required. A software tool that helps organize the educational software development process is Designer’s Edge.11 It guides the project manager through the entire development process. It also includes a number of forms to help document and manage the project. A software tool used by many project managers is Microsoft Project.12 This software tool helps create detailed timelines such as Gantt charts and track resource usage, among numerous other management tasks.

Design The design phase results in a collection of documents known as the design specifications that describe on paper all of the program elements, instructional content, and interactivity. Specifications frequently include a working prototype of the program to help enhance communication. Design guides, such as the Guidelines for the Design of Educational

November 2003

■


Software and Web Style Guide, can help ensure that all relevant design issues have been addressed.5,6,13 Use brainstorming techniques. In order to develop an innovative and creative design, brainstorming techniques are essential. A brainstorming session usually involves from three to five team members. During brainstorming, all ideas are welcome, and none are judged or eliminated. The emphasis is on the quantity of ideas. Afterwards, all ideas are listed in no particular order and with no indication of ownership. Then each one is discussed and either eliminated or kept. The revised list is examined again, except that the remaining ideas are listed in order of priority based on how well they meet the goals of the program. Some ideas will be discarded, based on the characteristics of the audience, relevance to the goals, restriction of the delivery system, ability of the production team, or expense. During the process of review, the faculty member and instructional designer will collaborate to determine the best instructional strategy (tutorial, drill, simulation, hypertext/ hypermedia [a format in which textual, image, or other media information related to a topic can be directly accessed from that topic], etc.). Determine the look and feel. It is not unusual for a faculty member’s ideas about the “look and feel” of a project to differ from the instructional designer’s. To prevent misunderstandings, the look and feel should be developed as early as possible and as a product of team collaboration. Sketches can illustrate the designer’s ideas, or a rapid prototype can be created, using an authoring package such as Authorware.14 Navigation samples should be programmed to the level that the faculty member understands how the program “works.” Graphical elements should match the delivery medium. CD-ROMs have few restrictions on the complexity of graphics, but the web needs to account for the user’s connection speed to prevent lengthy download times.15 Thus, complex and detailed (rich) graphics may need to be traded for fast download times. Media (animations, video, and audio) should be used sparingly and only to reinforce the instruction. Animations are effective for attracting attention. Video is very powerful for tasks like demonstrating the effects of patient interactions and dental procedures. But a “talking head” video is rarely essential to the instruction. Instead, you should consider replacing it with a still photograph accompanied by audio and/or text. When video is used, try to keep the segments short (usually twenty seconds or less) to maintain student interest.

1213

Develop a project style manual. The instructional designer develops a document that contains the presentation specifications for the project. The manual may include the use of a specific logo, font, color scheme, etc. It could also include expectations such as scrolling conventions, writing style, image size, use of frames, video formats, and screen size. Sequence and detailed design. Together the faculty member (content expert) and the instructional designer conduct a highly detailed analysis of the content. Usually, either a task analysis for procedural skills or a concept analysis for principles and rules is completed.16,17 A task analysis takes a complex skill and breaks it into component skills. The component skills are individually taught and then combined. This is frequently how the hand skills of dentistry are taught. A concept analysis first identifies the relevant and incidental characteristics of a concept followed by a number of examples and nonexamples. For example, a concept such as “diagnosis of sore mouth” can be taught using three characteristics (onset, healing time, and location) to distinguish between different diseases. (See Table 4.) Flowcharts and storyboards. The instructional designer and programmer use flowcharts to communicate the major elements of the program sequence and interactions. The instructional designer creates storyboards to visualize the details. Storyboards (see Figure 1 for a sample) contain the instructional messages the students will see, including information presentations, questions, feedback, directions, graphics, and prompts. Flowcharts and storyboards are developed simultaneously because a change to one requires modification to the other. This technique works well for most educational software program except for simulations, games, and hypermedia programs, whose navigations are not linear and therefore cannot be easily planned using flowcharts and storyboards. In these cases, a concept-mapping tool

such as Inspiration can be used to demonstrate the program structure.18 Another option is to use a database to create a template with the dynamic information represented in a database field. Expert review. Content experts other than those working on the project team should review the flowcharts, storyboards, concept maps, databases, etc. for pedagogical quality, content accuracy and sequence, and suitability for the designated hardware and software. Reviewers should feel free to ask questions and record comments. Revisions are made based on this information.19

Development Development is the process of producing, refining, and validating the program. The key steps are described below. Write audio/video scripts. The storyboards or database templates for many educational programs include video or audio scripts with spoken text and stage directions. However, script writing is so different from most other software development tasks that it is a good idea to hire a professional scriptwriter to make sure the script accomplishes the program goals and results in a professional product. Create graphics and animations. All graphics should have parallel levels of richness, character, and theme. Three types of computer-based graphics tools are available depending upon the task to be completed. These are summarized in Table 5. There are also a number of special purpose programs such as Strata 3D for creating three-dimensional images. Many of these packages require special expertise that usually only trained digital artists possess.20 Media creation. Animations, video, and audio should be able to pause, continue, repeat, and skip. Animations for CD-ROM are usually created using Director, and a program named Flash is frequently

Table 4. Concept analysis of “diagnosis of sore mouth” 1. Distinguish between the onset characteristics:

Is the onset slow (e.g., erosive lichen planus, pemphigus) or sudden and recurring (e.g., aphthous ulcers, recurrent herpes, and erythema multiforme)?

2. Distinguish between the healing characteristics: Do the sores heal in approximately the same amount of time for an individual patient (e.g., aphthous ulcers and recurrent herpes) or does the healing time vary (e.g., erythema multiforme)? 3. Distinguish between the location characteristics: Are the sores found on bound-down or keratinized mucosa (the gingiva, hard palate, and dorsum of the tongue) as in recurrent herpes, or are they found on movable nonkeratinized mucosa (everywhere except the gingiva, hard palate, and dorsum of the tongue), as in aphthous ulcers?

1214


Table 5. Summary of digital graphics tools Illustrator & Freehand21,22

Object-oriented or drawing software allows you to construct images with shapes, text, shading, and colors.

Painter23

Bit-mapped or paint software gives you control over every pixel on the computer screen, but requires artistic abilities to be used well.

PhotoShop10

Photo editing software has advanced capabilities for modifying scanned images or pictures taken with a digital camera.

used for web animations.24,25 Even more than graphics, digital video requires special expertise, hardware, and software. Decisions must be made about the video capture technique, compression method, video window size, frame rate, audio compression, and the final movie format. All attempts should be made to keep the video segments short (twenty seconds or less) to maintain student attention.26 When creating

media for the web, the team will ensure that it plays at an adequate speed by testing it on various computers and browsers with differing Internet connection speeds. Code the program. The specifications are translated into a language the computer can understand and that can be displayed as the instructional lesson. Approaches include using an authoring system, such as Authorware, a programming language such as HTML or Java, or an application that serves as an interface between the author and the computer code such as Dreamweaver and Front Page.14,27-29 Currently, interactive programming is more difficult on the web than on CD-ROM, but new software tools are changing that. As individual pieces of the program such as graphics and media are completed, they are added to the program. When all of the pieces are assembled and all of the interactivity has been included, you have the first version of the program. Throughout the development process, there will be numerous revisions and program improvements. The project manager and programmer will keep track of the large number of changes through version control or

Figure 1. Sample storyboard and final screen for Treatment Planning in Dentistry

November 2003

■


1215

versioning. This is a very difficult process especially when more than one person is making changes. For large programs, software dedicated to versioning, such as Visual SourceSafe, is essential.30 For smaller projects the project team should have an agreed-upon protocol for renaming and dating files in each version of the program. A single document should log all assets and files, their contents, and their changes. Programmers should insert comments explaining their code throughout the program. These comments, also known as documentation, should provide sufficient information so that another programmer does not have to spend a lot of time figuring out what you have done. All developers should anticipate that someone else will have to make changes to the program. This documentation will save time and expense for future programmers. Remember: the documentation is more important than the code. Prepare support materials. The specifications will state what documentation will accompany the final project. Larger programs will have documentation for the learner, the instructor, and/or the technical staff.31 Denton and Kelly32 have written a seminal book on good computer documentation, and Alessi and Trollip include checklists for educational software manuals.33 Alpha and beta testing. There are two levels to software testing. The project development team does alpha testing, and the evaluation expert working with students does beta testing. Both are essential steps that should be conducted with great planning and attention to detail. During alpha testing the development team carefully checks the program against the specifications including look and feel, style conventions, and functionality. Emphasis should be placed on trying the unexpected, not just what is in the specifications. The team should deliberately try to “break” the program. Based on the results, revisions are made and these revisions are tested. This process continues until errors are no longer found. Beta testing is often called formative evaluation. This is when the program is used by a student, so that design or programming flaws can be recognized and resolved. While beta testing is described here as a separate process, it is not unusual for it to be intertwined with the later stages of alpha testing. Carefully select beta testers so that they represent the typical user, the advanced user, and the novice user. Five to six students are usually sufficient. Ask each student to complete the program as if they were taking it for credit and to speak their thoughts aloud

1216

as they navigate through the program. These thoughts should be recorded by an observer (such as the project manager, instructional designer, or faculty) or on videotape. Also, if the student does anything unexpected, note it and ask about the action afterwards. After the student completes the program, the observer should discuss the user’s comments and the notes they took. For example: OBSERVER: “I noticed that you seemed puzzled by the list of menu options after you completed your review of the radiographs.” STUDENT: “I wasn’t sure if I was supposed to check each of those options in sequence or if I could select any option I wanted.” At all times the student’s perspective should be obtained. An important outcome of the session is that you obtain the student’s opinion about the structure and flow of the program, the clarity of the directions, the level of student control, and other operational functions. Once again revisions are made. If major revisions are made, another round of beta testing is conducted; stop beta testing when only minor refinements are made.

Implementation and Maintenance Implementation means ensuring flawless, convenient, and secure access to the educational program by the student and instructor. Planning for implementation should occur early during the analysis phase. It is not unusual for a project, especially innovative ones, to use software and/or hardware that may not currently be in place within an institution. This may give rise to cries of “Think of the new support headaches this will cause!” from information technology support personnel. However, this should not dissuade you from undertaking an educational technology project. Instead start working with your technology support staff early on. Good communication can help smooth the implementation hurdles you will eventually encounter such as student access to new hardware. Maintenance is the ongoing process of making sure the content and software are up to date. After implementation, the instructional developer or programmer usually performs the maintenance, monitoring, administration, and updating. They will require the solid documentation that you have written to complete these tasks. Lastly, there is always the flurry of activity at the finish of a project during which routine tasks may be left undone. Once the Journal of Dental Education ■ Volume 67, Number 11

project is implemented, the project manager should finish any cleanup documentation, archive assets (images, video, etc.), and file for copyright.

Summative Evaluation Summative evaluation determines the extent to which the goals of the program are met.34 In education, those goals might include improvements in educational outcomes, such as increased long-term retention; economics, such as decreased cost; or efficiency, such as faster acquisition of content. The first step in conducting a summative evaluation occurs during the analysis phase when the project goals are defined. In conjunction with the evaluation specialist, the faculty member determines how to measure the success of meeting these goals.16 A few methods available for this purpose include standardized tests, oral exams, assignments, direct observation, simulations, and competency exams on patients. Observational or exploratory studies may also be appropriate. Care needs to be taken to separate the effects of the delivery medium, in this case the instructional technology, from the educational method(s).* For instance, comparing a new educational software program to a traditional lecture course entails not only a change in the delivery medium, but also in the educational method. Many decades of educational research have yet to produce significant evidence that the medium alone increases learning.36,37 However, the delivery medium can affect the economics, logistics, and cognitive efficiency of learning. Summative evaluation studies quantifying these outcomes provide valuable support for educational software.

Case Study: Treatment Planning in Dentistry Patient Simulations We have described an approach to educational software development, but in reality no single project is exactly like another project. Although it is important to have a process, there is no substitute for experience to help manage those differences. The following case study takes the described development process and overlays it with a healthy dose of reality.

A CD-ROM containing five interactive patient simulations was developed by the University of Iowa College of Dentistry to accompany the textbook Treatment Planning in Dentistry.38 Key highlights from the development of that project are described to illustrate the process of developing educational software. Summative evaluation of this project has not been implemented because it will take several years to completely put into place these simulations in the curriculum. Thus, this case study will address the following components of development: assembling the project team, analysis, design, and development with closing comments concerning implementation and maintenance.

Project Team, Analysis, and Design The core team for Treatment Planning in Dentistry consisted of two subject matter experts (for this project they are also the book authors and so will be referred to here as the authors), the project manager, an instructional developer, a multimedia artist, an editor (who also assisted with the book), and a narrator. Specialized expertise was called upon from a video production studio for capturing the video and audio narration. The project was completed in approximately fifteen months although, due to other time demands, no single person worked on it fulltime. The book and the patient simulations had the same audience—dental students and practitioners— but different goals. The goal of the book was to teach treatment planning; the goal of the patient simulations was to provide students and practitioners with opportunities to practice applying the treatment planning process described in the book to patients. Many dental patient simulations focus on gathering patient information and giving a diagnosis.39,40 Very few dental patient simulations contain a treatment planning module.41 These simulations present all of the required patient information, thereby allowing a focus on treatment planning. To provide a high level of interactivity that would not be constrained by modem speed and to provide the broadest possible distribution, it was decided to develop a CD-ROM that would operate on both the Macintosh and Windows operating systems. The content resources for the patient simulations were the book and the authors’ patients who met specified criteria. Over the course of the project, four management tools were used: a

*Salomon, in Interaction of Media, Cognition, and Learning (1994), defines educational methods as “any way to shape information that activates, supplants, or compensates for the cognitive process necessary for achievement or motivation.”

November 2003

■


1217

timeline (developed using Microsoft Project©), a specifications document (developed using a word processor), a flowchart (developed using Inspiration©), and a Patient Simulation Definition Worksheet (developed using a spreadsheet) that contained textual information about the patients. The authors, the project manager, and the instructional designer used brainstorming techniques to complete the design of the patient simulations. The authors then selected a typical patient case that would become the first simulation to “test” and revise the design. The final specifications included the simulation elements (Patient Information, Diagnosis, Treatment, Phase, and Sequence), the functionality of each element, the flow of the simulations, and the CDROM components (Patient Simulations, Introduction, Instructions, Glossary, and Help). During the design phase, decisions were made that impacted the entire program. For example, at the end of each program element the user’s decisions are compared with the authors’, feedback is given, and the program continues using the authors’ decisions. Thus, if the user selects a treatment for a diagnosis that does not match the authors’, they receive the authors’ answer with an explanation. The user then proceeds to the next step using the authors’ treatment decisions. This prevents a user from making an incorrect initial decision and then following an incorrect pathway. During the design, the type of patients to be represented as simulations was finalized, and sample screen designs were constructed, reviewed by the authors, and revised. Additionally, the authors sought input from their colleagues regarding the program’s design.

Development The authors continued to gather patient information (textual, intraoral photos, charts, radiographs) during the design and development phases. Textual patient information was organized in a spreadsheet. The multimedia artist finalized the look and feel, completed the charts, and refined the intraoral photographs and radiographs. To prevent identification of the patients, separate facial photos were taken with special releases. The script for the introduction video and instructions audio was recorded by external professionals and digitally edited by the multimedia artist and instructional developer. Prior to production, the editor reviewed the scripts and the program text for consistency with the book.

1218

The instructional developer wrote the program code, placed the media assets (video, audio, graphics, photos, and charts) into the code, and created and revised the text files for each simulation. The beta testing overlapped the alpha testing. During the alpha testing, the project manager and authors verified that each simulation worked properly. As significant portions of the initial simulation were completed, additional dental experts and dental students completed it and suggested revisions. Once the first simulation satisfied the project team, the remaining simulations were programmed. The beta testing of each simulation was completed by a total of four to six dental students at two dental schools. Based on their feedback, refinements were made. Finally, the software was tested on a number of workstations, each with various hardware configurations and operating systems, to ensure that it would work as specified. Once again, revisions were made. The instructions section of the program was written, programmed, and tested, and the printed instructions for the book written. Finally, a CD-ROM containing the entire program was delivered to the book publisher for replication. Throughout the development process, tradeoffs were made that affected cost and program quality. The strategy used to code this program was based on preserving quality at the lowest cost. Because all five simulations would have the same interface, three programming strategies were considered: 1) write a separate program for each simulation, 2) write one program that would read patient specific files for each simulation, or 3) write one program that would call upon a database containing the patient information. The first option was eliminated immediately because when a revision was made to one simulation, that same revision would have to be made to all other simulations, meaning it would be very difficult for the program manager to verify the revisions. Due to the small number of patient simulations, it was decided to program a single program that would read the files for each simulation (option #2). While developing a database of patient information was a task the project team could accomplish, it was decided that ensuring that it operated smoothly on several types of computers and operating systems would take too long and would increase costs.

Implementation and Maintenance This program is designed for students to independently work through the case prior to a large group


class or seminar where the case is to be discussed. The visual information (clinical images, radiographs, and charts) from the case can be displayed during the discussion. Because this project was to be included with a textbook, implementation issues were addressed up front in the analysis phase. It was determined early on that the program should work on the most common computers found in dental schools and in practices. This eliminated any network or modem issues. The code was documented as the program was developed and is archived on CD-ROMs with images, graphics, etc. The simulations will be used in treatment planning curriculums over the next few years. At that time it is anticipated that an evaluation of the product will be conducted.

Comments and Lessons Learned

velopment process and not wait until completion. Combining all of these phases will increase the likelihood of a project being of high quality. Many faculty members are able to learn enough about software development to create simple educational programs. However, the broad range of expertise required to complete the development tasks described in this article makes it difficult for a single person to create effective educational software that takes full advantage of today’s technology. Again we emphasize instructional quality. A small team is most likely to have expertise in a few areas, while a larger team, consisting of faculty, instructional designers, multimedia artists, programmers, and evaluation specialists, will naturally have broader expertise. All educational software development teams, no matter what their size or skill set, should recognize their shortcomings and, in order to develop a better product, consider outsourcing the work that they are least skilled at. The bottom line should always be the same: do what it takes to create the highest-quality learning experience for students.

If the project team had been able to concentrate on the CD-ROM, the product could have been finished comfortably in less than one year. In reality, the authors’ time constraints meant the book and its accompanying CD-ROM were produced over the course of three years. This lengthy time frame resulted in personnel turnover. It also meant that multiple people fulfilled one role. For example there was more than one instructional developer. This turnover and role duplication reinforces the need for good project documentation. Good documentation decreases the amount of time required for a new person to understand the work that had been completed and shortens the learning curve, thereby reducing costs.

The contributions of the Treatment Planning in Dentistry development team are acknowledged including Stephen Stefanac, Samuel Nesbitt, Danny Novo, Phil Bailey, David Rubright, Nellie Kremenak, and George Stratton. The authors also wish to acknowledge our colleagues with whom we have worked to develop educational software that meets the teaching and learning needs of our colleagues and students.

Conclusions

REFERENCES

The development of high-quality educational software is a multiphased process that requires a broad array of skills. This article attempts to succinctly describe the six phases of educational software development and the personnel required. Careful analysis and planning are important for the success of a project. Time should be invested early on in a complete and thorough project design. Too much haste will inevitably result in a less than effective product or in time wasted during development. Development comprises a number of activities that all must be executed in a coordinated fashion. Planning for implementation, maintenance, and summative evaluation should occur early in the de-

November 2003

■


Acknowledgments

1. 2. 3. 4. 5.

6.

7.

8.

Blackboard. 6th ed. Blackboard, Inc., 2003. WebCT. Vista ed. WebCT, Inc., 2003. TopClass. WBT Systems, 2003. Alessi SM, Trollip R. Multimedia for learning: methods and development. 3rd ed. Boston: Allyn and Bacon, 2001. Schleyer T, Johnson L. Guidelines for the design of educational software. Temple University and the University of Iowa, 2003. Johnson L, Schleyer T. Developing standards for the design of educational software. Quintessence Int 1999;30:16. Schleyer T, Johnson L. Developing a protocol for an educational software competition. Proceedings of the 2001 AMIA Fall Symposium, Washington, D.C. J Am Med Informatics Assoc 2001:603-6. Friedlin A. Web project management. San Diego: Academic Press, 2001.

1219

9. McCarthy J. Dynamics of software development. Redmond, WA: Microsoft Press, 1995. 10. Photoshop 6. 7.0 ed. Adobe Systems Inc., 2003. 11. Designer’s Edge. 3.5 ed. Allen Communication Learning Services, 2003. 12. Microsoft Project 2002. Microsoft Corp., 2003. 13. Lynch P, Horton S. Web style guide: basic design principles for creating web sites. New Haven: Yale University Press, 1999. 14. Authorware. 6.5 ed. Macromedia Inc., 2003. 15. Weiman L. Designing web graphics. Indianapolis, IN: New Riders Publishing, 1999. 16. Smith PL, Ragan TJ. Instructional design. 2nd ed. Upper Saddle River, NJ: Prentice-Hall, 1999. 17. Engelmann S. Direct instruction. In: Dilles CR, Romiszowski AJ, eds. Instructional development paradigms. Englewood Cliffs, NJ: Educational Technology, 1997:371-89. 18. Inspiration. 7th ed. Inspiration Software, Inc., 2003. 19. Flagg BN. Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum Associates, 1990. 20. Strata 3Dpro. 3.8 ed. Strata Corporation, 2003. 21. Illustrator. 10th ed. Adobe Systems, Inc., 2003. 22. FreeHand. MX ed. Macromedia, Inc., 2003. 23. Painter. 7th ed. Corel Corporation, 2003. 24. Director. MX ed. Macromedia, Inc., 2003. 25. Flash 5. 6th ed. Macromedia, Inc., 2003. 26. Swan K. Exploring the role of video in enhancing learning from hypermedia. J Educational Technology Systems 1996;25:179-88. 27. Hall M, Brown L. Core web programming. 2nd ed. Palo Alto, CA: Prentice Hall PTR/Sun Microsystems Press, 2001.

1220

28. Macromedia Dreamweaver. MX ed. Macromedia, Inc., 2003. 29. FrontPage. Microsoft, Inc., 2003. 30. Visual SourceSafe. 6.0 ed. Microsoft, Inc., 2003. 31. Johnson LA, Cunningham M, Finkelstein MW, Hand J. Geriatric patient simulations for dental hygiene. J Dent Educ 1997;61:667-77. 32. Denton L, Kelly J. Designing, writing, and producing computer documentation. New York: McGraw-Hill Companies, 1993. 33. Alessi SM, Trollip SR. Computer-based instruction: methods and development. 2nd ed. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1991. 34. Schleyer T, Johnson L. Evaluation of educational software. J Dent Educ 2003;67(11):1221-8. 35. Friedman C, Wyatt J. Evaluation methods in medical informatics. New York: Springer, 1996. 36. Clark RE. Media will never influence learning. Educational Tech Res Dev 1994;42:21-9. 37. Kozma RB. Will media influence learning? reframing the debate. Educational Tech Res Dev 1994;42:7-19. 38. Stefanac S, Nesbit S. Treatment planning in dentistry. St. Louis, MO: Mosby, Inc., 2001. 39. Finkelstein MW, Johnson LA, Lilly GE. Interactive videodisc patient simulations of oral diseases. J Dent Educ 1988;52:217-20. 40. Cunningham M, Rubright D. Assessment of geriatric patients. Iowa City: University of Iowa, 2001. 41. Johnson LA, Wohlgemuth B, Cameron CC, et al. Dental interactive simulations corporation (DISC): simulations for education, continuing education, and assessment. J Dent Educ 1998;62:937-46.
