Development and VaWation of an Evaluation Tool for ..... Development of the tool . ..... today's desktop personal cornputen can perform 16 - 66 million operations ...
THE UNIVERSITY OF CALGARY
Development and VaWation of an Evaluation Tool
for Multimedia Resoarces in Health Education
by
Kalyani Premkumar
A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES
m PARTIAL FULFILMENT OF THE REQ~REMENTSFOR THE DEGREE OF MASTER OF SClENCE
DEPARTMENT OF MEDICAL S C I E N C E
CALGARY, ALBERTA APRIL, 1997
Q Kdyani Premkumar 1997
National Library
BibIioth&que nationale du Canada
Acquisitions and BibliographicServices
Acquisitions et s e ~ k e bibliographiques s
The author has granted a nonexclusive licence allowing the National Libmy of Caoada tcs reproduce, loan, distniute or sell copies ofhidher thesis by any means and in any fonn or format, making this thesis available to interested persons.
The author retains ownership of the copyright in M er thesis. Neither the thesis nor substantial extmcts &om it may be printed or otherwise reproduced with the author's permission.
L'auteur a accord6 m e licence non exclusive pennettant S la Biblioth&penationale du Canada de reproduire, Hter,distniuer ou vendre des copies de sa Wse de quelqye maai&e et sous quelqye fome que ce soit pour mettre des exemplaires de cette these a la disposition des personnes interessees.
L'auteur cooserve la propriM du droit d'auteur qui proege sa these. Ni la t h h ni des eXfiZU*tssubstantiek de celle-cine doivent &re imprim& ou autrement reproduits sans son
ABSTRACT The last decade has seen an increase in the use of multimedia in health education, Since one of the roles of educators is to assess and select le-ng
resources, development of standatdised
methods for multimedia evaluation becomes vital. To the learner, it is important for reviews of the quality of the resource to be readily available. Recognising the need, an evaluation tool was
developed Validity of the tool was tested using experts in technology and education. Reliability was determined using faculty and students who reviewed the same software, using the tool. In
addition, graduate students reviewed two versions of a nursing program, of varying quality The learning outcome of students using each of the versions was also assessed The results indicate that the tool is reliable and valid It is envisaged that this tool can be utilized by health educators for evaluating muitimedia resources and setting up a much needed clearing
house.
ACKNOWLEDGEMENTS The enthusiastic participation of the Faculty, graduate nurses, undergraduate nursing and
medical student volunteers are gratefully acknowledged I wish to aclrnowledge the support, encouragement, mentoring and guidance provided by Drs Joseph Davison, P e ~ Jye ~ e tand t William Hunter. I wish to thank Bilan Bodhaa and Darcy Norman of the Faculty of Nursing for permission to use, and for modifyiag the versions of the Peritoneal Dialysis CDROM developed by them.Finally, the assistance of Gisela Engels for
analysis and my
husband James Premkumar for editorial assistance is also acknowledged+ This study was partly fimded by the Thesis Research Grant, University of Calgary.
TABLE OF CONTENTS Approval Page...........................................................................................................
Abstract ................................................................................................................. Acknowledgements ................................................................................................. Table of Contents ....................................................................................................
v
List of Tables ..........................................................................................................
viii
List of Figures .........................................................................................................
ix
CHAPTER ONE .....................................................................................................
I N T R O D U ~..O .....N ............................................................................................... What is mdtirnedia? ...................................................................................... Increase in the use multimedia ...................................................................... Unique qualities of multimedia .................................................................. Integtution ........................................................................................... Multi-ievei approach ............................................................................ Branching capabiiitiesurui individmfitation ....................................... Sk&ipCenearner c o d .....................~.~~.~.~..~~~~..~~~.................~~....~....... Avaihbifitymd accessibifity ................................................................
.......................................................................... Conrpuatio~i tool ............................................................................. h e of tecwd-keeping ........................................................................ k s o n infegrity ..................................................................................... Avaihbiiity of teachers ....................................................................
Simulation wpbiiities
Common limitations of multimedia ..............................................................
-
Teacher and Multimedia a powerful instructional system ............................. Importance of evaluating multimedia resources ............................................. CHAPTER TWO .........................., . .................................................................... L~TERATUREREVIEW ..............................................................................................
Role. Characterisitcs and Design of Multimedia .............................................
9
Rare of ntultirnedia in imtruction and learning.......................................
9
Characterisitcsof eflective muItimedirr resources ..................................
II
Design and [earning theory.....................................................................
12
Modes and designs of muftimedia........................................................... . . .. Measurement pnnc~plesr e v w t d....................................................................
18
Learning outcome using multimedia................................................................
20
16
-
Evaluation of multimedia current resources avdable....................................
21
CHAPTER THREE ..................................................................................................
23
PROBLEMSTATEMENT...................................................... . . .
23
.......
Purpose.............................................................................................................. 23 CHAPTER FOUR .....................................................................................................
25
................................................................................................................. M~THOD.-, Development of the tool..................................................................................
..
Validation ofthe tool .......................................................................................
. .
Description of the tool.......................................................................................
...
-
Rehabthty ofthe tool i....................................................................................
- ..
Reliability ofthe tool u...............L.................................................................. Learning Outcome............................................................................................. Post-test Validation........................................................................................
Ethical Approval............................................................................................... Statistical Analysis............................................................................................
CHAPTER FIVE...................................... . . . . ....... 31
RESULTS ................................................................................................................... . Face and Content Val~dldty............................................................................. *
Reliability of the tool - i .................................................................................
31
31 31
...
Reliablhty of the tool .11...................................................................................
34
Learning Outcome.............................................................................................
35
.........................................................................................................
41
Drscussro~..............................................................................................................
41
Face and Content Validity ................................................................................
42
Reliability of the tooi - i ...................................................................................
44
Reliability of the tool - ii ..................................................................................
45
Learning Outcome ............................................................................................
46
C
R S
**
CHAPTER SEVEN ...................................................................................................
48
............................................................................................................ CONCLUSION
48
GLOSSARY ...............................................................................................................
50
REFERENCES ...........................................................................................................
53
APPENDICES ................... ..,,,
.............................................................................
-
Appendix 1 Evaluation Tool original ............................................................ Appendix 2 Assessment form for Experts ........................ . ...........................
-
Appendix 3 Evaluation Tool modified ........................................................... Appendix 4 Post-test ......................................................................................... Appendix 5 Optical Score sheet .......................................................................
Appendix 6 Consent form for Student Volunteers ........................................... Appendix 7 Consent form for Evaluators........................................................
LIST OF TABLES Table 1: Face and Content Validity :The d e m k background ofthe eight experts. Table 2: Reliability A cornperison of the aggregate sams (mean scores of four
examinations taken by nursing students during the first year of nursing) of the three
-
groups Control, Study Group A and Study Group B.
Table 3: Reliability: A comparison of the pst-test scores of the Control and
combined (A & B) Study Groups. Table 4: Results of the correlation between the aggregate and post-test scores of all participants, Control Group, Study Group 4 Study ~ r o u pB and combined (A & B)
Study Groups.
LIST OF FIGURES Figure I: The mean scores of various subsectiom of the evaluation tool, as rated by the eight experts.
Figure 2: Comparison of the overall scores given by the nursing students (n=4), medical students (n=4) and faulty (n=4) for the CardiovascularPhysiology CD-ROM Figure 3: A comparison ofthe aggregate scores of the Control a d combined (A & B)
Study Groups. Figure 4: A comparison of the post-test scores of the Control and combined (A & B) Study Groups.
CHAPTER
ONE
" EDUCATORS MUST TAKE THE ELECTRONIC REVOLUTION SERTOUSLY is
the title of a recent special article in Academic ~edicine'. The primary reason given for this warning is the fat that computing power is great enough to permit creation of multimedia packages in health education that challenge the foundations of our educational practices and institutions! Truly, with the advent of the world wide web' and
online learning3, complex modelling and simulation4, and other rapid advances in technology this is a revolutionand educators need to take it seriously! What is mu1timedia? The word medium; media is derived from the Latin medim, %%ween9'. The term
refers to anything that carries informationbetween a source and a receives. This could be film, television, radio, audio recordings, photographs, pmjected visual, printed materials and the me6.When two or more of these media are combined they are termed
multimedia In the generic sense, multimedia refers to such resources that integrate video, audio and graphical material with text and number operations such as CDROMs, interactive videodiscs and learning packagess. With the field evolving so rapidly, it also implies that the application may go beyond a sin& computer to include natiod and intematioad networks- Agreemeat bas to be reached on proper definitions
for terms used in the field, in order to avoid misuse (a gl-
has been added to clarify
meanings of tern used in this thesis).
Increase in the use of multimedia
Undoubtedly, the use of multimedia in health education has increased. Many reasons can be cited for this. The extent to which knowledge must be assimilated by those
involved in health care has increased markedly and an efficient way of accessing
infomation bas become vital7- The improvement in technology and the evolution of
CD-ROM,with their capacity to store about 600,000 pages in one disc, CD - I (Compact Disc Interactive), D M @igital Video Interactive), WORM (Write Once Read Many), rewritable compact discs, video discs and optkal storage systems, have paved the path for doing so'.
Over the last decade, the cost of computers has decreased tremendously. Computers
are also easily available and accessible to evev student, A recent swey done at the University of Calgary, s h o d that 90.A of medical shdents possessed computers aud used them for accessing coursewareg.The size of the computers has decreased without the loss of power'0. The p o w of prcsent
day computers is astounding Most of today's desktop personal cornputen can perform 16 66 million operations per second, and some more than LOO million! The most sophisticated of supercomputers can
-
perform nearly 32 billion calculations per secondLL.Such computers attain these speeds
through the use of advanced engineering techniques. For example, the circuitry is supercooled so that electrons can move at the speed of light! The portability of the equipment has eaabled the user to access resources anywhere and at any time. The progress made in technology has been aptIy desaibed by Michael Blumenthal: "If automobiles had pursued the same technologicalpath that computers have, Americans would now own halfpound cars, costing less than $4.50, and getting 1.5 million miles to the
The availability of numerous authoring systems such as HyperCard, Linkway, Tool book etc. has encouraged the development of muitimedia packages by individuals for classroom use, and for sharing across itrstitutions. W
e a few years ago, considerable
training was required for using these programs, today, the skills can be mastered by educators in a matter of hours.
The introduction of Open Systems which allows for running a particular program on
any brand of computer has also contniuted to the increasing use of
Hypermedia technology, with its capability of introducing powedid new ways for creating and using biomedical information is yet mother reason". With the advent and
affordability of connections to the Internet, resources anywhere in the globe can be accessed! Programs and infommtion can also be shared by Ieamers across the globe
creating victual classrooms with a tremendous soope for promoting continuing educati0d3.
The possibilities of networking computers - Local Area Networks and Wide Area ~etworks'*l4 and therefore the cost reduction and impact on the economy has encow aged recent political support". Governments have begun to set aside h d s to encourage research and use of Computers in ducati ion"* 16. Recognising the unique qualities of multimedia, the P a d on the General Professional Education ofthe Physician of the Association of American MedicalColleges (AAMC) in its ~eport' strongly recommends its use in health education. Unique Qualities of Multimediau,17-21 Integration
In the realm of learning, progress in technology has given educators the potential to
remove various barriers that have existed in traditional teaching methods- By having
-
the ability to combine all fonns of information text, pictures, film,animation, sound,
touch etc, technology allows students to leam and teachers to present informaton in a variety of ways. This caters to the individual learning styles of students.
Mulri-level approach By removing the barriers that force educators using traditional teaching methods, to
move from simple to complex, no matter what level each student is in, technology has
blurred the distinction between elementary and advanced, making it easy for learners to learn as indepth as necessary in any subject.
Branching capabilities and individdisation One of the unique features of multimedia is the capacity to introduce branching at
various points acoordingto student needs and level of learning. Multimedia courseware can be designed in such a way that learners can be tested and routed according to their
understandingofthe material presented Sey- pacdearner controi
Technology also enables learners to move at their own pace. It allows students to review and practice as many times as they want without losing patience. Also, it gives students the authority to determine the order of the units, when to take spot tests, the
number of practice sessions required etc. Avaiiabiliry and accessibility
Multimedia allows for learning to occur at m y time and in any place that an appropriate computer, power source and soAware can be found. Simdution cupabilities
For those situations where practice in real life is not always possible, the simulation capabilities of multimedia can be lltilixdp o w e ~ yIn . high quality courseware, the learners can be given control over the variables so that they can predict, test and refine their understanding. With the introduction of v & d reality, specific skUs can be practised to perfectioa Computatiod tool
Multimedia applications can be used to store databases and to perform complex calculations and in three-dimensional modelling - a boon to statistical and research methodology courses. Euse of record keeping
Computer capabilities allow for an accurate recording of individual and group progress,
enabling teachers to keep easy track of individual students and suggest remedial measures. Lesson integrity
Multimedia prognuns can be designed in such a way that irrelevant idorma-on does not underinhe the learning process.
Accessibility of Tecrchers
By its capacity to access students and teachers virtually, technology bas broken down
the temporal organisation of teaching by altering the reIationship between class time
and out of class time. Technology W c a n also change the relationship between
students and teachers. It enables students to access teachers more easily, changing the role of teachers to facilitators and mentorsCommon Limitatioas of Multimedia
As in any situation, there is controversy over the potentiat of multimedia applications. Atkins and 0'~alloran~ in their paper on Evaluating Multimedia Applications for
Medical Education, have cited some common limitations. For example, in programs such as tutorials, although multimedia bas branching capabilities - capacity to skip information or to backtrack as deemed desirable, the needs of a particular learner cannot be met, as, all routing decisions and f
w k messages are pre-programmed.
Thus, responses of the user, unanticipated by the programmer cannot be intelligently
interpreted. Unlike a human teacher who can gauge the misconceptions,and partially correct knowledge of individual learners, the program cannot tailor itself in such a unique way to each learner. In addition, it limits incidental learning that usually occur when a teacher and student interact directly".
Although it is possible to create applications thet are flexible to individual learning often, designers fall into the trap of creating programs which favour one type of learners.
In those areas which require learning of skills, abstract concepts etc, whm visual demonstfations and simulations play on important role, multimedia though applicable. has its limitations. Often, the simulations cannot reach as close to realityas desired For
example, in simulations, users cannot opt for solutions that have not occurred to the designer- In research applicationss it is difficult to create a scenario that is close to
-
reality with its deadads and ~ulde-sacs~ We are yet to understand the extent to which experiences of multimedia simulation is trsnsferred to real Life situations. As an individual learner moves fiom the stage of novice to expert, the usefihess of
multimedia applications seems to decrease*. This, perhaps could be attributed to our
lack of complete understanding of "expert". S
k experts seem to hction at an
unconscious level unless challenged by a novel problem2s,it is difficult to iden*
the
usefbi.uessof multimedia in this situation,
Another limitation that is rapidly being overcome is the need for complex, expensive
hardware. Also, multimedia relies heady on reading, visual and auditory skills. Unlike traditional instruction in the form of lectures, the faculty require additional develop
ment skills and need to spend a longer time on development Unfortunately, while the technology advances, backers are also progressing in their development of software. in an effort to sabotage other people's computers. The existence of logic bombs, viruses and worms cannot be ignored as a timitation since it can cause the entire computer system to crash!! This is of major concern where networking is employed
-
Teacher and Multimedia a powerful instructionalsystem On cornwag multimedia with the strengths of teachers, several limitations can be
identified. Multimedia while being able to execute a course, cannot plan a course, develop instruction, bond with students, respond to enquires outside what has been
programmed, model behaviom and values, empathise with and motivate students as a teacher can, nor can it adapt to changing demands. Therefore, it is important to realise that while individually, teacherdmuitimedia have Limitations,
a combination of
teacher and multimedia can provide a very powerfid instructional system by compensating for the limitations of each other.
Importance of walluting multimedia resources Given the capabilities of multimedia resources, there is a speedy turnover of new packages such as computer-assisted learning (CAL), computer managed instmc-
-
t i o n s ( 0 , drills, tutorials and simulations in the market, In 1987 a decade ago, it was estimated that there were 8000 to 10,000 educational software titles in the market
and about 200 titles were appearing in the market each year? This would extrapolate to
over 12,000 titles today! A 1996 resource catalogue of software for health sciences
education, lists over 820 programsn- Although the software available in health education is much less, it adds up to a phenomenal number. In the health education field, the availability of more than ow product addressing the same subject, o w product addressing more than one subject, a single product catering to more than one target
population, poses a problem for purchasers. Earlier sweys of software available for schools, indicated that majority of the
packages were of poor q u a l i v applying little of the proven educational and evaluation principles. Although the quality of multimedia has improved tremendously, the issue of
quality still remains. The purchaser is still left with the unenviable task of sifting through many products before being able to identify the one that is superior andlor caters to specific needs. Unlike books, it is often difficult to obtain preview copies of multimedia packages. One
of the reasons being the ease with which the whole package can be illegally copied onto discs. To soive this, many producers require non-refimdable deposits, or require
-
teachers to use preview centres a cumbersome procedure.
The importance of evaluating multimedia resources and setting up a clearing house is reiterated by the issue of cost. The cost of some multimedia packages may be as high as S 1000. Hence, a purchaser needs to cruelidly evaluate what he/she contemplates to buy.
In medicine, as the subject matter overlaps a great deal, it is often difficult to identify the target population. Also,
since the subjects are becoming more and more specialised,
an opinion of a content expert is required Many of the packages require sophisticated
-
and complex capabilities of the computer as they need to incorporate videos, high
resolution pictures, capacity to view three dimensiodly etc. Hence a technical expert is required to judge the capacity of the computer that is going to be used and the
requirements of the p r o m In addition, a specialist in insEnrctiona1 design is needed to evaluate how well the program incorporates educational principles.
Although c l e a ~ houses, g on-line services and review journals exist for elemen-, primary and secondary school^^, in medical education, there is no clearing house set up to objectively evaluate all such resources. This need was recognised by the Steering
Committee of the Association of American Medical colleges". There ate few peer-
reviewed journal^^' that offer reviews of health education resources. Even these are subjective in nature, giving an overall impression. Evaluation checklists that are
available seldom have published analyses of their validity or reliability? Hence there is an urgent need to develop a standardised software evaluation tool for resources in health education. This thesis strives to address these lacunae.
CHAPTER TWO
Role, Characteristics and Design of Multimedia Role of muttimedh in instmdon and f40n~*ng Since the introduction of computers in education, extensive research has been done to
identify the conm'bution of multimedia to the teaching and learning process. But, generalizability of the research finding is diflicult due to the diversity of multimedia and, many of the technologies are relatively new and have not been l l l y analyzed'6. Numerous comparative studies betweea multimedia and other traditional methods have also
been done6, 32-34 but, many of the findings contradict each other. Much of the
research in technology in education has focussed on computer based instruction and interactive video and many reviews of these studies have been report&
32-38.
Yet,there
is evidence to show that multimedia does contribute uniquely to education, Multimedia courseware have a high degree of interest to students3*4* *'
-
"- They can
""
offer a reality of experience more so with the possibilities of virtual reality39* ? They can also provide experiences that are not easily secured by other material4*'7 In addition, they have been shown to supply a concrete basis for conceptual thinking6 contribute to ?he efficiency, depth and variety of learning and provide flexibility
according to student needs" a*41. As such, multimedia can be utilised in a variety of
situations
- as a support for teacher-based instruction,
student drill and practice,
discovery learning, management of instnrction, individualised instruction, special edu-
cation among others6. Multimedia helps incorporate various learning theories in an efficient and effective
manner. Just as a child learns to dsfferentiate by experience and thus develop concepts, generalisations and abstractions, it should be remembered that any learner of any age
has to do the same in a new subject area It is the role ofthe teacher to organise the experiences of learners in a way that helps them change their behaviour in a meanin@
way. Jerome Bnmer, the psychologist proposes that instruction should proceed
-
through direct experience or repcesentations of experience stimulating various senses6-
Being multisemry, multimedia may allow students with varying learning styles to benefit Research in learning styles indicate that some students learn better through specific modalities such as audio, visual or kiae~thetic~. In other words, some
students may benefit most when instruction is delivered through sound and narration while others may Learn better with pictures and texts. Learning style inventories such as the one developed by Renzulli and smitha, are available which help identify prefer-
ences of students to specific activities such as independent study, simulation, drills, lectures, projects, teaching games etc. For a comprehensive individualization program, activities have to be organized to accommodate different learning styles and multimedia allows for that.
Instruction in multi-seawry channels also incorporates Howard Gardner's multiple intelligence theory? Gardwr has established a criteria as to what C O ~ an M ~ S intelligence and has identified seven intelligences (each with its sub-intelligences)
-
linguistic, logical-mathematical, spatial, bodily-kinesthetic, musical, interpersonal and intrapersod His research has shown that most people appear to excel within one or two intelligences. This indicates that students have unique cognitive profiles and when
instruction is delivered in many modes
- integrating the 1 I 1 spectrum of
human
strengths, more opportunities for leaming are provided.
Multimedia not only provides the necessary concrete experience but also helps students integrate prior experiences. Multimedia, allows for individual students to start learning
at a level appropriateto the experientialbackground of the student Thus, it incorporates the principles of situated learning?
The modcl of situated cognition is based on the
notion that knowledge is contextually situated and is hdarnentally influenced by the activity, context and culture in which it is d3: Multimedia progratns have the power
and flexibility to support key components of this mode1 such as coaching, multiple practice, reflection, dlabomtion and provision of a highly realistic environment
Multimedia also allows for active participation in the pocess of gathering information rather than passively receiving it1*-If desigued well it also facilitates immediate
feedback to the student Both active participation and immediate feedback - have been shown by behavioural psychologists such as BF Skinner and others as being essential ingredients that encourage fwther leamingi7.By providing opportunities for learners to
work co-operativeiy, multimedia encourages cooperative learning-Research has shown that group interactions enhance effectiveness and efficiency of learning? Literature
review indicates that use of multimedia in a network can affect learning by providing opportunities to improving communications skills4'.
In terms of economy, somea have found significant economic benefits especially in terms of saved time in fsculty preparation and lecture time.
char acre^ of effitfbe tmu&te&
rewumes
17.19. CIJ1
Simplistically, an effective multimedia package can be said to be one % a t does what it was designed to
The characteristics of an efffetive multimedia can be better
illustrated by considering that of an ineffective one. Factors associated with the latter
include - failure to capitalise on the unique interactive and individualising capabilities of multimedia; usage of weak forms of interaction; heavily text or picture dependent presentations; utilisation of entertaining or attractive material that have no educational objective; lengthy instructions; difficult to execute; highly dependent on accompanying printed material and material that fails to keep the student's attention. In short, an
effective multimedia would take into account the scientific theories behind learning, instructional design, cognitive psychology and proven educatiof principles. The characteristics described by ~ o b ~ l can e r ~be awIied appcopriateIy. Robyler,
classifies the characteristics under three headings - essential, aesthetic and differentid
characteristics, Essential characteristics are critical and include criteria that bas to be used by any educational medium, such as statement ofobjectivesype--requisites, design
of learning activities, tests and presentation, Aesthetic characteristics include spacing, format, usage of coloun and sound Differential characteristics include criteria such as learner control, feedback and response formats, use of graphics and animation among others. In summary, some ofthe characteristics to be looked for in an effective multimedia are: Is it based on instructional objectives? Does it match learner characteristics? Does it
maximise interaction? Does it use individualising capabilities? Does it maintain leamer interest? Does it use reliable and valid questions for evaluation? Does it evaluate the
performance of the leamer properly? Does it provide a variety of feedback? Does it approach the learner positively? If used in a crowded setting does it fit the instructional
environment? Does it incorporate the principles of inshuctional design? However, it
has to be undentoodthat the importance given to the characteristics needs to be varied according to the type of multimedia being evaluated. ~esigrta d ~eorning tllre~ty~'''~~~
Researchers in educational psychology have identified several ways by which the probability of learning can be improved The field of instructional design has applied these findings One o f the theories that has been advanced is the behavioural learning theory or
Skimerian or stimulus-response psychology. This theory states that learning results
when a response is paired with a stimuli.
The key concepts here ace contiguity,
repetition and reinforcement Learning is improved ifthe response immediately follows
the stimulus (contiguity), if there is scope for practice (repetition) and some form of feedback on the appropriateness of the response is present (reinfor~ement)'~? Another theory is cognitive theory which is based on the premise that stimulation affects the learner's sensory receptors and
produces patterns of neural activity- It
expands on how i n f o d o n is gained, represented and traasfonned as knowledge, and
on how it is stored and retrieved (shod and long term memory) as and when required
The key concepts here are that leaming involves the synthesis of prior inforation that must be recalled to active memory (orientation and r e d ) ; Iearning is facilitated by
using existing processes and strategies e-g. recalling the process of how similar objectives were learnt earlier and utilising and improving it (intellectual skills); and
learning is more efficient when the instruction caters to the needs and profiles of individual learners (individualisation)?
Other related principles are that learning is related to the time that the student spends actively engaged in the activity. It is also related to how alert, interested and responsive the learner is throughoutthe activity. Leamhg is also influenced by the motivatioa and
attitude of the shdent If productive attitudes are fostered, learning is enhanced. Basic
assumptions of the humanistic theory are that the learner is the best judge of the developmental and momentary needs and bas the responsibility to choose the methods
and goals of learning. The various leanring theories have several implications for design of multimedia In accordance with the internal processes of laming, the resource should be designed in
-
such a way that it gains the learner's attention such as amactive graphic sequences and intellectual challenges. The learner can be informed of the objectives. This is likely to influence the recall of information, skills and strategies to working memory.
The multimedia should stimufate d
l of prior learning. This can be achieved by
stating prerequisites and testing than if aecessary. In order to direct the learner's perception, key elements caa be highlighted in some way. The multimedia should allow
for adequate practice and shape chired bebaviour by assessing performance, providing
feedback and reinforcement, thus enhancing retention Since the learning process is related to prior experiences, the multimedia should be designed to cater to individual learners. This can be achieved by dowing for individualisationof content, pace, practice and feedback Capitalising on the interactive capabilities of multimedia, learning can be enhanced. Interactioa ensures that messages are received and key elements are focussed upon It also encourages responses to
strengthen cognitive ties, facilitates recall, i&ntifies and comts misconceptions. As well, by holding the interest ofthe leamer, it increases the time spent in learning.
Multimedia can be designed to use feedback in an effective manner based on research on feedback53? It has been shown that negative feodt#lck (feedback provided when the response is incorrect), is more effkctive than positive feedback (feedback provided
when the response is comet). Also, response. The timing off-k
feodbgck
is not aecessuy after every correct
should be varied accordingto the mastery level of the
learner. For example, it can unduly slow the progress of a learner with high mastery level if given too fkquently. On the other hand, wntinuous, immediate f e c k is beneficial for leamers with limited background l c n ~ w l e d g e ~ ~ Since success is an enormous motivating fator in learning, multimedia should be
deigned to ensure success. This can be achieved by stating the objectives, moving from
simple to complex, monitoring the performance of the learner and adjusting the itlstnrctional steps according to the characteristicsof the leamer and the content
Another educational principle that has to be applied to the design of multimedia is to
ensure that there is a congruence betmen the objective, instruction and assessment AIthough elemental, this principle is ofien overlooked- Objectives have to be carefirlly written (a discussion on writing objectives is beyond the scope of this thesis) as
objective writing is an art and science in itself? With regards to assessment, consistent
assessment format, anticipation of as many answers as passiile, judgement of answers as an expert would, and variatl* in presentation of questions are characteristics of multimedia that can enhance learning The issue of learner control is still unresolved Research indicates that learner control has to be varied according to the learnern- For effective and appropriate learner control,
multimedia should be designed in such a way that the learner is presented with a concept map of topics; has the facility to review and access segments, start, stop and resume according to desire, can hypedink to choice topicdsubtopics, and is informed about the time required to complete each module. Principles underlying screen design should also be incorporated This includes, amount of text on each frame,colour, fonts, use of graphics among othersLg. Another important aspect of multimedia, though obvious, is the appropriateness of the
-
imtructional media to the content The media be it text, video or voice should be selected according to the capacity of that specific medium for achieving the instructional objectives. For example, a video clipping rather than text, may be more
appropriate where motor skills need to be leamt The media for response such as verbal, m - t i n touch screen also has somc effect on the learning process. Media selection guides - tools developed to determine the best medium or combination of media for an instructional task are available1ss1. 5 5 9 Over twenty media selection models exist, but the similarities outweigh the differences? The most common form of selection model uses the matrix format which
matches media attn'butes with specific instruction events or pdcal
benefits6'. Others
use wo~heetr/check-listssL, or flow charts or algorithms to guide the selection process? Some of the factors that need to be taken into account include choice of a particular instructional method, the type of learning task facing the student, learning
characteristics of the student, practical constraints and teacher attitudes and skills. For example, if the instructiotliil method or the learning task involves group discussion, opportunities for feedback and interaction have to be increased Administrative and
economic factors may limit the choice of media The attitudes and skills ofthe teacher have been identified as one of the key factors that affect the integration of media into the cruricullum60.
Modes and Desrggrtsqf MuItiirnedla
The major classifications of multimedia include tutorials, drill and practice, simulations, instructional games, decision support systems, and exploratory environments
among others. Hybrid designs - a combination of two or more designs also exist In tutorials, the information is taught, verified and reinforced through interactiom It is generally used to present new information to learners. Effctive tutorials should have orientation of information, guidance to learner and appropriate feedback Interaction is a key concept that should be included. The principal limitations of tutorials is related to
the time required to design and develop an effective one that attains higher-order
objectives. Drill and practice designs provide for defined skills, immediate feedback to the student and some form of remediation They are usually used to reinforce skills taught
elsewhere. Drill and practice has the advantage of providing an opportunity to refine skills by providing immediate feedback and keeping accurate assessment of the learner's respoase. Although considered boring, this design can be enriched by appropriate incorporation of colour, sound and other features.
Simulations are designs which appmrdmate, replicate or emulate the features of some task, setting or context
a where the learner is a
very much part of the
scenario. Such a design comes in useW in those situations where the costs ofaltemative teaching methods are very high, when it is impossible to study the concept in "real time", when the risks of demoaseating in a real setting are high, endangerment of patient or when the instructional content c a ~ obe t observed by the naked eye, or the
process is too slow or too f~ Some of the areas where simulation has found extensive use include pilot training (using flight simulators), training of smgeons, anaesthetists and nurses. An effective
simulation should provide a realistic scenario, clear options for learner participation., have plausible consequences and guidance for completion of the scenario. This design
allows for development of higher order skills such as application, problem-solving and synthesis. The limitations of simulations may be related to the necessity of advanced
computer systems and significant challenges for lesson designers. In the health education field, the learner may not accept simulation as a substitute for reality. Also, the speculative concepts or competing theories behind biological models cannot be fully
explained as can be done by a teacher. Assessment of the outcome of simulation is also
difficult as it afFects a participant in many dimensions. Another design of multimedia is the use of decision.support systems. These systems attempt to emulate experts at problem solving These are tools that can provide health professionals with strategies to analyse, evaluate, develop and select solutions to complex problems. Such systems can be used for training students of medicine in
history taking physical examination and diagnosis of disease. Presently, the limitation is the challenge for designers.
One of the newer designs of multimedia is virtual reality64, 65. This utilises computer based technology for simulating visual, auditory and other sensory aspects of complex
environments Here the users can move and act as if they are part of the world It also allows for complex mechanisms to be studied three dimentionally. At present, the cost of equipment and the need for powerfirl and complex technological requirements are the Limiting fmtors.
In summary, multimedia has great potential in health education and programs can be designed in many different ways to suit specific needs. However, the characteristics of
effective multimedia have much in common, though the importance of each characteristic varies according to the design. Hence, it is feasible to have a standardid evaluation tool to assess the quality of the product The purpose of this thesis is to develop and
validate such a tool. While the development ofthe tool was based on research on the characteristics of effective multimedia, the validation was done based on scientific measurement principles. Measurement principles revisited-
Two important characteristics of measurement in establishing the appropriateness and usellness of any measurement instrument are validity and reliability Vofidr'ty of measurement is the extent to which the instrument measures what it is
designed to measure. Reliabiliky is the consistency of the instrument in measuring whatever it measures. Validity refers to the appropriateness of the interpretation of the results of a test and it
is specific to the intended use. There are differeat types of evidence of validity. Content-relatedevidence or content validity, is the process of establishing the represen-
tation of the items with respect to the domain of whatever is being measured Face validity refers to the evaluator's appraisal of what the content of the test measures. Criterion-rehed evidence
-
concurrent and predictive validity are approaches to
establishing validity in which the relationship between the test scores and measures of performance on an external criterion are determined In evaluating concurrent validity,
the test scores and criterion are collected at or abut the same time, while for predictive validity, the test scores and criterion are collected after an intervening period Constmc~~re~ated evidence or construct validity is used in situations where theoretical
attributes such as moral reasoning, personality traits and creativity are involved Several procedures can be used to estimate rdMili@. PuraIIel form or alternate fonnrprocedure involves using two or more equivalent forms to the test The two forms are given to a group of individuals with a short interval between administrations. If the
test is reliable, the pattern of scores for individuals will be the same if there is a high
positive associatioos as indicated by a statistical measure ofcorrelatio~
In the test-retest procediwe, the same test is administered on two or more occasions to the same individuals. If reliable, there will be a high positive association between the
results obtained on the two administratioas.
In the split-harfprocoCwe the test is administered once and the test items are divided into two halves of q u d difficulty and content and the scores of the two halves are
compared. If reliable, there will be a high positive 8ssociation between the results obtained for the two halves.
In the Kuder-Richardson procedwe, one of the two formulas - KR-20 or KR-21 are used. These provides the mean of all possible splithalf coefficients.Cronbach alpha is another formula used for estimating reliability. In terms of establishing reliability of questionnaires, inter-rater and intra-rater reliability
may be estimated Inter-rater reliability is established by correlating scores given by two or more raters judging the same event Intmrater reliability correlates scores given by
the same rater on two different accasions while judging the same or similar events.
a estabIish the validity and reliability of hstmnents in educational
It is important
research, before dissemination.
Learning outcone using multimedia
While there are tnany studies that have compared the learning benefits o f different Cla& in 1992, on reviewing hundreds of these comparative
instructional m&&,
studies indicates that measured achievement gain that was attn'buted to a medium was more due to filure to control o w or more of the codomding factors than to actual
gains? Three common codomding fhctoa have been identified. First, different
-
instructioaal methods were used in difEeretlt treatments this influenced learning in the treatments that employed more effecfive methods. Second, different informational contents were presented in the compand treatments
- learning gains can be found
associated with treatments where one finds information necessary to answer post-test items but not in treatments wbere critical infomation is absent Tbird, the novelty of newer media tends to increase the persistence and effort students invest in the early stages of new programs, but this is short-livd
Studies &re
these confounding h r s have been conmIIed show that there were
significant student eflciency learning benefits when compared to text-based version6.
-
But no student profiiency gains were found i.e. students reached acceptable levels of
achievement with both versions. In the few studies which have addressed learning retention, no conclusions could be drawn70. 71.
Many studies3'
'"
'O
have shown that less time may be spent in learning using
multimedia although the learning outcome has not been shown to differ concretely in comparisonto other forms of instruction,
In multimedia designs such as simulation and virtual reality, although comparative
studies are few, it can be stated that even if learning outcomes are not different, the that such designs bring to the field of educatioa
unique capabilities and
especially in the problem-solving domain, is reason enough to udilise them. lark?, in his reviews on dangers of evaluating media, concludes that it is likely that any
instructionalmethod and any information-a can be delivered by more than one medium and a rational approach to the problem may be to do an economic analysis of all instructional alternatives, in addition to taking enjoyment and other factors listed by existing media selection models into wnsideration On doing SO, it may be found that some portions of a training program such as delivery of routine information, can be economically delivered by inexpensive media, while other parts such as highly complex diagnostic and problem-solving lessons may require more expensive media
-
Evaluation of Multimedia current resources available
While the learning outcome of multimedia is debatable, there is no doubt that it has
-
great potential in education especially in health education, where one of the mandates
is to train individuals to be self&ected aad life long learners. With the rapid increase and changes in knowledge, it is important for health professionals to keep current with the changes, and technology seems to be the answer. As a result, multimedia packages
have flooded the market aad educators and students require a method to judge the suitability and quality of the multimedia A number of disciplines have produced evaluation tools26.2s.
47. R-77
The. Computer Assisted Learning Evaluation to#
developed by the University of
Calgary, is common for all disciplines and educational levels and uses a checklist
format The Council of Ministea of Education in Canada4', also has a checklist tool for evaluating primary school resources. The widely usad York Educational Software
Evaluation scalesab,is mainly applied to primary and secondary educational programs. It consists of guidelines which are used by a panel of reviewers. In the health sciences, the Council on Collegiate Education for Nursing in affiliation with the Southern
Regional Education Board have developed the SREB software evaluation tool'6 to assess resources for nurses.
The Faculty of Medicine, University of Calgary has
developed a Medical Software Evaluation tooln to help committees select and purchase medical software. The tool evaluates different aspects of s o h e using items each to be rated on a 5 point scale. Atkins and 0'EIaUorans discuss the general guidelines that can be used for evaluating multimedia applications for medical education. But no tool
is presented Xakellis a a p descni the use of students to evaluate soffware,but this is
not feasible all the time- While all ofthese evaluation tools have much in common, not all address the unique needs of health education. Evaluation of resources for health education have to be done by more than one reviewer due to the presence of numerous subspecialties. For a global evaluation, expertise in content, educational theory and technical aspects is required Also, one package may
have subject matter that can be used at different educational levels, e.g, first year
medicai student or trained clinician, physiologist or anaesthetist. Hence the evaluator
has to assess in the light of more than one (stated or illlstated) target population the package addresses. In health education, althoughjournals do exist that review multimedia29' 31
these are offen subjective, coloured with the prejudices of individual reviewers.
"
The few cataloguesn* that are available, though very usefbl as a compilation of what is "out there", do not have reviews for many ofthem, Hence, there is an urgent need for a standardized evaluationtool and a clearing house.
CHAP'CER THREE
There has been a rapid increase in recent times in the use of multimedia in education
especially in the tmhhg of health profmionsls. This increase is not only due to the easy availability, accessibility, low cost of technological resources, but also to the
expanding body of research on the role of multimedia in nursing, medical, patient and continuing education13.14.20
-
Given the above evidence, there is a speedy turnover of new multimedia packages in the market Numerous resources are also available in the world wide web,
like the
Virtual ~ o s ~ i t a lThe ? quality ofmany of these resources is debatable. Since the role of educators is to assess and select learning resources based on curriculum goals and
student needs, development of standardised methods for multimedia evaluation using
sound educational and evaluation principles becomes vitaL Evaluation is hampered by various ktors. For one, the packages ace not freely
available for independent review. Sometimes,the b a r d m required are available only
-
in larger institutions of come, this picture is changing rapidly. In addition, many distributors require a wn-reftndable deposit even to review the material. In health
education ,there is no clearing house set up to objectively evaIuate all such resources. The few peer-reviewedj o u t n a l ~that ~ ~ -offer ~ ~ reviews of softwate are o h n subjective in nature, giving only an overall impression. Hence there is an urgent need to develop a
standardised evaluation tool, Purpose
The purpose of this study was to develop an evaluation tool and establish its' validity
and reliability. Specifically, the study: +
+
expands on the process ofdevelopment ofthe tool
describes the evaluation tool
* details the process of validation ie. -judgement of the contents ofthe tool by a group ofexperts
- comparisonofthe assessment ofthe same multimedia product by faculty and students - ability ofthe developed tool to discriminate between quality mdtimedia resources fiom others
- identi@ the Ieamhgoutcomeof students using multimedia products of Mering qualities but addressing the same content
CHAPTER FOUR
Development of the tool
The evaluation tool was developed after extensive research of the literature5*'
16*
19*
26,47,76.n and several discussions with educationaland technological expects. Evaluation tools already develop#
"
"*
n* specifically, that developed by the Faculty of
Medicine, University of caI&aryn and books on design, development and evaluation of multimedia1g* were consulted. Instructional design and educational psychology b k s 6 , 19.
were referred to in order to ensure that the too1 addressed these areas
adequately and appropriately. The characteristics of simulation4*
" were
studied and
the section on simulation was developed Aftw repeated revisions, the first version of the tool was developed (Appendix 1).
Validation of the tool
In order to ver@ faoe and content validity, copies of the tool were distributed to 11 experts. An "expert" was defined as an experienced hculty member, who was actively
involved in teacbing at the university level and vesy howledgeable in Educationat Technology, Educational Psychology, Infomatics &or
Medicine. Of the 11, eight
participated as three were unavailable at the time of study. The expects were asked to comment on each of the sections of the tool and rate them on a five point scale using a
separate form that was constructed for the purpose (Appendix 2). Every suggestion made was compiled and changes were made to the evaluation tool in accordance to the
suggestions, after delikratioa and discussion with an educational expert and the reviewer concerned The modified tool ( ~ b d i 3) x was then utilised for other aspects
of the study. Description of the tool (Appeadu 2):
The evaluation tool was designed to serve as a comprehensive guideline for reviewers.
It begins with an introductionthat explains the purpose of the tool highlights the
contents and strategies that reviewers can employ while assessing the resource. The rest of the sections consist of items that are rated either on a five point scale or as yedno. To
avoid ambiguity, descriptors identify the 1 and 5 point levels of the d e The content has 5 sections:
* Defru'hufrevtewer
- addresses the date o f review; name, academic background and
area of expertise of reviewer
* MuffiimediorQsource &n&*ty
- addresses the title; name of publisher, publication
year. type of multimedia resource e.g tutorial, drill and practice etc; subject addressed; target population and setting e-g. independent learning etc.
-
Tccirnial requirements addressees the type of computet; monitor and system requirement; network capabilities, copyright details, availability of technical support and price
* Detailed ewoludon - addresses the overall @ty
of the resource in terms of user
interface, content, educational principles and simulation
* Evaluation
Summrrry
- is a compilation of the overall rating in each of the
subsections of the detailed evaluation; gender/raciaVethnic/religiousbias and a summary of how this multimedia product adds to existing single medium products.
The tool has been designed such that it can be easily adopted eventually as a world wide website, to be directly accessed by a reviewer. The preliminary design of the website has been developed
-
Reliability of the tool (i)
In order to determine the reliability, four volunteer nursing students, four medical students and four members of the faculty were requested to view the same CD-ROM
and assess it with the help of the evaluation tool.
The Cardiovascular Physiology CD-ROMdeveloped by A D A U Software Inc. and Benjamin/CUmmings ~ublishinflwas used for the putpose. The four faculty members were chosen on the basis of their medical background, active involvement in teaching
for at least ten years, and interest in innovative teaching- The nursing and medical students were volunteers. The subjects were video taped while they viewed the resource so that verbal comments made by them on specific aspects of the resource could be captured As well, the video could be used, ifn-,
for stimulating recall at a later
date. The scoring and comments of the students and faculty were then compiled, compared and analysed.
-
Reliability of the tool (ii)
The discriminatory power o f the tool in identifying subtle differences between two resources of the same content, but of varying qudity was determined next A CD-ROM addressing the topic o f Peritoneal ~ i a l ~ s was i s ~identified for this purpose. The
resource is published by the Faculty of Nursing, University of Calgary. This particular resource was chosen as the publishers were accessible and it was possible to manipulate the quality of the CD-ROM With the help o f the programmers, certain elements such
as references, quality of screen d o u r background, some aspects of forward and
backward movement, formative quiz, and feedback for correct and incorrect answers
were removed Four volunteer graduate nurses were requested to assess the altered and unaltered
CD-ROM using the evaluation tool. Two of the nurses wem given the unaltered CD first, while the other two were given the altered CD. Later, the other version was assessed by the nurses. It was ensured
that at least one week elapsed between the
assessment of each of the CD-ROMs.In addition, the nurses were asked to identity qualitative differences, if any, between the two versions. The two sets of evaluation tool
From each of the nunes were then cornpared and analysed.
Learningoutcome
To identify if the quality of the CD-ROM affiected the learning outcome of students,
further studies were conducted First year nursing students enrolled in the anatomy and physiology course conducted by the Faculty of Medicine, were requested to volunteer for this study. The twenty students who volunteered were matched according to an aggregate of scores obtained in four anatomy and physiology course examinations and
divided into two groups. One group (Study Group A) was requested to view the unaltered CD while the other group (Study Group B) viewed the altered one. Two students were given one CD ROM to share. They were encouraged to view the CD at home, or at the Learning Resources Centres of the University or any convenient location and time. A log sheet pasted on the front ofthe CD case enabled students to easily keep
Log of the time they spent viewing the CD. At tfK end of three weeks, a post-test was given to the participants. Of the twenty students, 17 (7- group A; 10 - group B)
completed the study. An additional 15 volunteer nursing students who had not seen the CD took the post-test This latter group served as a controL Post test Validation
The post-test (Appendix 4) was constructed afkr consultation with a Psychometrician It was based on a table ofspecEcatiom preparedafterviewing the CD.Thirty multiple
choice questions were constructed and this was field tested on three volunteer students. It was e n s d that the test contained no technical errors.
The post-test was administered to all volunteers of the control group and some students of the study group in one sitting Other participants did the post-test individually. All students answered on score sheets (Appendix 5 ) which were then optically scanned
The scores were then compiled and statistically a n a m i Item analysis and test reliability were also calculated The scores obtained by the Control Group, Study Group A and Study Group B weie then compared-
Ethical Approval Ethical cleac~~lice for the study and approval of the content of the consent forms designed for all volunteers (Appendix 6,7X was obtained from the Ethics Committee, University of Calgary prior to the study. An informed consent was obtained fiom all participants. Statistical Analysis All statistical analysis was done using the SPSS version 6.1 for Windows and SPSS
version 6.1 for IBM Risc 6000 m*e
machines? Discussions were held with a
Statistical Consultant for employing appropriate procedures.
Descriptive statistics and reliability analysis, using the alpha model, was done for the
scores given by the eight experts for &e digerent components ofthe evaluation tool (see Appendix 1-3). Cases were deleted if they had missing values on any variable. A total
of 17 items were used for the reliability analysis.
In addition to descriptive statistics, we-parametric tests were used for anaiysing the
data obtained fiom the four nursing students, medical students and fmulty who bad viewed the same cardiovascular CD-ROM.The Kruskal-Wallis one-way analysis of
variance was used for comparing the three groups. The Mann-Whitney U test was used to compare faculty with the combined data ofthe students.
The data of the four graduate nurses, who viewed two versions of the same CD and scored them using the evaluation tool, were analysed using non parametric tests. The
Wilcoxon Matched-Pairs Signed-Raaks test was used for this purpose. One-way analysis of mhmce was done to compare the performance of students of
Study group A and Study group B, and those who did not see the CD (control) at all.
Pearson's correlation coefficients were computed for comparing scores obtained by
each student in the post-test with the aggregate ofthe scores offour tests the student had taken dwhg the first year nursing course. Descriptive statistics of the student answers to questions h u t the peritoneal CD that they had viewed were also analysed
Face and Content Validity Table 1 gives the academic background of the eight experts. AU except two experts bad
PhD or MD as their highest degree. The experts bad an average of 27.75 years experience as an educator in the field ofmedicine or technology. Figure 1 indicates the mean and standard deviations of the scores given by the eight experts for various subsections o f the evaluation tool using the assessment form (Appendix-2). A scale of 1-5 was
used with 1 indicating adequacy. All subsections
except simulation were given a score above 3, the midpoint
The suggestions made by the reviewers were mainly relaud to clarification of words and changes in sentence formation to improve understanding. Two reviewers indicated that the content o f the "simulatioa" was inadequate and confbing, while another suggested removal of the simulation section This is reflected in the mean score of simulation in Figure I. In general, the reviewers indicated tbat the tool was thorough, comfortable to use and addressed all aspects.
-
Reliability of the toot i Contparisson of the evaluation 4 a Cardiovascular Physiology CD by four nursing st dents. four
m e d i d students a dfour faculty.
Subjects:
The four nursing students (two males) were in their first year of undergraduate nursing. The four medical students (two males) were in their first (n=3) or second @=I) year of medicine. The faculty (3 males) were professors in clinical medicine (n=l) or special-
32 #
cardiovascular physiology ( ~ 3 ) The . subjects took approximately two hours 114.55; S.D. 60.85 minutes) to view and evaluate the resourcece There was no
significant difference in the time taken by students and fmulty.
On comparing the way the three groups perceived their expertise in the three areas of content, education and tedmology, the faculty had rated themselves low in expertise in the area of technology. However, all three groups did not assess the CD-ROMin terms of technological requirements W
e usingthe evaluation tool-
There was ambiguity between the groups in identifLing the type of multimedia resource, and target population. While the faculty agreed among themselves in that the resource was a tutorial, the students indicated that the resource could be drill and practice, reference, electronic book or combination of many types despite the presence of definitions of each of these design modes in the evaluation tool. The students also had difficulty identiwg the suitability of the resource for different target populations.
Figure 2 compares the overall scores given by the three groups for content, educational
principles and user interface, where 5 indicates adequacy. There was no significant difference in the way the three groups scored the CD-ROM in any of the criteria (e.g. technical requimments,gd, information etc) when analysed using the Kruskal-Wallis one-way Anova test On combining the scores of the students and comparing with the
scores given by the faculty using the Mann-Whitney U test, no significant difference was seen In general, thece was higher agreement between the students and faculty for
items related to documentation, statement of goals, feedback and gender bias.
-
Reliability of the tool ii Comparison of the scores given by four graduate nvses while viewing two different versions of the Peritoneal Dialysis CD using the evaluation tool.
No significant difference was seen in the way the four nurses evaluated the two versions, of the CD-ROM, when analysed using the Wilcoxon Matched-Pairs Signed
33
-
Table 1: Face and Content Validity The academic backgroundof the eight experts
EXPERT
YEARSAs 50 EDUCATOR
25
34
Figure 1: The mean scores or various subsections of the evaluation tool, as rated by the
r
eight experts. (The stmdmd deviation is given above the bars) 4-0
--
MTLENGH
IengthofintroQlttion
NTmFO
ratbrmrtionin hrtmduction
D
ovarllseaefb~farevinrras
m
REsRCmD
rcsapce-
m H R E Q
tcl?lar'M'-
DEUSER
rbllrrl~-l~~~iatafret
DEUSERSCR
ovarll~forusain!afhc
DECONGOAL
darilcdeurl~-~goal
DECONINFO
dariledcvshutiar -colltcat: i n f i
DECONLANG
&ailalcvduruiau -amtcnt: language
DECONSCR
o v d score tor amtcnt
DEEDPRAPP
detailed cvaiuatim educational principles: appcaranu
DEEDPRNAV
derailcd evaluation - educational ptinciptcs: navigation
DEEDPRSCR
ovcrall scorc for educational principles
DESIM
simulation
DESIMSCR
overall sum for simulation
EVASUM
evaluation summary
FMALSCR
fmal scorc
-Ranks test However a trend was seen in the way the nurses assessed the altered CD. All aspects except overall content, language and gender bias were given a lower score
by the nunes for the altered CD. On being questioned as to whether they found any
difference in the quality ofthe two versions of the CD, three of the nurses indicated that the altered CD was inferior to the unaltered The fourth nurse indicated that she was
unable to remember if there was any difference as three weeks had elapsed between the viewing of the two versions. It was noteworthy that the four graduate nurses, like the faculty in the previous case, evaluated themselves as low in technology expertise. The nunes too did not evaluate the technical aspects of the CD-ROM while using the evaluation tool.
Learning Outcome
The post-test was field tested and found to be valid and devoid of technical errors. The
reliability of the items in the post-test was high if the scores of the study groups were used. An alpha coefficient of -75 was obtained This indicates that the performance of the students was consistent However, the reliability of the items was reflected as very
low (alpha coefficient = -0.57) if the scores of the control group were used, indicating that the performance of the better and poor students was unpredictable in this group. Table 2 compares the aggregate scores (i-e. mean scores of four exams done by the nursing students during the first year of nursing) and Table 3 the post-test scores of the three groups - control, study group A and study group B using analysis of variance. No two groups were significantly different at the -05 level. However, F probability of the post-test scores was -0663as compared to only -6816 of the aggregate scores. A higher sample size may have identified significant differences in the post-test scores.
Figure 3 and Figure 4 compare the aggregate and post-test scores of the control and study groups. The results of the study group A and B were combined as no significant
Figure 2: Comparison ofthe overall scores given by the medical students (n=4), nursing
students (n=4) and faculty (n=4) for the Cardiovascular Physiology CD-ROM.
Content 7. %
Wdstudcnb
Educatlod Principles aumiogstudentr
User lnterfrce thcuft~.
Table 2: A comparisono f the aggregate scores (mean scores o f four examinations taken by nursing students during the first year of nursing) o f the Control, Study Group A and
Study Group B.
DF
Sum of Squares
Mean Squares
F ratio
F probability
Betweea Croups
Within Groups Total
Table 3: A comparison of the post-test scores o f the Control and combined (A & B) Study Groups.
I DF Between
Groups Within Groups
Sum of Squares
Mera Squares
F ratio
F probability
(
Figure 3: A comparison of the aggregate scores of the Control and combined (A & 8 )
Study Groups. The shaded boxes extendfiorn the 25th to the 75thpercentile (quurtilcs) of rhe dafa The dark horizontai h e at the centre ofihe box marks the 50th percentile. The lines projecting out from the box extend to the adjocent vuiues. The acQacenf values are the most extreme observations rhaf are not more that 1.5 times the height of the box beyond eilltcr quartile. All points outside this range (outliers) are represented by circles.
Control vs- Study Group
Figure 4: A comparison of the post-test scores of the Control and combined (A & B)
Study Groups.
I
0
11
502 PI
.
15
IT
Control
Sludy
Control vs Study Group
difference was found in the way the two groups performed in the post -test Although the difference was not statistically significant, while the performance was similar in the aggregate score (Figure 3) the combined (A & B) study groups performed better in the
post-test (Figure 4) .
Table 4 shows the results of correlation between aggregate scores and post-test scores of all participants, control group, study group A, study group B and study group A and B together. In gene*
those who bad higher scores in the aggregate scores performed
better in the post-test- This is reflected in the results ofstudy group B and the combined scores of the whole group. There was no correlation between the aggregate scores and post-test scores of the control group and study group k
h addition to the post-test, the students in the study groups were asked to comment on various aspezts of the CD they viewed In summary, the students felt that the CD was easy to follow, clear, concise and interesting. They expressed the view that the
presentation of the material in a variety of formats such as text and video, with good colour and sound, helped to enhance learning. The stepby step approach (sequencing) and the inclusion of questions, guidelines and m-onale (orgauisation) also helped. Other positive comments were that this resource allowed the learner to pace themseives,
and was accessible even at home.
Some of the negative comments were that the start-up information (documentation) and degree of control (branching capabilities, navigation) was insuffkient Difficulty in accessing computers made the whole process inconvenient for some studeats. There were also comments about the repetitiveness, lack of variety and feedback in the quiz
Some felt that there was too much information in each frame. One student expressed that this form of learning may not cater to all learning styles and that she, personally learnt better using text and demonstrations.
Table 4: Results of the correlation between aggregate scores and post-test scores of all
participants, control group, study group A, study group B and combined (A and B ) study &roupS-
P (kvdof significance)
All participants (n= 32) Control group (n= 11) Study group A
(n= 7) Study group B
(n=lO) Combined study group
(A & B) (n= L7) I
* significant
CHAPTER SIX
Given the repid strides made in technology and its applications in health care, health
care research, training and education, use of multimedia is on the rise. Research in education also indicates that the use of mdtimedia in learning bas its own advantages3'.
Therefore there is a need for standardisingevaluation tools that can asses the quality of the resource. Although many guidelines exist *'
q none ofthem are standardisxi.
The key to any measurement process is the determination of validity and reliability?
Face, content, concurrent, predictive and construct validity are the major types of validity that have been recognised in educational research Face validity is subjective and refers to the evaluator's appraisal of what the content measures. Content validity reflects the extent to which the different compoaents of the content universe are
represented, taking into account the importance and relevance of the component Coacumnt validity, commonly used in different achievewnt or aptitude tests relates test scores of a group ofsubjects to a criterionmeasure administered at the same time or
within a short interval of time M
e predictive validity relates to codinnation of test
results to behaviour of the subjects at a later date. Construct validity, more commonly used in psychology, refers to the extent to wbich a particular test can be shown to
measure a hypothetical construct The importance of determining each of the types of validity varies according to the type of researchbeing conducted..In this study, faoe and content validity is of greater importance. In terms of the evaluation tool, how do experts appraise the contents of the tool? How was the conteat of the tool arrived at? In terms of prediction, what was the learning outcome?
Reliability is another measure that should be catehlly considered. It determines the
level of internal consistency or stability of the measuring device over time. The level of
reliability expected is determined largely by the nature of the research High reliability should be expected if the research requires very small differences to be determined Le.
if higher sensitivityand discriminatory power is required Reliability is expressed as a reliability coefficient which reflects the extent to which the
measuring device is free of measurement error. The closer the reliability efficient is to the value of 1.00, higher the reliability. Ln this study, the reliability of the eveluation tool as well as the post-test had to be determined In terms of the evaluation tool, how do different individuals using the tool compare while measuring the same multimedia
resource? How discriminative is the tool is differentiating small differences in the quality of the resource?In terns of the post-test, how well does it discriminate between
performance of good and better students? Another aspect of standardisation is the determination of reference points.
The stan-
dards may be norm-referenced or domain-referenced Nonn-referend measures com-
pares performance ofone with another. Dornain-reference compares pwforrnance with a precisely defined content area In terms of the evaluation tool presently, there are no existing gold standard for quality multimedia However, the qualities that are desirable
in a multimedia resource have been compiled by reference to literature in educational psychotogy and hstmctional designz
Multimedia reviewss* 3' indicate quality to
some extent. But, the reliability of the reviews can be questioned as the quality of the multimedia resource has been determined by individual reviewers using no predeter-
mined, statrdardisedcontent domain,
Face and Content Validity The experts who appraised the tool were all educators with many years of experience in fields such as instructional design, educational technology, informatics and medicine (see Table 1). There was general agreement between the experts in all aspects of the
evaluation tool as indicated by the scores given using the assessment form.
On looking at the reliability of subsections ofthe tool the section on simulation was the
lowest. Inte~*ews with individual experts provided some insights. One of the experts
thought that the section on simulation should be removed and the items interspersed in other sections as many questionsthere were related to simulation software. It was
explained that this section was to be considered in adition to the other sections if the software was one of simulation. The oldest of the experts mentioned that many of the
terms used in this section were unclear and coafusing to h i m This may be so, as
simulation is one of the more recently developed innovative methods. The items were
-
modified and simplified due to this revelation Two other experts who were involved in clinical simulations used in standardised Royal College Medical Examinations thought that this section was superficial and required additional details. The suggestions
made by them were incorporated in the modified tool. The sections on educational principles and user interface had the highest reliability coefficients. In general, the experts thought tbat the tool appraised all desirable characteristics of quality multimedia. The validity of the tool may have been enhanced if the experts had scored the same resource using the evaluation tool. But this, though ideal, was not possible due to time constraintsts
It was noteworthy that each expert was able to bring hidher own unique contriiution to improve the tool. This indicates that a team approach is required for quality evaluation tools to be developed, and that literature review done does not su5cece It also indicates the importance of face and content validity.
One limitation of the tool that can be rectified is the fact that the tool itself is not multimedia. Conversion of the tool into a multimedia form opens up great possibilities. For instance, the tool can have branching features so that it gives the reviewer an option to evaluate a resource in any sequence - evaluation summary to details or vice versa. Also, it opens up avenues for varying the emphasis on different aspects of the resource
according to the type of multimedia -viz drill & practice7 tutorial, simulation etc. For
-
exampie, in simulation realism, decision points, coosequences of decisions made need emphasis.
In addition, the tool in a multimedia format can make it easier for reviewers. Reviewen in different parts of the globe can access the tool if it is oa a website. It may also be possible to split the monitor screen to view the resource in one part of the screen and
input reviewer comments into the multimedia tool using the other. As well, input of reviewers can be pooled in seconds and disseminated to a website and access to the interested public. A preliminary design ofthis evaluation tool in multimedia format is underway.
Reliability of the tool 4
To add to the validity and inter-rater reliability as determined using experts, reliability was further explored using faculty and students to evaluate a cardiovascuku physiology CD. Overall, no significant differences were seen in the way WuIty and students scored this CD. In general, the faculty tended to be conservative in their ratings. The variability between the scoring pattern of f d t y and students was Low in the areas addressing general issues such as documentation, appearance and navigation. Notably, only the faculty were able to identifil errors in content A number of comments were made on the infonnatioa presented by the faculty whereas the students had little or no comments. It can be concluded that general areas such as navigation and appearance can be reliably assessed by students or faculty, but areas related to wntent(educati0d pinciples and
the content of f&k
is better done by faculty experts. Another striking finding was
that both faculty and students alike did not assess the resource in terms of the technical
requirements. Although no one perceived themselves as experts in this area, the lack of
an attempt in the part of the subjects to fill questions related to this indicates that for a complete assessment, the input of a technical expert is required The advances in the
capabilities of computers and the introduction of open systems bas made assessment of this aspect of multimedia less important, at present Assessment of this aspect is objective and can be done by any technicat expert. However, if more sophisticated multimedia resources such as virtual reality become easily accessible this section of the tool will have to be modifiedand emphasised.
-
Reliability of the tool ii. An attempt was made to determine the discriminatory power o f the tool by requesting
participants to view two versions of a CD addressing the same content. Although the
number of participantswere too few for generdizability, interesting findings were seen. The CD addressing the topic peritoneal dialysis was developed locally. Hence, it was
possible to produce another version of the CD with the same content but without some
aspects such as references, glossary and fomtive quizzes. Using the evaluation tool as a guideline, these elements were identified and the programmers were requested to remove these elements in the modified version It was hypothesised that the evaluation
tool would be able to identifythese missing elements. Three o f the four subjects felt that
the quality of the modified CD was inferior. This is reflected to some extent in the
scores given by the subjects in these areas. One of the drawbacks of this aspect of the study was that even the unmodified peritoneal dialysis CD had some deficiencies. The original CD did not have written or online instructions for easy navigation. No goals were stated. Prerequisites were not
mentioned Too much information was presented in certain fiames. A major drawback was that it did not have the facility for fhst forwarding the video segment which
-
appeared on every screen this waz pointed out by many of participants. No summary was given It did not have a reading list and references were few in number. The
assessment questions - both formative and summative were not directiy related to the contents presented. The options of one of the questions was wrong. The scoring system was also inferior. Hence, in spite of the differing qualities, it was not a comparison of a
high quality multimedia with another of inferior quality. The absence of basic elements in the peritoneal dialysis CD reiterates the importance of using an evaluation tool not only for assessing quality but also in the design and development process. Learningoutcome
A review ofthe literature indicates that the learning outcome is one ofthe more dBcult
areas to assess.
lark^^ states that this is mainly because of the difficulty in controlling
confounding factors.
In educational research, different approaches have been employed to determine learning outcomes8.Pre-test-post-test control group design, post-test-ody control group design, Solomon four-group design, factorial design, repeated measures designs are some of them. The post-test-only control group design was employed for this study. Though the
students could not be randomly assigned, they were matched in accordance to the aggregate scores obtained in four examhations held during the nursing coursecuSe The
validity and reliability of these examhatiom have been proven over the years. The control group was recruited only on the day of the post-test and later matched with the study group. This avoided the element of cross contamination between groups. One of the limitations of the design is the small number of participants and inability to
randomly allocate groups. Due to paucity of h d s , incentives could not be provided to attract more participants. Despite the well planned design, a few variables could not be
controlled Firstly, the students were not hrlly motivated to see the CD and write the post-test as this was not part of their scheduled program. Although the CDs were duplicated such that it
could be shared between two people, many of the students did not possess computers
and had to rely on the ones available in the campus, which made the whole process
inconvenient for some. The time lapse between viewing of the CD and taking of the post-test was variable as some students had viewed the CD on the day it was given to
them while others could find time to view it only much later. Moreover, the students were not motivated enough to view it over again before the post-test due to lack of
incentive. Due to
in.classschedules, not all student volunteers could take the
post-test at the same time, hence the administration of the test was not uniform.
Therefore, the results of the post-test have to be interpretedcautiously, despite the fact that the post-test was validated
Although there were no statistidy significant differ-
ences, Figure 3 and 4 show that some learning had occurred in the study group. Further studies with a larger sample are required for generalizabilityand better interpretation of
the results, Analysis of the comments given by the study groups (A & B), gave further insights. All except two students found the experience interesting- One of the reasons given by a
student, who was also one of the oldest, for not enjoying the program was that the CD did not suit her learning style. Other positive comments such as this method catering to different learning styles, a variety of stimuli being evoked, advantages of self pacing,
accessibility at any time and place, cornparrdwith those cited in the literatured?
CHAPTER SEVEN
Many evaluation criteria have been formulated in the form of check Lists for evaluation of software, a few specific for health education. Most of them have not been validated and serve only as a guideline for reviewers. With the increasing use of multimedia in healtheducation, a validated evaluationtool is neceSSBcy.
There is also an urgent need for a cleating house for assessing the rapid turnover of multimedia resources. It is envisaged that this tool can till these existing gaps as the present endeavour provides evidence that it is valid aud reliable. The agreement between the 8 experts has established the fhce and content validity of the tool. The
agreement between the nursing students, medical students and fxulty while evaluating the same CD-ROM indicates that the tool is reliable. The tool was able to discriminate between programs of varying quality. Though the d t s were not statistically significant, a trend was detected
The obvious next step is to disseminate this evaluation tool. Presently a website is
being developed for the tod. It is hoped that in the near future,reviewers will be able to use the tool to evaluate multimedia and submit their input to a common website where
-
the comments can be compiled and be made available on-line the dawn of a clearing
house! This evaluation tool w ill be presented to other health educators at an international conference on Health Informatics to be held later this year, and then published in print. For the tool to be kept cunent with the rapid advances in technology, it is
understood that it will have to be modified and validated fiom time to time, As for the learning outcome of multimedia, further carefully designed research using a
larger sample size is required. However, the focus of the question in the mind of all
educators should be on qualitatively analysing how this medium can be ldilised to
enhance learning rather than how much laming occumd with this medium as compared to traditional methods.
-
artificial intelligence the ability of a robot or computer to imitate human actions or skills
such as problem solving,decision making, learning, reasoning and self-improvement branching
- the capacity to execute
an alternative path through the lesson; allows
adaptation to the needs of individual learner, i-e.enables slow learners to review, fkst ones to skip and uses menus from which students can select aspects of the topic according to individual needs, preferences or performance differences
CAI - computer-assisted instnrctioa; the application of computer technology to solve an byte the basic measuring unit for memory capacity or storage capacity
instructional problem
-
CBT computer-based training; instruction delivered primarily via computer
CMI - computer managed instruction; application of computer technology to monitor, track, and report on student and lesson performance CAL -computer-assisted learning; synonymous with CAI
-
CD-ROM computer disc-read only memory
-
Decision Support system a program that draws on a collected data base of knowledge via
an expert system to suggest courses o f action based on known idormation
-
documentation the orderly presentation, organisation, and communication of informa-
tion
-
driil and practice program designed to reinforce and repeat skills learned elsewhere
-
feedback information that tells students about the accuracy ofthe response given by them
iastructiom
- a debirate arrangement of experience!@)to help a learner achieve a
desirable change in performance interactive - two-way commUnic8tionbetween the student and computer
-
interface a device or program that permits one part ofa computer system to work with another, as when making a connection betweena cassette tape recorder and the computer
-
kilo, mega byte prefixes that represent certain multipliers. kilo, when talking about computer memory is changed to 1024 (as compared to 1000 in scientific notation); a megabyte
represents 1024 x1024 (1048,5760 bytes)
-
Krusk.1-Wallis test ranks d l cases fiom the different groups in a single series, computes the rank sum for each group, and computes the Kruskal-Wallis
H statistic, which
has
approximately a chi-square distriiution
-
learner control degree to which the learner can control the pace and progress of the lesson
-
learning a general term for a relatively lasting change in performance caused directly by experience
-
local area network (LAN) a way to connect computers that are physically close together
so that information, programs and equipment can be shared.
-
mainframe large, extremely fast, multi-user computers Mann-Whitaey U test
- m&sall the cases in order of increasing size and computes the
test statistic U, the number of times a score from Group 1 precedes a score from Group 2
-
media means of communication between a source and a receiver
-
multimedia use of more than one media for communication memory -internal part of computer (found in the Central Processing Unit) where programs and data are stored
-
objective the unambiguous description ofbehaviotus that is expected of the student at the
end of the instruction
-
RAM Random Access memory;this memory has no information in it, but is available for
any program to store information ROM
- Read -Only Memory, this memory has information already stored in it by the
computer manufacturer and nothing is allowed to change that information
-
simulation the process of using the computer to model a process or system. It is an operating model of central features or elements of a real or proposed system, process, or environment
-
software general tem for sets of computer instructions(prograrns) which manage the general facilities of the computer and control the operation of application programs system
- a group of actions or procedures which together are logically connected by their
operation and products and which accomplish a connectedset of organisational objectives
tutorial - a type of instruction where the computer serves as the primary instructor, evalu-
ating the the progress ofthe student and providing remediation if necessary
-
Wilcoxon matched-pairs signed-mnb test computes the differences between the pair of variables, raaks the absolute differences, sums the positive and negative ranks, and computes the test statisticsZ from the positive and negative rank sums.
1. Chodorow S. Educators must take the electronic revolution seriously. Acad Med.
1996; 7 I@): 22 1-226. 2. Shotsberger PG.Instructional Uses o f the World Wide Web: Exemplars and Precau-
tions. Educ Tech. 1996; March-April: 47-50. 3. Kearsley G, Lynch W, Wizer D. The effectiveness and impact of odine learning in
graduate education. Educ Tech. 1995; November-December 37-42. 4. Mattoon JS. Modelling and simulation- A rationale for implementing new training
technologies. Educ Tech. 1996; July-August 17-25. 5. Atkins MJ, O'Hallom C. Evaiuating Multimedia Applications for Medical Educa-
tion. Med Teach, 1995; 17: 149-160. 6 . Heinch R, Molenda M, Russell JD. Instructional Media and the new technologies
of insbuctioe New Yo&- John Wiley & Sons, 1982. 7. Muller, S (Chairman). Physicians for the Twenty-FirstCentury. Report of the
Project Panel on the General Professional Education of the Physician and College Preption for Medicine. J Med Educ. 1984; 59: Part 2. 8. Millman A, Lee N. ABC of medical computing
- CD-ROMs,Multimedia and
Optical Storage Systems.BMJ. 1995; 3 11: 675678. 9. Hannah R What is Multimedia and are we using it effectively?April 1997
(unpublished data).
10. Fishman S
M Palmtop Computers on the medical wards. J A W 1992; 267(1): 169.
1 1. Compton's Interactive Encyciopaedia, Ver 3. Compton's New Media Inc, 1995.
12. Piemme TE.Computer-Assisted Learning and Evaluation in Medicine. JAMk 1988; 260(3): 367-372.
13. tIanoah KI, Ball UI,Edwards Ulk Introduction to Nursing IaforrnaticsNew York: Springer-Verlag, 1994. 14. Frisse ME. The case for hypermedia. Acad Med. 1990; 65(l): 17-19. 15. Iennett P7Buell T (Coordinators) Learning Enhancement Envelope @ hmJ/
www.ucalgary.cal-IidsttE/LEE/htm, 16. Myers JD (Chairman). Evaluation of Medical Information Science in Medical
Education Executive Council of the Association of American Medical Colleges. J Med Educ. 1986; 6 1:488-543. 17. Ference PR. Vockell EL. Adult learning characteristics and effaaive software
instruction Educ Tech. 1994; July-August 25-31.
-
18. Barron AE, Orwig GW.New TechnoIogies for Education A beginner's guide
Colorado: Libraries Unlimited Inc, 1993. 19. Hannafin UI,Peck KL.The design, development and evaluation of instructional software. New Yoik Macmillan Publishing Company,1988.
20. Macfarlane AGJ. Interactive computing: a revolutionary medi urn for teaching and
design. Comp and Control Eng L 1990; July: 149-158. 2 1. Gagne RM,Briggs U.Principles ofIastnrctioaa1Design- New York: Holt, Rinehart and Winston, 1979.
22. Gregorc AF. Gregorc style deiineator. Development, technical and administrative manual. Mayoard, MA: Gabriel Systems, 1982. 23. Renzulli JS, Smith LK The learning styles inventory, a measure of student performance for instructional techniques. Mhsfield Center, CT:Creative Learning
Press, 1978. 24. Tobias S. Learner characteristics. In Gagw R M (ed). Instructional Technology:
Foundations. London: Lawrence Erlbaum Associates, 1987. 25. Schmidt HG, Norman GR, Boshuizen HPA A cognitive perspective on medical expertise: theory and implications. Acad M d 1990; 65(10): 6 11-621.
26. Owston RD.Software Evaluation a criterion-based approach Ontario: Prentice-Hall Cmada Inc, 1987.
27. Software for Health Sciences Education a resource catalogue. 7th Ed Michigan:
The University of Michigan Medical Centre, 1996. 28. Alberta Education, Computer Courseware Evaluations: June 1985-March 1986,
Edmonton, Alberta: Alberta Education, 1986. 29, Medical Software Review. June 1995, 30. Software review. Bull Med Libr Assoc. 1995; Oct. 83(4): 529-530.
3 1. Books, Joumals and Software. JAMA. 1996; Jan24/3 1 275(4): 328-332.
32. Merola AJ, Pengov RE, Stokes BT.Computer supported independent study in the
basic medical sciences. In- DeLand EC (ad): Information technology in health science education, New York Plenum Press, 1978, 17-43. 33. Barnett GO.The use of a computer-based system to teach clinical problem solving.
In: Waxman B, Stacy R(ed): Computers in Biomedical Research. New York: Academic Press, 1974, (4) 301-3 19. 34. Friedman RB. A computer program for simulating the patient-physician encounter.
J Med Educ. 1973; 48: 92-97. 35.Clancy WJ.Intelligent Tutoring. Cambridge: MIT Press, 1987. 36. Clark RE. Dangers in the evaluation of ~ c t i o n amedia. l 1992; 67(12): 8 19-820.
37. Kulik JA, Kulik CC, Shwalb BI. Effectiveness of computer-based adult education. J Educ Comp Res. 1986; 2(2): 235-252. 38. Clark RE and Sugrue BM Research on Instructional Media 1978-1988. In:
Educational Media and TecbnoIogy Yearboak 1988. Colorado: Libraries Udimited, 1989, 19-26. 39. Waugh TA, Mayer JW,Ewy GA, Felner JM,Issenberg BS, Gessner IH,Rich S,
Sajid AW, SafFord RE. Multimedia computer-assisted instruction in cardiology. Arch Int Med 1995 Jan; 1SS(2): 197-203. 40. Kauhan D, Lee S. Formative computer-assisted evaluation of a multimedia CAL
program in an ophthalmology clerkship. Med Teach 1993;lS(4): 327-340. 41. Lee W,Ault I!& Kirk JS,Cornstock C H Interactive multimedia for prenatal
ultrasound tmining. Obs Gyn 1995;Jan 85(1): 135-140. 42. Gardner HE, Multiple Intelligences; the theory in practice. New York: Basic Books, 1993,
43. Mckllam H.Situated m i n g continuing the conversation. Educ Tech. 1994;
October: 7-8. 44. Schlechter T.The relative instructional efficiency of small group computer-based
training. J Ed Comp. 1990; 6(3): 329-341. 45. Cohen M, Riel M The effect of distant audiences on student's writing. An Educ
Resident l. 1991; 26(2): 143-159. 46. Lyon HC,Healy JC, Bell JR,O ' h ~ e l IF, l Shultz EK, Moore-West M, Wigton RS,
Hirai F, Beck JR PlanAlyzer, an interactive computerassisted program to teach clinical problem solving in diagnosing anemia and coronary artery disease- Acad
Meb 1992; 67(12): 821 -828. 47. Software Evaluation Criteria for educational computer software evaluation Coun-
cil of Ministers of Education, Canada 1985. 48. Knapper CK Evaluating Instructional Technology. London: Croom Helm Ltd1980.
49. Blease D.Evaluating educational software. New Hampshire: Croom Helm Ltd
1986.
50. Jacobs PI, Maier MH, Stolurow LM.A guide to evaluating self-instructional
programs. Holt, Rinetwt and Winston Inc, 1964. 51. Locatis CN,Atkinson FD.Media and Technology for Education and Training Ohio:
Charles E Menill Publishing Company, 1984. 52. Roblyer MD.When is it "good coufsewacen?Problems in developing standard for
microcomputer courseware.Educ Tech.1981;2l(10): 47-54.
-
53. West CK Farmer JA, Wolff P M Instructional design implications fiom cognitive science. MA: Allyn & Bacon, 1991. 54. Cohen VB. A reexamination o f feedback in computer-based instruction: implica-
tions for instructional design Educ Tech. 1986; 25(1):33-36. 55. Lynch PJ. Visual design for the User Interface Part 1: Design fundamentals.IBC.
1994; 21(1): 22-30. 56.Skimer BF. Science and human behaviour. New York: Macmillan Publishing
Company, 1953. 57. Bruner L Beyond the information given New York: Mouton, 1973.
58. Mager W.Preparing instructional objectives-2 ecL Belmont, CA: Pitman, 1975. 59. Leiblurn MD.A media selection model geared toward CAL. Technological horizons
in education 1980; 7(2): 29-33. 60. Rorni~szo~ski kl-The selection and use of instructional media 2 ed. New York:
Nichols Publishing, 1988. 6 1. Gagne RM The conditions of learning. New York: Holt, Rinehart & Winston, t 965.
62. Greenblat CS. Designing games and simulations an illustcated handbook California: SAGE publications, 1988. 63. Jones K Simulations, a handbook for teachers. New York: Nichols Publishing
Company, 1980. 64. Field MI. Telemedicine a guide to assessing telecommunications in health care.
Washington DC: National Academy Press, 1996. 65. Rosen
Soltanian H, Laub DR, Mecinski A, Dean WK. The evolution of virtual
reality from surgical training to the development of a simulator for health care delivery - a review. In: Weghorst SJ,Sieburg HE3, Morgan KS(eds). Medicine meets virtual reality 4. Health care in the information age, future tools for transforming
medicine. Amsterdam: IOS Press-, 196; 89-99.
-
66. Borg WR, Gall MD.Educational Research an inaoduction 3rd Ed. New York:
Longman Inc., 1979. 67. Cohen C, Manion L. Research Methods in Education 2nd Ed. New South Wales:
Crmm Helm Ltd, 198768. Wiersma W. Research methods in Education:an introduction. 5th Ed. MA: Allyn
and Bacon, 1991. 69. Clark RE. Dangers in the evaluation of instructional media 1992; 67(2): 8 19-820.
78.Xakellis GC, Gjerde C. Evaluation by second-year medical students of their computeraided instruction- Acad Med 1990; 65(1): 23-26. 79.Spedick
Ingalls EC, Walters RE A ~ur~culum-oriented catalog of multimedia
learning resources. InProceedings of the MUMPS Users' Group. College Park, Md., MUMPS User's group. 1974, 163-170. 80. Virtual Hospital -http~/indy.radiology.uio~~~.eddvirtual hospital.html
-
8 1. Impact of New Technologies http~~liss.berkeley.eddirnpact/main. html
82. World lecture hall - h t t p ~ / ~ ~ ~ ~ u t e x a s . e d u / w o r I d / ~ c t o r s / l 83. Marieb E,Bmnstrom UT. interactive Cardiovascular Physiology. ADM S o h a r e Inc. & Benjamin ICummings Publishing Company, 1995.
84. Peritoneal Dialysis. Virtual Dream Team,Faculty of Nursing, University of Calgary,
1996, 85. Norusis UI.SPSS for Windows: Base system user's guide release 5. Chicago: SPSS.
1992. 86.Pagano M, Gauvreau K. Principles of Biostatistics. California: Dwbury Press, 1993,
-
Appendix 1 Evaluation Tool original
An Evaluation Tool for Multimedia Resources in Medical Education Introduction: This tool has been designed to be used by reviewers of multimedia resoutccs with special refirence to medical education- The purpose is to assess the overall quality of the tcsouccc and to identifj. the target population that the resource can serve- In addition, using educational and scicntificalty tested principles, identi@ weaknesses and strengths. The tool has been designed in such a way that all arpects of the specific multimedia resource may be reviewed in a structured and systematic manner-
Strategies for reviewers:
*
After reading the accompanying written documents ofthe resource (if present), run the multimedia resource several times to test t&t branchin&'' possibilities and ease of running the program.
* Use the program as you imaginethe most capable student would use it* Use the program as you imagine the leust capable student would use it-
*
Check the on-line help (ifpresent)-
Make dellkrate data entry errors (eg. strike multiple keys).
* Check the way responses ate made to erroneous answers, if the resource provides opportunities for self assessment Complete the Details of Reviewer, Multimedia resource identification and the Technical requirements section k pages 14-
Then anwer each question under the Detailed Evaluation, systematlCdy.
*
Finally, having obtained an ovewicw of every asped of thc product, 6ll out the Evaluation summary-
* Use the space ptovided to comment on any specific aspect of evaluation that might require more elaboration,
*
111
If the software crashes repeatedly, discontinue evaluation and stmte this in the evaluation summary.
Branching refers to the capacity of the resource to skip informationor to backtrack, as deemed desirable- [t enabtes slow learners to review, fast ones to skip, and uses menus fiom which students can select aspects of the topic according to individual needs, preferences or performance differences.
" 3
1-1
M
D
Details of Reviewer
Y
Date of review
Academic background
fscudenr
1-
program
( e g medical; nursing etc-)
year of study
0 (eg. lN3)
Area of expertise Please circle where your expertise lies with respect to the following on a d e of 1-5 (I =incompetent and 5= very competent)
I Multimedia resource identity I Title
1111111111111111111llllllllllll
Name of Publisher
1111111111111111111lllllll
Publicationyear
Namcfs of Authods:
I. 2.
contribution o f author, if known (please check) content educational technology
Type of muNimedia resource (pfease check the appropriafe bar you tncry check more than one ifappficabkJ r
Tutorial ii
Drill& Practice ;l
Reference (eg. encyclopediia) Q
Simulation i ;
Elecbonic book 5
Others 0 please state type
Subject addressed
Primarv Subiect @leasecheck one)
Sccdndarv subiectfs (please chock as many as applicable)
Anatomy E Physiology 0 Pathology P I3iochernistry 0 Pharmacology I7 Clinical assessment 0 SWFw a Medicine Q (difEerent aspects of one or more diseases) B iostatistics i Z Others-please state 2
Summary of description of product (eg. lhir program ir d&gned for nurses, to a s i s t in understanding the purpose mdmction of a nasogastric Nbe. the dtrwence m e e n nasoenfericmcd nasogasfrc tubes. the insertion and maintenance. and monitoring of the fube and providing pacare wery 4.8 and 24 havs.)
Target Population (checkas mrvry a applicable) as indicated bv authorls Pre medical 0 Medical 0 Clinical clerk 0 Resident 0
Nursing Allied health professionals
0 C1
bv reviewer 3
u
0 0 ! I
Ci
Other Setting by authods
Independent learning Ciass demonstration colIaborative or small group learning Other (please state)
C ..
i.I
by reviewer
I Technical requirements 1Type of Computer= PC 3
Mac D
Both U
Other
RAM required:
kilobytes/rnegabytes/gigabytes
Disc space required:
kilobytes/megabytes/gigabytcs
Monitor requirement:
EGA C;
VGA 3
SVGA 0
other 2
state type
System:
UNIX u"
Windows 3.1 Cl
yes 0
no 0
IimitedC
Is network version available? yes 0
no 0
Is there an update policy?
yes 0
no Z
Copyrighted:
Yes 0
no Z
Technical support available:
Yes 0
no 0
DOS El
OS/2 0
Network cornpatrile:
Mac 0
Please comment on ray other technicat tcquinmeats:
Basic Price (state currency):
Wmdows 95 S
(
Detailed Evaluation
1
USER FRENDLINESS This denotes the overail qud/int of the procrlirct ih terms of user &pug, ou@ut rutd +run errors (teclirnicoi). h i addition, i2 ilclirdcs the ducumu~titt&n mailtabfeboa on-liire and in prht, &at describes the producl, setup and use ( ~ o c u M M ~ P ~ ~ R ) .
Pleuse circle one of the numbers in the 5 poinr scaie that you think 13qppropriate, Vthe numberfafk b e e n rwo poim, cirde rk lowcr of the two. The ~ ~ S C T Q I Ofor ~ S the two extremes are giwt on either en&-
System error kquent
1 2 3 4 5
Documentationprovided inadequate for setup
no system enor
documentation very helphl for set up
1 2 3 4 5
Please circle yes or no.
Is the documentation clear ?
Yes
no
Does the documentation con- information on how to integrate into teaching ?
Yes
no
Does the documentation contain references ?
yes
no
Is the documentation adequate for navigating through program ?
Y e
no
Are there clear cues for input?
Ye
no
Overall score for User Friendliness Please rate the userfiiendiness in tenns of rhe overall qualie of the progrmn s ' technicPl arpects and doctlmentation using a scale of i to 5. Scottof l='lhtsynanarr&sottar~dis dificult LO navigate Uumgh, The information in he docwncn~-on b inadquact for setting up a~ weit u the Itarningprocess.
1
2
3
4
5
Scoreof5=Thcsyslcmdosna
