Evaluation of Network Based Learning
Proceedings JENC8
A. Bittorf
Evaluation of a Network Based Learning Environment for Dermatology Andreas Bittorf Thomas L. Diepgen
Abstract The importance of computer networks in a non technical environment is steadily increasing. Especially medicine is becoming a major user with its high demand in current information. We show here, how dermatology education can win by new network based applications. Because visual information is very important in dermatology, we have done in depth examinations for image quality considering compression, resolution, colour display, and transfer times. The basis layer for all our teaching resources is an image database, that is used to offer dynamic lecture scripts, case reports, an atlas and a quiz system. For evaluation purposes a generic user survey was used. The result of our evaluation and access statistics are very encouraging.
I. Introduction Dermatology is a visually oriented discipline because diagnosis and differential diagnosis is mainly based on the recognition of the clinical and histopathological features. Therefore, images of diseases are extraordinary important in education and patient care. Conventional media like books or journals offer only limited support as they are restricted in size, expensive, often hard to get, static, only allow one dimensional searches, and do not provide acceptable linking mechanisms. Due to this an extensive image database has been developed starting in 1994 and is available in an online and offline version since the middle of 1994. To provide high quality images we have done in depth examinations for image quality considering compression, resolution, colour display, and transfer times. The database is the basis layer for all our teaching applications. We offer dynamic lecture scripts, case reports, an atlas, and a quiz system through the WWW. A user survey was performed and the access statistics were analysed.
A resident picked more than 2,000 35mm slides from the 40,000 preselected of our slide archive. Some images were also submitted by external physicians. The slides were digitised by a commercial service provider and stored on Kodak Photo Compact Discs (PCD). The images were enhanced in Photoshop or Photostyler and JPEGcompressed with a quality factor of 60 (= 1:37) with the Ulead-implementation available in Photostyler 1.1a. After image enhancement, anonymisation, and data entry a senior dermatologist controlled the quality of the selection and the appropriateness of the description. For a correct display of the colours we provide an image for gamma adjustment to the settings we used and also have implemented a server based correction using the PNM-tools. Documents were authored utilising HTML Assistant, Microsoft Internet Assistant, simple text editors, and Visual Basic. We used HTML 2.0 and a few extensions proposed for HTML 3.0, like tables. As a frontend, we used Netscape Navigator and NCSA Mosaic.
II. A. Local Database For local use a relational database [1] was designed carefully, which can be accessed by a Visual Basic frontend. As Content Based Retrieval (CBR) [2] is in the very beginning we had to develop key systems for the standardised description of the images. We decided to use the following categories: Localisation, localisation modifiers (proximal, dorsal, etc.), relevant attributes of the localisation (exposed to sun, seborrhoic, etc.), expanded description of lesions, attributes of the lesion (different categories, e.g. form, colour, relationship), histopathological attributes, and expanded ICD-9 compatible diagnosis key. All keys were constructed hierarchically which permits to chose the required degree of accuracy, e.g. a search for face may result in all images showing exactly the face, parts of the face (e.g. ear, eyelid), supersets of the face (whole body, head) depending on the mode of search.
II. Methods
II. B. Dermatology atlas
As WWW-server we used a Sparc 20 workstation running Solaris and the Netscape Commerce Server. All programs except the database (DB) demon (=daemon; "Disk And Execution MONitor") were performed using the common gateway interface (CGI) of the WWW-server and were written in the Practical Extraction and Report Language (PERL).
Using the database and key systems three different methods to access the images of the Dermatology Online Atlas (DOIA) are offered: a) query interface, b) static access structure, c) HARVEST. The query interface was implemented with a slightly modified Generic Structured Query Language (GSQL) that provides the WWW-user
Evaluation of Network Based Learning
Proceedings JENC8
interface and the mapping to SQL. A table based query form is offered where the user can define restrictions and configure how results should be displayed. The database queries are performed connecting to a database demon process that uses an INFORMIX database that contains a subset of the information stored in the local database. 20.000 15.000 10.000
KNAW Library, Netherlands VOL, Italien Mie U, Japan Erlangen, Intern Erlangen, Extern
5.000 0 09/94 01/95 0 5/95 09/95 0 1/96 0 5/96 09/96 Figure 1: Access numbers per day. A static access structure interconnected by hyperlinks based on the diagnosis (alphabet, ICD) was generated from the local database using Visual Basic scripts. Mirrors of the documents and images are stored on WWW-servers at other universities, e.g. Mie University, Japan and UC Davis, USA. Further, full text retrieval was supported using HARVEST [3,4] (with GLIMPSE as search subsystem) on the basis of the static access structure and additional META-tags, which were included for structured querying.
II. C. Dynamic lecture scripts and case reports For our students, we have developed lecture scripts which summarise the most important facts given in the lecture. They contain text and all slides shown in the lecture. In a similar way physicians report about rare and interesting diseases and effects of therapy in so called "case reports". Special case reports so called "Grand Rounds" were audio recorded with a digital audio tape (DAT) and are offered through the web as a slide shows with sound. There is an automatic mode, where the browser loads automatically the next slides using the refresh tag, when the spoken text is finished for the corresponding slides. For WWW-usage of the audio files we used Toolvox with a compression factor of about 300:1. In the lecture scripts sometimes one static link to an illustrative image and another link to all related images (database query) were included. Such a query could be »select * from image where diagnosis_text like '%eczema%' and age < 4« for all eczema in infants. This query could be coded like:
click for more images.
A. Bittorf
II. D. Quiz system The quiz system [5] was implemented using PERL, PERL "databases" that can store associative arrays, and text files (due to the limitation in the length of records in PERL). Two kinds of users exist, students who use a quiz, and teachers who ask questions and construct a quiz. A user may be a student and a teacher at the same time. Every user has to have one account that specifies his rights and a password. For every user and network address a configuration exists that contains colour correction information for computer and monitor. A user may be author of none, one, or multiple questions, but there is one and only one author responsible for a question, whereas a quiz may have multiple authors. A quiz consists of at least one question. For each student his previous answers on questions in a given quiz are stored. Questions may be related to none, one, or more topics. The topics help with the selection of questions when constructing a quiz. Each question may use none, one, or more images, and images may be used in no, one, or more questions. For all questions at least two answers and accompanying explanations exist. In addition a general explanation for the question is given. In a written examination of our students there are four different kinds of questions all using single-choice answers. The quiz system can be used and maintained completely with a Web browser. Three different modes of presentation may be chosen for a quiz: a) single question, b) all questions, c) random. Answers are given in a single choice mode by selecting a radio button. The answers of the user are evaluated and scored. A general explanation to the question is given provided that the answer was right. Additionally, specific explanations are given for wrong answers. Right and wrong answers are recorded. The next time the student starts a quiz he has already been working on, he has to answer all questions that he did not yet answer right. Once a quiz is completed the student may reset his prior score and try the quiz again.
II. E. Evaluation The whole CBT-system is offered through the WWW. We evaluated the system by conducting an open user survey. The link to the adaptive questionnaire was shown from 10th May till 10th July 1996 on our homepage and the starting page of the image atlas DOIA. Further, we analysed the access statistics using getstats 1.2.
III. Results III. A. Access statistics
Evaluation of Network Based Learning
Proceedings JENC8
To date the WWW-server containing the dermatology resources is reachable without restrictions. The access numbers have drastically increased during the last two years (Figure 1). In October 1996, we received over 15 000 hits per day. DOIA got by far the most hits (77%), followed by case reports, lecture scripts, and finally the quiz system. About 10% of the requests were originated by users of the big commercial online providers (e.g., T-Online, CompuServe, AOL, Inktomi) compared to about 5% half a year ago. We received roughly 25% of the hits from Germany, 25% from other countries in Europe, 25% from USA, and 15% from other North American countries.
III. B. Survey 199 questionnaires were completed correctly, that is 3 per day in average. The pages pointing to the questionnaire were downloaded about 18 700 times during that time. The most important results were: 〈
The country statistics of the questionnaire corresponds very well with the domain distribution of the access statistics.
〈
About 50% of our users are physicians and 25% work in a healthcare related area.
〈
DOIA is used most enthusiastically (Figure 3).
〈
The CBT modules are mainly used for the intended purposes (Figure 4).
〈
〈 〈
The rating of the server ranged from "good" to "excellent", only 2% used "sufficient" or "not sufficient". DOIA received the best rating followed by the lecture scripts, case reports, and the quiz system. Also the image quality was judged "very good" in average.
IV. Discussion One of the first applications for dermatology [6] on CD-ROM has been published four years ago. It is already taking advantage of the different access paths possible in a computer, but does not provide an extensive image collection and sufficient display quality. A comprehensive overview of computer applications in dermatology can be found in [7]. During the last few years a lot of CD-ROM products for Dermatology have been developed using images. All these applications are more or less stand-alone applications and do not use the power of global cooperation and continuous updating through the Internet. Most WWW-based applications for Dermatology have been developed starting in 1995/1996. Some of the most substantial are: 〈
Searchengine 41% 50% 45% 40% 35%
20%
Africa Asia
20% 70%
America - USA
60% 10%
USA
50%
Europe - Ger.
0% 40% < 9,6 K > 10 K = 64 K 20 K 10 M 30%
Germany
20%
Figure 5: Connection speed distribution
Lectures
Quizsystem
Figure 3: What was used
Other
Figure 2: How our WWW-site was found
5% 0% Continuing Education
Reference Teaching
Own disease
Friends' disease
Other
Figure 4: Reason for using our server 〈
University of California at Davis , including the Dermatology Online Journal (DOJ)
〈
University of Iowa , including the DermPathTutor Dictionary & Images
〈
Mie University , including the proceedings of the 1st and 2nd Internet World Congress in Biomedical Sciences
〈
Internet Dermatology Society , including the electronic text book for Dermatology
〈
Vanderbilt University, including the Contact Dermatitis Home Page
0% Cases
Directory 7%
10%
10%
DOIA
Friend 5%
15%
Australia 30%
Journal 17%
30% 25%
At the time of the survey most users were using a 28.8 kbaud modem. ISDN is only popular in Germany (Figure 5). Country
American Academy of Dermatology , including patient oriented information and professional information
Link 30%
About 75% of our users learnt about our offer online (Figure 2).
40%
A. Bittorf
Evaluation of Network Based Learning
Proceedings JENC8
〈
Cutaneous Drug Reaction Database
In other medical fields there are also a lot of applications emerging, just one example: 〈
Virtual Hospital at the University of Iowa [8,9] And two indices:
〈
Martindale's Health Science Guide
〈
Jonathan Tward's Multimedia Medical Reference Library
A. Bittorf
distributed image database, for example image colour settings, quality settings, and technical know-how in other departments of dermatology. Further, the problem with a common and multilingual description system for querying has not yet been solved sufficiently. In a simple way, search engines can be seen as an unfocused query interface to all dermatologic data [11], but they are not able to deal with the medicine specific problems. We also offer lecture scripts and case reports on the WWW that use images from our database. When viewing these images the user can easily navigate to similar images in the database for comparison. Lecture scripts can contain database queries as a link thus making it dynamic. With this mechanism the complete collection of relevant images in the database is visible upon mouse-click.
IV. A. Dermatology Atlas DOIA
IV. B. Image quality
Most online applications in medicine still rely on static HTML-documents and are maintained manually. The generation of documents from a database we used, offers big advantages, especially considering maintenance:
The high teaching quality of the images was ensured by the strict selection and the quality control in the second step. We decided to use photographic slides as basis because they offer the best dynamic range of any media and a very high resolution, that is about 6000x4000 points for a 35mm slide. Digitising slides has proven to be the best way to acquire high quality images. It is clearly superior to video plus framegrabber and also to medium prized digital cameras. A high photographic quality was achieved by the choosing the PCD.
〈
Consistency
〈
Improvement or change of diagnosis key
〈
Change of presentation (e.g., features)
〈
Change of server structure
〈
Inclusion of META tags
〈
Generation of a CD-ROM version
〈
etc.
new HTML
A complex structure like DOIA cannot be maintained sufficiently without using a database. It has proven to be useful to offer a hierarchical access structure to our images using the diagnosis, because no CGI-programs are necessary, and server load is reduced. Further, all data can be indexed by any search engine and used for distribution mechanisms like HARVEST. It allows the novice user to get an impression of what kinds of images are available and how to ask queries just by following hyperlinks. Implicit or explicit relationships from the database can be mapped to links thus offering meaningful means for navigation. Both implementations for querying in DOIA, the database integration and the HARVEST [3,10] based search mechanism, have advantages. The database interface is very fast and supports structured querying. Further, the display of the result can be configured in a very flexible way and server load is moderate. HARVEST is not as fast and the situation gets worse with the number of records to be queried. On the other hand it offers functionality for text retrieval and is a single and flexible query interface to heterogeneous distributed data [11]. Unfortunately there are quite a lot of problems associated with a
As resolution 768x512 dots using 16.8 million colours has been used because studies have shown that the digital image in this resolution on the computer screen is comparable to slides with respect to diagnostic accuracy (e.g., [12,13]). The compression factor of 1:35 was chosen on the basis of a study [13] and seemed to work well, considering the rating of the image quality during the survey. The resulting image size of about 30KB permits fast transfer even over international computer networks. Gamma correction seems to be the essential part of the colour adjustment. Further, the whitepoint is very important, because photographic images are used in Dermatology. The chromacities (display red, green, and blue) can be ignored without problems in our application domain. For images of the skin a low colour temperature has proven to be useful, for example 6500 K.
IV. C. Dynamic lecture scripts and case reports A lot of universities have started to set up teaching material for their students. However, we have used dynamic scripts which link to an underlying database. In our application this is very helpful, because the complete collection of images available in the database can be shown upon mouse click. Thus, a nice side effect of adding images to the atlas is that the scripts may become more complete. By following the links provided with the images, a
Evaluation of Network Based Learning
Proceedings JENC8
student may find interesting case reports. Usually case reports are used for the continuing medical education, however, interested students may use this resource as well, for example, to view exceptional images.
IV. D. Quiz system The great advantage of the PERL solution compared to a relational database is the easy portability as PERL is available on nearly every Web-server. However, the PERL solution has some drawbacks. Database integrity control is absent, there are no mechanisms for recovery, the data structures cannot be changed easily, and there are no mechanisms provided to deal with big collections of questions, that is the performance decreases proportional to the number of questions. Our quiz system can be maintained with a Web browser without writing any line of Hypertext Markup Language (HTML). We consider this to be important because we do not agree with the statement that due to the simplicity of HTML, teachers do not need any special skills [14]. This may be true in a technical environment, but it is definitely not valid for physicians. The single question mode is comparable to an oral examination. This mode is recommended only if a fast network connection is available to avoid dissatisfaction by long transfer times. Comparable to a written examination is the all questions mode, where the student can select the questions he wants to answer first and no direct feedback is provided before the end of the examination. We also recommend this mode with a slow connection as a student may do other work, for example in a text processor, while the whole quiz is downloaded. The random mode is intended for a fast repetition of the lecture few days before the examination. By keeping track of the questions answered right and wrong, difficult questions may be asked again. Within the WWW it turned out that an artificial user name is not acceptable. It is hard to remember the user id the next time. Due to this we decided to use given name and last name together with a password.
IV. E. Evaluation The mirrors we offer allow users to access our data at a much accelerated speed, i.e. in the USA and Japan. However, the mirrors receive only about 10% of the hits. In our opinion this is due to the current structure of the Web. Search engines and links, which are responsible in about 75% for finding our webserver, point mainly to the master site. Mirrors cannot be detected automatically. Perhaps an improved domain name service (DNS) that maps names to IP-addresses depending on the location of the client will provide a better solution for this in the future. In radiology a system has been implemented [15] with a somehow similar approach,
A. Bittorf
that mirrors only the images and generates the HTML documents from a central server. The documents contain the images as links to the closest mirror depending on the domain of the user. Another way to provide fast access across slow connections is to use caching. Unfortunately caches are currently organised by organisational units and not depending on the content the users are interested in. That works well for the most demanding users, that are usually interested in technical topics, but not for users e.g. in medicine. So we propose to set up a national infrastructure of content dependent caches, e.g. a cache for dermatologists and allergologists. This should work, because the network infrastructure within a research community, e.g. the "Breitband Wissenschaftsnetz" provided by the German Research Net Association ("DFN-Verein"), is usually fast. The acceptance of our WWW-server is surprisingly high. The image atlas part is by far the most popular, which shows the importance of images for education and reference. Images are usually downloaded within 5-10 seconds, which is acceptable for most users. However, to enable a truly interactive application at least a 128 kbaud connection is required.
V. Conclusion An online resource for medicine professionals and students was successfully implemented. Acceptance is very high, because the service offers an added value for our users compared to conventional media like books and journals. High quality images can be used with acceptable transfer times. Future work will include the expansion of the database, improvement of the description systems, client side colour correction using a JAVA applet, integration of other resources, setup of a patient information system, and improvement of user interface and access times.
VI. Acknowledgements We want to thank the German Research Net Association (DFN-Verein)/German Research Ministry (BMBF) for financial support (grant no. TK 598 - VA/I3) and Dr. N. C. Krejci-Papa, Prof. Dr. M. Simon as well as all other physicians that worked very hard on the content of the CBTmodules.
VII. References [1] Bittorf A, Krejci-Papa NC, and Diepgen TL. Development of a dermatological image atlas with worldwide access for the continuing education of physicians. Journal of Telemedicine and Telecare. 1. 1. pp.45-53. 1995;
Evaluation of Network Based Learning
Proceedings JENC8
[2] Köthe U. Content Based Retrieval (CBR). Computer Graphik Topics. 6. pp.17-18. 1993 [3] Bowman CM, Danzig PB, Hardy DR, Manber U, and Schwartz MF. The Harvest Information Discovery and Access System. 1994 Oct; Chicago, Illinois. 1994; p. 763; Proceedings of the Second International World Wide Web Conference. [4] Bowman CM, Danzig PB, Hardy DR, Manber U, and Schwartz MF. The Harvest Information Discovery and Access System. Computer Networks and ISDN Systems. 28. pp.119-125. 1995; [5] Herrmann S. Dermatologisches Quiz, Farbdarstellung und Benutzerkonfiguration im Word Wide Web -Verteilte Dermatologische Bilddatenbank. Universitätsklinik Erlangen, Lehrstuhl für Betriebsysteme IMMD IV und Dermatologische Universitätsklinik. 1995 [6] General Dermatology, CD-Derma Series 1.[computer program]. Dockx PFE. Antwerp: Lasion; 1992; CD-Rom. PC. MS-Windows. [7] Stoecker WV. Computer applications in dermatology. New York, Tokyo. IGAKUSHOIN Medical Publishers, Inc. 1993 [8] D`Alessandro MP, Galvin JR, Erkonen WE et al. The Instructional Effectiveness of a Radiology Multimedia Textbook (Hyperlung) versus a Standard Lecture. Invest. Radiol. 28. pp.643-648. 1993; [9] D`Alessandro MP, Galvin JR, Santer DM, and Erkonen WE. Hand-held Digital Books in Radiology for Convenient Access to Information. AJR. 164. pp.485-488. 1995; [10] Bowman CM, Danzig PB, Hardy DR, Manber U, and Schwartz MF. Harvest. 1996; [11] Bittorf A, Krejci-Papa NC, and Diepgen TL. Building a unified global access structure to distributed dermatologic images. Mie University; 1995 Dec 05; 1995; 2nd Internet World Congress on Biomedical Science '95 [12] Perednia DA, Gaines JA, and Butruille TW. Comparison of the Clinical Informativeness of Photographs and Digital Imaging Media With Multiple-Choice Receiver Operating Characteristic Analysis. Archives of Dermatology. 131. pp.292-297. 1995
A. Bittorf
[13] Bittorf A, Fartasch M, Diepgen TL. Auflösung und Kompression digitaler Bilder Anforderungen an die Bildqualität in der Dermatologie, Abstract Band der 41. Jahrestagung der GMDS, p. 124; 1996; [14] Lai MC, Chen BH, and Yuan SM. Toward a New Educational Environment. O'Reilly & Associates, Inc. 1995 Dec 11; p. 221; 1995 [15] McEnery MD, Roth SM, Walkup MD, and Walkup RV. Radiology CME on the Web Using Secure Document
Author Information Andreas Bittorf, born 1969, received a first class honours degree in computer science from the University of Erlangen in 1993. From 1993 till 1997 he worked in the Department of Dermatology, University of Erlangen, where got a Ph.D. degree for his work on digital images in Dermatology. His Dermatology Online Atlas (DOIA) won 1995 the medical software award of the MEDICA. He gave tutorials on "Web image databases" at major conferences. His research interests include digital images, databases, the World Wide Web, and medical informatics. He is member of the GI and the GMDS and technical editor of the Dermatology Online Journal. In spring 1997 he joined SerCon an IBM daughter. Thomas Ludwig Diepgen studied economics from 1974-1977 at the Universities of Regensburg and Munich, and medicine from 1977-1983 at the Universities of Regensburg and Erlangen. He worked as a research fellow at the Institut of Medical Statistics and Informatics from 1981 till 1986. In 1984 he completed his doctoral thesis dealing with a statistical analysis of genetic encyme polymorphisms. Since 1986 he is working in the department of Dermatology of the University of Erlangen. 1990 he received his board certification for dermatology, 1991 for allergology and 1996 for environmental medicine. 1991 he completed his professorial thesis on a biometric analysis of atopic diathesis. Since 1991 he is senior lecturer for dermatology and assistant professor at the Department of Dermatology, University of Erlangen. He is vice chairman of the department and heading the unit of dermatoepidemiology and medical informatics. Thomas L. Diepgen is president of the German Society of Occupational and Environmental Dermatology and Secretary of the European Contact Dermatitis Society. He is author of over ninety original papers and forty book chapters.
