Recording and evaluating end-user searches on a personal computer.

BRIEF COMMUNICATIONS Recording and evaluating end-user searches on a personal computer* By Marie A. Abate, Pharm.D. Associate Professor of Clinical Pharmacy Director, West Virginia Drug Information Center School of Pharmacy

James M. Shumway, Ph.D. Assistant Professor of Medicine and Pediatrics School of Medicine Arthur I. Jacknowitz, M.S., Pharm.D. Professor and Chairman, Department of Clinical Pharmacy School of Pharmacy

George Sinclair Data-Entry Operator School of Pharmacy West Virginia University Morgantown, West Virginia 26506

Although several studies have evaluated the utility of user-friendly, computerized medical information services by physicians, these studies generally have been conducted in a university or medical center environment [1-4]. However, limited information exists regarding the need for and use of computer information databases by other health care professionals or by private practice physicians who are employed in non-university-associated community settings. Therefore, a long-term study of the utility of two commercially available database services, BRS/Colleague® and DIALOG® Medical Connection, was undertaken. This study evaluated the drug information needs of clinical pharmacists, physicians in private practice settings, and their nurses. Each service was to be used and evaluated for a twelve-month period of time. In addition to attitude data, the authors obtained data on the questions or topics that were the focus of the searches, search strategies, databases selected, time spent searching, and success rates. Since the study was designed to examine the unsupervised use of these systems in practice settings, it was necessary to develop a method for automatically recording the searches and collecting other needed data in

a manner that minimized the participants' time com-

This research was supported in part by U.S. Public Health Service Grant No. R29 LM04658 from the National Library of Medicine. OBRS/Colleague is a registered trademark of BRS Information Technologies; DIALOG is a registered trademark of Dialog Information Services. *

Bull Med Libr Assoc 77(4) October 1989

mitment. Although not specifically a goal of this study, this method would also be valuable for educational purposes. This report describes the method developed to record automatically the computer information searches and the initial results.

PROJECT DESCRIPTION The communication software used for this project was PROCOMM VERSION 2.4.2, which was installed on the fixed disk of IBM PS/2 Model 30 or 50 computers at each practice site. PROCOMM was used because of its individual customization features [5]. A batch file (Figure 1) was prepared for each of the computer information services. By having the participant simply type in either "BRS" or "DIALOG," the batch file changed the directory to PROCOMM and called up a command file for the appropriate information service. The command file, prepared using PROCOMM, automatically connected the participants to either BRS or DIALOG, and permitted recording of the searches. An example of one of the command files ("BRS.cmd") is shown in Figure 2. In order to create the command files, the questions asked by the telecommunication services during the connection procedures (e.g., "Please log in"), the responses needed, and any additional required information were identified. The command files first opened a file called "capture.dat"; this file recorded the information shown on the computer screen from the point opened through logging-off. After opening "capture.dat," the command file displayed a message to enter the current time (e.g., 2:05 P.M.) and then checked whether the entered time was in the appropriate format. If not, it displayed an error message, "Invalid response-type in current time as shown in example." The time format was selected so that it contained a space (between the number and A.M. or P.M.) that the computer could look for. This was done to prevent participants from bypassing the time step simply by pressing the "Enter" key. The command file then prompted for first initial and last name (e.g., "G. Smith"), and checked for the proper format. It was again necessary for the computer to look for a space to prevent participants from bypassing this step. If the proper format was not found, the computer displayed the error message "Invalid response-type first initial, a space, then last name." The "dial" command dialed the telecommunication service Tymnet from the PROCOMM directory (Figure 2, entry number 3 in the dialing directory ) from which either BRS or DIALOG was accessed. The command file also could have included instructions to enter automatically participants' passwords for the information services. However, given the large num381

Brief communications Figure 1 Batch files BRS.Bat Echo off cd\procomm procomm / fbrs.cmd

Figure 2 BRS command file DIALOG.Bat Echo off cd\procomm procomm / fdialog.cmd

ber of participants (fifty) in this project and the number of passwords, it was decided to end the command file at the password step. After participants had completed their search, logged-off from the information service, and exited PROCOMM, the "capture.dat" file was closed and recorded on the fixed disk. Although the participants were shown how to manually exit PROCOMM (ALT-H to hang up and ALT-X to exit to DOS), these steps could also be incorporated at the end of the command file (with the commands "hangup" and "quit"), so that PROCOMM would be exited automatically after logging-off. Each time a participant typed in "BRS" or "DIALOG" and the command file was run, search data was added to the "capture.dat" file. Every few days, the file was renamed to the current date, e.g., "11-2-88.dat," copied to a floppy disk, and erased from the fixed disk to conserve space. Renaming the file to the date on which it was copied provided both a method of cataloging the files according to date and a means for later reviewing and analyzing a participant's search. After erasing the "capture.dat" file from the fixed disk, the system was ready to start another such file to record searches. The information in the "capture.dat" file was saved in "text form," meaning it could be viewed by any textediting or word-processing program. The captured search files were reviewed and the following information compiled for each search: the person performing the search, the time spent in each database, the total time spent per search session, the database(s) used, the library (libraries) used (for BRS), the number of search statements used, the exact search terms and other commands used, the number of documents found, and the number of documents displayed. RESULTS The capture file system generally worked very effectively. However, the "capture.dat" file was not recorded on the fixed disk if the computer was turned off before exiting PROCOMM. Since an occasional participant would turn off the computer without exiting PROCOMM, the command file has since been modified using the newest version of PROCOMM, PROCOMM PLUS; this allows for automatic exiting and recording of the "capture.dat" file immediately 382

log open "capture.dat" message" message" TIME: message "Please enter the current time (for example: 2:05 pm)" get s5 9 FIND S5" IF NOT FOUND MESSAGE "INVALID RESPONSE- TYPE IN CURRENT TIME AS SHOWN IN EXAMPLE" GOSUB TIME ELSE message" message s5 message" message" NAME: message "Enter your first initial and last name and then press enter." get s6 18 FIND S6" IF NOT FOUND MESSAGE "INVALID RESPONSE-TYPE IN FIRST INITIAL, A SPACE, THEN LAST NAME" GOSUB NAME ELSE message" message s6 dial "3" pause "3" if connected transmit "a" waitfor "please log in:" transmit "brs" transmit "^m" endif

at the conclusion of a search. In order to exit automatically, the computer needed to know exactly when to implement the "hangup" and "quit" commands, i.e., it had to know what phrase it should wait for just prior to exiting. Therefore, the "waitfor" command, followed by a phrase DIALOG or BRS prints out just after logging-off, was added to the command file prior to "hangup" and "quit." Thus, after loggingoff from DIALOG or BRS, the computer detects the appropriate phrase and then exits the command file and PROCOMM. Table 1 summarizes the pertinent data collected Bull Med Libr Assoc 77(4) October 1989

Brief communications Table 1 Summary of computer searches

Information service Data collected Number (%) who used services Number of search sessions Total number of times databases accessed Mean (±SD) [and median] total time (minutes) per session Mean (±SD) [and median] number of search statements Mean (+SD) [and median] number of documents found Mean (±SD) [and median] number of documents displayed Mean (+SD) [and median] time (minutes) spent in each database Number of questions for which information found partially or not at all

from the study of participants who used the computer during an initial 2½h-month training and practice period for BRS or DIALOG. During this period, small groups of participants attended a session of at least one hour in which instructions for use of the services were reviewed. Afterward, participants were free to use the databases as desired. User manuals for the services were also readily available. A computer log sheet asked each participant performing a search to indicate whether they completely, partially, or not at all found the information for which they were looking. For those searches in which participants did not find all of the information they needed, the investigators reviewed the captured search files to identify any errors that may have occurred. Table 2 summarizes the errors identified that most probably caused incomplete information retrieval during BRS or DIALOG use. Potential searching errors were identified for almost two thirds of the questions that were incompletely answered.

BRS

DIALOG

14(42%) 40 71 23.2 ± 19.9 [17] 3.1 ± 2.6 [2] 3.8 ± 9.9 [1] 2.3 ± 7.0 [0] 11.8 ± 12.4 [8]

9(53%) 45 56 12.5 ± 7.1 [11] 2.7 ± 2.8 [2] 15.5 ± 17.2 [9] 13.0 ± 12.7 [9] 8.1 ± 5.2 [7]

26

28

sonal computer. This method allows for the characterization and evaluation of searches performed in the absence of direct supervision, such as in private practice settings. Participants were informed that the searches were "captured" and that individual confidentiality would be maintained. In addition, the method can be employed during end-user searching training sessions or for student education. On several occasions, the study participants attributed unsuccessful searches to a lack of available information without realizing that searching errors might have occurred. Since the study was not specifically designed as an education program, no formal evaluation of the searches with individuals was undertaken. However, as part of an educational program, evaluation of search strategies by trained information specialists and review of the unsuccessful searches with trainees or students should lead to more productive future information retrieval and enhanced system use.

REFERENCES SUMMARY A simple, user-transparent method was developed for recording computer information searches on a per-

Table 2 Errors made in searches Number

Type Used inappropriate search terms (e.g., a phrase, not all appropriate synonyms) Did not search all appropriate databases Could not use "zoom" command in BRS full-text databases Misspelled search term(s) Could not use "expand" command in BRS Unable to display document(s) Used inappropriate connector Used brand name rather than generic name for a drug Total

Bull Med Libr Assoc 77(4) October 1989

of times 12 8 4 3 2 2 2

1. GoTTINGER HW. Computers in medical care: a review. Meth Inform Med 1984 Apr;23(2):63-74. 2. LUDWIG L, MixrER JK, EMANUELE MA. User attitudes toward end-user literature searching. Bull Med Libr Assoc 1988 Jan;76(1):7-13. 3. HAYNES RB, McKIBBON A, WALKER CJ, MoussEAu J, et al. Computer searching of the medical literature: an evaluation of MEDLINE searching systems. Ann Intern Med 1985 Nov;103(5):812-6. 4. COLLEN MF, FLAGLE CD. Full-text medical literature retrieval by computer. JAMA 1985 Nov 15;254(19):2768-74. 5. CLANcy S. How to download and use PROCOMM-communications software with power, performance, and price. Online 1988 Jan;12(1):97-114.

Received December 1988; accepted April 1989

1 34

383