Institute of Research (LAIR) uses a Data General. Eclipse computer for its scientific work, so LAIR ... particular, Dr. William Baker of NIH and William. Sibley of ... New York: McGraw-. Hill, 1975. 4. Karpinski, R.H.S., and Bleich, H.L.: MISAR: A.
CLINFO
-
AN INDEPED1ENT USER'S EVALUATION
Major Thomas A. Marciniak, MC, USAR Tri-Service Medical Information System8-Army Walter Reed Army Medical Center Washington, DC
CLINFO is a clinical research data management and analysis computer system developed for the NIR General Clinical Research Centers (GCRC) program. The prototype version of CLINFO was implemented at three GCRC's in 1976 and 1977, and a production is now being extended to additional version GCRC's. These systems have been implemented as total packages, including dedicated hardware, software, and a full-time systems manager. In addition, a CLINFO terminal has been operational on a trial basis at Walter Reed Army Medical Center during 1980. This paper presents the results of the latter experience and describes an on-going effort to adapt the prototype version of CLINFO to a time-sharing environment.
of Research and the Institute of Surgical Research. These institutes and others contribute substantially to advances in such areas as tropical medicine, environmental medicine, medical evacuation, and burn therapy. At the care delivery level, all Army medical centers include an autonomous clinical investigation service to foster and coordinate clinical research regarding all types of medical problems. At all levels, clinical research is encouraged as a means of improving patient care and promoting professional growth. At Walter Reed Army Medical Center this comitmebt to research translated into 232 protocols supported during FY 1979.
The Tri-Service Medical Information System (TRIMIS) Program has the mission to "improve the effectiveness and economy of health care delivery administered by the Military Departments, through the application of standardized automatic data processing (ADP) techniques to health care information systems". 7 TRIMIS-Army, the Army organization responsible for coordinating the TRIMIS Program within the Army, has provided consultative services to Army clinical investigators with regard to selecting and using computer systems. In addition, TRIMIS-Army has maintained a continuous surveillance of new developments in automated support for clinical research.
Introduction Clinical research is an endeavor highly suited automation: Written protocols define in advance the data acquisition methodologies. Data collection is methodical, precise, and structured. Data analysis is exhaustive and frequently involves standard statistical techniques. Not surprisingly, some of the most popular computer software packages are the general purpose statistical programs with exotic names such as "SAS" ,1 '\BMDP", In and "SPSS" .3 addition, computer systems have been developed specifically for supporting clinical research. These systems also have exotic names, such as "MISAR",4 "PROPHET",5 and "SEARCH" .6 With such a menagerie of mnemonics to choose from, to which system should the clinical investigator entrust his priceless research data? to
In connection with this surveillance of new developments, TRIMIS-Army has conducted informal evaluations of computer systems applicable to clinical research. Three general criteria have been used in the evaluations: (1) Suitability. Obviously, a general ledger package would not be as suitable for clinical research as a statistical package. (2) Ease of use. If technical expertise and time are required to use a research system,
The Army Medical Department is committed to supporting clinical research. This commitment is shown by the its organizational structure: At the highest level, the Army Assistant Surgeon General for Research and Development oversees research of direct relevance to the soldier at dedicated installations such as Walter Reed Army Institute
then the value of that
is
diluted.
The
does not have enough programmers to support custom programming for even a small fraction of its clinical investigators. Usability by naive users is difficult to achieve. Even well-designed, mature systems such as SAS are more complex than desirable for direct use by physicians. (3) Availability. Regardless of theoretical desirability, a system is worthless if is it absolutely unavailable because of circumstances or relatively unavailable because of
The opinions and assertions contained herein are the private views of the author and are not to be construed as official or as reflecting the views of the Department of the Army or the Department of Defense.
U.S. Government work not protected by U.S. copyright.
system
Army Medical Department
cost.
1267
Initial Evaluation of CLINFO
and any investigators with studies ready for analysis within a three month period were invited to try using CLINFO. Investigators responding were given an additional one-hour training in system use and provided consultative services on an on-call basis. Experience with CLINFO was evaluated closely, particularly during the initial usage from February 4 to May 11, 1980.
One system noted during the TRIMIS-Army surveillance of clinical research systems was CLINFO.8 CLINFO is a clinical research data management and analysis system sponsored by the Division of Research Resources of the National Institutes of Health (NIH) for the General Clinical Research Center (GCRC) program. The the Rand prototype system was developed by Corporation with NIH funding and implemented at three GCRC's in 1976 and 1977. A production version is now being extended to additional GCRC's. CLINFO documentation was reviewed in depth, and on paper CLINFO was judged to be extremely compatible with the criteria discussed above:
This trial use of CLINFO at WRAMC differed significantly from the GCRC installations: CLINFO in the GCRC program has been implemented as a dedicated, in-house system, including a full-time systems manager. Using CLINFO in a teleprocessing mode presents several minor nuisances: Line errors and limitation of communication rate (1200 baud, rather than the 4800 baud preferred based on the early GCRC experience) were tolerable problems. Unavailability of some of the more advanced printing capabilities of CLINFO (better formatting of schemas and worksheets on a central line plotting rather than character printer; plotting by a special slave printer) was also found to be a tolerable deficit. More limiting was the lack of an experienced systems manager: Operations management was not required, but more in-house expertise with CLINFO use would have been helpful, particularly with regard to the data management side of CLINFO.
(1) Suitability. CLINFO was designed based upon an extensive survey of the information processing needs of GCRC investigators.9 The design evolved from that survey, and the system was continuously operationally tested in the GCRC environment. Although the GCRC program does not include military medical facilities, it has its counterpart in the Army. For example, the Kyle Metabolic Unit at Walter Reed Army Medical Center is a 16-bed research ward identical to the typical GCRC unit.
(2) Ease of use. CLINFO was "designed to be used by investigators and members of their staffs who have no previous experience with computers". 10 A screen-oriented, menu-selection type of manmachine interaction is used that is largely selfinstructional. Other design enhancements were introduced to promote usability. For example, to minimize special function keys are used keystrokes and facilitate use by poor typists.
The trial was conducted under several severe constraints: (1) Use of CLINFO was forbidden from 1100 to 2200 EST in order not to interfere with Rand use of their system. (2) All participating clinical investigators were thoroughly briefed about the uncertainty of system availability. The system was presented as being available for trial use from January to April, 1980, but that no guarantee was offered regarding preservation of entered data after April, 1980. Furthermore, if Rand file use became excessive or other problems were encountered, the trial could end prematurely. (3) A similar problem regarding uncertainty of continued operation of the CLINFO terminal was noted, since no maintenance contract or routine service was available for that terminal. The net effect of these constraints was the participation in the trial of a small set of investigators who were particularly interested in computerized
(3) Availability. The software for the prototype version of CLINFO is in the public domain. However, that version was not intended for general distribution and is not supported. A commercial version of CLINFO is available that is fully supported with regard to hardware and software maintenance. A CLINFO Trial
analysis. Although the evaluation summarized above is very favorable towards CLINFO, its value is limited by its theoretical nature--One must always be suspicious of paper analyses of computer systeim. Fortunately, through support provided by the NIH Division of Research Resources and the Rand Corporation, a trial use of CLINFO was conducted at Walter Reed Army Medical Center (WRAMC) during 1980. This trial enabled a more realistic evaluation of the capabilities of the CLINFO system.
Six investigators used CLINFO for seven studies, logging about 91 hours of terminal time in 62 sessions during the initial three months. Five of the seven studies produced usable analyses during the trial period. All five of these studies were handled by means of the data analysis side of CLINFO alone. The other two studies made u-se of the data management side (roughly comparable to a specialized data base management system with a "schema" and logical retrieval options). One of the studies was withdrawn from CLINFO for a combination of reasons: the initial data entered into CLINFO contained lab errors; sufficient time was not available to both meet a publishing deadline and to learn to use the system; and CLINFO did not support ideally the study design. The other study with data management is continuing on CLMFO.
Durin$ the trial a CLINFO display terminal, slave printer, and modem were installed in the conference room area of the Kyle Metabolic Unit at *RAMC. This terminal complex was used for dial-up access to the Rand Corporation CLINFO system in Santa Vdnica, California. The system was demonstrated to groups of clinical investigators, 1268
The response of the participants to CLINFO was enthusiastic. The participants were asked to rate their experience oA scale from 1 (highly unsatisfactory) to 5 (highly satisfactory)--The majority responded "highly satisfactory." (The other two responses were "satisfactory.") There was good agreement that CLINFO saved time and permitted more analyses to be done. Time savings estimates ranged from "10-15 hours" for a small study, to "120 hours" for a large one, with an average of about 50 hours. Opinions were neutral regarding any benefits of improved quality or quantity of research--Other factors were judged to be overriding.
question of whether the prototype version software, which is available in the public domain, is a viable option. The question of continuing usage of CLINFO within the Army was considered in three phases: First of all, could the prototype version software easily be converted to run on one of the computers available at WRAMC? The prototype version software consists of about 110 modules written in Data General Extended BASIC. BASIC is not a standardized language. The ANSI standard for minimal BASIC specifies only core capabilities. Most production BASIC's, including Extended BASIC, extend and enhance the core in various non-compatible ways. Extended BASIC was compared to other vendors' BASIC's and significant incompatibilities were found, particularly with regard to input/output processing, string handling, and error recovery. Furthermore, although user documentation for CLINFO is excellent, available is somewhat system documentation for CLINFO scanty. Hence an initial decision was made not to attempt to convert CLINFO to another vendor's version of BASIC.
Equally favorable was the publication rate. Six papers or abstracts have been accepted or Three submitted using analyses done with CLINFO. of the papers addressed different facets of the same study, a seventh paper is being finalized on an eighth paper is in and another study, preparation regarding a study with one other paper submitted. If the study mentioned above that was removed from CLINFO is excluded, only one study does not currently have a paper using CLINFO analyses in its final stages.
After the decision was made not to convert the software, the next question was "Is there hardware compatible with the prototype version of CLINFO available within the AMEDD?" The Letterman Army Institute of Research (LAIR) uses a Data General Eclipse computer for its scientific work, so LAIR was targeted as the initial site for bringing up CLINFO. However, hardware compatibility problems do not end with the processor: The prototype version of CLINFO uses a unique 80 character by 40 line display terminal with an integral printer port. The CLINFO software uses special control character sequences embedded in PRINT statements to accomplish screen insertions and graphs (character-plots) of research data. Conversion to the current standard 80 character by 24 line display is infeasible. Since the terminal model used for the CLINFO prototype sites is no longer a being manufactured, an alternative was found: terminal with a 80 programmable intelligent character by 48 line display will be converted to emulate the features needed for CLINFO. Due to delays encountered because of the complicated federal computer procurement process, the terminal hardware has not yet been acquired.
The results of the trial confirmed the initial paper evaluation of CLINFO with regard to the issues of suitability and ease of use. Suitability for clinical research is well demonstrated by the publication rate. Certain limitations of CLINFO were encountered: Participants' complaints included unavailability of some analyses and awkward use for some study designs. In addition, many WRAMC investigators desired computer support for research activities not suited for CLINFO: real-time data acquisition and reduction; and global, prospective surveillance (e.g., a data base on all patients seen in rheumatology). Ease of use of the data analysis side of CLINFO was outstanding: The typical investigator received one-hour of personal instruction and reported feeling familiar with the system after 3-4 hours of terminal time. Experience with the data management side of CLINFO was less certain: The data management side of CLINFO is more complicated than the data analysis side. This complexity of clinical research data mangement is not unique to CLINFO. According to other developers of similar retrieval systems, it is unavoidable. Because of time constraints and an incomplete familiarity with the system, the data manageratnt side was not fully tested.
The third question is "How portable is CLINFO?" The BASIC source modules include calls to assembly language routines to perform functions not available within BASIC. In addition, the vendor-supplied operating system and BASIC interpreter were mdified by the Rand Corporation to correct errors and to support special features of CLINFO. Since the LAIR system uses a different processor and a different operating system than the prototype sites and supports time-sharing of other applications, conversion of the assembly language routines and accommodation of the interpreter and operating system changes are being The final appraisal of the accomplished. portability of the prototype version of CLINFO will occur when CLINFO becomes operational at LAIR.
A more detailed analysis of the trial use of CLINFO at WRAMC is available in a separate
document.ll Prototype Version Evaluation The trial use of CLINFO at WRAMC suggests that CLINFO is a valuable system. However, the WRAMC trial did not definitively answer the question "How valuable?" How many of their scarce research dollars should clinical investigators contribute toward support of a system like CLINFO? Furthermore, the WRAMC trial did not address the related
1269
Conclusion In summary, a review of CLINFO documentation suggested that CLINFO could be of value to Army trial WRAMC The investigations. clinical The LAIR that initial impression. confirmed effort is addressing the question of whether the prototype version of CLINFO is a viable option. Regardless, we look forward to further experiences with CLIiFO and with systems like CLINFO in the Army Medical Department.
Acknowledgements We wish to thank the NIH Division of Research Resources and the Rand Corporation for providing In the support necessary to carry out the trial. particular, Dr. William Baker of NIH and William Sibley of the Rand Corporation were generous with their time and support.
References SAS User's Guide. 1. SAS Institute: SAS Institute, 1979.
Cary, NC:
2. Dixon, W.J., and Brown, M.B.: BMDP-77. Angeles: U. of Cal. Press, 1977.
Los
SPSS - Statistical Package 3. Hie, N.H., et al: New York: McGrawfor the Social Sciences. Hill, 1975.
4. Karpinski, R.H.S., and Bleich, H.L.: MISAR: A Minature Information Storage and Retrieval System. Comp Biomed Res 4: 655-660, 1971. The PROPHET System and Resource 5. Raub, W.F.: Sharing. Fed Proc 33: 2390-2392, 1974.
6. Buckley, J.D., et al.: SEARCH - A Language for Retrieval and Statistical Analysis of Medical Records. Comp Biomed Res 6: 235-244, 1973. 7. DODI 6000.5: Tri-Service Medical Information System (TRIMIS) Program. June 11, 1976. 8. Palley, N.A., et al.: CLINFO User's Guide: Release Three. Santa Monica: Rand, 1977.
A Survey of 9. Palley, N.A., and Groner, G.F.: Clinical Investigators and Their Information Santa Monica: Rand, Processing Activities. 1974.
10. Groner, G.F., et al.: A Plan for the Development and Evaluation of a Data Management and Analysis System for Clinical Investigators. Santa Monica: Rand, 1974. 11. Narciniak, T.A.: Report of a Trial Use of a Clinical Research Computer System (CLINFO) at Walter Reed Army Medical Center. Washington, DC: TRIMIS-Army, 1980.
1270
