This study aimed to test the usefulness and reliability of text-to-algorithm conversion ... Israel (NG, CZM); and Harvard Pilgrim Health Care, Brookline,.
Text-to-algorithm Conversion to Facilitate Comparison n of Competing Clinical Guidelines NURIT BARAK, CARMI Z. MARGOLIS, MD, MA, LAWRENCE K. GOTTLIEB, MD, MPP This study aimed to test the usefulness and reliability of text-to-algorithm conversion in comparing competing clinical guidelines and defining their differences. Two pairs of competing guidelines for measles immunization, published in 1989 and 1994, were analyzed and compared. Five categories of differences were detected: differences in recommendations, excluded elements, logical inconsistencies, nonspecific phrases, and approaches to contraindications. On a scale of O-10 (where identical = 10). the overall comparison scores were 6.01 for the guidelines published in 1989 and 5.54 for the guidelines published in 1994. Text-to-algorithm conversions performed by three different persons on the 1989 guidelines were compared and found similar. Text-toalgorithm conversion is an important step in facilitating comparison of competing guidelines. It has the potential to assist in making rational and systematic choices between competing guidelines before actual field testing takes place. Physicians can use it to analyze and to learn a prose clinical guideline, to critique existing guidelines, and to simulate hypothetical patients for teaching and evaluating clinical management. Key words: medical decision making; clinical guidelines; clinical algorithms; health services evaluation; semantic analysis; measles immunization. (Med Decis Making 1998;18: 304-310)
Recent emphasis on the development and deployment of medical guidelines has led to the introduction of multiple practice guidelines addressing the same clinical problem. Competing guidelines reintroduce some of the uncertainty and variability that guidelines are supposed to reduce. Medical guidelines can be unreliable in the sense that both: 1) two groups of experts may come up with different guidelines for managing the same problem, and 2) different experts may interpret the same guidelines differently when applying them to the management of a particular patient. Methods for comparing competing guidelines may help determine any potential advantages of one over the other by pinpointing the differences between them. A central problem in developing such methods is that most guidelines are written in prose, which is not well suited to describ-
ing the complex interrelationships between decisions that are the basis for, or the result of, other decisions. Prose guidelines, although more accommodating to many clinicians, are generally somewhat vague and ambiguous. The attribute of prose that is attractive to readers of literature, i.e., that a reader invests his own personality, experience, and knowledge to create a different story each time a text is read, becomes a liability when reading clinical prose guidelines that are expected to convey similar recommendations to every clinician using them. One way of reducing the vagueness and ambiguity of guidelines to a minimum is to convert the logical flows of their interdependent decisions, what might be called their “clinical-algorithm logic engine,” to a flow-chart format. Such flow-chart guidelines, more precisely called “algorithmic map guidelines” or “clinical algorithms” seem clearer and more accurate than prose for describing clinical logic and the connections between decisions. In fact, almost all clinical guidelines are driven logically by clinical algorithms, in that they present stepwise approaches to care using conditional logic. While the need for clear, concise, rigorous methods for describing the approach to a clinical problem has motivated the use of clinical algorithms in clinical education, the need for standardization of care and for methods of evaluation and comparison of clinical management strategies supports their use
Received December 29, 1995, from the Center for Medical Decision Making, Ben-Gurion University of the Negev, Beer-Sheva, Israel (NG, CZM); and Harvard Pilgrim Health Care, Brookline, Massachusetts (LKG). Revision accepted for publication October 22, 1997. Supported by the Institute of Medicine of the National Academy of Sciences, Office of the Forum for Quality and Effectiveness in Health Care, Agency for Health Care Policy and Research, and by the Harvard Community Health Plan Foundation. Address correspondence and reprint requests to Dr. Margolis: Faculty of Health Sciences, Ben-Gurion University of the Negev, P.O. Box 653, Beer-Sheva 85105, Israel.
304
VOL 18/NO 3, JUL-SEP 1998
Text-to-algorithm Conversion for Guideline Comparison
305
recent sets of guidelines for measles immunization are the AAP guidelines that appeared in the 1994 Red Book 3 and the 1994 MMR immunization guidelines of the Israel Ministry of Heath (IMH).4 These sets of guidelines for measles immunization, although not created with the same provider audiences in mind, were thought of as competing guidelines only for the purpose of this study, which is to describe a method of guideline comparison. The four sets of guidelines were chosen as examples for the following reasons: they address the same pressing problem; considerable resources were invested in their development; and, while it is apparent that they differ, it is unclear how significant these differences are. Text-to-algorithm conversion consists of converting practice guidelines written in prose or other non-flow-chart formats into algorithm maps, using a standard flow-chart algorithm format suggested by the Society for Medical Decision Making Committee on Standardization of Clinical Algorithms.’ Key steps in the conversion process include:
as health-care tools. Clinical algorithms can be an-
alyzed more accurately and competing versions can be compared using semantic analysis techniques. The aims of this study were 1) to perform and evaluate a systematic conversion of competing prose descriptions of clinical management strategies to algorithm maps and 2) to use the constructed algorithm maps to determine the differences between the various sets of guidelines.
Methods TEXT-TO-ALGORITHM CONVERSION Two versions of recommendations for measles immunization were published in 1989, one by the Committee on Infectious Diseases of the American Academy of Pediatrics (AAP)1 and the other by the Advisory Committee on Immunization Practices (ACIP) of the Center for Disease Control.’ Two more
FIGURE 1.
l
Excerpt from the Measles Immunization Algorithm Map Guidelines.
+
1
1. Immune giobuline (IG) (4 2. Vaccine a few months later
no
I
8 Student or sibling born after 1957
(B) who had 2
Middle school
shots, 1 month
apart, after ;ige 12 mos?
yes Annotations: A. Esplain use of IG. B. Persons born before 1957 are considered to have hiltI mc;lsles. C. Explain why MMR is preferred over monovalent measles. Monovalent measles may be substituted for MMR if cost is a factor.
6 no
Immunize at le;W 1 dose MMR
(C)
,
yes IO
/
I
Medical facility employee, burn after 1957, with lWient contact
I
I
Table 1
I
l
Part of the Clinical-rule Analysis, AAP Measles Immunization Algorithm Mad If
I, Documentetl measles or +Ab titer or 2 shots, 1 mo apart?
9 Do not immunize
yes
no ‘I 12 Immunize or
Then
1
Measles outbreak + exposed to measles + cl year of age
Administer immune globulin Vaccinate a few months later
2
Same as 1 above - cl year of age + exposed ~72 hours
Immunize with at least 1 dose of MMR
3
Same as 2 above
..
Comments
give 2nd MMR - exposed ~72 hours
Unclearnot specified
306
l
Barak, Margolis, Gottlieb
Studying the guidelines carefully; aiming to get an overall picture as well as to understand content details (this might include translating the prose guidelines into English) Identifying any algorithmic (conditional “if . . . then” statement) sections of the guidelines Placing problem definitions; decision steps, and action steps in their appropriate boxes Sequencing the diagnostic and action boxes in a clinically rational order. The four prose guidelines for measles immunization were converted into algorithm maps, using text-to-algorithm conversion. A part of one algorithm map is shown in figure 1. The text on which this algorithm map is based, excerpted from the original prose guidelines, appears in the appendix. CLINICAL-ALGORITHM PATIENT ANALYSIS Once the algorithm maps were constructed, they were compared using the clinical-algorithm patient abstraction procedure described by Pearson et al.” This consists of three steps. First, clinical rule analysis reduces the algorithms to lists of “if . then” conditional statements. Every branch of even the most complicated algorithm can be reduced to these conditional logic pieces, as shown in table 1 for the beginning of the map in figure 1. Hypothetical patient (fictional person that fits the qualifications of a particular algorithm map pathway) abstraction was the second step. Each algorithm map was broken down into separate management pathways, each of which begins at the first box of the algorithm and continues until a particular terminal “action” box is reached. For example, in figure 1, the pathway created by following boxes 1, 2, 3, 5, and 6 might be considered to represent the hypothetical case of a 15-month-old child (box 3, “
