Computerized Clinical Decision Support Systems in Family Medicine: proposed principles. Authors: Carl Steylaerts1, MD, Jef Goris2, MD, Jean Karl Soler3, MD, Didier Duhot4, MD, Teng Liaw, MD5, 5
PhD, Ilkka Kunnamo6, MD, PhD, Emmanuel Bottieau MD7, Jef Van den Ende, MD, PhD7,8 1 General Practitioner, Belgian Society of General Practioners/Family Physicians, Belgium 2 General Practitioner, Domus Medica, Belgium 3 General Practitioner, Malta College of Family Doctors, Malta 4 General Practitioner, Société Francaise de Médecine Générale, France
10
5 General Practitioner, FRACGP, FACHI, School of Rural Health, The University of Melbourne, Australia 6 General Practitioner, The Finnish Medical Society Duodecim, Helsinki, Finland 7 Institute of Tropical Medicine, Antwerp, Belgium 8 University Hospital, Antwerp, Belgium.
15
Version 3.6 28 February 2007
Word count: 3247 Key words: decision support systems; family medicine; Bayesian; electronic patient records; 20
transition project; Isabel; Kabisa; Dépican.
Corresponding author Carl Steylaerts ,
[email protected]
Abstract (234) 25
Introduction This paper investigates the latest trends in the literature about computerized clinical decision support systems (CCDSS) in primary care and family medicine and identifies some working examples. It outlines some important principles to consider. Material and methods
30
An extensive literature search, a focus group discussion of GP experts who gathered in Prato and a 7months e-mail discussion with experts from outside the GP field were used as method to outline the principles. Results Seven 7 major principles were identified: 1) the overall aim of CCDSS is to decrease uncertainty;
35
2) CCDSS should support clinical reasoning that is based on iterative testing of diagnostic hypotheses and weighing them probabilistically against thresholds; 3) evidence-based guidelines should underpin CCDSS support; 4) highly structured electronic patient records featuring episodes of care are required; 5) support for diagnostic and therapeutic decisions should be generated from these structured electronic patient
40
records; 6) CCDSS should also support chronic disease management, providing guidelines and up to date information; 7) socio technical issues associated with workflow should be addressed, e.g., CCDSS should avoid excessive or irrelevant reminders and information overload. Several software programs were identified where elements of the above are at work. No single program addresses all.
45
Discussion It is concluded that to support decision support in family practice, CCDSS should aim at several axes encompassing the broad field of Family Medicine. “The” decision support system does not yet exist.
2
Introduction This paper is one of a series prepared for this journal during a workshop by the World Organization of 50
National Councils and Academies (of general practitioners/family physicians; WONCA) in Prato, Italy, 2006. This paper is intended for expert readers of medical informatics journals, particularly software developers and clinicians, users of computerized clinical decision support systems (CCDSS).
Clinical decisions In the last decades, the emphasis in clinical work has shifted from “making the diagnosis” to “decide 55
correctly”. A clinical decision often starts with the recognition of a pattern emerging from the presenting symptoms and signs. A finding being present given a diagnosis, gives a likelihood function that modifies the prior or baseline probability into the posterior probability. Obtained evidence for the most relevant or probable hypothesis is then compared to decision thresholds: “test threshold” and “test-treatment threshold” as defined by Pauker and Kassirer (1;2) and “reference threshold” specific
60
for general practice. Other competing diagnoses are explored to bring them all down the test (and reference) threshold. Further, the clinician should select the appropriate therapeutic intervention, taking into account efficacy, cost, drug interactions, drug allergies, use during pregnancy and breastfeeding, equipotent doses of related drugs and adverse effects.(3-8) Finally, he should decide on the best management plan and follow-up scheme.
65
Rationale for CCDSS This process draws on the clinician’s acquired knowledge and experience, and published evidence. It may be guided by formally developed evidence-based protocols and guidelines. Computers do not have an intuition and are effective only where rules can be defined and phenomena measured in defined ways. The question arises if we might approach the clinicians’ intuition by introducing fuzzy
70
logic in suggesting diagnosis and decisions. This supposes a thorough analysis of the clinicians’ logic, which has been the subject of extensive research in the past. (9-12) Clinical decision support is focused on the gathering of information required for the diagnostic decision making process, the management of the patient and problem solving during (real-time) or outside (off-line) the clinical encounter. Its aim is to support and augment the skills of the general
75
practitioners/family physicians rather than substitute them.(13) CCDSS developed for family medicine should comply with validity, relevance and utility. In respect to validity, the CCDSS should be explicitly evidence-based on empirical data from family practice. For relevance, the CCDSS should be applicable in day-to-day family medicine and to common and important conditions. Finally, in respect to utility, the CCDSS should be shown to result in improved
80
clinical outcomes.(13)
3
Additionally to clinical decisions concerning diagnosis and treatment two more corollaries might result from a CCDSS: given a defined longitudinal management plan, what are the lessons to be learned? And: what is the impact on population health of such a management system? Shortliffe (1989, 2001) described 3 overlapping types of DS functions/tools: managing information, 85
focusing attention and providing patient specific recommendations.(14) The most successful CCDSSs have been simple rule-based systems that draw attention to data and facts e.g., drug-drug interaction or abnormal values that the user already knows but may have forgotten at the moment of decision.(15) In addition, a lot of effort has gone into the development of more sophisticated systems where mathematical modelling or fuzzy logic goes beyond the clinician’s immediate capacity to process data.
90
The most successful examples of the latter include Bayesian systems for supporting the diagnostic process.(16-18)
Building blocks for CCDSS An appropriate classification for symptoms, processes and diagnoses is essential to describe the events and knowledge within a profession or discipline. A clinical terminology may be conceptualized as 95
comprising reference, interface and aggregating terminologies. These terminologies are not formal entities on their own, but are formal transformations of one another, reflecting 3 overlapping functional dimensions of a terminology spectrum. The reference terminology is “an all-encompassing superset representation that either links to every other interface or reporting terminology or supports most of the useful interface or analytic functions within its own structure”. The interface terminology,
100
comprising preferred terms and synonyms, may be part or independent of but mapped to corresponding concepts in the reference terminology. Similarly, classifications or aggregating terminologies are separate and parallel terminology structures, related to reference terminology concepts by maps. WONCA’s International Classification for Primary Care (ICPC),(16) accepted as the most appropriate
105
classification for family medicine, is the primary care member of the World Health Organisation Family of International Classifications.(19) The ICPC proposes a data model of episodes of care and, because it allows unique classification of concepts, has been used for diagnostic and therapeutic decision support.(16) However, because it does not capture data at a sufficient level of detail for clinical care, it has been mapped to International Classification of Diseases 10th edition (ICD10) to
110
allow a greater level of detail and specificity (WONCA Thesaurus) in coding clinical information. In many situations, even ICD 10 is not specific enough. The Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT) has been selected as a reference terminology in the United States of America and Australia, because it has more unique health concepts than ICD10. A large nomenclature such as the SNOMED CT includes terms that can
115
describe almost any symptom or diagnostic finding. Such specificity or granularity allows building decision support tools for very specific situations. However, the more terms there are for the clinician
4
to choose, the lower is the reliability in coding: different doctors will choose different terms to describe the same condition, and even the same doctor can choose different terms at different times. On the other hand, the Unified Medical Language System (UMLS) brings together a number of 120
terminologies (clinical/user and aggregated) and is very comprehensive. It is the largest available metathesaurus – it also includes the SNOMED CT. Instead of inventing new terms or codes, the developers of structured patient records and decision support systems should first check whether the term or code they plan to use already exists in the UMLS. From the metathesaurus they can also find synonyms (in several languages), definitions, and hierarchies, which are all useful.
125
Apart from a correct thesaurus, a CCDSS needs a) a basic set of evidence-based rules for clinical decisions b) that can be written into a computer-readable form, c) structured (coded) data from the electronic patient record, d) the construction of trigger events that feed data from the electronic patient record, e) and a decision support engine that applies the rules to the electronic patient record data. The building blocks are different for every decision point.
130
The building blocks of a CCDSS are exemplified by the demo website of the Finnish EBMeDS decision support project (http://www.mrex.fi/EBMeDS/demo.asp?lang=en). Clinical data of a number of virtual patients have been coded and can be modified by the users utilizing different coding systems. Multilingual decision support as well as links to guidelines is provided. In addition to a rules-based decision support (inference) engine and a structured patient database, it has
135
been argued that more sophisticated CCDSS should include interfaces to allow a dialogue between the system and the clinician to promote ongoing learning and feedback as part of Quality Assurance & Continuous Professional Development (QA & CPD).
Objective The authors aim to promote awareness and to provide guidance about what is available and what is 140
desirable. Based on discussions and on a literature review, we propose a set of principles to guide clinician users and software developers. We present briefly some of the available expert systems and we examine if they comply with these principles.
5
Methods 145
A qualitative research method was used: first, in a meeting 3 key examples of CCDSS dealing with databases, diagnostic support, therapeutical advice and screening planning were presented and analysed: the Transition Project, Kabisa and Dépican. Next, after delineating the field of discussion, a literature search was performed. We searched Medline with “computer* clinical decision support” (the asterisk served to avoid problems with –ised or ized). Given the enormous amount of articles about
150
this topic in the last decade, we focused only on recent papers, published in 2005-2006. Experts were asked to suggest important articles not included in the search. We wrote the findings and expert opinions in a draft article and we submitted it to the group and to 3 other experts. During further e-mail discussion, we found two more key examples, Isabel and the Soja project. The process resulted finally in a list of principles.
6
155
Results Literature The Medline search came up with 139 articles and 11 reviews published in 2005-2006. Our experts suggested another 4 key articles. (20-23) Rochon et al. state that CCDSS is a promising feature to improve care but the results as yet are conflicting, (24) although the majority of recent articles and
160
reviews suggest a bright future. (3;5-8;24-29) Kumar et al. emphasize the role of standard terminology systems like UMLS, it constitutes one of the core components of such models. (21) Moreover, structured recording is a prerequisite for good results (30-32). Pisanelli and Gangemi find evidence for the fundamental role played by conceptual frameworks or models (ontology) when integration and interoperability of differing knowledge sources are needed, in particular in the field of clinical
165
guidelines and evidence-based medicine.(22) Plaza et al. state that cost-effectiveness of interventions may be more easily considered when a CCDSS is in place.(33) In a systematic review Kawamoto et al. suggest that clinical practice can be improved when decision support is embedded in the clinical workflow, offered as recommendations rather than assessments, provided at the time and location of decision making, and computer-based.(34) Unexpectedly,
170
providing periodic performance feedback, sharing recommendations with patients, and requesting documentation of reasons for encounter did not lead to compliance with recommendations.
Principles The following set of principles for the development of a CCDSS for family medicine was distilled:
175
1. CCDSS is about decreasing uncertainty – not eliminating it.(35) 2. Human diagnostic decision making should be simulated by repetitive serial testing of hypotheses and weighing them against thresholds. 3. Evidence-based guidelines should underpin CCDSS support. 4. Only if clinical data, test data, treatment data etc. are presented in numerical or categorical
180
format, Bayesian or pseudo Bayesian logic can be applied. (36) Highly structured electronic patient records are an essential element for the effective implementation of a CCDSS. Data should be structured in episodes of care, to allow the study of co-morbidity and chronic disease care within an appropriate data model. Data based on a single encounter does not reflect the complexity and richness of the family doctor’s approach to continuity of care. (17-
185
19;37) 5. When a threshold is reached for a certain condition, there is “a best available treatment for the moment” in a given country (based on specific evidence-based guidelines) that will change over time depending on changing evidence and cost-effectiveness data. Incorporating best
7
evidence, cost-effectiveness and patient preference or values requires complex Markov 190
models or probabilistic (Bayesian) analysis.(38) 6. In chronic disease management, there is “a current best available method of follow-up” in a given country (according to specific evidence-based guidelines). Family doctors should find a reliable (possibly electronic) source for up to date information, and it is desirable for such solutions to be linked to electronic patient record systems.
195
7. CCDSS could include reminders prompting doctors to intervene, especially for preventive or health promotion issues. Acceptability to doctors and patients should be optimal. Paternalism and disease-centred models of care are not appropriate. Information overload is an important cause of sub-optimal decision-making. (23)
Examples 200
First, we describe a project that focuses on how to get the necessary data to find the likelihood ratios that are appropriate in a local general practice facility. Then we will give examples that we feel are exemplary for 3 very important decision points (diagnostic, therapeutic, longitudinal management). Where do we get the necessary data? The best and biggest database for decision aids in family medicine is certainly the Transition Project,
205
managed by ICPC authors Henk Lamberts and Inge Okkes together with Sibo Oskam. This network research project started in the mid-80’s and continues to the present day, with data being collected using ICPC from episodes of care in day-to-day practice coded by family doctors in the Netherlands, Japan, Poland, Malta and Serbia.(16;17) Table 1 shows an example of a dataset on prevalences. The data on patients’ presenting complaints and doctors’ interventions and diagnostic labels allows
210
research into diagnosis in family medicine. The positive and negative likelihood ratios of a symptom against a diagnosis with an episode of care can be calculated for each pair of presenting symptom and diagnostic label. Diagnostic Decision Support The Transition Project generated an electronic patient record (Transhis – www.transitieproject.nl ) that
215
allows one to calculate Bayesian prior and posterior odds of a diagnosis given a presenting complaint. (16-18;37). Table 2 shows an example of diagnostic aid for the diagnosis of otitis media, based on new episodes of care of this condition in children aged 0-4 years in the Netherlands. Sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, odds ratio, prior (pretest) odds and posterior (post-test) odds are displayed. Ear pain increases the probability of a diagnosis of otitis
220
media in new episodes of care in young children more than its absence excludes this diagnosis.
8
An novel diagnostic aid has been developed in 2006.(Figure 1) Kabisa (www.kabisa.be) is an “intelligent” expert system for imported diseases based on a database of 2000 patients coming back from a stay in the tropics, gathered from 2000 to 2005.(39) For the power of findings, the expert 225
system relies on the mathematics and logic built in the original teaching program for clinical work in (sub-)tropical countries.(40) It gives a final ranking of data entered by a clinician, but on demand it will question the clinician, looking for strong confirmers and excluders for the diseases with probability > 5% and a low threshold (dangerous and treatable diseases).(Figure 2) The final “verdict” will be given only if a high probability is reached, after exhausting all strong findings, and if there is
230
sufficient difference between the highest ranked diagnoses. Isabel (http://www.isabelhealthcare.com/) acts as a diagnosis reminder system. For a given set of clinical features (entered through a free language natural text), Isabel instantly provides a checklist of likely diagnoses including bio-terrorism conditions, related diagnoses and causative drugs. The diagnoses are arranged by body system only, and in no particular order. In addition, the software
235
provides a knowledge mobilizing system which helps answer clinical questions with up to date knowledge (through a “quick consult” or through access to reference textbooks). Therapeutic decision support The Transition Project provides a prescribing engine (Prescriptor – www.digitalis.nl ) that links the Dutch College of General Practitioners prescribing guidelines to diagnostic labels, and incorporates
240
warnings for drug interactions, intolerances and contra-indications based on the individual patient’s problem list. Another dimension of incorporating evidence in practice is the sensitivity to patient values and preferences. An alternative or complementary approach to prescribing choices that are sensitive to patient preferences is exemplified by the Soja project (www.sojaonline.nl) that uses decision analysis
245
models based on expert consensus to select the most appropriate drug for a patient suffering from a selected disorder (e.g. diabetes, hypertension, hyperlipidaemia, etc.) by giving various parameters (e.g., cost, side-effect profile, dosing frequency, efficacy, etc.) a weighting depending on the patient’s values (e.g. prioritizing cost over side-effects, or vice versa). Thus the choice of drug may be both evidence based and sensitive to the patient’s values.
250
Longitudinal patient management While breast cancer mass screening is easy in theory (only two risk factors: age and sex) its application is not as widespread as expected (in France only 45% of women 50 to 74-years old participate). For individual screening the situation is worse with multiple risk factors and risk groups. Without help, the GP fails to assess the correct risk group in one third of the cases.
255
(http://www.sfmg.org/SFMG%20Anglais/poster_wonca_dpio.pdf)
9
Aware of this problem, the French Society of General Medicine has developed the Dépican software to allow general practitioners to realize individual screening of 6 cancers on a scientific base (with the support of the Fund for help of Ambulatory Care Quality). These cancers are: mouth, colon, skin (melanoma and epithelioma), cervix and breast. For each cancer and based on a bibliography of 400 260
articles, the French Society of General Medicine screening scientific committee has selected risk factors, risk groups and the most appropriate screening actions to be taken regarding each risk group. (41) During a regular consultation or a specific screening consultation, the doctor collects the individual and hereditary risk factors for the patient. The software calculates the risk group and suggests a screening procedure.(Figure 4) The free distribution of this software will enable the
265
acculturation of patients and doctors to screening and mass screening, and provide suggestions for individual screening for patients with a particular risk.(www.sfmg.org) The Dépican second version will be plugged directly with the electronic health record software.
10
Discussion 270
Due to the fact that qualitative research was used, this study does not pretend to be exhaustive, for literature nor for analysed programs. The list of principles is in fact complementary to Shortliffe’s. This list of CCDSS principles draws on the decision making and support work done in the USA, European Community and Australia over the past decades. The fundamental concepts of CCDSS and issues associated with the development and implementation of CCDSS are very similar in these
275
environments. A difference is that the EU and Australia had a strong GP/FP and primary sector compared to the USA, where most of the CCDSS work has been hospital-based; in fact most of the innovative USA work has been done in 4 benchmark institutions".(42) Do the analyzed expert systems satisfy the above mentioned principles? The transition project is certainly the best example of a large and reliable database. A thesaurus has been used, and the data are
280
presented in a format that is ready for use in a Bayesian diagnostic decision system. Some questions remain: how “hard” are the final diagnoses, as they are often made on clinical grounds alone? And are predictors evaluated in a multivariate way? Kabisa has the advantage over the transition project diagnostic aid that diagnoses are proven by paraclinical investigations. It includes also the “waning specificity”, the increase of false positives in
285
the course of a consultation, since competing diseases initially become more probable as the diagnostic panorama narrows. Finally it gives an intelligent help to the clinician, asking for findings he possibly has not readily in mind, also suggesting further testing, thereby offering also some training. This additional logical tool had been suggested also by Berner in an editorial on gold standard in CCDSS.(43) On the other hand, it covers only a very small array of all patients, and data does not
290
originate in the general practice. It is difficult to imagine such a tool functioning in the background of and electronic patient record. No information is given about the source of the data of Isabel, nor of the logical engine. The main strength is the links to information about the listed conditions, but there resides the danger, in information overload.
295
Both systems for therapeutical support we analyzed follow the aforementioned principles. They can effectively promote patient participation. However, successful communication with the patient is pivotal, and requires both time and skills. The Dépican program offers an intelligent proposal for screening, based on an extensive database, without regular built-in warnings, which many doctors would detest.
300
We might conclude that CCDSS should follow strict rules. Most available programs apply to just one field of the 5 fields where electronic decision support might be valuable. Trying to cope with the field of CCDSS will cost several years ... and require probably a tense relationship with the Medical Decision Making Society (http://www.smdm.org/main.html)
11
12
305
Responsibility of the authors The group managed to recruit the expertise of authors TL, IK and JVE in this project through retrieving some of their publications and inviting their participation. TL provided input on the basic principles of decisions. IK provided input on the application of guidelines in CCDSS.
310
JVdE and Emmanuel Bottieau provided input from the Kabisa project (www.kabisa.be) and participated in analysis of diagnostic systems and in the final editing. Carl Steylaerts and Jef Goris did the literature search and most of the writing. Jean Karl Soler participated in the writing. Didier Duhot explained and commented on Dépican.
13
315
Tables Table 1 The age distribution of diagnoses in new episodes of care (first consultation for a new health problem) for patients presenting with cough to Dutch family doctors. Acute bronchitis and pneumonia are much more likely in younger and older patients. Rates per 1000 patient years denominator. 3
320
Episodes for RFE R05 N (n=23933) Code Label
1
R78
Acute
Total
0-4
5-14
15-
25-
45-
65-
24
44
64
74
75+
30.4 101.8
25.9
13.0
16.0
26.6
50.1
64.2
bronchitis/bronchiolitis 2
R74
URI (head cold)
29.4 135.1
33.1
18.3
18.3
20.0
34.1
31.7
3
R05
Cough
24.1
71.1
25.5
13.5
17.3
22.9
27.0
32.8
4
R77
Acute laryngitis/tracheitis
10.0
19.0
7.5
5.3
8.5
10.1
17.1
13.4
5
R75
Sinusitis acute/chronic
4.7
3.3
4.2
3.7
5.1
4.7
7.1
3.9
6
R81
Pneumonia
3.3
10.2
4.5
1.2
1.6
2.5
4.7
8.1
7
R80
Influenza (proven)w/o
2.9
3.4
2.4
2.1
2.6
3.0
4.1
4.4
pneumonia 8
R96
Asthma
2.8
16.7
4.2
2.0
1.3
1.6
2.3
1.5
9
A77
Other viral diseases NOS
2.3
13.7
3.7
1.2
1.1
1.2
1.8
1.6
Whooping cough
1.0
6.4
3.7
0.6
0.3
0.2
0.2
0.1
4804 2957
1586
4997
4410
10 R71
Total
23933
14
2747 2432
Table 2 Example of characteristics of a predictor: ear pain as predictor of otitis media.
Row %
Row %
Episode of
Other
otitis media
episode
Total
With ear pain
995
59.4
680
40.6
1675
With other reasons
840
8.5
9054
91.5
9894
Total
1835
15.9
9734
84.1
11569
Sensitivity 0.54
LR+: 7.76
LR-: 0.49
PV+: 0.59
Odds ratio 15.77
Pretest Odds 0.19
95% CI.: 7.14-
95% CI: 0.47-
8.44
0.52
for encounter
Specificity 0.93
95% CI: 13.98PV-: 0.92
17.79
Posttest Odds 1.47
325 Example of diagnostic decision aid of the Transition Project: ear pain as predictor of otitis media in children aged 0-4 years in the Netherlands. LR+: positive likelihood ratio; LR-: negative likelihood ratio; PV: predictive value; CI: confidence intervals. Ear pain increases the probability of a diagnosis of otitis media in new episodes of care in young children more than its absence excludes this 330
diagnosis.
15
335
Figures Figure 1 General form if Kabisa travel expert (patients coming back from (sub-)tropical countries with fever. Different categories of findings can be chosen as present or absent. The program computes the probability of different hypotheses. The “analyze” button offers help, as shown in figure 2.
340 Figure 2 Example of dialogue between tutor and clinician in the kabisa expert system.
16
Figure 3 Dépican: entry of personal data in the warning decision support system for breast cancer.
345
17
Figure 4 Result screen of the Dépican warning system for breast cancer
18
References 350
Reference List
(1) Pauker SG, Kassirer JP. Therapeutic decision making: a cost-benefit analysis. N Engl J Med 1975; 293(5):229-234. 355
(2) Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med 1980; 302(20):1109-1117. (3) Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293(10):1261-1263.
360
(4) Bomba D, Land T. The feasibility of implementing an electronic prescribing decision support system: a case study of an Australian public hospital. Aust Health Rev 2006; 30(3):380-388. (5) McGregor JC, Weekes E, Forrest GN, Standiford HC, Perencevich EN, Furuno JP et al. Impact of a computerized clinical decision support system on reducing inappropriate antimicrobial use: a randomized controlled trial. J Am Med Inform Assoc 2006; 13(4):378384.
365
370
(6) Berner ES, Houston TK, Ray MN, Allison JJ, Heudebert GR, Chatham WW et al. Improving ambulatory prescribing safety with a handheld decision support system: a randomized controlled trial. J Am Med Inform Assoc 2006; 13(2):171-179. (7) Samore MH, Bateman K, Alder SC, Hannah E, Donnelly S, Stoddard GJ et al. Clinical decision support and appropriateness of antimicrobial prescribin: a randomized trial. ACP J Club 2006; 295(15):1771. (8) Peterson JF, Kuperman GJ, Shek C, Patel M, Avorn J, Bates DW. Guided prescription of psychotropic medications for geriatric inpatients. Arch Intern Med 2005; 165(7):802-807. (9) Elstein AS. Heuristics and biases: selected errors in clinical reasoning. Acad Med 1999; 74(7):791-794.
375
380
(10) Elstein AS, Schwarz A. Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. BMJ 2002; 324(7339):729-732. (11) Hamm RM. Medical Decision Scripts: Combining Cognitive Scripts and Judgment Strategies to Account Fully for Medical Decision Making. In: Hardman D, Macchi L, editors. Thinking: Psychological perspectives on reasoning, judgment and decision making. Chichester, UK: John Wiley & sons, Ltd, 2003. (12) Norman G. Research in clinical reasoning: past history and current trends. Med Educ 2005; 39(4):418-427.
385
(13) Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6(4):313-321.
19
(14) Musen MA. Clinical Decision-Support Systems. In: Shortliffe EH, Perrault, editors. Medical informatics. 2007.
390
395
400
(15) Musen MA, Shahar Y, Shortliffe EH. Clinical Decision Support Systems. In: Shortliffe EH, Perrault LE, editors. Medical informatics: computer applications in healthcare and biomedicine. New York: Springer Verlag, 2001: 573. (16) Okkes IM, Oskam SK, Van Boven K, Lamberts H. Episodes of care in Dutch Family Practice. Epidemiological data based on the routine use of the International Classification of Primary Care (ICPC) in the Transition Project of the Academic Medical Center/University of Amsterdam (1985-2003). Okkes IM, Oskam SK, Lamberts H. ICPC in the Amsterdam Transition Project. CD-Rom. ed. Amsterdam: Academic Medical Center/University of Amsterdam, Department of Family Medicine, 2005.: 2006. (17) Lamberts H, Oskam SK, Okkes IM. The clinical relationship between symptoms and the final diagnosis in general practice, determined by means of posterior probabilities calculated on the basis of the Transition project. Comment in: Ned Tijdschr Geneeskd.2006 150[3], 165-166; author reply 166. 2006. Ref Type: Journal (Full) (18) Okkes IM, Oskam SK, Lamberts H. The probability of specific diagnoses for patients presenting with common symptoms to Dutch family physicians. J Fam Pract 2002; 51(1):3136.
405
410
(19) Anonymous. World Health Organisation. Family of International Classifications. 2006. Ref Type: Internet Communication (20) Bisig B, Moreira J, Combes M, Asiimwe A, Bisoffi Z, Haegeman F et al. Comparison between an algorithmic vs. a panoramic clinical approach: effect of different logical approaches on diagnosis and treatment. 2006. Ref Type: Unpublished Work (21) Kumar A, Ciccarese P, Smith B, Piazza M. Context-based task ontologies for clinical guidelines. Stud Health Technol Inform 2004; 102:81-94. (22) Pisanelli DM, Gangemi A. If ontology is the solution, what is the problem? Stud Health Technol Inform 2004; 102:1-19.
415
420
(23) Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163(21):26252631. (24) Rochon PA, Field TS, Bates DW, Lee M, Gavendo L, Erramuspe-Mainard J et al. Clinical application of a computerized system for physician order entry with clinical decision support to prevent adverse drug events in long-term care. CMAJ 6 A.D.; 175(4):385. (25) Subramanian U, Fihn SD, Weinberger M, Plue L, Smith FE, Udris EM et al. A controlled trial of including symptom data in computer-based care suggestions for managing patients with chronic heart failure. Am J Med 2004; 116(6):375-384.
425
(26) Van-den-Ende J, Blot K, Kestens L, Van-Gompel A, Van-den-Enden E. Kabisa: an interactive computer-assisted training program for tropical diseases. Med Educ 1997; 31(3):202-209. (27) Ramnarayan P, Roberts GC, Coren M, Nanduri V, Tomlinson A, Taylor PM et al. Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasiexperimental study. BMC Med Inform Decis Mak 2006; 6:22.
20
430
(28) Morrison RS, Meier DE, Fischberg D, Moore C, Degenholtz H, Litke A et al. Improving the management of pain in hospitalized adults. Arch Intern Med 2006; 166(9):1033-1039. (29) Steele AW, Eisert S, Davidson A, Sandison T, Lyons P, Garrett N et al. Using computerized clinical decision support for latent tuberculosis infection screening. Am J Prev Med 2005; 28(3):281-284.
435
(30) Bleeker SE, Derksen-Lubsen G, van Ginneken AM, van der LJ, Moll HA. Structured data entry for narrative data in a broad specialty: patient history and physical examination in pediatrics. BMC Med Inform Decis Mak 2006; 6:29. (31) Sutton DR, Taylor P, Earle K. Evaluation of PROforma as a language for implementing medical guidelines in a practical context. BMC Med Inform Decis Mak 2006; 6:20.
440
(32) Schattner P. Using computers to work smarter--a guide for GPs. Aust Fam Physician 2006; 35(1-2):28-31. (33) Plaza V, Cobos A, Ignacio-Garcia JM, Molina J, Bergonon S, Garcia-Alonso F et al. Costeffectiveness of an intervention based on the Global Initiative for Asthma (GINA) recommendations using a computerized clinical decision support system: a physicians randomized trail. Med Clin (Barc) 2005; 124(6):215-216.
445
(34) Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330(7494):765. (35) Eddy DM. Variations in physician practice: the role of uncertainty. Health Aff (Millwood ) 1984; 3(2):74-89.
450
(36) Van-den-Ende J., Moreira J, Basinga P, Bisoffi Z. The trouble with likelihood ratios. Lancet 2005; 366(9485):548. (37) Wonca. The International Classification of Primary Care. Version 2 (Revised). ICPC-2-R. Oxford UK: Oxford University Press, 2005.
455
(38) Guyatt G, Rennie D. Users' guides to the medical literature. A manual for evidence-based clinical practice. Chicago: American Medical Association, 2001. (39) Bottieau E, Clerinx J, Van den EE, Van Esbroeck M, Colebunders R, Van Gompel A et al. Fever after a stay in the tropics: diagnostic predictors of the leading tropical conditions. Medicine (Baltimore) 2007; 86(1):18-25.
460
(40) Van-den-Ende J., Blot K, Kestens L, Van Gompel A, Van-den-Enden E. Kabisa: an interactive computer-assisted training program for tropical diseases. Med Educ 1997; 31(3):202-209. (41) Richard P, Gibily A, Grichy J. Dépistage et Prévention individualises Organisés. Doc Rech Med Gen 61. 2005. Ref Type: Journal (Full)
465
(42) Chaudry B, Wang J, Wu S, Maglione M, Mojica W, Ronth E. Systematic Review: impact of health information technology on quality, efficiency, and costs of medical care. 2006; 144: E12-E-22. Ann Internal Medicine 2006; 144(E):12-22. (43) Berner ES. Diagnostic decision support systems: how to determine the gold standard? J Am Med Inform Assoc 2003; 10(6):608-610.
21
470
22
