inductive learning method

Aflificial Intelligence in Medicine 5 (1993) 213-223

2L3

Elsevi€r

ARIMED 205

Medical decision making based on inductive learning method Josipa Kern, Gjum htdrija Sumpar

Deieli( Theodor Diirrigl and Silviie Vuleti6

School of Public Heatth Medical school, Ilniversity of Zagreb, Zagreb, Croatia

April 1992 Revis€d D€cember 1992 Received

Abstract

Kem, J., G. Deieli6, T. Dtinigl and S. Vuleti6, Medical decision making based on inductive leaming mothod, Artificial Intelligence in Medicne 5 0993)213-223. Modical decision making based on inductivo leamiog has be&[ studied in orde. to collect exporienco necessary for practical us€ of such methods in clinical aDd epidemiological work. The decision trees have beoo constructed by using the modifled Quiolan's appoach based on choosing rclevant attributes accoding to their iniormativity. AD inductive loaming software tool, ASSISTANT Pofessioqal, has been used for expeiimenting. The variability in results has beon studied undervarying leaming conditions. Two setrs of data have been chosen for leaming experimentrs: ftom a study on rheumatoid factors in patients with rheumatoid arthritis, and ftom an epidemiological irvestigation of ageing. The rcsults of this study indicate the oecessity to determine inductive leaming parameters for oach padcular problem. The pruniog procedure is always recommended as it eliminates redundant elements in the tree. In problems with greater number of aftribute$, however, pruning its€lf is not guara[teeing satishctory solutions. Interventions like the change of the minimal weight threshold might improve the situation. If these prccautions are met, the method of inductive leaming s€ems to b€ a useful guide in pmctical clinical and epidemiological decisions.

Keywor^.

N{e.dical decision-making; inductive loaming; rheumatiod arthritis.

1. Intmduction Decision making is broadly defined to include any choice or selection of alternatives. Within the complicated world of medical problem-solving and decision-making, any assistance for lhe physician could be helpful. It is hardly amazing, therefore, that much effort has been spent in developing systems to support doctors in their difficult tasks [17]. Computer-aided medical decision making systems are expert systems that directly aid a physician in evaluating and managing a patient [19]. Correspondence to: Dr. Josipa Kem, Andrija Stampar School of Public Health, Medical School, University of Zgreb, Rockefellerova 4, P.O.B. 770, HR41000 ZAGREB, Crcatia, fax & phone: + 38 41274 742, amail: [email protected]

azs-Zzn1m1W.m @

1993

-

Elsevier Scienc€ Publish€rs B.V.

All

righrs res€rv€d

2t4 Problem solving in medical decision making are primarily problems of diagnosis, prediction, interpretation, and management. There are two ingredients to the logical concepts inherent in medical decision making: they are (i) data, and (ii) medical knowledge. Medical data are available from the patient medical record. Medical knowledge is a result of leaming. For example, in the case of teaching medical students, when students are taught by experts, they become able to cope with a variety of, more or less, similar or dissimilar examples. After finishing this process of professional maturing, they are able to apply their 'knowledge base' and make decision or prognosis.

The knowledge acquisition phase has a very important role in the development of a computer-aided medical decision making system. There are two ways of knowledge acquisition:

(i) (ii)

directly, through communication with experts, and indirectly, from examples.

Knowledge acquisition is further complicated by the inability of experts to formulate explicit rules from their own knowledge and decision making and by the lack of automated rule generation techniques which incorporate the knowledge ofthe specialized domain forapplying to computer systems. Therefore, automated leaming, machine learning or inductive learning techniques, offering a possibility ofacquiring knowledge from complex data, usually in noisy domains, can play an important role in helping the medical doctor understand key factors in clinical diagnosis. This would enable clinical events and outcomes to be better forecast resulting in a better understanding of the clinical situation [3, 5, 8, 15]. There is a lot of algorithms being used as inductive learning techniques: indentification methods like different types of discriminant analysis, regression tree construction for quantitative output prediction, Bayesian principle, minimization of entropy, pattern recognition techniques, etc. ll4, 16, 13, 10, 31. One of the first and well-known algorithms is ID3 [16] as well as some of its modifications. The main idea of Quinlan's approach is a decision tree construction based on examples used for leaming. All examples are described by the values of relevant attributes and by the class to which the examples belong. The decision tree is constructed by choosing attributes according to their informativity. The leafs of the tree are assigned a class containing all the conesponding examples. The procedures coltinue until the tree classifies correctly all the examples used for leaming [2]. In such a way the rules for problem-solving are established and can be applied to the classificationof new examples. In our recent papers [7, 11, 12] some different problems have been studied by applying the software tool for inductive learning of decision trees, named ASSISTANT Professional developed by Bratko and collaborators [4], based on Quinlan's ID3 algorithm. While experimenting with the inductive learning of decision rules [7], some different results have been obtained for the same data. Therefore it seems to be of interest to study more thoroughly the variability in inductive leaming results obtained with the same or different groups of data under varying learning conditions. The variability in inductive leaming results reflects in:

(D (ii)

different decision trees and conesponding variability in prognostic accuracy, regarding different subsamples for leaming and testingi the variability of prognostic accuracy with regard to changing system parameters like:

-

ratio of learning and testing examples in the sample we have,

Med,ical decision making based on inductiue learning method

(iii)

2t5

minimal weight threshold which makes direct influence on generating the subtrees, or pruning or not pruning the decision tree;

the differences in variability regarding the complexity of problems, i.e. number of attributes describing the examples.

Such investigation might give more insightinto performance ofthe inductive learning method used, thus allowing to collect experience necessary when decision trees are used in practical clinical and epidemiological work.

2. Materials and methods The data used for experimenting with the inductive learning of decision rules are from the studies:

(i) (ii)

on clinical evaluation ofserologic tests for rheumatoid factors in patienls with rheumatoid arthritis [7], and

on epidemiological investigation of ageing [20, 9].

Data of 377 patients of the first study include the course of disease and present diagnostic criteria of the American Rheumatism Association (ARA). As the data had been collected in mid-seventies, these are the 1958 ARA diagnostic criteria [19]. The criteria used in the present work are: morning stiffness (ARA-l), pain in at least one joint (ARA-2), joint swelling (persisting for at least six weeks, ARA-3), swelling of an additional joint (during three months, ARA-4), symmetric swelling of joins (ARA-5), subcutaneous nodules (ARA6), typical x-ray changes (ARA-7), and positive test for rheumatoid facton (ARA-8). The remaining three criteria (mucin clot, synovial and nodule biopsy) have been omitted on practical grounds [6]. ARA revised lhe criteria in 1987 [1]. These new criteria differ ftom the old list in three points: ARA-2, ARA-3 and ARA-4, which are replaced with two more precise definitions: arthritis of three of more joint areas, and arthritis of hand joints. It is not possible to corelate in full extent findings based on old criteria with the new ones, but the fact lhat five of the criteria in both sets are essentially the same allows us, in the scope of this work, to reach sound conclusions. The course of disease, described as 'continuously progressive' and'episodic or intermittent', is an outcome, the class which shouldbe predicted. The original classes 'episodic', 'intermittent' and 'continuously progressive' have following meanings. 'Episodic' means a course of disease which is characterized with short evolutive attacks with longer remission intervals, in the 'intermittent' course the phases of inflammatory activity and remissionexchange regardless of the length ofthe periods ofactivity or remission, whereas 'continuously progressive' means a steadily existing inflammatory activity during which neither spontaneously nor by treatment an objectively provable remission could be reached. The data for all examinees was complete. The second study includes 5511 examinees. Two samples were drawn, one of 5'J7 examinees described by sex, age, fasting glucose, postload glucose, cholesterol, triglycerides, blood pressure, liver disease, heart disease, gastrointestinal disease, kidney disease, diabetes, other diseases, cigarette smoking, physical activity, body mass index, oil/fat consumption, alcohol consumption - all the attributes measured in one time point - and, cause of death or

216

,1.

Kern et al.

the fact that the examinee is still alive. The cause of death or'still alive' is an outcome, the class which should be predicted. The second sample of 1411 examinees, all passed away, is described by all the same attributes, predictors, and by the time elapsed from examination to death, as the class. There is a number of examinees without complete data.

The gold standards are outcomes doctors noticed (in the clinical example the course of rheumatoid arthritis disease, and in the epidemiological example the cause of death and time elapsed from examination to death). Method ofinductive leaming applied here is that described by Kononenko et al. [13] built in

ASSISTANT Professional, the software developed by Bratko and his group [a]. ASSISTANT belongs to TDIDT class (top Down Induction of Decision Trees). It presumes existence of an appropriate number of learning examples described by a set of attributes and by classes representing conditions (diagnoses, courses of disease, etc.). The algorithm for decision tree construction searches for the most informative attribute as follows. The information amount necessary to classify an example, -8, equals to

E = -E;p;logr(p;), where p; is the a priori probability that the observed example belongs to class i. If the root of the tree is attribute A with V different values, the new amount of information necessary to classify an example appears to be

r(A) = -2,p, 10,;/p,)tos2(p,;lp,)

,

p, is the a priori probability that the observed example has the yth value of attribute A, and p,; the probability that the observed example has the yth value of attribute ,4 and belongs to class i. The best attribute is the one minimizing the function I(A) as it renders the

where

maximum of information. The informativity of attribute A is defined as

Inf(A)=E-I(A). The system includes the possibility to work with continuous attributes, incompletely specified learning examples, automatic choosing of good learning examples, tree pruning, etc. Before startingwith a decision tree construction ASSISTANT Professional offers to put some parameters. Its authors have recommended, according to their experience with the system and various data seb, some of the parameter values. The paramete$ are. as follows: selection of only good instances (good instance means that it is conectly classified according to Bayesian classification principle), percentage of instances used in the leaming process (707o is recommended by the authors of ASSISTANT Professional), informativity pruning factor (with values between 0 and 9; 3 is recommended), class frequency threshold (percentage of instances at a given node having the same value; LNTo is recommended), minimal weight threshold (if in a certain node the actual percentage of weight of learning set falls below the declared threshold, this node is considered as a leaf; weight of an example determines the importance of it for the classification problem; the usual weight is 1, or, between 0 and 1 for examples with 'unknown' values, or greater than I for examples with 'don't care' values for some attributes; 2Va is recommended) and pruning (yes or no). A decision tree can be evaluated according to the absolute accuracy, Ao, and relative accuracy, Ar, with which it classifies examples.

A"= KIN

,

Medical d.ecision ma.k'ins based on inductiae learnins method

where

I(

is the number of correctly classified examples,

A, = (I|N)E"K,l@P,)

N

is the number

2t7

oftesting examples.

,

where .B is the number of classes, 1{. is the number of correctly classified examples from rth class, and P, is a priori probability of the rth class. A detailed description of the algorithm is given in the paper by Kononenko et al. [13]. In our experiments we decided to use not only good instances, the recommended pruning factor of3, and class frequency threshold of 100%. Other parameters were varying.

3. Results 3.1 Experiments with different random sarnples

Such type of experiments has been performed with two samples of data:

(a) (b)

rheumatoid arthritis data, and

epidemiological data with 'time elapsed from examination to death' as the class.

Fig, 1. A tree constructed without pruning ftom rheumatological data.

214

J. Kern et al.

(a) Rheumatoid ar thr i tis

(I)

7O7o of instances have been used as learning examples, and the 307o instances served for testing of decision rules;

of remaining

Q)

50% of instances have been used as leaming examples, and the instances for testing of decision rules.

of remaining

SOVo

Fig. 2. A tree constructed with pruning from rheumatological data.

The learning examples are chosen at random, so each new tree construction can be expected to produce a different result. Experiments were made both without and with pruning. Testing have been performed in three ways: 'Tree only', 'Fint Tree Then Bayes' and 'Bayes In Tree'. The method 'First Tree Then Bayes' uses a decision tree to calculate prior probabilities for the Bayes method and then classifies examples with Bayes method considering only attributes that do not appear in the corresponding branch of the tree. The method 'Bayes In Tree' uses a decision tree for classification, taking into account already calculated Bayesian probabilities in the leaves. Each testing was done ten times. Tables 1 arld 2 present a summary of results on automatic tree construction obtained in our experiments. Experiments show that there is more variability in trees constructed on the basis of 507a learning and 507o testing examples than with 7O% leaming all.d 3O7o testing examples. At the same time, much less identical trees are constructed during the learning process. However, the prognostic accuracy remains nearly the same in both cases. (b ) Epidemio lo gic al dat a

f)7o of instances were used as learning examples, and, tre 1O/o of remaining instances served for testing of decision rules. The leaming examples are chosen at random, both with and without pruning, in 'Tree Only' mode. Each testing was done ten times. hble 3 presents a summary of results on automatic tree construction obtained with pruning. Experiments

219

M edical decisi.on makins based on inductiue learnins method Table

1.

Summary of experiments in construction of decision tiees for classification of rheumetoid arthritis patients regarding the course of disease (lOEo le rl]ing, 30% testing examples).

constructio[ attribute

of

With

k€es

pruning:

resulting

fteq. of ideotical trces

Tree

o

No

prognostic

(%)

(qo)

^ccnftcyrelative absolute ^c.Dftcy rclative absolute

ly

ARA-8 5 ARA6 5 ARA{ 9 ARAS 1

Yes

prcgnostic

6a.1J7.9 6t.645;t

2

72.5

68.1J52 6L.444.4 12.6 70.s-76.1 61343.1 73.3

3 7

17.O

1

63.6

63.0 62.4

61.2

First Tree Theh Bayes

ARA.8 ARAS ARA.8 ARA6

No Yes

Bayes

No

I

ARAS

Thble

2.

71..6

63.0

2

2

63.234.9

76.1

64.O

8 2 8 2

5

63;1J7.9 fi.244.2 7t;7-18.4 .845.t 655J52 59343.4

7t.4

61.9 63.4 61.6 62.7

4 8

3 3

16.t

72.9 72.4

64.4 63.1

Tree

ARA.8 ARA-6 ARA-8

Yes

5

69.0-71.0 63.2-($.4 70.4-73.4 62.343.4 64.6-75.2 60.444.1

6

0 5

752-76.1 622..632

2

15.3

71.7 75.7

Summary of experiments iD construction of decision tr€es for classification of rheumatoid arthritis patients regarding the couNe of disease (5O% leaming,50% testiDg examples).

conshuction attribute ofresulting

freq.

With

identical

pruning: Tree

trees

of

prognostic

ac.vacy

(Eo)

prognostic acc,uracy (Ea)

trees

absolute .elative absolute

relative

0 2 0 2 2

71.8-72.9 62.0-43.1

62.6 62.2

o

69.2-75.0 62.544.1 7t.9 75.5-77.9 62.6-63.2 76.7

o ly

No

ARA€

Yes

ARA.6

7

ARA-4 ARA-8

8

ARA.6

1

1

(92-77.1 61.5-63.1

71.3

62.0

64.4-750 55.9-63.7 71.9-74.5 622.42.4

72.5

74.2

70.t 73.2

60.5

623

First Tree Then Bayes

ARAA 3 ARA6 6 ARA.4 1 ARA.8 7 ARA.6 2 ARA-4 1

No

Yes

Bayes

No Yes

In Tree ARA.8 ARA.6 ARA-8

4 6 7

ARA-6

3

3

o 0 0 0

0 0 3 3

7t.8

63.4

'70.0-74.5 6t.844.5 7t.6 64.6-76.1 61.244.4 73.4 70.7

63.0 63.0

6t.6

.6-72.3 59 .&425 69 .4 70.L76.1 fi.642.3 72;t 69.7J4.5 ffi.743.t 71.9 '1L.3J6.6 6t.743.O 74.1 67

63.3

62.9

6L.2 61.4 61.9 62.4

220 made wilhout pruning have produced a lot of nodes in any decision tree, but the prognostic accuracy in both cases, with and without pruning, was nearly the same. Table

3.

Summary of experimeDts in construction ofdecision hees for classification

of €pidemiological examples regarding the time elapsed ftom examination to death (9OZa leaming, 10% testing exampleq with pruning).

attribute ofr€sulting

h€es

Region

4

fieq. of

prognostic

prognostic

identical accnr?,cy (Va) acaracy (Vo) trees absolute rclative absolute relative

2

3t.6-39.7 29.2-31..1

34.4

29.A

3.2 Experiments with changing the minimal weight threshold

These experiments were performed with all three data sets. Experiments with changing the minimal weight threshold on the same leaming/tesling samples, i.e. changing the required minimal relative sum of weights of examples in a current node with respect to the sum of weights in the root of the tree (if the threshold increases, the decision tree becomes smaller), have been performed in order to analyze three problems:

-

with the outcome where the course of disease was either episodic/intermittent or continuously progressive, with the outcome cause of death or still alive, and with the outcome, the time elapsed from examination to death.

In the case of rheumatoid arthritis data, obtained on the basis of 70% learning examples, there was no significant variation, both in the prognostic accuracy and the number of nodes, regardless of the tree pruning procedure. Iable 4 shows results obtained by experimenting with the classes ofoutcomes declaring the cause of death or being alive. The process of decision tree constuction was carried out with the same random sample, sample size of 7O% of the set of examples-instances (n aO4). Testing was performed with the remaining 30% examples (n = 173). As it can be seen, the prognostic accuracy, absolute as well as relative one, is very high, from 8.447o to X).75Vo for absolute, and, from 86.35/o to 88.18% for relative accuracy, but it is varying with regard to the minimal weight threshold.

=

?able 5 shows results obtained by experimenting with the classes of oulcomes declaring the time elapsed from examination to death, which was detemined on the basis of attributes measured at one moment. The process of desicion tree construction was carried out with the same random sample, sample size of X)Vo of the set of examples-instances (r, = 1288). Testing was performed on the remaining lOVo examples (n - 143). As it can be seen prognostic accuracy, absolute as well as relative accuracy, is rather low, from 36.O1/o lo 45.80% for absolute, and, from 28.59/o to 32.457o for relative accuracy, but it is varying with regard to minimal weight threshold.

Medical decision makins based on ituducti& learnins nethod. Thble

4.

221

of decision trees obtained with epidemiological data, with caus€ of death aDd still alive as the outcome O0% leaming, 30% testing Characteristics

examples).

threshold (%)

2 4

10

Thble

5.

ebsolute relative

nodes

leaves

89.98 87.53 75 89.98 88.13 53

38 27

Yes

90.17 88.18 35 89.60 47.62 23

18 12

Yes

90.75 90.t7

47.76

23

t2

No

;13

13

7

Yes

90.75 90.L7 a9.o2 88.,14

87.60

2t

11

No

.73

13

7

Yes

a1.B

17

9

No

86.35

9

5

Yes

No

No

Chamcteristics of decision trees obtaioed with epidemiological data, with time elapsed ftom examination to death

as

thg o,utcome(goEo

le ming, lol%

testiDg oxamples).

threshold (%)

2

absolute

229

leaves

115

L9t

No Yes

45.45

32.45

t07

45.80

30.62

65

43.35 43.36

30.69

61

31

No

29.33

4t

2t

Yes

47

u

No

L2

Yes

20

No

41.96 4t.96 10

Elative nodes

36.0t 30.15 36.71 30.66

4t.96 4L.96

29.97 ?A.59

29.6 24.59

)a

39 23

54 33

No Yes

12

4. Discussion Results of all experiments show that the prognostic accuracy is nearly constant for a defined problem, no matter whether the accuracy is high or low. Taking into account changes in the learning/testing sample with rheumatological data, the resulting decision trees were varying in contents but not very much in size, prognostic accuftrcy and the attributes included in the tree. It can be speculated, and this is often heard from professionals, that two or three attributes (as in our case e.g. ARA criteria), no matterwhich of them, are relevant for prediction ofthe course

222

of the disease (rheumatoid arthritis). Experimenting with low predictable causes of death on the basis of many in-one-time-point measured indicators, i.e. attributes, the obtained decision trees became huge if the minimal weight threshold was put lo 2Ea (as recommended by the authon of ASSISTANT Professional). Even more, there is no possibility of cutting the tree, in spite ofincreasing the pruning factor to its maximum. The situation improves by increasing the threshold. The decision trees are smaller with the same or better prognostic accuracy, thus more suitable for use. This can be seen in Frg. 3 where the curves are constructed from Table 5. The maximum in prognostic accuracy is achieved at minimal weight threshold which is higher than that recommended by the system designers. A general trend can be observed as follows: the number of nodes and leaves in the trees decreases rapidly, reaching a low stable condition at a particular threshold value (e.g. at 4/o inFig.3); the absolute and relative accuracy are slowly increasing, but maxima can be observed in both cuwes.

Minimal weight thr€shold (%)

Fig.3.

Numb€r of nodes and leaves as well as absolute and relative prognostic accuracy (%) in trEes constructed with pruning ftom epidemiological data (Table 4')

in

derP-ndence of the minimal weight threshold.

The results of our experiments indicate that tr€e construction mechanisms are influenced by the number of attributes and their mutual dependence. So in the case of rheumatological problems where the number of attributes is small, there has been almost no influence of the minimal weight threshold on tree construction. In our previous paper [8] we suggested that the variability in tree construction could arise from a rather small number instances used for learning and iesting. The results collected in Tables 7 and 2 indicate that the increase of the number of learning instances stabilizes tree construction which follows from the increase of the number of identical resulting trees, especially when the trees are pruned. The average prognoslic accuracy becomes also more stable by increasing the number of learning instances. Here, tree pruning does not affect very much the overall prognosfic accuracy. In summarizing the experimental evidence of this study, the necessity to determine inductive learning parameten for each particular problem has to be pointed out. The pruning procedure is always recommended as it eliminates redundant elements in the tree. In problems with a greater number of attributes, however, pruning itself is not guaranteeing satisfactory solutions. Interventions like the change of the minimal weight threshold might improve the

M edical d.ecision makins based on;tuduct'iue learn;ns method

223

situation. If these precautions are met, the method of inductive learning seems to be a useful guide in practical clinical and epidemiological decisions.

References

[1]

F.C. Amett, S.M. Edwodhy, D.A. Bloch, D.J. McShaDe, J F. Fries, N.S. Coopet L.A. Healey, S.R. Kaplan, M.H. Liang, H.S. Luthra, T.A. Medsger, Jr., D.M. Mitchell, D.H. Neustadt, R.S Pinals, J.G. Schaller, J.T. Sharp, R.L. Wolder and G.G. Hunder, The American Rheumatism Association 1987 revised criteria for the classification of fieumatoid arthritis, ,{rrrlirii.i Rheun 3l (1988) 315-324 .

[2] I. Bratko and P. Mulec, An experiment in automatic leaming of diagnostic rules, lnlorm atica 4 (l9AO) 18-25 , [3] C.A. Cameron, G.V Conroy and M.D. Kaogavary Machine leaming techniques fot Patient and proglam [4]

management io rcnal replacement/transplantation therapy, in: R. O'Moore, S. Bengtsson, J.R. Bryant and J.S. Bryden, eds., Prac. MIE 90, Glasgow (SPinget \99O) 286-297. B. Cestnik, I. Kononenko aod L Bratko, ASSISTANT 86: A knowledge elicitation tool for sophisticated use6, in: l. Bratko and N. tavraE, eds., Progress il Machi e Learhing WSL 87 (Sigma Press, Wilmslow,

1987)3t45.

[6]

I

Bratko and N. I-avrat, eds., Progress ifl Machi e Learning, EWSL 87 (Sigma Prcss, Wilmslow, 1987) 11-30. Gj. DoZeli6, Th. Dtirrigl, N. Dezeli6, v. Z€rgollem, H. Jurak, M. vitaus aod S. Androi6, The photomehic latex test for rh€umatoid frctoN in patients with rheumatoid arthritis, II, Cllinical evaluation,Z. Rheurrlatol.

[5] P Clark and T. Nible$, Induction in ooisy domains, in:

37

(97A) tD-122.

Gj. DeZelid and J. Kem, Inductive loaming as a method for medical decision making, in: K,-P. Adlassnig, G. Grabner, S, Bengtsson and R. Hansen, eds.,Proc. MIE gl,v,enna (Springer 1991) 327-331. [8] A.S. Houston, RJ. Ioms and M.A. Macleod, The use of induction in the design of an expert system for thyroid tunction shrdies,Nuclear Med. Com un- 12(1991) 497-506. [9] D. Ivankovii, J. Kem, Z. Soii6, M. KujundZi6 and S. Vuleti6, Ageing $'ithout impairment and disease Crhort study: II. Twenty year's mortality and causes of death (iD prcss). [10] B. Karali6 and B. Cestnik, The Bayesian approach tree-structured regression, in: V ieri6, V Dobri6' V LuZar and R. Paul, eds., Proc. 13th Interut. Conf. oa Informatiott Tech ology Interfaces, University C-omputing

[7]

Centre, Zagreb. (1991) 155- 160. S. Vuletii, Inductive leaming method as a tool for forecasting in epidemiology, in: V. Cori6, lrrforriatiot Technolog) Interfaces, V. Dobri6, V Luzar and R. Paul, ods., Proe 13th te at. Cortf. University Computing Crntre, Zagreb (1991), l7ll7A. [12] J. Kem, M. TeZik-Ben6i6, S. Vuleti6, Gj. Dezeli6 and Kujundzi6, An inductive leaming method versus the common statistical appoach to outcome forecastinl,Coll. A trop 15 (1991)283-249.

[11] J. Kem and

[13] I. Kononenko,

I

o

I. Bratko and E. Rodkar, A system for i[ductive leaming ASSISTANT (original in Slovenian),

Infornatica 70 (19 ) 4152. fL4l L.B.lnsted,lhtrodltction to MedicalDeciion Making $homasBooks, Springfield, 1968) [15] T. Niblett, CoDshuction decision trees io noisy domains, in: I Bmtko and N. I'avraa, eds., Progless itt ElrySL 87 (Sigma Press, Wilmslow, 1987) 67-78. Dichotomizor 3 (ID3), Artiicial Intelligence l-abomtory, Stanford University, CaliJ.R. lterative Quinlan, [16]

Machi e Learning, fomia, 1979.

[17] J. Ridenikotr, A diagnostic decision support system for routino cliDical pEctis€, J. Clinical Comput. 17 (1988) 1-16. [18] M.W. Ropes, GA. Bennett, S. Cobb, R. Jacox and R.A. Jessar, Revision ofdiagnostic criteria for rheumatoid Bull. Rheum. Drs. 9 (1958) 175-176. ^tthi,tis, S. Tuhrim and J. Reggia, Feasibility of physician-development expert systems, Med Decis, Making 6 (7986) [19] 23-26.

[20]

Vuletii, J. Kem, D. Ivankovi6, M. Kujundzi6 and Z. Sosie, Ageing without imPairment and disease Crhort shrdy: I. Baseline cohort chatacten$lics, Coll. Antropol 15 (1991) 197-212.

S.

-