Decision-making and diagnosis in disease ... - Wiley Online Library

Plant Pathology (1999) 48, 147–153

LETTER TO THE EDITOR

Decision-making and diagnosis in disease management G. Hughesa*, N. McRobertsb and F. J. Burnettc a Institute of Ecology and Resource Management, University of Edinburgh, Edinburgh EH9 3JG; bPlant Biology Department, Scottish Agricultural College, Auchincruive KA6 5HW; and cCrop Health Department, Scottish Agricultural College, Edinburgh EH9 3JG, UK

Introduction Suppose a decision is to be made on whether or not to apply crop protection measures, based on the use of a risk algorithm. The term ‘risk algorithm’ is used here to refer to any calculation that uses observations of one or more components of the ‘disease triangle’ – the host crop, the pathogen population and the environment – to make an assessment of the need for crop protection measures, judged by comparison of the result of the calculation with some predetermined threshold value. Depending on the outcome of this comparison, a decision on whether or not to treat a crop will be made. It has to be accepted that decision-making, based on whatever risk algorithm is employed, will not be perfect. Along with correct decisions to treat when treatment is required, and not to treat when treatment is not required, incorrect decisions will sometimes be reached. That is to say, sometimes the decision to treat will be made when treatment is not required, and sometimes the decision not to treat will be made when treatment is required. Obviously, in order to be of practical use, it is a requirement that risk algorithms lead to correct decisions most of the time. Some method of evaluation is therefore required. The main objectives of this article are to draw attention to a paper by Murtaugh (1996) on the statistical evaluation of ecological indicators, and to try to explain its significance in the context of disease management decisionmaking. An indicator is an easily measured substitute for a property of a system that is difficult to measure directly. Murtaugh (1996) discussed the use of receiver operating characteristic (ROC) curves to assess the usefulness of indicators in the general context of monitoring environmental quality. In the context of disease management decision-making, we can think of risk algorithms as indicators. Economic yield loss cannot be measured directly until it is too late to prevent. The purpose of a * To whom correspondence should be addressed. Accepted 2 November 1998. 䊚 1999 BSPP

risk algorithm is to provide a substitute for the measurement of economic yield loss, allowing an earlier assessment of the need for crop protection measures. Yuen et al. (1996) suggested the use of ROC curves as a means of evaluating risk algorithms. This is discussed, along with some related problems, including the assessment of accuracy, the use of discriminant function analysis for assessment of the need for crop protection measures, and the use of data obtained by sampling in the establishment of evaluation methods.

Receiver operating characteristic curve analysis ROC curve analysis is widely used by clinicians as a means of evaluating diagnostic tests for decision-making in the context of patient management (see, for example, Metz, 1978; Zweig & Campbell, 1993; Schulzer, 1994). We outline the analysis in that context before discussing its application in disease management decision-making. ROC curve analysis of a laboratory test proposed as a basis for clinical diagnosis proceeds as follows. First, from a group of subjects, two subgroups are established. One subgroup comprises the ‘cases’ – all those individuals known definitively to be suffering from the particular condition in question; the other subgroup comprises the ‘controls’ – all those individuals known definitively not to be suffering from the condition. The classification into cases and controls is made independent of the diagnostic test, the performance of which is being evaluated. The diagnostic test is then performed on all the individuals in both subgroups. Typically, this procedure results in two overlapping frequency distributions of test scores, one distribution for the cases, the other for the controls (Fig. 1). Since the two distributions overlap, the test does not provide perfect discrimination between cases and controls. In Fig. 1, most of the cases have test scores above the indicated threshold (these are true positives), but some have test scores below the threshold (these are false negatives). Most of the controls have test scores below the threshold (these are true negatives), but some have 147

148

G. Hughes et al.

Figure 1 Frequency distributions of test scores for an hypothetical diagnostic test. By convention, the frequency distribution of test scores for cases is shown above the test score axis and the frequency distribution of test scores for controls is shown below the test score axis.

test scores above the threshold (these are false positives). The question therefore arises as to what threshold test score should be adopted for the implementation of treatment in a subject where the test score is the main information available (the definitive condition being unknown and, say, too risky, time-consuming, or expensive to establish at the outset). Setting the threshold at a lower test score than the one indicated in Fig. 1 will reduce the number of false negatives but, at the same time, increase the number of false positives. Conversely, setting the threshold at a higher test score than the one indicated in Fig. 1 will reduce the number of false positives but, at the same time, increase the number of false negatives. Before an ROC curve analysis, some notation is required. The cases are all definitively disease positive (D þ) individuals, the controls are all definitively disease

negative (D –) individuals. Most of the cases, and some of the controls, provide test scores (T) above the threshold value (T > Tthresh). Most of the controls, and some of the cases, provide test scores (T) at or below the threshold (T ⱕ Tthresh). We make the following definitions: the true positive proportion (TPP) is the number of true positive decisions divided by the total number of cases; the false negative proportion (FNP) is the number of false negative decisions divided by the total number of cases; the true negative proportion (TNP) is the number of true negative decisions divided by the total number of controls; and the false positive proportion (FPP) is the number of false positive decisions divided by the total number of controls. Then: TPP is an estimate of Prob(T > Tthresh | D þ) (read as ‘the probability of a test score above the threshold, given the presence of disease’), and similarly, FNP is an estimate of Prob(T ⱕ Tthresh | D þ), FPP is an estimate of Prob(T > Tthresh | D –), TNP is an estimate of Prob(T ⱕ Tthresh | D –). The TPP is often referred to as the ‘sensitivity’ of a diagnostic test, and the TNP is often referred to as its ‘specificity’. An ROC curve is a graphical plot of TPP (sensitivity) against FPP (1 – specificity), the values of TPP and FPP being calculated by allowing the threshold test score (Tthresh) to vary over the whole range of test scores (T). The ROC curve shown in Fig. 2 is based on the distributions of test scores for cases and controls shown in Fig. 1. The plot passes through the points (0, 0) (which in decision-making terms corresponds to never treating) and (1, 1) (which in decision-making terms corresponds to always treating). For the purposes of evaluating a diagnostic test, an ROC curve that passes close to the point (0, 1) (in the top left-hand corner of the plot) shows the test has both desirable sensitivity and specificity characteristics (that is, relatively high values of both can be achieved with an appropriate choice of threshold test score). A straight line joining the points (0, 0) and (1, 1) is the ‘no discrimination’ line. On an ROC plot, this line is indicative of a diagnostic test that does not provide a basis for discriminating between cases and controls (Hanley & McNeil, 1982).

Application of ROC curve analysis to plant disease data

Figure 2 The receiver operating characteristic (ROC) curve derived from the frequency distributions of test scores for cases and controls shown in Fig. 1. The point indicated by a solid circle (X) corresponds to the threshold test score set as indicated in Fig. 1. The diagonal line (– – –) is the ‘no discrimination’ line.

There is an important distinction between diagnosis and decision-making as practised by plant pathologists, and as practised by clinicians. For both plant pathologists (see, for example, Grogan, 1981; Miller, 1998) and clinicians, diagnosis is concerned with the problem of disease identification. However, while clinicians treat disease at the same level at which they diagnose it (that is, the individual patient), plant pathologists often diagnose at the level of the individual (the plant), but treat at the level of the population (the crop). In the context of plant disease management decision-making, a ‘diagnostic test’ is not synonymous with an ‘indicator’ of the need for crop protection measures. This raises some 䊚 1999 BSPP


Decision-making and diagnosis in disease management

problems for the use of ROC curve analysis in the evaluation of risk algorithms, which are indicators of the need for treatment at the crop level. Consider the data of Yuen et al. (1996) in this context. The final incidence of Sclerotinia stem rot was estimated from a random sample of 200 plants taken from untreated plots in each of 267 fields, in which various covariates were also recorded. On economic grounds, it was established that treatment was required in order to prevent final disease incidence > 0·2, but not in those with a final disease incidence ⱕ 0·2. In threshold theory (Stern, 1973; Pedigo et al., 1986), a final disease incidence of 0·2 corresponds to the ‘economic injury level’ (EIL). Thus, the fields were divided into two subgroups, depending on whether final disease incidence was above the EIL (these were the cases) or less than or equal to the EIL (these were the controls). Two risk algorithms, based on covariate data, were to be compared. For each algorithm in turn, an indicator score (referred to as the ‘risk point sum’) was calculated for each field, and frequency distributions of indicator scores were compiled separately for the cases and the controls. A comparative evaluation of the two risk algorithms was then carried out by plotting the ROC curve for each. These curves plot the TPP as a function of the FPP at all decision thresholds (Yuen et al., 1996). The term ‘decision threshold’ was used to denote the threshold value of the indicator score (the ‘risk point sum’) and corresponds to the term ‘economic threshold’ (ET) as used in the development of threshold theory (Stern, 1973; Pedigo et al., 1986); the latter terminology is adopted here. The application of ROC curve analysis described by Yuen et al. (1996) does not correspond exactly with the application of ROC curve analysis by clinicians in the evaluation of diagnostic tests, as outlined above. Clinicians have definitive subgroups of cases and controls. In circumstances such as those described by Yuen et al. (1996), plant pathologists have subgroups of cases and controls defined by reference to an EIL. Were the EIL for Sclerotinia stem rot to be increased to a final disease incidence of, say, 0·25, a field with a final disease incidence of 0·22, that previously would have been regarded as a case, would now be regarded as a control. Thus, in disease management decision-making, crops are not classified definitively (as either D þ or D –), but only in relation to the adopted EIL (as either D > Dthresh or D ⱕ Dthresh). The definitions of the various outcomes in the establishment of the risk algorithm are then: TPP is an estimate of Prob(I > Ithresh | D > Dthresh) (read as ‘the probability of an indicator score above the adopted ET, given that disease is above the adopted EIL), and similarly, FNP is an estimate of Prob(I ⱕ Ithresh | D > Dthresh), FPP is an estimate of Prob(I > Ithresh | D ⱕ Dthresh), TNP is an estimate of Prob(I ⱕ Ithresh | D ⱕ Dthresh). The implication of this is that the ROC curves plotted by Yuen et al. (1996) are not complete descriptions of the performance of the risk algorithms they evaluated. They 䊚 1999 BSPP


149

are descriptions of the performance when the EIL is set at a final incidence of 0·2. Since an EIL is established primarily on economic grounds, it may change with, for example, the cost of treatment or the potential value of the crop. A complete description of the performance of a risk algorithm for use in disease management decisionmaking requires that TPP and FPP are calculated by varying Ithresh (i.e. ET) over the whole range of indicator scores (I) and varying Dthresh (i.e. EIL) over the whole range of disease levels (D). This problem was addressed by Murtaugh (1996) in the context of environmental monitoring. In situations like those with which Yuen et al. (1996) dealt, where the response (here, either D > Dthresh or D ⱕ Dthresh) is actually a dichotomization of a continuous variable (here, D), ‘sensitivity and specificity can be thought of as double integrals of the conditional density of the indicator, given the value of the continuous response’ (Murtaugh, 1996). That is to say, an ROC surface, rather than an ROC curve, can be plotted. In effect, such a surface would show a sequence of ROC curves, each curve corresponding to a different choice of Dthresh. When a satisfactory basis for discriminating between cases and controls is provided by an indicator, the advantage of the ROC curve format is that it makes explicit the implications of choosing any particular ET, in terms of the risks involved. Data from an ongoing series of trials devoted principally to forecasting the need for fungicide treatment of wheat to control eyespot disease (caused by Pseudocercosporella herpotrichoides) are presented as a brief illustrative example. These data were collected during 1992–97 from experimental crops in 32 unsprayed plots of winter wheat (cv. Beaver or cv. Riband) and comprise the eyespot index at growth stage (GS) 85 (from which the potential yield loss, and so the need for treatment, may be determined) and disease incidence based on a (visual) assessment of the percentage tillers affected at GS 30/31/32 (which is the indicator). All assessments were made in accordance with Anonymous (1986) and Goulds & Polley (1990). Retrospectively, the experimental crops were divided into subgroups comprising cases and controls on the basis of the eyespot index at GS 85. The level of this index that corresponds to the EIL depends on the cost of treatment and the potential value of the crop, among other things. For this illustration, four different EILs were investigated: eyespot index at GS 85 equal to 20, 30, 40 or 50. For each of these four EILs, frequency distributions of scores for the early assessment of disease incidence were compiled, separately for cases and controls. These are shown in Fig. 3 (a–d). As the EIL increases, more of the 32 crops are classified as controls, rather than cases. For each EIL, an ROC curve was then produced by allowing the ET to vary over the range of observed indicator scores, and calculating the corresponding TPP and FPP values. The four ROC curves in Fig. 4 (a–d) represent slices through an ROC surface. They are shown separately here in order to illustrate the correspondence between the four ROC curves in

150

G. Hughes et al.

Fig. 4 (a–d) and the four EILs by which the crops were classified in Fig. 3 (a–d), respectively. Generally, the ROC curves shown in Fig. 4 indicate that visual assessment of disease incidence at GS 30/31/ 32 was a poor indicator of whether or not a crop required treatment, whichever of the four EILs was used. Only when an EIL of eyespot index at GS 85 of 40 was used (Figs 3c and 4c) did the visual assessment of disease incidence at GS 30/31/32 offer any basis for discriminating between cases and controls. This could be shown more formally by comparing the areas under the ROC curves (see, for example, Hanley & McNeil, 1982). The data set on which this illustration is based is too small to reach any definitive conclusions, either about the general value of early visual assessment of disease incidence as an indicator for use in eyespot disease management decision-making or the particular use of the currently recommended threshold level. However, eyespot assessment in spring has long been known to be an unreliable indicator of subsequent disease development (Scott & Hollins, 1978). In view of changes in fungicides, wheat cultivars and in the pathogen population since the currently recommended threshold was devised (Fitt et al., 1988; Jones, 1994), the results shown in Fig. 4 are perhaps not surprising. On the basis of the data presented here, the currently recommended threshold seems to provide specificity at the expense of sensitivity, resulting in a relatively low FPP but a relatively high FNP (Fig. 4). A high FNP results when the adopted ET is such that treatment tends not to be recommended when in fact it would be justified. Jones (1994, see Table 11) discussed the ‘accuracy’ of the ET for use of fungicides to control eyespot disease. Data from 58 sites were presented. The case and control subgroups were identified on the basis of the increase in yield resulting from prochloraz treatment at GS 30–31. For cases, this increase was ⱖ 0·2 t ha –1; for controls, this increase was < 0·2 t ha –1. The data set comprised 41 cases and 17 controls. Eyespot incidence at GS 30–31 was the indicator, with the ET set, in this example, so that the decision was to treat when ⱖ 20% of tillers were affected, and not to treat when < 20% of tillers were affected. Of the 41 cases, 28 had ⱖ 20% tillers affected (true positives), so sensitivity (TPP) was 0·68. Of the 17

Figure 3 Frequency distributions of cases and controls, determined according to whether the eyespot index at GS 85 (Goulds & Polley, 1990) is greater than the economic injury level (EIL) (the cases) or less than or equal to the EIL (the controls) in each of 32 plots of winter wheat. (a) EIL set at eyespot index ¼ 20; (b) EIL set at eyespot index ¼ 30; (c) EIL set at eyespot index ¼ 40; (d) EIL set at eyespot index ¼ 50. The scale on the ‘frequency’ axis is such that each division represents a single case (above the ‘indicator score’ axis) or control (below the ‘indicator score’ axis). The indicator score is based on an early visual assessment of disease incidence (number of tillers affected out of 25; Anonymous, 1986; Goulds & Polley, 1990). The scale on the ‘indicator score’ axis shows 6–10 tillers affected as ‘10’ and 11–15 tillers affected as ‘15’.

䊚 1999 BSPP



controls, seven had < 20% of tillers affected (true negatives), so specificity (TNP) was 0·41. Jones (1994) calculated accuracy from the proportion of total decisions that were correct, giving 35/58, or 60%. The problem with this calculation is that, because the indicator is more accurate for cases than for controls, the calculation of accuracy depends on the proportions of cases and controls in the data set. Consider the following hypothetical data set. Of 58 sites, 19 were classified as cases and 39 as controls. Of the 19 cases, 13 were correctly identified by the indicator (true positives), so sensitivity (TPP) was 0·68. Of the 39 controls, 16 were correctly identified by the indicator (true negatives), so specificity (TNP) was 0·41. This hypothetical data set thus comprises the same number of sites as the data set of Jones (1994), and has the same TPP and TNP. However, in this case, the accuracy is only 29/58 (50%), because the proportion of cases in the hypothetical data set is lower than in the data set of Jones (1994). Sensitivity and specificity represent two kinds of accuracy, respectively, for cases and controls (see, for example, Johnson et al., 1998). Unlike the calculation of accuracy from the proportion of correct decisions, an ROC curve analysis does not depend on the proportions of cases and controls in a data set, because sensitivity and specificity are independent of these proportions (Metz, 1978).

Discriminant function analysis Discriminant function analysis is another technique used by clinicians in classification and diagnosis that has attracted some attention from plant pathologists (see Hau & Kranz, 1990). Snedecor & Cochran (1967) summarized as follows: ‘With two diseases that are often confused, it is helpful to learn what measurements are most effective in distinguishing between the conditions, how best to combine these measurements, and how successfully the distinction can be made.’ As with ROC curve analysis, a clinician begins by identifying individuals, definitively, as members either of the subgroup comprising cases or the one comprising controls. The discriminant function is constructed, from data comprising the covariates measured on the individuals in both groups, in such a way as to produce as accurate a prediction as possible of the disease status of an individual for whom only the (relevant) covariates have been measured. Ahlers & Hindorf (1987) applied discriminant function analysis in forecasting Sclerotinia stem rot of winter rape. Without going into the same

Figure 4 Receiver operating characteristic (ROC) curves derived from the data shown in Fig. 3. Curves (a)–(d) correspond, respectively, to the frequency distributions of cases and controls shown in Fig. 3 (a–d). On each plot, the point indicated by a solid circle (X) corresponds to a threshold indicator score of five tillers affected out of 25 (i.e. 20%, as recommended; Anonymous, 1986) and the diagonal line (– – –) is the ‘no discrimination’ line.

䊚 1999 BSPP


151

152

G. Hughes et al.

level of detail as with the above discussion of ROC curve analysis, we note that the same problem arises when cases and controls are identified according to whether a population (crop) is above or below an EIL, as in the study by Ahlers & Hindorf (1987), rather than definitively at the level of the individual, as in most clinical studies. The discriminant function as formulated applies only to the adopted EIL. If economic considerations dictate that this EIL should be, say, increased, then some crops that were classified as cases in the data set from which the function was originally formulated may be reclassified into the control subgroup, necessitating reformulation of the discriminant function.

Uncertainty in the classification of cases and controls Classification of cases and controls by reference to an EIL is one way in which the use of ROC curve analysis and discriminant function analysis by plant pathologists in the context of disease management decision-making differs from usage by clinicians; there is another. In the establishment of an ROC curve or a discriminant function, clinicians usually have a method of classifying each individual definitively as either a case or a control. However, in plant pathology, there is uncertainty attached to the classification of crops into the case and control subgroups, because this information is derived from sampling. For example, Yuen et al. (1996) classified fields as cases or controls on the basis of a random sample of 200 plants taken from an untreated plot in each field. Inevitably there is some uncertainty attached to this classification. In fact, it is a relatively simple

Figure 5 A sampling likelihood for the sampling scheme adopted by Yuen et al. (1996). The likelihood shows, for any true disease incidence, the probability that disease incidence assessed by sampling will be less than or equal to the adopted economic injury level (EIL) of 0·2. This probability is calculated from ProbðX ⱕ 40Þ ¼

40 X

ProbðX ¼ x Þ

x ¼0

where there are X diseased plants in a random sample of 200 plants and Prob(X ¼ x) is based on the binomial distribution.

matter to quantify this uncertainty, given the details of the sampling scheme. For any particular EIL adopted, a sampling likelihood can be plotted. This shows the probability of a decision (correct or otherwise) that disease incidence is less than or equal to the adopted EIL, for any actual value of incidence (Fig. 5). Up to and including the EIL, the curve gives the probability of a true negative; above the EIL, the curve gives the probability of a false negative. Typically, such curves show probability values near 1 when the actual incidence is much less than the incidence value adopted as the EIL, near 0 when the actual incidence is much larger than the incidence value adopted as the EIL, and near 0·5 when actual incidence is near the EIL. Further, the probabilities of false positives and true positives are, respectively (1 – the probability of a true negative) and (1 – the probability of a false negative). Thus, when crops are being classified as cases or controls on the basis of information obtained by sampling, there is a chance that some crops will be wrongly classified (as either false negatives or false positives), and this chance is greatest for crops near the adopted EIL. There is further discussion of the problem of evaluation when there is uncertainty in the classification of cases and controls in Zweig & Campbell (1993) and Schulzer (1994).

Diagnosis and decision-making Many new assays are being developed that make possible very accurate and early detection and diagnosis of infection by plant pathogens (e.g. Duncan & Torrance, 1992). These include assays for the causal agent of eyespot of cereals (for example, Poupard et al., 1993; Priestley & Dewey, 1993; Beck et al., 1996; Gac et al., 1996; Nicholson et al., 1997) and for the causal agent of Sclerotinia stem rot of oil seed rape (Jamaux & Spire, 1994). Such assays can contribute to earlier and more detailed diagnoses than were previously possible (for example, Anonymous, 1996), but where a threshold approach to disease management decision-making is adopted, pathogen detection at the level of individual plants is not the only issue. In this context, an indicator of the need for treatment of the crop is required. It is the extent to which such an indicator, together with an appropriate choice of ET, allows discrimination between crops in which the adopted EIL would subsequently be exceeded, and those in which it would not, that is crucial. The proper evaluation of risk algorithms therefore remains the basis for good disease management decision-making.

References Ahlers VD, Hindorf H, 1987 Epidemiologische Untersuchungen u¨ber den Schaderreger Sclerotinia sclerotiorum an Winterraps im Hinblick auf eine Prognose. Nachrichtenblatt des Deutschen Pflanzenschutzdienstes.(Braunschweig) 39, 113–9. Anonymous, 1986 Use of Fungicides and Insecticides on

䊚 1999 BSPP



Cereals. Booklet 2257 (86). Alnwick, UK: Ministry of Agriculture, Fisheries and Food. Anonymous, 1996 Take careful aim at eyespot. Farmers Weekly 125, 55. Beck JJ, Beebe JR, Stewart SJ, Bassin C, Etienne L, 1996 Colorimetric PCR and ELISA diagnostics for the detection of Pseudocercosporella herpotrichoides in field samples. In: Proceedings of the Brighton Crop Protection Conference, Pests and Diseases, 1996, Vol. 1. Farnham, UK: BCPC, 221–6. Duncan JM, Torrance L, eds. 1992. Techniques for the Rapid Detection of Plant Pathogens. Oxford, UK: Blackwell Scientific Publications. Fitt BDL, Goulds A, Polley RW, 1988 Eyespot (Pseudocercosporella herpotrichoides) epidemiology in relation to disease severity and yield loss in winter wheat – a review. Plant Pathology 37, 311–28. Gac ML, Montfort F, Cavelier N, 1996 An assay based on the polymerase chain reaction for the detection of N- and Ltypes of Pseudocercosporella herpotrichoides in wheat. Journal of Phytopathology 144, 513–8. Goulds A, Polley RW, 1990 Assessment of eyespot and other stem base diseases of winter wheat and winter barley. Mycological Research 94, 819–22. Grogan RG, 1981 The science and art of plant disease diagnosis. Annual Review of Phytopathology 19, 333–51. Hanley JA, McNeil BJ, 1982 The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36. Hau B, Kranz J, 1990 Mathematics and statistics for analyses in epidemiology. In: Kranz J, ed. Epidemics of Plant Diseases 2nd edn. Berlin: Springer-Verlag, 12–52. Jamaux I, Spire D, 1994 Development of a polyclonal antibody-based immunoassay for the early detection of Sclerotinia sclerotiorum in rapeseed petals. Plant Pathology 43, 847–62. Johnson DA, Alldredge JR, Hamm PB, 1998 Expansion of potato late blight forecasting models for the Columbia Basin of Washington and Oregon. Plant Disease 82, 642– 5.

䊚 1999 BSPP


153

Jones DR, 1994 Evaluation of fungicides for control of eyespot disease and yield loss relationships in winter wheat. Plant Pathology 43, 831–46. Metz CE, 1978 Basic principles of ROC analysis. Nuclear Medicine 8, 283–98. Miller SA, 1998 Impacts of molecular diagnostic technologies on plant disease management – past, present and future. In: Abstracts – Vol. 1, 7th International Congress of Plant Pathology. International Society for Plant Pathology, Edinburgh, August 1998, 9–16. Murtaugh PA, 1996 The statistical evaluation of ecological indicators. Ecological Applications 6, 132–9. Nicholson P, Rezanoor HN, Simpson DR, Joyce D, 1997 Differentiation and quantification of the cereal eyespot fungi Tapesia yallundae and Tapesia acuformis using a PCR assay. Plant Pathology 46, 842–56. Pedigo LP, Hutchins SH, Higley LG, 1986 Economic injury levels in theory and practice. Annual Review of Entomology 31, 341–68. Poupard P, Simonet P, Cavalier N, Bardin R, 1993 Molecular characterisation of Pseudocercosporella herpotrichoides isolates by amplification of ribosomal DNA internal transcribed spacers. Plant Pathology 42, 873–81. Priestley RA, Dewey FM, 1993 Development of a monoclonal antibody immunoassay for the eyespot pathogen Pseudocercosporella herpotrichoides. Plant Pathology 42, 403–12. Schulzer M, 1994 Diagnostic tests – a statistical review. Muscle and Nerve 17, 815–9. Scott PR, Hollins TW, 1978 Prediction of yield loss due to eyespot in winter wheat. Plant Pathology 27, 125–31. Snedecor GW, Cochran WG, 1967 Statistical Methods 6th edn. Ames, Iowa, USA: Iowa State UP. Stern VM, 1973 Economic thresholds. Annual Review of Entomology 18, 259–80. Yuen J, Twengstro¨m E, Sigvald R, 1996 Calibration and verification of risk algorithms using logistic regression. European Journal of Plant Pathology 102, 847–54. Zweig MH, Campbell G, 1993 Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry 39, 561–77.