Intensive Care Med (2009) 35:616–622 DOI 10.1007/s00134-008-1286-2
Barbara Metnitz Eva Schaden Rui Moreno Jean-Roger Le Gall Peter Bauer Philipp G. H. Metnitz on behalf of the ASDI Study Group
ORIGINAL
Austrian validation and customization of the SAPS 3 Admission Score
R. Moreno Unidade de Cuidados Intensivos Polivalente, Hospital de St Anto´nio dos Capuchos, Centro Hospitalar de Lisboa Central, E.P.E., Al. de St Anto´nio dos Capuchos, Lisbon, Portugal
illness. This was true for both available equations, the General and the Central and Western Europe equation. For this reason a customized country-specific model was develPhilipp G. H. Metnitz, Rui Moreno, Peter oped, using cross-validation Bauer, Barbara Metnitz and Jean-Roger Le Gall are all authors of the main SAPS 3 techniques. This model showed J.-R. Le Gall report [4, 5]. Dept. Re´animation Me´dicale, excellent calibration and discriminaHoˆpital St Louis, Universite´ Paris VII, tion in the whole cohort (Hosmer– Electronic supplementary material Paris, France Lemeshow goodness-of-fit: The online version of this article (doi:10.1007/s00134-008-1286-2) contains Hˆ = 4.50, P = 0.922; Cˆ = 5.61, supplementary material, which is available P = 0.847, aROC, 0.82) as well as in Abstract Objective: To test the to authorized users. the various tested subgroups. Conprognostic performance of the SAPS clusions: The SAPS 3 Admission 3 Admission Score in a regional Score’s general equation can be seen cohort and to empirically test the need and feasibility of regional cus- as a framework for addressing the problem of outcome prediction in the tomization. Design: Prospective general population of adult ICU multicenter cohort study. Patients and setting: Data on a total of 2,060 patients. For benchmarking purposes, patients consecutively admitted to 22 region-specific or country-specific B. Metnitz P. Bauer intensive care units in Austria from equations seem to be necessary in Department of Medical Statistics, October 2, 2006 to February 28, 2007. order to compare ICUs on a similar Medical University of Vienna, level. Vienna, Austria Measurements and results: The database includes basic variables, E. Schaden P. G. H. Metnitz ()) SAPS 3, length-of-stay and outcome Keywords Severity-of-illness Department of Anesthesiology data. The original SAPS 3 Admission Outcome Prognosis and General Intensive Care, Score overestimated hospital mortal- Customization SAPS 3 Medical University of Vienna, ity in Austrian intensive care patients Vienna, Austria e-mail:
[email protected] through all strata of the severity-ofReceived: 28 April 2008 Accepted: 30 July 2008 Published online: 10 October 2008 Springer-Verlag 2008
Introduction Severity-of-illness assessment has been established in intensive care research since the 1980s, when the first scoring systems were introduced in clinical practice [1]. Their aim was to stratify patients by assigning to each patient an increasing score proportional to the severity of
his or her illness. Thus, these systems, by adjusting for the risks of death of individual patients, allow the comparison of groups of patients at a normative level. Without the help of these systems, most modern outcome-related research in intensive care would not be possible or would be highly biased. Any study, be it a randomized controlled trial or a purely observational
617
study, must prove that patients in different arms of the study carry a comparable risk of death—otherwise, no conclusions can be drawn from the differences in the observed outcomes. In addition, the evaluation of the performance of intensive care units (ICUs) and benchmarking has increasingly become a focus of interest. Thus, risk adjustment is the basis of any type of performance measurement in the critical care setting. Recent studies have shown that several problems exist with currently used systems. There is a large variability in the prognostic performance of general severity scoring systems [2, 3]; this variability has several different roots. For example, new treatment options and changing patient cohorts have altered the prognosis of patients in recent decades, and thus the risk-adjustment systems developed several decades ago have become dated. For this reason, a new risk-adjustment system was developed in a worldwide effort, with the participation of more than 300 ICUs. This effort resulted in the development and publication of the SAPS 3 Admission Score [4, 5]. The system was designed to be used on multiple levels: besides a general equation, regional equations for the calculation of predicted hospital mortality were developed, thus allowing ICUs to compare themselves with others on a more common level. The aim of this study was to evaluate the prognostic performance of the SAPS 3 in a regional cohort and to empirically test the need and feasibility of regional customization.
Materials and methods Database
lacked an entry in the field ‘‘hospital outcome’’ (n = 28) were excluded. Patients where a SAPS 3 score was not documented were excluded as well (n = 144). The final study cohort consisted thus of 2,060 patients. Since no additional interventions were performed, the need for informed consent was waived by the institutional review board. Data quality To assess the reliability of data collection, we sent an independent observer to each unit to obtain data from the clinical charts of a random sample of patients. Variancecomponent analyses with the random factors ‘‘units,’’ ‘‘patients within units,’’ and ‘‘observers within units’’ were performed (SAS, procedure varcomp) as described previously [6]. To assess the completeness of the documentation, we calculated the number of missing physiologic parameters. Statistical analysis Statistical analysis was performed using the SAS system, version 9.1 (SAS Institute Inc., Cary, NC, USA). A P value of \0.05 was considered significant. Unless otherwise specified, results are expressed as median and interquartile ranges (quartiles). To evaluate prognostic performance of the different models, the following tools were used: • Observed-to-expected (O/E) mortality ratios: were calculated by dividing the number of observed deaths per group by the number of expected deaths per group (as predicted by the SAPS 3). To test for statistical significance, we calculated 95% confidence intervals (CI) according to the method described by Hosmer and Lemeshow [10]. In a perfect model they should not be different from one. • Calibration: evaluates the degree of correspondence between the estimated probabilities of mortality and the actual mortality in the analyzed sample. The Hosmer– Lemeshow goodness-of-fit Hˆ- and Cˆ-statistics [11] were used for this purpose. • Discrimination: evaluates the capability of the model to distinguish between patients who die from patients who survive. This was tested by measuring the area under the receiver operating characteristic (aROC) curve, as described by Hanley and McNeil [12]. • Calibration curves: are used to graphically compare differences between observed and predicted mortality rates.
Data were collected by the Austrian Centre for Documentation and Quality Assurance in Intensive Care Medicine (ASDI), a nonprofit organization that has established an intensive care database and benchmarking project in Austria [6, 7]. The prospectively collected data included sociodemographic data, such as age, sex and chronic conditions; the reason for admission, which was based on a pre-defined list of medical and surgical diagnoses [8]; severity-of-illness, as measured by the recently published SAPS 3 Admission Score [4, 5]; level of provided care, as measured by the Simplified Therapeutic Intervention Scoring System (TISS-28) [9]; length of ICU and hospital stay; and outcome data, including survival status at ICU discharge and at hospital discharge. Data on all patients consecutively admitted to 22 Austrian ICUs from October 2, 2006 to February 28, 2007, were included in this study. A total of 2,363 patients were A new predictive model was developed, using the admitted to the 22 ICUs during the study period. Patients who were readmitted (n = 122), those who were SAPS 3 Admission Score as the independent variable and \16 years of age (n = 9) and those with records that hospital outcome as the dependent variable. The
618
validation procedure used recursive half-sampling. The total sample was randomly partitioned into two samples (each, n = 1,030). Both samples were comparable with respect to age, length of stay, severity-of-illness, and observed mortality (ESM, Table E1). Alternatively each of the two samples was used to construct a customized model using the original SAPS 3 score type of model function (which has been shown to be stable in the crossvalidation procedure). Validation was done two times in the respective second half of the sample. Results can be found in the ESM, Table E2. To check the results of the half-splitting, cross validation was done through partitioning the whole sample into five equal-sized parts. It was thus possible to run the customization process five times, each time taking four parts of the sample as a development set and the remaining one as the validation set. The results (ESM, Table E3) showed that the same type of function as applied for the SAPS 3 score works sufficiently well with the customized parameter estimates in the five validation samples. Given the good validation results, the final customized Austrian model was then calculated in the total sample and validated for specific subgroups.
Results Demographics A total of 2,060 patients admitted to 22 ICUs were included in the study. Mean age of the patients was 64.9 years, and about 42% were female. Median length of ICU stay was 4 days (2–9). Almost 60% were surgical admissions. The most common comorbidities were chronic renal failure and chronic respiratory failure, each having a prevalence of approximately 10%. Basic demographic data are shown in Table 1.
Table 1 Basic demographic data
Number of patients Age (years; mean ± SD) Female sex ICU length of stay (days) LOD (day 0–1) Type of admission Medical Scheduled surgical Unscheduled surgical Comorbidities Malignant tumor, without metastases Malignant tumor, with metastases Chronic renal failure Chronic respiratory failure Heart failure Alcoholism Insulin-dependent diabetes TISS-28 TISS-28 score per patient TISS-28 score per patient per day SAPS 3 Admission Score SAPS 3 predicted hospital mortality (G) SAPS 3 predicted hospital mortality (C&WE) Observed ICU mortality Observed hospital mortality
N
%
2,060 64.9 ± 16.8
100.0 42.0
4 (2–9) 4 (2–6) 42.5 28.3 29.2 7.6 3.8 10.2 10.8 7.2 3.8 4.2 112 (58–304) 31.3 (26.3–36.0) 52 (40–65) 28.6 25.1 14.5 21.7
Unless otherwise indicated, numbers are given as median and interquartile range LOD Logistic Organ Dysfunction Score
P \ 0.001; Cˆ = 45.61, P \ 0.001). However, in some of the subgroups a good calibration was present, e.g., for planned admissions (Hˆ = 6.87, P = 0.738; Cˆ = 9.08, P = 0.525, Table 2). Discrimination was the same as with the original equation, with an aROC of 0.82. O/E ratio increased with this equation to 0.86 (0.80–0.92). Performance of SAPS 3—Austrian equation
Performance of SAPS 3—General equation Calibration of the original SAPS 3 on all patients was poor (Hˆ = 100.18, P \ 0.001; Cˆ = 90.29, P \ 0.001), and it was not improved by grouping patients according to the type of admission (Table 2). Discrimination was good, with an aROC of 0.82. O/E ratio was with 0.79 (0.74–0.85) similar compared to the original SAPS 3 report [5]. Performance of SAPS 3—Central and Western Europe equation
Since discrimination was not a problem, we opted to use a first-level customization procedure for the Austrian equation, as described previously [13], changing just the logistic coefficients and keeping the relative weights of the variables unchanged. Using the same function of relationship between the SAPS 3 and vital status at hospital discharge as in the original SAPS 3 paper [5] the following equation was derived: LogitðPÞ ¼ 20:2604 þ logðSAPS 3 score þ 0:8601Þ 4:6735ðP being the probability of dying in hospitalÞ
The respective probability of mortality is thus given Calibration of the Central and Western Europe model on all patients was also unsatisfactory (Hˆ = 51.56, by the equation:
619
Table 2 Performance of the SAPS 3 Admission Score N
General equation Hˆ
ASDI cohort 2,060 100.18 Admission type at ICU admission Planned 606 13.05 Unplanned 1,377 91.38 Surgical status at ICU admission Nonoperative 585 27.51 Scheduled 613 36.60 Unscheduled 645 77.80
Central and Western Europe equation
P
Cˆ
P
aROC
Hˆ
P
Cˆ
P
aROC
\0.001
90.29
\0.001
0.82
51.56
\0.001
45.61
\0.001
0.82
0.221 \0.001
14.48 83.21
0.152 \0.001
0.82 0.81
6.87 45.81
0.738 \0.001
9.08 43.29
0.525 \0.001
0.82 0.81
0.002 0.001 \0.001
27.98 27.89 71.30
0.002 0.002 \0.001
0.80 0.83 0.80
18.87 20.94 47.29
0.042 0.022 \0.001
21.06 17.93 41.92
0.021 0.056 \0.001
0.80 0.83 0.79
Hˆ Hosmer–Lemeshow goodness-of-fit Hˆ-test, Cˆ Hosmer–Lemeshow goodness-of-fit Cˆ-test. aROC, area under the receiver operating characteristic curve
Probability of death ¼
elogit 1 þ elogit
The new model showed an excellent calibration on the general cohort (Hˆ = 4.50, P = 0.922; Cˆ = 5.61, P = 0.847) and in the various tested subgroups (Table 3, Fig. 1). Figure 1 shows the different calibration curves from the two different validation sets (each n = 1,030). Discrimination was identical to the performance of the original equation, with an aROC curve of 0.82. Customization of the score changed predicted mortality, and therefore the O/E ratio for the whole cohort increased to 1.00 (0.93–1.07). Moreover, O/E ratios for subgroups ranged after customization from 0.80 to 1.11 (Table 3).
Discussion To our knowledge, this is the first multicentric prospective validation study of the prognostic performance of the SAPS 3 Admission Score. The results of this study are in accordance with data originating from the Austrian participation in the SAPS 3 study [14].
It should be noticed that a low O/E ratio was also present in some other developed countries, such as Switzerland, Australia, The Netherlands, and the territory of Hong Kong [14]. How can we explain this? The SAPS 3 project was aimed at assessing the strong variability that can be found in both, intensive care treatments and outcomes. This variability might potentially be related to factors not addressed in classical severity scoring systems, such as variations in lifestyle and baseline characteristics of the populations. Moreover, major differences in the provision of critical care in very different health care systems might further contribute to this phenomenon. Regional differences were thus expected and were one of the major reasons for the multi-level approach [5], which integrates several different equations and thus allows the use of different factors for comparison at different levels. Two such levels were already integrated into the original report: general and regional equations to calculate the predicted hospital mortality. In the original SAPS 3 study, O/E ratios ranged from 0.84 to 1.30 for the seven defined geographical regions. To explain these differences as a ‘‘lack of performance’’ of the score would, however, be to oversimplify: the differences in outcomes probably reflect unmeasured but very real differences in health care, particularly the
Table 3 Performance of the customized SAPS 3 Admission Score for Austria n ASDI cohort 2,060 Admission type at ICU admission Planned 606 Unplanned 1,377 Surgical status at ICU admission Nonoperative 585 Scheduled 613 Unscheduled 645
H
P
C
P
aROC
O/E
95% CI
4.50
0.922
5.61
0.847
0.82
1.00
0.93–1.07
3.06 3.35
0.980 0.972
5.44 9.10
0.860 0.523
0.81 0.80
0.97 0.99
0.77–1.18 0.92–1.07
10.94 16.06 16.97
0.362 0.098 0.075
16.45 12.28 11.88
0.087 0.267 0.293
0.80 0.83 0.79
1.11 0.80 0.85
0.99–1.22 0.59–1.01 0.74–0.97
Hˆ Hosmer–Lemeshow goodness-of-fit Hˆ-test, Cˆ Hosmer–Lemeshow goodness-of-fit Cˆ-test. aROC, area under the receiver operating characteristic curve
620
500
a
400 300 200 100 0 500
1
2
3
4
5
6
7
8
9
c
400 300 200 100 0 1
2
3
4
5
6
7
8
9
100 90 80 70 60 50 40 30 20 10 0
120
100 90 80 70 60 50 40 30 20 10 0
120
10
b
100 80 60 40 20 0 1
2
3
4
5
6
7
8
9
10
d
100 80 60 40 20 0
1
2
3
4
5
6
7
8
9
10
100 90 80 70 60 50 40 30 20 10 0
100 90 80 70 60 50 40 30 20 10 0
Fig. 1 Comparison of expected and observed hospital mortality in the two respective validation sets (half-splitting). a, c Grouping according to the Hosmer–Lemeshow Hˆ statistics. b, d Grouping according to the Hosmer–Lemeshow Cˆ statistics. The graphs show the calibration of the customized SAPS 3 Admission Score for Austria in the two half-splitted validation sets (each n = 1,030).
Estimated risk of hospital death (grouped in deciles), observed hospital mortality rate, and the corresponding number of patients per decile are shown. Columns number of patients. Squares mean SAPS 3-predicted mortality per decile. Triangles mean observed mortality per decile. GOF goodness-of-fit
provision of intensive care. For example, availability of structures and resources (such as ICU beds and ventilators) and availability of intermediate and other high-care units are all known to affect prognosis [15, 16]. In fact, in some respects the differences between observed and expected mortality rates reflect the match (or mismatch) between required and provided health care. Other factors that are well known to have an effect on prognosis (but can hardly be quantified) are cultural differences—e.g., therapeutic behaviors such as withholding or withdrawing life-sustaining therapies in terminally ill patients [17, 18] and genetic predispositions [19, 20]. To be able to compare ICUs in the same geographical area, regional equations were developed and integrated into the SAPS 3 project. These equations performed well in at least some cohorts, such as for patients in Brasil [21] and Belgium [22]. For our cohort, the Central and Western Europe equation was closer to reality than the original equation but still not accurate enough. The explanation for this can be found through a glance at the composition of the region of ‘‘Central and Western Europe’’ in the original report: it included seven other countries with O/E ratios ranging from 0.7 to 1.5, with only one country in this group exhibiting a lower O/E ratio than Austria [14]. Thus, even regional accumulation of countries which have many things in common does not ensure that outcome is similar. This again highlights the need for regional adaptation of a severity-of-illness system through customization. Customization presents in this
context not a ‘‘last possibility’’ to improve a non-working system, but rather a prerequisite to obtain different views about the same content. Although the development of country-specific equations was planned for the original SAPS 3 report, it was finally not possible because of the lack of representative samples of ICUs and patients within each country. The current study is based on a sample of 22 ICUs in Austria. Although the sample might not be representative (not all geographic regions from Austria are represented), our study cohort seems to be very similar to the general Austrian ICU population. For this reason we used the available data to generate a country-specific equation. Our results show that this undertaking was both possible and reasonable: the excellent fit of the derived Austrian equation in the validation data sets means that the SAPS 3 Admission Score is a valid instrument for use in our cohort of critically ill patients, as it incorporates the correct variables with their appropriate prognostic weights. If the score lacked construct validity, our customization would not have worked: discrimination, calibration and uniformity-of-fit would have remained poor. We have however, to accept that the limited size of our sample did not allow us to look at more subgroups in detail as has been recently shown, samples must be large enough to detect the lack of agreement between predicted and observed mortality rates [23]. It should moreover be noted, that our study presents a single country study and thus our results are not necessarily applicable to other
621
countries. Moreover, the obtained equation is not a static variable: it is a necessary part of the SAPS 3 project to update the equations regularly to ensure that the most accurate equations are used. Finally, with respect to the subject of benchmarking— and thus the comparison of ICUS, it should not be forgotten that O/E ratios have their limitations when they are used for benchmarking purposes [24]. They seem not to be the perfect tools to measure performance, since they tend to average all variability towards a mean. However, as long as we do not have other easy-to-use tools available, O/E ratios seem to be the best alternative. It is thus important to note that external quality comparisons should never be focussed on one single parameter (such as standardized mortality rates). Over the past years many more important quality indicators have been established in the intensive care setting. They include both, process- and outcome-related indicators, such as readmission- and reintubation-rates, device-related infections or appropriate use of therapeutic interventions (e.g., mild hypothermia after cardiac arrest). It is increasingly recognized, that more than one figure is necessary to be able to judge what we call the ‘‘clinical performance’’ of a certain institution.
Conclusion In conclusion, the SAPS 3 Admission Score’s general equation can be seen as a framework for addressing the problem of outcome prediction in the general population of adult ICU patients. For benchmarking purposes, more differentiated levels of comparison are needed. Regional equations, as provided in the original SAPS 3 project are a starting point, but may not be specific enough in some settings. Thus, region-specific or country-specific equations are a necessary approach to further enhance the capabilities and possibilities of the SAPS 3 Admission Score as a benchmarking tool.
We thus propose that the SAPS 3 Admission Score should be further evaluated on a regional basis. Where the regional equation as published in the original report seems not to represent a particular country, the development of a country-specific equation is warranted. This approach is both conceptually and operationally more transparent than the revision of the coefficients for the global database, as it has been done in the past for other systems. Acknowledgments We thank the members of the ASDI study group and their respective study co-ordinators in each ICU: Gu¨nter Sagmu¨ller, UKH Meidling, Vienna; Franz Schwameis, Landesklinikum Thermenregion Baden, Baden; Brigitte Pichler, Landesklinikum Weinviertel Mistelbach/Ga¨nserndorf, Mistelbach; Felix Ernst, Landesklinikum Weinviertel Mistelbach/Ga¨nserndorf, Mistelbach; Thomas Bauer, Landesklinikum Krems, Krems A. D. Donau; Franz Sterrer, A. o¨. LKH Vo¨cklabruck, Vo¨cklabruck; Helmut Trimmel, A. o¨. KH Wiener Neustadt, Wiener Neustadt; Walter Klimscha, Donauspital SMZ Ost, Vienna; Dieter Linemayr, Landesklinikum Mo¨dling, Mo¨dling; Johannes Schuh, A. o¨. KH Wiener Neustadt, Wiener Neustadt; Gabriele Sprinzl, Landesklinikum Donauregion Tulln, Tulln; Kurt Do¨rre, A. o¨. KH Waidhofen a.d. Thaya, Waidhofen A. D. Thaya; Helmut Trimmel, A. o¨. KH Wiener Neustadt, Wiener Neustadt; Gu¨nther Frank, A. o¨. KH der Barmherzigen Bru¨der Eisenstadt, Eisenstadt; Heinz Malle, UKH Klagenfurt, Klagenfurt; Ingrid Schindler, Sozialmedizinisches Zentrum Floridsdorf, Vienna; Sylvia Fitzal, Wilhelminenspital der Stadt Wien, Vienna; Reinhard Schuster, Donauspital SMZ Ost, Vienna; Gottfried Locker, AKH Wien Universita¨tskliniken, Vienna; Helmut Schneller, A. o¨. KH der Barmherzigen Bru¨der Linz, Linz; Hubert Artmann, Kardinal Schwarzenberg’sches Krankenhaus, Schwarzach/Pongau; Oswald Schuberth, A. o¨. LKH Steyr, Steyr; Statistical analysis was supported by a grant from the Fund of the Austrian National Bank, Project # 12690 ONB. Conflict of interest statement The authors declare that there exist no financial or other conflicts of interest. None of the authors has recently been employed or received reimbursements, fees, funding, or salary by/from an organization that may in any way gain or lose financially from the publication of this manuscript. No author holds any stocks or shares or patents in/from an organization that may in any way gain or lose financially from the publication of this manuscript.
References 1. Moreno RP, Metnitz PhGH (2007) Severity of illness scoring systems. Tools for the evaluation of patients and ICUs. In: Parrillo JE, Dellinger RP (eds) Critical care medicine: principles of diagnosis and management in the adult, 3rd edn. Elsevier, Amsterdam 2. Moreno R, Morais P (1997) Outcome prediction in intensive care: results of a prospective, multicentre, Portuguese study. Intensive Care Med 23:177–186
3. Metnitz PG, Valentin A, Vesely H, Alberti C, Lang T, Lenz K, Steltzer H, Hiesmayr M (1999) Prognostic performance and customization of the SAPS II: results of a multicenter Austrian study. Intensive Care Med 25:192–197
4. Metnitz PhGH, Moreno RP, Almeida E, Jordan B, Bauer P, Abizanda-Campos R, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR, on behalf of the SAPS 3 investigators (2005) SAPS 3—from evaluation of the patient to evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description. Intensive Care Med 31:1336–1344
622
5. Moreno RP, Metnitz PhGH, Almeida E, Jordan B, Bauer P, Abizanda-Campos R, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR, on behalf of the SAPS 3 Investigators (2005) SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med 31:1345–1355 6. Metnitz B, Metnitz PhGH, Bauer P, Valentin A on behalf of the ASDI Study Group (2008) Patient volume affects outcome in critically Ill patients. Wien Klin Wochenschr 120 (in press) 7. For further information see also http://www.asdi.ac.at 8. Metnitz PhGH, Steltzer H, Popow C, Valentin A, Lenz K, Neumark J, Sagmu¨ller G, Schwameis F, Urschitz M, Hiesmayr M (1997) Definition and evaluation of a documentation standard for intensive care medicine: the ASDI pilot project. Wien Klin Wochenschr 109:132–138 9. Miranda DR, De Rijk A, Schaufeli W (1996) Simplified therapeutic interventions scoring system: the TISS28 items—results from a multicenter study. Crit Care Med 24:64–73 10. Hosmer DW, Lemeshow S (1995) Confidence interval estimates of an index of quality performance based on logistic regression models. Stat Med 14:2161–2172 11. Lemeshow S, Hosmer DW (1982) A review of goodness-of-fit statistics for use in the development of logistic regression models. Am J Epidemiol 115:92–106
12. Hanley JA, McNeil BJ (1981) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29–36 13. Moreno R, Apolone G (1997) The impact of different customization strategies in the performance of a general severity score. Crit Care Med 25:2001–2008 14. Table E10 of the ESM of the SAPS 3 admission score main report, to download from: http://www.springerlink.com/content/ t78746238246416j/ 15. Miranda DR, Ryan DW, Schaufeli WB, Fidler V (1998) Organization and management of intensive care. In: Vincent JL (ed) Update in intensive care medicine, vol 29. Springer, Berlin 16. Metnitz PhGH, Kopp A, Jordan B, Lang Th (2004) More interventions do not necessarily improve outcome in critically ill patients. Intensive Care Med 30:1586–1593 17. Azoulay E, Jordan B, Sprung C, Timsit JF, Lemaire F, Schlemmer B, Moreno R, Metnitz P (2007) End-of-life practices in intensive care units: worldwide data from the SAPS 3 database. Intensive Care Med 33(Suppl 1):A779 18. Vincent JL (1999) Forgoing life support in western European intensive care units: the results of an ethical questionnaire. Crit Care Med 27:1626– 1633
19. Yende S, Kammerer CM, Angus DC (2006) Genetics and proteomics: deciphering gene association studies in critical illness. Crit Care 10:227 20. Kellum JA, Angus DC (2003) Genetic variation and risk of sepsis. Minerva Anestesiol 69:245–253 21. Soares M, Salluh JI (2006) Validation of the SAPS 3 admission prognostic model in patients with cancer in need of intensive care. Intensive Care Med 32:1839–1844 22. Ledoux, Canivet JL, Preiser JC, Lefrancq J, Damas P (2008) SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 34 (in press) 23. Zhu B-P, Lemeshow S, Hosmer DW, Klarm J, Avrunin J, Teres D (1996) Factors affecting the performance of the models in the mortality probability model and strategies of customization: a simulation study. Crit Care Med 24:57– 63 24. Metnitz PhGH, Vesely H, Valentin A, Lang T, Le Gall JR (2000) Ratios of observed to expected mortality are affected by differences in case mix and quality of care. Intensive Care Med 26:1466–1472
