Medicine, University of Melbourne, and 3Centre for the Study of Clinical Practice, ... Model fit was evaluated using receiver operating characteristic curve i.e. c.
International Journal for Quality in Health Care 1999; Volume 11, Number 1: pp. 29–35
Inter-hospital comparison of mortality rates M. Z. ANSARI1, M. J. ACKLAND1, D. J. JOLLEY2, N. CARSON1 AND I. G. McDONALD3 1
Epidemiology Unit, Health Care Evaluation, Department of Human Services, 2Department of Public Health and Community Medicine, University of Melbourne, and 3Centre for the Study of Clinical Practice, St Vincents Hospital, Melbourne, Victoria, Australia
Abstract Objective. To compare crude and adjusted in-hospital mortality rates after prostatectomy between hospitals using routinely collected hospital discharge data and to illustrate the value and limitations of using comparative mortality rates as a surrogate measure of quality of care. Methods. Mortality rates for non-teaching hospitals (n=21) were compared to a single notional group of teaching hospitals. Patients’ age, disease (comorbidity), length of stay, emergency admission, and hospital location were identified using ICD9-CM coded Victorian hospital morbidity data from public hospitals collected between 1987/88 and 1994/95. Comparisons between hospitals were based on crude and adjusted odds ratios (OR) and 95% confidence intervals (CI) derived using univariate and multivariate logistic regression. Model fit was evaluated using receiver operating characteristic curve i.e. c statistic, Somer’s D, Tau-a, and R2. Results. The overall crude mortality rates between hospitals achieved borderline significance (v2=31.31; d.f.=21; P=0.06); these differences were no longer significant after adjustment (v2=25.68; P=0.21). On crude analysis of mortality rates, four hospitals were initially identified as ‘low’ outlier hospitals; after adjustment, none of these remained outside the 95% CI, whereas a new hospital emerged as a ‘high’ outlier (OR=4.56; P=0.05). The adjusted ORs between hospitals compared to the reference varied from 0.21 to 5.54, ratio=26.38. The model provided a good fit to the data (c=0.89; Somer’s D= 0.78; Tau-a=0.013; R2=0.24). Conclusions. Regression adjustment of routinely collected data on prostatectomy from the Victorian Inpatient Minimum Database reduced variance associated with age and correlates of illness severity. Reduction of confounding in this way is a move in the direction of exploring differences in quality of care between hospitals. Collection of such information over time, together with refinement of data collection would provide indicators of change in quality of care that could be explored in more detail as appropriate in the clinical setting. Keywords: administrative database, comorbidities, in-hospital death, inter-hospital comparison, quality of care, severity adjustment
In an atmosphere of cost-containment with demands on providers for increased accountability, any valid information which may be relevant to quality of care in hospital practice is valuable, especially if it can be readily collected. In-hospital mortality for a surgical procedure determined from discharge data is clearly important information; interpreted cautiously, such mortality can sometimes be attributed to differences related to quality of care [1,2]. Some correction for differences for case-mix must be made [3–6] but even then, evidence that the residual
variation is related to quality of care is conflicting [7,8]. The Victorian Inpatient Minimum Database (VIMD) provides routinely collected hospital discharge data. If a regression model is used to make the necessary adjustments for differences in illness severity and length of stay, outlier hospitals could be tracked over time and their quality of care explored in more detail as required. The first step, however, is to demonstrate the effect on crude death rates of adjustment for variables known to influence hospital mortality. This is the purpose of this study.
Address correspondence to M. Z. Ansari, Senior Clinical Epidemiologist, Epidemiology Unit, Health Care Evaluation, Department of Human Services, 18/120 Spencer Street, Melbourne 3001, Victoria, Australia. Tel: +61 3 96374244. Fax: +61 3 96374251. 1999 International Society for Quality in Health Care and Oxford University Press
29
M. Z. Ansari et al.
Table 1 Patient and hospital characteristics used for adjusting logistic regression analysis Characteristic Categories ....................................................................................................................................................................................... Age distribution (years) 40–60, 61–65, 66–70, 71–75, 76–80, >80 Length of stay (weeks) 1, 2, 3, 4, >4 Type of admission Planned, emergency Type of prostatectomy Transurethral, open Location of hospital Rural, metropolitan Cancer of prostate ICD-9-CM (1850) Other malignancy (except prostate) ICD-9-CM (1400–1840, 1860–2090) Cardiovascular disease ICD-9-CM (3900–4590) Respiratory illness ICD-9-CM (4600–5190) Endocrine disorders (including diabetes) ICD-9-CM (2400–2790)
Patients and methods The VIMD records demographic and other details of all public hospital inpatient separations for the State of Victoria [9]. Clinical data are stored as ICD-9-CM codes in diagnosis and procedure fields in the VIMD [10]. Patients who had undergone prostatectomy between 1987/88 and 1994/95 were identified according to the relevant procedures for prostatectomy: transurethral resection of prostate (ICD-9CM code 60.2), suprapubic (60.3), retropubic (60.4), and radical (60.5). The presence or absence of comorbid illnesses or disease at the time of hospitalization were determined by the presence of specific ICD-9-CM codes within any of the first five diagnosis fields of the VIMD. Covariables used to control for severity of illness were patients’ age, disease status (presence of prostatic or other malignancies, other serious comorbid illness or not), and admission (location of hospital, emergency admission or otherwise, length of stay) (Table 1). Length of stay is an important covariate and has been used as a surrogate global measure of disease severity [11]. With the exception of age and length of stay, which were grouped into discrete ranges, the patient and hospital covariables were coded into categories indicating the presence or absence of the factor. Mortality was defined as in-hospital death associated with prostatectomy. Hospitals performing 40 operations or more (n= 36) were selected for further analysis from the VIMD. Fifteen were teaching hospitals identified according to the groupings of the hospitals by the Department of Human Services: these were aggregated into a single notional hospital category and served as the comparison group in multivariate analysis. The remaining (n= 21) were Regional General and Area hospitals with 500 or more episodes of care per annum. Analytic methods Statistical analysis was performed using the Statistical Analysis System software (SAS, 6.10 release). The outcome (dependent) variable was defined as the total number of inpatient deaths divided by the total admissions for prostatectomy. Patient level logistic regression analysis was used to compare
30
death rates between hospitals, an appropriate method for binomial outcomes [12–13]. Details of coding of the variables used in the logistic model are shown in the appendix. Odds ratios (ORs) for death for each hospital versus a reference were based on maximum likelihood techniques using the SAS procedure LOGISTIC [14]. Identification of high and low outlier hospitals compared to the reference was based on a 0.05 significance level. Two models were used for comparing mortality rates across hospitals. Model 1 described the variation in mortality rates between hospitals based on univariate logistic regression (crude OR) i.e. without accounting for patient and hospital characteristics. Because we wished to examine the variation in mortality rates across hospitals after adjusting for differences in case-mix, we report model 2 where we added age, measures for severity of illness, and hospital characteristics (Table 2) into the logistic model, and then assessed the variation in mortality rates across hospitals (adjusted OR). Adjusted ORs were obtained by exponentiating appropriate coefficients in the fitted model. An asymptotic 95% confidence interval (CI) for the OR was constructed by exponentiating the limit of ±1.96 standard errors about the coefficient. As the usual binomial model does not account for inter-hospital variability, a parameter of overdispersion was estimated by the deviance divided by the degrees of freedom [12–14]. As the factor of overdispersion was less than one, we are therefore reporting the results of a fixed effect logistic model. The predictive accuracy of the logistic model was assessed using the three rank correlation indexes (‘c’ statistic, Somer’s D, Tau-a) and R2 [15–21]. The ‘c’ statistic is the area under the receiver operating characteristics (ROC) curve, and it represents the probability that a randomly selected pair of subjects of opposite status (survivors/non-survivors) are correctly rated [15–17]. ‘c’ has a maximum value of 1.0 (perfect prediction). Somer’s D is mathematically equal to ‘c’ but rescaled to lie between –1.0 and +1.0 [18]. Tau-a is also similar to ‘c’ except that the randomly selected pair of subjects may have the same status (survivors/non-survivors) [19]. R2 is the adjusted generalized coefficient of determination obtained from PROC LOGISTIC [20–21]. The statistical
Inter-hospital mortality comparison
Table 2 Comparison of prostatectomy mortality rates between hospitals: adjusted odds ratios (OR) and 95% confidence intervals (CI) Crude mortality Adjusted OR,1,2 P rate (%) (95% CI) ............................................................................................................................................................................................................................. A 114 1 0.87 5.54 (0.73–41.7) 0.09 B 662 9 1.35 1.04 (0.51–2.10) 0.90 C 519 3 0.57 2.65 (0.80–8.72) 0.10 D 868 2 0.23 0.40 (0.10–1.67) 0.21 E 886 1 0.11 0.23 (0.03–1.67) 0.14 F 1201 8 0.66 1.11 (0.53–2.32) 0.77 G 1295 5 0.38 0.47 (0.40–1.30) 0.15 H 1560 7 0.44 0.55 (0.22–1.34) 0.18 I 425 5 1.17 1.79 (0.63–5.09) 0.27 J 301 2 0.66 0.81 (0.18–3.63) 0.78 K 480 2 0.41 0.60 (0.13–2.68) 0.51 L 954 6 0.62 1.04 (0.40–2.70) 0.92 M 571 1 0.17 0.21 (0.03–1.66) 0.14 N 234 1 0.42 1.04 (0.13–8.03) 0.96 O 1002 10 0.99 1.33 (0.60–2.99) 0.48 P 515 2 0.38 1.20 (0.27–5.35) 0.80 Q3 154 2 1.29 4.56 (1.00–20.67) 0.05 R 476 3 0.63 1.25 (0.36–4.38) 0.72 S 424 1 0.23 0.21 (0.03–36.27) 0.14 T 1015 8 0.78 1.07 (0.51–2.27) 0.84 U 534 4 0.74 1.22 (0.40–3.72) 0.71 Hospitals
Study size
Deaths (n)
1
Adjusted for patient and hospital characteristics in the logistic model. Reference=teaching hospitals. 3 High outlier hospital. 2
significance of the differences in the fit of the logistic models, i.e. their effectiveness in describing mortality, was assessed with the likelihood ratio v2.
Results Comparison of death rates for prostatectomy for hospitals A through U based on univariate logistic regression are shown in Figure 1. Before adjusting for age, severity of illness and hospital characteristics, the OR (crude) for each hospital varied from 0.11 to 1.32, i.e. the hospital (B) with the highest proportion of operative deaths experienced rates 12 times higher than the hospital (E) with the lowest rate. The variation in crude mortality rates between hospitals achieved a borderline significance (v2=31.31; d.f.=21; P=0.06). After adjustment, although the OR varied from 0.21 (hospital S) to 5.54 (hospital A), ratio=26.38, the difference in mortality rates between hospitals was not statistically significant (v2= 25.68; P=0.21). The crude ORs of deaths in two hospitals (A and Q) were close to reference (OR=0.84 and OR=1.26, respectively). Adjustment increased both ORs dramatically (OR=5.54 and OR=4.56, respectively) with one (Q) remaining outside the 95% CI (Table 2).
Figure 1 Comparison of mortality after prostatectomy between hospitals: crude odds ratio and 95% confidence intervals.
31
M. Z. Ansari et al.
Table 3 Ranking of hospitals based on odds ratio before and after adjustment for severity of illness Rank 11 based on Rank 21 based on crude odds ratio adjusted odds ratio ............................................................................................................ A 17 21 B 21 9 C 10 19 D 3 4 E 1 3 F 14 13 G 5 5 H 9 6 I 20 18 J 13 8 K 7 7 L 11 11 M 2 2 N 8 10 O 18 17 P 6 14 Q 19 20 R 12 16 S 4 1 T 16 12 U 15 15 Hospital
Figure 2 The impact of adjustment for severity of illness on death rates after prostatectomy in hospitals: a comparison of crude and adjusted odds ratios.
Based on univariate logistic regression, four hospitals (D, E, G and H) were identified as low outliers on the basis of crude mortality figures (Figure 1). After adjusting for hospital and patient characteristics in the logistic model, none of these hospitals remained outside the 95% CI (Table 2). Two hospitals (A and C) had high adjusted mortality rates with wide 95% CIs and P values of 0.09 and 0.1, respectively. The impact of adjusting death rates for severity of illness for each hospital is illustrated in Figure 2. The ranking of the hospitals based on OR before and after adjustment for severity of illness is shown in Table 3. The model with age, measures of severity of illness and hospital characteristics provided a better fit compared to the basic model (likelihood ratio v2=1990.78; d.f.=16; P1.0 and were significantly different from the reference. Correction for comorbidity from discharge data is relatively blunt and the relationship between higher grades of severity of illness and death is likely to be non-linear. Because an administrative database, rather than clinical information, is the main source of data it creates problems in assessing comorbidities. We did not have information on some useful clinical variables such as prostate size. In addition, it is known that comorbidities are under-recorded in hospital discharge data [21], particularly for surgical admission [29]. A recent analysis of coding errors in the VIMD supports the fact that the majority of diagnosis coding errors are omissions of codes for comorbid conditions [30]. Under-recording of comorbidities in hospital discharge data limits the effectiveness of statistical methods for eliminating case-mix bias [21]. For these reasons, caution should be exercised in interpreting residual variance after adjustment to differences in quality of care. The finding of an outlier should rather be
seen as a ‘signal’ suggesting the need for a more detailed clinical study. We have restricted our analysis to well-defined patient groups (all patients undergoing prostatectomy) instead of looking at overall mortality experience of the hospital which comprises a variety of patient subpopulations subject to distinctive risks of death. As with any composite index, the use of total in-hospital mortality can conceal excess mortality in one specific group if this was compensated by unusually low mortality among patients with other diseases. A better perspective on quality of care and performance can be assessed by restricting analysis to specific groups of patients. This will further control for differences in case-mix compared to using total in-hospital mortality rates. Ascertaining condition-specific mortality rates over long periods will provide a better way to screen hospitals with ‘unexpectedly’ high numbers of deaths compared to mortality rates either on different conditions or on all conditions [31]. However, the main limitation of restricting mortality analysis to a specific condition or a procedure is the availability of only small numbers of cases. The wide 95% CI for some hospitals observed in this study highlights this problem. Inter-hospital comparison of death rates for specific conditions may therefore be used only for large hospitals with high throughput of major procedures (or medical conditions). The aggregation of data over many years for small hospitals, as an alternative approach, may delay timely action for improving the quality of care. We have restricted our analysis to in-hospital deaths after prostatectomy. This is because our data only include events that occurred during the hospital stay. In-hospital deaths have been used by other investigators in comparing mortality rates between hospitals [32–33]. HCFA [34–35] included, in their report, all deaths that occurred within 30 days of the last admission during the calendar year because of their concern about the effect of differences in length of stay on in-hospital mortality. HCFA was criticized for this approach, as it is unclear if the hospitals can be held accountable for all deaths occurring after discharge. We have accounted for the variation that may occur due to differing length of stay by adjusting for this variable in the logistic model. As mortality rates for several hospitals are tested, the statistical issues of multiple testing may make interpretation of the data more difficult. There may be a higher probability of generating a false-positive result due to random error than the stated level of significance for individual comparisons [36]. A variety of methods exists for correction of P values but such corrections may reduce the statistical power [37–38]. Policy makers should therefore be careful in making judgements on quality of care based purely on the strength of evidence provided by the P values. Assuming that data collection and analysis are optimal, there are two ways in which information might be refined. Adding information relevant to severity of illness collected on admission and after initial treatment is useful in predicting subsequent morbidity and mortality. To what extent it will prove cost-effective to collect and process such information remains an unanswered question. Another way of improving
33
M. Z. Ansari et al.
the accuracy of discharge data such as ours is to analyse data cumulatively over time. Just as collecting in-hospital data on clinical indicators benefits from a progressive narrowing of CIs over time [39], so too would trends become apparent in discharge data over time. This would allow more confident identification of atypical performance as a statistical outlier. Hence routinely available hospital discharge data are valuable for flagging quality of care problems for procedures in specific hospitals. An atypical result for a hospital, or a puzzling trend in several could be the signal to explore locally and in more depth. Such an exploration would involve more accurate determination of severity of illness in the individual patient preferably using a combination of objective scales and implicit clinical judgements. Such a detailed prospective evaluation could also be extended to assessment of ‘preventability’ of serious morbidity and of death. This could be undertaken by a multidisciplinary research team comprising experienced clinicians, epidemiologists, quality assurance experts and hospital administrators. The quality assurance loop could then be closed when shortcomings were addressed locally with the assistance of those clinicians who had helped to expose them.
References 1. McKee M. Routine data: a resource for clinical audit? Qual Health Care 1993; 2: 104–111. 2. Roos LL, Roos NP, Sharp SM. Monitoring adverse outcomes of surgery using administrative data. Health Care Financ Rev 1987; Suppl.: 5–16. 3. Rubin HR, Wu AW. The risk of adjustment. Med Care 1992; 30: 973–975. 4. Iezzoni LI, Ash AS, Shwartz M et al. Judging hospitals by severity adjusted mortality rates: the influence of the severityadjustment method. Am J Publ Health 1996; 86: 1379–1387. 5. Landon B, Iezzoni LI, Ash AS et al. Judging hospitals by severity adjusted mortality rates: the case of CABG surgery. Inquiry 1996; 33: 155–166. 6. Iezzoni LI, Ash AS, Shwartz M, Mackiernan YD. Differences in procedure use, in-hospital mortality, and illness severity by gender for acute myocardial infarction patients: are answers affected by data source and severity measure? Med Care 1997; 35: 158–171. 7. Jencks S, Dobson A. Refining casemix adjustment: the research evidence. N Engl J Med 1987; 317: 679–686.
11. Roemer MI, Moustafa AT, Hopkins CE. A proposed hospital quality index: hospital death rates adjusted for case severity. Health Serv Res 1968; 3: 96–118. 12. Berry G, Mandryk J, Mock P. Presentation of results of logistic regression: analyses of cohort and cross-sectional studies. In Francis IS, Manly BJF, Lam FC, eds, Proceedings of Pacific Statistical Conference, Auckland, New Zealand, 1985. Amsterdam: Elsevier, 1986: 79–82. 13. Breslow NE, Day NE (eds) Statistical Methods in Cancer Research. Vol. 1. The Analysis of Case Control Studies (IARC Scientific Publication No. 32). Lyon: International Agency for Research on Cancer, 1980. 14. The LOGISTIC Procedure. SAS Inc., SAS/STAT Software: Changes and Enhancements, Release 6.10. Cary, NC: SAS Institute Inc., 1994: 41–60. 15. Green J, Wintfield N, Sharkey P, Passman LJ. The importance of severity of illness in assessing hospital mortality. J Am Med Assoc 1990; 263: 241–246. 16. Erdreich LS, Lee ET. Use of relative operating characteristic analysis in epidemiology. Am J Epidemiol 1981; 114: 649–662. 17. Haneley JA, McNeil BJ. The meaning and use of the area under a receiver-operating characteristic (ROC) curve. Radiology 1982; 143: 29–36. 18. Ross LL, Sharp SM, Cohen MM, Wajda A. Risk adjustment in claims-based research: the search for efficient approaches. J Clin Epidemiol 1989; 42: 1193–1206. 19. Siegel S, Castellan Jr NJ. Nonparametric Statistics for the Behavioral Sciences. New York, NY: McGraw Hill, 1988. 20. Nagelkerke NJD. A note on a general definition of the coefficient of determination. Biometrika 1991; 78: 691–692. 21. Jencks SF, Daley J, Draper D et al. Interpreting hospital mortality data: the role of clinical risk adjustment. J Am Med Assoc 1988; 260: 3611–3616. 22. Gonella JS, Goran MJ. Quality of patient care – a measurement of change: the staging concept. Med Care 1975; 13: 467–473. 23. Knaus WA, Zimmerman JE, Wagner DP et al. APACHE – acute physiology and chronic health evaluation: a physiologically based classification system. Crit Care Med 1981; 9: 591–597. 24. Horn SD. Measuring severity of illness: comparisons across institutions. Am J Pub Health 1983; 73: 25–31. 25. Brewster AC, Karlin BG, Hyde LA et al. MEDISGRPS: a clinically based approach to classifying hospital patients at admission. Inquiry 1985; 22: 377–387.
8. Ash AS. Identifying poor-quality hospitals with mortality rates: often there’s more noise than signal (Editorial). Med Care 1996; 34: 735–736.
26. Charlson ME, Pompei P, Ales KL, McKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis 1987; 40: 373–383.
9. The Victorian Inpatient Minimum Database: An Overview. Acute Health Services, Health and Community Services, Melbourne, Victoria, April 1994.
27. Hoore WD, Sicotte C, Tilquin C. Risk adjustment in outcome assessment: the Charlson Comorbidity Index. Meth Inform Med 1993; 32: 382–387.
10. ICD-9-CM. The Official NCC Australian Version. Sydney: National Coding Centre, Faculty of Health Sciences, University of Sydney, 1995.
28. Krakauer H, Bailey RC, Skellan KJ et al. Evaluation of the HCFA model for the analysis of mortality following hospitalization. Health Serv Res 1992; 27: 317–335.
34
Inter-hospital mortality comparison
29. Hargreave TB, Heynes CF, Kendrick SW et al. Mortality after transurethral and open prostatectomy in Scotland. Br J Urol 1996; 77: 547–553. 30. MacIntyre CR, Ackland MJ, Chandraraj E. Accuracy of injury coding in Victorian hospital morbidity data. Aust N Z J Publ Health 1997; 21: 779–182. 31. Chassin MR, Park RE, Lohr KN et al. Differences among hospitals in Medicare patient mortality. Health Serv Res 1989; 24: 1–31. 32. DesHarnais SI, Chesney JD, Wroblewski RT et al. The RiskAdjusted Mortality Index: A new measure of hospital performance. Med Care 1988; 26: 1129–1148. 33. Jencks SF, Williams DK, Kay TL. Assessing hospital associated deaths from discharge data: The role of length of stay and comorbidities. J Am Med Assoc 1988; 260: 2240–2246. 34. Medicare Hospital Mortality Information. Washington, DC: Health Care Financing Administration; 1987: publication 01–002. 35. Medicare Hospital Mortality Information. Washington, DC: Health Care Financing Administration; 1988: publication 00641. 36. Savitz DA, Olshan AF. Multiple comparisons and related issues in the interpretation of epidemiologic data. Am J Epidemiol 1995; 142: 904–908. 37. Cole P. The evolving case-control study. J Chron Dis 1979; 32: 15–27. 38. Jones DR, Rushton L. Simultaneous inference in epidemiological studies. Int J Epidemiol 1982; 11: 276–282. 39. Ansari MZ, Collopy BT, McDonald IG. Establishing thresholds for adverse patient outcomes. Int J Qual Health Care 1996; 8: 223–230.
Appendix
x1=x2=...=x21=0 for all teaching hospitals (reference) Age in years=x22,..., x26 x22=1 for age 61–65 years x23=1 for age 66–70 years x24=1 for age 71–75 years x25=1 for age 76–80 years x26=1 for age 81 years and over x22=x23=x24=x25= x26=0 for age 40–60 years (reference) Length of stay=x27... x29 x27=2 weeks x28=3 weeks x29=4 weeks x27=x28=x29 =0 for length of stay 1 week (reference) Cardiovascular disease=x30 0=absent (reference), 1=present Respiratory disease=x31 0=absent (reference), 1=present Endocrine disorders including diabetes=x32 0=absent (reference), 1=present Prostatic malignancy=x33 0=absent (reference), 1=present Other malignancy=x34 0=absent (reference), 1=present Type of admission=x35 0=planned (reference), 1=emergency
Coding of the variables used in the logistic model
Type of operation=x36 0=transurethral (reference), 1=open
Death status=y 0=alive (reference), 1=died
Location of hospital=x37 0=rural (reference), 1=metropolitan
Hospital=x1,...,x21 x1=1 for hospital A, x2=1 for hospital B,..., x21=1 for hospital U;
Accepted for publication 1 September 1998
35
