Regression based quasi-experiment when randomisation is not an option interrupted time series analysis
Evan(gelos) Kontopantelis Centre for Health Informatics Institute of Population Health University of Manchester
Manchester, 6 Sep 2016
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
1 / 22
Outline
1
Background
2
RCT vs ITS
3
Examples
4
Analysis
5
Summary
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
2 / 22
Outline
1
Background
2
RCT vs ITS
3
Examples
4
Analysis
5
Summary
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
3 / 22
Interrupted Time Series ITS
Quasi-experimental approach Longitudinal Observational
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
4 / 22
ITS focus
Downloaded from qualitysafety.bmj.com on September 14, 201
to account for pre-intervention trends
measured improving 2004, and the first y findings o ated impr tion group
Figure 2 Aggregate patient level Quality and Outcomes Framework care and predictions based on the pre-incentivisation trend.
Kontopantelis (Health Informatics)
Interrupted Time Series
Strengths a The main for individ tative sam to certain versally an mised exp entails tha 6 Sep 2016 5 / 22
Outline
1
Background
2
RCT vs ITS
3
Examples
4
Analysis
5
Summary
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
6 / 22
RCTs advantages
Randomisation is key to establish causal inference At the top of the pyramid evidence
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
7 / 22
RCTs disadvantages
Very expensive Possibly unethical Questionable generalisability (external validity) for various reasons Generally investigating short term effects Power to detect large effects and rarely interactions Not always possible, e.g. national policy change
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
8 / 22
ITS advantages
Makes full use of longitudinal nature of data and pre- trends Real populations rather than experimental settings Much cheaper than a trial Best method possible in the absence of randomisation
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
9 / 22
ITS disadvantages
No randomisation ⇒ unmeasured confounding Other external effect(s) Linearity Autocorrelation (seasonality) Best possible approach but no definitive answers
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
10 / 22
Outline
1
Background
2
RCT vs ITS
3
Examples
4
Analysis
5
Summary
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
11 / 22
Non-incentivised aspects of care Sample of 148 representative practices from the CPRD
Figure
Achievement rates improved for most indicators in the pre-incentive period Significant initial gains in incentivised indicators but no gains in later years By 2006-7 achievement rates significantly below those predicted by pre- trends
Mean achievement rate of 148 general practices for quality of grouped by activity and whether they were incentivised unde mean rate is the mean of the adjusted means for the individ RESEARCH BMJ 2011;342:d3590 doi: 10.1136/bmj.d3590
Page 1 of 12
Research
Effect of financial incentives on incentivised and non-incentivised clinical activities: longitudinal analysis of data from the UK Quality and Outcomes Framework Tim Doran clinical research fellow 1, Evangelos Kontopantelis research associate 1, Jose M Valderas clinical lecturer 2, Stephen Campbell senior research fellow 1, Martin Roland professor of health services research 3, Chris Salisbury professor of primary healthcare 4, David Reeves senior research fellow 1 1
National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL, UK; 2NIHR School for Primary Care
Research, Department of Primary Health Care, University of Oxford, Oxford OX3 7LF; 3General Practice and Primary Care Research Unit, University of Cambridge, Cambridge CB2 0SR; 4Academic Unit of Primary Health Care, University of Bristol, Bristol BS8 2AA
Abstract Objective To investigate whether the incentive scheme for UK general practitioners led them to neglect activities not included in the scheme. Design Longitudinal analysis of achievement rates for 42 activities (23 included in incentive scheme, 19 not included) selected from 428 identified indicators of quality of care. Setting 148 general practices in England (653 500 patients). Main outcome measures Achievement rates projected from trends in the pre-incentive period (2000-1 to 2002-3) and actual rates in the first three years of the scheme (2004-5 to 2006-7).
Kontopantelis (Health Informatics)
Interrupted Time Series
Results Achievement rates improved for most indicators in the pre-incentive period. There were significant increases in the rate of improvement in the first year of the incentive scheme (2004-5) for 22 of the 23 incentivised indicators. Achievement for these indicators reached a plateau after 2004-5, but quality of care in 2006-7 remained higher than that predicted by pre-incentive trends for 14 incentivised indicators. There was no overall effect on the rate of improvement for
Introduction Over the past two decades funders and policy makers worldwide have experimented with initiatives to change physicians’ behaviour and improve the quality and efficiency of medical care.1 Success has been mixed, and attention has recently turned to payment mechanism reform, in particular offering direct financial incentives to providers for delivering high quality care.2 In 2004 in the UK the Quality and Outcomes Framework (QOF) was introduced—a mechanism intended to improve quality by linking up to 25% of general practitioners’ income to achievement of publicly reported quality targets for several chronic conditions.3 Should these incentives succeed, the potential benefits for patients with the relevant conditions are considerable.4 Incentives might also improve general organisation of care, benefiting processes and conditions beyond those covered by the incentives.5 Financial incentives have several potential unintended consequences, however. For example, they might
6 Sep 2016
12 / 22
Patient level diabetes care
2000/1 2001/2 2002/3 2003/4 2004/5 2005/6 2006/7 new diagno 44.7 50.4 56.5 65.3 73.4 74.2 74.3 53.9 59.4 71.1 80.9 83 83.2 5-9 years 46.4 51.9 56.8 69.1 78.7 81.4 81.8 10+ years 45.4 50 55.1 66.7 77.6 79.3 80.4
1-4 yearsCPRD 48.4 Sample of 148 representative practices from the
By 2006-7 improvement above trend smaller at 7.3%
90 85
aggregate recorded QOF care score
In 2004-5 quality improved over-and-above this pre-incentive trend by 14.2%
80 75 70 65 60 55 50 45 40
Levels of care varied significantly for sex, age, years of previous care, number of co-morbid conditions
2000/1 2001/2 2002/3 2003/4 2004/5 2005/6 2006/7
new diagnoses
44.7
50.4
56.5
65.3
73.4
74.2
74.3
1-4 years
48.4
53.9
59.4
71.1
80.9
83
83.2
5-9 years
46.4
51.9
56.8
69.1
78.7
81.4
81.8
10+ years
45.4
50
55.1
66.7
77.6
79.3
80.4
Downloaded from qualitysafety.bmj.com on September 13, 2013 - Published by group.bmj.com
ORIGINAL RESEARCH
Recorded quality of primary care for patients with diabetes in England before and after the introduction of a financial incentive scheme: a longitudinal observational study Evangelos Kontopantelis,1 David Reeves,1 Jose M Valderas,2,3 Stephen Campbell,1 Tim Doran1
▸ An additional data is published online only. To view this file please visit the journal online (http://bmjqs.bmj.com) 1
Kontopantelis (Health Informatics)
Health Sciences Primary Care Research Group, University of Manchester, Manchester, UK 2 Health Services and Policy Research Group, NIHR School for Primary Care Research, Department of Primary Health Care, University of Oxford, Oxford, UK 3 European Observatory of Health Systems and Policies, London School of Economics, London, UK
Interrupted Time Series
ABSTRACT Background The UK’s Quality and Outcomes Framework (QOF) was introduced in 2004/5, linking remuneration for general practices to recorded quality of care for chronic conditions, including diabetes mellitus. We assessed the effect of the incentives on recorded quality of care for diabetes patients and its variation by patient and practice characteristics. Methods Using the General Practice Research Database we selected a stratified sample of 148 English general practices in England, contributing data from 2000/1 to 2006/7, and obtained a random sample of 653 500 patients in which
years were more modest. Variation in care between population groups diminished under the incentives, but remained substantial in some cases.
INTRODUCTION In the last 15 years the National Health Service in the UK has undergone a series of reforms aimed at improving the quality of care for people with chronic conditions. These include the creation of the National Institute for Health and Clinical Excellence, and the introduction of National Service Frameworks which set
6 Sep 2016
13 / 22
Withdrawing incentives 644 CPRD practices, 2004-5 to 2011-12
Financial incentives partially removed for aspects of care for patients with asthma, CHD, diabetes, stroke and psychosis Mean levels of performance generally stable after the removal of incentives Health benefits from incentive schemes may be increased by periodically replacing existing indicators with new ones
Fig 2 Trends and predictions for removed and related unremoved scores were compared with back transformed observed scores agree with raw scores fully in most cases, that might not be true RESEARCH scores are prevalent. This can lead to discrepancies due to emp lower score) and an “unfair” comparison between observed and Withdrawing performance indicators: retrospective were also plotted raw scores asunder no comparison with pred analysis of(using general practice performance UK and Outcomes Framework not plotted;Quality vertical lines indicate timing of indicator removal BMJ 2014;348:g330 doi: 10.1136/bmj.g330 (Published 27 January 2014)
Page 1 of 17
Research
OPEN ACCESS
12
13
Evangelos Kontopantelis senior research fellow , David Springate research associate , David 13 4 5 6 Reeves reader , Darren M Ashcroft professor , Jose M Valderas professor , Tim Doran professor NIHR School for Primary Care Research, Centre for Primary Care, Institute of Population Health, University of Manchester, Manchester M13 9PL, UK; 2Centre for Health Informatics, Institute of Population Health, University of Manchester; 3Centre for Biostatistics, Institute of Population Health, University of Manchester; 4Centre for Pharmacoepidemiology and Drug Safety Research, Manchester Pharmacy School, University of Manchester; 5 Institute for Health Services Research, UE Medical School, University of Exeter, Exeter, UK; 6Department of Health Sciences, University of York, York, UK 1
Abstract Objectives To investigate the effect of withdrawing incentives on recorded quality of care, in the context of the UK Quality and Outcomes Framework pay for performance scheme. Design Retrospective longitudinal study. Setting Data for 644 general practices, from 2004/05 to 2011/12, extracted from the Clinical Practice Research Datalink. Participants All patients registered with any of the practices over the study period—13 772 992 in total. Intervention Removal of financial incentives for aspects of care for patients with asthma, coronary heart disease, diabetes, stroke, and psychosis.
Kontopantelis (Health Informatics)
Interrupted Time Series
Main outcome measures Performance on eight clinical quality indicators withdrawn from a national incentive scheme: influenza immunisation (asthma) and lithium treatment monitoring (psychosis), removed in April 2006; blood pressure monitoring (coronary heart disease, diabetes,
Conclusions Following the removal of incentives, levels of performance across a range of clinical activities generally remained stable. This indicates that health benefits from incentive schemes can potentially be increased by periodically replacing existing indicators with new indicators relating to alternative aspects of care. However, all aspects of care investigated remained indirectly or partly incentivised in other indicators, and further work is needed to assess the generalisability of the findings when incentives are fully withdrawn.
Introduction As part of wider efforts to improve the quality and efficiency of healthcare, purchasers worldwide have experimented with linking performance indicators to financial incentives, reputational incentives, or both, within pay for performance and public reporting schemes. As the clinical evidence base and policy priorities change over time, indicator sets must be periodically reviewed and individual indicators modified,
6 Sep 2016
14 / 22
Outline
1
Background
2
RCT vs ITS
3
Examples
4
Analysis
5
Summary
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
15 / 22
Linear vs autocorrelation debatable
Dependent on analysis ‘bin’ Annual bins tend to be linear (although needs to be tested) Analyse using mainstream regressions
Monthly bins tend to be autocorrelated Analyse using more advanced AutoRegressive Integrated Moving Average or similar models Some consider including month as a covariate is enough
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
16 / 22
Mainstream inference generally assuming linearity
Linear, logistic or Poisson regression Basic model only includes three components Can be built up to account for changing population characteristics Or even incorporate ‘control’ factors in analyses
How many time points needed? Simple simulation to detect 2% jump due to intervention Two pre-intervention points: 91.5% power Three pre-intervention points: 96.2% power
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
17 / 22
Mainstream inference
Percentage
esearch Methods & ReportinG generally assuming linearity
100
Pre-trend Pre-trend Observed
Level change (obs-pre) Slope change (obs-pre) Uplift in 2006-07
80 60 40 20
Kontopantelis
* Intervention year
20 06 -0 7
06 20 05 -
20 04 -0 5
20 03 -0 4*
20 02 -0 3
20 01 -0 2
01
0
20 00 -
| Interrupted time es analysis components lation to the Quality Outcomes Framework rvention
Year
certain chronic conditions included in the scheme (for example, diabetes) were published in 2001 or earlier. (Health Informatics) Series This is where the Interrupted strengthTime of the ITS approach lies; to
accounting for the pre slope quantifies the vention and post-in assumption we have vention we set out trend would continue tion period and there cally affecting the tre We collected perfo and coronary heart d four time points: 199 2005 and 2007 (post for the 2009 analysi study.5 We generate and used linear reg allowed us to quanti recorded quality of c est, on top of what wo pre-intervention tren 6 Sep 2016 18 / 22 had an effect on qual
More advanced approaches for inference accounting for autocorrelation
ARIMA models (arima in Stata) Fitting model can be challenging (modelling autocorrelation) Can include intervention as a dummy or other covariates
Unobserved-component models (ucm in Stata) Newer alternative Decomposes time-series into trend, seasonal, other
itsa in Stata, paper available here
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
19 / 22
Outline
1
Background
2
RCT vs ITS
3
Examples
4
Analysis
5
Summary
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
20 / 22
Conclusions
Complex Exogenous effects and other biases Not conclusive But should complement RCT evidence RCTs are not enough especially if real life implementation varies
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
21 / 22
Thank you for listening
Comments and questions:
[email protected]
Kontopantelis (Health Informatics)
Interrupted Time Series
6 Sep 2016
22 / 22
