Monitoring obstetricians' performance with statistical process control ...

DOI: 10.1111/j.1471-0528.2007.01270.x

General obstetrics

www.blackwellpublishing.com/bjog

Monitoring obstetricians’ performance with statistical process control charts S Lane,a A Weeks,b H Scholefield,c Z Alfirevicb a Centre for Medical Statistics and Health Evaluation and b School of Reproductive and Developmental Medicine, University of Liverpool, Liverpool, UK c Liverpool Women’s NHS Foundation Trust, Liverpool, UK Correspondence: Dr S Lane, Centre for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool L69 3GS, UK. Email [email protected]

Accepted 19 December 2006. Published OnlineEarly 12 March 2007.

Objective The main objective of this study was to pave the way

towards proactive, continuous assessment of individuals and hospitals by demonstrating the application of evidence-based competency standards in maternity care using statistical performance monitoring. Design Retrospective study using data routinely collected by a

large maternity hospital.

Main outcome measures The recorded complication rates for aminocentesis and ventouse delivery. Results The SPC charts identified significant variation in complication rates within the team and showed the ways in which prospective data can be used to provide continuous feedback to individuals on their performance. Conclusion The study shows that statistical performance

Setting A large teaching hospital. Population Clinicians who routinely perform either amniocentesis

or ventouse deliveries. Method As a ‘proof of principle’, we have used statistical process

control (SPC) charts to compare the observed complication rates for amniocentesis and ventouse delivery with the expected complication rates based on published data.

monitoring and, in particular, the use of control charts can be a valuable tool in the continuous assessment of individuals and the healthcare service being provided. The control charts provide a more immediate indication of current performance and provide an alternative to performance-based league tables for the presentation of yearly performance data. Keywords Amniocentesis, control charts, performance monitoring,

statistical process control, ventouse delivery.

Please cite this paper as: Lane S, Weeks A, Scholefield H, Alfirevic Z. Monitoring obstetricians’ performance with statistical process control charts. BJOG 2007;114:614–618.

Introduction Several recent high-profile medical tragedies in the UK, including the crimes of Doctor Harold Shipman, have shaken public confidence in self-regulation of the medical profession.1 In obstetrics and gynaecology, the Ritchie Report2 on Rodney Ledward recommended that auditing of performance should be a core requirement for clinicians. It has been suggested3 that the analysis of routine hospital data could have raised concerns about his practice at an earlier stage. The chief medical officer has published his recommendations on appraisal and revalidation as a result of these, and the onus has been put on doctors to prove their competence.4 Monitoring the performance of individuals within a hospital is a particularly difficult aspect of any revalidation process. There have been a variety of methods suggested for monitoring, but each method has problems. Simple comparisons of

614

adverse outcomes and complications can be misleading and may encourage clinicians to practice risk-averse behaviour.5 Quality assessment based on the prospective evaluation of performance data is well established in the manufacturing industry. Shewhart first developed statistical process control (SPC) and the use of control charts in the 1920s.6 The control chart is a simple graphical representation of the current status of the process that includes warning and action lines set at 2 and 3 standard errors around the expected target value. If a single observation lies outside the action limit or two successive observations lie outside the warning limit, some form of adjustment is required.6 There have been a number of reported applications of SPC in medical settings.7–12 In most, the statistical assessments are retrospective and institution based. However, there is an increasing need for analyses that use data as soon as they become available. Here, the objective is to detect changes in

ª 2007 The Authors Journal compilation ª RCOG 2007 BJOG An International Journal of Obstetrics and Gynaecology

Performance monitoring

performance patterns as quickly as possible and allow remedial action to be taken before damage is caused. Several studies13–15 have suggested that control charts may be less threatening to healthcare providers than hospital league tables, as they place less emphasis on ordering and more easily understood by patients and auditors. As a ‘proof of principle’, we set out to compile the control charts for two relatively common obstetric procedures, amniocentesis and ventouse delivery, for which data have already been collected routinely. Instrumental delivery is a core competency skill for trainees in obstetrics and gynaecology. Ensuring that this competency is achieved and maintained remains a challenge for both trainees and trainers. Failure to deliver the baby with the ventouse is a relatively common complication and is associated with an increase in adverse perinatal outcome. Amniocentesis, on the other hand, is a procedure usually conducted by fetal medicine specialists. Earlier versions of the Royal College of Obstetricians and Gynaecology guidelines on amniocentesis recommend a minimum of 30 procedures per annum to ensure competency, but the latest version highlighted internal audit of complications as the best way to assess competency.16 The multiple needle insertion (MNI) rate was selected, as this is the most commonly recorded complication associated with adverse outcomes.

visor. A ‘failure’ was defined as any procedure when the ventouse was initially used but the final delivery was achieved through the use of forceps or caesarean section. Data relating to MNIs and the number of procedures performed were obtained from Liverpool Women’s Hospital Fetal Centre Database. Seven clinicians performed the procedure without supervision for the years 2001–05. The continuing performance of the individual operators for amniocentesis was plotted as a moving overlapping window of 20 consecutive procedures, and each circle represents the MNI rate for each consecutive window. In this way, the monitoring chart can be updated every time a new procedure is performed by plotting the complication rate for the most recent 20 procedures. Poor performance because of inexperience when the operator first begins to perform the procedure is eliminated from the monitoring statistic, and as time progresses, the operator becomes more proficient. Because of a relatively small number of ventouse deliveries performed by each operator, it was not feasible to use a moving window of 20 procedures, as for amniocentesis. The use of smaller windows produced warning and action limits that were very wide allowing a failure rate of more than 50% before poor performance was detected. To provide more sensitive control limits, the cumulative performance of the individual operators was plotted over the year.

Methods Results Ventouse delivery It was decided to focus on the clinicians who had undertaken ten or more procedures during 2004 because of the wide confidence limits associated with less than ten procedures. There were 278 procedures conducted by 16 doctors (10 specialist registrars [SpRs] in year 1–3, 4 SpRs in year 4+ and 2 senior house officers). The overall failure rate was 26%. The SPC charts for monitoring failed ventouse procedures (Figure 1) show that two clinicians were performing outside acceptable limits; two SpR 1–3 clinicians who performed 60 B

50

Failure rate (%)

The ‘best-practice’ complication rate was identified from a systematic review of the literature. We searched the Medline and PubMed databases using the keywords ‘instrumental delivery’, ‘ventouse’ and ‘amniocentesis’ and restricted the search to articles published from 2000 onwards. Two articles were identified that gave data relating to failed ventouse deliveries from at least 100 consecutive procedures. One article17 reported 81 (20%) failures from a total of 397 procedures, and a second18 article reported 100 (25%) failures from 404 procedures. The overall failure rate was therefore set at 22%. One relevant amniocentesis article was found19 that reported 3571 (97%) single insertions from a total sample size of 3681, with a MNI rate of 3%. The control charts for monitoring the performance of the individual operators were constructed using funnel plots.14 The control limits were constructed from the expected failure rate and the number of procedures performed by each operator using the Wilson ‘exact’ confidence interval.20 As it is important to detect potential outliers as quickly as possible, it was decided to set the warning limit at the 90% confidence level and the action limit at the 95% confidence level. A year’s data on individuals’ ventouse failure rates were retrieved from the hospital’s computerised databases. A ventouse delivery was recorded under an individual’s name whenever they were involved in a ventouse delivery, irrespective of whether they were the primary operator or the super-

40 30 20

A

10 0 10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

Ventouse procedures

Figure 1. Individual failure rates of ventouse deliveries. The solid horizontal line is the expected failure rate and the dashed lines represent the warning and action performance limits. Two clinicians are identified as being above the expected range.


615

Lane et al.

Amniocentesis There were 1176 amniocenteses performed over the 5-year period with an overall MNI rate of 4.5%. The individual results can be seen in the SPC chart (Figure 3). The performance of one operator was identified as being outside the acceptable control limits. This operator performed 128 procedures with 21 multiple insertions (16.4%), indicating that further investigation should be undertaken to ascertain the cause

D

16 14 12 10 8 6

C

4 2 0 50

100

250

300

of the high complication rate. The MNI rate of another clinician (5 from 51 procedures) was on the warning line. The rates of the five remaining clinicians, who have performed the largest number of procedures, were well within the control limits. Figure 4A,B uses SPC charts to show the continuing performance of operators C and D from Figure 3. Figure 3 shows that operator C’s performance has been within the control limits throughout.

35 30

MNI (%)

25 20 15 10

128

137

146

155

111

118

125

119

104

110

92

101

83

74

65

56

47

38

20

29

0

23

21

19

17

15

13

11

9

7

3

5

5

(B)

Amniocentesis procedures

100 90 80 70 60 50 40 30 20 10 0

(B) 35 30 25

MNI (%)

Figure 2. (A) Cumulative performance of operator A. Each circle represents the percent failure rate of the operator, who is shown to be performing well within the expected limits. (B) Cumulative performance of operator B. The operator is seen to ‘touch’ the warning limit after the fifth procedure and crosses both the warning and action limits after the 14th procedure. In figures 2–4 the solid horizontal line is the expected complication rate and the dashed lines represent the warming and action limits.

5 0 97

Ventouse procedures

10

90

10 11 12 13 14 15 16

83

9

76

8

69

7

62

6

55

5

48

4

41

3

34

2

15

27

1

20

20

Failuree rate (%)

200

Figure 3. MNIs during amniocentesis. Operator D is above both the warning and action line.

Ventouse procedures

616

150


(A)

100 90 80 70 60 50 40 30 20 10 0 1

Failure rate (%)

(A)

18

MNI (%)

16 and 18 procedures with eight and nine failures, respectively. The performance of two SpR 4+ clinicians who performed 19 and 22 procedures with eight and nine failures, respectively, lied between the warning and action limits. Figure 2A shows the performance of operator A who performed the highest number of ventouse deliveries (23) and had a low failure rate (4). The chart shows that the operator’s performance was within normal limits throughout the year. In Figure 2B, the cumulative performance of operator B who performed 16 procedures with eight failures (50%), is shown. This operator’s performance crossed the warning line after the fifth procedure, giving an early indication of problems. The action line was crossed after just 14 procedures.


Figure 4. (A) Performance of operator C presented as a ‘rolling window’ of 20 amniocentesis procedures. The operator’s performance is well inside acceptable limits. (B) The MNI rates of operator D first crosses the action limit after the 55th procedure. This would be the signal for further action.


Performance monitoring

Operator D, however, crossed the warning line after the 48th procedure and went on to cross the action line on four occasions subsequently. In a proactive setting, the first crossing of the line would be an indication for further investigation.

Discussion The monitoring of hospital performance is an issue of intense political interest. This study shows that SPC can be used as a basis for continuous performance monitoring and quality improvement within the healthcare profession. These charts provide an alternative to performance-based league tables for the presentation of yearly data. The major advantage of SPC is continuous monitoring of the individuals working within the hospital. To facilitate continuous monitoring and feedback, the clinicians would need to record each procedure and with the aid of a spreadsheet or software produce a graphical representation of their continuing performance. The examples presented show that individuals with high complication rates tend to show early warning signs. This should lead to further assessment into possible causes which may include inaccurate data, the individual’s case mix or, indeed, suboptimal skills requiring training interventions. All need to be explored with confidentiality and sensitivity and with an agreed policy outlining proposed actions in place before commencing the process.21 If the identified cause of the high rate is lack of skills, then further training could prevent many of the later complications. Conversely, the majority of operators will be shown to be in the acceptable range thus providing reassurance to public and staff regarding the quality of care they receive and provide, respectively. Crude data may be misleading as clinicians may develop a risk avoidance policy. An example of this is in cardiac surgery where cigarette smokers or obese patients may not be treated. In our ventouse data, two of those with the highest rates were quite experienced obstetricians (SpR 4+ doctors). This may be an example of protocol-driven institutional risk aversion whereby more senior doctors are required to perform or supervise all difficult deliveries (e.g. rotational deliveries). The more junior staff, therefore, perform proportionally more straightforward deliveries. Conversely, high failure rates may be avoided by automatic recourse to caesarean section instead of operative vaginal delivery. Reluctance to undertake instrumental delivery may increase maternal morbidity despite ‘improving performance’ on the SPC. The choice of procedure to monitor is crucial. It must involve an indicator that is common enough and with small enough variability to allow valid comparisons, is routinely and accurately monitored and is an indicator of quality. The choice for any unit will depend on the total number of procedures performed, the case mix and the frequency of the complication. Furthermore, different units may need their SPC charts to be set differently to reflect this.

In this study, we chose literature-based standards. Alternatives may include data from other similar institutions or historical data from our own unit. As a teaching hospital seen as a ‘centre of excellence’, we sought to find the data that would provide a challenging target to aspire to. Finally, there is the issue of monitoring individuals who perform a small number of procedures infrequently. The warning and action limits are dependent on the number of procedures being performed, but they may be too conservative when it comes to detecting poor performance in individuals with small number of procedures. To address this, we reduced the level of the warning and action lines to 90 and 95%, respectively, and removed from the analysis those conducting very few procedures. Obstetrics is a high-risk specialty and accounts for approximately half of the NHS’s clinical negligence bill. In the past, the performance of juniors was monitored by the consultant in charge of their clinical firm, but the introduction of shift working means that personal mentoring is difficult. Underperforming doctors can therefore easily slip through the net without formal strategies for monitoring performance. In this study, we have shown the potential of SPC charts for the continuous prospective monitoring of common obstetric procedures. Improvements in performance of obstetric procedures could have the potential to reduce adverse events and with it the human costs to patients, families and health professionals and the associated financial costs to the NHS. j

References 1 Baker R, Jones D, Goldblatt P. Monitoring mortality rates in general practice after Shipman. BMJ 2003;326:274–6. 2 Department of Health. An Inquiry into Quality & Practice Within the National Health Service Arising from the Actions of Rodney Ledward. (The Ritchie Report). London: Stationary Office, 2000. 3 Harley M, Mohammed MA, Hussain S, Yates J, Almarsi A. Was Rodney Ledward a statistical outlier? Retrospective analysis using routine hospital data to identify gynaecologists’ performance. BMJ 2005;330: 929–34. 4 Chief Medical Officer. Good Doctors, Safer Patients. London: Department of Health, 2006. 5 Bridgewater B, Grayson AD, Jackson M, Brookes N, Grotte GJ, Keenan DJM, et al. Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data. BMJ 2003;327:13–17. 6 Wetherill GB. Statistical Process Control: Theory and Practice. London: Chapman & Hall, 1991. 7 Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman, and clinical governance: Shewhart’s forgotten lessons. Lancet 2001; 357:463–7. 8 Morton AP, Whitby M, McLaws ML, Dobson A, McElwain S, Looke D, et al. The application of statistical process control charts to the detection and monitoring of hospital-acquired infections. J Qual Clin Pract 2001;21:112–17. 9 Ismail NA, Pettitt AN, Webster RA. ‘‘On-line’’ monitoring and retrospective analysis of hospital outcomes based on a scan statistic. Stat Med 2003;22:2861–76.


617

Lane et al.

10 Bridgewater B. Mortality data in adult cardiac surgery for named surgeons: retrospective examination of prospectively collected data on coronary artery surgery and aortic valve replacement. BMJ 2005;330:506–10. 11 Grigg AO, Farewell VT, Spiegelhalter DJ. Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Methods Med Res 2003;12:147–70. 12 CEMACH. Confidential Enquiry into Maternal and Child Health. Stillbirth, Neonatal and Post-natal Mortality 2000-2003 England, Wales and Northern Ireland. London: RCOG Press, 2005. 13 Adab P, Rouse AM, Mohammed MA, Marshall T. Performance league tables: the NHS deserves better. BMJ 2002;324:95–8. 14 Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med 2005;24:1185–202. 15 Simpson D, Roberts T, Walker C, Cooper K, O’Brian F. Using statistical process control (SPC) chart techniques to support data quality and information proficiency: the underpinning structure of high-quality health care. Qual Prim Care 2005;13:37–44.

618

16 Alfirevic Z, Walkinshaw SA. Amniocentesis and Chorionic Villus Sampling. Green Top Guidelines, RCOG Guideline No. 8 (Revised), London, 2005. 17 Sau A, Sau M, Ahmed H, Brown R. Vacuum extraction: is there any need to improve the current training in the UK? Acta Obstet Gynecol Scand 2004;83:466–70. 18 Groom KM, Jones BA, Miller N, Paterson-Brown S. A prospective randomised controlled trial of the Kiwi Omnicup versus conventional ventouse cups for vacuum-assisted vaginal delivery. BJOG 2006;113: 183–9. 19 Horger EO, Finch H, Vincent VA. A single physician’s experience with four thousand six hundred genetic amniocenteses. Am J Obstet Gynecol 2001;185:279–88. 20 Brown LD, Cai TT, DasGupta A. Interval estimation for a binomial proportion. Stat Sci 2001;16:101–33. 21 Parry G, Draper E, McKinney P. What to do with outliers? Qual Saf Health Care 2006;15:144.
