Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P et al. The.
RAND/UCLA Appropriateness Method User's Manual, RAND, Santa Monica, ...
es
xecutive
European Practice Assessment (EPA): Quality management in primary care
10 years NATIONAL PRIMARY CARE RESEARCH AND DEVELOPMENT
executive summary 35
CENTRE
ummary 35
Background While there is substantial research describing both the development and validation of clinical quality indicators, this is not the case for indicators describing the organisational aspects and management of primary care practices. There is general agreement within Western Europe about the importance of general practice. However, the structures, financing and role of primary care within the wider health care system differs considerably between countries. Whilst the health systems of the member states operated independently of each other, this was not an issue. However, the ongoing process of European unification, introducing free movement between countries for health care services, has resulted in an increased interest in the need for quality indicators that allow comparisons of health care facilities. The European Practice Assessment (EPA) project aimed to develop and validate a set of indicators and tools describing the organisational aspects of primary care practices, to enable such comparisons to be made (Grol et al 2004). This executive summary provides an overview of this project.
Objectives 1. To develop a framework and quality indicators describing general practice organization and management. 2. To develop and validate a European Practice Assessment instrument using this framework and these indicators.
1
es
xecutive
ummary 35
Countries taking part Nine European countries took part: Austria; Belgium; France; Germany; Israel; Netherlands; Slovenia; Switzerland and the United Kingdom (represented by NPCRDC for England and the Department of Primary Care, University of Wales Swansea Clinical School for Wales). The research was co-ordinated by the Centre for Quality of Care Research, WOK at the University of Nijmegen in the Netherlands, funded by the participating countries and supported by the Bertelsmann Foundation in Guttersloh, Germany (http://en.bertelsmann-stiftung.de/foundation_kp.html).
How we developed the indicators In 2002 and 2003 EPA partners in the participating countries utilised expert panels to develop a set of quality indicators representing primary care practice management. We identified 171 potential indicators by undertaking a comprehensive review of the published and grey literature. Grey literature is material which might not be formally published, such as organizational reports. The findings were reviewed and refined at a workshop for all participants in 2002. We then undertook a two-round postal Delphi survey between June 2002 and January 2003 (Engels et al 2005). A Delphi survey is designed to measure consensus among individual responses and normally consists of a series of rounds where information is fed back to panel members using questionnaires. It is the most commonly used method of developing quality indicators. An expert panel was established in each country, consisting of ten people for each panel, predominately general practitioners, but also practice managers in the UK and NL. They were asked to rate all of the indicators for usefulness on a scale of 1 to 9, with 1 being not useful at all and 9 being very useful. Analyses were based on the Rand Appropriateness Method (Fitch et al 2001), in which indicators with an average rating above 7 were considered valid for that panel/country, whereas those with a national average of 1-3 were considered invalid. All other indicators were rated equivocal. Only indicators that were rated valid by all six panels were included in the European set of indicators. We looked at how many indicators were rated valid and invalid by all countries and per country respectively.
2
executive summary 35 The response rate for the Delphi process was 95%. Sixty-two indicators (37%) were rated valid by all six panels. None of the indicators were rated invalid by any of the countries.
Creating a framework for the indictors High quality practice management was defined in terms of the systems (structures and processes), which help to deliver good quality patient care. This definition sets practice management apart from the actual clinical care provided by the practice and served as the basis for an organizing framework for the indicators. There were five key areas to this framework: infrastructure, people, information, finance and quality and safety. Each area was divided into a number of dimensions; for example, infrastructure included premises and medical equipment. All of the quality indicators were then allocated to the most appropriate dimension on the basis of discussions between the research partners.
Developing and validating the instrument EPA partners in all nine countries took part in this stage of the project (Grol et al 2004). The 62 indicators were operationalised into 202 questions. These questions were then allocated to one of 6 questionnaires which constitute the EPA instrument. Four questionnaires are designed to be self-completed (GP, staff, practice manager, and patient) whilst an external visitor had to complete the additional two during a practice visit, an observer checklist and a structured interview with the practice manager. In addition to these questionnaires, the EPA instrument contained two other quality assessment tools, EUROPEP and the Maturity Matrix. EUROPEP is an internationally validated instrument that enables patients to evaluate the quality of primary care (Wensing et al 2002). The Maturity Matrix Family Practice (TM) is an optional tool within the EPA instrument. It is a formative educational group designed to give practice teams new insights into into their organisation and performance. It is used by the whole practice team with the assistance of a trained facilitator (Elwyn et al 2004).
3
EPA: Quality management in primary care The EPA instrument was developed in English and translated into relevant languages by at least two independent translators per country (an EPA partner in cooperation with an independent translator). The whole instrument was piloted in 3-5 practices per country in Autumn 2003. Minor modifications were made as a result of this pilot and then between January and June 2004, data were collected from over 270 practices across the nine European countries (about thirty practices per country). As we used data collected from practices who volunteered to take part, rather than from a randomly selected sample, it would be inappropriate to present comparative data in this report.
What did we find? The data collected from this larger scale study was then examined to test the properties of each question. Sixty questions were excluded from our analysis because: • They were open to misinterpretation; • They were duplicates of other questions; • There was more than 5% missing data for self-completed questionnaires; • More than 95% of practices in all nine countries achieved the question, which meant it was not a useful discriminator between practices The remaining questions were analysed using factor analyses (this reduces the number of questions to a smaller set of indicators by allowing us to group related questions together) and reliability analysis (checking to ensure that each dimension contained the appropriate indicators). This process resulted in a set of 57 indicators, spread across the five areas of the practice management framework.
Implications for policy and practice The EPA study has successfully developed and tested a novel and comprehensive instrument which can be used to compare the organisational elements of general practice care in different European countries. We think that the instrument is useful for both formative (as an educational tool to help practices identify areas for improvement) and summative (to judge how well a practice is doing, usually compared to other practices) purposes.
4
executive summary 35 The challenges of undertaking a large scale international project such as this should not be underestimated. The participants from the different countries had a long track record of working successfully together and good will and commitment was required to overcome the significant challenges associated with different languages, understandings, cultures and health systems. Each country made an important contribution to the research process and to the final product and the participants gained new insights into their own health systems from discussions with others in the group and from examining the study findings. As far as the UK is concerned, its long established system of general practice, mature primary health care teams and professionalized managerial processes contributed much to the project. In addition, the NPCRDC’s expertise in developing and using quality indicators was highly valued by the other partners.
The future of EPA The EPA instrument is not a finished product and requires further development and testing. The focus will be on three areas: 1. Updating the indicators as new evidence or experience comes on line; 2. Using the instrument in a representative sample of practices so that meaningful comparisons can be made between organizations and countries; 3. Examining ways of supporting practices which make use of the instrument in order to maximize its potential to improve the performance of primary care practices.
5
EPA: Quality management in primary care The extent to which the instrument will be used in different countries will depend upon the local context. In the UK, for example, there is considerable overlap between the EPA indicators and the structural/organisational indicators within the Quality and Outcomes Framework of the new General Medical Services contract (BMA, 2004). Given that this is a high priority to British general practices, it is less likely that the instrument will be used widely in the UK in the short term. It is, however, more comprehensive than the contract indicators and will be a useful source of information when the contract is revised. In contrast, most other European countries are not currently using organizational indicators and so EPA is likely to be of significant use to them. Partners in the nine countries that took part in the EPA, including NPCRDC and the Bertelsmann foundation, founded the TOPAS-Europe Collaboration in January 2005, coordinated by Professor Richard Grol from the Centre for Quality of Care Research in Nijmegen. This collaboration aims to develop, validate and continuously revise and improve a wide range of indicators, instruments and tools for assessing health care practices and professionals, representing a range of dimensions of quality of care. Those who want to use the tools, including EPA, can apply to the TOPAS-Europe Collaboration to do so. For further information, visit (http://www.topas-europe.org).
References British Medical Association. Quality and Outcomes Framework, BMA, London, 2004 (Available at: http://www.bma.org.uk) Elwyn G, Rhydderch M, Edwards A, Hutchings H, Marshall M, and Grol R, Assessing organisational development in primary medical care using a group based assessment: The Maturity Matrix. Qual & Saf Healthcare, 2004. 13: p. 287-94. Engels Y, Campbell S, Dautzenberg M, van den Hombergh P, Brinkmann H, Szécsényi J et al. Developing a framework of, and quality indicators for, general practice management in Europe. Family Practice 2005; 22: 215-222. Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P et al. The RAND/UCLA Appropriateness Method User's Manual, RAND, Santa Monica, 2001. Grol R, Dautzenberg M, Brinkmann H (eds) Quality Management in Primary Care, Verlag Bertelsmann Stiflung, Gutersloh, 2004. Wensing M, Vedsted P, Kersnik J, Peersman W, Klingenberg A, Hearnshaw H, et al. Patient satisfaction with availability of general practice: an international comparison. International Journal of Quality in Health Care 2002; 14: 111-8.
6
executive summary 35 Authors (on behalf of the EPA project partners): • Centre for Quality of Care Research (WOK), Universities of Nijmegen and Maastricht, the Netherlands: Richard Grol, Yvonne Engels, Maaike Dautzenberg, Pieter van den Hombergh • National Primary Care Research and Development Centre, University of Manchester, UK: Stephen Campbell, Martin Marshall • Centre for Health Sciences Research, Cardiff University, UK (formerly Swansea University during the EPA study): Glyn Elwyn • Swansea University, UK: Melody Rhydderch • Bertelsmann Foundation, Gütersloh, Germany: Henrik Brinkmann, Andreas Esche, Jan Böcken • AQUA Institute, Göttingen, Germany: Joachim Szecsenyi, Ferdinand Gerlach, Björn Broge, Petra Wippenbeck • Société de Formation Thérapeutique du Généraliste, Paris, France: Marianne Samuelson, Hector Falcoff • Swisspep Institute for Quality and Research in Healthcare, Gümligen, Switzerland: Beat Künzi, Walter Oswald • Scientific Society of Flemish General Practitioners, Berchem (Antwerp), Belgium: Luc Seuntjens, Nicole Boffin
7
EPA: Quality management in primary care Further information A book has also been produced - Grol R, Dautzenberg M, Brinkmann H (eds) Quality Management in Primary Care, Verlag Bertelsmann Stiflung, Gutersloh, 2004. ISBN 389204-790-1. More details are available from the Bertelsmann Foundation at: http://en.bertelsmann-stiftung.de/media/English_Publications_2005_Flyer.pdf A report produced by Wensing et al for the EUROPEP group which summarises the process is available at: http://www.swisspep.ch/pdf/europep.pdf
Study team contact details For more information about the study please contact the UK project lead Dr Stephen Campbell at (email:
[email protected], or telephone: 0161 275 7655). About NPCRDC NPCRDC is a multi-disciplinary centre, established in 1995 to carry out policy related research in primary care. Our centre is a collaboration between the Universities of Manchester and York, with our main base at the University of Manchester. We are committed to excellence in primary care research and dissemination. If you would like further copies of this summary, please contact NPCRDC. National Primary Care Research and Development Centre, Williamson Building, The University of Manchester, Oxford Road, Manchester. M13 9PL. Telephone: +44-(0)161 275-7634 Fax: +44-(0)161 275-0611. Email:
[email protected] Full details of the indicators are available to download from: www.npcrdc.ac.uk
es xecutive
ummary 35
8
