monitoring health system performance at subnational level Candy Day ...

South Africa: monitoring health system performance at subnational level Candy Day South Africa is a unique and highly diverse country, described sometimes as ‘the world in one country’. The country has 9 provinces divided into 52 districts, and 11 official languages. There is massive diversity in the geography, climate, ethnic groups, social values, socio-economic status, burden of disease, population density and health outcomes across these areas. In addition to this, the political history and vast inequities which persist, make it imperative that the health system is monitored not only at provincial and national levels, but also at district level or lower where possible. There has thus been a huge need to develop and strengthen information systems which are capable or producing timeous, reliable information and low levels of aggregation, within severe resource constraints. The District Health Barometer (DHB) is a tool which has been developed to provide a regular snapshot of the overall performance of the public health sector across the provinces and health districts in South Africa, focusing on primary health care (PHC). It has contributed to understanding inequities in the health system, through the integration of detailed, disaggregated time series data from sources such as the District Health Information System (DHIS), National Treasury expenditure data, Electronic TB register (ETR.net), antenatal HIV seroprevalence surveys and Statistics South Africa surveys. The DHB seeks to highlight inequities in health outcomes, health resource allocation and outputs as well as track the efficiency of health processes between provinces and between all districts in the country, with particular emphasis on rural and urban (metropolitan) districts. The report also functions as a tool to monitor progress towards strategic health goals such as the Millennium Development Goals (MDGs) and to support the improvement of provision of PHC and the improvement of the quality of routinely collected health data. The analysis of indicators between districts assists in identifying successes, gaps and potential corrective measures within the health system. The DHB also fulfils some of the roles of a Public Health Observatory, by making population and health indicators readily available, and engaging with a wide range of stakeholders. Tracking trends in inequities at sub-national level Inequity can be assessed in terms of several dimensions including geographic area, socioeconomic status and individual characteristics such as race, gender or age. The latter are not available from routine aggregated data, and thus the deprivation index (DI), a composite measure of relative deprivation between areas, has been developed to facilitate comparison of health indicators according to socio-economic quintiles (SEQ), or need. The DI and SEQs have been calculated for a series of years, and also expanded to sub-district level in 2007.

Using ‘Non-hospital PHC expenditure per capita’ as an example, it can be seen that although there is an increase in the absolute difference between the highest and lowest expenditure by district, the relative difference is decreasing, and both the absolute and relative difference between the best and worst socio-economic quintiles are slowly decreasing, suggesting a gradual improvement in financing equity. (The absolute gap is the difference in indicator values for the disadvantaged group (SEQ 1) and the reference group (SEQ 5). The relative gap is the ratio or percentage difference between these values.)

2

On the other hand, when considering a key socio-economic determinant for health, the proportion of households with access to piped water, there has been remarkably little improvement in inequities between districts even though there has been an overall improvement at a national level. The most striking feature of a simple line graph of household access to piped water at district level, grouped by province, is the wide disparities in some provinces such as Eastern Cape (EC) and KwaZulu-Natal (KZN), compared to Free State (FS) and Western Cape (WC) – which are concealed if one only monitors the national trend. When considering the distribution of household access to piped water at district level according to district type (using a boxandwhisker plot) the disparity between ISRDP (rural) areas and metros is dramatic, and there has been very little improvement over time.

The power and pitfalls of routine data A thorough assessment of the usefulness and problems of routine or administrative data sources for health indicators is beyond the scope of this paper, therefore only a selection of key issues from our experience will be highlighted here. Despite relatively low levels of financial and human resources, the DHIS has been rolled out across the country, and is currently the main source of regular information for planning and management of health services. Although discrepancies do occur, in general there is standardisation across the country of the definitions of data elements and data flow policies. One of the major advantages of the system is that it provides relatively simple access to a wide range of integrated health indicators, including all levels of the public health system, and even integrating selected key information from surveys and other primary data sources. Data verification There is inadequate monitoring of indicators throughout the system, from facility to national levels. This has resulted in some districts having indicator values that are clearly implausible. 3

There has been very little regular and comprehensive verification of the data quality from routine systems, however published papers evaluating selected aspects of DHIS have found significant discrepancies in the completeness and accuracy of data. It appears that the main source of inaccuracy lies in the data collection tools at health facilities and the summary and transfer of these records into the software system.

4

Data validation with external data sources and information Despite the multitude of data quality problems, for some indicators there has been good correlation with external data sources (for example HIV prevalence among antenatal clients) and consistency over time (for example PHC utilisation rate, which shows consistent seasonal variation and changes corresponding to known events such as health worker strikes).

Data limitations Routine health information systems currently have virtually no patient-level clinical data. Limited private sector data are captured in the system. Certain indicators are not amenable to this type of data collection. For example monitoring of the implementation and outcomes of the PMTCT programme involve numerators and denominators which may be collected over the time period of the pregnancy, at different health facilities, since initial antenatal care may take place at a different location to the delivery. There are issues with disaggregated data for rare events such as maternal or perinatal deaths. Small data errors can dramatically affect indicator values, and at low levels of disaggregation indicator values do fluctuate widely, particularly for areas of low population density.

5

Looking at monthly data for an indicator like Perinatal Mortality Rate (PNMR) shows that the difference between districts is less than the range of variation in the data because perinatal deaths are relatively few at this level of disaggregation.

Integration of data for more explanatory analysis Although there are some national structures and initiatives involved with the co-ordination and harmonization of health information systems and data sources, it is widely acknowledged that overall integration and use of systems is inadequate, and many have substantial data quality problems. The DHB has worked with some of these data sources, developing coding to connect the DHIS with expenditure data and selected survey-based socio-economic determinants. There have been many challenges with this due to lack of standard coding for health facilities, geographic areas and a general lack of consistency or documentation. In addition the structure and content of the information changes each year, so the process of integration between data sources requires ongoing work. Some important areas of data integration have been highly problematic, in particular the human resource information system, due to fundamental deficiencies in how the data are collected. It is clear that despite the need for interoperability between information systems for effective health systems management and assessment, some systems have not been responsive in putting the fundamental architectural components in place to facilitate integration of data. Integration of data from other sectors is also vital for monitoring of multi-sectoral interventions, and further work is planned to develop and improve linkages between these sources.

6

Summary As part of HIS strengthening, South Africa undertook a national HIS assessment in March 2009, using the Health Metrics Network framework. These results are based on scoring by the participants present and each area covers a number of different data sources which may be of different adequacy making it difficult to generate an accurate result – however it does give an overall feel for perceptions of the adequacy of data sources. In general, surveys and StatsSA sources received higher ratings, while health and resource records were generally found to be problematic.

Data Source

Contents

Capacity & Practices

Dissemination

Integration and use

Total

Census

Highly adequate 100%

Highly adequate 75%

Adequate 71%

Adequate 56%

Highly adequate 75%

Vital statistics

Highly adequate 89%

Adequate 67%

Highly adequate 100%

Highly Adequate 83%

Highly adequate 85%

Population-based surveys

Adequate 57%

Highly Adequate 88%

Highly adequate 100%

Present but not adequate 33%

Adequate 70%

Health and disease records (incl. surveillance)

Adequate 56%

Adequate 59%

Present but not adequate 44%

Present but not adequate 28%

Present but not adequate 47%

Health service records

Not adequate at all 12%

Present but not adequate 41%

Highly adequate 78%

Adequate 50%

Present but not adequate 45%

Resource records

Adequate 63%

Present but not adequate 40%

Present but not adequate 33%

Present but not adequate 31%

Present but not adequate 42%

Total

Adequate 63%

Adequate 62%

Adequate 71%

Present but not Adequate 47%

Adequate 61%

7

References  Garrib A,Stoops N, McKenzie A, Dlamini L, Govender T, Rohde J, Herbst K. An evaluation of the District Health Information System in rural South Africa. 2008. South African medical journal 98 (7) p. 549-52  Chopra M, Lawn JE, Sanders D, Barron P, Abdool Karim SS, Bradshaw D, Jewkes R, Abdool Karim Q, Flisher AJ, Mayosi BM, Tollman SM, Churchyard GJ, Coovadia H; Lancet South Africa team. Achieving the health Millennium Development Goals for South Africa: challenges and priorities. Lancet. 2009 Sep 19;374(9694):1023-31. Epub 2009 Aug 24. Review.  Coovadia H, Jewkes R, Barron P, Sanders D, McIntyre D. The health and health system of South Africa: historical roots of current public health challenges. Lancet. 2009 Sep 5;374(9692):81734. Epub 2009 Aug 24. Review.  Day C, Barron P, Monticelli F, Sello E, editors. The District Health Barometer 2007/08. Durban: Health Systems Trust; June 2009.  Day C, Gray A. Health and Related Indicators. In: Barron P, Roma-Reardon J, editors. South African Health Review 2008. Durban: Health Systems Trust; 2008.  Day C, Gray A. Health and Related Indicators. In: Harrison S, Bhana R, Ntuli A, editors. South African Health Review 2007. Durban: Health Systems Trust; 2007.  Day C, Gray A. Health and Related Indicators. In: Ijumba P, Barron P, editors. South African Health Review 2005. Durban: Health Systems Trust; 2005.  Herbst K, Burn A, Nzimande N. A review of data sources for public health service planning and evaluation. In: Ijumba P, Ntuli A, Barron P, editors. South African Health Review 2002. Durban: Health Systems Trust; 2003.  Mate KS, Bennett B, Mphatswe W, Barker P, Rollins N. Challenges for routine health system data management in a large public programme to prevent mother-to-child HIV transmission in South Africa. 2009 PloS one 4 (5) p. e5483  Mars M, Seebregts C. Country Case Study for eHealth South Africa. Bellagio: Making the eHealth Connection. July 2008. http://www.ehealth-connection.org/content/country-case-studies  Rohde JE, Shaw V, Hedberg C, Stoops N, Venter S, Venter K, Matshisi L. Information for Primary Health Care. In: Barron P, Roma-Reardon J, editors. South African Health Review 2008. Durban: Health Systems Trust; 2008.

8