Evaluation of health system strengthening initiatives for improving the quality and outcomes of maternal, neonatal and child health care in four. South African ...
Evaluation of health system strengthening initiatives for improving the quality and outcomes of maternal, neonatal and child health care in four South African districts
Conducted by: School of Public Health UWC/SAMRC Health Services to Systems Research Unit University of the Western Cape Helen Schneider Andrew McKenzie Hanani Tabana Fidele Mukinda Asha George November 2017
Executive summary Background and methods Since 2014, the National Department of Health, in partnership with a number of players and agencies, has been providing targeted support to four districts to improve their maternal, neonatal and child health (MNCH) services and outcomes. These districts are: Waterberg in Limpopo; Gert Sibande in Mpumalanga; and OR Tambo (ORT) and Nelson Mandela Bay Metro (NMBM) in the Eastern Cape. Under the broad umbrella term “3 feet model” this support has involved two distinct but complementary entry points, strands of activity and processes of facilitation: 1) The establishment of sub-‐district and district “Monitoring and Response Units” (MRUs), drawing together players across levels (community, PHC, hospitals and district), and focused on monitoring and coordinated action to reduce key causes of maternal, neonatal and child mortality and morbidity. 2) Quality improvement tools and processes focused on enhanced data use and problem solving in order to achieve coverage targets, starting at community and facility level in the primary health care system (PHC), and integrated into the PHC supervisory mechanisms at sub-‐district and district level. In Waterberg and Gert Sibande both components were implemented, whereas in the Nelson Mandela Bay Metro, the emphasis was on the second component, due to the relative absence of district hospitals connected to the PHC system. In OR Tambo, after an initial phase of implementation of both components, further facilitation was put on hold as key supporters in the district management team left the district. In the other three districts, the 3 feet model initiatives, now in their third year of implementation, have generated attention, engagement and commitment from a range of actors who perceive the interventions to be beneficial. The School of Public Health, University of the Western Cape (UWC), through its MRC-‐funded extra-‐mural Health Services to Systems Research Unit, and with additional funding from UNICEF (for the Eastern Cape component), evaluated the 3 feet model in the four districts, focusing in particular on the three districts where the initiatives had been fully anchored. The aims of the evaluation were to describe and assess the impact of the 3 feet model in achieving improved MNCH quality and outcomes in the districts, how these were achieved, their likely sustainability and the lessons for scale up and implementation elsewhere. Specifically it assessed: 1. Trends in maternal, neonatal and under 5 programme coverage and health outcomes over the period of intervention, using the district health information system (DHIS) and other locally performed audits (e.g. death audits);
ii
2. Processes of entry, facilitation and implementation of the 3 feet interventions; 3. The perceptions of district actors, from frontline to senior on the value, impacts and likely durability of the 3 feet interventions; 4. The factors – intervention design, actors engaged, processes adopted and context conditions – underpinning success (or not); 5. Prospects for integration, sustainability and “scale-‐up” to other districts. The methods involved a post implementation assessment that combined analysis of trends in MNCH outcomes and coverage using routine data and additional district based audits, and more than 130 interviews with national and district players. These were supplemented by observations of meetings and reviews of documentation. The findings are presented in two parts: Part 1: MRU/3 feet in Gert Sibande and Waterberg Districts Part 2: PHC 3 feet in Nelson Mandela Bay Metro and MRU/3 feet in OR Tambo District (in two sections, Part 2:1 and 2:2) Part 1: MRU/3 feet in Gert Sibande and Waterberg Districts Underpinned by a set of values and principles (e.g. outcomes based orientation), the core elements of 3 feet in these two districts have been maternal, neonatal and child “real-‐time” death reporting, analysis and evidence-‐based responses, embedded in a system of district and sub-‐district governance in the form of Monitoring and Response Units (MRUs). MRUs bring together line managers (drivers), clinicians (experts) and programme managers (navigators), and crucially, enable new forms of collaboration between levels of the district health system: between primary health care, hospitals, ward based outreach teams and district management. These are “held together” with a set of tools, guidelines and structured processes and the use of metaphors such as the “open tap”, connecting different parts of the system and helping individual actors to see their part in the whole and to link prevention with care. They have been greatly enhanced by the 3 feet quality improvement methodologies introduced into the PHC system and integrated into the routine monthly review processes (described in more detail in Part 2). After three years, the MRU/3 feet model has become well anchored in the two districts and their sub-‐districts, with widespread support for the methods and approach, even if not implemented equally in all facilities. Changes in service delivery for maternal, neonatal and child health (most notably for severe acute malnutrition) were extensively reported in both districts. They entailed: 1) Enhanced screening in community and PHC settings, with early identification of problems (secondary prevention) 2) Better functioning referral systems across levels of care and between players within hospitals 3) Improved clinical practices within health facilities 4) Better continuity of care (most notably for child malnutrition). These have been made possible by a “bundle” of new organizational practices, which involve new forms of team work (across levels, sectors, professions), better uptake of guidelines, knowledge sharing and
iii
in-‐service training, better use of information to guide practice, and greater individual and collective accountability. Interviewees made frequent reference to changes in attitudes to death, to accountability for performance, and to use of information. These were, in turn, facilitated by new knowledge and understanding, in particular the development of a systems perspective and the linking of preventive actions in PHC and communities to care in hospitals. These service delivery, organizational and mindsets changes plausibly account for changes in outcomes – maternal, perinatal and child mortality and morbidity documented in the two districts. The changes were achieved with the relatively modest inputs of two, albeit high quality, facilitators, providing methodologies, “role-‐modelling”, training and mentoring to the districts on a periodic basis. The focus was on mobilizing existing resources and players, such as the District Clinical Specialist Teams (DCSTs) and programme managers, who have been key to successes, rather than adding new resources. Local contextual factors influencing implementation include stability and quality of leadership (at all levels), health workforce imbalances, gaps in service delivery infrastructure, and the challenging social contexts of the two districts. The interventions interface with a range of other quality improvement and audit processes in the two districts. They include, amongst others, the perinatal and child problem identification programme (PPIP and Child PIP), quality assurance (adverse event reporting, complaints mechanisms) and the Ideal Clinic. While the MRU/3 feet is perceived to add value, future sustainability depends on the extent to which it can align with these other improvement and accountability mechanisms, and on its ability to retain its transformative edge. Part 2:1 3 feet in Nelson Mandela Bay Metro District As indicated, the focus in NMBM has been on the PHC system component of the 3 feet model, the essence of which is improved information use and problem solving by PHC and community based players in order to achieve coverage targets.1 Supported by a well designed package of tools and software, it functions as a bottom-‐up problem identification and planning process, integrated into the routine supervision and review system, from facility, to cluster, to sub-‐district and district. A key aspect is the visual nature of the tools used, that include a set of agreed indicators and targets, dashboards/run charts to track progress of the indicators over time, the 3x4 matrix and the bottleneck analysis to identify and prioritise activities to improve performance, activity charts/dashboards to capture agreed activities and track implementation and a traffic light system (green, amber, red) to visually highlight progress. Through bottleneck analysis, actions required at higher levels of the district can be identified and fed into formal planning processes such as the District Health 1 Note this was implemented in all the intervention districts, by the same facilitators, and the
successes and lessons described therefore apply, for the most part, to the other districts.
iv
Plans and District Implementation Plans (for the “90:90:90” Strategy for HAST). Similarly, using the same methodology “outwards”, with ward based outreach teams and other community players (3 feet community component), action on bottlenecks related to demand (e.g. immunization) or social norms (e.g. breast feeding) can be facilitated. At the time of the review, the methodology had been implemented across the metro and institutionalised in the key supervision and review processes at facility, cluster, sub-‐district and district levels. It had the buy-‐in of, and was actively driven by the District Manager and District Management Team. It was in the process of being scaled up to other districts in the Eastern Cape. The 3 feet model was widely perceived to have significantly changed facility level practices in the PHC system of NMBM, with: -‐ An evidence-‐based approach to defining problems and action -‐ Ownership of data and problems, with PHC facility managers spurred to look for innovative solutions within their own sphere of influence, rather than blame the “perennial staff shortages” -‐ Closer collaboration and team-‐work forged between clinic and community services through the Ward Based Outreach Teams -‐ Greater peer learning across facilities and clusters -‐ The embedding of a new culture of collective responsibility and accountability for reaching targets in the PHC system This was enabled by a carefully managed and incremental facilitation process, integration of the interventions with core supervisory and review processes, and ownership and leadership of the processes by line managers. Analysis of routine data showed impressive improvements in virtually all the HIV and MNCH coverage (and some outcome) indicators examined in NMBM. However, these trends were also observed in other Eastern Cape Districts. Following the experiences of the NMBM, the 3 feet processes have been disseminated to other districts, formally since mid-‐2016. It is possible that there have been spill over effects, although without further interrogation of initiatives in other districts, it is not possible to draw any conclusions in this regard. Constraining factors include the limited financial and human resource delegations to tackle higher-‐level bottlenecks, particularly as a ceiling of improvement is reached at facility level; uncertainties with catchment populations at facility level and therefore denominators of indicators; the experience of “initiative overload” at the coal-‐face of the health system; and the lack of integration with maternal and perinatal audit systems in hospitals and with other quality improvement systems such as the Ideal Clinic. Part 2:2 MRU/3 feet in OR Tambo District As a high maternal, neonatal and child mortality District, OR Tambo was one of the first to be engaged with both MRU and 3 feet PHC interventions. However, after an initial phase, the loss of key players at District level left a gap in district drivers and champions and was compounded by key vacancies in the PHC line
v
functions of the District. External facilitation of, and support for, the MRU/3 feet activities were therefore not continued, although at the time of the review, there were plans were for a resumption as part of a broader provincial scale up. While recognizing that OR Tambo provided a useful counterfactual to the “successes” elsewhere, it was not possible to conduct a meaningful assessment within the timelines of the overall evaluation. Nevertheless, we sought to obtain some insights on developments in this District in five telephonic interviews with players who had been involved in 3 feet implementation in one way or another. From these, a few tentative observations can be made about experiences in this District and the lessons it offers for implementation and scale up elsewhere. Despite not receiving ongoing facilitation, the evidence suggests that the 3 feet model in OR Tambo did find relevance, and had been partially implemented and even sustained in certain parts of the District. Performance of some of the coverage indicators has improved, although in the context of multiple other interventions, attributions are hard to make. Where 3 feet was implemented, the nature and pathways of this were not dissimilar to that of the other districts evaluated. For example: • Across sub-‐districts, implementation of the 3 feet PHC tools and processes had been anchored in the monthly, quarterly and annual review processes. According to one interviewee: 75% of PHC facilities were “doing it fully” on a monthly basis. • The MRU design of joint review processes between clusters of PHC facilities and a district hospital was widely perceived as beneficial, was functional in at least one sub-‐district and in the process of being “resuscitated” in the other sub-‐districts. The 3 feet interventions in OR Tambo were kept alive by the energetic efforts of a few key individuals. However, it lacked the critical mass of players (drivers, navigators and experts) at district and sub-‐district level to ensure their sustained implementation across the district. OR Tambo District also has significant infrastructural and human resource challenges, and as an NHI pilot site, has had to implement numerous innovations one after the other. Low absorptive capacity, its size and complexity, and the legacies inherited from the past make the governance and leadership of systems strengthening in OR Tambo District particularly challenging. Conclusions In many respects, the 3 feet model interventions evaluated in the four districts acted to shift organizational culture – namely the accepted rules and ways of doing things – in positive ways. They offer important lessons for other districts. In transferring the lessons, the 3 feet toolkit and methodologies provide valuable guidance on how to unlock latent capabilities in local health systems. However, it is important to recognize that the “hard core” of interventions, however well designed, is embedded in a “soft periphery” of learning and implementation. The soft periphery includes, for example, how districts are engaged, partnerships and processes of facilitation that are both strategic and operational, the role of team-‐ work, processes of sense-‐making, informal alliances, and systems of
vi
accountability and support. These, in turn, are enabled or constrained by district governance and leadership capacity, degree of decentralized decision-‐ making and the broader provincial health system context. Acknowledgements Conducting an evaluation of initiatives with so many positive effects has been immensely rewarding. We are deeply grateful to the following for giving us this opportunity: -‐ Drs Joey Cupido, Sanjana Bhardwaj and Yogan Pillay for approaching us to do this work, approving the protocol and facilitating the evaluation in the four districts -‐ The District Managers and senior staff in the districts who welcomed us and agreed to the disruptive presence of external visitors and interviewees -‐ The key focal points in the districts who steered the logistics, arranging interviews and visits that made the evaluation smooth sailing: Ms Kholekile Mabunda, Mr Stephen Mosimege, Dr Jeannette Wessels, Ms Nadiema van der Bergh and Ms Nosipho Mdondolo -‐ The 130+ people – CEOs and other senior staff, PHC and hospital operational managers, clinicians, DCSTs, programme managers, EMS, dieticians, WBOT team members, and partners – who so generously gave of their time and insights -‐ they are this report -‐ UNICEF and the UWC/SAMRC Health Services to Systems Unit for their funding.
vii
Table of contents Executive summary ........................................................................................................................................... ii Background and methods ......................................................................................................................... ii Part 1: MRU/3 feet in Gert Sibande and Waterberg Districts ................................................... iii Part 2:1 3 feet in Nelson Mandela Bay Metro District .................................................................. iv Part 2:2 MRU/3 feet in OR Tambo District ........................................................................................ v Conclusions ..................................................................................................................................................... vi Acknowledgements .................................................................................................................................... vii Background ........................................................................................................................................................... 1 Methods .................................................................................................................................................................. 4 Limitations ....................................................................................................................................................... 7 Findings .................................................................................................................................................................. 9 PART 1: MRU/3 feet in Gert Sibande and Waterberg Districts ................................................... 10 Design ................................................................................................................................................................... 10 Implementation ............................................................................................................................................... 12 Entry and adoption ................................................................................................................................... 12 Actor roles ..................................................................................................................................................... 16 Impact .................................................................................................................................................................. 21 Changes in outcomes ................................................................................................................................ 21 Gert Sibande ............................................................................................................................................ 21 Waterberg District ............................................................................................................................... 26 Service delivery improvement ............................................................................................................. 29 New organisational practices ............................................................................................................... 32 Team-‐work .............................................................................................................................................. 32 Evidence-‐based practice .................................................................................................................... 33 Use of information ................................................................................................................................ 34 Improved accountability ................................................................................................................... 34 Mindsets ......................................................................................................................................................... 35 Local contexts impacting on change ....................................................................................................... 38 Leadership .................................................................................................................................................... 38 Health workforce ....................................................................................................................................... 39 Social contexts ............................................................................................................................................. 41 Service delivery infrastructure ............................................................................................................ 41 Integration and sustainability ................................................................................................................... 42 Summary and conclusions ........................................................................................................................... 44 PART 2:1 3 feet in Nelson Mandela Bay Metro ................................................................................... 47 Design ................................................................................................................................................................... 47 Essence of 3 feet ......................................................................................................................................... 47 Implementation .......................................................................................................................................... 49 Entry ........................................................................................................................................................... 50 Adoption ................................................................................................................................................... 51 Impact .................................................................................................................................................................. 52 Changes in outcomes ................................................................................................................................ 52 Changes in service delivery ................................................................................................................... 55 Changes in organisational practices and mindsets ................................................................ 56 Integration and sustainability ................................................................................................................... 60 Scale up ........................................................................................................................................................... 60 Integration .................................................................................................................................................... 60 Sustainability ............................................................................................................................................... 61 Discussion: Key success factors and challenges going forward .................................................. 62 Summary and conclusions ........................................................................................................................... 66 PART 2:2 MRU/3 feet in OR Tambo District ........................................................................................ 69
viii
Background ................................................................................................................................................... 69 Implementation and impacts ................................................................................................................ 69 Enablers and constraints ........................................................................................................................ 70 Conclusions ................................................................................................................................................... 71 OVERALL CONCLUSIONS AND IMPLICATIONS FOR SCALE UP .................................................. 72 Annexure 1: Quantitative data analysis ................................................................................................. 75 Mpumalanga ........................................................................................................................................... 77 Limpopo .................................................................................................................................................... 84 Eastern Cape ........................................................................................................................................... 90
Tables and Figures Table 1: Profile of districts studied* .......................................................................................................... 2 Table 2: Profile of data collection methods ............................................................................................ 4 Table 3: Profile of interviews ........................................................................................................................ 6 Table 4: List of indicators ............................................................................................................................... 7 Table 5: Participants in the MRU at district and sub-‐district level ............................................ 18 Table 6: Health system and social contextual factors ..................................................................... 38 Table 7: Other quality improvement and programmatic initiatives ......................................... 44 Figure 1: The open tap analogy (Source: Dr J Cupido) .................................................................... 11 Figure 2: Timeline of MRU/3 feet implementation .......................................................................... 13 Figure 3: Actor roles in the MRU implementation ............................................................................ 17 Figure 4: Sub-‐district middle manager informal alliances ............................................................ 18 Figure 5: Impact of MRU/3 feet ................................................................................................................ 21 Figure 6: Maternal mortality ratio and number of maternal deaths 2012-‐2016 ................ 22 Figure 7: Case fatality rate and cases of severe acute malnutrition
