Monitoring Implementation and Evaluating Performance - Working Paper

Monitoring Implementation and Evaluating Performance Experiences from cash social assistance in Moldova

Valentina Barca and Ludovico Carraro September 2013

ISSN 2042-1257 (Print) ISSN 2042-1265 (Online) ISBN 978-1-902477-14-5

Acknowledgements This paper analyses the structure and achievements of a new M&E system for the social support benefit established in the Ministry of Labour, Social Protection and Family (MLSPF) of Moldova under the UK Department for International Development (DFID) funded project “Support to the delivery of effective and sustainable social assistance services”, where Oxford Policy Management (OPM) worked to provide technical assistance to the Ministry. We are grateful to have participated in this reform process and we cherish the good working relationships established with staff of the Ministry, the National Social Insurance House and the National Bureau of Statistics. The views expressed here are those of the authors and not attributable to either DFID or the MLSPF.

Monitoring Implementation and Evaluating Performance

1

1 Introduction “It is tempting – but dangerous – to view M&E as having inherent value”, a recent World Bank report provocatively states. “The value of M&E does not come simply from conducting M&E or from having such information available”, but from “using the information to help improve government performance”. The report goes on to establish the key roles that government M&E systems should fulfil if they were set up appropriately: support policy making, performance budgeting, and national planning; help government ministries and agencies manage activities at the sector, programme, and project levels; and, enhance transparency and support accountability relationships (Mackay, 2007). But how do we set up an M&E system that fulfils these key roles? While a vast literature has been written on Results Based Management and the ideal structures for M&E (Gosling and Edwards, 1995; UNDP, 2002; Adams et al., 2003; Kusek and Rist, 2004; Gorgens and Kusek, 2009 among others), there are few successful examples of the complex processes that governments need to go through in 1 order to set up such systems in specific contexts – in our case, providing social assistance to the poor – and, most importantly, make them feed back into the policy process. For clarity, within this paper, we divide the M&E system into three main building blocks: the identification of key indicators and targets, ensuring that adequate data sources exist to collect information on the indicators and finally the institutional arrangements that guarantee the production and use of the data. The definition of key indicators and targets is perhaps the area where most guidelines have been provided, especially as the overall concepts are easily applicable (Adams et al., 2003). Importantly, the literature highlights that “indicators are only relevant when they measure against an objective” (Kusek and Rist, 2004) and that they should be: precise and unambiguous; appropriate to the subject at hand; available at a reasonable cost; provide a sufficient basis to assess performance; and, amenable to independent validation (Schiavo-Campo, 1999). Regarding the second – complementary – step of establishing data sources for the M&E process, the international research community has placed much emphasis in recent years on the use of ad hoc surveys with experimental evaluation designs that provide ‘rigorous’ evidence of programme impact. Since 2006, when the ‘evaluation gap’ was first denounced by Esther Duflo and her colleagues at the Centre for Global Development (CGD, 2006), over 800 costly impact evaluations of social policy 2 interventions have been carried out in low and middle income countries . We argue, together with a recent wave of literature on the matter (Bedi et al, 2006; Jones et al, 2009; Kusters et al. 2011; Weyrauch and Langou, 2011), that while ad hoc evaluations do provide important information, they seldom feed into programme improvement and planning and are therefore often less ‘useful’ than ongoing monitoring and less expensive forms of evaluation based on existing data. Interestingly, the use of administrative data sources for regular monitoring activities – and its matching with national household survey data to evaluate simulated impacts (as well as for research on social assistance systems) – is less of a ‘hot topic’ in the literature. A wave of important contributions on the matter was made in the late 1990s in the US, where researchers and practitioners started to document the huge potential of using social security administrative databases and management information systems (MIS). Compared to surveys, they argued that administrative data can: quickly respond to programme changes; offer a much larger sample size which allows for disaggregated geographic analysis; allow the tracking of individuals and households over time; be significantly less expensive; and, be matched to other data sources (including surveys). Important limitations, of course, include: the ‘denominator problem’ by which the “choice-, event-, or participation-based nature of administrative data limits inferences”; the fact that administrative data does not measure all outcomes (for example some indicators of well-being); and, that “data is only available when the client is ‘in the program’”, while little is known when they leave (Hotz et al, 1998). The scarce recent literature on the topic tends to focus on how to maximise the use of MISs to strengthen control and accountability mechanisms for social safety net programmes (Baldeon and Arribas, 2008) and on practical issues around the cleaning and matching of administrative data (Goerge and Bong Lee, 2002), but it is difficult to find literature that provides guidance on how MISs Oxford Policy Management

Working Paper


2

should be designed and fit into the overall M&E system. In this paper we argue that administrative data from programme’s MISs can be a fundamental and unique source of information for management and can provide very quick feedback on adjustments necessary for the improvement of programme implementation. Moreover, matching such data to other sources – existing household surveys and ad-hoc qualitative studies among others – can provide a wealth of information for both monitoring and performance evaluation purposes. The third key aspect of creating a functional M&E system, we propose, is the establishment of institutional arrangements – the “formal and informal processes, procedures, rules, and mechanisms that bring monitoring activities into a coherent framework” (Bedi et al., 2006). In other words, what we refer to here is the people and institutions working behind the scenes, putting M&E theory into practice. Going back to the teachings of the forefather of utilisation-focused evaluations, Michael Quinn Patton, any M&E system should be focused on “intended use by intended users”. However, specialists now recognise that this is “one of the most difficult aspect in the design and implementation of monitoring systems”, requiring a combination of strong political leadership, coordination of actors with different incentives, links with line ministries, and involvement of national statistics agencies (Bedi et al., 2006). The importance of generating appropriate demand for M&E within line ministries and central government , including the capacity to interpret and act upon the evidence provided by the M&E system, cannot be over-estimated in this context. In this paper we hope to contribute to the literature by describing the experience of setting up a comprehensive M&E system for a new cash benefit in Moldova. As applicants enrolled for the benefit, their information was recorded by social assistants around the country in a comprehensive MIS to assess their eligibility. We argue that the use of that information, linked with other sources, was invaluable for the implementation of the reform. Helping to identify problems and misunderstandings at an early stage, we believe that important lessons were learned from this M&E system – an example of good practice that could be useful for many other countries intending to implement similar programmes. Other than providing a practical case study and framework for the development of future M&E systems of social protection programmes, we also hope to call attention beyond rigourous and costly impact evaluations. While these clearly have a role to play, they should be complementary to an even more important ‘rigorously’ designed M&E system. Perhaps, monitoring is considered less challenging from a technical point of view and less revealing in its insights, but M&E systems based on carefully planned MISs linked to other data sources for triangulation can be cheaper, more effective and more useful for the “intended users” than ad-hoc studies which too often end up not effectively interacting with the implementers and do not contribute to building capacity in national systems. The remainder of the paper is structured as follows. Section 1 provides an overview of the theoretical design of a monitoring system for a poverty-targeted cash benefit, identifying key indicators, data sources and institutional actors. Section 2 looks at how the monitoring system was actually implemented in Moldova for the social support benefit, while Section 3 sets out some practical examples of how the system was used, influencing the implementation of the benefit. We then provide some conclusions.

Oxford Policy Management

Working Paper


3

2 Designing an M&E system for targeted cash benefits According to the Organisation for Economic Cooperation and Development (OECD) and to commonly accepted DAC terminology, monitoring can be defined as a “continuous function that uses the systematic collection of data on specified indicators to provide management and the main stakeholders of an on-going development intervention with indications of the extent of progress and achievement of objectives, and progress in the use of allocated funds”. Evaluation, on the other hand, is defined as the “systematic and objective assessment of an on-going or completed activity, program or policy, its design, implementation and results. The aim is to determine the relevance and 3 fulfilment of objectives, development efficiency, effectiveness, impact and sustainability” . Based on these definitions, it could be stated that every M&E system needs to accomplish four main tasks: 1) determine relevant indicators; 2) measure those indicators; 3) set targets against which to compare updated measures of the indicators; and, 4) provide findings to stakeholders and decision makers. By achieving these objectives, a well-designed M&E system can perform its basic function of informing policy implementation, while helping to correct and adjust for unforeseen problems or implementation errors. Following a framework developed by Chen, we argue that the exact mix of monitoring-focused or evaluation-focused activities needed to effectively implement government programmes will depend on the level of ‘maturity’ of the programme itself. For example, at the planning stage, it will be important to “provide pertinent information (…) to help stakeholders in developing the programme’s rationale and plan”. This could be done through needs assessments and analysis of existing data. Similarly, at the initial implementation stages, the main focus will need to be on troubleshooting implementation problems through on-going “process monitoring”. Once activities have stabilised and the programme has reached its “mature implementation stage”, monitoring activities can start to be accompanied by more in-depth ‘evaluative’ studies to further improve implementation, assess user satisfaction, and strengthen accountability. Only when a programme has been operating for enough time to ensure the delivery of outcomes is it useful to develop a final evaluation aimed at assessing to what extent it has achieved its goals (Chen, 2005). As Chen succinctly puts it, “program evaluation has, across, much of its history, focused on outcomes. Lessons from the field, however, have plainly taught that programme failures are essentially implementation failures”. In this paper we hope to show how to tackle such failures through a carefully constructed system that integrates the advantages of both monitoring and evaluation, without putting the cart before the horse – evaluation before adequate monitoring. The rest of this section aims to provide an ideal framework that focuses on three of the largest practical constraints that ministries face when attempting to set up and M&E system for cash benefit programmes. First, the selection of appropriate and easily measurable indicators and targets to be used to monitor and evaluate the cash benefit programme. Second, the identification/adaptation/creation of data sources to be used for analysis. And third, the complex institutional set-up which is needed to coordinate the M&E process and make sure that the information generated is used effectively by policymakers.


Working Paper


4

2.1 Indicators and targets Developing indicators and targets for a programme’s monitoring system depends on two main considerations: the programme’s objectives and theory of change, and its administrative structure. These will be analysed in turn in the next few paragraphs. In this paper, we focus on poverty targeted cash transfers to which some conditionality may be attached as part of the underlying theory of change. Depending on the specific country context, targeted cash transfers are sought to protect the most disadvantaged, reduce inequality and to have a bigger impact on poverty with a limited amount of financial resources (see Besley and Kanbur (1993)). The main challenge of targeted cash transfers is therefore to try to identify the poor in the most effective way given the country circumstances (budget, decentralisation, capacity, etc). This could result in a variety of different targeting approaches, the description of which is beyond the scope of 4 this paper (see Coady et al. (2004) for an overview) . Nevertheless, no matter what the specific set-up of a targeted cash transfer, it is useful to describe the process through which indicators and targets are determined on the basis of programme characteristics. As a case study, we focus on (means tested) guaranteed minimum income benefits as this is the targeting approach that was adopted in Moldova. Case study: Guaranteed minimum income benefit This targeted benefit calculates a ‘guaranteed minimum income’ (GMI) for all people with incomes below a defined level, giving them the right to receive a benefit equal to the gap between the GMI and their income. If implemented properly, this programme design has the biggest impact on incomepoverty when compared to any other benefit that makes use of the same budget (Besley and Kanbur (1993)). However, implementation is usually complex as it requires an advanced administrative system and it also needs to be accompanied by incentives to work (see Grogger and Karoly (2005) for examples in the US, and Adam et al. (2006) for examples in the UK). The underlying theory of change is that, by safeguarding certain minimum income levels, destitution is prevented allowing people to overcome a possible period of difficulty and at the same time protecting some groups from chronic poverty. Such income-gap benefit can also be an entry point to offer other services that are necessary for a full personal development (employment services, social services, etc.). The essence of the income gap benefit can be captured in a graph. Ideally, at any point in time, all people with an income below the GMI should receive a benefit equal to the difference between the GMI and their actual income. Taking a snapshot of this situation, we could rank people based on their income from the poorest to the richest and put these on a graph with income per capita on the Y axis and the cumulative number of people on the X axis. We would then obtain a line that cuts the graph diagonally from the lower left corner up to the higher right corner (income line). The starting point, steepness of the curve and its irregularity depend on the specific circumstances of each country, and in Figure 2.1 we provide a simple exemplification for illustrative purposes. If this is a graph representing all people in the country/region of interest, the effect of the policy would be to take everyone to a minimum level of income and thus reduce income poverty.


Working Paper


Figure 2.1

5

Illustration of an ideal Guaranteed Minimum Income benefit

Income Overall budget + average benefit

GMI Number of beneficiaries

People

Source: authors

Estimations of the ‘income line’ can be made through an appropriately representative household survey, an analysis of which allows the calculation of overall targets of number of beneficiaries, the average amount of the benefit and overall required budget, as well as on the extent of poverty reduction at different poverty lines. However, the graph in Figure 2.1 only captures a specific snapshot of people’s circumstances. These need to be supplemented by information related to the extent to which people’s income changes over time. Indeed, we could imagine two very different scenarios. First, a simple scenario in which the same people maintain the same income throughout the year. Then, a more complicated scenario whereby the national income distribution remains the same throughout the year, but the people falling below the GMI are different in the first and second half of the year. In this second scenario, the theoretical number of people eligible for the benefit would be twice the number of the first scenario, while the overall budget expenditure could remain the same in the two scenarios, provided that changes in people’s circumstances are detected immediately. In practice, the required level of administrative flexibility will depend on the nature of the targeted poverty (households falling below the GMI), i.e. whether poverty tends to be a chronic or transient phenomenon and the duration of poverty spells. This will also depend on the general degree of mobility in the country, demographic factors (household changes), the degree of economic change, vulnerability and general attitudes towards risk (expectations on such dynamics should determine the rules of granting the benefit and for income reporting). In any case, when looking at the dynamics of the benefit, it is important to track indicators relating to how long people receive the benefit, while also considering the impact that the benefit could have on people’s attitudes towards risk and work. More specifically, we could think about the possibility that cash social assistance may generate dependency and unwillingness to keep working or looking for employment. It is in this context that it is possible to make an assessment of the role played by possible incentives to work included in the policy (such as the income disregard).


Working Paper


6

Implications for policy or ‘objective-related’ indicators From all this it emerges that the monitoring of a GMI benefit could include the following key policy indicators: the number of recipients, the average amount of the benefit, overall budget expenditure (including the administrative costs for the delivery of the benefit), poverty reduction (at appropriate poverty lines), poverty dynamics (including the period during which households fall under the GMI), and the dynamics of labour market participation and then any eventual indicator directly related to the theory of change that underpins the promotion of the specific cash transfer (see Table 2.1).

Table 2.1

Policy or ‘objective-related’ indicators

Objective

Indicators

Poverty reduction

Number of recipients, average amount of the benefit, poverty reduction, poverty dynamics

Stay within budget

Overall budget expenditure

Avoid creating dependency

Dynamics of labour market participation

Specific outcomes linked to the theory of change

Human capital, economic activities, reduction of inequality, etc.

Developing administrative or ‘process-related’ indicators However, all the above indicators are related to the policy objectives, but do not say anything about its administration. How do we identify, reach and serve people with income below the GMI (or in the case of other targeted benefits below a certain PMT score or meeting the definition of ‘vulnerable’)? Administrative systems can be very different and critically depend on the specific policy design, existing agencies and available human resources/budget of the implementing agency, as well as other country-specific factors. Although ‘process-related’ indicators must be based on the specific administrative system, there are some general steps involved in every administrative solution: •

training staff involved in the administration of the benefit,

•

informing the public,

•

determining eligibility and enrolling beneficiaries,

•

processing and making payments,

•

managing exit from the benefit, and

•

dealing with complaints.

For each of these steps there could be a range of specific indicators, aimed at tracking the various stages of the process. Some examples are the following: level of knowledge and awareness of the benefit by the programme’s staff and wider public, numbers involved in the administration of the benefit, workload, number of applications and their success rate, reasons for failure, processing time, payment delays, payments not collected, reasons for stopping applications, and satisfaction and complaints (by type). A summary of the key administrative steps involved in the set-up of a cash benefit alongside key process indicators is presented in Table 2.2.


Working Paper


Table 2.2

7

Administrative or ‘process-related’ indicators

Steps

Indicators

Training of administrative staff

Number and quality of trainings, testing of understanding

Informing the public

Type of media campaign, level of knowledge of the benefit (awareness and understanding)

Determining eligibility and enrolling beneficiaries

Overall number of applications, success rate, number of staff involved in the administration of the benefit, workload (time and number of applications per staff), reasons for negative applications, processing time, administrative running costs

Processing and making payments

Payment delays, payments not collected

Managing exit and renewal process

Number of blocked applications, number of nonrenewed and renewed applications, reasons for blockage/non-renewal

Dealing with complaints and enquiries

Number of complaints, reasons for complaints, satisfaction

2.2 Data sources “Too much data can be distracting, or worse, can lead to the data being completely ignored”. Moreover, original data collection for M&E can be an extremely “time-consuming and costly activity” (Morse and Struyk, 2006). It is exactly for these reasons that identifying existing data sources, establishing their usefulness for monitoring purposes, and planning them carefully to deliver exactly the indicators needed for M&E is such an important task. When it comes to identifying such data sources, a range of options is available, depending on the budget, the depth of analysis needed and the availability of data. In particular, we identify five main possible sources of information, described briefly below and in more depth in the following sections: household survey data available in the country; the programme’s administrative database; additional administrative information from other sources; ad hoc household surveys; and, ad hoc qualitative assessments. Household survey data, usually collected by the national or regional statistical office on a regular 5 basis, can be used to determine targets on most of the policy indicators from the time when a poverty targeted benefit is being designed. As this paper will describe in section 2, the depth of information obtainable from national surveys can be increased significantly by adding questions or specifically tailored modules on the programme to be monitored, including information on awareness and reasons for non-application. A second set of policy indicators – including the actual number of beneficiaries, the average benefit, and duration of assistance – cannot be calculated using household surveys alone, which can only provide a nationally representative estimate. Instead this information should be monitored using a programme’s administrative database, which is also the main source used to measure and track most administrative indicators. It should be noted that it is in the policy-maker’s interest to design this database in such a way as to capture all relevant information for the administration of the benefit.


Working Paper


8

Other than the main administrative database, there are usually other administrative sources of information for which it is important to secure access. These include the staff database, actual bank payments, information that can be used to verify declared incomes (for instance the pension database and other social transfer databases), and information from agencies otherwise involved in the process of determining eligibility. Depending on the specific design of the benefit, these could include employment agencies, schools, hospitals and banks. While administrative indicators and some of the policy indicators need to be continuously monitored, the assessment of complex policy indicators requires a more in-depth analysis that adopts an ‘evaluation perspective’, attempting to understand causal relationships and generate counterfactuals (e.g. what would have happened if the benefit did not exist). Policy indicators requiring this in-depth analysis include: the targeting of the benefit; its use; impact on consumption, simulated poverty, and access to education and health; duration of assistance and movements in and out of benefit; and, the effect on dependency and labour market participation. Although some of such assessments can be made with existing household survey data, rigorous impact assessments usually require the design of additional ad hoc household surveys. The need of such surveys should be assessed based on the importance of the programme and the survey’s actual value added, given their large cost. All of the data sources above provide quantitative data that can be used to statistically analyse the roll-out of a social benefit, providing invaluable insights into the overall functioning of the programme and often highlighting problematic trends. However, they do not help to understand why and how things are going wrong, and, most importantly, how such problems could be addressed and solved. The most effective way to collect such information is to use the insights from the analysis of quantitative data to design ad hoc qualitative studies. Based on interviews and focus groups with administrators, benefit recipients and other households, these studies quickly gather information at relatively low cost, helping to explain problems and trends and gain a practical understanding of the problems at hand.

2.3 Institutional arrangements Setting up the institutional arrangements to monitor cash transfer implementation depends on the existing institutions, their relative strength and the data sources available. Institutions and their relative strengths should be assessed taking into account the specific context of each country, so here we discuss some of the institutional pre-requisites in which data sources should be administered. Household survey data are usually organised and administered independently from the implementing organisation by national or provincial statistical offices. As explained above, the scope of analysis of these surveys can be increased significantly by adding relevant questions on the benefit to the standard modules. This requires continuous coordination with the offices responsible for the delivery of the surveys, including a substantial effort to argue the case for additional questions (which lengthen the questionnaires, increasing their cost). Administrative databases often only provide limited information on number of beneficiaries and amounts of benefit, but can be carefully designed to offer much more information useful for the ‘management’ of the benefit. In order to do this, it is essential for administrative data to be collected and generated as an easy ‘side-product’ of the application process itself. For instance, data collection will be ensured if applications must be entered in the database in order to decide eligibility and the size of the benefit. Conversely, if eligibility (and the size) can be determined independently, and there is a separate requirement of introducing information about applicants, a risk of missing, incomplete or less accurate information is inevitable. In fact, data collection will be seen as an extra task whose importance is definitely lower than actually providing the benefits. The responsibility for designing and creating a programme’s main administrative database usually lies within the agency in charge of implementing the benefit. To do this, it can either be incorporated in an existing database (of other benefits) or be developed independently, while ensuring that it is connected to a specific infrastructure for the payment of the benefits. Information on payments and actual banks payments will need to be accessed regularly and cross-checked with administrative information. Similarly it will be important to identify other administrative databases for which it Oxford Policy Management

Working Paper


9

would be useful to have access to (staff database, employment database, pension database, etc.) whereby adequate institutional arrangements should be made in advance. Finally ad hoc studies, whether quantitative or qualitative, will need to be implemented within national or provincial Monitoring and Evaluation units of the implementing agency, able to either directly conduct or commission such studies from third parties. Such units, together with statistical offices, will also need to be responsible for most of the data analysis, and be in charge of the coordination of information systems, data sharing and consolidation of findings. However, the institutional set-up must allow space and procedures to share and explain relevant findings and reports with implementation managers and policy-makers depending on the messages that need to be delivered. This should be done both at the national and local level. All these institutional relationships need to be carefully defined to ensure good coordination and data access and, at the same time, responsibilities need to be clearly assigned to persons and units within the different agencies. When required, information is held in different agencies the difficulties involved in data access and sharing should not be underestimated. Finally, in order to make the system work, it will often be necessary to invest in strengthening human resources, while creating adequate opportunities for sharing information with managers and decision makers who are all too often separated from the monitoring and implementation process.


Working Paper


10

3 Implementation of Moldova’s new cash benefit 3.1

Overview of the social assistance reform

In 2008, the Government of Moldova (GoM) undertook an ambitious reform of its social assistance 6 system, moving gradually from a system of category-based nominal compensations for individuals to a system of poverty-targeted cash benefits for households.

3.1.1 Background and objectives of the reform The drive behind the reform was to improve the targeting of the previous system, under which a large percentage (21%) of the population received relatively small amounts of social assistance (‘nominative compensations’ and child allowances). According to estimates performed on the country’s Household Budget Survey (HBS) only 20% of the social assistance budget was received by the poorest 10% of population. Moreover, the average amounts received varied between MDL 11 and 50 (about 2USD per capita on average) (Carraro, 2008) – an extremely low amount to have a relevant impact on consumption levels of the poor: in fact even for the poorest decile on average such compensations represented only 8.6% of consumption expenditure (Carraro, 2008). In order to address this problem, Moldova’s Ministry of Labour Social Protection and Family (MLSPF), with support from Oxford Policy Management and EveryChild Moldova within a DFID-SIDA funded project and in coordination with development partners active in Moldova (DFID, SIDA, WB, UNICEF, EU, etc), designed a ‘zero cost’ reform to reduce the number of social assistance recipients, include only poor people and substantially increase the size of the benefit. Eligibility for the new social support system (commonly known as ‘Ajutor Social’) is determined by an 7 innovative set of criteria. First, household income must be lower than a certain threshold, the GMI. Second, households need to satisfy a ‘Proxy Means Test’, based on key household characteristics which match those found to be typical of very poor households (low ownership of durable assets, low 8 levels of education, etc). Third, specific incentives to work are built into the policy design. For example, in order to receive the benefit, able-bodied people declaring they are unemployed must be registered with the employment agency and actively looking for jobs. Once households are selected as eligible, the amount of their entitlement is automatically calculated using an ‘income gap’ approach: each household receives the difference between the amount they are entitled to using the GMI and their overall income. Given high levels of social mobility in Moldova, 9 the new benefit is only awarded for a period of six months at a time. Once that time is expired, households have to re-apply and supply new information on their household circumstances in order to continue receiving the benefit.

3.1.2 Implementation of the reform Implementing such a complex and innovative reform was not an easy process, especially given the practical and psychological expectations involved in the previous system. However, unlike other cash 10 transfers in the country, its administration was made possible thanks to a new network of more than 1,000 social assistants deployed across communities in Moldova (as of 2007) and managed by the raion-level Social Assistance Departments (SADs). The application process for the new benefit is relatively simple. Households apply through social assistants in their own communities who help them to fill in the application (a 6 page form with questions on income and household characteristics) and make sure it is complete and contains all the 11 necessary supporting documents. Social assistants then take the forms to the SAD in the district centre, where a designated operator enters all the relevant information into a computer database called the Moldova Social Assistance System (MSAS). On the basis of the information provided, MSAS processes the applications, determining their outcome based on the eligibility criteria described above. All applicants then receive a letter explaining whether they are entitled to receive the benefit, Oxford Policy Management

Working Paper


11

and, if so, its amount and duration – a major change compared to previous benefits in the country where applicants had to enquire personally whether their application was successful and what amount they were entitled to. In the meantime, lists of recipients are compiled nationally and passed to the bank responsible for processing the payments, with the help of the Territorial Social Insurance Houses (TSIH), remnants of the previous system of social assistance. As a result of this process, a wealth of information is stored in the benefit’s administrative database, MSAS. This paper argues that the use of that information, together with other sources, to set-up a comprehensive M&E system that tracked the roll-out of the new benefit, was invaluable for the implementation of the reform. Helping to identify problems and misunderstandings and thus allowing for early intervention and improvement of policy effectiveness. We believe that important lessons were learned from this M&E system – an example of good practice that could be useful for many other countries intending to implement similar policies. The next few paragraphs follow the Moldova case study, highlighting how the conceptual framework introduced in the previous section was applied to monitor the new minimum income cash benefit. This will include practical examples of how indicators were developed, data sources identified and institutional arrangements established.

3.2 Identifying targets and indicators: the logframe The M&E system for the new social support in Moldova was built around a logical framework that identified the main goals of the new benefit, the necessary outcomes to reach those goals, as well as the required activities (inputs) and outputs. The logframe identified indicators for each stage and the relevant data sources that had to be created or modified. Table 3.1 presents an adaptation of the original logframe, showing the main groups of indicators and relevant sources of information.


Working Paper


12

Table 3.1 Simplified logframe for the minimum income benefit Objectives

Indicators/targets

Data sources

Extreme poverty indicators (head-count, poverty gap, severity of poverty)

Household Budget Survey

Goal Extreme poverty reduction

By 2010 compared to a situation without the benefit poverty is reduced by 40%

Outcomes Cash benefits targeted to the poorest Encourage self-dependency among people able to work

Coverage of the poorest decile (target by 2010 = 40%)

Household Budget Survey (new questions)

% of people in poorest decile among recipients (target by 2010 = 50%)

Social support administrative database (MSAS)

% of recipients in working age (excluding disabled, students, caregivers) who are not working

Labour Force Survey (new question)

Outputs Money is disbursed and households receive the benefit

Number of beneficiaries (individuals and hhs)

Social support administrative database (MSAS)

Functional database

Amount disbursed

Equipment is used appropriately by social assistants

Number of social assistants

Internal data from the Ministry of Labour, Social Protection and Family

Network of trained social assistants is formed

Ability of social assistants (qualifications)

Good management/ efficient use of resources/time

% of social assistance departments/directorates with adequate qualified personnel

Awareness of the new benefit

case load per social assistant

Household Budget Survey (new questions)

processing time social assistants’ time allocation % of people aware of new cash benefit

Activities 1) Legal framework (law and regulations) 2) Database/infrastructure (creation of payment and information system, development of administrative procedures) 3) Training (social assistants and other staff involved in the delivery of the benefit) 4) Media campaign


Working Paper


13

3.3 Data sources in Moldova 3.3.1 The Household Budget Survey and Labour Force Survey In Moldova, the nationally representative household survey chosen for monitoring the benefit is the 12 Household Budget Survey (HBS), a continuous activity of the National Bureau of Statistics (NBS) that has a panel component (ensuring that a certain percentage of households is interviewed for consecutive years). The survey regularly collects comprehensive information on income, consumption, household composition, housing and the ownership of assets, health status, the education of household members, as well as other relevant indicators. Analysis of the HBS proved invaluable at various stages of the policy cycle, from design all the way through to implementation. During the design stage the HBS helped to: •

determine the GMI that the Government could afford with the available budget, estimate the approximate number of households eligible for support, and the overall budget required to fill the income gap. In other words, the HBS was used to estimate the empirical income line reported in Fig 2.1;

•

develop a set of indicators of household living standards to be used as a proxy means test 13 aimed at preventing income under-reporting;

•

calculate the number of eligible households in current and future years and the poverty impact 14 of the benefit using a simulation model;

•

on this basis, identify a set of targets concerning poverty reduction, targeting of the benefit, number of eligible households, average benefit and overall required budget.

In view of the implementation of the benefit, moreover, the Ministry of Labour, Social Protection and Family asked the National Bureau of Statistics to include some new questions in the HBS to understand whether people were aware of the benefit, applied for it (and if not, why not), and received it. Thanks to this wealth of data, analysis of the HBS during implementation was used to provide: •

information on the main household characteristics of eligible households (demographic characteristics, income sources, housing and assets). These could then be compared to that of applicant households so as to analyse any discrepancies and understand issues around benefit take-up;

•

external monitoring on awareness of the new system and take-up of the benefit (including reasons why people had not applied if they hadn’t)

•

M&E of the benefits targeting and simulated impact on poverty.

Another important household survey conducted by the National Bureau of Statistics in Moldova is the Labour Force Survey, which monitors employment and unemployment rates. This is also a continuous survey with interviews conducted every month. Moreover, since the same people are interviewed more than once (panel data), it enables an understanding of the population’s movements in and out of employment. As with the HBS, the Ministry asked the NBS to include a simple question on receipt of the benefit within the survey questionnaire, in order to investigate potential problems of disincentives to work for benefit recipients. It is important to note that, in this context, the existence of a relatively good and flexible system of national household surveys with satisfactory sample sizes reduced the necessity for conducting a separate impact evaluation survey. In fact, both the HBS and LFS capture the most important questions on the effectiveness of the new benefit.


Working Paper


14

3.3.2 MSAS and other administrative databases Before the new benefit was introduced the main source of information on cash benefits in Moldova was a separate institute, the National and Social Insurance House, which still has the task of processing benefit payments. However, the NSIH database only used to provide information on the number of recipients and money disbursed. The new system was carefully designed to improve the quality and variety of information stored for monitoring and administration purposes. Use of the MSAS database is an integral part of the application process, as eligibility is determined on the basis of the information that is entered. This means that much more data is collected and stored, with no extra data-entry burden, ensuring the sustainability of the system.

Box 3.1

MSAS, in practice

The backbone of the new administrative database is a 13 digit personal identification number called the IDNP. This number uniquely identifies individual applicants and their adult family members (who constitute the targeted ‘household’), preventing the risk of duplicates and ghost applications while also simplifying matches with other databases. In order to do this, every time that the system processes an application, it verifies that there are no other ‘live applications’ containing the same IDNPs, effectively stopping the possibility of having the same person in more than one application at the same time. As applications are processed, each is given a second ‘programme’ ID: a number made of a district number, the social assistant identification number and the progressive number of applications processed by the same social assistant. This allows the system to verify and check the work of each social assistant, at least for applications that arrive to the stage of being processed. The processing of each application through the MSAS database consists of three basic steps: 1)

registration of the application (where identities are checked through IDNPs, and in case of re-applications old applications are stopped);

2)

entry of all application data into the MSAS database and checking the consistency of completeness of data (automated checks), and;

3)

decision on the application with issuing of a letter for the applicant communicating the outcome of the application.

In the process of determining eligibility and entitlements MSAS generates and stores a number of variables capturing declared income, proxy test, and work requirements. The programme also guarantees the possibility to update a set of parameters over time: GMI, the income disregard, proxies and associated coefficients, as well as cadastre scales used to assess agricultural income based on amount of land and its fertility. Finally, based on entered applications, MSAS can generate payment lists that are then transmitted to the TSIHs. Such payment lists are compiled nationally by the NSIH and then communicated to the banks, which execute the payments. MSAS retains information from all applications irrespective of whether an application has a positive or negative outcome. Ministry staff then compile data files on a regular basis into a national database, 15 generating an automated set of tables and conducting ad hoc analysis where possible.


Working Paper


15

‘Administrative’ tables that are automatically generated by MSAS include: •

the number of beneficiaries per month;

•

the average benefit amount (and distribution of the benefit amount);

•

the number of applications and percentage of successful and negative applications, including the main reasons for the negative outcomes;

•

the characteristics of applicants and beneficiaries (household composition, income sources, housing and asset ownership), and;

•

benefit dynamics - people’s entry and exit into the programme and the average length of receipt of benefit.

While these tables are very useful in their own right, they can shed further light when they are compared to other data sources. For example, payment lists are separately available from the NSIH. Similarly, banks provide information on executed and not collected payments. When such information is cross-checked against MSAS data, it helps to identify any eventual discrepancies, as well as the size of uncollected payments. Most importantly, administrative tables on beneficiaries, average benefit amount, and characteristics of beneficiaries can also be compared to similar information from the HBS. This can be done both on recipients and on simulated eligible households, to check whether there are significant differences and providing very useful feedback on socio-economic groups that may be hard to reach, or insights on possible problems in income under-reporting. An example of this can be seen in Table 3.2 below.

Table 3.2

Example of HBS-MSAS comparison All eligible households (HBS) (%)

Actual beneficiaries (MSAS) (%)

Rural

85.1

84.1

Urban

14.9

15.9

Families without a disabled household member

82.3

74.9

Families (households) with 1+ members abroad

7.9

5.1

Source: OPM calculations

However, in making such comparisons, one should always be aware of the differences existing in the two data sources. For instance, in determining eligibility and amount of the benefit, differences between the two sources can be related to the time in which assessments are made, to the viscosity (slow adjustment) of administrative payments, and to the different ways assessments and measures are made (for example, if there are delays in processing payments, initial payments could be made of cumulative entitlements, so that disbursements divided by number of recipients could provide the wrong information on average benefit amounts).


Working Paper


16

MSAS also produces a second set of tables related to the management of the administration of the benefit. These include: •

the number of social assistants processing applications;

•

the number of applications per social assistant;

•

the time required to process applications (backlog);

•

the type of mistakes in applications;

•

the presence of suspicious applications (positive applications, but with information that appears to be unusual);

•

the re-application process (% of expired applications that were not renewed); and,

•

positive applications blocked as a result of checks.

All such information can be created flexibly for the whole country or for specific districts, even allowing the analysis of the performance of individual social assistants. This enables managers to identify problems and correct them with targeted explanations, training or measures.

3.3.3

Ad hoc rapid assessment studies

In a few occasions, analysis of administrative and HBS data posed questions that could not be answered by the analysis of data itself. In these cases, additional qualitative research was needed to triangulate information and explain why certain trends were being picked up in the quantitative analysis. As highlighted by Garbarino and Holland (2009), qualitative research can “explain relationships/ trends/ patterns emerging from surveys” and other quantitative sources. In Moldova, research was conducted through rapid assessments with interviews to beneficiary and non-beneficiary households, social assistants and SAD specialists. These activities included investigations on the reasons for the low take-up of the benefit and poor retention (lack of reapplications), as well as a qualitative assessment on the use of the benefit and its preliminary impact. Although these assessments could not by definition provide fully representative national estimates, they were based on the sampling of representative cases based on HBS and MSAS data. Moreover, they followed clear interview protocols and continuously triangulated information, thus providing a rigorous and reliable understanding of the implementation issues faced in the various communities, 16 with the advantage of providing quick and detailed answers.


Working Paper


17

3.4 Institutional arrangements In the Ministry of Labour, Social Protection and Family two units are involved in the monitoring of the benefit: the “Social Assistance Department” and the “Policy Analysis, Monitoring and Evaluation Department”. The Social Assistance Department is in charge of the implementation of the policy, whereas the Monitoring and Evaluation department has an overall view of the legal framework and its impact on policy. The Social Assistance Department supports the districts in the implementation of the benefit, providing supervision, instructions, and advice. It is also responsible for the compilation of all the MSAS data and for the generation of the automated tables described above. The Monitoring and Evaluation Department liaises with the NSIH and the National Bureau of Statistics, obtaining the NBS’s household survey data. It is responsible for analysing the tables that come both from MSAS and the NBS, and – together with the Social Assistance Department – identifies the need for possible ad hoc rapid assessments.

3.4.1 Scope for improvement While this process now works relatively smoothly, it was not easy to set up and required considerable capacity building of ministry staff (complicated by very high staff rotation) and negotiations with external actors such as the NBS. At an institutional level, it has also not been easy to set up incentives for the full potential of the monitoring system to be taken advantage of in day to day administration. In particular there were human resources constraints with few overburdened Ministry staff. Moreover, there is scope to improve the current monitoring system in two main areas: actual payment execution and grievance system. On one hand, payment information from the NSIH and banks is not being sufficiently used to detect payment delays and possible fraud. On the other, there is currently no systematic procedure for the monitoring system to capture and address complaints. Systematic complaints assessments through complaint forms could be used to investigate possible errors in the targeting criteria and, more in general, could be an additional source of information on possible implementation problems.


Working Paper


18

3.5 Summary overview of the M&E framework A summary of the overall M&E approach is reported in Figure 3.1. As clearly shown, the main indicators are derived from the triangulation between various different data sources.

Figure 3.1

Summary of M&E approach for the social benefit

Analysis of administrative data (MSAS + NSIH)

Analysis of national sample survey (HBS + LFS)

Payments

Total eligible households

Percentages of successful applications (why?)

Characteristics of eligible households

Characteristics of applicants/beneficiaries Average entitlements Suspicious applications

Expected entitlements Comparison Issues around take-up inclusion and exclusion

Actual targeting of the benefit Poverty impact Dependency problems (LFS)

Performance of DASs and individual social assistances processing social applications Design Understanding

Understanding Qualitative research and in depth studies

Source: Authors

The MIS administrative database provides the bulk of the information for programme management, including detailed information on payments, percentage of successful applicants and reasons for 17 rejection, characteristics of applicants and of beneficiaries , average entitlements, suspicious 18 applications , and performance of Social Assistance Departments and individual social assistants. This information is then matched and compared to data from national sample surveys such as the Household Budget Survey and the Labour Force Survey, that give an overview of the number and characteristics of eligible households across the country. The comparison allows understanding 19 issues around take-up , targeting and poverty impact, among others. Problems highlighted by the quantitative data – where a trend is visible but there is no clear understanding on why it is happening or how it could be solved – can then be explored in more detail through qualitative research by small teams from the ministry. Respondents for these studies can be purposively sampled according to their characteristics, using the MIS to sample applicant households and the national sample survey to sample other respondent types (for example eligible nonbeneficiary households).


Working Paper


19

4 How the M&E system improved the implementation of the new benefit In the previous section, we described the data sources available for the M&E of Moldova’s new social benefit and how these were ideally set up to quickly and flexibly gather information for monitoring. In this section, we present a few cases where such information and a triangulated approach to M&E helped to solve serious problems in the implementation of the benefit.

4.1 Low take-up and retention of the benefit 4.1.1 The problem About a year into the roll-out of the benefit, in December 2009, the monitoring numbers that were being automatically produced through MSAS started to highlight a stall in applications. Moreover, when data from MSAS was compared to data from the HBS, it became quickly apparent that of the 71,000 households estimated as eligible, only some 21,000 were receiving the benefit in December 2009 – that is 29.5% of the expected outcome. More alarmingly still, during the course of the whole year, some 31,500 households successfully entered the programme, but 10,500 stopped receiving 20 the benefit because they did not re-apply. Most of these never re-applied though they received an 21 average monthly benefit of just under MDL600 lei (Barca et al., 2010). Figure 4.1 below helps to highlight the presence of these two large problems in the implementation of the new benefit: in December 2009 about 39,500 households still needed to be reached (low takeup) and a further 10,500 households needed to re-apply to continue receiving the benefit (low retention).

Figure 4.1

Take-up and retention problems – December 2009

Thousands of households

75

No take-up

60 39.5 45

No retention 71 30

10.5 31.5

15 21 0 Eligible

Recipients in Dec.

Overall recipients

Source: MSAS and HBS (2009) based on analysis of the authors


Working Paper


20

4.1.2 M&E as a solution The problems of low take-up and retention described above could only have been uncovered thanks to the on-going monitoring of the number of applicants and actual recipients (successful applicants) using data from MSAS, comparing this to the number of eligible households from the HBS. However, identifying the problem was only the first step. In order to find a solution, it was necessary to understand why so few people were applying and why many households were dropping out of the programme after having received the benefit for six months. The first hint was given by the HBS. One of the additional questions the Ministry had asked the NBS to add to their survey was on the ‘reasons for non-application for social support’. In other words, all households interviewed who had never applied for the benefit were asked the reason for not applying. The results, shown in Table 4.1, gave a very clear picture of a worrying (though improving) trend: awareness of the benefit was still extremely low in the fourth quarter of 2009, and especially so among eligible households. Interestingly, some 35% of eligible households in Q4 of 2009 believed they were not eligible for the benefit.

Table 4.1

Reasons for non-application for social support Total population

Reason for non-application (%)

Q1

Not aware Not eligible

Eligible

Q2

Q3

Q4

Q1

Q2

Q3

Q4

47.3

36.3

33.0

28.0

70.1

47.9

46.1

30.7

38.9

46.1

49.8

57.1

12.5

14.8

26.1

35.2

No need assistance

7.0

6.5

7.1

4.6

1.6

1.9

1.4

3.6

No necessary documents

1.8

3.3

3.2

3.0

4.8

8.9

2.5

5.3

Office too far

0.3

0.3

0.1

0.1

1.0

0.0

0.0

0.0

Administration too complicated

2.8

4.0

3.9

4.3

3.1

7.1

9.5

6.1

Other

0.1

0.0

0.0

0.0

0.0

0.0

0.0

0.0

Total

100

100

100

100

100

100

100

100

Source: HBS (2009), developed by OPM

However, the information provided by the HBS was still not sufficient. Why were households still so unaware of the benefit despite the mass media campaigns organised by the Ministry? And why were so many eligible households convinced they were not? The only way to find an adequate answer was to organise an ad hoc qualitative assessment, based on findings from the quantitative data. First, administrative data from the M&E helped to formulate a set of research hypotheses for the assessment. For example, MSAS data was clearly showing that current applicants for the benefit mostly belonged to those same categories of people that were targeted by the previous nominative compensation system: disabled people and mothers with many children. Moreover, it was showing that only people who were entitled to very high benefits were applying, so much so that the average benefit in December 2009 was higher than the average benefit projected if all eligible households had applied for a difference of almost MDL 100. Second, data from the HBS and from the MIS were used to sample specific types of households to be interviewed and the localities to be visited. Specifically, the HBS was used to sample eligible but nonapplicant households (to look into issues of low take-up) and the MIS was used to sample households that had applied once and not yet reapplied for the benefit (to analyse low retention). The cost of such an undertaking – with a total of 51 semi-structured interviews – was low, and the results were immediate and unequivocal. In short, the assessment found that the poorest people were the least likely to be reached by or to understand the Ministry’s TV and radio media campaign. It also Oxford Policy Management

Working Paper


21

found that social assistants around the country were either completely passive in spreading the word on the new benefit, or at times actively discouraging people from applying. At the same time, many households were finding it difficult to interpret the bureaucratic language of the automatically generated letter they received from MSAS after applying, meaning that they did not understand that they were entitled to re-apply if they were still in need after six months. While details of the findings and recommendations can be found in the assessment’s report, ‘Study into Reasons for Low Take-Up and Retention’ (Barca et al, 2010), it should be noted that a set of measures were set in place as a direct consequence of the evidence generated. Among other things, a poverty-targeted communication campaign was rolled out in local communities, using simple and non-stigmatising posters for village billboards and user friendly leaflets; social assistants received tailored training; and the MSAS letters were simplified. Once again, thanks to the benefit’s monitoring system, the results of these interventions were immediately quantifiable – testifying to their success. Figure 4.2 reports the number of households receiving the social support benefit from January 2009 until October 2010. Four clear phases are immediately visible, modelling the trends described above. The first phase, until July 2009, marks the initial roll-out of the benefit, with the number of beneficiaries slowly increasing. The second phase shows the subsequent stagnation (because of lower take-up and low retention) which called for the qualitative assessment. The success of the new poverty-targeted communications campaign – designed on the basis of careful M&E – is testified in the third phase, with applications rising sharply again in January/February 2010. Nevertheless, a new phase of stagnation was evident at the end of 2010, mostly linked to new automated checks performed by the Employment Agencies (which led to some 5,500 blocked applications).

Figure 4.2

Number of beneficiaries and average household benefit Communications campaign

Initial roll-out 40,000

Number of beneficiaries

35,000 30,000 25,000 20,000 15,000 10,000 5,000

10/10

09/10

08/10

07/10

06/10

05/10

04/10

03/10

02/10

01/10

12/09

11/09

10/09

09/09

08/09

07/09

06/09

05/09

04/09

03/09

02/09

01/09

0

Source: MSAS (November 2010)


Working Paper


22

4.2 Other implementation problems and their solutions On-going M&E of the benefit highlighted a large set of other problems, kick-starting processes to resolve them. These included: •

Under-reporting of income: highlighted by the analysis and comparison of administrative and survey data, it became necessary to question some of the self-declared information with home visits. This also resulted in a review of the proxies to identify poor households and complement income declarations.

•

Identification of suspicious applications: using administrative data, lists of households with particularly low incomes or household members abroad and no remittances, were drawn and sent to the relevant districts for checks.

•

Clarifying the status of pregnant women: pregnant women don’t work, but cannot yet be considered as looking after a household member. To circumvent this problem, social assistants were considering these women as employed, but they then reported an unrealistically low income. This prompted a change in the legislation, which was then reflected in the administrative/computer system.

•

Modifying calculation of agricultural income through cadastre scales: income from selfemployment in agriculture is computed based on the amount of land owned by the household through specific cadastre scales that take into account the location and quality of the land. From the analysis of HBS data, but also from the interviews with households and social assistants, it became clear that it was not correct to attribute cadastre income to households whose members are all unable to work. This resulted in an amendment of the legislation, reflected in the administrative/computer system, allowing the system to ignore cadastre income in the calculation of household income for those households whose members are all elderly or disabled.

•

Problems with the Employment Agencies: monitoring activities revealed that Agencies often refused to register people who were close to retirement age or without certificates to prove their education. Moreover, in the summer period, they refused registration on the assumption that people ‘should be able to find jobs in agriculture’. While this problem has not been resolved to date, the Ministry is now hoping to reduce it through better information and the enforcement of improved checks on the working of employment agencies.

•

Identification of areas where applications were either too few or where there were delays in processing applications: in the first case, the problem was eased by the substitution of absent social assistants with other personnel in the town hall. In the second case, extra computers were deployed to reduce the backlog.

4.3 Providing quick feedback on targeting and initial impact While the focus up to now has been on the resolving of issues arising from the day-to-day implementation of the benefit, this last example relates to a wider issue in the background: government’s support for the new benefit. As with every reform process, some criticism is expected by those who are not directly benefitting. In this case, the government of Moldova swiftly came under significant pressure from the country’s middle classes and farmers’ associations, who questioned the targeting of the new benefit, complained about the risks of dependency, and spread stories around misuse of the benefit on alcohol. In cases like these, it is very important to have information to back the results of the policy in question, going beyond singular cases or anecdotal evidence to show what is actually being achieved. HBS data from 2010 provided an initial solution to the problem, allowing to determine the progress made by the social support benefit in targeting social assistance resources to the poor and to reduce poverty.


Working Paper


23

Figure 4.3 shows the share of budget spent by the new benefit on the poorest 10% of the population and compares it with the same percentage that was achieved by the previous benefit system (‘Nominative Compensations’). The figure also simulates the impact of the benefit on extreme 22 poverty, using a poverty line related to the level of GMI. The simulation computes the level of poverty with and without the receipt of social support, assuming that households’ income without the benefit would be reduced by exactly that amount, resulting in a reduction of consumption of the same size.

Figure 4.3

Targeting impact of social support

Targeting of resources to poorest 10%

Poverty reduction (low poverty line)

Percentage Severity of of poor Poverty gap poverty

80% 68% 0% 60%

-10% -20%

40%

-30% 20%

18%

-15% -26%

-40% -50% -50%

0% Nominative Social Support compensations

-60%

Source: HBS (2010, first 9 months)

The figure shows the three main poverty indicators: head-count (the percentage of people falling below the poverty line), poverty gap (which takes into account the distance from the poverty line), and 23 severity of poverty (which considers the level of inequality among the poor). In all three cases, the simulated reduction of poverty achieved by the new social support benefit is very high. This confirms 24 that the benefit is well targeted and mainly received by the poorest. However, the information provided solely by quantitative data was once again not enough to understand whether the accusations around misuse of the benefit were founded or not. In order to answer the question and assess the overall use of social support money (understanding possible changes in behaviour as a consequence of the social support benefit), a new qualitative assessment was designed, interviewing about 50 households in different localities across Moldova. The study clearly showed that the large majority of recipients use the benefit to satisfy basic necessities such as buying food, medicines and paying for basic utilities. In a limited number of cases, households also made small investments in livestock, while many used the benefit to avoid negative coping strategies that would lead them to deplete their assets. Two of the largest concerns/criticisms expressed on the social support benefit – the risk of generating dependency and the misuse of the benefit for drinking – appeared to be largely overstated (See Barca, Carraro and Sinchetru (2011): Social Support beneficiary assessment).


Working Paper


24

Conclusions Although the social support benefit described in this paper is tailored and specific to the circumstances of Moldova, there are some important similarities with many income gap benefits and poverty-targeted cash transfer programmes in other countries. Therefore, the design of Moldova’s comprehensive M&E system and its significant contributions to the cash benefit’s implementation presents an interesting case-study that can provide useful insights for similar applications. In particular, the way the M&E system was structured, the types of indicators and data sources it used, and the institutional arrangements set in place can all be relevant for the M&E systems of similar benefits. The key winning points of Moldova’s M&E system, as we see it, include its simplicity, low cost, strong links to policy, immediacy in delivering results and the importance it gives to the triangulation of data from different sources, with the administrative data from the programme’s MIS acting as a backbone. The success of the system was testified by the influential role played by M&E in guiding a whole range of improvements/corrections during the implementation of the social support benefit in Moldova. These included: •

a communication strategy better targeted to eligible households and poor communities (using posters, and leaflets rather than stigmatising television ads) that significantly increased takeup of the benefit

•

targeted instructions to social assistants to rectify their misunderstanding on the application and re-application process

•

focus on Social Assistance Departments and individual social assistants that were ‘underperforming’

•

fostering outreach activities to identify potentially eligible households in areas with low take-up

•

identification of suspicious applications that triggered checks on the actual household living conditions

•

recognition and fixing of loopholes in the legislation that were affecting certain groups of people (e.g. pregnant mothers, disabled landowners)

•

activities aimed at addressing the misconduct of employment agencies and other partners

Moreover, adequate information from the monitoring system was also influential in defending the new social benefit from criticisms that were often based on anecdotal evidence of beneficiaries being drunk or unwilling to work. As a famous saying powerfully says, “a single falling tree makes more noise that a whole forest growing”. In order to counterbalance such ‘noise’, M&E data that present objective information on the achievements of a programme or policy are very important. Indeed, M&E data showed a very good targeting of the benefit and use of the benefit money to pay for subsistencelevel household expenses. While the implementation of the new M&E system was not always smooth, with several problems linked to high staff turnover and low capacity at the Ministry of Labour Social Protection and Family, overall there were several important lessons learned that could be usefully applied in other contexts. Among others, these include: •

Developing indicators and targets based on a) the overall policy objectives; b) the cash benefits’ theory of change; c) the whole administrative process and structure.

•

Careful planning of data sources from the very first stages of the policy design. For example, MIS administrative databases can give much more information than the number of beneficiaries and overall disbursement, and provide very useful feedback for management (including workload, processing time, backlog, common errors, payment delays, etc.)

•

Standardizing MIS data to match other data sources in order to facilitate comparison with other national databases (in Moldova’s case the Household Budget Survey). This includes using an individual ID number in the administrative database that can be easily matched (for example a national ID number), allowing for easy triangulation of sources


Working Paper


25

•

Establishing connections with the national statistical agency and negotiating the addition of a short module on the programme to national sample surveys (including the equivalent of the Household Budget Survey and Labour Force Survey). These can then be used to monitor eligible households and take-up, perform targeting analysis, estimate simulated impacts on poverty, etc.

•

Making sure software requirements for the management of the data generated by the Management Information System are as simple as possible (and as close as possible to staff’s preferences). MSAS, Moldova’s basic MIS system, is an Excel-based programme that allows decentralized social assistance operators to enter the data and perform basic checks.

•

Making sure additional administrative data for M&E is collected and generated as an easy ‘side-product’ of the application process itself, not to increase the burden on staff and ensuring higher data quality.

•

Exploring issues highlighted by the quantitative analysis in more depth using qualitative interviews and focus groups (to understand the whys and hows). Interviewees can be purposively sampled using quantitative data.

To conclude, it is not yet common to find a satisfactory and reliable M&E system of social assistance benefits, and consequently M&E data are not always used as they should to strengthen policy implementation and its effectiveness. We hope that this small contribution can guide the design and implementation of robust M&E systems in other countries – proving that randomised impact evaluations are not necessarily the only way to provide ‘rigour’ – and encourage further strengthening of the monitoring system in Moldova itself.


Working Paper


26

References Adam S., Brewer M. and Shepard A. (2006): Financial work incentives in Britain: comparisons over time and between family types; IFS, WP06/20. Adams J., Bakewell O. and B. Pratt (2003): “Sharpening the development process: a practical guide to monitoring and evaluation”, INTRAC publication Akerlof, G. A. (1975). The economics of ‘‘tagging’’ as applied to the optimal income tax, welfare programs, and manpower planning. American Economic Review, 68(1), 8–19. Baldeon C. and Arribas-Baños M. D. (2008) ‘Management information systems in Social Safety Net Programs: A look at accountability and control mechanisms’, TSP Discussion paper No. 0819, World Bank. Barca V., L. Carraro and A. Sinchetru (2011): “Social Support beneficiary assessment”, project report, Oxford Policy Management. Barca V., L. Carraro and A. Sinchetru (2010): “Study into the reasons for low take-up and retention”, project report, Oxford Policy Management. Bedi T. et al (2006): “Beyond the Numbers, Understanding the Institutions for Monitoring Poverty Reduction Strategies”, World Bank publication Besley, T., & Kanbur, S. M. R. (1993). The principles of targeting. In M. Lipton and J. Van der Gaag (Ed.), Including the Poor (pp. 67–90), The World Bank, Washington, D.C. Besley, T., & Coate, S. (1995). The design of income maintenance programmes. Review of Economic Studies, 62(2), 187–221. Carraro L., A. Hurrell, V. Barca and L. Pellerano (2009): “Social Support Administrative Data: Database Creation and Analysis”, project report, Oxford Policy Management. Carraro L., A. Hurrell, V. Barca and L. Pellerano (2009): “Monitoring the Performance of the Social Support Programme: Analysis of MSAS and HBS Data” project report, Oxford Policy Management. Casley D. and K. Kumar (1987): “Project monitoring and evaluation in agriculture”, World Bank and John Hopkins University CGD (2006): “When will we ever learn: improving lives through impact evaluation”, Report of the Evaluation Gap Working Group, May 2006 Chen H.T. (2005): “Practical programme evaluation: assessing an improving planning, interpretation and effectiveness”, Coady D. et al. (2004): Targeting of transfers in developing countries: review of lessons and experience; The World Bank, Washington, D.C. Foster J., Greer J., and Thorbecke E. (1984): “A class of decomposable poverty measures”; Econometrica, Vol. 52, pp. 761-765. Garbarino S., J. Holland (2009): “Quantitative and Qualitative Methods in Impact Evaluation and Measuring Results”, GSDRC Issues Paper Gorgens M., Kusek J.Z. (2009): “Making monitoring and evaluation systems work: a capacity development toolkit”, World Bank publication Goerge R.M., Bong Joo Lee (2002): “Matching and cleaning administrative data”, chapter in The National Academy of Sciences volume “Studies of Welfare Populations: Data Collection and Research Issues”, National Academy Press, Washington D.C. Gosling l. and M. Edwards (1995): “Toolkits: a practical guide to assessment, monitoring, review and evaluation”, Development manual 5, Save the Children, UK Grogger J. and Karoly L. A. (2005): Welfare reform: effects of a decade of change, Rand Corporation. Grosh M. E. and Baker J. L. (1995): Proxy means tests for targeting social programs: simulations and speculation; LSMS Working Paper no. 118, The World Bank, Washington, D.C. Grosh M. E. et al. (2008): For protection and promotion: the design and implementation of effective safety nets; The World Bank, Washington, D.C. Hotz J. et al (1998): “Administrative data for policy-relevant research: assessment of current utility and recommendations for development”, Report of the Advisory Panel on Research Uses of Administrative Data of the Northwestern University/University of Chicago Joint Center for Poverty Research Oxford Policy Management

Working Paper


27

Jones N. et al (2009): “Improving impact evaluation production and use”, ODI Working Paper 300 Kusek J.D, R. C. Rist (2004): “Ten steps to a results-based monitoring and evaluation system : a handbook for development practitioners”, World Bank publication Kuster C. et al (2011): “Making evaluations matter: A practical guide for evaluators. Centre for Development Innovation, Wageningen University & Research centre, Wageningen, The Netherlands. Mackay K. (2007): “How to build M&E systems to support better government”, World Bank publication Morse K. and R.J. Struyk (2006): “Policy analysis for effective development: strengthening transition economies”, Lynne Rienner Publishers Inc Schiavo-Campo S. (1999): “‘Performance in the Public Sector.” Asian Journal of Political Science United National Development Programme - UNDP (2002.) “Handbook on Planning, Monitoring and Evaluating for Development Results”. New York: UNDP Evaluation Office. U.S. Social Security Administration (2008): “Uses of administrative data at the U.S. Social Security Administration”, prepared for the International Seminar on the Use of Administrative Data for Economic Statistics and the Register-Based Population and Housing Census Vaitsman J., R.W.S. Rodrigues, R. Paes-Sousa (2006) “The System for Evaluating and Monitoring Social Development Programs and Policies: the case of the Ministry of Social Development and the Fight against Hunger in Brazil”, Policy Papers 17, 2006, UNESCOWeyrauch V., G.D. Langou (2011): “Sound expectations: from impact evaluations to policy change”, paper by 3ie and the Center for the Implementation of Public Policies Promoting Equity and Growth (CIPPEC)


Working Paper


28

Endnotes 1

One of such examples is a 2006 case study that describes the “process and methodology for the creation of the evaluation and monitoring system of the Ministry of Social Development and the Fight against Hunger in Brazil” (Vaitsman et al. 2006)

2

This information was disclosed on occasion of the 2011 conference of the Network of Networks of Impact Evaluations (NONIE). See http://nonie2011.org/?q=content/post-2 for details

3

These definitions are evolved from previous version developed by Casley and Kumar in 1987 and Gosling and Edwards in 1995.

4

See also Grosh et al. (2008) for an overview of different cash transfers in developing countries, Besley and Coate (1995) for minimum income benefits, Grosh and Baker (1995) for proxy means tests and Akerlof (1975) for group targeting also referred as ‘tagging’.

5

Examples include household budget surveys, living standards measurement surveys, labour force surveys, etc.

6

In the previous (Soviet) system, nominative compensations were provided to certain categories of people to support consumption of utilities (heating, gas, electricity and other compensations for communal services). Receiving categories included the disabled, WWII veterans and their spouses, individuals and families affected by the cleaning operations that followed the Chernobyl disaster, single pensioners, families with more than four children, and people who lived in Leningrad during the blockade (WWII). Moreover, child allowances of MDL 50 per month per child were provided for poor households with children under 16.

7

The household level GMI is calculated by taking into consideration all household members and applying carefully calculated adult equivalence scales (so that, for example, a higher income is guaranteed for disabled people and a lower income for children, etc). For details see Carraro, 2008.

8

Note that this test does not automatically exclude or include people based on any of these characteristics; it simply applies a formula that considers them together to determine the probability of each household being poor.

9

For households whose members include only senior citizens, the benefit is awarded for 24 months since income is presumed to be more stable than for other households.

10

For most benefits, notably Nominative Compensations and Child Allowances, administration is provided by the National Social Insurance House through their network of Territorial Social Insurance Houses. These are present in each raion and in every sector of Chisinau. People apply and present relevant documentation directly at these offices. 11

All documents required can be obtained at the local town-hall, with the exception of the registration to the employment agency in case of unemployment for which people need to travel to the district centre.

12

Every month a number of households all over the country are interviewed.

13

This was mostly due to Moldova’s large informal economy (primarily agriculture and foreign remittances)

14

However, it was not possible to estimate the average duration in which households do need support, but there was evidence suggesting that it was necessary to check households’ circumstances at least every 6 months. 15

It is important to generate at least some ‘essential’ tables in an automated fashion, since these can be created quickly and offer a constant comparison that specifically-trained Ministry staff can gradually become familiar with, making them more likely to be interpreted. However, it should also be important to generate some capacity to conduct ad hoc analysis, though the complexity of working with large databases should not be underestimated.

16

Although carrying out such exercises is less expensive than conducting a quantitative survey, they still need to be carefully designed and adequate expertise is required both in understanding the policy and administrative system and in conducting interviews. 17

This includes a whole range of socio-economic information, from household size and structure to disability status, marital status, education levels, sex, age, employment type, ownership of assets, etc. 18

For example, this includes households who have declared members abroad and no remittances, salaries below the minimum income and other irregularities. 19

This includes comparing characteristics of beneficiary households and eligible households, to assess what type of people are still being “left out” by the programme (given that applications are on-demand). 20

Note that, though the benefit is set up to continuously support households in extreme poverty, households are required to re-apply to the programme every six months in order to keep on receiving the benefit. 21

This is a relatively high amount: it should be remembered that the previous system of nominative compensations distributed much lower entitlements. Oxford Policy Management

Working Paper


29

22

Such poverty lines were initially computed with 2006 HBS data and determined at a level that would have resulted in eradicating poverty using a budget of MDL 292 million in 2006 (this was the budget available for reform considering Nominative Compensations and some child allowances). 23

These indexes are also known as P0, P1 and P2 indexes (see Foster J, Greer J., and Thorbecke E. (1984): “A class of decomposable poverty measures”; Econometrica, Vol. 52, pp. 761-765).

24

However, it should be noted that as a consequence of a still relatively low coverage, the head-count reduction is limited.


Working Paper

Monitoring Implementation and Evaluating Performance - Working Paper

Monitoring Implementation and Evaluating Performance - Working Paper

Suggest Documents

Monitoring Implementation and Evaluating Performance - Working Paper

Monitoring and evaluating civil service performance - GSDRC

Monitoring and evaluating civil service performance - GSDRC

An Implementation and Performance Monitoring of ...

working paper working paper - CDKN

working paper working paper - CDKN

analyzing firm performance heterogeneity - Working Paper Series

NBER WORKING PAPER SERIES PERFORMANCE ... - SSRN papers

Working paper on the Conception and implementation of rich ... - Tecfa

Monitoring and Evaluating the Performance of - University of Glasgow

Monitoring and the Portability of Soft Information Working Paper

Monitoring and the Portability of Soft Information Working Paper

Monitoring and the Portability of Soft Information Working Paper

Monitoring 'monitoring' and evaluating 'evaluation': an ...

Evaluating Barriers to Effective Implementation of Project Monitoring

WORKING PAPER Psychological change ... - Working Paper Series

DEM Working Paper Series The Clothing Export Performance and ...

MSE working paper: Evaluating Asian FTAs: What do Gravity Equation ...

Working Paper No. 13 EVALUATING THE IMPACT OF ALTERNATIVE

MSE working paper: Evaluating Asian FTAs: What do Gravity Equation ...

WORKING PAPER SERIES Working Paper No. 7 Civil Society and ...

Working Paper

Working Paper

Working Paper