Thus, in urban areas, 2% in Romanian children compared with 27.3% in ...... The purpose of this evaluation, in accordance with the ToR (see Annex 7.2), is to collect evidence ..... Distribution of the sample by community types according to social services availability (%) ...... In addition, occasionally they receive food staple.
Helping the ‘invisible’ children (HIC) Second Evaluation Report
Project coordinator: Voichita Pop Author: Manuela Sofia Stănculescu (coord.) Simona Anton Cătălina Iamandi-Cioinaru Georgiana Neculau Bogdan Corad Andreea Trocea
October 2013
List of acronyms The acronyms of the Romanian institutions are in the Romanian language
CBS – Community-Based Services CCS – Community Consultative Structure CERME – Romanian Centre for Economic Modeling CJRAE – Country Centre for Educational Resources and Assistance ICCV – Research Institute for the Study of the Quality of Life (Romanian Academy) DGASPC – General Directorate for Social Assistance and Child Protection DAC – Development Assistance Committee DSP – Direction of Public Health HIC – Helping the ‘invisible’ Children (UNICEF project) ISJ – County School Inspectorate MDRAP – Ministry of Regional Development and Public Administration MMFPSPV – Ministry of Labour, Family, Social Protection and Elderly MS – Ministry of Health NGO – Non-governmental organization NUTS – Nomenclature of territorial units for statistics SAI – School Attendance Initiative (UNICEF project) SPAS – Public Social Assistance Service (within mayoralty) UNICEF – The United Nations Children's Fund US$ – United States dollar
The findings and interpretation expressed in this paper are those of the authors, and do not necessarily represent the views of UNICEF.
2
Contents 1 2
Executive summary ................................................................................................................................... 5 Description of the HIC Project .............................................................................................................15 2.1 Context .............................................................................................................................................15 2.2 Objectives and program theory.....................................................................................................18 2.3 Geographical coverage ...................................................................................................................20 2.4 Stakeholders .....................................................................................................................................21 2.5 The minimum package of community services ..........................................................................24 2.6 Theory of change ............................................................................................................................25 2.7 Flow of activities .............................................................................................................................28 2.8 Desired outcomes............................................................................................................................29 3 Evaluation Profile ....................................................................................................................................30 3.1 Evaluation overview .......................................................................................................................30 3.2 Research questions ..........................................................................................................................31 3.3 Scope of evaluation and counterfactual .......................................................................................32 3.4 Data of evaluation ...........................................................................................................................33 3.4.1 Consolidated database of ‘invisible’ children ..........................................................................37 3.4.2 Survey on a random sample of ‘invisible’ children and their families .................................38 3.4.3 Focus groups with social workers employed in HIC project ................................................39 3.4.4 Interviews at county level ...........................................................................................................39 3.4.5 Opinion survey on community representatives ......................................................................40 3.5 Method of data analysis ..................................................................................................................41 4 Evaluation Findings ................................................................................................................................43 4.1 Effectiveness ....................................................................................................................................43 4.1.1 Identification and 're-disappearance' of the 'invisible' children within HIC .......................43 4.1.2 Diagnostic of vulnerabilities ......................................................................................................47 4.1.3 Planning for intervention ...........................................................................................................66 4.1.4 Providing a response to the vulnerabilities of the worst-off .................................................67 4.1.5 Community support and participation .....................................................................................81 4.1.6 Main institutional bottlenecks and barriers for services supply ............................................83 4.1.7 Added value of micro-grants .....................................................................................................86 4.1.8 Monitoring the 'invisible' children ............................................................................................88 4.2 Relevance ..........................................................................................................................................91 4.2.1 Relevance from beneficiaries’ point of view............................................................................91 4.2.2 Relevance from community representatives’ point of view ..................................................92 4.2.3 Relevance of the minimum package of services .....................................................................93 4.3 Efficiency..........................................................................................................................................95 4.3.1 Project cost-efficiency .................................................................................................................95 4.3.2 Time allocation by activities .......................................................................................................96 4.4 Sustainability ....................................................................................................................................97 4.4.1 Sustainability of the project activities .......................................................................................97 4.4.2 Community Consultative Structures ...................................................................................... 100 4.4.3 Resource centre at DGASPC providing methodological support for communities ...... 101 3
5 6 7
4
Conclusion, lessons learned and recommendations ........................................................................ 104 References.............................................................................................................................................. 117 Annexes.................................................................................................................................................. 118 7.1 List of micro-grants ..................................................................................................................... 127 7.2 Terms of Reference...................................................................................................................... 129 7.3 Evaluation team ............................................................................................................................ 139 7.4 List of people interviewed........................................................................................................... 139 7.5 Site visits ........................................................................................................................................ 140 7.6 Evaluation matrix ......................................................................................................................... 142 7.7 Descriptive results by research component.............................................................................. 146 7.7.1 Focus groups with social workers .......................................................................................... 146 7.7.2 Interviews with county stakeholders ..................................................................................... 159 7.8 Research instruments ................................................................................................................... 183 7.8.1 Questionnaire for ‘invisible’ children and their families ..................................................... 183 7.8.2 Focus group guide .................................................................................................................... 193 7.8.3 Interview guide for DGASPC ................................................................................................ 198 7.8.4 Interview for Prefecture .......................................................................................................... 201 7.8.5 Questionnaire for social workers ........................................................................................... 202 7.8.6 Questionnaire for community representatives ..................................................................... 205
1 Executive summary Background The project Helping the ‘invisible’ Children (HIC), prepared in April-June 2011 and implemented during June 2011-September 2014, is part of the UNICEF's Community Based Services (CBS) Programme in Romania, which focuses on the development of the preventive approach in the social protection system, particularly in rural areas. The HIC project has targeted the 'invisible' children defined as the children who face one or more types of vulnerabilities1 and have been reached by social workers through fieldwork activities. Within HIC, the ‘invisible’ children refer also to children acknowledged as being in a vulnerable situation, but with ‘invisible’ vulnerabilities (also identified through fieldwork activities).2 The purpose of the project Helping the ‘invisible’ children is to increase the impact of social protection policies for poor and socially excluded (‘invisible’) children and families. Taking into account the underdevelopment of the social assistance services at community level, the HIC programme theory has considered that children’s welfare in Romania will improve only if and when the children, especially the worst-off (‘invisible’ ) children, will have enhanced access to social services (education, health, and social assistance services). For this purpose, in 2011, social workers were employed in 96 communes from eight counties (Bacău, Botoşani, Buzău, Iaşi, Neamţ, Suceava, Vaslui and Vrancea). After a short training, these social workers identified the ‘invisible’ children within community and mobilized the Community Consultative Structures (CCS),3 under the supervision of the County General Directorate of Social Assistance and Child Protection (DGASPC). In 2012, the project coverage was limited to 64 communes. A basic package of community preventive social assistance services was piloted: the social workers have carried out outreach activities including needs-assessment, monitoring, informing and counselling, and have provided appropriate social assistance services to the worst-off children and families. This model is supposed to be implementable at the national level (with a total of 2,858 communes in the country) and to impact progressively the main gaps of the social protection system at four layers: i) legislative provisions, ii) institutional building, iii) resources allocation, and iv) social control mechanisms, including monitoring and evaluation.
The HIC project has used the following list of vulnerabilities: (1) Children in households with many children, in poverty and precarious housing conditions; (2) Children left behind by migrant parents, living in poverty or other difficult situations; (3) Children at risk of neglect or abuse; (4) Children with suspicion of severe diseases; (5) Relinquished or at risk of child relinquishment; (6) Children out-of-school and children at risk of school dropout; (7) Teenage mothers who left school and/or are at risk of relinquishing the new-born child; (8) Children without ID papers or documents; (9) Other cases of vulnerable children. 2 That is children who have been already known at local level as being in a vulnerable situation, but about whom the field visit offered new insights (such as abuse, neglect, etc.), irrespective if his/her family have received some social benefits or services before the start of the project (e.g. social aid, heating allowance etc.). 3 They include local decision-makers such as the mayor/vicemayor, secretary of the mayoralty, social worker, doctor, policeman, school representative, priest, etc. Although they are set on paper in the Romanian legislation, the CCS were not really functioning in most localities at the start of the project. 1
5
Objective and Methodology of the Evaluation This is the Second Evaluation Report of the HIC Project in Romania. The report presents the results of an extended field research carried out in December 2012- January 2013. This evaluation focuses on the relevance, effectiveness, efficiency and sustainability of the HIC project. The OECD-DAC evaluation criteria as the humanitarian ones are not relevant for this task. Also, some elements of impact in terms of outputs are inventoried. The evaluation has covered all key stakeholders, all counties (8), all communities (95)4 and all activities carried out in the project in 2012 (in 64 rural communities that remained in the project). The evaluation has included a counterfactual comprising the ‘invisible’ children identified in the 32 communities which were not anymore part of the project in 2012. These children were supposed to benefit of the services existing in these communes (e.g. SPAS regular services, Roma mediator, community health nurses, etc.) but not of the services provided within HIC project. Also, in these 32 communities, the outreaching activities of identification of the most vulnerable children and their families were ceased in 2012. In contrast, in the 64 communities remained in the project, the identification of the most vulnerable has been kept as a component of on the ground activities and the ‘invisible’ children have been supposed to benefit of the services provided under the minimum package approach. Thus, the evaluation compares:
‘invisible’ children in both types of communes and how their situation evolved in 2012; the services in both types of communes, including support provided from county level to communes in the project versus virtually no support in communes not included in the project; the results for children in both types of communes.
The evaluation used a mixture of primary and secondary sources of information so that to ensure high quality and credibility. However, most data were collected with the special purpose of this evaluation. Primary data were collected both at the micro (individual, household) and meso (community) levels, through quantitative and qualitative research techniques. The field research was carried out by CERME, in December 2012 - January 2013. The following data sources have been used: The 2011 evaluation report (Stănculescu and Marin, 2012), including quality assessment comments and recommendations; All reports carried out by supervisors and by social workers; All UNICEF reports (including field monitoring); Databases for case management filled-in at community level; Legislation in place: law on social assistance (2011) and revised law on child protection (2012); Consolidated database of ‘invisible’ children (5,758 cases); Survey on a random sample5 of ‘invisible’ children and their families; Focus groups with From the 96 communes included in the project, Trifeşti (Iaşi) has never offered any data. Out of the consolidated database, a random sample of 1,006 ‘invisible’ children was extracted, with a margin error of ± 2.8% at 95% confidence level. The total response rate was 99%. Thus, the sample consists of 708 households with valid questionnaires, including 923 children registered as ‘invisible’, 1,724 siblings below 18 years old (not registered as ‘invisible’ children) and 1,441 adults (18 years or over), located in 60 rural communities. 4 5
6
social workers employed in HIC project;6 Interviews at county level;7 Opinion survey on community representatives.8 The evaluation methodology has several advantages, including the use of a mix of quantitative and qualitative methods, the fact that the quantitative data refer to representative samples, the utilization of a participatory approach (all key stakeholders took part in the evaluation activities), the collection of data for specific-evaluation purposes. Thus, the methodology is in line with previous researches that highlight the importance of the engagement, interaction, and communication between evaluation clients and evaluators for a meaningful use of evaluation (Johnson et al., 2009). Nevertheless, there are also limitations. The main one is given by the difficulty to separate the project outputs/outcomes of the results of other interventions in the area of preventive social services, such as the School Attendance Initiative9 or the national program for the development of preventive medical services in rural area.10 Consequently, the situation as measured in January 2013 is a combined result of all such interventions, which were implemented only in some communities (and not in all), have not been coordinated or consistently applied, some were only sporadic, and most are not documented at all. However, the counterfactual reflects the situation and developments in the absence of the Helping the ‘invisible’ Children project. Key Findings and Conclusions The project has produced overwhelming proofs that the issue of 'invisible' children is highly relevant for the rural communities from Romania and it represents a serious problem that needs an urgent and determined policy response. In the same time, the project has demonstrated that: (i) the development of preventive community services is possible in spite of the limited human resources at local level and of the insufficient local budgets; (ii) outreach activities are possible and essential for ensuring the right to social security for children (and other vulnerable groups). Moreover, it presents clear evidences that the preventive community services are more effective and much cheaper in real life and not only in theory. In one year and a half (2011-2012) the project identified 5,758 'invisible' children who face a complex cumulus of vulnerabilities. Based on the minimum package of community services approach, over 3,400 children and their families, from 64 rural communities, received a variety of services from diagnostic to information, counselling, accompaniment and support, referral, as well as monitoring and evaluation. Thus, access to health, education, social protection, and opportunity to develop into the natural family has been enhanced for many children at risk. There were realized 4 focus group discussions with 37 social workers, from all 8 counties. There were conducted 24 interviews with DGASPC directors, supervisors and Prefecture representatives, from all 8 counties. 8 The sample included 60 HIC social workers and 235 CCS members, from 60 communes. 9 Since mid-2010 UNICEF has financed the School Attendance Initiative (SAI) with the aim of getting children of school age back to school and supporting them to complete the compulsory years of schooling. SAI has been implemented in over 230 rural communities among which 14 communities covered by HIC. 10 Since 2006, the Ministry of Health has financed the development of a network of community health nurses and Roma health mediators with the aim to improve the preventive medical services in rural area. This national program has covered 43 communes included in HIC (in 2012-2013). 6 7
7
In addition, people's attitudes toward children and their rights has changed for the better, the preventive community services have improved, the number of cases of violence and abuse within communities has declined. The community activation and participation (particularly through the CCS) has improved compared with 2011. So, current results indicate that project objectives may be achieved in the next 2-3 years. Hence, the project is on the right track. Yet many areas need adjustments. The project is very relevant and highly efficient. Problematic aspects relate to effectiveness and sustainability. For improving effectiveness and sustainability, the evaluation points out two priority areas for action, namely (1) diagnostic of vulnerabilities and (2) monitoring and evaluation activities. The effectiveness of the project needs improvements. At the national level, the identification of vulnerable cases (children or not) is severely underused as there is no official methodology (standards, methods, activities, estimated costs) for this service. HIC has addressed this gap. Identification was the major activity of the phase I of the project (in 2011) and it has remained a constant activity since then. Thus, the HIC consolidated database comprises 5,758 'invisible' children from 95 communes. Nonetheless, 33.5% (1,929 cases) of all cases were identified in 2011 but have not been recorded in the databases for case management implemented in 2012, hence 're-disappeared' from the HIC focus. This phenomenon was the result of a combination of factors related to: (1) adjustment of the project coverage, (2) discontinuity of activities and changes in the project instruments, without a proper training of the social workers, (3) introduction of new specifications regarding the target groups, (4) results of the activities carried out in 2011, (5) insufficient capacity of response at the local level and (6) the community characteristics. The quality of the consolidated and cleaned database still needs serious improvements. Thus, 7% of all registered children could not be identified in the field and no local stakeholder has had updated information about them. Furthermore, out the 'invisible' children found in the field, another 4% are not living with their families, but information about them and their situation are available. The distribution of population of 'invisible' children by types of vulnerability did not change between November 2011 and December 2012. The accuracy of the diagnostic is of utter importance even more so given that about 40% of 'invisible' children have to face multiple vulnerabilities.11 The most extended vulnerability refers to neglect or abuse, which affects more than half of the 'invisible' children. This vulnerability is the least accurately12 assessed by the HIC social workers, being strongly under-appreciated. The second vulnerability refers to large households with many children that live in poverty, which is also poorly assessed by social workers, but in the opposite sense, being over-appreciated. The third vulnerability relates to education. Children out-of-school or at risk of school dropout impinge on almost a quarter of the 'invisible' children. This vulnerability is According to the data from the consolidated database at December 2012. The accuracy rate computed using the formula (confirmed presence of the vulnerability + confirmed absence of the vulnerability)*100/total sample (N=923). 'Confirmed' means that data collected within household survey correspond to the social worker' assessment from the project consolidated database. 11 12
8
more accurately assessed by HIC social workers, but only three out of four registered cases in the databases for case management are confirmed in the field. At the national level, in most SPAS, the step of diagnostic of vulnerabilities is limited to a summary check of the documents included in applications for various cash benefits. HIC has addressed this gap but further steps are needed. Specifically due to the lack of a methodology for assessing and recording vulnerabilities and insufficient training of the social workers, the accuracy of project database is rather weak. Only three out of four registered cases in the databases for case management are confirmed in the field. The issue of a complete and systematic diagnostic of all vulnerabilities for each child is of utmost importance even if in some cases solutions at the community level seem to be too distant or not at all. – 30% of the 'invisible' children belong to households in which the adults (18 years or more) completed only primary education. An additional 47% of 'invisible' children live in households with adults who achieved only gymnasium. – In terms of cash incomes, all 'invisible' children live in poor family. The cash incomes of the 'invisible' children's families are extremely low: 80% of them live in absolute poverty, with less than USD$ 1 per person per day, while the other 20% are in households with monthly cash income per person below the national threshold of relative poverty. – Children who live in large households, in poverty and precarious housing represent 43% of all 'invisible' children, according to survey data. They are boys and girls of all age groups, from all types of households, but with a considerably over-representation of Roma children (51% compared with 42% of Romanians). – Almost three in every ten 'invisible' children do not grow up in a nuclear family, fact with long term effects on their development, even more so considering the chronic poverty they live in.13 – 50% of 'invisible' children face neglect, abuse or violence.14 The survey indicates that children aged 6 to 14 years old (who go to school) are more exposed to this vulnerability compared with those under 6 years as well as with teenagers over 14 years old. Also, Roma children have a considerably higher probability to be the victim of domestic violence compared with the Romanian children. However, most children in this situation are Romanians and not Roma. The adults from households in which one or more children are at risk of neglect or abuse are married couples (legally or not), predominantly aged 30-49 years, with 8 classes at most, in which women are housewives, while men and young work by the day especially in agriculture. – The risk of child abuse and neglect is strongly associated with alcoholism. Of all 'invisible' children: (i) 19% belong to single parent families, but overall 25% live together with only one parent (some being part of large multigenerational households and some single parent families); (ii) 4% are have no parent at home but live together with relatives (most often at a grandmother) or are in foster care. 14 (A) risk of violence and abuse - the most commonly used methods of disciplining children as declared by parents include shouting to, threatening, humiliating, demeaning or beating the child; (B) risk of neglect - the child (0-14 years) is left home alone (together with sisters and/or brothers) for at least several hours, without being supervised by an adult. 13
9
– Only 54% of the cases were correctly assessed by HIC social workers. This reflects the deep cultural tolerance of violence against children that dominate the entire country and not only HIC communities. – Almost 24% of the 'invisible' children are out of school or at risk of school drop-out. Only one in every three 'invisible' children aged 3 to 5 years go to kindergarten and even fewer attend it daily. One in every ten 'invisible' children of 6 to 14 years old is not enrolled in school and a significant part of those enrolled in school do not attend school daily, not to mention that only about a half of them have medium-good results. After 15 years, the school enrolment of 'invisible' children diminishes to 53%, with 42% who attend school daily and only 26% with medium-good school performances. – Children relinquished or at risk of relinquishment represent 7.9% of all 'invisible' children.15 The risk of child relinquishment as assessed by the HIC social workers is much more frequent among the Romanian children compared with Roma, particularly at the small ages (0-2 years old). It is also significantly higher for children from poor households with many offspring. The children living with relatives and with no parent at home have a probability four times larger than the other 'invisible' children. In addition, there are significant differences between counties. In Bacău virtually no 'invisible' child is at risk of abandonment, while in Vrancea the proportion is almost double the sample average. In 2012, HIC has piloted the minimum package of community services, thus providing a variety of services. HIC has operated along with other interventions and community services (in collaboration or not, coordinated or not) in many communities. Furthermore, in some, the HIC social worker was in the same time SAI facilitator or community health nurse or school mediator or SPAS employee. Therefore, HIC impact cannot be clearly delimited of the results of the accompanying interventions. The coverage of services was important, much higher than in the control communes in which HIC was no longer implemented in 2012. The targeting varies from a service to another, but in this respect the total performance is rather low, especially because the link between the diagnosed vulnerabilities and delivered services is weak. – In 2012, HIC together with the other interventions provided services to 89% of the identified 'invisible' children in HIC communities. Furthermore, 97% of 'invisible' children in severe poverty who suffer of hunger or cold a few times a month benefited in 2012 of at least one service. HIC reached 91% of 'invisible' children from households with 3 or more children. Children from single parent families were covered in a proportion of 94%. At least one service was provided to 90% of children at risk of neglect or abuse, 88% of children out-of-school or at risk of school dropout, 86% of children in need for medical services. – In HIC communities, 43% of girls and boys aged 3-17 years old benefited in 2012 of services of facilitating the access to education. The proportion of beneficiaries decreases from 54% at 6-10 15
According to the data from the consolidated database at December 2012.
10
years to 49% of children 11-14 years and only 35% for teenagers of 15-17 years. The interest was also reduced for early education, only 20% of 'invisible' children aged 3-5 years receiving assistance for going to kindergarten. Roma children had significantly higher chances to have benefited of services of facilitating the access to education compared to Romanian children. By contrast, in the control group, virutally no Roma child benefited of such services. – In the communities in which the UNICEF financed School Attendance Initiative (SAI) was implemented in parallel with HIC, the proportion of beneficiaries of educational services was significantly higher (50%). The existence of a Roma health mediator produced the same positive effect. By contrast, the presence of a school mediator within community has made no difference with respect to the educational services delivered to 'invisible' children. – 41% of 'invisible' children from all 96 initial communes benefited of some services of facilitating the access to health care, during the year 2012. Almost a third of 'invisible' children were registered with a family physician, almost a quarter were assisted in making an appointment for a medical specialist, 5% were granted free transportation to a specialist in a nearby city and 2% were helped to get a disability certificate. In HIC communities, girls and boys, Roma and Romanians had the same probability to benefit of such services. By contrast, in the control group, Romanians had more than two times more chances than Roma children to benefit of medical services in 2012. – Nonetheless, 7.5% of all 'invisible' children still need services of facilitating the access to health care: 2.9% are not registered with a family physician, 1.4% need to obtain a disability certificate, and 3.2% need to see a medical specialist. Although 38% of them need medicines for continuous treatment, only 9% afford it. Within HIC, 75% of cases of 'invisible' children either ameliorated or were solved in 2012. Moreover, in HIC communities, 82% of 'invisible' children belong to households with a 'good' or 'very good' relation16 with the social worker and that are 'satisfied' or 'very satisfied'17 with the social worker's activity. From the HIC social workers' point of view, the main benefit of the project on children's welfare in their community has been better information first and most of all. The project is considered highly relevant by all stakeholders. Nearly all community representatives know the HIC objectives and activities within their community. The project visibility has increased between 2011 and 2012. The minimum package of services is considered highly relevant in addressing vulnerable children’s needs and it is seen as a real support in preventing separation of the child from family at the community level. The project is very efficient. Over 3,400 children benefited of one or more services. Moreover, 58 out of 70 children vulnerable to relinquishment or at risk of child relinquishment were prevented from separation from family. This was ensured with a cost per child (and his or her family) per year 16 17
According to the opinion survey on social workers. According to the survey on the 'invisible' children and their families.
11
of lei 250. By comparison, the standard cost in the child protection system, per child per year, varies from lei 14 to 35 thousand, which is 56-to-140 times higher.18 Thus, preventive community services are not only more effective in protecting children, but are also much cheaper compared with the specialized protection services. Sustainability is the most problematic dimension, although 70% of community representatives declared that HIC activities would be assumed and continued at community level after the project stop. 57% of community representatives believe that the HIC social worker will be hired within the mayoralty with the same attributions as in the project. In HIC communities with micro-grants,19 the project appears to be more sustainable than in the other communes or, at least, the community representatives are more optimistic in this respect. Nonetheless, data from the control group show that this optimism should be cautiously considered. Recommendations The general recommendation is to continue the intervention and to enhance the advocacy efforts so that the model of community preventive service piloted within HIC to be undertaken and scaled up at national and county levels. Availability of human resources at the community and county levels is critical both for project sustainability and effectiveness. The project has proven that the implementation of minimum package is possible even based on a network of social workers with less experience in social assistance. Many HIC social workers are not specialists but are hearty and tenacious people willing to work in this field and with good communication abilities. They performed the activities as they could and in many situations they found creative solutions to children's problems and difficulties. Nonetheless, in order to improve the project performances they need better training, increased methodological support from the county specialists, but also better tools (database, guides, methodologies and procedures) for improving their activities. Two factors may considerably boost sustainability, namely: (1) the HIC social workers to be employed within mayoralty with the same attributions, which ensures the transfer of knowledge and practices to the local SPAS and (2) the existence of preventive services other than HIC within community (e.g. community health nurse, sanitary mediator, school mediator, roma mediator). The available data indicates that regardless the local decision makers’ openness to integrate the HIC social worker within mayoralty the current legislation and the budgetary constraints represent factors that seriously impede the project staff sustainability after the UNICEF funding will cease. So, in the
KPMG, 2010, Standard cost in the child protection system – comparative analysis. Standard cost per child per year varies as follows: lei 13,608 for maternal assistances with 3 children in foster care (p. 12); lei 16,639 for maternal assistances with 2 children in foster care (p. 10); lei 23,947 for maternal assistances with one child in foster care (p. 8); lei 34,589 for residential services (p. 29). 19 The activities conducted within the projects financed through the micro-grants included: support groups for parents, information activities for parents, educational activities and support for the children, thematic trips, and leisure activities. Within these projects children had the opportunity to play with toys and games that otherwise they would not afford, go on trips for the first time in their life, and spend quality time with parents. Also, for the parents and especially for mothers, the activities were a good occasion to go out, to meet other parents, share their problems and their experiences and spend time (even playing) with their children. 18
12
next phase of the project, the advocacy efforts for changing the regulations that block posts in the public sector needs to be enhanced. Better collaboration between institutions is another key factor. Improvements are needed regarding the vertical and horizontal collaboration between institutions, professionals and stakeholders. According to the community representatives, the Community Concultative Structures (CCS) became more active in 2012 and the frequency of CCS meetings increased compared with 2011. However, from the county stakeholders’ perspective, more needs to be done in order to increase the participation of CCSs members in the project activities. In this sense, more clear regulations need to be set on how the CCSs function and on their responsibilities. The micro-grants and the Centres of Resources20 from DGASPC have had a very positive impact. We recommend that in the next phase of the project to continue providing micro-grants and to continue financing the Centres of Resources from DGASPC as coordinating units that supply methodological support for communities. Besides the coordination of social workers from the county level (ensured by the DGASPC supervisors), constant monitoring at the central level is needed in order to reduce the significant gap in performances between counties. Sustainability is affected by the tolerant attitudes towards alcohol abuse, violence, school dropout or early motherhood. These attitudes hamper the process of identification of ‘invisible’ children and, more generally, the way in which vulnerable children are treated. It is needed a clear strategy to address local knowledge, attitudes and practices at community level, both in terms of public institutions and population as a whole. For improving project effectiveness, the next phase of the project should start with a consolidated database of 'invisible' children acknowledged by all stakeholders, this implying the cleaning of the already existent consolidated database so that to ensure that the phenomenon of 're-disappearance' of 'invisible' children from HIC attention is stopped. A procedure regarding the registration/deletion in the database should be developed in order to ensure the fact that every case is regularly monitored at least for 1-2 years. Also, to better understand the needs of a child it’s indicated to see him in his natural environment and not in isolation. In this regard a new database format is necessary, which to require systematic information about all household members. In this way the intervention plan would consider also the relevant others who decide and are responsible for the child. To avoid discrepancies on the identification of ‘invisible children’, a methodology for a complete diagnostic of vulnerabilities based on a grid of indicators needs to be applied unitary for all children and across all communities. Given the complexity of vulnerabilities faced by the 'invisible' children Both supervisors and DGASPC Directors consider that the Centre of Resources is necessary not only within the project, but in the collaboration with all local SPAS and stakeholders, from all rural communities. However, for extending the Centre of Resources would be needed (1) new regulations, (2) clear definition of the relation between DGASPC and local SPAS, as well as working procedures and (3) a distribution of the contributions that DGASPC and rural mayoralties should provide for running this county centres. Under the current budgetary allocation for the DGAPSC, the extension of the Centres of Resources is not sustainable and even the maintenance of the current Centre of Resources is endangered after the project finish. 20
13
the approach of the project need to be integrated and not to treat each vulnerability separately. A methodology for linking the diagnostic of vulnerabilities with the individual plan of intervention is definitely necessary. In order to help and support the supervisors and social workers, the new methodologies, procedures, formats and indicators should be integrated in an intelligent online application. This kind of application would also allow real time monitoring of the activity in the field as well as data aggregation at various levels (community, county, project level) at any moment, which would support evidence-based adjustments of the project in a timely manner. Besides tools, social workers and supervisors need proper training and constant guidance and support for using the new instruments in an effective manner. Regarding the basic package of community services, the outreach activities that ensure the active presence of the social worker among the community members need to remain an important component of the minimum package of services. The services of accompanying and guidance included in the minimum package need to be extended and diversified. The addition of two components would significantly add value, namely: (i) a material component which to address the multiple deprivation aspects; (ii) a component dedicated to parents as without literacy programs, rehab and employment for parents of the 'invisible' children, the project results have very poor chances to be effective and sustainable. The local stakeholders consider that HIC effectiveness may be improved by developing other preventive services at community level, such as day-care centres. Beyond the project, at the level of national or regional policies, the household survey on 'invisible' children brings into light the issue of lack of data at the national level about this segment of population. The 'invisible' children and their families are predominantly atypical cases, statistical outliers as compared with the official data, on almost all dimensions from demography to economy, from number of children and composition of households to employment, level of education, incomes or consumption. The population of 'invisible' children and their families cannot be assessed using the standard indicators and methodologies. Therefore, for the social inclusion and poverty reduction policies, the standard official data21 should be completed with dedicated studies based on representative samples of households such as those of the 'invisible' children.
21
E.g. Household Budget Survey implemented by the National Institute for Statistics.
14
2 Description of the HIC Project The project Helping the ‘invisible’ Children (HIC), prepared in April-June 2011 and implemented during June 2011-September 2014, is part of the UNICEF's Community Based Services (CBS) Programme in Romania, which focuses on the development of the preventive approach in the social protection system, particularly in rural areas. The purpose of the project Helping the ‘invisible’ children is to increase the impact of social protection policies for poor and socially excluded (‘invisible’) children and families. In 2011, social workers were employed in 96 communes from eight counties (Bacău, Botoşani, Buzău, Iaşi, Neamţ, Suceava, Vaslui and Vrancea). After a short training, these social workers identified the ‘invisible’ children within community and mobilized the Community Consultative Structures (CCS),22 under the supervision of the County General Directorate of Social Assistance and Child Protection (DGASPC). In 2012, the project was renamed into First Priority: No ‘invisible’ Child! Development of Basic Social Services at the Community Level. The project coverage was limited to 64 communes, in which a basic package of community preventive social assistance services was piloted with the purpose to reach the worst-off children and families. This is the Second Evaluation Report of the HIC Project in Romania. The report presents the results of an extended field research carried out in December 2012- January 2013 by the Romanian Centre for Economic Modeling (CERME). 2.1 Context In Romania, there are nearly 3.8 million children under 18 (3.2 million aged 0-14 years, NIS data as of 1 January 2012), living in 3.2 million households.23 After 2009, in the context of the global economic crisis, Romania has seen a rise in poverty rates and vulnerability. Children and youth have constantly had the highest poverty risk of all age groups, irrespective of the poverty estimation method, and in their case the depth of poverty has also been bigger. Post-2009 trends indicate that the economic recession has strongly affected children and young people and the negative tendency is likely to continue; almost 320,000 children (0-17 years) were living in absolute poverty in 2010. The crisis hit rural and Roma children the hardest.24 Recently, three independent research teams (European Commission and UNICEF) 25 showed that: (a) As regards children, income poverty (or monetary poverty) is higher in Romania than in any other European country (33% versus 20% the EU-27 average); the depth of income poverty is They include local decision-makers such as the mayor/vicemayor, secretary of the mayoralty, social worker, doctor, policeman, school representative, priest, etc. Although they are set on paper in the Romanian legislation, the CCS were not really functioning in most localities at the start of the project. 23 Most children come from complete nuclear families (58% of households with children) or multigenerational households including grandparents and/or other relatives, with or without parents (36%). The other children live in single-parent families, accounting for nearly 6% of all households with children. (NIS data as of 1 January 2012) 24 If in urban areas the absolute poverty rate was only 3.5%, in rural area it reached 12.4%. For Roma children, the absolute poverty rate is extremely high. Thus, in urban areas, 2% in Romanian children compared with 27.3% in Roma children, and 10.6% versus 41.1% in rural communities, respectively. Preda (coord.) (2011) Situation Analysis of Children in Romania. UNICEF Report. HBS data, NIS. 25 UNICEF Innocenti Research Centre (2012) Measuring child poverty; EC (2012) Measuring material deprivation in the EU indicators for the whole population and child-specific indicators; Frazer and Marlier (2012). 22
15
greater in Romanian children than in any other European country; (b) Nearly 72-78% of Romanian children suffer from severe material deprivation, which is significantly greater than in all the other European states, for almost all goods.26 In addition, large numbers of children continue to be separated from their natural families or subjected to various forms of violence or social exclusion. Thus, the general context has been marked by widespread and severe needs among children as well as alarming phenomena27 such as ‘hidden’ single-parent families,28 ‘hidden hunger’, ‘co-morbidity,’29 low birth weight caused by mothers’ undernutrition or precarious health, children suffering from protein-calorie malnutrition, non-enrolment in school, school drop-out or household practices that are a threat to children’s healthy development. As response, Romania has greatly developed the system of social services, which nevertheless is still lagging behind many European Union Member States (in terms of employment in the sector: 4% versus 9.6% in the EU).30 The application of current legislation concerning the development of primary social services is hindered by lack of flexible forms of employment (for example, part-time), a very reduced share of social service expenditure in the GDP (Romania ranks bottom in the EU for this indicator), the headcount31 and training of human resources. The study32 on the State of Social Service Development in 2011 in Romania, conducted in all counties of the country, concludes that the social protection system is still faced with many lacks and inconsistencies, being unevenly developed especially in rural areas. Local public authorities occasionally engage in system organisation, being active only in cities and towns and solely where there is a strong non-governmental presence. The capacity of local government decision makers to employ or retain specialised staff in social assistance services is very low, which leads to serious discrepancies between rural and urban communities.33 At the community level, although the number of social services provided by NGOs has grown, certain vulnerable groups are poorly covered, such as former detainees, drug users or alcoholics, homeless people or HIV/AIDS-infected people. The development of community-based social services is hampered by lack of financial resources at local level, but also by lack or shortage of specialised staff.
Children lack (due to insufficient money): to a particularly great extent, indoor games, outdoor equipment and participation in school trips and events that cost money; to a very great extent, children’s books and family car; to a great extent, fresh fruit and vegetables as well as a meat- or fish-based meal everyday, celebrations, the possibility of inviting friends over from time to time, regular leisure activities (sports, organisations for children and young people, etc.), new clothes and two good pairs of shoes, computer and Internet, as well as a desk/adequate space for homework. 27 Stănculescu (coord.) (2012), UNICEF research based on 2011 data. 28 Households with children where one or both parents are missing, regardless of the number of the other adults present. 29 ‘[…] overnutrition in the form of obesity and non-communicable diseases in the same communities or even families where we find child undernutrition.’ (UNICEF, 2006, p.26) 30 Romania has to submit twice a year to the European Commission administrative and statistical information alongside monitoring reports on social service developments based on the European Commission Recommendation on the active inclusion of people excluded from the labour market (2008). 31 In Romania, in 2010, one social worker corresponded to 4,300 inhabitants compared with 1/300 in Sweden, 1/600 in United Kingdom or 1/1,600 in Italy. Campanini et al. (2010) in Preda (coord., 2011). 32 IRECSON (2011), Strategic Operational Program on Human Resource Development, project implemented by MMFPS. 33 Preda (coord., 2011), UNICEF project; IRECSON (2011), MMFPS and SERA Romania (2012). 26
16
In general, at community level, identification and early intervention services and referral systems for most vulnerable groups are poorly developed or completely missing. Cooperation between different sectors and services is relatively limited. There are no mechanisms of coordination between competent ministries/institutions. The prevailing social attitudes and practices that foster discrimination (particularly against the Roma) as well as tolerance toward negative behaviours (such as domestic violence or alcohol abuse) add more obstacles to identification and referral. Local prevention services continue to be preponderantly centred on social benefits (the guaranteed minimum income, winter fuel allowance, child allowance and family benefits), to the detriment of developing and implementing prevention and counselling social services, of life skill development, job-finding assistance, etc. Of all SPAS beneficiaries (children) in 2010, only 24% benefited from services while the others received only benefits, although 21% of the beneficiaries of cash benefits were children at risk of being abandoned by their parents.34 The Public Social Assistance Service (SPAS) has the task to apply social assistance policies and strategies at community level. Among SPAS, only 30% are accredited as social service providers, and 70% fall short of accreditation.35 A MMFPSPV and SERA Romania study (2012) shows that, as regards the human resources allocated to social assistance activities in rural areas, some mayoralties have included in their organisational charts a social assistance department, others just one or two persons with related attributions. These persons who perform social work at commune level neither deal exclusively with social work, nor do they have clearly defined attributions. Usually, these people concurrently hold most varied and different responsibilities: agricultural registrar, librarian, taxes, emergencies, etc. Their workload is huge and often they are strictly limited to granting social benefits with everything that involves – making and checking files, handling monthly payment records, house calls (especially in large and very large communities, with a population of over 3,000) – without carrying out prevention and counselling activities for at-risk persons. A qualitative research36 carried out by UNICEF has indicated that, in areas where professionals in social assistance were already hired, about 85% of their activities in selected communities were office-based and bureaucratic. The reduced training of social assistance staff at local level constitutes another major problem. 37 Practically, over 60% of those who should implement prevention programmes (61% of SPAS staff and 64% of persons with social work attributions) don’t have specialised studies. Only about one in every four persons with social assistance attributions or employed by SPAS has specialised higher education. There are considerable gaps between rural and urban areas: on average 1 employee (irrespective of education and whether s/he is with SPAS or not) in each commune versus an average of almost 25 employees with social assistance attributions in town-based SPAS.
FONPC (2012). MMFPS and SERA Romania (2012). 36 Magheru (2009). 37 Preda (coord., 2011), UNICEF project. 34 35
17
Since at system level the human resources are insufficient and not properly trained, at SPAS,38 social assistance activities are carried out without clear procedures or criteria of eligibility/admission of beneficiaries. There are no methodologies to evaluate or monitor the circumstances of different vulnerable groups in the territorial administrative unit and no methodologies for the early detection of at-risk cases. Social assistance is not performed based on clear plans and objectives but only when cases turn up. SPAS social workers don’t know the child protection system at county level in terms of existing services and can’t make an informed referral; in general, they refer children and families to services based only on case analysis without correlating the child/family’s needs assessment with existing services in the county. 2.2 Objectives and program theory The ultimate objective of HIC has been to increase the impact of social protection policies for poor and socially excluded (‘invisible’) children aged 0-17 years and their families. The main specific objectives of HIC have referred to: i) Strengthening the national strategy for prevention services in terms of effectiveness and efficiency; ii) Increasing the national capacity to deliver basic services with emphasis on identification of children and families at risk and strengthening the monitoring and assessment mechanisms and iii) Extending, in the end, the access to essential services to 30,000 poor, excluded, vulnerable children. Taking into account the underdevelopment of the social assistance services at community level, the HIC programme theory has considered that children’s welfare in Romania will improve only if and when the children, especially the worst-off (‘invisible’ ) children, will have enhanced access to social services (education, health, and social assistance services). For this purpose, in rural areas (particularly in the poorest communities), social workers need to be hired and trained to carry out mainly outreach activities including needs-assessment, monitoring, informing and counselling, and to provide appropriate social assistance services to the worst-off children and their families. To this end, HIC has aimed to pilot in 64 communes from the NE and SE regions (the poorest in the country and Europe) a model of community preventive services with the purpose to reach the ‘invisible’ children and families. UNICEF contribution to the modelling project consisted of: i) the total budget of HIC for the year 2012, for piloting the minimum package of services, was lei 1,152 thousand (almost US$ 345 thousand);39 and ii) technical assistance for the design of the model, minimum package of services and instruments, capacity building activities at local and county level, and monitoring and evaluation activities. This model is supposed to be implementable at the national level (with a total of 2,858 communes in the country) and to impact progressively the main gaps of the social protection system at four layers: MMFPS and SERA Romania (2012); FONPC (2012). This budget contains: (1) monthly wages of about US$ 255 for 8 county supervisors; (2) a monthly budget of US$ 90 per county for transportation of supervisors to visit the communities included in the project; (3) about US$ 1,200 per county per year for covering the maintenance costs of a resource centre (within DGASPC) for communities included in the project; (4) monthly wages of about US$ 300 for 64 social workers employed with the project; (5) 24 micro-grants of US$ 3,000 per year per community for covering the costs of a community centre for children and parents; (6) a budget of US$ 600 per year for experience exchange. 38 39
18
i) legislative provisions, ii) institutional building, iii) resources allocation, and iv) social control mechanisms, including monitoring and evaluation (Magheru, 2010). From the design phase, HIC was aligned to the Government’s National Strategy for the Protection and Promotion of Children’s Rights (2008-2013), which highlights the need for development of prevention mechanisms instead of interventions in specialized services,40 as promoting actions that keep children in the family rather than in the protection system is more effective, more in line with the child rights approach and it is cheaper. Moreover, the project is in line with the Government's priorities in the field of Family, Child Protection and Equality of Chances: ‘increasing the quality of life for the vulnerable children and fulfilment of minimum quality standards in the services for the children in difficulty’.41 Even though national priorities and strategies anticipate the shift from protection to prevention and therefore the development of community based services, until present the main response was limited mainly to the creation of some day care centers. The modelling project was designed to contribuite to more effective and efficient, easy to implement and sustain, community-based solutions, complementing thus the prevention action model at graasroot level. In the same time, the project's aim and objectives are highly compliant with the provisions of the UNICEF Child Protection Strategy, which states that the successful child protection begins with prevention, as well as with the Convention on the Rights of the Child with its fundamental principle of 'the best interests of the child'. The HIC project is founded on the following principles: i) the fulfilment of developmental rights is fundamental not only for children but also for the communities in which they live; ii) focusing on rural areas is an important means of reaching the most vulnerable and deprived; iii) a cross-sectoral approach is needed to make an enduring impact; iv) partnering with stakeholders at all levels ensures ownership and commitment; and v) wide coverage ensures critical mass, which in turn influences processes related to policy reform. (Foreword in Stănculescu and Marin, 2012) In 2012, in parallel with the social assistance services at community level, an additional initiative was launched aiming to contribute to the improvement of the legislative and regulatory framework for community health care/services (in relation to the new health law proposal). This intervention included: (1) Mapping42 of community health care/services and available resources with focus on the same geographical area as HIC; (2) Analysis of legislative and regulatory framework for community health care, including data of the mapping exercise; (3) Formulation of recommendation for the primary and secondary legislation for community health care (improving regulations regarding community care functioning, financing, reporting and coordination within the present decentralization framework). In 2013, the two initiatives merged into the program First Priority: No ‘invisible’ Child! Development of Basic Social Services at the Community Level. National Strategy for the Protection and Promotion of Children’s Rights, p.2, www.copii.ro Government’s Priorities for 2012, http://www.gov.ro/prioritatile-anului-2012-br-capitolul-9-familia-protectia-copilului-siegalitatea-de-sanse__l1a116016.html. The same goal is to be found in the Strategy for reforming the social assistance, 2011. 42 The mapping included data collection on the organization, delivery and regulation for the community health services, database analysis for health issues, inventory of available staff, recommendations for the local authorities for better integration of basic health and social services at community level, etc. 40 41
19
2.3 Geographical coverage The project has been implemented in eight counties, including all six counties of the North-East development region (Bacău, Botoşani, Iaşi, Neamţ, Suceava and Vaslui) and two counties from the South-East region (Buzău and Vrancea).43 In the first phase (2011), HIC covered 96 communes, while in the second phase (2012) it has been reduced to 64 communes. In the next phase (20132014), the project is planned to focus only on 32 communes (see table A.1 in Annex). Figure 1 Geographical coverage and the implementation phases
Preparation
Implementation
Rural communities from 8 counties in NE and SE regions
656 136
Community censuses for identification of 'invisible' children
96
32 Invisible' children
32
Counterfactual
32
Invisible' children
undertaken by local SPAS
64 Invisible' children benefited of the package of basic community services piloted within HIC
undertaken by local SPAS
32
Adjusted package of integrated socio-medical services HIC
2011 Jan.
Mar.Apr.
2012 Dec.Jan.
2013-2014 Dec.Jan.
Sept.
Note: Boxes present the number of rural communities included within the project.
In the preparation phase, 136 communes were selected out of all 656 communes in the covered regions, based on a secondary data analysis of the official statistics (National Statistical Office) as well as of data collected by previous research.44 Selection was refined by using data regarding children's vulnerabilities, at the commune level, for 2010-2011 (March), which were collected in each county from three institutions: County Inspectorate for Education, Directorate of Public Health and the General Directorate for Social Assistance and Child Protection. In the third step, the quantitative The North-East region and the Macroregion 2 (grouping North-East and South-East regions) are among the poorest in Europe. Source: Eurostat database, At-risk-of-poverty rate by NUTS region [ilc_li41], date of access: December 3, 2011. 44 Basically, three groups of variables were identified as the best fit to guide the selection process: (1) Social risk factors index, which was computed based on the Community Development Index (Sandu et al., 2009) and a Local Economy Development Index; (2) Index of the mayoralty's attitude towards social problem; and (3) Share of children in commune's population. (Stănculescu et al., 2011). 43
20
data were triangulated with qualitative information obtained through interviews with the key county stakeholders (General Directorate for Social Assistance and Child Protection - DGASPC and Prefecture). In the fourth step, UNICEF in cooperation with the relevant local stakeholders (DGASPC, Prefectures and agreements of mayoralties) selected the 96 communes that entered the project (11 or 13 communities per county, see table A.1 in Annex). In 2012, after the first formative evaluation, a second selection was carried out. Thus, 64 communes (8 communities per county, see table A.1 in Annex) were chosen based on criteria related to: (a) number of ‘invisible’ children and their proportion in total population of children; (b) performances of the social worker employed in the project in 2011; (c) participation of mayoralty and envisaged support for 2012. For the next period (2013-September 2014), 32 communes (4 communities per county, see table A.1 in Annex) were selected taking into account: (a) the results of this second evaluation; (b) the professionals in social fields (social worker, community medical assistant, school mediator, sanitary mediator, Roma mediator) that are available within the community; (c) supervisors' assessments regarding the relations with the social worker employed in the project as well as with the mayoralty. In these communities, an adjusted package of integrated socio-medical community services will be piloted in 2013-2014. 2.4 Stakeholders At the outset, HIC covered more than 409 thousands persons, of which over 114 thousands children (0-19 years). Due to the dropout of 32 localities, HIC coverage diminished in 2012 to almost 285 thousands persons and about 82 thousands children respectively, located in 64 communes (NIS, population data for 1 January 2010). The project coverage by county and implementation phase is shown in table A.2 in Annex. The key stakeholders implied in this project are located at three layers: national, county and local. Level:
Key stakeholders of HIC:
Community
-
County (judet) National
-
The ‘invisible’ children (0-17 years) and their families Social workers employed with the project Social workers working in SPAS Professionals providing social services, including community health nurse, school mediator, Roma health mediator, Roma mediator etc. Mayors and Community Consultative Structures DGASPC (supervisors and executive directors) and Prefecture Ministry of Labour, Family and Social Protection through its General Directorate for Child Protection (MMFPSPV, DGPC), Ministry of Health, Ministry of Education
The project reached all major groups of population: men and women, young and elderly, all ethnic groups, all social strata. Nevertheless, HIC has targeted the ‘invisible’ children who 'are disappearing 21
from view within their families, communities and societies and to governments, donors, civil society, the media and even other children’ (UNICEF, 2006, p. 35). In operational terms, HIC has defined
the ‘invisible’ child as a child that faces one or more types of vulnerabilities and has been reached by social workers through fieldwork activity. This working definition was developed during the first evaluation of the project (Stănculescu and Marin, 2012). By consensus, the social workers, supervisors and UNICEF together with researchers have agreed on a common definition for the ‘invisible’ children to be used in a standard manner across all project activities and communities. In addition, the types of vulnerabilities that make children ‘invisible’ have been determined based on lack of fulfilment of children’s rights, including the child’s right to education, health and ‘the right of every child to a standard of living adequate for the child's physical, mental, spiritual, moral and social development’.45 Consequently, within HIC, the ‘invisible’ children were identified through a census at community level, carried out by social workers employed in the project, based on the following list of vulnerabilities: (1) Children in households with many children, in poverty and precarious housing conditions; (2) Children left behind by migrant parents, living in poverty or other difficult situations; (3) Children at risk of neglect or abuse; (4) Children with suspicion of severe diseases; (5) Relinquished or at risk of child relinquishment; (6) Children out-of-school and children at risk of school dropout; (7) Teenage mothers who left school and/or are at risk of relinquishing the newborn child; (8) Children without ID papers or documents; (9) Other cases of vulnerable children. The ‘invisible’ children have included the cases newly identified in the field, but also children who have been already known at local level as being in a vulnerable situation, but about whom the field visit offered new insights (such as abuse, neglect, etc.), irrespective if his/her family have received some social benefits or services before the start of the project (e.g. social aid, heating allowance etc.). Thus, within HIC, the ‘invisible’ children refer also to children acknowledged as being in a
vulnerable situation but with ‘invisible’ vulnerabilities. In the second phase of implementation (in 2012), the main categories of ‘invisible’ children targeted by the project were: children without IDs, with migrant parents, living in precarious conditions and in crowded houses, in single parent families, not registered to medical doctors and minor mothers. For all these categories the project has strived to provide the appropriate support in order to address vulnerabilities, to fight against inequalities, and to prevent violence against children, including the separation of the child from her/his parents. For addressing the complex issue of ‘invisible’ children, UNICEF has developed partnerships with the Ministry of Labour, Family and Social Protection through its General Directorate for Child Protection, the General Directorates for Social Assistance and Child Protection, and the Institution of Prefectures in 8 counties and Public Local Authorities from 96 disadvantaged communes. At the national level, the General Directorate for Child Protection has participated in the project preparation and in designing the package of basic community preventive services. At the county 45
Article 27 of the Convention on the Rights of the Child.
22
level, the General Directorates for Social Assistance and Child Protection have played the role of coordination, resource centre and technical support for the social workers employed in the project. At the local level, the mayoralties have actively participated to project implementation, along with the Community Consultative Structures. Also, since 2012, partnerships have been developed with the County Public Health Directorates for the collaboration and cooperation with the community health nurse and Roma health mediator network at community level. At the institutional core of HIC has been the Public Social Assistance Services (SPAS after the Romanian acronym), as the main provider of social services and social benefits at local level. The way in which the SPAS are functioning impacts the whole system. Enhancing SPAS institutional capacity to reach the worst-off children and their families with appropriate protection and support represents the only effective way to improve children's welfare in Romania. To this end, HIC has developed the minimum package of community services and has piloted it (in 2012) in 64 of the poorest rural communities. Table 1. Typology of communities according to the availability of social services
Counterfactual - only local SPAS Counterfactual - local SPAS and social services other than HIC Local SPAS and HIC Only HIC HIC and other social services (no SPAS) Local SPAS and HIC and other social services Total
Participated in HIC only in 2011
Participated in HIC in 2011 and 2012
Total
17 15 32
16 7 6 35 64
17 15 16 7 6 35 96
Notes: HIC in 2011 consisted mainly in censuses focused on children's vulnerabilities and mobilization of the CCSs in the 96 selected communities. HIC in 2012 piloted the basic package of community services in 64 selected rural communities. Other social services refer to professionals in social fields, such as community health nurses, school mediators or Roma mediators. SPAS is either the Public Social Assistance Service or a person(s) with social assistance attributions within mayoralty. Schools and family physicians are not included in the typology.
However, on the one hand, in some rural communities included in the project, the Public Social Assistance Service or a person(s) with social assistance duties within the mayoralty did not existed in 2012 hence the social worker employed with the project covered all social assistance related activities. On the other hand, in other communities, the social worker employed with the project worked together with the local SPAS as well as with other various professionals in social fields, such as community health nurses, school mediators or Roma mediators. Consequently, the social services available within the community have varied considerable from one community to another. Six types of communities according to the availability of social services can be identified as shown in table 1. Thus, while in 7 rural communities the social worker employed with HIC covered all social assistance related activities, in 35 communes between 3 and 5 professionals were active in addressing the vulnerable groups. 23
2.5 The minimum package of community services Since 2005, UNICEF Romania has developed the Community Based Services Programme (CBS), which has been focused on the preventive approach in social protection system, especially in rural areas. During 2007 and 2008, UNICEF provided financial and technical assistance through the Child Protection program and expanded the prevention activities (with initial focus on hospital units) by developing pilot projects in 20 communities. In 2009, a Plan for Harmonisation of initiatives with a prevention focus was implemented. Since 2010, in line with the process of decentralisation and the reform of the social protection system, the focus towards enhancing the capacity of communities in providing preventive social services has accentuated further. Within this context, in 2011-2012, a group of experts elaborated the concept of the minimum package of community services for preventing separation of children from family as well as for better protecting the children against various risks and vulnerabilities. The minimum package of community services combines two complementary approaches: (i) social inclusion46 and (ii) children's rights.47 It results seven dimensions48 of social inclusion and fulfilment of children's rights. On each dimension, the main vulnerable groups49 and their specific needs were identified. Also, the social services for addressing these needs were determined (the available services at the community level, those that need to be improved as well as the ones that need to be created). As a result, for each dimension, a minimum package of community services was assembled, including both existing services, and some not granted or granted sporadically. The minimum package of community services is organized in six categories of services, namely: (1) identification, (2) needs assessment, (3) information, (4) counselling, (5) accompaniment and support, (6) referral, and (7) monitoring and evaluation. Both the minimum package of services and the minimum standards for preventing and combating the risk of social exclusion, at community level, can be and should be further developed in order to address other vulnerable groups, particularly with the entry into force of the Law on Social Assistance 292/2011, which aims to balance the cash social benefits and social services, in order to obtain consistent but also durable effects. Thus, this minimum package of basic social services will represent only a part of the social activity of local communities and social workers, namely the part related to the protection of children within family.
The social inclusion perspective refers to four dimensions of integration: civic, economic, social and interpersonal. Children's rights according to the Convention on the Rights of the Child, articles: 9 (right to develop in a family), 28 (right to education), 24 (right to the enjoyment of the highest attainable standard of health and to facilities for the treatment of illness and rehabilitation of health), 27 (right to a standard of living adequate for the child's physical, mental, spiritual, moral and social development), 26 (right to benefit from social security, including social insurance), and 31 (to rest and leisure, to engage in play and recreational activities appropriate to the age of the child and to participate freely in cultural life and the arts). 48 These dimensions are: civic integration and the right to an identity, social integration and the right to education /health /social protection/ development in a family, and economic integration and the right to an adequate standard of living. 49 For example, in relation to the dimension of civic integration and the right to an identity, the following vulnerable groups were considered: families that abandoned their child in a health care facility, children in the special protection system, single parents, teenage mothers, parents in divorce process, children with one or both parents left for work abroad, and parents without ID or civil status documents. 46 47
24
2.6 Theory of change The project ‘Helping the invisible children’ targets the situation of 'invisible' children who are ‘disappearing from view within their families, communities and societies and to governments, donors, civil society, the media and even other children’ (UNICEF, 2006, p. 35), with the ultimate objective of increasing their access to social services. Previous research has demonstrated that in Romania, 'invisible' children exist because (i) they do not have ID documents, (ii) community services are not available, which means that the response capacity at local level is very low and/or (iii) lack of awareness, lack of knowledge or fear of parents to address the existent services. Thus, 'invisible' children cannot be found in the administrative database or if they are registered they face 'invisible' vulnerabilities. Consequently, the first step in providing them support and protection would be to identify them through fieldwork activities based on risk indicators instead of administrative criteria. For example, several researches brought evidences that violence against children is quite spread but is 'invisible' in the administrative records either because the victims do not take action against the perpetrator or because the police does not take any administrative or legal action due to the tolerance towards this behavior, which is seen 'normal' or even 'educational'. Precisely to this aim, social workers need to be hired and trained for identifying the 'invisible' children through fieldwork activities. A community census has been considered the best option in this respect, for ensuring completeness of the identification process. Once the 'invisible' child is identified, his or her needs must be diagnosed and addressed in a systematic manner through a common effort of local actors. So, if the child has no ID papers than the local police should cooperate in order to obtain those. If the child is out of school or at risk of school drop out than the local school should work together with the social worker for finding the most appropriate way to enrol or to keep the child in school. So, the social worker should provide services with preventive character (such as information, counselling, accompaniment, support, and referral) together with the other relevant local actors, which usually participate in the Community Consultative Structures. To this aim, another line of activity of the social worker should be the activation of the Community Consultative Structure, which although were set up in accordance with the law aiming social inclusion are at a large extent dormant. On the other hand, in rural communities, especially in the poor ones, the human resources are scarce, particularly the specialized ones. For this reason, in order to develop a sustainable system built under severe budgetary and human resources contraints, the community should be left to decide on how and wherefrom to select and employ the social worker. As a result, it was expected that local authorities will find people who are not specialized in social work. For this reason, the social workers employed with the project need to be trained and also they need constant methodological support for carrying out the project activities. The best option is to build a partnership with the county DGASPC that is in the best position and have the best resources for assisting the change at the local level. In addition, in this way, the relation between the county DGASPC and the rural communities is strenghten and their institutional capacity is also enhanced. 25
In the same time, for ensuring the policy impact at the national level, the national structures in child protection were also invited as partner from the project onset. The entire project was designed as a learning process based on trial-error/success steps in which all stakeholders to be part since the initial phase. A scheme of the theory of change on which the project is based is presented in the next table.
26
Theory of change of HIC project
National/ central
County (judet)
Community
‘Invisible’ children and families
Activities -
-
Vulnerabilities of children and their families are assessed and addressed through individualized plan of services; Vulnerable children and families are informed about rights and entitlement and are assisted to access relevant services
64 social workers hired and trained: carry out mainly outreach activities and provide minimum package of services mobilise professionals within consultative community structures support for development project proposals and micro-grants organise experience exchange field trips -
Supervisors trained and equipped with instruments to monitor and ensure methodological guidance at local level Experience exchange organised
Mapping policies and strategies on prevention and community-based services and identifying bottlenecks and barriers for implementation
Expected outputs of HIC -
-
-
‘Invisible’ children (and their families) are identified and benefit of minimum package of services Children and families have more information about rights and entitlements and increased access to basic social services Increased capacity of social workers for identifying vulnerable children and their families Effective delivery of minimum package of services 24 community centres of support and counselling for children and parents are set-up 64 functional consultative community structures active Increased capacity of DGASPC to provide methodological support to local authorities
Evidence based justification for effective and efficient models of prevention services developed at community level
Expected outcomes of HIC Even in the disadvantaged rural communities: all children are visible in their families and in their communities for the health, education and social protection systems all children have access to primary health services all children of school age are enrolled in school all children are protected against separation from family all children are protected against all forms of violence (including neglect, abuse and exploitation) Improved community capacity to deliver social services (in 64 rural communities) Approximately 150,000 persons from rural areas better informed regarding the child rights as well as family rights and responsibilities 24 community centres of support and counselling for children and parents 64 functional consultative community structures that act in the benefit of the worst-off based on local action plans. Reduced pressure on the child protection system (in 8 counties) Improved capacity of DGASPC to provide methodological support to local authorities using the newly established county support centres for communities. Strengthening the national strategy in the prevention of separation of children from family and of violence against the child, through effective and appropriate budget.
Impac
Increased the impact of social protection policies for poor and socially excluded (‘invisible’) children aged 0-17 years and their families.
Poor children and their families are ‘invisible’ and accumulate vulnerabilities and thus increasing risk of social exclusion
Initial
2.7 Flow of activities Project preparation Selection of communities in the North-East and South-East regions. A (April-June 2011) number of 96 communes were included in the project, after Mayor’s commitment was obtained (see also section 1.3). Phase I (June-December 2011)
In all 96 rural communities, a social worker was selected by local authorities, he or she was employed with the project and participated in a 2-day training session. The project social workers carried out outreaching activities and mobilized the Community Consultative Structures (CCS). More precisely, they completed a community census for identifying the ‘invisible’ children and families. In November 2011, a first formative evaluation of the project was conducted by CERME.
Phase II (February-December 2012)
In 64 rural communities, a model of minimum package of community preventive social assistance services was piloted with the purpose to reach the worst-off children and families (the identified ‘invisible’ children), by the project social workers with the help of SPAS and Community Consultative Structures, under the coordination of county supervisors (DGASPC). All 64 communities were invited to participate to a call for proposals for a micro-grant up to 10,000 lei (about US$ 3,000) and 25 of them (3-4 communes per county) were selected (see Annex 6.1). In the 32 communities that left the project in 2012, supposedly the local SPAS took over the identified cases of ‘invisible’ children. This activity was not organized under the project, but was left to the decision of local authorities and DGASPC. In December 2012-February 2013 a second formative evaluation was conducted by CERME, which is presented in this report.
Phase III (June 2013-October 2014)
Revise the model of minimum package of community preventive services in rural areas in accordance with the results of the second evaluation. Complete the minimum package with: (1) health services carried out by community health nurses and Roma health mediators and (2) micro-grants for all included communities. Pilot this extended model of integrated community preventive (socio-medical) services in 32 rural communities. A summative evaluation is planned to be carried out for preparing the model of basic community services to be scaled up by the state.
Phase IV
The state takes over the minimum package model and pilots it.
(November 2014-2015)
An impact evaluation will be carried out by the end of 2015, including the use of the counterfactual and baselines built in 2011 and 2012.
2.8 Desired outcomes Level
Desired outcomes of HIC:
‘Invisible’ children and families
Even in the disadvantaged rural communities: - all children are visible in their families and in their communities for the health, education and social protection systems - all children have access to primary health services - all children of school age are enrolled in school - all children are protected against separation from family - all children are protected against all forms of violence (including neglect, abuse and exploitation)
Community
-
County (judet)
National/ central
-
Improved community capacity to deliver social services (in 64 rural communities) Approximately 150,000 persons from rural areas better informed regarding the child rights as well as family rights and responsibilities 24 community centres of support and counselling for children and parents 64 functional consultative community structures that act in the benefit of the worst-off based on local action plans. Reduced pressure on the child protection system (in 8 counties) Improved capacity of DGASPC to provide methodological support to local authorities using the newly established county support centres for communities.
Strengthening the national strategy in the prevention of separation of children from family and of violence against the child, through effective and appropriate budget.
29
3 Evaluation Profile 3.1 Evaluation overview The purpose of this evaluation, in accordance with the ToR (see Annex 7.2), is to collect evidence related to project effectiveness and efficiency, contributing to the understanding of its relevance and impact. In this respect, we use the OECD/DAC criteria for evaluating development assistance: (1) relevance - the extent to which the project is suited to the needs, priorities and policies of the target group, recipient and donor; (2) effectiveness - the extent to which the project activities attain its objectives; (3) efficiency - a measure of the outputs (qualitative and quantitative) in relation to the inputs (to what extent the project uses the least costly resources possible in order to achieve the desired results); (4) impact – the positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended; (5) sustainability – a measure of whether the benefits of an activity are likely to continue after the withdrawal of donor funding. (OECD, 1991) As the relevance of the programme is extended across the key stakeholders, the information provided under this report is useful for UNICEF, the national policy decision-makers, the HIC decision-makers as well as HIC beneficiaries. The specific objectives of the evaluation are: i) to contribute to the design of the model of minimum package. Information generated by the evaluation about the model of the minimum package will inform stakeholders at local level (social workers, community professionals, and mayors), county level (DGASPC and supervisors) and central level (MMFPSPV) through dissemination of the report. Consultation workshops to be organised with these stakeholders will input new design of the minimum package in 2013-2014 ii) to elaborate evidence based recommendations of which area to continue intervention and how to do it in the most appropriate way, besides funding availability, which will be used similarly as above; iii) to generate input for the evaluation of National Strategy for the protection and promotion of children’s rights 2008-2013, which need to be revised in 2013. The group of experts already working on the development of the new National Strategy for 2014-2020 are using preliminary findings of the current evaluation for setting priority strategies and for design of effective and efficient directions for operationalization, and iv) to gather reliable information to advocate for the model and to develop policy recommendations in the anticipation of UNICEF 2014 activities. As such, for the local stakeholders (both counties and communes), the evaluation offers evidence related to practices to be implemented or kept from previous interventions as well as on the ways in which to adjust the local action plans and to prioritize in a proper way. For all institutional stakeholders, this evaluation report presents evidence related to all categories of vulnerable children and families and makes their voice heard. This evaluation is financed by UNICEF and carried out by a team of consultants from the Romanian Center for Economic Modeling (CERME) and the Research Institute for the Study of the
30
Quality of Life (ICCV) – see Annex 7.3. This is a formative evaluation,50 which is a method for judging the worth of a program while the program activities are in progress. The focus is on the process, which permits the stakeholders to identify project strengths and weaknesses, and to target deficiencies that need revise and adjustments. 3.2 Research questions This evaluation focuses on the relevance, effectiveness, efficiency and sustainability of the HIC project. These criteria were choosen as: (1) relevance – is measurable at this phase and may explain to what extend the modelling project is suited to the needs of vulnerable children and families, and at the same time to the social inclusion priorities and policies to be designed at county and central level; (2) effectiveness – is as well measurable and may bring evidence for how the minimum package of services may contribute to the increase of impact of social protection policies for poor and socially excluded (‘invisible’) children and their families; (3) efficiency – is measurable and represents a powerful argument for scaling up the model especially if community based service, including the minimum package of services are least costly resources possible in order to increase impact of social protection policies; (4) sustainability – may be measured especially in the context of a future implementation of UNICEF’s exit strategy from the modelling project funded for a period of 3 years. The OECD-DAC evaluation criteria as the humanitarian ones (e.g. connectedness or coherence) are not relevant for this task. For measuring impact of the minimum package of services, the time of implementation is still too short. Nevetheless, some elements of impact in terms of outputs are inventoried. A rigorous impact evaluation, including the use of the counterfactual and baselines built in 2011 and 2012, is planned after at least one year of implementation of the revised package of socio-medical services (estimated at the end of 2015). The research questions guiding the evaluation have been the following: For the relevance: What is the value of the intervention in relation to the needs of the worst-off groups and reduction of inequities between the best-off and the worst-off groups? And in relation to the institutional developmental needs? And in relation to the equity approach as well as Human Rightsbased Approach to Programming and Gender Mainstreaming? And in relation to covered communes: 64 in 2012 versus 96 in 2011? For the effectiveness: Is intervention achieving satisfactory results in relation to stated equity-focused objectives? Were the contextual factors (political/social/economic/cultural) taken into account in the design and implementation of the intervention? Are the services provided effectively reaching the worst-off groups? Is there a child rights approach involved in all processes? Are any gender issues involved in providing the services? What are the main constraints on supply and on demand? Which parts of project are most and least effective? What factors explain success? What are the By contrast, a summative evaluation is a method of judging the worth of a program at the end of the program activities. In this case, the focus is on the outcome. All assessments have the potential to serve a summative function, but only some have the additional capability of serving formative functions. (Scriven, 1967) 50
31
differences between the 64 communes in 2012 and the 32 which were not kept anymore from the 2011 intervention? For the efficiency: Does the programme use resources in the most economical manner to achieve expected equity-focused results? Is this based on the principles of child rights? Are any other economical alternatives feasible? How cost-effective is the project for reaching worst-off groups? How do costs for reaching worst-off groups compare with public services costs? However, since there are not standard costs either in relation to prevention services or to the minimum package, the efficiency is limited on the existing costs within the project, which may differ from the costs in a different context. A proper cost-benefit analysis is not included within this evaluation. For the sustainability: To what extend the current context is more or less favourable to continuation of such approaches in the near future (short and mid-term)? Are the intervention and its impact on the worst-off groups likely to continue when external support is withdrawn? Are inequities between best-off and worst-off likely to increase, remain stable, or decrease when external support is withdrawn? Will the strategy be more widely replicated or adapted? Is it likely to be scaled up? What happened in the 32 communes dropped in 2012? 3.3 Scope of evaluation and counterfactual For answering the research questions, the evaluation has covered all key stakeholders, all counties (8), all communities (95)51 and all activities carried out in the project in 2012 (in 64 rural communities that remained in the project), as showed in the Figure 1 Geographical coverage and the implementation phases. The evaluation has included a counterfactual comprising the ‘invisible’ children identified in the 32 communities which were not anymore part of the project in 2012. These children were supposed to benefit of the services existing in these communes (e.g. SPAS regular services, Roma mediator, community health nurses, etc.) but not of the services provided within HIC project. Also, in these 32 communities, the outreaching activities of identification of the most vulnerable children and their families were ceased in 2012. In contrast, in the 64 communities remained in the project, the identification of the most vulnerable has been kept as a component of on the ground activities and the ‘invisible’ children have been supposed to benefit of the services provided under the minimum package approach. Thus, the evaluation compares:
51
‘invisible’ children in both types of communes and how their situation evolved in 2012;
the services in both types of communes, including support provided from county level to communes in the project versus virtually no support in communes not included in the project (taking also into account the typology of communities according to the social services availability as shown in table 1);
the results for children in both types of communes.
From the 96 communes included in the project, Trifeşti (Iaşi) has never offered any data.
32
3.4 Data of evaluation The evaluation used a mixture of primary and secondary sources of information so that to ensure high quality and credibility. Not all stakeholders collect, centralize and manage data in the same way, hence the evaluation has carefully analysed all available information and has drawn conclusion only after all data were compared and processed. As it concerns the existing sources of information, the evaluation has relied on: i) the 2011 evaluation report (Stănculescu and Marin, 2012), including quality assessment comments and recommendations; ii) all regular monitoring reports carried out by supervisors and by social workers; iii) all UNICEF reports (including field monitoring); iv) databases for case management filled-in at community level; v) legislation in place: law on social assistance (2011) and revised law on child protection (2012). Social workers employed with the project were monitored throughout the project by the county supervisors. The aim was to observe the implementation process, the timing, involvement of local stakeholders (especially the local authorities and Community Consultative Structures) and the potential changes generated by the project. The monitoring process served as a quality-assurance tool and a source of data for evaluation. Thus, as we have mentioned, the previous evaluation, the regular monitoring reports as well as local case management databases, which form the monitoring framework of the project, were used in this assessment both during interviews with representatives of the institutions that implement the project and for drawing (after cleaning and consolidation) the sampling frame, respectively the sample of 'invisible' children and their families (including the control group from the 32 communes that left the project in 2012). However, most data were collected with the special purpose of this evaluation. Primary data were collected both at the micro (individual, household) and meso (community) levels, through quantitative and qualitative research techniques. This external evaluation was conducted at UNICEF Romania's request by a team of local researchers from the Research Institute for Quality of Life (Romanian Academy) and a research non-governmental organization - CERME. The selection criteria, the responsibilities of the research team and the methodological milestones for the evaluation are outlined in the Terms of Reference (see Annex 7.2). The fieldwork was carried out by CERME, in December 2012 - January 2013. The data collection methods have involved stakeholders irrespective of their age, gender, political or religious beliefs, social or economic status.
33
Table 2. Research design
DATA SOURCE/ TARGET GROUP Preparatory phase Local databases for case management (Databases)
SAMPLING METHOD Exhaustive
Consolidated and cleaned database (sampling frame) Survey on the The identified ‘invisible’ children and their families ‘invisible’ children
95 databases with 8,347 ‘invisible’ children 5,758 ‘invisible’ children
Random
1,006 ‘invisible’ children from 60 communes
Exhaustive
8 interviews
DGASPC Directors
Exhaustive
8 interviews
Prefecture Representatives
Exhaustive
8 interviews
Geographical and case-diversity criteria
4 FG with 37 SWs, from 8 counties
Probabilistic, stratified two-stage
60 SWs and 235 CCS members from 60 communes
2 photos per surveyed household
280-300 selected photos
Interviews DGASPC Supervisors
Focus-groups HIC Social Workers (SWs) Opinion Survey
VOLUME
Social Workers (SWs) Mayoralty representatives (mayor, vice mayor, secretary, social worker not participating with the project) Community Consultative Structure (including teachers, doctors, policemen, priests etc.) Other local professionals
Photo voice Photos (with the parents' agreement)
Each research component is presented in a section below. Children have been involved in the data collection process only in the presence of their parents/relatives. They have been interviewed by experienced researchers at their homes. Their identity has not been disclosed in any of the materials produced in the evaluation. The interviews have been recorded on paper. The questionnaires and photographs are stored in a secured place with the research company conducting the field work, CERME, which is a certified research and development unit (by the National Authority for Scientific Research) and holds a license from the National Authority for Personal Data Protection. The Terms of Reference is the regulating document of the evaluation, which set up its objectives, schedule, the evaluation criteria, the reseach questions and parameters of the evaluation process. The methodology focused on collecting quantitative and qualitative data, drawing on all possible data sources (including all stakeholders and all project activities) and on upholding the highest ethical standards. This evaluation is part of the learning process and not an audit. As the project is implemented both at the individual level (intervention actors, 'invisible' children and their families) and at the institutional level (SPAS, DGASPC, local authorities, CSCs etc.), the evaluation is based on a mix of data and research methods that would allow the identification of the factors that can support or impede the scaling up of the model of minimum package of community preventive 34
services at the national level. In this way, the evaluation facilitates the learning process based on the acquired experience and offers inputs for the next phase of the project. In the same time, this evaluation provides detailed and objective information, thus being a tool of quality assurance which supports evidence-based decisions but also recommends solutions for improving the project. To this aim, the evaluation is guided by rigorous research standards as well as objectivity and impartiality of the evaluators. The evaluation methodology has several advantages, including the use of a mix of quantitative and qualitative methods, the use of a mix of data sources for reflecting the perspectives of all stakeholders, the fact that the quantitative data refer to representative samples, the utilization of a participatory approach (all key stakeholders took part in the evaluation activities), the collection of data for specific-evaluation purposes. Table 3. Stakeholders' participation in the evaluation process Categories of stakeholders 'Invisible' children and their families HIC Social Workers SPAS Social Workers CSCs Other local professionals Local authorities County DGASPC County DSP Prefecture MMFPSPV UNICEF
Preparatory phase X
Survey X X X X X X
X
Photo voice X
Interviews
Focus groups
X
X
X
Community visit X X X X X X
Validation workshops X X X X X X X
X X
X X
Note: Validation workshops are presented on page 112.
Besides the fact all key stakeholders took part in the evaluation activities, the research included the photovoice method. The families of 'invisible' children who agreed to participate were asked to take 1-2 photographs and to develop narratives for explaining the benefits that the project brought for them. In this way, the 'invisible' children and their siblings provided their voice and views about the services they received. So, they conceptualize their circumstances and their hopes and expectations from the project. This information might be used for operationalizing the tangible impact of the project in the impact evaluation planned for the end of 2015. Thus, the methodology is in line with previous researches that highlight the importance of the engagement, interaction, and communication between evaluation clients and evaluators for a meaningful use of evaluation (Johnson et al., 2009). The Evaluation matrix (see Annex 7.6), outlines the specific linkages between methodology, the mix of methods and data sources, including stakeholders participation, with the evaluation questions and the selected evaluation criteria in accordance with the Terms of Reference.
35
Nevertheless, there are also limitations. The main one is given by the difficulty to separate the project outputs/outcomes of the results of other interventions in the area of preventive social services. For example, since mid-2010 UNICEF has financed the School Attendance Initiative (SAI) with the aim of getting children of school age back to school and supporting them to complete the compulsory years of schooling. SAI has been implemented in over 230 rural communities among which 14 communities covered by HIC. Also, since 2006, the Ministry of Health (MS)52 has financed the development of a network of community health nurses and Roma health mediators with the aim to improve the preventive medical services in rural area. This national program has covered 43 communes included in HIC (in 2012-2013). In addition, non-governmental organisations (or nonformal charitable groups) have developed various initiatives in the region, providing support to vulnerable children and their families. Consequently, the situation as measured in January 2013 is a combined result of all such interventions, which were implemented only in some communities (and not in all), have not been coordinated or consistently applied, some were only sporadic, and most are not documented at all. However, the counterfactual reflects the situation and developments in the absence of the Helping the ‘invisible’ Children project. Other limitations are caused by the discontinuity of project activities (between January and March 2012).53 As a result of change in the project instruments, local stakeholders did not collect, centralize and manage data in a uniform manner. So, the reconstruction of the case history for the ‘invisible’ children identified with the project was rather difficult. In the 32 communes that participated only in the phase I of the project, the local databases for case management were not completed because the social workers employed in the project have no longer been available in the community (some cannot be contacted or have different work arrangements). In other communes, the databases for case management were only partially linked with the databases of ‘invisible’ children identified in phase I of the project. However, data related limitations were tackled as presented in the next section. Limitations in answering all evaluation questions exist as well. One of them is linked to differences between the 64 communes in 2012 and the 32 which were not kept anymore from the 2011 intervention – to be addressed when analysing effectiveness, relevance and sustainability criteria – which were analyses consistently, but not addressed necessarily separately. For example, in order not to duplicate information, the answers on “What happened in the 32 communes dropped in 2012?” under sustainability criteria, can be found in the chapter on findings regarding effectiveness. Similarly, addressing “Are inequities between best-off and worst-off likely to increase, remain stable, or decrease when external support is withdrawn?” was overlapping data and analysis already mentioned in effectiveness chapter, by including comparison in provision of services (1-7) for 'invisible children in terms of coverage, targeting (including child rights approach and gender issues) This national program has been based on a pilot project implemented starting with 2002 by UNICEF, UNFPA, USAID and John Snow Institute. Since 2009, the decentralization reforms have affected the development of the community health care givers (community health nurses and Roma health mediators). Thus, the network of community health care givers has considerably shrunk. At present, at the national level, there are 979 community health nurses and 389 Roma health mediators (MS 2013 statistics) compared to 1,228 and 498 respectively, in 2008, before their transfer to the local authorities. 53 The discontinuity of project activities occurred also between January and March 2013. 52
36
and data accuracy (registration) between communes active in 2011 and still implementing the project in 2012. Last, but not least, it is to be highlighted that designed as a formative evaluation in accordance to the Terms of reference, data and findings do not necessarily report at outcome level as it focuses on the process for achieving expected outcomes. While in some instances, the data may report at outcome level especially in terms of results at children, family and sometimes community level, most of the findings concentrate on analysing the project while activities are in progress in order to identify strengths and weaknesses, and to guide for adequate revision and adjustments. 3.4.1
Consolidated database of ‘invisible’ children
In the first step, a comprehensive monitoring exercise was developed in cooperation with the county supervisors (8), the project social workers (64) and the SPAS (95) from all HIC rural communities selected initially. All (64) local databases for case management have been cross-checked with the databases of ‘invisible’ children identified in November 2011 (Stănculescu and Marin, 2012). A cleaned consolidated database of vulnerable children (worst-off/ '‘invisible’ ') from all 9554 rural communities was prepared. This database was used as sampling frame. The database has included for each child the following information: name, address, problems/ vulnerabilities and social protection actions (by type of problem). In phase I of the project, a number of 3,041 ‘invisible’ children were identified as at November 2011. However, the process of finalizing and cleaning the community censuses continued until January 2012. Afterwards, the project was adjusted, the minimum package of preventive community services was defined, the targeted vulnerable groups were specified, new M&E instruments were elaborated, and a format of database for case management at the community level was implemented. Preparation of these new instruments took few months, during which the continuity of activities within the project was not ensured. Due to this gap, in some communities or even counties, the databases of ‘invisible’ children suffered all sorts of changes. For example, some communities included among the ‘invisible’ children all children receiving social benefits such as Minimum Guaranteed Income or family benefits as they have been also in vulnerable situations which require constant monitoring. However, the approach was not uniform across communities. Consequently, at the end of the phase II of the project (2012), the 95 local databases included 8,347 children out of which only a part were ‘invisible’. By confronting the local databases from 2011 and 2012, the history of each case was reconstructed. Based on this information, a consolidated database (cleaned and uniform) was prepared so that to include only the ‘invisible’ children defined as in section 1.4. The consolidated database comprises 5,758 ‘invisible’ children from 95 communes.
54
Commune Trifesti has never reported and did not participated with the project in 2012.
37
3.4.2
Survey on a random sample of ‘invisible’ children and their families
Out of the consolidated database, a random sample of 1,006 ‘invisible’ children was extracted, with a margin error of ± 2.8% at 95% confidence level. The total response rate was 99%.55 Thus, the sample consists of 708 households with valid questionnaires, including 923 children registered as ‘invisible’,56 1,724 siblings below 18 years old (not registered as ‘invisible’ children) and 1,441 adults (18 years or over), located in 60 rural communities. The sample is representative at the level of ‘invisible’ children, but not at the level of households with ‘invisible’ children; households with more children recorded as vulnerable have higher chances to be selected compared with households with only one recorded child, irrespective of the number of children in the household. For example, let's consider two households with three children. The first one is poor, have precarious housing conditions and parents make abuse of alcohol. Correspondingly, all three children from this household are registered as ‘invisible’ in the local database. The second household is not poor, parents provide proper care, but one child is suspected of a severe disease which is not diagnosed. Correspondingly, in this household, only one child is recorded as '‘invisible’ '. Therefore, the chances of the second household to be selected in the sample are three times lower than those of the first household. This is mainly the result of the present format of the database for case management at community level, which requires only the registration of few data about the 'invisible' child, but no information on the household/ parents or siblings. The sample covers all counties and all types of communities according to the availability of social services (table 3). Table 4. Distribution of the sample by community types according to social services availability (%)
Total
N % Counterfactual - Participated in HIC only in 2011 Counterfactual - only local SPAS Counterfactual - local SPAS and social services other than HIC HIC - Participated in HIC in 2011 and 2012 Local SPAS and HIC Only HIC HIC and other social services (no SPAS) Local SPAS and HIC and other social services
‘Invisible’ children
Siblings (0-17 years)
Adults (18+ years)
Total persons
923 100 13.5 5.2 8.3 86.5 24.6 4.6 9.2 48.1
1,724 100 14.7 5.7 9.0 85.3 23.8 3.9 9.9 47.7
1,441 100 13.1 5.3 7.8 86.9 24.9 4.5 7.2 50.2
4,088 100 14.0 5.5 8.5 86.0 24.4 4.3 8.8 48.7
Data: UNICEF and CERME, Survey on the ‘invisible’ children and their families, January 2013. Notes: HIC in 2011 consisted mainly in censuses focused on children's vulnerabilities and mobilization of the CCSs in the 96 selected communities. HIC in 2012 piloted the minimum package of community services in 64 selected rural communities. Other social services refer to professionals in social fields, such as community health nurses, school mediators or Roma (health) mediators. SPAS is either the Public Social Assistance Service or a person(s) with social assistance attributions within mayoralty. Out of the sample of 1,006 '‘invisible’ ' children, 935 (or 93%) were identified in the field, of which 923 answered the questionnaire, while the others refused to participate with the study. The rest of 7% of the sample could not be found; they were left abroad with their parents (2.4%), they were moved to other settlements (1.8%) or simply no information about them was available (2.8%). 56 Out of the '‘invisible’ ' children, 23 reached 18 years old and 3 of these are enrolled in education. 55
38
Data collection at the level of the worst-off children and families was based on a questionnaire (see Annex 7.8.1), applied face-to-face at the respondents' homes. Unlike the database for case management at community level, the evaluation questionnaire has included information about all household members (all ages) with respect to: (a) socio-demographics (b) vulnerabilities; (c) the social assistance services received by the household members; (d) beneficiaries’ assessment of these services; (e) perceived impact over the child/ household; (f) incomes and housing conditions. The interviews with the worst-off observed all ethical standards. 3.4.3
Focus groups with social workers employed in HIC project
Four focus groups with 37 social workers involved in phase II of HIC project were conducted in January 2013 (table 4). The selection of participants to focus-groups has been based on the supervisors’ recommendations. Women have been over-represented among participants as they represent 81% of all social workers employed in the project (in 96 communes). Table 5. Location and number of participants at the focus-groups discussions with social workers County Neamţ Botoşani Vaslui Vrancea
Number of participants, social workers employed in the project 9 11 8 9
Number of participants per county 4 Bacău and 5 Neamţ 8 Botoşani and 3 Suceava 4 Vaslui and 4 Iaşi 4 Vrancea and 5 Buzău
The social workers gave information about: (a) how the minimum package of preventive community services was implemented in their commune: main difficulties in addressing effectively the ‘invisible’ children's problems, institutional bottlenecks and barriers in the area of prevention services, the changes in people behaviour and in opportunities associated with the project that are valued within community, risk factors that may hamper the positive impact, elements of the minimum package that need revision, time allocated by service included in the minimum package; (b) opinions about the relevance of the minimum package of preventive community services for the specific problems and vulnerabilities of the children within community; (c) time spent with the HIC and non-HIC social assistance activities; (d) support received from DGASPC; (e) micro-grants financed within HIC57 and how effective have been these in improving the vulnerable children's lives. The annex 7.8.2 presents the focus group guide. 3.4.4
Interviews at county level
Structured interviews were conducted with all county key stakeholders based on the guides presented in annexes 7.8.3 and 7.8.4. Thus, the project supervisors, DGASPC directors, and Prefecture representatives (under-prefects or directors), from all eight counties, provided their view on topics related to: (a) implementation of the minimum package of preventive community services: Three-four communes in each county (a total of 25 communes) benefited in 2012 of a micro-grant of 9,000 lei (aprox. US$ 2,700) for implementing a project developed at community level. The list of micro-grants is presented in Annex 6.1. 57
39
main difficulties in addressing effectively the ‘invisible’ children's problems, institutional bottlenecks and barriers in the area of prevention services, impact of the minimum package of services by commune, risk factors that may hamper the positive impact, elements of the minimum package that need revision; (b) the support provided by DGASPC to communities and possibilities to scale-up the resource centre for covering all communes within county; (c) micro-grants financed within HIC58 and how effective have been these in improving the vulnerable children's lives; (d) capacity of the county stakeholders to uptake and extend the project in more communes within county. 3.4.5
Opinion survey on community representatives
The survey is similar to the one completed in the first formative evaluation (November 2011). The survey was conducted in the period of December 2012 – January 2013, by specialized field interviewers of the Romanian Centre for Economic Modeling. Data collection method: face-to-face interviews based on questionnaire. Type of sample: probabilistic, two-stage stratified, representative across types of communes59 and stakeholders (see tables A. 3 and A. 4 in Annex). The selected communes resulted from the sample of ‘invisible’ children (60 rural communities). Within each chosen commune, 3-4 community representatives were selected out of the following eight categories of relevant local stakeholders: (1) mayors; (2) vice-mayors; (3) secretaries of mayoralty; (4) teaching staff, school mediators; (5) doctors, nurses, sanitary mediators, community health nurses or Roma health mediators; (6) priests, business owners, Roma mediators; (7) policemen; (8) social workers. At the county level, the field operators were instructed to have a balanced distribution of questionnaires across all categories of stakeholders. However, no quota was provided per county. Overall, 235 community representatives responded in the opinion survey. In addition, all 60 social workers from the selected communities participated in the survey.
See footnote 35. Types of communes regarding both the typology according to social services availability and the level of community development (measured against the index IDC elaborated by Sandu et al. 2009). 58 59
40
Figure 2. Distribution of respondents in the opinion survey by category of local stakeholders 70
1200 Number of res pondents
60
1006
Number of 'i nvi s i bl e' chi l dren a bout whom the res pondents provi ded i nforma tion 50
1000
800
40
686
691 600
517
30
501 446
413
461 400
20 231
200
10 29
27
Ma yors
Vi ce ma yors
42
38
31
19
23
26
60
Doctors , nurs es , s a ni tary medi a tors , communi ty medi ca l a s s i s tants
Pri es ts , l oca l bus i nes s es , Roma medi a tors
Pol i cemen
Loca l SPAS (Soci a l workers not pa rtici pa ting wi th the project)
HIC s oci a l workers
0
0 Secretari es of Tea chi ng s taff, ma yora l ty s chool medi a tors
Data: UNICEF and CERME, Opinion survey, January 2013. N=235 community representatives and 60 HIC social workers from 60 rural communities.
Similar to the first evaluation, the questionnaire for community representatives has covered topics related to: (a) the level of knowledge about the project, (b) relevance of the minimum package of preventive services for the community needs and (c) the benefits for community - perceived and expected. In addition, for each ‘invisible’ child selected in the sample, the community representatives provided information about: (a) knowledge about the case and (b) participation in finding solutions for the case. On the other side, the social workers employed with the project assessed the degree of involvement and role played by each local actor, the relation and support received from the DGASPC resource centre, main problems and difficulties encountered in the implementation of the minimum package of services. Also, for each ‘invisible’ child selected in the sample, they provided information about: (a) knowledge about the case and (b) the relation with the ‘invisible’ children's families. The annex presents the two questionnaires used for community representatives and for the HIC social workers. 3.5 Method of data analysis Data analysis (bivariate and multivariate) has systematically tested the following multi-level set of predictors: -
Individual level: children's gender, age, ethnicity, health status. Regarding the caretakers, parents, relatives and other adults living in the households with ‘invisible’ children we also use data on presence at home, marital status, education and occupation (employment); 41
-
Household level: type of household; household size; number of children within household; monthly cash income per person; in-kind incomes (food from own garden/ land); social benefits received by household; (absolute and relative) poverty; extreme poverty (hunger and insufficient heating during winter); subjective assessment of income and of housing conditions related to family needs;
-
Community level: county, community typology according to the availability of social services, number of ‘invisible’ children within community, type of social worker (only HIC, HIC and SPAS, only SPAS), and the level of community development (IDC index).60
The report presents only the results that are statistically significant at p
