Children and Youth Services Review 61 (2016) 353–358
Contents lists available at ScienceDirect
Children and Youth Services Review journal homepage: www.elsevier.com/locate/childyouth
What's the difference? Using descriptors to classify the care provided to children and adolescents with behavioral and emotional problems K.E. Evenboer c,⁎, A.M.N. Huyghen a, J. Tuinstra b, E.J. Knorth a, S.A. Reijneveld b a b c
Department of Special Needs Education and Youth Care, University of Groningen, Grote Rozenstraat 38, 9712 TJ Groningen, The Netherlands Department of Health Sciences, University Medical Center Groningen, University of Groningen, PO Box 196, 9700 AD Groningen, The Netherlands University Medical Center Groningen - Health Sciences (FA10) Ant. Deusinglaan 19713 AV Groningen The Netherlands
a r t i c l e
i n f o
Article history: Received 31 August 2015 Received in revised form 12 January 2016 Accepted 12 January 2016 Available online 14 January 2016 Keywords: Mental health care Child and youth care Primary health care Behavioral problems Emotional problems
a b s t r a c t More detailed information concerning the content of interventions for children with behavioral and emotional problems may help to improve their effectiveness. In this study, we made a distinction between “well-defined” and “poorly defined” interventions that were being provided in a catchment area. Well-defined interventions are included in the Dutch “Effective Youth Interventions” (EYI) database; poorly defined interventions are not. We aimed to assess (1) to what extent “well-defined” interventions had similar content, that is, could be grouped together, and (2) whether the proportion of those interventions that could be grouped was smaller for “well-defined” than for “poorly defined” interventions. The interventions were scored by professionals in terms of the degree to which the activities entailed in that intervention were covered by the 20 descriptors that represent that specific type of care. Those interventions with similar scores on descriptors were then grouped together. The percentage of interventions that could be grouped was then compared with that found in an earlier, comparable study concerning “poorly defined” interventions. “Well-defined” interventions could be classified into 19 groups; this represented a reduction in interventions of 44%, with the largest reduction found in those interventions within the main types “individual child support” and “family support.” This reduction was somewhat smaller than for the “poorly defined” interventions (52%), where the largest reduction was found in the main type “family support.” The descriptors then allowed interventions offered to children to be grouped within and across care organizations. In this way, we were further able to distinguish differences and similarities in the content of grouped interventions per main type of support. © 2016 Elsevier Ltd. All rights reserved.
1. Introduction The effectiveness of the interventions offered to children with behavioral and emotional problems depends mostly on their content (Abraham & Michie, 2008; Ballinger, Asburn, Low, & Roderick, 1999; Garland et al., 2010; Lloyd-Evans, Johnson, & Slade, 2007). A system to describe and compare the content of interventions (e.g., the techniques used by professionals such as prompting a child to express emotions, or teaching a young person how to deal with setbacks and frustrations, etc.) may help policymakers and practitioners in making evidencebased decisions regarding the choice of care provided to children with behavioral and emotional problems (Chorpita & Daleiden, 2009; Cjaza, Schulz, Lee, & Belle, 2003; Ezell et al., 2011; Harden & Klein, 2011; Lee & Barth, 2011; Marsh, Angell, Andrews, & Curry, 2012; Miller & Rowe, 2009; Yohalem & Wilson-Ahlstrom, 2010). The labeling of interventions frequently does not reflect the actual content of an intervention accurately. Identical labels are often used for quite different treatments, while similar treatments may be given ⁎ Corresponding author. Tel.: +31503632848. E-mail address:
[email protected] (K.E. Evenboer).
http://dx.doi.org/10.1016/j.childyouth.2016.01.010 0190-7409/© 2016 Elsevier Ltd. All rights reserved.
different labels (Van Yperen, Van Rest, & Vermunt, 1999; Lloyd-Evans et al., 2007). Therefore, more knowledge on a detailed level is needed in terms of the content of care and treatment. In some earlier studies, the focus was on characterizing the content of care by using a list of behavioral/psychological change techniques (Abraham & Michie, 2008; Chorpita & Daleiden, 2009; Michie, Hyder, Walia, & West, 2011). However, in psychosocial care for youth, other aspects of the care process, such as duration and intensity of the care, are also important and so should be classified as well. We previously reported about our application of this method for “poorly defined” interventions (Authors' own, 2014b). “Poorly defined” meant that an intervention had not been included in the database of “Effective Youth Interventions” (EYI) (Netherlands Youth Institute, 2013). The EYI database was developed in The Netherlands to document information on the effectiveness of interventions. These interventions were assessed by an independent committee of national experts who evaluated interventions in the context of accreditation for the EYI database (Zwikker, Van Dale, & Kuunders, 2009). They did so using four criteria: (1) whether a protocol description of the intervention was available, (2) whether the intervention was theoretically well-founded, (3) whether peer-reviewed articles had been published on the
354
K.E. Evenboer et al. / Children and Youth Services Review 61 (2016) 353–358
intervention, and (4) whether research had been done on the intervention (in those cases where no peer-reviewed articles had been published) (cf. Veerman & Van Yperen, 2008). Interventions were included in the EYI if they met at least the first two criteria. The results of this analysis showed that this method enabled us to identify similarities and differences in the content of these interventions, and to group together interventions with a highly similar profile of activities (Authors' own, 2014b), leading to a reduction (52%) in the number of distinct interventions. The application is further detailed in the Methods section of this paper. In the current study, this specific assessment procedure was extended to the group of “well-defined” interventions, that is, those interventions included in the EYI database. Our aim was to assess: (1) to what extent “well-defined” interventions had similar content, that is, could be grouped together, and (2) whether the proportion of interventions that could be grouped was smaller for “well-defined” than for ‘poorly defined’ interventions. Based on the studies of Hibbs (2001) and Van der Linden and De Graaf (2010) we expect that the proportion of interventions that could be grouped is larger for the “poorly defined” interventions than for the “well-defined” interventions. The latter group of interventions is included in the EYI database based on sufficing the criteria for accreditation regarding having been documented and having been studied. We therefore expected that these interventions have been defined more specifically regarding their contents, resulting in less overlap with other interventions. 2. Method 2.1. Sample Four care organizations in primary health care (PHC, offering N = 7 interventions), child and youth care (CYC, N = 42), and mental health care (MHC-A, N = 31; MHC-B, N = 11) participated in the context of the Collaborative Center on Care for Children and Youth (C4Youth). These organizations provide most of the psychosocial care for children, adolescents, and their families in a catchment area in the northern part of the Netherlands. The manuals concerning the interventions offered by these four organizations (in total N = 91) were therefore used to obtain more detailed information about the content of the care. There is no potential conflict of interest with respect to the research, authorship, and/or publication of this article. 2.2. Procedure We first assessed whether the interventions that came up were conceptually and empirically well-founded by using four criteria that had been formulated and applied by an independent committee of national experts evaluating interventions in the context of accreditation for the EYI database (Zwikker et al., 2009). The four criteria used were, (1) whether a protocol description of the intervention was available, (2) whether the intervention was theoretically well-founded, (3) whether peer-reviewed articles had been published on the intervention, and (4) whether research had been done on the intervention (in those cases where there were no peer-reviewed articles published) (cf. Veerman & Van Yperen, 2008). Interventions are included in the EYI if they meet at least the first two criteria. Of the 91 interventions, 35 were “well-defined” interventions that were part of the EYI database; the other 56 interventions were labeled as “poorly defined” and were not part of the EYI database (Authors' own, 2014a). In the current study, we will focus on these “well-defined” interventions – such as, for instance, “Families First” and “Triple P” – and compare the findings on these with findings on “poorly defined” ones — such as, for instance, “parent counseling” and “individual support.” We then went on to categorize the 35 “well-defined” interventions by main type of support, a term indicating the most important activities carried out to improve the functioning and development of children,
adolescents, and their families. Categorization was made based on the names of the interventions and the treatment manuals available. Regarding these latter, the terminologies and descriptions of the treatment manuals were the leading indicators. Next, we collected descriptors for each main type of support using all the interventions included in the EYI database (Netherlands Youth Institute, 2013). An example of one descriptor found is the following expression: “regulate emotions” (which was, for reasons of standardization, reformulated as “prompting client to regulate emotions”). All manuals of the interventions from this database were analyzed in order to achieve a good representation of the types of support offered within the four care organizations participating in our study. The 20 most frequently used descriptors per main type of support in these intervention manuals were collected, resulting in predefined lists for each main type of support (Authors' own, 2014a). Subsequently, the list of standardized descriptors for categorizing the various types of “well-defined” interventions offered to children with behavioral and emotional problems was applied. Professionals working at the participating care organizations scored the intervention descriptors using a seven-point Likert scale ranging from (1) “very poor” to (7) “very good.” “Very poor” meant that a descriptor was very inaccurate, while “very good” meant that the descriptor fully represented (one of) the activities to be carried out. A set of 20 descriptors was found to be a feasible amount for categorizing the content of the interventions (cf. Abraham & Michie, 2008). For scoring the activities found in all the interventions carried out by the PHC, CYC, and MHCs, we randomly selected two professionals experienced in that type of care. We deliberately chose to use two experienced professionals, because the rating was done based on activities carried out in the course of daily practice and was not based on scoring the intervention manuals. To control for the amount of bias in the scoring, we asked two professional to rate the same intervention. The background of these professionals varied from psychiatry and psychology to behavioral and family counseling. Prior to scoring the interventions, professionals received instructions in how to fill in the score forms. 2.3. Analysis The interventions were compared to others within the same main type of support category in terms of similarities and differences in content. First, for each intervention we computed mean scores per descriptor, based on the scores of the two raters, and compared these means to the overall mean scores per descriptor for that main type of support. Next, when deciding on how to group the interventions, 60% of these mean scores per descriptor were allowed to differ, up to a maximum of 0.5 points, from the overall mean score per descriptor for that main type. Interventions that did not meet both criteria – and thus were not grouped with other interventions – were then compared pairwise using the mean scores per descriptor for those two interventions, instead of using the overall mean scores per descriptor for that main type. Subsequently, we compared our findings for “well-defined” interventions with previously reported findings for “poorly defined” interventions (Authors' own, 2014b). 3. Results In total 34 “well-defined” interventions were analyzed, derived from primary health care (N = 6), child and youth care (N = 13), and mental health care (MHC-A, N = 10; MHC-B, N = 5). One intervention of MHCA was not assessed, because it was no longer provided. Table 1 shows the number of interventions before and after the four steps, as well as examples of descriptors per main type of support. Included are also data on “poorly defined” interventions from our former study (see fifth column). As a result, the original 34 “well-defined” interventions offered by the four care organizations participating in the C4Youth study could be grouped into 19 distinct interventions.
K.E. Evenboer et al. / Children and Youth Services Review 61 (2016) 353–358
355
Table 1 Number of well-defined interventions for each main type of support, before and after grouping, and comparison of percentages of reduction with poorly defined interventions. Main type of support
Well-defined
Reduction (%)
Poorly defined reduction (%)
Examples of descriptors per main type
N=0
N.A.a
50%
N=3
N=3
0.0%
33.3%
Experiential learning support
N=1
N=1
0.0%
50.0%
Individual child support
N = 20
N=9
55.0%
50.0%
Independent living support
N=1
N=1
0.0%
33.3%
Parenting support
N=3
N=2
33.3%
43.7%
Family support
N=6
N=3
50.0%
63.6%
– Providing information about the situation – Recognizing, identifying, and acknowledging feelings – Addressing feelings of guilt – Teaching client how to set rules – Stimulating client to relive an event – Prompting client to express emotions – Showing client how to deal with unfamiliar situations – Facilitating positive experiences – Teaching client how to deal with setbacks and frustrations – Instruction in recognitive structuring – Stimulation motivation – Prompting client to express emotions – Training in self-regulation – Analyzing the client's social environment – Providing positive reinforcement – Showing clients how to use positive reinforcement – Stimulation of interaction – Teaching clients to use disciplinary rules – Stimulating interaction – Giving behavioral instructions – Prompting clients to practice behavioral instructions
Total
N = 34
N = 19
44.0%
52.0%
N = (before)
N = (after)
Foster care support
N=0
Trauma support
a
N.A. = Not Applicable for this main type of support.
The sample of 56 “poorly defined” interventions was able to be grouped into 27 distinct interventions. The interventions that could be grouped were those especially found within the “family support” and “parenting support” categories. The reduction in number of distinct interventions after grouping was somewhat smaller for “well-defined” interventions (44%) than for “poorly defined” interventions (52%). The largest reduction was obtained in the category “individual child support” of the ‘well-defined’ interventions, while in the group of ‘poorly defined’ interventions this concerned the category “family support”. The results further revealed that not only differently labeled interventions provided by the same care organization, but also interventions provided by different organizations, sometimes had comparable content. In Fig. 2, we provide an example of the main type “individual
Fig. 1. Overview of the procedure in terms of the four steps used to identify distinct interventions.
child support.” This example shows that two CYC, four MHC-A, and four MHC-B interventions could be considered similar in terms of their content, despite their different labels. However, several CYC and MHC-A interventions in “individual child support” were found to be distinct interventions that could not be grouped; these interventions were thus not rated, in terms of their descriptors, as being highly similar to other interventions within that main type of support. “Trauma support” contained three interventions that had to be considered as distinct types of interventions (see Fig. 1). 4. Discussion The aim of the study was to assess (1) to what extent “well-defined” interventions have similar content, that is, could be grouped together, and (2) whether the proportion of interventions that could be grouped together was smaller for “well-defined” than for “poorly defined” interventions. The results showed that the “well-defined” interventions included in a national database had substantially overlapping content. The number of interventions could be grouped into 19 distinct types of interventions across the participating care organizations. The total amount of the grouping of “well-defined” interventions (44%) was somewhat smaller than the total amount of the grouping of “poorly defined” interventions (52%). The relatively large proportion of “well-defined” interventions that could be grouped is in contrast with propositions made by Van der Linden and De Graaf (2010), and Hibbs (2001), concerning the “evidence-based practice revolution.” These propositions implied that the “well-defined” interventions that were included in the EYI database met stricter criteria, since used in the context of accreditation. On this basis, these authors assumed that these interventions included a more detailed in the description of their content and thus were more differentiated. But despite the fact that these “well-defined” interventions were theoretically well-founded and research had been done on them, they did not appear to be as unique as assumed. A considerable reduction could be attained, especially for the main type “individual child support.” For the group of “poorly defined” interventions, the reduction within the main category “family support” was the largest (63.6%). This is in line with outcomes published by Loeffen, Ooms, and Wijgergangs (2004), and Veerman, Janssens, and Delicat (2005); in the Dutch sphere of practice, the latter identified more than 90
356
K.E. Evenboer et al. / Children and Youth Services Review 61 (2016) 353–358
Fig. 2. Number of distinct interventions within the six main types of support. *Experiential learning support: Interventions aimed at supporting youth by actively engaging in activities within the context of treatment. **Independent living support: Interventions that prepare and stimulate adolescents to independently organize their own housing and lives.
differently labeled “family preservation programs,” indicating quite a bit of overlap in terms of methods and target groups. A substantial proportion of the interventions had highly similar content, especially – as mentioned before – within the main types “individual child support” and “family support.” Interventions that were grouped within the main type “individual child support” involved, for example, “Emotion Regulation Therapy (ERT),” “Aggression Regulation Therapy (ART),” and “Social Skills Training (SST).” Although these interventions focused on different psychosocial problems, their “ingredients” were rather similar. This implies that we could combine these interventions into one group with similar content. Nevertheless, the application
of the ingredients in daily practice may differ in terms of target group and aim of the intervention. The outcomes of this study showed that despite the varying terminologies that were used to describe the care offered, the content of a substantial proportion of the interventions was rather similar. This was true not only for interventions offered by different care organizations but also by the same care organization. It demonstrates the importance of using standardized descriptors for characterizing the content of care. Use of descriptors to characterize the content of interventions is relatively new in the fields of primary health care, child and youth care, as
K.E. Evenboer et al. / Children and Youth Services Review 61 (2016) 353–358
357
well as mental health care. Abraham and Michie (2008), and Michie et al. (2011) previously used sets of “behavior change techniques” to characterize interventions aimed at physical activity, healthy eating, or smoking cessation. Another example of classifying interventions is the Distillation and Matching Model (DMM) proposed by Chorpita and Daleiden (2009). The DMM classifies interventions based on their content, target group, and specific type of problem behavior to be tackled. In this way, evidence-based “practice elements” across interventions can be classified in a sophisticated way. Both studies use protocol descriptions as the source of information for scoring these practice elements. In contrast, our method focuses on characterizing interventions by asking professionals to score the applicability of the descriptors. The number of interventions available for children and adolescents with psychosocial problems has increased enormously over the past few years. Only for a small percentage of these interventions, however, there is evidence available for their effectiveness. Therefore, more research on the techniques and activities carried out during treatment is needed in order to make statements as to which parts of interventions are effective, and for whom. Studies by Abraham and Michie (2008); Michie et al. (2011), and Chorpita and Daleiden (2009) also showed that gathering more knowledge on the content of interventions is of importance for monitoring the care that is being provided, and to link this to the outcomes in the short and longer term.
choosing the appropriate care. In the longer term, this could enhance the number of positive outcomes for children, adolescents, and their families with behavioral and emotional problems (Maschi, Hatcher, Schwalbe, & Rosato, 2008; Miller & Rowe, 2009; Ten Brink, Veerman, De Kemp, & Berger, 2004), and may result in an optimizing of the care process.
4.1. Strengths and limitations
References
One strength of this study is that the descriptors used to classify the content of the interventions were defined in such a way that they were independent of the organization-specific labels used by the four care organizations themselves. These descriptors provided a standard that permitted greater insight into the content of the care offered. Another strength is that we were able to compare the content of the interventions used by different types of care organizations in the field of primary health care, child and youth care, and mental health care. In addition, since the interventions were characterized by two professionals, the likelihood of bias during the assessment of the interventions decreased. One limitation of the study might be that, despite using the EYI database's thesaurus, we may have overlooked some important descriptors for the care offered: Only the 20 most frequently used descriptors were selected. Nevertheless, 20 is a relatively large number of descriptors when characterizing an intervention. Moreover, the scores for the descriptors given by the professionals were generally rather high, suggesting that they were good representations of the activities and operations carried out as part of an intervention. 4.2. Implications We found that some “poorly defined” and some “well-defined” interventions may have rather similar content, which raises the question of overlap between these two categories. An interesting subsequent step might be to further involve professionals in the identification of relevant descriptors, by asking them not only which descriptors characterize interventions most accurately, but also which descriptors are the most important ones for achieving positive changes in the target group. Their responses could well provide support and enhance research on the effective components of interventions (Barth, 2014). One implication for practice is that the interventions available for treating children, youth, and their families within care organizations could then be grouped per main type of support. Grouping interventions that have highly similar content is a good starting point for doing more research. Which content works for whom, and why does the intervention work? More knowledge about the content of interventions could also provide a more transparent and systematic overview, and this might lead to better communication between professionals within care organizations as well as across organizations. In addition, this might actually serve to support clients and their caregivers in
4.3. Conclusion We conclude that interventions included in a national database have substantially overlapping content. Detecting this may serve to support further improvement in the array of treatment offered to children with behavioral and emotional problems. Acknowledgments This study has received grants from the Netherlands Organization for Health Research and Development (ZonMw), the province of Groningen, the University Medical Center Groningen, the University of Groningen, health insurance company Menzis, and youth care providers Accare and Elker.
Abraham, C., & Michie, S. (2008). A taxonomy of behavior change techniques used in interventions. Health Psychology, 27(3), 379–387. Authors' own, 2014a. Authors' own, 2014b. Ballinger, C., Asburn, A., Low, J., & Roderick, P. (1999). Unpacking the black-box of therapy. A pilot study to describe occupational therapy and physiotherapy interventions for people with stroke. Clinical Rehabilitation, 13, 301–309. Barth, R.P. (2014, September). Common Components of Evidence Based Practices. Keynote address at the 13th Biennial International EUSARF Conference, September 4, Copenhagen, Denmark. Retrieved from: http://www.sfi.dk/keynote_speakers-11620.aspx Chorpita, B.F., & Daleiden, E.L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566–579. Cjaza, S.J., Schulz, R., Lee, C.C., & Belle, S.H. (2003). A methodology for describing and decomposing complex psychosocial and behavioral interventions. Psychology and Aging, 18(3), 385–395. Ezell, M., Spath, R., Zeira, A., Canali, C., Fernandez, E., Thoburn, J., & Vecchiato, T. (2011). An international classification system for child welfare programs. Children and Youth Services Review, 33, 1847–1854. Garland, A.F., Brookman-Frazee, L., Hurlburt, M.S., Accurso, E.C., Zoffness, R.J., HaineSchlagel, R., & Ganger, W. (2010). Mental health care for children with disruptive behavior problems: A view inside therapists' offices. Psychiatric Services, 61(8), 788–795. Harden, B.J., & Klein, S. (2011). Infants/toddlers in child welfare: What have we learned and where do we go from here? Children and Youth Services Review, 33, 1464–1468. Hibbs, E.D. (2001). Evaluating empirically based psychotherapy research for children and adolescents. European Child and Adolescent Psychiatry, 10, 3–11. Lee, B.R., & Barth, R.P. (2011). Defining group care programs: An index of reporting standards. Child and Youth Care Forum, 40, 253–266. Lloyd-Evans, B., Johnson, S., & Slade, M. (2007). Assessing the contents of mental health services: A review of measures. Social Psychiatry and Psychiatric Epidemiology, 42, 673–682. Loeffen, M., Ooms, H., & Wijgergangs, H. (2004). An organizational model is a must for effective care of children and youth. Nederlands Tijdschrift voor Jeugdzorg, 7, 102–113 (in Dutch). Marsh, J.C., Angell, B., Andrews, C.M., & Curry, A. (2012). Client-provider relationship and treatment outcome: A systematic review of substance abuse, child welfare, and mental health services research. Journal of the Society for Social Work and Research, 3(4), 233–267. Maschi, T., Hatcher, S.S., Schwalbe, C.S., & Rosato, N.S. (2008). Mapping the social service pathways of youth to and through the juvenile justice system: A comprehensive review. Children and Youth Services Review, 30, 1376–1385. Michie, S., Hyder, N., Walia, A., & West, R. (2011). Development of a taxonomy of behavior change techniques used in individual behavior support for smoking cessation. Addictive Behaviors, 36, 315–319. Miller, J., & Rowe, W. (2009). Cracking the black box: What makes an arts intervention program work? Best Practices in Mental Health: An International Journal, 5(1), 52–64. Netherlands Youth Institute (2013). Dutch Youth Institute, retrieved 14-08-2013 from https://www.nji.nl. Ten Brink, L.T., Veerman, J.W., De Kemp, R.A.T., & Berger, M.A. (2004). Implemented as intended? Recording family worker activities in a Families First program. Child Welfare, 83, 197–215.
358
K.E. Evenboer et al. / Children and Youth Services Review 61 (2016) 353–358
Van der Linden, D., & De Graaf, I. (2010). State of the art. Acquaintance with and use of evidence-based interventions in child and youth mental health care services. Utrecht: Trimbos Institute. Van Yperen, T.A., Van Rest, E., & Vermunt, C. (1999). Programmes in child and youth care: The definition of core concepts. Utrecht: NIZW Publishers (in Dutch). Veerman, J.W., & Van Yperen, T. (2008). A perspective on effectiveness. Handbook on practice-driven outcome research in care for youth. Delft: Eburon (in Dutch). Veerman, J.W., Janssens, J.M.A.M., & Delicat, J.W. (2005). Effectiveness of intensive homebased child and family care: A meta-analysis. Pedagogiek, 25(3), 176–196 (in Dutch).
Yohalem, N., & Wilson-Ahlstrom, A. (2010). Inside the black box: Assessing and improving quality in youth programs. American Journal of Community Psychology, 45, 350–357. Zwikker, M., Van Dale, D., & Kuunders, M. (2009). Committee accreditation of interventions: Method and procedure. Utrecht, The Netherlands: Netherlands Youth Institute/National Institute for Public Health and Environment (in Dutch).
