Implementation of Goal Attainment Scaling in ... - Wiley Online Library

1 downloads 0 Views 192KB Size Report
Jun 2, 2006 - Correspondence: Mark Burton, Head of Service, Manchester Learning ... Melanie Chapman*, Mark Burton*, Victoria Hunt*, and David Reeves†.
Blackwell Publishing IncMalden, USAPPIPolicy and Practice in Intellectual Disabilities1741-11222006 International Association for the Scientific Study of Intellectual Disabilities and Blackwell Publishing Ltd? 200632119128Original ArticleGoal Attainment Scaling and Community Intellectual Disability ServicesM. Chapman et al.

Journal of Policy and Practice in Intellectual Disabilities Volume 3 Number 2 pp 119–128 June 2006

Implementation of Goal Attainment Scaling in Community Intellectual Disability Services Melanie Chapman*, Mark Burton*, Victoria Hunt*, and David Reeves† *Manchester Learning Disability Partnership; and †National Primary Care Research & Development Center, University of Manchester, Manchester, UK

Abstract The authors describe the evaluation of the implementation of an outcome measurement system (Goal Attainment Scaling – GAS) within the context of an interdisciplinary and interagency intellectual disability services setting. The GAS database allowed analysis of follow-up goals and indicated the extent of implementation, while a rater study evaluated the quality of goals. While staff were able to produce adequate goals and scales, fewer goals were set than anticipated, and the overall quality was not high. Although implementation resulted in a number of perceived benefits, various barriers to implementation were experienced. These hinged on staff perceptions of the value, ease of use, appropriateness, and soundness of the method. Widespread adoption of GAS in community intellectual disability teams is not supported by the findings of this study. The authors suggest that staff perceptions, ease of use, and the implementation process play a key role in the successful adoption of an outcome measurement system. They conclude that alternative ways of measuring individually oriented outcomes may be more useful. Keywords: community setting, Goal Attainment Scaling, outcome measurement

INTRODUCTION Health and social care organizations, policy, and professional guidance have emphasized the importance of outcome measurement and the monitoring of the quality of health and social care services (e.g., within the British context see Department of Health, 1997; 2001; NHS Executive, 1998; United Kingdom Central Council for Nursing, Midwifery & Health Visiting, 2001). An outcome focus can demonstrate the effectiveness and impact of existing and new services; improve accountability; enable users and practitioners to choose between alternative approaches; and help service providers improve services, make more effective use of available resources, and demonstrate effectiveness to commissioners (i.e., funders of services). Routine outcome measurement is particularly important when there is a lack of published information about the effectiveness of different approaches, as is often the case in intellectual disability practice. This paper describes an evaluation of the implementation of one outcome measurement system, Goal Attainment Scaling (GAS), as used in a British intellectual disability service. GAS

Received March 14, 2005; accepted April 25, 2005 Correspondence: Mark Burton, Head of Service, Manchester Learning Disability Partnership, Mauldeth House, Mauldeth Road West, Chorlton, Manchester M21 7RL, UK. Tel: +44 161958 4050; Fax: +44 161958 4149; E-mail: [email protected]

involves the service recipient (or his or her representative) agreeing to the most important and feasible goals with the service provider (Kiresuk & Sherman, 1968). The expected level of attainment, using a 5-point scale anchored at zero, is then constructed for each goal. This scale is used to score the outcome at a date specified in advance. Initially developed for use in North American community mental health centers, GAS has been used in a variety of service contexts, and a substantial literature has appeared (see Kiresuk, Smith, & Cardillo, 1994, for a review). It has been reported that GAS has particular virtues as a change measure, has adequate validity, and its reliability is at least as good as those of conventional rating scales (Cardillo, 1994; Cardillo & Smith, 1994a; 1994b). The pragmatic features of GAS also make it attractive for applications to community services. GAS uses a common metric for all types of goals, workers, and clients, anchoring the scale at the “expected level of attainment.” Cardillo (1994) reports that professional staff can identify likely levels of outcome, and so set the “expected level” anchor realistically across goals. Potential bias from the professional’s participation in scoring the outcome is reduced, as the criteria for outcome measurement are established in advance of the outcome being measured. The participation of the person with intellectual disabilities (or a key ally) in the goalsetting process should enhance goal relevance and commitment to the intervention. The measurement of outcomes through the goals deliberately set means that measurement has appropriate

© 2006 International Association for the Scientific Study of Intellectual Disabilities and Blackwell Publishing, Inc.

Journal of Policy and Practice in Intellectual Disabilities

Volume 3 Number 2 June 2006

M. Chapman et al. • Goal Attainment Scaling and Community Intellectual Disability Services

sensitivity to the intervention in contrast to broad-brush strategies that will inevitably measure aspects of the person’s circumstances or functioning that will be irrelevant to, or unlikely to respond to, the intervention. However, there have been limited applications of GAS in intellectual disability services, although there has been interest in and advocacy for its use (Burns & Tobell, 1997; Turnbull, 1998; Young & Chesson, 1997). Published studies have concluded that GAS is applicable to intellectual disability services, can be implemented at minimal expense in terms of staff time (Bailey & Simeonsson, 1988; Glover, Burns, & Stanley, 1994), and appears helpful in clarifying the purpose of therapy (Young, Harvais, Joy, & Chesson, 1997). However, these studies have taken place in residential or day service settings, or with goals restricted to behavioral change (Bailey & Simeonsson, 1988). Unpublished studies of GAS in the context of intellectual disability field services report difficulties in implementation, largely because of professional resistance or at least conflict with existing procedures and practices (Ball, 1996; Lemon & Burns, 1996). However, these studies indicated that staff could set realistic, observable, and attainable goals, and that service recipients valued involvement in goal setting. Ball (1996) concluded that the difficulties encountered could be minimized by adequate training, planning, and management. Although Ball found that measurable change was produced, and GAS was sensitive to small changes, the study was of limited scope, covering 30 clients with a mean of two goals each, set and monitored by eight staff. While Lemon and Burns (1996) reported that staff were able to set realistic, observable, and attainable goals, the diversity of goal types with such a small sample meant that quantitative analysis of the GAS scores was not attempted. Therefore, there has been surprisingly limited evaluation of the application of GAS in intellectual disability services generally and in intellectual disability field services specifically. Moreover, where GAS has been applied, it has been used with restricted goals that are broadly therapeutic in nature, whereas community intellectual disability teams typically work with a broader range of interventions involving a variety of types of outcome (e.g., behavior change, affective change, learning, status maintenance or change, social and physical environmental change, arrangement of other interventions, and services). In addition, there is a lack of research on the practitioner experience of using GAS. This study aimed to evaluate the implementation and usefulness of an outcome measurement system based on GAS within community intellectual disability teams, and how staff experienced the introduction and use of the system.

Health Service), and supports approximately 1,400 people with intellectual disabilities living in a large city (Manchester). At the time of this study, four community teams consolidated care management/social work, intellectual disability nursing, psychology, speech and language therapy, occupational therapy, physiotherapy, additional support workers, and domiciliary care under a single management structure. Each team served a catchment population of approximately 105,000 inhabitants, and had around 12 full-time equivalent staff, not including management, administration, and domiciliary care staff. Community intellectual disability team staff work with people in the family home and in service settings managed by the service or other provider organizations (e.g., day service provision, supported housing, and short-term support respite services).

Implementation of GAS The GAS system consists of the GAS form (reproduced as Figure 1), a manual and examples booklet for staff, and a database that allows entry of data for progress monitoring and analysis centrally. Figure 2 outlines the process of using GAS. In our study, the system was implemented by a project worker in cooperation with local managers. It was maintained by Author 3, who also had a research role in the project. Management ownership was considered “built-in” as a senior manager in the service (Author 2, who was then the head of Development and Clinical Services) initiated the project. Organizational ownership was encouraged through briefings and regular meetings with managers and senior members of each professional group. Adoption by staff was encouraged by regular updates included in the monthly service update for staff, clear messages about the nonnegotiability of the system, training sessions and materials, “GAS Clinics” and individual appointments, feedback on goals set, and use of “early adopters” to help other staff. The original GAS recording form was modified to include a rating of external factors affecting goal attainment and mode of delivery of intervention (a help/ hindrance scale).

EVALUATION METHODS The evaluation incorporated a number of methods. The quality of information provided by the GAS system was examined through statistical analysis and rating of the quality of goals, while implementation issues were examined through staff questionnaires and interviews, and by a journal kept by the researcher.

The Service Setting This project involved the implementation of GAS in the community teams of a British intellectual disability service. The service is a partnership between the local Social Services Department (local government) and community health services (National 120

Analysis of Data from the Outcome Measurement System A research database was established (using Microsoft ACCESS) to record the goals and scales, and to aggregate and

Journal of Policy and Practice in Intellectual Disabilities

Volume 3 Number 2 June 2006

M. Chapman et al. • Goal Attainment Scaling and Community Intellectual Disability Services

G.A.S. Form Name of Service User:

Goal:

Name of Person Setting the Goal: Occupation of Person Setting Goal: LEVELS OF ATTAINMENT

Date Goal Se t

Pre-G.A.S. Score ( Mark with a ‘X’)

Predicted Date for 6 Month Review

Ac tual Date of 6 Month Review

G.A.S. Score Date of G.A.S. Score at 6 Month Discharge at Discharge (if not Review (if not discharged at at 6 (Mark wit h a 6 Months) Months) ‘X’)

Much Le ss

Somewhat Less Expected Level of Attainment Somewhat More Much More

Mode of Delivery

Type of Goal

Help/Hindrance Score

Outc ome

The form should be completed as follows: 1.

Name of the Service User



The member who the goal is being set for.

2. 3. 4.

Name of Person Setting Goal Occ. of Person Setting Goal Goal

– – –

Member of the Joint Service. I.e. profession. The goal set.

5.

Date Goal Set



The date the goal is decided and written onto the form.

6.

PreG.A.S. Score



The level the service user is at before the work with them began.

7.

Pred. Date for 6 Month Review



8.

Actual Date of 6 Month Review



The date six months after the goal is set. The date when the 6 Month Review actually takes place.

9.

G.A.S. Score at 6 Month Review



10. Date of Discharge (if not at 6 Month Review)



11. G.A.S. Score at Discharge (if not at 6 Month Review)



The level the service user is at on the 6 Month Review. If the client is discharged either before or after the 6 Month Review, please write the date down in this column. The level the service user is at when discharged, if discharged before or after 6 Month Review.

NB. Numbers 10 & 11 (above) do not need to be completed if the service user is discharged at 6 Month Review. 12. Mode of Delivery 1) 2)

Direct Via Staff

3)

Via Family

4)

Via Referral to Another Service

– Please choose the option which best describes the way the goal will be achieved: (You, the goal setter, will do the work yourself) (Staff in a house/day centre, etc. have responsibility for work being completed) (The family of the service user are responsible for the the work being completed) (The outcome of the goal will now depend on the response of another service)

13.

Type of Goal 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12)

14.

Accommodation Arrange service Assessment Counseling Environmental Change Give or Obtain Information Health Related Maintenance of Function Mobility Skill Development Support Family/Carers Therapy and/or Behaviour Change Please rate whether external factors helped or hindered goal attainment, using the following guide: Considerable help from external factors Some help from external factors No significant effect from external factors Some hindrance from external factors Considerable hindrance from external factors

Help/Hindrance Score +2 +1 0 –1 –2

15.

Please choose the option which best describes the type of goal which has been set:

Outcome 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11)

= = = = =

Please choose the option which best describes the outcome of the goal at this time: Goal Completed Goal Continued Inappropriate Goal Reduction in Resources New Goal Written Service User Illness Service User Discharged Before Goal Completed Service User is Withdrawn Service User Withdraws Staff Illness Death of Service User

FIGURE 1 Goal Attainment Scaling (GAS) form. 121

Journal of Policy and Practice in Intellectual Disabilities

Volume 3 Number 2 June 2006

M. Chapman et al. • Goal Attainment Scaling and Community Intellectual Disability Services

TABLE 1 Examples of goals set

Referral to community intellectual disability team

To enable Lara to live in a residential environment which incorporates all her needs That Susan may attend chosen “drop in” with support every 4–6 weeks Gayle will be able to have a bath with support until her arm is out of the sling Derek will be provided with equipment that enables him to sit on the toilet safely and comfortably Assessment of independent living skills Yazid will attend the art group 2/3 times a month For Ellie to complete an occupational therapy skills assessment That Melanie has a new set of splints to meet her needs Jack will have increased access to leisure activities Mobeen will be able to step out of bath independently using a suitable handrail

Assessment

Complete GAS form(s): Service recipient (or their representative) and community team member agree on goals and expected level of achievement

After 6 months: Review situation

Goal achieved

Goal not achieved/ ongoing/inappropriate

New goal identified

Note: All names have been changed.

Questionnaires to Staff Complete GAS form, recording current position on leve l of achievement scale and any help/hindrance factors

End of input regarding goal(s)

Complete new GAS form(s)

FIGURE 2 The process of using Goal Attainment Scaling (GAS).

analyze this information. SPSS (SPSS, 2001) was used to conduct more detailed statistical analysis.

Three batches of questionnaires were used: Batch 1 (Month 12): Sent to all staff who had completed more than three goals. Batch 2 (Month 18): Sent to all staff who had completed more than three goals. Batch 3 (Month 20): Sent to all staff who had not submitted at least three goals. The questionnaires were anonymous and sought information about how staff viewed GAS, where staff first heard of the system, difficulties they experienced, and their suggestions for further training. Fixed response questions, open questions, and opinion rating scales were used, along with space for any further comments. Interviews with Staff

A Rating Study Exploring the Quality of Goals Some examples of goals are provided in Table 1. Three raters independently examined a selection of 60 goals, selected randomly and anonymously from those available. The raters involved were all connected with the project: the research assistant (Author 3), the project leader (Author 2), and the statistical advisor (Author 4). Goals were rated on clarity, specificity, relevance, practicality, and measurability. Scales were rated on whether they involved multiple goals, multidimensional scales, overlapping levels or gaps between levels, blank levels, achievability, and appropriateness of factor used to scale. For the rating study, the quality of goals was scored on a 5-point or 3-point rating scale; the quality of scales was scored on a “Yes/No” basis. 122

Two sets of interviews were carried out. Eight interviews were carried out relatively early in the project, and five further interviews at a later stage. The interview schedule explored opinions on GAS, training sessions, experiences of using GAS, benefits and drawbacks of the system, and suggestions for improvements to the implementation process. Interviews were taped and transcribed. Analysis consisted of “open coding” of interview transcripts by Authors 2 and 3 using a set of starter codes that were added to as required. Themes and patterns were then discussed and refined, paying particular attention to the different views expressed by participants, including “negative cases.” Matrices (Miles & Huberman, 1994) were used to summarize the information and to guide analysis of emergent

Journal of Policy and Practice in Intellectual Disabilities

Volume 3 Number 2 June 2006

M. Chapman et al. • Goal Attainment Scaling and Community Intellectual Disability Services

themes (Coffey & Atkinson, 1996). A thematic framework was then developed through familiarization, constant internal and external comparison (e.g., with literature and other project findings), and discussion with the research team. There was no attempt to make themes “representative” of the sample in a quantitative sense, but rather to reflect as full and complete as possible a picture of the issues perceived by participants (Strauss & Corbin, 1990). The two coders (Authors 2 and 3) made explicit their assumptions about the project in written statements prior to beginning the analysis. This was to increase reflexivity in terms of sensitivity to the “sense making” (Bannister, Burman, Parker, Taylor, & Tindall, 1994) by researchers who had specific interests in the project. Research Journal The researcher (Author 3) kept a journal throughout the project describing factual details concerning the everyday running of the project and reflexive analysis of the progress of the project. This was analyzed by identifying key themes, using a similar method to that outlined above.

FINDINGS This section describes the number of goals set during the research project to demonstrate the extent of implementation of GAS, and analysis of follow-up scores. Key issues related to implementation—identified from analysis of the questionnaires, interviews, and research journal—are then presented. Analysis of Data from the Outcome Measurement System The number of goals set provides an index of the extent of implementation and routinization of GAS in the work of the community teams. A total of 789 goals were set across the whole service over a 2-year period. As Figure 3 shows, there was a slow start, with all teams showing a similar initial pattern of slow setting of goals. The rate rose 1 year into the project, but tailed off again once the implementation phase drew to a close and the project devoted more attention to the evaluation. During the year 1998–99, 1,395 referrals to the relevant disciplines in the community teams were recorded. Of these, some 10% were classified as inappropriate or requiring no further

Cumulative goals set 800 700

Number of goals

600

Team

500

1 2

400

3 4 All

300 200 100

Ja nu Fe ary br 1 ua 99 ry 8 1 M ar 99 ch 8 1 Ap 99 ril 8 1 M 99 ay 8 Ju 199 ne 8 1 J u 998 Au ly 1 Se gu 99 pt st 8 em 1 b 99 O er 1 8 ct 99 N ob ov er 8 em 19 De be 98 ce r 1 m 99 b 8 Ja er 1 nu 99 Fe ary 8 br 1 ua 99 ry 9 1 M ar 99 ch 9 1 Ap 99 ril 9 1 M 99 ay 9 Ju 199 ne 9 1 J u 999 l A y Se ug 199 pt ust 9 em 1 b 99 O er 1 9 c 99 N to b ov er 9 e 1 D mbe 999 ec r em 19 be 99 r1 99 9

0

Tim e

FIGURE 3 Cumulative goals set by the four teams. 123

Journal of Policy and Practice in Intellectual Disabilities

Volume 3 Number 2 June 2006

M. Chapman et al. • Goal Attainment Scaling and Community Intellectual Disability Services

TABLE 2 Scores by discipline

Nurses Care managers Psychology Physiotherapy Speech and language therapy Occupational therapy All

Mean baseline scorea

Mean follow-up score

Mean change in score

−1.6 (0.6) −2.0 (0.2) −1.7 (0.6) −1.6 (0.6) −1.9 (0.3) −1.7 (0.6) −1.8 (0.5)

0.1 (1.4) 0.5 (1.0) 0.3 (1.1) 0.6 (1.4) 0.1 (1.1) 0.3 (1.3) 0.3 (1.3)

1.8 (1.4) 2.5 (1.0) 2.1 (1.0) 2.2 (1.6) 2.1 (1.1) 2.0 (1.3) 2.0 (1.3)

Note: Numbers in brackets are the standard deviation. a Means are on the scale of −2 (much less than expected) to +2 (much more than expected).

action. This would suggest a minimum expected number of 1,255 goals per year, if only one goal were set per referral; however, the project suggested that 2–3 goals would usually capture the main thrust of the work being done. Therefore, the number of goals set was far fewer than anticipated. Mean baseline and follow-up scores for 374 goals with both an initial and follow-up score are presented in Table 2, along with a breakdown of these scores by discipline. There were no statistically significant associations between the extent of change in GAS score reported and discipline (χ2 (5, n = 374) = 9.8, p > 0.05) or mode of delivery (χ2 (3, n = 372) = 1.8, p > 0.05). However, there were significant associations between change in GAS score and both help/hindrance (Spearman rank correlation coefficient: rs = 0.453, p < 0.01) and type of goal (χ2 (6, n = 372) = 20.1, p < 0.01). Smaller changes in GAS scores were associated with a greater hindrance factor, while higher changes in GAS scores were associated with a greater help factor. The data seem to indicate that smaller changes in GAS scores were associated with therapy/behavior change and skill development goals, whereas greater changes in GAS score were associated with goals around giving and obtaining information and environmental change.

Rating Study Findings Table 3 summarizes the results of the rating study relating to the quality of goals. Score means have been translated into a common scale ranging from 0 (lowest quality) to 10 (highest quality). These results are based on all three raters combined. Table 4 contains the average results (summed across all three raters) on the existence of multiple goals, and the quality of scales.

Staff Satisfaction with GAS Forty-four questionnaires were returned (a response rate of 38.3%). Table 5 gives details of satisfaction with and usefulness 124

TABLE 3 Quality of goals

Dimension

Clarity Specificity WHO Specificity WHAT Specificity WHEN Relevance Practicality Measurability Technical language

Average score out of 10

Acceptable to two or more raters (%)

6.9 7.5 6.0 4.0 7.0 9.0 8.0 8.5

92 100 93 40 100 100 98 75

TABLE 4 Multiple goals and quality of scales

Dimension

Multidimensional scale Other factors better used? Gaps between levels Multiple goals Overlapping levels Achievable? Blank levels

% Yes

Agreement between all ratersa(%)

47 42 23 16 15 11 7

61 47 75 73 85 79 97

a

That is, on 73% of goals, all three raters agreed that either there were or were not multiple goals implied in the goal statement.

of GAS as perceived by staff who returned questionnaires. Usefulness was scored on a 5-point scale (1 being “not at all,” and 5 being “very much so”), while satisfaction was scored on a 10-point scale (1 being “very happy,” and 10 being “very unhappy”).

Journal of Policy and Practice in Intellectual Disabilities

Volume 3 Number 2 June 2006

M. Chapman et al. • Goal Attainment Scaling and Community Intellectual Disability Services

Similar issues arose in interviews, the day-to-day talk recorded in the research journal, and the open sections of the questionnaires, providing a triangulated check on the plausibility of these themes. These shed light on potential reasons for the disappointing returns of GAS forms and the low levels of satisfaction among staff. Perceived benefits and drawbacks with GAS are summarized in Table 6. These broadly surround the implementation process, use of the information, value to clients and community intellectual disability team staff, appropriateness and methodological strength of GAS, and ease of use of the system. Of particular note with the current emphasis on personcentered approaches is that GAS was not being used in a dialogue with service recipients or families in the majority of cases. The

TABLE 5 Staff satisfaction with Goal Attainment Scaling and perceived usefulness

Batch 1 (>3 goals) Batch 2 (>3 goals) Batch 3 (