Development and Testing of a Scale Measuring Parent Satisfaction With Staff Interactions Janis E. Gerkensmeyer, PhD, RN, APRN,BC Joan K. Austin, DNS, RN, FAAN Abstract This study tested the psychometric properties of the Parent Satisfaction Scale (PSS), a new scale measuring parents' satisfaction with their child's mental health services, primarily focusing on staff's interpersonal interactions. Data were collected by postal surveys 5 to 10 months after a child's admission to 1 of 5 mental health treatment programs. Completed surveys were received from parents of 120 of a potential 232 children (52% response rate). Internal consistency reliability of the PSS was strong (or = .96). Construct validity was supported by significant relationships between the PSS and constructs considered to be antecedent variables influencing parent satisfaction, including met expectations (r = 0.62, P < .001), met desires (r = 0.32, P < .001), and met needs (r = 0.55, P < .001). Convergent validity was supported, with the PSS having Pearson correlations of .86 (P < .001) with the Client Satisfaction Questionnaire-8 and. 76 (P < .001) with parents' ratings of satisfaction with specific services received.
According to the Report of the Surgeon General's Conference on Children's Mental Health, our nation has been experiencing a crisis in children's mental healthcare, t An estimated 12% to 20% of children have mental health problems meriting treatment. 1-3 Fewer than a third of these children have received mental health services, and approximately half of those receiving services have received inappropriate services. 4-7 Unmet service needs for children have been reported to be as high now as they were 20 years ago. 1 In the past several years, new service delivery models have emerged, largely due to initiatives to contain cost and, at times, to improve services. 8-1° With concerns about inadequate services and the emergence of new care delivery models, there has been a pressing need to ensure that children with mental health problems and their families receive appropriate services.l' t0 Measurement of parent satisfaction has the potential to be an important component in evaluating service adequacy 1 because parents usually are (a) responsible for obtaining mental health services for their child, (b) key to the success of treatment through their participation, ll (c) the best source of information about the effects of caring for a child with mental health problems, 12 and (d) the primary caregivers after service completion. Research findings have consistently shown that the most
Address correspondenceto Janis E. Gerkensmeyer, PhD, RN, APRN,BC, Indiana University School of Nursing, 1111 Middle Dr, Room 403-G, Indianapolis, IN 46202. E-mail:
[email protected]. Joan K. Austin, DNS, RN, FAAN, is a distinguished professor at the Indiana University School of Nursing, Indianapolis, Ind.
Journal of Behavioral Health Services & Research, 2005, 32(1), 61-73. @ 2005 National Council for Community Behavioral Healthcare.
Scale for Measuring Parent Satisfaction With Staff Interactions
GERKENSMEYER, AUSTIN
61
important factor contributing to satisfaction in the healthcare context has been interpersonal relationships between staff and consumers. 13-16 Technical aspects of care, although an important determinant of clinical outcomes, have not been found to account for significant variance in consumer satisfaction. 17Focusing on interpersonal interactions of staff, therefore, served as the common denominator when developing items for the parent satisfaction scale described in this article.
Background A review of studies related to parent satisfaction with mental health services for their children from 1982 to 2001 resulted in 34 studies and 2 reviews of the literature. 18,19 An English-language search of MEDLINE, HealthSTAR, PsycINFO, CINAHL, Cochrane Collaboration, and WorldCat was used for the search, along with a hand search of the annual research proceedings from A System of Care for Children's Mental Health.
Methodological Issues In the two existing reviews, by McMahon and Forehand TMand by Young et all 9 and in the author's review 2° of the 34 parent satisfaction studies, problems identified were lack of psychometric support for many parent satisfaction instruments, as well as methodological variations across studies that limited the ability to compare findings across studies. 18-2° An example of such variations was the use of 28 different satisfaction instruments in the 34 studies from 1982 to 2001.2° The Parent Satisfaction Scale described in this article was one of these 28 instruments. Another methodological problem in parent satisfaction measurement has been the use of global items. Global measures only provide a general sense of the extent to which parents are satisfied. 21 With such general information, it has not been possible to identify what has contributed most to parent satisfaction and with which aspects of care parents were more or less satisfied. Having more specific information would provide a guide for efforts to improve parent satisfaction. On the other hand, when items within a parent satisfaction scale are too specific to a particular program or intervention, its usefulness is also limited. Satisfaction measures specific to a particular program or intervention preclude making comparisons in parent satisfaction across settings and programs or with different interventions or phases of treatment. For example, 3 studies were limited by using scale items that focused on specific attributes of initial assessment, 22 case managers, 23 or a particular approach to psychiatric inpatient c a r e . 24
Theoretical Issues The reviews by McMahon and Forehand TM and by Young et a119do not address the need for viable theoretical frameworks with well-constructed conceptual definitions to guide instrument development, methodology, and interpretation of findings. Furthermore, none of the 34 parent satisfaction studies reviewed by the authors had a conceptual framework, 2° and only 2 presented conceptual definitions of parent satisfactionY ,26 This lack of conceptual clarity led to something other than parent satisfaction being measured in some of the 34 studies. 27-36 For example, parents' ratings of improvement in their child's behavior was used as a measure of parent satisfaction in 1 study. 33 It is likely that parents could be satisfied with services without improvement in their child's behavior; for example, parents might be satisfied because of other benefits such as receiving support and information on how to cope with their child's behaviors. Another example was the faulty assumption that if parents made any criticisms or suggestions for improving services in response to an open-ended question, they were dissatisfied, and, if not, they were satisfied. 34 It is possible that parents making suggestions were generally satisfied with care and simply trying to be helpful.
62
The Journal of Behavioral Health Services & Research
32:1 January~March2005
From this review it is evident that scales are needed that (a) have validity and reliability, (b) contain items that provide specific information while at the same time are useable across settings and interventions, and (c) actually measure the construct of parent satisfaction.
Conceptual Framework This study, testing the psychometric properties of the Parent Satisfaction Scale (PSS), was part of a larger study testing a consumer satisfaction model. 2° The consumer satisfaction model was developed by the authors prior to this study in an effort to better define parent satisfaction, both conceptually and operationally, and to examine underlying determinants of parent satisfaction. The consumer satisfaction model, a discrepancy model, included consumers' perceptions about what was expected or valued as a baseline with which to compare their perceptions of services received. Satisfaction was determined by the difference between perceived actual services and consumers' perceptions of ideal, expected, or deserved services. 37 In the model, consumers' characteristics are proposed to influence how consumers define their situation of having a child with mental health problems. Consumers' characteristics and definition of their situation, in turn, are proposed to influence consumers' desired services, perceived care needs, and expectations. Desired services, perceived care needs, and expectations serve as consumers' psychological standards to compare with the care they experience. Comparing consumers' perceived expectations, needs, or desires with perceptions of care experienced is proposed to result in consumers' judgments about their level of met desires, met needs, and met expectations. Met desires, met needs, and met expectations are proposed to influence satisfaction. Parent satisfaction is defined as the degree to which a parent is pleased with particular aspects of the healthcare system, in this study related to staff interactions in the context of mental health services for children. The PSS, therefore, only measured the construct of parent satisfaction. Met needs, met desires, and met expectations are predicted to be significantly related to the PSS.
Purpose The purpose of this study was to test the psychometric properties of the author-developed Parent Satisfaction Scale (PSS), a scale to measure parents' satisfaction with their child's mental health services, primarily operationalized as staff's interpersonal interactions. Following a description of the development of the instrument, information is presented on the testing of its psychometric properties. Internal consistency reliability was explored using coefficient or. Unidimensionality of the PSS was also examined using principal components factor analysis with varimax rotation. Furthermore, the ability of the PSS to differentiate between programs and settings was examined using an analysis of variance with the post hoc Scheffe test. Validity of the PSS was examined using Pearson correlations. Construct validity was examined by exploring the relationships of the PSS with other variables proposed to have a positive relationship with parent satisfaction, including Met Expectations, Met Desires, and Met Needs. 2° Convergent validity was examined by exploring the relationship of the PSS with the Client Satisfaction Questionnaire-8, 38 a frequently used global measure of parent satisfaction for which there is evidence of reliability 24'3941 and validity.39 In addition, convergent validity was examined by exploring the relationship of the PSS with parents' satisfaction with specific services received.
Scale Development Item development The current PSS resulted from an iterative process of development and refinement. 42 The first step involved generating items from a review of summaries of consumer satisfaction research, 21,43-46
Scale for Measuring Parent Satisfaction With Staff Interactions
GERKENSMEYER,AUSTIN
63
specific consumer satisfaction research articles, 40'41'47-60 articles describing parent and staff interactions, 6v-63 and a group of 5 parent satisfaction scales. 64 These items were designed to measure parents' perceptions of parent and staff interactions but not the parents' satisfaction with those staff interactions. From these items, the Parent-Staff Interaction Scale (P-SIS) was developed, consisting of items related to parent satisfaction. Development and testing of the P-SIS has been discussed elsewhere. 65 The PSS was developed, based on items from the P-SIS, to measure parent satisfaction on 6 staffrelated, topical areas identified by the authors, including staff attitudes, availability, supportiveness, and helpfulness; inclusion of parents; and informing parents. A 19-item PSS was developed by creating items directly addressing parent satisfaction from each topical area, along with an overall statement of satisfaction. Each item was worded as "I was satisfied with [the particular aspect of care]" and each item was designed to be rated on a 5-point Likert-type scale from 1 (strongly disagree) to 5 (strongly agree).
Pilot Testing and Scale Revision In the first wave of pilot testing, leaders of several Indiana support groups for parents of children with mental health problems distributed the 19-item PSS, with 44 members completing it. The 19item PSS was highly correlated with the P-SIS (r = 0.97, P < .001). Seven items were deleted from the PSS that had interitem correlations above 0.70, indicating redundancy. In addition, the item-to-total correlation, the c¢ if deleted, judgment of conceptual desirability, variance, and mean were examined. To be retained, an item needed to have a correlation with the total scale as high as 0.60 or greater, the ot if deleted needed to decrease by .01 or more, and/or the variance needed to be broad enough to show it was sensitive in discriminating differences among individuals, el Following additional pilot testing, another item was deleted from the PSS, leading to the current 11-item scale. The final PSS is designed to provide information about specific aspects of care and to be used across studies and treatment settings. Aspects of the healthcare system addressed by the PSS are primarily related to staff's interpersonal interactions (Table 1).
Testing of Psychometric Properties The testing of the psychometric properties of the 11-item PSS was carried out in this study of parents whose children with mental health problems had been admitted to 1 of 5 treatment programs. Administrators from each of the programs described children and adolescents in their programs as having serious mental health problems. In addition, parents rated their child's mental health problem as serious (M = 4.12) on a scale from 5 (is very serious) to 1 (is not at all serious). 2° A brief description of the 5 participating programs follows. Program A was a "wraparound," community,based program that admitted children aged 5-18 who met the following criteria: (a) at risk for imminent placement for a long term in a psychiatric hospital or residential facility; (b) a resident of the county being served; (c) an impairment in 2 or more functional areas; and (d) a Diagnostic and Statistical Manual for Mental Disorders (DSM-IV) diagnosis. 66 Program B was a 42-bed, state-operated, inpatient program for children between the ages of 6 and 18. Admission criteria included a DSM-IV diagnosis. 66 Children with primarily a serious substance abuse problem or mental retardation were excluded from the program. Program C was affiliated with a not-for-profit hospital that provided a continuum of care for children less than 18 years old. Admission requirements included (a) a DSM-1V diagnosis66; (b) significant functional impairment in at least 1 of 4 areas; (c) duration of illness anticipated to be at least 12 months; and (d) families at 200% of the poverty level or below.
64
The Journal of Behavioral Health Services & Research
32:1 January~March2005
Table 1 Parent Satisfaction Scale items
Item Overall, I was satisfied with the staff. I was satisfied with the availability of the staff. I was satisfied with the way the staff helped me understand my child's problems. I was satisfied with the convenience of appointments with the staff. I was satisfied with the caring and concern the staff showed for my child. I was satisfied with how the staff treated me with respect. I was satisfied with how the staff listened to what I had to say. I was satisfied with how the staff kept me informed about changes in the care of my child. I was satisfied with how the staff helped me find the services my child needed. I was satisfied with how the staff included me in decision making about my child's treatment. I was satisfied with the support I received from the staff.
Mean
Standard deviation
Factor loading
Corrected item-total correlations
4.07 3.91 3.63
1.08 1.11 1.30
0.88 0.83 0.84
0.85 0.79 0.80
4.02
1.10
0.75
0.70
4.21
1.07
0.85
0.81
4.18
1.15
0.88
0.85
4.13
1.12
0.84
0.80
3.80
1.33
0.86
0.83
3.81
1.23
0.85
0.82
3.89
1.19
0.87
0.84
3.98
1.15
0.92
0.90
Program D served youth, aged 12 to 17, in 6 counties, providing an intensive psychotherapeutic, home-based intervention designed to offer an alternative to out-of-home placement, provide transitional support for youth discharged from juvenile detention, and decrease the rate of youth recidivism in the juvenile justice system. Eligibility requirements varied contractually in each of the counties; however, the primary criterion was the youth's involvement in the juvenile justice system. A DSM-IV diagnosis 66 was not required. Program E was an 18-bed, state-operated, inpatient program for adolescent boys between the ages of 13 and 17 with conduct disorder, oppositional disorder, or explosive disorder. 66
Methods Participants Participating parents were part of a larger study testing a consumer satisfaction model that proposed particular psychosocial determinants of parent satisfaction. 2° Both mothers and fathers of children admitted to 1 of the 5 treatment programs were invited to participate. A child's primary caregiver, who had cared for the child for 5 months or longer if the child was in residential care or for 3 months or longer if the child lived in the respondent's household, could complete the instruments in place of the child's parent. Data were collected from parents 5 to 10 months after their child's admission to a program to ensure adequate exposure to the program. After obtaining Institutional Review Board approval, surveys were sent to parents or caregivers of 278 children or adolescents, with attempted follow-up phone contacts for nonrespondents.
Scale for Measuring Parent Satisfaction With Staff Interactions
GERKENSMEYER, AUSTIN
65
Nonresponding parents of 46 children could not be contacted by phone owing to their not having a phone. This resulted in 232 family units invited to participate in the study by mail with a follow-up contact by phone if they had not returned a completed survey. Completed surveys were returned by at least 1 parent of 120 of these children and adolescents, for a 52% response rate. An additional 16 spouses also completed surveys. One parent was randomly selected from each of the 16 pairs of spouses to be included in data analysis. Each participant received $10.00 upon receipt of a completed survey.
Characteristics of parents The majority of participants were Caucasian (72.5 %) and female (90%). Most respondents were biological mothers (74.2%), with 6.7% biological fathers, 6.7% grandmothers, 4.2% adoptive mothers, 3.3% foster mothers, 1.7% stepmothers, 0.8% foster fathers, 0.8% grandfathers, and 0.7% adoptive fathers. Respondents' ages ranged between 21 and 71, with an average age of 40. Most were married (40%) or divorced (27.5 %), with the remainder single (13 %), separated (10.8%), living with a partner (5.8%), or widowed (1.7%). The average total annual household income fell between $20,000 and $29,999.
Description of children and adolescents Children's ages ranged from 3 to 18, with an average age of 12.8. The average grade was seventh. The majority of children were Caucasian (70%), with 21.7% African American, 7.5% biracial, and 0.8% Asian. Consistent with previous research on children with serious mental health problems, the vast majority (69.2%) were males. 67 Most children (60.8%) lived at home at the time the survey was completed. The remaining children lived in a residential treatment facility (10%), an inpatient psychiatric unit (10%), foster care (5.8%), a group home (4.2%), with a relative (3.3%), or in another setting such as a nursing home or independent living situation (5.8%). Parents reported an average length of their child's mental health problem of 7 years. Parents rated the seriousness of their child's problem on average as 4.12 on a 5-point scale ranging from 5 (very serious) to 1 (not at all serious). There was no significant difference among program sites on this variable.
Instruments for related concepts
Met Desires Met desires was defined for this study as the degree and type of congruence between parents' self-reported desired care and their perceptions of actual care experienced. It was proposed to directly influence parent satisfaction. To measure desired care, parents completed the Mental Health Services Scale (MHSS), a scale that lists 42 services for children with mental health problems and their families. For scoring desired care, parents' responses were weighted with 2 points (a lot), 1 point (a little), and 0 points (not at all and don't know). Parents also checked if they had received each service. If a parent indicated he or she desired a service and had received it, that represented a met desire. If a parent reported he or she desired a service and did not receive it, that represented an unmet desire. Parents' Met Desires was measured by subtracting the sum of unmet desires from the sum of met desires, thus taking into account not only the contribution of met desires, but also unmet desires.
Met Expectations Met expectations, defined as the degree of congruence between expectations for care and perceptions of actual care experienced, was proposed to directly influence parent satisfaction. Parents' Met Expectations was measured by a combination of 2 items ("I expected to be treated well by staff and the staff treated me the way I expected them to."). The combination of these items ranged from service
66
The Journal of Behavioral Health Services & Research
32:1 January/March 2005
anticipated to be extremely negative but actually being experienced as being extremely positive (12 points) to services anticipated to be extremely positive but actually being experienced as extremely negative (0 points). The highest Met Expectations score would result when parents responded that they strongly disagreed that they expected to be treated well, and that they strongly disagreed that they were treated the way they expected to be. In this case, parents' negative expectations would have been greatly surpassed.
Met Needs Met needs, defined as the degree of congruence between self-reported needs and perceptions of actual care experienced, was also proposed to directly influence parent satisfaction. Parents' Met Needs was measured by ratings on the MHSS. Parents responded to the question, "How well were your needs met?", for each of the 42 listed services by checking met, sometimes met, not met, or don't need. Met Needs was a mean score determined by the sum of the weighted points, 2 points for each time met was reported and 1 point for each time sometimes met was reported, divided by the number of points met and sometimes met was checked.
Client Satisfaction Questionnaire-8 The Client Satisfaction Questionnaire-8 (CSQ-8), 39 a global measure of parent satisfaction in the healthcare context, is a shortened version of the 31-item Client Satisfaction Questionnaire. 38 It was originally standardized on adult patients and was later used in parent satisfaction studies. 68,69 Stevenson and Srebnik69 reported that internal consistency reliability was good (o~ = .93). In this study, coefficient ot for the CSQ-8 was .96. Satisfaction with Services Received
Satisfaction with Services Received was rated on the MHSS for each of the services that parents reported they received. Satisfaction with Services Received was scored on a 4-point Likert-type scale ranging from 1 (not at all) to 4 (very).
Results Relationship of demographic variables to the PSS No significant relationships were found between demographic variables and the total mean score of the PSS, with 1 exception: parents of children living at home when the survey was completed reported significantly higher levels of satisfaction than those with children living out of home (r = 0.32, P < .01). Other demographic variables examined included child's gender, race, age, grade in school, length of stay in the treatment program, length and severity of mental health problem; and parents' gender, race, age, highest year completed in school, employment status, household income, marital status, and relationship to child.
Factor analysis and internal consistency reliability Factor analysis found that the 11-item PSS was a unidimensional scale with factor loadings ranging from 0.75 to 0.92 (N = 120, see Table 1). The standardized a coefficient for the PSS in this study was .96. Corrected item-to-total correlations ranged from 0.70 to 0.90 (Table 2). Standard deviations for each of the 11 PSS items were all greater than 1, indicating adequate variability (see Table 1). The preceding reliability analysis was rerun on subgroups of subjects stratified by major demographic groupings including child's gender, child's age, child living at home versus out of home, length of child's problems, program, parents' race, parent's level of education, and household
Scale for Measuring Parent Satisfaction With Staff Interactions
GERKENSMEYER, AUSTIN
67
Table 2 Correlations among study variables
1. 2. 3. 4. 5. 6.
Parent Satisfaction Scale Met Expectations Met Desires Met Needs Satisfaction with Services CSQ-8
1
2
3
4
... 0.62* 0.32* 0.55* 0.76* 0.86*
... 0.04 0.34* 0.38* 0.45*
... 0.32* 0.33* 0.33*
0.58* 0.60*
5
6
° . o
... 0.86*
*P < .001.
income; and the pattern of reliability was similar, ranging from .90 to .98, with 18 out of 19 of the subgroups having an c~ of .93 or more.
Differences in parent satisfaction between programs As can be seen in Table 3, there were significant differences among the 5 programs for 6 of the 11 PSS items. Parents from Program A, the wraparound program, and Program C, the program with a continuum of services, reported higher satisfaction with how staff (a) listened to what parents had to say, (b) kept parents informed about changes in their child's care, (c) included parents in making decisions about their child's care, and (d) helped parents find needed services for their child than did parents with children being served in the 2 state, tertiary inpatient programs. This may have been confounded with or related to the finding that parents of children with mental health problems who were living at home when they completed their survey reported significantly higher levels of satisfaction than those with children living out of home (P < .01). A general linear, univariate model was used to examine the differences between program means on the PSS items while controlling for the covariance of the child's living situation (in-home versus outof-home) at the time of survey completion. The patterns of differences between specific programs remained the same, with some changes in levels of significance. All 5 items on Table 3 that had nonsignificant differences remained nonsignificant (Items 2, 3, 4, 6, and 11). A decrease from P < .01 to P < .05 was seen in 2 items (Items 1 and 5), while in item 7 the decrease was from P < .05 to P becoming nonsignificant. Two items (Items 8 and 10) remained the same (P < .05). One item's level of significance decreased from P < .0l to P becoming nonsignificant. (Item 10).
Construct validity of the PSS Construct validity of the PSS was supported by the direction and significance of the correlation of the PSS to proposed antecedent variables: Met Expectations, Met Desires, and Met Needs (Table 2, P < .001). The preceding analysis was rerun on subgroups of subjects including child's gender, child's age, child living at home versus out of home, and length of child's problems. The PSS remained significantly related to proposed antecedent variables except for a nonsignificant relationship between the PS S and Met Desires when the child was female (r = 0.19, P -----NS ) versus male (r-----0.38, P < .01).
Convergent validity Convergent validity was strongly supported by the positive relationship of the PSS with the CSQ-8 (r = 0.86, P < .001). Items in the PSS, however, provided more specific, and therefore more clinically
68
The Journal of Behavioral Health Services & Research
32:1
January~March 2005
Table 3 Differences b y p r o g r a m o n Parent Satisfaction Scale items* Mean Item 1. Overall, I was satisfied with the staff.+ 2. I was satisfied with the availability of the staff. 3. I was satisfied with the way the staff helped me understand my child's problems. 4. I was satisfied with the convenience of appointments with the staff. 5. I was satisfied with the caring and concern the staff showed for my child, t 6. I was satisfied with how the staff treated me with respect. 7. I was satisfied with how the staff listened to what I had to say.~ 8. I was satisfied with how the staff kept me informed about changes in the care of my child.~ 9. I was satisfied with how the staff helped me find the services my child needed.~ 10. I was satisfied with how the staff included me in decision making about my child's treatment, t 11. I was satisfied with the support I received from the staff. Total PSS Score~
Program A Program B Program C Program D Program E (n ---- 31) (n = 23) (n : 18) (n = 4) (n : 44) 4.14 4.09
3.68 c 3.58
4.70 bd 3.96
3.67 e 3.89
4.50 4.25
3.75
3.29
3.87
3.72
3.25
4.14
3.74
4.30
3.89
3.75
4.30 b
3.74:
4.83 b
3.94
4.50
4.11
3,87
4.78
4.06
4.25
4,23
3,74 c
4.70 b
3.83
4.00
4.07
3.29
4.26
3.50
3.50
4.11
3.29
4.17
3.50
3.75
4.09
3.48
4.43
3.56
3.25
4.02
3.71
4.43
3.78
4.00
4.10
3.58 c
4.39 b
3.76
3.90
*Superscript letters indicate significantmean differencesat the .05 level between the program and the letter of the program(s) listed using a post hoc Scheffe test. t p < .01. P < .05.
useful, information than the global satisfaction items in the CSQ-8. Additional support for convergent validity was provided by the positive relationship of the PSS to ratings o f parent satisfaction with specific services received as listed on the M H S S (r = 0.76, P < .001).
Discussion The purpose of this study was to test the psychometric properties o f the Parent Satisfaction Scale. Results showed that the PSS was a unidimensional scale with high internal consistency reliability in this sample of parents with children receiving mental health services. There was also support for both construct and convergent validity. The P S S ' s high correlations with variables proposed to have a direct relationship to parent satisfaction (Met Expectations, M e t Desires, and Met Needs) supported construct validity. The high correlations of the PSS with the CSQ-8 and with the
Scale for Measuring Parent Satisfaction With Staff Interactions
GERKENSMEYER, AUSTIN
69
satisfaction ratings of services received on the Mental Health Services Scale supported convergent validity. The PSS was able to differentiate among programs and settings. There were significant differences among programs on most of the items. These differences, relating to topic areas such as including parents in decision making, informing parents about changes in the child's care, and listening to what parents had to say, were not surprising when comparing community-based programs to state hospital programs. In the 2 community-based programs with higher parent satisfaction ratings, parents were actively and regularly involved in the care experience, whereas in the state hospital system parents often lived at a great distance from the hospital and were less likely to be actively involved. In addition, significant differences were found when the PSS was completed while the child was living at home versus out of home. Parents whose children lived at home were significantly more satisfied than those whose children were in out-of-home placement. Parents' being more satisfied when their child lived at home during survey completion might have confounded increased satisfaction with community-based services. There was support that the patterns of differences between programs remained the same when controlling for the child's living situation; however, for several items there was a decrease in the level of significance. It is also likely that many of the children living in out-of-home placements would not be functioning as well as those at home and, therefore, parents would be less satisfied with services. Finally, with no significant differences in the PSS total mean score by demographic variables, except for children living at home versus out of home during survey completion, it would appear that these variables do not unduly influence the PSS results. Limitations Limitations of this study that could interfere with generalizability of findings included use of 1 geographic location, possible response bias due to not obtaining information about nonresponders, and the use of a cross-sectional design. It was possible that parent satisfaction varied during different stages of the treatment process. In addition, having data collected at only 1 point in time and anywhere from 5 to 10 months after admission could have affected results both within a given program and across programs. Additional limitations included the 52% response rate and the highly positive mean score for parent satisfaction that might indicate that it would be difficult to show improvement. Future research recommendations Examination of test-retest reliability is needed to test the stability of the scale. At least 1 item on cultural competence needs to be added to the PSS to provide information about this important aspect of care. An item being considered is, "I am satisfied with how staff respected my family's values and beliefs" Efforts to examine nonrespondents' level of satisfaction with services are also needed for determining if nonrespondents are more dissatisfied with services than respondents or if there are particular aspects of care that nonrespondents tend to rate more negatively. In addition, whether parent satisfaction changes over the course of treatment needs to be examined. Although we found no significant differences in parent satisfaction and demographic variables, except for living at home versus out of home during survey completion, research is also needed to clarify if interprogram score differences are entirely due to differences in the programs themselves or if child and parent characteristics, such as the child living at home or out of home, may account for these differences. None of the PSS items were designed to be specific to the mental health context, although it has only been tested with parents of children with mental health problems. There is no reason to believe that PSS would not be relevant for children receiving services in other contexts such as acute or chronic physical healthcare settings, and therefore it should be tested in these settings.
70
The Journal of Behavioral Health Services & Research
32:1
January~March 2005
Implications for Behavioral Health The PSS provides evaluative information from the parents' perspective about how pleased they are with services for their child, primarily focusing on staff's interpersonal interactions. Parent satisfaction is a very important factor to measure because it is likely to be highly related to parents' being active partners in therapeutic efforts to address their child's health needs. 58 There is increasing evidence in the mental health context that parent satisfaction with services is related to mental health clinical outcomes for children. 7° The current 11-item PSS is not only suited for research studies, but also has the potential to provide relevant information for clinical practice. To obtain open and honest responses, however, participants' confidentiality needs to be maintained to prevent them from feeling at risk for negative consequences when sharing their opinions. The information provided by the PSS can be useful to inform quality improvement efforts and program development. Because the items in the PSS relate primarily to staff interactions, however, other complementary measures might be indicated if focusing on improvements in areas other than staff interactions. Except for 1 global satisfaction item, the PSS consists of items that provide information about specific aspects of parent interactions with staff. Despite their specificity, the items should be applicable across programs and settings serving children with health problems and their families. The use of specific items provides information about which aspects of care the parents are most and least pleased with, which is particularly useful in identifying targets for improving parent satisfaction. Findings from this study, for example, indicated that staff in state-operated inpatient settings needed to focus on ways to work more closely with parents. The PSS has strong psychometric support and is based on a conceptual model that has been empirically tested. 2° Findings from this study suggested that the PSS is a valid and reliable scale that can be used both in research studies and to provide clinicians with useful and specific information to evaluate services and guide program development in the context of children's mental health. Further testing in other contexts will determine if the PSS is valid and reliable in other healthcare contexts.
Acknowledgments The research was supported by grants from the Association for the Advancement of Mental Health Research and Education, Inc, and the Indiana University School of Nursing Graduate School. The authors thank the program administrators and staff for their assistance, the parents for their participation, Phyllis Dexter for editorial comments, and Susan M. Perkins for statistical consultation.
References 1. U.S. Public Health Service. Report of the Surgeon General's Conference on Children's Mental Health: A National Agenda. Washington, DC: U.S. Government Printing Office; 2000. 2. Substance Abuse and Mental Health Services Administration. Mental Health, United States. Rockwell, Md: Center for Mental Health Services; 1996. 3. U.S. Department of Health and Human Services. NationalPlanfor Research on Child andAdolescent Mental Disorders: A Report Requested by the U.S. Congress. Washington, DC: National Advisory Council; 1990. 4. Boyd LA. Integrating Systems of Care for Children and Families: An Overview of Values, Methods and Characteristics of Developing Models, With Examples and Recommendations. Tampa, Fla: Florida Mental Health Institute; 1994. 5. Junek W, Thompson AH. Self-regulating service delivery systems: a model for children and youth at risk. Journal of Behavioral Health Services and Research. 1999;26(1):74-79. 6. Knitzer J. Children's mental health policy: challenging the future. Journal of Emotional and Behavioral Disorders. 1993; 1:8-16. 7. National Mental Health Association and the Federation of Families for Children's Mental Health. All Systems Failure: An Examination of the Results of Neglecting the Needs of Children With Serious Emotional Disturbance. Washington, DC: National Mental Health Association; 1993. 8. Greeneich D. The •ink between new and retum business and qua•ity •f care: patient satisfacti•n. Advances in Nursing Science. 1993;16(1):6272. 9. Megivern K, Halm MA, Jones G. Measuring patient satisfaction as an outcome of care. Journal of Nursing Care Quality. 1992;6(4):9-21.
Scale for Measuring Parent Satisfaction With Staff Interactions
GERKENSMEYER, AUSTIN
71
10. Plante TG, Couchman CE, Diaz AR. Measuring treatment outcome and client satisfaction among children and families. The Journal of Mental Health Administration. 1995;22(3):261-269. 11. Delaney KR, Engels-Seianna B. Parents' perceptions of their child's emotional illness and psychiatric treatment needs. Journal of Child and Adolescent Psychiatric Nursing. 1996;9(4):15-24. 12. Dickey B, Wagenaar H. Evaluating mental health care reform: including the clinician, client, and family perspective. The Journal of Mental Health Administration. 1994;21(3):313-319. 13. Eriksen LR. Patient satisfaction with nursing care: concept clarification. Journal of Nursing Measurement. 1995;3(1):59-76. 14. Hsieh M. Importance and Anticipation as Expectation and Their Relationship to Patient Satisfaction With Health Care [dissertation]. Chicago, Ill: University of Illinois; 1988. 15. Lin C. Patient satisfaction with nursing care as an outcome variable: dilemmas for nursing evaluation researchers. Journal of Professional Nursing. 1996;12(4):207-216. 16. Sheppard M. Client satisfaction, extended intervention, and interpersonal skills in community mental health. Journal of Advanced Nursing. 1993;18:246-259. 17. Williams B, Wilkinson G. Patient satisfaction in mental health care. British Journal of Psychiatry. 1995;166:559-562. 18. McMahon RJ, Forehand RL. Consumer satisfaction in behavioral treatment of children: types, issues, and recommendations. Behavior Therapy. 1983;14:209-225. 19. Young SC, Nicholson J, Davis M. An overview of issues in lesearch on consumer satisfaction with child and adolescent mental health services. Journal of Child and Family Studies. 1995;4(2):219-238. 20. Gerkensmeyer J. Examining Parent Satisfaction With Services for Children and Adolescents With Mental Health Problems [dissertation]. Indianapolis, Ind: Indiana University; 1999. 21. Lebow J. Consumer satisfaction with mental health treatment. Psychological Bulletin. 1982;91(2):244-259. 22. Gowers S, Knshlick A. Customer satisfaction in adolescent psychiatry. Journal of Mental Health. 1992; 1:353-362. 23. Measelle JR, Weinstein RS, Martinez M. Parent satisfaction with case managed systems of care for children and youth with severe emotional disturbance. Journal of Child and Family Studies. 1998;7(4):451--467. 24. Palisin H, Cecil J, Gumbardo D, Variey C. A survey of parents' satisfaction with their children's hospitalization on a psychiatric unit. Children's Health Care. 1997;26(4):233-240. 25. Gerkensmeyer J, McBride AB, Feaster CE Austin J. Evaluating mental health services for children: the parents' perspective. In: Liberton C, Kutash K, Friedman R, eds. The 9th Annual Research Conference Proceedings: A System of Care for Children's Mental Health. Tampa, Fla: University of South Florida; 1997:153-158. 26. Webster-Stratton C. Systematic comparison of consumer satisfaction of three cost-effective parent training programs for conduct problem children. Behavior Therapy. 1989;20:103-115. 27. Baker BL, Landen SJ, Kashikma KJ. Effects of parent training on families of children with mental retardation: increased burden or generalized benefit? American Journal on Mental Retardation. 1991;96(2):127-136. 28. Bradley EJ, Clark BS. Patients' characteristics and consumer satisfaction on an inpatient child psychiatric unit. Canadian Journal of Psychiatry. 1993;38(3):175-180. 29. Charlop MH, Parrish JM, Renton LR, Cataldo ME Evaluation of hospital-based outpatient pediatric psychology services. Journal of Pediatric Psychology. 1986;12(4):485-503. 30. Chang WS, Pardeck JT, Murphy JW. Factors associated with premature termination of psychotherapy by children. Adolescence. 1995;30(119):717-721. 31. Furey WM, Basili LA. Predicting consumer satisfaction in parent training for noncompliant children. Behavior Therapy. 1988; 19:555-564. 32. Hillier S, Loshak R, Rahman S, Marks E An evaluation of child psychiatric services for Bangladeshi parents. Journal of Mental Health. 1994;3:327-337. 33. Homer B, Rey JA. Preliminary findings from program evaluation research on the North Idaho Child And Adolescent Service System Project. In Willis J, Liberton C, Kutash K, Friedman R, eds. The 6th Annual Research Conference Proceedings: A System of Care for Children's Mental Health: Expanding the Research Base. Tampa, Fla: University of South Florida; 1993:19-28. 34. Kopec-Schrader EM, Marden K, Rey SW, Touyz PJ, Beumont, PJV. Parental evaluation of treatment outcome and satisfaction with an inpatient program for eating disorders. Australian and New Zealand Journal of Psychiatry. 1993;27:264-269. 35. Szykula AA, Morris SB, Stadweeks C. Child-focused behavior and strategic therapies: outcome comparisons. Psychotherapy. 1987 ;24(35):546-551. 36. Wi~iams PD~E~derJH~Griggs C. The e~ects ~f fami~y training and supp~rt ~n c~i~dbebavi~r and parent s~tisfa~ti~n~Archives ~f Psychiatric Nursing. 1987;1(2):89-97. 37. Pascoe JM. Patient satisfaction in primary health care. Evaluation and Program Planning. 1983;6:185-210. 38. Attkisson CC, Zwick R. The client satisfaction questionnaire: psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluation and Program Planning. 1982;5:233-237. 39. Attkisson CC, Greenfield TK. The Client Satisfaction Questionnaire-8 and the Service Satisfaction Scale-30. In: Maruish ME, ed. The Use of Psychological Testing for Treatment Planning and Outcome Assessment. Hillsdale, NJ: Erlbaum; 1994:402~-20. 40. Taylor E Nelson-Wernick E, Lois C. Development and use of a method of assessing patient perceptions of care. Hospital and Health Services Administration. 1981 ;Winter:89-104. 41. Vandvik IH, Hoyeraal HM, Fagertun H. The first stay in a pediatric rheumatology ward. Scandinavian Joutrtal of Rheumatology. 1990; 19:216-222. 42. DeVellis RE Scale Development: Theory and Applications. Newbury Park: Sage; 1991. 43. Corrigan P. Consumer satisfaction with institutional and community care. Community Mental Health Journal. 1990; 26(2):155-165. 44. Locker D, Dant D. Theoretical and methodological issues in sociological studies of consumer satisfaction with medical care. Social Science and Medicine. 1978;12:283-292. 45. McPhee C, Zusmen J, Joss RH. Measurement of patient satisfaction: a survey of practices in community mental health centers. Comprehensive Psychiatry. 1975;16(4):399~-04.
72
The Journal of Behavioral Health Services & Research
32:1
January~March 2005
46. Petersen M. Measuring patient satisfaction: collecting useful data. Journal of Nursing Quality Assurance. 1988;2(3):25-35. 47. Dawson K, Mogridge N. Parental perceptions of paediatric inpatient care. New Zealand Medical Journal. 1991; 104:12-13. 48. Edwards D, Yarvis RM, Mueller DP, Langsley DG. Does patient satisfaction correlate with success? Hospital and Community Psychiatry. 1978;29(3):188-190. 49. Eriksen L. Patient satisfaction: an indicator of nursing care quality. Nursing Management. 1987;18:31-35. 50. Essex D, Fox J, Joseph M. The development, factor analysis, and revision of a client satisfaction form. CommunityMental Health Journal. 1981;17(3):226-235. 51. Fiester A, Fort D. A method of evaluating the impact of services at a community mental health center. American Journal of Community Psychology. 1978;6(3):291-302. 52. Goldsmith G. Patient satisfaction with a family practice clinic: comparison of a questionnaire and an interview survey. The Journal of Ambulatory Care Management. 1983;May:24-31. 53. Grella C, Grusky O. Families of the seriously mentally ill and their satisfaction with services. Hospital and Community Psychiatry. 1989;40(8):831-835. 54. Kotsopoulos S, Elwood S, Oke L. Parent satisfaction in a child psychiatric service. Canadian Journal of Psychiatry. 1989;34:530-532. 55. Reynolds I, Jones J~ Berry D, Houtt J. A crisis team for the mentally ill: the effect on patients, relatives, and admissions. American Psychologist. 1990; 152:646-652. 56. Risser N. Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings. Nursing Research. 1975;24(1):45-52. 57. Stamps P, Lapriore E. Measuring patient satisfaction in a community hospital. Hospital Topics. 1987;65(5):22-26. 58. Wilier B, Miller GH. On the relationship of client satisfaction to client characteristics and outcome of treatment. Journal of Clinical Psychology. 1978;34(1): 157-162. 59. Williams F, Elder J, Griggs C. The effects of family training and support on child behavior and parent satisfaction. Archives of Psychiatric Nursing. 1987;1(2):89-97. 60. Zyzanski S, Shulka B, Cassell J. Scale for the measurement of "satisfaction" with medical care: modifications on content, format, and scoring. Medical Care. 1974;12(7):611-620. 61. Armstrong-Esther C, Lacy B, Sandilands M, Browne K. Partnership in care. Journal of Advanced Nursing. 1987;12:735-741. 62. Collins B, Collins T. Parent-professional relations in the treatment of seriously emotionally disturbed children and adolescents. Social Work. 1990;35(6):522-527. 63. Modcrin M, Robinson J. Parents of children with emotional disorders: issues for consideration and practice. Community Mental Health Journal. 1991 ;27(4):281-292. 64. Kutash K, Robbins T. Measures of Satisfaction With Child Mental Health Services. Tampa, Fla: Florida Mental Health Center: Research and Training Center for Children's Mental Health; 1992. 65. Gerkensmeyer J, McBride AB, Finke L, Austin J. Parent satisfaction with mental health staff interactions: the development, validity, and reliability of the parent satisfaction scale and the parent-staff interaction scale. In: Liberton C, Kutash K, Friedman R, eds. The 8th Annual Research ConferenceProceedings:A System of Carefor Children's Mental Health. Tampa, Fla: University of South Florida; 1997:153-158. 66. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th ed. Washington, DC: American Psychiatric Association; 1994. 67. McDermott PA. A nationwide study of developmental and gender prevalence for psychopathology in childhood and adolescence. Journal of Abnormal Child Psychology. 1996;24(1):53-66. 68. Byalin K. Assessing parental satisfaction with children's menial health services. Evaluation and Program Planning. 1993;16:69-72. 69. Stevenson J, Srebnik D. Congruence between parent-professional ratings of level of functioning: collaboration and satisfaction. In: Liberton C, Kutash K, Friedman R, eds. The 8th Annual Research ConferenceProceedings: A System of Carefor Children's Mental Health. Tampa, Fla: University of South Florida; 1997:167-170. 70. Gerkensmeyer J. Parent satisfaction with mental health services: relationship to outcomes. In Liberton C, Kutush K, Friedman R, eds. The 14th Annual Research Conference Proceedings: A System of Care for Children's Mental Health. Expanding the Research Base. Tampa, Fla: University of South Florida; 2001:259-266.
Sca le f o r M e a s u r i n g Parent Satisfaction With S t a f f Intera c t i ons
GERKENSMEYER, AUSTIN
73
