Evaluation of Keystone STARS Quality Rating System: Preliminary Results Richard Fiene, Ph.D. Penn State University April 27, 2006
Summary Keystone STARS, Pennsylvania’s Quality Rating System, helps to improve child care quality. Preliminary results from a statewide evaluation clearly indicate that Keystone STARS helps child care programs improve their overall quality. An independent evaluation being conducted by Richard Fiene, a Penn State faculty member, demonstrates that this innovative quality rating system is having its intended impact on child care quality in Pennsylvania. Keystone STARS is reversing the negative trend in child care quality that was so evident in the late 1990’s. Today, child care programs are operating at a much higher quality level because of the Keystone STARS program. Introduction Research has clearly demonstrated that high-quality early care and education can help prepare children for success in school, work and life (Schweinhart, 2004). Because of this research a growing number of states and communities have begun to develop Quality Rating Systems (QRS), a strategy to improve the quality of early care and education programs by providing “star ratings” (Mitchell, 2005). Pennsylvania is one of those states. The Keystone STARS system, Pennsylvania’s version of their quality rating system, has been in place for four years. Administrators in the Pennsylvania Office of Child Development felt it was time to evaluate the effectiveness of their particular program. This brief report presents the preliminary results of the evaluation being conducted in 2006. This evaluation needs to be put into historical context because Pennsylvania has a rich history of assessing the overall quality of their early care and education program. The most recent example was completed in 2002 (Fiene, Greenberg, Bergsten, Fegley, Carl, & Gibbons, 2002) but other research studies were completed in 1996 (Fiene, Iutcovich, Johnson, & Koppel, 1998; Iutcovich, Fiene, Johnson, Koppel, & Langan, 2001), 1990 (Melnick & Fiene, 1990), 1984 (Kontos & Fiene, 1986, 1987), and 1978 (Fiene & Aronson, 1979). The 2002 Early Care and Education Quality Study had been the most comprehensive study completed in Pennsylvania to date with approximately 400 sites accessed. It helped to establish a baseline to quality so that future quality initiatives could be measured to determine the relative effectiveness of these initiatives. So in order
EVALUATION OF KEYSTONE STARS Fiene
1
to put these 2006 preliminary findings into perspective, a brief review of the 2002 Early Care and Education Quality Study is needed. The 2002 Early Care and Education Quality Study, commissioned by then Governor Schweiker as part of his Task Force on Early Care and Education, clearly demonstrated that the overall quality of child care had decreased significantly from previous studies completed in the 1980’s and 1990’s. Significant decreases were noted in both child care centers and family child care homes. In both cases, quality scores (ERS—Environmental Rating Scales) as measured by the Early Childhood Environment Rating Scale (ECERS-R)(Harms, Clifford, & Cryer, 1998) and the Family Day Care Rating Scale (FDCRS)(Harms and Clifford, 1989) had dropped from a 4.50 level to just below 4.00 level. So there was great concern in Pennsylvania in attempting to reverse this decrease in child care programs’ quality. The answer was the Keystone STARS quality rating system. Methodology Forty (40) programs have been randomly selected from across the state. Twentyone (21) were child care programs, while 19 were family or group child care homes. They were proportionately divided amongst Start with STARS (5) and STAR 1-2 programs (20); and STAR 3-4 programs (15). These programs were representative of the state in that they were geographically dispersed. Seven data collectors trained on the ECERS-R and FDCRS collected the data on all 40 programs. All data collectors were determined reliable (.85 or above) before they were allowed to collect data for the study. Data were collected from February 2006 until the April 2006. Results The results from the study were very significant in that the higher a program was in the Keystone STARS system the higher their ERS rating. The average ERS scores for Star 4, the highest rating was 5.99 followed by 5.89 for Star 3. Star 1 and 2 attained scores of 4.41 and 4.78 respectively. Start with STARS, a readiness type of program for STARS was 3.40. These results are statistically significant utilizing an ANOVA OneWay analysis (F = 8.484; df = 39; p < .0001)(see Figure 1). Figure 1: ERS Scores by Stars Level Stars Level ERS Scores Start with Stars 3.40 Star 1 4.41 Star 2 4.78 Star 3 5.89 Star 4 5.99 In comparing the 2006 results with previous early childhood quality studies in Pennsylvania there were statically significant differences as well (See Figure 2). In 1978, the Child Development Program Evaluation Scale (CDPES) was used to measure quality. Comparing it to the ERS Scales, programs scored at a 4.75 level. In 1984 and in 1990,
EVALUATION OF KEYSTONE STARS Fiene
2
Figure 2: Comparisons to Previous Early Childhood Quality Studies Year ERS Scores 1978 4.75 (CDPES) 1984 4.00 1990 4.00 1996 4.50 2002 3.90 2006 3.40 Start with Stars 2006 4.60 Star 1 & 2 Levels 2006 5.95 Star 3 & 4 Levels the average ERS score was 4.00; in 1996, the average ERS score was 4.50; in 2002, the average ERS score was 3.90. In 2006, Start with STARS was 3.40; STAR 1 or 2 was 4.60; and STAR 3 or 4 was 5.95. The STAR 3 and 4 programs were significantly better than in previous years while the Start with STARS and STAR 1 or 2 were not (F = 9.970; df = 411; p < .0001). Discussion These results clearly demonstrate the effectiveness of the Keystone STARS quality rating system. Child Care Programs, both child care centers and homes, improved significantly as they moved up the STAR continuum. The Keystone STARS system has reversed the drop off in quality that was so evident in the late 1990’s and early 2000’s. This is a major reversal that has helped to improve the overall quality of child care in Pennsylvania. Once a program gets engaged in the STARS program they start to improve. This was evident from the STAR 1 and 2 levels. Once they move into a STAR 3 or 4 level, their overall quality scores as measured by the ERS scales improves significantly. From these results, it is obvious that the Keystone STARS program is a success. From a national perspective, these are very important findings in demonstrating that the quality rating systems being developed in other states should be evaluated in a similar fashion in order to determine their relative effectiveness. Conclusion Although these are preliminary findings that are the beginning stage of a much larger study involving evaluations of over 400 child care programs, these results are representative of the full number of programs. The results were so statistically significant it is important to report on these data as preliminary findings. The remaining program data are being collected now and will be completed this summer, 2006. At that point, a report detailing the results of all the child care programs participating in the evaluation will be produced.
EVALUATION OF KEYSTONE STARS Fiene
3
References Fiene & Aronson (1979). The Child Development Program Evaluation Study, Harrisburg, Pennsylvania: Bureau of Child Development. Fiene, Greenberg, Bergsten, Carl, Fegley, & Gibbons (2002). The Pennsylvania early childhood quality settings study, Harrisburg, Pennsylvania: Governor’s Task Force on Early Care and Education. Fiene, Iutcovich, Johnson, & Koppel (1998). Child day care quality linked to opportunities for professional development: An applied community psychology example. Community Psychologist, 31(1), 10-11. Harms, T., Clifford, R., & Cryer, D., 1998. Early childhood environment rating scale, New York: Columbia University Teachers College Press. Harms, T., and Clifford, R., 1989. Family day care environment rating scale. New York: Columbia University Teachers College Press. Iutcovich, Fiene, Johnson, Koppel, & Langan (2001). Professional development and the quality of child care: An assessment of Pennsylvania's child care training system. In Early education and care, and reconceptualizing play, Elsevier Science Ltd., Volume 11, 115168. Kontos & Fiene (1987). Child care quality, compliance with regulations, and children's development: The Pennsylvania Study, in Quality in Child Care: What Does Research Tell Us?, Phillips, editor, Washington, D.C.: National Association for the Education of Young Children. Kontos & Fiene (1986). Predictors of quality and children's development in day care, in Licensing of Children's Services Programs, Richmond, Virginia: Virginia Commonwealth University School of Social Work. Melnick, S., & Fiene, R., 1990. Licensure and program quality in early childhood and child care programs. Paper presented at the annual meeting of the American Educational Research Association, Boston, Massachusetts. Mitchell, A., 2005. Stair steps to quality, Alexandria, Virginia: United Way Success by Six, Schweinhart, L., 2004. The High/Scope Perry Preschool Study through age 40: Summary, conclusions and frequently asked questions. Ypsilanti, Michigan: High Scope Research.
EVALUATION OF KEYSTONE STARS Fiene
4
For additional information regarding this study: Richard Fiene, Ph.D. Associate Professor and Director Capital Area Health and Human Development Institute The Pennsylvania State University 2001 North Front Street, Building 2, Suite 221 Harrisburg, Pennsylvania 17102 717-233-5276
[email protected] http://www.hhdev.psu.edu/cahhdi/
EVALUATION OF KEYSTONE STARS Fiene
5
