A Case Study of the Philadelphia Public School's School ... - CiteSeerX

CPRE

CONSORTIUM FOR POLICY RESEARCH IN EDUCATION

University of Pennsylvania •Harvard University •Stanford University University of Michigan •University of Wisconsin-Madison

A Case Study of the Philadelphia Public School’s School-Based Performance Award Program

Eileen Kellor, Allan Odden, and Eric Conti Consortium for Policy Research in Education University of Wisconsin-Madison 1025 W. Johnson Street Madison, WI 53706 (608) 263-4260 http://www.wcer.wisc.edu/cpre

September, 1999 The research reported in this paper was supported by grants from the Pew Charitable Trusts (No. 97001184000), from the U.S. Department of Education, Office of Educational Research and Improvement, National Institute on Educational Governance, Finance, Policy-Making and Management, to the Consortium for Policy Research in Education (CPRE), the Wisconsin Center for Education Research, School of Education, University of Wisconsin-Madison (Grant No. OERI-R3086A60003) and from the Carnegie Corporation (No. B6520). The opinions expressed are those of the authors and do not necessarily reflect the view of the Pew Charitable Trusts, the National Institute on Educational Governance, Finance, Policy-Making and Management, Office of Educational Research and Improvement, U.S. Department of Education, the Carnegie Corporation, the institutional partners of CPRE, or the Wisconsin Center for Education Research.

UW

Wisconsin Center for Education Research, University of Wisconsin-Madison 1025 West Johnson Street, Room 659, Madison, WI 53706-1796 •Phone 608.263.4260 •Fax 608.263.6448

Introduction Effective with the 1996-97 school year, the Philadelphia school district implemented a school-based performance award program as part of a broader strategy to improve student achievement throughout the district. This case will provide basic background information about the district and the circumstances leading up to the decision to establish a school-based performance award program. It will summarize the process used to design the program and provide an overview of the program’s design details and enablers that support the program’s goals. Finally, some evaluation findings will be reported and observations made regarding possible considerations for other jurisdictions considering a school-based performance award program. Background The Philadelphia Public Schools enrolls 214,000 students distributed across 259 schools. The student population is 65% African American, 19% white, 11% Latino, and 5% Asian. The district employs approximately 12,000 teachers at an average salary of $46,659, compared to the state of Pennsylvania average of $47,147. Per pupil spending in 1997 was approximately $6,860. Benchmark tests administered in 1996 established that less than half of the students displayed basic knowledge and skills in mathematics, reading, and science (Partnership for Reform, 1996). In 1983 the district entered into a desegregation agreement in which it agreed to address educational disparities as well as racial isolation (Schmidt, 1994a). Through the efforts of the district and other public and non-profit organizations, a variety of reform strategies were tried over the years. For example, beginning in 1988 the Philadelphia School Collaborative began to assist the district in its effort to restructure the city’s

UW


neighborhood high schools. The effort included implementing school-based management, shared decision-making at the school level, and generally identifying the administrative and procedural obstacles to restructuring the high schools and working with the district to remove those obstacles. One of the strategies included the creation of “charters”, or semiautonomous schools, within the high schools. Many of the problems encountered in the high school restructuring effort were mirrored at the lower grade levels, e.g., high student and teacher mobility, challenging labor-management relations, promotion policies that resulted in large numbers of students being over-age for their grade level, and chronic financial problems (Bradley, 1992). Early in 1994 a judge ruled that the district had failed to desegregate or to provide minority and poor students with equal educational opportunities. Although the district claimed that the disparities in achievement were separate from and irrelevant to the desegregation case, the judge stated that the educational differences were “the paramount and most fundamental issues presented.” The judge appointed a panel to help the district develop a plan that would address the achievement issues by some specific strategies identified by the judge, as well as others developed by the panel (Schmidt, 1994). Thus, the issue of improving student achievement became not only a socially desirable goal but also a court-mandated goal for the district. This backdrop contributed to the high level of importance and visibility of the district’s efforts to improve student achievement. At this same time the district was in the process of hiring a new superintendent. The previous superintendent had retired in the summer of 1993 after 11 years as superintendent. In June 1994, the Board announced that it had selected David Hornbeck, an educational consultant well known for promoting systemic, standards-based education

UW


reform, including the state of Kentucky’s comprehensive education reform program. Hornbeck had been one of two finalists for the position and was the only candidate remaining after the second candidate withdrew his application (Bradley, 1994). The community generally reacted positively to his selection and his strong commitment to improving student achievement. In fact, Hornbeck’s own contract included provisions that tied salary increases to student achievement goals and he made his appointment contingent upon being allowed to implement an accountability program (Bradley, 1994). One of Hornbeck’s first projects as superintendent was the development of a strategic plan for the district; the plan was titled Children Achieving. Around the same time the court-appointed panel issued its recommendations for restructuring Philadelphia’s schools, Hornbeck made public his proposal for a strategic plan for the district. The proposal was generally consistent with his statements about what he would do to improve Philadelphia’s schools that he made during the selection process. The strategic plan was developed through the efforts of seven task forces involving hundreds of educators and citizens. It included many of the same suggestions made by the courtappointed panel, including downsizing central administration and regional offices; implementing school-based management throughout the district, including local school councils of parents, teachers, and administrators; extending the paid work year for principals and teachers to allow for more professional development opportunities; reducing teacher-student ratios and creating smaller “learning communities” within schools; providing full-day kindergarten and prekindergarten; developing alternative schools and other programs for the most disruptive students; and also developing standards, assessment and accountability systems. The most racially isolated schools

UW


would be targeted initially and most of the reforms would be implemented within three years (Schmnidt, 1994b). In early 1995 the Annenberg Foundation announced that it was awarding a $50 million challenge grant to the Philadelphia schools to support the comprehensive and systemic reform agenda laid out in Children Achieving. The grant required an additional $100 million to be raised, with $50 million to come from both the public and private sector. The Greater Philadelphia First organization, a coalition of leaders from major local corporations, would be the fiscal agent for the grant. The strong leading role of the superintendent in planning Philadelphia’s reforms, rather than forces external to the schools, was noted as a relatively unique aspect of Philadelphia’s Annenberg proposal. In fact, an individual involved in the drafting of Philadelphia’s proposal stated that the drafters made an effort to “find the elements of Mr. Hornbeck’s strategic plan that fit the best with Mr. Annenberg’s interests, in order to have a single comprehensive reform effort in Philadelphia in which all the pieces fit” (Sommerfeld, 1995). Ten major reforms were articulated in the strategic plan, Children Achieving. They were: 1. Set high expectations for everyone. 2. Design accurate performance indicators to hold everyone accountable for results. 3. Shrink the centralized bureaucracy and let schools make more decisions. 4. Provide intensive and sustained professional development. 5. Make sure that all students are ready for school.

UW


6. Provide students with the community supports and services they need to succeed in school. 7. Provide up to date technology and instructional materials. 8. Engage the public in shaping, understanding, supporting and participating in school reform. 9. Ensure adequate resources and use them effectively. 10. Be prepared to address all of these priorities together and for the long term. In February 1995, Hornbeck announced plans to go ahead with aggressive reform measures (Schmidt, 1995). One of the specific strategies (Strategy 2) developed to achieve the strategic plan’s goals acknowledged the need for assessments and an incentive system. That strategy included three specific components: 1. Implement a system of performance-based assessments tied to the new high standards for students; 2. Design valid assessment for students of diverse language background. 3. Design an incentive system for staff that links achievement by all students, including those whom the schools have historically failed, to real rewards and penalties. These three components created the basic framework for Philadelphia’s school-based performance award program. The focus on results and their publication, coupled with extrinsic incentives in the form of rewards and sanctions, are central to Philadelphia’s accountability system. The district believes that these factors are engines that drive school improvement for all students. Insisting that the 10 Children Achieving goals must work together as an

UW


integrated, comprehensive package, the thinking behind the accountability component is that it contributes to achievement in one of two ways: teachers will teach better because they desire the reward (either cash or public recognition) obtained for higher test scores, and teachers will teacher better because if student performance fails to improve, the teachers will be subject to various sanctions (Luhm, Foley and Corcoran, 1998).

UW


Program Design Although the basic principles of the program were set forth in the strategic plan, the development of the program’s details included hundreds of partners. The design process, including development of the actual curriculum content and student performance standards, made steady progress, with the work formally beginning in December 1995 and the actual accountability measurement system completed by the end of November 1996. The participants included 1,000 administrators, teachers, and parents from the district as well as over 2,500 citizens who attended 22 community meetings (one in each district cluster). During this time, a Philadelphia team was part of CPRE’s first schoolbased performance award design seminar (Odden, Heneman, Wakelyn and Protsik, 1996). The actual design of the details began when the Standards Writing Teams began their work. In April and May of 1996, draft standards in English language arts, mathematics, science and the arts, were distributed for review, with review teams formally commenting on the first four sets of standards in the summer of 1996. In July 1996, concurrent with the review of the draft standards, the district held a four-day professional development session on standards-based instruction for teams of teachers. The second draft of standards was distributed to all teachers in August 1996 and curriculum resources guides for grades K-4, 5-8 and 9-12 distributed to teachers in September. During October and November the details of the accountability system, the Performance Responsibility Program, were developed and approved by the Board of Education and public hearings on the proposed standards were held in each of the district’s 22 clusters. By the end of 1996 the Board adopted the first four sets of

UW


Recommended Content Standards, Benchmarks and Performance Examples. A resource guide for standards-based assessment and instruction was distributed to schools in February 1997. Development of the standards in the remaining areas (health and physical education, social studies, and world languages) continued, with final Board approval in July 1997 (CPRE, 1998). Elements of Performance The district’s accountability system is based on three performance indicators, which are combined into a single measure called the Performance Responsibility Index (PRI). The PRI includes three separate elements: student achievement scores in reading, match and science (60% of total score), graduation and promotion rates (20% of total), and student and staff attendance (20% of total score). For elementary and middle schools, a school’s promotion rate must average at least 95%; the graduation/persistence rate for high schools requires that an average of 95% of first-time 9th graders must graduate from a Philadelphia school district high school four years later. With respect to student attendance, an average of 95% of the students in a school must be absent fewer than 10 days per school year to be considered “proficient” and an average of 95% of the school staff must be absent fewer than 10 days per school year (CPRE, 1998). The district selected the ninth edition of the Standard Achievement Test (SAT-9) to measure student achievement. The district chose the SAT-9 assessment because it is a criterion-reference test based on national standards and because it has a performancebased feature, not relying exclusively on multiple choice. Because the district’s own standards were based on national standards the district believed the SAT-9 fit well with its local standards. The first year of the program included the SAT-9 scores in reading,

UW


mathematics and science in grades 4, 8, and 11. The second year of the program added grades 3, 7, and 10 but these grades were not included in the accountability program during the first cycle (CPRE, 1998). The student and staff attendance measure was included to address concerns about serious teacher and student attendance problems. The persistence to graduation rate and promotion rate were included to ensure that students were not just dropping out and thus excluded from the accountability measures. In addition, those measures help to demonstrate that students are actually achieving, i.e., making progress and moving from grade to grade and eventually graduating. Calculating Change in Performance The following section documents how change in performance, i.e., improvements in student achievement) are calculated for each school. It also summarizes the methods the district uses to make the calculations fair, such as adjustments made or precautions taken to address special measurement issues and concerns. How change is measured A key concept of Philadelphia’s accountability program is that schools are not compared to other schools; each school is compared to its own baseline performance over time. Each of the three performance elements noted above has multiple performance levels. The SAT-9 scores are categorized into seven performance levels and the other two performance elements are categorized into six performance levels. Each performance level is given a weight, e.g., ranging from 0.0 points for Not Tested to 1.2 points for Advanced level for the SAT-9 score categories. The value of the performance

UW


level is then multiplied by the percentage of students categorized at that level and the total score derived from the sum of the scores at each level. The twelve-year (one student generation) goal is that every school will have an average score of 95% across the three performance elements. This is measured by means of the PRI, which is a summary of a school’s results compared to its baseline performance in the 1995-96 school year. Using the 95% average as the ultimate goal, the district sets a target for each school every two years. The current accountability plan anticipates six two-year cycles. The SAT-9 academic scores are reported using seven performance levels. These performance levels are determined by the SAT-9 test reports and are similar to the National Assessment of Educational Progress (NAEP) performance levels. Harcourt Brace, the test developer, provides teacher evaluators with a rubric that outlines students’ performance levels. The standard report includes four levels of performance from advanced to below basic. However, the district was concerned that students making significant progress in the below basic performance level but not enough to attain the basic performance level would not be measured as having improved. To remedy this problem, the district and Harcourt Brace agreed to further delineate below basic performance into three separate performance levels. Thus, the seven levels are as follows: 1. Advanced: superior performance on world-class standards. 2. Proficient: solid performance on world-class standards. 3. Basic: partial mastery of world-class standards. 4. Below Basic III: inadequate mastery.

UW


5. Below Basic II: little mastery. 6. Below Basic I: very little mastery. 7. Not tested: students not having a valid test score. It should be noted that the ability to identify improvements in student achievement for students in the below basic category was an issue in Kentucky’s accountability program. That experience likely influenced Philadelphia’s decision to differentiate performance within the broader Below Basic category. In addition, Philadelphia’s program requires that the percentage of students categorized as performing below the “basic” level on the SAT-9 must decrease by at least 10 points, which means that at least some portion of students must move into at least the basic category for a school to receive an award. A school cannot meet its performance target unless performance improves significantly among the lowest performing students, thus a school may not “skim” the higherperforming students and rely on movement of these students from Basic to Proficient to Advanced levels of performance. The teacher and student attendance measures include the assignment of attendance levels to six categories with the same titles as those for student achievement. The calculation of the student attendance component of the PRI is as follows: 1. Each student’s individual attendance rate and the student’s days (enrolled) are placed in one of the following categories: advanced (96-100%); proficient (95%); basic (85-94%); below basic III (80-84%); below basic II (75-79%); and below basic I (10-74); 2. The total number of days enrolled by students in each category is summed and divided by the school’s total number of possible days to arrive at a PRI score

UW


for each category. For example, if the total number of possible days for a school was 30,000 and the number of days enrolled by students in the advanced category was 3,000, students in the advanced attendance level would account for 10% of the total possible days for the school 3. Attendance rates in each category are then weighed by a factor. For example, the factor for the advanced category is 1.2, so the total subscore in the example above would be 12 (10 x 1.2). Staff attendance rates are based on the total number of staff and the percent of school days they attended. All teachers are included in the statistic, as well as administrative and instructional support staff. Food service workers and custodians are not included in the score. The same weighting system noted above is used by category, i.e., advanced is weighted 1.2; proficient is weighted by 1.0; basic is weighted by 0.8, etc., but the cut scores for each category are different. For example, any staff attendance under 93% is considered below basic, while student attendance under 85% is considered below basic. The differences between basic and advanced attendance are quite small, e.g., a teacher who attended 169 days out of a 180 day school year, one who attended 171 days, and one who attended 173 days would each be included in a separate category. For example, a school in which 60% of staff attended 96% of school days would receive 72 points toward its overall staff attendance goal, while the same proportion of teachers attending 94% of school days would receive only 48 points toward a school’s overall score (Luhm, Foley & Corcoran, 1998). An aspect of the staff attendance score that was controversial in some quarters involves who is included in the calculation. As noted above, some school support staff

UW


such as food service workers and custodians are excluded, but teaching staff on long-term leave for illness or maternity leave are not excluded. The district reports that this policy struck many principals as unfair. However, the intention of this policy was not to judge the validity of the absence but rather to stress the importance of having staff present to provide services to students. Moreover, by weighting attendance in the highest category by a factor greater than 1 (1.2), the effect of long-term absences on a school’s PRI can be balanced out by having more people in the “advanced” category of attendance. A final detail of the attendance scores is how they contribute to the overall calculation of the PRI. While the SAT-9 results for each subject area count individually as one component of the PRI score, staff and student attendance are averaged to create the Enabling Score. These two different indicators, staff and student attendance, were collapsed into one component to make a single composite score that counts for one-fifth of the total index score, with academic measures counting 60%. The remaining 20% of the PRI score is calculated using student promotion and persistence scores. Promotion measures the ability of staff to enable students in grades 1 through 8 to move through at their appropriate grade level until high school. Persistence involves measuring the four-year graduation rate of the district’s 9th grade students. The persistence rate measures the proportion of first-time 9th graders in school year 1993-94 who graduate from a Philadelphia School District high school four years. The first group to which the persistence measure applied were 1992 9th-graders. Multiple schools can get credit for student persistence if the student transfers between schools and remains on schedule to graduate. Making change fair

UW


A small number of schools did not participate in the first accountability cycle because they were new and did not have a baseline for 1995-96. A few other schools did not participate because they served unique student populations for which the PRI would not be an appropriate measure of school progress. The new schools are included in the second accountability cycle because baseline performance data is now available. For the remaining schools, appropriate performance indicators and measures are being developed so that they, too, will eventually be included in the accountability program. For the first year of cycle 1 of the program (1996-97), the SAT-9 score of every 4th, 8th and 11th grade student was counted toward the score of a school. Because of the issues of student mobility for the baseline year (1995-96) a student was assigned to the school where she/he spent the most time. Students in grades 4, 8 and 11 who do not complete have a composite score on the subject being measured are given a score of zero which affects how a school performs in terms of the accountability index. This policy is designed to reduce gaming by ensuring that school administrators do not inflate their scores by testing only those students whom they believe will perform well. Other scenarios for incomplete tests include: scores missing due to absence; students not taking both the multiple choice and the open-ended parts of the test; students not having met the publisher’s criteria for a serious attempt at the test; and students having been excluded from last spring’s testing because of special education or language minority status. Students who are classified as severely and profoundly impaired, as trainable mentally retarded, or are in the lowest level of English proficiency are exempted from testing and are not counted in the untested category. The policy for students classified at the next level of English proficiency is that they should be tested with appropriate

UW


accommodations, unless the teacher feels that the testing would be harmful to the child or the student enters the district after December 1. These students and those with mild disabilities are counted as untested if they do not take the test and students with mild disabilities are to be tested at their age-appropriate level with accommodations unless their individual education plan indicates differently. A school may decide to give a disabled student an out of grade level test, but this disqualifies the student and categorizes them as non-tested. The program’s method of calculating change in student achievement is intended to encourage the inclusion of all but a small proportion of students in the testing program. This goal may not have been well understood by everyone because there were concerns about possible manipulation of the results by excluding certain students or by deliberately keeping some students out of the testing program the first year. However, the program as a whole is intended to make the district accountable for the achievement of all students and not to “excuse” the district from improving the achievement of students with special needs, disabilities, or those with limited English proficiency. As noted earlier, the three SAT-9 tests in reading, math, and science comprise 60% of the PRI score. The district reported that many educators made focused efforts to improve their students’achievement on these tests. Despite these efforts to focus instruction to help students do better on the tests, there appeared to have been some concerns or lack of understanding about how these scores were used to calculate the overall index. One such controversy concerned the number of students considered not tested. In order to encourage the testing of all students regardless of ability the district made a

UW


policy decision to count all students categorized as “not tested” in each subject area as scoring zeros in calculating the overall score for a school. This discouraged schools from attempting to raise their performance on the SAT-9 by pushing low performers out. Yet, the “not tested” category is somewhat of a misnomer and implies that all of the students in that category were either absent or not included in the testing.

The district reported

that many of the students who were counted as not tested did participate in the examinations, but either did not complete all of the sub-tests for that subject area or for various reasons they did not receive a valid score on the test. The exact criteria applied varies by item type, but in general, to receive a valid composite score in each subject a child must: 1. Attempt either three out of the first six items, or any ten multiple choice questions AND get one correct answer; and 2. Attempt one open-ended question and be credited at least one point by the scorer. Some educators in the district believe that this policy unnecessarily penalizes schools with large special education populations whom they hypothesize had a higher tested, but invalid rate than other schools. The district indicated that many teachers expressed the frustrations their students experienced during the first year the tests were administered. However, the district also reported that other educators believed special education students are penalized if the system does nothing to encourage their inclusion in the testing process. Another way this issue could be looked at as it relates to the incentive program is that by including these students in the testing program from the outset, a school’s initial performance might indeed be lower than if the students were not

UW


included, however, in future years the school just needs to make improvement from that initial, lower base. (Refer to the “Evaluation Results” section for more discussion of this and related issues.) Reward System Philadelphia’s overall accountability program provides some form of rewards and sanctions for all levels of employees. Because the focus of this case is on the schoolbased performance award element of the accountability program, primary emphasis will be given to the reward system in that program. However, several other types of rewards and sanctions will be noted to demonstrate how the school-based performance award program is integrated into other elements of the overall accountability program. School-based performance award program Under the Performance Responsibility Program, individual schools may receive cash awards based on their performance and the extent of improvement demonstrated over the two year cycle. Schools that exceed their targets will receive public congratulations and an award of $1500 for each teacher and $500 for each other staff member. Schools that meet but do not exceed their targets are publicly recognized for their accomplishments but receive no award. Schools which improve beyond their baseline (1995-96) scores but fall short of their targets receive help from a team of educators who will review school information, assess school resources, and help find ways for schools to improve. Unlike some school-based performance award programs that grant awards to individual teachers, Philadelphia’s program grants the award to the entire school. Schools that receive an award are allocated a total amount that is available over a two-

UW


year period and may spend up to half of the amount the first year. The amount of money an eligible school receives is based on the number of staff members in the school’s staff allocation for the second year of the two-year cycle; for the first cycle, the staff allocation for 1997-98 was used to determine the amount of a school’s award. For teaching staff that are less than full-time the amount is pro-rated by the teacher’s full-time equivalency; for non-teaching staff the amount is pro-rated based on the number of hours the person worked at the school that year. Other than the prohibition on using the funds for staff bonuses the schools have wide latitude in deciding how the funds will be used. The decision on how to spend the money is made by the School Council with the general principle that the money must be spent on initiatives that will help the school continue to improve instruction and student achievement. In schools where there is no School Council, the decision on how to use the funds must be made by the principal, staff representatives and parents. Their involvement in the decision-making process is documented by their signatures on the form the principal submits to request release of the funds to the school’s budget. The forms must be received by the district before any funds are released to a school’s budget. The forms include detailed unit cost information for specific staff functions and other types of expenditures such as overtime, substitute costs, and professional development costs. The cost information helps the schools make informed decisions about how to use their award money and also provides a record of how the bonus funds were spent by the school. As regards sanctions for schools that do not achieve the specified improvement goals, to some extent the failure to meet the targets and to therefore miss the public recognition and bonus can be considered a sanction. The more meaningful sanction,

UW


however, is probably that schools that continue to decline can be reconstituted and at least 75% of the school’s staff replaced if a school fails to meet its short-term goals for two consecutive two-year accountability cycles. One final program element, the School Support Process, should be noted. The purpose of the program is to provide participating schools with intensive support and to monitor their ongoing progress in order to catch schools that appear to be on the road to decline and thus to prevent them from falling into sanction status. Each participating school is assigned a School Support Team, which is chaired by a district leader outside the school and the cluster and includes district staff, cluster staff, and parent and union representatives. The Team identifies the school’s areas of strength and need and makes specific recommendations to the school, cluster and central office as to what is needed to support student learning, including identification of progress milestones. The 13 schools participating in this program the first year were those that had a decline in Performance Index score and an increase in the proportion of students scoring below the Basic level on the SAT-9. The second year 12 schools were included, of which six were schools that had participated the previous school year and did not meet both parts of the Performance target. The other six were schools that declined on one of the performance targets; either the Performance Index fell or the percentage of students scoring below basic increased. At present, the district has the resources to support 12-15 schools as the program currently is designed (Office of Standards, Equity and Student Services, 1998). System-wide rewards and sanctions A related part of the overall accountability program is that the superintendent and his Cabinet are held accountable for improving student achievement through a set of

UW


yearly performance goals. System-wide SAT-9 scores account for 50% of their performance and their salary increases and bonuses are tied to these targets (Luhm, Foley and Corcoran, 1998). In addition, if system-wide achievement falls the superintendent and Cabinet can be penalized up to 5% of their annual pay. Cluster leaders are held accountable in a similar manner: their annual bonus and increases are directly tied to student achievement within their cluster, student achievement systemwide, and specific goals agreed on between the cluster leader and the Superintendent. Thus, for the top district management the progress (or lack thereof) towards the student achievement goals can have a personal, financial and relatively immediate impact. Program Funding For the first accountability cycle, $11,150,000 was allocated over two years. At $5,575,000/year in the operating budget, it represents .37% of the operating budget. This amount is significantly less than the 1% of budget amount often suggested by the CPRE Teacher Compensation Project as a possible guide. Awards totaling the full budgeted amount were granted based on the results of the first accountability cycle. It should be noted, however, that although the funding for the school-based performance award program in Philadelphia is not part of the $150 million created by the Annenberg Challenge, the Annenberg funds were undoubtedly influential in the individual school’s ability to meet the student achievement goals. In particular, the Annenberg funds were used to create the Children Achieving Challenge, which provides a wide range of technical support to the district and its partners. Thus to some extent the district’s direct costs for the school-based performance award program were largely the

UW


cost of the awards themselves, with many of the related costs, e.g., technical assistance to schools, supported at least in part by outside funds. Program Enablers As noted previously, the explicit shift to an accountability system for Philadelphia’s education system began in 1995 as part of the comprehensive reform plan outlined in the district’s strategic plan. However, as noted in the Background section, the district had begun various types of reform efforts even prior to the new superintendent’s arrival and the implementation of the strategic plan. Because the strategic plan outlined a comprehensive reform strategy of which the Performance Responsibility Program was but one element, it implicitly included many enablers that support the accountability program. The district-specific standards and curriculum resources mentioned previously were two types of supports for the program. Other supports are summarized briefly below. The Office for Best Practices was created with responsibility for identifying and disseminating information on current best practices in curriculum and instruction. Twenty-two (22) geographically based clusters were created; each cluster includes 8-15 elementary and middle schools and a comprehensive high school. The clusters help support the decentralization of the district’s management structure and to reinforce the concept of smaller learning communities within the district and within schools themselves. Each cluster has staff to provide leadership and support for reform as well as an Equity Coordinator. The Equity Coordinator works in collaboration with the cluster team to ensure that all students have equal access to educational opportunities such as

UW


those mandated by law and program such as Title I, ESOL/Bilingual, and Desegregation (CPRE, 1998; Philadelphia Schools, 1999). Summer professional development institutes for teachers were provided and additional professional development opportunities provided throughout the year. Decisions for specific professional development opportunities became a school-level responsibility, with brokering and support at the cluster and district level. Family Resource Center Networks and associated staff were created and housed in the high schools to help forge connections between the families, the schools, and the social service system. More school nurses were authorized and placed in the schools on a full-time basis; child-care centers for school-age parents were created. These efforts and others were part of a broader strategy to have school financial and human resources work in unison with families, social services agencies and the community as a whole to help solve the social, medical, and financial problems that work against children’s learning (Cohen, 1996; CPRE, 1998). More recent reform measures include full-day kindergarten for all five-year olds; a 300% increase in computers for a computer:student ratio of 1:10; 15,000 new student volunteers, and the establishment of small learning communities in all schools. The 1994 contract with the teachers’union included important new language that allowed the district to reconstitute schools that were continually low performing (Bradley, 1995). The district attempted to reconstitute two schools in 1997, although an arbitrator ruled that summer that the proper procedure had not been followed and the reconstitutions were voided. Nonetheless, that action resulted in the district and the union reaching some alternate agreements on seniority and transfer and apparently

UW


moving closer to a conceptual understanding that drastic changes might be needed to bring about the needed improvements to student achievement (Reinhard, 1997). Private and philanthropic funds have helped the district implement some of the reform plans that it otherwise might have been unable to. In addition, prominent business leaders have worked with the district to identify administrative and other types of changes that could save the district money. Although the district’s strategic plan is premised on the ability to provide and maintain these and other enablers, chronic financial issues have made it difficult for the district to carry out all of its plans and have threatened some of the initiatives already in place. The district has been engaged in a longstanding battle with the state of Pennsylvania over the state funding level; several lawsuits have been filed by the district against the state but thus far the district has not won any of them. Loans from two banks in June of 1998 ensured that the district’s budget would balance (White, 1998). Nonetheless, a report prepared for state lawmakers on the district’s financial situation suggested that the district could not sustain its operations in the long run and that projected expenditures would fall far short of the district’s revenues (Blair, 1998). In summary, the district has planned a comprehensive set of enablers that would support improved student achievement. Many of those enablers are already in place and others are pending. However, due to current and projected future financial concerns, it is uncertain whether all of the supports can be sustained in the long run. Evaluation Results The evaluation of the strategic plan has been ongoing since the program’s inception in 1995. Evaluations were conducted by the Consortium for Policy Research in

UW


Education (CPRE) from the University of Pennsylvania, as well as Research for Action (RFA), a non-profit organization engaged in education research and reform. In addition, an external advisory panel was appointed in 1998 and has provided additional review and analysis of how the accountability program is working. Highlights of these two formal evaluations are noted below, as well as certain concerns that were identified by the community. Finally, the results of the first accountability cycle are reported. CPRE-Penn Evaluation The following findings were taken from the evaluation of the program’s second year, 1996-97 (CPRE, 1997). The evaluators recommended that an expert panel should be appointed to review the PRI, monitor it over time, and advise the district. This would help district officials prove the quality of the system and build confidence in the results. It might also reassure teachers that they are not being treated unfairly. In addition, the district should put more effort into explaining the components of the PRI. For example, although only 51% of the teachers surveyed reported understanding the purpose of the PRI, 63% of those teachers believed that it had the potential to benefit their students. These data suggest that if district officials were to make a greater effort to educate teachers about the purpose of the PRI more teachers might see the potential benefit. Other recommendations included changing the calculation of the staff attendance variable so that long term illnesses would no longer be included. The qualitative data suggested that this might improve teacher and administrator attitudes toward the index. The SAT-9 categories (advanced, proficient, basic, and below basic) should not be used for the non-cognitive indicators in the accountability index because there is no empirical basis for the cut points used to assign schools to these categories. Their use eliminates

UW


much of the actual variation, yet exaggerates the importance of minor differences among schools. With respect to the academic areas, long-term targets might be based on state or national data and interim targets could be based on reasonable progress toward these goals. The actual changes in performance could be used in the index. To that end, a local panel of experts, including teachers and the union should work with the SAT-9 test publisher to review the alignment of the “revised” SAT-9 with Philadelphia’s standards. In addition, school district officials should move with “due haste” to pilot additional student performance indicators that can supplement the SAT-9, such as portfolios and course exams. The evaluators’final in-depth analysis concluded that new programs often take more than two years to produce effects or increase achievement. Thus, it follows that a school that has adopted an appropriate course of action and is working hard to implement it may fail to reach its numerical target. For this reason, when district officials publicly identify schools as “low progress,” they should include information about action taken to improve performance. Community Concerns Between the 1995-96 and 1996-97 school years Philadelphia’s public school students significantly increased their achievement on the SAT-9 test, which led to increases in the overall school scores on the PRI. Ninety-two (92) schools reported more than a 5-point gain at the basic level or above, and over one-half of those had more than a 10-point gain. The superintendent and others viewed these improvements as a major

UW


achievement and as affirmation that the Children Achieving reform agenda is a sound one (CPRE, 1997). However, some board members, press and teacher union representatives questioned the legitimacy of the increases. They variously argued that the scores increased simply because more students took the test, because the baseline was very low, or because of the methods used to calculate the index, e.g., testing different students each year and the lack of specific controls on testing accommodations for special education, disabled and limited English proficiency students. These questions were raised publicly when the superintendent met with the school board in January 1998 and have continued as chronic concerns, especially among teachers. These were reasonable questions to ask about the system, and the external advisory panel has since then considered most of those concerns (see following section). External Advisory Panel As recommended by the CPRE-Penn evaluators, an external advisory panel was appointed to review assessment and accountability practices in Philadelphia, with three specific objectives in mind: 1) reviewing the extent to which the practices promote the strategic plan objectives; 2) providing guidance to promote continuous improvement; and 3) providing advice on whether or not and how the practices meet professional practices for assessment and accountability (Porter, 1998). The panel has five members and is chaired by Andrew Porter, a professor of education psychology at the University of Wisconsin-Madison and the director of the Wisconsin Center for Education Research. Other panel members are experts on accountability and assessment issues from several higher education institutions and research programs. Representatives from the school

UW


district participate in the meetings to provide information and respond to questions from the panel. The panel met three times in the 1998-99 school year and a summary was developed after each meeting to report the Panel’s deliberations to the Board of Education. The panel will continue to meet in the next school year to continue to review issues brought to them by the district as well as new issues that arise as the panel continues its work. The panel identified many elements of the accountability program that it views as strengths. The strengths include: • The general goal and approach, noting that if all 10 components of the strategy are accomplished there will definitely be a marked improvement in the quality of education provided to students in the district; • The strategy of putting different pieces in place over time rather than all at once is appropriate for the district; • The commitment to testing all core academic subjects rather than only one or two is appropriate; • Multiple parties (teachers, students, parents, community) will be held accountable; • The district is aligning its assessments with the district’s standards; • The program shows appropriate concern for providing accommodations to students with disabilities or limited English proficiency, within the broader goal of including and being held accountable for the performance of all students; • The strategy for getting all but a few students to participate in the accountability system seems promising and already seems to be having the intended effect;

UW


• More than one test will be used for the student accountability portion, rather than relying solely on one test to determine a high stakes outcome; • The establishment of incremental improvement levels within the “below basic” performance category is appropriate and allows the progress that schools make with the lowest-performing students to be recognized (A.C. Porter, personal communication, October 21, 1998). The panel has reviewed a number of aspects of the accountability program ranging from specific technical details to broad policy decisions. Several that are most germane to this case are summarized below. • Reliability of the school performance index: Concerns had been expressed about the stability with which schools are placed into the categories for reward, help, and reconstitution. The panel noted that the consistency will be improved in the second accountability cycle because school performance will be based on two adjacent grade levels, rather than the single grade level used in the first accountability cycle. The panel also recommended that the district and the panel continue to study this issue. (A. C. Porter, personal communication, October 21, 1998). • Treatment of nontested students: A primary concern in this area was the potential of artificially inflating the gains in the number of students tested by deliberately limiting the number of students tested in the first year of the accountability cycle and then increasing the number in the second year. The panel did not find evidence that such gaming had occurred. In addition, it noted that even if such gaming had occurred, it would only work in the first year because all students tested in the

UW


second year would be included in the baseline for the second accountability cycle (A. C. Porter, personal communication, January 22, 1999). • Accommodations of disabled, special education and limited English proficiency students: The panel believed that the district’s set of accommodations for these students seemed reasonable. It further stated that if the guidelines were internalized throughout the district they would lead to excellent procedures. As noted previously, this area had been controversial, with the undercurrent seemingly that there were no guidelines in place and that potentially too many opportunities for exclusion of these students existed and could be manipulated to improve a school’s performance (A. C. Porter, personal communication, January 22, 1999). • “Gaming” concerns in general: A major question posed by the Board of Education to the panel was whether the accountability system was set up and being administered in a way that offered opportunities for serious manipulation by school personnel and/or students. The panel concluded that if there had been any serious gaming problems they were a thing of the past. Regardless, data at the system level showed that even in the first accountability cycle there was positive movement both in the number of students tested and the student achievement levels. The panel also observed that the opportunities for gaming in the future are few, if any (A. C. Porter, personal communication, April 19, 1999). • Communication about the program: The panel noted that the accountability program is not particularly complicated and is based on reasonable pieces of school information. The panel would like to know more about how and what information

UW


about the program is being communicated to the public (A. C. Porter, personal communication, October 21, 1998). Among the issues yet to be reviewed by the panel include the reasonableness of the student performance target of 95% proficient by the end of 12 years. It characterized that target as “enormously challenging” and further questioned whether a straight-line performance projection was appropriate. The panel is scheduled to begin meeting again the fall of 1999 and will continue to consider issues relating to the accountability and assessment program. The consideration of accountability program goals and details by an entity with no connection to or vested interest in the district should help build support for the program by providing an objective review forum (A. C. Porter, personal communication, October 21, 1998). First accountability cycle results The first two-year accountability cycle was completed at the end of the 1997-98 school year. The initial progress indicated at the end of the first year of the two-year cycle generally was sustained and improved upon significantly. Schools received awards if, by June 1998, they exceeded two-year performance targets pre-set based on the 199596 school year. Of the 249 schools included in the accountability program, 145 received awards totaling $11,150,000, with up to half of the award amount available for expenditure in 1998-99. The amount each school received was determined based on $1500 per teacher and other professional staff and $500 per para-professional and noninstructional staff, pro-rated on the basis of full-time equivalency. As noted previously, the awards could not be used for staff bonuses but other than that restriction the individual school had considerable latitude on how to spend the money. According to the

UW


district there was considerable overlap between the schools earning awards under the state’s two accountability programs (one for improvement in state test results and one for improvement in attendance) and those earning awards under the district’s program. Of the 53 schools that earned achievement awards from the state, all but one made substantial progress in the district’s accountability system and 41 of them also earned district awards. Evaluation Summary The entire reform agenda, including the accountability and incentive program, were controversial from the time Hornbeck was hired. Nevertheless, there has been continuous progress to implement all elements and student achievement appears to be improving. In fall 1998 the Philadelphia school board with strong support from the current mayor voted to extend Hornbeck’s contract until August 2001 despite some protests from the union, the city council, and Democratic mayoral contenders. In contrast to those who were unhappy with his performance, Hornbeck’s supporters praised his efforts to focus public schools on the mission of improving student achievement, of which the school-based performance award program is one mechanism (Keller & Olson, 1999). Thus, it is likely that Philadelphia’s program will continue at least for the duration of Hornbeck’s tenure as superintendent which will allow at least one more accountability cycle to be completed and the continuing progress of Philadelphia’s students towards the long-term student achievement goals evaluated.

UW


Conclusion Philadelphia’s school-based performance award program is part of a comprehensive reform package, thus the program’s separate and direct effect on student achievement is difficult to isolate and it is arguable whether it would be appropriate to even attempt to do so. Many of the enablers that contribute to the success of the award program are appropriately intended to improve student achievement regardless of whether the results are tied to a formal performance award program. Finally, Philadelphia’s program is relatively new, with only one complete award cycle having been completed. There has been improvement in the achievement of Philadelphia’s students, including large increases in the number of students being tested, but the extent to which it will be sustained obviously cannot be predicted. Nonetheless, several observations can be made that other jurisdictions that are considering a school-based performance award program may want to consider. First, Philadelphia’s accountability program is part of a comprehensive education reform program that includes changes to the district’s administrative and decision-making structure, as well as specific curricular and instructional changes. As such it is not the sole or even the primary, means by which the district intends to improve student achievement, which we believe is the appropriate role for such a program. The performance award program was created to complement and support the district’s overall reform efforts with their emphasis on assessment and accountability, rather than to be the driving force of the reform effort.

UW


Second, Philadelphia’s context appears to pose some unique challenges that could influence the success of its performance award program. As noted previously, David Hornbeck made his acceptance of the superintendent position contingent on being able to implement an accountability program. Thus, although the community as a whole, as well as the teachers and administrators were involved in the design process, the initial impetus for the program did not necessarily come from within the district and the community itself. In addition, there was a relatively short time frame in which the program was designed and implemented, with all but a few schools participating from the outset. In retrospect, a pilot year might have allowed more support for such a program to solidify among all key stakeholders as well as for some of the more controversial and complex program details to have a public airing prior to the full implementation. However, given the low level of student performance in the district, it could just as easily be argued that speedy implementation was needed. Third, although the cost of the rewards provided as part of the Performance Responsibility Program was not that high, the costs of the enablers related to the education reform program were significant. These costs must be considered in the context of a district with chronic and longstanding financial concerns which regularly created a situation where the district had to cut existing programs, to say nothing about embarking on new programs. The presence of external funds to support the broader accountability program was undoubtedly an important factor in the schools’ability to succeed and the ability of the district to sustain that level of funding may influence the long-term success of the program.

UW


Fourth, it appears that the labor-relations climate between the district and the teachers’union is not one that could be characterized as “cooperative.” The union leadership does not appear to be supportive of the overall reform approach and does not believe it will succeed. In such a situation, it will be difficult for teachers to have a strong sense that they will be able to meet the improvement goals, or that there is even a reasonable expectation that they can meet the goals. The union leadership is not in complete agreement with the path the district is taking to education reform, with the strongest area of disagreement seemingly the accountability program. However, the union and the district do share some elements, such as lower class size, importance of maintaining discipline in the schools, etc. Nonetheless, without the full support of the involved parties the current reform effort, including the school-based performance award program, faces many challenges. Finally, Philadelphia’s accountability program is relatively new, with only one complete two-year award cycle under its belt. Thus, regardless of the challenges it has faced or the level of improvement to student achievement that remains, it must be acknowledged that it is still early in this particular education reform effort. It is probably not reasonable to expect that a large urban school district with a long history of poor student achievement and ineffective reform efforts can turn around a generation or more of problems in just a few years. Rather, the district’s progress and the role of the schoolbased performance award program in this progress, should be evaluated over a longer period of time.

UW


References Bradley, Ann (1992). New arrangements: reforming Philadelphia’s high schools from within. EdWeek [On-line]. 18 November. Available: http://www.edweek.com. Bradley, Ann (1997). Teachers: the reluctant recruits. EdWeek [On-line]. 19 February. Available: http://www.edweek.com. Cohen, Deborah L. (1996). Pencil me in. EdWeek [On-line]. 31 January. Available: http://www.edweek.com.

Consortium for Policy Research in Education (1998). A second-year evaluation report of Children Achieving: Philadelphia’s education reform; executive summary 199697. February. Phildelphia: author. Keller, Bess & Olson, Lynn (1999). Philadelphia extends Hornbeck’s contract for 2 years. EdWeek [On-line]. 10 February. Available: http://www.edweek.com. Luhm, Theresa, Foley, Ellen, and Corcoran, Tom (1998). The accountability system: defining responsibility for student achievement. April. CPRE: authors: Manzo, Kathleen Kennedy (1996). Philadelphia plan links student achievement, teacher pay. EdWeek [On-line]. 30 October. Available: http://www.edweek.com Miller, Julie A. (1993). Philadelphia embraces whole-school approach wholeheartedly. EdWeek [On-line]. 4 August. Available: http://www.edweek.com. Odden, Allan, Heneman, Herbert, Wakelyn, David J., and Protsik, Jean (1996). Schoolbased Performance Award Designs: A Case Study. October. University of Wisconsin-Madison. Office of Standards, Equity, and Student Services (1998). School Support Process: Interim Report. 12 February. Philadelphia: School District of Philadelphia

UW


Partnership for Reform (1996). Professional Responsibility Issue. Straighttalk About Philadelphia School Reform. Philadelphia: author Philadelphia Schools (1999). Supports for reaching the target. [On-line]. 16 June.. Available: http://www.philsch.k12.pa.us. Reinhard, Beth (1997). Arbitrator rejects overhaul plan for 2 Philadelphia schools. EdWeek [On-line]. 6 August. Available: http://www.edweek.com. Schmidt, Peter (1994a). Philadelphia schools shortchange minorities, court rules. EdWeek [On-line]. 16 February. Available: http://www.edweek.com. Schmidt, Peter (1994b). Philadelphia leaders join call for overhaul of schools. EdWeek [On-line]. 28 September. Available: http://www.edweek.com. Schmidt, Peter (1994c). Hornbeck setting sights on a new course for Philadelphia. Edweek [On-line]. 23 November. Available: http://www.edweek.com. Schmidt, Peter (1995). Hornbeck to push ahead with Philadelphia reforms. EdWeek [On-line]. 15 February. Available: http://www.edweek.com Sommerfeld, Meg. (1995). $50 million Annenberg Grant expected for Philadelphia. EdWeek [On-line]. 1 February. Available: http://www.edweek.com.

UW
