Organizational Report Cards - Wiley Online Library

Download PDF
14 downloads 0 Views 126KB Size Report
Comment
Despite a recent spate of organizational report cards, rela- tively little is known about their impact on consumers ' choices or public policy. This study identifies 32 ...
Charles K. Coe James R. Brunet North Carolina State University

Organizational Performance

Organizational Report Cards: Significant Impact or Much Ado about Nothing?

Charles K. Coe is a professor in the Department of Political Science and Public Administration at North Carolina State University. His current research focuses on budgeting, financial management, and performance management. E-mail: [email protected].

Despite a recent spate of organizational report cards, relatively little is known about their impact on consumers’ choices or public policy. This study identifies 32 report cards that compare government performance across states in a variety of policy domains. These report cards fall into four categories according to their issuer: governments, commercial enterprises, academics, and advocacy groups. Government-generated report cards are directed at improving consumer choice and enhancing service quality. Commercial enterprises seek to increase profits and readership. Academics generally take a value-neutral approach, looking to stimulate public policy debate. Public interest groups, think tanks, and foundations indirectly measure public policy impact by the amount of media attention generated.

James R. Brunet is an extension assistant professor in the Department of Political Science and Public Administration at North Carolina State University. He is the author of Drug Testing in Law Enforcement: Social Control in the Public Sector (LFB Scholarly Publishing). E-mail: [email protected].

A

n “organizational report card” has been defined as a regular effort by an organization to collect data on two or more other organizations, transform the data to assess performance, and transmit the information to an external audience (Gormley and Weimer 1999, 3). As such, they have early origins. In 1845, the Boston School Board administered a uniform test to elementary school students to gauge performance. In 1861, Florence Nightingale persuaded London hospitals to publish mortality statistics. In 1917, during the Progressive Era, the American College of Surgeons issued a scathing report on the performance of 692 hospitals. Later, the federal government began to require hospital report cards as a means of slowing the rising Medicare and Medicaid health care costs (Gormley and Weimer 1999, 44). Despite these early efforts, report cards remained fairly uncommon until recently, when a spate of report cards were implemented in a wide range of policy areas, including the issued environment, K–12 education, fire protection, health and social welfare, economic development, state management, and fiscal policy. Table 1 lists 32 report cards that compare performance across states.1 These are all the interstate report cards the authors know of, but doubtless more exist. Of the 32, 26 emerged after 1990 and 13 since 2000. Table 1 90

Public Adminisration Review • January | February 2006

does not include a host of intrastate report cards, such as those evaluating public schools, nor are international report cards included, such as the Corruptions Perception Index issued by Transparency International, which ranks countries’ corruption levels. Gormley and Weimer (1999) argue that a good report card should be valid, comprehensive, comprehensible, and relevant; have reasonable preparation costs; and be functional, which they define as encouraging a desired behavior. The bottom line, of course, is whether a report card accomplishes the issuer’s end. This article focuses on the functionality of report cards. Previous research has suggested that report cards generally serve one of two purposes (Gormley and Weimer 1999). Micro-level report cards facilitate consumer choice. For example, state-required hospital, health maintenance organization (HMO), physician, and Medicare report cards assist consumers in making health care decisions. Similarly, federally required airline punctuality and safety report cards are customer driven. Macro-level report cards, on the other hand, are designed to influence public policy. Finally, some report cards fulfill both purposes; for example, college report cards influence both the school choice of parents and students (Monks and Ehrenberg 1999) and educational policy decisions. Considerable research has centered on the functionality of micro-level report cards in the areas of health care (Gormley 1998, 2003), public education (Clotfelter and Ladd 1994, 1996), and toxic emissions (Hamilton 1995; Konar and Cohen 1997; Vasu 2003). This article suggests a new way of assessing functionality that is based on four types of report card issuers: governments; commercial enterprises; academics; and public interest groups, think tanks, and foundations. Although report cards may ultimately affect consumers’ decisions and public policy, these issuers tend to assess functionality more narrowly.

Table 1 Report Cards Policy Area

Report Card

Organization/Author

Environment

Toxic Release Inventory Total maximum daily load program evaluation 100 Worst Corporate Criminals of the 1990s State of the Air Quality Counts No Child Left Behind Research: Doctoral programs America’s Best Colleges Student Engagement Measuring Up

Public Interest Research Group National Wildlife Federation

1992 1997

Mother Jones

1999

American Lung Association Education Week Federal government National Research Council U.S. News and World Report Indiana Center for Survey Research National Center for Public Policy and Higher Education The Center, University of Florida Technology Review Insurance Services Office Annie E. Casey Foundation United Health Group United Way of America Stephen Jencks et al. Morgan Quitno Press Federal Administration for Children, Youth, and Families Health and Human Services National Women’s Law Center and Oregon Health and Science University Corporation for Enterprise Development Small Business Survival Commission Progressive Policy Institute Beacon Hill Institute Cato Institute Maxwell School and Governing magazine

2000 1996 2002 1982 1983 2000 2000

1986 1997 1999 2001 1992 1999

National Park Trust

2000

K–12 education Higher education

Fire protection Health/social welfare

Top Research Universities University Research Scorecard Fire Protection Ratings KIDS COUNT State Health Report Card State of Caring Index Medicare beneficiaries Safest, Smartest, Healthiest Foster care Home health care Women Smoking

Economic development

State management/policy

Development Small Business Survival Index New Economy Index State Competitiveness Report Governors’ Fiscal Conservatism Management: financial, capital, human resources, managing for results, information technology Land conservation

Methodology A literature review and Internet search were conducted to identify report cards and ensure that the identified report cards met our operational definition of a report card. Twenty-seven organizations were found to issue report cards; these organizations were grouped into the issuer types shown in table 2. Interviews were then conducted with these organizations, except the federal and state governments and four commercial enterprises: U.S. News and World Report, Mother Jones Magazine, Technology Review, and the Insurance Service Office. Respondents were asked to comment specifically about their impactmonitoring efforts in three areas: media coverage, Website visitation, and policy influence. We will begin with a description of the report cards, followed by a discussion of how report card issuers assess functionality. Issuer Types Governments

Governments issue two types of report cards: those with and without funding consequences if standards are not met. A recent and increasingly controversial example of the former comes from the federal No Child Left Behind Act passed in 2002, which mandates

Year Begun

2000 2000 1916 1989 1989 1990 2000 2002 2003 2003 2003

that states administer end-of-grade reading and math exams. Public schools that receive federal Title I funds aimed at low-income students (about one-half of all schools) and fail to meet federal standards for two consecutive years must offer students the choice of another school, if one is available. Schools that fail to meet standards for three consecutive years must offer extra services to low-income students. After six years of noncompliance, school systems must implement an alternative governance plan. Another federal report card that will have funding teeth was issued in 2003 by the Administration for Children, Youth, and Families. This report card measures state performance with regard to foster children’s safety (two measures), permanence of living situations (two measures), and well-being (three measures). Initially, 32 states were graded as pass–fail on each of the seven measures. After all of the states are graded, states will lose federal funding if they do not correct identified problems. Governments also issue report cards in which compliance with the measured criteria is optional. Organizational Report Cards

91

Table 2 Types of Report Card Issuers Type

Issuer

Report Card

Governments

Federal government

No Child Left Behind Foster care Home health care Health-related K–12 schools America’s Best Colleges 100 Worst Corporate Criminals of the 1990s University Research Scorecard State Management: financial, capital, human resources, managing for results, information technology Quality Counts State Health Report Card Safest, Smartest, Healthiest Fire Protection Ratings Student Engagement Top Research Universities Research: Doctoral programs Medicare Beneficiaries Toxic Release Inventory

State governments Commercial

Academics

Public interest groups, think tanks, and foundations

U.S. News and World Report Mother Jones Technology Review Governing magazine

Education Week United Health Croup Morgan Quitno Press Insurance Services Office Indiana Center for Survey Research The Center, University of Florida National Research Council Stephen Jencks et al. Public Interest Research Group National Wildlife Federation American Lung Association National Women’s Law Center and Oregon Health and Science University National Center for Public Policy and Higher Education Annie E. Casey Foundation United Way of America Progressive Policy Institute Corporation for Enterprise Development Beacon Hill Institute Small Business Survival Index Cato Institute National Park Trust

Such report cards attempt to reduce the information asymmetry between service providers and consumers so that consumers can make better-informed decisions. For example, in 2003, the U.S. Department of Health and Human Services issued a national report card on the performance of home health agencies certified by Medicare and Medicaid. The report card measured patients’ improvement in mobility (4 measures), activities of daily living (4 measures), medical emergencies (2 measures), and mental health (1 measure). Performance was benchmarked against state and national averages for each measure. Likewise, some states have issued health report cards related to hospital, nursing home, and health maintenance organizations (Gormley 1998). In a somewhat different category, at least 35 states publish report cards on individual public schools (Gormley and Weimer 1999, 1). Although parents cannot readily vote with their feet, as they can with privately provided health care services, such report cards can encourage parents and other stakeholders to get more involved in school governance and related policy making. 92

Public Administration Review • January | February 2006

Total maximum daily load program evaluation State of the Air Women Smoking Measuring Up KIDS COUNT State of Caring Index New Economy Index Development State Competitiveness Report Small Business Survival Commission Governors’ fiscal conservatism Land conservation

Commercial Enterprises

Whiteman (1985) found that members of Congress were more likely to use policy reports addressing highly salient issues. Report cards likewise resonate with varying intensity for stakeholders. Clearly, an issue of pressing concern to parents and their children is where to attend college. Indeed, as the country shifts inexorably from a manufacturing- to a knowledge-based economy, college choice takes on added significance. The best-known college report card is the America’s Best Colleges Report Card, first issued by U.S. News and World Report in 1983.2 Mother Jones, another magazine that is available both on newsstands and to subscribers, issued a Top Environmental Criminals Report Card listing the top 100 corporations in order of the fines they paid for environmental wrongdoing during the 1990s. Of the top 100 firms, 38 paid fines totaling between $200,000 and $125 million. Three other magazines are not available on the newsstand but issue report cards to their subscribers. Technology Review, an enterprise of the Massachusetts

Institute of Technology aimed primarily at members of high-tech corporations, issues the University Research Scorecard, which ranks universities by income generated from technology transfer (6 items) and technological strength (4 items). Governing magazine issues biannual report cards on state governments’ financial (71 items), capital (41 items), human resource (24 items), managing for results (31 items), and information technology (34 items) performance. In Education Week, Editorial Projects in Education annually issues the Quality Counts Report Card, which evaluates performance regarding student achievement (14 items), standards and accountability (9 items), school climate (13 items), improving teacher quality (10 items), and resources (22 items). Morgan Quitno Press sells monthly reports of state statistical trends regarding poverty and social welfare, corrections, and annual reports, including State Rankings (13 items), Health Care Rankings (7 items), City Crime Rankings (3 items), State Crime Rankings (6 items), and State Education Rankings (5 items). The United Health Group, a publicly traded company offering health services to consumers, issues the State Health Report Card, which compares states’ performance regarding quality of life (5 measures), access to health care (4 measures), occupational safety and disability (2 measures), disease (3 measures), and mortality (3 measures). A different type of commercial report card is issued by the Insurance Services Office (ISO), a publicly held, for-profit company that rates fire departments to enable insurance companies to set premiums for homeowners insurance and fire insurance.3 The ISO evaluates fire departments’ water-supply systems, operational abilities, and alarm systems.4 Departmental ratings range from class 1 (most desirable) to class 10 (least desirable). Academics

Kingdon (1995) found that although academics rarely affect policy agenda setting, they can influence the discussion of policy alternatives. Two research groups have prepared report cards expressly to stimulate more discussion about higher education policy. Specifically, the report cards were designed to address and improve on perceived deficiencies in the U.S. News Best Colleges Report Card. Critics have validly argued that the Best Colleges Report Card unduly weights academics’ perceptions of quality rather than learning outcomes (Graham and Thompson 2001). Accordingly, the Pew Forum on Undergraduate Teaching and the Carnegie Foundation for the Advancement of Teaching engaged the Indiana University Center for Survey Research to prepare a biannual Student Engagement Report Card, which compiles the perceptions of freshmen and seniors

regarding their institution’s level of academic challenge, active and collaborative learning, students’ interactions with faculty, enriching educational experiences, supportive campus environment, and student learning. Likewise, redressing perceived deficiencies in the U.S. News report card, researchers at the University of Florida annually prepare a Top Research Universities Report Card (Lombardi et al. 2000). They limit their study to 79 research universities with at least $20 million in annual federal research expenditures that also rank among the top 25 universities on at least one of nine measures: total research expenditures, federal research expenditures, endowment assets, annual giving, faculty members in the national academies, faculty awards, doctoral degrees, postdoctoral appointees, and entering freshman SAT scores. Arguing that public and private universities operate in significantly different contexts, the researchers evaluate public and private universities separately (Lombardi et al. 2000). Moreover, the Top Research Universities Report Card neither uses reputational evaluation nor ranks institutions numerically, arguing that such a comparison would give a distorted sense of precision. In the area of health care, Jencks et al. (2000) evaluated the quality of Medicare services. This research team randomly sampled and analyzed up to 750 inpatient records per state for patients having acute myocardial infarction, stroke, and pneumonia; up to 800 records per state for patients with heart failure; and all claims for breast cancer and diabetes. Their findings were published in the 2000 and 2002 volumes of the Journal of the American Medical Association. Public Interest Groups, Think Tanks, and Foundations

Public interest groups, think tanks, and foundations issue many kinds of report cards directed at a wide array of stakeholders, including policy makers, service providers, consumers, the media, and their own members. Three environmental public interest groups issue report cards. First, the Public Interest Research Group ranks each state according to toxic discharges generated by firms (see http://pirg.org/reports/enviro/ waters98). In 1986, the U.S. Environmental Protection Agency (EPA), concerned about the chemical disaster in Bhopal, India, and a near disaster at a Union Carbide plant in West Virginia, mandated that prescribed firms must report toxic emissions annually to the EPA, which creates a Toxics Release Inventory. The Public Interest Research Group uses this inventory to prepare its report card. Second, the National Wildlife Federation (NWF) issues report cards on states’ plans to correct water Organizational Report Cards

93

pollution. The Clean Water Act requires states to prepare “303(d) lists” that summarize the percentage of waterways whose condition is good, threatened, fair, or poor, as well as total maximum daily load watershed-restoration plans for cleaning polluted waters. The NWF grades states’ performance on meeting the EPA’s minimum requirements (6 items), as well as public participation, preparing comprehensive lists, considering and prioritizing pollutants, and developing and implementing total daily maximum loads. Third, the American Lung Association annually issues its State of the Air Report Card on air quality in counties. The National Center for Public Policy and Higher Education, a nonprofit, nonpartisan organization dedicated to improving higher education performance, issues the Measuring Up Report Card, which grades states regarding student preparation (10 items), participation in higher education (3 items), college affordability (5 items), completion of the first year of college (3 items), and benefits to the state (5 items). Three organizations issue report cards related to health and well-being. The Annie E. Casey Foundation issues the KIDS COUNT Data Book, which ranks states’ performance regarding child well-being (10 measures); demographic change among children under 18 (1 measure); economic characteristics (4 measures); child health and education (4 measures); child care (6 indicators); juvenile justice (2 measures); and children’s access to telephones, computers, and the Internet (3 indicators). The United Way of America issues a State of Caring Index that evaluates states’ economic and financial well-being (6 items); education (8 items); health (9 items); volunteerism, charity, and civic engagement (6 items); safety (2 items); and natural environment and other factors (4 items). Finally, the National Women’s Law Center and Oregon Health Science University issue a Women Smoking Report Card that assigns grades regarding states’ progress toward achieving 11 performance targets set by the Department of Health and Human Services to reduce lung cancer among women. Four think tanks issue report cards. The Progressive Policy Institute, a think tank founded by the Democratic Leadership Council to promote the “third way” movement popularized by Bill Clinton and Tony Blair, publishes the New Economy Index, a report card measuring state performance regarding knowledge jobs (4 items), globalization (3 items), economic dynamism (4 items), digital economy (6 items), and innovation infrastructure (5 items). The Beacon Hill Institute of Public Policy Research issues the annual State Competitiveness Report Card, which ranks states’ fiscal policies, institutions, infrastructure, human resources, technology, finances, openness, 94

Public Administration Review • January | February 2006

domestic competition, and environmental policies. The Small Business Survival Committee, a lobbyist for small businesses, issues the Small Business Survival Index, which ranks states according to the taxes and other costs they impose on businesses. Finally, the Cato Institute, a conservative nonprofit research foundation that promotes limited government, individual liberty, free markets, and peace, issues the biannual Governors’ Fiscal Conservatism Report Card, ranking governors’ conservatism regarding state expenditures (four items), revenues (6 items), and taxes (7 items). The National Park Trust, a privately funded land conservancy, ranks 32 states by state park directors’ perceptions of how threatened their state parks are by development and increased traffic. Assessing Functionality Governments

The federal government issues report cards both with and without sanctions. For instance, the No Child Left Behind report card includes explicit sanctions for each year of noncompliance. Such report cards are more likely to induce improvement than voluntary report cards. Still, specific provisions of the No Child Left Behind Act may make performance assessment ambiguous. Each state has designed its own tests and passing thresholds. States that experience unusually high failure rates may lower their passing bar. To ensure grading consistency, the federal government compares states’ performance to their performance on the National Assessment of Education Performance standardized test, which is given to a random sample of students in each state. States issue health report cards that measure outcomes by the choices consumers make. Researchers have found evidence that state report cards can affect consumer choice. For example, from 1991 to 1992, deaths attributable to bypass operations in hospitals dropped 41 percent after New York State instituted public reporting of in-state hospital death rates, presumably because consumers began to make better decisions about their choice of medical services (Glazer 2000). Moreover, Mukamel and Mushlin (1998) found that the market shares of New York physicians and hospitals with better health outcomes grew. State education report cards may use the carrot, stick, carrot and stick, or neither. North Carolina, for instance, employs both carrots and sticks to influence school performance. Teachers whose schools exceed state standards receive cash bonuses, but failing schools face sanctions, including the removal of principals and takeover by the state until performance meets an expected standard. Report cards have improved school performance in North Carolina and Texas (Clotfelter and Ladd 1994, 1996).

Commercial Enterprises

Commercial enterprises such as Morgan Quitno Press and U.S. News have a clear measure of functionality: the sales of magazines or special reports that have report cards. Critics have claimed that U.S. News arbitrarily changes its criteria to stimulate sales (Gottlieb 1999).5 Not surprisingly, U.S. News demurs, claiming that its changes have improved the reliability of the results (Lombardi et al. 2000). In fact, most of the changes have been tune-ups, not overhauls (Thompson 2000). A notable exception occurred in 2000, when some schools experienced wide swings in their rankings. For instance, the California Institute of Technology rose from ninth to first. Responding to criticism that year, U.S. News readjusted its criteria the next year, and the same institutions that had risen dramatically fell markedly to their previous locations. Still, the U.S. News rankings inevitably change somewhat each year. The “best” university in the country typically changes and the top universities are reshuffled, stimulating reader interest and sales, which are sizable indeed. In 2000, though U.S. News’s overall magazine and advertising sales fell 21 percent, the Best Colleges issue sold 40 percent more copies than a typical issue (Kuczynski 2001). Though such publications have primarily commercial intentions, researchers use their findings to measure their impact on public policy. Monks and Ehrenberg (1999) found that U.S. News’s rankings affected the quantity and quality of students applying for admission to colleges. Darnall (2003) found that being ranked among Mother Jones’s “Top Environmental Corporate Criminals” best explained why firms voluntarily went through the costly and elaborative process of meeting the international standard for environmental management systems (known as ISO 14001 certification). She found that firms on the Mother Jones hit list adopted ISO 14001 certification 541 percent to 568 percent faster than firms not on the list (Darnall 2003, 19). Other magazines with less commercial interests make their report cards publicly available on the Internet. Although they are interested in stimulating subscriptions, these magazines also desire to influence public policy. The mission of Education Week is to improve public education; Governing magazine encourages the adoption of governmental best management practices; Mother Jones promotes social justice; and issuers of the Student Engagement Report Card want to enhance consumers’ choice of colleges. As evidence of their report card’s impact, issuers of the Student Engagement Report Card point to the fact that after two subsidized report cards, colleges now offset the cost of report card issuance by paying fees to have the survey conducted on their campuses. The Insurance Services Office likewise offsets its costs with fees, but it has a captive audience. Insurance

companies must pay the Insurance Services Office a fee to obtain the fire rating classifications of fire departments as well as commercial and industrial buildings, which are used to underwrite insurance policies. Academics

Academics measure report card functionality in two ways. First, they have the hard measure of the number and quality of publications of research related to report cards in books, refereed articles, reports and on the Internet. Most successfully, academics at the Maxwell School generated two books, three book chapters, three refereed articles, and numerous academic presentations from the state management report cards they helped Governing magazine to publish (see www.maxwell.syr.edu/gpp/publications. htm). Similarly, the Medicare Beneficiaries Report Card is published annually by the Journal of the American Medical Association. Academics such as those at Indiana University, University of Florida, and Syracuse University assess the impact of report cards by the number of visitors to their Websites. Public Interest Groups, Think Tanks, and Foundations

Measuring the impact of report cards issued by academics, commercial enterprises, and governments is relatively straightforward, whereas public interest groups, think tanks, and foundations have a more difficult time measuring the functionality of their report cards, partly because they have a wide range of stakeholders that includes policy makers, consumers, service providers, the media, and their own membership. The respondents indicated they do want to influence public policy or firms’ behaviors. One such direct measure is how legislators vote on key policy issues. At the federal level, 100 interest groups directly measure their success by scoring the votes of representatives and senators on particular issues. These interest groups publicize congressional scores based on individual voting records (Davidson and Olesrek 2004, 380). Although senators downplay the influence of scorecards on their voting or reelection, these measures likely affect the voting of House members facing close elections (Cochran 2003). State lobbyists also score telltale votes, but none of the surveyed organizations reported any direct link between their report cards and voting records. Instead, they measure functionality indirectly by counting Website hits, media coverage, references in governors’ state addresses, and state chapters created. All of the organizations counted visitors to their Websites. Issuers noted a spike in Website activity soon after the report card was posted. For example, immediately after the publication of its Safest Cities book, Morgan Quitno Press recorded more than one Organizational Report Cards

95

million hits on its Website (Morgan 2003); the site typically receives 10,000–15,000 visitors per day. Staff at the University of Florida (Top Research Universities Report Card) monitor Web hits by geographic region: 83 percent of Web visitors are from the United States, with significant online viewing coming from Asia (7 percent) and Europe (7 percent) (Reeves 2003). Measurement of media coverage usually depends on money available to measure coverage. Organizations with limited funds monitor print coverage with inhouse staff. The Progressive Policy Institute (New Economy Index), United Way (State of Caring Index), and Beacon Hill Institute (State Competitiveness Report) track newspaper and newswire stories using subscription-based databases such as LexisNexis. Others routinely search for press coverage using freely available Web browsers such as Yahoo and Google. Generally, there is little structured analysis pertaining to the quality of press coverage (i.e., accuracy of reports, positive versus negative slant, how many people are actually reached), just a simple count of hits. Organizations with greater financial resources track both print and television coverage more extensively. The KIDS COUNT report card exemplifies state-ofthe art print media tracking. Researchers track the number and type of print stories (editorials, Associated Press wire stories, and locally produced stories) carried in each state and determine the number of adults exposed to the report card using newspaper circulation figures, or penetration rates (Reynolds and O’Hare 2002). Although most people get their news primarily from television, only three issuers reported they have the resources or the will to hire a firm to track television and radio coverage. These exceptions are the Annie E. Casey Foundation (KIDS COUNT), the United Health Foundation (State Health Ranking), and the Maxwell School (State Management), which all use an outside contractor (Burrelle’s Information Services) to monitor television and, in some cases, radio coverage. Stories related to the 2002 release of the KIDS COUNT report card generated 460 television and 600 radio stories (Reynolds and O’Hare 2002). In addition to resource availability, the way an organization defines functionality can affect the extent of media tracking. For instance, the Cato Institute, which has ample financial resources, is only marginally concerned with media penetration and impact because its ultimate goal is simply to “make a point” (Edwards 2003). The institute thus publishes reports even when readership is small, simply to be on record on a particular issue. Perhaps the Cato Institute’s cavalier attitude toward media coverage is justified. Research indicates the 96

Public Administration Review • January | February 2006

effort and expense required to attract media attention may not be worthwhile. In a study of 250 federal policy makers, Kingdon (1995) found that only 26 percent mentioned the media as important to their work, and only 4 percent regarded the media as very important in how they set their own policy agenda. The policy makers ascribed their discounting of the media to the press’s short attention span for policy issues (Kingdon 1995, 58–59). State legislators likewise ascribed little influence to news reports and even less to editorials (Gray and Lowery 2000). Not to dismiss the media altogether, Kingdon surmised the media act as communicators within a policy community, magnifying policy trends and indirectly affecting public policy agendas (1995). To measure functionality more directly, the authors of the New Economy Report Card reviewed governors’ State of the State address messages to determine whether their report card was mentioned (Atkinson 2003). They found few references, which is borne out by other research. Analyzing 36 State of the State addresses delivered in 2002, Gormley found that although 56 percent of the governors compared their states statistically to other states, only six explicitly mentioned a particular report card (Gormley 2003). Two report cards reported a tangible result that they had encouraged. The KIDS COUNT report card, prepared by the Annie E. Casey Foundation, stimulated the founding of four statewide organizations to prepare similar report cards in California, Rhode Island, Illinois, and Maine (Kemper 2003). Likewise, local United Way chapters have devised their own “state of caring indices” for their communities. Design Strategies

Issuers consciously or unconsciously select alternative approaches to designing report cards. One approach is to use valid criteria and distribute grades reasonably. Another tactic is to set a very high bar for success and use unreasonably difficult criteria, ensuring a low rate of success. The approach selected will probably affect both media attention and the impact of the report card on the organization’s membership. Balanced Reporting Tactic In assessing the validity of a group of report cards, Coe (2003) found that some took extreme pains to design report cards that validly measured the construct in question and fairly assigned grades. Among the report cards discussed in this article, several used a balanced reporting approach: Quality Counts, Student Engagement, Measuring Up, KIDS COUNT, the New Economy Index, and State Management. These report cards used experts to select measures that were both objective and current, reflected program outcomes, completely and accurately

measured the construct, weighted measures according to their relative importance, and stated the rationale for the weights. Dramatizing failure tactic Balanced reporting may be less likely to gain media attention and energize the membership. Indeed, getting media attention, particularly for state policy issues, is difficult because most local papers do not have full-time state government reporters, and few television stations have camera crews that regularly cover state government (Gormley 1978). Moreover, media attention to state government has declined generally. From 1990 to 1998, print coverage of state government was down in 27 states, including large states such as New York, Texas, Florida, and Illinois (Allen and O’Brien 1999). The media are inclined to report stories they routinely cover, come from official sources, fit within dictated time frames, connect easily with social trends, and have dramatic plot possibilities (Bennett 1997, 184). Although report cards are routinely covered, come from official sources, fit within convenient time frames, and connect easily to social trends, some lack the most important element: dramatic plot possibilities. If the dramatic angle is not clear, reporters are less likely to report a story (Bennett 1997, 181). A negative slant dramatizes plot possibilities, which may explain why some report card designers establish grading systems that appear to guarantee high failure rates. Table 3 shows the four most frequently used grading systems, which vary with respect to dramatic impact, and several less frequently used systems. The A–F system is the most attention getting and easiest to understand. Similarly attention getting is a pass–fail system used in the Child Welfare and Women Smoking report cards. Simple to understand but with less dramatic impact is the ranking of state performance from 1 (best) to 50 (worst) based on a composite score of numerous measures. A variation of this method is to sequentially rank organizations (e.g., universities, corporations, or fire departments). The KIDS COUNT report card uses a valid but difficultto-dramatize grading system that ranks states on a variety of measures but gives no overall composite grade. In particular, the environmental report cards dramatize failure. For instance, the National Wildlife Federation sets its bar so high that in the most recent report card (issued in 2000), no state received an A; 6 received a B, 6 a C, 19 a D, and 19 failed. Moreover, this report card includes an unfair measure. Although the EPA requires that states only prepare total maximum daily load plans—not implement them— the NWF awards an A only to states that implemented at least 10 of their plans.

Likewise, the American Lung Association (ALA) does not indulge in grade inflation. In 2003, 8 percent of counties received an A, 6 percent received a B, 9 percent received a C, 4 percent received a D, and 58 percent failed. The ALA also harshly grades any violation. An EPA violation is considered to have occurred if ozone levels exceed 0.086 parts per million during an eight-hour period over more than 3.3 days within a three-year period. The ALA gives an F to any county if a single air-quality monitor exceeds the EPA standard. Thus, states and localities that voluntarily install more monitors than are required by the EPA to more closely track pollution levels are more likely to be penalized by the ALA grading system. Conservative think tanks have vehemently and appropriately argued that the ALA methodology is unfair (Clarke 2003; Cordato 2001; Schwartz and Hayward 2003). The libertarian Cato Institute unsurprisingly found that few governors were sufficiently fiscally conservative for its taste. Its 2003 grades were as follows: A, 5 percent; B, 26 percent; C, 21 percent; D, 38 percent; and F, 10 percent. Harshest of all, the Women Smoking Report Card gave the following grades in 2001: passing (0 states), satisfactory minus (1 state), unsatisfactory (9 states), and failing (40 states). Conclusion Report cards have become a common device for assessing the performance of governments, organizations, and individuals in a range of policy arenas. Governments, commercial enterprises, academics, and public interest groups, think tanks, and foundations are the primary issuers of report cards, with much of the recent growth occurring in the last-named category. The hoped-for impacts are numerous and vary by issuer type. Governmentgenerated report cards are often directed at increasing consumer choice (e.g., foster care, nursing homes, home health care) or improving service quality (No Child Left Behind, K–12 Schools). For-profit publishing houses seek to increase sales with their report cards. Research organizations generally take a value-neutral approach to their offerings, looking to provide unbiased information to a more academic audience. Rankings of top research universities and college student engagement fit this description. For these, the scoring criteria are widely accepted as valid. In direct contrast, public interest groups actively seek to influence public policy through their report cards. Environmental report cards are especially amenable to a distinctive report card design—the dramatic failure. Some report cards also serve as a forum for policy debate. On at least one issue, groups on the left and right sides of the political spectrum have promulgated dueling report cards (New Economy versus State Competitiveness and Small Business Survival). Organizational Report Cards

97

Table 3 Types of Grading Systems Grading System

Report Card

Organization

A–F

Total maximum daily load program evaluation State of the Air

National Wildlife Federation American Lung Association

Quality Counts

Education Week

Measuring Up Development

National Center for Public Policy and Higher Education Corporation for Enterprise Development

Governors’ fiscal conservatism

Cato Institute

State Management State Health State of Caring

Governing magazine and the Maxwell School United Health Group United Way of America

Safest, Smartest, Healthiest

Morgan Quitno Press

Small Business Survival

Small Business Survival Commission

New Economy

Progressive Policy Institute

State Competitiveness

Beacon Hill Institute

Land conservation

National Park Trust

Medicare beneficiaries

Stephen Jencks et al.

KIDS COUNT Research: Doctoral programs 100 Worst Corporate Criminals of the 1990s America’s Best Colleges

Annie E. Casey Foundation National Research Council Mother Jones

Top Research Universities

The Center, University of Florida

University Research Scorecard

Technology Review

Fire Protection Ratings

Insurance Services Office

Overall state ranking

States’ ranking on individual measures Ranking on a measure or measures

Other types Percentage of students following best practice Pass–fail

Student Engagement

Indiana Center for Survey Research

Child Welfare

Satisfactory minus, unsatisfactory, fail

Women Smoking

Federal Administration for Children, Youth, and Families National Women’s Law Center and Oregon Health and Science University

This research has important implications for both practitioners and academics. Public administrators should be aware that all report cards are not created equal. Some are based in good science and altruistic intentions; others attempt to promote a particular policy perspective at the expense of empiricism. Practitioners should consider the source and rating criteria when evaluating the usefulness of a report card. Interviews with report card issuers indicate they feel their efforts are effective, but the evidence is more anecdotal than scientific—which is where researchers can lend their assistance. We need to better understand the impact of these report cards on public policy, government performance, and consumer choice. Thus, future research should examine the following questions: Micro-level report card dissemination and interpretation. Report cards issued by magazines such as U.S. News and Mother Jones have been shown to influence the behavior of parents and firms, respectively. Such evaluations gain wide O

98

U.S. News and World Report

Public Administration Review • January | February 2006

public attention. On the other hand, other micro-level report cards have received relatively little public exposure or are complicated to understand. For instance, few state governments make data on child care facilities’ performance available to parents (Gormley 2003, 19). Likewise, report cards on hospitals, HMOs, and nursing homes only slightly affected these organizations’ market shares, probably because these cards were inadequately disseminated to patients (Gormley and Weimer 1999, 120). Moreover, even if they are well disseminated, some report cards need to be interpreted for consumers. Focus groups reported they could not understand the Toxic Releases report card found on the Public Interest Research Group’s Website unless an expert interpreter clarified its findings (Vasu 2003). Further research should investigate best practices for disseminating and explaining micro-level report cards to consumers. O Functionality of macro-level report cards. Issuers principally measure their impact on public

policy indirectly through media impact, which is the method used by think tanks generally (Abelson 2002). Though no direct evidence has been presented that macro-level report cards affect public policy, they nevertheless may do so. Coggburn and Schneider (2003), for instance, found that states’ report card management grades related to managers’ policy priorities. Following this lead, future research should construct models to determine whether report cards explain variance in state government performance in areas such as the environment, health care, and economic development. This will be no easy task, however, as it is difficult to disentangle the impact of report cards from other policy-influencing initiatives such as lobbying, issuing other reports, and holding media events (Nolan 2003). Also of interest would be a survey of public policy makers to ascertain the extent to which they perceive that report cards influence public policy. O Report card design strategies. This article identified two approaches to the design of report cards. Left unanswered is whether designers consciously take either approach. That is, designers who grade harshly and use unreasonable criteria may indeed perceive their system as fair. On the other hand, they may be taking a calculated strategy aimed at gaining media attention, bolstering and rallying their faithful followers, or getting the attention of policy makers to change public policy.

and training. Fire-alarm criteria include dispatch operations, staff size, and amount of equipment. Weights include water supply, 40 percent; fire department, 50 percent; fire-alarm system, 10 percent. 5. Considerable scholarship exists on rankings (see www.library.uiuc.edu/edx/rankbib.htm).

References Abelson, Donald. 2002. Do Think Tanks Matter? Montreal, Canada: McGill-Queen’s University Press. Allen, David, and Sinead O’Brien. 1999. State of American Newspaper Capital News. American Journalism Review 21(6): 57–63. Atkinson, Robert D. 2003. Telephone interview with the author, May 8. Bennett, W. Lance. 1997. Why Government Innovation Is Not News: The View from the Newsroom. In Innovation in American Government, edited by Alan Altshuler and Robert Behn, 177– 201. Washington, DC: Brookings Institution Press. Clarke, Jody. 2003. “State of the Air” Report Deserves a Failing Grade. www.cei.org/gencon/003, 03458. cfm [accessed September 28, 2005]. Clotfelter, Charles, and Helen Ladd. 1994. Information as a Policy Lever: The Case of North Carolina’s School Report Card. Paper presented at the Annual Conference of the Association for Public Policy Analysis and Management, October 28, Chicago, IL. ———. 1996. Recognizing and Rewarding Success in

Notes

Public Schools. In Holding Schools Accountable:

1. The management report cards published by Governing magazine are five separate report cards. 2. Following a classification scheme devised by the Carnegie Foundation for the Advancement of Teaching, U.S. News divides institutions into four categories: national universities, national liberal arts colleges, regional universities, and regional liberal arts colleges. Regional schools are then divided into four geographic regional groups: North, South, Midwest, and West. Twelve of the 16 criteria measure input (average faculty salary, percentage of full-time faculty, faculty degrees, students’ test scores, students’ high school class standing, acceptance rate, yield, class size, student– faculty ratio, educational expenditures, and other expenditures), and four measure output or outcome (freshman retention, graduation rate, value added, and alumni donations). Reputation is a global measure capturing some aspects of inputs, process, and outputs. 3. Mississippi, North Carolina, Hawaii, Utah, and Washington use their own personnel. 4. Water-supply evaluations are based on hydrant availability and condition, as well as water pressure. Fire departments are graded on their staffing levels, geographic distribution of stations, equipment,

Performance-Based Reform in Education, edited by Helen Ladd, 23–63. Washington, DC: Brookings Institution Press. Cochran, John. 2003. Interest Groups Make Sure Lawmakers Know the Score. CQ Weekly Congressional Affairs 61(16): 924–29. Coe, Charles. 2003. A Report Card on Report Cards. Public Productivity and Management Review 27(2): 53–76. Coggburn, Jerrell D., and Saundra K. Schneider. 2003. The Quality of Management and Government Performance: An Empirical Analysis of the American States. Public Administration Review 63(2): 206–13. Cordato, Roy. 2001. Clearing the Air: ALA’s Misleading Attack on Air Quality. Spotlight No. 193, John Locke Foundation. http://johnlocke. org/spotlights/2001052342.html [accessed September 28, 2005]. Darnall, Nicole. 2003. Why Firms Adopt ISO 14001: An Institutional and Resource-Base View. Best Paper Proceedings of the 2003 Academy of Management Conference, August 1–6. Seattle, WA. Davidson, Roger, and Walter Olesrek. 2004. Congress and Its Members. 9th ed. Washington, DC: Congressional Quarterly Press. Organizational Report Cards

99

Edwards, Chris. 2003. Telephone interview with the author, June 17. Glazer, Sarah. 2000. Do Public “Report Cards” Improve Medical Care? CQ Researcher 10(17): 1–31. Gormley, William T., Jr. 1978. TV Coverage of State Government. Public Opinion Quarterly 42(3): 354–59. ———. 1998. Assessing Health Care Report Cards. Journal of Public Administration Research and Theory 8(3): 325–52. ———. 2003. Using Organizational Report Cards. Paper presented at the National Public Management Research Conference, October 11, Washington, DC. Gormley, William T., Jr., and David Weimer. 1999. Organizational Report Cards. Cambridge, MA: Harvard University Press. Gottlieb, Bruce. 1999. Cooking the School Books: How U.S. News Cheats in Picking Its “Best” Colleges. Slate, September 1. http://slate.msn.com/ id/34027/ [accessed September 28, 2005]. Graham, Amy, and Nicholas Thompson. 2001. Bottom Ranks: U.S. News’ College Rankings Measure Everything but What Matters. Washington Monthly, September. Gray, Virginia, and David Lowery. 2000. Where Do Policy Ideas Come From? A Study of Minnesota Legislators and Staffers. Journal of Public Administration Research and Theory 10(3): 573–97. Hamilton, James. 1995. Pollution as News: Media and Stock Market Reactions to the Toxics Release Inventory. Environmental Economics and Management 28(1): 98–113. Jencks, Stephen, Timothy Cuerdon, Dale Burwen, Barbara Fleming, Peter Houck, Annette Kussmaul, David Nilasena, Diana Ondin, and David Arday. 2000. Quality of Medical Care Delivered to Medicare Beneficiaries. Journal of the American Medical Association 284(13): 1670–76. Kemper, Diane. 2003. Telephone interview with the author, February 12. Kingdon, John. 1995. Agendas, Alternatives, and Public Policies. New York: Longman.

100

Public Administration Review • January | February 2006

Konar, Shameek, and Mark Cohen. 1997. Information as Regulation: The Effect of Community Right to Know Laws on Toxic Emissions. Environmental Economics and Management 32(3): 109–24. Kuczynski, Alex. 2001. “Best” List for Colleges by U.S. News Is Under Fire. New York Times, August 20. Lombardi, John, Diane Craig, Elizabeth Capaldi, and Denise Gater. 2000. The Top American Research Universities. Gainesville, FL: The Center, University of Florida. Monks, James, and Ronald Ehrenberg. 1999. U.S. News and World Report’s College Rankings: Why Do they Matter? Change 31(6): 42–51. Morgan, Scott. 2003. Telephone interview with the author, August 1. Mukamel, Dana, and Alvin Mushlin. 1998. Quality of Care Information Makes a Difference. Medical Care 36(2): 945–54. Nolan, Janice. 2003. Telephone interview with the author, November 13. Reeves, Kristy R. 2003. Telephone interview with the author, June 17. Reynolds, Meghan, and William O’Hare. 2002. Media Coverage of the 2002 KIDS COUNT Data Book. www.aecf.org/kisdcount/databook_coverage_ 2002.pdf [accessed September 28, 2005]. Schwartz, Joel, and Steven Hayward. 2003. Four Questions Reporters Should Ask about ALA’s “State of the Air” Report. http://www.rppi.org/ fourquestions.html [accessed September 28, 2005]. Thompson, Nicholas. 2000. Playing with Numbers: How U.S. News Mismeasures Higher Education and What We Can Do about It. Washington Monthly, September. www.washingtonmonthly. com/features/2000/0009.thompson.html [accessed September 29, 2005]. Vasu, Michael. 2003. Project Director: EPA Toxic Release Study. Telephone interview with the author, November 12. Whiteman, David. 1985. The Fate of Policy Analysis in Congressional Decision Making: Three Types of Use in Committees. Western Political Quarterly 38(2): 294–311.