Aiding Welfare-to-work Transitions: Lessons from ... - Semantic Scholar

Aiding Welfare-to-work Transitions: Lessons from JTPA on the Cost-effectiveness of Education and Training Services

Carolyn J. Heinrich, Ph.D.

JCPR Working Paper, October 1998, #3

This research was sponsored by the Northwestern University/University of Chicago Joint Center for Poverty Research, Small Grants Program. I thank Rueben Snipper and Peter Mueser for helpful comments and Annie Zhang for computer programming assistance.

Carolyn Heinrich is the Associate Director of the Center for Social Program Evaluation at The University of Chicago and Research Director for the Pew Study on Public Management and Governmental Effectiveness. She can be contacted at The University of Chicago, Irving B. Harris Graduate School of Public Policy Studies, 1155 East 60th Street, Chicago, IL 60637, or by phone or e-mail at: (630) 527-9048 or [email protected].

ABSTRACT A growing consensus is emerging that under PRWORA, more disadvantaged welfare recipients are failing to retain jobs and are not earning enough to rise above the poverty level, even when working fulltime. In this study, I draw primarily on the experiences of welfare recipients who participated in JTPA programs to address the central research question: What are the most cost-effective education and training services to aid individuals making the transition from welfare to economic self-sufficiency? The study findings suggest that if afforded access, formal on-the-job training and vocational training in technical/professional fields might substantially and cost-effectively increase the labor market success of more disadvantaged TANF recipients. Before or while they are engaged in these training activities, however, they need to acquire the basic educational skills that will enable their success in training and unsubsidized employment. Younger participants with little or no work experience or labor attachment, (and lower opportunity costs of participating), should be targeted for these services.

1.0

Introduction The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) replaced individual entitlements to welfare with a single block grant to states (Temporary Assistance for Needy Families or TANF) and established strong "work-first" requirements to move families from dependence on government aid to economic self-sufficiency. Even before the enactment of PRWORA, a majority of states experienced significant declines in their welfare caseloads, in part due to welfare-towork initiatives adopted in the late 1980s and early 1990s, but mainly as a result of robust economic conditions throughout the U.S. (Ziliak, Figlio, Davis and Connolly, 1997.) Under PRWORA and continued economic strength, most states have seen their welfare caseloads decline further and sometimes dramatically. As anticipated, the first cases to leave welfare have been those with fewer barriers to employment (Harris and Edin, 1998; Dresang, 1997.) Employers have been more than willing to provide the most job-ready welfare recipients with opportunities to work. Many current aid recipients attempting to make this transition, however, lack the basic education, employment skills and work experience necessary to acquire and retain employment. Employers have indicated in recent gatherings, including the U.S. Conference of Mayors and the American Association of Community Colleges, that a government role in raising aid recipients' education and skills levels will continue to be important. While PRWORA legislation encourages states to continue providing education and training to high school dropouts and long-term welfare recipients, there are no federal requirements regarding the level of services made available. The Act also requires that any services provided be directly related to, or concurrently provided with, work or job search activities. Many states are still in the process of determining how much education and training will be made available to TANF recipients, and equally important, how much will be cost-effective and affordable to provide. The central question I address in this research is: What are the most cost-effective education and training services to aid individuals making the transition from welfare to economic self-sufficiency? In this research, I draw primarily on the experiences of individuals who participated in programs administered under Title 2A of the Job Training Partnership Act (JTPA), the largest federally-funded job-training program. Nationally, 95 percent of JTPA participants have a family income equal to or below the federal poverty line and nearly half are receiving some form of public assistance at the time they apply to the program. In addition, three billion dollars in welfare-to-work grants are being administered through JTPA over the next two fiscal years to provide services to TANF recipients who cannot be directly placed into jobs, as well as to arrange job placements for others. In the next section, I review prior research on the effectiveness of employment and training programs and indicate what contributions I expect this work will make. In section 3, I describe the study population and data sources and discuss the advantages and limitations of these data. Section 4 presents the research hypotheses, and section 5 follows with the research findings. Finally, in section 6 of this paper, I discuss the findings and their policy implications.

1

2.0

Prior research on the effectiveness of employment and training programs Following an extensive review of evaluations of both voluntary and mandatory training programs, Friedlander, Greenberg and Robins (1997) identified some key questions that they believe should be addressed in future evaluations of training programs for the economically disadvantaged. The broadest question they pose asks how the aggregate effects of government training programs can be cost-effectively increased. Another question pertains to the development of targeting strategies to ensure that those most in need of training are those who participate in training programs. A third question asks if additional intensive, skills-building activities can be designed to produce large enough, lasting earnings effects that make their use cost-effective on a larger scale, particularly for the more disadvantaged. The research I present adds to our base of knowledge about the cost-effectiveness of training programs for the economically disadvantaged and sheds some light on each of these three questions. As Friedlander, Greenberg and Robins (1997) also report, previous evaluations, including studies of welfare recipients participating in training programs, have produced some conflicting findings on program cost-effectiveness. For example, Carol Romero (1994) of the National Commission for Employment Policy studied JTPA programs and found that "there is a payoff to training as well as job placement" in terms of employment outcomes and raising the incomes of welfare recipients above the poverty level. On the contrary, a 1992 report to the Ford Foundation by the Institute for Women's Policy Research concluded that "job-training appears to have no significant impact on the likelihood of work...or on an AFDC mother's ability to raise her family out of poverty." The findings of the National JTPA Study indicated that only AFDC recipients who engaged in on-the-job training and job search assistance achieved significant earnings gains (Orr, Bloom, Bell, Lin, Cave and Doolittle, 1995.) A follow-up to this study, however, by Erik Beecroft and Larry Orr (1996) showed that welfare recipients' earnings gains were not large enough to diminish their need for public aid. Greenberg and Wiseman's (1992) evaluation of welfare-to-work demonstrations similarly showed very small program effects in terms of employment outcomes, earnings and reduced transfer payments. Evaluations of four welfare-to-work programs by the Manpower Demonstration Research Corporation (MDRC) (Friedlander and Burtless, 1995), including the California GAIN program (Friedlander, Riccio and Freedman, 1994), indicated that the main achievements of these programs have been quicker job-finding and increased work hours for participants. Friedlander and Burtless report that most of the programs did not increase the quality of jobs obtained by welfare recipients, and no significant differences in wage/earnings rates were found. O'Neill and O'Neill (1997) showed that even the large percentage earnings gain attained by Riverside GAIN participants was due primarily to increases in amount of time they worked per year. In addition, Riccio and Orenstein (1996) found in the GAIN evaluations that a programmatic emphasis on quick job entry was correlated with larger earnings impacts, but they acknowledged that the Riverside GAIN site may have been driving the findings. Without Riverside in the sample, there was a negative correlation between a strong emphasis on quick job entry and earnings gains and welfare savings. Therefore, it is not surprising that when the initial effects of more rapid transition into the job market and increased work hours diminished through normal employment transitions over time, no significant lasting program impacts were found. 2

The majority of these welfare-to-work programs have provided little in the way of formal education services and more intensive, skills-building activities. Yet in Friedlander and Burtless' evaluations (1995), the most persistent earnings impacts were found in the Baltimore Options program, which placed less emphasis on job search activities and more on human capital development through education and training activities. They also found that quarterly earnings gains in the Baltimore program grew over time, suggesting that "human capital effects develop gradually from a lengthy investment period." While the net cost of the Baltimore program was the highest of the four programs they studied, compared to at least one of the other programs (San Diego SWIM), the cost differential was very small. One of the main challenges to obtaining more information about what employment and training strategies for welfare recipients are most cost-effective is the significant time and resources involved in conducting comprehensive, long-term program evaluations. The National JTPA Study (NJS), the largest experimental evaluation of employment and training programs, has produced a large body of research findings that Friedlander, Greenberg and Robins (1997) identify as "the most credible evidence to date about training program effectiveness." The NJS evaluation cost a staggering thirty million dollars, however, and researchers have pointed to numerous problems and limitations of these findings (Heckman and Hotz, 1989; Heckman and Smith, 1995; Heckman, Hohmann, Khoo, and Smith, 1997; Orr, Barnow, Lerman and Beecroft, 1997). In the National JTPA Study, experimental analyses of the relative effectiveness of services received by participants were complicated by the fact that the analyses were based on service recommendations, rather than actual services received. For some individuals, the services recommended were different than those which they received. In follow-up nonexperimental analyses, Orr et al. (1997) attempted to resolve this problem but encountered additional difficulties due to small sample sizes for individual services. While data from sixteen sites might enhance the generalizability of the NJS research findings, the additional sources of variation in participants, administrative and service delivery practices, and economic conditions across the service delivery areas made working with small samples in these analyses questionable. Heckman, Hohmann, Khoo and Smith (1997) also investigated this problem and found that a large proportion of individuals in the NJS experiment who were recommended to receive classroom training services but were subsequently assigned to the control group, (i.e., who were excluded or "randomized out" from participation in JTPA), obtained training anyway from alternative programs. Moreover, the types, intensity and duration of this training were similar to those delivered through the JTPA programs. Combined with the fact that some members of the treatment group dropped out of programs without receiving classroom training as assigned, Heckman et al. found that the difference in the fraction of treatment versus control group members receiving this training was reduced to approximately 0.20, rather than 1.00 as assumed. Using nonexperimental methods and accounting for the true difference in participation rates, Heckman et al. (1997) found a sizable negative effect on earnings of being in training, and a large positive effect on earnings of completing classroom training. The individual net returns to classroom training, estimated under a variety of conditions, were in the thousands of dollars. These estimates of 3

the cost-effectiveness of classroom training were substantially higher than the earnings impact estimates for classroom training generated in the National JTPA Study. Also complicating the NJS experimental analyses was the fact that program impacts were computed for 18- and 30-month periods following random assignment, which occurred after individuals were recommended for services. The period of program participation is therefore included in the follow-up period. Since participants were enrolled for varying lengths of time in different JTPA program activities, the NJS analyses do not estimate impacts for all participants over the same posttermination period. Furthermore, the opportunities to engage in paid employment during the training period vary considerably by the type of training received. Including the training period in the follow-up period for estimating program impacts likely further skewed NJS analyses of the relative costeffectiveness of different JTPA program services. For example, Heckman et al.'s findings of a large negative effect on earnings while participating in classroom training, (which is a relatively longer activity), suggests that the NJS experimental impact estimates of potential gains from this training are probably substantially downward biased. Overall, thirty experimental and nonexperimental evaluations of voluntary training programs and a little over half as many evaluations of mandatory training programs have deduced that, on average, adult women experience significant earnings gains through participation in training programs, including those whose participation is mandatory (Friedlander, Greenberg and Robins, 1997.) However, we still need to know much more about who among these women to target for specific types of program services, (e.g., basic/remedial education, vocational training, on-the-job training, job search and other employment skills training services), and how cost-effective different "packages" of employment and training services might be for different subgroups of women receiving welfare. The research I now present addresses these questions and specifically aims to add to the base of more "practical" knowledge available to and essential to those currently engaged in implementing welfare-to-work programs. 3.0 3.1

The study population and data used in this research Data sources, comparative advantages and limitations The data I used to address these research questions were collected from the Cook County President's Office of Employment and Training, the government agency that administers JTPA programs in Suburban Cook County, Illinois. These data include all of this agency's program administrative records for JTPA participants over the years 1984 (the first full JTPA program year) through 1994, or more than ten consecutive years of program operations. They contain detailed background information on program participants collected at the time of application and enrollment, complete records of their training activities and supportive services received, information about their employment status and earnings at the time of discharge, and follow-up earnings and employment data. In total, I obtained over 44,000 participant training records. I also obtained data from the performance-based contracts between this agency and its subcontracted service providers, beginning with the first full JTPA program year through June, 1993. These contracts (over 750) contain information about the service providers, their participant enrollment 4

plans, planned program activities and costs, performance standards and other information. The contract data were linked to the program administrative records to develop controls for and facilitate analyses of the relative costs and effectiveness of different types of service provider organizations. Together, the contract and administrative data allowed me to estimate program costs for the various service categories with unprecedented accuracy. In addition, through an independent, advisory relationship I maintained with the Cook County JTPA agency over a four-year period, I was able to observe all of the administrative processes involved in contracting for and delivering program services. Although these data were obtained from a single JTPA agency, I find there are a number of advantages to using them in this research. First, the administrative structure of this agency and the JTPA-eligible population it serves are typical of a majority of local job-training agencies. By focusing on a single geographic area, and conceding some loss of generalizability, it is easier to control for the influence of economic conditions on participant outcomes. Heckman and Roselius (1994) showed that a primary cause of the tainted findings of earlier JTPA program studies was the absence of (or failure to use) data on local labor market characteristics in the estimation of program outcomes and impacts. Information on annual changes in total employment, exactly matched to the Suburban Cook County service delivery area boundaries, was merged with these data to develop accurate controls changing economic conditions over the study period. Since these data span a period of more than ten years, they allow for the examination of changes in service delivery and program costs over time and the implications of these changes for program outcomes. Their repeated cross-section character also facilitates analyses of repeat enrollments of program participants. A study of enrollments in Illinois JTPA programs (Trott and Baj, 1993) found that by 1990, repeat enrollments constituted 20 percent of all JTPA enrollments. The studies by Romero and the Institute for Women's Policy Research drew data from a single year in the mid-1980s, prohibiting these kinds of analyses. In addition, the National JTPA Study data were collected over a three-year period in the late 1980s, prior to the precipitous funding declines of the 1990s and other important policy changes. Another advantage of these data are the precise estimates of program costs they have generated. Complete access to the Cook County agency's contracts with service providers was key to acquiring the information necessary to produce these estimates. (See Heinrich and Zhang, 1998, for a review of previous training cost estimation efforts and for details about how cost estimates for this study were computed.) Despite the extensive (and expensive) data collection and research activities of the National JTPA study, NJS researchers were still compelled to use a number of different sources to impute training costs, and the reliability of their estimates was questionable. The cost estimates I generated for the different types of training were analyzed in conjunction with estimates of participants' earnings gains (or losses) to produce more useful information on training service cost-effectiveness. A fourth advantage of these data is the availability of detailed background information on welfare recipients receiving JTPA program services, which allows for the examination of program outcomes and the effectiveness of different types of training for different subgroups of the public aid population. One of the more challenging aspects of designing programs to aid individuals making the transition from welfare to economic self-sufficiency is the diversity among public aid recipients. While 5

Romero (1994) analyzed training program outcomes by the age, education and race of participants, there are many other participant characteristics, including their employment, earnings and public aid histories, that may be even more likely to influence their training program experiences and outcomes. If characteristics of welfare recipients who are most likely to benefit from education and training services are identified and their anticipated gains quantified, policymakers can more effectively allocate their limited program resources among these groups. These data also include uniquely available information about organizations delivering employment and training services in this service delivery area. Using these data, I was able to examine whether different types of organizations, with different missions, performance records and responses to performance incentives, have a significant influence on participants' employment and training program outcomes. As we have already seen, a diverse array of for-profit and not-for-profit organizations have entered the competition for contracts to operate and manage state and local welfare-to-work programs. Salamon (1993) showed that for-profit organizations have contributed to a disproportionate share of the growth in the social services field. Many maintain the perspective, however, that nonprofits are more trustworthy, charitable, equitable and accountable in their delivery of social services. If this is true, one would expect them to serve the more disadvantaged JTPA clientele and to place more emphasis on raising participants' skills levels and helping them achieve long-term employment and earnings gains.

Finally, this study shows what we can learn from data that are at the fingertips of JTPA program administrators, using resources that are a negligible fraction of the cost of experiments such as the National JTPA Study. One of the primary reasons for undertaking costly experimental evaluations is to create a control group identical to those receiving program services (the treatment group), so that unbiased estimates of program impacts may be produced. In this study, I focus on estimating the relative cost-effectiveness of different types of training, (employing controls for participant characteristics, attributes of service providers and services delivered and local economic conditions.) A nonparticipant control or comparison group is not required to perform these analyses, although not having a control group also constrains the types of program effects I am able to evaluate. For example, I measure the change in participants' earnings from pre-program to post-program periods, but I do not obtain estimates of participants' earnings in the counterfactual state of "no program participation," or in the absence of program services. (These technical issues are discussed in more depth in Appendix A.) Some experimental evaluations are also designed to allow for a longer-term follow-up, such as the evaluations discussed by Friedlander and Burtless (1995) and the National JTPA Study. Evaluation research has consistently shown that longer-term follow-up periods of three to five years or more allow for more accurate measures of program impacts. In this study, I obtained four quarters of pre-program earnings data for JTPA participants and four quarters of post-program earnings. I found that the number of quarters of follow-up information I obtained, however, was not substantially different from that of the NJS, (when the follow-up period is defined as beginning after participants' discharge from the program.) The NJS included program participation time in the follow-up period, and after eighteen follow-up months, many cases were missing earnings data. While the single year of follow-up data I use precludes an examination of long-term outcomes, these data are consistent with a more realistic timeframe for evaluations of ongoing programs. Many program administrators and policymakers require 6

more timely information to assess program performance and guide implementation. State- and locallevel government agencies are increasingly making use of technological advancements to facilitate datasharing and manipulate administrative data in ways that will aid routine program management as well as major programmatic decisionmaking (Bowman, 1993.) 3.2

JTPA participants in comparison with the welfare population In her 1994 National Commission for Employment Policy Report, Carol Romero points out that AFDC recipients who participate in JTPA programs may be a "select group," particularly with respect to their motivation to obtain employment. In comparing JTPA participants without employment in the prior year to the general AFDC population not employed in the prior year, she found that almost four times as many general AFDC recipients reported that they were not in the labor force, (i.e., not looking for work.) A concern in making generalizations based on the experiences of welfare recipients participating in JTPA is that this group's higher level of motivation might make them more successful in training programs than average public aid recipients or those less motivated to enter the labor force. In the Suburban Cook County, Illinois JTPA service delivery area I focused on, approximately 80 percent of the enrollees were unemployed at the time they applied to JTPA, suggesting they were looking for work. About one-fifth of those on welfare at the time of application to JTPA indicated they were not in the labor force. Attempting to address this issue, I analyzed program outcomes for two groups of welfare recipients participating in JTPA: (1) all adult welfare recipients, and (2) Job Opportunities and Basic Skills (JOBS) participants, whose enrollment in employment and training programs was mandatory. O'Neill and O'Neill (1997) show that in 1992, 57 percent of adult AFDC recipients in Illinois were mandatory registrants for work and training programs. During the 1991-1993 JTPA program years, about 42 percent of the adult AFDC recipients who participated in JTPA in this service delivery area were JOBS registrants. AFDC clients who refused to participate or refused what they believed were "unsatisfactory" program activities risked losing their benefits. Orr et al. (1997) suggest that because JOBS participants who enrolled in JTPA were mandated to participate, welfare recipients who are relatively disadvantaged and enroll in JTPA by their own choosing may be "more motivated and otherwise employable than recipients with the same measured characteristics in mandatory work programs." While a sizable proportion of the Illinois caseload was still exempt from these participation requirements, (i.e., mainly persons with disabilities or very young children), I found that JOBS participants in JTPA were distinctly different from the average JTPA adult welfare recipient (see Table 1). In particular, they were much younger and were significantly more likely to be minorities, to be out of the labor force, to have minimal work experience and to have received AFDC for two or more years. While about 19 percent of all welfare recipients participating in JTPA reported they were not in the labor force, over 43 percent of JOBS participants were not in the labor force. Although I have no empirical evidence to show that JOBS participants are significantly less motivated than welfare recipients who apply to JTPA on their own, I find they are more disadvantaged and assume that the threat of sanctions is a serious concern influencing their enrollment into JTPA. In addition, under the 7

new Temporary Assistance for Needy Families (TANF) program, very few aid recipients will be exempt from work and training requirements, suggesting that many more will be similarly motivated as JOBS participants by the possible loss of benefits to engage in work and training activities. The JOBS participants I study are probably not the most disadvantaged among public aid recipients, (i.e., the physically or mentally disabled or those with sick or disabled children), but they are clearly among the more disadvantaged welfare recipients participating in JTPA. The findings presented in section five uncover significant differences in the types of training that were effective for JOBS participants compared to those that were most beneficial for the average adult welfare recipient participating in JTPA. In addition, based on extended observations of training program service delivery and my interactions with employment and training professionals and some welfare-to-work program administrators, I find that most have a realistic appreciation of the minimum capabilities required of individuals to enter training programs with some chance of success. It does not seem realistic to expect the physically or mentally disabled or those with sick children, (i.e., the most disadvantaged among welfare recipients), to participate in these programs without first addressing, if possible, their barriers through other health and social services. 4.0

Research hypotheses I advanced four main hypotheses in this research. The first two hypotheses and their corollaries address the types of training welfare recipients received and the employment and earnings outcomes they realized following participation in JTPA. Although eligible persons cannot be turned down for TANF services, some "selection" among agencies subcontracted to deliver services may still occur with respect to the types of services made available to participants. In separate research, both Brodkin (in her study

of JOBS programs administered through public aid offices, 1997) and Heinrich (1995) found that an important component of "selection" into work and training programs is the process of assigning participants to program services. Brodkin described how welfare client preferences for JOBS services were "severely circumscribed" and management preferences were asserted in determining the mix of recipients to be enrolled and the services to be offered. The number of education, training and supportive service "slots" were limited in a strategy that sought to minimize state program costs. Failing to consider the role of participant service assignments, one might erroneously determine that employment and training services are ineffective, when possibly only specific types of training are less cost-effective for certain subgroups of participants. Hypothesis 1: In JTPA programs, are welfare recipients assigned to different types of employment and training services than those who are not receiving welfare at the time of program application? (Corollary): What other characteristics of welfare and non-welfare participants have a significant influence on the types of training they receive? Hypothesis 2: Are certain types of education and training services relatively more 8

effective for welfare recipients than others, in terms of costs and employment and earnings gains? (Corollary): How does their relative effectiveness differ for various welfare subgroups, including aid recipients with weaker employment histories, longerterm aid recipients, high school dropouts, and other subgroups? The next hypothesis asks whether the outcomes of JTPA participants who enroll more than once improve over subsequent enrollments. The information generated through these analyses is likely to be important to policymakers as they consider how much training should be made available and whether TANF recipients who continue to struggle in securing and/or retaining employment should be allowed repeated access to employment and training services. Hypothesis 3: Do the post-program outcomes of welfare recipients who enroll in and complete training in JTPA programs more than once improve over subsequent training periods? (Corollaries): How long is the time between their training spells? Do they receive different types of services when they re-enter the program? Since many states are "contracting out" the delivery of services designed to move aid recipients to work, it is essential for them to have some idea of what types of organizations provide different types of services more effectively, (in terms of costs as well as gains to participants.) The final hypothesis I posed asks whether certain types of organizations are more effective in working with public aid recipients or specific segments of the public aid population. In addition, one billion dollars in performance bonuses will be distributed to states, ($200 million per year, beginning in this fiscal year), to increase the level of resources available to successful programs. Since JTPA programs operate under a well-established performance standards system and the Suburban Cook County JTPA agency I studied included performance standards in its service provider contracts, I also examined the implications of using performance-based contracts with these different types of organizations. Hypothesis 4: Are certain types of organizations, (e.g., private, for-profit firms, nonprofit agencies or community-based organizations, and public entities), more effective in working with welfare recipients? Corollary: How do these various types of organizations differ in who they target among welfare recipients, the types of services they offer to welfare recipients, and the amount of money they budget per participant?

9

5.0 5. 1

Research findings Analyses of training service assignments and service effectiveness I begin by presenting analyses about welfare recipients' JTPA training service assignments. Table 2 shows the ten most common combinations of training services provided to adult JTPA participants and to the welfare adult subgroup in the Suburban Cook County service delivery area. Six of the ten most common combinations are "categorical" or single-service strategies in which participants receive only one type of training before being discharged from the program. About three-fourths of all adults and 80 percent of welfare adults received only one type of training during their enrollment. The most common type of training received by both adults and welfare adults was vocational training. Adults receiving welfare were significantly more likely to receive vocational training only (÷2=332.34, p=0.000) and were significantly less likely to receive on-the-job training (÷2=70.39, p=0.000) compared to all adult participants. To examine what factors influence adult JTPA program service assignments, I estimated a multinomial logit model of participant assignments to core program services, (vocational training, remedial education, on-the-job training, and job search/job club activities.) I included controls in this model for participant demographic characteristics, education levels, recent employment and public aid histories, and economic conditions -- factors I found in earlier research to be related to intake counselors' service assignment decisions (Heinrich, 1995.) (All of the variables included in these models are defined and described in Appendix B.) I was particularly interested in determining if, controlling for these factors, welfare recipients are assigned to different types of services than those who are not receiving welfare at the time of program application. The findings from this model are reported in Table 3. Although the pseudo R2 value indicates that only about 13 percent of the variation in service assignments is explained, a number of highly significant findings emerge. First, the results confirm that welfare recipients are significantly more likely to receive vocational training as well as job search assistance than other adult JTPA participants in this service delivery area. On the contrary, welfare recipients are significantly less likely to receive on-the-job training, the service that unequivocally contributes to the best earnings and employment outcomes for all adult participants, including those on welfare and participating in JOBS. Those who had been receiving welfare for two or more years at the time of application were significantly more likely to be assigned to the less intensive services of employment counseling and job search or job club, (with the exception of JOBS participants.) Overall, vocational training recipients are significantly more likely to be women, welfare recipients and high school dropouts. Those with the most serious barriers to employment -- basic skills deficiencies, high school dropouts, minimal work histories, not in the labor force at application and no earnings in the year prior to enrollment -- as well as younger persons and minorities, are significantly more likely to be assigned to remedial education. On the contrary, those receiving the most productive type of training (on-the-job training) also enter the program most prepared for work. Enrollees assigned to receive OJT are significantly more likely to be white or Hispanic males with work experience, earnings in the year prior to enrollment and high school degrees. 10

The multinomial logit model findings also indicate that JOBS participants are assigned to the least intensive job preparation activities, (i.e., the default categories of case-management, supportive services and some "soft skills" job preparation activities.) The coefficients on vocational training and on-the-job training are statistically insignificant, and O'Neill and O'Neill show that in 1992, only 7.5 percent of all JOBS participants (in the U.S.) received vocational training, and a tiny 0.5 percent engaged in on-the-job training. Blank (1997) points out that when the implementation of JOBS finally got underway, the early 1990s recession was also beginning, with the result that most budget-squeezed states introduced only minimal programs to satisfy federal requirements. Noteworthy among the findings on JTPA service assignments was the near absence of combinations of training services designed to sequentially or concurrently build the skills levels of more disadvantaged participants. For example, not a single adult participating in JTPA Title 2A programs in this SDA over program years 1984-1993 received a combination of basic/remedial education and vocational training, even though a high school degree or GED are required for entry to many vocational schools or community college vocational programs. Only 18 participants received basic/remedial education in combination with on-the-job training. In the very small fraction of cases it was made available with other services, basic/remedial education was most likely to be combined with employment counseling services. Tables 4 and 5 show employment, wage and earnings outcomes for a number of adult JTPA participant subgroups, defined either by their characteristics at enrollment or the training they received. Statistics are also shown for adult participants in the National JTPA Study (in the first row) to provide a baseline for comparison of program outcomes in this service delivery area. The second through fourth rows of these tables compare the outcomes of all adult JTPA participants and adult welfare recipients as well as the subgroup of JOBS participants. In Table 4, a pattern emerges in which welfare recipients achieve poorer outcomes than all adult JTPA participants, and JOBS participants fare worse than both of these. Welfare recipients and the subgroup of JOBS participants were significantly less likely (compared to all adult participants) to be employed at the time they were discharged from JTPA, and they were also significantly less likely to sustain employment throughout the first post-program year. The percentage of JTPA participants realizing increases in earnings during the first post-program year relative to earnings at placement is small for all three groups and is less than ten percent for JOBS participants. Table 5 shows a slightly different pattern, with JOBS participants attaining slightly higher average wages and first post-program quarterly earnings than welfare recipients overall. Examining participant employment and earnings outcomes across five core service types listed at the bottom of these tables, it is evident that those who received on-the-job training achieved outcomes far superior to those enrolled in any other type of program activity. Close to one-third of on-the-job training recipients experienced an increase in their quarterly earnings relative to earnings at placement during the first post-program year. On the contrary, participants who received basic/remedial education realized the poorest outcomes in the first post-program year, which is not surprising given that less than one-half were employed at the time they 11

completed training. The major problem with comparing the relative effectiveness of different employment and training services using the measures in Tables 4 and 5 is that one cannot separately evaluate the influence of participant characteristics versus the effects of training services provided. For example, Table 3 showed that males with more work experience were significantly more likely to be assigned to on-the-job training, while high school dropouts, minorities, and individuals with basic skills deficiencies, poor work histories, and no earnings in the pre-enrollment year were significantly more likely to receive basic/remedial education. To generate more definitive evidence on the relative effectiveness of different JTPA services provided in this service delivery area, I estimated multiple and logistic regression models for the seven program outcomes shown in Tables 4 and 5 and for three JTPA participant groups: all adult participants, welfare recipients participating in JTPA, and the JOBS participant subgroup. (See Appendix A for a review of the construction of these models and related technical issues.) In addition to indicators for the type of employment and training service(s) received, these models included controls for participant characteristics, service provider characteristics and service provider contract incentives, and yearly economic conditions. (All of the outcome and control/explanatory variables included in these models are defined in Appendix B.) The complete sets of model results are shown in Appendix C. I present two different model specifications, with the main difference between the two being the inclusion of an indicator for service provider contract performance incentives. In the first set of models (Tables C.1.a-C.4.a), information about service provider contract performance incentives is not included. In the regression set including the performance incentives indicator (Tables C.1.b-C.4.b), the number of observations is smaller because participants who were not served by a subcontractor, but rather by the SDA office (about 22 percent), were excluded from these analyses. In addition, the indicator for public subcontractor is omitted as the default (base) category for service provider type in the second model set. Table 6 summarizes findings on the relative contributions of different types of employment and training services to participants' employment and earnings outcomes and one-year net earnings gains or losses, (as estimated in the models shown in Tables C.1.a-C.4.a in Appendix C). The effects of four service types, (vocational training, basic/remedial education, on-the-job training and job search/job club), are shown relative to the base or default categories of employment counseling and casemanagement/support services, (i.e., the least intensive service activities.) About two-thirds of the training service type coefficients are statistically significant, and some very salient patterns emerge. The effects of on-the-job training on participant outcomes (see the third column) are almost all positive, relatively large and highly significant. On the contrary, the coefficients indicating the contributions of basic/remedial education services to participant outcomes are all negative and most are statistically significant as well. For vocational training, the findings are mixed for the three participant groups. To further investigate vocational training outcomes, I classified vocational training services received by participants into three subcategories using the standard occupational class codes contained in their training service records. The three subcategories of vocational training are: (1) low-skilled services/production, (e.g., food service workers, nurses aides, counter clerks, production assemblers, 12

etc.), (2) skilled business and other services (e.g., office manager, bookkeeping, social worker, police/protective services, etc.), and (3) skilled technical occupations, (e.g., registered nurses, health technicians, electronic repair, computer programmer, engine mechanics, etc.). Table 7 shows some characteristics of persons enrolled in these three vocational training subcategories, as well as their outcomes following program participation. Two points are obvious in the findings presented in Table 7. Those who receive vocational training in skilled business/other service and technical occupations enter JTPA more prepared for employment. Participants receiving technical training are especially more likely to be males with more education and work experience and are less likely to be receiving public assistance. Second, the outcomes for participants receiving skilled technical training are clearly superior to those for participants receiving other types of vocational training. While the job placement rate for this group is actually several percentage points lower than that for the low-skilled occupational training recipients, job retention is higher for this group and wages and earning outcomes are also significantly higher. In fact, the mean hourly wage at termination is higher for the technical vocational training recipients than it was, on average, for recipients of all other types of training (see Table 5 to make this comparison.) I also estimated a regression model of participants' earnings changes similar to that shown in Appendix C (Table C.3.a), with the only difference being the inclusion of three separate indicator variables for the different vocational subcategories, (instead of a single vocational training indicator.) The results showed positive, statistically significant contributions of vocational training in professional services ($230 per quarter, on average) and technical occupations ($183 per quarter, on average) to earnings changes for all adult participants. These findings suggest, as one might expect, that participants gain more through training that advances their technical/professional skills and prepares them for higherskilled jobs. Returning to Table 6, other interesting patterns are also evident in examining the relative effectiveness of training services across the three groups (all adult participants, welfare adults and JOBS participants.) For example, focusing on the two types of training more oriented toward increasing specific occupational or job skills (i.e., vocational and on-the-job training), I find that these services are substantially more effective for JOBS participants compared to all adults or adult welfare participants in general. For the vocational training category (as a whole), only JOBS participants consistently realize significant, positive effects on their employment and earnings through receipt of these services. In addition, the effects of on-the-job training on JOBS participants' employment and earnings outcomes are two to three times larger than those for all adult participants and welfare recipients in general. Only JOBS participants realized large, statistically significant net earnings gains in the first post-program year through participation in JTPA program activities, specifically, vocational training and on-the-job training. Overall, the relative contributions of job search and job club activities to participants' employment and earnings outcomes are poor. JOBS participants in particular appear to perform much more poorly in the labor market when they receive less intensive services that are designed to get them employed as quickly as possible. While Table 4 shows that on average, participants receiving job search or job club services attained the highest hourly wage at placement and the highest first postprogram quarter earnings, Table 3 suggests that the more able were also more likely to receive these 13

services. When controlling for participant characteristics and other service-related factors, job search and job club activities emerge as relatively ineffective services. Table C.3.a (in Appendix C) presents regressions of participants' pre- to post-program quarterly changes in earnings. The model for all adults that includes coefficients for welfare receipt and participation in JOBS (column four) suggests that, overall, JOBS participants gain relatively more from participating in JTPA, particularly through receipt of on-the-job training ($1387 per quarter, on average, as seen in column six.) The regressions estimating one-year net earnings gains (Table C.4.a) show similar results. The outcomes of JOBS participants in the labor market, however, are still, on average, lower than all adult participants and the general JTPA adult welfare group. An examination of the intercepts in the regression models shown in Appendix C confirms this. Table 1 suggests that JOBS participants and other more disadvantaged welfare recipients are building skills capacity from a "lower base," as more of these individuals have minimal, (if any), work experience and longer terms of public aid receipt, and proportionately fewer are in the labor force. Given their lower skills levels and meager labor market opportunities, it seems logical that the opportunity costs of participating in JTPA would be lower for more disadvantaged welfare recipients, (although I do not have data to measure this directly.) For all of the participants studied, a longer training period is negatively associated with employment and earnings outcomes (see the regression models in Appendix C.) This relationship is strong and statistically significant for all adults and welfare adults but is insignificant, however, for JOBS participants. I believe this may reflect the lower opportunity cost of time for the JOBS participants who are less likely to be attached to the labor force and whose skills (or lack thereof) have lower value in the labor market. If we could explicitly factor in opportunity costs of training program participation into net earnings gains measures as well, we might find that the relative net gains are even higher for more disadvantaged welfare recipients than the results in Table C.4.a suggest. 5. 2 Analyses of local economic factors and participants' repeat enrollments into JTPA One of the most pressing concerns of those engaged in welfare reform is about what happens to those former welfare recipients who obtain jobs but do not earn enough to escape poverty, or to those who become employed but experience serious difficulties retaining employment. Research by Harris and Edin (1998) shows that women at the bottom of the wage-skill distribution, earning minimum or close to minimum wages, simply cannot support their families on their earnings. How much training should be made available to TANF recipients who continue to struggle economically? If the economy slows or recesses and job availability declines severely, are education and training programs a productive alternative to employment for TANF recipients? I now address these questions and show what happens to the post-program outcomes of welfare recipients who enroll in and complete training in JTPA programs more than once. The participant outcome regressions (see the tables in Appendix C) revealed that changes in local economic conditions have significant implications for participants' labor market success following participation in JTPA. Data on changes in total employment in the local service delivery area showed 14

that during the mid-1980s through 1988, employment in the SDA was increasing. Between 1988 and 1989, however, total employment declined. Employment began to increase again and continued growing until the early 1990s recession, when employment subsequently fell between 1992-1993 and 1993-1994. The regression findings showed that even small changes of -0.0015 percent to +0.337 percent had significant implications for participants discharged from the program during those years. Table 8 summarizes the effects of two local employment declines on adult participant and welfare adult outcomes, (calculated using the coefficients on percent change in employment variables from the participant outcome regressions and the actual percentage changes in employment.) Declining employment has the strongest implications for participants' retention of jobs during the first year following program discharge, with the probability of being employed all four quarters reduced by as much as one-half during these periods. Post-program earnings are also significantly affected by economic downturns, with earnings lowered by over $200 per quarter for participants entering the labor market during these times. These findings on the strong effects of changes in local employment conditions lend support to those of Spalter-Roth et al. (1992), who estimated (using SIPP and other economic data) that for every one percent increase in a state's unemployment rate, the probability of being poor for single mothers receiving AFDC increased by 31 percent. Overall, these findings suggest that during economic slowdowns or recessions, significantly more TANF recipients might have difficulty obtaining or retaining jobs, and their earnings will likely be lower as well. During these periods, should recipients be allowed to enroll in programs providing additional education or employment and training services? It seems logical that during economic downturns, their opportunity cost of participation would be even lower. Were JTPA participants who enrolled more than once more successful in the labor market following subsequent enrollments? Table 9 shows that the average adult JTPA participant with more than one enrollment (column 2) is very similar to those who enroll only once (column 1), and welfare recipients who enroll more than once also seem no more disadvantaged than other welfare recipients participating in JTPA. An obvious difference between these groups, however, is in the percentage of participants and welfare recipients with more than one enrollment who are employed at the time of their first discharge from the program. For all adults with more than one enrollment, the job placement rate is over ten percent lower than those who enroll only once, and for welfare recipients with more than one enrollment, the job placement rate is lower by another five percentage points. In addition, the employment retention rates of those with multiple enrollments (all adults and welfare recipients) are significantly lower than those who enroll only once, as indicated by the percentages who were employed all four quarters of the first post-program year. As one might expect, this information seems to suggest that those who re-enroll are more likely to be struggling in the labor market than those who do not. Welfare recipients waited an average of almost two years before they re-enrolled in JTPA, and about two-thirds of this group were still receiving welfare when they enrolled again. Were participants who enrolled a second time assigned to the same services or to services that built upon their previous training, or did they receive only minimal assistance to get them back into the labor market? In Table 10, I present the conditional probability of receiving a specific employment and training 15

service during the second enrollment, given the type of training participants received during their first enrollment. The findings in Table 10 indicate that all adults and adults receiving welfare at their first enrollment were most likely to be assigned to the same services they received during their first enrollment when they enrolled the second time. The main exception to this pattern was for those who received job search during their first enrollment; they were most likely to receive vocational training in a later enrollment. The logical question that follows is: do the outcomes of these participants improve over subsequent enrollments, or do the same services prove to be equally (in)effective the second time around? Table 11 shows that for those receiving welfare at the time of their first enrollment into JTPA, their outcomes generally improved over subsequent enrollments. The most dramatic changes occurred in the percentages of participants who retained employment all four quarters of the first post-program year. Improvements in employment retention were particularly large for those who were receiving welfare at their first enrollment or both their first and second enrollments. For all three groups examined, hourly wages increased over subsequent enrollments, but the patterns in quarterly earnings were mixed. Those receiving welfare at the time of their first enrollment (or both their first and second enrollments) earned more per quarter following subsequent enrollments than they did in the quarter following their first enrollment. The same is not true for the average adult participant with more than one enrollment. In general, the findings suggest that welfare recipients in particular are significantly more likely to retain their jobs and to increase their wages over multiple program enrollments. The conditional probabilities in Table 10 indicated that the majority of participants with multiple enrollments received vocational training during their first and second enrollments. It is unclear, however, whether the gains made by "repeaters" are the result of increases in human capital generated through training or more so due to the work experience some gain through prior job placements. In addition, since most recipients received the same type of training during their multiple enrollments, there was little data to evaluate the effects of receiving a different type of training in subsequent enrollments, such as on-the-job training that might build upon prior vocational training. In section six, however, I discuss findings from a demonstration program in this same service delivery area that was specifically designed to concurrently and sequentially provide these combinations of training services. 5.3

Analyses of JTPA service provider effectiveness The final set of analyses I present used data on service providers and their subcontracts with this JTPA service delivery area to examine whether different types of organizations -- public, nonprofit and for-profit -- are differentially effective in working with public aid recipients and producing cost-effective participant outcomes. Since JTPA programs operate under a well-established performance standards system and the service delivery area I studied used performance-based contracts, I was also able to generate information about the implications of using performance incentives in subcontracts with these different types of organizations. Over the ten-year study period, the administrative agency in this service delivery area directly provided about 22 percent of program services to adult participants, while the remaining services were

16

delivered by subcontractors. Among these, nonprofits provided about 28 percent of services to adults, while public subcontractors delivered 26 percent of adult program services and for-profits accounted for the remaining one-fourth of adult services. These numbers show that these organizations have roughly equal shares in service delivery to adult JTPA clients. Table 12 shows demographic and other characteristics of JTPA participants, (all adults and adults receiving welfare at the time of enrollment), by service provider type. Focusing on welfare adults, a cursory review suggests that there are relatively few differences among the welfare recipients served by these organizations. Public subcontractors were significantly more likely to serve welfare recipients with minimal work histories (÷2=20.12, p=0.000) and with no (zero) earnings in the pre-enrollment year (÷2=71.30, p=0.000.) On the contrary, public subcontractors served significantly fewer welfare adults who were high school dropouts (÷2=69.35, p=0.000) and more with a post-high school education. This may be due in part to the focus of many public subcontractors (e.g., local community colleges) on the delivery of vocational training, which frequently requires participants to have a high school diploma or general equivalency degree. Across most other characteristics, the clientele of these organizations were very similar.

Table 13 shows the proportions of services delivered to all adult JTPA participants and to the subgroup of adult participants receiving welfare by service provider type (the SDA offices, public, nonprofit and for-profit subcontractors.) While nonprofit subcontractors were not dominant providers of any of these services for adults in general, they were the primary providers of basic/remedial education services to welfare recipients. For-profit subcontractors provided the majority of on-the-job training services to all adult participants and welfare recipients, and they were also the primary provider of vocational training services to welfare recipients. As seen in Table 13, for-profit providers were the least likely to serve welfare recipients with minimal work histories and zero earnings in the preenrollment year. This is consistent with the findings in Table 3 that showed persons with minimal work histories and zero earnings in the pre-enrollment year were significantly less likely to be assigned to receive on-the-job training. A strong pattern among these numbers reveals that compared to all adults, adult JTPA participants receiving welfare were significantly more likely to be served by the SDA offices than by subcontractors across all types of training (see column 1 of Table 13.) For example, while about 38 percent of all adults participate in job search through the SDA offices, 82 percent of welfare recipients receive job search through the SDA. Overall, SDA offices served a little over one-fifth of all adult JTPA participants, but they served about one-third of the adult welfare recipients. Depending on the relative effectiveness of SDA offices in serving welfare recipients, these findings might have important implications for welfare recipients' program outcomes. If we assume that the type of training subcontractors provide is exogenous and include controls for training service type in participant outcome models (see Appendix C, Tables C.b-C.4.b), statistically significant differences in the contributions of public, nonprofit and for-profit providers emerge in fewer than half of these models.1 Table 14 shows that, in general, for-profit subcontractors contribute to 1

The omitted or default service provider category in these regressions is the SDA's job-training offices, (actually another public service provider, although not a subcontracted one.)

17

significantly higher wage and employment outcomes for adult participants at termination and during postprogram quarter one. On the contrary, JOBS participants, who are more disadvantaged than the average welfare recipient participating in JTPA, appear to be better served by public organizations (including the SDA offices.) When one assumes service provider type and training service assignments are jointly determined, (which seems reasonable since many trainees apply for services directly with subcontractors who frequently specialize in a particular training service), the findings on service provider contributions to participant outcomes are stronger (see Table 15.) Adult participants and welfare adults served by for-profit subcontractors achieved significantly higher hourly wages at termination and earnings in the first post-program quarter and also experienced larger quarterly earnings changes. They were also significantly more likely to be employed at termination, in the first post-program quarter and all four quarters of the first post-program year. The consistently larger contributions of for-profit providers are most likely due in a large part to the role of for-profits as the primary providers of on-thejob training in this service delivery area. On the other hand, even when evaluating the type of training provided jointly with service provider type, JOBS participants still appear to achieve better postprogram outcomes through services delivered by public organizations. Why were the more disadvantaged welfare recipients more effectively served by public service providers or the SDA offices, even when taking into consideration the type of training provided? One possible explanation may lie in the amount of resources these different providers allocated per participant in service delivery. In examining the unit costs negotiated and budgeted in the service providers' contracts, overall, the average unit costs budgeted by public service providers were significantly higher than those of other (for-profit and nonprofit) subcontractors (see Table 16.) These differences were even larger and more highly significant for the subset of contracts serving public aid recipients. On the other hand, for-profit subcontractors serving public aid recipients budgeted unit costs that were significantly lower than those of public and nonprofit subcontractors combined. Numbers estimating the actual costs spent per participant (using individual training records linked to expenditure data) by different service provider types reveal substantially greater differences in average training dollars spent per participant (see Table 16 again.) While the average per participant costs of nonprofit and private subcontractors were very similar, the mean costs of public service providers were significantly higher by a large margin. Public subcontractors served comparatively few welfare recipients, but of those who did receive services from them, nearly all received vocational training. This is one of the longer and more expensive types of services. It is possible that public subcontractors were also more generous in the levels and types of supportive services provided to more disadvantaged participants during the time of their training. The SDA's records of supportive services did not have adequate information to fully assess the types and duration of these services provided to participants. Although a direct link between the amount of resources expended and the relatively better employment and earnings outcomes of the more disadvantaged welfare recipients served under public providers cannot be established, it does not seem unreasonable to assume a positive relationship between training expenditures and participant outcomes. The final column in Table 14 also shows the coefficients on performance incentive indicators 18

included in the regressions presented in Tables C.1.b-C.4.b. These variables indicate the presence of performance benchmarks in service providers' contracts with the SDA, (which typically included minimum hourly wage rate and employment retention specifications.) A little over two-thirds of the service provider contracts contained performance benchmarks. The findings on the effects of including performance incentives in subcontracts under which participants are served are somewhat surprising. For adult participants and the welfare subgroup, performance incentives appear to have some strong negative implications for participant wage and employment outcomes, (contrary to what one would expect if performance incentives were having the intended effects on service delivery and outcomes.) On the other hand, the subgroup of JOBS participants achieved some significantly higher outcomes when they were served by subcontractors who faced performance incentives. For example, all else being equal, JOBS participants who received services from subcontractors that were required to attain specific wage and employment benchmarks in order to be reimbursed for training expenditures were over 1.5 times more likely to be employed all four quarters following discharge from JTPA. These findings on the relationship of contract performance incentives to JTPA program outcomes for the JOBS subgroup seem to suggest that the performance incentives foster better service delivery. However, if this is true, the question remains as to why there were negative, statistically significant coefficients on the performance incentive indicators in models for all adults and adult welfare recipients. Particularly since the incentives are tied to outcomes at termination and for short periods (10-90 days) after termination, one would expect that their influence on the shorter-term outcomes achieved would be positive rather than negative. One possible explanation is derived from related research findings of Heinrich (forthcoming, 1998), which showed that JTPA staff in this service delivery area seemed to emphasize "quantity," i.e., getting as many participants placed in jobs as possible, while sometimes foregoing potentially higher wages or earnings at placement. The job placement standards used by the SDA in subcontracts were more stringent (relative to state and federal standards) and more challenging to satisfy than the hourly wage/earnings at placement standards. This resulting emphasis on "quantity" versus "quality" in service delivery may help to explain the negative influence of performance incentives on adult/welfare adult participant outcomes. Courty and Marschke (1997) and Brodkin (1997) have also suggested that performance incentives induce a "numbers game" that diverts case workers' attention away from delivering quality training and more toward accounting. On the contrary, given the significantly poorer labor market experience and other disadvantages of JOBS participants relative to other JTPA participants, it is possible that contract performance incentives had a different effect for this group. To satisfy contract requirements for wage and earnings outcomes that might be relatively weak for the average JTPA participant, it is likely that subcontractors serving JOBS participants would have had to more carefully monitor and make a greater effort to raise the wages at placement of this group to obtain full reimbursement of training costs. The performance incentives, therefore, likely have a positive influence as intended on service strategies and the outcomes of more disadvantaged participants. 6.0

Discussion of findings and policy implications 19

6.1

Discussion of research findings In general, the outcomes for welfare recipients and JOBS participants receiving training through JTPA seemed meager. Approximately one half of welfare and JOBS participants were employed at their time of discharge from JTPA. Proportionately more welfare and JOBS participants had no earnings at all in the first post-program year (38 and 41 percent, respectively) than were employed all four quarters of the year following discharge (34 and 26 percent, respectively.) Based on these average outcomes, one might conclude that JTPA services such as those made available in this service delivery area are not sufficiently effective to warrant their extension to a broader, welfare-to-work population. The change in quarterly earnings from the fourth pre-program quarter to the fourth postprogram quarter showed, however, that welfare recipients and JOBS participants were, on average, earning substantially more following their participation in JTPA ($409 per quarter and $996 per quarter, respectively.) JOBS participants, who were more representative of those welfare recipients likely to struggle in making the transition from welfare to work, realized some of the largest earnings gains of the JTPA subgroups examined. On-the-job training was clearly the most effective (and cost-effective) training service provided for adult participants, as evaluated across seven employment, earnings and cost-effectivness measures. An important finding among these results (presented in Table 6) is that the contributions of on-the-job training (OJT) to the outcomes of JOBS participants were as much as three times larger than the contributions of OJT to the average adult participants' outcomes. In addition, only JOBS participants achieved positive, significant net earnings gains in the first year following program participation through participation in OJT and vocational training. Net earnings gains projected and discounted over two- and three-year periods are shown in Table 17 to suggest potential gains over longer post-program periods if the earnings levels participants attained in the fourth post-program quarter continued, on average, in subsequent quarters (see Appendix A for further discussion of these measures.) What is most noteworthy among these estimates of net earnings gains are the substantial, increasing gains realized over time by JOBS participants who received on-the-job training, relative to other adults and other types of program services. These models estimate that, on average, on-the-job training would raise the earnings of JOBS participants by more than fourteen thousand dollars over a three-year period, compared to a little less than three thousand dollars for all adult participants. Unfortunately, the multinomial logit analyses of training service assignments showed that welfare recipients were significantly less likely to be assigned to on-the-job training, the most cost-effective service. The examination of vocational training subcategories also revealed that welfare recipients and JOBS participants were significantly less likely to receive vocational training in skilled, technical occupations, which unquestionably produced the largest average wage, employment retention and earnings change outcomes for participants receiving vocational training. Participants who received onthe-job training and vocational training in technical fields entered JTPA more prepared for employment; they were significantly more likely to be white or Hispanic males with high school degrees and/or posthigh school education, work experience and earnings in the year prior to enrollment. 20

Those JTPA participants with the most serious barriers to employment, (i.e., basic skills deficiencies, high school dropouts, little or no work experience, not in the labor force and no earnings in the year prior to enrollment), were, as one might expect, significantly more likely to receive basic/remedial education. While the returns to basic/remedial education were the lowest in terms of employment and earnings outcomes, research and practical experience suggest it is an essential activity for many welfare recipients. Olson and Pavetti (1996) found that a low level of basic skills is the most common barrier to employment for welfare recipients. They report that one-third of welfare recipients score in the bottom decile of the women's distribution of the Armed Forces Qualifying Test (AFQT), and another third score in the 10th-25th percentiles. Orr, Barnow, Lerman and Beecroft (1997) point out that adult basic education and General Educational Development (GED) classes are available through community colleges and other schools, and low-income persons may obtain federal Pell grants to pay for these classes. Program administrators and others engaged in service delivery contend, however, that without the additional supports and motivating factors that the context of a training program sometimes provides, individuals may be more likely to drop out of GED programs. In addition, Skricki (1990) found that 12 percent of 375 JOBS participants she interviewed had defaulted on student loans from previous educational experiences and were therefore unable to obtain financial aid to attend colleges and vocational schools. Many practitioners further agree that if individuals with low basic skills levels do not obtain remedial education and/or a GED, their chances of attaining success in the labor market following program participation are dismal. Romero (1994) also points out that while basic skills training does not lead to jobs, it may prepare individuals for formal job training. While it might seem logical that more of the highly disadvantaged could be prepared through activities such as remedial education and employment skills training to enter on-the-job training activities, over the 10-year period of this study, only 0.4 percent of all adults and 0.4 percent of welfare recipients in JTPA received these combinations of services. Furthermore, none of the adults received a combination of basic/remedial education and vocational training over this same period, even though a high school diploma or GED are required for entry to many vocational schools and community college programs. Program administrators in this service delivery area and others who were interviewed in conjunction with the National JTPA Study submit that it is too costly to provide multiple services such as these to participants. The regression analyses of participant outcomes also indicated that for the average adult JTPA participant, a longer training period had a significant, negative effect on program outcomes. This finding is consistent with that of other employment and training research (Heckman, Hohmann, Khoo and Smith, 1997; Jacobson, LaLonde and Sullivan, 1997.) Jacobson et al. studied displaced workers who received re-training through community colleges and found that during the time they were enrolled in courses, their earnings were lower the more hours they spent in classes. After leaving school, there was a transition period in which the schooling effects were usually substantially lower (and frequently negative) in the short-term than they were in the long-term. With a long follow-up period, they found modest earnings increases. However, the returns varied by the type of schooling and individuals' opportunity costs of training. Individuals with more than a few years of tenure with their previous 21

employer (more experience) incurred substantially larger permanent earnings losses that were not offset by schooling returns. For the JOBS participants in this study, who were significantly more likely to be out of the labor force and to have minimal work experience, the relationship between length of training and program outcomes was much weaker. This implies that the opportunity costs of training for this group are probably considerably lower. JOBS participants might not be negatively effected by longer training periods if services such as basic/remedial education were provided in combination with on-the-job training and/or vocational training. Some demonstration research involving job-training programs has already shown that training strategies combining basic/remedial education, employment skills training and on-the-job training can be cost-effective even for the most disadvantaged participants (Berg, Olson and Conrad, 1991; Heinrich, forthcoming.) In a demonstration program implemented in the Suburban Cook County service delivery area during the early 1990s, participants received basic skills training and/or vocational training services that were customized to on-the-job training (OJT) in which they simultaneously or sequentially participated. By making effective use of community resources and working directly with employers to coordinate educational activities with OJT duties/responsibilities, the demonstration program staff were able to maintain program costs at a level comparable to standard JTPA programs, while at the same time generating better program outcomes for the most disadvantaged (Heinrich, forthcoming.) The close working relationships program staff developed with employers were especially important to understanding the types of skills that were of value to employers and also to providing the guidance and supportive services necessary to foster employment retention among participants. While these more intensive service combinations would cost more to provide in the short-term than job search/placement strategies, the findings of this study also showed that for less prepared welfare recipients such as JOBS participants, job search assistance alone produced little or no benefits. Other research has similarly suggested dismal prospects for more disadvantaged TANF recipients who are routed directly to jobs. Harris and Edin (1997) found that women with more human capital obtained the better jobs and were more likely to stay off welfare, while women with less human capital were more likely to find unstable or low-paying jobs and to cycle between welfare and work. Rebecca Blank's (1995) research also suggests that lower-skilled women will continue to face dismal labor market opportunities, i.e., "bad jobs" at "bad wages" with comparatively fewer nonwage benefits such as health insurance and pensions. O'Neill and O'Neill (1997) add that when employment is intermittent, the jobs held are less likely to provide on-the-job training and to build skills that are rewarded in the labor market through increasing earnings. Building employment skills is essential, but first participants must have a base from which to build them. In the majority of cases, low-wage, low-skilled employment is not going to provide this base. The performance incentives introduced to welfare-to-work programs should also be designed to encourage human capital development and foster long-term earnings capacities among more disadvantaged TANF recipients. This study showed that the use of performance incentives in service provider contracts was particulary effective in improving the outcomes of more disadvantaged participants (i.e., the JOBS subgroup.) On the contrary, these and related findings also suggest that 22

JTPA performance incentives do not correlate well with measures of program gains or impacts. Heckman (forthcoming, 1998) shows that neither measures of employment and earnings at the time of program discharge nor the same measures taken 90 days later are positively correlated with long-term program impacts. Orr et al. (1997) also found a weak relationship between estimated program impacts and measured performance (e.g., monthly earnings and hours worked) in the National JTPA Study. While their exact form is still vague, the four performance measures planned for TANF programs seem superior to those used in JTPA programs. The TANF measures, intended to evaluate programs' success in moving recipients into jobs and toward self-sufficiency, include: (1) "job entry rates," (i.e., the proportion of adult welfare recipients entering employment for the first time each year, whether or not they leave the welfare rolls), (2) measures of "success in the work force," based on job retention and improvements in earnings, (3) improvement in job entry rates, and (4) improvement in work-force success. Although JTPA performance standards also include job entry rates and job retention over a 90-day period, earnings are measured in terms of levels at a point in time, rather than improvements relative to participants' pre-program earnings. If job retention in welfare-to-work programs is measured over a period longer than 90 days, (e.g., six months to one year), this measure may also prove to be better than that used in JTPA programs. 6.2

Policy implications A growing consensus is emerging from welfare-to-work evaluations that the more disadvantaged welfare recipients are failing to retain jobs and are not earning enough to rise above the poverty level, even when working full-time (Harris and Edin, 1998; Blank, 1997; Pawasarat, 1997; Burtless, 1995; Friedlander and Burtless, 1995; Glazer, 1995.) Most of these programs have not provided formal education and training services. In fact, in Milwaukee County where the Wisconsin Works (W-2) welfare reform program is now underway, thousands of welfare recipients have dropped out of technical colleges to satisfy work requirements.2 The "work-first" approach established under PRWORA may definitely be moving people to work, but it does not appear to be moving the more disadvantaged TANF recipients toward economic self-sufficiency. The findings of this study suggest that if afforded access, on-the-job training and vocational training in technical/professional fields might substantially and cost-effectively increase the labor market success of the more disadvantaged TANF recipients. Before or while they are engaged in these training 2

Since the 1995-'96 school year when the W-2 precursor, Pay for Performance, went into effect, enrollments of welfare recipients declined over 95 percent from 6,455 to 274 in the fall of 1997 when W-2 officially began (Johnson, 1998.)

23

activities, however, they need to acquire the basic educational skills that will enable their success in training and unsubsidized employment. Younger participants with little or no work experience or labor attachment, (and lower opportunity costs of participating), should be targeted for these services. At least one demonstration program has shown that when working closely with employers and utilizing community resources, basic education can be cost-effectively combined with on-the-job and vocational training. Other research examining the effectiveness of these training services over a longer period also suggests that we can expect the gains from more intensive program services to grow over time. Based on these study findings, I make the following suggestions for TANF programs. States should consider waiving "work-first" requirements for TANF recipients with minimal or no work experience and/or basic skills deficiencies and who struggle to find employment that enables them to support their families. In this study, JTPA participants with these characteristics performed much more poorly in the labor market when they received less intensive services designed to get them employed as quickly as possible. For these individuals, on-the-job training or vocational training, (in combination with basic education, if necessary), should be made available to increase their basic, occupational and job-specific skills levels and to prepare them for better-paying jobs. States should support participation in vocational training programs for as long as two years, to facilitate training opportunities in vocational programs that build professional/technical skills with higher labor market payoffs. In most JTPA programs, participants are enrolled for less than one year, which constrains the types of training programs they may enter. This may also explain, in part, why some individuals enrolled in JTPA multiple times, (the majority of whom received vocational training during first and second enrollments), and realized increases in earnings and retention of employment retention over subsequent enrollments. Contrary to the low-skilled, transitional or community service jobs these individuals might otherwise be obliged to take,3 these training opportunities could build the skills levels of the more disadvantaged and their long-term potential earnings growth as well. Although basic/remedial education is not cost-effective when provided alone, for the most disadvantaged among TANF recipients, further skills-building training may not be possible without the prior or concurrent provision of basic education services. The need for remedial education among adults, viewed as a financial burden by both public and private sector entities, manifests the deficiencies in our public education system. For example, in Chicago, 96 out of every 100 public school graduates who take the placement test to enroll in degree programs at Chicago's City Colleges need one or more remedial education courses before beginning college-level coursework (Jones, 1998.). Very few private sector employers have the resources to provide basic skills training themselves, and particularly among smaller businesses, some entry-level positions go unfilled for lack of individuals with the minimum 3

As of March 31, 1998, over 70 percent of Wisconsin's W-2 participants were placed in community service or transitional jobs.

24

capabilities necessary to acquire job-specific skills. Federal and state governments must bear the costs of our public failure to provide individuals with basic reading, math and writing skills that enable them be productive workers and citizens. While the government should provide additional support to the more disadvantaged in obtaining basic skills and vocational training, the private sector should have a more formative role in developing and improving on-the-job training programs and policies for TANF recipients. Small businesses seem to have been neglected in JTPA programs, partly because many are unaware of their potential for participating in service delivery, and also because of a lack of coordination between local program administrators and private sector businesses that was not entirely ameliorated by the establishment of Private Industry Councils. Cooperative efforts between public and private sector organizations are essential to making on-the-job training (and other policies to increase job-specific training opportunities) more cost-effective. In a new, edited work, Freeman, Gottschalk and their contributors (1998) discuss some "demand-side" policies that might also encourage employers to provide more on-the-job training to less-skilled, disadvantaged workers. In the Freeman and Gottschalk book, the importance of increasing the skills levels of lowskilled, low-paid workers through education and training emerges in nearly every essay. While some demand-side policies might contribute to some short-run wage and employment gains for these individuals, in the long-run, only education and training will significantly improve their earnings and employment potential. The PRWORA legislation emphasizes "work-first," and while it does not discourage state-level education and training investments, the experiences of some welfare recipients suggest that the work requirements imposed on most TANF recipients will decrease education and training investments among those who most need them and for whom, this research shows, they would be most cost-effective.

25

REFERENCES

Ashenfelter, Orley. "Estimating the Effect of Training Programs on Earnings." Review of Economics and Statistics, 1978, 67: pp. 47-57. Beecroft, Erik and Larry Orr. "Why Do Earnings Gains for Welfare Recipients not Lead to Benefits Reductions? Evidence From the National JTPA Study." Presented at the Association for Public Policy Analysis and Management Annual Research Conference, November 1996. Berg, Linnea, Lynn Olson, and Aimee Conrad. "Causes and Implications of Rapid Job Loss Among Participants in a Welfare-to-work Program." Presented at the Annual Research Conference of the Association for Public Policy and Management. Bethesda, MD. October 1991. Blank, Rebecca M. It Takes A Nation: A New Agenda for Fighting Poverty. New York: Russell Sage Foundation, 1997. Blank, Rebecca M. "Outlook for the U.S. Labor Market and Prospects for Low-Wage Entry Jobs." In The Work Alternative: Welfare Reform and the Realities of the Job Market, Demetra Smith Nightingale and Robert H. Haveman, (eds.). Washington, D.C.: The Urban Institute Press, 1995. Bowman, William R. "Evaluating JTPA Programs for Economically Disadvantaged Adults: A Case Study of Utah and General Findings." Report prepared for the National Commission for Employment Policy and Utah Department of Commerce and Economic Development. June 1993. Brodkin, Evelyn Z. "Inside the Welfare Contract: Discretion and Accountability in State Welfare Administration." Social Service Review, March 1997: 1-33. Cook, Thomas D. and Donald T. Campbell. Quasi-Experimentation. Boston: Houghton Mifflin Company, 1979. Courty, Pascal and Gerald Marschke. "Do Incentives Motivate Organizations? An Empirical Test." In Performance Standards in A Government Bureaucracy: Analytical Essays on The JTPA Performance Standards System, ed. James J. Heckman. Kalamazoo, MI: W.E. Upjohn Institute, forthcoming, 1998. Dresang, Joel. "Welfare's Plunge Raises New Questions." Milwaukee Journal Sentinel, December 16, 1997, pp. 1, 16A. 26

Freeman, Richard B. and Peter Gottschalk, (eds.). Generating Jobs: How to Increase Demand for Less-Skilled Workers. New York: Russell Sage Foundation, 1998. Friedlander, Daniel, David H. Greenberg, and Philip K. Robins. "Evaluating Government Training Programs for the Economically Disadvantaged." Journal of Economic Literature, December 1997, 35(4): 1809-1855. Friedlander, Daniel, James Riccio and Stephen Freedman. "GAIN: Benefits, Costs and ThreeYear Impacts of a Welfare-to-Work Program." New York: Manpower Demonstration Research Corporation, 1994. Friedlander, Daniel and Gary Burtless. Five Years After: The Long-Term Effects of Welfareto-Work Programs. New York: Russell Sage Foundation, 1995. Glazer, Nathan. "Making Work Work: Welfare Reform in the 1990s." In The Work Alternative: Welfare Reform and the Realities of the Job Market, Demetra Smith Nightingale and Robert H. Haveman, (eds.). Washington, D.C.: The Urban Institute Press, 1995. Greenberg, David and Michael Wiseman. "What Did the Work-Welfare Demonstrations Do?" Institute for Research on Poverty Discussion Paper No. 969-92, April 1992. Harris, Kathleen M. and Kathryn Edin. "From Welfare to Work and Back Again: A Quantitative and Qualitative Perspective." Presented at the Northwestern/University of Chicago Joint Center for Poverty Research Seminar, February 1998. Heckman, James J. "Do Short Run Performance Goals Predict Long Run Impacts?" In Performance Standards in A Government Bureaucracy: Analytical Essays on The JTPA Performance Standards System, ed. James J. Heckman. Kalamazoo, MI: W.E. Upjohn Institute, forthcoming, 1998. Heckman, James J., Neil Hohmann, Michael Khoo and Jeffrey Smith. "Did We Learn the Right Lesson from the National JTPA Study? Substitution Bias in Social Experiments." Unpublished manuscript, The University of Chicago, 1997. Heckman, James J. and V. Joseph Hotz. "Choosing Among Alternate Nonexperimental Methods for Estimating the Impact of Social Programs." Journal of the American Statistical Association, December 1989, 84(408): pp. 862-874. Heckman, James J. and Rebecca L. Roselius. "Evaluating the Impact of Training on the Earnings and Labor Force Status of Young Women: Better Data Help A Lot." Unpublished manuscript, The University of Chicago, 1994. Heckman, James J. and Jeffrey A. Smith. "Experimental and Nonexperimental Evaluation." In International Handbook of Labor Market Policy and Evaluation, G. Schmidt, 27

J. O'Reilly and K. Schomann, (eds.) Cheltenham, U.K.: Elgar Publishers, 1996. Heckman, James J. and Jeffrey A. Smith. "Assessing the Case for Social Experiments." Journal of Economic Perspectives, Spring 1995, 9(2): pp. 85-110. Heckman, James J. and Jeffrey A. Smith. "Ashenfelter's Dip and the Determinants of Participation in a Social Program." Unpublished manuscript, The University of Chicago, 1994. Heinrich, Carolyn J. "Returns to Training for the 'Hard-Core' Unemployed: What Does It Take to Make an Impact?" Forthcoming, Evaluation Review. Heinrich, Carolyn J. "The Role of Performance Standards in JTPA Program Administration and Service Delivery at the Local Level." In Performance Standards in A Government Bureaucracy: Analytical Essays on The JTPA Performance Standards System, ed. James J. Heckman. Kalamazoo, MI: W.E. Upjohn Institute, forthcoming, 1998. Heinrich, Carolyn J. and Annie Zhang. "The Costs of Training: Estimates from Employment and Training Programs Under the Job Training Partnership Act." Unpublished manuscript, The University of Chicago, 1998. Heinrich, Carolyn J. "Public Policy and Methodological Issues in the Design and Evaluation of Employment and Training Programs at the Service Delivery Area Level." Dissertation, The University of Chicago, December 1995. Jacobson, Louis S., Robert J. LaLonde and Daniel G. Sullivan. "The Returns from Community College Schooling for Displaced Workers." Federal Reserve Bank of Chicago Working Paper Series (WP-97-16), December 1997. Johnson, Mike. "In W-2 and Forced Out of School." Milwaukee Journal Sentinel, February 8, 1998, Section B, pp. 1, 3. Jones, Patrice M. "A Help or A Hindrance?" Chicago Tribune, May 3, 1998, Section 4, pp. 1, 3. Kornfeld, Robert and Howard S. Bloom. "Measuring Program Impacts in Earnings and Employment: Do UI Wage Reports from Employers Agree With Surveys of Individuals?" Joint Center for Poverty Research working paper, 1997. LaLonde, Robert J. "The Promise of Public Sector-Sponsored Training Programs." Journal of Economic Perspectives, Spring 1995, 9(2): 149-168. Meyers, Marcia K. "Gaining Cooperation at the Front Lines of Service Delivery: Issues for the Implementation of Welfare Reform." Presented at the Association for Public Policy 28

Analysis and Management Nineteenth Annual Research Conference, November 1997. Olson, Krista and LaDonna Pavetti. "Personal and Family Challenges to the Successful Transition from Welfare to Work." Prepared for the Office of the Assistant Secretary for Planning and Evaluation and the Administration for Children and Families. The Urban Institute, May 1996. O'Neill, Dave M. and June Ellenoff O'Neill. Lessons for Welfare Reform: An Analysis of the AFDC Caseload and Past Welfare-to-Work Programs. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 1997. Orr, Larry, Howard Bloom, Stephen Bell, Winston Lin, George Cave and Fred Doolittle. The National JTPA Study: Impacts, Benefits, and Costs of Title IIA. Bethesda, MD: Abt Associates, 1995. Orr, Larry, Burt S. Barnow, Robert I. Lerman and Erik Beecroft. Follow-Up Analyses of the National JTPA Study Sample. Final report prepared for the U.S. Department of Labor, July 1997. Pawasarat, John. "Employment and Earnings of Milwaukee County Single Parent AFDC Families: Establishing Benchmarks for Measuring Employment Outcomes Under 'W-2'." University of Wisconsin-Milwaukee Employment and Training Institute, 1997. Pawasarat, John and Lois M. Quinn. "Evaluation of the Wisconsin WEJT/CWEP Welfare Employment Programs." University of Wisconsin-Milwaukee Employment and Training Institute, April, 1993. Riccio, James A. and Alan Orenstein. "Understanding Best Practices for Operating Welfareto-Work Programs." Evaluation Review, February 1996, 20(1): 3-28. Romero, Carol J. "JTPA Programs and Adult Women on Welfare: Using Training to Raise AFDC Recipients Above Poverty." National Commission for Employment Policy, Research Report No. 93-01, June 1994. Salamon, Lester M. "The Marketization of Welfare: Changing Nonprofit and For-Profit Roles in the American Welfare State." Social Service Review, March 1993: pp. 16-39. Spalter-Roth, Roberta M., Heidi I. Hartmann and Linda Andrews. "Combining Work and Welfare: An Alternative Anti-Poverty Strategy." A Report to the Ford Foundation from the Institute for Women's Policy Research, April 1992. Trott, Charles E. and John Baj. "An Analysis of Repeating in JTPA Illinois." Report Prepared for the Illinois Department of Commerce and Community Affairs, 29

Northern Illinois University, August 1993. United States General Accounting Office. "Welfare Reform: States' Early Experiences With Benefit Termination." Report to the Ranking Minority Member, Committee on Finance, U.S. Senate, May 1997.

Ziliak, James P., David N. Figlio, Elizabeth E. Davis and Laura S. Connolly. "Accounting for the Decline in AFDC Caseloads: Welfare Reform or Economic Growth?" University of Wisconsin-Madision Institute for Research on Poverty discussion paper, 1997.

30