APPLICATION OF THEORY-BASED EVALUATION ON A ...

APPLICATION OF THEORY-BASED EVALUATION ON A VOLUNTARY NONFORMAL ADULT EDUCATION PROGRAM

by WILLIAM GARY HUBBARD (Under the direction of Lorilee R. Sandmann) ABSTRACT This purpose of this study was to investigate the relationships of mediating, moderating, and outcomes elements of a theory-based impact evaluation on a voluntary nonformal adult education program. Mediators (knowledge change, use of informal education, use of professional assistance and products, use of social networks) and outcomes (forest management activity) were measured and their relationships studied. Several moderators (age, gender, education, etc.) were also measured to determine influences on mediators and outcomes. A multiple mediator-moderator model was developed to measure these relationships. Six hundred and forty-seven participants of the 2004 Master Tree Farmer Program served as the population for this study. The response rate was 38%. Mean age of the respondents was 61 years. Eighty-five percent of the respondents were male. Ninetysix percent of the respondents were Caucasian. Seventy-five percent of the respondents earned at least a Bachelor’s degree, and 58.4% had at least an annual household income of $75,000.

Key findings include that knowledge change was a powerful predictor of increased forest management activity and explained 32% of its variance. Mean increases for the four mediator variables and one outcome variable ranged from low to moderate on a scale of 1 (no change) to 4 (substantial change). Of fifteen moderator variables studied, age class and the importance of the non-economic objectives of managing for wildlife, for recreation and beauty, and for the next generation were the only statistically significant predictors of change in mediator and outcome variables. There high statistical correlations between the mediator variables and outcome (r’s greater than or equal to .57). Based on these findings, recommendations for research and practice include: the need for qualitative research regarding the relationships between knowledge change and mediators, additional use of program theory-based models and evaluation techniques to better account for feedback loops and causality, and increased interaction with participants following program participation to understand motivations and barriers to use of mediators.

INDEX WORDS:

program theory-based evaluation, mediator variables, moderator variables, impact evaluation, program evaluation, non-industrial private forest management, Extension, voluntary nonformal adult education.

APPLICATION OF THEORY-BASED EVALUATION ON A VOLUNTARY NONFORMAL ADULT EDUCATION PROGRAM

by

WILLIAM GARY HUBBARD B.S., The University of Florida, 1985 M.S., The University of Florida, 1987

A Dissertation Submitted to the Graduate Faculty of The University of Georgia in Partial Fulfillment of the Requirements for the Degree

DOCTOR OF PHILOSOPHY

ATHENS, GEORGIA 2010

© 2010 William Gary Hubbard All Rights Reserved

APPLICATION OF THEORY-BASED EVALUATION ON A VOLUNTARY NONFORMAL ADULT EDUCATION PROGRAM

by

WILLIAM GARY HUBBARD

Electronic Version Approved: Maureen Grasso Dean of the Graduate School The University of Georgia May 2010

Major Professor:

Lorilee R. Sandmann

Committee:

Bradley C. Courtenay Khalil M. Dirani Ben D. Jackson

ACKNOWLEDGEMENTS Through the ups and downs of the doctoral career of this ‘Student Over the Traditional Age’, or ‘SOTA’, I have had several chances to think of individuals who I would like to acknowledge and thank. First and foremost is my loving wife Joni, who listened to my ideas, encouraged me during the difficult times, and pushed me to ‘finish what you started’! She never doubted me, even when I doubted myself. My four little boys, now young men, have also had to put up with a lot over the last several years and I am certain they are thankful to have their Dad’s full mental capacity back (if that was ever possible). Liam, Duncan, Keenan, and Sean, you guys are the greatest. I could not have asked for a better major professor and advisor than Dr. Lorilee Sandmann. She has been a tremendously strong, positive force throughout this whole process. As we reviewed and researched more than half a dozen potential evaluation theories and frameworks, her expertise in Extension and understanding of the Engaged Institution philosophy, were invaluable resources to draw upon. Thank you, Dr. Sandmann. In addition to Dr. Sandmann, I was fortunate enough to have five other members of my doctoral committee who took the time and effort to provide much needed guidance and oversight. These were Drs. R. Cervero, B. Courtenay, K. Dirani, B. Jackson, and T. Valentine. There is a saying that things happen for a reason, and I believe that because I had the opportunity to interact with not just four committee members over the last several

iv

years, but six, resulted in a much better research study and dissertation. Thank you to everyone. Others who provided assistance, advice and encouragement included Dr. George Kessler (Professor-Emeritus, Clemson University), Dr. George Hernandez (Forest Regeneration Specialist, USDA Forest Service, soon to be Dr. Kris “Burbs” Irwin (Service Learning Professional, Warnell School of Forestry & Natural Resources, UGA), and my staff with the Southern Regional Extension Forestry office, especially Candee Golden and Whitney Howell, who assisted with survey deployment, data entry, and checking. Also, Dr. Jien Chen and Dr. Jaxk Reeves with the UGA Department of Statistics Consulting Center were helpful with advice concerning proper statistical analyses. Thank you to all. In addition, this project would not have been possible without the financial support of the USDA Forest Service. Last but not least, I’d like to acknowledge the influence of my parents. As Wilbur Wright said it best in 1910: “If I were to be giving a young man advice as to how he might succeed in life, I would say to him ‘pick out a good father and mother, and begin life in Ohio’.” Thank you!

v

TABLE OF CONTENTS Page ACKNOWLEDGEMENTS............................................................................................... iv LIST OF TABLES............................................................................................................. ix LIST OF FIGURES .......................................................................................................... xii CHAPTER 1. INTRODUCTION .........................................................................................................1   Background of the Problem.......................................................................................2   A Case Study: The Master Tree Farmer Course .......................................................5   Program Evaluation and the Potential of Program Theory-Based Evaluation ..........7   Statement of the Problem ........................................................................................11   Purpose of Study and Research Questions ..............................................................12   Significance of the Study ........................................................................................12   2. REVIEW OF THE LITERATURE .............................................................................16   Introduction .............................................................................................................16   Program Evaluation Background and Schema ........................................................17   Program Evaluation Methods..................................................................................21   3. METHODOLOGY ......................................................................................................44   General Conceptual Framework..............................................................................44   Measurement Framework........................................................................................48   Description of the Program and the Program Theory .............................................51   Instrumentation........................................................................................................56  

vi

Finalizing the Pilot Survey Instrument....................................................................66   Pilot Study ...............................................................................................................68   Data Collection Strategy .........................................................................................70   Sample Identification and Sampling Procedure ......................................................71   Data Preparation ......................................................................................................75   Data Analyses Procedures .......................................................................................76   Limitations of the Study ..........................................................................................85   4. FINDINGS...................................................................................................................88   Findings Related to Research Question #1..............................................................88   Findings Related to Research Question #2..............................................................99   Findings Related to Research Question #3............................................................110   5. INTERPRETATION OF FINDINGS AND CONCLUSIONS .................................119   Summary of the Study...........................................................................................119   Interpretation of the Findings ................................................................................122   Conclusions ...........................................................................................................133   Implications for Practice .......................................................................................137   Implications for Theory and Research ..................................................................141   Recommendations for Future Study......................................................................142   Summary ...............................................................................................................144   REFERENCES ............................................................................................................... 145 APPENDICES ................................................................................................................ 157 A

SURVEY INSTRUMENT .............................................................................. 157

B

REQUEST FOR PARTICIPATION ............................................................... 164

C

SECOND REQUEST FOR PARTICIPATION .............................................. 166

D

FINAL REQUEST FOR PARTICIPATION .................................................. 168 vii

E

IMPLIED CONSENT FORM.......................................................................... 170

F

ITEM RESPONSE FREQUENCY TABLE: KNOWLEDGE CHANGE....... 172

G

ITEM RESPONSE FREQUENCY TABLE: USE OF INFORMAL EDUCATION .................................................................................................. 175

H

ITEM RESPONSE FREQUENCY TABLE: USE OF SOCIAL NETWORKS .......................................................................................................................... 178

I

ITEM RESPONSE FREQUENCY TABLE: USE OF PROFESSIONAL ASSISTANCE AND PRODUCTS.................................................................. 180

J

ITEM RESPONSE FREQUENCY TABLE: FOREST MANAGEMENT ACTIVITY....................................................................................................... 182

viii

LIST OF TABLES Page Table 1: Definition of Research Constructs..................................................................... 49   Table 2: Master Tree Farmer Session Topics and Objectives ......................................... 53   Table 3: Survey Items Measuring Knowledge Change ................................................... 58   Table 4: Survey Items Measuring Use of Informal Education ........................................ 59   Table 5: Survey Items Measuring Use of Professional Assistance and Products............ 61   Table 6: Survey Items Measuring Use of Social Networks............................................. 62   Table 7: Survey Items Measuring Forest Management Activity ..................................... 63   Table 8: Moderator (Personal and Contextual) Variables ............................................... 67   Table 9: Data Collection Steps ........................................................................................ 70   Table 10: Summary of Sample Composition of Personal Characteristics....................... 73   Table 11: Summary of Sample Composition - Forest Management Characteristics....... 74   Table 12: Summary of Sample Composition - Landowner Objective Characteristics.... 75   Table 13: Statistical Summaries and Reliability Measures.............................................. 77   Table 14: Mediator and Outcome Variable Intercorrelations .......................................... 83   Table 15: Rank Order List of Variables Based on Mean Item Mean .............................. 90   Table 16: Frequency Percentages by Response Category ............................................... 91   Table 17: T-Test Results Mean Comparisons (M=1, M=2.5) ......................................... 92   Table 18: Rank Order Listing of Knowledge Change Items ........................................... 94   Table 19: Rank Order Listing of Use of Informal Education Items ................................ 96  

ix

Table 20: Rank Order Listing of Use of Social Networks Items..................................... 97   Table 21: Rank Order Listing of Use Professional Assistance and Products Items ........ 97   Table 22: Rank Order Listing of Forest Management Activity Items ............................. 98   Table 23: Correlation Between Age Class and Knowledge Change ............................. 102   Table 24: Regression Results for Predicting Knowledge Change ................................. 103   Table 25: Correlations Between Moderator Variables and Use of Informal Education 104   Table 26: Regression Results for Predicting Use of Informal Education...................... 105   Table 27: Correlations of Independent Variables with Use of Professional Assistance and Products........................................................................................................................... 106   Table 28: Regression Results for Predicting Use of Professional Assistance and Products ......................................................................................................................................... 106   Table 29: Correlations of Independent Variables with Use of Social Networks........... 107   Table 30: Regression Results for Predicting Use of Social Networks........................... 108   Table 31: Correlations of Independent Variables and Forest Management Activity .... 109   Table 32: Regression Results for Predicting Forest Management Activity................... 109   Table 33: Correlations Between Knowledge Change and Forest Management Activity ......................................................................................................................................... 110   Table 34: Regression Results for Predicting Forest Management Activity................... 111   Table 35: Correlations of Knowledge Change and Independent Intermediate Mediators ......................................................................................................................................... 112   Table 36: Regression Results for Predicting Use of Informal Education...................... 113   Table 37: Regression Results for Predicting Use of Professional Assistance and Products ......................................................................................................................................... 113  

x

Table 38: Regression Results for Predicting Use of Social Networks........................... 113   Table 39: Correlation of Intermediate Mediator Variables and Forest Management Activity ........................................................................................................................... 115   Table 40: Summary of Major Test Statistic Results ...................................................... 117   Table 41: Significant Correlation Coefficients (and Percent Variance Explained)....... 131   Table 42: Significant Intermediate Mediator-Outcome Correlation Coefficients ......... 132  

xi

LIST OF FIGURES Page Figure 1: Basic Program Theory Model (Chen, 2005) ...................................................... 8   Figure 2: Conceptual Framework for the Study (Adapted from Donaldson, 2007) ........ 46   Figure 3: Multiple Mediator-Moderator Impact Model: The Master Tree Farmer Program ........................................................................................................................................... 50   Figure 4: Distribution of Knowledge Change Performance Index .................................. 78   Figure 5: Distribution of Use of Informal Education Index ............................................ 79   Figure 6: Distribution of Use of Professional Assistance and Products Index ................ 80   Figure 7: Distribution of Use of Social Networks Index ................................................. 81   Figure 8: Distribution of Forest Management Activity Index ......................................... 82   Figure 9: Proximal Mediator, Intermediate Mediators and Outcome Variables to be Measured........................................................................................................................... 89   Figure 10: Summary of Moderator, Mediator and Outcome Relationships .................. 100   Figure 11: Proximal Mediator-Outcome Relationship .................................................. 110   Figure 12: Proximal Mediator-Intermediate Mediators Relationship............................ 112   Figure 13: Intermediate Mediator-Outcome Relationship............................................. 114   Figure 14: Multiple Mediator-Moderator Impact Model Results: The Master Tree Farmer Program........................................................................................................................... 123  

xii

CHAPTER 1 INTRODUCTION Education is a principal means by which adults are exposed to new ideas, concepts and knowledge. Adult education programs exist in society to meet the demands of adults wishing to increase their knowledge. From a human behavior perspective, these educational programs are posited to influence knowledge, self-efficacy, and empowerment, that affect attitude, skills, decision-making and, ultimately, adoption of one or more desirable practices or behaviors. Basic evaluation techniques have been developed to measure program outputs and outcomes (Bennett, 1977, 1995; Bennett & Rockwell, 1995; Boone, 1989; Cason, 2005; Muth & Hendee, 1980). However, theory and practice are still looking to provide a more holistic and defensible account of how educational programs influence outcomes. Evaluation theory and practice have matured over the years to include concepts and tools that better account for the dynamic and complex nature of adult education programs. (Alston & Reding, 1998; Cervero, Rottet, & Dimmock, 1986; King & Rollins, 1995). Factors that have been statistically tested for their influence on program outcome have included those that relate to the individual, the educational program, and the social environment in which the program participant lives and works. Factors that have not been fully investigated however, include post-programming influences of learning, and selfempowerment activities such as the pursuit of additional information, social network participation, and the use of professional assistance and professionally developed

1

educational products. An understanding of the extent to which factors such as these postprogram activities occur, and their relationship to program outcomes can provide evidence that an educational program can stimulate participants to become active learners and to take empowered action with their newfound knowledge. The personal and contextual manner in which these activities and relationships take place can also inform administration, planning, implementation, and evaluation of adult education programs. Background of the Problem Since the inception of adult education as a recognized field of study, programs have been classified in several ways. These have included categories such as formal, nonformal, and informal adult education (Merriam & Brockett, 2007). Formal education is considered to originate from existing established organizations such as colleges and universities, vocational and technical schools and other institutions. Formal education often includes the granting of degrees, credits or official certificates following participation and testing. Nonformal education exists in many parts of the world as “any organized educational activity outside the established formal system….that is intended to serve identifiable learning clienteles and learning objectives” (p. 11) (Coombs, Prosser, & Ahmed, 1973). Informal education is similar to nonformal education in that it takes place outside the established formal systems but it is different in that it involves unorganized and unplanned activities (from the program provision perspective) such as reading, listening to audio tapes, viewing video, conversations with friends and neighbors and other learning experiences (Merriam & Brockett, 2007). Informal education from the perspective of the individual may range from intentional, self-directed learning, to incidental or unanticipated knowledge gain via interactions with others (Cross, 2007).

2

Adult education has also been described as being either compulsory, mandatory or voluntary (Lowenthal, 1981). Whereas the laws of a government require compulsory education, mandatory education is often necessitated by the requirements of a profession. Voluntary education on the other hand is neither and its provision is due to the demand for, and supply of, such services. One common type of program is the voluntary nonformal adult education program. This approach includes programs designed for those who wish to utilize a structured format to improve their knowledge of a subject matter area. These consist of classes, courses, workshops, training, mentoring, or a host of other activities. In many cases, behavior change in the form of an adoption of one or more desired practices or skills is considered the program outcome (Barao, 1992; Beal, Bohlen, & Anderson, 1962; Bennett, 1995). The Cooperative Extension Services (Extension) of the land-grant university system are among the agencies involved in the creation and delivery of both voluntary nonformal adult education and avenues for pursuit of informal education activities. In addition, there is recognition of the importance and need for social networks and resource professionals who can offer more specialized assistance to clients (Wise & Ezell, 2003). These features of Extension: nonformal programming, informal educational tools, and guidance towards social networks and additional professional assistance, offer a menu of alternatives for the interested, motivated adult learner yet their affect on adoption of practice has not been studied to any noticeable extent. Forestry Extension provides an illustrative example of how nonformal and informal education, use of social networks, and professional assistance are important considerations when planning and

3

implementing a voluntary nonformal adult education program, how they can influence outcome, and how they have not been studied to any extent. First, due to the detailed and complex nature of forest management, landowners who participate in educational programs are often encouraged to learn more about specific subject matters of interest. Many forestry Extension programs have facilitated this self-directed educational process by developing publications, Internet sites, videos, television programs, decision support tools, and other products from the general to the specific. These products have been made available in a wide array of formats that range from traditional print and video to new online digital forms. Extension forestry has also facilitated knowledge gain and behavior change by encouraging landowners to participate in social networks. Many state and county Extension units have been involved in organizing and leading county forest landowner associations, and developing peer-to-peer networks and other options to increase social interaction. Furthermore, the existence of online mechanisms such as social media networks, websites, listservs and web portals have added to the menu of alternatives for landowners to become more involved with others who have mutual interest, needs and concerns. Additionally, Extension forestry promotes the use of professional assistance by providing pathways for landowners to engage and use the services of those experts knowledgeable in various aspects of forest management. Forestry Extension often promotes the use of state forestry agency representatives, forest industry, finance, tax and legal expertise and others for technical assistance and regularly provides directories of professionals to interested landowners.

4

Another factor to consider in forestry Extension programming is the tendency for lengthy time delays between an educational programming experience and the implementation of a forest management practice. It may be several months or even years before a landowner actually puts into place a practice learned during an educational program. This time difference increases the importance of keeping the program participant engaged with informal education, use of social networks, and professional assistance options lest they lose interest or become distracted by other activities. Finally, forestry Extension program participants are a diverse group with varying backgrounds, circumstances, knowledge bases, worldviews, and landholding objectives. There are several factors such as age, education, land tenure, personality characteristics, ownership objectives, etc. that have been tested with regard to their possible influence on forest management behavior (Butler, et al., 2007; Haymond & Baldwin, 1988; Lorenzo & Beard, 1996). A review of the literature in forestry revealed that program evaluation inclusive of the influence of factors such as these post program activities and personal and contextual factors was not evident. Extension forestry, and many adult education programs for that matter, have been conducted for decades with little thought to relevant post program activities, and the influence of personal and contextual characteristics. A Case Study: The Master Tree Farmer Course The case used in this research study illustrates the potential utility of studying the relationships between adult education programming, post program activities and personal and contextual factors that can relate to program outcome. It also provides the opportunity to develop a model to investigate the importance of these factors and their

5

relationships to program outcomes. In the year 2000, a live, forestry satellite program was launched from the studios of Clemson University in Clemson, South Carolina. It was an Extension-sponsored event targeted at private forest landowners in the southeastern United States. The program involved twenty-one hours of instructional material and interaction with local professionals. The program’s objectives were to provide a detailed overview of various forestry principles and practices. In addition, special attention was given to providing an awareness of the types and sources of additional educational information, use of social networks, and professional assistance and products availability. The goal of the program was to make aware, educate, and motivate. Final desired program outcomes included increased forest management activity and improved forest conditions on the ground. These conditions were assumed to be beneficial for both individuals and society. In total, the program reached a documented level of over 10,000 private owners in at least 12 states between the years 2001 and 2007 (Kessler, 2004, personal communication). Exit surveys by program participants have rated the program very highly over the years with a majority responding that they felt their participation was worthwhile. Many participants planned to implement a forestry practice on their land in the future based on what they had learned. A one-year post workshop survey was conducted on participants in South Carolina with positive results. A high percentage of participants (over 90%) had completed at least one forestry practice (Kessler, 2002, unpublished evaluation results). These results, while useful from a basic impact perspective, failed to provide information on the processes and factors that followed the educational program and their relationship to program outcomes. The dynamics of the

6

relationship between any knowledge change that occurred, any informal education, use of social networks, and use of professional assistance that took place and information relating to the participant’s unique personal situation can provide more insight into the understanding of the impact of the program. This example of the Master Tree Farmer course highlights a program with defined objectives, coordinated stakeholder input, and a delivery strategy that utilized new technologies. It also highlights a program suitable for inquiry into any knowledge change, use of informal education, use of social networks, and use of professional assistance that results directly or indirectly from the program and whether these factors are related to program outcomes. Program Evaluation and the Potential of Program Theory-Based Evaluation Traditional program evaluation has typically involved a one input (education program), one outcome (change in behavior) model (Abrussese, 1987). Within the last few decades, more holistic and comprehensive evaluation frameworks have emerged to better map the dynamic relationships between educational programs, intervening and contextual factors, and program outcomes (Rogers, 2007). These frameworks have included discussion and testing of new theories and concepts involving program evaluation. Early evaluation research textbooks included discussion of the need to surface assumptions, record activities that are conducted, record the effect of each activity that is conducted and follow through with formal evaluations (Suchman, 1967; Weiss, 1972). A majority of program evaluation work, including those involving continuing and adult education programs, however, have still focused on final program outcomes with very little attention to the intervening mechanisms that may influence program outcome and the dynamic relationships in the system to be evaluated (Chen, 2005).

7

One solution to better understanding the dynamics of programs has been the introduction of program theory and program theory-based evaluation (Donaldson, 2007). Program theory and its related theory-based evaluation framework provide the flexibility to incorporate components of many evaluation frameworks, concepts, methods and models . A general program theory model can be summarized graphically in Figure 1. This change model includes a theory or set of theories that drives various key components of the program. Through implementation of the program via goals and objectives, the program theory presumably works as planned and the social intervention will succeed.

Figure 1 Basic Program Theory Model (Chen, 2005)

The Intervention box can be defined as the bundle of services or activities that are undertaken by program managers to achieve the pre-specified program goals. This includes the action model component of the program theory. The action model includes the details and activities necessary to implement the intervention. The Determinants box includes the mediator factors and the moderator factors. These factors have been 8

identified via the predetermined program theory as having an influence on program outcome. Specifically, mediator factors are believed to influence program outcome through implementation of the change aspect of the model (Donaldson, 2007). Examples of mediator factors include knowledge change, improved self-efficacy, empowerment, involvement in additional programs, etc. Mediator factors, or simply ‘mediators’ are also known as intervening mechanisms or proximal or intermediate outcomes (Chen, 2005; Donaldson, 2007). Moderator factors, or simply ‘moderators’ are those that affect program outcome but are considered extraneous and uninfluenced by the program itself. Examples of moderators include age, education, income, personality, gender and a host of other factors. The inclusion of determinants such as mediators and moderators provides one of the major strengths and improvements of this framework over previous evaluation frameworks (Chen, 2005). These points will be explained in more detail in the Review of the Literature and Methodology chapters. Finally, the Outcomes box can be defined as the results and impacts of the intervention (behavior changes such as control of an addictive habit, acquisition of a new skill, etc.) (Donaldson, 2007). Program theory offers a framework to plan and implement many aspects of social programs such as those offered by adult education organizations such as Extension. Program theory-based evaluation in turn, offers a framework for inclusion and examination of the importance of various mediators and moderators posited to influence program outcomes (Gascon, 2006). A practical application of program theory, the logic model offers one example from which to implement both a program’s theory and to evaluate it (Wasserman, 2010).

9

In its simplest sense, the logic model consists of defining program inputs, activities, outputs and outcomes; concepts that are very similar to those in program theory-based evaluation. Evaluations can be conducted through a single methodology to identify and measure the actual inputs, activities, outputs and outcomes (Torghele, et al., 2007) or to measure, through multi-method evaluation, the integrative nature of the model (Cooksy, Gill, & Kelly, 2001). It should be noted that logic modeling has focused on attainment of the various inputs, activities, outputs, and outcomes, and not on the measurement and relationships between mediators, moderators, and outcomes as is typically included in program theory-based evaluation. In addition, the review of the literature has resulted in no known inferences to informal education, use of social networks, or use of professional assistance as potentially key mediators to include in the models such as the logic model. The Cervero Model, used more specifically in the professional development field, also includes concepts within the realm of program theory by studying factors that influence program outcome. This model was used to test several categories of factors that were believed to influence performance. The model incorporates many aspects of the diffusion of innovations framework developed by Everett Rogers (Abrussese, 1987) and discussed more fully in the Literature Review chapter. The major components of the Cervero Model include groups of variables relating to the educational program under investigation, the proposed practice(s), the individual, and the work environment. This framework has been empirically tested, particularly in the continuing health profession field, with statistically significant results (Cervero & Rottet, 1984; Cervero, et al., 1986; Umble, Cervero, Yang, & Atkinson, 2000).

10

The Cervero Model’s focus on the content and delivery of the program, the individuals desire to learn, and the environmental context has contributed to the theoretical and practical discussion regarding a program’s theory and program planning. The model’s temporal scope involved short to intermediate-term impacts but did not include informal education, use of social networks, and professional assistance activities that may be undertaken by the participant following the educational program. Statement of the Problem Knowledge change, use of informal education, use of social networks, and use of professional assistance and products are several activities that, if undertaken by participants following course participation are presumed to influence the outcome of an educational program. In addition, factors such as age, race and gender are also believed to influence program outcome. The relationships of these mediators and moderators to program outcomes are important theoretical and practical aspects of adult education program evaluation research. Weenig and Midden’s (1991) in “Communication Network Influences on Information Diffusion and Persuasion” summarizes a key component of the problem: Finally, like research on persuasive communication, research on diffusion of innovations has almost exclusively investigated the final stage of the innovation decision process, that is, positive adoption decisions. The preceding stages of the innovation diffusion process, such as awareness of the innovation and the subsequent seeking of extra information have received little research attention. (p. 734)

11

In addition to the limited research on knowledge (i.e. ‘awareness’) change, and the seeking of information processes, a review of the literature on research concerning the relationships between social networks, professional assistance, and program outcomes also provided limited results. A comprehensive model inclusive of these variables would provide useful insight into whether these factors are important intervening mechanisms that influence program outcomes and whether components that make up the programs are successful in influencing these intervening mechanisms, especially in unique situations such as adult education in forestry. Purpose of Study and Research Questions The purpose of this research study was to investigate the relationships between mediators, moderators and outcomes of a voluntary nonformal adult education program. The following research questions were answered in this investigation: 1. To what extent was change exhibited in the mediators (knowledge change, use of informal education, use of social networks, use of professional assistance and products), and the outcomes (increased forest management activity) of a voluntary nonformal adult education program? 2. To what extent can variations in the mediators and outcomes be explained by identified moderators of a voluntary nonformal adult education program? 3. What relationships exist between the mediators and the outcomes of a voluntary nonformal adult education program? Significance of the Study This study was significant from a number of theoretical, policy, and practical or operational perspectives. From a theoretical perspective, this research study offered an

12

opportunity to advance discussion of the utility of program theory and program theorybased evaluation within the context of voluntary nonformal adult education programming. This will be done through the empirical testing of a type of program theory-based impact model. Specifically, a multiple mediator-moderator impact model (Donaldson, 2007) will be implemented to assess an adult education program with the goal of understanding the utility of the theory and identifying any gaps that may exist in the conceptual thinking of program theory-based evaluation. In addition to basic theoretical opportunities, much of higher education is beginning to more formally embrace the policy of the engaged institution. Key features of the engaged institution concept include partnerships, mutuality, reciprocity and collaboration (Kellogg Commission on the Future of State and Land Grant Universities, 2001).This type of research has policy implications for higher education’s engagement with communities of various types through the informal education, use of social networks and professional assistance (mediators) realm. Through the interaction with program participants before, during, and after the educational event, and a better understanding of personal and situational characteristics (moderators for example), the higher education professional opens up the desired ‘two-way street’ of communication, collaboration, and joint problem solving that characterizes an institution (Bull, Cote, Warner, & McKinnie, 2004). In addition, stakeholders who are in the position to affect program policy will have an improved understanding of the dynamics of the educational program system. This can lead to improvement in how programs are funded, planned, delivered and evaluated. From a practical perspective, a basic understanding of the importance, and relationships of mediators and moderators in evaluation research is useful. An

13

educational program’s impact might not be fully recognized if it fails to take into account the mediators or moderators that affect program outcomes. Practical questions involving the potential relationships between the mediators, moderators, and program outcomes are accounted for through this research. Information from studies such as this can cause educators, to be conscious about what happens following a program they developed and implemented. Significant resources, both human and fiscal are being invested in audience analysis, program planning and program evaluation. Typically, programs are developed based on one or more perceived needs. Awareness of the relationship between the theory behind the program, the program participant’s personal and contextual situations, and the participant’s post-event activities (mediators) and final program outcomes can lead to more effective programming. For example, forest owners might be taught a certain wildlife management practice but not given advice on sources of more information, connecting with others doing similar work on their land, or with professionals who can provide more assistance. Factors such as the availability of information or of assistance and social networks opportunities and professional assistance can be incorporated into the planning of the educational program if they are believed to be important determinants of program outcomes. Research such as this can be an impetus for the Extension educator to be more intentionally systematic in planning, delivering and evaluating programs by viewing processes that occur after the education program. Also, from a practical perspective, an understanding of moderator variables that influence outcome can be important from a programming perspective. Programs that are inclusive of age or gender differences can be planned and implemented.

14

Chapter 2 presents a summary of the literature reviewed to inform the research study. Literature relating to program evaluation, informal education, social networks, diffusion of innovations, and other related concepts is reviewed.

15

CHAPTER 2 REVIEW OF THE LITERATURE Introduction The purpose of this research study was to gain an understanding of the value and use of program theory-based evaluation on a voluntary nonformal adult education program. To conduct this research it was necessary to answer three research questions relating to defining and determining mediators, moderators and outcomes. These research questions were: a) to what extent did participants increase their knowledge, use of informal educational activities, use of social networks, use of professional assistance and products, and forest management activity due to or following participation in the educational program, b) to what extent can variations in knowledge change, use of informal education, use of social networks, use of professional assistance and products and forest management activity be explained by the moderator variables that are personal and contextual in nature; and c) what is the relationship between changes in knowledge, use of informal education, use of social networks, use of professional assistance and products, and forest management activity. The review of the literature focused on program evaluation theory, research and application. This involved literature pertaining to traditional approaches to evaluation as well as evaluation that included the broader dynamics of the system being evaluated. In addition, a historical perspective of diffusion of innovations and adoption of practice is included due to its potential utility in the field of program evaluation (Hubbard &

16

Sandmann, 2007). Research on this perspective is especially valuable for program evaluation because it involves human and social behavior and characteristics that are expected to have an affect on program outcomes. Program Evaluation Background and Schema Program evaluation can be conducted for a variety of reasons and can serve several purposes. Evaluations can be conducted to improve existing programs (formative evaluation), or summarize the results of a recently conducted program (summative evaluation). They can be conducted on various aspects of a program from the planning phase, to the implementation and outcome phase. They may consist of experimental design treatments, quasi-experiments, documentary or case studies, pre-post program knowledge gain surveys. Program evaluation has progressed in the last several decades to include both theoretical and practice concerns. The following review summarizes the literature on program evaluation from several different perspectives. Categorical Perspectives Frameworks and approaches to program evaluation have developed from several different perspectives over the years. One schema, outlined in a popular textbook on program evaluation includes six categories based on the orientation of the evaluation and the types of questions to be asked/answered. These include ‘objectives-based’, ‘management-based’, ‘consumer-based’, ‘expertise-based’, ‘adversary-based’ and ‘participant-based’ . Objectives-based program evaluation is useful for determining the extent to which a program’s outlined goals and objectives are achieved. Several techniques have been

17

used to implement objectives-based evaluation. Objectives-based program evaluation originated with Tyler’s approach to evaluation, which included several steps from goal establishment to collection and use of data to compare results with objectives. ‘Curriculum-based evaluation’, a closely related evaluation framework is used to achieve an understanding of how closely outcomes of a program track with a predefined curriculum (Howell, Fox, & Morehead, 1993). In this manner, if objectives or curricula can be identified and articulated, then assessment and evaluation questions or other techniques can be developed to address the outcomes and their match with the program’s intent. Management-based evaluation includes approaches that assist with management decision-making. These include evaluation procedures that look at organizational context, the inputs that are included, the process that is undertaken and the products that are produced. An evaluation framework that utilizes the management-based evaluation scheme is one with an acronym of CIPP (Context evaluation, Input evaluation, Process evaluation & Product evaluation) (Stufflebeam, 2000). From the consumer-based evaluation perspective, the evaluation is focused on evaluating the program or product from the user perspective. This perspective is unique in that the end-user is in control of the evaluation rather than the provider or a contractor chosen by the provider (Scriven, 1991). Preset standards or guidelines are often used in the evaluation and can include processes, content, transportability, and effectiveness (Worthen, Sanders, & Fitzpatrick, 1997). Expert-based evaluation utilizes the opinions and experiences of highly qualified and knowledgeable individuals to evaluate a program. In this framework, experts may or

18

may not contact end-users for their input. Experts would review program objectives and other relevant documentation and provide input based on their experience and expertise regarding the program (Worthen, et al., 1997). In adversary-based evaluation, individuals with potentially opposing opinions and worldviews would evaluate a highly charged or controversial program. This consensusbuilding effort is designed more for social programs than educational programs although certain social marketing campaigns including AIDS, teen pregnancy and abstinence programs include education, as a key component can be controversial in their approaches and objectives (Worthen, et al., 1997). Finally, participatory-evaluation involves a hands-on approach by participants and developers and includes multiple realities of the situation based on the lens’ of the participant (Fitzpatrick, Sanders, & Worthen, 2004) . The multiple reality approach to program evaluation is useful for complex programs with multiple potential outputs. It also includes participants in the program planning stage to ensure that their needs are met and can be evaluated. Another schema, includes goals-based, process/output, outcome/impact, or utilization-focused perspective (Patton, 2008). Goals-based evaluation, similar to objectives-based evaluation involves formal or informal queries of program developers, funding sources and others involved in the program development phase of the project. Questions relating to how goals were established, whether the personnel have adequate resources to implement the goals, current status of the goals in relation to accomplishments, and other related concerns, are typically answered through interviews,

19

one-on-one discussions, questionnaires and other mechanisms (McNamara, 1998). This type is similar to the objectives- and curriculum-based evaluations discussed in earlier. Process evaluation includes examining the processes involved with program design and logic, program implementation and evaluation (Rossi, Freeman, & Lipsey, 1999). In this manner, light can be shed on what went right and what went wrong during these phases. Output evaluation includes an accounting of the activities performed in a program, the numbers and types of participants, and their reactions. These numbers have provided useful information for stakeholders and administrators and are typically a major component of the logic model, which is discussed in further detail in the next section. Impact or outcome evaluation, as opposed to process or output evaluation, is defined as “establishing, with as much certainty as possible, whether or not an intervention is producing its intended effects” (Rossi, et al., 1999). These can be distinguished from the outputs discussed previously in that outcomes are based on a change in learning, action or condition. These outcomes are also a typical component of the logic model for programming. Impact evaluation can also be categorized as being short-term, intermediate-term, and long-term. In the short-term, change in awareness, knowledge, attitudes, skills, opinions, aspirations or motivations are measured. Intermediate-term outcomes are often those relating to an action. These are observable actions such as a change in behavior or the implementation of a practice that is seen as desired by the change agent. Other changes involve policy or social actions. Finally, long term-impacts are those affecting the individual, culture or society over the long run. These are social, economic, or

20

environmental impacts and can be calculated using modeling, expert predictions and long-term time series studies. Long-term impacts are difficult and expensive to measure, but can be very valuable for program justification and expansion. Forest, Brack and Moss also organized approaches according to assessment methods (Forest, Brack, & Moss, 1994). These include the objective-based model, the experimental model, adversary evaluation, decision-centered model and experiential evaluation. Program evaluation can also be categorized as being formative (for program improvement and enhancement), summative (program justification or continuation) or some combination of these two (Patton, 2008). Program Evaluation Methods Evaluation science is replete with methods and models from simple to complex. Methods can be either quantitative, qualitative or a mixture of both (Worthen, et al., 1997). Bennett (1977), an evaluation specialist with the USDA Extension Service summarized traditional assessment methods. They included experiments, post-then-pre studies, time-trend studies, surveys, case studies and hybrid methods (1977). A short summary of the quantitative and qualitative methods used in program evaluation is described below. Experiments offer a systematic way to evaluate changes in behavior or adoption of practice under randomly or non-randomly controlled circumstances (Rossi, et al., 1999). In the random experiment, a control group and treatment group are chosen from the same population. The nonrandomized approach involves comparing program participants with another group of individuals who have similar attributes, but are not currently or did not ever participate, in the educational program under question. This type

21

of program is sometimes called a matched-set design (Bennett, 1977). According to Rossi, et. al. (1999), matched-set designs have been enhanced and supplanted recently by incorporating statistical controls. In a time-trend study, participants of a program are followed for several months or years to identify any changes in knowledge, behavior or adoption of practice. The changes observed by the participants may be compared to program objectives or a comparison trend based on general projections that would have occurred regardless of the program. Other approaches to evaluation include the before-after study and the post-thenpre evaluation. In the before-after approach, participants provide information relating to knowledge, attributes, skills, motivations, aspirations, practice adoption potential, and so forth, before and after the program. Following the completion of the program, the participants are again asked these same questions (Bennett, 1977). They can also be contacted several weeks or months later and asked similar questions or observed to determine change in behavior or practice. The net effect is found by comparing the differences in these variables. There are no control groups in these approaches. An alternative to the before-after evaluation is the post-then-pre evaluation. In this form an attempt is made to remove what is known as the response-shift bias. This bias occurs when the program participant is not knowledgeable enough to answer preconference questions concerning a practice or behavior and thus is unable to provide accurate information that can be used for comparison following the program. By placing the frame of reference at the end of the program, the participant has a much better understanding of the concept(s) and can give a more accurate reflection of the knowledge

22

changed or practices adopted (Rockwell & Kohn, 1989). Rockwell and Kohn (1989) used a post-then-pre test with an Extension nutrition course. They found the results useful for program improvement, program accountability and program promotion purposes. Similar to many non-experimental studies, these studies exhibit a weakness in their ability to provide strong evidence of attribution (Diem, 2002). Surveys and case studies are other program evaluation methods that are typically conducted after program participation; however, on occasion data can be collected before or during the education program. The survey provides the opportunity for the researcher to assess a number of topics including participant perception of program relevance, satisfaction, attribution and impact. The survey can be implemented to determine what change in knowledge or behavior/practice has occurred since the program occurred. The survey approach is challenged by the fact that no a priori data is collected and no direct effects can be concretely attributed to the program. Advantages of the survey include relative flexibility, and simplicity (Bennett, 1977). The case study can be used to track changes in knowledge, attitudes, skills, aspirations, practice or other changes implemented by one or more participants more closely over an extended period of time. Qualitative research methods such as observation, interviews, and review of records are often used in case studies. Extension often uses the “success story” which is a weak form of case study (Bennett, 1977). These are often situations that are chosen by the program developer to highlight the impact of the Extension program. This evidence is fairly easy to collect and showcases what one individual, community or group did as the result of participation in the shortcourse.

23

An Integrated Approach to Evaluation Bennett was among the first to encourage a systematic approach of integrating evaluation with program development (1977). By combining Tyler’s curriculum development concepts with Kirkpatrick’s training evaluation model, Bennett, created a seven-level hierarchy model for use by Extension employees in states and counties. These levels were, from “lowest” to “highest”: inputs, activities, people involvement, reactions, knowledge/attitude/skills/aspirations changes (oftentimes summarized by the acronym KASA), practice change and SEE change (SEE is an acronym for social, environmental and economic). According to Bennett, various evaluation techniques could and in fact were being conducted at each of these points in the process; however, tradeoffs between cost, complexity, value and potential use of the results varied based on the level under investigation (1977). At the process and output levels, for example, input, activities, people involvement and reaction would primarily be descriptive and fairly easy and inexpensive to collect, while those at the higher levels that include knowledge, attitude, behavior, practice, and SEE changes would be more explorative, costly and complex to collect but potentially more valuable. Bennett revised his hierarchy model throughout the next two decades, first with the addition of the Reflective Appraisal of Programs framework (RAP) in the early 1980s, (Bennett, 1982) and then with the Targeting Outcomes of Programs (TOP) concept in the 1990s (Bennett, 1995). These approaches furthered program developmentprogram evaluation integration. Many of the concepts in these models are compatible and complimentary with an effort that today is known as the logic model process (W.K. Kellogg Foundation, 2001).

24

This model, like Bennett’s and Rockwell’s TOP model, involves an approach to both program development and evaluation. It is increasingly being used within the Extension System, nonprofit community, public service, and other sectors as a planning and accountability tool, however it has not been used to any great extent as an evaluation tool (Arnold, 2002; Torghele, et al., 2007). In its simplest form, the logic model consists of inputs (i.e. investments in the program), outputs (i.e. activities and participation levels), and outcomes (i.e. impact). Inputs, for example, include staff, volunteers, time, money, research-based information, materials, equipment, technology and partners. Outputs are characterized by what educators provide and whom educators reach. These include workshops, curriculum, training, publications, Internet sites, and so forth, for participants, clients, agencies, decision-makers and others. According to the logic model, outputs also include customer satisfaction and immediate reactions (W.K. Kellogg Foundation, 2001). Outcomes, according to the logic model, can be broken down into short-, medium- and long-term impacts. Short-term impacts include learning outcomes. These might be, for example, changes in awareness, knowledge, attitudes, skills, opinions, aspirations and motivations. Medium-term impacts relate to action. Action impacts are behavior, practice, decision-making, policy implementation and social action. Finally, long-term impacts involve changes in conditions as Bennett outlined in the hierarchy model: social, economic, civic and environmental (W.K. Kellogg Foundation, 2001). The logic model also incorporates assumptions (beliefs, social context, etc), and external factors. For incorporating program development into the logic model, needs assessments, stakeholder engagement, and priority setting are included in the stages

25

before input identification and evaluation. The logic model includes a framework for including program evaluation. (McNamara, 1998). In 2007, a study was published documenting the use of the logic model as a basis for developing a survey instrument for program evaluation (Torghele, et al., 2007). This study utilized a predetermined logic model developed for conducting emergency preparedness summits for schools of nursing in Georgia. A Likert-Scale survey was developed to measure the attainment of various inputs, activities, outputs and outcomes. No attempt to correlate results between components of the logic model was made; that is, the survey, although more complex and tied to the logic of the program did not reveal relationships between knowledge change, outputs, outcomes and any personal or contextual factors that may influence the outputs or outcomes. Other Approaches to Program Evaluation Adult education professionals have utilized other methods to determine the impact of programs. These include focus groups, Delphi studies, direct observation, expert testimony and reviews of documents and existing data (Diem, 2002). Also, cost-benefit (O'Neill & Richardson, 1999) and social impact assessments (Score, 1995), have been undertaken within the realm of adult education evaluation. In the health field, Cervero and Rottet (1984) created an instrument designed to empirically test a diffusioninnovation-evaluation model hypothesized previously by Cervero (Abrussese, 1987). Specifically the study’s objective was to analyze the impact of the training program on employee performance using the Cervero Model. A fifty-one-item survey instrument was designed, tested for validity and implemented. Aside from the survey data, external data was collected from a review of charts, interviews with participants and supervisors and

26

observation. Results indicate that the framework was highly explanatory; between thirtynine percent and eighty-one percent of the variance of the dependent variables could be explained (Cervero & Rottet, 1984). This suggests that this design and framework could be useful in other fields. The Cervero Model did not specifically include topics such as post-education learning and inquiry, and social network factors, nor did it include factors relating to the pursuit of additional expertise on matters of interest or importance to the students. Program Theory and Program Theory-Based Evaluation Recently, the concept of program theory has received the attention of many scholars within the evaluation arena because of its flexibility and compatibility with other program evaluation theories and techniques. Chen (1990) and others believe that much of program evaluation, while exceptional in the advancement of these above referenced methods has been devoid of theory, which he defines as ‘a set of interrelated assumptions, principles, and/or propositions to explain or guide social actions’. By excluding theory, program evaluation using many of the above referenced methods become simple input-output, or “black-box”, forms of evaluation (Weiss, 1997). The fairly recent inclusion of program theory and theory-based methodology into program evaluation has improved its value and usefulness because of greater attention to program purpose, intervening mechanisms, causal variables and a host of other factors that are described in more detail in the following section. Following is a more detailed description of program theory, program theory-based evaluation, and its contributions to evaluation science.

27

As outlined in Chapter 1, program theory can be understood as the underlying assumptions and delivery mechanism of how a program should work (Rogers, Petrosino, Huebner, & Hacsi, 2000). These relate to the development of program goals and objectives as addressed by formal or informal needs assessments. They also relate to how the program is implemented through education, funding, mentoring, rules and regulations, technical assistance, etc. In addition, the assumptions relate to how the program’s outcomes can be measured or assessed via program determinants. Program theory-based evaluation uses the program’s theory as the backdrop from which various aspects of the program can be assessed. Program theory-based evaluation has also been called theory-oriented evaluation, program theory-driven evaluation, program logic, logic modeling and other terms (Donaldson, 2007). In Donaldson (2007), three steps are used to “operationalize” the concepts behind program theory-based evaluation science. These are developing program impact theory, formulating and prioritizing evaluation questions, and answering evaluation questions. In order to evaluate programs, Chen (1990) offered a typology of program theory, which includes two subtheories; ‘normative’ and ‘causative’, with three theories under each. Normative theory of the program is also known as ‘prescriptive theory’ and includes the goals and objectives of the program and the interventions and actions that will be used to achieve a desired outcome. Under the normative program theory he explains that there is a ‘normative treatment theory’, a ‘normative implementation theory’ and a ‘normative outcome theory’. These each then can be evaluated based on program theory by comparing what was supposed to occur versus what actually occurred. To explain more fully, with the normative treatment evaluation, the treatment that was

28

supposed to be developed and delivered can be compared to what was actually developed and delivered. Similarly with the implementation theory, an evaluation can be conducted that compares what was programmatically planned vs. what was actually implemented. Finally, under the normative program output subtheory, planned outputs can be compared to actual outputs. The other subtheory Chen describes is causative theory (1990). Causative theory is also known as ‘descriptive theory’ and involves a description of how and why the program works the way it does through cause and effect expectations. Within this theory are three more subtheories including ‘impact theory’, ‘intervening mechanism theory’, and ‘generalization theory’. Evaluations that are developed from these theories include ‘impact evaluation’, ‘intervening mechanism evaluation’, and ‘generalization evaluation’. In impact evaluation, strategies are developed to determine causation. In intervening mechanism evaluation, strategies are developed to quantify the effect of processes that occur between program implementation and any outcomes. Finally, generalization evaluation develops strategies to broaden the evaluation results to other programs and stakeholders (Chen, 1990). In developing program theory-based evaluation, terms such as direct effect, mediator effects and moderator effects are discussed (Donaldson, 2007). Direct effects are basically those effects that are the direct result of the program under investigation. Mediating effects follow program participation and presumably have a positive influence on final outcomes. These effects can be proximal or intermediate outcomes such as knowledge change or change in behavior that may influence final program outcomes. Moderatoring effects are the qualitative (race, gender, age) or quantitative (external

29

factors such as markets and other programs) that influence the impact of the program and outputs (Donaldson, 2007). Practically speaking, program theory-based evaluation has shown increasing promise. It has the potential to explicate which processes lead to observed outcomes, and which processes and activities, that are not borne out, lead to program failure (Weiss, 1997). A literature review in the mid 1990’s found that over 30 program theory evaluation studies had been conducted (Weiss, 1997). In the same article, Weiss (1997), states: Theory-based evaluation is demonstrating its capacity to help readers understand how and why a program works or fails to work. Knowing only outcomes, even if we know them with irreproachable validity, does not tell us enough information to inform program improvement or policy revision. Evaluation needs to get inside the black box and to do so systematically (p. 42). From a discipline perspective, theory-based evaluation has increased “significantly” in the areas of health promotion and risk prevention (Rogers, 2007), but not necessarily from an educational applications perspective (Gascon, 2006). Gascon (2006) used program theory-based evaluation to study several hypotheses concerning quantification of achievement test scores based on implementation of a value-added assessment score. The results of the study, utilizing ANOVA hierarchical linear models and a program theory-based framework showed a positive, linear relationship between these assessment scores and the test scores (Gascon, 2006). To summarize, program theory has shown increasing promise in the area of evaluation. To implement program theory-based evaluation, an understanding of how the

30

black box of evaluation might be opened up is needed. One major area of program theory-based evaluation is in understanding how the above referenced mediators and moderators might influence outcome. One lens from which to view this from is from the ‘diffusion of innovations’ and ‘adoption of practice’ perspective. Both of these frameworks offer an opportunity to provide a better understanding of how outcomes are achieved based on the program’s theory and the context within which the program is delivered. Diffusion of Innovation and Adoption of Practice Introduction A literature review of the fields of diffusion of innovation and adoption of practice was conducted to obtain a more thorough understanding of factors that influence change following participation in an educational program such as the one conducted in this research project. The reasons were at least two-fold. First, these areas of literature were reviewed to discern if and how use of informal education, social networks and professional assistance were directly, or indirectly included in these frameworks. Second, this study utilized the program theory-based evaluation framework, which includes many moderator aspects of these two fields of study. Components of the diffusion of innovations framework have also been utilized in several studies that have been previously mentioned in this study such as those conducted using the Cervero Model. In addition, Hubbard and Sandmann (2007) outlined the potential for using this framework to evaluate adult education programs such as those conducted by Extension. Social scientists, marketers, agriculture specialists, technology transfer experts, and others are among those who have investigated the factors that affect adoption of

31

practice following exposure to information and technology. In particular, several fields of study have emerged to explore how new information, technology, and practices are adopted. These are the adoption of practice (adoption), diffusion of innovations (diffusion), and transfer of learning (transfer) frameworks. These frameworks consist of several hypotheses relating to how and why people adopt new technologies or implement new practices. These frameworks borrow heavily on the human psychology and sociology fields (Rogers, 2003). The diffusion and adoption pioneers began by studying traditional Extension programs in the 1940’s and 1950’s and making generalizations and enhancements that continue today. The transfer of learning framework is a more recent concept that includes the study of various types of near (proximal) and far (distal) learning and how it is incorporated and implemented by learners (educational program participants). By combining characteristics outlined in these frameworks with traditional program evaluation procedures and new program theory-based evaluation frameworks, we can open up the black box of a program’s influence on outcomes (Cervero & Rottet, 1984). Diffusion and adoption are two closely aligned terms that define a special kind of communication and decision-making process that occurs between a change agent or agency and an individual, organization or social group (Beal, et al., 1962). In particular, the characteristic of the communication process that makes it special is the “newness” of the idea or innovation being communicated (Lamble & Seaman, 1994). Typically, diffusion has included the dissemination of information from research while adoption has involved transforming the information into practice (Boone, 1989). Most of diffusion

32

research can be categorized as either adoption-related or “macro-diffusion related” (Attwell, 1992). Multiple Perspectives Scholars have discussed the concepts of diffusion and adoption from historical, topical, economic, geographic and other perspectives throughout the last one hundred years. Two fairly recent literature searches of diffusion and adoption related materials have resulted in anywhere from 5,000 (Rogers, 2003) to 10,000 (Backer, 1991) references to these concepts. Some of the discrepancy in these two estimates relates to definitions. Diffusion and adoption have often been used interchangeably with concepts such as knowledge utilization, technology transfer, information dissemination, consumer behavior, communication theory, and many other related terms (National Center for the Dissemination of Disability Research [NCDDR], 1996). While there may be significant differences in these terms depending on the perspective of the researcher or user (Rogers, 2002), they illustrate the tremendous growth and diffusion of the theory and empiricism behind this broad field of study (NCDDR, 1996). One of the original and longstanding scholars on the subjects of diffusion and adoption was Dr. Everett M. Rogers, a rural sociologist who spent an early portion of his career working alongside researchers and Extension agents studying the diffusion and adoption of hybrid seed corn in Iowa (Stephenson, 2003). According to Rogers, diffusion and adoption are two key elements of a framework he and others developed and refined over the past fifty years called the diffusion of innovation framework. This framework incorporates several components and concepts that attempt to explain how new ideas or technologies are spread throughout a social system via various communication channels

33

over time. Because of this, sociology, psychology and communication behaviors are key attributes of the framework. The framework has been used, empirically tested, critiqued and updated extensively since its inception in the 1950s. It has been adopted by researchers and practitioners in the areas of marketing and social marketing, communication and technology, Extension, business and finance, health care and medicine, politics and government, international development and countless other disciplines as a means to understand, plan and evaluate the diffusion of innovations, ideas, technologies and even social change. The diffusion of innovation framework as a meta-theory draws from several component theories that are useful for understanding the numerous different dimensions of the diffusion and adoption processes. These components are the innovation-decision theory, the individual innovativeness theory, the theory of rate of adoption and the theory of perceived attributes (Yates, 2001). The innovation-decision theory is temporal and includes five stages that the adopter potentially goes through in the innovation-decision process. These are the knowledge, persuasion, decision, implementation, and confirmation stages (Stephenson, 2003). These have been updated from the originally named phases of awareness, interest, evaluation, trial, and adoption (Bohlen & Beal, 1957). Research has indicated that not all individuals will go through each stage or go through the process in a linear fashion (Rogers, 2003). The theory of perceived attributes is based on assumptions that relate to the innovation’s perceived complexity, compatibility, trialability, relative advantage and

34

observability (Rogers, 2003). In other words, the more profitable, simple, compatible, observable and testable the innovation is perceived to be, the higher the potential for adoption. The theory of individual innovativeness suggests that there are innovators, early adopters, early majority adopters, late majority adopters and laggards in most social systems. These five categories are often visualized through graphical representation of the S- and bell-shaped curves (Lamble & Seaman, 1994). The S-shaped curve denotes the cumulative number of adopters from innovators to laggards while the bell-shaped curve is the resulting frequency curve that is normally distributed. The bell-shaped graph is useful in graphically representing the different types of adopters and utilizes standard deviations to depict category (i.e. early and late majority adopters are one standard deviation away from the average adopter and the innovators, and early adopters and laggards are two to three standard deviations from the mean) (Stephenson, 2003). Various related fields explain this phenomenon in a similar manner. In marketing, for example, these categories have been called enthusiasts, visionaries, pragmatists, conservatives and skeptics. Regardless of the terminology the bell-shaped curves and associated concepts remain similar (Moore, 1991). The S-shaped graph helps explain the theory of rate of adoption; that innovations and new ideas start slowly and then reach a point of inflection where their adoption/implementation increases at an increased rate. Finally, as the market for the new innovation is saturated, the number of new adopters tapers off to zero. Characteristics relating to the communication channel, social networks and external factors can also influence the adoption rate of an innovation or practice (Rogers, 2003). Communication

35

channels relate to the interaction between the individual and the change agent or agency, and attributes of the communication program (for example, educational program type or means of dispersing information). Social networks and systems relate to the availability of support systems such as a local farmer organization or associations. Social network analysis, as a theoretical field has grown substantially in the last few years and has provided a large amount of insight into networks and relationships and their importance in adoption (Cross & Parker, 2004). Finally, external factors such as markets, weather, policy and unanticipated events all can have an effect on adoption. Dr. Ronald Havelock, another well-known diffusion scholar who has continued to research this area, began his work in the late 1960s and 1970s by studying dissemination and knowledge utilization shortly after Roger’s model became widespread (1971). Through an analysis of the literature he found three perspectives on how information is diffused. These were the research, development, and diffusion perspective, the problem solving perspective, and the social interaction perspective (Havelock, 1971). These perspectives varied based on the characteristics of the innovation/idea/problem, the social system and the change agent involvement. In the research, development and diffusion model, knowledge is created in a laboratory and delivered to consumers. The agricultural research and Extension model is one of the more popular examples of this perspective. In this perspective, the change agent delivers the information, usually en-masse to willing clients who most likely have not been involved in the research and development phase. There is rarely follow-up by the change agent or research and development entity (Havelock, 1971).

36

The second perspective, the problem solving perspective, is a more interactive, collaborative perspective. The problem may be from the individual, group, community or societal point of view. This perspective involves successive interactions of need reduction involving substantial interaction between the change agent and client. Follow-up by and with the change agent is fairly regular until the problem is solved (Havelock, 1971). The third perspective, the social interaction perspective involves community networks. This perspective focuses on personal and system interactions. The importance of opinion leaders, personal contacts and social integration are emphasized with this perspective. The assumption here is that change agents are actively involved in the social networks (Havelock, 1971). The advent of the field of social network analysis has allowed researchers to more fully study the relationships between change agents, users of technologies, and other forms of engagement (Knoke & Kuklinski, 1982). These perspectives, or models, are not mutually exclusive of one another. Because of this Havelock suggests a fourth perspective, a linking perspective. “Linkage is seen as a series of two-way interaction processes which connect user systems with various resource systems including basic and applied research, development, and practice” (1971, Summary para. 5). Many other scholars, including Egon Guba, from the educational community who combined development, diffusion and evaluation concepts, Paul Lazarsfeld, a sociologist with communication expertise, and James Coleman, also a sociologist who provided important information on network analysis in the medical community, also contributed to the early state-of-knowledge of diffusion and adoption (Rogers, 2003).

37

Diffusion of Innovations in Rural Sociology, Agriculture and Forestry Although twenty percent of all diffusion research over the years has related to rural sociology and agriculture, the last twenty-five years have seen relatively few published works in the United States. According to Rogers, this is primarily due to the fact that “the major research problems had been solved, anomalies appeared, and intellectual criticisms arose during the late 1960s and early 1970s” (Rogers, 2003, p. 5859). Noticeably absent from Roger’s review of the history of diffusion research are discussions of any recent rural sociology, agriculture or natural resource studies. In agriculture, King & Rollins utilized the diffusion of innovation framework in their survey of Pennsylvania farmers who had participated in an educational program concerning the use of a new soil testing technology (1995). Variables they found to be statistically significantly different between adopters and nonadopters included the economic variables of “saving money” and being “inexpensive to use” (even though the technology wasn’t designed to promote more efficient use of nitrogen which may or may not increase the profitability of the farming activity). Also significant were the communication and information sources needed to create the necessary awareness and training to use the technology. Additionally, technical efficiency such as the reliability of the technology, conflicting practices, and the knowledge, skills and time required for taking and drying soil samples were found to be barriers to adoption of practice. Another barrier involved the attitude of the change agent (Extension agent/specialist) and their commitment to the innovation. Similarly, Drost and others found barriers to adoption of practice to be maintenance of profits or other financially related concerns, lack of knowledge or skill and federal farm programs (1996). They also

38

found older farmers were more resistant to adoption of an innovation. These farmers felt the practice was either not feasible or impractical. Finally, a general lack of concern for environmental issues was listed as a barrier to adoption of practice in this study (Drost, Miller, Campbell, Long, & Wilson, 1996). Another study looked at factors that influence the adoption of practice and the participation in educational outreach of integrated pest management (IPM) in Utah. Characteristics that influenced farmers and producers to adopt practices or participate in educational programs included major source of income (on-farm or off-farm), farm size, market destination (in-state or out-of-state), diversity of crop produced, past intensity of IPM outreach efforts and development of commodity organizations (Alston & Reding, 1998). Related to factors that affect the adoption of innovation is the concept of audience segmentation. While Rogers postulates that the diffusion process increases once innovators and early adopters reach a critical mass, he also supports the concept of audience segmentation. Audience segmentation means that targeting individuals who have similar beliefs, socioeconomic status, education and other characteristics for programming may be more effective than targeting just the innovators, opinion leaders or other early adopter categories. In other words, a blanket approach to meeting the needs of different types of potential adopters is not necessarily the most effective way to develop an Extension program (Drost, Long, & Hales, 1998). Extension professionals involved in the delivery of innovation to forestry clients have had similar concerns as their agricultural counterparts. These concerns have ranged from broad interests in program impact and evaluation to more specific interests in

39

diffusion of innovation and adoption of practice. A 1980 article in the Journal of Forestry introduced the diffusion of innovation framework to the forestry profession. This article announced the creation of a technology transfer office within the USDA Forest Service and explained the various components of Rogers’ theories on diffusion patterns and characteristics. In particular they referenced how these theories could be utilized to conduct a greater amount of quality educational events for forestry clients. Mention was made of the important role that the Cooperative Extension System played in this arena (Muth & Hendee, 1980). This introduction was followed with empirical work on the subject within Extension forestry. Dr. Jackie Haymond, a Clemson University Extension forester and researcher was among the first foresters to research the diffusion topic. She reported on the results of identifying and surveying opinion leaders (early adopters) and quantified their characteristics. More than half of the sample valued lifestyle enhancement benefits of their forestland more than timber production and economic benefits (Haymond, 1987). In addition, age, profession, income, gender and other characteristics were summarized for these opinion leaders. No effort was made to compare opinion leaders to “non opinion” leaders in the field although numerous other studies have characterized the forestland owner and have found that not only opinion leaders, but most landowners, have objectives for their land that rate higher than timber production (Butler & Leatherberry, 2004). Cross and Green (1996) conducted a survey of opinion leaders who were provided training to determine their actions following the intervention. This training, grounded in Rogers’ early adopter theory, was designed to provide education to leaders for use in

40

their respective communities. They compared results to non-opinion leaders who had also participated in the training and found significant differences in adoption behavior. These leaders adopted more forestry and wildlife training than their counterparts, however they did not have a significantly different diffusion behavior than non-opinion leaders (i.e. number of acres impacted through neighbors or other contacts). A leadership model for selecting opinion-leading forest landowners was not successful in predicting adoption or diffusion behavior. In a study comparing early and late adopters of a forestry innovation, Graesser and Force found that late adopters tended to be older, less educated and have less income than those considered to be early adopters (Graesser & Force, 1996). No differences in perception of profitability were found between the groups suggesting that those participating in forestry exercises such as these have non-financial goals. The researchers also found that later adopters tended to have longer land tenureship and spent less time on their property than early adopters. Finally, issues with complexity, compatibility and relative advantage, were also noted by the late adopters. It is important to note that this study did not involve participants of an Extension program and did not look at effects of knowledge change on adoption. Another area that has been studied in the forestry arena relates to Rogers’ work on communication, networks and change agents. Baldwin and Haymond found that the forestry community was homophilic; that is, that researchers communicated with researchers, professional foresters communicated with professional foresters and landowners communicated with landowners (Baldwin & Haymond, 1994). These findings highlight a major barrier that must be overcome if diffusion of innovation and

41

adoption of practice are to occur in this field. There is additional information in the forestry literature concerning targeting audiences, increasing communication, understanding landowner behavior and related diffusion of innovation studies (Broderick, Snyder, & Tyson, 1996; Jones & Harmon, 1996; Snyder & Broderick, 1992). Another potential tool being used for evaluation research related to diffusion of innovation is social network analysis. Social network analysis involves mapping and measuring qualitative and quantitative relationships within specific social systems with the purpose of understanding their impact on various factors. The analysis can be between individuals, groups, organizations or other entities. Social network analysis for evaluation purposes involves utilizing techniques and methods to understand the influence these relationships have on the program outcomes. Graphs, matrices, sociograms and other tools are used to quantify the relationships. In summary, Chapter 2 summarized the literature on program evaluation and program theory-based evaluation with particular attention to other theories that support the implementation of theory-based evaluation. These include the diffusion of innovations, adoption of practice, transfer of learning and social network theories. The review of the literature reveals that a more robust era of program theory-based evaluation is currently underway and that much of the progress has been made due to the inclusion of these other fields of study. The review of the literature also reveals that while program theory-based evaluation, as a theoretical science, has been fairly well developed and documented, its potential for use by practitioners has not been tested to any great extent (Weiss, 1997). In particular, its use for studying causal affects of social programs such as

42

Extension education programs and the various factors and relationships that may influence outcomes have not been empirically tested to any extent. Chapter 3 outlines the methodology for this research project. It focuses on the development and deployment of a program theory-based evaluation of an adult education course. Specifically, program participants will be queried on several aspects relating to the program they participated in, their subsequent actions, and a variety of other aspects that are conjectured to influence their decision to increase forest management activity.

43

CHAPTER 3 METHODOLOGY The purpose of this research study was to investigate the relationships between mediators, moderators and outcomes of a voluntary nonformal adult education program. The following research questions were answered in this investigation: 1. To what extent was change exhibited in the mediators (knowledge change, use of informal education, use of social networks, use of professional assistance and products), and the outcomes (increased forest management activity) of a voluntary nonformal adult education program? 2. To what extent can variations in the mediators and outcomes be explained by identified moderators of a voluntary nonformal adult education program? 3. What relationships exist between the mediators and the outcomes of a voluntary nonformal adult education program? This chapter is organized into six sections that include a review of the conceptual and measurement frameworks, the instrumentation development, identification of the study population, data collection, preparation and analysis procedures, and study limitations General Conceptual Framework As referred to in Chapters 1 and 2, program theory-based evaluation offers an opportunity to envision evaluation from a theoretical framework versus a methods-based framework. Additional meta-frameworks explored in Chapter 2 such as diffusion of

44

innovation and adoption of practice also provided input into developing the study’s conceptual framework. This was primarily due to the fact that program theory in and of itself is not a theory; it is the use of theory in programming. The potential for use in program evaluation comes from the fact that multiple theories and methods can be used to determine the best approach from which to develop a conceptual and measurement framework. A potential disadvantage though is that any and all theories can potentially be included under the umbrella of program theory and program theory-based evaluation, leading to expensive and complex models (Cook, 2000). Several other theories and evaluation frameworks were investigated as discussed in the Literature Review for inclusion into the framework for evaluating an educational program that may suit the needs of this research study. These included objectives-based evaluation, curriculum-based evaluation, domain-based learning evaluation, transfer of learning, and social learning network theories models. From these, a composite theoretical framework including components of program theory and program theorybased evaluation, logic modeling, the Cervero Model, and diffusion of innovations/adoption of practice concepts were used to develop a conceptual framework. Based on the review of the literature, the general conceptual framework chosen for this study is based on a modification of Donaldson’s (2007) complex multiple mediation model with inclusion of moderator variables. A basic visual conceptualization of this refined model is summarized in Figure 2.

45

Figure 2 Conceptual Framework for the Study (Adapted from Donaldson, 2007)

In this model, certain program features are expected to have a direct effect on what are termed ‘proximal mediators’. The proximal mediator(s), also known as proximal outcome(s) (Donaldson, 2007), are then conjectured to affect one or more ‘intermediate mediators’ (behaviors and activities such as acquisition of new information or assistance that occur at some point more distantly from program participation). The proximal mediators and the intermediate mediators are posited via the predetermined program’s theory to influence outcome(s) (i.e. it is through these potential mediators or ‘enabling factors’ that desired outcome is achieved or enhanced). In this particular model, the relationships between the intermediate mediators and the outcomes are dynamic; that is, there is a potential for ‘loopbacks’. The intermediate mediators affect the outcomes and the outcomes can in return affect the intermediate mediators. These increases can be looked at as either increases in use of intermediate mediators or they can be viewed as

46

additional outcomes (Chen, 2005). Also in this framework, moderators such as age, gender education, income, landholding characteristics, etc. are included because they are also believed to influence the proximal and intermediate mediators and outcomes. In this conceptual framework, the inclusion, direction and nature of the arrows are also important and indicate either correlation or causation. This conceptual framework is based on a dynamic program impact theory, which can be summarized as follows: First, the program itself can have no direct impact on outcomes, therefore there is no arrow directly from the program to the outcome. This is due to the Law of Indirect Effect which, is described by Hansen and McNeal (1996): This law dictates that direct effects of a program on behavior are not possible. The expression or suppression of behavior is controlled by neural and situational processes, over which the interventionist has no direct control. To achieve their effects, programs must alter processes that have the potential to indirectly influence the behavior of interest. Simply stated, programs do not attempt to change behavior directly. Instead they attempt to change the way people think about the behavior, the way they perceive the social environment that influences the behavior, the skills they bring to bear on situations that augment risk for the occurrence of the behavior, or the structure of the environment in which the behavior will eventually emerge or be suppressed. The essence of health education is changing predisposing and enabling factors that lead to behavior, not the behavior itself (p. 503).

47

Therefore, there are no arrows leading directly from the ‘Program’ box to the ‘Program Outcome’ box. The program’s impact theory, which is often developed by stakeholders and the programming team, includes the belief that the program leads to changes in the proximal mediators. These changes in the proximal mediators affect the intermediate mediators and outcomes as are shown by the arrows leading from the proximal mediator. Additionally, in this model, prior research or stakeholder input provides information regarding relationships between the intermediate mediators and the program outcomes; and in this case, the relationship is dynamic; hence the double-ended arrows between the intermediate mediators and the program outcomes. Finally, the moderators are believed to have an influence on the proximal mediators, the intermediate mediators, and the program outcomes. This is shown via the one-way arrows directing away from the moderators. Measurement Framework To effectively implement and test the conceptual framework presented above, it was necessary to first develop measures for each identified model construct: the proximal mediator(s), the intermediate mediator(s), the moderator(s), and the program outcome(s). Following development of measures for each of these components, an analytical framework utilizing appropriate statistical tests was necessary. Proximal mediators in this research study became the knowledge change from participation in the course and was simplified by the creation and use of the variable knowledge change. Variables were created for each of the intermediate mediators and were defined as use of informal education, use of social networks, and use of professional assistance and products. Finally, one program outcome variable was defined and became a composite forest

48

management activity variable. These elements became the study’s main research constructs and more detailed justification for inclusion in this model is provided within the next section of this chapter. More detail regarding these constructs and their definitions and are listed in Table 1.

Table 1 Definition of Research Constructs Variable Type and Name

Definition

Proximal Mediator: Knowledge Change

Change in the participant’s awareness, understanding, and skill set with regard to the subject matter content. The participant’s change in critical thinking skills relating to a subject matter.

Intermediate Mediator: Use of Informal Education

The use of subject-matter related education activities related to topics taught in the course.

Intermediate Mediator: Use of Social Networks

Involvement with communities of similar interest such as interaction with fellow classmates, neighbors, and others involved with similar situations. Engagement in social activities such as clubs or associations related to forestry.

Intermediate Mediator: Use of Professional Assistance and Products

Utilization of services and products provided by professionals.

Outcome: Forest Management Activity

Adoption of new or enhanced forestryrelated practices on participant’s forest.

In addition to these constructs (referred hereafter as mediators and outcomes), several moderators were expected to influence the outcome. In this case, a review of the literature provided a number of personal and contextual characteristics that were believed to have an influence. Personal characteristics were related to the individual participant 49

and contextual characteristics were related to their landholding situation. A simple graphical representation that reflects more detail than Figure 2 regarding the program’s theory is shown in Figure 3.

Figure 3 Multiple Mediator-Moderator Impact Model: The Master Tree Farmer Program

A summary of the specific research project’s program theory is now presented in more detail. This includes the background of the program’s theory and each subcomponent of the study model listed in Figure 3. This is followed by an explanation of how variables were specifically developed from these constructs so that they could be used to answer the research questions.

50

Description of the Program and the Program Theory According to Chen (2005), a program’s theory includes “a specification of what must be done to achieve the desirable goals, what other important impacts may also be anticipated, and how these goals and impacts would be generated” (p. 16). The following summary of the Master Tree Farmer Program elucidates this definition of program theory. The Master Tree Farmer Program consists of three separate short courses: The Master Tree Farmer level one course (MTF1), the Master Tree Farmer level two course (MTF2), and the Master Wildlifer (MW) course. These courses have been broadcast live via commercial satellite or via tape-delayed sessions to Extension offices in states in the Southern region in the following sequence over the last seven years: 2000 – MTF1 (1,250 participants, 10 states), 2001 – MTF1 (2,500 participants, 14 states), 2002 – MTF2 (1,500 participants, 12 states), 2003 – MW (4,800 participants, 12 states), 2004 – MTF1 (1,800 participants, 10 states), 2005 – MW (2,000 participants in 9 states) and 2006 – MTF2 (500 participants in 6 states). The courses involved a team teaching approach including national experts from universities, Extension, industry and associations. The courses were seven-week, 21-hour programs designed with the objectives of: (1) increasing the knowledge and awareness of basic forest management principles; (2) increasing the interest for inquiry into conducting forest management on the participant’s own land; (3) increasing the use of professional forestry services and resources; and 4) increasing the adoption of forest management activity by course participants. Based on data from state and local

51

coordinators, over 15,000 people have participated in these courses either live or via videotape delays since 2000. In 2000, the Master Tree Famer Level 1 design committee undertook the development of the course by determining the need for such an effort, the course content, the curriculum and individual session objectives, finding the most reputable and qualified speakers available, and investigating the best teaching format to deliver the information. In addition, course participants and stakeholders provided information each year via a short post-workshop evaluation to the organizers, for purposes of improving the course. Table 2 lists the specific session objectives relating to knowledge and awareness change as retrieved from the official website of the course (http://www.mastertreefarmer.net). This course’s committee included an advisory committee with representatives from the professional forestry and forest landowner communities including universities, nonprofits, forest industry, Extension, adult education, distance learning experts and others. In addition to the advisory committee, each participating state in the south was invited to add one or more experts to a “state coordinators” committee. These groups were invited to meet monthly via telephone conference calls to discuss various logistics of the program including program goals and objectives, topics, speakers, marketing, notebook materials, and other program aspects. In the end, a seven-week forest management foundations and applications course was designed and developed by this group. In 2004, the third iteration of this particular course; nine states chose to participate; these states held one or more live or tape-delayed sessions at strategic Extension offices throughout their state.

52

Table 2 Master Tree Farmer Session Topics and Objectives Session Number 1

Master Tree Farmer Session Topics Forestry terms and concepts, introduction to forest management planning, importance of managing your land, and defining objectives.

Individual Session Objectives

2

Forest economics, forest taxation, and estate planning.

Introduce forest economics and how forestry differs from other investment opportunities and explore the elements of risk in forestry investments. Introduce tax structure for timber and demonstrate how landowners can use these tax programs. Introduce the concept of estate planning and make landowners aware of complexities and demonstrate a few estate-planning strategies.

3

Pine management, managing stands for marketable products, natural pine management, intensive pine management.

Provide an overview of management strategies for even and uneven-age management systems for pine stands. Review natural stand dynamics over the life of a forest. Review management practices such as cutting and vegetation control practices that alter stand dynamics, and demonstrate application of management practices to stands over time and follow stand development through time.

4

Hardwood management.

Provide an overview of management strategies for hardwood stands. Review habitat requirements for different hardwood species for upland and bottomland areas. Review natural stand dynamics over the life of a hardwood forest. Review management practices such as cutting and vegetation control practices that alter stand dynamics. Demonstrate application of management practices to stands over time and follow stand development through time.

5

Marketing and harvesting.

Cover the basics on effective marketing of forest stands. Discuss products and when to begin marketing and harvesting activities, place emphasis on marketing small diameter wood, cover market trends, and harvesting systems for tracts of 20 acres and up.

6

Management opportunities,

Introduce landowners to management possibilities available on forestland. Allow landowners to

Become familiar with the overall field of forestry, understand meaning of basic forestry terms, exposure to basic forestry concepts, familiar with the components of a management plan for land, begin to think through personal objectives.

53

Session Number

7

Master Tree Farmer Session Topics agroforestry, wildlife management, timber management, recreational opportunities including tourism, pine straw, Shiitake mushrooms, medicinal plants, greenery, and firewood.

Individual Session Objectives

Forestry services, current issues, and getting engaged

Acquaint landowners with forestry services available to landowners. Touch on current and future issues that may influence ownership and management of forestland. Discuss groups that provide landowners with opportunities to become active in forestry issues. Provide landowners with activities they can undertake to become involved with the management of their land.

determine some objectives they have for their land. Discuss compatibilities, conflicts, and tradeoffs of managing for multiple objectives. Demonstrate how different acres can be managed for different objectives.

source: Master Tree Farmer website: http://www.mastertreefarmer.net/MTF2001Fall/MTFI/index.htm retrieved on June 15, 2008

Program Theory In the case of Master Tree Farmer program, several studies and several informal surveys have documented the need for and interest in information and education relating to forest management (Boyd, 1984; Downing & Finley, 2005; Williams, Voth, & Hitt, 1996). The course was targeted to forest owners or potential forest owners with interest in improving the management of their lands. The consensus of the advisory committee and coordinating group was to provide both foundational and applied information on several specific forest management topics, and awareness relating to sources of additional information and assistance. According to the program’s theory, landowners with an increased understanding (knowledge) of foundational principles and sound forest management applications as well as an increased awareness of available information and assistance would become more interested in learning more, would contact professionals

54

or other sources of assistance and information, would start to interact with other forest owners through various social networks, and ultimately, increase forest management on their land. This consensus of the advisory board is substantiated by forestry literature on the value and importance of the use of informal education (Radhakrishna, Nelson, Franklin, & Kessler, 2003), use of social networks (Broussard, Sagor, & McDonough, 2009), and use of professional assistance and products (Boyd, 1984; Hubbard & Abt, 1989). The MTF course developers also felt that the purpose of this course was not to make experts out of the participants, but rather to provide them with enough information to make them more knowledgeable and interested, less uncertain, and more empowered to seek more information, assistance and advice to the point where they would become further involved with the management of their land. In addition, stakeholders and program designers realized the potential iterative and building nature of this education, information seeking, assistance, networking and behavior change model. Forest landowners, who became engaged via this program, would hopefully become ‘lifelong’ learners, and as they increased forest management activity, would continue to increase their learning, use of social networks and use of professional assistance and products. Based on this rationale behind the program theory, the theory-based evaluation considered the extent to which the program features led to an increase in a critical proximal mediator variable (knowledge change), the extent increases in this variable lead to increases in critical intermediate mediator activities (use of informal education materials, use of social networks and use of professional assistance and products), and the extent to which increases in the proximal mediator variable and intermediate mediators correlated to the outcome of increased forest management activity. In addition,

55

according to the program’s theory, moderator variables such as age, ownership size, educational level of the participant, etc., were also believed to have an influence on the variables in the program theory. Instrumentation Survey research was undertaken to address the research questions. The type of survey research was a questionnaire for use on participants in the 2004 Master Tree Farmer Program Level 1 course (Appendix A). This involved developing a post-program, self-administered questionnaire designed by the researcher, doctoral advisory committee, and regional forestry educators including the course coordinators. The participants were responsible for answering questions based on their personal reactions to the course, and any activities and interaction they recalled undertaking following the course. The survey instrument was a mixture of multiple-choice, demographic, and phrase completion response questions. The research instrument was designed to obtain information to develop the mediator, moderator and outcome variables so that the relationships between them could be quantified and explored. These included sections that were used to identify the participant’s personal perception of their change in knowledge, change in the use of informal education activity, change in the use of professional assistance and products, change in the use of social networks and change in forest management activity following course participation. In addition, information on personal and contextual characteristics that were believed to influence these behaviors and outcomes was collected. The following sections summarize the methods that were used to develop measures for these constructs.

56

Identifying Knowledge Change Items Objective- or curriculum-based evaluation measures how well outcomes meet pre-determined objectives as defined by stakeholders and program managers (Howell, et al., 1993). Measurement items for knowledge in this study come directly from the program’s objectives as listed in Table 2. These objectives were the result of several meetings with program stakeholders, funders, program managers, and session instructors. Further, the studies’ methodologists provided assistance with refinement and rewording of these measurement items. The objectives and a brief summary of the survey item language are summarized in Table 3. They were listed below the following stem statement: “As the result of my participation in the 2004 Master Tree Farmer Program my knowledge of”. The specific survey language is listed in Appendix A. Identifying Use of Informal Education Items Similar questions were then asked of program participants with regard to any additional topic-specific learning that may have been undertaken for the same 19 forest management topics listed above. Information collected on these items would be useful for determining whether informal education on specific topics was pursued. This became Section 2 of the survey instrument. They were listed below the following stem statement: “Since my participation in the Master Tree Farmer 2004 Program my learning about:”. The survey items used to measure the construct of basic learning change are listed in Table 4.

57

Table 3 Survey Items Measuring Knowledge Change Proximal Mediator Knowledge Change

Survey Item Number 1

As the result of my participation in the 2004 Master Tree Farmer Program my knowledge of: Forestry terms and concepts

2

The importance of defining forest management objectives

3

The purpose and elements of a forest management plan

4

Forestry business aspects such as forest economics, forest finance and forest risk

5

Federal timber taxation rules and regulations

6

Federal, state and local (county or parish) tax programs for forestry purposes

7

Estate planning and estate planning strategies

8

Pine management strategies such as thinning, applying fertilizer, site preparation, etc.

9

Hardwood management strategies such as crop tree management, improving degraded stands, managing for high quality hardwoods, etc.

10

Marketing forest products

11

Forest logging systems commonly used to harvest products

12

Forest management possibilities available to me on my land

13

The compatibilities and tradeoffs of managing different acres for varying objectives

14

Forestry services available to me such as professional forestry advice, government cost-share programs, etc.

15

Forestry information and technologies available to me

16

Forestry issues that can affect the ownership of my land

17

Forestry issues that can affect the management of my land

18

Organizations that can provide me with opportunities to become more active on forestry issues

19

How to become more involved in the management of my land

58

Table 4 Survey Items Measuring Use of Informal Education Intermediate Mediator Informal Education

Survey Item Number 20

Following my participation in the 2004 Master Tree Farmer Program my learning about: Forestry terms and concepts

21

The importance of defining forest management objectives

22

The purpose and elements of a forest management plan

23

Forestry business aspects such as forest economics, forest finance and forest risk

24

Federal timber taxation rules and regulations

25

Federal, state and local (county or parish) tax programs for forestry purposes

26

Estate planning and estate planning strategies

27

Pine management strategies such as thinning, applying fertilizer, site preparation, etc.

28

Hardwood management strategies such as crop tree management, improving degraded stands, managing for high quality hardwoods, etc.

29

Marketing forest products

30

Forest logging systems commonly used to harvest products

31

Forest management possibilities available to me on my land

32

The compatibilities and tradeoffs of managing different acres for varying objectives

33

Forestry services available to me such as professional forestry advice, government cost-share programs, etc.

34

Forestry information and technologies available to me

35

Forestry issues that can affect the ownership of my land

36

Forestry issues that can affect the management of my land

37

Organizations that can provide me with opportunities to become more active on forestry issues

38

How to become more involved in the management of my land

59

Identifying Use of Professional Assistance and Products Items In addition to monitoring general changes in knowledge and topic-based informal education, stakeholders were interested in the extent to which participants increased their use of professional assistance and educational products. While the variable relating to use of informal education measured the ‘what’ that was being pursued, this variable measured the ‘who and how’ of the pursuit. A review of the program’s videotapes, Powerpoints ®, and supporting materials provided an idea of the types of professional assistance and programs and products that were promoted by session speakers. There are countless sources of professional assistance and educational products available to private forest owners from the forestry community and many of these were reviewed in the program (Hubbard, 1997). Table 5 provides a summary of the professional assistance and products available to forest landowners. They were listed below the following stem statement: “Since my participation in the 2004 Master Tree Farmer Program”. These items were compiled following a review of the program’s videotapes and consultation with the program development committee, program stakeholders and study methodologists. Identifying Use of Social Networks Items Use of social networks was measured by increases in participation in forestry-related associations, events and activities. Table 6 summarizes the survey items utilized to measure the change in use of social networks following participation in the program.

60

Table 5 Survey Items Measuring Use of Professional Assistance and Products Intermediate Mediator Use of Professional Assistance and Products

Survey Item Number 39

Since my participation in the 2004 Master Tree Farmer Program my learning about: Use of professional forester or Extension agent

40

The use of tax, finance, legal or other business professionals for forestry purposes

41

The participation in educational events related to forestry

48

The use of forestry information (fact sheets, newsletters, magazines, videos, computer software, etc)

49

The use of forestry Internet sites

50

The use of a forest management plan to make decisions regarding my land

51

The use of a forest management plan to implement practices on my land

The items for this scale were developed following a review of the program videotapes, the literature and discussions with professional forestry educators. As summarized in the literature review, several sources noted the availability and potential value of social networks in agriculture and forestry (Baldwin & Haymond, 1994; Broderick, et al., 1996; Haymond, 1987; Haymond & Baldwin, 1988). They were listed below the following stem statement: “Since my participation in the 2004 Master Tree Farmer Program”. Table 6 lists the survey items used to measure the use of social networks variable. Identifying Forest Management Activity Items Forest management activity items were generated from course content and available forest management literature which defines what constitutes forest management on private forestland (Williston, Balmer, & Tomczak, 1988).

61

Table 6 Survey Items Measuring Use of Social Networks Intermediate Mediator Use of Social Networks

Survey Item Number 42

Since my participation in the 2004 Master Tree Farmer Program my learning about: Membership and active participation in a forest landowner association

43

The interaction with neighbors or friends who practice forestry

45

The participation in other forestry events and activities

46

The use of my forested property for teaching or other purposes (youth, neighbors, other landowners, politicians, news media, etc.)

47

The engagement with issues at the local, state or national levels

59

The participation in one or more forest recognition or certification programs such as the American Tree Farm System®, the Forest Stewardship Program, the Treasure Forest Program®, or others on my land

It was necessary however to simplify the list of possible forest management activities in this research project due to the fact that forest holdings and the landowners who own them are highly unique and diverse in nature. Forest management activities recommended for young stands, for example, are entirely different than recommendations for older stands. In addition, landowners are diverse in their views of their objectives; some may wish to grow trees intensively while others plan for a more natural forest succession. The course covered both intensive and natural stand management alternatives, however stakeholders and program managers did not expect full adoption of each practice presented. Developing a measurement tool that included some of these practices that are land and landholder dependent therefore would not serve the purpose well.

62

To alleviate this concern, only those practices that were discussed in the course and could be conducted on an owner’s land regardless of their objective or stage of management were included in the survey. The stem statement was “Since my participation in the 2004 Master Tree Farmer Program:”. Table 7 lists the survey items used to measure the forest management activity variable.

Table 7 Survey Items Measuring Forest Management Activity Outcome Forest Management Activity

Survey Item Number 52

Since my participation in the 2004 Master Tree Farmer Program my learning about: The use of federal, state or county forestry tax programs for my forestry investment

53

The implementation of forest management practices on my land (for example, reforestation thinning, fertilization, herbicide applications, etc)

54

The monitoring, maintenance, or improvement of wildlife habitat on my land

55

The monitoring, maintenance, or improvement of water quality on my land

56

The maintenance of records relating to such things as timber inventory, costs, returns, etc. on my land

57

The creation or use of an estate plan that is inclusive of my property

58

The use of financial analyses relating to my forest investment

Developing the Response Scale Once the individual statements that comprised each of the five constructs were identified, it was necessary to develop a response scale to measure the extent of change. From these change responses, a composite index for each of the construct variables in the

63

study could be developed by either summation or average. There were several alternatives to choose from that could suit the questions asked and the data analysis. These ranged from a numbered scale where 1 might be little to no change in the proximal or intermediate mediator, or the program outcome, to 10, which would represent a substantial increase. A prototype survey instrument using responses tied to simple stemcompletion phrases rather than a number or frequency scale was developed after a review of the literature and discussion with the methodologist. The instrument contained a fourpoint response scale that captured the participant’s reaction to how much change they recalled for each of the stem statements referenced in Tables 3 through 7. This four-point scale was identified as the following: has not changed, has increased minimally, has increased moderately, and has increased substantially. This response scale was utilized to capture change for all five variables (knowledge change, use of informal education, use of social networks, use of professional assistance and products, and forest management activity) The belief by both the researcher and the study methodologist was that survey subjects would be more accepting of a four-point scale, especially if the survey were lengthy. In addition, the amount of recall after five years would probably be fairly limited, and a four-point scale would more accurately reflect the respondents’ ability to recall such things as knowledge change and activities following participation in a course that occurred five years hence. The pilot study could be used then to detect whether there was enough variation in the responses, and whether the resulting composite indices would be useful for data analysis and discussion of results.

64

Moderator Variables: Personal and Contextual Factors In addition to the various outcomes, there are characteristics relating to the program participant (personal) and their landholding (contextual) that are believed to influence the proximal mediator, intermediate mediator and outcome variables. Rogers (2003), for example outlines several characteristics that may have an influence on adoption of a behavior. These have included personal motivational factors (innovator, early adopter, early majority, late majority and laggard, for example), education, total income, discretionary income, personal objectives, and a host of others. Others have empirically tested some of these characteristics in the agriculture and forestry arenas (Alston & Reding, 1998; King & Rollins, 1995; Lorenzo & Beard, 1996). Several characteristics chosen for this study that have been found in one or more landowner or Extension studies to have influenced adoption of practice include age, race, gender, total, income, length of land tenure (in years), size of landholdings (in acres), ownership type (sole proprietor vs. corporation for example) and highest educational level achieved (high school, college, graduate, specialized, for example). Investigating the effect of these variables is important from a variety of perspectives. Under the assumption that survey participants have all taken the full Master Tree Farmer course and are therefore aware and more knowledgeable about common forestry practices, it becomes important to understand what additional effects such as age, education, landholding size, etc. have on the learning, pursuit of services and resources, use of social networks, and forest management activity. Answers to these questions will shed light on better programming methods, better audience segmentation including social marketing

65

opportunities, impact of programming on different audience segments and even posteducational event needs of participants. Table 8 presents a summary of the characteristics that are believed to impact program output from a review of the literature, discussions with the program committee, and personal experience. Finalizing the Pilot Survey Instrument Incorporating Expert Review into the Survey Instrument The survey methodologist and other members of the researcher’s doctoral advisory committee reviewed several drafts of the survey instrument prior to pilot study implementation. Following this, the researcher sought the insight of other forestry education professionals in the South, including the co-course instructor, Dr. George Kessler, Professor Emeritus at Clemson University, the forestry faculty member of the doctoral advisory committee, a USDA Forest Service representative, and several state coordinators. Several suggestions were made to improve the cognitive and motivational features of the survey including simplifying the language of several of the questions. These suggestions were integrated into the survey instrument and the methodologist again reviewed and approved the instrument for pilot testing.

66

Table 8 Moderator (Personal and Contextual) Variables Type Personal

Moderator Variable Age (Age Class)

Variable Type Categorical

Rationale for Inclusion Age has been shown to be a proxy for years of experience, generational differences in land use and management philosophies, and even time available to implement outcomes.

Personal

Race (Caucasian, African-American, etc)

Categorical

Past studies have looked at racial differences with respect to access and use of forestry services and resources and social networks.

Personal

Gender (male/female)

Dichotomous

There is a great interest today in the forestry community in understanding the role of gender in forestry decision-making and planning and implementation. Many widowers are now responsible for the family’s forestland.

Personal

Annual income level (thousands of dollars per year)

Categorical

Income level can be used as a proxy for discretionary resources to be used to implement forest management practices.

Personal

Education level (none through advanced degree)

Categorical

The role of education level has been studied for market segmentation purposes as well as the ability to understand and implement complex forestry practices.

Contextual

Importance of landowner objective (timber, wildlife, recreation, etc)

Continuous

Landowner objective has shown to be associated with level of forest management activity.

Contextual

Distance to forestland (miles)

Continuous

Those owners who live on the property or live close to the property have been shown in certain studies to be more active managers.

Contextual

Possession, updating or creating a management plan (yes/no)

Dichotomous

Studies have shown that those who posses and utilize a management plan are more active managers.

67

Type Contextual

Moderator Variable Participation in Master Tree Farmer Level 2 (yes/no)

Variable Type Dichotomous

Rationale for Inclusion Participation in the advanced level program should lead to more enabling behavior and management activity.

Contextual

Landholding size (acreage)

Continuous

Size of acreage and size of forested acreage have been shown in past studies to be an influencing characteristic on forest management activity.

Contextual

Length of ownership (years)

Continuous

Understanding relationships between land tenure and active forest management is critical due to the increased levels of forest transition in the South.

Pilot Study A pilot study was conducted on the participants of the 2004 Master Tree Farmer program in Louisiana. The pilot study was used to test the survey methods, the validity of the data, and the reliability of the survey instrument. The pilot study was conducted to determine whether there was enough variability in the response scales, whether data could be grouped and analyzed appropriately, whether an adequate response rate could be obtained, and whether the personal and contextual questions were worded appropriately to return usable results. The 6-page survey was printed, numbered and mailed out to 82 participants of the Louisiana course. Dillman’s Total Design Method (Dillman, 2000) was utilized and following an initial mailing and two follow-up reminder activities, a 39% response rate (32 surveys) was achieved. The study methodologist had recommended a response rate of at least 25% to study the variability and validity of the survey instrument. From the data analysis of the 32 responses, Cronbach’s Alpha was high for all scales indicating a strong correlation between the individual items measuring the

68

construct. From these results, the initial thought was that since these numbers were high and resulted in potential redundancy in the questions, many could have been removed without a serous loss in construct measurement. Discussions with the methodologist however revealed that the inclusion of all the current items on the survey would be useful from both the assessment perspective of the course, and the construct development. The detailed information on the individual items showed promise to be useful for reporting impacts on specific topics and activities, and in answering Research Question #1 concerning how much change had occurred following participation in the course. From the 32 resulting questionnaires it was not apparent that respondents had difficulty understanding or answering the questions. One question “My knowledge of forest management possibilities available to me on my land and the compatibilities and tradeoffs of managing different acres for varying objectives “ may have been too long or confusing for people because several respondents left this blank. After discussion with the methodologist, it was decided that this question did not add anything to either the descriptive results or construct development component of the study. This question was deleted. Also, there seemed to be some confusion on how the questions regarding management plan development and improvement were organized on the survey instrument. This was corrected for the final survey instrument. Other suggestions by the methodologist included inserting a title on the actual survey instrument, and incorporating two to three sentences to introduce the purpose of the survey. Both of these recommendations were included in the final survey instrument.

69

Data Collection Strategy Collection of participant names and addresses began concurrently with questionnaire design and testing. The questionnaire was submitted and approved by the Human Subjects Office at the University of Georgia. The following steps in Table 9 were undertaken for data collection.

Table 9 Data Collection Steps Step

Task State or site coordinators were contacted to obtain mailing lists of past MTF course participants.

Notes This information was available either on a regional website or from Clemson University archives. This took several emails to those who provide addresses.

2

Organized names and addresses into database for formatting and mailing label development.

674 names and addresses were provided from three states (Alabama, Florida, and South Carolina).

3

Duplicate required number of questionnaires for mailing was made.

USDA Forest Service agreed to assist with the costs associated with this step.

4

Mail first round of questionnaires mailed in August-September, 2009.

USDA Forest Service has agreed to assist with the costs associated with this step.

5

Data collected and entered as surveys arrived in mail. Bad addresses were corrected where possible and survey instruments were resent.

Nonresponse, coverage and measurement errors were minimized to the extent possible (Salant & Dillman, 1994).

6

Second mailing was sent in October, 2009.

As suggested by Dillman (2000)

7

Third mailing was sent in November, 2009.

As suggested by Dillman (2000)

8

Final data entered into PASW. Data cleaned up and incomplete surveys were removed from database.

9

Utilization of PASW statistical tests to quantify results.

10

Report findings through peer-reviewed literature/dissertation.

1

70

Sample Identification and Sampling Procedure As noted earlier, the three different courses, offered seven times have been attended by close to 15,000 participants since 2000. The courses have been conducted at hundreds of sites across the region (anywhere from 50 sites to 150 sites depending on the year). A database of participants who took the course in Alabama, Florida and South Carolina in 2004 was used due to the ready availability of the information, and the fact that these states and Louisiana were the four most active participants in the course in 2004. Other states participated at a minimal level with only one or two active sites and an estimate of less than 100 landowners participating. The total participation in the three states represented 674 participants. This data does include many family units (husbandwife partners, parents with children, as well as several forestry and natural resource professionals, loggers and other affiliated people). This means that the actual number of participants with decision-making responsibilities and who actually owned land was somewhat less. This course represented the most recent installment of the Master Tree Farmer level I course and was chosen because of the number of practices outlined in the program and the likelihood that participants in the earlier courses may have difficulty recollecting certain aspects of the course. A concern with studying any of the later courses conducted in 2005, 2006 or 2007 is the time necessary to observe reasonable outcomes due to the fact that forest management often involves multiyear decision-making and adoption. It is advisable for example to plant trees during winter months and conduct certain practices after the forest achieves a certain growth stage.

71

A survey conducted five years after the program had the advantage of capturing the main impacts of the course. Since this course was a major commitment for participants, it was conjectured that participants would be able to adequately recall major reactions and immediate follow-up activities. Attention to adequate recall was monitored during the pilot test and only one of the 32 respondents commented that they had trouble responding to a few of the questions because of the time lag. Personal Characteristics of the Study Respondents Of the 674 questionnaires sent out, 290 responded to the study for a 43% total response rate. Some of these responses were blank or were not completely filled out. This resulted in a refined list of 271 surveys (40.2%). Several surveys with outlier responses to at least one question were removed (based on results of scores that were more than 3 standard deviations from the standardized mean) (Vaske, 2008). The questions with outliers included size of tract, age, and tenure. The final number of usable surveys was 255 for a response rate of 37.8%. In addition, some respondents did not fill in entire sections of the survey. This is evident in Table 10, with some questions receiving a 34% response rate (income) and others receiving upwards of a 38% response rate. The respondents’ age ranged from 30 years to 87 years, with a mean age of 60.7 years. Age was also broken down into four different, distinct categories to better analyze this effect on the various dependent variables. These age classes were ‘less than 51 years old’ (46 respondents), ‘51-60 years old’ (71 respondents), ‘61-70 years old’ (80 respondents), and ‘greater than 70 years old’ (53 respondents). Of the respondents who answered the gender question 84.6% were male and 15.4% were female. A vast majority of the respondents were Caucasian (96.1%), with 2.6% being African American, and the remainder either of

72

Indian descent (1.3%) or did not respond. A majority of the respondents earned at least a Bachelor’s degree (75.2%), and 58.4% had at least an annual household income of $75,000 or more. A more detailed summary of these personal characteristics is listed in Table 10.

Table 10 Summary of Sample Composition of Personal Characteristics Personal Variable Age Class (n=250)

Category Less than 51 years old 51-60 years old 61-70 years old Greater than 70 years old Male Female

n 46 71 80 53

% 18.4 28.4 32.0 21.2

209 38

84.6 15.4

Race (n=233)

Caucasian African American Other

224 6 3

96.1 2.6 1.3

Highest Level of Education (n=250)

No degree High School Associate’s degree Bachelor’s degree Graduate degree Other

4 40 18 88 83 17

1.6 16.0 7.2 35.2 33.2 6.8

Approximate Annual Income (n=226)

Less than $25,000 $25,000 to $49,000 $50,000 to $74,999 $75,000 to $99,999 $100,000 to $124,999 $125,000 to $149,999 Greater than $149,999

7 37 50 48 28 18 38

3.1 16.4 22.1 21.2 12.4 8.0 16.8

Gender (n=247)

Contextual Characteristics of Study Respondents The mean ownership size of respondents was 452 acres with a standard deviation of 860 acres and a range between 0 acres and 6700 acres. The mean tenure, or length of

73

ownership was slightly over 19 years (19.1 years) with a standard deviation of 16.0 years and a range of 0 years to 94 years. The mean distance to the participant’s forest property was 35.7 miles with a standard deviation of 53.6 miles and a range of 0 miles to 900 miles. Other relevant contextual characteristics of the study respondents are summarized in Table 11. Of those responding to the question (243), 43.1% had developed a management plan before participating in the class. Of those, 62.2% updated their plan following the course. Of those who did not have a management plan prior to participating in the course (56.9% of the respondents), 53.0% developed a management plan following participation in the shortcourse. 35.4% of respondents indicated that they had participated in the Master Tree Level II course (the advanced 7 week course) in addition to the Master Tree Level I course being evaluated.

Table 11 Summary of Sample Composition - Forest Management Characteristics Contextual Variable Possession of Management Plan Before Course (n=243)

Category Yes No

n 110 33

% 45.3 54.7

Updated Plan Following Course Participation (n=111)

Yes No

69 42

62.2 37.8

New Plan Developed Following Course Participation (n=134)

Yes No

71 63

52.9 47.0

Participation in Advanced Master Tree Farmer program (MTF 2) (n=243)

Yes No

86 157

35.4 64.6

74

Participants were also asked to rate each of several landowner objectives as to their importance. The possible responses were ‘Not Important’, Slightly Important’, ‘Important’ and ‘Very Important’. They were coded for analysis as 1=Not Important; 2=Slightly Important; 3=Important; 4=Very Important. Table 12 displays the means and standard deviations for the responses. ‘Importance of Planning for the Next Generation as an Objective’ ranked the highest with a mean of 3.52. This indicates a high relatively high level of importance (between Important and Very Important). ‘Importance of Economic Return as an Objective’ ranked the lowest with a mean of 3.16; still important for landowners, but not as important as the other factors.

Table 12 Summary of Sample Composition - Landowner Objective Characteristics Contextual Variable Importance of Planning for Next Generation as an Objective (n=251)

M 3.52

SD .750

Importance of Other Objective (n=27)

3.48

.975

Importance of Wildife as an Objective (n=252)

3.35

.817

Importance of Recreation as an Objective (n=251)

3.33

.798

3.16

.895

Importance of Economics as an Objective (n=251) Note: These variables were coded using the following possible range of answers: 1=Not Important; 2=Slightly Important; 3=Important; 4=Very Important.

Data Preparation There were four steps involved with the preparation of the data for analysis. These included the following. First, following closure of the mail survey, the data were entered

75

into a Predictive Analytical SoftWare (PASW) 18.0 data set. Second, the data set was examined and invalid surveys were removed. Third, reliability for the core measures was calculated. Finally, indexes for each of the research constructs were calculated. Scales and indexes are useful data reduction techniques as well as useful in providing more comprehensive and accurate assessment of a given desired variable (knowledge, behavior and activity in this case) (Babbie, 1990). A case-by-case review of the data was undertaken to ensure no more than a minor number of questions were missing responses for each respondent. Five indexes were then created to measure the research constructs in this study (a) knowledge change, (b) use of informal education, (d) use of professional assistance and products, (d) use of social networks, and (e) forest management activity. Data Analyses Procedures Overview To answer the research questions, a variety of basic statistical analyses were conducted depending on the specific research question, type of variable and the relationships being tested. These included frequency tables, means and standard deviations, rank-ordering, and bivariate and multivariate analyses. PASW 18.0 was used to calculate the frequencies, means and standard deviations for the questionnaire items. Following this, these results were reviewed by hand to ensure responses were logical given the expected responses. As in the pilot study, the coefficient alpha for each of the construct indexes was calculated to evaluate reliability of the data. A summary of the reliabilities as well as other relevant data is provided in Table 13.

76

Table 13 Statistical Summaries and Reliability Measures Variable

Number of Items

M

SD

Alpha

13.05

Mean Item Mean 2.76

Knowledge Change

19

51.99

Use of Informal Education

19

45.83

15.67

2.42

.97

Use of Professional Assistance and Products

7

18.19

6.31

2.28

.89

Use of Social Networks

6

11.87

4.79

1.97

.89

Forest Management Activity

8

17.01

6.47

2.13

.90

.96

The histograms of the frequency of each variable’s item mean indexes are displayed in Figures 4-8. These figures illustrate the distribution of the indexes that were created from averaging each of the individual item responses for each of the research constructs. From these figures, it is apparent that there is spread in the response for each of the research constructs. In addition, the relative shape of each of the histograms shows a variation in central tendency. Figures 4 and 5 show relatively normally distributed histograms of knowledge change and use of informal education for example, while Figures 6 and 7 show right skewness. Finally, Figure 8 shows a high number of average responses at or around 1 for forest management activity increase. This represents a high number of respondents who have not increased activity on their land in any of the subcomponents that define forest management activity.

77

Figure 4 Distribution of Knowledge Change Performance Index

78

Figure 5 Distribution of Use of Informal Education Index

79

Figure 6 Distribution of Use of Professional Assistance and Products Index

80

Figure 7 Distribution of Use of Social Networks Index

81

Figure 8 Distribution of Forest Management Activity Index

Next, the intercorrelation between the variables was determined to test whether there was a high degree of colinearity. The coefficient between each of the variables was significant at the .01 level. The numbers indicated high colinearity between several of the variables including the intermediate mediator variables. Table 14 summarizes the findings.

82

Table 14 Mediator and Outcome Variable Intercorrelations Knowledge Change Knowledge Change Use of Informal Education Use of Professional Assistance and Products Use of Social Networks Forest Management Activity Note. **p < .01

Use of Informal Education

Use of Professional Assistance and Products

Use of Social Networks

Forest Management Activity

-.68**

--

.63**

.76**

--

.48**

.69**

.81**

--

.57**

.72**

.84**

.84**

--

Summary of Specific Statistical Analyses Used to Answer the Study Research Questions The research questions identified in this study focused on the measurement of outcomes and mediator and moderator variables related to an educational program, the influences of moderator variables on the outcomes and activities, and finally, the relationships between the mediator variables and outcomes. To answer the first research question, which was focused on measuring any knowledge change, use of informal education, use of professional assistance and products, use of social networks, and forest management activity, data from Sections 1-3 of the survey instrument was averaged across items and across participants to develop

83

mean increase indexes for each variable. In addition, averages for each individual topic or activity were also calculated and rank ordered to provide detailed information on which items on the survey were ranked highest under each variable. These statistical analyses were carried out using several functions provided in the PASW 18.0 software package. To answer research question two, the extent that the moderator variables influence the proximal and intermediate mediator variables and outcome, bivariate and multivariate statistical analyses were used to determine any relationships that exist separately or in combination. The specific statistical analysis depended on the type of data collected. Data can be collected and categorized as dichotomous, categorical or continuous. To conduct these tests, the data entered into the PASW 18.0 software package was identified as scale, nominal or ordinal. Guidelines for selecting an appropriate analysis strategy were reviewed from Vaske (2008). This resource provided the analysis strategy based on the type of independent and dependent variables under investigation, the number of variables under investigation, the analysis strategy and the appropriate test statistic to employ (t vs F for example). Similarly, bivariate and multivariate analyses was used to answer research question three regarding the relationships between knowledge change, use of informal education, use of professional assistance and products, use of social networks and forest management activity. Figure 3, which summarizes the specific model under investigation provided conceptual guidance regarding the type of relationships to study (correlation vs. regression). Again, Vaske’s (2008) resource book was used as a guiding reference based on the type of variable, the number of variables under investigation, the analysis strategy

84

and the appropriate test statistic to employ. PASW 18.0 was then utilized to conduct the various tests. Limitations of the Study As an organizer and presenter in the Master Tree Farmer program, I had a personal interest in the outcome of this study however, I believe that the development of the survey instrument that was reviewed by peers in my field and peers in the field of adult education and program evaluation temper any bias potential. Statistical analysis of the data provided the opportunity to remove bias as well. Moreover, because of my knowledge of the program and my experience with working with landowners and education, I brought insight into the results that others, more removed, might not be able to do. Another limitation that could be argued is the length of time that has passed since the participants attended the course. The course under study was conducted in 2004. In response to this concern and as described earlier in this chapter, it is believed that five years offered an appropriate amount of time to quantify any behavior change or change in forest management activity. Measuring self-assessed change in knowledge four to five years later could be problematic, however it is my belief that since the course was a fairly major undertaking for participants, that they would remember the influence the course had on their comprehension of major forest management topics. In addition, major events, activities and decisions following participation in the course should also be fairly easy to recollect. From a practical perspective, it often takes several years to fully implement forestry practices. This delayed time is often necessary for program participants to undertake self-directed learning activities and decision-making. The

85

possibility does exist however that people might not recollect some events and activities. As mentioned previously, the pilot test was conducted to gauge participant’s abilities to remember key events and activities. No problems were noticed following a review of the data. There was also a potential for response-shift bias. This bias occurs when the answers to the questions are affected by the experiences that have occurred due to factors unrelated to the program or at least perceived by the survey respondent as unrelated to the program. This bias would be of greatest concern when measuring the result of the change in knowledge section. Some participants might attribute their change in knowledge to something other than the course and therefore underreport any impact. While of some concern, Bennett (1982) and others hold that collecting this information is still important and useful. It is also difficult to avoid in human subject survey research (K. Diem, personal communication, February 18, 2005). For many purposes, perception of impact is as critical as impact itself. Perceptions can allow for interconnecting events, identifying the cumulative impact of larger programs and can be easier to understand than more experimental approaches. While these errors may occur, there was no evidence to show that they were biased one way or another individually or in relation to each other (Bennett, 1977). For example, respondents were just as apt to over-attribute as underattribute impact and these over- or under- attributions will probably be consistent across the variables. The respondent were just as likely to attribute adoption of a practice to an educational program as a cost-share payment received or discussion with a neighbor about a practice that was observed.

86

One last potential concern is organizational bias; that is, the reflective information may be based on the participant’s overall attitude towards Extension and not so much the course (Bennett, 1982). This was dealt with by minimizing the use of Extension logos and verbage beyond what is necessary to gain access to the participant. In summary, the research study involved the development of a self-administered survey to determine knowledge change of forestry concepts and practices as well as post educational event self-directed learning and the implementation of any forestry practices. In addition, the survey instrument was designed to collect relevant personal and contextual factors that are believed to influence change in behavior or adoption of practice.

87

CHAPTER 4 FINDINGS The purpose of this research study was to investigate the relationships between mediators, moderators and outcomes of a voluntary nonformal adult education program. The following research questions were answered in this investigation: 1. To what extent was change exhibited in the mediators (knowledge change, use of informal education, use of social networks, use of professional assistance and products), and the outcomes (increased forest management activity) of a voluntary nonformal adult education program? 2. To what extent can variations in the mediators and outcomes be explained by identified moderators of a voluntary nonformal adult education program? 3. What relationships exist between the mediators and the outcomes of a voluntary nonformal adult education program? Findings Related to Research Question #1 The first research question asked, “To what extent was change exhibited in the mediators (knowledge change, use of informal education, use of social networks, use of professional assistance and products), and the outcomes (increased forest management activity) of a voluntary nonformal adult education program?” To answer research question one, the resulting means of each of the mediators and outcomes were calculated for each item and for each survey respondent. Figure 9

88

provides a graphical representation of the variables to be measured to answer this research question.

Figure 9 Proximal Mediator, Intermediate Mediators and Outcome Variables to be Measured

The results are listed in Table 15. The mean item mean has a possible range of 1 (has not changed) to 4 (has increased substantially). They are also listed in rank order to depict the relative average response for each construct. The order of ranking was as follows: knowledge change (M=2.75, SD=.68), use of informal education (M=2.42, SD=.83) use of professional assistance and products (M=2.28, SD=.79), increased forest management activity (M=2.13, SD=.81), and lastly use of social networks (M=1.99, SD=.81). These numbers represent composite indexes averaged from each research construct’s individual item means. In addition, frequency counts for the items used to develop each of the variables were calculated to examine the spread across response rate (‘Has Not Changed’, ‘Has Changed Minimally’, ‘Has Changed Moderately’, and ‘Has Changed Substantially’).

89

Individual response counts and frequency percentages for each of the 19 items used to compose the knowledge change index, 19 items to compose the use of informal education index, six items used to compose the use of social networks index, seven items used to compose the use of professional assistance and products index and eight items used to compose the forest management activity index were calculated using the PASW 18.0 frequency procedure. These results are listed in tabular form in Appendices F through J.

Table 15 Rank Order List of Variables Based on Mean Item Mean Rank

Variable

Number of Items 19

Mean Item Mean 2.75

Standard Deviation .68

1

Knowledge Change

2

Use of Informal Education

19

2.42

.83

3

Use of Professional Assistance and Products

7

2.28

.79

4

Forest Management Activity

8

2.13

.81

5

Use of Social Networks

6

1.99

.81

Note. Means were calculated based on the following coding system: 1=Has Not Changed, 2=Has Increased Minimally, 3=Has Increased Moderately, 4=Has Increased Substantially. The observed Range was 1-4.

A summary of the response rate frequencies for each variable is listed in Table 16. Sixty two percent of the respondents noticed either a moderate or substantial knowledge change due to participation in the course. Forty six percent of the respondents increased their use of informal education either moderately or substantially following the course, while forty one percent of the respondents increased their use of professional assistance and products either moderately or substantially following the course. Thirty percent of the respondents increased their use of social networks at least moderately following

90

participation in the course while thirty six percent of the respondents increased their forest management activity at least moderately following participation in the course. Also, from Table 16 relatively high frequencies are noticed in the “Has Not Changed” categories for use of social networks (40%), and increased forest management activity (38%).

Table 16 Frequency Percentages by Response Category Variable

Knowledge Change

Frequency Percentages by Response Category* Has Has Has Has Not Missing Increased Increased Increased Changed Minimally Moderately Substantially 11 26 39 23 1

Use of Informal Education

21

31

30

16

2

Use of Social Networks

42

26

19

11

2

Use of Professional Assistance and Products

30

27

25

16

1

Forest Management Activity

38

24

22

14

1

Note. *Rows may not add to 100% due to rounding From another perspective, two additional tests were conducted. The first test was to determine whether these means presented in Table 15 were statistically different than 1, which was no change in knowledge, use of informal education, use of professional assistance and products, use of social networks and forest management activity from the course. The second test was to determine whether these means were statistically different from 2.5, which would be an average score on each of the scales (this number was calculated as that being halfway between ‘has not changed’ (1) and has increased

91

substantially’ (4)). The results of these tests would provide insight into how statistically significant the changes in the various variables were. These analyses were accomplished using the one sample t-Test to compare these means to 1 (no change) and 2.5 (an average level). The results of these analyses are summarized in Table 17. These data show that: a) there was a significant difference between the individual index construct means and the mean that would occur if no change had occurred (1), b) there were significant differences in several of the construct means and a hypothetical ‘average’ score of 2.5 however their absolute value of their differences were important.

Table 17 T-Test Results Mean Comparisons (M=1, M=2.5) Variable Knowledge Use of Informal Education Use of Professional Assistance and Products Use of Social Networks Forest Management Activity

T-Test Statistic1

df1

p1

T-Test Statistic2

df2

p2

41.03

253

.00

5.90

253

.00

27.24

252

.00

-1.60

252

.11

25.73

251

.00

-4.43

251

.00

19.43

251

.00

-10.11

251

.00

22.13

251

.00

-7.31

251

.00

Note. 1 T-Test statistics comparing sample mean to ‘no change’ (1); 2 T-Test statistics comparing sample mean to average change mean (2.5).

92

Knowledge change was the only construct index mean with a statistically significant positive difference in the mean, while use of professional assistance and products, use of social networks, and forest management activity were all significantly different from the hypothetical average for the course but were statistically significantly lower. The use of informal education construct mean was not statistically different than the hypothetical average for the course. In addition, means and standard deviations for each of the individual knowledge, use of informal education, use of social networks, use of professional assistance and products and forest management activity items were calculated. Tables 18 through 22 depict the rank order results of this exercise. The means range from slightly over 3 (has increased moderately) to under 2 (has increased minimally) depending on the construct and the specific topic. Under the construct of knowledge change from the course, four of the nineteen items had means slightly greater than 3.00 (‘has increased moderately’). These four related to forest ‘management objectives’ (M=3.08, SD=.89), ‘forestry terms and concepts’ (M=3.05, SD=.91), ‘pine management strategies’ (M=3.04, SD=.94), and ‘forest management plans’ (M=3.03, SD=.88). The relative ranking of these results were logical as these principles were proposed as the basic building blocks of the course. The lowest ranked items related to ‘tax assistance programs’ (M=2.39, SD=.88) ‘federal timber taxation’ (M=2.47, SD=.94), ‘forestry information and technology’ (M=2.57, SD=.91), ‘marketing’ (M=2.57, SD=.92) and ‘estate planning’ (M=2.58, SD=.90). Forest landowners are typically not tax, technology or marketing experts, and these topics are challenging subjects for

93

participants that require specialized training. These rank ordered means are summarized in Table 18.

Table 18 Rank Order Listing of Knowledge Change Items Rank

Item

1

2

Topic Forest management objectives

M 3.08

SD .89

2

1

Forestry terms & concepts

3.05

.91

3

8

Pine management strategies

3.04

.94

4

3

Forest management plan

3.03

.88

5

4

Business aspects

2.91

.86

6

17

Forestry issues that affect management

2.80

.86

7

19

Becoming more involved

2.80

.98

8

11

Forest logging systems

2.79

.96

9

9

Hardwood management strategies

2.75

.98

10

12

Forest management options

2.75

.91

11

14

Forestry services

2.72

.93

12

13

Forest management tradeoffs

2.70

.92

13

18

Forestry organizations

2.66

.90

14

16

Forestry issues that affect ownership

2.64

.91

15

7

Estate planning

2.58

.90

16

10

Marketing forest products

2.57

.92

17

15

Forestry information & technologies

2.57

.91

18

5

Federal timber taxation

2.47

.94

19

6

Tax assistance programs

2.39

.88

94

Similar rankings were noticed for items making up the use of informal education construct although means were on average lower. These rank ordered means are summarized in Table 19. The highest-ranking topics included ‘forest management planning’ (M=2.64, SD=1.03), ‘pine management strategies’ (M=2.59, SD=1.05), ‘forest management objectives’ (M=2.59, SD=1.03), and ‘forestry terms and concepts’ (M=2.57, SD=1.02). These results are at the approximate midpoint between ‘has increased minimally’ and ‘has increased moderately’. The lowest ranking topics included ‘tax assistance programs’ (M=2.19, SD=.91), ‘federal timber taxation’ (M=2.25, SD=.94), ‘forestry information and technology’ (M=2.27, SD=.97), ‘estate planning’ (M=2.33, SD=.99), and ‘marketing forest products’ (M=2.33, SD=.97). These were in similar order as those items in the knowledge change construct closely in order of rank. Under the use of social networks construct, the highest-ranking item included ‘interacting with friends and neighbors’ (M=2.24, SD=1.00), and the lowest ranking item was ‘use of forestland for events’ (M=1.68, SD=.93). These means and standard deviations are summarized in Table 20. From these results it is clear that course participants preferred interacting with others who had similar needs and issues to other social network opportunities. The ‘use of forestland for events’ social network item for example opens up issues of privacy and liability; two subjects for which many landowners have traditionally voiced concern with.

95

Table 19 Rank Order Listing of Use of Informal Education Items Rank

Item

1

22

2

Topic

M

SD

Forest management plan

2.64

1.03

27

Pine management strategies

2.59

1.05

3

21

Forest management objectives

2.59

1.03

4

20

Forestry terms and concepts

2.57

1.02

5

38

Becoming more involved

2.52

1.04

6

31

Forest management options

2.49

.98

7

33

Forestry services

2.48

.98

8

36

Forestry issues that affect management

2.47

1.00

9

23

Business aspects

2.47

.96

10

30

Forest logging systems

2.39

1.03

11

35

Forestry issues that affect ownership

2.39

.98

12

28

Hardwood management strategies

2.36

1.03

13

32

Forest management tradeoffs

2.36

1.02

14

37

Forestry organizations

2.35

1.00

15

29

Marketing forest products

2.33

.97

16

26

Estate planning

2.33

.99

17

34

Forestry information and technologies

2.27

.97

18

24

Federal timber taxation

2.25

.94

19

25

Tax assistance programs

2.19

.91

Under the use of professional assistance and products construct, the highestranking item was ‘use of a professional forester’ (M=2.54, SD=1.15) and the lowest ranking item was use of ‘forestry Internet sites’ (M=1.88, SD=.94). While the use of both of these forms of assistance were encouraged throughout the course, it is clear that most

96

preferred face-to-face interaction than online activities. These are summarized in Table 21. Table 20 Rank Order Listing of Use of Social Networks Items Rank

Item

1

43

2

Topic

M

SD

Interacting with friends & neighbors

2.24

1.00

45

Participation in forestry events

2.18

1.00

3

42

Participation in association

2.10

1.10

4

47

Engagement in forestry issues

1.87

.92

5

59

Participation in forest recognition or certification program

1.85

1.10

6

46

Use of forestland for events

1.68

.93

Table 21 Rank Order Listing of Use Professional Assistance and Products Items Rank

Item

1

39

2

Topic

M

SD

Use of professional forester

2.54

1.15

41

Participation in educational events

2.40

1.05

3

51

Use of forest management plan for implementing practices

2.35

1.09

4

50

Use of forest management plan for decision-making

2.31

1.06

5

48

Use of forestry information

2.24

1.00

6

40

Use of business professional

2.13

1.03

7

49

Use of forestry internet sites

1.88

.94

97

Finally, under the forest management activity construct, ‘monitoring, maintaining and improving wildlife habitat’ ranked highest (M=2.58, SD=1.13), and ‘using financial analyses’ ranked the lowest (M=1.80, SD=.88). The fact that wildlife ranks high is not surprising given the fact that this is a high priority for many landowners (Butler & Leatherberry, 2004). Although knowledge change and increased use of informal education in business aspects increased, financial analyses are abstract concepts for most landowners, and are difficult to understand and implement. These results are summarized in Table 22.

Table 22 Rank Order Listing of Forest Management Activity Items Rank 1

Item 54

2

Topic Monitor, maintain and improve wildlife habitat

M 2.58

SD 1.13

53

Implementation of forest management practices

2.47

1.10

3

55

Monitor, maintain and improve water quality

2.26

1.10

4

56

Maintenance of timber records

2.04

1.00

5

60

Implementation of a recreation or wildlife enterprise

2.01

1.14

6

57

Creation or use of an estate plan

1.96

1.03

7

52

Use of tax assistance programs

1.89

.99

8

58

Use of financial analyses

1.80

.88

98

Findings Related to Research Question #2 The second research question asked, “To what extent can variations in the mediators and outcomes be explained by identified moderators of a voluntary nonformal adult education program?” To answer this research question, statistical tests such as simple correlation, multiple regression, t-Tests and ANOVA analyses were implemented via the PASW 18.0 software program. To answer research question two, it was necessary to understand the variables in a different light. The moderator variables, defined in Table 8 were conjectured to have influence on each of the proximal and intermediate mediator and outcome variables. In the analyses then, the moderator variables became the independent variables and the mediator and outcome variables became the dependent variables. Figure 10 summarizes these variables and the posited relationships. Statistical significance tests were calculated between the 15 independent moderator variables shown in the left oval and the five dependent mediator and outcome variables shown in the right oval. In all, 63 tests were run to determine the relationships between the moderator variables and the mediator and outcome variables. Twelve tests were not conducted due to their illogical nature as explained in the next section.

99

Figure 10 Summary of Moderator, Mediator and Outcome Relationships

Correlation, particularly the Pearson product-moment correlation, was utilized to determine any statistically significant relationships between the continuous moderator variables and the continuous proximal mediator, intermediate mediator and outcome variables. These variables included the independent moderator variables of acreage, tenure, and distance to forestland, importance of economics as an objective, importance of wildlife as an objective, importance of recreation and beauty as an objective, and importance of planning for the next generation and the continuous dependent variables of the proximal mediator, intermediate mediator and outcome variables. Independent sample t-Tests were utilized to determine whether there were differences in the means of the dichotomous independent variables and the continuous dependent variables (Vaske, 2008). The dichotomous independent variables in this study 100

were gender, possession of management plan, creation of management plan, and updating a management plan. The one-way analysis of variance (ANOVA) and stepwise regression analysis were utilized to determine any relationships between the categorical independent moderator variables and the continuous proximal mediator, intermediate mediator and outcome variables (Vaske, 2008). These categorical variables included age class, highest level of education achieved, and income. One personal characteristic, race, was excluded from analysis due to lack of variation in the sample. This resulted in a total of fourteen moderator variables. In cases where the resulting tests showed statistical significance, coefficients of determination were calculated by squaring the correlation coefficients (Vaske, 2008). These coefficients show the proportion of the construct variance explained by the personal or contextual subject variable. In addition, stepwise linear regression was utilized to develop models for predicting change in the dependent variables. Personal and Contextual Moderator Variables and Knowledge Change Not all of the moderator variables were analyzed to determine their relationship with the dependent proximal mediator variable knowledge change. Only age, education, and possession of a management plan prior to participation in the course were analyzed. These were the only moderator variables to have a logical influence on change in knowledge. Of these three, only age was found to correlate with knowledge change at the statistically significant level of .01%. The results of this analysis are displayed in Table 23. The Pearson’s Coefficient is .20. Additionally, age class explained 4.1% of the observed variance in knowledge change.

101

Table 23 Correlation Between Age Class and Knowledge Change Independent Variable Age Class

r .20

p .01

r2 .04

Note: r=Pearson’s Coefficient, p=significance level, r2=coefficient of determination The model from the stepwise regression procedures explaining the greatest amount of variance of knowledge change included age class. This model provides for 4.1% of the observed variance in the dependent variable knowledge change. Model results are summarized in Table 24. Both the unstandardized (b) and standardized (Beta) coefficients are provided. The unstandardized estimates do not take into account the differences in measurement and scale of the independent variables (parameters) while the standardized estimate accounts for this through a standardization procedure involving the means and standard deviations. In this table, the t statistic for each parameter provides an indication of whether the parameters are significantly different than zero. The p values are .00 indicating that the parameter estimates are statistically significant. The F value for the model is 10.61. This value provides an idea of the predictive capability of this model as a whole. A relatively large value such as this indicates the probability of the model having no predictive power (i.e., all parameter estimates are simultaneously zero) is unlikely. Finally, the overall P value for the model provides an indication of the potential for multicollinearity amongst the parameters.

102

Table 24 Regression Results for Predicting Knowledge Change Independent Variable Age Class Model Statistics:

Parameter Estimate (b) .13

Standardized Estimate (Beta) .20

t

p

3.26

.00

R2=.041; F=10.61; P=.001

Personal and Contextual Moderator Variables and Use of Informal Education All of the moderator variables were analyzed in this situation to see if any were significantly correlated with use of informal education. Four of these variables were found to be significantly correlated with this dependent variable. These included age class, importance of wildlife as an objective, importance of recreation as an objective and importance of planning for the next generation as an objective. The moderator variable showing the strongest correlation with use of informal education was Age class. This variable provided for 7.3% of the observed variance in use of informal education. Importance of planning for the next generation (6.4%), importance of wildlife as an objective (3.8%), and importance of recreation as an objective (2.1%) were also statistically significant correlates. Table 25 provides a summary of the statistically significant correlations between the independent moderator variables and the dependent mediator variable use of informal education. In addition, the probability levels are listed as p (the probability that these relationships would occur due to chance, or random errors in the populations studied - for example .01 equates to a 1/100 of a possibility). Finally, the coefficients of determination are listed, which are the squares of the correlation coefficient and represent the amount of variation in the dependent mediator variable (use of informal education) explained by the independent moderator variables (age class, etc.). 103

Table 25 Correlations Between Moderator Variables and Use of Informal Education r

p

r2

Age Class

.27

.01

.07

Importance of Wildlife as an Objective

.20

.01

.04

Importance of Recreation as an Objective

.15

.05

.02

Importance of Planning for the Next Generation

.25

.01

.06

Independent Variable

Multivariate analysis was utilized to understand the predictive ability of these moderator variables on the use of informal education. Only those variables that exhibited statistical significance through the bivariate procedures were included in the analysis. Multiple regression was chosen as the statistical technique, specifically stepwise regression tools in PASW 18.0 were utilized. The choice of this statistical procedure was made because of its ability to add variables sequentially based on the relative size and strength of correlation between the independent and dependent variables (Vaske, 2008). The model from the stepwise regression procedures explaining the greatest amount of variance of use of informal education included age class, and importance of planning for the next generation as an objective. This model provides for 13.1% of the observed variance in the dependent variable use of informal education. Model results are summarized in Table 26.

104

Table 26 Regression Results for Predicting Use of Informal Education Independent Variable Age Class

Parameter Estimate (b) .21

Standardized Estimate (Beta) .27

t

p

3.49

.00

.27

.24

4.03

.00

Importance of Planning for the Next Generation as an objective Model Statistics:

2

R =.131; F=18.25; P=.000

Personal and Contextual Moderator Variables and Use of Professional Assistance and Products Again, all of the potential moderator variables were analyzed to detect any significance in correlation with use of professional assistance and products. The same four variables were significantly correlated with this construct however the strength of the relationships varied. These were age class, importance of wildlife as an objective, importance of recreation as an objective and importance of planning for the next generation as an objective. The strongest independent moderator variable in this case was importance of planning for the next generation as an objective, which explained 10.9% of the observed variance in dependent mediator variable use of professional assistance and products. Importance of wildlife as an objective (4.8%), importance of recreation as an objective (4.8%) and age class (5.4%) were also found to be statistically significant correlates. Table 27 provides the statistically significant correlations for use of professional assistance and products.

105

Table 27 Correlations of Independent Variables with Use of Professional Assistance and Products r

p

r2

Age Class

.23

.01

.05

Importance of Wildlife as an Objective

.22

.01

.05

Importance of Recreation as an Objective

.22

.01

.05

Importance of Planning for the Next Generation

.33

.01

.11

Independent Variable

Using stepwise regression, the best model for explaining the construct of use of professional assistance and products included age class and importance planning for next generation as an objective. This model explained 15.2% of the observed variance in the dependent variable use of professional assistance and products variable. Model results are summarized in Table 28.

Table 28 Regression Results for Predicting Use of Professional Assistance and Products Independent Variable

Age Class

Parameter Estimate (b) .17

Standardized Estimate (Beta) .22

t

p

3.78

.00

.34

.32

5.46

.00

Importance of Planning for the Next Generation as an Objective Model Statistics:

R2=.158; F=22.44; P=.000

Personal and Contextual Moderator Variables and Use of Social Networks All potential moderator variables were analyzed to detect any significance in correlation with use of social networks. As in the previous two situations, the same four variables were the only significantly correlated variables with this construct. Again, 106

however the strength of the relationships varied. Those significant were age, importance of wildlife as an objective, importance of recreation as an objective and importance of planning for the next generation as an objective. The strongest independent moderator variable was age class, which explained 8.4% of the observed variance in use of social networks. Importance of planning for the next generation as an objective (7.8%), importance of wildlife as an objective (4.7%), and importance of recreation as an objective (3.8%) were also found to be statistically significant. Table 29 provides a summary of the statistically significant correlations for the independent moderator variables and the dependent mediator variable use of social networks.

Table 29 Correlations of Independent Variables with Use of Social Networks r

p

r2

Age Class

.29

.01

.08

Importance of Wildlife as an Objective

.22

.00

.05

Importance of Recreation as an Objective

.20

.00

.04

Importance of Planning for the Next Generation

.28

.00

.08

Independent Variable

Using stepwise multiple regression, the best model for explaining use of social networks included age class and importance of planning for the next generation as an objective, and importance of wildlife as an objective. This model explained 17.7% of the observed variance in the dependent variable use of social networks. Model results are summarized in Table 30.

107

Table 30 Regression Results for Predicting Use of Social Networks Independent Variable

Parameter Estimate (b) .24

Standardized Estimate (Beta) .30

4.18

.00

Importance of Planning for the Next Generation as an Objective

.23

.21

3.77

.00

Importance of Wildlife as an Objective

.15

.15

.33

.02

Age Class

Model Statistics:

t

p

2

R =.177; F=17.15; P=.000

Personal and Contextual Moderator Variables and Forest Management Activity All potential moderator variables were analyzed to detect any significance in correlation with forest management activity. As in the previous three situations, the same moderator variables were the only significantly correlated variables with this construct. Again, however the strength of the relationships varied. Those significant were age, importance of wildlife as an objective, importance of recreation as an objective and importance of planning for the next generation as an objective. The strongest independent moderator variable again was importance of planning for the next generation as an objective, which explained 9.2% of the observed variance in the dependent forest management activity variable. Age class (6.0%), importance of wildlife as an objective (7.3%), and importance of recreation as an objective (5.3%) were also found to be statistically significant. Table 31 provides a summary of the statistically significant correlations between the independent moderator variables on the dependent mediator variable forest management activity.

108

Table 31 Correlations of Independent Variables and Forest Management Activity r

p

r2

Age Class

.25

.01

.06

Importance of Wildlife as an Objective

.27

.00

.07

Importance of Recreation as an Objective

.23

.00

.05

Importance of Planning for the Next Generation

.30

.00

.09

Independent Variable

Using stepwise multiple regression, the best model for explaining forest management activity included age, importance of planning for the next generation as an objective, and importance of wildlife as an objective. This model explained 17.1% of the observed variance in the dependent variable forest management activity. The model is summarized in Table 32.

Table 32 Regression Results for Predicting Forest Management Activity Independent Variable

Parameter Estimate

Standardized Estimate

(b)

(Beta)

Age Class

.20

Importance of Planning for the Next Generation as an Objective Importance of Wildlife as an Objective Model Statistics:

2

t

p

.25

4.22

.00

.25

.23

3.84

.00

.17

.17

2.33

.02

R =.171; F=16.43; P=.000

109

Findings Related to Research Question #3 The third research question asked, “What relationships exist between the mediators and the outcomes of a voluntary nonformal adult education program?” To answer this research question, correlation analyses were first used to determine the relationships between the proximal mediator variable of knowledge change and the outcome variable: forest management activity. In this case, knowledge change became the independent variable and forest management activity became the dependent variable (see Figure 11). The results are listed in Table 33. The relationship was significant at the .01 level and the variable knowledge change accounted for 32% of the variation in the change in the forest management activity variable. Model statistics are provided in Table 34.

Figure 11 Proximal Mediator-Outcome Relationship

Table 33 Correlations Between Knowledge Change and Forest Management Activity Independent Variable Knowledge Change

110

r

p

r2

.57

.01

.32

Table 34 Regression Results for Predicting Forest Management Activity Independent Variable

Parameter Estimate

Standardized Estimate

(b)

(Beta)

.67

.57

Knowledge Change Model Statistics:

t

p

10.84

.00

2

R =.32; F=117.56; P=.000

Next, the bivariate relationships between the proximal mediator variable knowledge change and the intermediate mediator variables of changes in use of informal education, use of professional assistance and products, and use of social networks were calculated. In this situation, the proximal mediator variable knowledge change became the independent variable and the other three intermediate mediator variables became the dependent variables (see Figure 12). Each of the relationships was significant at the .01 level. In addition, knowledge change accounted for 46% of the variation in the change in use of informal education, 40% of the change in the use of professional assistance and products, and 23% of the variation in the use of social networks. The results are listed in Table 35. Since the proximal mediator (knowledge change) was considered to have an influence on the intermediate mediators (use of informal education, use of social networks and use of professional assistance and products) as well (see Figure 3), and each one of the correlations were significant at the .01 level (see Table 35), these variables were next regressed to determine parameter, betas and coefficient of determinations (R2’s).

111

Figure 12 Proximal Mediator-Intermediate Mediators Relationship

Table 35 Correlations of Knowledge Change and Independent Intermediate Mediators r

p

r2

Use of Informal Education

.68

.01

.46

Use of Professional Assistance and Products Change

.63

.01

.40

Use of Social Networks

.48

.01

.23

Dependent Variable

The results are displayed in Tables 36-38. These two variable models with knowledge change as the independent variable and the intermediate mediators as the dependent variables explained 23% of the variance in use of social networks, 40% of the variance in the use of informal education, and 40% of the variance in the use of professional assistance and products. In each model, the t values were significant at the

112

.00 level for the knowledge change parameter. Also, for each of the models, the F statistics were large and the P values were .00.

Table 36 Regression Results for Predicting Use of Informal Education Proximal Mediator

Knowledge Change Model Statistics:

Parameter Estimate

Standardized Estimate

(b)

(Beta)

.82 R =.47; F=217.97; P=.000 2

.68

t

p

10.84

.00

Table 37 Regression Results for Predicting Use of Professional Assistance and Products Proximal Mediator

Parameter Estimate

Standardized Estimate

t

p

(b)

(Beta)

.73

.63

12.89

.00

Parameter Estimate

Standardized Estimate

t

p

(b)

(Beta)

.57

.48

8.72

.00

Knowledge Change Model Statistics:

2

R =.40; F=166.30; P=.000

Table 38 Regression Results for Predicting Use of Social Networks Proximal Mediator

Knowledge Change Model Statistics:

2

R =.23; F=75.99; P=.000

113

Finally, the bivariate relationships between the intermediate mediator variables and the outcome variable forest management activity were calculated using Pearson’s Coefficient (Figure 13).

Figure 13 Intermediate Mediator-Outcome Relationship

The results are displayed in Table 39. Each of these relationships was significant at the .01 level. The strongest relationships were between use of professional assistance and products and forest management activity and use of social networks (.84), and forest management activity (.84). The other relationship, that between use of informal education and forest management activity was also strong (.72).

114

Table 39 Correlation of Intermediate Mediator Variables and Forest Management Activity Use of Informal Education

Forest Management Activity

Use of Professional Assistance and Products

.72**

.84**

Use of Social Networks

.84**

Note. **p