Increasing Student Interest in Science, Technology, Engineering ...

Increasing Student Interest in Science, Technology, Engineering, and Math (STEM): Massachusetts STEM Pipeline Fund Programs Using Promising Practices Prepared for the Massachusetts Department of Higher Education March 2011

Acknowledgements

Increasing Student Interest in STEM

Acknowledgements The concept for this report evolved from valuable conversations with the former Pipeline Director, David McCauley and from Program Manger, Keith Connors. Information in this report has been compiled from reports and data submitted by Pipeline funded programs over the past five years. Many stakeholders have been involved with these programs including students, teachers, parents, industry professionals, and others throughout the state. It is their support, the ongoing work of the dedicated professionals leading these programs, and the thoughtful work of their local evaluators that have made this overview possible. Special thanks should be given to the thousands of students and teachers who participated in these programs all over the state and provided their feedback and insights.

For further information regarding this report, please contact: Sonia Bouvier, MPH Research Analyst/Coordinator University of Massachusetts Donahue Institute Email: [email protected] Telephone: (774) 455-7373

For more information on any of the projects reviewed in this report, or the STEM Pipeline Fund in general, please contact Keith Connors at the Department of Higher Education at: Keith Connors STEM Pipeline Fund Program Manager One Ashburton Place, Room 1401 Boston, MA 02108 E-mail: [email protected] Telephone: 617-994-6911

UMass Donahue Institute Research and Evaluation Group

i


Contents

Contents Acknowledgements ...........................................................................................................................i Executive Summary .........................................................................................................................iv Introduction .......................................................................................................................................1 Background: ............................................................................................................................1 Program Evaluation Requirements: ........................................................................................2 Literature Review: ...................................................................................................................2 Purpose of the Report: ............................................................................................................6 Criteria and Methodology.................................................................................................................8 Criteria for Inclusion in this Report: .........................................................................................8 Additional Explanation of Program Outcome Criteria:.............................................................8 Methodology:...........................................................................................................................9 Organization of the Report: .....................................................................................................9 What is Measureable Change: ..............................................................................................10 Program Profiles .............................................................................................................................11 Programs that Conducted Pre-Post Assessments: Measureable Results ............................11 Biotech Career Pathways......................................................................................................11 DIGITS ..................................................................................................................................13 Engineering is Elementary ....................................................................................................16 Got Math?..............................................................................................................................18 STEM Summer Camps .........................................................................................................20 Programs that Conducted Pre-Post Assessments: No Measureable Results.......................22 Engineering the Future by Design.........................................................................................22 STEM Fellows and Leaders Project ......................................................................................24 STEM RAYS (Science, Technology, Engineering & Mathematics Research Academies for Young Scientists) ..................................................................................................................26 Programs that Used Post-only or Qualitative Assessments..................................................28 Berkshire STEM Career Fair .................................................................................................28 Family Science Programs .....................................................................................................30 MA Region I Middle & High School Science Fairs ................................................................32 Saturday STEM Academy Middle School Project .................................................................35 STEM Career Awareness Conferences ................................................................................37 Summary of Results and Conclusion ...........................................................................................40 Appendix A: Detailed Explanation of Program Headings ...........................................................44


ii


Contents

Appendix B: Examples of Questions and/or Instruments...........................................................46 Biotech Career Pathways......................................................................................................46 DIGITS ..................................................................................................................................47 Sustainability Camp...............................................................................................................48 Got Math?..............................................................................................................................49 Engineering the Future by Design.........................................................................................50 STEM Fellows .......................................................................................................................52 STEM RAYS After-School Science Program ........................................................................55 Berkshire STEM Career Fair .................................................................................................56 Berkshire Family Science Program .......................................................................................58 MA Region I High School Science and Engineering Fair ......................................................60 Student Survey 2010.............................................................................................................60 Saturday STEM Academy .....................................................................................................63 Men in STEM Conference .....................................................................................................65


iii

Executive Summary


Executive Summary The purpose of this report is to profile STEM Pipeline funded programs that have employed promising practices to increase student interest in STEM (Science, Technology, Engineering and Mathematics). More specifically the intent of this report is to:

● Describe how the programs were designed ● Document what the programs accomplished ● Summarize program evaluation methods used

Programs were selected for inclusion in this report if they met all of the following criteria: I. Program Design Criteria

II. Program Outcome Criteria

A. Designed to increase student interest in STEM B. Employed practices that research indicates are effective in affecting STEM interest, such as: o Content was related to real-world application, ideally through hands-on learning o Critical thinking, collaboration, and small group work was encouraged o Caring adults (i.e., parents, teachers or STEM professionals) participated o Information about STEM career opportunities was provided

A. Provided data on student interest in STEM B. Assessed changes in STEM interest in relation to program participation C. Assessed interest in STEM more generally, beyond the program itself D. Collected student interest data directly from students

Programs funded by the STEM Pipeline during the 2009-2010 year (whether by the 2007 Networks Grants or by the 2008 Student Interest Grants) were considered for inclusion. The review included all current and historical materials submitted to the Department of Higher Education as part of a statewide evaluation process, as well as reports completed by local evaluators. Frequently, a single organization implemented two or three programs with different approaches and objectives and local evaluators conducted tailored evaluations specifically for each program. As a result, each program was considered separately for inclusion in this report. Program Profiles: Each program profile is two to three pages long, and begins with an overview of the program’s purpose and design. Following the overview, information on the target population, program length, frequency (i.e., one-time offering or series), and program setting (i.e., in-school or out-of-school) is provided. Profiles include descriptions of the evaluation methods used, and provide summaries of the student interest and other outcome data collected. They conclude with brief commentary on the program’s use of the research-supported promising practices discussed in the Introduction, and a summary of challenges and lessons described in program reports. The profiles are presented in three groups according to whether program outcome data indicated that a “measureable” change in student interest had taken place: (1) Programs that conducted pre-post assessments where results showed measureable increases in student interest, (2) Programs that conducted pre-post assessments where no measureable change in student interest was identified, and (3) Programs that used post-only or qualitative assessments where no measureable change was identified. Definitions of measureable and other terms are at the end of this section. Brief summaries of each program profiled, and its evaluation results, are provided below. Programs that Conducted Pre-Post Assessments: Results Showed Measureable Increases in Student Interest BioTech Career Pathways: Norfolk County Agricultural High School partnered with a hospital, college and veterinary school to offer two opportunities for students to be exposed to biotech and other STEM careers. A six-week internship


iv


Executive Summary

for juniors included observing surgeries, using advanced radiography, performing necropsies and tissue harvesting. Workshops for seniors covered agro-terrorism, hematology, laboratory biosafety, zoonosis and public health. Results: High school seniors’ and juniors’ average rating of their level of awareness of, interest in and motivation to pursue STEM related careers was higher after their internship and workshop experiences.

DIGITS: Professionals employed in STEM fields visited sixth grade classrooms in order to increase student interest in, and understanding of, STEM careers. These “STEM Ambassadors” lead discussions about the importance of math and science, shared their experience working in STEM, and lead hands-on activities designed to stimulate discussion and challenge stereotypes about STEM careers. Additional elements included a poster, website and music video. Results: Results of pre-post surveys of students showed a measureable improvement in students’ positive attitudes toward STEM subjects and jobs. The strength of students’ agreement with statements that math, science or technology (each asked separately) was “interesting” or “fun” increased as well as the percentage of students who viewed STEM jobs as fun and exciting.

Engineering is Elementary: Over 200 elementary teachers from four districts were trained in this curriculum, which uses hands-on, inquiry-based experiences to encourage learning in STEM subjects, particularly engineering and technology. It is available from the Boston Museum of Science and is aligned with state education frameworks. Results: A review of raw data indicated there was a positive shift in students’ sense of self-efficacy in math and science. Student responses demonstrated more positive views of becoming an engineer on the post survey. While only 25% of students initially agreed with the statement “When I grow up I want to be an engineer,” 35% agreed with this statement after the program.

Got Math?: Elementary and middle school students participated in a series of math after-school instructional activities. Students then visited local businesses where they had the opportunity to apply the math skills learned to a real life situation. Activities were intended to improve students’ skills and attitudes toward math. Results: Pre-post survey results showed a measureable increase in students’ sense of self-efficacy regarding math. Responses to other questions that asked how much students liked math, enjoyed math puzzles or would like to participate in activities such as math club, did not demonstrate changes.

STEM Summer Camps: Middle school students participated in two, five-day long camps: “Sustainability and the Environment” and “Biotechnology and Forensics.” The camps were designed to engage students in fun, hands-on, content-based inquiry science that would develop their subject knowledge and awareness of careers in STEM fields. Results: The percentage of students who stated they viewed science as “fun” rose after participation in each camp, from 45% to 66% after the Sustainability camp and 61% to 72% after the Forensics camp.

Programs that Conducted Pre-Post Assessments: No Measureable Change in Student Interest Identified Engineering the Future by Design: Middle school teachers attended a two-day summer engineering curriculum workshop. They were encouraged to implement the curriculum, which included hands-on activities, with their students the following school year. Subjects included: Wind Turbines, Bottle Rockets, Structures & Supports, and Hydroponics. Results: Pre-post surveys that were designed to assess students’ preference for STEM or engineering activities over other activities did not show any measureable changes after the program. However, the evaluator reported observing a high level of student participation and interest in the classroom activities. Teacher participants also reported a high level of satisfaction with the curriculum.

STEM Fellows and Leaders Project: Teachers received thirty hours of professional development to increase content knowledge and pedagogical skills. They then worked in teams to design and implement strategic “Capstone Plans” to increase the numbers and performance of students interested in STEM subjects and careers in their districts. Results: Results of pre-post surveys designed to assess the program’s effect on students’ interest in STEM were inconclusive. According to the local evaluator, students generally indicated positive attitudes and beliefs about STEM subjects prior to the program, which made it difficult to show a positive change in interest. Teachers who responded to a post-only survey reported an increase in their own sense of self-efficacy and their belief in the importance of promoting and teaching STEM.


v


Executive Summary

STEM Rays: Students in fourth through eighth grade participated in after-school science clubs, conducted authentic scientific research throughout the school year. Groups were led by certified teachers who received professional development and monthly guidance from scientists and researchers at local colleges. Results: Pre-post survey results did not indicate any measureable change in students’ interest in pursuing STEM careers. However, other qualitative information gathered from surveys, observations and interviews with teachers, parents and administrators suggest participation may have increased student interest in STEM.

Programs that Used Post-only or Qualitative Assessments: No Measureable Change Identified Berkshire STEM Career Fair: This annual career fair introduced high school students to STEM careers and professionals. Each year, 15 to 25 area business/employer representatives participated as exhibitors, provided displays with information about their companies, and talked with students about potential career opportunities. Results: Nearly eighty percent (79%) of students who responded to a post-program survey reported the fair helped them realize that the math and science they take in school will affect their future career options. Over one third (35%) indicated they were thinking more about going into a high-tech career than before the fair. More than two thirds reported the career fair made them realize that a STEM career could be interesting.

Family Science Programs: Middle school students and their parents participated in hands-on learning activities together. Activities were led by STEM professionals and were intended to introduce students to STEM careers. Programs were offered in out-of-school environments with activity topics, timing, and frequency varying greatly. Results: Students surveyed after the program reported that their interest in and understanding of STEM subjects and careers increased as a result of their participation. Parents’ responses to pre-post surveys did not indicate expectations for their child’s science course taking in high school had changed.

Massachusetts Region I Middle and High School Science Fairs: 350 students from western Massachusetts entered projects on science, math or engineering subjects. Students often spent months doing research before they presented their exhibit at the fairs. Top projects could advance to compete at the state, national and international fairs. Results: When high school students responded to a survey conducted after the fair, 41% indicated that participating in the fair was a “very important” or “important” benefit in "building my confidence in my science abilities." 72% reported that they are planning on majoring in a STEM field in college and 74% indicted they were considering a career in a STEM field.

Saturday STEM Academy: Eighth-grade students and their parents participated in STEM-related activities over four Saturdays. Each day involved a field trip to a different location to participate in hands-on activities and meet STEM professionals. The program was to increase underserved students’ interest in STEM academics and careers. Results: Two-thirds of student respondents to a post-program survey agreed they were now more motivated to “study math and science in high school” and “to prepare myself to go to college.” All students agreed with “Math and Science is important for me to be successful in life” and over 80% of agreed that the subjects of math, science and engineering are important and interesting.

STEM (Central) Career Awareness Conferences: These Saturday conferences were designed to provide middle school students with STEM career information and inspire interest in STEM. Each one included a large group keynote address or panel discussion in which one or more professionals shared how and why they became employed in STEM, and the challenges and rewards they encountered. Students rotated through small-group workshops delivered by STEM professionals who interacted with students and often offered hands-on activities related to their work. Results: According to open-ended questions asked after the program, the majority of students reported their interest in STEM subjects and careers had increased. The percentage of respondents indicating increased interest was not reported.

-----------------------------------------------------------------------------------------------------------------------------------------------------------Results and Recommendations: The thirteen programs profiled in this report highlight that the Pipeline Fund has supported a wide variety of programs across the state, reaching students from elementary through high school. All of the programs profiled in this report employed practices that have been described as promising in research literature. The diverse approaches of the


vi


Executive Summary

programs, and the resulting variation in evaluation methods and outcome data, made it challenging to make other broad-based statements regarding their effect on student interest in STEM. Five of the thirteen programs profiled in this report provided data that demonstrated a measureable increase in student interest after program participation. While the remaining programs did not demonstrate a measureable change in student interest, results often suggest that changes in student interest occurred, but were not fully captured by the evaluation methods used. These results mirror similar studies across the country, as others have found it challenging to assess student interest and to make generalized statements about the effect of STEM education programming, particularly in informal settings. The same variety that makes it difficult to make broad statements about the influence of STEM programming on student interest in Massachusetts may also be the state’s strength. No single program will ever be able to meet the needs of all students when it comes to encouraging student interest in STEM. However, the range of Pipeline programs offered throughout the state made it possible for students of all ages and levels of interest to participate in local programming appropriate to their age, ability and interest level. Those who implement STEM programs in the future have a range of programs and evaluation methods to draw upon as potential models. Below are recommendations to strengthen and support implementation and assessment of student STEM interest programs. These recommendations, which come from recent literature, could benefit future Massachusetts programs: 1. Compile a list of tools to assess STEM programming for future programs to draw upon. This report provides examples of instruments and questions that were used by the programs to assess student interest. Linking program developers to other existing sources of instruments commonly used in the field could also be helpful. The National Science Foundation and others have supported the creation of such compilations. 2. Standardize a small number of student interest evaluation questions to ask across all programs, even if program designs vary. Massachusetts could encourage the use of identical, age-appropriate questions to be asked of students who participate in STEM programming. This would make is easier to assess the effect of Massachusetts’ programming as a whole and to compare program outcomes. 3. Consider the creation of a larger mechanism for sharing information on STEM program development and evaluation. Programs could benefit by communicating with others implementing STEM programming. Sharing resources, ideas, methods, lessons learned and evaluation strategies will reduce duplication of effort. Ideally such a mechanism would also enable programs in Massachusetts to access current research information and assessment resources outside of the state. It is our hope that this report will be a first step in following through on the above three recommendations by providing shared information to support future STEM programming and evaluation. -----------------------------------------------------------------------------------------------------------------------------------------------------------Definition of Terms: For the purposes of this report, the term “measureable” means that two conditions were met. First, that an assessment of student interest in STEM was conducted before and after program implementation (referred to as a pre-post assessment). Second, that there was an observable difference in levels of student interest documented by the two assessments. Assessments of student interest that were administered only once at the end of the program are referred to as “post-only” (i.e., no baseline, or “pre” results were established). By definition, postonly results cannot document measureable change, as there are no results from before the program against which to compare. However, post-only results can still provide valuable information. Note on Inclusion: Programs did not have to show evidence of increasing student interest to be included in this report. Inclusion does not imply that the program was effective – only that it presented some data on student interest and met other report criteria. Other programs may also be effective, but did not collect data on student interest or did not have adequate time to collect outcome data. While this report asserts that some program results showed a measureable change in student interest, this does not mean that the change was statistically significant or proven to be a direct result of the program.


vii


Introduction

Introduction Background: The Pipeline Fund was established through a Legislative appropriation under the Acts of 2003 Economic Stimulus Trust Fund and refunded under the 2006 Economic Stimulus Bill. The Massachusetts Department of Higher Education (DHE) administers the Fund with a focus on the following three goals: (1) To increase the number of Massachusetts students who participate in programs that support careers in fields related to mathematics, science, technology, and engineering (STEM); (2) To increase the number of qualified mathematics, technology, engineering and science teachers in the Commonwealth; and, (3) To improve the mathematics, technology, engineering and science educational offerings available in public and private schools. This report reviews programs that received funding through two STEM Pipeline Requests for Proposals (RFP). The first RFP was titled “A Grant Opportunity for 2007 Regional PreK-16 Networks to Advance Interest and Learning in Science, Technology, Engineering and Mathematics.” This RFP encouraged programming addressing all three of the Pipeline Fund’s goals in conjunction with the formation of regional K-16 STEM collaboratives/networks. The second RFP (funded in 2008) was titled “Enhancing Student Interest and Retention in STEM Fields.” This RFP focused primarily on the goal of increasing the number of students who participate in STEM programs. Programs funded under the first RFP are commonly called “Network” programs, and projects supported by the second RFP are often referred to as “Student Interest” programs. In addition to focusing on increasing the number of students who participate in STEM programming, the second, 2008, RFP asked applicants to concentrate on increasing student interest through both STEM awareness and STEM learning. The RFP suggested that to increase STEM awareness, programs should address misconceptions about STEM careers, and promote STEM career opportunities through guest speakers, field trips to STEM businesses, student internships, and other activities. It further recommended that programs increase awareness of STEM subjects and STEM careers among not just students, but also parents, teachers and school administrators. To increase STEM Learning, the RFP suggested programs design activities that would “improve student and teacher content knowledge and teacher pedagogical skill.” Programs were encouraged to use innovative strategies both in and out of the classroom, including small group settings and the use of hands-on activities and experiments to promote inquiry and curiosity. Finally, the RFP recommended connecting learning to the real world through an emphasis on “the application of STEM subjects to everyday life, employment and the surrounding environment.” The RFP specifically encouraged active employer participation in project activities. The majority of programs profiled in this report were supported through student interest funding. Both funding streams were initially expected to last for three years (programs under the first RFP were to be funded spring 2007-2010 and programs under the second RFP were to be funded spring 2008-2011). However, the state legislature had to transfer almost $2M from the STEM Pipeline Fund to the General Fund as one of many recession-related budget cuts. As a result, the DHE had to eliminate the final year of student interest funding, forcing the projects to conclude by the spring of 2010 (a full year earlier than they originally planned). As might be expected, the early cut in funding had a significant effect on program implementation and evaluation efforts.


1


Introduction

Program Evaluation Requirements: The University of Massachusetts Donahue Institute (UMDI) has served as the statewide evaluator to the Massachusetts STEM Pipeline Fund since its inception in 2005. In this capacity, UMDI has coordinated standardized evaluation and reporting requirements across all projects, analyzed information submitted to DHE in fulfillment of these requirements, and overseen a statewide indicators project designed to benchmark the Commonwealth’s progress in STEM education. Funded organizations were required to submit bi-annual reports (mid and year-end) that included information such as: ● Participant numbers (total, student or teacher, gender, race/ethnicity, grade level, other special attributes, etc.) ● Schools/districts involved (including data on MCAS performance and the percentages of low-income and minority students) ● Attributes of program activities (type, length, frequency, kinds of learning experiences, etc.) ● Educational background/experience of adults who participated in teacher professional development activities (years teaching, undergraduate and graduate degrees, licenses held, etc.) Programs were expected to submit the results of any tests or surveys that assessed content knowledge, attitudes or feedback, as well as the instruments. Reports included a narrative in which program directors described program highlights and accomplishments over the most recent reporting period, challenges encountered, lessons learned and planned future activities. Most programs hired a local evaluator to design and conduct detailed evaluations. This requirement “include[d] both formative (i.e., yielding information that informs project improvement) and summative (i.e., documenting outcomes and progress toward goals) components.” UMDI has collected and reviewed the statewide and local program evaluation data, providing summaries and analysis as requested by DHE.

Literature Review: As a foundation for this report, a literature review was conducted to summarize answers to the following questions: ● ● ● ●

Why is student interest important to the STEM pipeline? What is student interest and how can it be measured? What are promising practices in STEM programming and how have they been identified? How have successful STEM programs been profiled by others?

Question 1: Why is student interest important to the STEM pipeline? While a number of STEM Pipeline Fund programs were designed to increase teachers’ and students’ STEM knowledge and skills, the focus of this report is on programs designed to increase student interest in STEM. A recent model of the STEM education pipeline highlights the importance of student interest to increase the number of students choosing to enter STEM fields. In April 2010, the Business High Education Forum (BHEF) published a working paper 1 describing a “system dynamics model of the U.S. STEM Education system” developed by the Raytheon Company. The model recognizes that to increase the number of individuals entering STEM majors in college for eventual employment in STEM fields, students must be both proficient and interested in STEM. Ongoing research sponsored by the DHE and conducted by the UMDI reinforces this model. Analysis of Massachusetts’ public school students’ answers to survey questions from the SAT (Scholastic Aptitude Test) has shown that students who attend higher performing schools (as determined by schools’ aggregate scores on the Massachusetts Comprehensive Assessment System tests) are, in general, less interested in STEM college majors than students who attend lower performing schools. As a result, there are many students who are STEM capable (i.e., attend high schools that are strongly college-prep oriented) but not interested in STEM college majors.

1

Business Higher Education Forum. (April, 2010) Increasing the Number of STEM Graduates: Insights from the U.S. STEM Education and Modeling Project. (As retrieved from: http://www.bhef.com/solutions/documents/BHEF_STEM_Report.pdf.) UMass Donahue Institute Research and Evaluation Group

2

Introduction


It is important that students become interested in STEM before they enter high school so they will enroll in courses that can prepare them for STEM majors or careers. According to one study, 2 94% of eighth graders make coursetaking decisions related to preparing themselves for postsecondary education or a career. According to another, 3 (1) science achievement correlates with attitudes toward science, and (2) positive perceptions of science are lower among older students than younger. In light of these studies, middle school students who do not consider a STEM major or career as a possible option, or at least maintain a positive orientation toward STEM subjects, may not enroll in the necessary high school coursework that would allow them to properly prepare for, or enter, those fields later on. Ensuring that students take a critical mass of STEM preparatory coursework in high school is especially important for girls. According to a 1988 study, “…the number of mathematics and science courses taken in high school was the predominant factor…” in females’ selection of quantitative undergraduate majors: “It had the greatest direct influence 4 on field of study and served as the mediating variable for all indirect influences.” So, middle school interest in STEM is important to students’ becoming prepared for STEM majors, which, in turn, is important to students’ following through in majoring in a STEM field. In addition, this study found that being interested in a quantitative major in tenth grade had a significant effect on whether females chose quantitative majors in college. Consequently, not only is it important that interest in STEM be developed early, but that it be sustained. Question 2: What is student interest and how can it be measured? The Noyce Foundation commissioned a 2008 study to “review and report on the current state and needs of the after5 school science assessment world.” The study was conducted by the Program in Education, Afterschool & Resiliency (PEAR) at Harvard University and McLean Hospital. PEAR researchers adapted and modified a framework that has been used by the National Science Foundation (NSF) for evaluating the effect of informal science education programs. 6 The framework described five domains in which programs can have an affect on participants. While these domains were originally intended to assess the effect of informal science education programs, they have relevance for programs in technology, engineering and math as well. The five domains were:

● Interest and Engagement ● Attitude and Behavior ● Content Knowledge

● Competence and Reasoning ● Career Knowledge and Acquisition

PEAR researchers identified components of student outcomes that could be assessed under each domain, then used the domain framework to group and classify instruments commonly used to evaluate informal science education programs. This framework was used to determine what constituted assessments of student interest for this report. The first two domains align most closely with STEM interest while the other three domains align most closely with STEM proficiency. However, one modification was needed to clarify what could be considered student interest data for the purposes of this report, namely a clearer distinction between assessments of career knowledge versus the intent to engage in a STEM career. Since the PEAR study placed the desire to become a scientist under the Attitude and Behavior domain, Pipeline program assessments of the intent to engage in a STEM career or participate in STEM activities were also considered to be measures of student interest under the Attitude and Behavior domain. Assessments of knowledge about STEM careers and pathways remained under the last domain, and were not considered to be assessments of student interest. These clarifications are added in bold italics below.

2

Akos, Patrick; Lambie, Glenn W.; Milsom, Amy; Gilbert, Kelly. (2007). Early Adolescents' Aspirations and Academic Tracking: An Exploratory Investigation. From Professional School Counseling, Vol. 11, Issue 1. 3 Neathery, M. Faye. (1997) Elementary and Secondary Students’ Perceptions toward Science and the Correlation with Gender, Ethnicity, Ability, Grade, and Science Achievement. From Electronic Journal of Science Education Vol. 2, No. 1. 4 Ethington, Corinna A., and Lee M. Wolfe. (1988). Women’s Selection of Quantitative Undergraduate Fields of Study: Direct and Indirect Influences. From American Educational Research Journal. Vol. 25, No.2. 5 Hussar, K., Schwartz., Boiselle, E., & Noam, G. (2008). Toward a Systematic Evidence-base for Science in Out-of-School Time: The Role of Assessment. Retrieved from Harvard University and McLean Hospital, Program in Education, Afterschool & Resiliency website: http://www.pearweb.org/pdfs/Noyce-Foundation-Report-ATIS.pdf. 6 Friedman, A. (Ed.). (March 12, 2008). Framework for Evaluating Impacts of Informal Science Education Projects [On-line]. (Available at: http://insci.org/resources/Eval_Framework.pdf) UMass Donahue Institute Research and Evaluation Group

3

Introduction


Domains:

Components:

Interest/ Engagement

Curiosity in STEM–related activities/issues Excitement about / Enthusiasm for engaging in STEM activities Fun / Enjoyment / Interest in STEM Activities

Attitude / Behavior

Desire to become a scientist / engage in a STEM career Level of Participation Intent to Participate in STEM Activities Belief that science / math is sensible, useful and worthwhile Belief in one’s ability to understand and engage in science and math Reduced anxiety / trepidation around STEM Positive scientific / math identity Pro-social / adaptive learning behaviors in relation to STEM

When reviewing program outcome data to determine which programs would be included in this report, results were considered to be assessments of student interest if they concerned any of the components described above. For example, if a survey question asked students whether they found STEM subjects or activities to be fun, exciting or enjoyable, the results would be considered to be student interest data. Assessments of students’ intent to participate in a STEM activity (or engage in a STEM career) or measures of student perceptions regarding the importance of math or their own STEM abilities were considered to be assessments of student interest. Assessment results that did not concern the domains and components listed above were not considered to be measures of student interest. Data on changes in student interest could be quantitative or qualitative in nature, and collected through a variety of surveys (i.e., pre-post tests, or post-only assessments) or other tools. Question 3: What are promising practices in STEM programming and how have they been identified? A number of methods have been used to determine what can be considered a “promising practice” in STEM education programming in general, and in student interest programming in particular. In general, researchers in the field have sought evidence that the domains and components discussed under Question 2 have been affected by programming. Three leading efforts to identify promising practices have been taken by Building Engineering and Science Talent (BEST), Girl Scouts of America, and Project Tomorrow. After a description of each of these efforts, we provide a synthesis of the promising practices they identified. 1. Building Engineering and Science Talent (BEST) : A direct review of programs BEST (a public-private partnership established on the recommendation of a Congressional Commission) identified a number of promising practices in 2004, after a review of STEM programs nationwide to identify evidence of their success. 7 BEST’s goal was to review a sample of K-12 STEM Education programs designed to reach underrepresented students in STEM fields and identify those with research-based evidence of effectiveness. Over 200 programs across the country were nominated for the project. However, only thirty-four were eventually selected for review, based in part on whether they had a track record of data or other studies that demonstrated their effect. BEST reviewed formal (i.e., structured in-school curricula) and informal programs (i.e., afterschool programming) that were designed to affect students on a variety of outcome levels, including achievement and performance, behavior (such as course or activity selection) and attitude toward or interest in STEM subjects or careers. BEST reviewed the evaluation data from these programs to identify those that had research-based evidence of effectiveness. BEST then developed and published a listing of common design principles shared by the successful programs. 2. Girl Scouts of America: A literature review and survey of educators A study commissioned by the Girl Scouts of America and the National Center for Women Information and Technology (NCWIT), sought to identify promising practices in programming to increase girls’ interest in, and engagement with STEM, particularly in informal education settings. 8 Researchers from the Puget Sound Center for Teaching, Learning

7

Building Engineering and Science Talent. (2004) What it Takes: Pre –K-12 Design Principles to Broaden Participation in Science, Technology, Engineering Math. Retrieved from http://bestworkforce.org/index.htm on September 18, 2010. UMass Donahue Institute Research and Evaluation Group

4

Introduction


and Technology (Liston, 2008) conducted a literature review of effective programs, reviewed multiple studies in existing research and distilled 33 practices that researchers frequently cited as “promising” in the literature. They then surveyed 123 programs in 36 states that implemented STEM programming. Respondents were asked to rate which practices they felt most strongly contributed to the success of their programs. 3. Project Tomorrow: A survey of students, parents and teachers Project Tomorrow released a report in 2008 with the results of its nationwide Speak Up survey of students, parents and teachers on science education programs. 9 This survey asked students in grades three through twelve what experiences they felt would be most helpful to increase their interest in STEM careers. Students identified a number of strategies most likely to boost their motivation to pursue a STEM field. While the methods employed to identify promising practices varied among these studies, the themes reflected in their summaries are similar and consistent. BEST identified the following as shared features of programs with positive outcomes: ●

Challenging content: “Content is related to real-world applications; students understand the link between the content they study and career opportunities later in life.” Programming encourages critical thinking and problem solving and team work.

●

Defined outcomes and assessment: Assessment tools are designed to measure outcomes and provide both quantitative and qualitative information, a basis for research, and continuous program improvement.

●

Sustained commitment and community support: Continuity of funding, stakeholders are involved and provide support at multiple levels

●

Engaged adults: Adults who believe in the potential of students stimulate interest and create expectations. Educators play multiple roles and active family support is sought.

●

Personalization of approach: Student-centered teaching and learning methods are core approaches. Mentoring and peer interaction are important parts of the learning environment.

BEST has been careful to point out that, “These design principles are not intended as a causal explanation of what works, yet are derived from reasonable inferences” but “Nevertheless, they are a first approximation of what it takes to succeed over time.” (2004) Similarly, the study commissioned by the Girls Scouts, identified the following practices as promising in promoting student interest and engagement as a result of the study’s literature review (Liston, 2008): ● ● ● ●

Hands-on learning activities Projects having real life context and relevance Opportunities for collaborative work Increasing confidence in STEM fields

● Providing contact with role models and mentors ● Parental involvement ● Providing career information

When asked what they perceive to be the most important to cultivate their interest in STEM, young people surveyed by Project Tomorrow (2008) echoed these themes. So did program staff who responded to the survey conducted for the Girl Scout study (Liston, 2008) mentioned above.

8

Puget Sound Center for Teaching, Learning and Technology (Liston, C., Peterson, K., & Ragan, V).(2008) Evaluating Promising Practices in Informal Science, Technology, Engineering and Mathematics (STEM) Education for Girls. Girl Scouts of the USA. Retrieved from http://www.girlscouts.org/research/resources/evaluating_promising_practices_in_informal_stem_education_for_girls.pdf.

9

Project Tomorrow and PASCO Scientific (2008) Inspiring the Next Generation of Innovators: Students, Parents and Educators Speak Up about Science Education. National Findings on Science Education from Speak Up 2007 Author Kim Farris-Berg. Sponsor PASCO Scientific. Data from Project Tomorrow’s Speak Up Initiative.


5

Introduction


Students and program staff indicated that the top strategies most likely to boost motivation to pursue a STEM field would be: ● Hands-on learning opportunities ● Conversational interactions with professionals ● Visits to STEM companies

● College scholarships ● Using tools professionals use

According to the Project Tomorrow survey, the “College scholarships” and “Using tools professionals use” strategies listed above were more important for students who had already indicated having a high level of interest in STEM. Question 4: How have successful STEM programs been profiled by others? As part of the literature review for this report, profiles of programs using best practices were reviewed and considered as possible models for this report. The BEST Commission and the Bayer Foundation have each published profiles of programs employing promising practices in STEM programming for youth. BEST’s profiles included the following components in each program profile: (1) general program description; (2) purpose/objective; (3) evaluation summary (including methods, targeted population and reach, subject matter); (4) evidence base (including outcomes examined, results and implications for future research). (BEST, 2004) The Bayer Foundation published, “A Compendium of Best Practice K-12 STEM Education Programs” across the country in 2006. Bayer’s profiles were very similar to BEST’s, but also included a listing of community partners, a 10 description of the learning environment, and a final section on opportunities for replication.”

Purpose of the Report: The purpose of this report is to profile STEM Pipeline funded programs that have employed promising practices to increase student interest in STEM (Science, Technology, Engineering and Mathematics). More specifically the intent of this report is to:

● Describe how the programs were designed ● Document what the programs accomplished ● Summarize program evaluation methods used

Development of a Logic Model: This report is focused on the Pipeline Fund’s first strategic goal to “increase the number of Massachusetts students who participate in programs that support careers in fields related to mathematics, science, technology, and engineering” with the ultimate goal of increasing the number of students who eventually choose to enter a STEM career. The preceding literature review reinforces the Pipeline’s theory of change. This theory of change is described in detail below and is also visually represented in a logic model format. Inputs: The Pipeline Fund served as the major “Input” in this logic model. While the Pipeline Fund clearly provided financial support for professional staff to coordinate programming, the intent was also to leverage additional resources from stakeholders. One intent of the Network funding in particular was to serve as an impetus to encourage the coordination of other available resources to support STEM programming. A review of the program profiles in this report shows that most programs did in fact leverage additional resources. (See the Stakeholder section of each profile.) Such additional resources went beyond funding to include facilities, equipment and space as well as the knowledge, expertise and experience that STEM professionals shared with students. These are listed as additional inputs in the model. Activities: Pipeline funding and the additional resources provided by various stakeholders enabled the implementation of the STEM programming described in these profiles. These programs utilized many of the practices that research literature indicated are effective in increasing student interest in STEM. Features of the programs 10

Bayer Foundation (2010) Planting the Seeds for a Diverse U.S. STEM Pipeline: A Compendium of Best Practice K-12 STEM Education Programs” as retrieved from http://www.bayerus.com/msms/web_docs/Compendium.pdf. UMass Donahue Institute Research and Evaluation Group

6

Introduction


included opportunities to: (1) interact with STEM professionals and learn about STEM careers; (2) apply STEM skills in real-life, hands-on situations; (3) employ critical thinking and problem-solving skills; and (4) collaborate within small group settings. Participants: While programming primarily focused directly on students as the main targets of this initiative, programming also targeted other individuals that could influence students’ interests and decisions. Research indicates that parents, teachers and other adults, including STEM professionals, can exert a strong influence on students’ interest in STEM. (BEST, 2004; Liston, 2008) As a result, several of these initiatives were designed to involve parents and teachers as participants alongside the students. STEM professionals were encouraged, and in some cases trained, to serve as role models and mentors for the students. Short-term Outputs: The intent of the programs was to stimulate students’ curiosity while engaging them in STEM activities that give them an opportunity to have fun and exciting experiences. It was also hoped that the experiences would encourage their belief in the importance and relevance of STEM subjects to their lives, as well as to increase their sense of self-efficacy and reduce anxiety related to their own perceived ability to apply STEM skills. Essentially, the programming was intended to influence the components of the domains described as “Interest/ Engagement” and “Attitude/Behavior” as described earlier by the National Science Foundation (Friedman, 2008) and the PEAR study (2008). The Pipeline Fund required the use of local evaluators to design survey questions and instruments to assess changes in these areas. Long-term Outcomes: As a whole, it was hoped that overall the initiatives would increase student interest in STEM subjects and careers. This logic model, along with the BHEF model, recognizes that there is an interaction and potentially mutually reinforcing dynamic between interest and proficiency in STEM fields. Increased interest can lead one to seek more knowledge and skill, while increased knowledge/skill may also stimulate further interest. Figure 1: Logic Model Pipeline Fund Inputs

Short-term Outputs

Network Support

Activities

Increase Curiosity in STEM

Interact with STEM Professionals / Learn about STEM Careers

Participants

Project Support

Apply STEM Skills

Teachers

Additional Resource Inputs

Employ Critical Thinking Skills

Funding

Collaborate in Small Groups

Students Parents

STEM Professionals Other Adults

Facilities Equipment STEM Professionals

Increase Belief in the Importance / Relevance of STEM Subjects Increase Sense of SelfEfficacy in STEM Increase Other Components of STEM Interest / Engagement and Attitude / Behavior

Long-term Outcomes Increase Interest in STEM Increase SEM Knowledge / Skills Develop Positive Feedback Loop between Interest and Knowledge / Skills Increase Number of Individuals Interested in, and Prepared for, Entering STEM Careers

According to this model and the BHEF model of the STEM education pipeline (2010), increased interest in STEM subjects and fields, along with proficiency in STEM, should lead to an increasing number of students engaging in STEM majors in higher education, who will eventually seek employment in STEM fields. The has been the overall strategic intent of the Pipeline Fund. UMass Donahue Institute Research and Evaluation Group

7


Criteria and Methodology

Criteria and Methodology Criteria for Inclusion in this Report: Themes that emerged from the literature review on promising practices and student interest formed the basis for the development of the criteria for inclusion in this report. The literature review indicated that programs can be determined to have promising practices through assessments in two complementary areas: Program Design and Program Outcomes. While BEST used an outcomes-based approach to work backwards from programs with good outcomes to promising practices, others, such as the Bayer Foundation, used identified promising practices (along with outcome data) as criteria for report inclusion. Still others, such as Project Tomorrow, surveyed program administrators, teachers and students regarding which practices respondents have found to be most effective. Both program design and program outcome criteria were used to select the programs profiled in this report. I.

Program Design Criteria: A. Designed to increase student interest in STEM B. Employed practices that research indicates are effective in affecting STEM interest, such as: ○ Content was related to real-world application, ideally through hands-on learning ○ Critical thinking, collaboration, and small group work was encouraged ○ Caring adults (i.e., parents, teachers or STEM professionals) participated ○ Information about STEM career opportunities was provided

II. Program Outcome Criteria: A. B. C. D.

Provided data on student interest in STEM Assessed changes in STEM interest in relation to program participation Assessed interest in STEM more generally, beyond the program itself Collected student interest data directly from students

Additional Explanation of Program Outcome Criteria: Provided data on student interest in STEM: As the first two domains described in the PEAR study align most closely with STEM interest, these domains, and their relevant components, formed the basis for determining what outcomes could be considered student interest data for the purposes of this report. As a result, programs needed to utilize instrument items or survey questions to assess STEM interest as described in those components (see earlier Literature Review). Programs that only collected data on STEM proficiency (such as content knowledge, competence or reasoning) were not included. However, if a program did collect data on STEM proficiency in addition student interest data, then the information is summarized in the program’s profile. The standard for the amount of data collected was “any” data addressing the components of student interest as described above. Programs funded through the second, 2008, RFP with the particular focus on student interest, had only received Pipeline funding for a little over a year at the time of this review. As a result, some had not yet had the opportunity to collect much data on student interest. Evidence may have been as limited as one post-survey where most students self-reported whether they felt the program had affected their interest in STEM. The data collected could be qualitative or quantitative in nature. Programs did not have to show evidence of increasing student interest to be included in this report. Inclusion does not necessarily imply that the program was effective – only that it presented some data on student interest and met other report criteria. The converse is also true. Not being included does not mean a program was ineffective – only that it did not meet the above criteria. It is quite possible that a very effective program may not have specifically collected data on student interest, or that the program had been in existence for too short a time and data was not yet available. UMass Donahue Institute Research and Evaluation Group

8



Programs funded by either Pipeline Fund RFP may have focused on teacher professional development or student content knowledge, and as a result, they may not have collected data on student interest. Assessed STEM interest in relation to program participation: Survey instruments or questions had to be designed or collected in a manner that assessed changes in student interest as a possible result of participation in the program. For example, a survey question that asked whether students plan to enroll in STEM majors in college would only be acceptable if it was asked both before and after a program. Asking this as a pre-post question would enable evaluators to assess whether STEM interest changed following the program. However, when asked as a post-only question, the information gathered would not meet the inclusion criteria. While answers to this question asked after a program would provide an assessment of student interest at that moment in time, it would not provide any information about whether students’ interest may have been influenced by participating in the program. Post-only questions were only acceptable if they asked students to reflect on the program’s effect on their interest. For example, data collected in response to a post-only question asking, “Did participation in this program strengthen your interest in majoring in STEM subjects in college?” would be acceptable. Assessed interest in STEM more generally, beyond the program itself: Survey items needed to gather information about student interest in STEM beyond interest in the program activity alone. Questions that asked whether students enjoyed the program activity (i.e., would you like to participate in another “science club” or “summer camp”) were not acceptable. While such questions are useful to assess student satisfaction with participation, they are not sufficient as assessments of interest in STEM subjects more generally. The question needed to imply that the interest gained from participation in this activity would be transferrable to other STEM related activities. For example, a question that asked about interest in a different STEM-related activity, or an item that assessed interest more generally, such as one asking students to indicate how strongly they agree or disagree with statements such as “I like math “ or “I like science more after this program” would be acceptable. Collected student interest data directly from students: Information collected from observers of students, such as teachers, parents or external evaluators, was not considered sufficient. Reports of student engagement in activities by instructors or levels of interest in STEM reported by parents can be valuable. However, evidence of interest collected from third parties cannot be considered as strong as information gathered directly from students. So, for the purposes of this report, only those programs that presented data collected directly from students were included. However, if data was collected from observers as well, then summaries of data from both sources are presented in the profiles.

Methodology: All Pipeline programs funded during 2009-2010 were reviewed to determine if they met the program design and outcome data criteria for inclusion in this report. All report materials were reviewed, including materials submitted as part of the DHE’s statewide evaluation and reporting requirements as well as assessment instruments, data, and reports completed by local evaluators. All information submitted since the inception of Pipeline funding through June of 2010 was reviewed. In some cases, programs were contacted after June and asked to provide updated data on program outcomes if an extension had been granted or data had not yet been available at that time. Each program was reviewed individually to determine whether it met report criteria. Frequently a single lead organization coordinated multiple programs with their STEM Pipeline funding. For example, Northern Essex Community College implemented three programs: Engineering is Elementary, Viva Computers for Latinas, and GEMS (Girls Exploring Math and Science). Each program had different approaches and objectives and, as a result, the local evaluator designed a unique evaluation process, and collected separate data, for each program. Each of these programs was reviewed individually to determine whether it met the report criteria. The Engineering is Elementary program was chosen to be profiled in this report as it met all of the above criteria.

Organization of the Report: Profile fields were modeled after those presented by the Bayer Foundation and BEST Commission. Each program profile is approximately two to three pages long, and includes the following:


9



            

Program Name Lead Organization and Region of the State Served Pipeline Funding Source (Network or Student Interest) and Duration of Funding Program Overview (Including the Program’s Purpose and Design) Target Population and Reach Frequency (One Time or Series) Length of Program In-school or Out-of-school Stakeholder Involvement Evaluation Method Evaluation Results and Interest in STEM Additional Evaluation Results Commentary and Implications for Future Application

Detailed descriptions of each of these fields can be found in Appendix A.

What is Measureable Change: Data collection results are summarized in each profile, with a specific statement on whether any changes observed were measureable or not. For the purposes of this report, “measureable” means that two conditions were met. First, that an assessment (such as a survey or test) was administered before and after the program. Second, that there was an observable difference when comparing results from before and after the program. Measureable does not imply that results were statistically significant or solely caused by program, only that the data collected before and after the program indicated a change in student interest. Limited time and funding for local evaluation efforts often did not allow for rigorous methodology such as randomized sampling or a control group for comparison. Please note that for the purposes of this report, the term “pre-post” will be used to refer to measures administered before and after a program is implemented. When pre-post measures were taken, but no observable difference was evident when comparing the results from before and after the program, results will be described as “not measureable.” In cases where the local evaluation showed no clear difference between pre-post results, the evaluator usually gave a written interpretation of the outcome. This report includes these interpretations. In either case, if clear data was available, the report summarizes the results and provides a description of any change observed in student interest. The term “post-only” means that no measure was taken before the program and refers to assessments conducted, or data collected, only after a program concluded. By definition, post-only results cannot document measureable change, as there are no results from before the program against which to compare. While post-only surveys cannot document measureable change, they can still be useful by asking participants to reflect on and assess how they feel their attitudes or plans may have changed as a result of participation in the program. While programs were not required to demonstrate a change in student interest to be included in this report, program profiles are organized and presented in this report according to the type of outcome data each collected, and whether the data indicated a measureable change. Program profiles are presented in the following three categories. Within each group, profiles are presented alphabetically by program name. 1. Measureable Change Assessed:

Programs conducted pre-post assessments and results suggested measureable change.

2. No Measureable Change Assessed:

Programs conducted pre-post assessments, but results did not indicate measureable change.

3. Qualitative Results Only:

Programs did not use pre-post assessments, but used post-only or qualitative assessments.

Please note that copies of survey instruments or sample questions from each profiled program have been included in Appendix B.


10

Program Profiles


Program Profiles Programs that Conducted Pre-Post Assessments: Measureable Results Biotech Career Pathways Lead Organization/Region: Framingham State University / MetroWest Region Funding/Duration: Network Funding – 2008 to 2010 Program Overview: The Biotech Career Pathways project offered two types of opportunities for students at Norfolk County Agricultural High School (NCAHS) to gain exposure to careers in the biotech industry and other STEM areas. Juniors could participate in a six-week long internship while seniors could participate in five days of intensive workshops. Students completed applications and a series of permission forms to participate in the program. The high school worked in partnership with Massachusetts General Hospital (MGH), Mass Bay Community College (MBCC), and Cummings Veterinary School at Tufts University to coordinate the internships and workshops. Juniors took a two-credit college course in Laboratory Biotechnology Skills at MBCC to prepare them for the internship. The internship began with a three-day orientation and training in bio-security and biohazard control. During the summer of 2009, students rotated through a series of five stations at a MGH research facility. Each station was located in a different area and designed to expose students to different research methods, project designs and biotechnical skills. Internship activities included observing surgeries, using advanced radiography, performing necropsies, and harvesting tissue. According to program reports, these activities were intended to give students the opportunity to see STEM skills in use during daily functions at the facility. Students also visited other industry facilities, including the New England Primate Center, Charles River Labs, and New England Ovine (NEO). Seniors participated in five days of intensive workshops at the Cummings Veterinary School at Tufts University. Subjects included agro-terrorism, hematology, laboratory biosafety, zoonosis and public health. Both juniors and seniors were accompanied by school instructors who participated in the events and were trained on technical subjects along with the students. An additional objective of this project was to update the animal science program at Norfolk County Agricultural High School to include laboratory animal research, biomedical research, and an introduction to contemporary scientific methods and research. Initial curriculum development projects resulted in two courses being offered: Research Animal Technology I and II. The most recent program report stated that due to a high level of student interest and participation, the number of classes offered has increased from two to four sections and a new major of Research Animal Technology has been added to the Animal and Marine Science program. Target Population/Reach: In the initial Pipeline application, the program targeted middle school students. However, th th given the complexity of the subject matter and the partners, the project concept evolved to target 11 and 12 grade students. During the summer of 2009, twenty-one students (11 juniors and 10 seniors) and their teachers participated. One Time/Series: Series Program Length: Six week internship for juniors and five day seminar for seniors. In/Out of School: The internship and seminars are offered out of school during summer vacation.


11


Program Profiles

Stakeholder Involvement: As described above, many local partners were involved in creating and sustaining this project, including the Norfolk County Agricultural High School (NCAHS), the Cummings School of Veterinary Medicine at Tufts University (Tufts), Massachusetts General Hospital (MGH) and Mass Bay Community College (MBCC). Evaluation Method: Students completed surveys designed to assess their interest in, and awareness of, STEM subjects before and after they participated in this program. Students were asked to rate their level of awareness about the job market and employment opportunities in research, biotechnology and animal management. They were also asked about their level of interest in, and motivation to pursue, careers in these areas. Recent program reports described additional evaluation efforts planned for the end of the summer 2010 internships. Program coordinators intended to ask students to critique their experience and provide input and suggestions for improving future programs. Focus group discussions were planned to gather information from staff and students to review the program, and devise ways to improve the internship for the following year. They planned to ask instructors to complete a Massachusetts Work Based Learning Plan Assessment for each participant at the beginning and end of the internship to assess their skill levels. They also intended to ask teachers to evaluate students on the CVTE (Career Vocational and Technical Education) state competencies relevant to the internships and workshops. Information from these additional evaluation efforts was not yet available at the time this report was written. Evaluation Results/Interest in STEM: On average, juniors and seniors rated their awareness about job market and employment opportunities in research, biotechnology and animal management as higher after participating when compared to pre-program results. The average level of students’ self-reported level interest in, and motivation to pursue, careers in these areas also rose after participation. Additional Evaluation Results: Additional evaluation information concerning results of the teacher focus group and results of the second round of internships during 2010 was not yet available. Commentary/Implications for Future Application: The Biotech Career Pathways program employed nearly all of the promising practices for increasing student interest in STEM that were described in the introduction to this report. This experience offered students the opportunity to do hands-on learning with the tools professionals use at a STEM facility with STEM professionals. Compared to many other STEM programs across the state, this program offered a particularly in-depth experience to students. Providing this intensive experience for students required the coordination and use of many resources. For example, the project reported that collecting and compiling the paperwork and documentation required for student participation alone was a significant challenge. The paperwork needed for clearance into a highly restricted animal research facility, and the releases necessary to allow students under the age of 18 to be involved in the program, resulted in a 44 page “parental permission document” that was compiled collaboratively by the host site and the high school. Because of the demanding permission process, the most recent evaluation report recommends that “…students and their parents should be given a minimum of two months to obtain all the required information, with incremental deadlines during this time to keep the flow of paperwork moving.” Project staff also cautioned that processing the paperwork took significant staff time. This level of coordination, staff time and resources may not be possible for all programs. The project also recommended that a large student body be introduced to the internship experience to encourage a greater number of applicants. To accomplish this, the school provided students with an opportunity to attend career day sessions with biotech speakers and to take a tour of the internship facilities, all with the goal of stimulating student interest in the program. According to a recent report, “The stereotypical impressions of the biotech industry held by students prior to their exposure to this project were dispelled by their formal educational introduction to the program.” Project reports also stated that additional challenges were posed by current cooperative education guidelines for the Massachusetts Department of Elementary and Secondary Education (MADESE) Department of Career and Technical Education. As payment was made to the MBCC for course credit, the course fulfilled the academic requirement for students in the program, allowing them to be released for full days. The project asserted that without funding to pay for this, it would have been difficult to meet current state requirements for cooperative education. Despite the challenges involved, and the resource-intensive nature of this program, it clearly utilized many promising practices and provided an intensive experience for students. Initial results suggested it may be effective in stimulating student interest in STEM.


12

Program Profiles


DIGITS Lead Organization/Region: Massachusetts Technology Leadership Council / Statewide Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: DIGITS was designed to raise sixth graders’ interest in, and understanding of, STEM careers. A DIGITS program generally involved a structured visit to a classroom by a professional employed in a STEM field. These “STEM Ambassadors” volunteered to lead a discussion and share some of their work experiences. STEM Ambassadors were encouraged to describe their journey to their current STEM position, including what they have found to be interesting, exciting and fun and about their jobs. They were also encouraged to talk about the positive aspects of their careers, including opportunities for travel and higher income. STEM Ambassadors were intended to serve as positive role models to counteract negative stereotypes about people employed in STEM fields. STEM Ambassadors participated in an hour-long training session (conducted by phone before or after regular business hours) and were provided with a “STEM Ambassador Guide” which provided a script and instructions for leading discussions and activities during classroom visits. STEM Ambassadors visited anywhere from 2- 8 individual classes at their assigned schools. Classroom visits usually included two hands-on activities designed to generate discussion between students and their STEM Ambassador about the rewards and benefits of working in STEM fields. For one activity, students used stickers of letters embedded with STEM–related icons to spell their names. Students were then asked to consider which icon in their name represented a career they might consider. Program designers have indicated they felt this was critical as it may have been the first time some of the students had ever envisioned themselves in a STEM career. The other activity was a card sorting exercise where students sorted the letters/icons into one of four STEM sectors – information technology, life sciences, engineering and energy. STEM Ambassadors were encouraged to bring examples of their work or equipment to the classroom for students to see. This provided some classes with additional opportunities for hands-on STEM experiences. Research on how young students learn, how they perceive their own abilities in STEM subjects, and what they value when thinking about a career was used to develop the program design, name, logo, themes, and presentations. The program also conducted its own research among students and teachers from Massachusetts. Focus groups and other methods were used to gather information to develop the DIGITS concept, presentations, and activities. DIGITS was initially tested as a pilot program with 400 students. The program was then offered, on an opt-in basis, to several hundred schools across the state. Upon request of teachers/principals, the DIGITS program was introduced to four schools with larger populations through a large-group assembly. The assemblies included a brief introductory presentation, a presentation from a STEM Ambassador who started out as a materials engineer at NASA’s Kennedy Space Center and now works for a Massachusetts-based video game company and, in 3 of 4 cases, a live performance from Tezz Yancey, who is the recording artist of the original DIGITS music video. STEM Ambassadors, who were assigned to these schools, were introduced at the assemblies if they were able to attend. Congressman John Tierney participated in one assembly in his district. The Ambassadors were encouraged to schedule individual classroom visits within two weeks of the assembly. Additional related elements included a poster, website, a preview video (which demonstrates how DIGITS is deployed in the classroom) and music video. th Target Population and Reach: DIGITS was designed to reach 6 grade students. The program report submitted in June 2010 indicated the program had reached over 10,000 students at 118 schools (over 20% of the schools in MA that have a sixth grade). DIGITS programs were provided in central, northeastern and southeastern MA (including the Cape and Martha’s Vineyard) and in the Greater Boston suburbs. Programs had not yet been conducted in Boston or western MA. However, the report expressed the intent to expand the geographic reach of DIGITS. The average number of students reached per program varied according to classroom size.

One Time/Series: One time Program Length: Presentations were typically 40-50 minutes long, the length of an average class session. UMass Donahue Institute Research and Evaluation Group

13


Program Profiles

In/Out of School: In school during a class Stakeholder Involvement: The project was the result of the collaboration of six leading science and technology industry trade associations known as the STEMTech Alliance. Members included the Mass Technology Leadership Council, the Massachusetts Biotechnology Education Foundation, The Massachusetts Medical Device Industry Council, The Massachusetts Networks Communications Council, The Engineering Center collaborative, and the New England Clean Energy Council. The STEMTech Alliance represents “over 1,500 companies and 300,000 people who work in software, Internet, telecommunications, biotechnology, medical devices, engineering and clean energy” according to the DIGITS website, which also provides information on each organization. According to the June 2010, report, over 150 volunteers from multiple industry sectors visited classrooms throughout Massachusetts (software/robotics 36%, telecommunications/wireless 23%, engineering 18%, biotechnology 10%, clean energy 5%, medical devices 3% and other 5%). The report stated that volunteers came from diverse backgrounds (41% women; 23% minority) and represented 44 different companies from the six participating industry sectors. Evaluation Method: Data evaluating the effects of the DIGITS rollout to over 10,000 students during the 2009-2010 school year were not yet available when this report was written. However, formative and summative results from the 11 pilot program (conducted from April through July of 2009) were available. The evaluation involved multiple methods. Students, teachers and STEM Ambassadors were surveyed regarding their experiences. Surveys of teachers and students involved both baseline and follow-up assessments. Other evaluation methods included formal observation of classroom presentations, as well as interviews with teachers, ambassadors, and program staff. Pre- and post-surveys of students were designed to assess student perceptions of the STEM subject areas of mathematics, science and technology. Students were asked to respond to separate statements on how strongly they agreed or disagreed that each subject was “fun” or “interesting.” (Students were given a four-point scale, with options of strongly disagree, disagree, agree or strongly agree.) Survey questions also asked about their intentions to take Algebra or participate in STEM related activities, and their perceptions regarding STEM careers. Students were also asked to list jobs that involve the subject areas of mathematics, technology and science. Evaluation Results/Interest in STEM: Seventy-two percent of the students who participated in the pilot program (289 of 400) completed both the pre- and post-visit surveys. The survey evidence indicated that there was an increase in student interest in, and positive attitudes toward, STEM subjects, activities and careers following the DIGITS presentation. Students were more likely to report that mathematics, science, and technology is “fun” and “interesting” on the post-survey. The evaluation of the pilot project also indicated that students “were slightly more likely to report they would take algebra in the future” and they were nearly 50% more likely to agree that people who take algebra I or II are more likely to get higher paying jobs. The percentage of students who indicated they would be interested in participating in a science fair or a math or computer club also increased after the classroom presentations. The evaluator’s report stated that, “Qualitative analysis of open-ended questions reveals that students’ awareness of careers that involve mathematics, science, and technology expanded after the STEM Ambassador visits.” When asked to list jobs that involved mathematics, science and technology, the types of jobs that students were able to list after the DIGITS presentation generally reflected a wider array of choices and a greater awareness of more specific types of jobs in those fields. For example, many students gave more general examples prior to the visit such as “scientist” or “teacher” prior to the presentation, and then gave more specific examples of jobs such as “meteorologist,” “pediatrician” or “computer programmer” after the visit. Additional Evaluation Results: The local evaluator found that teachers and ambassadors who responded to the surveys generally had positive responses. Of the nine teachers who responded to the evaluation survey for the pilot program, when given a four- point scale with options of “very low quality,” “low quality,” “high quality” or “very high quality” three said it was “very high quality,” and six said it was “high quality.” The evaluator’s report stated that, “While most teachers agreed or strongly agreed that the presentation would help students understand the importance of mathematics, comments indicated that some Ambassadors did not explicitly make the connection between their jobs and middle school mathematics; 1 respondent disagreed with this statement and 1 respondent gave a neutral response.” Similarly, nine out of 10 STEM Ambassadors who responded to the pilot program survey about the training 11

STEM Tech Alliance Pilot Evaluation Report, by Written by Diane Schilder, Principal Evaluator, August, 2009, Evaluation Analysis Solutions, Inc. Cambridge, Ma One Mifflin Place, Suite 400, 02138. UMass Donahue Institute Research and Evaluation Group

14


Program Profiles

and materials they received prior to visiting the classrooms indicated that they perceived the DIGITS materials to be of high or very high quality. All ambassadors indicated that they felt prepared for their classroom visits. Students also reacted positively to the programs. The evaluator’s report stated that “(O)ver 70 percent of students who completed both the pre and post surveys gave the visits the highest rating” with an average score of 3.65 out of 4 points. A copy of the original evaluation report is available on the program website, www.digits.us.com. Commentary/Implications for Future Application: While a DIGITS presentation was designed to be implemented during a single classroom session (typically 50 minutes), DIGITS managed to utilize several of the promising practices described in existing research. Students were provided with information on the real-life application of the STEM skills they were learning in school, particularly as they are relevant to future careers. They were also given an opportunity to meet and interact with a STEM professional. While the sticker and sorting exercise described above did not give students the kind of experience that is usually described as a “hands-on” activity for STEM programs (such as the opportunity to practice applying science, math or related skills or knowledge to a situation or problem) given the limitations of a one-time classroom visit, the use of the sticker activity and sorting exercise appear to have been effective mechanisms to generate interactive discussion. Since STEM ambassadors were encouraged to bring examples of their work, it is likely that many students were given additional opportunities for hands-on experiences that were not accounted for in the program’s evaluation. One advantage of a one-time, shorter program like DIGITS is the opportunity to offer it to greater numbers of students. DIGITS has reached more than 10,000 students across the state, which is more than any other Pipeline funded program. As initial results from pre- and post-surveys of the pilot program indicated a change in attitude toward STEM, this program may be an effective model for reaching large numbers of students with a positive STEM message. The STEMTech Alliance went on to implement the program throughout much of the state, as described above. It is likely that evaluative information gathered from wider program implementation, once available, will add value to the current body of evidence on the effectiveness of this program. Should funding for a more extended evaluation of the DIGITS program be available, it would be interesting to do a follow-up survey with a sample of some students six months to a year after their STEM Ambassador’s visit to see if any of the short-term changes in beliefs, attitudes, knowledge, etc. are retained over time. It may also be useful for future evaluations of similar efforts to ask not only about attitudes, but to also ask about behavior relating to participating in STEM activities or classes. For example, the original survey asked students about their interest in participating in STEM-related activities such as math or science clubs. Follow-up surveys that assess whether students actually joined such clubs might also provide some insight. Following the pilot presentations, some of the teachers indicated that they would be interested in additional presentations. Again, resources may not allow this, but it would be interesting to see if teachers who participated in the larger roll-out of this program over the past year have similar interests in follow-up programming, which could include additional visits from STEM Ambassadors or other programming.


15

Program Profiles


Engineering is Elementary Lead Organization/Region: Northern Essex Community College / Northeast Region Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: Engineering is Elementary is available from the Boston Museum of Science (MOS). According to their website, “The Engineering is Elementary® (EiE) project fosters engineering and technological literacy among children. EiE is creating a research-based, standards-driven, and classroom-tested curriculum that integrates engineering and technology concepts and skills with elementary science topics. EiE lessons not only promote K-12 science, technology, engineering, and mathematics (STEM) learning, but also connect with literacy and social 12 studies.” EiE was implemented by Northern Essex Community College (NECC) and its partners to encourage student interest and proficiency in STEM in northeast Massachusetts. The program was introduced to four different public school districts. During the summer of 2008, 22 teachers attended a three-day workshop to learn about the curriculum. These “lead teachers” were then expected to both implement their EiE units and teach other teachers how to implement these lessons. During the 2008-2009 and 2009-2010 school years and during the fall of 2010 (after the grant funding ended), the lead teachers trained 80 additional teachers, resulting in over 100 teachers being trained in the curriculum. Teachers’ guides and other curriculum materials for use with students were purchased and distributed to the trained teachers. Target Population/Reach: This program was to reach third through fifth grade students by training teachers in the EiE curriculum. More than 120 students reportedly received the curriculum during the initial year of the project (from the 22 lead teachers trained in 2008). While the number of students reached in the 2009-2010 school year was not yet available when this report was written, initial data indicated more than 500 students likely received programming. One Time/Series: Series Program Length: The EiE curriculum has 15 different units, each designed for a standard class period. In/Out of School: In school Stakeholder Involvement: Administrators and teachers from the four school districts were partners in this program. The Northeast Network, a consortium of schools, colleges and STEM businesses provided support as well. Evaluation Method: A number of methods were used to evaluate the effect of the program, including an on-line preand post-survey of students’ attitudes toward engineering subjects, activities and careers. The survey contained statements where students were asked whether they, “strongly agree,” “agree,” “disagree,” or “strongly disagree.” Three statements directly concerned student interest as defined in this report. Two addressed students’ perceptions of math and science self-efficacy: “I am good at solving problems in mathematics” and “I am good at solving problems in science.” The third asked “When I grow up I want to be an engineer.” In the first year, the student attitude survey was administered only once, usually after completion of the EiE units, and was therefore unable to capture change in interest. In the second year, the program distributed the survey earlier, with instructions encouraging teachers to administer it both before and after implementation. Unfortunately, only a small number of students completed both pre- and post-surveys. The program attributed this partially to teachers who were aware that year three funding was cut and therefore lacked any real incentive to capture program data. Following their training, teachers completed an on-line survey assessing their attitudes toward, and confidence in, teaching STEM subjects, with a focus on engineering. This pre/post survey also assessed content knowledge and asked their perspectives on the value of the professional development activities.

12

As retrieved from the Engineering is Elementary page at the Boston Museum of Science webpage http://www.mos.org/eie/index.php, September, 2010. UMass Donahue Institute Research and Evaluation Group

16


Program Profiles

Evaluation Results/Interest in STEM: While full analysis of the student attitude surveys was pending at the time this report was written, four classes had completed both pre- and post-surveys. Raw data indicated there was a positive shift in students’ sense of self-efficacy in math and science. All four classes had an increase in the number and percentage of students that agreed with the statement “I am good at solving problems in mathematics.” There was also a shift from “agree” to “strongly agree” among responses to this statement. Students’ responses to the statement “I am good at solving problems in science” also indicated improvements in students’ sense of self-efficacy. While students as a whole did not initially perceive their abilities in science to be as high as they did in math, the largest shift from pre- to post-program results was the increase in the number of students who agreed with this statement. Many students shifted from “agree” to “strongly agree” when asked about their science abilities. There was also a decrease in those who disagreed with the statement. Students’ responses to “When I grow up I want to be an engineer” demonstrated more positive (or less negative) views of becoming an engineer after the program: 25% of students initially agreed with this statement while 35% agreed with it on the post survey. Similarly, the percentage of students who disagreed decreased from 75% before the program to 65% after. Within those that disagreed, there was a shift away from “strongly” disagree. While 38% strongly disagreed on the pre-survey, only 21% strongly disagreed on the post-survey. For those students that took the survey only once, results indicated that a majority (over 80%) agreed that they were good at solving problems and math and science. Over 90% of students indicated they liked school and that “Being a student at my school is important to me.” A majority (over 60%) of students indicated that when they grow up they want to “solve problems that help people” and “design different things.” Additional Evaluation Results: Results of the pre- and post-tests assessing student gains in content knowledge had not been fully analyzed as of the writing of this report. However, it is worth noting that some teachers expressed concerns that they felt some test questions for one unit did not accurately cover its content. They also felt the scoring method for some questions was not effective, and that the overall scores were skewed as a result. As a result, they felt the instrument was not sufficiently effective in reflecting the true level of knowledge gained by students. Anecdotal responses from teachers indicated they found the implementation of the units to be useful, with one teacher commenting, “Overall, the students and I very much enjoyed this unit and I really felt that they learned a lot!” According to the local evaluator, teachers that attended the EiE trainings indicated on post-program surveys that they had a “high degree of interest in improving their skills in teaching engineering design.” Teachers generally expressed a strong interest in learning more about teaching engineering, and strongly agreed with statements that “pre-service education (in engineering and technology) is essential for the preparation of tomorrow’s teachers.” Teachers identified lack of teacher knowledge and lack of teacher training as two strong barriers to integrating engineering in their teaching. They indicated only a moderate level of preparation in this area during their own pre-service training. Commentary/Implications for Future Application: EiE utilized many promising practices, most notably the opportunity to do hands-on learning and problem solving and having engaged adults involved. Evaluation results indicated that participating in this training may have improved teacher confidence and abilities in teaching engineering overall. Such capacity building may be difficult to measure, but it can be quite valuable over time. EiE also used the promising practice of providing information about engineering as a career to students. While the lack of year-three funding may have affected the ability of this program to collect extensive assessment results, the available information indicated a positive change in students’ sense of math and science self-efficacy, and willingness to consider an engineering career. These results are reinforced by additional research conducted on EiE generally. As this curriculum has been offered by the MOS since 2004, more resources have been put into evaluating its effects than most other Pipeline-funded programs. Effects have been reported as positive in several research articles. A study published in June, 2010, stated that “Results indicate that students who completed the EiE curriculum were significantly more likely to report interest in being an engineer on the postsurvey than control students. They were also significantly more likely than control students to report interest in and comfort with engineering jobs and 13 skills, and to agree that scientists and engineers help to make people’s lives better.” This study and other information on the curriculum can be found on the MOS’s website and at: http://www.mos.org/eie/index.php.Given these positive results, it may be worth exploring whether this program could be offered to more elementary students across the state, especially considering that most Pipeline programs have targeted older students 13

The Impact of Engineering is Elementary (EiE) on Students’ Attitudes Toward Engineering and Science. Dr. Christine M. Cunningham, Dr. Cathy Lachapelle, Engineering is Elementary, The National Center for Technological Literacy, Museum of Science, Boston (June 2010)


17

Program Profiles


Got Math? Lead Organization/Region: Massachusetts College of Liberal Arts / Berkshire Region Funding/Duration: Network Funding – 2007 to 2010 Program Overview: The primary goal of Got Math? was to improve elementary and middle school students’ skills in, and attitudes toward, mathematics. A secondary goal was to support instructors’ use of applied mathematics during regular daytime classroom sessions. This after-school program provided a series of math instructional activities then connected students with local businesses where they had an opportunity to apply the math skills they learned. While the instructional activities and types of application varied greatly, the program’s signature characteristic was that students met with a local business or employer and had the opportunity to apply math skills in a real life setting. The Got Math? program was implemented at four schools in Berkshire County during the 2009-2010 school year. At one school, the program topic was sports and the business partner was a college. Students applied the math concepts they learned to track scores and determine standings at a basketball game. Another school’s program partnered with a local architecture firm. An employee from the firm worked with students over two sessions to apply measurement and problem solving skills to design and build a foam-model house to scale. A third program topic was grocery shopping and baking. A major chain grocery store invited the group to take a tour of its bakery and use coupons at the register. Students used math problem solving skills to estimate quantities, solve problems regarding proportional relationships, and convert measurement units. The fourth school’s program topic was fitness and the business partner was a physician who was also a tri-athlete and a supporter of children’s fitness. He presented to the class about how he used math to set fitness goals, including how he planned his training and determined his speed by tracking distances and time. Target Population/Reach: Elementary and middle school students in grades three through six participated in this program. During the 2009 - 2010 school year, 85 elementary students participated in programs at 4 different schools. Generally, 12 to 25 students participated in the Got Math? program at each school, however one school had 39 participants. One Time/Series: Series Program Length: 1-1 ½ hours, often twice a week for 6 weeks In/Out of School: After-school program Stakeholder Involvement: Several area businesses assisted with curriculum development and provided opportunities for students to apply math skills in real-life settings. A local higher education institution coordinated the programs with elementary and middle school math teachers. Evaluation Method: Attitudes toward math were assessed in pre-post surveys that were administered to all student participants. Students were asked whether they liked math and whether they felt they were good at math. Other questions asked whether students liked activities such as math puzzles or would participate in a math club. The postsurvey had two additional questions that asked how often students enjoyed the program and whether they felt participation had improved their math abilities. Content tests assessed students’ abilities to solve math problems and were tailored to assess the skills and knowledge gained from participation in each Got Math? program. Finally, interviews were conducted with instructors to determine whether and how participation with the Got Math? program had affected their regular classroom instruction. Evaluation Results/Interest in STEM: Nearly half (42 or 49%) of the 85 students that participated in the program completed the post-participation survey. The majority of students indicated they felt the program had improved their UMass Donahue Institute Research and Evaluation Group

18


Program Profiles

math skills, with some reporting more improvement than others. Sixteen (38%) reported that participation improved their math skills “a lot.” Twenty-three respondents (55%) said that participating in Got Math? improved their math skills “a little.” Nearly all students (95%) reported they usually or always enjoyed the Got Math? program. A notable difference between the pre- and post-survey results concerned students’ perceptions of their math abilities. According to the evaluator’s report, “(T)here was a significant improvement in the percentage of students who rated themselves good at math.” Before the program 50% of participants rated themselves as “very good or pretty good” at math and after the program 79% rated themselves in those categories, indicating an increase in perceived selfefficacy regarding their math skills. Little difference was apparent in student responses to questions before and after the program regarding math-related activities. From an analysis of these results, the program’s local evaluator stated that, “It is difficult to draw conclusions from this small sample regarding the effect of the Got Math? program on students’ attitudes toward mathematics” pointing out that most of the differences between the pre- and post-tests indicated a change in the attitude of only one or two students. The evaluator speculated whether the minimal change was due in part to the fact that students reported a high level of liking math and enjoying math activities prior to participating in the program. Additional Evaluation Results: Evaluation data did indicate some improvement in student math content knowledge. Average performance on the Got Math? knowledge tests improved from 39% correct on the pre-tests to 52% on the post-tests during the 2009-2010 school year. However, one of the school groups did have a slight decrease in achievement. Another goal of the program was to encourage instructors to use more applied mathematics in regular classrooms. According to the evaluators report, “All of the teachers reported a positive impact upon their classroom instruction. Changes in their approaches to teaching included being more aware of creating ‘themes’ rather than narrowly focusing on the standards in order to make the material more interesting and relevant to students and including more hands on activities” including using varied approaches to solving math problems. Commentary/Implications for Future Application: Got Math? employed several of the practices that research indicate are promising for increasing student interest in STEM, such as the opportunity to do hands-on learning, to apply a STEM skill (math) in a real-life setting, to work in small groups to solve problems, and the chance to interact with a STEM career professional. As students met repeatedly, Got Math? provided an ongoing experience with more exposure to these learning experiences than a one-time program could provide. Programs that target elementary students and surveys designed to assess student interest in STEM at the elementary level are fairly rare in the Pipeline program and in research on STEM interest nationwide. This, combined with the positive outcome data on student interest, may merit further exploration of this program for wider application with young students. As students self-selected to participate in Got Math?, conducting this program as a more generic after-school program, without a particular focus on math, might provide information about the program’s potential to affect students that are not pre-disposed to enjoy math.


19

Program Profiles


STEM Summer Camps Lead Organization/Region: CONNECT Partnership / Southeast Region Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: CONNECT Partnership offered two, five-day summer camp experiences to middle school students in 2008 and 2009. A goal of the camps was to “increase the number of Massachusetts students who participate in programs that support careers in fields related to science, technology, engineering and mathematics.” 14 The camps were designed to engage students in hands-on, content-based inquiry science that would develop students’ content knowledge, and provide them with a motivating and fun experience with science. Other goals were to develop student awareness of careers in STEM fields and strengthen students’ “21st Century Skills” such as collaboration, communication, and problem-solving. Two different camps were offered each summer. One was “Sustainability and the Environment” and the other was “Biotechnology and Forensics.” Each camp was led by a licensed teacher and an assistant. The Forensics camp gave students an opportunity to work as a Crime Scene Investigator and use scientific skills and approaches to solve a mock crime scene. Activities involved DNA sequencing, blood analysis, fingerprinting, and hair analysis in a lab at Durfee High School in Fall River. Students that participated in the Sustainability camp worked as environmental specialists and designed a model community using environmentally friendly technologies and approaches. Campers collected and analyzed soil samples and pond water. Students presented their models of a future sustainable town and green community center to over 25 parents, UMass Dartmouth staff, and community leaders. Target Population/Reach: The program intended to enroll 60 middle school students from high-need districts in each camp. Due in part to challenges with recruitment, this initial target – both in number and composition – wasn’t fully realized. Instead, approximately 40 students attended each camp with a substantial percentage of attendees at one camp coming from private schools. Half or more of participants at each camp were female. The student composition changed in 2009, when more than half of the students came from high-needs districts. One Time/Series: Series Program Length: Five days for each camp In/Out of School: Out of school during the summer Stakeholder Involvement: Program reports stated that the camps were made possible through the collaborative working relationships between K-12 educators and higher education partners. The Fall River Public Schools’ Office of Instruction/Professional Development organized the Forensics camp which was held at the Durfee High School in Fall River. The Sustainability camp was coordinated by UMass Dartmouth. The Southeast STEM Network program director coordinated publicity and communications, and provided other resources as needed. Evaluation Method: All summer camp attendees completed pre- and post-surveys assessing their perceptions in the following three areas: scientific knowledge, level of experience with science skills, and feelings about science. Students were asked to assess their level of scientific knowledge as it related to the general topic of the camp. For example, the survey for the Sustainability camp asked, “Rate your knowledge about Sustainability and the Environment” and provided four answer choices (“very high,” “high,” “some” and “a little”). The post-program survey question was similar, but not identical (emphasis added): “After participating in this camp, please rate your knowledge about Sustainability and the Environment.” Students were also asked to self-assess their level of knowledge in specific scientific areas relevant to the camp. For example, the pre-test asked, “In school, how much have you learned about these science topics?” Topics listed for the Forensics camp included structure of cells, blood types and finger printing. The carbon cycle, renewable energy and the ecology/ environmental science were listed as topics for the Sustainability camp. Campers were given a three-point answer scale of “A lot,” “Some,” and “Little or None.” The post-survey asked the same question with the slight change (emphasis added), “During this camp, how much have you learned about these science topics?” 14

Initial application from the Southcoast to the Pipeline, RFP, 2007. UMass Donahue Institute Research and Evaluation Group

20


Program Profiles

The second part of the survey asked students to rate their level of experience with science skills such as working in teams, conducting experiments, collecting and analyzing data and using microscopes. Again, students were given a three point answer scale of “A lot,” “Some,” and “Little or None.” A final part of the survey assessed students’ feelings about studying science. Students were asked, “Which word BEST describes your feelings about studying science?” with the following four choices: “Fun,” “Challenging,” “Hard,” and “Boring.” During the post-test, students were asked an additional question on whether their level of interest in learning about the subjects covered in camp had increased. For example, “After attending this camp, are you interested in learning more about sustainability and renewable energy than you were before?” Parents were also surveyed after camp and asked how much knowledge and skill they felt their child had gained and whether they felt camp had increased their child’s level of interest in science. Evaluation Results/Interest in STEM: The number of students who stated that they viewed science as “fun” increased after both 2009 camps, from 45% to 66% after the Sustainability camp and from 61% to 72% after the Forensics camp. At all four camps, students’ self assessment of scientific knowledge about the camp topic increased greatly after participation. For example, in 2009, 45% of attendees at one camp rated their knowledge of “Sustainability and the Environment” as high or very high before the camp and 85% after camp. 27% of students rated their knowledge of Biotechnology and Forensic Science as high or very high before camp compared to 95% after camp. Similarly, students’ self-perceived levels of knowledge in specific topic areas consistently increased after camp. The percentage of students who stated they had learned “A lot” about specific topics during camp (compared to their answers about what they learned in school) rose in all topics listed on the 2009 surveys. In response to post-only questions, most students (80 -85% of campers in 2009) indicated they were interested in learning more about the subject covered at their camp. Additional Evaluation Results: When students were asked to rate their level of experience with specific science skills, they also consistently rated their level of experience in all areas (working in teams, conducting experiments, collecting and analyzing data, and using microscopes ) as higher after camp. Parents seemed to share their child’s perceptions of increased knowledge, skills and interest in science. The parents’ survey had a 20% response rate. Most parents (88%) stated they felt their child had learned “A Lot” at camp. The remaining 12% of parents indicated their child had learned “something” from camp. All (100%) parents agreed with statements that participating in the camp had increased their child’s interest in science. Commentary/Implications for Future Application: These summer camps utilized several promising practices for encouraging student STEM interest. Students participated in hands-on learning activities with clear applications for the real world. They used professional scientific tools, and interacted with STEM professionals, including scientists and a police officer during the “mock” crime scene analysis. Students’ learning experiences included small group work and collaborative problem solving. Parent involvement was encouraged: parents were invited to attend their child’s presentation on a model eco-friendly community as part of the Sustainability camp. As the camps were five days long, students’ engagement with these “promising practices” was more in-depth than that offered by a one-day workshop. A number of Pipeline grantees have conducted STEM summer camps. However, this program conducted the most comprehensive evaluation of student interest and perceived levels of knowledge. Results from student and parent surveys clearly indicated that students felt they learned a lot, gained experience with scientific skills, and were interested in learning more. Reports indicated that the biggest challenge in camp implementation was meeting recruiting goals, both in terms of total number of students and the number from high-needs districts. During the second year, the program prioritized enrollment from high-need school districts, and focused outreach efforts to publicize the camps in them. However, despite distributing flyers about the camps to schools, the flyers did not always reach the students. Furthermore, some students who registered did not attend. Other methods of program recruitment, such as distributing information earlier in the school year, possibly directly to science teachers or parent groups (such as the PTO), as well as using more electronic methods (school website, e-mail, or social networking sites), could provide students with additional avenues to learn about the camps. Asking students and/or parents about the best way to publicize these camps may also yield some useful suggestions. This group used grant funding to off-set the cost of attending the camp and to support transportation to camp, so that in many cases, children did not need to make any payment to participate. Corporate sponsorship may be another way to promote and support student attendance at future camps.


21

Program Profiles


Programs that Conducted Pre-Post Assessments: No Measureable Results Engineering the Future by Design Lead Organization/Region: Andover, Brookline, and Worcester Public Schools Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: Middle school teachers attended two-day summer workshops at Northeastern University to learn the implementation of engineering modules featuring hands-on learning activities. The purpose of these workshops was to improve student attitudes and achievement in science and engineering. During summer 2008, teachers were trained using a wind turbine module. The program expanded in 2009, with the introduction of three additional modules: “Bottle Rockets,” “Supports and Structures,” and “Sustainable Food Engineering.” Two engineering teachers from Andover Public Schools designed the program. According to the project’s local evaluator, these teachers “…understood the needs of the teachers, designed well-organized and effective sessions, and provided the encouragement and expertise necessary for the teachers to successfully replicate the projects in their own classrooms.” Curriculum implementation started fall 2008, but was more widely implemented in the Andover, Brookline and Worcester school districts during fall 2009. Target Population/Reach: While the program is targeted to teachers, middle school students are direct beneficiaries of teachers’ improved knowledge and attitudes about engineering. According to the program’s June 2010, report, a total of 23 middle school teachers received training in one or more engineering modules. Of those teachers, 20 implemented one or more modules with their students. As a result, nearly 500 students received instruction in these modules over the 2009-2010 school year with approximately 51% of the students being female. For every teacher who implemented the curriculum, approximately 21 students received instruction. One Time/Series: Series Program Length: Students received multiple weeks of instruction aligned with state curriculum frameworks, estimated at between 7 and 18 hours of instruction. In/Out of School: In school for students, out of school for teachers. Teachers participated in two-day training workshops during the summer, then provided classroom instruction over the following school year to students. Stakeholder Involvement: Local schools and higher education institutions (Northeastern University) were involved in program implementation. One challenge grantees reported was their effort to involve local science business partners. Because most projects were initially implemented with apparent success without local business partners, the impetus to seek further business partners was weak. Evaluation Method: Most of the 500 students who received the curriculum were provided with pre-post assessments to gauge their interest in engineering, interest in other STEM subjects, and their content knowledge. Program administrators used a “comparison evaluation” to assess student attitudes. This survey asked students to choose “subjects they liked better” out of 26 word pairs describing activities that they would prefer doing. For example, students were asked if they liked “Building Computers or Painting” or “Multiplication or Foreign Languages” or “Designing Things or Playing Music.” The intent of the test was to determine if students would be more likely to choose STEM-related activities after participating in the program. Evaluation methods also included a survey for teachers following each professional development session. The survey assessed teacher’s perceived utility of the training. Additionally, the local evaluator conducted teacher interviews, focus groups and classroom observations.


22


Program Profiles

Evaluation Results/Interest in STEM: Results did not show any measureable difference in interest in STEM subjects before versus after the program. According to the local evaluator, “After examining the data from 491 students, the evaluators find that there is no discernable increase in the number of STEM subjects chosen in the post-test over those chosen in the pre-test. The evaluators find, therefore, that by this measure, student attitudes toward STEM subjects did not improve.” However, the evaluator asserted that this did not necessarily mean that interest in STEM did not increase. It is also possible that the instrument, as designed, while assessing interest in contrasting activities, did not actually measure interest in pursuing the study of STEM subjects or interest in a future STEM career. The evaluator indicated that even prior to the intervention, more students chose STEM than non-STEM activities, making an increase in STEM interest more difficult to assess. Finally, the evaluator stated that limited changes may also have been due to a short time frame between the pre- and post-interest surveys, which in some cases was reported as being as short as three weeks. Qualitative data gathered by the local evaluator through classroom observation and interviews indicated that teachers and students had high levels of satisfaction and enjoyment with the curriculum. According to the June 2010, program th report, “The evaluators conducted classroom observations of five 8 grade classes doing the wind turbine unit in Brookline and Worcester… In all of these observations the teachers were found by the evaluators to be comfortable using the … materials and tools with their students. They also seemed confident about helping their students solve open-ended design problems. The students seemed engaged in the activities, and exhibited a good understanding of the underlying concepts for the wind turbine they were building.” In follow-up interviews, teachers reported that students “loved doing the wind turbine unit.” Additional Evaluation Results: Analysis of student content knowledge gains was only partially completed when this report was written. However, the analysis that was completed and presented in the June 2010, report asserted that, “(G)ains were made in science content areas and … these gains are statistically significant.” Commentary/Implications for Future Application: Engineering the Future by Design used most of the promising practices recognized in the field of STEM programming. For example, the program used adult teachers who received two days of professional development and training and then implemented the curriculum with their students. The content was related to current real world challenges, such as the need for clean energy and ample food supplies. The curriculum also involved the application of engineering skills to challenges facing engineering professionals today. Students were involved with hands-on learning in which they worked collaboratively in small groups to solve problems. Finally, career information on engineering was provided to students. Since students received multiple weeks of instruction, this program provided students with a longer experience than most typical, after-school activities or a one-time event such as a science fair or conference. According to the local evaluator, one unforeseen positive outcome was the development of collegial relationships among teachers across the districts that participated. The June 2010, report indicated that a network of teachers created by this project continued to work together, to share teaching approaches and ideas, and to train additional teachers in their districts to implement the curricula. The report asserted that one effect of this change in culture was the doubling of engineering projects incorporated in one district. Given this initial input of resources, the project managers indicated that the program will be very sustainable and will require little funding to continue student instruction in the future.


23

Program Profiles


STEM Fellows and Leaders Project Lead Organization/Region: UMass Lowell / Northeast Region Funding/Duration: Network Funding – 2005 to 2009 Program Overview: The STEM Fellows and Leaders program was one of the more broad-based and overarching programs funded by the Pipeline. At a basic level, the program was to provide professional development for teachers to increase their content knowledge and pedagogical skills in STEM areas. However, the larger intent of the program was for teachers to work in teams to create a “Capstone Plan” for their districts. These Capstone Plans were to be strategic plans focused on increasing student performance in STEM and increasing the number of students interested in STEM subjects and career pathways. The design and implementation of a Capstone Plan was expected to affect the entire culture of a school district, creating a community of support for STEM education both within each district and across the northeast region of the state. The following quote from the original application to the Pipeline Fund explained the program’s intent: “By developing teacher expertise in local STEM issues, encouraging each team to create a strategic agenda supported by school district administrators, and supporting the implementation of those agendas under the guidance of STEM Leaders, the project expects to continue building regional capability to fuel the STEM talent pipeline sufficiently to keep the Massachusetts economy competitive.” The project was the recipient of two rounds of funding. Eight school districts participated in the initial project. Teacher participants in the first cohort (2005-2007) were called STEM Fellows. These teachers completed 30 hours of individual professional development pertaining to their subject area. As a district team of 4-6 teachers, they spent 20 hours working developing a curriculum-focused Capstone Plan for STEM in their districts. In addition, the Fellows attended four, full-day project workshops focused on STEM careers and cutting edge STEM focused industries. The culminating event showcased the STEM Fellows Capstone Plan. STEM Fellows were then charged with implementing the Capstone Plans in their districts. During 2007-2009, seven new districts entered the STEM Fellows project with a new cohort of Fellows who would have the same experiences as the first cohort. Fifteen of the former STEM Fellows were chosen to become STEM leaders. The Leaders guided the new teacher teams in the Capstone planning process as well as being trained in teacher leadership skills. Each district team was expected to implement at least one student-focused STEM career awareness project as part of their larger district Capstone Plan. Target Population/Reach: The target population for the first cohort of STEM Fellows was 40 STEM middle and high school teachers from 8 districts. At the end of the first round of the program, 33 teachers from 16 schools in six districts completed the STEM Fellows program. The next round of the project included 15 STEM Leaders chosen from the original cohort of Fellows and 38 STEM Fellows from the new participating districts. Each participating district also had a least one district administrator involved in the project as a representative to the Northeast Network Advisory Committee. Budget cuts prevented the program from further expanding the professional development opportunities for STEM Fellows. However, efforts have continued to focus on supporting and encouraging STEM Fellows to move into leadership roles in their districts. While the total number of students reached by participating teachers could not be fully tracked, over 700 students completed surveys on experiences with a STEM Fellow. One Time/Series: A series of professional development trainings and workshops were part of the preparation for STEM Fellows. However, since the focus was largely on planning and capacity building, this required on-going work beyond a one-time event or series of meetings. Program Length: STEM Fellows participated in 30 hours of individual professional development, 20 hours of team time, and four, full-day project events over the course 18 months. STEM Fellows and STEM Leaders spent additional time working on the implementation of their Capstone Plans in their school districts throughout the year. In/Out of School: This program was designed to affect students primarily in-school activities with their teachers. However, some teachers developed and implemented out-of-school activities as part of their Capstone Plans. Stakeholder involvement: The STEM Fellows and Leaders project was supported by a high level of stakeholder support and involvement, largely organized through the Northeast Network. The Network’s Advisory Committee UMass Donahue Institute Research and Evaluation Group

24


Program Profiles

included representatives of seven institutions of higher education,; Workforce Investment Boards from the North Shore, Merrimack Valley, and the Greater Lowell area; multiple industry partners, including Nantero, Inc., Jtech, Inc., Raytheon, Inc., Millipore Corp., Netfrastructure Solutions and Kronos, Inc.; and 16 school districts. Evaluation Method: Evaluation activities for this initiative were both formative and summative, including pre- and post-surveys, focus groups with STEM Fellows and Leaders, and a brief online survey conducted among students taught by STEM Fellows. The student survey asked several specific questions about their attitudes toward science, math and technology and their interest in eventually having a job that involves STEM skills. Survey and focus group questions also assessed the STEM Fellows’ and Leaders’ perceptions regarding the effectiveness and usefulness of the project in a number of areas and asked questions to determine whether there had been changes in their teaching methods. The student surveys were designed to assess students’ interest in STEM subjects and professional fields. Evaluation Results/Interest in STEM: Evidence of the effect on students’ interest in STEM was inconclusive, as there were no practical differences in results from pre- and post-surveys. According the March 2009 report, “(T)he students taught by the Fellow(sic) generally had positive attitudes and beliefs about education, and science, technology and mathematics education at the start of the year” leaving little room for improvement during the year. Most students agreed or strongly agreed that they enjoyed learning about science and mathematics. They generally agreed they needed to do well in science to get into the college of their choice and that learning science and mathematics will help them in their daily lives. Students were unsure whether they need to do well in science to have the job they want or whether they would like a job that involves using science. Additional Evaluation Results: According to the local evaluator’s report, “(T)he majority of Fellows felt that the Initiative was moderately successful or very successful for creating a cadre of STEM teacher leaders and for increasing the level of understanding of STEM-related issues in their district. The majority of Fellows also reported that the Initiative was moderately or very successful for fostering collaboration among STEM teachers in their district, for providing local and regional resources outside that can be leveraged in the future, and for helping to create a 15 sustainable Capstone Plan.” All of the STEM Leaders agreed that the initiative was moderately or very successful at providing professional development opportunities with effects that carried-over into classroom practices and “all the Leaders agreed or strongly agreed that through the Northeast STEM Pipeline Initiative they have developed a greater understanding of STEM-related issues in their school, district and the Northeast region of Massachusetts.” There was evidence that teachers changed their behaviors regarding talking with students about STEM careers. The percentage of Fellows who reported talking to their students about STEM career opportunities, and researching career opportunities in STEM fields, increased from the pre- to the post-survey. Qualitative results from open-ended portions of the surveys, as well as focus groups, indicated that many of the teachers felt the program had a deep and long–lasting effect on their perceived abilities to effectively teach STEM subjects, their teaching practices, their understanding of STEM fields, and their ability to convey this information effectively to their students. Commentary/Implications for Future Application: All of the promising practices described in the introduction to this report were promoted in the STEM Fellows and Leaders program. However, the specific ways in which these practices were implemented varied among districts, as teachers developed their Capstone Plans according to the individual needs and challenges of their districts. Many teachers’ responses to open-ended evaluation questions suggested that they were employing promising practices in greater measure in their own activities/teaching following their participation in this program. While the program did not collect data that clearly indicated an effect on student interest, this may be due to instrument design or other factors. Quantitative and qualitative results from the local evaluator’s assessment indicated that this program had the potential for long-lasting effects on teachers’ confidence and ability to teach STEM subjects in a manner that may affect student interest and performance in STEM fields. Additional follow-up with the STEM Fellows and Leaders to determine whether implementation of the Capstone Plans has continued in the districts after Pipeline funding ended would be valuable in determining this program’s potential for developing long-term STEM teaching, learning, and interest capacity.

15

2008 End of Year, Evaluator’s Report: Laura M. O’Dwyer Educational Consulting Associates, Inc. March, 2009 Priming the Pump of the STEM Pipeline in the Northeast Region: The Northeast STEM Fellows Network Findings from Participant Surveys p.20. UMass Donahue Institute Research and Evaluation Group

25

Program Profiles


STEM RAYS (Science, Technology, Engineering & Mathematics Research Academies for Young Scientists) Lead Organization/Region: University of Massachusetts, Amherst / Pioneer Valley Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: STEM RAYS provided after-school science clubs where students could conduct authentic scientific research during the school year. The goal of the program was to increase student interest in STEM fields. This program differed from some others in that it pre-dated the Student Interest funding provided by the Pipeline. It started in 2006, with funding from the National Science Foundation (NSF), which supported the program through June 2009. Pipeline funding allowed for further program evaluation and supported the program through early 2010. This program differed from many other afterschool programs, in that the groups were led and guided by certified teachers. These teachers participated in a professional development session at the beginning of each academic year that was specifically geared to the STEM RAYS program. During this kick-off training, faculty from local colleges helped teachers to gain content knowledge and practice research methods, as well as to explore potential research threads to pursue with their students. Teachers then worked with the students in their science club to conduct original research in a particular area throughout the year. Teachers met monthly with the professors they had been matched with to receive ongoing guidance. At the end of the school year, students presented the results of the research they had conducted at an annual symposium, which was attended by parents, administrators, state leaders and the media. No specific curriculum or materials kit was provided. Teachers and students worked together to determine research threads. During the 2009-2010 school year, four research topics were covered: (1) Engineering, (2) Sustainability, (3) Birds (identification and migration patterns) and 4) Global environmental change. Grant funds were used for teacher stipends, materials for the research clubs, and to support the assistance of faculty and curriculum specialists from Smith College. Target Population/Reach: The program targeted students in grades four through eight, as well as teachers that led the STEM RAYS clubs. Each club had up to 12 students. During the 2009–2010 school year, there were STEM RAYS programs at 15 different schools in the Pioneer Valley. Over 200 students participated, including 131 elementary and 88 middle school students. Nearly half (47%) of the elementary students were girls, and 13 percent were minority. Thirty-eight percent of the middle school students were girls while 16% were minority. Research groups were facilitated by nine elementary and six middle school teachers. Each club had between eight and 12 students. One Time/Series: Series Program Length: Clubs generally met weekly for two hours for the entire school year. In/Out of School: Out of School (after-school) Stakeholder Involvement: As stated earlier, 15 different schools participated in the program over the 2009-2010 school year. College faculty supported the program by providing content knowledge and guidance in practicing research methods with teachers. Smith College, the University of Massachusetts at Amherst, and Greenfield Community College all had professors who mentored teachers leading these groups. Evaluation Method: Multiple evaluation methods were used to assess the effect of the program. Students completed pre-post tests assessing their knowledge and understanding of general science concepts. (Questions assessing students’ understanding of the nature of science generally were adapted from a previously used instrument titled, “Views of the Nature of Science” which asked true/false questions such as, “Scientific theories are explanations and not facts” and “Scientists often try to disprove their own ideas.”) Pre-post tests asked students an open-ended question about their future career aspirations: “When I am 25 years old I would like to be working as a …” Teachers,


26


Program Profiles

administrators and parents were surveyed regarding their perceptions of the effect of the program. Information was also gathered through external observation of science clubs and focus groups conducted with teachers and students. Evaluation Results/Interest in STEM: While fully analyzed results for the 2009-2010 school year were not yet available at the time of this report’s writing, results from previous years that assessed the effect of the program on both student and teacher participants were available. Results from pre-post surveys did not indicate any measureable change in students’ interest in pursuing STEM or non-STEM careers. However, other qualitative information gathered from surveys, observations and interviews with teachers, parents and administrators suggests that participation may have increased student interest in STEM. SageFox Consulting, the local evaluator, reported that “Parents consistently noted an increased interest and understanding of important environmental questions” and that “There were numerous comments about students’ increased interest in studying particular scientific disciplines in college and pursuing careers in science.” Program reports suggested that surveys of administrators indicated an increase in students' interest in science. The report also stated that the program manager witnessed “a high level of engagement, good utilization of science and engineering methods and tools, and great enthusiasm.” Additional Evaluation Results: While pre-post test results for the 2009-2010 school year had not been fully analyzed, during the previous year, student content knowledge scores, and the percentage that answered questions correctly, increased in each of the four research areas pursued by students. Initial results from 2009-2010 suggested that student content knowledge increased, quite dramatically in some cases. Students were also asked, “What was the best thing you learned in STEM RAYS this year?” Over half of the students mentioned “the importance of working in a group to being able to achieve good science results.” Teachers completed a self-assessment of their research skills at least twice and sometimes three times over the course of the project. According to program reports, results from this self-assessment indicated improvements in nearly all areas. Teachers also completed surveys and participated in focus groups. A report submitted in October 2009, stated that, “Overall we found that the teachers increased their knowledge of science content and methods, they increased their comfort with open-ended science projects, and they reduced their reliance on pre-packaged science activities and worksheets. The teachers were also more likely to discuss their research projects and findings in scientific terms and language when in their group.” Another indicator that the program was successful in generating student interest was that it had consistently high levels of participation and low drop-out rates for three years. The number of students seeking enrollment in the program has often exceeded the number of spaces available. Commentary/Implications for Future Application: The STEM Rays program provided an intense and long-lasting experience of several promising practices in STEM programming for elementary and middle school students. Students worked in small groups to conduct authentic scientific research over the course of a full school year. While the level of interaction with a STEM professional (other than their science teacher) was not clear and would have varied between projects, it is clear that information about STEM careers was provided to students, and that their experiences reinforced the real-life relevance of their work. Teachers also gained skills and knowledge as a result of participating. Evaluation results suggested that teachers’ comfort with having open-ended, hands-on discovery as learning activities for their students increased. When further local evaluation results for STEM RAYS become available, they may yield additional insights into this program’s effect on student interest. Going forward it would be very helpful to ask a number of additional, specific questions assessing student interest. Questions asking students about their interest, skill and confidence in multiple STEM subjects, and about their plans to take STEM-related courses, would yield much more information about the potential for this program to affect student interest. The local evaluator did indicate that it was challenging to develop tools and questions that accurately assess student interest or students’ general conception of the nature of science in an age-appropriate manner. This may be particularly challenging for this program as it involves students who vary greatly in age and development.


27

Program Profiles


Programs that Used Post-only or Qualitative Assessments Berkshire STEM Career Fair Lead Organization/Region: Massachusetts College of Liberal Arts / Berkshire Region Funding/Duration: Network Funding – 2007 to 2010 Program Overview: The purpose of the Career Fair was to introduce Berkshire high school students to area STEM professionals in order to increase student interest in STEM subjects and careers. The fair has been held annually every fall since 2007. Generally, between 15 and 25 representatives of area businesses and employers have participated as exhibitors. STEM professionals were encouraged to interact with students, to answer questions about their work, and to tell students about their company and potential future career opportunities. The fairs were organized over a four hour period during one school day, with two separate two-hour sessions. Target Population/Reach: While the program initially targeted juniors and seniors, due to requests from teachers, ninth and tenth graders were also invited to attend the fair in 2009. During this fair, 357 high school students from 10 different school districts attended. Similar numbers attended each of the previous two years. According to the local evaluator’s report on the 2009 fair, “Gender, race, and grade level breakdowns were available for only 5 of the 11 schools participating. Based on the schools reporting, 42% of participants were female, 9% were minorities, 2% were freshman, 4% sophomores, 55% juniors, and 35% seniors. These breakdowns mirror those from the 2008 Career Fair, when data from all schools was available.” One Time/Series: One-time Program Length: Four hours (two consecutive two-hour sessions). In/Out of School: In school. Technically, the career fair was held off school grounds at Berkshire Community College. However, it was not an after-school, weekend or summer program as most “out-of-school” activities are. The fair took place during a regularly scheduled school day as a field trip. Students were chaperoned and bus transportation was provided. Stakeholder Involvement: Clearly multiple stake-holders have been involved with this event each year, as 25 exhibitors from STEM businesses participated in 2009. Eleven high schools from ten different school districts sent students to the fair. Higher education institutions have also been involved, as Berkshire Community College has hosted the event and the Berkshire Tech Prep Consortium (based at BCC) coordinated the programming. UMassAmherst and Massachusetts College of Liberal Arts (MCLA) have also participated, sharing information about their respective institution’s STEM offerings. Evaluation Method: Multiple methods were used to provide both formative and summative evaluation results for this program. Students, chaperones and exhibitors were surveyed regarding their experiences. Students were asked to indicate how strongly they agreed or disagreed with a number of statements on whether participation changed their interest in STEM careers or their plans to take science or math courses in the future. Students were also asked whether they would be interested in additional opportunities to learn more about STEM careers. Chaperones were asked how strongly they agreed or disagreed with a number of statements regarding student engagement, presenter preparation, whether exhibitor presentations were age-appropriate, and whether they would recommend participation in this fair to others. Exhibitors were asked how strongly they agreed or disagreed with statements regarding whether the communication of expectations and logistics were clear, whether they felt participation was useful for their companies, and whether they would participate again or recommend participation to other employers. Both chaperones and exhibitors were asked open ended questions about what they thought the


28


Program Profiles

benefits of participation had been either for the students (asked of chaperones) or for the participating companies (asked of exhibitors). Evaluation Results/Interest in STEM: According to the local evaluator’s report, 85% of students stated they were planning to take more math and science courses in high school than they had been before participating in the career fair. Two-thirds (66%) indicated they were planning to work harder in their math and science courses. Nearly eighty percent (79%) reported that the career fair helped them realize that the math and science they take in high school will affect their career options. Over one-third (35%) indicated they were thinking about going into a high-tech career more than they were before. More than two-thirds reported that the career fair made them realize that a STEM career could be interesting. Nearly ninety percent (88%) reported that the career fair helped them learn about career opportunities in Berkshire County that they didn’t know existed before. The local evaluator reported that many of the 253 students who completed a survey expressed an interest in additional opportunities to learn about STEM careers. Between 42% and 55% (depending on the question) of respondents indicated they would be interested in “hanging out with” or having a tour/internship at a local employer. Over one-third (34-39%) said they would like more information about careers and/or salaries in STEM fields. Twenty-one percent said they would be interested in pursuing an in-depth, career-oriented project. Additional Evaluation Results: The local evaluator’s report stated that, “Feedback from STEM Career Fair chaperones was uniformly positive.” All of the chaperones stated they felt the presenters were well prepared and that the content of the presentations was appropriate for high school students. Nearly all (17 of 18) indicated they felt students were engaged by the presentations, that they would bring their students back to the career fair again, and that they would recommend this fair to other schools. Chaperones also provided suggestions for additional local employers to include in future fairs, with some encouraging more female presenters and others suggesting that summer internship opportunities be offered by the companies. Among the exhibitors, all respondents indicated that career fair expectations were clear and that set-up and implementation ran smoothly. All indicated they would participate in the fair again and would recommend participation to other companies. Most (22 of 25) felt that participation was a useful activity for their company and nearly all (24 of 25) agreed that communications regarding logistics were organized and provided sufficient details. Commentary/Implications for Future Application: The career fair employed some commonly used promising practices to encourage students’ interest in STEM, most notably the provision of career information by STEM professionals who interacted with students. While this activity may not have utilized as many promising practices as some of the other initiatives described in this report, it had the benefit of reaching a large number of students (over 300 annually). Extensive hands-on learning activities or small-group research and problem solving would not have been practical or possible for a brief career fair reaching so many students. However, student responses to surveys indicated that the program may have increased their awareness of, and interest in, STEM careers. As such, this is quite possibly a valuable activity that, if implemented as part of additional regional efforts to engage students in STEM, could be an effective initial strategy to spark student interest in STEM. Others interested in implementing similar fairs may benefit from lessons learned from the Berkshire Career Fair. According to the local evaluator’s report, several chaperones suggested that, “due to the large number of student attendees, efforts (should) be made to relieve congestion at booths, such as staggering students’ entrance into the exhibition area or spreading tables out more. Other suggestions to enhance student interest included displaying products, including more female presenters, and including summer internship opportunities. Chaperones also offered many suggestions of local employers to include in future fairs. Other recommendations for improving fairs included encouraging employers to offer more hands-on activities at each booth. One exhibitor suggested the students should be encouraged to treat this like a real job fair and “dress professionally, introduce themselves.” This exhibitor noted that “there were a lot of one-sided initial discussions” and that perhaps “descriptions of the exhibitors should be included in the student materials.”


29

Program Profiles


Family Science Programs Lead Organization/Region: Massachusetts College of Liberal Arts / Berkshire Region Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: The Berkshire Family Science Programs were designed to introduce middle school students and their parents to STEM careers and professionals in order to increase student interest in STEM. While the format and topic of program implementation varied greatly across the Berkshire region, the characteristic that all Family Science programs shared was that students participated in hands-on STEM projects with their parents. The programs gave students and their parents opportunities to interact with the STEM professionals who led the learning experiences. In some cases, programs were offered once while in other cases the program was offered as a series over several weeks. The topics and timing of the programs varied as well. Programs were offered after-school, on weekends, and during school vacation. In one case, students and their parents chose one of three workshops on biology, chemistry or robotics. These one-time workshops lasted three hours. Another Family Science program, called “Designing a Green Community” involved four, three-hour math workshops offered after school on a weekly basis. In these workshops, children and their parents worked together using math to build, design and display a green community. Other workshops on robotics, chemistry and microbiology were offered by professors on Saturdays at the Fisher Science Center at Bard College at Simon’s Rock. Each workshop was two and one-half hours long and involved 20 sixth graders and their parents in hands-on learning activities. According to their June 2010 report, coordinators redesigned the program in 2010, in order to reach more families. Data assessing students’ reported interest in STEM careers and subjects as a result of the newly formatted program were not available at the time of this report’s writing. As a result, this profile describes the older model used during 2008 and 2009. Target Population/Reach: The primary target population for this initiative was middle school students and their parents. Originally, program coordinators hoped to have 40% of participants come from low-income households. (Low income status was determined through a survey which asked parents if their child participated in the free or reduced school lunch program.) Since 2008, 5 to 20% of participants (depending on the workshop) came from low-income families. Approximately 12 -25 students and their parents participated in each workshop. Of the 24 students that participated during 2009, data indicated that seven (29%) were female and one (4%) was minority. Twenty-two parents participated during 2009. One Time/Series: Both. Some workshops were offered once and others were offered in a series. Program Length: Generally workshops were three hours long, however the length varied. In/Out of School: Out of school. Programs were offered after school, evenings, and weekends primarily. A more recent version of the program added an in-school component. Stakeholder Involvement: Multiple stakeholders were involved in this program. Business and industry professionals helped sponsor and facilitate the workshops. Higher education institutions provided space and faculty to lead workshops. A local science museum and library hosted workshops. Evaluation Method: Students took a post-survey asking about the following three aspects of STEM interest: -

Student intent to take science courses in high school Student interest in science classes in school and science as a future career Interest in participating in another Family Science program.

Students were also asked to reflect on how they would have answered such questions “last year,” or prior to participating in the program, as well as to report whether they thought their interest or intent may have changed as a result of participating. UMass Donahue Institute Research and Evaluation Group

30


Program Profiles

Parents took pre- and post-surveys that asked about how many years of science classes they expected their child to take in high school. Parents were also asked to offer their perception of their child’s aptitude in science and to indicate how likely they would be to participate in another Family Science program in the future. Evaluation Results/Interest in STEM: Seventy percent of students stated that participating had increased their interest in pursuing a career in science. Only half of the respondents indicated they had given any thought to the amount of science they would take in high school. Of those that had thought about their plans, seventy-five percent (eight of 12) indicated that participating in the program did not change their educational plans for taking science classes. Additional Evaluation Results: Over ninety percent (91%) of students reported that the program gave them a better idea of the different things scientists do and seventy percent (70%) said the program improved their understanding of what scientists are like. According to the local evaluator, there was no reported change in parents’ aspirations for their child’s participation in science classes in high school, or in their views of their child’s science aptitude, as many already viewed their child’s ability as good or excellent. There was a high reported interest among students (88%) and parents (95%) to participate in another Family Science program. Commentary/Implications for Future Application: The varying activities, structure, and length of these programs made it difficult to make any generalized statements about their use of promising practices. However, it is clear that the programs usually encouraged hands-on learning and problem solving in small groups. The program also made use of promising practices by having learning experiences that were led by STEM professionals, and that often took place in a STEM workplace/environment, such as a college science center. The involvement of a caring adult is one promising practice that is frequently mentioned in the literature, and this program’s consistent involvement of parents as participants was unusual. Only three of the 25 parents (12%) who participated in this program indicated that they had ever participated in a science program along with their child before. As most of the student interest programs funded by the Pipeline did not involve a parent component, and parents often play a substantial role in influencing their children’s interests in career goals, it could be useful for other initiatives to consider integrating a parent component into their STEM outreach and education activities. As noted above, more recent programming was re-formatted to reach more students. During spring 2010, the program added an in-school component at two Pittsfield elementary schools, followed by a two-day event over the following weekend at Pittsfield State Forest. The program was re-named “Bio-Blitz” and was organized in collaboration with the Berkshire Museum. During the program, children were encouraged to bring their parents to collect and identify plant and animal specimens with the assistance of area scientists and specialists. Program numbers under this new model increased to 129 elementary students (generally ages 8 and 9), 10 teachers, and 50 parents. Evaluation data for this newly designed program was much more limited, possibly due to the younger age of the student participants. Initial evaluation results of this BioBlitz model were that 62% of participating students rated the program as “very interesting” and 79% of students indicated they learned “quite a few new things.”


31


Program Profiles

MA Region I Middle & High School Science Fairs Lead Organization/Region: Massachusetts College of Liberal Arts / Berkshire Region Funding/Duration: Pipeline Funding (Network and Project) – 2005 to 2010 Program Overview: The Massachusetts Region I Middle and High School Science and Engineering Fairs were hosted by the Massachusetts College of Liberal Arts (MCLA) and overseen by the Berkshire STEM (Science, Technology, Engineering and Math) Pipeline Network. The Region I fairs have been held at MCLA from 2005 through 2010. The goals of the Berkshire science fairs have been to increase student interest in science, to increase subject matter mastery among participants (assessed through the average scores judges’ gave to projects), and to impact science teaching and curricula. Region 1 includes the following western counties: Berkshire, Franklin, Hampshire and Hampden. Students could enter projects on any scientific, mathematical, or engineering subject and could compete as individuals or in a team of up to three members. Students were required to pre-register for the fair. All research projects required approval by a Scientific Review Committee, and adult sponsors or supervisors to guide students in their research projects and act as chaperones during the fair. Many students spent months on their research projects prior to exhibiting their work. The fairs generally started at 8:30 with registration and setup, followed by judging. Students presented their work to professional judges as well as to other attendees and each other. Each project was evaluated and scored by five judges. The day ended with a key note speaker and an awards ceremony. The top 40 projects could advance to the statewide science fair at the Massachusetts Institute of Technology (MIT). After taking down exhibits, the fairs typically ended in the late afternoon. Target Population/Reach: High school students (grades 9 through 12) and middle school students (grades 6 through 8). During 2010, more than 300 students, 80 business people and 50 teachers participated. Adults from the science, math and engineering business and teaching communities participated as judges and chaperones. One Time/Series: One-time (One fair for middle school students and one for high school students). Preparation for the fair takes place over an extended period of time, often several months. Program Length: One full day for fair, several months for research & project development In/Out of School: Out of School. While some schools and teachers allowed students to use classroom time to prepare their projects for the fair, most students spent time out of the designated school day to prepare their projects. The fairs themselves were often held on school days at MCLA. For some schools the science fair was part of the curriculum. Stakeholder Involvement: The regional fairs were supported in part by the MSSEF’s (Massachusetts State Science and Engineer Fair) Curious Minds Initiative and the Gelfand Endeavor in Massachusetts Schools (GEMS) to promote inquiry-based learning and science fair participation in schools. The planning committee for the fair has invited individuals with degrees and/or work experience in STEM fields to serve as judges for the event. Evaluation Method: A number of methods were used to evaluate the science fairs. High school students were asked to complete a brief on-line survey about their experience after participating in the fair. Three questions in the survey assessed student interest as defined in this report. Two questions asked whether students planned to major in or have a career in a STEM field. A third question asked students to reflect on their experience, and to indicate, through a number of preset statements, possible benefits of participating in the fair. One of these statements addressed student interest as it may have changed over time. The statement asked students to indicate how important the fair was to “Building my confidence in my science abilities.” UMass Donahue Institute Research and Evaluation Group

32


Program Profiles

The student survey also asked questions about their experience preparing for the fair, the benefits of participating, and whether they planned to participate in the science fair in the future. Students were asked whether participation in the fair had been a requirement for their science class, where and when they prepared for the fair, how much assistance they received from teachers, whether they consulted with local businesses or local experts in the field and whether they had participated in a science fair at their school. Questions on possible benefits of the fair asked students to reflect on their experience and indicate how important participation had been for them in a number of specific areas, including the “Getting to do hands-on science work” and “Working closely with a teacher or mentor” or increasing their content knowledge by “Learning science that I didn’t know before.” Students were also asked to reflect on whether the fair was well organized and whether they had any suggestions for improving the fair. A brief survey of school-level science fair coordinators was also conducted to assess the schools’ roles in supporting and encouraging participation of students. Survey questions asked if schools required the participation of any students, whether any after school programs devoted time to project preparation, and whether schools hosted their own science fair. The number of participants at each fair and the average scores of the projects given by the judges was also tracked and reported. Evaluation Results/Interest in STEM: Ninety-two (92) of the 183 high school student participants in the fair completed the online survey regarding their experience. According to the local evaluator 41% of the respondents indicated that the fair was a “very important” or “important” benefit in "building my confidence in my science abilities." This indicates that a large proportion of the participants felt that preparing for and participating in the fair increased their sense of self-efficacy in science, which is indicator of student interest. The majority of students reported having an interest in STEM: 72% reported that they were planning on majoring in a STEM field in college and 74% indicted they were considering a career in a STEM field. Since these questions were not asked prior to the fair, it is not possible to determine whether the percentage of students interested in a STEM major or career changed. Additional Evaluation Results: The local evaluator’s report stated that, “Over the four years of science fairs funded under the Pipeline initiative, participation in the high school fair by Berkshire County students grew by 158% and participation in the middle school fair grew by 243%,” to 183 high school students and 135 middle school students in 2010. While the average scores given to projects by judges did not substantially change at the high school level, middle school project scores increased steadily from 52.7 in 2007 to 83.1 in 2010. One-third of student respondents reported consulting with a local business or expert (e.g. scientist or college faculty) regarding their project. Two-thirds of students indicated that they completed their projects mostly at home, with little or no assistance from teachers. Most respondents (79%) indicated that “Building my resume for college” was an “important” or “very important” benefit. Students rated a number of other benefits of participating in the fair as “important” or “very important,” including: “Getting to do hands-on science work” (75% of respondents), “Working closely with a teacher/mentor” (17%), “Working with other students” (41%) and “Learning science that I didn’t know before” (68%). 91% of students indicated that the fair was “pretty well” or “very well” organized and 53% indicated they would like to participate in the regional science fair during the following year Commentary/Implications for Future Application: Results of the student survey indicated that preparation for and participation in the fair enabled many students to experience a number of the promising practices described in this report. Interestingly, these survey questions also assessed students’ perception of the importance of these experiences to them, which was unusual among Pipeline project evaluation instruments. Most respondents indicated that the opportunity to do hands-on scientific work and to work with other students was an important benefit of their experience. Nearly one out of five also indicated that the opportunity to work with a teacher or mentor was an important benefit for them. Results of the student survey suggest that there was a strong interest in STEM subjects (indicated by the high percentage of students planning to have a STEM major in college or considering a STEM career) among respondents UMass Donahue Institute Research and Evaluation Group

33


Program Profiles

after the fair. However, the results provide little information on whether student interest in STEM may have changed as a result of participation. One indicator suggested that the experience did increase student interest in STEM as defined in this report, as 41% indicated they felt an important benefit was increased confidence in their own science abilities. This benefit, the high level of STEM interest documented after the fair, and the positive benefits that students reported from engaging in promising practices, suggest that the science fair experience may indeed be increasing student interest in STEM. Future evaluations could provide more information on changes in student interest as a possible result of participation in the fair. Asking identical survey questions before and after the fair (if possible) would enable a comparison of student interest data. A survey which asks students to reflect on whether they believe the experience affected their interest in STEM (particularly regarding the various components of student interest as outlined in the introduction to this report) would provide more information about the science fairs’ potential effects on student interest in STEM. A number of lessons have been learned over the years of coordinating the regional science fairs. In 2010 the Berkshire County STEM Steering committee made a decision to limit the number of student entries per school to 12 in order to increase the diversity in school participation. School-level survey results also suggested a way to increase the diversity of participants: schools that required some level of participation in the fairs entered more of projects. The network may wish to encourage schools that have not participated regularly to provide strong incentives for student participation or to consider making participation in the fair a requirement for some classes. Students also provided information on lessons learned in their responses to survey questions. When asked to offer advice to future participants, they recommended that future participants pick topics they are interested in, start early on their projects and work hard. Finally, it should be noted that the Berkshire Network took important steps to sustain the science fair over the long term by forming active partnerships and seeking the support of STEM focused organizations.


34


Program Profiles

Saturday STEM Academy Middle School Project Lead Organization/Region: Framingham State University / MetroWest Region Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: This program targeted underserved, minority eighth grade students and endeavored to increase student interest in, and motivation to pursue, STEM careers. It was designed to increase awareness of STEM careers and the need to participate in STEM academic subjects to prepare for these careers. Students and their parents were invited to participate in a series of STEM-related activities over four separate Saturdays. Two program orientation sessions were held to introduce the program to parents. One was held at a local church, and one was for local parents recruited through the METCO program. (Note: METCO is a grant funded program of the Commonwealth of Massachusetts “intended to expand educational opportunities, increase diversity, and reduce racial isolation, by permitting students in certain cities to attend public schools in other communities that have agreed to participate” according to the state’s website at http://www.doe.mass.edu/metco.) Each Saturday involved a separate field trip, where students were given the opportunity to participate in hands-on activities and to meet professionals employed in STEM fields as role models. During the first half of 2010, the program included four field trips during late winter and early spring to the Christa McAuliffe Learning Center, the Olin College of Engineering, the U.S. Army Soldier Research, Development and Engineering Center in Natick, and Framingham State University. A June 2010, program report described students’ experiences at the four sites: (1) The Christa McAuliffe Center: “Following a short pre-mission briefing, participants took their positions in either Mission Control or the Space Station, and assumed control of the Center’s two-hour ’Mission to Mars’ simulation. Working in teams, participants were able to experience both roles during this period with exposure to life-like engineering problem solving in a space exploration environment.” (2) Olin College of Engineering: students were exposed to an engineering environment and took part in hands-on design-and-build activities led by Olin College engineering students. (3) U.S. Army Research, Development and Engineering Center: “…students witnessed demonstrations in various laboratories and listened to presentations emphasizing the importance of STEM to the Army, society, and the U.S. economy.” The experience also included an opportunity for students to ask questions of scientists and engineers employed at the site. (4) During the fourth and final session, student teams were engaged in “building, programming, and testing Lego robotic vehicles for the remote-controlled exploration of a simulated Mars terrain.” Parents participated in this session, and observed a “PowerPoint presentation on Lego robotics and its relevance to the engineering design process.” At the end of the day, students were able to share and demonstrate the use of their completed robots to their parents and each other. Target Population/Reach: The program’s primary target population was underserved and minority eighth grade students. During the spring of 2010, 31 minority students (including 17 girls) participated. All students were recruited through the METCO program. Between 20 and 31 students participated in each of the four Saturday workshops. While the program initially intended to work with Newton, Needham, Natick and Wayland public school districts, the most recent report stated that they conducted additional outreach only through the Lexington METCO program. One Time/Series: Series Program Length: The program was conducted over four Saturdays throughout spring, 2010. Each session was approximately 6 hours long. In/Out of School: Out of School Stakeholder Involvement: According to the June 2010, program report, in addition to the host organizations for the field trips listed above, additional partners included the Metro South/West Regional Employment Board (MSWREB); the METCO programs of Lexington, Newton, Needham, Natick and Wayland public school districts; and the Greater Framingham Community Church. Business partners Raytheon and Intel also provided financial support for programming.


35


Program Profiles

Evaluation Method: At the conclusion of the fourth Saturday session, students were given post-program surveys consisting of 17 questions that, according to a program report, “…were intended to gauge their awareness and interest in STEM topics and careers and their motivation to continue engaging in these experiences.” The post-only survey had a number of questions where students could indicate how strongly they agreed or disagreed with statements about engineering, perceived self-efficacy and interest in pursuing a STEM career, their plans to continue studying math and science, and the importance of engineering. Evaluation Results/Interest in STEM: The local evaluator found that results of the post-survey conducted with one cohort of students indicated that students had a strong interest in, and awareness of, the importance of STEM subjects and career possibilities. All of the students indicated they agreed or strongly agreed with statements like “Engineering is a very important profession” and “Engineers do lots of different and interesting things.” Most students (86%) agreed that “Being an engineer can be exciting” and 81% agreed or strongly agreed that “Engineering is a good career for both minorities and women.” According to the program’s local evaluation results, “…students are now very aware of STEM careers and the importance of Math and Science.” All of the students agreed with the statement “Math and Science is (sic) important for me to be successful in life” with 76% strongly agreeing. Some questions attempted to assess whether students’ perceptions had changed as a result of participation in the program. Two-thirds (67%) of students agreed or strongly agreed with the statements “I am now more motivated to study math and science in high school” and “I am now more motivated to prepare myself to go to college.” However, when students were asked specifically about their interest in pursuing a career in STEM, the local evaluator summed up the results by saying, “There is far less certainty when asking students of this age about pursuing a STEM career.” Despite this, nearly one-quarter of students agreed with “I would like to become an engineer” and “I would like to go to an engineering college after high school.” Additional Evaluation Results: The local evaluator conducted observations at two of the four Saturday sessions. The evaluator summarized the experience as follows: “Students were observed to be actively engaged in the lunarlanding simulation and the project-based robotics activity at the Christa McAuliffe Center at Framingham State College. The hands-on activities provided a well-matched experiential opportunity to assume the role of an engineer.” Commentary/Implications for Future Application: While students’ experiences varied from week to week, most of the promising practices described in the literature review for this report were employed in this program over the course of the four Saturdays. Participating students and their parents were given career information and the opportunity to visit STEM companies and interact with STEM professionals. Three of the fours weekends involved hands-on, learning activities with the two involving students in designing and building projects. Parents were also encouraged to attend the programs, which is another promising practice recommended in the literature. Program reports shared a number of challenges and lessons learned during the implementation and evaluation of this program. Program staff reported they initially had trouble recruiting enough students to fill 25 slots for the workshops, so they expanded outreach to additional school districts. Program reports also recommended sharing a planned itinerary of program activities at the beginning of the first session with students, as coordinators found that the information they had forwarded to the parents did not consistently reach the students. With regard to evaluation results, the local evaluator suggested that one possible reason for a lack of evidence of strong interest in pursuing a career in engineering may be related to the students’ young age and questioned whether this is an age appropriate goal for junior high students, stating “It well may be that developmentally, awareness and interest are appropriate goals for students of this age, while interest in pursuing any career may not be.” To improve future evaluation of this program, the local evaluator suggested that “the evaluation questions need to be fine-tuned so that they more accurately measure student’s interest in STEM careers.” However, it is also possible that the level of interest students expressed was high and an indicator of a successful program. As stated above, nearly onequarter of the students indicated an interest in becoming an engineer or attending engineering school. This level of interest in any school population might be considered high. However, since no survey was conducted before the program, it cannot be determined if there was a change in the assessed level of interest, or student perceptions regarding the importance of engineering, science and math. In the future, if an identical survey could be administered before the start of the program, it would be possible to assess any change or growth in students’ reported interest in STEM careers. UMass Donahue Institute Research and Evaluation Group

36

Program Profiles


STEM Career Awareness Conferences Lead Organization/Region: University of Massachusetts Medical School / Central Region Funding/Duration: Student Interest Funding – 2008 to 2010 Program Overview: Several STEM Career Awareness Conferences have been offered by the Central STEM Network since March 2008, as part of its Career Awareness Campaign, including five separate conferences in 2009, alone. The conferences were designed to provide middle school students with STEM Career information and to inspire interest in STEM careers. Two of the conferences were for girls only, titled “Women in Science” and one was for boys only, titled “Men in STEM.” Each day-long, Saturday conference hosted from 85 to 200 students and their teachers. During 2009, the five conferences offered a total of 58 workshops and affected 650 middle school students. All conferences were coordinated in conjunction with local school districts and teachers. Most conferences started with a brief, large-group session, followed by smaller workshops, lunch, another large-group session, and a closing activity. Conferences often included a keynote address or panel discussion delivered by professionals who shared how they came to be employed in STEM. Presenters often described the challenges, successes, and rewards they experienced in their work. Small-group workshops were delivered by volunteers from STEM businesses who offered information about their careers and had interactive discussions with students. During 2009, volunteer presenters were employed in engineering, computers, forensics, robotics, marine life, meteorology, veterinary science, biotechnology, green chemistry, renewable energy, public health and nursing. Many workshops enabled students to experience real-life aspects of STEM careers through hands-on activities. Students reported they enjoyed feeding foxes, making casts and splints, building a tower using a tennis ball, and holding/observing sea creatures. Conferences were often held at science-related sites, including the Worcester EcoTarium, Intel, and WPI. In spring 2010, the Blackstone Valley conference provided an additional session for parents. Forty parents attended a one-hour session in which the Executive Director of the BioTech Council spoke about STEM in general and a representative from the Massachusetts Educational Finance Authority (MEFA) gave information about financing for postsecondary and retraining opportunities. Target Population/Reach: The intended target population was middle school students, with a special emphasis on girls and under-represented minorities, first-generation college, and low-income families. In addition, some volunteer partners in this initiative, (e.g. WPI) expressed a particular interest in hosting opportunities for women, and underrepresented minorities. The conferences reached large numbers of girls and students of color (both underrepresented groups in STEM fields) from urban districts. On average, between 85 and 200 middle school students and their teachers attended each conference. The Women in Science conferences in Worcester and Shrewsbury-Westborough reached 140 and 200 girls respectively. Mixed gender conferences usually had about 50% female attendance. The conferences in the Worcester, Fitchburg, and Leominster school districts were offered in more urban setting. According to program reports, 55% to 70% of participants from these districts were minorities. Conferences were also offered in more suburban school districts. Reports on the Shrewsbury-Westborough conference indicated that 20% of the students were minorities while 7% of attendees at the Blackstone Valley conference were students of color. One Time/Series: One time Program Length: Approximately 6 hours In/Out of School: Out of School / Saturdays. However, programming was facilitated in part through teacher recruitment. Bussing was generally arranged from schools and teachers attended the conferences with their students. Stakeholder Involvement: The STEM Career Awareness conferences have had some of the highest levels of business involvement of all Pipeline-funded projects. Local company employees from Abbott Pharmaceuticals, Intel, and EMC have participated in, and provided in-kind donations to, the conferences. The value of in-kind donations was $62,000 in just one year. There was also a high level of higher education involvement, as WPI, UMass Medical School, and Tufts/Cummings School of Veterinary Medicine have all participated by providing presenters and/or UMass Donahue Institute Research and Evaluation Group

37


Program Profiles

space for the conferences. According to program reports, many of the collaborative relationships that helped make these events successful evolved from stakeholder participation in Central STEM Network. Evaluation Method: Students, teachers and volunteer presenters were asked to complete post-conference evaluation surveys in which they rated and commented on their experience participating in the conferences. Surveys of students asked them to answer open-ended questions about what they learned, and whether their career goals had changed as a result of participating in the conference. The conferences had observation protocols in place for teachers, staff, and a local evaluator to assess the level of student engagement in workshop activities. A focus group was conducted by the local evaluator to get feedback from conference organizers. A debriefing meeting was generally held with participating teachers two to three months after a conference. Teachers who attended the debriefing sessions were encouraged to ask their students ahead of time what they remembered most from the conference. Evaluation Results/Interest in STEM: Since student surveys asked open-ended questions, all of the data assessing student interest in STEM and career goals was qualitative in nature. One example of survey results was from the Women in Science conference in 2010. Students were asked to respond to the question, “Now that I’ve participated in this conference, my career goals have changed/not changed because…” Results from the 115 surveys collected indicated that 60% of respondents felt that the conference had changed their career goals and that they were now interested in a STEM profession. 28% indicated that participation had strengthened their pre-existing interest in a STEM career, and 13% stated that participating in the conference had no impact on their career goals. Many students stated that they became aware of new career opportunities and that they had learned more details about familiar STEM careers during the conference. Additional Evaluation Results: According to program evaluation reports, “Feedback for all of the conferences was consistently positive.” Students’ average rating of the workshops at the 2009, conferences was 2.49 (3 being excellent, 2 being okay and 1 being poor). Students rated keynote presentations from 2.3 to 2.77 (depending on the conference). Panel discussions tended to be rated lower than solo speakers who shared their story in-depth. According to the June 2010, evaluation report, all of the students who attended the conferences the preceding year reported they would recommend it to their friends. Results from observer and teacher feedback on evaluation forms indicated there was a high level of student engagement. Both students and teachers described the workshops that had hands-on activities as the most successful. According to the June 2010, evaluation report, “Very few ratings (of workshops) were below ‘good’, but for those that were, students explained that they were ‘boring’ or too much ‘lecture.’ Observers noted that students responded to enthusiastic presenters: those who talked about their backgrounds and personal journeys. Participating teachers reported they found the experience to be worthwhile: that students remembered many specific details about the workshops, especially those that focused on hands-on activities. Many students reported they enjoyed being in the settings were the conferences were offered. Commentary/Implications for Future Application: These conferences used many promising practices, especially for a one-time, day-long conference. Clearly, career information was provided in a variety of formats by several STEM professionals. Students had the opportunity to interact with several STEM professionals, and frequently were able to engage in some hands-on activity where the presenter explained and demonstrated how STEM skills were used in the “real world.” Program reports and evaluation results suggested that, in some cases, students were involved in brief, problem-solving exercises (such as building a tower). Teachers participated alongside students and were able to learn about the varied types of STEM career opportunities. Significant resources are available for others to replicate these programs in the future. Conference organizers developed a manual titled “Framework to Develop a Science/Engineering Career Awareness Conference” which provides information and directions on how to implement similar conferences. Many lessons have been learned during the implementation of these conferences over the years. According to recent reports, high student engagement during the conferences was associated with well-organized, hands-on activities and a pace that allowed time for questions and ample interaction. Organizers also developed materials to support volunteers in offering effective workshops. They developed a video of effective, engaging presentations which could be shown to presenters to help them see examples of interactive presentations for students. In addition to this video, conference organizers recommended that


38


Program Profiles

an event should be held for volunteer presenters prior to a conference to provide training on how to develop a presentation that is hands-on, engaging and age appropriate. According to program reports, conference organizers have discussed whether the approach to selecting keynote speakers for the conference should change depending on whether the program targets girls, boys, or both. Data did not show that the boys or girls reacted differently to the conferences. However, some organizers questioned whether some of the keynote speakers would be equally appealing to boys and girls. For example, the young woman who spoke at the 2010, Women in Science conference told of her journey. Some questioned whether boys would have been as interested in this presentation as the girls were. A number of steps could be implemented to help gather more specific information from student evaluations of the conferences. For example, if possible, conducting a survey on students’ career goals prior to the conference, and comparing results to a second, post-conference survey, would more accurately assess whether career goals and/or reported interest in STEM subjects had changed. Questions asking students to reflect on a few of the components of student interest (presented in the introduction of this report) would give more specific information on the aspects of student interest that could have been impacted. Use of a rating scale where students could indicate how strongly they agreed or disagreed with statements about the conference’s impact on their interest in STEM would also be helpful in assessing levels of interest. Finally, a follow-up survey, several months after the conferences would provide information on whether interest in STEM careers persisted.


39


Summary of Results and Conclusion

Summary of Results and Conclusion A Rich Variety of Programming The programs profiled in this report clearly demonstrate that the Pipeline Fund has supported a wide variety of programming across the state. Pipeline-supported programs have used multiple methods, have been implemented in a variety of settings, and have offered a range of experiences for students. Programs targeted students of all ages, from early elementary through high school. Programs involved students, teachers, parents and STEM professionals to varying degrees. Some were held in-school and others in afterschool, weekend or summer environments. Some, such as the BioTech Career internship program provided a lengthy, in-depth experience to a small number of students, while others like the DIGITs program reached thousands of students with a brief, one-time experience. Figure 1 (next page) provides an “at a glance” reference summarizing basic attributes of the programs as described in the profiles, including the program’s name, region where it was implemented, target population(s), grade level(s), average enrollment per activity, one-time or series, length, in- or out-of-school, evaluation method(s), and assessment results. Utilization of Promising Practices As might be expected, there was also a great variety in how promising practices were utilized by each program. Some programs employed a greater number of promising practices than others. For example, while nearly all programs provided STEM career information, not all provided opportunities for team work or small group collaboration. Similarly, some programs provided participating students with a more in-depth exposure to certain promising practices. For example, while nearly all programs related content to real-world applications, some programs provided this information verbally to students, some demonstrated the application and others involved students in a learning experience where they applied STEM skills to a real-world situation. There was no clear or easy way to quantify or measure the utilization of promising practices, except to state whether programs employed the practice, and describe the depth or nature of the experience. Pipeline Funded Programs: Was there an Effect on Student Interest? Individually, the data from many of these programs indicate that they may have had a positive effect on student interest. But given the variety of the programming, and the varying methods employed by local evaluators to assess them, it is difficult to make broad statements about the effect of the Pipeline Fund programming as a whole. Five of the thirteen programs profiled in this report demonstrated some evidence of measureable change in student interest. All of the programs profiled in this report provided some data - qualitative, and in some cases quantitative - that suggested their programs may have had an effect on student interest in STEM. While the data were not always measureable, the findings were generally positive. The Challenge of Demonstrating Effect in STEM Education Programming While it may initially seem discouraging that only five of the Pipeline programs demonstrated some evidence of measureable change in student interest, a review of the current literature in the field suggests that this is a common experience, particularly in informal STEM programming. As stated in the introduction to this report, the BEST Commission embarked on an effort to identify STEM programs for K-12 students that had evidence of effectiveness in reaching out to underrepresented populations. Over 200 programs across the country were nominated to be reviewed by the BEST Commission. Only thirty-four were eventually selected for review, based in part on whether they had a track record of data or other studies demonstrating their effect. Of these 34, only 20 had research-based or descriptive evidence of success beyond the anecdotal level. Only nine had at least one rigorous independent study demonstrating positive findings. Of the nine, not one was identified as “verifiably effective” (having five or more independent studies with positive findings). However, two were considered “probably effective” (with at least two rigorous independent studies with positive findings) and seven were considered “notable,” having at least one independent study with positive findings. UMass Donahue Institute Research and Evaluation Group

40



Figure 1: Summary of Basic Program Attributes Program Name

Lead Organization / Region

Target Population

Grade Level Targeted

Average Enrollment Per Activity

One Time or Series

Length

Environment

Evaluation Method

Student Interest Assessment Results

Biotech Career Pathways

Framingham State / MetroWest

Students

High School

10 -11 per internship

Series

19-30 hours

Out-of-school

Pre-post

Measureable difference identified

DIGITS

MA Technology Leadership Council / Statewide

Students

Middle School (6th grade)

Average class size

One-time

Average class period

In-school

Pre-post


Engineering is Elementary

Northern Essex Community College / Northeast

Students & Teachers

Elementary School (Grades 3-5)

Average class size

Series

Average class period

In-school

Pre-post


Got Math?

MA College of Liberal Arts/ Berkshire

Students primarily, Some Instructors

Elementary and Middle 12-25 students per (Grades 3-6) group

Series

1- 2 hours

Out-of-school

Pre-post


STEM Summer Camps

CONNECT Partnership/ Southeast

Students

Middle School

40 students per summer camp

Series

Five days

Out-of-school

Pre-post


Engineering the Future by Design

Andover Public Schools / Northeast, Boston & Central


Middle School

Average class size

Series

7-18 hours for students

In-school

Pre-post

No Measureable difference determined – qualitative data available

STEM Fellows and Leaders Project

UMass Lowell/ Northeast

Teachers

Mostly Middle, Some High School

45 teachers

Ongoing

90+ hours

Both

Pre-post

No measureable difference determined qualitative data available

STEM RAYS (Science, Technology, Engineering and Mathematics Research Academies for Young Scientists)

UMass Amherst / Pioneer Valley


Elementary and Middle School (Grades 4-8)

8-12 students per club

Series

2 hours per week for entire school year.

Out-of-school

Pre-post

No measureable difference determined qualitative data available

Berkshire STEM Career Fair

MA College of Liberal Arts/ Berkshire

Students

High School

350 students

One-time

Two 2-hour sessions

Off-site, during school day

No measureable data Post-only and/ or collected - Qualitative data qualitative assessments available

Family Science Programs

MA College of Liberal Arts / Berkshire

Students primarily, Some Parents/ Guardians & Instructors

Middle School

12 - 25 students per program

Both onetime and series

3 hours on average

Both


Massachusetts Region I Middle and High School Science Fairs

MA College of Liberal Arts/ Western MA

Students

Middle and High School

100-195 students per fair

One-time

8 hours on average plus prep time.

Out-of-school


Saturday STEM Academy Middle School Project

Framingham State / MetroWest

Students

Middle school (8th grade)

20-31 per session

Series

24 hours (4 sixhour sessions)

Out-of-school


STEM Central Career Awareness Conference

UMass Medical School / Central

Students

Middle School

65 – 200 per conference

One-time

6 hours

Out-of-school



41



Evaluation of the Pipeline programs both individually and as a whole mirror how evaluation is conducted across the country for similar programs. The programs profiled in this report used varied approaches to affect student interest, and as a result, varied methods were used to capture and measure potential change in interest. The PEAR study (cited in the Introduction) summarized some of the challenges in evaluating STEM programming, particularly for Outof-School time (OST) science education programming: To date, many of the studies that show promising results have used “homegrown” assessment tools to demonstrate impact. Although this practice has its benefits (e.g., instruments relate directly to a specific program), it results in two challenges. First, the use of program-byprogram assessments calls into question the validity and/or reliability of the studies (since they traditionally lack norms, psychometric properties, and peer-reviewed reports), and this can render them less persuasive in the eyes of researchers, funders and policymakers. Second, because many programs and program evaluators create their own tools, it is difficult to compare or summarize results across programs or evaluators. As a consequence, there is very little comparative data available to support the claim that OST science programming is effective, or to support best practices for the field.

16

Assessing student interest in STEM adds another dimension to the challenge of evaluating STEM programming. Student interest in STEM is increasingly being recognized as an essential component to increasing the number of students who major in STEM fields in college and engage in STEM careers. 17 Ironically, the science of research on whether particular programs are “evidence-based” and shown to increase student interest in STEM subjects is still in its infancy. Research assessing content knowledge and achievement is much more common and data on competence or achievement in math or science is readily available through numerous standardized tests such as the Scholastic Aptitude Test (SAT), the National Assessment of Educational Progress (NAEP) tests, and the Massachusetts Comprehensive Assessment System exams. A review of National Science Foundation funded programs (cited in the Introduction) found that many OST programs were limiting their evaluations primarily to measures of achievement and content-knowledge. The reviewers encouraged a broader approach to evaluation, including assessing student interest. The PEAR study noted that even when programs sought to assess interest, the instruments used were often not appropriate for the elementary and/or middle school aged participants: ”The majority of measures assessing attitude and interest were intended for high 18 school and college students.” Meeting Student Interest Evaluation and Assessment Challenges The same variety that makes it difficult to make broad statements about the influence of STEM programming on student interest in Massachusetts may also be the state’s strength. No single program will ever be able to meet the needs of all students when it comes to encouraging student interest in STEM. However, the range of Pipeline programs offered throughout the state made it possible for students of all ages and levels of interest to participate in local programming appropriate to their age, ability and interest level. Those who implement STEM programs in the future have a range of programs and evaluation methods to draw upon as potential models. Below are recommendations to strengthen and support implementation and assessment of student STEM interest programs. These recommendations, which represent a synthesis of recent literature, could benefit future programs, both in Massachusetts and elsewhere: 1. Compile a list of tools to assess STEM programming for future programs to draw upon. This report provides examples of instruments and questions that were used by the programs to assess student interest. Linking program developers to other existing sources of instruments commonly used in the field could also be helpful. The National Science Foundation and others have supported the creation of such compilations.

16

Hussar, K., Schwartz., Boiselle, E., & Noam, G. (2008). Toward a Systematic Evidence-base for Science in Out-of-School Time: The Role of Assessment. Retrieved from Harvard University and McLean Hospital, Program in Education, Afterschool & Resiliency website: http://www.pearweb.org/pdfs/Noyce-Foundation-Report-ATIS.pdf. Page 8. 17 Tai, R. H., Liu, C. Q., Maltese, A. V., & Fan, X. (2006). Planning early for careers in science. Science, Vol 312, 26 May, 2006. 18

Hussar, K., Schwartz., Boiselle, E., & Noam, G. (2008). Toward a Systematic Evidence-base for Science in Out-of-School Time: The Role of Assessment. Retrieved from Harvard University and McLean Hospital, Program in Education, Afterschool & Resiliency website: http://www.pearweb.org/pdfs/Noyce-Foundation-Report-ATIS.pdf. Page 51. UMass Donahue Institute Research and Evaluation Group

42



2. Standardize a small number of student interest evaluation questions to ask across all programs, even if program designs vary. Massachusetts could encourage the use of identical, age-appropriate questions to be asked of students who participate in STEM programming. This would make it easier to assess the effect of Massachusetts’ programming as a whole and to compare program outcomes. 3. Consider the creation of a larger mechanism for sharing information on STEM program development and evaluation. Programs could benefit by communicating with others implementing STEM programming. Sharing resources, ideas, methods, lessons learned and evaluation strategies will reduce duplication of effort. Ideally such a mechanism would also enable programs in Massachusetts to access current research information and assessment resources outside of the state. After reviewing the first recommendation, it became clear that a compilation of the student interest assessment measures used by the Pipeline programs in this report might benefit future programming and evaluation efforts. This would provide an array of survey questions and items that could be drawn upon to assess student interest in the future. As a result, copies of instruments and sample questions for each program are included in Appendix B. While these instruments and items have not been tested for reliability or validity, they may still serve as helpful resources. The Massachusetts Governor’s STEM Advisory Council, its Operations Board, and subcommittees may wish to consider the second recommendation and take interim steps to standardize some evaluation questions that could be asked across all programs regarding student interest, even if program designs vary. The development and use of ageappropriate questions for elementary and middle school students would be particularly useful. Others in the field are currently in the process of implementing the third recommendation: developing tools or instruments that could be used across the country. The National Science Foundation’s Math and Science Partnership – Motivational Assessment Program at the University of Michigan is in the midst of reviewing and assessing available tools and will make recommendations for use in the future. While awaiting their recommendations, the PEAR’s compilation of nationally used instruments with questions assessing student interest and other STEM-related domains can serve as a resource for assessments of STEM programming. It is our hope that this report will be a first step in following through on the above three recommendations by providing shared information to support future STEM programming and evaluation.


43


Appendix A

Appendix A: Detailed Explanation of Program Headings Program Name: This is the name of the program at the time it was conducted. In some cases, lead organizations have updated or changed the names of the programs over time. Lead Organization/Region: This is the organization that was awarded funding by the Pipeline Fund and that held primary responsibility for reporting on the program’s accomplishments and outcomes. In several cases, multiple organizations were involved in program implementation. Region is the geographic area served by the project. Funding/Duration: States which Pipeline funding stream supported the program, either the grant titled “A Grant Opportunity for 2007 Regional PreK-16 Networks to Advance Interest and Learning in Science, Technology, Engineering and Mathematics” referred to as “Network” funding or the grant titled “Enhancing Student Interest and Retention in STEM Fields,” referred to as “Student Interest” funding. Duration provides the start and end year for when the program received funds. Program Overview: A brief summary of the program’s design and purpose, as gathered from the program’s initial application to the Pipeline Fund and as stated in program reports. Target Population/Reach: This section includes a description of the target population that the program initially intended to reach, as well as a description of the actual participants as described in program reports. In some cases programs initially intended to reach out to a particular group or demographic but ended up implementing the program with a somewhat different group. The reasons for these changes are often described as they can yield useful insights into the experience of the program coordinators. Demographic information, if available, is presented. This section also includes the grade level(s) and total number of students and teachers reached over the last year, along with the average number that participated per program. One Time/Series: Some programs were designed to be implemented once, such as a one-time career fair or conference, while others were intended to be offered as a series of sessions over time. Some programs such a summer camps or after-school science clubs met repeatedly over a week or more. Others, such as those integrated into school curricula, may have met repeatedly over a school year. The number of sessions offered in a series, if available, is reported in the profile. Program Length: The amount of time used for each session and total hours overall for the program is presented in the profiles when available. The length of the program and its frequency (whether it was offered once or over several occasions) together suggest the depth of the nature of the experience for students, and may also reflect the level of resources involved in program implementation. In/Out of School: Whether the program was held in an in-school our out-of-school environment. Recently, there has been an increasing interest in Massachusetts for STEM programming for out-of-school settings. Stakeholder Involvement: Sustained stakeholder support and involvement was identified as one of the common features of successful programs by the BEST Commission. While the level or length of stakeholder involvement could not be easily assessed, information on stakeholder involvement as presented in program reports is summarized. Evaluation Method: The evaluation methods that were used by the local evaluator are summarized in every profile. These summaries include a description of the types of instruments used, the questions asked, and whether the information was gathered before and after programming. A description of all efforts to evaluate the program is provided, including efforts to collect data on student interest and as well as other evaluation efforts. Copies or examples of instruments or assessment questions used are included in Appendix B for reference.


44


Appendix A

Evaluation Results/Interest in STEM: All evaluation data was screened to determine whether they fell under the domains and components of STEM Interest as outlined in the logic model in Figure 1 (Page 7). Any program evaluation data that did not fall within this domain were summarized and presented under “Additional Evaluation Results” as described below. A summary of the data collected on student interest is provided first, along with statements on whether the information collected indicates a measureable change in student interest or not. Any nonmeasureable or qualitative information on student interest that was collected was also presented here. Additional Evaluation Results: Typically, many methods were used to evaluate the effect of programs. Often programs collected information on the effect of their efforts beyond student interest in STEM. Student knowledge or competence in STEM subjects was sometimes assessed through tests. Teachers, parents or other adults who participated in initiatives were also often asked for their feedback and reaction to the programming. When programs offered additional evaluation data beyond assessments of student interest in STEM, the information has been summarized in each profile. Commentary/Implications for Future Application: Each profile concludes with commentary on its utilization of promising practices, including a brief statement on which practices were used by the program according to program reports. A review of any lessons learned or insights offered by the program directors with implications for program replication in other sites is also provided. Finally, any clear suggestions for future program implementation or opportunities for improved evaluation methods that became apparent through this review are also provided.


45


Appendix B

Appendix B: Examples of Questions and/or Instruments Biotech Career Pathways Attitudinal Surveys for Juniors and Seniors (Note: This instrument was given to students before and after participating.)

On a scale of 1-4, with 1 being “low level”, 2 being “average”, 3 being “moderate” and 4 being “high level”. 1. What is your level of awareness about careers in research, biotechnology and research animal management? 2. What is your level of awareness about the job market and employment opportunities in research, biotechnology and research animal management? 3. What is your level of interest in careers in research, biotechnology and research animal management? 4. How motivated are you in pursuing a career in research, biotechnology and research animal management?


46

Appendix B


DIGITS Pre and Post Survey Information (Note: The questions below were inferred from a review of the local evaluator’s report and represent a sample of the questions used. This is not a complete representation of the actual instrument that was used.) Students were asked to indicate how strongly they agreed or disagreed with the following statements:

Math is fun.

Strongly disagree

Disagree

Agree

Strongly agree

Science is fun.

Strongly disagree

Disagree

Agree

Strongly agree

Technology is fun.

Strongly disagree

Disagree

Agree

Strongly agree

Math is interesting.

Strongly disagree

Disagree

Agree

Strongly agree

Science is interesting.

Strongly disagree

Disagree

Agree

Strongly agree

Technology is interesting.

Strongly disagree

Disagree

Agree

Strongly agree

If given a choice, I will take an Algebra class.

Strongly disagree

Disagree

Agree

Strongly agree

People who take Algebra I Strongly disagree and Algebra II are more likely to get higher paying jobs.

Disagree

Agree

Strongly agree

Students were also asked about their level of interest in participating in the following clubs: Science Fair

Chess club

Computer club

Math club

Please list jobs that involve math: ______________________________________________________________ Please list jobs that involve science: ____________________________________________________________

Students were also asked whether they viewed STEM jobs as: Fun

High paying

Flexible

Easy

Exciting

Creative

Challenging

Team-work


Make the world a better place

47

Appendix B


Sustainability Camp STEM Pipeline Student Interest Grant Evaluation Survey Questions 1. Gender:

Male

Female

2. Grade & School you will attend in September: 3. After participating in this camp, please rate your knowledge about Sustainability and the Environment: Very High

High

Some

A Little

4. After participating in this camp, which word BEST describes your feelings about studying science? (Choose only ONE) Fun Challenging Hard Boring 5. List THREE things you think of when you hear the word scientist or architect: 6. During this camp, how much have you learned about these science topics? TOPIC Ecology/Environmental Science Renewable Energy Sources Carbon Cycle How you can help protect our planet Natural foods & Products “Green” building/technology Human Impact on Environment

5 = A Lot

3 = Some

1 = Little or None

7. During this camp, how would you rate your experience with these science skills? SKILL Working in teams Conducting Experiments Collecting & Analyzing Data

5 = A Lot

3 = Some

1 = Little or None

8. After attending this camp, are you interested in learning more about SUSTAINABILITY and the ENVIRONMENT? Yes No 9. The BEST part about this camp was:

10. My least favorite part about this camp was:

11. Would you recommend this camp to a friend? _______


Why or why not? __________

48

Appendix B


Got Math? Elementary School Participant Survey Name: _______________________________ Grade: ______

School: ___________________

1. How do you feel about math? I like math. Math is ok. I don’t like math. 2. How good are you at math? Very good Pretty good Ok Pretty bad Very bad 3. If my school had a math club, I would participate. Yes Maybe No 4. I enjoy doing math puzzles and games. A lot Sometimes Not Really Never have

(Note: The following questions were only included in the post-program survey.) 5. Did you enjoy Got Math? Always Usually Sometimes Not Really 6. After participating in Got Math my math skills: Improved a lot Improved a little Stayed the same Got worse 7. The best thing about Got Math was ____________________________________________________________ _____________________________________________________________________________________________ 8. The worst thing about Got Math was ___________________________________________________________ _____________________________________________________________________________________________


49

Appendix B


Engineering the Future by Design Student Survey Your INITIALS __________

Circle:

School ___________________

Male / Female

Grade: 5 6 7 8

Date ___________________

Please check any of the activities that you have attended: Summer Engineering Camp at WPI

Women in Technology after school program

Women in Science

Robotics Program

Men in Science and Technology

Design Camp at ULowell

Engineering Camp

How Things Work

Which subject do you like better? Circle one of each pair. (You may like both, but tell us which one you’d rather spend the morning doing.) Multiplication or Foreign Language

Music or Chemistry

Social Studies or Weights and Measures

Computer Programming or Chorus

Botany (plants) or Foreign Languages

Reading or Addition and Subtraction

Electricity or Creative Writing

Biology or Spelling

Division or Health

Learning About History or Writing Software

Estimating or Vocabulary

Building Circuits or Reading

Geography (maps) or Physics

Experimenting or Studying People and Cultures

Building Computers or Painting

Geometry or Learning about History

Algebra or Writing

Problem Solving or Writing Book Reports


50

Appendix B


Engineering the Future by Design Student Survey continued…

Designing Things or Playing Music

Writing or Astronomy (Planets)

Creating Models or Studying Literature

Word Problems or Health Class

Creating Art or Testing Ideas

Taking Things Apart or Independent Reading

Building Things or Spelling

Foreign Language or Geology (Rocks)

Circle the kind of activity you like better: Independent challenges

or

Team Challenges

Problems with one correct answer

or

Problems with more than one solution

Problems with instructions to follow

or

Experiments

What’s the best way to learn about something? (write your answer in the space below)

If you had the choice, how likely would it be that you would sign up to take MORE Engineering courses/classes in school? (choose one of the options below)

__ Definitely! ___ Good Chance

____Maybe

____Probably Not ______ No Way!

Thank You for Your Time!


51

Appendix B


STEM Fellows Information about the STEM Fellows Student Survey (pre and post) 1. Demographic information Are you a boy or a girl?

A. B.

Boy Girl

What grade are you in? How old are you? What is your teacher's last name?

2. Relevance and Importance of school work

How true are each of the following statements? School is important for my future My education will create many future opportunities for me School work is important to complete Going to college is important to me Going to college is important to my family What I'm learning in my classes will be important to my future

My parents think it is important for me to Do well in mathematics at school Do well in science at school Do well in technology at school My friends think it is important to Do well in mathematics at school Do well in science at school Do well in technology at school I think it is important to Do well in mathematics at school Do well in science at school Do well in technology at school


Answer Choices were: Not true at all Kind of true True Very true

Answer Choices were: Agree a lot Agree Not sure Disagree Disagree a lot

52

Appendix B


STEM Fellows Survey continued… 3. Attitude Towards Science How much do you agree with these statements about learning science? I usually do well in science I would like to take more science in school Science is more difficult for me than for many of my classmates I enjoy learning science Science is not one of my strengths I learn things quickly in science


How much do you agree with these statements about science? I think learning science will help me in my daily life I need science to learn other school subjects I need to do well in science to get into the university or college of my choice I would like a job that involves using science I need to do well in science to get the job I want

How often do you do these things in your science lessons? Apply what we are learning about science to our daily lives Talk about the types of jobs and careers that use science

Answer Choices were: Every or almost every lesson About half the lessons Some lessons Never

4. Attitude Toward Mathematics How much do you agree with these statements about learning mathematics? I usually do well in math I would like to take more math in school Math is more difficult for me than for many of my classmates I enjoy learning math Math is not one of my strengths I learn things quickly in math How much do you agree with these statements about mathematics? I think learning math will help me in my daily life I need math to learn other school subjects I need to do well in math to get into the university or college of my choice I would like a job that involves using math I need to do well in math to get the job I want

How often do you do these things in your math lessons? Apply what we are learning in math to our daily lives Talk about the types of jobs and careers that use math




53

Appendix B


STEM Fellows Survey continued… 5. Attitude Toward Technology How much do you agree with these statements about learning technology? I usually do well in technology I would like to take more technology in school Technology is more difficult for me than for many of my classmates I enjoy learning technology Technology is not one of my strengths I learn things quickly in technology


How much do you agree with these statements about technology? I think learning technology will help me in my daily life I need technology to learn other school subjects I need to do well in technology to get into the university or college of my choice I would like a job that involves using technology I need to do well in technology to get the job I want

How often do you do these things in your technology lessons? Apply what we are learning about technology to our daily lives Talk about the types of jobs and careers that use technology


6. Knowledge about STEM Career Options In your school do you take classes in engineering or study the design process? Notions about people in STEM careers: (samples follow – example is not complete) A scientist usually works alone Scientific work is dangerous Scientific researchers are dedicated people who work for the good of humanity Scientists don’t get as much fun out of life as other people do Scientists are helping to solve challenging problems Scientists are apt to be odd and peculiar people Most scientists want to work on things that will make life better for the average person Scientists have few other interests but their work


The above questions were also asked regarding “an engineer” and “a person who works in the field of technology.” Do you know an engineer? Do you know a mathematician? Do you know a scientist?


54

Appendix B


STEM RAYS After-School Science Program Pre- and Post-Assessment Survey Name 1. Who uses science in their work?

A. B. C. D. E.

A nurse A crime scene laboratory expert A weather reporter None of them All of them

2. Who uses engineering in their work?

A. B. C. D. E.

A bridge designer An astronaut A building inspector None of them All of them

3. Scientific theories are explanations and not facts.

A. B.

True False

4. Scientists sometimes try to disprove their own ideas. A. B.

True False

5. Scientific theories

A. B. C. D. E.

Are never wrong Have to be accepted if experts like them Are useful if they explain a lot of experiments Cannot be tested Don’t change

6. Scientific discoveries come from

A. B. C. D. E.

Science books Television The Internet Observations from the real world All of these

7. Engineering mostly deals with

A. B. C. D.

Trying to understand how the world works Making useful things The same things as science None of these is correct.

8. Science mostly deals with

A. B. C. D.

Trying to understand how the world works Making useful things The same things as engineering None of these is correct.

The pre- and post-test asked the students to complete the following: “When I am 25 years old I would like to be working as a …” The responses were coded into categories of types of careers, and then coded according to whether the responses indicated a STEM or Non-STEM career.


55

Appendix B


Berkshire STEM Career Fair Student Survey Thank you for taking the time to complete this survey! Your answers are confidential. On a scale of one to five (one is “very bad” and five is “very good”), how would you rate your experience at the STEM Career Fair? 1

2

3

4

Please indicate how much you agree or disagree with the following statements: Strongly Agree Not Sure Disagree Agree

5

Strongly Disagree

The Career Fair made me realize that a math, science, and technology career could be interesting. At the Career Fair, I learned about career opportunities in Berkshire County that I didn’t know about before. After this Career Fair, I'm thinking about going into a high-tech career more than I was before. The Career Fair helped me realize that the math and science I take in high school will affect my career options. I'm planning to take more math and science courses in high school than I was before. I'm planning to work harder in my math and science courses. Are you interested in the following? (Check all that apply. Please contact your guidance counselor, or internship coordinator to see which of these are already available through your school.)

 Hanging out for a few hours with an employee at his or her workplace  Touring a local employer  Participating in an internship one or more days a week at an area employer  Earning high school credit for learning a trade at a company  More information about salaries in high-tech jobs  Finding out more about jobs in math, science and technical fields  Pursuing an in-depth career-oriented project After high school I plan to go to:

 Four-year college  Two-year college  Technical, business or trade school  Apprenticeship  Find a full-time job  Don't know  Other: UMass Donahue Institute Research and Evaluation Group

56

Appendix B


Berkshire STEM Career Fair Survey continued… Gender:

 Male  Female Race:

 White  African American  Native American  Asian  Hispanic  Native Hawaiian, Pacific Islander Grade:

 11  12 On your report card, do you get mostly?

 A's & B's  B's & C's  C's & below THANK YOU for completing the survey!


57

Appendix B


Berkshire Family Science Program Student Questionnaire (Post-Program) 1. Before participating in this Family Science Program had you thought at all about the science classes you would like to take in high school?  Yes  No (skip to Question # 4 ) 2. Before participating in the Family Science program, how many science courses did you plan to take in high school?  I did not plan take science in high school  Just enough science courses to graduate  Enough science courses to get into college  As many science courses as I could 3. After participating in the Family Science Program do you plan to take…  Less science in high school than I planned to before  More science in high school than I planned to before  My plans have not changed 4. How do you feel about your science classes in school? Think about last year and this when you answer.  I generally don’t find them interesting  They are okay  I really like my science classes 5. I think science classes in high school will be  Less interesting than the science classes I have taken so far  About as interesting as the science classes I have taken so far  More interesting than the science classes I have taken so far 6. Before participating in this in Family Science program, I would describe my interest in having a career or job in a scientific field as…  Not much interest  Some interest  A lot of interest 7. After participating in the Family Science Program my interest in having a career in science has…  Decreased  Stayed the same  Increased


58

Appendix B


Berkshire Family Science Program Survey continued…

8. After participating in the Family Science Program, do you know more about all the different things scientists do than you knew before?  No  Yes  Not sure 9. Has participating in the Family Science program has changed your idea of what scientists are like?  No  Yes  Not sure 10. How interested are you in participating in another Family Science Program in the future?  Not interested  Somewhat interested  Very interested 11. Are you…  Male  Female 12. Are you…  White  African American  Native American  Asian  Hawaiian or Pacific Islander 13. Are you …  Hispanic  Not Hispanic 14. What grade are you in? ________ 15. What school do you attend? ___________________________________

Please rate the Family Science Program you have just completed: Poor

Fair

Good

Great

The program’s topic The instructor(s) The activities you did The room or physical space in which the program was held The amount of science you learned

Please share any other comments that would help us improve the program:


59


Appendix B

MA Region I High School Science and Engineering Fair Student Survey 2010 1. What grade are you in?  9  10  11  12 2. Is your science fair project an individual project or a group project?  Individual  Group 3. Was participation in the science fair a requirement for one of your science classes?  No  Yes 4.     

Did you prepare for the science fair… Mostly at home Mostly during the school day Mostly after school (using school facilities) Equal combination of home and school (either in school or after school) Equal combination of in school and after school

5. How would you describe the assistance, if any, you received from teachers at your school as you prepared you project?  None  Very Little  Some  Quite a bit  A lot 6. Did you consult with any local businesses, or individuals from businesses to help you prepare your project?  No  Yes 7. Did you consult with any local businesses, or individuals from business to help you prepare your project?  No  Yes 8. Did you participate in a science fair at your school this year prior to coming to the Region I Science Fair today?  No UMass Donahue Institute Research and Evaluation Group

60




Appendix B

Yes

9. How challenging was it for you to complete your project?  Not really challenging  A bit challenging  Somewhat challenging  Quite challenging  Extremely challenging 10. What obstacles, if any, did you encounter in doing your project? (check all that apply)       

Finding the time to do the work Getting adequate input from my mentor or advisor Getting the equipment/materials I needed Learning the science I needed to complete the project Making the project “work” Having a place to work on the project Other: ____________________________________________

11. Below is a list of the possible benefits of participating in the science fair. Please rate the importance of each in your experience this year (Set up as table with the following response categories: Not important, somewhat important, very important)         

Learning science that I didn’t know before Getting to do hands-on science work Working closely with a teacher or mentor Working with other students Seeing other students’ work at the Regional Fair Having others see my project Building my resume for college The chance to move on to the state competition Building my confidence in my science abilities

12. Please list any other personal benefits you received. (Free text response)

13. How would you describe the organization of today’s Region I Science Fair   

Not very well organized Pretty well organized Very well organized


61


Appendix B

14. What one or two changes would you make to improve the organization of today’s Science Fair? (Free text response) 15. Would you like to participate in the Region I Science Fair again next year?  No  Maybe  Yes 16. Are you planning to major in math, science, technology, or engineering in college?  No  Maybe  Yes, definitely 17. Are you considering a career in math, science, technology, or engineering ?  No  Possibly  Yes, definitely 18. If you could give advice to another student who is interested in participating in a science fair, what would you tell them?


62

Appendix B


Saturday STEM Academy Survey Questions 1. Engineering is a very important profession. Strongly Disagree

Disagree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Agree

Strongly Agree

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

Not Sure

Agree

Strongly Agree

2. Engineers do lots of different and interesting things. Strongly Disagree

Disagree

3. Being an engineer can be exciting. Strongly Disagree

Disagree

4. Engineering is a good career for both minorities and women. Strongly Disagree

Disagree

5. Engineers really enjoy their work. Strongly Disagree

Disagree

6. Engineers have fun in their jobs. Strongly Disagree

Disagree

7. Engineers are very well paid in their jobs. Strongly Disagree

Disagree

8. Going to an engineering college can be enjoyable and exciting. Strongly Disagree

Disagree

Not Sure

9. Knowing math and science is important for me to be successful in life. Strongly Disagree

Disagree

Not Sure

10. Engineers get to do many of the things that I like to do. Strongly Disagree

Disagree

11. Engineering is something I can be good at. Strongly Disagree

Disagree

12. I would like to become an engineer. Strongly Disagree

Disagree

13. I would like to go to an engineering college after high school.


63

Appendix B


Strongly Disagree

Disagree Not Sure Agree Saturday STEM Academy Survey continued…

Strongly Agree

14. I am now more motivated to study math and science in high school. Strongly Disagree

Disagree

Not Sure

Agree

Strongly Agree

Agree

Strongly Agree

15. I am now more motivated to prepare myself to got to college. Strongly Disagree

Disagree

Not Sure

16. What do you like most about what engineers do: Building Things Solving Problems Using Math & Science Working in Teams Making things work Always learn new things Getting paid well Getting to travel Have fun Money Helping people Making new things


64

Appendix B


Men in STEM Conference Student Evaluation Form Let us know what you think! Please rate each aspect of the conference. Workshop

5= Excellent

4=Very Good

3 =Good

2=Fair

1=Poor

Title: Nursing

5

4

3

2

1

Title: Science Exploration

5

4

3

2

1

Title: Robotics

5

4

3

2

1

Title: Engineering Design

5

4

3

2

1

Title: Computers

5

4

3

2

1

Title: CSI

5

4

3

2

1

Lunch

5

4

3

2

1

Keynote Panel

5

4

3

2

1

Tour on Intel

5

4

3

2

1

Please finish the following phrases: My favorite part was…

The worst part…

If I had to describe this conference to another, I would tell him… I never knew…

Now that I’ve participated in this conference, my career goals have changed/not changed because…


65