Auton Robot (2011) 30: 265–280 DOI 10.1007/s10514-011-9218-3
Reflections on iCODE: using web technology and hands-on projects to engage urban youth in computer science and engineering Fred G. Martin · Michelle Scribner-MacLean · Sam Christy · Ivan Rudnicki · Rucha Londhe · Colleen Manning · Irene F. Goodman
Received: 11 December 2009 / Accepted: 6 January 2011 / Published online: 26 January 2011 © Springer Science+Business Media, LLC 2011
Abstract More than 200 middle school and high school students from underserved urban communities in Boston, Lowell, and Lawrence, Massachusetts, participated in afterschool and summer enrichment programs over a three-year period, using hands-on learning materials and web resources to complete hands-on microcontroller-based projects. Program content was based on a suite of robotics and electronics kits developed by the University of Massachusetts Lowell and Machine Science Inc., together with on-line instructions, a web-based programming tool, and a shared electronic portfolio of student projects. Participating students worked with classroom teachers and undergraduate mentors
This material is based upon work supported by the National Science Foundation under Grant Numbers DRL-0624669 and DRL-0624631. F.G. Martin () · M. Scribner-MacLean University of Massachusetts Lowell, Lowell, USA e-mail:
[email protected] M. Scribner-MacLean e-mail:
[email protected] S. Christy · I. Rudnicki Machine Science Inc., Cambridge, USA S. Christy e-mail:
[email protected] I. Rudnicki e-mail:
[email protected] R. Londhe · C. Manning · I.F. Goodman Goodman Research Group, Inc., Cambridge, USA R. Londhe e-mail:
[email protected] C. Manning e-mail:
[email protected] I.F. Goodman e-mail:
[email protected]
to complete a series of projects, and took part each year in a non-competitive robotics exhibition and a competitive robot sumo tournament. Goodman Research Group assessed learning outcomes and attitudinal changes using a variety of measures, including observations of program sessions, group interviews with participating students, pre- and postprogram student surveys, and educator feedback. The program was found to effectively engage participants, give them real engineering and programming skills, improve their attitudes toward science, technology, engineering, and mathematics (STEM) subjects, and increase their interest in STEM career pathways. These results are presented, along with lessons learned from the program implementation, technology development, and evaluation. Keywords Robotics · Education · K-12 · Informal · After school · Microcontroller · Programming · Logo · Sensors · Crafts · Evaluation · Competition · Career · Computer science · Engineering
1 Introduction From 2006 to 2009, the University of Massachusetts Lowell (UML) and a non-profit partner, Machine Science Inc. of Cambridge, Massachusetts, put technology on the web to support the growth of a community of young people using tangible, microcontroller-based projects to learn about science, technology, engineering, and mathematics (STEM). This work, titled Building an Internet Community of Design Engineers (iCODE), was funded by a three-year grant from the National Science Foundation’s (NSF’s) Information Technology Experiences for Students and Teachers (ITEST) program.
266
Auton Robot (2011) 30: 265–280
– Student responses to the iCODE project’s two tangible electronic hardware platforms: one a printed circuit board, the other a built-it-yourself, breadboard-based system. – A discussion of our community-building work, including on-line and face-to-face components. – Student learning outcomes and educator feedback, based on the three years of analysis by an external project evaluator.
2 Background
Fig. 1 iCODE program participants
Working with an open-source course management platform, the iCODE team developed a web system (www. icodeproject.org) offering dozens of illustrated design challenges, interactive quizzes, and other instructional materials. The system features a unique web-based programming tool—a Java applet that enables users to develop code for microcontrollers in a web browser, without any locally installed software. The two organizations also developed a suite of physical hardware, including electronics and robotics kits, to work in conjunction with the iCODE web site. Using the iCODE web system and tangible learning materials, the project partners facilitated year-long enrichment programs for students from local schools and community centers during the 2006–2007, 2007–2008, and 2008–2009 academic years. Comprising weekly after-school sessions, weekend robotics exhibitions and competitions, and intensive summer camps, the programs were effective in engaging the project’s inner-city youth population, teaching them computer programming and STEM workforce skills, and enhancing students’ interest in STEM subjects and careers. Figure 1 shows participants in the program. This paper describes the iCODE project in detail and presents lessons learned from its implementation, addressing the following topics: – Challenges associated with retaining middle school and high school students in the enrichment program, during the course of each year and from year-to-year. – Issues associated with supporting program leaders in different venues, including both school sites and neighborhood community centers. – The use of web technology to deliver instructional materials, enable programming of microcontroller projects, and build a community of users.
The use of robotics in education has become widespread in informal settings, with a blossoming of robotics kits and related K-12 curriculum materials over the last decade. Foundational work included Seymour Papert’s introduction of the robot turtle in his implementations of the Logo programming language (Papert 1980). This line of work led to commercial implementations of robotics materials launched by the LEGO Group (Martin and Resnick 1993; Martin et al. 2000) and numerous follow-on products. At the university level, robotics competitions are now common. Early instances, such as the IEEE Micromouse contest, have led to a wide range of national and international contests, such as RoboCup, the Intelligent Ground Vehicles Competition, and the Trinity College Home Firefighting Robot Competition. The competition format has migrated back to the K-12 learning environment, with examples such as RoboCupJunior (Sklar et al. 2003), FIRST Robotics, FIRST LEGO League, and Botball. Over the past few years, the growing popularity of web sites featuring user-generated content has led to the formation of “maker” communities—hobbyists who create electronics and crafts projects for fun, and share them with each other at sites like http://instructables.com. At the same time, the decreased cost of microcontroller hardware and related electronic components, coupled with the increased availability of free compilers and other software tools, have prompted a surge in user-inspired microcontroller-based projects. This merging of craft and computing is a rich area of study, and was underway even before the Web 2.0 phenomenon gave it a boost (Eisenberg and Eisenberg 1998). The iCODE project was inspired by these trends and drew on the experience and respective strengths of the partnering organizations. Prior to devising iCODE, co-author Martin conducted work in contemporary educational robotics, contributing to in-school implementations and university competition formats (Martin 1996). More recently, Martin developed the Super Cricket and promoted its use in K12 outreach programs at UML, including the university’s highly successful summer DesignCamp program (Baker 2009).
Auton Robot (2011) 30: 265–280
Co-author Christy served as manager of the first Computer Clubhouse, a network of community centers for underserved youth focusing on developing creative technical skills (Kafai et al. 2009). In 2001, Christy founded Machine Science—a non-profit that operated year-long after-school engineering programs for underserved Boston youth, using a breadboard-based microcontroller development platform. Previously, Machine Science had also developed a number of innovative web-based resources to support these programs, including both on-line design challenges and the online code development environment.
3 Project goals The iCODE project was one of more than 130 projects that NSF has funded under the ITEST grant program, which was launched in 2003 in response to concerns at the time about the availability and skills of the domestic information technology and STEM workforce. As of 2009, NSF had disbursed a total of more than $140 million through ITEST, supporting technology development, program implementation, educational research, and scale-up of effective models. Having received its initial funding in 2006, iCODE was in the fourth cohort of funded projects. It was among 22% of ITEST projects with a primary focus on engineering. Others have focused on environmental science (30%), computer programming (28%), computer gaming and simulation (10%), and bioscience (9%). The iCODE project’s overall objective was to increase the likelihood that participating middle school and high school students would pursue IT and STEM careers later in life. Within that context, the project had the following specific goals: – Goal #1: Enhance participating students’ information technology fluency. Throughout the year, students completed a range of IT-intensive microcontroller-based projects, requiring real engineering and computer programming skills. – Goal #2: Increase awareness among participating students about educational and career opportunities in IT and STEM. Students visited university campuses during the iCODE summer sessions and met with IT professionals at biannual IT and STEM career events. – Goal #3: Connect participating students to a community of like-minded peers and adults. Participating students worked side by side with undergraduate mentors in the year-long iCODE after-school programs. Participating students were also encouraged to create on-line portfolios of projects within an existing web database of inventions. Table 1 presents the underlying theory of change for each project goal.
267
4 Program structure The iCODE enrichment program was designed to meet guidelines established by the ITEST grant program, which mandated a minimum of 120 hours of contact time for each participating student, including a summer component lasting at least two weeks. In keeping with these guidelines, the iCODE experience encompassed the following elements: – After-school sessions. Every participating school and community center offered weekly after-school sessions, led by a teacher or staff member, with assistance from an undergraduate mentor, using the design challenges and other resources available on the iCODE web system (24 sessions @ 2 hours = 48 hours). – Weekend events. In each grant year, iCODE students participated in UML’s Botfest, a non-competitive exhibition on the UML campus, and Machine Science’s Robot Sumo Tournament, an autonomous robotics competition held at the Museum of Science Boston (2 events @ 6 hours = 12 hours). – Summer camps. The iCODE experience culminated in two-week summer camps at the University of Massachusetts campuses in Boston and Lowell. The camps gave students the opportunity to take their computer programming skills and creative abilities to the next level, completing collaborative projects of their own design (10 days @ 6 hours = 60 hours). Within this framework, UML and Machine Science offered two separate, but parallel, iCODE content tracks. Middle schoolers from Lowell and Lawrence, Massachusetts, worked with the Super Cricket, a printed circuit board controller programmed in Logo. High school students from Boston, Massachusetts, worked with Machine Science’s breadboard development kits, physically wiring components to a PIC microcontroller and programming the chip in C. Despite these differences in the hands-on materials, students in both content tracks were challenged to master a similar progression of knowledge and skills (Table 2). Moreover, twice a year, participants from both tracks came together to exhibit their robots and competed against one another in the robot sumo contest on equal footing.
5 Student recruitment and retention The iCODE enrichment programs were run at five school and community center sites in Boston and Lowell/Lawrence during 2006–2007, and at 10 sites during 2007–2008 and 2008–2009. In each year, the programs served a racially diverse and economically disadvantaged urban population. Over the three-year grant period, a total of nearly 250 students participated, with many staying involved for multiple
268
Auton Robot (2011) 30: 265–280
Table 1 iCODE theory of change
years. Recruitment of students was left to the teachers at participating schools. Teachers were encouraged to identify students with an interest in engineering and technology, but received no formal recruitment guidelines. In every year of the grant, male students outnumbered female students by a ratio of 2:1 or greater. This enrollment pattern was a disappointment to the project team, but comparable to other programs with a strong focus on engineering, computer science, and robotics. Among the relatively small number of female students who joined the program, many excelled. Girls consistently ranked among the top performers in the annual sumo robotics tournament. In addition, in their responses to pre- and post-program surveys,
some female students appeared to make stronger gains than their male counterparts on items dealing with attitudes toward STEM subjects and careers. Table 3 presents the demographic characteristics of the iCODE student population. In planning for the project, the iCODE project team had predicted that roughly half of each year’s participants would return to the program the following year, producing the target retention figures shown in Table 4. This prediction proved to be roughly accurate, and the project met its overall student recruitment goals. For both middle school and high school students, 120 hours of participation represented a considerable time commitment. Although nearly all students enjoyed and ben-
Auton Robot (2011) 30: 265–280 Table 2 iCODE program: required skills
269 Level
Objective
Skills
Novice
Learn to use tools and build simple circuits.
– identify basic components: e.g., wires, switches, batteries – construct basic circuits: e.g., simple circuit games – understand basic concepts: e.g., power, ground, short circuit, voltage, resistance, amperage – use tools properly and safely
Beginner
Build programmable circuits and write simple programs.
– identify components in programmable circuits: e.g., microcontrollers, programming linkages, USB cables – connect input and output devices to the microcontroller – write a program and download it to the microcontroller – understand persistence of program in the controller
Intermediate
Complete guided building and programming projects.
– write programs to control output devices: e.g., LEDs, speakers, motors, LCDs – write programs to use input from sensors: e.g., temperature, light, infrared distance sensors – declare and manipulate variables
Advanced
Apply building and programming skills to structured design challenges.
– write programs that use advanced structures: e.g., sub-procedures, digital and analog inputs, loops and conditionals, logical operators (AND/OR), variables – develop a working solution to a structured design challenge, using an appropriate subset of the above programming elements
Master
Apply skills in independent projects and mentor peers.
– formulate and complete a novel design challenge, connecting hardware in new ways – teach circuit building and programming to others – render project guidance or debugging assistance to peers – teach others how to use tools
efited from the program experience, some found it difficult to maintain their attendance in weekly after-school sessions through both the fall and spring semesters, as well as a twoweek summer session. At some sites, the full-year iCODE program commitment conflicted with seasonal sports schedules and other extracurricular activities. Convincing students from the after-school sessions to attend the program’s summer component was challenging, particularly for the Boston-based programs, whose low-income, high schoolaged participants were strongly motivated to seek summer employment. For some students, mandatory summer school precluded their participation in the summer program. The iCODE program teachers were asked to be somewhat flexible about attendance, so as not to lose students with extracurricular conflicts. However, this was a delicate balance to strike, as students were expected to complete an ambitious sequence of increasingly complex hands-on
projects and to acquire a substantial set of new engineering and programming skills. Students who missed sessions would get out of step with their peers, and require extra assistance to catch up. The project team did anticipate attrition over the course of the school year; a recent Massachusetts study estimated that the typical after-school program has an attendance rate of 59% (United Way of Massachusetts Bay & Merrimack Valley 2005). Program teachers over-recruited at the beginning of each academic year in anticipation of this, and planned recruitment goals were met (Table 4). To analyze retention rates for males and females, retention was defined as the number of program participants who completed both the pre- and post- program surveys, as compared to the total number students who completed the presurvey. With this definition, the overall retention of students from the iCODE school-year program was 47%. This esti-
270
Auton Robot (2011) 30: 265–280
Table 3 iCODE student demographics: 2006 to 2009 Characteristic
Attribute
2006–2007
2007–2008
2008–2009
Gender
Female
32%
18%
21%
Male
68%
82%
79%
6th
6%
2%
7%
7th
27%
35%
37%
8th
12%
13%
23%
9th
17%
10%
10%
10th
25%
14%
3%
11th
8%
19%
0%
Grade
12th
6%
8%
11%
Ethnic and
American Indian or
3%
2%
0%
racial
Alaska Native
background
Asian
19%
20%
25%
Black or
35%
23%
24%
19%
31%
37%
26%
26%
22%
African-American Spanish/Hispanic or Latino White
Table 4 iCODE student recruitment and retention: 2006 to 2009 Student
2006–2007
Group
Target
Actual
Target
Actual
Target
Actual
Target
Actual
1st year
50
40
75
66
51
103
176
209
2nd
2007–2008
2008–2009
Total
year
–
–
25
12
37
23
62
35
3rd year
–
–
–
–
12
5
12
5
Total
50
40
100
88
100
131
250
249
mate is likely to be low, because it does not take into account students who may have been absent from the program on the days during which the post-program survey was administered. The retention rate was comparable for males (46%) and females (49%).
6 Support for program sites In rolling out the iCODE program, UML and Machine Science found few teachers at the partnering program sites with knowledge of the program’s core content: electronic circuitry, embedded computing, and C and Logo programming. The iCODE program did not simply ask teachers to teach their STEM subjects in a new, computer-intensive way; it asked them to teach an entirely new body of knowledge and skills. As a result, most teachers required significant training and support to implement the programs. Since the partnering schools and community centers were
all located in Boston, Lowell, and Lawrence, UML and Machine Science were able to provide this support directly. Teachers received individual on-site guidance in setting up the programs prior to the start of the academic year, and assistance from undergraduate mentors throughout the year. The iCODE partners had more success with programs that were based in schools than with those that were based in community centers, such as Boys and Girls Clubs. The classroom teachers who led the school-based programs were able to draw on their history of interactions with particular students during the school day when recruiting participants for the program. They also were also well-positioned to remind students about upcoming iCODE program sessions and events, and experienced at encouraging students to persevere when confronted with academically challenging material. In contrast, at community center sites, the relationships between staff and youth were generally more informal, and youth visitors were not always accustomed to attending after-school programs on a consistent basis from
Auton Robot (2011) 30: 265–280
271
Fig. 2 On-line programming window
week to week. To succeed in this environment, programs like iCODE would need to articulate very clear expectations in terms of student attendance and benchmarks. Also, they might benefit from a more-intensive schedule, with meetings held on more days of the week for a shorter overall time period.
7 Web technology A major goal of the iCODE project was to use web technology to facilitate the enrichment programs at each partner site. Students used the iCODE web system to access project instructions, as well as the on-line programming tool. Available from any Java-enabled web browser, the programming tool presents students with a full-featured code development environment, complete with syntax highlighting for supported languages (C and Logo). With a single mouse click, a student can send a code file to the server to be compiled. The server instantly returns the compiled code, along with any compiler messages. At that point, the student either modifies the code or, with another mouse click, downloads it to the microcontroller. The applet communicates directly with a bootloader on the chip through a virtual serial port on the student’s computer. The programming window supports multiple hardware platforms. Using the options menu, students can move seamlessly between developing programs in Logo for the Super Cricket and developing programs in C for the PIC microcontroller. Figure 2 shows the on-line programming window. After students had completed projects, they were encouraged to post images, videos, and descriptions to the iCODE system’s invention database, an on-line portfolio of projects. The project database enabled students to show off their work to friends, siblings, and parents, comment on one another’s finished projects, and find inspiration for future projects. Moodle, an open-source course management system originally developed for universities, provided the framework for student account management and web content delivery. It was well suited to this purpose, offering a range of useful built-in features, including a WYSIWYG editor for
Fig. 3 Super Cricket (left) and Machine Science breadboard kit (right)
course content, quiz development tools, threaded discussion forums, and a highly configurable scheme for setting teacher and student permissions. The iCODE team had hoped that using the Java-based programming tool would avoid some of the difficulties associated with installing and maintaining software in schools, including the need to support multiple operating systems, the logistical problem of distributing software updates, and the understandable reluctance of some school administrators to permit the installation of large software packages (like C compilers). In practice, the use of the web-based programmer involved challenges of its own. Frequent updates were needed to keep up with Java releases and new versions of the PC and Mac operating systems. Also, it was not completely free of installation hassles: teachers needed an administrative log-in in to install a small DLL library file and the USB drivers that create the virtual serial port for communication with the microcontroller.
8 Hands-on materials The iCODE project’s hands-on materials included two unique microcontroller development platforms (Fig. 3), together with related robotics hardware and building materials. UML’s Super Cricket is a printed circuit board, programmed in Logo; Machine Science’s platform is a breadboard development kit, with a PIC microcontroller intended to be programmed in C. The physical design of the Cricket and the breadboard were thought to be suitable for the different grade levels in-
272
Auton Robot (2011) 30: 265–280
gramming, as a learning challenge. The mentor working with these students reported that students’ work was successful. Each partner brought a pre-existing hardware platform to the iCODE collaboration. The research team did not investigate whether the two different hardware platforms resulted in differences in student learning and outcomes. Figure 4 illustrates the range of projects completed by students in the iCODE program, including robots, games, electronic fashion accessories, and other programmable creations.
9 Community building
Fig. 4 iCODE student projects
volved in the iCODE project. The Cricket’s dedicated connectors make circuit-building relatively straightforward— students simply plug in sensors and actuators to the appropriate jacks. Machine Science’s breadboard approach challenges students to build circuits by hand, physically wiring components to the microcontroller. The computer code for the two devices is another key differentiator: Logo is a simple programming language whose commands and syntax can be quickly learned by novice programmers; C is a powerful language widely used by professionals for embedded computing applications. In general, the iCODE program’s hands-on materials held high appeal for participants. Students seemed intrigued by the opportunity to learn about technologies that have become ubiquitous in modern life, embodied in devices such as cell phones, MP3 players, handheld electronic games, GPS receivers, etc. Some high schoolers found Machine Science’s breadboard-based projects frustrating, since these projects involved both wiring and coding challenges. As anyone with experience in embedded computing will attest, it can be very difficult to troubleshoot a project without knowing whether the hardware, the software, or both are the source of the problem. At the same time, many students appreciated having the chance to work with real, off-the-shelf electronic components, and felt that this made the program experience more authentic. During the project’s third year, students who had worked in previous years with the Cricket and Logo programming chose to transition to the breadboard platform and C pro-
Various aspects of the iCODE project were intended to build a sense of community among students and connect them with adult mentors and role models. Participating middle and high school students were given the chance to work on their iCODE projects with help from undergraduate engineering and computer science students. They met with college students and admissions staff at IT and STEM-focused career events, and shared their work with parents, siblings, and the broader community at UML’s BotFest and Machine Science’s Robot Sumo Tournament. Also, participants were encouraged to form an on-line community through use of a threaded discussion forum and the web-based invention database. According to educator and student feedback, the presence of undergraduate mentors in iCODE classrooms was one of the more successful program elements. Depending on the host teacher’s abilities, the mentor’s role ranged from classroom assistant to lead instructor. Mentors helped explain the iCODE content, assisted students with their hands-on projects, challenged students to expand their project ideas, and provided programming and troubleshooting expertise, as needed. Mentors were particularly adept at gauging the potential of individual students and challenging them with questions and activities to help them go beyond their acquired expertise. Because the mentors were relatively close in age to the participating students, and because the mentors were themselves still in school, the iCODE program’s middle school and high school students were inquisitive about the mentors’ college classwork and career plans. Machine Science’s annual Robot Sumo Tournament was consistently well attended and cited by educators as a strong motivator for students to build sophisticated autonomous robots. Holding this event in exhibit space at the Museum of Science Boston provided visibility for the iCODE program, attracted news media attention, and increased the excitement level for competitors. Over the three-grant period, more than 120 robots were entered in the competition. As for the on-line invention database, students made some limited use of this system feature, posting more than
Auton Robot (2011) 30: 265–280
273
200 project images and descriptions, but it never became a popular gathering point for iCODE participants. The invention database suffered from security and other technical problems, and its appeal to students may have paled in comparison to other social networking alternatives, such as Facebook and MySpace, which grew rapidly in popularity during the three-year grant period.
10 Program evaluation Goodman Research Group (GRG) of Cambridge, Massachusetts, served as the project’s external evaluator. Over the course of the project, student learning outcomes were assessed using a variety of measures: structured observations of iCODE program sessions; group interviews with participating students; pre- and post-program student surveys, containing both Likert-scale and open-response items; educator surveys and interviews; and content-specific quizzes, administered three times annually. Results were reported in three broad areas: (1) appeal and effectiveness of iCODE program components; (2) gains in engineering and programming skills; and (3) preparation for STEM careers. The qualitative data from open-ended survey questions, interviews, and observations were coded. Themes were clustered and, where appropriate, counted. Then narrative accounts emphasizing patterns, commonalities, and differences were generated. Where possible, data from different sources were triangulated to corroborate findings. Results regarding students’ understanding of the work of engineers, presented in Sect. 10.3, are an example of insights gleaned from the qualitative evaluation work. 10.1 Appeal and effectiveness of program components In their survey responses, students rated the major components of the iCODE program highly (Table 5). Students particularly enjoyed completing hands-on technology projects Table 5 Student ratings of iCODE program components
Program component
and working closely with the program’s undergraduate mentors. Both of these components earned a mean rating near or above 4.0 on the survey’s five-point scale (1 = Poor, 2 = Fair, 3 = Good, 4 = Very Good, 5 = Excellent). Based on discussions with educators and students, GRG reported that the following program components were most effective: – Hands-on activities. A common theme throughout the three years of the iCODE program was the high appeal among the students of the hands-on aspect of the various projects. The students enjoyed working on the projects and experiencing success with a project. The projects gave them an opportunity to be creative within the structured program curriculum. – Project progression. Participating educators were satisfied with the structure of the program, which involved a gradual progression from introductory activities to more open-ended projects. Educators encouraged students to reflect on their own work, communicate with others, and document their progress, leading to an increase in student engagement. – Balance between autonomy and collaborative work. There was consensus among the educators that the iCODE system’s on-line guides and programming portal allowed the students to be independent. During the after-school sessions, students closely followed instructions from the project guides and loaded code files from the guides into their projects. Nevertheless, students felt responsible and took ownership for their projects. At the same time, the students had a number of opportunities to work in small or large groups and with the mentors, leading to community building among students and the mentors. – Culminating events. The participation in Botfest and the Robot Sumo competition, which signified the evolution of students’ hands-on projects, were important aspects of the program, according to the educators. The summer session, being more intensive, increased the students’ engineering knowledge and skills to a large extent. Work-
2006–2007
2007–2008
2008–2009
(N = 43)
(N = 52)
(N = 69)
Hands-on projects
4.40
4.50
4.30
On-line resources
3.70
4.05
3.60
3.86
3.90
3.60
3.70
3.90
3.50
4.10
4.12
3.80
In-person visits from engineering and technology experts Internet-based interactions with industry mentors Collaboration with student mentors
274
Auton Robot (2011) 30: 265–280
A focused group of mostly girls using creativity and imagination to engineer high-tech garments and accessories. A GRG researcher observed a summer camp session for about an hour. Nine girls and three boys, all from the 9th grade, participated. An educator and an undergraduate mentor were present to instruct the students. The students worked on projects that had been approved by the educator earlier in the week. Projects included: lamps with LED lights, LED night-lights, picture frames with etched Plexiglas and LEDs, and purses and T-shirts embellished with LEDs. The students had new computers and ample supplies. There was some structure to the session, but the teacher gave the students a lot of flexibility, which they seemed to appreciate. During the observation period, the educator supervised the class and helped one-on-one when needed. The educator explained to the GRG researcher that, during a previous session, students had struggled to embellish T-shirts with LEDs. After that day, the students learned the value of making a design plan and using it to progress step-by-step through a project. They researched and developed plans for their projects, and the educator approved each plan individually. By the day of the observation, the students were all a few steps into their projects, revising their design plans as needed. The GRG researcher observed students wiring lights, wiring their project items, creating the structure for their projects, testing lights, and writing code to control their lights. They seemed to be enjoying the activity and the fact that they got to choose a project and make their own personal design choices. The students also got to take home whatever they made, which seemed to have a positive influence on their progress and enthusiasm for finishing the projects. Fig. 5 Observation of UMass Lowell summer camp session
ing on goal-oriented projects, whether for the competition or through the summer camps, added immensely to the learning experience of the students. Figures 5 and 6 encapsulate evaluator observations of the summer camp sessions. 10.2 Gains in engineering and programming skills For most students, the iCODE experience provided their first opportunity ever to engage in this type of activity (Table 6). The vast majority of participants (80%) had never written a computer program prior to joining the iCODE program. There were statistically significant differences between the baseline experience of the middle school students involved in the programs run by UML and that of their high school counterparts in the Machine Science programs. By the end of the program, nearly all students reported at least a little increase in their understanding of computer programming and electronic devices. In both of these areas, 59% of students felt their understanding had increased a great deal (Table 7).
Notably, students’ previous experience did not affect their perceptions of what they learned from iCODE. Students who had previous experience building electronic circuits were just as likely to report gains in their understanding of electronic devices at the end of the program as those who had no prior experience. The same was true with respect to writing computer programs. On average, participants in each iCODE cohort left their summer camp feeling that they could perform the required engineering and programming skills pretty well and, with enough time to review, they could lead friends in performing the skills. Table 8 shows mean student responses on a fivepoint scale (1 = I cannot do this, 2 = I can do this but only with assistance, 3 = I can do this well enough for my own personal use, 4 = I can do this pretty well, and could show a friend how to do it if I had time to review, and 5 = I can do this very well and could show a friend how to do it). 10.3 STEM career preparation The iCODE program provided students with opportunities to work together, increased their perceived problem-solving
Auton Robot (2011) 30: 265–280
275
Students building and programming wireless text messaging devices, supported effectively by their teachers and strongly motivated by the project technology. A GRG researcher observed a session during the summer camp at the Machine Science site. Three members of Machine Science staff were present as instructors. There were a total of six male and one female high school students, representing grades 10 to 12. There were plenty of computers and space for everyone. Each student had a computer to work on and the other required supplies. There were two projectors at the front of the room for the teacher to post directions and demonstrate code development. There was also a white board for the teacher to use for additional visual aids. The web technology seemed to be working well. On the day of the observation, students were starting a new project: building wireless text message devices. The educator introduced the code and the instructions, wrote notes on the white board, and talked through some of the web site content. Then, the students went to work on their own, going through the instructions with the machines. During this time, Machine Science staff members went around the room multiple times helping students and doing individualized instruction. The educator had a very gentle and supportive approach with the students. She seemed to encourage the students to figure out their own problems by doing a combination of asking scaffolding questions and explaining new concepts. The students were at the computers with their machines, writing the code and trying it out. They mostly used the directions on the Machine Science website, although they also got assistance from the instructors. They were learning about how phones use codes to translate numbers into letters to do text messaging. Later that day, they were going to learn to add a radio frequency component to the machines and try sending texts to each other. The students were very engaged and focused on learning the code and programming their text messengers.
Fig. 6 Observation of Machine Science summer camp session
Table 6 Relevant experience prior to iCODE
ap
< .001
bp
< .05
Table 7 Perceived extent to which iCODE increased students’ IT understanding
N = 145
Experience
UML
Machine Science
Total
N = 154–157
N = 104–107
N = 261–262
Completed an engineering project before iCODEa
27%
52%
37%
Built electronic circuits before iCODE
41%
41%
41%
Wrote a computer program before iCODEb
15%
27%
20%
Content Area
Not at all
A little
Some
A great deal
Computer programming
2%
6%
32%
59%
Electronic devices
1%
9%
31%
59%
276
Auton Robot (2011) 30: 265–280
Table 8 Students’ perceived skill level post summer camp Rating
UML
Machine Science
2007
2008
2009
2007
2008
2009
(N = 11)
(N = 34)
(N = 35)
(N = 11)
(N = 13)
(N = 7)
Mean
3.74
4.25
3.64
3.56
3.79
4.07
Minimum
2.30
2.64
1.18
2.22
2.30
3.00
Maximum
4.70
5.00
5.00
4.80
4.80
5.00
Table 9 Educator and student comments on iCODE program impact
Source
Comment
Educator
“The iCODE program is a total package of designing, building, coding, testing, documenting, and assessment. This approach is very effective in accomplishing the program objectives.”
Educator
“The online guides allow the students to be very independent (learn without a teacher), and also encourage them to work with their peers.”
Student
“I liked the fact [that] no matter what we did, it was ours, whether we failed or achieved, it was what we did, there was no handholder or anything, aside from the basic explanation, so it was kind of being your own boss and getting what you wanted to get done.”
Educator
“I enjoyed having the mentor at the program. She was knowledgeable about the projects, and could help troubleshoot when students were having problems. The students felt comfortable asking her questions, and she was very good at explaining the concepts in the project.”
Student
“If you’re at all interested in electronics or anything, it’s [iCODE] something hands-on. Instead of looking it up on the internet and having to read everything about it, you can actually learn about it [electronics] here.”
Student
“iCODE far exceeded my expectations. I was looking for a program to allow me to build, and program. iCODE is far more than that.”
Table 10 Perceived extent to which iCODE increased students’ workforce skills
N = 143–145
Not at all
A little
Some
A great deal
Provided students with opportunities to work together with other students (N = 97)
5%
12%
29%
55%
Increased students’ problem solving abilities (N = 99)
9%
17%
34%
40%
Connected students with professionals in the fields of engineering and technology (N = 145)
8%
18%
27%
48%
abilities, and connected them with professionals in the fields of engineering and technology. All of these are important factors in encouraging workforce readiness. Qualitative evidence in the form of selected educator and student comments are presented in Table 9, and quantitative evidence from analysis of survey replies is presented in Table 10. As shown in Table 11, more than three-quarters of participants reported that the iCODE program exposed them to information about careers in science and technology either some (30%) or a great deal (48%). Likewise, a strong majority of respondents indicated that the program experience
improved their attitudes about careers in science and technology either some (36%) or a great deal (40%). As an additional indicator of iCODE effectiveness in exposing students to STEM careers, on the pre- and postprogram surveys each year students were asked “What do engineers do?” There were 101 cases in which students answered at both pre and post. These responses were coded for accuracy, completion, and sophistication. As Table 12 illustrates, about half of the students (N = 50) gave more accurate or sophisticated responses after the program than they did before it. In most of these cases (N = 35), the postresponse was substantially better than the pre-response; in
Auton Robot (2011) 30: 265–280
277
Table 11 Perceived extent to which iCODE increased STEM career preparation
N = 144–145 Table 12 Student responses to “What does an engineer do?”
N = 101
Not at all
A little
Some
A great deal
Exposed students to information about careers in science and technology
6%
16%
30%
48%
Improved students’ attitudes about careers in science and technology
8%
17%
36%
40%
Post-response
Post-response
Post-response
less adequate
same as
marginally better
substantially better
than pre-
pre-
than pre-
than pre-
10% (N = 10)
40% (N = 41)
15% (N = 15)
35% (N = 35)
the other cases (N = 15), the post-response was marginally better than the pre-response. Marginal improvements were defined as slightly more accurate, complete, or sophisticated responses at post- than at pre-. Examples included: – Changes from one- or two-dimensional responses at pre to two- or multi-dimensional responses at post (e.g., “build” to “build or create”; “create and fix things” to “design, create, and fix things”) – Identification of a tool at post (e.g., “build stuff” to “create things with computers”) – Identification of a purpose at post (e.g., “They fix stuff or make stuff” to “They design and build stuff for people’s lives”) or identification of the focus of building at post (e.g., “design stuff” to “design bridges, roads”) Substantial improvements fell into similar categories but were quite a bit or a great deal more accurate or complete/sophisticated responses at post- than at pre-. These included eight students’ responses that changed from “I don’t know” at pre- to, at post-, giving an explanation of what an engineer does (e.g., design, build, program computers, fix things, work with technology). Other changes that qualified as quite a bit or a great deal more sophisticated started with simple responses such as “make stuff” and ended with somewhat more elaborate responses, such as “design things and solve problems to make human life easier.” The other half of students gave fairly similar responses pre- and post- or gave a scantier response at post-.
11 Discussion The iCODE project produced significant outcomes for participating students. The program effectively engaged participants in meaningful design projects. In completing these
Post-response
projects, students developed real engineering and programming skills, and their attitudes toward STEM subjects improved. The iCODE program also provided students opportunities to work closely with undergraduates and to spend two weeks per year at a college campus. Taken together, all of these program elements increased many students’ interest in STEM career pathways. To conclude, we reflect on the challenge of evaluating programs like iCODE, the sustainability of the project, and some larger issues raised by this work. 11.1 Reflections on evaluation As in other studies on informal education with robotics (Nourbakhsh et al. 2004; Nugent et al. 2009), the iCODE evaluation examined student learning in the areas of engineering and programming, teamwork and other workforce skills, and attitudes toward STEM subjects and careers. Students self reported their learning in written and online surveys. The iCODE study also tested two other data-collection methods. During the first summer camp, students completed journals and during the second and third years of the schoolyear program, students had the opportunity to complete online quizzes. Similar to other informal robotics program studies (Weinberg et al. 2007), the iCODE evaluation featured a pre- and post-test design in which participating students completed on-line or written surveys at the beginning and end of the program. Had budget permitted, the inclusion of a comparison or control group would have allowed for stronger attribution of observed outcomes to the program, as in the Nugent et al. (2009). The iCODE study did not include any mid-point or follow-up surveys, as some other studies have (Nourbakhsh et al. 2004), although a small number of students participated in the program for more than one year, thus providing a limited longitudinal data set. In general, the more data collection points, the better for determining the trajectory of
278
student learning. In addition, follow-up surveys are desirable for examining maintenance of program effects or for observing effects beyond the boundaries of program participation (e.g., pursuit of relevant activities). The iCODE evaluation was “whole-program” focused: it included all students and educators at all sites. There was no in-depth evaluation of specific students or sites as has been done in other research. Had it been possible, such a supplement may have resulted in more detailed understanding of student learning. In addition, iCODE program recruitment, particularly for the Machine Science curriculum, did not exclude students who had previously participated in the program. The presence of experienced students may have limited the observed impact of the program. 11.2 Sustaining iCODE Going forward, the iCODE partners are committed to maintaining the iCODE web site as a resource for educators, students, hobbyists, and others interested in learning about microcontroller-based projects. In addition to the more than 300 teachers and students from the 2006–2009 iCODE enrichment programs, some 1,800 additional users have registered accounts on the web site, in order to access instructional materials and the on-line programming tool. This registered user base includes educators and students from public and private secondary schools, as well as colleges and universities. Machine Science’s breadboard kits, in particular, have attracted interest for use as college-level teaching tools at Northeastern University, MIT, Tufts University, Norfolk State University, Henderson State University, and the University of North Texas, among others. UML and Machine Science have also made the project’s tangible learning materials available for sale on the web. It is hoped that revenues from sales of these materials will help sustain the project’s momentum, and continue the enrichment programs in local public schools. 11.3 Larger issues The results of our work and those of many other studies clearly demonstrate that students enjoy hands-on, designrich experiences with robotics, and show strong gains in content knowledge as a result of these programs. Having self-selected for participation, many students entered the iCODE program with generally positive attitudes toward engineering and technology. This phenomenon has also been reported by the other researchers previously cited (Nourbakhsh et al. 2004; Miller et al. 2005; Weinberg et al. 2007). In the case of iCODE, these attitudes were maintained, but more importantly, they were also given much clearer shape
Auton Robot (2011) 30: 265–280
and definition. Students understood much better after completing the program what professional engineers and programmers actually do. The iCODE program’s impact on career intentions varied from student to student. For some participants, we believe the iCODE program was truly transformative, leading to changes in their perceptions of technology, their educational aspirations, and their college and career plans. This is evidenced by students who stayed with the program for multiple years. When they finished middle school, a small group of iCODE students organized a robotics club at their high school to continue developing their interest. It seems clear that the larger structural challenge is to provide students with a continuum of learning experiences such as those developed by the iCODE project. Several of our teachers have commented that they enjoy teaching iCODE because it allows them to engage students in technical content in a deeper fashion that is possible in the conventional school day. In the American system of education, opportunities for students to pursue interests in depth tend to occur outside of the school day—for example, in sports practice or music rehearsals that can consume 15 or more hours per week. But there exist comparatively few opportunities to engage students in such a powerful fashion in science and technology. Furthermore, the separation of academic preparatory schools from vocational institutions is problematic. Students on the academic track are deprived of hands-on experiences, while students in the vocational track are not offered the opportunity to connect their concrete knowledge with the symbolic skills with which they are surmised to struggle. Neither group benefits from this arrangement. We believe that offering a range of students the honest, challenging, and sometimes frustrating experience of engineering in a significant way is a critical need. Students who never knew they had this interest will discover it. It is then up to us to continue to provide them with opportunities to nurture this interest towards meaningful, satisfying career paths.
References Baker, D. (2009). Design camp: an alternative model for D&T education? D&T Practice. URL www.data.org.uk. Eisenberg, M., & Eisenberg, A. N. (1998). Shop class for the next millennium: education through computer-enriched handicrafts. Journal of Interactive Media in Education, 98(8). URL www-jime.open.ac.uk/98/8. Kafai, Y., Peppler, K., & Chapman, R. (2009). The computer clubhouse: constructionism and creativity in youth communities. Teachers College Press, New York. Martin, F. G. (1996). Ideal and real systems: a study of notions of control in undergraduates who design robots. In Y. Kafai & M. Resnick (Eds.), Constructionism in practice: designing, thinking, and learning in a digital world. Mahwah: Lawrence Erlbaum Associates.
Auton Robot (2011) 30: 265–280 Martin, F., & Resnick, M. (1993). LEGO/Logo and electronic bricks: creating a scienceland for children. In D. Ferguson (Ed.), Advanced educational technologies for mathematics and science (pp. 61–90). Berlin Heidelberg: Springer. Martin, F., Mikhak, B., Resnick, M., Silverman, B., & Berg, R. (2000). To mindstorms and beyond: evolution of a construction kit for magical machines. In A. Druin & J. Hendler (Eds.), Robots for kids: exploring new technologies for learning (pp. 9–33). San Mateo: Morgan Kaufmann. Miller, L., Shearer, S., & Moskal, B. (2005). Technology camp 101: stimulating middle school students interests in computing. In ASEE/IEEE frontiers in education conference, S1F–26. Nourbakhsh, I, Crowley, K., Hamner, E., & Wilkinson, K. (2004). Formal measures of learning in a secondary school mobile robotics course. In IEEE international conference on robotics and automation (Vol. 2, pp. 1831–1836). Nugent, G., Barker, B., Grandgenett, N., & Adamchuk, V. (2009). The use of digital manipulatives in K-12: robotics, gps/gis and programming. In ASEE/IEEE frontiers in education conference, M2H-1. Papert, S. (1980). Mindstorms: children computers, and powerful ideas. New York: Basic Books. Sklar, E., Eguchi, A., & Johnson, J. (2003). Robocupjunior: learning with educational robotics. In Lecture notes in computer science, RoboCup 2002: robot soccer world cup VI (pp. 238–253). Berlin: Springer. United Way of Massachusetts Bay & Merrimack Valley (2005). Pathways to success for youth: what counts in after-school. Tech. rep., Intercultural Center for Research in Education and National Institute on Out-of-School Time. URL supportunitedway.org/ asset/mars-massachusetts-after-school-research-study. Weinberg, J. B., Pettibone, J. C., Thomas, S. L., Stephen, M. L., & Stein, C. (2007). The impact of robot projects on girls’ attitudes toward science and engineering. In Robotics science and systems (RSS) workshop on research in robots for education. URL www.roboteducation.org/rss-2007/.
Fred G. Martin is an Associate Professor in the Computer Science department at the University of Massachusetts Lowell. He directs the Engaging Computing Group, which develops and studies the use of technological materials for engineering and science education at the K-12 and university levels. Previously, Fred was a research scientist at the MIT Media Laboratory, where he developed a series of educational robotics materials, and laid the foundation for the breakthrough LEGO Mindstorms Robotics Invention System, which was launched in 1998. Fred helped launch the worldwide robot contest phenomenon by cofounding the MIT Autonomous Robot Design Competition, and by publishing the textbook, “Robotics Explorations” (2001), along with the Handy Board robot controller platform. Fred holds a B.S. in Computer Science (1986), an M.S. in Mechanical Engineering (1988), and a Ph.D. in Media Arts in Sciences (1994), all from the Massachusetts Institute of Technology. Fred also contributes to Gleason Research, a robotics company he cofounded with his wife Wanda Gleason, has consulted on a educational projects in the United States and across the world, including Ireland, Brazil, Thailand, Mexico, and Colombia.
279 Michelle Scribner-MacLean is an assistant professor of science and math education at the Graduate School of Education at the University of Massachusetts Lowell. She holds an Ed.D. in Science and Math Education, an M.Ed. in Curriculum and Instruction from the University of MA Lowell and a M.S. in Biology from the University of Nebraska Kearney. Her research interests include assessment and evaluation of science, math, and engineering learning in elementary and secondary students. She currently serves as an assessment expert and curriculum developer on two STEM-related NSF grants with Dr. Fred Martin. Dr. Scribner-MacLean has extensive teaching experience in formal and informal settings. As a science educator at the Boston Museum of Science for over 20 years, she designed and implemented courses for preschoolers through adults, designed teacher professional development experiences, and led safaris to East Africa. She has also taught in the elementary and secondary settings, in addition to her teaching experiences as a faculty member specializing in curriculum and instruction and assessment. Sam Christy is a co-founder of Machine Science Inc., overseeing the organization’s technology development and product commercialization. Sam began his career with the John F. Kennedy Library Foundation, where he coordinated community-service projects performed by Boston-area teenagers. In 1992, Mr. Christy founded Science Bridge—an after-school science workshop for high school students. Operating from a storefront in Chelsea, Massachusetts, Science Bridge served over 50 students each week with mentoring assistance from undergraduate and graduate students. Mr. Christy went on to become the first manager of the Boston Computer Museum’s Clubhouse Program, where he led the development of a computer learning center visited annually by some 4,000 students. In 1996, Mr. Christy founded WordStream—a private company based on a language processing technology that he invented and patented. As the company’s chairman, he raised over $5 million in venture capital funding and grew the company to more than 30 full-time staff members. Ivan Rudnicki is a co-founder of Machine Science Inc., where he supervises the company’s curriculum development, strategic partnerships, and grant fundraising. Since 2001, Machine Science has worked with more than 25 schools and more than 100 teachers, serving a total of more than 2,000 learners. Mr. Rudnicki is part of the management team for two National Science Foundationfunded education research projects, Building an Internet Community of Design Engineers (iCODE) and the Internet System for Networked Sen-
280
Auton Robot (2011) 30: 265–280
sor Experimentation (iSENSE), both collaborations with the University of Massachusetts Lowell. Prior to joining Machine Science, Mr. Rudnicki held editorial and project management positions at two Bostonarea consulting firms. Rucha Londhe Ph.D., works as a Research Associate at the Goodman research Group (GRG), Inc., a Cambridge, MA based firm that specializes in evaluating programs, materials, and services. At GRG, Dr. Londhe manages a variety of evaluation projects across a number of content areas. Dr. Londhe received her doctorate from the School of Family Studies, University of Connecticut in Human Development and Family Studies, with a specialization in child development. She has a graduate degree in Psychology from Bombay University, India, where she worked as a clinician before moving to the U.S. She has also served as a visiting lecturer, teaching undergraduate classes in Human Development and Psychology, at the University of Connecticut, Storrs and the Framingham State College, MA. Colleen Manning is Director of Research at Goodman Research Group, Inc. Her areas of expertise are evaluation of informal education programs and research on early care and education. She received her M.A. in Child Study from Tufts University in 1994 and her B.A. in Psychology from Mount Holyoke College in 1989. She is currently completing her Ph.D. in Public Policy at the University of Massachusetts, Boston.
Irene F. Goodman Ed.D. is the founder and president of Goodman Research Group, Inc. (GRG), a research firm specializing in the evaluation of educational programs, materials, and services that has been operating since 1989. Aside from overseeing 40+ research evaluation projects per year, Dr. Goodman provides consultation and gives lectures and workshops on evaluation. Currently, her particular areas of interest are informal education opportunities for all ages, and the intersection between formal and informal science education for youth. Prior to founding the company, she was an evaluation consultant to various local and national organizations, taught courses at Dartmouth College and Tufts University as Visiting Lecturer, developed instructional materials, and carried out regional training sessions on public policy issues. She also had stints as senior research associate at other research institutes. In the mid 1970s, she was on the faculty at the University of Wisconsin-Madison and was the statewide specialist in child development for University of Wisconsin-Extension. She holds a BA in Psychology from UCLA (1973), MA in Child Development from Washington State University (1975) and a doctorate in Human Development and Psychology from Harvard University (1984).
