Interactive, Web-Based Workshops on Educational Development Proposals ...
since the Solicitation has a component calling for projects devoted to developing
...
Interactive, Web-Based Workshops on Educational Development Proposals and Projects Introduction Repeated national reports have called for fundamental changes in STEM education that substantially strengthen the focus on student learning and emphasize authentic, engaging instructional approaches.19 Reports on engineering education share this perspective, for example, the National Academy’s reports on the Engineer of 2020 described the changes in the knowledge and skills that graduates will need.20, 21 Ultimately, significant change will involve adjustments in the complex higher education enterprise and will have to deal with the widely accepted research-based faculty reward structure and the community’s limited understanding of cultural factors and change processes.3, 10, 23 Significant reform will require a better understanding of these factors but, in the end, all approaches will rely on effective strategies for changing the way faculty members approach their teaching. It is not clear that the current professional development models for accomplishing this will be adequate to enable the substantial changes in attitude, knowledge, and skills on a scale that will meet the need. Although some traditional engineering faculty workshops have had positive results as reported by Felder and his colleagues, 2, 7, 8 several investigators have identified some important issues with the short-term, face-to-face model. 6, 9, 13 Specifically, such workshops do not allow time for faculty members to go through the transitions from awareness to action, 9 can cause an adversarial relationship between the presenter and the participants, 6 and do not encourage participants’ motivation and commitment.13 The inadequacy of existing faculty development models is reflected in: 1) the slow adoption of engaging, active-learning methods that have been systematically tested and shown to improve student learning among diverse students: 1, 11 2) the stalling of innovation in STEM education:15 and 3) the wide-spread reliance on lecture and the receive/memorize cognitive demand as the primary instructional strategy.12 Thus alternative faculty development models are needed. From a NSF Division of Undergraduate Education (DUE) perspective, faculty development has been a key component in the Transforming Undergraduate Education in STEM (TUES) and Course, Curriculum and Laboratory Improvement (CCLI) Programs since the Solicitation has a component calling for projects devoted to developing faculty expertise. Further, with many of the proposals focused on the curriculum development, faculty workshops play a major role in the dissemination plan. For a number of years, DUE engineering Program Directors have had concerns about the cost-effectiveness and reach of these faculty development efforts, and they have urged the community to explore the use of Internet technology to create more cost-effective implementations that rely on the rapidly developing Internet conferencing and social networking tools for both synchronous and asynchronous interactions. After promoting these ideas for a number of years, DUE engineering Program Directors realized that their own outreach faculty development program aimed at increasing participation and improving the responsiveness of submitted proposals, had similar issues with cost and reach. They undertook an effort to convert their interactive workshops to a web-based approach to determine if the Internet could be used effectively in faculty development in their outreach. They also
1
hoped that success with this approach would encourage the community to explore the use of these web-based conferencing and social networking tools in their faculty development and dissemination efforts. Although the DUE engineering Program Directors recognized the community’s historical dependence on and strong preference for face-to-face models and skepticism about the effectiveness of web-based approaches, they felt that web-based approaches offered the only scalable approach for widespread faculty development. Thus, an effort was undertaken to examine the viability and faculty response to a program of web-based faculty development workshops. The purpose of this report is to increase the community’s awareness of this NSF interactive web-based workshop (IWBW) program, to share some of the evaluation data from these efforts, and in the process stimulate the community to begin exploring webbased approaches for faculty development and dissemination of the results of TUES projects. Virtual Approaches to Faculty Development In an earlier web-based effort, Courter and her associates5 described an on-line professional development effort involving 20 faculty members at 10 different colleges or universities. The effort included weekly conversations on readings selected by the rotating facilitators, and participants also developed and presented a project. The investigators reported 100 % retention of the participants with high participation at the weekly sessions and found the delivery mode convenient, efficient, and free of technical difficulties. Although the number of reported virtual faculty development implementations is small, a number of reports have discussed the advantages and characteristics of them for faculty and teacher development.4, 16,24 For example, Coto et al4 discussed how virtual approaches eliminate place-based restrictions and how technology makes the community of practice experience possible in a cost effective way to geographically-dispersed individuals. Earlier, Sherer and his colleagues24 noted that the use of the Internet, through a wide variety of services and tools, could overcome many of the limitations of faculty face-to-face communities of practice and enhance many of the advantages. Early efforts had more of a passive portal focus that managed vast amounts of information and linked a community easily and efficiently; whereas, later efforts utilized a highly interactive implementation enable by the evolution of the technology. More literature is available on the effectiveness of on-line learning models for students than for faculty development. The effectiveness of these models supports the premise that web-based approaches can provide effective and efficient learning experiences for faculty participants. A large scale, systematic meta-analysis of published reports of online learning models from 1996 to 2006 included only studies with random-assigned or controlled quasi-experimental designs and with objective measures of student learning, discarding studies based on perception data.18 The study, which identified only 45 reports over all education levels from K-12 through higher education to professional educational, showed a statistically significant improvement in learning with the on-line students
2
relative to the face-to-face students. However, some later studies suggested that the effectiveness of the on-line approach might depend on the students’ academic preparation.27 A systematic survey of Journal of Engineering Education articles published from 1999 to 2008 found only three articles that dealt with student collaboration and communication over the Internet.17 One compared performance and satisfaction of face-to-face and virtual teams in a random control experiment in a graduate course on lean manufacturing.26 The investigators reported similar performance on the project and in the group process patterns but the face-to-face teams were more satisfied with the experience. A second report, involving virtual and face-to-face design teams, found no difference in completion times and quality of task outcome.16 The third article reported that first-year students who participated in a learning community, where they communicated via email and Internet conferencing, had a better understanding and confidence in the technical aspects of the design project than those who did not participate.22 DUE’s Interactive Face-To-Face Faculty Development Model Outreach to the community is part of the NSF Mission and most Program Directors engage in faculty development efforts to expand the application pool and to improve the quality of submitted proposals. Over the last decade, DUE engineering Program Directors developed a series of interactive workshops related to the CCLI and later the TUES Program on various factors related to proposal development and project implementation, such as proposal writing strategies, project evaluation, and dealing with broader impacts. These were offered successfully at many national and regional meeting and other venues by many different program directors (see http://nsf.gov/events/event_summ.jsp?cntn_id=111743&org=NSF for an example of the workshop at the 2010 ASEE Annual Conference). After running these for several years, DUE engineering Program Directors noted that funded PIs had many questions about project evaluation and dissemination. In response to this perceived need, they began to offer these interactive workshops at NSF PI Conferences which are national meeting held for awardees in a particular program or set of related programs. The amount of participation and the evaluation results indicated that these sessions were very well received. When DUE engineering Program Directors began developing proposal writing workshops almost a decade ago, they set out to use real examples, to have participants recall their current understanding, to engage in discussion and reflection, and to share the perspective of experienced Program Directors with the participants. These workshops consisted of a series of short activities where participants considered some aspect of the larger topic. For example, in the proposal-writing workshop, the first activity focused on the project’s goals and expected outcomes, and this was followed a sequence of activities dealing with the project’s rationale, evaluation, broader impacts, and dissemination efforts. Each activity asked participants to respond to some question about a proposal section and usually took one of the following forms: outline that section for a
3
hypothetical project, critique that section from an sample proposal, write guidelines for the content in that section, or list the questions that reviewers’ would expect that section to address. We developed a three-step process where we asked participants to write down their personal response to the question (the Think phase), to share these with one or two others (the Pair-Share phase), then have selected individuals report to the group (the Report phase), which we referred to as the Think-Pair-Share-Report (TPSR) process. Once the participants had considered each question, the presenter summarized the response of a group of NSF Program Directors that had been developed earlier through a similar exercise. This model incorporated the attributes of an effective adult learning environment -presenter expertise, content relevance, application choices, praxis (action followed by reflection), and group work.26 Our approach also incorporated many of Felder’s guidelines for engineering faculty development: (1) use facilitators with expertise in both engineering and pedagogy, (2) use engineering-related examples and demonstrations, (3) target content to the needs and interests of the participants, (4) provide choices in applications of recommended methods, (5) model recommended techniques, (6) provide opportunities for formulating and practicing their own applications, and (7) actively engage participants.2, 7, 8 Topics presented at the workshops included: proposal writing strategies; introduction to the TUES Program; project evaluation and broader impacts (collectively and individually); and a mock panel review. The workshops were presented by DUE engineering and computer science Program Directors. DUE’s Interactive Web-Based Faculty Development Model Realizing that the reach of these workshops was limited to the small fraction of the community that could travel to meeting and conference sites, DUE engineering Program Directors began developing and offering web-based versions of these interactive workshops. In the face-to-face workshop, the presenter interacted directly with the participants and served as the facilitator for the TPSR activities by organizing the participants into small groups and guiding them through the activity. In transitioning to a web-based format, two major changes were needed. First of all, the new approach required a satisfactory Internet conferencing software system that would allow two-way communication between the presenter and the various participant sites. This did not pose a problem because a number of satisfactory systems were available. However, there were several logistical and technical difficulties that had to be overcome including managing the registration process for the workshops, converting to a voice-over-Internet protocol (VoIP), minimizing feedback from open microphones and telephones at local site, and facilitating the reporting process. The second (more major) change involved replacing the presenter’s role as the facilitator in the TPSR activities and identifying a local facilitator at each participating institutions to organize and lead the TPSR activities. One of the coauthors (Seals) received several small NSF grants to provide the needed logistical and technical support to recruit and register the sites and to coordinate the
4
interactions during the workshops. The workshops were developed and presented by DUE engineering and computer science Program Directors. Over a period of three years, they offered 24 interactive web-based workshops (IWBWs) aimed at engineering and computer science faculty. These treated 6 different topics with up to 35 sites participating in a session and, over all, the workshops reached over 1000 participants at more than 50 sites. Detailed guidelines for moderation at the local sites and copies of the PowerPoint presentations were provided to the local facilitators in advance of the workshops. Orientation sessions were provided to allow all sites to check the adequacy of their facilities. IWBW Structure: Each IWBW began with a brief introduction to the TSPR framework emphasizing the importance of participants’ engagement in a learning process and the role that active collaborative learning plays in this. Following this introduction, the presenter asked participants to take a few minutes and respond individually to a question intended to get them to reflect on their current understanding of the IWBW topic. These questions took several forms; for example “What are the most important aspects of the topic?” or “What would you tell a colleague to consider if she asked about the topic?” or “What confuses you about the topic” or “What do you want to know about the topic?” At the end of the IWBW, the presenter again engaged the virtual community in an identical reflection activity and had them compare their initial and final responses to recognize the changes or learning that occurred. After the initial reflection activity, the presenter indicated the goals and expected learning outcomes for the IWBW with the latter expressed in terms of what the participants will be able to do after the workshop. Participants then completed a sequence of activities using a modified TSPR process where the report was divided into two steps -- a local report coordinated by the local facilitator and a virtual report coordinated by the presenter. As in the face-to-face version, each activity concluded with a feedback session in which the presenter summarized the views of a group of NSF Program Directors that were developed earlier through a similar exercise. Each IWBW was divided into two 60-minute segments with a 15-minute break in between them. Typically, each segment had about four TPSR activities with a few shorter ones included to break the routine. The format allotted six minutes for each TPSR activity. Usually, two Q&A sessions were included in each segment with one in the middle and one at the end. Since the presenter had no control once a TPSR activity was initiated and could not react to question or provide guidance, the task statements needed to be clear and precise and describe challenging but doable tasks that fit within the time constraints. Local facilitators were encouraged to limit time on each of the three phases of the activity to about two minutes. Sample IWBW Activities: In order to give the reader some perspective on the nature of these activities, the next few paragraphs present some of those that have been used in the Project Evaluation IWBW. These activities were based on a brief description of a
5
hypothetical TUES project provided in the workshop material. The workshop developers realized that having participants work on their own project had some merit, but they believed that working on a common project made the discussion in the TPSR structure more focused and cohesive and the Program Director’s feedback more relevant. Also, many of the participants will have little experience with planning a project and would struggle to define a realistic project in a short amount of time. In the first activity, participants were asked to write a cognitive and an affective goal for the project. In setting up the activity, the IWBW presenter provided a brief introduction to goals and expected outcomes and additional explanation so that the participants understood the task. As feedback, the presenter provided a number of alternative goals for this project and a few of these were used in the next two activities. Participants were then asked to write expected measurable learning outcomes (again, some background was provided) for a cognitive and an affective goal (e. g., “Improve the students’ understanding of the fundamentals in engineering mechanics.” and “Improve the students’ self-confidence in solving engineering mechanics problem.”) and then to write an evaluation question for these two expected measurable learning outcomes. In later activities, participants were told to assume that they were considering the use of an existing instrument, such as a concept inventory, for use in their project’s evaluation and then asked to identify the questions they would consider in deciding if the tool was appropriate. (Note, a brief description of concept inventories was provided.) Next, participants were presented hypothetical pre- and post-intervention data on a few items (i. e., questions probing the understanding of a single concept) from a concept inventory instrument with a range of pre-post differences. They were asked which items indicated changes in student understanding and this led to a discussion of variability, statistical testing, and so on. For one item with a large pre-post difference, they were asked to identify some alternative explanations that could explain the change, leading to a discussion of confounding factors. In the final six-minute activity, participants were asked to identify the topics that needed to be addressed in a project evaluation plan. Evaluation Results A voluntary anonymous post-workshop survey of both participants and local facilitators (with both forced-choice and open-ended questions) was conducted after each session to evaluate the effectiveness of the approach. The survey results encompass responses from almost 600 participants and facilitators. Since Government regulations make it very difficult to collect survey data in an NSF organized session, there are no comparative data from the earlier face-to-face workshops. However, an analysis of the data from the webbased sessions indicated that the participants were satisfied with the format, and judged that the workshop learning outcomes were generally achieved. Participant Demographics: Table 1 shows the demographics of the participants that completed post workshop evaluations for all IWBWs. The distribution of ethnicity, race, and residence status generally matched the engineering faculty population. However,
6
females and Assistant Professors were somewhat over represented in comparison to the actual numbers in the faculty ranks. Table 1. Participant Survey on Demographics (N= 376) Gender
Ethnicity
Race
Residence Status
Academic Rank
Male
60%
Female
40%
Hispanic or Latino
6%
Not Hispanic or Latino
94%
American Indian or Alaskan Native
0%
Asian
25%
Black or African American Native Hawaiian of Other Pacific Islander
6%
White
69%
U. S. Citizen
72%
Permanent Residence
14%
Other non U. S. Citizen
13%
Ph. D. Student
6%
Post Doctoral Fellow
2%
0%
Instructor
5%
Assistant Professor
45%
Associate Professor
17%
Professor
24%
Table 2, which shows the distribution of the participants’ institutions by the Carnegie Basic Classification, indicates that all types of institutions were represented with a strong component of research institution, which is not surprising since most of the participants were from engineering programs, which are generally housed in research institutions. Table 2. Participants' Institution Type (N = 323) RU/VH: Research University (very high research activity)
20%
RU/H: Research University (high research activity)
19%
DRU: Doctoral/Research University
17%
Master’s L: Master’s Colleges and Universities (larger programs)
15%
Master’s M: Master’s Colleges and Universities (medium programs)
9%
Master’s S: Master’s Colleges and Universities (smaller programs)
7%
Bac/ A&S: Baccalaureate Colleges- Arts and Sciences
4%
Bac/Diverse: Baccalaureate Colleges- Diverse Fields
4%
Bac/Assocs: Baccalaureate/Associate’s Colleges
3%
Spec/Engg: Special Focus Institutions- Schools of engineering
3%
Responses to the survey question on whether the participants were from a Minority Serving Institutions (MSI) or not indicated that 40 % of the participants were from an MSI. Table 3, showing the participants‘experience with the CCLI and TUES program, indicates that a large majority had never submitted a proposal. This suggests that many of them probably had very limited exposure to the ideas presented in the workshops,
7
especially those dealing with project evaluation and broader impacts. Thus we were reaching an audience that could benefit greatly from the workshop. The degree of participant inexperience needs to be considered when looking at the intended outcome achievement data presented later. Table 3. Participants' CCLI and TUES Experience (N = 304) Current or Prior CCLI (TUES) Funding as a PI or Co-PI
9%
Submitted CCLI (TUES) proposals but not funded
14%
No prior submission of CCLI (TUES) proposals
77%
Facilitators’ Perceptions of IWBWs: The results provided in Table 4 shows the data on the number of participants at the local sites. The data indicate that nearly half of the sites had more than five participants, providing the opportunity for good discussions. Slightly more than half of the sites represented small groups with some of them being single participants, as noted from the site registration information. Table 4. Facilitators' Survey on Number of Participants (N = 264) 1116
