Development of a structured undergraduate ... - Wiley Online Library

Article Development of a Structured Undergraduate Research Experience: Framework and Implications

Anne M. Brown† Stephanie N. Lewis‡ David R. Bevan†*

From the †Department of Biochemistry, Virginia Tech, Blacksburg, 24061, Virginia, ‡Office of Undergraduate Academic Affairs, Virginia Tech, Blacksburg, 24061, Virginia

Abstract Participating in undergraduate research can be a pivotal experience for students in life science disciplines. Development of critical thinking skills, in addition to conveying scientific ideas in oral and written formats, is essential to ensuring that students develop a greater understanding of basic scientific knowledge and the research process. Modernizing the current life sciences research environment to accommodate the growing demand by students for experiential learning is needed. By developing and implementing a structured, theory-based approach to undergraduate research in the life sciences, specifically biochemistry, it has been successfully shown that more students can be provided with a highquality, high-impact research experience. The structure of this approach allowed students to develop novel, independent projects in a computational molecular modeling lab. Students engaged in an experience in which career goals, problem-

Keywords: undergraduate research; mentoring; activities; development and implementation

high-impact

Introduction Participation in an engaging undergraduate research experience (URE) can be an invaluable opportunity for undergraduate students to develop both academic and professional skills [1]. Undergraduate student researchers, defined as undergraduates who choose to participate in the scienVolume 44, Number 5, September/October 2016, Pages 463–474 Abbreviations: URE, undergraduate research experience; STEM, science, technology, engineering, and mathematics; URT, undergraduate research trainee; SURS, senior undergraduate research student; PI, principal investigator; GRM, graduate research mentor; CTC, critical thinking and research communication skills *Address for correspondence: 201 Fralin Life Sciences Center, Blacksburg, VA 24061. E-mail: [email protected] Received 11 September 2015; Revised 30 January 2016; Accepted 3 March 2016 DOI 10.1002/bmb.20975 Published online 28 April 2016 in Wiley Online Library (wileyonlinelibrary.com)

Biochemistry and Molecular Biology Education

solving skills, time management skills, and independence in a research lab were developed. After experiencing this approach to undergraduate research, students reported feeling challenged to think critically and prepared for future career paths. The approach allowed for a progressive learning environment where more undergraduate students could participate in publishable research. Future areas for development include implementation in a bench-top lab and extension to disciplines beyond biochemistry. In this study, it has been shown that utilizing the structured approach to undergraduate research could allow for more students to experience undergraduate research and develop into more confident, independent life scientists well prepared for graduate schools and C 2016 by The Internaprofessional research environments. V tional Union of Biochemistry and Molecular Biology, 44(5):463–474, 2016.

tific research process to gain experience and inform career goals traditionally in science, technology, engineering, and mathematics (STEM), can have a variety of experiences and outcomes. The implications of a research experience now extend to areas of medicine, business, and the arts as well [2]. While summer research experiences provide an abbreviated but in-depth exposure to the research process, greater benefits and gains have been observed in yearlong UREs, highlighting a need for more students to become involved in undergraduate research during the academic year [3]. However, increasing undergraduate student participation places a greater burden on faculty time and resources, ultimately leading to a small number of students in a given lab and increased competition for yearlong UREs [4]. In addition, little change or development in the basic structure and curriculum related to undergraduate research has been reported in the past decade, especially in the biomedical research environment [5, 6], despite the trend in technological advances and professional needs in the research arena. There is a need to increase the number

463

Biochemistry and Molecular Biology Education of students participating an academic year URE and to develop the URE to better fit skill-sets needed in a biomedical research environment and post-graduate education. There is a shift toward the mindset that UREs need to focus more on student interpretation of results and development of critical thinking skills, in addition to mastery of experimental techniques [4]. However, assessment of skills, level of competency gained, and overall perceptions from the URE are not well studied and often lack pedagogical backing [5]. In order to advance the state of UREs and improve student outcomes, a new approach is required to meet the growing demand from students and university initiatives. The presented approach identifies increased student participation and higher-level engagement as room for improvement in an URE and shows that with a structured, pedagogical-based approach to URE, more students are able to participate in the URE and develop both research methods and critical thinking skills at a higher level than before implementation.

Implications for Increased Student Participation in an URE Studies suggest that UREs are high-impact activities that improve the educational outcomes of students who participate [7]. This suggestion is also supported by student responses from an alumni feedback survey, which found that students who participated in UREs reported higher satisfaction and overall personal growth as compared with those who did not participate [5]. Additionally, students who participated in novel research exploration derived a greater understanding and appreciation for the research process and application of basic science knowledge, while contributing something worthwhile to the scientific community [8]. Depending on the type of institution and goals of the department curriculum, there are multiple approaches to undergraduate research. These approaches have been detailed and state the variety of benefits for having students being both active and passive in the URE, [9, 10] and how UREs can fit into the curriculum in a variety of ways.

Current Obstacles Teaching and research are often isolated endeavors at research universities [11]. As described by Elsen et al. (2009), if universities strengthen their “research-teaching nexus,” then more students will be able to participate in the URE, regardless of discipline [12]. Healey (2005) proposed that there are two main focus areas in most UREs, with the experience usually favoring one rather than both. A focus on content in an URE is geared more toward written and oral presentation deliverables, while a focus on research process is attentive to student engagement in all

464

parts of the research cycle with students leading and designing experiments [10]. Furthermore, there is a need to address the pressures surrounding undergraduate research, as defined by Beckman and Hensel [13]. Such pressures include the URE being either student initiated or faculty initiated, biased at student levels (first-year vs. senior, honors vs. non-honors students), and focusing on either the process or products of research rather than learning outcomes. In addition, universities often lack the resources to involve a majority of the undergraduate population in undergraduate research [14, 15]. The ratio of total student population to number of available lab positions, which is based on materials, lab space, mentoring capacity, and number of projects, translates to a small percentage of students who can participate in an engaging URE for an extended period of time. With these obstacles present in most undergraduate research programs at both the departmental and university level, we sought to explore a method that would address both sides of the pressures mentioned above as related to the life sciences field. The promotion of similar teaching pedagogies across disciplines and better discourse on how to improve the overall undergraduate research environment in the life sciences can be achieved [9]. Training effective and successful junior research scientists should incorporate independent research projects and reflective discussions on how students learn in order to improve the efficacy of the research training experience and the perceptions of the students regarding that experience [11]. Conversations with students regarding their expectations for research experiences in the biochemistry field led to the development of a program within the Bevan Molecular Modeling Lab at Virginia Tech, which is a researchintensive university. These conversations provided insight into the varying experiences across campus for undergraduate students. The responses were in alignment with reports that stated that students had difficulty learning research methodology and that varying levels of autonomy greatly influenced the URE [16, 17]. In order to fully prepare students for their next career stage, students needed to participate in both focus areas proposed by Healy [10]. To this end, concepts in educational theory should be incorporated into the URE and underlay the basis of our framework for increasing the number of students participating in a high-quality URE. The purpose for developing this structured practice and framework was to provide a consistent, individualized, self-sustaining research experience that trains undergraduates to become independent researchers in the life science fields.

Development of an URE Framework to Increase Student Participation and Improve Experience Outcomes The purpose of this article is to discuss the method of implementation of a student-centered research program in

Development of a Structured Undergraduate Research Experience

an individual lab that is backed in theory by a studentcentered framework [9]. We sought to increase the number of students participating in a high-quality URE. In our report, a high-quality URE experience is determined both by student feedback and analysis of critical thinking skills presented in final deliverables as determined by professionals in the field. In addition, we provide resources and detailed discussion of such implementation of a studentcentered research program and how it can be implemented on the individual lab, departmental, and college-wide levels. The Bevan Molecular Modeling lab has been established as an environment for computational modeling that encourages collaboration with experimental labs. Molecular modeling techniques have increasingly been recognized as a means to predict biological outcomes and elucidate details at an atomistic level. The lab participates in many collaborative projects involving therapeutic drug discovery and development for cancer, allergies, and chronic inflammatory diseases. The breadth of projects and speed of the techniques makes the lab perfectly poised to provide a 1- to 2-year research experience for undergraduates. Projects can be easily divided into segments that can be accomplished by a single student, a group of students, or a series of students over longer periods of time. The established collaborative environment is conducive to assembling the proposed URE structure. In order to address the obstacles described above and to engage more students in a student-centered URE in the life sciences field, the model introduced and described by Brew in 2013 [9] was used as a template. Adjustments to the design included incorporation of students at all academic levels (first-year to senior) into the URE and establishing autonomy in the laboratory across the levels. In addition, various pedagogical-based implementations were made, specifically related to learning objectives and assessment. Explicit attention to implementation and assessment of curricular and pedagogical goals needs to be determined before any type of implementation of an URE. By informing the students of the accepted practices, purpose and goals of the URE, the individual lab/department/college can better meet the educational needs and expectations of students who would be apprehensive of such experiences. Establishing a self-sustaining, self-propagating lab environment allows students to become experts who can in turn train more students in the lab. At the onset of the implementation of this framework, the “structured” approach came from the idea that students are accustomed to course syllabi and the standardized outline for courses. At the start of every semester, all courses disseminate a syllabus that gives the student a timeline with course expectations. However, this practice is often neglected for an URE because it is not a traditional course or course environment. Most undergraduate institutions utilize a system in which undergraduate research is done for credit and graded (A–F). While it is necessary to

Brown et al.

be mindful of different strategies to assess undergraduate research across disciplines, establishment of a standard approach for the department, especially related to assessment, will benefit students and set a baseline for learning outcomes and graded assignments [9]. A standard set of expectations across a department/discipline, which is common practice for traditional courses, would negate variations in expectations while improving learning outcomes. In this regard, we have structured the approach to undergraduate research to be consistent with a typical course format while maintaining elements that make the URE autonomous, student centered, and focused on development of skills related to critical thinking, oral presentation, and written dissemination of scientific work. The following sections breakdown the structured URE developed to increase student participating and engagement. These sections provide detailed rationale for incorporation into the URE.

Syllabus When developing the syllabus, the first goal was to move away from an unstructured research experience, which is defined as students working in a lab on an individual project with no explicit knowledge of learning objectives, grading system, or overall timeline. A syllabus was established to eliminate “unknowns” for the students. The syllabus was structured to allow students to have the experience of executing independent research projects and partaking in research process workshops. Learning objectives were stated (Table I), and a breakdown of graded elements and assessment methods, deadlines for deliverables, and details on weekly workshops were provided. A lab objective was stated for the URE and gave students perspective into the types of approaches used and the overall research focus for the lab. “Undergraduate research experience for students in their freshman, sophomore, junior, and senior years of study. Bevan Lab focuses on using computational biology and bioinformatics techniques, including molecular dynamics, molecular docking, homology modeling, and biomolecular visualization, to understand protein structure and dynamics, and aid in novel drug design.” The learning objectives for the course were also discussed and related to both the individual projects of the students and the deliverables each student was expected to submit to the graduate students and PI. All assignments were used to assess progression of learning and retention of research process as well as individual project knowledge. A proposed agenda of the weekly group meetings was also included in the syllabus and gave students the opportunity to see what topics would be discussed, plan their presentation dates, and have input into what topics are presented at the weekly workshop sessions. Deliverables,

465

466 Compose and present a research poster at university undergraduate research symposium.

Present at a research symposium.

Learn pedagogical practices.

Coursework in pedagogy courses and/or workshops at the university or at conferences.

Discussions with SURS about mentoring techniques.

Develop mentoring skills.

Develop teaching style.

Discussions with SURS about research progress.

Collaborate with peers in lab.

Progression of research projects by URTs and SURS in the lab.

Peer evaluation rubrics completed during presentations.

Critique oral presentations of peers.

Manage projects and personnel.

Final research presentation at the end of the semester.

Present scientific findings in oral format.

Conversations with PI about mentoring techniques. Participate in mentoring workshops at the university or at conferences.

Check point documents during semester: Abstract, Introduction, and Final paper.

Present scientific findings in written format.

Develop mentoring skills.

Research notebook checks at weekly meetings.

Weekly check-in meetings with GRM and PI.

Collaborate with peers in lab.

Maintain a lab notebook.

Peer evaluation rubrics completed during presentations.

Critique oral presentations of peers.


Final research presentation at the end of the semester.

Present scientific findings in oral format.

Progress toward completion of independent research project.

Check point documents during semester: Abstract, Introduction, and Final paper.

Present scientific findings in written format.

Participation in weekly workshops and presentation of one workshop.

Research notebook checks at weekly meetings.

Maintain a lab notebook.

Learn basic research practices.


Progress toward completion of independent research project.

Each investor (column 1) participates in research following a list of learning objectives (column 2) and is assessed (column 3) continuously throughout the semester.

Graduate Research Mentor (GRM)

Senior Undergraduate Research Student (SURS)

Participation in weekly workshops.

Assessment

Learn basic research practices.

Learning objective

Learning objectives for participants and mentors in the URE

Undergraduate Research Trainee (URT)

Investor

TABLE I



TABLE II

Grading rubric to assess content of URE final papers

such as the final paper and final presentation, required the generation of grading rubrics (Tables II and III), which were provided to students at the beginning of the semester.

Brown et al.

Providing the rubrics early conveyed clear expectations with established grading metrics. This aided in the students’ progression throughout the semester and improved

467


TABLE III

Grading rubric for URE final presentations

the quality of their check-in deliverables (e.g., the abstract and introduction section for the final paper turned in half way through the semester). We also sought to maintain elements in our rubrics that would be essential to students in

468

other molecular biology and biochemistry courses. Some elements that were easily transferable to other coursework were the use of proper citations, development of a scientific voice, and practice in writing the methods sections of


TABLE IV

Proposed schedule for fall semester

Week 1

Orientation: syllabus, introductions, questions

Week 8

Graduate student presentation

Week 2

Lab notebooks, research goals, file organization

Week 9

Graduate student presentation

Week 3

Predatory reading, How to write Abstracts & Intros, Journal club paper assigned

Week 10

Undergraduate Presentations (3)

Week 4

How to give a good presentation

Week 11


Week 5

Journal Club – based on paper assigned week 3

Week 12


Week 6

Fall Break

Week 13


Week 7

Peer review of Abstracts/Intros

Week 14

FINAL PAPERS DUE

manuscripts. Such skills are required for courses and associated labs, and tied their experiences in the research lab to experiences in their traditional core courses.

Grading and Assessment A section on grading and assessment was added to the syllabus for two reasons. The first reason was to provide the student with a clear understanding of expectations for the URE and what would constitute the grade received at the end of the semester. Secondly, setting comprehensive expectations would allow for grade equality regardless of academic level and number of credits enrolled for the URE. If a student is not performing as expected, the faculty has a clear “contract” with the student indicating what a student should expect to gain from working with the faculty member and his/her lab. If there are issues in participation and performance, these issues can be quantified as a grade, which students are accustomed to interpreting as an assessment of their skills development process. In the URE discussed here, the grade distribution was as follows: Notebook Documentation (20%), Participation in lab and lab meetings (20%), Final Presentation (30%) and Final Paper (30%). Participation was defined as “attendance at all group lab meetings, substantial progress on individual research projects, and clear communication with mentors on any issues.” The inclusion of participation in the grade clearly conveyed the level of effort needed to fully contribute to the URE and see academic and professional gains. The deliverables require proficiency in the research process (i.e., writing, discussion, methodology, and data analysis), understanding of project-specific biological facts, and critical thinking in the research process. These deliverables provide the foundation of combining the multiple levels of an URE as discussed in the Current Obstacles section. This learning process and assessment is denoted in the

Brown et al.

grading rubrics provided for the final paper and final presentation assignments (Tables II and III). As a complete document, the syllabus addresses the goals of UREs: students should be held accountable for the critical thinking aspects of the project and the methodological skills development necessary to become well-rounded scientists of the future [4].

Group Meetings Weekly lab group meetings focused on honing research process skills while providing a platform for students to practice oral presentation skills. The weekly workshop allowed for the dissemination of key information related to the progression of the student in a manner that would require minimal repetition by the PI. For example, instead of detailing how to write an abstract for students in a oneon-one meeting, all students participate in a group discussion about the writing process. These workshops could be facilitated by a graduate student or a senior undergraduate student. Scheduling such sessions meant that interactions with the PI were reserved for individual meetings that included meaningful conversations about the progress of individual research projects. Additionally, students were able to exhibit research process and content proficiency through using these presentation opportunities to prepare for honors thesis defenses and professional conferences. An example agenda is shown in Table IV with key workshop topics related to building skills in the research process. Once faculty members commit to using a structured approach to UREs, a standard syllabus and schedule generated for the fall and spring semesters can be used in subsequent years. Topics for workshop topics can include professional development (scientific writing and presentation skills) and techniques specific to the research lab. For example, a spring workshop on a commonly used technique in lab, Basics of Molecular Dynamics Simulations,

469

Biochemistry and Molecular Biology Education was given to students in order to improve their base knowledge on techniques and jargon in the field. Potential topics for other lab environments could include review of titrations and making buffer solutions, mass spectrometry, and high-performance liquid chromatography. By presenting this information to the group and allowing for discussions where questions about the topics can be addressed, more students benefit than in one-on-one conversations detailing these techniques. Once the general framework for the URE is established, the workload for implementation becomes very manageable. One critique of establishing a single agenda to be reused over multiple years is that some students will redundantly experience the same topics multiple times. In the URE implemented and discussed here, students reported enjoying multiple workshops on the same topic because it helped them master the skill. In addition, senior students in the lab can lead workshops, thereby giving them the ability to develop a deeper mastery of the subject. The usage of senior students in the teaching process can also lead to positive peer-to-peer mentoring relationships in a collaborative environment.

Mentorship Undergraduate research students were broken down into subset groups based on level and duration in the lab: undergraduate research trainee (URT) and senior undergraduate researcher (SUR). Each student level had specific learning objectives and deliverables that focused on the progression of the student’s own undergraduate research project, the development of scientific writing and presentation skills, and mentoring (Table I). Graduate research students were also included as a group, classified as graduate research mentor (GRM). In this URE, SURs mentored URTs, graduate students mentored both levels of undergraduate students, and the PI mentored all students (graduate and undergraduate). An added benefit to this mentoring framework is that the research experience and mentoring support from both principal investigators (PIs) and graduate students is thought to be supportive of women and minority students, giving them access to faculty and peers in the field [18]. Our structure allows graduate and undergraduate students in the sciences to benefit both academically and professionally from a research mentoring relationship. It was important to the development of this URE to include graduate students as research mentors in order to provide career preparation, teaching, and communication skills development. In the role of research mentor, graduate students have the opportunity to discuss and develop pedagogical techniques relating to teaching undergraduate students about research methods. The benefits of being a mentor and mentee for both undergraduate and graduate students has been explored previously [19] and was observed in stu-

470

dent feedback from our preliminary study. The undergraduate mentee sees senior undergraduates as peers who have achieved a breadth of knowledge in their major that they may aspire to achieve. The senior undergraduate mentor gains mentoring experience while achieving an increased level of understanding of a scientific subject (e.g. biochemistry). The graduate student as a mentor participates in practical training for effective research mentoring and project management. The graduate student presents his/her research to the undergraduate students, which serve as an example to the undergraduate students of graduate level achievement while providing the chance for professional development. As a mentee, the graduate student is able to explore career option conversations with the PI to assess how working with undergraduate researchers can add to his/her professional skill set. The PI maintains the benefits of providing UREs to more students, training and graduating research-focused undergraduates, potential matriculation of undergraduates into graduate programs at the home institution, and graduating pre-doctoral candidates prepared to succeed as science professionals.

Effectiveness of Structured URE Assessment of Scalability and Improved Student Outcomes As the students progress through the semester, the learning process should ensure that the final product for the semester is a reflection of how much a student has advanced in learning the research process. The URE method presented was tested on a small pilot group of eight participants during the fall semester of 2013. This group of students was selected for survey because half had been members of the lab before the implementation of this method and other students were new to the lab as of fall 2013 (defined as pre- and post- implementation). Surveys were given to all students who participated in the URE during the 2013–2014 academic year. Survey questions were composed to determine if the URE influenced academic performance in core courses, gain feedback on changes related to critical thinking, assess gains in scientific paper comprehension and analysis, assess progression in scientific writing and presentation skills, and gauge comprehension of project background and scope. Analyses on the outcomes for this URE are based on qualitative methods and review of student survey data, in addition to outside review of student deliverables pre- and post- implementation of this structured method. Taken together, these analysis metrics provided valuable insight regarding the effectiveness of the structured approach. The preliminary observational study discussed in this article was conducted on achieved student deliverables. The student artifacts were obtained prior to establishing a need for research data on a structured method. As the method was developed and a need to quantitatively assess


FIG 1

Quantified outcomes of structured UREs. The frequency (y-axis) of students in each category per cohort year (x-axis), where the number students for a given year (blue bars), number of students participating in conference (green bars), and number of students listed as authors on peer-reviewed publications (gray) are shown. The number of publications includes manuscripts that have been published, submitted, and that are in preparation. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

effectiveness arose, it was determined that approval for work with human subjects should be obtained. The reflection questionnaire and purpose for formalizing the collection of additional student data were submitted to the institutional review board (IRB) for human-subject research

FIG 2

Brown et al.

and was approved for further data collection (IRB# 14680). The posed reflection questions about the student experience were made a part of the reflection process for SURs, and student deliverables were scored for data collection after students completed their UREs. Of the deliverables completed by each student each semester, the final papers are the strongest indicator of the scientific work completed and knowledge gained from the provided experiences. The final papers were used as a tool to assess the development of critical thinking and research communication skills (CTC) (Table V). A score was assigned to a subset of the papers submitted by students who were in the lab during the transition to the updated URE approach (Fig. 2A) and two groups of students who were in the lab either before or after the transition (Fig. 2B). We saw an increasing trend in CTC score in both groups of data. We also calculated a statistically significant difference between the before and after scores. These data considered together suggest that students perform at a higher level of proficiency, as determined by outside review, under the updated URE structure. Finally, another measure of effectiveness for this URE approach is the ability to accommodate an increased number of participating undergraduate students compared with our previous approach. Between 2010 and 2015, we were able to increase participation from 4 to 15 undergraduates ranging from first-year students to seniors, an increase of almost three-fold over 5 years (Fig. 1). In addition, we were able to increase the number of students presenting research at professional conferences and co-authoring peer-reviewed publications (Fig. 1). The method discussed in this article was first implemented in 2013 as an update to previous practices that worked for junior- and senior-level students on a small scale. We saw the need to also open up research opportunities for first-year

Comparison of critical thinking and communication (CTC) scores. The CTC score is based on an assessment rubric applied to the final papers submitted by students. Each paper was scored by three individuals and the average scores are reported here. (A) Scores for the same student (each line) before and after implementation of the proposed method (N 5 2). (B) Scores for different students (each circle) before and after implementation of the proposed method (N 5 4). Standard deviation for each point is represented by an I-shaped error bars. The asterisk (*) represents statistical significance (Student’s t test, p