Session 13c3 Assessment of Technology-Based Learning ... - CiteSeerX

Session 13c3 Assessment of Technology-Based Learning Tools in an Introductory Physics Laboratory Teresa L. Hein, Sarah E. Irvine Department of Physics/School of Education American University Washington, DC Abstract - The computer and other related technologies are currently being used with vigor within many domains of science, mathematics, engineering, and technology (SMET) education. In this paper we will address strategies designed to assess student learning following instruction that utilizes technology-based learning tools in the introductory physics laboratory. To address this issue, we will describe an interactive laboratory experiment designed to teach the concept of momentum and impulse to introductory physics students. The laboratory activity makes use of a collision apparatus and computer interface to allow students to collect relevant data. An interactive software tool allows students to perform a series of analyses of various elastic and inelastic collisions. We will link the discussion of assessment of student learning to learning gains to address how the use of technology as an interactive learning tool impacts student learning. The associated discussion should have broad applications in a wide range of areas in science, mathematics, engineering, and technology education.

Introduction A growing number of technology-based educational learning tools currently exist within the domains of science, mathematics, engineering, and technology (SMET) education. In addition, the use of educational technologies is growing both in and out of the classroom and laboratory. Certainly technology has the potential to serve as a powerful tool to improve the educational process for students as well as teachers 1. However, educational technology is only as good as the content it supports 2. Therefore, it is important to address such issues as learning goals and curriculum objectives before one implements any form of technology as a learning tool. Many traditional teaching methodologies have clearly been shown to put students in the role of passive rather than active learning 3. Traditional instructional methods have also been shown to be inadequate in terms of promoting deep learning and long-term retention of important physics concepts. More often than not physics is taught in a typical lecture-style format in which the instructor provides information to the students by talking to them. This style of instruction focuses primarily on the instructor, the only active participant in the class. Although optimum for some, this mode of instruction is deficient in many ways for most

students. One outgrowth of much research in physics learning is the basic idea that in order for meaningful learning to occur, the learner must be given the opportunity to actively interact with the material to be learned 4 – 6. The explosion in the availability of technological tools is literally forcing those who teach physics as well as other SMET educators to change the way they instruct students. These changes, however, must involve much more than simply implementing technology for technology’s sake. The recent advances in computer-based technologies and their use in SMET education provides an opportunity for educators to take a critical look at how these tools are being integrated into the classroom and laboratory. Research has shown that these technological tools can only be effective in promoting student understanding if used in a pedagogically sound way 7. Furthermore, once we have established what it is we want our students to understand, we must determine a way to help him/her do just that! The use of technology may serve as one tool to help students gain the knowledge that has been ascertained as important. It is essential to note at this point that the integration of computer-based technologies into the classroom and laboratory is not enough. Strategies must be employed which are designed to assess student understanding following the use of any new type of learning tool, computer-based or otherwise. Furthermore, effective strategies must be developed and implemented to assess overall student learning gains. Teachers need to make wise decisions about how and why computer-based technology should be used in the classroom 8. Along with providing students an opportunity to be more active learners, the use of computer-based technologies in the classroom and laboratory may also help educators accommodate a wider range of students’ learning styles. A growing body of research on adult learners suggests that increased learning gains can be achieved when instruction is designed with students’ learning styles in mind 9 - 11. In addition to increased learning gains, when students are given the opportunity to learn new material through their individual learning styles they often become more motivated to learn. Thus, the use of computer-based technologies has the potential to provide other benefits to students by offering them a different type of learning tool – which may serve to accommodate a wider range of learning styles.

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-1

Session 13c3 One purpose of this paper is to describe a computerbased learning strategy employed in the introductory physics laboratory at American University. A primary focus will be on the assessment of student learning following implementation of the computer-based tools. Several strategies were developed to assess student understanding. A brief description of the assessment strategies will be provided along with a summary of results obtained to date. This paper will be concluded with some suggestions for continued study in this growing area of educational research.

The Introductory Physics Course at American University The introductory course for non-science majors at American University in Washington, D.C. is a one-semester, algebrabased course and is entitled Physics for the Modern World. Topics covered in the course typically include Kinematics, Newton’s Laws, Conservation of Momentum and Energy, Rotational Motion, Fluid Mechanics, Waves, and Sound. Although traditional in its content, the course is not taught in a “traditional lecture format.” Numerous teaching strategies have been developed which correspond to the accommodation of students’ diverse learning styles 12. In addition, the course includes both strong conceptual and problem solving components. Physics for the Modern World is a 3-credit course and consists of both a lecture and a laboratory component. Students meet twice a week for class sessions that are 75 minutes long. On alternate weeks students meet for a twohour laboratory. Approximately 120 students, with 60 students in each of two sections, enroll in the course each semester. Many students who enroll in Physics for the Modern World are liberal arts majors. A typical class consists of a mixture of students from the College of Arts and Sciences, the School of Public Affairs, the School of International Service, and the Kogod College of Business Administration. Students enroll in Physics for the Modern World to satisfy a portion of the Natural Science requirement for graduation at American University. Students may choose to satisfy this requirement with a general Physics, Chemistry, Biology, or Psychology course. Due to the wide range of majors in the course, one could assume that the diversity of students enrolled in Physics for the Modern World closely parallels the diversity of students enrolled at American University. The 1995 - 96 American University catalog describes its student population as being “... cosmopolitan and multicultural ...” 13. There were 110 students enrolled in Physics for the Modern World during the spring 1999 semester. This number included students from 24 states and 25 countries. Nearly 40% of this class was made up of international students.

Description of the Laboratory Activity The laboratory activity developed for use in this study is entitled Conservation of Momentum and was designed to help students learn the concepts of momentum and impulse. The apparatus involved the use of a dynamics cart accessory track set available from Pasco Scientific 14. This apparatus includes a 1-m track with a set of two gliders used to create elastic and inelastic collisions. Photocells interfaced to a computer were used to help determine the velocities of each glider before and after the collisions. The software package used is called Logger Pro 15. In the Conservation of Momentum laboratory activity, students made use of the track and gliders to establish elastic and inelastic collisions. Students also made use of the photocells and software to help them determine the velocities of the gliders before and after each collision. Knowing the masses of the gliders and their corresponding velocities, students were able to determine the total momentum of their system before and after each collision. From this determination, students could verify that momentum has been conserved for each collision. Of particular interest is student ability to understand the entire collision process. This process involves a fundamental understanding of the force and time involved for each collision. The product of the force and collision time yields a quantity called impulse. Impulse, by definition, is equal to change in momentum. In addition to understanding the concept of momentum conservation, students are also challenged to demonstrate their understanding of impulse in a collision. The concept of impulse is one that tends to be problematic for many students.

Assessment of Student Learning: Data Collection Strategies The assessment strategies designed for use in this project were both formative and summative in nature. Formative data was collected from student writing activities and through a specially designed post-lab activity. Students were asked to demonstrate their understanding of the collision process, in writing, before and after they had completed the formal laboratory activity. These activities were designed, in part, by adapting classroom assessment techniques developed by Angelo and Cross 16. Cross 17 has shown that the use of classroom assessment techniques provides powerful assessment information while the learning is actually taking place. Students in this study were presented with two writing activities. The first writing activity was given before they had received any formal information on the concept of momentum and the collision process in general. Students were given a second very similar writing activity after all

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-2

Session 13c3 formal classroom and laboratory instruction was completed. In this way, students’ pre- and post- conceptions regarding the concept of momentum and their understanding of the collision process were assessed. A post-lab assessment activity was also given to all students immediately after they had completed the collision laboratory activity. This activity was designed to assess whether their experience with the collision laboratory experiment had enhanced their understanding of the collision process. Summative data was collected from a more traditional assessment measure, namely the second classroom examination. The examination was given to all students after they had completed both of the writing activities, the laboratory experiment, and the post-lab activity.

Results In this section of the paper each of the assessment measures utilized will be briefly described. Following this description, a summary of results obtained will be presented. In addition to the summary, a typical student response to each of the assessment measures will be presented. Note that each response shared in the sections that follow came from the same student at different stages of the learning process. This will facilitate a demonstration of how student understanding was enhanced before, during, and after the collision laboratory experiment had been performed. The Writing Strategy: Assessment of Student Conceptions The writing activity used to assess student understanding is called a folder activity and was developed by the lead author for use with introductory physics students 18. As part of their homework assignments, students are required to keep a twopocket paper folder. Students make one entry in their folders approximately every other week. Students receive specific written feedback designed to help them uncover potential problems in their understanding of a particular concept or idea. Students are encouraged to reflect on the feedback they receive and then work to confront any potential difficulties they are having with a particular concept. When students take time to reflect on their writing and on the instructor-given feedback, the folder becomes a highly effective tool in helping them uncover and then wrestle with their misconceptions while the learning is taking place 19. The writing activities provide a window into understanding how well students are integrating the new knowledge into their existing knowledge schemas, thus providing a valuable tool in encouraging deeper understanding and retention. The folder activities are not graded based on students’ correct use of physics but are still a required component of the course. Essentially the folder

activities provide students an opportunity to make mistakes, confront, and then correct them before they are asked to perform on a quiz or exam. An additional aspect of the writing activities is that students are permitted to be as creative as they would like to be when they submit these assignments. Students are encouraged to write their responses in a fashion that allows them to make use of their individual learning styles. For example, some students like to enhance their writing through the use of manipulatives and artistic drawings. Other students might choose to write their responses in the form or a story or short play. The students know that they have complete control of this activity and are free to put their individual learning styles to good use! For the current study, students were given a writing activity associated with the concept of momentum before they had performed the actual laboratory experiment, and before this topic had been addressed in class. In this manner, student misconceptions that were present before the laboratory activity was performed could be uncovered. This writing activity served as a pre-assessment of students’ understanding of the collision process before they had received formal instruction on this topic. The first writing assignment given students was as follows: If a Mack truck and a Honda Civic have a head-on collision, upon which vehicle is the impact force greater? Which vehicle experiences the greater acceleration? Explain your responses. Next, do your best to thoroughly describe what happens to the momentum of each vehicle before, during, and after the collision. The correct response to this assignment is that the force on each vehicle should be the same via Newton’s 3rd Law. Further, the Honda Civic should have the greater acceleration due it’s smaller mass (via Newton’s 2nd Law). In addition, the momentum of each vehicle changes by the same amount during the collision. Student 1’s response to these questions is representative of the responses received by many students and is presented below: “In a head-on collision between a Mack truck and a Honda Civic, the impact force would be greater upon the Civic. The truck has the greater mass and therefore the greater force (F = mass × acceleration). If the vehicles have the same acceleration before the collision, the car will undergo a greater impact force due to the greater mass of the truck. The car also experiences greater acceleration. This is due to the fact that all forces in nature have an equal force acting back on it. So as the truck has a greater force on the car during impact due to its greater mass, the car has a greater force after the force with a greater acceleration. The larger mass of the truck and the larger acceleration of the car

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-3

Session 13c3 balance out the forces of the collision. Before the collision, momentum is unchanging in both the truck and the car if friction is not considered. Momentum (M = mass × velocity) is constant because the masses and the velocities of the vehicles were unchanging. During the collision, the momentum is still the same. The velocity is decreasing but the mass has increased because for a moment the two vehicles are connected and the mass becomes greater. After the collision, the vehicles separate and their masses return to the same quantity as before the collision. Their velocities are decreasing because of the exerted forces and thus the momentum is decreasing.” In general, responses received from students clearly indicate that some common misconceptions existed in their understanding of the collision process before they had received any type of formal instruction. In fact, upon inspection of the initial responses to this activity, only one student was able to write a flawless response to the above questions at this point in the learning process. The most common misconceptions involved confusion between the mass of the vehicle and the magnitude of the impact force during the collision. Many students felt that the Mack truck would exert a greater force on the Honda Civic because of its larger mass. Most students had some idea about the concept of momentum, but were unable to extend this to an actual head-on collision. Since this writing assignment was given before students had received any formal classroom or laboratory instruction on the concept of momentum, it is not surprising that many students had difficulty with it. The second writing activity was given students after they had received formal classroom and laboratory instruction on the topic of momentum. The assignment was as follows: If a Mack truck and a Honda Civic have a head-on collision, which vehicle will experience the greater force of impact? The greater impulse? The greater change in momentum? The greater acceleration? The correct response here is that the truck and the Civic will experience the same impact force (via Newton’s Third Law). Since the force on each vehicle is the same, and the contact time during the collision is also the same, then each vehicle will also experience the same impulse (I = Fave∆t). Since impulse is equal to the change in momentum, each vehicle would also experience the same change in momentum (I = ∆p). The Civic will experience the greater acceleration, however, due to its smaller mass (via Newton’s Second Law). Student 1’s response to these questions is presented below:

“According to Newton’s Third Law, there are action and reaction forces and these forces act in pairs. The collision between the Mack truck and the Honda Civic creates an action reaction pair of forces. These two forces are equal in magnitude and opposite in direction. They exert the same impact force on one another. The impulse, or change in momentum, is dependent upon the mass and the velocity of the body in motion. Impulse can also be calculated with the average force and the time. Since the forces exerted are equal and the vehicles are in contact for the same period of time, the impulse will be equal. The change in momentum is equal to the impulse, so therefore the changes in momentum would also be equal. The Civic would have the greater acceleration after the collision because acceleration is equal to the net force divided by the mass. The two vehicles have the same net force, but the Civic has the smaller mass. Thus, the Civic has the greater acceleration because the net force is divided by a smaller value.” As this response illustrates, Student 1 was now able to give a correct response to each of the questions asked. In fact nearly half of the students were able to give a flawless response to these questions on the second writing activity. Since there were essentially no completely correct student responses given to the first writing activity, this is certainly a substantial improvement. Many students who failed to give a correct response to this activity failed to see that the time given in the impulse relationship is the contact time during the collision and not the time it takes for each vehicle to come to a stop after the collision. Thus, the writing activity permitted the elicitation of students’ remaining misconceptions regarding the collision process after formal instruction had been completed. Once their remaining misconceptions were uncovered, the feedback provided by the instructor (lead author) assisted students in confronting and dealing with them. Had the students not been able to share their understanding (or misunderstanding as the case may be) in writing, these remaining misconceptions may not have been detected until the classroom examination had been taken. This is obviously much too late for the students. The Post-Lab Activity Immediately following the laboratory experiment, students were asked to respond to some questions on a post-lab questionnaire. Furthermore, students performed the laboratory activity after they had completed the first writing assignment and before they had completed the second writing assignment as described above. Several questions were presented to students on this questionnaire. The two most pertinent questions have been combined and summarized as follows:

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-4

Session 13c3 The momentum of each glider changed during the collisions you performed. In physics we call the change in momentum of an object “impulse.” How does the impulse of glider 1 acting on glider 2 compare to the impulse of glider 2 acting on glider 1 in each collision? Are they the same magnitude or different? Do they have the same sign or different signs? Furthermore, impulse can also be defined as the force acting on an object multiplied by the time through which the force acts. In light of this definition of impulse, how does the impulse of glider 1 acting on glider 2 compare to the impulse of glider 2 acting on glider in each collision? Are they the same magnitude or different? Do the have the same sign or different signs? Explain. Many students had difficulty responding to these questions at this point in the learning process. However, their responses did show that progress was being made. It is important to note that the concept of “impulse” had not been formally discussed during class before students had performed the laboratory activity. Hence, it is reasonable to expect students to have some difficulty responding to these questions. Student 1, however, was able to give a reasonable response to this question: “The impulse depends on the mass and velocity of each glider. They have the same magnitude but different signs. The change in momentum, the impulse, is the same for both. But the two gliders have different masses and velocities. The two impulses are equal. The forces and the times are equal for both gliders.” The Classroom Examination As a final check on students’ understanding of the collision process, the same questions given students on the second folder activity were repeated on the second classroom examination and will not be repeated again here. Results from the student responses to this examination question show that over half of the students received perfect marks on these questions. No student missed all of the questions. In fact, most students who had a flaw in their thinking received at least half of the points allotted for this question on the exam.

Summary Technology, such as that used in this computer-based laboratory activity, must be carefully evaluated before being implemented into the curriculum. Once effective implementation is achieved, strategies designed to assess student learning must be developed. The assessment of student understanding as well as learning gains is a critical

component of any instructional strategy. When new computer-based technologies are utilized, these assessments become a vital and necessary component. The assessment tools developed for use in this study made it possible to monitor student learning gains throughout the entire instructional process. In this way students were given multiple opportunities to expose, confront, and correct any misconceptions they had regarding the concepts of momentum and impulse. In this way the instructor was able to more effectively monitor student progress while the learning was actually taking place. So often problems with student understanding aren’t uncovered until after an examination has been given. In addition, the assessment tools allowed for ongoing changes to be made in the instructional process to further assist student learning throughout the entire instructional sequence.

Directions for Further Research At this writing data analysis is ongoing. Future plans include the addition of other forms of data collection techniques to assess the utility of these computer-based learning tools. For example, we plan to collect empirical data by observing students at work with the new technologies. In addition, we plan to collect empirical data through the use of structured, open-ended interviews as well as individual demonstration interviews. As appropriate, these interviews will involve having students work with a particular computer simulation or tutorial related to the concepts of momentum and impulse and will follow a standard “think aloud” protocol. Data collected from these observations and interviews will be analyzed using protocol analysis techniques 20. Finally, we intend to continue our studies into the role(s) that learning styles may play in terms of enhancing student understanding following instruction that makes use of computer-based learning tools. The assessment of individual learning styles does not link itself directly to effective instructional design. The best teaching strategies involve the incorporation of a variety of instructional techniques. The individual learning style assessments ARE important in that they may allow for possible connections to be made between specific learning style strengths and student learning gains that result from instruction that includes the use of computer-based technologies. Once the current study is completed, learning style data will be linked to student performance on the various strategies used to assess student understanding. For example, the laboratory activity involves a hands-on approach. We would like to be able to determine whether students who have a tactile learning style preference perform better when given the opportunity to perform the laboratory activity as opposed to traditional teaching strategies. Thus, we plan to use the data collected to help us determine the role(s) that learning style may play in terms of student

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-5

Session 13c3 understanding of the collision process after exposure to the interactive laboratory experiment on collisions. In addition, we plan to further analyze the relationship between learning style and strategies of instruction that may or may not result in differences in student performance on the variety of assessment measures used. These results will be shared in a future publication.

7.

8. Acknowledgments 9. Partial support for this work was provided by the National Science Foundation’s Division of Undergraduate Education through grant #DUE 9850570. Any opinions expressed in this article are the authors’ and do not necessarily represent the opinions of the NSF. We would like to extend a special thank you to Jami Burgess, an undergraduate student at American University, for her assistance in developing the laboratory activity described here. In addition, we would like to thank our graduate assistants, Amani Ishaq, Yongguang Liang, and Yan Wang for their invaluable assistance in teaching the laboratory component of the Physics for the Modern World course. References 1.

2.

3.

4.

5.

6.

Edwards, V. B., Editor’s introduction in Education Week, November 10, 1997. Washington, DC: Editorial Projects in Education. Hein, T. L., and S. E. Irvine, “Assessment of Student Understanding Using On-line Discussion Groups,” Proceedings, 1998 Frontiers in Education Conference, FIE, 1998, pp. 130 – 135. IEEE Catalog No. 98CH36214. ISBN 0-7803-4762-5. Meyers, C., and T. B. Jones, Promoting Active Learning: Strategies for the College Classroom. San Francisco: Jossey-Bass Publishers, 1993. Larochelle, M. and Desautels, J. “The Epistemological Turn in Science Education: The Return of the Actor,” In R. Duit, F. Goldberg, and H. Niedderer (Eds.), Research in Physics Learning: Theoretical Issues and Empirical Studies. Proceedings of an International Workshop, University of Bremen, Germany, 1992. Niedderer, H., Goldberg, F., and Duit, R. “Towards Learning Process Studies: A Review of the Workshop on Research in Physics Learning,” In R. Duit, F. Goldberg, and H. Niedderer (Eds.), Research in Physics Learning: Theoretical Issues and Empirical Studies. Proceedings of an International Workshop, University of Bremen, Germany, 1992. Scott, P. H. “Pathways in Learning Science: A Case Study of the Development of One Student’s Ideas Relating to the Structure of Matter,” In R. Duit, F. Goldberg, and H. Niedderer (Eds.), Research in Physics Learning: Theoretical Issues and Empirical Studies.

10.

11.

12.

13. 14. 15. 16.

17. 18.

19.

20.

Proceedings of an International Workshop, University of Bremen, Germany, 1992. Kulik, J. A., “MetaAnalytic Studies of Findings on Computer-based Instruction.” In E. L. Buker and H. F. O’Neill, Jr. (Eds.), Technology Assessment in Education and Training, Hillsdale, NJ: Lawrence Erlbaum Associates, 1994. Healy, Jane M. (1998). “Failure to Connect: How Computers Affect Children’s Minds – For Better or Worse.” New York, NY: Simon & Schuster. Dunn, R., K. Dunn, R. I. Sklar, and J. Beaudry, “Effects of Matching and Mismatching Minority Developmental College Students’ Hemispheric Preferences on Mathematics Scores,” Journal of Educational Research, vol. 83, no. 5, 1991, pp. 283 – 288. Miller, J., “The Effects of Traditional Versus Learning Style Presentations of Course Content in Ultrasound and Anatomy on the Achievement and Attitudes of College Students,” Doctoral Dissertation, St. John’s University, Jamaica, New York. Ranne, T. M., “Hawthorne Un-capped: The Relationship of Adult Learning Styles to the Academic Achievement of Nursing Students.” Dissertation Abstracts International, vol. 57, no. 9, 1996, p. 3771. Hein, T. L., “Learning Style Analysis in a Calculusbased Introductory Physics Course,” 1995 ASEE Annual Conference, ASEE, 1995. The American University Catalog, Washington, DC: University Publications and Printing, 1995 – 1996. Pasco Scientific: Rosehill, CA. Logger Pro (windows version 1.0.7). Tufts University and Vernier Software. Portland, OR. Angelo, T. A., and K. P. Cross, Classroom Assessment Techniques: A Handbook for College Teachers. San Francisco: Jossey-Bass Publishers, 1993. Cross, K. P., “On College Teaching,” Journal of Engineering Education, vol. 82, no. 1, 1993, pp. 9 – 14. Hein, T. L., “Using Student Writing as a Research and Learning Tool,” AAPT Announcer, vol. 27, no. 4, 1998, pp. 79. Hein, T. L., “Using Writing to Confront Student Misconceptions in Physics,” European Journal of Physics, Vol. 20, 1999, pp. 1 – 5. Ericcson, K. A., and H. A. Simon, Protocol Analysis. Cambridge, MA: The MIT Press, 1984.

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-6