Session F3A ON-LINE ENGINEERING MATHEMATICS TESTING AND

Session F3A ON-LINE ENGINEERING MATHEMATICS TESTING AND ASSESSMENT1 M. Sami Fadali2, N. Henderson3, J. Johnson, J. Mortensen4// J. McGough5 Abstract  We are currently developing a new strategy for mathematics assessment and remediation at the University of Nevada Reno (UNR) and the South Dakota School of Mines and Technology (SDSMT). As part of this strategy, electrical engineering students take an on-line test of their ability to use mathematics to solve context-based problems. In this paper, we report on the result of the computer test for students in an electrical engineering orientation class at UNR. The results indicate a reluctance on the part of the students to accept their mathematics deficiencies, which hinders any attempt at remediation to reduce attrition.

INTRODUCTION There is currently an unsatisfied need in industry for more qualified engineers coupled with a disturbing decline in college students pursuing technical degrees [1]. In addition, the attrition rate among engineering majors is unacceptably high [2]. There is strong evidence that suggests the high attrition rate and the decline in enrollment are due in part to weak skills in mathematics [3]. In response to the retention problem, the Electrical Engineering Department and the Mathematics Department at UNR teamed to develop a retention strategy targeting freshmen electrical engineering students [4]. SDSMT is a partner institution in this project. A proposed retention strategy was developed and a formative study done to identify high-risk students defined as those exhibiting weak mathematics skills, and to provide remediation through peer tutoring. This preliminary study sought to answer the question: will an intervention plan using online computer testing, advising and peer tutoring provide a viable retention strategy for freshman electrical engineering students? This paper describes the trial results of student testing at UNR. Section II describes the student participants in the test. Section III is a description of the on-line test. Section IV is a report of the testing results. Section V gives conclusions and future work.

PARTICIPANTS AND SETTING Participants consisted of 48 freshman students enrolled in an introductory electrical engineering orientation class, EE101 1 2 3 4 5

Introduction to Electrical Engineering, at UNR (a Carnegie I university). The data was collected over a 16-week semester using several methods including (a) formal personal interviews (see Appendix A) with 6 students in EE101; (b) an examination of university and college of engineering student demographic records such as attrition rates, ethnicity, gender, grade-point average, etc.; (c) completion of two different surveys, an Engineering Attitude Survey (see Table 1) and a Math Anxiety Test (see Table 2), each administered at the beginning of the semester and at the end of the semester; and, (d) an online math test (http://devnull.math.unr.edu/webtest), developed by the project directors and administered at the beginning of the semester. All students were asked to take the online math test again at the end of the semester but there were no volunteers. An examination of university data books located at: http://www.vpaf.unr.edu/pba/ia/databook/index.html . yielded several data groupings such as enrollment by year (freshman, sophomore, junior, senior, graduate), age, gender, ethnicity, college, and degree. From this data, freshman attrition rates for the last ten years (1990-1999) were obtained for the university overall (26.7%). However, the attrition rate for students in the electrical engineering program is 58% over a two-year period [2]. The high attrition rate for electrical engineering majors appears to be primarily for students enrolled in science and mathematics classes. Circumstantial evidence indicates that one of the major reasons of attrition is failure to acquire the mathematical skills necessary to continue in the program. An Engineering Attitude Survey [5] was administered during the third week (pre) and again at the end of the fall 2000 semester (post) to all students in the introductory electrical engineering class (EE101). The instrument was a 25-item, six-point Likert scale used to measure attitudes and experiences associated with engineering. In addition, the EE101 class completed a Math Anxiety Test [6], during the fourth week and again during the final week of the fall 2000 semester. The instrument was a 21item, six-point Likert scale used to measure attitudes and experiences associated with mathematics. The higher the score, the more positive the attitude toward learning mathematics. Since the instrument used in this study was

This work was supported in part by NSF grant number DUE 9980687. M. Sami Fadali is with the Electrical Engineering Department, University of Nevada, Reno. Norma Henderson is with the Curriculum and Instruction Department, University of Nevada, Reno. J. Johnson, J. Mortensen are with the Mathematics Department, University of Nevada, Reno. J. McGough is with the Mathematics Department, South Dakota School of Mines & Technology.

0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31 th ASEE/IEEE Frontiers in Education Conference F3A-15

Session F3A modified, the validity and reliability will be assessed using Cronbach’s alpha coefficient to examine internal consistency and a common factor analysis to examine its construct validity. The results of this survey are used to determine whether students experienced any change in level of math anxiety over the course of the fall semester. Results from both of these instruments were used to compare students’ attitude toward engineering and mathematics between the beginning and end of the semester, and perception of tutoring expressed by students in formal interviews. Additional data collection and analysis included an online math test administered the fifth week of the semester. The purpose of the online math test was to identify students who were experiencing deficiencies in mathematics early in the semester. The test is located at: http://devnull.math.unr.edu/webtest. Each student logged on to the World Wide Web and accessed t he math test using their login name and password, which consisted of their name and student identification number. As the students completed each of the 8 online math test questions, they would also immediately answer an online survey consisting of 6 questions related to the specific test question using a six-point Likert scale. An item analysis was performed on the math test and will be used to refine the online test questions for use in the summative portion of this study. The online math test was scored instantaneously and all students who did not achieve a passing score on the test were offered help in the form of peer tutoring through the University’s Math Center. Participation was strictly voluntary and students were even offered a monetary reward to participate in the tutoring program. There were no punitive consequences, such as a grade reduction, for not participating in the tutoring program.

ON-LINE TEST During October, 2000 an exam consisting of eight questions was administered to 43 electrical engineering majors in EE 101. For a complete sample test please see devnull.math.unr.edu/webtest/ The computer test questions covered elementary precalculus mathematical skills. One of the goals of the program is to include mathematics problems in the context of engineering applications. Colleagues in the College of Engineering have observed that students who have otherwise successfully completed a mathematics course have difficulty applying it in engineering classes. We believe that this is partly because the students are seeing the mathematics in an unfamiliar setting with different notation. The web-based tests are designed to couch elementary mathematics in engineering terminology. Here are two simple examples:

1. Capacitance is a circuit property used to model the storage of energy in an electrostatic field by a device called a capacitor which has two conductor plates separated by an insulator. Capacitance is measured in units of Farads (F) although practical values are less than one microFarad (10−6F). Capacitance is associated with two conductor plates separated by an insulator. The capacitance is given by C = εA d where ε is the permittivity of the insulator, A is the area of the plates, and d is the separation between them. The permittivity of mica is 5×10-11 F/m. If a capacitor with a mica insulator has a capacitance of one microFarad and the area of the plates is 15 square centimeters, what is the distance between the plates? 2. Capacitance is a circuit property used to model the storage of energy in an electrostatic field by a device called a capacitor which has two conductor plates separated by an insulator. The capacitance is directly proportional to the area of the plates and inversely proportional to the distance between the plates. What happens to the capacitance if the area of the plates is doubled and distance between the plates is cut in half? (a) it is quadrupled (b) it is doubled (c) it stays the same (d) it is half as much (e) it is 1/4 as much The test software allowed the exam designer to select a number of questions from a bank of questions that we have developed. The numerical values used in the questions were randomly selected by the computer software. The software was designed to grade the tests and record the time that it took each student to complete the test. However, students were given essentially no time limit and the duration of the test was only recorded for assessment purposes. Although the questions were mostly context -based, two questions were included that test the ability of students to avoid "fatal errors." These are questions where an incorrect answer would indicate a complete lack of understanding of basic algebra such as incorrectly stating that

x2 + y2 = x+ y .

The scores were as follows: 3 scores of 1, 9 scores of 2, 9 scores of 3, 4 scores of 4, 7 scores of five, 8 scores of 6, no scores of 7, and three scores of 8. Both the average and median scores were 4.

RESULTS Data collected on the students enrolled in electrical engineering 101 in the fall 2000 semester yielded a few interesting observations. Although 24 (57%) of the 42 students who took the online math test scored at or below 50%, they did not seek tutoring through the resources provided. Six students were selected for formal interviews to

0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31 th ASEE/IEEE Frontiers in Education Conference F3A-16

Session F3A determine why there was a lack of motivation for mathematics for success in an engineering program. Many participation in the tutoring program. students spent a longer time attempting to complete the exam These six students consisted of five males and one than one would expect for a student with a solid precalculus female. All were first semester freshmen and ranged in age background. We take this as evidence that poor performance from 18-21. Interviews were transcribed word for word and was not due to lack of effort. items were categorized by common themes. For instance, all Fourteen of the students had previously taken a of the students interviewed expressed a positive attitude precalculus algebra and trigonometry course (Math 128). toward mathematics—a heading labeled attitude was created Their average score on the Web test was 3.43. Twenty-eight for analysis and each student’s attitude noted. In this of the students had not. Their average score was 4.32. It is manner, the data was reduced and cross-referenced with the important to realize that the Web test required no attitude surveys collected in the classroom and with the mathematics skills beyond solving two simple equations in online math scores. By triangulating the students’ data in two unknowns and an understanding of ratio and proportion. this fashion, it was apparent whether their own personal One might conclude that students who had to take a perceptions during the interview were correlated with the test precalculus class had deficiencies that persist. results. Another interesting result is the strong correlation Unfortunately, due to time constraints, interviews between student performance in the on-line test and their Sat were not available for all 48 students in the EE101 class. scores (Figure 1) and ACT scores (Figure 2). However, the six students selected represented the entire spectrum of scores achieved on the online math test. For CONCLUSION example, two students received a score of 2 or less, 2 students received a score in the 4-5 range, and the final 2 Due to a limited time constraint of one semester, the findings students received a score above 6. Interestingly, the scores of this preliminary study are representative of a snapshot in were not significant indicators of the students’ selftime. However, this was the exploratory stage of a more perception toward math skills or engineering attitude. They comprehensive study planned for the next year. Additional all had the same level of attitude toward both—very positive. longitudinal data collection would yield a richer picture of Student interviews revealed that many students had an students’ freshman experience, as well as for freshmen inflated opinion of their mathematical abilities. Even electrical engineering students in particular. We would like students wh o committed the "fatal errors" believed that they to test the effectiveness of tutoring on student performance, did not have any deficiencies in mathematics. but their reluctance to participate in the tutoring program has The Engineering Attitude Survey was administered hindered us so far. We plan to include a form of incentive for to determine students’ attitude toward engineering. Fortystudents to participate in the tutoring program through eight students completed the pretest and thirty-seven grades or possibly a monetary compensation. Without students completed the posttest. As shown in Table 2, there student participation, it is difficult to assess the viability of a was no significant change in students’ attitude toward peer tutoring retention strategy. engineering. The higher the score, the more positive the It would also benefit the study to investigate other attitude toward engineering with the exception of questions strategies that might contribute to a higher retention rate. At 2, 5, 12, 16, 19, 22 and 23 which are negative and should show this stage of the assessment, our results do not shed light on a lower score to indicate a more positive attitude. the relationship between weak math skills, remedial tutoring, Students scored high (positive attitude) on the math and attrition. anxiety test. Low scores on negatively worded questions 1, 6, 8, 9, 12, 14, 16 through 21 also indicate a positive attitude toward math (low math anxiety). Grades for the students’ first semester were unavailable at this time. Accordingly, their GPA will be used in the assessment performed in the next semester (spring 2001). The six students will be tracked through the remainder of their engineering coursework. Because none of the students participated in the tutoring program, interviews were not performed with the math tutors. However, interviews with a math tutor will be done at the beginning of the next semester to provide insight into the tutoring process, and will be included in the next phase of the study. The results suggest that a significant number of students did not have the necessary background in 0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31 th ASEE/IEEE Frontiers in Education Conference F3A-17

Session F3A Table 1 Engineering Attitude Survey Results. Question 1. Most engineers have poor social skills. *2. Engineers spend most of their time doing complex mathematical calculations. 3. Engineering would be a highly interesting profession for me. 4. A problem with engineering is that engineers seldom get to do anything practical. *5. Engineers deal primarily with theory. 6. Engineers spend relatively little time dealing with other people. 7. Engineers spend most of their time working in offices. 8. Engineers spend most of their time working with computers. 9. Engineers seldom get involved in business decisions. 10. Engineers have little need for knowledge about environmental issues. 11. Engineers have little need for knowledge about economics. *12. Engineers have little need to deal with questions about behavior that is morally right or wrong. 13. Engineers have little need for knowledge about political matters. 14. To be a good engineer requires an IQ in the genius range. 15. Engineering is a poor career choice because job availability is dependent on defense spending. *16. Engineers need a great deal of inborn aptitude for science and mathematics. 17. Most engineers have very narrow outside interest. 18. Engineering is important to future US economic success in the world. *19. Engineers typically have

Pretest Mean 3.00

Posttest Mean 3.32

3.55

3.62

4.93

4.70

2.33

2.38

3.17

3.19

2.48

2.76

2.98

3.03

3.56

3.49

2.69

2.76

very little common sense. 20. A career in engineering would be financially rewarding. 21. Most of the skills learned in engineering would be useful in everyday life. *22. Engineers are not typically people who are fun to be around. *23. Engineers do not tend to be appreciative of the arts. 24. Engineers are frequently those individuals who were regarded as “nerds” in high school. 25. If I had to do it over again, I would consider a career in engineering. KEY:

4.97

5.19

4.40

4.62

2.42

2.70

2.69

2.78

3.02

3.78

4.76

4.73

1=Very Strongly Agree; 2=Strongly Agree; 3=Agree; 4=Disagree; 5=Strongly Disagree; 6=Very Strongly Disagree *Negative questions—lower scores indicate a positive attitude

SAT vs Web

2.31

2.35

2.27

2.41

2.46

2.30

2.77

2.59

2.59

2.81

800 700 600 Series1

500 400 300 200 100 0

Linear (Series1)

0

5

10

Figure 1 Plot of SAT scores vs. web test scores. ACT vs WEb

2.02

2.16

3.96

3.86

2.71

2.65

4.79

5.24

35 30 25 20 15 10 5 0

Series1 Linear (Series1)

0

5

10

Figure 2 Plot of ACT scores vs. web test scores. 2.38

2.62

0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31 th ASEE/IEEE Frontiers in Education Conference F3A-18

Session F3A Table 2 Math Anxiety Test Results. Question *1. Mathematics is very interesting to me and I enjoy my math courses. 2. My mind goes blank, and I am unable to think clearly when doing math. 3. I feel a sense of insecurity when doing math. 4. Mathematics makes me feel uncomfortable, restless, irritable, and impatient. 5. I approach math with a feeling of hesitation, resulting from a fear of not being able to do math. *6. Mathematics is a course in school which I have always enjoyed studying. 7. It makes me nervous to even think about having to do a math problem. *8. I feel a definite positive reaction to mathematics; it’s enjoyable. *9. If I am confronted with a new mathematical situation, I can cope with it because I have a good background in mathematics. 10. I get flustered if I am confronted with a problem different from the problems worked in class. 11. I do not attempt to work a problem without referring to the textbook or class notes. *12. I can draw upon a variety of mathematical techniques to solve a particular problem. 13. I do not feel that I have a good working knowledge of the mathematics courses I have taken so far. *14. I believe that if I work long enough on a mathematics problem, I will be able to solve it.

Pretest Mean 2.35

Posttest Mean 2.27

4.81

4.81

4.86

4.51

4.98

4.81

5.05

4.54

2.44

2.32

5.14

4.92

2.53

2.41

2.24

2.38

4.12

3.89

15. I have forgotten many of the mathematical concepts that I have learned. *16. I learn mathematics by understanding the underlying logical principles, not by memorizing the rules. *17. If I cannot solve a mathematics problem, at least I know a general method of attacking it. *18. Mathematics problems are a challenge; solving problems provides satisfaction similar to those of winning a battle. *19. Problem solving fascinates me. *20. I have more confidence in my ability to deal with mathematics than in my ability to deal with other academic subjects. *21. Mathematics classes provide the opportunity to learn values that are useful in other parts of daily living. KEY:

4.45

4.01

2.79

2.73

2.30

2.36

2.48

2.58

2.40

2.45

2.68

2.65

2.29

2.43

1=Very Strongly Agree; 2=Strongly Agree; 3=Agree; 4=Disagree; 5=Strongly Disagree; 6=Very Strongly Disagree *Negative questions—lower scores indicate a positive attitude

REFERENCES [1] [2] 4.17

3.82

[3]

2.66

2.59

[4]

4.88

4.27

[5] [6]

2.07

2.24

P. Mendels, "Report Indicates Decrease in High -Tech Degrees", New York Times, May 5, 1999. Walter Johnson, 1998 College of Engineering Annual Report, College of Engineering, University of Nevada, Reno, NV, 1998. D. Budny, G. Bjedov and W. LeBold, "Assessment of the impact of the freshman engineering courses", Proc. Frontiers in Education Conf., Pittsburgh, PA, Nov. 1997. M. Sami Fadali, J. Johnson, J. Mortensen, J. McGough, "A New On-line Testing and Remediation Strategy for Engineering Mathematics"Proc. FIE 2000, Kansas City, MI, Oct. 2000. M. Robinson, M. S. Fadali, J. Carr, C. Maddux, "Engineering Principles for High School Students", Frontiers in Education, San Juan Puerto Rico, Nov. 1999. L. Ludlow and K. Bell, “Psychometric characteristics of the attitudes toward mathematics and its teaching (AMAT) scale”, Educational and Psychological Measurement, 56, 864-880, 1996.

0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31 th ASEE/IEEE Frontiers in Education Conference F3A-19

Session F3A Appendix A

9.

Personal Interview Name of Interviewer______________________________________ _ Date___________________________________________ _ Name of Interviewee_____________________________________ __

10.

Interview with EE101 Student

14.

This interview is being conducted to get your input about the math/engineering study being done through a grant from the National Science Foundation. The purpose of the study is to try and determine how the engineering/math department can help reduce the dropout rate of electrical engineering students. Your input is extremely valuable and there are no right or wrong answers. This is only for the development of a retention program.

11. 12.

13.

15.

16. 17.

It appears that this study has been received negatively by the students. In your opinion, why? How could the study be changed to help motivate students? What are your feelings about tutoring? Do you see it as something negative or positive for yourself? Have you ever received tutoring for any subject? If so, what age? For how long? Who did the tutoring? Have your ever tutored anyone? If so, what subject? What age group? For how long? What was your experience like? How do you feel about your math instructor? Is he/she effective? Are you thinking of changing majors? Dropping out of school? If so, why? (grades, money, personal, etc.) Is there anything else you would like to comment about with regards to your engineering studies or this study? How could we more effectively reach students who might need some outside support academically? Would you be willing to talk to me at the end of the semester? Next semester?

If it is okay with you, I will be tape-recording our conversation to save time. This will help me get all the details but at the same time pay close attention to what you are saying. I assure you that your comments will remain confidential. Once I transcribe the interview, I will erase the tape. No one else will hear what was said here, so it pays to be very honest. I appreciate your time and effort in this. If you agree to this taped interview, please sign this consent form. 1.

2. 3. 4. 5.

6. 7. 8.

What made you decide to pursue electrical engineering? Your motivation? (teachers, friends, family, money, etc.). How far into the program are you? (freshman, sophomore, junior, senior, etc.) Are you a transfer student, or is this your first college experience? Are you here on scholarship? Any other financial aide? How many credits are you carrying? If this is not your first semester, what is your average credit load? How many engineering courses have you completed? Do you work? How many hours per week? What was your feeling about the online math test? Did you take it seriously? Why or why, not? Did the score on your test prompt you to change your study habits? If so, how? (study group, peer tutor, outside help) How much time are you spending? If not, why not?

0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31 th ASEE/IEEE Frontiers in Education Conference F3A-20