Computers & Education 37 (2001) 53–66 www.elsevier.com/locate/compedu
Experiences of using computer assisted assessment in engineering mathematics A.C. Croft *, M. Danson, B.R. Dawson, J.P. Ward Loughborough University, Loughborough, LE11 3TU, UK Received 3 February 2001; accepted 19 February 2001
Abstract This article discusses issues associated with large-scale computer assisted assessment (CAA) of the mathematical skills of first-year undergraduate engineering students. It explains the background to this testing as it relates to a wider open-learning project in engineering mathematics. Details of issues specifically associated with mathematics testing are given. Practical issues of constructing appropriate tests, their implementation and their delivery are described. Finally, the results of a survey of students’ perceptions of CAA and the influence of these on future developments are presented. # 2001 Elsevier Science Ltd. All rights reserved. Keywords: Applications in subject areas; Evaluation of CAL Systems; Interactive learning environments; Pedagogical issues
1. Introduction This paper describes the implementation of a computer-based testing regime which has been used as a vehicle to encourage first-year students of engineering to practice routine mathematical techniques. At first-year level it is important that students develop confidence and competence in a diverse range of basic mathematical techniques which will be required later in their engineering studies. A major challenge for educators is to encourage these students, whose primary interest is not mathematics and who do not come to university to study mathematics, to engage actively in learning mathematics. To be successful this engagement must begin sufficiently early in the course and there must be sufficient motivation to continue to practice throughout the year. Computer Assisted Assessment (CAA) has made this possible with large groups of students through a regular testing regime. In the project described here approximately 400 first-year engineering students have been tested three times each semester. * Corresponding author. Fax: +44-1509-223969. E-mail address:
[email protected] (A.C. Croft). 0360-1315/01/$ - see front matter # 2001 Elsevier Science Ltd. All rights reserved. PII: S0360-1315(01)00034-3
54
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
Section 2 establishes the importance of developing basic mathematics skills in engineering students, and gives details of some particular difficulties associated with trying to teach mathematics to these groups. Section 3 describes the way in which CAA has been used to encourage students to practice and develop these skills. Sections 4 and 5 describe how libraries of test questions have been constructed using Question Designer software. Section 6 gives details of the practical implementation of the tests. Finally, the results of a survey of students’ perceptions of CAA and the influence of these on future developments are presented in Section 7.
2. Mathematics in first-year engineering courses The importance of mathematics in engineering and science is well recognised (see IMA, 1999). A typical first course in engineering mathematics at university is about ‘mathematical methods’. These form a body of techniques or tools which students will be expected to be able to apply in other, engineering, modules later in their degree programme. In 1997, Loughborough University, through its Development Fund for Teaching Innovation, funded a major project known as the Open Learning Project: Mathematics for Engineers. This initiative has resulted in an open learning environment rich in resources to instill these mathematical methods. It uses a mix of traditional and modern computer technologies to ensure that students can benefit from a learning regime which can be configured to suit their interest, starting point, and their preferred mode of working. Full details of this project have been published elsewhere (Croft & Ward, 2001). Because mathematics is a linear subject and one in which one topic builds upon the knowledge gained in earlier topics, it is vital that students keep apace with the subject otherwise the new material becomes increasingly obscure. Mathematics modules might only make up one sixth of a university engineer’s first-year programme. Despite its importance, it has to be recognised that students come to university to read engineering and not mathematics. From the perspective of an uninterested or poorly motivated student mathematics doesn’t appear to be that significant, takes low priority, and, unless action is taken, students have a tendency to ‘switch off’ from a topic they traditionally find difficult. By the time they switch back on, it is often too late to catch up. What are needed are regular prompts to the students to stay switched on throughout the module. One way of doing this is to test at regular intervals.
3. Using computer assisted assessment to encourage practice At first year level, assessment strategies are generally based upon verifying that a sufficient number of these mathematical methods have been learned. Traditional written examinations are held at the end of each semester. Because of the time required to practice and develop mathematical skills those students who do not start to engage in the learning process sufficiently early find themselves doing poorly in these examinations. Regular testing of large groups of students throughout the whole year is only feasible if some form of computer assisted assessment is used. In the project described here there are three formal CAA examinations per semester. These are used to focus the students’ minds from the outset and so drive them to practice mathematics throughout their course.
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
55
Of course, in an open learning environment regular testing is vital anyway. Many students will choose to absent themselves from lectures and tutorials in order to study at a time of their choosing and at a pace that suits them. This is to be encouraged and is a central theme within an open learning regime. For the better motivated, and generally better able, group of students regular testing is required so that they can assess what progress is being made and better gauge their level of working. The average engineering student faced with a heavy work schedule, will not choose to take a mathematics test, in any form, purely for his or her educational development. The test result must make a contribution to the overall assessment mark for the module. Students will more easily accept a regular testing regime if they know that they will have a significant proportion of real marks to show at the end. In this project the CAA element contributes 20% to the final module mark. Traditional written examinations are still used for the remaining 80%. However, the Open Learning Project aims for more than mere reluctant acceptance of a regular testing regime. It aims to get students to positively engage in this approach. This is achieved by convincing students that with a good work strategy they can perform ‘very well’ in CAA. In this project students are allowed to ‘trial’ each of the three assessed tests, as often as they want over a period of 10 days before the ‘real test’. In practice this meant that the real test was available on Thursday–Friday in the the week of the test, and throughout the preceding 10 days the trial test was available. In the trial tests students gain familiarity with the area of mathematics being covered and with the type of question being asked. Every trial question has an explicit worked solution or an outline of the method from which the solution can be obtained. These worked solutions are presented as feedback should the student fail to answer correctly. The important point is that the trial test is in form and in level of difficulty identical to the real test. Both will have the same number of questions and corresponding questions will cover the same subject material and be posed at the same level in the same sequence. The strategy that students employ is that they do the trial test two or three times. If an adequate score (i.e. one the student is happy to accept) is obtained then they proceed to the real test. If not, they can take the trial test repeatedly to improve performance. Our experience and student feedback tells us that this positive engagement in this aspect of the learning process would have been very unlikely with traditional assessment regimes. We stress again that this first course in engineering mathematics is essentially one of mastering tools, and repeated practice assists in developing this mastery.
4. Questions, question libraries and tests The software package that has been used exclusively within this project for developing CAA is Question Mark Designer. Within this environment a large number of question libraries have been constructed by members of the project team. Each library covers a specific mathematics topic. Each question in a given library has the same level of difficulty. Although the designer software allows many question types (multiple choice, hot-spot, wordmatch, explanation, etc.) use has been restricted here to numeric input questions together with a small number of multiple choice questions. In the numeric input type of question either integer input or real number input (to a specified accuracy) is allowed. Using this software it is not possible to accept more general (algebraic, symbolic) mathematical input. This is a shortcoming for mathematics education. Attempts have been made to overcome this problem by development of
56
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
input tools for algebraic and symbolic input but these can be extremely cumbersome for the student to use because of the levels of complexity required in mathematical notation. The notation needed by engineers in mathematical calculations is particularly sophisticated, with frequent use of algebraic subscripts, superscripts, greek characters and other symbols, a requirement to use common keywords, (e.g. log, sin), complicated bracketed expressions which can change their meaning radically following a simple transposition, and so on. All of these factors add further levels of sophistication to what, for many, is a difficult topic and so this approach has not been pursued. Instead, despite shortcomings, we have aimed for simplicity in respect of user input. Furthermore it is not normally feasible to test higher level skills, such as handling multi-stage problems, in this way, and traditional examinations are used for this purpose. In the trial tests the feedback is given if a student submits a wrong answer. Figs 1 and 2 show a typical question together with its feedback page. Over 1200 questions, in this general form, have been constructed by the project team and appended to question libraries which are held on a central file server. A CAA test is similar to traditional tests in that it has a (modified) rubric as shown in Fig. 3. In the real test students navigate through the material using ‘tape-recorder’ controls (top left-hand corner). The first screen they meet (not shown here) is standard to all tests delivered at Loughborough University. The test will not be accessible unless students agree to abide by the University Code of Practice on academic conduct. At any stage students may amend an answer they have typed in. When a student has responded to all of the questions they can review their answers before declaring that they have finished, and submit their answers for electronic marking. Each test has a generous time limit. A clock which counts down to show the time remaining is shown in top right-hand corner of the screen. Depending on the subject area a typical test might contain 5–15 questions. Each question is chosen from a library of up to 30 questions so that the likelihood of any two tests being the same is very small. For example a test on vectors and matrices might extract questions from libraries in which topics relate to: 1. simple matrix products; 2. evaluating determinants; 3. missing terms in an inverse matrix;
Fig. 1. Typical question involving vectors.
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
Fig. 2. Feedback — if required.
Fig. 3. CAA rubric for a specific test.
57
58
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
4. 5. 6. 7. 8. 9.
finding a single unknown in a system of linear equations; obtaining the component of a vector in a given direction; finding a scalar product; finding the area of a triangle; finding the distance between two points; and working out a single component of a cross product.
All of these questions can be formulated so as to require a single numerical input. Multiple choice questions have been used when, for example, students are asked to identify the characteristic polynomial and eigenvectors of a matrix. This question form is useful in cases when an answer takes the form of a sophisticated algebraic expression or when an answer can be written in many different yet equivalent ways. We encourage students to work together on the trial tests. They will need to bring along pens, paper for their calculations, and calculators. Many will bring text-books, notes and formulae books along just as they would if they were solving a real problem in a real situation. Some will use other computers equipped with symbolic computing packages to work some of the problems. The aim is to get students to do mathematics regularly. This is a considerable improvement over the traditional approach in which a single mid- to late-semester test was given and for which feedback followed much later. To a student, such tests can be relegated to the back-burner and there is little impetus to keep working consistently. However, irrespective of the type of question that is being asked, we are ensuring that this testing regime actively encourages students to engage in the learning process. Our argument is that by continually exercising students in this way they will develop mastery of the techniques they require. Application of these techniques follows in their engineering subjects and in later mathematics modules. Students are examined in the traditional manner at the end of each semester. The higher level skills are assessed at this stage.
5. Constructing questions A question goes through a number of stages before it is added to a library. 1. First it is necessary to decide which skills or methods are to be tested and then construct an appropriate question. This decision is informed by the use of learning outcome statements in the workbooks and computer assisted learning materials to which the students have access and which have guided the learning undertaken prior to testing. Questions should not be too verbose, as they should fit easily on the screen. Some care should also be given to ensuring that the student should not have to go through an excessively detailed calculation to obtain the answer, although this is not always possible. It would be regarded as unfair to award marks for a particularly convoluted question on the basis of a single numerical input. 2. In mathematical work, both printed and electronic, there are always difficulties associated with the extensive range of symbols required and their relative positioning. It was felt desirable that questions, even though presented on screen, should use the same fonts and symbols that students would recognise if they were reading a text although font sizes will have to be considerably
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
3. 4.
5.
6. 7.
59
larger than those normally used for printing. The computer typesetting system LaTex which is specifically designed for typesetting mathematics, was used to typeset each question. The LaTex output can be saved in the form of Portable Document Format files (pdf files). Question Designer software is used to create the testing environment. Here the question type is chosen. Control statements are set which govern navigation dependent upon correct/near correct/wrong answers being entered by the student. For example, in a trial test feedback is given when a student submits an incorrect answer. Acrobat Reader is used to open the pdf files containing the mathematics questions. Appropriate segments from the pdf document can be copied into a ‘rich text’ area within Question Designer. Rich Text Format (RTF) is used to ensure that questions are not distorted when delivered on client computers having various screen resolutions. Question Designer allows a ‘supplementary question’ if required. This is used to provide feedback in the case where students select the wrong answer. The correct answer (plus the accuracy required if real number input is used) is set in Question Designer together with the mark to be allocated for a correct/near correct answer.
6. Issues of delivery and operation 6.1. Central CAA support The CAA aspect of the open-learning project has made extensive use of central support facilities and in particular the CAA Unit which is located in a central learning and teaching support centre known as Learning & Teaching Development. The Unit is responsible for ensuring the effective integration of protocols and the implementation of a campus wide strategy. The Unit employs a CAA Officer who, in addition to having other duties, is responsible for installing and scheduling tests, and for processing the results. One outcome of the work of the unit is the centrally-based Question Mark for Windows Service (QMWS; Phelps & Danson, 2000). Details of Question Mark software are given in Dempster (1994). However, it is of interest to note that a web-based product (Question Mark Perception) has recently become available which the CAA Unit is currently implementing, with a view to the possibility of delivering this type of testing via the World Wide Web. The QMWS allows the effective management of screen-based assessment. Tests are written by members of the Open Learning Engineering Mathematics Project team as described in Section 5, and installed by the CAA Officer. Once taken, marking is immediate, and feedback can be given during or at the end of the test if required. In the real tests feedback is given in the form of percent correct at the end of the test. In the trial tests feedback is given after each question. There is one central copy of the test so any updates need only be made once.
7. Integrity and authentication The CAA system is reliant on IT equipment. Students must be able to log into the campus network, that is, they must know their user-name and password. Tests are available only on campus. All universities are likely to have existing mechanisms for authenticating users of a
60
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
campus network. The system described here uses a Novell NDS system. It was decided that the QMWS should use the user-name and password information to authenticate candidates rather than have candidates enter their identification manually. Student names are not unique and manual input allows for spelling mistakes and the input of false information. On booting a networked PC users must enter their NDS login ID and password. This information is written into the answer file once a test is launched. Post-processing allows the substitution of real name and student ID number from the central student records database. This open-access approach is of course prone to both plagiarism and collusion as with any form of open access coursework. Both the trial tests and coursework tests comprise questions drawn randomly from a number of question libraries. The scale of these libraries means that no two tests are likely to be identical. In addition, a time limit is set for each session. This limits the opportunity for plagiarism. In this project students are still required to sit a formal end-of-semester examination which accounts for 80% of their mathematics module mark. Consequently these issues were not of paramount concern to the team. One benefit of the CAA approach is the level of detail recorded in the server logs. Student behaviour can be tracked to include details of the number of attempts made at the various tests, the time and date the tests are accessed, and the time spent taking each test. 7.1. Access and usage patterns At Loughborough University there are over 600 seats in open-access networked PC laboratories. In addition the majority of student halls of residence are networked and 3077 students currently have networked PC’s in their study bedrooms (about 60% of Hall residents, and five times as many Hall PC’s as laboratory PC’s). Since the PC laboratories may be booked for teaching purposes and in order to increase accessibility the tests were made available 24 h a day, 7 days a week and no serious problems with access have been reported. Table 1 details the number of attempts made. Trial 1 relates to all the attempts for the first trial test, Course 1 relates to all the attempts for the first real coursework-assessed test and so on. The last row in the table shows the average number of attempts per student. The first trial test, for most students, was their first experience of computer-based testing, and it is not surprising to find that students attempted this test several times, partly to familiarise themselves with the system. Usage waned slightly from trial test 1 to trial test 2 as students gained experience. Of interest is the trend for students to take the coursework test more than once even though they know that the first attempt made is taken as their performance for module credit. These students are taking time to improve their score without any module credit incentive. Fig. 4 shows the pattern of usage during the first two tests, both trial and real, in Semester 1 of 1999/2000. Note that, given a choice, students will take tests late into the night. Usage stops Table 1 The number of attempts made on trial and coursework tests Test
Trial 1
Course 1
Trial 2
Course 2
No. of attempts Average attempts per student
2420 6.1
552 1.4
1887 4.7
570 1.4
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
61
entirely at 5.00 pm Wednesday when the first trial test is removed from the server (Point A), and resumes the following day at 9.00 am when the first real test is made available. Students were told that the real test would be available until 5 pm on Friday, but in practice the test is still available until the following Wednesday. This is a safety net which is not advertised, designed to allow students with genuine reasons for missing the test to still take it. Note also disruption at point B. This corresponds to a major disc failure on the CAA server shortly after 12.00 pm. Service resumed in the evening of the same day. As compensation for this a decision was made to allow students to continue to submit real test attempts until the following Wednesday. 7.2. Reporting and results The students are given instant feedback on these tests. The lecturer is provided with the following reports at the end of the test period. 7.2.1. Answers.csv This is a csv file to be opened in Microsoft Excel with details of date, time, test name, NDS ID, Student ID Number, surname, initials, score as a percentage. Lecturers can filter the information as desired, tracking the behaviour of a particular student, sorting by chronological order, by score, or alphabetically by surname. 7.2.2. Fullreport.txt A text file showing each individual attempt at the test (each test script) with details of test name, date and time, test file name, answer file name, NDS ID, Student ID number, student name, the time limit, and the time to complete the test.
Fig. 4. Distribution of test attempts during two cycles of testing.
62
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
7.2.3. Summary.txt A text file with details of test name, number of attempts made, maximum score, minimum score, mean score and standard deviation, the maximum, minimum and average time to complete the test, maximum, minimum, mean and standard deviation of the scores for each question library. 7.2.4. Analysis.txt A text file showing the performance of each question delivered in terms of maximum, minimum, mean and standard deviation of the scores achieved, question facility and discrimination. The computer-based testing described here accounts at most for 20% of one out of six module marks. As such it counts for just a small portion of time credit a student will receive for their work. Resources are not available to carry out detailed analyses of student performance, track individual performance, etc. At the present time the results of the tests are simply carried forward as raw marks which contribute to the final module mark.
8. Student feedback In the academic year 1999/2000 nearly 400 engineering students were regularly tested using CAA. Depending upon the programme followed some students were tested twice per semester though most were tested three times per semester, typically in weeks 3, 7 and 10. At the end of the second semester students were given a questionnaire allowing them to comment (anonymously if necessary) on the CAA approach to testing. They were asked to react to 12 questions graded from ‘A: Strongly disagree’ through to ‘E: Strongly agree’. A total of 250 replies were received. Their responses are recorded in Figs. 5 and 6. The overwhelming conclusion from this feedback is that CAA is generally viewed positively by students, so much so that the majority would be pleased if such a regime were used in other modules (question 7). This is certainly in line with the anecdotal evidence collected throughout the year. However (question 6), there is not much enthusiasm for increasing the weighting allocated to CAA. Question 8 reinforces this with more than 50% of responses indicating that they would prefer to keep (in some form) the traditional method of examining. The results of questions 4 and 5 reveal that more students than not stated that they had worked harder than they would have done without CAA, and more than not thought that CAA had encouraged them to work consistently. To this extent, one of the main aims of the project had been achieved. On this questionnaire students also had the opportunity to make written comments in order to develop their scored responses. The written comments are closely correlated with the graphical data. In particular the flexibility of CAA (question 10) was seen as one of its major advantages. Typical comments were:
flexibility — you have lots of time to prepare and flexibility to take exams when you like The flexibility is good, it allows you to take the test when you‘re ready and confident.
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
Fig. 5. Students’ perceptions of CAA.
63
64
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
Fig. 6. Students’ perceptions of CAA (continued).
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
65
I liked the time flexibility involved. It was a great help that the test could be taken at any time in a 24 hr period What negative comments there were often refered to the question format students were most often presented with. Many students felt uneasy with numerical input questions, feeling that too many marks were lost if they made slight numerical errors: Only problem with computer assessment is there is no allowance for slight method errors meaning you have to get the correct answer to get the mark Many students liked the open access nature of the regime at Loughborough, many making explicit their opposition to having supervised tests: It was good to keep on working on math throughout the year. I think supervision would remove the relaxed atmosphere from the tests and would require them to be taken at a set time, removing the ‘open-access’ nature of CAA No supervision decreased the pressure, so it-was easier to concentrate and not rush In can be argued that such CAA tests do not adequately assess the higher level skills or understanding in mathematics. To a large extent this is true. Students are well aware of this, distinguishing as they do in feedback, between credit gained for method and credit gained for a single numerical input. Typical comments (by a small number of students) are: I found the assessments to be an unrealistic reflection of my work. The fact that marks are not awarded for working like every other written test I have taken. The slightest error which would lose you 1 or 2 marks in a ‘real’ exam would get you a big fat zero in the computer exam I feel they are quick and good for your time management but they may not reflect your true ability — which you are able to do more so in the normal written exams. I was originally taught throughout my A-levels to show working. This destroys all this as you only show answers. Would prefer if it was abolished and traditional means implemented (i.e. pen and paper). Please. . . We attempt to offset this criticism by ensuring that the majority of questions address one quantum of a subject demanding an answer which would follow from a small amount of working. With 80% of a module mark being achieved from the formal written examination we do not believe that this is a serious cause for concern, and any disadvantages are outweighted by the benefits of a such a system. Overall comments were positive: I think that it’s an excellent way to ensure that the basics of core material of any course is mastered
66
A.C. Croft et al. / Computers & Education 37 (2001) 53–66
I found the trial test extremely helpful as a practise of what I could expect in the real test The testing method was enjoyable, and the trial tests enabled me to learn constructively Good idea. Some small glitches, but generally successful [1 liked] the fact that we were given over a week for practise tests before the ‘real’ tests In response to this feedback the project team is following up by constructing further tests with a variety of input types other than numerical, for example multi-choice, hot spot, etc. This should alleviate the main concern of students which is that they can lose all marks for a question by making a simple arithmetical error.
9. Conclusions This paper has described the implementation of a CAA system for first-year engineering mathematics. Details have been given of some of the problems associated with attempting to test mathematical skills using a computer, and with construction of appropriate tests. This system has been subject to largescale testing during the autumn of 1999. It has been found that the QMWS works well for non-supervised self-assessment (24 h a day/7 days a week), as a learning tool, and as part of coursework credit. On the whole, student feedback has been positive and student responses indicate that they have worked harder and more consistently than they might have done without such a system. Given that one of the main aims of the project was to encourage consistent participation, the project team believe that the CAA has been a successful component of the Open Learning Environment. Acknowledgements The authors would like to acknowledge the support of Loughborough University who have funded the Open Learning Engineering Mathematics Project. References Croft, A., & Ward, J. P. (2001). A modern and interactive approach to learning engineering mathematics. The British Journal of Educational Technology, 32(2), 195–208. Dempster, J. A. (1994). Question mark designer for Windows (a review). Active Learning, 1, 47. IMA (1999). Engineering mathematics matters: curriculum proposals to meet SARTOR3 requirements for chartered engineers and incorporated engineers. The Institute of Mathematics and its Applications, September 1999. ISBN 0 905091 09 4. Southend-on-Sea. Phelps, J. M., & Danson, M. (2000). Journal of quality assurance in education. Special Edition on Learning Technologies, February 2000.
