Website. Description. Easy Test Maker http://www.easytestmaker.com/. Online test generator to create, format, publish in the web and grade multiple question ...
COMPUTER-BASED TOOLS FOR THE ASSESSMENT OF LEARNING PROCESSES IN HIGHER EDUCATION: A COMPARATIVE ANALYSIS J.M. Fuentes, A.I. García, Á. Ramírez-Gómez, F. Ayuga BETI Teaching Innovation Group, Universidad Politécnica de Madrid, E.T.S.I. Agrónomos (SPAIN)
Abstract Assessment of learning processes is an essential part of the educational practice. On the one hand, it allows evaluating the knowledge, skills and attitudes achieved by the students; on the other hand, results can be used to make decisions for the improvement of the educational practices. The success of e-learning has allowed the emergence of new computer-based tools designed to enhance each stage of the educational process. Information technologies applied to education and particularly the use of Internet allow the creation of interactive self-assessment tests that can be useful tools to motivate students in learning tasks and to check their progress previously to be definitely examined. Computer-based assessment (CBA) offers the following advantages against traditional paper-based exams: grades and feedback can be immediately provided to the student; greater flexibility in the location and the time to carry out the exam is allowed; interactive activities and multimedia tools can be incorporated to the assessment process; teacher’s interpretations and legibility problems are avoided; and time and cost effectiveness, among others. Disadvantages include: authoring and security possible failures, additional work for teacher to generate tests, possible technical failures and difficulties to carry out and automatically mark certain types of exams (e.g. essay questions or engineering problems, etc.). The paper discusses the technical features required by computer-based assessment software to be used in higher education courses. A comparative analysis of different applications to conduct on-line and off-line exams has been carried out, with particular attention to the available types of questions, the possibility of incorporating multimedia content to the exams, the possibility of creating and managing a bank of questions, available test generation options (i.e. maximum number of questions, time control, number of attempts,…), scoring options or the ability to provide feedback to the students on their answers, among some other questions. Keywords: e-assessment, computer-based assessment, CBA, higher education.
1
INTRODUCTION
Assessment of learning processes is an essential part of the educational practice, as it allows measuring the achievement of learning objectives by students, and improving the quality of the teaching practice [1]. It implies that assessment may affect decisions about marks, placement, instructional needs, and curricula. Learning assessment can be defined as a process designed to identify, collect and interpret information on the knowledge, skills and attitudes achieved by the students [2]. Nevertheless, the concept has undergone significant evolution over the past few decades, moving from an isolated activity taking place at the end of the learning process with the only purpose of giving a mark to the student to become a continuous and systematic process directed toward making decisions for the improvement of the educational practices [3]. Considering the aims and the way to be developed two major types of assessment can be identified: formative and summative assessment [4]. Formative assessment is a systematic and continuous activity during the learning process aims to provide feedback to the student on their progress and identify weaknesses and gaps to be corrected. So, the emphasis in formative assessment is to provide the student with written or oral feedback from the teacher rather than a numerical mark. This type of assessment can be developed through selfassessment activities performed by the students themselves and observations of the teacher to the work carried out by the student.
Proceedings of INTED2014 Conference 10th-12th March 2014, Valencia, Spain
0976
ISBN: 978-84-616-8412-0
On the other hand, summative assessment is given to summarize the level of competence the student achieved at the end (or at certain point) of the learning process by means of a particular grade or certification. In higher education, summative assessment is usually based on exams, exercises, oral presentations and individual or group dissertations. In both assessment methods, the teacher assumes two different roles. In the case of formative assessment, the teacher plays the role of a coach and facilitator, so as to encourage students’ learning; but in summative assessment, he performs the role of a judge about students’ achievements at a given period. Taking into account the discussion above, it is quite clear that formative assessment is generally favoured over summative assessment since it gives students an opportunity to have an idea about their strengths and weaknesses as they proceed through a given course and provides teachers with pertinent information concerning the students’ progress as well as the efficiency of the teaching methods and materials. In any case, assessment requires an objective judgment, far from mere incidental knowledge, intuitions and opinions. The use of Information and Communication Technologies (ICT) in university is a currently fully consolidated fact. The increasing success of e-learning has allowed the emergence of new computerbased tools particularly designed to enhance each stage of the educational process. Nevertheless, although online assessment tools are gradually being introduced and expanded, its use may be still considered at present a minority activity [5]. The paper discusses on the advantages and disadvantages that computer-assisted evaluation offers against conventional paper-based methods. Technical features required by computer-based assessment software to be used in higher education courses are also examined in the manuscript and a comparative analysis of 20 applications to conduct on-line and off-line exams is carried out, with particular attention to the available types of questions, the possibility of incorporating multimedia content to the exams, the possibility of creating and managing a bank of questions, available test generation options (i.e. maximum number of questions, time control, number of attempts,…), scoring options or the ability to provide feedback to the students on their answers, among some other questions.
2
COMPUTER-BASED EXAMS VERSUS TRADITIONAL ASSESSMENT METHODS. ADVANTAGES AND DISADVANTAGES
Information and Communication Technologies (ICT) have become an essential component in higher education for the last decade. At present, Spanish universities put more than 100.000 computers at students’ disposal, which means a ratio of 1 computer for every 10 students, and more than 60% of the university classrooms have a multimedia projector [6]. Due to the rapid and outstanding development of ICT in the last few years, e-learning has become a fully consolidated reality in the vast majority of higher schools and universities around the world. In the US, for example, over 6.7 million university students took at least one online course in 2012. This figure means a 32 per cent of the total enrolled university students in this country [7]. In Spain, meanwhile, almost 90% of universities have an institutional e-learning platform and over 60% of the traditional on-site courses use it as an additional learning resource [8]. The increasing success of e-learning has allowed the emergence of new computer-based tools particularly designed to enhance each stage of the educational process. Information technologies applied to education and particularly the use of Internet allows the creation of interactive selfassessment tests that can be useful tools to motivate students in learning tasks and to check their progress previously to be definitely examined [9-11]. Practical benefits of computer-based assessment tools, mainly based on multiple choice questions, have been previously reported in literature [12-16]. Among some others, Computer-based assessment offers the following advantages against the traditional examination methods: -
automatic marking: marks and feedback can be immediately provided to the students, for even the largest class sizes;
-
less time is needed for marking;
-
greater frequency of assessment is allowed, providing detailed information on the students’ progress and helping teachers to early detect possible problems;
-
students have greater flexibility in the location and the time to take the exam;
0977
-
interactive activities and multimedia tools can be incorporated to the assessment process;
-
impartiality in the results, as teacher’s interpretations and legibility problems are avoided;
-
time and cost effectiveness advantages;
-
motivation of the students is increased and skills in the use of ICTs are encouraged.
But, although credited with numerous advantages, online assessment also seems to suffer from a number of shortcomings [12, 15 and 17]. Disadvantages include: -
possible authoring and security failures;
-
additional work for teacher to generate a broad bank of questions;
-
possible technical failures in the server, computers, network or software when performing the test;
-
technical difficulties to automatically mark certain types of exams (i.e. essay questions or engineering problems);
-
the use of objective online assessment tools can have direct negative effects on students’ approaches to learning by encouraging narrow reproduction rather than the development of higher order cognition abilities of synthesis and evaluation.
From the above it can be concluded that computer-assisted assessment is beneficial for both students and lecturers. For students, it serves to motivate them and improve their competence in the use of ICT, at the same time that they can be provided with greater flexibility in the location and the time to take the exam. On the other hand, lecturers can provide the student with immediate comments or guidelines to conduct their learning. In any case, computer-based or traditional assessment must be carefully planned to make it truly effective for student learning. The mere fact of using computer techniques for assessment is not going to produce significant improvements in the learning process without a careful design of the tests to be conducted and the formative feedback to be provided on students’ responses [18].
3
COMPARATIVE ANALYSIS OF SOFTWARE TO CONDUCT COMPUTER TESTS
At present, there is abundant software to generate and administer computer-based exams. Most of the e-learning platforms (Moodle, WebCT, Dokeos, .LRN, etc.) have specific modules to prepare and conduct on-line tests [8]. On the other hand, numerous commercial and freeware computer programs specifically designed to generate and conduct online and offline tests have been developed for the last decade [10-11]. A selection of software is shown in Table 1. Table 1. List of selected applications to conduct computer-based tests Software
Website
Description
Easy Test Maker
http://www.easytestmaker.com/
Online test generator to create, format, publish in the web and grade multiple question types tests.
Exam Professor
http://www.examprofessor.com/
Exam Professor is a web-based tool to build and manage exams, tests or quizzes.
ExamView
http://www.einstruction.com/produ cts/examview/examviewassessment-suite
ExamView® allows creating, administering, and managing online or offline assessments using multiple question formats. Content compatible with publisherprepared question banks for over 10,000 textbooks.
Hot Potatoes v6
http://hotpot.uvic.ca/
The Hot Potatoes freware suite includes six applications to create interactive multiple-choice, short-answer, jumbled-sentence, crossword, matching/ordering and gap-fill exercises for the World Wide Web.
iGivetest v2
http://www.igivetest.com/
iGiveTest is a comprehensive solution for creating and administering a bank of questions and providing tests on the Internet and Intranet.
0978
Software
Website
Description
Itest
http://itest.sourceforge.net/
iTest is an application for computerised examination. It consists of two programs: iTestServer (question/answer database editor and exam server) and iTestClient (used by students to connect to the server and pass tests).
Learnclick
http://www.learnclick.com/
Allows creating online quizzes and gap-filling exercises, sharing them with students and managing grades.
Lime Survey
http://www.limesurvey.org/
LimeSurvey is an open source tool for developing online surveys and tests.
ProProf Quizmaker
http://www.proprofs.com/quizschool/
ProProfs is an advanced, free online tool to create online tests, quizzes and exams.
Powerquizpoint v.1.5
http://www.digitalofficepro.com/Po werpoint/quiz-creator.html
This application allows creating Flash Quizzes from PowerPoint presentations and Publish them to Adobe Flash format.
QuestBase
http://www.questbase.com
QuestBase is a free, cross-platform application to create and manage assessments, tests, quizzes and exams, both on-line and printed.
Question Mark v.5.4
http://www.questionmark.com
This commercial software has been designed to author, schedule, deliver, and report on surveys, quizzes, tests and exams.
QuestionPro
http://www.questionpro.com/
Question Pro allows easily creating online surveys or questionaires and collecting responses.
Question Tools
http://www.questiontools.com
Question Tools provides instantly available eassessments on a wide range of topics, plus the opportunity to create your own online tests or exams.
Quizcreator
http://www.wondershare.com/pro/ quizcreator.html
Wondershare QuizCreator is a powerful quiz maker to create and publish on-line Flash-based quizzes and surveys with multimedia.
Quizmaker
http://www.articulate.com/products /quizmaker.php
Commercial software for on-line or offline assessment.
Quizstar
http://quizstar.4teachers.org
Test Creator
http://centronsoftware.com/tcpage .html
This software allows creating a database of questions and delivery online and off-line tests.
Test Generator
http://www.testshop.com/
Test Generator is a professional online test maker and assessment application designed for test creation, delivery and administration.
http://www.eduware.com/
This computer tool allows creating tests from own questions or a database over 180,000 questions Asssign test for students to take online. Generate reports based on their grades and print tests.
Test Wizard
QuizStar allows creating, disseminating automatically grading online quizzes for students.
and
According to the way how the tests are created and distributed, two types of applications can be differentiated: web applications designed to conduct online tests (questions database and tests are hosted in a web server and lecturers and students can access the application through a web browser) and offline tools (computer programs to be installed and operated in a computer or a network of computers). The advantage of introducing the questions directly through the web is that the teacher can do it from any computer at home or on campus, but has the disadvantage of possible problems associated with the operability of the network while the exam is conducted. Although there are some free applications to create, conduct and grade computer tests, such as Hot Potatoes, iTest, QuestBase, QuestionPro or QuizStar, among some others, most of the existing tools are commercial software that require a monthly or annual payment. In some cases, the licence fee to be paid depends on the number of enrolled students or the number of tests that will be performed. In general, as expected, the most complete and higher educational valued tools are the most expensive.
0979
A comparative analysis of applications to conduct on-line and off-line exams has been carried out, with particular attention to the available types of questions, the possibility of incorporating multimedia content to the exams, the possibility of creating and managing a bank of questions, test generation options (i.e. maximum number of questions, time control, number of attempts,…), scoring options or the ability to provide feedback to the students on their answers, among some other questions. Results of the comparative analysis are summarized in Tables 2 and 3 Table 2. Comparative analysis of computer-assisted assessment applications: General features.
Software
Type
Types of questions
Multimedia
Allows creating a bank of questions
Easy Test Maker
Online
Multiple choice; true/false; fill-inthe-blank; matching; short answer.
Images
No
Exam Professor
Online
Multiple choice
Images
Yes
Examview
Offline
Multiple choice; bimodal; yes/no; true/false; modified true/false; fill-in-the-blanks; matching; numeric response; problem; short answer; essay; case; others
Images; equations
Yes
Hot Potatoes 6
Offline
Multiple choice; gap-fill; short answer; jumbled sentence; crossword; matching/ordering
Images
No
iGivetest v2
Online
multiple choice; multiple correct answers; true/false; short answer; essay
Images, equations, audio, video, animations
Yes
Itest
Offline
Multiple choice; multiple correct Graphics (SVG answers files)
Yes
Learnclick
Online
Multiple choice; multiple correct answers; fill-in-the-blanks; matching; drag & drop
Images, animations, video, links
No
Lime Survey
Offline
28 different question types
Images, tables, video, animations
Yes
Proprof Quizmaker
Offline
Multiple choice; check boxes; true/false; fill-in-the-blanks; matching; short text; essay
Images, tables, animations, equations, links
Yes
Powerquizpoint 1.5
Offline
Multiple choice
Images, audio, video, animations
Yes
QuestBase
Online
Multiple choice; multiple correct answers; true/false; fill-in-theblanks; essay
links
Yes
0980
Type
Types of questions
Multimedia
Allows creating a bank of questions
Question Mark 5.4
Offline
22 different types of questions, including: multiple choice; true/false; fill-in-the-blanks; matching; ordering; drag & drop; short text; etc.
Images, equations, animations, audio, video, links
Yes
Question Pro
Online
Multiple choice; open text; ordering; drag & drop; other advanced types
Images, audio, video, animations
No
Multiple choice; true/false; multiple answer; short text; open text; hotspot
Images, video, audio
No
18 question types, including: Multiple Choice; fill-in-theblanks; matching; Likert scale, short text; essay; etc.
Images, flash video and audio, including voice-over, textto-speech, etc.
Yes
25 question types, including; Images, video, multiple choice; true/false; fill-inaudio, flash the-blanks; word bank; animations, web matching; ordering; drag & objects drop; hotspot; etc.
Yes
Software
Online (An Question Tools
Quizcreator
offline test editor is also available)
Offline
Quizmaker
Offline
Quizstar
Online
Multiple choice; multiple answer; true/false; short text
Links to multimedia files
No
Test Creator
Online and offline versions
multiple choice; true/false; fill-inthe-blanks; short text
Images and sounds
Yes
Test Generator
Offline
Test Wizard
Online and offline versions
11 question-types, including: Images, audio, multiple choice; true/false; fill-inPDF files, flash the-blanks; short answer; animations or essay; ordering; matching; video hotspot or essay
Yes
Images, audio, video, equations
Yes
Multiple choice; free text
0981
Table 3. Comparative analysis of computer-assisted assessment applications: Test options. Allows Number of Automatica negative attempts lly grading grading on control options mistakes
Shuffle questions & answers
Time control options
Easy Test Maker
Yes
No
No
Yes
No
No
Exam Professor
Yes
Yes
Yes
Yes
No
No
Examview
Yes
No
Yes
Yes
No
No
Hot Potatoes 6
Yes
Yes
Yes
Yes
No
Yes
iGivetest V2
Yes
Yes
Yes
Yes
No
Yes
Itest
Yes
Yes
No
Yes
Yes
No
Learnclick
No
No
Yes
Yes
No
Yes
Yes / No
Yes
Yes
No
No
No
Proprof Quizmaker
No
No
No
Yes
Yes
Yes
Powerquizpoint 1.5
No
No
Yes
Yes
No
Yes
QuestBase
Yes
Yes
Yes
Yes
Yes
Yes
Question Mark 5.4
Yes
Yes
Yes
Yes
Yes
Yes
Question Pro
No
No
Yes
Question Tools
Yes
Yes
Yes
Yes
Yes
Yes
Quizcreator
Yes
Yes
Yes
Yes
Yes
Yes
Quizmaker
Yes
Yes
Yes
Yes
No
Yes
Yes / No
Yes (limited)
Yes
Yes
No
Yes
Test Creator
Yes
Yes
No
Yes
No
No
Test Generator
Yes
Yes
Yes
Yes
Yes
Yes
Test Wizard
Yes
Yes
No
Yes
No
No
Software
Lime Survey
Quizstar
No (in free No (in Free version) version)
Allows, including feedback
No
Regarding the type of questions available in the different applications there are some significant differences. Multiple choice questions with only one correct answer are available in all the applications analysed. Other types of questions, such as true/false, fill-in-the-blanks, multiple correct answers, matching, ordering or short answer are also available in most cases. Nonetheless, more advanced options, such as hotspot or drag-and-drop exercises are currently only available in commercial applications. ExamView, for example, allows formulating calculation algorithms, so that different problems can be reformulated one and again by changing the input data. On the other hand, Question Mark allows recording the participant's voice to answer open-text questions. Most of the applications evaluated allow lecturers to create a bank of questions that can be used to generate examinations at random. Some programs, such as ExamView, Question Mark or iTest, allow classifying the questions included in the database according to different criteria (i.e. topic, skill, level of difficulty, etc.). So, the sequence of questions to be included in a test can be automatically selected according to different criteria suggested by the educator, such as the grade obtained by the student on previous attempts. This possibility allows converting the computer tests in true learning tools, as the type and amount of questions offered to students are based on their results. As shown in Table 2, the different applications allow incorporating multimedia content to the tests. However, there are some differences in the way how these resources are included. In some cases
0982
images, video or animations can be directly displayed together with the question statement; in some other cases, multimedia content should be accessed through a link and an appropriate player need to be installed in the student’s computer. Applications such as ExamView, Question Mark or Test Wizard allow incorporating equations when formulating the questions, what can be of great interest in technical or engineering courses. Although almost all of the applications analysed offer the option to automatically grading the tests, differences can be found in the way to do it. So, for example, the possibility of negative scoring when student’s answer is wrong is not a frequent feature. The chance of providing feedback to the student’s answers is an interesting option in formative assessment. Virtually all the computer programs offer the possibility of displaying the correct answer after answering each question or when the exam is completed. Additionally, some applications enable lecturers to provided custom-made comments and explanations of why it is right or wrong a particular answer. Some of the programs analysed can also provide clues to the student to answer a question, if decided by the educator. Aid may consist of mere clues, such as the first letter of the word he is looking for, equations, or elaborate explanations.
4
CONCLUSIONS
A review of the state of the art and a comparative analysis of different computer applications to conduct on-line and off-line exams are presented in this paper. The main conclusions of this work are summarized below: -
Assessment of learning processes is an essential part of the educational practice, as results can be used to improve the educational practices.
-
Information technologies applied to education and particularly the use of Internet allows the creation of interactive self-assessment tests that can be useful tools to motivate students in learning tasks and to check their progress previously to be definitely examined. Benefits and constrains of computer-based assessment are discussed on the paper.
-
Numerous commercial and freeware computer programs particularly designed to generate and conduct online and offline tests have been developed for the last decade. Although basic types of questions (multiple choice, true/false, fill-in-the-blanks, multiple correct answers, matching, ordering and short answer) are available in most cases, more advanced options (i.e. drag & drop, hotspot or oral answers) are only available in the most sophisticated commercial software.
-
Classification of the questions included in the database according to topics or level of difficulty can be used to personalize the tests to be conducted according to the particular needs of each student. The chance to include multimedia content is also a distinctive and particular interesting option. Equations can be useful in technical or engineering courses.
-
Feedback on students’ answers and aids for answering the questions can also be used with formative purposes. Most of the programs analysed offer this possibility to a great or a lesser extent.
ACKNOWLEDGEMENTS Financial support for this research has been provided by Universidad Politécnica of Madrid, via the project IE13-14-02005 “Desarrollo de materiales de auto-aprendizaje y sistema de tutorías on-line en asignaturas del antiguo plan de estudios de la ETSI Agrónomos” [Development of self-learning materials and online tutorial tools for non-in person courses at the Technical School of Agricultural Engineering of Madrid].
REFERENCES [1]
Villardón, L. (2006). Evaluación del aprendizaje para promover el desarrollo de competencias. Educatio Siglo XXI, 24(1), pp. 57-76.
[2]
Berry, R. (2008). Assessment for Learning. Hong Kong: Hong Kong University Press.
[3]
Earl, L.M. (2003). Assessment as Learning: Using Classroom Assessment to Maximize Student Learning. Thousand Oaks (CA, USA): Corwin Press.
0983
[4]
Scriven, M. (1967). The methodology of evaluation. In: Tyler R.W. et al. [Ed]. Perspectives of curriculum evaluation, pp. 39-83. Chicago: Rand McNally.
[5]
Gipps, C. (2005). What is the role for ICT-based assessment in universities? Studies in Higher Education, 30(2), pp. 171-180.
[6]
CRUE (2012). UNIVERSITIC 2012: Descripción, gestión y gobierno de las TI en el Sistema Universitario Español. Madrid: Conferencia de Rectores de las Universidades Españolas (CRUE).
[7]
Allen, I.E.; Seaman, J. (2013). Changing Course: Ten Years of Tracking Online Education in the United States. Massachusetts: Babson Survey Research Group.
[8]
Fuentes, J.M.; Ramírez-Gómez, Á.; García, A.I.; Ayuga, F. (2012). Web-based education in Spanish Universities. A Comparison of Open Source E-Learning Platforms. Journal of Systemics, Cybernetics and Informatics, 10 (6), pp. 47-53.
[9]
Mogey, N.; Watt, H. (1996) The use of computers in the assessment of student learning. In: Stoner, G. [ed.]. Implementing Learning Technology. Edinburgh: Learning Technology Dissemination Initiative, Heriot-Watt University.
[10]
Lara, S. (2001). La evaluación formativa en la universidad a través de Internet: Aplicaciones informáticas y experiencias prácticas. Pamplona: EUNSA.
[11]
Olmos, S. (2008). Evaluación formativa y sumativa de estudiantes universitarios: Aplicación de las Tecnologías a la evaluación educativa. Ph.D. Tesis. Universidad de Salamanca.
[12]
Brown, S.; Bull, J.; Race, P. (1999). Computer-assisted Assessment in Higher Education. London: Kogan Page Ltd.
[13]
Thelwall, M. (2000). Computer-based assessment: a versatile educational tool. Computers and Education, 34(1) pp. 37–49
[14]
Bull, J.; McKenna, C. (2004) A Blueprint for Computer-Assisted Assessment. London: Routledge
[15]
Chiheb, R.; Faizi, R.; El Afia, A. (2011). Using objective online testing tools to assess students’ learning: potentials and limitations. Journal of Theoretical and Applied Information Technology, 24(1), pp. 69-72.
[16]
Loewenberger, P.; Bull, J. (2003). Cost-effectiveness analysis of computer-based assessment. 11 (2), pp. 23-45.
[17]
C.G. King, R.W. Guyette, C. Piotrowski (2009). Online exams and cheating: an empirical analysis of business students’ views. The Journal of Educators Online, 6 (1) (2009).
[18]
Taras, M. (2003) To feedback or not to feedback in student self-assessment. Assessment & Evaluation in Higher Education, 28 (5), pp. 549-565.
0984
