WEB-BASED EVALUATION SYSTEM FOR ONLINE COURSES AND LEARNING MANAGEMENT SYSTEMS Chakkrit Snae1 and Michael Brückner 2 1,2
Dep. of Computer Science and Information Technology, Naresuan University, Phitsanulok, Thailand, e-mail :
[email protected],
[email protected]
Abstract— Online course websites must be of high quality in order to help and stimulate students gathering knowledge, understand the meaning, interpret problems and solutions, use and apply concepts and methods, analyze and classify material, synthesize structures and patterns, and make judgments about the value of materials and ideas. In this paper the authors present an extended version of an earlier evaluation framework for online courses and Learning Management Systems (LMS). The new version of the evaluation framework deals with several shortcomings, which the authors found when applying it in various e-learning courses. The main features added to the existing framework are: (1) web-based questionnaire in Thai and English to facilitate easier use and evaluation of the results, (2) an authorization process for accessing the questionnaire, (3) a role-based evaluation process, and (4) online calculation of the evaluation results. The new framework will be incorporated into O-DEST (OntologyDriven E-learning System for Thai environment) which the authors described elsewhere. Index Terms—web-based assessment, e-Learning, online course, learning management system
I. INTRODUCTION Electronic learning or e-learning generally refers to computer-enhanced learning. Systems for e-learning can be placed in a computer network, in the Internet (sometimes called web based training), or they can be stand-alone systems. The materials used in e-learning are manifold and reach from simple study texts passages to enhanced graphics and videos as well as computer-generated quizzes and tests. A Learning Management System (LMS) is a set of software tools designed to manage learning interventions by users. LMSs extend conventional training records management and reporting by functions, such as learner self-service (e.g. self-registration on instructor-led Training), training workflow (e.g. user notification, manager approval, waitlist management), the provision of on-line learning (e.g. Computer-Based Training, Read & Understand), on-line assessment, management of Continuous Professional Education (CPE), collaborative learning (e.g. application sharing, discussion threads), and training resource management (e.g. instructors, facilities, equipment), are some of the additional dimensions to leading Learning Management Systems. Most LMSs are web-based to facilitate "anytime, any place, any pace" ac
cess to learning content and administration. For a list of features provided by various LMSs refer to [1], [2]. Learning Management Systems (LMS) are of great importance for an organization to set up an online course environment. For universities, which mostly work under tight budgets, the upcoming open source LMS systems are of increasing interest. For an overview of open source LMS see [3]. The term quality has been defined in many ways. ISO 9000 says quality is the "degree to which a set of inherent characteristic fulfills requirements". The American Society for Quality (Source: http://www.asq.org/glossary/q.html) states that quality is "a subjective term for which each person has his or her own definition. In technical usage, quality can have two meanings: (1) the characteristics of a product or service that bear on its ability to satisfy stated or implied needs, (2) a product or service free of deficiencies." Gilbert (1992) [4] sees quality assurance as “the assembly of all functions and activities that bear upon the quality of a product or service so that all are treated equally, planned, controlled and implemented in a systematic manner.” Planning, controlling and implementing product or service quality is then obviously part of a management process, such as Total Quality Management (TQM). Focusing on the quality of Learning Management Systems and online courses we can extract several components to be considered in the management process (based on [5]), such as system performance, features, reliability, conformance and compatibility, maintainability, and aesthetics. Elsewhere, the authors proposed an Ontology-driven Elearning System for Thai learning environment (O-DEST) which is based on ontologies to represent conceptual descriptions of the specific content as they help to identify appropriate items and relationships in a given knowledge domain. O-DEST provides a unified platform for logging, assessing, planning, delivering contents, managing records, and reporting. These activities help both the self-paced and the instructor-led learning processes. All these activities are represented to the end user (or organization) as web based applications. In fact, learning management systems are highly distributed systems in the web. One course presents an integrated structure of many learning resources that can be hosted on different web locations. The same resources can be combined with others to set up different courses. Also, more student groups can learn many courses at the
same time. In these conditions, the system must have powerful management features. This means that a system needs specialized e-learning system properties and the capacity to perform the described components and metadata of the elearning system [6]. In this paper the authors concentrate on the web-based evaluation framework of online courses and LMS, based on web-based questionnaires, which are directed at different target groups, such as students (for the course contents) and administrators (for the web design and maintainability). This paper is organized as follows: after a brief introduction into the O-DEST system in Sect. 2, the evaluation criteria are described in Sect. 3. Sect 4 shows some of the results of the evaluation processes. The we draw conclusions from this study and deal with further work, which the authors feel useful for this subject.
• •
•
• • • •
II. O-DEST The design and implementation of O-DEST has been described in [6]. Here we present only a few features of the system. O-DEST is designed with three basic modules: (1) tools for the teachers, (2) tools for the students, (3) tools for the administration. (1) comprises functions which help the teachers in creating learning objects, connecting the objects to existing ones, re-using objects, and capturing the metadata. As teachers are responsible for entering the students´ data and observing the progress of the class there is a need for administrative elements in this module. Teachers can monitor results of the tests, quizzes, assignments, and sessions. (2) is designed to enable students to master the learning material and meet the learning goals of the course. Students can communicate their expectations and interests, favorites, predispositions and real skills. In the use of system they are guided through the learning material and can actively mark important points, get contextual help and are subject to skills measurement. Students can also collaborate with other students, teachers, and administrators. (3) supports different management functions and tasks of the system. For example: maintaining and updating student and teacher records, administering the domain knowledge and functions for the system and data security. The client side has the following modules: •
•
• • •
Register: new users/members fill in the information about themselves to get access to the elearning system Login: checking and confirming username and password of users/members before granting access to the system Edit personal data: for updating personal information of users/members Student usage record: showing the history of access and learning of each user/member Select subject: is for choosing subjects/lessons which users want to learn
•
Select chapter: is for choosing chapters of subjects/lessons that users want to learn View quiz score: is for checking all marks/scores and results of learning given in percent, it shows the status of the learning process Download documents: is for downloading learning material, e.g. PowerPoint, Word, PDF, and multimedia files Take course: is for learning the subject that users have chosen Take quiz: is for taking quiz/exam after learning the subject Review quiz solutions: is for going through explanations and solutions of quiz/exam Evaluate: filling out a role-based questionnaire (depending on being a student, teacher or admin) about online course and LMS Logout: is for exit from the system
The server side comprises the following modules and contents: • •
• • • • • • • • • •
Available courses: is for displaying all subjects for which course material is available for learners, Courses descriptions: is for an overview of all subjects and lessons, e.g. how many chapters each subject might have and what the details in each chapter are, Courses contents: is for displaying all detail of subjects/courses, Lecture documents: is for downloading documentation for each subject or course, Multimedia: is for studying by multimedia lessons which can be done using Macromedia Captivate Quiz score: is for displaying learning results of oneself Pre-teaching quiz: is for testing before taking courses/lessons Post-teaching quiz: is for testing after taking courses/lessons Time counter: is for displaying time during examination Quiz solutions: is for displaying the solution of the tests, examinations, and quizzes All usage records: is for displaying history of oneself Edit database: is for adding or updating information and data in database
The teacher side contains the following modules and contents:
• • •
Add subjects: is for adding more subjects/lessons Add chapters: is for adding more chapters in each subject that is currently available Add exam: is for adding, editing and designing quizzes and exams in each subject
The admin side contains the following modules and contents: • •
All usage records: is for displaying all or some of the logs of the users/learners Edit database: is for adding and updating the database
In order to get results students are forced to work through the students’ questionnaire after they have finished all the course materials and after they have taken the tests. The actual grade of the students is held back until they have finished the answer process of the questionnaire. Figure 1 shows how the questionnaire is being used during the evaluation process. In terms of client/server the webbased questionnaire is being held at the server side. If students want to fill out the questionnaire they are being checked whether or not they have finished the online materials and have taken the necessary tests (authorization process). If they meet all necessary conditions, the questionnaire is sent to the client side for being filled out. The response is sent back to the server, where it is processed by a PHP script for calculating the marks and results. A feedback is sent back to client stating that the questionnaire has been received. The results are then stored in a database.
(in the following referred to as the UMES standard) as a guideline for evaluation of edutainment enhanced e-learning courses [7]. As we pointed out earlier [6, 8] this standard matches our e-learning course development and management process quite well. The main categories of the UMES standard are (1) course prerequisites, (2) course objectives and learning results, (3) course material, (4) interaction between students and instructors, and (5) students’ assessment. These main categories are subdivided further: (1) is split into what the students have to know to start the course and which technology requirements are there (3) is split into the course contents, multimedia presentation, pacing, and reusability of the contents (4) is split into tools for interaction, supporting students, feedback, chat & online discussions, navigation of the e-learning website, display (labeling according to information architecture), time the instructor devotes to the course feedback. In the course of the role-based evaluation process for various online courses the authors have outlined a more detailed catalog of criteria which are shown below. The criteria correspond to the Learning Management System and its properties as well as to the online course materials presented and worked through by the students. 1. Web Design o Components -
o
- Standard font type – Font type should be no more than 3 types in one page and no need to install more fonts and easy to read and clear - Font format (bold, italic, underline) should be proper e.g., use bold format for heading and normal for details - Font size is suitable for page - Font type and format are harmonious every page in the system
Figure 1 Web-based evaluation system
III. EVALUATION CRITERIA We use the UMES (University of Maryland Eastern Shore) Online Course Learning Object Evaluation Rubric
The name of website is related to the details Each page title is related to details There is a member login system A web board for suggestions and discussion is provided for the LMS as well as for the individual course contents - A list with frequently asked questions (FAQ) is provided for the LMS - A web counter for counting page visits helps to find the favorite web pages - Help desk and user manual menu for use of elearning - Attractive and creative design, should be different from other pages - No infringement of copyrights of pictures, texts, and other content Fonts
o
Colors, e.g. font, background, picture and other use in the system and web pages
o
- Use beautiful color and not too many colors in one page - Use color to convey meaning, e.g. red for new event - Suitable color for background text and picture colors - Different color between text and link - Color should be harmonious every page in the system Graphical elements
o
- Picture coveys to the purpose or can be explain better than text - Size of picture should be suitable with text and page - Picture Type and size should not be too big Animations
o
- should coveys to the purpose or can be explain better than text - Size of animation should be suitable with text and page - animation type and size should not be too big - Speed of movement should be smooth and nature - Amount of animation should be proper not too many in one page Videos
o
- Size of video should be suitable with text and page - video type and size should not be too big - Speed of movement should be smooth and nature and looks real - show how to open video or program that can be used to play Sound
o
- Clear sound both oral presentation and background - Proper sound not annoying users - sound type and size should not be too big - sound should be harmonious to detail - show how to open sound or program that can be used to play - option to choose for sound or no sound Buttons and icons
o
- proper size - proper and same position in each web page - easy to understand and standard - same format and size in every page Position of components (picture, text etc)
o
- Accuracy - Display Thai and English character correctly Support search engine
o
Accuracy of links (no dead links) - Links to file documents within website - Links to other web sites/web pages - File download and program
3. Instructional Design o
Lesson objectives
o
- behavior objective which can be assessed - precise and clear, cover all lesson - suitable for students Lesson/detail presentation
o
- sequence of lessons should be from ease to difficulty - overview from general to common - from concrete object to abstract - from advantage to theory that need to be learnt Use of language - clear an suitable for students - polite writing not jargon - correct spelling and grammar, especially technical terms, or mathematics formula - has been arranged in order for user atmosphere
o
o
- Suitable for students - Interactive levels including contact, e.g. mail, chat, web board - Emphasize student activity via web, e.g. do assignment - Emphasize teacher role activity, e.g. assign coursework and evaluate via web - Support and develop creative ideas - No need more facilities Time use
o
- Limit enough/appropriate time for learners - Flexible time for ability level of learners - No time limit but should state standard use and record time use Test/Evaluation
o
- Right to the objective and cover all lesson detail - Test format is appropriated to learners - Various tests/methodologies, e.g. multiple choices, matching, filling, explanation - learners can choose test level, e.g., easy, medium, difficult test levels - Evaluate intervals, e.g. before, start, during and after learning - Amount of tests are suitable for time limit - Tests are accurate and fair - Solution with additional explanation/discussion - Report result of tests Advantage of further application/ use
- suitable and balance - standard, e.g. left to right or top to bottom - harmonious in each page 2. Website development techniques o
Structure
o
- Files and folders naming/title - Specify main folders and sub folders Display of Thai and English language
Activities in teaching (activity for gaining understanding, skill, and objective lesson)
- For further research or give widely general knowledge or specific knowledge - Learner can learn by themselves 4. Details of lessons o
Lesson levels suitable for learners
o
Formally Accurate
o
Complete lesson and each details relate to one another
o
Appropriate lesson, precise and not too short or long
o
Up to date lesson/modernity
o
Should include ethical behavior
o
Reliability sources or references of lesson
o
Creative ideas and analysis for learners
IV. ANALYSIS OF RESULTS In this section the authors show results of the web-based evaluation process for a course on Software Engineering. This represents a course, which is held every term for students of Computer Science, Information Systems and Law & Information Systems of Naresuan University. The students of this course together with four instructors and two administrators (web designer and website developer) answered the questionnaire.
Table 1 Results of the web-based questionnaire Criteria
Percentage
Website design and website development techniques
87.8 %
Instructional design
80.7 %
Details of lessons
85.0 %
Overall
88.6 %
From results like this we can find the courses and areas of the LMS that need further improvement. For the Software Engineering we should concentrate on instructional design
V. CONCLUSION AND FURTHER WORK This study was about a new web-based framework for the role-based evaluation of online courses and Learning Management Systems. As a result of the evaluation we can draw conclusions about the necessary improvements of the online courses covered and of the Learning Management System. More improvements can be reached when we implement a monitoring system of the students’ learning behavior, e.g. by measuring the time they use for a particular topic. An-
other interesting point is to provide a consulting system based on the students results in training tests before midterm and finals as well as the measured learning development, e.g. by predicting the final grade or by suggesting specific subjects and areas the student should concentrate on.
VI. REFERENCES [1]http://www.edtechpost.ca/pmwiki/pmwiki.php/EdTechPost/OpenSource CourseManagementSystems (last viewed Oct. 15, 2007) [2] http://elearning-india.com/content/blogcategory/19/38/ (last viewed Oct. 3, 2007) [3] EDUCAUSE EVOLVING TECHNOLOGIES COMMITTEE (2006). OVERVIEW OF OPEN SOURCE LEARNING MANAGEMENT SYSTEMS. SEPTEMBER 15, 2006, available online as PDF [http://www.educause.edu/ir/library/pdf/DEC0602.pdf] (last viewed Sep. 30, 2007) [4] Gilbert, J. (1992). A slice by slice guide to Total Quality Management. Tudor. [5] Nichols, M. (2002). Development of a quality assurance system for elearning projects. Proceedings of ASCILITE 2002, Auckland, New Zealand, available online as PDF [http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/00 4.pdf] (last viewed Sep. 28, 2007) [6] Snae, C & Brueckner, M 2007, Ontology-Driven E-Learning System Based on Roles and Activities for Thai Learning Environment, Interdisciplinary Journal of Knowledge and Learning Objects, vol. 3, no. 1, pp. 1-17. available online as PDF [http://ijklo.org/Volume3/IJKLOv3p001017Snae.pdf] [7] Buzzetto-More, NA & Pinhey, K 2006, Guidelines and standards for the development of fully online learning objects, Interdisciplinary Journal of Knowledge and Learning Objects, vol. 2, pp. 95-104. [8] Snae, C & Brueckner, M 2006, Web Based E-Learning Framework for Thai Learning Environment, Proceedings of the International Conference on “e-Learning: Learning Theories vs Technologies?”, Ramkhamhaeng Univ., Thailand, pp. 22.1-22.11.
