Assessing Quality in E-learning including learner with ...

Assessing Quality in E-learning including learner with Special Needs Latifa Ben Arfa Rabai and Neila Rjaibi Department of computer science, ISG, Tunis, Tunisia [email protected] ; [email protected]

Abstract. Quality in e-learning encompasses many criteria such as curricula, students, teaching methods, learning platform and teachers. Assessment is now a growing market, stimulated by the considerable development of e-learning platforms and distance learning need. A literature review has shown that there is no detailed e-learning assessment process that adapts to all teaching contexts, encompasses the needed assessment dimensions and their criteria. Our focus is on going with all users category of the considered e-learning systems including learners with special needs in the visual, hearing or cognitive disabilities. In this paper, we intend to improve the quality of e-learning educational processes, by presenting a new refined assessment model and listing its respective assessment criteria that increase student’s satisfaction. Keywords: assessment in e-learning, assessment criteria, quality in e-learning, student’s satisfaction, special needs, usability, accessibility.

1.

INTRODUCTION

Nowadays, e-learning has become a popular way of learning, it has increased exponentially in recent years. In order to support the improvement of e-learning quality, a variety of quality measures are developed by researchers and organisations. Several assessment factors need to be considered. Assessment models and frameworks vary from one context to another according to the specific output and purpose measure. They focus on technological aspects or on pedagogical view or on the training course or on increasing learner’s satisfaction. In the other point of view diverse assessment criteria and dimensions are proposed and used for evaluation. For example, twenty-four benchmarks were developed by the institute for higher education policy; they cover seven dimensions which are: institutional support, course development, teaching and learning, course structure, student support, faculty support, and evaluation and assessment [5]. Moreover, the assessment of e-learning quality as a service is presented in [5]. The elearning Service Certification Program consists of three quality aspects which are: personnel, course and system. It covers eight quality criteria as follows.

1. Learner support 2. Faculty support 3. Curriculum development 4. Instructional design 5. Instructional process 6. Organizational support 7. Technology 8. Assessment and evaluation In addition, the e-learning Courseware Certification Program presented in [5], examined the educational quality of e-learning courseware and focuses on four quality factors which are: contents, navigation, instructional design and instructional media and encompasses 15 quality criteria. Also, Barker presented quality concepts such as quality of learning materials, availability of materials and support for students [1]. We summarize quality indicators and criteria as follows: • Quality of curricula • Quality of students • Quality of infrastructure • Learning time • Teaching methods, assessment and feedback • Human resource, teachers. Despite the need and the widespread of e-learning in our modern society, a consolidated assessment methodology for e-learning systems is not available [11, 12]. Our focus is to produce a full detailed assessment model incorporating all the needed assessment components and their related sub-criteria, in addition our focus is on going with all users categories of the considered e-learning systems including learners with special needs in the visual, hearing or cognitive disabilities. This leads to bring major solutions and improvement for peoples with disabilities. Learner with special need covers: physically disabled, trouble distinguishing colors, visual, hearing, motor and language or cognitive disabilities. This type of learner with particular focus on the deaf and hard of hearing is enabling to access the e-learning content. Evaluation of the learning process should include [12]: • User-friendliness the e-learning content • The usefulness of the system with regard to learner with special needs • The accessibility of the content, • The effectiveness of the entire course. In the present study, we focus on e-learning quality assessment criteria based on the perspective of need to increase learner’s satisfaction and to produce learning situation with higher quality. To our knowledge, many studies have integrated e-learning in educational processes [2]–[4]. Among e-learning indicators we mention e-learning course quality, perceived ease of use, diversity in assessment, learner interface, system content, learner and instructor. In order to improve quality in distance learning, we need to assess learning and teaching processes and to find criteria affecting the success of a given learning situation for all users. In this work, recent assessment models in e-learning field are discussed then our elearning assessment model is proposed and refined.

The paper is organized as follows. In Section 2, we review the recent educational research models used for evaluation in e-Learning. In Section 3, a new model for assessing quality teaching processes is proposed and refined. Finally 4, we conclude this paper with directions and future work.

2.

RELATED WORK IN E-LEARNING

Sun et al. [3] provided a clear definition of e-learning concept as the use of technology to deliver information for training. This modern education is useful in creating interactions between learners and instructors, or learners and learners without considering time and space constraints. They discussed dimensions and factors that affect the user’s satisfaction in an e-learning system. The proposed assessment model covers six dimensions which are: learners, instructors, courses, technology, design and environment. As a critical insight, we mention that this model ignored the synchronous learning assessment dimension which is fundamental in assessing the quality of online courses. Also, based on the analytic hierarchy process (AHP), this model was presented by Wang and Shee to evaluate web based e-learning system (WELS) and to assess learner's satisfaction [4]. The model includes four dimensions: • The learner interface focuses on usability, friendliness, easiness and stability of the interface. • The learning community focuses on the easiness of discussion, the share and exchange of data between learners and their teacher. • The system content focuses on updating, sufficiency and usefulness. • The personalization focuses on controlling learner progress and recording learner performance. Chao and Chen [2] discussed an assessment model which includes five primary criteria and sub-criteria in order to evaluate the efficiency of an e-learning system as follows: A. The e-learning material focuses on quality and contents such as easiness, structure, contents and interaction. B. The quality of web learning platform focuses on quality of web connection and learning platform. C. The synchronous learning is about the synchronization between teacher and students with a minimum of one hour per week or one hour per month, the use of web discussion and the possibility to organize meetings when necessary. D. The learning recording concerns the record of student learning material. This will be useful to teachers to monitor learning and to students to evaluate themselves. E. The self-learning focuses on criteria that correspond to individual learning like feeling comfortable, controlling learning progress. The two last models present some insufficient aspects, they neglect learner and instructor dimensions which are essential in an e-learning process evaluation. From the perspective of learners’ satisfaction, Sevgi and Refika [9] propose the hexagonal e-learning assessment model (HELAM), it covers six assessment

dimensions as follows: (1) system quality, (2) service quality, (3) content quality, (4) learner perspective, (5) instructor attitudes, and (6) supportive issues. The dimension supportive issue focuses on promotion, trends, ethical issues and cost. The service quality dimension focuses on student tracking, course authorization, course management, knowledgeable and maintenance. The last two dimensions which are supportive issue and service quality are not interesting in our context because we concentrate on the quality of learning and teaching processes from the perspective of learner’s satisfaction, the hexagonal elearning assessment model focuses on the quality of e-learning system effectiveness. The framework for assessing e-learning readiness of Iranian universities proposed by Darab and Montazer in [8] covers the human resources which are learner and instructor, the content, the learning platform as network, equipments, and security. Some assessment dimensions are not appropriate in our context such as finance, culture, policy and standards. The Darab and Montazer model used to assess elearning readiness in Iranian universities taking into account a variety of external and internal assessment dimensions from the perspective that e-learning is a system. The two last assessment models adopted from the literature present some shortcomings. In fact, they ignore the dimension synchronous learning that cover the web discussion and the capacity of sharing data. Moreover, they ignore the dimension personalization of learning which covers control, record and content review. These two neglected assessment dimensions are fundamental in our context to assess an online course. We are reviewing related work of recent assessment models in distance learning. Every model mentioned a set of assessment components or dimensions in order to assess the learning and teaching processes. Therefore, table 1 presents a comparative study between the implementation assessment criteria of e-learning models. Assessment in e-learning is produced by a variety of detailed models. Our focus is to study, discuss and use all of the essential assessment components and criteria presented separately to adopt them in order to produce a full refined model that assesses an online learning and teaching process. We remember that we focus on studying the quality of learning and teaching processes for all users. The proposed models of the literature neglect quality assessment criteria for learners with special needs. In order to make a holistic assessment model we should integrate the three fundamental quality criteria which are usability, accessibility and flexibility/adaptability: • Usability Is considered among the most important quality factor in order to assess the e-learning system, it means that the software is able to achieve the specified goals with effectiveness, efficiency and satisfaction. And it is the way to measure how it is easy to use the product [10, 11, 12]. • Accessibility It is a primordial quality indicator for e-learning and especially for learners with special needs. It means the increasing of quality of e-Learning materials with better didactic material matching those needs [13, 14, 15]. When we talk about accessibility we refer to typical didactical contents for example a graph, a diagram, an image or a table. They form a critical didactical content for learner with cognitive disabilities.

“Accessibility is defined as meaning that people with disabilities have access” [18]. Accessibility need to be certified by a variety of procedures [15], for example to underline the need for the introduction of e-learning to deaf students. Also another definition of the term accessibility provided in [19] by Tim Berners-Lee, W3C Director and inventor of the Word Wide Web: “The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect.” A variety of standard are specific to guarantee the accessibility quality criteria, the best known is the World Wide Web Consortium’s (W3C) Web accessibility Initiative (WAI). The W3C is a set of guidelines that make the site more accessible [20]. • Flexibility & Adaptability Flexibility and adaptability are considered among the primary quality factor. The concept of flexibility in e-learning means suppleness in terms of time and location because a lot of people want to learn but restricted by working time, family and economy [15]. In other aspect the flexibility degree covers also security and authentication. E-learning needs to be flexible and adaptable especially for learners with special needs to promote freedom, independence and to individualize the learning process. It helps in varying learning style, communication formats, so the electronic text or the printed text is suitable for blind learner, vision impaired dyslexic [19]. Table 1. Comparing The Implementation Criteria of E-learning Models Dimension/ model

Chao and Chen model

Wang and Shee model

The learner

Sun et al model

Sevgi and Refika model

Darab and Montazer model

 Learner

Learner

Human

perspective

Resource

Instructor

Human

perspective

Resource

Content quality

Content

attitude toward e-learning  Learner computer Anxiety  Learner Internet selfefficacy The instructor

 Instructor attitude toward e-learning  Instructor response timeliness

 The course  System content  E-learning material

E-learning material:  Has a good structure  Has exercise

System content:  Up to date

 Good content

 Sufficient  Usefulnes s

The course :  E-Learning course flexibility  E-Learning course quality

 Design  Environmen t  Technology  The interface  E-Learning

E-Learning material:  Easiness  Containing multimedia design  Interactive mode  Assessment

learning interface:  Easiness

Design:  Ease of use

 Friendline

 Assessment

ss  Stability  easiness

E-Learning platform :  Friendly user interface

of

material  Platform

 A good web connection

ding

 Learning

Synchronous learning:  Student use the web

community  Synchronou s learning

discussion  A synchronous learning with teacher  Student meet his teacher in class

 Environment:  Interactive

System

 Network

quality

 Equipme nts  Security

mode  Technology  Internet quality

understan

Learning community:  Discussio n with learner  Discussio n with teacher  Share of data

 Personalizat ion  Learning record  Self

Learning record:  Catch learning record  Control learning progress  Self learning : student review content

Personali zation:  Recording and controllin g learning

learning

3.

THE PROPOSED ASSESSMENT MODEL IN E-LEARNING

Given the shortcomings of available theoretical models, we first study all assessment components presented in the literature and their related criteria. Therefore, we propose a detailed assessment model for distance learning. A. The Proposed Model Figure 1 presents our proposed assessment model in distance learning to assess elearner’s satisfaction including learner with special needs. It is based on recent investigations on the assessment of e-learning system in [2]–[4]. We carefully studied dimensions which are Instructor (A), Learner (B), E-learning platform (C), System contents: the course (D), Synchronous learning (E) and Self learning (F), Usability (G), Accessibility(H), Flexibility and Adaptability (I) .

Fig. 1. The proposed assessment model for distance learning B. Refinement of the Proposed Model In distance learning model, we carefully studied nine dimensions and their relative criteria. Our proposed refinement is presented in figure 2. Dimension (A) is the instructor studied in [3]–[6]. It is the most important component in our assessment model because it is the maestro of the learning situation. The instructor organizes online content, motivates student and urges them to use the web site. This dimension contains 2 criteria which are attitude toward e-learning and response timeliness. The first refers to the possibility of the teacher to encourage and motivate students to participate in virtual learning. The second refers to his real support and fast response. Dimension (B) is the learner studied in [3]–[6]. It is the engine of learning situation; we remember that we focus on student’s satisfaction. This dimension contains 2 criteria which are attitude toward e-learning and computer Anxiety. The first measures student’s motivation to use an e-learning system for a course while the second, refers to all feeling of apprehension, tension, discomfort, fear or feeling of insecurity due to the use of online courses. Dimension (C) is the e-learning platform; it is related to the technical and pedagogical aspects. This dimension contains 8 criteria studied in [2]–[4] which are described as follows. The first criteria focuses on ease of use in navigation, inscription and connexion, the second focuses on containing multimedia design like video, flash animation, sound and image, the third concentrate on providing interactive mode

between tutor and student, the fourth is interested in providing assessment like multiple choice question and other exercises or activities, the fifth is user friendliness interface which means the ability of the platform to be satisfactory and easy to learn and easy to use, the sixth criteria is the necessity to provide good web connection, the seventh is the stability of the platform without technical problem, finally the eighth and last sub-criteria is the ease of understanding, it refers to the capacity of the platform to be easy to student. Dimension (D) is the system content. It focuses on four characteristics of the course presented in the platform. It should be up to date and sufficiency [4], useful [3], [4] and finally structured [2]. Dimension (E) is the synchronous learning, it focuses on 3 criteria. The first discussed in [2]–[4] which refers to the possibility for a student to contact his teacher by the use of web discussion. The second criteria is the possibility to have a synchronous learning between teacher and other students and finally the possibility to meet in class when necessary [2]. Dimension (F) is the self learning. It includes three criteria, it focuses on the possibility to record learning, the learning progress control [2]–[4] and finally the review of the learning contents [2]. Dimension (G) is usability [10, 11]. It includes six quality sub factors as follow: • Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design? • Efficiency: Once users have learned the design, how quickly can they perform tasks? • Memorability: When users return to the design after a period of not using it, how easily can they re-establish proficiency? • Errors: How many errors do users make, how severe are these errors and how easily can they recover from the errors? • Satisfaction: How pleasant is it to use the design? • Pedagogical usability : which includes factors such as learner control, • Learner activity, motivation and feedback. [15] Dimension (H) is Accessibility [13, 14]. It covers three quality criteria: courseware, Library resources and content to be accessible to variety of users including students with disabilities. When we talk about accessibility we promises to be free, accessible to all. • Courseware [19] : is the software or the application containing instructional modules • Library resources (electronic reserves and digital collections ) [19] : covers the online databases, so images, tables, frames, forms, animations and graphics should be easy in reading and navigation for users with vision and mobility disabilities. • Content [19] Dimension (I) is Flexibility and Adaptability [15, 16]. It includes ten quality sub factors as follow: • Starting times • Location (where studies are pursued), • Study period (duration) • Study pace (full time/part time) • Security and authentication. • open course (no formal prerequisites),

• Content and tasks (students can select and specialize) • Flexibility of study method (communicated through many channels/modes) • Language(s) of instruction and content, • Adaptation f methods to disabled people To guarantee the success of E-learning process for persons with special needs, content should be designed minutely and nicely specifically for the deaf and hard of hearing or visually handicapped. Other requirements of the content, it should be designed with higher quality to guarantee more simplicity in navigation and content. We noted that accessibility is considered important, it allow the e-learning course to be more comfortably for every learner with its special need.

Fig. 2. The new refined model for assessing learner’s satisfaction in an e-Learning system

Our proposed model is useful to assess a distance learning situation including learner with special needs in the visual, hearing or cognitive disabilities. This evaluation is recommended during the development phase of e-learning content; it can be used as recommendations or quality requirements. The refined assessment model need to be validated throughout a survey to explore the relationship between learner’s satisfaction and the proposed e-learning environment criteria, it will bring the useful information to move the field forward. Empirical results show that some of the related criteria of assessment dimensions are significant and affect learner’s satisfaction. This leads us to develop the most important criteria related to special needs cases. For example, in order to improve usability dimension we need to work more on the possibility to make content more easy to learn, more efficient and more memorable with a variety of motivational activity and feedback.

Conclusion The advancement of education in many ways depends on assessment. Quality assessment frameworks and models can provide organizations, individuals and educators with useful information to move the field forward throughout the evaluation of student’s performance and the measure of student’s satisfaction.. In this work the educational processes are assessed to find the needed assessment criteria that increase student’s satisfaction and adapt to all online learning process evaluation and all category of users. Our holistic model is inspired from the literature and refined to assess learner’s satisfaction with nine dimensions and 41 criteria. This evaluation is useful during the development phase of the e-learning content; it can be used as recommendations or quality requirements. For learners with special needs further efforts should be taken in order to design more usable and interactive tools for the presentation of accessibility guidelines. Future research may include demographic characteristics, such as computer ownership and internet access, which may influence students’ attitudes toward online learning environments. The improvement of quality learning and teaching processes need more collaborative works from the perspective of pedagogic experts, didactic experts, teaching experts and web-technology specialists.

References 1.

K.C. Barker, “E-learning quality standards for consumer protection and consumer confidence: a Canadian case study in e-learning quality assurance,” Educational Technology and Society, vol. 10, pp. 109-119, 2007.

2.

R.J. Chao and Y.H. Chen,“ Evaluation of the criteria and effectiveness of distance elearning with consistent fuzzy preference relations,” Expert Systems with Applications: An International Journal Elsevier, vol. 36, pp. 10657-10662, 2009.

3.

P.C. Sun, J.T. Ray, G. Finger, Y.Y. Chen, and D. Yeh, “What drives a successful eLearning? an empirical investigation of the critical factors influencing learner satisfaction,” Computers and Education, Elsevier, vol. 50, pp. 1183-1202, 2008.

4.

Y.S. Wang and D.Y. Shee, “Multi-criteria evaluation of the web-based e-learning system: a methodology based on learner satisfaction and its applications,” Computers and Education, Elsevier, vol. 50, pp. 894-905, 2008.

5.

M.P. Chen, “An evaluation of the ELNP e-learning quality assurance program: perspectives of gap analysis and innovation diffusion,” Educational Technology & Society, vol. 12, pp. 18-33, 2009.

6.

H.M. Selim, “Critical success factors for e-learning acceptance: confirmatory factor models,” Computers and Education, Elsevier, vol. 49, pp. 396–413, 2007.

7.

I.Sahin, “Predicting student satisfaction in distance education and learning environments,” Turkish Online Journal of Distance Education-TOJDE, vol. 8, pp. 113-119, 2007.

8.

Darab and Gh.A. Montazer, “An eclectic model for assessing e-learning readiness in the Iranian universities,” Computers and Education, Elsevier, vol. 56, pp. 900–910, 2011. 9. O. Sevgi and K. Refika, “Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation,” Computers and Education, Elsevier, vol. 53, pp. 1285–1296, 2009. 10. J. Nielsen, “Usability Engineering, Morgan Kaufmann”, San Francisco, USA, 1994. 11. M. Debevc, JL Bele, “Usability testing of e-learning content as used in two learning management systems”, Assessment, 2012 - eurodl.org 12. Ardito, M. F. Costabile, M. De Marsico, R. Lanzilotti, S. Levialdi, T. Roselli, V. Rossano: “An approach to usability evaluation of e-learning applications”, Universal Access in the Information Society, Vol. 4, No. 3, 2006. 13. S. Gabrielli, V. Mirabella, S. Kimani, T. Catarci, “A Boosting Approach to eContent Development for Learners with Special Needs”, Educational Technology & Society, 9 (4), 17-26, 2006. 14. Unterfrauner, C. Weiermair-Märki, “User Requirements for Adult Learners with Special Needs in Accessible Lifelong Learning”, Paper presented at elearning Forum, Paris, France, 2008. 15. National Agency’s Department of Evaluation, Eva Åström, Report 2008:11 R, Elearning quality Aspects and criteria for evaluation of e-learning in higher education, Published by the Swedish National Agency for Higher Education, 2008. 16. M. Bandalaria, “Impact of ICTs on open and distance Impact of ICTs on open and distance learning in a developing country setting: The Philippine experience”, The International Review of Research in Open and Distance Learning, 2007.

17. Efthimiou, F. Stavroula, G. Sapountzaki, “E-accessibility to educational content for the deaf”. The European Journal of Open and Distance Learning (EURODL),2006. 18. COM (2000) 284 final: “Towards a Barrier Free Europe for People with Disabilities”, Communication from the Commission to the Council, the European Parliament, the Economic and Social Committee and the Committee of the Regions, Brussels, 2000. 19. J. Seale, “E-Learning and Disability in Higher Education: Accessibility research and practice”, Taylor & Francis, 2006. 20. http://www.w3.org