Session F4E
Work in Progress - Open Source Usability Evaluation: The Case of Moodle Hoda Baytiyeh The American University of Beirut,
[email protected] Abstract - The open source course management system, Moodle, is designed to help instructors deliver course materials to their students from a social constructivist perspective. This study evaluates the usability of Moodle based on the perceptions of 189 professors and 1867 students at the American University of Beirut. Participants were invited to participate in an online survey reflecting the performance of the system in teaching and learning by rating a set of 30 items related to five usability attributes: Learnability, Efficiency, Memorability, Error Prevention, and Satisfaction. Besides testing the usability attributes and evaluating the course management system, the study aims to attest the open source software quality at academic institutions in the Middle East region. Index Terms – Course Management system, Free Open Source Software, Middle East, Moodle, System usability. INTRODUCTION In today’s electronic society, using computer applications is no longer a choice, but a necessity. However, with the abundance and the variety of applications, it is not easy for academic institutions to determine what applications to adopt on campus such as a primary operating system, Internet browser, or an office suite. In addition, other essential applications are expected to be implemented on campus for academic use, such as a Course Management System (CMS). In this regard, an academic institution can choose among several competing CMS, from both proprietary manufacturers and open source applications. A CMS is a software application that provides specific features to assist professors in delivering learning material to students while meeting pedagogical goals. The primary competitor is the commercial application, Blackboard, especially after its acquisition of WebCT. It is now estimated that more than 20% of institutions of higher education in the United States use Blackboard as their official CMS [1]. Among CMS products, free open source applications have been widely used because they can be obtained free of charge. These applications can be configured to run on most operating systems. Open source software are developed from contributors worldwide, driven mostly by altruistic values [3]. FOSS developers’ aspirations are to ensure that education is available to everyone, regardless of financial
ability, and to help provide the applications necessary for people to gain the life-long skills they need to succeed [3]. Moodle, the most popular free open source CMS was developed from a social constructivist perspective by Martin Dougmias in Australia [4]. This free application provides instructors with useful features such as the ability to embed resources, activities centered on a topic of study, and a variety of modes of operation. The main advantage of integrating a FOSS on campus, such as Moodle, is the discharge of license costs. In addition, system administrators have the ability to modify and customize the product. On the other hand, adopting a FOSS can be a challenging procedure because no guaranteed maintenance is offered, whereas with proprietary applications the software manufacturers provide support. Therefore, administration and maintenance costs must be considered [7]. Still, Blackboard is certainly much more expensive to maintain because it incurs a yearly licensing cost. Several studies have discussed the benefits and risks of adopting an open source application versus a commercial one. Most of these studies have compared the existing CMS, mainly to explore the advantages and disadvantages of both Moodle and Blackboard. As such, Bremer and Bryant [5] reported that 80% of students preferred Moodle over Blackboard. Machado and Tao [6] showed that Moodle was rated higher than Blackboard regarding the organization of course material and communication. The purpose of this study is not to compare Moodle with another CMS; rather, this study will inspect the usability of the system from the users’ point of view. In fact, successfully implementing CMS depends not only on providing training and support for instructors, but also on users’ satisfaction with the application. THEORETICAL FRAMEWORK Nielsen [7], a leader in the field of usability, considers usability as the primary aspect that influences the product’s acceptance. Nielsen presented operational criteria and defined usability using five attributes: • Learnability refers to how easy it is to learn to use the system. • Efficiency refers to the way the system supports users in carrying out their tasks. • Memorability refers to how easy the system is to remember how to use. • Error prevention refers to a low error rate in the system.
978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD 41st ASEE/IEEE Frontiers in Education Conference F4E-1
Session F4E •
Satisfaction refers to the users’ satisfaction upon using the system. CONTEXT OF THE STUDY
Very little is known about adopting CMS in Lebanese academic institutions and how these applications are used in practice. The purpose of this study, therefore, is to examine professors’ and students’ attitudes toward the use of the CMS program; Moodle; at the American University of Beirut (AUB). The American University of Beirut was established in 1866 by American Protestant missionaries and consists of six colleges: Agricultural and Food Sciences, Arts and Sciences, Engineering and Architecture, Health Sciences, Medicine (including the School of Nursing), and the School of Business. In the summer of 2005, the Academic Computing Center (ACC) at AUB decided to switch from the proprietary application WebCT to the FOSS Moodle. The implementation process was completed in the spring of 2007, and Moodle became the official CMS at AUB. Numbers show that 65% of all courses sections taught in spring term 2010 used Moodle. This reveals that students and professors are using this free software more and more as the main medium to manage their courses. Professors upload learning materials, grades, and quizzes, while students upload their assignments and receive feedback from professors through the same system.
questions were included to reflect users’ perceptions using Moodle in teaching and learning. On the last question of the survey, participants were asked to rate 30 likert-scaled items on a scale of 5 (1=Strongly disagree, 5=Strongly agree). These items were based on the five usability attributes based on Nielsen’s theory [7]. CONCLUSION Besides testing the usability of the CMS Moodle, the implications of this research help in spreading awareness about the importance of competitive Free Open Source Software. Further studies are needed to investigate the spread of FOSS in general and Moodle in particular in the Middle East region. REFERENCES [1]
Mazard, C. “On line learning market may be monopolized”. Retrieved April 2010, from http://www.antitrustlawblog.com/articleon-line-learning-market-may-be-monopolized.html, 2005.
[2]
Baytiyeh, H., & Pfaffman, J. “Open source software: A community of altruists”. Computers in Human Behavior, Vol.26, 2010, 1345-1354.
[3]
Baytiyeh, H., & Pfaffman, J. “Volunteers in Wikipedia: Why community matters”. Educational Technology and Society, Vol.13, No.2, 2010, 128-140.
[4]
Dougmias, M., & Taylor, P. “Moodle: Using learning communities to create an open source course management system”. Proceedings of the 15th World Conference on Educational Multimedia and Hypermedia and World Conference on Educational Telecommunications, Norfolk, VA, 2003, 171-178.
METHOD I. Participants and Procedure As of fall 2010, 548 professors and 7204 students; attending all colleges at AUB; possessed active accounts on Moodle. Using their email addresses, users were invited to participate in an online survey, while ensuring them of complete anonymity. A first call for participation was sent to the designated population in the first week of November 2010. After two weeks, 115 professors and 1078 students had completed the survey and a second invitation was sent as a reminder. Toward the end of November, 189 professors and 1867 students had completed the questionnaire. This resulted in 34.5% and 26% response rate, respectively. Data from partially completed surveys were discarded. II. Instrumentation Participants were invited to complete a 20-question survey. The survey was based on previous studies available in the literature related to website usability, using both quantitative and qualitative approaches [8, 9]. The same instrument was used for both students and professors, but the questions were adjusted for each sample. The first section included questions related to demographics and general information such as gender, major, and the number of courses using Moodle. Other
[5] Bremer, D., & Bryant, R. “A comparison of two learning management systems: Moodle vs Blackboard”. Proceedings of 18th Annual Conference of the National Advisory Committee on Computing Qualifications, Tauranga, New Zealand, 2005,135–1390. [6] Machado, M., & Tao, E. “Blackboard vs. Moodle: Comparing user experience of learning management systems”. Proceedings of the 37th ASEE/IEEE Frontiers in Education Conference, 2007, Milwaukee, WI. [7]
Nielsen, J. “Heuristic Evaluation”. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods. 1994, New York, NY: John Wiley & Sons.
[8]
Kirner, T., Custodio, C., & Kirner, C. “Usability evaluation of the Moodle system from the teachers' perspective”. Proceedings of IADIS International Conference, 2008, 371-378.
[9]
Oztekin, A., Kong, Z. J., & Uysal, O. “UseLearn: A novel ckecklist and usability evaluation method for elearning systems by critically metric analysis”. International Journal of Industrial Ergonomics, Vol. 40, 2010, 455-469.
AUTHOR INFORMATION Hoda Baytiyeh, Assistant Professor, The American University of Beirut, Department of Education,
[email protected].
978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD 41st ASEE/IEEE Frontiers in Education Conference F4E-2
