Quality Evaluation of Web-based Educational Software A Systematic Mapping Aparecida M. Zem-Lopes1, 2, Laís Z. Pedro1 1
University of São Paulo Jahu Faculty of Technology (Fatec Jahu) {cidazem, laiszp}@icmc.usp.br
2
Abstract— Nowadays, with the spread of massive open online courses (MOOC) there is a significant increase in the number of students learning through the Web. Consequently, there is a growing concern about the quality of the Web-based Educational Software (WES). Despite of various mechanisms to evaluate the quality of this software, most of the quality criteria are spread in different articles found in the literature. Therefore, this work aims to conduct a Systematic Literature Mapping to identify the criteria to assess the quality of WES, and classify them according to their contribution. A total of 78 studies were analyzed and classified into four categories of quality: pedagogical, technical, organizational and social. The results contribute to support the maintenance and improvement of the quality of the WES available to the community. Keywords – web-based educational software, semantic web, quality assessment.
I. INTRODUCTION The evolution of Information and Communication Technologies (ICT) has promoted changes in how products and services are designed, developed and distributed on the market. In education, several improvements in the teachinglearning process are due to the advances in ICT, which facilitate the incorporation of educational tools to the activities developed by teachers and student. Furthermore, some universities have revolutionized higher education to encourage more students to seek knowledge elsewhere. Currently, major universities around the world, such as Cambridge, Harvard, MIT, have adopted the Massive Open Online Courses (MOOC) offering their courses on sites like Coursera, Udacity and edX. The differential of MOOCs is to provide courses through the Web, with no charge to a large number of people without having to be regularly registered in any university [1]. In this context, it is also important the development of approaches and models to assess the quality of these Webbased Educational Software (WES). Pressman [3] defines software quality as “accordance with functional and nonfunctional requirements explicitly stated and documented during software specification”. To determine the quality, it is necessary to perform adequate measurements in the software product. Nevertheless, quality evaluation it is time consuming and complex, since involving human interactions and several criteria that are spread across different articles and instruments in the literature. Thus, the main objective of this work is to gather the criteria for quality assessment of WES by carrying out a
Seiji Isotani1, Ig I. Bittencourt3 3
Federal University of Alagoas
[email protected],
[email protected] Systematic Literature Mapping (SM). It is also our goal to classify these criteria according to their contribution. II. METHOD SM is a research method that consists of methodological steps to search, interpret, synthetize and analyze the information presented in published papers related to the desired domain [5]. The purpose of this method is to provide an overview of the field giving a better control during the literature review process and avoiding possible mistakes that may cause misleading or inaccurate conclusions. In this work, we used the steps proposed by Petersen [5] to carry out the systematic mapping: (i) definition of research questions; (ii) conduct search for primary studies; (iii) studies screening by inclusion and exclusion criteria; (iv) classification process; (v) data extraction and mapping. A. Definition of Research Questions The SM carried out aimed at identifying models, methods, guidelines, frameworks, techniques, approaches developed for quality assessment of WES. Based on this goal, we defined the following research question: • RQ1. Is there a model, method, framework, guideline for assessing quality of WES? • RQ2. Within this context, is there some method or specific evaluation model for Semantic WES? B. Conduct Search for Primary Studies The search string was built by identifying the main keywords derived from the research questions: “Model”, “Quality evaluation” e “Educational software”. Each keyword forms a category that contains their respective synonyms [6]. The final search string is shown below: (Checklist OR Criteria) AND ("Quality Evaluation" OR "Quality Evaluating" OR "Quality analysis" OR "Quality validation" OR "Quality Assessment" OR "Evaluate Quality" OR "Assessing Quality" OR "Quality Rating") AND ("educational software" OR "intelligent tutoring system" OR "educational system" OR "educational technologies" OR "educational technology" OR "CSCL" OR "CAI" OR "intelligent virtual environment" OR "virtual learning" OR "educational environment" OR "educational tool" OR "computer aided instruction" OR "computer assisted learning" OR "Learning Management System").
We selected the five most important databases in Computer Science to search for papers: IEEE Xplore, ACM Digital Library, Science Direct, Scopus and Springer. We also performed some specific searches, based on the control group defined by expert in the field.
C. Studies Screening - Inclusion and Exclusion Criteria The studies screening consist of applying a set of five Inclusion (I) and six Exclusion (E) criteria to add or remove papers from our analysis. According to them, after reading the titles and abstracts of each paper in order to identify studies considered relevant to our work, 78 studies were elected and remained for classification, analysis and data extraction. Then, we read carefully each papers to extract, analyze and categorize them. The final set of studies and the analysis is presented in section III. D. Classification Process The papers were analyzed and classified into categories based on the research question. Most of the studies analyzed presented models, frameworks or methods to assess the quality of WES and other approaches have also been identified as guidelines, checklists etc. as shown at Figure 1. 20 20 18 16 14 12 10 8 6 4 2 0
Initially, by conducting the search in the selected databases 603 papers returned. From these, 67 of them were duplicated. Thus, after applying the inclusion and exclusion criteria in title and abstract, 174 papers remained to be analyzed in the next step. Finally, there were only 78 studies. A list containing the references of these 78 studies is available at http://www.icmc.usp.br/e/31355. In this list each study is referenced by an ID number. In order to differentiate these references from those cited in the reference section, after this point of the paper, we will adopt the ID number between parentheses to indicate a paper in the list. Before answering the questions of this research, it is important to give an overview of where and when the studies were published. Figure 3 shows the number of studies per year of publication, which suggests the concern of the scientific community with quality issues in educational software has increase considerably in recent years, with 13 publications in 2012.
18 16 13
3
2013
13
2012 9
2011
12
2010 4 1
3
1
11
2009 2
6
2007, 2008 3
2006
4
2005 3
2002 2
1999, 2001
Figure 1. Type of studies according to research question
E. Data Extraction and Mapping of Studies The data extracted from papers were stored and subjected to qualitative and quantitative analysis. This analysis aimed at finding evidences to answer the questions defined in Section A. In order to organize the findings, we used a spreadsheet to document the data extraction process, which allowed us to also carry out other statistical analysis. III. RESULTS AND ANALYSIS The SM was conducted over the period of four months, from October, 2013 to March 2014. The main research question of this work was answered when we analyzed deeply each of the 78 studies: 82% of them were about evaluating WES and none were specific to evaluate Semantic WES, thus responding to the second research question. The Figure 2 shows the SM steps.
Figure. 2. Overview of the systematic mapping process
1
1996, 1998,2000,2003 0
2
4
6
8
10
12
14
Figure 3. Number of publication per year
We analyzed the number of papers by publication type and database where the papers were retrieved: 56 were published in conference proceeding, 19 in Journal 3 in book chapters, highlighting the importance of conferences for dissemination of research on this topic. We also found that the studies are spread over several countries, especially in Europe. China is the country with most significant number of selected studies, 18 of them while Spain and England follows with 14 studies. In order to present in further detail the contributions of studies in the SM we made a categorization of the criteria, factors and requirements necessary to evaluate WES, which identify four important aspects in the quality assessment: pedagogical, technical, organizational and social (7). The pedagogical aspects take into account the learning process quality and are strongly related to the teaching and learning objectives for the triple Student-Teacher-Institution. Among analyzed studies, 37% used or proposed pedagogical aspects to assess the quality of a WES. For example, studies (8), (9) and (10) present factors related to usability to propose ways of assessing the quality of WES both from the point of view of the student and teachers. Another example is the approach that makes use of text mining to identify the perceptions and attitude of the students in the environment and to propose a method for quality assessment (11). The technical aspects are related to infrastructure and
functionality that support learning through WES. In this context, 34% investigated technical aspects when evaluating WES. Among them, we highlight the reusability of learning objects as one of the important criteria for quality evaluation as it increases the life cycle of the teaching material and their subsequent use in different situations and contexts (12). Another aspect frequently mentioned by several authors is the flexibility and adaptability of the software to meet both a diverse group of users and to handle different types of teaching materials and teaching-learning methodologies without losing the consistency of the interaction between the user and the software (13), (14, (15). Other criteria such as security, privacy and responsiveness are also important technical issues identified in (16) and (17). The social aspects are related to the potential that a software offers to enable the interaction among users as well as being accessible to people with special needs. It is also related to the support for creating communities of practice, and assist collaborative learning activities. We found that 11% of the selected studies had some quality evaluation criteria aimed at social behaviors or accessibility. Students with any special needs have more difficult to interact in the virtual environment than in the classroom, because this kind of the interaction is based on a variety of media and there are few of them adapted to handle these users (18), (19). To increase the inclusion of users this factor becomes an important criteria for assessing the quality of WES and we believe more researcher should be done in this direction. Finally, the organizational aspects refer to the efficiency and cost to manage the various “pieces” involving the educational environment such as actors, technology, processes and environments. The effectiveness of using the WES by users is heavily dependent on organizational aspects. It was observed that 15% of the studies investigated the evaluation of WES from this perspective. For example, (20) has proposed a framework based on a modified version of SEEQUEL used to build quality characteristics trees extracted from common experience with various e-learning environments. Furthermore, some others factors like accuracy, integrity, flexibility, testability, interoperability, ability to monitor learning, technical and aesthetic quality are also part of this group of criteria and they are present in several SM studies identified such as (21) and (22). We also have found a study that discuss a collaborative process to assess new and emerging educational technologies (21). In this study, a discussion was promoted to understand how this type of assessment may cover interdisciplinary stakeholders. It also showed the application of these technologies in real contexts of teaching and learning, integrating the disciplines and learning objects. Another approach aims to help in the selection of the reusable educational materials from web repositories and develop a learning objects reusability indicator (12). Other studies identified in the literature contribute facilitate the process of quality assessment with frameworks for assessing quality of e-learning based on current standards and best practices including: standards compliance, user satisfaction and program evaluation (19).
IV. FINAL REMARKS The advancement of ICT and widespread dissemination of MOOC leads to a significant increase in the number of students learning through the WES. As a result, there is also an increasing concern about the quality of WES content. Although there are various mechanisms to assess the quality of educational software, we note that quality criteria are spread in different studies and instruments available in the literature. In order to be able to use them properly, we need to gather and classify these evaluation criteria. Therefore, we conducted SM to identify the criteria and methods, models, frameworks, techniques to assess the quality of WES. We also classify them according to their contribution. We analyzed a total of 78 studies and performed a categorization of the criteria, factors and requirements necessary to evaluate WES which identified four important aspects in the quality assessment: pedagogical, technical, organizational and social. Our findings indicate that most of the analyzed studies are related to pedagogical (37%) and technological aspects (34%) of WES. We could observe that there is a lack of quality criteria and studies related to social aspects of WES indicating a research gap that can be explored in future work. We believe that the findings of this work is the first step to contribute to support the maintenance and improvement of the quality of courses and WES available to the community. ACKNOWLEDGEMENTS The authors wish to thank CNPq, CAPES and FAPESP for the financial support. REFERENCES [1]
[2]
[3] [4]
[5]
[6]
E. I. Allen, J. Seaman, “Grade Change: Tracking Online Education in the United States”. Babson Survey Research Group. Jan, 2014. Avaiable:. Acessed in Dec. 2014. G. H. B. Campos, F. C. A. Campos, “Quality of hypermedia applications”. Chapter published in the book Quality Software: Theory and Practice. Campinas: Makron, 2001. R. S. Pressman, “Software Engineering”, 6 ed. Porto Alegre: AMGH, 2010. G. O. B. Roque, M. V. A. Fonseca, G. H. B. Campos, “The second generation of quality assessment of distance learning courses and attributes of know-how”. B. Tec. Senac: R. Educ. Professor, Rio de Janeiro, vol. 38, n. 2, May / August. 2012. K. Petersen, R. Feldt, M. Shahid, M. Mattsson, “Systematic Mapping Studies in Software Engineering”. In: Proceedings of Evaluation & Assessment Software Engineering, pp. 1-10, 2008. B. Kitchenham, “Procedures for performing systematic reviews”. Keele, UK, Keele University, 2004. Avaialble: < http://www.inf.ufsc. br /~awangenh/kitchenham.pdf>. Acessed in Dec. 2014.
