Towards a model of quality for learning objects - Semantic Scholar

Towards a Model of Quality for Learning Objects Antonio Sarasa Cabezuelo1, Juan Manuel Dodero Beardo2 Universidad Complutense de Madrid, [email protected] 2 Universidad Carlos III de Madrid, [email protected]

1

Abstract This paper presents the scene and the elements that take part in the definition of a quality model in the learning objects arena and describes the difficulties that appear during its definition, summarizing the current open research issues.

1. Introduction The quality concept is key in the software industry due to the direct relation it has with competitiveness, reduction of costs and increase of profit. Quality issues can be addressed either from the organizational level or from the product level. From the organizational level, a major concern of quality is the creation of an appropriate organizational structure that foments the quality of the work of personnel and departments, and the improvement of the development processes [1]. This is usually approached by means of quality systems and standards. On the other side, from the product level , the focus is on the adaptation of the quality system elements to enhance the productive activities, which makes necessary the application of software evaluation and quality control techniques to the different phases of the product life cycle. Since elearning industry is not unaware of such issues, quality must be an important factor to consider for the elaboration of their products, i.e. learning objects.

2. Learning Objects as products Some Computer-Based Education (CBE) systems put emphasis in the achievement of a sophisticated personalization of the system [3], while others prioritize the maintenance of the system and the independence of technology. Within the latter tendencies, worth mentioning is the proposal of creating learning resources from minimal, re-usable information units or learning objects. These respond to

the need of reducing the development time of a CBE system. A major objective is then to transform the creation of resources to a mere assembly of simpler pieces that can be located either local or remotely, and that may be constructed for heterogeneous platforms.

3. Generation of contents as the productive process In the context of CBE systems, the productive process consists in the generation of learning contents. This process assumes two premises: first, contents are in any digital format to allow for reconstruction, if required, by any instructional designer, to be adapted to different students; and secondly, the generation of contents is not a task of a unique instructional designer, but other stakeholders from the educational process can take part, like instructors and even students, as result of their accumulated expertise and experience [3]. The basic elements of such productive activities are the following [4,5]: • Contents, which can be packaged and more properly described by standardized metadata. These include also sequencing rules to guide the flow of the instruction through the contents. • Previously produced learning objects, or any digital resource deployed for a certain learning environment and educational purpose. • Didactic methods, to have a rich model of information and associations in order to describe the contents and a better-adapted processing style during every learning process instance. The set of requirements to describe what conditions must objects satisfy from the particular needs [6] of a learning actor can be revealed by different actors (i.e., instructional designers, instructors, students, etc.) Such requirements can be varied [9] like, for instance, user profiles (descriptions of the type of student); educational objectives (formalized descriptions of the knowledge or levels of learning that students should

Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT’04) 0-7695-2181-9/04 $20.00 © 2004 IEEE

achieve in each learning stage); timing/planning (time constraints required to carry out the process); learning strategies (formal descriptions of the style used to transmit the knowledge, hence to carry out the learning); and evaluation (description of the form to assess the attainment of students). The result of this activity must be a compound object or a set of assembled objects that satisfy those requirements [7]. In order to carry out this activity, a series of tasks are due. The first step is to analyze the degree of compatibility of the established requirements, or to study the viability of the generation of a learning object that fulfills them in a high proportion. Thus, to make the decision to realize or not the development, the cost of accomplishment should be necessarily considered. If the former estimation is positive, an analysis of local repositories of learning contents is carried out to look for existing objects or contents satisfying the raised requirements. In case of not existing such re-usable contents, you should look for in connected remote systems through communication networks. If that search gives positive results, the cost of acquisition of the remote objects should be compared to those incurred in a custom development, so choosing the more beneficial [8]. Once chosen the learning object, either custom-built or re-used, a study of its quality should be realized. Several types of actors/roles can take part in these activities, as depicted in figure 1. They define the temporary and economic requirements Producer They define the educational requirements

It controls the development process Securing of the quality

Software Engineer Content Authoring

Estimator of the effort Contents Authoring System

Specialist

Planner

Contents Authoring Search/Composition of contents and learni... Sequencer

Reasoner System

Evaluater

They do the development

Translation of requirements Profiler

Expert in education

Translator System

Searching Composition system System

Figure 1. Use case diagram a) Software engineer: those with expertise in software and computer science aspects, who do not have knowledge about the domain or learning subject. Some tasks involved by this role are: planning, effort estimation, and quality assurance.

b) Domain experts: specialists and expert persons in a certain subject. This role can have some variety: expert in contents, evaluator, instructional expert, instructional designer of sequencing and profiles, and a producer or financial person c) Producers: They finance the development of contents, and influence the settling of requirements as far as run time, personnel and acquisitions costs are concerned, as well as define the overall economic cost that is ready to assume. d) An authoring software tool can help in processing the requirements, estimating costs and effort, and eventually establishing the viability of producing a learning object with the requested features. If it is viable, then the tool can help in the search, production and composition of the learning objects. Afterwards, the system will analyze the quality of the developed objects, to be considered in further developments.

3. The quality concept in learning objects In order to define a model of quality for learning objects, it is essential to define what is meant by quality of a learning object. The quality of a product or service can be defined like the “degree in which the characteristics of a product or service can cover the felt or pre-felt needs of users in a period of time” [9]. On this basis, the quality issue can be studied from two levels [10]: (1) the quality of the product, and (2) the quality of the productive process. From the product side, the potential user and needs must be firstly identified. Then, a profiling task should support the formal specification of user needs and the selection of interesting attributes from products. If there is a nonempty subset of product attributes that satisfies a significant set of user needs, we have to define methods to numerically measure how much the product satisfy those needs. Therefore, mathematical tools can offer an objective measurement of the satisfaction of the user about the product. From the point of view of quality in the productive process, methods should be provided to guarantee that the development process of the product is the most efficient and cost-beneficial. Then it is necessary to implement a protocol (involving phases, reports, documents and experts) to describe how the product must be built. In general, a quality product requires a quality productive process, and vice versa. Several initiatives are based in these ideas to study the quality of learning objects. Some are generic, but specifically adapted for educational processes initiatives, like the European Foundation for Quality Management (EFQM) Excellence Model [11], the


Quality Mark of the British Association for Open Learning, the Higher Education Funding Council for England, or the Australian Universities Quality Agency. Others are direct applications of quality assurance in the field of education, like the Essen Learning Model [12]. Another class of approaches cover only specific aspects of the educational process, like the University of Wisconsin Online Resource Centre, which offers guidelines for instructional designers to follow before using a particular learning object; the Digital Library Network for Engineering and Technology (DLNET) [2] seeks to set standards for building-up and maintaining a quality collection of learning objects. The latter is done through the screening of content for quality or pedagogical value; the Australian Flexible Learning Framework Online Product Development Review and Evaluation [13] project identified some critical factors associated with the design and development of flexible online learning resources. They identified the following factors: interoperability, organisation, scalability, use, customisation, and learning design, and stated that a big problem was the issue of granularity; the Multimedia Educational Resource for Learning and Online Teaching (MERLOT) deals with the quality issue by relying on a peer review process. The evaluation standards are there broken down into quality of content, potential effectiveness as a teaching tool, and ease of use; the University Concordia [13] presents a guide to evaluate and select learning objects; the International Research and Training Center of UNESCO for Information Technologies and Systems suggests that an important component in the quality of a collection of learning resources depends on two characteristics: compactness and completeness. In order to measure these characteristics, each resource is represented as a point in a multidimensional metadata space, and distribution of resources within this space is analyzed. The results of such analysis may be used to improve management and maintenance of the collection and thus its quality for the users [14]. Finally, some global initiatives, like the European Quality Observatory (EQO) aims to build an European-level repository of learning contents and to define a conceptual framework for quality management, assurance, and assessment that can provide recent information, search, and adaptative mechanisms. These are complex tasks that are tried to be standardized in the bosom of CEN/ISSS and ISO/IEC JTC1 SC36 groups.

4. Future work

Our investigation is centered in the quality of learning objects as a product. In that sense, our initial objective is to extend metadata standards to room the learning object quality and quality management concepts, as well as the definition of metrics that, supported by these metadata, can give an accurate, comparable measurement of the quality in existing and developed learning objects.

5. References [1] ISO/IEC 9126-1. Software engieneering- Product qualityPart :1 Quality model. ISO/IEC.Suiza, 2001. [2]Shih, T.K. “Distance education technologies: current trends and software systems” Proceedings of the First International Symposium on Cyber Worlds, 2002. [3] Wiley, D. A.. OpenCourseWare. In A. Kovalchick & K. Dawson (Eds.) Educational Technology: An Encyclopedia. Santa Barbara: ABC-CLIO, 2002. [4] Mahadevan, S.; Rahman, S.; Wiesner, P. “An online process for posting and meta-tagging learning objects and integrating contents into digital libraries “.FIE 2002. 32nd Annual Frontiers in Education, 2002. [5] SantaCruz-Valencia. L, Aedo. I, Breuer. T, Delgado Kloos C.” A Framework for Creation, Integration and Reuse of Learning Objects”. Learning Technology newsletter. Vol 5. Issue 1. [6] Kawai, H.; Takayama, F.; Anzai, T.; Manome, T.; Yoshida, H. “Objectives and features of e-learning oriented programming courseware for freshmen”. Proceedings. 23rd International Conference on Distributed Computing Systems Workshops, 2003. [7] Quarati, A. “Designing shareable and personalisable elearning paths”. Proceedings. ITCC 2003. International Conference on Information Technology: Coding and Computing [Computers and Communications], 2003. [8] Ching-Tang Hsieh; Shih, T.K.; Wen-Chih Chang; WenChieh Ko. “Feedback and analysis from assessment metadata in E-learning”. Proceedings of the 17 th International Conference on Advanced Information Networking and Applications. AINA 2003. [9] Taguchi Methods. A hands-on approach to quality engieneering. Addison Wesley,1995. [10] Stephen H.Kan. Metrics and Model in Software Engineering. Pearson Education. Boston, 2003. [11] Consortium for Excellence in Higher Education (2001) Mapping the QAA Framework and the Excellence Model. Final Project Report GMP 143/QAA, Sheffield Hallam University, July 2001. [12] Pawlowski, J.M. . The Essen Learning Model – A Multi-Level Development Model. In Staff and Education Development International 4 (2000). [13] Lowerison.G.; Gallant. G.; Boyd. G. “Learning Objects in Distance Education: Addressing issues of Quality, Learner Control and Accessibility”, Conference Proceedings CADEACED. St.Johns, 2003.


[14] Rovinskiy. D; Synytsya.K.; Podgornov.A. “ Quality assessment for collections of learning resources”. Conference Proceedings ICALT 2003
