Int. J. Learning Technology, Vol. 3, No. 1, 2007
Quality mark e-learning: developing process- and product-oriented quality for learning, education and training Jan M. Pawlowski University of Duisburg-Essen Institute for Computer Science and Business Information Systems Information Systems for Production and Operations Research Universitätsstr. 9, 45141 Essen, Germany E-mail:
[email protected] Abstract: Quality marks intend to improve the quality of organisations and/or products and services by certification. Their scope, objectives and outcomes, however, vary. Therefore, it is necessary to match the objectives of a quality mark with the requirements of an educational organisation. Sometimes, the only outcome of a quality mark is the marketing effect. Quality marks, however, can start and maintain a quality development process in organisations. Therefore, the main research question is to design a quality mark that is adaptable to the requirements of organisations and takes the strengths of existing approaches into account. The Quality Mark E-Learning (Qualitätssiegel E-Learning, QSEL) is a holistic quality mark: It is derived from requirements of organisations and integrates existing approaches. It analyses the quality of an organisation and its products and services. In this article, the requirements for a holistic quality mark are derived. The paper shows how the quality development process can lead to a change process. The objective is to establish a learning, quality-aware organisation. Keywords: quality mark; quality mark e-learning; Qualitätssiegel E-Learning; QSEL; Quality Management; QM; quality standards; quality development; ISO/IEC 19796-1; DIN PAS 1032-1; learning technology. Reference to this paper should be made as follows: Pawlowski, J.M. (2007) ‘Quality mark e-learning: developing process- and product-oriented quality for learning, education and training’, Int. J. Learning Technology, Vol. 3, No. 1, pp.51–71. Biographical notes: Dr. Jan M. Pawlowski is the Director of the workgroup on e-learning and knowledge management within the Institute for Computer Science and Business Information Systems. This includes the research coordination of several German and European projects (e.g., Essen Learning Model, Virtual Education in Business Information Systems, European Quality Observatory, European Foundation for Quality Management for E-Learning, Quality Initiative E-Learning in Germany). He is the Editor of the International Handbook on Information Systems for Education and Training and the Handbook on Quality and Standards for E-Learning and has published numerous articles in books, journals and conference proceedings. His main research interests and activities are in the field of e-learning, modelling learning-related processes, procedural models, learning technology standardisation, quality management and quality assurance for education and mobile/ambient learning. He is on the Board of the German E-Learning Association D-ELAN, responsible for international cooperation. He is the Chair Copyright © 2007 Inderscience Enterprises Ltd.
51
52
J.M. Pawlowski for Education in the IEEE Technical Committee for Learning Technology. He is actively involved in research organisations (AACE, GI, etc.) and coordinates the quality workgroups within standardisation organisations (DIN, CEN, ISO/IEC JTC1 SC36).
1
Introduction
Quality marks are a popular method to assure the quality of products and services. A quality mark is an instrument of quality management or quality assurance providing a certification. This certification testifies the conformance to certain aspects and rules based on a review process for a certain domain, in this case the domain of e-learning. On the one hand, quality marks provide guidelines or rules for organisations and individuals to achieve quality. On the other hand, they are an instrument to document quality for potential customers and partners. However, they vary in their scope, applicability and usefulness. This means that quality marks do not always meet the quality requirements of organisations and their customers. Therefore it is necessary to analyse the scope of quality marks and the benefits and added value for organisations using them. Because of the different scopes and methodologies, the main research question is how to design a quality mark that covers the relevant aspects for organisations. Additionally, the quality mark should take into account existing practices. Finally, it should take into account the adoption and implementation aspects: The effort to implement a quality mark has to be reasonable. In this article, the Quality Mark E-Learning (Qualitätssiegel E-Learning, QSEL) will be described as an example of a holistic quality mark. It covers the variety of requirements and scopes of quality management. In contrast to existing quality marks, this approach should be adaptable to the requirements of educational organisations integrating different methodologies and the strengths of existing approaches. As a first step, existing quality marks for learning, education and training are analysed. The result of the analysis is a comparison of the scope and methodology of quality approaches. Secondly, the main requirements for a harmonised, adaptable approach are identified. Based on this review, the main principles for a harmonised quality mark are defined. QSEL follows a descriptive, adaptable approach increasing transparency of processes, products and services in the field of learning, education and training. Even though QSEL contains specific components for e-learning, it can be applied to other scenarios of education and training, such as face-to-face settings. The main concepts and categories of the quality mark are shown in detail. A first evaluation has been done using a case study approach. The paper concludes with an outlook on future activities, specifically in the field of quality standards.
Quality mark e-learning
2
53
Quality for learning, education and training
2.1 Quality approaches In this section, a brief review of existing quality approaches will be given. As already mentioned, quality approaches vary in their scope and methodology. How can those aspects be combined into a single quality mark? How can different methodologies be integrated to fit the needs of practice? Those are the main research questions of this section. Therefore, this section will show the variety of choices for organisations. Approaches will be classified and analysed based on their scope and methodology. The main result is the identification of requirements for quality marks derived from the analysis. The number of quality approaches that have been developed for all educational fields, along with the large number of methods (such as criteria lists, guidelines and management approaches), has grown to an unmanageable amount (a comprehensive overview is provided by Ehlers et al. (2005)). It is not just the number, however, that makes it difficult to select a suitable approach, but the often divergent requirements and recommendations of the approaches as well. Within the European context, there is now a vast multitude of diverse regional and differentiated sectoral approaches. The spectrum of competing approaches to determining and improving quality in the educational field, as well as in e-learning, ranges from internal evaluations to external certifications or accreditations. Generally, quality approaches can be distinguished according to their main subject: process-orientation or product-/service-orientation. Dippe et al. (2001) add the view of competency-orientation. However, this classification just gives a rough classification, which cannot be used to identify the usefulness of an approach for a certain context. In the following, the main methods are presented and analysed towards their usefulness within a quality mark. Quality Management (QM) generally aims at improving the quality of all aspects of an organisation and its processes (cf. Deming, 1982). QM approaches are directed at management issues on the strategic, tactical and operational level. This includes, for example, business strategies, processes and results. On the process level, the main aspects of QM, such as creation, implementation or realisation processes are analysed. Furthermore, QM approaches focus on the customer and, specifically, learner requirements for planning, implementing and providing a product (e.g., an educational product such as an e-learning course). Owing to their generic scope, these approaches can be applied in different branches and contexts. For e-learning, there are currently no generally recognised quality management approaches. Some generic approaches, such as EFQM (2003) or ISO 9000:2000 (ISO, 2002) are applied to the field of learning education, and training (cf. Cruickshank, 2003). Specifically, generic concepts need to be extended regarding educational processes. As an example, the generic, process-oriented concepts of HEFCE (2003) have been specifically adapted for the educational field or e-learning (cf. EFMD, 2004a–b). As a conclusion, QM approaches can be helpful to improve educational organisations internally (cf. Cruickshank, 2003). Most of the approaches, however, are not comparable and therefore not transparent externally. This leads to the conclusion that a new quality mark should harmonise existing approaches to provide a common base for quality
54
J.M. Pawlowski
management in educational organisations. One solution is the use of a reference process framework specifically for learning, education and training. This reference framework is provided in the ISO/IEC 19796-1 Standard (ISO/IEC, 2005). Secondly, Quality Assurance (QA) aims at the improvement of specific, mostly operational aspects, in most cases, products and services. QA systems based on criteria lists and checklists are widely used (cf. Kefalas et al., 2003; Stella and Gnanam, 2004). Those tools are relevant and popular because they seem to make it relatively easy to evaluate learning effectiveness, although this is usually only possible with time-consuming empirical methods. As a result, these tools enable people to assess the quality of a learning arrangement or learning software without prior empirical studies. Quality criteria are defined as characteristic attributes of a learning programme whose learning effectiveness has been proven in a validity study. Many quality criteria and checklists, however, contain criteria that have not been validated or are limited to a certain aspect (Tergan and Schenkel, 2003). It is solely assumed that they are effective for learning. Many quality criteria lists mainly contain criteria from the areas of usability and design. Pedagogical and didactic criteria, however, are often underrepresented. Metastudies on the learning effectiveness of multimedia learning environments, though, show that in particular, the didactic concepts implemented in e-learning-modules and learning arrangements have a greater effect on the learning process than the so-called ‘delivery technology’ that is used (cf. Kulik and Kulik, 1991; Kulik, 1994). As a conclusion, criteria and checklists can be helpful as a guideline for the design and development of learning, education and training because of their ease of use and the corresponding acceptance. Quality criteria, however, need to be adapted to the context of an assessment or an evaluation. For this adaptation process, ISO/IEC (2005) suggest to use reference criteria. Those criteria represent a comprehensive list of quality criteria and can be adapted to the requirements of an organisation. These criteria are then transparent and comparable. In addition to the previously described quality approaches, there are other methods for managing, assuring and assessing quality: •
Benchmarking systems (cf. Bacsich, 2005), such as ‘Quality on the Line’ (Institute for Higher Education Policy, 2000) to compare different offers – for example, from e-learning providers – on the basis of specified criteria. Those approaches are helpful to determine the position of an organisation and/or its products. However, it does not provide methods to improve the quality. Therefore, it should be used as a tool to analyse the current state.
•
Accreditation and certification approaches are mostly regulatory approaches (cf. Harman, 1998). Within the accreditation process, organisations usually undergo a self-assessment to provide the current state of development. In a second phase, external experts evaluate certain aspects within an audit process. Accreditation can be seen as a quality mark in a wider sense. Most accreditation processes, however, are rather expensive and time-consuming. Therefore, they are usually just done by larger organisations or in cases of legal or branch regulations.
Quality mark e-learning
55
From the brief survey of quality approaches, it is clear that the approaches differ in scope, methodology and context of use. On the one hand, the approaches are not easily comparable. A holistic quality approach should be generic enough to cover all educational contexts. However, it needs to include specific aspects and requirements of the field, in this case, e-learning. Secondly, it should incorporate the requirements of the users, which will be presented in the next section.
2.2 Quality marks Quality marks are certifications for a certain level of quality based on an assessment. The main aspect is to transparently show a certain level of quality. Internally, to improve the operations of staff, and externally for marketing and trust-building purposes. They certify quality on different levels: There are product- and process-oriented quality marks. Like the above-mentioned QM and QA approaches, they differ in scope, context and usefulness. Therefore, decision-makers have to choose amongst those quality marks to find an adequate solution for their organisation. To illustrate this decision problem, I will present selected quality marks and their different focuses: •
The EFMD (2004a–b) Certification for E-Learning (CEL) (Euler et al., 2004) is a quality mark focusing on managing and ensuring quality for management programmes in higher education. It provides criteria, recommendations and indicators for processes, as well as products of e-learning. Even though the criteria are based on validated evidence, it is limited to the domain of business schools (MBAs) and programmes using more than 20% of e-learning.
•
The Gütesiegel E-Learning (Bruder et al., 2004) focuses on products such as courses and modules. It provides a comprehensive list of criteria for assessing quality. This quality mark takes into account a variety of studies, e.g., on usability. It focuses on, and is therefore limited to, the product level.
•
The Quality Marks of ArtSet LQW (Zech, 2003), eQcheck (Chang Barker and Leginsky, 2003) and also the quality mark of the British Learning Association (2005) provide a framework for institutions mainly based on self-assessment and one external audit. They cover a variety of components, from strategies to products, aiming at training institutions. They are mainly targeted towards training organisations, ArtSet has some extensions for schools. They do not, however, provide sufficient specific e-learning aspects. Additionally, they have not yet reached a wide acceptance in the community (cf. Ehlers et al., 2005).
•
The criteria of WebKollegNRW (2003) define criteria that have to be fulfilled to submit (blended learning) materials to a repository/educational portal. These criteria are minimum requirements for courses and modules. In the view of a comprehensive quality mark, it is not sufficient just to provide minimum (low-level) quality.
•
Several quality standards (DIN, 2004; ISO/IEC, 2005; CEN/ISSS, 2003) provide frameworks for quality, which have to be adapted, e.g., concerning quality objectives, methods or criteria. They can serve as a guideline to develop individual approaches but do not contain recommendations or procedures for the e-learning domain.
56
J.M. Pawlowski
The brief survey of quality approaches and quality marks shows that the variety leads to complex decision process. The solution is to provide a comprehensive quality mark, which covers the aspects of other approaches and can be adapted to the needs and requirements of an organisation. Especially, there should be a focus on e-learning while keeping the generic structure of existing and successfully implemented quality marks.
2.3 User requirements The main question of this section is to analyse the perspective of the user regarding quality marks. It is based on our survey on quality with European stakeholders (cf. Ehlers et al., 2005). The online survey (N=650) was done to identify quality approaches being used or considered for use and to determine user requirements. First of all, it was shown that various quality strategies and concepts are used in European e-Learning. Table 1
Quality approaches in educational organisations
Quality concept ISO 9000 EFQM Excellence Model SCORM TQM DIN PAS 1032-1 AFNOR, Code of Practice AICC BAOL Quality Mark Learning Object Metadata EQO-Analysis Model IMS-Learning Design
Approach used or considered for use (N=650) 127 51 38 25 23 19 19 12 12 11 9
Source: Ehlers et al. (2005)
This result also shows the huge variety of choices and alternatives for organisations that consider using a quality approach or quality mark. Even learning technology standards are considered as quality approaches even though they only focus on assuring quality in regard to interoperability. This variety of approaches illustrates the need to have an instrument specifically designed for e-learning, which can be adapted to the needs of an organisation. Such an instrument should cover a wide range of quality aspects and the possibility for organisations to use the aspects in which they want to develop quality. As an example, one organisation might only want to improve its course design, while a different organisation might want to include e-learning quality on a strategical level. Therefore, when choosing a quality mark, an organisation should clearly state what their quality requirements are. As a next step, they should choose a quality mark meeting those needs (cf. Pawlowski, 2003). Secondly, current weaknesses of quality approaches should be improved. First of all, our study identified a lack of quality awareness: Although organisations have identified the need for quality, they have not yet overcome the barrier to implement an approach. Furthermore, it was identified that guidance for learner orientation, marketing effects, cost reduction and process optimisation are important requirements.
Quality mark e-learning
3
57
Developing a quality mark for e-learning
3.1 Requirements for a quality mark As a first step, requirements for a comprehensive quality mark are defined. Those requirements are (1) based on the above-shown analysis of existing approaches and (2) based on the needs of quality in practice, which have been identified in a European survey (cf. Ehlers et al., 2005). The principles for a comprehensive quality mark are: •
Holistic approach The quality mark should include concepts to ensure quality for processes (e.g., design, development, evaluation), products (e.g., modules, platforms), and services (e.g., support, administration). It shall cover all levels of quality development, from quality awareness and strategies to quality of learning materials. Products and services should not be analysed separately without taking production, realisation and the organisational context into consideration.
•
Transparency The quality mark shall provide means to inform all stakeholders about the quality of processes, products and services. It will not prescribe concepts, but describe the way organisations act to achieve high-quality solutions.
•
Learning orientation The quality mark should contribute to a better learning experience. Many organisations request guidance in this field.
•
Adaptability The quality mark shall provide methods to adapt the concept to the needs of an organisation. As an example, a small content provider might just need very specific guidelines for content creation, whereas a university needs complex quality systems to improve its operations.
•
Extensibility The quality mark should be extensible taking into account new concepts, technologies, or innovations. Applying this principle means that not only the current state of the art is implemented but also the innovations.
•
Quality development The quality mark should lead to a continuous quality development in organisations. It shall not focus on maintaining the current state but aims at continuously improving quality.
•
Participation Quality is seen as a negotiation process among all stakeholders. It shall ensure participation on all levels to improve acceptance within the organisation.
58 •
J.M. Pawlowski Harmonisation The quality mark does not solely aim at developing new solutions. It should also take existing practices and quality marks into account.
•
Standardisation The quality mark should be based on recent standards, such as ISO/IEC 19796-1 (ISO/IEC, 2005), to ensure that there is a common base to provide a comparable, transparent solution.
Following these principles, a quality mark needs to be generic enough to serve a variety of scenarios, but is flexible enough to fulfil the individual requirements of organisations.
3.2 The Quality Mark E-Learning (QSEL) The QSEL combines both process- and product-orientation. The main question is how to combine different quality marks. On the one hand, common categories were chosen where previous research indicates that those aspects influence quality of educational organisations. Secondly, it takes existing quality marks into account and combines aspects that contribute to the quality of an educational organisation. The previous section has shown the different scopes and methodologies, as well as the variety of requirements of organisations. In the following, I will show how those aspects can be included in a quality mark leading to a holistic approach. QSEL follows a common distinction within the field of quality management and is therefore divided into two main categories: (1) Quality of educational organisations and (2) Quality of components. The Quality of educational organisations contains quality development concepts for an organisation, including its vision, strategy, processes and results. One of the main focuses is the analysis of processes related to the conception, development, use and optimisation of learning environments. To reach consistency with other process-oriented quality marks, this part takes several important approaches into account: the EFQM model and its application in the quality mark of the British Learning Association (2005), ArtSet LQW2 (Zech, 2003), and CEL (EFMD, 2004a–b). Accordingly, the following categories are common to the above-mentioned approaches and serve as the structure of the quality mark: •
policy and strategy
•
management
•
resources
•
processes
•
learner orientation
•
staff management
•
innovation
•
results.
Quality mark e-learning
59
As shown in Section 2.3, however, it has become clear that organisations have specific requirements concerning their quality development: First of all, organisations have different quality objectives (e.g., improvement of the whole organisation, improvement of specific areas or external documentation of a quality level). The quality mark needs to be flexible enough to take these into account. Additionally, our survey (Ehlers et al., 2005) has shown that specifically, small organisations can only spend a limited effort on the process of quality management. Therefore, a second distinction is introduced: •
QSEL Basic specifies the minimum quality requirements of an organisation. Those minimum requirements include the quality of processes, learner orientation and results.
•
QSEL Excellence covers all aspects of the quality of an organisation.
Furthermore, a variety of instruments can lead to quality improvements in all categories. As an example, a variety of process modelling languages can be used to document processes. Therefore, QSEL does not prescribe the instruments but gives guidance as to which instruments might be used. By this method, organisations receive guidance but can still flexibly develop their own procedures based on their needs and contexts. In the following, the categories of QSEL and the quality development process is described in detail.
3.2.1 QSEL Basic First of all, QSEL Basic is described. The main aspects are processes, results and learner orientation. Those aspects assure that the general operations are included in the quality management process. Processes The influence of managing processes on the overall quality of an organisation is well documented (cf. Zairi and Sinclair, 1995). It is obvious that the process of designing and developing learning environments influences quality. In the field of learning, education and training processes are of much greater importance than the final product: in most cases, the final product cannot be tested in a real-life test because the effects on the learner depends on individuals (Ehlers, 2004). Therefore, the success of a learning environment cannot be predicted (e.g., learning performance, learner satisfaction). Managing processes can, however, improve learning environments (Marshall and Mitchell, 2002; v. Brocke, 2005). The main objective of this category is to assure that an organisation manages the quality of its processes. It should be aware of what quality means for each process. This means that all staff members are aware of their roles in the quality management process. This is done by preparing precise descriptions of all processes of the e-learning life cycle in the organisation. From the identification of educational demand, the conceptual design and rollout to the final optimisation, all processes should be transparently described. For e-learning, specific processes, which differ from traditional classroom training, should be taken into account. A useful instrument to take specific e-learning processes into account is the process model of ISO/IEC (2005).
60
J.M. Pawlowski
It is obvious that a mere description does not mean that the processes are efficient and effective. Moreover, employees should be aware of the quality goals and their responsibilities. This can be reached by, for example, involving staff members in the process design. Learner orientation Learners are often not involved in the design of learning environments or educational programmes. This often results in misunderstanding or neglecting the learners’ expectations. Ehlers (2004) shows the influence of the learners’ participation on the success of e-learning. The participation of the learner plays an important role in all phases of the life cycle. In order to meet the learners’ expectations, they need to be involved in the process. The main objective of this category is to draw focus on learners’ needs and participation in educational processes. Therefore, an organisation should involve learners in the early phases of the e-learning life cycle (e.g., requirements analysis), as well as during the learning process, to assure a continuous involvement. Secondly, support services are important for learners’ success (Dirr, 1999). Learners need to be informed on the kinds of services provided (Guest, 2004) (e.g., tutoring, course information). In the field of e-learning, this gains an even higher importance than in traditional classroom information (CEN/ISSS, 2006): Because of the variety of choices, students have to be well informed about the learning environments to assure that they meet students’ expectations, prerequisites and preferences. This also addresses the aspect of consumer protection for learners (Chang Barker, 2005). Results Even though processes play an important role for the quality of e-learning, the final results have to be taken into account as well (Khan, 2004). Secondly, the overall results of an organisation also need to be analysed to determine an organisation’s performance (cf. Breitner and Hoppe, 2005). This measure is usually checked in business-to-business relationships to determine the reliability of a partner. The main objective of this category is to create awareness for the need of continuous development of the organisation considering both financial and educational outcomes. Defined and measurable outcomes should directly lead to follow-up actions and improvement procedures. The three above-mentioned aspects assure the basic mechanisms of process-oriented quality management in organisations. The aspects are consistent with the main quality management approaches mentioned above. Table 2 summarises the components and shows examples for instruments, which can be used to fulfil the requirements.
61
Quality mark e-learning Table 2
Sample elements of QSEL Basic QSEL Basic
Component
Requirement
Instruments
The organisation should have detailed descriptions of following processes, including quality objectives, actors involved, methods to assure quality, and indicators to measure the success of a process:
Process description using ISO/IEC 19796-1 (ISO/IEC, 2005)
Processes Processes
•
Needs analysis
•
Framework analysis
•
Conception/Design
•
Development/Production
•
Implementation
•
Learning Process
•
Evaluation/Optimisation
Learner orientation Service
A concept for customer service, consulting and management activities exists
Service concept (cf. Fritsch, 2004)
Learners’ participation
Learners actively take part in the design of educational processes (e.g., objectives, methods, materials, competencies and the definition of learning success)
Concept for learners’ participation (cf. Ehlers, 2004)
1 The products and services offered are consistent with the strategy and the mission statement of the organisation.
Proof of balance between product portfolio and strategy/mission statement.
2 The products and services fulfil the requirements and expectation of the learners.
Product/Service evaluation using QSEL components.
The monetary outcomes are positive and indicate a positive development for the future.
Proof of outcome according to controlling instruments/indicators (cf. Breitner and Hoppe, 2005)
Results Performance
Financial results
3.2.2 QSEL excellence The goal of the aspects within QSEL excellence is to establish a complete quality system within an organisation. This means that all areas should be involved leading to an organisation that is fully aware of quality and has adequate procedures in place. The following items are part of QSEL excellence:
62
J.M. Pawlowski
Policy and strategy The success of e-learning programmes significantly depends on its embedding in the organisation’s strategy (cf. Zellweger, 2003). The support of the top-level management and the commitment to e-learning can improve the awareness concerning quality in an organisation (cf. Bates, 2000). This quality awareness is a main success factor of quality management (Ehlers et al., 2005). Therefore, it is the main objective of this category to assure the awareness and strategic embedding of education and training within an organisation’s strategic planning. Without the support on the strategic level, operational issues will not be successful. Management The main objective of this category is to ensure successful management. Organisational quality mainly depends on successful management (Juran, 1989; Samson and Terziovski, 1999). The diverse organisational concepts should be consistent, related to each other, and should be continuously updated. Therefore, all management processes should be clearly defined and measured. Staff management Generally, the competencies of staff influence the performance of an organisation, and specific competencies for the field of e-learning are needed (Souleles, 2005). In the field of learning, education and training, it is even more important than in the service sector in general that teachers, trainers and tutors themselves are continuously trained. The staff in educational organisations need to be up to date concerning, e.g., contents, didactics and technologies. Only educated employees can provide meaningful learning experiences. Therefore, a systematic staff management is essential (cf. Gunnarsdóttir et al., 2004). The main objective of this category is to reveal the skills and potentials of the working staff and to ensure their participation. To reach organisational success, professionally trained staffs are needed. Therefore, guidelines for standardised competence classification, measurement of staff contentment and individual professional training are examined. Additionally, access should be given to e-learning programmes, e.g., through the access to Learning Object Repositories (cf. Neven and Duval, 2002). Innovation One aspect of critique concerning quality management is that in many cases, only the current state is sustained (Benner and Tushman, 2003). In contrast, organisations need to focus on innovations and their sustainability (Seufert and Euler, 2004) to strengthen their market position. The main objective of this category is to define processes for innovation monitoring and meaningful integration of organisational, technological and didactical advances. Secondly, a consistent concept for cooperation, marketing and adequate representation of the organisation should exist to promote their business externally and internally. Those categories give advice on which areas should be included to develop a complete quality system within an organisation. The following table summarises the requirements and shows examples of instruments that can be used on the operational level.
63
Quality mark e-learning Table 3
Sample elements of QSEL excellence QSEL excellence
Component
Requirement
Sample instruments
Strategy
Learning is anchored within the organisation’s culture
Participatory strategy development (e.g., Redtray, 2005), Balanced Scorecard (Kaplan and Norton, 1996)
Definition of successful learning and e-learning
A consensual definition of ‘Successful Learning and E-Learning’ exists, which was negotiated with representatives of all stakeholder groups.
Document/Documentation about ‘Successful Learning and E-Learning’ (e.g., Zech, 2003)
Leadership
A concept for declaration of objectives and for monitoring of results exists
Leadership and management concept
Quality
A concept for quality management exists, which integrates all stakeholders and contains concerns of politics, planning, control, protection and optimisation
Quality management concept
Finances
Proof of an existing financial programme, which ensures financial sustainability for running educational offerings
Financial programme and control instruments for budget monitoring; Controlling instruments (cf. Ash and Bacsich, 2002)
Staff
A staff list and a staff development plan exist
Staff list, staff development programme
Professional training
Staff members are counselled and informed about professional training offerings and continuously take part in professional training activities
Staff consulting concept and list of relevant training actions
Staff skills
Textual, methodical and social staff skills are documented, continuously updated and enhance the competence field of the organisation. Proof of analysed skill overviews, which form a starting point for human resources development
Consistent companywide documentation and taxonomy for staff skills (e.g., Schmiedinger et al., 2005)
Innovation
Development of organisational, technological and didactical aspects is monitored. A development concept exists for implementation of organisational, technological and didactical innovation.
Concepts for monitoring and implementation of innovation
Transparency
A concept exists to promote the organisation’s activities externally and internally
Marketing and public relation concept
Policy and strategy
Management
Resources
Staff management
Innovation
64
J.M. Pawlowski
The presented categories help to ensure quality in all parts of an organisation covering a variety of aspects. It is also possible to only certify selected areas (such as processes or components). The main goal of the quality mark, however, is to start and maintain an awareness-building process within an educational organisation. This process can be started using a quality mark. Furthermore, the process is a continuous and ongoing process, which can even change the culture of organisations towards a learning organisation. As a result, quality should not be seen as additional work, but as an integral part of everybody’s daily operations.
3.3 Quality of components In practice, organisations also require a quality mark for products and services, such as programmes or courses. This is mainly used for marketing purposes and to establish trust between providers and customers (Ehlers et al., 2005). We recommend, however, not to solely rely on criteria catalogues and product certifications (see Section 2); product-oriented quality assurance should always be related to a quality management system within an organisation (Khan, 2004). This part can also be used to evaluate the category ‘Results’ from QSEL Basic. This section of the QSEL quality mark analyses the quality of components of services and products of an educational organisation (such as e-learning courses or learning management systems). QSEL is based on the criteria of different approaches, such as WebKollegNRW (2003) and Gütesiegel E-Learning (Bruder et al., 2004), as well as the reference criteria of DIN PAS 1032-1/ISO/IEC 19796-1 (DIN, 2004; ISO/IEC, 2005). QSEL follows the approach of (ISO/IEC, 2005) to improve transparency. In this approach, reference criteria are provided as a guideline on how products and services in the field of e-learning can be evaluated. There is no fixed evaluation scheme, but criteria are selected from a reference list based on the purpose of the evaluation. QSEL therefore performs an evaluation based on the requirements of an organisation. This means that the evaluation purposes are defined first. In a second step, the products/services are analysed based on the criteria. The result is not a simple quality mark but a transparent product analysis based on the ISO/IEC reference criteria. The criteria cover products, as well as services of e-learning (e.g., modules, platforms, support services), and include the following items based on the classification of ISO/IEC (2005): •
learning objectives
•
textual design
•
didactical design
•
roles and activities
•
organisational design
•
technical design
•
use of media
•
communication alternatives and modes
•
testing and examination
•
maintenance and care.
65
Quality mark e-learning
To provide elaborated quality assurance for the above-mentioned items, the component criteria are divided into three sections: 1
Common criteria are criteria that are applicable to all products and services.
2
Learning Management Systems (LMS) criteria applicable to platforms and LMS.
3
Course/Module criteria are applied to e-learning programmes and modules.
The following table shows the main component-related units of the quality mark. In total, 86 items are examined. The following table contains only examples of components that are checked in the certification process by the audit team. Table 4
Sample criteria for components
Category
Components
Common criteria
Didactical design Use of media Tracking Communication alternatives and modes Maintenance and care
Learning Management Systems (LMS)
General Function Modularisation Information
Courses
Learning objectives Textual design Didactical design Roles and activities Organisational design Technical design Communication alternatives and modes Testing and examination Information Billing Certification
The component-related categories presented above ease the process of component certification by providing common, as well as specific criteria. Through the persistent use of an elaborated vocabulary, the component certification process becomes transparent. Together with the organisational components, the described criteria constitute a multifaceted view on both the organisations’ operative and strategic e-learning alignment and contribute to assist the organisation’s continuous learning development.
66
4
J.M. Pawlowski
Evaluation
Quality mark e-learning has been evaluated in different contexts in two case studies. This methodology for evaluation was selected because the quality mark highly depends on the context, i.e., it has to be adapted to the specific requirements of an organisation. Therefore, a case study is the adequate method for a context-aware evaluation. Additionally, this method was used to test the quality mark in a real situation because of the difficulty to gain insights into the process in laboratory situations. The main goal was to observe characteristics and collect qualitative data about the implementation and adoption processes and their impact on the organisation and their staff. First of all, we chose the context of a large industry enterprise. In this organisation, the quality mark was tested within the training department. Members of the training department were already familiar with quality management concepts; however, there was no quality approach implemented before QSEL. Their main task is to evaluate and purchase courses and develop and maintain their own Learning Management System. Therefore, all parts of the quality mark have been tested. Five members of the organisation were questioned during and after the process using semi-structured interviews and a group discussion. The main findings of the evaluation are summarised as follows: •
QSEL is a starting point for organisational change. The aspect of strategy/policy required most efforts because a variety of actors (including the top management) had to be involved and had to agree on a change of strategies. This led, however, to changes on the management and process levels. As a conclusion, this was a first step towards a quality culture and increased quality awareness in all organisation levels involved. Long-term changes might be achieved, but cannot yet be validated.
•
Previous work on quality management (such as process documentations, benchmarking studies) can be integrated and used within QSEL. This is specifically useful to avoid redundant efforts.
•
The efforts to fulfil the requirements of QSEL are reasonable compared to expected benefits. This is the case because the organisations involved have already been involved previously in quality management activities. The most time-consuming activities have been the work on strategy changes and the process specifications.
•
The structure of QSEL (such as allowing different instruments and methods) leads to various solutions to improve quality. This means, however, that employees need to be aware of potential solutions.
•
QSEL can be directly linked to core business processes: Existing process descriptions have been extended by specifying quality objectives and quality measures. This enables a direct connection of quality and business tasks.
•
The implementation process has been accompanied by several workshops in which alternative solutions were discussed with the evaluation team. Those workshops led to more organisational changes than the audit itself. Therefore, instruments to promote participation and discourse should be continuously held to improve the acceptance of the stakeholders.
Quality mark e-learning
67
In a second case study, we evaluated QSEL in the context of higher education in a department of computer science. In this case, the department develops e-learning courses and an LMS, which are widely used at the university. This organisation was new to the field of quality management: Neither the whole institution nor the department has been involved in quality management processes before this case study. In this case, the implementation has been evaluated through three expert interviews once before, once during, and once after the implementation procedure. The main results are summarised as follows: •
Generally, the quality management process has led to very contrasting reactions and impacts. On the one hand, it was seen as a productive tool to improve course development and software development. However, there were many barriers to overcome: Specifically, some members of the organisation were convinced that quality management cannot be introduced in higher education institutions. The reactions were similar to studies summarised by Cruickshank (2003).
•
In this case, huge organisational changes were needed (e.g., in the areas of strategy development, innovation or staff participation). Those changes were initiated but not fulfilled within the short time period of the case study.
•
The organisation was not involved in quality development before. They need intense support by experts. The workshops clarified open questions and led to organisational changes. However, even though employees saw potential benefits, their attitude remained sceptical. This means that an adequate accompanying implementation and adoption process should be available to facilitate the awareness-building process.
•
Higher education institutions need a different terminology than that of companies. In the context of higher education, especially the categories of strategy and staff development might be misunderstood because the terminology and procedures differ significantly from the corporate sector. Therefore, guidelines are necessary to translate the concepts into the terminology of higher education.
•
The quality mark was seen as an adequate instrument to prepare programmes for their accreditation. Most of the documents can be used directly in the accreditation process. However, it was discussed to promote QSEL in accreditation agencies in order to ease the transfer process.
The results of the case studies have been used to improve the quality mark and to get a first glance on the results of the quality mark. The initial results are very promising. They show that QSEL can lead to organisational changes and improvements, if the organisation is willing to adopt new concepts and change towards a learning, quality-oriented organisation. However, if an organisation is new to the field of quality development, a step-by-step implementation and adoption process is necessary. In this process, a variety of actors has to be involved and committed to the project. Additionally, individual support and moderated discourse about quality issues seem to have a very positive impact on the adoption process. Therefore, the certification process should not only include audit, but also provide support for the participating organisations and individuals.
68
5
J.M. Pawlowski
Summary and future developments
In this article, several problems in the field of quality management for learning, education and training have been addressed. First of all, there is a variety of quality approaches, both generic and specific, for e-learning. In a brief survey, the differences of quality approaches regarding scope, methodology and outcomes were discussed. This variety of approaches leads to a decision problem in organisations: Which quality approach should be used within the organisation’s context? To solve this problem, it was suggested to develop a harmonised approach. This approach should take existing quality marks into account and should be comparable and transferrable to other quality approaches (e.g., accreditation). Secondly, from the analysis of existing approaches and from a survey in European organisations, requirements for a quality mark were derived. Those requirements were formulated as principles for the design of the QSEL. The task of this research was to design a holistic, harmonised approach, the QSEL. QSEL is directed to improve the quality of an educational organisation, as well as its products and services (such as courses or learning management systems). Additionally, QSEL covers both generic aspects that are applicable to all educational organisations, as well as specific aspects of e-learning. QSEL was evaluated in two different contexts: in the training department of a large industry organisation, and within a higher education department. In the evaluation, it was shown that QSEL can be used as a starting point to establish a complete quality management system. In the evaluation process, organisational changes (such as strategy changes and process modifications) were initiated. Specifically in higher education, however, employees were still very sceptical to utilise quality management systems. For both contexts, supporting mechanisms (such as moderated discussion in the quality discourse) positively influenced the implementation process. The result of the evaluation was very promising. However, it raised many research questions for further analysis: First of all, research should focus on supporting mechanisms. Within our case studies, the efforts to reach consensus were very high. Therefore, support structures, including workshops and user trainings, should be analysed to find a reasonable balance of support and self-responsibility. Secondly, the harmonisation process of quality approaches within the field of standardisation should be promoted. The quality mark will be submitted to standardisation organisations (CEN/ISSS, ISO/IEC JTC1 SC36) in order to further harmonise quality marks and to ease the orientation for e-learning providers and customers.
References Ash, C. and Bacsich, P. (2002) ‘The costs of networked learning’, in C. Steeples and C. Jones (Eds.) Networked Learning: Perspectives and Issues, London: Springer. Bacsich, P. (2005) Theory of Benchmarking for e-Learning: A Top-Level Literature Review, March, http://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdf (retrieved 1 March 2006). Bates, A.W. (2000) Managing Technological Change: Strategies for College and University Leaders, San Francisco: Jossey-Bass. Benner, M.J. and Tushman, M.L. (2003) ‘Exploitation, exploration and process management: the productivity dilemma revisited’, Academy of Management Review, Vol. 28, No. 2.
Quality mark e-learning
69
Breitner, M.H. and Hoppe, G. (Eds.) (2005) ‘A glimpse at business models and evaluation approaches for e-learning’, E-learning – Einsatzkonzepte und Geschäftsmodelle, Heidelberg. British Learning Association (2005) Quality Mark Profiles, http://www.british-learning.org.uk/ qualitymark/pages/profiles.htm (retrieved 1 March 2006). Bruder, R., Offenbartl, S., Osswald, K. and Sauer, S. (2004) Third Party Certification of Computer-based Learning Environments, EISTA, Orlando. CEN/ISSS Workshop Learning Technologies (2003) CWA14644 Quality Assurance Standards, CEN, Brussels. CEN/ISSS Workshop Learning Technologies (2006) Learning Experience Transparency, Interim Report, Brussels. Chang Barker, K. (2005) ‘Quality Standards for Consumer Protection’, in A. Hope and P. Guiton (Eds.) Strategies for Sustainable Open and Distance Learning, London: COL/RoutledgeFalmer Press. Chang Barker, K. and Leginsky, V. (2003) eQcheck Procedures, http://www.eqcheck.com/eq/ eQcheckProducers.pdf (retrieved 01 March 2006). Cruickshank, M. (2003) ‘Total Quality Management in the higher education sector: a literature review from an international and Australian perspective’, TQM & Business Excellence, Vol. 14, No. 10. Deming, W.E. (1982) ‘Out of the crisis’, Quality, Productivity and Competitive Position, Cambridge. DIN (2004) DIN PAS 1032-1 Learning, Education and Training Focussing on e-Learning – Part 1: Reference Model for Quality Management and Quality Assurance – Planning, Development, Realisation and Evaluation of Processes and Offers in Learning, Education and Training, Berlin: Beuth Verlag. Dippe, G., Eltén, A., Kollia, V., Lindholm, J., Lindström, B. and Tsakarissianos, G. (2001) Research on Quality Assessment Management and Selection Criteria regarding Content for Schools, Project Report, Version 1.0, European Treasury Browser, IST-1999-11781, D3.1, WP3. Dirr, P. (1999) Putting Principles Into Practice: Promoting Effective Support Services for Students in Distance Learning Programs – A Report on the Findings of a Survey, http://www.wcet.info/ projects/studentservices/Survey%20Report.pdf (retrieved 1 August 2005). Ehlers, U. (2004) ‘Quality in e-learning. The learners perspective’, European Journal of Vocational Training, CEDEFOP, Thessaloniki. Ehlers, U.D., Hildebrandt, B., Görtz, L. and Pawlowski, J.M. (2005) Use and Distribution of Quality Approaches in European e-Learning, CEDEFOP, http://www2.trainingvillage.gr/etv/ publication/download/panorama/5162_en.pdf (retrieved 1 March 2006). Euler, D., Seufert, S. and Wirth, M. (2004) ‘Gestaltung des Qualitätsmanagements zur Zertifizierung von E-Learning-Programmen (Design of quality management for the certification of e-learning programs)’, in D. Euler and S. Seufert (Eds.) E-learning in Hochschulen und Bildungszentren (e-Learning in Higher Education and Education Centers), Oldenbourg, München, Wien. European Foundation for Management Development (EFMD) (2004a) ‘EFMD Certification of e-Learning (CEL)’, CEL Criteria, Indicators, Standards, http://www.efmd.org/attachments/ tmpl_1_art_050606zxrc_att_050712rpku.pdf (retrieved 1 August 2005). European Foundation for Management Development (EFMD) (2004b) ‘EFMD Certification of e-Learning (CEL)’, CEL Quality Criteria Overview, http://www.efmd.org/attachments/ tmpl_1_art_041115geyj_att_050725hfzk.pdf (retrieved 1 August 2005). European Foundation for Quality Management (EFQM) (2003) ‘EFQM excellence model’, European Foundation for Quality Management, Brussels. Fritsch, H. (2004) ‘Quality assurance from the students’ point of view’, International Conference on Supporting Students in e-Learning Systems, Rome, Italy, 24 September.
70
J.M. Pawlowski
Guest, S. (2004) ‘How to be an educated e-learning consumer’, Learning Circuits, March 2004, http://www.learningcircuits.org/2004/mar2004/guest.htm (retrieved 1 March 2006). Gunnarsdóttir, S., Kieslinger, B., Küchler, T. and Simon, B. (2004) ‘From e-learning to learning management: results of an international survey’, Proceedings of I-KNOW ‘04, Graz, Austria. Harman, G. (1998) ‘The management of quality assurance: a review of international practice’, Higher Education Quarterly, Vol. 52, No. 4. Higher Education Funding Council for England (HEFCE) (2003) Embracing Excellence in Education – A Summary of the Learning Gained from Applying the EFQM Excellence Model® in Further and Higher Education, Sheffield Hallam University. Institute for Higher Education Policy (2000) Quality on the Line, Washington, DC. International Standards Organisation (ISO) (2002) ISO 9000 and 14000, http://www.iso.org/ iso/en/iso9000-14000/iso9000/iso9000index.html. ISO/IEC (2005) ‘ISO/IEC 19796-1:2005’, Information Technology – Learning, Education and Training – Quality Management, Assurance and Metrics – Part 1: General Approach. Juran, J.M. (1989) Juran on Leadership for Quality, New York, NY: The Free Press. Kaplan, R.S. and Norton, D.P. (1996) ‘The balanced scorecard’, From Strategy to Action, Harvard Business School Press. Kefalas, P., Retalis, S., Stamatis, D. and Theodoros, K. (2003) ‘Quality assurance procedures and e-ODL’, International Conference on Network Universities and e-Learning, Valencia, Spain, 8–9 May. Khan, B.H. (2004) ‘People, process and product continuum in e-learning: the e-learning P3 model’, Educational Technology, Vol. 44, No. 5. Kulik, C-L. and Kulik, J. (1991) ‘Effectiveness of computer-based instruction: an update analysis’, Computers in Human Behaviour, Vol. 7. Kulik, J. (1994) ‘Meta-analytic studies of findings on computer-based instruction’, in E.L. Baker and H.F. O’Neil (Eds.) Technology Assessment in Education and Training, Hillsdale. Marshall, S. and Mitchell, G. (2002) ‘An e-learning maturity model?’, Proceedings of ASCILITE 2002, Auckland, New Zealand. Neven, F. and Duval, E. (2002) ‘Reusable learning objects: a survey of LOM-based repositories’, International Multimedia Conference, Proceedings of the Tenth ACM International Conference on Multimedia, France. Pawlowski, J.M. (2003) ‘The European Quality Observatory (EQO): structuring quality approaches for e-learning’, ICALT 2003, Athens, Greece, July. Redtray (2005) The e-Learning Strategy Workbook, http://www.redtray.co.uk/docs/REDTRAY _elearning_strategy_workbook.pdf (retrieved 1 March 2006). Samson, D. and Terziovski, M. (1999) ‘The relationship between total quality management and operational performance’, Journal of Operations Management, Vol. 17. Schmiedinger, B., Valentin, K. and Stephan, E. (2005) ‘Competence based business development – organizational competencies as basis for successful companies’, Journal of Universal Knowledge Management, Vol. 0, No. 1. Seufert, S. and Euler, D. (2004) ‘Sustainability of e-learning innovations – findings of expert interviews’, SCIL Report 2, SCIL, University of St. Gallen. Souleles, N. (2005) ‘Staff use of e-learning and graduate competencies for the knowledge economy: a study of the misalignment between rhetoric and practice’, E-learning, Vol. 2, No. 3. Stella, A. and Gnanam, A. (2004) ‘Quality assurance in distance education: the challenges to be addressed’, Higher Education, Vol. 47. Tergan, S-O. and Schenkel, P. (2003) ‘Assessing the instructional power of e-learning applications: a learner-centred checklist approach’, in D. Lassner and C. McNaught (Eds.) Proceedings of the ED-Media 2003 World Conference on Educational Multimedia, Hypermedia & Telecommunication, Honolulu.
Quality mark e-learning
71
v. Brocke, J. (2005) ‘Multi-channel-based dissemination of e-learning systems: a reference process model for the design of applications, technologies, methods and organisations’, Proceedings of the 4th International Symposium on Information and Communication Technologies, ACM International Conference Proceeding Series, Vol. 92. WebKollegNRW (2003) WebKollegNRW – ein zeitgemäßes medienbasiertes Bildungsangebot in NRW, Zulassungsordnung (WebKollegNRW – Contemporary Media-based Education & Training, Admission Regulations), http://www.webkolleg.de/anbieterInfo/Zulassungsordnung/ WebKolleg-Zulassungsordnung-Gesamt-0309021.pdf, 2003 (retrieved 1 March 2006). Zairi, M. and Sinclair, D. (1995) ‘Business process re-engineering and process management: a survey of current practice and future trends in integrated management’, Business Process Management Journal, Vol. 1, No. 1. Zech, R. (2003) Lernerorientierte Qualitätstestierung in der Weiterbildung. LQW2: Das Handbuch (Learner-oriented Quality Assessment in Further Education – LQW2: Handbook), Hannover, Expressum. Zellweger, F. (2003) ‘E-learning strategy: findings from an empirical study of “innovative” higher education institutions’, E-Learn 2003, Phoenix, Arizona.
