Quality improvement of technical education in Saudi ... - Ingenta Connect

The current issue and full text archive of this journal is available at www.emeraldinsight.com/0968-4883.htm

Quality improvement of technical education in Saudi Arabia: self-evaluation perspective Zakarya A. Alzamil Software Engineering Department, King Saud University, Riyadh, Saudi Arabia Abstract Purpose – In Saudi Arabia, technical education is managed by Technical and Vocational Training Corporation. However, there is no independent accrediting body to accredit the technical institutes and assure its quality, which causes the lack of unified quality assurance standards and manageable quality improvement processes. The purpose of this paper is to propose self-evaluation standards to help the technical institutes to evaluate their performance. Design/methodology/approach – The author has used a brainstorming technique of local practitioners in quality assurance. Such technique was performed by adopting DACUM that stands for “Developing A CurriculUM”. The author has used a heuristic educational and training process as the foundation of the standards and processes development, namely; curriculum, environment, training managements, and instructors. A team has been established that has been trained locally and abroad on the quality assurance standards and processes. The team studied several quality frameworks of the different countries to come up with guidelines for quality self-evaluation and standards. Findings – The author has found that such an approach is a very effective tool for improving the institutes’ performance and gives them the flexibility to decide about their missions. Most of institutes’ staff is reluctant to participate in the self-evaluation process because of the fear it may reveal their weaknesses, but with encouragement and motivation especially from the top management they tend to participate in such a process. Self-evaluation helps the educational institutes to be benchmarked with other international institutions, in which good practices may be adopted by the institutes’ managements to achieve their vision. Research limitations/implications – The proposed approach can help the technical education institutes to manage their quality system. In addition, the implementation of such an approach might be the starting point to develop a quality system framework for the technical education in Saudi Arabia. However, further investigation is needed to measure whether applying such standards may help the technical education institutes to meet the required quality standards to attain an accreditation from the international quality agencies. Originality/value – The proposed guidelines for quality standards and processes is a contribution in the accreditation and quality assurance processes for many public and private institutions in Saudi Arabia. In addition, it is an important step to standardize the quality processes.

Technical education in Saudi Arabia 125 Received 14 December 2011 Revised 6 September 2012 4 February 2013 Accepted 18 September 2013

Keywords Quality assurance, Technical education in Saudi Arabia, External evaluation, Internal evaluation, Self-evaluation standards, Technical education Paper type Research paper

The author would like to express thanks to the general supervisors at TED department as well as the heads of TED’s branches for their involvement and participation in this project. Also, thanks are expressed to other local and international experts and quality assurance practitioners for their valuable comments on the initial self-evaluation standards. This work was done when the author was the Director General of Training Evaluation Directorate, TVTC, Riyadh, Saudi Arabia.

Quality Assurance in Education Vol. 22 No. 2, 2014 pp. 125-144 q Emerald Group Publishing Limited 0968-4883 DOI 10.1108/QAE-12-2011-0073

QAE 22,2

126

1. Introduction In Saudi Arabia, technical education has been supported by the government as an important contributor to the national economic and sustainable development. Technical education in Saudi Arabia is managed and supervised by the Technical and Vocational Training Corporation (TVTC). TVTC has established over 45 technical colleges and over 64 vocational industrial institutes and these are managed by Saudi Engineers. In its strategic plan, TVTC emphasized the importance of offering training with sufficient quality, and the plan’s third strategic objective stated that, TVTC shall offer its educational and training programs with sufficient quality that trainees will qualify to attain a proper job in the labor market (TVTC, 2007). Therefore, TVTC established the Training Evaluation Directorate (TED) that is responsible for assuring quality improvement and sufficiency in the technical education institutes (colleges of technology and vocational institutes) to satisfy the growing needs of the labor market for qualified graduates. It should be recognized that attaining quality requires the continuous improvement of performance. In addition, such improvement requires the availability of a process framework within the TVTC to manage quality assurance and improvement, especially with the fast expansion of the TVTC within all regions of Saudi Arabia and spread the technical education provision all over the Kingdom. Although most of the technical education institutes recognize the necessity of establishing a quality system and the importance of gaining performance improvement, they lack a unified quality assurance system that enables them to assure the quality of learning and training. As a result, TED has taken a forward step towards establishing a quality system by preparing a set of quality standards and guidelines that assure and improve the quality of the training and administration activities at the training institutes to achieve the training objectives. As part of their ongoing effort, the TED has developed a Self-Evaluation Standards for Training Units” guideline (TED, 2010a) which aims to guide the leaders of technical education institutes to assess and improve their performance and to continue their improvement. The aim of this paper is to present self-evaluation standards and an internal process to help the technical education institutes to evaluate their performance, and to understand their advantages and weaknesses, to assure and improve the learning and training quality. In addition, such proposed standards and processes provide technical education institutes with objective data that assists the leadership to identify the priority improvement plans to achieve sufficiency in providing learning and training that satisfy the labor market needs. Also, the work presented may help in identifying the degree of compliance of the technical education institutes’ practices with the quality standards, identifying the performance strengths and weaknesses with respect to the quality and accreditation standards, and identify the starting point to build and implement the improvement plans to achieve such quality standards. The proposed work presents a guideline with nine standards that can be used for evaluating the internal processes for achieving the intended goals. These standards were built based on the basic elements of the training processes (curriculum, environment, training management, and instructors). These standards can be used to measure the major elements of the training process to improve performance and to assure the continuous improvement of what is achieved by the students in developing knowledge, skills, tasks, views, values, thinking styles, problem solving, etc. These

standards are incorporate the vision and mission of the institute, institute’s administration, instructors’ performance, students’ assessment, students’ services, human resources, program evaluation, buildings and equipment, and relations with community. Every standard has been given a weight based on its importance, and every standard has several indicators used for assuring improvement and the extent to which such change has been achieved. In addition, to simplify the process, a unified form has been developed as a template for all standards including the indicators, evidence, and the improvement achieved. 2. Motivation Post-secondary education in Saudi Arabia is divided into two main tracks: (1) higher education; and (2) technical education. Higher education (universities) is managed by the Ministry of Higher Education (MOHE), while technical education (colleges of technology) is managed by the Technical and Vocational Training Corporation (TVTC). Each of these governmental agencies has its own rules, regulations, and qualifications framework. The MOHE has an independent accrediting and qualifying body, namely, the National Commission for Academic Accreditation and Assessment (NCAAA) which is responsible for the quality and accreditation of higher education institutions and is headed by the minister of the MOHE. NCAAA focuses on the higher education track and has developed quality and accreditation standards for the public and private higher education institutions (NCAAA, 2009a, b). However, NCAAA is distant from technical education and, as a result, has no standards for accrediting and assuring quality in technical education. In addition, compliance with NCAAA rules and regulations is optional for public universities and colleges, and recently, the MOHE has required all private institutions to comply with the NCAAA accreditation standards and regulations. Although, a national qualifications framework for the higher education in Saudi Arabia has been developed (NCAAA, 2009c) and proposed as a unified quality assurance system for post-secondary education in Saudi Arabia (Darandari et al., 2009), NCAAA has no authority to impose such a system on technical education. In addition, the phenomenon is that such standards and frameworks were designed for higher education institutions and concentrated on the academic programs, activities, and processes which may not be appropriate for the technical education. Irrespective of the accuracy of such hypothesis, the NCAAA framework and its related quality standards were not adopted by the technical education institutes, which caused the lack of an integrated and unified post-secondary qualifications framework. Because there is no “independent” accrediting body that is responsible for accreditation and quality assurance for technical education (private/public), most of the quality assurance processes of the technical education institutes are not unified and so, are hard to manage. Therefore, the TED established a team to work to develop and propose self-evaluation standards and internal processes for assuring and improving quality in technical education. In this paper we present the proposed standards and processes that, we believe, will lead to continuous improvements in technical education in which skilled graduates that satisfy the requirements of the labor market can be provided.

Technical education in Saudi Arabia 127

QAE 22,2

128

3. Related work Most of the quality assurance models use self-evaluation or internal evaluation processes as their foundation for quality improvement in any organization, e.g. The Australian Quality Training Framework (AQTF, 2010), New Zealand Qualifications Authority (NZQA, 2009), Commission of the Council on Occupational Education in USA (COE, 2009), European Network for Quality Assurance in Higher Education (ENQA, 2009), Scottish Qualifications Authority (SQA, 2004), and Quality Assurance Agency for Higher Education (QAA) in UK (Villanueva, 2012). In some countries, internal evaluation is mandatory for the institutions, such as in the UK systems, in which the technical education institutes are required to have an internal verification system which is responsible for reviewing the assessments and the assessment judgments, in addition, an external verification of assessment decisions is done by an external verifier (Gunning, 2010). Although these models have different forms of internal or self-evaluation, institutions are expected to have an audit process in which data on various aspects of institutional conditions and operations is gathered, evaluated and presented in a self-evaluation report. In Barnabe` and Riccaboni (2007) a quality measurement and evaluation mechanisms within the Italian context was discussed in which a quality evaluation project named VAI is presented that was developed at the University of Siena, Italy. This project aimed at the implementation of a system for measuring and evaluating quality. The VAI project consisted of a two-stage assessment, a self-evaluation phase and external peer-reviewing visits. The results of this project showed that, in spite of the cost and time consuming nature of such a two-phase assessment approach, it proved to be very effective. However, the authors noticed that such an approach focused on “what” higher education institutions should have done rather than on “how” they actually should have been done. The VAI project introduced a regulation approach named meta-regulatory that the authors claimed may avoid using rigid parameters and adopt a more flexible, qualitative approach to evaluation. In Villanueva (2012) the quality assurance practices of higher education institutions in Belizean were explored. The author recommended a development of a well-defined internal and external quality assurance framework. Also, this study recommended improving the structures and processes of the internal quality assurance system within the educational institute to utilize and benefit from an external quality assurance system. An evaluation of 17 higher education institutions in Paris was conducted by the National Evaluation Committee (CNE) in France based on the self-evaluation reports that were prepared by assigned teams at the institutions. The self-evaluation was an internal process that was used to understand the status of the institution by gathering information about its activities. The participation of the institute’s staff in the self-evaluation process depended on the encouragement they received from the institution’s top management. The study has found that, the self-evaluation is a necessity for the higher education institutions as well as an important tool for improving the institute performance to attain a highly competitive position (Bollaert et al., 2007). A study aims to emphasize the importance of benchmarking in the higher education quality assessment has been presented in (Achim et al., 2009). The evaluation methods have combined self-evaluation with external evaluations that focus on the institution

as a whole, and support the institutions to reshape their mission, objectives, activities, and organization, to achieve their vision as well as the expectations of their stakeholders. This study recommended benchmarking to identify and adopt good practices as well as to attain continuous improvement and innovation (Achim et al., 2009). A study of the quality system proposal in Portugal was presented in (Pile and Teixeira, 1997). The Portuguese quality system aimed to improve quality by providing less quality regulations and giving the institutions more flexibility to have their own internal systems and processes. Such flexibility helped the universities to decide about their missions and hold them accountable with respect to quality of education. This quality system was performed in two phases, internal and self-assessment performed by the institute according to the guidelines of the national coordination institution (Fundac¸a˜o das Universidades Portuguesas-FUP), and the external assessment performed by FUP through a group of experts. A study of the Faculty of Education at the University of Granada, Andalusia/Spain aimed to verify the need to implement a quality assurance system in the university and determine the need to implement an internal quality assurance system. The study found that there exists a need to implement an internal quality assurance system in the different grades (Hidalgo et al., 2011). A case study of the internal evaluation procedures of the 23 academic departments of a university in Iran aimed to identify the benefits and the obstacles that could arise from using internal evaluation. The study surveyed the perceptions of faculty members and department directors and their reactions to the effects of the internal evaluation on their work and improvement in department quality. This study found that the internal evaluation processes are valuable in Iranian universities. However, internal evaluation should be an integral part of the framework of quality assurance and management especially in case where there is an absence of external evaluation. Also, in order to have effective internal evaluation, the management and decision-making structures should be changed accordingly (Mehralizadeh et al., 2007). As presented in the earlier paragraphs, there is increasing emphasis on self-evaluation which is expanding to cover different quality processes such as programs, learning, and administration. Self-evaluation is required because it fulfills the accountability and development functions which improve the learning and development capacity of higher education institutions ( Jackson, 1997). Although the self-evaluation is a common process for many educational institutes within their quality assurance system to assure the compliance with the standards of their national accrediting body, most of these studies present self-evaluation within a quality system framework, in which a national or international accrediting body is deployed as an external evaluator within such quality system which supports and strengthens adopting the self-evaluation and internal evaluation process. As described earlier, Saudi Arabia lacks a unified quality assurance system for technical education. In addition, there is no national accrediting body for the technical education which adds an additional challenge to adopting and implementing self-evaluation within a quality assurance system. In our proposed approach, we have selected an internal evaluation approach as a foundation for our quality improvement system for two reasons: first, internal evaluation has been used and practiced by most of the quality assurance systems and approaches in many countries and has shown its


QAE 22,2

130

effectiveness and influence in improving quality of educational processes. Second, the absence of an independent quality assurance agency for technical education in Saudi Arabia stresses the need to propose an appropriate quality assurance system that can be adopted by the technical education institutions. We have designed a quality improvement model that can be adopted in cases where no national accrediting or external body exists. In addition, we have integrated the external evaluation within our proposed model so that the external evaluation is performed by external experts outside the institutions to represent an external body. 4. Methodology In order to develop the quality standards and processes we used a brainstorming technique with local practitioners in quality assurance at different technical education institutions. Such technique was performed by adopting DACUM that stands for “Developing A CurriculUM” which is a traditional curriculum development method that has been proposed by the Center on Education and Training for Employment, the Ohio State University, USA (Norton and Moser, 2008). Originally, DACUM was proposed as an occupational analysis tool performed by expert workers in an occupation, in which an occupational skill profile was developed and could be used for several purposes such as program planning, curriculum development, training materials development, needs assessment, etc. The DACUM process for occupational analysis involves practitioners from certain jobs, working on a short term committee assignment with a qualified DACUM facilitator. Workers are recruited directly from business and industry. These workers meet in a form of panel brainstorming discussion to develop a DACUM research chart which contains a list of general areas of competence called duties and the tasks that define that duty. Also, the Panel identifies the general knowledge and skills required of successful workers (DACUM, 2011). Although such an approach was designed for curriculum development of a certain occupation, we believe that such an approach is a suitable choice for understanding the quality standards and processes that are used and practiced in different educational institutes. In addition, it was reported that DACUM had multiple uses such as job analysis, occupational analysis, process analysis, functional analysis, and conceptual analysis. Also, it has been used by many companies, community colleges, and government agencies, and has shown the process to be very effective, quickly accomplished, and low cost (DACUM, 2011). We formed a main team of the general supervisors of the head quarter office of TED department. The team members were trained locally and abroad on quality assurance standards and processes in technical and vocational education, during which they visited several foreign organizations in USA, Great Britain, Europe (e.g. Germany, France, and Finland), Australia, and New Zealand to explore and learn about quality standards and processes in the technical and vocational education at those countries. The team studied several quality frameworks of the countries visited as well as other countries to come up with guidelines for quality self-evaluation and standards. In addition, the TED department has 18 branches located at different geographical places covering all regions in the Kingdom of Saudi Arabia. Each branch office is headed by a quality assurance practitioner with engineering or science qualification as well as quality assurance training certificates. Those practitioners act as a quality assurance supervisor for the technical education institutes under their regional office in which

they are responsible for assuring quality of 35 colleges of technology, 11 girls’ higher technical institutes, and 64 vocational institutes. We believe that the general supervisors and quality assurance practitioners are qualified and experienced practitioners that can produce a sound process for quality improvement. When using DACUM, the general supervisors represent the facilitators of the panels and workshops, and the quality assurance practitioners represent the expert workers according to DACUM approach. We have used a heuristic educational and training process that consists of four basic building blocks as the foundation of the standards and processes development, namely; curriculum, environment, training management, and instructors. Such a common process has been considered as the basic building block for any education and training process. In order to make the brainstorming panels and workshops effective, we formed a committee for each one of the four processes (curriculum, environment, management, instructors) that consisted of one general supervisor as facilitator and four to five quality assurance practitioners as expert workers. Each committee team is responsible to define the standards and guidelines for each process, in which each standard is subdivided into its basic elements, and for each element the team should develop the suggested indicators and/or best practices to measure such an element. We organized eight workshops for each committee, of which three were at the headquarters office and five workshops at the branches which were supervised and facilitated by the main team with the participation of the 18 quality practitioners as expert workers. As described earlier, each committee is responsible for developing the standards and guidelines for one of the four educational processes (curriculum, environment, management, instructors). The data collected from these workshops describe those four educational processes in terms of the suggested standards for each process, the basic elements of each process, and the suggested indicators and/or best practices to measure such elements. The purpose of these workshops was to understand the quality standards and processes used and practiced, analyze the data collected, and benchmark them to international standards and processes to develop self-evaluation quality standards and process guidelines that are achievable and satisfy the desired and required quality of education and training. Each committee that is responsible for one process of the educational processes mentioned earlier (curriculum, environment, management, instructors) organized one or more workshops to achieve the abovementioned objective. Several panel discussions were organized for all committees to discuss the results of their workshops in which an integrated model was designed for all educational processes that consist of the procedures of the internal and external evaluations. In addition, experts from numerous international organizations were consulted to review the proposed standards, such as the New Zealand Qualifications Authority NZQA in New Zealand and the Scottish Qualification Authority in Scotland. In our approach, the data were collected mainly by interviews, workshops, and panel discussion. In addition, some data were collected from the quality manuals of several international quality agencies and some others were obtained by observation during the international visits that the main team conducted. The data were studied and analyzed by the committees during the workshops and the panel discussion sessions. The facilitator of each committee was responsible for the documentation of the data analysis for each workshop, and the main team was responsible for


QAE 22,2

132

documenting the data obtained from the panel discussion sessions. The workshops and panel discussions were continued until the participants agreed on sound processes and guidelines. 5. Self-evaluation: concept and objectives Self-evaluation means the set of procedural steps that are performed by the institute’s staff to assess the performance of their institute based on the given standards. Such assessments are conducted using data about the current state of the teaching and administrative activities, and then measuring the compliance with the stipulated quality standards. The self-evaluation study of the institute includes an entry for teaching and curriculum improvement. This evaluation is an institutional evaluation that was at the level of the institute; therefore, no lower level of evaluation was conducted i.e. evaluation of the academic programs and courses offered. The self-evaluation aims to determine the degree of compliance between the current practices in the institute and the various quality standards. Also, it identifies the strengths and the weaknesses as well as the aspects of improvements to satisfy the requirements of accreditation to achieve the desired quality. Additionally, it helps to identify the starting point for planning, building, and implementing continuous improvement to achieve the excellence. Although the main purpose of the self-evaluation was to support continuous improvement, it is a forward step that improves the institute’s state and supports its status when applying for accreditation (locally or internationally). 6. Self-evaluation process Among the main obstacles to conducting the self-evaluation is the complexity of the process and its documentation that sometimes requires arduous and long training for the evaluation teams. Therefore, we proposed a simple process that helped the institute’s administration to conduct the evaluations using a simple and well-defined process that required minimal training. The self-evaluation process was designed to be performed in two phases. The first phase was the internal evaluation process which was conducted by the institute’s staff. The second phase, was the external evaluation which is done by an external team i.e. peer-reviewing visits, based on the output of the internal evaluation. That is a self-study report, to validate the evaluation conducted by the institute’s team. This phase was proposed due to the lack of independent external accrediting and evaluation bodies for technical education in Saudi Arabia. In addition, it may motivate the self-evaluation process. Although the external evaluation team may not be independent, it simulates the role of the external evaluation body. The use of the two-stage approach (internal self-evaluation and an external peer-review) is a common approach and has usually been identified as a valid and reliable model for quality assessments that does not require the use of rigid or quantitative standards. The self-evaluation aims to spread the quality culture and contribute to quality improvement and efficiency within the institution. In contrast, the external evaluation (peer-review) seeks to identify the strengths and the weaknesses of the organizational systems and/or processes (Barnabe` and Riccaboni, 2007). In some European countries, the educational institutes are responsible for the quality assurance of their programs through an internal evaluation process which includes external peers involved in evaluating programs and/or institutions. However,

other European countries require an external evaluation or accreditation (Orsingher, 2006). External evaluation is used in most of the quality assurance approaches for the purpose of program accreditation and institutional accreditation. Accreditation certifies to the public that an institution or program meets minimum educational standards (Villanueva, 2012). In other words, external evaluation is being used by external independent body to award a certificate of accreditation to a program or an educational institute based on a set quality standards (Orsingher, 2006). 6.1 Internal evaluation process We have defined a clear process for the internal self-evaluation in which the institute can conduct the process in a way that assures the participation of the institute’s leaderships, staff, and students. This process consists of seven steps that include an orientation activity that supports quality culture awareness among institute’s staff and students. In addition, we developed a simple template to be used for presenting the compliance of every aspect of the quality standards. Figure 1 depicts the internal evaluation process steps. In the following paragraphs we present the steps of the internal self-evaluation along with a brief explanation of each. Evaluation’s lead team formation. In this phase the institute formed a team to act as the main committee that lead and monitored the self-evaluation process, this lead team is headed by the institute dean and consists of the vice deans, department heads, instructors’ representative, and students’ representative. The responsibilities of the lead team are shown in Table I. Evaluation plan preparation. In this step, the lead team designs an executive plan to implement the self-evaluation process, in which a timetable for tasks, deadlines, responsibilities and roles, assigned persons, etc., is depicted. In addition a follow-up system should be enforced to assure meeting the deadlines. Also, the lead team should arrange for scheduled meetings to discuss the proposals for assuring the proper implementation of the plan. Self-evaluation study orientation and advertisement. Preparing and orienting the institute’s staff and the students to implement the self-evaluation procedures is very important step towards the success of the self-evaluation procedure. Also, it builds the staff and students’ trust in the process and improves their positive participation to achieve the evaluation goals. Such a phase is a quality culture awareness initiative, in which various tools and approaches can be used to prepare the institute’s staff and students for the self-evaluation, such as workshops, seminars, posters, pamphlets, unofficial meetings, and institute web site. Subcommittee formation and training. In this phase, subcommittees were formed in which every committee specialized in certain tasks of the evaluation process, or a committee was assigned to one or more of the standards of self-evaluation standards. Subcommittees may be established for every department as well to help in the data collection and analysis process. In addition, subcommittees should be trained on the best practices of the internal evaluation processes. Such training may be conducted by organizing local workshops or by formal training via specialized agencies. The institute should consider training and provide the necessary financial support. Data collection. At this stage, the lead team identified the types of data needed, and helped the subcommittees to get them precisely. Generally, the evaluation process required two types of data: quantitative and qualitative data. Quantitative data


QAE 22,2

134

Figure 1. Internal evaluation process steps

represents the numerical data such as students’ results, results percentiles, students’ dropout, staff statistics, students’ statistics, laboratories, and equipment. Qualitative data is descriptive data that can be reached by analyzing the institute mission and vision, opinions and perspectives of all stakeholders such as instructors, administrative employees, students and their parents, institute council members, and the labor market.

The process of data collection should be performed by the subcommittees and monitored by the lead team. The lead team should also provide the subcommittees with simple templates to simplify the data collection process. Data analysis. This step was very important because it described the current state of the institute. The lead team and the subcommittees should consider the following when analyzing the data collected to achieve the most accurate results that may help in understanding the current state of the institute’s performance: . The numerical processing and qualitative analysis of the data should be done based on the evaluation standards. . All members of subcommittees should participate in the process of data analysis, and should be reviewed by the members of the lead team. . The lead team should guide the data analysis process to draw the picture of the current state of the institute in the different fields, and compare it with the desired state based on the measures of accreditation standards. . When extracting the significance via the processing and analysis of the data, every standard should be studied alone and linked to the other standards to create a comprehensive picture of the whole state of the institute quality process.


Self-evaluation report. The self-evaluation study report preparation is a very important phase of the evaluation process and should document the evaluation’s procedures and any related tools, procedures, and challenges. Such reports should help to understand the actual state of the institute’s performance based on the quality standards and the requirements of performance improvement. This report should include the institute basic data, result of the evaluation of every standard, points of strengths, the challenges and their improvement priorities. The self-evaluation report may follow the suggested template in the Appendix. Improvement plan preparation and deployment. At this stage, and as a result of the internal evaluation report, the institute should come up with an improvement and correction plan. A team should be formed by selecting one member from each subcommittee to develop an improvement and correction plan based on the recommendations of the self-evaluations report as well as the recommendations of the external evaluation report as will be described later. The improvement and correction plan should be implemented by deploying the required improvement and applying the corrective actions. The internal evaluation process is a cyclic process, in which the end of each cycle is the start of another cycle. In order for the institute to achieve its vision, the internal No.

Tasks

1 2 3

Planning the evaluation process Forming the subcommittees for the self-evaluation Supervising and supporting the subcommittees in performing their tasks, such as providing them with the needed training Leading and guiding the subcommittees in the process of data collection and analysis Supervising the self-evaluation study final report Taking the proper decision on the process improvement priorities

4 5 6

Table I. Lead team tasks and responsibilities

QAE 22,2

evaluation process must continue and the performance improvement must be deployed as well. The software development spiral model (Boehm, 1988) may be adopted with slight modification, for the internal evaluation process in which the final product or target is achieving the institute’s vision. In this model, the evaluation process is performed in a spiral fashion, in which, for each cycle, the institute gets closer to achieving its vision.

136 6.2 External evaluation process The external evaluation, as described earlier, aims to simulate the role of an external evaluation body and to motivate the self-evaluation process. External evaluation is performed by forming an external evaluation team. To understand the external evaluation team formation, we describe the structures of the TVTC. TVTC consists of a headquarters, which is headed by a governor, and thirteen regional technical and vocational training (TVT) councils in which each council supervises the technical institutes of their region. In our proposal we assigned the external evaluation to the TVT council to simulate the role of the external body. The external evaluation team was headed by the chair of the (TVT) council, and consisted of a supervisor of the TED (from the headquarters office), the training evaluation unit director, students’ services unit director, and staff’s services unit director. In addition, the team may include representatives of various headquarters’ directorates such as students’ services directorate, staff’s services directorate, curriculum development directorate, projects and maintenance directorate, equipment directorate, and community services directorate. The tasks of the external evaluation team are presented in Table II. The external evaluation team members belong to the same organization and this may cause some conflicts of interest; therefore, in order to reduce such influence, the member should abide by the code of ethics listed in Table III. The external evaluation process steps are depicted in Figure 2. As has been described in the internal evaluation process, the self-evaluation report should be submitted by the institute to the TVT council. When the self-evaluation report is submitted, the external evaluation should start by enabling the external evaluation team to deeply study the institutes’ self-evaluation reports. Then, the external evaluation team should document its comments and feedback on the institute’s reports; No. 1 2 3 4 5 6 7 Table II. External evaluation team tasks

8 9 10 11

Tasks Preparing the site visits plan for the external evaluation Conducting the external evaluation’s visits to the institutes Reviewing the policies and procedures of self-evaluation and external evaluation Verifying and reviewing all data and reports prepared by the internal evaluation team Providing the support and consultation to the internal evaluation team Inspecting and assessing the improvement plan of the institute prepared by the internal evaluation team Proposing the recommendations to the institute to improve its performance and achieve its goals effectively Preparing the external evaluation report, and providing the feedback to the institute Developing the system to follow-up and update the improvement/correction actions Developing the quality assurance process/program Sharing the good practices between the institutes managed by the TVT council

such comments should be constructive by considering strengths and weaknesses. Next, the external evaluation team should develop an institute site visit plan, and schedule such a visit and conduct it. The institute may be provided with the site visit plan that describes the activities to be performed during the visit to allow the institute to prepare for the event. During the visit, the external team should meet the self-evaluation team and a sample of the academic staff, students, and administrative staff of the institute. After the site visit finishes, the external evaluation team should prepare its report based on the review of the self-evaluation report as well as the site visit observations. Then, the external evaluation report is provided to the institute and the TED. The external evaluation team should develop a follow-up plan to observe the institute in implementing the recommendations of the self and external evaluation reports as well as implementing the improvement/corrective action plan. Finally, the evaluation reports (internal/external) should be archived and recorded at the institute and TVT council for future self and external evaluation process.


7. Self-evaluation standards The self-evaluation standards represent various aspects of the educational and training processes. they includes the following standards: institute’s vision and mission, institute’s administration, instructors’ performance, students’ assessments, students’ services, human resources, programs’ evaluation, buildings and equipment, and relations with the local community. Every standard can be given a certain weight based on its importance to the institute, and every standard consists of set of indicators that can be used to measure the standard. These standards have been described in more detail in (TED, 2010a). In the following subsection, the standards are described briefly. 7.1 Institute’s vision and mission This standard measures the degree of clarity or ambiguity of the vision and mission, and its suitability to deliver the institute goals. Also, it measures how the vision and mission influence the planning and other processes within the institute. 7.2 Institute’s administration This standard measures the performance of the administration processes according the standards and best practices of the management and development of processes, such as internal auditing and planning. Also, how the administration contributes to spread the quality culture among the institute’s staff, as well as measuring the level of human skills and communication to improve the work environment.

No.

Tasks

1 2 3 4 5

Perform the evaluation in a professional and neutral way without any alignment Achieve the external evaluation tasks with delicate care using the best practices Be an independent of all the activities that you are evaluating Assure the neutrality and credibility in your evaluation and avoid any conflict of interest Respect the privacy and confidentiality of your work, and use the data only for the purpose of evaluation

Table III. External evaluation team code of ethics

QAE 22,2

138

Figure 2. External evaluation process steps

7.3 Instructors’ performance This standard measures the instructors’ performance by identifying the instructor’s competences through the learning and teaching processes, and how they are designed and implemented. Also, it measures the effects of such processes in improving the students’ learning (knowledge and skills). In addition, it helps to identify the commitment of the instructors to their roles.

7.4 Students’ assessment This standard measures whether a variety and adequate students’ assessment tools have been used to measure students’ achievement, and determines whether the assessment rules and regulations have been followed. Also, it establishes that the students’ results have been produced and analyzed, and the feedback has been provided to the students in a timely way to enable them to improve their performance.


7.5 Students’ services This standard measures the services provided to the students when they enrolled at the institute, such as the placement tests, competence tests, occupational tests, etc. that supported the students in choosing the proper occupation. Also, it assesses the continuous follow-up through the record-keeping of all students’ activities (academic, behavior, medical, etc.), as well as other services provided to the students such as sports, socials, scouts, etc. In addition, it assesses the services provided to the students on graduation, such as careers opportunities. 7.6 Human resources This standard measures the utilization of the human resources through assuring fairness and equality among the institute’s staff as well as motivating and supporting them by identifying their training and development needs to improve their efficiency and performance. 7.7 Programs’ evaluation This standard measures the institute’s efforts in evaluating and improving its programs’ curricula to meet the local labor market needs. Also, it measures the institute interaction with the stakeholders. 7.8 Buildings and equipment This standard assesses the institute’s utilization of its buildings, workshops, laboratories, equipment, and facilities. In addition, it helps the institute with the approach to buildings and facilities preservation, maintenance, and updating to provide the proper educational environment that satisfies the stakeholders’ needs. 7.9 Community relations This standard assesses the engagement of the institute with the local community via community training programs and services. Also, it measures how the institute interacts with the local community such as community activities involvement, raising the profile of the institute vision and mission in the community, making partnership with the community etc. 8. Benchmarking and evaluations We decided to benchmark the self-evaluation standards developed to have confidence in their validity, completeness, and comprehensiveness with respect to the major aspects of quality assurance and improvement. The standards were benchmarked against some local and international standards. For instance, the standards were benchmarked against the local Saudi’s National Commission for Academic Accreditation and Assessment (NCAAA, 2009a, b) and some international standards

QAE 22,2

140

such as the New Zealand Qualifications Authority (NZQA, 2006), European Network for Quality Assurance in Higher Education (ENQA, 2009), USA’s the Council on Occupational Education and the Southern Association of Colleges and Schools (COE, 2009; SACS, 2009), The Australian Quality Training Framework (AQTF, 2010), and Scottish Qualifications Authority (SQA, 2007). The benchmarking was performed by investigating whether our defined standards were recognized as a focal quality assurance issue at other local and international quality assurance agencies. Table IV shows the mapping between our proposed standards with the standards of other local and international agencies. In the cases where we were able to map our standard to a standard or an element of a standard at an agency we marked it with the (U) symbol; however, if no mapping could be identified, we marked it with the ( £ ) symbol. It should be noticed that the names of the standards may vary in the categorization and in scaling of these standards and their subcategories. For instance one standard at one agency may be considered as an element of a standard at other agency and vice versa. In addition, numerous international organizations were consulted, for instance overseas workshops were organized with experts from Scottish Qualification Authority as well as New Zealand Qualifications Authority NZQA (TED, 2010b) in which, the proposed standards were reviewed and some feedback gathered and used to improve the standards. As depicted in Table IV, all of our proposed standards met the quality standards of other international agencies; however, some standards such as the standards related to the vision and mission, and community relations could not be identified in the case of some quality models. For instance, we could not match the vision and mission standard of our proposed standards to the Australian’s AQTF quality standards. Also, we could not match the community relation standard of our proposed standards to the Scotland’s SQA standards. It should be noted that we did not include one important standard that is required by all of the accrediting and quality assurance agencies that is related to the financial resources. This standard was omitted because the technical education institutions in Saudi Arabia, which are targeted by this quality assurance system, are governmental institutions in which the funds are guaranteed by the government. Also, there are other standards that were not explicitly identified within our proposed standards, and these have been clearly identified within the quality standards of other international agencies, such as the library, learning resources, and distance education. Such standards have now been included in our proposed standards as elements within the students’ services standard.

No. Standard

Table IV. Benchmarking with local and international standards

1 2 3 4 5 6 7 8 9

Vision and mission Administration Instructors performance Students assessment Students services Human resources Programs evaluation Building and equipment Community relation

NCAAA COE SACS SQA ENQA AQTF NZQA Saudi USA USA Scotland Europe Australia New Zealand U U U U U U U U U

U U U U U U U U U

U U U U U U U U U

U U U U U U U U £

U U U U U U U U U

£ U U U U U U U U

U U U U U U U U U

9. Conclusions In this paper we proposed self-evaluation standards for technical education in Saudi Arabia. A traditional analysis approach namely, DACUM, was used to develop a guideline that can be used for judging the internal processes for achieving the educational institute’s intended goals. The presented standards were built based on the basic elements of the training processes (curriculum, environment, training managements, and instructors). We have benchmarked these standards to some local and international standards. It has been reported that (Bollaert et al., 2007; Barnabe` and Riccaboni, 2007), in spite of the cost and time consuming nature of such a two-phase assessment approach, such an approach is a very important and effective tool for improving technical education institutes’ performance. Also, it gives the institutions the flexibility to decide on their missions, and holds them accountable with respect to quality of education. In a pilot project related to the instructors’ performance standard, TED department developed a trainer’s quality manual (TED, 2010c) using a self-evaluation approach to help instructors to improve their performance by documenting and evaluating their performance according to suggested guidelines. We have observed the implication of such an approach via the observation visits that were conducted by the general supervisors in TED department to the different technical institutes all over the Kingdom before and after applying the trainer’s quality manual, and, as indicated by the observation visits’ reports, the instructors’ performance improved (TED, 2010d). Although encouraged by the observation’s results, we have noticed that, most of institutes’ staff were reluctant to participate in the self-evaluation process because they thought that such processes aim to reveal their weaknesses, but with encouragement through several workshops the aim to spread quality culture, they were motivated to participate especially when top management participated in the process. Some studies have recommended benchmarking to adopt good practice as well as attain continuous improvement and innovation, we have found that the institution management and staff were eager to be compared with other international institutions to achieve their vision. Heuristically these standards should cover all major aspects of quality assurance and improvement in technical education institutions, and currently they are in the implementation stage and are offered as an optional to the technical education institutes. We believe that, due to the lack of a unified quality system, the proposed approach can help technical education institutes to manage their quality system. In addition, the implementation of this type of approach might be the starting point to develop a quality system framework for technical education in Saudi Arabia. Although we are confident that these standards match the standards of several international quality assurance agencies and they should be standardized, further investigations should be planned to assess the proposed processes for the internal and external evaluations with respect to their applicability and preciseness in assisting technical institutions in assuring and improving quality of teaching and training. A pilot project of several technical education institutes might be implemented by measuring quality before and after incorporating such an approach to understand its implications for quality assurance and improvement better as well as understanding the influence of the external evaluation on the self-evaluation due to the lack of an independent external evaluation body. In addition, further investigation is needed to measure whether applying such standards may help the technical education institutes to meet the required quality standards to reach the standards to qualify for accreditation from international quality agencies.


QAE 22,2

142

References Achim, M.I., Ca˘bulea, L., Popa, M. and Mihalache, S.-S. (2009), “On the role of benchmarking in the higher education quality assessment”, Annales Universitatis Apulensis Series Oeconomica, Vol. 11 No. 2. Australian Quality Training Framework (AQTF) (2010), Users’ Guide to the Essential Conditions and Standards for Continuing Registration, Commonwealth of Australia, Australia. Barnabe`, F. and Riccaboni, A. (2007), “Which role for performance measurement system in higher education? Focus on quality assurance in Italy”, Studies in Educational Evaluation, Vol. 33 Nos 3-4, pp. 302-319. Boehm, B. (1988), “A spiral model of software development and enhancement”, IEEE Computer, Vol. 21 No. 5, pp. 61-72. Bollaert, L., Brus, S., Curvale, B., Harvey, L., Helle, E., Jensen, H.T., Komljenovic˘, J., Orphanides, A. and Sursock, A. (2007), Embedding Quality Culture in Higher Education; A Selection of Papers from the 1st European Forum for Quality Assurance, The European University Association, Brussels. Colleges of the Southern Association of Colleges and Schools (SACS) (2009), The Principles of Accreditation: Foundations for Quality Enhancement, Commission on Colleges of the Southern Association of Colleges and Schools, Decatur, GA. Council on Occupational Education (COE) (2009), The Self-Study Manual, Commission of the Council on Occupational Education, Atlanta, GA. DACUM (2011), available at: www.dacumohiostate.com/ (accessed 12 December 2011). Darandari, E., Al-qahtani, S., Allen, I., Al-yafi, W., Al-sudairi, A. and Catapang, J. (2009), “The quality assurance system for post-secondary education in Saudi Arabia: a comprehensive, developmental and unified approach”, Quality in Higher Education Journal, Vol. 15 No. 1, pp. 39-50. European Network for Quality Assurance (ENQA) (2009), Standards and Guidelines for Quality Assurance in the European Higher Education Area, European Association for Quality Assurance in Higher Education, European Union. Gunning, D. (2010), “Quality Assurance in Vocational Education and Training”, International Encyclopedia of Education, 3rd ed., Elsevier, Amsterdam, pp. 482-488. Hidalgo, E., Villoria, J. and Romero-Cerezo, C. (2011), “The necessity and challenge of setting up a quality assurance system in the higher education system in Andalusia (Spain)”, Procedia Social and Behavioral Sciences, Vol. 15, pp. 2972-2976. Jackson, N. (1997), “Internal academic quality audit in UK higher education: part II - implications for a national quality assurance framework”, Quality Assurance in Education, Vol. 5 No. 1, pp. 46-54. Mehralizadeh, Y., Pakseresht, M.J., Baradaran, M. and Shahi, S. (2007), “The dilemma of internal evaluation in higher education: a longitudinal case study”, Quality Assurance in Education, Vol. 15 No. 3, pp. 352-368. National Commission for Academic Accreditation & Assessment (NCAAA) (2009a), Standards for Quality Assurance and Accreditation of Higher Education Institutions, National Commission for Academic Accreditation & Assessment, Saudi Arabia. National Commission for Academic Accreditation & Assessment (NCAAA) (2009b), Standards for Quality Assurance and Accreditation of Higher Education Programs, National Commission for Academic Accreditation & Assessment, Saudi Arabia.

National Commission for Academic Accreditation & Assessment (NCAAA) (2009c), National Qualifications Framework for Higher Education in the Kingdom of Saudi Arabia, National Commission for Academic Accreditation & Assessment, Saudi Arabia. New Zealand Qualifications Authority (NZQA) (2006), Self-Evaluation Guide and Workbook, New Zealand Qualifications Authority, New Zealand. New Zealand Qualifications Authority (NZQA) (2009), Using Evaluation to Strengthen Organisational Self-Assessment, New Zealand Qualifications Authority, New Zealand. Norton, R.E. and Moser, J. (2008), DACUM Handbook, Center on Education and Training for Employment, The Ohio State University, Columbus, OH. Orsingher, C. (2006), Assessing Quality in European Higher Education Institutions Dissemination, Methods and Procedures, Physica-Verlag, New York, NY. Pile, M. and Teixeira, I. (1997), “The Importance of Quality Assessment in Higher Education Institutions”, Technical Report, available at: http://gep.ist.utl.pt/files/artigos/The_ Importance_Quality_Assessment.PDF Scottish Qualifications Authority (SQA) (2004), Guide to Assessment and Quality Assurance for Employers and Training Providers, Scottish Qualifications Authority, Scotland. Scottish Qualifications Authority (SQA) (2007), Awarding Body Criteria, Scottish Qualifications Authority, Scotland. Technical and Vocational Training Corporation (TVTC) (2007), “The Strategic Plan for the Technical and Vocational Training in Saudi Arabia 2008-2030”, Technical Report, Technical and Vocational Training Corporation, Saudi Arabia (In Arabic). Training Evaluation Directorate (TED) (2010a), Self-Evaluation Standards for Training Units, Guidelines, Technical and Vocational Training Corporation, Saudi Arabia, (In Arabic). Training Evaluation Directorate (TED) (2010b), Self-Evaluation Standards Workshop, Technical Report, Technical and Vocational Training Corporation, Saudi Arabia, (In Arabic). Training Evaluation Directorate (TED) (2010c), Training Evaluation Directorate (TED) Trainer Quality Manual, Guidelines, Technical and Vocational Training Corporation, Saudi Arabia, (In Arabic). Training Evaluation Directorate (TED) (2010d), Supervision Visits’ Reports, Academic Year 2010-2011, TED Archive, Technical and Vocational Training Corporation, Saudi Arabia, (In Arabic). Villanueva, N.N. (2012), “Assuring quality in Belizean higher education: a collective case study of institutional perspectives and practices”, PhD dissertation, University of Nebraska-Lincoln, Lincoln, NE. Further reading Jackson, N. (1996), “Internal academic quality audit in UK higher education: part I: current practice and conceptual frameworks”, Quality Assurance in Education, Vol. 4 No. 4, pp. 37-46.


QAE 22,2

Appendix

144

Figure A1. Self-evaluation form

About the author Zakarya A. Alzamil earned his PhD in Computer Science from the Illinois Institute of Technology, Chicago, Illinois, USA. His research interest is in the field of software engineering where he is specialized in software performance analysis and software testing as well as investigating new approaches in software engineering education and information security. In addition, he has several contributions in the field of education quality improvements. He was the Director General of Training Evaluation Directorate, TVTC, Saudi Arabia. Currently he is an Associate Professor at the Software Engineering Department, King Saud University, Saudi Arabia. Zakarya A. Alzamil can be contacted at: [email protected] To purchase reprints of this article please e-mail: [email protected] Or visit our web site for further details: www.emeraldinsight.com/reprints