Session F3A DEVELOPMENT AND IMPLEMENTATION OF A NEW MODEL FOR ACADEMIC EVALUATION César Cárdenas1 Abstract — Finishing an engineering program with the highest academic standards, and giving a diploma certifies the graduate skills to apply pertinent knowledge to the practice of engineering. Traditionally, the way of certifying the knowledge is through isolated academic evaluations whose results are supposed to estimate the overall student performance. The necessity for predicting and diagnosing the students performance and for improving the teachinglearning processes through the use of academic evaluations, in a continuous way, is imperative in educational institutions with the commitment of continuous quality improvement. Moreover, with these results, the administration could take actions on time, especially if the institutions have defined, (a requirement for graduation) to pass an academic evaluation. This document shows the academic results of the Monterrey Tech Querétaro Campus versus Monterrey Tech System (ITESM System) and CENEVAL national evaluations (National Center for Evaluation) in different curricula of Electronics Engineering programs. A methodology is proposed for implementing an academic adaptive evaluation system using the technological platform Lotus Notes / Learning Space. The objective is to certify the academic performance of the students when they finish their engineering studies.
Introduction There are many points of view for a successful graduate: society, industry, faculty and students, and they are very diverse [2]. The university has the responsibility for assuring that graduated students have the characteristics require by these constituencies (EC2000 requirement). On the other hand, the university must use a set of methodologies in order to assure that students met the constituencies' requirements; some examples of those methodologies are examinations, projects, homework, etc. The more efficient method should predict the students success, and so the graduate, not only in the academic side but also in the social one, where other skills such as attitudes and values are required. Having those attitudes and values could be a reason for which some times excellent academic results do not imply professional success. The ITESM System (Monterrey Tech) applies a Middleof-Program Examination (EMC) and an End-of-Program Examination (EIC) to all the students. The ITESM Querétaro Campus (CQ) has initiated the application of national 1
examinations with the objective for obtaining a more precise academic evaluation. With the purpose of having a complete evaluation form, that includes the academic and extra-academic education (hidden curriculum), a formal work has been initiated which intends to consolidate all the efforts centered in the students education, and that contemplates the adaptive evaluation in order to react quicker and improve the use of resources.
The Educational System The ITESM Querétaro Campus (CQ) is part of the national private educational system Monterrey Tech, accredited by SACS; The ITESM CQ is one of the two only institutions in Mexico with engineering programs recognized as substantially equivalent university programs accredited by ABET, as well as with programs accredited by CACEI, the Mexican counterpart of ABET. The periods of study consist of 16 weeks, 6 subjects average, 8 credits hours and 48 hours-class-semester each. Policies define that there should be 3 partial evaluations (generally exams) and a final exam per course. The partial exams are applied approximately each month and the final exam is determined a priori. Additionally, since 1994, the ITESM System initiated an academic effort to apply the EMC and the EIC examinations. These exams are applied each semester, and is intended for students that met the requirements established by the national academies of each academic program.
Current Examinations There are several instruments for evaluating the students learning level, for instance: partial and final examinations, projects, portfolios' evaluation, home-works, general examinations, admissions examinations, national examinations, etc. Besides these evaluations the ITESM System has developed the EMC and EIC which are describe bellow. The EMC of the ITESM System is applied to the students that have concluded their education in Basic Sciences, typically after the fourth semester; the EMC has the following objectives: • To know the level of the academic performance, find opportunity areas and standardize the academic education among the campi.
César Cárdenas, Academic Program Director, Electronic School, ITESM Querétaro Campus, Santiago de Querétaro, Querétaro, México, 76130,
[email protected]
0-7803-6424-4/00/$10.00 © 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference F3A-15
Session F3A • •
Measure the learning of the student in the career inside his/her campus and at the system level. Improve the teaching-learning processes.
The EIC of the ITESM System is applied to the graduate candidate students (last semester) and has the same objectives that the EMC. Students that have initiated their studies in August of the 2000, will require to pass these exams in order to be graduated. Given the importance of this statement it is imperative to initiate a process of organizational learning in the evaluation subject. The elaboration of the table of contents and the items for the EMC and EIC for each program correspond to the national academies, they delegate this responsibility to the director of the corresponding program of each campus and he/she at the same time requests it's elaboration to the local academies. The CENEVAL (National Center for Evaluation) was founded in 1994 as a non governmental institution, selffinanced through the base of the offer of its services and which most important goal is to provide a good service for the members of the educational sector. CENEVAL is a member of The International Association For Educational Assessment. CENEVAL measures, evaluates, analyzes and disseminates the students and professionals' academic results, especially whit regard to their academic abilities and learning. The center offers four types of evaluation instruments with a national, indicative and unique character: • National Admission Examination for High School (EXANI I). • National Admission Examination for Higher Education (EXANI II). • National Admission Examination for Graduate Studies (EXANI III). • General Examination for Graduating Licensee (EGEL) The EGEL were made for recent (one to three years) bachelor's degree graduates. The CENEVAL examination, are set up and elaborated with the collaboration of experts groups integrated in technical councils, whose determinate the criteria and contents of each evaluation instrument. The ITESM CQ applied in October 1998 to the EGELIEo (Electronics Engineering) with 15 applicants and in November of 1999 with 22 applicants. The second application has also applied to 66 students from the academic programs of software engineering, agronomy, accountancy, administration and chemistry. The Results
Some results are showed in the page six. Correlation could be obtained in two ways: through the time in similar test and through different tests. It is important to mention that the EGEL-IEo is an open book examination with 2 days (16 hours) duration, while the EMC and the EIC are 3 hours duration in one day and closed book. In both exams supported formularies are prohibited. In the EGEL examination the use of programmable calculator is prohibited while in the EMC and EIC any type of calculator can be used. The table 1 shows basic differences among the three types of exams. TABLE 1 COMPARISON BETW EEN EMC, EIC AND EGEL EXAMINATIONS. EXAM Time Days Calculator Books Cost Data Base Questions Objective
EMC EIC (IEC/ISE) 3 hrs 3 hrs 1 1 Programmable Programmable No No 20% of a course 20% of a course Aprox. 100 Aprox. 100 Aprox. 100 Aprox. 64/100 Intern Intern
EGEL 16 hrs 2 Simple Open Aprox. USD$55 Aprox. 700 312 National Profile
A revision of the EGELs' specification tables and our curricula was elaborated. Results show that only an average percent of 38.5% of all the subjects in EGELs' examinations are taught. See table 2. TABLE 2 EGEL AND EIC SUBJECT COMPARAISON. AREA Mathematics Physics Chemistry Electromagnetic Theory Theory of Circuits Control Theory Electronic Devices Analog Electronics Digital Electronics Electrical Measurement Programming Languages TOTAL
EGEL 144 39 45 22 39 21 25 76 66 28 20 525
ITESM 79 17 18 10 12 4 10 6 35 9 2 202
% COV 54.9% 43.6% 40.0% 45.5% 30.8% 19.0% 40.0% 7.9% 53.0% 32.1% 10.0% 38.5%
We obtained the third national place in the 1998 and 1999 EGEL applications; 6 of 17 and 5 of 11 High Performance Testimonies (first place) in 1998 and 1999 EGELs' applications. There are some differences comparing these results with the EIC examination. These results made us though the possibility that we are carrying out a work which we are not aware or even we are not evaluating. We have called the hidden curriculum to this other part of the profile that it is not evaluated. A model of this curriculum is shown in the figure 1. The hidden curriculum is the part without shadow. It can be observed that applying a greater quantity of
0-7803-6424-4/00/$10.00 © 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference F3A-16
Session F3A evaluations, certification of the knowledge is more accurate. To obtain a more accurate evaluation we should develop our own evaluation tool based on our campus profile.
GRADUATE PROFILE
EIC
EMC
CENEVAL FIGURE. 1 T HE HIDDEN CURRICULUM.
We propose an integral evaluation, which includes the hidden curriculum, perhaps not in a 100% but in a significant degree of coverage. While more complete evaluation more understanding of the educational process impact.
The Profile From the systemic point of view, two profiles are found in the educational process: the income profile and the outcome profile. See figure 2. This figure is the same in case of an academic program, a subject or a class [8].
The outcome profile is the curricular intend, it establish the knowledge, abilities, attitudes and values that the students should acquire through the educational process. The profile is the expression of what we want and think the student obtains at the end of the educational process and in the mission of the institution. On the other hand, the income profile is the precise setting, congruent and clear of the contents that a student should know to study with greater possibilities of succeeding in a specific curricular unit. All the profiles are constituted by complex contents. In other words there are contents that include other contents, and so on. From this perspective we can spoke of first order contents, second order contents, etc. It will be necessary to evaluate also the contents to different levels; in other words a meta-assessment approach [9]. A method to define profiles that facilitates the process of evaluation, especially in a subjective domain, is based on the observation of expected conducts. The definition of a profile is not reduced only to the establishment of a component or a product (knowledge, abilities, attitudes and values), which students must have upon entering or must obtain upon graduation; it also includes the process (types of actions that "guarantee" the product) we need to achieve the profile and the necessary mechanisms to confirm if the product is being obtained or not (evaluation). Profiles are the origin and the end of the instructional objectives, of the learning activities and of the evaluation processes.
Model of the Integral Profile (MPI) INPUT PROFILE
CONTENTS
OBJECTIVES
LEARNING ACTIVITIES
EVALUATION
OUTPUT PROFILE
FIGURE. 2 T HE EDUCATING PROCESS, INCOME AND OUTCOME P ROFILES.
The outcome profile is the exact setting, congruent and clear, of the competencies (general content or combination of contents) that the students should learn in a specific program, subject or class.
It is very clear that program profiles are each time more integrated, complete and including. For example the mission of the 2005 ITESM System is summarized in the following paragraphs: The mission will guide the Institute activities over the next ten years and is the product of a consultation process that includes all ITESM constituencies, which included members of the board of trustees of each of the cities where the Institute has a campus, presidents, vicepresidents, directors, faculty, alumni, and students across the country. The ITESM System has the mission of educating individuals who are committed to the social, economic, and political improvement of their communities, and who are internationally competitive in their areas of specialty. Carrying out research and extension relevant to Mexico's sustainable development is also part of the Institute's mission. The Institute provides students with academic training to make them ni ternationally competitive in their chosen disciplines.
0-7803-6424-4/00/$10.00 © 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference F3A-17
Session F3A Values and Attitudes The institute has defined a set of values, attitudes and abilities as part of the desired profile of the students. Through all its activities, the Institute promotes to be: honest, responsible, leaders, entrepreneurs, innovators and imbued with a commitment to personal development; encourage the development of a commitment to ethic work, a clear awareness of national and regional needs, a commitment to the sustainable development of the country and their local communities, a commitment to being agents for change, respect for human dignity and such inherent rights and responsibilities as the right to truth, freedom and protection under the law, respect for the environment, appreciation of culture, commitment to maintaining good physical condition and international outlook. Abilities During the teaching-learning process, the Institute develops the following skills and abilities in its students: learning on their own, analysis, synthesis and abstraction, critical thinking, creativity, identifying and solving problems, decision-making, team work, hard work, a culture of total quality, efficient use of telecommunications and information technology, proficiency in english, good oral and written communication skills. For these reasons we should have very clear the integral concept of evaluation, which consists of considering all the possible ways of educating a student during its educational process. An outcome evaluation focus on conducts observation. A model that clarifies the integral profile could be a fifth dimension curricula (knowledge, abilities, attitudes, values and time), each subject is related by developed connections taking into account the same conduct. Conducts are integral humankind expressions (holistic point). An extension of this model will continue to be a research work; this model is showed in the figure 3.
Level 4: Values Level 3: Attitudes Level 2: Abilities Level 1: Knowledge terms
time graduation
FIGURE. 3 MODEL OF THE I NTEGRAL P ROFILE (MPI)
The MPI helps us to select the type of evaluations that should carry out, but not the periodicity of the application. This theme will be commented in the following point.
The Adaptive Evaluation The traditional evaluation consists of isolated exams, which in overall are the result of the academic fulfillment of the students. It is well known that not always the academic result implies professional success. To have a form of evaluating the fulfillment in an individual adequate way given the chance to determine areas of opportunity in a quicker form [1][7]. Today, the concept of adaptive testing applies more to a family of test administration and scoring strategies that to a particular testing procedure. An adaptive test can be considered, as a test in which the examinee is presented with a set of items, which is the most appropriate for his or her level of ability. The main goal of an adaptive test is to administer an optimal sample of items by selecting them from a well-calibrated item pool. The word "adaptive" in adaptive testing implies that the items to be presented, depend on the response the examinee provides to all or some of the previously administered items. It is no accidental then that early work on adaptive testing called it a response-contingent approach. It is urgently needed to go on adaptive evaluation because next generations will have to pass EMC and EIC examinations in order to graduate. This means we have to predict the performance before the students apply to the examinations. On the other hand, better measurement is thought to be provided by a test if it contains items appropriate to the examinee's ability level. The adaptive evaluation has some repercussions in the common practices of the academic administration, for instance the application of asynchronous evaluations, with particular number of questions and number of evaluations applied, etc. The adaptive test is part of the qualification referred to criterion, which indicates a form of assigning the judgment to the fulfillment of a person that is submitted to one or several tests. This type of evaluation differs of the norm referred, where the group fulfillment defines the levels of acceptance [4]. Taking into account the MPI it will be necessary to apply academic evaluations, test of abilities, attitudes and values, monitoring the development of abilities, attitudes and values among others. The adaptive evaluation suggests the use of computational tools in order to facilitate the educational work. The platform that we will use is Lotus Notes and the application will be Learning Space.
0-7803-6424-4/00/$10.00 © 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference F3A-18
Session F3A The Technological Platform In 1995, the ITESM System initiated the re-engineeering of the teaching-learning process as a premier strategy defined in its mission. The re-engineering process changed the teaching paradigm, converting the teaching centered process in a learning centered process; and developing abilities, attitudes and values in a systematic and planned way. At present 49% of overall subject are delivered under this new model. The ITESM Querétaro Campus has 349 courses under this new model [6]. The technological platform used for the implantation of this new model is Lotus Notes/Learning Space (LN/LS). LN/LS consists of five databases: Schedule, Media Center, Course Room, Profiles and Assessment Manager [5]. Students enter courses through the Schedule, which operates as a course syllabus. The Media Center holds the materials students need for their classes. The Course Room is the asynchronous classroom an interactive, facilitated environment for secure collaboration among students and instructors. The Profiles area helps students and instructors get to know each other by providing information on each participant's interest and background. The Assessment Manager lets instructors create a variety of assessments including tests, surveys, and quizzes using new questions or by drawing on a bank of questions created previously. In live classroom sessions as well as asynchronously, instructors can give online true/false, yes/no, multiple choice, and short-answer quizzes, assessing progress and checking the level of student engagement [6]. A simple implantation methodology has been started to support the re-engineering process of individual subjects; it consists on a teachers sensibilization process, evaluation training, item elaboration, implantation and on going improvement.
Methodology of Implantation There are four strategies for implementing and institutionalizing assessment in the university [3]. The first one is the most cultural transforming: initiate a structured process to involve faculty and staff in the ongoing planning, development, and monitoring of the program. We would like to enrich the academic life and we think this is the one we must select. The participation of the professors of the ITESM Querétaro Campus in the elaboration of items for national examinations has been traditionally poor, as an example of the approximately 80 items of database of the EIC of Electrical Engineering (IEC), the 80% was elaborated by Monterrey Campus. One of the main barriers by which professor does not participate in the elaboration of items is the training.
Given that results of the EMC and EIC are performance indices among all the other campi of the ITESM System, the first phase consisted of doing a national revision of the examinations. This first phase was carried out in January 2000 at the ITESM CQ. Another form of carrying out this phase is that professors should apply also these examinations. Profiles Definition To have a congruent evaluation with the profile, which includes several levels, a second phase is used to know the profile of the student. Although there was already defined one, the ITESM CQ has initiated a consult of constituencies in order to define a new, actualized one. The profile development is dynamically defined with the participation of the academies. This job intends to have the income and outcome profile with its respective methodologies of evaluation. This work was carried out on June. Training A very important aspect to increase the participation in the elaboration of items is the training. The third phase consists of receiving this training. We will focus primarily in academic issues and secondly in other levels. Items Elaboration The following phase consists of elaborating items by subject in a way that has the database by subject theme and subtheme. Implantation This phase will be relatively simple since the majority of the professors were working in the re-engineering of the teaching-learning process, where the training is not limited. We expect to have the first tests during the first semester of january-may, 2001.
Conclusions • •
•
EMC, EIC and EGEL examinations should convert to a term evaluation, even in an asynchronous adaptive evaluation with predictions. We must to learn better ways to evaluate abilities, attitudes and values and how to incorporate the evaluation as a methodology for developing critical thinking. To better diagnostic and define the profile we should apply to Fundamental Engineering Examination.
Sensibilization 0-7803-6424-4/00/$10.00 © 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference F3A-19
Session F3A •
Having automatic academic evaluation could increase professors' time for preparing courses, making research and, continuing education and consulting. To obtain a more accurate evaluation we should develop our own evaluation tool based on our campus environment. The Model of the Integral Profile (MPI) could be modeled as a bayesian network that generates knowledge through the academic program process and so we could finally measure the knowledge in terms of intellectual capital.
• •
References [1] [2] [3] [4]
De la Torre, R, "The Development And Evaluation Of A System For Computerized Adaptive Testing, " IOWA University, Vol. I, 1991, pp 2-7. Piket -May J., Chang L., Avery P., "Understanding What 'Success' Means in Assessment", FIE 98, vol. I, pp 20-22. McGourty J., "Strategies for Developing, Implementing, and Institutionalizing a Comprehensive Asessment Process for Engineering Education", FIE 98, vol I, pp 117-121. Tristán A., "Modelo de Calificación y Análisis por Computadora de Cuestionarios Referidos a Criterio ", III Foro Nacional de Evaluación Educativa 1998, pp 237-247.
[5] [6] [7] [8] [9]
ITESM, "Manual de Learning Space para Profesores, Diseño, Visión y Ación de la Práctica Directiva", 1999, pp 2. ITESM, "Annual Inform 1999" ITESM-CQ, "Carpeta de la Carrera", Intern Document, Jan 2000. ITESM-CQ, "Perfil de Ingreso y Egreso, Intern Document", Dec 1999. Kellogg S. "A Meta-Assessment Model for Industrial Engineering", FIE 99. Pp 11b1 -13.
Biography César Cárdenas is the Director of the Electronics School (IEC/ISE academic programs) and President of the Accreditation Committee at ITESM Querétaro Campus. He holds a Master Degree in Satellite Communications in France, a Summer Session Program from the International Space University in USA and currently a Master in Business Administration in México. Since 1998 he coordinates CENEVAL applications for some programs on campus. Prior to joining academy he worked for two years as a product engineer in a cable factory. He is Committee Consultative Member of EXPOCOMM Mexico and IEEE Member since 1998.
TABLE 3 GENERAL COMPARISONS IN CURRICULAR ISSUES OF EIC AND CENEVAL EVALUATIONS BETWEEN TWO ELECTRONIC ACADEMIC PROGRAMS CAREER
GLOBAL % DEO
EAN EDIG -
-
-
-
-
-
-
-
-
60.00 55.00
-
55.55 42.50 32.50 37.77
-
-
-
-
-
-
49.33
IEC-CQ 9E
38.40
-
41.67 44.44
-
48.15 45.83 20.83 33.33
-
-
-
-
-
-
31.11
IEC-CQ 9A
41.60
-
48.04 29.41
-
32.03 48.53 26.47 39.87
-
-
-
-
-
-
48.63
-
-
ACUS COMP INT*
-
-
-
SDIG ICO
49.20
-
-
SCOM SEO
41.00
-
-
TEL
IEC-CQ 8A
-
-
TCIR TCON MEL
IEC-CQ 8E
-
-
PSE
IEC-SYS 8E
45.50
-
-
-
-
-
-
-
-
IEC-SYS 8A
48.40
-
51.03 58.25
-
53.61 45.10 28.09 43.18
-
-
-
-
-
-
-
49.48
IEC-SYS 9E
44.20
-
47.18 49.04
-
51.70 47.73 23.14 34.80
-
-
-
-
-
-
44.79
IEC-SYS 9A
44.60
-
48.99 43.62
-
40.87 51.96 33.26 41.26
-
-
-
-
-
-
44.87
ISE-CQ 98E
38.40
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
ISE-CQ 99E
44.60
-
32.00
-
-
-
-
-
-
52.00
-
-
-
-
66.67 29.63
-
34.56
-
-
-
-
-
-
45.56
-
-
-
-
33.71 29.01
ISE-CQ 9A
35.30
EGEL 8A IEC
53.71
63.33 39.86 73.41 60.64 56.19 31.83 70.20 65.71 50.21 42.40 51.75 49.03
-
EGEL 8A CQ
55.75
63.73 40.98 75.23 53.05 58.85 36.03 67.87 57.14 50.21 42.40 55.51 49.03
-
EGEL 8A MX
42.16
52.33 33.89 60.40 30.40 46.77 32.93 49.60 41.76 39.53 39.56 39.61 39.52 39.49
EGEL 9A IEC
52.27
59.72 37.93 64.01 50.50 69.17 36.31 55.31 55.05 48.34 53.16 60.32 74.17 21.74
-
-
-
48.35
61.54 51.74 39.51 39.54
EGEL 9A CQ
51.07
60.26 38.24 66.39 45.08 48.92 35.99 56.14 48.83 48.34 54.50 53.53 74.17 21.74
56.04
-
EGEL 9A MX
41.68
48.14 31.28 52.92 31.98 42.95 30.66 43.21 43.69 43.70 43.78 43.82 43.81 43.78
43.42
-
ISE-SYS 98E
40.20
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
ISE-SYS 9E
44.30
-
43.01
-
-
-
-
-
-
49.78
-
-
-
-
48.87 36.44
-
48.69
-
-
-
-
-
-
ISE-SYS 9A
47.80
54.00
-
-
-
-
47.46 42.18
EGEL 8A ISE
58.80
63.33 42.64 77.97 41.67 62.85 42.34 64.39 44.28
-
-
58.01
-
-
61.54 59.78
EGEL 9A ISE
47.87
61.73 39.08 72.72 30.63 31.58 35.13 58.33 32.27
-
66.58 38.61
-
-
56.04
0-7803-6424-4/00/$10.00 © 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference F3A-20
-
