Proceedings TEEM’14
Second International Conference on Technological Ecosystem for Enhancing Multiculturality Salamanca, Spain October 1st - 3rd, 2014
Editor: Francisco José García-Peñalvo University of Salamanca
TEEM’14 is organized by the Research GRoup in InterAction and eLearning (GRIAL) and Research Institute for Educational Sciences (IUCE) at the University of Salamanca.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Designed by: Felicidad García Sánchez Research GRoup in InterAction and eLearning (GRIAL)
ISBN: 978-1-4503-2896-8
II
TEEM’14 is organized by the Research GRoup in InterAction and eLearning (GRIAL) and Research Institute for Educational Sciences (IUCE) at the University of Salamanca and is supported by:
III
Software As A Service (SAAS) in Learning Processes ..................................................................................... 393 The Cloud hovering over the virtual campus Marc Alier Forment; Enric Mayol; María José Casañ Guerrero ............................................................................... 395 The Effects of Massiveness on the Participation in Social Technologies: A MOOC in Secondary Education Daniel Amo Filvà; María José Casañ Guerrero; Marc Alier Forment ...................................................................... 397 Recommendations for the Design and Deployment of MOOCs: Insights about the MOOC Digital Education of the Future Deployed in MiríadaX Carlos Alario-Hoyos; Mar Pérez-Sanagustín; Carlos Delgado Kloos; Pedro J. Muñoz-Merino ............................... 403 TSUGI: A Framework for Building PHP-Based Learning Tools Nikolas Galanis; Marc Alier Forment; María José Casañ Guerrero; Enric Mayol; Charles Severance ................... 409 Atmospheric Physics Group Open Data (GFA Open Data): Meteorological Data and Tools for Learning Analytics Ángel Manuel Guerrero Higueras; Laura López Campano; Vicente Matellán Olivera; Andrés Merino Suances; José Luis Sánchez Gómez ...................................................................................................................................... 415 Evaluation in education and guidance ................................................................................................................ 421 Evaluation in Education and Guidance María-José Rodríguez-Conde; Susana Olmos-Migueláñez; Susana Nieto-Isidro ................................................... 423 A Proposal for Skill Evaluation Via Complex Tasks in Virtual Learning Environments Jaione Cubero Ibáñez; María Soledad Ibarra Saiz; Gregorio Rodríguez Gómez .................................................... 429 A Global Approach to Improve the Mathematical Level of Engineering Students Susana Nieto-Isidro; Higinio Ramos ........................................................................................................................ 435 Mathematics Competences Assessment at Lab sessions of Aerospace Engineering Freshmen Fernando Mínguez; Santiago Moll-López; José Antonio Moraño; M. Dolores Roselló; Luis Manuel Sánchez Ruiz 441 Educational Qualitative Assessment of Augmented Reality Models and Digital Sketching applied to Urban Planning Ernest Redondo; Francesc Valls; David Fonseca; Isidro Navarro; Sergi Villagrasa; Adriana Olivares; Alberto Peredo ..................................................................................................................................................................... 447 Reducing Student Risks in the Laboratory Protocol Assessment Ana Belén González Rogado; Ana María Vivar-Quintana; Isabel Revilla-Martín .................................................... 455 The Socioeconomic Index in the Analysis of Large-Scale Assessments: Case Study in Baja California (Mexico) Fernando Martínez-Abad; Alicia Alelí Chaparro Caso López; Luis Lizasoain Hernández ....................................... 461 Digital Competence’s Frameworks in Europe: an Approaching to Spanish and Norwegian Framework Ana Pérez-Escoda ................................................................................................................................................... 469 Selection of Key Standards to Create an Instrument for Information Literacy Assessment in Compulsory Secondary Education Marcos Bielba Calvo; Fernando Martínez-Abad; Mª Esperanza Herrera García .................................................... 475 Peer Tutoring at School with Migrant Students: Intercultural Mentoring Programme Paloma No-Gutiérrez; Mª José Rodríguez-Conde; Valentina Zangrando; Antonio M. Seoane-Pardo; Lorenzo Luatti483
XVIII
A 3roposal for 6kill (valuation 9ia &omplex 7asks in 9irtual /earning (nvironments Jaione Cubero Ibáñez
María Soledad Ibarra SDiz
Gregorio Rodríguez Gómez
Universidad de Cádiz Facultad de C. de la Educación. República Saharaui, s/n 11510 0034 956016481
Universidad de Cádiz Facultad de C. de la Educación. República Saharaui, s/n 11510 0034 956016481
Universidad de Cádiz Facultad de C. de la Educación. República Saharaui, s/n 11510 0034 956016481
[email protected]
[email protected]
[email protected]
This document is the presentation of an innovative proposal to evaluate the prior skill level of university students who are participating in the EDECOM-DevalSimWeb Training Program in “Evaluation and Development of Professional Skills in Higher Education”. Said Training Program, aims to develop, in final year students, five professional skills which are considered transversal to evaluation. It will done via their participation in e-assessment processes. In order to carry out a diagnostic assessment of the students’ skill level, the “EDECOM-DevalSimWeb Skill Level” virtual campus was designed. This is a space through which the students will be able to give a response to different situations/problems which are related to the selected skills. Their responses will be evaluated by external evaluators using e-rubrics so as to determine their initial skill level.
The EDECOM-DevalSimWeb blended-learning Program, contextualised in the DevalSimWeb Project1 , aims to train final year university students in e-assessment and to develop professional skills which are transversal to evaluation, in order to improve their learning. It is based on the idea that training students in assessment skills favours the self-regulation of their learning and therefore their academic success and employability. The five specific skills to which we refer are: analytical and critical thought, problem solving, sense of ethics, decision making and teamwork. The students will actively participate in eassessment processes via the use of the EvalCOMIX web service [6] and other electronic tools such as forums, wikis and simulation games, all via a virtual learning environment which combines “resources, interactivity, support and structured learning activities” [1].
Categories and Subject Descriptors
1.1 e-Assessment of skills
ABSTRACT
K.3.1 [Computers and Education]: Computer Uses in Education – Distance learning.
General Terms Measurement, Design, Standardization, Theory, Verification.
Keywords Professional skills, Higher Education, e-Assessment, e-Learning, Blended-Learning, Skill Evaluation.
1. INTRODUCTION The growing relevance of Information and Communication Technologies (ICT) in all aspects of society means that university education must adapt to new learning environments which are conditioned by virtualisation and digitalisation. “The use of eassessment in higher education is a relatively new education practice that has been more frequently studied in recent years” [10]. It is, however, already recognised as a fundamental part of elearning [14]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. TEEM '14, October 01 - 03 2014, Salamanca, Spain Copyright 2014 ACM 978-1-4503-2896-8/14/10...$15.00 http://dx.doi.org/10.1145/2669711.2669935
When we speak of a skill, we understand it as "a latent attribute, knowledge, attitude, ability, proficiency or faculty for the practice of a profession, a job or an academic task, carrying out the required labour or academic tasks and activities properly and correctly” [9]. We started with the idea that in order to evaluate the students’ skill level, we need an assessment system “based on the active participation of the students, allowing them to use their knowledge creatively to resolve real problems” [13]. Similarly, it can be argued that “a skill supposes the integration of a series of elements (knowledge, techniques, attitudes, procedures, values) which a person puts into use in a specific problematic situation, demonstrating that they are capable of resolving it” [8]. Bearing in mind these characteristics, when designing our skill evaluation we used the presentation of complex activities or problematic situations as a reference and, as indicated in [5], these must involve certain conditions such as:
1
-
The assessment requires the observation of dimensions and situations
-
Application to more than one situation to facilitate transference
-
Being close to reality (authentic)
-
Mobilisation of various resources, both internal (knowing, knowing how and knowing how to be) and external (requested autonomously by the student)
-
Utilisation of rigorous judgement and appraisal
The DevalSimWeb Project - Development of professional skills via participative evaluation and simulation using web tools. ALFA III (2011)-10. Financed by the European Commission.
Second International Conference on Technological Ecosystems for Enhancing Multiculturality - TEEM’14 -
Development indicators (levels of command)
-
Involvement of various agents in the assessment process (concordance)
Skills
Definition collaborates with other people in the achievement of common objectives, shares resources and knowledge, harmonises various interests, evaluates the opportunities and therefore contributes to a working environment which facilitates the implementation and development of tasks.
Given that we understand that a skill is an action which puts into play a series of elements such as knowledge, attitude, ability and proficiency, it is logical that the starting point of its evaluation should be “more or less real situations which exemplify in some way those which may found in reality”[15].
Table 2. Levels of skill development. [7]
1.1.1. e-rubrics In today’s technological world, “the e-rubric has been established as a useful instrument for carrying out an assessment process which is adapted to current educational needs and demands” [4], seeing as it “makes it possible to evaluate the degree of compliance of an attribute by offering the description of the requirements for each level” [9], thus determining the quality of the student’s performance [11]. This instrument is considered especially useful for “facilitating the qualification of the student’s performance in matters or subjects which are complex, imprecise or subjective” [3], i.e. it is suitable for evaluating the students’ responses when they are faced with complex tasks.
Level Everyday
Thoughtful (technical)
Comprehensi ve (inclusive and creative)
2. METHODOLOGY In this section, we briefly explain about the design of the initial skill evaluation for the participants in the EDECOMDevalSimWeb Training Program. Firstly, the designers of the proposal, various experts with experience in the design of skill development and assessment programs, started with the DevalSimWeb Catalogue of Transversal Evaluation Skills [7], a document which features the definition of the skills to be developed in the Training Program (see Table 1) and their various levels of development (see Table 2). The content of this catalogue was validated through project members, through experts in Evaluation Orientated at Learning and through various university students.
As previously mentioned, the basis of the skill assessment is the setting of complex tasks or problematic situations [5]. A complex activity was designed for each of the skills and the student must address and respond to the established questions. The nature of these tasks varies depending on the specific skill and they are based on videos, opinion articles, moral dilemmas, etc. As an instrument to certify the skill level, we suggest the use of rubrics, using the proposal for the assessment of generic skills in [12] as an example. Following this suggestion, an e-rubric of each of the five skills was designed using the EvalCOMIX web service, bearing in mind the various levels and facets described in the DevalSimWeb Skills Catalogue. Figure 1 shows an example of the rubric designed to assess ethical sense skill.
Table 1. Definition of the skills, according to the DevalsimWeb Catalogue of Skills. [7] Skills Analytical and critical thought
Problem solving
Sense of ethics
Decision making
Teamwork
Description Shows an intuitive knowledge and basic grasp of assessment, allowing the development of elementary and complex activities concerning the assessment actions to be carried out. Establishes analysis and connections between the actions, using procedures, strategies and resources which allow sensible explanations based on the obtained results. There is integration, application, creativity and critical thought in the decisions made concerning the actions and results of the evaluation; they are discussed and information is provided which is objective, reliable, useful and oriented towards improvement.
Definition Acts systematically, establishes hierarchies and uses tools for analysis; questions the situations which arise and discusses the pertinence of the judgements in order to make decisions within his/her role. Identifies, analyses, builds criteria and applies knowledge when facing specific situations, makes decisions and evaluates possible risks. Acts responsibly according to universal axiological principles and respects diverse cultural contexts. Acts opportunely, efficiently and efficiently, agilely and proactively; uses support from relevant information which facilitates the identification, understanding, evaluation and acceptance of risks in situations of uncertainty; chooses the best alternative via consultation with the most appropriate sources, comparing and contrasting them and integrating them.
It was proposed that an external evaluator should revise the students’ responses and use said responses to establish their initial skill level in each of the skills. Using these rubrics, the evaluator will make an inference of this level, seeing as the skill cannot be observed through this presentation of problems. We must not forget that all the steps in this assessment design are conditioned by its method, a completely virtual skill assessment. Therefore the entire assessment procedure was integrated into the “EDECOM-DevalSimWeb Skill Level” virtual space.
Makes responsible commitments and
430
A proposal for skill evaluation via complex tasks in virtual learning environments
Figure 1. e-rubric to evaluate the initial skill level of the Sense of ethics skill.
3. RESULTS Presented in this section is the proposal to evaluate the initial skill level of students participating in the EDECOM-DevalSimWeb Training Program and it specifically describes the “EDECOMDevalSimWeb Skill Level” virtual campus (see Figure 2), designed via the Moodle platform and shown in various screen captures. Figure 3. Screen capture of the explicated text in EDECOMDevalSimWeb Skill Level Virtual Campus. In the previous image it can be seen how an explicative text has been inserted to inform the participants that they must carry out five different types of activities which are related to the resolution of practical cases in which they will be provided with a situation or a case to resolve via the guided questions. Each activity will be submitted in pdf format via a “task submission” section on the same platform. At no point is the student told which skills are going to be assessed and each activity is only indentified with a name related to the presented case to which a response must be provided, as shown in Figure 4.
Figure 4. Screen capture of the 5 complex activities. Table 3 specifies, in order to better understand the proposal, the skill to be developed, the name of the activity and the resource through which the complex task is presented. Figure 2. Screen capture of the EDECOM-DevalSimWeb Skill Level Virtual Campus.
Table 3. Activity and resource used for each skill.
The virtual campus is fully oriented towards students and it contains all the information required for each participant to carry out the proposed activities autonomously.
431
Skill
Name of activity
Resource
Decision making
It’s time to decide
Video
Sense of ethics
What would you do?
Moral dilemma
Analysis and critical thought
Robot invasion!
Opinion article
Teamwork
What if we set up a restaurant?
Video
Second International Conference on Technological Ecosystems for Enhancing Multiculturality - TEEM’14 Skill
Name of activity
Resource
Problem solving
What’s the problem?
Written problem
development level once they have completed their training in the EDECOM-DevalSimWeb Training Program.
5. ACKNOWLEDGMENTS This paper has been possible thanks to DevalSimWeb project ‘Development of professional skills via participative EVALuation and SIMulation using Web tools’ (ALFA III (2011)-10). Financed by the European Commission.
All participants in the Training Program must provide a response to the five complex tasks as an obligatory requirement to be able to access the EDECOM-DevalSimWeb Training Program. The estimated time for the completion of the five tasks is two hours. However, bearing in mind the personal characteristics of each student and the difference between the times required for the completion of the tasks, a period of one week has been established so that they can freely access the activities at any time and give their responses when it is most convenient for them. This provides flexibility as it does not require that the assessment be carried out at one specific time.
6. REFERENCES [1] Belloch, C. Entornos virtuales de aprendizaje. Unidad de tecnología Educativa (UTE). Universidad de Valencia. Consultado el [15/06/2014] en http: http://www.uv.es/bellochc/pedagogia/EVA3.pdf [2] Boyle, A. & Hutchison.D. (2009). Sophisticated tasks in eassessment: what are they and what are the benefits? Assessment & Evaluation in Higher Education, 34:3, 305319. DOI= http:// dx.doi.org/10.1080/02602930801956034
As can be seen in Figure 5, each activity presents three unavoidable elements to assist the students’ comprehension: -
A short introduction
-
The material which must be watched or read (videos, articles, etc.)
-
The submission area, which details the specifics of how to respond to the activity.
[3] Buján, K., Rekalde,I., y Aramendi, P. (2011) Las rúbricas de evaluación en formación de competencias. En Bujan, K.; Rekalde, I. y Aramendi, P. (Coords.) La evaluación de competencias en la educación superior. (75-97). Sevilla: MAD, S.L. [4] De la Cruz Flores, G. y Abreu Hernández, L.F. (2014). Rúbricas y autorregulación: pautas para promover una cultura de la autonomía en la formación profesional terciaria. REDU: Revista de Docencia Universitaria, Número monográfico dedicado a Evaluación formativa mediante Erúbricas, (12), 31-48. [5] Fernández March, A. (2011) La evaluación orientada al aprendizaje en un modelo de formación por competencias en la enseñanza universitaria. En Bujan, K.; Rekalde, I. y Aramendi, P. (Coords.) La evaluación de competencias en la educación superior. (37-57). Sevilla: MAD, S.L.
Figure 5. Screen capture of one the complex activities on the VC
[6] Ibarra Sáiz, M.S. et al. (2010). EvalCOMIX en Moodle: un medio para favorecer la participación de los estudiantes en la e-Evaluación. RED, Revista de Educación a Distancia. Número especial dedicado a SPDECE 2010. 12 de noviembre de 2010. Consultado el [15/06/2014] en http: http://www.um.es/ead/red/24/Ibarra_Cabeza.pdf
Having submitted their responses to the five complex tasks, the students’ responsibility will be complete and the evaluation by the external expert will begin. The evaluator will review the students’ responses and, using the e-rubrics designed with EvalCOMIX, they will be able to determine the skill level (everyday, thoughtful or comprehensive) of each student in each of the facets of each skill.
[7] Ibarra, Sáiz. M., Rodríguez, Gómez. G. y García, Jiménez, E. (2013). Catálogo de competencias transversales a la evaluación- Devalsimweb. Proyecto ALFA III 2011- (10). [8] Poblete, M. y Villa, A. (2007). Aprendizaje Basado en Competencias. Una propuesta para la evaluación de competencias genéricas. Bilbao: Mensajero.
4. CONCLUSIONS This paper has provided a brief description of the methodological design for the evaluation of the prior skill level of students participating in the EDECOM-DevalSimWeb Training Program.
[9] Rodríguez, Gómez G. e Ibarra, Sáiz M.S. (Eds.) (2011). eEvaluación orientada al e-Aprendizaje estratégico en la Universidad. Madrid: Narcea.
This proposal was shared in order to provide new strategies to address skill assessment via virtual learning environments, which are appearing ever more frequently in a university setting. The evaluation questions and tasks used in e-assessment must evolve into sophisticated tasks [2] which are understood to be: media-rich stimulus materials which are interactive and complex - aspects which feature in the tasks designed in our proposal.
[10] Stödberg.U. (2012) A research review of e-assessment, Assessment & Evaluation in Higher Education, 37:5, 591604. DOI=http://dx.doi.org/10.1080/02602938.2011.557496 [11] Vera Vélez, L. (2008). La rúbrica y la lista de cotejo. Consultado el [15/06/2014] en: http://www.tecnoedu.net/lecturas/materiales/lectura10.pdf
Although only one model for the initial skill evaluation of the students was presented, the aim is to create a pre-test/post-test evaluation system via the resolution of complex tasks. The final goal is to be able to discover and compare the students’ final skill
[12] Villa, A y Poblete, M. (2011). Evaluación de competencias genéricas: principios, oportunidades y limitaciones. Bordón. Revista de Pedagogía.Vol. 63, 1: 147-170.
432
A proposal for skill evaluation via complex tasks in virtual learning environments [15] Zabala, A. y Arnau, L. (2008). IDEA CLAVE 11. Evaluar competencias es evaluar procesos en la resolución de situaciones problema. En: 11 Ideas clave: como aprender y enseñar competencias. Ed. Graó, 4ª reimpresión 2008. Barcelona España. ISBN: 978-84-7827-500-7
[13] Villardón, L. (2006). Evaluación del aprendizaje para promover el desarrollo de competencias. Educatio Siglo XXI, 24. Disponible en: http://revistas.um.es/educatio/article/view/153 [14] Whitelock, D. (2009), Editorial: e-assessment: developing new dialogues for the digital age. British Journal of Educational Technology, 40: 199–202. DOI= 10.1111/j.1467-8535.2008.00932.x
433
