Using Rubrics for Monitoring and Evaluating Degree Projects in ...

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización.

Using Rubrics for Monitoring and Evaluating Degree Projects in Industrial Management Engineering Barbera-Ribera T1, Estelles-Miguel S2, Dema Perez C.M3, Garrigos-Simon F.J4. Abstract This paper presents an experience developed during three academic years in the Industrial Management Engineering, in the School of Design Engineering at the Universitat Politecnica de Valencia (UPV). The main aim has been to deepen on the role of the director of the Degree Project (PD), in order to improve the effectiveness and efficiency in carrying out this last step before the student can face the labor market and demonstrate its professional capabilities. To do this, we have modified the traditional methodology by introducing objective tools 1

Teresa Barberá Ribera ( e-mail: [email protected]) Grupo de Investigación e Innovación Educativa en Metodologías Activas para el Desarrollo y Evaluación de Competencias Genéricas Interpersonales (MACGI). Proyecto PIME : Innovaciones Tecnológicas en el aula: uso del Crowdsourcing. Dpto Organización de Empresas. Universitat Politècnica de Valencia (SPAIN). Edificio 7DCamino de Vera S/N 46022 Valencia 2

Sofía Estellés Miguel ( e-mail: [email protected]) Grupo de Investigación e Innovación Educativa en Metodologías Activas para el Desarrollo y Evaluación de Competencias Genéricas Interpersonales (MACGI). Proyecto PIME : Innovaciones Tecnológicas en el aula: uso del Crowdsourcing. Dpto Organización de Empresas. Universitat Politècnica de Valencia (SPAIN). Edificio 7DCamino de Vera S/N 46022 Valencia 3

Fernándo Garrigos Simón ( e-mail: [email protected]) Grupo de Investigación e Innovación Educativa en Metodologías Activas para el Desarrollo y Evaluación de Competencias Genéricas Interpersonales (MACGI). Proyecto PIME : Innovaciones Tecnológicas en el aula: uso del Crowdsourcing. Dpto Organización de Empresas. Universitat Politècnica de Valencia (SPAIN). Edificio 7DCamino de Vera S/N 46022 Valencia 4 José Onofre Montes Andrés( e-mail: [email protected]) Grupo de Investigación e Innovación Educativa en Metodologías Activas para el Desarrollo y Evaluación de Competencias Genéricas Interpersonales (MACGI). Proyecto PIME : Innovaciones Tecnológicas en el aula: uso del Crowdsourcing. Dpto Organización de Empresas. Universitat Politècnica de Valencia (SPAIN). Edificio 7DCamino de Vera S/N 46022 Valencia

929

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización.

(rubrics) that have been implemented over the entire duration of the DP, in order to evaluate the development and acquisition of competences, highlighting the transversal or generic ones. In addition, several questionnaires have been designed, with Liket scales and open-ended questions, with the purpose that students could assess competences, their evolution throughout the process, and also the rubrics in their own process of continuous improvement.

Keywords: Degree Project, Transversal Competences, Evaluation, Rubrics.

1 Introduction The DP, belonging to the oldest degrees of engineering, is actually the last subject after having already passed all the other subjects of the engineering degree. On the other hand, it has to be understood as a mainly personal work, corresponding to a type of experiential learning, where the project manager performs a formation that has important similarities with the called as “coaching”. In fact, the director’s mission is to lead the learning process, by supporting students and directing them to the sources of information. The director will be the one that will decide if DP is ready to present and will assign the qualification that deserves. In the field of Engineering Schools there is a long tradition in performing DPs. The “natural” professional activity of the engineer is “making projects”, that is, to conceive, design, implement, maintain and improve products or processes, giving to the term “Project” a meaning very specific in the Engineering context (Valderama 2008). Thus, in the Engineering field, it should be refined the sentence of RD 1393/2007 concerning the purpose of the DP. “DP will have between 6 and 30 credits, must take place in the curriculum final stage and it will be aimed at assessing the degree associated skills through performing a job that is similar to a real professional project”. DP is not the place to practice something for the first time, but it is the final test (Valderama 2008). Among scholars, the DP is seen as “the culmination of a degree program” (Todd, Banniser & Clegg 2004) or a “pinnacle” of undergraduate studies (Calvert and Casey 2004). From students’ point of view, it is the single most substantial and autonomous work of all those developed at the university (Webster, Pepper & Jenkins, 2000).

930

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización.

2 Design and Description of Experience

2.1 Contextualization The present educational innovation must be contextualized at the School of Design Engineering in the Universitat Politécnica de Valéncia. Specifically we focus on the Industrial Management Engineering (2nd cycle), and during the last three academic years. In total, 21 DC have implemented this methodology.

2.2 Objectives The objectives of this experience are as follows:  Find out the perception that students have about the introduction and use of the rubric in the preparation and monitoring of the DP.  Identify the difficulties observed in the implementation process of the rubrics.  Collect suggestions for an improved use of the rubric as a resource for learning, assessment and mentoring.  Determine the number of advisable tutorials to make a good direction of DP.  And finally, check whether there is a relation between the assessment given by the director/s of DP, the self-assessment done by the students, through the rubrics, and the mark assigned by the examining board of DP.

2.3 Tutorials DP carries associated a tutoring work, understood as an educational space for reflection where students can express their perceptions, thoughts, problems and goals, and the tutor can guide their process. Minimum numbers of tutorials were: Topic selection and choice of director and co-director, if necessary. Then, the Presentation Guide of the DP is given to the student. This guide is made by the authors of this paper. We provide and explain also in this tutorial the different rubrics to be used throughout the process. Presentation of the proposed objectives of the project and index. Generally this is an online tutorial. Submission, online or in person, of the different sections that the student will develop, and also of the rubrics that he/she is filling. These rubrics will be contrasted with those evaluated by the project manager. 931

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización.

Full presentation of the draft project of DP. At this time the rubrics are filled together, and the latest proposals are performed for necessary improvement. Final presentation of the DP Document and slides in PowerPoint (or similar program), which will be used for oral defense of the DP. The student gives a score to his document of DP and the director also qualifies it. Running of one or more trials or essays of the oral presentation of the DP to the director, who will do questions about it. Together with the student, we qualify the DP that is going to be presented and explained or defended.

2.4 Evaluation To define continuous evaluation process, we have taken into accounts those given by different authors. Some, like (Rullán et al. 2010), specify three time points: starting, follow-up and final. On the other hand, (Mateo et al. 2009) distinguish five phases, by incorporating a first choice and justification of the subject by students, and other where the student is asked to undertake a critical analysis of the learning process followed in preparing the DP. We have chosen the second proposal, by considering it the most suitable type for the DP that we conduct.

2.5 Evaluation Tools Used The main instruments used were: rubrics to assess transverse competences, questionnaires with Liker scales, and open questions.

2.5.1 Rubrics The rubric is a versatile tool that provides, on the one hand, of a reference point that gives him/her feedback on how to improve his/her work and, secondly, provides teachers with the opportunity to express their expectations about the learning objectives set (Mertler 2001; Roblyer & Wiencke 2003). Its use, according to (Kan 2007) permits the internalization of the evaluation criteria, promote activities under quality criteria, as well as a reflection on performance and errors. They enable students' self-assessment, being this process of the greatest importance for the student to manage, organize and edit his performance effectively (Zimmerman & Moylan 2009). Due to its versatility and teaching' potentiality, the assessment rubric allows conducting an authentic assessment (Blanco 2008). The generic competences evaluated are:  Ability to organize and work planning. 932

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización.

   

Capacity for analysis and synthesis. Ability to review and critisize the own work. Written communication skills. Oral presentation.

For instance, we present the latest versions of the rubrics used for assessment of the next competences that we have elaborate: Ability to organize and work planning (table 1) and oral presentation in the defense of the DP (table 2). Table 1 Rubric: Ability to organize and planning the DP Indicators

Evaluation of the director and other faculty / student self-assessment

The student is able to structure the stages of DP

1

2

3

4

5

The student is able to define the most appropriate methodology for developing the stages of the work.

1

2

3

4

5

The student is able to correctly temporalize phases of DP

1

2

3

4

5

The student analyze the degree of final fulfillment of the initial planning, the causes. deviations and their consequences

1

2

3

4

5

Table 2 Rubric: Oral presentation in the defense of the DP Criteria / Achievement level

1

2

3

4

5

Presentation Introduction Structured exposure of the content Posture, gestures, movements and eye contact Clarity and Fluency Volume, Intonation and Speed Use of technical language Audiovisual resources employed Use of the time allotted Conclusions Response to questions formulated

2.5.2 Questionnaire with Likert Scale and Open-ended Questions In order to know the student opinions about the degree of usefulness, satisfaction, difficulties and suitability in implementing the rubrics for the preparation of their DP, a short questionnaire with a Likert-type scales was designed (scale 1 to 5, with 1-Totally disagree and 5-Strongly agree). The questionnaire also included open-ended questions so that students could express their opinions. 933

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización. Table 3 Liker scale and open-ended questions to elicit the views of students on the usefulness of rubrics for the preparation of the DP 1. In the following statements, indicate your opinion by marking an X It was helpful to ...

1

2

3

4

5

Understand the criteria of valuation Develop accurate expectations about what was demanded Plan the preparing and presentation of the DP Guide the development of the project Assess the progress of my work Reduce my anxiety in the defense of the DP Confirm the level of competence acquired 2. What do you think about the fact of being assessed by rubrics? Justify your answer 3. About the format of rubrics used (, Was it comfortable for you? Justify your answer 4. What difficulties have you encountered in applying your self-assessment?. Justify your answer 5. Do you think it is suitable in the preparation and monitoring of your DP? Why? 6. Other comments and suggestions you want to suggest

3 Results The objectives achieved, according to the proposed, have been the following:  Find out the perception that students have about the introduction and usefulness of the rubric in the preparation and monitoring of the DP. Table 4 Liker scale and open-ended questions to elicit the views of students about DP rubrics. 1. In the following statements, indicate your opinion by marking an X This was helpful for….

Mean

Deviation

Understand the criteria of valuation

4

(0.80)

Develop accurate expectations about what was demanded

4

(0.83)

Plan the preparing and presentation of the DP Guide the development of the project

4,4 3.9

(0.61) (0.89)

Assess the progress of my work Reduce my anxiety in the defense of the DP

4.1 2.7

(0.75) (0,98)

Confirm the level of competence acquired

3,5

(0.79)

 Valuations are higher than the theoretical mean of the response scale (3.0) on all items except "reduce my anxiety in the defense of the DP" (mean 2.7). Students especially highlight its usefulness to "plan the preparing and presentation

934

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización.

of the DP "(mean 4.4)," assess the progress of my work "(mean 4.1)," and understand the valuation criteria "(mean 4). The main difficulties were: 29.4% reported the lack of time to self-evaluate themselves by rubrics, 27.93% reported having some difficulty for performing a critical analysis of their own work and 11.5% found little initial difficulties in understanding the rubrics. The suggestions proposed were: reducing the number of criteria (31.4%) and study the possibility of grouping some of them (21.5%). Most students (83.7%) believe that the optimal number of tutorials is 6. (82.5%) consider adequate a combination of face to face and online tutorials. The Table 5 reflects the scores assigned by the three evaluation agents (student, director/s and board) for each project, starting with the most recent projects.



  

Table 5 Punctuations or scores assigned by the three evaluation agents N er Student Director Tribunal N er Student Director Tribunal N er Student Director Tribunal 1

9

9

9

8

8,5

9

9,3

15

8,5

9

9,3

2 3

9 8,5

9,5 9

9 9,5

9 10

8 9

8,5 9,5

9 9,5

16 17

8,5 8,5

9 9,5

9 9,8

4 5

8,5 8,5

9 9

9 9,5

11 12

8 9

8 9

8,5 9

18 19

8,5 8,5

9 9

8,5 9

6 7

8,5 8,5

9 9

9,5 9,5

13 14

8,5 9

9 9,5

9 9,8

20 21

8 7,5

8 8

8,5 8.5

One can see that the self-evaluation scores of students, in most cases is slightly below that the ones granted by the director of the project and by the tribunal. This difference is larger in some of the projects better rated by the tribunal. For instance, in the case of the project with the highest rated by the court (9.8) is where there is a major difference with the self-assessed point of the student (8.5). The director of the project, except in one case, granted a score equal to or slightly less than that granted by the tribunal.

4 Conclusions The experience obtained with the use of the rubrics in the orientation and monitoring of the DP reveals an amount of conclusions about the goodness of this resource. First of all, we have demonstrated that it is a useful tool for providing feedback to students during tutorial sessions, as they can be provided with detailed information on its level of performance in the preparation of their DP. Moreover, the director of the project is provided with the opportunity to express its expectations about the learning objectives established. Secondly, the implementation has 935

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización.

revealed that, between rubrics and the tutorials, there is a bidirectional and interdependent relationship. Thirdly, one has to highlight the excellent perception showed by students in the use of evaluative rubrics. Apart from this, the instrument narrows and clarifies the general framework of the work performed. It also helps to the guidance and assistance in monitoring, and refines the conversations and divergences of students with their project manager. Moreover, we have corroborated that both, the students and the director of DP, coincide, to a large degree, on the assessments issued by the rubrics. There is also a close match with the assessment by the examination board of the DP. In addition, and according to the opinion of the students, it is considered as a very useful tool for evaluation, although they are conscious of the extra effort they assume with their fulfillment. We think that this experience is relevant, as it can be useful to any director who wishes to carry out an evaluation process of their students. In this sense, it enables being aware of their own learning process for the acquisition of transferable skills or competences throughout the development of the DP. Finally, we have to highlight the high relation between the scores assigned by students, directors, and the examination board of the DP. After this experience, we suggest its continuous implementation. In this vein we maintain our purpose of developing and refine our research in the development of the next DPs. We are conscious of the limitations of our work, mainly because of the reduced sample and methodology used to contrast our statements. However, the experience has shown us the high potentiality of the use of rubrics. A further step would be refining its development, extending this pilot experience, and improving this research with new methodology tools. We suggest also the improvement of its use, with the combination of this tool with new techniques, which could efficiently apply and implement the use of new technological advancements.

5 References Blanco, A. 2008, Las rúbricas, un instrumento útil en la evaluación de competencias. In L. Prieto (coord.). La enseñanza universitaria centrada en el aprendizaje: estrategias útiles para el profesorado (p. 171-188). Barcelona: Octaedro-ICE de la Universidad de Barcelona. Calvert, B. & Casey, B. 2004, Supporting and assessing dissertation and practical projects in media studies degrees: towards collaborative learning, Art Design and Communication in Higher Education 3(1), p. 47-60. Kan, A. 2007, An Alternative Method in the New Educational Program from the Point of Performance-based Assessment: Rubric Scoring Scales. Educational Sciences: Theory & Practice: 7 (1), p. 144-152. Mateo, J. et al. (coord.) 2009, Guía para la evaluación de competencias en el trabajo de fin de grado en el ámbito de las ciencias sociales y jurídicas. Barcelona: AQU Catalunya.

936

Book of Proceedings of the 7th International Conference on Industrial Engineering and Industrial Management - XVII Congreso de Ingeniería de Organización. Mertler, C.A. 2001, Designing scoring rubrics for your classroom. Practical Assessment, Research y Evaluation, 7 (25). http://pareonline.net/getvn.asp?v=7&n=25. Cited 15 Sep 2007. Real Decreto 1393/2007, de 29 de Octubre, por el que se establece la ordenación de las enseñanzas universitarias oficiales. BOE 29 de Octubre de 2007. Roblyer, M.D. & Wiencke W.R, 2003, Design and Use of a Rubric to Assess and Encourage Interactive Qualities In Distance Courses. American Journal of Distance Education 17(2): 77– 97. Rullán, M., Fernández-Rodríguez, M. Estapé, G. & Márquez, M.D. 2010, La evaluación de competencias transversales en la materia trabajos de fin de grado. Un estudio preliminar sobre la necesidad y oportunidad de establecer medios e instrumentos por ramas de conocimiento. Revista de Docencia Universitaria, 8(1), p. 74-100 Todd, M. Bannister, P. & Clegg, S, 2004, Independent inquiry and the undergraduate dissertation: perceptions and experiences of final-year social science students. Assessment and Evaluation in Higher Education 29(3), p. 335-355. Valderrama, E. (coord.), 2008 Guía para la Evaluación de las competencias en los Trabajos Fin de Estudios de las Ingenierías. Barcelona: AQU Catalunya. Webster, F., Pepper, D. & Jenkins, A. 2000, Assessing the undergraduate dissertation. Assessment & Evaluation in Higher Education 25(1), p. 71-80. Zimmerman, B.J. & Moylan, A.R., 2009 Self-regulation: Wherw metacognition and motivation intersect. In D. Hacker; J. Dunlosky, A. Graesser (Eds.) Metacognition in educational theory and practice (p. 299-315). Hillsdale, NJ: Erlbaum.

937