Rethinking engineering education practices with

Rethinking engineering education practices with active learning and digital technologies: Pilot experiences at the University of the Andes, Bogotá Alvaro H. Galvis University of los Andes, Bogotá, Colombia [email protected]

Francisco A. Galvis, Juan P. Casas University of los Andes, Bogotá, Colombia [email protected] [email protected]

Diego M. Valencia, Angélica Avalo, Alexa T. Jiménez University of los Andes, Bogotá, Colombia

[email protected] [email protected] [email protected]

Abstract. The Dean of the Faculty of Engineering invited professors to explore the use of pedagogies aimed at transforming their practice by making active use of Information and Communication Technologies (ICT)supported learning strategies. Three courses and their respective professors were chosen to participate in the innovation process. With the support of pedagogical and technological advisors, the professors redesigned, tested, implemented and researched the effects of this change on the actors. Studies were conducted to answer questions related to possible lessons from this pilot experience and the lessons learned will be used to create competencies-based academic programs led by principles of quality, innovation, flexibility, internationalization and sustainability. This paper presents the rationale and design of the pilot experiences, the studies that were conducted about the intervention, findings, conclusions, and recommendations.

Introduction Engineering Education is a field of practice where using digital technologies is a must. Engineers must master both their domain of knowledge as well as the use of up-to-date technologies –most of them digital—to be competitive in their professional lives. In spite of this, teaching engineering is not an endeavour that is always enriched by technologybased pedagogies (Forté & Wentland, 1998). Most traditional engineering faculty members teach as they were taught, reinforcing the idea that the professor’s role is to transfer relevant mental models to help students understand and solve problems with engineering methods and tools. This approach does not prepare them for today’s engineering field where professionals deal with incomplete data and competing (often conflicting) demands from clients, government, environmental groups and the general public (Mills & Treaagust, 2003). In spite of this, non-traditional teaching ideas have had relatively little impact on mainstream engineering education so far (Bates, 2015).

To break with traditional teaching practices in engineering at our University of the Andes (or UniAndes), we undertook a two-year pilot project in which three engineering courses belonging to different engineering programs became laboratories for the integration of new technology-based innovations and active pedagogy. A multidisciplinary team from the Center for Innovation in Technology and Education (ConectaTE) coached faculty members as they evaluated their teaching practices and reflected on the impact of different aspects of the intervention. This paper presents initial results and lessons from the pilot experiences.

Context of the study The Engineering portal of the University of the Andes (UniAndes) proudly states that since its foundation in 1948, it has been a national leader in training engineers and researchers at the forefront of the discipline. It also states that its undergraduate, masters and doctoral programs are based on the best international practices of the discipline, which focus not only on the acquisition of technological competencies, but also on the development of soft skills such as communication, leadership, management and interaction with interdisciplinary teams (UniAndes, Faculty of Engineering, 2017). This characterization describes the level of academic leadership that the university intends to maintain. As is stated in objective 1.4 of the institution’s 2016-2020 Development Plan, the university will continue “to have academic programs based on the development of competencies that are evaluated comprehensively and systematically against international standards and criteria such as quality, innovation, flexibility, interdisciplinarity, internationalization and sustainability” (UniAndes, 2015). Within this inspiring framework, the Dean of the Faculty of Engineering made a call for explorations in the use of transformative pedagogies, namely active teaching strategies supported by digital technologies. The targets of these interventions were very representative courses of those in engineering curriculum, and the participants were professors who were highly distinguished for teaching in their discipline. Three courses, Probability and Statistics (PS), Structural Systems Analysis (SSA), and Rigid Solid Mechanics (RSM), and their respective professors, were chosen to participate in a twoyear pilot experience. The courses are described below. Probability and Statistics (PS) Probability and Statistics is an undergraduate course in the area of the fundamentals of engineering offered by the Department of Industrial Engineering to all students of the faculty. While the course is taught in fourth semester for Industrial Engineering majors, in the rest of the engineering programs, it is taught either in the third or fifth semester. It is a course with high demand every semester, which is why it has at least ten sections with an enrollment of 40 to 100 students per section. The course has one coordinator and the sections are taught by a variety of professors. PS is a three-credit course traditionally taught in two hour and twenty-minute lecture sessions, complemented by independent assignments equivalent to six hours of work per week. The course director, Mario Castillo, and one of the professors, Astrid Bernal, took part in the pilot experience. They were each in charge of one section that met for three-hours a week. The Probability and Statistics course had thirteen topics. Now it consists of two core modules that integrate the two themes: probability and statistics. The intervention was piloted and the flipped learning strategies were implemented. Before class, students were asked to study the subject and do preparatory exercises; during class, their understanding of digital material seen before class was evaluated, and feedback and

more practice was given. After class, it was suggested that students attend optional supplementary classes where additional practice of concepts and feedback was provided. The remaining sections did not undergo any intervention. Tensions at the beginning of the redesign: (1) Curricular tensions: Because the course integrates two thematic areas, and each one is broad, this affects how deep one can go into the content. (2) Instructional tensions: Lecture classes with many students do not necessarily favor active participation. (3) Contextual tensions: Each section has students from different engineering programs, with differences in interests, previous training, and maturity. Given these conditions, the examples and the exercises do not necessarily address the needs of each student. (4) Reorientation tensions: Homework exercises are long, so by the time students receive feedback, the class has often moved on to another topic. This does not allow students to reflect on their mistakes nor to try new comprehension / study strategies. Areas of redesign: (1) Curricular area: The course structure was rearranged around four fundamental concepts that meet the requirements of basic engineering training, instead of the original fifteen topics. Redefined learning goals include: (a) recognize and understand events and phenomena of a probabilistic nature; b) acquire skills to construct probabilistic models to quantify risk; c) manage and analyze data to identify patterns and make inferences; and d) build basic statistical models. (2) Instructional, contextual and reorientation areas: Active pedagogy, through the use of the flipped classroom and a variety of short and focused exercises, was used. Operationally, this approach requires three complementary efforts: (a) Students must study topics and test what they learned before the face-to-face session. This required the creation of video clips to complement the printed material of each learning unit, for example https://youtu.be/A3PC1E4nZoY ; (b) During classroom sessions, the idea was to complement what was learned in video clips, books and previous exercises, through peer evaluation of answers to practice problems. Additionally, the classroom sessions were used for professorfacilitated group discussion and feedback on exercises that students completed on their phones or computers within an immediate response and feedback tool; (c) In after-class activities, the student had the chance to practice with exercises situated in the different areas of engineering and receive feedback in breakout sessions. Structural Systems Analysis (SSA) SSA is a compulsory undergraduate course in Civil Engineering taken predominantly by students of this program. It is part of the general fundamentals of engineering curriculum and is taught in the fifth semester. It is the first course that offers the student a real context in which to apply skills in the field. The main objective of SSA is for students to interpret reality from the perspective of civil engineering. Before the intervention process started, the SSA course had six thematic units and twenty-nine theoretical sessions. The course is now divided into two main modules that correspond to the general concepts about civil structures and methods of analysis. The director of the course, Juan Carlos Reyes, and a professor of the course, Francisco Galvis, took part in the pilot study, where blended learning, a flipped classroom approach, and active pedagogy strategies

complemented with ICT were put into practice. Multimedia materials (video clips with study guides) were created in keeping with this approach. Total class time of some learning units was reduced, and virtual laboratories replaced the sap2000 face-to-face laboratories. The complementary classes were exchanged for optional office hours. This represented a total reduction of 30% in face-to-face sessions compared to the traditional course. The redesign was implemented as materials and activities were developed. Tensions at the beginning of the redesign: (1) Since the course has broad thematic content, it is not possible to allocate class time for the students to analyze and solve problems with the guidance of the teacher. These tasks, which are central to the course, should be done outside of class. (2) Experience shows that students approach the problems from mathematical operations perspective, disregarding the analysis of the results they obtain in their calculations, which makes mathematics an ends, not a means, to solving real world problems. (3) Many students do not master the prerequisite subjects of the course making it necessary to add such thematic content. Areas of redesign: (1) To provide digital, contextualized content to complement course materials and enable the student to explore, understand and apply concepts in a real context before the class sessions, and to review and reinforce previous knowledge about concepts through this channel as well (Example: https://youtu.be/PTUWj4Nznjk ). (2) To focus not only on the analysis of the problem and the mathematical solution to it, but to concentrate on the interpretation of the results of that solution through rubrics and peer evaluation. (3) To create alternatives for how to approach the theoretical concepts of the course so that the student can apply them in the context of civil engineering. This includes the systematization of exercises administered by the computer and accompaniment of the professor in the analysis and discussion of solutions. Rigid Solids Mechanics (RSM) RSM is the first of a series of three compulsory courses in the mechanical systems area of the undergraduate Mechanical Engineering program; however, the concepts taught are transversal to all areas of the Engineering program. The course focuses on the basic concepts of applied mechanics in engineering and is oriented towards the achievement of a central objective, which is the interpretation of reality in the frame of mechanical and civil engineering (Casas & Leon, 2016). The RSM course was composed of ten units which were distributed into sixteen weeks of class. Currently, the course has four main modules that coherently articulate the course contents. It includes two mandatory lectures per week and an optional supplementary session. The former is by given by professor Juan Pablo Casas and the latter by graduate assistants. Tensions at the beginning of the redesign: (1) The enrollment and performance statistics of the course were lower than the rest of the Mechanical Engineering Department's courses. In the four years prior to the redesign, the percentage of withdrawals was higher than those of the department's other courses, and the average grades were also lower (Casas & Leon, 2016, p. 6).

(2) The lecture class sessions used conventional pedagogy and were centered on the teacher’s explanations and demonstrations; theory predominated over practice. (3) Feedback on what was learned was deferred over time as it depended on grading, so the focus of the classroom sessions was not on the learning problems that arose during the process; rather, feedback came after the fact. (4) The learning resources were closely linked to what the texts said and to the teachers’ presentations in class. Areas of redesign: (1) To use active pedagogy through a flipped classroom approach making sure students prepare for and participate in the class by using exercises that have the student face real situations that transcend the theoretical context. (2) To promote practice by using exercises with increasing levels of difficulty (three levels: textbook-related situations; simplified, real situations; complex real situations) which receive timely feedback about students’ degree of understanding. The information serves as a basis for developing new strategies to reinforce the proposed learning goals. In order for this to happen, weekly solutions to a set of problems that represent the level of difficulty that the student must be able to handle are delivered and revised. (3) To create multimedia resources (Video clips with study guides such as https://youtu.be/ZpBFBMbeZPk ) that the student can access repeatedly to see how the thematic expert interprets the situation and proposes ways of addressing it using what was learned in each learning unit.

Literature review In the literature review that follows, we seek to clarify concepts that guide the design of each of the pilot experiences. Engineering Education Studies about engineering education indicate that “the task of higher education is to educate students to become effective modern engineers—able to participate and eventually to lead in aspects of conceiving, designing, implementing, and operating systems, products, processes, and projects. To do this, students must be technically experts, socially responsible, and inclined to innovate” (Crawley, Malmqvist, Östlund, Brodeur, & Edström, 2014, p. 2). The same authors also point out that “there is a seemingly irreconcilable tension between two positions in engineering education. On one hand, there is the need to convey the ever-increasing body of technical knowledge that graduating students must master. On the other hand, there is growing acknowledgement that engineers must possess a wide array of personal and interpersonal skills, as well as product, process, and system building knowledge and skills required to function on real teams to produce real products and systems” (2014, p.3). The previous tension is present in each of the engineering courses. In some cases, PS for example, concepts that will be useful throughout one’s career and life may not be clearly connected to a specific discipline. This means that achieving competency in this course demands going beyond understanding each concept and method to being able to apply these in diverse contexts. In cases such as the RSM or SSA courses, concepts have empirical/concrete references that make it possible to grapple with real-life

problems associated with them and whose solutions require higher level thinking (i.e. analysis, evaluation and synthesis in Bloom’s taxonomy) and the use of interpersonal skills to search for, test and socialize valid solutions (Churches, 2009). Engineering Education Course Design In the search for anchoring principles and methods for the design of engineering courses, we discovered the following frames of reference: 1) design courses considering the students’ needs; 2) use curricular objectives suggested in the study guides; 3) base the design on thematic frameworks proposed by disciplinary authorities; 4) follow processes or sequences for solving problems; 5) use disciplinary norms or standards as a guide (Ziegenfuss, 2007). The aforementioned practices are not exclusive, as is the case with curricular tensions. That said, we decided to begin designing from a multidimensional needs analysis that combines the Engineering Education frameworks with the students’ perceptions; this marriage of perspectives provides the framework for the course redesign (Galvis & Pedraza, 2013). It was decided to use non-traditional approach for the design. According to Hansen (2011, pp. 17-18), course design usually focuses on teaching, aligning the content of the course with the needs it must satisfy. Afterwards, the evaluation criteria and tests are defined and designed. Finally, the learning objectives that guide the treatment of the topics that the instructor will present are defined in a linear and logical sequence. In contrast, in the design of courses focused on learning, after clarifying the needs that the course must attend, everything that the student should learn in order to achieve the objectives is detailed, and a plan to promote deep understanding of concepts in a nonlinear way is designed. The students are guided by essential questions whose answers must be found in groups through experimentation with content, rather than by simply memorizing concepts. This distinction between the logic of teaching and the logic of learning is the basis for design for understanding (Wiggins & McTighe, 2005) and can be implemented with the help of what is known as the "backward design model" (Erickson, 2007). Following this systematic model and proposals for understanding big ideas (Wiggins & McTighe, 2005 and Hansen, 2011 adapted by Galvis & Pedraza, 2013 for active learning supported with digital technologies), the first step is to determine learning outcomes that students should be able to demonstrate, in terms of big ideas, also called fundamental concepts, they should understand in depth, that is, not only outcomes based in curricular content. A detailed "what to learn" design is made, taking into account the concepts that students are supposed to have mastered and, in particular, the misconceptions they usually have. Guiding questions are used as bridges for inquiry and collaboration and aim at filling conceptual gaps, which are the difference between what should be and what is in the student’s mind. Only then is the assessment system defined, including both authentic performance tests for measuring observable comprehensive results and ongoing formative assessments during the process to measure strengthen the understanding of complementary concepts. Learning units are organized by big idea and give the learner control over how to approach the themes. This follows the principles of active pedagogy and is supported by the flexibility that virtual or blended learning environments provide. The student can also participate in learning experiences that only virtual learning and collaborative learning environments offer.

Conditions for Efficient and Effective Courses Redesign Professors from research universities such as UniAndes are usually focused on their teaching and doing disciplinary research, either by themselves, or with support from graduate assistants. Thus, several questions arise: Why should they spend time redesigning courses that are well-structured? Why should they reform courses that, according to the typical retention and passing rates, are successful in developing the expected competencies? Why do such a thing when the courses are well-evaluated by students? These questions were present when the dean opened the call to participate in REDINGE1-Engineering Courses Redesign 1 project. The objective of the pilot was to learn about a paradigm focused on fostering learning and not just teaching (Bates, 2015), to learn about the different means to learn (Forté & Wentland, 1998), to complement the lecture class, books, articles and laboratories with virtual learning environments implemented with digital learning tools that provide flexibility in accessing knowledge, and to foster changes in the professor's traditional role, as a facilitator of the process (Collison, Elbaum, Haavind, & Tinker, 2002). This promise of value would not have been possible had there not been conditions for change that favored its materialization, such as those studied by Galvis (2016). The dean of the Faculty of Engineering was the champion of the REDINGE1 process. He made resources to the departments available such as reducing the course load of participating. He also provided funds to sponsor the support of the multidisciplinary team from the Center for Innovation in Technology and Education (ConectaTE) of the Faculty of Education during the innovation. This made the process viable, but it was not enough. The university issued a new faculty handbook which recognized the importance of innovation in teaching as an academic outcome related to progress in one’s professional trajectory (UniAndes, General Secretariat, 2015, p. 19). This is precisely what the pilot aims to recognize, and this change in the handbook made this pilot initiative acquire academic value for the professors, which is something that motivates them and generates the possibility for them to create learning resources making the value of their teaching experience evident. The innovation initiative was also made possible by the experience of ConectaTE in providing accompaniment to innovative teachers, in creating Virtual Learning Environments (VLEs) and Virtual Learning Objects (VLOs), developing software, videos and websites in particular (UniAndes, ConectaTE, 2016).

Research Questions Considering that these pilot experiences aim at transforming engineering teaching, it was considered appropriate to try to answer this question: What can the Faculty of Engineering learn about the development and implementation of pedagogical innovations supported by digital technologies (DT) to achieve what is proposed in Objective 1.4 of the Academic Leadership Plan of the university? To answer this question, the following sub questions were formulated: From the students’ perspective: 1. What effects on the retention and results of the students of the redesigned courses is attributable to the transformations of educational conceptions, media and practices in the courses? 2. What changes in students’ perceptions are attributable to the processes and products of the innovation process?

From the professors’ perspective: 3. What changes in the conceptions, means and educational practices of innovating teachers can be attributed to their participation in the redesigns and its implementation? What lessons does the implementation of active pedagogy leave for the innovative professors? Considering the interactions in the redesign: 4. What are the key success factors in these innovation processes from the perspective of the professors and sponsors of the transformation process?

Research Design To answer the questions related to learning outcomes, the research possibilities in each of the courses with interventions were taken into account considering the extent to which the innovation was implemented in each semester of the redesign. In 2015-1 and 20152, interventions were implemented in one or more modules of each course according to the stage of the redesign of activities or materials. In 2016-1, all modules from each course were involved. In all three courses, attention was paid to content validity in the performance tests, ensuring coherence between the learning objectives that were proposed and taught and the grades reflecting achievement levels. In the PS course, the final exam was standardized, so it was possible to collect data for two semesters. A total of 882 undergraduate students from the different majors registered in the ten sections participating during 2015-2 and 2016-1, 127 of whom were enrolled in the sections with the intervention. Results were categorized by major, entry level of the student, and whether or not the student was repeating the course. The students’ cumulative average and grades in pre-requisite math courses were also considered (UniAndes, Faculty of Education Evaluation Center, 2016). We controlled for variability in the results by tracking record of students enrolled and withdrawn from the three courses over time. Regarding students’ perceptions of changes in their learning autonomy due to the innovation, soft skills were measured. We applied a survey that measures academic self-efficacy, methodological flexibility, support of autonomy, support of ICT in the teaching-learning process, and self-regulated learning whose scales show good internal validity. See Table 1 Table 1. Internal validity Instrument Academic self-efficacy Methodological flexibility Support de autonomy Support of ICT in the T-L process Self-regulated learning (SRL) Planning Self-monitoring Assessment Reflection

No. items 4 4 4 4 19 5 5 4 5

Alfa 0.83 0.89 0.85 0.87 0.88 0.75 0.76 0.68 0.83

Source: (Uniandes, Centro de Evaluación de la Facultad de Educación, 2016, pág. 9)

The constructs in this survey were chosen because they build on research that shows that when the learning environment is autonomous, students’ motivation and selfregulation are strengthened. This translates into greater interest and confidence, and thus, increased performance, persistence, creativity, and general well-being of students (Deci & Ryan, 2000). A quasi-experimental design was used in which control and experimental groups were compared. A pre-test and post-test measured students’ ability to self-regulate their learning and their academic self-efficacy. Students in five sections of the three courses (two PS, one RSM and two SSA sections) and twelve non-intervened sections (eight of the PS course and four of the RSM and SSA related courses) participated. In addition, three variables related to the pedagogical intervention were explored: a) promotion of autonomy; b) methodological flexibility; c) support of ICT in the teaching-learning process (UniAndes, School of Education Evaluation Center, 2017, p. 6). To understand students’ perceptions of the innovations in their courses, in 2016-1, 110 individuals from the intervention sections (14 of the MSR course, 65 from the SSA course, and 31 from the PS course) took an online survey. The survey asked participants to rate the quality of the resources, the feasibility of the activities, the contribution of the methodology to promoting interaction in the lecture classes, and the overall impact on learning. Information from the surveys was complemented with information from three focus groups, two of which had RSM students and one with SSA students with eight, nine and seven students respectively. Interviews were carried out with the professors implementing the intervention in their courses. The objective here was to understand: i) the changes generated in their teaching by the accompaniment they received; ii) the effects the accompaniment had on the redesigned courses; iii) and finally, the impact of the support in the other courses they are teaching. These interviews were transcribed, analyzed, categorized and the responses were compared. The perceptions described complemented those written by some of the teachers in their reports at the end of the pilot. Online surveys were also applied to administrators in the Faculty of Engineering and to the professors innovating in their courses to find answers to the fourth sub-question.

Findings Cognitive outcomes for students The findings will be discussed for each course separately. In the PS course show that active learning and flipped pedagogy supported by digital technologies had a positive effect on performance in the final exam of the class. Scores were approximately a quarter of a standard deviation higher when other academic variables of interest (e.g. semester, repetition, GPA, and grades of two previous courses) were controlled for (UniAndes, Faculty of Education Evaluation Center, 2016, p. 4). In the SSA course, it was not possible to have a control group since both of the two sections received the intervention. However, a historical review of the course grades in iterations taught by the same professor before and after the innovation was conducted. The percentage of failures in the course during the last twelve semesters was also

analyzed. Before the innovation, the failure rates of the SSA course where around 10%. In the first semester of implementation of the innovation, that is, when the course transitioned from conventional to active pedagogy, it showed the highest historical failure rate (20%). It was detected that the workload was very high during this period. Subsequently, the size of the tasks was reduced, and support material was continuously reduced; this led the percentage of failures to decrease by half in the most recent cohort (Reyes & Galvis, 2017, p.4). Reyes & Galvis (2017, p. 5) comment that one of the problems identified in the implementation of the methodology is the professor’s misunderstanding of the students’ comprehension of the video content before class. This essentially meant that in the first version of the course after the innovation, more than half of the class time was spent reviewing the content of the video. This practice resulted in a loss of motivation for many students, which was evident due to low attendance (close to only 50%). To overcome this problem, virtual quizzes which evaluated students’ understanding of the videos were given in each class. This assessment practice allowed the professor to minimize the time spent reviewing and at once improve attendance. In the RSM course, it was also impossible to have a control group since there was only one course per semester. Thus, students’ opinions and achievement scores from 2015-2 were gathered; this was during the semester of the first implementation in which materials were still in production. In 2016-1, the semester of the full implementation, the status of these variables was verified. Per a report by Casas and León (2016), there was a significant improvement between the two implementations. This was due to the fact that the second implementation corrected defects in the first and that students were more comfortable with the active, flipped pedagogy in the second iteration, is shown in the following figure:

Figure 1. Student grades for some RSM activities on a 0.0 to 5.0 grading scale

When contrasting the achievement indicators obtained in the RSM course with the other mechanical engineering courses over time, Casas and León (2016, pp. 20-23) found that: • Although the percentage of failures in RSM was still higher than those of mechanical engineering courses, there was a tendency for retention to improve as the innovation progressed. • Average RSM scores were still below the average grades of the Mechanical Engineering Department, but there was a tendency to improve as the innovation progressed. • The percentage of withdrawals also continued to have a decreasing trend.

•

The final grade of the course depended on the class work grade: in 2015-2, 77% of the people who passed class work passed the course, and the same happened with failing grades. In 2016-1 this happened for 80% of the cases.

Casas and León (2016, p.23) concluded that the active learning methodology used before and during the class allowed the student to take greater responsibility for their learning process and also to organize their study time more effectively. It permitted the professor to have an idea of the problems that the students have before each class, which helped them to orient the class in such a way that they dealt only with the issues that caused the students the most difficulty. On the other hand, students received constant feedback that informed them of their strengths and weaknesses. Non-cognitive student outcomes The results of the soft skills study show the following (UniAndes, Faculty of Education Evaluation Center, 2017, pp. 19-20): • There were no clear significant differences in self-regulation of learning in the sections with the intervention. • Significant differences in academic self-efficacy cannot be attributed to intervention. • There were no significant differences in the promotion of autonomy in the sections with the intervention. • The greater methodological flexibility in the sections with the intervention compared to the non-intervened sections is statistically significant. • Students’ positive perceptions of the support of the ICTs to the teaching and learning process were significantly more positive in the group with the intervention. • Of the three courses, the students perceived that the one that was the most flexible and most supported by the ICTs was the Structural Systems Analysis course. To interpret these results, it is important to bear in mind that the development of soft skills can be affected not only by the intervention, but also by other factors related to the learner. As the university puts it, the institution “has students who, within an holistic, interdisciplinary and flexible learning environment, are the main agents in their educational processes” (UniAndes, 2015). When inquiring in 2016-1 about students’ perceptions of the active methodology (blended learning in SSA, flipped classroom in RSM and PS, all supported by the use of video clips and study guides), the following was found (Valencia, 2017): •

•

In RSM, all respondents considered the learning process to be good or excellent. In SSA, the great majority (87%) shared this opinion, while the remainder rated the process as regular or poor. In PS a good majority (64%) considered the methodology to be good or excellent. In the focus groups, it was determined that the length of the class session affected the differences in perception. In terms of class preparation activities, in the three courses the vast majority agreed that videos and pre-class exercises were important (100% in RSM, 93% in SSA, 90% in PS). The focus groups confirmed the positive rating that video clips received as well as rating of the importance of doing the preparatory exercises.

•

As for the interaction in the lecture classes, the following table shows the very positive evaluation of the active pedagogy approach in class in general, though with a lower score for the PS group, likely due to the length of the sessions in this course.

Table 2. Student’s opinions about active methodology in class

Changes for professors Professors were interviewed about the changes and lesson that came from their participation in course redesigns. Specifically, they reflected on the changes in conceptions, in media they use to teach, and in the educational practice transformations attributable to their participation. Below are some testimonies, which help to understand certain effects of the innovation piloted. Changes in professor´s conceptions I do not think that I have changed the concept that I have of what pedagogy is or how people learn, but what I see is that the process occurs with a different dynamic. The same steps, but with a slightly faster dynamic and I think better, because [the student] does not wait until the last day to try to study everything and sit down to do a test (Reyes, 2016). In my case I do not think it is a change in me as a teacher as such, or in my teaching style, but in the relationship of the teacher with the student, especially in class, during class ... I have always emphasized application in my courses. That is not new for me, and I have also insisted a lot on bringing the abstract concepts down to earth, but now I communicate them better (Castillo, 2017). In my regular classes I try to do things throughout the class (...) so that [the students] are connected. So, I ask questions (...) In the innovation course, this definitely happens because students must work during the whole class (Bernal, 2017). I have just become a teacher. This is my second semester of full-time teaching. I did not really have a previous experience of traditional classes. Rather, what I had was the

opportunity to do this course blended and another course not blended. That’s where I have been able to contrast things (Galvis, 2016).

Changes in media for teaching Without the resources of this project, it was impossible to think that the student could come to class with content knowledge, because there is no textbook or guide [for SSA]. One had to write the book first. In this case, what has been done is to write the book in video form, so the student watches the video, which is a little more pleasant than reading the book (Reyes, 2016). Both the opportunity and the need to make the videos themselves is a new experience, and it is very interesting because we thought it was not so easy to reach the students with a probability video taught by a university professor. We have found that, in general, they really appreciate the videos, which they think are of good quality (Castillo, 2017). The real situations used as examples during the course motivated the students and increased their interest in the course. These examples allow the student to understand the usefulness of the concepts he/she is learning and also to create a valuable link between theory and practice (Casas & Leon, 2016, p. 23).

Changes in educational practices I feel that the role of the teacher in the three moments of the flipped class [before, during, after] is more valuable, makes more sense... When I'm reviewing a quiz in class and asking why they did something, or elaborating on some content they have already looked at ... one feels that that time is more useful than if one were lecturing for two and a half hours, which is what often happens. One really does see students are more focused on the activity one is doing (Castillo, 2017). The redesigned version of the PS course makes the process of guiding them on the topic easier because the methodology allows them to participate actively in an hour and twenty-minute-long workshop, and because they have a quiz in which they have to think and I, after, I can tell them what they got wrong and they can raise their hands and say, I still don’t understand (Bernal, 2017). The new methodology creates a challenge for both the student and the teacher. For the student, it creates the challenge of assuming a great responsibility for his or her preparation, and for the teacher it creates the challenge of taking a different role in which he is not the owner of knowledge, but rather a guide during the process (Casas & Leon, 2016, p.23)

The experience of the instructors involved in the innovations is broad. The youngest instructor was in charge of one section of the SSA course; however, academic results from the students of his sections was equivalent to the performance of the students of the other section, suggesting that the skill of the instructor was not a factor in the outcomes of the redesign. Outcomes of interactions in the redesign process We surveyed administrators in the Faculty of Engineering and the professors who took part in the pilot about what must be done well to succeed in an initiative like this. We also asked about the things that should also not be done, or the key success factors in an educational innovation supported by ICT. The administration was very generous in sharing their ideas which are presented in the following synthesis.

Table 3. Critical success factors stated by administrators and professors

Category

Administrators’ Opinions

Professors’ Opinions

Institutional Commitment

Explicit political will is required. High priority must be given to the innovation, and budget should be available to cover costs of lightened work load for professors, accompaniment, production of resources, and evaluation of the pilot.

It is important to lighten the professor’s course load, especially during the semester of the innovation design and planning, as is assigning assistants to support research into / creation of digital resources.

Professor’s Commitment

The professor must be intrinsically motivated to innovate in his/her practice and participate voluntarily.

Innovation requires commitment from the teacher and the program director as well as dedication throughout the project.

Focus

Active and reflective students are the focus of innovation. The fact that students are digital natives is an exploitable resource.

We must always keep the academic workload of the student in mind and consider the strategy for introducing the change in teaching style.

The innovation is a good opportunity to reflect on teaching in the field of engineering.

The professor should have the autonomy to define the changes that he makes in the courses.

The pedagogical innovation with ICT must be part of a broader, long-term institutional goal.

Leadership by the innovation leader helps to define and refine the scope and strategy of innovation.

Giving the opportunity to innovate in course units/modules, courses themselves, and at the program level.

Starting by thinking about the desired lessons for the students’ professional training rather than planning beginning with content makes a positive difference.

The pedagogical and technological accompaniment must be ensured throughout the innovation process.

The accompaniment of educational consultants from ConectaTE throughout the innovation process makes a positive difference.

Scope and target

Accompaniment

Workshops to improve the fluency and quality of the presentations in the videos should be included. Formative assessment

It is important to assess innovation set up and to learn from mistakes

Generating information from the beginning permits one to show advances clearly and with scientific support, which encourages replication of this type of experiences.

Assessment of outcomes

It is interesting to know if the amount of face-to-face instruction can be reduced in the blended modality.

From the outset, it is important to define how the impact of the innovation on students and teachers will be evaluated. This includes the entry and exit profiles, as well as information about the inputs and the process.

Socialization

Socialization via publications and presentations helps to motivate other professors to innovate.

It is important to publicize cases of success and to reflect on them in order to invigorate the process of change.

Outcomes in the Faculty of Engineering Here we return to the central question of this research: What can the Faculty of Engineering learn from this pilot experience with the aim of assuring competency-based academic programs, quality, innovation, flexibility, internationalization and sustainability? Below we present what we can confirm so far from our data: 1. Reflection on teaching practice and the pedagogical, as well as on technological innovation in engineering courses, serves as the basis for designing and implementing innovations that seek to cause changes in the professors and students. 2. The three pilots carried out in REDINGE1 make the collaboration between the innovating teachers and the consultants that accompanied them evident. The objective of this collaboration was for students to gain competencies that go beyond the disciplinary and to the development of competencies necessary to participate in the knowledge society. 3. In the professors’ opinion, the accompaniment by ConectaTE throughout the innovation process enriches their ideas about how to promote learning. It impacts the media they use in their teaching. Because of the process, they shift from being the center of their classes to having the role of facilitator. 4. The cognitive impact of the innovation was significant as was shown by the significant differences between the control and experimental groups. In the cases where an experimental design was not possible, there was improved student retention and passing rates. This was largely due to changes in participation before, during and after class because of the use of easily accessible media to transmit knowledge. 5. The evaluation of the non-cognitive effects does not show significant differences for soft skills though there is not a definitive explanation of why. One could imagine that this result is not only the effect of participating in a course, but of the educational ecosystem that leads students to study in a university like UniAndes. The feedback about the learning processes shows that students were satisfied with the media and methodologies used. There were variations in these perceptions depending on how the innovation was carried out. 6. What has been learned in relation to the key success factors for pedagogical transformation with ICT support provides guidelines for the expansion and sustainability of initiatives such as those studied here.

Recommendations The pilot experiences in engineering education carried out at the University of the Andes, Bogotá, can serve as inspiration for other initiatives within and beyond this university. Internally, only a few departments of the Engineering Faculty at UniAndes have benefited from initiatives such as this, which means that it is necessary to capitalize on this experience to create innovation guidelines to expand the initiative. This experience can also be a reference for exploring such opportunities in other engineering faculties. Keeping this in mind, we make the following recommendations to the academic community: 1. Encourage teachers from different engineering programs to learn about innovation experiences such as these by reading publications about innovations, talking with other professors innovating in their courses, and exploring the digital resources produced. For these purposes, the Faculty of Engineering has made the video clips produced for each of the redesigned courses open access. 2. Analyze the critical success factors identified by the different actors in the process to define or refine an institutional strategy to foster pedagogical innovation with ICT. 3. Explore interest in participating in experiences such as these, both among faculty members and the directors of their departments; this information can be the starting point for defining the scope of the innovations and determining the human and financial resources required to carry out what is proposed. 4. Ensure that the accompaniment is directed not only at the design and implementation of the innovation, but also toward formative assessment and evaluation of its impacts.

Acknowledgments This work is the result of collaboration between the Faculty of Engineering at the University of the Andes and the Center for Innovation in Technology and Education (ConectaTE) from the Faculty of Education. The Dean of Engineering, Eduardo Behrenz, led the process in 2015 and 2016, and his successors, Carlos Francisco Rodríguez and Alfonso Reyes, continued. Professors Mario Castillo and Astrid Bernal (Industrial Engineering), Juan Pablo Casas (Mechanical Engineering), and Juan Carlos Reyes and Francisco Galvis (Civil and Environmental Engineering), were the professors who implemented the innovations in the redesign their courses. Luz Adriana Osorio, director of ConectaTE, coordinated resources and monitored the process. Alvaro Galvis (ConectaTE) was the academic director of the process and techno-pedagogical facilitator as well. Mónica Patiño, coordinator and technological advisor, Alexa Tatiana Jiménez and Angélica Avalos, pedagogical advisors, Diego Valencia, consultant in formative assessment, also participated from ConectaTE. The production of educational software was led by Ricardo Calle’s team. Video production and creation of digital leaning environments was done by the team led by Mariela Rivero. The Faculty of Education’s Assessment Center performed the summative assessment with contributions from Samir Cure and Julian Mariño. Many thanks to everyone involved in the process.

References Bates, A. W. (2015). Teaching in a digital age. Guidelines for designing teaching and learning for a digital age. Tony Bates Associates Ltd. Retrieved 11 03, 2016, from https://opentextbc.ca/teachinginadigitalage/ Bernal, A. (2017, 2 10). Reflexiones acerca de mi docencia con pedagogía activa y TIC. (D. Valencia, Interviewer) Casas, J. P., & Leon, J. A. (2016). Pedagogía activa con TIC para aprender Mecánica de Sólidos Rígidos. Bogotá: Uniandes - Ingeniería Mecánica (Borrador de informe). Castillo, M. (2017, 02 10). Mis reflexiones sobre el uso de pedagogía activa en Probabilidad y Estadística. (D. Valencia, Interviewer) Churches, A. (2009, 10 01). Taxonomía de Bloom para la era digital. Retrieved from Eduteka: http://eduteka.icesi.edu.co/articulos/TaxonomiaBloomDigital Collison, G., Elbaum, B., Haavind, S., & Tinker, R. F. (2002). Facilitating online learning: effective strategies for online facilitators. Madison, WI: Atwood Publishing. Crawley, E. F., Malmqvist, J., Östlund, S., Brodeur, D. R., & Edström, K. (2014). Rethinking Engineering Education: The CDIO Approach. Springer Science & Business Media. Deci, E., & Ryan, R. (2000). The "What" and "Why" of goal pursuits: Human needs and the selfdetermination of behavior. Psychological Inquiry, 11(1), 227-268. Erickson, H. L. (2007). Concept-based curriculum and instruction for the thinking classroom. Thousand Oaks, CA: Sage Publications Ltd. Forté, E., & Wentland, M. (1998). The ARIADNE project: knowledge pools for computer-based and telematics supported classical, open and distance education. AAUC Ariadne Academic Users Group Conference (pp. 1-23). Lucerna, Swtizerland: AAUC. Galvis, A. H. (2016). Ambientes de aprendiaje híbridos, o mixtos, para educación superior y a lo largo de la vida. Bogotá: Uniandes - Facultad de Educación - Conecta-TE. Galvis, A. H., & Pedraza, L. C. (2013). Desafíos del bLearning y del eLearning en educación superior. In N. Arboleda Toro, & C. Rama Vitale (Eds.), La educación superior a distancia y virtual en Colombia: nuevas realidades (pp. 113-154). Bogotá, DC: Virtual Educa y ACESAD. Retrieved from https://goo.gl/jBNMpN Galvis, F. A. (2016, 10 13). Reflexiones acerca de mi docencia con pedagogía activa y TIC. (D. Valencia, Interviewer) Hansen, E. J. (2011). Idea-based learning: A course design process to promote conceptual understanding. Sterling, VA: Stylus Publishing, LLC. Mills, J. E., & Treaagust, D. F. (2003). Engineering education - is problem-based or project-based learning the answer? Australian Journal of Engineering Education, 1-16. Reyes, J. C. (2016, 10 21). Reflexiones acerca de mi docencia antes y después de REDINGE1. (D. Valencia, Interviewer) Reyes, J. C., & Galvis, F. A. (2017). Implementation of blended learning in a Structural Analysis course: Paper 163 accepted for presentation in REES 2017. Bogotá: Research in Engineering Education Conference Proceedings (julio 6 a 8, 2017). Uniandes. (2015, 12 9). PDI 2016-2020 Versión final. Retrieved from Universidad de los Andes, Colombia: https://goo.gl/biCHcW Uniandes, Centro de Evaluación de la Facultad de Educación. (2016). Evaluaciòn del efecto de una innovación tecnológica y pedagógica sobre el aprendizaje en un curso de Probabilidad y Estadística. Bogotá: Facultad de Educación - Centro de Evaluaciòn.

Uniandes, Centro de Evaluaciòn de la Facultad de Educación. (2017). Evaluación de innovaciones educativas de ConectaTE sobre habilidades blandas en la Facultad de Ingeniería. Bogotá: Centro de Evaluaciòn, Facultad de Educación. Uniandes, ConectaTE. (2016). Portal del Centro de Innovación en Tecnología y Educación. Retrieved from ConectaTE : https://conectate.uniandes.edu.co/ Uniandes, Facultad de Ingeniería. (2017). Facultad de Ingeniería, Información General. Retrieved 02 16, 2017, from Universidad de los Andes: https://goo.gl/vM44a6 Uniandes, Secretaría General. (2015, 04 15). Faculty Handbook - Approved by the Steering Commitee session 105-15, April 15, 2015. Bogotá, DC, Colombia. Retrieved from https://goo.gl/1kTxnW Valencia, D. (2017). Borrador de informe sobre opiniones de estudiantes y docentes acerca de cursos de REDINGE1. Bogotá: ConectaTE - documento de trabajo. Wiggins, G., & McTighe, J. (2005). Understanding by Design. Alexandria, VA: Association for Supervision and Curriculum Development. Ziegenfuss, D. (2007). A phenomenographic analysis of course design in the academy. Journal of Ethnographic and Qualitative Research(2), 70-79.

Copyright statement Copyright © 2017. Alvaro Galvis, Francisco Galvis, Juan Pablo Casas, Diego Valencia, Angélica Avalo, Alexa Jiménez; The authors assign to the REES organisers and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive license to REES to publish this document in full on the World Wide Web (prime sites and mirrors), on portable media and in printed form within the REES 2017 conference proceedings. Any other usage is prohibited without the express permission of the authors.