capitulo 9. learning performance with an augmented

Please Cite this paper as: Bacca, J., Baldiris, S., Fabregat, R., Clopés, J., & Kinshuk. (2016). Learning Performance with an Augmented Reality application in the Vocational Education and Training programme of Car’s Maintenance. In Proceedings of the VIII International Conference of Adaptive and Accessible Virtual Learning Environment (pp. 90–102). Cartagena de Indias, Colombia: Sello Editorial Tecnológico Comfenalco. Retrieved from: http://www.cava-conference.info/cava/cava2016/wp-content/uploads/2016/10/REAumentados.pdf

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión

CAPITULO 9. LEARNING PERFORMANCE WITH AN AUGMENTED REALITY APPLICATION IN THE VOCATIONAL EDUCATION AND TRAINING PROGRAMME OF CAR’S MAINTENANCE Jorge Bacca PhD student at the UdG (Spain) Silvia Baldiris Postdoctoral student in Athabasca University (Canada) Professor in the Fundación Universitaria Tecnológico Comfenalco (Colombia). Ramon Fabregat Director of the research group BCDS Joan Clopés Professor of the Institut Montilivi in Girona. Kinshuk Full Professor in the School of Computing and Information Systems at Athabasca University, Canada Member, IEEE

I. INTRODUCTION Augmented Reality (AR) technology is a topic of increasing interest today in manufacturing and industrial maintenance. One of the main advantages is the possibility that this technology offers for providing access to augmented information in real time for tasks of maintenance and repairing in a wide variety of fields and using many devices such as HMD (Head Mounted Displays), tablets, smartphones among others. This increasing interest in AR at the industrial level has also moved to the educational level, in particular, to the Vocational Education and Training (VET) level where many students are trained to work in many fields in the industry such as logistics, transport, manufacturing, electricity, automotive service maintenance and so on. However, as pointed out by Anastassova and Burkhardt the Automotive Service Technicians (AST) training cannot be considered a wellstructured, closed and fully working learning system. Consequently, the research on this topic cannot be studied in strictly controlled experiments but instead a description of how this open learning system works and how the formal and informal learning processes occur in this context are needed. Moreover, Borsci, Lawson and Broome claim that more studies

ABSTRACT The concept of Augmented Reality (AR) was coined in the context of industrial maintenance and nowadays AR is being applied to support teaching and learning process at different levels of education. One of these levels is the Vocational Education and Training (VET) which covers programmes that train people to develop a particular occupation or trade. At this level of education, AR is starting to show its benefits for the learning process. However, more empirical research is needed to uncover the benefits of AR for the students’ learning performance at this level of education. In this regard, this paper aims to contribute to the state-of-the art in the impact of AR in VET education in terms of the students’ learning performance. With that aim, this paper reports a quasi-experiment with a group difference study in three VET institutes with 69 students and explores the impact on the students’ learning performance when they use an AR application in the context of a VET programme about car’s maintenance. KEY WORDS Augmented Reality, empirical research, Learning Performance, Vocational Education and Training, car’s maintenance, automotive service technicians.

90

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión in a safe environment [6]. This view is supported by Emmanouilidis, Papathanassiou, Pistofidis and Labib who state that AR provide problembased maintenance training at low cost without going to the real-world environment.

that systematically explore the effectiveness of Virtual Reality (VR) and Mixed Reality (MR) in the training of service operators are needed. Mixed reality is defined as a subclass of VR technologies that combines the real world with virtual worlds In that regard, this paper reports a study on the learning performance of students from a VET programme of car’s maintenance when they use the mobile AR-based Paint-cAR application for learning. A quasi-experiment design with a group difference study was conducted in three VET institutes in Catalonia (Spain) to identify the impact of using the PaintcAR application on the students’ learning performance.

However, one of the disadvantages of AR-based training systems is that the content cannot be modified easily in the AR application. In terms of learning performance in VET levels using AR, Westerfield, Mitrovic and Billinghurst claim that most of the AR systems have focused on improving the user performance rather than focusing on teaching how to perform the task. The researchers introduced an AR system that combines an ITS (Intelligent Tutoring System) with an AR interface and the results of an experiment showed that the system improved the learning performance by 25% and the task performance by 30% in the process of assembling a computer motherboard. Likewise, Cubillo, Martin, Castro and Boticki developed an AR authoring tool for teachers to create AR learning experiences. The learning experiences created with the tool were tested with a group of VET students and the results show that students who studied with the AR experience had better results than the students who did not use it.

The rest of the paper is organized as follows: Section II describes the related work on AR in VET education and learning performance on ARbased learning experiences. After that, section III describes the Paint-cAR application used in the quasi-experiment and section IV describes the research design. Section V presents the results of the testing in the three scenarios which are discussed in section VI. Finally section VII presents the conclusions and future work. II. RELATED WORK Recently, there has been an increasing interest in research on the use of AR at VET levels of education. As pointed out by Yuk-kwan & Yeeshun, there is an increasing need of using mobile and flexible technologies to enhance learning experiences and technologies that serve as a complement of teaching and learning strategies in the evolving VET field. This need has boosted the research on AR in VET education. Yuk-kwan & Yee-shun state that some advantages of using AR to facilitate learning in VET are: AR is a low cost technology. It is easy to change and customize a virtual environment instead of a physical environment for learning practices; and, AR allows repeated practice for a large number of students before doing the work in a real-world environment. Other advantages reported are as follows: AR applications can be used to deliver different learning contents and AR allows to practice skills

With the aim of exploring the effectiveness of Virtual Reality (VR) and Mixed Reality (MR) applications used for training operators in procedural skills and maintenance, Borsci, Lawson and Broome conducted a survey and concluded that more studies are needed that systematically explore the effectiveness of VR/MR in the training of service operators. Borsci, Lawson and Broome also argue that in the field of training car service operators two challenges will be faced by researchers: the first challenge is to explore the training of sequential operations to reach a service procedure, and the second challenge is the design and assessment of tools for training car service maintenance.

91

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión people with cognitive disabilities. The framework was developed in the context of the European Project POSEIDON. They conducted a case study with 13 people with Down Syndrome (DS) for teaching them how to follow a route to reach a destination. The results are promising and show that the augmented information can help participants to follow a route and found that people with DS can make connections between the virtual world and the real world using VR and AR.

In contrast, Anastassova and Burkhardt argue that AR is an emerging technology that is in a state of searching potential applications. This state is the cause of a technology-driven research that put aside user’s requirements and its effectiveness. Consequently, some empirical results do not clearly report the benefits of AR for training. Besides that, the authors conducted 2 field studies and identified a set of requirements for future AR teaching aids. In short, the requirements are: AR applications for automotive service technicians (AST) training should be easy to use, facilitate the construction of shared representations, cost-effective, compatible with other technologies used in training and should collect and save the field experiences in the form of narratives.

However, these studies have focused on the research of particular disabilities such as cognitive disabilities. But a broader perspective may be adopted not only to address special needs of those in the margin but also the rest of students, including support for advantaged students. In this regard, Tolentino, Birchfield and Kelliher argue that special education has had limited opportunities for including digital technologies in the classroom and as a consequence many students in special education schools have not had the same exposition to digital technologies as students in other institutes. To overcome this issue, the researchers developed SMALLab (Situated Multimedia Arts Learning Lab), which is a mixedreality environment that implements the Universal Design for Learning (UDL) to provide multiple means of representation, expression and engagement. SMALLab was used to create a mixed-reality learning environment for chemistry and the results show significant gains in conceptual understanding, spatial abilities and reasoning.

However, these studies have not considered issues that are directly related to the training itself such as the barriers that the training material may impose to some learners during the training process. This means that the ARbased training material may impose some barriers to students during the learning process. In recent years some authors have begun to study the impact of AR in vocational education and training of people with disabilities. For instance, Chang, Kang and Huang developed ARCoach, a marker-based AR system for vocational skill-training of people with cognitive impairments. ARCoach uses a prompting system with audio and visual cues that helps participants to learn how to prepare a meal. The results showed an increasing success rate by using the system and better knowledge retention. Similarly, Chang, Kang, and Liu developed a marker-based AR game for vocational skill-training for people with cognitive impairments in the context of recycling. The results show that the gaming system has potential for facilitating training in vocational jobs.

III. THE PAINT-CAR APPLICATION Since this study aims to provide insights into the students’ learning performance when they use an AR application for learning, this section describes the Paint-cAR application, a markerbased mobile AR application developed to support the learning process of repairing paint on a car, in particular, in the context of the Vocational Education and Training (VET) programme of car’s maintenance (Car bodywork). Repairing paint on a car is a complex

Covaci, Kramer, Augusto, Rus and Braun introduced a framework based on AR and VR for education and training on daily routine tasks for 92

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión mode, they do not have help and assistance. Students are challenged to complete the process using the knowledge they acquired in the guided mode. The second activity is to answer some multiple-choice questions related to theoretical background for each step. The questions are validated by the system automatically and students need to approve the test to successfully complete the activity. The two activities need to be completed to continue with the next step.

process that involves a sequence of more than 30 steps and the use of many chemical products and tools in the workshop that need to be used by students in an adequate and careful way. The Paint-cAR application was developed under a Co-creation process in which teachers, software developers and educational technology researchers participated. The application was developed to address some difficulties faced by teachers in this VET programme such as: students’ lack of motivation, attention difficulties, lack of background knowledge and the need of a tool that complements the learning process of students by practicing the process without using real elements that are expensive until they reach certain expertise level to deal with real elements in the workshop. The application is based on the Universal Design for Learning (UDL) framework to address students’ special educational needs and to foster expert learning for all. Besides that, the application was designed to work in three modes: 1. Guided Mode: In this mode, students are guided step by step through the process of repairing paint on a car. For each step, students have some activities, such as: watch videos about the step, answer multiple-choice questions and search for the chemical products or tools they need to use for that step. The last activity includes an AR experience in which students need to move around the workshop and scan the markers associated to the tools and chemical products. In the guided mode, students can ask for help in the application at any time and the application will provide hints and information to help them to find the appropriate tools and products in the workshop. All the activities for each step need to be completed so that they can continue to the next step.

3. Informative Mode: In this mode, students have all the steps available and they can see the information of any step at any time. This is intended for students that want to practice the process as many times as they want to remember some particular pieces of information of some specific steps. Students can restart the evaluation mode and the guided mode at any time to practice the process from the beginning again. The Paint-cAR application was tested in a pilot study where the levels of motivation when using the application were explored and the results were promising and indicate a positive impact on students’ motivation as described in [16]. A booklet was created and contains the AR markers and the images of the chemical products and tools so that students can also use the application at home and not only in the workshop. The PaintcAR application was developed in C# with the Unity platform with the support of the Vuforia SDK for the AR module. The application works in tablets and smartphones with the Android operative system and the responsive design has been tested for virtually any screen resolution. IV. RESEARCH DESIGN A quasi-experiment was conducted to explore the learning performance of students when using the Paint-cAR application. Participants were divided into control group and experimental group to conduct a group difference study. Students in the experimental group used the Paint-cAR application in the learning process and students in the control

2. Evaluation Mode: In this mode students need to go through the process step by step and they have to complete two activities. The first one is to search for the tools and products for each step (as in the guided mode) but in this 93

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión by administering the questionnaire to 9 students of the advanced cycle in Car’s Maintenance from the institute 3 in Girona (Spain). The Cronbach’s alfa was 0,57 and the questions that affect the reliability were identified. These questions were analyzed with the expert teacher in this field and they were adjusted in terms of clarity, wordiness and general style to make them more understandable. Besides that, two questions were removed from the questionnaire because they caused confusion to students.

group followed a traditional class with the support of some online material in Moodle. The content equivalence was ensured for the control group. The independent variable was the group in which each student was assigned (control or experimental) and the dependent variable was the students’ learning performance. The learning performance was measured with a test for knowledge assessment as described in sub-section B. The null hypothesis H0 is that the two populations from which scores were sampled are identical and the alternative hypothesis is that the two populations from which scores were sampled are different. This difference may explain that the Paint-cAR application affects the learning of students in the experimental group.

In the second phase of the reliability analysis, the adjusted questionnaire was administered to another group of students from an advance cycle in Car’s Maintenance from another institute in Spain. In this phase 11 students from another institute in Barcelona participated in this process. The KR-20 method was used again to evaluate reliability. The Cronbach’s alfa was 0,77 and 2 questions were identified as problematic questions. These questions were adjusted.

A. Participants Participants in this study were 69 students enrolled in the intermediate training cycle in “Car Bodywork” in the context of the Vocational Education and training programme of Car’s Maintenance. The 69 students came from 3 different VET institutes in Catalonia (Spain), identified as Institute 1, Institute 2 and Institute 3. In the institute 1, 43 students participated (26 students were assigned to the control group and 17 to the experimental group), in the institute 2, 13 students participated (6 students were assigned to the control group and 7 to the experimental group) and finally in the institute 3, 13 students participated (6 students were assigned to the control group and 7 were assigned to the experimental group). These three VET institutes are the three Testing scenarios considered in this study.

In addition, a second test of knowledge was designed by teachers for an extended validation that was carried out in one of the VET institutes. The test consists of 22 questions that evaluate students’ learning performance. C. Procedure In this section, the procedure of the group difference study is described. The control group followed the traditional classes and students had access to additional learning material in Moodle. With regard to the experimental group, the procedure was as follows: first, an introduction about AR and a general explanation about the application and its purpose were provided to the students. Then, assistance was provided so that they can download and install the application on their own smartphones. The students were also provided with the booklet so that they could use the application at home. Then, the students were advised to explore the application and use it in guided mode for one week. During this

B. Instruments A test for knowledge assessment was created by teachers to evaluate the students’ learning performance in the control and experimental groups. The test consisted of 38 multiple-choice questions with only one correct answer. The reliability of the questionnaire was analyzed with the KR-20 (Kuder-Richardson) in two phases. In the first phase of analysis the questionnaire had 40 items and was conducted 94

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión conducted to identify if data of each VET institute have been drawn from a normal distribution.

phase, the researchers received information about the activity of the students in the mobile application and the teachers were informed if some of them were not using the application.

A. Results of the testing scenario in the institute 1 The results of the normality test confirmed that data followed a normal distribution. Then, a parametric independent samples t-test was conducted on the scores. Participants in the control group obtained slightly better scores (M=45.34, SD=10.89) than those in the experimental group (M=42.56, SD=11.25). The difference between means was not significant, t(41)=0.806, p>0.05, two-tailed.

After the first week of using the application, a testing in the workshop was conducted. The AR markers were placed on the real tools and products in the workshop. The testing lasted 2 hours on average for each vocational education institute. The students used the application in the guided mode to remember some concepts and then they were advised to use the application in evaluation mode. After the testing, the students were advised to use the application again at home for one week more in guided mode. After the second week of using the application at home, a second testing in the workshop was carried out. The testing lasted 2 hours on average for each vocational education institute. In this testing, the students used the application in evaluation mode.

This result means that the null hypothesis cannot be rejected, so it seems that there is no difference between the students that use the AR application and those who did not use it in terms of learning performance. B. Results of the testing scenario in the institute 2 The results of the normality test confirmed that the data followed a normal distribution. Then, a parametric independent samples t-test was conducted on the scores. Participants in the control group obtained lower scores (M=42.10, SD=8.4) than those in the experimental group (M=47,36, SD=16,7). The difference between the means was not significant, t(11)= -0.693, p>0.05, two tailed.

Two days after the last testing in the workshop, the students in both the control and the experimental groups answered the test for knowledge assessment. In order to extend the analysis of the students’ learning performance, it was decided to conduct an extended validation in one of the VET institutes for a longer period to identify if an increase in the time using the Paint-cAR application would change the scores in the learning performance. The extended validation was carried out in the institute 3 in Girona (Spain) and it lasted 4 weeks more. For each week, a real testing in the workshop was conducted.

The results show that the students in the experimental group (those who used the PaintcAR application) got better results than those in the control group in terms of learning performance. However, this result is not significant to reject the null hypothesis and this means that the groups do not seem to be different.

V. RESULTS Data collected from the tests of knowledge assessment was used to conduct a group difference study. An analysis for each one of the three VET institutes was conducted and the results are described in the following subsections. The Kolmogorov-Smirnov and the Shapiro-Wilk test for normality were also

C. Results of the testing scenario in the institute 3 The results of the normality test confirmed that the data followed a normal distribution. Then, a parametric independent samples t-test was 95

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión validation was: M=59.39, SD=17.17 and the mean of scores at the end of the extended validation was: M=62.40, SD=25.7. The difference between the means was not significant, t(6)=-0,563,p>0,05,two-tailed.

conducted on the scores. Participants in the control group obtained lower scores (M=41,66, SD=30,4) than those in the experimental group (M=59,39, SD=17,1). But, the difference between means was not significant, t(11)=1.32, p>0.05, two-tailed.

Together these results show that both groups (control and experimental) obtained better results at the end of the extended validation.

The results show that the students in the experimental group (those who used the PaintcAR application) got better scores than those who did not use it as it can be seen by comparing the mean of each group. However, the test was not significant and the null hypothesis cannot be rejected showing that the difference between the means is not enough to assert that the two groups are different in terms of learning performance.

Then, a comparison between the results of the control and experimental group at the end of the extended validation (a between-subjects comparison) was carried out to determine differences between the groups at the end of the extended validation. The results show that participants in the control group obtained better results (M=75.43, SD=17.12) than those in the experimental group (M=62.40, SD=25.77). The difference between the means was not significant, t(8)=0,452, p>0,05, two-tailed.

D. Extended Validation in the institute 3 At the end of the testing in the institute 3, the students answered the test of knowledge. Then the extended validation started and lasted 4 weeks. After this time, a second test of knowledge was applied. A comparison between the scores of the two tests was made in order to determine if an increase in the time using the Paint-cAR application would change the scores in the learning performance. In this regard, first the normality test was applied to identify if data was drawn from a normal distribution. The results of the normality test confirmed that the data followed a normal distribution.

VI. DISCUSSION A quick overview of the results may suggest that there is no difference between the students that used the Paint-cAR application and those who did not. However, Anastassova and Burkhardt claim that Automotive Service Technicians (AST) training cannot be considered a well-structured, closed and fully working learning system. Consequently, the research on this topic cannot be studied in strictly controlled experiments but instead a description on how this open learning system works and how the formal and informal learning process occur in this context are needed.

Then, a parametric paired sample t-test was conducted on the scores. As a result, all participants (from the control and experimental groups) obtained better results at the end of the extended validation. With respect to the control group, the mean of scores at the beginning of the extended validation was: M=62.28, SD=25.55 and the mean of scores at the end of the extended validation was: M=75.43, SD=17.12. The difference between the means was not significant, t(2)=-1,25,p>0,05,twotailed.

By drawing on this assumption, we argue that the analysis of the results of students’ learning performance in the VET institutes must consider the conditions of each VET institute in terms of infrastructure, materials for learning, teachers and the diversity of students to provide insights that drawn on empirical studies with respect to the use of AR technology in VET education. In terms of the learning performance, the results obtained in the institute 1 seem to show

With respect to the experimental group, the mean of scores at the beginning of the extended 96

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión difficult to understand the contents and answer the multiple-choice questions. This suggests that AR applications should be designed in a way that can support multiple languages spoken by the students or at least provide mechanism for them to understand the contents.

no difference between the students’ learning performance in the control and experimental groups. In this institute, there were two groups of students, one that took classes in the morning and the other that took classes in the afternoon. For logistic reasons, the teachers decided that the group in the morning would be the control group and the group in the afternoon would be the experimental group. However, it seems that the group in the morning (control group) had students with a better background knowledge than those in the afternoon (experimental group). This was the result of an internal selection of students that the institute made at the beginning of the course and the students with better background knowledge were in the group that took classes in the morning.

The results of the institute 2 show that students in the experimental group (using the Paint-cAR application) obtained better results than those who did not use the application. In this institute, the assignation of the students to the control and experimental groups was almost randomly because the students who did not have an Android device were assigned to the control group. Despite the fact that the difference between the score means of the control and experimental groups was not significant, it is worth noting that the use of the application seems to somehow help students to get better results. In this institute, the infrastructure is small for learning this topic and the students do not have enough materials. The consequence of these conditions was that some of the markers that are recognized by the Paint-cAR application had to be stuck in places that simulate the products and tools. However, in this kind of environments, AR offers the opportunity to interact with virtual materials without the need of using real materials, which reduces costs as stated by Emmanouilidis, Papathanassiou, Pistofidis and Labib Another limitation in this testing was the internet connection because the students did not have access to the Wi-Fi, so they needed to use the 3G and 4G connections.

In terms of the infrastructure, this institute has two workshops with enough space and enough materials for students to practice the real process of repairing paint on a car. However, one of the disadvantages during the testing was that the students who finished the testing with the Paint-cAR application started to work with real materials in the workshop and the students who were still working with the application felt that they missed real practice because they were using the application and they felt discouraged. Another limitation in the institute 1 was that the students did not have access to the Wi-Fi and they had to use their 3G and 4G connections. However, the 3G and 4G signals are very low and sometimes cannot be used. Since the application uses internet connection to send and retrieve data from the servers, some students felt upset because sometimes they could not use the application.

Despite the conditions in the institute, once the students started working in the application, they were engaged in the activities and they made an effort to complete the process as required by the teacher. This is in-line with the findings in other studies that recognize the benefits of AR for increasing attention, motivation and engagement.

With respect to the type of students, for most of them Catalan is not their first language. Some of them are not proficient enough in this language despite of the fact that most of the training is done in this language because it is the official language of teaching in Catalonia. Thus, the contents of the application were in Catalan language. For some of these students, it was

Taking into account that the students almost never used the application at home, they did not have enough background knowledge to 97

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión In this institute, some of the students in the experimental group had used the Paint-cAR application before the first testing in workshop (as recommended by the teacher in the first session of introduction and installation of the application), and during the testing, those students showed an improved performance than those who had not used the application before. From the observation of the testing, the teacher concluded that most of the students focused on the virtual information more than on the real objects they were exploring. This confirms one of the disadvantages of AR applications that is about paying too much attention to the virtual information as reported in previous studies [20] [25]. This suggests the need of a correct balance between virtual and real information in the AR applications so that students pay attention to real and virtual objects at the same time to achieve the link between virtual and real information.

complete the process in the evaluation mode during the testing in the workshop. Therefore, they felt disengaged because they could not advance in the process. When students needed to use the application at home, they did not use it. They claimed that they did not have time or they forgot it. The teachers shared their opinions about this and they claim that it is very difficult to motivate students at this level and it is always needed to tell them what to do and how to do it. Otherwise they are not going to do it by themselves. Thus, we recommend that future learning scenarios that include AR applications should encourage students to use the AR application at home, or more time in class should be provided so that the students can use the application for learning. Surprisingly, in this institute, some of the students who were not using the application (because they were in the control group) showed their interest in the application and they tried to help their classmates to complete the activities. Another relevant observation made during the testing was the fact that although the students were advised to pay special attention to the information in the guided mode so that they could complete the evaluation mode, some of the students still depended a lot on the help and assistance provided by the application in order to complete the task in the evaluation mode. This suggests that an important recommendation in the design of AR applications for supporting learning performance is to include a scaffolding mechanism to assist students in completing the tasks. This is in-line with findings in other studies that remark the importance of using a scaffolding strategy in AR applications.

Another issue identified during the testing is that the workshop in this institute sometimes has light conditions that are not optimal for an AR learning experience. The result was that some students had problems when trying to scan the markers. This led to disengagement in the task because the students felt that they could not use the application properly. We recommend that the use of AR applications in this type of contexts should take into account that light conditions in the workshops need to be optimal to create a successful AR learning experience. Besides that, the students were confused at the beginning because they did not know if the marker was not correct or the camera was not recognizing the AR marker. Another disadvantage was that sometimes students felt that they were competing with their classmates. The result of this situation was that they just tried to complete the task faster than their classmates and they did not focus on the information provided by the application. This may have led to negative or poor learning performance. We recommend that future studies should have more control over student’s

On the other hand, the results in the institute 3 show that the students in the experimental group obtained better scores than those in the control group but the difference between the means was not statistically significant.

98

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión and Burkhardt [3] highlight that the costeffectiveness is a requirement for the development of future applications for teaching to automotive service technicians.

activity and teachers will need to control future AR learning experiences in this regard. In contrast to the other VET institutes, the internet connection in this institute was very good. This facilitated the AR learning experience.

It is important to note that the scores obtained by the students ranged from 40 to 75 out of 100. One may think that these results are very low for a test of knowledge assessment. However, the teachers from the VET institutes claimed that the results are as expected because this is the average score for students at this level and in this particular VET programme.

In terms of using the application at home, most of them used it because the teacher continuously reminded them to use the application at home. Consequently, it was evident in their performance in the testing carried out in the workshop and they outperformed students from other VET institutes in terms of the number of activities completed in less time and with fewer errors. For this reason, the extended validation was conducted in this VET institute in order to determine if an increased exposure to use of the Paint-cAR application may improve the students’ learning performance.

Overall, these results provide insights into the students’ learning performance in AR learning experiences in the process of repairing paint on a car. Moreover, some recommendations have been identified for future studies in the context of AR learning in VET educational research. VII. CONCLUSIONS The results presented in this study together with the description of the conditions in which the three testing scenarios were carried out provide insights into the students’ learning performance in mobile AR learning experiences, leading to various recommendations for future studies in this field. This paper contributes to the state-of-the art on the use of AR in VET education, in particular in the topic of repairing paint on a car.

In terms of the learning performance as highlighted in the results (section 5), the difference between the means of the control and the experimental group was not statistically significant. However, both of the groups improved their learning performance after the extended validation. In particular, the students who used the Paint-cAR application showed remarkably better results in the 3 additional testing carried out during the extended validation. By analyzing the responses to the self-evaluation tests, an improvement was evident after each session because the students completed more tests with fewer errors. This means that the students had improved their knowledge after each session.

In terms of using the AR technology in the VET programme of Car’s maintenance, it is worth noting that the AR-based applications are aimed to provide support for the learning process and may help students to practice processes at home, using virtual products cost-effectively and save resources especially in VET institutes where the resources are limited.

As a conclusion of this extended validation, the teacher in this VET institute highlighted that the AR learning experience with the Paint-cAR application should be complementary to the traditional class and a tool that the students should use for practicing at home, with virtual tools and products which is cost-effective. With respect to the cost-effectiveness, Anastassova

In terms of the learning performance, the results show that statistically there were no significant differences between the control and the experimental group. This is in line with the findings of Rashid and Asghar [26] who found that the use of technology is not a predictor of 99

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión disabilities. For that purpose, the Universal Design for Learning (UDL) provides guidelines to address the variability of students and foster expert learning.

students’ performance but it is a predictor of students’ engagement and students’ selfdirected learning. However, this does not mean that the application is not good for learning, because in two out of the three testing scenarios (Institute 2 and Institute 3) the results of the experimental group are slightly better than the results of the control group. These results provide evidence of the positive effect of the Paint-cAR application in the learning performance. However, further research is needed in terms of identifying the impact of AR on the learning performance in VET programmes of car’s maintenance.

VIII. ACKNOWLEDGMENT Authors would like to thank to teacher Narcis Vidal (Institut Narcis Monturiol) that participated in the design of Paint-cAR application and to the teachers Sergi García (Institut Les Vinyes), Francesc Badia and Francesc Comas (Institut Castellarnau) and Manel Tomas (Institut La Mercé) that participated in the testings on the different VET Institutes.

A possible explanation of the results obtained is that students are used to working with real products and tools and sometimes the use of an application seems to be unreal for them and therefore the interest in the application may sometimes be low. To overcome this issue, it would be interesting to explore the introduction of complex simulations with AR that can be used to attract students’ attention in the topic of car maintenance and to explore new subjects in this field in which AR can be used to support the learning process. Besides that, more studies need to be conducted in other VET domains to uncover the benefits of this technology for the learning process at this educational level.

VIII. BIBLIOGRAPHY A. Covaci, D. Kramer, J. C. Augusto, S. Rus, and A. Braun, “Assessing Real World Imagery in Virtual Environments for People with Cognitive Disabilities,” in 2015 International Conference on Intelligent Environments, 2015, pp. 41–48. Á. Di Serio, M. B. Ibáñez, and C. D. Kloos, “Impact of an augmented reality system on students’ motivation for a visual art course,” Comput. Educ., vol. 68, pp. 586–596, Oct. 2013. A. M. Kamarainen, S. Metcalf, T. Grotzer, A. Browne, D. Mazzuca, M. S. Tutwiler, and C. Dede, “EcoMOBILE: Integrating augmented reality and probeware with environmental education field trips,” Comput. Educ., vol. 68, pp. 545–556, Oct. 2013.

Another important implication of this study is the fact that the AR application needs to be designed to support the learning process of students with a wide variety of learning preferences, needs and interests.

A. Syberfeldt, O. Danielsson, M. Holm, and L. Wang, “Dynamic Operator Instructions Based on Augmented Reality and Rule-based Expert Systems,” Procedia CIRP, vol. 41, pp. 346–351, 2016.

An in-depth analysis of testing scenarios, that include mobile AR in VET education, needs to involve the analysis of the context, teachers, students, infrastructure and other conditions to uncover details that may give a better explanation of the results obtained.

C. Emmanouilidis, N. Papathanassiou, P. Pistofidis, and A. Labib, “Maintenance Management e-Training: What we Learn from the Users,” IFAC Proc. Vol., vol. 43, no. 3, pp. 36–41, 2010

Moreover, future AR applications in VET domains should consider a broader perspective in terms of the attention to special educational needs instead of focusing on particular 100

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión K.-E. Chang, C.-T. Chang, H.-T. Hou, Y.-T. Sung, H.-L. Chao, and C.-M. Lee, “Development and behavioral pattern analysis of a mobile guide system with augmented reality for painting appreciation instruction in an art museum,” Comput. Educ., vol. 71, pp. 185–197, Feb. 2014.

C.-H. Tsai and J.-Y. Huang, “A Mobile Augmented Reality Based Scaffolding Platform for Outdoor Fieldtrip Learning,” in 2014 IIAI 3rd International Conference on Advanced Applied Informatics, 2014, pp. 307–312. D. Furió, S. González-Gancedo, M.-C. Juan, I. Seguí, and N. Rando, “Evaluation of learning outcomes using an educational iPhone game vs. traditional game,” Comput. Educ., vol. 64, pp. 1– 23, May 2013.

K. Chao, C. Lan, Y. Lee, Kinshuk, K. Chang, and Y. Sung, “Mobile augmented reality in supporting peer assessment : An implementation in a fundamental design course,” in Workhop proceedings of the 22nd International Conference on Computers in Education, 2014, pp. 867–878.

G. Westerfield, A. Mitrovic, and M. Billinghurst, “Intelligent Augmented Reality Training for Motherboard Assembly,” Int. J. Artif. Intell. Educ., vol. 25, no. 1, pp. 157–172, Nov. 2015.

L. Tolentino, D. Birchfield, and A. Kelliher, “SMALLab for Special Needs: Using a MixedReality Platform to Explore Learning for Children with Autism,” in The Future of Interactive Media: Workshop on Media Arts, Science, and Technology (MAST), 2008.

G. Singh, D. A. Bowman, D. Hicks, D. Cline, J. Todd Ogle, E. D. Ragan, A. Johnson, and R. Zlokas, “CI-Spy: Using mobile-AR for scaffolding historical inquiry learning,” in 2014 IEEE International Symposium on Mixed and Augmented Reality - Media, Art, Social Science, Humanities and Design (IMSAR-MASH’D), 2014, no. September, pp. 73–74.

M. Anastassova and J.-M. Burkhardt, “Automotive technicians’ training as a community-of-practice: Implications for the design of an augmented reality teaching aid,” Appl. Ergon., vol. 40, no. 4, pp. 713–721, Jul. 2009.

J. Bacca, S. Baldiris, R. Fabregat, Kinshuk, and S. Graf, “Mobile Augmented Reality in Vocational Education and Training,” Procedia Comput. Sci., vol. 75, no. 00, pp. 49–58, 2015.

M. Dunleavy, C. Dede, and R. Mitchell, “Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for Teaching and Learning,” J. Sci. Educ. Technol., vol. 18, no. 1, pp. 7–22, Sep. 2009.

J. Bacca, S. Baldiris, R. Fabregat, S. Graf, and Kinshuk, “Augmented Reality Trends in Education: A Systematic Review of Research and Applications,” J. Educ. Technol. Soc., vol. 17, no. 4, pp. 133–149, 2014.

P. Milgram and F. Kishino, “Taxonomy of mixed reality visual displays,” IEICE Trans. Inf. Syst., vol. E77, no. 12, pp. 1321–1329, 1994.

J. Bacca, S. Baldiris, R. Fabregat, Kinshuk, and S. Graf, “Mobile Augmented Reality in Vocational Education and Training,” Procedia Comput. Sci., vol. 75, pp. 49–58, 2015.

S. Borsci, G. Lawson, and S. Broome, “Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance,” Comput. Ind., vol. 67, pp. 17–26, Feb. 2015.

J. Cubillo, S. Martin, M. Castro, and I. Boticki, “Preparing augmented reality learning content should be easy: UNED ARLE-an authoring tool for augmented reality learning environments,” Comput. Appl. Eng. Educ., pp. 1–12, Apr. 2015.

S. Yoon, K. Elinich, J. Wang, C. Steinmeier, and S. Tucker, “Using augmented reality and 101

RECURSOS EDUCATIVOS AUMENTADOS - Una oportunidad para la inclusión knowledge-building scaffolds to improve learning in a science museum,” Int. J. Comput. Collab. Learn., vol. 7, no. 4, pp. 519–541, Aug. 2012. T. Rashid and H. M. Asghar, “Technology use, self-directed learning, student engagement and academic performance: Examining the interrelations,” Comput. Human Behav., vol. 63, pp. 604–612, Oct. 2016. Y.-J. Chang, Y.-S. Kang, and P.-C. Huang, “An augmented reality (AR)-based vocational task prompting system for people with cognitive impairments.,” Res. Dev. Disabil., vol. 34, no. 10, pp. 3049–56, Oct. 2013. Y.-J. Chang, Y.-S. Kang, and F.-L. Liu, “A computer-based interactive game to train persons with cognitive impairments to perform recycling tasks independently.,” Res. Dev. Disabil., vol. 35, no. 12, pp. 3672–7, Dec. 2014. Y. Kim and I. Moon, “E-Training Content Delivery Networking System for Augmented Reality Car Maintenance Training Application,” Int. J. Multimed. Ubiquitous Eng., vol. 8, no. 2, pp. 69– 80, 2013. Y. N. Ricky and Y. L. Rechell, “Using mobile and flexible technologies to enable, engage and enhance learning in Vocational Education and Training (VET),” in 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), 2015, no. December, pp. 96–101.

102