Available online at www.sciencedirect.com
ScienceDirect Procedia CIRP 38 (2015) 58 – 62
The Fourth International Conference on Through-life Engineering Services
Secure Access Augmented Reality Solution for Mobile Maintenance Support Utilizing Condition-Oriented Work Instructions Matthias Neges*, Mario Wolf, Michael Abramovici Chair for IT in Mechanical Engineering (ITM), Ruhr-Universität Bochum, Universitätstrasse 150, 44780 Bochum, Germany * Corresponding author. Tel.: +49-234-32-22109; fax: +49-234-32-14443. E-mail address:
[email protected]
Abstract In this paper, we present an approach that offers service technicians an interactive, intuitive and documentable way to maintain machinery. To realize context awareness, we use graph-based algorithms to guide the technician in achieving the machine’s desired operating condition after evaluating the current state of the machine. Our approach makes use of Augmented Reality not only as a tool for visualization, but also as interactive input method to capture the current machine state and to generate multimedia maintenance reports, e.g. by taking pictures with redlined components and adding further textual details. © 2015 2015 Published The Authors. Published by This Elsevier © by Elsevier B.V. is anB.V. open access article under the CC BY-NC-ND license Peer-review under responsibility of the Programme Chair of the Fourth International Conference on Through-life Engineering Services. (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Programme Chair of the Fourth International Conference on Through-life Engineering Services. Keywords: Augmented Reality; mobile devices; marking technology; authentication; maintenance
1. Introduction Throughout the last decade mobile devices (smart phones, tablets) have saturated the market [1] fully. These devices facilitate new oppertunities in the areas of mobile computing, mobile communication and mobile internet usage. Communication standards like Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), and Bluetooth, bundled with marking technologies like QR-Code or Near Field Communication (NFC) enable the vision of the Internet of Things (IoT) [2]. To fulfill a maintenance task, a service technician must be able to get an overview over the maintenance object and its functionalities, which causes delay to the maintenance itself [3]. The use of smart devices and technologies like Augmented Reality can improve complex maintenance processes [4]. In [5] a study has shown that work time and error rate can be halved by utilizing AR-Applications. Authentication is a viable tool to ensure that critical tasks are conducted on the right machine and by the right technician [5]. For that matter, Auto-ID-procedures with smart devices enable faster and more accurate identification, not vulnerable to errors typical for the manual entering of identification numbers [6]. Referencing multimedia content is a key component in the vision of the IoT, because it links physical elements to digital
ones [7], [8]. [3] lists several well-tested ways to store maintenance related data and their respective ways of accessing them. Today, maintenance data is stored in so called Computerized Maintenance Management Systems (CMMS), which are sometimes referenced as Enterprise Asset Management (EAM). CMMS manage working schedules, tasks, assets, stock, offer statistics and support the continuous process improvement by providing key performance indices [9]. Typically CMMS are desktop applications, which causes only limited usability on smart devices. [10] and [11] offer concepts for overcoming the shortcomings, but even then it offers no additional value over the standard CMMS. Schlick et al. [5] analyzed training success based on the instructional media. They found out that graphical instructions are superior to textual ones whenever the repetition rate is low. These findings confirm our intentions to enhance today’s work instructions. Work instructions are usually limited to simple decisions, like leaving out a few steps or jumping to another. In [12] we presented several ways to enhance this decisionmaking with greater interactivity as a goal. Augmented Reality (AR) applications enable us to enhance our digital approach and to combine digital 3D models with an information overlay that can be projected to the physical
2212-8271 © 2015 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Programme Chair of the Fourth International Conference on Through-life Engineering Services. doi:10.1016/j.procir.2015.08.036
Matthias Neges et al. / Procedia CIRP 38 (2015) 58 – 62
machine. This can be achieved by utilizing smart devices like smart phones, tablets or head-mounted devices. International projects like ARVIKA, STAREMATE or ARTESAS investigated the industrial use of AR, with focus on proprietary hardware. In cases where maintenance is to be conducted without heavy, proprietary hardware in predefined scenarios, user-friendly consumer hardware and flexible software will be advantageous. There are some recent examples of AR-applications for maintenance procedure support. Henderson and Feiner explore the benefits of AR documentation in maintenance and repair by evaluating the required time and quality of the maintenance tasks compared to traditional LCD and HUDs [13]. Lee and Akin show an AR based solution, that uses artificial markers for maintenance instruction support [14]. The authors provided an approach in 2013 by using CAD-Models to track the real maintenaince object instead of artifical markers [15] and they evaluated the potential and the ruggedness of this tracking approch in [16]. This tracking approch is also used by Sanna et al. [17] and Manuri et al. [18] to demonstrate the decrease of work time and error rate during AR supported maintenance tasks. One of the weak points is the static data provision of current approaches. Zhu et al. present a bi-directional authoring tool to enable the maintenance technican to edit and update AR content based on the on-side state [19]. Lamberti et al. focused on a stepwise approach to support disassembly tasks during the maintenace process [20]. Both explore the significance of modular and status-dependent work steps, but do not take into account the automatic generation of AR-based work instructions corresponding the real status of the maintenance object on-side. In this paper, we present a solution that enhances today’s maintenance documention. Our goal is not only to show predefined data, but to give the technicians a tool to analyze the machine at hand, and to receive appropriate instructions, based on the machine’s status. To realize this goal, we make use of a combination of 3D-Models, graph-based logic models and their association. Futhermore we support the maintenance documenation process by offering interactive tools for multimedial content creation on site. 2. Previous work We had the chance to collaborate with the German Technisches Hilfswerk (THW) Sektion Bochum to conduct a case study concerning step-wise quick guides for startup operations and maintenance of heavy salvage units. The machines were marked with optical markers, which could be read by smart devices to ensure that the right guideline was shown to the user. In [21], [15] the authors published approaches for the creation of stepwise AR work instructions, which include the use of a 3D Viewer to generate single instructions for selected components and associating them with visual hints. These could be arrows, turn direction animations etc. While working in the project Kompetenzzentrums für hydraulische Strömungsmaschinen (Competence center for hydraulic machinery) the authors developed concepts for interactive maintenance support tools , which were published
in [12]. These concepts are based on the capture of the machines status through the technician’s data enterings. Extensive textual instructions are precise and are still being used with success today. The only problem lies within the time spent by the user to convert the theoretical knowledge to the real world scenario, and the lack of reference point in the real world. AR work instructions connect the real world with the digital information overlay and ensure a direct link for better usability. To react to unforseen defects or states of the machinery, the maintenance support application must be context-aware and flexible. The creation of a logic module that is able to simulate a machine and the use of AR methods to create simple but clear input dialogs enables us to offer assistance in nearly every scenario. 3. Concept Our overall approach is divided into six phases (Fig. 1).
Fig. 1. Six phases of the overall approach
The first phase is the authentication phase, in which the service technician has to validate his access to the maintenance documentation trough authentication. This could be achieved by reading a marking on the machine, such as a QR-Code or a NFC-Tag, to ensure the position of the technician. After authentication, the digital 3D models, which are required for the subsequent AR visualization, are synchronized (phase 2) to the smart device. Furthermore it includes the logic model, which is the basis of the state analysis, and also any documentation, such as maintenance reports or technical data sheets. Based on this information and data, the service technician gets a holistic overview of the present system.
59
60
Matthias Neges et al. / Procedia CIRP 38 (2015) 58 – 62
As proposed by the authors in [21], phase 3 can be initiated by a recognition of the real object and the context-specific display of information and digital 3D models. As proposed, this is ensured by a combination of point-based and edge-based detection methods. By overlaying the real piping system with the digital 3D model, the service technician is also able to record the actual status of the system by selecting the corresponding elements on the display, highlighting them and selecting the respective status. For this purpose, the condition of the elements, such as e.g. "valve open", "closed valve", "valve malfunction" are provided. The interactive machine state recording with the ARdialogs is possible because of the mapping between the 3D model and graph based logic model, which allows an algorithmic analysis and problem determination (phase 4).
x edges: Piping and valves are both shown as edges. For a better overview in the graph both should be reported distinguishable. x node or edge costs: Potential for modeling other fluid mechanical characteristics, which are not yet implemented.
3.1. Linking reality, virtuality and logic In previous approaches, the unique identification of virtual and real components was achieved primarily for the mapping of information, instructions and animations, the relationship between the function block diagram, logic representation (directed graph) and the physical system will be further clarified here (Fig 2).
The results of the analysis can be transferred for the generation of work instructions. This requires a strong modularization of steps to sort them automatically based on the present machine state (phase 5). In [21], the authors already proposed an approach for the visualization of AR work instructions and AR-service reports. Are changes, such the changes of valve positions, required to produce the correct functioning of the system, then the recorded current machine state can also be used for a part of the service report. The report thus contains all valve positions or a statement about the functioning of the individual elements for example. Additionally, utilizing continuous AR-support can ensure the availability of the position and angle of view of the operator in relation to the physical system. This makes it possible to create position-related photos and to enhance them with comments and redlining, which in turn can be traced back to the virtual model (phase 6).
Fig. 2. Relationship between the function block diagram, logic representation and physical system
If the exact workings of a hydraulic machine or system are known, it can be represented as a graph, which in turn can be evaluated by algorithms at runtime. Piping systems are particularly easy to represent. Shortest paths algorithms (exact Single Pair Shortest Path, SPSP) can be used to calculate a certain flow. Furthermore, the algorithmic approach also allows a functional analysis based on the current system configuration, such as valve positions, defects, etc. Fig. 3. Exemplary linking of values
In all representations considered in this approach so far, unique names for (sub-)elements, such as component names or node / edge names must be chosen. As a prerequisite for the functionality of the graph-based calculations, all the representations of a single system must be consistent, so that all essential elements are present. As graph-theoretical algorithms are chosen for the simulation of hydraulic systems, the following three basic elements come to mind: x node: Each type of fitting, branching, pump, and the like is recognized as a node in the graph.
To clarify this, a simple valve is presented as an example in a complex hydraulic system. In the block diagram this valve is marked with a standard symbol and labeled “valve 1”. In the directed graph, this valve is an edge between two nodes, for example, node K1and node K2 (Fig. 3). Said valve can be identified by its position in the hydraulic system and its connections. Particularly the logical representation and the projection in reality are relevant to the intended approach, so that the association of the physical component must be consistent to the corresponding element in the graph.
Matthias Neges et al. / Procedia CIRP 38 (2015) 58 – 62
As the AR support is implemented as published in [15], the digital 3D model, it also provides its component structure in addition to its geometry. The data model of the component structure takes the exact names of the components into account, the assignment to assemblies and also additional metadata, such as versions or release dates. This data model is flexible enough to also contain references to each stored system graph, to show the number or name of the specific element (edge or node) for each component. With the widely-known graph-theoretic algorithms of the SPSP class, the shortest valid paths can be calculated in the hydraulic system.
For the AR implementation the metaioSDK 6.0 is used on the android platform. The interaction with the AR overlay is enhanced to allow the recording of the current state, so the selection of individual elements can be done via the touch display (Fig. 6) of our test device (8” Tablet). Evaluation of the captured machine status is then done via an implementation of the Dijkstra-Algorithm, working on the graph shown in Fig. 5 and the valve-status (open, closed) representing the edge costs within said graph. The algorithmic analysis of the actual state allows a selective generation of work instructions and thus a high degree of maintenance support for the service technician. The detailed recording of the actual state is used not only for the analysis and problem solving, but also flows into the documentation as shown in [15]’s concept of service reports.
4. Prototype implementation and validation As a demonstration of the validity of the proposed approach, our application is prepared for the complex hydraulic system, which was also used in the last case study (Fig. 4).
Fig. 6. Prototype implementation: recording the actual status Fig. 4. Frontside (a) and backside (b) of the demonstrator
The hydraulic system has two centrifugal pumps, two reservoirs and twelve switchable valves. The construction of the hydraulic system allows the pumping of the fluid through both of each of the two pumps through each of the two containers, as well as through both pumps in series. The respective fluid circuit depends on the combination of the respective valve positions. In the use case considered here, the case of failure of the primary pump is simulated. The goal is to continue to pump fluid through container 1. The underlying graph, and the algorithmic analysis to solve the problem described is exemplified in Fig. 5. The graph is provided along with the rest of the relevant data for further use on the mobile device.
Fig. 5. Algorithmic graph analysis for the given problem
5. Conclusion Although the current approaches take the use of AR for the visualization of information into account, they relate to static work instructions. However, there are ways for the algorithmical analyzing of a machine or system and so our application is able to provide condition-based troubleshooting assistance, after capturing the current state of the relevant parts. We have shown these possibilities with the example of a complex piping system, which is analyzable for an automated problem solving. In our approach, AR is used as an intuitive tool for recording the current machine state with a mapping of logic model, to reality and to the virtual 3D model. Our future work will pick up the idea of smart products and enhance the approach presented in this paper with interaction based on the state of the machine being captured by sensors. The part of the maintenance technician is then to evaluate the situation based on the suggestions made by the new assistant system.
61
62
Matthias Neges et al. / Procedia CIRP 38 (2015) 58 – 62
References [1] Abramovici, M., Göbel, J.-C., Neges, M. Smart Engineering as enabler for the 4th Industrial Revolution. In: M. Fathi, eds. Integrated systems. Innovations and applications. Cham: Springer International Publishing AG; Springer Verlag, 2015, S. 163-170. ISBN 978-3-319-15897-6. [2] Coskun, V., Ok, K., Ozdenizci, B. Professional NFC application development for Android. Chichester, UK: John Wiley & Sons, 2013. ISBN 978-1-118-38009-3. [3] Wohlgemuth, W., Friedrich, W. Advanced Augmented Reality Technologies for Industrial Service Applications: ARTESAS. BMBFLeitprojekt ARTESAS; zusammenfassender Schlussbericht: Siemens AG, 2007. [4] van Krevelen, D.W., Poelman, R. A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality, 2010, 9(2), 1-19. [5] Schlick, C.M., Moser, K., Schenk, M. Flexible Produktionskapazität innovativ managen. Handlungsempfehlungen für die flexible Gestaltung von Produktionssystemen in kleinen und mittleren Unternehmen. Berlin, Deutschland: Imprint: Springer Vieweg, 2014. ISBN 978-3-642-39895-7. [6] Finkenzeller, K. RFID-Handbuch. Grundlagen und praktische Anwendungen induktiver Funkanlagen, Transponder und kontaktloser Chipkarten. 3., aktualisierte und erw. Aufl. München, Deutschland: Hanser, 2002. ISBN 3-446-22071-2. [7] Fleisch, E. Internet der Dinge. Ubiquitous computing und RFID in der Praxis: Visionen, Technologien, Anwendungen, Handlungsanleitungen. Berlin, Deutschland: Springer Verlag, 2005. ISBN 978-3-540-24003-7. [8] Eisenhauer, M., Oppermann, R., Prinz, W. Internet der Dinge. Anwendung von RFID- und Tracking-Technologien zur intelligenten kooperativen Assistenz im Arbeitsprozess. In: H.-J. Bullinger, M.t. Hompel, eds. Internet der Dinge. Www.internet-der-dinge.de. Berlin, Deutschland: Springer Verlag, 2007, S. 49-62. ISBN 978-3-540-36729-1. [9] Deutsches Institut für Normung e.V.: DIN 31051, Grundlagen der Instandhaltung. Berlin, Deutschland: Beuth Verlag GmbH. [10] Schulz, D., Gitzel, R. Seamless maintenance. Integration of FDI Device Management & CMMS. In: C. Seatzu, eds. Proceedings of 18th International Conference on Emerging Techologies & Factory Automation. ETFA'13. Piscataway, USA: IEEE, 2013, S. 1-7. ISBN 978-1-4799-08622. [11] Bernhard, J., König, A., Krupp, M., Pflaum, A., Strauß, W., Bamberger, R. Mobile Technologien im Vor-Ort-Service. Mobile Servicewelten im MaschinenbauVDMA-Verlag, Frankfurt a. M, Deutschland, 2010, 70-132.
[12] Abramovici, M., Krebs, A., Wolf, M. Approach to ubiquitous support for maintenance and repair jobs utilizing smart devices. In: I. Horváth, Z. Rusák, eds. Proceedings of the 10th International Symposium on Tools and Methods of Competitive Engineering. TMCE'14. Budapest, Hungary, 2014, S. 83-95. ISBN 978-94-61861-76-4. [13] Henderson, S., Feiner, S. Exploring the benefits of augmented reality documentation for maintenance and repair [online]. IEEE transactions on visualization and computer graphics, 2011, 17(10), 1355-1368. ISSN 19410506. doi:10.1109/TVCG.2010.245 [14] Lee, S., Akin, Ö. Augmented reality-based computational fieldwork support for equipment operations and maintenance. Automation in Construction, 2011, 20(4), 338-352. [15] Abramovici, M., Neges, M. Potentiale der Anwendung von kommerziellen Smartphones und Tablets für Augmented Reality unterstützende Wartungsaufgaben. In: J. Gausemeier, M. Grafe, Meyer auf der Heide, Friedhelm, eds. Tagungsband Augmented & Virtual Reality in der Produktentstehung. Paderborn, Deutschland: HNI-Verlag, 2013, S. 261-272. ISBN 978-3-942647-30-3. [16] Neges, H.-M. Smart Augmented Reality Lösung für die IndoorNavigation und Instandhaltung. Bochum, Deutschland: Bochumer Universitätsverlag, 2015. ISBN 978-3-89966-762-2. [17] Sanna, A., Manuri, F., Lamberti, F., Paravati, G., Pezzolla, P. Using handheld devices to support augmented reality-based maintenance and assembly tasks. In: Proceedings of the IEEE International Conference on Consumer Electronics (ICCE). Piscataway, USA: IEEE, 2015, S. 178-179. ISBN 978-1-4799-7542-6. [18] Manuri, F., Sanna, A., Lamberti, F., Paravati, G., Pezzolla, P. A Workflow Analysis for Implementing AR-Based Maintenance Procedures. In: L.T. de Paolis, A. Mongelli, eds. Proceedings of the first Augmented and Virtual Reality Conference. AVR 2014: Springer International Publishing AG, 2014, S. 185-200. ISBN 978-3-319-13968-5. [19] Zhu, J., Ong, S.K., Nee, A. A context-aware augmented reality assisted maintenance system [online]. International Journal of Computer Integrated ISSN 0951-192X. Manufacturing, 2014, 28(2), 213-225. doi:10.1080/0951192X.2013.874589 [20] Lamberti, F., Manuri, F., Sanna, A., Paravati, G., Pezzolla, P., Montuschi, P. Challenges, Opportunities, and Future Trends of Emerging Techniques for Augmented Reality-Based Maintenance [online]. IEEE Transactions on Emerging Topics in Computing, 2014, 2(4), 411-421. ISSN 2168-6750. doi:10.1109/TETC.2014.2368833 [21] Abramovici, M., Neges, M. Durchgängige Lösung zur Unterstützung von Wartungsprozessen durch Augmented Reality mit Smart Devices. In: R. Stelzer, eds. Tagungsband Entwerfen, Entwickeln, Erleben. Methoden und Werkzeuge in der Produktenentwicklung. EEE'2014. Dresden, Deutschland: TUDpress, 2014, S. 199-212. ISBN 978-3-944331-67-6.
