A model-based approach for data integration to ...

Computers in Industry 64 (2013) 376–391

Contents lists available at SciVerse ScienceDirect

Computers in Industry journal homepage: www.elsevier.com/locate/compind

A model-based approach for data integration to improve maintenance management by mixed reality Danu´bia Bueno Espı´ndola a,*, Luca Fumagalli b, Marco Garetti b, Carlos E. Pereira c, Silvia S.C. Botelho a, Renato Ventura Henriques c a b c

Center of Computational Sciences, Federal University of Rio Grande – FURG, Rio Grande, RS, Brazil Dipartimento di Ingegneria Gestionale – POLIMI (Politecnico di Milano), Milano, Italy Department of Electrical Engineering, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil

A R T I C L E I N F O

A B S T R A C T

Article history: Received 18 December 2011 Received in revised form 13 November 2012 Accepted 11 January 2013 Available online 28 February 2013

Facilitating interaction with maintenance systems through intuitive interfaces is a competitive advantage in terms of time and costs for industry. This work presents the CARMMI approach, which aims to integrate information coming from CAx tools, mixed/augmented reality tools and embedded intelligent maintenance systems. CARMMI aims to provide support to operators/technicians during maintenance tasks through mixed reality, providing an easier access, understanding and comprehension of information from different systems. Information about where, when and which data will be presented in interface are defined by CARMMI. The paper presents three test cases that were performed using the proposed concepts and infrastructure. The main benefit of the approach is to provide an extensive and generic model for the integration and management of maintenance data through the use of CARMMI. ß 2013 Elsevier B.V. All rights reserved.

Keywords: Industrial maintenance Mixed reality Product data management Data modeling/visualization

Industrial maintenance is a knowledge intensive field based on different disciplines involved by the various technologies included in modern industrial equipment. Thus, until now, the complexity of maintenance has been the main limitation to substantial improvements in the maintenance discipline. In fact, breakthrough advances in supporting technologies are needed to allow the maintenance engineering approach to fully develop its potential. Concerning this, new technologies based on ICT and microelectronics (such as intelligent data capture, advanced visualization interfaces, wireless and IoT (Internet of Things), and intelligent sensors) are supplying maintenance with the environmental intelligence needed to re-invent the way to do maintenance [1]. For instance, the combination between real and virtual components in a visualization environment, which is enabled by mixed/augmented reality techniques [2], has already been tested for facilitating maintenance activities such as guided repairing/ spare part substitution activities [3,4], inspections and guided troubleshooting [5,6]. However, the potential of these techniques could be deployed through a integration between data from

several sub-systems. The use of MR in the maintenance field is not new [7–9], though previous research works do not deal with how to manage the maintenance information or what data is necessary to be presented. The use of intelligent maintenance systems for predicting and monitoring the health of machines and their components integrated with mixed reality interfaces can provide a valuable support to maintenance operators to isolate and fix the equipment components that possibly will fail, based on the analysis of their behaviors’ degradation. This would help to reduce the maintenance time and costs caused by unpredicted breakdowns of critical industrial equipment, in which the failures may severely compromise the production process. In this sense, the paper intends to provide an approach for the development of a mixed reality interface not only supporting the visualization, but also providing an information management interface about diagnostics, planning and safety in maintenance tasks. CARMMI (an acronym with the Portuguese words for ‘‘CAx models integrated to Mixed Reality and Intelligent Maintenance’’) aims to integrate information coming from three different systems and allow the visualization of the information using mixed reality interfaces:

* Corresponding author. Present address: Av. Ita´lia, s/n, Campus Carreiros, FURG, Rio Grande, RS, Brazil. Tel.: +55 53 84512191/32935134. E-mail address: [email protected] (D.B. Espı´ndola).

- Maintenance systems for assessing the reliability of monitored product/equipment; - CAD systems for modeling and describing the physical information related to the structure of the product/equipment;

1. Introduction

0166-3615/$ – see front matter ß 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.compind.2013.01.002

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

- Virtual modeling systems for content creation about the information related to product/equipment. The contribution of CARMMI is to describe the integration between CAD, MR (mixed reality) and IM (intelligent maintenance) data. As Larsen [10], this paper proposes the solution development in three phases: idea phase (conceptual model), creation phase (CARMMI implementation), usage phase (testing case studies). Idea phase consist of idea percolation and molding activities (Section 3); creation phase is practically related with the development of the idea (Section 4). Finally, the usage phase is related to the initialization and maturation of solution. In this paper, Section 5 introduces the testing phase preliminary to usage phase. Future works will implement the interface in factory floor to validate the usage phase. Describing the relationship between these data is possible to provide an efficient improvement of the maintenance interfaces. This approach has been validated in two real industrial projects: (i) within the scope of an applied research project developed in Brazil and entitled ‘‘Intelligent maintenance applied to electrical actuators’’, in partnership with Transpetro/Petrobras, the major Brazilian company responsible for oil transport and storage, which manages thousands of kilometers of oil and gas pipelines in Brazil; (ii) within the scope of an Italian project involving Politecnico di Milano and Balance Systems company. Different test cases have been considered within these projects, addressing different maintenance situations and different kinds of manufacturing processes, in order to support the testing phase. An important challenge of this study is to provide a tool for the implementation of a mixed reality interface for maintenance systems able to adapt to different industrial applications. The CARMMI model proposes the intersection/integration between three different domains: CAx data, maintenance data and virtual (computer-based) data. This integration (as seen in Fig. 1) allows providing the virtual data of components monitored by maintenance system. Fig. 1 describes the three domains, detailing the meaning of each intersection among the domains. In the following sections, the main research questions are presented and discussed by presenting the domain integration. Based on these three domains, the CARMMI model describes the integration of the three systems for generating information to be shown on mixed reality interface. Through the management and integration model, it was possible to develop three types of interfaces for maintenance: (i) diagnostics, (ii) planning and (iii) safety interfaces.

377

(i) The diagnostics interface: allows a complete re-design of the troubleshooting approach for easily moving from the failure mode to the failure cause, allowing the reasoning on the faults, are shown. In fact, thanks to a complete immersion of the maintenance operator into the troubleshooting process, he/ she is an active part of the local or remote computer system managing the diagnostic process. Also, based on the deployment of the faulty condition and related equipment information from the maintenance system, the visualization interface is able to suggest/plan the repair/recovery solution. In this way, a computer-aided diagnostic system is supported by the interface. (ii) The planning interface: supports the setting up of all elements (i.e. fixtures, components, spare parts, etc.) needed for performing a maintenance activity. This will be supported by mixed reality through the interactive visualization of all materials needed to perform the maintenance activity on the real factory environment. Thus, the maintenance manager can visualize in advance any interference among factory equipment and the required maintenance hardware, verifying, with the support of the computer system, which equipment needs to be stopped, how the maintenance activity can be optimally organized, etc. This approach is similar to the ones used for planning operations for assembly purposes [11,12]. The refined plan is saved in the computer system, together with all the procedural and graphical information for later support of the maintenance operator when he/she needs to start the maintenance activity. (iii) The safety interface: provides a substantial contribution for solving one of the biggest remaining problems in the maintenance activity, which is the risk factor of human safety in maintenance work. In fact, even if many advances have been made over the last years thanks to the use of safety equipment and safety procedures, in many cases, maintenance work is still a dangerous activity. Regarding this, mixed reality interfaces, together with full information about the factory floor status, allow the maintenance operator to know and see the risk zones (i.e. pieces of equipment at high temperature, high voltage, high pressure, parts that could move, etc.). In order to keep workers safe, the real time availability of information about the status of field equipment is necessary. The configuration interface related to the monitored parameters by intelligent maintenance systems is presented through mixed reality interface.

Fig. 1. Domains involved in the definition of CARMMI model.

378

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

Although in this work the use of mixed reality interfaces has been applied to security, planning and diagnostics activities, the CARMMI model was conceived to be a generic solution for portable and general industrial applications that require the presentation of information from maintenance systems in order to streamline maintenance tasks and assist planning activities. Among the challenges of CARMMI’s conception were identified: to define which systems are necessary to provide the data useful for operator; to process previously some data (i.e. 3D model) to run the mixed reality interface. For instance: to use the management and integration model, it is necessary to have the equipment’s 3D model available through dwg, wrl and x3d format; monitoring data from maintenance system is also needed in a specific local previously configured in visualization interface. This means that a pre-processing of data must occur before the mixed reality interface to be started. CARMMI’s definition and description defines which data from CAx and IM systems are processed in real time and which must be pre-processed. The remainder of this paper is structured as follows: In Section 2, the mixed reality interfaces applied to maintenance systems and data models for maintenance processes are discussed. Section 3 introduces the CARMMI model and Section 4 describes the CARMMI model implementation: the tools, the hardware devices and visualization system. Section 5 validates the implementation through three case studies, while Section 6 discusses practical issues related to the test cases. Section 7 presents the future work perspectives and Section 8 concludes the work. For a good understanding of the proposal, it should be noted the main acronyms used in this paper: MR (mixed reality), AR (augmented reality), VR (virtual reality), IM (intelligent maintenance), CAx (computer-aided design/manufacturing/engineering CAD/CAE/CAM) and CARMMI (CAx models integrated to mixed reality and intelligent maintenance). 2. Theory and background Although there are several works related to the application of augmented/mixed reality techniques to maintenance (see for example: [13,14]), there are no studies that describe in detail the data management for mixed reality systems. In other words, it is not common to find research providing details on how, when, where and which information is brought to visualization interface or on how to relate the real and virtual environment. Indeed, before defining the data/information shared between real and virtual environment, it is necessary to know which data will be shown on the interface [15]. It is necessary keeping in mind that maintenance operations are performed in noisy environments, such as industrial plants, and the maintenance operator needs mobility and free hands to perform the operations, for these reason it is essential to develop intuitive interfaces and non-invasive. Thus, mixed reality interfaces are presented as a good possibility for these constraints, in order to simplify the maintenance activity and therefore reduce time and cost of these operations. The challenge regarding the definition of what virtual data/ information would assist the maintenance operator can be considered the main issue of this research, and it is specifically related to how, when, where and which information are presented to the operator. As a result, a literature analysis about the use of mixed reality interfaces and data models for managing industrial maintenance processes is presented. 2.1. Mixed reality in industrial maintenance Solutions of mixed reality that could be applied to maintenance applications in different industries were searched. Also proposals

that address the integration and management of information into mixed reality interfaces were investigated. The ARVIKA project [16] is one of the most cited in the literature. ARVIKA’s main objective was to use augmented reality technology for implementing an user-oriented and applicationdriven support of working procedures for the development, production, and servicing of complex products and systems during the product lifecycle. Differently of ARVIKA, the AMIRE project [17] described a model, based on use case diagrams, which shows the relationship between actors and tasks involved in the maintenance process. This approach aims to improve flexibility and reusability through a component-based framework in which the object oriented paradigm hides the programming complexity. Two case studies were presented: training in an oil refinery [18] and a guided museum tour. AMIRE results are tailored for users not skilled in programming. To this concern, it is worth considering that also when dealing with common users working in the maintenance field, it is quite common to have users not skilled in programming. With respect to AMIRE, citations were not found about data acquisition from maintenance and CAx systems, in the context of information extraction, i.e. which information to bring or to use. In terms of hardware and architectures solutions, two large projects are often cited: the ULTRA and the INT-MANUS. ULTRA [19] developed a lightweight, compact and portable hardware system that applies techniques of augmented reality in handheld PCs (Tablet and PDA). In this, the major contribution is the development of solutions for wireless connection and integrated remote support to mobile phones. The applications of this project were extended in multiple domains, from maintenance of complex machinery to construction, production and education. Also INT-MANUS [20] developed a similar platform called SCCP (smart connected control platform). SCCP is supported by an architecture based on agents, open and distributed, which contributes mainly on the flexibility and adaptability issues of industrial systems. The architecture provides three specific interfaces: a web-based interface, a mobile devices interface and an interface for virtual reality systems. The description of the architecture is divided into two layers: one that integrates machines, robots and other devices and another one integrating services that are connected via TCP–IP interface. The main focus was on communication issues. Concerning relationship models and integration solutions between maintenance and virtual systems, some studies were identified as useful for our research work. In Quarles et al. [21] an augmented reality application, developed for training on anesthesia machines for hospitals is shown. Even if not related to maintenance, the work described in Fishwick [22] relates to an integrative multi-modeling technique based on XML for establishing a relationship among different types of data. The integrative multi-modeling provides a human– computer interaction environment that allows components of different model types to be linked to each other. The concepts about models linkage, ontologies and data integration described in Fishwick [22] and Park et al. [23] were used and adapted as base for our proposal. Toro et al. [9] present knowledge based industrial maintenance system proposing the UDKE platform, which is based on four layers (User, Device, Knowledge and Experience – UDKE). This system focuses mainly on the information modeling, proposing two new ontologies extending the SOUPA (Standard Ontology for Ubiquitous and Pervasive Applications) [24]. An ontology for modeling the user experience called SOE (Set of Experience Knowledge Structure) and the AR ontology to model the augmented reality environment used to improve the maintenance experience. The proposed platform is described at a high level of abstraction and the paper does not

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

provide a detailed description of models and data related to maintenance. Another studies that focused on model definition and data integration is discussed in [25,26]. Lecakes et al. [25] presented partial results of a project whose goal was to show the advantages of using virtual reality for integrating measurements from multiple sensors. The model defined three data types for creating the virtual reality environment: graphs data, measurements data and functional data. Even if the data integration from multiple sensors were within the scope of this work, only data applied to virtual reality systems were related. To sum up, many systems and projects on mixed reality systems applied to maintenance were found. However, most of the studies are oriented to specific applications and, quite often, do not include a more detailed discussion on how the information management in mixed visualization interfaces is done. The next section describes the main models, platforms and standards related to industrial maintenance systems in order to investigate which maintenance data models can be used in this proposal in order to improve the content quality of proposed interface. 2.2. Models, standards and platforms for maintenance management The use of devices such as sensors, embedded systems and computing devices has given a level of intelligence to maintenance systems. This has led to a change from the ‘‘fail and fix’’ to a ‘‘predict and prevent’’ approach. This can be evidenced by the transformation map of maintenance strategies presented by Lee et al. [27]. The map presents an overview of future maintenance systems in terms of complexity and uncertainty of the equipment. In his work, trends in new methods for maintenance management such as resilient systems and self-maintenance systems are presented, which are able to better face fault situations. The basis for the development of these solutions is the Condition Based Maintenance (CBM) paradigm [28]. CBM policy can significantly reduce the number of unnecessary maintenance, since it is based on maintenance actions carried out only if needed and at the right time, avoiding untimely stoppages production stops. Integration of such policy into existing information system for the use in industry has been discussed in [29]. Systems that support this policy use sensor data to monitor the operation of the production machines. Basically, the analysis of the machine’s behavior is performed by the acquisition, processing and analysis of sensor signals, enabling mainly the understanding of the machine’s health state [30,31]. Within this context, several standards were developed to assist engineers in the development of platforms and architectures to implement CBM on the factory floor. A complete review of the state of the art, concepts, standards and platforms for maintenance was presented by Muller et al. [32]. The ISO-13374 (Condition Monitory and Diagnostics of Machines) [33] provides a specification for information flow and data processing for monitoring and diagnosis of equipment health. This standard divides the CBM systems in six functional modules: data acquisition (DA), data manipulation (DM), state detection (SD), health assessment (HA), prognostics assessment (PA) and advisory generation (AG). The OSA-CBM architecture (Open Systems Architecture for Condition-based Maintenance) [34] is an implementation in UML (Unified Modeling Language) of the ISO-13374 functional specification. The description using UML provides a model that allows multi-technologies implementations, which can be developed according to the specific application. The standard OSA-CBM describes the information content and information flow in the six specified modules; it can support analysis and deployment of CBM systems, as discussed in [34], and

379

support use of MR for maintenance, as discussed in [35]. However, its practical implementation requires the existence of a middleware such as CORBA or RMI, where the communication is not specified by the standard. The absence of standards for information exchanging between factory floor and the maintenance system was the factor that triggered the creation of the MIMOSA (Machinery Information Management Open Standards Alliance, www.mimosa.org) organization [34] and the subsequent development of the MIMOSA OSA-EAI (Open Systems Architecture for Enterprise Application Integration). The OSA-EAI standard is a relationship-based platform whose core is the CCOM (Common Conceptual Object Model) conceptual model. This model is implemented by the CRIS (Common Relational Information Schema) scheme. The MIMOSA-CRIS provides an implementation as a XML schema that allows the information sharing among different systems. The XML schema (XSD) is a relational format commonly used in most database systems for creating tables, e.g. in Oracle and SQL Server Microsoft through scripts. The MIMOSA contributions address mainly topics such as collaborative maintenance, integration and interoperability (see for instance [36,37]). However, the pattern does not support the information visibility of BOL (Begin Of Life) and EOL (End Of Life) as instead postulated by the scientific community, i.e. [38] and do not support models from CAx systems [39,41]. Since the idea of this work is to relate the virtual information with the maintenance information and CAx systems, the present work was based on PDKM (Product Data and Knowledge Manager) model. With this respect, the PDKM model [40,42] developed in the context of the PROMISE project (Product Lifecycle Management and Information tracking using Smart Embedded systems) was considered. It is implemented using UML and it is focused on the information representation of product items. The PDKM allows the identification and tracking of the different components of a physical product and to describe its characteristics of use and operation during the product lifecycle. Thus, during the literature review was possible to identify the main research questions and the following proposition was defined: ‘‘A mixed interface applied to maintenance operations must be controlled by a data management system that integrates three types of information domains: virtual data (which answers the question ‘‘how’’); data from intelligent maintenance systems (defining ‘‘when’’) and CAx data (respond to ‘‘where’’)’’. The integration of three data types will reply the main research question: ‘‘Which data/information provide to the operator?’’. The virtual data integrated maintenance systems can provide real time information about health status of the equipment, while CAx data provides detailed information on the physical system and thus provides information about where to operate. Moreover, data can be presented in a dynamic or static way. Dynamic data is information that changes over time; the static data are those that remain unchanged. Given the proposition is possible to describe the main data types and its characteristic static/dynamic and qualitative/ quantitative:  virtual data: can be both as in a textual form (txt), i.e. instructions to guide a given task (static and qualitative); as well in graphical form (jpg), i.e. temperature behavior of the component (dynamic and quantitative); 3D virtual models (wrl), i.e. the physical structure of the equipment (static and qualitative), etc.;  intelligent maintenance data: text (txt), i.e. maintenance historical (dynamic and qualitative); graphics (jpg), i.e. operation degradation of the component (dynamic and quantitative); spreadsheets (xls), i.e. values of temperature sensors, pressure (dynamic and quantitative);

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

380

 CAx data: 3D models (dwg), i.e. engine structure (static and qualitative); model attributes (xml), i.e. material, size, etc. (static and quantitative). However the integration of these three data types is not a trivial process. CARMMI is a proposal to address this challenge; it is presented in the next section. 3. The CARMMI model 3.1. Conceptual model The conceptual model represents, in the idea phase (see framework in [10]) a complete view of the proposed approach for integrating mixed reality and maintenance systems is depicted in Fig. 2. The diagram represents the functions by rectangles and the information flow using arrows. The three data domains (virtual data, CAx data, and intelligent maintenance) are represented in the conceptual model of Fig. 2 using circles. The circle at the extreme left (yellow) represents the functions and data flow of mixed reality system and central-left (blue) circle concerns CAx system. The extreme right (green) circle depicts the function and data from maintenance systems. Fig. 1 intersection represented in Fig. 2 by central-right (red) is, then, the CARMMI model. The mixed reality interface provides two kinds of visualization: automatic and guided visualization. The automatic visualization is triggered when the intelligent maintenance system identifies the possibility of a failure in some component. A guided visualization is performed by user command. This interaction is represented in Fig. 2 by bi-directional arrow between the user and the mixed reality interface (or mixed interface). The four areas delimited by circles in Fig. 2 details the idea presented in Fig. 1. The focus of this work is to develop the central circle (red) of Fig. 2. This area represents the integration and management functions that will determine which data to bring and present in the mixed interface. The integration process of the

models is divided into two phases to detail relationship and description. In order to relate the information, a XML description of the equipment structure and the data paths generated by the configuration interface are used. The phase related with description generates an XML file that lists the identifiers with the maintenance, virtual and CAx data. This XML file of relationship is used for information management. The other three circles of conceptual model represent the functions of CAx, IM and MR systems. The CARMMI (in Fig. 2 – red central circle) will extract the information from these systems to place them in interface. The extreme-left (yellow) rectangles in Fig. 2 describe the functions of the mixed reality system. Firstly the real environment is ‘‘captured’’ by the system and the tracking process of the real environment begins. The ‘‘tracking’’ phase should be responsible for calculating the position of virtual objects on the real scene. The ‘‘superimpose’’ phase of virtual components is the last phase to be carried out. To perform the overlay is necessary that the virtual content has been identified through information management model. Currently, the capture and tracking process, the data from CAx, the virtual and maintenance systems, are processed. The data relationship begins after the interface configuration is set by the user. At this stage the user defines and relates an identifier (marker, sensor or RFID) with the component name of which he/she wishes to visualize the information. From this stage, the system automatically sets the paths of maintenance, CAx and virtual data. The information management process involves two phases: ‘‘extract content’’ and ‘‘generation content’’. The ‘‘extraction’’ phase uses the XML file as a relationship map for information tracking. It is worth mentioning that the overlay of virtual elements is made in real time, but not the content generation. The audio, video and text models must be previously stored in specific locations of the database for subsequent presentation. The central-left (blue) rectangles in Fig. 2 describe the functions of the CAx systems. The ‘‘to describe structure’’ phase generates an

Fig. 2. The conceptual model. (For interpretation of the references to color in text, the reader is referred to the web version of this article.)

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

XML hierarchy of equipment parts which is used to relate information from several systems. The 3D virtual models (wrl or x3d) and components properties are used in the information management stage and they are generated respectively by ‘‘to convert CAD-RV’’ and ‘‘to describe attributes’’ stages. The functions of the maintenance system are represented by extreme left (green) rectangles. Basically the functions ‘‘data acquisition’’, ‘‘to manipulate data’’, ‘‘to assess’’, ‘‘to diagnose’’, ‘‘to prognosticate’’ and ‘‘to support decision-making’’ are based on the ISO13374 standard used in the most part of maintenance systems. Once extracted the information from these systems, the MR interface is created in accordance with requests made by the operator or by intervention of the maintenance system through alert messages when the failure prediction is identified. The next step is to technically represent the conceptual model by UML (Unified Modeling Language [43]), in order to further molding the idea. To this concern, proper class diagram is presented in next subsection. 3.2. The CARMMI model The CARMMI model describes technically the conceptual model. This model uses CAx (CAD/CAE/CAM) data for the integration of the intelligent maintenance (IM) system and mixed reality (MR) techniques. Fig. 3 represents the simplified CARMMI model. All data relationship of CARMMI is based on configuration and tags of the environment. To use a solution based on the CARMMI model, the first step for the operator is to use the visualization interface to configure the linking between tag (marker placed in the real environment) with the component and the respective sensor. When setting up the relationship between the component, the marker and the sensor, the main semantic link of CARMMI is established. This relationship is represented by the association of CONFIGURATION class with MANAGER_INTEGRATOR class and by aggregation relation of TAG_CONF class with the MANAGER_INTEGRATOR. The management and data integration class (MANAGER_INTEGRATOR) will relate the data to describe the relationships and to extract information from IM, CAx and MR systems to generate content to be displayed in the interface. The MANAGER_INTEGRATOR class searches the MI, CAx and MR data from the respective classes: MONITOR_MI (describes data and functions of

381

the intelligent maintenance system), DESCRIPTOR_MI (describes data and functions of CAx systems) and MIXER_MS (describes data and functions of the mixed reality system). The intelligent maintenance data are generated by MONITOR_MI class and are based on the PDKM description model represented by white classes (see next Fig. 4). In Fig. 4 only the PDKM model classes that have direct relationship with the CARMMI model are presented. The data from FIELD_DATA and RESOURCE classes of PDKM are the basis for maintenance planning. The central class of the CARMMI model is the MIXED_SCENE class (Figs. 3 and 4). This class describes the attributes and operations of the object mixed_scene. Objects of this class are composed by virtual products/components, real products/components and multimedia content. To render the scene, the MIXED_SCENE class inherits the methods of the mixer class (MIXER_MS). MIXER_MS is responsible for the capture (capture()), tracking (tracking()) and overlay (superimpose()) of content generated by the MANAGER_INTEGRATOR class in the visualization scene. The MONITOR_MI classes describe the data maintenance. For this, it has association with four classes of the PDKM model (FIELD_DATA, RESOURCE, EVENT and PHYSICAL_PRODUCT). From FIELD_DATA class is generated a text file with maintenance history that can be viewed in the listbox component of mixed reality interface. The data of RESOURCE class are used for maintenance planning. From EVENT class the status of maintenance activity can be obtained. And from PHYSICAL_PRODUCT class of PDKM model describes the physical component and has information about all stages of product lifecycle. The DESCRIPTOR_MI, MONITOR_MI and MIXER_MS classes describe three data domain. These objects: descriptor, monitor and mixer represent respectively: virtual and structural description of the CAD model; maintenance data related to CAD components, mixed visualization of scene. 4. CARMMI model implementation The procedures for developing the visualization system, using mixed reality, were the implementation of the methods/routines described in the model classes (Fig. 3). The main routines developed for implementation of this proposal are listed in Table 1 and refer to integration, visualization, maintenance and

Fig. 3. The simplified CARMMI model.

382

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

Fig. 4. The CARMMI model.

description routines. The maintenance and description procedures are implemented respectively by intelligent maintenance system and CAx system. The generated information by these systems are used by routines of management and visualization data. 4.1. Tools Despite the focus on maintenance operations, the need to integrate intelligent maintenance tools, CAx tools and mixed reality tools in a programming model, leveraging on the resources provided by the CARMMI, made it very interesting to address from a general point of view. To this concern, Table 2 outlines the used tools and their corresponding functionalities.

The modeling issues were implemented through three steps: (i) modeling and creation of the virtual model, (ii) complexity reduction and (iii) format conversion. Concerning the first step, the UML language was chosen to represent the systematic of MR use in the intelligent maintenance system. The programming interface was done using the C++ language on the Visual Studio platform. The implementation related to the superimpose of virtual elements in the real scene in runtime was performed using the OpenGL and ARToolkit libraries [44]. The OpenGL graphics library provided routines for rendering the scene and the ARToolkit library was used for the tracking and placement of virtual elements. The ARToolkit uses functions of image processing to detect markers in the captured scene. These markers

Table 1 Main classes and functions of CARMMI. Classes

Functions

MANAGER_INTEGRATOR (routines of data management) MIXER_MS (routines of visualization) MONITOR_MI (routines of maintenance) DESCRIPTOR_MI (routines of structure description)

relate(); describe(); extract(); generate_content() capture(); tracking(); superimpose() view_mandata(); view_access(); view_diag(); view_prog(); view_decision(); view_hist_FD(); monit_event() descr_attributes(); descr_structure(); convert_CADRV()

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391 Table 2 Tools used to CARMMI development. Domain

Tool/function

Mixed reality

ARToolkit (tracking, superimpose) OpenGL (rendering) VRML (virtual modeling)

CAx

SolidWork (cad model, CAD-RV conversion) Vizup (complexity reduction)

Intelligent maintenance

Watchdog (maintenance system) MATLAB (methods implementation)

CARMMI

UML (description) Visual Studio, C++ (interface programming)

are printed labels placed on the real components for which it is required to visualize information. 4.2. Hardware devices The use of hardware devices for interaction such as Head Mounted Display (HMD), i.e. three-dimensional goggles, data gloves, Cave Automatic Virtual Environment (CAVEs) and mobile devices such as mobile phones, Tablet PC and Personal Digital Assistants (PDAs) increase the ability of user interaction with the visualization interface [45]. These devices are commonly integrated into mixed visualization systems to provide a greater immersion sense, communication facility and human–computer interaction [46]. The input devices are particularly important for the development of mixed reality environments, such as: camera (or webcam), microphones, keyboards and mouse. The camera function is for capturing the real environment. The keyboard and mouse are conventional devices to interact and are often used to send commands to the MR systems. The microphones are used to send requests through voice commands. The microphones use enables operations on the environment where the operator needs to be hands free. The sensors are used for data acquisition of vibration and temperature from machines components and are monitored by the data acquisition system. The placing position of virtual objects is accomplished by using devices such as RFID, sensors or markers (printed labels detected by image processing). The output devices used for the mixed reality visualization can be: Tablet PC, mobile phone, PDAs, etc. Another possibility are the three-dimensional glasses with transparent lenses called HMD see-through, which allow visualizing the real scene with overlapping of virtual objects. The headphones are used to receive audio instructions in guided interaction in the mixed environment. In test cases presented in this paper, were used Head Mounted Display and notebook as output device, connected to processing hardware (notebook). 4.3. The visualization system The visualization system has three types of predefined application: diagnosis, security and planning, according to the support to be given to the main areas in which MR can allow a step change in maintenance. For each application there are two visualization modes: mixed and virtual, which can present two types of navigation: automatic or guided. In this paper we will focus on system interfaces that present mixed reality visualization for diagnostic and security applications. It is worth of mixed reality is when the real environment is captured by camera and virtual elements are superimposed by output devices in real scene, while the virtual reality shows only virtual elements in scene. The automatic navigation mode presents

383

the virtual contents in the interface without any user intervention, while the guided mode is managed by commands from the operator. The whole system has been developed taking into account the fact that operators are not programmers neither experts in the computing field. Moreover, the fact that the environment is designed for several types of users could affect the display options. Reports, meetings and experiments on the factory floor have shown that as much as reduced and simple the menus are, better is the interaction, greater is the acceptability, shorter will be the subsequent training time for tool use and more efficient is the use of the visualization system (according discussed in Table 4). Therefore, the design of each interface has been built jointly with specialists and maintenance operators from the different industries that collaborated for the testing phase. The initial interface of CARMMI system has three menus: the Application, Configuration and Help menu. The Help menu displays the system manual. The Configuration menu can be changed when needed by the user; however its values are set up as default. In addition, the visualization and navigation mode of application can also be configured on this interface. The menu, the navigation type, the TAG, and device type can be configured. The default values for applications are respectively: guided navigation, TAG marker, mixed visualization and desktop device. For each marker the operator should relate the equipment part, of which he/she wants to visualize the information, and the sensor that monitors the component conditions and send them to the intelligent maintenance system. The paths (location) for each data type must be previously configured. Fig. 5 shows the configuration interface. The parameters for diagnostic, security and planning applications can also be updated via the configuration interface. With a practical point of view, in the remainder of this section, the steps for possible use of the system based on CARMMI model are presented. Clicking in the menu, the user can open a previously saved application or open a new type of application: diagnostic, security or planning. The diagnostic application allows the operator to identify the plant devices (real scene) that require maintenance in an urgent way. With this visualization, it is possible to start the previously planned maintenance tasks in accordance with the current state of equipment operation. The CAx and maintenance data integrated into the CARMMI system are generated currently. The execution mode of the CAx system and the intelligent maintenance system is offline, i.e., the IM and CAx outputs are not generated at runtime (during the mixed reality visualization). In fact, there is a pre-processing activity in which maintenance data and CAD models used by the CARMMI system are previously stored and mapped in a XML relationship file. On the other hand, the mixed reality system is online, this means that the virtual data are superimposed at runtime, according to the commands and navigation type chosen by the user. After identifying the components that are in degradation condition in the scene, it is possible to request information from each machine individually. Once chosen the equipment, the following contents are provided: maintenance guide, maintenance history and equipment manual. The presentation order of content in the MR interface is managed by CARMMI model, more specifically by the CONTENT_ TAG class. This sequence can be random or ordered. When navigation is guided, the order depends on commands, i.e. the content is presented according to user request. When navigation is in the automatic mode, the presentation order is given by the configuration order of the application. This implies that the system must be designed and configured with the aid of maintenance

384

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

Fig. 5. The configuration interface.

specialists, who must define in advance the presentation order of content. The maintenance guide is textual instructions for the user that are shown following the execution order. These instructions are presented in text format and must be previously stored in the default locations of maintenance data. Virtual CAD models of the internal structure of components and assembly and disassembly models are presented along with the guide, when available. Moreover, to better understand the instructions, audio and video resources can be made available, audio and video icons can be enabled on the interface to this end. The other visualization interface for the equipment is the maintenance history. This resource is part of the diagnostic

contents and must be used to visualize the equipment behavior along time. In this interface, the data of FIELD_DATA class concerning the maintenance history are presented and the operator can update the values during the operation. Three visualization types are available (see Fig. 6): (i) the mixed visualization (up-right), which presents textual data overlaid markers and 2D graphics related to TAGs; (ii) the virtual model (3D CAD) of the component (down-right) chosen in the listbox (list located down-left); (iii) the monitoring data regarding the selected component (2D graph, down-left).

Fig. 6. Diagnostic interface.

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

The data update by operator must be performed through the interface in guided navigation mode using the desktop device as can be seen in Fig. 6. The following information of maintenance history are presented in the interface: COMPONENT (component name), WHO (operator identity), WHAT (defect type), WHERE (defect location), WHEN (maintenance date), WHY (defect cause) and WHICH (parts required for operation). Concerning safety resource, the system has a similar diagnostic interface for visualizing the equipment conditions. The difference is that the operator will see only the equipment location whose values exceed the thresholds defined in the configuration interface (see Fig. 5). By monitoring these values, it is possible to visually identify the danger areas on the factory floor. The parameters presently monitored by the system are: torque (in order to check possible overload), temperature and voltage. The user according to the tested environment can calibrate the thresholds. As for the diagnostic interface, the visualization of security application points out by arrows the devices that exceed the thresholds of torque, temperature or voltage, indicating the corresponding plant locations as possible dangerous areas for the user. It should be noted that every location is identified through the TAG. An important feature of the CARMMI is the information availability: the efficiency of visualization depends on the amount and quality of information provided by the operator and on the outputs provided by the CAx and maintenance system. This implies a higher possibility of knowledge generation and a greater number of available resources for visualization. It means, the efficiency of visualization depends directly on the feedback data made by specialists, engineers and operators of maintenance area. 5. Testing phase Laboratory tests were performed for three types of interface: conventional interface (desktop), virtual reality interface (desktop) and mixed interface (HMD). These tests are presented in subsequent sections and the end of each subsection is presented issues from dimension and variables (see Table 3) identified in Section 2 and further described. Once identified from the literature the potential usefulness of the virtual and mixed reality environment for visualization of the maintenance systems, it is worth identifying some variables to express the efficiency, safety and operability of the interface, in order to assess if this proposal satisfy the expectations mentioned in literature for VR and MR. In this sense the following table describes the variables used in this study as baseline for testing the solution and make comparisons across the cases. Variables defined will be used for testing. The obtained results depends on developed interface. The variables assess the efficiency of the visualization system as well as operator safety and operability of the interface. To quantify the efficiency of the proposal we measure the operating time. The operator safety during the maintenance activity is determined by the mobility and

Table 3 Dimension/variables of the MR interfaces. Dimension

Variable

Efficiency

Operating time

Safety

Operator mobility Invasiveness of technology

Interface operability

Navigability Appearance Usability Content quality

385

Table 4 Evaluation of interface operability. Usage

Navigation

User User User User User

1 2 3 4 5

Appearance

Content

T1

T2

T3

T1

T2

T3

T1

T2

T3

T1

T2

T3

3 3 3 2 2

4 4 3 3 3

4 3 4 4 2

2 3 2 2 2

4 3 2 4 2

4 4 3 4 2

3 2 2 2 2

2 3 2 3 2

4 3 4 3 3

3 3 2 3 3

3 3 2 3 3

4 3 3 4 2

invasiveness of the display device used. Finally, the interface operability is tested by applying a questionnaire in each test case regarding the navigation, usability, appearance and content quality discussed in Section 6 – Table 4. 5.1. Test case 1 – conventional interface The first tests were conducted to evaluate visualization interfaces of maintenance systems based on use condition of the equipment (CBM systems). Most surveyed solutions provide a visualization output through 2D graphics. These graphical should be interpreted by specialists and operators, which lead to data knowledge and interpretation needs and an increase learning time of the visualization tools and complicate understanding the results of data processed by CBM systems. Furthermore, conventional interfaces do not provide characteristics of operability desirable for use in the factory floor. Fig. 7 shows the interface of the intelligent maintenance system (Watchdog Agent was used, see [47,48] for further information on this tool). In this interface is possible to select the desired actuator from actuator list or to train and verify the confidence value of all actuators. For each actuator, a confidence value chart that indicates the behavior of the analyzed component is generated. Ranges of values determine three states: Normal, Degradation and Failure. In this example it is possible to observe the behavior of the HV20073 actuator whose threshold confidence value for normal behavior is 0.7 (Fig. 7 – right). For maintenance data visualization on the factory floor, this type of interface can compromise the efficiency of the system in terms of operating time since the data are not related to the physical component and should be interpreted by the maintenance operator before performing task. Regarding the operator safety, being the display device a monitor connected to processing hardware, the interface does not supports devices such as 3D glasses that give mobility to the maintenance operator. The interface operability is easy to navigate and easy to use. As for content quality, this interface not integrate CAx data and virtual to provide meaning for information. 5.2. Test case 2 – virtual interface The second test case is related to the use of the CARMMI model for integrating CAD data with maintenance data. The equipment manual and the maintenance instructions handbook of the machine-tool produced by Balance Systems were used as source of maintenance data. After building the models, the operation of removal and placement of the spindle motor located in the milling_head was attended and filmed. Fig. 8 shows the operation of removing the motor on the machine during a maintenance task of the equipment (left) and the virtual interface tested to support maintenance activity. Once stored the data supplied by the company, it was possible to validate the implementation stages of the proposed approach. First, the XML description of the equipment components from CAD model of the machine was generated and the relationships of

386

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

Fig. 7. Conventional interface of maintenance system.

components with maintenance and virtual data were established. Because the equipment did not have an integrated intelligent maintenance system, maintenance data were extracted from the text file documentation provided by the company. As for the virtual data, the virtual 3D models in wrl and 3dxml format were used. These steps covered the modeling and monitoring stages of implementation. The interface programming stage provided the diagnostic interface in virtual visualization mode with guided navigation based on a desktop device. The TAGs placement for mixed visualization has not been performed in this test case. Through the use of this interface, the operator can select the equipment component in the listbox and load the visualization using the button. Clicking the button the virtual models of the component and equipment are presented. At this point, if there are videos about the way to perform the maintenance task, they are shown. Moreover, in the textbox, the maintenance operations related to the component are presented in the interface. The interface appearance is closely related to the number of features it offers. The greater the number of visualization options, navigation menus, checkboxes, etc., becomes more polluted interface impairing understanding and shifting the operator attention focus. Thus, the interface that will be used in factory

floor must have a minimum of resources to make navigation fast, intuitive and useful. Despite the content quality of this interface is good, because it has integration to CAD systems and generate virtual models for the maintenance operation, the polluted appearance has damage the operation. The operator safety depends on the device type used to help task and warnings about dangerous in the plant. 5.3. Test case 3 – mixed interface This test case was carried out within the scope of the intelligent maintenance (IM) project with the companies Coester and Petrobras. A bench instrumented for testing torque and vibration on electric actuators by Coester was used and the instrumentation is based on Kistler accelerometers, positioned along the bench, in the actuator CSR6. The overall bench is controlled by the CompactRIO NI cRIO-9004 device of National Instruments. The purpose of creating these benches is to simulate various situations that occur in the factory floor by providing the possible equipment behaviors and thus allowing to simulate different situation where to test an intelligent maintenance system to provide support for better repair or properly facing degradation situations. It is worth mentioning that, in a larger perspective that the one presented in this paper, on these test benches, presently,

Fig. 8. Virtual interface using CARMMI model.

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

387

Fig. 9. Mixed interface.

tests of systems integration and advanced visualization use in interfaces for better planning and execution of maintenance operations are being carried out continuously with periodic visits of workers/engineers of the industrial company. This test was done to validate the system based on CARMMI model. The degradation process in the activity of opening and closing the valve was simulated for to test the mixed reality interface showing the data from IM system. This type of degradation is one of the main problems that occur in Petrobras, resulting in high damage for each minute of downtime, which justifies the need for the establishment of an intelligent maintenance system. To simulate the degradation process, primarily, the axis of the electric actuator was aligned with the central axis of the bench. To generate a situation capable of producing an increased torque on the spindle of the electric actuator, the bench was purposely misaligned. The data generated by the intelligent maintenance system, are accessed through the mixed reality interface. Through the CARMMI model, data from the CAD model of the actuator, developed in Solidworks, are integrated with information of maintenance system, coming from the CompactRIO device, and finally they are presented by the visualization interface. One of the system interfaces was tested by user in mixed visualization mode using the HMD device, as shown in Fig. 9 (right). Each one of the five markers visualized in the scene (see Fig. 9) was related to components of the bench during the process of interface configuration. The ‘‘D’’ marker was related to the gearbox of the actuator and its corresponding accelerometer for torque measurement. As technical detail, the sensor related to this component was the 8705A50M1 accelerometer which guarantees a correct transformation of mechanical vibration of valve body in electrical voltage. The data shown on the scene are: 2D graphics from IM system, and textual information from the CAD model providing the name of the component. For interaction, the conventional device type keyboard is used. The keys PgUp, PgDn, Home, End and Enter are used for navigation in the interface. Voice commands will be used in future for a more flexible interaction. The lower left menu presents the virtual model of the equipment as decomposed in disassembly order. Besides, a listbox component presents the history of equipment degradation in text format. The navigability, usability and appearance of the interface showed good results in tests with maintenance professionals. Because the mixed interface presents a predominance of the real environment and few virtual elements to aid in the operation, this makes the interface less polluted and more intuitive for the

operator providing thus a decrease in operating time. Furthermore, the utilization of HMD device provides security and allows the operator to keep hands free during operation. The integration with the maintenance system ensures safety on operations risk and integration with CAx system allows to relate the real environment with virtual information. 6. Discussion In this section practical issues concerning implementation on the field of the proposed system are discussed. In the test cases performed in a laboratory the risk parameters were controlled easily. It was enough to configure the security interface and use the intelligent maintenance system integrated to the visualization system. However, in the real experiment in the plant environment, the equipment is run with all other equipment and it is necessary to monitor the whole plant. Real data concerning temperature and pressure variables are essential to operator security. These parameters must be continuously controlled and acquired during the maintenance operation through devices such as RFIDs and sensors and then compared with standard factory values. In this context, the CARMMI interface must guarantee a safe environment during operator activity and ensure that actions are performed without risk. Through the test cases, it was possible also to validate the CARMMI model using the visualization system. However, more use tests on the factory floor are necessary for improving the interfaces’ designs, in addition to obtaining operator requests for enhancing system preparations for implementation. It is worth mentioning that operators should be trained to operate with the support of the VR and/or MR environment. Proper procedures should also be established in order to include the utilization of the new tools in the daily practice of the operators. This can somehow result in an initial difficulty in adopting the solution and working with the VR and MR environment. Nevertheless, the new environments will enable easy access to necessary information to execute the maintenance task in plants. Indeed, the information will be improved in the following ways: (i) the facility to access the right information, avoiding wasting time in research, (ii) the possibility to access information at the factory floor level, without looking for the information in the company intranet or in another repository where proper documents can be found (e.g. CAD model/drawing). Moreover, once the system is properly set up, the information available in the VR or MR environment is selected automatically for the task objective of the operator. This means that only the part of

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

388

information really important for the tasks of the maintenance operator are shown to her/him. In fact, in the common industrial practice, the difficulty to find the proper information often leads the operators to work based on their experience. In order to analyze the proposed system, a short questionnaire to assess the operability of the interface was applied in terms of navigation, usage, content and appearance of the interface.  Navigation: the menus and buttons displayed on the interaction interface serve the demands of the maintenance guide?  Usage: the system is easy to understand and presents intuitive use?  Appearance: the screen presented is sufficient for the navigation environment?  Content: the data acquisition and the presented data are enough for maintenance activity?

Five users were interviewed, four from maintenance and one from computer graphics. To describe these analyses, qualitative answers were solicited in the following model: (1) unsatisfied (2) a little satisfied (3) satisfied (4) very satisfied. Table 4 shows the level of satisfaction indicated by users. Then, besides consideration about the possibility for operators to access more information, in order to summarize the comparison between the test cases in terms of more detailed parameters, the dimensions presented in Table 3 are considered. Table 5 presents the variables analyzed in test case. The symbol ‘‘+’’ means major contribution and ‘‘ ’’ means minor contribution. It is important to mention that the dimensions are analyzed in laboratory tests, but considering the common constraints of applications that involve maintenance activities on factory floors (i.e. hard environment for safety and mobility of the operators). As mentioned when introducing the dimensions for the comparison, the interface operability is tested by applying a questionnaire (see Table 4) in each test case, regarding the navigation, usability, appearance and content quality. It is observable in the above comparison that the mixed interface, due allowing the use of mobile devices, enables mobility and safety for the operator. The predominance of the real scene and the use of a few virtual components provide good navigability and an intuitive use of the interface. The integration with CAx and maintenance systems provides good quality content, since the data relationship adds value to the information presented. As already discussed, only correct information and images useful to the operator are shown. Finally, through the third test, it was possible to confirm that the use of the mixed interface using a data management and integration model (i.e. CARMMI model) provides greater efficiency for the operator during the maintenance activity. Table 5 Comparison of test cases. Test case 1 – conventional interface

Test case 2 – virtual interface

Test case 3 – mixed interface

Efficiency Operating time

+

Safety Operator mobility Invasiveness of technology

+ +

Interface operability Navigability Usability Appearance Content quality

+

+

+ +

+ + + +

7. Future work The use of advanced visualization interfaces with virtual and mixed reality support is becoming a practical alternative due to the popularization of mobile devices. Devices such as tablets and mobile phones enable a way to visualize, in plants, virtual information during activities on the factory floor. Thus, mobile technologies can work as display devices for the use of mixed reality techniques. The CARMMI model importation into operating systems, such as Android and OSx, is being investigated for the application development of mixed visualization for mobile devices. Some challenges of CARMMI use on mobile devices are cited: speed/processing power and data transmission in real time over wireless networks. The 3D rendering of models and data importation from IM and CAx systems, through wireless networks, require speed, processing and storage capacity. However, advances in smaller, lighter solutions that have high processing power should overcome technological barriers soon. Future researches about markless techniques for tracking are necessary. The ability to superimpose virtual elements in real environments without the use of markers provides less intrusive solutions; i.e., through image processing in real time (by detecting edges, shapes or patterns of their own image), virtual elements are superimposed on real scenes. This study is necessary because placing labels in industrial environments is not a simple task. The presence of industrial waste, high temperatures of equipment, exposure to weather, noise and magnetic fields sometimes impede the placement of markers. Researches about communication and storage areas should be investigated to improve the CARMMI model. Solutions through sensor grids for wireless communication and expert databases for information storage should be developed to provide support for CARMMI. The data organization and information extraction is critical for efficient data mining and fast information access. Moreover, the feedback from maintenance specialists during the execution of tasks, in plants, should be stored in a database for posterior generation of historical procedures. Regarding the maintenance planning aspect, it is important to consider data related to the equipment lifecycle; i.e., PLM (product lifecycle management) data must be analyzed for effective planning. Future work concerning the PLM database integration with the CARMMI model also should be considered. The emergence of intelligent products with storing and processing capacity is another area of research, which advances looking for solutions in communication and interoperability in order to make different systems, humans and machines communicate and understand each other [49]. Although there is a long way to go before achieving this interaction level between humans and devices, research on intuitive interfaces, semantic web and ontologies plays an important role in the future of pervasive computing. Thus, the semantic description of classes from CARMMI must be reconsidered to provide integration with product embedded information devices. Finally, assessment tools of mixed interface managed by the CARMMI model must be created and applied in the industrial environment for opinion report acquisition from technicians and maintenance managers. Based on usability testing of interface, improvements in terms of design and content can be enhanced. Feedback from interface users provides means to identify needs and establish requirements for a user-centered approach of interface. Thus, interviews, surveys and inspections of heuristic evaluation should be conducted in the future for systematic data analysis and quantitative analysis about interface usage. Thus, a more accurate conclusion can be achieved through statistical analyses of the results.

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391

8. Conclusions The possibility to visualize virtual objects mixed with the factory environment in real-time during the maintenance activity was made possible by the proposed approach based on the CARMMI model. The mixed scenarios based on the CARMMI model incorporated information from CAx, intelligent maintenance system and virtual systems. The adopted approach proposed models for the concept, the required hardware devices and software tools, the data structure and the visualization platform. The conceptual model of CARMMI defined required functionalities, through a diagram describing functions, information flow and involved systems. The hardware devices and software tools necessary for data processing and related to the visualization, storage and communication activities were also defined. Finally, the UML language was used for describing the integration and management of data, based on the conceptual model. This data model (described by CARMMI), was used as a data structure for the development of the visualization system using mixed reality. Three test cases were conducted to validate the proposal: the first one was developed in laboratory based on conventional interface of a maintenance system; the second experiment was based on a testing activity performed on a machine-tool from the Balance Systems company and the last was based on oil valves used by the Petrobras company. The test cases are still in progress and further results are to be expected as part of future work to implement the solution in the plant, making it available for possible daily usage by the operators. Among the challenges encountered during the conceptual stage, including testing and validation of this research work, some stand out and are worthy of mentioning:  the order of content presented in automatic visualization mode;  the interface interactivity;  the information availability from different systems involved.

In the automatic visualization mode, the operator visualizes the sequence of maintenance operations without any intervention by the user. This type of visualization was created for situations when the intelligent maintenance system detects the failure possibility (degradation) and automatically shows failure locations and maintenance operations to be performed to prevent the failure from occurring. However, the programming of this type of interface must be developed with the operator. In fact, the operator or maintenance expert must define (set) the presentation order of the content in the configuration interface before using the system. Therefore, future work could refer to the implementation of cause-effect relationships or another priority evaluation approach as sorting methods for content presentation. Another implementation issue is related to interactivity that depends directly on the type of interaction devices. Using the HMD device, the options for user interaction in mixed visualization mode are limited to the keys PgUp, PgDn, Home, End and Enter, thus, limiting the input values provided by the user and restricting the interaction ways. Implementations for using voice commands with mobile devices can be considered; however, communication challenges also arise with this technology. It requires the use of HMD see-through (translucent) glasses with wireless communication for greater flexibility of interaction in mixed visualization interfaces. In this context, future work in the area of communication solutions through the use of IoT (Internet of Things) technologies [48], should be investigated as a possible solution for mixed reality interfaces applied to maintenance.

389

An important aspect identified in this approach validation is the information availability from CAx, maintenance and virtual systems. The visual resources provided by CARMMI depend directly on the output of the three systems involved in the proposed solution. The more complete the system databases are the better the interface potential is, i.e., if the databases are frequently updated by the maintenance operator, the greater the extracted knowledge from mixed visualization interfaces will be. A solution to this problem can be to integrate the CARMMI model with the database server from the three systems. During the development of the study, the contributions from operators and maintenance experts were essential for the definition of the fundamental requirements of visualization interfaces. In most cases, it was required to simplify the menus and buttons for interaction, in such a way that the operator would need minimum knowledge about the system to put in use. Thus, the interface becomes intuitive and available for various types of users. Besides, improvements of the proposed approach are needed to evaluate the use of other tools and techniques related to mixed reality and to improve issues of interactivity. IoT technologies, RFID tags for component localization and advanced tools for intelligent maintenance should be assessed. Furthermore, usability tests of the interfaces should be performed when implementing the system, not in a lab environment, but on the plant with the support of maintenance specialists for defining a interface standard. The combination of mixed reality, with embedded maintenance and prognostics systems and their integration with CAx models, as proposed in our approach has a very positive impact on the operator’s role. By recognizing specific markers on devices and equipment, maintenance information related to that equipment/ device can be automatically accessed and graphically presented as an AR view, superimposed on the real devices. This allows operators a better and faster understanding of the situation by making some information, which would usually be hidden inside the equipment. The proposed concepts and tools not only allow a reduction in the execution time of operators’ tasks, but they can also increase the reliability of operators’ actions (due to a more precise indication of the degradation process of some components) as well as improve operators’ safety (given that dangerous situations and areas can be better communicated). At last, it is important to note that the approach aims at automating the mixed visualization process through the use of a relationship solution of three system proposed by the CARMMI model in order to make this process independent of specific platforms and tools. References [1] L. Fumagalli, D. Elefante, M. Macchi, B. Iung, Evaluating the role of maintenance maturity in adoption of new ICT in the process industry, in: In IMS’08: 9th IFAC Workshop on Intelligent Manufacturing Systems, Szczecin, Poland, October 9–10, 2008, 2008. [2] P. Milgram, F. Kishino, A taxonomy of mixed reality visual displays, in: IEICE Transactions on Information Systems, vol. E77-D, No. 12, December, 1994. [3] H. Regenbrecht, G. Baratoff, W. Wilke, Augmented Reality Projects in the Automotive and Aerospace Industries, Published by the IEEE Computer Society, IEEE, November/December 2005 (0272-1716/05, 2005). [4] S. Henderson, S. Feiner, Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret, in: IEEE International Symposium on Mixed and Augmented Reality 2009 Science and Technology Proceedings, 2009. [5] S. Henderson, S. Feiner, Opportunistic tangible user interfaces for augmented reality, IEEE Transactions on Visualization and Computer Graphics 16 (1) (2010). [6] J. Song, Q. Jia, H. Sun, X. Gao, Study on the perception mechanism and method of virtual and real objects in augmented reality assembly environment, in: The 5th IEEE Conference on Industrial Electronics and Applications, 2009. [7] B. Schwald, B. Laval, An augmented reality system for training and assistance to maintenance in the industrial context, in: The 11th International Conference in

390

[8]

[9]

[10]

[11]

[12] [13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24] [25]

[26]

[27]

[28]

[29]

[30]

[31]

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391 Central Europe on Computer Graphics, Visualization and Computer Vision’2003, Plzen, Czech Republic, 2003. S. Hassaine, K. Dhambri, H. Sahraoui, P. Poulin, Generating visualization-based analysis scenarios from maintenance task descriptions, in: 5th IEEE International Workshop on Visualizing Software for Understanding and Analysis, 2009, VISSOFT, 2009. C. Toro, J. Sanı´n, Vaquero, J. Posada, E. Szczerbicki, Knowledge based industrial maintenance using portable devices and augmented reality, in: 11th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems – KES2007, Springer-Verlag, Berlin, Heidelberg, 2007, pp. 295–302. T.J. Larsen, Information systems innovation: a framework for research and practice, in: T.J. Larsen, G. McGuire (Eds.), Information Systems Innovation and Diffusion: Issues and Directions, Idea Group Publishing, Hersey, PA, 1998 , pp. 411–434. A. Liverani, G. Amati, G. Caligiana, A CAD-augmented reality integrated environment for assembly sequence check and interactive validation, Concurrent Engineering 12 (67) (2004), http://dx.doi.org/10.1177/1063293.X04042469. P. Georgel, P. Schroeder, N. Navab, Navigation tools for augmented CAD viewing, in: IEEE Computer Graphics and Applications, 2009. S. Benbelkacem, et al., Augmented reality system for e-maintenance application, in: Mediterranean Conference on Intelligent Systems and Automation, 2, 23–25 March 2009, Zarzis, Proceedings, Melville, vol. 1107, (2009), pp. 185–189. , http:// dx.doi.org/10.1063/1.3106470. M. Macchiarella, D. Vincenzi, Augmented reality in a learning paradigm for flight and aerospace maintenance training, in: IEEE Digital Avionics Systems Conference, vol. 23, 2004, Salt Lake, Proceedings, IEEE, Pistacataway, 2004, pp. 5.1.1– 5.1.9. , http://dx.doi.org/10.1109/DASC.2004.1391342. D. Espı´ndola, C. Pereira, R. Henriques, S. Botelho, Using mixed reality in the visualization of maintenance processes, in: 1st IFAC Workshop on Advanced Maintenance Engineering, Services and Technology – IFAC A-MEST’10, 2010, Lisboa, Proceedings, 2010, pp. 35–40. W. Friedrich, D. Jahn, L. Schmidt, ARVIKA – augmented reality for development, production and service, in: Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR’02), 2002, pp. 3–4. P. Grimm, M. Haller, V. Paelke, S. Reinhold, C. Reimann, J. Zauner, AMIRE – authoring mixed reality, in: The First IEEE International Augmented Reality Toolkit Workshop, Darmstadt, Germany, 29 September 2002, 2002. M. Tra¨skba¨ck, et al., Mixed reality training application for an oil refinery: user requirements, in: ACM SIGGRAPH International Conference On Virtual Reality Continuum and its Applications in Industry – VRCAI2004, Singapore, (2004), pp. 324–327. A. Makri, J. Weidenhausen, P. Eschler, D. Stricker, O. Machui, C. Fernandes, S. Maria, G. Voss, N. Ioannidis, ULTRA light augmented reality mobile system, in: Proceedings of the International Symposium on Mixed and Augmented Reality – ISMAR2005, 2005. M. Foursa, D. D’Angelo, R. Narayanan, Innovative control system for industrial environment, in: Fourth International Conference on Autonomic and Autonomous Systems IEEE, 2008. J. Quarles, S. Lampotang, I. Fischler, P. Fishwick, B. Lok, A mixed reality approach for merging abstract and concrete knowledge, in: IEEE Virtual Reality, Reno, NV, (2008), pp. 27–34. P.A. Fishwick, Toward an integrative multimodeling interface: a human–computer interface approach to interrelating model structures, Simulation 80 (9) (2004) 421–432. M. Park, P.A. Fishwick, Integrating dynamic and geometry model components through ontology-based inference, Simulation (2005) 795–813. , http://dx.doi.org/ 10.1177/0037549705064359. H. Chen, T. Finin, A. Joshi, The SOUPA ontology for pervasive computing, in: Whitestein Series in Software Agent Technologies, Springer, 2005. G. Lecakes Jr., J.A. Morris, J.L. Schmalzel, S. Mandayam, Virtual reality platforms for integrated systems health management in a portable rocket engine test stand, in: IEEE International Instrumentation and Measurement Technology Conference, Victoria, Vancouver Island, British Columbia, Canada, 2008. M. Russell, G.D. Lecakes, S. Mandayam, J.A. Morris, M. Turowski, J.L. Schmalzel, Acquisition, interfacing and analysis of sensor measurements in a VR environment for integrated systems health management in rocket engine tests, in: IEEE Sensors Applications Symposium, New Orleans, LA, USA, 2009. J. Lee, Design of self-maintenance and engineering immune systems for smarter machines and manufacturing systems, Invited Keynote Paper for IMS 2010 – 10th IFAC Workshop on Intelligent Manufacturing Systems (IMS’10), which will be held in Lisbon, Portugal, July 2010. M. Lebold, M. Thurston, Open standards for condition-based maintenance and prognostic system, in: Proceedings of MARCON 2001 – Fifth Annual Maintenance and Reliability Conference, Gatlinburg, USA, 2001. L. Fumagalli, M. Macchi, M. Rapaccini, Computerized maintenance management systems in SMEs: a survey in Italy and some remarks for the implementation of Condition Based Maintenance, in: IFAC Proceedings Volumes (IFAC-Papers Online), vol. 13, Issue Part 1, 13th IFAC Symposium on Information Control Problems in Manufacturing, INCOM’09, Moscow June 3–5, (2009), pp. 1615–1619. J. Lee, J. Ni, Infotronics-based intelligent maintenance system and its impacts to closed-loop product life cycle systems, in: Invited Keynote Paper for IMS’2004— International Conference on Intelligent Maintenance Systems, 2004, Arles, France, 2004. J. Lee, L. Liao, E. Lapira, J. Ni, L. Li, Informatics platform for designing and deploying e-manufacturing systems, in: Collaborative Design and Planning for Digital Manufacturing, 2009, 1–35.

[32] A. Muller, A.M. Crespo, B. Iung, On the concept of e-maintenance: review and current research, in: Engineering and System Safety, Elsevier, 2007, http:// dx.doi.org/10.1016/j.ress.2007.08.006, pp. 1165–1187. [33] ISO, 13374 – International Organization for Standardization ISO 13374-1: condition monitoring and diagnostics of machines: data processing, communication and presentation, Part 1: general guidelines, Geneva, 2003. [34] OSA-CBM, Specification 3.3.0, Machinery Information Management Open Systems Alliance, 2009. Disponı´vel em: http://www.mimosa.org/?q=resources/ specs/osa-cbm-v330 (acesso em: 10 dez. 2011). [35] J. Campos, Development in the application of ICT in condition monitoring and maintenance, Computers in Industry 60 (1) (2009) 1–20. [36] D. Espı´ndola, L. Fumagalli, M. Garetti, S. Botelho, C. Pereira, AnaAdaption of OSACBM architecture for human-computer interaction through mixed interface, in: INDIN 11 Conference, Caparica, Lisbon, Portugal, 26–29 July, 2011. [37] J. Campos, An ontology for asset management, in: Proceedings of the IFAC CEA (Conference on Cost Effective Automation in Networked Product Development and Manufacturing) 2007, Monterrey, Mexico, 2007. [38] M.A. Lopez Campos, L. Fumagalli, J.F. Gomez Fernandez, A. Crespo Marquez, M. Macchi, UML model for integration between RCM and CBM in an e-Maintenance architecture, in: 1st IFAC Workshop, A-MEST’10, Advanced Maintenance Engineering, Services and Technology, Lisboa, Portugal, (2010), pp. 133–138. [39] S. Terzi, H. Panetto, G. Morel, M. Garetti, A holonic metamodel for product traceability in Product Lifecycle Management, International Journal of Product Lifecycle Management 2 (3) (2007) 253–289. [40] J. Cassina, M. Tomasella, M. Marquard, A. Metin, A. Matta, M. Taisch, Development of the semantic object model for a PDKM system, in: Proceeding at ICE 2006 Conference, Milan, 26–28 July, (2006), pp. 383–390. [41] L. Fumagalli, E. Jantunen, M. Garetti, M. Macchi, Diagnosis for improved maintenance services: analysis of standards related to Condition Based Maintenance, in: Proceedings of the 4th World Congress on Engineering Asset Management, 2009, Athens, Greece, 2009. [42] J. Cassina, M. Tomasella, M. Taisch, A. Matta, A new closed-loop PLM Standard for mass products, International Journal of Product Development 8 (2) (2009) 141– 161 (21). [43] G. Booch, J. Rumbaugh, UML: guia do usua´rio – Livraria Tempo Real Informe, 2006. [44] H. Kato, M. Billinghurst, I. Poupyrev, ARToolKit version 2.33 [S.1:s.n.], Manual de software, 2000. Disponı´vel em: http://www.tinmith.net/lca2004/ARToolkit/ ARToolKit2.33doc.pdf (acesso em: 20 mar. 2011). [45] M. Fiorentino, G. Monno, A.E. Uva, Tangible digital master for product lifecycle management in augmented reality, Proceedings of the International Journal On Interactive Design And Manufacturing 3 (2) (2009) 121–129. [46] M. Foursa, D. d’Angelo, R. Narayanan, Innovative control system for industrial environment, in: Fourth International Conference on Autonomic and Autonomous Systems IEEE, 2008. [47] D. Djurdjanovic, J. Lee, J. Ni, Watchdog Agent, an infotronics-based prognostics approach for product performance degradation assessment and prediction, in: Advanced Engineering Informatics, Elsevier, 2003, pp. 109–125. [48] IMS Center – Center for Intelligent Maintenance Systems. Disponı´vel em: http:// www.imscenter.net (Acesso em: 15 nov. 2011). [49] D. Espı´ndola, N. Duarte, S. Botelho, J.T. Carvallho, C. Pereira, Internet of things to provide scalability in product-service systems, in: The Sixth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies, 2012, Barcelona, vol. 1, Elsevier, Sao Diego, 2012, pp. 1–6.

Danu´bia Bueno Espı´ndola was born in Porto Alegre, RS, Brazil, in 1979. She received the M.Sc. degree in Oceanic Engineering, with emphasis on industry naval and virtual manufacturing, in 2007 and B.S. degree in Computer Engineering in 2005 both from Federal University of Rio Grande (FURG) and Ph.D. in Electrical Engineering (2011) from the Federal University of Rio Grande do Sul (UFRGS), Brazil. She is currently professor of the Center of Computational Science (C3) at the Federal University of Rio Grande (FURG). Main research interests are: virtual/mixed reality, digital manufacturing, 3D user interfaces, information visualization, intelligent maintenance systems and learning environment applied to education and industry. Luca Fumagalli holds a post-doc position at the Department of Management, Economics and Industrial Engineering of Politecnico di Milano, working mainly in two projects related with the industrial application of the electric signature analysis for maintenance purposes and mathematical models for spare part management. He is also responsible of the annual research of the Observatory TeSeM (Technology and Services for Maintenance), www.tesem.net. His research interests are focused on innovations in maintenance management. Luca Fumagalli has been lecturer and teaching assistant in courses at undergraduate and post graduate level, such as the courses of Reliability Analysis, Maintenance Management and Modeling of Production Systems.

D.B. Espı´ndola et al. / Computers in Industry 64 (2013) 376–391 Marco Garetti is full Professor of Industrial Technologies at Politecnico di Milano. He developed his activities since 1971 at Alfa Romeo cars and since 1974 at Politecnico di Milano. He is member of IFIP WG 5.7 (Advances in Production Management Systems) as well a founder member of IFIP WG 5.1 (Global Product Development in the whole lifecycle). He is member of the editorial board of Production Planning and Control (Taylor & Francis edit.) and of the International Journal in Product Lifecycle Management (Inderscience edit.). He has been project leader of many research projects on domestic and world-wide scale. He is author and coauthor of several books and more than 100 papers concerning modeling and simulation, product lifecycle management, industrial plant engineering and operations management. Carlos Eduardo Pereira was born in Porto Alegre, Brazil, in 1965. He received the B.S. degree in Electrical Engineering and the M.Sc. degree in Computer Science from the Federal University of Rio Grande do Sul (UFRGS), Porto Alegre, in 1987 and 1990, respectively, and the Dr.-Ing. degree in electrical engineering from the University of Stuttgart, Stuttgart, Germany, in 1995. He is currently an Associate Professor with the Electrical Engineering Department, UFRGS. His research focuses on methodologies and tool support for the development of distributed real-time embedded systems, with special emphasis on industrial automation applications and the use of distributed objects over industrial communication protocols. Dr. Pereira is the Chair of the International Federation of Automatic Control (IFAC) Technical Committee on Manufacturing Plant Control (TC 5.1). He has acted as a member of International Program Committees for several conferences in the field of industrial automation,

391

manufacturing, industrial protocols, and real-time distributed object computing. He is also an Associate Editor of the journal Control Engineering Practice. He is currently the Chair of the Brazilian Automation Society, the IFAC’s national member organization in Brazil. Silvia Silva da Costa Botelho was born in Rio Grande, Brazil, in 1971. She received the B.S degree in Electrical Engineering in 1991 and M.Sc. degree in Computer Science in 1996 from Federal University of Rio Grande do Sul (UFRGS), Porto Alegre. She is Ph.D. in Informatics and Telecommunications from Centre National de la Recherche Scientifique, France, in 2000. She is currently Associate Professor of the Center of Computational Science (C3) at the Federal University of Rio Grande (FURG) and has experience in Computer Science, acting on the following subjects: intelligent automation, Internet of Things and robotic. Renato Ventura Bayan Henriques received his B.Sc. in electrical engineering from Pontificia Universidade Catolica do Rio Grande do Sul, Brazil (1992) and M.Sc. in electrical engineering from the University of Sa˜o Paulo – USP, Brazil (1996) and Ph.D. in mechanical engineering from Universidade Federal de Minas Gerais – UFMG, Brazil. He is currently a professor at Federal University of Rio Grande do Sul (UFRGS) teaching electrical engineering with emphasis in electrical process automation and electronics industry, focusing on the following topics: position control, cooperative robots, robotic welding, intelligent maintenance, Kalman filters, manufacturing systems and distributed control.