From virtual reality to web-based multimedia ... - Springer Link

Int J Interact Des Manuf (2013) 7:183–190 DOI 10.1007/s12008-013-0185-0

TECHNICAL PAPER

From virtual reality to web-based multimedia maintenance manuals Giuseppe Di Gironimo · Rocco Mozzillo · Andrea Tarallo

Received: 8 February 2013 / Accepted: 25 February 2013 / Published online: 13 March 2013 © Springer-Verlag France 2013

Abstract This paper focuses on a structured methodology that uses virtual reality (VR) and digital human modeling (DHM) to study maintenance procedures of industrial products. VR technologies help to highlight the most critical aspects of maintenance operations, while DHM tools allow detailing working sequences. Data coming from these analyses are then used to draw up a multimedia maintenance manual based on digital video animations, audio comments, explanatory images and written recommendations. Information is available to maintenance personnel directly on the working site through portable electronic devices. Further, web-based multimedia manuals can be updated on-line and help to shorten learning time and maintenance downtimes. Keywords Design for maintainability · Virtual design review · Digital human modelling · Multimedia maintenance manuals

1 Introduction Digital mock-up (DMU) and numerical methods, such as FEM or CFD analyses, have become very common in the industrial design field (Virtual Product Development). However, the full integration of these analyses with other aspects mainly related to the context of use of the product and its G. Di Gironimo (B) · R. Mozzillo · A. Tarallo University of Naples Federico II, Piazzale Tecchio 80, 80125 Napoli, Italy e-mail: [email protected] R. Mozzillo e-mail: [email protected] A. Tarallo e-mail: [email protected]

life cycle (e.g. ergonomics, maintainability, usability, etc.) is still quite a challenge for designers. Furthermore, with the increasing complexity of industrial products, critical evaluation of these aspects needs very specialized skills that may not belong to a single individual, but instead they are shared among the entire production system, which is increasingly being distributed internationally. This means that design review activities must include a two-way horizontal information flow on the project, capable of interfacing skills that would otherwise be distant from each other. For instance, although many software applications help designers to perform maintainability analyses and to draw-up technical documentation, structured methods to summarize all the information coming from maintenance staff, human factors experts and maintenance engineers are needed. In order to face this kind of problems, our research group has been working since years on developing a design review methodology (Virtual Design Review) based on the synergy between virtual reality (VR) and digital human modelling (DHM). VR technologies allow designers to interact in realtime with digital mock-ups inside a single virtual environment (VE), while DHM tools make possible an objective and measurable analysis of the human factor in the context of use of the future product. On the one hand, one of the first applications of VR in the industrial design field is shown in [1] that investigates the use of this technology to simulate assembly and maintenance processes on a digital mock-up. Nowadays, these technologies are very sophisticated and are proved to appreciably reduce time and cost of product development [2]. In particular, organizations using VR can reduce lead time to market by 30 % and the number of design changes by 65% [3,4]. On the other hand, published literature agrees that the capabilities of DHM tools go over and above the mere graphical simulation [5,7] but instead they can be effectively used

123

184

to conduct very detailed ergonomic analyses [6]. Several authors [8,9] have demonstrated that ergonomic simulations based on DHM tools provide good estimations of the workload in real-life tasks, correctly predicting ergonomics issues for standing and unconstrained working postures. The synergy of VR and DHM can be particularly useful in the fields of maintenance. This is a quite subtle, but important problem. For instance, referring to aeronautic industry, many authors have shown that the 45 % of maintenance time is spent in reading technical documentation [10,11]. Thus, new modalities for developing and administering technical documentation to maintenance staff are needed. Many researchers are working on augmented reality (AR) systems. For instance, Schwald [12] focused on an AR system for training and assisting in maintaining equipment in industrial context. Azuma [13] describes the characteristics of AR systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Toro [14] presented a system implementation for the exploitation of embedded knowledge in the domain of industrial maintenance in a mobile context. As test case, it implemented an approach in different portable devices with video input capabilities such as PDAs and Tablet PCs. However, augmentation techniques are not always suitable for actual industrial contexts, since they need a proper technological framework. For this, we focused on a simpler but effective technology that can be introduced with minor efforts in industry. In particular, we show how VR and DHM simulations can be used not only to optimize maintenance procedures during the design phase, but also to draw up web-based maintenance manuals. Actually, several examples of multimedia manuals exist in the published literature, [15,16]. However, generally these manuals just consist of a set of short videos that only show mounting sequences and assembly of parts. In general, the movements of workers needed to perform a certain task are not considered at all, or, at least, they are not shown. Instead, the design approach presented in this paper assumes the human factor to be central since the very first steps of product design, according to the doctrine of Concurrent Engineering [17]. It is worth noticing that every good product need, in any case, the study of its assembly and its disassembly procedures, just as accessibility, ergonomics and working-time analyses. Thus, as a part of the product design process, technical documentation of a product can be made at virtually no cost.

2 Virtual design review and multimedia manuals Virtual design review (VDR) is a methodology that uses VR technologies to improve the development and the critical

123

Int J Interact Des Manuf (2013) 7:183–190 Table 1 IDEAinVR lab equipment for virtual design review Name

Description

Qty

Depth-Q HDs3D

3D projector

1

Sanyo PDG DWL-2500

Short optics 3D projector

1

IoTracker

IR Tracking cameras

4

Workstation

2

DELL Precision

Finger tracking device

1

Nintendo WiiTM

Wireless controller

1

Microsoft KinectTM

Motion sensing device

1

Cyberglove

review of projects. VDR allows designers to see and manipulate virtual products in real-time, evaluate several design solutions, simulate assembly and maintenance tasks, take measures, verify visibility, etc. Furthermore, by means of digital human models, VDR make it possible also to conduct ergonomic analyses on a product or a certain task in real-time. Historically, the main barriers to the spread of VR in industry was the cost of the equipments and the size of their operating space. Therefore, the authors have spent some efforts in the study and development of low-cost VR solutions, yet ensuring high quality standards. From this point of view, our VR laboratory IDEAinVR (Interactive Design and Ergonomics Applications in VR), equipped with two stereoscopic frontal projection systems and advanced motion capture systems, is definitely an affordable way for the diffusion of VR technology into the industrial field. IDEAinVR lab equipment is described in more detail in Table 1. This configuration has at least 3 significant advantages over ordinary frontal projection systems: – Thanks to the short optics, user can come close to the screen during the simulation, up to a distance of 20 cm without casting shadows, – Those involved in the design review session can follow the analysis on a second screen, larger than the former, without visual occlusions due to the user standing in front of the first screen (see Fig. 1), – The two screens can also be used to display different information. For instance, while the first screen is displaying the simulation, the second one can show CAD models, results of FEM/CFD analyses, etc. This solution is cheaper than a rear projection system and can be easily installed even in small environments. VR based analyses have well-known benefits, if compared with traditional desktop-based design review. In particular: – Many design solutions can be evaluated in real-time, – Complex CAD models can be handled in real-time,

Int J Interact Des Manuf (2013) 7:183–190

185

Fig. 1 Dual 3D projection system of IDEAinVR lab

– Stereoscopic visualization makes easier to detect technical problems, – A collaborative environment improves the information flow about the project, – Possible ergonomics issues can be evaluated in real-time, – Quite complex human activities can be simulated inside a virtual environment (e.g. maintenance operations).

However, apart from these clear benefits, VR-based analyses has some weaknesses. Just for instance, the backward propagation of information coming from design review activity is still an open issue, due to the lack of a generally accepted protocol for transferring data between PDM and VR environments and vice versa. Indeed, feedback data are often very heterogeneous (e.g. FEM, CFD, illumination, ergonomics analyses, etc.), involving entire processes and more than one CAx tool of different software vendors. This is particularly penalizing for complex projects developed on international basis. Nevertheless, at least with reference to maintenance analyses, a possible way of propagating the results of VR simulations to maintenance operators is through the so-called “multimedia manuals”. In particular, here we show a structured methodology that uses VR and DHM to study maintenance procedures. Then, the information coming from these analyses are used to draw up a web-based maintenance manual. The case study concerns the maintenance analysis of the pneumatic brake and air conditioning systems of a regional train.

2.1 Methodological approach The methodology that is being discussed here is essentially based on EDIVE [18] approach, that is summarized in Fig. 2. More precisely, in a first phase, CAD modules aimed at collision-free path planning are used to verify the detachability of assembly components. After that, DHM tools are used to perform more detailed analysis about the actual feasibility of the considered maintenance task.

Fig. 2 Main steps of EDIVE methodology

On the one hand, EDIVE methodology can answer in a precise and an objective way to the problems related to maintainability of industrial products and it can provide objective information about ergonomics of the posture assumed by workers, considering the human comfort as the fundamental requirement to observe, DHM tools also allows estimating maintenance time as function of a desired comfort level for critical tasks. Therefore, maintenance sequences can be defined considering the trade-off between the comfort perceived by maintenance staff and time/cost requirements. In this way, several assembly and disassembly procedures can be digitally evaluated until the final maintenance procedure is defined. Unfortunately, on the other hand, such methodology can be quite hard to implement, mostly because of the time needed for the animation of the human figures. In fact, despite different software solutions have been developed fit for purpose, quick simulation of human behavior in constrained environments is still a challenge, at least with reference to complex scenarios. In these cases, very specialized skills are

123

186


Fig. 3 VR-based maintenance simulation

needed, similar to those required to cartoons animators of entertainment industry, [19]. A proper set of “key postures”, needed to achieve the desired task, have indeed to be defined for the digital human, taking account of many physical constraints, such as the presence of obstacles along the disassembly path. It is understood that the simulation of complex scenarios can be very time consuming and can represent a significant cost for the industry and a barrier to the use of such simulation tools. Therefore, when maintainability analyses involve large assemblies (that means simulating hundreds of tasks) EDIVE methodology becomes definitely unsuitable. However, it must be noticed that, even for complex assemblies, generally just a small subset of maintenance operations really deserves the use of specific DHM tools. VR technologies are, indeed, known to be an effective way of simulating human movements in real-time [20], thanks to motion capture systems and VR devices (e.g. virtual gloves) that allow users to naturally interact with virtual objects inside a computergenerated environment (see Fig. 3). This kind of real-time interaction can be used to simulate maintenance tasks very quickly (VR-based analysis). For instance, a digital mock-up can be entirely disassembled by means of “virtual tools”. Therefore, the workflow of EDIVE can be modified as shown in Fig. 4. DHM tools will be used only when VR-based analysis is not applicable because the human-product interaction is very complicated (e.g. complex kinematic mechanisms) or when the results of VR simulation are uncertain from an anthropometric point of view. In other words, VR technologies act as sort of “filter” on the great number of possible maintenance operations. Then, DHM tools can be used to conduct more detailed ergonomics analyses only on a limited subset of maintenance operations (critical maintenance tasks). It must be noticed that both VR-based and DHM-based analyses have a useful “side-effect”: video animations can be recorded during the simulations and then properly edited

123

Fig. 4 Integration of VR and DHM for maintenance simulation

with subtitles. Thus, the development of multimedia content is a matter of course. Moreover, collaborative environments like our IDEAinVR lab help maintenance engineers to get involved in maintenance simulation. In this way, digital animators can solve many issues related to production and maintenance operations. The synergy between VR software and DHM tools can be used to identify design errors that affect the maintainability of the product and to propose design solutions that better take into account the human factor. Moreover, possible design errors can be highlighted before physical prototypes are built. This is particularly important because, at this stage, the proper design changes can be done at very reduced costs. Finally, all this information can be collected to become the bases of a multimedia maintenance manual. 2.2 Web-based maintenance manuals The use of multimedia applications for information transmission is a quite common practice in industry. However, just an electronic manual could not significantly reduce learning time and therefore maintenance cost [21] because maintenance operators, nevertheless, have to read the instructions, understand them, remember them and apply them correctly. Moreover, all these processes are prone to errors, especially omissions [22]. In order to face this kind of issues the authors have developed a web-based maintenance manual based on the information coming from VR and DHM analyses (see Fig. 5).


187

– Head area – Rear under-body area – Under-ceiling area Thus, these areas were further studied by means of a specific DHM tool (Siemens Jack ). In order to accurately simulate manual operations, also maintenance tools were digitally reproduced. Moreover, since the train was still under development, some missing CAD models were replaced with simplified geometries. 3.1 Head area VR-based analysis highlighted that some fittings in the head area were not accessible at all. For this, a proper hatch (see Figs. 6, 7) has been added to the original design. After that, Jack was used to verify the maintainability of the fittings. Fig. 5 Multimedia manual information flow

One of the main goal of the methodology is to shorten the consultation and learning time of maintenance operations. Indeed, video animations help maintenance operators to learn assembly and disassembly sequences, since they can pause and watch videos any time until they well understand every phase of the maintenance task [23]. As aforementioned, all the information previously produced becomes part of a web-based application that improves the accessibility and availability of the contents. In this way, possible flaws in documentation can be detected and fixed on-line. The changes are then propagated in real-time to maintenance staff. Finally, with reference to industrial processes, multimedia manuals can significantly shorten maintenance downtimes and therefore reduce production costs.

3 Case study As aforementioned, the case study concerned the maintainability analysis of both the pneumatic and the air conditioning systems of a regional train. The main goals of the work were: – – – –

detecting the critical maintenance operations, defining a step by step maintenance work flow, determine the maintenance time, drawing up a multimedia maintenance manual.

The first objective has been pursued by means of VR technologies. Several designers, human factors experts, maintenance engineers and operators got involved in the design review activity. In particular, with reference to pneumatic and air conditioning systems, just three areas of the train have been considered critical from an ergonomic point of view:

Fig. 6 New hatch has been added in head area for fittings inspection

123

188


Fig. 7 Most critical joint in head area Fig. 9 Maintenance of under-ceiling fittings

Fig. 8 Maintenance task simulation in under-body area

components to be maintained. Each component has further sub-components and each of these has several maintenance operations associated with. Finally, once user select a maintenance task, a digital video animation is displayed along with a detailed report on every phase of the selected maintenance operation. Figure 10 shows a screenshot of the resulting web-based maintenance manual developed for the case study. As aforementioned, the multimedia manual contains all the information coming from previous stages of product design. In particular, it includes:

3.2 Rear under-body area Rear under-body area needs some preventive maintenance. VR analysis showed that most of the fittings can be inspected visually, but their actual accessibility has been verified with Jack (see Fig. 8). 3.3 Under-ceiling area VR simulation showed that under-ceiling area needs further maintenance analyses. Jack has been used to simulate the tightening of some connections (see Fig. 9). DHM analysis highlighted some flaws in the original design that was consequently modified. Moreover, several possible maintenance sequences have been compared on the level of comfort perceived by the operator. 3.4 Multimedia manual The data coming from the previous analyses have become the matter for the multimedia manual. The information has been arranged in a tree structure. A certain number of thematic areas are on the top of the hierarchy. The user can select any of them by clicking on their thumbnails (or even touching on it, if a touch-screen is used). Each area have certain

123

– short video animations that show assembly and disassembly operations and the correct human movements, – flowcharts of the entire assembly cycle, – images showing human working postures in detail, – accessibility and visibility reports, – ergonomic reports.

Video animations clearly illustrate the proper execution of the maintenance tasks. They also turned out to be very useful to show all the suggestions coming from accessibility, visibility, postural and working time analyses. The flowcharts of the maintenance cycle describe the workflow and the tools needed for each operation. Moreover, they specify the quality checks to be performed after each maintenance operation. The evaluation of man-hours needed to accomplish each maintenance task has been based on methods-time measurement (MTM) [24]. Accessibility and visibility reports show disassembly paths and highlight the critical tasks (characterized by limitations both in working space and visibility). Finally, ergonomic reports include recommendations regarding equipment position, working postures, work patterns, etc.


189

Fig. 10 Multimedia manual (screenshot)

4 Conclusions In this paper, the authors focused on the development of multimedia maintenance manuals based on the synergy between VR, DHM and web-based frameworks. Multimedia manuals can reduce maintenance time and cost, improving at the same time the quality of working. However, the same technology could be extended to any assembly task as well as any industrial process involving human workers. Web-based manuals potentially may give better support to workers than a paper guide. Operators can look through the most updated version of the maintenance sequences and procedures by means of portable electronic devices (PDAs, tablet PCs, etc.) connected with a remote data repository. It is worth emphasizing that this technologies still require good expertise in how to use DHM tools and also in how different assembly tasks are (or should be) performed. Future work will focus on the standardization of the procedures for the development of multimedia manuals based on VR technologies. A still open issue is time needed to draw up the multimedia manual and to develop simulations using digital humans. In this regard, our research group is studying new interaction paradigms based on Natural User Interfaces (NUI) in order to reduce the time needed for digital characters animation with common DHM tools.

Acknowledgments The work presented in this paper has been partially funded by MIUR (Italian Ministry of University and Research) and Campania regional authority within, respectively, the PON project “Digital Pattern” and the Network of Excellence framework “INSIST”.

References 1. Gomes de Sá, A., Zachmann, G.: Virtual reality as a tool for verification of assembly and maintenance processes. Comput. Graph. 23(3), 389–403 (1999) 2. Di Gironimo G., Patalano S., Tarallo A.: Innovative assembly process for modular train and feasibility analysis in virtual environment. Int. J. Interact. Des. Manuf. 3(2), 93–101. ISSN: 1955–2513. doi:10.1007/s12008-009-0066-8 (2009) 3. Teresko, J.: Digital Manufacturing: Not Only Manufacturing. Industry Week (2006) 4. Chaffin, D.B.: On simulating human reach motions for ergonomics analyses. Human Factors Ergon. Manuf. 12, 235–247 (2002) 5. Demirel, H.O., Duffy, V.G.: Digital human modelling for product lifecycle management. In: Lecture Notes in Computer Science, vol. 4561, pp. 372–381 (2007) 6. Demirel, H.O., Duffy, V.G.: Applications of digital human modelling in industry. In: Lecture Notes in Computer Science, vol. 456, pp. 824–832 (2007) 7. Di Gironimo, G., Patalano, S.: Re-design of a railway locomotive in virtual environment for ergonomic requirements. Int. J. Interact. Des. Manufact. 2(1), 47–57 (2008). ISSN:1955–2513. doi:10. 1007/s12008-007-0035-z 8. Fritzsche, L.: Ergonomics risk assessment with digital human models in car assembly: simulation versus real life. Human Factors Ergon. Manuf. Serv. Indus. 20(4), 287–299 (2010)

123

190 9. Lamkull, D., Hanson, L., Ortengren, R.: A comparative study of digital human modelling simulation results and their outcomes in reality: a case study within manual assembly of automobiles. Int. J. Indus. Ergon. 39, 428–441 (2009) 10. Ott, J.: Maintenance executives seek greater efficiency. Aviat. Week Space Technol. 142(20), 43-44 (1995) 11. Neumann, U., Majoros, A.: Cognitive, performance, and system issues for augmented reality applications in manufacturing and maintenance. In: Proceedings of IEEE the Virtual Reality Annual International Symposium (VRAIS), pp. 4–11 (1998) 12. Schwald, B., de Laval, B.: An augmented reality system for training and assistance to maintenance in the industrial context. J. WSCG 11(1) (2003) 13. Azuma, R.T.: A survey of augmented reality. Tele-oper. Virtual Environ. 6, 355–385 (1997) 14. Toro, C., Sanan, C., Vaquero, J., Posada, J., Szczerbicki, E.: Knowledge based industrial maintenance using portable devices and augmented reality. In: Lecture Notes in Computer Science, vol. 4692, pp. 295–302 (2007) 15. Peng, G., Hou, X., Gao, J., Cheng, D.: A visualization system for integrating maintainability design and evaluation at product design stage. Int. J. Adv. Manuf. Technol. 1–16 (2011) 16. Hanson, L., Blomé, M., Dukic, T., Högberger, D.: Guide and documentation system to support digital human modelling applications. Int. J. Indus. Ergon. 36, 17–24 (2006) 17. Di Gironimo, G., Lanzotti, A.: Designing in VR. Int. J. Interact. Design Manuf. 3(2), 51–53 (2009). ISSN: 1955–2513

123

Int J Interact Des Manuf (2013) 7:183–190 18. Di Gironimo, G., Monacelli, G., Patalano, S.: A design methodology for maintainability of automotive components in virtual environment. In: Proceedings of International Design ConferenceDesign, pp. 723–734. Dubrovnik (2004) 19. Badler, N.I., Phillips, C.B., Webber, B.L.: Simulating Humans: Computer Graphics Animation and Control. Oxford University Press, Oxford (1993) 20. De Santis, A., Di Gironimo, G., Pelliccia, L., Siciliano, B., Tarallo, A.: Biomechanically-based motion control for a digital human. Int. J. Interact. Des. Manuf. 6(1), 1–13 (2012). doi:10.1007/ s12008-011-0132-x 21. Brinkman, W.P., Buil, V.P., Cullen, R., Gobits, R., Van Nes, F.L.: Design and evaluation of online multimedia maintenance manuals. Behav. Inform. Technol. 20(1), 47–52 (2001) 22. Wright, P.: Manual dexterity: a user-oriented approach to creating computer documentation. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 11–18. Boston, MA, USA (1983) 23. Reason, J.T.: Human errors and error handling. Appl. Psychol. 43(4), 427–432 (1994) 24. Di Gironimo, G., Di Martino, C., Lanzotti, A., Marzano, A., Russo, G.: Improving MTM-UAS to predetermine automotive maintenance times. Int. J. Interact. Des. Manuf. 1955–2505 (2012). doi:10.1007/s12008-012-0158-8