Virtual Laboratory for Automation and Robotics Study Michał Smater 1,a,Jacek Zieliński 1,b 1
Przemysłowy Instytut Automatyki i Pomiarów PIAP, Al. Jerozolimskie 202, 02-486 Warszawa, Poland a
[email protected],
[email protected]
Keywords: robotics, virtual laboratory, augmented reality, modern learning.
Abstract This paper describes the concept of Advanced Virtual Laboratory of Automation and Robotics using stereoscopic 3D video transmission augmented with computer generated virtual reality elements supporting students to understand and construct their personal conceptual knowledge of robotised and automated facilities design and control. It presents the assumptions and analysis of modern virtual robotics laboratory improving the process of education and responding to the needs of the modern market. Introduction The development of Automation and Robotics (A&R) products and technologies is presently enormously fast and the number of different solutions present on the market is uncountable. Additionally the new strategy 'Europe 2020' which main goal [1] is the advancement of the European Union economy is going to be implemented in the near future. To achieve this ambitious goal and to keep high position of Europe among other world economies the EU have to, in near future, intensify actions to reverse the declining role of industry. The post industrial Europe cannot effectively fight against economic crisis. Latest research of The VDI [2] shows that there were, only in Germany, 80500 vacancies for engineers in industry. The dynamics of changes in manufacturing environment and the new 'Europe 2020' strategy goals requirements results in necessity of rapid specialists (engineers, well qualified workers) training, preparing them to develop/maintain/operate automated and robotised production lines. However mentioned factors in conjunction with traditional study techniques may result in arising of significant gap between what students learn in theory during years spent at the university and the reality they are faced in factory as A&R engineers. A new, innovative approach to the study subject in the field of Automation and Robotics is highly required. The virtual laboratory which concept is presented in this work will allow to (1) minimise the mentioned gap between theory and reality and (2) meet the market demand for highly skilled, well prepared engineers-specialists in A&R field. Developing a new study methodology is a necessary but is not sufficient condition. Supporting technology, utilising beyond the state-of-theart ICT and visual techniques, must be a complementary, inherent element to the new methodology. Future laboratories should aim at creating a networking environment that fosters crossorganisational learning. It shall promote evolution of creative teams by developing networked access to laboratory with few conquering robots that facilitate questioning and challenging, foster imaginative thinking, widen the perspectives and make purposeful connections with people and their ideas. Also using virtual reality environment, the applications appear to be promising for any supporting E-learning tasks to be more nature and interactive [3]. Concept and structure of the Virtual Laboratory The Virtual Laboratory of Automation and Robotics will be using stereoscopic 3D video transmission augmented with computer generated virtual reality elements supporting students to understand and construct their personal conceptual knowledge of robotised and automated facilities design and control. Live video transmission in 3D of operated in real time by student robot, working in remote, safe location, will support the creative applications of the theory gained on Automation
and Robotics related faculty. Innovative stimulating applications of robot control theory in real world, augmented gmented with virtual elements of designed by student automated production line will take the learners through the complexity of a subject, activating and feeding curiosity and reasoning. Distinctly from training utilising entirely computer generated word of of virtual reality, students operating the real, working robot will have a fillings and emotions generating real perception of the work with robots. Operating the real robot, even it is installed in distant, secure laboratory, results in different, more tactile tile level of perception of the objective. The main, central object on the laboratory scene is the real industrial robot, robot, installed in a remote location and meeting the restrictive safety requirements preventing unauthorised access to robot working space, it can be operated by distant operator. To assure as much as possible close to reality robot operation/programming environment the distant operator equipment should reflect the robot legacy operator panel. As optimal solution all signals sent/received (including (including signals sent/received by legacy operator panel) by the robot must be accessible immediately with low latency to distant operator location. Combining this with real time 3D video of working robot will rise in the operator (student) the emotions and impression mpression as operating the robot directly. The presence of this real, significant element of virtual laboratory will help to initiate in students sense of real experience and respect for a robot as advanced, complex and potentially dangerous machine. The real r emotion accompanying the robot programming/operation will significantly strengthen the students process of theoretical knowledge acquiring and conceptual knowledge building, resulting in higher intrinsic motivation for the learning content and the elaboration elab process. The central, main part of laboratory is to be surrounded by a number of virtual objects representing real automation equipment. Students will have possibility to build, program and simulate complex automated and robotised applications. The wide spectrum of available building blocks, covering elements from small simple sensors through pneumatic actuators and PLC systems to other industrial robots. High precision, hyper realistic rendering of building blocks will also have significant influence ce on students education process. The structure of complete virtual laboratory is based on client-server server architecture as shown on schema below:
Fig. 1 Virtual Laboratory architecture schema
Virtual laboratory comprises of the following modules: 1. Industrial robot with control unit adapted to cooperate with other laboratory modules; equipped with computer for all robot I/O signals pre-processing. 2. 3D cameras set with raw video stream pre-processing and camera position control unit. 3. Main Processing Unit & Server for video stream combination with augmented reality elements, signal processing and acquisition, robot and camera control signals receiver/transmitter and processor. 4. Client unit for HMI interface. 5. Video Presentation facility. The modular design of virtual laboratory allows its structure variants and assures system scalability. The basic structure as shown on the schema above concentrates main laboratory elements in secure location while HMI and presentation facility are distant elements. This solution allows the existence of several client terminals utilising one central server. Main advantage is a low cost of the client side; disadvantage is the large amount of data to be transferred by the Internet. It is possible to build the laboratory in a different way where only modules 1 and 2 are in secure location while modules 3, 4 and 5 are distant elements. Main advantage is limited amount of data to be transferred in comparison to basic structure; disadvantage is high cost of the client side. The minimalistic version comprises only modules 3, 4 and 5. There is no Internet connection required as all elements can be installed in one place. This solution can be used for system demonstrator only because the absence of the main laboratory element, the robot, drastically decreases the educational ability of the system. The system can be easily expanded by multiplying the number of modules 1 and 2 as well as number of modules 5. To assure constant robot accessibility without necessity of introducing restrictive access time scheduling, new modules 1 and 2 can be added to the system. From the other side when complex A&R applications will be built the system can be expanded by additional modules 5 to increase the presentation area. Technical requirements The main requirements consists in letting the student interact with the environment through a 3D model of the workplace. Through this interface, operators elaborate robot trajectories, play them and control their execution in an intuitive way. Virtual environment offers graphic assistances, making easy robots programming and control. It interfaces to robots and tools through an execution controller, allowing updating model state according to the real situation. In proposed solution, specific processes functions (welding, ultrasounds inspection, grinding) are gathered in dedicated trade modules. Moreover, collision test algorithm warns the operator, preventing collisions with the modelled environment. This feature is called passive anti-collision but can only be used to prevent collision (warns the operator or stops execution of the task) and so may not be of any help in case of use in very constrained environment. Virtual reality makes it now possible to interact with an accurate mechanical model using a haptic device. Main applications are currently for training with digital prototyping (accessibility, maintenance). Concerning the teleoperation field, the main benefit of using a “physics engine” type software with a teleoperated system consists in providing the operator with an active collision avoidance feature. New functions differ from the passive anti-collision feature as it ensures collision avoidance by generating repulsive efforts, thus maintaining execution of the teleoperation task. Operator uses a 6 DoF (degrees of freedom) mouse to generate movement on the simulation and the robot. Because the simulation not only detects the collisions but also simulates the robot behaviour, the operator can then handle the robot safely, without taking care of obstacles of the environment. The learner is now able to explore different actions and their consequences in a simulated process instead of theoretically try to mentally follow a described process.
Future work shall be done to link the system with a force reflective master arm, and then provides the operator with force feedback on simulated collision. This functionality will allow using the master/slave system in a real force feedback mode, preventing collision of known, modelled objects. The issue of localization of objects will also be addressed. Augmented reality system for training The main task for the augmented reality system will be to produce synthetic images in real-time consisting of artificial 3D objects and a real scene. 3D geometry and radiometric properties of artificial objects will be given, visual and depth information about the real scene will be captured by 3D video and/or depth cameras. The system has to be capable to calculate the accurate light transport simulation in real time to produce the video of a real scene augmented with synthetic objects. Therefore the main focus shall be put to achieve high image quality and rendering speed. High-quality ray-tracing techniques such as photon mapping [4] in combination with rasterisation to calculate global illumination will have to be used in order to properly simulate the light interaction between real and virtual objects while preserving high system performance. Therefore the technique proposed by McGuire and Luebke [5] while porting all calculations to fully run on the GPU will have to be improved. The OptiX [6] ray tracing engine will be used to trace incoherent rays through the scene and the OpenGL rasterisation pipeline to trace coherent light rays in the scene. A first approach of our GPU based physically correct depth of field implementation is described in [7]. Moreover correct material properties have to be simulated to achieve realistic looking virtual objects. A complex BRDF model will be used, which can properly calculate light reflection on the surface to accurately simulate materials of objects in the scene. The rendering system will benefit from modern GPUs, which offer massive parallel computing power to gain the maximum performance and rendering speed. A depth camera will be used to obtain 3D information about the scene (e.g. such as in KinectFusion [8]). This allows to overcome the problem of using static, predefined geometry of real world objects. Predefined real geometry presents the current state of the art in high qualitative AR rendering. In this application scenario dynamic changes of the real scene are possible and likely. Based on the point cloud registration the position of the camera can be tracked. Virtual objects will be registered accurately with the real 3D scene according to the calculated camera position. Instead of using a depth camera, 3D reconstruction could be accomplished by using the 3D video to acquire depth information of the scene in real time. IMS has recently published an approach for real time stereo matching in [9]. In order to capture the environmental lighting situation of the real environment, an HDR camera with a fish eye lens will be used. This camera will produce a hemispherical environment map and will allow a computation of the lighting situation in order to correctly illuminate virtual objects. An Image Based Lighting [10] approach will be used. The environment map will be convolved with the cosine kernel in real-time to get the Irradiance Environment Map [11], which helps simulating the diffuse reflection of the environment light. This will focus on presenting the resulting 3D video with projector on relatively large screen allowing displaying the industrial robot in its real dimensions. Such approach will help to create impression of intercourse with the real robot and the surrounding automated environment build by students in virtual space. To enlarge the presenting area another projectors can be added to the system. As the alternative the head mounted displays could be used to present the resulting 3D video in order to immerse users e.g. the Sony HMZ-T1 head mounted display. Training approach 3D virtual training transforms the relationship between the learner and the training event. 3D virtual training functions quite differently compared with classic training delivery methods, such as classroom training and even traditional e-learning. Following a hybrid learning approach, it’s
applications provides new ways to develop skills competency and mastery. Therefore instructional designers must think differently about their training strategies. In traditional training media, such as classroom training, there is often a significant gap between the time that the learner develops skills in the school and when the same learner puts these skills into practice. At the end of the class, the learners complete a qualification test and demonstrate competency with the task. Then, the learners are sent out to factory to put their new skills to use. However, competency in the classroom does not translate to competency in the factory. Unless people put their new skills to repeated use, they will soon forget what they have just learned. Skills learned in the classroom become a distant memory, and the brain may even create false memories about process and procedure. This may lead to significant undesired consequences: procedural errors, delays, or needless risk to personnel or valuable equipment. To lower these risks, education sciences foster a close collaboration of theoretical teaching in the classroom and on the job training, namely hybrid- or blended learning [12]. Different forms of blended learning were implemented, e.g. a division in blocks of theory and practice, visits of practitioners in the school and – on basis of digital media – theoretical learning content was presented via e-learning platforms to the office, on the job. Very successful training curricula were developed for the financial and the insurance sector. For other sectors like health care and engineering it was harder to do, since it was too dangerous to integrate novices into the running processes in the relevant institutions. The presented solution enables blended-learning in a way that it was originally meant by education scientists: To integrate existing problems and applications into apprenticeship and therewith prepare the student for the relevant tasks in the job without interrupting daily business in the companies and without danger. Within the world of 3D virtual training, real robot situations are remote from trained operator done in a safe place but operator can see robot on real 3D visual transmission system. Real 3D virtual training expands the ways that can be delivered training to geographically distributed learners who need more training in any hour of the day and night. Students are encouraged to solve design problems by modelling the robot environment with 3D virtual models of real elements applied in real production lines. A skilled instructional designer can also shape the content and data flows that occur within 3D virtual training and 3D virtual task training applications. The solutions can incorporate a control menu with reset, help and exit functions and students can exchange their problems and knowledge on blogs using internet. Knowledge assessment The concept of modern Virtual Laboratory aims at supporting students in constructing conceptual knowledge by providing a 3D virtual training environment for robotised and automated facility design and control, a systematic method for obtaining evidence about this increase is required. This will be realized by using innovative assessment techniques and methods. In general, assessment uses questions to draw inferences about the knowledge, skills, attitudes and other characteristics of people. Due to the fact that 3D virtual training functions are quite different compared to classic training delivery methods, multiple-choice questions and the like are not sufficient. They do not allow creativity in answering and mainly evaluate the students' ability in memorising and recalling of factual knowledge. They focus on what is easy to measure rather than what is important. In order to avoid switching between training at the highly-interactive 3D virtual environment and assessment in limited contexts as measured by conventional paper and pencil tests, innovative assessment techniques and methods are applied [13,14]. This allows assessing students in the environment in which they learn and additionally requires and encourages them to use learned knowledge in appropriate situations without being prompted and to recognize situations where the information is relevant. Students should actively thinking, problem-solving, decision-making, etc. in contrast to choosing an answer from a list. This strengthens the process of theoretical knowledge acquiring and conceptual
knowledge building of students. Moreover, in order to avoid treating all students in the same manner, individual aspects (e.g., contexts, prior knowledge and preferences) will be considered in the assessment process as well. Conclusion Authors presented their analysis of requirements for realization of modern virtual robotics laboratory improving the process of education and responding to the needs of the modern market. The lab should enable the simulation of laboratory exercises in real time, under specific pre-defined conditions and in the most precisely simulated environment. It must be emphasised that the virtual reality environment is becoming more and more popular in the field of education and therefore capability to create virtual learning facilities simulating technological process related to, for instance, automation and robotics, will support the learning process, making it more effective and fulfilling the increasing needs of modern market. References [1] [2] [3] [4]
Europe 2020 strategy priorities: http://ec.europa.eu/europe2020/europe-2020-in-anutshell/priorities/index_en.htm The Association of German Engineers (VDI) labour market research: http://www.vdi.eu/economy-policy/labor-market/observing-the-labor-market/ H. M. Abdul-Kader, E-Learning Systems in Virtual Environment, 2008 ITI 6th International Conference on Information and Communications Technology (ICICT 2008)
Henrik Wann Jensen, Realistic image synthesis using photon mapping, A. K. Peters, Ltd., Natick, MA, USA, 2001. [5] Morgan McGuire and David Luebke, Hardware-accelerated global illumination by image space photon mapping, Proceedings of the 2009 ACM SIGGRAPH/EuroGraphics conference on High Performance Graphics (New York, NY, USA), ACM, August 2009. [6] Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich, Optix: A general purpose ray tracing engine, ACM Transactions on Graphics, 2010. [7] Peter Kan, Hannes Kaufmann, Physically-Based Depth of Field in Augmented Reality, submitted to Eurographics 2012. [8] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 559-568, 2011. [9] Hosni, A., Bleyer, M., Rhemann, C., Gelautz, M., Rother, C., "Real-time local stereo matching using guided image filtering," In Proceedings IEEE International Conference on Multimedia and Expo (ICME), pp.1-6, July 2011. [10] Kusuma Agusanto, Li Li, Zhu Chuangui, and Ng Wan Sing, Photorealistic rendering for augmented reality using environment illumination, Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality (Washington, DC, USA), ISMAR '03, IEEE Computer Society, pp. 208-218, 2003. [11] Ravi Ramamoorthi and Pat Hanrahan, An efficient representation for irradiance environment maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques (New York, NY, USA), SIGGRAPH '01, ACM, pp. 497-500, 2001. [12] Bielawski L., Metcalf B.: Blended eLearning, HRD Press Inc. Amherst Massachusetts 2005 [13] Saul, C., & Wuttke, H.-D. (2011). Personalized Assessment of Higher-order Thinking Skills. Proceedings of the CSEDU Special Session on Assessment Tools and Techniques for e-Learning (ATTeL 2011) (pp. 425-430). Noordwijkerhout. [14] Saul, C., & Wuttke, H.-D. (2011). Feedback Personalization as Prerequisite for Assessing Higherorder Thinking Skills. European Journal of Open, Distance and E-learning (EURODL) - Special Issue: Best of EDEN, 15
