REMOTE LAB IN VIRTUAL WORLD FOR REMOTE ... - Semantic Scholar

REMOTE LAB IN VIRTUAL WORLD FOR REMOTE CONTROL OF INDUSTRIAL PROCESSES J. Fayolle, C. Gravier

N. Yankelovich, E. Kim

Télécom Saint-Etienne Université de lyon 25 rue Annino 42000 Saint-Etienne, France

Open Wonderland Foundation P.O. Box 44926 Eden Prairie MN. 55344, USA

ABSTRACT This paper addresses ways users can remotely control industrial hardware devices over the Internet. We employ a J2EEbased Remote Lab platform to describe the interface of the remote devices and to relay commands and results between users and devices. Collaboration is accomplished by embedding the remote lab platform in an Open Wonderland virtual world. Index Terms— Remote Lab, Virtual world, ontology, collaboration, e-manufacturing 1. INTRODUCTION The manufacturing industry worldwide has been facing unprecedented challenges due to the globalization of markets. E-commerce and Internet technologies have enabled companies to shift their manufacturing operations from traditional factory settings to supply-chain-based e-factories, transforming companies from local factories to global enterprises. To compete in this environment, companies started to pool their resources and form groups of economic interest, often referred to as an “extended enterprise.” Those companies in the extended enterprise can now compete against otherwise wellfunded, well-resourced, highly capitalized firms. This situation is particularly applicable in research and development, where companies often need quick access to a given resource, but may only need that resource for a limited time. This may not justify the investment to purchase the equipment, but it is nonetheless a critical step in the development cycle. By gaining access to equipment through groups of economic interest, firms can conduct work that would otherwise be out of reach economically. Also, different phases of work such as “production” and “monitoring” may not always be co-located. That is to say, expensive, specialized equipment is not always controlled in the same place or locale as the operator. This concept of e-manufacturing is well defined in [1] as the use of advanced and emerging information technologies to provide automated, data-driven productivity optimization.

One challenge of e-manufacturing is providing data at the right time, in the right place, and in a understandable format. The assumption in the past has been that one does not have control of remote systems and, therefore, is reduced to doing one’s best with the data collected. We challenge this with a different approach. We believe that people can now interact directly with production systems and remote information systems in order to build an entire remote e-manufacturing system from beginning to end. The objective of our work, therefore, is to be able to remotely control devices in order to design and build new products, train operators, and maintain systems. Two factors lead us to believe that this is now possible: the development of remote lab technologies and the availability of collaborative virtual world environments. The pillars of a reusable e-manufacturing framework for monitoring and maintenance of remote systems rely on: • A generic method to support a collection of remotely controllable heterogeneous devices. • Security and sufficient bandwidth to support geographically distributed machines and personnel. • Robust data privacy controls to ensure that data is not lost or compromised. • Collaboration tools to understand what others are doing, to interact with others, and to provide remote training capabilities. To ensure the generalizability of this framework, we need to identify a level of abstraction for both the hardware and client devices. Below we describe our approach to creating a general e-manufacturing framework that relies on remote laboratory technology in association with a 3D virtual world environment that uses multimedia to enhance collaboration. 2. RELATED WORK A primary challenge in developing an end-to-end e-manufacturing system is to merge the technologies used in the manufacturing

enterprise with modern Internet technologies including multimedia, virtual worlds, and semantic descriptions. While a number of projects have investigated the use of virtual worlds in the manufacturing context [2, 3, 4], even for use in a chocolate factory [5], none have combined control of remote laboratory equipment with both collaboration and training. Recently, many device manufacturers have started to offer web services to collect sensor data. A survey of these systems can be found in [6]. In addition to web services, these companies provide access to their device data through the network using methods such as Open GeoSpatial Consortium’s (OGC) framework of open standards for web-connected sensors[7], Cristaldi et al’s multi-sensor agent platform [8], or even semantic approaches to sensor access [9]. Collaboration is a vital component of our e-manufacturing solution. One of the main drawbacks of a remote lab is isolation of the remote user (a caricatured geek, alone in front of his computer ;) ). We were inspired to consider the use of virtual worlds as a mechanism for remote laboratory collaboration after reading the Wonderblog article, “Reviewing Wonderland Code In-World” [10]. By situating the remote laboratory in a virtual world, we can provide an authentic context for working together with laboratory equipment. As Dede points out, the value of collaborating together in a computer-mediated space includes building “social network capital” (access to other people with relevant skills), “knowledge capital” (quick access to people who can answer questions on the spot), and “communion” (moral support from people who share a similar circumstance and understanding of the current situation) [11]. In terms of remotely training personnel to operate laboratory equipment, a number of groups offer web-based distance education courses ([12, 6, 13, 14]), sometimes using multimedia streaming [15] or integrated environments such as the Matlab toolkit [13]. The problem with these pure web-based approaches is that students are not adequately prepared to operate real equipment. They lack practice in manipulating real devices. To address these shortcomings, we are adopting McCarthy and Wright’s “felt-life” approach [16], merging the remote laboratory with a virtual world to promote collaboration and reproduce a felt-life sensation. The use of multimedia tools is key to obtaining this goal. Another important aspect of our work is designing a general remote control framework such that the remote user interface presented in the virtual world closely mirrors the physical interface in the real world. Our objective is to create a generic framework that will work with any device since we do not want to hand-code user interfaces for a large number of laboratory devices. Fortunately, the information needed to control most devices is similar (commands, answers, parameters, etc.) and sometimes, this information is normalized. Hao, et. al. propose a Service Oriented Architecture approach to modeling devices[17] in order to achieve the adaptability we are aiming for. We have adopted this approach cou-

pled with semantic modeling, using an ontology to describe the attributes of devices[18]. This allows us to describe the appearance of the remote control panel at the same time as the functionality of the device (linked to the command widgets). While the benefits of remotely controlling devices are numerous, it is evident remote control can easily create security risks. If we are to use remote control inside real manufacturing operations, we must validate that we are strictly adhering to security policies. The issue of security in e-manufacturing is rarely addressed in the literature, although Lujiong, et al do discuss authentication[19], promoting a role-based access scheme for networked manufacturing systems. In a paper dealing with e-health and e-business for m-commerce, Tan et al[20] put the focus on the same point: the issue of security, privacy, and integrity of information and transactions being exchanged from one point on the network to another is seen as a key barrier to making mobile solutions a reality. An obvious solution to mitigating security risks is to limit the number of entry points. The use of a virtual world can help in this regard since all the collaboration tools and commands can be embedded in the same channel between the virtual world server and the rest of the enterprise information system. In the related work presented above, we have found some partial solutions, but nothing as comprehensive as our emanufacturing framework which encompasses security, user interface, remote control, training, and collaboration. 3. REMOTE LAB IN VIRTUAL WORLDS To create a remote laboratory in a virtual world, we first have to specify the interfaces to each device. To do this, we use a Web Ontology Language file [21] to describe the graphical user interface (location, size, skill level of each widget) and the functionalities of each external device. An Enterprise Java Bean loads and interprets the semantic description. Commands and results are broadcast to each participant through message-oriented middleware. The client is a standalone application which is deployed on the client side through Java Web Start technology. On the server side, we use a Jonas J2EE server and JORAM Message Middleware [22]. We have taken a semantic web approach to describing the collaboration policy on the remote lab platform. To do that, we used the Semantic Web Rule Language [23]. This tool allows us to represent a typical collaboration situation. We identify who is the actor, and who are the spectators. We further identify who is requesting to become the actor (we assume that there is only one actor in the remote lab at a time). We describe the collaboration policy, for example as, “User in late has the higher priority to access the lab, however, instructor or superuser still gains preemptive access.” Consider the following scenario. Suppose that Dave and Alice, two co-workers, have to run an experiment on device 1, whereas Bob (another worker) has to work on device 2.

Using the web-based remote lab technology available today, Dave, Alice and Bob launch the remote lab software (for example, an instance of an Ocelot client using Java Web Start) on their own computers, which then loads the semantic file corresponding to the device they wish to use. Group awareness/interaction between Dave and Alice is facilitated by the JORAM messaging service which relays the commands made by Dave to Alice’s screen, for example by displaying the button pushed by Dave in a specific color. It is difficult, however, for Dave to know what Alice is doing. She could be reading a lab report, taking notes, or watching him run the experiment. We rate this solution as good but not optimal. By integrating and using the device control software in a virtual world such as Open Wonderland, collaboration between Dave and Alice is manifested by avatar actions and interaction. Now, when carrying out the actions described above, Dave can clearly see the level of Alice’s engagement in the process. He can now see if she is watching the remote device, reading a document exposed as a PDF on a virtual wall, or taking notes on the shared whiteboard. Another advantage of working collaboratively in the virtual world is that they are now working on the same instance of the Ocelot remote client. This reduces the number of client launches, simplifies communication requirements between multiple instances of the client, and removes competitive/conflicting commands sent to the remote hardware (see figure 1). Moreover, this scheme allows us to use the suite of collaboration tools provided by the Open Wonderland toolkit, such as Voice over IP channels, text chats, multi-user whiteboards, sticky notes, webcam video, or shared applications such as a web browser, a spreadsheet, or a data analysis tool. Apart from needing a dedicated server to host the Open Wonderland world and the Ocelot server, there are no substantial additional cost to an enterprise for integrating devices into Wonderland. All these tools are based on Java server technology. The main additional overhead is network bandwidth. The bandwidth requirements for the use of avatars and audio communication adds some data transfers and increases network traffic. 4. INDUSTRIAL PROCESS WITH REMOTE LAB IN VIRTUAL WORLDS The primary considerations for adopting a remote manufacturing system in real industry settings are inclusion of effective collaboration and communication tools and remote laboratory realism (a realistic laboratory user experience). In terms of realism, we create and assign a remote user interface to the laboratory equipment that is as close as possible to the physical interface. A user familiar with how to operate the real equipment in a lab setting should be able to easily and seamlessly use the same device via the remote user interface in the virtual world. The use of a virtual world helps us reach this goal since it allows 3D representations of objects and,

Fig. 1. Scheme of data exchanges between: users, avatars, the remote lab server and the devices inside the enterprise. therefore, realistic renderings of the physical devices (figure 2). To date, we have created a working prototype that includes a mechanism for describing a wide range of ISIcompliant electronic devices in an ontology, remote controlling those devices through the OCELOT remote lab platform, and embedding the OCELOT user interface in an Open Wonderland virtual world. Many steps of this process, however, currently require manual intervention. In the next phase of the project, we will focus on adapting the system for use in a real industrial e-manufacturing setting. Once achieved, we believe our approach can be used to support to a large set of devices, including devices related to electronics, robotics, mechanics, and a variety of other fields. 5. REFERENCES [1] J. Lee, “E-manufacturing-fundamental, tools, and transformation,” Robotics and Computer Integrated Manufacturing, vol. 19, no. 6, pp. 501–507, 2003. [2] Prashant Banerjee and Dan Zetu, “Virtual manufacturing,” in John Wiley and Sons, 2001. [3] T. S. Mujber, T. Szecsi, and M. S. J. Hashmi, “Virtual reality applications in manufacturing process simulation,” in Journal of Materials Processing Technology, 2004, vol. 155-156, pp. 1834–1838. [4] L Jin, IA Oraifige, and PM Lister, “E-manufacturing in networked virtual environments,” in IEEE International Conference on systems man and Cybernetics, 2001, vol. 3, pp. 1845–1849.

turing,” Robotics and Computer Integrated Manufacturing, vol. 20, no. 6, pp. 563–571, 2004. [13] A. R. S. Castellanos, L. H. Santana, E. Rubio, I. S. Ching, and R. A. Santonja, “Virtual and remote laboratory for robot manipulator control study,” The International Journal of Engineering Education, vol. 22, no. 4, pp. 702–710, 2006. [14] Z. Yanga and Q. Liu, “Research and development of web-based virtual online classroom,” Computers & Education, vol. 48, no. 2, pp. 171–184, 2007.

Fig. 2. E-Manufacturing prototype showing three remote users in an Open Wonderland virtual world using collaboration tools and a Network Analyzer displayed with OCELOT.

[15] H Abdel-Wahab, K Maly, A. Youssef, E Stoica, M Overstreet, K Wild, and A Gupta, “The software architecture and interprocess communications of IRI: an Internetbased interactive distance learning system,” in 5th IEEE Int. Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 1996, p. 4.

[5] M. Back et al, “The virtual chocolate factory: mixed reality industrial collaboration and control,” in Proceedings of the international conference on Multimedia, 2010, pp. 1505–1506.

[16] J. McCarhty and P. Wright, “Putting felt-life at the centre of human-computer interaction.,” Proceedings of In Reflective HCI Workshop, 2004.

[6] Christophe Gravier, Jacques Fayolle, Bernard Bayard, Mikael Ates, and Jeremy Lardon, “State of the art about remote laboratories paradigms - foundations of the ongoing mutations,” iJOE : International Journal of Online Engineering, vol. 4, no. 1, pp. 19–25, February 2008. [7] Mike Botts, George Percivall, Carl Reed, and John R sensor web enablement: Overview Davidson, “Ogc� and high level architecture,” in Lecture Notes in Computer Science, 2008, vol. 4540, pp. 175–190. [8] L. Cristaldi, M. Faifer, F. Grande, and R. Ottoboni, “An improved m2m platform for multi-sensors agent application,” in ISA and IEEE Sensors for Industry Conference, 2006, pp. 79–83. [9] Vincent Huang and Muhammad Kashif Javed, “Semantic sensor information description and processing,” in Second International Conference on Sensor Technologies and Applications, 2008, pp. 456–461. [10] Jonathan Kaplan, “Reviewing code in wonderland,” reviewing wonderland code in world on Wonderblog from blogs.sun.com, Nov 2009. [11] Chris Dede, The Evolution of Constructivist Learning Environments: Immersion in Distributed, Virtual Worlds, Constructivist Learning Environments: Case Studies in Instructional Design, 1996. [12] L. Wang, P. Orban, A. Cunnigham, and S. Lang, “Remote relatime cnc machnining for webbased manufac-

[17] Q. Hao, W. Shen, and L. Wang, “Towards a cooperative distributed manufacturing management framework,” Computers in industry, vol. 56, no. 1, pp. 71–84, 2005. [18] Christophe Gravier, Jacques Fayolle, and Martin O’Connor, “Adaptive system for collaborative online laboratories,” IEEE Intelligent Systems, vol. To be published, December 2011. [19] W. Lujiong, L. Aiping, and X. Liyun, “A security model for networked manufacturing system,” in International Conference on Computational Intelligence and Security, 2007, pp. 745–749. [20] J. Tan, H.J Wen, and T. Gyires, “m-commerce security : the impact of wireless application control (wap) security services on e-business and e-health solutions,” International journal of Mobile communications, vol. 1, no. 4, pp. 409–424, 2003. [21] Frank van Harmelen Deborah L. McGuinness, “Owl web ontology languageoverview,” W3C REcommendation, 2004. [22] OW2 Consortium, “http://www.ow2.org/,” . [23] Ian Horrocks, Peter F. Patel-Schneider, Harold Boley, Said Tabet, Benjamin Grosof, and Mike Dean, “Swrl: A semantic web rule language combining owl and ruleml,” W3C Member Submission, 2004.