Human-Computer Interaction Experiments - Immersive Virtual Reality Applications for the Mining Industry Tomasz P Bednarz, Con Caris, Jeremy Thompson, Chris Wesner, Mark Dunn Exploration and Mining CSIRO Queensland Centre for Advanced Technologies Brisbane, Australia
[email protected] Abstract—The present work concerns immersive virtual reality (IVR) experiments carried out for prospective applications in the mining industry. Demonstrated technologies are also suitable for use in generic training and e-learning fields. Visualisation is handled by the Unity3D 2.5 multiplatform game development tool, which communicates over .Net socket servers with data feeds from a 5DT Data Glove Ultra that measures finger flexures, and an iPhone based touchpad device. The iPhone also provides orientation data pertaining to acceleration and rotational attributes, such as, pitch, roll and yaw. The user is placed at the focal point of a 4-m dome and experiences the immersive virtual reality environment display. Navigation and object manipulation by the user is made possible through the combination of hardwired buttons, in-world selection techniques and gesture recognition. The techniques are directly applicable to the monitoring of mining environments, in which mining equipment is surrounded by various sensor networks. Immersive Virtual Reality; Teleoperation; Dome; Data Glove; iPhone; Sensor Networks; Data Communication
I.
INTRODUCTION
The potential of immersive virtual reality (IVR) technology and human-computer interaction (HCI) techniques as a major technical advance for supporting teleoperation scenarios has been already well recognized by multiple industries [1-3]. One of its unique advantages is the capability to allow operators to visualize abstract concepts, to visit environments and to interact with objects or events that distance, time, scale or safety factors make difficult or impossible to access. In addition, teleoperation scenarios offer safety advantages compared to physical presence of people in harmful environments. The same techniques can also be used in eeducation and for fully interactive training purposes, where all possible scenarios can be presented through a virtual reality world. Therefore, early investigations of possible technologies that can be used in such scenarios are beneficial. It is also important to realise that the same IVR can be used for training applications. Regian, Shebilske and Monk [4] reported on empirical research that explored the instructional potential of immersive VR as an interface for simulation-based training. According to these researchers, VR may hold promise for simulation-based training as the interface preserves both the visual-spatial characteristics of the simulated world and the linkage between motor actions of the student and resulting simulated effects. VR offers the possibility of presenting both small-scale (can be viewed from a single vantage point at a
single instant) and large-scale (experienced across time) [5] spatial information in a three-dimensional format, eliminating the need for users to translate between different visual modalities. Van Dam, Laidlaw and Simpson [6] have provided in their article a snapshot of immersive virtual reality use for scientific visualization. They have emphasized especially a great potential of IVR for dealing with the serious problem of exponentially growing scientific datasets. Their examples have shown IVR to be a potent tool for humans to “see” the patterns, trends and anomalies in data well beyond what can be done today using conventional 3D desktop displays. The Cell Biology Project [7] serves as an example of a study focusing on the impact of VR defined in terms of immersion, natural interaction (via hand trackers) and interactivity on the effectiveness of informal education (i.e., self-directed, unstructured learning experiences). The application was developed in three formats (immersive VR, desktop VR and video-tape viewing) to compare the various visual modalities. Statistics of the modalities highlighted that immersive users scored higher on post-testing of symbolic and graphic retention and were much stronger in terms of level of engagement. However, the non-immersive groups were able to retain more cognitive information (at least for the short term). Pouliquen, Bernard, Marsot and Chodorge [8] have addressed the problem of VR applications for a higher level of integration of safety and health requirements designs. They have reviewed industrial applications using VR and presented virtual hands, which are coupled with a virtual press-brake by using a system of motion capture and a force feedback device. As a result, they were able to estimate the risk level of machine tools. Adamo-Villiani, Carpenter and Arns [9] have presented their development of immersive 3D learning environments designed to increase core mathematical skills in deaf children. Their method taught mathematical concepts and sign language through the user's interaction with fantasy 3D signers and with the virtual environments. Bell and Fogler [10] suggest that VR is a good medium for presenting 3D objects and relationships as well as for illustrating concepts that have been covered elsewhere. However, they did not find it to be an appropriate medium for getting across factual information nor to substitute for traditional educational methods. Yet, when used properly, it can augment traditional methods and lead to improved longterm retention of material. These benefits are also directly applicable to visually oriented students, helping them to more fully engage with the learning process.
II.
CONFIGURATION AND OPERATION
A. Apparatus Fig. 1 shows the logical flow chart of the existing system for the human-computer interaction (HCI) experiments in a virtual reality environment. The user is the first element in the diagram interacting with the system via two input devices: a 5DT Data Glove 5 Ultra (Fifth Dimension Technologies) and an iPhone 3G (Apple Inc), which are attached to one hand. The data glove is connected wirelessly to a PC using the 5DT Ultra Wireless Kit. The data glove measures the flexure/bending of all five digits, reporting them as integer numbers as a function of the angle of intermediate and proximal phalanges. Prior to running the experiments, the glove is calibrated using an algorithm that estimates the minimum and maximum flexion and extension for each digit sensor and then calculates the normalized floating point numbers from 0 to 1.0 i.e., from maximum extension to maximum flexion. The glove calibration is subsequently used in the experiments to characterize the finger/hand movements.
significant bit. For example, the index finger point gesture will indicate number 1 and the little finger point gesture will indicate number 8. If the gesture cannot be recognized, -1 integer value is returned. All gestures were tested in the following way: a picture of a specific gesture is held on the screen for 1500 ms with 500 ms transitional time between successive gestures. These tests provide 90% of the required responses. Although better results can be achieved when the user has more time to prepare specific gestures, different dynamic imitations are achievable as described in detail in [11]. For prototype testing, a simple Paper-Stone-Scissors game was included within the virtual world. In this mode, the user interacts with the game by hand gestures only. The user has three seconds to prepare one of three game gestures: stone (gesture 0), paper (gesture 15) and scissors (gesture 3). The computer randomly picks one figure and compares it with the user's input. The users win when they pick stone against scissors, paper against stone or scissors against paper. If the user and the computer picked the same figure, it is a draw, and in the remaining cases the user loses. The game was tested with ten participants providing 100% correct system's recognition of the input gestures 0, 3 and 15. The orientation data is captured using three accelerometers from the iPhone 3G: one acceleration sensor mounted along each of the primary axes. This combination enables the detection of device movements in any direction and tracks the device's orientation relative to gravity. Access to raw data takes place through the iPhone's SDK UI Accelerometer object. For the experiments, the reporting occurs at 80 Hz as recommended in the iPhone Application Programming Guide for "suitability in applications needed detection of high-frequency motion such as hitting the device or shaking it very quickly" [12]. The raw data needs to be filtered in order to get the current orientation of device and detecting instantaneous motion. Therefore, simplified low-pass and high-pass filters were also implemented.
Figure 1. Logical flow chart of the system.
The iPhone’s touch screen is utilised to display dynamic information and to handle button presses. The iPhone's built in multi-axis accelerometer is used to measure orientation information. Unity3D 2.5 multiplatform game development tool is used to display the user interfaces and 3D objects and to process the interactions transmitted from the input devices. The final output is displayed in an immersive 3D environment on a four metre hemispherical dome screen using two Christie HD3 professional projectors. The projectors are mounted on a ceiling 1.5 metres apart and are located three metres from the focal point of the screen. B. Data and User Interface Interactions Using the 5DT SDK (Software Development Kit) supplied with the Data Glove, sixteen different hand gestures are easily implemented (excluding thumb readings). Gesture number 0 is defined as all the fingers being closed and gesture number 15 as all fingers open. The index finger indicates the least
The user interface and user interactions are implemented via a schema of interaction modes for our testing system as shown in Fig. 2. The system implements two modes of operation: (a) a view mode and (b) an interactive mode. In view mode, the user is not interacting with VR objects and only watches the scenes presented by the graphics engine. The view mode switches to the interactive by one of the following actions: if the user at anytime shakes the iPhone, touches the mode button on the iPhone's screen, or makes the hand gesture number 8 using the data glove. In interactive mode, the main menu is displayed on both the dome screen and the iPhone's touch screen. Input selection by a variety of hardwired buttons on the iPhone’s touch screen enables the user to have access to predetermined controls. These were limited to behaviours such as camera movement and selection, object interaction and menu based functionality. The inbuilt accelerometer allows the user to change the orientation of selected virtual objects by physical translation or rotation of the iPhone.
For gesture supported commands, a small icon is displayed next to the dynamic element. The icon demonstrates the specific gesture associated with the command, enabling ongoing integrated user-interface tuition.
The HSIP protocol has been designed to accommodate a base set of sensors whilst allowing for the simple addition of new devices. Data is transmitted using a set of common XML command strings containing a command type identifier. The general form is data . Table I provides examples of commonly used command strings. TABLE I.
EXAMPLE OF HSIP PROTOCOL
Example command string
Sensor
x, y, z
Android / iPhone / HTC
Acceleration vector
x,y,s
Android / iPhone / HTC
Touch at coordinates (x,y) with a state s (1=single, 2=double etc.)
aabbcc…
Webcam
p
Android / iPhone / HTC
Description
Binary image data Control command to navigate the display to page
The combination of command and the type identifier defines how the data should be interpreted and processed. This arrangement allows for the addition of new sensors by simply defining new command type identifiers. III. Figure 2. User interface interactions modes.
C. Multi-sensor Communication Protocol Fig. 3 depicts the communication flow within the system and outlines the major components. The central element in the system is the Data Server, which acts as an information proxy routing data between sensors and clients. For simplicity and universality, the data server has been implemented in Ruby 1.8.6 using GServer Class [13]. Sensors connected to the data server are either hard-wired or using wireless connection and, after registration, are able to transmit information by a custom XML protocol. This was developed specifically for the experiments and is known as the Human Systems Integration Protocol (HSIP).
IMMERSIVE VIRTUAL REALITY SCENARIOS
The main goal of the experiment presented in this paper is to employ the apparatus and methods described above to create a realistic and intuitive experience within immersive virtual reality scenarios. The system is immersive in that the 3D scenes fill the user's horizontal and vertical peripheral vision. This enhances situational awareness and reduces possible distractions when compared to viewing of the same virtual world on the monitor screen. When the field of view is only partially covered by the content, the user's eyes can easily catch motion changes in the background not belonging to the visual system (such as a person passing the room). The user attaches the data glove and iPhone to the same hand/wrist for interaction inputs (gestures and orientation). Two different scenarios are described: a three-dimensional (3D) mine engineering e-book and a teleoperated surface mine. A. Virtual Mine E-book The first example of IVR arrangement presents a 3D mine engineering e-book (Fig. 4) for an underground mining equipment system called "Longwall Shearer Guidance" as developed by the CSIRO Mining Automation team [14]. The 3D virtual book responds to the inputs provided by the user: the user can flip pages, zoom in and out of the pictures in the ebook and access/navigate the virtual worlds linked to the text. This functionality makes this system unique when compared to standard books or book viewers simply because the user can experience the content, potentiating longer-term recall.
Figure 3. Multi-sensor communication protocol.
content including web-pages, multimedia and other 3D VR scenes. The user can change linked content to full-screen display by making a specific hand gesture or by touching corresponding menu item visible on the iPhone's screen.
(a)
As mentioned previously, hyperlinks are able to load additional 3D VR scenes, potentially expanding the users experience by layering additional features. These include different camera modes for 3D navigation, the ability to record experiences (so they can be re-played later) and also ability to pick and change object features or orientations. Fig. 4c shows a snapshot of a linked 3D world that allows dynamic interaction with a previously static scene. The user can return to the source e-book by making the "stone" gesture or by touching the "exit" button on the iPhone.
(b)
(c)
B. Surface mine teleoperated scenario The immersive virtual reality of the surface mine is another working example of teleoperated scenario. In this case, the landscape of the real surface mine is reconstructed based on Geographic Information System (GIS) data and converted to a format suitable for Unity 3D. The mine surface equipment is then geo-referenced using GPS and passed to a virtual reality data server to construct a calculated virtual mine representation. Real-time tracking of equipment can be improved by using coupled IMU/GPU devices as presented in [15]. Input devices, such as, the data glove and iPhone, are used to interact with the camera viewports and objects and, eventually, to send specific safety commands to running equipment, where required. The cameras’ paths and views are dynamically optimised to include the location and orientation of equipment so that picked objects display consistently and correctly. The geo-referenced locations of the objects are stored in a database for replaying scenarios or for referencing purposes. Fig. 5 shows example of reconstructed surface mine scenario. A user is located at the front of the 4m spherical dome with the 5DT data glove and the iPhone attached to his right hand. Using touch screen or hand gestures, the user is able to pick up different options and send commands to displayed objects. In this example, the user is observing a truck moving along the virtual mine landscape and is able to check intrinsic information, such as, carried tonnage on the iPhone screen.
Figure 4. Virtual Mine E-book: (a) inside Virtual Mine Centre (VMC) at CSIRO QCAT, (b) virtual e-book, demonstrating gesture icons, (c) inside the virtual underground mine.
As showed in Fig. 2, the user can be in two modes: view mode and interactive mode. The view mode is a safe/lock mode i.e., the user can only read the book pages and cannot do any unexpected actions or send random inputs to the system, for instance, when changing pose or talking to someone using hands to gesture an alternative concept. In the interactive mode, the users interact directly with the e-book; they can flip the pages using gestures displayed in the lower right-hand corner of the screen or touch equivalent icons on the iPhone's touch screen. Discrete elements of the e-book's pages can be configured as live hyperlinks, enabling the user to access other
Figure 5. Surface Mine Scenario.
IV.
SUMMARY
In this paper, we have demonstrated the use of IVR applications for delivering an interactive environment for possible training purposes in underground coal mining and for teleoperation in surface mining environment. The first example has facilitated an e-learning experience by providing a familiar book object as the target for user’s interaction. Importantly, this IVR system has the potential for providing an "enhanced" teaching medium as humans tend to remember events/situations better when they experience those events in person. The IVR environment enables a user to physically interact with various 3D objects and behaviours, which, in turn, reinforces the idea that doing leads to efficient learning. Our prototype system can be implemented as an efficient and cost effective e-learning and training platform for different engineering domains. In addition, the same technology could be used in the teleoperation of mining equipment as presented in the second example. CSIRO has already developed VR systems to remotely monitor mining equipment (LASC Longwall Automation [14], and the Nexsys risk management system). Collaborative industry projects are currently underway that will see CSIRO leverage IVR technology to demonstrate that teleoperation can be a reality for everyday use. It is important that next generation e-learning systems are developed to provide new training media for the effective use of advanced control systems in the future. REFERENCES [1]
[2]
D. Weidlich, L. Cser, T. Polzin, D. Cristiano, and H. Zickner, “Virtual reality approaches for immersive design”, CIRP Annuals – Manufacturing Technology, vol. 56, pp. 139-142, 2007. J. Nomura, and K. Sawada, “Virtual reality technology and its industrial applications”, Annual Reviews in Control, vol. 25, pp. 99-109, 2001.
[3]
[4]
[5]
[6]
[7] [8]
[9]
[10]
[11]
[12] [13] [14] [15]
M. W. Bowyer, K. A. Streete, G. M. Muniz, and A. V. Liu, “Immersive virtual environments for medical training”, Seminars in Colon and Rectal Surgery, vol. 19, pp. 90-97, 2008. J. W. Regian, W. L. Schebilske, and J. M. Monk, "Virtual reality: An instructional medium for visual-spatial tasks". Journal of Communication, vol. 42, pp. 136-149, 1992. A. W. Siegal, "The externalization of cognitive maps by children and adults: In search of way to ask better questions". In I. Liben, A. Patterson, & N. Newcombe (Eds.), Spatial representation and behaviour across the life span , New York: Academic Press, pp.163-189, 1981. A. van Dam, D. H. Laidlaw, and R. M. Simpson, “Experiments in immersive virtual reality for scientific visualization”, Computers & Graphics, vol. 26, pp. 535-555, 2002. E. Gay, “Is virtual reality a good teaching tool?”, Virtual Reality Special Report, Winter, pp. 51-59, 1994. M. Pouliquen, A. Bernard, J. Marsot, and L. Chodorge, “Virutal hands and virtual reality multimodal platform to design safer industrial systems”, Computers in Industry, vol. 58, pp. 46-56, 2007. N. Adamo-Villani, E. Carpenter, and L. Arns, "3D sign language mathematics in immersive environment", Proc. of the 15th IASTED International Conference, Rhodes, Greece, June 2006. J. T. Bell, J. S. Fogler, "Preliminary testing of a virtual reality based educational module for safety and hazard evaluation training", Proc. American Society for Engineering Education Annual Conference, Indiana Sectional Meeting, Peoria, IL, 1996. B. J. Gold, M. Pomplun, and N. J. Rice, “A new way to quantify the fidelity of imitation: preliminary results with gesture sequences”, Experimental Brain Research, vol. 187, pp. 139-152, 2008. Apple Inc., “iPhone Application Programming Guide”, 2009. www.ruby-doc.org/stdlib/libdoc/gserver/rdoc/classes/GServer.html LASC Longwall Automation http://www.lascautomation.com/ T. P. Bednarz, C. Caris, C. Wesner, P. B. Reid, G. Falco, G. Einicke, and J. Malos, “Experiments utilizing data glove and high-performance INS devices in an immersive virtual mining environment”, International Global Navigation Satellite Systems Symposium Symposium 2009, Gold Coast, 1-3 December 2009.
