LâOréal used technology from CrowdOptic Inc. to allow users at the annual Luminato Festival in. Toronto to ...... basic ARVIKA systems, Wolfsburg: SMAR. Pang ...
A CRITICAL INVESTIGATION INTO THE POTENTIAL OF AUGMENTED REALITY APPLICATIONS IN THE MINING INDUSTRY
Jónatan Jacobs Submitted in partial fullfilment for the degree
B.Eng (Hons) (Mining Engineering)
IN THE FACULTY OF ENGINEERING, BUILT ENVIRONMENT AND INFORMATION TECHNOLOGY DEPARTMENT OF MINING ENGINEERING UNIVERSITY OF PRETORIA
Year: 2015
1
DECLARATION I hereby declare that this dissertation is my own unaided work. It is being submitted for the degree B.Eng(Hons)(Mining Engineering) at the University of Pretoria. It has not been submitted before for any degree or examination in any other University. This document represents my own opinion and interpretation of information received from the mine or people on the mine.
__________________________________(signature document) NAME OF AUTHOR Dated this ________day of _____________
i
on
copy
of
final
UNIVERSITY OF PRETORIA FACULTY OF ENGINEERING, BUILT ENVIRONMENT AND INFORMATION TECHNOLOGY DEPARTMENT OF MINING ENGINEERING The Department of Mining Engineering places great emphasis upon integrity and ethical conduct in the preparation of all written work submitted for academic evaluation. While academic staff teach you about systems of referring and how to avoid plagiarism, you too have a responsibility in this regard. If you are at any stage uncertain as to what is required, you should speak to your lecturer before any written work is submitted. You are guilty of plagiarism if you copy something from a book, article or website without acknowledging the source and pass it off as your own. In effect you are stealing something that belongs to someone else. This is not only the case when you copy work word-by-word (verbatim), but also when you submit someone else’s work in a slightly altered form (paraphrase) or use a line of argument without acknowledging it. You are not allowed to use another student’s past written work. You are also not allowed to let anybody copy your work with the intention of passing if of as his/her work. Students who commit plagiarism will lose all credits obtained in the plagiarised work. The matter may also be referred to the Disciplinary Committee (Students) for a ruling. Plagiarism is regarded as a serious contravention of the University’s rules and can lead to expulsion from the University. The declaration which follows must be appended to all written work submitted while you are a student of the Department of Mining Engineering. No written work will be accepted unless the declaration has been completed and attached.
I (full names)
_____________________________________________________
Student number _____________________________________________________ Topic of work
_____________________________________________________
Declaration 1.
I understand what plagiarism is and am aware of the University’s policy in this regard.
2.
I declare that this ______________________ (e.g. essay, report, project, assignment, dissertation, thesis etc) is my own original work. Where other people’s work has been used (either from a printed source, Internet or any other source), this has been properly acknowledged and referenced in accordance with departmental requirements.
3.
I have not used another student’s past written work to hand in as my own.
4.
I have not allowed, and will not allow, anyone to copy my work with the intention of passing it off as his or her own work.
Signature __________________________________
ii
Language Edit
I, _____________________________________ hereby declare that I performed an English language edit on the final version of this project report.
_____________________
___________________
Signature
Date
iii
ABSTRACT A CRITICAL INVESTIGATION INTO THE POTENTIAL OF AUGMENTED REALITY APPLICATIONS IN THE MINING INDUSTRY JÓNATAN JACOBS Supervisor: Professor Ronny Webber-Youngman Department: Mining Engineering University:
University of Pretoria
Degree:
B.Eng (Hons) (Mining Engineering)
The purpose of the study was to gain a thorough understanding of Augmented Reality technology, to determine how it has already been applied in other non-mining areas and what potential future applications have been identified. This information was used to critically look at the potential of using AR technology in the mining industry and to what extent it would add to an efficient mining process. Multiple potential applications were identified through various means. The applications identified for the mining industry were further evaluated to determine what value it could potentially add to mining. A SWOT analysis was conducted on the applications that were deemed most likely to be implemented successfully at present. This helped to identify numerous benefits as well as challenges associated with the technology. The general conclusion is that AR could bring multiple benefits to the mining industry and greatly add value to an efficient mining process. However, careful investigation would be required into how an application should be designed in order to ensure an efficient system that would deliver the required benefits while minimising the associated challenges.
iv
ACKNOWLEDGEMENTS I wish to express my appreciation to the following organisations and persons who made this project report possible: 1
Professor RCW Webber-Youngman, my supervisor.
2
The University of Pretoria.
v
TABLE OF CONTENTS DECLARATION .............................................................................................................................. I ACKNOWLEDGEMENTS ............................................................................................................. V LIST OF FIGURES ..................................................................................................................... VIII LIST OF TABLES ......................................................................................................................... X LIST OF SYMBOLS ..................................................................................................................... XI CHAPTER 1: INTRODUCTION ...................................................................................................... 1 1.1
Background to AR Technology ......................................................................................... 1
1.2
History of AR .................................................................................................................... 2
1.3
Project Background .......................................................................................................... 3
1.4
Problem Statement ........................................................................................................... 5
1.5
Scope of the Study ........................................................................................................... 5
1.6
Objectives and Methodology ............................................................................................ 6
REFERENCES ........................................................................................................................... 7 CHAPTER 2: LITERATURE SURVEY ........................................................................................... 9 2.1
Introduction...................................................................................................................... 9
2.2
Technology ....................................................................................................................... 9
2.2.1
Hardware................................................................................................................... 9
2.2.2
Display Technology ................................................................................................. 10
2.2.3
Input Devices........................................................................................................... 17
2.2.4
Tracking .................................................................................................................. 18
2.2.5
Computer ................................................................................................................ 18
2.3
Applications of Augmented Reality ................................................................................ 18
2.3.1
Archaeology ............................................................................................................ 19
2.3.2
Architecture ............................................................................................................. 20
2.3.3
Art Interventions ...................................................................................................... 21
2.3.4
Beauty .................................................................................................................... 22
2.3.5
Commerce (Sales and Marketing) ........................................................................... 22
2.3.6
Construction ............................................................................................................ 24
2.3.7
Education ................................................................................................................ 25
2.3.8
Emergency Management/Search and Rescue......................................................... 29
2.3.9
Everyday ................................................................................................................. 30
2.3.10
Gaming.................................................................................................................... 31 vi
2.3.11
Industrial Design ...................................................................................................... 33
2.3.12
Maintenance, Repair and Task Support................................................................... 34
2.1.13
Medical .................................................................................................................... 35
2.3.14
Military ..................................................................................................................... 37
2.3.15
Navigation ............................................................................................................... 38
2.3.16
Social Networking .................................................................................................... 40
2.3.17
Translation .............................................................................................................. 40
2.3.18
Mining...................................................................................................................... 41
2.4
Conclusions .................................................................................................................... 52
REFERENCES ......................................................................................................................... 57 CHAPTER 3: RESULTS AND ANALYSIS OF RESULTS ............................................................ 64 3.1
Drilling Applications ........................................................................................................ 64
3.2
Navigational Aid and Operator Assistance ...................................................................... 70
3.3
Maintenance and Repair................................................................................................. 75
3.4
Real-Time Information .................................................................................................... 79
3.5
SWOT Analysis of Augmented Reality Applications in the Mining Industry ..................... 85
REFERENCES ......................................................................................................................... 90 CHAPTER 4: CONCLUSIONS ..................................................................................................... 91 CHAPTER 5: RECOMMENDATIONS .......................................................................................... 99 CHAPTER 6: SUGGESTIONS FOR FURTHER WORK ............................................................. 100
vii
LIST OF FIGURES Figure 1.3a: LandForm, an experimental landing assistant system for helicopters (futurescienceleaders.com, 2014) ....................................................................................... 3 Figure 1.3b: BMW Augmented Reality Maintenance (imponderablethings.com, 2013) ................... 4 Figure 2.2.2a: Augmented Reality glasses, Wrap 920AR, from Vuzix (technologyreview.com, 2015)................................................................................................................................. 11 Figure 2.2.2b: Google Glass (allthingsd.com, 2015) ..................................................................... 12 Figure 2.2.2c: The Technology Partnership: "Heads-up" prototype AR glasses (theguardian.com, 2015)................................................................................................................................. 13 Figure 2.2.2d: Z800 Pro AR HMD (vrealities.com, 2013) .............................................................. 14 Figure 2.2.2e: Microsoft HoloLens (informationweek.com, 2015) .................................................. 15 Figure 2.2.2f: Augmented tabletop map (newscientist.com, 2005) ................................................ 16 Figure 2.2.2g: Avegant VRD prototype (engadget.com, 2013) ...................................................... 17 Figure 2.3.5a: Augmented furniture (dezeen.com, 2013) .............................................................. 23 Figure 2.3.5b: AR used for marketing to display additional content (Gigaom.com, 2012) .............. 24 Figure 2.3.7a: Letters Alive teaching kids the building blocks of literacy (prweb.com, 2011) ......... 26 Figure 2.3.7b: An augmented 3D display of a human heart with Anatomy 4D (qualcomm.com, 2015)................................................................................................................................. 27 Figure 2.3.7c: An augmented 3D display of a human body with Anatomy 4D (qualcomm.com, 2015)................................................................................................................................. 27 Figure 2.3.7d: AR used to aid in the visualisation of biology (mahei.es, 2015) .............................. 28 Figure 2.3.10: Minecraft Reality illustration (rockpapershotgun.com, 2015) .................................. 31 Figure 2.3.13: Treating cockroach phobia with AR (technologyreview.com, 2010) ........................ 36 Figure 2.3.17: Future of social networking with AR – Concept investigation (matthewbuckland.com, 2010)................................................................................................................................. 40 Figure 2.3.15: Maptek's PerfectDig displaying augmented design plans overlaid onto actual excavations (sparpointgroup.com, 2013)........................................................................... 47 Figure 3.1a: Augmented drill hole with indicated deflection on the drill head ................................. 65 Figure 3.1b: End of directional drill head (Schlumberger, date unknown)...................................... 68 viii
Figure 3.1c: Cross-section of drill end used in directional drilling (Schlumberger, date unknown) . 68 Figure 3.2a: An augmented outline of an approaching haul truck ................................................. 71 Figure 3.2b: Augmented overlays for operators ............................................................................ 72 Figure 3.2c: AR overlaid information during loading process ........................................................ 73 Figure 3.3: Example of an AR app for maintenance of a haul truck (simfusionar, 2015) ............... 77 Figure 3.4a: AR specification information on a haul truck tire........................................................ 80 Figure 3.4b: Real-time AR information in vehicles ........................................................................ 81 Figure 3.4c: An AR display of information on mining equipment ................................................... 82
ix
LIST OF TABLES Table 1.6: Objectives and Methodology .......................................................................................... 6 Table 3.1a: Strengths and Weaknesses Analysis for AR drilling applications ............................... 69 Table 3.1b: Opportunities and Threats Analysis for AR drilling applications .................................. 70 Table 3.2a: Strengths and Weaknesses Analysis for navigational aid & operator assistance........ 73 Table 3.2b: Opportunities and Threats Analysis for navigational aid & operator assistance .......... 75 Table 3.3a: Strengths and Weaknesses Analysis for maintenance and repair .............................. 78 Table 3.3b: Opportunities and Threats Analysis for maintenance and repair ................................ 79 Table 3.4a: Strengths and Weaknesses Analysis for real-time AR information ............................. 83 Table 3.4b: Opportunities and Threats Analysis for real-time AR information ............................... 84 Table 3.5a: Strengths and Opportunities Analysis for AR applications in the mining industry........ 85 Table 3.5b: Opportunities and Threats Analysis for AR applications in the mining industry ........... 88 Table 4a: Summary of findings from the literature study ............................................................... 91 Table 4b: Summary of findings from the results ............................................................................ 95 Table 4c: Summary of SWOT analysis of potential AR applications for the mining industry .......... 97
x
LIST OF SYMBOLS
3D
Three dimensions/Three dimensional
APP
Computer Program/Application
AR
Augmented Reality
FoG
Fall of Ground
GPS
Global Positioning System
HMD
Head Mounted Display
HUD
Head-Up Display
IHPC
Institute of High Performance Consulting
MEMS
Microelectromechanical systems
PC
Personal Computer
RDO
Rock Drill Operator
SOP
Standard Operating Procedure
xi
CHAPTER 1: INTRODUCTION Motivation for this Study Augmented Reality (AR) technology has been significantly improved over the last few years. Although AR is a technology that has been around for decades (examples to follow), it has become more important in the visual technology age that we currently find ourselves in. It has the potential to make significant contributions, and the purpose of this investigation was to explore and evaluate this aspect, with specific reference to applications of AR in the mining industry.
1.1
Background to AR Technology
Augmented Reality (AR) is either a direct or indirect, live (real-time) view of the physical (realworld) environment. Elements such as any object or environment viewed in the real world, are augmented (supplemented) by a computer-generated sensory input, including video, graphical displays, sound or Global Positioning System (GPS) data (Graham, 2012). AR is related to the concept known as ”mediated reality”, where a person‟s view of reality is similarly modified by a computer or hand-held device such as a smartphone. This modification entails adding to, subtracting information from, or otherwise manipulating a person‟s perception of reality through a computerised device that is able to display these digital, visual alterations. The difference between these two technologies is that AR is able to draw the digitalised information, for its augmentation purposes, from an “online” source (such as the Internet or another server containing information, or even through the use of GPS location tracking). Mediated reality, on the other hand, predominantly functions “offline”, therefore it utilises only the information available on the device itself (Rey, 2011). Both Augmented and Mediated Reality technologies function by enhancing a person‟s current perception of reality (Graham, 2012). In contrast to AR is a concept known as Virtual Reality (VR), a technology in which the real world is completely replaced by a simulated virtual world through the use of a computer system (Steuer, 1993). The real-time overlay of sport scores (and related information such as player names and penalty durations) over a live sports match video broadcast is considered to be one of the most basic modern-day examples of AR in our day-to-day lives (Maxwell, 2010). This example is something which most people would, undoubtedly, not recognise as augmented reality. This simply illustrates how
long
this
technology
has
been
explored
(Augmentedrealityon, 2012).
1
in
numerous
types
of
applications
In recent years, AR has advanced much further than simple overlays of television-based information. The technology now allows information in the real world to be interactive and amenable to digital manipulation through the use of computer vision-and-object recognition. Artificial information about the environment and the identified objects (such as car parts, highwalls, mining equipment, etc.) can be overlaid on the real-world environment. Such overlays can then be interacted with via a variety of different devices (Chen, 2009).
1.2
History of AR
The term “Augmented Reality” is attributed to a T. Caudell, a former Boeing researcher, who coined the term in 1990. However, before then, Ivan Sutherland had already created a headmounted display in 1968. This headset was put onto the wearer‟s head to function as a digital window into a virtual world. Limited computer processing power was available at the time when this was done (Lee, 2012). One of the first functioning AR systems known as “Virtual Fixtures” at the U.S. Air Force Research Laboratory was developed in 1992 (Rosenberg, 1992). AR was first used to identify space debris with “Rockwell Worldview” by overlaying satellite geographic trajectories on live telescope video (Abernathy et al., 1993). A software program named LandForm was successfully tested in 1999, and was a video map shot from a helicopter at Army Yuma Proving Ground in the US. It had video overlays with runways, roads, road names and taxiways (Delgado, 2000). Another big step in the development and improvement of AR application took place in 2004 when Trimble Navigation and the Human Interface Technology Laboratory (HITL) demonstrated a “highly accurate outdoor helmet-mounted AR system”. This technology from New Zealand demonstrated to the public how the real world could be digitally augmented. Other AR inventions included the Wikitude AR Travel Guide launched in 2008. This was done with the G1 Android phone in Austria, as well as the ARToolkit which was ported to Adobe Flash by Saqoosha in 2009. This was a definite turning point in the evolution of AR (Cameron, 2010). Today some of the latest AR technologies available include Google Glass (open beta test) and Microsoft HoloLens (augmented reality headset) (Microsoft, 2015). These were launched in 2013 and 2015 respectively. Google Glass connects to the Internet through Bluetooth to the user‟s cellphone.
This is where the information comes from that is displayed on augmented reality
glasses. The Microsoft HoloLens headset represents a great leap in AR technology as it provides endless potential for AR applications. This headset utilises various sensors along with a processing unit to blend high-definition holograms with the real-world environment observed by the wearer (Miller, 2013). 2
1.3
Project Background
AR is new technology with potential which still needs to be explored. The digital augmentation of reality, along with interaction with holographic displays that can be achieved with this technology, have countless implications for the way we view, and use, technology. AR has already found many applications in marketing, such as the Augmented Reality Catalogue from IKEA. This catalogue, along with a certain app (apps are computer-based applications which have been designed to run on smartphones, tablet computers and other mobile devices) that users can download on their smartphones or tablet PCs, allows them to see what IKEA‟s furniture would look like in the space they have in mind for it. Through the use of the camera on one of these devices, the selected furniture can be digitally augmented over the real-world view seen on the screen via the camera. It then appears on the screen as if the piece of furniture were in fact situated in the position being viewed (Andrews, 2013). Augmentation of an environment can be found where AR was applied as a digital, visual aid for navigation. An experimental landing assistant was developed in the 1990s for helicopters (also similarly for other aircraft), to aid pilots to execute safer landings. This system used AR to display beacons for the pilot. Figure1.3a shows the beacons that were displayed on a head-up display (more information regarding the associated AR hardware will be provided later). The aircraft could then be navigated towards the augmented beacons to approach the landing zone or runway at the best altitude and angle (Rougeau, 2014). Other systems include iOnRoad, which uses a smartphone app to warn drivers of potential collisions and the AR glasses developed by Mini. The glasses allow drivers of Mini vehicles to see augmented information such as navigational aids, or view their mileage or current speed.
Figure 1.3a: LandForm, an experimental landing assistant system for helicopters (futurescienceleaders.com, 2014)
3
Another area where AR has found multiple applications is in the maintenance of vehicles. Large motor manufacturers such as BMW and Audi have developed AR systems that aid users visually (even with or without additional audio assistance) through AR-based displays. These systems function to assist in and also improve the work of mechanics who need to perform repairs or maintenance on vehicles. These systems generally use object recognition to identify the component that needs to be replaced. The user is then shown visually how to perform the removal process and how to assemble the parts, as shown in Figure 1.3.b. Additional information such as which way to turn the nut and the appropriate tension required can also be displayed.
Figure 1.3b: BMW Augmented Reality Maintenance (imponderablethings.com, 2013) In Germany the coal mining company RAG Mining Solutions is already successfully using this AR application for maintenance purposes. Other AR mining applications have also been found, such as SHEQSCAN that was developed by IHPC. SHEQSCAN is an integrated audit and inspection tool which simplifies regulatory governance and compliance from a facilities management perspective by using AR technology. Finally, another big AR mining application is the PerfectDig AR-based software from Maptek. PerfectDig is able to provide the user with augmented visuals in order to display the correlation between the highwall being viewed (in open cast mining operations) and the planning data of this highwall. Differences between the plan and the actual can then be displayed (not in real time) with AR in order to monitor the progress of the mining operation. In identifying potential applications of AR and the benefits it may have in future, it was shown that the technology has advanced extensively in recent years. Several applications have already been found in various fields. However, compared to other industries, it was found that the mining sector 4
was lagging behind in finding suitable uses for this technology. It is possible that some of the existing AR applications could be applied directly, or be modified to be applied to the mining industry. New ideas may further emerge for mining-based applications when an understanding is developed of existing applications in other non-mining environments. The purpose of the literature search was therefore to investigate the potential applications for AR in mining and also in some way prove that its application and use in the mining industry has been lacking. AR-related research is important for technology and mining companies alike. Companies that provide a mining-related AR technology product or service, will open up a new and expandable market. In this way mining companies will then also gain access to technology that adds value. The main objective of the study was to explore the potential of AR technology applications in the mining industry and to what extent it could enhance efficiency in mining operations. Technology is constantly developing, and AR is a new phase of technology application in the mining industry. It is important that major industry sectors, such as mining, conduct thorough research on how AR can be applied and exploited.
1.4
Problem Statement
The purpose of this investigation was to critically look at the potential of using AR technology in the mining industry and to what extent it would improve the efficiency of the mining process. It was therefore important to evaluate various current AR applications in other industries and explore their potential adaptation and use in the mining industry.
1.5
Scope of the Study
It should be noted that this study does not consider the technical aspects involved in the functionality of AR in detail (nor the shortcomings associated or possibly associated with the current development of this technology). The study focused purely on what potential applications and gains could be found for the technology under the assumption that it is, or will be, a suitable technology for use in the mining industry. The efficiency of the potential applications resulting from this study will also not be measured in practice. It is assumed that AR can be implemented to fulfil the applications as described, without any technical constraints. The functionality and limitations of the technology itself are therefore not considered.
5
1.6
Objectives and Methodology
Table 1.6: Objectives and Methodology Objective
Methodology
Establish a thorough, general understanding of
A literature study on AR was completed in order to:
Augmented Reality.
-Develop a detailed understanding of AR (including what it entails). -Determine what technology and equipment are required to implement AR.
Determine what uses have been found for AR in
The literature study further investigated what
different environments and settings, as well as how
current AR applications can be found in different
it has been applied thereto. Then investigate other
environments. Other potential future applications
potential
and technological ideas that are available, or in
applications
of
AR
in
various
environments.
development, were also identified.
Determine what applications of AR are currently
The literature study included the applications of AR
available in mining.
that are currently available in the mining sector.
Determine how current AR applications from other
It was determined how current AR applications
environments can be applied to (or be modified to
from other environments can be applied to (or be
be applied to) the mining sector.
modified to be applied to) the mining sector at the end of each research section on the identified applications.
Investigate what potential new AR applications can
Further
be found – with the specific focus on the mining
conducted in order to identify other ways in which
sector.
AR could potentially be utilised in mining.
Develop challenges
an
expectation
of
what
the implementation of
independent
idea
generation
was
potential
A SWOT analysis was done on the identified
the above
potential applications that are most likely to be
applications in mining could be faced with. In
implemented at this point in time.
addition, consider what the possible positive and negative impacts associated with AR applications could be.
6
REFERENCES Abernathy, M. et al. 1993. Debris Correlation Using the Rockwell WorldView System, Maui: Air Force Maui. Andrews, K. 2013. Ikea launches augmented reality catalogue - Dezeen. [ONLINE] Available at: http://www.dezeen.com/2013/08/05/ikea-launches-augmented-reality-catalogue/
[Accessed
21
Available
at:
April 2015]. Augmentedrealityon.
2012. Augmented
Reality
On.
[ONLINE]
http://www.augmentedrealityon.com/. [Accessed 11 April 2015]. Cameron, C. 2010. Readwrite – Flash-based AR gets High-Quality Markerless Upgrade. [ONLINE] Available
at:
http://readwrite.com/2010/07/09/flash-based_ar_gets_high-
quality_markerless_upgrade. [Accessed 12 April 2015]. Chen, B. X. 2009. If You're Not Seeing Data, You're Not Seeing | WIRED. [ONLINE] Available at: http://www.wired.com/2009/08/augmented-reality/. [Accessed 12 April 2015]. Claire, M. 20 February 2013. New York Times. Delgado, F. et al. 2000. Virtual Cockpit Window for the X-38, SPIE Enhanced and Synthetic Vision 2000, Orlando Florida. Proceedings of the SPIE, vol.4023, pp. 63-70. Driscoll, S. 2013. Imponderable Things (Scott Driscoll's Blog): Comparison of Augmented Reality Glasses,
Google
Glass,
Meta,
castAR.
[ONLINE]
Available
at:
http://www.imponderablethings.com/2013/09/minority-report-and-terminator-vision.html. [Accessed 7 June 2015]. Graham, M. et al. 2012. Augmented reality in urban places: contested content and the duplicity of code. Transactions of the Institute of British Geographers, vol.38, no.3, pp. 464-479. hitlabnz.
2008. Outdoor
AR
-
YouTube.
[ONLINE]
Available
at: https://www.youtube.com/watch?v=jL3C-OVQKWU. [Accessed 10 April 2015]. Lee, K. 2012. Augmented Reality in Education and Training. Colorado: University of Northern Colorado. Maxwell, K. 2010. Meaning of Augmented Reality, BuzzWord from Macmillan Dictionary. [ONLINE] Available
at:
http://www.macmillandictionary.com/buzzword/entries/augmented-reality.html.
[Accessed 12 April 2015]. Microsoft. 2015. Microsoft HoloLens - Transform your world with holograms - YouTube. [ONLINE] Available at: https://www.youtube.com/watch?v=aThCr0PsyuA. [Accessed 12 April 2015].
7
Rey, P. J. 2011. Virtual, Mediated, and Augmented Reality » Cyborgology. [ONLINE] Available at: http://thesocietypages.org/cyborgology/2011/03/10/virtual-mediated-and-augmented-reality/. [Accessed 7 June 2015]. Rosenberg, L. B. 1992. The Use of Virtual Fixtures as Perceptual Overlays to Enhance Operator Performance in Remote Environments. Technical Report AL-TR-0089. USAF Armstrong Laboratory. Rougeau,
J.
2014. Augmented
reality.
What
is
it?
[ONLINE]
Available
at:
http://www.futurescienceleaders.com/jmaster/2014/09/augmented-reality-what-is-it/. [Accessed 7 June 2015]. Steuer, J. 1993. Defining Virtual Reality: Dimensions Determining Telepresence. Stanford: Stanford University.
8
CHAPTER 2: LITERATURE SURVEY 2.1
Introduction
It should be noted that this study has a more anthropocentric approach, in that less focus will be on the technological characteristics of Augmented Reality (AR) and a greater emphasis will be placed on the effect AR has on the user. This includes the applications and associated benefits of the technology. AR already has various applications to date. Some of the most extensive include military, industrial and medical applications, along with many other applications in the commercial and entertainment environments. In order to understand how AR applications can be applied to another environment, such as the mining sector, a general understanding of the technological functionality of AR is required along with examples of existing applications. Therefore an overview of the technology involved in utilising AR will be provided before different applications in numerous environments are investigated.
2.2
Technology
AR encompasses systems capable of combining the real world with virtual or digitised aspects. It then provides a means to interact with this augmented environment created by combining these two elements, and lastly it offers 3D registration (process of finding a spatial transformation that aligns two point sets) – the latter, however, is commonly accepted as not a requirement for an AR system to exist. In this sense, AR functions to enable a user to carry out sensorimotor and cognitive activities in a mixed space, which is a combination of the real and an artificially created environment (Hugues, 2013). In order to achieve this, a number of technological devices or pieces of equipment are utilised. These primarily include sensory devices that are able to stimulate the user‟s senses (such as helmets with head-mounted displays (HMD), glasses equipped with augmentation features, or speakers, etc.) together with motorised or computerised devices (such as a video camera, location detection system, computer processing system, etc.) that are able to detect and process either the environment, the user or both. 2.2.1
Hardware
The required hardware components for AR include a sensor, a processor, a display and input devices. Mobile computing devices such as smartphones and tablet computers contain these 9
hardware requirements, which is why they are currently so popular with the use of AR-based apps. These types of
devices are further made
into
suitable platforms
as they contain
microelectromechanical systems or MEMS (the technology of very small devices), sensors such as accelerometers (devices that measure real acceleration), solid state compasses (small compasses usually built out of two or three magnetic field sensors that provide data for a microprocessor) and GPS. The MEMS sensors are essential in determining the movement, orientation and position of the device used to process and display AR (Metz, 2012). 2.2.2
Display Technology
Various different display technologies are used for AR rendering (the process of generating an image or holographic 3D model from a 2D or 3D model by means of computer programs). These include optical projection systems, monitors, such as computer screens, hand-held devices, such as smartphones or tablets and other display systems worn on one‟s person, such as glasses designed to project AR technology. Some of the current and in-development display systems will be investigated. Contact Lenses A team of researchers at the University of Washington, led by Professor Babak Parviz, have been working on developing contact lenses that will be able to display AR. The team has been investigating whether these bionic contact lenses could be integrated with light-emitting diodes (LEDs) and circuitry for miniature AR displays, as well as antennas for wireless communication with other devices, such as the wearer‟s cellphone (Greenemeier, 2011). A contact lens has already been developed by Parviz and his team which, when worn, is able to display a single pixel to the wearer. They also stated that their plan was to eventually have a fullyfledged display with reasonable resolution and colour that can receive images from an external device and superimpose those images over what you would normally see (Greenemeier, 2011). Similarly, Belgian researchers have also been working on developing a spherical, curved LCD display. The display is being designed to fit onto contact lenses and cover an entire lens‟ worth of text, images and other visuals (Rosen, 2012). Another contact lens version in development for the U.S. Military is being designed to function with AR spectacles. The purpose is to retain wide-angle vision of the background while viewing AR displays. This will be achieved by having two different focusing lenses to give humans the ability to focus on the near foreground and far background. One lens focuses the foreground light into the
10
middle of the pupil, while the other lens focuses the background light on the edge of the pupil (Anthony, 2012). At the Augmented World Expo Conference in 2013, a futuristic video named “Sight” won the best futuristic augmented reality video award. The video showed benefits of having AR through contact lenses in everyday life. Eyeglasses AR displays can be rendered on devices that resemble normal eyeglasses. Different versions of eyewear include devices that employ cameras to intercept the real-world view and overlay the augmented view through the eye pieces. These devices have an augmented view which is projected through, or reflected off, the surfaces of the lenses themselves. This can either be accomplished by small projectors projecting on the lenses, or projecting directly onto the retina of the eye. The “Wrap 920AR” from Vusix shown in Figure 2.2.2a is an AR eyewear device that uses cameras. It is heavier than a regular pair of glasses but far lighter than a head-mounted display. In the device the direction of the wearer‟s visual attention is tracked through hardware such as gyro sensors and magnetometers. These glasses are equipped with ports that allow users to connect them to mobile smartphones for portable power and controls (such as the selection and loading of a particular AR object or environment for display on the eyewear). In addition to the abovementioned,
Vuzix
software
has
the
ability
to
recognise
and
track
visual
markers/identifications (Grifantini, 2010).
Figure 2.2.2a: Augmented Reality glasses, Wrap 920AR, from Vuzix (technologyreview.com, 2015) Arguably the most well-known AR eyewear (also categorised as a HUD (head–up display) display) is shown in Figure 2.2.2b. “Google Glass”, also known as “Project Glass”, is the invention of the 11
multinational technology company, Google Inc. The prototype of Google Glass illustrates the wearer‟s ability to control music, obtain directions, take photographs, issue voice commands and even conduct video-based conversations. The glasses, however, do not cover the wearer‟s entire eye. As a result, the wearer can be observed looking at the miniature screen, which is located on the face of the frame (Gannes, 2012).
Figure 2.2.2b: Google Glass (allthingsd.com, 2015) A Cambridge-based company in the United Kingdom have developed their own prototype AR eyewear device, shown in Figure 2.2.2c. The device from The Technology Partnership (a dedicated technology development company) is able to seamlessly project information into the scene or landscape in front of the viewer. These “glasses” incorporate a tiny projector in one arm of the spectacles. This projector reflects imagery onto the centre of the lenses which are, in turn, etched with a reflective pattern. The lenses are then able to beam the image directly into the viewer‟s eye. This means that the image can be incorporated directly into the wearer‟s frontal view. With Google Glass on the other hand, the wearer is required to look up or down to see the screen. In addition to the above, this imagery is entirely invisible to anyone observing the wearer of Google Glass (Arthur, 2012).
12
Figure 2.2.2c: The Technology Partnership: "Heads-up" prototype AR glasses (theguardian.com, 2015)
Handheld Displays Handheld AR displays are the first commercial success for AR-related technologies thus far. This is evident from the multiple, and increasing, amount of AR application software available in the market today. The main advantages of handheld AR displays lie in both their portability and the ubiquitous nature of mobile phone cameras. The disadvantages relate to the physical constraints associated with these devices. This is due to users being required to hold the devices out in front of them when attempting to generate an augmented view. Another disadvantage is the distorting effect of normal wide-angle cellphone cameras, as compared to the real-world view through one‟s own eyes (Sung, 2011). Currently there are three distinct classes of commercially available handheld devices for AR display, namely smartphones, PDAs (Personal Digital Assistant) and tablet PCs. The designs of these devices have specific trade-offs between size, weight, cost and computing power. Whilst smartphones are more portable and widespread in use, they currently lack adequate network connectivity, sufficient camera quality and the required processing power (when compared to AR‟s often arduous requirements). This renders these devices suboptimal for the purposes of rich and meaningful AR applications. Furthermore, their small screen display sizes and limited data input capabilities are less than ideal for the purposes of 3D user interfaces. Although tablet PCs do not share the aforementioned drawbacks to the same extent, they are much more expensive and often considered too heavy for single-handed (or even prolonged twohanded) use. PDAs fall somewhere between smartphones and tablet PCs in terms of their size, weight, processing power and price. Despite this fact, they have less of a ubiquitous social nature 13
than modern-day smartphones and are, as a result, not as widely used by the average consumer (Wagner & Schmalstieg, 2006). Head-Mounted Display (HMD) An HMD is a display device that is paired with a headset in one form or another (helmet, harness, etc.). HMDs provide an enclosed view of augmented objects overlaid onto the user‟s entire field of view. Modern HMDs often employ sensors for six degrees of freedom (refers to the freedom of movement of a rigid body in a 3D space) monitoring. This allows the system to align virtual information to the physical world and then adjust according to the wearer‟s head movement. HMD devices are capable of providing a wide view that can be augmented along with potential high resolution and crisp, clear displays. This trait allows these devices to provide users with immersive, mobile and collaborative AR experiences (Rolland et al., 2005). An example of the abovementioned device is the Z800 Pro AR HMD shown in Figure 2.2.2d. This HMD provides realistic, richly coloured 3D effects at a variety working distances. With this device designers, publishers and engineers can view multiple drawings and renderings as if they were laid out on the table in front of them (Vrealities, 2013).
Figure 2.2.2d: Z800 Pro AR HMD (vrealities.com, 2013) Another example of an HMD is the “Oculus Rift”, which is a bulky device capable of immersing the viewer into a full virtual reality. In this case, however, the real world cannot be seen and as a result the two are not intertwined as with AR. The distinctions between an HMD and an HUD are generally foggy; however, HMDs tend to be bigger, heavier and generally allow a larger span of view to be augmented.
14
Head-Up Display (HUD) HUDs are devices that are cable of augmenting only part of a wearer‟s field of view. Most eyewear devices fall under this category, whether in the form of eyeglasses or devices that look like hightech monocles. Google Glass, discussed earlier, is an example of a HUD device (Hill, 2013). An AR display device that currently seems to be top of its class in the established market, is Microsoft‟s HoloLens, shown in Figure 2.2.2e. This headset recognises the wearer‟s vocal communication, eye movement and hand gestures to help facilitate interaction between the virtual and the real world. The HoloLens projects imagery in mid-air and onto surrounding objects, which then appear to the wearer as 3D holograms. The wearer is then able to interact with these holograms and manipulate them depending on the type of application the augmented hologram serves. Examples include creating a 3D model and resizing or restructuring it, or creating drawings on top of real-world objects for illustration purposes (Sheridan, 2015). Brian Blau, director of consumer technology at Gartner, stated: "Out of all the head-mounted displays that I've tried in the past couple of decades, the HoloLens was the best in its class. There's a lot of promise for this kind of technology” (Sheridan, 2015).
Figure 2.2.2e: Microsoft HoloLens (informationweek.com, 2015)
Spatial Augmented Reality (SAR) SAR displays are capable of augmenting real-world objects and scenes by using digital projectors to display imagery and related information onto physical objects. Thus this type of display can be perceived with the naked eye and does not require additional hardware such as monitors, HMDs 15
or handheld devices. The key difference in this category of display relates to the fact that the display system is fully separated from the user (therefore not attached to, or worn by, the person perceiving the augmented view). Due to this separation, SAR scales naturally with groups of users and thus allows collocated collaboration between users (Bimber et al. 2001). Examples of the abovementioned include table and wall projections. One innovation is the “Extended Virtual Table”. This device separates virtual and real-world elements by including adjustable angled, beam-splitter mirrors (optical devices that split a beam of light in two) attached to the ceiling. Virtual showcases that employ beam-splitter mirrors in conjunction with multiple graphics displays provide an interactive means of simultaneously engaging with virtual and real elements (Bimber, et al. 2001). A research team at Cambridge University, UK, developed a system which augments an ordinary tabletop map with additional information, by projecting it onto the map‟s surface. The system utilises an overhead camera along with image recognition software to identify a region of the map‟s topographical features. An overhead SAR display projector then overlays relevant information onto the physical map, as shown in Figure 2.2.2f. Possible overlays include features such as traffic accidents, flooded areas and moving helicopters (Knight, 2005).
Figure 2.2.2f: Augmented tabletop map (newscientist.com, 2005)
Virtual Retinal Display (VRD) An entirely different approach from the displays described above is a technology known as virtual retinal display (VRD). VRD is a personal display device that has been under development at the University of Washington‟s Human Interface Technology Laboratory for AR application (Slideshare, 2014). 16
The technology aims to scan a display directly onto the viewer‟s retina. The viewer will then see what would appear to be a conventional display that floats in space in front of his/her view (Anthony, 2014). A prototype Virtual Reality Device (VRD) from Avegant is shown in Figure 2.2.2g. This VRD offers extremely sharp definition of its visuals. Although the device has no screens, the brain interprets the signal as an 80-inch (203.2 cm) screen which is located 8 feet (243.8 cm) away (Endgadget.com, 2013).
Figure 2.2.2g: Avegant VRD prototype (engadget.com, 2013)
2.2.3
Input Devices
Input devices include cameras capable of object, facial or movement recognition, as well as, to a lesser extent, devices capable of speech recognition. Other input devices include systems that can interpret a user‟s body movements by visual detection or from sensors embedded in a peripheral device (such as a stylus pen, motion sensor, glove and other body wear). Systems exist that can translate spoken words into text or computer instructions. It is, however, important to note that these systems are not considered to be extremely accurate at this point in time (Branscombe, 2013). One such input device for motion detection is a wearable AR system prototype, developed at the Glasgow Caledonian University. This prototype is based on a “full-body-motion capture system” using low-power wireless sensors. The system uses body motion to implement a whole-body 17
gesture-driven interface in order to visualise, interact and manipulate VR objects in AR settings. Gestures are mapped to corresponding behaviours for VR objects. This includes, for example, controlling the volume of audio equipment, or displacing or manipulating a virtual object‟s metadata such as its structural information (Barrie et al., 2009). 2.2.4
Tracking
Mobile AR systems utilise one or more tracking technologies such as digital cameras and/or other optical sensors. These include GPS systems, gyroscopes, accelerometers, solid state compasses and radio-frequency identification. These offer different levels of accuracy and precision in determining a device‟s and thus the user‟s position and orientation. 2.2.5
Computer
A computer analyses the sensed visual data, orientation, movement and positioning data perceived by a computerised device (either a camera and/or the tracking devices mentioned above). It then synthesises and positions the augmentations on the display in relation to the perspective of the viewer. It can also be noted that computer systems are ever-improving in terms of performance capabilities and speed, and are constantly decreasing in size as technology progresses. Importance of the study The significance of the study of technology used for AR is that it provides an understanding of current and perceived limitations. These perceptions can then be viewed in relation to what the future might hold in terms of these constraints. Understanding what the hardware component requirements are for a technology is essential in order to determine applications. Often applications are limited by the type of hardware it would require, or by the limitations of the associated hardware itself. Multiple drives toward developing and refining AR-based technology were identified, which indicates positive movement for the technology as a whole. Numerous possibilities of futuristic technology have been identified that are in development, all of which could bring major changes to a diverse field of technology, not limited to AR alone. With this in mind, the applications of AR that are currently in use, as well as futuristic ideas for potential applications, will be considered.
2.3
Applications of Augmented Reality
In order to investigate potential applications of AR in mining, it is imperative to comprehend how the technology is being and can be applied. The following section will therefore describe current as 18
well as potential AR applications in various fields. From these applications, ideas could then be generated for mining purposes. 2.3.1
Archaeology
AR can aid in archaeological research through the augmentation of archaeological features onto the modern landscape. This can aid archaeologists to form conclusions on the placement and configurations of an archaeological site. Traditionally, explorations of perception using GIS (Geographic Information System) were based on vision and analysis, which was carried out in a computer laboratory (not on site). Phenomenological analysis of archaeological landscapes, on the other hand, are normally carried out on the landscape itself. Computer analyses afterwards, away from the landscape in question, have often been a hindrance leading to difficulties in reaching conclusions. With AR however, the landscape can be merged with virtual elements, such as 3D models, that will bring aspects of the GIS analysis to the site itself. This leads to better visualisation that aid in reaching conclusions (Eve, 2012). Another application includes the possibility of rebuilding ruins, buildings, towns or landscapes to the state in which they had once been in the past through the use of AR. The Ludwig Boltzman Institute, together with several international partner organisations, is working to create a network of archaeological scientists. The aim of these programmes is the development of large-scale, efficient, non-invasive technologies. The technologies will be used for the discovery (using groundpenetrating radar), documentation, visualisation (using AR as a means to recreate and display the discoveries in the state in which they had been) and interpretation of Europe's archaeological heritage. This international team of archaeologists received worldwide attention after the recent discovery of the school of gladiators at Roman Carnuntum (Austria) and the work they did to digitally recreate this unique find (Neubauer, 2015). Significance of the available information The significance of the information relating to AR applications is that it spurs ideas on potential similar applications of AR for mining purposes. Apart from creating an augmented view of a historical site, plans for future site developments for mines could similarly be augmented for display purposes. Similarly AR can be used for historical mining site recreational purposes or to enhance mining museum experiences. Accidents, incidents or other historical events that need to be investigated (such as events that led up to a fatality) can also be recreated on the site of the incident itself. This will aid investigators to draw more concise conclusions and will be explored further in this investigation.
19
2.3.2
Architecture
AR is capable of aiding the visualisation of building projects and planning. Models and plans can be computer generated and superimposed onto a real-life local view of the property. This can be done before construction has even begun, as was demonstrated by Trimble Navigation in 2004 (Hitlabnz, 2004). The Faculty of Architecture of the University of Western Australia has been working on harnessing the new digital technology of AR. Their aim is to accomplish achievements such as allowing people to walk through a cathedral that exists only in virtual space, or to be able to fully inspect a 3D home display, all on one‟s smart phone or digital tablet. "Visitors with smart phones or tablets will be capable of interacting with 3D, augmented reality buildings. By downloading a free Aurasma Lite app on their handheld devices, they can view the virtual displays from every angle." (Van Meeuwen, 2012). In this way, architectural sight-seeing could be enhanced by AR applications which allow users viewing the building‟s exterior to virtually see through its walls. The interior objects and layout of the building itself can then be viewed (including general information relating to the aforementioned, e.g. estimated structural age, etc.) (Meeuwen, 2012). AR can further be employed in the architects‟ workspace by rendering animated 3D visualisations into their view during the inspection of 2D drawings. Matsuda (2010) is of the opinion that the restrictive nature of specialised professions will become less so as new technologies develop. This could potentially lead to fusions within separate professions to incorporate a more technologically based approach. “AR is spatial and experiential, but also temporal and interactive, so the future discipline of „information architecture‟ is probably a new hybrid somewhere between architecture, film-making and game design.” (Divecha, 2010). Significance of the available information The significance of this information is that it holds potential for enhancing the work of the planning phase prior to mine construction, and also for the planning departments of mines during the operation phase. The use of AR 3D visualisations can aid understanding and communication, and increase efficiency in planning of all scales or magnitudes. AR illustrations of mine planning and rehabilitation measures can also greatly aid companies during the process of applying for mining licences. The approval process of exploration or mining licences can benefit from clear and
20
concise illustrations of how a company intends to use such an approval, as well as function as a regulatory aid in the form of a 3D, augmented, solidified commitment. In conclusion, the statements by Matsuda (2010) might hold true for numerous other professions that incorporate technology in their day-to-day undertakings, in that professions could become more and more dependent on the use and application of different technologies. Therefore professions such as engineering need to apply themselves to fully utilising and understanding technology as far as is reasonably practical. This highlights the need to understand AR and seek out ways to maximise benefits that could be gained from using it. 2.3.3
Art Interventions
AR has been applied to allow disabled individuals to create art by using eye-tracking technology (this is a process of measuring the point of gaze or the motion of the eye relative to the head). The technology is able to translate a user‟s eye movements directly into drawings on a computer screen in real time. The EyeWriter uses low-cost eye-tracking glasses along with open source software (software that can be freely used, changed and shared), to allow people who suffer from a neuromuscular syndrome to do just that (Webley, 2010). Items such as commemorative coins can be designed so that additional objects and layers of information are displayed when scanned with an AR-enabled device. This is demonstrated by the Royal Dutch Mint with the new 5 florin silver collector‟s coin, which displays information that can only be seen when viewed with a mobile app called Layer (Alexander, 2012). L‟Oréal used technology from CrowdOptic Inc. to allow users at the annual Luminato Festival in Toronto to view a virtual gallery while enjoying the sights of art displays in the streets. In doing so, the users created a giant, digital version of the Lancôme rose as they were guided along certain paths. CrowdOptic tracked the users via GPS, who had downloaded the application for the festival in order to view augmented information and art displays (Wadhwa, 2013). Significance of the available information The significance of this information is that it illustrates how current technologies can be utilised, such as how eye tracking is developing and how it can be applied in connection with computer systems to create drawings or text on a screen. This has potential use for equipment operators who wish to interact with a computer system during operation. Uses could include bringing up augmented displays or navigational aids. This can be done without unnecessary hand movements during non-ideal or risky conditions. Further into the future this type of technology might also be
21
applied to signal for help during distress situations where a person‟s body is unable to function after an incident. The coin that is able to provide augmented information indicates that small objects can also be used for an AR platform. A potential application is viewing a person‟s ID card that will augment additional information. The information displayed will then depend on the relationship and hierarchy of authority difference between the ID card holder and the viewer. Lastly, tracking of a device, such as an ID card, could be mapped to indicate where workers had been during working hours for control purposes. 2.3.4
Beauty
L‟Oréal has created a mobile application named Makeup Genius, which utilises the device‟s frontal camera to allow users to see how a variety of make-up products would look on them. The application augments different beauty products (such as blushers, eyeliners, lipsticks, false eyelashes, etc.) directly onto the user‟s face in real time, even if the person is moving. This allows products to be inspected digitally on oneself without the need to leave the comfort of your own home. This saves both time and money (Tanase, 2015). Significance of the available information The above information goes to show how precise some of the latest commercially available technology is. Accurate real-time augmented overlays can be achieved with normal smartphone cameras. Similar to this, another application could be to augment what certain pieces of mining PPE would look like on one‟s face prior to placing orders. 2.3.5
Commerce (Sales and Marketing)
Product preview and demonstration can be enhanced in several ways through the use of AR. As a basic example, AR would allow a customer to inspect the inside of a product package without having to break packaging seals. Images or products that are scanned with a smart device can provide augmented views of additional content or features of a product. This would include customisation and personalisation configurations, as well as tutorials on how to use the product and illustrative examples displaying the product in use. In this way AR can aid in product selections at manufacturers, stores or even from paper and digital catalogues. One such example is the 2014 IKEA Augmented Reality Catalogue (Andrews, 2013). This catalogue allows the user to view IKEA‟s furniture in his or her own home by using a smart device with a camera and the associated application (specified by the catalogue to be downloaded). The user can then use the catalogue as a reference point to augment a chosen item 22
into position over the catalogue‟s location. This occurs when the space above and around the catalogue is viewed through a tablet or smart phone (Andrews, 2013). Augmented furniture displays are shown in Figure 2.3.5a.
Figure 2.3.5a: Augmented furniture (dezeen.com, 2013) Printed marketing material can also be designed with certain “trigger” images which, when scanned by an AR-enabled smart device with image recognition capabilities, activate a video version of the promotional material (Kats, 2012). AR is used to integrate print and video marketing in this way. The fundamental difference between AR and normal image recognition in this sense is that AR makes it possible to overlay multiple layers of media on the screen simultaneously. This can include in-page video or audio, 3D object and even social media „share buttons‟. Figure 2.3.5b shows the display of the interactive content done with the digital advertising services of Telefónica (Meyer, 2012).
23
Figure 2.3.5b: AR used for marketing to display additional content (Gigaom.com, 2012) Significance of the available information Similarly, products and materials required for mining can be visualised on a mining site. Direct, visual communication with suppliers allows for better customisations and more accurate orders pertaining to each supply need. This can reduce the risk of wrongful orders and wrong or suboptimum parts being sent to the mine. This application will most likely save on unnecessary logistics costs and time. 2.3.6
Construction
Businesses (and possibly also municipal services in the future) are able to use AR to visualise georeferenced models of construction sites, underground structures, pipes and cables. This can be done through the use of smart mobile devices that are also able to utilise GPS systems. A technician who went to Augview (an augmented technology company located in New Zealand) to connect the company‟s phones, was able to locate the connection point underground inside multiple access pits. The technician was equipped with a smart device enabled with AR technology which enabled him to identify the exact location of the underground services he required in less than a minute, a process that could take up to 30 minutes or even longer to complete under normal circumstances (Churcher, 2013). After the 2011 Christchurch earthquake in New Zealand, the University of Canterbury released an AR mobile app named CityViewAR. This app allowed city planners and engineers to visualise
24
buildings that had been destroyed in the earthquake. It also provided planners with tools to reference the previous cityscape (Lee et al., 2012). Significance of the available information AR can be applied in the same manner for mine constructions as well as during progressive mining. Underground visualisations of entire mining sections could also be possible while standing on the surface and looking down with an AR-enabled display device. This could aid new employees or contractors to visualise the mine layout and gain a quicker grasp of the problem area that needs to be addressed. The layout of this area, and how to get there when venturing underground, could also be understood and planned more easily. Furthermore, the escape routes that are in place could also be clearly visualised in this way. Lastly, augmented reconstruction of areas where FoGs occurred prior to the incident, could also be created. This will help to reference previous layouts for reconstruction of damaged sites. 2.3.7
Education
A standard educational curriculum could be greatly supplemented through the use of AR. Text, graphics, video and audio (sensory aids) could be superimposed on students‟ real-time environment to help improve their knowledge and understanding of a variety of concepts. The visualisation of concepts provided by AR can benefit students greatly by enabling them to form an accurate perspective and understanding of a problem. It also largely eliminates problems involved with allowing students to form their own perceptions and understanding of various concepts when faced with a lack of sensory aids. Logical Choice Technologies (an expert in educational technology) has released AR-based software known as Letters Alive. This is a supplemental reading curriculum that uses AR to “bring letters to life” in the form of 3D interactive animals. These animals are easily understood and enjoyed by younger children. By using this program, students more easily master the building blocks of literacy: phonemic awareness, phonics, concepts of print and comprehension. Sentences are built by using an animal card (3D image of a talking, interactive animal) and “three sight word cards” that demonstrate the concepts of punctuation and print. The AR technology also prompts the animal to react to, and answer, the questions and statements that students form through the use of these cards (Brown, 2011). Figure 2.3.7a shows the three sight word cards from Letters Alive. Similar to this, textbooks that support AR applications have been created in Japan by a company named Tokyo Shoseki. These textbooks form part of an English course known as New Horizon. The English course focuses predominantly on adults interested in learning English at high school 25
level. The AR technology imbedded in the course brings interactive learning to the student by having animated people, conversing in English, “pop up” at certain page sections. This occurs when the pages are viewed with the AR app through a smart phone or similar device (StewartSmith, 2012).
Figure 2.3.7a: Letters Alive teaching kids the building blocks of literacy (prweb.com, 2011) In this way students can participate interactively with computer-generated simulations. Another example is Construct3D, a “Studierstube” system, which was designed for mathematical and geometrical education. It forms an active learning process where students learn to study with technology whilst maximising the transfer of learning. The system is easy to understand and encourages experimentation with geometric constructions while also improving spatial skills (Kaufmann, 2000). AR can also assist students to understand chemistry by providing visual aids of spatial structures of molecules. The Technische Universität München, Germany, designed a system called “Augmented Chemical Reactions” to do just that. The virtual models displayed by the system exist in an augmented space and can be interacted with and controlled in an intuitive way. The models are rendered on a camera picture at the position of special markers held in the user‟s hands. The “intuitive controlling” of the position and orientation of the molecule is done by moving and rotating these markers in front of a camera. This system can also be applied to speed up the process of designing new molecules. Scientists are able to inspect created molecules and determine whether the molecules meet the special requirements for specific reactions (Maier et al., 2009). Students in physiology are also able to visualise different systems in the human body in 3D, as shown in Figures 2.3.7b and 2.3.7c.
26
Figure 2.3.7b: An augmented 3D display of a human heart with Anatomy 4D (qualcomm.com, 2015)
Figure 2.3.7c: An augmented 3D display of a human body with Anatomy 4D (qualcomm.com, 2015) 27
Figures 2.3.7b and 2.3.7c show the augmentations of an app known as Anatomy 4D (mobile software by Daqri, a leading AR software developer). The application allows students to learn anatomy quickly and accurately, anywhere and any time. In addition thereto, this application has been designed in such a way so as to make it useful to a wide variety of people. This would include anyone from the medical professional and students to the curious individual without any prior knowledge of the subject matter. The augmented 3D display empowers the user with an interactive 3D display that acts and feels like real space (Dempsey, 2015). According to the product marketing manager of Daqri, Gaia Dempsey (2015), "AR can help us learn much more quickly. It can help us spread and share information. It can help us be more productive together and work collaboratively.” Figure 2.3.7d shows the use of an app called iSkull.
Figure 2.3.7d: AR used to aid in the visualisation of biology (mahei.es, 2015) In Figure 2.3.7d the use of iSkull is demonstrated with a tablet. iSkull is an application that is used to aid teaching anatomy to students. As can be seen in this example, the application displays a hyper-realistic, augmented 3D skull. Students can mark sections of the augmented skull on the tablet and even use the application‟s “zoom in” functionality to view the fine lines and grooves inside the human skull (Mahei, 2015). Furthermore, AR technology also permits learning via remote collaboration. Instructors/teachers and students are able to share a common virtual learning environment without the need to be in the same physical location. The virtual environment is populated with virtual objects and learning 28
material which interact with one another within that setting. This technique is particularly useful when users are co-located and have a natural means of communication at their disposal (speech/voice transfer, physical gestures, etc.). It can also be mixed successfully with immersive VR or remote collaboration. A psychological factor of importance, however, is that some users might feel unsafe or tense if their view is “locked” in an immersive virtual (non-real) world. AR on the other hand allows users to “keep control” as they are able to see the real world around them while they still have augmented virtual elements that provide the required information or setting (Kaufmann, 2003). Significance of the available information The importance of this information is that it seems clear that AR is a significant teaching aid. This is due to the visual nature of the technology, which provides a means for students to grasp a concept much faster and with much more clarity. In this way it is further less likely for someone learning with the help of AR-based visuals to misunderstand concepts, layouts, designs or the way something functions. Based on this it could then be hypothesised that AR-based learning or training should reduce the risk of misunderstanding by students/trainees. It could also allow them to learn significantly faster. This will in turn lead to reduced costs and time consumption. The accurate visuals displayed in this section indicate further potential applications for on-the-spot guidance on how to apply CPR, first aid treatments or safety procedures when such scenarios arise. 2.3.8
Emergency Management/Search and Rescue
AR systems are used in public safety situations and, as a result, a drive to incorporate this technology in this area is currently being experienced. AR technology can aid in almost any safety situation, from flooding and super storms to chasing down a suspect at large. As an example, aircraft search and recovery, including camera-equipped UAV (Unmanned Aerial Vehicle) systems, can use AR technology in search-and-rescue operations. This can also be carried out in rugged terrain such as forests or mountains. An AR system is capable of providing aerial camera operators with a geographic awareness of the forest or mountain roads and locations blended with the camera footage. This aids the camera operator to search for a missing person as the operator is equipped with a better geographic context through the use of this camera footage. When the missing person has been found, the operator can more efficiently direct rescuers to the person‟s location (Cooper, 2007).
29
According to Crowe (2013), “technologies like augmented reality and the growing expectation of the public will continue to force professional emergency managers to radically shift when, where, and how technology is deployed before, during, and after disasters.” Significance of the available information This type of application can be used to aid rescue/proto teams in mine rescues and searches during emergencies. In underground situations no aerial view is available; however, the position tracking systems provided by AR can function similarly in order to determine the locations of those in dire need. Together with clear, augmented mine layout visualisations, emergency planning and rescue operations can be conducted in the most optimal and fastest way humanly possible. It is reasonable to hypothesise that in the future it would only be human capabilities that limit such (and many other) operations. This is because technology-based systems would be able to almost instantly calculate the best course of action to take. 2.3.9
Everyday
Since the 1970s and 1980s, Steve Mann has been developing technologies meant for everyday use. That is, “horizontal” applications, instead of specific applications or markets, i.e. “vertical” applications. Mann designed the EyeTap Digital Eye Glass, which is a general-purpose seeing aid that does dynamic-range management, HDR (High Dynamic Range) vision overlays, underlays, simultaneous augmentation and diminishment (Davies, 2012). According to Mann (2004), EyeTap can, through computer control, augment, diminish or otherwise alter a user‟s visual perception of the viewed environment. The diminishing factor can be used to diminish the electric arc while looking at a welding torch. To do this Mann created a so-called “quantigraphic camera” for his WeldCam HDRrchitecture helmet. The helmet is able to use the HDR photographic techniques to pick out the details the wearer most needs to see, instead of simply masking the bright light produced during the welding process. This allows the user to observe details that cannot be seen by the human eye or any existing commercial camera. New research into AR could make industrial work safer and improve visibility for those with partial or limited sight (Davies, 2012). Significance of the available information From the information above, this type of system can be applied on a mining site to aid highly vision-dependant tasks and maintenance and repair work. This system application could lead to better safety and higher quality work with fewer errors occurring. It could further aid visually impaired workers in many different types of tasks, from general office work to engraving gold 30
coins. The enhanced vision aid will also be able to help and retain invaluable older artisans, technicians, mechanics, specialists, etc., many of whom have irreplaceable knowledge and experience, but whose eyesight is fading. 2.3.10 Gaming AR enables gamers to experience digital play in a real-world environment. Players are immersed in a virtual environment through VR technology. However, AR is capable of blending the virtual aspects of a game into the real-world environment the player sees around him. An example of this is Minecraft Reality, in which the popular game Minecraft can be played through an AR device such as the Microsoft HoloLens. In this game you have the ability to create augmented Minecraft environments. These include different structures, buildings, environments or mining operations (to source the materials you need to create all of this), anywhere and at any time. Figure 2.3.10 illustrates how Minecraft Reality is overlain into a person‟s living room (Hudson, 2015).
Figure 2.3.10: Minecraft Reality illustration (rockpapershotgun.com, 2015) Many improvements in technology have been made which complement the application of AR as well as the efficiencies of using it. In gaming, the improvements in direct detection of a player‟s movements have led to numerous new ideas and uses of AR. An example is Wii, with games that detect the movements of a player and apply them in the game he/she is playing, such as a boxing match or tennis game (Day9, 2013). Another example of an application improvement in technology is eye tracking, which has become a widely used technology in gaming as well. This technology uses a camera that follows a person‟s gaze to detect where the user is looking on a screen. It can also detect head movements, 31
i.e. left, right, up or down, allowing players to manipulate their in-game avatar‟s (game character representing the player) angle of view and sight. TrackIR is an example of a system which can be implemented in games such as vehicle racing, flight simulators (piloting helicopters or old war-time aeroplanes, etc. within a game), first person shooter games (e.g. playing a soldier on a rescue mission or manning an anti-aircraft gun) and many more. Other newer games such as “Swarm!” combine this with AR technology to give players real immersion in the game while still remaining within their real environment. This eliminates the need to sit in front of a computer or console connected to a TV – the augmented game can be played anywhere the user finds himself in everyday life (AR23D, 2013). This kind of eye-tracking and movement detection technology, especially when combined with AR, can also be used to aid disabled people with communication. It can also aid flight simulations in the airline industry, and has numerous military applications. There are many examples of such applications. Significance of the available information The game “Minecraft” shows how well current technology can overlay augmented displays. This information provides ideas for similar systems that will have many applications for mine designs. This does not only have to be applied by highly specialised mine planning professionals, but it can also promote “playing” at home. This way some mine designers might broaden their perspective if they are inclined to have other mining-based influences in their lives outside of office hours. It could also spark interest in mining by younger generations and students. If they play a game such as Minecraft Reality, the number of new mining-based workers could be increased. As the workforce on mines is dwindling, this could greatly aid the mining sector in the future. The kind of eye-tracking and movement detection technology described in this section (especially combined with AR) can also be used to aid impaired people with communication or performing tasks. This could extend further to those who were injured during mining activities but still do other work on the mine. This type of technology finds application in flight simulations in the airline industry (not to mention numerous military applications), and it could also be applied to mining. Examples include training programmes, for example in how to use certain mining equipment and tools. Gesture recognition could be translated into instructions which could be relayed to machine operators or the machines themselves. An example would be the ability to shut down a piece of heavy machinery before it causes damage or injury by making a “stop” gesture with one‟s hand.
32
2.3.11 Industrial Design AR can help industrial designers to experience and test a product‟s design and operation before completion. This allows a good view of the product, and through proper visualisation certain flaws might be detected before it is completed. In the automotive industry crash tests are mandatory for all car models. AR can be used to compare calculated and actual crash test imagery of vehicles. It allows computer-generated simulations to be evaluated against the results of physically crashed cars. The augmented simulation results are displayed on a projection wall in stereo. In this way, AR offers a new way of evaluating simulation results. Development engineers are immediately able to recognise differences such as deviations from simulation results when they are congruently overlaid on the deformed components. In this way, faster and more qualified analyses can be realised with AR. This allows developers to cut the costs and time spent on various development processes (Noelle, 2002). The technologies and methodologies of assembly design and evaluation in the early design stages are of fundamental importance to product development. AR can be used to visualise and modify car body structure and engine layouts. It can further compare digital mock-ups of designs (of cars or other types of designs) with physical ones in order to find discrepancies between them (Pang, 2004). Although physical prototypes are considered the traditional method of product assembly, design and evaluation, virtual designs have great potential as a replacement method. This is due to the increased efficiency and reduced time requirements. Constructing a complex assembly environment with VR technology, however, requires a great deal of computation resources. It is often difficult to satisfy the requirements of a real-time simulation. AR, on the other hand, offers the concept of mixed prototyping, where parts of the design are available as physical prototypes and the remainder exist in a virtual form. With such an interface, it is possible to combine some of the benefits of both physical and virtual prototyping in a single project (Pang, 2004). Significance of the available information Crash tests can similarly be used for modelling of support to optimise designs, layouts and materials. The results from such AR-aided crash tests can further optimise mining method designs and layouts. All these tests can be done virtually, and therefore results can be obtained much faster and at a much lower cost than with prototypes.
33
2.3.12 Maintenance, Repair and Task Support BMW has developed AR glasses to assist their mechanics in performing maintenance activities on BMW vehicles. These glasses have the ability to read the wearer‟s field of view and indicate parts that need to be replaced. In addition thereto, other functions include the capability to identify screws or caps that require tightening. These AR glasses have the ability to indicate the required motions to tighten or undo such screws and caps. This is achieved via the use of virtual arrows or hands overlaid onto the real view of these parts (Dillow, 2009). “BMW Augmented Reality” techniques provide ideal support to the highly technical work on the complex innovations associated with BMW vehicles. The AR glasses have wireless access to the required information and are capable of the powerful computer processing required to aid mechanics in the workplace. Mechanics can receive 3D information on vehicle engines while they are in the process of repairing them. This aids them diagnose and correct faults. Apart from the real environment, the mechanic sees virtually animated/augmented components and the tools required to carry out the task. In addition to this, the glasses include an audio-based guidance system to allow the mechanic to hear instructions through the built-in headphones of the glasses (BMW, 2015). Another AR maintenance system example includes ARMAR (Augmented Reality for Maintenance and Repair). ARMAR was developed by Steve Feiner, a professor of Computer Science at Columbia University, and funded and partnered by the US Marine Corps. This system works with an HMD that has a virtual set of instructions built into the unit. The system functions as a guidance system for soldiers out in the field when performing repairs and/or maintenance on the LAV-25A1 armoured personnel carrier (Sung, 2011). A pre-mapped environment is stored on the computer using the ARMAR system. In order to overlay the virtual environment onto its real-world counterpart, one of two options are available. The first option is to detect the mechanic‟s head position and then to accurately track the movements of the HMD as the mechanic completes his work. The other option involves object recognition software, including adjusting the computer-generated components in real time as the work progresses (Sung, 2011). The effectiveness of this system had been tested. It was reported that even with trained mechanics, those who used the ARMAR system performed their tasks significantly better than those who did not use the system. The subjects‟ head movements were tracked to determine the efficiency of the system. It was noted that those using the ARMAR system used 37 per cent less head motions in comparison to those using PDF format instructions (Sung, 2011). 34
Other examples of similar maintenance-oriented AR applications include:
MARTA (Mobile Augmented Reality Technical Assistant), which is an accompanying mobile application for Volkswagen‟s XL1 model. The app provides instructions on the hybrid diesel‟s mechanical operations to assist the user to identify and fix problems (Recinos, 2013).
eKurtzinfo (German for “short info”) is an iPhone application for Audi‟s A1 and A3 models. The app is able to identify the vehicle parts that you choose to point your phone at and then offers relevant information relating to those particular parts in turn (it functions more as an AR user manual) (Rollenhagen, 2013).
Boeing, BMW and Volkswagen have also incorporated similar types of AR systems into their assembly lines to improve their manufacturing and assembly processes. “In 1992, Tom Caudell coined the term augmented reality when he was working at Boeing on a project to make it easier to assemble large bundles of electrical wire for aircrafts on the factory floor.” (King, 2009). Significance of the available information This study showed that numerous large, well-established companies with significant research funding capacities are not only researching AR applications, but are already using them. This lends credibility to the technology as a whole, as well as to its applications for maintenance work. The information in this section highlights clear potential applications for maintenance of equipment and vehicles on mines. The benefits this would bring, along with a great potential for increased efficiency, are substantial. AR can further be applied to improve other processes on mines, such as ore processing and assembly of equipment by enhancing communication and hastening the transfer of knowledge and instructions. On more basic levels AR can function as general task support, such as visually augmented guidance for trainee artisans or fitters. This could increase the rate of learning and lead to increased efficiency. 2.1.13 Medical AR can provide a surgeon with additional information such as blood pressure, heart rate or even the state of the patient‟s organs. This information can be overlaid onto a screen in real time or a video of the internal organs can be provided by a camera inside the patient. AR can also be applied to allow a doctor or surgeon to look inside a patient by combining one source of images with another, such as X-ray footage with video footage. Some examples include a virtual X-ray view based on prior tomography (imaging by sections or sectioning, through the use of any kind of penetrating wave). Furthermore, real-time ultrasound 35
images and confocal microscopy (optical technique for increasing optical resolution and contrast of a micrograph) probes could be used for optical biopsy mapping for minimally invasive cancer screening (Mountney, 2009). Other examples include the visualisation of the position of a tumour in the video of an endoscope (Camma Icube, 2014). Radiation exposure risks from X-ray imaging devices can be determined by overlaying 3D risk mapping scenes in an AR manner to give staff an accurate representation of radiation exposure (Rodes & Padoy, 2014). AR has also been used for cockroach phobia treatment. This was done based on exposure therapy through the use of augmented displays of cockroaches. This treatment method does not have the same high risk associated with real-world exposure treatment. Clinically phobic people often drop out of treatment due to their inability to face their greatest fears in the real world. Figure 2.3.13 shows how a patient can perceive cockroaches around his/her hands in an AR-created environment (Mims, 2010).
Figure 2.3.13: Treating cockroach phobia with AR (technologyreview.com, 2010)
36
Significance of the available information The scanning systems can potentially be expanded to detect underground geological features for more accurate AR displays of what the ground looks like. Designs can be done accordingly for optimal layouts in terms of safety, extraction and productivity. The cockroach example indicates that AR technology could have psychological impacts on users that may or may not yet be fully understood or predicted. 2.3.14 Military The military played a significant role in the early development of AR and continues to do so. A Chicago-based company known as Tanagram Partners has been developing military-grade AR technology. This technology is believed to have the potential to change the face of military combat as we know it (Cameron, 2010). In combat situations, AR can serve as a networked communication system that is able to render critical battlefield information onto a soldier's HMD in real time. The soldier can see, within his realtime viewpoint, various objects and people that can be marked with special indicators to warn of potential dangers. Virtual maps as well as 360° view camera imaging can also be rendered to aid a soldier's navigation and battlefield perspective. This can be transmitted to military leaders at a remote command centre for processing. In doing so, responses can be relayed back to the soldiers, regarding their surroundings, in the form of overlaid augmented information (Cameron, 2010). Another AR application occurred when Rockwell International created video map overlays of satellite and orbital debris tracks. This was overlaid on video footage from space surveillance telescopes. The map overlays indicated trajectories of various objects in geographic coordinates. This allowed telescope operators to identify satellites as well as to identify and catalogue potentially dangerous debris. The application aided in space observations at Air Force Maui Optical System (Abernathy, 1993). In 2003 the US Army started to integrate the SmartCam3D AR system into the Shadow Unmanned Aerial System. The purpose was to aid sensor operators using telescopic cameras to locate people or points of interest. The system combines geographic information (as well as street names, points of interest, airports and railroads) and information constructed from databases (such as terrain, cultural features, pre-mission plan, etc.) with live video from the dynamic camera system. This system offers a “picture-to-picture” mode that enables it to show a synthetic view of the area surrounding the camera‟s field of view. This helps solve the problem where the field of view is too narrow to include all important context (often described as “looking through a soda 37
straw”). The system further displays real-time friend/foe/neutral location markers that are blended with live video to provide the operator with better situational awareness (Calhoun, 2005). Researchers at USAF Research Laboratory found that the speed at which Unmanned Aerial Vehicle (UAV) sensor operators found points of interest when using this system increased approximately two-fold. The ability to maintain geographic awareness quantitatively enhances mission efficiency and is therefore a great asset due to AR technology. The system is in use in the US Army RQ-7 Shadow and the MQ-1C Gray Eagle Unmanned Aerial Systems (Calhoun, 2005). Significance of the available information This type of application can be applied in real-time instructions for, and adaptations to, planned rescues during crisis situations. In future, AR-based detection systems could potentially recognise dangerous and failing areas that could lead to collapse, and provide real-time aid in avoiding an incident.
2.3.15 Navigation Jon Fisher, CEO and co-founder of CrowdOptic, is one of many entrepreneurs advocating AR in a more mainstream setting. His start-up software is capable of detecting the direction in which a crowd of people have their phones pointed while taking photos or videos of objects or events. The software then allows users to invite others in the crowd to communicate and share content. The software uses the phone‟s accelerometer and gyro to calculate the user‟s position and line of sight. It then triangulates with other phones to determine exactly what everyone is looking at (Metz, 2012). This software has been used in several applications, amongst which one was used at a NASCAR race. During the race, fans who were unable to see the entire track could point their phones at distant turns. They could then receive photos and videos from other users at more convenient locations as the vehicles moved along the racetrack (Metz, 2012). Another company, iOnRoad, offers an AR collision-warning application for drivers with smartphones. The application uses the phone‟s camera stream, in combination with image processing software, to identify relevant objects (such as the lane the driver is in and the position of vehicles ahead). The phone‟s GPS is used to determine the vehicle‟s speed and the application measures the distance between the driver and the vehicle in front of him. The data is then used to calculate whether the driver is keeping a safe following distance and warns the user of a possible
38
collision if the gap closes at an unsafe rate. The app is also able to give a warning if the driver starts to drift out of his own lane (Metz, 2012). The automobile manufacturing company Mini (owned by BMW) has created its own way to incorporate AR in order to provide drivers with an enhanced driving experience. The Mini-branded AR glasses provide the wearer with augmented data such as messaging, mileage or speed. The AR overlay can also display navigation information and act as a console display to form a secondary rear-view mirror. The passenger side of some of the newer Mini vehicles vehicle has additional cameras which provide the driver with an “X-ray vision” of sorts. These cameras allow the driver to view video feeds when observing the passenger side of the vehicle. In essence, the driver obtains an overlay of what is happening behind the closed passenger door via an augmented overlay inside the HUD‟s lenses onto the real door (Wenz, 2015). Jaguar Land Rover created a very interesting concept system which intends to use a vehicle‟s windscreen as a HUD display device. Although Jaguar Land Rover stated that the technology is not yet available to accomplish this, the virtual windscreen concept they created provided interesting interactive virtual displays. Amongst these displays were virtual racing lines that change colour to indicate optimum braking positions for drivers when they approach treacherous curves. Another example includes virtual traffic cones that can be placed along the side of the road (in real time) to aid in the training of novice drivers to increase their competence and driving skill (Strange, 2014). Significance of the available information Similarly, linked and shared visualisation can be established when required for better collaboration between members, departments and different specialist roles. Better correspondence can be achieved with real-time updates. AR-based navigation systems could greatly assist operators as has already been mentioned. This concept could, however, be extended further by providing increased vision for operators in conditions of poor visibility (such as stormy winds that create dust clouds over surface mining operations). Virtual indicators can indicate the edges of roads, berms and sidewalls as well as other equipment or people. The maximum distance to reverse when discharging a load can also be shown accurately. This could prevent unnecessary damage to machinery, and lead to increased safety and productivity, reduced risk and higher utilisation of equipment.
39
2.3.16 Social Networking With the help of facial recognition technology, AR is able to display all types of information regarding any person at whom you point your smart device. This information can be sourced from, and linked to, a variety of social media. An example is shown in Figure 2.3.17, in which a smart phone is able to identify an individual (observed through the device‟s camera) and then, in turn, provide links to his/her social media profiles (Buckland, 2010).
Figure 2.3.17: Future of social networking with AR – Concept investigation (matthewbuckland.com, 2010) Significance of the available information This type of technology also has potential applications in other areas such as the work environment, where an employee is able to obtain information regarding his colleagues or subordinates, such as their position and seniority. This type of application does, however, have numerous privacy issues.
2.3.17 Translation AR systems have also been designed that can interpret the foreign text on signs and menus. These systems are able to re-display the text into translated languages (e.g. English) in an augmented view. Spoken words can also be translated and displayed in a user‟s view as subtitles in text format. “Word Lens” is an example of such an AR application (Tsotsis, 2010). 40
Significance of the available information Mining is internationally a major contributor to both the public and private sectors. International collaboration and correspondence often forms an essential part of task, projects or business relationships. This type of technology could aid understanding and communication between different language groups. Examples include communication with equipment or parts suppliers from other countries. This application could further benefit mining company culture on mines located in foreign countries. This will help in breaching communication barriers, and may extend to better acceptance by the surrounding and affected communities. 2.3.18 Mining When the above study was completed, current and potential AR applications in mining were then investigated. This research study contains the research and/or idea generation for potential AR applications of different sources, as well existing applications that are already in use in the mining sector. It will thus form a basis for further idea generation by providing insight into the point at which this study currently stands internationally. Health and Safety In the 2009 Annual review of the Safety and Health in Mine Research Advisory Board (SHMRAB) in the UK stated that a need for AR research exists. This need was identified in order to increase health and safety in mining. The research that followed this statement was based on “Enhanced miner-information interaction to improve maintenance and safety with augmented reality technologies and new sensors” (EMIMSAR). It was further stated that the research was aimed at accomplishing the following: to develop, implement and demonstrate AR devices and applications, enhanced marker systems and real-time location systems. The purpose was to improve the interaction of mine personnel with computerstored information in different fields of work. The research project also involved condition-oriented preventative maintenance. This was addressed through the development of novel sensors for online monitoring of critical parts of AFC and plough systems. Initial work was also undertaken on identifying and evaluating suitable positioning technologies for underground navigational aids (SHMRAB, 2009). In the EMIMSAR report released in 2012, and according to Catalina et al. (2012), the project contained the following areas of work (those which are most applicable to this study):
Identification, development or adaptation of suitable hardware;
Development of AR software for maintenance and repair processes; 41
Development of AR software for emergency operations;
Pilot and demonstration implementation of developed systems;
Integration of an AR informational and navigational aid.
Identification, development or adaptation of suitable hardware The project had been working on:
Identifying and evaluating usable positioning technologies to function as navigational aids in underground mines;
Identifying and adapting commercial display systems suitable for AR interfaces in mining conditions;
Developing ergonomic AR goggles that are appropriate for underground mining use;
Identifying an adequate mobile computing device suitable for use in mining environments and to develop a suitable software interface between the positioning system and the mobile device;
Developing an enhanced marker system that can perform automated identification of large machinery by means of a handheld PC. The purpose here is to gain immediate access to the status and operating conditions of the machines;
Evaluating the use of RFID technology for automated identification of machine parts or assemblies, along with people for maintenance and repair processes;
Selecting and adapting a tracking system to provide local positioning in the vicinity of a large machine.
Development of AR software for maintenance and repair processes A solution was implemented in the form of an AR-based assistant for Maintenance of Mining Machines (ARMM). Both software modules of the system, named “AR Creator” and “AR Viewer” respectively, were created from scratch through the use of various programming tools. This was done as no tools dedicated to creating AR software were available in the market at the time. In this system different materials can be assigned to markers. These include descriptions, labels, voice messages, images, animations, video recordings, etc. It is possible to create two types of resources, namely for aiding machine component identification and aiding maintenance activities. ARMM can also be used by two classes of mobile devices, which include tablet PCs as well as HMDs and wireless presenters.
42
Development of AR software for emergency operations An emergency application of AR-based emergency support was developed and evaluated. Results from other post-disaster, highly resilient data transmission media were studied. It was then decided to use an AR-enabled mobile node for the rescue team together with a rapid deployment communication system equipped with reliable communication and data transmission. The same technology was also used in the development of the positioning modules with the purpose of position determination. Integration of an AR informational and navigational aid Work on this task commenced with the study of hardware problems relating to navigation and positioning. The AR display software under development (discussed earlier) was then integrated with this information. An AR-based positional and navigation device was then integrated with the software, after which optimisations were carried out from an ergonomic perspective (Catalina et al., 2012). Pilot and demonstration implementation of developed systems The work entailed:
Integration of information and navigation capabilities with AR display systems in order to create an AR-based informational and navigational aid. The aid will be capable of integration with specific PDAs in an underground environment;
Pilot and demonstration implementation of AR informational and navigational aids for underground personnel in normal operations, such as maintenance technicians and supervisors, etc. Emergency actions, such as emergency rescue teams or first aiders, were also included;
Pilot and demonstration implementations of administration tools for position detection and sensor localisation;
Development of an AR system compatible with local PDA requirements that can also provide mapping-data acquisition and storage capabilities;
Pilot and demonstration implementation of AR-based human-computer interaction systems;
Pilot and demonstration implementation of the condition-oriented maintenance system.
Significance of the available information The above information identified a big move towards investigating applications of AR in mining and the types of work that had been undertaken. 43
Maintenance of Mining Machines Through a study of applying AR and RFID technologies in the maintenance of mining machines, further benefits of AR utilisation came to light. This study was the result of the EMIMSAR projects and, as a result, innovative training solutions were developed. Improvement of worker‟s skills during AR and RFID technology-based training was effected through the use of a real machine, which could be used on site in the future. Training that can be conducted directly at the workplace develops both knowledge and practical skills simultaneously (Michalak, 2012). Due to AR‟s ability to superposition computer images onto real-world objects (observed through display devices), the development of proper operative behaviour during machine breakdowns is made possible. In doing so, the application of AR in the mining industry increases safety and similarly minimises the risk of human error. Real-work conditions in real-work environments can be simulated on mobile stands due to the immersive nature of this technology. RFID technology enables the linking of knowledge resources to each mining machine, which simplifies and improves application of the knowledge during training. The technology can further be applied to training conducted in real, unfavourable mining conditions underground. RFID technology has been shown to aid mining machine maintenance. RFID delivers information about current technical conditions and knowledge about completing proper maintenance at the work site in real time. The system therefore provides more efficient maintenance of machinery due to the availability of electronic information and guidance. This is achieved on site through the use of AR computer images (and instructions) overlaid on the observed machine parts. This guidance includes forms of textual descriptions, static imagery, 3D models and interactive animations on methods of operation. It also contains information on conducting maintenance activities, including assembly and disassembly of parts. The system enables better translation of tacit knowledge into explicit knowledge. It also improves understanding and provides quick, on-hand search features of knowledge resources (Michalak, 2012). A software vendor named Chocolate Coded stated that they were in the process of developing an AR platform. The purpose of this AR platform is to streamline maintenance and site induction procedures on Australian mines. Crozier (2014) stated that Chocolate Coded had been in discussions with the earth-moving equipment supplier Komatsu Ltd. The discussions included
44
incorporating the specification data of Komatsu‟s equipment fleet into the Chocolate Coded AR tool (Crozier, 2014). Significance of the available information This section noted that AR-based aid for maintenance activities had already been identified and investigated in European countries. An extensive search led to no results found for South African mines using AR technology in the maintenance of mine vehicles at present. The study therefore, as indicated above, wanted to look at the potential future use thereof. Production and Safety “AR systems are making miners safer and more productive in dark, dangerous and confined spaces. Such systems are in commercial mines following the results of the EU-funded EMIMSAR project” (European Commission, 2014). By using systems developed by EMIMSAR, miners are able to “see” more clearly and, by doing so, work both faster and more safely. Miners can view AR versions of equipment on handheld computers and HMDs. They are able to scroll through different images of the machines and their components. They can recall historical and present data on the condition of the machines, or receive guidance on how to repair or maintain separate components. Sensors on the machinery provide useful insights into wear and tear on heavy-duty components, such as chains and gears. The system records and analyses temperatures and rates of acceleration. It also takes sound/noise samples from the sprockets on mining ploughs and flexible conveyors. From the noise samples wear can be detected due to alterations in the sound during operation. The sensor data is fed into a knowledge-based maintenance system. These data are combined with background data on the machinery and components along with associated visualisations. The data are then combined to create information-rich, real-time visualisations of the machinery to be viewed by those working on it. Voice-overs are also present to provide step-by-step instructions on maintenance procedures. According to EMIMSAR‟s coordinator, Juan Carlos Catalina (Catalina et al.,2014), the system greatly improves the productivity of maintenance teams. The information obtained can also be used at the mine‟s control centre to improve maintenance planning. Catalina et al. (2014) state that the EMIMSAR system is currently in use throughout the entire mining operations of RAG, Germany‟s largest coal mining corporation. RAG is an EMIMSAR 45
partner and the system is used by them for maintenance planning on longwall equipment, belt conveyors, loaders and other machinery. Other successful developments include a patented pick force sensor which measures loads on the road header (a boom-mounted excavator). Pick force sensors are currently being built into a range of equipment by the Austrian subsidiary of the Swedish mining and construction company, Sandvik (another EMIMSAR partner). EMIMSAR also came up with a solution to poor underground positioning and navigation technologies. It entailed an assembly of carefully fixed nodes in the mine tunnels that communicate with mobile nodes worn by miners. The closer the person is to a fixed node, the greater the signal intensity becomes. Measuring this intensity provides reasonably accurate positional measurement in the tunnel (European Commission, 2014). According to Catalina et al. (2014) this could be useful in emergencies such as a fire, where miners could identify and follow alternative routes to safety, even in thick smoke. Significance of the available information This section noted some of the movements brought about by the EMIMSAR research group and the progress it had reached up to this point. PerfectDig Maptek Pty Ltd. has created one of the first official and marketable AR applications for mining. The software, named PerfectDig, allows users to use handheld devices to rapidly compare laserscanned mine surface data during excavation against 3D mine plan designs. Conformance measurement within the field can then be conducted in real time (Sparpointgroup, 2013). The software automatically combines mine plans and design information from Maptek Vulcan with laser scans of the mining area in an AR application. The mine design is compared to 3D data to generate imagery which immediately identifies areas of non-conformance. This is accomplished via Maptek‟s I-Site laser scanner which captures surface data as mining takes place. Users can then assess the 3D visual and spatial analysis comparison on a handheld device and select which overlays to display. Such overlays include layers, depths, overdig, underdig, volumes and measurements. Figure 2.3.15 shows the augmented overlays that can be viewed for comparison between planned design and actual real-time design (Maptek, 2013). PerfectDig streamlines communication between mining engineers, surveyors and equipment operators to ensure adjustments to excavations are made correctly and as required. This facilitates efficient removal of soil, overburden and other material for optimal mineral recovery. An 46
excavation can be monitored as it progresses. A digital audit trail can be maintained, and required adjustments can be identified to allow better design conformance. This promotes efficient resource allocation and leads to improved wall stability and safety (Maptek, 2013).
Figure 2.3.15: Maptek's PerfectDig displaying augmented design plans overlaid onto actual excavations (sparpointgroup.com, 2013)
Significance of the available information The above information describes one of the first actual applications of AR in mining and what can be done with it. SHEQSCAN Another ground-breaking (and also one of the first official and marketable) applications of AR in mining is a mobile technology and AR collusion named SHEQSCAN. This AR software was created by Unfolding AR (Pty) Ltd. in partnership with IHPC (Institute for High Performance Consulting SA (Pty) Ltd.). Both are South African-based companies. SHEQSCAN is designed to operate on any Android-based touch-screen device and utilises easily navigable menus to assist users in completing their tasks. The software is an integrated audit and inspection tool which functions to simplify regulatory governance and compliance from a facilities management perspective (Unfolding AR, 2014). SHEQSCAN offers various digital media to specifically assist in training the user to become competent in any complex, real-world task or situation. This is achieved through on-the-spot, stepby-step digital tutorials. These coaching functions are available while conducting inspections or audits in the mining environment. SHEQSCAN is able to track construction progress and project performance. It can further also integrate complex structures and material usage during assembly.
47
It also captures visual proof of inspection, work and repairs to aid in managerial control (Unfolding AR, 2014). This software displays AR-based digital information on-screen, on what the device‟s camera recognises during inspection (what the user points the device‟s camera at during inspections or audits).
This object/device recognition can also be triggered by the GPS coordinates of the
SHEQSCAN-enabled device during use. Compliance with structural and other requirements, as well as baseline risk assessments, can then be conducted. Such compliance can further be proven and traced through the GPS system. When non-conformance is identified during an inspection, maintenance defects and areas requiring improvements can be addressed immediately through a work order functionality on the system (Unfolding AR, 2014). Through the use of GPS coordinate tracking, it can also be traced and regulated that an audit or inspection has indeed taken place. This tracking also captures which device, linked to which individual, did the inspection (Unfolding AR, 2014). SHEQSCAN is pushing towards paperless maintenance and control and might very well form a stepping stone to a new era of technology utilisation in the world of AR. Facilities management is but one of numerous applications for the mobile inspection and audit tool. Others include safety, health, environment, quality, risks, aviation and maintenance (Unfolding AR, 2014). Significance of the available information The significance of the information above is yet another step towards applying AR in mining and provides good insight into what can already be achieved. Subsurface Data Visualisation In 2002 the University of Nottingham in the UK conducted research on AR systems that would allow people to look into the ground and see underground features. These features could be major geological structures, gas or water pipe layouts or zones of contaminated land. The prototype they developed was successful in underground visualisation of known data, such as buried pipes and cables beneath a tar road (Roberts et al., 2002). In the research it was noted that an important aspect of the work was the accuracy of the data to be visualised. A user is only able to augment the virtual image where the system shows it to be. Therefore accurate survey data of the features in question are fundamental. Surveys are conducted during installation or through non-intrusive seismic or ground-penetrating radar surveys, otherwise the user will effectively visualise a virtual image in the wrong position – “the system is not a pair of x-ray glasses” (Roberts et al., 2002). 48
Significance of the available information The section above ties in with a previously mentioned potential application pertaining to the prospect of augmented views of obstructed underground factors. This can be further expanded to underground use. A miner could possibly use an AR-enabled device for a 360 degree view of a mining excavation. The AR-generated view could then contain 3D visuals of all the geological elements that need to be considered for safe and productive mining. Identified Potential Applications in Mining Hugues et al. (2012) explored potential uses of AR in the mining sector. A specific focus was placed on open-pit mining. Although they identified AR‟s great potential in the exploration and planning phases, only the operation phase was considered. The article identifies several potential applications from the perspective of needs that were identified on the Osisko mine in Canada. Osisko is faced with many operational constraints, amongst which are governmental regulations regarding vibration, noise and dust generation. In addition, the mine is required to clean up previously polluted operations as part of an operations rehabilitation project. Following this, one of the first uses of AR identified was the management of the site from an environmental perspective. However, no details of how to accomplish this were identified, but operational applications for AR were listed and described verbatim as follows: Heavy Machine Operators “Operators must deal with one major problem: Lack of visibility. On board these giant machines, which weigh several hundred tonnes, operators must deal with significant blind spots. In many situations, drivers must manoeuver “blind”, guided only by their experience. The use of augmented reality to solve this type of problem is attractive.” Drillers and Dynamiters “Similar to what is done in the field of public works, we think that augmented reality could be used to enable drillers to ensure that the positioning of the drill holes is correct. The goal of using augmented reality in this context would be to limit the different transits currently needed, between the site and the paper or electronic plans, for definition of the drill pattern. The use of a portable device can be envisioned in this context.” Heavy or Fixed Machinery Mechanics “The first uses of augmented reality rightly took place in the field of manufacturing or maintenance aid. Besides the fact that augmented reality minimizes reliance on paper media or on the direct 49
consultation of a computer tool during the maintenance task, augmented reality makes it possible for an expert to remotely guide an onsite technician in conditions which are an improvement upon the use of a simple telephone.” Mining Technicians “The tasks of a mining technician are varied. As planners, mining technicians are called on to plot the mine’s ramps and access points. In some cases, the mining technician will need to participate in the diversion of existing roads. The mining technician also participates in drawing up drilling plans for drillers and dynamiters. The use of computer tools is now generally accepted practice, enabling technicians to use computer-aided design software in doing so. During exterior work, the use of augmented reality could help technicians deal with all of the constraints related to public works, excavation work or those related to artwork.” Geotechnology “During the mining operations phase, sometimes boundaries are moved, without the geotechnicians’ approval. They must therefore ensure that the position of the lines is consistent with the plans, which requires sometimes very exacting, point-by-point verifications. The use of augmented reality enables the technician to easily verify that the boundary points on the site are still in place.” The article further states that even though the operations phase of the AR applications were discussed, it was hypothesised that AR could be a great asset in terms of the environment, safety, productivity and the other phases of the mining cycle. The list of potential applications in mining was expanded by Srinivasan et al. (2012). They state that the application of AR in the mining industry would help overcome many of the challenges currently faced by maximising the safety and effectiveness of the personnel on a mine site. The application of AR in the mining industry would help overcome many of the challenges currently faced by maximising the safety and effectiveness of the personnel on a mine site. The list of potential applications in mining can also be expanded, or some of those already pointed out in previous sections could be added to, such as those mentioned by Srinivasan et al. (2012): Digital Instructions The concept here is step-by-step instructions that are overlaid via an AR display device on the actual item a worker is working on. Maintenance, repair and overhaul tasks are believed to be grasped more easily. It was further hypothesised that: 50
Work instructions should be easier to understand;
Untrained workers should be likely to perform tasks as well as experienced workers with help from such an AR system;
Tasks are also likely to be completed faster and with fewer errors in this way.
The benefits derived from this application are potentially:
Improvement in equipment availability;
Reducing reliance on key personnel;
Better health and safety conditions.
Remote Collaboration This aspect relates to shared real-time views and helpful augmented overlays between someone performing a task and another providing guidance. Areas or objects can be highlighted remotely instead of describing them, and actions can be visually demonstrated instead of purely talking a person through them. This can significantly reduce the time spent on a task by reducing lag periods associated with waiting for guidance and assistance. This way workers with more experience can still be used on a retainer basis (even if they are retired) to provide their knowledge to other workers. Distances can be breached with this application of AR as well, which will reduce travelling and salary costs. Operator Guidance This involves wearing a display device such as a HUD that provides the operators with necessary and helpful information in the form of AR overlays. The information can be anything from basic control or manoeuvring guidance to bucket or load fill percentages. It is believed that this application will improve safety due to better situational awareness. It is also believed that it will increase productivity due to better equipment utilisation and more efficient operating of the equipment. It is further hypothesised that such a system will reduce the cognitive workload placed on operators. Potential hazards can also be highlighted on the display (e.g. bringing up camera views of hazardous spots and nearby personnel or equipment). This will increase safety and remove the need to constantly check mirrors and cameras which results in attention fluctuations. Training Based on the success of VR-based training, it is believed that AR can make training more effective, realistic and cost effective. AR provides significant advantages over VR as it is able to 51
provide training in real time in the real world and still retain the benefits of virtual scenarios. ARbased training would also not require trainees to travel to a certain location in order to receive training, as it can be provided on site from a remote location. Incident Investigation This entails reconstructing major incidents in order to determine what the root causes were that led to the incident. Understanding how an incident occurred and what caused it could help prevent a recurrence. AR is able to help recreate scenarios with virtual elements and even virtually reconstruct destroyed equipment or structures. All of this can be done at a relatively low cost compared to physical reconstructions. Significance of the available information The significance of the information above is that it provides good insight into work that is in progress and investigations that have already been conducted. In order to avoid reinventing the wheel, this section is of paramount importance and will comprise a large portion of future work to identify potential AR applications in the mining sector.
2.4
Conclusions
Numerous technological advances were identified that are in development and are being researched, some of which could drastically impact the use of AR technology in the mining sector as the associated hardware become more useful and practical. From the current AR eyewear that has already been developed, it seems certain that the hardware can be adapted and made similar to current PPE eyewear. It also seems possible that in future AR-enabled contact lenses will become practical for use in certain work environments (environments that have a low risk of foreign objects getting into one‟s eye). This means that the hardware should not be a limiting factor when investigating scenarios or environments for AR applications, such as the strenuous use of tablet PC devices. Numerous potential AR applications for the mining sector were identified from the literature survey. The survey was conducted on existing AR applications as well as on other applications either being investigated or noted as possibilities. These applications were predominantly from nonmining environments as well as existing AR applications in mining. In order to further critically investigate potential mining applications, it is important to first note those that already exist. The identified existing AR applications for the mining industry are listed below:
AR-based assistant for Maintenance of Mining Machines (ARMM). European initiative. 52
o
Includes viewing AR versions of equipment on handheld computers and HMDs.
o
Scrolling through different AR visuals of the machines and their components.
o
Recalling historical and present data on the condition of machines, in real time, as augmented overlays.
o
Receiving guidance on how to repair or maintain separate components.
o
Combining wear and tear detection systems with AR to create information-rich, realtime visualisations of machinery and its status.
PerfectDig AR system compares laser-scanned mine surface data against 3D AR mine plan designs for conformance measurement (not real time). Available globally, including in South Africa.
SHEQSCAN, an integrated AR audit and inspection tool. It also offers various digital media to specifically assist in training the user to become competent in any complex, real world task or situation. Available in South Africa.
Some potential mining AR applications that were already identified by various minds are listed below:
Emergency support – the use of AR-enabled mobile nodes for the rescue team. The nodes function as position detection devices. Information regarding the node being tracked can then be overlaid on a display device. Includes a rapid deployment communication system with reliable communication and data transmission.
The use of RFID technology for automated identification of machine parts or assemblies, along with people for maintenance and repair processes (ties in with ARMM).
Integration of information and navigation capabilities with AR-based systems for informational and navigational aid.
AR systems that will allow people to look into the ground and see underground features. These features could be major geological structures, gas or water pipe layouts or zones of contaminated land.
AR-based system for visual aid and guidance for heavy machine operators.
AR assistance for accurate drill hole positioning – will limit different transits required between planning and the drill site.
AR-assisted “art work” to assist mining technicians and engineers with design and planning. 53
Tracking of planning to ensure that mining boundary points are in place when moved: in short, AR geotechnology conformance visualisations.
Digital instructions. The concept is step-by-step instructions that are overlaid via an AR display device on the actual item a worker is working on. o
Maintenance, repair and overhaul tasks.
Remote collaboration. This aspect relates to shared real-time views and helpful augmented overlays between someone performing a task and another providing guidance. o
On-site AR guidance for heavy or fixed machinery mechanics between technicians and experts via a remote system.
AR-based training – believed to make training more effective, realistic and cost effective.
Incident investigation. This entails on-site reconstruction of major incidents with AR visuals to determine what the root causes were that led to the incident.
Potential AR applications that were identified for the mining sector, based on other types of applications investigated in the literature study:
Augmented displays can be generated of an entire underground mine, with the ability to zoom in on different sections to show accurate details. These can then be viewed while standing on the surface at the mine or even while sitting in a boardroom. The visual aids of AR can greatly help planning and grasping of layouts without having to physically visit every section.
AR-based views of obstructed underground factors which expand to an all-round view when underground to observe the hidden details within the host rock. The geological features, ground conditions, rock types, discontinuities and other relevant information can be displayed in the form of AR holograms or 3D models.
Display of mining site plans – can be used for planning during the development phase. Also for visual representations to stakeholders or when applying for a mining licence. A complete walk-through can be conducted in an augmented environment as if the mining site had already been constructed around the viewer. Can also be used as a model to solidify a commitment in terms of design methodology or treatment processes, etc.
Recreating historical mine sites or equipment – even creating a holographic display of a historical mine in operation.
54
Recreating historical events, such as incident recreation for investigation purposes. Can be done on site with minimal disturbance to the site of the incident.
Combining eye-tracking technology with AR technology to create displays or issue commands for numerous purposes. The purposes may include operators interacting with an AR system for guidance, navigation, support functions or requesting information. May also include sending out signals, such as transferring information regarding warnings or production parameters, or sending out distress signals.
Augmented views of personnel ID cards to provide additional AR-overlaid information. Levels of access to different information can be controlled based on the hierarchy difference between the viewer and the person being “scanned”. Examples include general personal information, as well as section/area of work, expertise, superior and subordinates, experience and even personal ratings.
RFID tracking can also be applied to both equipment and ID cards for monitoring purposes. The location history for these can then be recalled in an AR display when and where required.
Mining materials, equipment, components and consumables can be visualised on site (in the form of 3D AR visuals, such as full-scale models). This can be used to accurately determine the applicability, level of accuracy and conformance of an item prior to placing orders.
Direct visual communication with AR-aided interactions between the parties can be established. This can be used for precise communication between different stakeholders, which can include communication between management, with suppliers, or between trainees and experts as if the parties were standing next to one another in real time.
AR-based training programmes, such as emergency procedures, operator training, task performance, etc. The visual nature of the technology can greatly aid understanding and reduce training/education duration. Concepts can be grasped quicker and more accurately, with less room for inaccurate perceptions.
Combining tracking systems (such as RFID and personnel ID card tracking) with mine layout augmentations can aid rescue operations. A system can be developed to provide real-time assistance for the best course of action, escape routes to take or ways to navigate towards those requiring assistance. Alternatively these applications can be used for optimal human-generated planning for the same scenarios.
55
AR-assisted visual aids that not only enhance and amplify vision, but also provide augmented information on details that the human eye is unable to detect.
Mining games or software development that promotes at home “play” and/or easier mine planning and design in the professional environment. Examples include designing mines, playing with configurations, layouts and designs, equipment matching, etc.
AR-constructed crash tests. For modelling purposes of support, designs, material selection etc.
Linked and shared AR visualisation can be established when required. The sharing of information and visuals will provide better collaboration between personnel, departments and various specialists. Better correspondence can be achieved with real-time updates.
AR-based navigation systems. This could provide increased vision for operators in conditions of poor visibility (such as stormy winds that create dust clouds over surface mining operations). Virtual indicators can indicate the edges of roads, berms and sidewalls as well as other equipment or people. The maximum distance to reverse when discharging a load can also be shown accurately.
Integration of AR-assisted language translations (both voice and text) for increased communication efficiency. This is accomplished with object (text) recognition software and will aid international mining companies to breach language barriers.
56
REFERENCES Abernathy, M. et al. 1993. Debris Correlation Using the Rockwell WorldView System, Maui: Air Force Maui. Ackerman,
J.
2000. Ultrasound
Visualization
Research.
[ONLINE]
Available
at: http://www.cs.unc.edu/Research/us/ [Accessed 25 April 2015]. Alexander, M. 2012. Arbua Shoco Owl Silver Coin with Augmented Reality | Coin Update. [ONLINE] Available at: http://news.coinupdate.com/arbua-shoco-owl-silver-coin-with-augmentedreality-1490/. [Accessed 20 April 2015]. Andrews, K. 2013. Ikea launches augmented reality catalogue - Dezeen. [ONLINE] Available at: http://www.dezeen.com/2013/08/05/ikea-launches-augmented-reality-catalogue/.
[Accessed
21
April 2015]. Anthony,
S.
C.
2012. US
military
developing
multi-focus
augmented
reality
contact
lenses|ExtremeTech. [ONLINE] Available at: http://www.extremetech.com/computing/126043-usmilitary-developing-multi-focus-augmented-reality-contact-lenses. [Accessed 09 May 2015]. Anthony,
S.
C.
2014. Virtual
retinal
display.
[ONLINE]
Available
at:
http://www.edutalks.org/downloads/VIRTUAL%20RETINAL%20DISPLAY.pdf. [Accessed 09 May 2015]. Arthur, C. 2012. UK company's 'augmented reality' glasses could be better than Google's| Technology|
The
Guardian.
[ONLINE]
Available
at:
http://www.theguardian.com/technology/2012/sep/10/augmented-reality-glasses-google-project. [Accessed 10 May 2015]. AR23D. 2013. Mining | Augmented Reality News by AR23D Studio. [ONLINE] Available at: http://blog.ar23d.com/tag/mining/. [Accessed 20 April 2015]. Barrie, P. et al. 2009. A Pervasive Gesture-Driven Augmented Reality Prototype using Wireless Sensor Body Area Networks, Glasgow: Glasgow Caledonian University. Bimber, O. et al. 2001. The Extended Virtual Table: An Optical Extension for Table-Like Projection Systems. Massachusetts Institute of Technology, vol.10, no.6, pp. 613-631. BMW. 2015. BMW Augmented Reality: Workshop applications. [ONLINE] Available at: http://www.bmw.com/com/en/owners/service/augmented_reality_workshop_1.html. [Accessed 13 May 2015].
57
Branscombe, M. 2013. Beyond the mouse: how input is evolving | TechRadar. [ONLINE] Available at: http://www.techradar.com/news/computing/beyond-the-mouse-how-input-is-evolving-626794/1. [Accessed 11 May 2015]. Brown, C. 2011. Groundbreaking Augmented Reality-Based Reading Curriculum Launches . [ONLINE] Available at: http://www.prweb.com/releases/2011/10/prweb8899908.htm. [Accessed 21 April 2015]. Buckland, M. 2010. The Future of Social Networking – a Concept Investigation with Augmented Reality. [ONLINE] Available at: http://matthewbuckland.com/?p=1041. [Accessed 21 April 2015]. Calhoun, G. L. et al. 2005. Synthetic Vision System for Improving Unmanned Aerial Vehicle Operator Situation Awareness. Proceedings of SPIE Enhanced and Synthetic Vision, vol. 5802, pp. 219–230. Catalina, J. C. et al. 2012. EMIMSAR, Brussels: European Commission. Cameron, C. 2010. Military-Grade Augmented Reality Could Redefine Modern Warfare. [ONLINE] Available at: http://readwrite.com/2010/06/11/military_grade_augmented_reality_could_redefine_modern_warfa re. [Accessed 26 April 2015]. Camma Icube. 2014. 3D Global Estimation and Augmented Reality Visualization of Intra-operative X-ray Dose - YouTube. [ONLINE] Available at: https://www.youtube.com/watch?v=pINE2gaOVOY. [Accessed 25 April 2015]. Churcher, J. 2013. Augview | Blogs | Augmented Reality: Future or Fad? [ONLINE] Available at: http://www.augview.net/News/Blog/archive-13September2013.html. [Accessed 20 April 2015]. Cooper, J. 2007. Supporting Flight Control for UAV-Assisted Wilderness Search and Rescue Through Human Centered Interface Design, Provo: BYU ScholarsArchive. Crowe, A. 2013. What Does the Future Hold for Emergency Management? [ONLINE] Available at: http://www.emergencymgmt.com/emergency-blogs/disasters20/What-Does-the-Future-Hold-forEmergency-Management.html. [Accessed 22 April 2015]. Crozier, R. 2012. Aussie Miners Toy With Augmented Reality - Software - News - iTnews.com.au . [ONLINE]
Available
at:
http://www.itnews.com.au/News/316460,aussie-miners-toy-with-
augmented-reality.aspx. [Accessed 13 May 2015]. Davies, C. 2012. Quantigraphic camera promises HDR eyesight from Father of AR - SlashGear. [ONLINE] Available at: http://www.slashgear.com/quantigraphic-camera-promises-hdr-eyesightfrom-father-of-ar-12246941/. [Accessed 22 April 2015]. 58
Day9. 2013. Your thoughts about augmented reality in video games. Day[9]TV. [ONLINE] Available
at: http://day9.tv/d/Lineste/your-thoughts-about-augmented-reality-in-video-games/.
[Accessed 23 April 2015]. Dempsey,
G.
2015. Anatomy
4D
|
Qualcomm.
at: https://www.qualcomm.com/products/vuforia/case-studies/anatomy-4d.
[ONLINE] [Accessed
Available 22
April
2015]. Dillow, C. 2009. BMW augmented reality glasses help average Joes make repairs. [ONLINE] Available
at: http://www.popsci.com/scitech/article/2009-09/bmw-developing-augmented-reality-
help-mechanics. [Accessed 22 April 2015]. Divecha, D. 2011. Augmented Reality (AR) used in architecture and design | Design Middle East. [ONLINE]
Available
at:
http://www.designmena.com/inspiration/augmented-reality-ar-part-
architecture-design. [Accessed 19 April 2015]. Endgadget.com. 2013. Virtual Retinal Display (VRD) prototype. [ONLINE] Available at: http://www.engadget.com/2013/10/09/avegant-retinal-hmd/. [Accessed 19 April 2015]. European Commission. 2014. Augmented reality improves safety and production. [ONLINE] Available at: http://ec.europa.eu/programmes/horizon2020/en/news/augmented-reality-improvessafety-and-productivity-mines. [Accessed 12 May 2015]. Eve, S. 2012. Augmenting Phenomenology: Using Augmented Reality to Aid Archaeological Phenomenology in the Landscape. Journal of Archaeological Method and Theory, vol.19, no.4, pp.3-10. Gannes, L. 2012. Google Unveils Project Glass: Wearable Augmented-Reality Glasses - Liz Gannes - News - AllThingsD. [ONLINE] Available at: http://allthingsd.com/20120404/googleunveils-project-glass-wearable-augmented-reality-glasses/. [Accessed 10 May 2015]. Greenemeier, L. 2011. Contact Lenses Can Display Your Text Messages. [ONLINE] Available at: http://mashable.com/2012/12/08/contact-lenses-text-messages/. [Accessed 09 May 2015]. Grifantini, K. 2010. Augmented Reality Goggles | MIT Technology Review. [ONLINE] Available at: http://www.technologyreview.com/news/421606/augmented-reality-goggles/. [Accessed 09 May 2015]. Hill, S. 2013. Every Google Glass Smart Eyewear Competitor in Development | Digital Trends. [ONLINE] Available at: http://www.digitaltrends.com/mobile/google-glass-alternatives/. [Accessed 10 May 2015].
59
Hitlabnz. 2008. Outdoor AR - YouTube. [ONLINE] Available at: https://www.youtube.com/watch?v=jL3C-OVQKWU. [Accessed 10 April 2015]. Hudson, L. 2015. We've Played HoloLens Minecraft: Hands On With Microsoft's Augmented Reality Headset | Rock, Paper, Shotgun. [ONLINE] Available at: http://www.rockpapershotgun.com/2015/01/22/minecraft-hololens-minecraft-reality/. [Accessed 23 April 2015]. Hugues, O. et al. 2013. Towards the Application of Augmented Reality in the Mining Sector Open Pit Mines. International Journal of Applied Information Systems, vol. 4, no. 6, pp. 1-7. [Accessed 05 May 2015]. Kaufmann, H. 2000. Construct3D: An Augmented Reality Application for Mathematics and Geometry Education, Vienna: Institute of Software Technology and Interactive Systems. Kaufmann, H. 2003. Collaborative Augmented Reality in Education, Vienna: Vienna University of Technology. Kats,
R.
2012.
Mobile
Marketer:
Software
Technology.
[ONLINE]
Available
at:
http://www.mobilemarketer.com/cms/news/software-technology/13810.html. [Accessed 14 May 2015]. King, R. 2015. Augmented Reality Goes Mobile - Businessweek. [ONLINE] Available at: http://www.bloomberg.com/bw/stories/2009-11-03/augmented-reality-goes-mobilebusinessweekbusiness-news-stock-market-and-financial-advice. [Accessed 13 May 2009]. Knight, W. 2015. Augmented reality brings maps to life - tech - 19 July 2005 - New Scientist. [ONLINE] Available at: http://www.newscientist.com/article/dn7695#.VU-ZA_mqqko. [Accessed 10 May 2005]. Lee, A. et al. 2012. CityViewAR: A Mobile Outdoor AR Application for City Visualization, Canterbury: The Human Interface Technology Laboratory New Zealand, University of Canterbury, New Zealand. Mahei.es.
2015. Mahei:
Augmented
reality
for
mobile
devices.
[ONLINE]
Available
at: http://mahei.es/iskull.php?lang=en. [Accessed 21 April 2015]. Maier, et al. 2009. Augmented Reality for Teaching Spatial Relations, München: Technische Universität München. Mann,
S.
2004. EyeTap
Personal
Imaging
Lab.
at: http://www.eyetap.org/research/eyetap.html. [Accessed 22 April 2015].
60
[ONLINE]
Available
Maptek. 2013. Maptek - PerfectDig. [ONLINE] Available at: http://www.maptek.com/products/perfectdig/. [Accessed 12 May 2015]. Matsuda, K. 2015. Projects | Keiichi Matsuda. [ONLINE] Available at: http://km.cx/projects/. [Accessed 19 April 2010]. Metz, R. 2012. Augmented Reality Is Finally Getting Real | MIT Technology Review. [ONLINE] Available at: http://www.technologyreview.com/news/428654/augmented-reality-is-finally-gettingreal/. [Accessed 5 May 2015]. Meyer, D. 2012. Telefónica bets on augmented reality with Aurasma tie-in | Gigaom. [ONLINE] Available
at: https://gigaom.com/2012/09/17/telefonica-bets-on-augmented-reality-with-aurasma-
tie-in/. [Accessed 20 April 2015]. Michalak, D. 2012. Applying the Augmented Reality and RFID Technologies in the Maintenance of Mining Machines, San Francisco: World Congress on Engineering and Computer Science. Mims, C. 2010. Treating Cockroach Phobia With Augmented Reality | MIT Technology Review. [ONLINE] Available at: http://www.technologyreview.com/view/419653/treating-cockroach-phobiawith-augmented-reality/. [Accessed 25 April 2015]. Mountney, P. et al. 2009. Optical biopsy mapping for minimally invasive cancer screening. Medical Image Computing and Computer-assisted Intervention, vol. 83, no.90, p. 12. Neubauer, W. 2015. School of Gladiators discovered at Roman Carnuntum, Austria – Ludwig Boltzmann Institute for Archaeological Prospection and Virtual Archaeology. [ONLINE] Available at: http://archpro.lbg.ac.at/press-release/school-gladiators-discovered-roman-carnuntum-austria. [Accessed 18 April 2015]. Noelle, S. 2002. Stereo augmentation of simulation results on a projection wall by combining two basic ARVIKA systems, Wolfsburg: SMAR. Pang, Y. et al. 2004. Assembly Design and Evaluation in an Augmented Reality, Singapore: National University of Singapore. Recinos, E. 2013. Volkswagen XL1 Gets an Augmented Reality Service App. [ONLINE] Available at: http://mashable.com/2013/10/05/volkswagen-augmented-reality/. [Accessed 13 May 2015]. Roberts, G. W. et al. 2002. The Use of Augmented Reality, GPS and INS for Subsurface Data Visualisation, Washington: FIG XXII International Congress. Rodes, N. L. & Padoy P. N., 2014. 3D Global Estimation and Augmented Reality Visualization of Intra-operative X-ray Dose, Unknown: US National Library of Medicine. 61
Rolland, J. P. et al. 2005. Development of Head-Mounted Projection Displays for Distributed, Collaborative, Augmented Reality Applications. Presence, vol.14, no.5, pp. 1-2. Rollenhagen, L. 2013. Audi App Turns Your iPhone Into a Robust User Manual. [ONLINE] Available at: http://mashable.com/2013/08/21/audi-app-car-user-manual/. [Accessed 13 May 2015]. Rosen, K. 2012. Contact Lenses Can Display Your Text Messages. [ONLINE] Available at: http://mashable.com/2012/12/08/contact-lenses-text-messages/. [Accessed 09 May 2015]. Sheridan, K. 2015. Microsoft HoloLens vs. Google Glass: No Comparison - InformationWeek. [ONLINE]
Available
at:
http://www.informationweek.com/mobile/microsoft-hololens-vs-google-
glass-no-comparison/d/d-id/1318851. [Accessed 10 May 2015]. SHMRAB, 2009. Safety and Health in Mines Research Advisory Board - Annual Review 2009, Sheffield: Health and Safety Executive. Slideshare.
2014. Virtual
Retinal
Display
(VRD).
[ONLINE]
Available
at: http://www.slideshare.net/asertseminar/virtual-retinal-display-vrd. [Accessed 09 May 2015]. Sparpointgroup. 2013. Maptek – Augmented reality reaches mining industry | SPAR Point Group. [ONLINE]
Available
at: http://www.sparpointgroup.com/news/digital-visualization/augmented-
reality-reaches-mining-industry. [Accessed 12 May 2015]. Srinivasan, V. et al. 2012. The Augmented Mine Worker - Applications of Augmented Reality in Mining, Sydney: CSC Australia (Pty) Ltd. Stewart-Smith. 2012. Education with Augmented Reality: AR textbooks released in Japan (video) | ZDNet. [ONLINE] Available at:http://www.zdnet.com/article/education-with-augmented-reality-artextbooks-released-in-japan-video/. [Accessed 21 April 2015]. Strange, A. 2014. Jaguar Concept Windshield Shows Off Augmented Reality in the Car. [ONLINE] Available at: http://mashable.com/2014/07/14/jaguar-augmented-reality/. [Accessed 13 May 2015]. Sung, D. 2011. Augmented reality: a long way off? - Pocket-lint. [ONLINE] Available at: http://www.pocket-lint.com/news/108949-augmented-reality-interview-steve-feiner.
[Accessed
10 May 2015]. Tanase, A. 2015. Augmented Beauty: L'Oréal Makeup Genius App | TechAcute. [ONLINE] Available at: http://techacute.com/augmented-beauty-loreal-makeup-genius-app/. [Accessed 20 April 2015].
62
Tsotsis, A. 2010. Word Lens Translates Words Inside of Images. Yes Really. | TechCrunch. [ONLINE] Available at: http://techcrunch.com/2010/12/16/world-lens-translates-words-inside-ofimages-yes-really/. [Accessed 13 May 2015]. Unfoldingar. 2014. Unfolding AR | Augmented Reality Leaders | South Africa. [ONLINE] Available at:
http://www.unfoldingar.co.za/#!SHEQSCAN-MOBILE-TECHNOLOGY-USES-IN-FACILITIES-
MANAGEMENT/c1jmk/E3485F4E-F368-4282-9381-AE3F0364A53F. [Accessed 12 May 2015]. Van Meeuwen, R. 2015. Architectural dreams in augmented reality. University News: The University of Western Australia. [ONLINE] Available at: http://www.news.uwa.edu.au/201203054410/events/architectural-dreams-agumented-reality. [Accessed 19 April 2015]. Vrealities.
2013. New
Augmented
Reality
HMD!
Z800.
[ONLINE]
Available
at:
https://www.vrealities.com/products/augmented-reality/z800-pro-ar. [Accessed 10 May 2015]. Wadhwa, T. 2013. CrowdOptic and L'Oréal to make history by demonstrating how augmented reality can be a shared experience - Forbes. [ONLINE] Available at: http://www.forbes.com/sites/tarunwadhwa/2013/06/03/crowdoptic-and-loreal-are-about-to-makehistory-by-demonstrating-how-augmented-reality-can-be-a-shared-experience/. [Accessed 20 April 2015]. Wagner, D. &. Schmalstieg D. 2006. Handheld Augmented Reality Displays, Graz: Graz University of Technology, Austria. Webley, K. 2010. EyeWriter - The 50 Best Inventions of 2010 - TIME. [ONLINE] Available at: http://content.time.com/time/specials/packages/article/0,28804,2029497_2030618_2029822,00.ht ml. [Accessed 20 April 2015]. Wenz, J. 2015. Behold BMW's Augmented Reality Driving Glasses. [ONLINE] Available at: http://www.popularmechanics.com/cars/a15023/bmw-is-coming-out-with-augmented-realitydriving-glasses/. [Accessed 13 May 2015].
63
CHAPTER 3: RESULTS AND ANALYSIS OF RESULTS For the purpose of this study, focus was placed on potential Augmented Reality (AR) applications in the mining environment that were identified as having the greatest likelihood of implementation at this point in time. The likeliness of implementation stems from current readily available technology that could hinder the adoption of a “new” technology. Other factors could also play a part in the adaptation of such a relatively unknown technology, for example an associated lack in advancement of equipment, financial funds allocated to R&D, and bringing in a potentially mind-set changing technology (as well as numerous physical changes such as equipment, electronic devices, software usage, and even training methods). Four potential AR applications, along with an overview of how they could be functionally implemented, will now be discussed. Three of the aforementioned applications were either identified or inspired by the literature study. The first application regarding drilling applications, however, was an independently generated idea. A SWOT analysis is provided for each application separately, as well as an additional SWOT analysis on Augmented Reality technology applications in mining (with the focus predominantly on the South African mining industry).
3.1
Drilling Applications
AR has the potential to greatly aid and improve both the accuracy and efficiency of drilling practices. An AR system is envisaged which would be able to help guide drilling operations by providing the operators with visual aids. The envisaged system would make use of a virtual column that represents the physical drill rod and another column that represents the planned drill hole. The operator would then be able to see the real-time location of the drill rod in relation to the planned drill hole, despite visual obstructions such as the ground and the machine casing itself. In this way the deflection angle of the drill rod could be visually displayed and monitored as drilling progresses in order to maintain the desired drilling angle (e.g. 90 degrees to surface). If the drill rod deviates from its desired course, the colour of the two virtual columns would change to indicate the inaccuracy (and digital information would be provided to explain how and which way the rod needs to be steered or readjusted to correct the error). The drill rod could then be steered back towards the correct angle within the virtual column representing the desired drill hole. Adjustments could be made until the colour changes back again to indicate an acceptable angle (e.g. a green column for acceptable (planned) angle, moving to orange for acceptable deviations and a red column for unacceptable deviation representations). This could result in higher accuracy
64
in drilling practices. Figure 3.1a shows a basic example of how AR could show what is happening underground, along with instructions on how to rectify inaccuracies.
Figure 3.1a: Augmented drill hole with indicated deflection on the drill head In the example in Figure 3.1a the drill head was deviating too much and the virtual drill hole turned to red, with the drill rod inside the hole shown in yellow. For illustration purposes the drill rod is shown as having a smaller diameter than the hole, but for practical reasons one might have the virtual drill rod (yellow) replace the virtual drill hole column (green-orange-red) as drilling progresses. Such an AR system, which is aimed at improving drilling accuracy, is envisaged to work with another system that is able to track the real-time location of the drill bit (drill head) itself. Numerous systems exist that are capable of relaying the exact location of the drill head as drilling progresses. Many surgical procedures, particularly those used in orthopaedic surgery, require holes to be drilled in the bone of a patient. Surgical drills have long been used for this purpose along with various mechanical guidance instruments. These instruments are designed to enable a surgeon to accurately drill a hole to the desired depth and without the danger of damaging surrounding tissue. Such drill guides often provide a visual depth gauge which requires the surgeon to read the depth of the drilled hole off a graduated scale on the instrument (Courture et al., 2005). At the forefront of pin-point drill monitoring is directional drilling in oil and gas extraction. Some of the patents in this industry date back as far as 1972. Elwood (1974) describes (verbatim) how the system works : “The invention relates to a new and improved low-frequency drill bit locating and tracking apparatus and method of detecting the location of a drill collar as it moves through the earth by 65
transmitting and receiving a low-frequency electro-magnetic wave below 5,000 Hz. The transmitter is placed adjacent to the drill bit in the drill collar. The transmitter includes an alternator driven by a turbine in the mud supply conduit in the drill collar. The alternator functions as the transmitter for continuous operation during the drilling operation. The transmitter may also be a battery powered very low-frequency power oscillator. The drill collar is fitted with two electrodes or a wire loop connected to the transmitter output to cause the drill collar to act as an electrical or magnetic dipole. A plurality of directional indicating receivers with a magnetic or electric dipole antenna system is located in the low-frequency signal and the direction of the signal. A computing means is utilized to manipulate the data in order to plot the movement of the drill collar by triangulation. The method of operation is to place three receivers at known points with respect to the well head which acts as a reference for the system. The arrangement of the three receivers is basically an equilateral triangle with the well head in the center. All relative angles and distances between stations and the well head are measured and entered into the computer as a basis of computation. The transmitter is connected to the drill collar dipole for continuous operation during the entire drilling operation. The low-frequency electromagnetic signal from the drill collar dipole passes relatively undistorted through the complex inhomogeneous media to each receiver antenna. A vertical magnetic dipole comprising a small wire loop, buried or located at some depth beneath the earth’s surface, will produce an electromagnetic field on the earth’s surface if an alternating current is injected into the loop. The conductivities of the overburden, which may be quite complex, will modify the geometrical character of the vertical and horizontal magnetic field components observed at the surface. However, this effect is quite small provided the burial depths involved are small compared with the free-space wavelength, or electrical skin depth.” The drill tracking technology used in directional drilling in the oil and gas industry has been modified and applied to the mining industry. Drilling systems are at present available that allow automated alignment, reducing the reliance on and need for geologists and surveyors to align drill rig set-ups. The Azimuth Aligner® from Minnovare in Australia is an example of such a system. It uses military grade, north-seeking gyro technology, along with an autonomous, driller-operated, drill rig alignment system. The system can align a drill rig to tolerances within 0.1 degree of azimuth accuracy and dip accuracy, while also streaming “live” drilling information to a display unit (Minnovare, 2015). AR can also be applied when creating the blast pattern before drilling starts. The burden and spacing measurements can be inserted into a system and a virtual blast pattern could then be overlaid over the planned drill area (e.g. the overburden on surface or the face in a development end underground). This will greatly increase the accuracy with which hole spacings are placed 66
from one another and improve blasting efficiencies. The time to set up a drill pattern would also be significantly reduced as no marking would be required. By pressing a few buttons, drill rig operators or RDOs could see the exact location of each planned drill hole, thereby increasing the accuracy of hole placement. For drilling, an AR system can be developed to aid drill rig operators in achieving higher drilling accuracies and operator awareness by utilising aspects from directional drilling. This can either be a system as described earlier, which makes use of visual aids to help an operator to guide the drill inside a desired “virtual drill hole”, or AR can be applied to assist drill rig operators to use an automated system (such as the Azimuth Aligner) or to train them. In addition, AR could also be applied in drill rig setup procedures prior to drilling. Virtually overlaid measurements and alignment indicators could help greatly to decrease the setup time while improving the accuracy of the procedure. When it comes to “steering” the drill rod towards the desired accuracy, or “targeted” end of the drill hole, principles from directional drilling could also be applied. See Figures 3.1b and 3.1c for representations of a drill bit used in directional drilling. “Steerable drill ends” would yield higher accuracies with greater down-the-line benefits. Another method of controlling drilling accuracy in such a system might be to limit the amount of pressure that can be exerted on the drill rod to avoid excessive deflection. This limitation will be based on calculations for the geology the drill head finds itself in, which could also extend to realtime adjustments as drilling progresses (in this instance another system could be incorporated which measures the density of the rock, e.g. by measuring the vibrations in the drill head). An example of a system that is able to provide detailed feedback on the rock mass (in real-time as drilling commences), is the ROCKMA system (Rockma, 2015). AR could provide an integral means of combining these different technologies by assisting the operators to use the new system. The visual information that an AR device is able to deliver would greatly aid operators in becoming familiar with such a system, as well as guide them on how to use it efficiently.
67
Figure 3.1b: End of directional drill head (Schlumberger, date unknown)
Figure 3.1c: Cross-section of drill end used in directional drilling (Schlumberger, date unknown) Higher drilling accuracy leads to numerous downstream benefits (e.g. cost savings and improvements in production efficiency). Some examples are less wear on the lining of crushers and mills, reduction or elimination of secondary blasting, and optimal fragmentation (includes improved loading cycles, better fill factors, less material loss, less damage to conveyor belts, etc.). This leads to an overall reduction in disturbances in mining processes from mine to mill. All of this points to both cost savings and an increase in revenue from numerous different down-the-line beneficial implications. 68
SWOT Analysis A SWOT analysis is shown in Table 3.1 on the AR-based drilling application described above. Many of the SWOT factors involved are for AR applications in general and they will therefore be listed at the end under “SWOT Analysis on Augmented Reality Applications in the Mining Industry”. General factors are therefore not included under each individual application analysis. Table 3.1a: Strengths and Weaknesses Analysis for AR drilling applications Strengths
Weaknesses
Reduce or eliminate setup errors – when AR is Dependent on other technological systems, used as a visual guidance system for the setup devices or patents – An AR drilling application process (especially when coupled with a system needs to combine other technologies for it to designed for accurate drill rig setup), the setup work effectively. process could be done much faster and with greater accuracy.
High costs – the additional systems required may be expensive if the AR system is to be
Increased drilling accuracy – this leads to many highly effective. This could increase capital downstream benefits. Increased
drilled
metres
layout and possibly increase maintenance –
when
higher
accuracy can be achieved, the total depth of drilling can also be increased accordingly. A system such as the Azimuth Aligner (due to its high
accuracy
alignment
capabilities)
costs. Retraining or additional training for operators may be required when introducing a new technological application – the extent may vary.
can
increase total drilling metres by up to 10% alone (Minnovare, 2015). Reduced reliance on surveyors and geologists; lower personnel requirements – as described, the application could aid both in the setup and the drilling process, reducing the need for human assistance. Lower drilling costs – due to a reduction in personnel requirements, faster setup times, and faster all-round drilling, less secondary drilling requirements should be experienced.
69
Table 3.1b: Opportunities and Threats Analysis for AR drilling applications Opportunities
Threats
A reduction in secondary blasting could be Poor synergy between components – it is expected which would save on production possible
that
the
different
systems
(e.g.
costs.
directional drill head used to steer the drill rod,
Better fragmentation – leads to other associated
or a tracking system such as the Azimuth Aligner) do not work together effectively.
benefits as well (e.g. better fill factors and cycle times). Increased production efficiencies – decreased drilling periods paired with a reduction in time wasted due to poor drilling practices could enhance overall efficiencies. Better operator awareness – the visual aids of AR could help with operator awareness during all phases of the drilling process. Overall cost savings – from downstream benefits.
3.2
Navigational Aid and Operator Assistance
As seen in the literature survey section of this research study, AR has already found many uses for navigational purposes in vehicles. The overlaid virtual data could include a GPS system and additional information such as mileage, speed, fuel consumption, caller ID (even video call), etc., all in the form of AR while driving. These inputs can be combined with other navigational aids such as a clear AR-based view of that which an operator of a massive piece of equipment is unable to see from his cabin (such as smaller vehicles, people, the exact distance to the discharge point, etc.). Surface mines with high amounts of dust or rain generally have poor visibility for certain periods of time. It is possible to create an application that shows hazardous scenarios and objects, such as the side berm, other approaching vehicles, or distance to reverse to the dumping site or crusher. These navigational aids can be provided in the form of digital displays over the real-time (perhaps shrouded) view of the operator. Figure 3.2a indicates high dust conditions where an augmented view is provided of an approaching haul truck and the road boundaries. 70
Information that is hidden from the view of the operator could also be shown on an AR-enabled device. This could, for example, be in the form of a live camera feed of the rear and sides of the haul truck, by overlaying a window frame onto the real-time view of the operator.
Figure 3.2a: An augmented outline of an approaching haul truck Many other useful informational features can be overlaid to aid operators in their work. Some of these informational overlays include remaining fuel, current load, estimated duration until the next fuel refill or a timer displaying a count-down to the end of the operator‟s shift. These visual aids will help an operator to perform his tasks more efficiently and with increased accuracy and safety. 71
Figure 3.2b shows some of the informational overlays that can be retrieved (or “called-up”) by the operator on his AR-enabled eyewear (AR contact lenses are represented in this example, although it is a futuristic idea; however, many other AR devices could be used or adapted for use in this environment).
Figure 3.2b: Augmented overlays for operators Some of the informational displays could include the use of the piece of equipment or the efficiency of the operator (normally measured in loads per hours). This could encourage operators to increase their performance, especially where increased efficiencies are linked to a performancereward system. Different pieces of mining equipment could also be interlinked to enable real-time information sharing. This could facilitate improved information sharing between operators and equipment, which would ultimately aid them in their work. Figure 3.2c shows the information that a loader operator could see while loading a haul truck. The overlaid information could include details on the truck being loaded (e.g. the truck‟s identification number) as well as the real-time display of the truck‟s current load weight in relation to its maximum load capacity. A reminder of how many scoops have been discharged onto the truck can also be shown. This could aid loader operators to reach higher fill factors on haul trucks if they realise that previous bucket loads were under-filled. 72
Figure 3.2c: AR overlaid information during loading process An AR-based navigational system for mining equipment (primarily for haul trucks) could greatly increase safety and operator efficiency. Increased efficiency could further lead to higher utilisation and productivity. SWOT Analysis A SWOT analysis is shown in Table 3.2a on the AR-based navigation aid and operator assistance application described above. Table 3.2a: Strengths and Weaknesses Analysis for navigational aid & operator assistance Strengths
Weaknesses
Increase in safety – the visual aids and operator Large system requirements – the application guidance could greatly enhance the safety of described would require several other systems operating
mining
equipment.
By
virtually and technologies (such as GPS or other
highlighting, indicating and providing warnings tracking
systems,
distance
measurement
and/or showing the distances to hazardous systems, information senders and receivers, AR objects, other vehicles or personnel, many visuals along with the required hardware and dangers can be avoided.
complex software, etc.) in order to function as
73
Better operator awareness – unknown or hidden information can be shown with such an AR
system.
This
could
greatly
enhance
intended. Achieving synergy between these systems would pose its own challenges and have cost implications.
operator awareness and reduce the risk of High costs – the initial costs of such a large accidents.
system could be massive, and as with all
Enhanced operator efficiency – the assistance
implementations
which such an AR application could provide could enhance the efficiency of machine
of
new
systems/
technologies/equipment etc. a good trade-off study will be required.
operators due to the provision not only of Limited visual space – the amount of space that helpful but essential information.
can be overlaid with additional information is
Increased equipment efficiency – the individual
limited. This means that only a set amount of
efficiency of each piece of equipment can be expected to increase with the associated
Better fleet utilisation – the aforementioned application has the potential to increase the utilisation of the entire mining fleet (can be expanded beyond mere haul trucks and loaders as well). Each individual piece of equipment could experience higher utilisation and greater efficiency. By linking the different equipment and sharing information, the fleet could further
Fewer mishaps – accidents often occur due to or
momentarily
priority of importance. Difficult to filter through or call-up the required information – where most informational displays have some form of a trigger (e.g. GPS location or a “marker” which the software recognises), it is often necessary to request/retrieve specific information. Busy equipment operators might experience difficulty
be optimised and work together more efficiently.
attention
Additional beneficial information could be lost as information will be displayed based on
operator efficiency increase.
wandering
information can be displayed at any given time.
slow
memory recall. AR provides visual aids that could greatly reduce the risk of these mishaps. Operators (or many other AR users) are able to
time in
constraints
sifting
through
and
general
masses
of
information. In addition, it should be noted that data retrieval poses an entirely separate challenge on its own. One solution could be the incorporation of an eye-tracking system as described in the literature review section of this research paper.
glance at the virtual information to realign their thoughts and focus. This has the potential to reduce a variety of risks, including the risk of collisions.
A system is only as good as the software coding that runs it – If there is an error in the software coding, the entire system is inherently flawed.
74
Table 3.2b: Opportunities and Threats Analysis for navigational aid & operator assistance Opportunities
Threats
Higher functionality – since many informational Reliance on technology – operators could overlays are incorporated into the operators‟ become too reliant on the technology. Their view, their functional capacity could increase as own sense of intuition and logic could be dulled. a variety of tasks will be handled by the AR system
directly.
This
frees
up
thought
processes and enables operators to focus their attention on more pressing/complex matters. It is possible for the brain‟s processing to synchronise with AR applications. This enables operators to more accurately calculate typical working scenarios and to reach optimised and safer conclusions for many given situations. Increased productivity – increased operator and overall fleet efficiency could lead to downstream benefits such as increased productivity.
3.3
Maintenance and Repair
General maintenance and repair of equipment and conveyor belt systems can be conducted with greater efficiency through the utilisation of AR. This leads to improved maintenance and reduced repair durations, which in turn results in greater availability of equipment. An application can be designed to provide visual and audio assistance for maintenance tasks in the form of a tutorial or guidance system. Numerous types of information can be overlaid, including required pressure, torque or fluid levels, or guidance on “how to remove/replace”. An AR based software system is envisaged which sources its information from a server located on a mine‟s premises. This system could provide the following:
Guidance on maintenance and repair of the primary and secondary equipment of a mine. This would be in the form of: o
Tutorials, comprising digitally overlaid information, images or videos of how-toguides when conducting maintenance or repair work; 75
o
Manuals in the form of digitally overlaid information that can be drawn from the local server. These could contain information on the equipment and its parts as provided by the suppliers (e.g. scan the pdf on site, at machine); or
o
Real-time information, such as video chats with experts to aid in the task at hand, or even with the maintenance experts of the equipment suppliers (e.g. Caterpillar or JoyGlobal).
Such an AR software system could also be linked to the appropriate lock-out system. The mineworker working on the piece of equipment could then see, on his AR-enabled eyewear, whether the piece of machinery/equipment that is being worked on has been locked out or not. This would further increase safety by providing a final warning within the user‟s line of sight when the equipment has not yet been made safe for labour to commence. Such a system could be expanded to include a magnitude of other details, some of which include:
Inspection of equipment and/or machinery. This would be for specific applications, such as haul truck inspections, haul truck tire inspections, or conveyor belt inspections, etc. The purpose would be to prevent or reduce unwanted failure/breakdowns, and to improve availability.
Maintenance of other types of machinery such as conveyor belts, pumps and pumping systems, piping systems, ventilation shafts, headgear, winders, or processing plants, etc.
Conducting tasks or real-time correspondence with other employees or selected external experts (such as artisanal work).
Other integral maintenance tasks directly related to available production time. An example is pick changes on a continuous miner to increase the cutting time and production for the CM.
AR-assisted guidance for these integral activities could increase the efficiency of the maintenance or repair activity, availability of the system and the associated production output. It could also assist with familiarisation of new equipment or technology in order to reduce training periods of personnel where they lack the experience to maintain the new equipment/technology. Figure 3.3 is an example of such an AR application which could assist with the maintenance of equipment.
76
Figure 3.3: Example of an AR app for maintenance of a haul truck (simfusionar, 2015) Another aspect which could be incorporated is “pro-active inspections”, which can be done with an application programmed to guide the user by indicating what visual cues he/she should be looking for. An example would be to highlight, via digitally overlaid data, certain areas that require inspection. In addition to this, the application could include pictures or video guides of both correct and incorrect examples for each key location. This could help reduce breakdowns or unscheduled maintenance activities. Some of the areas where a pro-active inspection approach could be applied include conveyor belt systems, piping systems, ventilation fans, metallurgical plants, haul truck tyres, and other general primary and secondary mining equipment (LEDs, shovels, trucks, etc.). SWOT Analysis A SWOT analysis is shown in Table 3.3 on the AR-based application for maintenance and repair of mining equipment.
77
Table 3.3a: Strengths and Weaknesses Analysis for maintenance and repair Strengths
Weaknesses
Increased task efficiency –guidance provided by Specialist knowledge required – to create such the described application could enhance the a maintenance system, specialist maintenance task efficiency of the user (mechanic doing the knowledge would be required (generally offered maintenance, or artisan working on a piping by the
equipment manufacturers
to their
system, etc.) by showing clear-cut procedures clients). Sourcing this information and capturing for the task at hand.
it on a software-driven system could be a
Shorter maintenance and repair periods – with
challenging and costly exercise.
higher task efficiency it can also be expected A system is only as good as the software that a reduction in task completion times will coding that runs it – if there is an error in the follow.
coding, the entire system would be flawed.
Faster training periods – the duration of training Reduces human skill improvement – if people periods could be reduced if much less focus become dependent on a technological system needs to be placed on memory and experience, they are unable to grow and improve their and
more
emphasis
competencies
and
can
be
placed
utilising
on personnel skills much. With a specialist task
different such as mining equipment maintenance, the
technological systems. Lower
risk
of
poor/wrong
required human expertise (and the growth practices
or
thereof) will still be essential.
procedures – when providing step-by-step Dependent on human interaction – a specialist guidance on how to complete tasks, it can be AR system (or application) will be dependent on expected that the associated safety would humans to improve it. Specialists will therefore increase for both the correct and safest still be required for check-ups, maintenance, practices.
and upgrades of the system.
Reduced dependency on specialist mechanics – with an AR-based maintenance system as described, most of the specialist knowledge and experience can be contained within the system itself. This can then be carried over to nearly any mechanic with much less training required.
78
Table 3.3b: Opportunities and Threats Analysis for maintenance and repair Opportunities
Threats
Increased availability of machinery and other Blindly follow system instructions – users could parts or devices – better maintenance activities, blindly follow the instructions of the AR system coupled
with
a
pro-active
approach
to without questioning it. An error or malfunction in
inspections and maintenance activities could the coding could result in serious problems and lead to higher availability. Reduction
in
unplanned
possible mishaps. maintenance
or Inaccurate information input – it is possible that
breakdowns – The pro-active maintenance the information coded into such an application approach described above could also reduce could be inaccurate. This could either result unscheduled
maintenance
requirements
or from
miscommunications
breakdowns by detecting and correcting issues programmers beforehand.
and
the
experts
between (or
other
information sources) providing the information, or
from
people
purposefully
providing
inaccurate information. Regular changes in methods/procedures – regular updates/changes in the methods or procedures for the associated tasks would mean that the AR system could often become outdated. It is also possible that companies (e.g.
equipment
manufacturers)
could
deliberately change procedures, components, etc. to force such technological systems to become outdated.
3.4
Real-Time Information
This type of AR use has, quite possibly, the most diverse applications. Any real-time information imaginable can be overlaid over the real-world view of the user. Applications include virtual information, images, audio or video overlays on eyewear (or other AR devices). Some of the areas where AR can be applied will now be discussed. Hazardous situations, equipment or devices could potentially be detected with an AR-based system, for example, the user could identify dangers such as the possibility that an overhanging object might fall, or contraband close to flammable substances, etc. Such pre-emptive approaches could greatly reduce the risk of loss or damage to equipment and materials or injury to personnel. 79
Video tutorials and real-time examples/guidance (visual aids) could be provided, or video calls could be conducted where employees are able to receive on-the-job training. An example would be an artisan learning how to fit pipes or carrying out maintenance procedures on hydraulics while an expert in a different location provides him with real-time assistance. Such correspondence could involve experts working from home on a retainer basis, or across borders. This could be further applied in other communicational situations, such as communication between different managerial levels. For example, a CEO could witness first-hand the conditions of a mine on another continent while consulting with a manager who is walking him through the mine with his own AR device. Additional helpful information could be overlaid in real time, for example the amount of torque required to fasten a certain nut or clamp on a specialised device. In addition, useful information could be displayed during inspections, such as on a haul truck tire for “pro-active maintenance”, as shown in Figure 3.4a. Hidden details can be displayed via AR as well. Examples would be the water level in a tank/reservoir, the direction a fan or other rotating device is turning inside a casing, or simply the length of the bolts that fasten the radiator on a certain haul truck. This could act as an “X-ray vision” of sorts, to enable users to see real parts and their locations despite being hidden directly from the human eye.
Figure 3.4a: AR information on a haul truck tire specifications Real-time information ties in with just about every AR application, including operator assistance as previously described. This could be extended to any vehicle on a mining operation to provide 80
workers with helpful (and potentially lifesaving) information. Figure 3.4b is an example of how an AR system could be combined with a proximity detection system to provide a warning signal to a driver. A distance measurement between two following vehicles could then also be shown, along with the minimum allowable following distance as per the mine‟s SOPs.
Figure 3.4b: Real-time AR information in vehicles An AR system for overlaying real-time information could be linked to numerous other technological or software systems. One such example is shown in Figure 3.4c, where an AR system detects an error with a backhoe loader. The operator is provided with information to inform him that an error has occurred with his machine. He is then able to halt his actions to avoid further possible damage to the machine. Following this the AR system could direct the operator to the software system of the machine itself to identify the problem. Additional features could be added to a point where the AR system provides step-by-step assistance to solve or diagnose the problem.
81
Figure 3.4c: An AR display of information on mining equipment Information on employees could also be embedded in their individual clock cards (or another identification system used by a specific mine for its employees). When a more senior employee, such as a manager, chooses to view a certain person‟s information, he can do so with his AR device (such as AR-enabled glasses or a normal smartphone) in real time. The information could include, among other things, the employee‟s name, proficiency, qualifications, section of work, medical history, time clocked in, and even his historical location data. The location data could be used for several purposes, including efficiency monitoring, security or real-time rescue operation coordination during emergencies. Real-time rescue operation coordination on surface could also be coupled with AR eyewear of the rescue team venturing underground. The team could be equipped with a system capable of guiding them through dust and smoke by providing augmented visuals, and ultimately rescue operations could be led to the locations of trapped or injured miners. The tracking information from the ID cards of mineworkers could serve as beacons that lead rescue operations directly to their locations. Combined with a computer system that is able to calculate the best route for the safest and fastest extraction, or simply communicating with the team on surface who are planning the route, rescue operations could be completed in a far safer and more effective manner. The Mine Improvement and New Emergency Response Act of 2006 forced mining companies in the US to install two-way communications and wireless systems by 2010 to keep personnel on 82
surface aware of mine workers‟ locations at all times. This created a drive for Real-Time Location System (RTLS) implementations. Even with the unpredictable and dangerous conditions in underground mines, it was reasoned that any tracking system was better than none at all (Evans, 2007). With two-way enabled tags, trapped miners can raise the alarm directly and relay their location instantly and directly to a control room. Furthermore, with certain “tags” (such as a radio beaconing system built into the ID card of an employee) now being able to access the Internet via nodes on the local network (in a mine), the tags can also communicate directly with location servers anywhere within the proximity of the mine. This makes remote monitoring via web portal increasingly practical and further extends the possible uses of mine-based WiFi (Evans, 2007). With safety and efficiency being compelling drivers for any mining operation, the rise of real-time tracking information and surveillance technologies has spurred increasingly innovative applications to track the work of employees and their locations. AR provides another innovative means of processing and displaying aforementioned systems‟ tracking data. SWOT Analysis A SWOT analysis is shown in Table 3.4 on the real-time information for mining that AR could provide. Table 3.4a: Strengths and Weaknesses Analysis for real-time AR information Strengths Instant
access
to
Weaknesses
information
–
the
AR Arranging the information that should be
applications described could provide instant displayed according to the highest priority, can access to information for mineworkers, right at be difficult – when the AR system described the workplace.
becomes
Information is in “real time” – it is shown in real
amounts of data, it will become a challenge to
value
and
contains
massive
be the highest priority.
tasks are being completed adds
big
get the system to know what the user deems to
time, which means it can be accessed while
Information
too
–
the
overlaid
information could add a great amount of value in the mining industry and lead to numerous
A system is only as good as the software coding that runs it – if there is an error in the software coding the entire system is inherently flawed.
benefits. AR applications can be adapted to function with and connect to numerous other systems and
Limited visual space – the amount of space that can be overlaid with additional information is
83
technologies – by using AR, different systems limited, which means that only a set amount of and technologies can be combined into new information can be displayed at a time. The systems/applications. AR
adds
value
information that could be beneficial or that is to
the
other
systems/technologies that it is linked to – the new system/application is then often able to add more
value
than
previous
individual
essential will therefore need to be shown according to priority. Whereas each individual application of real-time displays described above can be tremendously useful, bombarding a user with all of them at once would simply
technologies.
nullify all the benefits. AR could increase safety and also rescue operation efficiencies – some of the applications mentioned
have
the
potential
to
greatly
enhance both safety in the workplace as well as the efficiency with which rescue operations are
hardware and transmission system such as WiFi) are limited in their capacity to send and receive information. This means that the number and size of the files that can be
conducted. Can overcome long distances – long-distance communication
Size limitations – the systems (e.g. the
and
collaboration
can
be
achieved with AR. These experiences could be
displayed at any given moment will also have a limit. The information will have to be adapted accordingly
(e.g.
format,
size,
length
of
videos/tutorials, etc.)
far more personal (such as video calls) and collaborative than many existing technologies (such as the telephone). During such an AR communicational experience, visual markers (or other forms of information) could be relayed along with the verbal information.
Table 3.4b: Opportunities and Threats Analysis for real-time AR information Opportunities Higher task efficiency –
Threats
with useful AR Abusing the privacy of employees – when
information at the required time it is possible tracking systems are incorporated into an AR that the overall task efficiency of users could application, care should be taken not to abuse increase.
the privacy rights of employees.
Increase safety to zero harm – This would Information overload – the method of recalling become an even stronger possibility if artificial information, and/or the amount of information intelligence and object recognition capabilities displayed at a time, should be carefully 84
developed to the point where hazardous regulated. scenarios can be identified by an AR computer system.
3.5
SWOT Analysis of Augmented Reality Applications in the Mining Industry
A SWOT analysis is shown in Table 3.5 on potential AR-based applications that were found for the mining industry. Table 3.5a: Strengths and Opportunities Analysis for AR applications in the mining industry Strengths
Weaknesses
Fewer mishaps when completing tasks – often Colour blindness factor – AR in its visual form mishaps occur due to attention wandering or could be less effective in the presence of colour momentarily slow memory recall. The visual blindness or a similar condition. An AR aids that AR provides could greatly reduce the application has to be designed with these risk of these occurring as AR users are able to obstacles in mind. glance at the virtual information to realign their thought processes.
Job loss/reduction – with the implementation of technology
incorporation
and
applications
Expandable new technology – many of the (including AR), the use of warm-bodied user applications that exist along with those that are skills becomes increasingly redundant, resulting envisaged can be expanded and improved in staff reduction and job losses. upon. AR has many benefits far beyond the mere
application
of
visual
overlays.
The
maturing phase has not yet been reached as this is a very new technology, which means that many improvements can be expected. AR
Dependent on other technological systems, devices,
knowledge/information
source
or
patents – in most instances, AR applications have
to
be
combined
with
additional
technologies to enable them to work effectively.
brings what is often referred to as the “Virtual Sphere”, which has opened the way to a whole
technologies need to be acquired and combined
new platform of digital media applications. Consists
of
several
technologies
– the other systems, technologies, information,
and/or
systems – the fact that an AR application typically
consists
of
several
Moderately high overall costs when various
different
technologies means that the technologies are
etc. mentioned above are often expensive. R&D on improving AR application and the required hardware could also result in high costs at present.
often substitutable. It is possible to replace one technology or system with another in order to Internet or a local server access is required – in 85
increase the synergy between the various order to transfer the information that an AR components.
application or system provides, some form of
Faster task completion times – the assistance
Internet access (such as WiFi) or server located on a mine‟s premises will be required. This
and/or guidance from an AR application is able to greatly increase the rate at which people are
Can bridge long distances – long-distance and
other
IT
concerns
needs
to
be
considered and a typical dependency on the
able to complete their tasks.
communication
means
collaboration
can
be
achieved with AR. If the connection to a server or the Internet is strong enough, long distances
associated technologies can be expected. Additional hardware and software requirements – new hardware and software needs to be acquired for an AR application. In general, the software will also differ for each specific
would prove to be far less of a hindrance. Can bridge language barriers – through its visual nature, technologies such as AR and VR are able to bridge language barriers by
application. IT systems that are already in place will be able to fulfil some of the requirements; however, in some instances additional capital outlay can be expected.
providing clear-cut visual (or other sensory based) messages. Along with text recognition capabilities, AR could greatly help overcome
Increased awareness – the visual aids could direct the attention of mineworkers to numerous issues and errors. This is even more true for awareness,
such
as
can be overlaid with additional information is limited, which means that only a set amount of
issues relating to multi-lingual environments.
operator
Limited visual space – the amount of space that
equipment
operators (e.g. primary production equipment or
information can be displayed at a time. The information overlay should further be kept to a prescribed minimum (additional research might be necessary to determine this minimum) to avoid user view obstructions which could lead to distractions and information overload.
drill rig operators). Instant access to information – AR applications could provide instant access to information for mineworkers right at the workplace.
A system is only as good as the software coding that runs it – if there is an error in the underlying system code the entire system would be flawed.
Information is in real-time – the information shown is in real-time, which means it can be accessed while completing tasks.
Reduces individuals
human become
skill too
improvement
–
if
dependent
on
a
technological system they might be unable to
Can work with and connect numerous other grow and improve their personal skills and systems and technologies – by using AR, competencies in an effective manner. different systems and technologies can be
A computer system is unable to improve and
86
grow like a person – a computer system
combined into new systems/applications. Adds value to the other systems/technologies that it is linked to – in most instances the new system/application is able to add more value for end-users
than
the
system/application.
previous
Interactive
standalone technological
applications – AR applications bring a new level and to, means of interaction with, the virtual (or
(considering the current position of artificial intelligence in the public domain) is unable to improve upon its own skills as a warm-bodied user would. A developed AR system would, for the most part, be dependent on human influence
and
intervention
to
facilitate
improvements and upgrades.
digital) world. The ability to manipulate and Retraining or additional training requirements – interact with virtual elements for seemingly it might be required to develop and implement unlimited purposes brings a new approach to various personnel retraining courses when task completion and technology usage.
introducing a radical or complex technological
Better memory retention – The visual and
application. The extent of training may vary
interactive nature of the technology allows far better memory retention and facilitates faster,
risk
of
inaccurate
or
wrong
perceptions – by using visuals in the workplace, the
risk
of
skewed
perceptions
and
misinterpretations can be significantly reduced. Less needs to be left to the imagination and
Elimination or reduction of mundane tasks – many AR applications are able to eliminate or mundane
tasks,
which
systems of most applications (e.g. the hardware and the transmission system such as WiFi) are limited in their capacity to send and receive information. This means that the number and overall size of files that can be displayed at any given moment will also be limited by storage
much more can be visualised directly.
reduce
the overall complexity of the AR system. Size/storage limitations – the components and
improved grasping of concepts. Reduced
depending on current staff competencies and
increases
and transmission limitations. Hardware limitations – the applications and way in which AR can be used is subject to various hardware limitations. Examples include the
capacity for more important functions and tasks. Increased task efficiency – an increase in task efficiency can be expected where an AR application has been designed to aid the user in
quality of regularly available cameras, and the display quality on devices that are also practical for the working environment (e.g. PPE goggles in mining operations).
completing his/her work. This could be for the specific task targeted by the application, or it could extend to other related tasks as well (e.g. when a haul truck driver receives AR navigation and
assistance,
the
navigation
aimed
at
Dependent on human interaction – a specialist AR system (or application) will be dependent on humans to improve it. Specialists will therefore still be required for check-ups, maintenance and
87
improving his driving efficiency and safety might upgrades for the system. improve his overall productivity as well). Reduced
operating
–
costs
many
AR
applications could reduce running costs and other downstream costs. This is due to increases
in
productivity,
personnel
and
equipment efficiency, and general safety, all of which reduce overhead costs. Reduced
labour
–
requirements
the
incorporation of technology into everyday tasks, coupled with an increase in personnel efficiency could reduce the number of personnel required in certain areas (e.g. mechanics, artisans, etc.). Reduced
reliance
personnel/contractor
on –
skills
specialist many
AR
applications aim to reduce the complicated nature of certain tasks while providing guidance on other more intricate tasks. Some of these systems could reduce the reliance on (or replace/substitute) the specialist skills for which they aim either to provide aid or “how-to” guidance. Reduced risk of human error – The clear-cut guidance and task assistance reduces the risk of human error during task completion.
Table 3.5b: Opportunities and Threats Analysis for AR applications in the mining industry Opportunities Higher task efficiency –
Threats
with useful AR New/foreign concept – AR might be foreign to
information at the required time, it is possible many mine workers and as a result, the that the overall task efficiency of users could technology could experience difficulty being increase. Higher
brain
accepted. People may treat AR applications functionality
–
since
many
with distrust, similar to when the computer first
88
informational overlays are incorporated into the came into use. view of the AR user, their functional capacity could increase as a variety of tasks will be handled directly by the AR system. This frees up thought processes and enables operators to focus their attention on more pressing/complex matters. It is possible for the brain‟s processing to synchronise with AR applications. This enables operators to more accurately calculate typical
working
scenarios
and
to
reach
New ground/unknown territory – because of the relative newness of the technology (especially in mining), it is possible that unforeseen obstacles
can
implementations. compatibility
be An
issues
encountered example with
would the
be
mining
environment (e.g. if tablet PCs or smartphones need to be used in a rough and dirty environment).
optimised and safer conclusions for many given AR
situations.
with
often
means
radical
change
–
AR
applications, as can be imagined, might bring Increase safety to zero harm – if artificial intelligence and object recognition capabilities develop to the point where hazardous scenarios can be identified by an AR computer system, zero harm might become a reality.
radical changes to the environment it is introduced into. Many people may oppose this change, especially if they feel it will replace them. Technological incompetency – although people might want to learn and use an AR application in one form or another, it is possible that a lack of understanding and learning capability might hinder the use, or reduce the effectiveness, of the technology. Too much reliance on technology – people could become too reliant and dependent on the use of a technology such as AR. This could dull their own sense of logic, intuition and problemsolving abilities. The opposite could, however, also be true. Poor synergy between components – since AR often incorporates different components and/or technologies, the possibility exists that not all match-ups will work together with optimal synergy.
89
REFERENCES Courture, P. et al. 2005. Patent US6887247 - CAS drill guide and drill tracking system - Google Patents. [ONLINE] Available at: https://www.google.com/patents/US6887247. [Accessed 27 September 2015]. Elwood, A. 1974. Patent US3828867 - Low frequency drill bit apparatus and method of locating the position of the of the drill head below the surface of the earth - Google Patents. [ONLINE] Available at: https://www.google.com/patents/US3828867. [Accessed 20 September 2015]. Evans, G. 2007. The Real Deal - Mining Technology. [ONLINE] Available at: http://www.miningtechnology.com/features/feature1436/. [Accessed 25 September 2015]. Minnovare - Azimuth Aligner. 2015. Laser Alignment System - Mining Industry | Minnovare. [ONLINE] Available at: http://minnovare.com/azimuth-aligner/mining-industry/. [Accessed 20 September 2015]. Rockma. 2015. Rockma. [ONLINE] Available at: http://www.rockma.se/en/ . [Accessed 29 September 2015]. Simfusion. 2015. Simfusion | Augmented Reality and Virtual Reality custom applications. [ONLINE]
Available
at: http://www.simfusionar.net/.
90
[Accessed
24
September
2015].
CHAPTER 4: CONCLUSIONS It was identified in this research study that augmented reality (AR) has found multiple applications within a diverse range of environments. Further possible applications and futuristic ideas were also investigated in order to obtain a concise understanding of the technology and how it can be practically applied. Following this, it was identified that the mining industry, especially in South Africa, has been lagging behind in finding applications to utilise this sensory (more commonly known as visual) based technology. The visual aspect of AR was then explored to generate ideas for potential AR applications in the mining industry. These ideas stemmed either from independent thinking, or were inspired by the literature study of applications in non-mining areas. Three existing AR applications were highlighted which had already been implemented in the mining industry. During the research twelve potential mining application ideas from various independent thinkers were also found. Following this research, an additional eighteen potential applications, either futuristic or currently achievable, were generated. Refer to Table 4a for a summary of the findings from the literature study. Table 4a: Summary of findings from the literature study Existing AR applications identified for the mining industry
AR-based assistance for Maintenance of Mining Machines (ARMM): o
Includes viewing AR versions of equipment on handheld computers and HMDs.
o
Scrolling through different AR visuals of the machines and their components.
o
Recalling historical and present data on the condition of machines, in real time, as augmented overlays.
o
Receiving guidance on how to repair or maintain separate components.
o
Combining wear and tear detection systems with AR to create information-rich, realtime visualisations of the machinery and its status.
PerfectDig AR system – compares laser-scanned mine surface data against 3D AR mine plan designs for conformance measurement.
SHEQSCAN, an integrated audit and inspection AR tool. Also offers various digital media to specifically assist in training the user to become competent in any complex, real-world task or situation. 91
Some potential mining AR applications that had already been identified by various people
Emergency support. Use of AR-enabled mobile nodes for the rescue team. The nodes function as position detection devices. Information regarding the node being tracked can then be overlaid on a display device. Combined with a rapid deployment communication system with reliable communication and data transmission.
The use of RFID technology for automated identification of machine parts or assemblies, along with people for maintenance and repair processes (ties in with ARMM).
Integration of information and navigation capabilities with AR-based systems for informational and navigational aid.
AR systems that will allow people to look into the ground and see underground features. These features could be major geological structures, gas or water pipe layouts or zones of contaminated land.
AR-based system for visual aid and guidance for heavy machine operators.
AR assistance for accurate drill hole positioning. Will reduce different transits required between planning and the drill site.
AR-assisted “art work” to assist mining technicians or engineers with designs and planning.
Tracking of planning to ensure that mining boundary points are in place when they are moved. In short, AR geotechnology conformance visualisations.
Digital instructions. The concept is step-by-step instructions that are overlaid via an AR display device on the actual item a worker is working on: o
Maintenance, repair and overhaul tasks.
Remote collaboration. This aspect correlates to shared real-time views and helpful augmented overlays between someone performing a task and another providing guidance: o
On-site AR guidance for heavy or fixed machinery mechanics, between technicians and experts via a remote system.
AR-based training. Believed to provide opportunities for making training more effective, realistic and cost effective.
Incident investigation. This entails on-site reconstruction of major incidents with AR visuals in order to determine what the root causes were that led up to the incident.
Potential AR applications identified for the mining sector, based on other types of 92
applications investigated in the literature study
Augmented displays of an entire underground mine can be generated, with the ability to zoom in on different sections with accurate details. This can then be viewed while standing on the surface or sitting in a boardroom. The visual aids of AR can greatly help in planning and grasping of layouts without having to physically visit every section.
AR-based views of obstructed underground factors. Expanded to an all-round view when underground, in order to observe the hidden details within the host rock. The geological features, ground conditions, rock types, discontinuities and other relevant information can be displayed in the form of AR holograms or 3D models.
Display of mining site plans. Can be used for planning during the development phase. Also used for visual representations when presenting to stakeholders or when applying for a mining licence. A complete walk-through can be conducted in an augmented environment as if the mining site had already been constructed around the viewer. Can also be used as a model to solidify a commitment in terms of design methodology or treatment processes, etc.
Recreating historical mine sites or equipment, or even a holographic display of a historical mine in operation.
Recreating historical events, such as incident recreation for investigation purposes. Can be done on site with minimal disturbance to the site of the incident.
Combining eye-tracking technology with AR technology to create displays or issue commands for numerous purposes. The purposes may include operators interacting with an AR system for guidance, navigation, support functions or requesting information. May also include sending out signals, e.g. to transfer information regarding warnings or production parameters, or sending out distress signals.
Augmented views of personnel ID cards to provide additional AR overlaid information. Levels of access to different information can be controlled based on the hierarchy difference between the viewer and the person being scanned. Examples may include general personal info along with section/area of work, expertise, superior and subordinates, experience and even personal ratings.
RFID tracking can also be applied to both equipment and ID cards for monitoring purposes. The location history for these can then be recalled in an AR display when and where required.
93
Mining materials, equipment, components and consumables can be visualised on site (in the form of 3D AR visuals, such as full-scale models). This can be used to accurately determine the applicability, level of accuracy and conformance of an item, prior to placing orders.
Direct visual communication with AR-aided interactions between the parties can be established. This can be used for precise communication between different stakeholders, which can include communication between management, with suppliers, or between trainees and experts as if the parties were standing next to one another in real time.
AR-based training programmes, such as emergency procedures, operator training, task performance, etc. The visual nature of the technology can greatly aid understanding and reduce training/educating durations. Concepts can be grasped quicker and more accurately, with less room for inaccurate perceptions.
Combining tracking systems (such as RFID and personnel ID card tracking) with mine layout augmentations can aid rescue operations. A system can be developed to provide real-time assistance for the best course of action, escape routes to take or ways to navigate towards those in need. Alternatively these applications can be used for optimal human-generated planning for the same scenarios.
AR-aided visual aids that not only enhance and amplify vision, but also provide augmented information on details that the human eye is unable to detect.
Mining games or software development that promote at home “play” and/or easier mine planning and design in the professional environment. Examples include designing mines, playing with configurations and layouts, designs, equipment matching, etc.
AR-constructed crash tests. For modelling purposes of support, designs, material selection, etc.
Linked and shared AR visualisation can be established when required. The sharing of information and visuals will provide better collaboration between personnel, departments and different specialist roles. Better correspondence can be achieved with real-time updates.
AR-based navigation systems. This could provide increased vision for operators in conditions of poor visibility (such as stormy winds that create dust clouds over surface mining operations). Virtual indicators can indicate the edges of roads, berms and sidewalls as well as other equipment or people. The maximum distance to reverse when discharging
94
a load can also be shown accurately.
Integration of AR-aided language translations (both voice and text) for increased communication efficiency. This is accomplished with object (text) recognition software and will aid international mining companies to breach language barriers.
Four of the aforementioned ideas were chosen that were deemed to have the highest potential of adding value, while being practical to implement at present. The likeliness of successful implementation was based on current readily available technology that could hinder either the adoption of a “new” technology, or the practicality of designing a multifarious technological system. The four identified potential AR applications were broadly classified as Drilling Applications, Navigational Aid & Operator Assistance, Maintenance and Repair and the provision of Real-Time Information. These were then elaborated on. A description of how such systems could add value to mining and how they would function was provided. These four concepts were further expanded into multiple sub-applications, each of which would have the potential for practical implementation as a value-adding application. Table 4.b summarises some of the most prominent identified potential applications and sub-applications. Table 4b: Summary of findings from the results Identified potential AR applications, categorised into four main groups, each with various sub-applications Drilling Applications
Digital overlay of drill/blast pattern.
Instructions and assistance with drill rig set-up (existing technologies can be used, AR could be incorporated into their use).
Real-time visuals as drilling commences to track drill rod (existing technologies can be used for the physical tracking).
Steering of the drill head (existing technologies can be used and AR could provide assistance in terms of how to manoeuvre the drill head).
The means of combining the different systems by providing AR visuals for training and operator assistance.
95
Navigational Aid & Operator Assistance
AR visuals for GPS navigation.
Numerous helpful informational overlays (e.g. operator efficiency, fuel capacity, live camera feeds for blind spots, digital outlines of hazardous objects or other equipment, etc.).
Efficiency monitoring and guidance for optimisation.
Maintenance and Repair
Guidance for maintenance and repair tasks on mining equipment – includes digitally overlaid information such as tutorials, video or image guides, digital manuals for equipment and parts, real-time correspondence (e.g. video chats on the working site), etc.
For additional safety the envisaged system could be linked to lock-out systems in order to provide a warning if equipment/machinery is not safe to work on.
Pro-active inspections with AR instructions, to improve availability of equipment, machinery or other systems.
Real-time Information
Hazardous scenarios, equipment or devices that could be detected with an AR-based system, or an example could be provided which allows the user to identify such dangers on the working site.
Real-time examples/guidance for on-the-job training.
On site correspondence with external (knowledgeable) persons for assistance, communication or training purposes (coupled with AR visual indicators for interactive collaboration).
Overlay of helpful information to aid in working practices – could also include information hidden from the eye to act as “X-ray vision”.
Visual overlays for danger warnings or other breaches of SOPs.
Can be combined with tracking technologies to map historical location data of employees. Can be used for efficiency monitoring and the optimisation of real-time rescue operations.
Rescue operations could be extended to the emergency response team, where team members could receive valuable information to aid in their success (such as the shortest/safest route to trapped mine workers, rate at which fire is progressing, dangerous gases in the air, etc.).
96
A SWOT analysis was done for each of the four main application groups to identify the possible challenges and benefits of each for mining. Lastly, a general SWOT analysis was completed on the application of augmented reality in the mining industry, with some of the main findings listed in Table 4c. Table 4c: Summary of SWOT analysis of potential AR applications for the mining industry Strengths
Weaknesses
Colour blindness factor.
tasks.
Job loss/reduction.
Expandable new technology.
Dependent
Consists of several technologies
Fewer mishaps when completing
systems,
Faster task completion times.
Can bridge long distances.
Can bridge language barriers.
Increased awareness.
Instant access to information, right at
Information is in real time.
Can
Adds
with other
value
and
connect
systems
and
to
the
other
to.
Predicted moderately high overall costs
Internet
or
local
server
access
is
required.
Additional
hardware
and
software
Reduced risk of inaccurate or wrong perceptions. Elimination or reduction of mundane tasks.
Reduced operating costs.
Reduced labour requirements.
System is only as good as the software coding that runs it.
Reduces human skill improvement.
Computer system is unable to improve
Retraining
or
additional
training
requirements.
Better memory retention.
Increased task efficiency.
Limited visual space in human field of vision.
and grow like a person.
Interactive technological applications.
knowledge/
requirements.
systems/technologies that it is linked
devices,
acquired and combined.
technologies.
technological
when various technologies need to be
the workplace.
numerous
other
information source or patents.
and/or systems.
work
on
97
Size/storage limitations.
Hardware limitations.
Dependent on human interaction.
Reduced
reliance
on
specialist
personnel/contractor skills.
Reduced risk of human error. Opportunities
Threats
Higher task efficiency.
New/foreign concept.
Higher brain functionality.
New ground/unknown territory.
Increase safety to zero harm.
AR often means radical change.
Technological incompetence.
Too much reliance on technology.
Poor synergy between components.
The general conclusion is that AR could bring numerous benefits to the mining industry and greatly add value and efficiency to the mining process. Careful investigation would, however, be required as to how an application should be designed in order to ensure an efficient system that would deliver the required benefits while minimising the associated challenges.
98
CHAPTER 5: RECOMMENDATIONS All of the identified potential applications for AR in mining stemmed from idea generation and are concepts which can be developed further. Each of these requires extensive investigation before they can be practically implemented. It is therefore recommended that a thorough investigation be conducted on each of the concepts provided before attempting to create a real system or software application. Further research into the psychological effects of technologies such as AR on the human mind is advised. Very little is currently understood of the impact of using AR for extensive (e.g. eight hours per day) periods. As was seen in the SWOT analysis of AR applications in the mining industry, it may be possible that extensive use of this reality-shaping technology could affect the human mind in one of two (perhaps more) ways. One possibility is a detrimental effect, where people become too reliant on technology. Their own sense of judgement and ability to make accurate calculations or predictions could suffer. The other possibility is that the human mind could reach a higher state of functionality as an AR application is capable of providing different forms of aid in numerous environments. The last mentioned involves the human brain “synchronising” with the technology. This could ultimately lead to increased processing capacity due to an AR system taking over certain processing requirements (e.g. mundane calculations or reminders). For safety reasons it is recommended that extensive research be done on the psychological effects to determine how the threats mentioned above can be avoided and the potential opportunities exploited.
99
CHAPTER 6: SUGGESTIONS FOR FURTHER WORK The results section contains some of the identified AR applications with the highest potential to add value to the mining industry at present. It is suggested that further investigative work should first be done on these applications to take them from concept to practical implementation. Most of the applications described contain sub-categories of other potential applications and each should be investigated individually. It is the author‟s opinion that each application mentioned in this study should be investigated in detail before physical implementation is attempted.
100
