Real-Time Location-Based Rendering of Urban Underground ... - MDPI

International Journal of

Geo-Information Article

Real-Time Location-Based Rendering of Urban Underground Pipelines Wei Li 1,2 , Yong Han 1,2, *, Yu Liu 1,2 , Chenrong Zhu 1,2 , Yibin Ren 1,2 and Ge Chen 1,2 1

2

*

ID

, Yanjie Wang 1,2

College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China; [email protected] (W.L.); [email protected] (Y.L.); [email protected] (C.Z.); [email protected] (Y.R.); [email protected] (Y.W.); [email protected] (G.C.) Laboratory for Regional Oceanography and Numerical Modeling, Qingdao National Laboratory for Marine Science and Technology, Qingdao 266237, China Correspondence: [email protected]; Tel.: +86-0532-6678-6365

Received: 29 October 2017; Accepted: 18 January 2018; Published: 21 January 2018

Abstract: The concealment and complex spatial relationships of urban underground pipelines present challenges in managing them. Recently, augmented reality (AR) has been a hot topic around the world, because it can enhance our perception of reality by overlaying information about the environment and its objects onto the real world. Using AR, underground pipelines can be displayed accurately, intuitively, and in real time. We analyzed the characteristics of AR and their application in underground pipeline management. We mainly focused on the AR pipeline rendering procedure based on the BeiDou Navigation Satellite System (BDS) and simultaneous localization and mapping (SLAM) technology. First, in aiming to improve the spatial accuracy of pipeline rendering, we used differential corrections received from the Ground-Based Augmentation System to compute the precise coordinates of users in real time, which helped us accurately retrieve and draw pipelines near the users, and by scene recognition the accuracy can be further improved. Second, in terms of pipeline rendering, we used Visual-Inertial Odometry (VIO) to track the rendered objects and made some improvements to visual effects, which can provide steady dynamic tracking of pipelines even in relatively markerless environments and outdoors. Finally, we used the occlusion method based on real-time 3D reconstruction to realistically express the immersion effect of underground pipelines. We compared our methods to the existing methods and concluded that the method proposed in this research improves the spatial accuracy of pipeline rendering and the portability of the equipment. Moreover, the updating of our rendering procedure corresponded with the moving of the user’s location, thus we achieved a dynamic rendering of pipelines in the real environment. Keywords: augmented reality; underground pipelines; occlusion method; differential correction; Visual Inertial Odometry; dynamic tracking

1. Introduction 1.1. Background Urban underground pipelines are known as “urban blood vessels”, consisting of a series of pipelines for water supply, drainage, gas, heat, electricity, communication, television, and industry [1]. Reasonable planning and effective management of underground pipelines guarantee the normal operation of urban functions. Fortunately, the rapid development of Geographic Information System (GIS) technologies, such as network analysis, topology validation, and spatial mapping, provides reliable support for intelligent pipeline management, analysis, and planning [2]. However, with the acceleration of urban construction, an increasing number of pipelines buried underground are ISPRS Int. J. Geo-Inf. 2018, 7, 32; doi:10.3390/ijgi7010032

www.mdpi.com/journal/ijgi

ISPRS Int. J. Geo-Inf. 2018, 7, 32

2 of 17

forming a concealed and complex spatial network system, which brings new challenges to traditional pipeline management. As an extension of GIS functions, Virtual Geographic Environments (VGEs) can present representations and visualizations of augmented reality (AR) that are interactive, immersive, and close to nature [3]. Recently, VGE has been applied to various fields, such as simulating and analyzing air pollution [4], collecting real-time dynamic traffic information [5], and measuring noise levels [6]. To perceive and understand the huge body of pipeline data, using a form of VGE to represent it will generate a feeling of actually being there. As one of four subtypes of VGE, expressing the environment gives the opportunity to observe the virtual environment with tools such as helmets and gloves [7]. Simulations made by personal computer based VGEs are unable to provide timely and intuitive references for decision-making. Therefore, it is crucial to develop new methods with new technologies to improve the efficiency of managing underground pipelines. Through computing a device’s real-time position and posture, AR can update the corresponding image model with the device’s perspective, aiming to combine the virtual world with the real world [8–10]. The rise of smartphones has provided favorable conditions for the development of mobile-based VGE, especially in mobile AR. On the one hand, mobile devices such as smartphones, HoloLenses, etc., are more convenient to carry, which makes them suitable as moving carriers of mobile AR systems. On the other hand, the popularity of high-quality mobile devices is growing rapidly, with many sensors such as gyroscopes and accelerometers integrated with stable cameras and powerful central processing units, providing a wealth of hardware to support developing AR software. When it comes to pipeline management, mobile AR is able to display on-site pipeline information vividly, intuitively, and in real time. This provides a new way for us to meet the current challenges in pipeline management. 1.2. Related Work Some research in recent years has proposed methods of providing “X-ray vision” to see where underground infrastructure lies. For the first time, Roberts et al. [11] combined Real-Time Kinematic (RTK) Global Positioning System (GPS) and an internal navigation system (INS) to allow robust and precise real-time positioning and orientation in an unprepared environment. This data was then used in the AR system to superimpose buried pipes and cables onto the real-world view of a head-mounted display. Schall et al. [12] designed a 3D scene-graph rendering on the client side, referred to as Geography Markup Language (GML), exported from the GIS server. Additionally, they integrated GPS, an inertial measurement unit, and an ultramobile PC into a handheld device to achieve relatively accurate real-time tracking and added some functions such as excavation, labeling, and filtering. They also explained the transcoding process from a GML encoding consisting of context and geometric properties to scene-graph (Open Inventor) visualization data [13]. Chang et al. [14] designed a system that can also work indoors, based on ARToolKit, where pipelines can be displayed on a laptop. They combined GPS and INS to obtain the outdoor position and orientation. Taking into account the accuracy and refresh rate of the GPS, it is more appropriate to adopt markerless tracking based on a visualization with higher outdoor accuracy. In recent years, with the development of AR, a series of products such as ARToolKit [15] and Vuforia [16] have been derived. The current main solution of AR is based on the marker: users need to paste a certain marker in the AR scene beforehand and obtain the transformation parameters between the marker coordinates and the camera coordinates [17,18]. Even if the position and posture of a mobile phone changes, the phone can maintain stable tracking, keeping the virtual object in the same position in the real world. On the other hand, the software development kit (SDK) [19] provides ordinary programmers with more freedom and flexibility to use their own inspiration to design excellent AR applications, even if they do not understand the algorithm implementation, such as AR red packet, etc. However, most of these SDKs must be based on simple image identification techniques, which cannot apply to complex scenarios.

ISPRS Int. J. Geo-Inf. 2018, 7, 32

3 of 17

The inevitable direction of future development is simultaneous localization and mapping (SLAM) [20] technology, which can build a model simulating the real environment through the background process based on panoramic 3D reconstruction. The model is then rendered to a display device after scene-graph fusion of the virtual world and the real world so as to achieve an interaction of the two worlds. Localization and mapping are the key functions of SLAM, which can be divided into two separate threads [21–23]. Motion tracking based on MonoSLAM [24] has problems such as extensive calculation, long work time, existing scale ambiguity [21], and difficulty detecting feature points when the device is moving fast. The integration of inertial measurement unit IMU [25] to get six degrees of freedom (6DOF) [26] of the device plays a complementary role in improving its refresh rate and accuracy. For example, Google Tango [27] and the updated ARKit in iOS 11, released in June 2017, utilize the Visual-Inertial Odometry (VIO) [28] method, which combines vision and inertia to gain 6DOF of the mobile device. The difference is that Google Tango achieves 3D reconstruction with hardware, such as a depth camera and a fisheye camera, while optimization based on the ARKit’s algorithms allows most iOS devices to have AR capabilities. Without depth perception, ARKit currently can only detect planes, which means it cannot achieve reconstruction of the environment, like Tango, or complete a complex environment interaction. 1.3. Problems Aiming to improve the efficiency of managing underground pipelines, based on an analysis of the existing AR underground pipeline systems, we must solve three key problems. The first is how to retrieve and load pipelines near the user accurately and conveniently. Location accuracy determines the effect of pipelines’ real-time loading, and the mobile device draws pipelines within its perspective referring to the user’s location, but it is worth finding a way that does not cause fatigue. The second problem is how to keep track of the model stably and precisely. Some methods are solely based on image registration in computer visualization or GPS positioning. If light changes result in a lack of feature points or a multipath effect occurs, it will not allow development of outdoor underground pipeline systems based on AR. The last problem is how to improve the immersive experience of underground pipelines in real time. Currently, due to the limitations of AR technology, the confusing relation of occlusion gives users the illusion of pipelines floating on the ground. 1.4. Our Idea As stated before, the application of AR to show underground pipelines can fully reveal its real-time and vivid characteristics. We can get people’s precise real-time locations based on the BeiDou Navigation Satellite System (BDS) [29], which can provide a relatively high-precision positioning service to mobile devices without having to carry large equipment. According to the user’s location, we can transmit pipeline data by the network; the mobile device then renders the pipelines after data conversion. Scene identification based on visualizations can further improve the accuracy of the pipeline position. At the same time, we can make some improvements to pipeline rendering. Feasible methods such as VIO can ensure robustness outdoors and improve the performance in a markerless environment. In addition, we can design some details to make the pipelines vivid, such as the shape of the buried pit, the occlusion of walls, and the texture of the round and smooth junction between the pipeline and the wall. Based on the above analysis, we propose an original method to load the data and render nearby underground pipelines in real time. To sum up, there are three novel contributions of this research: (1) we designed a method to load pipeline data based on highly precise positioning service and scene recognition, and transformed the plane coordinates and elevation into the data of Unity3D; (2) we adopted the VIO to track camera motion to make pipelines look stable; moreover, we studied the theory of world coordinates in Area Description File (ADF) and VIO, then designed a method to reduce drift; and (3) we applied occlusion building based on the real-time 3D reconstruction to the pipeline rendering and designed a method to extract the mesh of the ground that hides the pipelines, and we designed details of the pipeline.

ISPRS Int. J. Geo-Inf. 2018, 7, 32

4 of 17

The rest of this paper is organized as follows. Section 2 introduces the overall architecture of our system. Section 3 introduces the key methods in detail. Section 4 shows tests and analyses of the experiments performed. The last section gives our conclusions and future research directions. 2. System Design 2.1. Hardware Composition Our system adopts Google Tango Phab 2, which has a 16 million pixel camera and integrates the fisheye camera to capture its own motion information. The fisheye camera ensures an angle of view wide enough to catch more environmental features, which makes model tracking more stable. The device is also equipped with a depth camera, which can launch and receive infrared structure light. By working with the sensor algorithm, the mobile device can obtain the object’s distance and size. It can also support 3D reconstruction of the surrounding environment [30,31]. Instead of RTK, the handheld differential device is connected to the mobile phone via Bluetooth to calculate the differential corrections received by the phone, which is the same size as a cellphone and can be easily held with one hand. 2.2. Data Design As shown in Table 1, the raw measured data exported from the GIS server to the client according to the position information mainly includes spatial and attribute data. The spatial data is used to transform the position, angle, scale, and texture of the game object in Unity3D, and the attribute data is used for data retrieval. We transformed the geographic coordinates into world coordinates through seven-parameter coordinate transformation [32]. The field of the material corresponds to the identification of the cached texture map in the system. Finally, we drew pipelines to the mobile device according to the information. Table 1. Structure of pipeline spatial data. Field

Type

Mean

Id Start_lng End_lng Start_lat End_lat Start_ele End_ele Start_dep End_dep Diameter Material

int float float float float float float float float float varchar

Pipeline number Start point longitude End point longitude Start point latitude End point latitude Start point elevation End point elevation Start point depth End point depth Pipeline diameter Pipeline material

2.3. General Framework As shown in Figure 1, the real-time detection system of underground pipelines draws nearby underground pipelines based on the user’s location. The specific steps are as follows: 1.

2.

The mobile device sends the final coordinates calculated from the differential corrections to our data server. According to the user’s location, the server transmits the pipeline data around the people and the corresponding ADF [27] to the mobile phone. The mobile phone transforms the latitude and longitude coordinates into the world coordinate data of Unity3D and draws the pipelines. Then it loads the ADF. After identifying the scene, it adjusts the pipeline position.

ISPRS Int. J. Geo-Inf. 2018, 7, 32

5 of 17

3.

In static mode, the mobile device achieves tracking of the model by calculating its own position using the VIO algorithm. In dynamic mode, we complete the dynamic rendering by the camera orientation and position output by the VIO. 4. According to the 6DOF of the device, the phone registers the point clouds obtained by the RGB-D sensor to complete the 3D reconstruction of the environment, and the mesh of the ground in the real is 2018, split.7, 32 ISPRS Int.world J. Geo-Inf. 5 of 17 5. The system attaches the 3D reconstruction rendered by the transparent occlusion texture to the 5. pipeline The system attaches the 3D reconstruction rendered by the transparent occlusion texture to the model. pipeline model.

Figure1.1. Architecture of the with solutions for the three proposed: we (a) locationFigure Architecture of system the system with solutions for problems the threeweproblems proposed: based data transmission, (b) tracking based on the Visual-Inertial Odometry (VIO), and (c) occlusion (a) location-based data transmission, (b) tracking based on the Visual-Inertial Odometry (VIO), and built by the 3D reconstruction. (c) occlusion built by the 3D reconstruction.

3. Methods 3. Methods 3.1. Pipeline Data Loading Based on High Accuracy Differential Positioning and Scene Identification Based 3.1. Pipeline Data Loading Based on High Accuracy Differential Positioning and Scene Identification Based on Visualization on Visualization As mentioned above, the first problem is how to accurately retrieve and load pipelines near the As mentioned above, the first problem is how to accurately retrieve and load pipelines near the user. We can obtain location data by the BDS and network transmission without complex and large user. We can obtain location data by the BDS and network transmission without complex and large equipment. The user’s location information is used to complete data retrieval. As shown in Figure 2, equipment. The user’s location information is used to complete data retrieval. As shown in Figure 2, the framework for pipeline data loading consists of three steps: the framework for pipeline data loading consists of three steps:

ISPRS Int. J. Geo-Inf. 2018, 7, 32 ISPRS Int. J. Geo-Inf. 2018, 7, 32

1.

6 of 17 6 of 17

Obtaining differential corrections 1.

Obtaining differential corrections

We need to rely on precise BeiDou real-time positioning based on the Ground-Based We need to rely on precise BeiDou real-time positioning based on the Ground-Based Augmentation System [33], which uses the theory of real time differential (RTD) [34], which reaches Augmentation System [33], which uses the theory of real time differential (RTD) [34], which reaches submeter precision. TheThe phone sends andlatitude latitude coordinates to the customized submeter precision. phone sendsthe theinitial initiallongitude longitude and coordinates to the customized location service in Global Positioning (GGA)format format and receives differential location service in Global PositioningSystem System Fix Fix Data Data (GGA) and receives the the differential corrections in Radio Technical Commission (RTCM)[26] [26] format. corrections in Radio Technical Commissionfor forMarine Marine services services (RTCM) format. 2.

2. Providing differential corrections solutions Providing differential corrections solutions

The smartphone transmits thedifferential differential corrections corrections totothe differential device and and The smartphone transmits the thehandheld handheld differential device obtains the user’s precise coordinates via Bluetooth. The solution of the precise coordinates using obtains the user’s precise coordinates via Bluetooth. The solution of the precise coordinates using differential corrections is completed thedifferential differential chip. chip. differential corrections is completed bybythe

3.

3.

Loading data

Loading data

The mobile device transmits the precise coordinate parameters to the data server by HTTP

The mobile device transmits the precise coordinate parameters to the data server by HTTP protocol. According to the range of the location, the data server provides the corresponding ADF and protocol. According to the range of the location, the data server provides the corresponding ADF and pipeline data around the user according to the user’s position, in JavaScript Object Notation format. pipeline data the user according theand user’s position, in JavaScript Object and Notation format. Then the around mobile device downloads the to data transforms it into world coordinates loads the Then ADF the mobile device data and transforms into world coordinates loads the for the area thatdownloads has enoughthe visual features such as the itroadside, the area around aand building, ADF for etc. the area that has enough visual features such as the roadside, the area around a building, etc.

Figure 2. The procedures of pipeline loading: (a) obtaining differential corrections, (b) providing Figure 2. The procedures of pipeline loading: (a) obtaining differential corrections, (b) providing differential corrections solutions, and (c) loading data. differential corrections solutions, and (c) loading data.

ADF contains mathematical descriptions of visual environmental features and is analogous to

ADF descriptions ofmake visual environmental features and is analogous spatialcontains memorymathematical based on loop closure [35]. We ADFs in advance with the application offered to spatial based onofloop closure [35]. We make ADFs in advance with the application offered by bymemory Tango. The origin the ADF’s coordinates is where we first launch the application, and its angles of The axes origin are the of same the camera orientation when the is launched. After computing the of Tango. theasADF’s coordinates is where weapplication first launch the application, and its angles probability of as whether the current image comes the already perceived scene, the smartphone axes are the same the camera orientation whenfrom the application is launched. After computing the can localize itself by obtaining theimage ADF’s coordinate parameters and connect the pipelines to asmartphone precise probability of whether the current comes from the already perceived scene, the location in the real world by aligning the device’s world coordinates with the ADF’s coordinates. can localize itself by obtaining the ADF’s coordinate parameters and connect the pipelines to a precise We made an ADF that recorded visual features of an area five meters long and five meters wide. location in the real world by aligning the device’s world coordinates with the ADF’s coordinates. The memory of the ADF was about 10 mb, considering the network transmission. Taking into account We made an ADF that recorded visual features of an area five meters long and five meters wide. the impact of sunlight on recognition, we made several ADFs for the same area under different light The memory of the about 10 considering network Taking into account conditions, andADF userswas can choose themb, ADF according tothe their currenttransmission. light conditions. Therefore, by the impact of sunlight on recognition, we made several ADFs for the same area under different light conditions, and users can choose the ADF according to their current light conditions. Therefore, by initializing the ADF, we can ignore the error of satellite positioning. On the other hand, satellite positioning narrows the range of ADF retrieval.

ISPRS Int. J. Geo-Inf. 2018, 7, 32

7 of 17

ISPRS Int. J. Geo-Inf. 2018, 7, 32

7 of 17

3.2. Spatial Conversion of the Pipeline Coordinates

initializing the ADF, we can ignore the error of satellite positioning. On the other hand, satellite

After the data transfer, we need to transform positioning narrows the range of ADF retrieval. the raw pipeline data with the spatial descriptions of lines received in JavaScript Object Notation format into the rendering data. Designing a method of data 3.2. Spatialand Conversion the Pipeline management loadingofcan make itCoordinates convenient for managers of underground pipelines rather than programmers, making it easy to pipeline data andpipeline changedata thewith position of pipelines visually. After the data transfer, weupdate need to transform the raw the spatial descriptions of lines received in JavaScript Object Notation format into the data.longitude Designingcoordinates a method of(B, L, First we loaded attributes of the pipeline and transformed the rendering latitude and data management loading can make itcoordinate convenient system, for managers underground(X, pipelines rather H) into (X, Y, Z) in theand spatial rectangular thenoftransformed Y, Z) into world than programmers, making it easy to update pipeline data and change the position of pipelines coordinate data through seven-parameter coordinate transformation [36]. visually. First we loaded attributes of the pipeline and transformed the latitude and longitude Figure 3 shows the conversion from the spatial rectangular coordinate system to the world coordinates (B, L, H) into (𝑋, 𝑌, 𝑍) in the spatial rectangular coordinate system, then transformed coordinate system. When Tango is initialized, the world coordinate system and the camera coordinate (𝑋, 𝑌, 𝑍) into world coordinate data through seven-parameter coordinate transformation [36]. system have the same origin and axes, so the current position and rotation of the camera are Figure 3 shows the conversion from the spatial rectangular coordinate system to the world (0, 0, coordinate 0). We recorded location M (X , YM , Z M ) the and world the posture parameters pitch, azimuth) system.theWhen Tango is Minitialized, coordinate system (roll, and the camera obtained from the direction sensor of the phone, expressed as (ω , ω , ω ). The scale of coordinate x y z coordinate system have the same origin and axes, so the current position and rotation of the camera transformation world coordinates to(𝑋 the spatial rectangular coordinate system was m. Assuming are (0, 0, 0).from We recorded the location M the posture parameters (roll, pitch, azimuth) 𝑀, 𝑌 𝑀 , 𝑍𝑀 ) and obtained from the direction sensor of the phone, as (ω𝑥axes , ω𝑦 , ω scale of coordinate that point M was the origin of world coordinates andexpressed its coordinate were approximately parallel to 𝑧 ). The transformation from world coordinates the spatial rectangular coordinate system m. the coordinate axes of the spatial rectangular to coordinate system, the world coordinates can was be regarded Assuming that point wasworld. the origin of world coordinates and its coordinate were the as local coordinates in theMreal In order to meet the requirements, when weaxes initialize approximately toshould the coordinate axes of spatial direction rectangular system,screen the world system for the first parallel time, we try to keep thethe vertical ofcoordinate the smartphone parallel coordinates can be regarded as local coordinates in the real world. In order to meet the requirements, to the north and then transform the pipeline coordinate data in the spatial rectangular coordinate when we initialize the system for the first time, we should try to keep the vertical direction of the system into world coordinates through seven-parameter coordinate transformation. Equations (1)–(4) smartphone screen parallel to the north and then transform the pipeline coordinate data in the spatial 0 , Z 0 ) in world showrectangular the conversion from (X,system Y, Z) ininto the world spatial coordinates rectangular coordinate system to (X 0 , Ycoordinate coordinate through seven-parameter coordinates. After the conversion, calculated the world pipeline start point transformation. Equations (1)–(4)we show the conversion fromcoordinates ( 𝑋, 𝑌, 𝑍 ) in of thethe spatial rectangular 0 0 0 0 0 0 ′ ′ ′ (Xs , Ycoordinate end point Zeworld ) fromcoordinates. the raw data. process we of calculated Unity3D the development, to (𝑿 (X , 𝒀e,, 𝒁Ye),in After In thethe conversion, world s , Zs ) and system ′ ′ pipeline start (𝑿′𝒔 , 𝒀position, end point scale, (𝑿′𝒆 , 𝒀′𝒆and , 𝒁′𝒆 ) material from the raw data. In the objectcoordinates renderingofisthe determined by point the pivot of mesh. However, 𝒔 , 𝒁𝒔 ) and rotation, process of Unity3D development, object rendering is determined by the pivot position, rotation, scale, the height anomaly of satellite positioning has an influence on the height of the pipeline coordinates. and must material of mesh. However, the heightand anomaly of satellite positioning hasfloor an influence on distance the The user place the phone horizontally the depth sensor toward the to get the height of the pipeline coordinates. The user must place the phone horizontally and the depth sensor between the phone and the floor through point clouds, then the height of the floor and the height of toward the floor to get the distance between the phone and the floor through point clouds, then the the pipeline can be calculated by the depth under the floor. As shown in Equations (5)–(8), the pivot height of the floor and the height of the pipeline can be calculated by the depth under the floor. As coordinates (X 0p , Yp0 , Z 0p ),(5)–(8), scale and orientation (α x , α(𝑿 y ,′ α z )′ of′ the game object in Unity3D are calculated shown in Equations the pivot coordinates 𝒑 , 𝒀𝒑 , 𝒁𝒑 ), scale and orientation (𝜶𝒙 , 𝜶𝒚 , 𝜶𝒛 ) of the 0 , Y 0 , Z 0 ) and end point 0 from game the start point coordinates (X coordinates (Xe0 , (𝑿 Ye0′,,Z ′ the pipeline. In this s s s from the start point e′ ) of object in Unity3D are calculated coordinates 𝒔 𝒀𝒔 , 𝒁𝒔 ) and end point ′ equal ′ case, coordinates the pivot point to the midpoint of the pipeline. The length and diameter determine the (𝑿′𝒆 , 𝒀is 𝒆 , 𝒁𝒆 ) of the pipeline. In this case, the pivot point is equal to the midpoint of the threepipeline. components of theand pipeline’s in Unity3D. The length diameterscale determine the three components of the pipeline’s scale in Unity3D.

Figure 3. Coordinate conversion fromspatial spatial rectangular rectangular coordinate system to world coordinates. Figure 3. Coordinate conversion from coordinate system to world coordinates.

𝑋

𝑋𝑀

𝑋′

 [𝑌 ]= [ 𝑌𝑀 ]+ （1 + m）R1 (𝜔𝑥 )R 2 (𝜔𝑦 )R 3 (𝜔𝑧 ) [𝑌 ′   ] 0 X X X ′ M 𝑍 𝑍 𝑀 
  𝑍   Y  =  YM  + (1 + m)R1 (ωx )R2 ωy R3 (ωz ) Y 0  0 ZM Z Z 

(1)

(1)

ISPRS Int. J. Geo-Inf. 2018, 7, 32

8 of 17



1 0 0  R1 (ωx ) =  0 cos(ωx ) sin(ωx ) 0 − sin(ωx ) cos(ωx ) 

cos ωy 0 − sin ωy
 R2 ωy =  0 1 0

sin ωy 0 cos ωy  cos(ωz ) sin(ωz ) 0  R3 (ωz ) =  − sin(ωz ) cos(ωx ) 0 0 0 1       X 0p Xs0 Xe0  0   0   0   Yp  = ( Ys  +  Ye )/2 Zs0 Ze0 Z 0p     αx cos−1 (∆X/R)      αy  =  cos−1 (∆Y/R)  − 1 αz cos (∆Y/R)

R= ∆X = Xe − Xs

p

     

(3)

  

∆X 2 + ∆Y 2 + ∆Z2

∆Y = Ye − Ys

(2)

∆Z = Ze − Zs

(4)

(5)

(6)

(7) (8)

3.3. Tracking Technology Based on VIO Algorithm As mentioned above, the second problem is how to keep object tracking stable. Stable object tracking ensures that the virtual model can be updated precisely at any perspective of the mobile device, which results in the visual effect that the virtual model is fixed to a specific position. We adopted the tightly coupled VIO algorithm that combined visualization with inertia to obtain the 6DOF of the mobile device. In Equation (9), the Sim(3) [37] defines the group of similarity transformations between adjacent frames in three-dimensional space, which consists of three parameters: t ∈ R3 is a translation, R ∈ SO(3) is a rotation, and s ∈ R is a scale between current frame and previous frame. " Sim(3) = {S =

sR 01×3

t 1

#

∈ R 4×4 }

(9)

By extracting the feature points of adjacent frames, which are commonly based on Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), etc., and matching them, we can calculate the 6DOF of the camera. Optimization mainly refers to reducing noise in the SLAM process. The visual odometry (VO) algorithm estimates the motion between the two images. By integrating the acceleration and posture of the device, we can obtain the distance. The previous estimation error accumulates to the motion estimation in the back, which is called drift [38]. The classic VO algorithm has a frame rate of only 10–30 Hz. The feature points are easy to lose when the camera moves fast, and there will be scale ambiguity. The IMU can get the acceleration and posture of the device directly and has a frame rate that reaches 200 Hz, ensuring faster speed and solving the problem of scale ambiguity. IMU overcomes the inherent shortcomings of VO, which makes up for the inherent noise and errors of IMU [39]. Therefore, the VIO has more advantages. Moreover, the fisheye camera ensures that model tracking is stable. In Figure 4, the camera’s perspective is in a subway station model, which is the same size as reality. Due to the stable tracking of the camera, the position of the virtual model is consistent with reality, and its refresh rate reaches 50 Hz. In order to evaluate the performance for long-distance use, we held the Tango device and walked through a fifty-meter-long corridor, and measured the distance of the corner at the end of the corridor between the screen and the real world; the average distance was about 1.23 m.

ISPRS Int. J. Geo-Inf. 2018, 7, 32

9 of 17

ISPRS Int. J. Geo-Inf. 2018, 7, 32

9 of 17

ISPRS Int. J. Geo-Inf. 2018, 7, 32

9 of 17

Figure 4. Camera tracking in a virtual subway station (a) at the corner, (b) near the elevator. Figure 4. Camera tracking in a virtual subway station (a) at the corner, (b) near the elevator. 4. Camera tracking in a virtual station (a) at theuse corner, (b) near the elevator. We Figure designed a method to reduce drift subway due to long-distance of VIO. As described in Section We designed a method to reduce drift due to long-distance use of VIO. As described in Section 3.2, 3.2, when Tango initializes, the world coordinate and camera coordinate systems have the same We designed to coordinates reduce drift initialize. due to camera long-distance use of VIO. described in Section when Tango initializes, world coordinate and coordinate systems have theclearing same origin origin and axes, soa method thethe world We can reduce drift byAs regularly the 3.2, when Tango initializes, the world coordinate and camera coordinate systems have the same and axes, so the world coordinates initialize. We can reduce drift by regularly clearing the previous previous estimation of orientation and position. origin and axes, sotwo themodes world coordinates initialize. We canstatic reduce by regularly clearing the estimation of orientation and position. We designed for the rendering of pipeline: anddrift dynamic mode. Considering previous estimation of orientation and position. Wecontinuous designed requests two modes for the rendering dynamic mode. that for location service canof bepipeline: a burden static on theand battery, by asking for Considering the service We designed two modes for rendering of an pipeline: static and dynamic mode. Considering that continuous for can location service canfor be a burden on the battery, byten asking forwide the service only once, the requests smartphone get the pipeline data area ten meters long and meters from that continuous requests for location service can be a burden on the battery, by asking for the service only once, the smartphone can get pipeline data for an area ten meters long and ten meters wide from the server and render the pipelines. Tango initializes and restarts tracking each time the rendering only once, the smartphone can get pipeline data for an area ten meters long and ten meters wide from and therender world the coordinates reinitialize and have aand newrestarts origin in the realeach worldtime according to the thestarts, server and pipelines. Tango initializes tracking the rendering the server and instead render the pipelines. Tango initializesa and restarts tracking eachintime the5a, rendering user’s location of using the same origin shown Figure in static starts, and the world coordinates reinitialize andover have along newdistance. origin inAsthe real world according to the starts, and the world coordinates reinitialize and have afrom new origin in the real world according to the mode, the user can watch pipelines in a fixed area different perspectives. The tracking is user’s location instead of using the same origin over a long distance. As shown in Figure 5a, in static user’s location instead of usingobservation the same origin overFigure a long 5b distance. As shown rendered in Figure in 5a,dynamic in static accurate enough for people’s distance. shows pipelines mode, the user can watch pipelines in a fixed area from different perspectives. The tracking is accurate mode, the get userthecan watch pipelines in a fixed area from in different perspectives.through The tracking is mode;for we user’s real-time position and orientation world coordinates VIO mode; and enough people’s observation distance. Figure 5b shows pipelines rendered in dynamic accurate enough fordata people’s observation distance. Figure shows pipelines rendered in dynamic query the pipeline near the user inorientation a fixed range from5b the data we receive. Thus, the userthe wemode; get the user’s real-time position and in world coordinates through VIOonce andVIO query we get the user’s real-time position and orientation in world coordinates through and obtains the data from the server, the following rendering will not be achieved through the network. pipeline data near thedata usernear in athe fixed range from range the data wethe receive. Thus, onceThus, the user obtains the query the pipeline user in a fixed from data we receive. once the user When the user walks out of the area of pipeline data, the system will remind him or her to ask for data from the server, the following rendering will not be achieved through the network. When the user obtains the data from thethe server, thecoordinates. following rendering not beand achieved through the update network. service again to initialize world Thus, the will pipelines world coordinates in walks outthe of user the area ofout pipeline data, thepipeline systemdata, will the remind him or remind her to ask for service again to When walks of the area of system will him or her to ask for correspondence with the moving of the user’s location. initialize the world coordinates. Thus, the pipelines and world coordinates update in correspondence service again to initialize the world coordinates. Thus, the pipelines and world coordinates update in with the moving ofwith the the user’s location. correspondence moving of the user’s location.

Figure 5. Ways pipelines are rendered: (a) in static mode, (b) in dynamic mode. Figure Wayspipelines pipelines arerendered: rendered: (a) (a) in in static Figure 5.5.Ways are static mode, mode,(b) (b)inindynamic dynamicmode. mode. 3.4. Pipeline Rendering Based on Occlusion

As mentioned above, the last problem is how to improve the immersive experience of Pipeline RenderingBased Basedonon Occlusion 3.4.3.4. Pipeline Rendering Occlusion underground pipelines in real time. The ideal situation is that the mobile device builds the model of Asmentioned mentioned above, the last problem is how howrelation to experience problem is to improve improve the immersive immersive experience theAs environment inabove, real timethe andlast judges the occlusion by the the model’s render queue. Thenof it of underground pipelines in real time. The ideal situation is that the mobile device builds the model of makes the reconstruction model and produces reasonable occlusion underground pipelines in real time.transparent The ideal situation is thatthe the correct mobile and device builds the model of the environment in real time and judges the occlusion relation by the model’s render queue. Then it theeffect. environment in real time and judges the occlusion relation by the model’s render queue. Then it makes reconstruction model underground transparent and produces thedifferent correct and reasonable occlusion Forthe example, people watch pipelines from Parts of effect. the makes the reconstruction model transparent and produces the correct andperspectives. reasonable occlusion effect. pipelines and walls are hidden the ground. Ifpipelines all the walls and pipelinesperspectives. are shown, there willofbethe For example, people watchby underground from different Parts For example, people watch underground pipelines from different perspectives. Parts of the pipelines and walls are hidden by the ground. If all the walls and pipelines are shown, there will be the pipelines and walls are hidden by the ground. If all the walls and pipelines are shown, there will be

ISPRS Int. J. Geo-Inf. 2018, 7, 32

10 of 17

ISPRS Int. J. Geo-Inf. 2018, 7, 32

10 of 17

the that illusion that the pipelines are floating the ground. As shown in Figure we designed the of illusion the pipelines are floating on the on ground. As shown in Figure 6, we6,designed the shape shape of the buried pit as an arc instead of a square; the irregular shape made the buried pit look the buried pit as an arc instead of a square; the irregular shape made the buried pit look logical and ISPRS Int. J. Geo-Inf. 2018, 7, 32 10 of 17 true. logical and true. In addition, we made the texture the same as the ground at the edge of the wall, the In addition, we made the texture the same as the ground at the edge of the wall, which enhanced which enhanced layered effect between aboveground and underground. Meanwhile, we used a the illusion that the pipelines are floating on the ground. As shown in Figure 6, we designed the layered effect between aboveground and underground. Meanwhile, we used a transparent occlusion transparent surrounding areathe of irregular the groove to simulate effectpitoflook the shape of theocclusion buried pitcovering as an arcthe instead of a square; shape made thethe buried covering the surrounding area of the groove to simulate the effect of the ground. Then we hid the part ground. Then we In hidaddition, the part we of the inner that would be as covered by realatground. The logical and true. made thewall texture the same the ground the edge of occlusion the wall, of the inner wall that would be covered by real ground. The occlusion hid the pipelines that stretched hid theenhanced pipelines the thatlayered stretched outbetween along theaboveground wall, which and made the junction of the pipeline thea which effect underground. Meanwhile, weand used out along the wall, which made the junction of the pipeline and the wall seem more natural. wall seem more natural. transparent occlusion covering the surrounding area of the groove to simulate the effect of the ground. Then we hid the part of the inner wall that would be covered by real ground. The occlusion hid the pipelines that stretched out along the wall, which made the junction of the pipeline and the wall seem more natural.

Figure 6. Occlusion for pipelines and walls.

Figure 6. Occlusion for pipelines and walls.

Similarly, if pipelines are rendered on real objects, such as flower beds and dustbins, they will

Similarly, if pipelines areare rendered on realthe objects, such flower beds and dustbins, they will have the illusion that they floating above ground. It and is as more reasonable to make the entities Figure 6. Occlusion for pipelines walls. on the real ground hide the pipelines. An RGB-D depth camera that launches and receives infrared have the illusion that they are floating above the ground. It is more reasonable to make the entities structure light provides measurements ofobjects, the surrounding environment Then the 6DOF Similarly, ifhide pipelines are rendered real such as flower beds and[40]. dustbins, they will on the real ground thedepth pipelines. Anon RGB-D depth camera that launches and receives infrared output from the VIO is used to complete the registration of point clouds acquired by the camera. have light the illusion thatdepth they are floating above of thethe ground. It is moreenvironment reasonable to [40]. makeThen the entities structure provides measurements surrounding the 6DOF Thus, can obtainhide models of the entities and place them in thethat right place. This not suitable for on from thewe real the to pipelines. An the RGB-D depth camera andisreceives output theground VIO is used complete registration of pointlaunches clouds acquired byinfrared the camera. items with a reflective or black surface. Considering that the 3D reconstruction contains the mesh of structure light provides depth measurements of the surrounding environment [40]. Then the 6DOF Thus, we can obtain models of the entities and place them in the right place. This is not suitable for the ground that can hide the pipelines, we designed the method to extract the entities from the whole output from the VIO is used to complete the registration of point clouds acquired by the camera. itemsmodel. with aAs reflective black7,surface. thatphone the 3D reconstruction contains the mesh shown inor Figure firstentities we Considering hadand to place them the and is the depth sensor Thus, we can obtain models of the in thehorizontally right place. This not suitable for of thetoward ground can hide the pipelines, wethedesigned the method to extract the entities from thethat to get theblack distance between phone and the floor through point clouds, ① a the items with afloor reflective or surface. Considering that the 3D reconstruction contains thethen mesh of wholevirtual model. As shown in Figure 7, first we had to place the phone horizontally and the depth sensor transparent would be placed on the real according to the the entities distance. Thethe ②whole floor the ground that can plane hide the pipelines, we designed the floor method to extract from 1 a toward the to get distance between the phone and thethe floor point clouds, then and ③floor garbage caninthe inFigure the real were as a whole by 3D through reconstruction. We sensor raised model. As shown 7, world first we hadbuilt to place the phone horizontally and the ① depth the plane m in get order make itbetween higher the mesh of the through floor, highlighted in Figure 7a.a and 2 floor toward the0.01 floor to theto distance andaccording the② floor point clouds, then ① virtual transparent plane would be placed on than the phone real floor to the distance. The

As shown in 7b, wewould applied texture tofloor ①, and ③,to then the virtual transparent plane betransparent placed the real according theset distance. ②queue floor 3 garbage 1 rendering

can inFigure the real world were built as on a whole by the ②, 3D reconstruction.

WeThe raised the plane in the shade to make the plane ① hide the plane ② by the depth, but the pipelines showed. And ③ and ③ garbage can in the real world were built as a whole by the 3D reconstruction. ① We raised 2 the floor, highlighted in Figure 7a. As shown 0.01 m in order to make it higher than the mesh of beyond the plane ①order couldtohide theitpipelines by the depth. It would have ahighlighted better effectin toFigure sit on the the plane 0.01 m in make higher than the mesh of ② the floor, 7a. 1 , 2 and , 3 then set the rendering queue in the in Figure 7b, we applied transparent texture to , flat floor. Furthermore, ourapplied methodtransparent does not require precise textures, sorendering it is less affected As shown in Figure 7b, we texture to ①,and ②, detailed and ③, then set the queue 1 hide the plane 2 by the depth, but the pipelines showed. And 3 beyond shade to make the plane by modeling precision and minimizes in the shade to make the plane ① hidethe theworkload. plane ② by the depth, but the pipelines showed. And ③ 1 could hide the pipelines by the depth. It would have a better effect to sit on the flat the plane beyond the plane ① could hide the pipelines by the depth. It would have a better effect to sit on the floor.flat Furthermore, our method does does not require precise and detailed it isisless lessaffected affected by floor. Furthermore, our method not require precise and detailedtextures, textures, so so it modeling precision and minimizes the workload. by modeling precision and minimizes the workload.

Figure 7. The procedure of occlusion building: (a) the mesh component, (b) the visual effort of occlusion. Figure 7. The procedure of occlusion building: (a) the mesh component, (b) the visual effort of

Figure 7. The procedure of occlusion building: (a) the mesh component, (b) the visual effort occlusion. of occlusion.

ISPRS Int. J. Geo-Inf. 2018, 7, 32 ISPRS Int. J. Geo-Inf. 2018, 7, 32

11 of 17 11 of 17

4. Experiments and Discussions 4. Experiments and Discussions 4.1. 4.1.Spatial SpatialAccuracy Accuracy InInthe thefirst firstexperiment, experiment,ininorder ordertotoevaluate evaluatethe theinfluence influenceofofthe thenetwork networkRTD RTDand andADF ADFon onthe the accuracy of the pipeline position, we tested our system by calculating the location error in accuracy of the pipeline position, we tested our system by calculating the location error insome some notable notableplaces, places,such suchasasintersections, intersections,school schoolgates, gates,etc. etc.We Wechose chose40 40different differentplaces placestototest testthe thespatial spatial accuracy. The handheld differential device worked in dual mode (GPS + BDS). As shown accuracy. The handheld differential device worked in dual mode (GPS + BDS). As shown in Figure in 8b, Figure 8b,the weactual had the actual coordinates of pipeline ③ , in and marked ③ ground on the 3 point 1 area 3point we had coordinates of pipeline point in area and ①, marked point on the ground according to the virtual point on the of screen of the phone. mobile phone. Assuming the of according to the virtual point on the screen the mobile Assuming that thethat error oferror the data the data measured by the transit can be ignored, we recorded the coordinates of marked point ③ 3 and calculated measured by the transit can be ignored, we recorded the coordinates of marked point and the mean distancedeviation and standard deviation between the actual coordinates andofthe the calculated mean distance and standard between the actual coordinates and the coordinates the coordinates of the marked point. As shown in Table 2, the spatial accuracy by RTD reached about marked point. As shown in Table 2, the spatial accuracy by RTD reached about 0.613 m; the total error 0.613 m;came the from total the error mainly came from the of GPS positioning and Moreover, the coordinate mainly error of GPS positioning anderror the coordinate transformation. using transformation. the positioning function the mobile phone, we found that the the positioning Moreover, function of using the mobile phone, we found thatofthe mobile device rendered pipelines that mobile device rendered pipelines that were actually located in area ② according to the coordinate 2 according to the coordinate data. were actually located in area data. Table 2. Standard deviation of the pipeline position. Table 2. Standard deviation of the pipeline position. Method

Method RTD (BDS + GPS) RTD (BDS + GPS) RTD + ADF RTD + ADF

Mean (m)

Standard Deviation (m)

Mean (m) Standard Deviation (m) 0.613 0.054 0.613 0.054 0.127 0.074 0.127 0.074

Wealso alsoevaluated evaluatedthe theaccuracy accuracyofofADF ADFand andRTD RTDatatthree threespecific specifictimes: times:08:30, 08:30,11:30, 11:30,and and16:30. 16:30. We After the ADF download was complete, we made the camera look at a featured scene. The average After the ADF download was complete, we made the camera look at a featured scene. The average successrate rate was was about about 0.73. mean to to identify thethe scene andand initialize the ADF. success 0.73. ItIt cost costabout about1523 1523ms msbyby mean identify scene initialize the It performed betterbetter in a place Then the position of the pipeline adjusted ADF. It performed in a with placestable with light stableconditions. light conditions. Then the position of the pipeline slightly. slightly. As shownAsin shown Table 2,inhowever, identifying scene, drift reduced and adjusted Table 2,by however, by the identifying theisscene, drift is environmental reduced and aspects such asaspects sunlight andastemperature an influence onanaccuracy. environmental such sunlight andhave temperature have influence on accuracy.

Figure8.8.Test Testofofpipeline pipelineposition: position:(a) (a)2D 2Dmap mapofofunderground undergroundpipelines; pipelines;(b) (b)the thecorrect correctpipeline pipeline Figure position;(c) (c)the thewrong wrongpipeline pipelineposition. position. position;

Roberts Robertsetetal. al.[11], [11],Schall Schalletetal. al.[12], [12],and andChang Changetetal. al.[11–14] [11–14]proposed proposedvisualization visualizationofofpipelines pipelines by combination of ofRTK RTKGPS GPSand andINS, INS,and and they had higher location accuracy bytracking trackingbased based on on the the combination they had higher location accuracy and and relatively poor flexibility and portability. In particular, Schall et al. [13] proposed a method relatively poor flexibility and portability. In particular, Schall et al. [13] proposed a method whereby the whereby the user could send a query to the Web service to get GML based his location and transcode the GML into visualization data, and accuracy reached 0.3 m by the RTK technology. The user’s

ISPRS Int. J. Geo-Inf. 2018, 7, 32

12 of 17

user send a2018, query ISPRScould Int. J. Geo-Inf. 7, 32to the Web service to get GML based his location and transcode the GML 12into of 17 visualization data, and accuracy reached 0.3 m by the RTK technology. The user’s position determines the loading of the pipeline data.ofThe high precision of high RTK precision ensures high accuracy the accuracy pipeline position determines the loading the pipeline data. The of RTK ensuresofhigh location. In ourlocation. system, In theour location is the further applied to theapplied transformation of pipeline coordinates of the pipeline system, location is further to the transformation of pipeline and the ADF and download. Thedownload. accuracy ofThe the RTD reaches about 0.5 m. Successful of the coordinates the ADF accuracy of the RTD reaches aboutinitialization 0.5 m. Successful ADF further increased thefurther location accuracythe of the BDS. accuracy This indicates theThis combination of RTD initialization of the ADF increased location of thethat BDS. indicates that the and visualization could better pipeline location accuracy. combination of RTD andachieve visualization could achieve better pipeline location accuracy. 4.2. 4.2. Tracking TrackingAccuracy Accuracy In tracking accuracy by observing the translation of virtual objects from Inthis thisexperiment, experiment,we wetested tested tracking accuracy by observing the translation of virtual objects different angles at the same time every day. Figure 9 shows four snapshots from when we observed from different angles at the same time every day. Figure 9 shows four snapshots from when we the pipelines an intersection at Ocean University in China from fourfrom different The area observed the at pipelines at an intersection at Ocean University in China four viewpoints. different viewpoints. 2 , and its depth was about 3 m. The position, diameter, depth, and texture of the hole was about 10 m 2 The area of the hole was about 10 m , and its depth was about 3 m. The position, diameter, depth, of thetexture pipeline were basically consistent the actual wereThere drainage and of the pipeline were basicallywith consistent withsituation. the actualThere situation. werepipelines, drainage gas pipelines, and street lamp pipelines. The texture at the edge of the wall was similar to the ground, pipelines, gas pipelines, and street lamp pipelines. The texture at the edge of the wall was similar to which made the border between the ground and the buried part look natural. The edge of the hole the ground, which made the border between the ground and the buried part look natural. The edge was at hole the same height as the floor,asenough to show thetoreal vertical relationship between pipelines, of the was at the same height the floor, enough show the real vertical relationship between thus we did not evaluate the accuracy of elevation. pipelines, thus we did not evaluate the accuracy of elevation.

Figure Figure 9. 9. Underground Underground pipelines pipelines in in aa road: road: (a) (a) on on the the top top of of the the pipelines; pipelines; (b) (b) at at the the edge edge of of the the hole; hole; (c) far away from the hole; (d) the whole area of the pipelines. (c) far away from the hole; (d) the whole area of the pipelines.

Furthermore, we we evaluated evaluated drift drift quantitatively quantitatively by placing a striking striking arrow arrow on on the the floor floor and and Furthermore, recording the translation when people walked in aninarea five meters long and five meters recording translationofofthe thearrow arrow when people walked an area five meters long and five wide inwide 13 min different perspectives 10 times.10 Every time the experiment finished, we turned meters in from 13 min from different perspectives times. Every time the experiment finished, off turned the phone andphone cooledand it down. 3 and Figure 10, we measured translation we off the cooledAsit shown down. in AsTable shown in Table 3 and Figure 10, we the measured the according to the location of location a particular point between currentthe time and the The translation according to the of a particular pointthe between current timeoriginal and thetime. original accumulation of drift ledoftodrift an increase translation. Working for a longfor time made Tango overheat time. The accumulation led to anin increase in translation. Working a long time made Tango and influenced accuracy. It was better to confine the tracking time to 10 min and avoid long-distance overheat and influenced accuracy. It was better to confine the tracking time to 10 min and avoid tracking. long-distance tracking.

ISPRS Int. J. Geo-Inf. 2018, 2018,7,7,32 32 ISPRS Int. J. Geo-Inf. 2018, 7, 32

13 of 17 13 of 17

Figure 10. Positions of arrows at different times and two different angles: (a) at the 1st min; Figure 10. Positions different andmin. two different angles: (a) at the 1st the Figure 10. Positions of at(d) different times and two different angles: (a) min; at the(b) 1statmin; (b) at the 5th min; (c)ofatarrows thearrows 9thatmin; at times the 13th 5th min; atmin; the 9th the 13th (b) at the(c) 5th (c) min; at the(d) 9thatmin; (d) atmin. the 13th min.

Table 3. Translation of pipelines in 13 min. Table Table 3. 3. Translation Translation of of pipelines pipelines in in13 13min. min.

Time (min)

Time1(min) Time (min) 1 5 9 13

51 95 9 13 13

Translation (m) Translation (m) (m) 0.022Translation ± 0.013 0.022 ± 0.013 0.044 0.022 ± 0.017 ± 0.013 0.0440.044 0.017 ± 0.017 0.090 ±± 0.021 0.090 ± 0.021 0.090 ± 0.021 0.123 ± 0.020 ± 0.020 0.1230.123 ± 0.020

As shown in Figure 11, touching the black pipeline on the screen opened a window showing As shown shown in Figure Figure 11, material, touching usage, the black black pipeline onfrom the screen screen opened a window window showing As in 11, touching the on the opened showing attribute data such as type, etc.,pipeline received the server. By astable tracking, the attribute data such as type, material, usage, etc., received from the server. By stable tracking, attribute attached suchtoas type, position material,inusage, etc.,world. received server. By stable tracking, window a fixed the real We from couldthe watch the information fromthe a window attached to ato fixed position in in the real We watch the the window attached a fixed position the realworld. world. Wecould could watch the information information different perspective. Moreover, we could touch the buttons on the window to achievefrom basica differentperspective. perspective.Moreover, Moreover, could the buttons on the to window achieve basic different we we could touchtouch the buttons on the window achieveto basic interaction. interaction. interaction.

Figure 11. Display of the pipeline information. Figure 11. Display of the pipeline information. Figure 11. Display of the pipeline information.

Existing methods, such as [11–14], adopted GPS and INS to obtain position and orientation, with methods, such [11–14], and to position with Existing methods, such as as [11–14], adopted GPS and INS INS to obtain obtain position and and orientation, with a lowExisting precision and refresh rate. As isadopted known, GPS tracking based on visualization hasorientation, sub-pixel-level a low precision and refresh rate. As is known, tracking based on visualization has sub-pixel-level a low precision and refresh rate. As is known, tracking basedthe on robustness visualization has sub-pixel-level accuracy; the combination of visualization and inertia improves of outdoor markerless accuracy; the combination of visualization and inertia improves the robustness of outdoor markerless accuracy; the combination of visualization and improves robustness performance. However, when it works for a long time, VIO suffers the accumulation of drift. From performance. However, when it works for a long time, VIO suffers the accumulation of drift. From the performance. it works for a range. long time, VIO suffers the accumulation of drift. From the test results,However, the error when is within acceptable test results, the error is within acceptable range. the test results, the error is within acceptable range. 4.3. Occlusion Building 4.3. 4.3. Occlusion Occlusion Building Building In this experiment, we tested occlusion building based on depth perception outdoors under In experiment, tested building based on perception outdoors under In this this experiment, we we tested occlusion occlusion building on depth depth perception different environmental conditions. Considering that based the phone overheats whenoutdoors it does under huge different environmental conditions. Considering that the phone overheats when it does huge different environmental that the phone when huge computation, we designedconditions. a switch toConsidering turn the scanning function onoverheats and off and stop itthedoes scanning computation, we designed aa switch to turn the scanning function on and off and stop the scanning computation, we designed switch to turn the scanning function on and off and stop the scanning after the environmental entity was modeled. It is reasonable to keep the phone a distance of 0.5 to after the environmental entity was modeled. It is reasonable to keep the phone a distance of 0.5 to

ISPRS Int. J. Geo-Inf. 2018, 7, 32

14 of 17

ISPRS Int. J. Geo-Inf. 2018, 7, 32

14 of 17

after the environmental entity was modeled. It is reasonable to keep the phone a distance of 0.5 to 4 m in front of theof entity. We set We the set resolution of the modeling as 0.03 masto0.03 ensure a balance efficiency 4m in front the entity. the resolution of the modeling m to ensure aofbalance of and quality, and the system’s refresh raterefresh was around 45 Hz. By walking and changing viewpoint efficiency and quality, and the system’s rate was around 45 Hz. By walking andthe changing the of the camera, we camera, could enhance theenhance model’s the missing details. In Figure 12a, depth got viewpoint of the we could model’s missing details. In the Figure 12a,camera the depth point clouds of the surrounding environment for 3D reconstruction, which was applicable to the static camera got point clouds of the surrounding environment for 3D reconstruction, which was applicable scene. The automotive reflected the reflected infrared structure light, and there wasand no there mesh was covering the to the static scene. The glass automotive glass the infrared structure light, no mesh glass. A 3D reconstruction model restored the outline of the car, though there was error in the complex covering the glass. A 3D reconstruction model restored the outline of the car, though there was error or position; the mesh coverthe is about 0.013 is mabout above0.013 the surface a real car. of InaFigure 12b, in hidden the complex or hidden position; mesh cover m aboveofthe surface real car. In the texture of the model adopted the transparent occlusion material, which occluded the pipeline and Figure 12b, the texture of the model adopted the transparent occlusion material, which occluded the gave the and realistic that the car covered thecovered pipelines a verticalindirection. Because we were pipeline gaveeffect the realistic effect that the car theinpipelines a vertical direction. Because concerned about the about 3D reconstruction of the contact position the entities on entities the ground instead of we were concerned the 3D reconstruction of the contactofposition of the on the ground the whole body, the contact positions of things on the ground were enough to achieve the occlusion. instead of the whole body, the contact positions of things on the ground were enough to achieve the In short, it is use occlusion by 3D reconstruction the surrounding environment to occlusion. Inreasonable short, it istoreasonable to use occlusion by 3D of reconstruction of the surrounding enhance the feeling of pipelines being in thebeing ground. environment to enhance the feeling ofsunk pipelines sunk in the ground.

Figure 12. 12. Underground Underground pipeline pipeline based based on on 3D 3D reconstruction reconstruction occlusion: occlusion: (a) (a) the the 3D 3D reconstruction reconstruction of of Figure the environment; (b) effect of transparent occlusion material treatment. the environment; (b) effect of transparent occlusion material treatment.

Roberts et al. [11] and Chang et al. [14] displayed wide-ranging underground infrastructure that Roberts et al. [11] and Chang et al. [14] displayed wide-ranging underground infrastructure that appeared to float on the street. In order to provide a plausible interpretation of depth, Schall et al. [12] appeared to float on the street. In order to provide a plausible interpretation of depth, Schall et al. [12] designed a cubic hole in the ground for the placement of pipeline. Additionally, we changed the designed a cubic hole in the ground for the placement of pipeline. Additionally, we changed the shape of the hole and applied materials from the real world to the underground infrastructure. shape of the hole and applied materials from the real world to the underground infrastructure. Furthermore, we utilized depth perception to accomplish 3D reconstruction of real-world entities Furthermore, we utilized depth perception to accomplish 3D reconstruction of real-world entities and and made the spatial relationship clear. In this way, we made a scene-graph fusion of virtual pipelines made the spatial relationship clear. In this way, we made a scene-graph fusion of virtual pipelines and real ground. The results show that our efforts achieved a more vivid and real portrayal of and real ground. The results show that our efforts achieved a more vivid and real portrayal of underground infrastructure. underground infrastructure. 5. Conclusions 5. Conclusions In this research, aiming to solve new challenges in underground pipeline management, we In this research, aiming to solve new challenges in underground pipeline management, leveraged the power of AR technologies and proposed a rendering procedure for pipeline we leveraged the power of AR technologies and proposed a rendering procedure for pipeline management. A mobile-based VGE system has been developed to reproduce real physical pipelines management. A mobile-based VGE system has been developed to reproduce real physical pipelines and and provide spatial decision support capabilities. First, in order to improve location accuracy, we provide spatial decision support capabilities. First, in order to improve location accuracy, we adopted adopted precise positioning based on the BDS to obtain the user’s coordinates. Then we transformed precise positioning based on the BDS to obtain the user’s coordinates. Then we transformed the raw the raw data into attributes in Unity3D through seven-parameter coordinate transformation and data into attributes in Unity3D through seven-parameter coordinate transformation and rendered rendered the pipelines. The successful initialization of ADF further improved location accuracy. the pipelines. The successful initialization of ADF further improved location accuracy. Second, Second, in order to achieve stable object tracking, we adopted Google Tango to achieve the VIO in order to achieve stable object tracking, we adopted Google Tango to achieve the VIO algorithm, algorithm, which improved the robustness and authenticity of the outdoor pipeline rendering. To which improved the robustness and authenticity of the outdoor pipeline rendering. To improve the improve the immersive experience of underground pipelines in real time, we applied occlusion based immersive experience of underground pipelines in real time, we applied occlusion based on the depth on the depth perception of the pipeline rendering and designed pipeline details. perception of the pipeline rendering and designed pipeline details. We tested our rendering procedure outdoors, and discuss the improvement over existing methods quantitatively in terms of spatial accuracy, tracking accuracy, and the effect of occlusion. Through these experiments, we reached three conclusions: (1) Compared with the RTK GPS, the RTD BDS obviously reduces the physical burden and makes development of the mobile phone convenient,

ISPRS Int. J. Geo-Inf. 2018, 7, 32

15 of 17

We tested our rendering procedure outdoors, and discuss the improvement over existing methods quantitatively in terms of spatial accuracy, tracking accuracy, and the effect of occlusion. Through these experiments, we reached three conclusions: (1) Compared with the RTK GPS, the RTD BDS obviously reduces the physical burden and makes development of the mobile phone convenient, though it has relatively lower accuracy. Through the integration of scene recognition, it can provide high-precision positioning accuracy. (2) Compared with the GPS-INS tracking technology that is usually adopted outdoors, the VIO achieved higher robustness in markerless environments and had relatively high tracking accuracy. However, over time there still exists an obvious drift; by making the VIO initialize, we can control the error. (3) Occlusion modeling based on depth perception and some detail changes achieved a clearer spatial relationship. We give full play to the advantages of VGE that can simulate spatial phenomenon whose occurrence is difficult for people to perceive directly. Therefore, we chose AR technology to show pipelines on the site and in a timely manner in the expression environment. The VGE emphasizes multidimensional perception and interaction beyond reality and closer to true nature [7]. Our work aims to express the position and appearance of underground pipelines closer to true nature; we also designed some necessary windows that go beyond reality for interaction. The results show that VGE is an ideal form for representing underground pipelines that matches the nature of human cognition. However, we have to reduce the resolution to ensure a stable refresh rate. This method still requires further optimization, because real-time 3D reconstruction has a relatively large workload. Strong outdoor light may affect the success rate of scene identification. Temperature also has a certain influence on IMU, which has an impact on tracking precision. All these aspects will be considered in our future work. Acknowledgments: This work is supported by the Primary Research and Development Plan of Shandong Province (Grant No. 2017CXGC0608), Qingdao major projects of independent innovation (No. 16-7-1-1-zdzx-xx), Qingdao source innovation program (No. 17-1-1-6-jch), and the Fundamental Research Funds for the Central Universities (No. 201762005). Author Contributions: Wei Li designed and implemented the method, wrote the paper. We Li and Chenrong Zhu did the experiments. Yong Han conducted the experiments and helped Wei Li to analyze experimental results. Ge Chen, Yu Liu and Yibin Ren read and modified the paper. Yanjie Wang drew the result figures of the experiments. Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2. 3. 4. 5.

6. 7. 8.

Shi, L.; Liu, Z.; Fu, J.; Yan, S. Research on the utility tunnel location technology based on ZigBee communication network. C e Ca 2017, 42, 167–171. Qingquan, L.I. Research on 3D Visualization of Underground Pipeline; Editorial Board of Geomatics & Information Science of Wuhan University: Wuhan, China, 2003. Lu, G. Geographic analysis-oriented Virtual Geographic Environment: Framework, structure and functions. Sci. China Earth Sci. 2011, 54, 733–743. [CrossRef] Lin, H.; Lu, G. Virtual Geographic Environment: A Workspace for Computer-Aided Geographic Experiments. Ann. Assoc. Am. Geogr. 2013, 103, 465–482. [CrossRef] Lv, Z.; Li, X.; Hu, J.; Yin, L.; Zhang, B.; Feng, S. Virtual geographic environment based coach passenger flow forecasting. In Proceedings of the IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications, Shenzhen, China, 12–14 June 2015; pp. 1–6. Che, W.; Lin, H.; Hu, M.; Lin, T. Reality-virtuality fusional avatar-based noise measurement and visualization in online virtual geographic environment. Ann. Gis 2014, 20, 109–115. [CrossRef] Hui, L.; Huang, F.; Xuejun, L.U.; Mingyuan, H.U.; Bingli, X.U. Preliminary study on virtual geographic environment cognition and representation. J. Remote Sens. 2010, 14, 822–838. Azuma, R.; Baillot, Y.; Behringer, R.; Feiner, S.; Julier, S.; MacIntyre, B. Recent Advances in Augmented Reality. IEEE Comput. Graph. 2001, 21, 34–47. [CrossRef]

ISPRS Int. J. Geo-Inf. 2018, 7, 32

9. 10. 11.

12. 13. 14. 15. 16. 17.

18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31.

32. 33. 34.

16 of 17

Feiner, S.; Macintyre, B.; Hollerer, T.; Webster, A. A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment. Pers. Technol. 1997, 1, 208–217. [CrossRef] Sutherland, I.E. A head-mounted three dimensional display. In Proceedings of the Fall Joint Computer Conference, San Francisco, CA, USA, 9–11 December 1968; pp. 757–764. Roberts, G.W.; Evans, A.; Dodson, A. The Use of Augmented Reality, GPS and INS for Subsurface Data Visualisation. In Proceedings of the FIG XXII International Congress, Washington, DC, USA, 19–26 April 2002; pp. 1–12. Schall, G.; Mendez, E.; Kruijff, E.; Veas, E.; Junghanns, S.; Reitinger, B.; Schmalstieg, D. Handheld Augmented Reality for underground infrastructure visualization. Pers. Ubiquitous Comput. 2009, 13, 281–291. [CrossRef] Schall, G.; Schöning, J.; Paelke, V.; Gartner, G. A survey on augmented maps and environments. Adv. Web-Based GIS Mapp. Serv. Appl. 2011, 9, 207. Yong, C.; Xue, L.M. Research on Constructing Outdoor Augmented Reality System Based on Improved ARToolKit. J. Syst. Simul. 2009, 21, 2273–2276. Wagner, D.; Schmalstieg, D. ARToolkit on PocketPC platform. In Proceedings of the 2003 IEEE International Augmented Reality Toolkit Workshop, Tokyo, Japan, 7 October 2003; pp. 14–15. Cushnan, D.; El Habbak, H. Developing AR Games for iOS and Android; Packt Publishing Ltd.: Birmingham, UK, 2013. Kato, H.; Billinghurst, M. Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. In Proceedings of the IEEE and ACM International Workshop on Augmented Reality, San Francisco, CA, USA, 20–21 October 2002; p. 85. Siltanen, S. Theory and Applications of Marker Based Augmented Reality; VTT: Espoo, Finland, 2012; pp. 45–50. Amin, D.; Govilkar, S. Comparative Study of Augmented Reality Sdk’s. Int. J. Comput. Sci. Appl. 2015, 5, 11–26. [CrossRef] Davison, A.J. Real-Time Simultaneous Localisation and Mapping with a Single Camera. In Proceedings of the IEEE International Conference on Computer Vision, Nice, France, 13–16 October 2003; p. 1403. Engel, J.; Schöps, T.; Cremers, D. LSD-SLAM: Large-Scale Direct Monocular SLAM; Springer International Publishing: Berlin, Germany, 2014; pp. 834–849. Durrant-Whyte, H.; Bailey, T. Simultaneous Localization and Mapping: Part I. IEEE Robot. Autom. Mag. 2006, 13, 99–110. [CrossRef] Bailey, T.; DurrantWhyte, H. Simultaneous localisation and mapping (slam) part 2: State of the art. IEEE Robot. Autom. Mag. 2006, 13, 108–117. [CrossRef] Davison, A.J.; Reid, I.D.; Molton, N.D.; Stasse, O. MonoSLAM: Real-Time Single Camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 29, 1052–1067. [CrossRef] [PubMed] Jones, E.S.; Soatto, S. Visual-inertial navigation, mapping and localization: A scalable real-time causal approach. Int. J. Robot. Res. 2011, 30, 407–430. [CrossRef] Leutenegger, S.; Lynen, S.; Bosse, M.; Siegwart, R.; Furgale, P. Keyframe-based visual-inertial odometry using nonlinear optimization. Int. J. Robot. Res. 2015, 34, 314–334. [CrossRef] Marder-Eppstein, E. Project Tango; ACM SIGGRAPH 2016 Real-Time Live: Anaheim, CA, USA, 2016; p. 40. Li, M.; Mourikis, A.I. High-precision, consistent EKF-based visual–inertial odometry. Int. J. Robot. Res. 2013, 32, 690–711. [CrossRef] Li, M.; Qu, L.; Zhao, Q.; Guo, J.; Su, X.; Li, X. Precise point positioning with the BeiDou navigation satellite system. Sensors 2014, 14, 927–943. [CrossRef] [PubMed] Klingensmith, M.; Dryanovski, I.; Srinivasa, S.; Xiao, J. Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields. Robot. Sci. Syst. 2015, 4. Schops, T.; Sattler, T.; Hane, C.; Pollefeys, M. 3D Modeling on the Go: Interactive 3D Reconstruction of Large-Scale Scenes on Mobile Devices. In Proceedings of the International Conference on 3d Vision, Lyon, France, 19–22 October 2015; pp. 291–299. Jili, Y. Rigorous Formula for Direct Calculating Parameter in 3D Transformation. Bull. Surv. Mapp. 2006, 5, 57–60. Lee, J.; Jung, S.; Pullen, S. Enhancements of Long Term Ionospheric Anomaly Monitoring for the Ground-Based Augmentation System. Proc. Int. Tech. Meet. Inst. Navig. 2011, 8034, 930–941. Yunck, T.P.; Bertiger, W.I.; Lichten, S.M.; Mannucci, A.J.; Muellerschoen, R.J.; Wu, S.C. Robust Real-Time Wide-Area Differential GPS Navigation. U.S. Patent 5,828,336, 27 October 1998.

ISPRS Int. J. Geo-Inf. 2018, 7, 32

35. 36. 37.

38. 39. 40.

17 of 17

Angeli, A.; Filliat, D.; Doncieux, S.; Meyer, J. Fast and Incremental Method for Loop-Closure Detection Using Bags of Visual Words. IEEE Trans. Robot. 2008, 24, 1027–1037. [CrossRef] Wang, L.; Shaojie, N.I.; Wang, F. Development and Application of GPS Ground-based Augmentation System. Gnss World China 2014, 39, 26–29. Salas, M.; Latif, Y.; Reid, I.D.; Montiel, J.M.M. Trajectory Alignment and Evaluation in SLAM: Horn’s Method vs Alignment on the Manifold. Available online: http://webdiis.unizar.es/~msalasg/docs/rss15.pdf (accessed on 18 January 2018). Scaramuzza, D.; Fraundorfer, F. Visual Odometry: Part I: The First 30 Years and Fundamentals. IEEE Robot. Autom. Mag. 2011, 18, 80–92. [CrossRef] Hesch, J.A.; Kottas, D.G.; Bowman, S.L.; Roumeliotis, S.I. Camera-IMU-based localization: Observability analysis and consistency improvement. Int. J. Robot. Res. 2014, 33, 182–201. [CrossRef] Newcombe, R.A.; Izadi, S.; Hilliges, O.; Molyneaux, D.; Kim, D.; Davison, A.J.; Kohi, P.; Shotton, J.; Hodges, S.; Fitzgibbon, A. KinectFusion: Real-time dense surface mapping and tracking. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, Basel, Switzerland, 26–29 October 2012; pp. 127–136. © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).