Framework for Real-Time Three-Dimensional Modeling of Infrastructure

Framework for Real-Time Three-Dimensional Modeling of Infrastructure Jochen Teizer, Changwan Kim, Carl T. Haas, Katherine A. Liapi, and Carlos H. Caldas over time, may become an irregular four-sided polygon to fit a distorted wooden beam or a cylinder that may grow a joint to represent a bent pipe. Nonparametric forms include wire nets that may represent contour data, polylines that can represent cracks and occupancy arrays, or octrees that can represent amorphous volumes or deformed objects. These forms can generally be derived from range point data that contains the distance information in an array of pixels of the original scene image. Thus, the need for fast and accurate geometric modeling requires using existing and emerging laser ranging technologies such as laser detection and ranging (LADAR), Total Stations, and Flash LADAR. These new sensor technology innovations allow the highest priority problems in the transportation and construction area to be addressed. The laser ranging technologies and typical applications are defined in following sections.

Modeling transportation infrastructure assets in three dimensions (3D) is becoming increasingly necessary for good management. Condition assessment, maintenance, operations, and construction activities are exploiting 3D models for improved visualization, communications, and process control. Acquiring 3D models rapidly can improve safety and productivity and is becoming feasible through approaches based on sparse range point clouds; however, although this approach has contextual advantages, it is ultimately limited in speed. Emerging Flash laser detection and ranging (LADAR) technology is opening up the possibility of 3D modeling at rates better than 1 Hz (real time). A framework for 3D modeling is presented that includes the dimension of time. In particular, the performance of the Flash LADAR technology is examined, and potential applications are explored. Technologies such as Flash LADAR will play an important role in real-time modeling of infrastructure assets in the near future.

Having the ability to locate, describe in three dimensions (3D), control, and track objects within a field of view has become an important factor in transportation construction, maintenance, and asset management, as well as in intelligent transportation systems. This usually requires that assets be scanned and then modeled in 3D at varying frequencies, depending on the application. Asset management may require no more than biannual updates, whereas construction activities may require real-time updates (1). For these cases and the continuum in between, it is useful to have a framework for the process of data acquisition and 3D model building. An overall framework for this process is presented in Figure 1. Figure 1 illustrates that, in practice, 3D modeling requires combinations of topdown design, bottom-up data acquisition, and comparison of both sources of information in many cases for individual assets. In addition, 3D models can be represented in three basic forms. Typically, design processes provide well-defined information, including perfectly parallel, perpendicular, flat, and the like forms (strong forms) such as pipes, beams, columns, and floors, whereas weak and nonparametric forms are produced from existing infrastructure conditions. As defined by Hirschberg and Streilein (2), “weak forms” often are related to strong forms, but previous design information was improperly documented. Examples are a rectangle that,

STATE OF THE ART Laser range scanners have been widely used to obtain 3D range data for construction site scenes. Unlike traditional survey instruments, laser range scanners require no target reflectors. Unlike ultrasonic and stereo vision sources, they provide a large amount of precise, dense data. With significant cost and time benefits in other industries for applications such as inspection of objects on manufacturing lines, one true value of range sensors lies in their ability to rapidly build virtual representations to facilitate automation tasks (3). Work related to the use of laser range scanners in the construction of transportation assets has focused on issues such as capturing 3D as-built conditions (4). Figure 2 illustrates the process of modeling a bridge. In construction projects, the tracking of progress is required to enable project control and to integrate the construction process information into life-cycle data. Laser range scanners also have proven to be beneficial for teleoperable control of semiautomated or automated equipment on large construction sites where timely, on-site decisions require rapid recognition and accurate measurement of objects in the workspace (5). A limitation of most workspace modeling applications is their reliance on analyzing dense point cloud data. Leica Geosystems HDS, Inc., (formerly Cyra Corporation) developed a powerful information management system that requires computationally intensive processing. However, the slow speed in extracting objects from dense point clouds is a limitation of current modeling systems. It makes them unusable for real-time equipment control. For real-world automated, semiautomated, or robotic equipment, the geometric information of target objects must be obtained rapidly (6). Current workspace modeling applications demand large amounts of computation because of the large data sets and scanning

J. Teizer, C. T. Haas, and C. H. Caldas, Department of Civil Engineering, University of Texas, Austin, TX 78712. C. Kim, Department of Architecture, Chung-Ang University, Seoul, South Korea. K. A. Liapi, University of Patras, Panepistimioupolis, Rio, 26500, Patras, Greece. Transportation Research Record: Journal of the Transportation Research Board, No. 1913, Transportation Research Board of the National Academies, Washington, D.C., 2005, pp. 177–186.

177

178

Transportation Research Record 1913

FIGURE 1

Illustration of framework for real-time 3D modeling of infrastructure.

(a)

(b)

(c) FIGURE 2 Bridge in Dallas: (a) I-35 and I-40 interchange, (b) range data representation from LADAR, and (c) modeling process from original to range model and 3D visualization (4) (images courtesy of Leica Geosystems HDS).

Teizer, Kim, Haas, Liapi, and Caldas

of contiguous points on the target object’s surface. They may even have unfortunate weaknesses with respect to civil engineering applications. For example, one generally very powerful approach is not appropriate for symmetric objects (3). Existing range scanning techniques for 3D modeling can be categorized into dense point cloud and sparse point cloud approaches. A sparse point cloud approach enables magnitudes more rapid modeling than dense point cloud approaches. Both approaches are focused on static environments and have their own merits. The Flash LADAR approach can address static and moving environments.

Dense Point Cloud Approach LADAR is primarily used commercially to accurately model as-built conditions because its acquisition of a very dense precise range point clouds (7 ). Taking single, dense range images requires anywhere from 20 s to several hours, because each image may consist of millions of points. Postprocessing time is in the order of days or weeks (4). Purchase costs of between $25,000 for lower-end systems and up to $1 million for high-end systems are often hard for small businesses to handle (5), despite the potentially high accuracy of 1 mm and range of up to 350 m. This technology is powerful but not appropriate for identifying and tracking people, equipment, or objects in real time or for updating models of assets more frequently than every few weeks.

Sparse Point Cloud Approach The sparse point cloud approach focuses on selected points to avoid high computational costs of dense range point cloud information and therefore requires only a few minutes to model a scene. The sparse point cloud approach is based on three basic transformations: (a) fitting sets of range points to computer-aided design (CAD) primitives, (b) creating bounding objects, and (c) merging and compliance checking. It can produce strong, weak, or nonparametric forms. Human intervention is needed in all three steps to select meaningful points from a cluttered scene. Figures 3a to 3d demonstrate that an abstract model with a collection of primitive strong forms will suffice as building blocks for a description for many applications (8, 9). Bounding algorithms allow for grouping all range points into bounded objects such as convex hulls (Figures 3e and 3f ). This is a process of abstracting or simplifying nonparametric and weak forms or cluster forms. It is useful for real-time applications, because it minimizes computational burden. Use of obstacle avoidance calculations in the background in real time can improve equipment operation and safety. When objects are related to tasks, object fitting, matching, and merging algorithms can be used to extract precise geometrical information from workplace scenes. Such spatial modeling can be applied in obstacle avoidance operations of heavy equipment (10). The major limitation (and power) of the sparse point cloud approach is the requirement for human judgment and the focus on static environments. Judgment is used in the process of acquiring distinct range point clouds. This enables rapid modeling of the static elements of work spaces; however, moving objects can not be captured without distracting the operator. Thus, detection and avoidance of moving obstacles still requires full operator attention (11). A coherent view and idea of objects is based on relating individual parts to a world model. Figures 3g to 3j illustrate the use of developed

179

algorithms for integrating merging and compliance checking capabilities into site modeling methods to improve the model’s value for communication. The limited view of the range-finding instrument from a single reference point makes multiple sets of range data and their corresponding model subsets necessary in modeling a workspace. Merging all subsets from different locations into a world model of the workspace requires that transformations and merging take place at the range point, geometric primitive, convex hull, and subset levels.

Flash LADAR Approach Using sparse point clouds acquired with a single axis laser ranger can satisfy most of the requirements for local area 3D modeling. However, it cannot adequately handle moving actions of vehicles, workers, or equipment that are moving from one location to another. Therefore, the capability of Flash LADAR in the detection and efficient representation of static and moving objects (including people) could be used to complement the 3D graphical-modeling approach described here. Model subsets derived from sparse range point clouds are thereby integrated in practice with those acquired from Flash LADAR. Still, some of the disadvantages of the Flash LADAR, as common in all optical range scanning devices, are the need for line-of-sight and the influence of weather conditions on the quality of the scanned image. Weingarten et al. (12) demonstrated some promising preliminary results in using the Flash LADAR approach. The range information acquisition process through a Flash LADAR is faster and safer than a LADAR, because it does not require surveying or inspection personnel to set up and operate instruments in dangerous traffic zones. Mounted on a vehicle, Flash LADAR sensing does not require traffic lane closings and can provide real-time information on bridges, interchanges, railroad crossings, flyovers, power lines, off- and on-ramps, and building clearances, as seen in Figure 2. Having access to a rapid-generated 3D model, field personnel are instantly able to understand features and geometry they would have not seen from conventional surveying methods. 3D models generated from Flash LADAR range information increase options and mobility to make the best transportation decisions. For instance, Flash LADAR facilitates the analysis of highly congested areas to assess numerous existing structures quickly. This can expedite highway expansion, reconstruction of interchanges and bridges, and construction of drainage systems and improved sidewalks for pedestrians and bicycles (4).

Summary of Features, Capabilities, and Limitations of Existing Systems Dense point cloud, sparse point cloud, and Flash LADAR methods have different characteristics and produce different results. A meaningful comparison of the three modeling methods can be made on the basis of the following criteria: • Density of data used in modeling, • Frequency of updating of the derived model, • Precision and accuracy (how well the model reproduces the actual scene), and • Richness of the derived model (information quantity and quality incorporated into the model).

180

Transportation Research Record 1913

(a)

(b)

(c)

(d)

High Voltage Wire

Structure

Materials Division from Heavy Traffic Road

(e)

(g)

(f)

(h)

(i)

(j)

FIGURE 3 Sparse point cloud approach: (a) fitted and matched cuboid, (b) actual object, (c) fitted and matched pipe, and (d) actual object. Bounding box generation: (e) actual objects and (f ) bounded objects. Merging primitives can improve model display: (g) Step 1, no object recognition process; (h) Step 2, only object distinction process; (i ) Step 3, object distinction and object reconstitution process; and (j ) picture of scanned scene.

Figure 4 presents a schematic diagram of the existing methods according to these criteria. Whereas dense point cloud approaches are precise but slow and expensive, the sparse point cloud approach tends to achieve a compromise between accuracy and speed that is useful for some real-time field applications and that can be performed at a much lower cost (10). On the basis of the current technology development, the biggest and most unique advantage of using Flash LADAR is to track moving objects in real-time of up to 30 Hz. The given accuracy in ideal condition is in the range of millimeters and is expected to change in constructionlike environments. The Flash LADAR achieves an average-to-high-data density (data array of 128 × 160 points)

compared with up to 20 cumulative single measurements taken by single axis range finders used in the sparse point cloud approach. Ultimately, the array of measured points allows users to draw complete objects in real time. Because applications define the requirements for choosing sensing equipment, the scale for richness of the model is not quantitatively described. Table 1 summarizes the characteristics of the sparse point cloud, dense point cloud, and Flash LADAR approaches (7, 11, 12). Ideal values for the target applications in transportation and construction also are indicated in the table, described as follows. The field of view of the sensor must not be limited, but if limited, the sensor should

Teizer, Kim, Haas, Liapi, and Caldas

181

FIGURE 4 Schematic diagram to illustrate existing sparse and dense point cloud approaches [after Kwon et al. (8)].

TABLE 1

Ideal and Existing Attributes of Range Sensing Solutions

a

Leica, Riegl, Sick, etc.a Shape, static 232–904 nm 360°×310° 300 4 mm @ 50 m 0.001°–0.658° 1–0.0111 Hz 38×34×43 15 20–1,000 N/A

Leica Distometer or Total Stationa Dimension, Static 690 (670) nm Sparse point 200 (300) 1.5 mm @ 100 m N/A