3D Reconstruction from UAV-acquired Imagery for Road Surface Distress Assessment Chunsun Zhang Department of Geomatics, The University of Melbourne, Parkville VIC 3010, Australia Tel: +61 3 8344 9183; Fax: +61 3 9349 5185 Email:
[email protected] Ahmed Elaksher Faculty of Civil Engineering, Cairo University, Giza, Egypt Email:
[email protected] KEY WORDS: UAV, 3D Reconstruction, Road, Distress Assessment ABSTRACT: Road condition data are very important in transportation management system. Conventional data collection approach is time-consuming and labor intensive. This paper reports the developed techniques for 3D road surface reconstruction in support of unpaved road distress assessment using imagery acquired from a Unmanned Aviation Vehicle (UAV)-based imaging system. This system, produced in a project for rural road condition monitoring sponsored by the US Department of Transportation, consists of a low cost model helicopter equipped with a GPS/IMU, Flight Control System, a Ground Control System, and a non-metric camera. To generate road surface model, a coarse-to-fine hierarchical strategy with several image matching algorithms has been employed in this work. The road surface model is constructed by combining the matching results of feature points, normal image points and grid points. The matching strategy takes use of the short baseline of stereo imagery to avoid large image distortion while in 3D computation images with larger separation are exploited to benefit from the favorable geometry for image ray intersection. For grid point matching, a technique of cross-correlation along the vertical direction is developed. This approach simultaneously determines the correspondences cross images and the 3D position of the matched points. The coarse-to-fine strategy has been designed to initially reconstruct the surface model from matched feature points and normal image points. Afterwards, evenly distributed grid points are created, with the matched grid points improving the initial surface model in areas with poor or no texture. These dense 3D points reproduce detailed road surface which characterizes the details of the road distresses, thus facilitating road condition evaluation. Experiments have been conducted over several road segments and 3D models of different types of road distresses are reproduced to demonstrate the capability of the developed algorithms. 1.
INTRODUCTION
Efficient monitoring of transportation infrastructures is challenging in transportation management worldwide. The predominant method in conducting road condition survey and analysis is still largely based on extensive field observation by experts. Recent efforts include the development of a pavement survey vehicle coupled with sensor technologies and data-processing onboard. Some such systems have been used by highway agencies (Kenneth, 2004). Nevertheless, data collection using a moving vehicle still remains an expensive and troublesome survey, while cost and safety considerations require that it be done at regular intervals. Over last years, remote sensing technologies have been introduced for pavement assessment. The initial study has shown that the high ground resolution is critical in identification of road surface features. Aerial images can be a choice for high resolution road image source, but the limited maneuverability of the platform to acquire the image data and the associated high costs are shortcomings. Recently, UAV-based imaging system has been introduced into transportation for unpaved road condition evaluation (Zhang, 2009). UAVs are highly flexible, collecting image data at lower cost, faster and more safely. Moreover, UAVs are able to operate rather close to the object and acquire images with very high resolution, providing sufficient detail for identification and extraction of road condition parameters. This paper reports the developed technology for 3D reconstruction of road surface from UAV-acquired imagery in support of the road distress measurement and condition assessment. In the next section, the UAV-based imaging system architecture is briefly outlined. Then, the techniques of 3D reconstruction of rural road surface are discussed in detail. Experiments have been conducted and 3D reconstruction has been performed with image data collected in several mission over several road segments. Experimental results are presented and discussed. The paper concludes with discussion and outlook. 2.
SYSTEM ARCHITECTURE
The UAV-based imaging system consists of a UAV helicopter, on-board flight control system and ground
control station (GCS). The helicopter (Figure 1) features an electric engine. It has a payload of 15lb, and is capable to fly around 25 minutes with fully charged battery. This mini UAV can reach 200m above the ground and travel at a maximum speed of 10m per second.
GPS antenna
Computer
INS
Camera
Figure 1. UAV-based imaging system. The flight control system includes a GPS/ INS and an on-board computer. The computer fuses the GPS and INS observations in real time to navigate the helicopter. Thus, it supports both assisted mode for speed commands through a joystick and mission mode for pre-programmed flight trajectory tracking. An autopilot software is installed on the GCS computer, allowing for mission plan and guiding the flight course. During mission, the GCS can communicate with the on-board flight control system and automatically trigger the camera at the preset positions and acquire imagery. In this project, a Canon EOS Digital Rebel XTi digital camera has been used. The camera has an approximate focal length of 50mm, a 10.1Mpixel resolution (3888x2592 pixels), and a pixel size of 5.7µm. More details of the project and the UAV imaging system can be found in Zhang (2009). 3.
DATA ACQUISITION
Data collection using the above UAV system was conducted in summer over several unpaved road segments near the city Brookings, South Dakota, USA. The roads have usually been maintained well, and only have a few moderate distresses such as potholes or ruts. The UAV flew at an altitude of about 45m above ground, capturing details of the road surface with image scale of ~900. The ground resolution is about 5mm. The UVA travelled at 4m/s, acquiring road images with 80% overlap along the path. Figure 2 is an example of the road imagery collected over a road segment with ruts. This road segment was imaged four times in a single mission, providing highly redundant information for the evaluation of road condition, and also allowing for precise 3D measurement of road surface features. Thanks to the very high spatial resolution, the fine details of the road surface distresses are clearly presented in the image, allowing for detailed evaluation of the road surface condition.
Flight direction Figure 2. Example of road images acquired by UAV over a road segment. The segment was captured 4 times in a single mission. The flight directions are indicated by arrow lines. The bottom image shows the fine details of road feature (ruts) in original resolution.
4.
3D Reconstruction of Road Surface Distresses
After the flight mission, the acquired road imagery undergoes photogrammetric orientation, resulting in image block for road condition survey with orientation parameters for each image and 3D coordinate of points sparsely distributed on the road surface. This procedure is detailed in Zhang and Elaksher (2009). With the orientation parameters, 3D measurement of image features can be conducted manually on stereo and multiple images. To efficiently asses the road condition, automated procedures are necessary. One of the key procedures is 3D reconstruction of road surface which allows for three-dimensional visual inspection and supports automatic distress detection for road condition evaluation. An approach to process the UAV-acquired imagery to derive 3D road surface fully automatically using the combination of digital photogrammetry and computer vision techniques has been developed in this work. Since a point on a road is captured in consecutive images, thus, by reversing the imaging process, its 3D position can be computed through space intersection of the image rays (Figure 3). The fundamental process to automate this procedure is to locate the corresponding points in image space by automated image matching. O2
O1
a1 1
O3 a2
a3
2
3
A Figure 3. Imaging process and 3D reconstruction from images. Ground point A is imaged at a1, a2 and a3 in images 1, 2 and 3 respectively from perspectives O1, O2, O3. Point A can be reconstructed by the intersection of image rays of O1a1, O2a2, and O3a3. Automated image matching is an active research topic in photogrammetry and computer vision for automated mapping (Zhang, 2003), object construction and recognition (Chehata et al., 2009), and engineering applications (Wang et al., 2007; Chi et al., 2009). In this work, a coarse-to-fine hierarchical strategy with several image matching algorithms (Zhang and Gruen, 2004; Zhang and Fraser, 2008) has been employed. This strategy has been proved to be efficient to generate dense 3D points to reproduce detailed surface which is also necessary in this project to characterize the road distresses. The road surface model is reconstructed by combining the matching results of feature points, normal image points and grid points. The matching strategy takes use of the short baseline of stereo imagery to avoid large image distortion while in 3D computation images with larger separation are exploited to benefit from the favorable geometry for image ray intersection. For grid point matching, a technique of cross-correlation along the vertical direction is developed. This approach simultaneously determines the correspondences cross images and the 3D position of the matched points. The coarse-to-fine strategy has been designed to initially reconstruct the surface model from matched feature points and normal image points. Afterwards, evenly distributed grid points are created, with the matched grid points improving the initial surface model in areas with poor or no texture. The outcome is a road surface model with dense 3D points and this forms the detailed shape of the road surface. The feature points are distinct image points with rich texture. They are detected with the SIFT algorithm (Lowe, 2004) and are matched by the comparison of the point attributes. These points are also generated for automated image block orientation (Zhang and Elaksher, 2009). Afterwards, cross correlation is performed to the remaining points in the images. Geometric constraint is applied and the search for the conjugate points is restricted in a narrow region around the epipolar line. The matched points are then transferred to object space. 4.1 Matching of Normal Image Points Due to the inherent errors in the image orientation and the camera calibration, the 3D positioning accuracy is associated with the intersection angle. In the current system, the base to height ratio is small due to the use of the armature camera, resulting in small intersection angle in the immediately neighboring images. On the other hand, an object point is usually photographed in three or more consecutive images. Small intersection angles lead to higher uncertainty in 3D computation. Thus, for 3D reconstruction purpose, images with larger separation are in favor. However, with increasing separation, image distortion is large. This creates difficulty in
automated matching for point correspondences. To solve this problem and efficiently use the UAV-acquired images, image matching in immediately neighboring stereo images is first conducted (e.g. 1 and 2, 2 and 3 in Figure 3) to find the correspondences (e.g. a1 and a2, a2 and a3). Since the image distortion is small between images 1 and 2, the matching success rate is high. Thus, by fusing the matching results, the correspondences between images with larger separation is also determined (e.g. a1 in 1 and a3 in 3). For the computation of the 3D point A on the ground, a weighted multiple image ray intersection approach is employed which involves all the images containing point A. The weight is determined with the proportion to the image separation. Therefore, in the case shown in Figure 3, three image rays are intersected, while the highest weight is given to the image rays O1a1 and O3a3. This strategy efficiently explores consecutive images for image matching and 3D computation, thus providing reliable and accurate 3D model of road surface, necessary for measurement of surface distresses. 4.2 Matching of Grid Points The feature points and 3D points generated in the previous process deliver an initial road surface model. This model is further augmented with grid points. The grid points are expected to fill the void area in the initial surface model where the image texture is poor and the previous matching process fails. A regular grid is produced for the road segment. Therefore, each grid point has a fixed X and Y coordinates, and its initial Z coordinate can be derived from the generated initial surface model. To precisely determine the Z value, image matching with cross correlation technique along vertical direction has been developed. This is outlined in Figure 4. Assume A is a grid point with fixed horizontal coordinates XA, YA and initial Z value Z0. The vertical coordinate of A is ZA and is in the range (Z0-deltZ, Z0+deltZ). A is photographed at a1 and a2 in images 1 and 2, respectively. Since Z0 differs from ZA, the projections of (XA, YA, Z0) to images 1 and 2 (the black dots) deviate from a1 and a2 in the images. Thus, the image patches around the projections demonstrate large difference and therefore cross correlation on the image patches deliver a small coefficient which describes the level of similarity of the image signals. Cross correlation will achieve the largest value when Z0 is replaced with ZA. Based on this principle, the Z value from Z0-deltZ through Z0+deltZ is alternated progressively. At each step, cross correlation on the image patches around the projection points is performed and the correlation coefficients are recorded. The correct vertical coordinate of point A and correspondence across images based on the largest correlation coefficient is then determined. O2
O1 a2
a1 1
2 Z0+deltZ ZA Z0 Z0-deltZ A
Figure 4. Determination of Z coordinate of grid point A (XA, YA) with initial Z value Z0. ZA is in the range (Z0deltZ, Z0+deltZ). The above procedure delivers correspondences with pixel level accuracy. Finally a multiple image least squares matching (Elaksher, 2008) is conducted to further refine the matching results providing sub-pixel accuracy of point correspondences, thus further improving the quality of the Z coordinate of the grid points. In least squares matching, for a window of N points the intensity difference in each pair of images for each point should, ideally equal to zero. The previous statement can be stated as the condition equation in the least squares adjustment process. Equation 1 shows the mathematical form of this condition equation.
Gijpq = gijp − gijq = 0
(1)
where
Gijpq
is the least squares matching equation,
gijp , gijq are gray values for one pixel representing post (i,j) in images p and q respectively. The gray level in each image is the function of the position of the pixel. Hence, the adjusted pixel location could be determined. This is achieved by minimizing a goal function, which measures the differences between the gray levels in the template and the patch images. The goal function to be minimized is the L2-norm of the residuals of the least squares estimation. The above process is repeated for each grid point and thus produces high quality road surface model with dense grid points. The density of the model can be adjusted by changing the interval of the grid, thus this approach is also suitable for various engineering applications requiring different resolutions of 3D surface data. 5.
EXPERIMENTAL RESULTS AND DISCUSSIONS
The developed 3D reconstruction algorithms were applied to the acquired imagery for evaluation of road surface distresses, such as ruts and potholes. Figure 5 (left) shows an image patch of rut on a rural road section. Field survey with tape shows the distress is mild, with the depth around five centimeters in the most severe area. The ruts are caused by excessive heavy traffic. The rut area is dark and several ruts are observed with the largest one in the middle of the image. Shadows are present in the image and cover some parts of the ruts. The image resolution is very high, even the tyre prints are clearly visible. Image matching was then performed using the techniques described in the previous section. With the determined image orientation parameters, the 3D positions of the road points were computed. This resulted in dense 3D points allowing for measurement of the shape and size of the ruts as shown on the right of Figure 5. It can be seen the ruts were successfully reconstructed by the proposed approach. They are visible in the 3D model. However, the model is not complete and some errors still exist. Shadows create problem in image matching. Shadow area cannot be matched and can cause mismatches, resulting incorrect 3D points. Shadows can be avoided by selection of a proper imaging period, but this might not be realistic for practical application in routine transportation management. Mismatches are also caused by low variation of image gray scale and the lack of texture in some areas. The very high resolution also poses a challenge in selection of window size in image matching.
Figure 5. 3D reconstruction of rut. Left: an image patch of rut on a road. The blue dots represent grid points. Right: reconstructed 3D model of the rut area. The similar procedures were applied to a road segment with mild potholes (Figure 6). This road is constructed by specially treated gravel, resulting in a hard and compact surface. The pothole is caused by frequent collection of water on road. The depth of the potholes is just around three to four centimeters measured in field with tape. However, if it is not repaired promptly, it continues to collects water and gradually gets larger and deeper, progressively deteriorating road foundation and creating safety issue. The 3D model of the pothole is demonstrated on the right of Figure 6. Visual inspection reveals the pothole is generally reconstructed. The process tends to smooth the slope of the pothole. Again, the lack of the image texture and the low variation of image gray level cause some mismatches and thus incorrect 3D reconstruction.
Figure 6. 3D reconstruction of pothole. Left: an image patch of pothole on a road. Right: reconstructed 3D model of the pothole. 6.
CONCLUDING REMARKS
This paper reports the developed techniques for 3D road surface reconstruction in support of road distress assessment using imagery acquired from a UAV-based imaging system. The road surface model is constructed by combining the matching results of feature points, normal image points and grid points following a coarse-tofine hierarchical strategy. In image matching, short baseline of stereo imagery are used to facilitate locating point correspondences, while in 3D computation images with larger baseline are exploited to achieve higher accuracy. For grid point matching, a technique of cross-correlation along the vertical direction is developed. This approach simultaneously determines the correspondences cross images and the 3D position of the matched points. The matched grid points improve the surface model in areas with poor or no texture. Experimental results show the developed approach is capable to reproduce 3D models of different types of rural road distresses. The generated 3D models are currently evaluated by visual inspection due to lack of precise reference data. Nevertheless, these 3D models, coupled with other image processing techniques, can definitely facilitate automated road condition evaluation and thus improving the practice of transportation management. Some errors still exist in the models, largely due to shadows, mismatches caused by low gray scale variation of road surface and lack of texture. To fully benefit from the very high resolution of imagery from UAV system for 3D production, the selection of window size in image matching is an issue. These constitute the future research. 7.
ACKNOWLEDGEMENTS
This work is supported by the US Department of Transportation under grant DTOS59-07-H-0007. 8. REFERENCES Elaksher, A., 2008. A multi-photo least squares matching algorithm for urban area DEM refinement using breaklines, International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Beijing, China, CDROM. Chehata, N., Jung, F. and G., Stamon, 2009. A graph cut optimization guided by 3D-features for surface height recovery. ISPRS Journal of Photogrammetry and Remote Sensing, 64(2), 193-203. Chi, S., Caldas, C.H., D.Y., Kim, 2009. A methodology for object identification and tracking in construction based on spatial modeling and image matching techniques. Computer-Aided Civil and Infrastructure Engineering, 24 (2009), 199-211. Kenneth, H.M., 2004. NCHRP synthesis of Highway practice 334: automated pavement distress collection techniques. Transportation Research Board, National Research Council, Washington, D.C., 94p. Wang, K.C.P., Hou, Z., Watkins, Q.B. and S.R., Kuchikulla, 2007. Automated imaging technique for runny condition survey. Proc. of 2007 FAA Worldwide Airport Technology Transfer Conference, CDROM. Zhang, C., 2003. Towards an operational system for automated updating of road databases by integration of imagery and geodata, ISPRS Journal of Photogrammetry and Remote Sensing, 58(3/4), 166-186. Zhang, C., 2009. UAV-based remote sensing of road condition. Procs. Of ASPRS annual conference. March 913, Baltimore, Maryland. CDROM. Zhang, C., 2009. Photogrammetric processing of low-altitude UAV imagery. Procs. Of ASPRS annual conference. March 9-13, Baltimore, Maryland. CDROM. Zhang, C., Fraser, C.S., 2008. Generation of Digital Surface Model from High Resolution Satellite Imagery. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 3-11 July, Beijing, China. CDROM. Zhang, L. and Gruen A., 2004. Automatic DSM generation from linear array imagery data. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol 35 (Part B3), pp. 128-133.
