Real-time Tracking of Highway Linear Features Dorota A. Grejner-Brzezinska, Charles K. Toth1 and Qian Xiao Department of Civil and Environmental Engineering and Geodetic Science 470 Hitchcock Hall, 2070 Neil Avenue Columbus, Ohio 43210 Tel: 641-292-8787; Fax: 614-292-2957 E-mail:
[email protected]
BIOGRAPHY Dr. Dorota A. Grejner-Brzezinska is an Assistant Professor at the Department of Civil and Environmental Engineering and Geodetic Science, The Ohio State University. Prior to that she was a Research Specialist at the OSU Center for Mapping. She received an MS in Surveying and Land Management from the University of Olsztyn, Poland, and an MS and a Ph.D. degree in Geodesy from The Ohio State University. Her research interests cover precise kinematic positioning with GPS, GPS/INS integration for direct platform orientation, mobile mapping technology, and robust estimation techniques. Dr. Charles Toth is a Research Scientist at the Ohio State University Center for Mapping. He received an MS in Electrical Engineering and a Ph.D. in Electrical Engineering and Geoinformation Sciences from the Technical University of Budapest, Hungary. His research expertise covers broad areas of 2D signal processing, highresolution spatial imaging, surface extraction, modeling, integrating and calibrating of multisensor systems, multi-sensor geospatial data acquisition systems, and mobile mapping technology. Qian Xiao is currently a Master student, specializing in Geographic Information Systems (GIS) & Computer Mapping at the Department of Civil and Environmental Engineering and Geodetic Science, The Ohio State University. In 1992, he earned a M.S. in Computing Science and Mathematics from Sichuan University, Chengdu, Sichuan, China. ABSTRACT Road surface markings such as centerlines and lane marks are essential parts of the transportation network; their condition is critical for traffic safety. 1
Therefore, the position and the visibility of lane marks are of critical importance for any road maintenance activities. This paper presents the design of a multisensor system aimed at the precise positioning of lane marks with the major focus on the systems design, calibration, and preliminary performance analysis of a prototype system. A new approach to the application of navigation data to real-time processing of imagery is presented. 1. INTRODUCTION Direct georeferencing of imaging sensors by means of integrated GPS/INS continues to generate an increasing interest in the surveying/mapping, and remote sensing communities (He et al., 1994; Bossler and Toth, 1995; El-Sheimy and Schwarz, 1995; Schwarz, 1995). On one hand, the primary driving force behind this process is a need to accommodate the new spatial data sensors, such as LIDAR or SAR (airborne systems). On the other hand, a substantial cost decrease, a possibility of data reduction automation, and a short turn-around time are the most attractive features offered by this technology. The Ohio State University is currently developing a GPS/INS/CCD integrated system for precise (centimeter level) monitoring of highway center and edge lines, sponsored by the Ohio Department of Transportation. The prototypepositioning component of the system is based on a tight GPS/INS coupling, and the imaging component comprises a single down-looking, highperformance color digital camera. The high image capture rate (up to 15 frames per second) provides sufficient overlap of the subsequent images and thus, it allows for stereo data processing, which, to a large extent, is performed in real-time. From a navigation standpoint, the post processing of GPS/INS data provides more accurate orientation, as a benefit of forward and backward trajectory processing and precisely synchronized timing information. However, some navigation data
The Ohio State University Center for Mapping, 1216 Kinnear Rd., Columbus, OH 43212,
[email protected]
available in real-time (such as relative image orientation) can be used to process direct-digital stereo-pairs on-the-fly by extracting and storing only the necessary information, as opposed to the entire image (if only simple features, such as linear objects are needed). The precise time synchronization of image capture to the navigation estimates is one of the most challenging tasks of the real-time image pre-processing. Obviously, further post-processing can rectify the positioning and orientation data that should subsequently be used to provide precise georeferencing to the features extracted in real-time. This procedure adds more robustness to the system, allowing faster and more automatic data processing, saving storage space and processing time, as data acquisition can be combined with the image pre-processing. 2. MULTISENSOR SYSTEM CONCEPT The high-accuracy GPS/INS/CCD system designed for monitoring linear highway features is based on the concept of the sensor integration, combining post-processing with real-time image processing. The two primary components of the mobile mapping system currently being implemented are precise navigation and digital imaging, both allow for flexible and optimal system design, leading potentially to near-real time overall data processing. The navigation component, in essence, follows the structure of the AIMS system, developed earlier at the Center for Mapping, OSU (Toth, 1997). The imaging system provides the much-needed connection between the high-precision vehicle navigation data and the marks on the road surface, thus relieving the driver from the stress of having to drive a pivot wheel precisely on the lines as former mechanical systems required. For economic reasons, the processing of the images must be automated, preferably performed in real-time. Figure 1 shows the concept of the centerline mapping system. ZINS ω -YINS H Centerline
Road surface Y
Figure 1. Sensor geometry of the mapping system.
Under ideal conditions, the centerline offset from the vehicle position can be easily obtained from monoscopical image measurements, provided the vehicle geometry and the camera model are known. Vehicle position/attitude changes due to its motion and road unevenness introduce at the same time non-negligible errors in this model. To compensate for these errors, a stereo technique can be used, which can be easily achieved by acquiring overlapped imagery. Since a single camera collects images, the stereovision is realized by the platform motion, which, in turn, emphasizes the need for high-precision sensor orientation. 3. HARDWARE IMPLEMENTATION The prototype of the integrated GPS/INS/CCD system (Figure 2) designed for precision monitoring of the highway edge- and centerlines comprises two dual-frequency Trimble 4000SSI GPS receivers and a medium-accuracy and high-reliability strapdown Litton LN-100 inertial navigation system, based on Zero-lockTM Laser Gyro (ZLGTM) and A-4 accelerometer triad (0.8 nmi/h CEP, gyro bias – 0.003°/h, accelerometer bias – 25µg). The LN100 firmware version used in this project allows for access to the raw IMU data, with an update rate up to 256 Hz. Estimation of errors in position, velocity, and attitude, as well as errors in inertial and GPS measurements, is accomplished by a 21-state centralized Kalman filter that processes GPS L1/L2 phase observable in double-differenced mode together with the INS strapdown navigation solution. The estimated standard deviations are at the level of 2-3 cm for position coordinates, and 5-7 arcsec and ~10 arcsec for attitude and heading components, respectively. The imaging component is built around the Megaplus ES 1.0/MV Kodak digital camera or its color version with 9.07mm by 9.16 mm imaging area (9-micron pixel size) and 15 images per second acquisition rate (15 Hz), which allows for 60% overlap in image acquisition at normal highway speed. For testing and performance evaluation of the positioning component, a digital camera based on a 4,096 by 4,096 CCD with 60 by 60 mm imaging area (15 micron pixel size), manufactured by Lockheed Martin Fairchild Semiconductors was used. The imaging sensor of this experimental configuration, see Figure 2, is integrated into a camera-back (BigShot™) of a regular Hasselblad 553 ELX camera body, and the camera is installed on a rigid mount together with the INS (Toth, 1998).
Digital camera GPS antenna
Image j
Image j+1
Edges
Edges
Edge structures
Edge structures
Edge detection
Affine transformation
Collinearity
Matching
Sequential Orientation
Line Structures
Line Structures
LN 100 Affine tranformation
Tracking/Connectivity
2D Block System
Line Annotation Transfer to GIS
Figure 2. Hardware configuration of the prototype. Figure 3. Image processing workflow. The system design, including the sensors, dataflow, processing steps, except for the real-time image processing module is similar to the AIMS system, mentioned earlier and was presented in (Grejner-Brzezinska, 1998; and Toth and GrejnerBrzezinska, 1998). 4. EMULATION RESULTS FOR THE REALTIME IMAGING SYSTEM To assess the feasibility of automated line extraction with 3D positioning and consequently its real-time realization, a complete implementation of all the necessary image processing tasks was developed in a standard C++ programming environment. Figure 3 shows the overall dataflow and processing steps, which will be illustrated in more detail later. In short, the real-time image processing is feasible due to a simple sensor assembly and the limited complexity of the imagery collected. First, a single down-looking camera acquires consecutive image pairs, which cover only the road surface to the side of the vehicle; therefore, the object contents of the images are rather simple and predictable. In other words, the road surface is flat, the road edges, usually defined by solid or dashed bright lines, are parallel to the driving direction. As a result, the feature extraction (in this case center- and edge lines only) can be easy and reliable. For example, the edge detection algorithm would search for a sudden change in the grayscale pixel values in the direction perpendicular to the driving direction. Second, the simultaneous availability of navigation data, basically the change in position and attitude between two image captures – relative orientation in photogrammetric terms – has a dramatic impact on the search time for conjugate entities on image pairs since the usually two-dimensional search space is reduced to one dimension.
Color Separation Traffic signs, centerlines and the like have distinct colors to draw attention of drivers in an unambiguous way. Therefore, color images are preferred over monochromatic ones. Figure 4 illustrates an extreme case where the yellow solid lines are hardly visible in the B/W image. A simple histogram analysis in the RGB (Red, Green, Blue) color space easily reveals two peaks in the red and green channels representing the yellow color of the centerline.
Figure 4. Difficult to see centerlines in B/W image. Although there are a great many image processing algorithms working in various color data (multichannel gray-scale imagery), the great majority of the core functions work on simple monochrome image data. Therefore, if possible, a color space conversion is desirable. For example, moving from RGB to IHS (Intensity, Hue, Saturation) space can effectively decouple intensity and color information. After some experiments, we decided to use an RGB to S transformation, which is illustrated in Figure 6 under various road conditions. Obviously, dealing with one channel has a major benefit for the real-time implementation.
Scan line Raster line P Centerline Figure 6. Centerline, scan line, raster line, and midpoint.
Figure 5. Original image (top) vs. color space transformed version.
Centerline Extraction After the RGB to S transformation, centerlines are extracted from the raster images. Under the given dimensions of the mapping vehicle and the geometry of the camera, the centerlines, in transportation terms, are showing up in the images as of multi-pixel-width lines (20-30 pixels wide), usually referred to as ‘raster line’. All other terms for lines such as centerline, scan line, refer to onepixel-width lines, see Figure 6. For a raster line, its centerline is of primary interest. In some literature, similar terms such as skeleton (Pavlidis 1982, Murthy 1974, Nguyen 1986), or medial line (medial axis) (Montanvert 1986) are used. Skeleton is more like a one-pixel-width line, while centerline can be used to express a one-pixel-width line in both raster and vector lines. The mathematical definition of the centerline of a raster line varies depending on the algorithms used to generate the centerline. In thinning algorithms, the skeleton is a collection of such pixels, which has more than one nearest neighbor to the boundary of a raster line (Pavlidis 1982). The skeleton is then extracted by shrinking the raster line from its boundary in all directions until the one-pixel-width eight-connected line remains. In the medial axis transformation method (Montanvert 1986), the discrete medial axis pixels are the local maximum of a transformation value.
For centerline extraction, we selected a scan lineoriented method, which is simple and executes faster than other algorithms. In this one-pass process, the computation is linear to the number of pixels on a raster line. Taking advantage of the centerline direction, the optimal scan line direction, which is perpendicular to the centerline, can be easily achieved for most situations. During the scanning, the pixels along a scan line, which is a small segment of an image column, are processed in a top-to-down fashion. A robust recursive filtering technique eliminates noise such as gaps or smaller gray-scale irregularities, as well as provides segmentation for multiple centerlines such as double solid lines. Once midpoints, P0, P1,…, Pk are extracted, a line-following routine can generate centerline pixels. Figure 7 illustrates automatically extracted centerlines (displayed over the original images).
Figure 7. Automatically extracted centerlines. To achieve the highest accuracy possible, the 3dimensional centerline positions must be obtained from stereo imagery. Knowing the camera orientation, both interior and exterior, and the matching (identical) entities between the 2dimensional centerlines, the 3-dimensional centerline position can be easily computed. Since the external orientation is provided by the
navigation data, and similarly the interior orientation can be determined by a priori calibration, the primary task is reduced to finding conjugate points or features in overlapping images. Since centerlines are subject to shift invariance, they cannot be directly used for matching purposes. There is a number of methods used for image matching, including feature-based and area-based techniques (Toth and Schenk, 1992). Because of the special condition of the object space – near parallel planes – a simple correlation-based area method seems to be adequate for this purpose. For matching image primitives, feature points are considered.
Matching Feature Points
Feature Point Extraction
c( s1 , s 2 ) = λ
Feature points correspond to high curvature or high gray level variety points. For simplicity, a corner detector, devised by Harris and Stephens was selected to extract feature points, which is based on the following operator: R(x, y) = det[C] – k trace2[C] Where C is the following matrix
é ^2 I C = ê ^x ê ëê I x I y
^
IxIy
ù
^
I y2
and I denotes the smoothing operation on the gray level image I(x, y). Ix and Iy indicate the x and y directional derivatives respectively. Figure 8 depicts feature points extracted around the centerline region from overlapping images.
The matching of the feature points is accomplished through correlation. The search space is constrained by the availability of epipolar geometry. For a given feature point s, a correlation window of size n×m is centered at its image 1 location, at point s1. Then a search window around the approximated location of the same object point s in the second image is selected, point s2, and the correlation operation is performed along the epipolar line. The search window size and location are determined by navigation data. The correlation score is defined as n
m
i =1 j =1
[ I 1 (u1 + i, v1 + j ) − I1 (u1 , v1 )] ⋅
[ I 2 (u 2 + i, v2 + j ) − I 2 (u 2 , v2 )] Where Ik(uk, vk) is the average at point (uk, vk) of Ik (k=1, 2), and λ is a normalizing factor so that the score ranges from –1, for two correlation windows which are not similar at all, to 1, for two correlation windows which are identical. A point in the first image may be paired to several points in the second image. Several techniques exist for resolving the matching ambiguities. Due to the special case scenario of a near planar object surface, a 6parameter affine transformation provides the geometrical relation between two images. Therefore, by calculating the affine transformation parameters by conjugate points, straightforward blunder detection can be effectively used for disambiguating matches and removing outliers. Strip Formation After determining the transformation parameters between consecutive images, the centerline segments are connected and an approximate centerline can be incrementally formed. However, the final coordinates of centerline can be computed only in post-processing mode once the final navigation data have become available. To illustrate the fit between images, an image strip was built by transforming five consecutive images into the same frame as shown in Figure 10. 5. POSITIONING PERFORMANCE
Figure 8. Feature points extracted in the overlap area.
The multisensor system calibration is a key task to achieving the ultimate accuracy of the given sensors. System calibration is defined here as the determination of spatial and rotational offsets between the sensors (GPS/INS lever arm and INS/camera boresight misalignments) as well as
imaging sensor calibration. Continuous calibration of the INS system is provided by GPS and thus is very dependent on GPS anomalies such as satellite signal obstructions, multipath, interference, etc. To demonstrate the importance of proper system calibration, the boresight misalignment determination and the overall accuracy assessment are briefly discussed here. For system calibration and for performance assessment of the positioning component, a 4K by 4K digital-sensor equipped Hasselblad camera with a 50-mm focal length lens, tilted downwards by 5°, was mounted rigidly on the top of the LN100 in side-looking position; the offset from the GPS antenna was about ~1 m, as shown in Figure 2. Imagery with various overlap was collected at a calibration wall and along a surveyed road in several passes. The effective ground pixel size was about 2-4 mm. Figure 9 shows the calibration range, and the road area with control points is depicted in Figure 11.
Figure 11. Test road area. Table 1 shows the aerial triangulation results for six images. The residuals for the photo centers are less than one cm. Differencing the exterior orientation data with the navigation solution resulted in the boresight misalignment for the digital camera with about 2-3 cm offset and about 20 arcsec attitude accuracy. The ultimate accuracy test for the overall system performance is the comparison of ground coordinates obtained by the photogrammetric methods from the directly oriented imagery (using the above mentioned boresight parameters) to the GPS-measured positioning results. The control points used in this test were determined with an accuracy of ~1.5 cm per coordinate and were located about 18 m from the perspective center of the camera. The comparison performed on 4 control points is presented in Table 2.
Figure 9. Calibration range.
More details on the system calibration and positioning performance are available in (GrejnerBrzezinska and Toth, 1999, and 2000).
Figure 10. Automatically formed image strip from five overlapping images.
Point ID 14 15 24 25 34 35 RMS
E [m]
N [m]
H [m]
E residual [m]
N residual [m]
H residual [m]
553789.008 553789.426 553788.985 553789.407 553788.974 553789.390
221908.411 221909.427 221908.418 221909.438 221908.415 221909.437
212.473 212.448 211.366 211.356 210.272 210.248
-0.003 -0.006 0.010 0.009 -0.006 -0.004
-0.001 0.002 0.000 -0.002 0.002 0.000
0.002 0.000 -0.002 -0.001 0.001 0.000 0.001
0.007
Table 1. Aerial triangulation results of calibration wall images. Point 1 2 3 4 RMS
Object distance 17.25 16.30 18.57 17.86
Easting [m] 0.002 0.009 -0.019 -0.059 0.031
Northing [m] 0.029 0.015 0.029 0.018 0.007
Height [m] 0.008 0.000 0.010 0.009 0.005
Table 2. Coordinate difference between control points measured on the imagery and the ground truth.
7. SUMMARY AND CONCLUSIONS This paper introduced a concept of all-digital mapping system designed for precise mapping of highway linear features. The test results presented here indicate that an integrated, land-based system supported by a medium to high quality strapdown INS and dual frequency differential GPS offers the capability for automatic and direct sensor orientation of the imaging sensor with high accuracy. In addition, the concept of real-time extraction of highway linear features such as centerlines was demonstrated. The overall system performance was extensively tested by a prototype positioning module (for more details see References 4-8 and 16-17), while the feasibility of automated feature extraction was evaluated only by simulations. To assess the automation potential of highway linear feature extraction, various algorithms have been tested on two data sets with different roadway and roadway mark conditions. Based on performance, including both success rate and execution time, a group of processing functions has been selected. This final workflow incorporates the following processing steps: 1) color space transformation of input images, 2) scanning for centerlines, 3) feature point extraction around the centerlines, 4) feature point matching for connecting images, and 5) establishing an affine transformation between consecutive images with blunder detection. Although the algorithms are computation-intensive, a convincing performance has been achieved for diverse test data sets, proving that highway linear features can be extracted in a
totally automated way and thus can be implemented in real-time as on-the-fly processing. Another key factor of the successful real-time implementation, in additon to using a powerful computer system, is the quality of the estimates for the relative orientation of consecutive images. Therefore, by providing accurate navigation data online, the search space for time-intensive matching can be cut, resulting in substantially reduced execution time. The GPS/INS-based positioning component of the mapping system has been tested under normal operational conditions. The internal estimates, computed by the Kalman-filter showed standard deviations for pitch and roll at 5-8 arcsec level, while the heading standard deviation ranged between 8-12 arcsec, depending primarily on vehicle dynamics. With the help of a highresolution camera, the achieved accuracy in terms of measuring point coordinates through photogrammetric technique in object space was in the 1-3 cm range. It should be emphasized that the system calibration is essential in exploiting the accuracy potential of the system, since due to the nature of the direct orientation technique, there is no feedback in the processing, which would compensate for possible systematic errors. The highway test indicated that loss of lock is the single most critical threat to maintaining consistent performance. In fact, that is the primary reason why a high-performance INS was selected as it can effectively bridge the gaps when GPS data are not available. The INS attitude performance is not that critical for the imaging sensor due the rather large photo scale.
This research was partially supported by the Ohio Department of Transportation research grant.
12. Nguyen, Thinh V. and Sklansky, Jack (1986): A fast Skeleton-Finder for Coronary Arteries, IEEE Transactions on Pattern Analysis and Machine Intelligence, 481-483.
REFERENCES
13. Pavlidids, T. (1982): Algorithms for graphics and Image Processing, Computer Science Press.
ACKNOWLEDGEMENTS
1.
Bossler, J. D., and Toth, C., (1995): Accuracies Obtained by the GPSVan, Proc. GIS/LIS’95, Vol. 1., pp. 70-77.
2.
El-Sheimy, N., Schwarz, K. P., Wei, M. (1995): VISAT: A Mobile City Survey System of High Accuracy, Proc. ION GPS, pp. 13071315.
3.
El-Sheimy, N. and Schwarz, K. P. (1999): Navigating Urban Areas by VISAT – A Mobile Mapping System Integrating GPS/INS/Digital Cameras for GIS Application, Navigation, Vol. 45, No. 4, pp. 275-286.
4.
Grejner-Brzezinska D. A. (1998): “Direct Platform Orientation with Tightly Integrated GPS/INS in Airborne Applications”, Proceedings of ION GPS, pp. 885-894
5.
Grejner-Brzezinska D. A., Da, R., Toth C. (1998): GPS Error Modeling and OTF Ambiguity Resolution for High-Accuracy GPS/INS Integrated System, Journal of Geodesy, 72(11), pp. 628-638.
6.
Grejner-Brzezinska D. A. (1996): Positioning Accuracy of the GPSVanTM, Proceedings of 52nd Annual ION Meeting, Boston, June 1921, 1996, pp. 657-665.
7.
Grejner-Brzezinska, D. A., Toth, Ch.K., Direct Platform Orientation in Aerial and Land-based Mapping Practice, International Archives of Photogrammetry and Remote Sensing, Vol. XXXII – 2W1, 5W1, 5/3W, pp. 2-4/1-7, 1999.
8.
Grejner-Brzezinska, D. A., Toth, Ch.K. (2000): Precision Mapping of Highway Linear Features, International Archives of Photogrammetry and Remote Sensing, Vol. XXXIII – B2, pp. 233-240.
9.
He, G.P., Novak, K., and Tang, W. (1994): The Accuracy of Features Positioned with the GPSVan, Symp. ISPRS Comm. II, Vol. 30, Part 2, pp. 480-486.
10. Lam, L. (1992): Thinning Methodologies, IEEE Trans. On Pattern Analysis and Machine Intelligence, 14(9) 869-885. 11. Murthy, I.S.N. and Udupa, K.J. (1974): A search algorithm for skeletonization of thick patterns, Computer Graphics and Image Processing, (3) 247-259.
14. Schwarz, K. P. (1995): INS/GPS as a Georeferencing Tool for Multi-Sensor Systems, presented at the Mobile Mapping Symposium, May 24-26, Columbus, OH. 15. Toth, Ch.K. and Schenk, T. (1992): FeatureBased Matching for Automatic Image Registration, ITC Journal, pp. 40-46. 16. Toth, C., Grejner-Brzezinska, D. A. (1998): Performance Analysis of the Airborne Integrated Mapping System (AIMS™), International Archives of Photogrammetry and Remote Sensing,Vol. XXXII, part 2, pp.320326. 17. Toth, Ch.K. (1997): Direct Sensor Platform Orientation: Airborne Integrated Mapping System (AIMS), International Archives of Photogrammetry and Remote Sensing, ISPRS Comm. III, Vol. XXXII, part 3-2W3, pp. 148—155.
