Automation in Construction 8 Ž1999. 581–588
Automated detection of surface defects in water and sewer pipes Osama Moselhi ) , Tariq Shehab-Eldeen Department of Building, CiÕil and EnÕironmental Engineering, Concordia UniÕersity, 1455 de MaisonneÕue BouleÕard west, Montreal, Quebec, Canada H3G 1M8 Accepted 8 December 1998
Abstract Automation is gaining momentum in industry, particularly in rehabilitation and inspection works of underground infrastructure facilities. This paper describes a model for automating inspection and identification of surface defects in underground water and sewer pipes. The paper describes the current efforts in identification of surface defects in underground water and sewer mains, and presents an automated system designed to assist infrastructure engineers in diagnosing defects in this class of pipe networks. It describes the general architecture of the system and its basic components, and focuses primarily on four modules designed for automating image acquisition, image processing, features extraction and classification of defects. q 1999 Elsevier Science B.V. All rights reserved. Keywords: Neural networks; Automation; Pattern classification; Pipe defects; Sewer pipes; Water pipes; Rehabilitation
1. Introduction Considerable efforts have been made in construction automation w6,7,10,13x, including rehabilitation works w2,14x. Automating construction and rehabilitation processes is driven by the need for more controlled environment, cost reduction and higher quality and safety performance. A typical maintenance or rehabilitation process usually starts by collecting information about the utility itself. This set of data usually highlights many aspects and provides useful information about the condition of the utility itself such as the presence, number and location of defects. closed circuit television ŽCCTV. cameras
)
Corresponding author. Tel.: q1-514-8483190; fax: q1-5148487965; e-mail:
[email protected]
either mounted on robots or winched between two manholes, to scan the inner surface of a pipe, are used to capture these data. Currently, this process ends up by having a videotape that has to be visually and manually inspected in order to identify and locate the defects in a pipeline. The process is usually time consuming, tedious and expensive. It may also lead to diagnostic errors due to lack of concentration of human inspectors. Clearly, if this identification process can be automated, then not only significant time and money can be saved, but also higher accuracy and consistency in diagnostics can be achieved. Automating these processes can also provide an incentive for checking water and sewer mains regularly as a part of a preventive maintenance program. This can help municipality engineers to plan ahead and avoid unpleasant surprises. Automation in this field usually involves the application of computer vision, digitized
0926-5805r99r$ - see front matter q 1999 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 6 - 5 8 0 5 Ž 9 9 . 0 0 0 0 7 - 2
582
O. Moselhi, T. Shehab-Eldeenr Automation in Construction 8 (1999) 581–588
video images, image processing and analysis, pattern recognition and classification using neural networks ŽNNs. w2,8,14,15,17x. This paper describes methods used in identification of surface defects in underground water and sewer mains, and presents an automated system designed to assist infrastructure engineers in diagnosing defects in water and sewer lines. The system exhibits a number of interesting features: it utilizes image analysis, pattern recognition, neural networks and efficient techniques in processing digitized video images to produce a report that highlights the locations and types of defects in underground pipe lines. The paper describes the general architecture of the system and its basic components, and focuses primarily on four modules designed for automating image acquisition, image processing, features extraction and classification of defects.
2. Proposed system Fig. 1 illustrates the main components and modules of the proposed automated system. A CCTV camera scans the inner surface of a pipe and produces a videotape which is then played back using a VCR. The VCR feeds the information captured on the tape to a computer equipped with a frame grabber, image analysis and NN software packages. The frame grabber captures and digitizes the frames of the acquired images. The image analysis software,
Image Tool, Ždeveloped at the University of Texas Health Science Center at San Antonio, TX and available from the Internet by anonymous FTP maxrad6.uthscsa.edu. analyzes those captured frames and processes them in a manner so as to prepare a suitable input to a neural network pre-designed and trained to detect and classify the defects encountered. NeuroShell 2 is used for the design and training of the NN. The trained NN utilizes the attributes obtained from the image tool software. Once the classification process is completed, the output is forwarded to a printer where an analysis report is produced. As such, the proposed system involves four major steps: Ž1. image acquisition; Ž2. image processing; Ž3. image segmentation and features extraction; and Ž4. pattern classification. This paper focuses primarily on image analysis techniques and describes the three steps leading to the pattern classification stage Ži.e., image acquisition, image processing, image segmentation and features extraction.. 2.1. Image acquisition The first step in any rehabilitation process is to collect as much data as possible about the piping system. This is accomplished by using a CCTV camera mounted on a small robot. Different cameras and robots are now available in the market and they differ in size and capability. For applications concerned with pipe investigation, it has been recom-
Fig. 1. Schematic arrangement for the automated system.
O. Moselhi, T. Shehab-Eldeenr Automation in Construction 8 (1999) 581–588
Fig. 2. Smoothed image—averaging.
mended to use black and white cameras rather than colored ones w11x. The pictures captured in black and white are sharp and have good image contrast w11x. More details about the different types of robots and CCTV cameras can be found in Ref. w11x. 2.2. Data preparation NNs are recognized for its superior performance in pattern recognition and classification capabilities w4,9,14,16x. They usually consist of an input layer, an output layer and one or more hidden layers with each layer consisting of one or several neurons. As a rule of thumb, the number of neurons in the input layer has to be minimized so that the computation and conversion speed can be maximized w1x. Reducing the number of neurons can also help in improving the learning process of NNs w1x. The data captured on video images are used in this paper for training a neural network to detect and
Fig. 3. Smoothed image—median.
583
Fig. 4. Original image.
classify pipe defects. A typical video image may consist of 512 = 512 pixels. If this image is to be processed using NNs, then at least three alternatives can be considered. First is to digitize and feed one image or frame at a time with its huge number of pixels to the NN. Second, is to digitize and compress the video image or frame before feeding it to the NN. The third is to extract some feature victors that represent the different objects in the image and then feed them to the NN. Clearly, the first alternative is impractical since one neuron will be needed for each single pixel in the image. This will require a huge number of neurons in the input layer that cannot be handled efficiently by the NN. The second alternative might also be impractical for some applications since little details such as small cracks might be lost due to the nature of the compression process itself andror to the limited capabilities of the software system used
Fig. 5. Edge detection.
O. Moselhi, T. Shehab-Eldeenr Automation in Construction 8 (1999) 581–588
584
Fig. 6. Negative image.
in the process. The third alternative appears to be promising and will be considered in subsequent developments of this paper. That alternative has been found useful in pattern classification using NNs w5,8,15x. The technique basically minimizes the amount of data that has to be fed to a neural network and, accordingly, reduces significantly the number of neurons in the input layer of that network. It ultimately results in improving the learning speed as well as the classification capabilities of the network. 2.3. Image processing Image processing, in this paper, refers to the different techniques that are applied to an image so as to improve its quality for analysis purposes. Some of the useful techniques are: smoothing, edge detection, inverse transformation and thresholding w3,12x. Smoothing involves a number of operations used to reduce the effects that may be present in an image due to poor transmission or acquisition techniques.
Examples of these operations are neighborhood averaging and median filtering w3,12x. In the neighborhood averaging technique, a grid of 3 = 3 pixels is considered and the gray level of the center pixel is replaced by a new gray level that equals to the average of the gray levels of the original pixel and its eight nearby pixels. Median filtering technique, on the other hand, replaces the gray level of the center pixel by the median of the gray levels of the original pixel and its eight nearby pixels. In some cases the later technique may be more effective as it reduces the influence of noisy gray levels that might have been produced due to an error in transmitting images w3,12x. Figs. 2 and 3 depicts the difference in applying the averaging and median operations, respectively on the original image shown in Fig. 4. Edge detection ŽFig. 5. is performed by measuring the change in the gray level intensity between pixels. The rate of change in gray level in both the vertical and horizontal direction is set equal to the partial derivative g Xx s E g Ž x, y .rE x and g Xy s E g Ž x, y .rE y, respectively. These partial derivatives can be represented by some of the popular operators such as Robert’s, Prewitt’s and Sobel’s edge detectors w3x. Robert’s edge detectors are used for diagonal edges. Prewitt’s edge detectors combine uniform smoothing in one direction and edge detection in the perpendicular direction while, Sobel’s edge detectors combine binomial smoothing with edge detection w3x. For continuous images g Ž x, y ., the gradient is a vector that is defined by a direction and a magnitude. The direction is the direction of the greatest change in g Ž x, y ., while the magnitude equals to the rate at which g Ž x, y . increases in that direction. To obtain the magnitude of the gradient,
Fig. 7. Gray level histogram.
O. Moselhi, T. Shehab-Eldeenr Automation in Construction 8 (1999) 581–588
585
Fig. 8. Thresholded image.
Fig. 10. Original image.
the response of partial derivatives in both x and y directions are computed using Eq. Ž1. w3x.
value. to black and each gray level greater than T to white w3x. This threshold value ŽT . is determined from the gray level histogram that depicts the gray level distribution in an image ŽFig. 7.. The gray level is the brightness value associated with each pixel and it ranges from 0 for black to 255 for white. One way for establishing the threshold value ŽT . is to set it equal to the gray level that corresponds to the minimum point between two peaks w3x. Referring to Fig. 7, the Threshold value was found to be equal to the gray level of 121. It should be noted that this value ŽT . was used in developing the thresholded image shown in Fig. 8. It should also be noted that other points were also selected, but gray level of 121 was found to yield better results.
2
M s g Xx Ž x , y . q g Xy Ž x , y .
2 0.5
Ž 1. g Xx Ž x, y . g Xy Ž x, y .
where M is the magnitude of the gradient, is the partial derivative in the x direction and is the partial derivative in the y direction. Inverse transformation is a process in which the color table of each image is inverted, i.e., the dark becomes light and light becomes dark ŽFig. 6. w3x. This process is sometimes needed when the object’s gray level intensity is lighter than the background. It may also ease and enhance the thresholding and segmentation processes by converting the background to white and the objects to black. Thresholding is a process that sets each gray level that is less than or equal to some value T Žthreshold
2.4. Image segmentation and features extraction Segmentation refers to the division of an image into a number of regions, each of which is reason-
Fig. 9. Segmented image.
Fig. 11. Negative image.
586
O. Moselhi, T. Shehab-Eldeenr Automation in Construction 8 (1999) 581–588
Fig. 12. Gradient image.
Fig. 14. Thresholded image.
ably uniform in some characteristics such as the gray level value w3x. The simplest method of segmenting an image is to threshold it and then to consider each connected region as an object as shown in Fig. 9 w3x. Once the image has been segmented, the different parameters of the identified objects can be measured and analyzed. These parameters include area, width, length, diameter, perimeter, roundness, centroid, minimum gray level and maximum gray level. These parameters help describing the geometry of each object and constitute the training set of the NN used in classification of these objects. One way of training NNs is by using the supervised technique w1x. In this technique, the attributes that will be fed to the input layer have to be distinguished for each class, so that the classification process can be achieved with as maximum accuracy as possible. In the present case, if the attributes are selected to be the minimum axis length, elongation Žthe ratio of
major axis length to the minor axis length. and area, then the NN can be trained for classifying different objects based on their geometry. For example, if an object has a width Žminor axis length. of few millimeters and its length Žmajor axis length. is much greater than its width, then the object is classified as a crack. On the other hand, If the elongation is close or greater than one and its minor axis length is few centimeters, then the object is classified a hole rather than a crack. Another factor that may be useful in the classification process is the area of the object, expressed in absolute value or as a ratio with respect to the thickness of the pipe. It should be noted that the system should be flexible enough to detect objects based on a pre-specified minimum size of an object. Depending on the analyzed parameters or features of each object, the most suitable set of features which represents the characteristics of each object is selected. Those features constitute the input parame-
Fig. 13. Gray level histogram.
O. Moselhi, T. Shehab-Eldeenr Automation in Construction 8 (1999) 581–588
Fig. 15. Segmented image.
ters for the NN used to classify the detected objects. It should be noted that the process utilized in segmenting and extracting the representative set of features reduced significantly the number of attributes that are fed to the NN.
3. Example application The previously described techniques were applied on an image that was obtained from a sewer pipe and digitized by a scanner. The process was implemented on a Pentium computer with 233 MHz processor, 64 MB RAM and 16 M color monitor. The analysis was performed using the Image Tool program. Observing the original image reveals that several defects are available Ži.e., cracks and joint displacements.. The
587
pipe surface showed irregularities in its color Ždifferent intensities of gray levels. ŽFig. 10.. It should be noted that the intersection of the water and pipe surfaces created what appears to be a crack. Clearly this adds to the complexity of the classification process and makes it more challenging. The process started by creating an inverted image from the original one to enhance the appearance of the defects and to make it more effective for the operations to follow ŽFig. 11.. The defects were then highlighted by automatically applying the gradient in the north direction to improve the thresholding operation and to emphasize their locations ŽFig. 12.. It should be noted that the gradients in both the northwest and the west directions were tried but the resulted image from applying the gradient in the north direction was found more effective. The gray level histogram was then constructed to assist in determining the threshold value ŽFig. 13.. The threshold value was found to be 141. It should be noted that other values were tried, but gray level value of 141 was found to yield these results. Applying this threshold value to the gradient image ŽFig. 12., the image shown in Fig. 14 was generated to remove the effect of the background and to put more emphasis on the defects themselves. Then the image was segmented to identify and highlight the available defects to prepare them for the analysis phase ŽFig. 15.. The number of defects Ži.e., cracks and joint displacements. was identified by the system to be eight. A table that shows the different attributes for the detected defects was also obtained ŽTable 1.. This table shows the
Table 1 Results
Mean Standard deviation 1 2 3 4 5 6 7 8
Object
Area
Perimeter
Major axis length
Major axis angle
Minor axis length
Minor axis angle
Elongation
01 02 03 04 05 06 07 08
174.00 176.00 220.00 571.00 150.00 200.00 38.00 42.00 64.00 80.00
164.69 134.31 221.41 448.48 139.25 220.92 49.70 50.97 78.38 100.38
70.96 54.75 87.36 186.68 64.54 100.66 22.83 23.77 35.69 46.17
20.90 54.42 15.95 20.38 y12.53 13.21 151.19 y14.62 y11.31 4.97
2.72 1.77 5.10 4.12 2.00 5.10 1.41 1.00 1.00 2.00
1.04 86.66 y78.69 y75.96 90.00 y78.69 45.00 90.00 90.00 y90.00
26.64 10.21 17.13 45.28 32.27 19.74 16.14 23.77 35.69 23.09
588
O. Moselhi, T. Shehab-Eldeenr Automation in Construction 8 (1999) 581–588
area, perimeter, major axis length, minor axis length, and elongation. The output results shown in Table 1 are all in pixels. More attributes can be obtained from the Image tool but these shown here were selected for the sake of demonstration only. These extracted attributes can be fed to the NN for classification purposes. It should be noted that the proposed system was tested on a variety of pipe materials such as concrete and clay. It was found to be applicable for both materials, provided that the inner surface of the pipe is well exposed. Other image analysis techniques are being evaluated for the case of brick pipes.
4. Summary and concluding remarks A prototype model has been presented for automating the identification process of surface defects in underground water and sewer pipes. The model combines four main modules: Ž1. image acquisition; Ž2. image processing; Ž3. image segmentation and features extraction and Ž4. pattern classification. Some of the most common image processing and analysis techniques have been discussed. They were found to provide useful means for extracting essential features, from the captured images, that can be used to automate the process of identification and classification of defects in underground pipes. The results of image analysis processes were found to be greatly dependent on the quality of the original image, the intensity of the used lighting source and the combination of image processing techniques employed. The use of image analysis techniques improves the process of transferring data to the NN used in the classification of defects and, accordingly, can make it more efficient.
5. Notation CCTV M g Ž x, y .
Closed circuit television Magnitude of gradient Pixel locations in an image
g Xx Ž x, y . g Xy Ž x, y .
Partial derivative of an image with respect to x Partial derivative of an image with respect to y
References w1x C.G. Looney, Pattern Recognition Using Neural Networks, Oxford Univ. Press, NY, 1997. w2x D.A. Chmberlain, A remotely operated building inspection cell, Automn. Constr. 1 Ž1. Ž1992. 71–81. w3x E. Gose, R. Johansonbaugh, S. Jost, Pattern Recognition and Image Analysis, Prentice-Hall, NJ, 1996. w4x G. Smith, D. Longstaff, Neural network components in texture classification system, IEEE Trans. Neural Networks Proc. 1 Ž1995. 43–48. w5x J.S. Mashford, A neural network image classification system for automatic inspection, IEEE Int. Conf. Neural Networks Proc. 2 Ž1995. 713–717. w6x M. Hastak, M.J. Skibniewski, Automation potential of pipe laying operations, Automn. Constr. 2 Ž1. Ž1993. 65–79. w7x M.J. Skibniewski, Current status and key issues for construction automation and robotics in the USA, 13th Int. Symp. Automn. Rob. in Constr., 1996, pp. 51–58. w8x M. Kaseco, S.G. Ritchie, Comparison of traditional and neural classification for pavement—crack detection, J. Transp. Eng. 120 Ž4. Ž1994. 552–569. w9x N. Guglielmi, R. Guerrieri, G. Baccarani, Highly constrained neural networks for individual quality control, IEEE Trans. Neural Networks 7 Ž1. Ž1996. 206–213. w10x O. Moselhi ŽEd.., Robots and Automated Machines in Construction, The International Association for Automation and Robotics in Construction, Watford, England, UK, 1998. w11x P. Morici, Small cameras: diagnosing sewer lateral quickly and easily, Trenchless Technol. 6 Ž10. Ž1997. 40–45. w12x R.C. Gonzalez, P. Wintz, Digital Image Processing, 2nd edn., Addison-Wesley Publishing, 1987. w13x R. Navon, Y. Rosenfeld, Computer-vision control for a floor-tiling robot, Proc. 14th Int. Symp. Automn. Rob. in Constr., 1997, pp. 52–57. w14x S.G. Rtchie, M. Kaseko, B. Bavarian, Development of an intelligent system for automated pavement evaluation, Transp. Res. Rec. 1311 Ž1991. 112–119. w15x S.G. Ritchie, Digital image concepts and application in pavement management, J. Transp. Eng. 116 Ž3. Ž1989. 287–298. w16x S. Lawrance, C.L. Giles, A.C. Tosi, A.D. Back, Face recognition: a convolution neural-network approach, IEEE Trans. Neural Networks 8 Ž1. Ž1997. 98–111. w17x S.J. Broadhurst, G. Cockerham, N. Taylor, T. Pridmore, Automatic task modeling for sewer studies, Automn. Constr. 5 Ž1. Ž1996. 61–71.
