A New Technique for Automatic Detection and

A New Technique for Automatic Detection and Parameters Estimation of Pavement Crack Ghada Moussa, Ph.D. Civil Engineering Department, Assiut University, Assiut 71515, Egypt [email protected] Khaled Hussain, Ph.D. Faculty of Computers and Information, Assiut University, Assiut 71515, Egypt [email protected] ABSTRACT Pavement condition evaluation is a significant part of a good pavement management system for effective maintenance, rehabilitation, and reconstruction decision-making. One of the key components of pavement condition evaluation is the quantification of pavement distresses data. Cracking is the main form of early pavement distresses. Cracking of pavement affects road condition, driving comfort, traffic safety, and consequently reduce pavement service life. Once initiated, cracking increases in extent and severity and accordingly accelerates the rate of pavement deterioration. Therefore, the awareness about crack type, extent, and severity is essential to evaluate pavement condition and to determine timing and cost of pavement maintenance. Digital image-based automated pavement evaluation has been gradually replacing the manual pavement evaluation due to its improved efficiency and safely operating. In this paper, we are presenting a novel reliable automated pavement assessment system based on image processing techniques and machine learning methods. The proposed system has the ability to i) identify crack, ii) extract crack parameters, and iii) report the type, extent, and severity level of that crack in an output file. Actual pavement images were used to verify the performance of the proposed system. The results clearly demonstrated that the proposed system was able to automatically and effectively identify crack type and efficiently extract crack parameters from pavement images. Such information can be used by public road agencies to define maintenance plans and assist in pavement management decision-making, in accordance with real pavement condition. Key words: Crack Detection, Flexible Pavement, Image Processing, Support Vector Machine

1. INTRODUCTION Pavements are important infrastructures, they begin their life in excellent conditions and remain in excellent conditions for a few years without need of any maintenance. Over time, however, pavements exhibits distresses due to their constant usage and consequently their condition will worsen. Pavement distresses; visible undesirable imperfections on pavement surface, that affect pavement structural capacity, appearance and hence serviceability, are usually due to traffic loads, environmental conditions, and normal wear [1]. Pavement distresses represent a significant economic concern in any country. It is estimated that pavement distresses cause damage costing $10 billion each year in the United States alone [2]. Technically, cracks are the main form of early pavement diseases [3]. Unfortunately, if these early distresses were not treated, potholes are formed causing the pavement to become

more dangerous. Therefore, a better and timely evaluation of pavement condition is likely to lower maintenance cost, support planning schemes and effectively allocating resources and increase drivers’ safety and comfort [4]. To successfully conduct an adequate evaluation process, fast and reliable quantification of pavement distresses data using automatic systems are desired, instead of relying solely on the more conventional, time-consuming, labor-intensive and subjective, manual inspection procedures [5 and 6]. With the fast developments in computer technology, digital image acquisition, and image processing, many researchers have paid a great attention to use digital image-processing to produce automatic systems to assess pavement distress [7]. However, different types of distresses, complex texture and color of the pavement surface present a challenge in developing a precise yet reliable automated system for detection and evaluation of pavement distresses [8, 9, and 10]. To overcome the limitation of image-based automated systems, a novel reliable automated pavement assessment system based on image processing techniques and machine learning methods is proposed in this paper. Subsequently, the performance of the developed system is assessed and compared with ground truth data. The rest of the paper is organized as follows. Section 2 briefly presents previous work. The various steps of the proposed system are explained in section 3. Section 4 presents the results of various experiments done to confirm the performance of the proposed system. Finally, section 5 draws some conclusions and presents some hints for future work.

2. PREVIOUS WORK In the literature, different digital image-processing techniques such as, fuzzy set theory [11], Markov methods [12 &13], artificial life [14], neural networks [15, 16, and 17], and many other techniques have been used for crack detection and classification. Cheng, et al. (1999), proposed a novel pavement cracking detection algorithm based on fuzzy set theory [11]. The main idea of their method is based on the assumption that the cracking pixels are always darker than their surroundings. Delagnes and Barba (1995), proposed a Markov random field model for crack detection and extraction [12]. Another multiscale approach for crack detection, using Markov random field mode, is presented by Chambone et al., 2009 [13]. An artificial living system for crack detection is proposed by Zhang and Wang (2004) [14]. In their study, a bottom-up approach was used for searching artificial structures within the pavement image. Bright points in images were excluded and the contrast between crack and non-crack pixels was enhanced. Qualitative experimental results were given in their analysis.

The neural network technique for pavement distress detection and classification was extensively adapted. Chou et al., (1994), used moment invariant and neural network for pavement crack classification [15]. After calculating moment invariants from different types of cracks thus obtaining crack features, neural network was used to classify these features. They trained their neural network and reported a one-hundred percent classification accuracy results. Cheng et al., (2001), used the mean and standard deviation as parameters and trained a neural network to select a threshold for pavement image segmentation [16]. Lee and Lee (2004), present an integrated neural networkbased crack imaging system for crack type classification from pavement images [17]. They presented three neural; imagebased, histogram-based, and proximity-based networks. Their classification was based on the sub-images (distress region) rather than crack pixels in digital pavement images. Moreover, based on a spatial autocorrelation function, Lee and Oshima (1994) proposed an automated imaging procedure for crack identification and density measurement [18]. By calculating the autocorrelation function of a pavement image, the crack pattern in a noisy background could be identified. They concluded that their approach can identify crack type and density with a reasonable accuracy. Based on local binary pattern operator, Hu and Zhao (2010) proposed a pavement crack detection approach [10]. They concluded that thin cracks (less than 1 mm) and cracks in strong texture were correctly and efficiently detected. Based on image processing and pattern recognition techniques, Oliveira and Correia (2009) proposed an automatic system for crack detection and classification in flexible pavement images [19]. Their system was evaluated using real pavement images with their manually ground truth data. They reported promising results in both crack detection and classification. Based on both image processing and supervised classification techniques, Younes et al. (2009) introduced an algorithm to classify pavement deterioration images [20]. An evaluation was conducted using 80 real pavement images, achieving 80% success in pavement deteriorations classification.

3. SYSTEM DESCRIPTIONS A novel reliable approach for automatic crack detection, classification, and parameter estimation from flexible pavement images acquired during road surveys, based on image processing and machine learning techniques is presented. The proposed approach consists of four main stages: (1) segmentation, (2) feature extraction, (3) classification, and (4) parameters quantification. 3.1. Segmentation Our goal is to accurately segment a given image into “crack” and “background” regions. We used the Graph Cut segmentation technique [21 &22] because it gives the best balance of boundary and region properties. The solution to our ,…, ,…, , segmentation problem is a binary vector where Ar equals one for crack and equals zero for background. To improve efficiency, we deal with these regions instead of image pixels. We used watershed algorithm [23] which divides the image into small regions (R). We create a graph G = (V, E) with nodes corresponding to regions r ϵ R of the image. There are two additional nodes: a “crack” terminal (a source S) and a “background” terminal (a sink T). So, V becomes  

 

,

(1)

Each region r has two t-links (terminal links) {r, S} and {r, T} connecting it to each terminal. n-links (neighborhood links) connect each pair of neighboring regions {p, q} in N. Therefore,     , , , (2) We divide the regions into three types: crack, background, and unknown regions. One of the main characteristic of cracks is that it has a low intensity. Thus, the crack regions are identified by its mean intensity is less than a threshold T1. In general, the intensity of the background is greater than the intensity of the cracks. Therefore, the background regions are identified by its mean intensity is greater than a threshold T2. The K-means method is used to divide the crack and background regions into clusters. The mean intensity of the crack and background clusters are denoted as and respectively. For each region r, we compute the minimum distance from its mean intensity M(r) to crack clusters as min . Then we compute the minimum distance from the region r’s mean intensity M(r) to the background clusters as min . The following table gives weights of edges in E. Table1: Weights of edges in E Edge

Weight (We) ∞ 0

,

For r ϵ Crack Regions r ϵ Background Regions r ϵ Unkown Regions

∞ 0

,

r ϵ Background Regions r ϵ Crack Regions r ϵ Unkown Regions

{p, q}

|

1

|

{p, q} ϵ N

After defining the graph G, we need to calculate the minimum cost cut on the graph G. The cost of the cut is defined as ∑ . The minimum cost of the cut CH can be calculated in polynomial time using the maxflow algorithm in [24]. After calculating the cut H, the segmentation binary vector ,…, ,…, is calculated as follows: 1 (crack) if {r, T} H 0 (background) if {r, S} H

(3)

3.2. Feature Extraction One of the most difficult problems in the design of any computer vision system is the selection of a set of appropriate numerical attributes or features to be extracted from the region of interest for classification purposes. The success of any practical system depends critically upon this decision. Although there is little in the way of a general theory to guide in the selection of features for an arbitrary problem [25], it is possible to state some desirable attributes of features for identification of cracks; the features should be invariant with translation, scale, and light conditions. Let L be a two dimension binary array of size m x n corresponding to the segmentation binary vector A. Let  

∑

, ∑

 

, ,

   

1,2, …

, and

  ,   1,2, … . Seven features are extracted as follows:

(4) (5)

1. 2. 3. 4. 5. 6. 7.

PksNC is the number of local peaks in SC that are greater than T3 and that are separated by minimum distance of T4. PksNR is the number of local peaks in SR that are greater than T3 and that are separated by minimum distance of T4. PksMeanDistC is the mean of the distances between peaks in SC and it equals zero if PksNC equals one. PksMeanDistR is the mean of the distances between peaks in SR and it equals zero if PksNR equals one. PksStdDistC is the standard deviation of distances between peaks in SC and it equals zero if PksNC equals one. PksStdDistR is the standard deviation of distances between peaks in SR and it equals zero if PksNR equals one. PksLocMeanR is the average of the normalized locations of the peaks in SR.

3.3. Classification In this work, the Support Vector Machine (SVM) was used for the classification because it can produce accurate and robust classification results. The SVM is the state-of-the-art among classification algorithms [26]. The SVM is shown to be suitable for solving problems with a small sample set, nonlinearity, high dimension, over-fitting and local minima, eventually achieving a better generalization performance [27]. The proposed method is constituted essentially of two main phases; training phase and classification phase. As the name indicates, during the training phase is learned to recognize a set of different cracks. In its classification phase, it outputs the crack type (Transverse cracking, Longitudinal cracking, Block Cracking, or Alligator Cracking). The training of the SVM is carried out in the following sequential order: 1. Scale each feature to the range [0-1]. 2. Construct the training set of (Fi; yi); i=1,2,…, L where Fi is the feature vector after scaling and yi=(1,0,0,0) for Transverse cracking, (0,1,0,0) for Longitudinal cracking, (0, 0, 1, 0) for Block Cracking, or (0,0,0,1) for Alligator Cracking. ) is used. 3. The RBF kernel ( , 4. Use cross-validation to find the best parameter γ. 5. Use the best parameter γ to train the whole training set. 3.4. Parameters Quantification The automated assessment of the crack extent and severity based on crack parameters (lengths and widths) are a useful input to pavement condition evaluation [28]. Our proposed system automatically processes crack images to compute the crack length and the average crack width for extent and severity level identification (low, moderate, or high) as defined in the Distress Identification Manual [1]. The crack length in meter is calculated using the following equations: For Longitudinal crack, 1 0

   

,

WR W

∑

(6)

∑

(7)

For Transverse crack, 1 0

   

,

HR H

As explained in the manual, the average crack width is the mean crack opening width of the crack [1]. To calculate the average crack width, we first need to calibrate the camera. Then we fit line(s) along the crack points and calculate the average

crack width along the line(s). Our method is based on random sample consensus (RANSAC) method [29] and thinning process as shown in the following steps: 1. Let WR and HR be the actual width and height, respectively, in meter of the pavement captured by the camera. Let W and H are the width and height, respectively, in pixels of the pavement image. Since, the height of the camera is fixed at CH meter, the camera is pointed downward, and the camera parameters are fixed. We can assume that the width PW and height PH of a pixel in meter are equal to WR/W and HR/H, respectively. 2. Apply thinning process [30] for reducing the crack regions in the image A to skeletal lines that preserves the extent and connectivity of the original crack region while deleting most of the original crack pixels. The details of the thinning process is explained as follows: a. The image is divided into two different subfields in a checkerboard pattern. b. In the first sub-iteration, if conditions G1, G2, and G3 are all satisfied, then pixel p is removed. c. In the second sub-iteration, if conditions G1, G2, and G3' are all satisfied, then pixel p is removed. Let x1, x2… x8 are the values of the eight neighbors of p, starting with the east neighbor and numbered in CCW order. Condition G1 is satisfied if XH(p) =1 ∑ Where, and 1 0

 

0 

 

Condition G2 is satisfied if 2 ∑ Where,  

1  min     

 

1 , and

(8) 3

 

Condition G3 is satisfied if         0 Condition G3’ is satisfied if       0   3. For fitting the line y = mx+c, we need to estimated the parameters m and c that minimize E where ∑    a. Randomly select two points (xr, yr) and (xs, ys). b. Estimate the parameters m and c of the line that pass through the two points (xr, yr) and (xs, ys). | | c. For each crack pixel (xi, yi) caculate di where √ d. Count the number of outliers and inliers: the pixel (xi, yi) is consider outlier if di is greater than a threshold τ; the pixel is consider inliers if di is less than a threshold τ. e. Repeat steps a to d for N times and find the the parameters m’ and c’ of the line that have the maximum inliers. 4. The following steps show how we caculate the crack width, given the line y = m’x+c’. a. Set LineWidth =1 b. Draw the line y = m’x+c’ with width LineWidth in an empty image I1 of size W x H |A I | c. Set D ∑∑D d. Set e e. Increase the LineWidth (LW) by one f. Repeat steps a to e M times and find the the line width LineWidth’ (LW’) that has the minimam e1. g. Convert the line width from pixels to meter using the following equation:

′ cos

tan

1 m

WR W

′ sin

tan

1 m

HR H

4. TESTING AND RESULTS To test the reliability of the proposed system, a set of 87 pavement images (with 61 crack images) were collected from various road sections and used for the analysis. The implementation of our proposed method was carried out and the experimental results were recorded. Our system was trained to recognize two types of cracks: Longitudinal and Transverse. In Sahibsingh et al., it has been stated that “It is generally difficult to make a comparison of different recognition systems, even for the same problem, since different test sets are used for evaluating performance...'' [25]. Therefore, no comparisons are offered for our proposed system with other systems. Examples of pavement images for Longitudinal and Transverse cracking and their testing results are shown in Figure 1 and 2 respectively. For both Figures a) represents original pavement image, b) represents crack image after segmentation, c) represents line fitting for avg. width calculations. From Figure 1 and 2, It can be observed that cracks are well detected using our system.

a) Original image

b) Segmentation result

c) Line fit result Fig. 1: Pavement surface image with a longitudinal crack

crack type of the test sample in less than 1 minute on a 64-bit Intel Core 2 Duo, 2.13 GHz, Ram 4GB, personal computer. The results of this test are summarized in Table 2. The system was able to detect the presence or absence of cracks in pavement images. Only 2 images out of 49 images with cracks were missdetected. All the 21 distress-free pavement images were correctly predicted as no-crack. Therefore, the overall system prediction accuracy was found to be 97.1%. Based on these results, we can conclude that the proposed approach is able to detect pavement cracks effectively and accurately. Table 2: Summary of system performance in crack type prediction Actual crack type Long. Transverse No distress Total

Total System predicted crack type Accuracy no. Long. Transverse No Crack (%) 22 21 1 95.5 27 26 1 96.3 21 21 100 70 21 26 23 97.1

4.2. Crack Parameters Quantification As the awareness about crack type is important, identifying the crack extent and severity level is essential to evaluate pavement condition and to determine timing and cost of pavement maintenance. The crack extent can be defined as the total cracked length for transverse and longitudinal cracking, as explained in the manual [1]. Moreover, the crack severity level is classified into three severity levels; low, moderate, and high based on the average crack width, as defined in the manual [1]. To evaluate the system performance in estimating average crack width and crack extent (length), field inspection data (actual measurements) was used. Paired t-tests were applied to compare the differences between the system estimated values and field inspection data, as shown in table 3 and 4. The statistical results indicate that there were no significant differences in the average crack width and crack extent between the system and field inspection data, at 95% confidence level (i.e. α = 0.05). Note that the two miss-detected images were eliminated from the data set. Table 3: Summary of statistic t-test for avg. crack width (mm) N Mean (mm) S.D. t tα Actual Avg. Width 59 8.90 6.12 0.7 2.05 Predicted Avg. Width 59 8.83 6.16

a) Original image

b) Segmentation result

Table 4: Summary of statistic t-test for crack extent (m) N Mean (m) S.D. t tα Actual Crack Extent 59 1.190 0.514 0.25 2.05 Predicted Crack Extent 59 1.188 0.508 For the severity level prediction, table 5 presents severity-level comparison between in-the-field estimated values (actual severity level) and the system predictions ones. The results show that the system is able to predict the cracks severity levels in pavement images at in an average of 95.1% accuracy.

c) Line fit result Fig. 2: Pavement surface image with a transverse crack 4.1. Crack Type Identification In this section, results obtained when applying our system for classification of cracks depicted from real-life road images are discussed. The support vector machine was trained with 20% of the data set (17 images). Then we used this model to verify the other 80% (70 images). Our proposed system recognized the

Table 5: Summary of system performance in severity prediction Actual Severity Level Low Moderate High Total

No. of No. of images identified images by the system Low Moderate High 33 31 2 19 1 18 7 7 59 32 20 8

Accuracy (%) 93.9 94.7 100 95.1

5. CONCLUSIONS As the evaluation of pavement condition is the base of the pavement management, identification of pavement distresses and their parameters is an important step in this evaluation. Recently, the trend toward automatic distresses assessment has been widely spread, which is based largely on image processing techniques. However, the automatic recognition of pavement distresses from digital images is a difficult task, since pavement images can bring more information than those needed to detect a distress which can mislead the recognition process [20]. Based on image processing techniques and machine learning methods, we proposed a novel system for pavement distress assessment. Our proposed system was not only able to identify cracks but also classify cracks and quantify their parameters including extent and severity from digital pavement images, and present them in an output file. The proposed system attained promising performance results when tested via real pavement images with their ground truth data. An area of future work that would be considered is increasing distress types for analysis. Furthermore, adapting this approach to process video streams would be studied.

6. REFERENCES [1] J.S Miller and W.Y. Bellinger, Distress Identification Manual for Long-Term Performance Program (Fourth Revised Edition), Office of Infrastructure Research and Development, Federal Highway Administration, Report No. FHWA-RD-03-031, 2003. [2] R.J. Dilger, American Transportation Policy. West Virginia University, Department of Political Science. Westport, CT: Praeger Publishers, 2003. [3] A. Ouyang, C. Luo and C. Zhou, “Surface Distresses Detection of Pavement Based on Digital Image Processing, Advances in Information and Communication Technology, Vol. 347, 2011, pp. 368-375. [4] J. Kim, Development of a Low-Cost Video Imaging System for Pavement Evaluation, Ph.D. Thesis, Oregon State University, Oregon, United States, 1998. [5] H.D. Cheng and M. Miyojim, “Automatic Pavement Distress Detection System”, Journal of Information Sciences, ELSEVIER, Vol. 108, No.1, 1998, pp. 219-240. [6] K.A. Abaza, S.A. Ashur, and I.A. Al-Khatib, “Integrated Pavement Management System with a Markovian Prediction Model,” Journal of Transp. Eng., Vol.130, No.1, 2004, pp. 24-33. [7] K. McGhee, Automated Pavement Distress Collection Techniques, NCHRP Synthesis 334, National Cooperative Highway Research Program (NCHRP), USA., 2004. [8] L. Li, P. Chan, and R.L. Lytton, “Detection of Thin Cracks on Noisy Pavement Images”, Transp. Res. Record, No. 1311, pp.131-135, 1991. [9] K. Wang, “Design and Implementations of Automated Systems for Pavement Surface Distress Survey”, Journal of Infrastructure Systems, ASCE, Vol. 6, 2000. [10] Y. Hu, C.X. Zhao, “A Local Binary Pattern Based Methods for Pavement Crack Detection”, Journal of Pattern Recognition Research, Vol. 5, No. 1, 2010, pp. 140-147. [11] H.D. Cheng, J.R. Chen, C. Glazier, C. and Y. G. Hu, “Novel Approach to Pavement Cracking Detection Based on Fuzzy Set Theory”,Journal of Computing in Civil Eng., Vol.13, No.4, 1999, pp. 270-280.

[12] P. Delagnes and D. Barba, “A markov random field for rectilinear structure extraction in pavement distress image analysis,” in Proceedings of IEEE International Conference on Image Processing (ICIP ’95), Vol. 1, pp. 446–449, Washington, DC, USA, October 1995. [13] S. Chambon, P. Subirats, and J. Dumoulin, “Introduction of a Wavelet Transform Based on 2D Matched Filter in a Markov Random Field for Fine Structure Extraction: Application on Road Crack Detection”, in IS&T/SPIE Electronic Imaging, Image Processing: Machine Vision Applications II, San Jose, USA, 2009. [14] H.G. Zhang and Q. Wang, “Use of Artificial Living System for Pavement Distress Survey”, Industrial Electronics Society, 30th Annual Conference of IEEE, Pusan, Korea, Nov. 2004, pp. 2486-2490. [15] J. Chou, W.A. O’Neill, and H.D. Cheng, “Pavement Distress Classification Using Neural Networks”, Systems, Man, Cybernetics1, IEEE, 1994, pp.397–40. [16] H.D. Cheng, J.L. Wang, Y.G. Hu, C. Glazier, H.J. Shi, and X.W. Chen, “Novel Approach to Pavement Cracking Detection Based on Neural Network”, Transp. Res. Record, No. 1764, pp. 119-127, 2001. [17] B.J. Lee, and H.D. Lee, “Position-invariant neural network for digital pavement crack analysis”, Computer-Aided Civil and Infrastructure Engineering, vol. 19, no. 2, 2004, pp. 105–118. [18] H. Lee, and H. Oshima, “New crack-imaging procedure using spatial autocorrelation function” ASCE Journal of Transp. Eng., Vol. 120, No. 2, 1994, pp. 206-228. [19] H. Oliveira, and P.L. Correia, Supervised Crack Detection and Classification in Images of Road Pavement Flexible Surfaces, Chapter in Recent Advances in Signal Processing, Austria, 2009. [20] G. Younes, C. Hadda, and Z. Djellou, “Supervised Learning and Automatic Recognition of Asphalt Pavement Deteriorations”, MASAUM Journal of Basic and Applied Sciences, Vol.1, No. 2, September 2009, pp. 254259. [21] Y. Boykov, and M. Jolly, “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images”, in Proceedings of Eighth IEEE International Conference on Computer Vision (ICCV), Vol. 1, 2001, pp. 105-112. [22] Y. Li, J. Sun, C.K. Tang, and H. Y. Shum, “Lazy Snapping”, SIGGRAPH04, April 2004. [23] L. Vincent, and P. Soille, “Watersheds in digital spaces: an efficient algorithm based on immersion simulations”. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1991, pp.583-598. [24] Y. Boykov, and V. Kolmogorov, “An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision”, Energy Minimization Methods in Computer Vision & Pattern Recognition, 2001. [25] A.D. Sahibsingh, J.B. Kenneth, & B.M .Robert, “Aircraft Identification by Moment Invariants”, IEEE Transactions on Computers, Vol. 26 No. 1, 1977, pp. 39-45. [26] C.H. Lampert, "Kernel Methods in Computer Vision", Foundations and Trends in Computer Graphics &Vision, Vol.4, No.3, 2009, pp.193-285. [27] V. Vapnik, The Nature of Statistical Learning Theory, New York: Springer-Verlag, 1995. [28] A. Amarasiri, M. Gunaratne, and S. Sarkar, “Modeling of Crack Depths in Digital Images of Concrete Pavements using Optical Reflection Properties”, Journal of Transp. Eng., Vol. 136, No. 6, June 2010, pp. 489-499.

[29] M.A. Fischler, and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography”, Communications of the ACM, Vol. 24, No.6, 1981, pp. 381–395. [30] L. Lam, S.W. Lee, and C.Y. Suen, "Thinning Methodologies-A Comprehensive Survey", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No. 9, 1992, pp. 869 - 885.