algorithm development for real-time infrastructure

ALGORITHM DEVELOPMENT FOR REAL-TIME INFRASTRUCTURE DAMAGE DETECTION AND ANALYSIS by Tonmoy Roy

A Thesis Submitted to the Faculty of Purdue University In Partial Fulfillment of the Requirements for the degree of

Master of Science in Mechanical Engineering

Department of Mechanical Engineering West Lafayette, Indiana May 2017

ii

THE PURDUE UNIVERSITY GRADUATE SCHOOL STATEMENT OF THESIS APPROVAL

Dr. Kartik B. Ariyur, Chair Department of Mechanical Engineering Dr. Bin Yao Department of Mechanical Engineering Dr. Ganesh Subbarayan-Shastri Department of Mechanical Engineering

Approved by: Dr. Anil Bajaj Head of the Departmental Graduate Program

iii

Shahreen, I want to thank you for your continued presence, patience, and emotional support. This one is for you!

iv

ACKNOWLEDGMENTS

I want to take the opportunity to thank Dr. Kartik Ariyur for trusting me with the immense responsibility of completing this work. Throughout some of my graduate classes and during my research work, he has been an excellent mentor and professor. He truly understands that hard work and dedication can get a person anywhere. Despite having background in Mechanical Engineering with focus in Systems, Measurements & Controls, I completed a project that required extensive knowledge in topics such as Image Processing, Statistics, and Structure from Motion. This couldn’t have been possible without the guidance of Dr. Kartik Ariyur. He motivated me to take on the challenge of attempting damage identification even though my initial goal was to complete my work with successful damage detection. I am glad that he did, and because of this action, not only have I made my work more robust, but also have gained more knowledge. I am truly thankful for his support. I also want to thank Dr. Rezwan Rahman for the time he spent helping me understand Image Processing, Statistics and Machine Learning concepts. He has been a great resource to me throughout my research activities. I am glad that he pointed me towards the appropriate books and research materials when I had difficulties understanding complex topics.

v

TABLE OF CONTENTS

LIST OF TABLES ............................................................................................................ vii LIST OF FIGURES ......................................................................................................... viii ABSTRACT ........................................................................................................................ x CHAPTER 1

|

INTRODUCTION ................................................................................ 1

CHAPTER 2

|

OBJECTIVES & ASSUMPTIONS ...................................................... 3

2.1 Objectives .................................................................................................................. 3 2.2 Assumptions .............................................................................................................. 3 CHAPTER 3

|

PRIOR LITERATURE REVIEW ........................................................ 5

3.1 Image Quality ............................................................................................................ 5 3.2 Image Texture ........................................................................................................... 5 3.3 Image Feature Extraction & Selection ...................................................................... 6 3.4 Damage Detection ..................................................................................................... 7 3.5 Structure Estimation .................................................................................................. 8 3.6 Geometry Feature Estimation.................................................................................... 9 CHAPTER 4

|

EXPERIMENTAL SETUP & METHOD .......................................... 11

4.1 Data Acquisition & Description .............................................................................. 11 4.2 Development & Computation Environments .......................................................... 12 4.3 Data Pre-Processing ................................................................................................ 13 4.3.1

Greyscale Transform ................................................................................... 13

4.3.2

Histogram Equalization ............................................................................... 13

4.3.3

Feature Calculation ..................................................................................... 15

4.4 Dimensionality Reduction ....................................................................................... 19 4.5 Data Comparison ..................................................................................................... 23 4.5.1

Euclidean Distance ...................................................................................... 24

4.5.2

One-Class Support Vector Machine ............................................................ 25

4.6 Damage Detection & Quantification ....................................................................... 26 4.7 Geometry Generation .............................................................................................. 29 4.8 Feature Extraction ................................................................................................... 32

vi CHAPTER 5

|

RESULTS & DISCUSSION .............................................................. 36

5.1 Results & Discussion for Alpha & Beta .................................................................. 36 5.2 Results and Discussion for Gamma ........................................................................ 41 CHAPTER 6

|

CONCLUSION ................................................................................... 49

6.1 Practical Issues, Algorithm Limitations & Prospects.............................................. 49 6.2 Conclusion ............................................................................................................... 50 APPENDIX A. CODE ...................................................................................................... 51 APPENDIX B. ADDITIONAL DATA & TABLES ........................................................ 68 APPENDIX C. ILLUSTRATIONS OF ADDITIONAL TEST SETS PROCESSED BY ALGORITHM GAMMA ................................................................................................ 71 REFERENCES ................................................................................................................. 76

vii

LIST OF TABLES

Table 1: Features Calculated from Images ....................................................................... 15 Table 2: GLCM Texture Features at Varying Inter-Pixel Distances ................................ 18 Table 3: GLCM Texture Features at Varying Offset Angles ........................................... 19 Table 4: Contributions of Eigenvalues ............................................................................. 21 Table 5: Parameters & Computation Times for Comparison Methods ............................ 26 Table 6: Description of Validation Datasets for Geometry Generation ........................... 29 Table 7: Defined Metrics from Curvature Statistical Data, and Selected Thresholds for Classification.................................................................................................................. 34 Table 8: Performance Evaluation of Algorithm Beta on Test Images from Set C ........... 38 Table 9: Calculation Results Generated by Gamma for Crack Damage and Depression Damage Test Sets........................................................................................................... 43 Table 10: Calculation Results Generated by Gamma for Other Damage Test Sets ......... 48 Table 11: Statistical Properties of Positive Curvature Values in Seven Validation Sets of Surface Crack Data ........................................................................................................ 68 Table 12: Statistical Properties of Negative Curvature Values in Seven Validation Sets of Surface Crack Data ........................................................................................................ 68 Table 13: Statistical Properties of Flat Surface Curvature Values in Seven Validation Sets of Surface Crack Data .................................................................................................... 69 Table 14: Statistical Properties of Positive Curvature Values in Seven Validation Sets of Surface Depression Data ................................................................................................ 69 Table 15: Statistical Properties of Negative Curvature Values in Seven Validation Sets of Surface Depression Data ................................................................................................ 69 Table 16: Statistical Properties of Flat Surface Curvature Values in Seven Validation Sets of Surface Depression Data ........................................................................................... 70 Table 17: PF, NF and FA Ratios for Surface Crack Validation Datasets ......................... 70 Table 18: PF, NF and FA Ratios for Surface Depression Validation Datasets ................ 70

viii

LIST OF FIGURES

Figure 1: Dry and Wet Reference Material Surface Data at High and Low Illumination 11 Figure 2: Formation of the GLCM.................................................................................... 17 Figure 3: Arbitrary Inter-Pixel Distance and Offset Angle Combinations ....................... 17 Figure 4: Dataset Matrix X Containing Texture Values for Set A ................................... 17 Figure 5: Scree Plot........................................................................................................... 22 Figure 6: Projection of First & Second Principal Components ........................................ 22 Figure 7: Flowchart for Algorithm Alpha (or alpha.py) ................................................... 23 Figure 8: Flowchart for Algorithm Beta (or beta.py) ....................................................... 28 Figure 9: Flowchart for Performing Geometry Generation .............................................. 31 Figure 10: Flowchart for Algorithm Gamma (or gamma.py) ........................................... 31 Figure 11: Curvature Spectrum Data from Surface Crack Damage and Surface Depression Damage Validation Datasets ....................................................................... 33 Figure 12: Samples of Test Image (Left) and Result Image (Right) with Percentage of Possible Damage (PPD) and Time Taken for Algorithm Execution ............................. 37 Figure 13: Results for Blur Avoidance (Top) and Low Illumination Avoidance (Bottom) Criteria ........................................................................................................................... 39 Figure 14: Results for Unknown Material Rejection (Top) and Unknown Object Rejection (Bottom) Criteria ........................................................................................... 39 Figure 15: Results for Markings Rejection Criteria .......................................................... 40 Figure 16: Results for Material State Insensitivity ........................................................... 41 Figure 17: Point Cloud Model and Mesh Model for Surface Crack Damage Data from Validation Set................................................................................................................. 42 Figure 18: Point Cloud Model and Mesh Model for Surface Depression Damage Data from Validation Set ........................................................................................................ 42 Figure 19: Image and Mesh Model Representation for Crack Damage Dataset 1 ........... 44 Figure 20: Image and Mesh Model Representation for Crack Damage Dataset 2 ........... 44 Figure 21: Image and Mesh Model Representation for Crack Damage Dataset 3 ........... 45 Figure 22: Image and Mesh Model Representation for Crack Damage Dataset 4 ........... 45 Figure 23: Image and Mesh Model Representation for Depression Damage Dataset 1 ... 46

ix Figure 24: Image and Mesh Model Representation for Depression Damage Dataset 2 ... 46 Figure 25: Image and Mesh Model Representation for Depression Damage Dataset 3 ... 47 Figure 26: Image and Mesh Model Representation for Depression Damage Dataset 4 ... 47 Figure 27: Damage Test Set 1 ........................................................................................... 71 Figure 28: Damage Test Set 2 ........................................................................................... 71 Figure 29: Damage Test Set 3 ........................................................................................... 72 Figure 30: Damage Test Set 4 ........................................................................................... 72 Figure 31: Damage Test Set 5 ........................................................................................... 73 Figure 32: Damage Test Set 6 ........................................................................................... 73 Figure 33: Damage Test Set 7 ........................................................................................... 74 Figure 34: Damage Test Set 8 ........................................................................................... 74 Figure 35: Damage Test Set 9 ........................................................................................... 75

x

ABSTRACT

Author: Roy, Tonmoy. MSME Institution: Purdue University Degree Received: May 2017 Title: Algorithm Development for Real-Time Infrastructure Damage Detection and Analysis Major Professor: Kartik B. Ariyur

Extensive studies have been done in topics related to structural health monitoring using image processing and machine learning techniques. Trends in recent studies show increasing interests in the use of vision-enabled Unmanned Aerial Vehicle (or UAV) systems in structural health monitoring applications. Although there have been a lot of studies focused in theoretical aspects of image processing and machine learning, not much attention is given to the practical aspects. This thesis presents algorithm-based solutions to real-time infrastructure damage detection and damage identification with the intention of use with UAV systems. In this thesis, detection and identification results are quantified for intuition. In a practical scenario, a vision-enabled UAV system might be used to acquire image data of a damaged surface of interest with the goal of damage detection. However, unwanted information might be captured in the image data during such an exercise. The algorithm presented in this thesis is intended to remove the irrelevant regions from image data, detect damage, and quantify the results. The algorithm successfully detects damage in five out of seven tested categories. Generalization to different types of material surfaces is one of the objectives behind development of the algorithm. Identification of the type of damage is also attempted in another part of the algorithm. In this case, the algorithm shows reliable results in most test datasets and indicates the most prominent type of damage present.

1

CHAPTER 1

| INTRODUCTION

According to reports by the American Society of Civil Engineers (ASCE) (2013 & 2017), the U.S. infrastructure achieved a grade of D+, where D means “poor to fair condition and mostly below standard, with many elements approaching the end of service life…condition and capacity are of serious concern with strong risk of failure” (ASCE, 2017). Per both the reports, the U.S. roads achieved a grade of D. ASCE (2013) concluded that 32% of the U.S. roads are in poor condition costing motorists $67 billion a year in additional repair and operation costs; the report further stated that the accumulated cost of poor road conditions significantly exceeds the cost to maintain good conditions on the same roads. ASCE (2017) reported that “in 2016 alone, U.S. roads carried people and goods over three trillion miles” and that “Americans are driving more and vehicle miles travelled is the second highestever level, second only to 2007” (p. 1). ASCE (2017) further stated that “one out of every five miles of highway pavement is in poor condition and our roads have significant and increasing backlog of rehabilitation needs” (p. 1). Urban roads are reported to be in far worse shape than rural ones due to greater traffic. ASCE (2017) also stressed that “driving on roads in need of repair cost U.S. motorists $112 billion in extra vehicle repairs and operating costs in 2014” (p. 2). While it is important to maintain good conditions within any infrastructural entity, it is also crucial to monitor the infrastructure frequently to prevent failures through early detection and prevention. However, is it economically feasible to monitor on a large scale? Recent advances in energy storage, sensing capabilities, aeronautics with autonomous navigation make UAV’s affordable choices in monitoring applications (Liu et al., 2014). Ham et al. (2016) presented a collection of research endeavors led by the Civil Engineering, Computer Vision, and Robotics communities, and compared them in terms of their prospects in leading towards civil infrastructure monitoring applications using visionenabled UAV’s. Image-based analysis is a form of non-destructive assessment technique in structural health monitoring. Automation of certain aspects of the monitoring process can facilitate early detection. This thesis proposes a novel approach for damage detection and

2 quantification. Moreover, the thesis explores possibilities of identification of damage type for two types of damages. Chapter 2 of the thesis states the objectives and assumptions for this work. Chapter 3 provides a thorough study of prior literature on topics related to this work. Chapter 4 explains the algorithm development approach and Chapter 5 discusses the results derived. Chapter 6 discusses practical issues, limitations of the algorithm, and concludes with a summary for future work.

3

CHAPTER 2

2.1

| OBJECTIVES & ASSUMPTIONS

Objectives

Critical to UAV systems is the development of small, lightweight, and solid-state sensors. Advances in Micro-Electromechanical Systems (or MEMS) have enabled sensors to be smaller, more accurate, cheaper, and therefore, more appropriate for use in UAV’s (Beard & McLain, 2012). UAV systems are now capable of performing highly autonomous flights due to developments in Global Positioning Systems (or GPS), batteries, and microcontrollers. Vision-enabled UAV systems show promising results in civil infrastructure monitoring applications (Ham et al., 2016), and can fill the niche in spatial and temporal resolution necessary in monitoring applications (Lucieer et al., 2011). The objective of this thesis is to provide practical algorithm-based solutions to the problem of infrastructure monitoring and damage detection using vision-enabled UAV systems. One of the objectives of this thesis is to introduce a novel method for surface damage detection. Why is the method novel? The intention behind the development of this method is generalization to different types of materials, and hence to different types of surfaces. The algorithm is developed such that it considers both the intensity and texture features of any type of surface from train image data to first “train” and then “recognize” sections of that same surface in test image data. One of the goals of the algorithm is to remove information in sections of the test image data that it doesn’t “recognize” as the surface of interest. Throughout the thesis, the phrase reference material surface is used to refer to the surface of interest. The end goal is detection of damage in the reference material surface and calculation of numerical results for intuition. One of the other objectives of this thesis is to propose a method for further analysis after damage detection. Here, identification of the type of damage is attempted using geometry estimation techniques.

2.2

Assumptions

In the context of this thesis, let’s assume a possible scenario. The UAV system intended for use is a multi-rotor one, and is capable of localization (of self, goals, and obstacles),

4 path planning, navigation, altitude estimation, pose estimation (and correction), disturbance adaptation, and wireless communication. It has onboard data storage capabilities. It has two gimbal-mounted and gyro-stabilized vision sensors. The first vision sensor can acquire high resolution image data. The second vision sensor is used for pathfollowing activities (similar to the technique presented by Nguyen et al. (2014)). The UAV system has an onboard computer capable of executing all the necessary responsibilities discussed above. The UAV system receives directions (wirelessly) from a base station to move to a GPS location near an infrastructural entity, locate a start marker using the second vision sensor, align itself to the start marker, maintain a certain amount of distance from the surface of the infrastructure body, track a thin marked path on the infrastructure body, and follow the path until the UAV reaches an end marker. The path is laid such that it goes near critical regions of the infrastructure body that must be monitored frequently. While following the path, the UAV system uses the first vision sensor to capture image data and uses its onboard computer to process the data in real-time. If damage is detected, the UAV hovers in a circle to capture multiple overlapping images of the region, stores the images (for later processing), and resumes to follow the path until the end marker is reached. The UAV has wireless charging capabilities (similar to work presented by Jung et al. (2012)). When it runs low on power, the UAV stops over at any UAV landing zone along the path to charge its battery wirelessly. After the UAV completes the monitoring exercise, it returns to the base station, where the geo-tagged image data is downloaded into the base station computer for

further

analysis.

Using

structure

from

motion

techniques,

the overlapping images are used to construct point cloud models, and eventually mesh models of the damaged regions; finally, geometric and luminance properties are derived for further study. Based on severity of damage, the base station computer indicates whether repairs are required. If repairs are required, location meta-data from the geotagged images is used to send resources for repairs. While the UAV system is executing this hypothetical exercise, it is assumed that there are no severe weather phenomena or unavoidable disturbances. It is also assumed that there are no sources of image data noise.

5

CHAPTER 3

3.1

| PRIOR LITERATURE REVIEW

Image Quality

The ability to accurately estimate noise levels in image data is an essential step towards achieving robustness in computer vision algorithms. Liu et al. (2006) proposed a method of inferring noise as a function of image intensity (i.e. the noise level function, NLF) from a single (test) image. Liu et al. also derived accurate bounds on noise levels and showed the usefulness of using noise estimates from noise level functions (to set computer vision algorithm parameters or select training sets). Wang et al. (2004) developed a Structural SIMilarity (SSIM) index for use as an image quality measure, and stated that the measure can be calculated from the mean (luminance) and standard deviation (contrast) values of a reference image and a test image. The method described in this paper is a full-reference type, meaning that a complete reference image is assumed to be known.

3.2

Image Texture

In an inspiring paper, Haralick et al. (1973) showed fourteen different statistical texturefeature measures derived from grey-tone spatial-dependence matrices (known at present as grey-level co-occurrence matrices or GLCM). Haralick et al. used these texture-feature measures for category-identification of three different kinds of image data and achieved identification accuracies of more than 80% on each kind of image data test set. In the concluding remarks of the paper, Haralick et al. (1973) proposed that further investigations are necessary “to determine the size of the subimage region and the distances which should be used in computing the grey-tone dependence matrices” (p. 618). Some of proposed investigations recommended by Haralick et al. were carried out by Gebejes and Huertas (2013), and the study concluded that texture-feature measures such as contrast and dissimilarity (and not energy, entropy and homogeneity) show significant changes with changes in displacement (or distance) between pixels.

6 In an enlightening paper, O’Byrne et al. (2012) used the grey-level co-occurrence matrices (GLCM) from color images to develop a texture characteristics feature vectors consisting of seven measures – angular second moment, homogeneity, contrast, correlation, skewness, entropy, and kurtosis. Furthermore, O’Byrne et al. designed each texture measure in the feature vector to be the average of four texture measure values calculated in four GLCM matrices for each combination of inter-pixel distance, 𝑑, and angle of offset, 𝜃 between neighboring pixels. To reduce computational complexities, O’Byrne et al. quantized the grey levels between 1 – 12 instead of the typical scale of 1 – 255 for an 8-bit image. Even though GLCM was designed for greyscale images, O’Byrne et al. claimed that GLCM can be applied separately on each of the red, green, and blue channels for color images.

3.3

Image Feature Extraction & Selection

Padmapriya et al. (2012) presented a technique for deduction of bladder wall thickness using edge-based image segmentation. In this technique, images were first passed through the low pass filter to filter out high frequency image noise; then, thresholding was applied to convert images to binary images and the Sobel operator was used for edge detection. Zhao et al. (2012) proposed two methods (for road edge detection for visual navigation and control of mobile robots) based on color segmentation and threshold segmentation. In an informative paper, Dhage et al. (2015) presented an iris-pattern recognition method that involved pre-processing techniques such as Radon Transform for detection of lines and Top Hat Filtering for enhancement of gradient (or contrast) between brighter and darker pixels. Further on, Dhage et al. extracted salient iris features using Discrete Wavelet Transform (DWT) and Discrete Cosine Transform (DCT), and then selected the most important features for recognition using Binary Particle Swarm Optimization (BPSO). Finally, for classification, Dhage et al. used the Euclidian distance, and concluded that the algorithm worked successfully with low contrast and low illumination images. Varun et al. (2015), to tackle the problem of face recognition in varying pose and illumination conditions, proposed the use of Chirp Z-Transform and Goertzel algorithms

7 for combined pre-processing technique for image enhancement, division by blocks for feature extraction, Exponential Binary Particle Swarm Optimization for feature selection, and finally, the Euclidean for classification. To tackle a similar problem in varying illumination conditions, Yandunandan et al. (2015), in another paper, proposed a face recognition technique that involved the use of Hough Transform on blocks of images for feature extraction and Binary Particle Swarm Optimization (BPSO) for feature selection to accomplish the task of face recognition; finally, the Euclidian was used to classify test feature vectors in comparison to reference feature vectors. Xu et al. (2015) proposed a method of surface segmentation and edge feature lines extraction from 3D scans of fractured fragments. First, Xu et al. used a clustering algorithm to produce a rough surface segmentation; then, the concept of integral invariance was used to derive surface roughness, and difference between original and fractured surfaces. Finally, Xu et al. extracted edge feature lines. Xu et al. claimed that the algorithm showed robust performance with most fractured fragments. Abhishree et al. (2015) proposed a face recognition technique that involved the use of Anisotropic Diffusion as a pre-processing step for removal of noise, Gabor Filter for capturing facial features aligned at specific angles), and finally, Binary Particle Swarm Optimization (BPSO) for selection of features. Abhishree et al. stated that the use of discriminant features from BPSO resulted in high recognition rates and reduced feature subsets.

3.4

Damage Detection

In a well-rounded paper, Lim (2014) presented a robotic crack inspection and mapping (ROCIM) system that uses a sensor-equipped mobile robot to detect cracks and develop a crack map. In the ROCIM system, a sensor-equipped mobile robot is utilized to construct a 2D map of a bridge deck using a laser and to collect images of the bridge deck surface using a camera; Laplacian of Gaussian algorithm is used to detect local cracks in images. Fabio Pereira and Carlos Pereira (2015) investigated the possibilities of running image processing algorithms within computers inside UAVs, and used the Sobel Filter and

8 morphological operations such as image dilation, fill, and erosion to detect cracks (F. Pereira, 2015; C. Pereira, 2015). Sankarasrinivasan et al. (2015) proposed a combination of image processing algorithms such as HSV (or hue, saturation and value) Thresholding, Hat Transform, and Greyscale Thresholding for detection of structural defects such as cracks and surface degradations. O’Byrne et al. (2012) used Support Vector Machines (SVM), an unsupervised learning classifier, to classify regions of interest in images as either damaged or undamaged; the first step of the classification process involved training the algorithm using texture features to provide the basis for separation between the damaged and undamaged regions. Amza and Cicic (2014) proposed the use of Fuzzy Logic for automatic defect detection from segmented X-Ray images of industrial products. For each region of interest, Amza and Cicic extracted five features – area, perimeter, shape, roundness and togetherness. Using two Fuzzy modules, one to take automatic decisions to pinpoint defective regions and another to take final decisions based on logical criteria, Amza and Cicic presented results with higher reliability.

3.5

Structure Estimation

Hartley and Zisserman (2000) thoroughly described the raw basics of multiple view geometry and structure computation. Since then and especially over the recent years, Structure from Motion techniques have been implemented in a range of research fields. Westoby et al. (2012) proposed the use of a ground-based close-range terrestrial photogrammetry and computer vision approach to obtain spatial data for topographic reconstruction of landforms for geoscience applications. Kersten and Lindstaedt (2012) explored the feasibility for use of reconstruction techniques from multiple images for 3D documentation of architectural, cultural and archaeological objects. In the past, UAV systems have been used in research to investigate mapping and monitoring applications. Current trends in research indicate a growing interest in the use of Structure from Motion techniques on aerial image data collected using UAV systems.

9 Adams et al. (2016) attempted to explain the consequences of debris flow events with the help of a UAV system and photogrammetric techniques. Ryan et al. (2015) presented the application of a UAV and Structure from Motion techniques to investigate calving dynamics at a major marine-terminating outlet glacier that is currently draining the western section of the Greenland ice sheet. Ryan et al. acquired over 2000 overlapping, geotagged aerial images of a calving event at an approximate 40 cm sampling distance. Using this study, Ryan et al. observed changes in the glacier structure over daily and seasonal timescales, and reported key observations. Hackney and Clayton (2015) explored the use of UAV systems in geomorphological mapping applications. McLeod et al. (2013) investigated the feasibility of using video data acquired using a UAV and Structure from Motion techniques to obtain point cloud models for mining applications. Lucieer et al. (2012) used of a UAV system equipped with a sensor to acquire overlapping image data and used Structure from Motion techniques to derive dense point clouds to map the terrain of Antarctic moss beds. The visible color, near infrared, and thermal infrared sensors are used one at a time.

3.6

Geometry Feature Estimation

Curvature is one of the defining properties for surface geometry. A curvature metric for a surface measures the deviation of the surface from a tangent plane, and that there is more than one way to measure this deviation. Goldman (2005) presented a compilation of curvature formulae for implicit curves and surfaces. Kurita and Boulanger (1992) explained a method to compute Mean and Gaussian curvatures from range image data. Theisel et al. (2004) introduced an approach for estimating the curvature tensor of a triangular mesh. In this approach, Theisel et al. proposed the estimation of curvature for each mesh triangle instead of each vertex. Lack of analytical formulae makes it harder to calculate curvature from point cloud data (Foorginejad & Khalili, 2014). Foorginejad & Khalili pointed that curvature estimation methods can be classified into two categories – the surface fitting method and the discrete method. A function is usually chosen to model the local surface shape in the surface fitting method. On the other hand, the discrete method comprises either of a direct

10 approximation formula for curvature or an approximation of the curvature tensor. Zhihong et al. argued that in a general case, the discrete method leads to gain in computational time at the cost of accuracy (as cited in Foorginejad and Khalili, 2014, p. 348).

11

CHAPTER 4

4.1

| EXPERIMENTAL SETUP & METHOD

Data Acquisition & Description

A digital image can be described as a 2-D array, 𝑓(𝑥, 𝑦), having M rows and N columns, where (𝑥, 𝑦) are discrete coordinates (Gonzalez & Woods, 2008). Importantly, the number of image pixel (intensity) values, 𝐿, is an integer power of 2 such that 𝐿 = 2𝑘 . When an image can have 2𝑘 intensity values, it is conventional to refer to the image as a “𝑘-bit” image. In common practice, values of 𝐿 are defined within the interval [0, 𝐿 − 1]. For instance, for an 8-bit image 𝐿 ranges from 0 (black) to 255 (white). A CMOS 13megapixel sensor is used to acquire 3120x4160 pixel high resolution image data. The surface of interest chosen for validation of the algorithm consists of a section of a road surface made of concrete; this material surface is referred to as the reference material surface in this thesis. To maintain consistency, high resolution image data is collected from the same section of the road at distances ranging from 45 cm to 75 cm. The data is divided into three sets – Set A (Train Set), Set B (Validation Set), and Set C (Test Set).

Figure 1: Dry and Wet Reference Material Surface Data at High and Low Illumination Set A (Train Set) consists of image data of the reference material surface acquired from both dry and wet surfaces under high and low illumination conditions. Sections of the

12 Set A image data are cropped into 800x800 pixels representing the reference material surface under different conditions as illustrated in Figure 1. Set B (Validation Set) and Set C (Test Set) consist of unaltered image data (each 3120x4160 pixels in dimension). Set B is used to calibrate certain parameters in the algorithm and Set C is used to test the robustness of the algorithm.

4.2

Development & Computation Environments

Oracle VM VirtualBox Manager 2016 running a Linux Ubuntu 16.04 LTS system is the development environment chosen for this work. Oracle VM VirtualBox is used to allocate 8 Gb of overall system memory (or RAM) to the virtual environment running the Linux system. An Intel Core i7-6700 CPU running at 3.4 GHz is used. Advantages of development and validation in a Linux-based virtual environment include uninterrupted development, standalone storage allocation, and portable Linux installation that can be operated in any computer capable of running a virtual environment manager. Furthermore, running a Linux system in a virtual environment allows the ease of running Microsoft Windows and Linux Ubuntu at the same time; this allows ease of use of software in multiple operating systems simultaneously. Oracle VM VirtualBox Manager also allows creation of a file exchange channel between the Windows system and the Linux system. Using this method, large data files can be transported to the Windows environment for visualization in MATLAB 2016 and MeshLab 2016. Python is chosen to be the programming language of choice for this work. Python is an object-oriented language that is free and portable (since it compiles and runs on every major platform) (Lutz, 2013). It has a large collection of standard libraries and wide variety of third-party tools. In addition to standard libraries, NumPy and OpenCV are third-party libraries that are extensively used in the development of the algorithm. NumPy allows scientific computation to be done with ease. OpenCV is an open-source computer vision and machine learning library, which was constructed for computational efficiency with strong focus on real-time applications. Other significant libraries used are Pandas, ScikitLearn, and Scikit-Image.

13 4.3

Data Pre-Processing

4.3.1

Greyscale Transform

The image data in Set A, B and C is converted to greyscale using the simple transform as shown in (4-1). The transform reduces the amount of information in general, but retains most of the feature-related information in the image data. The resulting greyscale image data are weighted sums (of the red, green and blue color channels) with coefficients 𝛼, 𝛽, and 𝛾. The Open Computer Vision (OpenCV) library carries an implementation of the method where the coefficients are predefined. 𝑓𝐺𝑅𝐸𝑌𝑆𝐶𝐴𝐿𝐸 (𝑥, 𝑦) = 𝛼𝑓𝑅𝐸𝐷 (𝑥, 𝑦) + 𝛽𝑓𝐺𝑅𝐸𝐸𝑁 (𝑥, 𝑦) + 𝛾𝑓𝐵𝐿𝑈𝐸 (𝑥, 𝑦)

4.3.2

(4-1)

Histogram Equalization

The intensity values in an image can be thought of as random variables in the interval [0, 𝐿 − 1] (Gonzalez & Woods, 2008). A random variable can be described using a probability density function (or PDF), and a histogram approximates a PDF. Per Gonzalez and Woods (2008), histogram equalization automatically determines a transformation function that produces an output image with a uniform histogram and higher contrast. Equation (4-2) states the transformation function. 𝑠𝑘 = 𝑇(𝑟𝑘 ) = (𝐿 − 1) ∑𝑘𝑗=0 𝑝𝑟 (𝑟𝑗 ) =

(𝐿−1) 𝑀𝑁

∑𝑘𝑗=0 𝑛𝑗

(4-2)

𝑤ℎ𝑒𝑟𝑒 𝑘 = 0, 1, 2, … , 𝐿 − 1 Here, 𝑟𝑘 is the input image intensity value, 𝑠𝑘 is the output image intensity value, and 𝑇(𝑟𝑘 ) is the transformation called a histogram equalization. 𝑝𝑟 (𝑟𝑘 ) is defined as the probability of occurrence of intensity value 𝑟𝑘 , and is defined by (4-3). 𝑝𝑟 =

𝑛𝑘 𝑀𝑁

𝑤ℎ𝑒𝑟𝑒 𝑘 = 0, 1, 2, … , 𝐿 − 1

(4-3)

14 Solomon and Breckon (2011) argued that the overall histogram of an image may have a wide distribution of intensity values while local regions may be highly skewed towards one end of the spectrum; in such cases, it is preferable to enhance the contrast of these local regions independently. Adaptive histogram equalization achieves this by processing these regions independently. However, deriving histograms of local regions and equalizing the intensity values of those regions tend to amplify noise with too much increase in contrast. Solomon and Breckon (2011) further stated that equalizing the intensity histogram without control in each region can produce undesirable results. Contrast-limited adaptive histogram equalization (CLAHE) places a limit on the normalized increase in contrast in any given region. The implementation of this method in the OpenCV library is used in the algorithm development.

15 4.3.3

Feature Calculation Table 1: Features Calculated from Images INTENSITY FEATURE 1 𝑀𝑁

Mean

𝑀 ∑𝑁 𝑦=1 ∑𝑥=1 𝑙𝑥𝑦

(4-4)

where, 𝑙𝑥𝑦 represents intensity value, 𝑀 represents the rows and 𝑁 represents the columns.

Mean measures average intensity value of an image, where each intensity value of image 𝑓(𝑥, 𝑦) is represented by 𝑙𝑥𝑦 .

TEXTURE FEATURE Contrast

𝐾 2 ∑𝐾 𝑖=1 ∑𝑗=1(𝑖 − 𝑗) 𝑝𝑖𝑗

(4-5)

Contrast measures intensity contrast between a pixel and its neighbor. Contrast ranges from 0 (when 𝐺 is constant) to (𝐾 − 1)2 .

𝐾 ∑𝐾 𝑖=1 ∑𝑗=1 |𝑖 − 𝑗|𝑝𝑖𝑗

(4-6)

Dissimilarity is closest to Contrast except Dissimilarity grows linearly and Contrast grows quadratically.

(4-7)

Homogeneity measures the spatial closeness of distribution of values in 𝐺 to the diagonal. Homogeneity ranges from 0 to 1, with 1 being achieved when 𝐺 is diagonal.

(4-8)

Uniformity/Energy measures the uniformity in an image. Uniformity ranges from 0 to 1, with 1 being achieved for a constant image.

Dissimilarity

𝐾 ∑𝐾 𝑖=1 ∑𝑗=1

Homogeneity

Uniformity / Energy

𝑝𝑖𝑗 1+|𝑖−𝑗|

𝐾 2 ∑𝐾 𝑖=1 ∑𝑗=1 𝑝𝑖𝑗

𝐾 ∑𝐾 𝑖=1 ∑𝑗=1

(𝑖−𝑚𝑟 )(𝑖−𝑚𝑐 )𝑝𝑖𝑗

(4-9)

𝜎𝑟 𝜎𝑐

where, 𝐾

𝐾

𝑃(𝑖) = ∑ 𝑝𝑖𝑗

𝑎𝑛𝑑

𝑃(𝑗) = ∑ 𝑝𝑖𝑗

𝑗=1 𝐾

Correlation

𝑖=1 𝐾

𝑚𝑟 = ∑ 𝑖𝑃(𝑖)

𝑎𝑛𝑑

𝑚𝑐 = ∑ 𝑗𝑃(𝑗)

𝑖=1

𝑗=1

Correlation measures how correlated a pixel is to its neighbor. Correlation ranges from -1 to 1.

𝐾

𝜎𝑟 2 = ∑(𝑖 − 𝑚𝑟 )2 𝑃(𝑖) 𝑎𝑛𝑑 𝜎𝑐 2 𝑖=1

𝐾

= ∑(𝑗 − 𝑚𝑐 )2 𝑃(𝑗) 𝑗=1

Entropy

𝐾 − ∑𝐾 𝑖=1 ∑𝑗=1 𝑝𝑖𝑗 𝑙𝑜𝑔2 𝑝𝑖𝑗

(4-10)

Entropy measures the randomness of values of 𝐺. Entropy ranges from 0 to 2𝑙𝑜𝑔2 𝐾, with maximum occurring when all 𝑝𝑖𝑗 ’s in 𝐺 are equal.

16 Table 1 shows the two types of features that are extracted from images – intensity and texture. Equation (4-4) calculates the intensity feature, or the mean. Equations (4-5) through (4-10) calculate texture features such as contrast, dissimilarity, homogeneity, uniformity/energy, correlation, and entropy first defined by Haralick et al. (1973), and later, explained by Gonzalez and Woods (2008). Let 𝑄 be an operator that defines the spatial relationship between two pixels (called the reference pixel and the neighbor pixel) in an image 𝑓(𝑥, 𝑦) (Gonzalez & Woods, 2008). Generally, two variables 𝑑 and 𝜃 can define 𝑄, where 𝑑 is the inter-pixel distance and 𝜃 is the offset angle. Figure (3) illustrates three different 𝑄 operators with arbitrary combinations of 𝑑 and 𝜃. Let 𝐺 be a matrix whose element 𝑔𝑖𝑗 represent the number of times two pixels occur in 𝑓(𝑥, 𝑦) in specific intensity value combinations per the spatial relationship defined by 𝑄. Moreover, the elements of 𝐺 can be normalized to give

𝑝𝑖𝑗 =

𝑔𝑖𝑗 𝑛

(4-11)

where 𝑝𝑖𝑗 is the probability of occurrence of 𝑔𝑖𝑗 and 𝑛 is total number of pixel pairs that satisfy 𝑄. 𝐺 is referred to as grey-level co-occurrence matrix (GLCM), and calculations are performed using an implementation in the OpenCV library. Figure (2) illustrates an image on the left and the corresponding to the GLCM on the right with a specific relationship 𝑄. Texture features defined in Table 1 are evaluated from the GLCM. After all the texture features are calculated for all the samples in Set A, the values are accumulated into a matrix containing texture feature vectors for 𝑁 samples as shown in Figure (4).

17

Figure 2: Formation of the GLCM

Figure 3: Arbitrary Inter-Pixel Distance and Offset Angle Combinations

Figure 4: Dataset Matrix X Containing Texture Values for Set A

Table 2 shows GCLM texture feature values calculated from a randomly selected 800x800 pixels section of an image from the Validation Set (or Set B). The inter-pixel

18 distance is varied at different iterations to comprehend the variation of the different texture feature values. The data in Table 2 shows that at a constant offset angle, values of contrast and dissimilarity features increases with increasing inter-pixel distance; the values of the rest of the texture features decreases with increasing inter-pixel distance. Table 3 shows GLCM texture feature values calculated from another randomly selected 800x800 pixels section from Set B. The offset angle is varied at different iterations and the data in Table 3 does not show much variation in texture feature values. A Python subroutine for this method is shown in section A5 of Appendix A.

Table 2: GLCM Texture Features at Varying Inter-Pixel Distances Levels

256

Cell Dimension Offset Angle, 𝜃

800x800 0

45

0

45

450

450

450

GLCM Texture Features Iteration

1

2

3

4

5

Inter-Pixel Distance, 𝑑

2

4

6

8

10

Contrast

727.39

2660.67

3308.30

4177.57

4494.09

Dissimilarity

17.79

36.74

42.05

48.70

50.97

Homogeneity

0.092

0.041

0.034

0.027

0.025

Energy

8.65E-5

3.91E-5

3.44E-5

3.03E-5

2.92E-5

Correlation

0.897

0.622

0.530

0.407

0.362

Entropy

0.004

0.002

0.001

0.000

0.000

Iteration

6

7

8

9

10


12

14

16

18

20

Contrast

4754.36

5171.56

5351.16

5677.38

5820.05

Dissimilarity

52.82

55.65

56.84

58.99

59.93

Homogeneity

0.023

0.021

0.021

0.019

0.019

Energy

2.85E-5

2.75E-5

2.71E-5

2.66E-5

2.64E-5

Correlation

0.325

0.266

0.240

0.194

0.173

Entropy

0.000

0.000

0.000

0.000

0.000

19 Table 3: GLCM Texture Features at Varying Offset Angles Levels

256

Cell Dimension

800x800


10

10

10

10

3

4

10

GLCM Texture Features Iteration

4.4

1 0

2 0

0

5 0

Offset Angle, 𝜃

0

45

90

135

1800

Contrast

4413.95

4359.88

4268.06

4227.50

4431.95

Dissimilarity

50.36

50.13

49.43

49.08

50.36

Homogeneity

0.025

0.025

0.026

0.026

0.025

Energy

2.82E-5

2.83E-5

2.87E-5

2.88E-5

2.82E-5

Correlation

0.335

0.342

0.354

0.361

0.335

Entropy

0.000

0.000

0.0000

0.000

0.000

Dimensionality Reduction

The Karhunen-Loève Transform, commonly known as the Principal Component Analysis (PCA), has wide applications. Hotelling stated that PCA can be defined as the orthogonal projection of multivariate data onto a lower dimensional space such that the variance of the projected data is maximized (quoted in Bishop 561). Johnson and Wichern (2007) explained that although 𝐷 variables are required to produce the total variance in the system, much of the variability can be accounted for using 𝑀́ principal components, where 𝑀́ < 𝐷 ; the 𝑀́ principal components can then replace 𝐷 variables. Eventually, the original dataset containing 𝑁 measurements on 𝐷 variables is reduced to dataset comprising of 𝑁 measurements on 𝑀́ principal components. The goal of PCA is to find the mapping from the inputs in the original 𝐷-dimensional space to a new (𝑀́ < 𝐷)-dimensional space with minimum loss of information (Alpaydin, 2014). In this thesis, dimensionality reduction through PCA serves as the one of the prior steps. The intention behind the implementation of PCA is to reduce the algorithm’s computational cost further downstream. Figure (4) shows a dataset matrix 𝑋 of dimensionality 𝐷 = 6 (representing number of rows) and 𝑁 samples or observations (representing number of columns). In this case, 𝑁 = 200 for Set A. As explained by Bishop (2006) and in context of this thesis, let 𝑋 be represented as a dataset of observations {𝑥𝑛 } where 𝑛 = 1, … , 𝑁, and 𝑥𝑛 is a vector of

20 dimensionality 𝐷. The dataset is standardized before proceeding with the calculations to ensure equal weight given to each variable (“STAT 505 Applied Multivariate Statistical Analysis Lesson 11.5,” n. d.). The covariance matrix of the mean-centered (standardized) data is defined by 1

𝑇 𝑆 = ∑𝑁 𝑛=1(𝑥𝑛 − 𝑥̅ )(𝑥𝑛 − 𝑥̅ ) 𝑁

(4-12)

where, 1

𝑥̅ = ∑𝑁 𝑛=1 𝑥𝑛 𝑁

(4-13)

Variance of 𝑥𝑛 is defined by (4-14). Similarly, variance of 𝑥𝑛 projected on the direction of a 𝐷-dimensional vector 𝑤1 is stated by (4-15). 1

2 𝑉𝑎𝑟(𝑥𝑛 ) = ∑𝑁 𝑛=1(𝑥𝑛 − 𝑥̅ ) 𝑁

1

𝑇 𝑇 2 𝑇 𝑉𝑎𝑟(𝑤1 𝑇 𝑥𝑛 ) = ∑𝑁 𝑛=1(𝑤1 𝑥𝑛 − 𝑤1 𝑥̅ ) = 𝑤1 𝑆𝑤1 𝑁

(4-14) (4-15)

The next step is to maximize the variance of the projected data assuming the condition 𝑤1 𝑤1 𝑇 = 1 is satisfied. Using a Lagrange multiplier 𝜆1 , the constraint 𝑤1 𝑤1 𝑇 = 1 can be enforced as shown in (4-16). 𝑤1 𝑇 𝑆𝑤1 = 𝜆1 (1 − 𝑤1 𝑤1 𝑇 )

(4-16)

By setting the derivative of the above equation with respect to 𝑤1 equal to zero, the following is derived, which shows that 𝑤1 is the eigenvector (or the first principal component) of 𝑆 and 𝜆1 is the corresponding eigenvalue. 𝑆𝑤1 = 𝜆1 𝑤1

(4-17)

Additional principal components can be defined by choosing each new direction, which maximizes the variance of the projected data amongst all possible directions orthogonal to those already considered. Let the eigenvalues be defined such that 𝜆1 ≥ 𝜆2 ≥ ⋯ ≥ 𝜆𝐷 . Per Johnson and Wichern (2007), the variance-covariance structure of a set of variables in PCA can be explained algebraically using a few linear combinations (or principal components) of these variables. Equation (4-18) shows the linear combinations. Geometrically, these combinations represent the selection of a new coordinate system by

21 rotating the original system. The 2nd principal component and all subsequent components have the same property – they are linear combinations that consider as much of the remaining variation as possible with the additional constraint that they are not correlated with any other principal component such that 𝑐𝑜𝑣(𝑌𝑖 , 𝑌𝑗 ) = 0. A section of the Python subroutine executes this method as shown in section A1 the Appendix A. 𝑌1𝑛 = 𝑤1 𝑇 𝑥𝑛 = 𝑤11 𝑥1𝑛 + 𝑤21 𝑥2𝑛 + ⋯ + 𝑤𝐷1 𝑥𝐷𝑛

(4-18)

𝑌2𝑛 = 𝑤2 𝑇 𝑥𝑛 = 𝑤12 𝑥1𝑛 + 𝑤22 𝑥2𝑛 + ⋯ + 𝑤𝐷2 𝑥𝐷𝑛 𝑌𝐷𝑛 = 𝑤𝐷 𝑇 𝑥𝑛 = 𝑤1𝐷 𝑥1𝑛 + 𝑤2𝐷 𝑥2𝑛 + ⋯ + 𝑤𝐷𝐷 𝑥𝐷𝑛 𝑛 = 1, 2, … , 𝑁 Table 4: Contributions of Eigenvalues Proportion Component

Eigenvalue

of Contribution

Cumulative Contribution

1

4.513016

0.748409

0.748409

2

0.779996

0.129349

0.877758

3

0.557953

0.092527

0.970285

4

0.157380

0.026099

0.996384

5

0.021264

0.003526

0.999910

6

0.000541

8.97E-05

1

Table 4 shows the eigenvalues calculated from 𝑋, and their relative contributions in explaining the variance in 𝑋. The calculations show that the cumulative contribution up to the 3rd component is more than 97%, which means more than 97% of the variance is explained by the first three components. Figure (5) shows an illustration of the calculation results.

22

Figure 5: Scree Plot Figure (6) shows a plot of the projected data using the first principal component (or Y1n ) against the projected data using the second principal component (or Y2n ). Section A1 of the Appendix A shows the Python code named Alpha for performing the necessary calculations described above, and Figure (7) shows the flowchart for Alpha. Here, the ́ = 3 ) principal eigenvalue lower bound is set to 0.5 such that the first three (or M components are considered further downstream in the algorithm. Eventually, this helps reduce computation load further downstream, and in return for slightly reduced accuracy.

Figure 6: Projection of First & Second Principal Components

23

Figure 7: Flowchart for Algorithm Alpha (or alpha.py)

4.5

Data Comparison

This step involves the comparison between the data derived from the reference material surface images from Set A to that of sections of images from Set B (in the validation stage) and Set C (in the test stage). These sections are basically square image blocks or “grid cells” of a certain user-specified dimension; this dimension is chosen to be 400x400 pixels to draw a compromise between execution speed and decision efficiency. After dividing an image in either Set B or Set C into square image blocks, the algorithm compares computed variables from Set A data with those from each of the square image blocks in an image in Set B or C. A Python subroutine shown in section A3 of Appendix A helps to calculate number of square blocks to be compared and the coordinates of the square blocks. Now, the algorithm decides whether to classify an image block as a match or a mismatch with the reference material surface. As explained earlier in Chapter 2 of the thesis, it is important to remove or replace unnecessary information contained in sections of an image; these

24 sections might contain data that do not represent the reference material surface. If there is a match, the corresponding image block is left unaltered. However, if there is a mismatch, the information in the corresponding image block is removed and replaced with an intensity value chosen by the user. This intensity value is called the Maximum value for Binary Thresholding. Next, the algorithm generates a modified image based on the decision. Section A2 of the Appendix A shows the Python subroutine. Turk and Pentland (1991) described a face recognition method that operates by projecting new face images onto a feature space that spans the significant variations described by the significant eigenvectors (or principal components) of known face images. Turk and Pentland stated that the projection operation characterizes an individual face image by the weighted sum of the eigenvector features; for recognition of a face image, it is only necessary to compare the weights of that face image to those of multiple known face image classes. Following a similar approach, an (𝑀́𝑋𝑁)-dimensional texture-feature train weight vector is calculated from each image in Set A using (4-19). Then, mean is calculated over all 𝑁 Set A images using (4-20). 𝜔𝑡𝑟𝑎𝑖𝑛 = 𝑤𝑘 𝑇 (𝑥𝑛 − 𝑥̅ )

𝑘 = 1, 2, … , 𝑀́

1

𝜔 ̅𝑡𝑟𝑎𝑖𝑛 = ∑𝑁 1 𝜔𝑡𝑟𝑎𝑖𝑛 𝑁

(4-19) (4-20)

For image in each block cropped from original image data in Set B or Set C, the (validation or test) texture-feature vector 𝛤 is calculated in the subroutine section A2 of Appendix A; the texture-feature (validation or test) weight vector is calculated using (4-21). 𝜔𝑣𝑎𝑙𝑖𝑑𝑎𝑡𝑖𝑜𝑛/𝑡𝑒𝑠𝑡 = 𝑤𝑘 𝑇 (𝛤 − 𝑥̅ )

𝑘 = 1, 2, … , 𝑀́

(4-21)

Finally, two comparison methods are tested for algorithm speed – the Euclidean Distance and the One-Class Support Vector Machine.

4.5.1

Euclidean Distance

In this method, the algorithm follows a two-step method for detection of reference material surface in the image blocks from images in Set B or Set C. In the first step, “similarity” or “closeness” is measured between an averaged intensity value of images in Set A and

25 intensity values of each block from images in Set B or Set C using the Euclidean distance equation shown in (4-22). In the second step, a Minkowski distance of order 𝑝 = 2 is implemented to measure the similarity between the train and validation/test weight vectors as shown in (4-23). Please note that a Minkowski distance of order 𝑝 = 2 represents Euclidean distance. ̅ 𝑑𝐼𝑁𝑇𝐸𝑁𝑆𝐼𝑇Y = ‖𝑙𝑡𝑟𝑎𝑖𝑛 − 𝑙𝑣𝑎𝑙𝑖𝑑𝑎𝑡𝑖𝑜𝑛/𝑡𝑒𝑠𝑡 ‖ 𝑑 𝑇𝐸𝑋𝑇𝑈𝑅𝐸 = [∑𝑛𝑖=1|𝜔 ̅𝑡𝑟𝑎𝑖𝑛,

(4-22) 𝑝 1/𝑝

𝑖

− 𝜔𝑣𝑎𝑙𝑖𝑑𝑎𝑡𝑖𝑜𝑛/𝑡𝑒𝑠𝑡, 𝑖 | ]

(4-23)

𝑤ℎ𝑒𝑟𝑒 𝑛 = 1, 2, … , 𝑀́ Finally, (4-22) and (4-23) calculation results are compared against predefined thresholds (or upper bounds) to reach a decision. Due to high computation requirements and dependence on a priori selection of upper bounds, this method is not implemented for comparison.

4.5.2

One-Class Support Vector Machine

The problem at hand can be categorized as an image classification problem. Convolutional Neural Networks (CNNs) have been successfully implemented in image classification problems (Ciresan et al., 2011). Recent studies have shown computation speed improvements in classification problems using CNNs (Cong & Xiao, 2014). However, as stated earlier, Set A consists of image data for the reference material surface. This presents an issue of the data being class-imbalanced, which means there is more of data from one class (of reference material surface) and less or no data from other classes. Hensman and Masko (2015) acknowledged that imbalanced training data can have negative impact on the overall performance of CNNs; in contrast, balanced training data produces better results. Support Vector Machines (SVMs) are learning models that can be used in both classification and regression problems. A One-Class SVM can be used to “train” an anomaly detection model using dataset containing normal cases (“One-Class Support Vector Machine,” 2016). Schölkopf et al. (1997) stated that an SVM algorithm with a Radial Basis Function (RBF) approach automatically determines centers, weights, and threshold with the objective to minimize an upper bound on the expected test error. The

26 Scikit-Learn Machine Learning library in Python carries an implementation of the OneClass SVM (Pedregosa et al., 2011). Table 5: Parameters & Computation Times for Comparison Methods ONE-CLASS SUPPORT VECTOR MACHINE Radial Basis Function (RBF) 𝑘𝑒𝑟𝑛𝑒𝑙 = 𝑅𝐵𝐹 Input Parameters

𝜈 (𝑛𝑢) = 0.01 𝛾 (𝑔𝑎𝑚𝑚𝑎) = 0.01

Processor Computation Time for Comparison to an Image Block from a Set B Image

Upper bound on the fraction of training errors and lower bound on the fraction of support vectors in the interval (0,1] Kernel coefficient for ‘RBF’ 0.625847 s

Table 5 shows the input parameters and typical computation time for identification of a block as either match or mismatch. The One-Class Support Vector Machine method has the advantage of limiting subjective input since it automatically determines the necessary parameters with the objective to minimize an upper bound on expected test error.

4.6

Damage Detection & Quantification

The algorithm, in the previous step, removes sections of the test image that do not match the reference material surface. In this step, the algorithm seeks to further purify the image by applying the bilateral filtering and binary thresholding transforms on the greyscale version of the modified image. Per Tomasi and Manduchi (1998), the application of bilateral filtering smoothens an image while preserving edges by combining nearby image (intensity) values in a nonlinear fashion. After smoothing the image, binary thresholding is applied to the image to segment regions within an image based on pixel values of the image. Through application of these transforms, the algorithm reduces information in the image to two sets of pixels with different intensity values. One set of pixels represent the regions of the test picture with damage (which in this case is black with intensity value 255) and the other set of pixels represent the background (which in this case is set to intensity value 125. This procedure creates a clear segmentation between the background and damaged regions. Let 𝐿𝐷 represent the set of pixels that signify regions of damage, and let 𝐿𝐵 represent the set of pixels that represent the background. Therefore, the percentage of

27 possible damage detected (or PPD) by the algorithm on a test image can be calculated using (4-24). 𝑃𝑒𝑟𝑐𝑒𝑛𝑡𝑎𝑔𝑒 𝑜𝑓 𝑃𝑜𝑠𝑠𝑖𝑏𝑙𝑒 𝐷𝑎𝑚𝑎𝑔𝑒, 𝑃𝑃𝐷 =

100𝐿𝐷 𝐿𝐷 +𝐿𝐵

(4-24)

Section A2 of Appendix A shows the algorithm (named Beta) that carries out the computations listed in sections 4.5 and 4.6 of this chapter. The flowchart for Beta is shown in Figure (8).

28

Figure 8: Flowchart for Algorithm Beta (or beta.py)

29 4.7

Geometry Generation

There are two types of geometry acquisition approaches – active and passive (Fuhrmann et al., 2014). Time-of-Flight camera, Microsoft Kinect, and LiDAR (Light Detection and Ranging) are devices currently used for active geometry acquisition. Generally, such devices emit electromagnetic waves and calculate the distance to an arbitrary target by measuring the time taken for the waves to reflect off the target and come back. Clearly, these devices help to estimate the geometry of the nearby environment in real-time, and hence they are categorized as active geometry acquisition devices. On the other hand, the passive approach consists of capturing image data using electro-optical sensors such as CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), which is found in consumer-grade cameras, smart phones, etc. These devices are readily available, and in addition, cheaper and simpler in comparison to the active geometry acquisition devices. If the PPD calculation in the previous section indicates presence of damage, further scrutiny is necessary to gain a good understanding of the structure or geometry of the damage. Table 6 shows description of the validation datasets collected for geometry generation. Each of the datasets contains overlapping images acquired from slightly offset viewpoints. Table 6: Description of Validation Datasets for Geometry Generation Surface Crack Data

Surface Depression Data

Dataset Name

Number of Images

Dataset Name

Number of Images

Crack Dataset 1

17

Depression Dataset 1

22

Crack Dataset 2

14


14

Crack Dataset 3

11


12

Crack Dataset 4

11


13

Crack Dataset 5

8


8

Crack Dataset 6

12


8

Crack Dataset 7

9


9

In an inspiring paper, Fuhrmann et al. (2014) introduced an end-to-end multi-view geometry reconstruction software named Multi-View Environment (or MVE). MVE uses

30 multiple overlapping image data of a scene acquired from different viewpoints to construct a textured 3D mesh of the scene. The MVE software executes a Structure from Motion (SfM) protocol, computes dense depth maps using the Multi-View Stereo (MVS) algorithm (Goesele et al., 2007), constructs global surface mesh from dense MVS data using the Floating Scale Surface Reconstruction (FSSR) algorithm (Fuhrmann & Goesele, 2014), and finally generates texture for the mesh model using the TexRecon algorithm (Waechter et al., 2014). MVE carries out Structure from Motion in a few steps – extract intrinsic parameters (such as focal length and radial distortion of the camera) and extrinsic parameters (such as camera position and camera orientation), detect features using SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Feature) protocols simultaneously, match features between pairs of images and carry out incremental reconstruction of the geometry in the form of point clouds. Then, the MVS algorithm reconstructs dense 3D geometry by finding visual correspondences in image data; in this step, depth maps are created for every scene view to facilitate the generation of detailed geometry and overcome noise issues. In the next step, the FSSR algorithm merges the MVS output to create globally consistent mesh data. Finally, texture is applied to the mesh data suing the TexRecon algorithm. Waechter et al. (2014) proposed that this method is applicable for large-scale geometry models and claimed that the method addresses challenging scenarios such as large number of input images with varying scales and qualities. OpenDroneMap is an open-source toolkit developed mostly on C++ and Python languages, and is currently under development. Version 0.2 of this toolkit is readily available for download from GitHub. The OpenDroneMap toolkit relies on the state-ofthe-art texturing techniques developed by Waechter et al. (2014). Using the OpenDroneMap toolkit, 3D point cloud models and mesh models are generated in Polygon File Format (as .PLY data files). MeshLab 2016 can read this file format and produce visual representations of the point cloud data and the mesh data. A Python subroutine named Gamma (shown in section A6 of Appendix A) reads the data in the .PLY files and uses the data to perform further calculations.

31

Figure 9: Flowchart for Performing Geometry Generation

Figure 10: Flowchart for Algorithm Gamma (or gamma.py)

Figure (9) shows the flowchart for the geometry generation step, and Figure (10) shows the flowchart for Gamma.

32 4.8

Feature Extraction

In this step, curvature and luminance features are calculated from the mesh and point cloud model data generated in the previous step. The calculation of curvature relates to the calculation of surface normals from mesh data. However, this thesis only explains the curvature calculations. A point on a geometric surface can be defined in terms of Cartesian coordinates as (𝒙, 𝒚, 𝒛) with respect to a given origin. Per Rusu (2009), the problem of determining the normal to a query point on a geometric surface is approximated by the problem of estimating the normal of a plane tangent to the surface. Therefore, the problem can be thought of as a least-square plane fitting estimation problem. An analysis of eigenvectors and eigenvalues of a covariance matrix created from the nearest neighbors of a query point is required for this endeavor. For each query point 𝒑𝒊 , the covariance matrix 𝑪 is constructed per (4-25) (“Estimating Surface Normals in a Point Cloud,” n. d.). Here, 𝒌 is ̅ represents the 3D centroid of the the number of point neighbors considered about 𝒑𝒊 , and 𝒑 nearest neighbors. Per (4-26), 𝝀𝒋 is the 𝒋-th eigenvalue of the covariance matrix, and 𝒗𝒋 is the 𝒋-th eigenvector. 𝟏

̅). (𝒑𝒊 −𝒑 ̅)𝑻 𝑪 = ∑𝒌𝒊=𝟏(𝒑𝒊 − 𝒑 𝒌

𝑪. ⃗⃗⃗ 𝒗𝒋 = 𝝀𝒋 . ⃗⃗⃗ 𝒗𝒋

𝒋 𝝐 {𝟎, 𝟏, 𝟐}

𝒄𝒖𝒓𝒗𝒂𝒕𝒖𝒓𝒆 =

𝝀𝟎 𝝀𝟎 +𝝀𝟏 +𝝀𝟐

(4-25) (4-26) (4-27)

After the eigenvalues are calculated, (4-27) is used to calculate the surface curvature. Equation (4-27) shows the surface curvature as a relationship between the eigenvalues of the covariance matrix; here, 𝝀𝟎 is the smallest eigenvalue. As shown above, curvature can be estimated using the ratio between the minimum eigenvalue and the sum of the eigenvalues; this method of curvature calculation is known as surface variance (Foorginejad & Khalili, 2014). Foorginejad and Khalili argued that the surface variance method is appropriate for point cloud data as it doesn’t consume computation time, and only uses the coordinates of the points of concern and their neighbors. Curvature values are calculated for each of the datasets defined in Table 6. Following the curvature calculations, the curvature spectrum data from different validation

33 datasets are plotted in MATLAB 2016. Through inspection of the plots of some of the validation datasets, it can be concluded that most of the curvature values are between -2 to +2 range; in comparison, there are just a few curvature values that lie outside that range. This observation can be used to define a threshold 𝝍 = 𝟐, and use this threshold to divide curvature values from any dataset into three subsets – positive (with 𝒄𝒖𝒓𝒗𝒂𝒕𝒖𝒓𝒆 > 𝝍), negative (with 𝒄𝒖𝒓𝒗𝒂𝒕𝒖𝒓𝒆 < −𝝍 ) and flat surface (with −𝝍 ≤ 𝒄𝒖𝒓𝒗𝒂𝒕𝒖𝒓𝒆 ≤ 𝝍) . Figure (11) shows positive and negative curvature spectrum data for two validation datasets – one for surface crack and one for surface depression. The number of positive, negative, and flat surface curvature values can be defined as 𝝋𝑷 , 𝝋𝑵 , and 𝝋𝑭 respectively.

Figure 11: Curvature Spectrum Data from Surface Crack Damage and Surface Depression Damage Validation Datasets

34 Tables 11, 12, 13, 14, 15, and 16 in Appendix B show statistical data generated from the three subsets of curvature values. Scrutiny of the statistical data reveal that some metrics can be defined to provide intuitive understanding of the surface topography in the overall mesh data for each dataset. Table 7 shows the metrics and the thresholds defined for classification between two types of surfaces – surface with presence of crack damage and surface with presence of depression damage. The Positive-to-Flat (PF) ratio metric defines the ratio of the number of positive curvature regions to the number of flat surface regions. Similarly, the Negative-to-Flat (NF) ratio metric defines the ratio of the number of negative curvature regions to the number of flat surface regions. Finally, the Flat-to-All (FA) ratio defines the ratio of the number of flat surface regions to overall number of curvature values calculated from the mesh data for each dataset. Tables 17 and 18 in Appendix B show the values of PF, NF and FA ratios calculated for each dataset. Furthermore, luminance or color feature values are calculated using (4-28) from the point cloud model data. 𝒍𝒖𝒎𝒊𝒏𝒂𝒏𝒄𝒆 = 𝟎. 𝟐𝟏𝟐𝟔𝒍𝑹 + 𝟎. 𝟕𝟏𝟓𝟐𝒍𝑮 + 𝟎. 𝟎𝟕𝟐𝟐𝒍𝑩

(4-28)

Table 7: Defined Metrics from Curvature Statistical Data, and Selected Thresholds for Classification Classification Criteria Presence of Presence of Surface Surface Crack Depression

Metric Definition

Metric Symbol or Formulae

Selected Threshold

Number of Positive Curvature Regions Number of Negative Curvature Regions Number of Flat Surface Regions Positive-to-Flat (PF) Ratio Negative-to-Flat (NF) Ratio Flat-to-All (FA) Ratio

𝝋𝑷

300

𝝋𝑷 < 𝟑𝟎𝟎

𝝋𝑷 ≥ 𝟑𝟎𝟎

𝝋𝑵

300

𝝋𝑵 < 𝟑𝟎𝟎

𝝋𝑵 ≥ 𝟑𝟎𝟎

𝝋𝑭

99550

𝝋𝑭 > 𝟗𝟗𝟓𝟓𝟎

𝝋𝑭 ≤ 𝟗𝟗𝟓𝟓𝟎

(4-29)

0.003

𝑷𝑭 < 𝟎. 𝟎𝟎𝟑

𝑷𝑭 ≥ 𝟎. 𝟎𝟎𝟑

(4-30)

0.003

𝑵𝑭 < 𝟎. 𝟎𝟎𝟑

𝑵𝑭 ≥ 𝟎. 𝟎𝟎𝟑

(4-31)

0.995

𝑭𝑨 > 𝟎. 𝟗𝟗𝟓

𝑭𝑨 ≤ 𝟎. 𝟗𝟗𝟓

𝑷𝑭 = 𝑵𝑭 = 𝑭𝑨 =

𝝋𝑷 𝝋𝑭 𝝋𝑵 𝝋𝑭 𝝋𝑭

𝝋𝑷 +𝝋𝑵 +𝝋𝑭

Algorithm Gamma performs all the calculations discussed in this section. Using the six metrics defined in Table 7, Gamma performs six tests and a score-based classification. For example, if number of positive curvature regions for mesh data from an unknown test set falls within the 𝝋𝑷 < 𝟑𝟎𝟎, the presence of surface crack damage is “awarded” one

35 point. Subsequent tests are performed to derive total scores for both the presence of surface crack damage and the presence of surface depression damage.

36

CHAPTER 5

5.1

| RESULTS & DISCUSSION

Results & Discussion for Alpha & Beta

Alpha extracts intensity and texture information from train images in Set A (or the Train Set). Beta uses the extracted information to detect damage in test images in Set C (or the Test Set); moreover, Beta calculates the PPD (or Percentage of Possible Damage) metric. Then, an open-source software toolkit (based on the research by Fuhrmann et al. (2014), Fuhrmann & Goelese (2014) and Waechter et al. (2014)) processes multiple overlapping images of the damaged region and generates point cloud and mesh models. Finally, Gamma uses the point cloud and mesh models to calculate luminance and curvature values; Gamma also uses the curvature values to calculate metrics to indicate most prominent type of damage. The development of Alpha and Beta is influenced by objectives defined in Chapter 2 of the thesis. These objectives are driven by the motivation to implement the method in autonomous real-time damage detection applications. Chapter 4 of the thesis provides accurate details related to the development. The results are presented in this section. Table 8 lists the performance evaluation criteria for Beta and the results. The first criterion tested is the algorithm’s ability to detect damage in the reference material surface; Figure (12) shows the results produced by Beta. In the first performance evaluation criteria, Beta calculated PPD (or Percentage of Possible Damage) values that are consistent with the amount of damage one notices subjectively in the result images.

37

Figure 12: Samples of Test Image (Left) and Result Image (Right) with Percentage of Possible Damage (PPD) and Time Taken for Algorithm Execution

38 Table 8: Performance Evaluation of Algorithm Beta on Test Images from Set C Description of Algorithm Objective

Damage Detection in Reference Material Surface

Algorithm Performance Evaluation Criteria

Blur Avoidance

Low Illumination Avoidance

Unknown Material Rejection

Unknown Object Rejection

Markings Rejection

Material State Insensitivity

The algorithm should detect damage in any test image consisting only of the reference material surface with damage. The algorithm should “accept” blurred image blocks of any test image consisting only of the reference material surface. The algorithm should “accept” image blocks of any test image consisting only of the reference material surface in low illumination conditions. The algorithm (if trained with data from one type of reference material surface) should “reject" blocks consisting of unknown materials. The algorithm should “reject” blocks consisting of unknown objects. The algorithm should “reject” image blocks of any test image consisting of the reference material surface with markings. The algorithm should “accept” image blocks of any test image consisting only of the reference material surface in dry or wet state.

Number of Images Tested

Average Image Blocks Rejected

Average PPD Detected

Average Time Taken

22

8.23

3.71%

42.66 s

4

88

0%

33.33 s

8

23.63

9.90%

39.47 s

4

73.50

0.06%

36.52 s

4

4.75

5.58%

40.73 s

18

11.89

3.79%

42.18 s

8

6

5.27%

40.34 s

Figure (13) shows the results when tested for the blur avoidance and low illumination avoidance criteria. Beta produces poor results when tested for blur avoidance and low illumination avoidance. When blurry image data is presented, the algorithm rejects all test blocks from the test image, i.e. it “considers” all test blocks as mismatches with the reference material surface. Eventually, the algorithm calculates a PPD of 0% even though damage is present in the blurry image data. Similarly, Beta produces poor results when

39 tested for low illumination avoidance. The algorithm rejects or removes regions of reference material surface under low illumination conditions such as shadows.

Figure 13: Results for Blur Avoidance (Top) and Low Illumination Avoidance (Bottom) Criteria

Figure 14: Results for Unknown Material Rejection (Top) and Unknown Object Rejection (Bottom) Criteria

40 Figure (14) shows the results when tested for the unknown material rejection and the unknown object rejection criteria. It is clear from the results that Beta successfully detects blocks of images containing unknown materials, and rejects or removes them before calculation of the PPD metric. Additionally, Beta successfully removes sections or blocks of images containing unknown objects on the references surface material; in this case, the illustrations clearly show that sections of images containing the blue piece of paper are removed.

Figure 15: Results for Markings Rejection Criteria The illustrations in Figure (15) show the results generated for the markings rejection criterion. The objective for success in this criterion is to hinder the registration of traces of surface markings into the result image before calculation of the PPD metric. Beta successfully accomplishes this task. However, Beta fails to retain information containing sections of surface damage in a few locations. Furthermore, Beta also registers regions of the surface containing low illumination in the test images as regions of damage in the corresponding result images. This phenomenon is also noticeable in Figure (13). Figure (16) shows examples of the performance of Beta in the material state insensitivity test. The objective for success in this criterion is to detect surface damage regions in the result image regardless of whether the surface is dry or wet. Beta successfully

41 achieves this goal. However, Beta registers traces of regions of wet surface in the test images as sections of damage in the corresponding result images.

Figure 16: Results for Material State Insensitivity

5.2

Results and Discussion for Gamma As stated earlier, point cloud and mesh models should be generated prior to

luminance and curvature calculations. Figures (17) and (18) show illustrations of the point cloud models and mesh models generated using the open-source toolkit OpenDroneMap Version 0.2 and displayed using MeshLab 2016. The overall objective of Gamma is to differentiate between two types of damages – surface crack damage and surface depression damage – or at least numerically indicate with reasonable confidence the presence of one type of damage over the other.

42

Figure 17: Point Cloud Model and Mesh Model for Surface Crack Damage Data from Validation Set

Figure 18: Point Cloud Model and Mesh Model for Surface Depression Damage Data from Validation Set Table 9 shows the calculation results for four test sets of crack damage and four test sets of depression damage, and Figures (19) through (26) illustrate the damages and their corresponding mesh model representations. The points calculated for the crack damage sets show that Gamma identified all test sets successfully. However, Gamma failed to identify Depression Damage Set 1 and 4. Furthermore, visualizations of the luminance data calculated from point clouds models of surface crack and surface depression damages showed that luminance values cannot be used to distinguish between different types of damages.

43 Table 9: Calculation Results Generated by Gamma for Crack Damage and Depression Damage Test Sets

Test Set

Crack Damage Test Set 1 Crack Damage Test Set 2 Crack Damage Test Set 3 Crack Damage Test Set 4 Depression Damage Test Set 1 Depression Damage Test Set 2 Depression Damage Test Set 3 Depression Damage Test Set 4

𝝋𝑷

𝝋𝑵

𝝋𝑭

Points for

Points for

𝑷𝑭

𝑵𝑭

𝑭𝑨

Surface

Surface

Ratio

Ratio

Ratio

Crack

Depression

Damage

Damage

53

70

99906

0.00053

0.00070

0.99877

6

0

120

70

99834

0.00120

0.00070

0.99810

6

0

174

207

99643

0.00175

0.00208

0.99619

6

0

141

127

99761

0.00141

0.00127

0.99732

6

0

218

129

99669

0.00219

0.00129

0.99653

6

0

289

399

99347

0.00290

0.00401

0.99312

2

4

769

463

98803

0.00778

0.00469

0.98768

0

6

221

184

99610

0.00222

0.00185

0.99595

6

0

44

Figure 19: Image and Mesh Model Representation for Crack Damage Dataset 1


45



46

Figure 23: Image and Mesh Model Representation for Depression Damage Dataset 1


47



48 Table 10: Calculation Results Generated by Gamma for Other Damage Test Sets Test Set Damage Test Set 1 Damage Test Set 2 Damage Test Set 3 Damage Test Set 4 Damage Test Set 5 Damage Test Set 6 Damage Test Set 7 Damage Test Set 8 Damage Test Set 9

𝑵𝑭 Ratio

𝑭𝑨 Ratio

Points for Surface Crack Damage

Points for Surface Depression Damage

𝝋𝑷

𝝋𝑵

𝝋𝑭

𝑷𝑭 Ratio

85

63

99877

0.00085

0.00063

0.99852

6

0

361

446

99223

0.00364

0.00449

0.99193

0

6

198

228

99584

0.00199

0.00229

0.99574

6

0

1445

754

97845

0.01477

0.00770

0.97801

0

6

105

75

99843

0.00105

0.00075

0.99820

6

0

301

111

99634

0.00302

0.00111

0.99588

4

2

475

666

98904

0.00480

0.00673

0.98859

0

6

134

200

99681

0.00134

0.00201

0.99666

6

0

955

368

98695

0.00968

0.00373

0.98677

0

6

Table 10 shows results of additional tests performed on damage test sets. The intent here is to let Gamma indicate the type of damage instead of naming the damage set a priori to tests. Appendix C contains illustrations for the additional test sets. Some of these test sets contain multiple types of damages in the same set. Through this exercise, it was found that Gamma “awards” points to the type of damage that is most prevalent in the mesh model generated from the set of image data. However, it is not visually clear why Gamma classifies Damage Test Set 7 (containing only surface crack damage) as having surface depression.

49

CHAPTER 6

6.1

| CONCLUSION

Practical Issues, Limitations, Solutions & Prospects

Among the practical issues related to the objectives of the thesis is the possible inability of the algorithm to distinguish between a surface crack and a structural edge. Since both present sharp changes in intensity and curvature, practical solutions have to be devised. Another practical challenge is the detection of damage on transparent surfaces. Contemporary UAV systems do not have spherical fields of vision, i.e. image or video data can be captured from under and from the side of the UAV system, but not over. This is a limitation that might prevent contemporary UAV systems from being used to inspect or monitor structures from underneath. Low illumination also presents challenges that need to be addressed. Future work on algorithm development (for structural health monitoring applications using UAV systems) should focus on these issues. In retrospect to results presented in Chapter 5 of the thesis, the algorithm is not robust enough to produce accurate results when processing image data with regions of low illumination and blur. Data acquisition in the proper conditions can solve these issues. The algorithm execution speed of Beta is not fast enough for real-time applications. Possible improvements in hardware or algorithm development can solve this problem. The use of field-programmable gate arrays (or FPGAs) can solve the issue. In addition, the algorithm has several input parameters (defined in Figures 7, 8 and 10) that should be decided on by the user before execution of the algorithm. Self-calibrating algorithms are more independent of subjective input, and are often desirable. In summary, further development of the algorithm requires more image data for validation and test purposes, more trial runs in different scenarios, and eventually, more modifications and upgrades based on learnings. The knowledge presented in this thesis lays foundations for future work. The use of UAV systems not only present prospects in the field of structural health monitoring applications, but also other scenarios. In addition to applications discussed in Chapter 3 of the thesis, UAV systems could be used in other applications such as monitoring of biodiversity, detection of oil spill severity, inspection of forest fire damage, monitoring of shallow water coral health, etc.

50 6.2

Conclusion This thesis proposes algorithm-based solutions for damage detection, quantification

and identification. Different types of scenarios are considered during the validation process of the damage detection algorithm. The damage detection part of algorithm shows good results in most scenarios. The results also indicate room for improvement in some categories such as low illumination avoidance and blur avoidance. The damage identification part of the algorithm shows decent results in classifying two types of damage – surface crack damage and surface identification damage. It can be concluded that the algorithm “awards” points to the type of damage that is most prevalent in the mesh model. However, to understand the behavior the algorithm properly in terms of damage identification, more study is required.

51

APPENDIX A. CODE

A1.

ALPHA (alpha.py)

# ----// Alpha // ----------------------------------------------------------------------# --------------------------------------------------------------------------------------import cv2, glob, math, matplotlib.pyplot as plt, numpy as np, pandas as pd import sys, seaborn as sns, time # --------------------------------------------------------------------------------------from numpy import linalg as la from pandas import DataFrame # --------------------------------------------------------------------------------------from CALCULATE_mean_stddev import * from CALCULATE_prop import * from PERFORM_PCA import * # --------------------------------------------------------------------------------------Q_distance = [10] # VARIABLE // GLCM Calculation, Inter-Pixel Distance Q_angle = [np.pi/4] # VARIABLE // GLCM Calculation, Offset Angle levels = 256 # VARIABLE // Intensity Levels for GLCM Calculation eVals_thresh = 0.5 # VARIABLE // Eigenvalue Threshold for Selection of M Principal Components # --------------------------------------------------------------------------------------standardize = 1 # SWITCH // Standardize Data # [ROW > NUMBER OF FEATURES, COLUMN > NUMBER OF SAMPLES] # --------------------------------------------------------------------------------------# NOTE: This code reads Train (or Set A) image data, uses custom functions to calculate image # properties and to perform principal component analysis calculations, and finally # generates output data files and plots. # ---------------------------------------------------------------------------------------

start = time.clock()

# CALCULATE IMAGE PROPERTIES AND STANDARDIZE VALUES FOR PRINCIPAL COMPONENT ANALYSIS ---# --------------------------------------------------------------------------------------train_files = glob.glob('files/train_image/*') # GLOB // LOCATE REFERENCE MATERIAL IMAGE FILES TO CREATE MODEL glcm_matrix = np.array([]) # --------------------------------------------------------intensity = np.array([]) # --------------------------------------------------------contrast = np.array([]) dissimilarity = np.array([]) homogeneity = np.array([]) energy = np.array([]) correlation = np.array([]) entropy = np.array([]) # NUMPY // INITIALIZE EMPTY NUMPY ARRAYS for i in range(len(train_files)): train_im = cv2.imread(train_files[i]).astype(np.uint8) # OPENCV // READ IMAGE AS UNASSIGNED INTEGER (WITHIN 0 TO 255 PIXEL VALUES) mn, con, dis, hom, ene, cor, ent = CALCULATE_prop(train_im, Q_distance, Q_angle, levels) # CALCULATE_prop // EXTRACT FEATURE VALUES FROM ONE IMAGE intensity = np.append(intensity, mn) contrast = np.append(contrast, con) dissimilarity = np.append(dissimilarity, dis) homogeneity = np.append(homogeneity, hom) energy = np.append(energy, ene) correlation = np.append(correlation, cor) entropy = np.append(entropy, ent) # NUMPY // COLLECT GREY-LEVEL PROPERTY VALUES IN ROWS # END mean_intensity_data = np.mean(intensity, axis = 0)[np.newaxis] # NUMPY // CALCULATE MEAN OF INTENSITY PROPERTY

52

org_texture_data = np.vstack((contrast, dissimilarity, homogeneity, energy, correlation, entropy)) # NUMPY // STACK TEXTURE PROPERTY VALUES IN SEQUENCE VERTICALLY [ROW > NUMBER OF FEATURES, COLUMN > NUMBER OF SAMPLES] mean_texture_data = np.mean(org_texture_data, axis = 1)[np.newaxis] mean_texture_data = np.transpose(mean_texture_data) # NUMPY // CALCULATE MEAN OF ORGANIZED TEXTURE PROPERTY VALUES DATA AND TRANSPOSE [ROW > NUMBER OF FEATURES, COLUMN > 1] # --------------------------------------------------------std_texture_data = np.zeros(org_texture_data.shape) if standardize == 1: for i in range(org_texture_data.shape[0]): data_row_mean, data_row_stddev = CALCULATE_mean_stddev(org_texture_data[i]) for j in range(org_texture_data.shape[1]): std_texture_data[i][j] = (org_texture_data[i][j] - data_row_mean)/float(data_row_stddev) # END # END # END std_texture_data = np.nan_to_num(std_texture_data) # NUMPY // CONVERT NOT-A-NUMBER (NAN) DATA POINTS TO ZERO # --------------------------------------------------------# ---------------------------------------------------------------------------------------

# PERFORM PRINCIPAL COMPONENT ANALYSIS CALCULATIONS ------------------------------------# --------------------------------------------------------------------------------------projection, eVals, eVecs, eVecs_PC = PERFORM_PCA(std_texture_data, eVals_thresh) # PERFORM_pca // PERFORM PRINCIPAL COMPONENT ANALYSIS eVals = eVals[np.newaxis] eVals = np.transpose(eVals) # NUMPY // TRANSPOSE MATRIX eVals_sum = np.sum(eVals) # NUMPY // CALCULATE SUM OF EIGENVALUES temp = 0 proportion = np.array([]) # NUMPY // INITIALIZE ARRAY for j in range(len(eVals)): temp = (eVals[j])/(eVals_sum) proportion = np.append(proportion, temp) # NUMPY // CALCULATE PROPORTION VALUES # END cumulative = np.cumsum(proportion) # NUMPY // CALCULATE CUMULATIVE SUM VALUES # ---------------------------------------------------------------------------------------

# GENERATE OUTPUT DATA AND CONSTRUCT PLOTS ---------------------------------------------# --------------------------------------------------------------------------------------eVecs_df = pd.DataFrame(eVecs) eVecs_df.to_csv('eigenvectors.csv') eVals_df = pd.DataFrame(eVals) eVals_df.to_csv('eigenvalues.csv') eVecs_PC_df = pd.DataFrame(eVecs_PC) eVecs_PC_df.to_csv('principal_component_eigenvectors.csv') data_org_mean_df = pd.DataFrame(intensity) data_org_mean_df.to_csv('organized_intensity_train_data.csv') data_org_texture_df = pd.DataFrame(org_texture_data) data_org_texture_df.to_csv('organized_texture_train_data.csv')

53 data_mean_texture_df = pd.DataFrame(mean_texture_data) data_mean_texture_df.to_csv('mean_texture_train_data.csv') data_mean_intensity_df = pd.DataFrame(mean_intensity_data) data_mean_intensity_df.to_csv('mean_intensity_train_data.csv') # ---------------------------------------------------------------------------------------

end = time.clock()

print "// COMPUTATION TIME:", (end - start), "seconds"

54 A2.

BETA (beta.py)

# ----// Beta //------------------------------------------------------------------------# --------------------------------------------------------------------------------------import cv2, glob, math, matplotlib.pyplot as plt, matplotlib.font_manager, numpy as np, pandas as pd import sys, seaborn as sns, time # --------------------------------------------------------------------------------------from pandas import DataFrame from sklearn import svm # --------------------------------------------------------------------------------------from CALCULATE_mean_stddev import * from CALCULATE_prop import * from APPLY_grid import * # --------------------------------------------------------------------------------------Q_distance = [10] # VARIABLE // GLCM Calculation, Inter-Pixel Distance Q_angle = [np.pi/4] # VARIABLE // GLCM Calculation, Offset Angle levels = 256 # VARIABLE // Intensity Levels for GLCM Calculation cell_dim = 400 Cells/Blocks

# VARIABLE // Vertical/Horizontal Pixel Dimension of Cropped Test Image

# --------------------------------------------------------------------------------------OCSVM_kernel = 'rbf'# KERNEL // Kernel Type to be Used In Algorithm # // TYPES: 'linear', 'poly', 'rbf', 'sigmoid' OCSVM_nu = 0.01

# VARIABLE // An Upper Bound on the Fraction of Training Errors and # // a Lower Bound on the Fraction of Support Vectors in the # // Interval (0, 1]

OCSVM_degree = 2 # VARIABLE // Degree of polynomial kernel function 'poly' OCSVM_gamma = 0.01 # VARIABLE // Kernel Coefficient for 'rbf', 'poly', 'sigmoid' # --------------------------------------------------------------------------------------standardize = 1 # SWITCH // Standardize Data # [ROW > NUMBER OF FEATURES, COLUMN > NUMBER OF SAMPLES] # --------------------------------------------------------------------------------------d = 10 # VARIABLE // Diameter of Each Pixel Neighbor that is Used During Filtering sigmaColor = 150 # VARIABLE // Filter Sigma in Color Space, Large Value of the Parameter # // Means Farther Colors within the Pixel Neighborhood will be # // Mixed Together Resulting in Larger Areas of Semi-Equal Color sigmaSpace = 150 # VARIABLE // Filter Sigma in the Coordinate Space, Large Value of the Parameter # // Means Farther Pixels will Influence Each Other as long as their # // Colors are Close Enough # --------------------------------------------------------------------------------------thresh = 75 # VARIABLE // Threshold Value for Binary Thresholding maxValue = 125 # VARIABLE // Maximum Value for Binary Thresholding # --------------------------------------------------------------------------------------# NOTE: This code reads output from STEP_ABC.py, and performs calculations to classify blocks # in the image from Set B as match or mismatch. To accomplish this task, the code implements # a One-Class Support Vector Machine (or One-Class SVM). Based on the result, the code # removes information from the block of the image that the classifier predicts as a # mismatch. Finally, the code detects and predicts a quantitative value for the damage. # ---------------------------------------------------------------------------------------

start = time.clock()

# READ OUTPUT DATA FROM BUILD MODEL FUNCTION, AND SELECT NUMBER OF EIGENVECTORS --------# --------------------------------------------------------------------------------------eVecs = pd.read_csv('eigenvectors.csv') eVecs = np.array(eVecs) eVecs = eVecs[:,1:(eVecs.shape[1])] eVals = pd.read_csv('eigenvalues.csv') eVals = np.array(eVals) eVals = eVals[:,1:(eVals.shape[1])] eVecs_PC = pd.read_csv('principal_component_eigenvectors.csv') eVecs_PC = np.array(eVecs_PC) eVecs_PC = eVecs_PC[:,1:(eVecs_PC.shape[1])] org_int_train_data = pd.read_csv('organized_intensity_train_data.csv') org_int_train_data = np.array(org_int_train_data) org_int_train_data = org_int_train_data[:,1:(org_int_train_data.shape[1])] org_tex_train_data = pd.read_csv('organized_texture_train_data.csv') org_tex_train_data = np.array(org_tex_train_data) org_tex_train_data = org_tex_train_data[:,1:(org_tex_train_data.shape[1])]

55

mean_tex_train_data = pd.read_csv('mean_texture_train_data.csv') mean_tex_train_data = np.array(mean_tex_train_data) mean_tex_train_data = mean_tex_train_data[:,1:(mean_tex_train_data.shape[1])] mean_int_train_data = pd.read_csv('mean_intensity_train_data.csv') mean_int_train_data = np.array(mean_int_train_data) mean_int_train_data = mean_int_train_data[:,1:(mean_int_train_data.shape[1])] # ---------------------------------------------------------------------------------------

# CALCULATE TRAIN WEIGHT VECTOR (OR TRAIN "PATTERN" VECTOR) AND STANDARDIZE ------------# --------------------------------------------------------------------------------------omega_train = (np.transpose(eVecs_PC)).dot(org_tex_train_data - mean_tex_train_data) omega_train_mean = np.zeros([omega_train.shape[0], 1]) omega_train_stddev = np.zeros([omega_train.shape[0], 1]) # --------------------------------------------------------std_omega_train = np.zeros(omega_train.shape) if standardize == 1: for i in range(omega_train.shape[0]): omega_train_mean[i], omega_train_stddev[i] = CALCULATE_mean_stddev(omega_train[i]) for j in range(omega_train.shape[1]): std_omega_train[i][j] = (omega_train[i][j] omega_train_mean[i][0])/float(omega_train_stddev[i][0]) # END # END # END std_omega_train = np.nan_to_num(std_omega_train) std_omega_train = np.transpose(std_omega_train) # NUMPY // CONVERT NOT-A-NUMBER (NAN) DATA POINTS TO ZERO AND TRANSPOSE # --------------------------------------------------------# ---------------------------------------------------------------------------------------

# TRAIN WITH ONE-CLASS SVM CLASSIFIER --------------------------------------------------# --------------------------------------------------------------------------------------tex_classifier = svm.OneClassSVM(kernel = OCSVM_kernel, nu = OCSVM_nu, degree = OCSVM_degree, gamma = OCSVM_gamma) tex_classifier.fit(std_omega_train) # SKLEARN // DEFINE A TYPE OF ONE-CLASS SUPPORT VECTOR MACHINE FOR TEXTURE CLASSIFICATION tex_train_predict = tex_classifier.predict(std_omega_train) tex_train_error = tex_train_predict[tex_train_predict == -1].size # SKLEARN // PERFORM REGRESSION ON TRAIN SAMPLES AND FIND ERRORS # -int_train_mean, int_train_stddev = CALCULATE_mean_stddev(org_int_train_data) std_int_train_data = np.divide((org_int_train_data - int_train_mean), float(int_train_stddev)) # CALCULATE_mean_stddev // STANDARDIZE ... int_classifier = svm.OneClassSVM(kernel = OCSVM_kernel, nu = OCSVM_nu, degree = OCSVM_degree, gamma = OCSVM_gamma) int_classifier.fit(std_int_train_data) # SKLEARN // DEFINE A TYPE OF ONE-CLASS SUPPORT VECTOR MACHINE FOR TEXTURE CLASSIFICATION int_train_predict = int_classifier.predict(std_int_train_data) int_train_error = int_train_predict[int_train_predict == -1].size # SKLEARN // PERFORM REGRESSION ON TRAIN SAMPLES AND FIND ERRORS # ---------------------------------------------------------------------------------------

# CLASSIFY IMAGE CELLS/BLOCKS AS MATCH OR MISMATCH -------------------------------------# --------------------------------------------------------------------------------------test_files = glob.glob('files/test_image/*') # GLOB // LOCATE TEST IMAGE FILES for i in range(len(test_files)): test_im = cv2.imread(test_files[i]).astype(np.uint8) # OPENCV // READ IMAGE AS UNASSIGNED INTEGER (WITHIN 0 TO 255 PIXEL VALUES)

56

cell_count, grid_coordinates = APPLY_grid(test_im.shape[0], test_im.shape[1], cell_dim, cell_dim) # APPLY_grid // CALCULATE NUMBER OF CELLS AND GRID COORDINATES glcm_matrix = np.array([]) mn = np.array([]) con = np.array([]) dis = np.array([]) hom = np.array([]) asm = np.array([]) ene = np.array([]) cor = np.array([]) ent = np.array([]) # NUMPY // INITIALIZE EMPTY NUMPY ARRAYS elimination = np.ones([cell_count, 1]) # NUMPY // INITIALIZE ARRAY WITH ONES for i in w1 = h1 = w2 = h2 =

range(0,cell_count): grid_coordinates[i][0] grid_coordinates[i][1] grid_coordinates[i][2] grid_coordinates[i][3]

test_im_cropped = test_im[h1:h2,w1:w2] mn, con, dis, hom, ene, cor, ent = CALCULATE_prop(test_im_cropped, Q_distance, Q_angle, levels) # CALCULATE_prop // EXTRACT PROPERTY VALUES FROM ONE IMAGE im_tex_test_data = np.vstack((con, dis, hom, ene, cor, ent)) # NUMPY // STACK TEXTURE VALUES IN SEQUENCE VERTICALLY [ROW > NUMBER OF FEATURES, COLUMN > 1] omega_test = ((np.transpose(eVecs_PC)).dot(im_tex_test_data - mean_tex_train_data)) omega_test = np.reshape(omega_test, (len(omega_test),1)) std_omega_test = np.divide((omega_test - omega_train_mean), omega_train_stddev) std_omega_test = np.transpose(std_omega_test) # NUMPY // CALCULATE TEST WEIGHT VECTOR, RESHAPE THE VECTOR AND STANDARDIZE THE VECTOR std_mn = (mn - int_train_mean)/float(int_train_stddev) # CALCULATE_mean_stddev // STANDARDIZE ... # --------------------------------------------------------tex_predict = tex_classifier.predict(std_omega_test) int_predict = int_classifier.predict(std_mn) if tex_predict == -1 or int_predict == -1: elimination[i][0] = 0 # END # END # END # ---------------------------------------------------------------------------------------

# DETECT AND QUANTIFY DAMAGE -----------------------------------------------------------# --------------------------------------------------------------------------------------damage_im = cv2.cvtColor(test_im, cv2.COLOR_BGR2GRAY) # OPENCV // CONVERT IMAGE TO GRAYSCALE damage_im = cv2.bilateralFilter(damage_im, d, sigmaColor, sigmaSpace) # OPENCV // APPLY BILATERAL FILETERING thresh, damage_im = cv2.threshold(damage_im, thresh, maxValue, cv2.THRESH_BINARY) # OPENCV // APPLY BINARY THRESHOLDING for i in range(0,cell_count): if elimination[i][0] == 0: w1 = grid_coordinates[i][0] h1 = grid_coordinates[i][1] w2 = grid_coordinates[i][2] h2 = grid_coordinates[i][3] damage_im[h1:h2,w1:w2] = maxValue # END # END

57

cv2.imwrite('results/damage_image.jpg', damage_im) # OPENCV // GENERATE IMAGE damage_quantity = ((damage_im.shape[0])*(damage_im.shape[1]) - cv2.countNonZero(damage_im)) damage_percent = (damage_quantity/float((damage_im.shape[0])*(damage_im.shape[1])))*100 # OPENCV // CALUCLATE DAMAGE PERCENTAGE # ---------------------------------------------------------------------------------------

end = time.clock()

# PRINT RESULTS ------------------------------------------------------------------------# --------------------------------------------------------------------------------------print "----------------------------------------------------------------------------------" print "COMPUTATION TIME: ", (end - start), "seconds." print "----------------------------------------------------------------------------------" print "RESULTS:" print elimination[elimination == 0].size, "out of", cell_count, "blocks were classified as mismatch." print "Image has approximately %0.2f" %damage_percent,"% damage." print "----------------------------------------------------------------------------------" # ---------------------------------------------------------------------------------------

58 A3. # # # # # #

APPLY GRID (APPLY_grid.py)

----// APPLY GRID //--------------------------------------------------------------------------------------------------------------------------------------------------------NOTE: This function receives image height, image width, preferred grid cell height and grid cell width, and calculates and returns grid cell (block) count and grid cell (block) coordinates. ---------------------------------------------------------------------------------------

# --------------------------------------------------------------------------------------def APPLY_grid(im_height, im_width, grid_height, grid_width): ax = 0 ay = 0 bx = 0 by = 0 cx = 0 cy = 0 dx = 0 dy = 0 hor_cell_count = 0 # INITIALIZE GRID COUNT IN THE HORIZONTAL DIRECTION ver_cell_count = 0 # INITIALIZE GRID COUNT IN THE VERTICAL DIRECTION cell_count = 0 # INITIALIZE TOTAL GRID COUNT grid_coordinates = [] # INITIALIZE GRID COORDINATES while ver_cell_count 0: ax = hor_cell_count * grid_width bx = im_width cx = ax dx = bx # CALCULATION OF ay, by, cy, dy ARE NOT NEEDED, VALUES ARE BORROWED grid_coordinates.append([ax, ay, bx, cy]) # CALCULATION OF by, cx, dx, AND dy IS NOT NECESSARY cell_count += 1 # INCREMENT TOTAL GRID COUNT # END hor_cell_count = 0 # RESET HORIZONTAL GRID COUNT ver_cell_count += 1 # INCREMENT VERTICAL GRID COUNT grid_coordinates = tuple(grid_coordinates)

59 return (cell_count, grid_coordinates) # ---------------------------------------------------------------------------------------

60 A4.

CALCULATE MEAN & STANDARD DEVIATION (CALCULATE_mean_stddev.py)

# ----// CALCULATE MEAN & STANDARD DEVIATION OF DATA // --------------------------------# --------------------------------------------------------------------------------------import cv2 # ---------------------------------------------------------------------------------------

# CALCULATE MEAN AND STANDARD DEVIATION OF DATA ----------------------------------------# --------------------------------------------------------------------------------------def CALCULATE_mean_stddev(data): value = cv2.meanStdDev(data) return value[0][0][0], value[1][0][0] # ---------------------------------------------------------------------------------------

61 A5.

CALCULATE IMAGE PROPERTIES (CALCULATE_prop.py)

# ----// CALCULATE IMAGE PROPERTIES // -------------------------------------------------# --------------------------------------------------------------------------------------import cv2, math, numpy as np # --------------------------------------------------------from skimage.feature import greycomatrix # --------------------------------------------------------------------------------------clip_limit = 2.0 # VARIABLE // Clip Limit for CLAHE title_grid_size = (8,8) # VARIABLE // Grid Size for CLAHE # --------------------------------------------------------------------------------------# NOTE: This function receives image data and calculates properties such as mean, contrast # dissimilarity, homogeneity, angular second moment, energy, correlation # ---------------------------------------------------------------------------------------

# CALCULATE PROPERTIES (MEAN, CONTRAST, DISSIMILARITY, HOMOGENEITY, ENERGY, CORRELATION, ENTROPY) # --------------------------------------------------------------------------------------def CALCULATE_prop(im, Q_distance, Q_angle, levels): im = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY) # OPENCV // CONVERT IMAGE TO GRAYSCALE clahe = cv2.createCLAHE(clipLimit = clip_limit, tileGridSize = title_grid_size) im = clahe.apply(im) # OPENCV // PERFORM CONTRAST LIMITED ADAPTIVE HISTOGRAM EQUILIZATION mn = cv2.mean(im) mn = mn[0] # OPENCV // CALCULATE MEAN (INTENSITY) OF IMAGE AND USE SLICING TO EXTRACT MEAN VALUES glcm = greycomatrix(im, Q_distance, Q_angle, levels, symmetric = True, normed = True) # SKIMAGE // CALCULATE GREY LEVEL CO-OCCURANCE MATRIX # --------------------------------------------------------# 'im' represents the image [array] from which GLCM has to be calculated # 'Q_distance' represents the distance offset [array] between the reference and neighbor pixels # 'Q_angle' represents the angle (in radians) [array] between the reference and neighbor pixles # 'levels' represents the number of possible intensity levels with highest value of 256 # 'symmetric' # 'normed' # --------------------------------------------------------# --------------------------------------------------------glcm_matrix = np.array(glcm[:,:,0,0]) # NUMPY // EXTRACT i AND j COMPONENTS TO MAKE A 2D ARRAY FROM A 4D ARRAY OUPUT FROM greycomatrix con = 0 dis = 0 hom = 0 ene = 0 ent = 0 # VARIABLE RESET glcm_matrix = np.nan_to_num(glcm_matrix) for i in range(levels): for j in range(levels): temp = (i - j) con = con + ((glcm_matrix[i][j])*(temp**2)) # CALCULATE CONTRAST PROPERTY dis = dis + ((glcm_matrix[i][j])*abs(temp)) # CALCULATE DISSIMILARITY PROPERTY hom = hom + ((glcm_matrix[i][j])/(((temp)**2) + 1)) # CALCULATE HOMOGENEITY PROPERTY ene = ene + ((glcm_matrix[i][j])**2) # CALCULATE ENERGY PROPERTY if glcm_matrix[i][j] == 0: ent = 0 elif glcm_matrix[i][j] != 0: ent = ent + (glcm_matrix[i][j])*(np.log2(glcm_matrix[i][j])) # END # END # END

62

ent = -ent # CALCULATE ENTROPY PROPERTY # --------------------------------------------------------# --------------------------------------------------------cor = 0 mu_i = 0 mu_j = 0 var_i = 0 var_j = 0 std_i = 0 std_j = 0 for i in range(levels): for j in range(levels): mu_i = mu_i + (i)*(glcm_matrix[i][j]) # CALCULATE MEAN BASED ON REFERENCE PIXELS i mu_j = mu_j + (j)*(glcm_matrix[i][j]) # CALCULATE MEAN BASED ON NEIGHBOR PIXELS j # END # END for i in range(levels): for j in range(levels): var_i = var_i + (glcm_matrix[i][j])*(((i) - mu_i)**2) # CALCULATE VARIANCE BASED ON REFERENCE PIXELS i var_j = var_j + (glcm_matrix[i][j])*(((j) - mu_j)**2) # CALCULATE VARIANCE BASED ON NEIGHBOR PIXELS j # END # END std_i = math.sqrt(var_i) # MATH // CALCULATE STANDARD DEVIATION BASED ON REFERENCE PIXELS i std_j = math.sqrt(var_j) # MATH // CALCULATE STANDARD DEVIATION BASED ON NEIGHBOR PIXELS j for i in range(levels): for j in range(levels): cor = cor + (glcm_matrix[i][j])*((((i) - mu_i)*(((j) - mu_j)))/math.sqrt((var_i*var_j))) # MATH // CALCULATE CORRELATION PROPERTY # END # END # --------------------------------------------------------return mn, con, dis, hom, ene, cor, ent # ---------------------------------------------------------------------------------------

63 A6.

GAMMA (gamma.py)

# ----// GAMMA // ----------------------------------------------------------------------# --------------------------------------------------------------------------------------import glob, numpy as np, sys, matplotlib.pyplot as plt, pandas as pd, PIL, seaborn as sns from numpy import linalg as la from pandas import DataFrame from plyfile import PlyData, PlyElement from sklearn.neighbors import KDTree, NearestNeighbors # --------------------------------------------------------------------------------------n_neighbors = 10 # // NUMBER OF NEIGHBORS FOR NEAREST-NEIGHBORS ALGORITHM curv_thresh = 2 CURVATURE VALUES lumi_thresh = 245

# // CURVATURE THRESHOLD FOR DIVISION BETWEEN POSITIVE, NEGATIVE AND FLAT SURFACE

PF_thresh = 0.003 NF_thresh = 0.003 FA_thresh = 0.995

# // POSITIVE-TO-FLAT RATIO THRESHOLD # // NEGATIVE-TO-FLAT RATIO THRESHOLD # // FLAT-TO-ALL RATIO THRESHOLD

# // LUMINANCE THRESHOLD FOR SELECTION OF DARK PIXELS

PHIp_thresh = 300 # // THRESHOLD FOR NUMBER OF POSITIVE CURVATURE POINTS FOR SELECTED CURVATURE PHIn_thresh = 300 # // THRESHOLD FOR NUMBER OF NEGATIVE CURVATURE POINTS FOR SELECTED CURVATURE PHIf_thresh = 99550 # // THRESHOLD FOR NUMBER OF POINTS REPRESENTING NEARLY FLAT SURFACES FOR SELECTED CURVATURE # ---------------------------------------------------------------------------------------

# READ .PLY FILE AND EXTRACT DATA ------------------------------------------------------# --------------------------------------------------------------------------------------PTCL_data = PlyData.read('point_cloud_model.ply') # PLYFILE // READ POINT CLOUD DATA MESH_data = PlyData.read('mesh_model.ply') # PLYFILE // READ MESH DATA x_coordinate y_coordinate z_coordinate # PLYFILE // R_channel G_channel B_channel # PLYFILE

= MESH_data.elements[0].data['x'] = MESH_data.elements[0].data['y'] = MESH_data.elements[0].data['z'] EXTRACT COORDINATES

= PTCL_data.elements[0].data['red'] = PTCL_data.elements[0].data['green'] = PTCL_data.elements[0].data['blue'] // EXTRACT R, G, AND B VALUES

# ---------------------------------------------------------------------------------------

# CURVATURE/LUMINANCE ANALYSIS ---------------------------------------------------------# --------------------------------------------------------------------------------------data = np.vstack((x_coordinate, y_coordinate, z_coordinate)).T # NUMPY // ARRANGE DATA AS NEEDED neighbors = NearestNeighbors((n_neighbors + 1), algorithm = 'kd_tree').fit(data) # SKLEARN // FIT THE NEAREST NEIGHBORS MODEL USING 'data' AS TRAINING DATA distances, indices = neighbors.kneighbors(data) # SKLEARN // EXTRACT DISTANCES AND INDICES OF NEIGHBORS WITHIN INDICATED THRESHOLD IN 'n_neighbors' distances = distances[:,1:] # NUMPY // REMOVE SELF, AND KEEP DISTANCES TO NEIGHBORS indices = indices[:,1:] # NUMPY // REMOVE SELF, AND KEEP INDICES TO NEIGHBORS p = np.matlib.repmat(data, n_neighbors, 1) - data[np.ravel(indices), 0:] p = np.reshape(p, (len(data), n_neighbors, 3)) # NUMPY // REPEAT ARRAY OR MATRIX, AND RESHAPE C = np.zeros((len(data), 6)) # NUMPY // INITIALIZE COVARIANCE MATRIX C[:,0] = np.sum(p[:,:,0]*p[:,:,0], axis = 1) C[:,1] = np.sum(p[:,:,0]*p[:,:,1], axis = 1) C[:,2] = np.sum(p[:,:,0]*p[:,:,2], axis = 1)

64 C[:,3] = np.sum(p[:,:,1]*p[:,:,1], axis = 1) C[:,4] = np.sum(p[:,:,1]*p[:,:,2], axis = 1) C[:,5] = np.sum(p[:,:,2]*p[:,:,2], axis = 1) # NUMPY // CALCULATE ELEMENTS OF COVARIANCE MATRIX curvature = np.zeros(len(data)) luminance = np.zeros(len(data)) # NUMPY // INITIALIZE CURVATURE AND LUMINANCE neg_curv = [] pos_curv = [] flt_curv = [] neg_count pos_count flt_count drk_count

= = = =

0 0 0 0

for i in range(len(data)): C_MAT = np.array([[C[i,0], C[i,1], C[i,2]], [C[i,1], C[i,3], C[i,4]], [C[i,2], C[i,4], C[i,5]]]) # NUMPY // POSITION COVARIANCE MATRIX ELEMENTS INTO AN ARRAY eVals, eVecs = la.eig(C_MAT) # NUMPY // FIND EIGENVALUES AND EIGENVECTORS OF COVARIANCE MATRIX diag = np.diag(eVecs) # NUMPY // EXTRACT A DIAGONAL ARRAY lmda = np.min(diag) # NUMPY // RETURN MINIMUM # COUNT POINT CLOUDS THAT HAVE CURVATURE WITHIN A DESIRED RANGE # --------------------------------------------------------------------curvature[i] = lmda/float(np.sum(diag)) # NUMPY // CALCULATE CURVATURE VALUE if curvature[i] < -curv_thresh: neg_curv = np.append(neg_curv, curvature[i]) neg_count += 1 # END elif curvature[i] > curv_thresh: pos_curv = np.append(pos_curv, curvature[i]) pos_count += 1 # END else: flt_curv = np.append(flt_curv, curvature[i]) flt_count += 1 # END # --------------------------------------------------------------------# COUNT POINT CLOUDS THAT HAVE LUMINANCE ABOVE A DESIRED THRESHOLD # --------------------------------------------------------------------luminance[i] = 0.2126*float(R_channel[i]) + 0.7152*float(G_channel[i]) + 0.0722*float(B_channel[i]) # MATH // CALCULATE LUMINANCE VALUES if luminance[i] > lumi_thresh: drk_count += 1 # END # --------------------------------------------------------------------# END # ---------------------------------------------------------------------------------------

# # # # #

WRITE DATA -----------------------------------------------------------------------------------------------------------------------------------------------------------------neg_curv_df = pd.DataFrame(neg_curv) neg_curv_df.to_csv('negative_curvature.csv') PANDAS //

# pos_curv_df = pd.DataFrame(pos_curv)

65 # pos_curv_df.to_csv('positive_curvature.csv') # PANDAS // # flt_curv_df = pd.DataFrame(flt_curv) # flt_curv_df.to_csv('flat_curvature.csv') # PANDAS // # # # #

lumi_df = pd.DataFrame(luminance) lumi_df.to_csv('luminance245.csv') PANDAS // ---------------------------------------------------------------------------------------

# CLASSIFY SURFACE ---------------------------------------------------------------------# --------------------------------------------------------------------------------------CRK_point = 0 DEP_point = 0 # TEST POSITIVE-TO-FLAT RATIO # ----------------------------------------------if (pos_count/float(flt_count)) < PF_thresh: CRK_point += 1 else: DEP_point += 1 # ----------------------------------------------# TEST NEGATIVE-TO-FLAT RATIO # ----------------------------------------------if (neg_count/float(flt_count)) < NF_thresh: CRK_point += 1 else: DEP_point += 1 # ----------------------------------------------# TEST FLAT-TO-ALL RATIO # ----------------------------------------------if (flt_count/float(len(data))) > FA_thresh: CRK_point += 1 else: DEP_point += 1 # ----------------------------------------------# TEST NUMBER OF POSITIVE CURVATURE POINTS # ----------------------------------------------if pos_count < PHIp_thresh: CRK_point += 1 else: DEP_point += 1 # ----------------------------------------------# TEST NUMBER OF NEGATIVE CURVATURE POINTS # ----------------------------------------------if neg_count < PHIn_thresh: CRK_point += 1 else: DEP_point += 1 # ----------------------------------------------# TEST NUMBER OF FLAT REGION POINTS # ----------------------------------------------if flt_count > PHIf_thresh: CRK_point += 1 else: DEP_point += 1 # ----------------------------------------------# ---------------------------------------------------------------------------------------

# PRINT RESULTS ------------------------------------------------------------------------# --------------------------------------------------------------------------------------print "--CURVATURE COUNTS-----------------------------" print "Number of Positive Curvature Regions: ", pos_count print "Number of Negative Curvature Regions: ", neg_count print "NUmber of Flat Regions: ", flt_count print "-----------------------------------------------" print "--DARK LUMINANCE COUNTS------------------------" print "Number of Dark Luminance Regions above 245: ", drk_count

66 print "-----------------------------------------------" print print print print print

"--CURVATURE RATIOS-----------------------------" "PF Ratio: ", pos_count/float(flt_count) "NF Ratio: ", neg_count/float(flt_count) "FA Ratio: ", flt_count/float(len(data)) "-----------------------------------------------"

print "--CLASSIFICATION RESULTS-----------------------" print "Points for Presence of Surface Crack : ", CRK_point print "Points for Presence of Surface Depression: ", DEP_point print "-----------------------------------------------" # ---------------------------------------------------------------------------------------

67 A7.

PERFORM PRINCIPAL COMPONENT ANALYSIS (PERFORM_PCA.py)

# ----// PERFORM PRINCIPAL COMPONENT ANALYSIS // ---------------------------------------# --------------------------------------------------------------------------------------import numpy as np # --------------------------------------------------------------------------------------from numpy import linalg as la # --------------------------------------------------------------------------------------# NOTE: This function receives standardized texture data (with number of features in rows # and number of samples in columns), and calculates eigenvalues, eigenvectors and new # projected data in the direction of the principal components. # ---------------------------------------------------------------------------------------

# CALCULATE EIGENVALUES, EIGENVECTORS, AND NEW PROJECTED DATA --------------------------# --------------------------------------------------------------------------------------def PERFORM_PCA(data, eigenVals_thresh): data_mean = np.mean(data, axis = 1)[np.newaxis] data_mean = np.transpose(data_mean) data -= data_mean # NUMPY // CENTER THE ENTIRE DATASET BY SUBTRACTING THE MEAN FROM EACH COLUMN covariance_mat = np.cov(data, rowvar = True) covariance_mat = np.nan_to_num(covariance_mat) # NUMPY // CALCULATE COVARIANCE MATRIX FROM DATA WHERE EACH ROW REPRESENTS A FEATURE & EACH # COLUMN REPRESENTS A SAMPLE # NUMPY // CONVERT NOT-A-NUMBER (NAN) DATA POINTS TO ZERO eigenVals, eigenVecs = la.eig(np.mat(covariance_mat)) # NUMPY // CALCULATE EIGENVALUES AND EIGENVECTORS FROM COVARIANCE MATRIX # (HERE, DATA IS REPRESENTED AS MATRIX INSTEAD OF ARRAY) indices = np.argsort(eigenVals) indices = indices[::-1] eigenVals = eigenVals[indices] eigenVecs = eigenVecs[:,indices] # NUMPY // SORT EIGENVALUES AND EIGENVECTORS IN DESCENDING ORDER projection = np.dot(np.transpose(eigenVecs), data) # NUMPY // PROJECT DATA INTO NEW SPACE k = 0 for i in range(len(eigenVals)): if (eigenVals[i] > eigenVals_thresh): k = k + 1 # FIND THE NUMBER OF EIGENVECTORS TO CONSIDER BASED ON THRESHOLD # END # END eigenVecs_PC = eigenVecs[:,0:k] return projection, eigenVals, eigenVecs, eigenVecs_PC # ---------------------------------------------------------------------------------------

68

APPENDIX B. ADDITIONAL DATA & TABLES

Table 11: Statistical Properties of Positive Curvature Values in Seven Validation Sets of Surface Crack Data Positive Curvature Statistical Data for Surface Crack n

Mean

Median

Std. Dev.

Kurtosis

Skewness

Range

Min

Max

Set 1

270

9.645

3.087

50.569

195.676

13.412

769.368

2.090

771.458

Set 2

143

16.570

3.822

44.787

41.820

6.008

385.176

2.001

387.177

Set 3

78

30.277

5.457

142.736

62.591

7.712

1206.102

2.028

1208.130

Set 4

89

9.948

5.014

13.206

8.624

2.842

71.763

2.032

73.795

Set 5

113

3.244

2.394

3.341

33.800

5.439

26.379

2.002

28.382

Set 6

119

26.317

3.945

153.788

107.005

10.158

1644.824

2.013

1646.837

Set 7

139

10.584

4.374

22.019

40.752

5.803

193.522

2.002

195.524

Table 12: Statistical Properties of Negative Curvature Values in Seven Validation Sets of Surface Crack Data Negative Curvature Statistical Data for Surface Crack n

Mean

Median

Std. Dev.

Kurtosis

Skewness

Range

Min

Max

Set 1

188

-10.531

-4.165

30.366

83.409

-8.451

340.682

-342.699

-2.017

Set 2

135

-25.339

-4.009

182.089

131.563

-11.405

2110.101

-2112.140

-2.039

Set 3

112

-7.574

-3.637

14.013

33.614

-5.481

98.879

-100.880

-2.001

Set 4

146

-33.280

-3.151

299.054

143.530

-11.936

3605.007

-3607.030

-2.023

Set 5

9

-11.893

-8.491

9.464

1.359

-1.392

28.353

-31.857

-3.504

Set 6

111

-8.770

-3.423

15.466

22.280

-4.299

111.507

-113.518

-2.011

Set 7

296

-6.395

-3.036

13.686

112.304

-9.406

186.687

-188.689

-2.002

69 Table 13: Statistical Properties of Flat Surface Curvature Values in Seven Validation Sets of Surface Crack Data Flat Surface Curvature Statistical Data for Surface Crack n

Mean

Median

Std. Dev.

Kurtosis

Skewness

Range

Min

Max

Set 1

99573

-0.597

-0.655

0.340

2.847

1.006

3.981

-1.995

1.986

Set 2

99748

-0.615

-0.704

0.356

3.953

1.335

3.996

-1.996

2.000

Set 3

99843

-0.616

-0.673

0.317

1.509

0.721

3.985

-1.987

1.998

Set 4

99806

-0.627

-0.721

0.346

3.024

1.131

3.986

-1.997

1.989

Set 5

99913

-0.514

-0.575

0.433

5.063

1.691

3.955

-1.957

1.999

Set 6

99798

-0.582

-0.637

0.346

4.199

1.135

3.977

-1.981

1.995

Set 7

99609

-0.610

-0.687

0.357

3.154

1.076

3.956

-1.996

1.960

Table 14: Statistical Properties of Positive Curvature Values in Seven Validation Sets of Surface Depression Data Positive Curvature Statistical Data for Surface Depression n

Mean

Median

Std. Dev.

Kurtosis

Skewness

Range

Min

Max

Set 1

125

14.104

3.231

42.881

43.854

6.343

348.105

2.008

350.112

Set 2

6824

23.956

5.255

279.858

2584.559

47.773

16919.058

2.002

16921.060

Set 3

2556

61.740

8.509

1120.098

2265.490

46.474

54973.186

2.010

54975.197

Set 4

441

12.367

3.604

42.745

157.255

11.077

694.995

2.000

696.995

Set 5

319

10.419

3.307

34.949

116.105

9.797

482.400

2.001

484.402

Set 6

2676

33.605

6.329

323.518

777.071

26.759

10666.618

2.008

10668.626

Set 7

10622

38.016

4.214

1064.147

2599.983

50.022

59723.969

2.000

59725.969

Table 15: Statistical Properties of Negative Curvature Values in Seven Validation Sets of Surface Depression Data Negative Curvature Statistical Data for Surface Depression n

Mean

Median

Std. Dev.

Kurtosis

Skewness

Range

Min

Max

Set 1

169

-13.062

-3.937

32.025

28.654

-5.018

260.156

-262.159

-2.003

Set 2

9245

-22.272

-4.060

423.989

4274.853

-61.399

32668.242

-32670.242

-2.000

Set 3

6069

-17.172

-4.816

108.114

872.672

-26.753

4274.703

-4276.703

-2.000

Set 4

584

-12.564

-3.541

60.534

223.255

-13.571

1139.779

-1141.780

-2.001

Set 5

199

-14.306

-3.565

37.996

31.320

-5.313

293.183

-295.190

-2.007

Set 6

2856

-41.462

-5.608

597.547

1295.653

-34.521

24804.161

-24806.162

-2.001

Set 7

8898

-103.021

-4.351

6382.184

8124.471

-88.849

588216.878

-588218.879

-2.001

70 Table 16: Statistical Properties of Flat Surface Curvature Values in Seven Validation Sets of Surface Depression Data Flat Surface Curvature Statistical Data for Surface Depression n

Mean

Median

Std. Dev.

Kurtosis

Skewness

Range

Min

Max

Set 1

99732

-0.592

-0.662

0.352

4.162

1.295

3.965

-1.971

1.993

Set 2

83984

-0.434

-0.446

0.626

1.248

0.590

3.999

-2.000

2.000

Set 3

91434

-0.568

-0.582

0.480

2.429

0.732

3.999

-2.000

1.999

Set 4

99010

-0.605

-0.671

0.372

5.595

1.436

3.997

-1.997

2.000

Set 5

99503

-0.640

-0.754

0.364

4.916

1.512

3.999

-1.999

2.000

Set 6

94522

-0.546

-0.554

0.413

3.446

0.799

3.997

-1.999

1.998

Set 7

80535

-0.315

-0.384

0.729

1.164

0.880

4.000

-2.000

2.000

Table 17: PF, NF and FA Ratios for Surface Crack Validation Datasets PF

NF

FA

Set 1

0.002711578

0.001888062

0.995421419

Set 2

0.001433613

0.001353411

0.997220723

Set 3

0.000781227

0.001121761

0.998100627

Set 4

0.00089173

0.001462838

0.997650963

Set 5

0.001130984

9.00784E-05

0.998780427

Set 6

0.001192409

0.001112247

0.997700644

Set 7

0.001395456

0.002971619

0.995651913

Table 18: PF, NF and FA Ratios for Surface Depression Validation Datasets PF

NF

FA

Set 1

0.001253359

0.001694541

0.997060764

Set 2

0.081253572

0.110080492

0.839395121

Set 3

0.02795459

0.066375746

0.913800857

Set 4

0.004454096

0.005898394

0.989753586

Set 5

0.003205933

0.00199994

0.994821088

Set 6

0.028310869

0.030215188

0.944709857

Set 7

0.131892966

0.110486124

0.804907301

71

APPENDIX C. ILLUSTRATIONS OF ADDITIONAL TEST SETS PROCESSED BY ALGORITHM GAMMA

Figure 27: Damage Test Set 1


72



73



74



75


76

REFERENCES

Adams, M. S., Fromm, R., & Lechner, V. (2016). High-Resolution Debris Flow Volume Mapping with Unmanned Aerial Systems (UAS) and Photogrammetric Techniques. The International Archive of Photogrammetry, Remote Sensing and Spatial Information Sciences, 749-754. Alpaydin, E. (2014). Introduction to Machine Learning (3rd ed.). Cambridge, London: The MIT Press. Amza, C. G., & Cicic, D. T. (2014). Industrial Image Processing using Fuzzy-Logic. 25th DAAAM International Symposium on Intelligent Manufacturing and Automation, 492-498. Anbarjafari, G., Jafari, A., Jahromi, M. N., Ozcinar, C., & Demirel, H. (2015). Image Illumination Enhancement with an Objective No-Reference Measure of Illumination Assessment based on Gaussian Distribution Mapping. Engineering Science and Technology, An International Journal, 696-703. Anonymous. (2013). 2013 Report Card for America's Infrastructure. Amerian Society of Civil Engineers (ASCE). Amerian Society of Civil Engineers (ASCE). Retrieved October 2016, from http://infrastructurereportcard.org/a/documents/2013-ReportCard.pdf Anonymous. (2016, June 20). One-Class Support Vector Machine. Retrieved December 1, 2016, from Microsoft Azure: https://msdn.microsoft.com/enus/library/azure/dn913103.aspx Anonymous. (2017). 2017 Infrastructure Report Card. American Society of Civil Engineers (ASCE). American Society of Civil Engineers (ASCE). Retrieved April 1, 2017, from http://www.infrastructurereportcard.org/ Anonymous. (2017). What Makes a Grade? Retrieved April 5th, 2017, from American Society of Civil Engineers, ASCE: http://www.infrastructurereportcard.org/making-the-grade/what-makes-a-grade/ Beard, R. W., & McLain, T. W. (2012). Small Unmanned Aircraft - Theory and Practice (1st ed.). Princeton: Princeton University Press. Bishop, C. M. (2006). Pattern Recognition and Machine Learning (1st ed.). Springer.

77 Bradski, G. (2000). OpenCV Library. Dr. Dobb's Journal of Software Tools. Burger, W., & Burge, M. J. (2013). Principles of Digital Image Processing - Advanced Methods (Vol. III). Springer. Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., & Schmidhuber, J. (2011). Flexible High Performance Convolutional Neural Networks for Image Classification. 22nd International Joint Conference on Artificial Intelligence, II, 1237-1242. Cong, J., & Xiao, B. (2014). Minimizing Computation in Convolutional Neural Networks. Artificial Neural Networks and Machine Learning, 8681, 281-290. Dhage, S. S., Hegde, S. S., & Ramachandran, M. K. (2015). DWT-based Feature Extraction and Radon Transform based Contrast Enhancement for Improved Iris Recognition. Procedia Computer Science 45 - International Conference on Advanced Computing Technologies and Applications, 256-265. Downey, A. B. (2014). Think Stats - Exploratory Data Anlysis in Python (2nd ed.). Needham: Green Tea Press. Fangueiro, R. (2011). Fibrous and Composite Materials for Civil Engineering Applications. New Delhi: Woodhead Publishing Limited. Foorginejad, A., & Khalili, K. (2014). Umbrella Curvature: A New Curvature Estimation Method for Point Clouds. Procedia Technology, 347-352. Forsyth, D. A., & Ponce, J. (2011). Computer Vision - A Modern Approach (2nd ed.). Fuhrmann, S., & Goelese, M. (2014). Floating Scale Surface Reconstruction. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2014. Fuhrmann, S., Langguth, F., & Goesele, M. (2014). MVE - A Multi-View Reconstruction Environment. GCH 2014 - Eurographics Workshop on Graphics and Cultural Heritage, 11 - 18. Gebejes, A., & Huertas, R. (2013). Texture Characterization based on Grey-Level CoOccurance Matrix. Conference of Informatics and Management Sciences SECTION 12 Artificial Intelligence and Image Processing, 375-378. Goelese, M., Snavely, N., Curless, B., Hoppe, H., & Seitz, S. M. (2007). Collections, Multi-View Stereo for Community Photo. 2007 IEEE 11th International Conference on Computer Vision.

78 Goldman, R. (2005). Curvature Fromulas for Implicit Curves and Surfaces. Computer Aided Geometric Design - Special Issue: Geometric Modelling and Differential Geometry, 632-658. Gonzalez, R. C., & Woods, R. E. (2008). Digital Image Processing (3rd ed.). Upper Saddle River: Pearson Prentice Hall. Gonzalez, R. C., Woods, R. E., & Eddins, S. L. (2009). Digital Image Processing Using Matlab (2nd ed.). Gatesmark Publishing. Hackney, C., & Clayton, A. I. (2015). Unmanned Aerial Vehicles (UAVs) and their Application in Geomorphic Mapping. Geomorphological Techniques, Chapter 2 Section 1.7. Hall-Beyer, M. (2007, February 21). The GLCM Tutorial Homepage, 2.10. Retrieved July 1, 2016, from University of Calgary: http://www.fp.ucalgary.ca/mhallbey/tutorial.htm Ham, Y., Han, K. K., Lin, J. J., & Golparvar-Fard, M. (2016). Visual Monitoring of Civil Infrastructure Systems via Camera-Equipped Unmanned Aerial Vehicles (UAVs): A Review of Related Works. Visualization in Engineering. Haralick, R. M., Shanmugam, K., & Dinstein, I. (1973). Texture Features for Image Classification. IEEE Transactions on Systems, Man, and Cybernetics, SMC - 3(6), 610-621. Harrington, P. (2012). Machine Learning In Action (1st ed.). Shelter Island: Manning Publications Co. Hartley, R., & Zisserman, A. (2000). Multiple View Geometry in Computer Vision. Cambridge University Press. Hensman, P., & Masko, D. (2015). The Impact of Imbalanced Training Data for Convolutional Neural Networks. KTH Royal Institute of Technology. Stockholm: Digitala Vetenskapliga Arkivet. Retrieved November 1, 2016 Johnson, R. A., & Wichern, D. W. (2007). Applied Multivariate Statistical Analysis. Upper Saddle River: Pearson Prentice Hall. Jung, S., Lee, T., Mina, T., & Ariyur, K. B. (2012). Inductive or Magnetic Recharging for Small UAVs. In S. International (Ed.).

79 K., V. K., R., S. P., Manikantan, K., & Ramachandran, S. (2015). Face Recognition using Block based Feature Extraction with CZT and Goertzel Algorithm as a Preprocessing Technique. Procedia Computer Science 46 - International Conference on Information and Communication Technologies 2014, 1458-1467. Kersten, T. P., & Lindstaedt, M. (2012). Automatic 3D Object Reconstruction from Multiple Images for Architectural, Cultural, and Archaeological Applications Using Open-Source Software and Web Services. Photogrammetrie Fernerkundung - Geoinformation, 727-740. Kim, K., Chalidabhongse, T. H., Harwood, D., & Davis, L. (2005). Real-Time Foreground-Background Segmentation using Codebook Model. Journal, RealTime Imaging, 11(3), 172-185. Kurita, T., & Boulanger, P. (1992). Computation of Surface Curvature from Range Image Using Geometrically Intrinsic Weights. Lim, R. S. (2014). A Robotic Crack Inspection and Mapping System for Bridge Deck Maintenance. IEEE Transactions on Automation Science and Engineering, 11(2), 367-378. Liu, C., Freeman, W. T., Szeliski, R., & Kang, S. B. (2006). Noise Estimation from a Single Image. IEEE Conference on Computer Vision and Pattern Recognition. Liu, P., Chen, A. Y., Huang, Y.-N., Han, J.-Y., Lai, J.-S., Kang, S.-C., . . . Tsai, M.-H. (2014). A Review of Rotorcraft Unmanned Aerial Vehicle (UAV) Developments and Applications in Civil Engineering. Smart Structures and Systems, 13(6), 1065-1094. Lucieer, A., Robinson, S., & Turner, D. (2012). Unmanned Aerial Vehicle (UAV) Remote Sensing for Hyperspatial Terrain Mapping of Antarctic Moss Beds based on Structure from Motion (SfM) Point Clouds. 34th International Symposium on Remote Sensing of Environment. Lutz, M. (2013). Learning Python (5th ed.). Sebastopol: O'Reilly Media Inc. M., A. T., J., L., K., M., & Ramachandran, S. (2015). Face Recognition using Gabor Filter based Feature Extraction with Anisotropic Diffusion as a Pre-Processing Technique. Procedia Computer Science 45 - International Conference on Advanced Computing Technologies and Applications, 312-321.

80 Marden, J. I. (2015). Multivariate Statistics - Old School. CreateSpace Independent Publishing Platform. Marsland, S. (2009). Machine Learning - An Algorithmic Perspective (1st ed.). Boca Raton: Chapman & Hall/CRC. Miller, T. I., Spencer Jr., B. F., Li, J., & Jo, H. (2010). Solar Energy Harvesting and Software Enhancements for Autonomous Wireless Smart Sensor Networks. University of Illinois at Urbana-Champaign, Newmark Structural Engineering Laboratory (NSEL). Urbana: Newmark Structural Engineering Laboratory, University of Illinois at Urbana-Champaign. Retrieved October 31, 2016, from http://hdl.handle.net/2142/16300 Nguyen, T., Mann, G. K., & Gosine, R. G. (2014). Vision-Based Qualitative PathFollowing Control of Quadrotor Aerial Vehicle. Unmanned Aircraft Systems (ICUAS), 2014 International Conference on. Nixon, M. S., & Aguado, A. S. (2012). Feature Extraction & Image Processing for Computer Vision (3rd ed.). London: Elseiver Limited. O'Byrne, M., Ghosh, B., Pakrashi, V., & Schoefs, F. (2012). Image Processing based Damage Detection of Offshore Infrastructural Elements using Texture Information. 5th European Conference no Structural Control, 1-10. Padmapriya, B., Kesavamurthi, T., & Ferose, H. W. (2012). Edge Based Image Segmentation Technique for Detection and Estimation of the Bladder Wall Thickness. Procedia Engineering 30 - International Conference on Communication Technology and System Design, 828-835. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., . . . Duchesnay, E. (2011). 2.7 Novelty and Outlier Detection. Retrieved August 1, 2016, from Scikit-Learn: http://scikitlearn.org/stable/modules/outlier_detection.html#outlier-detection Pereira, F. C., & Pereira, C. E. (2015). Embedded Image Processing Systems for Automatic Recognition of Cracks using UAVs. International Federation of Automatic Control, 16-21. Petrou, M., & Petrou, C. (2010). Image Processing: The Fundamentals (2nd ed.). John Wiley & Sons Limited.

81 Point Cloud Library (PCL). (n.d.). Retrieved December 1, 2016, from Estimating Surface Normals in a PointCloud: http://pointclouds.org/documentation/tutorials/normal_estimation.php#normalestimation R., V., Kini, Y. V., Manikantan, K., & Ramachandran, S. (2014). Face Recognition using Hough Transform based Feature Extraction. Procedia Computer Science 46 International Conference on Information and Communication Technologies, 1491-1500. R., V., Kini, Y. V., Manikantan, K., & Ramachandran, S. (2015). Face Recognition using Hough Transform based Feature Extraction. Procedia Computer Science 46 International Conference on Information and Communication Technologies 2014, 1491-1500. Rusu, R. B. (2013). Semantic 3D Object Maps for Everyday Robot Manipulation. Springer Publishing Company, Incorporated. Ryan, J. C., Hubbard, A. L., Box, J. E., Todd, J., Christoffersen, P., Carr, J. R., . . . Snooke, N. (2015). UAV Photogrammetry and Structure from Motin to Assess Calving Dynamics at Store Glacier, A Large Outlet Draining the Greenland Ice Sheet. The Cryosphere. Sankarasrinivasan, S., Balasubramanian, E., Karthik, K., Chandrasekar, U., & Gupta, R. (2015). Health Monitoring of Civil Structures with Integrated UAV and Image Processing System. 11th International Multi-Conference on Information Processing, 508-515. Schölkopf, B., Platt, J. C., Shawe-Taylor, J. C., Smola, A. J., & Williamson, R. C. (2001). Estimating the Support of High-Dimensional Distribution. Neural Computation, 13(7), 1443-1471. Schölkopf, B., Sung, K., Burges, C., Girosi, F., Niyogi, P., Poggio, T., & Vapnik, V. (1997). Comparing Support Vector Machines with Gaussian Kernels to Radial Basis Function Classifiers. IEEE Transactions on Signal Processing, 45(11), 2758-2765.

82 Sharma, G., Laureyns, I. A., & Ariyur, K. B. (2010). The Use of Natural Signals for Localization and Navigation with Application to Centimeter sized UAVs. 2010 American Control Conference, 27-32. Solomon, C., & Breckon, T. (2011). Fundamentals of Digital Image Processing - A Practical Approach with Examples in Matlab. Wiley-Blackwell. Spencer Jr., B. F., & Yun, C.-B. (2010). Wireless Sensor Advances and Applications for Civil Infrastructure Monitoring. University of Illinois at Urbana-Champaign, Newmark Structrual Engineering Laboratory (NSEL). Urbana: Newmark Structural Engineering Laboratory, University of Illinois at Urbana-Champaign. Retrieved October 31, 2016, from http://hdl.handle.net/2142/16434 STAT 200 Elementary Statistics, Lesson 10: One-Way Analysis of Variance (ANOVA). (n.d.). Retrieved June 1, 2016, from Penn State University, Eberly College of Science: https://onlinecourses.science.psu.edu/stat200/node/66 STAT 505 Applied Multivariate Statistical Analysis, Lesson 11.5: Alternative: Standardize the Variables. (n.d.). Retrieved July 1, 2016, from Penn State University, Eberly College of Science: https://onlinecourses.science.psu.edu/stat505/node/55 STAT 505 Applied Multivariate Statistical Analysis, Lesson 11: Principal Component Analysis. (n.d.). Retrieved July 1, 2016, from Penn State University, Eberly College of Science: https://onlinecourses.science.psu.edu/stat505/node/49 Sun, J., Zhang, W., Tang, X., & Shum, H.-Y. (2006). Background Cut. Computer Vision ECCV 2006, 628-641. T. McLeod, C. S. (2013). Using Video Acquired from an Unmanned Aerial Vehicle (UAV) to Measure Fracture Orientation in an Open-Pit Mine. Geomatica, 67(3), 173-180. Taylor, Z. (2014, October 15). MathWorks. (Z. Taylor, Editor) Retrieved February 1, 2017, from Find 3D Normals and Curvature: https://www.mathworks.com/matlabcentral/fileexchange/48111-find-3d-normalsand-curvature

83 Theisel, H., Rossl, C., Zayer, R., & Seidel, H.-P. (2004). Normal Based Estimation of the Curvature Tensor for Triangular Meshes. PG '04 Proceedings of the Computer Graphics and Applications, 12th Pacific Conference, 288-297. Tomasi, C., & Manduchi, R. (1998). Bilateral Filtering for Gray and Color Images. Sixth International Conference on Computer Vision. IEEE. Turk, M., & Pentland, A. (1991). Eigenfaces for Recognition. Journal of Cognitive Neuroscience, 3(1), 71 - 86. Waechter, M., Moehrle, N., & Goesele, M. (2014). Let There Be Color! Large-Scale Texturing of 3D Reconstructions. Computer Vision - ECCV 2014, 836 - 850. Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 13(4), 600-612. Westoby, M. J., Brasington, J., Glasser, N. F., Hambrey, M. J., & Reynolds, J. M. (2012). 'Structure from Motion' Photogrammetry: A Low-Cost, Effective Tool for Geoscience Applications. Geomorphology 179, 300-314. Xu, J., Zhou, M., Wu, Z., Shui, W., & Ali, S. (2015). Robust Surface Segmentation and Edge Feature Lines Extraction from Fractured Fragments of Relics. Journal of Computational Design and Engineering 2, 79-87. Yang, F., & Ariyur, K. B. (2011). Laplacian Path Planning: Implementation and Generalizations. 1-9. Zhang, H. (2011). Building Materials in Civil Engineering. Beijing: Woodhead Publishing Limited. Zhao, Y., Wang, H., & Yan, R. (2012). Unstructured Road Edge Detection and Initial Positioning Approach based on Monocular Vision. AASRI Procedia I - 2012 AASRI Conference on Computational Intelligence and Bioinformatics, 486-491.

algorithm development for real-time infrastructure

algorithm development for real-time infrastructure

Suggest Documents

An Efficient Bandit Algorithm for Realtime Multivariate Optimization

a camera auto-calibration algorithm for realtime road ... - LISA - ULB

Infrastructure Development for the Economic Development in ...

Infrastructure Development Strategy for Sustainable

Realtime

Realtime

Tai Pan Realtime - Prognosis Software Development

Development of Triplex SYBR Green RealTime PCR for Detecting ...

ScienceDirect Algorithm development for minor

Flexible Architecture for the Development of Realtime ... - CiteSeerX

INDONESIA ELECTRICITY INFRASTRUCTURE DEVELOPMENT

Social Infrastructure Development

Social Infrastructure Development

Infrastructure for Component-Based DDS Application Development

Infrastructure Development for Strengthening the Capacity of ...

Infrastructure for Development of Intelligent Learning ...

Preliminary Infrastructure Development for Altair Sortie Operations

Development of Six Sigma Infrastructure for ...

Mobile phone infrastructure development: Lessons for ... - CyberLeninka

Infrastructure for Development of Intelligent Learning ... - IFIP

Infrastructure Development for Strengthening the Capacity of ...

Infrastructure Development Investment Program for Tourism (IDIPT ...

Infrastructure development for conversion to ... - Semantic Scholar

Infrastructure Development for Agro-Processing Cooperatives ... - SSRN