Roadway Pavement Anomaly Classification Utilizing

Roadway Pavement Anomaly Classification Utilizing Smartphones And Artificial Intelligence Charalambos Kyriakou Symeon E. Christodoulou Loukas Dimitriou Dept. of Civil and Environmental Engineering University of Cyprus [email protected] [email protected] [email protected]

Abstract - Presented herein is a study on the use of low-cost technology for the data collection and clasification on roadway pavement defects, by use of sensors from smartphones and from automobiles’ on-board diagnostic (OBD-II) devices while vehicles are in movement. The smartphone-based data collection is complimented with artificial intelligence-based (AI) pattern recognition techniques for the classification of detected anomalies. The proposed system architecture and methodology utilize eleven metrics in the analysis, are checked against three types of roadway anomalies, and are validated against hundreds of roadway runs (relating to several thousands of data points) with an accuracy rate of over 90 percent. Keywords- road anomaly detection; potholes; classification; artificial neural networks; smartphones

I.

INTRODUCTION

In recent years, the area of interest for transportation authorities, researches and practitioners has shifted from the construction of new roads to the management of existing ones [1]. One of the most important indicators for road quality is the pavement surface condition, which is defined by the anomalies in the road surface that have an effect on the ride quality of a vehicle. Not only can road anomalies damage vehicles, increase fuel consumption, and cause unpleasant driving, they are also in many cases the reason for traffic accidents, injuries and/or fatalities. Further, the condition of pavement surfaces is not time-invariant, as pavements deteriorate in time from causes related to location, materials used, traffic, weather, etc. Identifying pavement anomalies such as bumps, potholes, patching and cracking on the surface can contribute to surveying road surface quality. Pavement surface condition monitoring systems could improve road surface, care for vehicles from damage due to bad roads and improve traffic safety. As a result of the rapid and powerful development of smartphones in recent years, connected vehicle technology has received noteworthy attention within the transportation, infrastructure and automotive industries. This paper investigates the use of smartphones and connected vehicle applications, in order to improve the condition assessment and management of roadway pavements, and by extension it investigates the possibility that connected vehicle data may contribute to monitoring the condition of transport infrastructure. Modern

smartphones can be used to capture vehicle sensor data without integration with built-in vehicle systems, since they come with a range of built-in sensors such as accelerometer, gyroscope and GPS sensors. Further, some of the vehicle system (Controller Area Network, CAN bus) data can be collected through onboard diagnosis (OBD) Bluetooth connectors (ELM 327 Bluetooth Car Diagnostic Scanner) to a smartphone (e.g. DashCommand application). This combination of hardware and software components facilitates the monitoring of, among others, forward, lateral and vertical acceleration, vehicle roll and pitch, GPS latitude and longitude, GPS vehicle speed, engine RPM and current acceleration based on the last two speed readings. The prospect of roadway anomaly detection by use of smartphone technology is viewed in parallel with the premise that such technology can be used for crowd-sourced data collection and analysis in GIS-based pavement management systems (PMS). Contemporary and future primary datasets will be different in form from traditional pavement condition data, and in order to overcome this barrier, it is prudent that crowdsourced data be collected from a statistically significant number of probe-vehicles [2]. Crowd-sourced data would create a geocoded event at points where the car develops abnormal behaviour. Discrete ranges could be converted into a numerical (1-5, 1-10, etc.) scale or a good/fair/poor rating. Populating a database with crowd-sourced event points from a number of connected vehicles will allow engineers and PMS program managers to identify where vehicles are experiencing rough riding conditions. The work included herein presents ongoing research on road anomaly detection and classification by use of smartphones and artificial neural networks. Further to this short introduction, the paper provides a brief synopsis of existing work related to roadway anomaly detection using PMS and pothole detection using smartphones (Section 2), an outline of the utilized data collection structure and methods (Section 3), and the methods and tools used to classify the data and experiment results (Section 4). The paper concludes with a discussion of the results and an outline of related future research directions (Section 5).

II.

LITERATURE REVIEW

Pavements are roadway infrastructure assets of major significance. As stated by the American Public Works Association (APWA) [3], ''pavement management is a systematic method for routinely collecting, storing, and retrieving the kind of decision-making information needed to make maximum use of limited maintenance (and construction) dollars''. Undoubtedly, the investment in a Pavement Management System (PMS) is worthwhile as it provides the tools an agency needs for the rational resource allocation, optimal use of funds, pavement rehabilitation cost reductions, pavement treatment selection, pavement life extensions, and increased credibility with stakeholders [4]. In developed countries, PMS are specialized platforms equipped with expensive equipment installed on specialized Pavement Evaluation Vehicles [5]. The costs linked to a PMS include software (purchase and installation), data collection, database building and system maintenance, as well as, updates, consultant services, employee training, personnel time and actual expenditures on the pavement and rehabilitation [6]. The first step in developing a pavement management procedure is to define the roadway network [7]. The second step in pavement management design process is to verify the survey methodology for collecting the distress data. At present a PMS uses surveys, such as the Pavement Surface Evaluation and Rating (PASER) rating procedure, generally addressing an estimate or a detail measurement of distress [8]. The PASER rating procedure is based on a series of photographs and descriptions for each of the individual rating categories. A rater uses this series of photographs to evaluate the overall condition of an individual pavement segment [9]. After verifying the survey methodology for collecting the distress data, an agency must opt between the two main approaches of collecting road surface condition data; manual and automated [10]. In manual surveys, pavement surface condition data are collected from a moving vehicle (windshield surveys) or by “walking” the pavement. Automated surveys are performed using vehicles being equipped with specialized camera and other sensing devices that collect images and data relevant to the pavement being under investigation [11]. The third step concerns the prediction of the pavement surface condition. Pavement network conditions can be calculated using either average deterioration rates or prediction models via statistical modelling like regression analysis [9]. The fourth step of PMS design is to select the suitable treatments for the pavement network. The recommended treatments are arranged using cyclical treatment selection, ranking or benefit/cost/analysis. The fifth step refers to the development of reports comprising the results obtain from the previous steps. The sixth step is to select pavement management tool such as a pavement management software, customized spreadsheets and/or GIS software. The final step is keeping the process updated. Notably, pavement management is a dynamic process that requires regular updates [4]. Reference [12] proposed a public transport system called “BusNet” in order to monitor environmental pollution and pavement surface condition. BusNet implements a sensor network, placed on top of public buses. The acceleration sensors identify potholes through changes in the vertical acceleration and determine the car speed modification using the horizontal

acceleration. Reference [13] used seven taxis running in the Boston area and developed a mobile sensor system called (Pothole Patrol). Each taxi needed a computer running Linux, a WiFi card for transmitting collected data, an external GPS and a 3-axis accelerometer. In general, there are three main problems concerning the above systems. First, there is a large number of events (such as doors being knocked, unexpected swerves, traffic jam) and road anomalies (such as road expansion joints) that are difficult to distinguish from potholes. The second problem is that the systems cannot identify between a pothole that merits fixing and a bump in the road. The third problem is that the values reported by the sensors depend on a car’s speed and how the sensors are mounted on the car. Reference [14] proposed a method requiring an Android smartphone with GPS, 3-axis accelerometer and a communication channel (cellular or wi-fi). The system consists of two application components, one for the Android device and one for a data server. Reference [5] proposed a system that detects road anomalies using mobile equipped with inertial accelerometers and gyroscopes sensors, while [15] described a system which included a combination of a custom mobile application and a georeferenced database system. The roughness values computed and stored in a back-end geographic information system enable visualization of road conditions. Reference [16] suggested the gyroscope around gravity rotation as the main indicator for road anomalies, in addition to the accelerometer sensor, in order to avoid false-positive indications; especially when there is a sudden stop or sudden change in motion acceleration. The above systems, despite hardware differences in terms of GPS accuracy and accelerometer sampling rate and noise, they show that pothole detection is possible. The 2014 Mercedes-Benz S-Class used a Light-Detection-and-Ranging (lidar) scanner to measure pavement roughness as a component of an active suspension system. Recently, the Jaguar Land Rover automaker announced that is researching a new connected vehicle technology, which allows a vehicle to spot dangerous potholes in the road and then share these data in real time with other vehicles and road authorities [17]. III.

METHODOLOGICAL SETUP

As aforementioned, the work presented herein investigates the use of smartphone technology for the detection of roadway anomalies and for their classification. The study focuses on three types of common roadway anomalies (transverse depressions, Fig. 1a; longitudinal depressions, Fig. 1b; and potholes or manholes, Fig. 1c), which it examines first individually and then in tandem. Data on these types of roadway anomalies is collected in-situ by use of a car fitted with a smartphone (mounted on the car’s windshield) with its GPS and accelerometers activated, and with an OBD-II reader connected to it. The smartphone had also its video camera active, for recording the routes traveled and subsequently visually verifying the existence of roadway anomalies (as pinpointed by the data analysis and Artificial Neural Networks (ANN) classification), and was fitted with the DashCommand app for recording (and exporting) sensor readings, date/time stamps, and GPS locations of taken data.

(a)

(a)

(b)

(b) Fig. 2 (a) Smartphone’s roll and pitch directions and [18] (b) relation to car’s wheels’ differential [19]. TABLE I. DATA COLLECTED AND VARIABLES USED FOR CLASSIFYING ROADWAY ANOMALIES

Variable

VAR _8

Variable Name AUX.ACCEL.FORWARD (Gs) AUX.ACCEL.LATERAL (Gs) AUX.ROTATION.ROLL (°) ROLL 2 - ROLL 1(°) AUX.ROTATION.PITCH (°) PITCH 2 - PITCH 1(°) AUX.GPS.LATITUDE (°) AUX.GPS.LONGITUDE (°)

VAR _11 VAR _12

AUX.GPS.SPEED (km/h) SAE.RPM (rpm)

VAR _28

CALC.ACCELERATION (m/s²)

VAR _1

(c) Fig. 1 Roadway anomaly types examined for detection and classification: (a) transverse defect/anomaly; (b) longitudinal defect/anomaly; (c) potholes/manholes.

The collected case-study data are of high spatial resolution (at intervals of 0.1 seconds) and pertain to both uni-dimensional (e.g. X, Y, Z accelerations, speed, etc.) and two-dimensional indicators (e.g. the smartphone’s roll and pitch values, Fig. 2a, which can be related to the traveling host car’s roll and pitch values). The roll relates to a car’s acceleration differential between its left and right front wheels, and the pitch relates to a car’s acceleration differential between its front and rear wheels. In tandem, roll and pitch indicate how the host car is off balance, sideways and front to back. The field investigation included the collection of sensor data for a total of eleven parameters (as shown in Table I), presumed to influence the accuracy of detection and classification of the roadway anomalies examined. Most of the eleven factors examined were varied during the site investigation (e.g. speed, acceleration, engine RPM) and some were left constant (e.g. type of car/engine and fuel type). Two variables (VAR_4, VAR_6) were calculated posterior the data collection, as they relate to the difference in roll and pitch, respectively, between two data points (0.1 seconds in difference between the two data points).

VAR _2 VAR _3 VAR _4 VAR _5 VAR _6 VAR _7

IV.

Variable Description Forward Acceleration Lateral Acceleration Vehicle Roll

Vehicle Pitch

GPS Latitude GPS Longitude GPS Vehicle Speed Engine RPM Current acceleration based on the last two speed readings

RESULTS AND DISCUSSION

A first look at the raw data reveals the complexity of the problem, as variables, such as the X/Y/Z acceleration and vehicular speed, thought to indicate roadway anomalies are not as conclusive as originally thought. A plot of acceleration in one direction over time, such as the vertical acceleration (Fig. 3),

Gz (m/s²)

ROLL 2 – ROLL1 (°)

helps demonstrate the inconclusiveness of the taken readings. The vertical variability is random, and even at a point of known roadway anomaly the variability in the acceleration is not conclusive of the presence of the anomaly. A similar situation presents itself for various vehicular speeds (20, 40, 60, 80, 100km/hr.).

Time (s)

Known location of roadway anomaly Time (s) Fig. 3 Vertical acceleration over time

The situation can be improved, should one consider vehicular roll and pitch values over time (Fig. 4a, 4b), but even those indicators fail to safely indicate locations of roadway anomalies. Even though these plots indicate areas of concern (high and lows beyond the running average values), they are not conclusive. Further, the plots fail to report on other operating parameters which could influence the correctness of the data (e.g. the car’s speed). The difference in values between subsequent locations (Fig. 4c, 4d) seems to be a better predictor (points of high peaks indicate roadway anomaly), but even that is not fail-proof in the absence of other complimentary information.

PITCH 2 - PITCH 1 (°)

(c)

Time (s) (d) Fig. 4 Raw and processed sensor data: (a) roll, over time; (b) pitch, over time; (c) point-to-point roll variation, over time; (d) point-to-point pitch variation, over time.

Roll (°)

For that reason, the datasets are then fed into an artificial neural network (ANN), consisting of 11 inputs and 2 outputs (its architecture is as shown in Fig. 5) and implemented in MATLABTM. The ANN outputs are binary in nature (‘0’ for no defect, ‘1’ for defect), and they are used to classify data readings into classes of roadway anomalies. Time (s)

Pitch (°)

(A)

Fig. 5 ANN model schematic architecture and MATLABTM implementation.

Time (s) (b)

The ANN is first trained with each case of roadway anomaly (as given by Fig. 1), and then trained with all three roadway anomalies in tandem (ClassType_0, ClassType_1, ClassType_2, ClassType_3). For each case, 70% of the data are used for training, 15% for testing and 15% for validating the ANN, with a synopsis of the obtained classification results shown in Table II.

Fig . 1a Fig . 1b Fig . 1c Fig . 1a, 1b, 1c

0:Type_0 1:Type_1 0:Type_0 1:Type_2 0:Type_0 1:Type_3 0:Type_0 1:Type_1 2:Type_2 3:Type_3

Train . Accu racy 100% 100% 84% 95% 100% 100% 100% 100% 100% 100%

Valid . Accu racy 100% 96% 83% 98% 100% 100% 100% 100% 100% 100%

Test. Accu racy

Over. Accur acy

Total Accu racy

100% 100% 79% 94% 100% 100% 100% 94% 100% 100%

100% 99% 83% 95% 100% 100% 100% 99% 100% 100%

99.8 % 91 % 100% 99.9 %

The ANN confusion matrices enable us to investigate the numbers (and percentages) of not only accurate classifications (i.e. perfect matches between target and output classes) but also of erroneous and of false-positive classifications. As a backdrop, let us note that a good classifier yields a confusion matrix that will look dominantly diagonal, and that all off-diagonal elements on the confusion matrix represent misclassified data. TABLE III. TRAINING CONFUSION MATRIX

0 0 0.0% 0.0% 0 0 0.0% 0.0% 240 0 28.6% 0.0% 0 299 0.0% 35.6% 100% 100% 0.0% 0.0% 3 4 Target Class

100% 0.0% 100% 0.0% 100% 0.0% 100% 0.0% 99.9% 0.1%

1 2 3 4

Output Class

0 0.0% 109 13.0% 0 0.0% 0 0.0% 100% 0.0% 2

0 0 0 0.0% 0.0% 0.0% 22 0 0 12.2% 0.0% 0.0% 0 43 0 0.0% 23.9% 0.0% 0 0 72 0.0% 0.0% 40.0% 100% 100% 100% 0.0% 0.0% 0.0% 2 3 4 Target Class

100% 0.0% 100% 0.0% 100% 0.0% 100% 0.0% 100% 0.0%

1 2 3 4

TABLE V. TEST CONFUSION MATRIX

As can be seen, the ANN classification arrives at a high degree of accuracy, not only when examining the existence of a specific roadway anomaly, but also when examining for all roadway anomalies at once (last case in Table II). This high degree of accuracy is also evident when the ANN results are examined closer, by means of the produced ANN confusion matrices for training, testing and validating the classification of defects.

191 22.7% 1 0.1% 0 0.0% 0 0.0% 99.5% 0.5% 1

43 23.9% 0 0.0% 0 0.0% 0 0.0% 100% 0.0% 1

55 30.6%

0 0.0%

0 0.0%

0 0.0%

100% 0.0%

1

0 0.0%

22 12.2%

0 0.0%

0 0.0%

100% 0.0%

2

0 0.0% 0 0.0% 100% 0.0% 1

0 0.0% 0 0.0% 100% 0.0% 2

100% 0.0% 100% 0.0% 100% 0.0%

3

46 0 25.6% 0.0% 0 57 0.0% 31.7% 100% 100% 0.0% 0.0% 3 4 Target Class

4

Output Class

Classes

TABLE VI. ALL CONFUSION MATRIX

289 24.1% 1 0.1% 0 0.0% 0 0.0% 99.7% 0.3% 1

0 0 0 0.0% 0.0% 0.0% 153 0 0 12.8% 0.0% 0.0% 0 329 0 0.0% 27.4% 0.0% 0 0 428 0.0% 0.0% 35.7% 100% 100% 100% 0.0% 0.0% 0.0% 2 3 4 Target Class

100% 0.0% 99.4% 0.6% 100% 0.0% 100% 0.0% 99.9% 0.1%

1 2 3 4

Output Class

Ref ers to

TABLE IV. VALIDATION CONFUSION MATRIX

Output Class

TABLE II. ANN TRAINING AND VALIDATION STATISTICS FOR THE VARIOUS ROADWAY ANOMALY CASES EXAMINED

In essence, the ANN classifier detects and correctly classifies the three roadway anomalies (target classes 2, 3 and 4) while also detecting the ‘no defect’ condition (target class 1), thus distinguishing between normal and abnormal roadway pavement conditions. V.

CONCLUSIONS

The paper presented a study on the use of smartphones for the detection of roadway anomalies and on the use of artificial neural networks for the classification of such defects. The applied methodology is readily available and low-cost.

Furthermore, as evidenced by the case-study detection and classification of the three types of roadway anomalies, the utilized technology and method have a high-degree of accuracy (exhibiting accuracy levels higher than 90%). Finally, the proposed system architecture can be utilized in crowd-sourced applications leading to roadway assessment and pavement management systems. The proposed methodology is currently field-tested with larger datasets and a higher number of roadway defect types, with links created to GIS mapping and database management systems for use of the proposed methodology in pavement management systems. REFERENCES [1]

[2]

[3] [4]

[5]

[6]

[7]

[8]

I. M. Panagopoupou and P. A. Chassiakos, "An Optimization Model For Pavement Maintenace Planning And Resource Allocation," Transportation Research Circular, vol. Number E-C136, 2012. E. Dennis, Q. Hong, R. Wallace, W. Tansil and M. Smith. Pavement condition monitoring with crowdsourced connected vehicle data. Transportation Research Record: Journal of the Transportation Research Board (2460), pp. 31-38. 2014. C. Johnson, Pavement (Maintenace) Management Systems. APWA Reporter, 1983. Washington Department of Transportation, A Guide for Local Agency Pavement Managers, Washington State Department of Transportation, Trans Aid Service Center. The Northwest Technology Transfer Center, 1994. F. Seraj, B. J. Zwaag, A. Dilo, T. Luarasi and P. Havinga. RoADS: A road pavement monitoring system for anomaly detection using smart phones. 2014. AASHTO, Guidelines for Pavement Management Systems. Washington, D.C: American Association of State Highway and Transportation Officials, 1990. P. DeCabooter, K. M. Weiss, S. Shober and B. Duckert. Wisconsin's pavement management decision support system. Transp. Res. Rec. (1455), 1994. D. Walker, L. Entine and S. Kummer, Pavement Surface Evaluation and Rating: PASER Manual. Madison WI: University of Wisconsin, Transportation Information Center, 2002.

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

A. Wolters, K. Zimmerman, K. Schattler and A. Rietgraf, "Implementing pavement management systems for local agencies," Illinois Center for Transportation, Urbana, IL 61801, Tech. Rep. ICT-R27-87, 2011. J. M. McQueen and D. H. Timm. Part 2: Pavement monitoring, evaluation, and data storage: Statistical analysis of automated versus manual pavement condition surveys. Transportation Research Record: Journal of the Transportation Research Board 1940(1), pp. 53-62. 2005. AASHTO, "Asset management data collection guide, task force 45 report," American Association of State Highway and Transportation Officials, Washington, DC, 2006. K. De Zoysa, C. Keppitiyagama, G. P. Seneviratne and W. Shihan. A public transport system based sensor network for road surface condition monitoring. Presented at Proceedings of the 2007 Workshop on Networked Systems for Developing Regions. 2007. J. Eriksson, L. Girod, B. Hull, R. Newton, S. Madden and H. Balakrishnan. The pothole patrol: Using a mobile sensor network for road surface monitoring. Presented at Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services. 2008. G. Strazdins, A. Mednis, G. Kanonirs, R. Zviedris and L. Selavo. Towards vehicular sensor networks with android smartphones for road surface monitoring. Presented at 2nd International Workshop on Networks of Cooperating Objects, Chicago, USA. 2011. G. Alessandroni, L. Klopfenstein, S. Delpriori, M. Dromedari, G. Luchetti, B. Paolini, A. Seraghiti, E. Lattanzi, V. Freschi and A. Carini. SmartRoadSense: Collaborative road surface condition monitoring. Proc.of UBICOMM-2014.IARIA 2014. A. Mohamed, M. M. M. Fouad, E. Elhariri, N. El-Bendary, H. M. Zawbaa, M. Tahoun and A. E. Hassanien. "RoadMonitor: An intelligent road surface condition monitoring system," in Intelligent Systems' 2014Anonymous 2015. N. O'Donnell, K. McConomy, Jaquar Land Rover Announces Technology Research Project to Detect, Predict And Share Data on Potholes, [online] 2015, http://newsroom.jaguarlandrover.com/en-in/jlrcorp/news/2015/06/jlr_pothole_alert_research_100615/ (Accessed: 15 June 2015). Yaw Pitch & Roll to spherical Theta & Phi., Physics Forums, [online] 2015, https://www.physicsforums.com/threads/yaw-pitch-roll-tospherical-theta-phi.788531/ (Accessed: 15 July 2015). Motions on Formula 1 car., Formula1 Dictionary, [online] 2015, http://www.formula1-dictionary.net/motions_of_f1_car.html (Accessed: 15 July 2015). J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68-73.