Vision-based Detection of Power Distribution Lines in Complex Remote Surroundings Hrishikesh Sharma∗ , Rajeev Bhujade∗ , Adithya V∗ and Balamuralidhar P.∗ ∗
TCS Innovation Labs, Bangalore. Email:
[email protected],
[email protected],
[email protected],
[email protected] Abstract—Inspection and Maintenance of power lines is an important and costly legal responsibility, mainly for public safety, of a power distribution company. The traditional method of manual, on-foot inspections have many disadvantages such as long inspection cycles etc. Due to advances in sensors and flight technology, an emerging cost-effective solution is to employ Unmanned Aerial Vehicles (UAV) for the inspections. Automatic detection and complete extraction of power lines from UAVbased aerial imagery, having complex and varying natural surroundings, is first critical problem to be efficiently solved for enabling detection of line faults. In this paper, we propose a solution based on a novel morphological operator, and a robust image space heuristics for locating and complete extraction of power lines. Our algorithm is a three-stage algorithm, and it focuses on minimization of missed detection of line segments. The entire algorithm was tested on a real outdoor video shot for around 320 meters length of power grid using a fixed-wing UAV. No missed detection of important line segments across the sequence of overhead video frames, and < 3% false positives prove that our approach is very effective. We believe that our algorithm can be easily ported for line detection problem in any other real outdoor video as well. Index Terms–Morphological Detection; Heuristic Algorithm; Transmission Line Monitoring; UAV
I. I NTRODUCTION
I
N any country, there are many installations and systems which are located in outdoor areas. Some such systems are vast, and their installation area may run into hundreds of kilometers, e.g. a pipeline or a power grid. Some such systems are inapproachable by humans, e.g. a dam on a river in a tricky terrain, or power grid in thick forests. Maintenance, both preventive and breakdown, of all such installations is an obvious must, and typically a costly legal responsibility towards public safety in most countries. As is well-known, the current prevalent practice of periodic human inspections for maintenance of such systems are somewhat inefficient. Such inspection can sometimes lead to some zones being inspected more frequently than needed, and some not enough. Also, this task is difficult because humans cannot enter the vicinity of high voltage. Semiautomatic inspection using various non-human agents have been proposed to overcome these inefficiencies. The important agents so far proposed are satellite, helicopter, crawling robots and UAVs [1]. Inspection through helicopters is more efficient than manual inspection, but it is quite expensive [2]. The other option of hiring services of a remote sensing satellite entails the problem that satellites have a fixed schedule while c 2014 IEEE 978-1-4799-2361-8/14/$31.00
they move in their orbits [3]. Hence their visit time in a particular zone may not suit the scheduling requirements of an inspection, specially when there is an emergency. A third option is of employing a robotic machine which can hang and walk along the power cables, such as one by Hydro Quebec Inc. that is being used by India’s National Power Grid Corporation [4]. Generally, such robots are relatively heavy, and are limited to survey of just vegetation growth in line vicinity. An emerging option out of all the non-manual methods is to employ a UAV [5]. UAVs enable the in-depth reconnaissance and surveillance of an area. A small UAV can acquire highresolution images that could be used in line inspections and in fact, many other applications. By [6], the UAV option rd incurs approximately 13 the cost of employing helicopters for inspection. Also, unlike robots, it is more reliable, and covers detection of more faults. Finally, in the aftermath of emergencies such as cyclone, earthquake, landslides etc., it is quicker and easier to employ UAVs than to hire and employ a helicopter for damage to the grid, or do a manual survey [7]. The transmission lines run into tens of kilometers, and the image data acquired by UAV-mounted imaging sensor is correspondingly huge. It is impossible for any human observer to sift through all images/frames and locate a fault in real-time. Hence vision-based automatic extraction of power lines and fault detection along the extracted lines is a very important problem to be solved for UAV-based monitoring. Towards detection of power lines in overhead recorded video, especially in varying complex natural outdoor background, we have a designed a novel three-stage algorithm. We perform adaptive thresholding to first isolate the power lines in varying light conditions along the line, and also due to time of day. We then use a new erosion operator to select candidate set of edges from which power lines have to be detected. The operator retains the edges of a foreground region, especially straight line segments, while eroding the interior. In the final stage, we use robust heuristics on pixel intensity to detect “stripes” in image matrix that correspond to power lines. Overall, the algorithm aims to minimize the missed detections, or false negatives, since a minimum tracking of power line just below the overhead camera is guaranteed in all frames. It also additionally minimizes the mis-classification of random linear feature as a power line segment i.e. a false positive, since long linear stripes in image intensity is quite unique to power lines. In fact, < 3% frames show residual false positives.
In past, for line detection in visual band images, most of the researchers have used Hough Transform at the core of their detection algorithm [8], [9], [10], [11], [12]. It is wellknown that Hough Transform has computational complexity of the order of O(n3 ) i.e. very high. Since we work with video data of around 24 frames per second, faster processing is needed. The core stage of our algorithm has a complexity of just O(n2 ). Further, we have focused on detection and tracking on video data with around 1000 frames to prove the robustness and detection performance of algorithm, while most other papers work with limited set of discrete images. In works following other approaches, [13] uses data from multiple sensors mounted on UAV, including LiDAR, and focuses on somewhat different problem of detection of intruding vegetation to the transmission line corridor. [14] has a similar objective, and uses photogrammetry to measure the distance between nearby vegetation and the power line. A lengthy method for removal of multi-region background is given in [15], but we achieve the same using adaptive thresholding. Gabor filters have been used for edge detection stage in [9], but [7] uses canny edge detection for this stage which gives more accurate results. Our approach is similar to approach in [16], but we differ in the middle step in the sense that we use a simple erosion operator. Further, we deal with overhead line detection. [17] also uses image space heuristics to detect linear bands, but it does that in very beginning which results in various types of false positives which are then weeded out one-by-one, resulting in long operations. Clustering in Hough space for line detection has been used in [12], while we perform some form of clustering in image/pixel space. In the remaining paper, we first describe our algorithm in detail in section II. We then describe the nature of our experiments in section III, which is followed by the main section IV on results and analysis, before concluding this paper. II. P ROPOSED D ETECTION A LGORITHM For overhead monitoring, the UAVs can fly at a height above the power-lines, and in relative proximity. The video captured from this point will have the power-lines having good brightness due to reflection. However, the power-lines in remote areas run long distances over complex terrain. The image/video background hence can vary from trees and patches of greenery to different flat spaces such as barren patches, along with other objects like houses and roads. Accordingly, we have designed a new detection algorithm that can detect lines in any such varied background. Our detection algorithm consists of three stages as shown in Figure 1. The first two stages are related to preprocessing, that are tuned to cater to the core line detection algorithm in the final, third stage. As explained earlier, the first stage uses adaptive thresholding to isolate the power lines in varying light conditions. In the second stage, we then collect the significant edges of bright regions available after thresholding, as the candidate edge set. For that, we use a new erosion operator, which retains the edges of a foreground region, especially
Fig. 1: Functional Block Diagram of Detection Algorithm
straight line segments, while eroding the interior. In the final stage, we use robust heuristics on pixel intensity to detect linear bands, or stripes in image matrix that correspond to power lines. The choice of third stage is the key to performance of our algorithm. As will be shown in details in section IV, missed detections, or false negatives, are very less since a minimum tracking of power line just below the overhead camera is guaranteed. Similarly, mis-classification of a random linear feature as a power line segment i.e. a false positive is also very less, since long linear stripes in image intensity is quite unique to power lines or any other linear infrastructure in remote outdoor locations. Algorithm 1 Algorithm for Detection of Power Lines Perform a decomposition of the video into a sequence of frames. For Each Frame in the Sequence do Adaptive Thresholding in a 3×3 neighborhood using Gaussian Kernel Majority logic based erosion of remaining background, while retaining all linear features Heuristic Detection of Power Lines in Image Space, while simultaneously weeding out false positives end for A. Adaptive Thresholding Isolating the power lines from a background that varies in intensity becomes challenging with the use of a single global threshold. Also, between frames the intensity of the captured lines themselves vary according to the position of the UAV. Finally, the inspection could be done at any time in day i.e. different daylight brightness conditions, and the detection algorithm must be able to detect successfully in all conditions. An adaptive threshold algorithm that uses a different threshold for every pixel in each frame according to the pixel intensity in the neighborhood can address all these issues. The algorithm employs a blur function to calculate
(a) Synthetic Test image with lines 3 pixels thick
(b) Eroded image after applying operator with n=3 and threshold 2·n
(c) Eroded image after applying operator with n=5 and threshold 2·n
(d) Eroded image after applying operator with n=7 and threshold 2·n
Fig. 2: Erosion Performance on Synthetic Image with Linear Features
the neighborhood intensity. The choice of the blur function depends on the background characteristics, as well as its customization/tuning. The power lines are just 2-3 pixels thick in our case, and the background greenery, houses and roads are spread across more pixels. Hence using a Gaussian kernel, that calculates a weighted mean function in a 3×3 neighborhood is able to suppress more background while retaining the power lines when compared to any other kernel type or size. This is because the kernel covers sufficient foreground and background for a single pixel, and at times 2-3 pixel thick lines. B. Majority-based Morphological Erosion of Remaining Background The output of the adaptive thresholding algorithm is a binary image with the power line pixel thickness retained from the original image and certain bright background patches. The perspective view of the power lines from the front facing camera, during overhead tracking, results in lines of varying thickness across the image. The lines are thickest near the bottom edge of the frame and tapers off towards the horizon near the top part of the image. Having a line of uniform thickness makes it easier to distinguish it from the background still present in the image after the previous stage. To get such uniform thick lines, we use a novel majoritybased erosion operator. This operator is specific for line detection, as it operates by thinning down the line thickness to a single pixel. A n×n all-ones matrix is as the kernel of the operator is moved across the image. A sum-of-product (SOP) of ‘n’ or less was taken as a favorable candidate output and the center pixel was set to one, else it was set to zero. The condition here was that a line of any orientation cannot have more than ‘n’ as sum of product in a n×n kernel. That is, 1) If the sum of products after imposing all-1 n×n kernel is > n pixels, then the foreground contains something else than a power line. It should be suppressed. 2) If the SOP is = n, it has a power line crossing along any rotation of the kernels principal diagonal, and nothing else. It has to be retained. 3) If the SOP is < n, it has a power line crossing along any rotation of the one of the kernels minor diagonals, and nothing else. It has to be retained.
The morphological operator is used to retain lines in a binary image and remove anything else. Basically it is a shape retainer and in this application, the shape is a line. The interiors of the foreground patches are removed and only their edges are retained. The size of the operator was chosen based on the distance of separation between lines. It is obvious that for a n×n kernel, the distance between the lines should be a minimum of ‘n’ pixels, for both the lines to be retained. The performance of this operator for a synthetic test image, a real sideways power grid image, and a real overhead power grid image is shown in Figs. 2, 3 and 4. The images demonstrate that a kernel of size 3×3 reduces false negatives the most, while a kernel of size 7×7 reduces false positives the most i.e. a tradeoff. The images also demonstrate that the erosion operator works irrespective of the orientation of the power lines. To understand detailed parametric tuning of this operator, refer to [18]. C. Heuristics based line detection It is a known fact that metallic objects reflect majority of electromagnetic radiation in visual band that impinges upon it. Hence it is expected that other than visible sky above the horizon in a typical overhead frame, no other artefact within the frame will have high intensity, as captured within the pixel intensity matrix. Since we are able to erode sky easily due to usage of our erosion operator, power lines remain as the most luminant artefact in the post-erosion frames. The same has also been observed experimentally, as a predicate at all stages of processing: from colored images to HSV-represented images to binary images, post erosion. Specifically, in an HSV representation of image, power lines consistently showed intensity > 65% in the value dimension, along the surface of (thin) power line captured. The power lines captured in our video are 3-pixel wide. However, in the perspective view captured by the front facing camera, the parallel edges of power lines appear to meet at the horizon. Hence the line thickness slowly tapers off as its distance from the camera increases. The above property holds true after erosion also. Post erosion, the lines are single-pixel wide and also have an expected slightly staggered appearance due to overhead detection and discreteness of image capturing grid on the CCD backplane of the camera. The minor stagger, a uniform tilt of 1 pixel to
(a) Power Grid Sideway Image
(b) Eroded image after applying operator with n=3 and threshold n
(c) Eroded image after applying operator with n=5 and threshold n
(d) Eroded image after applying operator with n=7 and threshold n
Fig. 3: Erosion Performance on Grid Sideways Image
(a) Power Grid Overhead Image
(b) Eroded image after applying operator with n=3 and threshold n
(c) Eroded image after applying operator with n=5 and threshold n
(d) Eroded image after applying operator with n=7 and threshold n
Fig. 4: Erosion Performance on Grid Overhead Image
right or to the left, manifests as small vertical line segments that are stacked one above the other in a staircase manner. The more vertical the line during overhead detection, the bigger are the length of the line segments, and lesser it staggers. We use this consistent heuristics, which has been also used in [16], [17] in a different form, to trace the lines. In overhead detection, power lines typically run from bottom to top of the image matrix. To trace the lines, first, their starting locations have to be identified. The front facing camera has the closest view of the line at the bottom of the image. Hence the bottom portion of the image has the lines in greatest prominence in all frames, with no breakages in between. In the bottom few rows of the post-erosion matrix, we look out for vertical runs of all 1s. We also assume that lines are separated by at least 3 pixel distance, which can be tuned, and hence use windowing to locally identify starting positions. Once the starting positions are identified, row-wise scanning from bottom to top is continued to greedily detect and accumulate the line length, till a break in the line is encountered. III. E XPERIMENTS AND DATA C OLLECTION The image capturing can be done using many cameras e.g. visual or near infrared camera. The popular imaging is done by using visual camera. A video was captured as the visual band image data sequence that would be used for evaluation of our algorithm. A 320 meter section of line in the outskirts of Bangalore was chosen as the test site. A fixed-wing mini-UAV was flown at a speed of around 35 km/hour. The positioning of the UAV was done so that it could flow overhead to the power grid. This implied that the camera tilt was also front-facing. This minimized the amount of occlusion among power lines.
The camera captured the frames of (standard) size 640×480 pixels, at a frame rate of 24 fps. The UAV flew over lines between two poles separated by one intermediate pole. The overall video consisted of 782 frames, and lasted around 32 seconds. The background of the power lines was varying, and typically consisted of vegetation, sky, few unpaved roads and houses. For lack of space, while demonstrating the results, we only show few sample frames out of this video. IV. R ESULTS AND P ERFORMANCE A NALYSIS The algorithm was implemented using OpenCV software package. While the algorithm is tested over around 800 frames, to showcase our algorithm’s effectiveness, we show tracking of power line in two specific frames here. The two images have been chosen with different enough background. They were chosen in order to show that the proposed approach works well in different conditions. The background of figure 6a is more complicated than the one of figure 5a. The images after complete processing and tracking are shown in Figs. 5b and 6b, respectively. The results show qualitatively that we are able to detect majority of overhead power lines. For overhead monitoring, this is very useful result as we quantitatively show next. For fault detection which is the next stage in power line monitoring, this initial success of power line detection is important. For quantitative analysis of results in form of tracked line length, we present both the worst case and average case analysis of our algorithm. Before presenting the analysis, let us discuss a lower bound on the required length of line to be tracked between two successive frame captures, in units of pixels. The UAV was flown at a speed of approximately 10
(a) Sample Video Frame 1
(b) Frame Overlaid with Detected Power Lines
Fig. 5: Detection of Lines in Sample Frame 1
(a) Sample Video Frame 2
(b) Frame Overlaid with Detected Power Lines
Fig. 6: Detection of Lines in Sample Frame 2
m/s, and the frame capture rate was 24 fps. Hence between two frame captures, UAV flies over around 0.4 meters. Though not shown for brevity, it is easy to calculate that for two poles that are 100 meters apart as per the grid design, the wire between them covers 3/4 of image length i.e. 360 pixels. Hence 0.4 meter corresponds to 1.44 pixel distance. The inter-frame distance poses a very important requirement on line tracking. This is because in overhead detection, the detection of line segment just below the front-facing camera is very good. Hence if we can somehow always detect power line segment equal in length to the distance covered by UAV between two successive images from the bottom of each image, for all such successive pairs, then across the length of video, we can successfully claim that we have tracked the entire power line. For statistical analysis, we chose 281 sample frames, which are placed uniformly 3 frames apart in the real video sequence. From analysis of detected length data for all these images frames, it was seen that only 9/281 frames had few false positives. For all other frames, it was again seen that the
minimum length of line detected is around 17 pixels from bottom of the frame. The power line in each frame is around 300 pixels long. However, when we compare this to our important requirement of being able to track at least 1.44 pixels from the bottom of each frame, then we have definitely done around 10× better, even in worst-case.
Moving to average case analysis, we did a classification of long and short lines in each frame, by thresholding on 10% variation around the mean. It was found that only 13% of the lines have no. of detected long lines lesser than no. of detected short lines. This implies that in majority of frames, lines were detected that were long enough. Further, we also took a global mean of all such detected lines across all frames. A global average of 179 pixels, out of 300 pixel long representation of a true line in our video, is not only 125× better than required bound, but also covers 60% of the overall length of the visible grid till the horizon.
V. C ONCLUSION In this paper, we have provided an algorithm of detection of power lines in complex and varying outdoor surroundings. Automatic detection of power lines is an important first step in usage of increasingly popular solution of using UAV as a sensing platform for many applications, including power grid monitoring. The design of algorithm is based on a novel morphological erosion operator, and a robust image pixel space heuristics to extract power lines. The entire algorithm was tested on a real outdoor video shot for around 320 meters length of power grid using a fixed-wing UAV. In all frames, we are able to detect at least few lines for which detected initial length is minimum of 17 pixels, more than the minimum required length of 1.44 pixels for overhead detection in video monitoring. In fact, in 87% of the frames, long detected lines are in majority and cover upto 60% visible line length till horizon. Thus, we have 0% missed detection when it comes to important line segments. Also, false positives are far and few, and were found to be limited in only 9/281 analyzed frames. We are currently working on further improving the degree of detected line length in each frame on two counts. Whenever there is a bright barren patch beneath a power line, the greedy line extraction in last stage is found to stray into the patch. Also, in cases when two lines at different altitude nearly align in the same plane in which UAV is flying, the captured image renders them very close, typically 1-2 pixels apart. We are working on detecting all such closely rendered lines as well. Both of them require fine-tuning of one stage respectively, and not any redesign of algorithm. Hence, and overall, we believe that our algorithm for power line detection is robust enough with good performance, and hence can be also applicable to any line detection problem in any other real outdoor video as well. ACKNOWLEDGMENTS We thank Prof. Omkar and his research group from Dept. of Aerospace, Indian Institute of Science, Bangalore for collaborating and providing us with test video data to run our algorithm. R EFERENCES [1] D. Jones, “Power Line Inspection – a UAV Concept,” IEE Forum on Autonomous Systems, 2005. [2] X.-f. Zhang, G. Chen, W. Long, Z.-f. Cheng, and K. Zhang, “Current Status and Prospects of Helicopter Power Line Inspection Tour with LIDAR,” Electric Power Construction, vol. 3, p. 013, 2008.
[3] F. Nˇemec, O. Santol´ık, M. Parrot, and J. Bortnik, “Power Line Harmonic Radiation observed by Satellite: Properties and Propagation through the Ionosphere,” Journal of Geophysical Research, vol. 113, no. A8, 2008. [4] “India’s Power Grid Corp. to deploy robotics technology,” http://www.elp.com/articles/2011/11/ india-s-power-grid-corp-to-deploy-robotics-technology.html, 2011. [5] S. Adams and C. Friedland, “A Survey of Unmanned Aerial Vehicle (UAV) usage for Imagery Collection in Disaster Research and Management,” in International Workshop on Remote Sensing for Disaster Response, 2011. [6] T. Skrzypietz, “Unmanned Aircraft Systems for Civilian Missions: Policy Paper,” Brandenburg Institute for Society and Security, Tech. Rep., 2012. [7] Z. Li, Y. Liu, R. Hayward, J. Zhang, and J. Cai, “Knowledge-based Power Line Detection for UAV Surveillance and Inspection Systems,” in IEEE International Conference on Image and Vision Computing, 2008, pp. 1–6. [8] T. W. Yang, H. Yin, Q. Q. Ruan, J. D. Han, J. T. Qi, Q. Yong, Z. T. Wang, and Z. Q. Sun, “Overhead Power Line Detection from UAV Video Images,” in IEEE International Conference on Mechatronics and Machine Vision in Practice, 2012, pp. 74–79. [9] C. Mu, J. Yu, Y. Feng, and J. Cai, “Power Lines Extraction from Aerial Images based on Gabor Filter,” in International Symposium on Spatial Analysis, Spatial-temporal Data Modeling, and Data Mining. International Society for Optics and Photonics, 2009. [10] G. Yan, C. Li, G. Zhou, W. Zhang, and X. Li, “Automatic extraction of power lines from aerial images,” IEEE Geoscience and Remote Sensing Letters, vol. 4, no. 3, pp. 387–391, 2007. [11] W.-G. Tong, B.-S. Li, J.-S. Yuan, and S.-T. Zhao, “Transmission Line Extraction and Recognition from Natural Complex Background,” in IEEE International Conference on Machine Learning and Cybernetics, vol. 4, 2009, pp. 2473–2477. [12] J. Zhang, L. Liu, B. Wang, X. Chen, Q. Wang, and T. Zheng, “High Speed Automatic Power Line Detection and Tracking for a UAV-Based Inspection,” in IEEE International Conference on Industrial Control and Electronics Engineering, 2012, pp. 266–269. [13] S. J. Mills, M. P. Castro, Z. Li, J. Cai, R. Hayward, L. Mejias, and R. A. Walker, “Evaluation of Aerial Remote Sensing Techniques for Vegetation Management in Power-line Corridors,” IEEE Transactions on Geoscience and Remote Sensing, vol. 48, no. 9, pp. 3379–3390, 2010. [14] W. Zhang, G. Yan, N. Wang, Q. Li, and W. Zhao, “Automatic Block Generation and 3D Line Extraction in Photogrammetric Power Line Inspection,” in IEEE International Geoscience and Remote Sensing Symposium, 2007, pp. 2920–2922. [15] P. Mazurek and K. Okarma, “Application of Background Estimation and Removal Techniques for the Extraction of the Power Line Components on the Digital Images for the Automatic Power Line Inspection Systems,” Pomiary, Automatyka, Kontrola, vol. 54, pp. 698–699, 2008. [16] Z. Xiao, “Study on Methods to extract Transmission Line Information from High-resolution Imagery,” in International Symposium on Multispectral Image Processing and Pattern Recognition. International Society for Optics and Photonics, 2009. [17] L. Yuee and L. Mejias, “Real-time Power Line Extraction from Unmanned Aerial System Video Images,” in 2nd International Conference on Applied Robotics for the Power Industry (CARPI), 2012, pp. 52–57. [18] R. Bhujade, A. V, and B. P., “Design of a Novel Morphological Erosion Operator specific to Power Line Detection Problem,” Innovation Labs, Tata Consultancy Services Ltd., Tech. Rep., 2013.
