International Journal of Advanced Robotic Systems
ARTICLE
A 3D Laboratory Test-platform for Overhead Power Line Inspection Regular Paper
Chang-an Liu1, Ruifang Dong1∗, Hua Wu1, Guo-tian Yang1 and Wei Lin1 1 North China Electric Power University, Beijing, China ∗ Corresponding author(s) E-mail:
[email protected];
[email protected] Received 29 September 2015; Accepted 02 March 2016 DOI: 10.5772/62800 © 2016 Author(s). Licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract Using unmanned aerial vehicles (UAVs) for performing automatic inspection of overhead power lines instead of foot patrols is an attractive option, since doing so is safer and have considerable cost savings, among other advan‐ tages. The purpose of this paper is to design a 3D laboratory test‐platform to simulate UAVs' inspection of transmission lines and secondly, proposing an automated inspection strategy for UAVs in order to follow transmission lines. The construction and system architecture of our 3D test‐ platform is described in this paper. The inspection strategy contributes to knowledge pertaining to an automated inspection procedure and includes two steps: flight path planning for UAVs and visual tracking of the transmission lines. The 3D laboratory test‐platform is applied to test the performance of the proposed strategy and the tracking results of our inspection strategy are subsequently present‐ ed. The availability of the 3D laboratory test‐platform and the efficiency of our tracking algorithm are verified by experiments. Keywords 3D Laboratory Test‐platform, Overhead Power Lines, Inspection Strategy, Flight Path Planning, Visual Tracking, UAVs
1. Introduction Overhead power line inspection is an essential task for the maintenance of electrical grids, since the fault of power
transmission equipment can cause power outages, which can directly lead to the failure of nodes in other service providers such as Internet communication networks, hospitals, residential areas, etc. More importantly, power outages will result in large economic losses [1, 2]. Early detection of potential anomalies and timely maintenance can help avoid grid faults and reduce economic losses, further benefitting customers and electricity companies. The maintenance methodologies currently being used involve two main strategies: foot patrol and manned aerial vehicle inspection. However, these methodologies are in some ways inefficient and expensive. Moreover, helicopter assisted inspection can be dangerous [3]. Recently, meth‐ ods involving unmanned aerial vehicles (UAVs) have been widely used for various reasons, e.g., it decreases inspec‐ tion time, improves inspection quality and increases safety [4‐9]. It should be noted that observers always indicate defects in electrical equipment by watching videos taken by cameras mounted on UAVs. In our work, a 3D laboratory test‐platform is constructed to simulate the process of UAVs inspecting power lines. In [10], a one‐dimensional test‐bed is constructed for auto‐ matic power line inspection, in which a trolley runs along an aluminium track as a substitute for a helicopter. Since the inspection is always performed in a 3D space, a one dimensional test‐bed simulation differs substantially from a real case. Our 3D test‐platform, which allows movement in 3D space and which includes detailed scenarios for inspection, is designed to render the simulation more comprehensive. Int J Adv Robot Syst, 2016, 13:72 | doi: 10.5772/62800
1
In [11], the authors provide an overview of an aerial video inspection for power lines. They discuss in detail the motivation of pursuing video inspection techniques, especially in terms of the direction requirements for camera sightlines. Until now, the camera sightline direction has been controlled by an operator in order to keep desired object at the center of image. This is known as target selection and tracking. Target selection and tracking allows the observer to zoom in on an object for greater detail. However, manual target tracking imposes a severe work‐ load on the observer. Therefore, it is desirable to track the overhead power line automatically in order to have the target at the centre of images at all times. The methodologies of tracking overhead power lines can be categorized into two types. One is based on 3D coordi‐ nates (e.g., GPS) of the overhead power lines and UAVs, where the camera sightline direction can be set to be directed at the overhead power line, e.g., as shown in [12]. This approach depends heavily on the accuracy of the position measured, although a small error will result in tracking failed. For the second method, image‐based visual servoing is adopted. The most important thing in this method is that the overhead power line must be in the field of view, otherwise it will fail, e.g. [13‐14]. In this paper, an inspection strategy is proposed to track the overhead power lines for the camera on UAVs by combining these two strategies. First, a safe flight path is planned for UAVs according to inspection rules and the coordinates of overhead power lines; then, the pose of camera can be computed based on the planned safe flight path. Finally, a vision tracking technique is applied to make fine adjust‐ ments for the camera sightline direction in order to ensure the target is at the centre of the image. The primary contributions of this paper include: a novel design of the 3D test‐platform to simulate UAVs’ inspec‐ tion of overhead power lines; secondly, an inspection strategy is proposed for tracking overhead power lines automatically. The rest of this paper is organized as follows. Section II describes the architecture of our test‐ platform. Section III shows an overview of our inspec‐ tion strategy. Section IV discusses how to plan the flight path. Section V provides the visual tracking of overhead power lines for the camera. Section VI verifies the feasibility of the test‐platform and the inspection strat‐ egy, followed by conclusions in section VII. 2. Design of the 3D Test-platform 2.1 Architecture The architecture of our 3D test‐platform is shown in Figure. 1. It consists of three sub‐systems: a simulated flying system, an intelligent control system and an simulated inspection system. In the simulated flying system, as shown in Figure 1, the 3D motion module can move in 3D space (along the X, Y 2
Int J Adv Robot Syst, 2016, 13:72 | doi: 10.5772/62800
Simulated Inspecting System Navigation Module
Image Process Module
Miniature Scenery
Intelligent Control System Upper Computer
Controller
Safe-Protection Module
Simulated Flying system 3D Motion Module
Image Capturing Module
Hardware & Driver
Figure 1. The architecture of the 3D test‐platform
and Z axes). A PTZ (pan/tilt/zoom) camera is mounted at the end of the Z axis to capture images of inspection targets (as shown in Figure.2). The hardware and driver includes motors, drivers, gears and sensors. These work together to simulate UAVs in flight. In the intelligent control system, the upper computer aims to communicate with the simulated flying system and the displays the status of the entire system. The controller parses commands from the upper computer and controls the movement of the 3D motion module. The safe‐protec‐ tion module combines software and hardware techniques to avoid the motion module from moving out of bounds.
Y X Z
Figure 2. The structure of simulated flying system
Our simulated inspecting system consists of a navigation module, image process module and miniature scenery. The navigation module can plan a flight path automatically or manually. The image process module is able to capture images for targets, thereby detecting and tracking them; moreover, it sends tracking results to the intelligent control system. The miniature scenery (Figure 3) shows some power line equipment (e.g., overhead power lines, poles, insulators, fittings, etc.) and a complicated background for inspection (e.g. city, grass, fields, buildings, desert, river, animals, etc.). All of the modules work together to simulate UAVs’ inspection of overhead power lines.
k3 =
m3 × z3 × p , 360
(4)
where mi {i ∈ 1, 2, 3} and zi {i ∈ 1, 2, 3} represent the modulus of gears and the number of teeth in the X, Y and Z axes. Taking a derivative of (1), equation (5) is obtained: é x& ù é k1 ê ú ê ê y& ú = ê 0 ê z& ú ê 0 ë û ë
k2 0
0 ù éq&1 ù úê ú 0 ú êq&2 ú , k3 úû êëq&3 úû
(5)
From (5), a Jacobian matrix of translation for the 3D motion module is obtained as follows:
(b)
(a)
0
é k1 ê J1 = ê 0 ê0 ë
0 k2 0
0ù ú 0 ú, k3 úû
(6)
2.2.2 Analysis for the pose of camera
(c)
(d)
the 3D Amotion module toat capture images of 3D the target. PTZ camera is mounted the end of the Z axis of the motion module to capture images of the target. Figure 4 Figure 4 shows the Cartesian coordinate reference frame shows the Cartesian coordinate reference frame for the for the camera, camera, which can bein terms steered in(α) terms of yaw (α) which can be steered of yaw and pitch (β). (β). and pitch
Figure 3. Miniature scenery: (a) power lines over desert; (b) Figure 3. Miniature scenery: (a) power lines over desert; (b) power lines in power lines in the city; (c) power lines over grass; (d) power the city; (c) power lines over grass; (d) power lines near buildings lines near buildings.
2.2 Systems analysis
O
2.2.1 Analysis for 3D motion of the platform
Z
The 3D motion module moves along the X, Y and Z axes using three different motors.
Yp
The motion equation in 3D space is shown in (1):
)
Y
X Ya
Op
(d)
World
Oa Xp
Xa
Za
ù é k1 over 0 desert; 0 ù éq1 ù (b) ature scenery: (a) poweréê xlines ú ê úê ú y 0 k 0 ú êq 2 ú , = 2 ú ê grass; the city; (c) power linesê over (d) power ê z ú ê 0 0 k3 ú êq 3 ú ë û ë ûë û lines near buildings.
(1)
Zp Figure 4. Coordinate reference frames definition for the camera
where, x, y and z represent the moving distance of the 3D module along the X, Y and Z axes, ki {i ∈ 1, 2, 3} is the motion
Figure 4. Coordinate reference for the As noted above, it is desirable to frames keep the definition target centred. Here, yaw (α) and pitch camera. (β) can be adjusted to make the
alysis camera's sightline aim at the target. Assuming that the for 3D motion of the ofplatform coefficients gears in the X, Y and Z axes, respectively, and target locations in (a, b, c) are in the world coordinate frame θi {i ∈ obtainedalong from equations ki can be noted above, it iswith desirable to keep the target motion module moves the X,(2)~(4). Y and Z1, 2, 3} as the Asand its origin is coincident the camera frame, then the motion angle of gears in the X, Y and Z axes. axis yaw can be (α) rotated as shown Figure 5 in be order to centred.optical Here, and pitchin (β) can adjusted to ee different motors. point at the target. This rotation process is equal to rotate make the camera's sightline aim at the target. Assuming on equation in 3D space is shown in (1): m × z ×p 0 2 2 2
x
k1
y 0 z 0
0
k2 0
k =
0 11
0 2 ,
1
k = p × m × z k3 3 360 2
,
1
360
2
(2)
(1) 2
,
ndz represent the moving distance of the 3D
g the X, Y and Z axes, ki i 1, 2,3 is the
cients of gears in the X, Y and Z axes,
vector V (0, 0, a + b + c ) to V (a, b, c).
that the target locations in (a, b, c) are in the world The detailed process is conducted as follows. coordinate frame its origin coincident First, by rotatingand V0 counter clockwiseisaround the Y axiswith the 1 roughly α degrees, receive vector , as shown in camera frame, then we thecanoptical axisV can be rotated as (3) Figure 5(b), where α can be computed by (7). shown in Figure 5 in order to point at the target. This Dong, Wu, Guo-tian YangVand rotation Chang-an processLiu,isRuifang equal toHua rotate vector (0,Wei 0, Lin: a 3b A 3D Laboratory Test-platform for Overhead Power Line Inspection to V ( a, b, c ) . 0
The detailed process is conducted as follows.
2
2
c ) 2
Then, by rotating V1 counter clockwise around the X axis roughly β degrees, we can receive vector V, as shown in Figure 5(c), where β is computed by (8): æ a a = arc tan 2 çç 2 2 è b +c
ö ÷÷ , ø
1.
Flight path planning: plan the flight path of UAVs to guarantee a safe distance from overhead power lines and improve inspection quality;
2.
Pose planning for camera: plan the pose of camera to have the sightline aimed at target;
3.
Tracking overhead power lines: apply visual servoing techniques to make fine‐adjustments to the camera sightline, thereby locating the target at the centre of the image.
(7)
æ bö b = arc tan 2 ç - ÷ , è cø
(8)
4. Flight Path Planning Y
2. Position planning forpath camera: planalways the position Regular solutions for flight planning generate of the thecamera shortest path by connecting series of line 2. Position planning for camera: plan the position of the topossible have the sightline aimed atathe target; segments; following on, flight path can be produced camera have the sightline aimed at the target;apply visual by 3.toTracking overhead power lines: servoing smoothing out these line segments. In contrast, a number 3. Tracking overhead power lines: apply visual servoing techniques to make fine-adjustments to the of research works applied some curves (e.g. Dubins,camera techniques to make fine-adjustments to the camera Clothoid, PHthereby and B‐splined to plan path of the sightline, locatingcurve) the target at flight the centre sightline, thereby locating the target at the centre of the directly [15‐19]. However, the work. path planned by Dubins image. This is an essential image. This is an essential work. curves is not feasible for flight, due to its discontinuous curvature. Although Clothoid curves have continuous !Invalid Character Setting. path planning !Invalid Character Setting. Flight pathFlight planning curvature, when applying these, aplanning compound flight path Regular solutionssolutions for flight for pathflight always Regular path planning always with low flexibility is generated in a complicated manner. generate the shortest possible path by connecting series generate the shortest possible path bya connecting a series In this paper, a Nurbs curve is directly applied for planning of line segments; following on, flight path can be of line segments; following on, flight path can be the flight due to out its continuous and uniform produced by path, smoothing these line segments. In varia‐ produced by smoothing out these line process segments. In tions curvature. Thehave flight path planning contrast, aofnumber of studies directly applied curves twoasteps: planning the referenced flight path and curves contrast, number of studies have directly applied to includes plan flight path. Among these studies, Dubins, modifying using a Nurbs curve. Clothoid, PH it and B-splined respectively, to plan flight path.curves, Among these have studies, Dubins,
Y Y Y
( a , b, c )
( a , b, c )
( a , b, c )
( a , b, c )
O
O
X
O
V 0 (0, 0,0 a 2 b2 2 2c 2 ) 2
X
Z
Z
X
Optical axis
Optical axis
V a, 0, b c
V (0, 0, a b c )
Optical axis axis Optical
X
O 1
V 1 a, 0, b 2 c 2
ZZ
(a) (a)
2
2
(b) (b) Y
Y
V ( a , b, c )
V ( aaxis , b, c ) Optical
Optical axis X
O
O
X
[15-19]. However, the path been used to fit toPH flight paths Clothoid, and B-splined curves, respectively, have planned by Dubins curves isplanning not feasible for [15-19] flight, due to 4.1been Referenced flight path Z . However, the path used to fit to flight paths (c) its discontinuous curvature. Although Clothoid curves Figure 5. Rotating the camera to point at the target. planned by Dubins curves is not feasible for flight, due to Z have when flight applying a In continuous this work, curvature, the referenced paththese, planning for (c) its discontinuous Although Clothoid compound flight path withascurvature. low flexibility is generated in inspection is defined follows: given the coordinates of curves Figure 5. Rotating the camera to point at the target. !Invalid Character Setting. Overview of the inspection have continuous curvature, when curve applying these, a a complicated manner. Inand this paper, a Nurbs is Figure 5. Rotating the camera to point at the target transmission towers a safe inspection distance, the strategy directly applied for planning the flight path, dueinto3D its compound flight path with low flexibility is generated in shortest safe flight path can be planned space, inspection strategyOverview aims to track !Invalid Our Character Setting. of theoverhead inspectioncontinuous and uniform variations of curvature. The depending on inspection guidelines and the dynamic a complicated manner. In this paper, a Nurbs curve is 3. Overview the Inspection power linesofautomatically andStrategy keep them located at the strategy flight path planningofprocess includes two steps: planning characteristics UAVs. The path must guarantee that directly applied for planning the flight path, due to its centre of image. An overview of this process is shown in theUAVs referenced flightall path and modifying it using a Nurbs Our inspection strategy aimsoverhead to track overhead can pass transmission in order accom‐ Our inspection strategy aimsstages: to track power lines Figure 6 and includes three continuous and uniform towers variations of to curvature. The curve. power lines automatically and keep them located at the plish the inspection task. Due to the small size of UAVs automatically and keep them located at the centre of image. 1. Flight path planning: plan the flight path of UAVs to flight path planning process includes two steps: planning
centre of image. overview this process is shown Anguarantee overview ofAn this process isoverhead shown in Figure 6and and in a safe distance from of power lines Figure 6 and includes three stages: includes three stages: improve inspection quality;
(compared to overhead power lines), for analysis purposes, the referenced flight path and modifying it using a Nurbs they can be considered as particles in 3D space.
curve.
1. Flight path planning: plan the flight path of UAVs to guarantee a safe distance from overhead power lines and
X Motion space
improve inspection quality;
Y
Coordinates of the transmission line Inspection rules
Z Flight path planning
Coordinates of the transmission Dynamic line characteristics
X 3D motion
Motion space Y Z
of UAVs
Inspection rules
Pose planning for camera
Flight path planning
Pose planning for camera
Output image
3D motion
Adjust the pose of the camera
Target tracking
Dynamic characteristics 6. Overview of the overhead power line inspection. Figure 6. Overview of the overhead power lineFigure inspection of UAVs
4
Referenced flight path13:72 planning Int 4.1 J Adv Robot Syst, 2016, | doi: 10.5772/62800 In this work, the referenced flight path planning for inspection is defined as follows: given the coordinates of transmission towers and a safe inspection distance, the shortest safe flight path can be planned in 3D space,
Output UAVs (compared to overhead power lines), for analysis image purposes, Adjust thethey can be considered as particles in 3D space. Target pose of theare three reasons for planning the referenced There tracking camera flight path. First, UAVs have a maximum mission time, because they are battery powered. The shortest path
There are three reasons for planning the referenced flight path. First, UAVs have a maximum mission time, because they are battery powered. The shortest path therefore provides the benefit of reducing flight time. Secondly, inspection company guidelines state that the distance between UAVs and overhead power lines must be larger than simply a safe distance. Finally, contrary to the safe distance, the closer the UAV is to the overhead power line, the clearer the captured images will be. Therefore, it is necessary to plan a flight path that can not only guarantee a safe and effective inspection, but also ensure good images. The coordinates of transmission towers are represented in the Cartesian coordinates system as P = { p0(x0, y0, z0), ⋯ , pi (xi , yi , zi ), ⋯ , pn−1(xn−1, yn−1, zn−1)}, where n is the number of transmission towers and P is considered the initial control point of the Nurbs curve. Given a safe distance (represented by D) between UAVs and overhead power lines, the shortest safe flight path can be produced as shown in Figure 7. We begin by connecting the initial control points in turn to produce a set of line ¯ ¯ segments p¯ 0 p1, p1 p2, ⋯ , pn−2 pn−1; secondly, we move all the
line segments pertaining to distance D along the normal ¯ ¯ direction of p¯ 0 p1, p1 p2, ⋯ , pn−2 pn−1 to obtain a number of intersections points C = {c0, c1, ⋯ cn−1}, then, set C makes up
the final control points. Finally, we connect all the final control points to generate the referenced flight path as ¯ ¯ c¯ 0c1, c1c2, ⋯ , cn−2cn−1.
pn2
p n 1
pi 1
pi pi 1 p1 p1
p0
ci
ci 1
c n 2c n 1
ci 1 c1 c0
Figure 7. Generating the referenced flight path, p0, p1, ⋯ , pn−1 are initial control points, move them along the normal direction of ¯, ¯ p¯, 0 p1 p1 p2 ⋯ , pn−2 pn−1 a distance D to get c0, c1, ⋯ , cn−1as final control points, ¯, ¯ then connect them to be line segments c¯, 0c1 c1c2 ⋯ , cn−2cn−1 all of line segments ¯, ¯ c¯, 0c1 c1c2 ⋯ , cn−2cn−1 makes up the reference flight path
It must be noted that movement which illustrated in Figure. 7 from p0, p1, ⋯ , pn−1 to c0, c1, ⋯ , cn−1 is performed in the XY plane; thus, zi does not change after moving. Performing
such
movement
along the normal directions of ¯ ¯ p¯ p , p p , ⋯ , p p 0 1 1 2 n−2 n−1 has the following advantages: it
ensures that the produced control points satisfy the requirement of safe distance and that it maintains the geometric property of overhead power lines, which can contribute to capturing good quality images for targets.
4.2 Modify the referenced flight path using Nurbs curve The referenced flight path is composed of a series of line segments. If UAVs fly along it directly, this will give rise to serious effects on the stability of movement and result in a path drift. Therefore, the path should be modified to be a smooth one, where the state of motion (e.g., linear velocity, angle velocity) changes continuously to supply the control system with more accurate input. Here, the Nurbs curve is applied to modify the referenced flight path in order to remove sharp corners, thereby achieving the final flight path. The Nurbs curve is defined by (9): ì n 1 ïC(u) = n å N i , p (u)wi Pi ï å i = 0 N i , p (u)wi i = 0 ï ï ïì 1 if ui £ u £ ui + 1 , í N i ,0 (u) = í otherwise ïî0 ï ï u-u ui + p + 1 - u ï N (u) = i N (u) + N (u) i , p -1 ï i ,p ui + p - ui ui + p + 1 - ui + 1 i + 1, p -1 î
(9)
where p is degree, {Pi } are control points, {wi } are weights,
N i, p (u) are B‐spline basis functions of degree, p.
{
}
U = a, ⋯ a, u p+1, ⋯ , um− p−1, b, ⋯ b represents a knot vector p+1
p+1
and m is the number of knots. Final control points C = {c0, c1, ⋯ cn−1} are used to establish a safe flight path. 5. Tracking Overhead Power Lines Our tracking strategy aims to track the overhead power lines automatically by following the flight path planned in the previous section. Thus, the PTZ camera mounted on a platform is required for tracking overhead power lines automatically. According to camera projection theory [20‐21], the target will be projected at the centre of the image when the optical axis points at it. To achieve this, two operations are carried out for tracking the overhead power lines: 1.
Plan the pose of camera based on the planned safe flight path and the position of overhead transmission towers;
2.
Track the overhead power lines by applying visual servoing techniques to maintain targets at the centre of the image.
5.1 Pose planning for the camera We use a six‐value vector S(p(x,y,z), l(i,j,k)) to represent the pose of the camera, where p(x,y,z) and l(i,j,k) describe the position of the camera and the direction of the sightline. In this paper, p(x,y,z) is assumed as a coincident with the coordinates of UAVs along the flight path and l(i,j,k) is Chang-an Liu, Ruifang Dong, Hua Wu, Guo-tian Yang and Wei Lin: A 3D Laboratory Test-platform for Overhead Power Line Inspection
5
taken as the normal vector of the flight path. This can ensure that the optical axis points at the target. The normal vector of Nurbs can be obtained from equations (10~12) [22]. It must be noted that the direction of the normal vector is only considered in the XY plane; thus, so uz =0. Given the
direction of sightline l(x,y,z), yaw (α) and pitch (β) can be obtained according to the method introduced in section 2.2.2. æ ux ö ç ÷ C '(u) t(u) = ç uy ÷ = , ç u ÷ C '(u) è zø n n n n ì å N 'i , p (u)wi Pi å N i , p (u)wi - å N i , p (u)wi Pi å N 'i , p (u)wi ï i =0 i =0 i =0 i =0 '( ) C u = ï 2 æ n N (u)w ö ï ç å i ,p , i÷ í 0 i = è ø ï u u ï N ' (u) = (u) (u) N N ï i ,p ui + p - ui i , p -1 ui + p + 1 - ui + 1 i + 1, p -1 î
æ -uy ö ç ÷ n(u) = ç ux ÷ . ç 0 ÷ è ø
(10)
(11)
4.
Overhead power lines are approximately parallel to each other and their shape is similar, i.e., they are parabolas moving upwards.
The above characteristics can be applied as knowledge for building overhead power line extraction algorithms. In this work, Hough transform is used to detect the parabolas in image, in further to extract the power lines in image. Compared to the colour‐based method, detecting the parabolas can overcome the disturbances cause by a complicated background.
(
E = sin q ( x - h ) + cosq ( y - k )
)
2
(
)
- a cosq ( x - h) - sin q ( y - k ) = 0,
(13)
where a is curvature (h, k) is the vertex point of the parabola and θ is rotational angle. (12)
Ideally, planning the camera pose intends to make the optical axis point at targets; however, it cannot always ensure that targets are located at the centre, due to some sources of error in real applications. For example, the surveying error of transmission tower position in the map. Additionally, UAVs are subject to short‐term random motion due to air turbulence (e.g., wind gusting), where the sightline will ‘jitter’ due to the imbalance of the platform. In our work, visual tracking is applied to help the camera track overhead power lines. Two operations are carried out in the tracking process: target location in images and visual tracking. Target location aims to find the position of overhead power lines in an image. Hough transform is a popular way for detecting overhead power lines; this is effected by detecting straight lines [14,23,24]. The overhead power lines are projected as straight lines because the image is captured just above the power lines. However, considering safety, it is preferable that the UAV flies alongside the overhead power lines. In fact, when viewed from the side, power lines are parabolas; therefore, when flying alongside power lines, they can be projected as parabolas. The overhead power lines in aerial images have the following character‐ istics: 1.
They are generally the longest line to cross the entire image between two towers;
2.
The overhead power line is made from a special metal that has a uniform brightness in aerial images that is usually brighter than image backgrounds [25];
Int J Adv Robot Syst, 2016, 13:72 | doi: 10.5772/62800
The background is always complicated such as forest, fields, grassland, river, cities, etc.
Considering the projection character of a parabola, the rotation factor is applied to the Hough transform as (13)[26]:
5.2 Visual tracking for the camera
6
3.
Similar to detect straight lines using Hough tranform, we set a four‐dimension accumulator array Acc(a,θ,h,k) to be voted based on edge points, in further to compute the parameters (a,θ,h,k). However, it will take a significant time to search within a four‐dimension array while the inspec‐ tion requires real‐time data. [27] decreased the searching dimensions by adding information about the gradient directions of edges. Although computing time can in this way be decreased, this approach only works when the symmetry axis of the parabola is parallel to the coordinate axis, it cannot be used in overhead power line detection. Considering the features of overhead power lines, comput‐ ing time can be decreased by reducing the search range of the accumulator array. The detailed tracking process can be given as follows: 1.
Initial acquisition: the camera is pointed in such a way that overhead power lines appear somewhere in the field of view; the observer specifies the line to be inspected so that an F (θ0, a0, h 0, k0) can be initialized;
2.
Edge detection: Canny Edge Detection algorithm is applied to detect all edges in the image;
3.
Overhead power line extraction and tracking: the overhead power lines can be extracted based on their characteristics, e.g., their shape being similar (they all move upwards). It must be noted that when the inspection is performed according to the planned pose of camera, the variation of θ and a must be small. This means that the size of a‐dimension and θ‐dimension in array Acc(a,θ,h,k) will be reduced. In our work, a particle filter [28‐29] is employed to track the parame‐ ters (θ, a, h , k ) in Hough space, thus reducing the search time. Each pixel coordinate of edge point is plugged into parabola expression (13) to vote
Acc (θ, a, h , k ). The parabola can be obtained by finding the peaks of Acci (θi , ai , h i , ki ). Once the parab‐
Start
ola lines have been detected, the overhead power lines can be located. The PTZ camera we use is the VIVOTEK: PZ7121. This camera can steer automatically to make target at the centre of image once accepts the pixel coordinate of target in image. Therefore we just need to measure the position of target in image as described above. The entire visual tracking process is shown in Figure 8.
Initialize the pose of the camera to make the target in the center of the image
Read video
6. Experiment
Yes Is the first frame?
Our 3D test‐platform allows for the inspection strategy to be demonstrated and the behaviour of the system to be quantified. Figure 9 shows our 3D test‐platform. It provides a 25:1 scale model of a 150m span of overhead power line, which consists of four power towers. It also supplies significant clutter in the form of backgrounds including a desert, city, grassland, field, industry buildings, etc. The 3D motion module, which acts as a substitute for UAVs, moves in a 6m*2m*2m aluminium track located above the scenery. A PTZ camera (VIVOTEK PZ7121) is mounted at the end of the Z axis, as shown in Figure 10. A PC serves as a ground station.
No Canny edge detection
Hough transform
Particle filter
Obtained the location of the target
6.1 Tracking algorithm in the video sequence To verify the effectiveness of our power line tracking algorithm in a video sequence, six complex backgrounds (classified into ground backgrounds and backgrounds that are parallel to camera views) are captured. In addition, different lighting conditions are also considered. These scenes make tracking the overhead power lines more challenging. A correct tracking rate and a complete tracking rate are measured to evaluate tracking performance. The correct tracking rate indicates the number of frames that power lines are tracked during the sequence. The complete tracking rate indicates the number of frames that complete power lines are tracked within all of the frames which power lines are tracked. Figures 11‐12 and Table 1 show the tracking results for different backgrounds. Figures 11‐12 show that overhead power lines can be tracked successfully in the case of six complex backgrounds. The correct and complete tracking rate for the green grass, desert, mountain/field and indus‐ try building scene are better than for the city and residence buildings scenes. This is because parts of the city and residence building scenes have similar colours with power lines (especially the case for roads and residential build‐ ings), which misleads the canny. The Hough transform relies on edge detection and as such, a good edge detection approach is beneficial. Thus, in the real scene, where power lines are directly above roads, residential buildings or other elements with similar colours to that of the power lines will render tracking more challenging.
Initialize F
Is the target in the center of image?
Yes
No Adjust the pose of the camera
No
Is the last frame? Yes
End Figure 8. Diagram of visual tracking for power lines
In addition, in Figure 12(c) and (g) ‐(i), we can see that two power lines are not parallel due to different tension, some parts of the behind power line are occluded by the front one to camera; however, these cables could still be successfully tracked. Figure13 and Table 2 show the tracking results in different lighting conditions. Videos are captured according to three lighting conditions ranging from bright to dark for the green grass and city backgrounds. Figure 13 shows that power lines can be tracked successfully in different lighting Chang-an Liu, Ruifang Dong, Hua Wu, Guo-tian Yang and Wei Lin: A 3D Laboratory Test-platform for Overhead Power Line Inspection
7
Figure 9. Photograph of the 3D test‐platform
Figure 10. PTZ camera mounted at the end of the Z axis
conditions. Table 2 shows that the correct tracking rate and complete rate for bright lighting conditions are better than for darker lighting conditions. The darkest lighting condi‐ tion has the least correct tracking rate, as well as and the least complete rate.
segments (blue lines shown in Figure 14). Then, by moving all of line segments following the steps in section 4.1, the final control points can be obtained as points ①~④ and the reference flight path can be generated (the black lines shown in Figure 14(b)).
In conclusion, power lines can be tracked successfully according to different backgrounds and lighting conditions.
Step 2: smooth the reference flight path using Nurbs curves to obtain the final flight path.
6.2 Whole inspection strategy
Figure 14(a) shows the results planned in 3D space, while Figure 14(b) provides an overhead view of said results. We can see that sharp corners (at points ② and ③ in Figure.14 (b)) in the reference flight path are smoothed by Nurbs curves. Figure 14(a) shows the curvature of the reference flight path; two dramatic changes happen at points ② and ③, which results in the reference flight path not being able to fly. In contrast, the curvature of the final flight path is
To verify our entire inspection strategy, the experiment is performed as follows: Step 1: plan the reference flight path according to our algorithm described in section IV; Taking the coordinates of transmission towers as initial control points, connect them in turn to form a set of line 8
Int J Adv Robot Syst, 2016, 13:72 | doi: 10.5772/62800
Figure 10. PTZ camera mounted at the end of the Z axis. Figure 9. Photograph of the 3D test-platform.
(a) Frame 39.
(b) Frame 113.
(c) Frame 203.
(d) Frame 69.
(e) Frame 199.
(f)
Frame 416.
(g) Frame 40.
(h) Frame 130.
(i)
Frame 339.
Figure 11. Vertical view sequence shots
Figure 11. Vertical view sequence shots.
(a) Frame 40.
(b) Frame 206.
(c) Frame 320.
(d) Frame 63.
(e) Frame 218.
(f)
Frame 487.
(g) Frame 31.
(h) Frame 95. Figure 12. Horizontal view sequence shots.
(i)
Frame 141.
Figure 12. Horizontal view sequence shots
Table 1. Tracking performance for different backgrounds. Background
ground Parameter
Parameter
Green grass Green grass
City City
Desert Desert
Mountain Residential Industry and field building Mountain and field Residential building
Industry
Tracking correct rate
96.7%
85.3%
95.4%
97%
78%
93.5%
Tracking complete rate
94.3%
74.6%
94.7%
95.5%
70%
94%
Table 1. Tracking performance for different backgrounds
Chang-an Liu, Ruifang Dong, Hua Wu, Guo-tian Yang and Wei Lin: A 3D Laboratory Test-platform for Overhead Power Line Inspection
9
(a). Bright lighting.
(d). Bright lighting.
(f). Darkest lighting.
(e). Dark lighting
Figure 13. Different illumination sequence shots
Figure.13 Different illumination sequence shots.
City
Table 2. Tracking performance for different lighting conditions. Green grass
Illumination Parameter
(c). Darkest lighting.
(b). Dark lighting.
Illumination
City Bright
Dark
Darkest
Bright
Dark
Darkest
Tracking correct rate
96.7%
89%
83.4%
85.3%
78.5%
69.9%
Tracking complete rate
94.3%
86.7%
80.6%
74.6%
70.3%
64%
Parameter
Table 2. Tracking performance for different lighting conditions
160
The initial control points The reference flight path The flight path fitted by Nurbs
140 16 14
120
12
Z(m)
10
100
8
Y(m)
6
80
4
The initial control points The reference flight path The flight path fitted by Nurbs
2 0 150
60
40 150 100 100
20
50 50 0
Y(m)
0
0
X(m)
0
50
100
150
X(m)
(a) The flight path in 3D space.
Figure 14. The planned flight path
(b) A vertical view of the flight path in the XY plane. Figure 14. The planned flight path.
0
0 -0.2
-0.2
-0.4 -0.4 -0.6 -0.6
-0.8
-0.8
-1 -1.2
-1
-1.4 -1.2 -1.6 -1.4 -1.6
-1.8
0
50
100
150
200
250
300
350
(a) Curvature of the reference flight path.
-2
0
50
100
150
200
250
300
350
(b) Curvature of the flight path fitted by Nurbs curves.
15. The curvature Figure 15. The curvature of reference flight pathFigure and final flight path of reference flight path and final flight path.
continuous and almost without changes (as shown in Figure 14 (b)); when UAVs fly along it, a continuous status 10
Int J Adv Robot Syst, 2016, 13:72 | doi: 10.5772/62800
of motion (angle velocity and line velocity) can therefore be guaranteed.
Figure 16. Visual tracking for camera
(1) Frame 1.
(2) Frame 96.
(3) Frame 120.
(4) Frame 192.
(5) Frame 218.
(6) Frame 318.
(7) Frame 430.
(8) Frame 456.
(9) Frame 487.
Figure 16. Visual tracking for camera.
Figure 16(1-3) shows the tracking results after initializing roughly 200 frames. We can see that power
Step 3: initialize parameters as introduced section 5.2 and lines are located at the centre in of images at all times and that following they have beenthe tracked successfully. In Figure 16(4), perform the inspection path planned in the the power lines start to shift down from the centre, due to previous step. Figurethe 16perturbation shows sequence shots module of tracking of the 3D motion (vibration the camera to perturbthat the sightline). Figure 16(5) results, the red lines incauses images indicating the overhead shows power lines shifting to the bottom of image; power lines have been tracked. however, by applying the tracking strategy presented in this paper, the target returns to the centre, as shown in
16(6). In Figure 16(7), power shift up to the Figure 16(1‐3) showsFigure the tracking results afterlines initializing top of the image and after a few frames, the power lines roughly 200 frames. We can see that power lines are located return back to the centre, as in Figure 16(8-9). This indicates when and a target strays from have the centre of the at the centre of images at allthat times that they been our camera can automatically steer the target back tracked successfully. image, In Figure 16(4), the power lines start to the image's centre. Overall,due the 3D designed in this paper to shift down from the centre, totest-platform the perturbation of the is able to simulate UAVs' inspection of overhead power 3D motion module (vibration causes the camera to perturb lines by carrying out the strategy presented in this paper. It can therefore concluded thatshifting the target is the sightline). Figure 16(5) showsbepower lines totracked the successfully by applying this inspection strategy. bottom of image; however, by applying the tracking Character Setting. Conclusions strategy presented in!Invalid this paper, the target returns to the This paper presents a 3D laboratory test-platform for centre, as shown in Figure 16(6). In Figure 16(7), power lines simulating the inspection of overhead power lines using shift up to the top of the image and after a few frames, the power lines return back to the centre, as in Figure 16(8‐9). This indicates that when a target strays from the centre of the image, our camera can automatically steer the target back to the image's centre.
Overall, the 3D test‐platform designed in this paper is able to simulate UAVs' inspection of overhead power lines by carrying out the strategy presented in this paper. It can therefore be concluded that the target is tracked success‐ fully by applying this inspection strategy. 7. Conclusions This paper presents a 3D laboratory test‐platform for simulating the inspection of overhead power lines using UAVs. This platform can be used to develop, test and assess the performance of inspection strategies. Experimental results are presented to support the concept. An inspection strategy is proposed for UAVs to track overhead power lines. First, a safe reference flight path is
UAVs. This platform can be used to develop, test and assess the performance of inspection strategies. planned based position of power Experimental results on are the presented to support the towers; then, sharp concept. corners in the referenced flight path are smoothed by Nurbs An inspection strategy is proposed for UAVs to track curves. Thelines. final path flight has path continuous and uniform overhead power First,flight a safe reference is planned based onof thecurvature, positioning of power towers; then, variations which is beneficial for supplying sharp corners in the referenced flight path are smoothed the control system with accurate by Nurbs curves. The final flight path more has continuous and inputs. Secondly, a uniform of curvature, whichisisapplied beneficial for visualvariations tracking technique to track the target. Our supplying the control system with more accurate inputs. parabola detection algorithm, based Secondly, a visual tracking technique is applied to trackon the Hough trans‐ the target.is Our parabola detection algorithm, based on power lines and a form, employed to detect overhead the Hough transform, is employed to detect overhead particle filter is used to track thethe specific parameters. power lines and a particle filter is used to track specific parameters. verified Experiments verified the Experiments that the that algorithm is able to track algorithm is able to track power lines against complex power lines against complex backgrounds and in different backgrounds and in different lighting conditions. Furthermore, the power lineFurthermore, strays from the centre lighting once conditions. once the power line of the image, the camera can automatically steer the strays theExperimental centre ofresults the demonstrate image, the camera can auto‐ image backfrom on target. that power lines can the be tracked matically steer imagesuccessfully back onbytarget. Experimental employing our approach. results that power lines can be tracked We can demonstrate therefore draw the conclusion that the 3D test-platform presented this paper is able simulate successfully by in employing ourtoapproach.
We can therefore draw the conclusion that the 3D test‐ platform presented in this paper is able to simulate UAVs' inspection for overhead power lines and that the strategy proposed for tracking overhead power lines based on the accompanying visual technique is effective and feasible. 8. References [1] S. Bjarnadottir, Y. Li and M. G. Stewart, “Risk‐ based economic assessment of mitigation strategies for power distribution poles subjected to hurri‐ canes,” Structure and Infrastructure Engineering, 10(6):740‐752, 2014. [2] G. B. França, A. N. de Oliveira, C. M. Paiva, L. D. F. Peres, M. B. da Silva and L. M. T. de Oliveira, “A fire‐ risk‐breakdown system for electrical power lines in the North of Brazil,” Journal of Applied Meteorolo‐ gy and Climatology, 53(4):813‐823, 2014. [3] Group Conference. 2004: San Diego.3. Russell, B.D., et al., "Reliability Based Vegetation Management Chang-an Liu, Ruifang Dong, Hua Wu, Guo-tian Yang and Wei Lin: A 3D Laboratory Test-platform for Overhead Power Line Inspection
11
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
12
through Intelligent System Monitoring." In Power Systems Engineering Research Center. 2007, Texas A&M University: Texas. L. Dai, J. Qi, J. and Han et al., “Camera selection for unmanned helicopter power line inspection.” In Proceedings of the IEEE Innovative Smart Grid Technologies—Asia. ISGT Asia, 12:1‐4, 2012. G. Dong, X. Chen and B. Wang et al., “Inspecting overhead power lines with an unmanned fixed‐ wings aircraft.” In Proceedings of the 2nd Interna‐ tional Conference on Applied Robotics for the Power Industry. CARPI, 12:173‐174, 2012. C. Liu and L. Wang, “Mission planning of the fling robot for powerline inspection,” Progress in Natural Science, 19(10):1357‐1363, 2009. J. Katrašnik, F. Pernuš and B. Likar, “A survey of mobile robots for distribution power line inspec‐ tion,” IEEE Transactions on Power Delivery, 25(1): 485‐493, 2010. B. Yang, Y. He, J. Han and G. Liu, “Rotor‐flying manipulator: modeling, analysis and control,” Mathematical Problems in Engineering, vol. 2014, Article ID 492965, 13 pages, 2014. L. F. Luque‐Vega and B. Castillo‐Toledo et al., “Power line inspection via an unmanned aerial system based on the quadrotor helicopter.” In Proceedings of the IEEE 17th Mediterranean Electrotechnical Conference, pp. 393‐397, 2014. D. I. Jones, C. C. Whitworth, G. K. Earp and A. W. G. Duller, “A laboratory test‐bed for an automated power line inspection system,” Control Engineering Practice, 13(7):835‐851, 2005. D. I. Jones, “Aerial inspection of overhead power lines using video: estimation of image blurring due to vehicle and camera motion.” Proceedings of the IEE Vision, Image and Signal Processing, 147(2): 157‐166, 2000. Ming Lu, Gehao Sheng, Yadong Liu, Xiuchen Jiang, “Research on Autotracking Algorithm for Power Line Inspection Based on Unmanned Aerial Vehi‐ cle.” Power and Energy Engineering Conference (APPEEC), Asia‐Pacific. pp. 1‐5, 2012. Zhengrong Li and Yuee Liu et al. “Towards auto‐ matic power line detection for a UAV surveillance system using pulse coupled neural filter and an improved Hough transform”. Machine Vision and Applications, 21(5):677‐686, 2010. Jingjing Zhang and Liang Liu et al., “High speed Automatic Power Line Detection and Tracking for a UAV‐Based Inspection” International Conference on Industrial Control and Electronics Engineerin, pages: 266‐269. 2012. Madhavan Shanmugavel et al., "Cooperative path planning of multiple UAVs using Dubins paths with clothoid arc." Control engineering practice, 18:1084‐1092, 2010
Int J Adv Robot Syst, 2016, 13:72 | doi: 10.5772/62800
[16] Yucong Lin and Srikanth Saripalli., “Path planning using 3D Dubins Curve for Unmanned Aerial Vehicles”, on Unmanned Aircraft Systems (ICUAS), 2014 International Conference. Orlando, FL, USA. May 27‐30, 2014, pp. 296‐304. [17] I. K. Nikolos et al. "Evolutionary algorithm based offline /online path planning for UAV Navigation." IEEE transactions on systems, man, and cybernetics – part B: Cybernetics, 33(6): 898‐912, 2003. [18] Wang Yi, Zhu Xiaoping et al., "Path Planning Based on PH Curves for Unmanned Aerial Vehicles". Journal of Computer Simulation, 30(3):76‐79, 2013. [19] Ee‐May Kan and Meng‐Hiot Lim et al., "Contour Based Path Planning with B‐Spline Trajectory Generation for Unmanned Aerial Vehicles (UAVs) over Hostile Terrain". Journal of Intelligent Learn‐ ing Systems and Applications, 3:122‐130, 2011. [20] F. Ackermann, "Digital image correlation: perform‐ ance and potential application in photogrammetry." The Photogrammetric Record, 11(64):429‐439, 2006. [21] Chang‐an Liu, Rui‐fang Dong and Hua Wu., "Flying Robot Based Viewpoint Selection for the Electricity Transmission Equipment Inspection". In Mathe‐ matical Problems in Engineering, 2014:1‐10, 2014. [22] S. Elgeti and H. Sauerland et al., "On the usage of NURBS as interface representation in free‐surface flows". On International Journal for Numerical Methods in Fluids, 69(1):73‐87, 2012. [23] Qirong Ma., “Power Line Detection for Millimeter‐ Wave Radar Video”. Thesis (Ph.D.) – University of Washington, 2012. [24] Guangjian Yan and Chaoyang Li et al. “Automatic Extraction of Power Lines from Aerial Images”. IEEE Geoscience and Remote Sensing Letters, 4(3): 387‐391, 2007. [25] Zhengrong Li and Yuee Liu et al., “Knowledge‐ based Power Line Detection for UAV Surveillance and Inspection Systems”. 23rd International Confer‐ ence on Image and Vision Computing New Zea‐ land, IVCNZ:1‐6, 2008. [26] Young Kyoon Jang and Byung Jun Kang et al., “A study on eyelid localization considering image focus for iris recognition”. Pattern Recognition Letters, 29: 1698‐1704, 2008. [27] H. Wechsler and J. Sklansky. “Finding the rib cage in chest radiographs.” Pattern Recognition, 9:21‐30, 1977. [28] Branko Ristic, Sanjeev Arulampalam and Neil Gordon.” Beyond the Kalman Filter: Particle Filters for Tracking Applications”. Publisher: Artech Print on Demand (31 January 2004). [29] A. Hafez and E. Cervera., "Particle‐filter‐based Pose Estimation from Controlled Motion with Applica‐ tion to Visual Servoing." In International Journal of Advanced Robotic Systems, 11:177, 2014.
