Navigating a Miniature Crawler Robot for Engineered Structure ...

368

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 2, APRIL 2008

Navigating a Miniature Crawler Robot for Engineered Structure Inspection Weihua Sheng, Hongjun Chen, and Ning Xi

Abstract—This paper addresses the problem of how to navigate a miniature Crawler robot in a typical engineered structure inspection application—aircraft rivet inspection. First, a novel vision-assisted localization algorithm is developed to find the heading and position of the Crawler robot. Second, a new algorithm is developed to solve the path planning problem so that the Crawler robot can navigate through all the rivets. Experimental results validate the localization and path planning algorithms. This inspection system can be extended to other similar engineered structure inspection applications. Note to Practitioners—This paper was motivated by the problem of automated aircraft rivet inspection. Currently, a new nondestructive inspection technology has been developed for deep layer crack detection around the rivet area on an aircraft body. However, the efficiency and accuracy of the inspection system can not achieve the maximum if the inspection is manually conducted by human operators. Therefore, there is a need for automated rivet inspection. This paper develops a mobile sensor that integrates the sensor and a micromobile robot. The control, localization and path planning problems are solved so that the micro mobile robot can navigate on the airplane body surface to carry out rivet inspection without human involvement. This mobile robot based inspection system can be extended to different engineered structure inspection applications by integrating different sensors. The use of vision-assisted localization method requires that the position information of the inspected subject, such as rivets, should be known from a source such as a CAD model. Also, the surface of the structure to be inspected should be reasonably smooth to enable the robot to stick to it. Our future work will integrate multiple mobile sensors to speed up the inspection process. Index Terms—Climbing robot, localization, mobile sensor, nondestructive inspection (NDI), path planning.

I. INTRODUCTION A. Motivation For many civil, mechanical and aerospace structures, structural health monitoring (SHM) [1], [2] is very critical to maintaining their performance and preventing potential failures or disasters. For example, the body of most commercial aircraft has multiple rivets connecting multilayer skins. Because of the fatigue stress caused by the typical application of a commercial aircraft, cracks can be induced in the vicinity of rivet holes. These cracks should be detected as early as possible for necessary corrections [3]. In recent years, various sensing technologies and sensors for structural deformation or damage detection have been developed [4]. For example, a new nondestructive inspection (NDI) technique called flat geometry remote field eddy current (FG-RFEC) has been developed by Innovative Materials Testing Technologies, Inc. (IMTT, Inc.) [5] to Manuscript received April 24, 2006; revised December 23, 2006 and July 2, 2007. This paper was recommended for publication by Associate Editor M. Wang and Editor K. Bohringer upon evaluation of the reviewers’ comments. W. Sheng is with the School of Electrical and Computer Engineering, Oklahoma State University, Stillwater, OK 74078 USA (e-mail: weihua.sheng@ okstate.edu). H. Chen is with the Department of Electrical Engineering, Harbin Institute of Technology, Harbin 150001, China. N. Xi is with the Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI 48824 USA. Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TASE.2007.910795

accurately detect deep cracks around the rivets in multilayer aircraft structures with high sensitivity. However, these probes are mainly used in a manual way, where human operators carry the probes to conduct inspection. Therefore, the accuracy and efficiency of the inspection are greatly affected by the experience of the operator. To solve there problems, we propose to develop an automated inspection system using mobile crawler robots. B. Related Work Previous research in structure inspection using robotic devices has resulted in several prototypes [6]–[9]. In recent years, interests in the development of miniature climbing robots are growing rapidly [10], [11]. Siegel [12] gives a survey on the climbing robots used in aerospace structure inspection. These robots are usually small and have suction cups that enable them to climb on aircraft skins. The ROSTAM series robots developed by Benham et al. [13] aim to carry NDI sensors and video cameras for aircraft skin inspection. Seemann et al. [14] developed a tank-like, multisuction-cup-tracked robot called AutoCrawler for aircraft skin inspection. NASA JPL’s (Jet Propulsion Laboratory) MACS series robots [15] are small, lightweight mobile platforms that use suction cups for attachment and ultrasonic motors for motion. The Automated Nondestructive Inspector (ANDI) robot developed by Siegel et al. [12] was used for commercial aircraft skin inspection. Most of the robotic devices developed so far are remotely controlled. The lack of effective navigation capability greatly limits the scope of their applications. Our goal is to develop a mobile sensor capable of autonomous navigation for structure inspection applications. The focus of this paper is on how to navigate such a mobile sensor on an aircraft body. This paper is organized as follows. In Section II, we introduce the Crawler robot used in the inspection and its pose error model. Section III investigates the localization, or pose estimation problem. Section IV studies the path planning problem raised in the autonomous navigation. Experimental results are presented in Section V and Section VI concludes this paper. II. THE CRAWLER ROBOT AND ITS POSE ERROR MODEL A. The Crawler Robot The mobile sensor consists of a miniature FG-RFEC probe and a Crawler robot. Fig. 1 shows the prototype of the mobile sensor. The Crawler [16] is a bipedal robot with an underactuated mechanism to achieve a good balance between compactness and maneuverability. It is supported by two suction feet that can hold on anticipated smooth and nonporous surfaces. Each of its three joints is driven by a DC servomotor. Onboard touch sensors and pressure sensors are used for suction cup control. For localization, a CCD camera is mounted under the belly of the Crawler. There are two wireless communication channels. One is a low data rate channel to receive motion command from and return motion status to a remote host computer. The other is a high data rate channel to transfer camera view data to the remote PC. The dimension of the Crawler is about 80 mm 2 50 mm, and its weight is about 450 g. Due to the limited space, readers are referred to [16] for the dynamic model and control of Crawler. B. Robot Pose Error Model Before studying robot localization, we should analyze the error introduced in the rotation and translation movement of the Crawler robot. As shown in Figs. 2 and 3, the amount of rotation and translation in each step is limited. We assume the step size for rotation is s and the step size for translation is ds . For each rotation step, the rotation error can

1545-5955/$25.00 © 2007 IEEE Authorized licensed use limited to: Oklahoma State University. Downloaded on April 16,2010 at 16:27:17 UTC from IEEE Xplore. Restrictions apply.

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 2, APRIL 2008

369

Fig. 4. Crawler moves from rivet a to rivet b but ends at b due to the angle error and the position error. Fig. 1. The prototype mobile sensor. Note that the FG-RFEC probe is not mounted yet in this picture.

Fig. 5. Position error accumulated over a sequence of rivets.

As shown in Fig. 5, the robot position error at any rivet along a path can be calculated as

1di = Fig. 2. The angle error in the rotation movement.

i j =0

02

djj +1 djj +1

2 cos 3

+3

+3 j p=1

2

djj +1 t ds djj +1 t ds

p  s

+ djj2 +1

djj +1

1=2 :

(2)

The above equation shows that the position error significantly increases as the Crawler robot makes more turns and translations. The longest straight line distance that a Crawler can move without getting lost as the threshold distance is designated as dr .

Fig. 3. The position error in the translation movement.

III. VISION-ASSISTED ROBOT LOCALIZATION be modeled as a random error with a zero-mean Gaussian distribution. i.e., s  N ;  2 . To turn an angle of  , the Crawler needs to make a total of d =s e independent rotations, where dxe represents the minimum integer that is greater than or equal to x. Therefore, we have   N ; d =s e 1  2 . For each translation step, the translation error can also be modeled as a zero-mean Gaussian distribution, i.e., ts  N ; t 2 . To translate a distance of d, the crawler needs to make a total of d d=ds e independent translation steps. Similarly, we have t  N ; d d=ds e 1 t 2 . As shown in Fig. 4, when the Crawler moves from rivet a to rivet b, the position error caused by the rotation error and translation error is

1

(0 ) ( ) ) 1 (0 ( ) 1 (0 ) ( ) 1 (0 ( ) )

1dab = (dab + 1tab )2 + dab2 0 2(dab + 1tab )dab cos(1):

(1)

As shown in the above section, the Crawler robot has limited accuracy in positioning itself over a long distance because the position error grows as the travel distance increases. If the robot makes any turn, the position error will be significantly larger. Such position errors are undesirable. Therefore, a pose estimation and correction process is needed to find the position and heading of the Crawler robot. As in many engineered structures, each rivet has its exact geometric coordinate stored in the aircraft CAD model. Therefore, we develop an effective vision-assisted pose estimation approach. In this method, the motor encoders on the Crawler are used to provide rough pose estimation, while the onboard camera is used to provide fine pose estimation using the rivets as landmarks. Consider the movement of the Crawler between two consecutive rivets a and b. Initially, the center of the camera is right above rivet a and the heading of the Crawler is along the line connecting a and b. The pose estimation process consists of two steps: heading estimation and position estimation. Depending on the distance between

Authorized licensed use limited to: Oklahoma State University. Downloaded on April 16,2010 at 16:27:17 UTC from IEEE Xplore. Restrictions apply.

370

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 2, APRIL 2008

IV. PATH PLANING A. Problem Statement The vision-assisted pose estimation approach discussed above requires that the distance between two consecutive rivets on the Crawler’s path be less than the threshold value dr , beyond which the Crawler robot may not be able to find the target rivet. A typical dr is 15 in. It also requires the number of turns that the Crawler makes along the path that should be minimized, because it usually takes much longer time for a Crawler to make a turn than to move straight forward [17]. A formal statement of the single Crawler robot path planning problem is as follows. Given a set of n rivets, P = fp1 ; p2 ; . . . ; pn g, in a 2-D space with known locations f(x1 ; y1 ); (x2 ; y2 ); . . . ; (xn ; yn )g, find a sequence of the rivets so that a Crawler, starting from rivet s, can travel through all the n rivets at least once and get back to s. The sequence should satisfy the following requirements: 1) the distance between any two consecutive rivets is less than the localization threshold dr ; 2) the number of turns is small; and 3) the overall distance traveled is short.

Fig. 6. Estimating the pose using two rivets in the camera view.

Fig. 7. Estimating the robot pose using two one-rivet images.

the two consecutive rivets, we consider the following two cases: 1) two or more rivets are in a camera view and 2) only one rivet is in a camera view. 1) Two or More Rivets in a Camera View: When the distance between two consecutive rivets is short, two or more rivets can be viewed by the camera, as shown in Fig. 6, where (x1 ; y1 ) and (x2 ; y2 ) are the coordinates of the center of two rivets a and b in the camera view. The rivets can be recognized through image segmentation. It is straightforward to estimate the heading of the Crawler. The angle between the line connecting the two rivets and the horizontal line is the heading angle which can be calculated as  = arctan(y1 0 y2 =x1 0 x2 ). The position of the Crawler can be easily calculated based on the image and the known positions of the rivets. 2) One Rivet in a Camera View: When the distance between two consecutive rivets is bigger than the camera view, only one rivet is captured in the camera view. In this case, the distance from the encoders can help. To estimate the heading, the Crawler captures two images when it reaches rivet b. One image is captured when the Crawler stops and another image is captured one small step before it stops. When the two images are captured, the Crawler also records the corresponding encoder readings. In Fig. 7, c1 and c2 are the center of the images; Lab is the distance between the two rivets; Le is the distance calculated from the encoder reading corresponding to c1 ;  is the heading angle;  and are the internal angles as indicated in the figure. The heading angle  can be calculated as follows:

 =  0 = arccos jbc1 j

2

jc1 c2 j2 0 jbc2 j2 2jbc1 jjc1 c2 j 2 2 2 + Lab 0 Le : 0 arccos jbc1 j2jbc 1 jLab

+

(3)

Once the heading angle  is known, the position of the Crawler can be calculated based on the image and the known positions of the rivet. Then, the heading and position can be used to correct the pose of the Crawler. It can be seen that the heading and position error is limited between two rivets and will not accumulate to the next rivet along the path.

We name the above path planning problem the Crawler Traveling Salesman Problem (Crawler-TSP). The problem of traveling a set of points with turn concerns has been studied by several researchers [18]–[20]. Recently, Stein and Wagner [21] developed approximation algorithms for the Minimum Bends Traveling Salesman Problem (MBTSP), which minimizes the number of turns in the tour. However, in their work, it is assumed that there exists a path between any pair of points. This assumption does not hold in our Crawler-TSP because the Crawler is not able to travel between a pair of rivets that are more than distance dr away. Their algorithm does not minimize the overall length either. B. The Find-Crawler-TSP Algorithm The basic idea of our Find-Crawler-TSP algorithm is as follows. First, the minimum number of line segments that partition all the rivets are obtained, with no rivet covered by more than two line segments. It is clear that the number of line segments determines the number of turns a Crawler makes. Second, the minimum set of line segments are connected with a goal to minimize the overall traveling distance. The minimum set of line segments that partition all the rivets can be found by modeling it as a set-partition-problem [22]. To connect the resulted line segments, we first construct a segment graph, where each vertex represents a line segment, while the edge between two vertices represents the existence of connecting lines between the two corresponding line segments. The edge weight is the arithmetic average of the length of all the connecting lines between the two line segments. Christofides’ algorithm [23] is adopted to find the shortest tour on the segment graph. Once the order of line segments is determined, starting from rivet s, a rivet sequence can be found, which enables the Crawler to tour all the unvisited rivets in the segment that covers s and end up at a rivet in the next line segment with a minimum tour distance. This can be done by enumerating the different traveling possibilities between two consecutive line segments. The entering rivet in the next line segment becomes the starting rivet for the next round of this process, until all the rivets are visited and the Crawler goes back to rivet s. By applying a shortcut on the generated tour, a final tour can be obtained. Fig. 8 shows an example path generated by the Find-Crawler-TSP algorithm. V. EXPERIMENTAL TESTS The prototype rivet inspection system is shown in Fig. 9. The CMOS camera is black and white with a resolution of 380 TV lines and a field

Authorized licensed use limited to: Oklahoma State University. Downloaded on April 16,2010 at 16:27:17 UTC from IEEE Xplore. Restrictions apply.

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 2, APRIL 2008

371

TABLE I ESTIMATING THE HEADING ANGLE: TWO RIVETS IN CAMERA VIEW

TABLE II ESTIMATING THE POSITION: TWO RIVETS IN CAMERA VIEW

Fig. 8. The final tour.

Fig. 11. Pictures taken at the first rivet, at the second rivet before and after Crawler stops.

Fig. 9. The rivet inspection experiment system.

TABLE III ESTIMATING THE HEADING ANGLE: ONE RIVET IN CAMERA VIEW

TABLE IV ESTIMATING THE POSITION: ONE RIVET IN CAMERA VIEW

Fig. 10. Picture used in robot pose estimation: two rivets in camera view.

of view of 45 mm 2 34 mm. The host computer first runs the path planning algorithm, then it sends the position information of the next rivet to the Crawler robot. The Crawler robot receives the next target information and moves to it. Upon arriving at the target rivet, the Crawler sends the images back for vision-assisted localization. A. Localization We conducted the robot localization experiment in two cases: two rivets in camera view and one rivet in camera view. In the first case, the camera can see two rivets, as shown in Fig. 10. The experiment is independently conducted on a set of four rivets. The heading estimation results are shown in Table I, while the position estimation results are shown in Table II. It can be seen from this table that the heading and position can be estimated with high accuracy. In the second case, there is only one rivet in the camera view. Three images are shown in Fig. 11. The first one is taken when the robot is about to leave the starting rivet a. The other two images are taken before and after the robot stops at the target rivet b, respectively. The heading

estimation results are shown in Table III and the position estimation results are shown in Table IV. From these tables it is clear that both the heading error and the position error based on two one-rivet images are greater than those based on a two-rivet image. B. Path Planning To test the Find-Crawler-TSP algorithm, we run it on simulated rivet distributions. The set-partition-problem is solved using the software IPLOG Studio™. The test data is a collection of 2-D coordinates which represent the rivet locations. All the coordinate values are integer numbers between 0 and 10 (1 unit 10 in). The threshold distance dr = 1:5 (or 15 in) and the data set forms a connected graph under this threshold. We compare three algorithms. The first is Stein and Wagner’s Find-MBTSP [21]. The second one is called minimum-spanning-tree tour (MST-Tour) algorithm, which constructs the minimum spanning tree on all the rivets and then uses depth-first method to tour the minimum spanning tree. The third one is our Find-Crawler-TSP algorithm. Figs. 12–14 show the tours generated by the three algorithms. It can be seen that the Find-MBTSP algorithm generates the least number of turns. However, since it does not consider the distance

Authorized licensed use limited to: Oklahoma State University. Downloaded on April 16,2010 at 16:27:17 UTC from IEEE Xplore. Restrictions apply.

372

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 2, APRIL 2008

Fig. 12. The tour generated by the Find-MBTSP [21] algorithm. The total time cost is 95.6. Note that this path is not feasible.

Fig. 15. The total number of turns in different tests.

Fig. 16. The total distance in different tests. Fig. 13. The tour generated by the MST-Tour algorithm. The total time cost is 117.6.

Fig. 17. The total time cost in different tests. Fig. 14. The tour generated by the Find-Crawler-TSP algorithm. The total time cost is 94.6.

between two consecutive rivets, the tour is not feasible. The MST-Tour algorithm generates a feasible tour but it has too many turns. Our Find-Crawler-TSP algorithm obtains less turns and shorter travel distance, compared to the MST-Tour algorithm. We also calculated the total time cost due to the turns and straight movement. As we observed

in our tests, the time cost for one turn is roughly equal to the time cost for 10 inch straight movement. Therefore, this cost is set to 1. The total time cost of the three paths can be calculated according to the following formula: total time cost = number of turns + total distance (in 10 inches). Hence, the total time cost for the three paths are 95.6, 117.6, and 94.6, respectively. We also conducted tests on other random sets of rivets using the three algorithms. Figs. 15 and 16 show the number of turns and the

Authorized licensed use limited to: Oklahoma State University. Downloaded on April 16,2010 at 16:27:17 UTC from IEEE Xplore. Restrictions apply.

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 2, APRIL 2008

total distance obtained from tests on data sets with varying sizes, respectively. The total time cost for each of the algorithms is shown in Fig. 17. Since the Find-MBTSP algorithm cannot guarantee feasible tours, we are more interested in comparing the MST-Tour algorithm and our Find-Crawler-TSP algorithm. From the above figures, it is easy to see that our Find-Crawler-TSP algorithm outperforms the MST-Tour algorithm. VI. CONCLUSION In this paper, a Crawler robot is adopted to conduct the engineered structure inspections. The autonomous navigation of the Crawler robot is studied, with a focus on the localization and path planning problems. A vision-based algorithm is developed for localization and a Find-Crawler-TSP algorithm is developed for the path planning. Simulation and experimental results are provided. The proposed solution can be extended to other similar engineered structure inspection applications that involve autonomous robots such as the inspection of bridges, buildings etc. Our future work will investigate the integration of multiple Crawler robots so that they form a robotized sensor network for faster inspection.

REFERENCES [1] N. Xu, S. Rangwala, K. K. Chintalapudi, D. Ganesan, A. Broad, R. Govindan, and D. Estrin, “A wireless sensor network for structural monitoring,” in Proc. 2nd Int. Conf. Embedded Networked Sensor Syst., 2004, pp. 13–24. [2] D. R. Huston, “Adaptive sensors and sensor networks for structural health monitoring,” in Proc. SPIE—Int. Soc. Opt. Eng., 2001, vol. 4512, pp. 203–211. [3] D. Liu, H. Kirkham, A. Johnston, and L. A. Bergman, “Micro-sensor networks,” in Proc. SPIE, 1994, vol. 2215, pp. 29–38. [4] W. H. Prosser, T. L. Brown, S. E. Woodard, G. A. Fleming, and E. G. Cooper, “Sensor technology for integrated vehicle health managment of aerospace vehicles,” in Proc. AIP Conf., 2003, pp. 1582–1589. [5] Y. Sun, “Development of remote field eddy current (rfec) technique for aviation applications,” Innovative Materials Testing Technologies, Inc. 2003. [6] F. Kirchner and J. Hertzberg, “A prototype study of an autonomous robot platform for sewerage system maintenance, autonomous robots,” Autonomous Robots, vol. 4, no. 4, pp. 319–331, 1997.

373

[7] O. Adria, H. Steich, and J. Hertzberg, “Dynamic replanning in uncertain environments for a sewer inspection robot,” Int. J. Advanced Robot. Syst., vol. 1, no. 1, pp. 33–38, 2004. [8] W. Henning, F. Hickman, and H. Choset, “Motion planning for serpentine robots,” in Proc. ASCE Space Robot., 1998, pp. 1–7. [9] D. Huston, B. Esser, G. Gaida, S. Arms, and C. Townsend, “Wireless inspection of structures aided by robots,” in Proc. SPIE—Int. Soc. Opt. Eng., 2001, vol. 4337, pp. 147–154. [10] B. L. Luk, T. S. White, D. S. Cooke, N. D. Hewer, G. Hazel, and S. Chen, “Climbing service robots for duct inspection and maintenance applications in a nuclear reactor,” in Proc. 32nd Int. Symp. Robot., 2001, pp. 41–46. [11] D. Longo and G. Muscsito, “A modular approach for the design of the alicia climbing robot for industrial inspection,” Industrial Robot: An Int. J., vol. 31, no. 2, pp. 148–158, 2004. [12] M. Siegel, “Remote and automated inspection: Status and prospects,” in Proc. 1st Joint DOD/FAA/NASA Conf. Aging Aircraft, 1997. [13] B. Bahr and F. Wu, “Design and safety analysis of a portable climbing robot,” Int. J. Robot. Autom., vol. 9, no. 4, pp. 160–166, 1994. [14] University of Dayton Research Institute. Dayton, OH. [Online]. Available: http://www.udri.udayton.edu/general.htm [15] Y. Bar-Cohen and P. Backes, “Scanning large aerospace structures using open-architecture crawlers,” in Proc. Nat. Space Missile Mater. Symp., 2000. [16] J. Xiao, J. Xiao, N. Xi, R. Lal Tummala, and R. Mukherjee, “Fuzzy controller for wall-climbing microrobots,” IEEE Trans. Fuzzy Syst., vol. 12, no. 4, pp. 466–480, 2004. [17] J. Xiao, “Development of miniature climbing robots—Modeling, control and motion planning,” Ph.D. dissertation, Michigan State Univ., East Lansing, MI, 2002. [18] E. M. Arkin, M. A. Bender, E. D. Demaine, S. P. Fekete, J. S. B. Mitchell, and S. Sethia, “Optimal covering tours with turn costs,” in Proc. Symp. Discrete Algorithms, 2001, pp. 138–147. [19] A. Aggarwal, D. Coppersmith, S. Khanna, R. Motwani, and B. Schieber, “The angular-metric traveling salesman problem,” SIAM J. Computing, vol. 29, no. 3, pp. 697–711, 1999. [20] D. T. Lee, C. D. Yang, and C. K. Wong, “Finding rectilinear paths among obstacles in a two-layer interconnection model,” Int. J. Comput. Geometry Appl., vol. 7, no. 6, pp. 581–598, 1997. [21] C. Stein and D. P. Wagner, “Approximation algorithms for the minimum bends traveling salesman problem (extended abstract),” Dartmouth College, Hanover, NH, Tech. Rep. TR2000-367, 2000. [22] E. Balas, Combinatorial Optimization ch. 7, pp. 151–210, 1979. [23] N. Christofides, Worst-case analysis of a new heuristic for the traveling salesman problem Graduate School of Industrial Administration, Carnegie-Mellon Univ., Pittsburgh, PA, Tech. Rep. 338, 1976.

Authorized licensed use limited to: Oklahoma State University. Downloaded on April 16,2010 at 16:27:17 UTC from IEEE Xplore. Restrictions apply.