Robust welding seam tracking using image seam extraction

Robust welding seam tracking using image seam extraction J. S. Chen*1, G. D. Su1 and S. B. Xiang2 Extracting single pixel wide paths is the core image processing problem in many vision based welding seam tracking applications. Traditional techniques such as edge detection and image thinning may not be effective for solving this problem on low quality welding images polluted by process noises. Based on the understanding of the prior knowledge about the welding images, we present in this paper a welding seam tracking method using the image seam extraction technique. Owing to its optimisation nature, the proposed method is robust to strong image noises caused by environmental factors such as uneven illumination, light interferences and welding spatters. In addition, the proposed method is of low time complexity and is highly adaptive to different types of welding images. Experimental results on a variety of welding images validate the effectiveness and efficiency of the proposed method. Keywords: Seam tracking, Welding control, Image seam, Shortest path

Introduction Intelligent welding systems have attracted a lot of attention from both industry and academia due to the increasing demand for high welding quality and manufacturing flexibility. By incorporating certain kinds of sensors, an intelligent welding system acquires the welding data, which are further processed so as to extract information for calculating welding parameters to be used in the welding arm controlling. Previous researchers have proposed many intelligent welding systems using different sensors, tracking algorithms and control logic.1–3 In Ref. 1, a single ultrasonic transducer is adopted for sensing the weldment’s surface geometry, which is used for calculating subsequent spatial coordinates of the torch tip. In gas tungsten arc welding, Middel and den Ouden utilised the effect of SiO2 on arc voltage for welding seam tracking.2 Simpson proposed a through the arc estimation method, in which welding parameters are expressed as functions of the signature image, which is basically a histogram representation of the voltage and current signal collected over time.3 Recently, driven by the rapid development of optical imaging technologies, vision sensors have been more and more widely adopted for welding data acquisition due to its favourable characteristics of low cost and high precision, wide adaptability, robustness to electromagnetic interferences and the capability of capturing visual information for further quality inspection.4 In many

1

Department of Electronic Engineering, Tsinghua University, Beijing 100084, China Department of Electrical Engineering and Information, Sichuan Engineering Technical College, Deyang 618000, China

2

*Corresponding author, email [email protected]

vision based intelligent welding systems, the core image processing task is to extract single pixel wide edges or skeletons representing the welding seam or the structured laser stripe in the captured welding image.5–9 Feature points on these edges or skeletons are located and then used for the welding parameter estimation. Kou and Wu found that for steel and aluminium weldments, the best extraction result could be achieved by combining the Prewitt edge detector with an autoselecting threshold method.5 A similar approach was used by Shi et al., except that the Canny edge detector was used and the edge detection was performed iteratively on a local basis.6 Fang et al. used an image thinning algorithm and the Hough transform for extracting single pixel wide straight lines.7 For high quality welding images containing relatively regular welding seam or laser stripe patterns, template matching can be adopted.8,9 However, performances of these traditional image processing techniques may deteriorate seriously when the welding image is corrupted by process noises such as strong arc lights and welding spatters. Figure 1a shows a laser stripe image captured by an MT10 vision sensor in a Laser Pilot welding robot manufactured by Meta during a multilayer and multipass CO2 shielded V groove welding. The laser stripe was projected perpendicular to the welding seams. The corners of the stripe pointed by arrays are the intersection points of the welding seams and the laser stripe, indicating the centres of the welding seams. Figure 1b and 1c shows the Canny edge detection and the image thinning results of Fig. 1a respectively. The processing results are not satisfactory because the useful information of the laser stripe is strongly overwhelmed by the artefacts caused by noises. Compared with the original welding image, extracting the welding seam information from these processed images may be even more challenging.

ß 2012 Institute of Materials, Minerals and Mining Published by Maney on behalf of the Institute Received 27 July 2011; accepted 9 November 2011 DOI 10.1179/1362171811Y.0000000091

Science and Technology of Welding and Joining

2012

VOL

17

NO

2

155

Chen et al.

Robust welding seam tracking using image seam extraction

formalised by Avidan and Shamir in their work of content aware image resizing.13 Intuitively, an image seam is a connected single pixel wide path on an image from top to bottom (vertical image seam), or from left to right (horizontal image seam). As such, neighbouring pixels on an image seam are also neighbours in the image. Formally, given an n6m image I consisting of n rows and m columns, a vertical image seam S represented by an ordered sequence of image pixels can be defined by equation (1), in which si5(x(i),i) is the coordinate of the ith pixel on S and x is a mapping from the vertical coordinate to the horizontal coordinate: x:[1, …, n]R[1, …, m].13 Obviously, a vertical image seam contains one and only one pixel from each row of the image. Specifically in equation (1), the length of S equals the number of rows of the image I, and pixel I(si)5I(x(i),i) is from the ith row of I. A horizontal image seam can be defined similarly. Without losing generality, we only consider the vertical image seam because it can be easily transformed into a horizontal image seam through rotating the image by 90u. In equation (1), k is a non-negative integer constraining the connectivity of the pixels. An image column can be considered as a special case of the vertical image seam for k50. When k51, eight neighbours of image pixels are considered, and when k.1, the image seam may not be ‘strictly’ connected. S~fsi gni~1 ~fðxðiÞ,iÞgni~1 , s:t:Vi,jxðiÞ{xði{1Þj ƒk, (1)

0ƒkƒm

a original welding image; b Canny edge detection; c image thinning; d image seam extraction 1 Welding image and processing results

The aim of this study is to solve this problem by using image seam extraction, a technique widely used in various image processing tasks recently. In the remaining part of this paper, we will first introduce the basic concepts and algorithms of image seam extraction. The underlying rationale for applying this technique to the welding seam tracking task will then be explicated. We will also demonstrate through experiments that the proposed method can solve the welding seam tracking problem with considerably low complexity and high robustness. Furthermore, we will show that the proposed method can be applied to a variety of seam tracking problems on different types of welding images. Figure 1d shows the extracted single pixel wide skeleton of the laser stripe using the proposed method. The points indicating the locations of the welding seams can be located by applying a maximum curvature point finding algorithm10 to the skeleton. The resultant welding seam points (marked by crosses) are shown in the two pop-up windows left to the skeleton in Fig. 1d.

Image seam extraction Image seams have recently been widely used in digital image processing tasks such as editing, composition and resizing.11–13 The image seam on a single image was

Usually in image processing tasks, it is some special image seams that are of interests. For example, in the image resizing problem,13 an energy function E(N)is defined for describing the ‘noticeableness’ of image pixels. The total energy of an image seam is further defined in equation (2) as the sum of the energy values of all the pixels on the seam. TE ðS Þ~

n X

E ½I ðsi Þ

(2)

i~1

Among all the possible image seams, the image seam S* with the lowest total energy, as defined in equation (3), is extracted as the most unnoticeable part of the image to be removed or duplicated. Extracting such an optimal image seam is not as difficult as it seems to be. Actually it can be solved very efficiently using a dynamic programming algorithm, of which the complexity is linear to the total number of pixels in the image.13 S ~ argmin TE ðS Þ

(3)

S

Image seam extraction can also be formulated as a graph based problem. As illustrated in Fig. 2, each image pixel is a node in the graph. Coming out of each node, there are a number of weighted directed edges pointing to its neighbouring nodes in the next row. Apparently, the out degree of each node is decided by the value of k in equation (1). Additionally, a ‘start’ node is inserted, out of which equally weighted edges pointing to all the nodes in the first row are added. Similarly, an ‘end’ node is also inserted. By assigning weights to the edges instead of the nodes, the optimal image seam extraction can be equivalently transformed into a problem of finding the shortest directed path in the graph from the node ‘start’


2012

VOL

17

NO

2

156

Chen et al.


2 Structure of graph for image seam extraction

to the node ‘end’. Such a shortest path search problem can be easily solved.14 In Fig. 2, if E1[I(x,y)]5E2[I(x,y)]5E3[I(x,y)]5E[I(x,y)], the result of the shortest path search is identical to the solution of equation (3) using dynamic programming. However, such a graph based problem formulation provides more flexibility by allowing different weights for different edges coming out of the same node. In Fig. 2, if the edge weights E1[I(x,y)], E2[I(x,y)] and E3[I(x,y)] are not equal to each other, the shortest path search result may be significantly different from the solution of equation (3). The graph based formulation is adopted in our method and we use the unequal weighting scheme for enhancing the seam tracking performance. Without losing generality, we let k51 in our work. In fact, the proposed method can be directly extended to the cases where k.1.

Welding seam tracking As we have introduced, previous image based welding seam tracking approaches5–9 usually use traditional image processing techniques designed for general purpose applications. Thus, the prior knowledge about the characteristics of the welding seam or the laser stripe may not be fully utilised. In Fig. 1a, for example, there are at least three important priors that can be used for facilitating the welding seam tracking: (i) ideally, the laser stripe is continuous and its structure should be relatively simple. Therefore, all the spreading points in the middle and discontinuities along the stripe are probably caused by noises and should be ignored (ii) pixels with larger intensity values (range: [0, 1]) have higher probability of being on the laser stripe (iii) the points indicating the positions of the welding seams are single pixel points at the corners of the laser stripe. They can be located as the curvature maxima on the skeleton of the laser stripe. Consequently, for Fig. 1a, the welding seam tracking problem can be converted to a problem of extracting a continuous single pixel wide path from top to bottom

3 Image seam extraction using equal edge weights

which passes as many high intensity pixels as possible. That is to say, the major image processing task is to extract an optimum vertical image seam, of which the total pixel intensity value is maximised. To solve this problem, a graph G is constructed as illustrated in Fig. 2. The edge weights are positive and are inversely proportional to the intensity of the corresponding pixels. In our implementation, we simply use E[I(x,y)]5exp {2intensity[I(x,y)]}.The optimum image seam can then extracted by searching the shortest path in the graph G. Figure 3 shows the extracted optimum image seam when E1[I(x,y)]5E2[I(x,y)]5E3[I(x,y)]5exp{2intensity[I(x,y)]}. Although the overall structure of the laser stripe is well preserved, a close inspection reveals that the extracted image seam is rugged due to image noises, leading to noticeable inaccuracy in welding seam point location using maximum curvature point finding. According to prior (i) described above, the skeleton of the laser stripe should be relatively smooth, i.e. the geometrical length of the image seam should be as short as possible. In Fig. 2, the geometrical length of the diagonal edges (with weights E1[I(x,y)] or E3[I(x,y)]) is longer than that of the vertical edge (with weight E2[I(x,y)]). Therefore, we would favour vertical edges over diagonal edges during the image seam extraction. Instead of formulating a dual objective optimisation problem, we simply ‘penalise’ diagonal edges by multiplying a constant a (a.1) to their weights as shown in equation (4). We set a51?25 empirically. The resultant image seam is shown in Fig. 1d, in which the pop-up window on the right hand side demonstrates the smoothing effect. The welding seam points are then located as the maximum curvature points10 of the extracted image seam. More results of the image seam extraction and the welding seam point location on images captured in different stages of the multilayer and multipass welding are shown in Fig. 4. The corresponding experimental weldment is shown in Fig. 5


2012

VOL

17

NO

2

157

Chen et al.


a,b,c images captured in different welding stages; d,e,f corresponding image seam extraction results 4 Image seam extraction and welding seam tracking results

E2 ½I ðx,yÞ~expf{intensity½I ðx,yÞg,

(4)

E1 ½I ðx,yÞ~E3 ½I ðx,yÞ~aE2 ½I ðx,yÞ

5 Experimental weldment using seam tracking

For an n6m image, the graph G contains V5n6mz2 nodes and D5(2kz1)6(n21)6mz2m edges. According to Fig. 2, G is a directed acyclic graph14 with positive edge weights, so that the shortest path search can be performed with a time complexity of O(VzD), or simply O(n6m) when the value of k is relatively small and n&1, which are almost always true in practice. The complexity for constructing graph G is also O(n6m), and the complexity for the maximum curvature point finding10 is O(n). Hence, the overall complexity of the proposed method is linear to the number of pixels in the image. In this sense, our method is as simple as many fundamental image filtering operations, such as the Gaussian filtering or the median filtering, and thus can be implemented with high efficiency on most platforms. In addition, in practical systems, the region of interest can be chosen wisely to further improve the efficiency.


2012

VOL

17

NO

2

158

Chen et al.


implemented by removing all the edges connected to the pixels in the right half of the image. Similarly, a right image seam SR can also be extracted. SL and SR are considered as the left and right edges of the laser stripe respectively (iv) define a centreline SC between the two edges using equation (5). It can be easily proved that SC is also an image seam complying with equation (1). SC is used as the skeleton of the laser stripe for the welding seam tracking n xL ðiÞzxR ðiÞ ,i SC ~ , 2 (5) i~1 n n SL ~fðxL ðiÞ,iÞgi~1 and SR ~fðxR ðiÞ,iÞgi~1 Figure 6c shows the extracted image seams, SL, SC and SR, from left to right. It can be noticed that the extracted edges are significantly affected by the strong arc light interferences in the lower half of the image given the prior that the ideal laser stripe edge should be nearly straight. Xiong et al. solved this problem by changing the fuzzy membership function empirically. In our approach, this can be easily solved by changing the value of a in equation (4). Figure 6d shows the extracted image seams when a53. Figure 7 shows more experimental results on different welding images. Figure 7a–c are quoted from Kou and Wu5, Liu et al.16 and Fang et al.9 respectively. The corresponding extraction results are shown in Fig. 7d–f. It can be seen that the proposed method is robust to complicated shapes, uneven illumination, strong light interferences or even incomplete patterns. The bottom parts of the extracted image seams in Fig. 8f can be easily removed by inspecting the corresponding pixel intensity in the original welding image. a original welding image; b gradient image of a; c extraction result for a51?25; d extraction result for a53 6 Edge extraction in welding seam tracking

Edge extraction in welding seam tracking In certain welding applications, the welding seam or the laser stripe cannot be completely described using one single image seam.5,15 Figure 6a shows a laser stripe image presented in Xiong et al.15 Obviously, the width of the laser stripe cannot be ignored. For such welding images, a common practice is to first extract the edges and then use a centreline between the edges as the skeleton for welding controlling. For example, Xiong et al. used a fuzzy edge model for detecting the edges of the laser stripe in Fig. 6a.15 Actually, such an edge detection process can also be realised using the proposed image seam extraction based method. The basic idea is to extract image seams from the gradient image instead of the original welding image. The main steps of the edge extraction based welding seam tracking are as follows: (i) from the original welding image I, extract an initial image seam Sinitial, which divides the image into a left half and a right half (ii) calculate the gradient image IG5|dI/dx|z|dI/dy|. Figure 6b shows the gradient image of Fig. 6a (iii) extract an image seam SL left to Sinitial from the left half of the gradient image IG. This can be

Non-vertical seam tracking According to the definition, the geometry of an image seam is limited to some extent. For example, it can only be vertical (or horizontal) and its maximum slope is decided by the value of k in equation (1). As a result, the proposed method cannot be directly applied to free form non-vertical welding seams, of which an example quoted from Liu et al.17 is shown in Fig. 8a. Based on the observation that practical welding seams are usually smooth so that their local patterns are relatively simple, we propose a locally adaptive image seam extraction method for welding seam tracking. Our basic idea is to extract local image seams from a sequence of overlapping local subblocks of the welding image. The final result is a concatenation of all the extracted local image seams. The main steps of the proposed method as illustrated in Fig. 9 are as follows: (i) an initial rectangular window R0 is placed on the welding image. A local image seam S0 is extracted from the image block inside R0 (ii) select a point P on S0 and divide S0 into two parts: S0up and S0down (iii) construct a new rectangular window R1, of which the rows are perpendicular to the direction of S0down and P is the middle point of the first row. The direction of S0down can be defined as the slope of a straight line that fits S0down best


2012

VOL

17

NO

2

159

Chen et al.


7 Edge extraction results on different welding images

(iv) from the image block inside R1, extract a local image seam S1 that contains the point P (v) repeat steps (ii)–(iv) until the local image seam reaches the image boundary (vi) concatenate all the local image seams to form the extracting result: Scat ~S0up zS1up z:::. Strictly speaking, Scat may not comply with equation (1). However, it can be regarded as a more ‘general’ image seam, since it is a single pixel wide continuous path and neighbour pixels on the path are also neighbours in the image. The above process can be applied to the gradient image for detecting welding seam edges. Figure 8b demonstrates the initial rectangular window and several local rectangular windows constructed during the extraction of the lower edge of the welding seam. Extraction of the centreline can no longer be realised by applying

equation (5). One possible solution is to identify all the pixels in the image that has equal distances to the two extracted edges. However, the centreline thus extracted may be discontinuous. Therefore, we take an alternative approach by using a ‘distance map’, in which the value of a pixel equals the absolute value of the difference between the pixel’s Euclidean distances to the two edges. The locally adaptive image seam extraction method is applied to this distance map for extracting the centreline, as illustrated in Fig. 8c. Figure 8d shows the extracted welding seam edges and the centreline. It should be noticed that the complexity of the locally adaptive image seam extraction is determined by both the length of the welding seam and the size of the local window. The selection of the window size and the initial window position may affect the extracting result.


2012

VOL

17

NO

2

160

Chen et al.


a welding seam image; b edge extraction on gradient image; c centreline extraction on distance map; d extraction results 8 Non-vertical welding seam tracking

work, we will investigate the feasibility of applying this method to more complicated welding images.

Acknowledgements This work was essentially supported by the National Natural Science Foundation of China (grant no. 20111300754) and the Chuanxin Foundation from Tsinghua University (grant no. 110107001).

References

9 Locally adaptive image seam extraction

Empirically, for smooth welding seams, the local rectangular windows can be relatively large, and small windows are recommended for welding seams containing sharp turnings. Actually, adaptively changing the size of the window by analysing the curvature of the local image seam is also feasible.

Conclusions Extracting single pixel wide continuous edges or skeletons is a critical task in many vision based welding seam tracking applications. We reveal that given the prior knowledge about the welding image, such a problem can be efficiently solved using image seam extraction. A graph based problem formulation is presented, and its adaptability to different types of welding images and welding seam tracking requirements is demonstrated. The proposed method has very low time complexity, and experimental results show its high robustness towards noises and interferences. For future

1. C. Umeangukwu, B. Maqueira and R. Lambert: IEEE Trans. Ind. Electron., 1989, 36, 338–348. 2. W. Middel and G. den Ouden: Sci. Technol. Weld. Join., 1999, 4, 335–339. 3. S. W. Simpson: Sci. Technol. Weld. Join., 2008, 13, 80–86. 4. Y. H. Shi and G. R. Wang: Sci. Technol. Weld. Join., 2006, 11, 271–277. 5. H. Kou and L. Wu: J. Mater. Process. Technol., 2002, 120, 458–465. 6. F. Shi, T. Lin and S. Chen: Ind. Robot, 2009, 36, 277–283. 7. Z. Fang, D. Xu and M. Tan: Int. J. Adv. Manuf. Technol., 2010, 49, 519–526. 8. Zh. Y. Zhu, T. Lin, Y. J. Piao and S. B. Chen: Int. J. Adv. Manuf. Technol., 2005, 26, 784–788. 9. Z. Fang, D. Xu and M. Tan: IEEE/ASME Trans. Mechatron., 2011, 16, 540–550. 10. A. Rosenfeld and E. Johnston: IEEE Trans. Comput., 1973, C-22, 875–878. 11. A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D. Salesin and M. Cohen: ACM Trans. Graph., 2004, 23, 294–302. 12. J. Jia, J. Sun, C. K. Tang and H. Y. Shum: ACM Trans. Graph., 2006, 25, 631–637. 13. S. Avidan and A. Shamir: ACM Trans. Graph., 2007, 26, 10-1–10-9. 14. T. H. Cormen, C. E. Leiserson, R. L. Rivest and C. Stein: ‘Introduction to algorithms’; 2001, Cambridge, MA, The MIT Press. 15. Z. Xiong, W. Wan and J. Pan: Lect. Notes Electr. Eng., 2011, 8, (3), 323–330. 16. M. Y. Liu, N. S. Liu, C. R. Guo and B. F. Qian: Weld. Join. (China), 2007, 1, 28–31. 17. J. F. Liu, Z. W. Xiao, J. Z. Zou and X. L. Yang: Trans. China Weld. Inst. (China), 2008, 29, 61–64.


2012

VOL

17

NO

2

161