Evolving PCB Visual Inspection Programs using Genetic Programming

2013 28th International Conference on Image and Vision Computing New Zealand

Evolving PCB Visual Inspection Programs using Genetic Programming Feng Xie, Anh Hoang Dau, Alexandra L. Uitdenbogerd, Andy Song School of Computer Science and IT, RMIT University, Australia {feng.xie, dau.hoanganh, alexandra.uitdenbogerd, andy.song}@rmit.edu.au

Abstract—Automated optical inspection (AOI) is desirable in printed circuit board (PCB) manufacturing as inspecting manually is time-consuming and error-prone. This paper presents a study on evolving an AOI program with Genenetic-Programming (GP), an evolution-inspired technique. Using a GP-based approach, domain knowledge such as board design and lighting conditions are not required. Conventional feature extraction processes can also be avoided. The result demonstrates the evolved program capability to detect flaws under varied scenarios. Furthermore, it can be readily applied on different types of images without calibration or re-training. Index Terms—genetic programming; machine vision; defects detection; printed circuit board; automatic optical inspection.

I. I NTRODUCTION Commercial AOI systems are often expensive, which reduces their attractiveness to small-scale PCB manufacturers. They usually come with strict control of illumination, image acquisition and resolution. Moganti and Ercal [1] summarised six commonly used lighting techniques and described some rules on camera selection and resolution setting. Although techniques have matured, most studies still depend on highquality images for extracting typical features such as edges, lines and shapes. To meet the demands of small businesses, we aim to establish a cost-effective yet reliable visual inspection method. The program should be tolerant to a certain degree of variation in illumination and image quality, which is often the case in practice. Our inspiration is from GP’s effectiveness in machine vision problems [2], [3]. Most traditional methods require feature extraction, which is usually manually designed. GP, on the other hand, may not require this step and work on raw pixel values directly. By GP, computer programs with high accuracy can be automatically generated instead of manually coded. GP is an evolutionary computational method. In GP, a set of random solutions are generated and evaluated. The best ones are chosen for breeding the next generation. The iterative search for the best solution will stop once one of the predefined termination conditions is met. In this study, we use tree-shape GP, which represents a solution as a tree-structured computer program. Evolutionary Algorithms in general have been previously used to inspect PCBs, mostly in the design c 978-1-4799-0883-7/13/$31.00 2013 IEEE

and assembly phase [4], [5]. Mashohor et al. [6] used genetic algorithms with a library of high-level image processing functions to identify faulty components. Crispin and Vankov [7] used genetic algorithms with a component-based template matching approach to inspect the surface mount of circuit boards. However, these studies did not investigate how camera and light setting changes affect the performance. This paper is an extension to our earlier work on PCB defect detection with GP [8]. The main differences between this paper and the previous one are that 1) Image matching is automated which may introduce more difficulties, and 2) The robustness of the technique at handling different board designs and environment configurations is further explored. II. M ETHODOLOGY The basic process of our method is to train a program which can accurately detect PCB anomalies based on sample images by Genetic Programming. The input of this program is raw values of a sub-image from the difference between an input image and a perfect board. Its output is either “defect” or “no defect”. This program is effectively a binary classifier. Two batches of PCBs with defects are used in this study. The High-Q Batch, as the name suggests, is of higher quality. The Low-Q Batch is less sharp and pictured under poorer lighting, which hinders the inspection. The High-Q Batch is taken by a consumer-level Nikon 5000 camera with a Sigma 17-70mm lens. The Low-Q Batch is taken by the Basler GigE 5 megapixel camera. Only room fluorescent lighting was used, which, however, was not identical in each set. Boards were mounted on the gantry for the photo shoot. The photographic lens introduced image distortion observable with the naked eye. Moreover, the boards are not consistently positioned within the photos. Sample board images are shown in Figure 1(a) and Figure 1(b). The basic steps of our method are described below. A. Alignment To produce the differencing image, an input board needs to be aligned with the template board to overcome image variations. A Gaussian Blur is first applied on all images to reduce noise. These images are then cropped on all sides to remove parts of the image that don’t contain the actual board. A template matching algorithm from OpenCV is applied to align these two images. After this process, all the test boards

406

2013 28th International Conference on Image and Vision Computing New Zealand

(a) Example PCB Image from the High-Q Batch Fig. 1.

(b) Example PCB Image from the Low-Q Batch

Two Batches of PCB Photos with Different Quality

are adjusted according to the template. It is expected that the alignment will not be perfect. Even when the boards completely overlap, the position of individual components will still be slightly different. B. Differencing

The full image is of size 858 × 570 and 863 × 647 for High-Q and Low-Q batches respectively. During the sampling process, one window position is 1 pixel apart from its adjacent window. This is to maximize the number of windows generated through this sampling, hence, increases the classification accuracy. We train GP in a supervised learning manner. The defect areas on the differencing image are manually marked for training. The sliding window is then applied through this labelled differencing image. If a window overlaps with the defect for at least 70%, it is labelled as positive (“defect”), otherwise negative (“no defect”). D. GP Representation

Fig. 2.

Example of Differencing

The aligned images are converted into grey-scale for differencing, of which the result is the absolute difference between the two images. Figure 2 shows an example of a differencing result from the High-Q Batch. The differencing image will almost never be pure black (difference equal to 0) even if there is no fault on the test image. This is because perfect alignment is impossible to achieve. Moreover, differences from lighting conditions, shift in the legs of components, and board labels will also be captured. These non-defect regions may visually appear similar to true defects, which is only one component near the lower edge in our example. GP is trained to classify these two types. C. Sliding Window We use a sliding window of size 20 × 20 to sample the differencing image, which by itself is too large as input for GP.

GP evolves computer programs composed of pre-defined terminals(leaf nodes) and functions(internal nodes). A GP run is a competitive search for all the possible combinations of terminals and functions, favouring the fittest. We define five functions in total. Except for the four basic arithmetic operations, a specific function named “Region” is introduced to allow GP to work on a selected sub-area from the 20 × 20 input image. Three terminals can be attached to this function: 1) Area specifies the region width and height, 2) Coordinates specifies its top-left corner, 3) Operation indicates one calculation that is applied on selected pixels. There are four possible values for the terminal Operation, including Median, Average, Standard Deviation and Range (the difference between the maximum and minimum pixel value). In addition, the terminal VariableX enables GP to read any single pixel value from the input images. The range of X is between 0 and 399 as the input image of size 20 × 20 contains 400 pixels in total. E. Fitness Measure The evolved program is effectively a binary classifier. The fitness-based selection enables the solutions that fit better to pass its genetic materials to the next generation. In this particular problem domain, the faulty areas are usually rare, resulting in the number of “no defect” areas being significantly greater than “defect” ones. Such highly unbalanced data can

407

2013 28th International Conference on Image and Vision Computing New Zealand skew the fitness value, as a classifier can easily achieve high accuracy by categorising everything as the majority class. For example, if the ratio of positives and negatives in training data is 1 : 10, the accuracy of such classifier is 90% although it recognises none of the minority class instance. To address this issue, we evaluate fitness to be the average of True Positive Rate (TPR) and True Negative Rate (TNR). F. Integration The most accurate GP classifier found from the training process is used to examine new boards. A board to be inspected will go through the alignment and differencing process, against the template board. The classifier will then be applied on segments sampled by a sliding window of 20 × 20 to locate defect areas. The distance between adjacent windows is also 1, being consistent with the training process. Because of the overlapping window positions, a pixel will be classified multiple times. A voting strategy is therefore used to decide the final class of the pixel. Each pixel maintains a count, being the number of times it appears in a positive segment. If at least 60% of the windows visited for a given pixel say it is positive (count >= 240), the pixel is labelled positive and vice versa. This value is chosen based on the empirical experiment result. After this process, the defects will be highlighted as blobs on the image, which are groups of positive pixels adjacent to each other. Some simple rules such as the blob aspect ratio are applied to filter out false alarms. III. E XPERIMENTS AND R ESULTS The runtime GP parameters are such: 5% of programs in one generation are generated by mutation, 85% by crossover and 10% by elitism. The tree depth is limited between 2 and 8. A large population size of 1000 is used due to the complexity of the problem. The evolution process terminates when the generation number 100 is reached or a perfect classifier is found. Some small regions containing defects instead of the whole image as training data. The evolved program is later tested on the whole image. For the High-Q batch, extra tests were conducted to evaluate how our method would cope with variations in image quality and lighting conditions. Three image resolutions were simulated. The original images were reduced to 1/2, 1/3 and 1/4 size respectively, then enlarged to the original size, resulting in poor image quality. Examples are shown in Figure 3(b), 3(c), 3(d). In addition, the brightness of the original images was increased and reduced to generate two new test sets (see Figure 3(e), 3(f)). For the High-Q Batch, the training accuracy, TP rate and TN rate are 92.9%, 98.44% and 92.46% respectively. Table IV shows the test results of 11 High-Q Batch boards in six different scenarios. The six sub-rows for each board correspond to the original image and its five variations. The third column displays the detection ratio, being the number of detected defects out of all actual defects on that board. The fourth column is the number of false positive (FP) components. The last column lists the types of false alarms

and the corresponding number of FPs (in bracket). Each FP is a blob of neighbouring pixels classified as “defect“. More than one FP can be associated with a particular component. Table III displays five categories of false positive defect recognized by the PCB industry. Most of them can be easily filtered out by heuristic rules or manual inspection. We can see, that on the original images, all of the defects are detected. The program also shows tolerance to decreasing image quality. It manages to detect all faults at 1/2 and 1/3 resolution on all boards except for board No. 7. When image quality is further dropped down to 1/4, the classifier fails on 4 boards. The number of false positives increases considerably. This has shown to be challenging even to manual inspection due to the low standard images. On the other hand, the GP program is relatively tolerant to increased brightness. It only misses one defect on Board 8 when extra luminance is introduced. However, the performance on darkened images is rather poor. This is possibly because darkened boards minimize the difference between defects and non-defects on the differencing image. GP’s performance on the Low-Q Batch is not as high, which is expected. The training accuracy on this batch is 84.71%, with a TP rate of 78.22% and a TN rate of 87.3%. The testing result is shown in Table I. The evolved classifier worked properly on 7 out of 10 boards with 3 misses. We also tested the program trained on the High-Q Batch for this task. Table II listed the testing result. Although the condition is significantly different from what it is trained for, the classifier successfully identifies all faults on 5 out of 10 boards. By visual inspection, we can see that the misses from both of these experiments are quite difficult detect even by human experts (shown in Figure 4). TABLE I T EST R ESULTS ON THE L OW-Q BATCH (T RAINED ON L OW-Q BATCH ) Board No.

Detection Ratio

FP Components

1 2 3 4 5 6 7 8 9 10

5/5 0/0 1/1 1/1 0/1 0/1 0/1 0/0 0/0 4/4

4 2 5 4 5 4 4 3 2 3

FP Type I(1), III(6), IV(2) II(3), IV(2) I(1),III(3),IV(2) I(3), III(2),IV(2) I(3), III(4), IV(2) I(4),III(9) I(3), III(3), IV(2) II(1), III(1),IV(2) IV(2) I(1),III(5),IV(2)

IV. D ISCUSSION The program trained on the High-Q Batch is shown below. Although it uses two isolated pixels, the output mainly comes from a combination of median values of 10 selected regions. The instance is classified as positive only when the output is greater than 32767. These regions are overlapped and only 287 out of 400 pixels are used here.

408

Output = (Region(MED,,) + Att390 + Att152

2013 28th International Conference on Image and Vision Computing New Zealand

(a) High-Quality Board Image:Original Image

(b) High-Q: Quality Decreased to Resolution (1/2)

(c) High-Q: Quality Decreased to Resolution (1/3)

(d) High-Q: Quality Decreased to Resolution (1/4)

(e) High-Q: Increased Brightness

(f) High-Q: Reduced Brightness

Fig. 3. + * * * +

High-Q Image with Dynamic Settings

Region(MED,,)) * Region(MED,,) (Region(MED,,) + Region(MED,,)) (Region(MED,,) + Region(MED,,)) (Region(MED,,) + Region(MED,,) Region(MED,,))

if ((Output > 32767) defect = true; else defect = false;

The program trained on the Low-Q Batch follows a similar structure. It uses averages and medians. This program can be interpreted as follows: if the output is less than -250, the input image contains a defect, otherwise, there is no defect. It requires larger area (354 pixels in total) as its input. Both evolved solutions are not big in size, and contains no loops inside, which results in linear time complexity. Output = Region(AVG,,)/Att291

409

2013 28th International Conference on Image and Vision Computing New Zealand TABLE III T YPES OF FALSE P OSITIVES Type No.

Cause

Example 1. Components with different surface printing 2. Components or parts of the components with different colours

I

Component Variations

II

Ambient

III

Solder joints

Shapes of solder joints vary

IV

Alignments

1. Parts of components can not be aligned precisely 2. Solder mask cannot overlap well

V

Unknown

1. Shadow from other component 2. Shadow from holder

TABLE II T EST R ESULTS ON THE L OW-Q BATCH (T RAINED ON H IGH -Q BATCH ) Board No.

Detection Ratio

FP Components

1 2 3 4 5 6 7 8 9 10

5/5 0/0 0/1 0/1 0/1 0/1 0/1 0/0 0/0 4/4

1 1 3 2 3 2 3 1 1 1

PCB inspection programs are fairly simple in nature. Since feature extraction is also not needed, the visual inspection method is low cost and fast. The results from this study show many possibilities for future improvement. We will further improve the function and terminal set to address dark and unevenly lit images. Another aspect to be investigated is how to reduce false positives. A more sophisticated alignment technique that compensates for lens distortion may also yield better results. However, we have shown that it is possible to create a low-cost system for effective automatic inspection of circuit boards with genetic programming.

FP Type I(1),III(2) II(2) I(1),III(2) I(1), II(2),IV(1) II(1), III(2),IV(2) I(1), II(1),III(5) I(4), III(1), IV(1) IV(1) IV(1) I(1),II(1),IV(1)

ACKNOWLEDGEMENTS This research was funded by the Victorian Department of Business and Infrastructure, and Ad Hoc Software Pty Ltd. We thank John Harnett. R EFERENCES

Fig. 4. -

The Defects that GP Missed on Low-Q Batch

Region(MED,,) Region(AVG,,) Region(AVG,,) Region(MED,,)

-

Region(MED,,) Region(MED,,) Region(AVG,,) Att8

if ((Output