Research article
High-speed TFT LCD defect-detection system with genetic algorithm Chern-Sheng Lin and Yo-Chang Liao Department of Automatic Control Engineering, Feng Chia University, Taichung, Taiwan
Yun-Long Lay Department of Electronic Engineering, National Chinyi University of Technology, Taichung, Taiwan
Kun-Chen Lee Department of Automatic Control Engineering, Feng Chia University, Taichung, Taiwan, and
Mau-Shiun Yeh Chung-Shan Institute of Science and Technology, Tao-Yuan, Taiwan Abstract Purpose – The purpose of this research is to develop an automatic optical inspection system for thin film transistor (TFT) liquid crystal display (LCD). Design/methodology/approach – A new algorithm that accounts for the closing, opening, etching, dilating, and genetic method is used. It helps to calculate the location and rotation angle for transistor patterns precisely and quickly. The system can adjust inspection platform parameters according to viewed performance. The parameter adaptation occurs in parallel with running the genetic algorithm and imaging processing methods. The proposed method is compared with the algorithms that use artificial parameter sets. Findings – This system ensures high quality in an LCD production line. This multipurpose image-based measurement method uses unsophisticated and economical equipment, and it also detects defects in the micro-fabrication process. Originality/value – The experiment’s results show that the proposed method offers advantages over other competing methods. Keywords Inspection, Automation, Transistors, Pattern recognition Paper type Research paper
affecting the performance of the LCD product. Such defects include mask problems, over or under etching, TFT open, gate open, ITO layer open, data open, gate-ITO layer short. So, it is important for this automatic optical inspection system to detect defects at various phases of material-layer patterning. Mask operation, opening, closing, and image subtracting process are the most commonly used methods for image processing algorithms since they remove image noises with only a little distortion. The opening method, includes erosion and dilation process, is often used to cut acute angles on image edges (Hajimowlana et al., 1999; Kim et al., 2001). The relation is shown in equation (1) where A stands for the target image, and B stands for the opening sample:
1. Introduction This research combines digital image processing and electrical tests to send signals to a thin film transistor (TFT) liquid crystal display (LCD) by a signal generator, and then gets an image by a video camera and looks for defects during digital image processing (Lin et al., 2006, 2005, 2003). Figure 1 shows the machinery layout and the operationrelated equipment including monitor, keyboard, mouse, and start and stop buttons. The safety-related equipment includes an external protective hood that is able to ensure operation safety; an emergency stop button that allows operators to stop the machine immediately; and an optical axis sensor that enables operators to remove TFT LCD panels without opening the safety door. Figure 2 shows the TFT array substrate of the LCD device in the display area. Many production defects may occur
A+B ¼ ðAQBÞ%B
ð1Þ
where the symbol Q denotes erosion process and % denotes dilation process. When we are only interested in a specific image characteristic during digital image processing, we use a certain mask to heighten these specific image characteristics
The current issue and full text archive of this journal is available at www.emeraldinsight.com/0144-5154.htm
Assembly Automation 28/1 (2008) 69– 76 q Emerald Group Publishing Limited [ISSN 0144-5154] [DOI 10.1108/01445150810849037]
This work was sponsored by the National Science Council under Grant Number NSC 96-2221-E-035 -028 -MY3.
69
High-speed TFT LCD defect-detection system
Assembly Automation
Chern-Sheng Lin et al.
Volume 28 · Number 1 · 2008 · 69 –76
Figure 1 Machinery layout and the photograph of inner set-up
The system adopts a PC-BASED multiple threads software system. With the calculation results of image processing technology and a genetic algorithm, the system can adjust inspection function parameters automatically, and direct a motion card to control a servomotor when it moves in feedback mode. During the process of moving, it will yield a signal to notify the image capturing card and line scan charge coupled device. Under the dual CPU systems design, it will use multiple construction methods to calculate its gray value, and adjust its tolerance. Then, inspecting dim and bright points will begin. The evolution process of the genetic algorithms (Johnson and Rahmat-Samii, 1997; Haupt, 1995; Goldberg, 1999; Bhandarkar and Hui, 1999) adopted by this research is shown in Figure 3.
2. Genetic algorithm for the inspection function parameters adjustment 2.1 Choose first population During the test, there are seven influential variables including a, b, and c in the mask operation as well as gain factor G1, G2 and offset O1, O2 of the image improvement. The values are ranged as follows:
Figure 2 The defects of TFT array substrate of LCD device in the display area
210:0 # a; b; c; G1 ; G2 # 10:0;
Data open
2255 # O1 ; O2 # 255
An actual eight-bit value of the whole number ranges from 2 128 to 127, but the numbers are narrowed down to a range
ITO open Gate-ITO short
Figure 3 Evolution process for high-speed TFT LCD defect-detection system with genetic algorithm
TFT open
Array
Start Gate open
mask operation
adjustment of gain factors and offsets
Choose first population
Genetic crossover and mutation opening operation
TFT Substrate
Calculate the fitness first dilation operation and second dilation operation
for separation and improvement purposes. The expression can be presented as in equation (2) where C stands for the intermediate, A for the target image, M for the type of mask, and H for the result:
Filter out
XX Cðm; nÞ ¼ Aðm 2 k; n 2 lÞMðk; lÞ k
image subtracting
l
M ¼ ½a
b
c
b
a
ð2Þ
Hðm; nÞ ¼ Cðm; nÞ £ Gi þ Oi
location of defect is identified
where G1, G2 are gain factor and O1,O2 are offset. 70
High-speed TFT LCD defect-detection system
Assembly Automation
Chern-Sheng Lin et al.
Volume 28 · Number 1 · 2008 · 69 –76
from 2 12.8 to 12.7 by simulation to improve calculating efficiency. Values 2255 to 255 cannot be represented in an eight-bit pattern either; thus, the actual value will be converted into an eight-bit pattern. Although accuracy is given in this calculation, it will not have too much influence on the result. Equation (3) is used while a new gene is produced where NG stands for the new gene value, min for the minimum value, max for the maximum value, and R for the conversion rate:
equations to make the overall mutation rate conform to the settings. In MF1, a random number allows a ^ 10 percent variation of its range to be able to mutate. MF1, shown in equation (7), is the value after evolving through the first mutation method. MR stands for mutation rate and GV for the value before mutation. When the random number is greater than 0, the mutation will evolve, but keep its original value; when the number is 0, the mutation will evolve to produce a new value. R stands for the range of values: 8 < GV þ Random R5 2 R2 ; if RandomðMR £ 2Þ ¼ 0 MF1 ¼ : GV; if RandomðMR £ 2Þ . 0
NG ¼
min þ Randomðmax 2 min Þ R
ð3Þ
Each value, based on an eight-bit group, is converted into chromosomes through binary encoding; the seven pairs of chromosomes, the critical genes for forming living species, are designed in the programming since genes can be used in whole number forms and are good for the algorithm of all genes and chromosomes regardless of their different length and quantities. Rather than substantially adapting the program, new genes can be added by just adjusting the gene structures only.
ð7Þ In MF2, only some genes will be bit-reserved, producing a random RV value. After this process, the matching bits will be calculated through the bitwise XOR operation. In equation (8), the MF2 stands for the value after evolving through the second mutation method: ( XORðGV; RVÞ; if RandomðMR £ 4Þ ¼ 0 MF2 ¼ ð8Þ GV; if RandomðMR £ 4Þ . 0
2.2 Genetic crossover and mutation Genetic crossover forms new genes by gaining pieces from each parent. The combination of genes is decided by a random mask where the value ranges from whole number 0 to whole number 255. As equation (4) shows below, the CM stands for the mask during genetic crossover: CM ¼ Randomð255Þ
In MF3, a new gene is produced to fit with conditions for replacing the original gene with a new gene when mutation occurs. In equation (9), MF3 stands for the value after the third mutation method: ( NG; if RandomðMR £ 4Þ ¼ 0 ð9Þ MF3 ¼ GV; if RandomðMR £ 4Þ . 0
ð4Þ
where Random(n) stands for a random value from number 0 to number n. Once the mask is produced, paternal gene FG and mask gene CM, taken from the genetic pool are calculated through the bitwise AND operation. Maternal genes MG and CM which are bit-reversed, are taken from the genetic pool to be calculated through the bitwise AND operation. After the calculation, the results are combined with one another through the bitwise OR operation, forming Child(i), as shown as equation (5). Child(i) stands for the bit number in Child(i), in which (i) ranges from bits 1 to 8: ChildðiÞ ¼ max{min{FGðiÞ; CMðiÞ}; min{MGðiÞ; NOTðCMðiÞÞ}}
2.3 Calculating fitness function values Adaptation to the environment must also be considered. As for applying digital image processing, a sample image can be used as a learning model. We mark down defective positions on the original image, record those on sample S, and then compare the calculation’s result with sample S. The adaptation improves as the variation narrows. To distinguish individual variation, we multiply the sample by 100,000 where W and H stand for the width and height of the image, respectively, and Fn for the fitness function value. The equation for the fitness function value is shown in equation (10): P P jMðm; nÞ 2 Sðm; nÞj ð10Þ Fn ¼ 100; 000 £ m n W £H
ð5Þ
The calculation for Child2 (i ) is similar to the calculation for Child (i ), but the paternal genes and the maternal genes must be exchanged. The equation is shown in equation (6): Child2 ðiÞ ¼ max{min{MGðiÞ; CMðiÞ}; min{FGðiÞ; NOTðCMðiÞÞ}}
2.4 Filter out the defective breeds After calculating the fitness function values, the family competition genetic algorithm is used to remove defective breeds. The purpose is to find the minimum variation value and use the sorting method to arrange fitness function values, keep the ten minimum genes, and remove the rest.
ð6Þ
Through the test, it is found the evolution meets a chokepoint and the fitness function values will stop evolving if only one mutative method of evolution is adopted; thus, the evolution can be quickened if three mutative methods of evolution are adopted at the same time. MR is defined as the frequency of mutation in evolution. If we assume MR ¼ 100, then a mutation occurs in every 100 periods of evolution. To preserve MF1 $ 50 percent; MF2 $ 25 percent; MF3 $ 25 percent, MR are multiplied, respectively, by the numbers 2 and 4 in the mutation
3. Experimental results In a manual inspection, it took us much time to learn a ¼ 2 2, b ¼ 1 and c ¼ 2 were available parameters, while the results reported a combination as a ¼ 2, b ¼ 25, c ¼ 2 98 when the genetic algorithms process was used. The result of genetic 71
High-speed TFT LCD defect-detection system
Assembly Automation
Chern-Sheng Lin et al.
Volume 28 · Number 1 · 2008 · 69 –76
algorithms diverged greatly from and was better than manual adjustment. During the research, genetic crossover and mutations are processed with tournament selection and mask crossover. With a population size of ten, the evolution occurred up to 31 times until the 370th generation, and the fitness evolved from 17,784 to 4,212, as shown in Figure 4. At this stage, the parameters are able to inspect defects stably. Also, it takes only a few minutes for the system to figure out the parameters, exceeding manual adjustment in speed and stability. The number of fitness was reduced further to 3.877 in the 43,589th generation. The first calculation using pixel positioning was meant to work out numbers directly by even space plus rotation and translation compensation, as shown in Figure 5. The result, however, showed correct correspondence only on the very left and the very right side but not in the central area of the frame, resulting in the numbering mistake shown in Figure 6. Since, the final purpose of this testing procedure is to correctly identify a position that needs laser repair and replacement in the next stop after exerting a pixel number, numbering error will waste much time during manual searching by a high-resolution transmission microscope and
reduce overall efficiency. Thus, it is suggested more time be spent on image testing analysis to ease the use of a stable and 100 percent accurate margin detection function of sub-pixels for calculating pixel numbering. This is an innovative function when we compare it with equipment used now by TFT LCD plants that often needs manual numbering work. Figure 7 shows the search for the specified point position with the sub-pixel technology in the horizontal direction and calculating point-to-point distance. A re-measurement is done after repeating movements to a certain distance in a vertical direction (Figure 8). According to the result, the distortion of the first area is shown as Figure 9 and the change of each area is in Figure 10. In each procedure, we need to get four images and save time, after taking out the first one, the machines will continue running until four images are gained for further processing. The next image gained must be processed immediately afterwards. Therefore, commands must be given to drive the Adlink PCI-8164 movement control card into the platform, advise the frame grabber and trigger signal 6,000 points/s to switch to a different mode to begin testing the software function. The specialized controller achieves speeds of up to 500 mm/s with resolution 1 mm is obtained. To make this function work in a PC-Based system with i2S Horizon Link imaging card, we need to adopt multithreaded technology by different programs to quickly convert and effectively use its multifunctional work capacities. When multithreaded designed programs are used, the effectiveness of dualprocessor equipped computer is best utilized (Kumar and Pang, 2002). One image is the size of 8,192 £ 6,400 ¼ 52.4 MB and the needed time to process a dim point inspection is about 8 s. A light point inspection needs about 3 s, as shown as in Table I. If a multiprocessing system uses only one thread to process an image, before the completion, the processor must be loaded fully for 8 s but the other will be idle. To enable the multiprocessing system to work, we need to design small tasks. Thus, as well as multitasks such as image acquisition
Figure 4 The variations of the fitness in the genetic evolution 15,000 13,000
Fitness
11,000 9,000 7,000 5,000 3,000 1,000 –1,000
0
100
200 Generation
300
400
Figure 5 Calculation of even space
72
High-speed TFT LCD defect-detection system
Assembly Automation
Chern-Sheng Lin et al.
Volume 28 · Number 1 · 2008 · 69 –76
Figure 6 Calculation of numbering error resulting from image distortion
Figure 7 Automatic calibration screen
control and processing, task division design is needed inside the image processing software. The results of the sequential image processing are shown in Figure 11. There are many hard-to-find defects in the original image. After mask operation, adjustment of gain factors and offsets, opening operation, first dilation operation, second dilation operation,
and image subtracting (Lin et al., 2007), the location of defect is identified in the final image. Figure 12 shows the different time lengths needed by the two different processing methods. When using the synchronized method, each movement has to wait until the previous one finishes and then it has to be added to the 73
High-speed TFT LCD defect-detection system
Assembly Automation
Chern-Sheng Lin et al.
Volume 28 · Number 1 · 2008 · 69 –76
Figure 8 Calibration was completed
Figure 9 Distortion of single area
31
28
25
22
19
16
13
10
7
4
1
Spacing Pixel
TFT LCD Image distortion analysis 236 234 232 230 228 226 224 222 220
Sampling point
needed time to complete the test; while using a nonsynchronized method, several movements can be operated at the same time to reduce the time length from 51 to 27 s. Table II compares genetic algorithm-based approach with fuzzy logic methods and manual inspection in terms of factors such as accuracy and defects detected. We can see our genetic algorithms exceed fuzzy logic methods and manual inspection in speed and stability for TFT-LCD inspection function parameters adjustment.
235 234 233 232 231 230
1 60 119 178 237 296 355 414 473 532 591 650 709 768 827 886 945 1,004
Pixel
Figure 10 Measurement result of each area
Sampling point
Table I Time needed for each stage Stage Acquisition of the image Testing of dim point Testing of bright point Data output
4. Conclusions Handling time (s)
This research test proved it is able to improve production efficiency with a simple and small machine structure. It only took less than 30 s to check one TFT-LCD piece with 99.9 percent accuracy reducing manual work. Our technique is much better able to find hard-to-find defects, and it can work under varying conditions. The effective method is
5 8 3 4
74
High-speed TFT LCD defect-detection system
Assembly Automation
Chern-Sheng Lin et al.
Volume 28 · Number 1 · 2008 · 69 –76
Figure 11 The results of the sequential image processing
(a) original image
(b) mask operation
(c) adjustment of gain factor G1, G2 and offset O1,O2.
(d) opening operation
(e) first dilation operation
(f) second dilation operation
(g) image subtracting
(h) final result
Figure 12 Time difference between synchronized and nonsynchronized processing Output
Bright Point
Dark Image
Blue Dim Point
Blue Image
Green Dim
Green Image
Red Dim Point
Red Image
Table II Comparisons of genetic algorithm-based approach with fuzzy logic methods and manual inspection for inspection function parameters adjustment Item
Speed
Flexibility
Stability
Error (percent)
Genetic algorithm-based approach Fuzzy logic methods Manual inspection
27 s 31 s 5 min
Good Good Medium
Good Medium Medium
0.1 0.2 3.2
Required time for synchronized processing 5+8+5+8+5+8+5+3+4 = 51(s)
Required time for non-synchronized processing
essential for the inspecting system to improve the speed of the defect detection of LCD glass panels. Thus, it has the advantage of saving labor costs and improving production efficiency as well as providing a defect-detection function that meets production demand.
5+5+5+5+3+4 = 27(s)
75
High-speed TFT LCD defect-detection system
Assembly Automation
Chern-Sheng Lin et al.
Volume 28 · Number 1 · 2008 · 69 –76
References
Transactions on Systems, Man, and Cybernetics – Part B: Cybernetics, Vol. 32 No. 5, pp. 553-70. Lin, C.S., Cho, J., Lay, Y.L., Lin, C.H. and Lin, N. (2006), “The auto-measurement of the gap of LCD glass plates using sub-pixel accuracy estimation”, Optik, Vol. 117 No. 8, pp. 349-54. Lin, C.S., Lay, Y.L., Chan, S.X., Ho, C.W. and Chiou, Y.C. (2005), “The high-speed measurement of partial area imaging system applied to development processing”, The Journal of the Chinese Institute of Engineers, Vol. 28 No. 4, pp. 721-6. Lin, C.S., Lay, Y.L., Huan, C.C., Chang, H.C. and Hwang, T.S. (2003), “An image-based LCD positioning system utilizing the modified FHT method”, Optik, Vol. 114 No. 4, pp. 151-60. Lin, C.S., Lin, C.H., Wu, C.Y., Shieh, H.Z. and Lay, C.C. (2007), “The rotation rate measurement of microfluidic biodisk spinner and automatic adjustment for pulsed light source”, Optics and Laser Technology, Vol. 39 No. 1, pp. 202-9.
Bhandarkar, S.M. and Hui, Z. (1999), “Image segmentation using evolutionary computation”, IEEE Trans. on Evolutionary Computation, Vol. 3, pp. 1-21. Goldberg, D.E. (1999), Genetic Algorithms in Search, Optimization and Machine Learning, 12th ed., Addison-Wesley Longman Inc., Boston, MA. Hajimowlana, S.H., Muscedere, R., Jullien, G.A. and Roberts, J.W. (1999), “An In-camera data stream processing system for defect detection in web inspection tasks”, Real-Time Imaging, Vol. 5, pp. 23-34. Haupt, R.L. (1995), “An introduction to genetic algorithms for electromagnetics”, IEEE Antennas and Propagation Magazine, Vol. 37, pp. 7-15. Johnson, J.M. and Rahmat-Samii, V. (1997), “Genetic algorithms in engineering electro-magnetics”, IEEE Antennas and Propagation Magazine, Vol. 39, pp. 7-21. Kim, J.H., Ahn, S., Jeon, J.W. and Byun, J.E. (2001), “A high-speed high resolution vision system for the inspection of TFT LCD”, paper presented at Symposium on ISIE, Vol. 1, pp. 12-16. Kumar, A. and Pang, G.K.H. (2002), “Defect detection in textured materials using optimized filters”, IEEE
Corresponding author Chern-Sheng Lin can be contacted at:
[email protected]
To purchase reprints of this article please e-mail:
[email protected] Or visit our web site for further details: www.emeraldinsight.com/reprints
76
