Abstract. The role of color descriptors has been demonstrated to be quite remarkable in many visual inspection tasks. In some other tasks, texture measurements ...
Pattern Analysis and Applications
Optimizing color and texture features for real-time visual inspection Topi M¨aenp¨aa¨, Jaakko Viertola, and Matti Pietik¨ainen
1
Machine Vision Group Department of Electrical and Information Engineering University of Oulu, Finland {topiolli, jaskav, mkp}@ee.oulu.fi
Abstract. The role of color descriptors has been demonstrated to be quite remarkable in many visual inspection tasks. In some other tasks, texture measurements are needed because of unevenly colored or achromatic surfaces. In many applications, color and texture must be combined to achieve good performance. At the same time, the computational complexity of the methods must be kept as low as possible. In this paper, a methodology for combining and optimizing color and texture features is proposed. Feature sets are optimized using a deterministic and a randomized approach. Results are demonstrated in detecting and recognizing defect types on wooden surfaces. Keywords: optimization, beam search, genetic algorithms, wood inspection, color percentiles, LBP
1. Introduction Visual inspection plays an important role in the quality control of many manufacturing processes [1]. In many visual inspection tasks, color is the main information source. Color descriptors have been used in inspecting wood, ceramic tiles, food etc. [2, 3, 4]. Sometimes, however, color features alone do not provide sufficiently accurate information. Uneven illumination conditions, similarly colored but differently textured surfaces, small defects and many other phenomena call for support from texture analysis methods. On the other hand, in some applications like paper and textile inspection there is no color information, and thus only texture can be used (See e.g. [5, 6]). A number of methods of combining color and texture descriptions have been proposed [7, 8]. These and many other methods treat color and texture as a joint phenomenon. Recent research indicates, though, that it would be beneficial to measure color and textural patterns separately [9]. This approach also has the advantage that, if we can treat color and texture as independent properterties of an image, their optimization can be carried out irrespective of each other. 1
The financial support provided by Academy of Finland and the Graduate School in Electronics, Telecommunications and Automation is gratefully acknowledged.
2
Topi M¨ aenp¨ aa ¨, Jaakko Viertola, and Matti Pietik¨ ainen
In real-time visual inspection tasks, the length of a feature vector plays a major role irrespective of the classification principle used. In this paper we show how color and texture features can be optimized with respect to feature vector length and classification accuracy using beam search and a genetic algorithm. The method is independent of the features used, and can be applied to various visual inspection problems. As a result, a compact set of powerful features is selected that can be used by a classifier. The optimization of a classifier is beyond the scope of this paper. As an application example, the optimization methods are applied to a set of color and texture features in a parquet defect classification problem.
2. Parquet inspection experiment Color is usually regarded as a very important cue in wood surface inspection. This is hardly surprising as most defect types have fairly distinctive color properties compared to sound wood [10]. There are, however, certain defect types that are very difficult to distinguish based solely on their color. The color of sound wood also varies strongly. Thus, it can be presumed that shape and texture measures could be utilized. It has also been demonstrated that simple texture operators can be successfully utilized in enhancing the accuracy of a color-based defect classification system [11]. Parquet inspection therefore seems to be a good application for the proposed methodology. In classification, we used a non-segmenting block-based approach in which the images were divided into regular regions of 32×32 pixels. With this approach, some of the samples to be classified contained more than one type of surface. For example, samples with small defects usually contained lots of good wood. We were however willing to balance this accuracy versus speed trade-off towards speed. In the experiments, a set of 150 beech wood parquet slabs from ESPRIT-P21023 project were used [3, 4]. To obtain ground truth data, each sample was given a label indicating the type of defect in it (or, the absence of a defect). Defect types were assigned by experienced wood grading professionals. Even then, the resulting ground truth was not perfect. There were quite a few questionable region labels, and defect boundaries could not be found accurately in every case. Over half of the samples were marked as good wood. At the other extreme, some defect types, like streak and split, were represented by only a handful of samples. The total number of defect classes was 15. We selected 20% of all samples in each class for training purposes. A separate training set was needed to ensure an unbiased evaluation of the optimization results.
3. Features The features we used for defect classification were selected based on our previous experiences in visual inspection and texture research [3, 12, 13]. For texture description, we used simple statistics calculated in the spatial domain. Features included multi-dimensional distributions of co-occurrences and signed differences, local binary patterns, edge-based features and Hough transform features. As color features, we adopted color percentiles [14]. It should be noted that many other color and texture features have been successfully applied to real-time visual inspection where good performance has been verified in demanding tasks like textile web inspection [6]. However, in demonstrating the performance of the optimization approach there was no need to consider a large set of different features.
3.1. Color percentiles The color percentiles, albeit being very simple and computationally light-weight features, have been shown to perform well in visual inspection problems. In wood inspection, their discrimination performance has been shown to be quite independent of color space. Due to its computational simplicity and good performance in earlier wood inspection studies, we used the RGB space. [4] In the RGB color space, percentiles as such are sensitive to intensity changes, which may be an unwanted property. On the other hand, if illumination conditions can be properly normalized, as is often the case in industrial environments, the absolute intensity values carry lots of useful information. Invariance against intensity shifts in the RGB color space can be achieved by calculating differences between percentiles. Differences between percentiles in different channels can also be calculated. We made no assumptions as to which
Optimizing color and texture features for real-time visual inspection
3
of these types was the most suitable for our purposes. Instead, a set of 117 different types of percentiles was created including inter-channel and intra-channel percentile differences in addition to the absolute values.
3.2. Texture features The gray-level co-occurrence method is one of the best known and most widely used methods in texture analysis. We used the method of reduced multi-dimensional co-occurrence histograms [15, 16]. In this method, co-occurrences are collected into a single multi-dimensional distribution, which is then reduced to a onedimensional one with learning vector quantization (LVQ). In the experiments, eight displacement vectors corresponding to the eight-neighbors of a pixel were used, resulting in nine-dimensional co-occurrence distributions. The invariance of the co-occurrences against illumination changes can be enhanced by utilizing differences instead of the absolute intensity values. This leads us to signed gray level differences which have been shown to perform well in texture classification and wood inspection [13, 11]. Again, the eight-neighbors of a pixel were used and the resulting eight-dimensional distributions were reduced with LVQ. Although we did not consider optimizing the co-occurrence and signed difference features, they can serve as a reference for the LBP features. Optimization was not meaningful because the feature distributions were already in some sense optimally covered by LVQ. The local binary pattern (LBP) operator is a gray scale invariant simplification of the signed difference method [12]. By considering only the signs of the differences, invariance against any monotonic transformation of gray scale is achieved. The dimensionality of the feature space is reduced by representing the signs with binary numbers. An eight-dimensional space can thus be constricted to an eight-bit binary number. In addition to the basic LBP, we considered two variations, namely LBP8,1 and LBPu2 8,1 . LBP seemed to be a reasonable choice as it is computationally very cheap. It also provides intensity-independent pattern information that complements the color percentiles. The LBP operator has also been successfully applied to wood inspection [11]. Some defect types, especially cracks, are difficult to detect using color or the aforementioned texture features. The main reason may lie in the fact that cracks typically occupy only a small portion of the sample area. Texture or color statistics may thus be quite inaccurate. For this reason, we also tested features based on contrast edges and on the Hough transform.
4. Feature set optimization If all the aforementioned features were used in classification, both computational performance and classification accuracy would have been poor. Since the classification time increases linearly with the number of features, it is advantageous to discard “useless” features. Furthermore, discarding redundant information often enhances classification accuracy. Thus, the goal for feature set optimization is to reduce feature space dimensionality while not sacrificing classification accuracy. Three feature sets, namely color percentiles, local binary patterns and edge features, were optimized. The reason we thought LBP distributions could be subjected to optimization was that subsets of LBP codes have been shown to perform equally well or even better than the whole histogram with homogeneously textured surfaces [17]. To find the global optimum for color-texture descriptions, the features should have been optimized jointly. Nevertheless, because of the exhaustive number of possible feature combinations, we chose to optimize color and texture features separately. Furthermore, some recent empirical studies suggest that color information and texture patterns should be processed separately [9]. For optimization, a deterministic and a randomized optimization algorithm were considered. As a performance measure in the optimization procedure, a leave-one-out classification test was run on the training samples.
4
Topi M¨ aenp¨ aa ¨, Jaakko Viertola, and Matti Pietik¨ ainen
Table 1. The beam search procedure • Create set S so that |S| = 1 and s1 = • Loop
.
– Create an empty set L. – For feature set index b = 1 to |S| – Construct S 0 , the set of all possible feature sets s0i , by appending each feature in a − sb to sb in turn. – Set L = L + S 0 , remove duplicates. – End b loop – Obtain the classification errors against the training set for each feature set in L. – Order L by classification result in descending order. – Set S = {li | i = 1, . . . , B}. • until a stopping criterion is met.
4.1. Beam search Beam search was used for deterministic optimization. In this method, we iteratively increase the size of the feature set until a stopping criterion is met and only consider the best B feature sets produced so far. In the literature, B is typically called the “beam width”. In the following semi-formal description, a set of feature sets is denoted by a capital letter (e.g. S), and each feature set in this set by si , i = 1, . . . , |S|, where |S| denotes the number of items in set S. The set of all features is denoted by a. The pseudo-code in Table 1 describes the beam search procedure. The value for B was chosen to be five for it seemed to provide a good balance between computational burden and optimization accuracy. For statistical features like the LBP, discarding part of the features (histogram bins) would have been pestilent because the total number of entries in the histogram would have changed. To keep the number of entries constant, we collected all unused features into an additional histogram bin. Consequently, the number of features in search step i was actually i + 1. The maximum number of features to be selected needs closer consideration. As always, there is the learning accuracy versus generalization capability trade-off. As noted before, long feature vectors require more computation in the classification phase. At least in the case of color percentiles, they are also likely to contain a lot of redundant information. Therefore, the number of selected features should be significantly smaller than the number of candidate features. To limit the length of the resulting feature vector, a simple stopping criterion was employed. The optimization was stopped if the classification accuracy did not improve by more than 0.3 percentage units per iteration, averaged over the last five iterations. We call this situation saturation. As an optimization result, the feature set that performed best when saturation began was selected.
4.2. Genetic algorithm As an alternative to the deterministic approach, we also considered an optimization method based on the principles of genetic algorithms. The advantage in a randomized approach is that the whole feature space can be covered at once. On the other hand, there are quite a few tunable parameters. The parameters used are listed in Table 2. An individual in the population was represented as an N -bit binary number where each “one” indicated an enabled feature. N denotes the number of possible features. In selecting survivors and parents for a new population, individuals (sets of enabled features) were ordered in descending fitness order. “Fitness” in this case includes the classification result on the training set and a penalty term that favored short feature vectors. To be exact, the penalty term was defined to be 0.3 percentage units for each active feature. The first individual in the sorted population was selected as a survivor into the next generation with
Optimizing color and texture features for real-time visual inspection
5
Table 2. Parameters for genetic algorithm Parameter
Value
Population size Survivor selection Survivor count Mating selection Mutation probability
20 rank-based (p = 0.5) 4 rank-based (p = 0.5) 0.01
Table 3. Optimized feature vector lengths Feature
Original
Beam
GA
117 256 256 21
8 1 1 1
23 84 76 6
Color percentiles Basic LBP LBP8,1 Edge features
probability p = 0.5. If it was not selected, then the next one had a chance with the same probability, and so on. 20% of the population were selected directly for the next population, and the rest were filled in with new individuals. Individuals for mating were selected with the same procedure that was used in selecting survivors. Random crossing-over operations were performed on the “genes” of the selected parents. Finally, one percent of the genes in the population were randomly changed from 0 to 1 or vice-versa. The optimization was initialized with a random population, i.e. a set of randomly selected feature sets. The algorithm was stopped after a fixed number of 50 iterations, for the performance seemed not to improve beyond that point.
4.3. Optimization results In all cases, the beam search resulted in a shorter feature vector than the genetic algorithm (GA). The number of selected features for both optimization methods are listed in Table 3. As color features, the beam search selected three absolute percentiles, and five inter-channel differences. The genetically optimized set contained 11 absolute percentiles, one intra-channel difference, and 11 inter-channel differences. Three of the features were present in both sets, and four others were in some sense close to each other. For example, the 90% percentile of the blue color channel histogram (B(90)) was selected by the beam search. The genetic algorithm selected B(95). With both optimization methods, the reduction in feature space dimensionality was significant. In a setting where all defect types were separated in one pass, the edge and Hough transform features turned out to be useless. The beam search procedure did not gain any noticeable performance enhancement in five iterations, and finished with only one feature enabled. The genetic algorithm selected six features, but showed no considerable enhancement in classification accuracy. The LBP distribution, being a statistical texture measure, posed a difficult problem to the optimization algorithms. One of the patterns, namely the flat area and dark spot detector (255), which typically occupies a large portion of image data, showed superior performance over any other single LBP code. Adding five other codes to the feature distribution did not significantly enhance the discriminative power of the LBP histogram. Thus, the beam search exited with only one LBP code enabled. The genetic algorithm could not do any better with 76 codes enabled, even though code 255 was present in the set. Thus, it seems that in this particular application with very heterogeneous textures, optimized features do not produce as good results as the original LBP distribution.
5. Classification For classification, we chose a simple non-parametric approach. The k-NN classifier with k = 3 was used for its capability of handling highly nonlinear decision surfaces without prior knowledge of feature distributions.
6
Topi M¨ aenp¨ aa ¨, Jaakko Viertola, and Matti Pietik¨ ainen
Table 4. Classification error percentages Features
Beam
GA
9.7
10.2
16.3 13.0 12.5 11.8 10.0 9.6 8.8 9.7 9.6
14.5 13.7 13.2 12.5 10.1 9.2 8.6 10.7 10.3
Optimized color with unoptimized texture Hough and edge 16.6 Basic LBP 9.3 LBP8,1 8.9 u2 LBP8,1 8.6
16.5 9.6 9.4 9.2
Color percentiles alone Optimized color and texture features Hough and edge Co-occurrences (32 code vectors) Co-occurrences (64) Co-occurrences (128) Signed differences (32) Signed differences (64) Signed differences (128) Basic LBP LBP8,1
As a proximity measure for all but the LBP features, Euclidean distance was used. Because of the statistical nature of the LBP features, we utilized a log-likelihood dissimilarity measure [12]. The combination of statistical feature distributions and non-statistical feature vectors causes slight difficulties as the dissimilarities take very different value ranges. To solve this problem, we employed the standard method of scaling with average values. The dissimilarity between each pair of samples in the training set was calculated. The mean of these dissimilarities was then used as a scaling factor for each feature vector or distribution. The final dissimilarity between two samples was then defined to be the sum of the normalized dissimilarities [11]. To reduce the effect of mislabeled training samples, we cleaned the training set by running a leave-one-out test on it. All samples in the training that could not be correctly classified by the final optimized features were discarded. As a result of this cleaning, slightly different training sets were used for each feature combination.
6. Results The final results were obtained by classifying the samples that were not used in optimization, against the cleaned training set. There was a total number of 21491 testing samples. Classification errors for combined color and texture measures are listed in Table 4. The sole color percentile features achieved an error rate of 10.2% after optimization with the genetic algorithm. The beam search did even better with less features as the error rate was only 9.7%. With all color features enabled, the error rate was 10.7%. It can be seen that the performance of joint color-texture features improves as the correlation between color and texture descriptions gets smaller. In edge features and co-occurrences, the absolute intensity information is still present, which may be the reason for the fact that the classification accuracy is even worse than that of the sole color features. A strong correlation between color and texture features may also affect the optimization results as an assumption of independence was made when separating the feature spaces for optimization. Of all features, the signed differences quantized with 128 code vectors, alongside LBP u2 8,1 , perform best. The beam search seems to give a more compact set of features than the genetic algorithm. In most cases, the beam search resulted in better classification accuracy with less features. However, there is no rule without an exception. Namely, signed differences seem to yield better results with genetically optimized color features. The optimization of LBP features is not needed in this application, as the classification results with unoptimized feature distributions seem to be clearly better.
Optimizing color and texture features for real-time visual inspection
7
Table 5. Computational performance figures Operation Color percentiles Co-occurrences (32) Co-occurrences (64) Co-occurrences (128) Signed differences (32) Signed differences (64) Signed differences (128) Basic LBP LBP8,1 LBPu2 8,1 Hough transform Edge features
Speed (samples/s) 2000 23 14 8 19 10 6 15000 2300 2200 5 3
7. Computational performance As a representative example of a real-world inspection situation, an inspection system for the parquet production line presented in [4] should be capable of processing 4374 samples per second. The speed requirements are demanding, and they can only be met with parallel processing or special hardware. The color percentiles alone use the full capability of a modern general purpose cpu. The texture measures, except for the LBP, require even more processing. Table 5 summarizes the performances of different features. Execution times were measured on a Sun UltraSPARC-II 450 MHz cpu. The performance of an operation is measured in samples per second, the size of a sample being 32×32 pixels. For texture operations, it is assumed that a gray scale image is provided. With color percentiles, it is assumed that 10 percentiles are calculated for each color channel. The operations were implemented in C++, and in most cases there is still plenty of room for speed optimizations. In particular, the implementation of vector quantization in the co-occurrence and signed difference methods could be greatly optimized. On the other hand, the implementation of the LBP operator is a highly optimized one. The results indicate that in addition to color, only the LBP operators are applicable to a real-time setting. Even though the other texture features were simple as well, they still need a large amount of computational resources compared to the color percentile features. In detailed defect analysis their use might however be justified. We were able to implement the simultaneous calculation of both color percentiles and LBP texture features in real time on a Sun Enterpise server with four UltraSPARC-II 450MHz processors, showing that even general-purpose CPUs have enough computational power for a real-time setting.
8. Discussion and conclusions In this study, we have developed a method of combining and optimizing color and texture features for realtime use. Optimized color feature vectors seem to work well in parquet defect classification. The classification accuracy was enhanced while the length of the feature vector was reduced from 117 to 8. With most texture descriptors, however, the attained utility is not significant enough compared to the large computational needs. It turns out that even most of the simplest texture measures are of no practical value in a real-time environment as they are computationally very demanding compared to the simple but powerful color features. In some cases, texture measures even deteriorate classification accuracy. However, even the reported small increments in inspection quality are important, as the production volumes in the wood industry are huge. Even though it seems that in this particular application, the proposed feature set optimization methods do not provide good results with the LBP, they work very well with the color percentiles. However, the beam search method has been shown to give reasonable results also for LBP distributions with more homogeneous textures [17]. The accuracy of Hough transform and edge-based features was also enhanced, even though their overall performance was somewhat disappointing. The beam search method resulted in much more compact feature sets than the genetic algorithm, and did not lose in classification accuracy either; it therefore seems that in most cases it is the preferred choice. It can be argued, though, that for statistical features, the genetic
8
Topi M¨ aenp¨ aa ¨, Jaakko Viertola, and Matti Pietik¨ ainen
algorithm could be better as it does not start with a statistically unreliable distribution containing only one entry. There are however quite a few tunable parameters that must be selected with care. It should be emphasized that the proposed optimization methodology is limited neither to parquet inspection nor the selected color and texture features. It is intended to be a general way of reducing the dimensionality of feature spaces for speed and accuracy.
References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]
Newman T and Jain A. A survey of automated visual inspection. Computer Vision and Image Understanding 1995; 61(2): 231–262. Boukouvalas C, Kittler J, Marik R, and Petrou M. Color grading of randomly textured ceramic tiles using color histograms. IEEE Transactions on Industrial Electronics 1999; 46(1): 219–226. ESPRIT Project 21023. Catie - colour and texture inspection equipment. http://www.ee.oulu.fi/Catie/. Kauppinen H. Development of a color machine vision method for wood surface inspection. Dr. tech. dissertation, University of Oulu 1999. http://herkules.oulu.fi/isbn9514254244/. Baykut A, Atalay A, Er¸cil A, and G¨ uler M. Real-time defect inspection of textured surfaces. Real-Time Imaging 2000; 6(1): 17–27. Kumar A and Pang G. Defect detection in textured materials using Gabor filters. IEEE Transactions on Industry Applications 2002; 38(2): 425–440. Jain A and Healey G. A multiscale representation including opponent color features for texture recognition. IEEE Transactions on Image Processing Jan. 1998; 7(1): 124–128. Mirmehdi M and Petrou M. Segmentation of color textures. IEEE Transactions on Pattern Analysis and Machine Intelligence 2000; 22(2): 142–159. M¨ aenp¨ aa ¨ T, Pietik¨ ainen M, and Viertola J. Separating color and pattern information for color texture discrimination. In 16th International Conference on Pattern Recognition. Qu´ebec, Canada. Marszalec E and Pietik¨ ainen M. Colour analysis of defects for automated visual inspection of pine wood. In SPIE 1907, Machine vision applications in industrial inspection II. San Jose, California, USA, pp. 80–94. Kyll¨ onen J and Pietik¨ ainen M. Visual inspection of parquet slabs by combining color and texture. In IAPR Workshop on Machine Vision Applications. Tokyo, Japan, pp. 187–192. Ojala T, Pietik¨ ainen M, and M¨ aenp¨ aa ¨ T. Multiresolution gray scale and rotation invariant texture analysis with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002; 24(7). Ojala T, Valkealahti K, Oja E, and Pietik¨ ainen M. Texture discrimination with multidimensional distributions of signed gray level differences. Pattern Recognition 2001; 34: 727–739. Silv´en O and Kauppinen H. Color vision based methodology for grading lumber. In 12th International Conference on Pattern Recognition. Jerusalem, Israel, vol. 1 - Conference A: Computer Vision & Image Processing, pp. 787–790. Valkealahti K and Oja E. Reduced multidimensional cooccurrence histograms in texture classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998; 20: 90–94. Ojala T, Pietik¨ ainen M, and Kyll¨ onen J. Gray level cooccurrence histograms via learning vector quantization. In 11th Scandinavian Conference on Image Analysis. Kangerlussuaq, Greenland, pp. 103–108. M¨ aenp¨ aa ¨ T, Ojala T, Pietik¨ ainen M, and Soriano M. Robust texture classification by subsets of local binary patterns. In 15th International Conference on Pattern Recognition. Barcelona, Spain, vol. 3, pp. 947–950.
