This paper presents a defect segmentation work for bi-colored apple fruits performed by several artificial neural networks. Pixel-wise classification approach is ...
Apple Defect Segmentation by Artificial Neural Networks
Devrim Unay a
a
Bernard Gosselin
a
TCTS Lab., Facult´e Polytechnique de Mons, Belgium Abstract
This paper presents a defect segmentation work for bi-colored apple fruits performed by several artificial neural networks. Pixel-wise classification approach is employed to realize segmentation. Quantitative and qualitative evaluations showed that competitive networks were more erroneous while feed-forward and recurrent networks tested were more accurate in segmenting apple defects.
1
Introduction
Quality of apple fruits depends on size, color, shape and presence-type of defected skin according to the marketing standard of European Commission [1]. Visual inspection of apples with respect to size and color by machine vision is already automated in the industry. However, detection of defects is still problematic due to high variance of defect types, presence of stem/calyx concavities and natural variability of skin color. Defect defection, in particular, requires their precise segmentation. Majority of the works performing defect segmentation of apples used simple thresholding [4, 5, 13, 14, 17]. Wen and Tao [20] applied an adaptive spherical transform before simple thresholding to eliminate boundary light reflectance effect caused by the spherical shape of the fruit. Kim et al. [8] used globally adaptive thresholding (modified version of Otsu’s algorithm) to segment fecal contamination defects on apples. Region-based techniques segment images by finding coherent, homogeneous regions subject to a similarity criterion. Yang [21] used a region-based technique (flooding algorithm) to segment patch-like defects on apples. Classification-based techniques attempt to partition pixels into several classes using different classification methods. Bayesian classification is the most used method by researchers [3, 10, 12, 15], where pixels are compared to a pre-calculated model and classified as defected or healthy. On the other hand, Nakano [16] introduced a neural network based system to classify pixels of apple skin into six classes, one of which was ‘defect’. Unsupervised classification does not benefit any guidance in the learning process due to lacking target values. Such an approach was used by Leemans et al. [11] for defect segmentation. Above literature survey reveals that in segmenting surface defects of apple fruits, researchers have mainly focused on global thresholding-based approaches and Bayesian-based classification methods. Furthermore, defect segmentation on apples is a problematic task due to several facts. Hence, in this paper we focus on segmentation of skin defects of apples using sophisticated classifiers; namely artificial neural networks (ANN).
2
Methodology
We propose to segment skin defects of bi-colored apples as in Figure 1. After describing the image acquisition and the database, we will explain this system as individual subsections.
2.1
Image Acquisition and Database
Imaging of fruits is performed in a diffusely illuminated environment by a multispectral system based on a high-resolution monochrome camera and four bandpass filters (centered at 450, 500,
Figure 1: Architecture of the system used for defect segmentation. 750, and 800 nm with respective bandwidths of 80, 40, 80, and 50 nm). Figure 2 displays an illustration of the image acquisition system, which is capable of one-view inspection only. Each filter image is composed of 430x560 pixels with 8 bits-per-pixel resolution.
Figure 2: Illustration of the image acquisition system used. 246 Jonagold apples with various naturally and artificially created defects (russet, bruise, rot, scald,. . . ) are tested in this study. Jonagold variety is selected, because it has a bi-colored skin causing more difficulties in defect segmentation due to color transition areas. Defected skins in the image database are manually segmented by experts and used as reference in this work. Figure 3 displays some examples from the database with related manual segmentations. Assembly of the image acquisition system and collection of the database were done in the Mechanics and Construction Department of Gembloux Agricultural University of Belgium. Therefore, for more details please refer to [9, 10]. 450nm
500nm
750nm
800nm
segmentation
Figure 3: Examples of apple images and related manual segmentations. First four columns present images from different filters, while the last one shows corresponding manual segmentations. Rows display apples with different defect types (Top to bottom: bruise, flesh damage and russet).
2.2
Region-of-Interest Definition
As observed in Figure 3, background is lower than the fruit area in intensity. Therefore, fruit area can be separated from background by thresholding the 750nm filter image at intensity value of 30. However, this fixed thresholding can falsely remove some defects, stems or calyxes that are lower in intensity. Hence, morphological filling is applied to eliminate this error. Initial observations revealed that segmentation was problematic at the far edges of fruits probably due to illumination artifacts. Hence, after background removal, fruit area is eroded by a rectangular structuring element with size adaptive to fruit size (15 % of fruit’s bounding-box). Result of this erosion step gives the region-of-interest (roi ) that defines the fruit area to be inspected.
2.3
Feature Extraction
Segmentation of defects at pixel level requires each pixel to be represented by features. Thus, intensity values of each pixel from four filter images form its local features. In our previous work [18] we showed that an additional local feature related to pixels’ location relative to geometric center of roi improved segmentation of defects on the same database. In addition to the local features, average and standard deviation of intensity values over the roi are also calculated from each filter image, making up the global features. Hence, each pixel is represented by 13 features in the feature space (Table 1). Feature values are also normalized to fall into the range of [-1,+1] before being introduced to the defect detection step. category local global
description intensity of pixel pixel’s location relative to roi center average of intensities in roi standard deviation of intensities in roi
quantity 4 1 4 4
Table 1: Details of features extracted for defect segmentation.
2.4
Defect Detection
Emerging from the studies on how animal brains work, ANNs are composed of layers of neurons gathered in a parallel architecture with a high degree of interconnection between them [2, 6, 7]. Among various ANN algorithms provided in the literature, we selected the following ones for defect segmentation. 2.4.1
Feed-Forward Networks
In a feed-forward network data propagates in the forward (from input layer to output layer) direction, thus its neurons have only unidirectional connections (no feedback or same layer neuron-toneuron connections). Perceptron Perceptron (or McCulloch-Pitts neuron) is the simplest form of neural networks that can only solve linearly separable problems. It is a single neuron with threshold activation function. Multi-Layer Perceptrons (MLP) MLP is composed of several perceptrons in layered structure. It performs back-propagation learning, where two passes of signals through the network are employed. Forward pass: Input signals are propagated in forward direction, while weights at each layer are fixed and actual output of the network is produced. Error between the actual output and the desired output (label) is calculated. In backward pass, this error signal is propagated backward and weights are adapted a second time. Cascade Forward Neural Networks (CFNN) CFNN is similar to MLP, except that the neurons of each subsequent layer have inputs coming from not only the previous layers but also input layer. This results in more interconnections than MLP.
2.4.2
Competitive Networks
In competitive networks, the output neurons compete among themselves for being the one to be activated. Competitive networks can learn to detect correlations between inputs and adapt themselves for that. These networks can also include lateral connections (same layer neuron-toneuron connections) among themselves. Competitive Neural Networks (CNN) The most straightforward example of competitive networks is the CNN, which is an unsupervised method that can learn to recognize groups of similar inputs. CNN has lateral connections among neurons. Learning Vector Quantizers (LVQ) LVQ is composed of a competitive layer followed by a linear layer (a layer of neurons with linear activation function), where the former learns to classify inputs and the latter transforms the outputs of the former into labels defined by user. Therefore, these networks benefit from supervised learning. Self Organizing Feature Maps (SOM) SOM (Kohonen maps) is a competitive network composed of one or two-dimensional lattice of neurons. In SOM, presentation of similar inputs lead to the response of the same group of neurons (physically close to each other), while others stay quiet. It is an unsupervised algorithm. 2.4.3
Recurrent Networks
Feed-forward networks allow progression of inputs only in the forward direction. However, introduction of a feedback path between layers can unveil exclusive information, which is implemented by recurrent networks. Due to the feedback path, information flows in both forward and backward directions, which makes recurrent networks dynamic systems. Elman Neural Networks (ENN) ENN is a two-layered recurrent network with feedback from the first layer output to the first layer input. In addition to the feedback connection, it also permits supervised learning. All the ANNs used in this study have 2-layered architecture with 5 neurons in the hidden layer except for SOM, which has a 5x8 hexagonal topology. Sigmoid neurons are used as long as architecture permitted. Levenberg-Marquardt algorithm, learning rate of 0.01 and maximum epoch number of 25 are used for their training. Above parameters are found optimum after several trials. 2.4.4
Training of ANNs
Pixel-wise segmentation leads to excessive amount of data to be processed, which is computationally very expensive. Thus we performed our tests by hold-out method, where 12 , 61 and 31 of the fruits of each defect type are placed in training, validation and test databases, respectively, providing 116 fruits for training, 38 for validation and 92 for testing. This separation is found optimum after several trials. Training set is composed of a set of pixels that are randomly selected from fruits in the training database. Furthermore, defect types and classes (healthy-defected) are equally represented in the training set. We paid special attention on discarding the pixels in the training set that belonged to the fruit being tested, in order to prevent possible forced training. Cross-Validation is a training method for supervised classifiers, where a portion of training set is separated as validation data and training of the classifier is done on training set while evaluation on validation set. Perceptron, MLP, CFNN and ENN permitted cross-validation, thus they are trained with this method.
2.5
Stem/Calyx Removal
As orientation of fruits were not controlled during image acquisition, stem and calyx parts of fruits are also visible in the images. Discrimination of these natural parts from real defects by image processing is necessary to achieve more precise segmentation. Recently, we have introduced a highly accurate support vector machines-based method to solve this problem [19]. Hence, we benefit from this method to remove stem/calyx regions from segmented areas and provide refined segmentation (Figure 4).
Figure 4: Example of stem/calyx removal. Before the removal on the left, and stem/calyx removed on the right. Defected area displayed in white.
2.6
Performance Evaluation
The popular recognition error assumes that classes are equally represented, which is not true for our case where defect sizes highly vary within the database. Hence, we measure performance of segmentation by calculating class-specific recognition error (CSRE) as follows: CSRE = where TP FP TN FN
FN T P +F N
+
FP T N +F P
2
(1)
(True Positives): number of defected pixels correctly detected (False Positives): number of healthy pixels incorrectly detected as defect (True Negatives): number of healthy pixels correctly detected (False Negatives): number of defected pixels incorrectly detected as healthy.
If a class does not exist (e.g. no defected skin, i.e. fruit in perfect quality), then the error is computed using the other class only. Note that this measure is calculated for each test image, whereas error of a test is estimated as the average of measures of all test images.
3
Results and Discussion
We will first introduce quantitative analysis of segmentation results estimated by CSRE measure. Figure 5 displays segmentation performances of all the artificial neural networks. We observe that generally competitive networks are more erroneous than the others. Perceptron is the most inaccurate method with its 0.526 error rate, probably because it can only solve linear problems by definition. Feed-forward networks slightly outperform the recurrent one (ENN) in accuracy, which may be due to the feedback connection. Concerning learning, supervised methods surpass unsupervised ones in general, which shows that supervision through labelled training samples results in more accurate segmentation of defects. We also performed qualitative analysis by visual comparison of segmentation results with respect to manual segmentations. Figure 6 displays some examples of segmentations executed by ANNs with corresponding manual segmentations. As observed from segmentations, Perceptron cannot find the defected skin at all and this is why its error rates were relatively very high. Coherent with the observations of quantitative analysis, segmentations of competitive networks (CNN, SOM and LVQ) are quite erroneous, whereas those of feed-forward (MLP and CFNN) and recurrent (ENN) networks are very promising in general. When we focus on the segmentations performed by feedforward and recurrent networks we observe over-segmentation in limb rub; under-segmentation in russet, rot and bruise; and very accurate segmentation in flesh damage defects. Precision in the latter is probably due to the defect that is clearly distinct from healthy skin in 450 and 500nm images (see Figure 3). As seen in its manual segmentation russet defect is relatively very complex in shape. Despite this complexity, MLP, CFNN and ENN achieve very encouraging segmentations for this defect type. We observe false segmentations (e.g. quasi-circular ring in limb rub), which are due to the concavities around the stem or calyx areas. Pixels at these parts have relatively low intensities because of varying illumination caused by the curvature.
Figure 5: Performances of artificial neural networks for defect segmentation.
4
Conclusion
Detection of defects on bi-colored apple fruits by image processing is a problematic task due to stem/calyx areas and diverse types of defects present, as well as natural variability of skin color. In this paper we addressed this problem using an approach based on pixel-wise classification of fruit skin by different artificial neural networks. Following classification, segmentations are refined by an SVM-based stem/calyx removal method. Final segmentations are examined by qualitative and quantitative analysis, both of which showed that competitive networks were more erroneous while feed-forward and recurrent networks were quite accurate for apple defect segmentation. Some of the false segmentations were due to the natural variability of fruit shape, while others were probably because of the complexity of defects. A machine vision-based inspection system has to assign fruits to the corresponding quality categories. Therefore, a necessary future work is to perform fruit grading based on defect segmentations achieved here. Furthermore, evaluation of segmentations are accomplished with respect to a single set of manual segmentations in this work. Hence, repeatability of segmentations should be tested with a new database.
5
Acknowledgements
This research is funded by the General Directorate of Technology, Research and Energy of the Walloon Region of Belgium with Convention No 9813783. We thank Prof. M.-F. Destain, O. Kleynen and V. Leemans from Gembloux Agricultural University of Belgium for their support.
References [1] Anonymous. Commission regulation (ec) no 1619/2001 of 6 august 2001 on marketing standards for apples and pears. Official Journal L 215, pages 0003–0016, August 2001. [2] C.M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press Inc., New York, USA, 1995.
CNN
SOM
LVQ
Perceptron
MLP
CFNN
ENN
Figure 6: Examples of segmentations achieved by artificial neural networks, where the first column displays manual segmentations. In each image healthy skin is displayed in white color, while defected areas in gray. From top-to-bottom, examples are of fruits defected by limb rub, russet, rot, bruise and flesh damage.
segmentation
[3] J. Blasco, N. Aleixos, and E. Molt´o. Machine vision system for automatic quality grading of fruit. Biosystems Engineering, 85(4):415–423, 2003. [4] T.G. Crowe and M.J. Delwiche. Real-time defect detection in fruit - part ii: An algorithm and performance of a protoype system. Transactions of ASAE, 39(6):2309–2317, 1996. [5] A. Davenel, Ch. Guizard, T. Labarre, and F. Sevila. Automatic detection of surface defects on fruit by using a vision system. Journal of Agricultural Engineering Research, 41:1–9, 1988. [6] R.O. Duda, P.E. Hart, and D.G. Stork. Pattern classification. John Wiley & Sons, Inc., 2nd edition, 2001. [7] S. Haykin. Neural Networks: A Comprehensive Foundation. Prentice Hall PTR, Upper Saddle River, NJ, USA, 1994. [8] M.S. Kim, A.M. Lefcourt, Y.-R. Chen, and Y. Tao. Automated detection of fecal contamination of apples based on multispectral fluorescence image fusion. Journal of food engineering, 71(1):85–91, 2005. [9] O. Kleynen, V. Leemans, and M.-F. Destain. Selection of the most efficient wavelength bands for ‘jonagold’ apple sorting. Postharvest Biology and Technology, 30(3):221–232, December 2003. [10] O. Kleynen, V. Leemans, and M.-F. Destain. Development of a multi-spectral vision system for the detection of defects on apples. Journal of Food Engineering, 69(1):41–49, 2005. [11] V. Leemans, H. Magein, and M.-F. Destain. Defect segmentation on ‘golden delicious’ apples by using colour machine vision. Computers and Electronics in Agriculture, 20(2):117–130, July 1998. [12] V. Leemans, H. Magein, and M.-F. Destain. Defect segmentation on ‘jonagold’ apples using colour vision and a bayesian classification method. Computers and Electronics in Agriculture, 23(1):43–53, June 1999. [13] Q. Li, M. Wang, and W. Gu. Computer vision based system for apple surface defect detection. Computers and Electronics in Agriculture, 36(2-3):215–223, November 2002. [14] P.M. Mehl, K. Chao, M. Kim, and Y.R. Chen. Detection of defects on selected apple cultivars using hyperspectral and multispectral image analysis. Applied Engineering in Agriculture, 18(2):219–226, 2002. [15] E. Molt´o, J. Blasco, and J.V. Benlloch. Computer vision for automatic inspection of agricultural produces. In SPIE Symposium on Precision Agriculture and Biological Quality, Boston, MA, USA, November 1998. [16] K. Nakano. Application of neural networks to the color grading of apples. Computers and Electronics in Agriculture, 18(2-3):105–116, August 1997. [17] J.A. Throop, D.J. Aneshansley, W.C. Anger, and D.L. Peterson. Quality evaluation of apples based on surface defects: development of an automated inspection system. Postharvest Biology and Technology, 36:281–290, 2005. [18] D. Unay and B. Gosselin. A quality sorting method for ‘jonagold’ apples. In Proc. Int. Agricultural Engineering Conf. (AgEng), Leuven, Belgium, September 2004. [19] D. Unay and B. Gosselin. Stem and calyx recognition on ‘jonagold’ apples by pattern recognition. Journal of Food Engineering, to appear, DOI: 10.1016/j.jfoodeng.2005.10.038. [20] Z. Wen and Y. Tao. Building a rule-based machine-vision system for defect inspection on apple sorting and packing lines. Expert Systems with Applications, 16(3):307–313, April 1999. [21] Q. Yang. An approach to apple surface feature detection by machine vision. Computers and Electronics in Agriculture, 11(2-3):249–264, March 1994.
