Applying neural networks to automated visual fruit sorting - CiteSeerX

Applying neural networks to automated visual fruit sorting A. S. Simões; A.H. Reali Costa; A. R. Hirakawa and A. M. Saraiva 1

Abstract A common problem in fruit production systems is sorting and classification. A usual procedure to carry out this task is based on human visual inspection considering general fruit attributes like color and size. Color contains important information about fruit status and in some cases it is decisive for fruit quality differentiation. An adequate color classification can improve system accuracy and productivity. Large-scale utilization of automatic classification system for this purpose demands a robust color classification even under different color saturation, variations of environment lighting and light reflections. This paper provides an investigation on the applicability of color classification using an artificial neural network in the fruit-sorting domain. Using the well-known network generalization property we investigate the applicability of this approach to the segmentation of colored images represented by the RGB color system. Jointly with color analysis, we also use some shape analysis to generate a robust and real time system that was tested for orange classification according to a Brazilian standard and which was able to provide fruit classification under less restricted visual conditions. Keywords: machine vision, neural networks, agriculture automation, color inspection

Introduction Two main characteristics are decisive for visual inspection and classification of fruits: color and shape. An efficient autonomous system for fruit sorting must be able to adequately identify both parameters. Fruit’s shape can easily be obtained from a digital image using classical techniques for image processing. However, although apparently immediate for humans, color identification involves many physical and psychological concepts, making it difficult to properly model and process color in an image. In this work we focus on the automation of the color classification problem applied to orange sorting. When a human being observes wavelengths included in the range of visible light, he immediately groups them into classes, using psychological notions learned for color separations. In this way, the color classification process can be seen as an application of highly non-linear functions into a digital image function. A digital (discrete) image function f(x) is a mathematical representation of an image, given by f(x,y) = {fred(x,y), fgreen(x,y), fblue(x,y)}, where f(x,y) represents image color and brightness at a spatial coordinate (x,y), named pixel. In a previous work we have modeled, implemented and tested a color classification system based on Artificial Neural Networks (ANNs), which was applied to the robotic soccer domain. This system was investigated regarding its generalization to visual 1

Department of Computer Engineering and Digital Systems, Escola Politécnica da Universidade de São Paulo, São Paulo, Av. Prof. Luciano Gualberto – trav.3 – 355, Brazil. {assimoes, anna, arhiraka, amsaraiv}@pcs.usp.br. WORLD CONGRESS ON COMPUTERS IN AGRICULTURE AND NATURAL RESOURCES

situations not given before in the training universe (Simões, 2000). We also have estimated the performance of the classificator considering different color representation systems, which led us to the conclusion that the RGB color representation system is the one that presents the best accuracy in classification (Simões and Reali Costa, 2000a,b,c). These results corroborate with some previous works found in technical literature (Littman and Ritter, 1997). Motivated by the promising results achieved by our previous work, in this paper we investigate the use of the ANN color classification system to the orange sorting problem. In the next section we present an overview of the orange sorting and classification problem considering the agricultural domain. In the following section we describe the ANN color classification system used to solve this problem. Procedure and results are presented in the section that follows and finally we present our conclusions.

Orange sorting and classification According to Brazilian fruits classification patterns (Centro de Qualidade em Horticultura, 2000), oranges must be classified in several different aspects, such as size, color, and blemishes. Five different orange classes are defined: a) C1, corresponding to dark green; b) C2 to light green; c) C3 to yellow; d) C4 to light orange; e) C5 to dark orange. These orange classes are shown in Figure 1. Besides that, it is necessary to identify if the presence or absence of blemishes which is a reason for classifying as a rejected fruit. Two accepted orange of different classes and a rejected one (i.e., a blemished orange) are shown in Figure 2.

Using ANN for color classification Let us consider Figure 2. In this picture we can find oranges of three different classes sampled by a digital camera over a white background. In Figure 2a and 2b we can see a C3-class and a C1-class oranges, respectively, and in Figure 2c we can observe a typical rejected orange. Transformed to the RGB color space these figures remain as shown in Figure 3a, 3b and 3c respectively. In these figures the spatial location of the pixels in the original images is unknown and each point is pictured using its own color, that is, the coordinates of pixels are also its colors. Under a range of proper illumination conditions, the groups of colors (orange, green, brown, etc.) can be easily separated by edges. The problem of color classification can thus be seen as a problem of determination of optimum edges capable of a suitable partition of an RGB color space. These edges – capable of processing this separation – have some special characteristics: The edges are not necessarily regular; The edges of each class are not necessarily of same size; The edges must have some generalization level in such a way that pixels with small variations in color illuminations and saturation are evolved by the same edge. In order to fulfill these requirements, we propose the use of an ANN multilayer perceptron (MLP), trained using the back-propagation algorithm (Simões, 2000). The adopted network used is shown in Figure 4. In this network model, there are 3 input neurons (that receive the triple of color representation of each pixel in a frame), one hidden layer with 10 neurons, and 7 output classes, corresponding to dark green, light green, yellow, light orange, dark orange, blemished and white as a background class. To train the network we extracted some pixel examples of the typical colors (dark WORLD CONGRESS ON COMPUTERS IN AGRICULTURE AND NATURAL RESOURCES

green, light green, yellow, light orange, dark orange and white) using some frames with a graphical interface. A frame corresponds to a digital image of oranges in white background. Once trained, the totality of pixels in a frame was presented to the network (pixel by pixel). The network returned all image pixels classifieds as one of the system typical colors. The network classification was stored (pixel by pixel) and we got an output image from the initial frame. As shown in previous works (Simões, 2000), simple examples of the colors without brightness or saturation examples are enough to obtaining a satisfactory classification performance (about 97% with low illumination and color saturation variations) with low computational cost.

Oranges classification The totality of pixels of an orange image does not necessarily belong to the same color class. Since the model of Brazilian fruits classification was defined to color classification conducted by humans, there is no exact definition of the deviation ranges applied to each class. In order to adequately model orange classes, some typical oranges (classified by humans) of each class were presented to the network (pixel by pixel). Observing its classified images we obtained the percentage of each system color present in each orange class. Typical color vectors for all 5 classes (C1 to C5) are shown in table 1. So, the process of orange classification can be seen as a problem of vector approximation. The exception to this rule is the rejected orange: an orange that presents a minimum level of blemished pixels was considered a rejected orange.

Procedure and results Using the X-lib interface under a Linux system we developed a graphical interface for on-line viewing the images captured from a camera. Based on previous works of Scardua et. al, (1999) we developed a MLP class. For each new frame, the system looks for a colored region (fruit) using the well-known region-growing algorithm (Ballard and Brown, 1982). Once located the fruit, the system is able to classify the pixels of the region and to analyze its color composition. The colors vectors found were compared with the five previously stored patterns and the fruit was classified as belonging to the class that minimizes the distance from its color vector. After classification process, a visual indicator of the orange class is added to the fruits pixels in the graphical interface as shown in Figure 4. The rectangle color represents the chosen orange class. Testing the robustness of this approach, 5 oranges of each class classified by a human were presented to the system. The classification errors for each fruit class can be seen in Table 2. The average percentage of correct classification is about 93%. Errors in classification typically occurred in the following situations: oranges were spatially located in regions with highly non-homogeneous illumination; the correct orange classification was uncertain, also for humans.

Conclusions In this paper a new methodology for orange inspection and sorting was presented and implemented. The amount of color classes depicted in orange images was identified using an artificial neural network MLP with the error backpropagation algorithm. The

WORLD CONGRESS ON COMPUTERS IN AGRICULTURE AND NATURAL RESOURCES

obtained vector of colors depicted in a frame was compared to typical color vectors (defined by humans) of each orange class. The use of an artificial neural network as a color classificator allows a robust classification even under orange color saturation variations, brightness, and non-homogeneous ambient illumination conditions. The approach has proved to be robust with respect to color variations and consequently highly applicable to the proposed domain. It also can easily be applied to sorting systems of other fruits, such as apples, papayas, lemons, etc. Application and test of this methodology for sorting other fruits remains as future trends of this work. However, the approach presents several characteristics that can restrict future domains: all colors of the domain must be previously known; the network must be trained to each new color presented to the system; the computational cost is high. As shown, the detection of damaged parts of an orange can be a complex task if we have to consider all kind of damages. An approach to avoid network training for each new color is the adoption of color filters previously to the color processing for certain types of rejects. A second observation concerning this system is the necessity of turning the fruits for inspection, since only half side of an orange is under analysis in a single frame.

Acknowledgement Alexandre da Silva Simões was partially financed by CAPES (Escola Politécnica da Universidade de São Paulo), from 1998 to 2000. This project was partially financed by FINEP, RECOPE-MANET project grant number 77.97.0937.00 and by NSF/CNPQ ProTeM-CC project MAPPEL, grant number 680033/99-8.

References Centro de Qualidade em Horticultura – CEAGESP. 2000. Classificação da laranja – programa de adesão voluntária. São Paulo..(Orange classification – voluntary adoption program) Littman, E. and Ritter, H. 1997. Adaptive color segmentation – A comparison of neural and statistical methods. IEEE Transactions on Neural Networks. Vol. 8, N. 1, January. Scardua, L. A. and Reali Costa, A. H. and Da Cruz, J. J. 1999. Learning to Behave by environment reinforcement. International Workshop on Robocup (IJCAI’99). Stockholm, Sweden, IJCAI Press, p. 181-186. Simões, A. S. 2000. Segmentação de imagens por classificação de cores: uma abordagem neural. Dissertação de Mestrado. Escola Politécnica da USP. São Paulo.(Image segmentation by color classification: a neural approach. Master Thesis) Simões, A. S. and Reali Costa, A. H. 2000 (a). Segmentação de imagens por classificação de cores: uma abordagem neural para a representação RGB. In: Workshop de Computação WORKCOMP'2000, Carlos H. C. Ribeiro & Milton T. S. WORLD CONGRESS ON COMPUTERS IN AGRICULTURE AND NATURAL RESOURCES

Sakude (eds.), Instituto Tecnológico de Aeronáutica - ITA, São José dos Campos, SP, Pg.25-31. Simões, A. S. and Reali Costa, A. H. 2000 (b). Classificação de Cores por Redes Neurais usando a Representação HSV. International Joint Conference IBERAMIA'2000 and SBIA'2000, Open Discussion Track Proceedings, M. C. Monard & J. S. Sichman (eds.), November 19-22, 2.000, pp. 314-323. Simões, A. S. and Reali Costa, A. H.. 2000 (c). Using Neural Color Classification in Robotic Soccer Domain. International Joint Conference IBERAMIA'2000 and SBIA'2000, Workshop Proceedings, Meeting on Multi-Agent Collaborative and Adversarial Perception, Planning, Execution, and Learning. L. N. Barros, R. M. Cesar Jr., F. G. Cozman, A. H. Reali Costa (eds.). São Paulo, ISBN 85-901664-1-4, November 2.000, Atibaia, SP, Brazil, pp.208-213.

Tables Dark green Light green Yellow Light orange Dark orange

C1 80.3% 12.8% 0% 6.2% 0.5%

C2 44.5% 9.6% 13.4% 32.3% 0.1%

C3 0.4% 0% 52.0% 47.1% 0.3%

C4 0.1% 0% 44.3% 55.2% 0%

C5 4.3% 0% 0% 34.5% 58.3%

Table 1. List of numerical percentages of colors included in each orange class.

Class C1 C2 C3 C4 C5 Rejected

% Error 0 20 0 0 20 0

Table 2. Classification errors for each orange class (when compared with the same oranges classified by humans).

Figures

(a)

(b)

(c)

(d)

(e)

Figure 1.Orange class patterns: a) C1; b) C2; c) C3; d) C4; e) C5.

(a)

(b)

(c)

Figure 2.Orange images captured with a camera: a) C3 class; b) C1 class; c) rejected. WORLD CONGRESS ON COMPUTERS IN AGRICULTURE AND NATURAL RESOURCES

(a)

(b)

(c) Figure 3.The oranges of Figure 2 in the RGB color space.

white Dark green R

Light green

G

yellow

B

.. .

Light orange Dark orange blemished

Figure 4.The artificial neural MLP network trained for color recognition.

Figure 5. A typical system screen: three classified oranges.

WORLD CONGRESS ON COMPUTERS IN AGRICULTURE AND NATURAL RESOURCES