Since manual labor is costly and expensive, an automated weed control system may be economically ... structure, which includes a sprayer, a Logitech web digital camera, 12v motor coupled with a pump system ... Malaysia is a world leader in.
American-Eurasian J. Agric. & Environ. Sci., 3 (3): 451-458, 2008 ISSN 1818-6769 © IDOSI Publications, 2008
Machine Vision System for Automatic Weeding Strategy using Image Processing Technique Kamarul Hawari Ghazali, Mohd. Marzuki Mustafa and Aini Hussain Faculty of Engineering, Universiti Kebangsaan Malaysia, 43600, Bangi, Selangor, Malaysia Abstract: The most widely used method for weed control is to use agricultural chemicals (herbicides products). This heavy reliance on chemicals raises many environmental and economic concerns, causing many plantation companies to seek alternatives for weed control in order to reduce chemical usage in their plantation. Since manual labor is costly and expensive, an automated weed control system may be economically feasible. A machine vision precision automated weed control system could also reduce the usage of chemicals. In this research, an intelligent real-time system for automatic weeding strategy in oil palm plantation using image processing has been developed to identify and discriminate the weed types namely as narrow and broad. In machine vision technology, the main component of the system is image processing to recognize type of weeds. Three techniques of image processing, involving statistical approach GLCM and structural approach FFT and SIFT, have been used and compared to find the best solution of weed recognition for classification. The developed machine vision system consists of a mechanical structure, which includes a sprayer, a Logitech web digital camera, 12v motor coupled with a pump system and a small size CPU as a processor. Offline images and recorded video has been tested to the system and classification result of weed shows the successful rate is above 80%. Key words: Weed herbicide real time system GLCM FFT SIFT •
•
•
•
INTRODUCTION
•
climate. Due to the widespread lack of awareness of the hazards of herbicides, inhalation and skin absorption are the major causes of occupational poisoning cases among women sprayers. The survey results found that women sprayers are often not in good health. They suffer from acute and chronic ailments related to their work. The most common methods spraying pesticide in Malaysia is spraying the pesticide solution onto weed with knapsack sprayer. This technique has been proven to be inefficient as only about 20% of the spray reaches the plant and less than 1 percent of the chemical contributes to weed control, resulting in wastage and contamination of the environment. Health care considerations and environmental and economic factors are stimulating the development of sensors technologies for selective application of herbicide. Selective application of herbicides requires automatic detection and evaluation of weeds in the field [2]. Several methods are available for such automatic detection, among which are those based on machine vision [3-5]. Machine vision methods are based on digital images, within which, geometrical, utilized spectral reflectance or absorbance patterns to discriminate between narrow and broad weed. In [6], machine vision methods have
The plantation industry is a crucial part of the country’s development. Malaysia is a world leader in palm oil production and the cultivation of the crops is a major agricultural activity in the country [1]. The common herbicides used in oil palm plantation are dim ethylamine, diuron, DSMA (disodium methylarsonate) and paraquat. In 2004, Consumer Association of Penang (CAP) conducted a study of 11 oil palm plantations located in the northern states of Malaysia. The study focused on women herbicide sprayers, their working conditions and the consequent health impacts. Work on an oil palm plantation is backbreaking and hazardous. Women herbicide sprayers are expected to carry an 18-litre (4-gallon) drum containing herbicide and complete 14 to 16 rounds of spraying per day. Tractor spraying is also conducted on some plantations, where big drums of herbicide are placed on both sides of the tractor. Two women carry the pumps and spray as the tractor moves. In either case, the sprayers themselves are engulfed in a fine mist of herbicide. Recommended safety measures are rarely employed. The use of protective masks, gloves and boots is often impractical owing to the hot and humid tropical
Corresponding Author: Dr. Kamarul Hawari Ghazali, Faculty of Engineering, Universiti Kebangsaan Malaysia, 43600, Bangi, Selangor, Malaysia
451
Am-Euras. J. Agric. & Environ. Sci., 3 (3): 451-458, 2008
In conclusion, machine vision system is based on digital image processing is found to be the best sensor detection as its operation is similar to the human eye. In other words, machine vision is the desire to provide machines and robots with visual abilities. The main objective of the research presented in this paper is to develop real time system that can interface with the mechanical structure spraying system to be used as a weeding strategy in oil palm plantation. Three image processing techniques were considered and implemented namely; Gray Level Co-Occurrence Matrix (GLCM), Fast Fourier Transform (FFT) and Scale Invariant Feature Transform (SIFT).
been used to show shape features can be used to discriminate between corn and weeds. Other studies classified the scene by means of color information [5, 7]. Most of the reported study did not maximize and thoroughly analyze the usage of image processing technique but rather used the general technique on standard RGB color and shape analysis. In the present study, the proposed technique such as structural analysis which transforms the original image to the values called coefficient which is found very robust to common problem in image processing such as lighting, range of object and size of image. Many image-processing techniques were employed to extract texture features from captured images in various applications [8-10]. In [11] GLCM has been used for quality inspection of strawberry and they used four feature extraction parameters of GLCM as a feature vector to grade the tomatoes in group ripe, over ripe and under ripe. In [12], surface roughness problem such as friction, contact deformation, heat and electric current conduction has been analyzed using GLCM associated with its features parameters maximum occurrence of the matrix (MOM), maximum occurrence position (MOP), standard deviation of the matrix (SDM) and maximum width of the matrix (MWM). GLCM as proposed by Harlick have often been used in texture classification or texture detection, but have very rare used in weed detection. Fourier Transform is a technique to analyze frequency components in a signal. It is common processing technique in signal analysis and image processing to transform the original data into form that easily to process. Many methods have been developed using Fo urier Transform, such as non uniform Fast Fourier Transform (2-D NUFFT) which is used in [13] to analyze micro strip circuit. In [14], they used fourier transform fundamental to improve imaging speed by using data-sharing method. In biomedical imaging, Fourier Transform has been used to reconstruct the magnetic resonance imaging to compensate for field in homogeneities [15]. Most common uses of fourier transformation is in signal analysis and image processing mainly in image enhancement which is frequency component of the signal can be analyze for further process [16-19]. The last technique discussed in this paper is SIFT. Scale Invariant Feature Transform (SIFT) has been proven to be the most robust local invariant feature descriptor [20]. SIFT is an approach for detecting and extracting local feature descriptors that are reasonably invariant to changes in illumination, image noise, rotation, scaling and small changes in viewpoint. The SIFT algorithm is widely used for object recognition and detection which is invariant to illumination changes and affine or 3D projection.
MATERIALS AND METHODS With the advancement of computer technology, machine-vision systems have become a possible solution for weed classification [2]. The main component in machine vision system is a digital image processing as well as its interfacing with hardware structure system. The overall process of system development can be described in Fig. 1. The automated sprayer system consists of an image acquisition, software engine, electronic interfacing circuit and
Narrow broad
Image acquisition Logitech web cam camera
Software engine Image processing algorithm
Electronic interfacing circuit Buffer IC, SSR
Sprayer module 12v motor dc, Pump Fan nozzle, 2 tanks for herbicides
Fig. 1: Block diagram of real time system prototype development 452
Am-Euras. J. Agric. & Environ. Sci., 3 (3): 451-458, 2008 Electronic board
co-occurrence matrix is a square matrix in which it needs to be transformed into a close approximation of a probability table. This process is called matrix normalization. Normalization involves dividing the square matrix by the sum of the values, where i and j are coordinates of the co-occurrence matrix space.
Camera
Pi , j =
Vi , j
∑V
i , =j 0
Sprayer nozzle
(1)
N −1
i,j
The key approach to extract feature vector of texture co-occurrence analysis is by defining contrast and regularity as feature vectors to represents the weed images. Contrast analysis can be formulated as:
Tank for herbicide
Fig. 2: The sprayer structure of automated weeding strategy
N −1
∑P
= C
i , =j 0
sprayer modules. In the image acquisition module, a Logitech webcam (640x480 pixels) has been used as an image collector. The original image is then transferred to the software engine module or image pre -processing and feature extraction. This is a critical module because it must comply with the real time image processing requirements. Three techniques, GLCM, FFT and SIFT have been developed to enable the recognition of weed as either narrow or broad weed. In real time system, there are many uncertainties due to lighting conditioning, movements and sizes of the object etc. A new algorithm has been developed in this work that can respond positively to real time condition and it has been successfully tested as an engineering prototype system. As a result of image processing in software engine, the control will respond to sprayer module using electronic interface circuit. This interface circuits used to ensure the data can be transfer efficiently. Finally, the sprayer module will respond due to algorithm in Software Engine. A prototype of real time sprayer structure weeding strategy has been shown in Fig. 2. The core component in real time vision system is the image processing algorithm and its function is to detect the target object. In this research, we introduce and compare three techniques namely as grey level cooccurrence matrix, fast fourier transform and scale invariant feature transform.
i,j
(i − j) 2
(2)
Where the i and j represent the row and column of pixel value, respectively. Regularity equation can be defined as:
R=
N −1
Pi,j
∑ 1 + (i − j)
i , =j 0
2
(4)
Fast Fourier Transform (FFT): The second technique that used in this research is FFT. The FFT is an algorithm that computes the Discrete Fourier Transform (DFT) of a function. The DFT F of f is defined as follows,
F(k) =
1 N−1 ∑ f(n)e − 2Πi k n/ N N n= 0
(5)
The DFT of a two dimensional function f(x, y), where x and y both ranges from 0 to N - 1, is defined as follows:
F(k,l) =
2 Πi − (kx + ly) 1 N −1 N−1 f(x,y)e N 2 ∑∑ N =x 0 =y 0
(6)
Generally the derivation of FFT comes from the DFT definition. Here, we derive the DFT formula and let f(n) be a function of n that spans from 0 to N - 1. We first separate f into functions f1 and f2 , where f1 = f(2n) and f2 = f(2n + 1), with n ranging from 0 to (N/2 - 1). In other words, we simply split f into two parts, one with domain even n and the other with domain odd n. The FFT formula can be described as
Grey Level Co-occurrence Matrix (GLCM): GLCM is a tabulation of how often different combinations of pixel brightness (grey levels) occur in an image. Basically, GLCM considers the relation between two neighboring pixels at a time called the reference and the neighbor pixel. The grey value relationships in a target are transformed into a co-occurrence matrix space. The 453
Am-Euras. J. Agric. & Environ. Sci., 3 (3): 451-458, 2008 N / 2− 1
∑ f (n)e
= F(k)
n =0
e
−2πi kn /( N/2)
1
−2πi k / N
(7)
N / 2− 1
∑f
m =0
The difference of these two g’s is known as the difference of Gaussian (DoG) and the selection of σ can be referred as implementation of scale space to the image.
+ ...
2
(m)e
− 2πi km /( N /2 )
g1 (x,y) − g 2 (x,y) = G σ1 *f(x,y) − G σ2 *f(x,y)
Fourier transform of an image produced a complex valued function, which must be translated into a real valued function to display as an image. We can either display the magnitude, the real part, or the imaginary part of the complex values. For a complex number a + bi, the magnitude is defined as
a 2 + b2
= (G σ1 − G σ2 )*f(x,y)
The DoG as an operator or convolution kernel is defined as
DoG = Gσ1 − G σ2
(8) =
Where the real part is a and the imaginary part is b. The 2D-FFT coefficients represent the high and low frequency values of the 2 dimensional images of both narrow and broad weed type. It is difficult to classify the 2D-FFT coefficient as the size of intensity value is the same as the original image 240x320. In the image enhancement application, these coefficients are used to remove the high frequency components which are assumed as noise in the original image. However, for classification purposes, these coefficients are useful to produce a set of feature vectors that can uniquely classify the target object.
= G σ1 (x,y)
x 2 + y2 exp − 2 2πσ12 2σ
k − 1= x 2 + x 1M o + (s − smin )M o No
1 ≤ x 2 ≤M o − 2 , 1≤ x1 ≤ N o − 2 And
s min + 1 ≤ s ≤ smax − 1
(15)
The orientation θ of a key-point (x,σ) is obtained as the predominant orientation of the gradient in a window around the key-point. Key-point representation is a histogram of the gradient orientations augmented and 2-D location in the support of the SIFT frame.
(9)
(x, θ) ∈ R 2 x R / Z
(10)
(16)
Prototype system: The overall system of automatic weeding consists of two main parts, software system and the sprayer structure. The image-processing algorithm (GLCM, FFT and SIFT) needs to be interfaced with the hardware to control motor pump.
With different width σ2 , a second smoothed image can be obtained:
g 2 (x,y) = G σ 2 (x,y)*f(x,y)
(14)
Maximum and minimum pixel value can be detected as follows:
To get:
g 1 (x,y) = G σ1 (x,y)*f(x,y)
1 1 − (x2 + y2) / 2σ12 1 − (x 2 + y2 ) / 2σ 22 (13) − e e σ2 2π σ1
The construction of SIFT-C feature vector from the key-descriptor involves the determination of magnitude direction of every element of the key-points. For each element of the SIFT key-descriptors, we calculate the magnitude of similar angle directions and put them into the appropriate histogram bin. The transformation of SIFT-C formula onto key-descriptor produces values that can be used to classify object. The SIFT-C formula can then be defined as such, SIFT local maximum and minimum - As the octave represented by a 3D array, the formula returns indexed k that are to be mapped to scale space indexes of
Scale invariant feature transform: The last technique describe in this paper is scale invariant feature transform. Sift technique starts with the implementation of the Gaussian Filter. First, the RGB color code images had undergone the basic preprocessing stage of color conversion to gray and subjected to the filtering technique of Difference of Gaussian (DoG) technique. The DoG is similar to the Laplace of Gaussian technique, in which the image is first smoothed by convolution with the Gaussian kernel of certain width σ1
1
(12)
(11) 454
Am-Euras. J. Agric. & Environ. Sci., 3 (3): 451-458, 2008
We have used direct interfacing parallel port with Matlab coding by using the data acquisition toolbox. The following Fig. 3 shows how the software system was integrated with the mechanical structure to become a complete real time system for automatic weeding. The prototype consists of web camera Logitech (Pixel; VGA, Resolution; 640x480 (VGA), 30fps@352x288 and image format; RGB) to capture a narrow and broad weed images and sent to the data acquisition system through parallel port, processed by Matlab 6.5 GUI program. The brains of software engine are the GLCM, FFT and SIFT techniques which process the images in order to distinguish and classify the both type of weeds. An electronic system that consists of an interfacing circuit (buffer IC and solid state relay) interfaces the motor pump to receive 5V TTL control signals from the software engine. The TTL signals gives input to solid state relay to control the speed of 12-volt DC 7A electric motor of the pump so as to apply the herbicide through a fan nozzle. The complete system assemble is as shown in Fig. 2 and can be fixed onto a tractor for spraying action.
(a)
RESULTS AND DISCUSSION
(b)
The prototype real time system was tested using a recorded video which was taken at a location in the Universiti Putra Malaysia oil palm plantation. The recorded video of narrow and broad weed is considered to be a real time data and interfacing has been made to read the playback video file through Matlab6p5. The recorded video of 30 minutes in duration comprising of narrow and broad weed was captured under real condition with various lighting, movement of camera as well as range and position of target object. These conditions reflect the real time condition in a palm oil plantation. The system was also tested with 1000 sample of offline images to determine the classification error of both image samples for all techniques used. The main factor to key successful of the real time machine vision system is its processing technique to analyze an image. Narrow and broad weed images can be distinguish correctly by implementing very advance technique in image processing. This will contribute to the reliability of the machine vision system. All the three techniques have been tested with the available raw data to ensure high classification rate obtain. The most important thing in the classification stage is how the features of weed can be extracted using designated feature extraction technique. All the three techniques have their unique feature extraction values that represent the narrow and broad weed. Finally, these feature vectors are being use to design classification equation to classify the offline and
(c)
(d) Fig. 3: (a) Logitech webcam camera, (b) GUI of image processing algorithm, (c) electronic part to interface with motor pump, (d) motor water pump 455
Am-Euras. J. Agric. & Environ. Sci., 3 (3): 451-458, 2008 Table 1: Feature vector contrast of GLCM technique Contrast Broad
Narrow
1
2.124686
0.699542
2
3.710774
0.540285
3
3.142560
0.571875
4
3.142560
0.648287
5
2.932440
0.590311
6
3.142560
0.443057
7
3.713193
0.559545
8
2.936872
0.559545
9
3.713193
0.552419
10
3.713193
0.454799
Contrast
-------------------------------------------------Image
Number of Images
Fig. 4: Plot graph of GLCM contrast parameter
Table 2: Feature vector of FFT Narrow
Broad
-----------------------------
-----------------------
L
H
L
1
430
1256
451
2
870
1628
451
789
3
824
1800
888
1478
4
928
1815
1288
1826
5
1183
1821
765
1218
6
1057
1677
244
504
7
881
1653
179
337
8
881
1653
561
30
9
689
1502
612
146
10
679
1639
83
224
H 789 H
Fig. 5: Feature vector of FFT
St2
Image
L
Table 3: Sift feature vector Narrow
Broad
------------------------------
------------------------------
Image
St1
St1
1
0.65953
0.28271
0.81178
0.36157
2
0.69170
0.29375
0.83419
0.37490
3
0.63404
0.24282
0.82677
0.37105
4
0.64358
0.26702
0.82677
0.37105
5
0.64876
0.24651
0.80172
0.34511
6
0.65958
0.28214
0.82677
0.37105
7
0.66280
0.26866
0.83419
0.37490
8
0.66280
0.26866
0.80172
0.34511
9
0.65745
0.27276
0.83419
0.37490
10
0.68074
0.29060
0.83419
0.37490
St2
St1
St2
Fig. 6: Plot graph of SIFT feature vector Figure 4-6 are the plot graph of the feature vector of GLCM, FFT and SIFT. The values show that feature vector of narrow and broad weed concentrate at two different clusters. It is obviously seen that all the techniques can be easily classified using linear equation as the values clearly distribute at two different groups. However, the SIFT feature vector look more stable as the features concentrate at small range of x and y axes (St1 and St2) when compare to GLCM and FFT. Linear equations obtained from the feature vector were used to test the offline and recorded video of raw data images. Correct classification rate has been measured to find the best technique that gives high accuracy of classification. The following Tables (Table 4 and 5)
recorded video narrow and broad weed raw data. The following Table 1-3 shows the feature vector of respective image processing techniques. 456
Am-Euras. J. Agric. & Environ. Sci., 3 (3): 451-458, 2008 Table 4: Classification with offline images Type of weed
GLCM (%)
FFT (%)
SIFT (%)
Narrow
81.0
89.2
99.5
Broad
81.5
91.0
99.8
on-going to improve the system and developed a more accurate and robust algorithm in order to improve the performance of the current technique especially in real time condition. REFERENCES
Table 5: Classification wit h playback-recorded video GLCM (%)
FFT (%)
SIFT (%)
Narrow
70.4
80.6
90.2
Broad
72.5
81.1
91.7
1.
Azman, I., A.S. Mohd and N. Mohd, 2003. The production cost of oil palm fresh fruit bunches: The case of independent smallholders in Johor. Oil Palm Econ. J., Vol: 3(1). 2. Stafford, J.V., 1997. Machine-Assisted detection of weeds and weed patches. In: Stafford, J.V. (Ed.), Precision Agriculture’97. Technology, IT and Management, BIOS Scientific Publishers Ltd., 2: 511-518. 3. Wang, N., N. Zhang, F.E. Dowell, Y. Sun and D.E. Peterson, 2001. Design of an optical weed sensor using plant spectral characteristics. Trans. ASAE, 44 (2): 409-419. 4. El-Faki, M.S., N. Zhang and D.E. Peterson, 2000. Weed detection using color machine vision. Trans. ASAE 43 (6): 1969-1978. 5. Cho, S.I., D.S. Lee and J.Y. Jeong, 2002. Weedplant discrimination by machine vision and artificial neural network. Biosyst. Eng., 83 (3): 275-280. 6. Meyer, G.E.K. von Bargen, D.M. Woebbecke and D.A. Mortensen, 1994. Shape features for identifying young weed using image analysis. Am. Soc. Agric. Eng., St. Joseph, MI USA, 94: 3019. 7. Tang, L., L. Tian and B.L. Steward, 2000. Color image segmentation with genetic algorithm for in-field weed sensing. Trans. ASAE, 43 (4): 1019-1027. 8. Weszka, J.S. and A. Rosenfeld, 1976. An application of texture analysis to materials inspection. Pattern Recogn. J., pp: 195–199. 9. Garcia, P., M. Petrou and S. Kamata, 1999. The use of boolean model for texture analysis of grey images. Computer Vis. Image Understand, 74 (3): 227-235. 10. Tsai, D.M. and C.Y. Hsieh, 1999. Automated surface inspection for directional textures. Image Vis. Comput., 18: 49-62. 11. Gamal, A.B., ElMasry Ning, A. Wang, E.B. Adel and N. Michael, 2007. Hyper spectral imaging for nondestructive determination of some quality attributes for strawberry. A J. Food Eng., 81: 98-107. 12. Gadelmawla, E.S., 2004. A vision system for surface roughness characterization using the gray level co-occurrence matrix. NDT and E International, 37: 577-588.
show the result of correct classification rate of narrow and broad weed using offline images and real time condition. The sample images have been tested with all the three techniques described. The correct classification rate has been measured to find the best image processing method to be used in the system. From the results shown in Table 4 and 5, it can be seen that the SIFT technique give better classification performance compared to the GLCM and FFT for both offline images and recorded video. When tested with the offline images, the SIFT achieved 99.5 and 99.8% correct classification rate for narrow and broad weed recognition where as 81, 81.5, 89.2 and 91% classification rates were recorded for the GLCM and FFT. As for testing with the recorded video data, the SIFT scores 90.2 and 91.7% for narrow and broad weed recognition, respectively. The GLCM and FFT, on the other hand, only obtained 70.4, 72.5, 80.6 and 81.1% classification accuracy for the narrow and broad category. CONCLUSION The system consists of hardware and software engine was successfully built in engineering prototype as shown in Fig. 2 and 3. It was integrated with compact size of PC and assembled into mechanical structure. The whole system can be fixed into four wheel drive tractor sprayer to implement in real field. Testing to the real time simulation has been done in lab using recorded video as well as offline image. The core component of the machine vision system is the image processing technique. This paper has presented three techniques, namely the SIFT, FFT and GLCM, to recognize weed type as either narrow or broad. It was shown that the SIFT is better than the FFT and GLCM for the task. From the results, we anticipated that the developed system using SIFT can be realized for real time implementation. The developed system is quite robust considering that it can handle various conditions such as aforementioned above. Further work is 457
Am-Euras. J. Agric. & Environ. Sci., 3 (3): 451-458, 2008
13. Su, K.Y. and J.T. Kuo, 2005. Application of Two-Dimensional Nonuniform Fast Fourier Transform (2-D NUFFT) technique to analysis of shielded microstrip circuits. IEEE Transaction on Microwave Theory and Techniques, 53 (3): 993-999. 14. Liang, Z.P., B. Madore, G.H. Glover and N.J. Pelc, 2003. Fast algorithms for GS-model-based image reconstruction in data-sharing Fourier imaging. Medical Imaging IEEE Transactions, 22 (8): 1026-1030. 15. Bradley, P., C. Douglas and A. Jeffrey, 2003. Fast, Iterative Image Reconstruction for MRI in the Presence of Field Inhomogeneities. IEEE Transactions on Medical Imaging, 22 (2): 178-188. 16. Kadah, Y.M. and X. Hu, 1997. Pseudo-Fourier Imaging (PFI): A technique for spatial encoding in MRI. IEEE Transactions on Medical Imaging, 16 (6): 893-993.
17. Xu, X.M. and Q.H. Liu, 2001. Fast spectraldomain method for acoustic scattering problems. IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, 48 (2): 522-560. 18. Lin, H.C. and C.S. Lee, 2001. Enhanced FFTbased parametric algorithm for simultaneous multiple harmonics analysis. IEEE ProcGener. Tronsm. Disrvih., 14 (3): 209-214. 19. Zhang, F., Z. Geng and W. Yuan, 1996. The algorithm of interpolating windowed FFT for harmonic analysis of electric power system. IEEE Transactions on Power Delivery, 16 (2): 160-164. 20. Lowe, D.G., 2004. Distinctive image features from scale invariants keypoints. Intl. J. Computer Vision, 60 (2): 91-110. 21. Graniatto, P.M., H.D. Navone, P.F. Verdes and H.A. Ceccatto, 2002. Weed seeds identification by machine vision. Computers and Electronics in Agriculture, 33: 91-103.
458
