12th International Conference on Information Fusion Seattle, WA, USA, July 6-9, 2009
Automatic Optical and IR Image Fusion for Plant Water Stress Analysis Weiping Yang†, Xuezhi Wang*, Ashley Wheaton‡, Nicola Cooley‡, and Bill Moran* * Melbourne Systems Laboratory Faculty of Engineering, University of Melbourne, Australia
[email protected] Email:
[email protected], †ATRLAB, School of Electronic Science and Engineering, National University of Defense Technology, Changsha, Hunan, 410073, P.R.China Email:
[email protected] ‡Department of Land and Environment, University of Melbourne, Australia
[email protected] Email:
[email protected],
Abstract – Automatic registration of an optical image and an associated IR image is a key step toward to automation of canopy temperature measurement in the process of plant water stress analysis. In this context, the scene of the IR image is completely included in the optical image and the transform between the two images may involve translation and rotation by a small angle, though a small scale difference may also be present. This automatic registration of data from two quite different imaging regimes presents several challenges, and is not susceptible to several common image processing techniques. In this paper, an automatic image registration algorithm, based on the fundamental cross correlation method is designed, which can avoid human intervention in the alignment process and is suitable for the application to plant water stress analysis where significant numbers of images need to be processed. The effectiveness of the software design is demonstrated via our experiments and the registration error performance is compared to the cases where the similarity criterion is replaced by that of mutual information and correlation ratio respectively.
program to estimate spatial and temporal variation in water status of grapevines using data collected from remotely sensed digital images [5]. The project aims to build up an automatic controlled irrigation program with nondestructive sensing and measurement systems [6]. In the process of plant water stress analysis, the temperature profile of a specific area can be acquired from the software output of the IR camera. With a reference optical image, which is taken at the same spot using a normal camera to provide a true view of the IR image scene, the area of interest (e.g., plant leaves other than ground or sky) may be identified in the IR image. Subsequently temperature data associated with the area of interest is extracted and statistical analysis can be performed. In order to obtain statis-
Keywords: Automatic image registration, Optical and IR image fusion, Water stress analysis using temperature profile, Thermal imagery, Cross correlation.
1 Introduction
Figure 1: Example of image pair 1: Cabernet Sauvignon Advanced infrared (IR) sensing technology enables ac- grapevines. fIR = IR image (right) and fo = optical image quisition of high quality thermography imagery [1] to pro- of the scene (left). The scene of the IR image is in that of vide capability for analysis of plant water stress [2] from the optical image. canopy temperature data obtained from IR cameras [3, 4]. The Melbourne School of Land and Environment at The tically robust data, a large number of images over a time University of Melbourne is currently conducting a research course should be collected, processed and analyzed. It is de978-0-9824438-0-4 ©2009 ISIF 1053
sirable for image collection to be integrated into real time automated control irrigation programs and therefore essential that the image analysis process be automated. A key step towards this process is to automatically determine the overlap area between the IR image and associated optical image. An example of optical and IR image pair taken on grapevines is given in Figure 1, in which we need to identify the area overlapped with the IR image from the left optical image. Clearly, whether we can correctly register the overlapped area from the optical image can significantly influence of canopy temperature estimation outcomes. The major difficulties which arise in this type of image registration are listed below. • Taken by different type of sensors and possibly at different view angle and different time, the pair of optical and IR images are in general not matched exactly. • At the best matching overlap area, the intensities of both images can be quite different. Therefore, approaches involving image intensity are unlikely to obtain a satisfied registration performance. • Apart from some similarity of overall structures, it is difficult to identify consistent feature points from both images in some popular feature spaces via an automatic registration method, such as the scale invariant feature transformation (SIFT) method [7].
a method in [14] which applies mutual information (MI) to measure the statistical dependence or information redundancy between the image intensities of corresponding pixels in both images. Using correlation ratio (CR) as the similarity measure, Roche et. al. proposed a CR method in [15] which assumes the pixel-pair intensities between two registered images are functional dependent. These area-based methods were summarized in [16]. In this paper, we consider the registration problem between an optical image and an IR image1 . The images are taken using a hand held camera, and a hand held thermal imager, over a time span of less than a minute. In the situation studied, the scene of the IR image is completely included in the optical image and the transform between the two images may involve translation and rotation by a small angle, though a small scale difference may also be present. An automatic image registration algorithm based on the fundamental cross correlation method is designed, which avoids human intervention in the alignment process and is suitable for the application to plant water stress analysis where significant amounts of data are required. The effectiveness of the software design is demonstrated via our experiments and the registration error performance is compared to the cases where the similarity criterion is replaced by that of mutual information and correlation ratio respectively. The rest of the paper is organized as follows. The problem description is given in the next section. We present the automatic cross correlation (ACC) image registration approach in detail in the Section 3. Experimental results and discussions are given in Section 4, followed by the conclusion.
A variety of image registration techniques and algorithms which may be used in this automatic alignment application are available in the literature [8] (and references therein). The approaches fundamentally fall into two main categories, 2 Problem Description i.e., area-based methods and feature-based methods. Let Fo and FIR denote the optical and IR images of an The SIFT method perhaps is the most representative apimage pair respectively. From the application at hand, we proach in the feature-based automatic methods. The SIFT can reasonably assume that: implementation is able to find distinctive points that are invariant to location, scale and rotation, and robust to affine 1. The scene of FIR is completely within that of Fo . In transformations (changes in scale, rotation, shear, and poother words, there is an area in the optical image Fo sition) and changes in illumination for images of the same where the scene of the IR image FIR is approximately source or of the same type of sensors [7]. When this is the matched. case, the algorithm is particularly effective. However, for 2. The matching area in Fo may differ by a translation b, our application the success rate of SIFT is less than 10%. a small rotation θ [say θ ∈ (−10o , 10o )] and a slight In most cases, there is simply no SIFT point at all in the IR scaling s ≈ 1 as in (1). image. Solutions based on area correlation technique [9] seems to Therefore, a point (x, y) on the reference image FIR and the be more robust to our problem, except for those which use related point (xo , yo ) on the base image Fo are connected by intensity (or color) dependent functions as similarity meaa linear transformation. sures, such as the Fourier transformation type [10] and mu tual information type [11] approaches. The maximum correbx x cos θ, sin θ xo (1) + =s lation coefficient detection method was initially proposed in by yo y − sin θ, cos θ [12] and later extended in [9] by considering the correlation where s is the scaling factor, θ is the rotation angle and in a feature space. ′ [b x , by ] is a translation vector. Instead of using cross correlation coefficient, HuttenThe registration problem in the context of (1) is to find locher et. al. used the Hausdorff distance as a similarthe parameters s, θ and b and obtain the optical image part ity measure to register binary images that are the output of 1 To be precise, the intensity of the IR image is a function of the canopy an edge detector [13]. To deal with the problem of multimodality medical image registration, Maes et. al. proposed temperature. 1054
FOIR ∈ Fo associated with the IR image FIR using these parameters. In addition, we need the registration process to be completed automatically. Note that, in general there is no complete matching between the two images and we can only find the overlapping area of the best matching.
3 Automatic Cross Correlation Alignment Algorithm Implementation
R(u, v) =
fIR (i, j) − f¯IR
σu,v σIR
where f¯u,v
=
f¯IR
=
S(fu,v ) X 1 fu,v (i, j) S(fu,v ) i,j S(fIR ) X 1 fIR (i, j) S(fIR ) i,j
(2)
S(fIR ) X 2 1 fIR (i, j) − f¯IR S(fIR ) i,j
3. CC1 loop: (a) for each given search grid point (u, v), compute the cross correlation coefficient using (2). (b) weight the coefficient matrix R using the predefined probability distribution (for the location of the overlapped area). (c) find the control point CP1 – which is the location corresponding to the maximum value of R,i.e., cp1 = arg max (R) (3) i,j∈R
ˆ This is 4. CC2 loop: estimate the rotation angle θ. done by turning the candidate image window about the CP1 within a given θ range and accuracy (grid) and calculating the cross correlation coefficient Rθ using (2). The estimated rotation angle is then given by (4) θˆ = arg max(Rθ ) θi
• Most pairs of optical and IR images collected in this work can be manually registered via several control points identified by eye. Apart from potentially a larger registration error, this is time demanding and not suitable for processing large numbers of images.
fu,v (i, j) − f¯u,v
=
2. Binary filtering using the “Canny” type edge detector: Fo → fo , FIR → fIR .
• Two methods using freely available SIFT software were tried for the registration problem. However, as discussed in Section 4, neither of them were able to produce consistent results.
i,j
2 σIR
S(fu,v ) X 2 1 fu,v (i, j) − f¯u,v S(fu,v ) i,j
1. Input: Fo , FIR (possibly in reduced resolution) and predefined parameters: search grid, θ range and accuracy.
• As the intensities of the best overlapped pixels of the image pair are not consistent, direct application of the CC, MI or CR method simply does not work at all. Consequently, we used a binary edge detector filter to remove intensity information while retaining structure information. It was found in our experiment, by using the black and white (BW) edge images, the CC method has a very high success alignment rate over MI and CR methods even use resolution reduced images, though almost no common edge can be found in the images as shown in Figure 3.
PS(fIR )
=
(2) will be calculated for every point (u, v) over the search grid of size S(R).
The ACC algorithm designed and implemented in this work is based on the cross correlation (CC) method described in [12]. The algorithm is described in Table 1 and the process flowchart is given in Figure 2. In the algorithm, auxiliary filters are used to remove intensity information so that the correlation of the pair of images is purely based on the geometric structure over the area of interest. This is justified by the following observations:
Denote by S(f ), the size (or resolution) of the image f and fu,v , the (u, v)th image window of a given search grid over the image fo . The cross correlation coefficient matrix R in Table 1 is computed by
2 σu,v
5. based on estimated CP1 and θˆ estimate the overlapped area from Fo and output the image FOIR . Table 1: The steps of automatic image registration.
4 Experiment Results and Discussions The ACC algorithm was tested using many sample pair of images which were collected by the researchers from different plant fields. All of them can be successfully registered with tolerable errors2 . For the purpose of performance comparison, we also implemented the algorithms which use the normalized MI (NMI) and CR as a similarity measure described in [16], respectively. Both of them use a gray scaled Fo and FIR . 2 In fact the ground truth is unknown. So, what we mean “tolerable error” is that the registration result has passed visual examination conducted by an expert.
1055
FO
FIR
ED O
ED IR
(a)
f IR
fO
(b)
Figure 4: Color images of the aligned pair in Figure 3. CC1
Control Point Estimator
where
cp1 θ range Initial overlapped
f O part
VarIR =
X
i2 PIR (i) −
i
f ’O
X i
!2
i PIR (i)
and CC2
cp1
Rotation Estimator
X
j 2 POIR,u,v (i, j) Pu,v (i) j 2 X 1 − j POIR,u,v (i, j) Pu,v (i) j
^ θ
FO
1
Varu,v (i) = FOIR
Transform Parameters & Matching Output Output
Figure 2: Image alignment algorithm flowchart.
All summations in (5) and (6) are taken over image intensity space. As mentioned earlier, when taking the maximum value of the correlation matrix R, a probability weighting matrix is used to eliminate possible false maxima on the edge of the optical image. Implicitly, we assume that the correct alignment area is centered at the center of the optical image has a Gaussian distribution with the standard deviation σ. In the experiment, σ takes a value of approximately 1/3 of the (a) (b) image size. Figure 5 illustrates the weighted result for the Figure 3: Comparison of the aligned image pair filtered by example image pair 1 (see Figure 1), where the correlation a “canny” edge detector. (a) filtered optical image fo . (b) matrix is obtained using ACC. To demonstrate the image registration performance, we filtered IR image fIR . present two examples. Image pair 1, as shown in Figure 1, was taken from the scene of Cabernet Sauvignon grapevines. Image pair 2, as shown in Figure 6, was taken from a simIn the NMI method the correlation matrix RM I (u, v) is ilar scene to that of image pair 1 but with a tag box. The calculated by tag box provided a reference temperature for analysis purpose. The registration results are presented in Figures 7 ∼ P (Pu,v (i) log Pu,v (i) + PIR (i) log PIR (i)) 12. Since the ground truth is unknown, as given in Table 2, iP RM I (u, v) = we compared the registration error for NMI and CR methods i,j POIR,u,v (i, j) log POIR,u,v (i, j) (5) against the ACC results which were deemed to be the best where POIR,u,v (i, j) is the joint probability that the intensi- alignments by visual examination. The registration error is ties of Fu,v (i, j) and FIR (i, j) are at levels i and j respec- measured using the distance between the estimated location tively. Pu,v (i) and PIR (i) are the marginal probability of of the registered image and the “truth” on the optical image. the images Fu,v (i, j) and FIR (i, j). These probabilities can Image ACC NMI CR be computed from the normalized joint and marginal intenPair Deviation Error Deviation Error Deviation Error sity histograms. Pair 1 [0, 0] 0 [−12, 26] 28.64 [−36, 59] 69.16 In the CR method the correlation matrix RCR (u, v) is calculated by RCR (u, v) = 1 −
1 X Varu,v (i)Pu,v (i) VarIR i
(6)
Pair 2
[0, 0]
0
[1, 22]
22.02
[−40, 83]
Table 2: Registration error comparison
Discussions: 1056
92.14
(a)
Figure 6: Example image pair 2: Cabernet Sauvignon grapevines with a tag box.
(b)
Figure 7: ACC result: matched image pair 1, where θ = 0.3o and the shift b = [ 214, 134 ]′ .
(c) Figure 5: Weighted correlation coefficient matrix of size S(Fo ). (a) correlation coefficients; (b) weighting function; (c) weighted correlation coefficients. 1. Figures 7 ∼ 12 and Table 2 demonstrated that the proposed ACC algorithm provides the best and consistent alignment result compared to that of NMI and CR in the context of automatic registration of optical and IR image pairs. Using the same performance comparison technique, we considered 10 more pairs of images taken from various plants. As indicated in Table 3, the result is consistent with our claim. 2. From our experiments, we found that SIFT method produces mixed results with less than 10% success rate and hence is not suitable for our application.
water stress analysis. The proposed algorithm has used the information of coherent image structure but isolated image intensity impact in the alignment process and hence produces satisfactory and robust registration result. The ACC performance is compared to other area-based methods which can be used for the underlying application in the automatic registration context. We found that some image pairs taken from low altitude balloons were only partially matched with each other with large rotation angles as a result of limited sensor controllability. The automatic alignment of these images is an issue and will be addressed in our future work.
References [1] N.A.L.Archer, and H.G. Jones. “Integrating hyperspectral imagery at different scales to estimate component surface temperatures”, International Journal of Remote Sensing, vol. 27, no. 11, pp. 2141–2159, June 2006.
[2] I. Leinonen, and H. G. Jones. “Combining thermal and visible imagery for estimating canopy temperature In this paper, the ACC algorithm is proposed for regisand identifying plant stress”, Journal of Experimental tration of optical and IR image pairs automatically for plant Botany, vol. 55, no. 401, pp. 1423–1431, June 2004. 1057
5 Conclusion
Figure 8: ACC result: matched image pair 2, where θ = 0.4o and the shift b = [ 230, 145 ]′ .
Figure 9: NMI result: matched image pair 1, where θ = −0.4o and the shift b = [ 202, 160 ]′ . [3] C. Campillo, M. H. Prieto, C. Daza, M.J. Monino, and M.I. Garcia. “Using digital images to characterize canopy coverage and light interception in a processing tomato crop”, Journal of HortScience, Vol. 43, no. 6, pp. 1780–1786, 2008. [4] M. Saudreau, A. Marquier, B. Adam, P. Monney, and H. Sinoquet. “Experimental study of fruit temperature dynamics within apple tree crowns”, Elsevier: Journal of Agricultural and Forest Meteorology Vol. 149, pp. 362–372, 2009. [5] A. D. Wheaton, N. Cooley, G. Dunn, I. Goodwin, and S. Needs. “Evaluation of infrared thermography to determine the crop water status of Cabernet Sauvignon grapevines”, Poster paper of 13th Australian Wine Industry Technical Conference, Adelaide, 28 July – 2 August, 2007.
Figure 10: NMI result: matched image pair 2, where θ = −0.2o and the shift b = [ 231, 167 ]′ .
Figure 11: CR result: matched image pair 1, where θ = 0.5o and the shift b = [ 175, 75 ]′ . conductance over leaf surfaces”, Journal of Plant Cell And Environment,vol. 22, no.9, pp. 1043–1055, 1999. [7] D. G. Lowe. “Object recognition from local scaleinvariant features”, Proc. of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp.1150–1157, 1999. [8] B. Zitov´a, and J. Flusser. “Image registration methods: a survey ”, Journal of Image and Vision Computing (Elsevier), vol. 21 , pp. 977–1000, 2003. [9] W. K. Pratt. “Correlation techniques of image registration”, IEEE Trans. on AES, vol. AES-10, no. 3, pp. 353–358, May 1974. [10] H. Liu, B. Guo, and Z. Feng. “ Pseudo-Log-Polar Fourier Transform for Image Registration”, IEEE Signal Processing Letters, vol. 13, no. 1, pp. 17–20, Jan. 2006.
[6] H. G. Jones. “Use of thermography for quantitative [11] P. Viola, and W. M. Wells III . “Alignment by Maxistudies of spatial and temporal variation of stomatal mization of Mutual Information”, International Jour1058
Figure 12: CR result: matched image pair 2, where θ = 0.3o and the shift b = [ 190, 228 ]′ . ACC Deviation Error [2.5, 2.3] 3.40
NMI Deviation [6.9, 11.6]
Error 13.50
CR Deviation [32.4, 92.7]
Error 98.20
Table 3: Registration error averaged over 10 pairs nal of Computer Vision, vol. 24, no. 2, pp. 137–154, 1997. [12] P.F. Anuta. “Digital registration of multispectral video inagery”, Journal of Soc. Photo-Optical Instrum. Engs., vol. 7, pp. 168–175, Sep. 1969. [13] D. P. Huttenlocher, G. A. Klanderman, and W. J. Rucklidge. “Comparing images using the Hausdorff distance”, IEEE trans. on Pattern Analysis and Machine Intelligence, vol. 15, no. 9, pp. 850–863, Sep. 1993. [14] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens. “Multimodality image registration by maximisation of mutual information”, IEEE trans. on Medical Imaging, vol. 16, no. 2, pp. 187–198, April 1997. [15] A. Roche, G. Malandain, X. Pennec, and N. Ayache. “The correlation ratio as a new similarity measure for multimodal image registration”, Proc. First Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI’98), vol. 1496, pp. 1115–1124, 1998. [16] Y.H. Lau, M. Braun and B. F. Hutton. “Non-rigid image registration using a median-filtered coarse-to-fine displacement field and a symmetric correlation ratio”, Journal of Physics in Medicine and Biology, vol. 46, pp. 1297–1319, 2001.
1059
