IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 3, NO. 4, OCTOBER 2006
457
Markovian Fusion Approach to Robust Unsupervised Change Detection in Remotely Sensed Imagery Farid Melgani, Senior Member, IEEE, and Yakoub Bazi, Student Member, IEEE
Abstract—The most common methodology to carry out an automatic unsupervised change detection in remotely sensed imagery is to find the best global threshold in the histogram of the so-called difference image. The unsupervised nature of the change detection process, however, makes it nontrivial to find the most appropriate thresholding algorithm for a given difference image, because the best global threshold depends on its statistical peculiarities, which are often unknown. In this letter, a solution to this issue based on the fusion of an ensemble of different thresholding algorithms through a Markov random field framework is proposed. Experiments conducted on a set of five real remote sensing images acquired by different sensors and referring to different kinds of changes show the high robustness of the proposed unsupervised change detection approach. Index Terms—Data fusion, image thresholding, Markov random fields (MRFs), spatial context, unsupervised change detection.
I. I NTRODUCTION
F
ROM a methodological viewpoint, automatic unsupervised change detection is usually achieved through two key steps. First, a couple of coregistered multitemporal remote sensing images acquired at two different dates over the same geographical area are compared. The result of the comparison is an image usually termed “difference image.” In the second key step, changes are identified by analyzing the difference image. The problem of discrimination between the “change” and the “no-change” classes in the difference image can be viewed as an image binarization problem. The most common solution to this problem is based on the use of a thresholding algorithm to select the global threshold in the difference image histogram automatically [1]–[3]. However, in general, the choice of the best thresholding algorithm for a given difference image is not always clear, because: 1) in an unsupervised change detection process, no ground truth is available to represent the prior knowledge of the scene and therefore to guide the choice appropriately and 2) the effectiveness of a thresholding algorithm depends on the statistical characteristics of the difference image (i.e., the statistical distribution of the change and the no-change classes, the degree of overlap between them, and their prior probability). A possible approach to solve this problem is to fuse the results provided by an ensemble of different thresholding algorithms. In this way, we will be able to exploit the peculiarities of Manuscript received November 9, 2005; revised February 5, 2006. The authors are with the Department of Information and Communication Technologies, University of Trento, I-38050 Trento, Italy (e-mail:
[email protected];
[email protected]). Digital Object Identifier 10.1109/LGRS.2006.875773
the different thresholding algorithms synergetically and therefore reach more robust final decisions than with a single thresholding algorithm. It is noteworthy that the goal of the fusion is not to outperform the single thresholding algorithm but to obtain accuracies that, if not better, should at least be comparable to that of the best single thresholding algorithm, independently of the statistical characteristics of the difference image. The fusion approach has been studied extensively in the literature to solve challenging classification problems [4]. By contrast, compared to the literature on classifier fusion, this attractive approach has not received the attention that it deserves in the image thresholding literature in general and in the change detection community in particular. Although a thresholding problem can be viewed as a classical binary classification problem, the implementation of a fusion strategy for thresholding algorithms in an unsupervised change detection context is made difficult by two main factors, namely: 1) it should be carried out without training samples and 2) the conceptual heterogeneity of thresholding algorithms only leaves room for a decision-level-based fusion because of the difficulty in extracting compatible partial decision information such as posterior probabilities, which are often exploited in the classifier fusion. In this letter, we propose a novel robust unsupervised change detection approach based on a Markov random field (MRF) fusion of change maps provided by an ensemble of different thresholding algorithms. Our choice of MRFs to carry out the fusion task is motivated by two reasons, namely: 1) they represent a mathematically well-founded framework that has proved successful for fusing multiple sources of information [5] and 2) they allow to implement a complex but effective image analysis at a global scale through a model of the local image spatial properties. II. MRF F USION A PPROACH A. MRF Fusion Model Formulation Let X = {xmn : m = 0, 1, . . . , M −1, n = 0, 1, . . . , N −1} be the scalar M × N difference image with L possible gray levels (xmn ∈ {0, 1, . . . , L − 1}) generated from a couple of optical or SAR multitemporal images. Let us consider an ensemble of P different thresholding algorithms. Let Ti (i = 1, 2, . . . , P ) be the optimal threshold found by the ith algorithm of the ensemble. Let Ai (i = 1, 2, . . . , P ) be the thresholded image (i.e., change map) generated by the ith thresholding algorithm of the ensemble. The aim of the proposed approach is to reach the capability to generate, independently of the statistical properties of X, a
1545-598X/$20.00 © 2006 IEEE
458
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 3, NO. 4, OCTOBER 2006
Fig. 1. General block diagram of the proposed MRF fusion approach.
global change map that, if not better, is at least comparable to the best change map yielded by a single thresholding algorithm of the ensemble. As depicted in Fig. 1, to reach such an objective, we propose to combine the change maps obtained by the considered ensemble within a Markovian framework. Because a change detection problem can be viewed as a binary classification problem where each pixel (m, n) is assigned to a label ymn ∈ {ϕ1 , ϕ2 }, the optimal classification Y ∗ of all the pixels of the original image X, given the change maps Ai (i = 1, 2, . . . , P ), can be performed by applying the maximum a posteriori probability (MAP) decision criterion, which is defined by P (Y ∗ |A1 , A2 , . . . , AP ) = max{P (Y |A1 , A2 , . . . , AP )} . (1) Y
By adopting the MRF approach, one can greatly simplify the complexity of this maximization problem by passing from a global model to a model of the local image properties. The latter is defined both in terms of the potential function of the individual pixels and of the interactions among pixels in the appropriate neighborhoods. The combination of the MAP method with the MRF modeling makes our binary classification task equivalent to the minimization of a total energy function UT , which is expressed in the following relationship: P (Y |A1 , A2 , . . . , AP ) =
1 exp [−UT (Y, A1 , A2 , . . . , AP )] Z (2)
where Z is a normalizing constant. Under the Markovian approach, the total energy function UT (·) can be rewritten in terms of the local energy functions Umn using the concept of neighborhood [6] as UT (Y, A1 , A2 , . . . , AP ) =
−1 M −1 N
Umn
(3)
m=0 n=0
with Umn = U ymn , Y S(m, n),AS1(m, n),AS2 (m, n),. . . ,ASP (m, n) (4)
where Y S (m, n) and ASi (m, n) stand for the set of labels of the pixels of the image Y and the images Ai (i = 1, 2, . . . , P ), respectively, in a predefined neighborhood system S associated with pixel (m, n). The minimization of (3) can be carried out by means of different algorithms; the most popular being the simulated annealing (SA), the maximizer of posterior marginal (MPM), and the iterated conditional mode (ICM) algorithms [6]. In this letter, the ICM algorithm is adopted because it represents a simple and computationally moderate solution to optimize the MRF-MAP estimates, for it converges to a local, but usually good, minimum of the energy function. The ICM consists of minimizing iteratively the total energy function UT (·) through a pixel-based scheme until convergence is reached (i.e., where the pixel labels do not change much). In other words, the optimization process is reduced to the iterative minimization of the local energy function Umn associated with each pixel (m, n). As the true set of labels Y S (m, n) in (4) is unknown, at each iteration, the estimate of Y S (m, n) obtained at the previous iteration is used to generate a new estimate of the label set Y . At this point, the first problem to deal with is the decomposition of the local energy function Umn . This depends on the two kinds of sources of contextual information that contribute to the optimization process. They are: 1) the spatial contextual information source, which defines the spatial correlation in image Y between the label of pixel (m, n) and the labels of its neighbors and 2) the interimage information sources, which express the relationship between the image Y and each of the change maps Ai (i = 1, 2, . . . , P ). Similar to what is done in [5] in the context of multisource classification, for the sake of simplicity, it is assumed that the contributions from these sources of information are separable and additive. Accordingly, the local energy function Umn to be minimized for the pixel (m, n) can be written as follows: Umn = βSP · USP ymn , Y S (m, n) +
P
βi · UII ymn , ASi (m, n)
(5)
i=1
where USP (·) and UII (·) refer to the spatial and interimage energy functions, respectively, whereas βSP and βi (i = 1, 2, . . . , P ) represent the spatial and interimage parameters, respectively. B. Energy Functions The neighborhood system S = Y S ∪ AS1 ∪ · · · ∪ ASP adopted to define the two kinds of energy functions required to compute the local energy function in (5) is based on a second-order neighborhood. On the basis of this neighborhood system, the spatial energy function can be expressed as [5], [6] USP ymn , Y S (m, n) = −
ypq ∈Y S (m,n)
I(ymn , ypq )
(6)
MELGANI AND BAZI: MARKOVIAN FUSION APPROACH TO ROBUST UNSUPERVISED CHANGE DETECTION
459
Fig. 3. Histograms of the five test original difference images.
pixel level that can be incurred by the thresholding algorithms. The absence of an a priori knowledge about the scene (i.e., unavailability of training samples) makes the determination of these weights particularly tricky. A possible way to solve this problem consists in exploiting the threshold value found by each single thresholding algorithm of the ensemble to derive a confidence measure based on the idea that the larger the difference (i.e., distance) between the pixel and the threshold values, the higher the degree of confidence in the decision of the single thresholding algorithm. A simple weight function αi (·) that satisfies such requirement is given by αi (xmn ) = 1 − exp (−γ|xmn − Ti |)
Fig. 2. Set of real difference images used in the experiments. (a) Po image (northern Italy; agricultural changes; April–May 1994; Landsat-5 TM). (b) Trentino image (northern Italy; changes due to cloud contamination; May–July 2000; Landsat-7 ETM+). (c) Elba image (Elba Island, Italy; changes caused by forest fire; August–September 1994; Landsat-5 TM). (d) Bern image (near Bern, Switzerland; changes due to flooding; April–May 1999; ERS2SAR). (e) Pavia image (Pavia, Italy; changes caused by flooding; October 2000; ERS2-SAR). The brightness and contrast of the original difference images have been enhanced for better visualization.
where I(·, ·) is the indicator function, which allows to count the number of occurrences of ymn in Y S (i.e., the spatial part of S), and is defined as 1, if ymn = ypq I(ymn , ypq ) = (7) 0, otherwise. In a similar way as in the spatial correlation, we define the correlation between the image Y and the images Ai (i = 1, 2, . . . , P ) and, accordingly, the interimage energy function as follows: UII ymn , ASi (m, n) =− αi (xpq ) · I (ymn , Ai (p, q)) . (8) Ai (p,q)∈AS (m,n) i
The use of the weight function αi (·) aims at controlling, during the fusion process, the effect of unreliable decisions at the
(9)
where γ is a real positive constant controlling the steepness of the weight function. The possible misleading effects of the thresholding algorithms are further controlled at a global (image) level through the interimage parameters βi (i = 1, 2, . . . , P ), which are computed as follows: βi = exp −γ|T − Ti |
(10)
where T is the average threshold value, which is obtained by T =
P 1 Ti . P i=1
(11)
Accordingly, with this global weighting mechanism, a thresholding algorithm is penalized if it exhibits a threshold value that is statistically incompatible with those of the ensemble. C. Algorithm Step 1) Initialization step a) Apply each thresholding algorithm of the ensemble on image X to generate the set of change maps Ai (i = 1, . . . , P ). b) Initialize Y by minimizing for each pixel (m, n) the local energy function Umn defined in (5) without the spatial energy term (i.e., by setting βSP = 0).
460
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 3, NO. 4, OCTOBER 2006
TABLE I ERROR RATE (PE ), FALSE ALARM RATE (PF ), AND MISSED ALARM RATE (PM ) IN PERCENTAGE ACHIEVED ON EACH OF THE FIVE TEST ORIGINAL DIFFERENCE IMAGES
Step 2) Kth iteration a) Update Y by minimizing for each pixel (m, n) the local energy function Umn defined in (5) including the spatial energy term (e.g., by setting βSP = 1). Step 3) Stop criterion a) Repeat step 2 Kmax times or until the number of different labels in Y over the last two iterations becomes very small. III. E XPERIMENTAL R ESULTS A. Data Set Description Five different real multitemporal remote sensing images were used to conduct the experimental analysis of the proposed change detection approach. These images were acquired by different sensors and refer to different kinds of changes. The corresponding difference images are shown in Fig. 2. The histograms plotted in Fig. 3 point out clearly the statistical distributional variety of these difference images. Ground-truth images, which are not reported in this letter for reasons of space limitation, were used to evaluate the accuracy of the obtained change maps in terms of: 1) error rate (PE ); 2) false alarm rate (PF ); and 3) missed alarm rate (PM ). The first error measure is often considered as an important global detection performance criterion. The last two measures are more specific criteria whose importance depends on the application. B. Ensemble Construction To construct the ensemble, five different thresholding algorithms were considered. These are the Kittler and Illingworth [7], the Bazi et al. [3], the Otsu [8], the Huang and Wang [9], and the Kapur et al. [10] algorithms. The first two are parametric, that is, they assume that the change and the no-change classes follow a distribution of the exponential family (i.e., Gaussian for the Kittler and Illingworth algorithm and generalized Gaussian for the Bazi et al. technique). The last three are nonparametric, that is, they are based on the idea of finding the best threshold through the optimization of a criterion, which is of a statistical type for the Otsu algorithm, formulated under the fuzzy set theory in the case of the Huang and Wang technique or based on an entropy function as done in the Kapur et al. algorithm. For each of the five test images, the five algorithms were run so as to provide the threshold values Ti and, accordingly, the
change maps Ai (i = 1, . . . , 5). These outputs were exploited in the successive steps by the proposed Markovian fusion approach to produce the global change map Y ∗ . All the results obtained by the single thresholding algorithms are reported in Table I. In general, they confirm that the effectiveness of a thresholding algorithm depends on the difference image to be processed. An algorithm that may appear the best for one image may be a complete failure for another. C. Results of the MRF Fusion Approach Before running the proposed approach, it was necessary to set two parameters. One is the spatial parameter βSP introduced in (5). We carried out experiments varying the value of this parameter from 0.5 to 2. The results obtained did not change significantly for the five considered images, suggesting that the way this parameter is set is not critical. All the results reported in the following refer to βSP = 1. The second parameter is the steepness constant γ of the weight function described in (9), which was set to a value of 0.1 in all experiments. In a scale of 256 gray levels, thanks to this value, we can generate a reasonable confidence degree of at least 90% for a difference value of at least 25 between the threshold value and the considered pixel gray level. For the sake of comparison, we have implemented another fusion approach based on the well-known majority vote (MV) rule [4]. With this approach, it is possible to build the global change map by consensus. Said another way, each pixel of the change map will receive the label of the class that obtains the largest number of favorable decisions from the ensemble. In addition, we ran the change detection approach described in [1]. This is based on a spatial contextual analysis of the difference image through a Markovian framework that uses class models assumed to be Gaussian and determined by the iterative expectation–maximization (EM) estimation algorithm. In general, the quantitative results reported in Table I suggest that, whatever the statistical peculiarities of the difference image, with the proposed MRF fusion approach, we can achieve detection performances that are either better or close to the performances yielded by the best thresholding algorithm of the ensemble. This clearly underlines its high robustness despite the presence of misleading thresholding algorithms in the ensemble, as illustrated by the example of the obtained qualitative results in Fig. 4. By contrast, the simple MV fusion rule appears more sensitive to fusion scenarios with misleading
MELGANI AND BAZI: MARKOVIAN FUSION APPROACH TO ROBUST UNSUPERVISED CHANGE DETECTION
461
of different thresholding algorithms, as in the proposed MRF fusion approach. IV. C ONCLUSION In this letter, we have shown that a Markovian fusion of an ensemble of different thresholding algorithms is a promising way of achieving robust unsupervised change detection. This is supported by the fact that it leads to results that are either better or comparable to those of the best single thresholding algorithm of the ensemble, whatever the statistical characteristics of the difference image (i.e., statistical behavior of the change and nochange classes, overlap, and balance degrees between them). Robust unsupervised change detection is made possible by the proposed MRF fusion approach: 1) because it is capable of capturing the best peculiarities from an ensemble of different thresholding algorithms; 2) because it exploits spatial contextual information integrated in the fusion framework naturally; and 3) because of its weighting mechanism implemented at both the pixel and the image levels to handle the reliability of the results provided by each thresholding algorithm making up the considered ensemble. Concerning the preceding last point, we wish to point out that more sophisticated weight functions could be derived, for example, through correlation measures. Despite its simplicity, the adopted solution has proved attractive for its suitable handling of fusion scenarios with very misleading results. ACKNOWLEDGMENT The authors would like to thank U. Wegmuller (Gamma Remote Sensing, Bern, Switzerland) and P. Gamba (University of Pavia) for providing the multitemporal SAR images used in the experiments. R EFERENCES
Fig. 4. Change maps obtained by the ensemble of thresholding algorithms and the proposed MRF fusion approach on the Trentino image. (a) Kittler and Illingworth algorithm. (b) Bazi et al. algorithm. (c) Otsu algorithm. (d) Huang and Wang algorithm. (e) Kapur et al. algorithm. (f) MRF fusion. (g) Groundtruth map.
“experts.” This stresses the usefulness of integrating spatial contextual information in the fusion framework, as intrinsically carried out by the MRF fusion approach. The results shown by the other Markovian change detection method used for comparison point out that the contextual analysis of the difference image on which it is based can be greatly improved by appropriately exploiting the synergies between an ensemble
[1] L. Bruzzone and D. F. Prieto, “Automatic analysis of the difference image for unsupervised change detection,” IEEE Trans. Geosci. Remote Sens., vol. 38, no. 3, pp. 1171–1182, May 2000. [2] F. Melgani, G. Moser, and S. B. Serpico, “Unsupervised change-detection methods for remote-sensing data,” Opt. Eng., vol. 41, no. 12, pp. 3288– 3297, Dec. 2002. [3] Y. Bazi, L. Bruzzone, and F. Melgani, “An unsupervised approach based on the generalized Gaussian model to automatic change detection in multitemporal SAR images,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 4, pp. 874–887, Apr. 2005. [4] J. Kittler, M. Hatef, R. P. W. Duin, and J. Matas, “On combining classifiers,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 3, pp. 226–239, Mar. 1998. [5] F. Melgani and S. B. Serpico, “A Markov random field approach to spatiotemporal contextual image classification,” IEEE Trans. Geosci. Remote Sens., vol. 41, no. 11, pp. 2478–2487, Nov. 2003. [6] R. C. Dubes and A. K. Jain, “Random field models in image analysis,” J. Appl. Stat., vol. 16, no. 2, pp. 131–163, 1989. [7] J. Kittler and J. Illingworth, “Minimum error thresholding,” Pattern Recognit., vol. 19, no. 1, pp. 41–47, 1986. [8] N. Otsu, “A threshold selection method from gray-level histogram,” IEEE Trans. Syst., Man, Cybern., vol. SMC-9, no. 1, pp. 62–66, Jan. 1979. [9] L. K. Huang and M. J. Wang, “Image thresholding by minimizing the measures of fuzziness,” Pattern Recognit., vol. 28, no. 1, pp. 41–51, Jan. 1995. [10] J. N. Kapur, P. K. Sahoo, and A. K. C. Wong, “A new method for graylevel picture thresholding using the entropy of the histogram,” Comput. Vis. Graph. Image Process., vol. 29, no. 3, pp. 273–285, 1985.
