Contour-Based Multisensor Image Registration with Rigid Transformation Zhenhua Li Department of Electrical and Computer Engineering University of Calgary Calgary, Canada
[email protected] Abstract –This paper presents a contour-based multisensor image registration algorithm. The characteristic of this approach is that the registration parameters are calculated according to the centroids and the long axes of matched contour pairs in the images to be registered. It overcomes the difficulties of control point detection and correspondence in featurebased registration techniques. The geometrical deformation between the reference and sensed images is assumed to follow a rigid transformation. Salient contours are extracted from the reference and sensed images, respectively. After contour matching, open contour matches are changed to closed contour matches by linking the two endpoints of each open contour together with a line section. Registration parameters are then estimated according to the centroids and the angles of long axes of closed contour matches. Experiments using real data show that the proposed algorithm works well in multisensor image registration. Keywords: Image registration, contour, rigid transformation, chain code, invariant moments, centroid, long axis.
1
Introduction
Image registration is a fundamental task in image processing to overlay two or more images. Automatic image registration is required in many image processing applications when images of the same scene are acquired with different sensors, at different times, or from different viewpoints. Image registration is the process of geometrically transforming multiple images to the same reference image. The geometrical deformations between the reference image and the sensed image can be divided into rigid, affine, projective, and curved transformations [1]. In the proposed algorithm, we assume that the geometrical deformation between the reference image and the sensed image is rigid transformation. Only rotation (change of the orientation) and translation (change of the location) between images are to be registered. Registration methods can be loosely divided into the following classes: algorithms based on the raw intensity, e.g., mutual information methods [2,3]; frequency domain methods, such as fast Fourier transform [4,5]; algorithms based on features such as corners, contours or relations
Henry Leung Department of Electrical and Computer Engineering University of Calgary Calgary, Canada
[email protected] between features [6,7]. Multisensor images are usally obtained by disparate sensors with different intensity characteristics, noise levels, image geometries. For example, the polarity of contrast changes can be arbitrary between airborne infrared and airborne visible images. Standard techniques such as mutual information or area correlation cannot be applied directly. Feature-based methods, which extract and match the common structures from two images, have been shown to be more suitable for multisensor image registration. There are four main steps involved [1] : extraction of primitives from each image, matching of corresponding primitives from each image, determination of the registration parameters, and transforming the sensed image to the reference image. In feature-based image registration, we must extract a large number of features and matching those primitives to give a useful set of control points. Contours representing region boundaries are preserved in most cases. Li et al. proposed a basic contour matching scheme to multisensor image registration [6]. This contour matching scheme extracts closed contours and open contours from the reference image and the sensed image, respectively. For the closed contours, chain code correlation and invariant moments are used to do the pair matching. For the open contours, corners are detected and used in the matching process. The centroids of closed contour pairs and the corners detected from open contour pairs are used as control points to calculate the registration parameters. This method can register image pairs in which the contour information is well preserved, such as the optical images from LANDSAT and SPOT (Satellite Pour l'Observation de la Terre) satellites. However, the shape of contours imaged of the same region may slightly change from different sensors for arbitrary noise and different imaging mechanisms. Taking the corners of the open contours as control points may generate some errors in the registration parameters. Detection of control points from open contours are also difficult in realization. Dai and Khorram [7] presented a feature-based image registration algorithm using improved chain-code representation combined with invariant moments. Image segmentation is first performed for the purpose of obtaining contours of region boundaries. Closed contours are detected and sorted for further processing. After closed contours are detected from two images, regions are defined. Region matching is performed using region invariant moments and improved chain-code
representation. The centers of gravity are extracted from the matched regions and are used as control points. In real data, we may not get enough closed contours from the images to be registered. It is observed that the number of matched closed contours can greatly affect the precision of registration. An extreme case is that there are no matched closed contour pairs between the reference image and the sensed image. If no matched closed contour pairs can be detected, this approach will not work. In this paper, we propose extracting open contours and closed contours from the images to be registered. All the contours extracted from the reference and sensed images are matched according to improved chain code criteria and invariant moments. Open contour matches are changed to closed contour matches by linking two endpoints of each open contour together with a line section. This produces enough closed contour matches. The registration parameters are calculated according to the centroids and the angles of long axes of closed contour matches. Experiments show that the proposed contour-based multisensor image registration algorithm works well in real data even when no matched closed contour pairs can be extracted from the input images to be registered. The paper is organized as follows. Section 2 discusses the proposed contour-based multisensor image registration algorithm with rigid transformation. The experiment results are shown in Section 3. Conclusions are given in Section 4.
2
Contour-based multisensor image registration algorithm
The registration scheme is shown in Fig. 1.There are five steps in the proposed algorithm. The first step is to extract many salient contours from the source images to be registered. Contour matching using improved chain code criteria and invariant moments is then performed. In order to utilize the information of open contours, we change open contour matches to closed contour matches. There will be some mismatches in the closed contour pairs (including the closed contour pairs changed from open contour pairs). So the next step is to remove those mismatches. Finally, the registration parameters are calculated according to the well-matched closed contour pairs. 2.1.
Contour extraction
Feature extraction is an important step of feature-based image registration method. The ability to successfully match features from different images depends very much on the type, quantity and quality of the extracted features. In this paper, we take contours as the features used for the image registration. The contour extraction is divided into two steps: edge detection and edge linking. 1. Edge detection Edges in images are areas with strong intensity contrasts. The Canny edge detection algorithm [8] is well known to many as the optimal edge detector. In this paper we use it as the algorithm that implements the edge detection process. The Canny edge detector first smoothes out the
noise. It then employs the optimal 1st-derivative filter to compute the gradient field of the image and estimates the noise level as the RMS of the second-derivative of the filter response. This noise estimate determines two noise thresholds: Thigh and Tlow . The algorithm performs nonmaximal suppression by interpolating the gradient field, and edges are identified as points that are locally maximum in the gradient direction. Strong edges are those above Thigh ; weak edges are below Thigh , but above Tlow . An iterative process extends edges by progressively promoting weak edges to strong edges if they are contiguous with a strong edge (in the sense of pixel topology).
Reference Image
Sensed Image
Contour Extraction Open Contour Matching
Closed Contour Matching
Change Open Contour Matches To Closed Contour Matches Remove False Matches Calculate Centriods and Angles of the Long Axes of Matched Closed Contours Estimate Registration Parameters Fig. 1. Registration Scheme. 2. Edge linking Edge detectors yield pixels, which lie on edges. The next step is to collect these pixels together into a set of contours. The basic process is the tracking of a sequence of edge points. The practical problem may be much more difficult than the ideal one. This is mainly due to the reason that small pieces of edges may be missed in a contour. In this situation, we should connect the two contour segments together to form a whole contour. We make use of the similarity in the slopes of these two contour segments to perform edge linking. If two contour segments A and B are disconnected but the endpoint ( xiA , yiA ) of A is in the small neighborhood (normally 7 × 7 ) of the endpoint ( xiB , y iB ) of B , we will calculate the k-slope of A at ( xiA , yiA ) and the k-slope of
B at ( xiB , yiB ) . The k-slope of A at ( xiA , yiA ) is defined
d AB =
as
slope A ( xiA , y iA ) = arctan(
y iA− k − y iA ), xiA− k − xiA
∑ [φiA' − φ iB' ]2 .
(3)
i =1
(1)
where ( xiA− k , yiA− k ) is the k-th point next to the endpoint
( xiA , yiA ) in contour segment A , k is normally small (for example k = 4 ). Similarly, we can calculate the kslope of B at ( xiB , yiB ) , which is denoted as slopeB ( xiB , yiB ) . If slope A ( xiA , y iA ) − slope B ( xiB , yiB ) ≤ θ d ( θ d is a small threshold, for example θ d = π / 8 ), the two contour segments A and B will be conected to form a whole contour by linking the endpoints ( xiA , yiA ) and ( xiB , yiB ) together with a line segment. In order to improve the efficiency of this registration algorithm, we only preserve the contours that lengths are above a threshold L (its value may be large for large source images to be registered, in our experiments, L = 150 ). 2.2.
6
The smaller the distance is, the more similar the shapes of two regions are. If d AB is less than a threshold d (normally small, for example d = 0.05 ), the two closed contours A and B are selected as a matched closed contour pair. 2.3.
Change open contour matches to closed contour matches
If only closed contour matches are used to estimate registration parameters, poor registration accuracies may be obtained. In order to get enough closed contour matches, we change open contour matches to closed contour matches by linking two endpoints of each open contour together with a line segment. This process is illustrated in Fig. 2.
Contour matching
Open contour matching is based on the improved chain code criteria [6]. For two open contours A and B to be matched, the improved chain codes are calculated. According to the correlation measure of the improved chain codes, the best-fit N-point segments between contours A and B can be found ( N is less than the length of contour A or B but is larger than half of their minimum lengths). If the correlation measure of the bestfit N-point segments is larger than a threshold M T (normally larger than 0.95), these best-fit N-point segments will be selected as a matched open contour pair. For closed contour matching, we transfer the closed contour representation into the region representation by taking the closed contour as the region boundary. To matche two closed contours A and B , the correspondence region representation are marked as R A and R B . Six invariant moments [7,9] {φiA , i = 1, 2,…,6} and {φiB , i = 1,2, …,6} are calculated from region R A and region R B , respectively. We observe experimentally that normalized invariant moments are more suitable for closed contour matching. Taking the normalization of φ iA as an example, the normalization process is defined by
φ = ' iA
φ iA
, i = 1,2, …,6 ,
6
∑φ
(2)
2 jA
j =1
The invariant-moment distance between regions R A and
RB is defined by
(a)
(b)
(c)
(d)
Fig. 2. Change open contour match to closed contour match. (a) and (b) A matched open contour pair; (c) and (d) The closed contour pair after changing. 2.4.
Remove false matches
After contour matching and changing open contour matches to closed contour matches, many matched closed contour pairs {( Ai , Bi ) i = 1,2,…, N } ( N is the total number of matched closed contour pairs) will be generated. There are sometimes mismatched contour pairs and multiple contours may get matched to the same contour. We need to reduce the false matches before estimating the registration parameters. For good matched contour pairs, there is a characteristic that the included angle between the
long axes of this contour pair is approximatively equal to the rotation angle between the reference and sensed images. In this paper, we utilize this characteristic to remove false matches. The included angle between the long axes of each contour pair is calculated. The average included angle is calculated. If the included angle between the long axes of a contour pair have a large derivation from the average included angle, this contour pair will be removed. To calculate the included angles between the long axes of matched contours is a little complicated to be calculated, the angle of long axis of each closed contour is first calculated. The long axis of a closed contour is also called as the axis of minimum inertia. This is the axis of the least two-order moment axis [10]. Denoting the angle of long axis of contour Ai as θ i and the angle of long axis of contour Bi as θ i' , the included angle of the long axes of contour Ai and Bi is defined as
φi = θ i − θ i' .
(4)
the sensed image. The centroid of a closed contour is defined as
∑y _ ( x , y )∈D x = ∑1 ( x , y )∈D . x ∑ _ y = ( x , y )∈D ∑1 ( x , y )∈D
where D is a region defined by taking this closed contour as its boundary. Suppose the angles of long axes and the centroids of the matched closed contours in the reference image and the sensed image are (θ i , xi , yi ) and (θ i' , xi' , yi' ) ( i = 1,2, … , Num , Num is the number of matched closed contour pairs after removing the false matches), respectively, we define
φi = (θ i − θ i' ) ' ∆xi = xi cosθ + yi sin θ − xi . ∆y = − x sin θ + y cosθ − y ' i i i i
The average include angle is defined as
φ=
N −[ N ⋅α ] 1 ∑φ N : j , N − 2[ N ⋅ α ] j =[ N ⋅α ]+1
(5)
where φ N = {φ i i = 1,2,…, N } , φ N: j is the j-th element of the ascending sorted version of φ N , α is a quantile ( 0 < α < 1 , normally 0.2~0.3), [X ] rounds X to the nearest integer. _
If φ i − φ > φT ( φT is a threshold, usually small, such as
φT = π / 50 ), the contour pair ( Ai , Bi ) is then discarded as a mismatched contour pair. 2.5.
Registration parameters estimation
The geometrical deformation between the reference and sensed images is assumed to be a rigid transformation in the proposed algorithm. Rigid transformation preserves the distance between any two points in the domain of transformation. There is no scaling change in a rigid transformation. A rigid transformation can be expressed as
x' cos φ sin φ x ∆x (6) ' = s ⋅ + , − sin φ cos φ y ∆y y where ( x' , y ' ) is the new coordinates of point ( x, y ) after the transformation, the scaling factor s = 1 , (φ , ∆x, ∆y ) correspond to the rotation angle and translations along the two orthogonal directions, respectively. The original points of the Cartesian coordinate systems are the centers of the images to be registered. X-axis is in the horizontal direction (the positive direction is to the right). Y-axis is in the vertical direction (the positive direction is downward). Under rigid transformation, the registration parameters can be estimated by the centroids and the angle of long axes of matched closed contour pairs in the reference image and
(7)
(8)
The final registration parameters are estimated by Nmu −[ Num⋅α ] 1 φ= ∑φ Num : j Num − 2[ Num ⋅ α ] j =[ Num⋅α ]+1 Nmu −[ Num⋅α ] 1 ∆x = ∑ ∆x Num : j , (9) Num − 2[ Num ⋅ α ] j =[ Num⋅α ]+1 Nmu −[ Num⋅α ] 1 ∆y = ∑ ∆y Num : j − ⋅ Num Num 2 [ α ] j = Num ⋅α ]+1 [ where φ Num = {φ i i = 1,2, … , Num} , φ Num: j is the j-th
element of the ascending sorted version of φ Num , likewise for ∆x Num : j and ∆y Num : j , α is a quantile ( 0 < α < 1 , normally 0.2~0.3), [X ] rounds X to the nearest integer.
3
Experiment results
Two experiments are conducted to evaluate the proposed algorithm. In experiment 1, the test images are airborne visible image (Fig. 3(a)) and infrared (IR) image (Fig. 3(b)). These images are acquired from the website http://www.sic.rma.ac.be/Projects/Sahara/. Matched closed contours are shown in Fig. 3(c) and Fig. 3(d). There are three closed contour pairs detected before changing the open contour pairs to closed contour pairs. These three contours are used to test the algorithm of Dai and Khorram (1999) [7]. The airborne visible image after registration using the proposed algorithm is shown in Fig. 3(e). In experiment 2, the test images are LANDSAT TM image (Fig. 4(a)) and SAR image (Fig. 4(b)). Matched closed contours are shown in Fig. 4(c) and Fig. 4(d). There are no closed contour pairs detected before changing the open contour pairs to closed contour pairs. In this situation, the algorithm of Dai and Khorram (1999) can not be applied.
The SAR image after registration using the proposed algorithm is shown in Fig. 4(e). In Fig. 3 and Fig. 4, the centroids and long axes of closed contours are denoted by “+” and lines. In order to compare the result of the proposed algorithm with the results of manual registration scheme, we visually locate control point pairs using a magnifier on the images to be registered. These control point pairs are used to estimate the manual registration parameters. The centroids of matched closed contours are used to calculate the RMSE of registration accuracies of each registration scheme. The RMSE is defined as
Registration Parameters
The proposed method Manual Dai and Khorram (1999)
φ
∆x
∆y
1
9.8883
51.2614
55.8390
3.1005
1.0171
9.8606
51.3743
60.9024
4.8184
Cannot be applied in this experiment
N
RMSE =
∑ (errorxi2 + erroryi2 ) i =1
,
Num
(10)
with
errorxi = x i ⋅ s ⋅ cos φ + y i ⋅ s ⋅ sin φ + ∆x − xi' , errory i = − x i ⋅ s ⋅ sin φ + y i ⋅ s ⋅ cos φ + ∆y − y i' , where Num is the number of matched closed contour pairs after removing the false matches, ( xi , y i ) and ( xi' , y i' ) are the centroids of matched closed contours, s , φ , ∆x, ∆y are the registration parameters obtained by
(a)
each registration scheme. The performances of the registration algorithms in these two experiments are reported in Table 1 and Table 2.
4
Conclusion
In this paper, we present a contour-based multisensor image registration algorithm with a rigid transformation. Salient contours are extracted from the images to be registered. Contour matching is performed to obtain many contour pairs. Open contour pairs are then changed to closed contour pairs, and false matches are removed from these closed contour pairs. From Table 1 and Table 2, we conclude that the registration accuracy of the proposed algorithm is better than that of manual registration scheme and the algorithm of Dai and Khorram (1999) in the situation of rigid transformation. Future work includes the contour-based registration with affine transformation. In the situation of affine transformation, affine invariant moments may be used to match open contours. Table 1. Comparison of the registration accuracies
(b)
Registration Parameters RMSE
scale
φ
∆x
∆y
The proposed method
1
8.0139
20.0635
20.2048
1.2664
Manual
0.9996
8.1803
19.5400
20.1160
1.8973
Dai and Khorram (1999)
1.0012
8.0416
19.9384
19.4099
1.4832
Table 2. Comparison of the registration accuracies
RMSE
scale
(c)
(d) (b)
(e) Fig. 3. The registration of airborne IR image and airborne visible image. (a) The reference image – airborne IR image; (b) The sensed image - airborne visible image; (c) The centroids and long axes of matched closed contours in the reference image; (d) The centroids and long axes of matched closed contours in the sensed image; (e) The sensed image registered to the reference image.
(c)
(d)
(a)
[8]
J.F. Canny, A Computational approach to edge detection, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 8, pp. 679-698, 1986.
[9]
M.K. Hu, Visual pattern recognition by moment invariants, IEEE Trans. Inform. Theory, Vol. IT8(2), pp. 179-187, 1962.
[10] B.K.P. Horn, Robot Vision, MIT Press, 1986.
(e) Fig. 4. The registration of Landsat TM and SAR image. (a) The reference image - Landsat TM image; (b) The sensed image - SAR image; (c) The centroids and long axes of matched closed contours in the reference image; (d) The centroids and long axes of matched closed contours in the sensed image; (e) The sensed image registered to the reference image.
References [1]
L.G. Brown, A survey of image registration techniques. ACM Computer Surveys, Vol. 24, No. 4, pp. 325-376, 1992.
[2]
F. Maes, A. Collignon, Vandermeulen D., Marchal G., Suetens P., Multimodality image registration by maximization of mutual information, IEEE Trans. on Medical Imaging. Vol. 16, pp. 187-198, 1997.
[3]
P. Thévenaz, M. Unser, Optimization of mutual information for multiresolution image registration, IEEE Trans. on Imaging Processing. Vol. 9, pp. 2083-2099, 2000.
[4]
B.S. Reddy, B.N. Chatterji, An FFT-based technique for translation, rotation, and scaleinvariant image registration, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 5, pp. 1266-1270, 1996.
[5]
E. De Castro, C. Morandi, Registration of translated and rotated images using finite Fourier transforms, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. PAMI-95, pp. 700-703, 1987.
[6]
H. Li, B.S. Manjunath, S.K. Mitra, A contour based approach to multisensor image registration, IEEE Trans. on Imaging Processing, Vol. 4, pp. 320-334, 1995.
[7]
X.L. Dai, S. Khorram, A feature-based image registration algorithm using improved chain-code representation combined with invariant moments, IEEE Trans. on Geoscience and Remote Sensing, Vol. 37, pp. 2351-2362, 1999.
