Detecting Multiple Symmetries with Extended SIFT

1

Detecting Multiple Symmetries with Extended SIFT

2

Anonymous ACCV submission

3

Paper ID 388

4

Abstract. This paper describes an effective method for detecting multiple bilateral symmetries of planar objects under perspective projection. The method can detect multiple symmetrical objects and the multiple symmetry axes of each object. In this paper, an extended SIFT feature called “pseudo-affine invariant SIFT” is proposed for detecting symmetric feature pairs that show different appearance in images due to perspective projection. Candidates of symmetry axes are obtained by finding the two projected mid-points of every two symmetric pairs based on the cross-ratios of four points on a line. The symmetry axis candidate that has the greatest number of symmetric pairs fitting it is detected as the most relevant symmetry axis of a symmetrical object. Other symmetric axes of the object are detected from the symmetric pairs belong to the symmetry axis. The procedure is applied repeatedly to the symmetric pair set after eliminating the ones of the detected symmetrical object to detect all symmetrical objects.

5 6 7 8 9 10 11 12 13 14 15 16

17

1 Introduction

18

Bilateral reflection symmetry exists in many artificial objects, animals and plants. By detecting the multiple symmetries of objects, not only the global geometric properties such as symmetry axes can be obtained, but also the sets of pairs of image features at the symmetric positions on each symmetrical object can be obtained. This also has the effects of grouping image features and establishing pair-wise correspondence between features scattered in an image and thus, provides rich information about the global structure of objects. This paper proposes a powerful method for detecting multiple bilateral symmetries of planar objects under perspective projection, and grouping the associated symmetric pairs of features from a single view. We extend SIFT feature descriptor so that it can cope with affine transformations. This extension greatly increases the matching capability of the SIFT feature descriptor, and the symmetric pairs can be detected much more efficiently than the original SIFT while keeping low false positive detection rate. The candidates of symmetry axes are estimated by finding the two projected midpoints of every two symmetric pairs based on the cross-ratios of four points on a line. The symmetry axis candidate that has the greatest number of symmetric pairs fitting it is detected as the symmetry axis of a symmetrical object. All symmetry axes of the symmetrical object are detected by detecting the candidates of symmetry axes from the symmetric pairs associated with the object then selecting all the ones that have large number of symmetric pairs fitting them. This procedure is then applied repeatedly to the remained symmetric pairs after eliminating the ones associated with the detected symmetrical object to detect all symmetrical objects. The method can cope with the

19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

2

ACCV-10 submission ID 388

44

perspective distortions of not only the geometry arrangement, but also the appearance, of the symmetric pairs of features. The method is able to detect multiple symmetrical objects and the multiple symmetries within the same object. This method does not require that the geometry arrangement, the appearance, and the feature descriptor of the symmetric pairs of features remain symmetrical in input images.

45

2 Background

40 41 42 43

46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

There have been a lot of works dealing with the detection of symmetry in the computer vision for several decades (e.g. [1][2][3][4][5][6][7][8][9][10]. Symmetry detection has been used for many applications; including reconstruction[11][12], pattern recognition[13] and stereo vision[14]. Liu et al.[15] used edge-based features and exhaustive search to identify all single reflection symmetries. Since the algorithm was designed for analyzing artistic paper cutting patterns, the difficulty caused by perspective or affine distortion, image noise or complex background was not considered. Loy et al.[4] used SIFT to detected feature points and to find the pair wise matching. The symmetry was detected by voting for symmetry foci with Hough transform. It can detect reflection and rotation symmetry. In the algorithm of finding pair wise matching and estimating symmetric axes, the method used the orientations and the positions of image features directly without considering perspective or affine distortion of symmetrical objects. In contrast our method takes account of such distortions both in finding symmetric pairs and in estimating symmetry axes. Cornelius et al.[5] used several feature detection methods including SIFT, hessianaffine and Harris affine detector, to detect feature points and to find pair wise matching. The symmetry was detected by voting for symmetry foci with Hough transform. It can detect reflection symmetry in perspective projection. However, the detail of the algorithm for finding pair wise matching was not given in the paper.

3 Pseudo-affine invariant SIFT features for detecting symmetric pairs Due to perspective projection, the image patches of the parts of a symmetrical object at symmetrical positions show different appearances. The appearances may be different in scale, orientation and skew, as well as reflection. In order to detect symmetric pairs of feature points in images, we need a feature detector that can detect distinctive points with good repeatability, and give feature descriptors that can be used for estimating the similarities between features in the case that the perspective distortion occurs. Many symmetry detection methods used SIFT feature detector because of its good performance for detecting features and its information rich feature descriptor. While the SIFT feature descriptor is scale- and rotation-invariant, it is not invariant for skew distortion. Here we extend the SIFT descriptor so that it can be used to find pair wise matching among image patches that contain scale, rotation and skew distortion. We first use the SIFT to detect feature points in images. The SIFT gives the orientation and the scale of features in addition to the positions. In order to give the SIFT

ACCV-10 submission ID 388 81 82 83 84 85 86 87 88 89 90

feature descriptor the ability to estimate the similarity between two image features that contain skew distortions, we define several skewed coordinates systems and their mirrored versions (by flipping about the Y axis) for the image patch that was used to compute the SIFT descriptor (See figure 1). The Y axis is aligned to the orientation of the detected feature. We quantize and enumerate the angle between X-Y axes of the skewed coordinates systems. In the experiments, we let the angles be 60, 90 and 120 degrees. Each angle is then used to define a skewed coordinates system, which is then used normalize the image patch so that the skewed axes become orthogonal. We use each of the normalized image patches to re-calculate a SIFT descriptor. Thus, each feature point (p) will have a set of SIFT descriptors ( f (p) ). f (p) = {fi (p), mi (p)}, i = 1, 2, 3.

91 92 93

3

(1)

where fi (p) and mi (p) is the SIFT descriptor (a feature vector of 128 dimensions) computed using i-th skewed coordinates system and its mirrored version, respectively. We call this extended SIFT as “pseudo-affine SIFT features” or simply PA-SIFTs.

Fig. 1. Definition of pseudo-affine SIFT features (PA-SIFTs).

94

4 Detecting symmetric pairs with PA-SIFTs

95

In order to detect symmetric pairs of features, the difference between any two features is estimated by considering the feature difference and the scale difference between them. The feature difference between two feature points p and q is computed from the PASIFTs of them as

96 97 98

F (f (p), f (q)) =

min

i∈{1,2,3},j∈{1,2,3}

(fi (p) − mj (q), mi (p) − fj (q)) .

(2)

4 99 100

ACCV-10 submission ID 388

The scale difference between two feature points p and q is computed from the scales of them obtained from the SIFT detector as S(p, q) =

101 102

max(s(p), s(q)) , min(s(p), s(q))

(3)

where s(p) and s(q) are the scales of p and q, respectively. The difference between two feature points p and q defined by D(p, q) = F (f (p), f (q))S(p, q).

(4)

106

If D(p, q) is less than a predefined threshold T D , then the pair of feature p and q is detected as a symmetric pair. Figure 4 shows an example result. The detected feature points are shown in figure 4(a), and the detected symmetric pairs are shown in figure 4(b).

107

5 Detecting multiple symmetries in an image

108

118

After the symmetric pairs were detected, we estimate a symmetry axis candidate from every two symmetric pairs by assuming that they are on the same symmetrical planar object. For each candidate, we estimate an average “fitness” of all symmetric pairs to it. A symmetrical object can be detected by finding the symmetry axis candidate that has the highest evaluation. The symmetric pairs on this symmetrical object can be obtained by selecting the symmetric pairs that have high “fitness” value to the detected symmetry axis. Then we use these symmetric pairs to detect multiple symmetry axes of the detected symmetrical object. In order to detect other symmetrical objects in the image, we discard all pairs belong to the detected symmetrical object then apply the procedure for detecting one symmetrical object to the remained symmetric pairs again and again until no more candidates symmetry axes with high “fitness” can be found.

119

5.1 Detecting symmetric axis candidates from symmetric pairs

120

After detecting symmetric pairs, we detect an axis candidate of symmetry from each two symmetric pairs. Fig.2 shows two symmetric pairs {p i , qi } and {pj , qj } on the same symmetrical planar object thus sharing the same symmetry axis. This symmetry axis can be determined by estimating the two projected midpoints of 3D points of two pairs. Since the lines connecting symmetric pairs are parallel in 3D space, the vanishing point Sij of them is their intersection. The projected midpoints of the two symmetric pairs can be computed by using this vanishing point based on cross-ratios of four points on a line in perspective images. Let mi be the projected mid-point of {p i , qi }, and Mi , Sij , Pi , Qi be the 3D point of mi , pi , qi , respectively. According to the cross-ratios of four points on a line in perspective images, we have the following equations.

103 104 105

109 110 111 112 113 114 115 116 117

121 122 123 124 125 126 127 128 129 130

Qi − Pi Sij − Mi  qi − pi sij − mi  = , qi − mi sij − pi  Qi − Mi Sij − Pi 

(5)

ACCV-10 submission ID 388

5

Fig. 2. An axis candidate of symmetry is estimated from two symmetric pairs.

131

132

⎧ sij − mi  = qi − mi  + sij − qi  ⎪ ⎪ ⎨ Qi − Pi  = 2Qi − Mi  S ⎪ ij − Mi  = |Qi − Mi  + ∞ ⎪ ⎩ Sij − Pi  = 2|Qi − Mi  + ∞ Solving eq.(5) and eq.(6) we obtain: qi − mi  =

133

135 137

sij − qi  (qi − pi ). 2sij − pi  − qi − pi 

(8)

sij − qj  (qj − pj ). 2sij − pj  − qj − pj 

(9)

The line connecting m i and mj describes the symmetry axis. We use this line l ij and the vanishing point s ij that indicates the 3D orientation of the lines connecting the symmetric pairs, to describe the symmetry axis. Mij = {lij , sij }

138 139 140 141

(7)

Similarly, we also have mj = qj −

136

qi − pi sij − qi  2sij − pi  − qi − pi 

The projected mid-point m i can be obtained as mi = qi −

134

(6)

(10)

5.2 Detecting the most relevant symmetry axis and the associated symmetric pairs We estimate a set of candidates of symmetry axes and their parameters as described in sub-section 5.1 by using all combinations of any two symmetric pairs. For each

6 142 143 144

ACCV-10 submission ID 388

symmetry axis candidate, we estimate a degree for every symmetric pair that indicates how well it fits the symmetry axis candidate by considering the orientation of the line connecting the symmetric pair and the position of its projected midpoint.

(a)

(b)

Fig. 3. Estimating the fitness of {p, q} to the symmetry axis lij .

145 146 147 148

149 150

As shown in Fig.3(a), {p, q} is a symmetric pair, l ij is a symmetry axis candidate and sij is the vanishing point of the symmetric pairs used for estimating l ij . If {p, q} is a symmetric pair of l ij then the line connecting them should pass s ij . Here we compute two angles θp and θq :  θp = ∠sij pq (11) θq = π − ∠sij qp We define an angle θ e that describes the difference between the 3D orientation of the line {p, q} and the vanishing point s ij as follows. θe = max(θp , θq )

151 152 153 154 155 156 157

158 159 160 161

(12)

As shown in figure 3(b), M p is the intersection of the line connecting {s ij , p} and lij , and Mq is the intersection of the line connecting {s ij , q} and lij . If p and q is a symmetric pair of l ij then Mp and Mq should overlap at the projected mid-point of p and q. Since any feature point in a symmetric pair can be computed from the other feature point, the projected mid-point and the vanishing point based on the cross-ratios, the “correct” position of p and q can be computed as p  and q from {sij , q, Mq } and {sij , p, Mp }, respectively. We compute the two distances:  dp = p − p  (13) dq = q − q  In the case that p and q is a symmetric pair of l ij , p and q should overlap p and q, respectively and both d p and dq should be 0. We define a normalized distance d e that describes how well the symmetric pair fits lij by considering the position of its projected midpoint as follows. de =

max(dp , dq ) dpq

(14)

ACCV-10 submission ID 388 162 163 164

165 166

167 168 169

170 171 172

Here, dpq = p − q is for normalization. We use Mahalanobis distance to describe how well that the symmetric pair {p, q} fits the symmetry axis described by M ij as follows.   1 v(Mij , p, q) = exp − xT Σ−1 x , (15) 2 T

Where x = [θe de ] . Assuming that the correlations between θ e and de can be neglected, Σ can be expressed by a diagonal matrix as follows.

σθ 0 Σ= , (16) 0 σd where σθ and σd are the standard deviations of θ e and de , respectively. In order to estimate how well that a symmetry axis candidate fits the symmetric pairs, we compute the average fitness to all symmetric pair. nsp v(Mij , pi , qi ) V (Mij ) = i=1 , (17) nsp where nsp is the number of detected symmetric pairs. We compute V for each symmetry axis candidate described by M ij and select the one with the greatest value as the most relevant symmetry axis of the symmetric planar object. Mobject = argmin {V (Mij )}, Mij ∈Wm

173 174 175 176 177 178

7

(18)

where Wm is the set of all candidates of symmetry axes. We then compute the fitness of each symmetric pair to M object with eq.(15) and select the ones as the symmetric pairs of the symmetric planar object if v n (Mobject ) > Tsp . Here, Tsp is a threshold value. Fig.4 shows an example result. Fig.4(b) shows the detected symmetric pais, and Fig.4(c) shows the detected symmetry axis and the symmetric pairs of it.

(a) Detected feature points

(b) Symmetric pairs (c) Symmetry axis and its symmetric pairs

Fig. 4. Detecting the most relevant symmetry axis in an image.

8

ACCV-10 submission ID 388

179

5.3 Detecting multiple symmetries

180

205

In this section we expand the approach described in the previous section to be able to find multiple symmetries in the same image. In general, two cases exist: - There are two or more symmetrical planar objects in an image. - An object has more than one symmetry axes. In order to detect multiple symmetrical objects in an image, one may simply repeat the process of finding the symmetry axes until the average fitness of the found symmetry axis is less than a certain threshold. The main drawback of this method is that the evaluation of the next symmetrical object is affected by symmetric pairs that were already used by the previous symmetrical object and therefore, it can decrease the effectiveness of the detection. To minimize the error of the next symmetrical planar objects detection; we propose to discard all pairs that belong to the previous symmetrical objects and to not take it into consideration in the evaluation of the next symmetrical objects. For each detected symmetrical object, we constitute a rectangular bounding box aligned to the symmetry axis that just contains all the symmetric pairs of the object. This bounding box is an approximation of the area of the detected symmetrical object. All symmetric pairs having at least one of the feature points inside this area are discarded. Then we re-find the most relevant symmetry axis from the existing symmetric pairs and so on. The process is repeated until the estimated symmetry axis has less than 2 pairs (this value is determined experimentally). After all symmetrical objects have been detected; we try to find all symmetry axes in each symmetrical object by using the symmetric pairs belonging to it. We first detect the most relevant symmetry axis which has the highest evaluation of the fitness. We determine the pair that belong this symmetry axis. Then we determine the second highest evaluation symmetry axis by using always all pairs belonging to the object and so on. We repeat the process until the evaluation is less than a percentage of the maximum evaluation (the first maximum). In this paper we choose the percentage of 80%.

206

6 Experimental results

181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204

213

The effectiveness of the proposed method was confirmed through the experiments that used simulation images and real images. In the experiments, we let the number of feature points detected by the SIFT be about 500 points. The sizes of local image patches for computing PA-SIFTs was determined by the scale given by SIFT. The standard deviation of the angle error σ θ and the distance error σ d was set to 3.5(◦ ) and 0.15(pixel) respectively according to the results of some preliminary experiments. The threshold Tsp was set to 0.5.

214

6.1 The comparative experiments of detection accuracy

215

In this experiment, we compared the performance of three feature descriptors for detecting symmetric pairs: (a) original SIFT, (b) mirrored SIFT and (c) PA-SIFTs. We detected the symmetric pairs and used them to detect the symmetry axes by using (a), (b) and (c). Then we compared the accuracy of the symmetry axes detected with (a), (b)

207 208 209 210 211 212

216 217 218

ACCV-10 submission ID 388

(a) Generating simulation images

9

(b) Accuracy evaluation

Fig. 5. The experiment for evaluating detection accuracy

219 220 221 222 223 224

and (c). The skewed images are generated by rotating the vertical axis of the original image by θt . We let θt be 50, 60, 70, 80, 90, 100, 110, 120, and 130( ◦), and generate nine simulation images of 640×480 pixel, which were used as input images. The detection accuracy was evaluated by using the inclination error θ e between the true symmetry axis and the detected symmetry axis, and intercept error d e on the original vertical axis of the image (see Fig.5(b)).  θe = |θt − θd | (19) de = d

Table 1. Detection accuracy (de : pixel, θe : ◦ ) (a) de θe (b) de θe (c) de θe

mean Maximum Minimum Standard deviation 16.790 57.768 0.266 20.702 11.955 48.565 0.049 15.705 5.764 28.730 0.054 8.886 10.842 51.024 0.040 17.763 3.120 11.852 0.005 3.854 1.577 3.910 0.053 1.350

225 226 227 228 229 230 231 232 233 234

The experimental results of detection accuracy using the feature descriptor (a), (b) and (c) are summarized in table 1. These results showed that our PA-SIFTs give significantly smaller detection errors than the original SIFT and the mirrored SIFT. This indicates that the detection of symmetrical planar objects using the PA-SIFTs is robust and sufficiently accurate for all simulation images. Figure 6 shows some experimental results of symmetrical planer object detection using our PA-SIFTs. In this figure, the true symmetry axis overlaps with the detected symmetry axis. We confirmed that the symmetric pairs and the symmetry axis were detected successfully in each simulation image.

10

ACCV-10 submission ID 388

θt = 50

θt = 60

θt = 70

θt = 80

θt = 90

θt = 100

θt = 110

θt = 120

θt = 130

Fig. 6. Example results of detecting symmetry using PA-SIFTs.

235

6.2 Experiments using real images

236 238

Figure 7 shows some results of the symmetry detection in real images. We confirmed that the symmetrical planar objects could be detected successfully with our method even when the images showed significant deformation due to perspective projection.

239

6.3 Experiments using images in common databases

240

243

We also tested our method with images selected from some common databases: “Caltech256 Object Category Dataset”[16] and “MSRC Object class recognition database A and B1”[17]. Some results are shown in figure 8 where the symmetry axes and symmetric pairs were detected successfully.

244

7 Conclusion

245

In this paper, we have proposed an extended SIFT feature “PA-SIFTs” that can be used to detect symmetric pairs efficiently in perspective images. We also have proposed the method for detecting multiple bilateral symmetries of planar objects in perspective images, which can detect the multiple symmetrical objects and all the symmetry axes in echo of the symmetrical object.

237

241 242

246 247 248 249

ACCV-10 submission ID 388

11

Fig. 7. Many example results of symmetry detection.

253

Through the comprehensive experiments using not only the simulation images but the real images and the images in common databases, we have confirmed our method could detect symmetry axes and symmetric pairs of planar objects robustly and accurately in various input images.

254

References

250 251 252

255 256 257 258 259 260 261 262 263 264 265 266 267

1. M. Part, S. Lee, P.C.C., et al.: Performance evaluation of state-of-the-art discrete symmetry detection algorithm. Proc. of CVPR (2008) 1–8 2. Gupta, R., Mittal, A.: Illumination and affine- invariant point matching using an ordinal approach. Proc. of ICCV (2007) 1–8 3. Raymond, Y., Tam, P.: Application of elliptic fourier descriptors to symmetry detection under parallel projection. TPAMI 16 (1994) 277–286 4. Loy, G., Eklundh, J.: Detecting symmetry and symmetric constellations of features. Proc. of ECCV 3952/2006 (2006) 508–521 5. Cornelius, H., Loy, G.: Detecting bilateral symmetry in perspective. Proc. of POCV (2006) 6. Marola, G.: A technique for finding the symmetry axes of implicit polynomial curves under perspective projection. TPAMI 27 (2005) 465–470 7. Ha, V., Moura, J.: Affine-permutation invariance of 2-d shapes. Trans. on Image Processing 14 (2005) 1687–1700

12

ACCV-10 submission ID 388

MSRC object categorty dataset B1 (Category: Face)

Caltech-255 (Category: Tower)

Caltech-255 (Category: Umbrella) Fig. 8. Experimental results using images in common databases.

268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287

8. J.Podolak, P., et al.: A planar-reflective symmetry transform for 3d shapes. Proc. of ACM SIGGRAPH (2006) 549–559 9. H. Zabrodsky, S.P., Avnir, D.: Symmetry as a continuous feature. Trans. PAMI 17 (1995) 1154–1166 10. D. Reisfeld, H.W., Yeshurun, Y.: Context-free attentional operators: The generalized symmetry transform. IJCV 14 (1995) 119–130 11. K. Huang, W.H. Li, A.Y., Ma, Y.: On symmetry and multiple-view geometry:structure,pose,and calibration from a single image. IJCV 60 (2004) 241–265 12. D.Q.Huynh: Affine reconstruction from monocular vision in the presence of a symmetry plane. Proc. of ICCV 1 (1999) 476–482 13. Bigun, J.: Pattern recognition in images by symmetries and coordinate transformations. Computer Vision and Image Understanding 3 (1997) 290–307 14. Li, W., Kleeman, L.: Fast stereo triangulation using symmetry. Australasian Conference on Robotics and Automation (2006) http://www.araa.asn.au/acra/acra2006/contents.html. 15. Y.Liu, J.Hays, Y., Shum, H.: Digital papercutting. SIGGRAPH Technical Sketch, ACM (2005) 16. G. Griffin, A.H., Perona, P.: Caltech-256 object category dataset. Technical Report 7694 (2007) California Institute of Technology. 17. J. Winn, A.C., Minka, T.: Object categorization by learned universal visual dictionary. Proc. of ICCV 2 (2005) 1899–1807