Image Retrieval By Shape: A Comparative Study - CiteSeerX

Image Retrieval By Shape: A Comparative Study  Maytham Safar Cyrus Shahabi and Xiaoming Sun Integrated Media Systems Center Department of Computer Science University of Southern California Los Angeles, California 90089{0781 [safar; shahabi; xiaomins]@usc:edu November 17, 1999 Abstract

Besides traditional applications (e.g., CAD/CAM and Trademark registry), new multimedia applications such as structured video, animation, and MPEG-7 standard require the storage and management of well-de ned objects. This study compared four shape-based object retrieval techniques (FD, GB , DT , and TPV AS ). The similarity retrieval accuracy of our method (TV PAS ) was comparable to the other methods, while it had the lowest computation cost to generate the shape signatures of the objects. Moreover, it has low storage requirement, and a comparable computation cost to compute the similarity between two shape signatures. In addition, TPV AS requires no normalization of the objects, and is the only method that has direct support to RST query types. We also introduced a new shape description taxonomy.

1 Introduction Many applications in the areas of CAD/CAM and computer graphics require to store and access large databases [32, 31]. These object databases are then queried and searched for dierent purposes. A sample query might be \ nd all the scenes that contain a certain object". Therefore, given a query object, we would like to obtain a list of objects from the database which are most similar in some aspects to the query object. Shape of an object is an important feature for image and multimedia similarity retrievals. Therefore, this study focuses on shape-based object retrieval techniques. There is a variety of techniques that has been proposed in the literature for shape representation [17, 12]. In [12], shape representation techniques are divided into two categories: boundary-based and region-based. Boundary based methods use only the contour or border of the object shape and completely ignore its interior. On the other hand, the region based techniques take into account internal details (e.g., holes) besides the boundary details. The purpose of this paper is to evaluate and compare the performance of four boundary based methods for shape representation and retrieval: Fourier descriptors methods FD [22] (based on objects' shape radii), grid-based method GB [20] (based on chain codes), Delaunay triangulation method DT [18, 17] (based on corner points), and Touch-point-vertex-angle-sequence TPV AS (based on minimum bounding circles and angle sequences). TPV AS approach to shape representation and similarity measure was proposed by the authors in [10]. We compare its retrieval performance with that of the other 2 more established methods (i.e., FD, and GB methods) that were used in some commercial systems [12] and as a basis for dierent comparison studies in [20, 12]. In addition, it was compared to a new method based on a new indexing technique (i.e., DT) [17]. Although TPV AS method uses simple attributes such as minimum bounding circles and angle sequences (simple to extract), these attributes were shown to be translation, rotation, and scale invariant in [10]. Furthermore, TPV AS has low storage requirement and has the lowest computation cost to generate the shape signatures of the objects. Moreover, it has a comparable computation cost to compute the similarity between two shape signatures. In addition, the similarity retrieval accuracy of TV PAS comparable to the other methods.  This research was supported in part by NASA/JPL contract nr. 961518, and unrestricted cash/equipment gifts from Intel , NCR, and NSF grants EEC-9529152 (IMSC ERC) and MRI-9724567.

1

In some application domains, the support of RST query types in which the matching objects could be within a speci ed rotation angle, scaling factor, translation vector, or any combination of the three is important (see [10] for further details). An example is, searching for similar tumor shapes in a medical image database [32]. Tumors are presented by a set of 2D images that represent slices through it. A method for retrieving similar tumor shapes would help to discover correlations between tumor shape and certain diseases. Besides the shape, the size and the location of the tumor would help in the identi cation of patients with similar health history. With TV PAS method, a subset of the MBC features (e.g., radius r, center C , and start point SP) could be utilized to support RST query types. On the other hand, the other three methods lack the support of RST queries. This is due to either the use of normalization, or lack of features that could support such queries. In addition, this study introduces a new shape description taxonomy. Furthermore, we classify techniques that were not described in previous studies [12, 18, 17] (e.g., turning angle, collinearity, . ..etc.). The remainder of this paper is organized as follows. Sec. 2, provides some background material on shape description techniques, while Sec. 3 describes dierent types shape retrieval methods considered in this study. In Sec. 4 we provide cost analysis for the dierent methods and compares their eciency in terms of computation and storage requirements. Sec. 5 lists some of the drawbacks of the alternative shape description techniques. Sec. 6 reports on the performance results obtained from a set of experiments. Finally, Sec. 7 concludes the paper and provides an overview on our future plans.

2 Shape Description Techniques Shape description is an important issue in object recognition and its objective is to measure geometric attributes of an object, that can be used for classifying, matching, and recognizing objects. There are various methods for shape representation. An overview of shape description techniques is provided in [12]. It categorizes the techniques into boundary based and region based methods. Boundary based methods use only the contour of the objects' shape, while, on the other hand, the region based methods use the internal details (e.g., holes) in addition to the contour. The region-based methods are further broken into spatial and transform domain sub-categories depending on whether direct measurements of the shape are used or a transformation is applied. A drawback of this categorization is that it does not further sub-categorizes boundary based methods into spatial domain and transform domain methods. For example, Fourier descriptors method can be considered as a transform domain technique, while chain codes can be considered as spatial domain technique. Another drawback of this categorization is that it assumes that structural techniques (e.g., 2D-strings) are a sub-category of region-based, spatial domain techniques. However, we believe that structural techniques as 2D-Strings are spatial similarity based techniques, in which the retrieval of objects is performed based on the spatial relationships among objects and not shape retrieval techniques (see Fig.1). In another study [18], shape description techniques were broken into two dierent categorizes: transformation based and measurement based categories. They further breakdown the transformation based category into two subcategories: functional and structural categories, however, it is not clear what criteria is used to this end. Some drawbacks of this categorization are that it does not distinguish between boundary or region based techniques, and sometimes it miss categorizes some techniques. For example, chain code technique is categorized as a transformationbased technique, while it is a measurement-based technique. Another example is that silhouette moments could be used as a region-based technique but not as a boundary-based technique. Therefore, we introduce a new shape description taxonomy. Our complete taxonomy of shape description techniques is illustrated in Fig.1. We also added further techniques that were not described in [12, 18] (e.g., turning angle, collinearity, .. .etc.).

3 Object Shape Descriptions This study compares four NIP boundary based methods for object shape descriptions: Fourier descriptors method (FD) [22], grid-based method (GB) [20], Delaunay triangulation method (DT) [17], and Touch-point-vertex-anglesequence (TPV AS) [10].

2

Content Retrieval

Shape Description

Texture Description

Color Description

Spatial Location

Structural Boundary Based Methods Transform Domain

Region Based Methods

Spatial Domain (Geometric)

Transform Domain

Spatial Domain (Geomtric)

Hough Transform Walsh Transform Wavelet Transform

Discrete Fourier UNL Fourier NFD Fourier Fourier Descriptors Partial (Occlusion)

Partial (Occlusion)

Complete (No Occlusion)

Contour Segments Curvature Discontinuity Angle Right-Angelness Length Irregularity Complexity Sharpness Directedness

Contour Segments Breakpoints

Corner Points Chain Codes Shape Number Perimeter Area Elongation Compactness Significant Points Boundary Coordinates Turning Angle Collinearity Co-Curvilinearity Parallelism of Lines or Curves Convergence of Lines or Arcs Physical Proximity Length Radius of Shape Boundary Symmetry Fitting line Touch Points Touch Points Angle Sequence Vertex Angle Sequence Curvature Boundary Moment Invariants Circularity Aspect ratio Discontinuity Angle Right-Angelness Length Irregularity Complexity Sharpness Directedness Grid Cover Edge Direction

Primitives Rules 2D Strings 2D B -Strings 2D C -Strings 2D C+ -Strings Extended 2D Strings θR String Spatial Orientation Graph Quad-tree Spatial Arrangement

Complete (No Occlusion) Area Holes Euler Number Moment Invariants Silhouette Moments Zernike Moments Pseudo-Zernike Moments Max-Min Radii Compactness Elongation Symmetry Rectangle Cover Grid Cover Matrix Decomposition Polygon Cover Circularity Aspect ratio

Figure 1: Taxonomy of Shape Description Techniques

3.1 Fourier Descriptors Method The FD method [5, 12, 20, 21, 19, 26, 22] obtains the object representation in the frequency domain as complex coecients of the Fourier series expansion of the objects' shape signature. The method starts by obtaining a feature function of the object called shape signature f(k); which could be curvature based, radius based, or boundary coordinates based. f(k) is also called the Fourier descriptors (FDs) of the boundary. In the next step, a discrete Fourier transform of the shape signature is obtained. The Fourier coecients obtained are then used for shape representation, as index, and for shape similarity calculation. The discrete Fourier transformation (DFT) of a shape signature f(k) is given by:

X

N ?1 f(k) exp(?j2uk=N) u = 0; 1; : : :; N ? 1 F(u) = N1 k=0

(1)

Where N is the number of samples of f(k). Direct representation (e.g., radii) captures each individual detail of a shape, however, it is very sensitive to small changes and noise. As a consequence, a small change in the coordinates of the objects' boundary points may lead to a very dierent shape signature, hence, very poor retrieval performance. On the other hand, Fourier transformation captures the general feature of a shape by converting the sensitive direct representation measures into frequency domain. As a result, the data is more robust to small changes and noise. Therefore, Fourier transformation is used as shape representation instead of the direct representation (see [19] for further details). A popular shape signature is the shape radii (or centroidal distance) which computes the distance between points uniformly sampled along the object boundary and its centroid (its center of mass). When shape radii are used, only the distances between points are recorded and not the exact coordinates of the points. Thus, the shape signatures of the objects are translation invariant. Since the magnitude information of F(u) (i.e., jF(u)j) is invariant to rotation. Hence, rotation normalization is achieved taking only the magnitude information from F(u) and ignoring its phase information. Moreover, jF(0)j re ects the energy of the shape radii, thus jF(u)j = jF(0)j will be scale invariant. Therefore, in order to scale normalize the objects' FDs, the magnitude of all F(u)s are divided by the magnitude of F(0). As a result of 3

A sampling point r2

r1

r3 r4 r5

Centroid

r8 r7 r6

f(k) = [r1,r2,r3,r4,r5,r6,r7,r8]

(a): An object with 8 sampling points

(b): Fourier descriptor of the object

Figure 2: Fourier Descriptors Method normalization a new feature vector, which is invariant to translation, rotation and scale, can be generated as follows: FN = [jF(1)j=jF(0)jjF(2)j=jF(0)j :: : jF(N)j=jF(0)j]T [19]. The dierence between two objects is de ned as the Euclidean distance between their corresponding feature vectors. Hence, the similarity measure of FD method is that two objects are similar in shape, if and only if the Euclidean distance between their feature vectors is less than a prespeci ed threshold. In other words, they have the same set of distances between their centroids and their sampled boundary points. An example is illustrated in Fig.2, where Fig.2(a) shows an object with its 8 sampling points and its centroid, and Fig.2(b) shows the radii distances and the Fourier descriptors f(k) of the boundary.

3.2 Grid Based Method An alternative approach for shape representation is GB method [23, 19, 20, 21, 12] With this method, an object is rst normalized for rotation and scale. Then, the object is mapped on a grid of xed cell size. Subsequently, the grid is scanned and a 1 or 0 is assigned to the cells depending on whether the number of pixels in the cell which are inside the object are greater than or less than a predetermined threshold. Finally, a unique binary number is obtained as the shape representation by reading the 1's and 0's assigned to the cells from left to right and top to bottom. To improve the eciency of this method, another shape feature, called eccentricity [20], was used. Eccentricity of shape is the ratio of the number of cells used in x-direction to the number of cells used in the y-direction to represent a shape. Therefore, for two objects to be similar, their shape signatures and their eccentricities values should be similar. The dierence between two objects is the number of cells in the grids which are covered by one shape and not the other, which is the same as the sum of 1's in the result of the exclusive OR of their binary numbers. Hence, the similarity measure of GB method is that two objects are similar in shape, if and only if the dierence between their binary representations is less than a prespeci ed threshold, and they have similar eccentricities. An example is illustrated in Fig.3, where the objects are mapped on to a grid of xed cell size in a manner such that the objects are justi ed to the top left corner (i.e., assuming that the objects do not need to be normalized). The binary numbers obtained for the objects in Fig.3(a) and (b) are \110001110 111111111 111111111 111111110 111111100 001110000" and \000000110 000011100 001111100 111111100 111111110 011111111" respectively. Hence, the dierence between the objects is 20. Binary number obtained for the same object with a dierent orientation in space or with a dierent scale will be dierent. Therefore, the normalization of the object boundaries prior to indexing is crucial to meet the uniqueness criteria of the binary number.

3.2.1 Rotation Normalization Rotation changes the spatial relationships between the grid cells and the objects' boundaries, which leads to dierent binary number representations for the same object. Therefore objects must be normalized for rotation. This is achieved by obtaining the major axis of the shape, which is the line joining the two points on the shape boundary farthest away from each other. Subsequently, the shape is rotated to make the major axis parallel to the x-axis. An example of rotation normalization of an object is shown in Fig.4, where in Fig.4(a) an object is shown prior 4

1 1 1 1 1 0

1 1 1 1 1 0

0 1 1 1 1 1

0 1 1 1 1 1

0 1 1 1 1 1

1 1 1 1 1 0

1 1 1 1 1 0

1 1 1 1 0 0

0 1 1 0 0 0

0 0 0 1 1 0

(a): Object-1

0 0 0 1 1 1

0 0 1 1 1 1

0 0 1 1 1 1

0 1 1 1 1 1

0 1 1 1 1 1

1 1 1 1 1 1

1 0 0 0 1 1

0 0 0 0 0 1

(b): Object-2

Figure 3: Grid Based Method - Object Mapping and Representation x

x Minor Axis

Major Axis

Major Axis y

y (a): Original object

(b): After rotation normalization

Figure 4: Grid Based Method - Rotation Normalization to rotation normalization and Fig.4(b) shows the same object after being rotated so that the major axis becomes parallel to the x-axis.

3.2.2 Scale Normalization Both the grid and the object sizes aect the binary number derived for an object. Therefore objects must be normalized for scale. This is achieved by choosing a xed length of the major axis, called the standardized major axis. Subsequently, scaling normalization is achieved by scaling along the major axis of the object such that its major axis becomes equal to the length of the standardized major axis. To maintain the perceptual similarity of the object, it is also scaled along the minor axis proportionally. The minor axis is perpendicular to the major axis and of such length that a rectangle with sides of major axis and minor axis de nes the minimum bounding rectangle of the object. An example of scale normalization of an object is shown in Fig.5, where in Fig.5(a) and (b) two similar objects are shown prior to scale normalization while Fig.5(c) shows the same objects after being scaled along the x-axis and y-axis such that the length of the major axis becomes equal to the standardized major axis.

3.3 Delaunay Triangulation Method This approach of shape representation is histogram-based [18, 17]. Given an object, corner points are used as the feature points of the object. Corner points are generally high-curvature points located along the crossings of an object's edges or boundaries. Then a Delaunay triangulation of these feature points is constructed. Consequently, a feature point histogram is obtained by discretizing the angles produced by this triangulation into a set of bins 5

x

x

x

Major Axis

Standardised Major Axis

Major Axis y

(a): Original object

y

y

(b): A similar object

(c): After scale normalization

Figure 5: Grid Based Method - Scale Normalization

Number of angles

Feature Point Histogram 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Bin number

(a): An object with 10 points

(b): Delaunay triangulation

(c): Feature point histogram

Figure 6: Delaunay Triangulation Method and counting the number of times each discrete angle occurs in the triangulation. In building the feature point histogram, a selection criteria of which angles will contribute to the nal feature point histogram is chosen. For example, the selection criteria could be counting the two largest angles, the two smallest angles, or all three angles of each individual Delaunay triangle. The dierence between two objects is the Euclidean distance between the corresponding bins of the objects' feature point histograms. Hence, the similarity measure of DT method is that two objects are similar in shape, if and only if the euclidean distance between their feature point histograms is less than a prespeci ed threshold. In other words, they have the same set of feature points. Thus, each pair of the corresponding Delaunay triangles in the two resulting Delaunay triangulations must be similar to each other. The angles of the resulting Delaunay triangles of a set of points is invariant under uniform translations, scalings, and rotations. Therefore, the similarity measure is independent of object's position, scale, and rotation (i.e. objects do not have to be normalized). This approach also supports an incremental approach to image object matching, from coarse to ne, by varying the bin sizes. An example is illustrated in Fig.6, where Fig.6(a) shows an object that consist of 10 feature points (corner points). Fig.6(b) shows the resulting Delaunay triangulation. While Fig.6 shows the resulting feature point histogram built by counting all three angles of each individual Delaunay triangle with bin size of 10 degrees.

3.4 Touch Point Vertex Angle Sequence Method This approach, TPV AS, for shape representation was proposed by the authors in another study [10]. In that study, we presented a multi-step query processing method that eciently supports the retrieval of 2D objects by shape. We proposed three index structures ITPAS , IVAS , and ITPVAS based on features that are extracted from the objects MBC. Those features are the center coordinates of the MBC, the radius of the MBC, the set of touch points on MBC (TPs), touch points angle sequence (TPAS), vertex angle sequence (V AS), and the start point of the angle sequence (SP). Fig. 7 provides an example of an object with its six identi ed features. In Fig. 7(b) the MBC of an object is computed and its touch points are identi ed as the vertices that lays on its MBC (denoted by X). Figs. 7(c) and (d) show both the touch points and vertex angle sequences, respectively. For the algorithms to extract the MBC features as well as their formal de nitions see [10]. A major observation is that those features are unique per object. In addition, a subset of the MBC features (e.g., TPAS , V AS , and TP s) are translation, scaling and rotation independent. This subset of features could be utilized 6

X

X

C

X

ϕ3

(a): Original object

ϕ4 ϕ5 ϕ6

X

X

(b): MBC of the original object with it's center C, radius r. TP denoted by X and SP denoted by bullet

(c): TPAS: ['1;'2;'3;'4]

ϕ1

ϕ3

ϕ4

r

X

ϕ2

ϕ1

ϕ2

ϕ9 ϕ8 ϕ7

(d): VAS: ['1;'2;'3;'4;'5;'6;'7;'8;'9]

Figure 7: MBC Features to support query types in which the matching objects could be invariant with respect to translation, rotation and scaling. It could also be utilized to support query types in which the matching objects could be within a speci ed rotation angle, scaling factor, translation vector, or any combination of the three (RST queries). We use object's SP to check if it is rotated, use object's r to check if it is scaled, and the vertex coordinates of the object's C to check if it is translated (see [10] for further details). With TPV AS method, the shape descriptor of an object depends on a subset of the six MBC features. The method starts by obtaining a feature function of the object called shape signature f(k). To obtain f(k), four steps are required. The minimum bounding circles of the objects are rst found. Second, MBC features such as angle sequences (V AS) and number of touch points (TP) are identi ed. Third, the objects' shape signature is identi ed as the sequence combination of TP and V AS. Finally, as with the FD method, a unique object representation is obtained as the discrete Fourier series expansion of the objects' shape signature. The Fourier coecients obtained are used for shape representation, as index, and for shape similarity calculation. The discrete Fourier transformation (DFT) of a shape signature f(k) is given by:

X

N ?1 f(k) exp(?j2uk=N) u = 0; 1; : : :; N ? 1 F(u) = p1 N k=0

(2)

Where N is the number of points in the polygon representation of an object. Although TPV AS method uses simple attributes such as minimum bounding circles and angle sequences, these attributes were shown to be translation, rotation, and scale invariant in [10]. When angle sequences are computed, only the angle between two vectors connecting the center of the MBC and the objects' points were recorded. Thus, it does depend on the exact coordinates of the points. Therefore, the shape signatures of the objects are translation invariant. Rotating two intersecting vectors in space does not change the angle between them. In addition, by taking the magnitude information of F(u) (i.e., jF(u)j), we are insuring that the angle sequence is independent of the start point choice. Hence, TPV AS is also invariant to rotation. When applying uniform scaling to an object (assuming that the origin of the coordinate system is the center of the MBC), we are only changing its size and not its shape. Changing the size means that only the proportion of the distances between the points of the object and the center of its MBC changes. Therefore, the new object's AS is equivalent to the old object's AS. Hence, TPV AS is also scaling invariant. For further details refer to [11]. Therefore, the similarity measure is independent of object's position, scale, and rotation (i.e. objects does not have to be normalized). The dierence between two objects is the Euclidean distance between their corresponding feature vectors. Hence, the similarity measure of TPV AS is that two objects are similar in shape, if and only if the Euclidean distance between their feature vectors is less than a prespeci ed threshold. In other words, they have similar vertex angle sequences and number of touch points. An example is illustrated in Fig.8. Fig.8(a) shows an object with 9 vertices with its minimum bounding circle, and touch points. Fig.8(b) shows the vertex angle sequence of the object and its shape descriptors f(k).

7

TP

X ϕ2 ϕ1

MBC

X

TP and SP

C

ϕ3 X ϕ4 ϕ5 ϕ6

r

X

ϕ9 ϕ8 ϕ7 X

TP

VAS = [ϕ1,ϕ2,ϕ3,ϕ4,ϕ5,ϕ6,ϕ7,ϕ8,ϕ9] f(k) = [TP,ϕ1,ϕ2,ϕ3,ϕ4,ϕ5,ϕ6,ϕ7,ϕ8,ϕ9]

X TP

(a): An object with its MBC , center C , radius r, TP 's, and

SP

(b): Vertex angle sequence V AS and shape descriptor f (k)

Figure 8: Touch Point Vertex Angle Sequence Method

4 Computation and Storage Cost Analysis This section provides cost analysis for the dierent methods and compares their eciency in terms of computation and storage requirements.

4.1 Grid Based Method For the GB method, we assume that: the grid cell size gnXgn, the standardized major axis size mj, total number of pixels of the grid px = mj 2 , number of coordinates of object N, and binary sequence size SS = (mj=gn)2 =8 bytes. Then the following storage is required for the shape signatures:

   

8 bytes to store the eccentricity of each shape (one real number). SS = ((mj=gn)2 )=8 bytes to store one binary sequence. 2  n  SS + 8  n bytes to store n shape signatures (assuming at least 2 sequences per object). A temporary storage of px=8 bytes per object.

To compute the shape signature for the query object and computing the similarity between the query object and the objects in the database the following major operations are required :

    

To nd major axis O(N 2) To nd minor axis O(N) To rotate and scale O(N) To check if a point is inside a polygon O(N) To generate a binaryp number we need to check all the grid cells/ pixels to see if they are inside the shape O(px  N). Usually px  N, therefore O(px  N)  O(N 3).  For each similarity calculation we need: exclusive OR between the query binary sequence and the stored-shape binary sequence, and count the number of ones in the result. Hence, similarity calculation takes around 8  SS bits XOR operations and 8  SS bits additions.

4.2 Fourier Descriptors Method For the FD method, we assume that: number of radii equals to r, number of Fourier coecients is fn, number of coordinates of an object N. Then the following storage is required for the shape signatures: 8

 To store one shape signature: 8  fn bytes.  To store n shape signatures: 8  n  fn bytes. To compute the shape signature for the query object and computing the similarity between the query object and the objects in the database the following major operations are required :

   

To nd centroid O(NlogN) (we compute the Delaunay triangulation to this end). To nd all radii and to compute FD coecients O(r2 ). To generate shape signature O(r) (required for normalization step). For each similarity calculation we need: fn real-number subtractions, fn real-number multiplications, fn ? 1 real-number additions.

4.3 Delaunay Triangulation Method For the DT method, we assume that: number of bins nb, and number of coordinates of object N. Then the following storage is required for the shape signatures:

 To store one shape signature: 4  nb bytes.  To store n shape signatures: 4  n  nb bytes.  Temporary storage: 3  N (per object). To compute the shape signature for the query object and computing the similarity between the query object and the objects in the database the following major operations are required :

 To compute the Delaunay triangulation of a shape: O(NlogN).  To generate shape signature: O(N) (required to nd all angles).  For each similarity calculation we need: nb integer-number subtractions, nb integer-number multiplications, nb ? 1 integer-number additions.

4.4 TPVAS Method For the TPV AS method, we assume that: number of Fourier coecients fn, and number of coordinates of an object N. Then the following storage is required for the shape signatures:

 To store number of touch points 1 byte.  To store one shape signature : 8  fn bytes.  To store n objects: 8  n  fn bytes. To compute the shape signature for the query object and computing the similarity between the query object and the objects in the database the following major operations are required :

 To nd MBC O(N).  To compute angle sequence and nd all DFT coecients of an angle sequence O(N).  For each similarity calculation we need: fn real-number subtractions, fn real-number multiplications, fn ? 1 real-number additions.

9

5 Discussion Support for RST queries For some application, the concept of shape similarity is that the matching objects

are invariant to scale, size, and orientation. On the other hand, for other applications, it is important that the matching objects be within a speci ed rotation angle, scaling factor, translation vector, or any combination of the three (RST query). With TV PAS method, a subset of the MBC features (e.g., r, C, and SP) could be utilized to support (RST) query types in which the matching objects could be within a speci ed rotation angle, scaling factor, translation vector, or any combination of the three (see [10] for further details). On the other hand, the GB, FD, and DT methods are unable to support such query type. With the GB and FD methods, normalization is used so that the objects t into a prespeci ed mesh, or force the boundaries to have a standard size, and orientation, respectively. While DT method representation provides no information about the size, and orientation of the original objects. Normalization For the GB and FD methods, the shape representations obtained for the same object with dierent orientation in space or with a dierent scale are dierent. Therefore, the normalization of the object boundaries prior to indexing is crucial to meet the uniqueness criteria of the shape descriptor. The DT and TPV AS methods use simple shape attributes such as the angles of the resulting Delaunay triangles of a set of points, minimum bounding circles and angle sequences. However, these attributes were shown to be translation, rotation, and scale invariant. Therefore, their similarity measures are independent of object's position, scale, and rotation (i.e. objects does not have to be normalized).

6 Experiments In many applications, an important criterion for testing the ecacy of the shape retrieval methods is that for each query object the relevant items (similar shapes) in the database should be retrieved. Therefore, an experiment was designed to measure the eectiveness of the similarity retrieval of the four dierent shape representation methods (i.e., FD, GB, DT, and TPV AS) in terms of recall and precision that are commonly used in the information retrieval literature. We implemented a prototype shape retrieval system in Java, on a SUN UltraSparcII workstation. The shape database of the system consists of 296 sh and tool shapes provided to us by [27, 28]. In order to reduce the number of vertices used to represent the shapes while maintaining its general characteristics, we employed a three pass algorithm. In each pass, we try to identify the straight lines on the boundary of the shape and eliminate the extra points representing the line. Optimally, a straight line could be represented by its two end points. However, the edge detection algorithm that was used to generate the shape les introduced more than two points to represent a straight line. The extra points makes the straight line looks like a jagged line (like saw edge). Fifteen shape queries were selected randomly from the database, where each query object has 10 similar (relevant) objects in the database. The ten similar objects' variants were constructed as: three rotation variants, three rotation scaled up variants, three rotation scaled down variants, and one translated variant. For our experiments, we simply assume that each shape is relevant only to itself and to its ten variants. As advised in [20, 12, 17, 10], for the GB method we employed a grid cell size of 12X12 pixels and the length of the standardized major axis was xed at 192 pixels. With FD method, we used radius-based signature with 64 uniformly sampled boundary points. For both FD and TPV AS method, we kept 8 low frequency coecients of the Fourier transform. Finally, for the DT method, we used two largest angle histogram with 18 bins of size 10 degrees. Given a query object, the system retrieves relevant shapes from the database, in decreasing order of similarity to the query shapes. First, the shape features of the query object using the four dierent methods are extracted. Second, the Euclidean distances between the shape signature of the query object and all other objects' shape signatures are computed 1 . Third, the Euclidean distances are ordered in an ascending order for all the methods. Finally, the accuracy of the retrieval methods was calculated as precision-recall. Towards this end, we identify the ranks of the relevant objects to the query object inside the ordered list (according to Euclidean distance). The recall and precision results for 15 query shapes are averaged and shown in Fig.9. From Fig.9(a), it is observed that the FD method performs worse than the other three methods. This poor performance is explained by the fact that the shape signature is constructed using shape radii of points uniformly sampled along the object boundary. These points are not the exact vertices of the object, therefore, using dierent sampled points leads to dierent shape signatures. In other words, the rotation normalization is not truly achieved by taking only the magnitude information from the Fourier coecients (F(u)s) and ignoring its phase information as explained in [20]. Fig.9(b), shows the 1

The retrieval can be done more eciently using index structures, but since this is not the focus of this study we assumed the sequential

scan of the database

10

97

100

96

80 60

TPVAS GB

40

FD DT

20 0

Precision

Precision

120

1 0 20

30

40

50

60

70

80

95 TPVAS GB DT

94 93 92 91

90 100

10

20

30

Recall

40

50

60

70

80

90

100

Recall

(a): Four Methods

(b): Three methods

Figure 9: Recall-Precision Curves Table 1: Costs GB  2  n  SS

DT 4  nb 4  n  nb temp 3  N

GB

DT

8

a) Storage cost

SS

temp px=8

b) Shape signature generation cost

c) Similarity computation cost

O(N 2 ) O(N ) O(N ) O(N ) O(px  N )

GB 8  SS bit xor 8  SS bit add

O(NlogN ) O(N )

DT

nb int sub nb int mult nb ? 1 int add

FD 8  fn 8  n  fn FD

O(NlogN ) O(r2) O(r)

FD

fn float sub fn float mult fn ? 1 float add

TPVAS  fn 8  n  fn 8

TPVAS

O(N ) O(N )

TPVAS

fn float sub fn float mult fn ? 1 float add

recall-precision curves of the methods excluding FD method. From Fig.9(b), we make the following observations. First, that the accuracy of our proposed method (TV PAS) is comparable to the other methods. Second, unlike what was expected, the results obtained for the DT method were better than what was reported in [17]. This is due to the fact that in their approach, they reduce the number of vertices used to represent an object by identifying high-curvature points called corner points. On the other hand, in our experiments, we used the three pass algorithm. In their experiments, the number of vertices used to represent an object was approximately 20-26 points, however, in our approach it was from 50-90 points. Although our approach used more vertices to represent the objects, however, the representation is more discriminative in shape signatures. We can conclude that the DT method performance highly depends on the technique used to nd the representative vertices of the objects. Sec. 4 provides cost computation analysis for the dierent methods and compares their eciency in terms of computation and storage requirements. From Table 1(a) we can conclude that both TPV AS and FD have the lowest storage requirement, while DT method requires the largest storage requirement. GB requires larger storage than the DT method if the objects have multiple major axes. Both GB and DT, require some additional temporary storage to perform their algorithms. Table 1(b) shows the computation cost to generate the shape signatures of the objects. We can conclude that TPV AS has the lowest computation cost and GB the highest. Both DT and FD methods have comparable computation costs due to the use of Delaunay triangulation algorithm. However, Table 1(c) shows that GB requires the lowest computation cost to compute the similarity between two shape signatures, while the other three methods have comparable computation costs.

11

7 Conclusion and Future Work This study compared four shape-based object retrieval techniques (FD, GB, DT, and TPV AS). The similarity retrieval accuracy of our method (TV PAS) was comparable to the other methods, while it had the lowest computation cost to generate the shape signatures of the objects. Moreover, it has low storage requirement, and a comparable computation cost to compute the similarity between two shape signatures. In addition, TPV AS requires no normalization of the objects, and is the only method that has direct support to RST query types. We also introduced a new shape description taxonomy. We intend to extend this work in several directions. First, we plan to investigate how to handle objects with curved parts, holes, and open curves or lines. Second, partial similarity match is not supported by all the systems, therefore we intend to identify better features to be used to for partial similarity match queries. Third, in our experiments we assume that each shape is relevant only to itself and to its ten variants. However, human perceptions on shape similarity are appreciably dierent. Therefore, we intend to incorporate human perception in the future evaluation of the eectiveness of the shape matching method. Fourth, we intend to test the methods for robustness to noise by performing queries on a database of noisy shapes. Finally, we want to extend this work to support three dimensional objects. Our preliminary investigations show that analogous to MBC features for 2D objects, we can extract features from 3-D objects by using their minimum bounding spheres.

8 Acknowledgment The authors thanks Geo Leach for providing us with the Java code for implementing Delaunay triangulation algorithm (available at http://goanna.cs. rmit.edu.au/ gl/research/comp geom/delaunay/delaunay.html) . Special thanks also to F. Mokhtarian, X. Ding, and S. Abbasi for providing us with the sh shapes data set (available at ftp://ftp.ee.surrey. ac.uk/pub/vision/misc/ sh contours.tar.Z). We would also like to thank J. Gary for kindly providing us with the tool shapes data set.

References [1] C. Chung, S. Lee, S. Chun, D. Kim, and J. Lee Similarity Search for Multidimensional Data Sequences To appear in the Proceedings of the 16th International Conference on Data Engineering (ICDE), San Diego, CA, USA, February 29 - March 3, 2000. [2] C. Yang, and T. Lozano-Perez Image Database Retrieval with Multiple-Instance Learning Techniques To appear in the Proceedings of the 16th International Conference on Data Engineering (ICDE), San Diego, CA, USA, February 29 - March 3, 2000. [3] E. Welzl Smallest Enclosing Disks (Balls and Ellipsoids) \New Results and new Trends in Computer Science", Lecture Notes in Computer Science, 555(359-370), 1991. [4] L. Guibas and J. Stol Primitives for the Manipulation of General Subdivisions and the Computation of Voronoi Diagrams ACM Transactions on Graphics, 4:74-123, April, 1986 [5] G.H. Granlund Fourier preprocessing for hand print character recognition IEEE Transactions on Computers, C-21, pp. 195-201, Feb. 1972 [6] C.W. Richard Jr. and H. Hemami Identi cation of three-dimensional objects using Fourier descriptors of the boundary curve IEEE Transactions on Systems, Man and Cybernetics, SMC-4, pp. 371-378, July 1974 [7] T. Wallace and P.A. Wintz Fourier Descriptors for Extraction of Shape Information Final Report of Research for the Period Nov. 1, 1975 - Oct. 31, 1976., Contract No. F 30602-75-C-0150 [8] E. Persoon and K.S. Fu Shape discrimination using Fourier descriptors IEEE Transactions on Systems, Man and Cybernetics, 7, pp. 170-179, 1977 [9] C.S. Lin and C.L. Hwang New forms of shape invariants from elliptic Fourier descriptors Pattern Recognition, 20(5), pp. 535-545, 1987 12

[10] C. Shahabi, M. Safar, and A. Hezhi Ecient Retrieval and Spatial Querying of 2D Objects Proceedings of IEEE International Conference on Multimedia Computing and Systems (ICMCS), June 7-11, 1999, Florence, Italy. pp.611-617. [11] C. Shahabi, M. Safar, A. Hezhi Multiple Index Structures for Ecient Retrieval of 2D Objects University of Southern California Technical Report 99-694 ftp://ftp.usc.edu/pub/csinfo/tech-reports/papers/99-694.ps.Z [12] B.M. Mehtre, M.S. Kankanhalli, W.F. Lee Shape Measures for Content Based Image Retrieval: A Comparison Information Processing and Management, Vol.33, No.3, pp. 319-337, 1997 [13] H. Kauppinen, T. Seppanen, M. Pietikainen An experimental comparison of autoregressive and Fourier-based descriptors in 2D shape classi cation IEEE Transaction on Pattern Analysis and Machine Intelligence, 17(2):201207, 1995 [14] L. Guojun An approach to image retrieval based on shape Journal of Information Science, 23 (2) 1997, pp. 119-127 [15] T.P. Wallace, and P.A. Wintz An Ecient Three-Dimensional Aircraft Recognition Algorithm Using Normalized Fourier Descriptors Computer Graphics and Image Processing, 13, 99-126, 1980 [16] B. Scassellatie, S. Alexopoulos, and M. Flickner Retrieving images by 2D shape: A comparison of computation methods with human perceptual judgments SPIE Conference on Storage and Retrieval for Image and Video Databases II, February 6-10, 1994, San Jose, CA, USA. SPIE Proceedings Vol. 2185, pp. 2-14 [17] Y. Tao, and W.I. Grosky Delaunay triangulation for image object indexing: a novel method for shape representation Proceedings of the Seventh SPIE Symposium on Storage and Retrieval for Image and Video Databases, San Jose, California, January 1999, pp.631-642. [18] Y. Tao, and W.I. Grosky Object-Based image retrieval using point feature maps Proceedings of the International Conference on Database Semantics (DS-8), Rotorua, New Zealand, January 1999, pp. 59-73. [19] G. Lu, and A. Sajjanhar Region-based shape representation and similarity measure suitable for content-based image retrieval Multimedia Systems 7, Springer-Verlag, pp. 165-174, 1999(2) [20] A. Sajjanhar, and G. Lu A Grid Based Shape Indexing and Retrieval Method Special issue of Australian Computer Journal on Multimedia Storage and Archiving Systems, Vol. 29, No.4, November 1997, pp. 131-140. [21] A. Sajjanhar, and G. Lu A Comparison of Techniques for Shape Retrieval International Conference on Computational Intelligence and Multimedia Applications, 9-11 Feb. 1998, Monash University, Gippsland Campus, pp.854-859. [22] A. Sajjanhar, G. Lu and J. Wright An experimental study of moment invariants and Fourier descriptors for shape based image retrieval Proceedings of the Second Australia Document Computing Symposium, April 5 1997, Melbourne, Australia, pp. 46-54. [23] A. Sajjanhar and G. Lu Indexing 2D non-occluded shape for similarityretrieval SPIE Conference on Applications of Digital Image Processing XX, Proceedings, Vol. 3164, 30 July -1 August 1997, San Diego, USA, pp.188-197. [24] R. Mehrotra, and J.E. Gary Image retrieval using color and shape Second Asian Conference on Computer Vision , 5-8 December, Singapore. Springer, Berlin, ppII-529-II-533, 1995 [25] R.C. Gonzalez, and P. Wintz Digital Image Processing 2nd edition Addison-Wesley, Reading, Mass.1987 [26] Pitas, Ioannis Digital Image Processing Algorithms Prentice Hall, Englewood Clis, N.J. 1993 [27] J. Gary, and R. Mehrotra Similar-Shape Retrieval In Shape Data Management IEEE, Computer September 1995. [28] F. Mokhtarian, S. Abbasi, and J. Kitter Ecient and Robust Retrieval by Shape Content through Curvature Scale Space Proceedings of International Workshop on Image Database and Multimedia Search, pp. 35-42, Amesterdam, Netherlands, 1996. [29] S. Berchtold, D. Keim, and H.P. Kriegel Using Extended Feature Objects for Partial Similarity retrieval VLDB,1997. [30] S. Ghandeharizadeh Stream-Based Versus Structured Video Objects: Issues, Solutions, and Challenges In S. Jajodia and V. Subrahmanian, eds, Multimedia DB Systems: Issues and Res. Direct., Springer-Verlag, 1995 13

[31] J.P. Eakins Retrieval of trade mark images by shape feature Proc. First International Conference on Electronic Library and Visual Information System Research, de Montfort University, 1994, pp. 101-109 [32] F. Korn, N. Sidiropoulos, C. Faloutsos, E. Siegel, and Z. Protopapas Fast Nearest Neighbor Search in Medical Image Databases Proc. 22nd VLDB Conference, pp.215-226, Mumbai, India, 1996. [33] C. Sul, K. Lee, and K. Wohn Virtual Stage: A Location-Based Karoke System IEEE Multimedia, 1998, pp. 42-52

14