SYMMETRY TO REPRESENTATION: The abstraction of object information from images Michael F. Kelly, Martin D. Levine TR-CIM-94-12 July 6,1995
Computer Vision and Robotics Laboratory Centre for Intelligent Machines McGill University Montreal, Quebec, Canada
Postal Address: 3480 University Street, Montreal, Quebec, Canada H3A 2A7 Telephone: (514) 398-6319 FAX: (514) 398-7348 Network Address:
[email protected]
SYMMETRY TO REPRESENTATION: The abstraction of object information from images Michael F. Kelly, Martin D. Levine
Abstract
This report is one in a series which deals with the topic of using annular symmetry operators to extract object information from images. The two reports which precede this one discuss symmetric enclosure [11] and sampling with annular operators [10]. In this report, we discuss some of the computational issues surrounding the extraction and interpretation of symmetry data using annular operators. Annular operators are applied to edge data that has been extracted from a gray-scale image. We make the distinction between two types of enclosing edge con gurations - termed limbs and blobs. It is shown theoretically that each type has a unique signature that can be identi ed by examining extracted symmetry points within the three-dimensional S-space de ned by the two spatial coordinates and one scale coordinate. Based on this, we have developed a computational method for detecting blob and limb parts in images. An important factor which in uences our approach is the need to extract speci c part representations. In the case of blobs, the desired part representation corresponds to a single point in the S-space; whereas for limbs, the representation consists of a chain of points in the S-space. The computational method for identifying these part representations relies upon performing a constrained search through symmetry point clusters within the S-space. Results using real image data are used to verify the theoretical predictions. Several examples demonstrating the extraction of parts from images of natural and man-made environments are then presented.
Acknowledgements
This research is partially supported by The Natural Sciences and Engineering Research Council and by the National Centers of Excellence Program through IRIS. MDL would like to also thank CIAR and PRECARN for their support.
1
Contents 1 INTRODUCTION
1.1 Background : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1.2 The Medial Axis : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
2 PARTS
2.1 Blob and Limb Parts : : : : : : : : 2.2 Symmetry Volumes : : : : : : : : : 2.2.1 Blobs : : : : : : : : : : : : : 2.2.2 Limbs : : : : : : : : : : : : 2.3 Discrete Sampling : : : : : : : : : : 2.4 Parts and Symmetry Point Clusters 2.5 Part Representations : : : : : : : : 2.6 Summary : : : : : : : : : : : : : :
: : : : : : : :
: : : : : : : :
: : : : : : : :
: : : : : : : :
: : : : : : : :
: : : : : : : :
: : : : : : : :
3 EXTRACTING PART REPRESENTATIONS
: : : : : : : :
3.1 Limb Orientation : : : : : : : : : : : : : : : : : : 3.1.1 Discrete Angular Pro les : : : : : : : : : : 3.1.2 Limb Enclosure: Practical Considerations 3.1.3 Interpreting the Angular Pro le : : : : : : 3.2 Sampling De nitions: A Review : : : : : : : : : : 3.2.1 Quantifying Enclosure : : : : : : : : : : : 3.3 Consistency Constraints : : : : : : : : : : : : : : 3.4 Blob Part Representations : : : : : : : : : : : : : 3.5 Limb Part Representations : : : : : : : : : : : : : 3.5.1 Cluster Simpli cation : : : : : : : : : : : : 3.5.2 Path-Based Representations : : : : : : : : 3.6 Computational Method : : : : : : : : : : : : : : : 3.6.1 Blobs : : : : : : : : : : : : : : : : : : : : : 3.6.2 Limbs : : : : : : : : : : : : : : : : : : : : 3.7 Summary : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
: : : : : : : : : : : : : : : : : : : : : : :
3 3 6
10 10 12 14 18 19 25 26 27
29 29 31 33 35 38 40 40 42 44 45 50 63 63 64 66
4 RESULTS
67
A Determining Limb Orientation Resolution B Incorporating Edge Gradient Direction
87 89
4.1 Symmetry Point Distributions : : : : : : : : : : : : : : : : : : : : : : : : : : 67 4.2 Cluster Simpli cation : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 75 4.3 Part Extraction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 78
2
References
92
3
1. INTRODUCTION Annular operators have been previously introduced [10] [11] as a means of extracting symmetry information from gray-scale images. The extraction process can be described in two stages. First contrast edges are identi ed within an image at multiple scales. Then the edges are grouped using annular operators also applied at multiple scales. Speci c edge con gurations exhibiting 'symmetric enclosure' are identi ed based upon the position and extent of edges falling within a given annulus. Points corresponding to the centers of annuli are used to represent two sorts of symmetrically enclosing structures called 'limbs' or 'blobs'. In this report, we will examine how the latter might be used - either as a means for identifying where objects are located within an image scene, or as a means of decomposing and representing shape characteristics of objects within a scene. Both of these results will depend upon the ability to associate particular symmetry points with a common object structure or part. The analytical and computational steps which lead to this sort of grouping process are the topic of this report. It will be shown that it is possible to extract usable representations for object structures in images from raw symmetry data. The report is organized as follows. In the remainder of this chapter we will review methods previously introduced [10] for extracting symmetry points using multi-scale annular operators. It will also be necessary to review the Medial Axis [1] as it will be referred to often during the remainder of the report. In the following chapter, the notion of parts will be introduced. The analysis will make a direct analogy between parts and symmetry point clusters which are de ned based on a set of consistency constraints. In the nal part of Chapter 2, we will examine what types of information are required when attempting to represent blob and limb parts. Chapter 3 will discuss some of the details which apply when attempting to extract part representations from real symmetry data. The discussion will focus on the consistency constraints. The nal chapter presents experimental results which validate the theory described in previous chapters, and demonstrates the performance of an automatic system which is capable of extracting part representations.
1.1 Background The extraction of symmetry information from images using annular shaped operators has been introduced elsewhere (the reader is referred to [10][11] for a more detailed description of what follows). The application of these operators leads to the identi cation of symmetrically enclosed points which distinguish the spatial positions of symmetries existing between edges in an image. In this section we will brie y review the main features of the extraction process - starting from a raw gray-scale image, and leading to a set of symmetry points extracted over a range of scales. The extraction process consists of two stages. The rst involves extracting edge features over a range of scales. These scales are related to those used in the second stage where multi-scale annular operators are used to detect symmetries among edge components. Both stages will be brie y reviewed in what follows.
4 For the rst stage, we have chosen to employ a set of oriented edge operators based on a formulation similar to that used by Canny [5]. Operators having an elongated spatial extent are constructed. The spatial region associated with each operator is characterized by a major and minor axis corresponding to the widest and narrowest region dimensions respectively. A two-dimensional Gaussian function is used as a smoothing kernel within each elongated region, and the appropriate scale constants are chosen corresponding to the regions major and minor axes. Pronounced contrast changes are located by applying a di erential operation along the minor axis of the operator which results in the familiar Gaussian rst-derivative structure of [5]. When detecting large-scale symmetric structures, it is important that the edges derived from small-scale structures be discarded so that symmetry point clusters do not become too complex. We employ two forms of ltering for this. In the rst, the amount of smoothing is governed by the size (scale constants) of Gaussians chosen and has the e ect of blurring contrast edges which occur at scales much smaller than that of the operator. In the second, other small scale edges are suppressed by a non- linearity which requires that all valid edges have full support along an operator's major axis. Details pertaining to the method used for edge extraction are discussed in [12]. The extraction of multi-scale edge features is shown in Figure 1.1.
Multi-Scale Edge Features
Image
Figure 1.1: The Extraction of Edge Features at Multiple Scales.
Elongated edge operators are applied at locations throughout the original image over a range of orientations. Six di erent scales are shown here, each employing a di erent sized edge operator. The result of the edge extraction process is a set of edge maps. Orientation and magnitude (contrast) values are recorded at points in the edge map where edges are marked.
The multi-scale annular sampling structure has been discussed at length elsewhere [10].
5 It consists of a set of N layers, each of which contains a set of annular operators, all of the same size. As a set of layers is traversed, the size of the annuli increases in order that larger scale edge structures can be detected. Multi-scale annular operators applied to a set of extracted edge maps are shown in Figure 1.2. For any particular annular operator, Raw Symmetry Data
Multi-Scale Edge Features
Different Sized Annular Operators are Applied to Each Set of Edge Data
Figure 1.2: Annular Operators Applied at Multiple Scales. For each
edge map a corresponding set of annular operators is employed to detect symmetric enclosures at a given scale. All operators applied at one scale have the same size, and are applied to the entire edge map. The result is a set of symmetry maps, each consisting of a set of points found to be symmetrically enclosed at a given scale.
symmetries are identi ed by looking for edge contours which fall within the spatial extent of the operator's annular band. Break angles are computed, and based on these, a measure of symmetric enclosure may then be computed [11] (see Section 3.2.1). Two di erent types of symmetrically enclosing structures have been considered [11]. Blob enclosures are characterized by a set of edge contours which largely surround the central annular region without preference to direction. In contrast, limb enclosure is characterized by a pair of enclosing contours having one or more distinct gaps. The direction of these gaps may be used to de ne a mean orientation for the associated limb point (see Figure 1.3). After applying a set of annular operators at multiple scales, we obtain a set of symmetrically enclosed points for each of the N scales chosen. At this stage it is not yet possible to distinguish between blob and limb enclosing structures with any degree of con dence. Hence, all enclosed points are treated as limbs and an appropriate orientation value is assigned to each point. Given a single gray scale image, this annular sampling method is applied to obtain a set of symmetrically enclosed points. But how can these points be used to gain information about objects within the image? To answer this question we will rst consider the Medial Axis which is one of the principle ways that symmetry information has been traditionally used to represent two-dimensional pre-segmented objects. We consider the sort of information that is made explicit by a Medial Axis to be useful for representing object structures.
6 Edge Contours
~ Blob Enclosure
~ Limb Enclosure
Figure 1.3: Examples of Blob and Limb Enclosure. Blob enclosure
is characterized by a contour con guration within the annulus which does not favour a particular direction. Such a con guration may be viewed as approximating a closed circular contour. For limb enclosure, there is a distinct direction associated with the contour con guration. A limb enclosure better approximates two line segments which separate an enclosed space. The direction of the limb enclosure (modulo radians) is given by the `mean orientation' of these line segments, represented by a faint arrow in the gure.
1.2 The Medial Axis There is a direct relationship between the symmetry points extracted using the annular sampling structure and the set of points that comprises the Medial Axis [1]. The Medial Axis (MA) and variants thereof have been used extensively in the past by computer vision researchers to represent some of the symmetries present in two-dimensional objects. While having some signi cant disadvantages for practical applications, the MA does provide an intuitively appealing result which seems to capture much of what we as humans consider to be the characteristic `shape' for many two- dimensional objects. Due to its representational power, and also to its established place in computer vision, it provides a good point of reference when describing the symmetry points extracted using our method. The Medial Axis Transform (MAT) was introduced by Blum as a way to generate axes from any closed two-dimensional contours [1]. For any simple closed curve it is always possible to identify an interior region which is bounded by the curve. The Medial Axis (MA) is de ned by the locus of circle centers which are at least bi-tangent to the contour while at the same time being contained entirely within the contour's interior. This can equivalently be re-stated in terms of a wave process (the grass- re analogy) initiated on the contour, and ending at points within the curve interior where two wavefronts meet. These meeting locations correspond to points along the MA. It is possible to view the MA as a space curve within a continuous three-dimensional space spanned by the two spatial parameters x and y, and a third scale parameter s which corresponds to the radius of the bi-tangent circles used to de ne the axial locus. This method for visualizing symmetry information will be particularly helpful when analysing the
7 symmetry point clusters generated using annular operators. It is possible to represent the annular sampling space in similar continuous terms, as shown in Figure 1.4. In this case, the scale parameter s corresponds to the size of the annulus used. Recall that a set of layers is employed in the annular sampling structure[10]. As shown in the gure, these represent a discrete scalar sampling of the continuous space. scale s
Continuous Envelope
Operator Size layers
y
x
Figure 1.4: Annular Scale Space. The space is spanned by two spatial
parameters x and y , and a scale parameter s which relates to the size of annular operators used. In our implementation, a discrete number of layers is used. This may be viewed as slices taken through the space at a discrete number of positions along the scale axis. The xed operator size used within each layer must be chosen to fall somewhere within a speci c continuous range[10].
Each annulus in a given layer is de ned by an inner and outer circular boundary, having radii r and r , respectively. To demonstrate the connection between the MA and points extracted using annular symmetry operators, we consider the case in which the annular width (r ? r ) is allowed to shrink to zero and where sampling is performed within a continuous domain. This permits us to specify the position and size of the annuli as real numbers (x; y; s 2 < r ? r if r r ravg (r) = > r ? r if ravg r r (2:1) :0 otherwise Here, ravg = r1 r2 represents the average detectable radius value for an annulus. The expression in equation (2.1) may be used to predict the detection zone for a single annular operator. However, in our current discussion, we are interested in a slightly di erent quantity. Given a circle of radius r, we would like to predict which annular operators are capable of detecting it. The extent of operators responding to a circle will be shown to form a volume within the three-dimensional S -space, which characterizes 2D spatial and 1D scale variations. To begin we will examine only the spatial extent of operators responding to a circular contour. In Figure 2.6 the position of the annular operator is xed and the position of the circular contour is allowed to vary. In this case, we x the position of the circular contour, and then consider the range of positions of annular operators which are capable of detecting the circle (see Figure 2.7). We note that the associated region of activity is again circular. However, it is concentric with the contour rather than the operator as in Figure 2.6. This region will henceforth be referred to as the active region in order to distinguish it from the detection zone. By inspection, we note that although the active region and the detection zone are de ned di erently, they are both of the same size as de ned by the radius . For any circular contour, if we choose a set of operators at a particular scale s, and these operators are capable of detecting the contour, we know that there will be a spatial distribution of activity within the (continuously sampled) S -space which falls within a circular region concentric with the contour. We now consider how the size of this spatial distribution changes as we move along the scale dimension s. Recalling that the size of an annulus can be characterized by its average radius value s = ravg , it is now possible to express the radius of the active region in terms of the parameters r and s as follows: 8 >< r ? r if r r s (s; r) = > r ? r if s r r (2:2) :0 otherwise 1
1
2
2
+ 2
1
2
1
2
16
(a)
(b)
(c)
Figure 2.7: Range of Detecting Annuli for a Circle. A circle (dotted)
is detectable if it falls entirely within the annular region (dark shaded). Three di erent annular operator positions are shown for a given circular contour in a, b, and c. It is evident that the range of positions for detecting annuli (centers represented by a dot) will fall within a circular region that is concentric with the circular contour (light shaded).
Since the annular radii r and r may be expressed in terms of the annular width parameter and the scale s, a direct substitution leads to the following expression for (s; r) in terms of the parameter : 8 r ? s if r r s >< (s; r) = > s ? r if s r r (2:3) :0 otherwise Since the value of is independent of scale, it is possible to predict modi cations in active region sizes as the scale parameter s is varied . For a xed value of r, a plot of the active region radius as a function of scale s is shown in Figure 2.8. As might be expected, the two lines which de ne the active region radius intersect when s = r, and it is at this point that the value of (s; r) reaches its maximum. Substituting into equation(2.2) for r = s yields the following expression for the maximum active region radius max: max = r ?2 r = (2:4) By equating each of the nonzero terms in equation (2.3) to zero, we can determine the s axis intercepts to be s = r + in the rst case when r = r , and s = r ? in the second case when r = r . Hence, the lines which de ne the boundaries of the active region always intersect at right angles (see Figure 2.8). From the above analysis, it is now possible to predict the extent of operator response to a single circular contour of radius r. For a given scale s, we know that the active regions are bounded by circles. With a continuous change in scale, the size of these circles, as speci ed by the active region radius , increases linearly from zero, reaches a maximum, and then decreases linearly back to zero (see Figure 2.8). Therefore the surface which limits the extent of annular operators responding to a single circular contour in the S -space may be described by a pair of right circular conic surfaces connected at their bases. The distribution of operator responses corresponds to the volume contained by this surface, which we will refer to as a symmetry volume (see Figure 2.9). All annuli which are capable of detecting a 1
2
2 +1
2 1+
1
2
2
1
1
2
Recalling that s = r1 +2 r2 and r2 = r1.
17
β ψ =
r2 - r1 2
β max
s=r- ψ
s=r
s=r+ψ
s
Figure 2.8: Active Region Radius as a Function of Operator Scale s. A level curve for the function in Equation(2.3) for a xed value of r. Since
max = , we note that the the two lines
meet at right angles.
s
x
y
Figure 2.9: Symmetry Volume for a Circular Contour. The symme-
try volume is de ned by two right circular cones joined at their bases. The associated surface represents a boundary that can be later used to de ne the extent of point clusters when the S -space is sampled discretely.
18 circle of radius r will lie somewhere within such a symmetry volume. Since the practically realizable discrete sampling case may be viewed in terms of subsampling the continuous S -space, it is clear that no symmetrically enclosed point will reside outside of the surface bounding the symmetry volume. This fact will be used to predict the extent of symmetry point clusters in the discretely sampled space. By direct analogy, it is also possible to predict how detecting operators will be distributed for non-circular contours. For example, by distorting a circular contour by a small amount, we would expect a similar distortion to be re ected in the associated active region (see Figure 2.10(a)). For blob enclosure, where the distortion is small enough that the contour remains mostly within the annular region, we will use the volume in Figure 2.9 as an approximation to the actual symmetry volume. However, a blob which has been deformed to such a degree that there is a pronounced elongation in a single direction provides an example of limb enclosure. This is characterized by a correspondingly elongated active region (see Figure 2.10(b)). In the next section we will consider the symmetry volumes associated with such limb parts. contour
active region
(a)
(b)
Figure 2.10: Symmetry Cross Sections for Deformed Circles. (a)
A small distortion of the boundary yields a non-circular active region. (b) Larger scale distortions may cause an elongation of the active region. When the elongation is pronounced, this becomes an example of a limb enclosing structure.
2.2.2 Limbs
As mentioned previously (see Section 2.1), a limb part is identi ed by a set of symmetrically enclosed limb points. Such con gurations could arise, for example, from two parallel or diverging line segments. By allowing the lines to become curved, other possible con gurations can occur, as shown in Figure 2.11. These are all examples of the idealized limb parts previously introduced (see Figure 2.3). For each, it is possible to describe the shape of the associated symmetry volume by analogy with the results found for circular contours. Consider rst a part which consists of two straight parallel contours. An annular operator which is located so as to detect limb symmetry may be positioned at a range of locations, as shown by a, b, and c in Figure 2.12. The activity region in this case takes the form of a band. Using the results found for the circular contour, the width of this band may be calculated
19
(a)
(b)
(d)
(e)
(c)
Figure 2.11: Limb Con gurations. (a) Two parallel straight line seg-
ments. (b) Two diverging line segments. (c) Two diverging curve segments. (d) Bending applied to case (a). (e) Bending applied to cases (b) or (c).
based upon the distance w separating the two line segments. The band width 2 (s; r) is computed directly from equation (2.3) based on the substitution r = w . For variations in only the scale parameter s, we would expect to nd a behavior similar to that obtained for circular contours so that the band width in Figure 2.12 varies in proportion to the scale parameter s resulting in a triangular function (see Figure 2.8). The symmetry volume for this contour con guration may be visualized as the union of a set of such bands located continuously along the s axis. Alternately, this volume may also be represented as the volume swept out by a double-cone symmetry volume (see Figure 2.9), translated along the symmetry curve, as shown in Figure 2.13. It is not dicult to extend this analysis to some of the other contour con gurations shown in Figure 2.12. In the case of diverging lines, the blob symmetry volume used to generate the limb volume will have a size that depends upon the separation between the contours (since the maximum active region size max depends upon r in the case of circular contours). This size variation will result in the limb symmetry volume having larger cross sections as the operator scale increases. For curved contours, we would expect that correspondingly curved symmetry volumes should result. An example of a more general limb symmetry volume is shown in Figure 2.14. In the next section we will use our knowledge of symmetry volumes to predict how symmetry points are distributed within the discretely sampled S -space. 2
2.3 Discrete Sampling To practically implement an annular sampling scheme requires that the S -space be discretely sampled along both spatial and scalar dimensions. Therefore, we next consider a set of sampling layers which slice the S -space parallel to the x ? y plane. Within each layer, operators are located on a sampling grid whose spacing changes depending upon the scale s. The intersection of any given sampling layer with a symmetry volume will, in general, result
20 a
b
c contour
w
y x
Figure 2.12: Parallel Limb Example. (a) Two parallel line segments
separated by a distance w represent the simplest instance of a limb part. Three examples of annular operators which detect enclosing structure are given (labelled a,b, and c). Note that there is a range of positions in the y direction for which enclosure is detected. This corresponds to an activity region having the form of a band (region between dashed lines).
scale
Limb Symmetry Volume Blob Symmetry Volume
Figure 2.13: Symmetry Volume for Straight Parallel Contours. For the contour shown in Figure 2.12, we consider the associated symmetry volume. We note that the active region band width varies with scale s resulting in a behavior like that shown in Figure 2.8. Since each active region band corresponds to a planar slice taken through the symmetry volume parallel to the x ? y plane, we would expect the symmetry volume to widen linearly with scale up to some maximum value, and then narrow linearly with scale. The associated volume may be represented by the region swept out by a blob symmetry volume which is centered on and translated along the contour's symmetry curve. The volume has the form of a square prism with rounded ends (note that the contour itself is not shown in the gure).
21 scale
contour
y y x
x
Figure 2.14: Limb Symmetry Volume for a Contour. As the symmetry volume is traversed in the positive y direction there is an increase in scale for operators used to detect symmetry, and this results in a broadening of the symmetry volume.
in a bounded region which constrains the spatial position of operators detecting symmetry within that layer. By knowing how symmetry points are distributed within the discretely sampled space, it will be possible to devise computational methods for identifying blob and limb parts.
Blobs The relationship between symmetry point clusters in di erent layers for the case of `pure' blob symmetry (as represented by a circular contour) is easily understood. When the symmetry volume of Figure 2.9 is sliced parallel to the x?y plane, we recall that the result is a circular intersection. All such intersections will be concentric, and thus a set of these taken from several adjacent layers will appear as a concentric circles (see Figure 2.15). The existence of this `stack' of circular envelopes is a direct result of the inter-layer redundancy which was discussed at length in [10]. By appropriately selecting the amount of inter-layer redundancy, it is possible to ensure that some minimum number of layers (greater than one) always intersects the circle's symmetry volume. For a given blob symmetry volume, we have determined that there will be a stack of concentric circular regions which de ne the distribution of symmetry points within a set of adjacent layers. This stack represents a distinctive signature that may be associated with a given blob part. For example, assuming that the inter-layer redundancy is great enough, there will be distributions of symmetry points within at least two adjacent layers that will be approximately circular and concentric. This signature will provide an important means for distinguishing blob parts when we consider the extraction of part representations (see Chapter 3).
22 s
c1 c3 c2
c1 l4 l3 l2 l1
x
c2 c3
(a)
y (b)
Figure 2.15: Layer Intersections for a Blob Symmetry Volume. (a)
A blob volume within the continuous scale-space is sliced by four sampling layers, l1 to l4 , resulting in three circular sections, c1, c2, and c3. These circles constrain the extent of symmetry point distributions when the space is discretely sampled. (b) Projecting the three sections down into the x ? y plane results in a set of concentric circles having, in general, di erent radii.
Limbs An example of a discrete layer sampling in scale is shown in Figures 2.16. Figure 2.17 demonstrates a typical `pointed' intersection region for a single layer. We would expect that all operators responding to the contour structure for the scale (layer) shown should be located within the region shown. When a sampling layer intersects a limb symmetry volume, the result is generally a closed region bounded on four sides by curves (or line segments) meeting at vertices (see Figure 2.18). Since there is wide variability in the exact shape of these regions, we will henceforth refer to these as pointed quadrilateral regionsy When the annular scale-space is discretely sampled within a given layer, the distribution of limb symmetry points will, in general, be constrained to fall within the pointed quadrilateral envelopes that correspond to the symmetry volume/layer intersections. As in the case of blobs, these envelopes represent another form of signature that may be used in the process of identifying limb parts from raw symmetry data. Having some idea of how symmetry points are distributed within the discretely sampled S -space, it is now possible to devise a strategy for grouping symmetry points so that members of a given group all relate to one common part. The rules used in this grouping process which lead to the identi cation of symmetry point clusters is described in the following section. In the case of parallel contours, we recall that the active region takes the form of a band which (theoretically) need not be bounded in length. y Here the term quadrilateral is not used in the strict geometrical sense. Instead of a polygon having four edges, it will be used to refer (in the literal sense) to any bounded `four-sided' region where each side is represented by a distinct curve.
23
s
layers l 4 l3 l2
y
l1
x
Figure 2.16: Layer Intersections for a Limb Symmetry Volume.
The symmetry volume is shown shaded. Four layers, l1 to l4 slice through the scale-space and some intersect the symmetry volume.
Symmetry Volume Intersection Boundary
Layer
Layer
Figure 2.17: Intersection of a Symmetry Volume with One Layer. A slice taken through a limb symmetry volume results in a `pointed' intersection region. The boundary of this region limits the extent of symmetrically enclosed points detected by annular operators positioned on a discrete sampling grid.
24
(a)
(b)
(c)
(d)
(e)
(f)
Figure 2.18: Limb Intersection Regions. Recalling some of the limb
con gurations presented in Figure 2.11 (speci cally (b), (e) and (c)) we consider three contour con gurations represented here by dark curves in (a), (c), and (e). The range of annular operators which detect enclosure can be determined by construction (see dashed lines in rst column). The resulting active regions are shown by lled regions in the second column.
25
2.4 Parts and Symmetry Point Clusters We seek a means of relating subsets of extracted symmetry points to structural entities in the image. The concept of a part has already been introduced (see Section 2.1), although a formal de nition of part has yet to be given. In this section we will use a set of consistency constraints that will allow symmetry points to be grouped into clusters. In making a direct analogy between symmetry point clusters and parts, it will be possible to formalize our de nition of part in terms of the set of rules that are used to de ne symmetry point clusters. We are concerned with image structures that correspond to physical objects in the world. A distinction will be made between the image structure (which we will term object) and the physical structure (which we will term physical object). Certain properties may be associated with physical objects, and these same properties can be extended to our analysis of objects within images. For example, the fact that most physical objects exhibit some form of symmetry has been one of the principal motivations behind our using annular symmetry operators as a means for locating objects within images. Another important property associated with physical objects is material continuity. For a given physical object composed of a particular material (or set of materials), it is possible to consider points located within the volume of the object. A physical object exhibiting material continuity is demonstrated by the fact that for any two points located within the object, it is possible to nd a path which joins the points such that the path exists entirely within the object material. This path may be represented by a continuous space curve that is composed exclusively of points falling within the volume of the object. Assuming that we are dealing with only visible physical objects, it seems reasonable that we extend this notion of material continuity for physical objects to the characterization of object parts within an image. A set of extracted symmetry points will be partitioned into a set of clusters, where each cluster corresponds to either a blob or limb part in the image. In order to identify which symmetry points belong to a given cluster, we require that all points be mutually consistent. This consistency will be based upon a set of continuity constraints, some of which relate directly to the notion of material continuity. All symmetry points within a single cluster must satisfy the following requirements: 1. Enclosure Type Continuity 2. Spatial Continuity 3. Scalar Continuity 4. Orientation Continuity (for limbs) A set of consistency constraints follows directly from these requirements. The rst category requires that all points within a cluster be of the same type (either limb or blob exclusively). The remaining three forms of continuity can all be expressed in terms of adjacencies. Thus, for a given point to be included in a cluster, it must be adjacent to at least one other point in that cluster. This adjacency is de ned in terms of local spatial and scalar neighborhoods within the discretely sampled S -space. In the case of limbs, adjacency is also formulated in terms of neighborhoods within a discretely sampled orientation space. The details used to de ne these neighborhoods will be presented in a later section (see Section 3.5).
26 Based on the above discussion, we may summarize the rules used to verify consistency within a symmetry point cluster as follows: All points must exhibit the same type of enclosure.
Each point must be adjacent to at least one other point in the cluster. This adjacency
is de ned within the discretely sampled spatial and scalar domains, and in the case of limbs also within a discretely sampled orientation domain. This set of rules implicitly de nes blob and limb parts. A part corresponds to that contour structure which gives rise to a cluster of symmetry points that satis es the above consistency constraints. At this point we could directly apply the consistency rules to a set of extracted symmetry points in order to identify the parts. However, the results would only be partially useful since we would still require a means for representing the set of identi ed parts. In the following section we will consider the kinds of representations that are likely to be most useful for describing blob and limb parts.
2.5 Part Representations As introduced previously (see Section 1.2), the Medial Axis (MA) [1] is well established as a means for representing the regional structure of two-dimensional shapes. Due to its evident descriptive power in providing region-based representations, we have chosen to devise a part representation based upon a structure similar to the MA. In the previous section we introduced a set of consistency rules which could be used to identify symmetry point clusters associated with a given part. In an earlier section we also examined how the symmetry points within a given cluster are distributed within the three- dimensional S -space spanned by x; y and s. Recalling the idealized blob and limb parts that were introduced previously (see Section 2.1), in this section we will begin to consider how symmetry point clusters can be manipulated into a form which provides a representation similar to that produced by the MA. For any circular contour, the MA consists of a point (x; y) and a scale value corresponding to the circle's radius. In the S -space, this corresponds to a single point (x; y; s). We have shown that by employing annular operators, the result is actually a cluster of symmetry points within the S -space whose extent is limited by the surface of a symmetry volume similar to the one shown in Figure 2.9. Thus, in order to obtain a single point (MAlike) representation for an idealized blob part, we need to devise a means of collapsing the symmetry point cluster down to a single representative point. Intuitively, it makes sense that this point should lie somewhere towards the center of the cluster. The computational method used for locating this point from a set of raw symmetry data will be discussed in a later section (see Section 3.4). In the case of idealized limb parts, we again consider the corresponding MA representation. For the kinds of limb parts discussed thus far (see Figure 2.11), the MA will consist of a single continuous curve with a corresponding radius function de ned at all points along the curve. This corresponds to the space curve in the S -space which has previously been termed a symmetry curve (see Figure 2.4). We recall that since the symmetry volume may be viewed as a broadening about the symmetry curve, the latter lies somewhere within the
27 symmetry volume associated with a given limb part (see Figure 2.5). For limb parts, we seek to obtain an axial representation which approximates the symmetry curve. However, for our purposes, it will not be necessary that the symmetry curve be approximated with high precision. We consider a set of curves which have approximately the same locus as the symmetry curve in the S -space (see Figure 2.19). The central axis associated with a limb s Symmetry Volume approximating curves
central axis approximating curves
Figure 2.19: Axes Through a Limb Symmetry Volume. The central
axis of a symmetry volume is de ned by the locus of points which correspond to the geometrical centers of each pointed quadrilateral section found when slicing the S -space parallel to the x?y plane. We assume that the central axis provides a good approximation of the actual symmetry curve associated with a given symmetry volume. We therefore assume that the set of curves which have loci that fall close to the central axis provide adequate approximations to the symmetry curve.
symmetry volume is de ned by the centers of the pointed quadrilateral regions that result when the volume is sliced parallel to the x ? y plane. We will assume that the set of curves which lies close to this central axis will provide adequate approximations to the symmetry curve. In extracting a limb part representation from a symmetry point cluster, it is evident that we will require a means of thinning the cluster to produce a curve representation in the discretely sampled S -space. A curve in the discretely sampled space will be represented by a path or chain of points, where the adjacency used to de ne the chain will be based upon the consistency rules presented above. The computational method used for extracting a representation of limb parts will be presented in the next chapter.
2.6 Summary We have considered how the symmetry points corresponding to annular operator centers are distributed within S -space. The notion of blob and limb parts was introduced in order to make the distinction between two di erent sorts of enclosing contour geometries. By applying consistency constraints to sets of extracted symmetry points, it is possible to identify
28 symmetry point clusters. Blob and limb parts are de ned in terms of these symmetry point clusters. For each part in the image there is a corresponding symmetry point cluster which is identi ed based on the consistency constraints. Finally, it was possible to roughly outline how useful representations for blob and limb parts might be abstracted from symmetry point clusters. In the next chapter, we will explore the problem of extracting part representations in more detail so that a practical implementation may be devised.
29
3. EXTRACTING PART REPRESENTATIONS After applying annular symmetry operators, the result is a set of symmetry points located within a discretely sampled S -space. From the previous chapter, it is clear what kind of information will be needed for representing blob and limb parts. For blobs, we require that a single point be identi ed within the S -space so that the spatial position and scale for a given part may be represented. For limbs, we require that parts be represented by a set of points corresponding to paths in the S -space. In this chapter, we outline the details used in processing a set of raw symmetry data to arrive at useful part representations. We begin by discussing how symmetry points are identi ed and assigned an orientation value. After a brief review of some de nitions presented in our previous work on annular sampling [10], it will examine how part representations may be extracted from symmetry data. This will require that the continuity constraints introduced in the previous chapter (see Section 2.4) be re-examined in order to see how they can be applied to the discretely sampled symmetry data. Finally, the computational methods used to sequentially extract blob and limb parts based upon a physical support measure will be presented. Until now our main emphasis has been placed on a three-dimensional scale-space spanned by two spatial and one scaler parameter. For limb parts, it will be necessary to consider one additional parameter that speci es the orientation of symmetrically enclosed limb points. In the next section we will de ne limb orientation in terms of the information obtained through the application of annular operators. It will then be possible to examine the computational method used to extract limb orientation values from image data.
3.1 Limb Orientation The Extended S -space Limb symmetry points are characterized by an orientation which corresponds to the direction of openings at one or both ends of the angular pro le detected by an annular operator. Until now we have ignored this additional parameter which will be helpful in the analysis of symmetry point clusters. To specify a limb point uniquely, we require that four parameters be used - namely, (x; y; s; ). This four-dimensional space provides the basis for rules that permit the systematic grouping of symmetry points. Henceforth we will refer to this four-dimensional space as the extended S -space, in contrast to the previously de ned S -space which represents a three-dimensional projection along the axis. To introduce the concept of limb orientation, we rst consider the following simple example. The orientation of a symmetrically enclosed limb point is determined by examining the annular angular pro le (see [11] for an introduction to angular or `scan' pro les). In general, we will assume that a con guration of edges exhibiting limb enclosure will have at least one (usually two) predominant break(s) in its angular pro le. The direction of these breaks relative to the annular center may be assigned a value measured with respect to some xed global directional reference (see Figure 3.1).
30
contour
(a)
φ a1
θ1
θ2
a2 (b)
(c)
Figure 3.1: Orientation of Limb Enclosure. (a) A limb contour con guration detected by an annular operator. Dashed lines delineate the parts of the annulus that exclude the contour. (b) Two breaks are identi ed in regions where contour points are not found (light shaded). These are represented by the angular intervals 1 and 2 measured relative to the annular center. The direction of the bisector for each break interval de nes the break angle direction (a1 and a2 ). (c) The average of the two break angle directions can be used to represent the limb enclosure orientation, and is speci ed as an angle modulo radians.
31 This example demonstrates what information should ideally be conveyed by a limb orientation value. However, due to complicating factors found in practical settings, it will be necessary to initially consider an extraction method whereby multiple orientations are recorded at a given symmetry point. In later processing stages we will employ additional structural information to produce a single orientation value for each symmetry point in a limb cluster (see Section 3.5.1). This will allow symmetry points to be speci ed in terms of the extended S -space. We shall see that by applying continuity constraints within the extended S -space, it is possible to identify limb symmetry point clusters and to extract representations for them. For the moment, however, we need to focus on the method used to identify the multiple limb orientations associated with a particular enclosing structure. Before proceeding with this, we introduce the discrete angular pro les which are used as the basis for subsequent computations.
3.1.1 Discrete Angular Pro les
In all previous sections, it was assumed that the angular pro le was expressed as a function of a continuous angular position variable. For practical reasons, however, it is necessary that the angular position of edge fragments within a given annulus be represented in a discretized form. We have chosen to use a simple strategy whereby each annular region is partitioned into a set of N equal sized annular sectors (see Figure 3.2). To construct a discrete angular
Figure 3.2: Annular Sectors. The annulus is divided into N equal sized
sectors (here N = 20). Operations are rst performed separately on each set of data falling within a sector.
pro le requires that sampling be performed within each of the annular sectors so that any compatible edge fragments may be detected. For a given edge fragment located within a particular sector, this compatibility depends upon the orientationy of the edge fragment with respect to the mean annular tangent direction for that sector. Referring to Figure 3.3, we consider the mean annular tangent direction within a given sector and the orientation for an edge point p falling within that sector. As the di erence between the two orientation The choice of a particular value for N depends on many factors not yet discussed (see Appendix A) The orientation assigned to a given edge feature may be represented modulo radians, or it may be expressed as a function of the local edge gradient direction. Depending on which approach is adopted, it is possible to select di erent types of enclosing structures (see Appendix B). y
32 values decreases, an edge is considered to be more compatible with that particular annular sector. An edge is compatible with an annular sector if its local orientation does not di er from the mean sector orientation by more than a set amount. This amount will be termed the maximum angular deviation Dmax. Each annular sector is treated independently. If a single compatible edge is found within a given sector, this is recorded in the angular pro le as the presence of an edge at the corresponding angle for that sector (see Figure 3.4). It is possible, therefore, to detect edges for a maximum of N angular positions within the annulus. This e ectively limits the angular pro le resolution to increments of N radians. The value of Dmax is chosen such that it is slightly larger than this increment . 2
λ p
υ
contour
Figure 3.3: Edge Orientation Compatibility. A contour passes through
an annular region where N = 20. The detail for one sector is shown with the contour replaced by a dashed curve. The mean tangent direction within the annular sector is represented by the orientation . We consider an edge point p within the sector where the contour has an orientation . The compatibility between the edge orientation and the annular sector orientation increases as the two orientations become more similar.
A discrete angular pro le is constructed by recording the presence or absence of compatible edges for each annular sector (see Figure 3.4(b)). For the remainder of this report, since continuous angular pro les will no longer be considered, the term annular pro le will be used to refer to discrete angular pro les. Until now, the only quantity we have associated with edge points is their orientation, which has been represented by the orientation of a small line segment. We note that other quantities such as the edge gradient magnitude and direction also provide useful information. The gradient magnitude is carried into later processing stages where it will be used in the computation of the perceptual signi cance of parts (to be presented in a future report). This will lead to several di erent possibilities for both blob and limb enclosing con gurations. Speci cally, we can consider three basic types of parts - those which are darker than their background, those which are lighter than their background, and those for which there is a A typical sampling with N = 20 annular sectors results in 18 increments. In practice, by choosing = 18 we nd that signi cant enclosing structures may be missed. We therefore employ a slightly larger value of Dmax = 26 . An examination of the e ects of varying Dmax is given in Section 4.1.
Dmax
33 contrast reversal as the part boundary is traversed. A more detailed description of these con gurations is presented in Appendix B. 7
6
5
4 3
8
2
9 10
1
11
20 19
12 13
18 14
15
16 17
(a) φ compatible edge
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 * * * * * * *
(b)
Figure 3.4: Discrete Angular Pro le. (a) From the previous gure,
sectors containing compatible edges are shaded dark. The angular position is recorded by an integer corresponding to the sequential ordering of the sectors. (b) The discrete angular pro le is a binary function which indicates the presence or absence of a compatible edge at each angular position ^ 2 f1; 2; :::20g. Compatible edges are indicated by marks. Break angles i correspond to the size of empty intervals.
3.1.2 Limb Enclosure: Practical Considerations
It is possible to identify enclosing edge con gurations using the angular pro le. If we could be assured that limb edge con gurations were always as simple as the example shown in Figure 3.1, then it would be a straightforward matter to directly compute the orientation based on break angles as previously discussed. Unfortunately, the con gurations which arise in real imaging settings are rarely so simple. There are two signi cant sources of complication. The rst is due to additional edge fragments (see Figure 3.5), and the second is due to contour gaps (see Figure 3.6). In both cases, it is dicult to assign a single orientation value since, starting with an angular pro le, it is often not clear how to select the appropriate break angles. There are also examples of contour structures which naturally give rise to several di erent limb orientations at the same point in S -space (see Figure 3.5). For these reasons, it is necessary to select a method whereby several orientation values may be recorded at a given symmetry point. These values will be carried into the next processing stage. There it will be possible to employ the structural information present within symmetry point clusters to assign a single orientation value to each cluster point. In the following section, we discuss the operations used to extract a set of multiple orientation values from an angular pro le.
A common problem encountered when applying local edge detection strategies.
34 A ν χ
D
C B
(a)
φ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 compatible edge * * * * * * * * * * * *
(b)
Figure 3.5: Complicated Limb Enclosures (i). (a) Multiple edge fragments falling within an annulus may correspond to several di erent object structures. Here, the contours A and B are the predominant global structures. Yet based solely on the angular pro le (b), this is not directly evident. To identify a single limb orientation value would require that a particular pair of break angles be selected from the angular pro le. However, it is not clear how such a selection should be made. Rather than attempt to nd a single orientation value, we will instead carry all plausible orientation values (for example and in this case) forward for use in subsequent processing stages.
(a) φ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 compatible edge * * * * * * * * * * * * * * *
(b)
Figure 3.6: Complicated Limb Enclosures (ii). (a) Fragmented con-
tours make it dicult to accurately estimate a single orientation value, especially in the presence of added edge structure. From the related angular pro le (b) it is unclear which break angles should be used to estimate a limb orientation value. In fragmented situations such as this, we see that it may be necessary to again carry several orientation possibilities forward into the next processing stage.
35
3.1.3 Interpreting the Angular Pro le
When employing annular operators, all computations are based upon the discrete angular pro le discussed above (see Section 3.1.1). In this section we discuss how the angular pro le data is interpreted in order to eventually assign a set of limb orientation values to a given enclosure. To begin, we consider one important constraint that will allow us to formulate a simple method for detecting oriented limb enclosure. Consider the set of break angles [11] f ; ; :::ng found for a given angular pro le (see Figure 3.7). These represent an ordered sequence of decreasing values starting with the largest, . For an angular pro le to exhibit symmetrical enclosure, we require that the largest break angle be less than some value . 1
2
1
1
φ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 compatible edge * * * * * * * * θ3
θ1
θ2
Figure 3.7: Break Angles. Intervals corresponding to breaks in the angu-
lar pro le are assigned values i . These are ordered such that 1 represents the largest break.
The rationale for selecting a particular value for is as follows. We consider limb enclosures formed by two intersecting line segments of equal length, where the angle of intersection is . In Figure 3.8(a) we see a progression of limbs where the value of increases from left to right. For small values of it seems appropriate to label the structure as a limb based on our intuitive notion of what characterizes a limb. We can describe the dimensions of the enclosed region in terms of a nominal length and width - where the length is signi cantly greater that the width. As the value of increases, we observe that it becomes harder to identify an enclosed region as the contour approaches a single line segment (when = ). For large values of , any distinction between length and width also becomes less obvious since the enclosed region is less well de ned. In selecting an appropriate value for , we seek a maximum value of above which it no longer makes sense to apply the label of limb. We note that in the case where = , there is no way to di erentiate between length or width since the two segments de ne a perfectly symmetrical equilateral triangle (see Figure 3.8(b)). Therefore it seems reasonable to impose the constraint that in all cases
< . Geometrically, the largest possible break angle associated with this value of is given by = (see Figure 3.8(b)). Thus, we select the corresponding break angle threshold to be = . We have been able to use simple principles to choose precise values for the thresholds and Dmax . In a later section, we shall demonstrate how the set of symmetry points is 3
3
1
4 3 4 3
It is assumed that the largest break angles are associated with limb parts.
36 γ (a)
θ1 = 4π 3 π 3
(b)
Figure 3.8: Angular Constraints. (a) Limb parts are represented by intersecting pairs of line segments. As the angle between the segments increases, there is a point beyond which the structure ceases to manifest the qualities of a limb. (b) For the case where = 3 , since the contour corresponds to the sides of an equilateral triangle, the inherent symmetry makes it impossible to distinguish between width or length. The largest possible annular break angle for this contour is 1 = 43 , corresponding to a situation where the contour is tangent to the outer annular circle. Hence, for all enclosing structures, we require that 1 < 43 = . e ected by variations in both of these parameters (see Section 4.1). While we would expect that the number of extracted symmetry points should increase - particularly in the case of increasing Dmax - we will demonstrate that the overall part extraction process is not signi cantly e ected by small variations in either of the parameter values. In the forgoing analysis, we did not explicitly consider instances where limb structures had curved contours. Establishing a value for using limbs with straight sides made some intuitive sense, since these represent a kind of intermediate stage between inward and outward curving contours. We will demonstrate in a later section (see Section 4.1) that the threshold selected here will not lead to any noticeable performance degradation when curved limb contours are examined. Based upon our selection of the threshold , it is now possible to examine the method used in identifying oriented limb enclosures.
Determining Orientation We require a means for extracting multiple orientation values for a given enclosure. It was demonstrated (see Section 3.1.2) that, due to contour breaks, we cannot always rely on the set of break angles as a means of determining limb orientation values. In this section we propose an alternate approach for computing limb orientation values. Fundamentally, a limb enclosure can be represented by two separate sets of compatible edge data found within a given annular region. From our discussion above, it is evident that there are certain restrictions that de ne how these sets can be con gured. For example, we choose to neglect instances where two sets of data arise from a limb having > . A range of valid con gurations for pairs of limb data are shown in Figure 3.9. We do not attempt to 3
37
(a)
(b)
(c)
Figure 3.9: Valid Range for Limb Contours. Three limb con gurations
composed of line segments having = 3 (a), = 6 (b), and = 0 (c). A range of compatible edge values within each annulus is shown shaded dark. For each con guration, we note that there will always be a pair of such compatible regions. The limb orientation (arrow) in each case is the same.
detect each of these situations independently. Instead we detect all valid limb con gurations for a given orientation. Q1
Q2
φ Q1
Q2 (a)
(b)
Figure 3.10: Limb Detection Strategy. (a) The three contour con g-
urations from Figure 3.9 are shown superimposed within a single annular region (shaded). In order to detect any one of these, we require that edge compatibility tests be performed only within the dark shaded regions Q1 and Q2 . (b) A complimentary sampling where Q1 and Q2 have been rotated by radians so as to detect the same sorts of contour con gurations having an orientation + . For our purposes, all orientations will be expressed modulo radians, so that these two orientations are treated as equivalent. We note that in either case it is not necessary to extend the sampling regions beyond the center line (dashed) which divides the annuli since this region is sampled in the complimentary case.
We consider the operator con guration shown in Figure 3.10. For a given pair of regions Q and Q which correspond to an orientation , we detect the presence of enclosure if there is at least one compatible edge found within each region. It is clear that this method may be directly extended to the discretely sampled annular operator. For this, the regions Q and Q are directly implemented as a subsampling of the discrete angular pro le (see Figure 3.11). An example demonstrating the multiple orientations which result when detecting an enclosing structure is shown in Figure 3.12. Using this detection scheme it will be possible to extract limb orientation values in practical settings where there are additional edge 1
2
1
2
38
Q2
Q1
φ
Figure 3.11: Detecting Limb Orientation. An annular region (shaded) is divided into N sectors (here N = 20). For a given orientation ^ 2 f1; 2; :::N g, two sets of sectors Q and Q (dark shaded) are used to detect 1
2
limb enclosure. Limb enclosure is found when there is at least one compatible edge found within each region. The size of these regions is chosen so that the operator is capable of detecting limb enclosures where 0 3 . Here, a limb having = 3 and an orientation ^ is shown by a pair of dotted lines.
structures (see Figure 3.5) or when the contour is fragmented (see Figure 3.6).
Summary
In this section we have discussed how symmetry points are identi ed based upon information present in the discrete angular pro le of a given annular operator. The identi cation process leads to a set of discretized orientation values ^ that are assigned at a given symmetry point. In the next stage, when we attempt to identify and represent individual limb symmetry point clusters, it will be necessary to assign a unique orientation value to each symmetry point in a part cluster. Points will be located within the extended S -space where each symmetry point is speci ed by two spatial coordinates (x and y), a scalar coordinate (s), and an orientation coordinate . Before examining the next grouping stage in detail, we review the annular sampling scheme that has been described in previous work [10].
3.2 Sampling De nitions: A Review The analogy between discrete sampling and slices taken parallel to the x ? y plane in the continuous scale-space has already been made (see Section 1.2). A set of N layers L = fl ; l ; :::lN g is used to sample the scale-space for di erent values of s. For a given layer li, the corresponding scale value si speci es the size of operators used in that layer. The set of scale values S = fs ; s ; s ; :::sN g de nes a sequence in which the scale increases as the set is traversed from s to sN . In a previous report [10], the set of parameters used to specify the annular sampling structure was de ned. We recall that annular operators are positioned at the vertices of an equilateral triangular lattice, and that the lattice spacing d for the rst (smallest scale) layer 1
2
1
2
3
1
We observe that while the same set of symmetry points is used to identify both blobs and limbs, we require orientation information only in the case of limbs.
39 *
**
* * *
**
* * *
**
* *
**
* *
*
**
**
**
**
**
**
**
**
*
**
*
**
*
**
*
**
* *
**
**
* *
**
*
*
*
**
**
**
*
**
**
Figure 3.12: Multiple Limb Orientations. We detect several limb ori-
entations for a particular subset of edges illustrated in Figure 3.6. Sectors having compatible edge orientations are marked with an . Note that each vertical pair of results have equivalent orientations modulo radians. Each of these pairs will be treated as equivalent orientations in all subsequent analyses. Hence, the actual range of limb orientations is given by ^ 2 f1; 2; ::: N2 g.
is assigned the value d . Increasing in scale, the spacing in all other layers i is related to the previous layer by di = di, where > 1 is a xed constant. We note that although this scaling relationship exists, there is no direct spatial registration between points in two adjacent layers since the value of is typically chosen to be close to 1. For a given layer i, the radii which de ne the inner and outer annular boundaries are assigned the symbols ri and ri , respectively. The inner and outer annular radii are related by the parameter , a constant used in all layers: ri = ri (3:1) The inner annular boundary radius is related to the layer sampling spacing by: 1
+1
1
2
2
1
ri = di
(3:2)
1
The scale parameter s is de ned as the average annular operator radius for layer i as follows: i i si = r +2 r (3:3) Making the appropriate substitutions for r and r , the scale of layer i + 1 may be rewritten in terms of layer i as follows: si = si (3:4) This result indicates that, contrary to previous gures (see for example Figures 1.4,2.15, and 2.16), the spacing between layers along the s axis is not regular but instead increases with scale. 1
1
+1
2
2
40
3.2.1 Quantifying Enclosure
One nal topic requiring review is on the quanti cation of enclosure. By assigning the values f ; ; :::ng to the n break angles found in the angular pro le of a given operator (see Section 3.1 above), the degree of enclosure e [11] may be quanti ed as follows: m X e = 1 ? 21 (i) (3:5) i There is an associated enclosure value ei for each of the M symmetry points pi extracted from an image. We will refer to the set E = fe ; e ; :::eM g of all symmetry point enclosure values. Having nished this preliminary review, it is now possible to proceed with a more detailed examination of the previously de ned consistency constraints. This will make it possible to devise a computational means for extracting part representations later in the chapter. 1
2
2
2
=1
1
2
3.3 Consistency Constraints The consistency constraints used to de ne symmetry point clusters were described qualitatively in a previous section (see Section 2.4). Since these constraints will play a critical role in the extraction of part representations, as a preliminary, we need to discuss some of the practical implementation details. Consider a set of M extracted symmetry points P = fp ; p ; ::::pM g, and the set of N sampling layers L = fl ; l ; :::lN g. We de ne a function L : P ! L which for all points in P returns the layer lk = L(pi ) associated with a given point pi . Recall the constraints outlined in Section 2.4. The rst required that all points within a cluster be of the same type (either limb or blob exclusively). This constraint could be simply implemented by testing the type of enclosure found at each point within a symmetry point cluster. There is a problem with adopting such an approach, however, since it is dicult to di erentiate between blob and limb enclosing con gurations based on angular pro les alone. For example, to di erentiate between blob and limb structures based upon the enclosure value e would require that a threshold value be selected. There is, however, no principled means for selecting such a threshold. In developing a robust means for identifying blob and limb structures it will be necessary to employ other information relating to the spatial and scalar distribution of symmetry points within the S -space. Since it is not possible to reliably identify blob and limb enclosures based upon isolated symmetry points, the rst constraint will only apply insofar as ensuring that a given part must be classi ed exclusively as one type or the other. To implement the second continuity constraint requires several issues to be addressed. These relate to the de nition of adjacency for each of the parameters used in either the S -space for blob parts, or the extended S -space for limb parts. We will begin by de ning the neighborhoods used to de ne adjacency between points. Such adjacencies within the discretely sampled S -space are the basis for ensuring continuity within symmetry point clusters. We recall that a hexagonal neighborhood is employed during annular sampling within each layer [10], so that each point pi has six neighbors Ni = fg ; g ; :::g g which all fall 1
2
1
0
2
1
2
6
41 within layer lk = L(pi ). It is possible to identify similar neighborhoods within the layers that are immediately adjacent to layer lk , denoted by Ni and Ni? , corresponding to the next largest (+) and next smallest (-) adjacent scales lk and lk? . For a point pi in layer lk , it is possible to locate the spatially closest points p?i and pi + in the respective adjacent layers lk? and lk . For each of these, there is an associated local spatial neighborhood consisting of six points within the associated layer. This set of points is used to de ne the local spatial neighborhood for the point pi in its adjacent layers (see Figure 3.13). +
+1
1
1
+1
lk
pi
l k+1
lk
l k+1
pi
pi+
Figure 3.13: Spatial Neighborhoods in Adjacent Layers. A point pi
in layer lk has a local neighborhood de ned by six points in that layer. We select a point p+i which in layer lk+1 is spatially closest to pi . The spatial neighborhood Ni+ of p+i within the layer lk+1 is used to approximate the neighborhood of the point pi in layer lk+1 .
For spatial continuity to exist within a cluster of symmetry points, we require that a given point pj be contained within the neighborhood of at least one other cluster point pi. The particular neighborhood selected depends on which layer the point pj is found in. If the point pi is found in layer lk = L(pi ), then for scalar continuity to exist, we require that the point pj be found in an adjacent layer lm = L(pj ) such that (m ? k) 2 f?1; 0; 1g. Our choice of spatial neighborhood is then determined so that pj 2 Ni, where Ni represents the
42 appropriate neighborhood de ned by:
8 ? >< Ni if (m ? k) = ?1 Ni = > Ni if (m ? k) = 0 : Ni if (m ? k) = 1 0
+
(3:6)
This describes the constraints used to ensure that two points are adjacent spatially and across scale. If all points in a given set share this sort of adjacency with at least one other point in the set, then the previously de ned spatial and scalar continuity constraints are satis ed. The associated set C is referred to as a symmetry point cluster. It will be possible to directly extract blob part representations by searching for symmetry point clusters de ned in this way (see Section 3.4). In the case of limbs, a further orientation-based constraint needs to be imposed (see Section 3.5) to enable the di erentiation of parts whose symmetry volumes overlap in the S -space.
3.4 Blob Part Representations As previously discussed (see Section 2.5), the extraction of blob part representations may be viewed as a thinning process applied to the symmetry point cluster associated with a given blob part. This thinning involves removing all but one of the points found in a blob cluster, thereby producing a single point which can be used to represent the entire structure - both in terms of spatial location and scale. We start with the set of raw symmetry data that results from applying annular symmetry operators. Obviously we need to make a distinction between blob and limb points. However, to do this at such an early stage without any additional structural information would require an arbitrary decision threshold to be applied to the enclosure value ei at each point pi (see Section 3.3). Therefore, initially all symmetry points are treated in the same way, with limb orientation values recorded for each point. A given symmetry point pi located in layer lk = L(pi ) has an associated enclosure value ei. As discussed in Section 2.3, blob enclosures possess distinctive S -space signatures. These manifest themselves as symmetry point distributions con ned to fall within concentric circular regions within several layers. We have chosen to select a single representative point within a given symmetry point cluster by looking for enclosure extrema. This is based on the assumption that the center of a symmetry point cluster in a given layer may be approximated by the global extremum of the enclosure function. By locating enclosure extrema in a set of adjacent layers, the locus of the resulting extrema can be used to determine whether or not the cluster has concentric distributions in adjacent layers. A cluster in which distributions are concentric corresponds to a blob part. The point on the extrema locus having the highest enclosure value is chosen to represent the part. Recalling Figure 2.15, an example demonstrating the extraction of a blob part representation is shown schematically in Figure 3.14. The details describing how this is implemented are presented in what follows. To guarantee that symmetry points exist within at least two adjacent layers for any given blob enclosure, we need to ensure that there is adequate inter-layer redundancy. The sampling structure outlined in [10] was con gured so that circles having arbitrary position and size (within some bounded range) could be detected by at least one operator. We can extend this requirement to ensure that operators in at least two di erent layers will respond
43 central axis
(x,y,s)
(b)
(a)
(c)
Figure 3.14: Extraction of a Blob Representation. (a) The distribu-
tions of symmetry points in several adjacent layers are approximately circular and concentric. There is a central axis which approximates the common central axis for these distributions. The center of each distribution is approximated by locating the global extremum in the enclosure function within each layer. This makes it possible to de ne a locus of extrema consisting of one point from each layer (b). If this locus is found to have a small spatial variation, then we infer that the cluster corresponds to a blob part, and a single point - corresponding to an enclosure maximum within the locus - is chosen to represent the entire blob part (c).
by xing the minimum amount of overlap between annuli in adjacent layers. In [10] we chose a sampling strategy that produced no overlap between the restricted ranges of operators in adjacent layers. For scale continuity between two adjacent layers i and i + 1, the restricted range bounds, rmin and rmax, were de ned (see [10] equation (2.10)) by: i = ri rmax (3:7) min Since we would like to ensure that at least two layers respond to a given circle, we require that there be at least 50% overlap between the restricted ranges of operators in adjacent layers, resulting in the following equation: i + ri r min max i rmax = (3:8) 2 This allows us to modify the equation which dictates how the operator spacing d changes between layers. From [10] we recall that, based on equation (3.7), the layer spacing parameter was expressed as: p (3:9) = p3 + 1 3 + 1 Therefore the revised expression for based upon the 50% overlap indicated in equation (3.8) is given by: 2( ? p ) (3:10) = (1 + ) +1
+1
+1
1 3
44 As we would expect, for typical values of and , this revised expression gives lower values of due to the increased degree of overlap. Using an annular sampling structure based on this value of guarantees that there will always be adequate inter- layer redundancy for extracting blob representations using the following strategy. Using the previously de ned consistency constraints (see Section 3.3), a search is performed to sequentially examine all symmetry point clusters in the image. For a given cluster, we consider the computations employed to verify a blob symmetry cluster and extract a representation for the associated blob part. In order to detect a unique enclosure extremum within a given layer, a symmetric two-dimensional Gaussian kernel is convolved with the enclosure values of cluster points within the layer. Since the symmetry point distributions in each layer should be approximately circular, and since the extrema are assumed to roughly approximate the center of each distribution, each extremum can be tested to ensure that it has local support. The set of n extrema Q = fq ; q ; :::qng that are extracted in a set of adjacent layers represent an extremum locus. This locus is then tested to ensure that there is negligible spatial variation when moving from one layer to the next. Such a test corresponds to the condition that distributions within each layer are approximately concentric. Spatial variation is measured by computing the maximum spatial deviation along the extremum locus. For each point qi, there is an associated spatial position vector X~ (qi) in a common (image) coordinate system shared by all layers. The maximum spatial deviation Sdmax is computed as follows: Sdmax = qmax kX~ (qi) ? X~ (qj )k (3:11) i ;qj 2Q 1
2
Negligible spatial variation of the locus is indicated when the value of Sdmax does not exceed a constant which represents the largest spatial neighborhood radius found (corresponding to the spacing dk found within the largest scale layer k found in the locus). Having identi ed a locus which satis es this spatial variation constraint, we then choose a single point from the locus which will be used to represent the entire blob part. To achieve this, a one-dimensional smoothing kernel is convolved with the locus enclosure values to obtain a unique global maximum y. The point corresponding to the global enclosure maximum of the locus is then chosen to represent the entire blob part.
3.5 Limb Part Representations The extraction of limb part representations from symmetry data is somewhat more complicated than for blobs. This is because the actual limb representation is generally more complex (a curve as opposed to a single point). Thus it is necessary to simplify the symmetry data prior to determining a representation. This simpli cation, which will be discussed in the following subsection, is based upon computing an orientation consistency measure for each symmetry point. This will allow a single limb orientation value to be assigned to each symmetry point. It will also provide a means of reducing the number of symmetry points within a cluster based upon a thinning process. Following this, it will be possible to extract Recalling Section 2.2 we employ a scale constant which is proportional to the maximum expected symmetry point distribution radius max (see equation (2.4)). y Here a simple averaging operation is sucient for locating a unique maximum.
45 limb part representations based on the previous consistency constraints (see Sections 2.4 and 3.3) to de ne valid paths within symmetry point clusters.
3.5.1 Cluster Simpli cation Orientation Selection
The rst simpli cation stage involves assigning a unique limb orientation value to each symmetry point in the image. Recall that for each point there is a set of associated limb orientations that result from the detection of enclosing structures within a given annular region. Typically there are multiple orientations because, for a given enclosing structure, it is possible to infer limb enclosure in several directions (see Section 3.1). We seek to to assign a single orientation value to each symmetry point. The process whereby multiple orientations are reduced to a single value will depend upon nding consistent orientation con gurations in the neighborhood of each point. In a previous section, several consistency constraints were introduced (see Section 2.4). One of these, relating exclusively to limbs, proposed that orientation values should vary continuously within a symmetry point cluster. Although we have yet to de ne this precisely, we can use the idea of smoothness when performing local operations directed at choosing a single orientation value at each symmetry point. While there are several existing network-based optimization methods which could be used to solve this problem [23][20], they do not provide output data in a form readily suited to the second simpli cation stage (thinning). Thus, employing such a network-based approach would require that completely separate method for implementing a thinning process. Instead, we have chosen a computationally simpler method which can be used for both the selection of orientation values and thinning of point clusters. The results, though perhaps less accurate than those found using other approaches, have been found to be adequate for our purposes. The method used for choosing a single orientation value for each symmetry point may be viewed as a nonlinear operation applied at each symmetry point in a convolution-like manner. We wish to identify for each symmetry point pi , all other points located in the same point cluster based upon the consistency rules introduced previously (see Section 3.3). To do this, consider an elongated three-dimensional Gaussian kernel centered at the origin of a coordinate system de ned by the variables u, v and w. When centered at point pi in S -space, the ellipsoidal shape of the kernel is meant to locally approximate the symmetry volume (see Figure 3.15(a)). The kernel is expressed as follows: G^ (u; v; w) = Ke? au av aw (3:12) (
where
+
+
)
a% = %2%2 for % 2 fu; v; wg (3:13) and K is a constant. The Gaussians are truncated at 3% so that the kernel extent may be expressed as the ellipsoidal volume: (3:14) [au + av + aw ] 92 We choose a kernel that is elongated in the u direction. The w axis is aligned such that it 2
46 scale u v
w pi
y pi
y
x
x
(a)
(b)
Figure 3.15: Application of a Gaussian Kernel. (a) A three-
dimensional Gaussian kernel (shaded) centered at the point pi locally approximates the limb symmetry volume. (b) The projection of limb orientation values into the x ? y plane. Multiple limb orientation values at a point are shown as oriented line segments, the length of which varies with scale. The kernel is oriented so that its principal axis u (dashed) is aligned with one of the limb orientation values at that point. Orientation consistency considers all cluster points which fall within the kernel region.
corresponds directly to the scale variable s in the S -space. The other two variables, u and v, represent spatial coordinates that are independent of the S -space coordinates x and y. The scale constants which de ne the operator width are chosen in proportion to the maximum distribution width possible for operators in layer i (as represented by the annular width (ri ?ri ) = di(?1)). Since the extent of a limb symmetry volume is approximately the same p in both the spatial (r = x + y ) and scalar (s) dimensions (see Figure 2.13), both of the width scale constants are assigned the same value (v = w ). The scale constant u, de ning the operator length, is chosen to make the overall aspect ratio is greater than one. Kernels are centered on each of the symmetry points found in a given cluster. A computation is performed by using the kernel values G^ to weight the contributions of neighboring symmetry points. There are a number of limb orientation values at the point pi . For each of these, a kernel is positioned by rotating within the x ? y plane so that the long axis (u direction) is aligned with the limb orientation (see Figure 3.15(b)). A measure of the orientation consistency O (pi) is computed by considering all other cluster points having the same orientation value . The distance from the central point pi to another point pj is measured in the u; v; w coordinate system. For a given cluster, we consider the set A of all points which exhibit a limb orientation value . The orientation 2
1
2
2
We have used an aspect ratio of three. However, in practice it does not appear that a particular choice of aspect ratio will have a profound in uence on the results for ratios less than ve.
47 consistency value O (pi ) computed for the point pi is then given by:
O (pi ) =
X ^ G(uj ; vj ; wj )
pj 2A
(3:15)
where the distances uj ,vj and wj from the central point pi to another point pj are measured in the (rotated) u; v; w coordinate system. The value of O (pi ) indicates the amount of local support for an orientation at the point pi. Having computed a value of O (pi ) for all limb orientations found at pi, we select the maximum. The corresponding orientation
is chosen to represent the limb orientation at that point. The maximum value of the orientation consistency function O (pi) will be used in the following thinning stage. Since there is no longer a need to associate orientation values with the function O, the simpli ed notation O(pi ) will henceforth be adopted when referring to the maximum value O (pi ).
Thinning
For limb symmetry point clusters, thinning can be employed to reduce the total number of points, thereby facilitating the extraction of an axial representation. Consider the example of a contour consisting of two diverging straight line segments (see Figure 3.16). We recall
projection
Figure 3.16: Layer Intersections for a Limb Symmetry Volume.
The symmetry volume for two diverging straight line segments (outlined by dotted lines) is sampled discretely by layers positioned along the scale axis s. This results in `kite-shaped' intersection regions within each of the sampling layers. Overlap between intersection regions of adjacent layers is seen by projecting the result for each layer onto a common plane.
from a previous section (see Section 2.3) that the regions where a limb symmetry volume is intersected by the sampling layers are generally `kite-shaped' quadrilaterals. We consider the hypothetical situation where sampling is performed continuously within each of the layers.
48 By applying an ideal thinning process to these regions, so that both intra- and inter-layer redundancies are removed, we would expect to obtain a result like that shown in Figure 3.17. In this case, each kite-shaped region has been thinned down to a single line segment. If we were to consider the projection of these line segments onto the spatial plane, the result would be a single continuous line segment roughly corresponding to the Medial Axis of the contour. By parameterizing positions along this segment, the plot of scale as a function of position consists of a series of constant values separated by jump discontinuities. Although this example is not practically realizable, it does demonstrate an ideal that we hope to approach when thinning real data. In what follows we describe a computational strategy that attempts to approximate this performance.
s
projection l l (a)
(b)
Figure 3.17: Thinned Limb Symmetry Clusters. (a) An ideal version
of thinning which reduces the distributions within each layer to line segments (discrete sampling not shown). Projecting the resulting set of line segments into the spatial plane results in a single continuous line segment. Positions along this segment are parameterized by l. By plotting the scale values s versus l, the resulting function has a step-like appearance (b).
At each point in a given cluster, the local support for the chosen limb orientation value was indicated by the orientation consistency function O (see Section 3.5.1). From the summation in equation (3.15), we would expect that points located towards the center of the symmetry point cluster would generally be assigned higher values of O when compared with points located more peripherally. This e ect is further accentuated by using the Gaussian kernel of equation (3.12) to weight the point contributions. We can imagine a thinning method which seeks to preserve only the ridges of the function O within a given symmetry point cluster. Our approach has been motivated by the work of others [6] who have used a similar strategy for simplifying groups of extracted edge segments. By locating elongated Gaussian kernels on extracted edge data, the authors in [6] were able to de ne a `potential valley' which was used to force the two- dimensional positioning of a deformable curve representation towards the valley bottom. We have chosen to use a similar three-dimensional potential function to guide a thinning process used to remove redundancies from symmetry
49 point clusters. The process is quite similar in that we seek a means of identifying ridges in the potential function (not valley bottoms since the potential function is the inverse of that used in [6].). Numerous methods for thinning binary image data have been studied in the past [14]. However, in our case, we require a means of thinning data which also preserve ridges in the continuous potential function mentioned previously. Techniques for performing thinning under such conditions can be classi ed as `gray-scale' thinning algorithms, for which several examples may be found in the computer vision literature [9][15][16]. A particularly simple gray-scale thinning algorithm has been employed here based on the method proposed in [7]. We consider the set of all possible paths joining two points pf and pl in a symmetry point cluster. A path is de ned to be a sequence such that the rst point is pf , the last point is pl, and all other path points pi reside within the symmetry point cluster. In addition, for any two consecutive points pi and pj , it is necessary that both be spatially adjacent. It is possible to consider the minimum value of Omin when evaluating the function O over all points along a given path. The degree of connectedness is determined based upon the path which has a maximum value of Omin for all possible paths between the two points pf and pl . A thinning algorithm may be constructed in which symmetry points are removed only if their removal does not disturb the degree of connectedness of any pair of points in the cluster. This may be implemented based solely upon comparisons made between the values of O within local neighborhoods. Consider the point pi and its set of six adjacent neighbors Ni = fn ; n ; :::n g which de nes a local spatial neighborhood within the same layer lk . To perform thinning within the S -space, we have to consider the local scalar neighborhoods as well. For the point pi there are two other points which will be used to de ne the local neighborhood completely. For these we use the points pi and p?i which are spatially closest in each of the adjacent layers (see Section 3.3). The set N^i = fNi ; pi ; p?i g represents the complete local neighborhood for a point pi in the discretely sampled S -space (see Figure 3.18). To implement the thinning algorithm, we consider a point pi and its set of local neighbors N^i. Each of the associated nine points has an associated value of O. Maintaining global path connectedness may be implemented by testing connectedness within local point neighborhoods. Thinning is accomplished based on the following simple algorithm: For each cluster point pi, consider the degree of connectedness for all possible paths between pairs of neighboring points. We exclude all trivial cases such as the path between two adjacent neighbors ni and nj where i = j 1, or i = 1 and j = 6. 0
1
2
6
+
0
+
If the degree of connectedness is always greater than O(pi ), then changing the value of
O(pi ) has no e ect on the connectivity of any of its neighbors. As a result, the value of O(pi ) can be replaced by the lowest value within the local neighborhoody . The process is repeated iteratively for all symmetry points in a given cluster until a stable state is reached.
So that values may be compared across scale, it is necessary that all O values be normalized by scaling in proportion to the associated kernel size (u v w ) used for that layer. y We note that some neighbours may have a value of O(n) = 0 in which case, by setting O(p ) = 0, the i point pi is e ectively removed from the cluster.
50 s l k+1 pi+
pi
lk l k-1
pi-
Figure 3.18: Symmetry Point Neighborhood. The neighborhood for a
point pi consists of six points within the same layer lk , one point p+i in layer lk+1 , and another point p? i in layer lk?1 . A vertical axis is drawn through the point pi to denote the scale direction. Typically, the points p+i and p?i are not located directly along this axis due to sampling grid di erences between the di erent layers. Faint lines indicate all possible adjacencies between points.
This method of thinning is quite robust since it ensures that connectedness will be maintained in all cases. As a result, however, it is quite common that the end product will not always be a perfectly thinned chain (or a series of branching chains) of points. While the thinned limb may not be used to directly represent limb parts, it makes it possible to substantially reduce the number of points found in the associated symmetry point clusters. This limits the computational burden in the next stage where a search within symmetry point clusters will be required. Having considered two means for simplifying limb symmetry point clusters, we now turn to the problem of extracting limb parts representations. These will be in the form of paths which are found within symmetry point clusters. In the next section we introduce the constraints used in de ning valid paths for limb representations.
3.5.2 Path-Based Representations
In a previous section (see Section 3.5) we outlined the requirements for a useful limb part representation. For a given limb part and its associated symmetry volume, there is a central axis corresponding to the center of the volume. From this we hypothesized that nding a close approximation to the central axis would provide a useful limb part representation. This is because the central axis closely corresponds to the symmetry curve which represents the Medial Axis in the three- dimensional S -space. Starting with symmetry point data which has been simpli ed using the methods described above, we require a means of extracting It is not absolutely necessary to rst thin the cluster data, although in practice it signi cantly improves the overall computational eciency.
51 structures which approximate the curve corresponding to the symmetry volume's central axis. We begin by considering how such a curve may be represented in the discretely sampled S space. Based on this, we will devise a simple means of extracting limb part representations from symmetry data. Curves in the discretely sampled S -space are represented by a path which is de ned to be a sequence of points p ; p ; :::pn such that each pi is adjacent to pi? for 2 i n. Adjacency has already been discussed in the context of local spatial and scalar neighborhoods (see Sections 3.3 and 3.5.1). We recall the continuity constraints that were previously introduced when the concept of symmetry point clusters was introduced (see Section 2.4). Since these constraints were meant to apply to all points in a given cluster, it is clear that they should also apply to the paths within that cluster. The constraint which requires smooth limb orientation values has yet to be discussed in detail. It will become clear that this constraint is particularly useful when attempting to separate complicated limb con gurations - such as when limb symmetry clusters overlap in the S -space. Since the constraints will, in general, be important for de ning valid paths within a symmetry point cluster, we begin by reviewing all of the relevant constraints as they apply to limb part representations. 1
2
1
Path Constraints
In Section 2.4, three types of continuity were introduced as constraints to be applied when de ning symmetry point clusters. The rst required that points within a cluster be spatially continuous. This naturally extends to paths within a symmetry point cluster which are de ned in terms of adjacency relationships. An example of a spatially discontinuous limb con guration is given in Figure 3.19. The spatial continuity constraint is applied at a local s
Symmetry Curve
contour
y symmetry axis
x (b)
(a)
Figure 3.19: Spatial Discontinuity. (a) A limb part having a large contour break will result in a discontinuous axial representation. (b) This is manifested by a discontinuity in the S -space curve.
level by comparing pairs of consecutive points pi and pi along a path. In Section 3.3, speci c conditions were introduced which can be directly applied to test the spatial and scalar adjacency (and hence continuity) for any two such points. +1
For the sake of clarity, we have presented results in the continuous S - space
52 The second consistency constraint required that points within a cluster manifest continuity across scale. Again, this constraint may be directly applied to paths within a given symmetry point cluster. An example of a scalar discontinuity is shown in Figure 3.20{. While the symmetry curve is continuous in the spatial domain, there is a signi cant discontinuity present in the associated symmetry curve. s
Symmetry Curve
contour
y symmetry axis
x (b)
(a)
Figure 3.20: Scale Discontinuity. (a) A contour con guration with an
abrupt scale change between a large and a small scale region. The symmetry axis may be continuous in such cases. However, the symmetry curve has a signi cant discontinuity which distinguishes two separate limb parts (b).
The nal continuity constraint relates speci cally to limb parts and has yet to be fully described. In Section 2.4 we required that orientation continuity be present within limb symmetry point clusters. While this may not be such a meaningful constraint to apply when attempting to locate all the points in a symmetry point cluster, it is particularly useful in constraining the sorts of paths used to describe limb parts. We note that smoothness in the three-dimensional S -space is equivalent to orientation continuity for curves in the fourdimensional extended S -space. An example of a contour which gives rise to an orientation discontinuity is shown in Figure 3.21{ . The implementation of the limb orientation continuity constraint is substantially more involved than for the previous two constraints. In the remainder of this subsection we will examine the details pertaining to this implementation. Recall that limb orientations are quantized (see Section 3.1). An orientation ^ is given by an annular bin number representing the circularly periodic sequence of annular bins ^ 2 f1; 2; 3; ::: N g. Two orientations ^i and ^j are adjacent if their associated bin numbers are adjacent within the circularly periodic sequence such that: either ^i = 1 and ^j = N or ^i = N and ^j = 1 (3:16) or j ^i ? ^j j 1 2
2
2
The de nition of adjacency for paths in the discretely sampled extended S -space is complicated by the fact that for curves, there is a direct relationship between spatial position
53
φ contour
l
l
Figure 3.21: Smoothness and Orientation Discontinuity. A contour with a non-smooth but continuous symmetry curve in the S -space. The associated singularity manifests itself as a curve discontinuity in the extended S -space. This is demonstrated by plotting the orientation versus the parameterized position along the symmetry axis l.
C
A
B
A
A
B
C
B C
(a)
(b)
(c)
Figure 3.22: Adjacency in the Extended S -space. Sampling within a
single layer is shown by black dots on a triangular grid. Three limb points A, B, and C are identi ed with quantized limb orientation values indicated by arrows ( ^A , ^B , ^C ). (a) A valid path con guration where ^A = ^B and ^C = ( ^A + 1). (b) An invalid path con guration since ^C > ( ^B + 1). (c) Orientations as in (a), but path is invalid due to the spatial con guration of points.
54 and curve orientation. We consider the following simple example (see Figure 3.22). The rst example demonstrates a valid path con guration (see Figure 3.22(a)). In the second example, the path is not valid since the smoothness constraint is not satis ed between points B and C (see Figure 3.22(b)). In the nal example, although the orientation is continuous when traversing the three points, since the spatial con guration does not support a single curve having these orientations, the con guration does not represent a valid path (see Figure 3.22(c)). From this we see that, while satisfying the continuity constraints as stated above is a necessary condition for de ning paths in the discretely sampled extended S -space, it is not a sucient condition. In the following, we will attempt to characterize valid path con gurations by considering the relationship between spatial and orientation constraints. From this it will be possible to devise a simple way to locate valid paths within extracted symmetry data. As a preliminary to this, we rst consider some of the constraints which arise naturally due to the annular sampling scheme and which act to limit the range of possible curvatures for limb parts.
Limb Curvature
There are factors which limit the range of detectable limb curvatures for a given annular operator. By limb curvature we are referring to the curvature K of the (continuous) limb symmetry axis at a given point (see Figure 3.23). For each of the contour segments falling within the annulus, the curvature may be approximated by circular arcs having curvatures K and K . The sign of these curvature values is set by the direction of curvature relative to the detecting annulus so that curves bending towards the center are assigned positive values. In Figure 3.23, the curvature K of the upper contour takes on a positive sign, whereas K is negative. In general, the unsigned curvature value K at the annular center will fall within a range de ned by these contour curvatures. Two separate cases exist. In the rst, both K and K have the same sign, which results in a range de ned by: 0 K j min(K ; K ) j (3:17) 1
2
1
2
1
2
1
2
In the second case, where the two curvatures have di erent signs, the range is de ned by: min(j K j; j K j) K max(j K j; j K j) (3:18) 1
2
1
2
It is possible to set an upper limit on the maximum allowable curvature for a given limb symmetry axis. Several degrees of limb bending are shown in Figure 3.24. In principle, for limbs derived from parallel contours, the part can be bent until it is 'doubled over' (see Figure 3.24(c)). Any further bending of the limb beyond this produces more complex self-occluded situations. However, the maximum curvature along the symmetry axis is not altered (see Figure 3.24(d)). When the sides of a limb structure are converging, it is again possible to bend the limb until it is doubled over (see Figure 3.24(g)); however, if the limb is bent beyond this point, the maximum curvature value will be altered (see Figure 3.24(h)). For both the parallel and converging contours, it seems natural to choose the doubled over This represents a hypothetical situation in which the inner contour attains in nite curvature at a cusp point where the support for enclosure becomes in nitesimally small.
55
K1 = R1
1 R1
R1
R1
curvature K
K2 =
R2
R2
1 R2
R2
Figure 3.23: Limb Curvature. Limb enclosure is characterized by two separate annular regions in which edges are found. The contours in each of these regions may be approximated by circular contours having radii R1 and R2. Relative to the annular center, the sign of curvature depends on the direction of bending - either away from (?) or towards (+) the annular center. In this example, the symmetry axis at the annular center will have an unsigned curvature K which lies somewhere between the two curvature values K1 K K2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 3.24: Progressive Degrees of Limb Bending. (a-d) A contour with parallel sides. (e-h) A contour with converging sides.
56 con guration to represent the highest expected curvature for a limb. Due to nite sampling resolution at the cusps, since neither con guration is practically detectable, we can be sure that no detectable bent limb con guration will have a curvature exceeding the doubled-over limb. We will also assume that the di erence between maximum curvatures occurring for parallel contours will not be signi cantly di erent from converging contours. Then it is straightforward to use the con guration in Figure 3.24(c) to compute the maximum attainable symmetry axis curvature for a given scale. Figure 3.25 shows a doubled-over limb con guration where the largest possible annulus is used to detect limb enclosure. This con guration
W
Figure 3.25: Bent Limb Con guration. The largest annular operator detecting limb symmetry has an inner radius r1 such that r1 = W . represents the largest curvature Kmax that the operator would ever be (hypothetically) required to detect. The curvature of the limb axis in this case is W , so that the value of Kmax is related to the annular parameters by: (3:19) Kmax = r1 = 1 2+s For a given layer having a scale s, this equation provides a means of calculating the maximum limb curvature detectable using operators at that scale. With this result, it is possible to establish the appropriate number of annular sectors for a given sampling con guration (and hence the limb orientation resolution; see Appendix A). By considering the range of curvatures detected at a given scale, it is now possible to de ne more precisely the relationship between spatial and orientation constraints for valid paths. 2
1
Spatial and Orientation Constraints
The paths that will be used to represent limb parts have been introduced as approximations to space curves which lie inside the symmetry volume associated with a given limb part (see Section 2.5). For the sake of explanation, we will assume that there is one speci c curve which we are interested in approximating discretely, and we will refer to the symmetry axis which corresponds to the projection of this curve into the spatial plane. Within a given layer li, each annular operator is located at one point of a regular triangular grid. Which points are used to construct the approximating path will depend upon how close they are with respect to the symmetry axis. It is possible to construct a set of Voronoi cells (referring to
57 the regions enclosed by Voronoi polygons [17]) for the set of sampling points - each of which represents a region of in uence for the point located at its center. For all points located within a Voronoi cell, we know that the closest grid point is the one located at the center of the cell. If we consider the Voronoi cells associated with the sampling points in a given layer, and overlay the actual (continuous) symmetry axis that is to be approximated, it is possible to determine which of the sampling points will be used to approximate the axis. The Voronoi cells for a triangular grid are bounded by regular hexagons. Consider three consecutive points a, b, and c along a path used to represent the symmetry axis in the discrete domain. It is clear that only a certain range of spatial/orientation combinations will be possible for all potential symmetry axis curves(see Figure3.22). For each set of three consecutive limb points, we may formulate a relationship between the spatial position and the limb orientation at each of the points in order for them to be part of a valid path. To begin, we consider a set of path points which fall along a straight line as in Figure 3.26(a). Four
a
Φ
b
(a)
c
a
b
c
(b)
Figure 3.26: Neighborhood Constraints (i). All possible symmetry curves eliciting a response from operators a, b, and c are considered. For the triangular sampling grid, Voronoi cells are hexagonal in shape (shaded). (a) Circular arcs represent examples of curves having maximal curvature Kmax for the scale shown. These arcs are used to de ne a range of curves that can be represented by the three grid points. Upward curving boundaries are shown by dashed curves, and downward curving boundaries are shown by solid curves. (b) A range of orientations (black ./ regions) at each sampling point is established by the curves in (a). White line segments indicate the orientation ? about which each range of orientations (? ) is centered.
circular arcs with curvature Kmax are drawn so that they just glance the boundaries de ning the Voronoi cells for operators a and c. These limiting curves delineate an envelope for the the maximum possible range of curves - both spatially and in terms of curvature - that can be represented by a path consisting of points a, b, and c. At each point there will be a range of orientations which are consistent with the set of all curves which may be approximated by these three points (see Figure 3.26(b). The range of orientations ? at a given point
58 is de ned by a central orientation direction ? and a deviation from that orientation. For the case shown in Figure 3.26, if the line passing through the three grid points de nes a reference orientation, then the central orientation at each point will be de ned by ? = 0 (as indicated by white line segments in Figure 3.26). Although the deviations at a and c are slightly greater than at b, for simplicity we will assume that the deviation at b is the same as at the other two points. We require a means of estimating the deviation value that occurs in this setting. Consider the limiting curve labelled in Figure 3.26. From the symmetry that is present, it is clear that a deviation value computed based on applies to any of the four limiting curves. A detailed analysis of the geometry which gives rise to a computation of is shown in Figure 3.27. Here a set of curves f; ; g are related to since they all pass through the point O. By inspection, it is clear that these curves all represent limiting instances for the operator centered at c since any further translation of the curves upwards or rotation of the curves counterclockwise results in the curve falling outside of the Voronoi cell at c. In order to determine the deviation associated with this con guration, we seek the largest possible tangent angle, measured relative to ?, for any curve falling within the Voronoi cell at a. This value will correspond to the largest limb orientation value that we would expect to nd at point a for any path going through points a, b, and c. V
O d l a b
c
Λ Σ
U
Q
∆
P
Φ
Figure 3.27: Orientation Variation for Limiting Curves (i). A set of limiting curves f; ; ; g for Voronoi cell c such that each has a maximum
curvature value Kmax and passes through the origin O of a coordinate system (u; v ). Tangent angles are measured relative to the (horizontal) line passing through the three symmetry centers. To determine the maximum possible deviation we need to nd the maximum possible tangent angle for all such curves in cell a.
It is clear that the maximum tangent angle will occur at a point where one curve crosses the Voronoi cell boundary. The points P and Q are potential candidates for such a point. We consider the tangent angle of the curve at P and the curve at Q as measured relative to the u axis. For each of these curves, the chords OP and OQ are coincident with the u axis.
59 Thus, the tangent angle of the curves passing through points P and Q can be expressed as:
= arcsin[ 2ur ] (3:20) where the inner radius r corresponds to maximum limb axis curvature Kmax = r1 = d . Since the function in (3.20) increases with u, it is clear that the tangent angle reaches a maximum for the curve , which crosses the Voronoi boundary at the point Q which is farthest from the origin O. This result can be used to compute the maximum tangent angle based upon the lower right Voronoi cell boundary. However, we still need to test the curves ( and ) which cross the lower left cell boundary (represented by the ray l) to ensure that tangent angles do in fact reach a maximum at the point Q. Positions along the ray l are expressed as a function of the parameter t based on the following: u(t) = pd + tdp (3:21) v(t) = td 1
1
1
1
4
3
3
2
2
where 0 t 1. The chord segment between the origin and a point (u; v) on l is represented by X . From equation (3.20), the angle of the curve tangent at a point on l measured relative to the chord X is computed as:
^ = arcsin[ j2Xr j ] (3:22) In order to express the tangent angle relative to the u axis, we subtract the angle of the segment X relative to the u axis to arrive at the nal expression:
= arcsin[ j2Xr j ] ? arctan[ uv ] (3:23) Since both terms contain d in their numerator and denominator, the function (t; ) is independent of d, as we would expect. Using this, we may now demonstrate that tangent angles reach a maximum at the point Q - or equivalently that the function reaches a maximum at the point t = 0 on the interval 0 t 1. A plot of within this interval was obtained for several values of (see Figure 3.28) From these curves it is clear that the function (t; ) is monotonically decreasing on the interval for values greater than = 1:8. Since we typically employ values that are greater than 2.0, it is clear that the tangent angle will always be at the point Q on the Voronoi boundary. Using equation (3.20), it is possible to express the angular deviation in relation to the horizontal axis ?. The u axis in Figure 3.27 is oriented radians from the horizontal. By substituting the distance of Q along the u axis (u = pd ) into equation (3.20) it is possible to express the angular deviation relative to the horizontal axis ? as: = 6 + arcsin[ p2 ] (3:24) 3 With higher values of we recall that the sample spacing gets smaller relative to the annular size, thereby increasing the e ective sampling resolution. As we would expect, the deviation value is reduced (approaching ) as the value of is increased. 1
1
6
4
3
6
60
Tanget Angle (degrees)
60
1.4
50 1.6 40 1.8 30
2.0
20
2.5 3.0 3.5 4.0 4.5
10
0 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
t
Figure 3.28: Tangent Angle versus t (i). For the example shown in
Figure 3.27, we consider the tangent orientation at the point where curves cross the Voronoi cell boundary de ned by l. The parameter t corresponds to the position along the boundary edge. Curves are plotted for various values of (values shown to the right of graph).
Φ
a
a
b
b c
c
(a)
(b)
Figure 3.29: Neighborhood Constraints (ii). See caption for Figure 3.26. Here a `kink' in the path is manifested by a directional shift of 3 at the point b.
61 In the above examples, the three path points fa; b; cg were located along a straight line. Other con gurations are possible which involve a `kink' in the path, resulting in a directional change of radians at the central point (see Figure 3.29). We will not consider the other possible instances where the path has a kink of as these correspond to redundant situations. For kinks of it is possible to apply a similar analysis as in the previous case. From Figure 3.29 it is evident that, due to a high degree of symmetry which exists between the envelopes, the central orientation ? and the range at grid points a and c are the same. We will assume that these values are the same at b as well. Consider again just one of the limiting curves in Figure 3.29. Due to the symmetry which is present, the deviation value computed from this example can again be applied generally to all points in a kinked path such as this. In Figure 3.30 we see an example of one set of limiting curves passing through the u; v origin O. Again it is clear that the maximum tangent angle will occur at some point where 3
2 3
3
v
O
b
a
d c
l P Λ
Q Φ
∆ Σ
u
Figure 3.30: Orientation Variation for Limiting Curves (ii). A set of limiting curves f; ; ; g for Voronoi cell a, such that each has a maximum
curvature value Kmax and passes through the origin O of a coordinate system (u; v ). Tangent angles are measured relative to the orientation ? which is 2 radians from the horizontal (white line segments). To determine the maximum possible deviation we need to nd the maximum possible tangent angle for all such curves.
a curve crosses the Voronoi cell boundary. The most likely place for a maximum tangent angle to occur is along the Voronoi cell edge PQ, and it is unlikely that a maximum will occur along any of the other edges. As done previously, we consider how the tangent angle
measured relative to the u axis changes for curves whose crossing point falls somewhere Since we will eventually use these paths to estimate the length of a limb part, we require these redundancies be minimized.
62 along PQ. For this, the parameter t is used to locate points (u; v) along the ray l as follows: u(t) = d(2 ? t ) (3:25) v(t) = ?ptd where 0 t 1. Equation (3.22) may still be used to de ne the angle of the curve relative to the chord segment X . The expression for the curve tangent angle relative to the u axis is as follows:
= arcsin[ j2Xr j ] + arctan[ ?uv ] (3:26) In Figure 3.31, this function is plotted versus t for various values of . From these curves we 2
2
3
1
Tanget Angle (degrees)
50
45
1.4
40
1.6 1.8
35
2.0 30 2.5 3.0 3.5 4.0 4.5
25
20
15
10 0
0.1
0.2
0.3
0.4
0.5
t
0.6
0.7
0.8
0.9
1
Figure 3.31: Tangent Angle versus t (ii). For the example shown in
Figure 3.30, where curves cross the Voronoi cell boundary de ned by l. The parameter t corresponds to the position along the boundary edge. Curves are plotted for various values of (values shown to the right of graph).
observe that the function (t; ) is monotonically increasing on the interval 0 t 1 for values greater than = 1:8. Since we typically employ values that are greater than 2.0, it is clear that a local tangent angle maximum exists at the point P on the Voronoi boundary. We can compute the maximum value m by substituting into equations (3.25) and (3.26) for t = 1. The deviation is then computed relative to the reference orientation ? as: = 6 + m (3:27) This equation, along with (3.24), can be used to determine when a path satis es the continuity constraints. We note that both have been derived to give rather precise angular results, considering the coarse resolution at which limb orientation values are recorded. However, this exercise was necessary in order to provide some principled means for selecting a valid angular range, given a particular path con guration. Since the results of equations (3.24) and (3.27) are known to have a sound geometrical basis, it is then straightforward to apply these con dently in the discretely sampled orientation domain where the resolution is well de ned.
63
Valid Limb Paths: a Summary
When approximating curves using paths de ned on a triangular grid, we have demonstrated that a relationship exists between the set of spatial and orientation constraints. For a path to be valid, the following rules must be satis ed for any three consecutive path points a, b and c. The points a and c must be neighbours of b such that either: (a) The points a, b, and c lie along a straight line; or (b) A kink exists so that the path changes direction by radians at the central point b. 3
The limb orientation values at each of the three points must fall within the ranges
speci ed by equation (3.24) or (3.27), depending upon whether or not there is a kink in the path. The change in orientation values between a and b and between b and c must be continuous, as discussed previously (see Section 3.5.2). Any scale changes which occur between a and b or between b and c must be satisfy the continuity constraints as previously discussed (see Section 3.5.2). We have discussed how continuity in the four-dimensional extended S -space provides the necessary tools for limiting the types of valid paths used to represent limb parts. What remains to describe in this chapter is a computational scheme for extracting blob and limb part representations in typical imaging situations involving complicated symmetry point clusters.
3.6 Computational Method Previously, we examined the question of what constitutes a part in terms of the symmetry point clusters which were de ned based on a set of consistency constraints. Using these clusters it is possible to propose a method whereby part representations can be extracted. In this section, we discuss how these abstract ideas are incorporated into a practical scheme for extracting blob and limb parts from real image data.
3.6.1 Blobs
A method which can be used for identifying and representing blob symmetry point clusters was outlined (see Section 3.4). However, the sequential process that is used for selecting a particular symmetry point cluster has yet to be discussed. We have chosen to extract blob part representations based upon a hierarchical scheme which starts by nding those parts which are `least likely to be accidental'. In practice it has been observed that blob structures may arise in settings where there is no direct correlation with a physical object in the scene. These situations occur particularly in highly textured images where there are Note that if a scale change occurs between points, then a neighbourhood is based upon the de nitions in Section 3.5.2.
64 many edge contours which happen, by chance, to give rise to blob enclosures. We seek to characterize those situations which are least likely to be accidental, and therefore more likely to be associated with the physical objects in the scene. Such situations will be referred to as having a high degree of physical support. The following assumptions will be used when attempting to compare the physical support of two blob parts: 1. Parts having a higher degree of enclosure are less likely to be accidental, and therefore have a greater physical support. 2. Part which are physically larger (having a higher scale value s) have a greater physical support. The rst assumption makes sense from a quantitative standpoint. Since more edge data is required to be con gured in a highly structured manner for highly enclosed contours, it is less likely that that the contour has arisen without having any direct relation with objects in the scene. The second assumption is made based on the fact that (relatively) small blob parts often arise from image structures (for example, texture patterns) that have a small scale relative to the larger scale object regions. Both assumptions are also appealing from an intuitive standpoint. We construct a simple measure Sb = s! E which is used to quantify the degree of physical support for blobs, with the constant ! used to weight the relative contributions of the two parameters. The process whereby blob part representations are extracted is described as follows. A search is performed over all possible blob symmetry clusters to locate the part having the highest Sb value. Once found, a representation is extracted, and all symmetry points in the associated cluster are removed from the data set. Since there is a destructive aspect to this extraction process, it makes sense that the parts having highest physical support be extracted rst. Otherwise there would be the possibility of losing data pertaining to a more signi cant part by rst locating and removing data associated with an accidental part. The process continues until all blob part representations within an image are found. The result is a list of blob parts ranked in terms of the measure Sb.
3.6.2 Limbs
Although the method used to extract limb part representations is somewhat more complicated than it is for blobs, it employs a similar search- based strategy. Again, we consider a hierarchical extraction process which attempts to nd the part having the greatest physical support rst. Limb parts which are the least likely to be the result of accidental edge structures are again considered to be most likely source of structural information pertaining to physical objects in the scene. In the case of limbs, we assume that long, narrow structures are least likely to arise by chance in an image. We note the contrast with blobs, which favoured parts having a larger area. In practice we have found that many spurious largescale limb parts often arise. The associated path lengths for these may be relatively small in terms of the number of points. However, at large scales the absolute length of the part may be quite signi cant (even if there are only three points in the path). By favouring long narrow limbs, we are in fact looking for limb representations which have the greatest number of symmetry points in the associated path locus. This leads to a physical support measure for limbs having the form Sl = N , where N represents the number of points found along
65 a given path. Beginning with a set of thinned symmetry data, we perform a search which attempts to nd the part having the greatest physical support in the image. Once found, a representation (the path) is extracted, and the associated cluster data is removed. The rationale behind starting with the most signi cant part is the same as it was for blob parts. The search process is then repeated until all part representations are extracted. The result is a list of parts ranked based on the measure Sl. Finding the path having the greatest physical support within the extended S -space can be viewed as an optimization problem. However, the problem is not well suited to most mathematical approaches that are traditionally used in functional optimization. Since it is possible to view a path, and the set of all possible paths in the extended S -space as a graph, it seems reasonable to use a method speci cally suited to solving optimization problems within a graphical framework. For this we brie y review some of the graph theory which is relevant to the problem at hand.
Relevant Graph Theory
A graph is de ned by a set of nodes V = fv ; v ; :::vng and edges which represent connections between nodes. In our case, the set of symmetry points represent the nodes. For two nodes, edges are present only if the points are adjacent based upon the previously de ned consistency constraints (see Section 3.5.2). A path in such a graph is a set of edges and nodes such that each is distinct (there is no duplication). Thus, in the case of the extended S -space, a path would neither cross itself nor intersect itself at any point. If a set of nodes and edges intersects itself at the end point so that v = vn, then the set is referred to as a cycle. One of the diculties in nding optimal paths in the extended S -space will be in devising ways to deal with such cycles. It is possible to de ne the distance between any two nodes u and v as the minimum length of path joining the two nodes. Here path length is measured based on the number of nodes which comprise the path, and the shortest path is referred to as a geodesic. For a connected graph there is a path which connects each pair of nodes. In a connected graph V , the eccentricity e(v) is the distance to the node farthest from v. In other words, e(v) = maxfd(u; v)ju 2 V g. If we consider all possible eccentricities in a connected graph, the maximum value is referred to as the diameter of the graph. We will use the symbol D to represent the diameter so that D(V ) = maxfe(v ); e(v ):::e(vn)g. If we consider the perceptual signi cance of a limb path only in terms of the number of associated symmetry points, then in order to nd the most signi cant part in a cluster of symmetry points (that can be represented as a connected graph), we need to nd the path which corresponds to the diameter of the graph. In contrast to many graph optimization problems which require nding shortest paths, this requires nding the longest path in the graph, and as such, requires that all possible paths be examined [3]. Two standard approaches have been suggested for nding the diameter of a connected graph [3]. One method requires the compilation of an adjacency matrix for the graph. The diameter of the graph can then be computed by taking successive powers of the adjacency matrix. The second method involves an exhaustive search along all possible paths in the graph. In order to search for optimal paths in the extended S - space, we have chosen to use the latter method for the following reasons. By employing a thinning strategy to reduce the size of symmetry point clusters (see Section 3.5.1), 1
2
1
1
2
66 the resulting graphs have relatively few nodes and possible paths. Hence, in practice the exhaustive search required is not signi cantly expensive. Furthermore, the search method provides more exibility in terms of implementing the path constraints (see Section 3.5.2). We note that other parallel methods or neural networks [21] could be employed so that distances are propagated through a graph using nonlinear decisions at each node. Although these would provide a more elegant implementation, we have chosen to stay with the more `brute force' approach. This is because in the parallel case it is not clear how to deal directly with cycles in the graph. Our method for locating an optimal path is summarized in the following section.
Graph Search
A breadth- rst tree search is initiated at each symmetry point in the extended S -space. Search paths are traversed by ensuring that each pair of points is con gured in accordance with the continuity constraints described previously (see Section 3.5.2). The search continues until one of two situations occurs: either a path end is reached (the path cannot be extended any further), or the path intersects itself (thereby indicating a cycle). From any given starting point, the search proceeds in two opposing directions. When a complete path is found and the search has terminated in both directions, the number of path nodes is computed. By exhausting all search possibilities, it is possible for any two joined endpoints, to choose the geodesic having the fewest number of nodes. This gives the distance between the two. That path having the greatest distance is equivalent to the limb part having the greatest amount of physical support.
3.7 Summary Having de ned what kind of information is required to represent blob and limb parts, it was possible to develop a computational means for extracting such representations from raw grayscale image data. In the next chapter we will present results which demonstrate the various stages of this extraction process in order to validate some of the predictions made in this chapter, and to demonstrate the performance of an automatic part identi cation system.
67
4. RESULTS In this chapter, symmetry data extracted from several gray-scale images is used to demonstrate the predictions made earlier relating to the manifestation of object structural information in symmetry point data. We begin by examining the symmetry points that result from images of isolated blobs and limbs. Following this we will demonstrate the extraction of parts from both contrived and real images. We will show that we can identify object parts in practical imaging settings by employing the symmetry points extracted using annular operators.
4.1 Symmetry Point Distributions To demonstrate the symmetry point distribution for a blob, we have chosen the image of an apple shown in Figure 4.1(a). A set of edges extracted at a small scale are shown in Figure 4.1(b) to demonstrate the complexity of edge grouping. Based on the quantity of extracted edge data, it is possible to identify many enclosing structures which could potentially be interpreted as object parts. Since we use a multiscale edge extraction approach
(a)
(b)
Figure 4.1: Apple Image and Edge Map. (a) Apple image (384 288
pixels). Small scale edge features are extracted to produce the black points in (b). It is clear that large scale annular operators would produce a number of `false' enclosing structures having no relation to objects in the scene.
(see Section 1.1), small scale edge structure is overlooked when searching for large scale object structures. To demonstrate the validity of our prediction for the shape of the symmetry volume for a blob, we have employed an annular sampling having the following parameter
68 values: = 1:8, = 2:7, and = 1:1. In Figure 4.2, four adjacent layers within the sampling space are used to demonstrate the symmetry response associated with the apple. It is clear that as the S -space is traversed along the s dimension, the diameter of the symmetry point distribution points begins small, increases in size, and then decreases. Furthermore, the distributions remain approximately concentric. This result is consistent with the shape of the symmetry volume indicated in Section 2.2.1. There we predicted that blob symmetry point clusters would be constrained to fall within a surface formed by two circular cones joined at their bases (see also Figure 3.14). This characteristic signature is used as a basis for locating and representing blob structures in the images. We observe that the blob signature is clearly discernible even though the apple in not perfectly circular, and there is both internal and external structure present. Note that in this example, only those enclosing structures corresponding to dark objects on a light background are considered (see Appendix B). For the sake of visualization, this convention will be assumed in all other examples unless stated otherwise. Next we turn to the distribution of symmetry points associated with limb enclosures. In Section 2.3, we predicted that the shape of symmetry point clusters within a single layer would be constrained to fall within an pointed quadrilateral region. To demonstrate this, we consider the set of symmetry points which are found in a single layer in response to a pair of curved edge segments (see Figure 4.3). For two di erent levels of sampling resolution, we observe that the symmetry points are indeed constrained to fall within pointed quadrilateral regions as predicted. In the next example, we consider the set of responses obtained for a set of consecutive layers in an annular sampling space. Figure 4.4(a) shows a binary image which has been created to provide a simple example for demonstration. Figures 4.4(b) to (g) depict the resulting sets of symmetry points which are extracted for a set of six consecutive annular sampling layers. Although the sampling is relatively coarse, we observe that the symmetry point clusters within each layer are constrained to fall within pointed quadrilateral regions as predicted. Recalling Figure 3.16, we observe that there is also spatial overlap between the point distributions in any two adjacent scales. Also as predicted, the distributions become broader with an increase in scale. If we were to plot these points within the S -space, it is reasonable to suppose that they would be located within a volume similar to the one depicted in Figure 3.16. One further observation relates to the method's ability to detect curved limb enclosures. In Section 3.1.1, we derived a value for the maximum angular deviation parameter Dmax using a model where limbs were constructed from straight contours. In this example we observe that even in instances where contours are curved, the method is capable of detecting the associated symmetries. We recall the limb orientation value ^ which was intended to be used in the simpli cation of limb symmetry point clusters. An example showing the range of orientation values for a simple limb con guration is given in Figure 4.5 for two values of the maximum angular deviation Dmax. We observe that in general several di erent orientations can be associated with a given symmetry point. In a later simpli cation stage, we demonstrate how to reduce the data so that only a single orientation value is assigned to each symmetry point. We have chosen a very low value of to demonstrate the distribution of blob symmetry points in several adjacent layers. In practice, a higher value of is employed.
69
(a)
(b)
(c)
(d)
Figure 4.2: Distribution of Blob Symmetry Points. Results showing
four consecutive layers in the discretely sampled S -space where the scale s increases from (a) to (d). The edges used at each scale are shown in gray, with the symmetry points shown as white dots. Note how the annular sampling distance d changes between layers. Only those symmetry points exhibiting high enclosure values (e > 0:9) are shown. The spatial extent of the edge operators and the annular operators that were used for each layer are shown to the immediate right of each gure. We observe that the distribution of symmetry points is consistent with the predicted blob symmetry volume shown in Figure 2.9.
70
(a)
(b)
Figure 4.3: Symmetry Point Distribution for Curved Edges. (a) In order to demonstrate the extent of the symmetry point distribution at a high resolution, we have employed a very high value ( = 10). Individual points are dicult to discern due to the image resolution. However, the extent of the symmetry point distribution bounded by a pointed quadrilateral is clearly evident. (b) Symmetry points extracted at the same scale using more typical sampling with = 2:7. Symmetry points are now visible as white dots which are again observed to fall within a pointed quadrilateral region. One nal set of examples will be used to illustrate the variations which arise due to di erent annular sampling parameters. From Section 3.2 we recall that there are three parameters (, , and ) which specify the annular sampling structure [10]. We have observed in one previous example, how a variation in the parameter has the e ect of changing the spatial resolution of extracted symmetry points (see Figure 4.3). Higher values of correspond to a larger ratio between annular size and annular spacing. Hence, large values of result in a ner grain in the resulting symmetry point distributions within a given layer. The e ect of varying the parameter has also been shown indirectly in the example of Figure 4.2, and is obvious directly by its de nition. Here, by employing a small value of it was possible to increase the scalar resolution - thereby capturing the blob symmetry cross sections in several adjacent layers. We summarize the consequences of varying the parameters and as follows. Increasing the value of the parameter has the e ect of increasing the spatial resolution. Decreasing the parameter has the e ect of increasing the scalar resolution. We now consider the nal parameter which speci es the ratio between the outer and inner annular radii (see Figure 4.6). We would expect that for an increase in there would be a corresponding increase in the widths of symmetry point clusters that arise from annular sampling. We note that for small values of the parameter , adequate spatial resolution is required to capture all possible symmetric edge con gurations. The analysis of this relation We recall that by de nition, these parameters are not strictly independent due to the constraints imposed by the sampling structure. However, for the sake of demonstration, we will examine the e ects of each separately.
71
(a)
(b)
(c)
(d)
(e)
(f)
(g)
Figure 4.4: Symmetry Point Distributions at Di erent Consecutive Scales. (a) Binary image (378 277 pixels). (b)-(g) Symmetry points
(white dots overlayed onto original image) which result from applying annular sampling in six consecutive layers starting with the smallest scale in (b). The following set of parameters were used to de ne the sampling structure: = 1:8, = 2:7, = 1:4, and Dmax = 26 . Note how the sampling resolution varies with scale. As predicted, the distribution of points within each layer again falls within pointed quadrilateral regions. On close inspection we observe that there is spatial overlap between the distributions in any two adjacent layers. The presence of a second cluster in (f) is due to the enclosure at the right end of the object.
72
(a)
(b)
Figure 4.5: Limb Orientation Values. (a) Symmetry points have been
extracted at a single scale and are indicated by white dots. For each of these, there is a range of quantized orientations ^ at which enclosure has been detected based on a maximum deviation of Dmax = 18 when N = 20 (see Section 3.1.3). Each orientation is shown by an oriented gray line segment. We observe that in most cases the range of orientations covers two or three quantization levels. (b) For a larger value of Dmax = 45, more symmetry points are found and the range of orientations is generally greater than for the same points found in (a).
ship is discussed at length in [10]. Having employed several binary images to demonstrate the symmetry point distributions for limbs, we now turn to examples of gray-scale imagery. It will become clear that, while the symmetry point distributions are generally more complex, it will still be possible to identify characteristic limb clusters. The image of a bottle is shown in Figure 4.7 along with the resulting edge maps and symmetry data which were extracted for eight consecutive sampling layers. We note that for small scales, there is a signi cant quantity of symmetry data which relates to internal structures within the object (see layers 1 and 2). At larger scales, the majority of symmetry data relates to the gross shape of the object (see layers 7 and 8). Starting with such a set of raw symmetry data, the next computational stage involves extracting part representations. We can predict how such part information may be extracted by examining the symmetry data. From layers 7 and 8, it appears that the body of the bottle has the characteristic signature of a blob since the clusters are approximately circular and concentric. On the other hand, in layer 5 we see an example of a limb cluster signature corresponding to the neck of the bottle. In layers 4 and 5 there is also an additional blob signature corresponding to the top of the bottle. In terms of parts, at the coarsest level our method should produce three parts - speci cally, one large blob corresponding to the body of the bottle, one smaller blob corresponding to the bottle top, and a limb corresponding to the bottle neck. We will actually demonstrate this by simplifying the raw symmetry data using the methods developed in Section 3.5.1. In the next section we illustrate how the symmetry data is simpli ed to facilitate part identi cation and representation.
73
κ = 1.8 (a)
κ = 2.2 (b)
κ = 1.4 (c)
Figure 4.6: Variation in . For the object shown in Figure 4.4, three
di erent values of ((a)1.8, (b)2.2, and (c)1.4) are used to de ne the annular sampling structure. A representative annulus is shown to the right of each gure. From symmetry points cluster, it is clear that for large values of , the point distributions become quite wide (b). In the case of small values, the distributions narrow, and may become discontinuous (c).
74
8
7
6
3
5
2
4
1
Figure 4.7: Bottle Image. The set of symmetry points extracted by applying annular operators at eight scales (numbered 1-8 in the Figure) to the image (288 392 pixels) at the upper left. Symmetry points are indicated by white line segments whose orientation corresponds to the orientation of the enclosure at that point. The associated edge data are shown by isolated gray points. The shape and size of annular operators used for each is shown to the right of each gure, with the corresponding edge operator shown directly below. Note the scale dependent subsampling that takes place for both edge detection and symmetry extraction. Instances where there is a high variability in limb orientation within a group of symmetry points (see layers 5, 7 and 8) are examples of blob enclosure.
75
4.2 Cluster Simpli cation As discussed in Section 3.4, representations for a blobs can be derived based upon thinning processes applied to symmetry point clusters. In the previous section, the symmetry point distribution signature was demonstrated for an isolated blob (see Figure 4.2). This set of points corresponds to the symmetry point cluster associated with the principle blob in the image. The computational method used to simplify blob clusters requires that we rst locate a single extremum in the enclosure function e for each layer. See Figure 4.8 for the apple example(see Figure 4.1). Here we indicate the position of all the enclosure extrema for the same sampling layers shown previously (see Figure 4.8). We observe that the set of extrema has a maximum spatial deviation Sdmax (see Section 3.4) which does not exceed the spacing distance d used in the largest scale layer. This provides adequate evidence that these points are members of a blob symmetry point cluster. What remains is to choose a single point that will be used to represent the entire part. In the example shown in Figure 4.8, we nd that all enclosure values are equal (e = 1 in all cases). Since there is an even number of layers, the application of a smoothing lter will not identify a single enclosure extremumy. In this case we select the point from the smallest scale layer within the center of the range of applicable scales. For the example in Figure 4.8, since four scales are used, the center of the range is de ned by the points in (b) and (c). We therefore choose the point in Figure 4.8(b) to represent the entire part. Having demonstrated how a representation for a blob can be extracted from raw symmetry data, we next consider the same process for limbs. As outlined in Section 3.5.1, another thinning process may be used to reduce the number of symmetry points so that a single path-based representation remains. We begin by considering the simple binary image used previously in Figure 4.4. Each of the extracted symmetry points is located within the three-dimensional S -space, and a three-dimensional Gaussian smoothing kernel is applied at a variety of orientations (for details, see Section 3.5.1). To demonstrate the method, we rst examine in detail the operations performed on a single layer. We note, however, that in practise the computation is based upon information within a given layer i and within the neighboring layers i +1 and i ? 1. In Figure 4.9(a) we see the original set of symmetry points previously extracted at the second highest scale (corresponding to the data in Figure 4.4(f)). Here the limb orientations are shown as oriented line segments. A projection of the threedimensional Gaussian function used in the rst stages of thinning is shown in Figure 4.9(b). After applying a set of oriented Gaussian kernels, it is possible to select a single orientation at each symmetry point as shown in Figure 4.9(c). A thinning operation is then performed by preserving connectivity along the surface de ned by the orientation consistency function O (see Section 3.5.1). We observe that when compared with the original data in Figure 4.9(a), the result in Figure 4.9(e) is signi cantly simpli ed. For the sake of demonstration, we have chosen to show only those operations that are performed for a single layer. Although similar Recall that isolated extrema within a given layer are guaranteed since we apply local smoothing to the enclosure values. y In practice this is not a common occurrence. It happens in this example because we have chosen to use only four layers with a very small value of . If we had also considered data in other layers, there would likely have been non-maximal enclosure values (e 6= 1) which would have biased the distribution in favour of a particular point.
76
(a)
(b)
(c)
(d)
(e)
(f)
Figure 4.8: Blob Thinning Example. Each of the four symmetry point
distributions shown in Figure 4.2 is thinned down to a single point representing a local extremum in the enclosure function e. The result is shown in (a-d). A white dot indicates the location of the extremal operator center, and edges are shown in gray. The distribution of these centers projected into a common spatial plane is shown in (e) overlayed onto a bleached version of the original image. (f) The maximum spatial deviation (see Section 3.4) Sdmax is represented by the diameter of the black disc. White dots indicate the sampling grid used for operators in the largest scale layer which produced the result in (d). Since the spacing d is no greater than Sdmax, we conclude that negligible spatial variation exists for the points found in (a-d) and, therefore, that the distributions are concentric.
77
(b) (a)
(c)
(d) (e)
Figure 4.9: Limb Thinning Example. We consider the results of a thin-
ning operation applied to a single layer of symmetry data (see Figure 4.4(f)). (a) Limb orientation at symmetry points shown by oriented white line segments. Multiple orientations are indicated by fan shaped structures. (b) A set of oriented Gaussian kernels are applied to the data to select a single orientation value, and to provide a potential pro le on which thinning is performed. The two dimensional projection of a kernel representing one of N orientations is shown here. (c) After applying the kernels, the result yields a single orientation value at each symmetry point. (d) A detail showing the corresponding values of the orientation consistency function O for points in (c). Thinning is performed based on the values of O, resulting in the nal thinned form shown in (e).
78 operations are performed for symmetry data in all layers of the sampling space, we note that these operations are not layer independent. For a given layer i, the simpli cation operations performed will also depend on symmetry data in the two adjacent layers i ? 1 and i + 1. In the next section we will demonstrate how, using the set of simpli ed symmetry data, it is possible to identify parts within an image.
4.3 Part Extraction To demonstrate the identi cation of parts, we begin by examining results produced for a simple arti cial image. Following this we will show how blobs and limbs are identi ed in real images. Consider the 352 211 binary image shown in Figure 4.10 along with the set of symmetry data extracted from seven consecutive sampling layers. The parameters used in this sampling were = 1:8, = 2:7 and = 1:4. From the gure we see that symmetry data are distributed roughly within pointed quadrilateral regions within layers 1 through 4. We also recognize the characteristic distribution that signi es a blob enclosure within the round region at the left in layers 6 and 7. After simplifying the symmetry data, the set of thinned limb data is shown in Figure 4.11(a). The results for each of the layers are superimposed. By performing a search through all possible paths (which in this case is very simple, since there are not very many), it is possible to identify the limb structure having the greatest physical support as shown in Figure 4.11(b). In this simple example, this represents the only limb that is found. Blob structures are located using the same method demonstrated in Figure 4.8. Only one blob is found for e > 0:8, and this corresponds to the rounded region at the left of the shape. While still employing the same simple object, it is possible to demonstrate some of the appealing properties of our method when faced with problems such as fragmented contours or internal structure. In Figure 4.12, the same basic shape from the previous gure is shown, but now a curved internal structure has been added. This addition introduces the two problems that were discussed previously (see Section 3.1.2) - speci cally, internal structure and boundary fragmentation. While it is sometimes possible to use contour singularities such as corners or `T-junctions' to indicate the presence of contour fragmentation, we note that there are instances where such features are not present (see for example the right-most section of the object). Clearly, it would not be possible to employ a local boundary-based method to complete the object contour in such a case. Yet, using the annular sampling scheme, we can extract regional symmetry information which enables us to infer the global structural information independent of the contour fragmentation. In comparison with the previous Figure 4.10, the set of extracted symmetry points is noticeably more complex in appearance due to the presence of internal structures. When grouping is performed on this set of points, the results are also more complex than before. Speci cally in the case of limbs, we see that many additional structures are identi ed due to the internal structure (see Figure 4.13). The limb having the second highest physical support corresponds to the principal limb extracted in Figure 4.11. In order to construct a global representation for this object, it will be necessary to further analyze the parts obtained on the basis of the so-called perceptual signi cance. The topic of perceptual signi cance will be discussed in a future report.
79
1
2
3
4
5
6
7
Figure 4.10: Symmetry Points for a Binary Image. Image used to demonstrate the part extraction phase (upper left) followed by extracted symmetry points for layers 1 through 7 as labelled. Symmetry points are superimposed into the original image and are shown as white dots.
80
(b)
Layer (a) (c)
Figure 4.11: Part Extraction. (a) Thinned limb symmetry data, where each symmetry point is represented by a shaded square patch whose size and orientation is proportional to the associated scale and orientation of the enclosure. The results from layers 2 to 7 are gray coded to visually enhance the di erence between the structures arising at di erent scales (layer 1 is not shown due to resolution tolerances). We note that for the elongated section of the shape, the thinned data is consistent with our desired result. The thinning result in the rounded portion of the shape is more complicated as we would expect, since this corresponds to blob region. (b) The limb having the greatest physical support corresponds to the elongated section. Symmetry points along the associated path are represented by white squares which are scaled and oriented as above. No other limbs are found. (c) The blob having the greatest physical support is shown by a white circle whose size is proportional to the operator scale. Using an enclosure threshold of e > 0:8, no other blob structures are present.
81
1
2
3
4
5
6
7
Figure 4.12: Symmetry Point Extraction. Image used to demonstrate the part extraction phase (upper left) followed by extracted symmetry points for layers 1 through 7 as labelled. Symmetry points are superimposed onto the original image and shown as white dots. Ground intensity has been reduced to enhance the visibility of symmetry points.
82
Figure 4.13: Limb Identi cation. After thinning is employed, the set of
resulting symmetry points is shown in the top gure. Each symmetry point is represented by a lled square whose size is proportional to the symmetry point scale. The gures which follow depict the twelve most physically supported limbs in order of extraction from left to right, starting with the most supported part at the upper left. We note that the main limb that was previously identi ed in Figure 4.11 is the second most physically supported limb.
83 The result of extracting blobs from the example shown in Figure 4.12 is demonstrated in Figure 4.14. In this case, the part having the greatest physical support happens to correspond
(a)
(b)
Figure 4.14: Blob Identi cation. (a) Original image. (b) Darkened version of the original, with blobs overlayed as lled circles. Circles are coloured either white or gray to aid in visualization, and the size of circles is proportional to the scale at which symmetry was detected. Numbers indicate an ordering of part extraction based on physical support, such that blob 1 has the greatest physical support.
to the one extracted in Figure 4.13. We observe that the other blobs correspond to smaller scale structures within the object. Another example is used to demonstrate the method's robustness when faced with complicated internal structures on a binary image. In Figure 4.15(a) we see a checkered object on a white background. As we would expect, applying annular operators to this image produces a large number of responses spanning a range of scales. At small scales, the set of extracted limbs corresponds to the diagonal structures formed by the sets of connected black squares (see Figure 4.15(b)). We note that numerous blobs are also detected at these scales (not shown). These correspond to the centers of each black square. The object in Figure 4.15 has a small protruding limb which is successfully found within the tangle of limb responses at the smaller scales. At larger scales, the most physically supported limb structure corresponds to the major axis of the largest limb region (see Figure 4.15(c)). From this example, we observe that symmetric structures are identi ed at di erent spatial scales within the image. Using this set of extracted parts it will ultimately be possible to construct object representations based on the notion of perceptual signi cance. While the details of the method used for constructing such a representation will be discussed elsewhere, we can provide a rough sketch of the method here. For example, after identifying the limb in Figure 4.15(c) as the most perceptually signi cant part, all smaller scale structures falling within the corresponding limb region can be deleted from consideration. Following this we are left with only one other signi cant limb structure - namely the one corresponding to the small scale protrusion (see Figure 4.15(b)). This makes it possible to construct a coarse, intuitively appealing representation for the object in terms of its most predominant parts, while ignoring internal structural information. We consider the set of parts which are found for the previous Bottle image (see Figure 4.16). Again it is clear that part structures can be identi ed when there is internal
84
(a)
(b)
(c)
Figure 4.15: Checkered Object. (a) Original image. (b) Bleached ver-
sion of the original, with small scale limbs superimposed. Note the complexity of structures due to the diagonal arrangement of black squares. Each linear grouping of symmetry points is a separate limb. Note the single limb which corresponds to the lower protrusion. At small scales, blobs are found at the centers of each of the black squares (not shown). (c) At higher scales, the most physically supported limb de nes the principal axis of the object.
A
D
C
B
(a)
E
(b)
Figure 4.16: Bottle Parts. Limbs (a) and blobs (b) derived by grouping symmetry data extracted from the bottle image (see Figure 4.7). Speci c parts have been identi ed for discussion in text.
85 structure, as demonstrated by the limb B and the blob E . In Figure 4.16(a) we can also see how the e ects of the continuity constraints manifest themselves. Notably, limbs A and B are found to be separate limb parts due to a scale discontinuity found between the bottle body and its neck. In regions C and D the e ects of smoothness constraints are also evident. We observe that spatially continuous limb parts are segmented in two due to an orientation discontinuity (see black and white limbs in each case). Each of the parts labelled in Figure 4.16 would be considered as possible candidates when constructing an object representation since they all have high perceptual signi cance values (to be discussed in a future report). As a nal example, we have chosen an image of a natural scene to demonstrate how the method generalizes to more complex imaging settings (see Figure 4.17). In this example
(a)
(b)
(c)
Figure 4.17: Goat Image Example. (a) Original image. (b) Extracted
blobs superimposed onto the original image. Parts are indicated by lled circles whose size is proportional to the scale at which enclosure was detected. Numbers indicate the order in which blobs were extracted, starting with 1 corresponding to the part having the greatest physical support. (c) Extracted limbs. Parts are identi ed by lled squares whose size is proportional to the scale at which enclosure was detected, and whose orientation corresponds to the local limb orientation. Again, parts are numbered according to their relative physical support.
we observe many of the desirable properties that the annular symmetry detection method provides. For example, the large protruding rock structure (L4) is represented by a single limb in Figure 4.17(c) even though there is internal structure. The set of parts associated with the goat in the center are particularly signi cant since, if we consider the set of associated blobs and limbs together, we can conceive of a coarse representation for the overall object. In a future report, we will demonstrate how it is possible to construct this kind of representation, thereby producing a graph-based representation for objects in terms of their constituent parts. The fact that regional scale is used explicitly in the part identi cation process means that structures like the legs of the goat may be clearly distinguished from its larger body. This provides signi cant bene ts when attempting to derive representations for entire objects.
86
Summary
In this chapter we have demonstrated how the symmetric relationships between edge contours are manifested by sets of symmetry points extracted using annular symmetry operators. Speci cally, it was shown that for the two enclosure types considered (blobs and limbs), distinctive signatures can be found for the sets of extracted symmetry points. These signatures allow symmetry points to be grouped in order to identify blobs and limbs. Using both contrived (binary) and real images, we have demonstrated some of the desirable attributes that result from identifying parts in this way. The method makes it possible to extract parts in situations involving internal structure and when there is contour fragmentation. The fact that scale is employed explicitly in the extraction process also allows object structures to be separated based upon their regional scale properties. The set of parts extracted from natural images also provides a means of representing objects in an intuitively appealing way. Starting with a set of extracted parts, the method for constructing graph-based object representations will be presented in a future report.
87
Appendix A. Determining Limb Orientation Resolution When applying annular operators, it has been necessary to divide each annular region into a set of N equal sized sectors (see Section 3.1.1). The choice of N has a signi cant impact on the grouping of symmetry points since it limits the angular resolution of limb orientation values (see Section 3.1). In this appendix, we consider how to choose an appropriate value for N . To begin, consider the implications of choosing a value of N which is either very small or very large. In the case of small values, we would expect the resolution to be too coarse to be of any bene t during the grouping process. In other words, for two limbs whose symmetry volumes cross in the S -space it would not be possible to di erentiate the two structures since there is inadequate resolution in the extended S -space to separate the structures based on orientation. Therefore, if the value of N is set too low, there is no bene t gained by employing limb orientation values in the simpli cation process. In the second case, where N is chosen to be very large, we might assume this to be the ideal situation since the corresponding angular resolution would be high. In fact, we shall demonstrate that this is not the case. Recalling the method used to de ne paths within the extended S -space (see Section 3.5.2), by considering sets of three consecutive points at a time, there were only speci c con gurations of points that were valid. These con gurations are based upon both the spatial and orientation relationships between adjacent points. One important factor in identifying any three valid path points was that there be continuity in orientation when moving from one point to another. A smoothness test was performed by ensuring that the discrete orientation value at a given point di ered by at most one discretization level from a neighboring point along a given path. Clearly, if the resolution associated with limb orientation estimation is too great, it would no longer be possible to employ such a simple means for testing smoothness. We consider the upper limit on the angular resolution which still allows this simple smoothness test to be performed. Recall that the distance between any two symmetry points in a layer i is given by di . We have also determined that the maximum expected curvature for layer i is given by: max = r1 = d1 (A:1) i where is the sampling parameter specifying the ratio between ri and di . For a curve having curvature max, the change in tangent orientation along the curve between two points de ning a chord of length di (see Figure A.1) is given by: = 2 arcsin( di 2max ) = 2 arcsin( 21 ) (A:2) This change in orientation value corresponds to the largest orientation change that we would expect to nd for any curves detectable at the scale i. Thus, in order to ensure that we are able to detect such curves based on the smoothness assumption, we require that there be at most one orientation interval separating the two values and . We use the symbol = N 1
1
2
88 λ
di
γ 1
θ
κ max
Figure A.1: Orientation Change. For a curve having curvature max,
two symmetry points are located at the end points of a chord of length di. The limb orientation at these points is and . The change in orientation j ? j is equal to the angle as expressed by the relation sin( 2 ) = max2 di .
to represent the smallest angular interval resolvable when selecting N annular sectors. If we consider the number of angular intervals n = expressed as a real number n 2 : . Based on this constraint, a nal expression for N is given by: 1:5 (A:3) N < arcsin( ) 15
1 2
For a typical sampling with = 2:7, we compute the upper bound on the number of annular sectors to be N 25. In order to reliably di erentiate between overlapping limb structures in the extended S -space, it is advantageous to choose N as large as possible. Therefore, for this example we would select a value of N which is slightly less than 25 to achieve maximum resolution while at the same time satisfying (A.3).
89
Appendix B. Incorporating Edge Gradient Direction When constructing an angular pro le (see Section 3.1.1), we determine the compatibilities of edge elements within annular sectors. To compute this compatibility, we take into account the gradient direction associated with extracted edges. In doing so, it is possible to di erentiate between three fundamentally di erent types of enclosing structure (and hence parts). The two most common symmetric gradient con gurations occur when the gradient maintains the same direction relative to the inside of the object. Consider the closed boundary of an object. By traversing the boundary in one direction, the edge gradient along that boundary will always point in the same direction. This corresponds to either a dark object on a light background or a light object on a dark background (see Figure B.1). Previous method which have extracted symmetry either from edge maps [22] or directly from the gray-scale surface [9][4] have assumed that the imaged objects (or interesting regions) were represented by one of these two models. We nd, however, that while these examples represent some of the
(a)
(b)
Figure B.1: Objects with Constant Edge Gradient Directions. (a) A
dark object on a light background. The inset shows part of the contour falling within the region of an annular operator. The corresponding edge gradient directions (arrows) all point away from the center of the annulus. (b) A light object on a dark background where the corresponding edge gradient directions all point towards the center of the annular region.
objects found in imaging settings, they do not apply to a large class of objects. For example, one common situation occurs when there is a contrast reversal along the boundary. This often happens as a result of directed lighting. This phenomenon produces brightness on one side of an object and shadows (or darkness) on the other (see Figure B.2). Furthermore, with the addition of variable ground structure behind the object, it is dicult to make any general
90 assumptions about how the gradient direction might change as the occlusion boundary of an object is traversed.
(a)
(b)
Figure B.2: Objects with Edge Contrast Reversal. (a) An arti cial
image of an ellipsoid having uniform re ectance and illuminated by a directed light source. The gradient direction along the object boundary is clearly variable. (b) A natural image where the object is illuminated by the sun. Again the gradient direction around the bird's body is not constant.
Noting that many objects do not have a constant edge gradient direction along their boundaries, it is necessary to devise a computational means for dealing with these situations. Returning to Figure B.2(a), if we trace the edge gradient direction along the object's occluding boundary, it is clear that the value will change several times. In more complex images, when there is substantial external (ground) structure, it is not possible to predict any general principles that can be used to describe the edge gradient directions found along an object boundary (see Figure B.3).
Figure B.3: Object with Complex Ground Structure. This represents a very challenging problem for any existing computer vision approaches whether boundary or region-based.
To detect enclosing structures using annular operators, we will consider two possible approaches. In the rst, the edge gradient direction is actually employed when constructing an annular pro le. However, we make the distinction between symmetric structures with edge gradients pointing inwards or outwards relative to the annulus. In the second case, we do not employ the edge gradient direction when constructing the angular pro le. This makes it feasible to detect objects in settings like that shown in Figure B.3. However, there is an associated drawback. Since all possible symmetric structures are found, the quantity of
91 extracted symmetry data is in practice far greater than for the two simpler models. Therefore, it becomes necessary to deal with broader symmetry point clusters which are more dicult to isolate. An example demonstrating the result for an object when there is contrast reversal around the boundary is shown in Figure B.4.
(a)
(b)
Figure B.4: Contrast Reversal Example. (a) Original image. (b) The rst limb part identi ed (shown as a set of oriented white squares).
92
References [1] H. Blum. Biological shape and visual science (part I). Journal of Theoretical Biology, 38:205{287, 1973. [2] M. Brady and H. Asada. Smoothed local symmetries and their implementation. Technical Report A.I.Memo 757, Massachusetts Institute of Technology AI Lab, 1984. [3] F. Buckley and F. Harary. Distance in Graphs. Addison-Wesley, New York, 1990. [4] J.B. Burns, K. Nishihara, and S.J. Rosenschein. Appropriate-scale local centers: a foundation for parts-based recognition. In Proceedings of ARPA, 1994. [5] J. Canny. A computational approach to edge detection. IEEE Transactions on Pattern Recognition and Machine Intelligence, 8(6):679{697, 1986. [6] C. David and S. W. Zucker. Potentials, valleys, and dynamic global coverings. Technical Report TR-CIM-89-1, Center for Intelligent Machines, McGill University, Montreal, Canada, 1989. [7] C. R. Dyer and A. Rosenfeld. Thinning for gray-scale pictures. IEEE Transactions on Pattern Recognition and Machine Intelligence, 1(1):88{89, 1979. [8] M. Fleck. Local rotational symmetries. Technical Report A.I.Memo 852, Massachusetts Institute of Technology AI Lab, 1985. [9] J. M. Gauch and S. M. Pizer. The intensity axis of symmetry and its application to image segmentation. IEEE Transactions on Pattern Recognition and Machine Intelligence, 15(8):753{770, 1993. [10] M. F. Kelly and M. D. Levine. A sampling strategy using multi-scale annular operators. Technical Report TR-CIM-93-20, Center for Intelligent Machines, McGill University, Montreal, Canada, 1993. [11] M. F. Kelly and M. D. Levine. The symmetric enclosure of points by planar curves. Technical Report TR-CIM-93-1, Center for Intelligent Machines, McGill University, Montreal, Canada, 1993. [12] M. F. Kelly and M. D. Levine. Edge detection within an annular sampling framework. Technical Report TR-CIM-95-4, Center for Intelligent Machines, McGill University, Montreal, Canada, 1995. [13] B. Kimia. Toward a Computational Theory of Shape. PhD thesis, McGill University, Montreal, 1990. [14] L. Lam, S. Lee, and C. Y. Suen. Thinning methodologies - a comprehensive survey. IEEE Transactions on Pattern Recognition and Machine Intelligence, 14(9):869{885, 1992.
93 [15] F. Meyer. Skeletons and perceptual graphs. Signal Processing, 16:335{363, 1989. [16] S. Peleg and A. Rosenfeld. A min-max medial axis transform. IEEE Transactions on Pattern Recognition and Machine Intelligence, 3(2):208{210, 1981. [17] F. P. Preparata and M. I. Shamos. Computational Geometry. Springer-Verlag, New York, 1985. [18] A. Rosenfeld. Axial representations of shape. Computer Graphics and Image Processing, 33:156{173, 1986. [19] G. L. Scott, S. C. Turner, and A. Zisserman. Using a mixed wave/di usion process to elicit the symmetry set. Image and Vision Computing, 7(1):63{70, 1989. [20] D. W. Tank and J. J. Hop eld. Simple `neural' optimization networks: An a/d converter, signal decision circuitry, and a linear programming circuit. IEEE Trans. Systems Man and Cybernetics, 33:533{541, 1986. [21] S. Ullman. Low level aspects of segmentation and recognition. Phil. Trans. Royal Society of London (ser. B), 337:371{379, 1992. [22] Y. Yeshurun, D. Reisfeld, and H. Wolfson. Symmetry: a context free cue for foveated vision. In Neural Networks for Perception, volume I Human and Machine Perception, pages 477{491. Academic Press Inc., 1992. [23] S. W. Zucker, R. A. Hummel, and A. Rosenfeld. An application of relaxation labelling to line and curve enhancement. IEEE Transactions on Computers, C-26(4):394{403, 1977.
