a
UNIVERSITEIT GENT
Boundary Estimation for Multiply Digitized Objects Peter Veelaert and Robert A. Melter ELIS Technical Report DG 94-03 April 1994
VAKGROEP VOOR ELEKTRONICA EN INFORMATIESYSTEMEN ST.-PIETERSNIEUWSTRAAT 41 B-9000 GENT
1
Boundary estimation for multiply digitized objects Peter Veelaert1
Vakgroep ELIS Universiteit Gent, Belgium
[email protected]
Robert A. Melter2
Dept. of Mathematics Long Island University
[email protected]
Abstract. This paper considers the reconstruction of the boundary of a two-dimensional ob-
ject from one or more digital representations that have been obtained by successive displacements of a digital image sensor. We explain how the reconstruction can be done for known displacements as well as for unknown random displacements that are uniformly distributed. In the latter case we assume that we can solve a certain correspondence problem. The reconstructed boundary can be characterized by a sliding ruler property of which the well-known Lagrange interpolation is a special case. It is also shown that the reconstruction can be exact under certain circumstances. Although we only consider the reconstruction of functions of one variable, the technique can be extended to functions of two or more variables. The technique can also be applied to measurements generated by other digital sensors such as range sensors.
1 Introduction When a digital sensor is used to obtain a digitized representation of the contour or boundary of an object, information is inevitably lost due to the quantization e ect. By applying the well-known Sampling Theorem, we may be able to reconstruct some of the low-frequency information, but it will be impossible to reconstruct all the high-frequency information that is contained in the sharp turns and corners of the boundary [1]. There exist techniques however to obtain information that normally is beyond the theoretical limitations imposed by the Sampling Theorem. These techniques are also known as techniques for attaining superresolution or subpixel resolution. First, we can use the structural information contained in the image. For example, if a set of boundary pixels is recognized as 1 2
The authors are grateful for support by the NATO Collaborative Research Program. Robert Melter was also supported by a Faculty Research Award from Long Island University.
2
a straight line, then this information can be used to locate the original non-digitized line with subpixel accuracy [2, 3]. Second, we can combine the information from several measurements. By an early result of Fogel, we can improve the performance of a digital image sensor by not only digitizing the image function itself, but also by digitizing the derivatives of the image function [4]. Fogel's result has later been generalized, rst by Papoulis, then by Brown, to a multi-channel generalization of the Sampling Theorem [5, 6]. Recently, Ur and Gross have applied this generalized Sampling Theorem to obtain an image with improved resolution from a set of spatially shifted digital images [7]. In this paper we shall do both, that is, we shall use the structural information contained in the image, and we shall also use the information that is available from multiple spatially shifted images. In particular, we will assume that the boundaries that must be reconstructed are suciently smooth and that they can be represented by polynomials. For linear polynomials this amounts to assuming that the boundaries can be represented by digital straight lines. Furthermore, we will combine the information from several shifted images by taking averages of the discrete derivatives of these polynomials. As a result we nd a reconstruction curve whose average discrete derivative is equal to the average discrete derivative along the original boundary. This can also be expressed by a sliding ruler property. If we let a ruler slide along the original boundary, and if we let a similar ruler slide along the reconstructed curve, then both rulers will have the same average slope over an interval of unit length. Hence a sliding ruler property describes what kind of boundaries can be accurately reconstructed, and how the reconstruction actually proceeds. The advantages of the technique described in this paper are that it is simple, and that we can predict its performance by a sliding ruler property. Moreover, since we are taking averages, we do not have to know the exact relative displacements of the images; it suces that the displacements have uniformly random distributions, and that we can recover the rounded value of the horizontal displacement, which amounts to solving a correspondence problem. The paper is organized as follows. Section 2 describes the measurement setup, and gives a brief outline of the method that is used to reconstruct the contour of the original object. In Section 3 we start the investigation of the quality of the reconstruction by showing that it is exact for a restricted class of objects. Section 4 considers a more general class of boundaries for which we can still estimate the error although the reconstruction may no longer be exact. The most general case, where even the error can no longer be estimated, is discussed from a qualitative viewpoint in Section 5, which introduces a sliding ruler property. Finally, in Section 6 we discuss uniformly distributed random displacements. Although we will concentrate on the reconstruction of boundaries of two-dimensional digitized objects, our technique can be extended to digitized functions of more than one variable.
3
Figure 1: An object and its digital representation. By moving the rectangular grid relative to the object, we obtain several distinct representations.
2 Measurement setup In this section we describe the measurement setup that we shall use to reconstruct the original boundary from a set of digitized representations. In the most general case, we have a digital sensor, which provides us with discrete samples of a given function f . These samples are quantized, and we will assume that they are of either the form b f (k) c, or of the form d f (k) e, with k 2 Z, and where b c and d e respectively denote the oor operator and the ceiling operator. In fact, we shall state most results in terms of the oor operator, although it is clear that similar results will hold for the ceiling operator. The digital samples could be obtained by a range sensor or by a CCD camera. As an example, Figure 1 shows a digital representation of an object and the boundary of the object. The digital representation consists of those points of the object that are on the rectangular grid. The boundary of this object can be described by two functions; f2 (k) for the upper part of the boundary and f1 (k) for the lower part. Similarly, the digital representation of the boundary is described by the integral functions b f2 (k) c and d f1 (k) e. By moving the digital camera we can collect multiple digitizations of the object. We will assume that the camera is mounted perpendicular to the image plane, and that it can be moved to qm di erent positions in a plane parallel to the image plane. The rectangular grid, shown in Figure 1, is xed to the camera, and will move with it, while the object remains 4
stationary. Relative to some arbitrarily chosen reference point, the j th position of the camera is determined by the position vector ( j ; j ) 2 R2 . Let b f (k) c, with k 2 Z, be the digital representation of some part of the boundary when the camera is at position (0; 0). If we move the camera to position ( j ; j ), we obtain a new digital representation b f (k ? j ) ? j c. To reconstruct the original boundary from its multiple digital representations b f (k ? j ) ? j c, we shall use a polynomial Q(k) that satis es the following identity: qm ?1 m ?1 1 qX r b f (k ? ) ? c = 1 X r (Q(k ? ) ? ) : j j j j q q m j =0
m j =0
(1)
That is, we choose Q(k) such that the average value of its rth order di erence is equal to the corresponding average value obtained from qm digitized functions. We will see that all the coecients aj of Q(k) can be calculated from (1), except for a0 . The use of the above formula is based on the observation that it can reconstruct the original boundary function f (k) almost exactly when f (k) is a rational polynomial function, provided the displacement vectors are carefully chosen. In particular, we will see that when the boundary function f (k) has the form P (k)=q , there exists a set of displacement vectors such that qm ?1 m ?1 1 qX r b f (k ? j ) ? j c = 1 X r (f (k ? j ) ? j ) : q q m j =0
m j =0
(2)
In other words, for this particular kind of boundary function, the quantization noise due to the operators b c and d e can be removed by taking averages of di erences for carefully chosen camera positions. Once we have from (1) the average values of the di erences of Q(k), it is straightforward to calculate the coecients of Q(k). To this end, it is convenient to introduce falling factorial powers [8]. Let kj denote the j th falling factorial power of k; that is, kj = k(k ? 1) : : : (k ? j +1), for j 0. Then it is easy to show that kj = jkj ?1 . Thus taking di erences of falling factorial powers has the same e ect as taking derivatives of ordinary powers. In particular, let P Q(k) = nj=0 aj kj . Taking di erences we nd that r Q(k)jk=0 = r!ar . Thus, once the values of the di erences of Q(k) are known, the calculation of the coecients aj is straightforward for j 1. Note however that we cannot use these di erences to calculate the coecient a0 . In the next section, we will show that we often need a larger set of measurements to reconstruct a0. Although a set of measurements may be sucient to reconstruct the coecients aj for j 0, and hence the shape of the object, they may not be sucient to localize it exactly. In this paper we will concentrate on the reconstruction of shape. In fact, localization of the boundary will be assumed to be less important, since the relative displacements of the camera may even be unknown.
5
3 Exact reconstruction for rational polynomial boundaries In the previous section we have given an outline of a method that we can use to reconstruct a function f from its multiple digital representations. In this section we will start our investigation of the quality of the reconstruction. It turns out that, except for some small localization error, the noise due to the quantization operators b c and d e can be removed provided the function f has the form f (k) = P (k)=q , where P (k) is a polynomial with integral coecients. The noise can be removed by qm = q carefully chosen displacements of the camera. There are two di erent ways to choose the displacement vectors. For 0 j q ? 1, we can either use displacement vectors of the form ( j ; j ) = (0; ?j=q ), or we can use q displacement vectors of the form ( j ; j ) = (?j; 0). Under these particular circumstances (2) will hold. Note that since the displacement vectors of the second set are integral, the q digital representations can in this case also be obtained from a single digital image. The rst set of displacements will be called vertical displacements, while the second set will be called horizontal displacements.
Proposition 3.1 Let P (k) be an integer polynomial of degree n. For 1 r n, the following two identities hold :
?1 P (k) j 1 qX ?1 P (k) j 1 qX r + q = q r q + q q j =0 q j =0
(3)
?1 P (k + j ) 1 qX ?1 P (k + j ) 1 qX r r =
(4)
q j =0
q
q j =0
q
Proof. First, since we can reverse the order of the di erence and the summation operator, we nd that
qX ?1 j =0
qX ?1 qX ?1 r P (k) + j = r ( P (k) + j ? (P (k) + j ) mod q );
q
q
j =0
j =0
(5)
q
where we have used the identity b p=q c = (p?p mod q )=q . For j = 0; : : :; q ?1, (P (k)+j ) mod q lists all integers from 0 to q ? 1 exactly once, although not in order. Hence the right side of P (5) is equal to r ( (P (k) + j )=q ? (q ? 1)=2), and this yields (3). Second, we have qX ?1 j =0
r
P (k + j ) P (k + q ) P (k) r ?1 r ?1 = ? q q q = r?1 ( P (k + q ) ? Pq(k + q ) mod q ) ? r?1 ( P (k) ? Pq(k) mod q ) qX ?1 P (k + j ) = r q + r?1 P (k) mod q ? Pq (k + q ) mod q : j =0
Since P (k + q ) mod q = P (k) mod q , for any polynomial P (k), this yields (4). 6
2
If the boundary function has the form f (k) = P (k)=q , then, according to the above Proposition, we can eliminate the quantization operator b c, and thus reconstruct the original boundary from one or more of its digital representations. For example, from the digital representations we can calculate the values of the left side of (4). Once these values are known, it is straightforward to calculate the coecients aj of P (k). It suces to write the right side of (4) in terms of these coecients. Example. We will illustrate the use of (4). Suppose we have found the following digital representation of a function, which is known to have the form f (k) = P (k)=5, where P (k) is a polynomial of second degree:
b P (k)=5 c = 0; 0; 1; 3; 6; 9; 13 for k = 0; 1; : : :; 6: Hence b P (k)=5 c = 0; 1; 2; 3; 3; 4, and 2 b P (k)=5 c = 1; 1; 1; 0; 1. Applying (4) we nd that P P
4 P (k)=5 = 9; and 4 2P (k)=5 = 4: To calculate the coecients, we let P (k) = j =0 j =0P P a2 k2 + a1 k1 + a0 . We nd that 4j =0 P (k) = P (5) ? P (0) = 20a2 + 5a1 , and 4j =0 2 P (k) = P (5) ? P (0) = 10a2 . Solving for a2 and a1 yields a2 = 2 and a1 = 1. Hence P (k)=5 = (2k2 + k1 + a0 )=5: A value for a0 can be obtained from the inequality P (k)=5 ? 1 < b P (k)=5 c P (k)=5. We nd that either a0 = 2 or a0 = 3. In fact, both values give rise to the same digital
representation.
Remark. As mentioned before, the formulas (1) and (2) do not yield a value for a0. In fact,
the diculties in determining a0 are related to the solvability of congruence relations. It is easy to show that when the congruence P (k) + 1 0 (mod q ) has no solution for k, then b (P (k) + 1)=q c = b P (k)=q c holds for all k 2 Z. That is, if the above congruence has no solutions, then the curve (P (k) + 1)=q will never cross the vertical grid lines in a grid point. Therefore, the curve has the same digital representation as the curve P (k)=q , and it follows that at least two values for a0 give rise to the same digital representation. As a result the coecient a0 cannot be uniquely determined. Note however that, once that coecients aj are known for j 1, we can always determine an interval of possible values for a0 , by applying the relation P (k)=q ? 1 < b P (k)=q c = b f (k) c P (k)=q . Depending on the solution set of the above congruence relation, the interval may or may not consist of a single integer. The width of the interval depends on the solvability of congruence relations, and is therefore dicult to predict.
4 Error analysis for arbitrary polynomial functions In the previous section we have shown that for certain cases the quantization noise can be removed. In particular, we assumed f (k) to be equal to a rational polynomial whose denominator q is known exactly. In this section we will start generalizing this by considering functions of the form f (k) = Q(k), where Q(k) may be an arbitrary rational polynomial. In fact, any rational polynomial can always be written as Q(k) = P (k)=q , where q is the common denominator of 7
the coecients of Q(k). In this section however, we will assume that q is unknown. It follows that the number of measurements qm will di er from the unknown value of the denominator q . As a result the reconstruction of f (k) from b f (k) c will no longer be exact. Nevertheless, we can show that there is an upper bound for the error. The following proposition is proved in the Appendix.
Proposition 4.1 Let Q(k) = P (k)=q be a rational polynomial of degree n. For 1 r n, and qm 1, the following identities hold : m ?1 1 qX r Q(k) + j
qm j =0
qm
qX m ?1 q ) r?1 r (Q(k) + qj ) + (qm mod 2 ; = q1 q m j =0
m
m
qX m ?1 m ?1 r?1 1 qX r b Q(k + j ) c = 1 r Q(k + j ) + 2 ;
qm j =0
where is a real number satisfying jj < 1.
qm j =0
qm
(6) (7)
Proposition 4.1 gives the error bounds for the di erences of the polynomial Q(k) = P (k)=q when we use an arbitrary denominator qm to reconstruct these di erences from the di erences of b Q(k) c, instead of the real denominator q , which is unknown. Clearly, when we try to reconstruct the coecients of Q(k) from these approximated di erences we will obtain only approximations to the coecients. Since we may be more interested in error bounds for the coecients, than in the error bounds of the di erences, we examine the calculation of the coecients now in more detail. First, we consider the application of (6). If we let Q(k) = a0 + + an kn , then we have r Q(k)jk=0 = r!ar . From (6) it immediately follows that
m ?1 1 qX qm mod q r?1 r Q(k) + j 2 : ? qm r! j =0 qm k=0 r!qm Clearly, (qm mod q )=qm will approach zero when the number of measurements qm gets large. Hence, for every coecient ar , with r 1, the accuracy of the approximation will be enhanced when qm gets large. Example. For the polynomial 2k2=7+3k1=7, Figure 2 shows the approximated values of the coPqm ?1 ecient a1 . It shows, for qm ranging from 1 to 40, the values of 1=qm j =0 b Q(k) + j=q ck=0 , which are approximations of a1 . Figure 2 also shows the error intervals. According to Proposition 4.1, the approximation for a1 must lie in the region that is bounded by y = a1 (qm mod q )=qm . Note that the error is zero when qm is a multiple of q . It is also important to note that we would get a similar picture for the estimated values of a2 . Next we consider the use of (7) to reconstruct the coecients of the polynomial Q(k). Explicit calculation shows that
ar =
r?1 Q(k + qm ) k=0 ? r?1 Q(k) k=0 = 8
n X
j =r?1
j ?r+1
j r?1qm
aj :
1.5 1. 0.5 0. -0.5 10.
20.
30.
40.
Figure 2: The estimated values and the error bounds for the coecient a1 = 3=7 for qm increasing. Let wr denote the left side of this expression. Then, for a polynomial of second degree, we have w2 = 2qm a2 , and w1 = qm a1 + qm (qm ? 1)a2, from which the coecients a1 and a2 can be calculated. Let wj and aj denote an upper bound for the error that can occur on wj and aj respectively, for a given choice of qm . Then, since we must add the error bounds, we nd that a2 = w2=2qm , and a1 = w1 =qm + (qm ? 1)a2. According to Proposition (4.1), we have w2 = 2, and w1 = , and this nally yields a2 = =qm , and a1 = . Note that the error for a2 propagates to a1, and, by consequence, the error bound for a1 does not decrease for increasing values of qm , but instead remains xed. More generally, because of the propagation of errors, for a polynomial Q(k) of degree n, the error bound on the coecient ar depends on the degree of the polynomial. For example, for a polynomial of third degree a similar calculation shows that a3 = 2=3qm , a2 = , and a1 = (2qm2 + 1)=3qm. Example. Figure 3 shows, for increasing values of qm, the approximations for the coecient a3 = 3=8 of the polynomial 3=8k3 + 5=8k1. It also shows the error bound for a3 , which according to Proposition 4.1 is equal to 2=3qm . By contrast, Figure 4 shows that the error on a1 = 5=8 grows like (2qm2 + 1)=3qm for increasing values of qm . Note, however, that this does not necessarily mean that we will get inferior approximations for the polynomial Q(k). Actually, when qm gets larger, the decreasing estimation error of the most signi cant coecient a3 may compensate for the increasing error of a1 . The precise nature of this approximation will be made more clear in the next section where we will discuss a sliding ruler property.
9
1. 0.8 0.6 0.4 0.2 0. -0.2 10.
20.
30.
40.
Figure 3: The estimated values and the error bounds for the coecient a3 = 3=8 for qm increasing.
20. 10. 0. -10. -20. 10.
20.
30.
40.
Figure 4: The estimated values and the error bounds for the coecient a1 = 5=8 for qm increasing.
10
4. 3. 2. 1.
0.
4.
2.
6.
8.
Figure 5: One representative of a collection of digitized representations of an object.
5 Quality of reconstruction in the general case In the previous two sections we have seen that we can get good approximations for the coecients of boundary functions that can be represented by rational polynomials. In this section we will look at arbitrary boundaries. We will see that, for vertical displacements, the reconstructed curves are related to Lagrange interpolation. For horizontal displacements the reconstructions satisfy a so-called sliding ruler property. Furthermore, the observed periodicity of the error bound for the coecients, which is due to the occurence of the term qm mod q (see Figure 2), also reappears in this more general setting.
5.1 Vertical displacements and Lagrange interpolation Consider the digitized object of Figure 5. This object is the result of the digitization of the region de ned by the equation ((x ? p)2 + (y ? q )2 + a2 ) ? b4 ? 4c2(x ? p)2 0, where we have set a = 1:81, b = 1:9, c = 2:5, p = 4:14, and q = 2:2. This object was deliberately chosen because its boundary cannot be approximated well by polynomials of low degree. Now suppose we have not one but 10 di erent digital images of this object, where the j th image has been obtained by moving the camera vertically over a distance (j ? 1)=10 times the unit distance. Then, by using (1), we can reconstruct parts of the boundary of the object by polynomial curves. In this case we have performed a reconstruction for the lower part and one for the upper part of the boundary, where we have used polynomials of eighth degree. The resulting reconstructions are shown in Figure 6, together with the real boundary of the region, and one of its digital representations with the camera at position (0; 0). Figure 6 clearly shows that the reconstructed polynomial curves are not very good approx11
4. 3. 2. 1. 0. 0.
2.
4.
6.
8.
Figure 6: Boundary reconstruction by a large number of vertical displacements, which amounts to Lagrange interpolation. imations of the original boundary. However, they are close to the intersection points of the real boundary with the vertical grid lines. Note that the intersection points with the vertical grid lines are well de ned since we are moving the camera in the vertical direction, which leaves the vertical grid lines at a xed position relative to the object. Actually, the polynomial curves that we have found are close approximations of the Lagrange interpolation of the vertical intersection points. One can show that this must be valid in the general case. Let (ai ; bi) be the coordinates of the intersection points with the vertical grid lines. Then we can always nd very good rational approximations (pi ; qi ) (ai; bi). Consider a new object whose boundary coincides with the Lagrange interpolation of the points (pi ; qi ). Then by moving the camera in the vertical direction by a nite number of steps, the new object will give rise to exactly the same digital representations as the original object. Furthermore, the boundaries of the new object are rational polynomial curves. Hence, as shown in Proposition 4.1, if the number of measurements increases, the approximating curves will approach these rational polynomials. Therefore, they will approach the Lagrange interpolation of the vertical intersection points as shown in Figure 6. Thus by moving the camera we can get accurate information about the position of the intersection of the boundary with the vertical grid lines. Using (1) we can retrieve this information in an indirect way from the Lagrange interpolation of the intersection points. The polynomials may even be good approximations for the real boundary provided that this boundary can be approximated well by the Lagrange interpolation of its vertical intersection points. For a circle this is indeed the case; Figure 7 shows two approximating polynomials of third degree, which are the result of combining the digital images of 16 di erent camera positions. Note that we obtain a good reconstruction of part of the boundary, although it is not clear at all from each 12
4. 3. 2. 1. 0. 0.
1.
2.
3.
Figure 7: Reconstruction of the boundary of a circular object by vertical displacements. individual digital representation that the object is a disk.
Dominance of rational approximations. Above we have used the fact that we can always nd good rational approximations for real numbers. In fact, real numbers almost always have good rational approximations with small denominator. Hence the Lagrange interpolation also has a good rational approximation with small denominator. Suppose for example that the Lagrange interpolation polynomial is equal to P (k) = (2k(k ? 1) + 4k)= , where was chosen because of its irrationality. Nevertheless, has good rational approximations with small denominator such as 22=7. As a result, when we reconstruct the coecients of the above polynomial, we will nd the same periodicity pattern that we have found for rational polynomials, for values of qm that are not too large. In other words, for not too large values of qm , 22=7 dominates all the other approximations for , and the calculated values for the coecients will converge in the same way as if we had reconstructed P (k) = (2k(k?1)+4k)7=22. This e ect can be seen clearly in Figure 8, where we have good approximations for qm a multiple of 11.
5.2 Horizontal displacements and sliding ruler property If we use horizontal displacements of unit step, that is ( j ; j ) = (?j; 0), then, according to (1), the polynomial Q(k) that we use to approximate the digitized function b f (k) c satis es 1 r?1 Q(q ) ? r?1 Q(0) = 1 r?1 b f (q ) c ? r?1 b f (0) c : (8) qm
m
qm
m
Since the value of 0 is undetermined by (8), we may choose 0 = Q(0) = b f (0) c. For r = 1, it follows that Q(qm ) = b f (qm ) c. Hence the polynomial curve Q(k) will pass through both 13
1.25 1.2 1.15 1.1 1.05 1. 10.
20.
30.
40.
Figure 8: Estimated values for the rst order coecient of a real polynomial, for qm increasing. The exact value is 4= 1:27324. Note that 4= ? 14=11 0:0005. the endpoints of the digitized boundary represented by b f (k) c. Furthermore, one can easily verify that (8) implies Q(qm + k) ? Q(k) = b f (qm + k) c ? b f (k) c ;
for 0 k r ? 1. Therefore, if we consider pairs of points that are on vertical grid lines spaced qm apart, then, by construction, the di erence between the heights of the points will be the same for the polynomial reconstruction as for the digitized boundary. This is illustrated in Figure 9. The boundary of an object has been reconstructed by polynomials of degree 7, with qm = 2. Hence we have Q(k +2) ? Q(j ) = b f (k + 2) c?b f (2) c, for 0 k 6. Actually, we can connect two points on the curve Q(k) by a ruler such that the endpoints of the ruler are at a horizontal distance qm = 2 from one another. Then, as shown in Figure 9, if we let this ruler slide along the curve Q(k), each time that its endpoints are at vertical grid lines, the ruler will be parallel to a similar ruler that is sliding along the digitized boundary b f (k) c. Figure 10 shows the reconstruction with polynomials of fourth degree and a horizontal ruler length equal to 5. In this case the reconstructed curves turn out to be horizontal straight lines. One can verify however, that the horizontal line segments indeed satisfy the sliding ruler property; a ruler with length equal to 5 sliding along the digital representation remains horizontal all the time. Obviously, to obtain a good reconstruction the ruler must be of the right size. A long ruler, such as in Figure 10, may be too long to detect the small scale features of the boundary. Alternatively, we may use a short ruler with qm = 1. Then it is easy to see that, according to the sliding ruler property, the reconstructed curve must pass through all the digitized points 14
4. 3. 2. 1. 0. 0.
2.
4.
6.
8.
Figure 9: Reconstruction of the boundary of an object by horizontal displacements. Two parallel rulers are shown: one sliding along the reconstructed boundary and one sliding along the digital representation of the original boundary.
4. 3. 2. 1. 0. 0.
2.
4.
6.
8.
Figure 10: Reconstruction for a ruler of length 5.
15
4. 3. 2. 1. 0. 0.
2.
4.
6.
8.
Figure 11: Reconstruction for a ruler of length 1. This amounts to Lagrange interpolation for the digitized boundary. of the boundary. This is shown in Figure 11, where the polynomials have degree eight. Thus we obtain the Lagrange interpolation for the digitized points of the curve. Again this is not a good reconstruction of the original non-digitized curve. In fact, for rational polynomial boundaries, the best choice of the ruler has been discussed in the previous sections. We have shown that for a rational polynomial curve with denominator qm , the curve can be reconstructed exactly from its digital representation if we also use a ruler with horizontal length equal to qm . More generally, for arbitrary curves, the quality of the reconstruction with a given ruler length qm , will be related to the existence of a good rational approximation with denominator qm . Note that the Lagrange interpolation illustrated in Figure 6 can also be described by the sliding ruler property. We then have a ruler of length qm = 1 following the original boundary, in contrast with Figure 11 where the ruler follows the digitized boundary. In Figure 6 the ruler is able to slide along the points on the original boundary because their position has been determined by the vertical displacements.
Horizontal subpixel displacements. Thus, one way to nd a good reconstruction, pro-
vided it exists, is to employ a ruler of the right size. Unfortunately, the right size may not be known. There is however a second way to increase the quality of the reconstruction. To this end we will now use horizontal subpixel displacements of the form ( j ; j ) = (?j=dm; 0), where dm is also a parameter of the measurement. Let Q(k) be a polynomial satisfying dm ?1 qmX dm ?1 1 qmX r f (k + j ) = 1 r Q(k + j ) qm dm j =0 dm qm dm j =0 dm
16
Hence we perform qm dm digital measurements. The above identity can be rewritten as 1 Pdm ?1 1 ?r?1 b f (k + qm + j=dm) c ? r?1 b f (k + j=dm) c = dm j =0 qm 1 Pdm ?1 1 ?r?1Q(k + qm + j=dm) ? r?1 Q(k + j=dm) ; dm j =0 qm which can also be interpreted as taking average values of the left side and the right side of (8), where the average is taken over dm points in the unit interval. As before, the above identity implies that dm dm X 1 1 X (Q(qm + k + j=dm) ? Q(k + j=dm)) = d (b f (qm + k + j=dm) c ? b f (k + j=dm) c) : d m j =0
m j =0
For pairs of points spaced qm apart, the average di erence between the heights of the points will now, by construction, be the same for the polynomial approximation as for the digitized boundary. An example is shown in Figure 12, with approximating polynomials of degree 7, and where qm = 1 and dm = 64. The high value for dm was chosen to illustrate the limiting case, when the displacement step 1=dm becomes arbitrarily small. When dm approaches in nity, the average slope of a ruler sliding back and forth over one unit interval, k x < k + 1, along the reconstructed curve will be the same as the average slope of a similar ruler sliding along the curve b f (x) c. Put di erently, by moving the camera horizontally by very small steps, we obtain subpixel information about b f (x) c, for x 2 [k; k + 1[ [ [k + qm ; k + 1 + qm [. This information is conveyed to the reconstruction in an indirect way by a ruler that is sliding back and forth on the staircase shaped curve b f (x) c (or d f (x) e for the lower part of the boundary), which is also shown in Figure 12. The reconstructed polynomial curve is then de ned in such a way that the average slope of the ruler along the staircase is equal to the average slope of a ruler sliding along the reconstruction.
Combining horizontal with vertical displacements. A last step in increasing the quality of the reconstruction can be attained by combining subpixel horizontal displacements with (subpixel) vertical displacements. We choose Q(k) such that it satis es dm ?1 X em 1 qmX r f (k + j ) + i = r Q(k + dj ) + ei : (9) qm dm em j =0 i=0 dm em qm dm em j =0 i=0 m m We now have qm dm horizontal displacements and em vertical displacements of the camera. Using a similar analysis, we now nd that for em large enough, the average value of the ruler will be the same as for a ruler sliding along the original boundary f (x) of the object, instead of sliding along the staircase b f (x) c. The additional information has been determined by the vertical displacements. This is illustrated in Figure 13, for qm = 1, and dm = em = 8. By moving the camera horizontally and vertically by very small steps, we obtain subpixel information about
1
qmX dm ?1 X em
17
4. 3. 2. 1. 0. 0.
2.
4.
6.
8.
Figure 12: Reconstruction by horizontal subpixel displacements with a ruler of length 1. The ruler sliding along the reconstruction takes its average slope from a similar ruler sliding along the staircases.
4. 3. 2. 1. 0. 0.
2.
4.
6.
8.
Figure 13: Reconstruction by horizontal subpixel displacements as well as vertical (subpixel) displacements for a ruler of length 1. The ruler sliding along the reconstructed boundary takes its average slope from a similar ruler sliding along the original boundary.
18
b f (x) + i=em c, hence of f (x) when em gets large. Again this information is conveyed to the
reconstruction, but now by a ruler that is sliding back and forth on the original boundary f (x). Figure 13 shows that we obtain a good reconstruction for the boundary of the left side of the object. Gradually the error will become larger when we move to the right side of the object. This is due to fact that there is no good polynomial approximation of seventh degree for the entire upper or lower part of the boundary. On the contrary, if we try to improve the seventh degree reconstruction at the left side of the object, it will get worse at the right side, due to the particular shape of the object.
6 Uniformly distributed random displacements The reconstruction technique described in the previous sections proceeds by taking averages of di erences. Hitherto the displacements of the camera were deliberately chosen at well de ned regular positions. In this section we will consider the reconstruction from randomly distributed displacements, which are not known precisely. In particular, for a given displacement vector ( j ; j ), we will assume that we can only recover the integral part of j . We then choose Q(k) such that it satis es Z 1 m ?1 1 pX r b f (k ? f g) ? c = r Q(k + x) dx; (10) pm j =0
j
j
0
where f j g denotes the fractional part of j . Before taking averages, every digitized function b f (k ? j ) ? j c has been translated to b f (k ? ( j ? b j c)) ? j c, which is possible because the value of b j c can be recovered. It is not necessary to know the value of b c, since we are taking di erences, and r b f (k ? f j g) ? j c = r b f (k ? f j g) ? f j g + b j c c = r b f (k ? f j g) ? f j g c, for r 1. The sum at the right side has been replaced by an integral sign; we now take the average value of r Q(k + x) over an in nite number of displacements x, uniformly distributed over the interval [0; 1]. Again the coecients of Q(k) can be calculated by substituting Q(k) = a0 + + an kn into (10). The above reconstruction with random displacements will approach the reconstruction by known horizontal and vertical displacements as follows. We assume that f j g and f j g are uniformly distributed over the unit interval. Then, for the number of measurements pm increasing, the left side of (10) will approach the limit of the left side of (9), for qm = 1 and dm ; em ! 1. Similarly, the right side of (10) is equal to the limit of the right side of (9), for qm = 1 and dm ; em ! 1. Hence the reconstruction by random displacements corresponds to a reconstruction with ruler length qm equal to 1. The result is shown in Figure 14 for 10 digitized images obtained by random displacements both in the vertical and horizontal direction. The boundary of the left side of the object has been reconstructed by polynomials of third degree. Obtaining an accurate reconstruction of a larger part of the boundary by polynomials of higher degree, would require more random 19
4. 3. 2. 1. 0. 0.
2.
4.
6.
8.
Figure 14: Reconstruction of part of the boundary of an object by random displacements in the vertical and in the horizontal direction. measurements. For a very large number of measurements we would obtain the result shown in Figure 13. In (10) we consider vertical as well as horizontal displacements. Random displacements that are con ned to the vertical direction, can be handled by simply replacing the right side of (10) by r Q(k). For displacements con ned to the horizontal direction, (10) remains valid. Figure 15 shows the reconstruction of the boundary of a disk from 5 di erent digital images, where the camera has only been moved by uniformly distributed vertical displacements. If we increase the number of image samples, the reconstruction will become more and more resemblant to the result shown in Figure 7.
The correspondence problem. For random displacements that have non-vanishing horizontal components, we assumed b j c to be known. However, even if this value is not known,
we may be able to recover it by solving a correspondence problem. For example, in Figure 14 we may estimate the horizontal displacement from the position of the leftmost point of the digitized object. That is, we may say that b j c is equal to the horizontal component of the displacement of the leftmost point of the digitized object, which is integral. This corresponds to a coordinate system that is chosen such that, for j = 0, the vertical axis is a left tangent of the boundary of the object, although the precise shape of the boundary does not have to be known. An error will be made when the boundary has moved slightly across a vertical grid line while the leftmost point of the digital image is still on a grid line further to the right. This e ect is shown in Figure 16; for this particular object it will occur for less than 5% of the samples. For example, to obtain the reconstruction of Figure 14, the correspondence problem 20
4. 3. 2. 1. 0. 0.
1.
2.
3.
Figure 15: Reconstruction of the boundary of a circular object, by random displacements in the vertical direction. Only 5 digital representations were used. was correctly solved for each of the 10 images. More elaborate correspondence algorithms can be used to further reduce the error rate. If we restrict ourselves to horizontal displacements, then, for the object of Figure 14, the correspondence problem can even be solved without errors by considering the leftmost point on a given horizontal grid line. For displacements con ned to the vertical direction, the correspondence problem can be simply avoided by choosing j = 0 for all samples.
7 Conclusion We have found that it is possible to improve the reconstruction of the boundary of an object by combining several spatially shifted digital images. The quality of the reconstruction depends both on the properties of the original boundary and on the displacements that are made to obtain the images. This has been made more precise in several ways. For boundaries de ned by rational polynomials we have shown that the reconstruction either is exact or at least the error can be estimated; For arbitrary boundaries the reconstruction can be described qualitatively by the sliding ruler property. This property can be used as a guideline for what kind of displacements we have to make to reconstruct the boundary of a given object. The sliding ruler property appears in several forms. For displacements con ned to the vertical direction, it amounts to the Lagrange interpolation of a nite set of points on the original boundary. For unit step displacements in the horizontal direction, the reconstruction is related to the slope of a ruler sliding along the digitized boundary, while for subpixel displacements in the horizontal direction the reconstruction is determined by the average slope of 21
Figure 16: The correspondence problem is not correctly solved for this case. The original boundary crosses a vertical grid line, but this cannot be deduced from the digital representation. This occurs for less than 5% of the cases. the ruler. Finally, for subpixel displacements in both directions, the reconstruction is related to the average slope of a ruler sliding along the original boundary. Reconstruction is not only possible when the displacements are well known, but also when they are distributed randomly. In the latter case we assume that the distribution is uniform, and that we can solve a correspondence problem. In this paper we have used a simple technique to solve this problem with a 5% error rate for a given type of object. More sophisticated methods may lead to lower error rates, but their discussion is outside the scope of this paper.
Appendix Proof of Proposition 4.1. Let q0 denote the greatest multiple of q not greater than qm; that is, q 0 = qm ? qm mod q , or equivalently, q 0 = b qm =q c q . First we consider the digitized function b Q(k) + j=qm c. By varying j the original function Q(k) + j=qm will change, and its digitized representation will change accordingly. However, since the polynomial Q(k) = P (k)=q can only take rational values that have denominator q , and because 0 j=qm < 1, for every j there will be a j 0 such that j0 j = Q(k) + q ; (11) Q(k) + q m
for all k. Now we compare the sum
qX m ?1 j =0
r Q(k) + j
qm
22
(12)
with
qX ?1 j =0
0 r Q(k) + j
(13)
q
0
According to (11), every term that appears in the sum (12) will also appear in (13). In fact, every term in (12) appears either b qm =q c times or b qm =q c + 1 times in (13). More precisely, one can show that r b Q(k) + j=qm c appears b qm =q c + 1 times in (13) when either j = 0 or jqm mod q > q ? qm mod q . Thus, we have qX m ?1 j =0
r Q(k) + qj
m
qX ?1 0 X r Q(k) + jq + r Q(k) + qj = qqm
j =0
m
j 2S
0
;
where S is the set of indices j that satisfy either j = 0 or jqm mod q > q ? qm mod q . By P Proposition 3.1, it follows that the right side is equal to q 0r Q(k) + j 2S r b Q(k) + j=qm c : This can be rewritten as
X X j j qm r Q(k) ? (qm ? q 0 )r Q(k) + r (Q(k) + ) ? r Q(k) + ; qm j 2S qm j 2S
where fxg denotes the fractional part of the real number x; that is, fxg = x ? b x c. Since we have r (Q(k) + j=qm ) = r Q(k), and jS j = qm ? q 0 , the two terms in the middle cancel. Moreover, we have jr fQ(k) + j=qmgj < 2r?1 , which yields (6). For the second kind of displacements, we have qX m ?1 j =0
r b Q(k + j ) c =
qX ?1 0
j =0
r b Q(k + j ) c +
qX m ?1 j =q
r b Q(k + j ) c :
0
By a slight generalization of Proposition 3.1, one can easily show that qX ?1 0
j =0
r b Q(k + j ) c =
qX ?1 0
j =0
r Q(k + j ) = r?1 Q(k + q 0 ) ? r?1 Q(k):
Hence we have
qX ?1 0
j =0
r b Q(k + j ) c = r?1 Q(k + q
m
) ? r?1 Q(k) ?
qX ?1 j =q
r Q(k + j ):
0
Substituting this result in (14) yields qX m ?1 j =0
Since
r b Q(k + j ) c = r?1 Q(k + q
qm ?1 X j =q0
m
) ? r?1 Q(k) ?
qX m ?1 j =q
r fQ(k + j )g :
0
r fQ(k + j )g = r?1 fQ(k + qm )g ? r?1 Q(k + q 0 ) < 2 2r?2;
we nd that (7) holds.
2 23
(14)
References [1] C. Shannon, \Communication in the presence of noise," Proc. IRE, vol. 37, pp. 10{21, 1949. [2] L. Dorst and A. Smeulders, \Discrete representation of straight lines," IEEE Trans. Pattern Anal. Machine Intell., vol. 6, pp. 450{462, 1984. [3] C. Berenstein, L. Kanal, D. Lavine, and E. Olson, \A geometric approach to subpixel registration accuracy," Comput. Vision Graphics Image Process., vol. 40, pp. 334{360, 1987. [4] D. A. Linden, \A discussion of sampling theorems," Proc. IRE, vol. 47, pp. 1219{1226, 1959. [5] A. Papoulis, \Generalized sampling expansion," IEEE Trans. Circuits Syst. , vol. 24, pp. 652{654, 1977. [6] J. L. Brown, \Multi-channel sampling of low-pass signals," IEEE Trans. Circuits Syst. , vol. 28, pp. 101{106, 1981. [7] H. Ur and D. Gross, \Improved resolution from subpixel shifted pictures," CVGIP: Graphical Models and Image Processing, vol. 54, pp. 181{186, 1992. [8] R. L. Graham, D. E. Knuth, and O. Patashnik, Concrete Mathematics: A Foundation for Computer Science. Addison-Wesley, 1989.
24
