A Line-Tracker Arnold Jonk & Rein van den Boomgaard & Arnold W.M. Smeulders Faculty of Mathematics, Computer Science, Physics and Astronomy University of Amsterdam, The Netherlands fjonk,rein,
[email protected] A line-tracker is presented that is flexible and robust. Linepoints are detected locally in order to construct a tracker that can be fast and is able to use knowledge about the line to be tracked. This is especially needed in cases where the line is of poor image quality. An application is shown in which two trackers are combined into a parallel line-tracker.
1 Introduction Interpretation of line drawings is largely dependent on the quality of line-recognition, because lines form the building blocks of these types of drawings. Because of its importance, considerable effort has been put in line-recognition. There are two main approaches to line recognition, line-detection and line-tracking. Line-detection is a parallel proces of global-image operations such as segmentation, edge-detection and clustering, in principle using little a priori information where to find the line. The difficulty here is to assure connectivity in the detected line-segments (see [Jonk95]). Line-tracking is a sequential process starting from an a priori located point on the line. The difficultu here is reach the end of the line. Line-tracking may use an adaptive model of the profile, hence provides more information to the line-recognition step. When lines are of very poor quality global-image techniques are very costly, ineffective or give rise to a large number of false hits (observe figure 1 for an example). In such a case line trackers are the preferred method.
Figure 1: Example of a line that is difficult to detect with global image operators. Note that the line on the left is drawn on the backside of the paper, which explains its vague appearance.
2 Line modelling and detection The model of the line consists of two parts, namely a model describing the geometry of the line and a model describing the way the line is projected on the grey-valued image (the profile).
2.1 Shape modelling Line-tracking methods model lines differently than line-detecting applications. In [Dori93] curves are modelled as a series of arcs and vectors. The vector model and arc model can be useful for detecting lines. Vectors and arcs can be effectively detected and then clustered in complete line-descriptions. For line tracking however, another approach is needed. Describing the line as a spline yields a stable and precise description of a line. This is the reason most line-tracking applications use spline-descriptions. A B-spline is described by its set
of controlpoints fp~0 ; p~1 ; ; p~n g. The formula for interpolating a B-spline (see [Pavlidis87]) is given by:
Lk (t) = where
k X i=0
p~i Ni;m (t);
(1)
i t ti+1 Ni;0 = 10 totherwise (2) Ni;m(t) = t t ??ti t Ni;m?1 (t) + t ti+m+1??t t Ni1 ;m?1 (t): (3) i+m i i+m+1 i+1 In these equations m stands for the order of the spline, and k for the number of nodes. In our line-tracking method, a B-spline is used to describe the line.
2.2 Profile modelling & detection In this section we describe a method to detect linepoints. Linepoints are the centre-points of the line. To this end, first the model of the line is described, and then we describe a two-step detection method. This method is detailed in [Steger96]. In figure 2.a, a grey-value landscape is presented, figure 2.b shows a model of this landscape. This is modelled as a 1D-parabolic profile that is swept along a curved line. Finding the
A
B
Figure 2: a. Grey value landscape of a line. b. Model of a line. linepoints is done in two basic steps. First, the local direction of the line is estimated. Then, in the direction orthogonal to the line (let this direction be n(t)), the center of the line’s profile is sought. The local direction of the line can be estimated by looking at the second-order structure of the image (z (x; y )). As is clear from the example, the second order derivative is minimal is the line’s direction. Because lines are noisy, the data is convolved with the derivatives of a Gaussian smoothing kernel. The size of the smoothing kernel must be tuned to the expected width of the lines to get an optimal response (a technique called edge focussing). The direction in which the second derivative of z (x; y ) takes on its maximum absolute value will be used as the direction n(t). This direction can be determined by calculating the eigenvalues and eigenvectors of the Hessian matrix
zxy : H (x; y) = zzxx xy zyy
(4)
2 ?w x w ; f (x) = 0h (1 ? (x=w) )); otherwise
(5)
The direction n(t) is obtained by using the vector corresponding to the eigenvalue of maximum absolute value. Detection of the centre of the line is now straight forward. Figure 2.b shows our model of the line’s propile, a parabolic shaped profile. This is an effective first-order approximation of the observed real world profiles. The parabolic profile (with x the (local) coordinate in the direction n(t)) is defined with the equation:
with w being the width, and h the height of the line. The estimated 1D-derivatives (with Gaussian smoothing) are:
z(x; ; w; h) = (g f )(x) z (x; ; w; h) = (g f )(x) z (x; ; w; h) = (g f )(x): 0
(6)
0
00
(7)
00
(8)
It is clear from these equations that z (x; ; w; h) = 0 , x = 0 for all , and z (x; ; w; h) will take on its maximum negative value at x = 0. It is therefore possible to determine the precise location of the line for all . To apply the above analysis on discrete signals, two modifications need to be made. The first one is the choice of the convolution in discrete space, for which the integrated Gaussian is selected. This location can be determined with subpixel accuracy. 0
00
3 The Line-Tracker In figure 3 an overview is given of the basic steps of the tracking algorithm. In this figure the boxes denote steps in the line-tracking algorithm. So, there is an initializing step, a step to retrieve a list of possible next extensions, etc. POINT, DIRECTION
Start SPLINE STEPSIZE
CURVATURE
find extensions EVALUATED LINE-POINTS
Select extension EXTENSION POINT
extend
stop?
terminate
Figure 3: Overview of the tracking algorithm, with its main parameters. As input to the tracking-algorithm one point and a direction is needed. The data further consists of the grey-value image, and parameters needed to steer the extension-routine (detailed later). The increment of the line-description is done in two basic-steps. First the set of possible extensions is determined. Then each extension is evaluated, in order to be able to select the optimal extension. To extend a line with a certain stepsize, first potential extension points are detected. Then each point is evaluated by trying to find evidence in the image for the line that is formed by extending the spline with that point. This method of finding extension points is preferred over pixel by pixel extensions for two reasons. First, jumping along the tracked line gives better adaptive control on the curvature of the line. And secondly, jumping along the line can be more robust, for small line-ruptures can be handled. As can be seen in figure 4.a, extending the spline and looking for linepoints orthogonal to the extension is too simple an approach. The size of the extension is limited to the radius of the circle describing the maximal allowed curvature ( d0 in the figure).
In figure 4.b another approach is presented. Here, the possible new linepoints are sought on the circle with diameter d, between the intersections of this circle with the circles describing the maximum admissible curvature p. Using the points on this circle also ensures the length of the new line-fragment to be shorter with increasing curvature.
d d p
p d’
(A)
(B)
Figure 4: The maximally admissable curvature of the line is indicated by a with the circle with diameter p. (a). Seeking points orthogonal to the extension-direction severely restricts the stepsize. (b). Therefore, possible new linepoints are sought on the circle with diameter d.
The extension of a spline is calculated by obtaining the derivative-vector (d~) of the spline at its endpoint. The initial extension is a vector of length p (See figure 5.a). Linepoints are detected in the circular subimage using the detection-method described in section 2.2. Detection results in a set of linepoints S (figure 5.b). Now, the list of potential extension-points E is derived p
c
A
B
C
Figure 5: Definition of parameters. (a) The center c and stepsize p. (b) The set S of detected linepoints is a subset of the hatched area. (c) The set E of potential extensionpoints is a subset of the cross-hatched area.
from S by defining
E = fvjv 2 S \ d(v; c) p ? p g; (9) with d(v; c) the Euclidean distance between points v and c, and p the maximum distance be-
tween a detected line-point and the circle line. This is visualized in figure 5.(c). Now, each point in E needs to be evaluated, in order to be able to select the point that is the most likely extension of the line. Observe figure 6, to appreciate the need for an evaluation algorithm of potential extension points. other line c tracked line
p1 p0 p2
extension of tracked line
Figure 6: An evaluation based on evidence in the image is needed to distinguish between both detected linepoints. The evaluation is done by extending the spline Ln (representing a spline with n nodes) with each point e 2 E , and then checking the evidence in the image for the lines thus hypothesized.
This done by calculating the minimum squared distance between the extended spline, and the detected line-points. See for an example figure 7. p1
p1
p2
p2
A
B
Figure 7: The squared difference-function applied to an example. Figure (b) will produce a (much) larger error than figure (a).
The line tracker needs a stopping condition. Primarily, the tracker will halt if it can not find an extension point. In addition, the tracker must also be restrained from ’jumping’ lines (figure 8.a). This is achieved by imposing a maximum on the allowed value of c(Ln ; S ). Also, the line-tracker should halt when the line-structure changes abruptly (figure 8.a). This is tackled by allowing continuous changes in the parameters of the line: extensions that do not extend the line with a segment with significantly different contrast and/or width.
A
B
Figure 8: (a) the tracked line terminates, but another, similar, line falls within the stepsize. (b) Two different, connected, lines. The line-tracker should terminate tracking when it reaches the connection.
The tracker can fail in a number of circumstances. When line points remain undetected, for example because the line’s thickness is outside the parameters supplied to the algorithm, or its profile cannot be approximated by a parabolic shape, the tracker will incorrectly terminate. The design of the algorithm is such that small gaps can be handled. A gap larger than the stepsize, however, will still result in termination of the tracker.
4 Application of the linetracker In this section a detector based on two line-trackers is described. The task at hand is to track two lines that are largely parallel. The lines need to be tracked until one of the lines is lost. See figure 9 for an example, and figure 4 for the pseudo-code of the algorithm.
Figure 9: Example of two parallel lines, with one ending in an arrowhead. The input of the parallel line-tracker consists of two points and a direction. These parameters are sufficient to initialize both trackers, for the lines are presumed parallel. Both lines (called Bn1 and Bn2 ) are extended with their appropriate stepsize. This results in two sets of possible new points for both lines. The algorithm needs to decide which of the points will be selected to extend the lines. See figure 11. This decision is based on two factors, namely the quality of a hypotheses and the consistency in parallelism of the two lines.
procedure ParallelTracker (Point P1, Point P2, Vector Dir) initialize(L,P1,Dir), initialize(R,P2,Dir) LeftStepsize, RightStepsize DefaultStepsize do Left get extensions(L,LeftStepsize) Right get extensions(R,RightStepsize)
1
2
Min for Le Left for Ri Right This eval(L+Le,R+Ri) if This < Min LeMin Le RiMin Ri Min This endif if Min = extend (L,LeMin) extend (R,RiMin) LeftStepsize, RightStepsize adjust stepsize(L,R,DefaultStepsize) endif while Left = EMPTY .AND. Right = EMPTY
2
6 1 6
6
return L,R
Figure 10: Pseudo-code of the parallel line tracker The quality of the hypotheses of a new linepoint is described in the previous section. The consistency in parallelism is measured using the first-derivative vectors of both spline-descriptions between the last nodes in the following way:
P (Bn1 ; Bn2 ) =
Zn
n?2
(j Bn1 (i)0 ? Bn2 (i)0 j)2 d(i):
(10)
Note that we only need the last three nodes of the spline-descriptions because cubic splines are used. Combining these two factors is done by selecting the two potential linepoints of both lines maximizing the following equation:
max g (P ( (Bn1 ; e1 ); (Bn2 ; e2 ))) gc (c( (Bn1 ; e1 ); S1 )) gc (c( (Bn2 ; e2 ); S2 )): (11) e1 2E1 ;e2 2E2 p extension of backline
p1 c
p0 p2
backline
B galley backline
Figure 11: Extending the B-spline and locating valid backline-points can produce several hypothesis. Using constraints provided by the galley, the correct point is found.
In this equation, (Ln ; e) extends a spline Ln with a point e. c( (Bn1 ; e1 ); S1 ) is a function that measures the quality of an extension as described in the previous section. Note that usually there will be only one potential linepoint for each line, in which case those points are immediately selected. There are several ways to combine the terms in equation 11. Multiplication was chosen (instead of summation), after experiments demonstrated that it was best capable of trading off the quality of the line-detection, and the consistency in parallelism. Independently extending the two lines can lead to problems. This is due to the fact that lines are required to be parallel, but not straight. Observe figure 12.a. When on both lines the same stepsize is used, the detected points will not remain exactly opposite. Figure 12.b shows a solution to this problem. Before every extension, the normal to the end-point of the spline is intersected with the (straight) extension of the opposing spline. This produces an intersecting point. The distance between the endpoint and the intersection point is subtracted from the extension of the inner-spline.
Delta
A
B
A
(A)
B
(B)
Figure 12: (a). When tracking two curved, parallel lines, a uniform stepsize will cause the tracking to divert. (b) Calculating the size of the step needed on the ’inner-track’.
4.
This synchronization of the stepsize is done in the subroutine adjust
stepsize in algorithm
Increments are done until the stopcondition is met. The stopcondition is reached when one of the lines cannot be extended further. This can happen for example when one of the lines terminates, or runs out of the image. Additional stopcriteria can be envisioned. For example an upper bound can be enforced on the divergence of the lines. In the current algorithm, parallelism is only used as a selection-criterium when multiple extensions are available.
5 Experiments Using our algorithm, we undertook experiments to test the validity of our approach. It showed the algorithm succeeds in finding the set of parallel lines except for very specific circumstances. In figure 13 we present two of the tested situations, and the results obtained by the algorithm. In the presented experiments, two different lines are tracked. One line is drawn on the backside of the paper, while the other is a thick line drawn on the front. In figure 13.a a difficult but succesfully tracked situation is shown. Here, at the location the lines diverge (making the parallel-criterium less effective), the image is of very poor quality due to paper rupture. Line detection is also hampered by a crossing object. The tracker is, however, still able to detect te line. In figure 13.b a failed experiment is shown. Here, the line-detection step fails to find the line drawn on the backside of the paper. The figure displays (the white dots) detected linepoints (the center-line of the arrowhead), and the search area (indicated by the grey dots). The tracker can not operate when no linepoints are detected.
6 Comparison with profile matching There are other techniques for tracking lines. In this section, our method is compared with a profile matching algorithm. In [Noordmans97] an algorithm based on profile matching is described. Consequetive crosssections are matched against the model of the line (2D or 3D). This model is updated after each
(A)
(B)
Figure 13: Examples of tracked parallel lines. (a) Curved line with a close up of low quality image part. (b) Example of a tracking error.
match. The stepsize between consequetive matches is typically very small. The advantage of this method is the (potential) accuracy of the estimated parameters. Very specific structures can be followed, for example a 3D-ellips. A drawback of the profile matching method is its sensitivity to noise and gaps in the line. The example images in this paper show lines of very poor quality. Their profiles are not suited for high precision profile matching, and many gaps in the lines, as well as crossing lines, can be expected. Experiments showed that in this domain profile matching does not perform well.
7 Conclusions A line tracking algorithm was presented. We have shown the usefulness of the line point detection method in cooperation with our line tracker. The line tracker is designed such that is robust against line ruptures, and can accurately control the maximal allowed curvature of the tracked line. A strong point of this algorithm is its flexibility. This flexibility was demonstrated in an example, where two line-trackers were combined to form a parallel line-tracker.
References [Dori93] D. Dori & Y. Liang & I. Chai. Sparse Pixel Recognition of Primitives in Enigineering Drawings. Machine Vision and Applications, 6,pp69-82, 1993. [Jonk95] A. Jonk & A. Smeulders. An axiomatic approach to clustering line-segments. ICDAR, 1995. [Noordmans97] H.J. Noordmans & A.W.M. Smeulders High Accuracy Tracking of 2d/3d curved line structures by Consecutive Cross-Section Matching. Submitted to Pattern Recognition Letters, 1997 [Pavlidis87] T. Pavlidis. Algorithm for graphics and image processing. Computer Science Press, 1987. [Steger96] C. Steger. Extracting curvilinear structures: A differential geometric approach. In B. Buxton and R. Cipolla, editors, Fourth European Conference on Computer Vision, volume 1064 of Lecture notes in computer science, pages 630-641. Springer-Verlag 1996.
