Fast Random Sample Matching of 3d Fragments - Semantic Scholar

Fast Random Sample Matching of 3d Fragments Simon Winkelbach, Markus Rilk, Christoph Sch¨onfelder, Friedrich M. Wahl Institute for Robotics and Process Control, Technical University of Braunschweig, M¨ uhlenpfordtstr. 23, D-38106 Braunschweig, Germany {S.Winkelbach, M.Rilk, C.Schoenfelder, F.Wahl}@tu-bs.de

Abstract. This paper proposes an efficient pairwise surface matching approach for the automatic assembly of 3d fragments or industrial components. The method rapidly scans through the space of all possible solutions by a special kind of random sample consensus (RANSAC) scheme. By using surface normals and optionally simple features like surface curvatures, we can highly constrain the initial 6 degrees of freedom search space of all relative transformations between two fragments. The suggested approach is robust, very time and memory efficient, easy to implement and applicable to all kinds of surface data where surface normals are available (e.g. range images, polygonal object representations, point clouds with neighbor connectivity, etc.).

1

Introduction

The problem of surface matching is an important computer vision task with many applications. The field of application comprises object recognition, pose estimation, registration or fusion of partially overlapping range images or volumes, protein docking, reconstruction of broken objects like archaeologic artifacts and reduction of bone fractures in computer-aided surgery. The common challenge is to find a rigid geometric transformation which aligns the surfaces in an optimal way. This paper focuses on the reconstruction of broken arbitrary objects by matching their fragments. Nevertheless, the proposed method is easy adaptable to all other applications mentioned above. The problem of matching complementary fragments of broken objects is largely similar to the problem of matching partially overlapping surfaces (e.g. range images). The main common difficulties are: – The search space has six degrees of freedom (dof). – Digitized surfaces often are inaccurate and noisy. – Large data sets, i.e. a very high number of points in 3d. In the case of broken objects there are some additional difficulties: – A good initial guess, that can be used to iterate to the global minimum, generally is not available. – Very large surface areas without correspondence at the complementary part. – Object intersections must be avoided. – Material deterioration.

1.1

Related Work

An outline of all publications dealing with registration techniques would go beyond the scope of this paper. Therefore we will only give an overview of the most relevant work. A very popular surface registration approach is the iterative closest point (ICP) algorithm [1]. The approach iteratively improves an initial solution by finding closest point pairs and subsequently calculating the relative transformation which aligns the two point sets in terms of least square error [2]. Although many enhancements to the original method have been suggested, they still require a good initial guess to find the global minimum. Many approaches are using local surface features to find corresponding point pairs. Features vary from simple properties like surface normals or curvatures to complex vectors like point signatures [3], surface curves e.g.[4], or spin-images [5]. However, their use cannot guarantee unique point correspondences; nevertheless they can highly constrain the search space. A well-known category dealing with object recognition and localization are the pose clustering approaches (also known as hypothesis accumulation or generalized Hough transform e.g. [6],[7],[8]) The basic idea is to accumulate low level pose hypotheses in a voting table, followed by a maxima search which identifies the most frequented hypotheses. The drawback of voting tables is their relative high time and space complexity, particularly in case of large data sets. A simple and robust approach for fitting models, like lines and circles in images, is the RANSAC algorithm introduced in [9]. The repeated procedure is simple but powerful: First, a likely hypothesis is generated by random (uniform distribution) from the input data set. Subsequently, the quality of the hypothesis (e.g. number of inliers) is evaluated. The method has been applied to a wide range of computer vision problems. The most related work [10] applies the RANSAC scheme to the registration of partially overlapping range images with a resolution of 64x64 pixels. The approach highly depends on the fact, that a range image can be treated as a projection of 3d points onto an index plane; this is why it is not applicable to the general case. Moreover, it does not take advantage of surface normals or any other features. Many papers address matching of two-dimensional fragments like jigsaw puzzles or thin-walled fragments e.g. [11]. The problem of matching complementary three-dimensional object fragments (including the consideration of undesired fragment intersections) is rarely treated in the open literature. One of the recent approaches in this field of research is based on a pose error estimation using z-buffers of each fragment for each hypothesized pose [12]. The error minimization is performed by simulated annealing over a 7 dof search space of all possible poses of two fragments in relation to a separating reference plane. Although the approach makes use of the simulated annealing optimization, it degenerates to an exhaustive search if the optimal matching direction is limited to a small angular range, e.g. in case of matching a nut with a bolt. Another disadvantage is that the computation of each z-buffer has a time complexity of O(n).

2

Method Overview

Our suggested approach rapidly finds the ’interesting regions’ in the space of all possible relative rigid transformations of two fragments. Transformations where the parts do not get in contact are not considered at all. Moreover a desired surface contact is characterized by a preferably large touching area without a significant fragment penetration. Therefore the quality of a contact situation is determined by the size of the overlapping surface area and the absence of surface penetration, which is treated with a penalty. According to the RANSAC principle, the proposed method generates likely contact poses by random (see Sect. 2.1) and estimates the matching quality of each pose by an efficient combination of fast forecasting and by the use of a binary search tree (see Sect. 2.2). The basic procedure consists of the following steps: 1. Select a random surface point pi ∈ A and a random surface point pj ∈ B. A surface contact at pi and pj constrains three degrees of freedom. Another two degrees of freedom are fixed by the respective vertex normals npi and npj , which are directed against each other. Only the rotation around these normals remains unknown. Optional: Select only pairs which satisfy f eature(pi ) ∼ f eature(pj ). 2. Select a random second point pair (qi ∈ A, qj ∈ B) which forms a fully constrained two-point contact pose together with the first pair (pi , pj ). Optional: Select only pairs which satisfy f eature(qi ) ∼ f eature(qj ). 3. Estimate the matching quality of the pose. Efficiency highly can be increased by using a fast forecasting technique combined with a dropout if the expected result is considerably worse than the last best match. 4. Repeat steps 1-3 and memorize the best match until the matching quality is good enough or a time limit is exceeded. Here A ⊂ ℜ3 denotes the whole set of surface points of one fragment (B respectively). It is obvious, that the consideration of local surface properties or features can increase the likelihood that a pair corresponds to each other. In our experiments we compare the mean curvature of a point pair, which enables us to reject up to 90% as unsuitable. Curvatures can be obtained from all kinds of data sets which provide neighbor connectivity (see e.g. [13] for curvature computation on triangle meshes). The strength of the algorithm is that it is independent of the fragment shapes. The efficiency is directly proportional to the size of compatible surface area (fracture interface) and independent of the tolerance of matching direction, etc.; thus the method is even applicable in case of components and assemblies with narrow mating directions (like nut and bolt or plug and socket) where approaches which iterate to a local minimum may fail.

ri tangent plane

n

qi hi

h qi

hi

ri

p

rotation axis

r

spin-table

Fig. 1. Schematic illustration of the spin-table coordinates

2.1

Rapid Generation of Two-Point Contact Poses

The first step of the algorithm outlined above generates a random pose where the fragments get in contact and the surfaces touch each other tangentially (contrariwise surface normals ±n at contact point p). Since this step only generates random surface point pairs, it is very fast and simple. The second step tries to find a second point pair (qi , qj ) which can be brought into contact by rotating around an axis with origin p and direction n. A point pair which fulfills this condition must have an equal distance h to the tangent plane (see Fig. 1) and an equal distance r to the rotation axis. Referring to [5] we will call the parameter pair (ri , hi ) the spin-table coordinates of a point qi in relation to its rotation origin p. The following algorithm alternately adds point references of A and B into their respective spin-table Ma , Mb until a point pair with equal spin-table coordinates has been found, or until a counter exceeds the limit k, which is a strong indication for a contact situation with marginal surface overlap. 1. 2. 3. 4. 5. 6. 7. 8. 9.

Clear spin-tables Ma := 0, Mb := 0. Repeat k times Select a random point qi ∈ A Calculate its spin-table coordinates (ri , hi ) with respect to p Insert qi into its respective spin-table Ma (ri , hi ) := qi Read out the opposite spin-table qj := Mb (ri , hi ) If (qj 6= 0) terminate loop; the new contact pair is (qi , qj ) Repeat step 3-7 with reversed roles of A, B and Ma , Mb End-repeat

We achieve a good trade-off between accuracy and execution time with a table size of 64x64. The basic procedure can be improved by accepting only contact pairs (qi , qj ) with compatible features and roughly contrariwise surface normals nqi , nqj . Furthermore, the efficiency can be increased by reusing one of both filled spin-table Mi of a point pi for multiple assumed contact points pj on the counterpart. 2.2

Fast Quality Estimation of a Given Pose

After generating a hypothesis we must measure its matching quality. For this we estimate the proportion of overlapping area Ω where surface A is in contact

with the opposite surface B in the given pose. We assume that the surfaces are in contact at areas where the distances between surface points are smaller than ε. Ω can also be regarded as the probability that a random point x ∈ A is in contact with the opposite surface B. Thus Ω can be forecasted by an efficient Monte-Carlo strategy using a sequence of random points, combined with a dropout if the expected result is considerably worse than the last best match. Suppose that x1 , . . . , xn are independent random points with xi ∈ A. Then Ω is given by Pn i=1 contactB (xi ) Ω = lim (1) n→∞ n where contactB (x) is a function which determines whether point x is in contact with surface B ½ 1 if distB (x) < ε contactB (x) = (2) 0 else and distB (x) is a function which returns the minimal distance of a point x to surface B distB (x) = min |x − y| (3) y∈B

Our implementation of this function is based on a kd-tree data structure (see [15]) and therefore offers a logarithmical time complexity for the closest point search. In contrast to Eq. (1) it is only possible to try a limited number n of random points; thus Ω only can be approximated up to an arbitrary level of confidence. (Notice that this restriction only holds if A and B contain an infinite number of points). Considering the margin of error, for every additional random point, the approximation of Ω can be recomputed as Pn contactB (xi ) 1.96 ± √ (4) Ω ≈ i=1 n 2 n with a 95% level of confidence. If the upper bound of this range is worse than the last best match, we abort the computation and try the next hypothesis. Up to this point we have not considered surface penetrations at all. To ensure that the fragments do not penetrate each other we simply subtract a penalty for penetrating points (i.e. points which are more than ε ’below’ the complementary fragment surface). This can be done by using an alternative contact rating  (at surface)  1 if distB (x) < ε (5) contactB (x) = −4 if distB (x) ≥ ε ∧ (x − y) · ny < 0 (inside)  0 else (outside) Where y ∈ B denotes the closest point to x, and ny denotes the surface normal of y. Due to the fact that y can be determined simultaneous with the calculation of the minimal distance in Eq. (3), it is straightforward to test whether x lies inside or outside of fragment B. The execution time can be accelerated by an approximative early bound of the kd-tree search.

2.3

Matching Constraints for Special Types of Fractures

Up to this point, we did not use any knowledge about the object. However, in many applications, additional information about the parts to be combined are available and can be used to constrain the search space. This includes information about the object shape (e.g. [16]), geometric features like a sharp curvature transition from an intact surface to a broken one (e.g. [4]), or knowledge of the roughness of the fracture. An integration of these constraints into our approach is straightforward and can drastically limit the search space and thus reduce the execution time. Particularly suitable for this matching procedure is the previous knowledge of surface points at the fracture interface which must be brought into contact with the counterpart. These ’contact constraints’ can highly increase the efficiency of the contact pose generation of Sect. 2.1. A detailed discussion about these additional constraints is subject of further publications.

3

Experimental Results and Conclusion

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 2. Some matching results: (a) plug and socket; (b) broken boulder; (c) broken venus; (d) broken Stanford bunny; (e) broken bone model; (f) broken pelvis

The method has been evaluated on high-resolution triangle meshes of artificial and digitized objects with computer generated and real fractures. All computer

Table 1. Evaluation of 100 passes per experiment. The entries are: Total number of vertices; area of fracture interface (Overlap); added noise level in percent referring to the maximal object radius; lower limit (LRL), upper limit (URL), and mean execution time µ in seconds, and standard deviation of execution time σ

plug(a)

boulder(b) venus(c) bunny(d) bone(e) pelvis(f)

(a)

Vertices 41772

Overlap 29%

21068 51837 96170 35503 57755

38% 28% 30% 13% 1%

Add.Noise 0.0% 1.0% 1.5% 2.0% 0.1% 0.1% 0.1% 0.0% 0.0%

LRL 0.09 0.21 0.33 2.15 0.11 0.31 0.41 0.08 -

URL 10.28 154.31 186.27 263.96 22.58 27.42 64.31 114.62 -

µ 2.52 46.51 48.66 64.94 4.85 6.39 11.61 28.01 -

σ 2.85 43.92 45.27 63.86 4.53 5.79 10.68 28.83 -

(b)

Fig. 3. (a) Example with 2% noise distortion and smoothing result; (b) broken pelvis and the ’constraint matching’ result

generated fractures are distorted with some additive Gaussian noise and it is assured that the tesselation of one fragment is unrelated to its counterpart. Fig. 2 shows some representative test examples with different noise levels. To make the calculation of surface normals and curvatures robust against noise we apply a common mesh smoothing filter [13]. The left side of Fig. 3 visualizes the intensity of our maximal noise distortion, and the effect of smoothing. Some of our data sets are real human bone fractures (femurs and pelvis), which are extracted from computer tomographic scans. The medical relevance of a robust bone registration method is outlined in [16]. The required execution times for successful assembly of these fragments are listed in Table 1. The tests were performed on an AMD Athlon XP/1466MHz based PC. As can be seen, the method performs well with a variety of objects (3d fragments as well as industrial components). The desired matching accuracy is implicitly regulated by the input parameter ε (maximal contact distance of Eq. (2)) and the minimal matching quality Ω. In the unconstraint case the algorithm always finds the solution with the largest touching area, which is not always the desired one. This occurs if the portion of fracture interface is small in comparison to the total surface area, and if the intact object surface includes large smooth areas with similar regions on the counterpart (e.g. the pelvis in Fig. 2). In these cases additional constraints

(which are discussed in Sect. 2.3) are indispensable to prevent the algorithm from finding ”trivial solutions”. The desired solution of the pelvis (see the right side of Fig. 3) can be achieved by constraining the matching direction, or by selecting some vertices at the fractured surface which must be brought into contact with the counterpart. In further publications we will discuss usage of these additional constraints.

References 1. Besl, P., McKay, N.: A method for registration of 3-D shapes. IEEE Trans. PAMI 14(2) (1992) 239–256 2. Horn, B.K.P.: Closed-form solution of absolute orientation using unit quaternions, J. Opt. Soc. Amer. A 4(4) (1987) 629–642 3. Chua, C.S., Jarvis, R.: Point Signatures: A New Representation for 3D Object Recognition, Int’l Journal of Computer Vision, 25(1) (1997) 63–85 4. Papaioannou, G., Theoharis, T.: Fast Fragment Assemblage Using Boundary Line and Surface Matching. IEEE Proc. ICPR/ACVA (2003) 5. Johnson, A.E., Hebert, M.: Surface registration by matching oriented points. Proc. lnt’l. Conf. Recent Advances in 3-D Digital Imaging and Modeling (3DIM) (1997) 6. Stockman, G.: Object Recognition and Localization via Pose Clustering. Computer Vision, Graphics, and Image Processing, 40 (1987) 361–387 7. Linnainmaa, S.,Harwood, D., Davis, L.S.: Pose determination of a threedimensional object using triangle pairs. IEEE Trans. PAMI 10(5) (1988) 634–647 8. Barequet, G., Sharir, M.: Partial Surface Matching by Using Directed Footprints. Computational Geometry’96, Philadelphia PA, USA (1996) 9. Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with application to image analysis and automated cartography. Communication of the ACM, 24(6), (1981) 381–395 10. Chen, C.S., Hung, Y.P., Cheng, J.B.: RANSAC-Based DARCES: A New Approach to Fast Automatic Registration of Partially Overlapping Range Images. IEEE Trans. PAMI 21(11), (1999) 1229–1234 11. Leit˜ ao, H.C.G. , Stolfi, J.: A Multiscale Method for the Reassembly of TwoDimensional Fragmented Objects. IEEE Trans. PAMI, 24(9) (2002) 1239–1251 12. Papaioannou, G., Karabassi E.A., Theoharis, T.: Reconstruction of Threedimensional Objects through Matching of their Parts, IEEE Trans. PAMI, 24(1) (2002) 114-124 13. Desbrun, M., Meyer, M., Schr¨ oder, P., Barr, A.H.: Implicit Fairing of Irregular Meshes using Diffusion and Curvature Flow. Proc. SIGGRAPH 99, (1999) 317-324 14. Johnson, A.E., Hebert, M.: Control of Polygonal Mesh Resolution for 3-D Computer Vision. Tech. Report CMU-RI-TR-96-20, Robotics Institute, Carnegie Mellon University, April (1997) 15. Friedman, J.H., Bentley, J.L., Finkel, R.A.: An Algorithm for Finding Best Matches in Logarithmic Expected Time. ACM Trans. Mathematical Software 3(3), (1977) 209–226 16. Winkelbach, S., Westphal, R., Goesling, T.: Pose Estimation of Cylindrical Fragments for Semi-Automatic Bone Fracture Reduction. In: Michaelis, B., Krell, G. (Eds.): Pattern Recognition (DAGM 2003), Lecture Notes in Computer Science 2781, Springer (2003) 566–537