Delaunay, Fortune 4]. The circumcenter in the triangle plane, cT , is taken .... The teapot and soldier data sets are taken from multiple viewpoints in the horizontal ...
Building 3D Graphical Models of Complex Objects A. Hilton, A.J.Stoddart, J. Illingworth and T.Windeatt Abstract
A system is presented for the automated reconstruction of complete 3D models of arbitrarily shaped objects from multiple 2.5D range images. The system enables reliable reconstruction of 3D models of complex objects. This is achieved without a priori knowledge about the object to be modelled. Potential application domains include reverse engineering of 3D models for graphics and CAD. Geometric fusion is achieved by integration of multiple range images into a single 3D implicit surface representation. The implicit surface provides a framework for integrating redundant overlapping measurements based on the local surface geometry. This ensures accurate representation of the measured surface geometry and topology for complex objects. Triangulated 3D models are then reconstructed using an implicit surface polygonisation algorithm. `Marching Cubes' provides a standard algorithm for implicit surface polygonisation. However, this algorithm results in an inecient model and requires all measurements to be made prior to model construction. These limitations are addressed by the introduction of a new implicit surface polygonisation algorithm called `Marching Triangles'. The new algorithm is based on a local 3D constraint which ensures the resulting model is a 3D Delaunay surface triangulation. Marching Triangles constructs an ecient representation and enables dynamic integration of new measurements. Results are demonstrated of reliable 3D model reconstruction for complex objects.
1 Introduction Automatic reconstruction of integrated 3D models from multiple 2.5D range images is a primary goal of recent research in both computer graphics [8, 14] and machine vision [12, 11, 7]. Integration (or fusion) of multiple images is required to overcome the occlusions inherent in a single 2.5D range image. Reconstruction consists of four key stages: Data capture of 2.5D range images from multiple viewpoints. Data registration into a common coordinate frame. Data fusion into a single 3D model. Model optimisation for ecient representation. Model reconstruction is illustrated schematically in Figure 2. Reliable solutions exist for the problems of image registration and mesh optimisation. However, reliable image fusion remains an open problem. This paper focuses on the automated fusion of multiple view range images into a single integrated model. A novel algorithm for range image fusion is presented. This enables reliable reconstruction of 3D models with the correct surface geometry and topology.
Supported by EPSRC GR/K04569. `Finite Element Snakes for Depth Data Fusion'
Multiple 2.5D Range Image Capture
View 0 View 7
View 1
Registration View 6
View 2
Implicit Surface Fusion
Object
Polygonisation
Dynamic
3D Model View 5
View 3 View 4
Figure 1: Multiple View 2.5D Range Images Optimisation
Figure 2: 3D Model Reconstruction
1.1 Problem Statement
The goal of surface reconstruction is to estimate a manifold surface S 0 that approximates an unknown object surface S using a sample of points X = f~x0; :::; ~xN ?1 g in 3D Euclidean space ~x = (x; y; z), combined with knowledge about the sampling resolution ~x and the measurement con dence p(S 0=~xi ). Given a method we want to be able to specify conditions on the original surface S and sample X that allow S 0 to be a reliable model. The sampling resolution ~x imposes a lower limit on the feature size that can be reliably reconstructed.
1.2 Previous Work
Hoppe et al. [8] presented a general method for constructing an implicit surface representation from unstructured 3D points. Polygonal models were then generated using a `marching cubes' approach [2]. The algorithm is `static' in the sense that all the image data is required prior to the polygonisation process. Soucy et al. [12] integrated range images using canonic subsets of the Venn diagram. The canonic subsets each represent the overlap between a subset of the 2.5D range images and are associated with a 2D viewpoint reference frame. The 2D reference frames are used to eliminate redundant data and merge intersecting regions. Soucy et al. [13] extended this algorithm to be `dynamic' allowing the sequential integration of new range images. Turk et al. [14] integrated range image triangulations using a dynamic mesh `zippering' approach. Overlapping regions of meshes are eroded and the boundary correspondence found by operations in 3D space. A local 2D constrained triangulation is then used to join overlapping mesh boundaries to
form a single mesh. Boissonnat [3] and Rutishauser et al. [11] retriangulate overlapping meshes using local 2D constraints on triangle shape. The four integration algorithms are completely di erent in their approach to constructing a single triangulated model. This results in di erent complexity, limitations and failure modes. In practice this limits the reliability of the reconstructed surface model. Relative performance characteristics of existing integration algorithms are discussed in [6, 7].
1.3 Overview of New Integration Algorithm
This paper presents a new integration algorithm which addresses the inherent geometric limitations of previous approaches. The algorithm consists of two key stages. An implicit surface based representation is introduced in section 2. This provides a framework for the geometric fusion of multiple 2.5D range images in to a single 3D represention. The second stage is polygonisation of the implicit surface to obtain a 3D triangulated model, section 3. A new implicit surface polygonisation algorithm is introduced called `Marching Triangles'. This overcomes several limitations of previous implicit surface polygonisation algorithms such as `Marching Cubes'. In section 4, results are demonstrated for the reconstruction of models of complex objects. A comparison is also presented of the models constructed using the Marching Cubes and Marching Triangles polygonisation.
2 Implicit Surface based Geometric Fusion The implicit surface provides a framework for fusion of both geometric and topological information. Geometric fusion is achieved by combining multiple overlapping measurements to form a `consensus geometry'. Redundant measurements are combined according to their relative measurement con dence. Complex object geometries occur for thin parts, regions of high curvature and crease edges. Previous fusion algorithms failed to reliably reconstruct complex geometries. Implicit surface based fusion uses a rule based algorithm to correctly integrate overlapping measurements. Reliable fusion is achieved by ensuring that fused measurements have consistent local surface geometries. The resulting implicit surface is a piece-wise continuous representation of the object surface. Fusion of local surface topology information is achieved by integrating surface continuity information from individual images. The local surface continuity is estimated for a single image by a step discontinuity constrained triangulation in the 2D image plane. Object surface topology is de ned by explicit representation of the integrated surface boundary. The implicit surface correctly represents objects with complex geometry and holes.
2.1 Implicit Surface Representation of a Range Image
An implicit surface is de ned as the zero-set of a scalar eld function f (~x) = 0. The aim of representing a set of surface measurements, X = f~x0::::~xN ?1 g, as an implicit surface is to construct a smooth eld function, f (~x), such that the zero-set approximates the data X as closely as possible. A piecewise continuous implicit surface function for multiple range images is presented in this section. This is based on the integration of multiple overlapping 2.5D triangulations. Accurate representation of surface geometry is achieved by combining overlapping measurements according to their con dence. Boundary information is integrated from multiple meshes to obtain an explicit representation of the measured surface topology.
Meshes, Mi, are initially constructed from each range image using a step discontinuity constrained triangulation, [11, 12, 14]. Step discontinuities occur in the 2.5D range image due to occlusion. Step discontinuity constrained triangulation assumes that the measured surface, S , is locally continuous if the distance between adjacent measurements in the 2D image plane is less than a distance threshold, td. Range measurements are connected if the distance between them is less than the distance threshold, td = n~x where n 4. This de nes the local surface continuity for each range image. For a single mesh M the eld function, f (~x), is constructed as the signed distance to the nearest mesh point, ~p(x; y; z). A binary function, b(~x) = [0; 1], is used to explicitly label eld function values, f (~x), with nearest points on the mesh boundary. If the nearest point, p~, is not on the mesh boundary, b(~x) = 0, the signed distance is the dot product of the vector to the nearest point, (~x ? p~), with the surface normal, ~np, at the nearest point: f (~x) = (~x ? p~):~np. Alternatively, if the nearest point, p~b, is on the mesh boundary, b(~x) = 1, the signed distance function is evaluated as the sign of the dot product of the vector to the nearest points, (~x ? p~b), with the nearest point normal, ~np , multiplied by the Euclidean distance: f (~x) = sign[(~x ? p~b):~np ] j(~x ? p~b)j. The zero-set of the eld function, f (~x), for mesh, M , is piecewise continuous with the same topology. b
b
2.2 Fusion of Multiple Images
Integration of multiple range images requires the construction of an implicit surface function based on multiple overlapping meshes Mk where k = 0:::M ? 1. A eld function fk (~x) can be implemented as described above for a single mesh. The problem is then to integrate the individual eld functions into a single continuous surface, f (~x), and boundary label, b(~x). This is achieved by rst evaluating fk (~x) for each individual mesh and then integrating them using a simple set of rules based on local surface geometry: i) Evaluate the signed eld function,fk(~x), and boundary function, bk (~x), for each mesh Mk for k = 0:::m ? 1. ii) Find the nearest non-boundary mesh point, bk (~x) = 0, from the set fk (~x) , fmin (~x). iii) If a non-boundary point, fmin(~x), does not exist return the nearest boundary point, bk (~x) = 1: f (~x) = fmin (~x) and b(~x) = 1. iv) Else nd all non-boundary points, bk (~x) = 0, with the same orientation as the nearest point, fmin (~x): F = ffsame (~x)g i = 0:::Nsame , where nmin :nsame > 0 v) Find the nearest non-boundary point with opposite orientation, fopposite(~x), where nmin :nopposite < 0. vi) Eliminate all points in F , where fsame (~x) > fopposite(~x) . bound
i
i
vii) Evaluate P as a weighted average of all points in F: P the nearest point i
f (~x) =
k wk fk (~x) where
wk = 1 and b(~x) = 0
This set of rules enables the integration of overlapping meshes to de ne a continuous zero-set of the eld function, f (~x) = 0, in all regions where a mesh is continuous and non-zero elsewhere. The rules account for the special cases of two overlapping surfaces with di erent orientations and multiple overlapping surfaces. The eld function evaluation according to the rules given above is illustrated schematically in Figure 3 for the di erent cases of overlapping surface. Step (iii) explicitly de nes mesh boundaries, b(~x) = 1, providing an integrated representation of the local surface topology. Steps (iv|vi) eliminate ambiguity if there are multiple overlapping meshes corresponding to di erent surface regions. This enables correct representation for surface regions of high
Meshes
Zero Set of Implicit Surface
Signed Distance Fuction
A N0 (a)
M0
0
A
B
B A
N0 (b)
M0
N1
0
A
M1
B
B A
N0 (c)
M0
A
0
0
A
0
0
M1
B
N1 B N0 (d)
A
M0 M1 N2
0
B
N1
M2 B
Figure 3: Continuous Implicit Surface Function: (a) Single mesh, (b) Two meshes from same surface, (c) Two meshes from di erent adjacent surface with opposite orientation, (d) Three meshes from three di erent adjacent surfaces. curvature and di erent surfaces in close proximity. The weighted average of nearest points, step (vii), enables smooth integration of overlapping meshes using estimates of measurement con dence or blending functions, [6].
3 Implicit Surface Polygonisation Polygonisation of the implicit surface enables the generation of a triangulated 3D object model. Marching Cubes is a standard algorithm for polygonisation of closed manifold implicit surfaces, [1, 2]. A simple extension to this algorithm enables the polygonisation of implicit surfaces with holes, [8, 7]. However, marching cubes has several limitations: The algorithm is inherently static requiring all data a priori. Polygonisation is computationally expensive. Inecient representation due to the non-uniform triangle shape. The `Marching Triangles' implicit surface polygonisation algorithm is introduced to address the above limitations. Local operations in 3D space triangulate the implicit surface according to a 3D Deluanay surface constraint. This constructs a manifold surface triangulation with the same topology and geometry as the implicit surface. Ecient representation is achieved by the Delaunay constaint which ensures a uniform distribution of triangle shape. Dynamic integration is achieved as Marching Triangles constructs a new triangultated model from a seed model. In addition Marching Triangles is a surface based algorithm resulting in lower computational complexity.
3.1 Marching Triangles Theoretical Background
The aim in this section is to de ne the basis of a local 3D procedure for constructing a triangulated model, M , of an unknown object surface, S . We de ne a 3D suface based
constraint for the Delaunay triangulation of a manifold surface, S . The 3D Delaunay triangulation of an arbitrary point set X = fx1 :::xi :::xN g is composed of tetrahedral volumes, T (xi; xj ; xk ; xl ). Such that there exists a sphere which passes through each vertex, x, of T which does not contain any other interior points of X . In the case where the points, X , lie on a manifold surface, S , Boissonnat [3] derives the following important property. The manifold surface triangulation represented in the 3D Delaunay triangulation is de ned by the condition that it is composed of triangles T (xi; xj ; xk ) such that there exists a circumsphere passing through each vertex, x, that does not contain any other interior point of X . This result de nes a 3D triangulated manifold surface which correctly represents the measured surface geometry and topology. The manifold surface triangulation de nes an optimal geometric structure which is symmetric, isotropic and closely related to the metric of the surface. In particular the triangulation maximises the minimum angle of any triangle. This is the 3D analogue of the 2D Delaunay triangulation of a point set where the points, X , lie on a manifold surface in R3 rather than R2. The above de nition of the manifold surface approximation can be used as the basis for deriving a `surface-based' approach to model reconstruction. Given a partial model, M 0 , of the surface we de ne a local procedure for adding triangular elements to the model boundary. We de ne a local constraint based on the properties of the manifold surface triangulation represented in the 3D Delaunay triangulation: 3D Delaunay Surface Constraint: A triangle, T (xi ; xj ; xk ), may only be added to the mesh boundary, at edge e(xi ; xj ), if no part of the existing model, M 0 , with the same surface orientation is inside the sphere passing through the triangle vertices, x, about the center, cT . Where cT is the circumcenter of the triangle verticies, (xi ; xj ; xk ), in the plane of the triangle, T . Surface points of the same orientation are de ned by a positive normal dot product, nT :nM > 0. This constraint is illustrated in Figure 4. This constraint is sucient to guarantee that each triangle, T , in the resulting model, M , uniquely de nes the local surface. Hence, building a model using this constraint guarantees that the local surface does not over-fold or self-intersect. In addition by imposing a constraint that ensures the surface is locally Delaunay we ensure that the resulting triangulated model, M , is globally Delaunay, Fortune [4]. The circumcenter in the triangle plane, cT , is taken as the fourth point required to de ne a sphere for each triangular element. A local 3D Delaunay manifold surface is guaranteed provided the triangle, T , satis es the condition that no part of the existing model is inside an arbitrary sphere passing through the three triangle vertices, [3]. De ning the sphere about the equi-distant point in the element plane enables this constraint to be satis ed. If di erent surface regions are in close proximity the sphere about the center, cT , may intersect another surface region but still give a valid local triangulation. The condition that parts of the existing model, M 0 , with opposite orientation may intersect the sphere allows complex geometries to be modelled. Complex geometries which require this condition are illustrated in Figure 5. The 3D Delaunay Surface Constraint guarantees the correct local surface geometry where surface regions are in close proximity. 0
n
r
k
i
n
(a) Thin Surface Section
C j
n n
(b) Crease Edge
Figure 4: 3D Delaunay Surface Constraint Figure 5: Complex local surface geometry
3.2 Marching Triangles Algorithm
An implicit surface triangulation algorithm can now be developed based on the local 3D constraint de ned in the previous section. The algorithm proceeds as follows. Firstly an initial seed model, M = M0 , is de ned. This may be either a single triangular seed element or a previously constructed model to which we wish to incorporate new measurements. New measurements are represented as an intermediate implicit surface, f (x), section 2. The current model M is represented as a list of edges and nodes. The algorithm is implemented as a single pass through the edge list. New edges introduced by the addition of new elements to the model are appended to the end of the edge list. The algorithm does not terminate until all model edges have been tested once. The algorithm proceeds by testing each edge, ebound = e(xi ; xj ), on the current model boundary, M : i) Estimate a new vertex position, xproj , by projection perpendicular to the mid-point of the boundary edge in the plane of the model boundary element Tbound (xj ; xi ; xk ), by a constant distance, lproj . ii) Evaluate nearest point on the implicit surface to xproj : xnew = f (xproj ). iii) Terminate the mesh growing (step vii) for edge if: a) The nearest point is on the boundary. b) The implicit surface orientation at xnew , nnew , is opposite to the model orientation nT of T (xi ; xj ; xnew ): nT :nnew < 0. iv) Apply 3D Delaunay Surface Constraint to Tnew = T (xi ; xj ; xnew ). v) If Tnew passes the 3D Delaunay Surface Constraint (a) add xnew to M . (b) add Tnew to M . (c) add new edges e(xj ; xnew ) & e(xnew ; xi ). vi) If Tnew fails the 3D Delaunay Surface Constraint apply steps iv&v to adjacent boundary vertices, Tnew = T (xi ; xj ; xprev ) or Tnew = T (xi ; xj ; xnext ). vii) Testing of ebound terminates when one or no new elements, Tnew , have been added to the model, M .
The above algorithm de nes the mesh growing procedure using a local 3D constraint. The principal steps of the Marching Triangles algorithm are illustrated in Figure 6. Note that all new elements added to the triangulation must pass the local constraint. This algorithm enables the boundary of the initial model M to be grown to incorporate new surface regions de ned my the implicit surface. New mesh vertices must correspond to non-boundary points on the implicit surface, step (iiia). This constraint ensures that the measured surface topology is correctly
estimate
next
new
c lproj i
e
j
i
j
prev k (a) Step i: Estimate new vertex position
(b) Step iii: Apply 3D Delaunay surface constraint
Figure 6: Illustration in 2D of key steps in the Marching Triangles algorithm reconstructed. Explicit representation of the implicit surface boundary enables this constraint to be implemented, section 2. Step (iiib) ensures that the local model geometry corresponds to the implicit surface geometry. This is required where the estimated vertex position, xproj , may erroneously correspond to a di erent part of the object surface. This may occur for thin object parts with the simple estimation scheme of step (i). Step (vi) allows the local connection of existing model nodes to form a continuous surface representation. Marching triangles does not impose any constraint on the position of new vertices. In particular mesh vertices are not constrained to lie at measurement points unlike previous mesh growing procedures [3, 11]. This facilitates adaptive mesh growing by evaluating the projected distance, lproj , according to the local surface geometry.
3.3 Time Complexity and Geometric Limitations
De ning a general form for the computational complexity of any range image integration algorithm is not possible as it is a function of the particular image set, [13, 6]. The worst case computational complexity is approximated for integrating m images of N points assuming O(N ) redundant measurements between each pair of images. Static integration of m range images requires O(m) nearest point evaluations for each implicit surface function evaluation, f (x). Assuming the model is triangulated at approximately the same spatial resolution as the original measurements and using the Euler formula [9] the number of implicit surface function evaluations is O(mN ). Hence, the time complexity of the static Marching Triangles integration algorithm is O(m2N ). Dynamic integration of a single range image requires the evaluation of the implicit surface for a single mesh. Assuming no redundancy between the model and range image the worst case complexity for a single image is O(N ). The time complexity of dynamic Marching Triangles integration of m range images of N points is O(mN ). Step discontinuity constrained triangulation is used to estimate the local surface continuity from each range image using a constant distance threshold, td = nx. This imposes a lower limit on the feature size for correct topology reconstruction, nx. The Marching Triangles algorithm reconstructs the correct topology of implicit surface features greater than the constant projection distance, lproj . Hence, the lower limit on topology reconstruction is de ned by the measurement resolution provided lproj < nx. The 3D Delaunay Surface Constraint does not impose any limitations on the distance between adjacent surfaces. The use of this constraint enables correct reconstruction of arbitrarily thin object parts and crease edges. The Marching Triangles algorithm eliminates limitations on surface geometry identi ed in previous approahes [7].
Figure 7: Marching Triangle Reconstruction of 3D Models
4 Results Results of the Marching Triangles integration algorithm for four objects are shown in Figure 7. Each data sets contains approximately 10 range images. The telephone1 and bunny1 images were previously used to demonstrate the mesh zippering integration algorithm, [14]. The teapot2 and soldier2 data sets were previously used to demonstrate the integration algorithm of Soucy et al. [13]. The teapot and soldier data sets are taken from multiple viewpoints in the horizontal plane, resulting in regions of no data for horizontal surfaces and occluded regions. The results demonstrate that the integration algorithm correctly reconstructs the surface topology for features greater than the sampling resolution, x. Thin surface regions, crease edges and regions of high curvature are correctly reconstructed. This overcomes the geometric limitations of previous integration methods [11, 13, 14]. A comparison of the triangulation obtained with Marching Triangles and Marching Cubes polygonisation is illustrated in Figure 8. Both algorithms are applied at the same spatial resolution, x. The Marching Triangles algorithm uses a Deluanay surface constraint resulting in a uniform distribution of element shape. Marching Cubes uses a voxel based structure which results in non-uniform element shape. The number of triangles in the reconstructed model is approximately three times less for Marching Triangles relative to Marching Cubes for the same geometric accuracy. 1 2
Cyberware range scanner data registered using the ICP algorithm [14] NRCC laser range images [10] registered by M.Soucy using InnovMetric software [5]
Marching Triangles Triangulation
Marching Cubes Triangulation
Figure 8: Comparison of Representation Eciency for Implicit Surface Polygonisation
5 Conclusion In this paper we have presented a new method for integration of multiple 2.5D range images into a single 3D model. Results are demonstrated for the reliable reconstruction of 3D models of complex objects. Models are reconstructed without a priori knowledge about the object. The new integration algorithm eliminates geometric limitations inherent in previous approaches. Geometric fusion is achieved by integrating muliple range images into a single implicit surface representation. Accurate representation of surface geometry is achieved by combining overlapping measurements according to measurement con dence. Complex object geometries are correctly represented using a local rule based integration algorithm. Fusion of local surface topology information is achieved by explicit representation of surface boundaries. A new implicit surface polygonisation algorithm is introduced called `Marching Triangles'. This algorithm constructs the Delaunay surface triangulation of a manifold surface in 3D space. Triangulation is performed using a mesh growing procedure based on a local 3D Delaunay surface constraint. The Deluanay constraint results in an ecient representation due to the uniform distribution of triangle shape. Comparison of Marching Triangles with Marching Cubes implicit surface polygonisation demonstrates a factor three improvement in representational eciency for the same geometric accuracy. Marching triangles also enables dynamic integration of new measurements into an existing triangulated 3D model. The worst-case computational complexity for integrating m images of N measurements is O(mN ) for dynamic Marching Triangles compared to O(m2N ) for Marching Cubes.
References [1] J. Bloomenthal. Polygonization of implicit surfaces. Computer Aided Geometric Design, 5:341|355, 1988. [2] J. Bloomenthal. An implicit surface polygonizer. Graphics Gems ed. Heckbert,P.S., 4:324| 350, 1994. [3] J.D. Boissonnat. Geometric structures for three-dimensional shape representation. ACM Transactions on Graphics, 3(4):266|286, 1984. [4] S. Fortune. Voronoi diagrams and deluanay triangulations. In Computing in Euclidean Geometry, eds. Du, D.-Z. and Hwang, F., pages 193|230, 1992. [5] H. Gagnon, M. Soucy, R. Bergevin, and D. Laurendeau. Registration of multiple range views for automatic 3d model building. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition, pages 581{586, 1994. [6] A. Hilton. On reliable surface reconstruction from multiple range images. In VSSP-TR5-95. ftp://ftp.ee.surrey.ac.uk/pub/vision/papers/hilton-vssp-tr-5-95.ps.Z, 1995. [7] A. Hilton, A.J. Stoddart, J. Illingworth, and T. Windeatt. Reliable surface reconstruction from multiple range images. In ECCV. Springer-Verlag, 1996. [8] H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, and W. Stuetzle. Surface reconstruction from unorganised points. Computer Graphics, 26(2):71|77, 1992. [9] J. O'Rouke. Computational Geometry in C. Cambridge University Press, 1994. [10] M. Rioux. Laser range nder based on synchronized scanners. Applied Optics, 23(21):3837|3844, 1984. [11] M. Rutishauser, M. Stricker, and M. Trobina. Merging range images of arbitrarily shaped objects. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 573|580, 1994. [12] M. Soucy and D. Laurendeau. Multi-resolution surface modelling from multiple range images. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 348{353, 1992. [13] M. Soucy and D. Laurendeau. A general surface approach to the integration of a set of range views. IEEE Trans. Pattern Analysis and Machine Intelligence, 14(4):344{358, 1995. [14] G. Turk and M. Levoy. Zippered polygon meshes from range images. In Computer Graphics Proceedings, SIGGRAPH, 1994.
