Pierre Ermes, Frank A. van den Heuvel, George Vosselman. Faculty of Civil Engineering and .... Lowe. (1991) introduces the linearised observation equations.
A PHOTOGRAMMETRIC MEASUREMENT METHOD USING CSG MODELS Pierre Ermes, Frank A. van den Heuvel, George Vosselman Faculty of Civil Engineering and Geosciences Delft University of Technology Thijsseweg 11, 2629JA Delft, The Netherlands E-mail: {P.Ermes, F.A.vandenHeuvel, G.Vosselman}@geo.tudelft.nl Commission V, Working Group 2
KEY WORDS: parameterized CAD models, CSG, model based measurement, observation equations, analytical partial derivatives
ABSTRACT: The paper presents a measurement method for CSG (Constructive Solid Geometry) models in images for which the exterior orientation is assumed to be known. The visible edges of a volumetric CAD model with an initial pose are projected back into an image after a hidden line analysis. This analysis determines which object edges (or parts of edges) are visible, and computes the contour edges of curved surfaces. Observation equations are formulated that establish the relation between the position of the edges in the images, and the parameters of the CAD model. These equations are used for interactive manipulation of the model as well as automated fitting to the gradients in the images. For the formulation of the observation equations, the partial derivatives of the edge positions with respect to the model parameters are needed. Two methods for their computation exist. First, a partial derivative can be estimated numerically by incrementing a parameter with a small amount, computing the backprojection of the slightly modified model and computing the difference between this and the original backprojection. The second method is the analytical computation of the partial derivatives. For CSG models we have developed such a method. The major advantage of the numerical method is that it is applicable to any parameterized CAD model, whereas this analytical method is only suited for CSG models. On the other hand, the analytical method is faster and more robust. This paper focuses on the analytical method for the computation of the partial derivatives, compares the analytical and numerical methods, and shows how the observation equations can be used for interactive measurement of the CAD models in the images.
1. INTRODUCTION Photogrammetry is a measurement technique often used for the as-built reconstruction of piping installations. Existing photogrammetric techniques measure points or lines in the images from which a threedimensional object is reconstructed (Jones, et.al., 1996), (Veldhuis and Vosselman, 1998). However, it would be more straightforward to measure an object in one step, as one entity, instead of measuring separate points or lines and grouping them later. Lowe (1991) and Vosselman (1998) use parameterized CAD models and observation equations to estimate the parameters of the model. Lowe (1991) estimates only the pose parameters of a CAD model while Vosselman (1998) estimates the shape parameters of the model as well. This paper introduces a further extension to this method to make it applicable to a whole class of CAD models, i.e. Constructive Solid Geometry (CSG) models. Vosselman (1998) computes the partial derivatives in the observation equations by means of numerical simulation, but for complex models, it turns out to be a computationally intensive task. This paper introduces a faster, analytical method for the computation of the partial derivatives. The observation equations for CSG models can serve as the basis for automated image
matching (Tangelder, et.al., 1999) as well as for the interactive measurement of CSG models. The measurement of CSG models is embedded in a measurement system for piping installations, as is discussed in a broader context by Ermes and van den Heuvel (1998). The following section discusses the interactive measurements of CSG models, the third section elaborates on the formulation of the observation equations and discusses the differences between analytical and numerical partial derivatives. The fourth section describes how the partial derivatives are computed analytically, and in the fifth section the difference between the numerical and partial derivatives is demonstrated with a few examples. The final section summarizes the main results of the methods that are developed, and proposes possible enhancements.
2. INTERACTIVE MEASUREMENT OF A CSG MODEL A CSG model is a tree-like structure in which primitive shapes are combined using volumetric Boolean operations (union, intersection and subtraction) to form more complex shapes (Figure 1). A typical set of primitives contains shapes like a box, a sphere, a cylinder, a torus, a prism, etc. A CSG tree is
parameterized by the pose and shape parameters of each of its primitives.
a
b
c
d
Figure 1: (a) Boolean operations between two rectangles. (b) Union. (c) Intersection. (d) Subtraction.
The edges of an object are the most distinctive features in an image and therefore they are used for its measurement. It takes two steps to obtain the projected edges in an image from a parameterized CSG model. First, a boundary representation (B-rep) is computed, describing the geometry and topology of points, edges and surfaces, of the CSG model. Second, the B-rep model is projected onto the image, assuming that the interior and exterior orientation parameters of the image are known. This projection also computes the contour edges of curved surfaces and analyses which edges, or parts of edges, are visible from the viewpoint of the camera (see Figure 2). Although the purpose of this projection is to compute the visible edges of a model, it is commonly known as the hidden-line projection.
are solved using all the equations set up so far and a new projection is computed. This is repeated in a loop, using the current pointer position in each iteration, until the operator stops measuring. Experience shows that when the iterations are computed fast enough (less than one second per iteration, approximately) the operator gets the impression he or she is dragging the model edge, along with the pointer, to the corresponding position in the image. Measurements can be done to all the edges of the model, in an arbitrary number of images.
3. FORMULATION OF THE OBSERVATION EQUATIONS The observation equations relate a position on an edge of the model to the parameters of that model. In general, a set of these equations is too complex to solve analytically, therefore the equations are linearised and solved using an iterative least squares algorithm. Lowe (1991) introduces the linearised observation equations.
∂u ∆p j j =1 ∂p j n
∆u = ∑ Where
∆u is
(1)
the perpendicular distance from the
measured point to the selected edge,
p j are
the object
∆p j are the unknown changes parameters, and n is the number of parameters. parameters,
Vosselman derivatives
a
b
Figure 2: (a) A hidden-line projection of a T-junction. (b) Side view of the lines visible in (a). In the measurement process the operator interprets the images and selects a model from a database of CSG models. The selected model is projected with an initial pose and shape into the images, of which the interior and exterior orientation parameters are assumed to be known. The operator selects an edge of the projected model with a pointing device (e.g. a mouse) and a new observation equation, as discussed in section 3, is set up. Next, the operator moves the pointer to the corresponding edge in the image. During the movement, the distance between the position of the pointer and the selected edge is used to update the observation equation. The shape and pose parameters
∂u / ∂p j
(1998)
computes
the
to the partial
using a numerical simulation
method. For each parameter, a new hidden-line projection is computed with a slightly modified parameter. The partial derivative, with respect to the modified parameter at a point on an edge, is estimated by calculating the perpendicular distance between the point and the edge of the modified model. The distance is divided by the change of the parameter to obtain the partial derivative. When the equations are solved, the parameters of the model are added, as observations with a small weight, to the equations to stabilize the solution. This is useful for two reasons: When the first edges are measured, there are probably not enough measurements for a stable solution. An individual primitive of a CSG tree may not be completely visible and, therefore, its parameters cannot be determined accurately. Adding a parameter, as observation, to the equations maintains a default value for this parameter. For example, when a hole in an object is modeled by the subtraction of a cylinder, only one end of the cylinder is visible (at most), and therefore the length of this cylinder cannot be determined.
We implemented this algorithm, using the numerical partial derivatives, for the interactive measurement of CSG models and found that the hiddenline projection is the bottleneck for the fast measurement of a model. Especially for more complex CSG models, the time spent in each iteration, as mentioned above, increases rapidly. Not only the hidden-line projection takes more time to compute for a complex model, it also has to be repeated for more parameters to estimate all the partial derivatives. (And for all the images that are used for the measurements.) Lowe (1991) analytically computes the partial derivatives of the pose parameters of a fixed CAD model, i.e. without shape parameters. We extend this method by introducing the analytical partial derivatives for generic CSG models. The analytical method is far more efficient than the numerical one and thus greatly facilitates the interactive measurement of CSG models. Another drawback of the numerical method is its lack of robustness, since the projection of a modified model may show different edges than the original model and thus it will not be possible to compute the partial derivative of a point on such a 'disappearing' edge. On the other hand, the numerical method is applicable to any parameterized CAD model, while our analytical method can only be applied to CSG models, which is a subset of all parameterized CAD models. It is, however, a large subset and many man-made objects can be modeled accurately using CSG models. It is also not unlikely that, in the future, analytical partial derivatives for other classes of CAD models will be derived along the same lines as the partial derivatives for CSG models presented in this paper.
4. ANALYTICAL PARTIAL DERIVATIVES A CSG model is constructed from one or more primitives. Section 4.1 focuses on the partial derivatives of individual CSG primitives, while section 4.2 discusses the partial derivatives of complex CSG trees. How to get from a three-dimensional direction to the partial derivative used in (1), is explained in section 4.3. Section 4.4 gives some details about the practical implementation of the partial derivatives.
The position of a contour edge depends on the viewing direction towards the curved surface. Figure 4 shows that changing the position of the circle, shifts the contour points over the circle. Although this introduces an effect (non-linear) in the translation of a contour edge, it has no effect on the partial derivative of the translation because the direction of the instantaneous shift equals the viewing direction. This shifting effect may also occur due to a change in orientation or shape. As a contour edge is connected to a curved surface, the partial derivative of a contour edge equals the partial derivative of that surface.
a
b
c
Figure 3: (a) Partial derivatives of a translation parameter of a cylinder from a given viewpoint. The partial derivative is shown at sampled points along the edges of the projected cylinder. (b) Partial derivatives of a rotation parameter. (c) Partial derivatives of the radius, a shape parameter.
projectio n centre
projection centre contou r points
translation
new contour points
Figure 4: The contour points on a circle shift over the circle due to a translation.
4.1 Partial derivatives of CSG primitives There are two types of edges visible in the hidden-line projection of a primitive; the edges that are part of the primitive itself, and the contours of curved surfaces. In this paper they are referred to as primitive edges and contour edges, respectively. Figure 3 shows the partial derivatives of some parameters of a cylinder. Each CSG primitive has a pose in the world, and a point on a primitive can be expressed in its own, local, coordinates, as well as in world coordinates. The parameters of a CSG primitive fall into three categories; translation, rotation and shape parameters, for which each partial derivative is calculated in a different way. All yield a 3D vector in world coordinates.
Translation parameters The translation parameters (x,y,z) position the primitive in the world. The calculation of the partial derivative of a translation parameter is straightforward and applicable to both the primitive and the contour edges. For example, the partial derivative of the x parameter is (1,0,0). Rotation parameters The choice for a parameterization of the rotation matrix is an important one, because singularities in the representation need to be avoided (Shuster, 1993). Iterations of the linearised observation equations (1) at a
singular point, or in the neighborhood of a singular point, may result in slow convergence or no convergence at all. The quaternion parameterization is chosen because it provides a singularity-free representation with only one redundant parameter. This overparameterization can be removed by demanding that the length of the quaternion must equal 1 (Shih, 1990). The conversion from the four quaternion parameters, also known as the Euler-Rodrigues symmetric parameters,
q4 , to the 3*3 rotation matrix R
q1 , q2 , q3 and
is given by:
q12 − q22 − q32 + q42 2(q1q2 + q3q4 ) 2(q1q3 − q2q4 ) (2) R = 2(q1q2 − q3q4 ) − q12 + q22 − q32 + q42 2(q2q3 + q1q4 ) 2(q1q3 + q2q4 ) − q12 − q22 + q32 + q42 2(q2q3 − q1q4 )
R , with respect to the ∂R / ∂q i , are calculated by taking
The partial derivatives of quaternion elements
the partial derivative of each element in the matrix individually. The partial derivative of a point s on an edge of the hidden-line projection is then calculated as,
∂s ∂R s, = ∂qi ∂qi
(3)
in which the position of s is expressed in the local coordinates of that primitive. Expression (3) is applicable both to primitive edges and to contour edges. Several CSG primitives, such as cylinders, are symmetric around an axis and, in such cases, the quaternion representation is overparameterized even more. However, adding the initial values of the individual parameters as observations to the observation equations, as discussed in section 3, stabilizes this overparameterization. It is obvious that the rotation parameters can be omitted when the CSG primitive is a sphere. Shape parameters Each CSG primitive has different shape parameters and, within a primitive, each parameter will have a different effect on each edge and on each surface. The primitive has to be constructed in a local coordinate system, after which it is transformed to a world coordinate system, using the pose parameters described above. The partial derivatives of the shape parameters depend on the choice of the local coordinate system in the construction stage. As an example, the partial derivatives of a cylinder, with a radius r and a length l, are discussed here. The z-axis is chosen to coincide with the symmetry axis of the cylinder, the bottom of the cylinder is placed at z=0 and the top at z=l. For each surface and each edge this results in a partial derivative for each parameter. The partial derivative with respect to the radius r of a point s in local coordinates,
( x, y , z ) ,
on either the primitive
edges or the contour edges, becomes (Figure 3c):
x / r ∂s = R y / r ∂r 0
(4)
In which R is the rotation matrix mentioned above used to transform the directional derivative from local coordinates to world coordinates. The partial derivatives of the planar surfaces at the top and bottom of the cylinder are zero. The length parameter only affects the edge and surface on top of the cylinder; their partial derivative is (0,0,1), in local coordinates. The partial derivatives with respect to the length of all other edges and surfaces are zero. Although a planar surface of a primitive will never yield a contour edge, its partial derivatives are used when the surface is intersected with the surface of another primitive in the CSG tree, as is discussed in the next section. 4.2 Partial derivatives of a CSG tree Given the structure and operations of a CSG tree, and the type and parameters of the primitives, a Brep model can be constructed. In a Boolean operation between two primitives, surfaces of the primitives are intersected and new edges are constructed along the intersection curves. This type of edge needs to be distinguished from the primitive and the contour edge, and will therefore be referred to as an intersection edge. surface B
surface A shift of edge
shift of A
intersection edge A-B
Figure 5: Shift of an intersection edge. The intersection edge shifts over surface B due to a shift of surface A.
Since the intersection edge is created from the intersection of two surfaces from two different CSG primitives, its position and partial derivatives depend on the parameters of both primitives. Figure 5 shows the displacement of an intersection edge when one of the surfaces is shifted, the intersection edge slides upwards, following the second surface. Figure 6 shows the partial derivatives of a radius of a T-junction; note the intersection edge between the two cylindrical surfaces. It can be derived that the partial derivative of a point s on an intersection edge between surface
A
and surface
B,
pA ,
with respect to
surface
A , becomes
∂s = ∂p A in which
a parameter of the primitive of
nA B,
and
nB
nB ×
∂A × nB ∂p A
nB × nA
2
,
(5)
are the normals of surface
A
and
Coinciding surfaces and edges Special situations occur when two surfaces coincide, a surface coincides with an edge or when two edges coincide. See Figure 7 for some examples. Since the angle between the coinciding surfaces in 7a equals zero, no measurements can be done on that edge according to the criterion described in (6). In figure 7b, the coinciding edge can be classified as either a primitive edge or an intersection edge, while in 7c the coinciding edge can be classified as belonging to either one of the primitives.
surface respectively. It is assumed that all the directions in (5) are in world coordinates. In the above expression, the derivative
∂A / ∂pA
needs to be
perpendicular to surface A . If this is not the case, the part that is aligned with the normal of surface A can be calculated using the three-dimensional equivalent of expression (9). Note that a Boolean operation in the CSG tree does not affect the partial derivative of a surface or a primitive edge. The subtraction operation however, reverses the normals of the surfaces of the second operand. The direction of the intersection edge can be calculated by the vector product of the normals of both surfaces, which can be used in (9) to decompose the partial derivative in a parallel and perpendicular part. From Figure 5, or expression (5), it can be seen that the partial derivative of an intersection edge approaches infinity when the angle of intersection of both surfaces, α , approaches zero or 180 degrees. To avoid extreme values for a derivative and to avoid numerical singularities in the computation of (5) a threshold is introduced:
sin(α ) = n B × n A > 0.001 , assuming that
nA
and
nB
(6)
have unit length. If the two
surfaces at a point on the intersection edge do not pass this test, no partial derivatives can be computed and, as a result, this point cannot be used for measurements.
a
b
c
Figure 7: Examples of coinciding surfaces and edges. (a) Two surfaces touch. (b) An edge coincides with a surface. (c) Two edges coincide. All these situations can be seen as a transition stage between two 'regular' situations. In the classification of such an edge, there are two, or more, possible outcomes, but the choice is not relevant. See Figure 8 for an example.
a
b
c
Figure 8: Two regular situations (a and c), with the transition stage (b) in between. 4.3 Projection of direction in images The partial derivative of a point on an edge relates the change in position of that point to a change of a specific parameter. The partial derivative is a 3D vector indicating a direction and, with a known exterior orientation, it can be rotated to camera coordinates. (As it is a direction, it need not be translated.) Assuming a pinhole camera model, and a known interior orientation, a point
( xc , y c , z c )
in camera
coordinates can be projected into the image plane to produce an image point
( xi , yi ) :
xi = − f Figure 6: Partial derivatives of the radius of one of the cylinders of a T-junction.
xc zc
y yi = − f c zc
(7)
Where
f
is the focal length of the camera.
The
partial
derivative
parameter, with its elements
with
respect
( xu , yu , zu ) ,
to
a
is projected
into the image plane (Lowe, 1991), according to:
∂xi f x = − xu − zu c , ∂p j zc zc
(8)
∂yi f y = − yu − zu c ∂p j zc zc
The observations in (1) are perpendicular distances between measured points and model edges. Therefore, the perpendicular component of the partial derivatives with respect to the model edges must also be taken into account. The direction of the model edge is projected into the image plane, according to (8), in order to decompose the projected derivative perpendicular part,
d
into a parallel part
d par
and a
d perp . d par =
e⋅d e
2
e, (9)
d perp = d − d par In which
e
link to the CSG-tree structures. To identify the edges and surfaces in the B-rep model, all the edges and surfaces of a CSG primitive are labeled with a name and a link to the primitive. The name identifies the type of edge or surface, e.g. the edges of a cylinder are named 'top' and 'bottom'. Together with the link to the primitive, which also includes its parameters, the partial derivatives of the pose and shape parameters can be computed. The labels are assigned to the ACIS data structures when the B-rep of the primitive is constructed, and ACIS maintains these labels during the construction of the B-rep of the whole CSG tree. Since only primitive edges have labels attached to them, intersection edges can be identified by the absence of labels. The labels of the surfaces that are adjacent to the intersection edges carry the information for the computation of the partial derivatives. Which surfaces are adjacent to an intersection edge can be extracted from the B-rep data model of ACIS.
5. EXPERIMENTS To compare the difference in speed between the numerical and the analytical method for the computation of the partial derivatives, experiments were done with a cylinder and a flanged T-junction (Figure 9). Each iteration in the measurements, described in section 2, solves the observation equations, constructs a new model, computes a hidden-line projection of the new model in the images, and displays the new projections onto the images.
is the direction of the model edge projected
into the image plane. The signed length of
d perp
is used
in (1) as partial derivative, where the sign is given by the following expression:
sign (x perp y e − y perp x e )
(10)
x perp , y perp and xe , y e are the elements of d perp and e , respectively. The sign of the perpendicular distance from the measured point to the model edge is calculated in a similar way; comparing a vector from the model edge to the measured point, to the direction of the edge. 4.4 Implementation issues The application for the measurement of CSG models is built with the use of ACIS, which is an objectoriented software library (C++) containing a lot of CAD functionality. ACIS was used for the conversion from our CSG-tree structures to a B-rep model to a hidden-line projection. To compute the partial derivative of an edge projected in the image, a relation has to be established between that edge and the relevant CSG primitives. The data model of ACIS provides a relation between the visible lines of the hidden-line projection and the edges or surfaces of the B-rep model, but does not provide a direct
Figure 9: A flanged T-junction. For a single cylinder, one iteration with two images, using the analytical partial derivatives, takes about 0.3 seconds on a SGI O2 (R10000, 175 MHz) workstation. If the partial derivatives are computed numerically, a slightly modified cylinder has to be constructed for each parameter, and each modified cylinder has to be projected into the two images. The number of parameters for a cylinder is 9: the position (3 parameters), the orientation (4 quaternion elements), the length and the radius. The construction of a cylinder and the hidden-line projection are both done by ACIS, and take 0.5 msec and 12.5 msec, respectively. This adds up to (9*0.5 + 2*9*12.5) 230 msec, and together with the 0.3
seconds ‘overhead’ the update frequency slows down to less than 2 iterations per second. For models that are more complex, the difference in computation times becomes even bigger. An iteration of a flanged T-junction (Figure 9), with two images, takes 1.2 seconds using the analytical partial derivatives, in which the construction takes 140 msec and the hidden-line projection takes 112 msec to compute. The T-junction is modeled with 5 cylinders, each with 9 parameters, and, therefore, the computation of the numerical partial derivatives will take more than (45*140 + 2*45*112) 16 seconds. This is much too long to be used in an interactive measurement system and clearly shows the benefit of the analytical method for the computation of the partial derivatives.
6. DISCUSSION A method for the interactive measurement of CSG models from images, using observation equations, is discussed, and an analytical method for the computation of the partial derivatives in the observation equations is presented. This method is shown to be more robust and faster than the numerical method used previously. The method can be used in interactive measurements as well as in automated fitting procedures. The interactive measurement method, using the analytical partial derivatives, is implemented and can be used for the measurement of CSG models. The T-junction in Figure 9 is modeled with 5 cylinders, each with 9 independent parameters, resulting in a total of 45 parameters. For a real-world T-junction, however, several dependencies between the parameters can be formulated. For example, the lengths and the radii of the three flanges are the same, and the position and the orientation of the flanges depends on the pose of the two main cylinders, etc. The current research is directed towards the application of such constraints within CSG models and also between CSG models. The accuracy of the measurement method, in combination with the constraints, will also be addressed.
ACKNOWLEDGEMENT This research is supported by the Dutch Technology Foundation (STW).
REFERENCES P. Ermes, F. van den Heuvel, 1998, “Measurement of Piping Installations with Digital Photogrammetry”, International Archives of Photogrammetry and Remote Sensing, Vol. 32, part 5, pp. 217-220 M.A. Jones, D.P. Chapman, A.A. Hamid, A.T.D. Deacon, 1996, "Close Range Photogrammetry Using Geometrical Primitives for Efficient CAD Modelling of Industrial Plant", International Archives of Photogrammetry and Remote Sensing, Vol XXXI, Part B5, Vienna. D.G. Lowe, 1991, "Fitting Parameterized ThreeDimensional Models to Images", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 13, no 5, pp441-450. T.-Y. Shih, 1990, “The Duality and Critical Condition in the Formulation and Decomposition of a Rotation Matrix”, Photogrammetric Engineering and Remote Sensing ASPRS, Vol. 56, no 8, pp1173-1179 M.D. Shuster, 1993, "A Survey of Attitude Representations", Journal of the Astronautical Sciences, Vol 41, no 4, pp439-517. J.W.H. Tangelder, P. Ermes, F.A van den Heuvel, G. Vosselman, 1999, “Measurement of Curved Objects Using Gradient Based Fitting and CSG Models”, International Archives of Photogrammetry and Remote Sensing, Commision V, WG 2, Thessaloniki. H. Veldhuis, G. Vosselman, 1998, "The 3D Reconstruction of Straight and Curved Pipes Using Digital Line Photogrammetry.", ISPRS Journal of Photogrammetry and Remote Sensing 53, no 1, pp6-16 G. Vosselman, 1998, "Interactive Alignment of Parameterized Object Models to Images", International Archives of Photogrammetry and Remote Sensing, Commission III WG 3, vol. 32, no 3/1, pp272-278.
