ARTICLE IN PRESS
Pattern Recognition
(
)
– www.elsevier.com/locate/patcog
A graph-spectral method for surface height recovery Antonio Robles-Kellya,∗ , Edwin R. Hancockb a NICTA, Building 115, ANU, 0200 ACT, Australia b Department of Computer Science, University of York, York YO1 5DD, UK
Received 24 February 2004; received in revised form 1 December 2004; accepted 1 December 2004
Abstract This paper describes a graph-spectral method for 3D surface integration. The algorithm takes as its input a 2D field of surface normal estimates, delivered, for instance, by a shape-from-shading or shape-from-texture procedure. We commence by using the surface normals to obtain an affinity weight matrix whose elements are related to the surface curvature. The weight matrix is used to compute a row-normalized transition probability matrix, and we pose the recovery of the integration path as that of finding the steady-state random walk for the Markov chain defined by this matrix. The steady-state random walk is given by the leading eigenvector of the original affinity weight matrix. By threading the surface normals together along the path specified by the magnitude order of the components of the leading eigenvector we perform surface integration. The height increments along the path are simply related to the traversed path length and the slope of the local tangent plane. The method is evaluated on needle-maps delivered by a shape-from-shading algorithm applied to real-world data and also on synthetic data. The method is compared with the local geometric height reconstruction method of Bors, Hancock and Wilson, and the global methods of Horn and Brooks and Frankot and Chellappa. 䉷 2005 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. Keywords: Surface reconstruction; Markov chain; Curvature; Surface normals
1. Introduction Surface integration is a process that provides the means of converting a field of surface normals (i.e. the projection of the Gauss map of a surface from a unit sphere onto an image plane) into an explicit 3D surface representation. This problem of reconstructing the surface from its Gauss map arises when attempting to infer explicit surface structure from the output of low-level vision modules such as shape-from-shading and shape-from-texture [1]. Although the surface integration process is sometimes an integral part of the shape-from-shading or shape-from-texture algorithm,
∗ Corresponding author.
E-mail addresses:
[email protected] (A. Robles-Kelly),
[email protected] (E.R. Hancock).
it is of course an important and well-studied problem in its own right. The process involves selecting a path through the surface normal locations. This may be done using either a curvature minimising path or by advancing a wavefront from the occluding boundary or singular points. By traversing the path, the surface may be reconstructed by incrementing the height function using the known distance travelled and the local slope of the surface tangent plane. This is a matter of straightforward trigonometry. It must be stressed that the choice of the path can be critical. If the surface does not satisfy the integrability constraint, (i.e. the x derivative of the y-component of the surface normal must be equal to the y derivative of the x-component of the surface normal, and hence the Hessian must be symmetric), then the shape of the reconstructed surface is affected by the choice of integration path.
0031-3203/$30.00 䉷 2005 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.patcog.2004.12.005
ARTICLE IN PRESS 2
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
1.1. Related literature The analysis of the literature on the topic of surface integration is not a straightforward task. The reason for this is that surface recovery is frequently viewed as an integral part of the shape-from-shading or shape-from-texture process. However, with this caveat in mind we briefly discus some of the available algorithms. Broadly speaking, the methods available can be divided into those that are local and those that are global in nature. Local methods generally involve neighbourhood search to locate the best move for the integration path. Global methods pose the problem of recovering the integration path as that of minimising an energy function defined over the complete set of pixel sites. Of course, the energy minimisation problem although posed in a global setting, for computational reasons, usually relies on a search algorithm that will only locate a local optimum. We commence by considering local methods. The most direct method of height recovery is path-based. In the original work on shape-from-shading by Horn and Brooks [2,3], the surface height recovery process is applied as a postprocessing step. The process proceeds from the occluding boundary and involves incrementing the surface height by an amount determined by the distance traversed and the slope angle of the local tangent plane. Unfortunately, this process is highly sensitive to noise, and errors in surface normal direction accumulate rapidly as the front propagates inwards. In an attempt to regulate the errors, Wu and Li [4] average the surface normal directions to obtain height estimates. In recent work, Bors et al. [5] have developed a local method that uses the trapezoid rule to locate equal height commencing from reference points on the surface. More sophisticated path-based methods may be developed if differential geometry is used. For instance, Dupuis and Oliensis [6] have described a method which involves propagation in the direction of the steepest gradient from singular points. A fast variant of this algorithm is described by Bichsel and Pentland [7] who compute the relative height of the surface with respect to the highest intensity point. As noted above, the global approach to surface reconstruction poses the problem of height recovery as one of energy minimisation. Leclerc and Bobick [8] have developed a direct numerical method for height recovery from shading information which uses curvature consistency constraints. In related work, Tsai and Shah [9] describe a fast surface height recovery method, which works well except at the locations of self-shadows and numerically singular points. Recent work by Karacali and Snyder [10,11] has shown how surfaces can be reconstructed under partial integrability constraints, and this analysis is applicable when surface height discontinuities are present. The recovery of the surface height function can be also be posed as the solution of differential equation. In the levelset method of Kimmel, Bruckstein, Kimmia and Siddiqi [12,13] surface reconstruction is incorporated as an integral
(
)
–
component into the shape-from-shading process. They use the apparatus of level-set theory to simultaneously solve the image irradiance equation and recover the associated surface height-function under constraints provided by surface integrability. An elegant and ingenious solution is proposed by Frankot and Chellappa [14] who project the surface normals into the Fourier domain to impose integrability constraints and hence recover surface height. There have been a number of extensions to this work. For instance, Noakes et al. [15] have developed a so-called lawn-mower algorithm which can be used to impose integrability constraints on an initially nonintegrable field of surface normals. Wei et al. [16] describe a new regularisation framework for surface height recovery from potentially noisy fields of surface normals. Their cost function combines surface curvature and integrability constraints into a unified utility measure. Several of these methods are described in more detail and compared in the recent comprehensive review paper of Zhang et al. [17]. 1.2. Motivation The recovery of the integration path is clearly one of optimisation, which can be solved using a number of techniques including direct numerical methods and level set techniques. However, one of the methods that has not received attention is that of posing the method in a graph-spectral setting and using eigenvector methods to recover the solution. The idea underpinning graph-spectral methods is to abstract the problem in hand using a weighted graph. Here the nodes represent the basic image entities, and the weighted edges represent affinity relations between the entities. By computing the eigenvalues and eigenvectors of the weight matrix, it is possible to find groups or clusters of entities. The graph-spectral method is in fact one of energy minimisation since the eigenvectors can be shown to be minimisers of a quadratic form. In fact, graph-spectral methods have recently proved highly effective in image processing and computer vision. Perhaps the best-known method is that of Shi and Malik [18] which has shown how to locate image regions by recursively bisecting a weighted graph that represents the affinity of pairs of pixels. The method is based on the normalised cut. This is a measure of the relative weight of the edges connecting the two parts of the partition (the cut) to the weight assigned to the edges within the two parts of the bisection (the association). A relaxed solution to the bisection problem is found by locating the Fielder vector, i.e. the eigenvector associated with the second smallest eigenvalue of the Laplacian matrix (the degree matrix minus the affinity weight matrix). Although it is convenient to work with the Laplacian, since it is positive semi-definite, grouping and segmentation can also be performed using an edge-weight or affinity matrix. For instance, both Sarkar and Boyer [19] and Perona and Freeman [20] have developed matrix factorisation methods for line-segment grouping that use eigenvectors of an affin-
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
ity matrix rather than the associated Laplacian. The Sarkar and Boyer [19] method can be understood as maximising the association (i.e. the total edge weight) of the clusters. The methods described above all share the feature of using the eigenvectors of a Laplacian or an affinity matrix to define groups or clusters or objects. However, graph spectral methods can also be used for path analysis tasks on graphs. For instance, it is well known that the path length distribution can be computed from the spectrum of eigenvalues of the adjacency matrix [21]. Ideas from spectral graph theory have also been used to analyse the behaviour of random walks on graphs [22–24]. The observation underpinning this work is that random walks on a weighted graph can be represented as Markov chains in which the transition probabilities are computed from the normalised edge weights. The problem investigated is to compute the transition probability between pairs of pixel sites after a large number of time steps have elapsed. This study has lead to a number of interesting findings. Of direct relevance to this paper is the fact that the steady-state random walk on the graph is characterised by the leading eigenvector of the normalised edge weight matrix. In addition, there are important relationships between the eigenvectors of the edge weight matrix and other quantities related to random walks. These include the access time for a node (i.e. the expected number of time steps that must have elapsed before the node is visited) and the mixing rate (i.e. the rate at which the random walk converges to its steady state). One of the most significant recent results has been to establish the conditions under which the mixing is rapid [25]. The relationship between the leading eigenvector of the edge weight matrix and the steady-state random walk has been exploited in a number areas including routeing theory and information retrieval [26,27]. The advantage of graph-spectral methods is that they can be used to find approximate or relaxed solutions without the need for parallel iterative updates at the pixel sites. The method also obviates the need for complex search algorithms. However, although they have been applied to region segmentation and grouping problems, graph-spectral methods have not been applied to curve detection problems of the sort that arise in the determination of the optimal integration path. 1.3. Contribution Hence, in this paper we pose the recovery of the surface integration path in a graph-spectral setting. Our starting point is an energy functional that introduces penalties associated with the total squared surface curvature of the path. The required curvature may be computed from the change in surface normal direction between locations on the image plane. We use the energy function to compute an affinity weight matrix for the sites on the pixel lattice. According to our graph-based abstraction of the surface reconstruction process, each pixel-site is represented by a node, the edge structure is determined by connectivity on the pixel lattice,
(
)
–
3
and the edge-weights are determined by the change in surface normals directions. Our aim is to use the steady-state random walk on the graph as an integration path. With our construction of the weight matrix this path minimises surface curvature across the field of available surface normals. Using the degree matrix, i.e. the matrix whose diagonal elements are equal to the total weight of the edges connected to a node, we compute a transition probability matrix in which the elements in each row sum to unity. The steady-state random walk associated with the transition probability matrix is characterised by the leading eigenvector of the normalised affinity matrix. The components of the eigenvector are used to sort the pixel sites into an order which corresponds to the probability of visiting the pixel sites in the steady-state random walk. However, the path defined by the node order is not a connected path on the pixel lattice. To locate an edge-connected path on the pixel lattice, we use instead the components of the leading eigenvector to sort the pixels in each pixel neighbourhood. The integration path proceeds across the pixel lattice by always moving in the direction of the highest rank connected pixel. We perform surface height recovery by visiting the surface normals in the order defined by this path. As the path is traversed, then the height function is incremented by an amount determined by the slope of the local tangent plane. Following Sarkar and Boyer [19], the leading eigenvector may also be used to segment the surface into patches. Those pixel sites which are associated with negligible eigenvector components are not visited by the path, and are not associated with the most significant patch on the surface. Once the most significant patch has been identified and reconstructed in this way, then the elements of the associated pixel sites in the transition probability matrix are set to zero. We then repeat the procedure described above to extract each patch in turn. This process is halted when the size of the remaining patches becomes insignificant. The resulting surface integration algorithm has both local and global features. The graph-spectral method locates an ordering of the sites on the pixel lattice that minimise a quadratic energy function. However, this ordering is not guaranteed to be connected, and a local search procedure must be used to ensure connectivity. In subsequent work we have presented a shape-from-shading method that aims to overcome some of these shortcomings [28]. Rather than commencing from a specification of the recovery of the integration path in terms of a random walk, we couch the process in an energy minimisation setting. The minimisation problem is formulated so as to encourage path connectivity. We show how the recovery of the surface height function that best satisfies Lambert’s law can be posed as an approximate solution of this so-called graph seriation problem. The solution is recovered in an iterative manner using the eigenvector of a property matrix. The work reported in this paper represents a part of an ongoing and multi-pronged approach to the general problem of surface integration and its use in the context of
ARTICLE IN PRESS 4
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
shape-from-shading. Our work on the use of eigenvector methods was first reported in Ref. [29]. The work has since been considerably refined and developed in a number of directions [30,28]. Compared with the original conference account in Ref. [29], the work presented here represents an archival account of the method. It includes a detailed and definitive account of the underlying theory, a detailed development of the model, and an exhaustive set of experiments, which include comparisons with alternative methods. The work here is confined to surface integration, and takes a field of surface normals as input. In Ref. [28], we report an integrated approach to shape-from-shading where both, the surface normals and the integrated height map, are recovered. This is based on a graph-seriation technique, and we demonstrate how the solution to the underlying optimisation problem can be located using an eigenvector method. However, because of the way in which the optimisation process is posed, the matrix setting is rather different to the one reported in this paper. Hence, although the two methods share the feature of being based upon graph-spectral theory, the Markov chain setting used in this paper is probabilistic rather than an optimisation one. Further, by adopting a Markovian approach to the problem, the surface reconstruction process can be cast into a stochastic framework which allows the use of concepts such as reversibility, martingales and coupling for purposes of analysing further the behaviour of the surface height recovery method presented here. Finally, we note that both, the method described in this paper and the seriation method described in Ref. [28] are of more general applicability to problems concerning path analysis in graphs. Both methods have been applied to problems of graph-matching too [31,32]. The outline of this paper is follows. In Section 2, we present the differential geometry which underpins the construction of our curvature-based affinity matrix. Section 3 explains how the leading eigenvector of the affinity matrix can be used to construct an integration path. In Section 4, we demonstrate some of the properties of the integration path from energy minimisation and geometric perspectives. In Section 5 we describe how patches can be extracted from the field of surface normals. Section 6 outlines the geometry necessary to reconstruct the height function of the surface from this path and the raw needle-map information. Experiments on real-world image data and simulation data are described in Section 7. Finally, Section 8 offers some conclusions and suggests directions for future investigation.
2. Curvature-dependant weights Our aim is to reconstruct the height function for the surface S from a planar field of surface normals, under the assumption that the image of the surface is formed under orthographic projection. To realise this goal, we require an integration path. This path must traverse or connect the sites of the pixel lattice. By traversing the path, the relative
(
)
–
surface height function can be reconstructed. This is done using the trapezium rule to increment the height using the distance travelled on the path and the known slant and tilt angles of the surface normals at different locations on the image plane. In the work reported here the path is one that optimises a graph-spectral criterion that penalises high curvature. To this end, we require a means of gauging the affinity of pixels based on an image plane approximation to the surface curvature. Formally, our aim here is to find a curve across the image plane that can be used as an integration path to reconstruct the height-function of the surface S. The planecurve can be viewed as the projection of a geodesic curve S from the surface S onto the plane . Whereas the curve S has both curvature and torsion, the planar curve has only curvature. Further, suppose that (s) is the sectional curvature of the curve S at the point Q with parametric coordinate s. Dropping the iteration-dependant superscript used in the previous section, for the surface S the field of unit surface normals is sampled on the image plane , and the surface normal at the location Q ∈ is denoted by N Q . Accordingly, and following do Carmo [33], we let TQ (S) represent the tangent plane to the surface S at the point Q which belongs to the curve S . To compute the sectional curvature (s) we require the differential of the surface or Hessian matrix dNq : TQ (S) → TQ (S). The maximum and minimum eigenvalues 1 and 2 of dN Q are the principal curvatures at the point Q. The corresponding eigenvectors e1 ∈ TQ (S) and e2 ∈ TQ (S) form an orthogonal basis on the tangent plane TQ (S).At the point Q the unit normal vector to the curve S is n and the unit tangent vector is tQ ∈ TQ (S). The sectional curvature of S at Q is given by (s) =
(tQ · e1 )2 (1 − 2 ) + 2 , n · N Q
(1)
where (tQ · e1 )2 (1 − 2 ) + 2 is the normal curvature and = arccos n · N Q is the angle between the curve normal and the surface normal. Our aim is to approximate the curvature (s) using the field of surface normals on the plane . In practice, we deal with points which are positioned at discrete positions on the pixel lattice. Suppose that i and j are the pixel indices of neighbouring points sampled on the pixel lattice. To approximate the sectional curvature we make use of the surface normal directions. To commence, we note that if the path between the locations i and j can be approximated by a circle of radius R on the surface, then the approximate sectional curvature is |ˆ i,j | = 1/Ri,j . Suppose that the surface normal directions at the pixel locations i and j are respectively N i and N j . Further suppose that the line connecting the pixel sites on the image plane is of length si,j . The approximating circle connects the points i and j, and has si,j as the connecting chord. Hence, the change in direction of the radius vector of the circle is i,j = arccos Ni · N j , and
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
5
3. Graph-spectral analysis
Fig. 1. Illustration of the curvature computation and the integration path (see text for details).
as a result cos i,j = N i · N j . If the angle i,j is small, then we can make the Maclaurin approximation cos i,j 1 − 2i,j /2 = Ni · N j . Moreover, the small angle approximation to the radius of curvature of the circle is Ri,j =si,j /i,j and hence 2(1 − N i · N j ) ˆ 2i,j = . 2 si,j
(2)
The geometry outlined above is illustrated in Fig. 1a. In the next section of this paper, we will present a graphspectral technique for locating the integration path . The starting point for this analysis is a weight matrix that can be used to represent the affinity of pixels based on the curvature of the connecting path, or the difference in surface normal directions. To compute the elements of the affinity matrix, we associate with the pair of pixels a cost or energy that is equal to the squared product of the distance between the sites and sectional curvature of the connecting path, i.e. 2 . Ei,j = ˆ 2i,j li,j Using the approximation to the sectional curvature, we find that the cost associated with the step from the pixel i to the pixel j is Ei,j = 2(1 − N i .N j ).
(3)
With the energy function to hand, we could attempt to find the integration path S that minimises the path energy, El,l+1 E S = l∈s
ˆ
where Ni is the set of pixels-neighbours of the pixel i and is a constant. Hence, the curvature-dependent weight is only non-zero if pixels abut one-another. Moreover, the weights are unity if the sectional curvature is zero and tend to zero as the curvature increases. The set of pixel sites can hence be viewed as a weighted graph G = (V , E, W ) with indexset V, edge-set E ={(i, j )|(i, j ) ∈ V ×V , i = j } and weight function W : E → [0, 1]. The matrix W cannot be used as a transition probability matrix since its rows do not sum to unity. To normalise the rows of the matrix we compute the degree of each node deg(i) =
|V |
W (i, j ).
j =1
With the diagonal degree matrix D = diag(deg(1), deg(2), . . . , deg(|V |)) to hand, the transition probability matrix is given by P = D −1 W . The elements of the transition matrix are hence given by Pi,j = (1/deg(i))Wi,j . It is interesting to note that the transition matrix P is a row stochastic matrix. Moreover, it is related to the normalised symmetric positive definite matrix Wˆ = D −1/2 W D −1/2 = D 1/2 P D −1/2 . As a result, we can write
and satisfies the condition ˆ S = arg min E().
In the previous section we showed how to compute the energy associated with paths through the field of surface normals. In this section we explore how the graph-spectral analysis of a transition probability matrix computed from the energy can be used to locate an integration path for the purposes of height recovery. To pursue the graph-spectral analysis of the field of surface normals, we require a weight matrix which reflects both the sectional curvature and the connectivity of the pixel lattice. For the pixels indexed i and j we define the weight matrix to have elements exp[−Ei,j ] if j ∈ Ni , W (i, j ) = (5) 0 otherwise,
(4)
There are clearly a number of ways in which the energy could be minimised. These might include expectationmaximisation [34], relaxation labelling [35] or stochastic methods [36]. However, here we use the energy function to compute site transition probabilities, and from these probabilities we construct a Markov chain on the sites of the field of surface normals. We use the steady-state random walk associated with the Markov chain to locate the integration path. The steady-state random walk can be found by computing the leading eigenvector of the transition probability matrix for the Markov chain.
P = D −1/2 Wˆ D 1/2 . It is worth noting in passing that the matrix Wˆ is related to the normalised Laplacian L = D −1/2 (D − W )D −1/2 = I − D −1/2 W D −1/2 = I − Wˆ . Our aim is to use the steady-state random walk on the graph G as an integration path for surface height recovery. The walk commences at the pixel j1 and proceeds via the sequence of pixel sites = {j1 , j2 , j3 , . . .}. If the random walk can be represented by a Markov chain with transition
ARTICLE IN PRESS 6
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
matrix P, then the probability of visiting the pixel sites in the sequence above is PS = P (j1 )
l∈S
Pjl+1 ,jl =
Wjl+1 ,jl . deg(l)
l∈S
Substituting for the path energy, we have that exp −2 l∈S {1 − N l .N l+1 } P S = l∈S deg(l) 1 exp[−ES ], = Z S where ES = 2 and Z S =
l∈S
{1 − N l .N l+1 }
deg(l).
l∈S
Hence, the integration path is a Markov chain with energy function ES and partition function ZS . Further, let Qt (i) be the probability of visiting the pixel site indexed i after tsteps of the random walk and let Qt = (Qt (1), Qt (2), . . . )T be the vector whose components are the probabilities of visiting the sites at time t. After t time steps we have that Qt =P t Q0 . If Wˆ t is the result of multiplying the symmetric positive definite matrix Wˆ by itself t times, then P t = D −1/2 Wˆ t D 1/2 . To develop a spectral method for locating the steady-state random walk, we turn to the spectral decomposition of the normalised affinity matrix Wˆ Wˆ = D −1/2 W D −1/2 =
|V | i=1
)
–
The leading eigenvalue of Wˆ is unity, i.e. ∗ = 1. Furthermore, from spectral graph theory [23] provided that the graph G is not a bipartite graph, then the smallest eigenvalue |V | > − 1. As a result, when the Markov chain approaches its steady state, i.e. t → ∞, then all but the first term in the above series become negligible. Hence, the steady-state random walk is given by T
−1/2 Q . Qs = lim Qt = D 1/2 0 ∗ ∗D t→∞
This establishes that the leading eigenvector of the normalised affinity matrix Wˆ determines the steady state of the random walk. It is also important to note that the equilibrium equation for the Markov process is Qs = P Qs , where Qs is the vector of steady-state site visitation probabilities. Hence, since the leading eigenvalue of P is unity, then it follows that Qs is the leading eigenvector of P. For a more complete proof of this result see the book by Varga [37] or the review of Lovasz [22]. We aim to visit the pixel sites on the lattice in the order of their steady-state state probabilities. Suppose that the initial state vector for the sites is uniform, i.e. Q0 =
1 T 1 ,..., . |V | |V |
As a result the steady-state probability of visiting the pixel site i is |V | 1 deg(j ) Qs (i) = (i)∗ (j ) |V | deg(i) ∗ j =1
=
|V | 1 ∗ (i) deg(j )∗ (j ). √ |V | deg(i) j =1
T i i i,
are where the i are the eigenvalues of Wˆ and the i the corresponding eigenvectors. By constructing the ma | ˆ trix = ( 1 2 | . . . ||V | ) with the eigenvectors of W as columns and the matrix =diag(1 , 2 , . . . , |V | ) with the eigenvalues as diagonal elements, we can write the spectral decomposition in the more compact form Wˆ = T . Since, the eigenvectors of Wˆ are orthonormal, i.e. T = I , we can have that Wˆ t = t T . Substituting the spectral expansion of the matrix Wˆ into the expression for the state-vector of the random walk at time step t, we find Qt = D −1/2 t T D 1/2 Q0 |V | 1 T D 1/2 Q0 . = ti D − 2 i i i=1
(
Since the summation appearing above is the same for all pixel sites, the probability√rank order is determined by the ˆ (i) = (i)/ deg(i). Hence, it is the scaled quantity ∗ ∗ leading eigenvector ∗ = D −1/2 =
∗
(1) (|V |) T ,..., √ ∗ √ ∗ deg(1) deg(|V |)
that determines the probability rank order of the sites in the steady-state random walk. We can also view the recovery of the integration path as one of energy minimisation. The leading eigenvector of the matrix Wˆ satisfies the condition = arg max T Wˆ = arg max T D −1/2 W D −1/2 . ∗
We can make the relationship to the raw field of surface normals more explicit by introducing the matrix F =(N 1 |N 2 | . . . |N|V | ) with the surface normals as columns. When the constant is small, then making use of the
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
Maclaurin expansion of the exponential weighting function we can write W = eeT − (eeT − F T F ), where e = (1, 1, . . . , 1)T is the all-ones vector of length |V | matrix. In practice we use a 4-pixel neighbourhood to compute the weight matrix, and hence D 4I . As a result, when is small we can write D −1/2 W D −1/2 41 (eeT − [eeT − F T F ]). Hence, the path is the one that satisfies the condition = arg max TF TF ∗
= arg min
|V | |V |
i=1 j =1
N i .N j (i)(j ).
Hence, the integration path will minimise the change in surface normal direction. Our aim is to use the sequence of pixel sites given by the probability rank order to define a serial ordering for the sites on the pixel lattice. If we visit the sites of the pixel lattice in the order defined by the magnitudes of the coefficients of the leading eigenvector of the normalised affinity matrix, then the path is the steady state of the Markov chain. In this paper we aim to exploit this property to locate a connected path on the sites of the pixel lattice, and to use this path for surface integration and height recovery. Unfortunately, the path followed by the steady-state random walk is not edge-connected. Hence, we need a means of placing the pixel sites in an order in which neighbourhood connectivity constraints are preserved using the elements of the scaled . leading eigenvector ∗ To do this we commence from the pixel site associated with the largest component of ∗ , i.e. the largest site probability. We then sort the elements of the leading eigenvector such that they are both in the decreasing magnitude order of the coefficients of the eigenvector, and satisfy edge connectivity constraints on the graph. The procedure is a recursive one that proceeds as follows. At each iteration, we maintain a list of pixel sites visited. At iteration k let the list of pixel sites be denoted by Lk . Initially, L0 = j0 where j0 =arg maxj ∗ (j ), i.e. j0 is the component of ∗ with the largest magnitude. Next, we search through the set of first neighbours Nj0 = {k|(j0 , k) ∈ E} of j0 to find the pixel site associated with the largest remaining component of ∗ . The second element in the list is j1 = arg maxl∈Nj ∗ (l). 0 The pixel site index j1 is appended to the list of pixel sites visited and the result is L1 . In the kth (general) step of the algorithm we are at the pixel site indexed jk and the list of pixel sites visited by the path so far is Lk . We search through those first-neighbours of jk that have not already been traversed by the path. The set of pixel sites is Ck ={l|l ∈ Njk ∧ l ∈ / Lk }. The next site to be appended to the path
(
)
–
7
list is therefore jk+1 = arg maxl∈Ck ∗ (l). This process is repeated until no further moves can be made. This occurs when Ck = ∅ and we denote the index of the termination of the path by T. Theserial ordering of the pixel sites that results from this edge-based sorting is the integration path = LT . 4. Extracting patches In practice the surface under study may have a patch structure. We therefore take our spectral analysis one step further by showing how the patches may be identified by finding the blocks of the normalised affinity matrix Wˆ . To locate these blocks, we follow Sarkar and Boyer [19] and use the leading eigenvector of the weight matrix. We commence by constructing the thresholded normalised affinity matrix Wˆ T whose elements are defined as follows 0 if Wˆ T (i, j ) < T h, (6) Wˆ T (i, j ) = ˆ W (i, j ) otherwise, where Th is a threshold which allows the vanishingly small elements of the matrix Wˆ to be set to zero. Our aim is to identify groups of surface normals from a potentially noisy or normalised affinity matrix Wˆ T which correspond to surface patches. Stated formally, suppose that in an image with a normalised affinity matrix Wˆ there are m disjoint patches. Each such group should appear as a subblock of the matrix Wˆ T . However, as a consequence of noise or errors in the shape-from-shading method which delivers the field of surface normals, these distinct groups or patches may be merged together. In other words, their corresponding sub-blocks are no longer disjoint. Suppose that there are m distinct surface patches, each associated with an adjacency matrix B (i) where i is the patch index. If C represents a noise matrix, then the relationship between the observed matrix Wˆ T and the underlying blockstructured affinity matrix is Wˆ T = B + C,
(7)
where B = PBD PT , P is a permutation matrix and (1) B 0 BD = . .. 0
0 B (2) .. . ...
... 0 . .. . . . .. . 0 (m) 0 B
(8)
is a block diagonal matrix in which B (k) is the sub-block corresponding to the patch indexed k. To recover the matrix B, we turn to the eigenvector expansion of the matrix Wˆ T and write T Wˆ T = ∗ ∗ +
|V | i=2
T
i i i,
(9)
ARTICLE IN PRESS 8
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
is the where the leading eigenvalue is unity i.e. ∗ = 1, ∗ leading eigenvector and the eigenvectors are normalised to | = 1. To identify the patches, we be of unit length, i.e. | i use the following iterative procedure. We initialise the algo(1) (1) is the rithm by letting Wˆ = Wˆ T . Further suppose that ∗
T
(1) (1) (1)T leading eigenvector of Wˆ T . The matrix B (1) = ∗ ∗ represents the first block of Wˆ T , i.e. the most significant surface patch. The pixel sites with non-zero entries belong to the patch. These pixel sites may be identified and removed from further consideration. To do this we compute the resid(2) (1) ual transition matrix Wˆ U = Wˆ T − B (1) in which the elements of the first patch are nulled. The rows and column of (2) Wˆ U are no longer normalised. Hence, we compute a renor(2) malised matrix Wˆ by applying the normalisation method T
(2) (2) of the of Section 2 to Wˆ U . The leading eigenvector ∗ (2) normalised residual affinity matrix Wˆ T is used to compute (2)T . The process is repeated (2) the second block B (2) = ∗ ∗ iteratively to identify all of the principal blocks of Wˆ T . At (k)
is the leading eigenvector of the normalised iteration k, ∗ (k) residual transition probability matrix Wˆ T , and the kth block (k) (k)T is B (k) = ∗ ∗ . The patch indexed n is the set of pixel sites for which the components of the leading eigenvector (k) are non-zero. Hence, the index-set for the kth patch ∗
(k) (l) = 0} where (k) (l) is the lth component is Sk = {l| ∗ ∗ (k) . It is important to stress that the patches are nonof ∗ overlapping, i.e. the inner product of the block eigenvectors (k) . (l) = 0, where k = l. for different patches is zero ∗
∗
5. Geometry Our surface height recovery algorithm proceeds along the sequence of pixel sites defined by the order of the coefficients of the leading eigenvector associated with the separate patches. For the kth patch, the path is kS = (j1k , j2k , j3k , . . .) where the order is established using the method outlined at the end of Section 3. As we move from pixel-site to pixelsite defined by this path we increment the surface heightfunction. In this section, we describe the trigonometry of the height incrementation process. At step n of the algorithm for the path indexed k, we make k a transition from the pixel with path-index jn−1 to the pixel k with path-index jn . The distance between the pixel-centres associated with this transition is dn =
(x 2k − xj k )2 + (yj k − yj k )2 . jn n n−1 n−1
(10)
This distance together with the surface normals Nj k and n N j k at the two pixel-sites may be used to compute the n−1
(
)
–
change in surface height associated with the transition. The height increment is given by Nj k (x) dn N jnk (x) n−1 hn = + . (11) 2 N k (y) N k (y) jn
jn−1
If the height-function is initialised by setting zj k = 0, then 0
the centre-height for the pixel with path-index jnk is zj k = n+1 zj k + hn . n Once the surface normals that belong to the individual patches have been integrated together, then we merge them together to form a global surface. Suppose that Sk is the integrated surface for the kth patch. We compute the mean height for the pixels belonging to this boundary. We merge the patches together by ensuring that abutting patches have the same mean boundary height along their shared perimeter.
6. Experiments The experimental evaluation of the new surface reconstruction method is divided into two parts. We commence with a sensitivity study aimed at evaluating the method on synthetic data. In the second part of the evaluation, we focus on real-world data. The experiments were performed on a 2 GHz, dual-processor Xeon workstation. In all of our experiments the constant is set to unity. In addition, a threshold is used to separate the foreground object from the background. Only pixel-sites that have a normalised brightness greater than or equal to 0.3 are considered for the height recovery process. Finally, we have normalised the surface normals so that they have unit length, and hence satisfy the condition |N j k |=1. We have compared our results with those n delivered by three competing algorithms found elsewhere in the literature. These are the geometric integration algorithm described by Bors et al. [5] together with the methods of Horn and Brooks [2] and Frankot and Chellappa [14]. In each of these methods the surface integration method is separate from the surface normal process (in many shapefrom-shading methods the two components are integral parts of the algorithm which cannot be separated). Hence, the three alternative methods share the capability of taking fields of surface normals as input and returning a height map as output. The surface integration algorithm described by Bors et al. [5] is a local approach which was developed in the study of terrain reconstruction using radar shape-from-shading. This algorithm commences by locating a height reference on the pixel lattice. Once the reference is at hand, the method uses the trapezoid rule to increment the height from equal height reference contours. In contrast with this method [5], the remaining two algorithms are global approaches which aim to recover the surface height using the calculus of variations. The method of Horn and Brooks [2] hinges around solving a Poisson equation subject to boundary conditions imposed by
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
9
Fig. 2. Top row: artificially generated data; bottom row: field of surface normals.
the occluding object contour. However, the method is highly sensitive to initialisation and initial conditions are sometimes difficult to determine. Here, we initialise the method by setting the first pixel site in the path to a height equal to zero and, hence, our algorithm is devoid of the problems inherent to variable initialisation conditions. In the method developed by Frankot and Chellappa [14], integrability is enforced by posing the optimisation process in the Fourier domain. This method sometimes encounters difficulties when height recovery from polyhedral objects is attempted. In our implementation of the Horn and Brooks method, we have employed a finite element approach to recover the solution of the partial differential equation. For the Frankot and Chellappa method we have used the FFT. The algorithms of Frankot and Chellappa and Horn and Brooks are described, in greater depth, in the books by Horn [38] and Klette et al. [39]. 6.1. Synthetic data In this section, we provide some experiments on synthetic data. The aim here is to determine the accuracy of the surface reconstruction method. To this end, we have generated synthetic surfaces using simple analytical forms. From the analytical forms we have computed the field of surface normal directions. We have then applied our graph-spectral surface integration method to the field of surface normals to recover an estimate of the surface height. In Fig. 2, we show the surface height and the fields of surface normals for the shapes used in our experiments. The results of performing surface integration from the surface normals are shown in Fig. 3. From top-to-bottom, the panels in the first and second rows of the figure, respectively show the surface reconstructed from the field of surface normals and the absolute error between the ground-truth and recon-
structed surface height. The remaining rows of the figure show the corresponding results obtained with the alternative algorithms. Specifically, the third and fourth rows show the results obtained using the geometric integration method of Bors, Wilson and Hancock, the fifth and sixth rows those obtained using the Horn and Brooks method, and the seventh and eight rows those obtained using the Frankot and Chellappa method. In the columns of the figure, from leftto-right, the surfaces studied are a dome, a sharp ridge, a torus and a volcano. For all four shapes, our method outperforms the three alternatives. For the dome the height errors are greater at the edges of the surface where the slope is largest. In the case of the ridge, there are errors at the crest. For the volcano, there are some problems with the recovery of the correct depth of the “caldera”, i.e. the depression in the centre. For the reconstructed surfaces, the mean-squared errors are 5.6% for the dome, 10.8% for the ridge, 7.8% for the torus and 4.7% for the volcano. Hence, the method seems to have greater difficulty for surfaces containing sharp creases. We have repeated these experiments under conditions of controlled noise. To do this, we have added random errors to the fields of surface normals. The surface normal errors have been sampled from a Gaussian distribution with zero mean and known variance. In Fig. 4, we show the result of reconstructing the surfaces shown in Fig. 2 when random errors with a variance of 1 have been added to the field of surface normals. In the top row of the figure we show the noise-corrupted field of surface normals. In the second and third rows, we show the reconstructed height-function obtained from the noisy surface normals and the absolute difference between the ground-truth height and that recovered by the graph-spectral method. In the fourth and fifth rows, we show the results obtained using the geometric integration method of Bors, Hancock and Wilson, the sixth
ARTICLE IN PRESS 10
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
Fig. 3. From top-to-bottom: recovered surfaces and error plots for the noise-free fields of normals in Fig. 2 making use of our method, the geometric integration method of Bors et al. [5], the method of Horn and Brooks [2] and Frankot and Chellappa’s method [14].
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
11
Fig. 4. From top-to-bottom: noise-corrupted fields of surface normals (worst case with variance = 1), recovered surfaces and error plots delivered by our method, the local geometric integration algorithm of Bors et al. [5] and the methods of Horn and Brooks [2] and Frankot and Chellappa’s [14].
ARTICLE IN PRESS 12
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
)
–
ERROR AS A FUNCTION OF THE VARIANCE FOR THE RIDGE
ERROR AS A FUNCTION OF THE VARIANCE FOR THE DOME 30
60 GRAPH-SPECTRAL METHOD METHOD OF HORN AND BROOKS METHOD OF FRANKOT AND CHELLAPPA METHOD OF BORS, HANCOCK AND WILSON
25
(
GRAPH-SPECTRAL METHOD METHOD OF HORN AND BROOKS METHOD OF FRANKOT AND CHELLAPPA METHOD OF BORS, HANCOCK AND WILSON
50
20
40
15
30
10
20
5
10
0
0 0
0.2
0.4
0.6
0.8
1
45
0.2
0.4
0.6
0.8
1
45
GRAPH-SPECTRAL METHOD METHOD OF HORN AND BROOKS METHOD OF FRANKOT AND CHELLAPPA METHOD OF BORS, HANCOCK AND WILSON
40
0
ERROR AS A FUNCTION OF THE VARIANCE FOR THE VOLCANO
ERROR AS A FUNCTION OF THE VARIANCE FOR THE TORUS
GRAPH-SPECTRAL METHOD METHOD OF HORN AND BROOKS METHOD OF FRANKOT AND CHELLAPPA METHOD OF BORS, HANCOCK AND WILSON
40
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
0 0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
Fig. 5. Percentage surface reconstruction error versus noise variance for the four basic shapes.
Fig. 6. Results on real-world imagery.
0.8
1
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
13
Fig. 7. Results on real-world imagery.
Fig. 8. Results on real-world imagery.
and seventh rows show those obtained using the Horn and Brooks method, and the eight and ninth rows those obtained using the Frankot and Chellappa method. For all four methods, the gross structure of the recovered surfaces is maintained. However, in the case of the graph-spectral method, although the recovered height is clearly noisy, the height difference plots are relatively unstructured. This is an important observation since it means that our graph-spectral
method simply transfers errors in surface normal direction into errors in height, without producing structural noise artefacts. To investigate the effect of noise further, in Fig. 5 we plot the mean-squared error for the reconstructed surface height as a function of the standard deviation of the added Gaussian noise. From the plots for the different surfaces, it is clear that the mean-squared error grows slowly with
ARTICLE IN PRESS 14
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
Fig. 9. Results on real-world imagery.
Fig. 10. Results on real-world imagery.
increasing noise standard deviation. In general, the error rates for our algorithm are lower than those for the three alternatives. Moreover, for noise-corrupted fields of normals whose noise-variance is less than 0.7, the graphspectral method consistently outperforms the alternative algorithms.
Of course, the poor performance of the Horn and Brooks method [2] may be attributable to the initialisation scheme used. For instance, Worthington and Hancock [40] found that the method could be improved if the surface normals are positioned on the irradiance cone throughout the object under study, rather then being
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
15
Fig. 11. Results on real-world imagery.
Table 1 Summary statistics for the graph-spectral method Image
No. of pixel sites (foreground only)
No. of patches
% of reconstruction errors
Duck Volcano Urn detail Vase Mozart Block
6842 13924 8648 7664 9867 7456
57 76 145 132 174 47
7.3 0.7 7.8 5.4 9.6 8.1
initialised just at the occluding boundary. The Fronkot and Chellappa [14] method has difficulties in recovering depth discontinuities. However, some steps towards overcoming this problem are reported in the work of Wei and Klette [16]. However, detailed experimental comparison with this work is beyond the scope of this paper.
6.2. Real-world data The second part of our experimental evaluation of the new surface height recovery focuses on real-world imagery. Here, we have applied our surface recovery method to needle-maps extracted from real-world data using the shape-from-shading algorithm of Worthington and Hancock [40]. It should be stressed, however, that the method can be used in any situation where surface-data is presented in the form of a field of surface normals sampled on a plane. Hence, it can
also be used in conjunction with shape-from-texture and motion analysis. In Fig. 6, we show our first example. The image used in this study is of a toy duck and has been taken from the COIL database. The four panels to the left of the figure convey the following information. The two panels in the top row, show the raw image (left) and the extracted field of surface normals (right). Two views of the reconstructed surface are shown in the bottom row. The main features to note here are the well-formed beak, the spherical shape of the head, the well-defined cusp between the head and body, and, the saddle structure near the tail. The wing detail is also well reproduced. In far right-hand panel of the figure we show the integration path, i.e. the order of the components of the leading eigenvector, for each detected patch. The path appears to follow the main height contours on the surface patches. The main patches correspond to the head, the beak, the body, the wing and the tail.
ARTICLE IN PRESS 16
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
Fig. 12. Results of integrating the surface using the geometric approach of Bors, Hancock and Wilson.
Fig. 7 repeats this sequence of images for a synthetic image of a volcano. Here the method deals well with the transitions from convex to concave regions. Hence the “caldera” is well located. Hence, the method appears to cope well with the convex-concave reversal of the surface. The third example is shown in Fig. 8. This is the detail from a porcelain urn. Here both the convexities and concavities on the surface are well reconstructed. The principal surface patches correspond to the handle of the urn, the lower “bulge” and the cylindrical body of the urn. In addition, some of the ribbed detail on the surface of the urn is recovered. The fourth example is provided in Fig. 9 which shows the results obtained with an image of a wooden block taken from the COIL database. Here the cylindrical structure of the handle of the block is well recovered. The two planar faces of the block are well detected. However, the angle between the faces is somewhat greater than 90◦ . There are also problems with the planar faces where there is noise in the surface normal directions. However, the method copes well the depth discontinuity between the rectangular block and the cylindrical handle.
Finally, Figs. 10 and 11 show two images from the University of Central Florida database which is used in the recent comparative study of Zhang et al. [17]. The images are of a vase and a bust of Beethoven. The overall shape of the vase is well reconstructed and the fine detail of the face of Beethoven is well reproduced. In Table 1, for the examples described above, we list the image size, the number of patches extracted and the meansquared errors for the reconstructed surfaces. The meansquared errors are computed making use of a CAD model of the object under study. The errors may be due to artificial height discontinuities or locations where the surface becomes inverted (i.e. concave regions become convex, or vice versa). From the table, it is clear that only about 7–10% of the surface area of the objects studied is affected by such errors. In the case of the toy duck, the errors are dominated by the inversion of the wing. We have compared our algorithm with the three alternatives described earlier and used in our sensitivity study. Figs. 12–14, respectively show views of the reconstructed surfaces obtained with the geometric integration algorithm
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
17
Fig. 13. Results of integrating the surface using of the method of Horn and Brooks.
of Bors, Hancock and Wilson and the methods of Horn and Brooks method and Frankot and Chellappa. The images used are those already shown in Figs. 6–11, and there are two surface views for each object. Qualitatively, the most interesting point of comparison between the methods is their relative ability to recover discontinuities, and, the degree of distortion and over-smoothing introduced. In Fig. 13, the surface of the duck recovered by the Horn and Brooks algorithm is less detailed than that recovered using our graph-spectral method. For instance, the boundary of the neck and body is less well defined. Also most of the detail of the Beethoven bust is lost. In the case of the volcano, the result delivered by each of the alternative methods is distorted. In particular, singular points have been added, and this results in a “pimple” inside the caldera. Overall, the results obtained by the geometric surface integration algorithm of Bors, Hancock and Wilson are better than those obtained by the methods of the Horn and Brooks and Frankot and Chellappa, but do not contain the fine detail delivered by our graph-spectral method. For instance, the hair and fabric details of Beethoven are not
well reproduced and the rib structures on the vase are not well recovered. Moreover, the shape of the handle is not well recovered and has been distorted and over-smoothed. For the wooden block, there is an artificial depth discontinuity between the rectangular block and the cylindrical handle. A more quantitative analysis is presented in Table 2. We list the image dimensions and the mean-squared errors for the different surfaces. In contrast with our algorithm, the alternative ones yield error rates which are as high as 34% and never less than 7%.
7. Conclusions In this paper, we have described a new surface height recovery algorithm. The method commences from a surface characterisation which is based on an affinity matrix whose elements are related to path curvature and are computed from the difference in surface normal direction. We use the leading eigenvector of the normalised affinity matrix
ARTICLE IN PRESS 18
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
(
)
–
Fig. 14. Results of integrating the surface using the method of Frankot and Chellappa. Table 2 Summary statistics for the Bors, Hancock and Wilson geometric integration algorithm and the methods of Horn and Brooks and Frankot and Chellappa Image
Duck Volcano Urn detail Vase Mozart Block
Dimensions (pixels)
128 × 128 128 × 128 128 × 128 256 × 256 256 × 256 128 × 128
Percentage of reconstruction errors Geometric (%)
Horn and Brooks (%)
Frankot and Chellappa (%)
8.8 16.9 16.8 7.1 11.1 9.8
17.7 28.6 37.2 27.8 18.2 19.3
21.4 19.2 34.3 31.3 16.8 16.2
to define an integration path. By traversing this path and applying some straightforward trigonometry we reconstruct the surface height function. Experiments on real-world data reveal that the method is able to reconstruct quite complex surfaces. Noise sensitivity experiments on simulated data, show that the method is capable of accurate surface reconstruction.
Our future plans involve developing the work described here in a number of different directions. Firstly, there is clearly scope for using more elaborate differential geometry to construct the affinity matrix. One possibility here is to attempt to ensure consistency of the integration path direction, and that of the local minimum curvature direction. To do this we would need to compute an estimate of the
ARTICLE IN PRESS A. Robles-Kelly, E.R. Hancock / Pattern Recognition
surface Hessian from the field of surface normals, and the components of the affinity matrix can be determined by the inner product of the minimum curvature direction vectors at different locations on the pixel lattice. Another possible direction is to develop a more sophisticated model for the integration path. In this paper, we have sought the path that is the steady-state random walk of a Markov chain on a graph. This is a type of diffusion process. A more principled approach may be to pose the recovery of the integration path as the solution of a stochastic differential equation. In other words, the integration path may be posed as the solution of a Fokker–Plank equation. Finally, it may be interesting to investigate whether the idea of recovering a path using graph-spectral methods can be applied to 2D curve enhancement problems. The literature on curve enhancement is substantial and includes a number of alternative methodologies including relaxation labelling [35,41], variational methods based on the Mumford-Shah functional [42] and variants upon it, and, stochastic completion fields [36]. These methods all share the feature of attempting to represent the affinity of points to curves using probabilistic compatibility relationships. The compatibility model is invariably based on curvature and continuity information. However, most of the reported work uses local energy minimisation methods to locate the optimal grouping curve. The idea of using steadystate random walks clearly presents an interesting alternative which leaves considerable scope for further study. In particular, the behaviour of curves in noisy imagery is frequently posed as a diffusion process. Hence, the behaviour of the transition probabilities is governed by a stochastic differential equation, and the steady-state random walk is a natural way to characterise such systems. Finally, we note that in related work we have used eigenvector methods to develop an integrated approach to shapefrom-shading [28]. Here we have commenced from a specification of the problem as one of recovering a surface integration path that both satisfies connectivity constraints on the pixel lattice, and which minimises a brightness error. The work reported in this paper represents the starting point which lead us to this more sophisticated method, but nonetheless provides an interesting stand-alone surface integration method.
Acknowledgements The work presented here was done while Antonio RoblesKelly was with The University of York, York, United Kingdom.
References [1] A.P. Rockwood, J. Winget, Three-dimensional object reconstruction from two dimensional images, Computer-Aided Des. 29 (4) (1997) 279–285.
(
)
–
19
[2] B.K.P. Horn, M.J. Brooks, The variational approach to shape from shading, CVGIP 33 (2) (1986) 174–208. [3] B.K.P. Horn, Height and gradient from shading, Int. J. Comput. Vision 5 (1) (1990) 37–75. [4] Z. Wu, L. Li, A line-integration based method for depth recovery from surface normals, CVGIP 43 (1) (1988) 53–66. [5] A.G. Bors, E.R. Hancock, R.C. Wilson, Terrain analysis using radar shape-from-shading, IEEE Trans. Pattern Anal. Mach. Intell. 25 (8) (2003) 974–992. [6] P. Dupuis, J. Oliensis, Direct method for reconstructing shape from shading, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1992, pp. 453–458. [7] M. Bichsel, A.P. Pentland, A simple algorithm for shape from shading, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1992, pp. 459–465. [8] Y.G. Leclerc, A.F. Bobick, The direct computation of height from shading, in: Proceedings of Computer Vision and Pattern Recognition, 1991, pp. 552–558. [9] P.S. Tsai, M. Shah, Shape from shading using linear approximation, Image Vision Comput. 12 (8) (1994) 487–498. [10] B. Karaçali, W. Snyder, Reconstructing discontinuous surfaces from a given gradient field using partial integrability, Comput. Vision Image Understanding 92 (1) (2003) 78–111. [11] B. Karaçali, W. Snyder, Noise reduction in surface reconstruction from a given gradient field, Int. J. Comput. Vision 60 (1) (2004) 25–44. [12] R. Kimmel, K. Siddiqqi, B.B. Kimia, A.M. Bruckstein, Shape from shading: level set propagation and viscosity solutions, Int. J. Comput. Vision (16) (1995) 107–133. [13] R. Kimmel, A.M. Bruckstein, Tracking level sets by level sets: a method for solving the shape from shading problem, Comput. Vision Image Understanding 62 (2) (1995) 47–48. [14] R.T. Frankot, R. Chellappa, A method of enforcing integrability in shape from shading algorithms, IEEE Trans. Pattern Anal. Mach. Intell. 4 (10) (1988) 439–451. [15] L. Noakes, R. Kozera, R. Klette, The lawn-moving algorithm for noisy gradient vector fields, in: Proceedings of SPIE Conference, Vision Geometry VIII, 1999, pp. 305–316. [16] T. Wei, R. Klette, Depth recovery from noisy gradient vector fields using regularization, in: CAIP, 2003, pp. 116–123. [17] R. Zhang, P.S. Tsai, J.E. Cryer, M. Shah, Shape from shading: a survey, IEEE Trans. Pattern Anal. Mach. Intell. 21 (8) (1999) 690–706. [18] J. Shi, J. Malik, Normalized cuts and image segmentation, IEEE Trans. Pattern Anal. Mach. Intell. 22 (8) (2000) 888–905. [19] S. Sarkar, K.L. Boyer, Quantitative measures of change based on feature organization: eigenvalues and eigenvectors, Comput. Vision and Image Understanding 71 (1) (1998) 110–136. [20] P. Perona, W.T. Freeman, Factorization approach to grouping, in: Proceedings ECCV, 1998, pp. 655–670. [21] N.L. Biggs, Algebraic Graph Theory, Cambridge University Press, Cambridge, 1993. [22] L. Lovász, Random walks on graphs: a survey, Bolyai Soc. Math. Stud. 2 (2) (1993) 1–46. [23] F.R.K. Chung, Spectral Graph Theory, American Mathematical Society, Providence, RI, 1997. [24] D. Cvetkovi´c, M. Doob, H. Sachs, Spectra of Graphs: Theory and Application, Academic Press, New York, 1980.
ARTICLE IN PRESS 20
A. Robles-Kelly, E.R. Hancock / Pattern Recognition
[25] A. Sinclair, M. Jerrum, Approximate counting, uniform generation and rapidly mixing Markov chains, Inform. Comput. 82 (1989) 93–133. [26] Y. Azar, A. Fiat, A.R. Karlin, F. McSherry, J. Saia, Spectral analysis of data, in: ACM Symposium on Theory of Computing, 2000, pp. 619–626. [27] J. Kleinberg, Authoritative sources in a hyperlinked environment, in: Proceedings ACM-SIAM Symposium on Discrete Algorithms, 1998, pp. 668–677. [28] A. Robles-Kelly, E.R. Hancock, A graph-spectral approach to shape-from-shading, IEEE Trans. Image Process. 13 (7) (2004) 912–926. [29] A. Robles-Kelly, E.R. Hancock, A graph-spectral method for surface height recovery from needle maps, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. I 2001, pp. 141–148. [30] A. Robles-Kelly, E.R. Hancock, A graph-spectral approach to surface segmentation, in: International Conference on Pattern Recognition, vol. III, 2002, pp. 509–512. [31] A. Robles-Kelly, E.R. Hancock, Edit distance from graph spectra, in: International Conference on Computer Vision, 2003, pp. 234–241. [32] A. Robles-Kelly, E.R. Hancock, String edit distance, random walks and graph matching, in: SSPR 2002, Lecture Notes in Computer Science, vol. 2396, Springer, Berlin, 2002, pp. 104–112.
(
)
–
[33] M.P. Do Carmo, Differential Geometry of Curves and Surfaces, Prentice-Hall, Englewood Cliffs, NJ, 1976. [34] J.A.F. Leite, E.R. Hancock, Iterative curve organisation with the em algorithm, Pattern Recognition Lett. 18 (1997) 143–155. [35] S.W. Zucker, C. David, A. Dobbins, L. Iverson, The organization of curve detection: coarse tangent fields and fine spline coverings, in: Proceedings of the IEEE International Conference on Computer Vision, 1988, pp. 568–577. [36] L.R. Williams, D.W. Jacobs, Stochastic completion fields: a neural model of illusory contour shape and salience, Neural Comput. 9 (4) (1997) 837–858. [37] R.S. Varga, Matrix Iterative Analysis, second ed., Springer, Berlin, 2000. [38] B.K.P. Horn, Robot Vision, Mc-Graw Hill, New York, 1986. [39] R. Klette, K. Schluns, A. Koschan, Computer Vision: ThreeDimensional Data from Images, Springer, Singapore, 2001. [40] P.L. Worthington, E.R. Hancock, New constraints on datacloseness and needle map consistency for shape-from-shading, IEEE Trans. Pattern Anal. Mach. Intell. 21 (12) (1999) 1250–1267. [41] E.R. Hancock, J.V. Kittler, Discrete relaxation, Pattern Recognition 23 (1990) 711–733. [42] D. Mumford, J. Shah, Optimal approximations by piecewise smooth functions and associated variational problems, Comm. Pure Appl. Math. 42 (5) (1989) 577–685.
About the Author—ANTONIO ROBLES-KELLY received the BEng degree in electronics and telecommunications from the Instituto Tecnologico y de Estudios Superiores de Monterrey with honors in 1998 and the Ph.D. degree in computer science from the University of York in 2003. As a graduate student, in 2001, he visited the University of South Florida as part of the William Gibbs/Plessey Award to the best research proposal to visit an overseas research lab. He received his Ph.D. in Computer Science from the University of York in 2003 and, from 2002 to 2004, he worked as a Research Associate at York under the MathFit-EPSRC framework. Currently, he is a research Scientist at National ICT Australia (NICTA). His research interests are in the areas of computer vision, pattern recognition, and computer graphics. Along these lines, he has done work on segmentation and grouping, graph-matching, shape-from-X, and reflectance models. He is also interested in the differential structure of surfaces. About the Author—EDWIN HANCOCK studied physics as an undergraduate at the University of Durham and graduated with honours in 1977. He remained at Durham to complete a Ph.D. in the area of high-energy physics in 1981. Following this he worked for 10 years as a researcher in the fields of high-energy nuclear physics and pattern recognition at the Rutherford-Appleton Laboratory (now the Central Research Laboratory of the Research Councils). During this period he also held adjunct teaching posts at the University of Surrey and the Open University. In 1991 he moved to the University of York as a lecturer in the Department of Computer Science. He was promoted to Senior Lecturer in 1997 and to Reader in 1998. In 1998 he was appointed to a Chair in Computer Vision. Professor Hancock now leads a group of some 15 faculty, research staff and Ph.D. students working in the areas of computer vision and pattern recognition. His main research interests are in the use of optimisation and probabilistic methods for high and intermediate level vision. He is also interested in the methodology of structural and statistical pattern recognition. He is currently working on graph-matching, shape-from-X, image data-bases and statistical learning theory. His work has found applications in areas such as radar terrain analysis, seismic section analysis, remote sensing and medical imaging. Professor Hancock has published some 80 journal papers and 300 refereed conference publications. He was awarded the Pattern Recognition Society medal in 1991 and an outstanding paper award in 1997 by the journal Pattern Recognition. In 1998 he became a fellow of the International Association for Pattern Recognition. Professor Hancock has been a member of the Editorial Boards of the journals IEEE Transactions on Pattern Analysis and Machine Intelligence, and, Pattern Recognition. He has also been a guest editor for special editions of the journals Image and Vision Computing and Pattern Recognition. He has been on the programme committees for numerous national and international meetings. In 1997 with Marcello Pelillo, he established a new series of international meetings on energy minimisation methods in computer vision and pattern recognition.
