Qualitative Shape from Active Shading Michael S. Langer Steven W. Zucker TR-CIM-91-6 November 1991 1
1 2
Centre for Intelligent Machines McGill University, Montreal, Quebec, Canada
Mailing address: McConnell Eng. (rm. 410), McGill University, 3480 University Street, Montreal, Canada H3A 2A7. email:
[email protected] [email protected]. 2 Fellow, Canadian Institute for Advanced Research 1
1
Abstract We show how to actively compute qualitative shape properties directly from image intensities - that is, without having to rst \reconstruct the surface". Our approach diverges from classical active shape from shading [1] in two important ways. First, we do not attempt to compute a dense depth map of the surface. We rather detect the presence of certain qualitative geometric features. Second, we use two di erent types of lighting conditions rather than two di erent instances of a single type (the point source). Our two types of lighting conditions are a di use source and a point source positioned at the camera. The major result of this paper is that concave orientation discontinuities and smooth valleys share the same shading signatures. Under di use conditions, both produce local minima in the image intensity. Under the point source condition, both produce local maxima in the image intensity. One can detect these features by alternating between the lighting conditions.
2
1 Introduction Shape from Shading is the problem of computing geometric properties of a surface from measurements of the light which has re ected o the surface. Obviously, the distribution of light re ected o a surface depends on how the surface is illuminated. For example, if a surface were illuminated by a di use light source (such as the sky on a cloudy day) then it would have a di erent luminance pro le than if it were illuminated by a point source. We show in this paper that, by alternating the lighting conditions in the scene from a di use source to a point source, certain qualitative shape properties of the surfaces become salient. There are three separate components to this paper. The rst is that we compute qualitative shape properties from the image intensities themselves. We do not attempt to \reconstruct" the surface [3, 2] before computing its qualitative shape. The second component is an examination of the relationship between qualitative shape and shading under two different lighting conditions : a di use source, and a point source at the camera. We show how qualitative shape is revealed under these two conditions. In particular, we introduce a model of di use shading which is quite di erent from classical (point source) shading models. The third component is a proposal for actively manipulating the light sources so that features of interest will be salient.
2 Local Qualitative Shape We assume that our scene is composed of a set of piecewise smooth surfaces. In this domain, there exist two commonly held notions of qualitative shape. One notion applies to undulating 3
Figure 1: There are two notions of qualitative shape. One notion is that undulating surfaces can be described as a set of hills and valleys. A second notion is that surfaces can be described as a set of adjacent (or interpenetrating ) parts. In this scene, the folds of the blanket are best described using the former while the apples are best described using the latter. drapery surfaces, such as a blanket or a loose shirt. Here, qualitative shape refers to the spatial arrangement of surface hills and valleys, and is typically represented with a coarse coding of the surface curvatures (eg. convex, concave, hyperbolic) [9, 8, 13]. A second notion applies to surfaces which seem to be best described as a composition of a set of adjacent or interpenetrating convex objects [10]. Examples include a pile of oranges on a table, or the phalanges of a hand. Here, qualitative shape refers to a set of parts and how these parts are connected. Locally, the distinction between these two notions is unclear, since blurring a set of \parts" will produce an undulating surface. For example, how much does a pile of rocks 4
have to erode before it becomes a set of hills and valleys ? We take a uni ed position with respect to qualitative shape and maintain that a global theory is necessary to re ne the distinction [12]. With this background, we show in this paper that it is often possible to detect qualitative features of the surface shape directly from the image intensities. The idea is that the two notions of qualitative local shape described Figure 1 have something important in common. Hills and parts both refer to surface regions which protrude from the surface; Valleys and part boundaries both refer to surface regions which are often hidden by the surface.
The protruding surface regions usually receive direct illumination from the light source, whereas the hidden regions usually lie in shadow. Another way of saying this is that the protruding surface regions see a larger percentage of the sky than the hidden regions. We introduce a new geometric property of surfaces to capture this last idea. The aperture of a surface element is de ned as the percentage of the sky which is visible from that element. Thus, protruding surface regions have higher aperture than hidden regions. In this paper, we take advantage of the strong relationship between surface aperture and surface luminance to compute qualitative shape properties directly from image intensities. We prove in section 3 that concave orientation discontinuities have a di erent shading signature under di use source conditions than under point source conditions. Under di use conditions, surface luminance is a local minimum at the discontinuity. Under point source conditions, surface luminance is a local maximum at the discontinuity. This latter shading signature has been known for a long time [4] and has recently been analysed quantitatively [5, 6, 7]. We then claim that the same shading signatures typically occur for the smooth 5
Figure 2: This image was formed by overlapping a set of rectangles, each having a vertical intensity gradient. Most people perceive a set of at surfaces protruding from a ground plane. A skyline or a graveyard are two typical interpretations. These percepts cannot be explained by the Image Irradiance Equation which says that smooth variations in image intensity depict smooth variations in surface normal. The Image Irradiance Equation would suggest that the \tombstones" are cylinders! valleys (though we defer the proof to another paper), and show an example of a drapery surface under the two lighting conditions.
3 Shape From Shading on a Cloudy Day Classical Shape From Shading research [4] is based on the assumption that the intensity variations in an image depict unit surface normal variations in the scene. This depiction relation is speci ed by the Image Irradiance Equation. However, this model is not always applicable, as can be seen in Figure 2. Most people interpret this image as a set of of at surfaces protruding from the ground plane. This percept is clearly in con ict with the Image 6
Irradiance Equation. The main problem is that the Image Irradiance Equation was designed for scenes in which it makes sense to talk about "the illuminant direction". Most people perceive the scene in Figure 2 to be illuminated by a di use light source rather than a point light source. For this scene, the question, "Where is the illuminant?" [15] is meaningless. Although attempts have been made [16], one cannot model surface luminance under di use lighting conditions using the Image Irradiance Equation because the surface luminance is not determined by the direction of the local surface normal.
3.1 A Model of Surface Luminance Under Di use Light In this section we summarize some of the key elements of a new theory of Shape From Shading under di use lighting conditions. A more detailed analysis can be found in [17]. The theory is motivated by Gibson's observation [18] that the ambient light from the sky is of greater luminance than the ambient light from surfaces in the scene. We call this observation the Dominating Sky Principle. This model of surface luminance is a reasonable approximation to the actual surface luminance as long as the surfaces in the scene are Lambertian and the luminance of the di use light source is uniform across the sky. Let I (x) be the luminance of the surface at x, that is, the luminance scattered from a surface element at x. Let ID be the average luminance from the di use light source. Both
I (x) and ID are expressed in Watts per steradian per projected unit surface area [19]. Let (x) be the surface albedo, that is, the fraction of incident light energy which is scattered (rather than absorbed) by the surface (0 < (x) < 1). 7
N L ϕ
Figure 3: The hemisphere of incident directions above a point on a surface can be partitioned into two sets. D(x) denotes the set of directions pointing to the sky. H(x)nD(x) denotes the set of directions pointing to other surface elements in the scene. Let S denote a unit sphere and let H(x) S be the hemisphere of directions from 2
2
which the surface element at x can receive incident illumination. Formally,
H(x) = fL 2 S : N(x) L > 0g 2
Notice that for any L 2 H(x); L points either to the di use source (some region of the sky in the example of the cloudy day) or to some other surface element in the scene.
D(x) H(x) denotes the set of directions at x which point to the di use source, so that H(x)nD(x) is the set of directions at x which point to other surface elements in the scene. The Dominating Sky Principle says that the luminance from the directions D(x) is greater than that coming from the directions H(x)nD(x). This is proved in [17] and a lower bound is derived on the di erence between the luminance of the di use source and the surface luminance. 8
ID
IM _π _ 2
π _ 2
ϕ
Figure 4: Incident luminance is a function de ned on H. Directions on an arbitrary great circle (the dotted curve in Figure 3) can be indexed by . The above graph shows incident luminance de ned on this circle of directions. Notice that the incident luminance is much higher on D than on H(x)nD(x). The above de nitions suggest the following model of surface luminance. This model is similar in form to classical models which use only a single point source. The di erence is that we have to integrate over those regions of the sky which are directly illuminating the surface element. We use the symbol M to denote mutual illumination of the surfaces (as opposed to calling this term \ambient light"). Let dL be an in nitesimal region of H(x) which is centered at direction L.
I (x) = 1
Z D(x)
(ID ? IM) N(x) L dL + IM
(1)
The rst term in this model is due to the luminance, ID ? IM, of the di use source that is in excess of the luminance of the surfaces in the scene. The second term is due to the mean
luminance from other surface elements in the scene. 1I
1
M will vary with x but this variation is usually quite small over a local region of the surface [17]. 9
The model shows explicitly that the amount of the di use source seen from a surface element is an important determinant of the luminance of that element. This idea motivates the following de nition.
De nition: The aperture A(x) of point x on a surface is the ratio of the solid angle of D(x) to the solid angle of H(x). That is, A(x) = 21
Z D(x)
dL
:
(2)
Notice that, for all x; 0 A(x) 1. Intuitively, A(x) is the fraction of H(x) in which the di use source (eg. the sky on a cloudy day) is visible from x. It is important to observe that Aperture is a geometric property which cannot be de ned locally. We will see that it is a very natural property to use in discussing Shape from Shading on a Cloudy Day.
3.2 Di use Shading : An Simple Example Consider a scene composed of a sphere resting on a plane. We argue that the luminance is a minimum at the point of contact. Call this point of contact the south pole of the sphere, and call the opposite point of the sphere the north pole. Consider rst the shading on the plane. Because of the radial symmetry of the scene, shading occurs only along straight lines through the point of contact with the luminance decreasing continuously toward the point of contact. The reason for this decrease is simply that the sphere occludes a greater region of the sky for points on the plane which are closer to the sphere. Notice that there are no variations in surface normal along the plane, and yet 10
luminance variations clearly occur. As in Figure 2, such an e ect clearly cannot be accounted for with the Image Irradiance Equation. Now consider the shading on the sphere. Again, because of radial symmetry, shading occurs only along lines (of longitude) connecting the poles. The luminance is a maximum at the north pole, and a minimum at the south pole. The luminance varies continuously between these limits as the amount of the sky visible overhead varies. At the south pole, none of sky the is visible whereas at the north pole, all of the sky is visible. Thus, surface luminance is a minimum at the point of contact.
3.3 Di use Shading at Concave Orientation Discontinuities Suppose two convex surface regions are joined along a concave orientation discontinuity (see Figure 5). We claim that both surface luminance and aperture are a local minimum across this concave orientation discontinuity. More formally, let ~ (t) be a unit speed curve passing perpendicularly through the concave orientation discontinuity at t = 0, so that the tangent vector of ~ (t) is discontinuous at t = 0.
Claim : According to Equation 1, I ( ~ (t)) is a minimum at t = 0. The proof of this claim goes as follows. Imagine walking along the curve ~ (t), and consider what you see in the hemisphere of directions overhead. To be precise about the term "overhead", de ne a moving coordinate frame at ~ (t) by f ~ 0(t); N(t); N ~ 0(t)g. Let H denote the upper half of S as expressed in in this moving frame. Then H is 2
the set of directions "overhead". Notice that H does not depend on t since the coordinate axes are intrinsic. Let @ H be the boundary of H, that is, the "horizon". Let N 2 H be 11
a.)
α (t) concave orientation discontinuity
b.)
Figure 5: ~ (t) is a curve which is perpendicular to COD at t = 0. A. We wish to show that I ( ~ (t)) is a local minimum at t = 0. B. The idea behind the proof is that the amount of the di use source visible from ~ (t) is a local minimum at t = 0. The Dominating Sky Principle guarentees that if a region of the di use source is occluded by a surface in the scene, then the incident illumination from that direction is decreased. the unit surface normal (the direction directly overhead ). Let D(t) H be the directions overhead in which the di use source is visible from ~ (t). The variations in surface aperture and luminance along ~ (t) are entirely a result of the deformation in D (t). To see this, rewrite equations (1) and (2) as Z 1 I ( ~ (t)) = (ID ? IM)N
D (t)
LdL + IM
Z 1 A( ~ (t)) = 2 D d L (t)
(3) (4)
These equations are the integral of some xed scalar function over a domain which varies with t, so that luminance variations are determined entirely by variations in the domain
D (t). 12
1.5
height
1
0.5
0
-0.5 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
pixel
Figure 6: Height pro le of a wedge surface (in image coordinates) We can now complete the proof that luminance is a minimum at the concave orientation discontinuity. As we move along ~ (t) toward t = 0, we sees nothing but clear sky to the rear, while ahead we see a surface rising up and blocking regions of the di use light source. It follows immediately from Equation 3 that L( ~ (t)) decreases as t ! 0 2. Exactly the same argument shows that surface aperture reaches a minimum across the concave orientation discontinuity. An example of this luminance e ect is shown in Figure 6. The surface is planar with a wedge cut from it. Figure 7 shows the luminance pro le over a set of ve di erent 2
albedos and under a di use light source of unit luminance. The key point to observe is that luminance is a minimum at the concave orientation discontinuity. To see a real example of how surface luminance reaches a minimum at concave orientation discontinuities, perform the following experiment. Curl the palm of your hand slightly to make a cup. The skin of your hand will fold along wrinkles, and dark lines will emerge along 2
This luminance pro le was calculated exactly using a 1-d version of the Radiosity Equation [5]
13
Wedge : Diffuse Source 1 0.9 0.8 0.7
intensity
0.6 0.5 0.4 0.3 0.2 0.1 0 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
pixel
Figure 7: Image Intensity (or Surface Luminance) for ve di erent albedos (0.1, 0.3, 0.5, 0.7, 0.9) where the di use source has unit luminance. The Dominating Sky Principle guarentees that surface luminance is less than 1. This fact is easily observed in the gure. Notice also that luminance is a minimum at the concave orientation discontinuity. these wrinkles. Also observe that dark lines emerge where the ngers are pressed together. Dark lines are the luminance minima which separate the skin of the cupped hand into its convex regions. These observations are related to ideas mentioned in [20] concerning the stability of surface luminance structure over many di erent lighting conditions. In particular, observe that surface luminance is stable for surface elements with small aperture in the sense that when a scene is illuminated by a point source, surface elements with a small aperture tend to lie in shadows. Since a di use light source can be considered as the average of many di erent point sources, surface elements with small aperture will have small luminance under di use source conditions.
14
Wedge : Point Source 1.2
1
intensity
0.8
0.6
0.4
0.2
0 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
pixel
Figure 8: Image intensity for a point light source normal to the ground plane for ve di erent albedos.
4 Point Source Shading at Concave Orientation Discontinuities In the last section we showed that under di use lighting conditions, surface luminance attains a local minimum at the concave orientation discontinuities. In this section we simply state
a well known result [4, 7, 6] that when a concave orientation discontinuity is illuminated by a point source, surface luminance attains a local maximum at the discontinuity. This latter 3
e ect is due to the mutual illumination of the surfaces on each side of the concave orientation discontinuity. Figure 8 shows the luminance pro le of the wedge surface (from Figure 6) when it is illuminated by a point source normal to the ground plane (i.e. at high noon). The important point is that there is a local luminance maximum at the concave orientatation discontinuity. In fact, surface luminance in generally discontinuous at the orientation discontinuity. Our point is simply that luminance increases towards this discontinuity. 3
15
5 Active Shading Suppose the scene were illuminated by a di use source. We have shown that the surface luminance would then attain local minima at concave orientation discontinuities. Moreover, it can be shown that smooth valleys also typically contain local luminance minima [17] . A second image could be made by positioning a point source at the camera, and then subtracting the energy of the previous di use image. This di erence image would then be due entirely to the point source. Thus, wherever concave orientation discontinuities are present, the image intensity would switch from a minimum to a maximum as the di use image was switched to the point source image. Now, observe that the same shading switch would occur in the smooth valleys! Smooth valleys would typically contain local maxima in luminance in the point source case since, along the valley, there would be surface elements whose normals directly face the point source. To summarize, both concave orientation discontinuities and smooth valleys typically produce local luminance minima under di use conditions, and local luminance maxima when a point source is placed at the camera. The key result is thus, from a local perspective, concave orientation discontinuities (\part boundaries") have the same shading signatures as smooth valleys. Figure 9 shows an experiment to test this result about the shading signature of a smooth valley. A (smooth) drapery surface was illuminated rst by a point light source at the camera, and second by a di use source. For the (upper) point source image the locations of luminance maxima were detected with a positive contrast line operator. For the (lower) 16
Figure 9: The top image is a drapery surface illuminated by a point source at the camera. The bottom image is the same surface illuminated by a di use source. Initial estimates of the location of the local luminance maxima (for the point source) and minima ( for the di use source) are computed using line operators. These initial estimates are shown on the right side. Notice that these estimates generally include the locations of the valleys. A few thin shadows are present near occluding contours (in the upper left quadrant) because the point source is slightly displaced from the camera.
17
di use source image, the locations of luminance minima were detected using a negative contrast line operator. Notice how di erent the images are! In particular, the spatial variation of the shading is much more rapid in the point source image where both the valleys and the hills contain local luminance maxima. Despite this close packing of structure, the initial operators successfully detect most of the luminance maxima [21]. These estimates could be re ned with further processing [22], so that, for example, T-junctions could be detected. The initial operators perform even better on the di use image, where the spatial variation of the shading is much slower. Notice how well the qualitative shape of the surface is captured by the locations of the luminance minima i.e. the locations of the smooth valleys. A few points about the implementation should be noted. In practice, the point source will be near rather than at the camera. This may produce unwanted cast shadows. An example of this spurious e ect can be found in the upper left quadrant of the image. A second issue is that the locations of the luminance minima in the di use image might be slightly shifted from the locations of the corresponding luminance maxima in the point source image. Finally, the reader may wonder why we propose two types of light source in our active shading scheme when the di use source seems sucient to detect qualitative shape. The answer is that in real scenes, luminance minima may occur under di use light for reasons other than shading - there could be a smooth change in the albedo, for example. The point source is useful in that it provides independent shading evidence with which the hypotheses from the di use source image can be veri ed.
18
6 Conclusion In this paper we have addressed the question of how to compute qualitative shading information directly from the image intensities, that is, without rst \reconstructing the surface". We have shown that two popular qualitative shape features - concave orientation discontinuities and smooth valleys - have the same local shading signatures in the following sense. Under di use lighting conditions, local luminance minima are produced. Under point source conditions, where the point source is positioned at the camera, local luminance maxima are produced. By designing an environment such that images can be obtained under both these lighting conditions, one can detect these local qualitative shape features.
References [1] Woodham, RJ \Photometric method for determining surface orientation from multiple images", Optical Engineering 19 139-144 (1980). [2] Grimson W.E.L., From Images to Surfaces. (MIT Press, Cambridge, Mass., 1981) [3] Blake A and Zisserman A, Visual Reconstruction (MIT Press, Cambridge,Mass. 1987) [4] Horn B.K.P., "Understanding Image Intensities", Arti cial Intelligence 8 (1977). [5] Koenderink JJ and van Doorn AJ, \Geometrical Modes as a general method to treat di use interre ections in radiometry," Journal of Optical Society of America, 73 (1983). 19
[6] Nayar, S.K., Ickeuchi, K. and Kanade, T. \Shape From Interre ections" International Journal of Computer Vision 6 (1991).
[7] Forsyth D, and Zisserman A, "Re ections on Shading", IEEE PAMI 13 671-679, (1991). [8] Richards W, Koenderink J, Ho man DD, \Inferring 3-d Shapes from 2-d Silhouettes" Journal of the Optical Society of America (1987).
[9] Leyton, M. "A Process Grammar For Shape", Arti cial Intelligence, 34 213-247 (1988). [10] Ho man DD and Richards WA, "Parts of Recognition", Cognition 18 65-96 (1985). [11] Dudek, G and Tsotsos, JK, "Shape Representation and Recognition from Curvature", Proceedings of CVPR, Maui, Hawaii June 3-6, 1991.
[12] Kimia BB and Tannenbaum A and Zucker SW, "Toward a Computational Theory of Shape : An Overview", Computer Vision - ECCV 90 in Lecture Notes in Computer Science (Springer Verlag 1990) [13] Besl P.J. and Jain R.C. \Segmentation Through Variable Order Surface Fitting" IEEE PAMI 10 (1988).
[14] Knderink, J.J. Solid Shape (MIT Press, Cambridge, Mass. 1990). [15] Pentland A, \Finding the Illuminant Direction" J. Opt. Soc. Amer. 72 448-455 (1982). [16] Woodham RJ, and Lee TK, \Photometric Method for Radiometric Correction of Multispectral Scanner Data", Canadian Journal of Remote Sensing, 11, 132-161 1985.
20
[17] Langer, M.S. and Zucker S.W., \Shape From Shading on a Cloudy Day", McRCIM Technical Report 91-07, McGill Research Center For Intelligent Machines, McGill Uni-
versity, Montreal, Canada, 1991. [18] Gibson, JJ. The Senses Considered as Perceptual Systems (Houghton Miine, Boston, 1966). [19] Beckmann P and Spizzichino A, The Scattering of Electromagnetic Waves from
Rough Surfaces, (Macmillan, New York, 1963) [20] Knderink, J.J. and van Doorn, A.J. \Photometric invariants related to solid shape" Optica Acta 27, 981-996 (1980).
[21] Iverson, L. and Zucker, S.W. \Logical/Linear Operators for Measuring Orientation and Curvature" McRCIM Technical Report 90-06 McGill Research Center For Intelligent Machines, McGill University, Montreal, Canada (1990). [22] Parent, P. and Zucker, S.W., \Trace Inference, Curvature Consistency, and Curve Detection", IEEE PAMI 11, 823-829 (1989).
21
