Reconstructing shape from shading images under point light source ...

Reconstructing Shape from Shading Images under Point Light Source Illumination Yuji IWAHORI, Hidezumi SUGIE and Naohiro ISH11 Faculty of Engineering, Nagoya I n s t i t u t e of Technology, Nagoya, Japan 466 Abstract- A new photometric method is proposed f o r determining t h e 3-D shape of t h e object from multiple shading Images under t h e point light source illumination. When t h e surface is t h e perfect diffuser with t h e uniform reflectance, an algorithm f o r t h e determination of 3-D shape with positions is developed by using t h e method of least squares and basing on t h e principle of t h e monocular vision and the inverse square law f o r illuminance. In t h e proposed method, t h e number of t h e necessary images is four f o r t h e general surface, and can be reduced t o t h r e e f o r t h e continuous surface.

t h e necessary images can be reduced t o t h r e e for t h e continuous surface. 11. ILLUMINATING EQUATION

k Coordinate System Fig. 1 shows t h e coordinate system of t h e perspective projection. The center of t h e lens is placed a t t h e origin 0, and a point light source is located a t t h e coordinate (Xs.Ys,Zs). A surface element P(X,Y,Z) on t h e o b j e c t is mapped onto t h e image pixel p'(x.y) on t h e film. The symbol b represents t h e distance between t h e origin 0 and t h e film.

B. Definition of Vectors at SurPace Element

I. INTRODUCTION

W e define t h r e e vectors a t a surface element as follows.

As a method f o r estimating 3-D shape from 2-D shading

n : unit

surface normal vector s : unit vector from a surface element t o a point light source o : unit vector from a surface element t o t h e origin 0

information, Photometric Stereo [l], [Z] was proposed by Woodham, and has been studied f o r t h e practical applications [3]-[61. Photometric Stereo can determine t h e surface gradient of an object from t h r e e images illuminated by p a r a l l e l light beams from t h r e e different directions. However, Photometric Stereo can n o t determine t h e 3-D (X,Y,Z) coordinate of t h e o b j e c t by itself, since t h e illumination by p a r a l l e l light beams does not give t h e distance information. The process of integrating t h e gradient (p.q) is required t o get t h e coordinate (X.Y,Z) a t a surface element. On t h e o t h e r hand, when t h e object is illuminated by a point light source, t h e inverse square l a w f o r illuminance may be applied t o determine (X.Y,Z) coordinate of t h e object, because t h e distance information is included in t h e law. In t h i s paper, a new photometric method, named PSIS (Point Source Illuminating Stereo) which can determine t h e 3-D (X.Y,Z) coordinate, is proposed. PSIS uses multiple images of t h e object illuminated by a point light source positioned a t different (Xs.Ys,Zs) coordinate. If t h e surface of t h e object is t h e perfect diffuser, t h e illuminating equation becomes a nonlinear equation with four variables (p,q.Z,C). where (p,q), 2 and C are t h e surface gradient, t h e distance from t h e lens plane and t h e reflectance factor, respectively. By moving t h e point light source, w e can get a series of illuminating equations, which forms nonlinear simultaneous equations. The coordinate (X,Y,Z) a t a surface element can be d i r e c t l y obtained by solving these equations. Under t h e conditions t h a t t h e surface is t h e perfect diffuser with t h e uniform reflectance and t h a t a t least one local brightest surface element is observed in each image, an algorithm is proposed by using t h e method of least squares as follows. Firstly, t h e brightest surface elements a r e chosen from a l l images. and t h e value of t h e common reflectance f a c t o r C is determined from Z values of these elements. Secondly, t h e value of Z a t each surface element is determined by t r e a t i n g C as a known constant. In the proposed method, f o u r shading images a r e needed t o determine t h e unique solution f o r t h e general surface (i.e. Including t h e surface discontinuity ), and t h e number of

CH2898-5/90/0000/0083$01 .OO 0 1990 IEEE

v =

(-x,-y,-b) --------( x2+yz+b")le

(3)

where p and q represent t h e gradient components: (4)

The coordinate (X,Y) of t h e surface element is represented by t h e distance Z and t h e coordinate (x,Y) of t h e image pixe l as y = y - -Z b C. Illuminating Equation The luminous intensity I represents t h e luminous f l u x 4 per unit solid angle emitted by a point l i g h t source. (7)

where d o is t h e solid angle subtended t o t h e small a r e a dA of t h e surface element P by t h e point l i g h t source. d o is represented by do =

-dAc??-!.-

(8)

r2

where i is t h e incident angle. The illuminance E, in t h e direction of t h e surface normal n is given by

E,,

=

d 4 /dA = I d o /dA = I

r2

The luminous exitance M emitted from dA is represented in terms of t h e surface reflectance R.

83

n,

I

7y I t should be noted t h a t t h e r e are f o u r unhown variables. i.e.. (p,q,Z,C) In Eq.(20) and t h e equation is a nonlinear equation.

III. PRINCIPLE O F PSIS

I\

1

k Procedure The independence of t h e density D(x,y) can be held by taking images of t h e object under t h e different lighting conditions. The surface has t h e uniform reflectance R. t h e r e f o r e t h i s proposed method uses t h e constraint t h a t t h e value of C is constant f o r all of surface elements. So. t h e illumlnating equations can be solved by t h e following two steps.

Point Light

[Step 11 Choose t h e brightest surface element in each image, and estimate t h e value of C from 2 values obtained by solving t h e simultaneous illuminating equations of these surface elements.

Source

[Step 21 Determine t h e values of 2 and (p,q) at each surface element by using t h e value of C obtained in s t e p 1.

B. P r o p e r t y of Brightest Surface Element The b r i g h t e s t surface element is defined as t h e surface element on which t h e value of cos i /r" becomes maximum. and exists as one of t h e local brightest surface elements. The l o c a l brightest surface element has t h e gradient n given by COS i =1, n = s (23)

z

i.e. t h e surface normal vector d i r e c t s towards t h e l i g h t source. This can be explained by means of t h e reductive absurdum a s follows. From t h e physical constraint, cos i should be g r e a t e r than 0 and less than o r equal t o 1 at t h e observed surface element. A s shown in Fig.2, t h e surface element (a) is assumed t o be a local brightest surface element, and let n # s a t its surface element. If t h e local surface is assumed t o be a plain, t h e plain should cross t h e sphere surface whose center is located a t a point light source and on which t h e local brightest surface element exists. This means t h a t another surface element (b) exists near t h e local brightest surface element (a) and e x i s t s inside t h e sphere surface. Then, ( n ., s),, < ( n D. s d and r. > ra a r e obtained. Accordingly, ( n ., s .)/ra2 < ( n L, s a)/ra". Let t h e corresponding image densities be D, and D a respectively, then D. < Do. This Is in conflict with t h e definition t h a t t h e surface element (a) is t h e local brightest surface element. Therefore, n must be equal t o s a t t h e l o c a l brightest surface element.

Fig.1. The coordinate system of PSIS.

Assuming t h a t t h e reflectance p r o p e r t y of t h e surface is t h e perfect diffuser of t h e Lambert's Law,

M= nL

(11)

Therefore, t h e luminance L of t h e surface element is expressed by

cos i is t h e s c a l a r product of unit vector n and s , then

cos

-

P(Xs-X)+q(YS-Y)-(~s-~) r(p2+q2+1)

(13)

in which r is t h e distance between t h e surface element and t h e point light source. r2 Is represented by

r" = ( x ~ - x ) ~ + ( Y s - Y ) " + ( ~ s - ~ ) '

(14)

Substituting Eqs.(l3) and (14) i n t o Eq.(12) gives

n=Zs-2

(18)

Let D be t h e image density on t h e film. and assuming t h e Gamma Property t o be linear, then D=kL

Spherical Surface

r =raConst.

(19)

1

/J5/

where k is a proportional constant. From Eqs.(lS) and (19). t h e image density D is rewritten by (20) Fig.2. Explanation of t h e p r o p e r t y of brightest surface elements.

(21)

84

C. Step 1 C-1. Simultaneous Equations at Brightest Surface Elements Let k be t h e ordinal number of t h e selected brightest surface element illuminated by t h e k-th light source, and let j be t h e ordinal number of t h e image of t h e object illuminated by t h e j-th light source. Letting N be t h e number of input images, t h e illuminating equations a t t h e brightest surface elements become D ~ J C fJ(PkBqk,Zu)

(for k=l..N. j=l..N)

where, 1UJ

= XSJ

muJ

=

- Xu-&Zb

YSJ -

nkJ = ZsJ

zu

where P means t h e summation of N C 2 combinations. Eq.(33) becomes t h e following loth-order equation of Zk.

~ k - ~ -

- Zu

D k i 2 r k t 2huje = D k r 2 rkJ2huIe

The gradient (pk.qu) a t t h e selected surface element is represented by using Eq.(23) as pk =

-

and q u

nxu

-

!= nkk

(25)

Substituting Eq.(25) i n t o Eq. (24) gives fJ(pk$qU.ZU)= fJ(Zk)

D. Step 2 D-1. Relation between Gradient and Distance at Surface Element from Three Images In s t e p 2, t h e values of t h r e e variables (p,q,Z) a t each surface element a r e determined under t h e constraint which C is t r e a t e d as a known constant. The relation between (p,q) and Z can be obtained from t h r e e independent image density D J (j=1,2,3) a s follows. From two illuminating equations of D t and D2. t h e operat i o n Dl/D2 gives

By using f J ( Z k )of Eq.(26). t h e following simultaneous equations a t t h e brightest surface elements (k=l..N) a r e obtained

D u J = C fJ(Zu) ( f o r j=l..N.

k=l..N)

(27)

Let t h e number of simultaneous equations be ne and t h e number of unknown variables be n,. then n+=N" and n,=N+l. In s t e p 1. t h e values of (N+1) unknown variables. ( Z d . k-,. , N and C a r e estimated by using t h e method of l e a s t squares.

D I r13( P 12 + m 2 -n2 1 = D2 rZ3 ( p l I + q ml -nl )

C-2. Objective Function Let t h e objective function El be designated as N

Et =

N

( 3-1

DUJ

- c fJ(Zk)

(35)

This means t h a t only one minimum point of Et ~ Z U e)x i s t s when t h e number of light sources (images) is more than three, because more than t h r e e light sources produce more than two independent equations of ZU. Z k which minimizes E l l is adopted a s t h e i n i t i a l value of Z u which gives t h e minimum of some minimal points of El I.

(36)

Similarly, from two equations of D t and D3. D l r13( P 13 + m 3 -n3 1 = D 3 r3" ( P 11+ q ml -nl )

(28)

( 37)

These equations a r e linear equations of (p,q). Then,

Considering t h e quadratic expression of C in Eq.(28), and l e t t i n g fJ(zU) be f U J , t h e objective function El can be rewritten by

where, The f i r s t term of t h e right hand side of Eq.(29) equals zero when C satisfies t h e next relation.

For a s e t of t h e values of (Zu). U-I.. N. t h e value of C t h a t minimizes E l can be determined by Eq.(31). Under t h i s condition. E l equals E2. Hence, t h e searching problem of t h e minimum value of E l becomes equivalent t o t h a t of Ea.

A(Z) = a l 12 - a2 11 B(Z) = a l ma - a, m, C(Z) = al 13 - a3 11 D(Z) = al m3 - a3 ml E(Z) = a l n2 - a= nl F(Z) = at n3 - a3 n l a3 = DJrJ3 (j=1,2,3)

D-2. Determination of Distance at Surface Element from Four Images (p,q) given by Eqs.(38) and (39) a r e rewritten by

C-3. Inltial Values of Zu

N equations of Eq.(27) D U J = C fJ(ZU), j = l . . N

a r e held at t h e brightest surface element k. From two equations of D U I and D U J obtained from t h e different l i g h t source i and j (1 # j), t h e following high-order equation of Z, can be obtained as

Let C obtained in s t e p 1 be C., DJ /

then,

85

c.

= dj

(j=1..4)

and let (43)

v2 m i - va n i )" di2( VI2+ V 2 + Vz2 ( lr2+ m12+ nr2I3 (for i=1,2.3)

so t h a t t h e brightest surface element in each image exists within t h e commonly illuminated region in a l l images. L I : (XSl.YS1 ,zs 1)=(32.191.405) L2: (XS2,Y~2.Z~2)=(166,-108,419) La: (XSj.YSa.Z~3)=(-84,-130,386) L4: (X~4.Y~4,Z~a)=(180,11,387) [mm1 These values were computed photometrically by t h e calibration using t h e method of least squares from t h e image densities of original 256x256 pixels.

( V I 1, + =

(44)

is obtained f o r D, t o Ds by using relations of Eq.(20) t o Eq. (22). Similarly. f o r D4. (

VI

1J + v2 m J - v3 nJ )2 = dj"( vi2+ Vn2+ vj2 ) ( 1j2+mJ"+ (for j=4)

I~J')~

(45)

B. Reconstructed Results

is obtained. Eq.(44) and Eq.(45) a r e independent high-order

Input images with 8-bit quantizations and t h e commonly illuminated region are shown in Fig.4. In case of four input images, images (a)-(d) a r e used. On t h e o t h e r hand, in case of t h r e e input images, images (a)-(c) a r e used. The brightest surface element in each image is p l o t t e d by a black point in Fig.4. The obtained objective function E l I(Za).a-l.. N f o r N=4 is shown in FIg.5. Z a which minimizes E l i is determined by g e t t i n g t h e values of Z which give t h e minimal points of E l ,. Furthermore, using t h e obtained Z a as t h e i n i t i a l values, Powell method [71 is used t o search t h e minimum point of E 2 of Eq.(30). The objective functions E3(Z) in s t e p 2 a r e shown in Fig.G-(a) and (b) f o r N=4 and N=3 respectively. These a r e Ea(Z) a t t h e surface elements on t h e Y-Z plain. In case of f o u r input images, t h e number of minimum points of EdZ) is

equations of Z each other. Accordingly, an unique solution of Z f o r each surface element can be obtained when f o u r input images a r e used. From t h e above discussions, Z which minimizes t h e following objective function Ea(Z) is adopted as t h e solution of Z which gives t h e minimum of some minimal points of EdZ). EdZ)

P (DJ - c e f J ( ~ l ( z ) . ~ d z ) ,1"z )

(46)

where Z means t h e summation by t h e point light source ordin a l number j (j=1..4). When f o u r images a r e used, only one minimum point of E3(Z) exists.

E. Determination of Distance at Surface Element from Three Images In s t e p 1. more than t h r e e input images (N 2 3) a r e necess a r y t o obtain unique solutions of Z a and C. t h a t minimize Eq.(29). When t h r e e input images a r e used, t h e assumption t h a t Z values of t h e neighbouring surface elements are close t o each o t h e r (i.e. t h e assumption of t h e surface continuity) is used in s t e p 2. In Eq.(46), t h r e e terms f o r t h e summation a r e n o t dependent, then t h e number of t h e minimum point of E d Z ) is not one. Therefore, Z t h a t minimizes E3(Z) is orderly estimated a t from t h e brightest surface element towards t h e surrounding surface elements by using t h e value of Z a obtained in s t e p 1 as t h e i n i t i a l value. The i n i t i a l value of Z f o r minimizing E,(Z) a t t h e surface element is set t o t h e value of Z ob-

CCD CAMERA TI-22A b=25mm, Filmsize=?mm

Point ligvt Source

~Xs,;(S,Zsl

tained a t its neighbouring surface element. The searching direction is set t o t h e direction t h a t decreases E3(Z). Let t h e i n i t i a l value be Z and t h e moving s t e p of Z be dZ, then t h e sign of dZ t h a t satisfies Eo(Z+dZ)< Ea(Z)

I

(47)

Object R=45mm

Z=Zo Plane

is f i r s t l y determined. Z which gives t h e minimal point of E d Z ) is adopted as t h e solution of Z a t each surface element. This algorithm of t h r e e input images is effective f o r t h e continuous surface.

L

F. Arrangement o f Point Light Sources I t is necessary f o r solving simultaneous equations t h a t t h e constituent equations a r e independent each other. Furthermore, it is desirable t h a t each light source gives a g r e a t var i e t y of illumination. Since t h e luminance of a surface element is determined by t h e incident angle and t h e distance from t h e light source, t h e light sources must be arranged so t h a t both t h e incident angles and t h e distances a t t h e surface elements a r e much different. Accordingly, it is required t h a t t h e distance between light sources should be large.

Fig.3. Observation environment.

IV. Experiment k Method of Experiment

As an object, a p l a s t e r hemisphere with r=45[mm]was adopted, which w a s put on Z=ZO plane, where Zo=695[mm]. The film size w a s 7x7[mml. b was equal t o 25[mml, and the image plane w a s divided into 64x64 pixels. Fig.3 shows the observation environment. The diffused spherical light source with 30[mm] diameter w a s adopted. The distance r is more than ten times of t h e scale of t h e light source, then t h e light source may be regarded as a point light source. Each light source is located

( d )

( e )

Fig.4. Input images. ( a ) - ( d ) : f o u r input images. ( e ): commonly illuminated region in a l l images.

86

"I

300

600

900

1200

1500

2

nml

Fig.6. Objective function EdZ).

Fig.5. Objective function E1 I ( Z k ) .

The e r r o r s may be caused from some of f a c t s such as t h a t t h e assumption of t h e perfect diffused surface is not exactl y held and t h a t t h e gradients a t t h e brightest surface elements a r e calculated by basing on t h e limited image sampling resolution. I t is considered t h a t t h e reliability of t h e estimated value of C is reduced a little by these e r r o r s .

V. CONCLUSION A new photometric method named PSIS is proposed and discussed. The method adopts t h e principle of t h e monocular vision and t h e point light source illumination instead of t h e p a r a l l e l light illumination of Photometric Stereo. Conditions used in t h i s paper a r e as follows. (1) t h e inverse square law f o r illuminance (2) t h e o b j e c t with t h e perfectly diffused surface (3) approximation of t h e paraxial ray (4) neglecting t h e e r r o r of t h e precision of t h e coordinate of t h e image pixel An algorithm based on t h e method of least squares is developed t o determine t h e distance 2 of each surface element of an object from f o u r and t h r e e shading images. Under t h e above conditions, it is shown t h a t PSIS can reconstruct t h e shape of an o b j e c t with positions in 3-D space. I t should be noted t h a t f u r t h e r e f f o r t s a r e required t o extend t h i s method f o r r e a l complex objects, and several extensions of t h e work presented h e r e may be possible.

Fig.7. Reconstructed shapes f o r four input images

only one, while in case of t h r e e input images, t h a t is not always only one. The r e s u l t s of the distance distribution obtained in s t e p 2 are shown in Fig.7, where t h e coordinate (X,Y) of a surface element is determined from 2 by Eqs.(5) and (6). The shape reconstruction of t h e object is performed by joining (X,Y.Z) values of neighboring surface elements. In Fig.7. (Zo-Z) is adopted as t h e measure of t h e height of t h e surface element t o display reconstructed shapes, and t r u e curves of t h e object a r e also shown on t h e figure f o r t h e comparison. Under t h e condition t h a t t h e image quantization levels a r e about maximum 200. t h e mean absolute e r r o r s of t h e Z-distribution f o r t h e reconstruction a r e within 1[%1both f o r f o u r input images and three input images. Evaluating e r r o r s of t h e 2-distribution a r e a s follows. Mean absolute e r r o r s Mean square e r r o r s

Four Images 2.5 mm 9.1 mm"

REFERENCES

[ l ] R.J.Woodham. "Photometric Stereo : A reflectance map technique f o r determining surface orientation from image intensity," Proc. SPIE, vol. 155, pp.136-143, 1978 [21 R. J.Woodham. "Photometric method f o r determining surface orientation from multiple images," Opt. Eng., vol. 19. no.1. pp.139-144. 1980 [31 K.Ikeuchi, "Determining 3D shape from 2D shading information based on t h e reflectance map technique," Trans. of IECE JAPAN, Vol. J65-D, N0.7. pp.842-849, 1982 [41 H.Kitagawa, M.Suzuki, and H.Fujita, "Extraction of surface gradient from shading images," Trans. of IECE JAPAN. Vol. J66-D. NO. 1, pp.65-72, 1983 [51 H.Hong, T.Kawashima. and Y.Aoki. "A method t o reconstr u c t shape and position of 3-Dimensional o b j e c t s using photometricstereo system," Trans. of IECE JAPAN, Vol. J69-D, N0.3, pp.427-433, 1986 Y.Iwahori, N.Hiratsuka. H.Kame1. and S.Yamaguchi. "Extended photometric s t e r e o f o r an o b j e c t with unknown reflectance property," Trans. of IEICE JAPAN, Vol.J71D, No.1, pp.110-117, 1988 J.Kowalik and M.R.Osborne, "Methods f o r unconstrained optimization problems", American Elsevier Publishing Company, Inc., Chap.4. 1968

Three Images 2.7 mm 10.8mm"

Results from four images a r e b e t t e r than those from t h r e e images. The processing times in s t e p 1 a r e about 3.5 minutes f o r f o u r images and 1 minute f o r t h r e e images, and those f o r each surface element in s t e p 2 a r e about 3 seconds in case of four images and 2 seconds in case of t h r e e images. These processing times a r e measured under t h e environment of 80286 CPU + 80287 Arithmetic CO-Processor (10MHz clock) hardware and MS-DOS Turbo Pascal system software.

87