SFS Based View Synthesis for Robust Face ... - Semantic Scholar

SFS Based View Synthesis for Robust Face Recognition WenYi Zhao Rama Chellappa Center for Automation Research University of Maryland College Park, MD 20742-3275 Email: f wyzhao, rama [email protected] Abstract Sensitivity to variations in pose is a challenging problem in face recognition using appearance-based methods. More speci cally, the appearance of a face changes dramatically when viewing and/or lighting directions change. Various approaches have been proposed to solve this dicult problem. They can be broadly divided into three classes: 1) multiple image based methods where multiple images of various poses per person are available, 2) hybrid methods where multiple example images are available during learning but only one database image per person is available during recognition, and 3) single image based methods where no example based learning is carried out. In this paper, we present a method that comes under class 3. This method based on shape-from-shading (SFS) improves the performance of a face recognition system in handling variations due to pose and illumination via image synthesis.

1 Introduction

Face recognition has become one of the most active areas of research in image analysis and understanding. Many methods have been proposed for face recognition [1]. Even though many algorithms have been successfully applied to the task of face recognition/veri cation, they all have certain drawbacks. One of the major diculties is the pose variation problem, that is, the fact that system performance drops signi cantly when pose variations are present in the input images. This diculty is clearly revealed in the recent FERET test report, and solving the rotation-indepth problem has been suggested as a major research issue [2]. Moreover, when the illumination variation also appears in the face images, the task of face recognition becomes even more dicult. Researchers have proposed various methods to handle the rotation problem. Basically they can be divided into three classes: 1) multiple image based methods where multiple images per person are available, 2) hybrid methods where multiple training images are available during training but only one database image per person is available during recognition, and 3) single image based methods where no training is carried out. We have [9, 8, 17, 21] in the rst type, and [19, 16, 14, 11] in the second type. Up to now, the second type of approaches is the most popular one. The third type of approaches does not

seem to have received much attention in the context of face recognition. In this paper, we rst review some existing methods, and then carefully examine the problem of pose/illumination variations using a re ectance model with varying albedo. Using such a model we are able to assess the diculty of this problem and evaluate the ecacy of existing methods analytically. We then propose an SFS-based view synthesis technique for face recognition (the third type) which is robust to pose and illumination changes. Using this view synthesis technique, we can convert any input images into prototypes which are front-view and under frontal lighting. Finally, we feed these prototypes into an existing system such as the subspace LDA [6] to perform recognition. One major motivation for us to propose SFS for view synthesis is that we have recently developed an SFS-based illumination-insensitive face recognition method [6]. Combining the new symmetric SFS which assumes varying albedo for face images and a generic 3D face model, very good prototype images synthesized from front-view input images have been obtained.

2 Existing Methods

Among the rst type of work, one of the earliest work is by Beymer [9] in which a template-based correlation matching scheme was proposed. In this work, pose estimation and face recognition are coupled in an iterative loop. For each hypothesized pose, the input image is aligned to database images corresponding to a selected pose. The alignment is rst carried out via a 2D ane transformation based on three key feature points (eyes and nose), and then optical ow is used to re ne the alignment of each template. After this step, the correlation scores of all pairs of matching templates are used to perform recognition. The main restrictions of this method are 1) many images of dierent views per person are needed in the database, and 2) no lighting variation (pure texture mapping) and facial expressions are allowed, and nally 3) the computational cost is high since it is an iterative searching approach. More recently, an illumination-based image synthesis method [21] has been proposed as a potential method for robust face recognition handling both pose and illumination problem. This method is based on the illumination cone [20] and can handle

illumination variations quite well. To handle variations due to rotation, it needs to completely resolve the GBR (generalized-bas-relief) ambiguity when reconstructing the 3D shape. Numerous algorithms of the second type have been proposed and are by far the most popular ones. Possible reasons for this are: 1) it is probably the most successful and practical method up to now, 2) it utilizes prior class information. We review three representative methods here: the rst one is the linear class based method [19], the second one is the graph matching based method [15], and nally the third is the view-based eigenface approach [13]. The image synthesis method in [19] is based on the assumption of linear 3D object class and the extension of linearity to images which are 2D projections of the 3D objects. It extends the linear shape model (which is very similar to Cootes et al.'s active shape model [12]) from a representation based on feature points to full images of objects. To implement their method, correspondence between images of the input object and a reference object is established using optical ow. Also correspondences between the reference image and other example images of the same pose are computed. And nally the correspondence eld to the input image is linearly decomposed into the correspondence elds of the examples. Compared to the parallel deformation scheme in [11], this method reduces the need to compute the correspondences between images of dierent poses. This method was extended in [16] to carry an additive error term for better synthesis. In [15], a robust face recognition scheme based on Elastic Bunch Graph Matching (EBGM) is proposed. The authors basically assume a planar surface patch in each key feature point (landmark), and learn the transformation of `jets' under face rotation. And they demonstrated substantial improvement in face recognition under rotation. Also their method is fully automatic, including face localization, landmark detection, and nally a exible graph matching scheme. The drawback of this method is the requirement of accurate landmark localization which is not an easy task especially when illumination variations are present or face image size is small. The popular eigenface approach [3] for face recognition has been extended to view-based eigenface method in order to achieve pose invariant recognition [13]. This method explicitly codes the pose information by constructing an individual eigenface for each pose. More recently, a more general, uni ed framework called the bilinear model has been proposed in [18]. Despite their popularity, these methods have some common drawbacks: 1) they need many example images to cover all possible views, and 2) the illumination problem is excluded from the pose problem. Finally, there is the third class of approaches which include low-level feature-based methods, invariant feature based methods, and 3D model based methods. In [22], a Gabor wavelet based feature extraction method is proposed for face recognition and is robust to small angle rotation. There are a large number of papers on the subject of invariant feature in the computer vision literature. To our knowledge, serious application of this technology to face recognition has not been ex-

plored yet. However, it is worthwhile to point out that some recent work on invariant methods based on images [23] may shed some light in this direction. For synthesizing face images under dierent appearances/lightings/expressions, 3D facial models have been explored in computer graphics, computer vision and model-based coding communities [28, 27, 7]. In these methods, the face shape is usually represented either by a polygonal model or a mesh model which simulates tissue. However due to its complexity and computation cost, any serious attempt to apply this technology to face recognition has not been made yet except in [7] where a dataset of 24 range images of eight persons with three views of each was tested.

3 Problem De nition and Solutions

In order to systematically study the pose variation problem in face recognition, we use the Lambertian model with varying albedo to model the face shape. We divide the pose variation problem into three cases/scenarios, ranging from easy to hard.

3.1 Image Transformation Under 3D Rotation

Before we go into a discussion of dierent scenarios of pose variation problem, we study the transformation between images of dierent views. We use the Lambertian model combined with varying albedo to model the face shape. To simplify the analysis, we assume that the 3D rotation is about the y-axis without loss of generality. The Lambertian model we use here is commonly used in the SFS literature, and the standard equation is: 1 + pPs + qQs (1) I = 1 + p2 + q2 ; 1 + Ps2 + Q2s where (p; q) are the shape gradients (partial derivatives of the depth map z [x; y]), Ps = k sin  cos  and Qs = k sin  sin  , and k is the length of the lighting vector L~ . The light source is represented by the two angles: slant  (the angle between the negative L~ and the positive z -axis) and tilt  (the angle between the negative L~ and the x-z plane). Let us rst assume that the problem of illumination does not exit, i.e., lighting condition does not change across views. We rst prove the following Lemma on the changes in partial derivatives under 3D rotation.

p

p

Lemma 1

Suppose that the partial gradients (p[x; y]; q[x; y]) become (p [x0; y0 ]; q[x0; y0 ]) after the underlying surface is rotated in the x-z plane about the y-axis by ; then they are related by p [x0; y0 ] = tan( + 0 ) [x;y ] cos 0 q [x0; y0 ] = qcos( +0 ) ;

(2)

where tan 0 = p[x; y].

Proof Let us assume that the surface patch we are

interested has a surface normal ~n = (p; q; 1). Now the whole surface is rotated by  about the y-axis, so the surface normal vector is also rotated by  and

Theorem 2 Assuming an arbitrary single light

z z’

source (Ps ; Qs; 1), the rotated (in the x-z plane about the y-axis) image I  [x0; y0 ] is related to the prototype image Ip via

y/y’

q p

P

I  [x0; y0 ] = 1z; Ip [x; y](cos  ? p[x; y] sin ) p1+P12 +Q2 s s q cos(0 ) Q + 1]; [tan( + 0 )Ps + cos( s +0 )

1 θ0 x’ θ

(7)

x

Figure 1: Surface rotates by  about y-axis. becomes n~ (Fig. 1). With simple geometric interpretation, we have p = tan 0 . Since rotating the surface  is equivalent to rotating the coordinate system by ?, we rotate the coordinate system about the y-axis and keep the surface unchanged. From Fig. 1, it is easy to see that the projection point P of ~n onto x ? z plane and x0 ? z 0 plane are the same. Based on this, it follows that n~ = [ 1 + p2 sin( + 0 ); q; 1 + p2 cos( + 0 )]: (3) Renormalizing the vector n~ to be [p ; q ; 1], we have p = tan( + 0 ) q (4) q = p1+p cos( + ) :

p

p

p Replacing 1= 1 + p with cos  because tan  = p, 2

2

0

0

0

we prove the Lemma. Based on this Lemma we state the following theorem (proof omitted here)

Theorem 1 Assuming an arbitrary single light source (Ps; Qs; 1), the rotated (in the x-z plane about the y-axis) image I  [x0; y0 ] is related point-wise to the original image I [x; y] as I  [x0; y0 ] = 1z; I [x; y](cos  ? p[x; y] sin )    tan(+ pP)Pss++qQs +1

q cos(0 ) cos( + 0 )

Qs +1

(5)

; where tan 0 = p[x; y] and 1z; is the indicator function indicating possible occlusion determined by the shape and rotation angle. Assuming the straight single light source (Ps = 0 and Qs = 0), we have the following corollary: Corollary 1 For a straight single light source (Ps = 0 and Qs = 0), the rotated image is related to the original image as I  [x0; y0 ] = 1z; I [x; y](cos  ? p[x; y] sin ) (6) where 1z; is de ned in Theorem 1. 0

When only geometric transformation matters (no re ectance model), we can simply apply image warping technique to obtain the image transformation which is commonly used in [9, 19, 11]. To solve the correspondence problem [x; y] ! [x0; y0 ] optical ow is applied. However, when illumination problem is coupled with pose problem, these methods are not sucient anymore. To analyze this problem, we provide the following theorem (proof omitted here)

where the prototype image is the frontal image with the straight single light source L~ = [0; 0; ?1]: Ip [x; y] = [x; y]

p1 + 1p + q : 2

2

(8)

With these results in hand, we are ready to discuss the pose variation problem in dierent cases. We assume no illumination change for the rst two cases, and nally we address the case of pose variation coupled with illumination variation using a simple 3D face model. But before we discuss the various cases, we would like to study an interesting question which is particular to a class of face recognizers based on Fisher/Linear Discriminant Analysis [4, 5, 6].

3.2 An Interesting Question

Suppose we train only the (symmetric) front-view faced images fI1 ; I2;


; IM g with the straight light source and obtain a linear optimal discriminant mapping W . Then the question is if we can apply the same mapping to the rotated images fI1 ; I2 ;


; IM g after they are warped into perfect frontal (virtual) views? This question is not interesting at all if we do not consider a re ectance model since otherwise image warping is enough. However when the0 lighting model is included, the same point [x; y] ! [x ; y0 ] in the face appears dierently under dierent poses. And pure image warping does not resolve the re ectance dierence! But the interesting thing is that if we can assume frontal face symmetry as in [30], then we have the following proposition

Proposition 1 The optimal discriminant mapping W learned from front-view face images is also optimal for distinguishing among rotated face images provided that  all the eigenimages are symmetric, which is a direct result of training on symmetric/frontal images.  the light source is a single straight light: Ps = 0 and Qs = 0, and it produces symmetric frontal images from symmetric face shapes.  the rotated image is warped into frontal (virtual) view (the correspondence problem solved) with no occlusion.

This proposition is based on the following Lemma. Lemma 2 For a straight single light source (Ps = 0 and Qs = 0), if the correspondence problem [x; y] !

[x0; y0 ] has been solved, then the subspace projection coecient vectors between the frontal (symmetric) and 3D rotated () images of class (person) C are related as ( C + A ) = cos ( C + A ); (9) if we assume symmetric class (pLC [x; y] = ?pRC [x; y]), and that there is no occlusion, i.e. 1z;C = 1. We use A to denote the projection vector for the average image IA : [IA 1;


; IA n ], where denotes the sum of all element-wise products of two matrices (vectors).

Proof Consider the basic expression for the eigenspace decomposition of an image of class C : IC = IA +

Xn C;ii i=1

(10)

where i are the eigen-images and C;i are the projection coecients. These projection coecients are computed as C;i = (IC ? IA ) i (11)  C;i = (IC ? IA ) i : Applying Corollary 1 and dividing the image I into left and right halves I L and I R , we have C;i = ICL [x; y] Li [x; y] + ICR [x; y] Ri [x; y] ?IA [x; y] i [x; y]; (12) and  C;i = [cos  ? pLC [x; y] sin ]1zC ; ICL [x; y] Li [x0; y0 ] + [cos  ? pRC [x; y0] sin0 ]1zC ; ICR [x; y] Ri [x0; y0 ] ? IA [x; y] i [x ; y ]: (13) Now if we could solve the correspondence problem and the rotated images are warped into frontal ones, we can replace [x0; y0 ] with [x; y]. Utilizing the symmetric properties of images and class C (pLC = ?pRC ) and the assumption 1zC = 1, we have the following equation C;i = 2ICL [x; y] Li[x; y] ? IA [x; y] i [x; y] (14) for C;i , and  C;i = [cos  ? pLC [x; y] sin ]ICL [x; y] Li [x; y] + [cos  ? pRC [x; y] sin ]ICR [x; y] Ri [x; y] ? IA [x; y] i [x; y] = 2 cos ICL [x; y] Li [x; y] ? IA [x; y] i [x; y] (15) for  C;i . This leads to the following simple scale relation ( C + A ) = cos ( C + A ); (16) concluding our proof. Based on Lemma 2, it is obvious that Proposition 1 is true since scaling and adding constant do not change

the optimality of discriminant mapping W . However, we should notice that for Proposition 1 to be valid we need two conditions: 1) there is no occlusion for the whole image, and 2) the light source is a straight single light. On the other hand, Proposition 1 does not enforce further assumptions such as common class shape which is used in the literature (for example, in [29]); hence it is a fairly general statement.

3.3 Cases of Pose Variation Problem 3.3.1 Case I: Small Rotation Problem

We rst state the problem: suppose that we have (symmetric) frontal images fI1; I2 ;


; IM g and slightly rotated face images fI1 ; I2 ;


; IM g where the rotation angle  is small. The appropriate strategy to perform recognition for these slightly rotated images follows from Corollary 1. We have I  [x0; y0 ]  I [x; y]; (17) because sin   0, cos   1 and 1z;  1 for small angle .0 For small rotation, we can approximately align [x ; y0] and [x; y] by normalizing all images in a proper way. For example, the face images can be normalized with respect to the two eyes. Base on this, we conclude that pose variation is manageable when rotation angle is small.

3.3.2 Case II: Using Prior Class Knowledge

This case is the most commonly addressed one in many existing approaches [11, 19, 15, 13, 18]. It makes the following assumption: suppose that we have a full set of front-view face/symmetric images fI1; I2 ;


; IM g and partial set of rotated training images fI1 ; I2 ;


; IK g, then how do we classify the untrained image Ij where K < j  M ? The basic idea of solving this problem is to explore the pair-wise information for the available image pairs: frontal and rotated. That is we rst learn the transformation from the frontal image to the rotated one, i.e. !  , and then apply this learned transformation to the new testing/rotated image. This knowledge could be made explicit in the following manner:  

Corresponding point mapping between dierent images of the same pose [19] or dierent poses [11]. Transformation of \jets" between images of dierent poses [15].

This learned knowledge could be implicit, too. For example, in the view-based eigenface method [13] or the bilinear model method [18], the subspace projections of the frontal images into the front-view eigenface and of the rotated images into the rotated-view eigenface are implicitly assumed to be the same. To avoid the dicult correspondence problem, we propose the following (nonlinear) pure learning method:

Algorithm I

300 250 200 150 100 50 0 0 10 20

0 10

30

20 40

30 50

40 50

Figure 2: Image warping using a simple 3D face head. The images are arranged as follows: the rst column is the 3D head, the second column is the original frontal image, and the following (left-to-right) images are rotated by 50, 100, 250, and 350 respectively. 1. Apply nonlinear Principal Component Analysis (NPCA) [32] to map both the frontal and rotated views into the subspace of a nonlinear space. 2. Apply a linear learning method such as least-squareestimation to obtain a mapping between the NPCA coecients of the frontal view and the rotated view. 3. Use the linear mapping to perform recognition in the nonlinear space.

From a theoretical point of view, it is hard to verify the validity of this algorithm. So we have conducted experiments on real face images and report the result in the experimental section.

3.3.3 Case III: SFS-based problem

As the most challenging one, case III has the following assumptions: 1) we do not have enough prior class information, for example, test image has very dierent rotation angle from example images; 2) we need not only to include the illumination model but also consider the illumination change. We devote the next section to address this complex problem.

4 SFS-based Robust Face Recognition

Before we discuss the combined problem of pose and illumination variations, we brie y discuss a simple technique which is commonly used in the literature.

4.1 Image Warping

Without considering illumination, the rotation problem is just an image warping problem. Image warping has been used in [11] (called parallel deformation) where the correspondence is obtained by prior class knowledge and optical ow. In [15], exible graph matching is applied so that the correspondence problem is not so critical. And the correspondence problem is restricted to only landmarks (about 48 nodes for frontal view) where they are identi ed independently using some prior learned (face) class knowledge [14]. We propose using a simple 3D model (depth map) to solve the correspondence problem. Figure 2 shows such an example. To t individual images better, the simple 3D model can be deformed based on individual 2D features: face boundary, eye centers, the nose tip etc. Also we need to estimate the pose of a given image. We discuss the details in the following section.

Figure 3: Image synthesis under dierent rotations and illuminations using a generic head. The images are arranged as follows: in the row direction (left to right) the images are rotated by the following angles: 50, 100, 250, and 350; in the column direction (top to bottom) images are illuminated under the following conditions: ( = 00) and ( = 300,  = 1200).

4.2 Varying Pose and Illumination

This is the most dicult problem where we need to handle both pose and illumination variations. The technique used in computer graphics to model 3D face is too complicated for the task of face recognition with large database. In theory, we can also apply SFS or a recent work on symmetric SFS [30] to recover the complete 3D shape. However to be more practical, we propose using a simple 3D model to by-pass this 2Dto-3D process. This technique has been successfully applied in [30] to address pure illumination problem with pose xed.

4.2.1 Virtual View Synthesis for Recognition

The basic idea to improve the system performance against illumination and pose variations is to preprocess the input images. After preprocessing/synthesizing, we can feed the synthesized prototype images into an existing system such as the subspace LDA [6]. Using Theorem 2 we can synthesize virtual images at dierent views and illuminations as demonstrated in Fig. 3. To actually apply image synthesis techniques for face recognition, we need the illumination direction of the light source and the face pose. In this way, we do not need to synthesize all views and illuminations in our database in order to recognize input images under various conditions. Instead, we can synthesize the prototype view de ned in the database from an input image acquired under dierent views and illumination directions.

4.2.2 Light Source and Pose Estimation

Most existing pose estimation algorithms [25, 24, 10, 26] use some prior class knowledge, that is all face heads are similar. Instead of using 2D example images, we propose using a simple 3D model to estimate the pose. Hence the pose estimation problem becomes a parameter estimation problem as follows R ()) ? I R ]2; (18)  = arg min[IM F R where IR is the given image and IMF is the synthesized image generated from a simple 3D face model MF .

Moreover, we can also incorporate the estimation of the light source by adding two more parameters: ( ; ;  ) = arg;; min[IMR F (; ;  ) ? I R ]2: (19) However, as noticed in [30], such formulations ignore the reality of a varying albedo. To better address this problem, the albedo-less self-ratio image rI [x; y] was introduced in [30] for symmetric frontal face image I [x; y]: I [x; y] (20) = pPs ; rI [x; y] = ? I [x; y] 1 + qQ + s I [x;y]+I [?x;y] and I [x; y] = ? 2

where I+ [x; y] = I [x;y]?I [?x;y] . The advantage of this approach is that 2 it eliminates the eect of albedo, so comparisons between real and synthetic images are possible. However, in order to apply this method to rotated images of symmetric objects, we need additional processing. Using Theorem 1, we can formulate a new estimation problem: ( ;  ;  ) = arg;; min[rIMF (;  ) ? rI F (; ;  )]2; (21) where rI F (;; ) is the self-ratio image for the virtual frontal view generated from the original image IR via image warping and texture mapping using Theorem 1.

Figure 4: Some images used in the database. The rst row are the rotated views, the second row are the synthesized frontal views, and the third view are the real frontal views. The rst three columns are good image pairs, while the last three columns are bad pairs.

5 Experiments

Currently, light source and pose estimation (21) is not carried out. Instead very rough pose information is given manually. The light source is also assumed to be frontal though in fact it may be not. Basically we have only experimented on model-based pure image warping and plan to implement full SFSbased view synthesis in near future. However, as we have shown earlier, this is a good approach for eigensubspace based method even when Lambertian model and frontal lighting are assumed.

5.1 Database

The data set we used here is drawn from FERET and Stirling databases [6]. To compare, the quality of this dataset is lower than the one reported in [19] (which is also used in several subsequent works on 3D face recognition such as [29]) and the size of normalized images we are using is much smaller than those in [19, 14]. There are 108 pairs of face images: front view and quarter-pro le view, all normalized to the size of 48  42 w.r.t. the two eyes. The poses of these faces are not quite consistent, and we only apply a unique rotation angle picked manually to all images.

5.2 Visualization of Image Synthesis

There are two ways of synthesizing images for face recognition. One is to synthesize images from the database to match the view and lighting conditions presented in the test image. The other way is to synthesize the prototype view and lighting conditions for the database. It is obvious that the second approach is more ecient. For example, in the database, we usually have frontal views and the test image could be a rotated one. In such a case, we can generate the virtual frontal view from the test image and match it

Figure 5: Synthesis of virtual frontal view from another view: The rst and fourth columns represent all rotated views, the second and fth columns represent frontal views, and the third and sixth columns are virtual frontal views. against the whole database. As we mentioned earlier, the poses of these faces are not consistent and only one unique rotation angle is chosen for all the images. Hence some of the synthesis results are good (the rst three columns in Fig. 4) if the actual rotation angle agrees with the preset value, and some are bad (the last three columns in Fig. 4). More image synthesis examples are given in Fig. 5.

5.3 Comparison of Recognition Results

To test and compare the ecacy of various methods for robust face recognition, we have tried the following methods: I. Bi-Linear Model [18], II. Nonlinear Learning method (Algorithm I), III. Subspace LDA [6] on the original images, and IV. Subspace LDA on the synthesized frontal images. As mentioned before, the database we used has 108 pairs of images, of which only about 42 pairs are good images in terms of the correctness of rotation angle we manually picked (refer Fig. 4). We use all the frontal

views as the database, and all the rotated images as the testing images. For methods I and II, we further divided the images into training set (74 pairs) and untrained testing set (34 rotated test images). And the performance scores on training set and testing set are listed separately in Table 1. We also report the recognition performances of subspace LDA on the original images (method III) and on the virtual frontal images (method IV) in Table 2. Method I II Score (%) 14.7/100 9.3/100

Table 1: Performance comparison of two learning based methods (I and II). There are two correct scores for each method (each column), where the score on the right (100% in both methods) is for the training set (74 images) and the score on the left is for the testing set (34 images). Method III IV Score (%) 39.8/61.9 46.3/66.7

Table 2: Performance comparison of subspace LDA on original images (III) and on virtual images (IV). The scores on the right are for the 42 good images, and the scores on the left are for all 108 images. Unlike in methods I and II, there is no training at all, all 108 testing images are untrained. Some conclusions can be drawn here: 1) the rst two learning methods are very poor at generalization though they have achieved perfect scores on the training set, and this is especially true when the rotation angles in the training set and testing set are not consistent; 2) using the virtual frontal views the performance of subspace LDA, which does not have the generalization problem and does not need retraining of subspace and LDA [6], can be improved, and the extent of the improvement depends on the quality of the virtual views.

6 Conclusion

We have proposed a SFS based method to handle the pose variation problem possibly coupled with the illumination problem for face recognition. We studied the 2D image transformation under a 3D rotation using a Lambertian re ectance model. Based on this study, we assessed the problem and reviewed several existing methods. Finally, we proposed a SFS based method to address the most dicult case of this problem with the aid of a generic 3D face model. In future we plan to improve our view synthesis result by deforming the 3D model or using multiple 3D face models as in [31] and test the proposed light source and pose estimation algorithm (21). Also larger and better database are needed to evaluate the ecacy of our method.

References

[1] R. Chellappa, C.L. Wilson, and S. Sirohey, \Human and Machine Recognition of Faces, A Survey," Proc. of the IEEE, Vol. 83, pp. 705-740, 1995. [2] P.J. Phillips, H. Moon, R. Rauss, and S.A. Rizvi, \The FERET Evaluation Methodology for FaceRecognition Algorithms," In Proc. Conference on Computer Vision and Pattern Recognition, San Juan, PR, pp. 137-143, 1997. [3] M. Turk and A. Pentland, \Eigenfaces for Recognition," Journal of Cognitive Neuroscience, Vol. 3, pp. 72-86, 1991. [4] D.L. Swets and J. Weng, \Using Discriminant Eigenfeatures for Image Retrieval," IEEE trans. on PAMI, Vol. 18, pp. 831-836, Aug. 1996. [5] P.N. Belhumeur, J.P. Hespanha and D.J. Kriegman, \Eigenfaces vs. Fisherfaces: Recognition Using Class Speci c Linear Projection," IEEE Trans. on PAMI, Vol. 19, No. 7, pp. 711-720, 1997. [6] W. Zhao,, R. Chellappa, and P.J. Phillips, \Subspace Linear Discriminant Analysis for Face Recognition," Center for Automation Research, University of Maryland, College Park, Technical Report CAR-TR-914, 1999. [7] G.G. Gordon, \Face Recognition Based on Depth Maps and Surface Curvature," SPIE Geometric Methods in Computer Vision, Vol. 1570, pp. 234247, 1991. [8] S. Akamatsu, T. Sasaki, H. Fukamachi, N. Masui, and Y. Suenaga, \An Accurate and Robust Face Identi cation Scheme," In Proc. International Conference on Pattern Recognition, pp. 217220, The Hague, The Netherlands, 1992. [9] D.J. Beymer, \Face Recognition Under Varying Pose," MIT Arti cial Intelligence Laboratory, Technical Report 1461, 1993. [10] R. Brunelli, \Estimation of Pose and Illuminant Direction for Face Processing," MIT Arti cial Intelligence Laboratory, Technical Report 1499, 1994. [11] D.J. Beymer, T. Poggio, \Face Recognition from One Example View,\ In Proc. 5th Int. Conf. on Computer Vision, pp. 500-507, 1995. [12] T. Cootes, C. Taylor, D. Cooper, and J. Graham, \Active Shape Models-Their Training and Application," Computer Vision and Image Understanding, Vol. 61, pp. 18-23, 1995. [13] A. Pentland, B. Moghaddam, T. Starner, \ViewBased and Modular Eigenspaces for Face Recognition," in Proc. International Conference on Computer Vision and Pattern Recognition, Seattle, WA, 1994.

[14] L. Wiskott, J.M. Fellous, N. Kruger, and C.v.d. Malsburg, \Face Recognition and Gender and Gender Determination," in Proc. of the International Workshop on Automatic Face and Gesture Recognition, Zurich, pp. 92-97, 1995. [15] T. Maurer and C.v.d. Malsburg, \Single-View Based Recognition of Faces Rotated in Depth," in Proc. of the International Workshop on Automatic Face and Gesture Recognition, VT, pp. 176181, 1996. [16] E. Sali and S. Ullman, \Recognizing Novel 3-D Objects Under New Illumination and viewing Position Using A Small Number of Example Views or Even A Single View," In Proc. Conference on Computer Vision and Pattern Recognition, Bombay, India, pp. 153-161, 1998. [17] S. Ullman and R. Basri, \Recognition by Linear Combinations of Models," In IEEE Trans. on PAMI, Vol. 13, pp. 992-1006, 1991. [18] W.T. Freeman and J.B. Tenenbaum, \Learing Bilinear Models For Two-Factor Problems in Vision," In Proc. Conference on Computer Vision and Pattern Recognition, San Juan, PR, pp. 554560, 1997. [19] T. Vetter and T. Poggio, \Linear Object Classes and Image Synthesis From a single Example Image," IEEE Trans. on PAMI, Vol. 19, No. 7, pp. 733-742, 1997. [20] P.N. Belhumeur and D.J. Kriegman, \What is the Set of Images of an Object Under All Possible Lighting Conditions?" In Proc. Conference on Computer Vision and Pattern Recognition, San Juan, PR, pp. 52-58, 1997. [21] A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman, \Illumination-Based Image Synthesis: Creating Novel Images of Human Faces Under Diering Pose and Lighting," In Proc. Workshop on Multi-View Modeling and Analysis of Visual Scenes pp. 47-54, 1999. [22] B.S. Manjunath, R. Chellappa and C.v.d. Malsburg, \A Feature Based Approach to Face Recognition," in Proc. of Computer Vision and Pattern Recognition, Urbana Champaign, Illinois, pp. 373378, 1992. [23] R. Alferez and Y.F Wang, \Geometric and Illumination Invariants for Object Recognition," IEEE Trans. on PAMI, Vol. 21, pp. 505-536, 1999. [24] N. Kruger, M. Potzsch, and C.v.d. Malsburg, \Determination of Face Position and Pose With A Learned Representation Based On Labelled Graphs," Image and Vision Computing, Vol. 15, pp.665-673, 1997. [25] G.G. Gordon, \3D Pose Estimation of the Face from Video," Face Recognition: From Theory to

Applications, Eds. Wechsler, H., Phillips, P.J.,

Bruce, V., Soulie, F.F., and Huang, T.S. Springer: Berlin, pp. 433-445, 1998. [26] D.B. Graham and N.M. Allison, \Characterising Virtual Eigensignatures for General Purpose Face Recognition," Face Recognition: From Theory to Applications, Eds. Wechsler, H., Phillips, P.J., Bruce, V., Soulie, F.F., and Huang, T.S. Springer: Berlin, pp. 446-456, 1998. [27] D. Terzopoulos and K. Waters, \Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models," IEEE PAMI vol. 15, no. 6, pp. 569-579 , 1993. [28] T. Akimoto, Y. Suenaga, and R.S. Wallace, \Automatic Creation of 3D Facial Models," IEEE Computer Graphics and Applications,, Vol. 13, pp. 16-22, 1993. [29] T. Riklin-Raviv and A. Shashua, \The Quotient Image: Class Based Re-rendering and Recognition With Varying Illuminations," in Proc. of Computer Vision and Pattern Recognition, Fort Collins, Colorado, pp. 566-571, 1999. [30] W. Zhao, and R. Chellappa, \Robust Face Recognition using Symmetric Shape-from-Shading," Center for Automation Research, University of Maryland, College Park, Technical Report CARTR-919, 1999. [31] J. Atick, P. Grin, and N. Redlich, \Statistical Approach to shape from shading: Reconstruction of three-dimensional face surfaces from sigle twodimensional images," Neural Computation, Vol. 8, pp. 1321-1340, 1996. [32] B. Scholkopf, A. Smola and K.R. Muller, \Nonlinear Component Analysis as a Kernel Eigenvalue Problem," Neural Computation, vol. 10, pp. 12991319, 1998.