Blind Image Restoration With Eigen-Face Subspace - IEEE Xplore

1766

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 11, NOVEMBER 2005

Blind Image Restoration With Eigen-Face Subspace Yehong Liao and Xueyin Lin

Abstract—Performance of conventional image restoration methods is sensitive to signal-to-noise ratios. For heavily blurred and noisy human facial images, information contained in the eigen-face subspace can be used to compensate for the lost details. The blurred image is decomposed into the eigen-face subspace and then restored with a regularized total constrained least square method. With Generalized cross-validation, a cost function is deduced to include two unknown parameters: the regularization factor and one parameter relevant to point spread function. It is shown that, in minimizing the cost function, the cost function dependence of any one unknown parameter can be separated from the other one, which means the cost function can be considered roughly, depending on single variable in an iterative algorithm. With realistic constraints on the regularized factor, a global minimum for the cost function is achieved to determine the unknown parameters. Experiments are presented to demonstrate the effectiveness and robustness of the new method. Index Terms—Face recognition, image restoration, least-squares methods, singular value decomposition (SVD).

I. INTRODUCTION

I

MAGES can be blurred by atmospheric turbulence, relative motion between sensors and objects, longer exposures, and so on, but the exact cause of blurring may be unknown. In most cases, a linear operator and an additive noise process can be used to modeled the degradation process of an image as (1)

Here, the imaging equation is written in matrix-vector form; represent the observed degraded image, original image, and additive noise, respectively. is the number of pixels in one image. The matrix represents a linear distortion operator in the discrete form of point spread function (PSF), which is described in detail later. There has been extensive work on blind image restorations [1]. Existing blind restoration methods can be categorized into two main classes: methods which separate blur identification as a disjoint procedure from restoration, such as zero sheet separation [2], generalized cross validation (GCV) [3], and maximum likelihood and expectation Manuscript received July 2, 2003; revised October 27, 2004. This work was supported in part by the National Natural Science Foundation of China (60433030) and the National Grand Fundamental Research 973 Program of China (2002CB312101). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Thierry Blu. Y. Liao was with the Computer Science and Technology Department, Tsinghua University, Key Laboratory of Pervasive Computing, Ministry of Education, Beijing 100084, China. He is now with the School of Nuclear Engineering, Purdue University, West Lafayette, IN 47907-2017 USA (e-mail: [email protected]). X. Lin is with the Computer Science and Technology Department, Tsinghua University, Key Laboratory of Pervasive Computings, Ministry of Education, Beijing 100084, China (e-mail: [email protected]). Digital Object Identifier 10.1109/TIP.2005.857274

maximization (ML-EM) [4] based on the ARMA image model; and methods which combine blur identification and restoration in one procedure, such as nonnegative and support constraints recursive inverse filtering (NAS-RIF) [5], maximum likelihood and conjugate gradient minimization (ML-CGM) [6], ML-EM [12], and simulated annealing (SA) [7]. The current method belongs to the former class. Of the above methods, for simplification some assume the PSF with arbitrary elements is positive, symmetric and its DFT is zero phase. The others assume the PSF has a known parametric form consisting of only a few parameters. The current method inspects PSF of Gaussian, motion, and out-of-focus blur with only one unknown parameter. Because of the presence of noise, resolving (1) is an ill-posed problem, which means that the solution may be nonexistent, not unique or unstable. Regularization is a general and very effective method for solving ill-posed problems. Its basic idea is to use regularization parameter to trade off fidelity to the observed data and smoothness of the solution. Regarding the regularized solution of (1), there has been a lot of research done. When in (1) is precisely known, the solution focused on finding a good approximation of regularized factor [8], [9], and then a conand ventional method was followed. When an approximate its variance are known, regularized methods suggested in [10] and [11] are successful, but there are always no such knowledge about and its variance beforehand. In the present method, the blurred image is decomposed into the eigen-face subspace in Section II and then restored with a regularized total constrained least-squares method in Section III. With GCV, a cost function is deduced in Section IV to include two unknown parameters: the regularized factor and one parameter relevant to PSF. In Sections V and VI, it is shown that in minimizing the cost function, the solution of any one unknown parameter is very insensitive to the other one, which means the cost function can be considered roughly depending on only one variable in either case. So, with an iterative methodology and realistic constraints on the regularized factor, a global minimum for the cost function can be achieved to determine the unknown parameters. Experiments are presented to demonstrate the effectiveness and robustness of the new method in Section VII. Finally, in Section VIII, the size dependence of the new method on principal components and facial image training set are discussed, and its application to facial images with expressions is presented. One of the important application fields of human facial image restoration arises in viewing poor-quality security images, taken at airports, transportation vehicles, and crime scenes. The current method is robust in dealing with heavily blurred human facial images.

1057-7149/$20.00 © 2005 IEEE

LIAO AND LIN: BLIND IMAGE RESTORATION WITH EIGEN-FACE SUBSPACE

II. EIGEN-FACE SUBSPACE Principal component analysis (PCA) is applied to model facial images. PCA is a statistical technique that is useful for dimensionality reduction. Given a set of training facial images , where is the number of training images, the mean face is the average of all training images

1767

contains all eigen-faces filtered by high pass regularization operator. For blind image restoration, (7) contains unknown parameters. Since a parametric model is used for PSF and PSF is a function of only one variable, depends only on two unknown parameters (8)

(2)

Let . The eigenvectors can be computed with sinof matrix gular value decomposition (SVD). Eigen-face subspace is constructed from principal components: , where is the th eigen-face vector and is the reduced dimension. The facial image is decomposed into the eigen-face subspace

(3) , and is a with vector in the eigen-face subspace, representing the image coordinate in that subspace. III. REGULARIZED IMAGE RESTORATION To obtain a reasonable estimate for the degraded image, the problem in (1) is reformulated as a regularized minimization problem (4) (5)

where is the unknown parameter of PSF. In the next section, a GCV cost function is deduced to reand are determined by minimization of the late to and GCV cost function. From (8), the restored image coordinate in the eigen-face subspace can be calculated and then the image is restored with (3). IV. GENERALIZED CROSS VALIDATION GCV is a widely recognized technique in the field of image restorations. In conventional image restorations, it has been used to optimize the regularization parameter [8]. In blind image restorations, it has been used in the ML method [18], and in identifying many variables including the ARMA model parameters, the blur and the regularization parameter [3]. The GCV technique was also applied in image resolution enhancement [13], where the blur parameter was determined by minimization the GCV cost function by assuming the regularization parameter. To proceed with the GCV technique, the relationship between the “restored image” and “observed image” is derived with the image coordinate in (8). Combining (3) and (8), we get (9)

Here, are the regularization cost function, regularization parameter, and regularization operator, respectively. A smoothness requirement on the solution can be imposed by requiring that be a high-pass filter. is the two-dimensional Laplacian operator [17], given by

To get a more compact representation of (9) is to let

(10) From (9) and (10), and some matrix manipulation, we obtain

From (3)–(5), the cost function is

(6) (11) If the cost function in (6) is minimized with respect to , and deHere, notes a diagonal matrix with vector ’s elements on the diagand onal. (7) Here, degraded by blur operator, and

contains all eigen-faces

(12)

1768


Since and are two-column vectors representing the “restored image” and “blurred image,” respectively, (11) can be rewritten as .. .

.. .

.. .

(13)

with dimension . Here, is the th row vector of According to the methodology of GCV, in the th step, a partial restoration process leaving out the th pixel is Fig. 1. GCV solution of with respect to .

.. .

.. .

.. . (14)

.. .

.. .

.. .

ill-posed or ill-conditioned problems. The purpose of regularization is to provide an analysis of an ill-posed problem through the analysis of an associated well-posed problem, whose answer will yield meaningful answers and approximation to the ill-posed problem [16]. Therefore, regarding to (11), a left limit constraint is enforced on

Let represents the “restoration image” in the th step, then the validation error by using the th pixel is (15) where is the th row vector of the blur operator matrix in (1). Consequently, unknown parameters and are the solution to

(16)

(18) Here, means the condition number, and is a constant greater than 1. On the other hand, for the two items of the nominator on stands for the the right side of (11), the first item acts as blurred noisy image, and the second item smoothing item. However, if is too large, the smoothing item may introduce extra noise in restored image. Therefore, a right limit constraint is enforced on (19)

It has been shown that the asymptotically optimum solution of (16) according to GCV is given by [14] and [15] Here,

is a constant less than 1.

(17) VI. IMPLEMENTATION The estimation problem with multiple parameters is more difficult than that of a single parameter problem with GCV. The space over which the algorithm must search is much larger, and the presence of local minima may be more problematic. Since images are modeled with the eigen-face subspace and the matrix B in (3) can be calculated before the restoration process, a GCV cost function is deduced with only two unknown parameters, one for the regularization parameter and the other for PSF. Therefore, it is easier to find the minimum of the GCV cost function in (17) than those with multiple parameters based on ARMA image models. V. REGULARIZED IMAGE RESTORATION The purpose to enforce constraints on the regularization parameter is to limit the searching space and assure a global minimum for cost function. In mathematical terms, the inverse problem presented in (1) is ill posed. The ill-posed nature of this problem implies that small bounded deviations in the data may lead to unbounded deviations in the solution. Regularization theory is often used to solve

As shown by our experiments, if is within the limits specified in Section V, the solution to (17) can be considered roughly as an optimization problem for only one unknown parameter , and the solution of is approximately independent of . Fig. 1 shows our experiment result. A 144*108 facial image is blurred by Gaussian PSF with 9*9 support and equal to 3, and contaminated by additive Gaussian white noise at level of 20-dB signal-to-noise ratio (SNR). Then, assuming lies in the limiting range , the blurred parameter is restored with the present method. From Fig. 1, we can see that solution of is close to its actual value and insensitive to . Thus, can be approximated in the th iterative step as (20) Similarly,

can be approximated in the th iterative step as (21)


1769

TABLE I MOTION BLUR IDENTIFICATION

Fig. 2.

Original image.

Starting with arbitrary value of within limits constrained by (18) and (19), the restoration process iterates alternatively between (20) and (21). If iterative error of exceeds predefined accuracy, or exceeds its constraints, iterative process terminates. For numerical stability, the element value of diagonal matrix in (12) is constrained between [0,1] in each iterative step.

TABLE II PILLBOX BLUR IDENTIFICATION

VII. EXPERIMENT RESULTS In the experiments, the eigen-face subspace is constructed by frontal facial images with the size of 144*108 pixels. The 1350 images from 1350 different people are selected as training data. The Sobel edge operator is used to detect the boundaries of eyes and mouths, and then the position of eyes and mouths is identified by a form-matched method [19]. Before SVD, each training image is aligned with three points: the centers of eyeballs are used to calibrate the orientation, the horizontal distance between two eyeballs is used to calibrate the horizontal scale, and the vertical distance between the mouth center and middle of two eyeballs is used to calibrate the vertical scale. After solving the PSF unknown parameter, the image is restored with (3) and (7). In this final step, more details is obtained in the restored image with more principal components included in matrix . 768 principal components are chosen to retain 99% of the total eigen-value energy, which is sufficiently accurate. The constants defining regularization parameter constraints in (18) is equal to 25 and in (19) is equal are chosen as: to 0.1.

TABLE III GAUSSIAN BLUR IDENTIFICATION

TABLE IV IMPROVEMENTS FOR MOTION BLUR RESTORATION

A. Blur Identification The original image (Fig. 2), which is excluded from training data, is blurred by three common types of PSF: Gaussian, moving average, and pillbox (out-of-focus) blur. Then, blurred images are added by noise at various levels of 5-, 10-, 20-, and 40-dB SNR, respectively. For moving average PSF, the blur model is assumed to be a horizontal uniform linear motion blur. In this blur model, only the length must be identified. varies from 3 to 27 with increment steps of three to simulate motion blur. For pillbox PSF, only the radius must be identified. varies from 1 to 9 with increment steps of 1 to simulate the effects of varying degree of out-of-focus blur. For 9*9 Gaussian PSF, only the standard deviation must be identified. varies from 0.5 to 3.5 with increment steps of 0.5 to simulate different Gaussian blur. The blur parameters of all degraded images stated above have been recovered satisfactorily. Tables I–III summarize part of the results.

B. Improvement of Image Restoration for Face Recognition In face recognition [20], the face image is projected into the eigen-face space as a vector in (3). The simplest classification method is based on the nearest distance criterion. represent the original image vector, and the deLet graded image vector in the eigen-face space, respectively. With each restored blur parameter in Tables I–III, a reis calculated with (8). Let stored eigen-face space vector , then d is the distance between the original image and the degraded image in the is the distance between the origeigen-face space, and inal image and the restored image. Given the image vectors and are normalized to 1 and the dimension of the for eigen-face space is 512; Tables IV–VI compare and each image restoration process in Tables I–III, respectively. The comparison shows all of image restoration processes have improvement over the degraded image for face recognition.

1770


TABLE V IMPROVEMENTS FOR PILLBOX BLUR RESTORATION

TABLE VI IMPROVEMENTS FOR GAUSSIAN BLUR RESTORATION

Fig. 4. Comparison of ML-CGM and our method for Gaussian blur restoration

TABLE VII COMPARISON OF ML-CGM AND OUR METHOD FOR PILLBOX BLUR

TABLE VIII COMPARISON OF ML-CGM AND OUR METHOD FOR GAUSSIAN BLUR

ratio (ISNR) given knowledge of original image (neither method assumes such knowledge). ISNR is defined as

Fig. 3. Comparison of ML-CGM and our method for pillbox blur restoration.

C. Comparison of Image Restoration Methods The performance of the current method has been compared to ML-CGM [6] for eight cases: The original image (Fig. 2) is blurred by pillbox PSF with radius equal to 3, and Gaussian PSF with 9*9 support and standard deviation equal to 2. Each of the two blurred image is added by various noise at level of 5-, 10-, 20-, and 40-dB SNR. Figs. 3 and 4 show the comparing results for restoration of images degraded by pillbox blur and Gaussian blur, respectively. In Figs. 3 and 4, from left to right, the first row are blurred images added by noise at various level of 5-, 10-, 20-, and 40-dB SNR, respectively; the second row is restoration results with ML-CGM method; the third row is restoration results with our method. Tables VII and VIII compare the method performance by using increased signal-to-noise

where and are the original and degraded image pixel intensity values and is the restored image pixel intensity value. VIII. DISCUSSION The above experiments show that the new method is superior to ML-CGM in terms of either ISNR or human perception. Furthermore, Tables VII and VIII show that the new method is robust in noise removal: The heavier the noise level, the larger ISNR the new method can achieve. It is evident that the source of information can be used in ML-CGM method is confined to only a blurred image itself. Thus, the more heavily the image is blurred, the more detailed information is lost and the less performance such a method can achieve. Therefore, their application is limited to “slightly” blurred and noisy images. However, in the new method the lost information can be retrieved from human facial image training set, expressed as an eigen-face model. Thus, in spite of much information being lost from original blurred images, complementary information


1771

TABLE IX EFFECTS OF PRINCIPAL COMPONENT DIMENSION ON RESTORATION

TABLE X EFFECTS OF TRAINING SET DIMENSION ON RESTORATION

Fig. 5. Image restoration sensitivity with the size of training set. (a) Original. (b) Blurred. (c) Size = 1024. (d) Size = 768. (e) Size = 512; Size = 256.

can be retrieved from eigen-face subspace by some residual clue remaining in blurred images. As a model-based methodology, several important issues of the new method are discussed in this section. The size of principal components involved in GCV optimization technical is discussed in Section VIII-A. The size of facial images raining set involved in eigen-face space construction is discussed in Section VIII-B. Finally, the application of our methodology to facial images with expression is presented in Section VIII-C. A. Dimension of Principal Components The advantage of image model based on eigen-face subspace has been testified by above experiments. However, for large systems, numerators and denominators in (20) or (21) are very expensive to evaluate. For a training set of 1350 facial images, we reduce the dimension of matrix B in (7) to 128 principal components, retaining 90% of the eigen-value total energy, which dramatically accelerates numerical calculation while has little effect on solution of unknown parameters. This conclusion is drawn from Table IX, which shows the effects of the dimension of principal component on restoration of unknown parameters. In this case, the facial image in Fig. 2 is degraded by pillbox blur with radius equal to 3- or 5- and 20-dB SNR noise. As the dimension of principal component increases, the solution of radius is almost not affected. B. Dimension of Facial Image Training Set Obviously, the larger the size of the image training set, the more detailed message contained in the eigen-face space, thus conserved. To test the sensitivity of the present method with the size of the training set, the dimension of the training set, i.e., the value of in (2), varies by 1024, 768, 512, and 256 images. The original image in Fig. 2 is degraded by pillbox PSF of radius 3- or 5- and 20-dB SNR additive noise. Then the pillbox PSF radius is restored according the iterative algorithm. As shown in Table X, the PSF restoration is not sensitive to the training set size. However, the quality of restored image decreases as

Fig. 6.

Restoration for images with various expressions.

the training set size decreases, as shown in Fig. 5, where the degraded image is blurred by a pillbox PSF of radius 3- and 20-dB SNR additive noise, because in the final restoration step using (3), the dimension decreases also. C. Application to Facial Image With Various Expressions For facial images with moderate various expressions, Fig. 6 shows the experiment result. In each row, from left to right,

1772


TABLE XI COMPARISON OF ML-CGM AND OUR METHOD FOR FACIAL IMAGE WITH VARIOUS EXPRESSIONS

there are images representing the original image with expression, blurred image, image restored by ML-CGM and image restored by our method, respectively. In Fig. 6, from top to bottom, the various expressions are anger, fear, sadness, happiness, and surprise, respectively. The original images with moderate expressions come from Japanese female facial expression (JAFFE) Database. All blurred images are blurred by 9*9 windowed Gaussian PSF with standard deviation of 3, added by noise level of 20-dB SNR. Restoration results with both ML-CGM and the new method are compared in Table XI. However, since our method is model based, its flexibility is restricted to frontal facial images. For images that cannot be discomposed into eigen-face subspace, the assumption underlying our method is violated. Fortunately, due to high correlation in human facial images, most ordinary calibrated facial images can be expressed in eigen-face subspace with tolerable error, facilitating our method’s wide application. REFERENCES [1] D. Kundur and D. Hatzinakos, “Blind image deconvolution,” IEEE Signal Process. Mag., vol. 13, no. 5, pp. 43–64, May 1996. [2] M. M. Chang, A. M. Tekalp, and A. T. Erdem, “Blur identification using the bi-spectrum,” IEEE Trans. Image Process., vol. 39, no. 10, pp. 2323–2325, Oct. 1991. [3] S. J. Reeves and R. M. Mersereau, “Blur identification by the method of generalized cross-validation,” IEEE Trans. Image Process., vol. 1, no. 7, pp. 301–311, Jul. 1992. [4] R. L. Lagendijk, J. Biemond, and B. E. Boekee, “Identification and restoration of noisy blurred images using the expectation-maximization algorithm,” IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 7, pp. 1180–1191, Jul. 1990. [5] D. Kundur and D. Hatzinakos, “A novel blind deconvolution scheme for image restoration using recursive filtering,” IEEE Trans. Signal Process., vol. 46, no. 2, pp. 375–390, Feb. 1998. [6] D. Biggs and M. Andrews, “Acceleration of iterative image restoration algorithms,” Appl. Opt., vol. 36, no. 8, pp. 1766–1775, Mar. 1997. [7] B. C. McCallum, “Blind deconvolution by simulated annealing,” Opt. Commun., vol. 75, pp. 101–105, 1990. [8] N. P. Galatsanos and A. K. Katsaggelos, “Methods for choosing the regularization parameter and estimating the noise variance in image restoration and their relation,” IEEE Trans. Image Process., vol. 1, no. 7, pp. 322–336, Jul. 1992.

[9] A. M. Thompson, J. C. Brown, and J. W. Kay, “A study of methods of choosing the smoothing parameter in image restoration by regularization.,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 4, pp. 326–339, Apr. 1991. [10] V. Z. Mesarovic, N. P. Galatsanos, and A. K. Katsaggelos, “Regularized constrained total least squares image restoration,” IEEE Trans. Image Process., vol. 4, no. 8, pp. 1096–1108, Aug. 1995. [11] W. Chen, M. Chen, and J. Zhou, “Adaptively regularized constrained total least squares image restoration,” IEEE Trans. Image Process., vol. 9, no. 4, pp. 588–596, Apr. 2000. [12] A. K. Katsaggelos and K. T. Lay, “Maximum likelihood blur identification and image restoration using the EM algorithm,” IEEE Trans. Signal Process., vol. 39, no. 3, pp. 729–733, Mar. 1991. [13] N. Nguyen and P. Milanfar, “Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement,” IEEE Trans. Image Process., vol. 10, no. 9, pp. 1299–1308, Sep. 2001. [14] G. Golub, M. Heath, and G. Wahba, “Generalized cross-validation as a method for choosing a good ridge parameter,” Technometrics, vol. 21, pp. 215–223, 1979. [15] N. Nguyen and P. Milanfar, “A computationally efficient superresolution image reconstruction algorithm,” IEEE Trans. Image Process., vol. 10, no. 4, pp. 573–583, Apr. 2001. [16] M. R. Banhan and A. K. Katsaggelos, “Digital image restoration,” IEEE Signal Process. Mag., vol. 14, no. 3, pp. 24–41, Mar. 1997. [17] B. R. Hunt, “The application of constrained least-squares estimation to image restoration by digital computer,” IEEE Trans. Comput., vol. COM-22, no. 9, pp. 805–812, Sep. 1973. [18] N. Fortier, G. Demoment, and Y. Goussard, “GCV and ML methods of determining parameters in image restoration by regularization: Fast computation in the spatial domain and experimental comparison,” J. Vis. Comm. Image Represen., vol. 4, pp. 157–170, 1993. [19] K. R. Castleman, Digital Image Processing. Englewood Cliffs, NJ: Prentice-Hall, 1996. [20] M. A. Turk and A. P. Pentland, “Eigenfaces for recognition,” J. Cogn. Neurosci., vol. 3, no. 1, pp. 71–86, Mar. 1991.

Yehong Liao was born in Huizhou, China, in 1974. He received the B.S. degree in nuclear engineering and the M.S. degree in computer science from Tsinghua University, Beijing, China, in 1997 and 2003, respectively. He is currently pursuing the Ph.D. degree in nuclear engineering at Purdue University, West Lafayette, IN. His research interests include image restoration and construction and flow visualization.

Xueyin Lin was born in Shanghai, China, in 1940. He received the B.S. degree in automatic control from Tsinghua University, Beijing, China, in 1962. He is a Professor with the Department of Computer Science and Technology, Tsinghua University. He joined Tsinghua University in 1962 as a faculty member in the Department of Automatic Control and then went to the Department of Electronics. He was a Visiting Professor with the University of Cincinnati, Cincinnati, OH, from 1983 to 1985 and from 1993 to 1994, respectively. His main research interests are in the areas of computer vision, image processing, and pattern recognition. He has published more than 100 papers in these areas.