SPARSE REPRESENTATION BASED BLIND IMAGE DEBLURRING

SPARSE REPRESENTATION BASED BLIND IMAGE DEBLURRING Haichao Zhang†‡ , Jianchao Yang‡ , Yanning Zhang† and Thomas S. Huang‡ †

School of Computer Science, Northwestern Polytechnical University, Xi’an, China 710129 ‡ Beckman Institute, University of Illinois at Urbana-Champaign, IL, USA 61801 ABSTRACT

We propose a sparse representation based blind image deblurring method. The proposed method exploits the sparsity property of natural images, by assuming that the patches from the natural images can be sparsely represented by an over-complete dictionary. By incorporating this prior into the deblurring process, we can effectively regularize the illposed inverse problem and alleviate the undesirable ring effect which is usually suffered by conventional deblurring methods. Experimental results compared with state-of-theart blind deblurring method demonstrate the effectiveness of the proposed method. Index Terms— blind image deblurring, deconvolution, sparse representation 1. INTRODUCTION Image deblurring is a widely existing problem in image formation process, due to the imperfection of the imaging devices and remains an active research area in image processing communities [1, 2, 3, 4]. Possible factors causing the blur are atmospheric turbulence (in astronomy), defocusing, as well as the relative motion between camera and the scene [1] [2]. If we assume that the blur is global and translation invariant, then the observation process can be modeled as: Yj = kj ∗ X  εj , j = 1 · · · J

(1)

where k is the Point Spread Function (PSF) or blur kernel and ∗ denotes the convolution operator.  denotes the noise generation operator which can be additive or multiplicative. The adoption of convolution model is not too restrictive in the sense that it can approximate many real world blurring process well while enable efficient computation in the Frequency domain. The task of deblurring is to estimate the sharp image X given blurry and noisy observation(s) {Y i }Jj=1 . It is called a multi-frame deblurring problem if J > 1 and a single image deblurring problem when J = 1. Image deblurring is a well-known ill-posed inverse problem, which requires regularization to alleviate its ill-posedness and stabilize the solution. A lot of regularization methods have been proposed in the literature, using linear or non-linear regularization factors.

Recently, the sparse property of natural images is explored extensively in many works [1, 2, 5, 6] and has been proven to be an effective prior for natural image restoration. Levin et al. proposed a non-blind deblurring model using natural image statics to alleviating the ringing artifacts in the deblurred image [5] and proposed an Iterative Reweighted Least Square (IRLS) method to solve the sparsity-constrained least square problem. This model is further generalized by Cho et al. [7] by adapting the sparseness according to the image content using a learned regression model. Cai et al. proposed a blind motion deblurring method by exploiting the sparseness of natural images in over-complete frames, such as curvelet, to help with kernel estimation and sharp image estimation [8]. We focus on sparse representation based blind image deblurring method in this paper by exploiting the sparsity property of natural images in terms of learned redundant and overcomplete dictionary. In our previous work, we proposed a sparse representation based method for non-blind image deblurring in [9], which is shown to generate more desirable results than conventional deburring methods. This method utilizes the sparse representation of image patches as a prior to regularize the ill-posed inverse problem. In this paper, we further develop a blind image deblurring method based on sparse representation, which is based on the non-blind deblurring work [9] and the super resolution method recently proposed in [10]. Based on compressive sensing theory, Yang et al. assumes that the same sparse representation coefficients are shared for the high resolution and low resolution patches with respect to a high-resolution dictionary and a corresponding low-resolution dictionary, respectively. This method has been shown to generate state-of-the-art results for image super resolution. For blind image deblurring, however, this method can not be applied directly, due the unknown blurring kernel (PSF), thus the construction of the coupled dictionary is not an easy task. Very recently, Hu et al. proposed to construct the blurry-sharp dictionary couple via the blurry image and deblurred image using the current estimation of the kernel [11]. However, as the deblurring procedure will usually introduce severe artifacts, the dictionary pair constructed via this method is not desirable for deblurring. We propose in this paper another approach for blind image deblurring using sparse representation, which is a natural generalization of [9]. The rest of this paper is organized as follows. In Sec-

tion 2, we review briefly some related work based on sparse representation. We present the sparse representation based blind deblurring method and develop an effective algorithm in Section 3. Experiments are conducted in Section 4 and we conclude this paper in Section 5.

operator (such as handcrafted derivative filters [5] or filters learned from training images [12]). When W is orthonormal, we have α = W  x as the transform coefficients, and thus we can rewrite (4) as: ˆ = arg min k ∗ Wα − Y2 + λα1 + γk2 . (5) {α ˆ , k} 2 2 α,k

2. RELATED WORKS Assuming additive white Gaussian noise (AWGN) and uniform convolutional blur model [1, 8], the low quality image observation process (1) for a single image can be reduced to: Y =k∗X+ε

(2)

The problem of blind deblurring is to estimate both the latent sharp image X and the blur kernel k from the blurry and noisy observation y. With more unknowns than knowns, this is a typical ill-posed inverse problem, and thus requiring regularization techniques taking into account prior information to stabilize the solution, which can be formally formulated as: ˆ = arg min k ∗ X − Y2 + λρ(X) + γφ(k) ˆ k} {X, 2 X,k

(3)

where ρ(X) is a regularization term on the desired sharp image, which promotes some desired properties of natural images and can be linear or non-linear [7, 12]. φ(k) is a regularization term on the possible blur kernels. Most of the current restoration methods can be casted into such a regularization framework, which is equivalent to the Bayesian method in the broad sense [2]. Designing proper priors for the image is crucial for obtaining better restoration results, which is still an active field. Many different prior models have been proposed in the past, For example, Roth et al. developed a framework termed as ‘Field of Experts’ (FoE) for learning sparsifying filters for natural images; Cho et al. proposed to adapt the regularization term based on sparsity prior to the image under consideration [7]. Most of these priors are related somehow with the sparse property of natural images, which has been explored extensively, especially in many recent restoration works [5, 8, 9, 10]. In the past several years, almost all of the many image deblurring algorithms proposed utilized the image sparsity prior. Fergus et al. [1] proposed a blind kernel estimation method using Ensemble Learning (Variational Bayes) in the sparse derivative domain, where images have a heavy-tailed distribution. Shan et al. [2] proposed a probabilistic model for motion deblurring by fitting a parametric model for the sparse derivative distribution of images. Krishnan et al. presented a fast non-blind deblurring method with a hyper-Laplacian prior for natural images [13]. With the sparsity prior as regularization, we can derive the following formulation: ˆ k} ˆ = arg min X ∗ k − Y2 + λW X1 + γk2 {X, 2 X,k

To achieve better sparsity for the representation α, W can be generalized to be non-orthogonal over-complete, by either combining different orthonormal basis or learning from the data. In this paper, we model W as a dictionary trained from natural image, via, e.g., the KSVD algorithm [14]. In the sequel, we will propose our method using sparse representation for blind image deblurring. 3. SPARSE REPRESENTATION REGULARIZED BLIND IMAGE DEBLURRING In the Sparseland model [15], images are often decomposed into small patches to better fit the sparse representation assumption [14]. We also follow this convention in our method. The proposed sparse representation regularized image deblurring method can be √ modeled √as follows. Given the blurry obN , and√an over-complete dictioservation Y of size N × √ nary D ∈ Rn×k , trained on n × n patches sampled from natural images1 , we want to estimate the latent sharp image X and the blur kernel k simultaneously by ˆ k} ˆ = arg min E(X, k), {X, X,k

(6)

where E(X, k) = k ∗ X − Y22 +

I 

ηi Ri X − Dαi 22

i=1

+

I 

λi αi 1 +

(7)

γk22 .

i=1

Ri is an n × N matrix that extracts the i-th block from the image and I is the total number of patches in the image. In (7), the first term is the reconstruction constraint, i.e., the restored image should be consistent with the observation with respect to the estimated degradation model; the second and third terms together enforce that the representation of the patches from the recovered image should be sparse with respect to a proper over-complete dictionary; the last term is a 2 -norm based regularization to stabilize the blur kernel estimation. The proposed model can be optimized efficiently with the recent progress of sparse optimization techniques. We em(4) ploy the Alternating Minimization scheme, which is widely adopted when dealing with multiple optimization variables.

where W is some sparse transformation (such as Wavelet, Curvelet and Shearlet, among others) or sparsity inducing

1 These patches can either sampled from a separate training set or from the blurry image itself, as in [9] [14].

Following this scheme, we address each of the optimization variable separately and present an overall efficient optimization algorithm. We first initialize the sparse representation {α ˆ i }Ii=1 as that recovered from Y with respect to D, and the −1   I ˆ = I R Ri R Dαi . latent sharp image X i=1

i

i=1

i

Algorithm 1 Sparse Representation Regularized Blind Image Deblurring. 1: Input: a blurry image Y, dictionary D. 2: Initialization: sparse vector set {α ˆ } Ii=1 recovered from Y in terms of D, iteration number T . 3: For t = 1, 2, · · · , T • Kernel Estimation: update kernel k by minimizing Eq.(8);

3.1. k-subproblem: Blur Kernel Estimation In this subproblem, we fix all other variables and and optimize the image blurring kernel k. The model (7) reduces to the following form: ˆ = arg min k ∗ X ˆ − Y22 + γk22 . k

This is a least square problem with Tikhonov regularization, which leads to a close-form solution for k: ˆ = F −1 k

ˆ ◦ F(Y) F(X) ˆ ◦ F(X) ˆ + γI F(X)



,

3.2. X-subproblem: Latent Image Updating ˆ and sparse representaGiven the current kernel estimation k tion {α ˆ }Ii=1 , we want to update the estimation for the latent sharp image X. The optimization problem (7) becomes X

I 

ηi Ri X − Dαi 22 .

(9)

i=1

This optimization problem can be solved efficiently with Fast Fourier Transform (FFT) as [2, 4]:  ˆ = F −1 X

I

 ˆ ◦ F(Y) + F( F(k) ˆ i=1 ηi Ri Dα)  I  ˆ ˆ F(k) ◦ F(k) + F( i=1 ηi Ri Ri I)

 .

In practice, we follow the multi-scale estimation scheme for stable estimations of the blurring kernel k and latent sharp image X [1, 2, 4]. Conventional schemes such as structure prediction can also be incorporated into optimization [4]. 3.3. {αi }Ii=1 -subproblem: Sparse Representation ˆ for the image, minimization of Given current estimation X model (7) reduces to the following problem: {αi } = arg min

I 

ˆ − Dαi 2 + ηi Ri X 2

4: 5:

where F (·) denotes Fast Fourier Transform (FFT), F −1 (·) denotes inverse FFT, F (·) denotes the complex conjugate of F (·), and “◦” denotes element-wise multiplication.

ˆ − Y22 + ˆ = arg min X ∗ k X

• Sparse Representation: recovering the sparse coefficients by minimizing Eq.(11);

(8)

k



• Image Estimation: update the latent image X estimation via minimizing Eq.(9);

I 

End ˆ and deblurred image Output: estimated blurring kernel k ˆ X.

The deblurred patch after sparse projection on to the learned over-completed dictionary D can be computed as Dˆ α i . The overall algorithm optimizes over blurring kernel k, latent sharp image X and sparse representation set {α i }Ii=1 alternatively. Algorithm 1 describes our sparse representation regularized blind restoration algorithm. Although we do not offer convergent analysis of the proposed method in the current paper, we empirically observe that this iterative procedure usually converges fast, typically within in 10 iterations. In the following experiment, we set the number of iterations as 10. 3.4. Dictionary Learning In our algorithm, we use a redundant and over-complete dictionary D trained on image patches to help exploit the sparsity prior of natural images. In this subsection, we mention briefly the learning process for obtaining such a dictionary. Given the patch set {pj }Jj=1 , we can train a dictionary adapted to natural images via [14]: ˆ D = arg min P − DZ2F + λZ 1 D,Z

(12)

where P = [p1 , p2 , · · · , pJ ] is the matrix of all the training patches. Z are the matrix of sparse codes for all the patches where each column corresponding to one patch. 4. EXPERIMENTS

λi αi 1 .

(10) We conduct several experiments to verify the effectiveness of the proposed method. We set η i = 0.05, λi = 0.1ηi and γ = 5 in all our experiments. In the following, we first conIt is easy to see that (10) is decomposable over each patch duct experiments under noise-free condition and then test the Ri X, thus (10) can be solved by solving the problem of robustness of the proposed method under noisy setting. sparse recovery for each image patch separately: We first examine the performance of the proposed method ˆ 2 + λi αi 1 . α ˆ i = arg min ηi Dαi − Ri X (11) 2 under noise-free observation condition. In this experiment, αi {αi }

i=1

i=1

Table 1. Deblurring result comparison on estimation quality under different blur kernels (no noise). Methods fast deblur [4] proposed cameraman 30.03 18.22 Motion babara 15.74 9.71 boat 18.38 12.58 house 16.34 9.14 cameraman 30.29 17.88 Gaussian babara 16.57 12.92 boat 17.21 12.44 house 17.59 10.22 cameraman 32.14 21.54 CamShake babara 22.36 12.91 boat 27.25 19.83 house 20.92 13.76

Table 2. Deblurring result comparison on estimation quality under different blur kernels (additive Gaussian noise). Methods fast deblur [4] proposed cameraman 33.08 18.41 Motion barbara 22.34 11.05 boat 24.73 12.06 house 23.40 9.68 cameraman 31.85 21.41 Gaussian barbara 21.60 13.08 boat 21.28 13.91 house 21.57 10.44 cameraman 33.68 18.10 CamShake barbara 22.36 15.35 boat 30.94 11.47 house 26.00 15.04

we first generate blurry observations using different test images. Several different blur kernels are used in experimentations, including motion blur (direction 45 ◦ with motion length 5 pixels), Gaussian blur (standard derivation σ = 2 pixels) and a general camera shake blur as shown in Figure 1. We run our proposed method with 10 iterations and compared our delburring result with that of the fast deblurring method from [4], which is one of the state-of-the-art blind deblurring method. The experimental results in terms of Root Mean Square Error (RMSE) under this setting are presented in Table 1. As can be seen from Table 1, the proposed method performs better than the fast deblurring method in terms of RMSE under different kinds of blur kernels. The deblurring results for ‘cameraman’ test image as well as the estimated blurring kernel are shown in Figure 1 (TOP). As we can see Figure 1 (TOP), the deblurred image using the fast deblurring method has severe ‘ghost’ effet. The deblurring result using the proposed method has less artifacts due to the incorporation of the sparse representation, which is visually more appealing and agrees with the objective results in Table 1. To further evaluate the performance of the proposed method, we carry out experiments on blurry and noisy images in the following. In this experiment, we test the algorithms using blurry and noisy images generated with different blur kernel and additive Gaussian noise with standard deviation of 0.01.2 The RMSE results under this noisy setting are summarized in Table 2. As can be seen from Table 2, the proposed method again outperforms the fast deblurring method on different test images. The noisy and blurry observation used in this experiment under different blur kernels are shown in Figure 1 (2nd row to bottom). As can be seen, the deblurring results from the fast deblurring method suffered from amplified noise, implying its non-robustness to noise, which is an inevitable factor in real-world imaging process. The proposed

method, on the other hand, is robust to noise due to the sparse representation regularization, therefore, the deblurring results from the proposed method can recover details without noise amplification. The RMSE plots for estimated blurring kernel and the deblurred image for the blurry ’barbara’ image under motion blur and Gaussian noise are shown in Figure 2. As can be seen from Figure 2, the estimation errors for both the blurring kernel and the deblurred image are decreasing with increasing iterations, which empirically justifies the effectiveness of the proposed minimization scheme.

2 The

range of pixel value is [0, 1].

5. CONCLUSION An effective sparse representation based blind image deblurring method is presented in this paper. The proposed method exploits the sparsity prior of natural images to help alleviating the ill-posed inverse blind deblurring problem. Due to the incorporation of this sparsity regularization, the deblurred image suffers less from the undesirable ringing artifacts as well as noise amplifications. Experimental results under different observation processes demonstrate that the proposed method can generate desirable deblurring results. For future work, we would like to further explore sparsity regularization for deblurring and generalize the proposed method to video deblurring. Moreover, generalizing the current method to nonuniform and spatially variant blind deblurring [16, 17] is also an interesting research direction. Acknowledgement This work is supported by the U.S. Army Research Laboratory and U.S. Army Research Office under grant number W911NF-09-1-0383. This work is also supported by National Natural Science Foundation (No.60872145, No.60903126), Cultivation Fund from Ministry of Education (No.708085), National High Technology Program (No.2009AA01Z315) and Postdoctoral (Special) Science Foundation (No.20090451397, No.201003685)

(a)

(b)

(c)

(d)

Fig. 1. Deblurring results with additive Gaussian noise. (a) ground truth image and kernel, (b) blurry (and noisy) image, (c) deblurring results from [4], and (d) deblurring results from the proposed method.

Kernel Estimation Error

Image Estimation Error

0.75

25

0.7 RMSE

RMSE

20 0.65

15 0.6

0.55

1

2

3

4 5 6 7 Iterations

8

9 10

(a)

10

1

2

3

4 5 6 7 Iterations

8

9 10

(b)

Fig. 2. RMSE plots from the proposed algorithm: (a) Kernel estimation error plot. (b) Image estimation error plot. of China. 6. REFERENCES [1] Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, and William T. Freeman, “Removing camera shake from a single photograph,” in SIGGRAPH, 2006. [2] Qi Shan, Jiaya Jia, and Aseem Agarwala, “High-quality motion deblurring from a single image,” in SIGGRAPH, 2008. [3] A. Levin, Y. Weiss, F. Durand, and W.T. Freeman, “Understanding and evaluating blind deconvolution algorithms,” CVPR, vol. 0, pp. 1964–1971, 2009. [4] Sunghyun Cho and Seungyong Lee, “Fast motion deblurring,” in SIGGRAPH ASIA, 2009.

[10] Jianchao Yang, John Wright, Thomas Huang, and Yi Ma, “Image super resolution as sparse representation of raw image patches,” in CVPR, 2008. [11] Zhe Hu, Jia-Bin Huang, and Ming-Hsuan Yang, “Single image deblurring with adaptive dictionary learning,” in ICIP, 2010. [12] Stefan Roth and Michael J. Black, “Fields of experts: A framework for learning image priors,” in CVPR, 2005. [13] Dilip Krishnan and Rob Fergus, “Fast image deconvolution using hyper-laplacian,” in NIPS, 2009. [14] Michael Elad and Michal Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE TIP, vol. 15, no. 12, pp. 3736–3745, 2006.

[5] A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Deconvolution using natural image priors,” 2007.

[15] Alfred M. Bruckstein, David L. Donoho, and Michael Elad, “From sparse solutions of systems of equations to sparse modeling of signals and images,” 2009.

[6] Michael Elad, Mario A. T. Figueiredo, and Yi Ma, “On the role of sparse and redundant representations in image processing,” Proc. of IEEE, vol. 98, no. 6, pp. 972– 982, 2010.

[16] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, “Nonuniform deblurring for shaken images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2010.

[7] Taeg Sang Cho, Neel Joshi, C. Lawrence Zitnick, Sing Bing Kang, Rick Szeliski, and William T. Freeman, “A content-aware image prior,” in CVPR, 2010.

[17] Stefan Harmeling, Michael Hirsch, and Bernhard Sch¨ olkopf, “Space-variant single-image blind deconvolution for removing camera shake,” in NIPS, 2010.

[8] Jian-Feng Cai, Hui Ji, Chaoqiang Liu, and Zuowei Shen, “Blind motion deblurring from a single image using sparse approximation,” in CVPR, 2009. [9] Haichao Zhang and Yanning Zhang, “Sparse representation based iterative incremental image deblurring,” in ICIP, 2009.