Support driven wavelet frame-based image deblurring

Information Sciences 479 (2019) 250–269

Contents lists available at ScienceDirect

Information Sciences journal homepage: www.elsevier.com/locate/ins

Support driven wavelet frame-based image deblurring Liangtian He a,b,∗, Yilun Wang b,c,d, Zhaoyin Xiang b a

School of Mathematical Sciences, Anhui University, Hefei, Anhui 230601, PR China School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, PR China PrinceTechs LLC., Shenzhen, Guangdong 518101, PR China d Beijing institute of Big Data Research, Beijing 100871, PR China b c

a r t i c l e

i n f o

Article history: Received 13 December 2016 Revised 4 December 2018 Accepted 5 December 2018 Available online 5 December 2018 Keywords: Wavelet frame Image deblurring Support estimation Truncated 0 regularization

a b s t r a c t Wavelet frames have been widely applied in the field of image processing, due to their good capability for sparsely representing the piece-wise smooth functions which are suitable for describing natural images. In this paper, we propose a novel and efficient wavelet frame based sparse recovery model denoted as Support Driven Sparse Regularization (SDSR) for image deblurring. The partial support information of the wavelet frame coefficients is first attained via a self-learning strategy applied on a reference image, and this support prior is then exploited via a proposed truncated 0 regularization term. Moreover, existing off-the-shelf deblurring methods can be easily incorporated into the open interface of our flexible algorithmic framework, by providing the initial reference image for support detection. In the experiments, we compare our method with several state-of-the-art deblurring approaches. The results demonstrate the effectiveness of the proposed method in terms of PSNR and SSIM values. © 2018 Elsevier Inc. All rights reserved.

1. Introduction Image deblurring is a research topic of long-term interest in the fields of image processing and computer vision. Its primary purpose is to recover the ground truth image from a blurry and noisy observation. The image deblurring task can be mathematically modeled as the following linear inverse problem:

f = Au¯ +  ,

(1.1) 2 Rn

2 Rn

where A is a blurring matrix, images are assumed to be composed of n × n pixels, u¯ ∈ and f ∈ are lexicographically 2 stacked representations of the original clear image and the corrupted image, respectively, and  ∈ Rn denotes additive white Gaussian noise with a variance of σ 2 . Image deblurring is an ill-posed inverse linear problem, and so a naive implementation such as a direct pseudo-inversion of A will lead to an unsatisfactory output with amplified noise and smeared edges. Therefore, regularization is naturally adopted to introduce prior information of the latent image in order to achieve a robust recovery. The regularized image recovery model is often formulated into a variational formulation. As such, variational approaches [32,36,40] have been extensively studied. Besides variational approaches, varied framelets especially wavelet tight frames have also played an active role in designing effective image recovery models over the past decade [4–12,15,16,27,28,31,33,34]. The key idea of image recovery ∗

Corresponding author at: School of Mathematical Sciences, Anhui University, Hefei, Anhui 230601, PR China. E-mail addresses: [email protected], [email protected] (L. He), [email protected] (Z. Xiang).

https://doi.org/10.1016/j.ins.2018.12.005 0020-0255/© 2018 Elsevier Inc. All rights reserved.

L. He, Y. Wang and Z. Xiang / Information Sciences 479 (2019) 250–269

251

Fig. 1. Summary of the proposed method. Given a noisy and blurry image, we obtained an initial recovered image via an off-the-shelf method, e.g., the IDD-BM3D method. Then, we extracted the estimated support information from this recovered result and developed a truncated 0 regularization model based on wavelet frames. We solved this resulting optimization model and obtained a new recovered image. As such, our method is a multi-stage process that alternately implements support information detection and image recovery sub-processes.

models based on wavelet frames is that the underlying image is compressible in the transform domain. Therefore, the latent true image can be approximately obtained by solving an optimization problem to find the most sparse solution. The deep connections of image recovery methods based on wavelet frames with variational approaches have been investigated [5,9,11,13,24], and these studies have revealed why models based on wavelet frames often provide better image recovery performance than corresponding traditional variational approaches such as total variation based image recovery models. Most existing framelet-based regularization models have adopted the classical 1 norm, p (0 < p < 1), or 0 quasi-norm as the sparsity prior in their respective objective functions. The adoption of sparsity-promoting prior regularization is so widespread that it raises some questions, such as whether big room remains for further improvement of recovery performance, and what might be the most advantageous direction for future efforts to this end. For example, 1 norm might lead to a computationally efficient convex problem, but the solution is often suboptimal. p (0 ≤ p < 1) quasi-norm based model have a better solution in theory, but the optimal solution is hard to achieve due to the non-convexity of the corresponding models in practice. Along the way of further improving wavelet frame based sparsity regularized image deblurring methods, the exploitation of a greater number of priors of the latent image is well known to be essential. Accordingly, we seek a concrete way about how to effectively incorporate more priors into the sparsity regularization terms, such as the locations of nonzero wavelet frame coefficients1 . in the transform domain. Specifically, we proposed an efficient Support Driven Sparse Regularization (SDSR) model to incorporate the location information of large magnitude non-zero wavelet coefficients. The proposed SDSR algorithm, whose pipeline is illustrated in Fig. 1, consists of a multistage iterative procedure. First of all, the partial support information of the wavelet frame coefficients is obtained based on an initial reference image which can be a solution of any off-the-shelf deblurring method. This support information is then used to produce a truncated 0 regularization model, where the detected nonzero components are removed out of the 0 quasi-norm. The solution of this new model is then used as a new reference image for obtaining updated and more accurate partial support information that will be used to produce a new truncated 0 regularization model. This iterative process proceeds until a prescribed stopping criterion is satisfied. We would like to provide a preliminary illustration of the potential advantages of using support information of wavelet frame coefficients in image deblurring, because the key component of the proposed method is the estimation and adoption of support information, which will largely determine the final recovery performance. Fig. 2 presents some image recovery results of the proposed truncated 0 regularization model based on an exploitation of support information under the ORACLE case, i.e. the support information is obtained from the ground truth image. While it is infeasible to know the ground truth image beforehand in practice, the highly impressive performance demonstrates the potential benefits from making use of support information. Therefore, it is desirable that the detected support indices of the wavelet frame coefficients are close to the underlying true support set in practice. For this purpose, we expect to have a relatively high quality initial reference image from which to derive the support information. While this is a requirement, it is also an advantage of our algorithm on the other side. In particular, the proposed method can naturally adopt the result of almost any existing off-the-shelf deblurring algorithm as an initial reference image. In other words, we can make use of existing state-of-the-art deblurring results, e.g., the IDD-BM3D method [20] as a starting point for our proposed iterative algorithm. Overall, the main contributions of this paper are summarized as follows. • In contrast to most existing framelet-based regularization models, the SDSR model exploits the partial support information of the wavelet frame coefficients as a prior. It is the first time that a truncated 0 regularization model based on the self-learning of partial support information has been proposed. • While support information has been exploited to improve sparse recovery performance in the past studies, most of these efforts have assumed that the support information is available in advance, and they have been generally studied in the context of the specific compressive sensing field [26,35,37,39]. The SDSR model is one of the few to exploit support information in a self-learning procedure and it is applied to image deblurring instead of compressive sensing.

1 Please note that this is just a cursory definition. For an exactly sparse signal, the support prior is defined as the locations of its nonzero entries. However, in practice, the transformed coefficients of a natural image under a given wavelet frame are not exactly sparse, and are commonly compressible. The formal definition of the support prior in this case is included in Section 4

252

L. He, Y. Wang and Z. Xiang / Information Sciences 479 (2019) 250–269

Fig. 2. (a) blurred image, (b) recovered image of the classical 0 regularization model, (c) recovered image of the proposed truncated 0 regularization model.

• The SDSR algorithm is a self-contained iterative framework with an open interface that can seamlessly incorporate existing state-of-the-art image restoration methods. The detection, or learning, and exploitation of support information is a general idea, and can be readily incorporated into many existing sparsity-driven recovery methods. Therefore, this paper is expected to provide new insights into the development of other support driven sparsity-promoting regularization methods for more image restoration tasks. The remainder of this paper is organized as follows. In Section 2, preliminary information regarding wavelet tight frames is briefly introduced. In Section 3, we review several typical wavelet frame-based and nonlocal patch-based image restoration methods for subsequent comparison. Section 4 introduces the proposed SDSR model and summarizes the algorithmic framework. Section 5 presents the results of extensive experiments to demonstrate the effectiveness of the proposed SDSR model compared to several state-of-the-art alternatives. Section 6 presents concluding remarks, and also discusses some possible future research directions. 2. Notations and preliminaries Some preliminaries of the wavelet tight frames are introduced here, and interested readers are refereed to [7,17,34] for more details. Given a finite set of generators  = {1 , 2 , . . . , r } ⊂ L2 (R ), the wavelet frames for L2 (R ) can be generated by the shifts and dilations of them as follows.

X ( ) = {l, j,k , 1 ≤ l ≤ r, j ∈ Z, k ∈ Z},

(2.2)

where l, j,k = 2 j/2 l (2 j · −k ). If the following result holds:

f =



< f, ψ > ψ , ∀ f ∈ L2 (R ),

(2.3)

ψ ∈

the set X( ) is a tight frame of L2 (R ). According to the unitary extension principle (UEP) [5], given the 1D tight wavelet frame, the construction of the framelets for L2 (R2 ) can be obtained by the tensor products of 1D framelets.

L. He, Y. Wang and Z. Xiang / Information Sciences 479 (2019) 250–269

253

Table 1 The notations and representations. Notation

Representation

< α, β > A AT I W WT Wl, i, j u |Wl, i, j u| ||a||1, 1 ||a||1, 2 ||a||2 ||a||0 ||a||∞ O2 || ||0 O S T

Inner product of vectors α and β Blurring matrix associated with a known point spread function The transpose matrix of A Identity matrix framelet decomposition matrix framelet reconstruction matrix The frame coefficient of u correspond to pixel i in bands j at level l Absolute value (magnitude) of Wl, i, j u Anisotropic 1 norm of vector a Isotropic 1 norm of vector a Euclid norm of vector a Number of nonzero entries of vector a The largest absolute value of vector a n × n Cartesian grid. Cardinality of image singularities set in the spatial domain The feasible set for that imposes additional geometric constraints on Detected support index set of frame coefficients in the transform domain Supplementary set of the detected support set S Elements of α indexed in T Regularization parameters Penalty parameters Threshold parameter

αT λ, ν μ, γ ρ

2

In the discrete setting, we use the transform matrix W ∈ Rm×n (m ≥ n2 ) and WT to denote the framelet decomposition and framelet reconstruction, respectively. Then, the L-level framelet decomposition of the image u is denoted as:



W u = (. . . , Wl,i, j u, . . . )



for 0 ≤ l ≤ L − 1, i ∈ 1, 2, 3, . . . , n2 , j ∈ I,

(2.4)

where I represents the index set of all framelet bands. Wu is a 3-mode tensor, and Wl, i, j u is the frame coefficient of u corresponding to the pixel i in bands j at level l. Furthermore, to avoid confusion to the notations appearing in this paper, we give an elaboration of them in Table 1. 3. Related work The proposed truncated 0 regularization model can be considered a generalization of the traditional 0 regularization model, where the set of truncated components out of the 0 quasi-norm is empty as a special case. Therefore, we briefly revisit the wavelet frame-based 0 regularization image recovery model [16]. As similar counterparts, the cosparse analysis model [30], classical 1 norm regularization model [4], and piecewise smooth framelet-based image recovery models [9,27] are also reviewed. We also review some representative nonlocal patched algorithms, since the SDSR algorithmic framework employs a state-of-the-art nonlocal patched image restoration method to generate the initial reference image for support detection. 3.1. Wavelet frame-based 1 norm regularization model The fact that wavelet frames do not have an orthogonal property, i.e., WWT = I, has led to the development of various framelet-based models in the literature, including the synthesis model, the analysis model, and the balanced model. These models mostly penalize the 1 norm of the frame coefficients as the sparsity constraint in varied ways. A detailed description of these different models can be found at [33]. Here we mainly consider the analysis model (3.5), since numerical experiments [33] have demonstrated that the recovery performances of these models are approximately comparable in most cases.

min u

1 ||Au − f ||22 + ||λ · W u||1,p , 2

(3.5)

where p = 1 and p = 2 correspond to anisotropic 1 norm and isotropic 1 norm, respectively. The generalized 1 -norm is defined in a unified way as below.

⎡

||λ · W u||1,p =

 i

⎣

 1p ⎤



  l

λl,i, j |Wl,i, j | p

⎦.

(3.6)

j

If we introduce an auxiliary variable α = W u, (3.5) can be rewritten in the following equivalent form.

min u,α

1 ||Au − f ||22 + ||λ · α||1,p 2

s.t.

α = W u.

(3.7)

254

L. He, Y. Wang and Z. Xiang / Information Sciences 479 (2019) 250–269

Note that (3.7) is a convex optimization problem that can be solved using many efficient algorithms, e.g., the Split Bregman [3] or alternating direction method of multipliers (ADMM) [25]. 3.2. Wavelet frame-based 0 quasi-norm regularization model It is well known that the 1 norm regularization model obtains the sparsest solution only when the operator A satisfies particular conditions [14,18]. However, these conditions are not necessarily satisfied for image deblurring tasks. Therefore, the 1 norm regularization deblurring models often only achieve suboptimal performance. Recently, the p quasi-norm (0 ≤ p < 1) regularization model has attracted much attention, since it has a better theoretical property in terms of leading to more sparse solutions than 1 norm regularization. Particularly, Zhang et.al [41] proposed to penalize the 0 quasi-norm instead of the 1 norm of the frame coefficients.

min u

1 ||Au − f ||22 + λ||W u||0 , 2

(3.8)

where ||α ||0 represents the number of nonzero entries of α . Accordingly, the 0 minimization problem (3.8) was solved using the penalty decomposition (PD) algorithm [41]. More recently, Dong et al. [16] has developed a more efficient algorithm denoted as the mean doubly augmented Lagrangian (MDAL) algorithm for solving this same optimization problem. Note that compared with 1 norm regularization, the p quasi-norm (0 ≤ p < 1) regularization model often leads to a non-convex optimization problem, which is relatively harder to analyze and compute, than its convex alternatives. 3.3. Piecewise smooth wavelet frame-based model Let us denote a set of indices of an n × n Cartesian grid by O2 := {1, 2, 3, . . . , n}2 , and let ⊂ O2 be a set of image singularities. Correspondingly, the piecewise smooth framelet-based model proposed by Cai et.al [9] is formulated into an optimization problems as below.

min

u, ⊂O2

1 2

||λ · W c u||22 + ||ν · W u||1,2 + ||Au − f ||22 ,

where

 

||λ · W c u||22 =

i∈O 2 \

and

l

⎡ ||ν · W u||1,2 =



 2 λl,i, j Wl,i, j u ,

(3.10)

j

 12 ⎤



 

⎣

i∈

(3.9)

l

⎦.

νl,i, j |Wl,i, j |2

(3.11)

j

The optimization problem (3.9) was solved using an alternative scheme [9] as below. Specifically, given k , uk+1 is computed by:

min ||λ · W( k )c u||22 + ||ν · W k u||1,2 | + u

Given uk+1 , k+1 is estimated by:

k+1 =

⎧ ⎨

i:

⎩



 

 12

νl,i, j |Wl,i, j uk+1 |2

j

l

1 ||Au − f ||22 . 2

≤

 l

j

⎫ ⎬   2 λl,i, j Wl,i, j uk+1  . ⎭

(3.12)

(3.13)

The piecewise smooth model in the work of Ji et.al [27] is rewritten as:

min ||λ · W c u||22 + u,

1 ||Au − f ||22 , 2

s.t.

|| ||0 ≤ t and ∈ O,

(3.14)

where || ||0 represents the cardinality of , and O represents the feasible set for that imposes additional geometric constraints on . Note that the major difference between (3.9) and (3.14) is that is enforced as a constraint in (3.14), while it is formulated as a penalty term in (3.9). 3.4. Cosparse analysis model The cosparse analysis model in the work of Nam et.al [30] is formulated as follows.

min ||Du||0 , u

s.t.

Au = f,

(3.15)

L. He, Y. Wang and Z. Xiang / Information Sciences 479 (2019) 250–269

255

where D denotes the difference operator. Accordingly, a greedy analysis pursuit (GAP) method [30] was proposed to solve the minimization problem (3.15), which iteratively updates u as below.

uk+1 = argminu ||(Du )ck ||22 ,

s.t.

Au = f,

(3.16)

where k denotes the support estimation of Du, i.e., the index set is composed of the K largest entries (in terms of absolute values) of Duk . Specifically, the GAP algorithm was applied to the problem of computed tomography (CT) image reconstruction based on compressive sensing. In the context of compressive sensing field, the cosparse analysis model was demonstrated to obtain significantly improved recovery performance relative to the standard 1 regularization model. 3.5. Nonlocal patch based methods The key assumption behind nonlocal patch image restoration approaches is that similar structures in the spatial domain (i.e., nonlocal similarities) are more clearly detectable in small image patches. Therefore, these approaches aim to exploit the redundant information in natural images to better recover the textures. These approaches begin with the nonlocal means method (NLM) developed by Buades et al. [2] for image denoising. Then, the idea of nonlocal similarity has also been extended to many other image restoration tasks (see [11,19,20,40]). More recently, it has been incorporated into adaptive dictionary learning approaches, and has generated a diversity of state-of-the-art image restoration methods [1,21–23,29,42]. 4. Support driven wavelet frame-based model Our proposed SDSR model is formulated as follows.

min u

1 ||Au − f ||22 + λ||(W u )T ||0 , 2

(4.17)

where (Wu)T is the truncated form of Wu, i.e. a subvector of Wu after truncation. It is this truncation that primarily distinguishes this model from the standard 0 regularization model. Let us denote the support index set of frame coefficients to be truncated as S, which is an empty set in the standard 0 regularization model. In addition, let us denote T as the supplementary set of S, i.e., T = SC . It is noteworthy that S and T are not available in advance because the ground truth image is unknown. Thus, the critical component for the practical performance of (4.17) lies in the estimation of S in the transform domain. To avoid confusion to the above notations, we provide a simple example to illustrate them. Assuming that we want to recover a underlying sparse vector a¯ = (0, 10, 0, 25, 20, 0 ) via (4.17), the true support index set of a¯ is S¯ = {2, 4, 5}. A self-learning strategy is employed to obtain the partial support information, for example, the detected partial support set might be S = {2, 4}, and T = {1, 3, 5, 6} correspondingly. Accordingly, the elements corresponding to T are retained in the regularizer term while the elements corresponding to S are truncated out of the 0 quasi-norm. Intuitively, this means that nonzero entries, particularly those with large absolute values, should not be forced toward 0 by penalization. We achieve this goal by removing them out of the regularizer term. In summary, the multi-stage SDSR process alternatively implements the following two sub-processes. • Sub-process 1: Given a reference image, partial support information S is extracted based on it in the transform domain. • Sub-process 2: The truncated 0 regularization model (4.17) is solved with T = SC fixed. The recovery result then serves as the reference image in the next Sub-process 1 stage. 4.1. Sub-process 1: Support detection to determine T In this work, we adopted a heuristic yet effective support information detection method that is similar to a previously proposed strategy [37]. More precisely, S is computed as the indices of the wavelet frame coefficients whose magnitudes are greater than a prescribed threshold. Specifically, in the q-th stage, the initial reference image (q = 1) or the intermediate recovery result u(q) (q ≥ 2) is used for estimating the indexes of the support set as follows.





S(q+1) := (l, i, j ) : |Wl,i, j u(q ) | >  (q ) .

(4.18)

Correspondingly, the indexes of the frame coefficients remained in the 0 quasi-norm regularization are: T (q+1 ) = (S(q+1 ) )C . The threshold value of each stage is set in a heuristic way as below.

 (q) := ||W u(q) ||∞ /ρ ,

(4.19)

where ρ > 0. Empirically, the final recovery performance is quite robust regarding the parameter ρ . Remark: At first sight, the proposed thresholding based support detection strategy may seem overly simple for providing adequate detection. However, we would like to point out that reliable support information of the frame coefficients can be still obtained because the support detection is applied to relatively high quality reference images. Moreover, we note that a perfect detection accuracy is not necessarily required, because a modest number of wrongly detected support indices will not significantly degrade the final recovery performance.

256

L. He, Y. Wang and Z. Xiang / Information Sciences 479 (2019) 250–269

4.2. Sub-process 2: Solving the truncated 0 regularization model with known T Given T, the MDAL algorithm [16] originally developed for the standard 0 regularization model optimization problem can be slightly modified here for solving the non-convex problem (4.17) with respect to u. First, we introduce an auxiliary variable α = W u into (4.17), which yields the following equivalent constraint optimization problem.

min u,α

1 ||Au − f ||22 + λ||αT ||0 , 2

s.t.

α = W u.

(4.20)

Then, the MDAL algorithm is formulated as follows.

⎧ k+1 u = arg minu 12 ||Au − f ||22 + μ2 ||W u − α k + bk ||22 + γ2 ||u − uk ||22 ⎪ ⎨ α k+1 = arg minα λ||αT ||0 + μ2 ||α − (W uk+1 + bk )||22 + γ2 ||α − α k ||22 ⎪ ⎩ k+1 b = bk + W uk+1 − α k+1 .

(4.21)

Compared with the alternative procedure to solve the standard 0 quasi-norm regularization model, the main modification of the MDAL algorithm lies in the subproblem calculating α k+1 . Specifically,

α k+1 = HT,λ,μ,γ (W uk+1 + bk , α k ),

(4.22)

where the above support guided selective hard-shrinkage operator H is defined as follows.



(HT,λ,μ,γ (x, y ))θ =

θ ∈ T and | μxμθ ++γγ yθ |