Adaptive Multiple-Frame Image Super- Resolution Based on U-Curve

Adaptive Multiple-Frame Image SuperResolution Based on U-Curve IEEE Transaction on Image Processing, Vol. 19, No. 12, 2010 Qiangqiang Yuan, Liangpei Zhang, Huanfeng Shen, and Pingxiang Li Presented by In-Yong Song

School of Electrical Engineering and Computer Science Kyungpook National Univ.

Abstract  Super-resolution

reconstruction

– Recovery of high-resolution image from low-resolution images • Noisy, blurred and down-sampled

– Use of maximum a posteriori model (MAP) • Generation of noise and blurry depending on parameter • Adaptively selecting optimal regularization parameter

 Proposed

method

– Adaptive MAP reconstruction method based upon Ucurve

2 / 34

Introduction  Previous

method

– Frequency domain method • Tsai and Hung – Representation of multiple-frame SR problem

• Kim et al – Improving Tsai and Huang’ method » Considering observation noise and spatial blurring

• Bose et al. – Recursive total least squares method

• Rhee and Kang – Discrete cosine transform (DCT) based method

• Wavelet transform-based SR methods 3 / 34

– Spatial domain method • Non-uniform interpolation approach • Iterative back projection (IBP) approach • Projection onto convex sets (POCS) approach • Deterministic regularized approach • Maximum likelihood (ML) approach • Maximum a posteriori (MAP)approach • Joint MAP approach • Hybrid approach

4 / 34

 Proposed

method

– Based on MAP reconstruction model • Regularization parameter – Important role – Controlling tradeoff between fidelity and prior item

• Disadvantage – Time-consuming and subjective

• Avoiding disadvantage – Use of adaptive selection method

5 / 34

– Adaptive selection methods • Using classical method – Bose et al » Use of L-curve method – Nguyen et al. » Use of cross validation method (GCV)

• Using general Bayesian framework – Kang and Katsaggelos, He and Kondi, Molina et al. and Zibetti et al.

6 / 34

• Advantage and Disadvantage of each method – L-curve and GCV approach » Providing good solution » High cost – Bayesian framework method » Lower computational load » Attaching parameter and parameter distribution function

– U-curve method • Proposed by Krawczy-Stando and rudnicki • Selecting regularization parameter in inverse problems

7 / 34

Observation Model  Description

of observation model

– Degradation process form HR image to LR image – Assuming HR image to be shifted, blurred, downsampled, and additive noise

Fig. 1. Degradation model of the HR image. yk  Dk Bk M k x  nk

(1)

where l1 and l2 be the down-sampled factors for rows and columns M k stands for the warp matrix, Bk is the blurring matrix (PSF) Dk is the down-sampling matrix, nk is the noise vector. 8 / 34

– Assuming down-sample factors and blurring function • Same between LR images ( Dk , Bk  D, B )

– Representation of whole observation model y1  DBM 1 x  n1  y2  DBM 2 x  n2    y  DBMx  n   y p  DBM p x  n p 

(2)

where y  [ y1 , y2  y p ]T , M  [ M 1 , M 2  M p ]T and n  [n1 , n2  n p ]T .

9 / 34

MAP Reconstruction Model  Formulation

of MAP Reconstruction model

– Adding some prior information about HR image to regularize SR problem – Formulation of reconstruction function using MAP model and how to solve – Presentation of estimated HR image x  arg max  p ( x | y ) ^

(3)

– Using Bayes rule  p ( y | x) p ( x)  ^ x  arg max   p( y )  

(4)

10 / 34

– Independent of

p( y )

x  arg max  p ( y1  y p | x) p ( x)

(5)

^

where p( y1  y p | x) is the likelihood distribution of the LR images, and p( x) is the prior distribution of the HR images.

– Minus log of the functional in (8) x  arg min (log p( y  y | x)  log p ( x)) ^

1

(6)

p

– Zero-mean Gaussian noise and independent LR frame  1 p( y1  y p | x)    2 k 

p

 p yk  DBk M k x    exp   2   k 1  2 k  

2

   

(7)

11 / 34

– Use of Prior model in this paper p( x) 

 1 1 2 exp  Qx  C   

(8)

where  is a parameter that controls the variance of the prior distribution and Q represents a linear high-pass operation that penalizes the estimation that is not smooth.

– Substituting (7) and (8) in (6) • Cost function



x  arg min J ( x)  arg min y  DBMx   Qx ^

2

2



(9)

12 / 34

 Optimization

– Differentiating (9) with respect to x J ( x)  2 M T BT DT ( y  DBMx)  2 QT Qx  0

(10)

– Successive approximation iteration x n1  x n   n r n

(11)

1 r n   J ( x)  M T BT DT ( y  DBMx)   QT Qx 2

(12)

 

(r n )T r n

n

2

DBMr n

  Qr n

2

(13)

– Termination condition of iteration x n1  x n x

n 2

2

d

(14)

13 / 34

U-Curve Method  Description

of novel approach

– Estimating regularization parameter based on Ucurve • First, inducing principle of U-curve and its properties • Second, Use of u-curve method in SR problem

 U-Curve

and Its Properties

– Expression similar to traditional Tikhonov regularization model • Writing traditional model of Tikhonov regularization – Cost function in (9)



x  arg min y  Ax   Qx ^

2

2



(15)

14 / 34

– Using singular value decomposition (SVD) least squares method  0  A U  V g  U y T

0 0 r  gv ^ xa   2 i i i 2 i 1 ( i   qi )

(16)

– From functions (15) and (16)  2 qi4 gi2 R( )  y  Ax   2 2 2 i 1 ( i   qi )

(17)

 2 qi2 gi2 P( )  Qx   2 2 2 i 1 ( i   qi )

(18)

r

2

2

r

15 / 34

– From Previous discussion U ( ) 

1 1  R( ) P( )

(19)

Fig. 2. Typical U-curve.

– Three characteristic parts • (a) it is monotonically decreasing on the left side • (b) it is monotonically increasing on the right side • (c) in the middle it is almost “horizontal” with monotonous change 16 / 34

– U-curve properties • a) for   (0,( r / q1 )(4/3) ),the function U ( ) is strictly decreasing • b) for   (( 1 / qr )(4/3) , ), the function U ( ) is strictly increasing • c) limU ( ) 0   

• d) limU ( )    Remark: The optimal regularization parameter can be selected in the (4/3) (4/3) interval (( r / q1 ) ,( 1 / qr ) ) , and objective of the U-curve criterion for selecting the regularization parameter is to choose a parameter for which the curvature of the U-curve attains a local maximum close to the left part of the U-curve

17 / 34

 Selection

Steps

– Determining optimal regularization parameter  Singular value decomposition

Incorporate the result of Step 1

Plot the u-curve

Select the maximum curvature point

18 / 34

Experimental Results  Evaluation

of reconstruction results

– Use of mean square error (MSE) and structural similarity (SSIM) MSE 

SSIM 

1 x  x l1 N1  l2 N 2

2

(20)

(2u x u x  C1 )(2 x x  C2 ) (u  u  C1 )(    C2 ) 2 x

2 x

2 x

(21)

2 x

where x represents the original HR image, and x represents the reconstructed hr image.

– Comparing previous and proposed method

19 / 34



Simulated Data – Image of cameraman, boat, and castle – Classification of two cases • Degradation parameters known case • Degradation parameters unknown case

– Termination condition of iteration x n1  x n xn

2

2

 106

20 / 34

– 1) Degradation parameters known case • Case 1 – PSF of 3x3 window size » 0.5 variance – Down-sampled factor 2 – Zero-mean Gaussian-noise » 0.01 variance

• Case 2 – PSF of 5x5 window size » 1.0 variance – Down-sampled factor 2 – Zero-mean Gaussian-noise » 0.02 variane 21 / 34

– Parameter of image Table 1. Displacement Parameters of the Four LR Images.

Table 2. PSF and Noise Parameters of Case 1 and Case 2.

Fig. 3. L-curve, U-curve, and the selected regularization parameter of the “cameraman” image and “boat” image in Case 1 and 2. 22 / 34

– Result of Case 1

Fig. 4. Reconstruction results of the “cameraman” image in Case 1. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve (f) U-curve.

Fig. 7. Reconstruction results of the “boat” image in Case 1. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 23 / 34

Fig. 5. Detailed regions cropped from Fig. 4. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve (f) U-curve.

Fig. 8. Detailed regions cropped from Fig. 7. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 24 / 34

Fig. 6. Difference between the reconstruction results and the original image in Case 1. (a) BI. (b) Adaptive iteration. (c) L-curve. (d) U-curve.

Fig. 9. Difference between the reconstruction results and the original image in Case 1. (a) BI. (b) Adaptive iteration. (c) L-curve. (d) U-curve. 25 / 34

Table 3. MSE and SSIM value of different reconstruction methods in Case 1 (cameraman).

Table 4. MSE and SSIM value of different reconstruction methods in Case 1 (boat).

26 / 34

– Showing robustness of the proposed method

Fig. 14. Reconstruction MSE value under different SNR noise. (a) Cameraman image. (b) Boat image.

27 / 34

– Result of Case 2

Fig. 10. Reconstruction results of the “cameraman” image in Case 2. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve.

Fig. 12. Reconstruction results of the “boat” image in Case 2. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 28 / 34

Fig. 11. Detailed regions cropped from Fig. 10. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve.

Fig. 13. Detailed regions cropped from Fig. 12. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 29 / 34

Table 5. MSE and SSIM value of different reconstruction methods in Case 2 (cameraman)

Table 6. MSE and SSIM value of different reconstruction methods in Case 2 (boat)

30 / 34

– 2) Degradation parameters unknown case

Fig. 15. Reconstruction results of the “castle” image. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve.

Fig. 16. Detailed regions cropped from Fig. 15. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 31 / 34

– Real data

Fig. 17. Reconstruction results of the “text” image. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve.

Fig. 18. Detailed regions cropped from Fig. 17. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 32 / 34

– Optimal analysis • Showing efficacy of u-curve – Use of “cameraman” image in Case 1

Fig. 19. Change of the MSE value versus the regularization parameter.

33 / 34

Conclusion  Proposed

method

– U-cure method • Use of Selecting regularization parameter in MAP SR reconstruction model – First, Using data fidelity and prior model to construct function for regularization parameter » Plotting U-curve – Lastly, selecting optimal regularization parameter

• Advantage of method – First, computational efficiency – Second, Selecting more optimal regularization parameter than L-curve

34 / 34