2264
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Research Article
Single image super-resolution using locally adaptive multiple linear regression SOOHWAN YU, WONSEOK KANG, SEUNGYONG KO,
AND
JOONKI PAIK*
The Graduate School of Advanced Imaging Science, Multimedia and Film, Chung-Ang University, 84 Heuseok-Ro, Dongjak-Gu, Seoul, South Korea *Corresponding author:
[email protected] Received 7 August 2015; revised 16 September 2015; accepted 28 September 2015; posted 2 October 2015 (Doc. ID 247315); published 3 November 2015
This paper presents a regularized superresolution (SR) reconstruction method using locally adaptive multiple linear regression to overcome the limitation of spatial resolution of digital images. In order to make the SR problem better-posed, the proposed method incorporates the locally adaptive multiple linear regression into the regularization process as a local prior. The local regularization prior assumes that the target high-resolution (HR) pixel is generated by a linear combination of similar pixels in differently scaled patches and optimum weight parameters. In addition, we adapt a modified version of the nonlocal means filter as a smoothness prior to utilize the patch redundancy. Experimental results show that the proposed algorithm better restores HR images than existing state-of-the-art methods in the sense of the most objective measures in the literature. © 2015 Optical Society of America OCIS codes: (100.0100) Image processing; (100.3020) Image reconstruction-restoration; (100.6640) Superresolution. http://dx.doi.org/10.1364/JOSAA.32.002264
1. INTRODUCTION Image quality is determined by the sensor characteristics and the precision of an optical system. Since the spatial resolution of an image is determined by the pixel density in the sensor, a straightforward way to increase the image resolution is to increase the pixel density by reducing the pixel size at the cost of reduced sensitivity. On the other hand, it is not easy to increase sensor size because of the physical constraint of the sensor and high cost. Optical defocusing and aperture diffraction are critical factors in reducing the image resolution. In order to overcome the above-mentioned problems, signalprocessing-based superresolution (SR) algorithms can restore high-resolution (HR) images using one or more low-resolution (LR) images [1]. At first, many interpolation-based SR algorithms were proposed because of the simple computational structure and easy implementation. Among those approaches, Demirel and Anbarjafari’s work [2] was one of the most successful image resolution enhancement techniques. It uses the conventional interpolation algorithm in high-frequency wavelet subbands. However, interpolation-based SR methods cannot recover high-frequency components missing in the subsampling process. Existing SR algorithms are classified into two groups: multiple-image-based and example-based SRs. The former utilizes multiple LR images to restore the HR image, and provides a 1084-7529/15/122264-12$15/0$15.00 © 2015 Optical Society of America
theoretically promising result if a sufficient number of LR images are available with a proper restoration algorithm to deconvolve the low-pass filtering degradation in the subsampling process [3,4]. However, the multiple-image-based SR method cannot solve the registration and unwarping problems if object motions are complicated. On the other hand, the example-based SR method restores the HR image using a priori trained image patches [5]. To the best of authors’ knowledge, Freeman et al. first proposed the concept of the example-based SR approach using the hidden Markov model to select optimal LR patches to restore the HR patch [5,6]. However, Freeman’s original work requires high computational load to learn the patch dictionary, and cannot avoid the dependency on the characteristics of trained patches. To solve the problems in the original example-based approach, a single-image-based SR method was proposed using the smoothness prior in [7]. Glasner et al. proposed a more efficient way to find patches using the redundancy between in- and cross-scaled images in the scale space [8]. Freedman and Fattal, and Suetake et al. incorporated the similarity property between patches in the searching window and cropped region to improve the previous example-based SR methods [9,10]. However, Glasner et al.’s method still requires a significantly high computational load and a backprojection step to preserve the relationship with the input LR image, whereas
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Research Article
Freedman and Fattal’s method may result in patch-mismatch artifacts because of inaccurate estimation of local self-similarity. To overcome the above-mentioned limitations of recently developed example-based SR methods, the proposed SR method generates multiple down-scaled images from the initially interpolated image. More specifically, the input LR image is first expanded using a simple interpolation kernel. Next, multiple, differently low-pass filtered images are generated. After each image is downsampled to make a scale space, similar patches are selected from both in- and cross-scale image pairs. The final HR patch is then obtained using linear regression [11]. The high-resolution image generated by linear regression is used as a smoothness prior on each pixel [12]. Since the proposed method estimates the optimally similar patches in the multiple, differently downsampled images, significantly increased patch similarity is guaranteed, and the linear regressionbased patch combination can efficiently remove undesired SR artifacts. The paper is organized as follows. After briefly reviewing the related works in Section 2, the proposed SR algorithm is described in Section 3. Section 4 summarizes experimental results, and Section 5 concludes the paper.
JΘ arg min‖ˆy − y i ‖2 arg min‖ Θ
Θ
q X j0
2265
θj x ij − y i ‖2 ;
(4)
where yˆ is a predicted value computed by a linear combination of an estimated weight parameter θ and a set of ith independent variables. The optimization problem given in Eq. (4) can be performed using the gradient descent method as, for each i, ∂ JΘ Θt Θt1 Θt α ∂Θ X q q X yi − x ij ; α θk x ik (5) j0
k0
where α is a learning rate, and t the iteration number. B. Nonlocal Means Filter
The nonlocal means (NLM) filter was originally proposed to remove noise in the image under the assumption that the nonlocal similarity of image contents is redundant within the natural image [13]. The NLM filtered intensity value of the ith pixel is computed by weighted averaging of the similar neighboring pixels as X wx i ; x j NL zx j ; (6) zx i NL j∈S
2. BACKGROUND THEORY This section briefly revisits the basic theory of multiple linear regression and nonlocal means filtering, and summarizes technical terminologies for the following sections.
where S is a set of pixel coordinates satisfying the similarity condition, and zx j is the intensity value of the jth pixel. The similarity weight wx i ; x j NL between x i and x j is computed as 2
A. Multiple Linear Regression
wx i ; x j NL
Multiple linear regression is a statistical tool to analyze the relationship between p observed variables, y i , for i 1; …; p, and multiple independent variables, x ij , for j 1; …; q. The relationship can be expressed as Y XΘ E; where
2
3 y1 6 y2 7 6 7 Y 6 .. 7; 4 . 5
2
1 61 6 X6. 4 .. 1
yp
3 θ0 6 θ1 7 6 7 Θ 6 .. 7; 4 . 5
3 x 1q x 2q 7 7 .. 7; .. . . 5 x pq
and
3 ε1 6 ε2 7 6 7 E 6 .. 7: 4 . 5
(2a)
(2b)
εp
The linear regression model for the ith observation in Eq. (1) can be expressed, for i 1; …p, as yˆ i θ0 x i0 θ1 x i1 θ2 x i2 … θq x iq εi ;
where Cx i is the normalization factor defined as X −‖Pxi −Pxj ‖2G h2 e ; Cx i
(7)
(8)
j
where Px i and Px j , respectively, represent column vectors centered at x i and x j including the relevant neighbors, and h is a filter parameter that controls the decaying rate of the exponential function. G is a Gaussian kernel to assign the maximum weight at the center of a patch. The similarity weight between the target pixel x i and a neighboring pixel x j depends on the Euclidean distance between vectors Px i and Px j . 3. PROPOSED SR METHOD
2
2
θq
x 11 x 21 .. . x p1
(1)
j ‖G 1 −‖Pxi −Px h2 e ; Cx i
(3)
where x i0 1, Θ θ0 ; …; θq T represents a q 1 weight vector to be estimated, and εi the ith regression error with zero mean. To estimate the optimum set of parameters, the following cost function should be minimized:
This section presents a novel regularized SR method that combines multiple linear regression and the NLM filter. The proposed method consists of three steps: (i) selection of similar patches in multiple scale spaces that are generated in different ways, (ii) reconstruction of a desired HR patch using multiple linear regression with the selected patches in the previous step, and (iii) regularized optimization that combines the multiple linear regression and NLM filter, as shown in Fig. 1. A. Image Degradation Model
The SR problem is to estimate the original HR image from a given LR version that is generated by various degradation factors, including low-pass or antialiasing filtering, subsampling,
2266
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Research Article
Fig. 1. Block diagram of the proposed three-step SR method.
and noise. In this work, the image degradation model is defined as g SH f η;
(9)
where g and η are M 2 × 1 row-ordered vectors representing an M × M LR image and additive noise, respectively. f is the N 2 × 1 vector representing the N × N HR image, H an N 2 × N 2 block matrix representing a low-pass or antialiasing filter, and S an N 2 × N 2 block matrix representing the downsampling operator. B. Selection of Similar Patches in Multiple Scale Spaces
A natural image has similar patterns in a nonlocal manner, and similar patterns can be found in a differently scaled image [8,13]. Based on the cross-scale similarity property, the SR problem can be solved by reconstructing the HR patch that is similar to the input LR patch instead of directly estimating the inverse of the degraded image given in Eq. (9). However, incorrectly selected patches may result in undesired patchmismatching artifacts that degrade the quality of SR results. To solve the patch-mismatching problem, the proposed SR method first generates multiple scale spaces using both upsampling interpolation and Gaussian blurring kernels, and then selects multiple similar patches in the generated scale spaces, as shown in Fig. 2. Selected multiple patches and the input LR patch are used to reconstruct the HR patch using locally adaptive multiple linear regression. The details of the patch selection process is described in Subsection 3.C. The similar patches selected in multiple scale spaces are closer to the input LR patch than those selected in the external training database. In addition, Gaussian low-pass filtering
followed by downsampling by a small scale factor guarantees a sufficient amount of redundancy and self-similarity among multiple scale spaces [14]. For that reason, the selected similar patches in various scale spaces can be regarded as the multiple version of the original HR patch by low-pass filtering and downsampling. The input LR image is considered a degraded version of the original HR image f according to the image degradation model in Eq. (9). As shown in Fig. 2, the input image g is first upsampled to generate an enlarged image g up using a simple interpolation method. g up is then low-pass filtered and downsampled by various types of filters. Four Gaussian low-pass filters, B i , have different values of standard deviation, f0.1; 0.2; 0.3; 0.4g, for the experimentally best result. As a result, 25 images including g ij , for i; j 0; …; 4, form five different scale spaces. To reconstruct the HR patch corresponding to a target patch in g 00 , the most similar patch is selected in the remaining 24 images by minimizing the Euclidean distance, for i; j 0; …; 4, except i j 0, as ‖g P00 c x ; c y − g Pij c x − x; c y − y‖2 wx; y exp − ; (10) h2 where g P00 c x ; c y represents a 5 × 5 patch centered at c x ; c y in the upsampled image g 00 , and g Pij c x − x; c y − y represents the 5 × 5 patch centered at c x − x; c y − y in g ij . In other words, given a target patch in g 00 , the most similar patches are selected in the remaining 24g ij , as shown in Fig. 3. Since 24 selected patches come from images in multiple scale spaces, a proper combination of them can reconstruct the corresponding patch in the original HR image without jagging artifacts or unnaturally amplified high-frequency components.
Fig. 2. Generation of the proposed multiple scale spaces and similar patches: B i , for i 1; …; 4, are Gaussian low-pass filters with different standard deviations, and Di , for i 1; …; 4, is a downsampling operation by 1.25−i times.
Research Article
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
2267
methods, and an optimal combination of multiple candidate patches can provide better SR performance. This subsection presents a novel method to combine multiple candidate patches using learning-based linear regression. More specifically, after learning the similarity between the target and the 24 selected patches, the optimal set of weight parameters is estimated. The desired HR patch is reconstructed by the linear combination of selected patches using the locally adaptive weights, as shown in Fig. 4. The details of the proposed patch reconstruction method are described below. Assuming each patch is 5 × 5, the 24 selected patches are rearranged into the 25 × 25 matrix to make the independent variable X in Eq. (1): X b
gP 01
1; ; 1T
Fig. 3. Similar patches corresponding to an input patch: (a) input patch in g 00 and (b) similar patches in the remaining 24 images.
gP 02
gP 44 ;
∈ and is the column vector where b containing 5 × 5 pixels of g P ij c x ; c y in the raster scanning order. The dependent variable Y in Eq. (1) is also defined as P
More specifically, the reconstructed HR patch using the proposed SR method does not amplify the noise since it is a combination of multiple, differently low-pass filtered patches. In addition, an appropriate combination of multiple, differently scaled patches can provide a very similar result to the original HR patch without any synthetic artifacts, which is the common problem of most patch-based SR methods. C. Locally Adaptive Linear Regression
This subsection presents a locally adaptive linear regression method to reconstruct the optimal HR patch using 24 selected patches in multiple scale spaces described in the previous subsection. Let g Pij c x − x ; c y − y g P ij c x ; c y be the most similar patch in g ij to the target patch g P00 c x ; c y in the upsampled image g 00 , where x ; y represents the optimal shifting vector in g ij . The 24 selected patches, g P ij c x ; c y , for i; j 0; …; 4, i j ≠ 0, are differently scaled versions of the original HR patch, and have a similar pixel property [8,13]. The abovementioned similarity is widely used for multiple-frame SR
(11)
gP ij
R25 ,
Y g00high ;
P g00high
(12)
where is the column vector containing 5 × 5 highfrequency pixels of g P00 c x ; c y in the raster scanning order. In multiple scale spaces, the representative image features, such as corner, edge, and T-junction, are scale-invariant, if the image is downsampled by a very small nondyadic scale factor [9]. Accurate and similar patches can be found in a specific region using local self-similarity. Therefore, the high-frequency component of the input patch is estimated from the difference between the input patch and the patch that is downsampled by 1.25 times and then upsampled by the same ratio. Since the row of X can be considered the similar pixels in the HR patch to be estimated, the optimal weight parameter vector Θ is learned, for i 1; …; 25, as Gradient Descent Term
Θt1
zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{ X 24 24 X x il ; Θt α wi y i − θq x iq l 0
(13)
q0
where wi is the weight representing similarity between the input patch and the shifted patch in g ij by x; y as defined in
Fig. 4. Proposed multiple linear regression process using 24 similar patches.
2268
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Research Article
Fig. 5. Results of 4× magnification: (a) input image, (b) cubic-spline interpolation [16], and (c) locally adaptive multiple linear regression using the closed-form pseudo-inverse method, and (d) locally adaptive multiple linear regression using the iterative gradient descent method.
Eq. (10). In the gradient descent optimization process given in Eq. (13), we used the backtracking line-search method [15] to use the adaptive learning step size α. Since the direct inversion of matrix X of size 25 × 25 is numerically unstable, the weight parameter vector Θ of the multiple linear regression problem should be computed using either iterative minimization or the pseudo-inverse. Figures 5(c) and 5(d), respectively, show the result of the pseudo-inverse and iterative minimization to solve the multiple linear regression problem that enlarges an input image by four times using the estimated weight parameter vector Θ. Although Fig. 5(c) provides a better sharpened image than the cubicspline interpolation method shown in Fig. 5(b), it cannot avoid the jagging artifact near edges and the mean brightness value is decreased. On the other hand, the proposed method successfully removes jagging artifacts near the edges without unnatural amplification of high-frequency components in the SR result, as shown in Fig. 5(d). The result of locally adaptive linear regression can also be used as a smoothness prior in the regularized SR framework in the following subsection. D. Regularized SR Combining Nonlocal Means and Locally Adaptive Multiple Linear Regression
This subsection presents a regularized SR method by incorporating nonlocal means and locally adaptive linear regression as regularization constraints. Based on the image degradation model in Eq. (9), estimation of HR image f from the LR image g is an ill-posed problem, since a small noise or observation error in g significantly degrades the estimation results in an uncontrolled manner. A general regularization approach is to make this problem betterconditioned by incorporating a smoothness constraint in the estimation process: fˆ arg min‖g − SH f ‖2 λ‖Cf ‖2 ; f
(14)
where ‖g − SH f ‖ represents the data-fidelity data term, C a high-pass filtering operation, and ‖Cf ‖2 the smoothness constraint term. The relative weighting between the data fidelity and smoothness constraints is controlled by the regularization parameter λ [17–22]. Although the general regularization can suppress noise amplification in the restoration process to 2
a certain degree, it cannot always promise a successful result in the SR problem that includes the inversion process of combined anti-aliasing low-pass filtering and down-sampling. More specifically, since the SR reconstruction process is an ill-posed problem, it cannot directly estimate the inverse of the antialiasing low-pass filter, denoted as S, and the downsampling matrix H . For this reason, a restored HR image suffers from amplification of undesired artifacts, including noise, blurring, and jagging artifacts. To overcome the limitation of the general regularization approach, the proposed SR algorithm incorporates both nonlocal means and locally adaptive multiple linear regression results into the regularization process. To reduce noise amplification in the SR process while preserving details in the image, the nonlocal means filter estimates the noise-reduced version of the HR estimate as fˆ arg minf‖f − f NLM ‖2 g f
arg min f
(
X
i∈f Ω
) ‖f i − Wf i ; f j NL · Si ‖2 ;
(15)
where f i represents the ith pixel in the HR image, Si the column vector containing pixel coordinates that satisfy the nonlocal similarity condition of f i , f Ω the searching region of the candidate patch, and Wf i ; f j NL the row vector containing the nonlocal similarity weights between the HR pixel f i and its similar pixels. To incorporate the locally adaptive multiple linear regression result as a smoothness constraint into the regularization process, the following optimization can be used: fˆ arg minf‖f − f MLR ‖2 g f ( ) X arg min ‖f i − Mi · Θi ‖2 ; f
(16)
i∈f
where f MLR represents the initial HR estimate using locally adaptive multiple linear regression, Θi the weighting parameter to f i , and Mi the row vector of X containing the similar pixels to f i . By combining the NLM and multiple linear regression constraints, the proposed regularized SR method can be expressed as
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Research Article
fˆ arg minf‖g − SH f ‖2 λ1 ‖Cf ‖2 f
λ2 ‖f − f MLR ‖2 λ3 ‖f − f NLM ‖2 g;
(17)
where λ1 , λ2 , and λ3 represent regularization parameters for noise smoothing, linear regression, and NLM, respectively. Equation (17) can also be expressed as fˆ arg minf‖g − SH f ‖2 λ1 ‖Cf ‖2 f
λ2 ‖I − W MLR f ‖2 λ3 ‖I − W NLM f ‖2 g; (18) where I represents the identity matrix, and W MLR and W NLM are defined as follows: Θi ; j ∈ M i W MLR ; (19) 0; otherwise and
W NLM
Wf i ; f j NL ; 0;
j ∈ Si ; otherwise
f n1 f n τfH T S T g − SH f λ1 C T Cf λ2 I − W MLR T I − W MLR f (21)
and Eq. (21) can be rewritten as f n1 f n τH T S T g − SH f − βC T Cf
In this subsection, we evaluated the effectiveness of the proposed regularization constraints. In the experiment, we fixed the parameters as τ 1.0, β 0.2, and h 15. Figure 6 shows the performance of the SR method incorporating the proposed regularization constraints. Figures 6(b) and 6(c), respectively, show the results of using the multiple linear regression term with γ 0.5 and ω 0, and the NLM regularization term with γ 0 and ω 0.5. The multiple linear regression term can better restore the high-frequency details than the NLM term. The combination of the multiple linear regression and NLM regularization terms can restore the high-frequency details without undesired SR artifacts. B. Comparative Evaluation Using Simulated LR Images
− γI − W MLR T I − W MLR f − ωI − W NLM T I − W NLM f ;
interpolation [16], steering kernel regression (SKR) [24], patch-based SR [25], sparse coding-based SR (ScSR) [26], and Kim and Kwon’s method [27]. The objective image quality of the resulting SR images is assessed by a set of full-reference metrics such as peak-to-peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) [28]. To compute the similarity weight, we use the patch of size 5 × 5 and the searching region of size 15 × 15 in each scale space. The filter parameter h is set to 15, and the learning step size α is adaptively determined by the backtracking line-search method in multiple linear regression. For each test image, the proposed method uses the estimated similar patches in 24 scale spaces as mentioned in Section 3.B. In the regularization process, the learning rate τ for gradient descent optimization, and the regularization parameters β, γ, and ω are experimentally chosen for the best SR reconstruction results. A. Experiments on the Regularization Terms
(20)
where both Mi and Si represent the vectors containing the coordinates of pixels that are similar to f i . The optimization problem in Eq. (18) can be solved by the iterative gradient descent method as [23]
λ3 I − W NLM T I − W NLM f g;
2269
(22)
where n represents the iteration number, τ the learning rate, and fβ; γ; ωg the regularization parameters. The initial estimate of the iteration f 0 can be obtained using a simple interpolation method, such as bilinear or cubic-spline interpolation [16]. 4. EXPERIMENTAL RESULTS In this section, to evaluate the performance of the proposed SR method, the SR result is compared with cubic-spline
We used six test images, as shown in Fig. 7, to compare the performance of the proposed SR method with existing stateof-the-art methods. The input LR image is generated by the Gaussian low-pass filtering with σ 0.5, and then decimated using a cubic-spline interpolation kernel by factors of 2 and 4. The simulated LR RGB color images are converted to the YCbCr color space, and only the Y channel is magnified using various SR methods. Cb and Cr channels are simply expanded using the cubic-spline interpolation [16], and the resulting YCbCr channels are converted back to the RGB color space.
Fig. 6. Evaluation of the proposed regularization constraints in 4× magnification: (a) the original HR image, (b) result of using the multiple linear regression term (PSNR, 23.4589; SSIM, 0.7982), (c) result of the NLM regularization term (PSNR, 23.4661; SSIM, 0.7961), and (d) result of combined regularization terms (PSNR, 23.4859; SSIM, 0.8040).
2270
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Fig. 7. Six test images for comparative evaluation of various SR methods.
Research Article
data-fidelity term ‖g − SH f ‖2 in Eq. (14). To provide a fair comparison with Yang et al.’s method, an executable program with a test input is given, as we show in Code 1, Ref. [30]. Kim and Kwon’s method [27] learns the relationship between the input LR and corresponding HR images using kernel ridge regression and combined gradient descent and kernel matching pursuits. This method provides better results than the above-mentioned ones, but it generates the halo effect around the edge region. To avoid the backprojection artifacts, the proposed method iteratively estimates the SR solution in the HR image space using regularized optimization, and incorporates the patch redundancy among multiple scale spaces generated from the input LR image. Therefore, the proposed method can better reconstruct the high-frequency details, without unnatural artifacts, than existing SR methods. Objective quality assessments are summarized in Tables 1 and 2. C. Optimum Regularization Parameter Setting
Figures 8, 9, 10, and 11 show the results of various SR methods to enlarge simulated LR images by factors of 2 and 4. Since the SKR is originally proposed to remove noise, its SR performance is limited. The patch-based SR [25] generates undesired artifacts near the edge, and it cannot sufficiently recover details such as edge and texture. The result of He et al.’s method depends on the size of LR and HR patches. A pair of smaller patches provides better SR performance at the cost of increasing computational complexity and processing time. Yang et al. [26] used more sparsity-based prior information than the proposed method. However, the resulting SR image is degraded by interpolation artifacts, such as blurring and jagging, in the backprojection [29] process that minimizes the
In the proposed method, the optimal parameter selection step plays an important role in obtaining the best SR reconstruction result. In this experiment, we estimated the proper values of regularization parameters γ and ω using upsampled versions of the simulated LR image shown in Fig. 7 by two and four times. For 2× magnification, we set τ and β to 1.5 and 0.05, respectively, and γ and ω vary from 0.1 to 1.5 and from 0.1 to 1.5 with intervals of 0.1 and 30 iterations, respectively. For 4× magnification, parameters τ and β are set to 0.9 and 0.15, respectively. Parameters γ and ω in the multiple linear regression and NLM steps, respectively, vary from 0.1 to 1.3 and from 0.1 to 1.5 with intervals of 0.1 and 15 iterations. Optimally selected parameters γ and ω give the highest
Fig. 8. Results of various SR methods using a simulated LR image: (a) the original HR image, (b) the simulated LR image downsampled by a factor of 2, (c) result of cubic-spline interpolation [16], (d) result of the SKR method [24], (e) result of the patch-based SR [25], (f ) result of Yang et al.’s method [26], (g) result of Kim and Kwon’s method [27], and (h) result of the proposed method (τ 1.5, β 0.02, γ 0.2, ω 1.4, and n 30).
Research Article
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
2271
Fig. 9. Results of various SR methods using a simulated LR image: (a) the original HR image, (b) the simulated LR image downsampled by a factor of 2, (c) result of cubic-spline interpolation [16], (d) result of the SKR method [24], (e) result of the patch-based SR [25], (f ) result of Yang et al.’s method [26], (g) result of Kim and Kwon’s method [27], and (h) result of the proposed method (τ 1.5, β 0.02, γ 0.2, ω 0.4, and n 30).
Fig. 10. Results of various SR methods using a simulated LR image: (a) the original HR image, (b) the simulated LR image downsampled by a factor of 4, (c) result of cubic-spline interpolation [16], (d) result of the SKR method [24], (e) result of the patch-based SR [25], (f ) result of Yang et al.’s method [26], (g) result of Kim and Kwon’s method [27], and (h) result of the proposed method (τ 0.9, β 0.05, γ 0.6, ω 1.3, and n 15).
PSNR and SSIM values. Figures 12 and 13 show the comparison of PSNR values for each simulated test image shown in Fig. 7.
D. Subjective Evaluation Using Real Photographs
We tested the proposed SR method on the real photographs shown Figs. 14 and 15. As shown in the figures, the simple
2272
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Research Article
Fig. 11. Results of various SR methods using a simulated LR image: (a) the original HR image, (b) the simulated LR image downsampled by a factor of 4, (c) result of cubic-spline interpolation [16], (d) result of the SKR method [24], (e) result of the patch-based SR [25], (f ) result of Yang et al.’s method [26], (g) result of Kim and Kwon’s method [27], and (h) result of the proposed method (τ 0.9, β 0.05, γ 0.2, ω 0.6, and n 15).
Table 1. Comparison Values of PSNR and SSIM [28] on the Resulting SR Images Using Five Existing and the Proposed SR Methods for 2× Magnification Images
Methods
[16]
[24]
[25]
[26]
[27]
Proposed
Fig. 7(a)
PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM
24.23 0.832 25.96 0.867 27.38 0.903 28.93 0.883 26.60 0.848 22.15 0.760
21.42 0.683 23.41 0.786 24.25 0.820 25.71 0.775 24.18 0.744 19.73 0.613
22.23 0.780 23.66 0.826 24.29 0.861 26.10 0.831 24.32 0.803 19.55 0.667
26.81 0.904 27.69 0.900 29.81 0.931 31.25 0.924 28.30 0.894 23.44 0.821
27.17 0.910 27.84 0.902 30.03 0.935 31.52 0.929 28.40 0.870 23.69 0.827
27.30 0.913 27.96 0.903 30.05 0.939 31.71 0.934 28.58 0.904 23.92 0.838
Fig. 7(b) Fig. 7(c) Fig. 7(d) Fig. 7(e) Fig. 7(f )
Table 2. Comparison Values of PSNR and SSIM [28] on the Resulting SR Images Using Five Existing and the Proposed SR Methods for 4× Magnification Images
Methods
[16]
[24]
[25]
[26]
[27]
Proposed
Fig. 7(a)
PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM
18.31 0.475 19.90 0.662 20.35 0.679 21.75 0.579 21.30 0.607 16.47 0.377
16.97 0.397 18.24 0.560 18.49 0.634 19.34 0.544 18.91 0.518 15.69 0.343
19.60 0.544 21.41 0.710 21.39 0.722 23.51 0.647 22.14 0.642 17.60 0.438
20.67 0.613 22.71 0.733 22.92 0.771 24.92 0.713 23.35 0.687 18.93 0.526
20.90 0.621 22.82 0.744 22.89 0.775 24.92 0.712 23.49 0.697 18.84 0.512
21.51 0.663 23.43 0.769 23.49 0.804 25.70 0.752 23.89 0.723 19.16 0.553
Fig. 7(b) Fig. 7(c) Fig. 7(d) Fig. 7(e) Fig. 7(f )
Research Article
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
2273
Fig. 12. Comparisons of PSNR values on the regularization parameters γ and ω using upsampled versions of the six simulated LR test images shown in Fig. 7 for 2× magnification.
Fig. 13. Comparisons of PSNR values on the regularization parameters γ and ω using upsampled versions of the six simulated LR test images shown in Fig. 7 for 4× magnification.
interpolation method degrades the image quality by generating severe jagging and blurring artifacts. The SKR method gives a sufficiently smoothed result, but it cannot recover the highfrequency details. Although Yang et al.’s method provides sharper results than simple interpolation and SKR methods, it cannot avoid undesired SR artifacts due to the backprojection process. The proposed SR reconstruction method shows better SR results than other existing methods in the sense of both sharper details and minimized SR artifacts. For the computational efficiency on the weight matrix and estimation, the proposed method computes the weight matrix simultaneously with the initially interpolated image, and the NLM weights are computed in the specified limited region. These regularization prior terms produce promising SR results and guarantee the optimal patch selection.
The performance of the proposed method is compared with existing SR methods by using a personal computer with a 3.4 GHz CPU and 16 Gbytes of RAM. The experiment is adapted to enlarge the LR images of size 128 × 128 by a factor of 4. The steering-kernel-based method [24] takes only 15.96 s at the cost of a blurry result. The patch-based method [25] takes more than 1 h because its performance depends on the number of pairs of low- and highresolution patches. Yang et al.’s [26] and Kim and Kwon’s [27] methods, respectively, take 398.43 and 45.76 s, but they cannot avoid undesired artifacts near the edge. In contrast, the proposed method takes 261.68 s with 20 iterations, and successfully reconstructs the enhanced HR image with less undesired artifacts than existing state-of-the-art SR methods.
2274
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
Research Article
Fig. 14. Results of various SR methods using a real photograph: (a) the input LR photograph, (b) result of cubic-spline interpolation [16], (c) result of the SKR method [24], (d) result of the patch-based method [25], (e) result of Yang et al.’s method [26], (f) result of Kim and Kwon’s method [27], and (g) result of the proposed method (τ 1.5, β 0.02, γ 0.1, ω 1.0, and n 20).
Fig. 15. Results of various SR methods using a real photograph: (a) the input LR photograph, (b) result of cubic-spline interpolation [16], (c) result of the SKR method [24], (d) result of the patch-based method [25], (e) result of Yang et al.’s method [26], (f) result of Kim and Kwon’s method [27], and (g) result of the proposed method (τ 1.5, β 0.02, γ 0.3, ω 0.9, and n 20).
Research Article
Vol. 32, No. 12 / December 2015 / Journal of the Optical Society of America A
5. CONCLUSION Patch-based SR methods commonly assume that important image contents, such as edge, corner, and texture, repeat in both in- and cross-scale spaces. However, it is not easy to find the optimal HR patch from either single or multiple LR images. To solve this problem, the proposed method generates multiple scale spaces using low-pass filtering and downsampling with a small scale factor. In addition, the similar patches found in various scales are used to generate the optimally estimated HR patches using multiple linear regressions. A target HR pixel can be regarded as a linear combination of its similar pixels with optimum weight parameters. Therefore, multiple linear regression is incorporated into the regularization process as a local smoothness prior with NLM filtering and noise suppression constraints. Experimental results show that the proposed algorithm can successfully provide better HR images than existing state-of-the-art SR methods in the sense of most objective measures in the literature. Funding. Ministry of Science, ICT and Future Planning (MSIP) (B0101-15-0525, IITP-2015-H8501-15-1018). REFERENCES 1. R. Y. Tsai and T. S. Huang, “Multi-frame image restoration and registration,” Adv. Comput. Vis. Image Process. 1, 317–339 (1984). 2. H. Demirel and G. Anbarjafari, “Image resolution enhancement by using discrete and stationary wavelet decomposition,” IEEE Trans. Image Process. 20, 1458–1460 (2011). 3. S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Trans. Image Process. 13, 1327– 1344 (2004). 4. X. Li, Y. Hu, X. Gao, D. Tao, and B. Ning, “A multi-frame image superresolution method,” Signal Process. 90, 405–414 (2010). 5. W. T. Freeman and E. C. Pasztor, “Learning low-level vision,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 1999), pp. 1182–1189. 6. W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-based super-resolution,” IEEE Comput. Graph. Appl. 22, 56–65 (2002). 7. J. Sun, J. Sun, Z. Xu, and H. Y. Shum, “Image super-resolution using gradient profile prior,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8. 8. D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” in Proceedings of IEEE Conference on Computer Vision (IEEE, 2009), pp. 349–365. 9. G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” ACM Trans. Graph. 28, 1–10 (2010). 10. N. Suetake, M. Sakano, and E. Uchino, “Image super-resolution based on local self-similarity,” Opt. Rev. 15, 26–30 (2008). 11. X. Liu, D. Zhao, R. Xiong, S. Ma, W. Gao, and H. Sun, “Image interpolation via regularized local linear regression,” IEEE Trans. Image Process. 20, 3455–3469 (2011).
2275
12. S. Kolouri and G. K. Rohde, “Transport-based single frame super resolution of very low resolution face images,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 4876–4884. 13. A. Buades, B. Coll, and J. M. Morel, “A review of image denoising algorithms, with a new one,” Multiscale Model. Simul. 4, 490–530 (2005). 14. T. Linderberg, “Feature detection with automatic scale selection,” Int. J. Comput. Vis. 30, 79–116 (1998). 15. S. Boyd and L. Vandenberghe, “Gradient descent method,” in Convex Optimization (Cambridge University, 2004), pp. 464–469. 16. R. Keys, “Cubic convolution interpolation for digital image processing,” IEEE Trans. Acoust. Speech Signal Process. 29, 1153–1160 (1981). 17. J. Shin, J. Jung, and J. Paik, “Regularized iterative image interpolation and its application to spatially scalable coding,” IEEE Trans. Consum. Electron. 44, 1042–1047 (1998). 18. J. Shin, Y. Choung, and J. Paik, “Regularized iterative image sequence interpolation with spatially adaptive constraints,” in Proceedings of the IEEE International Conference on Image Processing (IEEE, 1988), pp. 470–473. 19. J. Shin, J. Paik, J. Price, and M. Abidi, “Adaptive regularized image interpolation using data fusion and steerable constraints,” Proc. SPIE 4310, 798–809 (2001). 20. A. K. Katsaggelos, “Iterative image restoration algorithms,” Opt. Eng. 28, 287735 (1989). 21. A. K. Katsaggelos, J. Biemond, R. W. Schafer, and R. M. Mersereau, “A regularized iterative image restoration algorithms,” IEEE Trans. Signal Process. 39, 914–929 (1991). 22. W. Kang, S. Yu, S. Ko, and J. Paik, “Multisensor super resolution using directionally-adaptive regularization for UAV images,” Sensors 15, 12053–12079 (2015). 23. K. Zhang, X. Gao, D. Tao, and X. Li, “Single image super-resolution with non-local means and steering kernel regression,” IEEE Trans. Image Process. 21, 4544–4556 (2012). 24. H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image processing and reconstruction,” IEEE Trans. Image Process. 16, 349–366 (2007). 25. H. He and W. Siu, “Single image super-resolution using Gaussian process regression,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2011), pp. 449–456. 26. J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19, 2861– 2873 (2010). 27. K. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image prior,” IEEE Trans. Pattern Anal. Mach. Intell. 32, 1127–1133 (2010). 28. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004). 29. M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graph. Models Image Process. 53, 231–239 (1991). 30. S. Yu, W. Kang, S. Ko, and J. Paik, “Comparison of super-resolution results,” figshare (2015), retrieved October 4, 2015, http://dx.doi.org/ 10.6084/m9.figshare.1545603.
