2013 Annual IEEE India Conference (INDICON)
Spatial Super Resolution Based Image Reconstruction using HIBP Rajashree Nayak Electrical Engineering Department NIT Rourkela, Odisha, India email:
[email protected]
S Monalisa Electrical Engineering Department NIT Rourkela, Odisha, India e-mail:
[email protected]
Abstract— Spatial image resolution explains about the pixel density in a digital image. As a result more the number of pixels more detailed visibility of information contained in the image. Hardware limitations restrict the increase in number of sensor elements per unit area in camera. Therefore an imaging system with inadequate sensor array will generate low resolution image which causes pixelization effect in them. This problem is solved in software level using signal processing techniques called super resolution based image reconstruction. In this paper super resolution based image reconstruction problem is addressed, which is used for resolution enhancement. Unlike interpolation, it takes information from multiple number of low resolution images with sub-pixel shifts and contain nonredundant data to generate a high resolution image. In this proposed reconstruction method, a hybrid iterative back projection technique is developed exploiting the notion of cuckoo search optimization algorithm in iterative back projection method. The high resolution solution from iterative back projection method is optimized using Cuckoo optimization algorithm. The performance of the proposed algorithm is found to be outperforming that of existing IBP and other interpolation based reconstruction techniques. Keywords—Super resolution; Iterative Cuckoo optimization algorithm; Levy flights.
I.
Back
Projection;
INTRODUCTION
Image reconstruction is a mathematical process that generates images from X-ray projection data acquired at many different angles. It is the attempt to retrieve information that has been lost or obscured in the imaging process itself. It retrieves images from degraded noisy, blurred and aliased images. Image reconstruction encompasses the entire image formation process and provides a foundation for the subsequent steps of image processing. In contrast to image enhancement, where the appearance of an image is improved to suit some subjective criteria, image reconstruction is an objective approach to recover a degraded image based on mathematical and statistical models. It has a wide application in the area of medical imaging, remote sensing, surveillance applications and satellite imaging where the accurate internal images can be obtained by combining images from different angles. Image spatial resolution depends on the physical properties of the sensor: the density of sensor pixels and the
Prof. (Mrs.) Dipti Patra Electrical Engineering Department NIT Rourkela,Odisha, India e-mail:
[email protected]
optics. Hardware limitations restrict modifications in sensor for resolution increment. More detailed information about scene could be achieved by obtaining multiple numbers of low resolution (LR) samples of the scene from a sequence of images. Image reconstruction takes sub-pixel shift information from the sequence of images and other sensor parameters to estimate the high resolution (HR) image. In the last two decades, the super-resolution reconstruction problem is well known, a challenging and extensively treated in the literature [1-5].Super-resolution based image reconstruction is a technique to reconstruct a high resolution images from a set of blurred and noisy low resolution images of a scene, thereby increasing the high frequency component and removing the degradations caused by the imaging process of the low resolution camera. The principal desire of high resolution image stems from two main reasons: enhancement of pictorial information for human perception; and helping representation for automatic machine perception. Earlier work on super resolution (SR) was carried by Irani and Peleg [1], who proposed an iterative algorithm for image registration having with sub-pixel accuracy. Later, the special case of super-resolution restoration (where the wraps are pure translations, the blur is space invariant and same for all the images and the noise is white) was proposed for a fast superresolution restoration in 2001[9]. A hybrid approach for single image super-resolution using IBP method with the edge preserving infinite symmetrical exponential filter (ISEF) for minimizing the reconstruction error significantly in iterative manner was proposed in [10]. The IBP method starts with an approximate estimation of the HR image and during registration procedure iteratively refines the displacement estimation. It also considers the blurring effect by using the Point Spread Function (PSF). The displacement estimation is done from a gradient image, which is calculated from the sum of the errors between each LR image and the estimated HR image [2] [3]. The gradient based technique used in IBP method, updates the HR estimate with a constant gradient during iterations, which solely depends on the initial guess of the high resolution image. If the initial guess is near about the optimum point in the search space then it converges efficiently otherwise the result is a local optimal solution even it fails to converge sometimes. This leads the need for meta-heuristic methods of optimization.
978-1-4799-2275-8/13/$31.00 ©2013 IEEE
An iterative approach to estimate the HR solution using iterative back projection (IBP) along with an evolutionary based meta-heuristic optimization technique called hybrid iterative back projection (HIBP) to optimize the HR solution is proposed in this paper. Here the optimization process is implemented with cuckoo optimization algorithm with Levy flights using Mantegna’s distribution which gives a better global optimum. This paper compares the results of Bi-Cubic Interpolation method, IBP and HIBP algorithms on various number of LR images by using mean squared error, PSNR and Structural Similarity (SSIM) index which is a useful metric for validation of image structural information. Examples are shown for grayscale images with increase in resolution showing clear high frequency details. II.
RECONSTRUCTION OF IMAGES USING SR
A. Image observation model The reconstruction based approach to spatial SR assumes sub-pixel level accuracy for registration and to start the observation model for the LR images yk from the unknown HR image , x, which the SR method wants to achieve is given as,
yk = ( DBkWk ) x + e
∀ k = 1, 2,
,N
(1)
The limited sensor density causes an aliasing effect in images leads to lower spatial resolution of images. This is modeled using down-sampling operator D . The finite aperture size in the camera causes sensor blur in the images which is modeled mathematically using point spread function (PSF). This blur is introduced using Gaussian blur in the blur matrix Bk . The sub-pixel shift information from multiple LR images is represented by translation or geometric warp operator Wk . B. Iterative Back Projection The iterative process starts with a rough estimation of HR 0
solution, called as initial guess x , which can be generated by one of the LR frames by decimating the pixels. Then iteratively a “gradient” image is added to it during the course of operation. The gradient image is calculated as, (2) Gradient image = ( y − y sim ) sim
Where, y is observed LR image and y is simulated LR image. The simulated LR images are compared with the observed ones and the error generated between them is back projected onto the initial guess after multiplying it with the back projection operator,
Where, •
yk : kth observed LR frame
•
D : Down sampling matrix
•
Bk : Blur matrix
•
Wk : Geometric warp matrix
•
x
•
N : number of low resolution images taken into
•
Figure 1 describes the image observation model.Input to the model is a reference HR image which undergoes affinetransformation, blur and down-sampling along with added noise gives a multiple number of low resolution images. The output of each block is shown in the corresponding camera man image.
X = X 0 + Abp ( y − y sim )
Where, Abp is back projection operator calculated from the image observation model, X is output HR solution.
: HR image desired
consideration
e : noise Geometric Warping -Translation - Rotation
(3)
Down-sampling
Blur
-Undersampling -Aliasing
Sensor averaging
Desired HR image Low resolution images
+ Sensor noise
Figure 1: The image observation model
III.
CUCKOO OPTIMIZATION ALGORITHM
Yang & Deb proposed a novel method for global optimization on cuckoo breeding behavior [6]. Egg laying and breeding style of cuckoos is the basis of this optimization algorithm. Cuckoos lay their eggs in the nests of host birds who hatch their eggs. The works of Civicioglu & Besdok, Rajabioun further confirmed that the “cuckoo search” algorithm, in its original or improved version, proves to be very effective [7] [8]. The cuckoo search algorithm based on Yang & Deb is replica of the idea of how cuckoos lay their eggs in host nest. The cuckoo eggs are hatched by host birds if and only if they are neither detected nor destroyed. The newly hatched cuckoo chicks join the population of cuckoos and the search algorithm continues to find the global optimum. The searching for new host nests is done by random walks via the Lévy flight. Yang & Deb formulated some guidelines to implement this (a) Each cuckoo lays a single egg into a randomly chosen host nest from among n nests; (b) The nests with better quality eggs (implying better fitness value of the function concerned), if not detected, would be hatched to grow into the cuckoo
chicks, who would join the next generation; (c) The number of available host nests is fixed. The host can detect the alien egg with a probability p a [0, 1] and, if detected, it will either
fact that there are fewer parameters to be fine-tuned in Cuckoo Search than in PSO and GA. In fact, apart from the population size n, there is essentially one parameter p a .Cuckoo Search is
abandon the nest and build a new nest elsewhere or destroy the
more promising to both GA and PSO in terms of both efficiency and success rate of finding the global optima.
egg; (d) when generating new solutions one
(x ) ; t i
The
Lévy
flight
is
( x ) from the old t +1 i
performed
with
the
IV.
SIMULATIONS AND RESULT ANALYSIS
parameter 1 < β < 3 and thus;
xit +1 = xit + α ⊗ Levy(β )
(4)
The Lévy flight is a type of random walk following Mantegna’s algorithm for step size [5]. In general, a random walk is a Markov chain process where the next location only depends on the current location (the first term in the above equation) and the transition probability (the second term). The product ⊗ means entrywise multiplications. This entrywise product is similar to those used in PSO, but here the random walk via the Lévy flight is more efficient in exploring the search space as its step length is much longer in the long run. The Lévy flight essentially provides a random walk while the random step length is drawn from a Levy distribution
(a)
(b)
(c)
(d)
Lévy flight ∼ u = t − β
( 1< β < 3) (5) The Lévy flight is a type of random walk which has a power law step length distribution with a heavy tail. The basic steps of the Cuckoo Search can be summarized as the pseudo code shown in Fig. 2. begin Objective function Generate initial population (i = 1, 2, ..., n) of n host nests while (t ), replace j by the new solution; end A fraction p a of worse nests
(e)
(g)
(j)
(f)
(h)
(i)
(k)
are abandoned and new ones are built; Keep the best solutions (or nests with quality solutions); Rank the solutions and find the current best end while Postprocess results and visualization End
Figure 3. Simulation Result for the image of an accu-pressure
Figure 2: Pseudo code of the Cuckoo Search via Lévy flight
ball using different SR based Reconstruction Techniques.
Cuckoo search start very well, with comparable with those by other methods such as GA and PSO; and it converges more quickly as the number of iterations increase, and much better results are obtained. This is partly due to
(l)
(m)
In figure 3.(a) a color image by the SAMSUNG GT-S7562 camera is captured. In (b) resizing the previous image with ( 512 × 512 ) resolution is done. In (c-f), 4 low resolution images each with unique information is being represented. Then by using bicubic interpolation of one of the LR image is being done and the corresponding reconstructed image is shown in (g).Super resolution solution using IBP and HIBP algorithm is obtained from the experimental set up and the reconstructed images are represented in fig 3.(h) and (i) respectively. Fig 3.(j-m) shows the enlarged and cropped image of the reconstructed images using different reconstruction techniques. (j) is one of the LR image, (k) bicubic interpolation of one of the LR image, (h) Super resolution solution using IBP method and (i) super resolved HR image from HIBP algorithm. The blur parameter for PSF is defined using Gaussian distribution with standard deviation 0.25.The proposed algorithm is also validated with 9 more real time image sets which are presented in figure 4. In the figure 4 for all test image sets(a-i) the left most images are the LR images ,the middle ones are the reconstructed images using IBP algorithm and the right most images are the reconstructed images using proposed HIBP algorithm in all the cases. A. Image quality assessment of proposed algorithm In simulation, set of ten test images of size (128×128) are considered for validating the proposed algorithm. For confirmation of proposed algorithm mean square error, PSNR and SSIM index are calculated. The error indexes are calculated for reconstructing a HR image with respect to reference image using IBP and HIBP(IBP+COA) algorithm. The table 1 shows a comparision of different performance evaluation parameters of reconstructed images using both IBP and HIBP algorithms. The corresponding equation used for the calculation of MSE, PSNR and SSIM are as follows
1 MSE = MN
∑∑ [ M
N
Xˆ (i, j ) − X (i, j )
i =1 j =1
]
2
(6)
⎛ MAX X2 ⎞ PSNR = 10 log10 ⎜ ⎟ ⎝ MSE ⎠ Figure 4. Resultant images for nine test image sets (a-i) using different super-resolution based reconstruction techniques. The simulations for IBP and proposed HIBP algorithm are done with MATLAB R2009a configuration. This section discusses the experimental results with a real time image of an accu-pressure ball taken with SAMSUNG GT7562 mobile camera of 5MP resolution. The color photograph then converted to grayscale and resized to (512 × 512) pixels.
(7)
⎛ (2 μ μ + C1 )(2σ xy+C2 ) SSIM ( x, y ) = ⎜ 2 x 2 y ⎜ μ + μ +C σ 2 +σ 2 +C y 1 x y 2 ⎝ x
(
)(
Where, • •
MSE= Mean square error PSNR= Peak signal to noise ratio
)
⎞ ⎟ ⎟ ⎠
(8)
•
SSIM=Structural similarity index
Xˆ (i, j ), X (i, j ) - are the HIBP superresolution based reconstructed image and desired high resolution image respectively.
• • •
MAX X2 = maximum possible pixel value of the image. x , y = images patches of which SSIM is to be calculated μ x , μ y = mean average of x and y
•
σ xy =
•
C1 , C2 = constants here the value is taken
respectively, each for 4, 9 low Resolution images. We have ignored the 16 LR image case as in some cases while using IBP algorithm we get the non-converging conditions. 0.16 0.14 0.12 Mean square values
•
0.1 0.08 0.06 0.04 0.02 0 1
co-variance of x and y
as 0.07.
MSE of IBP algorithm with 4 LR images MSE of Hybrid IBP algorithm with 4 LR images MSE of IBP algorithm with 9 LR images MSE of Hybrid IBP algorithm with 9 LR images
2
3
4
5 6 SL. No of images
7
8
9
10
Figure 5. MSE comparison curves as a function of no. of LR input images (4,9) for both IBP and HIBP algorithm. .
Table 1 Comparison of MSE, PSNR and SSIM of both IBP and HIBP algorithm for 4, 9 and 16 LR images.
110 105
PSNR(dB)
100
PSNR of PSNR of PSNR of PSNR of
IBP algorithm with 4 LR images Hybrid IBP algorithm with 4 LR images IBP algorithm with 9 LR images Hybrid IBP algorithm with 9 LR images
95 90 85 80 75 1
2
3
4
5 6 SL. No of images
7
8
9
10
Figure 6. PSNR comparison curves as a function of no. of LR images (4,9) for both IBP and HIBP algorithm. 1 0.99
SSIM Index
0.98 0.97 SSIM of IBP algorithm SSIM of Hybrid IBP algorithm
0.96 0.95 0.94 0.93 0.92 1
2
3
4
5 6 SL.No of images
7
8
9
10
Figure 7. SSIM comparison curves as a function of no. of LR images (4,9) for both IBP and HIBP algorithm.
To visualize and compare the MSE, PSNR and SSIM index of both of the techniques we have plotted the above information with respect to no of images and are shown in figure 5,6,7
B.Result Analysis In figure 5. MSE comparison curves as a function of no. of LR input images (4,9) for both IBP and HIBP algorithm is presented. From the graph it is observed that the mean square error is remarkably decreasing as in the case of HIBP as compared to IBP. It is because of random walks used in COA instead of constant gradients as in IBP method. Hence we are getting a lower MSE in proposed HIBP algorithm. Figure 6. is a plot of PSNR comparison curves as a function of no. of LR images (4,9) for both IBP and HIBP algorithm. PSNR improves by a factor of around 4-5dB as in the case of HIBP than that of IBP algorithm. The reason behind it is that by using HIBP algorithm the blurring as well as the image degradation due to sensor noise is efficiently
removed as compared to IBP algorithm, results in improvement of peak signal to noise ratio. SSIM comparison curves as a function of no. of LR images (4,9) for both IBP and HIBP algorithm is plotted in figure 7.SSIM index is more close to 1 ie. The case of perfect reconstruction without loss of generality is in the case of HIBP algorithm base reconstruction of images. Ideally the quality of the reconstructed high resolution image is directly proportional to the number of input low resolution images. For each test image as we increase the no. of LR images at the input side of forward model the performance of both the technique increases. But for 16 LR case it can be observed that some of the test images failed to converge for IBP technique, whereas for the case of HIBP, it efficiently converges for all of the test images. Even the result of the 16 LR image case is better than their 9 LR image counterparts, it is possible because of the multi-modal nature of the COA.
The results obtained using cuckoo search is better than the gradient based method. It is because of two reasons; First, cuckoo optimization uses random walk unlike IBP where constant gradient is used, while constant gradient has more chance to lose the best solution, the use of levy distribution for random walks give better result. Second, IBP method depends on single initial - guess and iteratively updates it with a constant gradient whereas COA starts with multiple initialguesses then randomly selects one of them to start the iterations and after iteration it updates the initial guess using Levy distribution. In back projection technique the solution also depends on the quality of the initial guess taken; sometimes the algorithm fails to converge when the initial - guess is of poor quality, whereas for cuckoo search multiple initial-guesses are taken from previous IBP iterations. REFERENCES [1]
Table 2 Comparison of computational time. [2] NO OF LR IMAGES PROCESSED
IBP Algorithm
HIBP Algorithm
4
10.3657 sec
13.9298 sec
9
23.1017 sec
27.3547 sec
16
38.3629 sec
46.8796 sec
[3]
[4]
[5]
The time taken by HIBP algorithm is little more than that of IBP algorithm.Trade off between the time complexity and rate of convergence is being considered .In certain cases IBP is unable to converge where HIBP converges efficiently. V.
CONCLUSIONS
SR technique is shown to be feasible when sub-pixel shift information can be computed from grayscale image sequences and other sensor parameters such as blur and down-sampling operators are approximated. HIBP, an evolutionary based global optimization algorithm for SR technique along with IBP method is presented. The suggested algorithm performed well for both real and synthetic image, and has been shown, theoretically and practically, to converge to the best solution possible. .
[6]
M. Irani, and S. Peleg, “Improving resolution by image registration,” Graphical Models and Image Processing, vol. 53, pp.231-239, 1991. Park, Sung Cheol, Min Kyu Park, and Moon Gi Kang. “Super-resolution image reconstruction: a technical overview,” IEEE Signal Processing Magazine, vol.20, no.3, 2003, pp. 21-36. Siddique, Muhammad, “Performance Evaluation of Super -Resolution Reconstruction Algorithms Based On Linear Magnifications ", 2012. K. Ng, M., & K. Bose, N. ,"Mathematical analysis of super-resolution methodology,". IEEE Signal Processing Magazine,vol-20,issue 3 , May 2003, pp. 62-74. S. Farsiu, M. D. Robinson, M. Elad and M. Peyman, "Fast and Robust Multiframe Super-resolution," IEEE Trans. on Image Processing, , Oct 2004, pp. 1327-1344.
Yang, Xin-She, and Suash Deb, “Cuckoo search via Lévy flights,” NaBIC, 2009. [7] P. Civicioglu and E. Besdok, “A conception comparison of the cuckoo search, particle swarm optimization, differential evolution and artificial bee colony algorithms,” Artificial Intelligence Review, Springer Science, 6 July 2011. [8] Rajabioun, Ramin, "Cuckoo optimization algorithm," Applied Soft Computing vol.11 no.8,2011, pp. 5508-5518. [9] Michael, E., and Yacov, H. , “A Fast Super-Resolution Reconstruction Algorithm for pure Translational Motion and Common Space-invariant Blur,” IEEE Trans on Image Processing,no.8,2001. [10] Patel, V., Modi, K. C., Paunwala, N., & Patanaik, S. , ''Hybrid Approach for Single Image Super Resolution using ISEF and IBP,” International Conference on Communication System and Network Technologies. IEEE Computer Society,2011.
