image is regarded as convolution result between latent image and point spread function (PSF). Image deconvolution is a helpful method to reverse blurring.
2015 8th International Conference on BioMedical Engineering and Informatics (BMEI 2015)
Inter-Level and Intra-Level Deconvolution Based Image Deblurring Algorithm for Wide Field Microscopy Yanzhi Ding
Iju Park
School of Communication and Information Engineering Chongqing University of Posts and Telecommunications Chongqing, China
Department of Biomedical Automation Advanced Technology Incorporated Company Incheon, South Korea
Xuenan Cui, Van Huan Nguyen, Hakil Kim*, Trung Dung Do, Wei Li School of Information and Communication Engineering Inha University Incheon, South Korea Abstract—This paper proposes an inter-level and intra-level deconvolution based image deblurring algorithm (ILILD) for microscopic images. Pyramid structure is used, and inter-level deconvolution is applied to estimate latent image from coarse level to fine level. The inter-level algorithm is based on total variation regularized Richardson-Lucy scheme, which can estimate latent image with artifacts suppressed. After inter-level deconvolution, intra-level deconvolution is applied. In each pyramid level of image, the residual deconvolution is done as the intra-level deconvolution scheme to recover image edges and details furtherly. Experiments show that ILILD algorithm can estimate latent images in less time and the results have better peak signal to noise ratio, higher image entropies and few artifacts. Keywords-wide field microscopic image; image deconvolution; ILILD; artifacts
I.
INTRODUCTION
Wide field microscopy is extensively used to visualize cellular structures and properties of biological specimens. Unlike confocal microscopy, wide field microscopy is prominent for its high instantaneity, little photo toxicity and low expense. In wide field microscopy imaging system, pure white light is emitted by a mercury lamp, and some optical components are used to select light with particular wavelength. The desired light is directed to specimen and then captured by camera. Because of the finite lens aperture, there is an annoying phenomenon of diffraction ring, which is so called Airy pattern. Images of wide field microscopy are blurred inevitably by the interference of both Airy pattern and some unfocused light noise. To get high quality images, image deblurring should be done [1]. For wide field microscopic image, the observed blurred image is regarded as convolution result between latent image and point spread function (PSF). Image deconvolution is a
helpful method to reverse blurring. Several image deconvolution algorithms have been proposed, which could be divided into two categories according to the prior knowledge of PSF: blind deconvolution algorithms and non-blind deconvolution algorithms [1, 2]. Image deconvolution algorithms are belonging to blind deconvolution algorithms when PSF is unknown previously. Suo et al. proposed maximum posterior (MAP) based multiobservation blind deconvolution, it estimates latent image by taking advantages of multiple blurred images [3]. In [4], coarse to fine method combing hybrid graph Laplacian regularization term is applied to solve the blind deconvolution problem. Michaeli et al. got derivation information from ideal patch recurrence, and used it for PSF estimation [5]. Sun et al. put up with an edge-based blind deconvolution algorithm [6]. Image deconvolution algorithms are classified as non-blind algorithm if PSF is known previously, and latent images can be estimated with the help of known PSF. A mount of non-blind algorithms have been proposed such as Wiener filter, constrained least squares, and Richardson-Lucy deconvolution. In [7], Wiener filter-based algorithms are used to build the nonblind deconvolution algorithm. Zhou et al. adopted constrained least squares filter to image restoration [8]. Richardson come up with Richardson-Lucy (RL) algorithm to do image deconvolution [9]. Among these algorithms, thanks to the adaption of Poisson noise, RL algorithm has been one of the most prevalent image deblurring algorithm for wide field microscopic images. However, RL has a problem of nonconvergence, and artifacts increased with the increase of iterations. Even when the iteration times is set as infinity, the result image will be noise only. This paper focuses on non-blind image deconvolution algorithm. Inter-level and intra-level deconvolution based image deblurring algorithm (ILILD) is proposed to estimate high quality latent images with known PSF. In the remainder of
This work was supported by Korea Institute of Energy Technology Evaluation and Planning (KETEP) research grant.(20151120100040)
978-1-5090-0022-7/15/$31.00 ©2015 IEEE
90
the paper, theoretical framework of ILILD algorithm is introduced in detail. Then experimental results of both synthetic blurred images and real wide field microscopic images are presented. Finally, conclusions are drawn. II.
PROPOSED INTER-LEVEL AND INTRA-LEVEL DECONVOLUTION BASED ALGORITHM
The flowchart of ILILD algorithm is shown as Fig. 1. Pyramid structures of blurred image and PSF are built. Interlevel deconvolution is adopted to estimate latent image progressively. Then intra-level deconvolution scheme is used to reveal edges and details information within each level of the pyramid structure. A.
Pyramid Structure
{b n }nN=1 and {k n }nN=1 are N-level pyramid structures of blurred image b and PSF k respectively. The pyramid structures are built using bicubic downsampling, and scale factors are equal to 2 . The deblurring algorithm is started with the lowest (coarsest) level b1 .
Estimated latent images with different number of pyramid levels are shown in Fig. 2.When number of levels is greater than three, the computing time increases and the estimated latent image is similar to the result of three-level pyramid structure. Therefore, this paper applies three-level pyramid structure experimentally. For the coarse level, images with fine edges and few ring artifacts can be gotten as PSF is small [10]. For middle level and fine level, the upsampled result from previous level is used by inter-level deconvolution. Thanks to the coarse-to-fine
approach, precious latent image can be estimated progressively. And within each level of the pyramid structure, intra-level deconvolution scheme is done to reveal enormous details information. B.
Richardson-Lucy (RL) Algorithm For microscopy, blurred image is modeled with latent image, PSF and noise, which can be described as following:
b = n(l ⊗ k ),
(1)
where b is the blurred image, l is latent image, k is PSF, ⊗ denotes convolution operator, and n (⋅) represents noise distribution. Applying Bayesian probabilistic model, the maximum aposteriori (MAP) description of latent image l is
l = arg max P(l | b) ∝ P(b | l )P(l ).
(2)
l
For RL algorithm [9], the blurred image b is viewed as Poisson distribution with mean (l ⊗ k )(X) P(b | l ) = ∏ X
(l ⊗ k )(X)b ( X ) exp( −(l ⊗ k )(X)) , b(X)!
where X is the pixel index. The solution of l that maximizing (2) is equal to the solution that minimizing the negative logarithmic function of (3), and the problem can be transformed into cost minimization matter. The cost function can be written as:
J RL (l ) = -ln P(b | l )
= ¦ X (l ⊗ k )(X) −b(X)lnP(l ⊗ k )(X) + ln(b(X)!).
Assuming the PSF is normalized, namely
{b } n
N n =1
,
{k } n
(3)
¦
X
(4)
k (X) = 1 .
N n =1
n = 0, l 0 = b1
f n +1 = TVRL ( l n ,k n+1 )
ǻb = b n+1 − f n+1 d = RL ( ǻb )
(a) blurred image
( )
l n = upsample lln
(b) two-level pyramid 34 (s)
(c) three-level pyramid 39 (s)
(e) five-level pyramid 48 (s)
(f) six-level pyramid 51 (s)
n +1 ln = f n +1 + d , n + +
l = llN
(d) four-level pyramid 44 (s)
Figure 2. Deblurred images and computing time for different levels
Figure 1. Flowchart of ILILD algorithm
91
Equation (4) is convex, so minimization of cost function can be calculated by differentiating (4) with respect to l and set value of derivative as zero. Using i to represent number of iterations, and the iteration scheme of RL algorithm is [9] lˆi +1 (X) = lˆi (X) ⋅ ( k m ⊗
b )(X), ( k ⊗ lˆi )
arbitrary term, which is helpful to suppress artifacts while preserving edges and details information. To get the optimized solution of the regularized cost function, Green’s one step late algorithm [11] is usually applied, and the iteration scheme can be gotten as:
(5) lˆi +1 (X) =
where k m (X) = k (-X) is the mirrored PSF, and lˆi (X) means the deblurred image with i times iteration.
lˆi (X) ⋅ ( k m ⊗ 1+λ (
b )(X) ( k ⊗ lˆi )
∂ J R )(lˆi ) ∂x
.
(7)
C. Inter-Level Deconvolution The basic idea of inter-level deconvolution is to recover latent image from coarse to fine level. The estimated result of a level is upsampled by bicubic upsampling and used for next level deconvolution initialization.
Total variation regularization term is known for its ability to preserve image details and edges while suppressing noise [12]. Total variation regularized Richardson-Lucy algorithm (TVRL) is used for inter-level deconvolution.
Estimated latent images by different algorithms are shown in Fig. 3. As can be observed in Fig. 3(b), the result of RL algorithm is influenced by amount of artifacts, especially ring artifacts. The artifacts are introduced unavoidably because of the ill-posed problem of inverse solution. Noises tend to be amplified with the increase of iteration times, and even when i → ∞ , the estimation of latent image consists of noise only. To deal with this problem, regularization term J R (l ) should be added to cost function J (l ) :
is
J (l ) = J RL (l ) + λ J R (l ).
(6)
λ is the weight of regularization term, J R (l ) can be
The cost term defined by total variation regularization term J TV = λTV ¦ X ∇l (X) = λTV ¦ X (Δ Xh l )2 + (Δ Xv l ) 2 ,
(8)
and the whole cost function of TVRL can be written as J = J RL + J TV = ¦ X (l ⊗ k )(X) −b(X)lnP (l ⊗ k )(X) + ln(b(X)!) + λTV ¦ X ( Δ Xh l ) 2 + ( Δ Xv l ) 2 .
(9)
where λTV is act as the weight of regularization, Δ hX l is fist order horizontal difference of latent image and Δ vX l is first order vertical difference of latent image. As shown in (8), total variation regularization term is useful to restrain false edge, especially ring artifacts. Applying Green’s one step late algorithm to (9), then the iterative scheme for inter-level deconvolution is:
lˆi +1 (X) =
(a) blurred image
(c) TVRL algorithm
(b) RL algorithm [9]
(d) ILILD algorithm
Figure 3. Estimated latent images with different algorithms
lˆi (X) ⋅ (k m ⊗
b
)(X) ( k ⊗ lˆi ) . ∇lˆi (X) 1 − λTV div( ) ∇lˆi (X )
(10)
The upsampled result l n from previous level deblurring is used as input image for the inter-level deconvolution, and rough latent image f n +1 is estimated. The value of λTV plays a significant role in the regularized iterative scheme, and it should neither be too small nor too large. If λTV is too small, the regularization will have little effect, and artifacts will not be suppressed efficiently. If λTV is too big, the result will be dominated by total variation terms, and denominator of (10) would near to zero, or even negative, which should be avoid. In our experiment λTV is set as 0.03, and ring artifacts are damped as is shown in Fig. 3(c). D. Intra-Level Deconvolution To recover more details information and suppress artifacts furtherly, residual information is used for intra-level deconvolution. The example of contrast maximized residual image and details image is shown in Fig. 4(a) and Fig. 4(b).
92
lˆ = f + d .
(12)
Details information can be gotten from residual image. Relationship between residual image and details information is Δb = d ⊗ k.
(13)
As can be observed, it is also a deconvolution problem to get details image from residual image. For intra-level image deconvolution, standard RL algorithm is applied to recover details image from residual image. The iterative scheme of details image is (a) residual image
(b) details image
d j+1 (X) = d j (X) ⋅ (k m ⊗
Figure 4. Residual image and details image
Residual image Δb is defined as: Δb = b − f ⊗ k,
(11)
where b is blurred image, f is rough latent image estimated from inter-level deconvolution, k is PSF. In the ideal situation of wide field microscopic image, the result of convolution between latent image and PSF should be closed to blurred image, namely Δb should approach to full zero. In practice, whereas Δb has some values, which contains details information. Lost details image d should be added into the estimated latent image and residual image should be damped. The modified estimated latent image is
(a) ground truth
Δb )(X), (k ⊗ d j )
where j is the number of intra-level iterations, details image is initialized as residual image, namely, d 0 = Δb . Taking both efficiency and accuracy into consideration, we set the number of intra-leve iterations as 2, so estimated residual image d = d 2 . III.
EXPERIMENTAL RESULTS
To demonstrate the efficiency of ILILD algorithm, both synthetic images and real wide field microscopic images are used for experiments. A.
Synthetic blurred images For this part, confocal microscopy image is used as ground truth image. The sharp image is blurred by synthetic wide field microscopic PSF and Poison noise. The PSF is generated
(b) blurred and noisy image
(c) synthetic PSF
PSNR : 27.30
(d) RL algorithm [9] PSNR : 27.99
(14)
(e) Fergus’s algorithm [14] PSNR : 26.10 Figure 5. Synthetic image deblurring
93
(f) ILILD PSNR : 29.52
following Gibson and Lanni model [13]. As shown in Fig. 5, ILILD algorithm is compared with standard RL algorithm and Fergus’s algorithm [14] in terms of peak signal noise ratio (PSNR).
PSNR = 10 ln(
(2n − 1) 2 ), MSE
(15)
where n is the number of pixels, i means the index of pixel, and MSE denotes mean square error:
MSE =
1 n ˆ ¦ li − li n i =1
(
)
2
.
(16)
As seen in Fig. 5(d), the edge artifacts are amplified by RL algorithm. The estimated latent image of Fergus’s algorithm is presented in Fig. 5(e). Fergus’s algorithm can recover edges information well, but the amplified noises make the estimated latent image more blurred. In contrast, as shown in Fig. 5(f), ILILD algorithm recovers edges and details information with noise reduced. And the evaluation in terms of PSNR also shows that ILILD algorithm has a convincing performance
(a) blurred image
(b) RL algorihtm [9]
dealing with microscopic image blurring. B.
Real Wide Field Microscopy images In this part, we use original wide field microscopic images offered by Advanced Technologies Incorporated (ATI) Company. PSF calculated by two-phase kernel estimation algorithm [15] is used by different image deblurring algorithms. Some estimated latent images are presented in Fig. 6. And as shown in Fig.7, entropies of images [16] are calculated. From Fig. 6(b) we can observe that the results of RL algorithm always are influenced by the amplified artifacts, including ring artifacts and edge artifacts. The results of Fergus’s algorithm are shown in Fig.6(c). Fergus’s algorithm can suppress ring artifacts and edge artifacts well, but the background of results are polluted by a mount of noises for some cases. In Fig. 6(d), the estimated latent images of ILILD algorithm restore more edges and details information with noised suppressed. Compared with RL algorithm and Fergus’s algorithm, the results of ILILD algorithms show better quality visually.
(c) Fergus’s algorithm [14]
Figure 6. Real wide field microscopy images deblurring
94
(d) ILILD algorithm
IV.
CONCLUSION
This paper proposes an inter-level and intra-level deconvolution based image deblurring algorithm for wide field microscopic images. TVRL algorithm is used as inter-level deconvolution, which is helpful to estimate latent image from coarse level to fine level. Residual deconvolution is applied as intra-level deconvolution, which is able to recover abundant edges and details information. The results of both synthetic blurred images and real wide filed microscopic images show that ILILD algorithm is a fast algorithm to estimate high quality latent images with artifacts reduced. REFERENCES [1]
Figure 7. Entropies comparison
For real wide field microscopic images, there is no ground truth images for evaluation. To assess deblurring algorithms quantitatively, entropies H of images are used.
H = −¦1 p(i) ln p (i), n
(17)
where n is the number of pixels, i indicates the index of pixel,
p(i) =
li
¦
n
l
.
(18)
[2] [3]
[4]
i =1 i
Entropy is an indicator to describe the amount of image information, and it can reflect quality of images. Higher entropy value means more information contained in images and better image quality. Experiments have done with 5 datasets, every dataset contains 20 wide field microscopic images of the same kind. The resulting entropies are shown as Fig. 7. The results of RL algorithm always have little improvement. For Fergus’s algorithm, the performance is unstable. The performance of Fergus’s algorithm depends on the kind of database. As ILILD algorithm, the results always have higher entropies, which indicates more details information is contained. Computing time is another important guideline to assess the performance of different algorithms. Computing time for images of different sizes is shown is in Fig. 8. As we can see, Fergus’s algorithm is the most time-consuming and ILILD algorithm is the fastest. Because of the use of pyramid structure, ILILD algorithm has more significant advantages for large images.
[5] [6]
[7]
[8]
[9]
[10]
[11] [12]
[13]
[14]
[15] [16]
Figure 8. Computing time comparison
95
M. Keuper, T. Schmidt, M. Temerinac-Ott, J. Padeken, P. Heun, O. Ronneberger, and T. Brox, “Blind deconvolution of widefield fluorescence microscopic data by regularization of the optical transfer function (OTF),” in IEEE International Conference on Computer Vision and Pattern Recognition, 2013, pp. 2179-2186. J. B. Sibarita, “Deconvolution microscopy,” Springer Microscopy Techniques, Berlin Heidelberg, pp. 201-243, 2005. Y. Suo, M. Dao, T. Tran, U. Srinivas, and V. Monga, “Hierarchical sparse modeling using spike and slab priors,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 3103-3107 X. Liu, D. Zhai, D. Zhao, G. Zhai, and W. Gao, “Progressive image denoising through hybrid graph laplacian regularization: a unified framework,” in IEEE International Conference on Image Processing, 2014, pp. 1491-1503. T. Michaeli, and I. Michal, “Blind deblurring using internal patch recurrence,” Springer ECCV, Berlin Heidelberg, pp. 783-798, 2014. L. Sun, S. Cho, J. Wang, and J. Hays, “Edge-based blur kernel estimation using patch priors,” in IEEE International Conference on Computer Computational Photography, 2013, pp. 1-8. F. Xue, F. Luisier, and T. Blu, “Multi-wiener SURE-LET deconvolution,” IEEE Transactions on Image Processing, vol. 22, pp. 1954-1968, 2013. X. Zhou, R.Molina, F. Zhou and A. K. Katsaggelos, “Fast iteratively reweighted least squares for l p regularized image deconvolution and reconstruction,” in IEEE International Conference on Image Processing, 2014, pp. 1783-1787 W. H. Richardson,“Bayesian-based iterative method of image restoration,” Journal of the Optical Society of America, vol. 62, pp. 5559, 1972. N. Dey, L. BlanćFeraud, C. Zimmer, P. Roux, Z. Kam, J. C. Olivó Marin, and J. Zerubia, “Richardson–Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution,” Journal of Microscopy Research and Technique,vol. 69, pp. 260-266, 2006. P. J. Green, “On use of the EM for penalized likelihood estimation,” Journal of the Royal Statistical Society, vol. 52, pp. 443-452, 1990. H. Kirshner, F. Aguet, D. Sage, and M.Unser, “3́D PSF fitting for fluorescence microscopy: implementation and localization application,” Journal of Microscopy, vol. 249, pp. 13-25, 2013. A. Griffa, N. Garin, and D. Sage. “Comparison of deconvolution software in 3D microscopy: A user point of view Part 1,” Journal of GIT Imaging & Microscopy, vol. 1, pp. 43-45,2010. D. Krishnan, and R. Fergus. "Fast image deconvolution using hyperLaplacian priors." Advances in Neural Information Processing Systems. 2009, pp. 1033-1041. L. Xu, and J. Jiaya. "Two-phase kernel estimation for robust motion deblurring," Springer ECCV, Berlin Heidelberg, pp. 157-170, 2010. D. Y. Tsai, Y. Lee, and E. Matsuyama. “Information entropy measure for evaluation of image quality,” Journal of Digital Imaging , vol. 3, pp. 338-347,2008.
