Efficient Restoration and Enhancement of Super-resolved X-ray Images M. Dirk Robinson∗ , Sina Farsiu† , Joseph Y. Lo‡ , Cynthia A. Toth§ ∗ Ricoh
Innovations, Menlo Park, CA, Email:
[email protected] Dept. of Ophthalmology, Duke University Medical Center, Durham, NC, Email:
[email protected] ‡ Dept. of Radiology and Biomed. Engr., Duke University Medical Center, Durham, NC, Email:
[email protected] § Dept. of Ophthalmology and Biomed. Engr., Duke University Medical Center, Durham, NC, Email:
[email protected] †
Abstract— Our previous work demonstrates the ability to reconstruct a single higher resolution image from fusing a collection of multiple extremely low-dosage aliased X-ray images. While this computationally efficient method eliminates aliasing artifacts associated with undersampling, it does not address the problem of deblurring the reconstructed image. In this paper, we present a fast nonlinear deblurring algorithm, specifically designed to address the nonstationary noise associated with multiframe reconstructed images. The algorithm uses a combination of Fourier sharpening and wavelet denoising similar to the ForWarD algorithm. Experimental results on enhancing digital mammogram images attest to the effectiveness of the presented method.
a: 226 mAs
b: 11.3mAs
Fig. 1. Mammogram X-ray images from a phantom breast. (a) Normal dosage at 226mAs, showing aliasing artifacts (SNR ≈ 13 db). (b) Extremely low-dosage at 11.3 mAS used in the proposed multi-frame reconstruction scheme(SNR ≈ 3 db). The total dosage of using 15 of these frames (15 × 11.3 = 170mAs) is still less than the normal dosage.
I. I NTRODUCTION While relatively new, digital mammography is rapidly replacing film-based mammography for the screening and diagnosis of early carcinomas in women. These digital X-ray imaging systems contain aliasing artifacts due to the large pixel sizes required to ensure adequate measurement SNR. In our previous work [1], we proposed a novel method for digitally combining multiple low-dosage images, each containing spatial shifts, to produce a single high quality image at a resolution greater than the sampling rate of the detector. To minimize the amount of patient radiation, we acquired the multiple images in a much lower dosage than normal. Our recent results demonstrate the ability to resolve details at greater resolution than a native system, while reducing total radiation dosage. Multiframe resolution enhancement, commonly called super-resolution, has received much attention in recent years in the image processing community. We refer the interested reader to [2], [3], [4] for a broad review of recent algorithmic development in this area. Accurate subpixel translation estimates allow us to unwrap the aliasing artifacts, reconstructing the high resolution image. Our previous work addressed the challenging task of multiframe block matching and subpixel registration. In this paper, we propose an efficient method for both restoring the lost contrast (due to the X-ray system’s point spread function) and reducing the noise in the reconstructed image. The computational efficiency is important due to the size of the reconstructed images which range from 40 to 160 megapixels depending on the resolution enhancement factor [1]. Finally, the captured data has extremely low SNR. To
minimize total exposure, we must use very low dosages of illuminating radiation. For example, Fig. 1 compares a normal dosage X-ray image (computed SNR 1 ≈ 13 db) with the very low exposure images (computed SNR ≈ 3db) used in our multi-frame scheme. We propose a fast restoration and denoising method which is a multiframe variant of the ForWarD algorithm [5]. The algorithm’s efficiency stems from separating the multiframe deconvolution or restoration step from the wavelet-based denoising step allowing us to achieve nonlinear processing in a non-iterative fashion. We demonstrate that the effectiveness of the two-stage restoration and denoising algorithm allows us to provide high resolution, high contrast, and low noise images at very low radiation dosages. In Sect. II, we describe the X-ray imaging model and the review the fast method for multiframe registration and reconstruction. In Sect. III, we describe the multiframe variant of the ForWarD algorithm we employ to restore and denoise the reconstructed images. Section IV presents experimental results using this new approach and Section V outlines some future directions of this work. II. M ULTIFRAME R EGISTRATION AND R ECONSTRUCTION In our previous work [1], we addressed the two problems of multiframe block matching and subsequent subpixel registration. We briefly review it to introduce the new multiframe restoration and denoising approach and the corresponding s 1 In this work, the SNR was computed numerically as SNR= 20log 10 n , where s is half the pixel value between the black and white signal regions and n is the noise standard deviation in flat regions.
notations. We start this work assuming that the block matching has successfully produced a collection of image tiles which contain only subpixel motions. In our forward model, we assume that we have a collection of K + 1 image tiles raster scanned into vector format and denoted yk . These low-resolution input image tiles are blurry, translated, and downsampled versions of an unknown high resolution image tile denoted by x. The forward model producing these tiles is given by the linear model yk = DS(mk )Hx,
(1)
where the vector yk represents NL × NL samples of the observed image undersampled with respect to the unknown high resolution image x by a factor of d in each dimension. The vector x represents samples of the unknown dB×dB high resolution image tile lexicographically ordered as a (dB)2 × 1 vector. The warping operator S(mk ) of size (dB)2 × (dB)2 represents the sub-pixel spatial shifts between similar tiles in the captured images. The spatial shifting is described by the vector mk for the kth frame. In our model, we assume that these spatial shifts are continuous values in the range of [−d, d]. This corresponds to the range of sub-pixel motions in the captured images. The downsampling operator D of size B 2 ×(dB)2 captures the undersampling of the detector. Finally, ek of size NL2 × 1 represents the noise inherent in the analogto-digital conversion. For our purposes, we assume this noise to be uncorrelated zero-mean noise with standard deviation σ. The general problem of super-resolution is to combine K + 1 captured low-resolution images and estimate the high resolution image x. To accomplish this task, the algorithm must also estimated the collection of unknown motion vectors. Without loss of generality, we assume that the initial image y0 defines the coordinate system of the high resolution image and hence we only have to estimate the unknown motion parameters for the remaining K images. To simplify the presentation, we use an underline notation to represent the larger set of unknown motion vectors m = [m1 , . . . , mK ]T . As shown in [6], a natural method for simplifying the estimation problem is in defining the image z = Hx which is the unknown high resolution blurry image. In our previous work [7], [1], we addressed the problem of estimating this high resolution blurry image by registering the set of images in a joint fashion. The joint estimation was based on some prior information about the unknown high resolution image covariance Cx . With this signal covariance model, the Maximum a-posteriori (MAP) estimate of the blurry high resolution image and motion parameters is formulated as
by
¢−1 ¡ ˆ = Q(m) g(m), ˆ z ˆ + (HCx HT )−1
where Q(m) ˆ = g(m) ˆ =
(3)
1 X T ˆ k )DT DS(m ˆ k ), S (m σ2 k 1 X T ˆ k )DT y ˜k . S (m σ2 k
Plugging this form back into Eq. 2, we obtain the variable projections formulation [8] ¡ ¢−1 JV P (m) = gT (m) Q(m) ˆ + (HCx HT )−1 g(m). (4) In [7], we described how to efficiently maximize this variable projections cost function using efficient means for storing and inverting the matrices in the Fourier domain. Upon completion of the nonlinear optimization, we have estimates of the set of subpixel translations m. ˆ In [1], we reconstructed the blurry ˆ using Eq. 3. high resolution image z III. M ULTIFRAME I MAGE R ESTORATION AND WAVELET D ENOISING The next step in the super-resolution process is restoring the contrast lost due to the blurring inherent to the imaging system. To achieve this, we derive a multiframe version of the fast ForWarD algorithm [5]. The ForWarD algorithm combines a Fourier-based regularized deconvolution algorithm with a wavelet-based denoising post processing step. The original algorithm applies to deconvolution and sharpening of a single image, while a very recent version addresses the multi-frame deblurring problem [9]. Application of the basic ForWarD algorithm assumes stationarity of the noise. In this section, we derive an extension of this approach to handle the nonstationarity inherent to the reconstructed multiframe image. The ForWarD algorithm first applies a Fourier-based regularized deconvolution filter, such as the Wiener filter [10]. In the case of the multiframe restoration, we apply a multiframe version of the Wiener filter. The multiframe Wiener estimate of the restored image is a variant of Eq. 3 given by ¡ ¢−1 T ˆ = HT Q(m)H x ˆ + λC−1 H g(m). ˆ (5) x
We compute the filter in the Fourier domain to simplify the computational complexity using similar tricks to those found in [7]. In Eq. 5, the term λ is used to control the influence of the prior information covariance matrix during reconstruction. The standard Fourier-based Wiener filter calls for λ = 1. In keeping with the spirit of the ForWarD deconvolution approach, we use values of λ < 1 to reduce the influence of this regularization to enhance contrast prior to wavelet 1 X 2 T T −1 k˜ yk −DS(mk )zk +z (HCx H ) z. denoising [5]. JM AP (z, m) = 2 σ The second step in the ForWarD algorithm performs a k ˆ for (2) discrete wavelet decomposition (DWT) of the estimate x We perform this joint registration and reconstruction efficiently either hard or soft thresholding of the coefficients. Similar to by formulating the problem as one of variable projections [8]. the original ForWarD algorithm, we perform a redundant (no To do so, we note that the MAP estimate of the high resolution downsampling) wavelet transform to the image represented by ˆ producing a set of scale coefficient images si and blurry image, given estimates of the set of translations, is given the vector x
a set of wavelet coefficient images wl . Convolving the input ˆ with a set of scaling Φi and wavelet Ψl functions image x produces the set of coefficient images. We achieve wavelet-based denoising by thresholding the wavelet coefficients followed by an inverse DWT to reconˇ to denote the denoised struct the denoised image. We use x image. As recommended in [5], we first apply a hard threshold as a means for coarse noise removal. The hard thresholding changes the wavelet (and scaling) coefficients according to ½ {wl }a , |{wl }a | > γ{σl }a ˜ l }a = , (6) {w 0, |{wl }a | ≤ γ{σl }a where γ is an input threshold and {σl }a represents the noise standard deviation at the lth wavelet space for the ath pixel. Then, the coarsely denoised image is reconstructed using the ˜ to create the image x ˜ . Next, the thresholded coefficients w DWT is applied using a different wavelet set to produce the ˜ 0 l . This second wavelet function is also set of coefficients w applied to the original image producing the set of coefficients ˜0 l , wl0 . Using the estimates of local signal strength given by w we apply the soft thresholding ˇ l }a = {wl0 }a {w
˜ 0 l }a |2 |{w . ˜ 0 l }a |2 + {σ 2 }a |{w l
(7)
ˇ. We apply the inverse DWT to obtain the denoised image x Unlike the traditional ForWarD algorithm, the multiframe ˆ has a random error field estimate of the deblurred image x which is not stationary. The single frame ForWarD algorithm requires only a single noise power σl2 for every pixel at a particular wavelet/scale space for Eqs. 6 and 7. In the ˆ has multiframe case, however, the colored noise field of x the covariance matrix ¡ T ¢−1 T ¡ T ¢ −1 −1 H Q(m)H ˆ + λC−1 H Q(m)H ˆ H Q(m)H+λC ˆ , x x where the spatially varying noise power depends on the collection of motion vectors m. That is, even though the noise fields nk are stationary, different pixels in the high ˆ have varying amounts of resolution reconstructed image x data. For example, suppose we combine 6 frames to enhance the resolution by a factor of d = 2 in both dimensions. If the set of motions were [0, 0], [0, 0], [0, 0], [ 21 , 0], [0, 12 ], [ 12 , 12 ], ˆ then some of the pixels in the high resolution image x would represent the average of three captured images, whereas other pixels represent only a single captured image. The spatial variation, however, is periodic every d pixels in both dimensions. To address this spatial variability, we estimate d2 noise powers corresponding to the d×d grid of high resolution pixel locations. IV. E XPERIMENT Our experimental imaging system is based on a Mammomat NovationDR digital mammography system. The system uses a stationary selenium-based detector having 85 µm pixels. Pixels with this size correspond to a Nyquist sampling rate of 5.6 line pairs per millimeter (lp/mm). We use a CIRS breast phantom to test out imaging algorithm. We introduce shifts in the image
by two methods. First, we allow the illumination tube to rotate by ± 1 degree. Second, we manually move the breast phantom to introduce motion into the system. The manual motion is completely uncontrolled. We acquire 15 frames at the low dosage level of 11.3 mAs at 28 kV tube voltage. As a point of reference, we also acquire a single frame at a more normal dosage of 226 mAs at 28 kV tube voltage (Fig. 1). We focus on the results of the test resolution chart to explore the contrast performance of the multiframe imaging system. We apply our algorithm to 100 × 100 pixel tiles in the captured image to estimate 400×400 pixel high resolution images (enhancement d = 4). We modeled our system point spread function (PSF) as a Gaussian shaped function having a standard deviation width of 1.5 pixels. We used Daubechies 2 tap and 6 tap filters respectively for the hard thresholding of Eq. 6 and soft thresholding of Eq. 7, respectively. Figure 2 gives a visual example of the performance of the restoration and denoising for a small portion of the resolution chart embedded in the breast phantom. Note that one of these the low-dosage (11.3 mAs) images is shown in Fig. 1.b, which shows the low SNR as well as the aliasing present due to undersampling. The image on the top left (a) shows ˆ after applying the the estimate of the high resolution image z multiframe registration and reconstruction algorithm of [1]. The aliasing artifacts are eliminated and contrast is restored above the Nyquist rate. The image is, however, still noisy and low contrast. The top right image (b) shows the result of applying the single frame ForWarD algorithm to the multiframe reconstructed image. We observe that the algorithm restores some of the contrast but incorrectly preserves some of the nonstationary noise as signal content. The bottom left ˆ after applying the multiframe image (c) shows the result x Wiener sharpening filter of Eq. 5. The image shows improved contrast with increased sharpness, but also amplified noise. The bottom right image (d) shows the result after applying the two-stage wavelet-based denoising algorithm of Eqs. 6 and 7. The resulting image preserves the contrast around the bar chart signal locations while eliminating most of the noise in the signal-free portions of the image. Figure 3 shows a scatter plot of the set of estimated motions ˆ k } on the high resolution image grid. The grid reflects {m the number of image samples per pixel in the reconstructed image. The example shows some pixel estimates combining as many as four measurements, whereas other pixels have no measurements underscoring the need for the multiframe denoising approach. To get an another perspective on the effects of the multiframe restoration and denoising, we plot slices through the resolution test chart region in Fig. 4. The top curve shows the slice through the average of the captured images containing only subpixel motion. The slice shows some aliasing as well as lost contrast for the bars about the Nyquist sampling rate of 5.6 lp/mm. The second graph shows a slice through the multiˆ. The reconstruction eliminates the frame reconstructed image z aliasing artifacts and effectively restores contrast beyond the sampling rate of the detector. The signal strength, however,
5 lp/mm
6 lp/mm
7 lp/mm
8 lp/mm
9 lp/mm
10 lp/mm
11 lp/mm
12 lp/mm
Multiframe Average
ˆ a: z
100
200
300
400
100
200
300
400
100
200
300
400
100
200
300
400
500
600
b: Basic ForWarD Multiframe Reconstruction 500
600
Multiframe Restoration
ˆ c: x
500
600
ˇ d: x
ˆ. (b) ForWarD [5] (c) Multiframe Fig. 2. (a) Multiframe reconstructed image z ˆ . (d) The multiframe wavelet denoised image x ˇ. restored image x
Multiframe Denoised 500
600
Fig. 4. The four curves show slices through the horizontal resolution bar charts. The Nyquist rate of the system corresponds to 5.6 lp/mm. The top slice shows a slice through an interpolated average of the captured images ˆ shows showing aliasing artifacts and lost contrast. The second slice through z enhanced resolution beyond the Nyquist rate, but poor contrast. The third slice ˆ shows restored contrast but with noise amplification. The bottom through x ˇ shows contrast preservation with significantly less noise. slice through x
Translation Estimates
4
3
mˆ 2 2
1 1
2
mˆ 1
3
4
ˆ k } on the Fig. 3. The scatter plot shows the set of estimated motions {m high resolution image grid. The grid locations correspond to measurements at individual pixels in the high resolution image estimate. Some high resolution pixels have as many as four measurements whereas others have none underscoring the need for a multiframe wavelet denoising.
above 8 lp/mm is very weak due to the blurry inherent to the imaging system. The third slice is from the multiframe restoration result (ˆ x). The sharping restores contrast out to the 12 lp/mm, more than twice the Nyquist rate, at the expense of ˇ after multiframe noise amplification. The bottom slice shows x wavelet denoising. We observe that the contrast is preserved while significantly eliminating the noise in between the bar chart signal regions. V. C ONCLUSION In this paper, we have proposed a novel method for restoring and denoising super-resolved low dosage X-ray images in a fast multiframe variant of the ForWarD algorithm of [5]. The proposed Fourier multiframe restoration and wavelet denoising algorithm provides high contrast super-resolved images while improving the extremely poor SNR of low-dosage images. The experimental results confirm that multiframe imaging can provide an alternative in the SNR versus resolution tradeoff for digital mammography. We note that our restoration algorithm
can be easily modified to further enhance the quality of other Shift-and-Add based super-resolution techniques [2]. Design of future X-ray imaging systems would benefit from a systematic analysis of the resolution and SNR required for mammographic screening and diagnosis. In the future, we explore the fundamental tradeoffs between radiation exposure, number of frames, and reconstruction performance. R EFERENCES [1] D. Robinson, S. Farsiu, J. Lo, P. Milanfar, and C. Toth, “Efficient multiframe registration of aliased X-ray images,” Proc. of the 41th Asilomar Conference on Signals, Systems, and Computers, Nov. 2007. [2] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” International Journal of Imaging Systems and Technology, vol. 14, no. 2, pp. 47–57, October 2004. [3] S. Borman and R. L. Stevenson, “Super-resolution from image sequences - a review,” in Proc. of the 1998 Midwest Symposium on Circuits and Systems, vol. 5, Apr. 1998. [4] S. Park, M. Park, and M. G. Kang, “Super-resolution image reconstruction, a technical overview,” IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 21–36, May 2003. [5] R. Neelamani, H. Choi, and R. Baraniuk, “Forward: Fourier-wavelet regularized deconvolution for ill-conditioned systems,” IEEE Transactions on Image Processing, vol. 52, no. 2, pp. 418–433, February 2004. [6] D. Robinson and P. Milanfar, “Statistical performance analysis of superresolution,” IEEE Transactions on Image Processing, vol. 15, June 2006. [7] D. Robinson, S. Farsiu, and P. Milanfar, “Optimal registation of aliased images using variable projection with applications to super-resolution,” The Computer Journal, vol. 51, doi:10.1093/comjnl/bxm007, 2008. [8] G. Golub and V. Pereyra, “Separable nonlinear least squares: the variable projection method and its applications,” Institute of Physics Inverse Problems, vol. 19, pp. R1–R26, 2003. [9] R. Neelamani, M. Deffenbaugh, and R. Baraniuk, “Texas Two-Step: A Framework for Optimal Multi-Input Single-Output Deconvolution,” IEEE TIP, vol. 16, no. 11, pp. 2752–2765, Nov. 2007. [10] A. K. Jain, Fundamentals of Digital Image Processing, 1st ed. Englewood Cliffs, New Jersey: Prentice Hall, 1989.
