IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005
567
Specification of the Observation Model for Regularized Image Up-Sampling Hussein A. Aly, Member, IEEE, and Eric Dubois, Fellow, IEEE
Abstract—Regularization is one of the most promising methods for image up-sampling, which is an ill-posed inverse problem. A key element of such a regularization approach is the observation model relating the observed lower resolution (LR) image to the desired higher resolution (HR) up-sampled image, used in the data-fidelity term of the regularization cost function. This paper presents an algorithm to determine this observation model based on a model of the physical acquisition process for the LR image, and the ideal acquisition process for the desired HR image, both from the same underlying continuous image. The method is illustrated with typical scenarios corresponding to LR and HR cameras modeled by either Gaussian or rectangular apertures. Experiments with some regularized image up-samplers demonstrate the importance of using the correct, adapted observation model as determined by our algorithm. Index Terms—Camera aperture, data fidelity, image up-sampling, interpolation, multidimensional signal processing, observation model, power spectral density (PSD), super-resolution.
I. INTRODUCTION
D
IGITAL image magnification yielding higher perceived resolution is of great interest for many applications such as law enforcement and surveillance, standards conversions for broadcasting, printing, and aerial and satellite image zooming. is acquired by a In such applications, a continuous image physical camera to produce a lower resolution (LR) sampled (i.e., lower resolution than desired). The physical image camera is modeled as a continuous-space(-time) filter followed by sampling on a lower density sampling lattice. It is desired to obtain a higher resolution (HR) version of that image sampled on a denser sampling lattice. In general, natural images are not bandlimited and so direct sampling will result in objectionable aliasing, even in the high-resolution images. Thus, the continuous image should undergo a continuous-space prefiltering prior to sampling on the HR lattice (which should not be an ideal lowpass filter that introduces objectionable ringing). If we know the display aperture for the HR image, then the best prefilter for that display can be determined [1], but, in general, we can specify any desired prefilter, whether physically realizable or not, to obtain the ideal HR image. Our proposed method
Fig. 1.
Scenario for image up-sampling.
will allow us to estimate the HR image that would have been obtained with this desired theoretical camera. The scenario is shown in Fig. 1. Many solution methods for the image magnification problem with a broad range of quality exist in the literature. In this paper, we categorize these solution methods into model-based and nonmodel-based ones. Nonmodel-based methods use linear or nonlinear (adaptive) interpolation. Linear interpolators range from straightforward zero-order-hold (pixel repeat), bilinear, or bicubic interpolation to embedding in spline kernel spaces [2]. Nonlinear methods adapt the interpolation method used according to the edges in the given LR image, and, hence, are called edge-directed interpolation [3]–[6]. Model-based image up-sampling methods rely on sophisticated regularization methods. An ill-posed inverse problem, image up-sampling [7] when approached in a regularization-based framework would generally be formulated as an optimization problem. that Two cost functions are set: a data fidelity term penalizes inconsistency between the estimated HR image and the observed LR image and a regularizer (a priori constraints) . Thus, the problem of obtaining an HR image estimate is given by (1)
Manuscript received February 4, 2004; revised June 11, 2004. This work was supported in part by the Ministry of Defence, Egypt, and in part by the Natural Sciences and Engineering Research Council of Canada. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Philippe Salembier. H. A. Aly is with the Ministry of Defence, Cairo, Egypt (e-mail:
[email protected]). E. Dubois is with the School of Information Technology and Engineering (SITE), University of Ottawa, Ottawa, ON, K1N 6N5, Canada (e-mail:
[email protected]). Digital Object Identifier 10.1109/TIP.2005.846019
where is a regularization parameter that controls the tradeoff and . Our analyses and contributions in this paper between serve to support the regularization-based methods which currently are the most promising methods for solving this problem. and spatiotemporal samNote that, for time varying pling lattice , the formulation of this paper is fully applicable to the class of super-resolution up-sampling methods [8].
1057-7149/$20.00 © 2005 IEEE
568
Much research has focused on designing the regularization cost function , which might involve different a priori constraints, in order to pick one solution with desirable properties from the infinite number of possible solutions. Our focus in this paper is to find and design an observation model that can best produce the LR image from the desired HR image, where the acquisition of the HR image is also modeled. This observation for the regularmodel is used in the data fidelity cost function ized up-sampling process. We study the possibility of obtaining such observation models for various scenarios determined by the models for both cameras. The results presented in this paper are for some standard physical camera models and arbitrary theoretical cameras. We presented preliminary work concerning this study and offered a generalized design of the observation filter for arbitrary scenarios in [9]. A data fidelity cost function used for most image interpolation and super-resolution research [10]–[12] is the special case proposed by Schultz and Stevenson [13]. Their model assumed that the LR image was obtained from the continuous image by a CCD camera whose aperture is modeled by a rect function [3]. If the HR image is also obtained by a rect aperture, then the filter in the observation model is the discrete moving average. Baker and Kanade [8] involved in their analysis a general form for the optical blurring effect convolved with the rect function modeling a CCD camera used to acquire the LR image. They represented the continuous image as a piecewise constant, piecewise linear, or quadratic function of the HR image samples, which is a simple reversible linear representation for the continuous image from discrete samples. This work does not clearly distinguish the HR image and the continuous image, and, so, basically uses the sampled LR camera aperture as the observation model. The motivation in pursuing our study is that an accurate data observation model leads to a better definition of the solution space, which is indeed a critical factor for a better quality up-sampling [14]. We have been able to obtain a closed-form objective criterion for the design of the observation model for arbitrary scenarios modeling the LR and HR camera prefilters and sampling structures. The design has been performed in an optimization framework using a power spectral density (PSD) model for the continuous image. Our designed observation model performance has been validated for some practical scenarios as well as in regularized image up-sampling using two different regularization techniques. The remainder of this paper is organized as follows. In Section II, we present the problem statement and formulation, followed by the design of the observation-model objective criterion. In Section III, we minimize the objective criterion using analytic optimization and arrive at the design formula. Section IV provides details used for the estimation of the PSD of the continuous image. In Section V, several scenarios are set up and the corresponding observation model is designed. Many comparative simulations are provided to evaluate the performance of the observation model with respect to other possible models. Practical implementation of the observation model for regularized image up-sampling is performed in Section VI using two different regularization techniques. The effect of using an accurate observation model versus a mismatched observation model is also investigated with experiments in this section. The conclusions of the study are presented in Section VII.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005
II. PROBLEM FORMULATION A continuous-space(-time) image is acquired by a physical camera giving a digital LR image sampled on a lattice that can be two-dimensional (2-D) or three-dimensional (3-D) for fixed or time-varying images, respectively. The physical camera with a linear-shift-invariant (LSI) is modeled as filtering of followed by sampling on . We wish to estimate a filter digital HR image obtained by sampling on a denser lattice with an arbitrary, ideal camera with aperture , using a model-based regularization method. In order to achieve this, we to , defining the data need an observation model relating fidelity cost . , In this paper, we assume that is a sublattice of , and that the observation model consists of LSI filtering of on with followed by down-sampling to . Our goal is to choose to make the model as accurate as possible. Referring to Fig. 1, we desire the modeled signal , obtained by filtering with followed by down-sampling to , to resemble as closely as possible in some sense. The problem is cast as sampling structure conversion for signals on lattices, following the , we could design notation of [15], [16]. Note that if on the least-dense common superlattice of and , and the observation model would be a linear, periodically shift-variant and are ideal system. It is known from [15] that if both low-pass filters with passband confined to a unit cell of the reand , respectively, then should be a digciprocal lattices cannot ital ideal low-pass filter. However, physical cameras be modeled by an ideal low-pass prefilter, and, hence, the analysis in this paper is necessary to find the accurate for practical cameras. The problem is formulated in the frequency domain. has a continuous-space Fourier transform Assume that , where for for space–time images, and destill images and with finite energy defined notes inner product. A signal on a lattice is denoted by and has a Fourier . The Fourier transform is periodic in the frequency domain, i.e, transform , , where is the reciprocal lattice to ; any unit cell of constitutes one period of . It should be noted that if is generated by a sampling matrix , , then the reciprocal , where the superscript denotes lattice is the transpose of the matrix inverse. The sampling matrix establishes an isomorphism between and . Since the sampling matrix for any given lattice is not unique [1] and is not needed for any of the theoretical derivations, we do in the following developments. not use any specific and are obtained from as in Fig. 1, we find that If they are related to in the frequency domain by
(2) where is the unit-cell volume of its argument lattice [15]. , then Assuming that
ALY AND DUBOIS: SPECIFICATION OF THE OBSERVATION MODEL
where are the coset representatives of in Fourier transform of is given by
569
, [15]. It follows that the
The problem addressed in this paper is to find the best modeling that minimizes the mean square modeling error. filter Thus, the best modeling filter in the MSE sense satisfies (8)
(3) Since for any and substituting from can be written (2) into (3), it follows that (4) with the Thus, can be obtained in one step by filtering LSI continuous-space(-time) filter whose frequency response is and sampling on . We can also write the Fourier transform of the error as
(5) so that can be obtained by filtering with the LSI continuous-space(-time) filter whose frequency response is followed by sampling on . This error is iden, i.e., if tically zero if , where it is assumed that the appropriate limit . However, and are arexists where must be pebitrary aperiodic frequency responses, while riodic, so this solution will not be possible in general. There are several cases where this solution can be used. One practical and are rect functions, where case is when both and will be sinc functions and, hence, is a moving average filter. There are other nonpractical cases, from the point of view of physical camera realization, such as when both and are sinc functions as mentioned earlier in Section I. Also, when they are B-spline functions, as described in [2], then will be a B-spline digital filter of the same order. However, in most cases of interest where appropriate models are used for and , will not have the required periodcannot be made identically zero. Then, icity and the error in the methods of this paper should be used to determine order to minimize the mean-squared error . is a realization of a continuousIf we assume that , then the space(-time) stationary random field with PSD error is a realization of a discrete-space stationary random field with PSD that can be written as [15]
, which is a finite where sum for FIR filters, and real for zero-phase filters. It is important to note that the model of the continuous image as a stationary random field characterized by its autocorrelation function is simply used to minimize the average-squared modeling error over a representative ensemble of images. As will be seen, the solution is not very sensitive to the precise form of the PSD. A more sophisticated prior model is required to achieve good up-sampling; such a model influences the choice of . However, that is not the object of this paper. We can assume that the integrals in (7) are nonnegligible for in the vicinity of . This is reasonable only a few because the aperiodic frequency response of the continuous filand will decay rapidly in the neighborhood of ters , , respectively. This is dictated by the physical aperture prefilter characteristics in cutting down aliasing. Fixing some of independent coefficients for , is just a number real function of the N unknowns, and our objective function (7) can be minimized with a general optimization routine, or optimized analytically since it can be written as a quadratic form in .
III. ANALYTIC OPTIMIZATION While the general formulation given is applicable to two or three dimensions, we now consider the specific case of spatial (2-D) image up-sampling. Assume that is a lattice that ad[17]. Supmits quadrantal symmetry pose that has quadrantal symmetry, and that the independent , where , are in the quadrant , coefficients . Assuming so that and are real, then substituting this expression that for into (7) and expanding the square term yields
(6) The corresponding modeling mean-square error (MSE) is
(7)
(9)
570
Fig. 2.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005
Image PSD estimates S (u) by two different methods. (a) Parametric estimate from an auto-correlation model. (b) Welch-modified periodogram.
where . Stacking (9) can be simply written as
into a lexicographic vector
,
(10) can be where the elements of the vector and the matrix determined by numerical integration of the appropriate terms in is easily minimized with (9). Then,
methods above as well as a flat (white) PSD. The resulting filter which coefficients were the same to within a difference of is almost 0.31% of the maximum filter coefficient’s value. This indicates that the results are not sensitive to the choice of the and . PSD model but rather are dependent on This is accounted for by the fact that and decay quickly in the vicinity of putting less weight on the high frequencies.
(11) V. VALIDATION OF THE OBSERVATION MODEL BY SIMULATIONS IV. PSD ESTIMATION Several models for the PSD of the continuous image exist. In the estimation of , we used parametric and nonparametric methods. A. Parametric Model for PSD Estimation Since is the Fourier transform of the autocorrelation , we used the basic model for the autocorrelation of continuous images defined by , where is the autocorrelation with zero lag and , are parameters [18]. Calculating the 2-D Fourier transform of and extending one of the one-dimensional (1-D) Fourier transform formulas in [19], we derived the PSD to be (12) , where A plot for the estimated PSD for (1/ph) denotes units per picture height, is given in Fig. 2(a). B. Nonparametric PSD Estimation In our simulations, we also used the Welch-modified periodogram method [20], using a Blackman–Harris window, to estimate the PSD from a very high-resolution image of size 3390 2436. The window size was 64 with eight samples of overlap. The estimated PSD is given in Fig. 2(b). In our design procedure to obtain an observation model for different scenarios, we used the PSD estimated by the two
Most physical camera apertures have an impulse response that is modeled by a standard 2-D Gaussian or rect function or, alternatively, as a circ function if the out-of-focus blur is the dominant degradation. Hence, we designed the basic experiment scenarios using different combinations of Gaussian and and . We will name each scerect functions for and , renario according to the aperture used for . For example, if is a spectively, and the value of is a rect function, and , then we call Gaussian, this scenario Gauss-Rect ( 25). The sample scenarios in this section are Gauss-Gauss ( 25), Rect-Rect ( 25), Rect-Gauss ( 25), and Gauss-Rect ( 25). The sampling lattices are given and , where is by measured in units of picture height (ph). For each scenario we obtained an FIR modeling filter described in the first row in Table I. Inspecting the magnitude of the frequency response of the modeling filters obtained for different scenarios, we can observe the following. 1) We only obtained the moving average filter as a modeling filter for the scenario Rect-Rect ( 25). 2) For the Rect-Gauss ( 25) and Gauss-Gauss ( 25) scenarios, we obtained modeling filters that are far from being a moving average filter. 3) The magnitude of the frequency response of the modeling filters obtained for scenarios having the same filter tend to have close characteristics in their passband, while the stopband shape is different. This is clear by comparing Fig. 3(c) and (d).
ALY AND DUBOIS: SPECIFICATION OF THE OBSERVATION MODEL
Fig. 3. Magnitude of the frequency response for the observation model (in decibels): (c) Rect-Gauss (# 25) and (d) Gauss-Gauss (# 25).
571
H (u) for two scenarios: (a) Rect-Gauss (# 25) and (b) Gauss-Gauss (# 25). Also plotted
4) The major difference among modeling filters obtained for different scenarios is in the stopband. This is due to the aliasing components introduced in the sampling processes involved. A general conclusion is that using an incorrect modeling filter in the observation model for the image up-sampling problem will have its greatest impact in measuring the high-frequency components whereas it will not have much effect in measuring the low-frequency components. This statement can be verified by the experimental results in the Section VI. In order to verify the results obtained, we ran a simulation of each scenario and used the modeling filter obtained to measure on images as follows. the actual modeling error 1) Start with a very high-resolution image simulating the continuous signal . by the appropriate digitally designed 2) Simulate , Gaussian or moving average filter. For Gaussian filters,
we can choose the appropriate cutoff angular frequency as given in [21]. and 3) The very-high-resolution image is filtered by providing 2 filtered versions of . 4) Down-sample the filtered images by different large factors. The images obtained are used to simulate , . The large down-sampling factors are used to minimize the error between the digital simulation and the reality of the continuous spectrum analysis, simulating the sampling on both lattices , , respectively. 5) Obtain by filtering with , obtained in different ways, followed by down-sampling from to . . 6) Measure the actual modeling error 7) Compute the peak-signal-to-noise ratio (PSNR) using . The computed PSNR value for each scenario is given in Table I. The PSNR results in Table I were obtained using a high-resolution 3390 2436 frame from the IMAX 3-D movie Space Station. A stereoscopic rendition of the image can be seen in [22,
572
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005
TABLE I PERFORMANCE OF MODELING FILTERS FOR DIFFERENT SCENARIOS: ke(x)k , MSE EXPRESSED AS PSNR (IN DECIBELS)
pp. 54–55]. For the sake of comparison, we also chose some other reasonable filters to be the modeling filter and measured the actual modeling error. The simulations were repeated for each of these different choices of for the four basic scenarios. The chosen filters are as follows: 1) the moving average and the Gaussian filters, because, intuitively, they seem to be suitable for some scenarios like the Rect-Rect and the Gauss-Gauss, respectively; 21 with maximally flat 2) a separable filter of order 21 passband and monotonically decreasing transition and stopband. This filter was chosen because it produces images with no ringing artifacts while not compromising resolution as much as many other filters like Gaussian ones. We chose two filters with two different cutoff and which are around the frequencies of for these case-study critical cutoff frequency scenarios. For all the Gaussian filters used as modeling filters, we optimized the variance parameter to maximize the PSNR of the result because we found that the results change drastically with the choice of the variance of this Gaussian function. The choice of the optimal variance of the Gaussian function was obtained by running a 1-D search that provides the maximum PSNR result for each scenario with a given image. The values of these
optimal standard deviations for each scenario are given in the footnote of Table I. in The order of each filter obtained by our model is each dimension and the value of is written between brackets in Table I for each filter. The minimum order was chosen such that it would provide the highest PSNR to a tolerance of the order of 0.1 dB. In these experiments, we started with and incremented it by one until there was no significant enhancement in the PSNR or if the PSNR value decreased by increasing the order. This latter case only happened for the RectRect scenario because, due to the simulation approximation er5 rors, it was not able to set the coefficients beyond the 5 kernel exactly to zero. We wanted to obtain the upper bound on the PSNR value that can be obtained for the simulation for each scenario. We achieved this by obtaining a modeling filter using the images , themselves, and solving a numerical optimization problem, directly from the images. Here, we used the least squares method because the problem is overdetermined. The problem is simply formulated as (13) where the matrix performs both the filtering by the argument is a block filter and down-sampling from to . Thus,
ALY AND DUBOIS: SPECIFICATION OF THE OBSERVATION MODEL
573
VI. IMPACT OF THE OBSERVATION MODEL ON REGULARIZED IMAGE UP-SAMPLING
Fig. 4. Magnitude of the frequency response for the observation model for Gauss-Rect (# 25).
Fig. 5.
H (u)
Portion of the cameraman image.
diagonal matrix formed from the coefficients of . The modeling filter so obtained was used to compute the modeling error which appears in the first row of results in Table I and is named simulated filter. This helped in obtaining an upper bound on the PSNR that can be obtained from the simulation experiment. However, it cannnot be practically used since, in real applications, is not available, and it results in a modeling filter that is biased toward the specific features of the underlying image. Comparing the results in Table I, we observe the following. 1) In the Rect-Rect scenario, the result is almost a moving average filter to some numerical approximation which agrees with the result in [13], [15]. The reason that we did not obtain exactly the moving average filter as a modeling filter is accounted by the numerical approximations involved in the numerical integration, and simulation apand to be disproximations such as using crete approximation of continuous-space filters and simulating the sampling process by sub-sampling with large factors. However, a PSNR of 58.39 as compared to 63.22 of the moving average is a very good result and the difference in results cannot be distinguished by human viewers. 2) For the Gauss-Gauss scenario, the method designed a Gaussian-like modeling filter with the optimized variance of the Gaussian function. The result is very similar to the manually optimized Gaussian filter, which is a good indication on the performance of the method in this scenario. It should be noted that in practice we cannot manually optimize the variance of the Gaussian filter because the is not available. HR image 3) For the Rect-Gauss and the Gauss-Rect scenarios, we obtained new modeling filters that are neither a simple moving average nor a Gaussian filter.
In this section, we verify the impact of the designed observation model (data fidelity) for the problem of image up-sampling by practical experiments. In the first analysis, we check the effect of using different observation models on different regularization methods. We used an image for which we did not know the acquisition process. We used two different regularized image up-sampling methods: a general standard regularization method of Tikhonov given by , where denotes the discrete Laplacian operator, and the advanced total-variation method in [7] given by , where denotes the spatial gradient and denotes the norm. For each method, we used two different data fidelity terms designed from two scenarios, specifically the Gauss-Rect ( 25) and RectGauss ( 25) scenarios. In all four experiments, we up-sampled a portion of the cameraman image given in Fig. 5 by a factor of 5 in each dimension. Note that, in this case, we have no knowledge of the LR camera model. We chose the regularization parameter empirically based on the best perceived image quality in our opinion as human observers. Although we can obtain using different methods, such as the L-curve for Tikhonov regularization, there is no adopted way to obtain it for the total-variation method. For the total-variation up-sampling method [7], for both scewe set the regularization parameter narios. The resulting up-sampled images using the method in [7] for the Gauss-Rect ( 25) and Rect-Gauss ( 25) scenarios are shown in Fig. 6(a) and (b), respectively. Clearly, there is a significant difference in the high-frequency contents (edges) between both images despite the fact that we used the same up-sampling method. The measured mean-squared difference between these two images expressed as PSNR is 28.91 dB. This indicates that choosing the appropriate observation model will have a significant impact on the results if a suitable regularizer utilizes it. The same experiments were performed for Tikhonov regularization method for . The resulting up-sampled images using the Tikhonov regularization method for the Gauss-Rect ( 25) and Rect-Gauss ( 25) scenarios are shown in Fig. 6(c) and (d), respectively. In this case, there is only a subtle difference in the results although we consider these results to be significantly inferior to either result from the total-variation method. The measured mean-squared difference between these two images expressed as PSNR is 42.55 dB, which is slightly above the distinction level by human observers. This is not a surprise because the Laplacian high-pass filter used in Tikhonov regularizer does not differentiate between noise activity and edges. In other words, it is a blind measure of high-frequency components to suppress them regardless of the underlying image structures. Hence, it does not attempt to make use of the extra information provided by the observation model in the high-frequency region. In the second analysis, we investigated the effect of using the appropriate observation model versus a mismatched one for the total-variation regularized image up-sampling of [7]. We simulated the Gauss-Gauss ( 25) imaging scenario by obtaining and from the very high-resolution test image “Woman” using
574
Fig. 6.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005
Up-sampling of a portion of the cameraman image, shown in Fig. 5, by a factor of 25 using the total variation approach [7] with a regularization parameter
= 0:05 and with two different observation models for two scenarios: (a) Rect-Gauss (# 25) and (b) Gauss-Rect (# 25). Up-sampling using Tikhonov regularization method with = 0:1 and with two different observation models for two scenarios: (c) Rect-Gauss (# 25) and (d) Gauss-Rect (# 25).
the appropriate Gaussian filters followed by down-sampling as in Section V. We chose this scenario because it provided us with images and that look visually better than those obtained by Rect-Rect scenario and naturally closer to what we are accustomed to see in images that are obtained by phys-
ical cameras. The ideal HR image is shown in Fig. 7(c) and the LR image is shown in Fig. 7(d). We ran the up-sampling experiments using the appropriate observation model obtained for the Gauss-Gauss ( 25) and a mismatched observation model obtained for Rect-Rect ( 25) scenario. Both exper-
ALY AND DUBOIS: SPECIFICATION OF THE OBSERVATION MODEL
575
Fig. 7. Up-sampling a portion of the test image “Woman” obtained by simulating a Gauss-Gauss (# 25) scenario by 5 in each dimension using the total variation method [7] with correct and incorrect observation model. (a) Up-sampled using a moving average observation model obtained by assuming Rect-Rect (# 25) scenario and = 0:02. (b) Up-sampled using observation model obtained by assuming Gauss-Gauss (# 25) scenario and = 0:02. (c) Ideal HR image f . (d) LR image f , both obtained from the very high-resolution original by two appropriate Gaussian prefilters.
iments used the method in [7] for up-sampling by a factor of 5 in each dimension and with the same regularization parameter which was chosen empirically to provide visually the
most pleasing up-sampled image using the matched observation model. The results are shown in Fig. 7 and the significance of the improved results using the appropriate observation model in
576
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 14, NO. 5, MAY 2005
Fig. 7(b) is evident over using a mismatched observation model shown in Fig. 7(a).1 VII. CONCLUSION This paper has demonstrated that proper choice of the observation model can have a significant impact on the performance of regularized up-sampling of still images. We can presume that this is also the case for up-sampling of time-varying imagery, which corresponds to the class of super-resolution methods. We have presented an algorithm to determine the observation model that relates the measured LR images to the desired HR images and shown that the observation model obtained can indeed lead to improved regularized up-sampling performance. With our optimal modeling filter tightening the relation between HR and LR images, we can reduce noise amplification during the regularized up-sampling process. This allows a relaxation of the smoothness (regularization) constraint (s) and helps in obtaining less blurred or over-regularized results if a suitable regularizer, like total variation, is used. Although the algorithm uses the PSD of the underlying continuous image in addition the LR and HR camera models, the results are very insensitive to this PSD and are essentially determined by the camera models. We now have the flexibility to choose an ideal camera model for the HR image, for example the model best adapted to the intended display device [1]. Future work will further examine the impact of the correct choice of the observation model using the total variation regularizer for both still and time-varying imagery. REFERENCES [1] H. A. Aly and E. Dubois, “Design of optimal camera apertures adapted to display devices over arbitrary sampling lattices,” Signal Process. Lett., vol. 11, pp. 443–445, Apr. 2004. [2] M. Unser, “Splines: a perfect fit for signal and image processing,” IEEE Signal Process. Mag., vol. 16, no. 6, pp. 22–38, Nov. 1999. [3] J. P. Allebach, “Image scanning, sampling, and interpolation,” in Handbook of Image and Video Processing, A. Bovik, Ed. San Diego, CA: Academic, 2000, ch. 7.1, pp. 629–643. [4] A. Biancardi, L. Cinque, and L. Lombardi, “Improvements to image magnification,” Pattern Recognit., vol. 35, pp. 677–687, Mar. 2002. [5] S. Carrato, G. Ramponi, and S. Marsi, “A simple edge-sensitive image interpolation filter,” in Proc. IEEE Int. Conf. Image Processing, vol. 3, Oct. 1996, pp. 711–714. [6] Q. Wang and R. Ward, “A new edge-directed image expansion scheme,” in Proc. IEEE Int. Conf. Image Processing, vol. 3, Oct. 2001, pp. 899–902. [7] H. Aly and E. Dubois, “Regularized image up-sampling using a new observation model and the level set method,” in Proc. IEEE Int. Conf. Image Processing, vol. 3, Sep. 2003, pp. 665–668. [8] S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 9, pp. 1167–1183, Sep. 2002. [9] H. Aly and E. Dubois, “Crafting the observation model for regularized image up-sampling,” in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 3, Apr. 2003, pp. 101–104. [10] B. Tom and A. Katsaggelos, “Resolution enhancement of monochrome and color video using motion compensation,” IEEE Trans. Image Process., vol. 10, no. 2, pp. 278–287, Feb. 2001. [11] D. Rajan and S. Chaudhuri, “Generation of super-resolution images from blurred observations using Markov random fields,” in Proc. IEEE Int. Conf. Acoustics Speech Signal Processing, vol. 3, Jun. 2001, pp. 1837–1840. [12] M. Elad and A. Feuer, “Restoration of single superresolution image from several blurred, noisy, and undersampled measured images,” IEEE Trans. Image Process., vol. 6, no. 6, pp. 1646–1658, Dec. 1997. 1It should be noted that the result in Fig. 7(a) can be enhanced by over regularization using a larger value of . We tried this, but the result was still of distinctly lower quality than that in Fig. 7(b). This represents an interaction between the appropriateness of the observation model and the regularization method which is beyond the scope of this paper.
[13] R. R. Schulz and R. L. Stevenson, “A Bayesian approach to image expansion for improved definition,” IEEE Trans. Image Process., vol. 3, no. 5, pp. 233–242, May 1994. [14] S. Borman and R. L. Stevenson, “Super-resolution for image sequences—a review,” in Proc. IEEE Int. Symp. Circuits and Systems, Jun. 1998, pp. 374–378. [15] E. Dubois, “The sampling and reconstruction of time-varying imagery with application in video systems,” Proc. IEEE, vol. 73, no. 4, pp. 502–522, Apr. 1985. , “Video sampling and interpolation,” in Handbook of Image and [16] Video Processing, A. Bovik, Ed. San Diego, CA: Academic, 2000, ch. 7.2, pp. 645–654. [17] S. Coulombe and E. Dubois, “Linear phase and symmetries for multidimensional FIR filters over lattices,” IEEE Trans. Circuits Syst. II, Analog Digit. Signal Process., vol. 45, no. 4, pp. 473–481, Apr. 1998. [18] A. K. Jain, Fundamentals of Digital Image Processing. Upper Saddle River, NJ: Prentice-Hall, 1989. [19] R. N. Bracewell, The Fourier Transform and Its Applications, 3rd ed. New York: McGraw-Hill, 2000. [20] P. D. Welch, “The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms,” IEEE Trans. Audio Electroacoust., vol. AU-15, no. 6, pp. 70–73, Jun. 1967. [21] O. Schade Sr., “Image reproduction by a linear raster process,” in Perception of Displayed Information, L. M. Biberman, Ed. New York: Plenum, 1973, pp. 233–278. [22] C. Covault, “IMAX 3D launches you to the space station,” Aviation Week Space Technol., vol. 155, no. 26, pp. 54–71, Dec. 2001.
Hussein A. Aly (M’00) received the B.Sc. degree (excellent with honors) in computer engineering and the M.Sc. degree in electrical engineering from the Military Technical College, Cairo, Egypt, and the Ph.D. degree in electrical engineering from the University of Ottawa, Ottawa, ON, Canada, in 1993, 1997, and 2004, respectively. He is currently with the Ministry of Defence, Cairo, Egypt. While at the University of Ottawa, he participated in a project on image magnification funded by the Royal Canadian Mounted Police (RCMP). His research interests are in image sampling theory and sampling structure conversion. His current research is focused on high-quality image magnification, interpolation of color filter array data (demosiacking), and the application of total variation for image processing. Dr. Aly is a Member of the Egypt Engineers Syndicate.
Eric Dubois (F’96) received the B.Eng. (honors, with great distinction) and M.Eng. degrees in electrical engineering from McGill University, Montreal, QC, Canada, and the Ph.D. degree in electrical engineering from the University of Toronto, Toronto, ON, Canada, in 1972, 1974, and 1978, respectively. He joined the Institut National de la Recherche Scientifique, University of Quebec, Montreal, in 1977, where he held the position of Professor in the INRS-Télécommunications Centre, Montreal. Since July 1998, he has been a Professor with the School of Information Technology and Engineering (SITE), University of Ottawa, Ottawa, ON. He is currently serving as Vice Dean and Secretary of the Faculty of Engineering. His research has centered on the compression and processing of still and moving images and in multidimensional digital signal processing theory. His current research is focused on archival document processing and compression, stereoscopic and multiview imaging, image sampling theory, and image-based virtual environments. The research is being carried out in collaboration with such organizations as the Communications Research Centre, the National Capital Institute of Telecommunications, the Royal Canadian Mounted Police (RCMP), and the Learning Objects Repositories Network (LORNET). He is a member of the editorial board of the EURASIP journal Signal Processing: Image Communication. Dr. Dubois is corecipient of the 1988 Journal Award from the Society of Motion Picture and Television Engineers. He is a Fellow of the Engineering Institute of Canada, a Member of the Order of Engineers of Quebec, and was an Associate Editor of the IEEE TRANSACTIONS ON IMAGE PROCESSING (from 1994 to 1998). He was Technical Program Co-Chair for the IEEE 2000 International Conference on Image Processing (ICIP) and a member of the organizing committee for the IEEE 2004 ICASSP.
