A recursive soft-decision approach to blind image deconvolution ...

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 2, FEBRUARY 2003

515

A Recursive Soft-Decision Approach to Blind Image Deconvolution Kim-Hui Yap, Student Member, IEEE, Ling Guan, Senior Member, IEEE, and Wanquan Liu

Abstract—This paper presents a new approach to blind image deconvolution based on soft-decision blur identification and hierarchical neural networks. Traditional blind algorithms require a hard-decision on whether the blur satisfies a parametric form before their formulations. As the blurring function is usually unknown a priori, this precondition inhibits the incorporation of parametric blur knowledge domain into the restoration schemes. The new technique addresses this difficulty by providing a continual soft-decision blur adaptation with respect to the best-fit parametric structure throughout deconvolution. The approach integrates the knowledge of well-known blur models without compromising its flexibility in restoring images degraded by nonstandard blurs. An optimization scheme is developed where a new cost function is projected and minimized with respect to the image and blur domains. A nested neural network, called the hierarchical cluster model is employed to provide an adaptive, perception-based restoration. Its sparse synaptic connections are instrumental in reducing the computational cost of restoration. Conjugate gradient optimization is adopted to identify the blur due to its computational efficiency. The approach is shown experimentally to be effective in restoring images degraded by different blurs. Index Terms—Blind image deconvolution, conjugate gradient, hierarchical neural network, soft-decision.

I. INTRODUCTION

T

HE PRIMARY objective of image restoration is to recover lost information from a degraded image and obtain the best estimate to the original image. Its applications include photography deblurring, sonar imaging, remote sensing, medical imaging, and multimedia processing. Linear image degradation processes are commonly modeled by (1) where , , and are the lexicographically ordered degraded is image, original image, and additive noise, respectively. the linear distortion operator constructed from the point spread function (PSF) or the blur [1]–[3]. Classical image restoration assumes that the complete knowledge of the blur is available. However, the assumption is elusive in real-world applications Manuscript received October 3, 2001; revised October 3, 2002. The associate editor coordinating the review of this paper and approving it for publication was Prof. Derong Li. K.-H. Yap is with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore (e-mail: [email protected]). L. Guan is with the Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada M5B 2K3 (e-mail: [email protected]). W. Liu is with the School of Computing, Curtin University of Technology, Perth, Australia (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2002.806985

as it is often too cumbersome, costly, or even improbable to obtain the actual blur. These could be due to various practical constraints such as the difficulty of characterizing air turbulence in aerial imaging or the potential health hazard of employing stronger incident beam to improve the image quality in X-ray imaging. These circumstances motivate and point to a blind image deconvolution scheme. Blind image deconvolution is an inverse problem of inferring the best estimates, and to the original image and the blur based on the degradation model. It is a difficult and ill-posed problem as the uniqueness and stability of the solution is not guaranteed [4]. A class of popular techniques for blind image deconvolution is a priori blur identification [5]–[8]. These approaches formulate blind deconvolution into two disjoint processes where the blur is first estimated, followed by classical restoration using the identified blur. They require either the blur to satisfy a parametric structure or the image to contain certain invariant edges or point sources. This restricts their flexibility as the schemes are unable to restore images degraded by untargeted blurs. In addition, the frequency null analysis used in blur identification fails to identify certain PSF structures, for instance, the Gaussian blur that does not show prominent nulls. Research into two-dimensional (2-D) linear stochastic systems has led to the suggestion of modeling a blurred image as an autoregressive moving average (ARMA) process. Under this assumption, blind image deconvolution is transformed into an ARMA parameter estimation problem. Maximum likelihood (ML) [9]–[11] and generalized cross validation (GCV) [12] are two popular methods that are employed to determine these parameters. Expectation maximization (EM) is often used in conjunction with ML to maximize the log-likelihood function of the parameter set from their solution spaces. In contrast, GCV determines the ARMA parameters by minimizing a weighted sum of predictive errors. The disadvantage of GCV lies in its high computational cost. In general, both ML and GCV require small AR and MA support sizes to have a manageable computational complexity, albeit at the cost of diminished modeling effectiveness. Motivated by the idea of joint image and blur estimations, several iterative algorithms employing deterministic constraint deconvolution have been proposed. These algorithms estimate the image and blur while imposing non-negativity and support constraints on them. Iterative blind deconvolution (IBD) [13] and simulated annealing (SA) [14] are two examples of iterative schemes. IBD alternates between spatial and frequency domains, whereas SA attempts to minimize a multimodal fidelity error using standard annealing process. Other deterministic constraint methods include non-negativity and support

1053-587X/03$17.00 © 2003 IEEE

516

constraints recursive inverse filtering (NAS-RIF) [15] and its extensions such as enhanced NAS-RIF [16] and regularized RIF [17]. These algorithms involve recursive filtering of the blurred image to minimize a convex cost function. They require the image object to have a known support and be located in a uniform background. A recent development in blind image deconvolution centers on proposals of multichannel blind algorithms. These techniques are inspired by one-dimensional (1-D) blind channel identification in digital communication where oversampled signals of multichannel systems are used to estimate the channel. Extending the concept to 2-D image deconvolution, several blind algorithms have been proposed, including eigenstructure-based multichannel restoration [18], direct blind estimation, and indirect least-square algorithms [19], and the greatest common divisor (GCD) approach [20]. Nevertheless, these techniques require multiple blurred versions of the same scene to be available. Another research direction involves extending regularization theory to address blind image deconvolution [21], [22]. The approaches attempt to minimize a cost function consisting of a restoration measure and some regularization functionals. They assume an implicit symmetry between the image and blurring function, resulting in underutilization of unique characteristics in each domain. The main principles of blind image algorithms can be described concisely as identification of useful a priori information such as the invariant features of input signals, integration of extracted knowledge into the schemes without compromising their flexibility, and development of appropriate computational techniques to optimize the cost function. We will develop an algorithm that addresses these requirements for systematic signal identification, intelligent information integration, and efficient optimization. A new scheme for blind image deconvolution based on soft-decision blur identification and hierarchical neural network is proposed. Unlike those traditional algorithms that require a hard decision on whether the blur satisfies a parametric model prior to their formulations, the approach offers a continual blur adaptation throughout the restoration. A cost function consisting of data fidelity measure, image and blur-domain regularization terms, and a new soft-decision blur error is introduced. A blur estimator is devised to determine various parametric models. The solution space of these estimates is constructed based on well-known blur structures or any available prior knowledge. The technique utilizes fuzzy concept to assimilate parametric information. The relevance of the blur models is evaluated, mapped onto a fuzzy measure, and integrated into the system accordingly. The method does not require assumptions such as image stationarity as in ARMA modeling. In addition, it is more flexible than constrained-based algorithms like NAS-RIF as it does not require the precondition that the support of the object to be known a priori or be located in a uniform background. The organization for the rest of the paper is outlined as follows. In Section II, the formulation of alternating minimization is explained. In Section III, the properties and algorithms for adaptive image restoration based on the HCM are described. In Section IV, a maximum a posteriori blur estimator is introduced. In Section V, the formulation of soft estimation error and blur

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 2, FEBRUARY 2003

identification using constrained conjugate gradient optimization are discussed. In Section VI, experimental results using the proposed method are discussed and compared with other technique. In Section VII, conclusions and further remarks are given. II. ALTERNATING MINIMIZATION Regularization theory has been used frequently to alleviate the ill-conditioned nature of image restoration [4] and, consequently, is integrated into our algorithmic development. A cost function consisting of a data fidelity measure, image, and blur-domain regularization terms, and the new soft blur estimation error is proposed as

(2) where and are the diagonal image and blur-domain reguand are standard Laplacian highpass larization matrices, is the soft-decision error beoperators [23], [27], and tween the computed blur and parametric estimate . When the PSF or is space-invariant, assumes the structure of block Toeplitz matrix. Similar to the formation of from , the macan be constructed by rearranging the coefficients of trix into block circulant structure, as described in [2] and [3]. The data fidelity term functions as a restoration criterion. However, the measure alone is sensitive to abrupt changes in visual activities and tends to form undesirable ringing over the smooth backgrounds. Therefore, regularization terms are incorporated into the cost function to introduce stability into the system. The image-domain regularization term encapsulates to provide an adaptive image the regularization matrix restoration. Likewise, the blur-domain regularization term to render piece-wise includes the regularization matrix smoothness in the blur. The soft error determines the proximity measure and modeling error of with respect to . If there exists a high degree of proximity, the confidence that the current blur assumes a parametric structure increases, leading to a gradual adaptation of toward . Otherwise, the fidelity and regularization terms will dominate the restoration criterion, and the soft error will wither away. This underlines the progressive nature of feature integration. The approach encapsulates the concept of fuzzy logic, where the relevance of the information is assessed and mapped into a membership function between zero and one. This is in contrast with the crisp logic employed by traditional a priori blur identification, where the parametric information is adjudged simply as either “fully relevant” or “totally irrelevant.” Due to the distinctive characteristics between the image and blurring function, the optimization objectives and priorities for them should be handled differently. Therefore, it is both algoonto the rithmically and intuitively sensible to project image and blur domains to form the following cost functions: (3)

(4)

YAP et al.: RECUSIVE SOFT-DECISION APPROACH TO BLIND IMAGE DECONVOLUTION

517

where and are the projection constants of onto the image and blur domains, respectively. The mathematical formulation, which is also known as alternating minimization de(AM) [21], [22], is summarized as follows, with and noting the solution spaces for and , respectively. to a random mask. i) Initialize th step, solve for and ii) For the (5)

Fig. 1. Flowchart of proposed algorithm.

(6) iii) Stop when convergence or a maximum number of iterations is reached. The overview of the proposed algorithm is illustrated by the flowchart in Fig. 1. The optimization procedure consists of two important steps: image restoration and blur identification. The restored image and identified blur are estimated by minimizing their respective cost functions. Image restoration involves the formation of HCM cluster structure and its energy minimization, which is to be discussed in Sections III-B and C. Blur identification handles soft parametric blur estimation, followed by conjugate gradient optimization, to be explained in Sections IV and V, respectively. The optimization process is employed to and , thereby improving the image and minimize blur estimates progressively. The proposed algorithm differs from the symmetric double regularization (SDR) approaches suggested by You and Kaveh [21] and Chan and Wong [22] in several ways. Their mathematical formulations decompose blind image deconvolution into two symmetrical processes of estimating the image and blur. Even though different weighting considerations are adopted during blur identification and image restoration, the underlying processes of their algorithms do not incorporate the unique characteristic of each domain. First, numerous real-world PSFs satisfy, up to a certain degree, a parametric structure. In comparison, we have little prior knowledge on most imaging scenes. Second, image restoration is inherently a perception-based notion, where human vision tends to recognize cluster-based image features. Third, most PSFs exist in the form of lowpass filters. In contrast, typical images consist of smooth, textured and edge regions, whose power spectral densities vary considerably from predominant low frequency content in the smooth regions to medium- and high-frequency content in the textured and edge regions. These observations illustrate the importance of performing image restoration and blur identification according to their characteristics. The proposed method attempts to address these asymmetries by integrating parametric blur information into the scheme and tailors image restoration and blur identification according to their properties. III. HIERARCHICAL NEURAL NETWORK IMAGE RESTORATION

Fig. 2.

Schematic diagram of the HCM.

tially finite PSF. As a result, only a finite local neighborhood is required to estimate each . This implies window of that image restoration is conceptually a collection of local parallel processes with a global coordination. A hierarchical cluster model (HCM) is a nested neural network consisting of parallel, distributed subnetworks [25], [26], [30]. It models the organization of the neo-cortex in the human brain, where functional groups of neurons organize themselves dynamically into multidimensional subnetworks. The schematic diagram of an HCM in Fig. 2 shows the structure of several interacting homogeneous subnetworks. Individual neurons form the basic level-zero clusters of the network. Those neurons sharing similar functionality coalesce into numerous level-one clusters, as illustrated in the diagram. Likewise, those level-one clusters with homogeneous characteristics in turn coalesce into level-two clusters. This process continues repeatedly to form a hierarchical neural network with multidimensional distributed clusters. The HCM bears close resemblance to image formation. We such that , define a functional mapping : , where and are the representative space for is the HCM neuron and image pixel , respectively, is the image dimension. the total number of neurons, and The mapping describes a correspondence between the HCM and the image. Each individual neuron of the HCM corresponds to an image pixel, with a level-one cluster corresponding to a homogeneous image region and a level-two cluster corresponding to a dynamic adjustment of the regions’ boundary. The main advantage of the HCM lies in its sparse intercluster synaptic connections that reduce the computational cost of restoration. The distributed structure of the HCM underscores the local parallel process of pixel formation. In addition, its hierarchical structure provides an overall coordination of the local parallel processes.

FOR

A. Structure and Properties of the Hierarchical Cluster Model

B. Optimization of Image-Domain Cost Function as HCM Energy Minimization

involves the The formation of each degraded image pixel neighborhood window by a spafiltering operation of an

We will formulate image restoration as an HCM energy min. imization problem based on the functional mapping :

518

The image-domain cost function as

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 2, FEBRUARY 2003

in (3) can be expressed

Consider the decomposition of (10) into the cluster level .. .

(11.1)

(7) We can simplify (7) by assigning and ignoring the constants

,

, and

:

.. .

.. .

(11.2)

.. .

.. .

(11.3)

(8) Typical images can be characterized into three common feature types, namely smooth, textured, and edge classes. These regions exhibit different characteristics and, as such, should be handled differently. Generally, smooth regions require large regularization parameters to prevent ringing and noise amplification, whereas textured and edge regions prefer small parameters to encourage detail preservations. The textured areas should be distinguished from the edges because the rich visual contents of the texture can provide more noise masking capability than the well-defined changes of gray levels at the edges. Motivated by this general characterization, we decompose the cost function into smooth , texture , and edge partitions by introducing

.. .

(11.4)

where and are the number of clusters for partition types and . Substituting (11.1)–(11.4) into (10) to express in terms of cluster levels, we obtain

(9.1)

(9.2)

(9.3)

(9.4)

are the rearranged pixels for smooth, texwhere , , and tured, and edge cluster types. The matrices , and vector are formed likewise into their corresponding matrices and vector, similar to [27], [30]. It is observed from (9.1)–(9.4) that there exists a cyclic symmetry between the smooth, texture, and edge partitions. Substituting (9.1)–(9.4) into (8) and expressing the result using the implicit cyclic symmetry, we obtain

(10)

are the cluster types. The first and where , second summation terms in (10) represent the contributions from similar and different cluster types, respectively.

(12) where , are the cluster designations. The first term in (12) refers to the intracluster contribution. The second and third terms correspond to intercluster contributions arising from similar and different cluster types. The energy function of a three-level HCM is given by [25], [26]

(13) are the state of the th neuron where is the intrain th cluster and th neuron in th cluster, and , cluster connection weight between neurons is the intercluster connection weight between neurons and , is the average strength of intercluster connections to inis the bias input to neuron , tracluster connections, is the number of gray levels, and is the number of bits used for first-level partitions with pixel representation. There are and representing the number of neurons in the th and th cluster, respectively. The first and second summation triplets in represent the intracluster and intercluster contributions to the overall energy function.

YAP et al.: RECUSIVE SOFT-DECISION APPROACH TO BLIND IMAGE DECONVOLUTION

The image-domain cost function in (12) and the energy function of HCM in (13) show a similar structure under close inspection. The HCM energy function is expressed in terms of generic clusters from smooth, texture, and edge partitions. Comparing these two expressions by mapping neuron to pixel and extending the generic clusters in (13) to encapsulate the detailed clusters in (12), we obtain (14) (15) (16) (17) (18) , , and are the intracluster connection where weight of cluster type , intercluster connection weight within cluster type , and intercluster connection weight between and . , , , , cluster types , are the th scalar entries of matrices , , , , , and , respectively. It is noted that diag , where is the th is cluster’s regularization parameter with th type. As similar within the same cluster, we can simplify (14)–(17) in terms of image degradation parameters to give (19)

(20)

519

adopt a logarithmic-based regularization function, as suggested in [27], as it is commonly used to model the nonlinear transformation of human perception. takes the form of a negative with gradient logarithm function of and offset . A negative logarithmic function is chosen due to the observation that textured and edge clusters, having high local variance, require small regularization parameters to preserve fine details. The reverse holds true for the smooth and take the form of shifted, scaled clusters. and are decaying exponential functions. As and , respectively, comparatively larger than this enables the restoration scheme to recover more details progressively as we become more confident about the identified blur and the restored image. It is noted that neighboring image clusters employ different regularization parameters for adaptive restoration. Therefore, in order to provide a smooth restoration across different regions, regularization parameters near the cluster boundaries are calculated from (23) based on local variance of boundary pixel neighborhood. C. Cluster Formation and Restoration We construct a three-level HCM from the blurred image. The image is first divided into smooth, textured, and edge regions, followed by neighborhood partitioning to form separate homogenous clusters. It is noted that only an approximate partitioning is required initially. This is because our approach provides continual adjustment to the partition structure as the image quality improves throughout deconvolution. In addition, the regularization parameters near the boundaries are estimated pixel-wise, hence, relaxing the condition. The representations for the cluster formation of an image are given as (26)

(21) (22) has a unity value as the space-invariance of The parameter the PSF causes the intercluster synaptic strength to be as strong as the intracluster synaptic strength. These results agree with the weight structure of other network-based algorithm [23], [27], [30]. The regularization parameter of the pixels in the th cluster is given by for the th recursion

(23) is the average local variance of the th cluster, and where , are the lower and upper regularization thresholds given by (24) (25) , , , and are the lower and upper regularization values in the beginning and final recursion. We

(27) are the th smooth cluster, th texture where , , and cluster, and th edge cluster, respectively. The blurred image is divided into different regions based on its local statistics. A standard Sobel operator is used to extract the edges as they exhibit distinctive change of gray levels compared to blurred textured regions. To further differentiate smooth backgrounds from textured areas, the blurred image is uniformly restored for an iteration using a uniform mask. As smooth backgrounds in the partially restored image are mostly monotonous, we adopt the proposed measures in [28], namely, contrast, entropy, and local variance to extract spatial activity information from the image. An expert voting technique is employed to discriminate whether a pixel belongs to smooth or texture regions. These regions are further divided into non-neighboring clusters to preserve their homogeneity. Throughout HCM formation, small fragmented clusters arising from misclassification are incorporated into larger neighboring clusters by morphological operations. The number of HCM clusters formed relies on the outcome of image partitioning and does not need to be determined a priori. Investigations in [27] and [29]–[31] show that multivalue neuron modeling with the gradient descent method is efficient

520

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 2, FEBRUARY 2003

in restoring the image. The neuron updating rule in 2-D image indices is given by

where is the log likelihood function, and is the conditional probability density function of given the observafollows a multivariate Gaussian tion . Assuming that distribution, we can rewrite (29) in terms of the covariance ma: trix

(28) is the update increment, is the where to , connection weight from neuron is the self-bias, and is the image size. The detailed restoration procedures are outlined in [27, Sec. II]. A restoration iteration is performed essentially by updating the neuron values and are obtained from using (28), where (19)–(22). At the end of each iteration, cluster boundaries are adjusted dynamically according to the mean and average local variance of each cluster. This renders a continual improvement in the cluster structure and quality of the restored image. The restoration process continues until convergence or a maximum number of iterations is reached. Equation (28) reveals that the update of a neuron or pixel of dimension value involves a weight kernel for an blur size. The combination of support and one-hop updating reduces the time optimal per iteration. In complexity of our algorithm to contrast, other network-based restorations [23], [24] require per iteration. Therefore, a time complexity of the technique provides a considerable computational advantage over traditional network-based methods. Notwithstanding its advantages, the restoration step in Section III can be adapted using other restoration algorithms, hence providing flexibility in its implementation. IV. SOFT PARAMETRIC BLUR ESTIMATOR The main challenge in addressing blind image deconvolution is to identify and extract extra information to compensate for incomplete blurring knowledge. In particular, the information should be integrated in such a way that the flexibility of the algorithm is not compromised. Studies show that numerous practical blurs feature certain parametric forms. The most prominent structures include motion blur, out-of-focus blur, sinc-square blur, pillbox uniform blur, and Gaussian blur [2], [3], [8]. These studies provide good justification for utilizing the parametric information in our formulation. In this section, a blur estimator is devised to model the computed blur with the best-fit parametric structure from a predefined solution space . This encourages a soft learning approach where relevant information is incorporated into the blur identification continually. The underlying issues for the estimator include the formulation of an unbiased estimator and the construction of the solution space. As we are interested in finding the soft estimate given the computed blur, the maximum a posteriori (MAP) estimator is ideal for the task. The MAP estimator can be expressed formally as (29)

(30)

(31) If the blur coefficients are uncorrelated, we can further simplify (31) by letting

(32)

(33) It is observed that the estimator determines the best-fit parafrom the solution space . A value of metric model is used to model the moderate match between and . Nevertheless, it has little impart on the estimator especially for blurs with similar dimension as the first two terms in (33) are independent of . The construction of the solution space involves incorporation of blur knowledge domain with carefully chosen constraints. To preserve the average intensity of the original image, we impose the unity and non-negativity constraints on : (34) (35) The construction of the solution space is flexible as it can be tailored for general or specific applications. The standard parametric blurs or certain targeted forms can be integrated into the scheme for various purposes. In this paper, we will focus on 2-D standard blur structures for general applications, namely, uniform blur, Gaussian blur, and concentric linear blur. Uniform blur is the 2-D extension of 1-D motion blur and is characterized completely by its dimension. The Gaussian blur is widely observed in applications such as X-ray imaging and is difficult to estimate using traditional blur identification approaches. The 2-D linear blur is implemented as a first-order estimation to the generic blur.

YAP et al.: RECUSIVE SOFT-DECISION APPROACH TO BLIND IMAGE DECONVOLUTION

The solution spaces of uniform, Gaussian, and concentric and are odd dimensions, are linear blurs, assuming both , , and , respectively: given by

521

estimate in the solution space, suggesting a strong likelihood that it satisfies the parametric structure, a continual adapis introduced. On the other hand, if and tation toward differs significantly, the parametric information becomes irrelevant. Motivated by the concepts of fuzzy logic, a proximity measure is developed as a soft value or probability with regard to the relevance of the parametric information:

(36)

if

(41)

otherwise (37)

(38) is the blur dimension, and are the standard where deviation and normalizing constant of the Gaussian blur, and , are the gradient and central peak value of the linear blur. The solution space is constructed by taking the union of the targeted subspaces (39) where is the blur support size, and characterizing the th blur type.

where and are the scaling factors, and is the matching threshold. The proximity measure corresponds to two scenarios. If a satisfactory match occurs between and , a likelihood-based , proximity measure is employed. We used – , and – in our experiment. The scaling factors and are determined by ensuring that the contribution of the soft estimation term is an order less than the fidelity or regularization term for medium matches and the same order for close matches. Experimental simulations show that it is unnecessary to evaluate their optimal values as long as they have the right order of magnitude. The second scenario describes those irregular that do not resemble any parametric structures. As their chances of assuming parametric form are very slim, a zero value is appropriately assigned to these cases. The raised cosine function assigns weighted emphasis to different regions of the blur

is the parametric vector

V. BLUR IDENTIFICATION A. Formulation of Soft Modeling Error The primary objective of soft-decision approach is to evaluate the relevance of the parametric structure and integrate the information into the learning scheme accordingly. A soft is introduced to achieve this obmodeling error is its proximity jective. An important criterion of interpretation in multidimensional vector space. The Euclidean distance-based measure is adopted due to its simplicity and is proximity-discerning capability. The soft error proposed as (40) where is the proximity measure, and is the weighted blur emphasis. The functionality of the soft error is analogous to reinforced learning in neural networks, where is the adaptive is the weighted learning learning step size, and error. The flexibility lies in its ability to incorporate and adjust the relevance of the parametric model throughout the restoration. To achieve this objective, is devised as a confidence criestimation. If the current resembles the soft terion in our

otherwise

(42) and where is the roll-off factor. The inner coefficients of are emphasized because estimates of circumferential coefficients are prone to distortions by undesirable ringing in the restored image, particularly in the early stage of restoration. A is chosen to provide a reasonable weighting value of emphasis. B. Conjugate Gradient Optimization th blur identification The mathematical objective of the can be obtained by combining (4), (6), and (40) to give

(43) The cost function consists of three criteria, namely, the data conformance measure, the blur-domain regularization term, and the soft estimation error. The data conformance and regularization

522

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 2, FEBRUARY 2003

terms ensure the fidelity and piecewise smoothness of the solutions. Their relative contribution is controlled by the regularizadiag , with tion matrix

iv) Calculate the th iteration’s adaptive step size to update the conjugate vector

(44)

(49)

are the lower where is the local variance at , and and and upper limit of the blur-domain regularization parameter. We adopt a regularization parameter assignment scheme similar to that of (24) and (25) for simplicity. The proximity measure renders a compromise between the soft error and the rest of the . If the correlation between and improves criteria in will become dominant and gradually locks significantly, into one of the parametric structures. Otherwise, the soft error will wither away, and the combination of fidelity and regularization terms will determine the estimation. Conjugate gradient optimization is chosen to minimize (43) ahead of other gradient-based approaches such as Quasi-Newton and steepest descent methods. This is due to its lower computational complexity and storage requirement compared to Quasi-Newton and faster convergence with more robustness with respect to steepest descent technique. The -dimensional blur can be achieved convergence of an steps. In practice, the algorithm in a maximum of is terminated when convergence or a maximum number of -dimensional gradient of iterations is reached. The is denoted as , where is with respect to , and the partial derivative of is given by

(45) are the th scalar entries of where , , , , , , and , , , , , , and , respectively. The mathematical formulations of blur identification based on conjugate gradient optimization are summarized as follows. : i) Initialize the conjugate vector with the gradient (46) ii) Calculate the th iteration’s adaptive step size to update the blur (47)

is the Toeplitz matrix formed by where diag . iii) Compute the updated blur

, and

(48)

v) Update the new conjugate vector (50) vi) Repeat steps ii)–v) until convergence or a maximum number of iterations is reached. Following the approach outlined in [22], the centrosymmetric constraint is adopted as it improves the convergence, robustness, and performance of the proposed scheme (51) The overall complexity of the algorithm can be analyzed by considering various stages of the method, namely, cluster formation, image restoration, soft parametric estimation, and blur identification. The time complexities for each stage, in order, are , , , and , is number of deconvolution iteration, and is the where dimensionality of blur knowledge domain. As for , , , , and , the overall complexity of the algorithm is given . As by ignores the multiplicative constant by definition, the final com. By complexity of the algorithm is given by parison, it is noted that the image restoration and blur identification steps of SDR algorithm in [21] each has a complexity of . Therefore, the overall complexity of the SDR , which method is given by is similar to the complexity of our proposed algorithm. The additional computation of our algorithm due to soft blur estimation and cluster formation is much smaller . The small overhead than the overall complexity is well justified, considering the improvement of the method, which is to be illustrated in the next section. VI. EXPERIMENTAL RESULTS The proposed algorithm was employed to deconvolve several degraded images, and their results were compared with those obtained using the symmetric double regularization (SDR) method in [21]. A three-level HCM was constructed using , , , (19)–(25) with , and the pixel updating rule in Section III-C and was performed to achieve restoration. On the other hand, conjugate gradient optimization was used to identify the blur by minimizing the blur-domain cost function. The regularization and . parameters were taken as The chosen image and blur domain regularization parameters are consistent with widely accepted values [21], [23], [27]. The deconvolution process continued until convergence or a maximum number of recursions was reached. The experimental results given in the following subsections cover blind deconvolution of images degraded under different blurring functions and noise levels. Due to its subjectivity,

YAP et al.: RECUSIVE SOFT-DECISION APPROACH TO BLIND IMAGE DECONVOLUTION

523

human inspection is the best assessment of the restored images. A meaningful measure called normalized mean square-error (NMSE) is also used to evaluate the performance of the identified blur [19]

and

(b)

(d)

(e)

(c)

(52)

NMSE

where

(a)

are the true and estimated blur.

A. Blind Deconvolution of Image Degraded by Uniform Blur The blind deconvolution of an image degraded by a uniform blur is illustrated in Fig. 3. The original “Lena” image used in Fig. 3(a) has a dimension of 256 256 with 256 gray levels. It was degraded by a 5 5 uniform blur with some quantization noise to form Fig. 3(b). The proposed algorithm was applied to the degraded image. The final restored image and the identified blur are given in Fig. 3(c) and (e), respectively. We observe that the restored image recovers the fine details near the feather of the hat. In addition, it suppresses noise and ringing in the smooth backgrounds effectively. The blur estimate in Fig. 3(e) shows that the algorithm is successful in identifying the uniform blur. The restored image and identified blur obtained using the SDR approach are given in Fig. 3(d) and (f), respectively. Comparing the restored images in Fig. 3(c) and (d), it is observed that our approach is more effective in suppressing ringing near the background. The HCM restored image also renders good contrast between different image regions. Comparison between Fig. 3(e) and (f) indicates that our algorithm is more successful in identifying the uniform blur. The incorporation of blur knowledge domain has enhanced the capability of our approach to identify the uniform blur. B. Blind Deconvolution of Image Degraded by Gaussian Blur With Additive Noise This subsection presents blind deconvolution of an image degraded by Gaussian blur to demonstrate the flexibility of the proposed algorithm. The original image in Fig. 3(a) was degraded by a 5 5 Gaussian blur with a standard deviation of 2.0, followed by a 30-dB additive noise to form the noisy, blurred image shown in Fig. 4(a). We applied the same approach as before to deconvolve the degraded image. The restored image in Fig. 4(b) shows that the overall sharpness of the image has been recovered, in particular, near the textured feather regions of the hat as well as the edges. Besides, there is no visible ringing in the smooth backgrounds. The restored image using the SDR approach is given in Fig. 4(c). Comparison between Fig. 4(b) and (c) reveals that our algorithm is superior in suppressing ringing near shoulder and background while preserving the fine details near the feather of the hat effectively. The identified blurs using the proposed algorithm and the SDR technique produce an NMSE of 0.032 and 0.038, respectively. The smaller NMSE value of our result shows that it achieves a closer resemblance to the Gaussian blur.

(f)

Fig. 3. Blind deconvolution of image degraded by uniform blur. (a) Original image. (b) Degraded image. (c) Restored image using our approach. (d) Restored image using SDR approach. (e) Identified blur using our approach. (f) Identified blur using SDR approach.

(a)

(b)

(c)

Fig. 4. Blind deconvolution of image degraded by Gaussian blur with additive noise. (a) Degraded image. (b) Restored image using our approach. (c) Restored image using SDR approach.

C. Blind Deconvolution of Image Degraded by Nonstandard Blur With Additive Noise We illustrate the capability of the proposed algorithm to handle nonstandard blur in Fig. 5. The original image in Fig. 3(a) was degraded by the 5 5 nonstandard exponential blur, followed by a 30-dB additive noise to the form the degraded image in Fig. 5(a). The exponential blur is given by , where is the decay is the normalizing constant to ensure condition factor, and (34) is satisfied. The proposed algorithm was applied to obtain the restored image in Fig. 5(b). It is clear that the algorithm is effective in restoring the image by providing clarity in the fine textured regions while suppressing noise and ringing in the smooth backgrounds. In comparison, the restored image using SDR method in Fig. 5(c) shows deterioration in terms of ringing and noise amplification in the smooth backgrounds. The identified blur using our algorithm yields an NMSE of 0.048 compared with 0.12 obtained by the SDR method. D. Blind Deconvolution of Degraded Flower Image We illustrate the flexibility of the proposed method by performing blind deconvolution on a new image. The original “Flower” image in Fig. 6(a) was degraded by a 5 5 Gaussian blur with a standard deviation of 2.0, followed by a 30–dB additive noise to form the noisy, blurred image shown in Fig. 6(b). The proposed algorithm was repeated as before to yield the restored image in Fig. 6(c). It is observed that the restored image

524

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 2, FEBRUARY 2003

It reconciles the dilemma faced by traditional blind schemes where a hard-decision on the certainty of blur structure has to be made before the algorithmic formulation. It is observed from experimental simulations that the method is effective in blind deconvolution of images under different circumstances. (a)

(b)

(c)

Fig. 5. Blind deconvolution of image degraded by a nonstandard exponential blur with additive noise. (a) Degraded image. (b) Restored image using our approach. (c) Restored image using SDR approach.

APPENDIX A CONVERGENCE ANALYSIS OF RECURSIVE DOMAIN OPTIMIZATION (ALTERNATING MINIMIZATION) We introduce a sequence by listing the consecutive cost functions (53) , , , . The following characteristics on the sequence observed: where

(a)

(b)

, are (54)

consists of norms, with and This is because being the diagonal matrices with positive entries and the proximity measure being non-negative by definition or equivalently and (c)

(d)

Fig. 6. Blind deconvolution of degraded Flower image. (a) Original image. (b) Degraded image. (c) Restored image using our approach. (d) Restored image using SDR approach.

recovers most of the visual information that is lost through the blurring process. On the other hand, the SDR restored image in Fig. 6(d) experiences ringing and noise amplification near the petal of the flower. The satisfactory results illustrate that the proposed technique is useful in blind deconvolution of images degraded under different circumstances, namely various blurring functions and noise levels. VII. CONCLUSIONS This paper proposes a soft-decision blur identification and neural network approach to adaptive blind image deconvolution. The technique incorporates blur knowledge domain without compromising its flexibility in restoring images degraded by other nonstandard blurs. A cost function consisting of a data fidelity measure, image, and blur-domain regularization terms, and a soft estimation error is proposed. An alternating minimization process is adopted where the cost function is projected and optimized with respect to the image and blur domains. The hierarchical cluster model is used to achieve perception-based restoration by minimizing the image-domain cost function. The blur identification is accomplished by applying conjugate gradient optimization to minimize the blur-domain cost function. The technique models the current blur with the best-fit parametric structure, assesses its relevance by evaluating the proximity measure, and incorporates the weighted soft estimate into blur identification.

(55)

are projected and The inequality in (55) holds true as minimized repeatedly with respect to its image and blur domains. This is equivalent to alternating minimization by part of the overall cost function. The combination of (54) and (55) gives rise to (56) based on Therefore, there exists a limit well-known limit existence theorem. To prove the convergence of the proposed algorithm, the following definition for convexity is used [32]: Definition 1 (Convex Function): A function : is said to be convex when, for all and all (57) . holds. The class of such functions is denoted by and can be shown to be The cost functions quadratic with respect to and , respectively. Therefore, it foland are convex [32]. As we have shown lows that corresponding to the limit , that there exists a vector ( , , it is apparent from the conand vexity property that the proposed algorithm will converge to a minima: local or global. Simulation results confirm with theoretical analysis. A thorough treatment of convergence analysis on general AM is given in [33]. APPENDIX B LIST OF KEY SYMBOLS Table I shows a list of key symbols used in this paper.

YAP et al.: RECUSIVE SOFT-DECISION APPROACH TO BLIND IMAGE DECONVOLUTION

525

TABLE I LIST OF KEY SYMBOLS

REFERENCES [1] H. C. Andrews and B. R. Hunt, Digital Image Restoration. Englewood Cliffs, NJ: Prentice-Hall, 1977. [2] R. C. Gonzalez and R. Woods, Digital Image Processing. Reading, MA: Addison-Wesley, 1992. [3] M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE Signal Processing Mag., vol. 14, pp. 24–41, Feb. 1997. [4] M. Bertero, T. Poggio, and V. Torre, “Ill-posed problems in early vision,” Proc. IEEE, vol. 76, pp. 869–889, Aug. 1988. [5] M. M. Chang, A. M. Tekalp, and A. T. Erdem, “Blur identification using the bispectrum,” IEEE Trans. Signal Processing, vol. 39, pp. 2323–2325, Oct. 1991. [6] R. Fabian and D. Malah, “Robust identification of motion and out-offocus blur parameters from blurred and noisy images,” CVGIP: Graphical Models Image Process., vol. 53, pp. 403–412, Sept. 1991. [7] S. K. Kim and J. K. Paik, “Out-of-focus blur estimation and restoration for digital auto-focusing system,” Electron. Lett., vol. 34, pp. 1217–1219, June 1998. [8] D. Kundur and D. Hatzinakos, “Blind image restoration,” IEEE Signal Processing Mag., pp. 43–64, May 1996. [9] R. L. Lagendijk, J. Biemond, and D. E. Boekee, “Identification and restoration of noisy blurred images using the expectation–maximization algorithm,” IEEE Trans. Acoust. Speech, Signal Processing, vol. 38, pp. 1180–1191, July 1990. [10] K. T. Lay and A. K. Katsaggelos, “Image identification and restoration based on expectation–maximization algorithm,” Opt. Eng., vol. 29, no. 5, pp. 436–445, May 1990. [11] A. M. Tekalp, H. Kaufman, and J. Woods, “Identification of image and blur parameters for restoration of noncausal blurs,” IEEE Trans. Acoust. Speech, Signal Processing, vol. ASSP-34, pp. 963–972, 1986. [12] S. J. Reeves and R. M. Mersereau, “Blur identification by the method of generalized cross-validation,” IEEE Trans. Image Processing, vol. 1, pp. 301–311, July 1992.

[13] G. R. Ayers and J. C. Dainty, “Iterative blind image deconvolution method and its applications,” Opt. Lett., vol. 13, pp. 547–549, July 1988. [14] B. C. McCallum, “Blind deconvolution by simulated annealing,” Opt. Commun., vol. 75, pp. 101–105, Feb. 1990. [15] D. Kundur and D. Hatzinakos, “A novel blind image deconvolution scheme for image restoration using recursive filtering,” IEEE Trans. Signal Processing, vol. 46, pp. 375–390, Feb. 1998. [16] C. A. Ong and J. A. Chambers, “An enhanced NAS-RIF algorithm for blind image deconvolution,” IEEE Trans. Image Processing, vol. 8, pp. 988–992, July 1999. [17] M. K. Ng, R. J. Plemmons, and S. Qiao, “Regularization of RIF blind image deconvolution,” IEEE Trans. Image Processing, vol. 9, pp. 1130–1134, June 2000. [18] H.-T. Pai and A. C. Bovik, “On eigenstructure-based direct multichannel blind image restoration,” IEEE Trans. Image Processing, vol. 10, pp. 1434–1446, Oct. 2001. [19] G. B. Giannakis and R. W. Heath, “Blind identification of multichannel FIR blurs and perfect image restoration,” IEEE Trans. Image Processing, vol. 9, pp. 1877–1896, Nov. 2000. [20] S. U. Pillai and B. Liang, “Blind image deconvolution using a robust GCD approach,” IEEE Trans. Image Processing, vol. 8, pp. 295–301, Feb. 1999. [21] Y.-L. You and M. Kaveh, “A regularization approach to joint blur identification and image restoration,” IEEE Trans. Image Processing, vol. 5, pp. 416–428, Mar. 1996. [22] T. F. Chan and C.-K. Wong, “Total variation blind deconvolution,” IEEE Trans. Image Processing, vol. 7, pp. 370–375, Mar. 1998. [23] Y.-T. Zhou, R. Chellappa, A. Vaid, and B. K. Jenkins, “Image restoration using a neural network,” IEEE Trans. Acoust. Speech, Signal Processing, vol. 36, pp. 1141–1151, July 1988. [24] J. K. Paik and A. K. Katsaggelos, “Image restoration using a modified network,” IEEE Trans. Image Processing, vol. 1, pp. 49–63, Jan. 1992.

526

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 2, FEBRUARY 2003

[25] J. A. Anderson and J. P. Sutton, “A network of networks: Computation and neurobiology,” in Proc. World Congr. Neural Networks, vol. 1, 1995, pp. 561–568. [26] J. P. Sutton, “Hierarchical organization and disordered neural systems,” Ph.D. dissertation, Univ. Toronto, Toronto, ON, Canada, 1988. [27] S. W. Perry and L. Guan, “Weight assignment for adaptive image restoration by neural networks,” IEEE Trans. Neural Networks, vol. 11, pp. 156–170, Jan. 2000. [28] J. S. Weszka, C. R. Dyer, and A. Rosenfeld, “A comparative study of texture measures for terrain classification,” IEEE Trans. Syst., Man, Cybern., vol. SMC–6, pp. 269–285, Apr. 1976. [29] L. Guan, “Model-based neural evaluation and iterative gradient optimization in image restoration and statistical filtering,” J. Electron. Imag., vol. 4, pp. 407–412, Apr. 1995. [30] L. Guan, J. A. Anderson, and J. P. Sutton, “A network of networks processing models for image regularization,” IEEE Trans. Neural Networks, vol. 8, pp. 169–174, Jan. 1997. [31] H.-S. Wong and L. Guan, “Adaptive regularization in image restoration using a model-based neural network,” Opt. Eng., vol. 36, pp. 3297–3308, Dec. 1997. [32] J. B. Hiriart-Urruty and C. Lemarecal, Convex Analysis and Minimization Algorithms I. New York: Springer-Verlag, 1993. [33] T. F. Chan and C. K. Wong, “Convergence of the alternating minimization algorithm for blind deconvolution,” Linear Algebra Appl., vol. 316, pp. 259–285, Sept. 2000.

Kim-Hui Yap (S’99) received the B.Eng. and Ph.D. degrees in electrical engineering from the University of Sydney, Sydney, Australia, in 1998 and 2002, respectively. Currently, he is a faculty member at Nanyang Technological University, Singapore. His research interests include adaptive image processing, computational intelligence, and multimedia signal processing.

Ling Guan (S’88–M’90–SM’96) received the Ph.D. degree in electrical engineering from University of British Columbia, Victoria, BC, Canada in 1989. From 1989 to 1992, he was a Research Engineer with Array Systems Computing Inc., Toronto, ON, Canada, where he was involved with machine vision and signal processing. From October 1992 to April 2001, he was a faculty member at University of Sydney, Sydney, Australia. Since May 2001, he has been a Professor with the Department of Electrical and Computer Engineering, Ryerson University, Toronto, where he was subsequently appointed to the position of Canada Research Chair. In 1994, he was a visiting fellow at British Telecom. In 1999, he was a visiting professorial fellow at Tokyo Institute of Technology, Tokyo, Japan. In 2000, he was on sabbatical leave at Princeton University, Princeton, NJ. His research interests include multimedia processing and systems, optimal information search engines, signal processing for wireless multimedia communications, computational intelligence and machine learning, and adaptive image and signal processing. He has published more than 150 technical articles and is the editor/author of two books: Multimedia Image and Video Processing and Adaptive Image Processing: A Computational Intelligence Perspective. Dr. Guan is an associate editor of the IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION and several other international journals. In 1999, he co-guest-edited the special issues on Computational Intelligence for PROCEEDINGS OF THE IEEE. He also serves on the editorial board of CRC Press’ Book Series on Image Processing. He has been involved in organizing numerous international conferences. He organized the inaugural IEEE Pacific-Rim Conference on Multimedia and served as the Founding General Chair in Sydney 2000. He is a member of IAPR and SPIE. He is currently serving on IEEE Signal Processing Society Technical Committee on Multimedia Signal Processing.

Wanquan Liu received the B.A. degree in mathematics from Qufu Normal University, Qufu, China, in 1985, the M.S. degree in control theory and operation research from Institute of Systems Sciences, Chinese Academy of Science, Beijing, China, in 1988, and the Ph.D. degree in electrical engineering from Shanghai Jiaotong University, Shanghai, China, in 1993. From 1994 to 1998, he was with the University of Western Australia, Perth, as a Research Associate. Between 1998 and 2000, he was with the University of Sydney, Sydney, Australia, as a U2000 Fellow and then as an ARC Research Fellow. Since 2000, he has been with the School of Computing, Curtin University, Perth. He has over 60 publications in control, optimization, and signal processing in reputed journals and conference proceedings. He has attracted several research grants from different sources in China and Australia. Currently, he is doing research on image processing and network communication.