Pattern Learning Based Image Restoration Using ... - Semantic Scholar

Pattern Learning Based Image Restoration Using Neural Networks Dianhui Wang+, Tharam Dillon+ and Elizabeth Chang* +

Department of Computer Science and Computer Engineering La Trobe University, Melbourne, VIC 3083, Australia * Department of Computer Science and Software Engineering Newcastle University, Australia

Abstract -This paper presents a generic pattern learning based image restoration scheme for degraded digital images, where a feed-forward neural network is employed for implementation of the proposed techniques. The methodology reported in this paper can be applied in different circumstances, for instance, quality enhancement as a post-processing of image compression schemes, blur image restoration and noise image filter, provided that the training data set is comprised of patterns rich enough for supervised learning, This paper focuses on the problem of coded image restoration. The key points addressed in this work are (1) the use of edge information extracted from source images as a priori knowledge in the regularization function to recover the details and reduce the ringing artifact of the coded images; (2) the theoretic basis of the pattern learning-based approach using implicit function theorem; (3) subjective quality enhancement with the use of an image similarity for training neural networks; and (4) empirical studies with comparisons to the set partitioning in hierarchical tree (SPIHT) method. The main merits of this model-based neural image restoration approach include strong robustness with respect to transmission noise and the parallel processing for real-time applications. The experimental results demonstrate promising performance on both objective and subjective quality for lower compression ratio subband coded images.

I. INTRODUCTION A degraded image may be caused by various factors such as atmospheric turbulence, distortions in the optical imaging system, lack of focus, sensor or transmission noise injection, coding techniques, and object or camera motion. The task of image restoration is to remove these degradations to enhance the quality of the image for further use in domain applications. Image restoration can be defined as a problem of estimating a source image from its degraded version. In the past, to solve this fundamental and important issue for image processing, considerable studies have been carried out using transform related techniques and algebraic approaches. The techniques involving an iterative method to minimize a degradation measure attracted many researchers and recently different models and approaches were developed such as maximum likelihood, constrained least square error and Kalman filter (see the references in [3], [10], [11]). To implement the minimization tasks for image restoration, however, we should take into account several practical factors, such as real-time requirement, or even many of the numerical optimization techniques can be used. Artificial neural networks, or simple neural networks, can be defined as a massively parallel and distributed processor that has a natural propensity for storing and recalling experiential knowledge [4], [5]. As potential tools, neural networks have

0-7803-7278-6/02/$10.00 ©2002 IEEE

been successfully applied in image processing mainly due to the ability to generate and recall an internal data representation through pattern samples learning [6],[7],[8],[9]. Some encouraging results on image restoration using neural networks have been reported in literatures [10],[11],[12]. So far, almost all of the related work used Hopfield network to find a stable point as the solution for a constrained least square error measure with a regularization term, which plays an important role in controlling the quality of restored image. If the regularization term is weighted too weaker in the error measure, the resultant restored image will contain noise artifacts. On the other hand, if the regularization measure is weighted too strongly in the error measure, the resultant restored image will be bluffed [2],[3]. Several approaches have been developed to adaptively vary the regularization parameter to achieve the optimal balance between removing edge ringing effects and suppressing noise amplification [10]. [13] presented an image restoration technique using neural networks, where a priori knowledge of the image dependent edge information was incorporated into the regularized error measure to improve the upper bound estimation of the high frequency content. A multiplayer perceptron model with single hidden layer architecture and a standard error backpropagation algorithm was employed. Although the results reported in [13] are interesting and promising, there are still some points to be emphasized and further studied. These include a proper statement of the Theorem, a modification of the cost function used for training neural networks based similarity learning, and a generalization of restoration model coded images. This paper aims at developing a pattern learning based neural image restoration technique for generic coded images. The edge information as the a priori knowledge about the source image to recover the details as well as to reduce the ringing artifact of the coded image is introduced in the regularization model to enhance the restoration performance both objectively and subjectively. A nonlinear relationship with some uncertainties among the restored image, degraded image and the edge information extracted from the source image is derived from the normal equation. It has been proved that a solution for restored image exists which is unique to certain classes of images, for given degraded image and edge information. To improve the subjective performance, an image similarity measure is applied for updating weights in neural networks. The remainder of the paper is organized as follows. Section II develops a generic image restoration model for coded images and establishes a mathematical basis for the proposed approach. Section III gives a detailed description of

the learning algorithm. Experimental results and discussions are reported in Section IV, and the last section concludes this work. II. MATHEMATICAL BASIS FOR PATTERN LEARNING BASED NEURAL IMAGE RESTORATION A. Generic Image Restoration Model Usually, the relationship between a degraded image g ( x, y )

and its original image f ( x, y ) can be expressed by g ( x, y ) = L[ f ( x, y )] + N ( x, y )

(1)

where L is a matrix operator, i.e., a linear mapping, and N ( x, y ) is an additive noise, respectively. The objective of image restoration can be formulated as “ to get an estimated image fˆ ( x, y ) from some class of images such that it will minimize the noise function N ( x, y ) ”. One popular approach for implementing this is to minimize the following objective function [3]:

J 0 = λ S ( fˆ )

2

+ g − L( fˆ )

2

(2)

where S is a regularizing operator of a appropriate dimension, which is generally a high-pass filter used to reduce the amount of noise in the restored image, x denotes the Euclidean norm of x , λ is a constraint factor which control the degree of smoothness of the restored image. Note that the linearity assumption on mapping L is not necessary although the model in (1) characterizes some image restoration problems. In this paper, we generalize the image restoration model (1) by replacing the linear mapping L with an unknown operator P (so it can be views as a nonlinear mapping in mathematics!), which may represent any distortion operation or process, for example, it can be a coding algorithm, spatial shift or an additive noise from sensor. To obtain a better upper bound in estimating the high frequency content, we also introduce a priori knowledge of the image dependent edge information in the objective function [2]. The modified objective function is given by

J = λ S ( f − fˆ )

2

+ g − P( fˆ )

2

(3)

As can be seen any numerical optimization techniques will lose their power in solving (3) due to the presence of the unknown nonlinear mapping P. To understand the framework better, we now show an illustrative example where the nonlinear mapping arises from subband coding algorithms. Let the source image f be decomposed into several narrow subbands by passing through an analysis filter bank. Subband images are then sub-sampled, and coefficients of different subbands are quantized with respect to different scalar constants and rounded to the nearest integer. The quantized coefficients are then entropy coded and transmitted. At the receiving side, the entropy-coded coefficients are decoded first and then de-quantized. The dequantized coefficients are up-sampled and filtered for interpolation with a synthesis filter bank. Finally, they are combined to form the reconstructed image. Mathematically,

0-7803-7278-6/02/$10.00 ©2002 IEEE

the aforementioned processing can be expressed as follows: Denote the source image as f, the wavelet transform operator W, the encoding operator E, the corresponding decoding operator D and the inverse wavelet transform operator W −1 . Then, a distortion operator H for the subband coding scheme can be expressed by (4) H = W −1  D  E  W where the symbol “  ” represents the composite operator. The operator D  E can be rewritten as D  E = Ι + Φ +ζ (5) where I represents the identity mapping, Φ represents an image dependent smooth nonlinear mapping with small norm, and ζ represents an error mapping. Thus, we have (6) H = I + Ψ + ξ = Hˆ + ξ

where Hˆ = I + Ψ consists of the identity mapping and a derived smooth nonlinear mapping Ψ = W −1  Φ  W with small norm, and ξ = W −1  ζ  W is a derived error mapping from ζ.. After going through the whole process, we can obtain the degraded image for a given image. This may be characterized by a nonlinear image degradation system: (7) Hˆ ( f ) = g + n where g, f and n are the degraded image, the source image and the additive noise, respectively. B. Mathematical Basis for Use of Neural Networks In this subsection, we establish a mathematical basis for use of the feed-forward neural networks to solve the optimization problem in (3). Minimizing J with respect to fˆ leads to the following normal equation: G ( fˆ , g , S t Sf ) = λS t S ( fˆ − f ) + ∇P t ( fˆ )[ P ( fˆ ) − g ] = 0

(8)

where M t denotes the transpose of matrix M, ∇P( fˆ ) is the Jacobian of the nonlinear mapping P with respect to variable fˆ . The term S t Sf in (8) is simply an edge extraction of the source images as S is taken as a Laplacian mask. For any given pair p={ g , S t Sf }, the corresponding solution of (8) offers a restored image. To proceed with our work, the following basic assumption is necessary and reasonable: ASSUMPTION: The unknown nonlinear mapping P in (3) can be written in a form as P = I + Ψ , where Ψ is an uniform bounded nonlinear functional mapping, that is, there exists a real positive constant δ such that for all fˆ || Ψ ( fˆ ) ||≤ δ

(9)

Also, let c > 0 be the uniform bound of the difference between the degraded image g and the restored one fˆ , that is, (10) || fˆ − g ||≤ c The following result is important and forms the basis of our work. Indeed, it characterizes a class of images in which the restored image for any given pair p={ g , S t Sf } is uniquely determined.

THEOREM: Let ρ ( M ), det( M ) and || M || E be the smallest singular value, determinant and the 2-norm of a matrix M, respectively. Denote that ℜ = {x :||

∂∇Ψ t ( x) λρ ( S t S ) || E < } ∂x c+δ

(11)

Then, for any fˆ ∈ ℜ and q = ( fˆ , g , S t Sf ) satisfying (8), we have (12) ∂G (q) )≠0 det( ∂fˆ Proof: From functional analysis, it is easy to know that (11) 2

defines a compact set in N -dimensional Euclidean space. To show the inequality (10), we calculate ∂∇Ψ t ( fˆ ) ∂G = λS t S + ∇P t ( fˆ )∇P( fˆ ) + [ P ( fˆ ) − g ] ˆ ˆ (13) ∂f ∂f Note that the operator S is a nonsingular matrix, so the term λS t S + ∇P t ( fˆ )∇P( fˆ ) is a positive matrix, and the following holds for any λ > 0 and all fˆ , that is (14) ρ[(λS t S ) −1 ] ≥ ρ[( λS t S + ∇P t ( fˆ )∇P( fˆ )) −1 ] Hence, for any fˆ ∈ ℜ and q = ( fˆ , g , S t Sf ) satisfying (8), we have ∂∇Ψ t ( fˆ ) ∂∇Ψ t ( fˆ ) || [ P ( fˆ ) − g ] || E ≤ (c + δ ) || || E ∂fˆ ∂fˆ (15) t t t ˆ ˆ < λρ ( S S ) ≤ ρ[λS S + ∇P ( f )∇P ( f )], which ensures the inequality (12). This completes the proof. By the implicit function theorem [14], we know that the solution, if any, of equation (8) for given pairs p is unique in ℜ . This implies that for certain classes of images the normal equation (8) may be explicitly rewritten as (16) fˆ = F ( g , S t Sf ) where F is a nonlinear continuous mapping. As the mapping F in (16) is available, the restored image may be calculated directly for any given pair p={ g , S t Sf }. So, we focus here on building an approximation of this unknown nonlinear mapping, denoted by N F , through pattern learning approach. Note that the N F obtained after sufficient training will only offer an expectation solution [4]. This is because of different original images may result in the same pair p={ g , S t Sf }. III. ALGORITHM DESCRIPTION The following provides an outline of the pattern learning based neural image restoration algorithm. It has been shown that the multiplayer perceptron networks (MLPN) with a single hidden layer and a nonlinear activation function can arbitrarily approximate any continuous nonlinear mappings on a compact set [15]. This forms the theoretical foundation for the domain application. The input information is fed to the input layer and distributed to the hidden layer and 0-7803-7278-6/02/$10.00 ©2002 IEEE

finally to the output layer via weighted connections. Each neuron in the network operates by taking the sum of its weighted inputs and passing the result through a nonlinear activation function. In this paper, the hyperbolic tangent function and the linear function are used as the activation functions in the hidden layer and the output layer, respectively. The linear activation function is only used in training phase and a hard limit is needed in the image restoration phase. The inputs of the neural network are the edge information of the source image and the degraded image, and the output of the neural model is the estimated image. Suppose that the image to be restored consists of N × N array of pixels. Thus, the neural network model should consist of 2 × N × N and N × N nodes at input and output layers, respectively. A remarkable drawback to this approach for realizing image restoration via a MLPN model is that the training time is prohibitively long and the training patterns required to build such a large network model may not be easily available. Note that the neural networks are usually trained on image data so that it develops an internal representation corresponding not to the image itself, but rather to the local statistic patterns and features of a class of images, since the restoration of images highly depends on these local patterns. Thus, instead of using the whole image related information as the training set, we partition the image (scene) into M × M disjoint sub-scenes and generically denote them as Rc for the degraded images and S f for the corresponding edge image, respectively. Each sub-scene contains p1 × p1 -dimensional pixel patches. These pixel patches are subsequently used as training patterns for the network model. Precisely, the training set is (17) {( R c (k ), S f (k )) → I m (k ), k = 1 : PLM } where PLM = L × M × M , L is the number of source images used in training set. A r -dimensional column vector, V (k ) = [a1 (k ), a 2 (k ), l , a r (k )]t , r = 2 × p1 × p1 , derived from the block sequence ( Rc (k ), S f (k )) forms the input vector to the neural network. The target output vector is denoted by I m (k ) = [T1 (k ), T2 (k ),m , Tr / 2 ]t , which is generated by the source images. To enhance the subjective performance, the neural network is trained by using the following image similarity measure (ISM) [17]: u

S=

v

∑∑ p =1 q =1

∆g ( p , q ) [δ ( p − m x )] 2 + [δ (q − m y )] 2 + ∆g ( p, q) 2 1+ δ

where u = v = r / 2 , δ is an adjustable parameter relating to the brightness of the images. If the average of all gray level intensities does not differ much it can be set to one; the pair (m x , m y ) is defined as an intensity center of the image, which satisfies the following relationships: mx

|

v

∑∑

u

∆g ( i , j ) −

i =1 j =1 mx

|

i =1 j =1

∑∑ ∆g (i, j) |⇒ min

i = m x j =1

v

∑∑

v

u

∆g ( i , j ) −

v

∑∑

i = m x j =1

∆g (i, j ) |⇒ min

(18)

where ∆g (i, j ) =| N F (i, j ) − I m (i, j ) | is the absolute difference between the neural network output block and the target block images, calculated pixel by pixel, and i, j represents the row and column index of the image. IV. EXPERIMENTAL RESULTS AND DISCUSSIONS Experiments have been carried out to evaluate the performance of the proposed approach in this paper. We used five standard images of resolution 256×256 pixels with eight bits per pixel (bpp) to generate the training data. The image Lenna (outside the training set) is employed as the test image for stopping the neural network training process. We used a block size of 4x4, i.e., p=4, with a total number of 40960 training patterns in our computer experiments. These training patterns were derived from the images Barbara, Airplane, Sailboat, Cameraman, Bridge, Building. The test images are used as Baboon, Goldhill, Germany, Lenna, and Peppers. In this work, a MLPN model with architecture 32-H-16 is employed to restore images blocks by blocks, where H represents the number of neurons at hidden layer. The authors belive it has less practical and theoretical results to effectively estimate the number H so that a given neural network gives good potential for both learning and generalization. By using similar arguments as used in [8] for considering the tradeoff between Generalization Constant (GC) and the Network Capacity (NC) [5], an estimated value, H=30, has been selected. It is worth of noting that before training commenced, the images data and the edge information were normalized so that all pixel values resides in the interval [-1,1]. R

MSE =

∑

1 ( || I o (k ) − I m (k ) || 2 ) K × R k =1

PSNR = 10 log10 (

255 2 ) (dB) MSE

(19)

A 3×3 Laplacian filter is used as the regularization operator S. SPIHT [1], which is one of the outstanding wavelet based codes, was used to code the image. As can be seen there exists a rate budget between the amount of transformed image data and the a priori edge knowledge. A GFA-based edge bitplanes coding scheme is used to compromise the amount of information carried by the image data and the corresponding edges and to ensure an adaptive image-independent encoding of the edge image [18]. Extensive experiments on the robustness of the image restoration with respect to the amount of edge data were conducted. It showed when the amount of the edge data is at about 30 percent of the total rate, the restored image would achieve the best perceptual quality and the best PSNR results. The amount of edge data could be controlled by varying the threshold parameter in the edgecoding scheme. An arithmetic coder was used as the entropy coder throughout the experiment. The mean square error (MSE) calculation was used as a measure of the performance for the training procedure, which was defined by (19), where K and R are taken as 16 and 4096 for the test set , and 16 and 40960 for the training set, respectively. The MSE is decreased at the beginning of the training phase for both the training set and the test set. However, at a certain point in the training, it begins to increase slightly for the test image although it may continue to decrease for the training data. Such a point is termed as the saturation point (SP). If the training work is continued after this point, the MSE performance on the test image becomes worse, which implies a decrease of the peak signal-to-noise ratio (PSNR) defined by (20). Therefore, the cross-validation scheme is employed in the training procedure to prevent over-fitting and terminate the training process when the SP is reached. Upon completing the training phase, the trained neural network can be applied to restore images.

(20) TABLE 1: OBJECTIVE PERFORMANCE EVALUATION AND COMPARISON FOR DIFFERENT COMPRESSION RATES PSNR(dB) for bit-rate Images

Figure 1. Part of the source images serving as the training data set: Airplane; Sailboat; Cameraman and Bridge

0-7803-7278-6/02/$10.00 ©2002 IEEE

at 0.25 bpp

PSNR(dB) for bit-rate at 0.5 bpp

SPIHT

NN

SPIHT

NN

Barbara*

23.83

24.67

26.57

26.96

Airplane*

26.29

25.91

29.88

30.07

Sailboat*

25.12

25.00

27.65

28.13

Cameraman*

26.95

26.60

30.46

30.61

Bridge*

24.09

23.88

26.04

26.19

Goldhill

26.85

27.00

29.17

29.26

Germany

30.78

30.72

33.25

33.20

Lenna

28.55

28.58

32.41

32.66

Baboon

20.84

21.52

22.54

23.41

Peppers

27.98

27.88

31.49

31.37

Table 1 shows the performance comparison of the proposed approach along with SPIHT on several images selected from both inside and outside the training data. In Table 1, the

images marked * were from inside the training data and others were completely outside the training data. Although these test images may have quite different statistics from the training data, the neural network still works well on them. This implies that the MLPN model has good potential for image restoration through pattern learning. Typically, PSNR is not a good metric for subjective measurement since the PSNR relates poorly to the responses of human visual system. In comparing the subjective performance of the two approaches, human objective evaluation is best. To evaluate the potential of the edge information used in our approach, we investigated the following aspects: edges representation via weights in the neural network, function and the effect on the PSNR performance. Firstly, let Rc in the input vector be null (i.e., zero component). The corresponding output images then can be observed as in Figure 2(b) for Barbara. Clearly, the edge information has been well stored and represented internally by neural network. Secondly, we ignore the edge information (i.e., take the S f as null vector in training patterns) and re-train the neural network by using the degraded image alone. A visual comparison of the results on Barbara at 0.25 bpp was shown in Figure 2(c) and (d). As can be seen in Figure 2 the function of edge information used in our approach is obvious, that is, it made the edges sharper in the reconstructed images. Lastly, we fixed the degraded image data amount at 0.25*70% bpp and then observed the effect on the PSNR performance when the edge information amount is shifted. The obtained results indicated that the PSNR performance may be greatly improved by increasing the edge information. However, this will result in an increase in the total compression rate. An optimal design method for assigning the bit-rate in both training data and test images is important so that the proposed approach can achieve higher performance. Further study on this topic is being carried out. It is also important to study the robustness of the neural image restoration algorithm with respect to the effect of input noise, weight changes and hidden nodes damage. This is because it is directly related to transmittal noise and hardware implementation issues. The image Baboon was employed here for this investigation. We first added a white noise with certain variance ρ into the input vectors. The restored image with ρ = 1 was shown in Figure 3(a). Although the performances were logically affected by the additive noise, they were still acceptable both subjectively and visually. Next, we tested the robustness of the neural network with respect to weight changes. A certain percentage of the trained weights were selected at random, and the values were changed within some specified tolerance ranges. As expected, the changed weights in neural model yielded lower PSNR performance. The images in Figure 3(b) and (c) shown the visual difference between the actual system output and the output obtained by altering all of the weights within a 10% tolerance range. In order to study the robustness of the system with respect to the network architecture, a number of hidden nodes were removed after training and the performances were calculated for some images. It was observed that the fidelity of the reconstructed 0-7803-7278-6/02/$10.00 ©2002 IEEE

image is greatly degraded as some hidden nodes are deleted. As can be seen in Figure 3(d), the image is still recognizable even up to a 20% removal of hidden nodes.

(a)

(b)

(c) (d) Figure 2. (a) Original image “Barbara”, (b) Reproduced edge image from NN model, (c) Restored image with CR=0.5 bpp, PSNR=26.96, (d) Restored image with CR=0.5 bpp and without using edge information, PSNR=23.94.

(a)

(b)

(c) (d) Figure 3. Robustness performance of the proposed approach for test image “Baboon” with CR=0.25bpp. (a) Restored image with input noises, PSNR=21.03, (b) Restored image without any perturbations, PSNR=21.52, (c) Reconstructed image with all weights perturbation within 10% tolerance range, PSNR= 19.67, (d) Restored image with deleted 6 hidden nodes, PSNR=15.28.

Lastly, we put some words in the regularizing factor. For fixed bounds in (9) and (10), if the regularizing factor in (11) is larger, the images in ℜ will have a little more sharp edges. This conforms to the implication expressed by (3), where a larger regularizing factor implies a heavier weight on the edge information. Then, how can we logically incorporate the meaningful regularizing factor into our pattern learning based image restoration approach? This problem still remains open and a further investigation on this issue is pending. V. CONCLUSION This paper presents a novel neural image restoration scheme by using machine learning approach. The feed-forward neural networks and an image similarity measure are employed to implement the algorithm. The proposed approach incorporates a priori knowledge about the edge information of the source image in reconstructing a higher quality image by means of minimizing a regularization model. The image restoration task can be views as a function approximation problem considering the images as points in a high dimensional space. It is important to realize that the MLP outputs would converge to the posteriori probability as the training data is sufficiently large. Therefore, the neural networks indeed approximate the expectation through training. The main purpose of this paper is to establish a mathematical foundation for the use of the neural networks, and also to result characterizing the class of restored images and the property of the regularizing factor. This new image restoration technique has potential in real-time applications because it can be effectively implemented by hardware. The neural network model used shows robustness with respect to not only the transmission noise and weight perturbations, but also the network model architecture. The robustness of this neural image restoration algorithm with presence of noise and uncertainties gives it reliable and practical approach. The scheme is evaluated using objective PSNR and the subjective visualization. The experiments demonstrate that performance comparable to SPIHT scheme can be achieved on both the objective and subjective quality for low compression ratio coded images. The experimental results also show that the image quality can be improved subjectively by reducing most of the ringing artifact and preserving more edge information. In particularly it works well on images with "random" texture, such as Baboon. Moreover, the trained images are independent of the trained neural network model as long as the learning patterns are sufficiently rich, spatially independent and representative of variety of images. REFERENCES [1] A. Said and W. A. Pearlman, A new, fast, and efficient image codec

[2]

based on set partitioning in hierarchical trees, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 6, No. 3, pp. 243250, June 1996. S. Teboul, L. Blanc-Feraud, G. Aubert and M. Barlaud, Variational approach for edge-preserving regularization using coupled PDE’s,

0-7803-7278-6/02/$10.00 ©2002 IEEE

[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]

IEEE Transactions on Image Processing, Vol. 7, No. 3, pp. 387-397, Mar. 1998. A. K. Katsaggelos and R. M. Mersereau, A regularized iterative image restoration algorithm, IEEE Transactions on Signal Processing, Vol. 39, No. 4, pp. 914-929, Apr. 1991. S. S. Haykin, Neural Networks: A Comprehensive Foundation, New York: Macmillan, 1994. B. Widrow and M. A. Lehr, 30 years of adaptive neural networks: Perceptron, Madaline, and Backpropagation, Proceedings of the IEEE, Vol.78, No.9, pp.1415-1442, 1990. M. Maillon, Neural approach for television image compression using a Hopfield type network, in Advances in Neural Information Processing Systems, New York: Morgan-Kaufmann, pp. 264-271, 1989. S. A. Rizvi, L. C. Wang and N. M. Nasrabadi, Nonlinear vector prediction using feed-forward neural networks, IEEE Transactions on Image Processing, Vol. 6, No.10, pp.1431-1436, 1997. A. Namphol, S. H. Chin and M. Arozullah, Image compression with a hierarchical neural network, IEEE Transactions on Aerospace and Electronics Systems, Vol.32, No.1, pp.326-338, 1996. C. Cramer, E. Gelenbe and H. Bakircioglu, Low bit-rate video compression with neural networks and temporal subsampling, Proceedings of the IEEE, Vol.84, No.10, pp.1529-1543, 1996. Y. Zhou, R. Chellappa, A. vaid and B. Jenkins, Image restoration using neural network, IEEE Transactions on Acoust., Speech, Sig. Proc., Vol.36, No.7, pp.1141-1151, 1988. J. Paik and A. Katsaggelos, Image restoration using a modified Hopfield network, IEEE Transactions on Image Processing, Vol.1, No.1, pp.49-63, 1992. S. Perry and L. Guan, Neural network restoration of images suffering space-variant distortion, Electronics Letters, Vol.31, No.16, pp.13581359, 1995. P. Bao and D. H. Wang, An edge-preserving image reconstruction using neural network, International Journal of Mathematical Imaging and Vision, Vol. 14 No. 2, pp.117-130, 2001. N. J. De Lillo, Advanced calculus with applications, Macmillan Publishing Co., Inc., USA, 1982. K. I. Funahashi, On the approximate realization of continuous mappings by neural Networks, Neural Networks, Vol.1, No.2, pp.183192, 1989. E. B. Baum and D. Haussler, What size net gives valid generalization?, Neural Computation, Vol.1, No.1, pp.150-160, 1989. K. J. Cios and I. Shin, Image recognition neural networks: IRNN, Neurocomputing, Vol.7, No.2, pp. 159-185, 1995. K. Culik II and J. Kari, Efficient Inference Algorithm for Weighted Finite Automata, in Fractal Image Encoding and Compression, ed. Y. Fisher, Springer-Verlag, New York, 1995.