Appendix: Processing Techniques for Linear Systems

Appendix: Processing Techniques for Linear Systems

A.

In tro d u c tio n

Linear processing techniques are the basis of many image enhancement and restoration algorithms. The purpose of this appendix is to introduce several of the important concepts and variations of linear system computation. For simplicity, only the one-dim ensional case without noise will be considered. The two-dim ensional extensions and noise considerations are covered in the text. The fundamental and familiar continuous linear system is first introduced. Both time variant and invariant systems are then considered, and it is shown that three forms, infinite im pulse response, fin ite im pulse response, and p erio d ic superposition are practical m odels for physical systems that may be closely approximated by discrete computations. The matrix representation for the time variant system is a superposition relation, and the forms of this superposition matrix for each of the three models is considered. The matrix form for the more widely used time invariant system is then developed. The important concept of a circulant matrix arises in the periodic convolution. It is next shown that the circulant matrix is diagonalized by a discrete Fourier transform and, consequently, an efficient solution to the periodic convolution equations is developed. N ext, it is shown that both finite and infinite impulse response convolutions may be computed via a circulant convolution, thus extending the efficient solution to all three cases. A linear system h is defined by the relation h [ a / i ( t ) + bf2{ t ) ] = ah [ / , ( t ) ] + bh [ f 2{ t ) ],

where f x( t ) and f 2( t ) are arbitrary inputs, and a and b are constants. Furthermore, since an arbitrary input function,/ ( / ) , may be expressed as a weighted sum of impulse functions, f(0= f

f(r)8(t-r)dr,

J —OO

555

Appendix

556

the response of the linear system to this arbitrary input may be computed by

/

OO

f(T)S(t-r)dr .

L ^ — oo - no

By the linearity property

8(0= [

f(r)h [8 (t-T )]d T .

J ——ran 00

Thus the response is characterized by the linear system response to an impulse function, which leads to the definition of the impulse response h(t , t ) as

A(/, t ) = A [ 5 ( / - t )]. Since the system response to an impulse may vary with the time of application of the impulse, the general computational form of the linear system is the superposition integral: 8 ( 0 =

( bf ( T ) h ( t , T ) d T .

Ja

This integral equation is called the Fredholm integral, and several books have been devoted to its solution [e.g., Rust and Burris (1972), see Chapter 4], The limits of integration are very important in determining the form of the computation. W ithout further assumptions about the system or input signal, the computation must extend over an infinite interval, —oo

An important condition of realizability of a continuous system is that the response be nonanticipatory, or causal, h(t,r) = 0

for

t —r < 0 .

The causality condition leads to the computation — OO

N ote that both equations require an infinite duration for the computation, and therefore cannot be directly computed numerically. Rather, three other forms that involve a computation only over a finite interval are more appropriate. These may be called infinite impulse response, finite impulse response, and periodic superposition to conform with the standard terminology in digital signal processing. In the following discussion, we shall consider these three forms for both continuous and discrete processing to emphasize the fundamental assumptions and permit the selection of the appropriate model for a physical system. The first computational form that can be practically implemented on a computer is the infinite impulse response, a causal system for which the input

Processing Techniques fo r Linear Systems

557

signal starts at t = 0. With the con d ition / ( ; ) = 0, / < 0 , the system is described by S(t)=

f ‘f ( T ) h ( t , T ) d r .

Jo

Since the output can be observed at any positive time, the system impulse response could be of infinite duration. A n infinite impulse response system can be implemented with a finite recursive filter computation. The previous com putation may be approximated by a numerical quadrature method for discrete implementation. This requires selection of evaluation or sampling times for both the input, T = i A t , and the output, t = j A t . N ote that an interesting class of problems could be generated by sampling either the input or the output but not both. These variations are not in the mainstream of image processing and will not be considered. A lso, to avoid an interpolation step in the evaluation of h( j At —i At ) , and rounding in the upper integral limits, the sampling increments will be assumed equal, i.e., A? = Ar. An interpolation, if required, can easily be accomplished. The discrete superposition for the infinite impulse response, causal system with input starting at t = 0 may be written as g ( k A t ) = 2 f ( j At ) h( k At , j A t ) A t 7 - 0

for k = 0 ,1 , 2 , . . . . By proper scaling so that A? = A t = 1 , the superposition simplifies to sO ) = 2 f(j)h{kj). 7 = 0

If the continuous system is time invariant, then h(t,T) = h ( t - r ) ,

and consequently, h( k, j ) = h( k —j ) . Thus the superposition reduces to a convolution of the form sO )= 2 f(j)h (k-j). 7 = 0

The above equations therefore describe the causal linear system with the input starting at / = 0 for the time variant and invariant systems. These computations may be used to model causal systems in which past, present, and future relationships are meaningful. In optical systems, causality in the time variable is applicable; however, spatial causality is not as appropriate. A point is usually blurred into its surrounding points in an unordered manner. Therefore another form of the linear system computation must be considered.

Appendix

558

The second important computational form is the finite impulse response system described by g ( t ) = f ‘+

f(r)h(t,T)dT.

J t-T /2

This form follows directly if the impulse response is assumed to be of finite duration, i.e., if \ t - r \ > \ T . h( t , t ) = 0 N ote that for this general noncausal system the present output depends on past, present, and future inputs. The assumption of a finite impulse response is usually reasonable and is especially appropriate for modeling blur in optical systems. Again using numerical quadrature, the finite impulse response computation may be converted to discrete form. The input and output sample increments will again be assumed equal, At = Ar, and the interval T divided into an even number M of subintervals with At = T / M . An odd number of points could also be used. The discrete com putation then becomes k + M /2

g(kAt) =

k = 0, ± 1 , ± 2 , . . . ,

f (J A t ) h ( k At J A t ) A t ,

2 j = k - M / 2

which simplifies to k + M/2

g(k) =

2

/C O M M -

j = k - M / 2

If the system is time invariant, the superposition reduces to the convolution k+ M /2

* (* )=

2

fU)h(k-j).

j = k - M / 2

Changing the subscripts of the above equation gives a more familiar form of the finite impulse response convolution as a moving window computation,

The final com putation to be considered is the periodic or circular superposition. The importance of this form stems, not from its appropriateness for m odeling a physical system, but rather from its discrete computational efficiency. Given finite durations for both the impulse response and the input to the linear system, i.e., f(t)

nonzero for

0 < t < T x,

and

h(t,T)

nonzero for

0 < t - r < T 2,

then periodic extensions with a com mon period, 7’ > m in (7 ’1, T2) may be defined as f p( t + T ) = f ( t ) ,

hp( t , T + T ) = hp( t + T, r ) = h(t, r).

Processing Techniques fo r Linear Systems

559

Then a com putation can be made of the form g p( t ) = f f p( r ) h ( t , r ) d r . *'0

The output g ( t ) will also be periodic with period T. To avoid wraparound or overlap of the functions requires only that T > T 1+ T 2. In this case an aperiodic computation may be performed with the periodic superposition. The discrete computation may be developed assuming A? = Ar and M A t = T, M —1 g P( k A t ) =

f P( j A t ) hp( k A t , j A t ) A t

2 j =o

or simply M- 1

g p( k ) =

f p U ) h p(k, j ),

2 7 = 0

where k = 0, ± 1, ± 2 and the conditions of periodicity are hp( k ± nM, j ) = hp( k , j ± n M) = hp(k,j), f p ( j ± n M ) = f p( j ) , g P( k ± n M ) = g p(k), for « = 0 , 1 ,2 ,.... For the time invariant system M- 1

g P( k ) =

2

j- 0

f p ( j ) hp( k - j ) -

At this point, the basic equations for the three computational forms have been developed. W e shall now consider matrix equations for each. B.

M a tr ix R e p re s e n ta tio n s

If only a finite number of output values are to be considered, then matrix forms of the discrete superposition or convolution computation may be easily developed. The matrix forms show definite structures for the linear systems. Consider the infinite impulse response convolution given earlier. The sequence of output values may be directly written from the equation as * (0 )-/(0 )A (0 ), g (l)-/(0 )A (l)+ /(l)A (0 ),

g(2) = /(0 )A (2 ) + /(1 )A (1 ) + /( 2 )/i( 0 ) ,

g ( M ) = f ( 0 ) h ( M ) + f ( l ) h ( M - 1) + • • • + f ( M ) h ( 0). If only M nonzero input values are given and N > M , then g(N)=f(0)h(N)+f(l)h(N- !)+••• + f(N )h (N -M ).

560

Appendix

This set of equations may be written in matrix form, g = H f, where

m

g(0)

M 0)

g (l) g( 2)

M l) M 2)

g{M)

h(M)

g(N) _

h(N)

M 0) M l)

/(I)

m

M 0)

f{M)

M 0)

_/(N )

L M N N ote that the matrix has nonzero terms only on the lower triangular portion. Also, the effect of all input terms being 0 after M terms simply places zeros in the input vector. If the impulse response h(j ) is zero for all N > j > L , then zero terms are added to the lower left portion of the matrix, giving the matrix a banded structure. The most important property of the H convolution matrix is that the diagonal terms are equal. Square matrices with this property are called Toeplitz. Thus, the convolution matrix is Toeplitzlike. N ow consider the superposition relationship given previously. In a similar manner, this equation may be written in the matrix form of g = H f. However, the H matrix no longer has the Toeplitzlike structure. H is given by

*(0,0)

A(1,0) MU) H=

h( 2,0)

h( 2,1)

A(2,0)

h( M, 0)

h(M,M)

This lack of Toeplitzlike structure indicates why superposition is more difficult than convolution. The number of degrees of freedom of the H matrix is on the order of M 2 rather than M. Suppose that the impulse response has L, the input f has M , and the output g has N nonzero terms. For a convolution computation, the number of nonzero output terms of g is N = M + L. The form of the H matrix in this case is of size N x L and thus has more rows than columns, representing a set of overdetermined equations, is banded, and nonzero diagonal terms are equal: m H=

h( i) h( L)

MO) ■■•

/i(0)

h( L)

■■•

M0)

Processing Techniques fo r Linear Systems

561

For a correlation com putation such as in template matching, the impulse response size L and input size M are given, and output responses g are computed only for the N = M —L + \ locations for which the template fits completely in the input sequence. In this case the H m afnx has more columns than rows, which represents underdetermined equations. The cases for which the H matrix is square, L = M, and of full rank are rare. N ow consider the matrix form of the finite impulse response computation. The output terms as com puted by the convolution for positive values of k are g(0) = / ( - \ M ) h { \ M ) + / ( 1

1) + • • • + / ( 0 ) / r ( 0 )

g(\)=f{\-\M )h(\M )+f(2-\M )h(\M -\)+-

• • + /(0 )A (1 )

g{N)=f(N-\M )h(\M )

1) + • • • + f ( N + \ M ) h ( - \ M ) .

+f(N + 1

Again, if only a finite number of output terms are computed, the equation may be written in the matrix form g = H f,

where * ( 0)

h(\M)

h ( \ M - 1)

h{ 0)

h(\M)

?(1)

h(\M)

g(N)

h(-\M )

/(-IM )

m /(i)

f(N + \M) N ote that the matrix is banded and Toeplitzlike since diagonal terms are

Appendix

562

equal. Also, for the convolution com putation the number of nonzero input terms assumed is N + M so that the set of equations is underdetermined. Another important point is that each row of the matrix except the first is a right shift of the row above it. This property will be very important with circulant matrices. The matrix form for the superposition given earlier is also expressable as M

g ( k ) = ? Jf ( j + k - \ M ) h ( k J + k - \ M )

or

g = H f,

7 = 0

where

H=

/i( l,|M + l) h(N,N+\M )

h(N,N-\M )

N ote that the diagonal terms are not equal for the superposition and that the equations are undetermined. The periodic convolution leads to the most interesting circulant matrix form. The output terms for the periodic convolution are S p ( 0 ) = / p ( 0 ) V 0 ) + / p ( l ) V - ! ) + • • • + f P( M - l ) h p( ~ M + 1).

By the use of the periodic property that hp( —k) = h p( M —k), S P( 0 ) = / p(0)A p( 0 ) + / p( l ) / . p( A / - 1) + • • • + / P( M - l) A p( l )

and g p( l ) = / p ( 0 ) A p( l ) + / p( l ) A p( 0 ) + • • • + / P( M - l)A p(2),

gp( M - l ) = / p(0)/!p( M - l ) + f p( l ) h p( M - 2 ) + - • • + f p( M — l)A p(0). In matrix form § p — Hpfp>

where gp(°) Sp(l)

=

Ap(0)

hp( M - l )

hp( M - 2 )

hp( 1)

/ P(0)

Ap( l )

*p(0)

hp( M - \ )

hp{ 2)

/p (l)

hp( M - l )

hp( M —2)

hp(0)

/ P( M - 1 )

The matrix H now has the property that each row is a circular right shift of the preceding row and H is square. Thus, H is a circulant matrix. An important property of a circulant matrix is that it is diagonalized by a discrete

Processing Techniques fo r Linear Systems

563

Fourier transform operation. This fact was applied by Hunt (1972) to convolution computations with great success. Follow ing Hunt’s development, let us determine the eigenvalues and eigenvectors of a general circulant matrix C, where

■■•

C (0 )

C( 2) C( 1)

c(2)

C (3 )

••

C (0)

C ( 1)

C ( N —1)

C ( 1)

C(N--1 )

■

C(N--2 )

■

C (0 )

Let W=exp(i2-rr/N), where i = V — 1 . Then W k is one o f the N distinct roots of unity, i.e., W kN= 1, A: = 0 , 1 , 2 , 1 . N ow consider

X(k) =

C (0 ) + C ( l)

Wk+

C (2 )

W 2k + ■■■ + C ( N —1) W (N ~ I)A:.

The value \ { k) also satisfies the following equations:

\ ( k ) W k = C { N - \ ) + C{Qi)Wk + C ( \ ) W 2k+ ■■■ + C ( N —2) w^N~ l^k, \ ( k ) W 2k = C ( N —2) + C ( N —1) W k + C(0) W 2K +

••• +

C ( N - 3 ) W (N~ l)k,

A(Jfc) W (N~ l)k = C (l) + C(2) W k + C(3) W 2K+ ■■■ + C( 0) W (N~ ')* . The significance of these equations is clear if they are written in matrix form

\(k)W(k)=CW(k), where W(A:) = [1, W k, W 2k, . . . , W (N~ 1) M + J — 1 is an important condition. N ote that all sequences are the same length. N ext construct a circulant matrix He as

He=

he( 0)

he( P - 1)

* e 0 )

* e 0 )

* e(°)

K{2)

hc( P ~ l )

A«(0)

N ow the computation of produces a vector ge of length P\ however, the first M + J —2 com ponents are exactly equal to the M + J —2 com ponents of g because enough zeros were added to prevent overlap of the nonzero terms added to the He matrix and the fe vector. Thus the aperiodic convolution may be com puted by a periodic convolution. N ow consider the com putation of the finite impulse response convolution given in matrix form. It was previously noted that the rows of this matrix, except for the first, were right shifts of the preceding rows. Thus, the matrix is almost circulant and must only be made square and circulant by adding \ M rows as the first and last rows. W e must therefore again define extended sequences

K (i)=

o,

h ( M + N + 1 —/),

i = ± M + N + l , . . . , N + M,

Appendix

566

and 0,

0,1

=

g e( 0 = -

= \M +l,...,\M +N +l,

l) ,

= \ M + N + 2 , N + M,

0,

then form the circulant matrix He with the sequence he as before. The resulting computation

is a periodic convolution. However, the desired g vector may be obtained exactly by selecting the center N terms of the g e vector. This is sometimes referred to as ignoring the edge effects. Thus, all three important convolution forms m ay be represented by circulant computations and thus computed via a fast Fourier transform algorithm. Let us consider as a final example the difficulties encountered if an impulse response of infinite duration is approximated by a finite discrete Fourier transform computation. Suppose the impulse response is given as (Kwoh, 1977, see Chapter 4)

h (0) = T r / 3 T 2, h ( n ) = — 1/ t7«27’2

for

« = ± 1 , ±2.

where T is the sample spacing. The frequency response can be shown to be H(u>) = |co| — T u 2/2-n for 0 < |co| < 2m / T and periodic with period 2-n/T. A discrete Fourier transform may be used to approximate this filter; however, the accuracy of the approximation must be considered. The D F T of the impulse response for finite N is D(k) = 2

h ( n ) W nk,

0 < k < N — 1,

where J F = ex p (—j l m / N) . Taking account of the Hermitian or folding frequency of the D F T gives M

D(k)= 2

h ( n ) W nk+

2

h ( n ) W nk,

n = N —M

where M can be set to the Nyquist frequency | N . Substituting the given impulse response gives

Processing Techniques fo r Linear Systems

567

Expanding and using the identity ^ 2

k-i

cosKx 7T2 mx x2 = ~Z-------o------- T ------1

K2

6

2

2

„ for

0