A Contribution to Nonlinear System Theory - Semantic Scholar

Proc. of the IEEE Workshop on "Nonlinear Signal and Image Processing", Halkidiki, Greece, June 1995

A Contribution to Nonlinear System Theory Hans Burkhardt, Hanns Schulz-Mirbach Technische Universitat Hamburg-Harburg Institut fur Technische Informatik I 21071 Hamburg, Germany ABSTRACT | This paper suggests a characterization of systems through equivalence classes of the input signals. Two input signals are called equivalent if they are mapped by the system to the same output. The corresponding equivalence classes for linear systems can be described by subspaces of the signal space. For the nonlinear case the description in terms of subspaces is no longer valid. We show how to construct nonlinear systems whose equivalence classes are generated by the action of a group on the signal space. This construction has interesting connections with the general Volterra theory of nonlinear systems. We discuss for a speci c group how to perform the computations eciently by using a nonlinear form of the FFT-graph. This gives general hints concerning the recursive calculation of the global system output from local computations.

1 Introduction

The paper will make a contribution to nonlinear system theory for signal and image processing. In linear system theory we can discriminate between a more local ltering analysis and the possibility of a more global spectral analysis with the Fourier transform. The behaviour of discrete local ltering can be described in a nite-dimensional vector space as multiplication with a Toeplitz matrix with a main-diagonal dominant band structure. Global linear transforms have in general a full featured matrix where each element of the input data in uences all the elements of the output data (like the Fourier transform). If this matrix can additionally be factorized into a logarithmic number of sparse matrices we can further formulate a fast algorithm (like the FFT). Nonlinear extensions came rst in such a way that local operators were extended to nonlinear operations like median and rank-order lters, morphological and general set operations, and polynomial or Volterra-type kernels. The topic of our paper will focus on the aspect of designing local and global, polynomial based

lters and transforms to extract invariants for certain group actions (like translations, translations and rotations etc.). Local nonlinear lters of Volterra type with appropriate invariance properties may be set up by averaging over the Euclidean Group (translation and rotation) [6, 3]. However, there are also global transforms, which are able to extract invariant features from images using a nonlinear form of the FFT-graph [1]. Each input pixel may in uence each output component of the feature domain over nonlinear functions. Using the C T -transform with the dyadic operations (+; ), we get global sums over monomials arranged hierarchically in increasing order which are invariant to the group of cyclic translations of the input data. We suggest in this paper a system characterization in terms of equivalence classes of the input signals. Two input signals are called equivalent if they are mapped by the system to the same output. A simple example for a one-dimensional signal space is depicted in Figure 1. The input signals are shown on the x-axis and the system output S(x) on the y-axis. In this case there are no nontrivial subspaces and the equivalence classes for linear and ane systems consist of single points. For nonlinear systems, however, the equivalence classes may consist of several discrete points and of entire intervals as shown in the lower part of Figure 1. In order to describe the systems and equivalence classes in the nonlinear case two seemingly dierent approaches come into mind: synthesis and analysis. The rst approach starts with a xed structure of the equivalence classes in the signal space and constructs systems which realize these equivalence classes. Using this point of view we will show how to construct nonlinear systems whose equivalence classes are generated by the action of a group on the signal space. On the other hand one can x the admiss-

S(x)

S(x)

Class 1

Class 2

x Class 1

x Class 2

Affine System Trivial Equivalence Classes

Linear System Trivial Equivalence Classes S(x)

Class 2

S(x)

Class 1

x

x Class 1

Class 2

Nonlinear System Discrete Equivalence Classes

Nonlinear System Discrete and Continuous Equivalence Classes

Figure 1: Equivalence classes for a linear, an ane and two nonlinear systems for onedimensional input signals. able class of mappings performed by the system (e.g. polynomials). Then the task is to examine the equivalence classes induced by these systems. Here it is possible to utilize the Volterra theory of nonlinear systems [4]. We will see that there are several links between these two approaches.

2 Systems and equivalence classes

A system S is a device which performs a mapping from the input space S to the output space R. We usually denote the system input by x and the system output by y and write y = S(x) for the system I/O relation. In order to facilitate the presentation we assume that both the signal space S and the result space R are nite dimensional vector spaces. We assume that the vector space dimension of S is dim S = N and that of R is dim R = M . Vectors x; y are written as column vectors. In this setting a system is a vector valued function and can be described by M scalar functions, i.e.

S(x) = (s (x); . . . s (x)) (1) where s : S ! C ; 1  i  M are complex valued 1

i

functions.

T

M

Given a system S we can introduce an equivalence relation in the signal space S . Two signals x1; x2 are called equivalent, x1  x2, if they are mapped by the system S to the same output, x1  x2 if S(x1) = S(x2): (2) In this framework we can distinguish between the problem of system analysis and those of system synthesis. System analysis means that we are given a xed system S and that we have to describe the equivalence classes in the signal space S generated by this system. This will be done in section 3 for linear systems. System synthesis means that we are given the equivalenve classes in the signal space S and that we have to design a system S whose equivalence classes coincide with the given ones. In this case we say that the system S realizes the given equivalence classes. We will describe in section 4 methods for system synthesis provided that the equivalence classes can be described by the action of a transformation group on the signal space S .

3 Equivalence classes of linear systems

A system S is called linear if for any x1; x2 2 S and ;  2 C

S(x1 + x2) = S(x1) + S(x2): (3) We denote by 0 the zero vector in R. Then the nullspace (sometimes also called kernel) N (S ) of the linear system S is de ned as N (S ) := fx 2 S j S(x) = 0g: Using the linearity of S it is easy to prove that N (S ) is a subspace of S . Therefore S can be written as a direct sum S = N (S )  N (S ) where N (S ) is the complement of N (S ). For a given linear system S we pick two equivalent signals x ; x . From the equivalence (eq. (2)) and the linearity (eq. (3)) of S we conclude S(x1 ? x2) = 0, i.e. x ? x is an element of the nullspace of S. Vice versa it follows from the linearity of S that two signals x and x = x + n with n 2 N (S ) are equivalent, i.e. x  x . This 1

2

1

2

1

2

1

1

2

proves the following Lemma. Lemma 1 Let a linear system S be given. Two signals x1; x2 are equivalent if and only if the difference x1 ? x2 is an element of the nullspace N (S ) of S. Although Lemma 1 follows from elementary linear algebra it has interesting implications for the synthesis of systems. We denote by k : k the

usual Euclidean norm in S . An equivalence class in the signal space S is called bounded if a  2 IR exists such that for every signal x of the given equivalence class k x k . It is impossible to realize nontrivial bounded equivalence classes by linear systems. This can be seen as follows. A nontrivial equivalence class contains at least two elements. This implies for a linear system S according to Lemma 1 that the nullspace is nontrivial, i.e. there is at least one signal n 2 N (S ) with k n k6= 0. However, since S is linear n 2 N (S ) implies n 2 N (S ) forall  2 C . Therefore the signals x and x + n are equivalent for all  2 C . Since k x + n k! 1 for  ! 1, nontrivial equivalence classes of linear systems are always unbounded. To illustrate this result we discuss a simple example where the equivalence classes in the signal space are induced by the action of the group G of cyclic translations. For a signal x = (x[0]; x[1]; . . . ; x[N ? 1]) 2 S the cyclic translation by k 2 ZZ places is given by y = g x with T

T

k

y[i] = x[(i ? k)mod N ] 80  i < N: (4) It is easy to see that the equivalence class of the signal x is bounded (use e.g.  =k x k). Therefore it is impossible to nd a linear system which realizes the equivalence classes described by eq. (4). Using the methods described in section 4 it is possible to construct nonlinear systems realizing these equivalence classes.

4 Nonlinear systems for equivalence classes generated by a group action Let G be a group acting on the signal space S by operators g . For a signal x the transformed signal is denoted by g x. The group action induces an equivalence relation in S . Two signals x1; x2 are called equivalent if a g 2 G exists with x1 = g x2. We want to construct a system S which realizes these equivalence classes (eq. (2)). This can be reformulated as

x = gx , S(x1) = S(x2): 1

2

(5)

One can also say that a system which ful lls (5) is invariant with respect to transformations of the input data. To construct such systems we proceed in two steps. First we explain how to construct for a given function f : S ! C a function A[f ] : S ! C which is invariant, i.e. A[f ](g x) = A[f ](x). Afterwards we show how to put several of these invariant functions together such that the corresponding system (eq. (1)) yields a realization of the equivalence classes.

The basic idea for constructing invariants is to integrate a given function f : S ! C over the transformation group G. This process is called group averaging and is explained in detail in [6]. If we assume that G is a nite group of order j G j then the group average is given by X A[f ](x) = 1 f (gx): (6)

jGj

g

2G

This has been applied [6] to the extraction of translation and rotation invariant features from gray scale images. In order to get a system S which realizes the equivalence classes we have to nd M functions f : S ! C ; 1  i  M such that the system S(x) = (A[f1 ](x); . . . ; A[f ](x)) ful lls eq. (5). If G is a nite group of order j G j it is shown in [5] that a system realizing the equivalence classes induced by G can be obtained by averaging (eq. (6)) all monomials f (x) = x[0] 0


x[N ? 1] N?1 with b0+b1 +. . .+b ?1 j G j over the group. The number M of? components of the resulting system
j j + . This may be prohibitively is at most large and it must be emphasized that this is only an upper bound which can be improved considerably in some cases. If f (x) is a polynomial then the group average A[f ](x) is also a polynomial (eq. (6)) and can therefore be written as X A[f ](x) = H [x]: (7) i

T

M

b

b

N

G

N

N

i

i

The operator H gives the part of the output corresponding to the polynomial degree i, i.e. i

H [x] =

Z

i

i

h (1; . . . ;  )x[1]


x[ ]d1


d i

i

i

i

(8) It is understood that (8) is a discrete sum since x is a vector. However, the notation with integrals is intuitively appealing and suggests to interpret eq. (7) as the Volterra series of A[f ] [4]. From the invariance of A[f ] with respect to the action of the transformation group G it follows that every H must be invariant too, i.e. H [gx] = H [x]. Transformations of x induce transformations of the kernel function h (1; . . . ;  ) in eq. (8) and H is invariant if the kernel is invarint with respect to these induced transformations. The important point to note is that invariant kernels can be determined from the group structure alone (i.e. this can be done o-line since no information about the actual system input is necessary). Appropriate kernels for abelian groups are e.g. given in [3]. IR

i

i

i

i

i

i

5 Local and global aspects of nonlinear systems Global nonlinear transforms utilize ambiguous functions to map all signals of an equivalence class into one feature vector and not only one spectral component as in the linear case where a spectral discrimination may be done only by selecting a certain subspace. In contrast to linear theory, where the isomorphism between convolution in the original space and the multiplication in the Fourier domain gives a strong link between both representations, there are in the nonlinear case still a number of open questions connecting both the local as well as the global domains. However, the concept of a recursive de nition provides a basic principle for the extension of local operations to higher dimensional spaces. In this section we present a global system based on a nonlinear form of the FFT-graph for calculating invariants for the group of cyclic translations (eq. 4). The class C T [1] based on two arbitrary commutative functions f1 and f2 is able to extract translation invariant features from the input data. As we can see from Figure 2 each element of the input data in uences by composition

x=x x0 x1 x2 x3 x4 x5 x6 x7

r r r r r r r r

(0)

A
 A
A A

 A A
 A A

A
A

A
AA
A
 AU
A
A
A

A
A AU

A
A

A AU

A
A AU
-

xH i

x

HH j *   j

x

n n n n n n n n fn

x

x = xe

n n n n n n n n

1

2

3

n x~ n x~ n x~ n x~ n x~ n x~ n x~ n x~

- f1 f1 @ ? * f1 H H ?- f  HH - f2j f1 @@ 1 ?@?  ? -f R @ f1 ?@ * f1?@ 2 HH   j R - f2  HH - f2f1 ? @ - f1 H f2 @ ? * f1 H   ?- f  HH j - f2f2 @@ 1 ?@?  ?@ - f1R f2 ?@ * ?@ f2 HH   HH R @ j ?  - f2f2 f2

k

-

0

1

2

3

4

5

6

7

f (x ; x ) commutative k

i

j

Figure 2: Signal ow graph for the class C T of global nonlinear transforms for dimension N = 8. of the functions f1 and f2 all elements of the output data. Hence the class C T has a similar "integral" or global character like FFT and FWT with similar properties like robustness against noise. Using the notation x1j2, x2j2 to divide the vector x in 2 subvectors we are able to give the fol-

lowing recursive de nition for ?x = C T (x) ?

x =

2 , 3

f (x ; x ) f2(x1j2; x2j2)

1 1j2 2j2 5 4 ,

2 3

=

x xj

(1)

6 1j2 7 4 5

(9)

(1) 22

f1 ; f2 commutative, beginning with the scalar variables x? = x . Here f1 2(x; y) denotes the componentwise application of the functions f1 2 to corresponding elements of x und y . This de nition corresponds topologically to a special form of the FFT called "decimation in frequency". Using e.g. the two operations f1 = + and f2 =  we get polynomial type translation invariant features. However, it has to be stressed, that these global features are not only invariant with respect to translations but also to other permutations [2]. Another global counterpart of nonlinear local set-type operations like rank-order lters (e.g. median lters) is the global M -transform using the dyadic operations (max; min). This transform is also known as bitonic sorter and is as well part of a complete parallel sorting network as global operation. It has to be stressed that the recursive de nition gives us the possibility to extend local to global operations (e.g. commutation of two elements may be extended to sorting of N elements). i

i

;

;

References

[1] H. Burkhardt. Homogeneous Structures for Position-Invariant Feature Extraction. In J. C. Simon, ed., From the Pixels to the Features, North-Holland, 1989. [2] H. Burkhardt, X. Muller. On Invariant Sets of a Certain Class of Fast Translation-Invariant Transforms. IEEE ASSP-28(5):517{ 523, October 1980. [3] G. Gheen Distortion invariant Volterra lters. Pattern Recognition, vol. 27, no. 4, pp. 569-576, 1994. [4] M. Schetzen The Volterra and Wiener Theories of Nonlinear Systems. Wiley 1980. [5] H. Schulz-Mirbach On the Existence of Complete Invariant Feature Spaces in Pattern Recognition. Proc. of the 11'th ICPR, vol.II, pp.178-182, 1992. [6] H. Schulz-Mirbach Constructing invariant features by averaging techniques. Proc. of the 12'th ICPR, vol.II, pp.387-390, 1994.