Orientation Selectivity: An Approach from Theoretical Computer Science

L

ISSN 0918-2802 Technical Report

Orientation Selectivity: An Approach from Theoretical Computer Science

Osamu Watanabe and Tadashi Yamazaki TR96-0008 November

Department of Computer Science Tokyo Institute of Technology ^ Ookayama 2-12-1 Meguro Tokyo 152, Japan

http://www.cs.titech.ac.jp/

c The author(s) of this report reserves all the rights.

Title:

Orientation Selectivity Problem: An Approach from Theoretical Computer Science Authors:

Osamu Watanabe and Tadashi Yamazaki Dept. of Computer Science, Tokyo Inst. of Tech., Meguro-ku Tokyo 152, Japan. E-mail: fwatanabe,[email protected] 1

Introduction

The brain consists of 10 to the 10 ganglion cells called \neurons", and it is believed that the function of each neuron is quite simple. However, once so many neurons are connected each other, they show several interesting behaviors on whole and on average. Orientation selectivity is a typical example of such behavior [WH63]. That is, in the primary visual cortex (area17) of cat and monkey, there is some group of neurons that reacts strongly to the presentation of light bars of a certain orientation. The reason why such preference is emerged has not been explained clearly yet. It should be noted that the process of developing orientation selectivity is not \learning" because the selectivity is developed from stimuli that are given uniformly at random, which contains nothing to learn. That is, mechanism that we need to nd out is not \learning" but \self-organization ". To explain orientation selectivity, a considerable number of mathematical model is investigated (see [CS92, Swi96] for some more explanation and the references on this topic). Among them, the pioneer research is the von der Malsburg's study [Mal73]. He developed a mathematical model of cortex and retina that receive light bars as inputs, and showed orientation selectivity is emerged under his model by computer simulations for the rst time. There are also other studies by dierent approaches from statistical physics [Tan90] or information theory [Lin86]. Furthermore, some of such models have been investigated mathematically [MM90, Tan90]. These researches conclude that the key factors are the following: (i) correlation between two types of inputs (i.e. ON- and OFF-center inputs) [Mil92, MT92], and (ii) some variation of Hebb's learning rule for modifying synaptic strength [MM94, SP95]. On the other hand, the main purpose of such study is the development of mathematical model of neural competition; thus, in many cases, its justi cation has been done through computer simulations. Even in the case that elaborated mathematical analysis is reported [MM90, Tan90], such analysis is made by assuming \average" inputs or stimuli under a certain distribution, and it has not been investigated the case where concrete stimuli are given under the distribution. Here we focus on this case, and investigate dynamic behavior 1

of neurons. Clearly, it is hard to investigate neurons' dynamic behavior; thus, for simplifying our analysis, we rst propose to separate the mechanism of selectivity into the following two parts: (i) the part that extracts bar stimuli from random noise, and (ii) the part that induces the selectivity when given bar stimuli of random orientation. In this paper, we focus on the second part, and propose a simple probabilistic game called \monopolist game", that is a simpli cation of neural competition. On this model, a learning rule corresponds to game's rule, and the selectivity is interpreted as that a single winner of the game | monopolist | emerges. The paper is organized in the following way. In Section 2, we make a brief overview of von der Malsburg's study. Section 3 is devoted to introduce the monopolist game and propose some updating rules. In Section 4, we see computer simulation for each updating rule and show some analytical results. Here we prove that monopolist emerges (which corresponds to the selectivity) with probability 1 under a game rule corresponding to von der Malsburg's learning rule. On the other hand, we prove that monopolist does not appear with high probability under a game rule with no constraint. Thus, we con rm that constraint is important for developing selectivity. Finally, the proof of results in Section 4 is given in Section 5. 2

Von der Malsburg's Computational Model

In his pioneer paper [Mal73], von der Malsburg gave the rst reasonable explanation to the orientation selectivity. Though his insight into the selectivity is still appreciated, the computational model that he used in his computer simulation is quite simple, and it has been taken over by more complicated and more biologically plausible models. Nevertheless, we use his model for our basic computational model. One reason is that it is much simpler. But we also believe [SW97] that correlation between ON- and OFF-center inputs would provides us with the situation like von der Malsburg's model. That is, the rst of the two key factors for the selectivity gives some justi cation of using von der Malsburg's computation model. Here we rst explain brie y the model considered by von der Malsburg. Neural Network Structure We consider two layer neural network. In particular, since we discuss here the orientation selectivity for one neuron, we assume that there is only one output cell. On the other hand, the input layer consists of 19 input cells that are (supposed to be) arranged in a hexagon like ones in Figure 1. We use i for indicating the ith input cell, and I for the set of all input cells. (Note that von der Malsburg studied the selectivity for a set of neurons; 2

our neural network is a basic component of his model.) Stimuli and Firing Rule We use 9 stimuli with dierent orientations (Figure 2), which are given to the network randomly. Here  indicates an input cell that gets input 1, and  indicates an input cell that gets input 0.

output

inputs

Figure 1: Neural network model. Figure 2: Nine stimuli. We use A to denote input value (either 0 or 1) to the ith input cell. Then output value V is computed in the following way: i

=

V

X

T hp

i

2

!

wi A i :

(1)

I

Where w is the current synaptic strength (which we simply call weight) between the output cell and the ith input cell. T h (x) is a threshold function that gives x 0 p if x > p and 0 otherwise, where p is given as a parameter. i

p

Learning Rule (Updating Rule) Initially, each weight w is set to some random number between 0 to some constant. Then weights are updated each time according to the following variation of Hebb's rule, which we call von der Malsburg's rule: i

wi0 wi

= =

+ cinc A V; and P 0 0 w 2 (W0 = 2 w ):

wi

i

k

i

I

(2)

k

Where cinc (which is called a growth rate) and W0 (total weight bound) are constants given as parameters. The rst formula is the original Hebb's rule; on the other hand, the second one is introduced in order to keep the total weight within W0 . (In fact, it is kept as W0 .) 3

Because of this second rule, some researchers do not consider it as Hebb's rule since it loses some feature of the original Hebb's rule. With this setting, von der Malsburg demonstrated that the selectivity occurs. Criticism to von der Malsburg's Model There have been several criticisms to the computation model of von der Malsburg. A crucial one is to the assumption of bar stimuli [Mil92]. This assumption may not be realistic; in fact, it has been reported that for some mammals, the orientation selectivity emerges before any visual stimulus is given. It is, however, by assuming a certain correlation between ON- and OFF-center inputs, we may be able to derive bar type stimuli (or, the eect of bar type stimuli) even from random noise on receptive elds [MM94, SW97]. 3

Our Mathematical Model: Monopolist Game

In this paper, in order to give a formal proof to von der Malsburg's rule, and to investigate some other computation rules, we simplify von der Malsburg's computation model yet further. We propose the following simple probabilistic game for our preliminary investigation. Monopolist Game

Rule: Initially n players are given the same amount of money. The game goes step by step, and at each step, one of the players win with the same probability. The winner gets some amount of money, while the other loses some. Goal: The game terminates if all but one become bankrupt. If the survived player keeps enough money at that point, then he is called a monopolist. The connection of this game to von der Malsburg's computation model is clear; player's wealth corresponds to total synaptic strength between the output cell and a set of input cells corresponding to one type of stimulus, and the emergence of a monopolist means that the network develops preference to one orientation. Thus, an updating rule of players' wealth corresponds to a rule of changing synaptic strength in the network. Here we consider updating rules described as below with some functions fdec and finc . (In the following, let i0 denote the player that wins at the current step.) 8 < w + finc (w ) 0 fdec (finc (w ); w ); if i = i0 , and w = : w 0 fdec (finc (w ); w ); otherwise. 0 i

i

i

i

i

i

i

(3)

i

Here finc and fdec are functions that determine respectively the amount of increment and decrement at each step. One could think of more general cases such as these function 4

values can depend on the status (i.e., wealth) of all players. But we will not go so further, because such general updating rules would lose biological justi cation. (We may use n0 , the number of currently survived players, for computing finc or fdec .) From the relation to von der Malsburg's computation model, we require the following basic constrains: (i) both finc and fdec are nondecreasing, and (ii) both finc (0) and fdec (; 0) are 0. Due to the second constraint, once a player loses all money, he stays forever in the 0 wealth state. (Note, on the other hand, that the player can still win with probability 1=n.) Here an updating rule is called a (variation of) Hebb's rule if it satis es finc (wi ) > fdec (finc (wi ); wi );

and

fdec (finc (wi0 ); wi ) >

0

(4)

for any w > 0. That is, a winning player gains some amount of money, while the others lose some. All updating rules considered here essentially satisfy this condition. (There are some exceptional cases in some updating rule.) On the other hand, an updating rule is called competitive if finc and fdec are de ned so that the total amount of wealth of all players is bounded. Now let us de ne some updating rules. In this paper, we consider the following three updating rules: (1) von der Malsburg's rule, (2) local rule, and (3) semi local rule. i

von der Malsburg's Rule We begin with a rule corresponding to von der Malsburg's rule. It is de ned as follows. finc (w )

=

cinc ;

and

fdec (; w)

=

=n0 :

(5)

(Here n0 denotes the number of currently survived players. Note that from our basic requirement, we should de ne finc (0) = 0 as an exceptional case. In the following, we omit specifying this exceptional case.) Note that with this rule, the total amount of wealth is kept constant. Thus, it is a competitive game. We can consider a similar rule such that finc (w) is not constant but proportional to w . (Similarly, fdec (w) is also proportional to w.) This rule might be closer to the original von der Malsburg's rule. But for discussing the probability of having a monopolist, the dierence of two rules is not important; the dierence disappears if we take log. Thus, we will discuss with our simpler rule. Local Rule In von der Malsburg's rule, updating function needs to know winner's pro t; that is, updating values cannot be determined locally. In other words, in order to be competitive, an updating rule must be local. Thus, to see the importance of competition, we consider 5

here the following purely local updating rule. finc (w )

=

cinc

and

fdec (; w )

=

(6)

cdec :

For this local rule (and the next semi local rule), the notion of \monopolist" is less clear than von der Malsburg's rule, because the notion of \enough amount of money" is not clear. In the following analysis, we will use W1 to denote this amount. (Cf. We often use W0 to denote the bound of total wealth, which is also the wealth of a monopolist under von der Malsburg's rule; thus, W0 = W1 in the context of von der Malsburg's rule.) When we need some speci c value for W1 , we will use as our working assumption, a half amount of the total initial wealth. That is, we regard a single surviver as a monopolist if he posses more than a half amount of the total initial wealth. Semi Local Rule As a third updating rule, we consider somewhat mixture of the above two rules. It keeps a certain amount of locality, but it is still competitive. This rule is de ned as follows. finc (w )

= min(cinc ; W0 0

X

w j );

and

fdec (; w )

=

cdec :

(7)

j

That is, we want to keep the total wealth smaller than W0 . Thus, a winner can gain cinc (in net, cinc 0 cdec ) if there is some room to grow. In this case, each player (more precisely, a winner) only needs to know the total amount of wealth, or the amount of space left. 4

Computer Simulation and Analytical Results

Here we rst show analysis of our three updating rules by computer simulation, and then investigate analytically the probability P3 that a monopolist emerges. For von der Malsburg's rule, we prove that that P3 = 1. On the other hand, for the local rule, our computer simulation shows that P3 is fairly small, and we prove some fact supporting this. For the semi local rule, we show that P3 is large through computer simulation. Moreover, we prove that for some appropriate parameters, P3 becomes large. (Theorems presented in this section will be proved in the next section.) In the following, we may assume without loss of generality that cdec = 1. von der Malsburg's Rule It is easy to express a game for n = 2 under von der Malsburg's rule by an one-dimensional random walk; thus, by applying standard analysis of random walks, we can prove that a monopolist emerges with high probability. Furthermore, the case where n > 2 is also provable by induction. Hence we have the following theorem. Under von der Malsburg's rule, a monopolist emerges (in nite steps) with probability 1.

Theorem 4.1.

6

Local Rule Let us rst see our result of computer simulation. Table 1 shows the results of 1,000 games for n = 10 and each cinc . We use 10 for initial wealth; that is, the initial amount of wealth for each player is 10. Here in order to avoid the game continues forever, we set an upper bound limit L = 10; 000, and stop the game as soon as a player's wealth reaches to L. The column N shows the number of games that eventually had one surviver. Based on the amount of wealth that surviver poses at the end, these games are classi ed into four groups, and the next four columns (N1 ; N2 ; N3 ; N4 ) indicates the number of games in each group. Since initial wealth is 10 (and thus, total wealth is initially 100), we would not call a surviver as a monopolist unless he has more than 50 (= 100=2) at the end; that is, N3 + N4 is the number of games that a monopolist emerges. First of all, we notice that the probability P3 that a monopolist emerges is quite low. It is also noticeable that N drops rapidly after some threshold. Small N means that many games stop because someone hits the limit L; on the other hand, if everyone goes into bankruptcy, then with very high chance, someone becomes the last one (with small amount of money). Notice that this threshold is a bit above the number of players n. This is reasonable. Because each player can win with probability 1=n, cinc must be larger than n (more precisely n + 1 because net increase is cinc 0 cdec ). On the other hand, if cinc is larger than n + 1, then it is more likely that everyone grows, and someone hits the limit soon or later. Table 1: Local rule with an upper limit. # of players : 10

cinc

N

7 8 9 10 11 12 13

925 959 973 990 470 130 36

1  30 31  60 61  90 90  815 110 0 0 573 369 17 0 435 387 127 24 232 276 197 285 46 136 107 181 3 38 20 69 2 2 8 24 N1

N2

N3

N4

Of course, it may still be the case that after someone hitting this very high limit, one player eventually becomes a monopolist. But we can see from the following results (Table 2) that such cases are rare. Table 2: Local rule with a time bound. 7

# of players : 10 cinc

N

5 86 6 92 7 89 8 95 9 97 10 100 11 44 12 7 13 0

N1

N2

N3

N4

86 92 80 53 43 20 2 1

0 0 9 39 35 24 8 0

0 0 0 3 15 18 8 1

0 0 0 0 4 38 26 4

This table summarizes the result of 100 games starting from initial value 10. Here we do not put any upper limit. Instead, we limit the time; that is, the game is terminated in 10,000 steps.

0 0 0 0

Though it seems hard to estimate P3 , the probability that a monopolist emerges, precisely, we can still get some rough bound analytically. First consider one player, say player 1, and investigate his average wealth. Here we have the following result. Consider the amount of wealth that player 1 has. For any t, it increases, 0 1 on average, after t steps.

Theorem 4.2.

by t

 cinc n



This supports our intuition; that is, if cinc is smaller than n, then every player dies quickly (and thus, no monopolist occurs). On the other hand, if cinc is larger than n, then the wealth of every player grows up (and thus, no monopolist occurs). Thus, the most critical case is the case cinc = n. Next, we discuss P3 for such a case. By our de nition of monopolists, P3 be the probability that, at some point in the game, all but one become bankrupt and that the survived player has wealth W1 . Here instead of this P3 , we consider the probability P30 that at least one player reaches to W1 and no more two players reach to W10 for suciently large W10 . If a monopolist emerges at some point, then clearly, someone needs to reach W1 in the game. Furthermore, it is unlikely that two players reach to some large value W10 . Thus, we may be able to regard P30 as an upper bound of P3 . For this P30 , we have the following bound. Theorem 4.3.

P30
t. Proof of Theorem 4.2.

1 n

1- 1 n

0

W0

x Cinc - 1

1

Figure 5: Random walk representing one player's wealth. Let S be the increment (or the decrement if it is negative) of player 1's wealth after t steps. That is, t

=

St

! X t t

i=0

i

1 

(icinc 0 t)

10

i

n

1 0 t

n

i

1

=

nt

X

!

t

cinc

i

i=0

Evaluate the coecient of n 010 (i 6= 0). Then we have t

t

!

1 = =

t

!

01 02 i

t

!

2

t i

(n 0 1) 0 t

i

0 t:

i

t

02

!

+ . . . + (01) 01+ t

i

!

i

t

i

i

i

!

i 1 1 1 (t 0 i) i! 0 (i 0 1)! + 2!(i 0 2)! 0 3!(i 0 3)! + . . . + i(!(0i1)0 i)! i 1h t 1 1 1 (t 0 i) (x 0 1) = 0: i! h1

t

1

1

1

i+1

i

x=1

Therefore St

=

1 nt

cinc tnt01

0 t:

2

To prove Theorem 4.3, we rst estimate, for a given W , the probability P that one player, say player 1, whose initial wealth is x obtains W at some point in the game. For this we use the following fact [Fel68]. x;W

Proposition 5.2.

x

W

+n02



For

Px;W



any x .

x

and

W,

W

11

the

above

probability

Px;W

is

Proof of Theorem 4.3.

Consider P30 with W1 and suciently large W10 . Note rst that

Prfless than two players' wealth reach to W10 g = (1 0 P Prfno player's wealth reaches to W1 g = (1 0 P

x;W

0 )n + nPx;W10 (1

1

x;W1

)

n

0P

x;W

0 )n01

1

:

Then by de nition of P30 , we have P30

= (1 0 P