Fuzzification of Spiked Neural Networks

Fuzzification of Spiked Neural Networks David Reid Liverpool Hope University [email protected]

Abstract Biological systems are slow, wide and messy whereas computer systems are fast, deep and precise. Fuzzy Neural Networks use fuzzy logic to implement higher level reasoning and incorporate expert knowledge into the system while neural networks deal with the low level computational structures capable of learning and adaptation. Whereas the first 2 generations of neural network are “rate encoded”, Spike neural networks (SNNs) are a relatively new type and potentially very powerful neural network (so called 3rd generation of neural network) that uses temporal encoding of information in a much more biologically realistic way than previous generations. This paper demonstrates how fuzzification of SNNs (FSNNs) may take place using interval Type-2 Fuzzy Sets (IT2FS).

Maybin Muyeba Liverpool Hope University [email protected]

2. Spiked Neural Networks Spiking neurons [6] are a third (3rd) generation of neural networks that use temporal data in order to code information. This new type of neural network can be defined as follows: Let N be a set of spiking neurons. Let P be a set of pre-synaptic neurons such that P ⊆ N , where the axon of each

p j ∈ P makes contact with the dendrites

or soma of each

ni ∈ N to form a synapse. A weight

w pj ,ni > 0 is given for a synapse, a response function

ε p ,n : R + → R for j

i

function

each synapse and a threshold

+

Θ ni : R → R for each neuron. K

Thus, there exists set

1. Introduction Spiking neural networks (SNN) are more biologically realistic models than their predecessor generations of neural networks (NN). As such neural learning is preferable using SNN. A number of researchers have recently been looking at several new types of Recurrent Neural Network (RNN) [19] that utilize their powerful temporal processing capability by using a randomly initialized, but static “reservoir” of spiking neurons in order to filter input patterns for classification. However powerful these are, hybrid systems that combine neural learning and fuzzy reasoning are more powerful [5, 10, 14]. Based on this, the paper addresses a hybrid neuro-fuzzy approach using spiking neural networks (SNN) and interval Type-2 Fuzzy Sets (IT2FS). The paper is organized as follows: Section 2 presents Spiked Neural Networks, section 3 presents Fuzzy Logic, section 4 presents Fuzzy Spiked Neural Networks, section 5 Entity Fuzzification, section 6 shows an implementation overview and a conclusion in section 7.

S = Υ{s k } of K synapses, k =1

with each synapse

s k forming a connection between each p j to each ni i.e. s k ⊆ PxN . Each ni receives a spike at a specific time t from each of the pre-synaptic neurons p j . The synapse

s k receiving this spike will have an

inhibitory or excitatory response (figure 1(a)). This is the Post Synaptic Potential (PSP). If the neuron has a threshold set to some specific k

value

Ti then it will fire if at time t

∑s

n

≥ Ti or vice

n =0

versa when this value is less (figure 1(b). When firing does take place (figure 1(c)), there follows a relative refractory period when the possibility of firing again is diminished and also an absolute refractory period where there is no possibility of firing. The PSP function usually approximates the synaptic function by using a sharp exponential rise and slower decay in response to each spike. In practice the following equation is a good approximation:

  s exp − k  τ )   ni

1

ε (sk ) =

τs

1− (

k

τn

   − exp sk  τ n   i

  H (sk ) (1)  

k

H ( S k ) is a Heaviside step function which vanishes for s ≤ 0 and has the value of 1 for s > 0 , where

τS

k

is a synaptic time constant and

τN

i

the firing times of its neighbours to fire in sympathy with the winning neuron. Again, as with SpikeProp, the firing time is shifted toward the winning neuron. Distant neurons firing times are moved away from the firing time of the winning neuron. A typical algorithm uses the physical distance to determine how firing times should be synchronized with the winning neuron

is a neuronal

time constant. In practice we have found that equation (2):  s  s  (2) ε ( s k ) = k 1 − exp − k   τ S  τ S k  k   is an adequate approximation. Learning can be both supervised and unsupervised in such a network. Learning in the supervised network is usually implemented using the SpikeProp method [7]. This is very similar to the error-backpropagation method by Rumelhart [8] except that the firing time of the neuron ni is shifted by altering its refractory period(s). Thus, learning is achieved by temporally co-coordinated firing of related neurons.. Excitatory delay

3. Fuzzy Logic Fuzzy logic is used to capture uncertainty in data and systems that are not modeled on crisp concepts (or bivalued logic) [1]. Relationships between membership values of elements in a set and a descriptive (linguistic) name defining the set is determined by a membership function. As such, type-1 fuzzy sets try to model words (linguistic names) and have been found to be insufficient for modeling various type of uncertainties [2]. Mendel [3] argues that rule-based fuzzy systems where antecedents and consequents of a rule are just words but modeled using mathematical functions which are completely certain (including membership degrees) are inappropriate. Type-2 fuzzy sets (T2FS) [3] tend to blur the membership degrees of type-1 fuzzy sets i.e. membership grades are also fuzzy. This blurring represents a Type-2 fuzzy set. A type-2 fuzzy set à then is characterized by three dimensions , x, u and

µ~ A~ ( x, u ) , ∀x ∈ X and ∀u ∈ J x ⊆ [0,1] with 0 ≤ µ~ A~ ( x, u ) ≤ 1. For a continuous universe

spike

of

discourse, this expressed as equation (3): Inhibitory

~ A=

~ ~ ( x, u ) /( x, u ) Jx ⊆ [0,1]} A

∫ ∫µ

(3)

x∈ X u∈Jx

(a) Synapse response to spike

Ti

(b) Non-Firing

Ti

Refractory Periods rel. abs.

(c) Firing Figure 1. Spiking neuron activity Unsupervised learning usually involves selecting a neighbourhood around a winning neuron and altering

For discrete data, the integral is replaced by

∑ . Type-

2 fuzzy sets thus have a footprint of uncertainty (FOU) which we shall show in section 5. Most neuro-fuzzy networks use continuously differentiable Radial-basis functions (RBF) [12] or Gaussian RBF, to model membership functions. We use this function in modeling the fuzzy spiked neuron using Type-2 fuzzy logic, which contrasts with Type-1 fuzzy logic [14, 15, 16]. We also use triangular fuzzy sets for modeling neuron neighbourhoods (see section 5.4).

4. Fuzzy Spiked Neural Network We propose that there are two fundamental ways to apply fuzzy logic to a neural network. The “traditional” way of producing a hybrid neuro-fuzzy system is by fuzzifying the rule set [5, 9, 10, 11, 12, 14]. This type

of neuro–fuzzy system is independent of the underlying structure. It focuses on fuzzifying the rules derived from the trained neural network. It also takes no account of the underlying functionality of the neural network itself. The alternative is to try and fuzzify the workings and entities in the neural network itself. This is a less popular way to create a neuro-fuzzy system although recently a number of researchers have been creating such systems . To our knowledge there are only a handful of researchers using type-2 fuzzy logic in conjunction with neural networks and there are currently no researchers using spiked neuro-fuzzy systems. In [5], a hybrid learning algorithm that uses an interval type-2 Takagi-Sugeno [13] fuzzy neural network model with learning rules is presented. The authors use t-norms and s-norms for aggregating membership values from interval type-2 fuzzy neurons. However, the authors use a traditional network of adaptive linear neurons (MADALINE) for adapting consequents and use backpropagation for learning to determine premise parameters. We propose that using type-2 fuzzy logic in modeling the entities and functionality of a spiked neural network system has the potential to be a powerful way to encapsulate knowledge in a neural system and is also biologically relevant. In essence type-2 fuzzy systems more accurately capture the uncertainties and randomness of the structure, behavior and functionality of real biological systems.

4.1. Neural Fuzzification and Rule Learning 4.1.1. Learning Fuzzy Rules Learning fuzzy rules using the “traditional” learning capability of a neural network to learn which rules to use or promote in a fuzzy system has been the standard way to implement neuron-fuzzy hybrid learning rules [9, 11, 12, 14]. These tend to be similar in structure to a multi-layer neural network with normal input and output layers that use crisp values. Usually this type of system. has 3 hidden layers that represent the membership functions and the fuzzy rules. In layer 1 each neuron transmits crisp signals to the next layer. In layer 2 the neurons in this set represent the fuzzy sets used in the antecedents of the fuzzy rules. Layer 2 takes layer 1 crisp values and determines to what degree the input belongs to this neuron’s fuzzy set. The activation function of this neuron is usually set to the function that defines the neurons fuzzy set (usually triangular or trapezoidal). In layer 3 each neuron corresponds to a single fuzzy rule and receives

input form layer 2 (the rule sets) The weights between these layers represent s the degree of confidence that a rule should be used for particular antecedents. Conjunction of rule antecedents is achieved by the fuzzy operation interaction. Layer 4 uses the fuzzy operator union to combine the fuzzy rule neurons in layer 3. This layer integrates the combined firing strength from the previous layer. Finally Layer 5 defuzzifies the result, usually by the sum-product composition method, ready for crisp output.

5. Entity Fuzzification Entity fuzzification attempts to fuzzify the various elements of the neural network itself. There are two main points to note; firstly, as we are using a relatively new type of (spiking) neural network, the fuzzification of the neuronal element is novel. Secondly, as we are using a neural system where the control of the temporal fluctuations of perhaps even a single spike may be significant, it is likely that there is both noisy data for the fuzzy inference system and uncertainty about what a sequence of spikes actually represents at a particular point in time (uncertainty about spike decoding). For this reason it is more appropriate to use a Type-2 Fuzzy Inference System. What follows are the main ways in which the various elements of a spiking neural network can be integrated into a type-2 fuzzy system.

5.1. Synapse Fuzzification The synapse can be fuzzified in a number of ways. The main characteristics that we can fuzzify are the delay before it fires and the function used to simulate the post synaptic potential. The modification of these parameters results in different area of response to an individual spike. This in turn influences consequent neuronal firing. It is suggested that this functionality is directly analogous to creating a number of different fuzzy sets which may subsequently influence firing. In other words by modification of parameters we are creating a fuzzy set of fuzzy sets, or a Type-2 Fuzzy System. The Footprint Of Uncertainty (FOU) in such a mapping represents the area where the PSPs overlap. This is illustrated in figure 2 where the shaded area represents the FOU:

Figure 2. A Synapse FOU

5.2. Neuron Fuzzification The neuron itself instigates a spike if the sum of the PSPs at a particular point in time crosses a threshold; after which a refractory period inhibits or prohibits further firing (discussed in section 2). In a similar way to the fuzzification of synapses, neuron functionality can also be fuzzified. This is illustrated in figure 3:

Ti + n Ti

be that neuron B spikes first. This later occurrence may or may not be interpreted as being significant (Case 1 or Case 2 in figure 4). Thus, again we have a fuzzy set of fuzzy sets; a Type-2 FS. Neuron A Neuron B Neuron C Case 1

Case 2

At ≤ Bt ≡ At At ≤ Bt ≠ At where At is the firing time of neuron A. Figure 4. Fuzzy Spike Encoding Times

Figure 3. Neuron FOU In the above example it is assumed that the PSP itself isn’t fuzzified as described in 5.1. It is clear that if this too is fuzzified then only a Type-2 FS would be able to represent this type of functionality.

5.3. Spike Train Fuzzification So far we have only examined the fuzzification of the neuron and the synapse. However, within the spiking neural network the greatest scope for fuzzification, and in particular Type-2 FS is in the spike encoding scheme itself. Individual spikes may or may not be important in a encoding scheme and as such could be either considered noise or be considered an integral part of a specific, but debatable, schema. In either case, a Type2 FS is well suited to these uncertainties. What follows are the main encoding schemes and a description of how each may be fuzzified. It is worth noting that all three encoding schemes have been observed in biological systems and that all three encoding schemes also have a significant amount of fuzziness in these systems. 5.3.1. Time to First Spike Encoding In figure 4, 3 neurons are used as input. The encoding scheme used is that the neuron which fires first signifies a specific input pattern as not only the time of the first spike to arrive equivalent to a singleton in a fuzzy set, but also the winning neuron. These consequent interpretations of the input pattern, subject to uncertainty, then this is best modeled by a Type-2 FS. For example the first spike (ringed for neuron A), could occur in a number of places (shaded), or it could

5.3.2. Spike Interval and Phase Encoding Repeating patterns of activations between neurons may be a way of encoding information. The intervals between phases may signify something about the input pattern. The length of the interval and the neurons used is again a fuzzy decision. As with the time to first spike (see example in figure 4), the interpretation of the significance of the phases is, again, also a fuzzy decision, a Type-2 FS. Neuron A Neuron B Neuron C

Interval Phase

Figure 5. Fuzzy Spike interval and phase encoding 5.3.3. Correlation and Synchronous Encoding Finally, there may be a strong correlation between patterns of spiking activity. Different neurons may spike in sync with each other at different times. Neuron A Neuron B Neuron C Neuron D Neuron E Figure 6. Fuzzy Spike correlation

The extent of the correlation, the neurons used and the interpretation for the synchronicity between neurons are all fuzzy decisions, Type-2 FS. In figure 6, neurons D and E seem to spike (in a fuzzily define interval) when neurons A,B and C fire (again, within a fuzzily defined interval).

5.4. Neighbourhood Fuzzification Recent work with spiked neural networks has started to look at recurrent neural networks [19] where the output of one time interval is fed back into subsequent training epochs. Most of these systems are similar in nature to Kohonen Self Organizing Maps [20], but use the temporal nature these systems to shift firing times of neighboring neurons in accordance to the distance of a winning neuron. The extent of the shift in firing times, the extent of the neighbourhood and the significance of the zones of sensitivity to input patterns are all fuzzy decisions and as such correspond to a Type-2 FS. Figure 7 illustrates how the neighbourhoods may be fuzzified using triangular Type-2 FS where the shaded areas show the FOU. Neighbourhoods

parallelization are favoured. As the Karnik-Mendel algorithm [17] computes centroids independently of each other, this iterative algorithm is capable of some parallelization. However, the algorithm that we are currently investigating for parallelization on CUDA uses minmax uncertainty bounds (Wu-Mendel uncertainty bounds). With this algorithm the type reduction stage, for example as shown in [5], is bypassed by sampling averaging the uncertainty bounds of the consequent interval Type-2 FS centroids. Mendel has reported a surprising high level of accuracy using this approximation. It is also possible to parallelize such a process much more readily rather than using the iterative Karnik-Mendel algorithms for type reduction. In short the system we propose is shown in figure 8 and a detailed design architecture in figure 9. Neural Entity Crisp Inputs

Rules Fuzzification Fuzzy Input Set

Inference

Fuzzy Output Set

Figure 8. Overview Type-2 Neuro-fuzzy system C

B

A

A

B

C

Figure 7. Neighbourhood fuzzification

6. Implementation Overview Currently we are using Mendel’s software [18] within MatLab to evaluate type-2 fuzzy systems. A number of projects to implement spiked neural networks to complement interval Type-2 FS are also progressing using Field Programmable Gate Arrays (FPGA) in HandleC and Stream-processing (in CUDA). As Type-2 fuzzy systems are computationally more expensive than type 1 FS, they require a type – reducer or a direct defuzzifier, which is often an iterative process, algorithms that are capable of

In figure 8, we fuzzify one of the neural entities as discussed earlier in the paper. Crisp input in this case could be for example, in the case of spike train encoding, the spikes activations. The fuzzy output set along with the crisp inputs is then sent to the output processor. The output processor averages the uncertainty bounds for both the upper membership function and lower membership function of the consequent centroids. Similarly the upper and lower firing level is calculated. These calculations provide us with the uncertainty bounds for the system. The left lower bound and upper bound are averaged, as are the right lower bound and upper bound. Finally these two averages are, in turn, averaged to provide a crisp output value (see figure 9). In this way the bounds of the FOU are taken into account during estimation of a crisp output value (see Mendel [18] for more details).

7. Conclusion In this paper, we introduced spiked neural networks with fuzzy properties and proposed a fuzzy Spiked Neural Network (FSNN) that shows how the structure and functionality of the neurons can be fuzzified using

Type-2 FS. This new approach seems to model biological neurons much more sensibly than previous neuro-fuzzy models using Type-1 FS than the spiked model. It is also our belief that such an approach renders itself more applicable to FPGA and has a parallelization potential. Output Processor Uncertainty Defuzzification Bounds Firing Level Crisp Input

Left LB

UFL LFL

Left UB

Centroids Fuzzy Output Set

UMFC LMFC

Avg Crisp Avg

Output

Right LB

Avg

Right UB

Complete System

Crisp Inputs

Neural Entity

Fuzzy Output Set

Output Processor Crisp Output

Figure 9. Fuzzy Spiked Neural Network (FSNN) architecture

9. References [1] Klir, G. J., and B. Yuan, Fuzzy Sets and Fuzzy Logic, Theory and Applications, Prentice-Hall, 1995. [2] J. M. Mendel, and R. I. B. John, “Type-2 Fuzzy Sets Made Simple”, IEEE Transactions on Fuzzy Systems 10(2), 2002, pp.117-127. [3] J. M. Mendel, “Type-2 Fuzzy Sets: some questions and answers”, IEEE Neural Networks Society, August 2003, pp. 10-13. [4] L. A. Zadeh, “The concept of a linguistic variable and its application to approximate reasoning - 1”, Information Sciences (8), 1975, pp. 119-249. [5] J. R. Castro, O. Castillo, P. Melin and A. RodriguezDiaz. “Hybrid learning algorithm for interval type-2 fuzzy neural networks”, IEEE Int. Conf. on Granular Computing, 2007, pp. 157-162. [6] W. Mass, “On the computational complexity of networks of spiking neurons”, Advances in Neural Information Processing Systems, 7, 1995, pp. 183-190.

[7] S. M. Bohte, J. N.. Kok and H. L. Poutre, “Errorbackpropagation in temporally encoded networks of

spiking neurons”, Neurocomputing, vol. 48, 2002, pp. 17-37 [8] D.E. Rumelhart, G. E. Hinton and R. J. Williams. “Learning representations by back-propagation errors”, Nature 323, 1986, pp. 533-536. [9] K. B. Cho and B. H. Wang, “Radial basis function based adaptive fuzzy systems and their applications to system identification and prediction”, Fuzzy Sets and Systems, vol. 83, no. 3, pp. 325-339, Nov. 1996. [10] D. Nauck, R. Kruse, “Neuro-fuzzy systems for function approximation”, Fuzzy Sets and Systems, vol. 101, no. 2, pp. 261-271, Jan. 1999. [11] C.S. Velayutham and S. Kumar, “Asymmetric subsethood-product fuzzy neural inference system (ASuPFuNIS)”, IEEE Trans. Neural Networks, vol. 16, no. 1, Jan. 2005, pp. 160-74. [12] C. Juang, and C. Lee, 2007. A fuzzified neural fuzzy inference network for handling both linguistic and numerical information simultaneously. Neurocomputing, vol. 71, no. 1-3, Dec. 2007, pp. 342352. [13] T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications to modeling and control”, IEEE Trans. Syst. Man. Cybern, vol. SMC-15, Feb. 1985, pp. 116-132. [14] C. Li, K. Cheng and J. Lee, “Hybrid learning neuro-fuzzy approach for complex modeling using asymmetric fuzzy sets”, IEEE Int. Conf. Tools with Artificial Intelligence, Nov. 2005, pp. 397-401. [15] W. Pedrycz, “Neurocomputations in relational systems”, IEEE Tran. on Pattern Analysis and Machine Intelligence , vol. 13, no. 3, Mar. 1991, pp. 289-297 [16] W. Pedrycz and A. F. Rocha, “Fuzzy-set based model of neurons and knowledge-based networks”, IEEE Trans. On Fuzzy Systems, vol. 1, pp. 254-266. [17] W. Dongrui, J. M. Mendel , “Enhanced KarnikMendel Algorithms for Interval Type-2 Fuzzy Sets and Systems” North American Fuzzy Information Processing Society, 2007. NAFIPS, vol. _no. 24-27, Jun. 2007, pp. 184 – 189. [18] J. Mendel, Uncertain Rule-based Fuzzy Logic Systems Introduction and New Directions, NJ: Prentice-Hall, 2001. [19] D. P. Mandic, D. P. and Chambers, J. A, “Recurrent Neural Networks for Prediction”, editor(s): Simon Haykin,, Wiley Series in Adaptive and Learning Systems for Signal Processing, Communications, and Control , 2002 [20] T. Kohonen, E. Oja, O. Simula, A. Visa and J. Kangas, “Engineering applications of the selforganizing map”, Proc. of the IEEE, vol. 84, no. 10, 1996, pp. 1358-84.