Biologically inspired features in spiking neural networks Michiel Hermans, Benjamin Schrauwen, Michiel D’Haene, Dirk Stroobandt ELIS research department, UGent: Sint-Pietersnieuwstraat 41 9000 Ghent Belgium e-mail:
[email protected]
Abstract—Neural networks have the power to deal with information which is very hard to process using ordinary approaches, e.g. speech recognition. A recent trend in applying neural networks is to use biologically realistic neuron models. Specifically, neurons are considered which communicate with discrete pulses instead of continuous signals: spiking neurons. In this paper we investigate a small selection of properties which are found in biological neurons and investigate their effect on the general computational performance of spiking neural networks (SNN). Firstly, we investigated the way in which the internal dynamics of the neurons and delayed communication improve the ability to recognize temporal patterns. Secondly we explored an unsupervised adaptation rule which helps to distribute the work equally over all the neurons in the network, so that all neurons are involved in the task they are supposed to solve. It turned out that these biologically inspired features often improved the performance for the tasks investigated.
I. I NTRODUCTION Neural networks (NN) consist of a large number of simple elementary processing units which are connected to each other through weighted interconnections called synapses. The global processing power of NN’s derives from the collective activity of the entire network. Many schemes for training and using neural networks exist, most of which are dependent on the exact way the neurons are connected. Recently, a very generic method to use neural networks has been proposed which is very easy to train and is especially powerful in processing temporal information, such as predicting a chaotic timeseries [11]. In general this technique is called Reservoir Computing (RC), and it can be used with analog as well as spiking neurons. When using analog neurons, the NN is known as an Echo State Network (ESN), and in the case of spiking neurons a Liquid State Machine (LSM) [10], [12]. Due to the fact that RC does not depend explicitly on the type of neuron used, it is very well suited for adopting properties which can enhance the overall computational power. Specifically, when using spiking neurons, one can integrate properties which are measured in biological neural networks and which might play a part in the brain’s massive computing power. This paper is structured as follows: in the next two paragraphs of the introduction we will describe spiking neurons and the nature of it’s internal dynamics, and the details concerning RC. In the following three sections we will investigate different biologically inspired aspects and their influence on the performance on a variety of temporal pattern-classification tasks.
Fig. 1. On the left: schematic representation of a biological spiking neuron. Spikes arrive at the synapses at the end of the dendrites. Spikes which are generated by the neuron itself will travel down the axon towards other neurons. On the right: a typical run of the electric potential over the cell membrane of the neuron. The sharp peaks are the spikes that the neuron itself generates.
Finally, we will look at conclusions which can be drawn from this work and discuss possible future directions. A. Spiking neurons 1) LIF-neurons with exponential synapses: Spiking neurons receive and emit discrete spikes. A spike is a single event which is only characterized by the time of it’s occurrence. In biological neurons, spikes are caused by quite complex electrical behavior, described by 4 non-linear differential equations [9]. Because of this complexity, a large variety of more simple spiking neuron models exist. In this work, we use the Leaky Integrate-and-Fire model (LIF), with exponential synapses. This model has one internal dynamic variable called the membrane potential, denoted by v. Spikes which arrive at the synapses cause a current I which is collected by the neuron, and in the meanwhile the membrane potential leaks away exponentially. The dynamics are described by: v(t) ˙ = I(t) −
v(t) . τm
(1)
Here, τm denotes the membrane time constant which determines the speed at which v decays to zero. Whenever v reaches a certain threshold value ϑ, v is reset to a certain value vr , and the neuron emits a spike (also called the ‘firing’ of the neuron). The current I is the sum of the currents generated by different synapses. Each single spike causes a current which rises immediately, followed by an exponential decay. If a neuron has N incoming synapses with corresponding weights wi , and we denote the set of times of the spikes arriving at the i-th
N X X wi tˆi − t H(tˆi ) exp . τs,i τs,i i=1
(2)
tˆi
Here, H(t) denotes the Heaviside-function, equal to zero when t < 0, and equal to one when t > 0. τs,i is the synaptic time constant of the i-th synapse. The division by τs,i ensures that the total amount of current is normalized with respect to wi , and does not depend on the synaptic time constant. It is now possible to solve equation 1. For notation simplicity we omit the Heaviside-functions, and assume that t > tˆi , i.e. we only add over past events: v(t) = v(0) exp
−t τm
+
N X X
wi i (t − tˆi ),
(3)
i=1 tˆi 200 ms. For the rest of this article we will hence discard this task and work with speech recognition and order recognition. B. With synapse model Next, we investigate whether delays combined with synapse models and fine-tuning the internal time scales can further improve performance. Rather than investigating the entire {τm , τs }-plain like in the previous section, we will consider fixed ratios between τs and τm , and investigate performance in function of τm . The value of dmax is always the optimal value for each task as found in the previous test. The result is presented in Figure 11. The order task surprisingly performs best without synaptic model, however, using a very short internal time scale (τm = 15 ms) does improve performance: 0.65% versus 0.8% when τm was 90 ms in the previous task. The speech recognition task performs best when τs = τm = 45 ms. Here the error rate is about 1.7%, which is slightly better then the optimal result when no delays were used. Delays can apparently improve temporal pattern recognition. Best performance is reached for short internal time scales and long delays. IV. I NTRINSIC PLASTICITY A. Introduction In a randomly constructed reservoir there will be a large variation in the total activity of individual neurons. Some neurons will get stimulated a lot and fire almost all the time, whereas some are inhibited all the time and hardly ever fire. Clearly, neurons that never fire do not contribute to the task they should perform, and neurons that fire all the time can have a disruptive influence on the rest of the reservoir. Biological neurons can adapt their sensitivity for input so that their average firing frequency is the same over long periods of time [17]. This quality is called intrinsic plasticity (IP). We implemented this property by a firing threshold ϑ(t) that decays in linear fashion. When the neuron fires, ϑ is increased
Fig. 12. Effect of IP. The solid black line is the firing-threshold, and the dotted grey line the membrane potential. If a neuron fires too much, the threshold will rise, making it more difficult to fire pulses, and if it fires too little the threshold will fall, making it easier.
parameters were chosen as the optimal values found in the previous section, except that we used τs = τm /2 instead of τs = τm for speech recognition due to the shorter simulation time. IP seems to work very well to reduce average firing ratevariability. The average firing rate itself remains fairly constant while the spread in average firing frequency reduces to a small fraction of that in an unadapted reservoir (Figure 13). The effect on the performance is depicted in figure 14, showing a noticable decrease in error rate for speech recognition and a very small decrease for the order task. V. C ONCLUSION
Fig. 13. Average firing frequency. Error bars depict the average spread in average firing frequency.
with a fixed amount ηIP . The desired average firing-frequency is denoted by νIP . The rate of linear decay of the threshold is then given by ϑ(t) = ϑ(0) − ηIP νIP t. B. Results We used IP to adapt reservoirs so that the variation in activity of individual neurons is reduced. We start off with an unadapted reservoir and measure the average variance in activity between each neuron (expressed as a mean firing frequency), and the error rate. Next, we introduce a plastic phase were IP is turned on and the reservoir is fed with the complete dataset. IP adapts the thresholds, and next we turn off IP, and train and test again and do the same measurements. This process is repeated a few times. ηIP is chosen to be 5 · 10−4 . In order to fairly compare performance before and after the adaptation-phase, νIP is set as the average firing frequency as measured in the unadapted phase. The reservoir
Fig. 14. Average error rate of the different tasks in function of the number of adaptive cycles
This survey of easily implementable features for spiking Reservoir Computing gives promising results. All features investigated in this article gave improvement in performance on temporal pattern classification. Synapse models as well as delays can be adjusted for the time scale neccesary for the given task, and intrinsic plasticity provides an easy way of regulating a reservoir to improve it’s performance. Future research can look at many other biologically realistic features such as dynamic synapses [14], spike-timing dependant plasticity [13], etc. ACKNOWLEDGMENT All simulations were carried out using the RC-Toolbox, which is an open source MATLAB-environment for easily setting up experiments and processing results. The actual simulation of spiking neurons was done with ESSpiNN [7], an event simulator written in C++, also open source. Both were developed at the ELIS-research department of Ghent University and can be downloaded from http://snn.elis.ugent.be/. This article is a summary of the work done in the masterthesis of the first author [8]. R EFERENCES [1] E. Antonelo, B. Schrauwen, and D. Stroobandt. Experiments with reservoir computing on the road sign problem. In Proceedings of the VIII Brazilian Congress on Neural Networks (CBRN), Florianopolis, 10 2007. [2] Catherine E. Carr. Processing of temporal information in the brain. Annual Review Neuroscience, 16:223–243, 1993. [3] L. A. Jeffres. A place theory of sound localization. Journal of Comparative and Physiological Psychology, 41:35–39, 1948. [4] Richard F. Lyon. A computational model of filtering, detection, and compression in the cochlea. Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP ’82., 7:1282–1285, 1982. [5] Benjamin Schrauwen and Jan Van Campenhout. BSA, a fast and accurate spike train encoding scheme. Proceedings of the International Joint Conferrence on Neural Networks, 4(20):2825–2830, 2003. [6] E. Antonelo, B. Schrauwen, and D. Stroobandt. Event detection and localisation for small mobile robots using reservoir computing. Neural Networks, 21(6):862-871, 2008. [7] M. D’Haene, B. Schrauwen, J. van Campenhout, and D. Stroobandt. Accelerating event based simulation for multi-synapse spiking neural networks. To appear in Neural Computation, 2008. [8] M. Hermans. Biologisch genspireerde aspecten in gepulste neurale netwerken. Master’s thesis, UGent, 2008. [9] A. L. Hodgkin and A. Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. Journal of Physiology, 117:500–544, 1952.
[10] H. Jaeger The echo state approach to analyzing and training recurrent neural networks. Technical Report University of Bremen, German National Research Center for Information Technology., 148, 2001. [11] H. Jaeger and H. Haas. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science, 304(5667):78–80, 2004. [12] W. Maass and T. Natschl¨ager. Real-time computation without stable states: a new framework for neural computation based on perturbations. Neural Computation, 14:2531–2560, 2002. [13] H. Markram, J. L¨obke, M. Frotscher, and B. Sakmann. Regulation of synaptic efficacy by coincidence of postsynaptic APS and EPSP’s. Nature, 375:213–215, 1997. [14] T. Natschl¨ager and W. Maass. Finding the key to a synapse. In NIPS, pages 138–144, 2000. [15] B. Schrauwen and J. van Campenhout. Extending Spikeprop. Proceedings of the International Joint Conferrence on Neural Networks, 1:471– 476, 2004. [16] D. Verstraeten, B. Schrauwen, D. Stroobandt, and J. van Campenhout. Isolated word recognition with the liquid state machine: a case study. Information Processing Letters, 95(6):521–528, 2005. [17] W. Zhang and D. J. Linden. The other side of the engram: Experiencedriven changes in neuronal intrinsic excitability. Nature Reviews Neuroscience, 4:885–900, 2003.
