(Knight, 1972; Gerstner, 2000; Vogels and Abbott, 2005), see also (Yu et al, 2002). Methods. For the network ..... In: Chalupa L, Werner. (eds) The Visual ...
JCNS manuscript No.
(will be inserted by the editor)
Accurate multiplication with noisy spiking neurons Panagiotis Nezis · Mark C. W. van Rossum
the date of receipt and acceptance should be inserted later
Abstract Multiplication is an operation which is fundamental in mathematics, but it is also relevant for many sensory computations in the nervous system. Nevertheless, despite a number of suggestions in the literature, it is not known how multiplication is implemented in neural circuitry. We propose a simple feedforward circuit that combines a rate model of neural activity and a realistic input-output relation to accurately and e ciently implement multiplication of two rate-coded quantities. Using a simulation of a network with integrate and re neurons we demonstrate the feasibility of the circuit.
Introduction After addition and subtraction, one of the basic computations in mathematics is multiplication. It is a core ingredient in many elds of mathematics. One example of the use of multiplication in mathematics is the StoneWeierstrass theorem, which states that any function can be accurately approximated by polynomials. Hence once a multiplication is implemented and thus powers can be calculated, it becomes possible to approximate any continuous function. Multiplication is also an essential operation in the nervous system, in particular in sensory systems. Many sensory computations such as motion detection (Reichardt, 1957), looming stimulus detection (Gabbiani et al, 2002), and auditory processing rely on multiplication operations. The strongest evidence that multiplication can be done accurately in the nervous system comes from the barn-owl auditory system (Peña and Konishi, 2001; Fischer et al, 2007), where it was observed that the sub-threshold responses of neurons accurately code the product of two stimulus parameters (the inter-aural level dierence and the inter-aural time dierence). Other evidence for approximate multiplication and gain control has been found in many systems, such as in binocular interaction (Freeman, 2004), in head and gaze direction gain elds (Andersen et al, 1985; Brotchie et al, 1995), attentional modulation (Treue and Trujillo, 1999; Institute for Adaptive and Neural Computation School of Informatics, University of Edinburgh 10 Crichton Street Edinburgh, EH8 9AB UK
2
McAdams and Maunsell, 2000) and in motor planning (Sohn and Lee, 2007; Hwang et al, 2003). Nevertheless, exactly how the nervous system multiplies signals is unknown. This question is intimately connected to neural coding. For instance, if ring rates do not code the actual signals but the log of the signals, multiplication becomes a trivial addition (see below). If instead population codes are used, radial basis functions can be used for computations (Haykin, 1998). Like most other work, we exclusively consider multiplication of ring rates in this study, which is presumably most relevant for early sensory processing. Addition and subtraction of ring rates is easily implemented by synaptic excitation and inhibition, however it is unclear how ring rates are multiplied. Various suggestions have been made how the neural circuits can multiply ring rates (see Koch and Poggio, 1992, for an overview). For instance, to calculate the product of the rate of two Poisson trains, one can use that the coincidence rate of both processes rate is the product of the rates (Srinivasan and Bernard, 1976). However, this method requires that only a few inputs are simultaneously active within a neuron's integration time-constant, which is unrealistic as most neurons receive many inputs. An alternative is to use that x.y = exp(log x + log y), i.e. multiplication can be done by addition in the log domain. Of course this needs to be followed by an exponentiation. Evidence for this multiplication algorithm has been found in the locust visual system (Gabbiani et al, 2002). A number of researchers have explored the possibility of doing non-linear computation in a single neuron using the interaction of excitatory and inhibitory inputs on the dendritic tree (Koch, 1999), and the voltage dependence of NMDA receptors (Mel, 1992, 1993). Gain modulation has been proposed to be implemented through dendritic interaction (Larkum et al, 2004; Mehaey et al, 2005), balanced (Chance et al, 2002) and unbalanced synaptic input (Murphy and Miller, 2003). Finally in population coding networks, recurrent excitation and inhibition can be used to multiply inputs that are additive to the single neuron (Salinas and Abbott, 1996). In this study we propose simple feedforward circuits that multiply two ring rates. The circuits are then implemented and simulated using noisy integrate and re neurons. Noisy integrate-and- re networks have received recent interest because, despite the presence of noise, they can accurately transmit and compute with rate-coded information while operating in a regime presumed comparable to cortical networks in vivo (Knight, 1972; Gerstner, 2000; Vogels and Abbott, 2005), see also (Yu et al, 2002).
Methods For the network simulations, we used integrate-and- re neurons, with parameters as in van Rossum et al (2002), except for the noise (see Results). The membrane potential V (t) obeyed, dV (t) τ = −(V (t) − Vrest ) + Iin dt with input resistance Rin = 100M Ω , resting potential Vrest = −60mV , and a membrane time constant τmem = 20ms. Whenever the membrane potential reached the threshold potential Vthr = −50mV , it was reset to the resting potential, Vreset = Vrest . We used 20 neurons per population; the inputs were also modeled as neural populations, thus the circuit depicted in Fig. 2a contained 120 neurons in total. The input current to each neuron was provided by a non-speci c noise current (see Results) and
3
synaptic input, Iin = Inoise + wexc
X
rev gexc (t)[V (t) − Vexc ] + winh
X
rev ginh (t)[V (t) − Vinh ]
where the sum runs over all excitatory/inhibitory inputs. AMPA-type excitatory synapses were implemented with a reversal potential of 0mV, and inhibition was implemented through GABA-A type synapses with a reversal potential equal to the resting potential, . The synaptic conductances jumped at the time of each presynaptic spike ti and decayed exponentially between spikes, that is, τsyn dg(t) dt = −g(t) + δ(t − ti ). We assume the same time-constant for both excitation and inhibition, τsyn = 5ms. In contrast to the excitatory synapses, the reversal potential of inhibitory synapses is close to the mean membrane potential. To obtain excitatory and inhibitory synaptic currents of similar magnitude, the inhibitory conductances were set seven times larger than the excitatory ones, winh = 7wexc . The strength of the excitatory synapses was set so that the output neurons had ring rates in a physiologically plausible range.
Results Approximate multiplication The central idea of this paper is that the product can be approximated by the minimum function. (The AND-function in the binary domain). This is illustrated in Fig. 1. The 2 ring rate along the x-axis changes according to a Gaussian curve fA (x) = e−x , while along the y-axis the ring rate is fB (y) = cos(y) + 1. For concreteness we use the same choice of inputs as in Peña and Konishi (2001), but our arguments are general. In panel 2 a the exact multiplication of these ring rates is plotted, i.e. fA×B (x, y) = e−x [cos(y)+ 2 1]. In panel b the minimum of these functions is shown, min(e−x , cos(y)+1). Although there are clear dierences between the minimum and the multiplication, the minimum function is a decent initial approximation to the multiplication. The approximation of the multiplication by the minimum function is useful because the minimum function can be implemented easily in feed-forward networks. Fig. 2 depicts two circuits that compute the minimum of two ring rates. The squares represent small populations of neurons, the activity of which is measured by their ring rate. By excitatory and inhibitory synapses, the various populations add or subtract the inputs fA and fB followed by a recti cation. First, we assume that the recti cation of the nodes, denoted by h(x) is simply a threshold linear relation. That is, h(x) = [x]+ , where [x]+ = max(x, 0). In that case the circuit in Fig. 2a calculates [[fA + fB ]+ − [fA − fB ]+ − [fB − fA ]+ ]+ = 2 min(fA , fB )
This is easily checked by assuming fA < fB , or, symmetrically, fA > fB . However, as we shown next, these circuits can approximate a multiplication much better. The sharp thresholding used above is only an approximation of a real neuron's input-output relation. It has been argued using data (Anderson et al, 2000) and models (Miller and Troyer, 2002; Hansel and van Vreeswijk, 2002) that noise smooths the inputoutput relation so that it approximates a power-law. That is, the input-output relation can be written as h(x) = ([x]+ )n
4
For very high ring rates one expects saturation to become important, however, in vivo ring rates seem mostly far from saturated. Importantly, when such a power-law transfer function is used for the circuit depicted in Fig. 2a, the approximation of the multiplication is much better, as is shown in Fig. 1c. To measure the accuracy of the circuit of Fig. 2a we compare its output (Fig. 1c) to the true multiplication (Fig. 1a). We calculate the root mean square (RMS) error, after scaling the output to have the same peak response. Numerically we nd that with h(x) = ([x]+ )1.45 , the multiplication is most accurate for the example inputs of Fig. 1. The error of the circuit is on average 3.3% of the maximal response, and nowhere more than 13%. If, in contrast, threshold linear units are used (corresponding to an exponent n = 1) so that the circuit calculates the minimum, the error is 14% on average and maximally 49%. Intuition for the optimal value for n can be gained by considering fA = fB , in that case only the left center node in Fig. 2a contributes to the output node; the function that the network then calculates becomes h(h(fA )) = fA 1.9 , close to the desired output fA 2 . Note, that when the transfer function of the rst 3 nodes is h(x) = [x]2+ , and of the nal node is h(x) = [x]+ , the output of the network is exactly a multiplication. However, as this requires dierent transfer functions for dierent nodes, this seems less biologically realistic. The other circuit, Fig. 2b, also calculates the minimum of the two ring rates, according to [fA − [fA − fB ]+ ]+ = min(fA , fB )
It uses less nodes, and is the smallest circuit we could nd that calculates a minimum. Nevertheless, this circuit approximates the multiplication less well when power-law transfer functions are used than Fig. 2a, (average RMS error 12%, maximal error 50%), and will not be considered further.
Implementation using a spiking network To examine the multiplication in spiking networks we use small populations of noisy integrate-and- re neurons connected in a feed-forward fashion with conductance based synapses to implement the circuit of Fig. 2a (see Methods). Apart from the synaptic input, the neurons are injected with a noisy bias current. As a result the input-output relation is smoothed and synchronization between neurons is suppressed. With this noise, information coded in ring rates is transmitted rapidly and with little distortion (Knight, 1972; Gerstner, 2000). We used Gaussian noise with a mean of 30pA and a standard deviation of 100pA, ltered with a 2ms time constant (compared to a mean of 55pA, and 70pA SD in an earlier study (van Rossum et al, 2002)). We reduced the bias current a bit to reduce spontaneous background rates and slightly improve the approximation of the multiplication. Synaptic input is provided by the neurons in the previous layer via either excitatory and/or inhibitory synapses. For simplicity we assume that input populations can make both excitatory and inhibitory synapses, rather than creating excitatory and inhibitory input populations separately (simulation with distinct excitatory and inhibitory populations gave identical results, not shown). The ring rate of a neuron receiving excitatory input at a rate fe and inhibitory input at a rate fi , was approximately h(fe − fi ). In other words, the inhibitory was subtracted from the excitation before the non-linearity converted the input current to a ring rate, cf. (Holt and Koch, 1997).
5
Performance of the spiking network In Fig. 3A the spike trains of the inputs and the output of the network are shown. As an illustration, we injected one input population with a staircase patterned current, while the other input received a sawtooth shaped current. As expected from the injected noise, both input and output spike trains are quite noisy and asynchronous. The response averaged over 20 trials, shown in Fig. 3B (dots), closely matches the true scaled multiplication (solid line). The latency in the network (not shown) is about 10ms, twice the synaptic time constant, as expected from earlier studies. Both systematic and random errors contribute to the dierence between the network's output and a true multiplication. The random errors are caused by the randomness of the activity and disappear after su cient averaging, while the systematic error is due to the network not quite implementing a multiplication. After averaging the remaining systematic error is 5.9% RMS of the maximal response. This is close to the error of the circuit of Fig. 2b, which yields an average error of 4.8% RMS using this choice of inputs. The systematic error is plotted in Fig. 3C as a function of the parameters of the noise current: mean and standard deviation. Both components of the bias current are important to make the multiplication as accurate as possible, while without any bias current the error is much higher. The mean current ensures that small inputs lead to some response and are not thresholded. The noise current (measured through the standard deviation) prevents synchronization of the neurons (van Rossum et al, 2002). It should be noted that even without noise, but an appropriately tuned mean current, the performance is reasonable, Fig. 3C. The reason is that the network is not very deep and therefore synchronization of the ring is limited. Finally, the minimum in Fig. 3C is quite broad, i.e. for an approximate multiplication the precise values of the bias current parameters are not very critical.
Discussion We explored how circuits of spiking neurons could implement a multiplication of two ring rates. We found that a simple feed-forward circuit can approximate a multiplication accurately, assuming a powerlaw neural transfer function. If the transfer function of the neurons were threshold-linear, the circuit would implement a minimum function, but the smoothing powerlaw transfer function typically observed in cortical neurons causes the network to accurately approximate a multiplication. A critical ingredient is the noise of the neurons which is essential to de-synchronize the neurons and smooth the transfer function. The origin of the noise could be manyfold: intrinsic noise, unspeci c other inputs, or random uctuations resulting from a chaotic network state. There has been a recent interest in networks of noisy spiking neurons, as they are assumed to be realistic models of cortical networks. These networks have been shown to have small latencies, realistic statistics (van Rossum et al, 2002) and can calculate various binary functions (Vogels and Abbott, 2005). The current work suggests that they are also useful for analog computations based on rate codes. As such this work might be useful for engineers that want to build sensory circuits with spiking neurons. The model makes the following predictions: when inhibition is blocked the output of the circuit is distorted in a predictable manner: the ring rates increase, and the
6
multiplication becomes less accurate. When inhibition is completely blocked, the network calculates an exponentiated sum of the ring rates (fA + fB )2n . Furthermore, the proposed network will become less accurate when noise or bias current are changed (Fig. 3c), but these parameters are hard to manipulate experimentally. Despite the abundance of multiplication and multiplication-like responses in many species and brain regions, current knowledge of the underlying circuitry is often scarse. A possible application of the circuit is to generate multiplicative binocular interactions in primary visual cortex (e.g. Freeman (2004) in references therein). Importantly, the proposed circuitry does not use any `exotic' elements, but only inhibitory-excitatory pairs, while a noise-induced expansive non-linearity has been observed in visual cortex (Anderson et al, 2000). Our result is not only relevant when sensory ring rate signals need to be multiplied. Arbitrary computations on population coded information can be done using radial basis functions (Haykin, 1998). Radial basis functions are often constructed using a multiplication operation (although the multiplication need not be very exact). A network similar to the one presented here, has been used to implement such computations (van Rossum and Renart, 2004). The current study can be viewed from a broader perspective. The classical results by McCullough and Pitts demonstrated that a network of simple binary threshold units can implement any binary computation (McCullogh and Pitts, 1943). Similarly, using multi-layer perceptrons any continuous function can be approximated (Haykin, 1998). Indeed, in barn-owl auditory processing the accurate multiplicative responses arise from summation of more diverse responses (Fischer et al, 2007). In light of this, the fact that a multiplication can be done at all, should not be surprising. However, the question that remains is whether the proposed mechanism is biologically plausible and e cient. We believe it is.
Acknowledgements We thank Matthias Hennig and Alfonso Renart for discussion. MvR is supported by the HFSP and EPSRC.
References Andersen RA, Essick GK, Siegel RM (1985) Encoding of spatial location by posterior parietal neurons. Science 230(4724):456458 Anderson JS, Lampl I, Gillespie DC, Ferster D (2000) The contribution of noise to contrast invariance of orientation tuning in cat visual cortex. Science 290:19681972 Brotchie PR, Andersen RA, Snyder LH, Goodman SJ (1995) Head position signals used by parietal neurons to encode locations of visual stimuli. Nature 375(6528):232235 Chance FS, Abbott LF, Reyes AD (2002) Gain modulation from background synaptic input. Neuron 35(4):773782 Fischer BJ, Peña JL, Konishi M (2007) Emergence of multiplicative auditory responses in the midbrain of the barn owl. J Neurophysiol 98(3):11811193 Freeman RD (2004) Binocular interaction in the visual cortex. In: Chalupa L, Werner (eds) The Visual Neurosciences, MIT Press Gabbiani F, Krapp HG, Koch C, Laurent G (2002) Multiplicative computation in a visual neuron sensitive to looming. Nature 420(6913):320324, DOI 10.1038/nature01190, URL http://dx.doi.org/10.1038/nature01190
7
Gerstner W (2000) Population dynamics of spiking neurons: Fast transients, asynchronous state, and locking. Neural Comp 12:4389 Hansel D, van Vreeswijk C (2002) How noise contributes to contrast invariance of orientation tuning in cat visual cortex. J Neurosci 22:51185128 Haykin S (1998) Neural Networks: A Comprehensive Foundation. Prentice Hall Holt G, Koch C (1997) Shunting inhibition does not have a divisive eect on ring rates. Neural Comp 9:10011013 Hwang EJ, Donchin O, Smith MA, Shadmehr R (2003) A gain- eld encoding of limb position and velocity in the internal model of arm dynamics. PLoS Biol 1(2):E25 Knight BW (1972) Dynamics of encoding in a population of neurons. J gen Physiol 59:734766 Koch C (1999) Biophysics of computation. Oxford University Press, New York Koch C, Poggio T (1992) Multiplying with synapses and neurons. In: McKenna T, Davis J, Zornetzer SF (eds) Single neuron computation, Academic Press, San Diego, pp 315345 Larkum ME, Senn W, Lüscher HR (2004) Top-down dendritic input increases the gain of layer 5 pyramidal neurons. Cereb Cortex 14(10):10591070 McAdams CJ, Maunsell JH (2000) Attention to both space and feature modulates neuronal responses in macaque area v4. J Neurophysiol 83(3):17511755 McCullogh WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bulletin of mathematical biophysics 5:115113 Mehaey WH, Doiron B, Maler L, Turner RW (2005) Deterministic multiplicative gain control with active dendrites. J Neurosci 25(43):99689977 Mel BW (1992) Nmda-based pattern discrimination in a modeled cortical neuron. Neural Comput 4:502517 Mel BW (1993) Synaptic integration in an excitable dendritic tree. J Neurophysiol 70(3):10861101 Miller KD, Troyer TW (2002) Neural noise can explain expansive, power-law nonlinearities in neural response functions. J Neurophysiol 87:653659 Murphy BK, Miller KD (2003) Multiplicative gain changes are induced by excitation or inhibition alone. J Neurosci 23(31):10,04010,051 Peña JL, Konishi M (2001) Auditory spatial receptive elds created by multiplication. Science 292:249252 Reichardt W (1957) Autokorrelationsauswertung als Funktionsprinzip des Nervensystems. Z Naturforsch 12b:448457 Salinas E, Abbott LF (1996) A model of multiplicative neural responses in parietal cortex. Proc Natl Acad Sci U S A 93(21):11,95611,961 Sohn JW, Lee D (2007) Order-dependent modulation of directional signals in the supplementary and presupplementary motor areas. J Neurosci 27(50):13,65513,666 Srinivasan MV, Bernard GD (1976) A proposed mechanism for multiplication of neural signals. Biol Cybern 21(4):227236 Treue S, Trujillo JCM (1999) Feature-based attention in uences motion processing gain in macaque visual cortex. Nature 399(6736):575579 van Rossum MCW, Renart A (2004) Computation with populations codes in layered networks of integrate-and- re neurons. Neurocomputing 58:265270 van Rossum MCW, Turrigiano GG, Nelson SB (2002) Fast propagation of ring rates through layered networks of noisy neurons. J Neurosci 22:19561966 Vogels TP, Abbott LF (2005) Signal propagation and logic gating in networks of integrate-and- re neurons. J Neurosci 25:10,78610,786
8
Yu AJ, Giese MA, Poggio TA (2002) Biophysiologically plausible implementations of the maximum operation. Neural Comp 14:28572881
9
a
b
c
2
2
1
1
6 4 2 0
0
4
-2
x
0
0 2
y
x
-4
0
4
-2 0
0 2
4
-2
y
x
-4
0
0 2
y
-4
2
Fig. 1 Multiplication of two ring rates. Along one axis the ring rate is de ned as e−x with
x = (−3 . . . 3), the other as cos(y) + 1 with y = (−5 . . . 5).
a) Mathematical exact multiplication of these two functions. b) The minimum of the two inputs approximates the multiplication, although not very accurately. The minimum is equivalent to the two layer network with rectifying neurons, with a transfer function h(x) = [x]+ . c) The approximation strongly improves when the nodes have a supra-linear input-output relation, h(x) = ([x]+ )1.45 .
a
b fA
f
fA fB
B
Fig. 2 Two neural circuits for the implementation of approximate multiplication. The sharp
arrows depict excitatory inputs, the stump arrows depict inhibitory inputs. The squares correspond to populations of neurons which sum and rectify the net input. A) Symmetric circuit that can do an accurate multiplication. B) The smallest approximate multiplication circuit that we found, which was however less accurate.
a
b
c
50
Output
RMS error
25
Firing rate (Hz)
0.3
Input B
0 50
0.2
25
0.1
0 50
0
M
Input A
25 0
0
1000
2000
Time (ms)
3000
4000
0
1000
2000
Time (ms)
3000
4000
ea 25 n no 50 ise 75 cu rre 100 nt (p A)
200 150 100
50 0
v. of
e Std.d
A)
nt (p curre noise
Fig. 3 Implementation of the multiplication network using integrate-and- re neurons.
a) Spike responses in the input and output populations. The inputs have a staircase and a sawtooth pattern. b) Firing rates corresponding to panel a, averaged over 20 runs. The response in the output layer (dots, top graph) matches the true multiplication of the rates (solid line) (100ms bins). c) The error of the network as a function of the noise parameters, revealing a broad range of parameters for which the network approximates a multiplication. The error was the mean RMS error relative to the maximum response calculated using the input and output ring rates.
