Learning by Stimulation Avoidance Scales to Large Neural Networks

Oral presentation

Learning by Stimulation Avoidance Scales to Large Neural Networks Atsushi Masumori1 , Lana Sinapayen1 and Takashi Ikegami1 1

The University of Tokyo, Japan [email protected]

Abstract Spiking neural networks with spike-timing dependent plasticity (STDP) can learn to avoid the external stimulations spontaneously. This principle is called ”Learning by Stimulation Avoidance” (LSA) and can be used to reproduce learning experiments on cultured biological neural networks. LSA has promising potential, but its application and limitations have not be studied extensively. This paper focuses on the scalability of LSA for large networks and shows that LSA works well in small networks (100 neurons) and can be scaled to networks up to approximately 3,000 neurons.

Introduction Learning is an important aspect of the neural system, and is crucial for animals as embodied neural systems to learn adaptive behavior to survive. One of the key concepts in studying adaptive behavior is homeostasis. For example, Ashby argued that adaptive behavior is an outcome of the homeostatic property of living systems (Ashby (1960)), and Di Paolo and Iizuka reported that adaptive behavior is an indispensable outcome of homeostatic neural dynamics (Iizuka and Di Paolo (2007); Di Paolo and Iizuka (2008)). However, those models are still too abstract to compare with the homeostatic property of biological neural dynamics. Biological neuronal cells in vivo are often implemented and tested as a model in artificial neural networks to understand learning mechanisms (Sejnowski et al. (1988)). Recently, the dynamics of artificial neural networks have become much closer to those of biological networks, by introducing biologically plausible models of spiking neurons (Izhikevich (2004); Brette et al. (2007)) and the synaptic plasticity ( Song et al. (2000); Dan and Poo (2006); Caporale and Dan (2008)). These more realistic models can lead to theoretical understanding of biological neural networks. Biological neuronal cells cultured in vitro have been used to study neural systems (Warwick (2010)) because such cultured networks are easier to study than biological neural networks in vivo, which are composed of relatively smaller numbers of neurons, and cultured in a more stable environment. Using real biological neural networks is advantageous because potential complexity features in neuronal cells can

be used, which is still difficult to implement in artificial neural networks. Although cultured neural systems are much simpler than real brain systems, they have essential properties, including spontaneous activity, various types and distribution of cells, high connectivity, and rich and complex controllability (Canepari et al. (1997)). Homeostatic control may be one such property. There are many studies on learning dynamics using cultured neurons (Potter et al. (2006); DeMarse and Dockendorf (2005); Kudoh et al. (2008)). In particular, Shahaf and Marom used cultured neuronal cells to study the learning principle for cultured neural networks is very insightful (Shahaf and Marom (2001); Marom and Shahaf (2002)). They demonstrate that a cultured neural network can learn a desired behavior by following protocols. First, an electrical stimulation with a fixed low frequency (e.g., 1-2Hz) was sent to part of the network. When the desired behavior appears, the stimulation is removed. Repeating these protocols, a network learns to produce the expected behavior in response to the stimulation. In practice, the authors showed that the network learned to produce spikes at a predefined output sections which is different from an input location in the network and in a predefined time window (within 40-60 ms after each stimuli), in response to a stimulation applied at the input location. The network form the sensory-motor coupling to avoid the external stimulation. Therefore, it seems the networks have some sort of homeostatic properties to learn adaptive behaviors for their environment. Marom explained these results by proposing the stimulus regulation principle (SRP) which is composed of the following two functions at neural network level. (1) Modifiability, in which stimulation drives the network to try to form the different topologies by modifying neuronal connections. (2) Stability, in which removing the stimulus stabilizes the network in its last configuration. Preliminary experiments suggested that cultured neural networks have these two functions at the network level. Although the findings of Shahaf and Marom are promising, it seems that the applications, limitations, and explanatory mechanisms of the method were not further studied

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

275

Oral presentation

in cultured or artificial neural networks until recently. In a previous study, we proposed a possible mechanism at the micro scale of neural behavior similar to Shahaf and Marom’s results, termed the principle of learning by stimulation avoidance (LSA; Sinapayen et al. (2015); Sinapayen et al. (2017)), using small networks. We claim that LSA is an emergent property of spiking networks coupled to Hebbian rules (Hebb (1949)) and external stimulation. LSA states that the network learns to avoid external stimulus by learning available behaviors. These behaviors are emerged from spike-timing dependent plasticity (STDP), which has been found in both in vivo and in vitro networks. STDP causes changes in the synaptic weight between two firing neurons depending on the timing of their activity. If the presynaptic neuron fires directly before the postsynaptic neuron, the synaptic weight increases, and if the presynaptic neuron fires directly after the postsynaptic neuron, the synaptic weight decreases. LSA can be lead to explain the mechanism of homeostatic adaptation using a realistic model: when external input stimulation patterns change, the network learns new behavior to avoid the new external stimulation, similar to a homeostatic adaptation. We believe that LSA has a promising potential, but its applications and limitations have not been studied sufficiently. This paper focuses on the scalability of LSA in simulated spiking neural networks. In previous studies using cultured neuronal cells (Masumori et al. (2015); Masumori et al. (2016)) on CMOS-electrode arrays (Bakkum et al. (2008)), we shown that even cultured neural networks with a small number of neurons (about 100 neurons) can learn a desired behavior using a simple protocol explained above, although Shahaf and Marom used a much larger network (10,000neurons). This suggests that such a homeostatic properties exists in the cultured neural networks, that the networks can form sensory-motor coupling to avoid external stimulation, and that the learning behaviors are scalable, at least biological neural scales. In this paper, using simulated spiking neural networks, we first show LSA works well in a small networks (100 neurons). Second we show this learning behavior can be scaled to larger networks (-3,000 neurons).

Method As explained above, the learning experiments in this paper are based on LSA which is an emergent property of spiking networks coupled to STDP and external stimulation. Therefore, for the experiments in this paper, we mainly use spiking neural networks and STDP for simulated neural networks as described below.

Network model Spiking neuron The model for spiking neuron proposed by Izhikevich (2004) was used to simulate excitatory neu-

rons and inhibitory neurons. This model is well know as it can be regulated to reproduce the dynamics of many variations of cortical neurons, and it is computationally efficient. The equations of the neural model are defined as: dv = 0.04v 2 + 5v + 140 − u + I, dt du = a(bv − u), dt ( v←c if v ≥ 30mV, then u←u+d

(1)

Here, v represents the membrane potential of the neuron, u represents a variable related to the repolarization of membrane, I represents input current from outside of the neuron as explained in detail below section, t is time, and a, b, c, and d are other parameters (Izhikevich (2003)). The neuron is regarded as firing when the membrane potential v ≥ 30mV. The parameters for excitatory neurons (regular-spiking neuron) are set as: a = 0.02, b = 0.2, c = −65mV, and d = 8, and for inhibitory neurons (fast-spiking neuron) are set as: a = 0.1, b = 0.2, c = −65mV, and, d = 2. The simulation time step ∆t is comparable to 1 ms. The resulting dynamics of these two neural types are shown in Fig 1.

Figure 1: Dynamics of regular-spiking and fast-spiking neurons simulated using the Izhikevich model. Regular-spiking neurons are used as excitatory neurons and fast-spiking neurons are used as inhibitory neurons. Network structure Randomly connected networks of 100-3000 neurons with 80% excitatory and 20% inhibitory neurons were simulated. This ratio of 20% inhibitory neurons is standard in simulations (Izhikevich (2003), Izhikevich (2004)) and similar to biological values (Cassenaer and Laurent (2007)). The excitatory neurons were divided into four groups: Input, Output A, Output B, and Hidden (Fig 2). Input consisted of 20% neurons and external stimulation is injected only to this group. Output A and B consisted of 10% neurons each and the desired behavior that the network learn was composed of these groups. Hidden consisted of 40% neurons. The connection rate between each neuron depended on the experiments. There were no connections between Input and Output group, and self-connections (e.g.,

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

276

Oral presentation

from neuron i to neuron i) were forbidden. The weight values w between each neuron were randomly initialized with uniform distributions as 0 < w < 5 for excitatory neurons, −5 < w < 0 for inhibitory neurons.

(100Hz). When more than 40% of Output A fired and less than 40% of Output B fired, the external stimulation to the Input is stopped in 1,000-2,000 ms (it is randomly chosen). Condition 2: When neurons in Output B fire more than neurons in Output A, Input is stimulated once at 30 mV (with a maximum frequency of 100Hz ). These conditions imply that the goal for the network is to avoid external stimulation by increasing the firing rate of Output A compared to Output B. (3) Synaptic current from other neurons: when a neuron a spiked, the value of the weight wa,b was added as an input to neuron b without delay. All these input were added for each neuron ni at each time step as: Ii = Ii∗ + ei + mi n X Ii∗ = fj (wj,i × sj ) j=0

Figure 2: Network diagram of the initial network. Network is composed of 80% excitatory and 20% inhibitory neurons. Input group consisted of 20% neurons, both Output A and Output B consisted of 10% neurons each and Hidden consisted of 40% neurons. There was no connection between Input and Output. The blue line represents connection with a plasticity and the gray dot line represents no plasticity.

Here, s represents short-term plasticity variables defined in (3). A phenomenological model of short-term plasticity (STP, Mongillo et al. (2008)) was used, and s varies for each neuron ni as: si = ui xi 1 − xi dx = − ui xi fi dt τd du U − ui = + U (1 − ui ) fi dt τf ( 1, if neuron i is firing fi = 0, otherwise.

Figure 3: STP stabilizes global burst synchronization and the global bursting level was stabilized depending not on network size with STP. Statistical results for N=10 samples with standard error. Input current to neuron There were three types of input to the neurons. (1) Zero-mean Gaussian noise m with a standard deviation σ = 3 mV was injected into each neuron at each time step. This is required for spontaneous firing of the neurons. (2) External stimulation e was injected into Input. There were two conditions for the stimulation. Condition 1: Input was stimulated with 10 mV and a fixed frequency

(2)

( 1, if neuron j is firing fj = 0, otherwise.

(3)

Here, x represents the amount of available resources, and u represents the resource used by each spike (Mongillo et al. (2008)). The parameters were set to τd = 200ms, τf = 600ms, and U = 0.2mV. STP is not necessarily required for LSA, but it is efficient to suppress global burst synchronization (Sinapayen et al. (2017)) and stabilizes the firing rate independent from the network size. Although LSA can be achieved only by a parameter tuning without STP, it can be easily achieved with STP. We thus used the STP model in this work. Fig 3, 4 show the results of preliminary experiments for the STP function. As adding an STP function, the burstiness index (an indicator for measuring global burst levels (Wagenaar et al. (2005))) and firing rate was relatively stabilized independent from the network size. Without STP, these variables significantly change with the network size. Synaptic plasticity STDP was used as a model for synaptic plasticity between the spiking neurons. STDP causes

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

277

Oral presentation

Figure 4: STP stabilizes the firing rate and the firing rate was stabilized depending not on network size with STP. Statistical results for N=10 samples with standard error. changes in the synaptic weight between two neurons depending on the timing of their spiking; when the presynaptic neuron fires right before the postsynaptic neuron, the synaptic weight increases, and when the presynaptic neuron fires right after the postsynaptic neuron, the synaptic weight decreases. The weight variation ∆w is defined as: ( A(1 − τ1 )∆t , if ∆t > 0 ∆w = 1 −∆t , if ∆t < 0 −A(1 − τ )

(4)

Here, ∆t represents relative spike timing between presynaptic neuron a and post-synaptic neuron b: ∆t = tb −ta (ta represents spike timing of neuron a, and tb represents a spike timing of neuron b). Other parameters are set to A = 0.1, and τ = 20 ms. Fig 5 shows the variation of ∆w depending on ∆t; ∆w becomes negative when the postsynaptic neuron fires first, and is positive when the presynaptic neuron fires first. In this paper, STDP is applied only between excitatory neurons; thus the weight of other connections does not change from the initial value during all experiments. The weight value between excitatory neurons w varies as:

wt = wt−1 + ∆w .

Results Using above settings, we conducted learning experiment in small and larger networks for examining scalability of LSA. First, the results of the small networks with 100 neurons are shown before the results of larger size networks.

Learning experiments in small networks In a previous study, we found that the learning rate of LSA changes depending on the connection size between neurons on more simple experiments (Sinapayen et al. (2017)). Therefore we first show learnability varies with connections size in the experimental setup. Fig 6 shows the relationship between learnability and the connection size.

(5)

The maximum possible weight is fixed to wmax = 20, and if w > wmax , w is reset to wmax . The minimum possible value of weight is fixed to wmin = 0, and if w < wmin , w is reset to wmin . In addition to STDP, a weight decay function was also applied to the weight between excitatory neurons. The decay function is defined as:

wt+1 = (1 − µ)wt

Figure 5: STDP function, the weight variation ∆w of the synapse from neuron a to neuron b depends on the relative spike timing (A = 0.1, τ = 20 ms).

(6)

In this paper, the parameter µ was fixed as µ = 5 × 10−7 .

Figure 6: Learnability with various connection sizes; when the connection size is 40-90 per neuron, learnability is significantly larger than the control experiment (random), and learnability is maximized at 50 connection per neuron. Statistical results for N = 20 networks with standard error. Here, learnability L is defined as L = (Fa − Fb )/N ; Fa represents the number of times Output A fired in last 20,000 ms, Fb represents the number of times Output B fired and N represents the number of Output A and B. When Output A is

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

278

Oral presentation

fired more than Output B, learnability becomes positive, and in the opposite case, learnability becomes negative. This is because the firing rate of Output A should be increased and that of Output B should be decreased to achieve LSA for external stimulation condition 1 and 2, explained above, simultaneously. As shown in Fig 6, when the connections per neuron is 40-90, learnability is significantly larger than the control experiments (random), and learnability was maximized at 50 connections. In the control experiment, stimulations of condition 1 and 2 were randomly injected into Input and other experimental settings were same as explained above. The figures below show more details of the results of the experiments with 50 connections.

and stabilized at higher value, on the other hand, the learnability did not increased in the control experiments . Fig 8 shows raster plot of one of the experiments with 50 connections. This actual spiking pattern shows qualitatively that the firing rate of Output A became larger than that of Output B, which implies that the network has high learnability. Fig 9 shows typical examples of network diagram for the learned network of experiments with 50 connections per neuron. Connections from Input to Hidden, and connections from Hidden to Output A were larger than initial values thus information from Input can be transmitted to Output A via Hidden. On the other hand, there were almost no connections from other groups to Output B. The more abstract network diagram in Fig 10 was drawn based on the network diagram in Fig 9. This network topology is a optimal for avoiding the external stimulations. Therefore, it suggest that through LSA, the networks autonomously yielded sensorymotor coupling to avoid external stimulation in the environments.

Figure 7: Time series of learnability in a small network (100 neurons, 50 connections). The learnability gradually increased and stabilized at significantly higher values than the control experiment (random). Statistical results for N = 20 networks with standard error.

Figure 9: Network diagram of a small learned network (100 neurons, 50 connections). The width of the edge represents the weight value, and color of the edge represents the direction of the edge, and it is directed to node with same color (e.g., green edge represents the edges to Output A). There is almost no edge from Hidden to Output B but many strong edges from Hidden to Output A.

Figure 8: Raster plot for the small network (100 neurons and 50 connections). The firing rate of Output A became larger than that of Output B. Fig 7 shows the time series of learnability. The learnability of LSA network gradually increased in 15,000-40,000ms

Learning experiments in larger networks Learning experiments with larger networks were conducted using the same experimental settings as the experiments with 100 neuron except for the network size. Fig 11 shows the relationship between learnability and a network size; the larger network is, the smaller the learnability.

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

279

Oral presentation

Figure 10: Network diagram of a small, learned intermodule network (100 neurons and 50 connections). There is a route from Input to Output A via Hidden, although, there is no route from Input to Output B. The blue line represents connection with plasticity and the gray dot line represents a connection with no plasticity. Fig 12 shows time series for the learnabilities; with this parameters (e.g., 50 connections), only in networks of 1,000 neurons, the learnability abruptly increased and stabilized at higher values. Longer time were required to learn it compared to smaller networks (100 neurons). However, even in larger networks than 1,000 neurons, using other parameters can increase learnability. For example, in networks of 3,000 neurons with 300 connections (other settings are same with above), learnability gradually increased, although the deviation was larger than in smaller networks (Fig 13). These results suggest that when the network size is larger, learning becomes more difficult compared to smaller networks because it takes more time to learn and the deviation is larger than smaller networks. In addition, we notice that the parameter region for high learnability differs from smaller networks.

Discussion The learning experiments were conducted based on LSA, and revealed that LSA can be scaled to larger networks (3,000 neurons) compared to previous works (-100 neurons). However, the parameter region where LSA works depends on network size and learning difficulties increase with network size. In the experimental setup proposed in this paper, there are no direct connections between Input and Output so that if the network size becomes larger with same number of connections, the average path length between Input and Output would becomes larger. It is assumed that this larger

Figure 11: The relationship between learnability and network size (50 connections). With this parameters (e.g., 50 connections), only networks with 1,000 neurons have high learnability. Statistical results for N = 6 samples with standard error. path length make a learning in larger network more difficult so that larger network needs larger connection size, getting the average path length smaller, for high learnability. Although there is no synaptic delay between each neuron in the model here, there is such a time delay in biological neural networks, indicating a limitation in network size for LSA because timing of injecting and removing external stimulation is crucial for LSA. If this hypothesis is true, although previous studies shows that cultured neuronal cells in vitro with 100-100,000 cells can learn by a similar protocol (Shahaf and Marom (2001), Masumori et al. (2016)), LSA can work only locally in huge networks in vivo. However, some animals seems to avoid the external stimuli (Hull (1943)). The relationship between LSA and more macrolevel adaptive behaviors should be studied. Adaptive behavior is and has been a major theme for artificial life and should be explored further. Previously, Di Paolo (2000), Iizuka and Di Paolo (2007), and Di Paolo and Iizuka (2008) studied the emergence of adaptive behavior as an outcome of neural homeostasis. However, these models are too abstract to explain homeostatic property in biological neural networks. In previous studies, we found that spiking neural networks with STDP has some sort of homeostatic properties that the networks can learn a behavior to avoid external stimulation from their environment (e.g., wall avoidance behavior). It suggest that the LSA can work in biological neural network in vivo, and LSA can be lead to explain homeostatic adaptation in a more realistic model. We need more study to show it as future work; e.g., learning ex-

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

280

Oral presentation

Figure 12: Time series for learnabilities in large networks (50 connections). Only in networks with 1,000 neurons, learnability gradually increased and stabilized at a high values. Statistical results for N = 6 samples with standard error. periment in which stimulation conditions change during the experiment. LSA can be considered to be similar to the Principle of Free Energy Minimization introduced by Friston (2010), which states that networks strive to avoid surprising input by learning to predict external stimulation. An expected behavior of networks obeying the Free Energy Principle, or LSA is that they fall into the dark room paradox, avoiding incoming input by cutting all sources of external simulation. There is no explicit prediction for incoming input in LSA unlike Free Energy Principle; however, recent studies suggest that spontaneous activities in biological neural networks can be regarded as prediction for the environment (e.g., external stimulation patterns) (Berkes et al. (2011)), thus a LSA can be related to macro-level active inference (Friston et al. (2016))) using such a framework. To show such a various spontaneous activities required for the framework, spiking neural network needs synaptic time delay (Izhikevich (2006)), which must be added to this model in future works. Although the previous and present studies tested LSA for simple tasks, LSA experiments with more difficult task have been conducted using an humanoid robot. As the results, it learned some more difficult tasks (see also Doi et al. (2017)). However, the limitations of learnability of LSA for more difficult tasks is still unknown, thus it must be explored in future works.

Acknowledgements This research was supported by Grant-in-Aid for JSPS Research Fellow (no.16J09357 to A. Masumori) andused computational resources of the K computer provided by the RIKEN Advanced Institute for Computational Science through the HPCI System Research project (Project

Figure 13: Time series for learnabilities in a large network (3,000 neurons with 300 connections). The learnability gradually increased, although the deviation was larger than in a smaller networks. Statistical results for N = 6 samples with standard error. ID:hp160264)

References Ashby, W. R. (1960). Design for a Brain: The Origin of Adaptive Behavior. Chapman and Hall, 2 edition. Bakkum, D. J., Chao, Z. C., and Potter, S. M. (2008). Spatiotemporal electrical stimuli shape behavior of an embodied cortical network in a goal-directed learning task. Journal of neural engineering, 5(3):310–23. Berkes, P., Orb´an, G., Lengyel, M., and Fiser, J. (2011). Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment. Science, 331(6013):83–7. Brette, R., Rudolph, M., Carnevale, T., Hines, M., Beeman, D., Bower, J. M., Diesmann, M., Morrison, A., Goodman, P. H., Harris, F. C., Zirpe, M., Natschl¨ager, T., Pecevski, D., Ermentrout, B., Djurfeldt, M., Lansner, A., Rochel, O., Vieville, T., Muller, E., Davison, A. P., El Boustani, S., and Destexhe, A. (2007). Simulation of networks of spiking neurons: a review of tools and strategies. Journal of computational neuroscience, 23(3):349–98. Canepari, M., Bove, M., Maeda, E., Cappello, M., and Kawana, A. (1997). Experimental analysis of neuronal dynamics in cultured cortical networks and transitions between different patterns of activity. Biological Cybernetics, 77(2):153–162. Caporale, N. and Dan, Y. (2008). Spike timing-dependent plasticity: a Hebbian learning rule. Annual review of neuroscience, 31:25–46. Cassenaer, S. and Laurent, G. (2007). Hebbian STDP in mushroom bodies facilitates the synchronous flow of olfactory information in locusts. Nature, 448(7154):709–13. Dan, Y. and Poo, M.-M. (2006). Spike timing-dependent plasticity: from synapse to perception. Physiological reviews, 86(3):1033–48.

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

281

Oral presentation

DeMarse, T. and Dockendorf, K. (2005). Adaptive flight control with living neuronal networks on microelectrode arrays. Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., 3:1548–1551. Di Paolo, E. A. (2000). Homeostatic adaptation to inversion of the visual field and other sensorimotor disruptions. From Animals to Animats VI: Proceedings of the 6th International Conference on Simulation of Adaptive Behavior, pages 440–449.

Mongillo, G., Barak, O., and Tsodyks, M. (2008). Synaptic theory of working memory. Science (New York, N.Y.), 319(5869):1543–1546. Potter, S. M., Wagenaar, D. A., and DeMarse, T. B. (2006). Closing the Loop: Stimulation Feedback Systems for Embodied MEA Cultures. In Advances in Network Electrophysiology, pages 215–242. Springer US.

Di Paolo, E. A. and Iizuka, H. (2008). How (not) to model autonomous behaviour. BioSystems, 91:409–423.

Sejnowski, T. J., Koch, C., and Churchland, P. S. (1988). Computational Neuroscience. Science, New Series, 241(4871):1299– 1306.

Doi, I., Ikegami, T., Masumori, A., Kojima, H., Ogawa, K., and Ishiguro, H. (2017). A New Design Principle for an Autonomous Robot. In The 14th European Conference on Artificial Life (ECAL 2017), (in press).

Shahaf, G. and Marom, S. (2001). Learning in networks of cortical neurons. The Journal of neuroscience : the official journal of the Society for Neuroscience, 21(22):8782–8.

Friston, K. (2010). The free-energy principle: a unified brain theory? Nature reviews. Neuroscience, 11(2):127–38.

Sinapayen, L., Masumori, A., and Ikegami, T. (2017). Learning by stimulation avoidance: A principle to control spiking neural networks dynamics. PLOS ONE, 12(2):e0170388.

Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O’Doherty, J., and Pezzulo, G. (2016). Active inference and learning. Neuroscience and Biobehavioral Reviews, 68:862– 879. Hebb, D. (1949). The Organization of Behavior: A Neuropsychological Theory. Wiley, New York. Hull, C. L. (1943). Principles of behavior: an introduction to behavior theory. Appleton-Century Company, Oxford, England. Iizuka, H. and Di Paolo, E. a. (2007). Toward Spinozist Robotics: Exploring the Minimal Dynamics of Behavioral Preference. Adaptive Behavior, 15(4):359–376. Izhikevich, E. (2003). Simple model of spiking neurons. IEEE Transactions on Neural Networks, 14(6):1569–1572. Izhikevich, E. M. (2004). Which model to use for cortical spiking neurons? IEEE Transactions on Neural Networks, 15(5):1063–1070.

Sinapayen, L., Masumori, A., Virgo, N., and Ikegami, T. (2015). Learning by Stimulation Avoidance as a Primary Principle of Spiking Neural Networks Dynamics. In The 13th European Conference on Artificial Life (ECAL 2015), (in press). Song, S., Miller, K. D., and Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature neuroscience, 3(9):919–26. Wagenaar, D. a., Madhavan, R., Pine, J., and Potter, S. M. (2005). Controlling bursting in cortical cultures with closedloop multi-electrode stimulation. The Journal of neuroscience : the official journal of the Society for Neuroscience, 25(3):680–8. Warwick, K. (2010). Implications and consequences of robots with biological brains. Ethics and Information Technology, 12(3):223–234.

Izhikevich, E. M. (2006). Polychronization: computation with spikes. Neural computation, 18(2):245–82. Kudoh, S. N., Kiyohara, A., Hosokawa, C., Taguchi, T., and Hayashi, I. (2008). Vitroid - a robot with link between living neuronal network in vitro and robot body. 2008 IEEE International Conference on Mechatronics and Automation, pages 375–378. Marom, S. and Shahaf, G. (2002). Development, learning and memory in large random networks of cortical neurons: lessons beyond anatomy. Quarterly reviews of biophysics, 35:63–87. Masumori, A., Maruyama, N., Sinapayen, L., Mita, T., Frey, U., Bakkum, D., Takahashi, H., and Ikegami, T. (2015). Emergence of Sense-Making Behavior by the Stimulus Avoidance Principle: Experiments on a Robot Behavior Controlled by Cultured Neuronal Cells. Proc. of the European Conference on Artificial Life (ECAL) 2015, pages 373–380. Masumori, A., Maruyama, N., Sinapayen, L., Mita, T., Frey, U., Bakkum, D., Takahashi, H., and Ikegami, T. (2016). Learning by Stimulation Avoidance Principle on Cultured Neuronal Cells. In Proc. of annual meeting of Physical Society of Japan 2016, pages 71(1), 3092.

Carole Knibbe et al, eds., Proceedings of the ECAL 2017, Lyon, France, 4-8 September 2017, (Cambridge, MA: The MIT Press, ©2017 Massachusetts Institute of Technology). This work is licensed to the public under a Creative Commons Attribution - NonCommercial - NoDerivatives 4.0 license (international): http://creativecommons.org/licenses/by-nc-nd/4.0/

282