FPGA Implementation of Spiking Neural Networks Supported by a

Preprints of the 18th IFAC World Congress Milano (Italy) August 28 - September 2, 2011

FPGA implementation of Spiking Neural Networks supported by a Software Design Environment A. Rosado-Muñoz*, A.B. Fijałkowski**, M. Bataller-Mompeán*, J. Guerrero-Martínez* 

*Dpt. Electronic Engineering. School of Engineering. University of Valencia. Dr. Moliner, 50. 46100 Burjassot. Valencia. SPAIN (e-mail: [email protected]). **University of Zielona Góra. Ul. Podgorna, 9. Zielona Góra. POLAND

Abstract: This paper is focused on the creation of Spiking Neural Networks (SNN) in hardware due to their advantages for certain problem solving and their similarity to biological neural system. One of the main uses of this neural structure is pattern classification. The chosen model for the spiking neuron is the Spike Response Model (SRM). For SNN design and implementation, a software application has been developed to provide easy creation, simulation and automatic generation of the hardware model. VHDL was used for the hardware model. This paper describes the functionality of SNN and the design procedure followed to obtain a working neural system in both software and hardware. Designed VHDL code is fully synthesizable; it has been tested with Xilinx ISE implementation software. Final results show some applications showing the ability of SNN for pattern recognition as well as their performance and occupation in FPGA. Keywords: neural network models, system synthesis. 

Hardware implementation for SNN is an interesting topic because real time operation can be achieved and standalone hardware systems can be created for on-line tasks. Some of the image recognition systems have been implemented in VLSI chips (Vogelstein et al., 2005). Since the Spiking Neural Networks are an implementation of real biological neural system, they can be applied to almost every field, for instance, its wide usage can be found in robotics for machine control movement as in (Floreano et al., 2001; Hagras et al., 2009). Other applications including navigation tasks or obstacle avoidance by robots (Roggen et al., 2004) have also been developed.

1. INTRODUCTION The use of Spiking Neural Networks (SNN) is growing due to their ability to solve different problems in the area of pattern classification, system prediction, machine control, image processing, etc. The term Artificial Neural Networks has two meanings, it can refer to the Artificial Neural Networks with sigmoidal neuron (Tommiska, 2003) which are not covered in this paper, or it can refer to an artificial implementation of different neural structures such as SNN. Spiking Neural Networks are inspired in the biological neural system, trying to obtain a computational system able to replicate the behavior of biological neurons for both information transmission among neurons and internal neuron signal processing. Nowadays, the behavior of the biological neural system is well known (Arbib et al., 2002) and thus, a modeling can be done.

Apart from adequate SNN algorithm modeling, hardware implementation requires specific design. A good background to SNN in FPGA structures is given in (Bellis et al., 2004). Some hardware description language aspects are covered in (Bailey et al., 2008; Gómez Casado, 2008) where a detailed hardware implementation is described. In this paper, Spike Response Model (SRM) is used (Gerstner et al., 2002).

Several applications have been developed using SNN, some authors use them for image recognition and operation (Meftah et al., 2008; Perrinet, 2009) or conjunction detection (Bothe et al., 2002) where an application solving so-called binding-problem is presented. In (Meftah et al., 2008), an interesting interpretation of RGB values of every image pixel by the neural structure is described. (Perrinet, 2009) presents retinal coding and image processing with the most information in the shortest time. The strong side of SNN is filtering, which can be applied to image processing with different receptive fields models (Wu et al., 2008). A very fast and adaptive image processing with multi-view pattern recognition can be achieved (Wysoski et al., 2008), two cameras were used as an input to obtain a three-dimension picture like a human.

Copyright by the International Federation of Automatic Control (IFAC)

This work was motivated by the need of developing a working spiking neural structure in hardware and also by providing a connection between it and the software application where a neural network can be easily designed, simulated and migrated to a hardware system by generating custom VHDL code according to the design specifications. This paper is not focused on the learning algorithm for weight update, typically, Spike-Timing Dependent Plasticity (STDP) algorithm (Song et al., 2002) is used as a learning rule for the network training process.

1934

Preprints of the 18th IFAC World Congress Milano (Italy) August 28 - September 2, 2011

The proposed syystem is testeed for a pattern classificatiion task similar to ((Gupta et al.,, 2007) but using u a differeent neu ural model. Some simiilar applicatiion of patteern reccognition is allso described in (Booji, 20 004). This papper desscribes the foollowed proceedure to desiign an SNN in harrdware, includding the desiign of the SR RM for neuroons and d the system iinterconnectioon among neurrons. This papper inccludes the testt for the desiggned structuree with a patteern claassification tassk.

n two neuronss for off the synapse, the travellingg time between a spike is obtain ned accordingg to (3).

t ji  t  t i( f )  d ji

The paper is struuctured as folllows: section 2 introduces tthe neu ural model, ccode and thee STDP rulee as a learniing alg gorithm used in training prrocess. Amon ngst all differeent mo odeling system ms, only thoose used in this work aare desscribed. Section 3 gives a quick ov verview on tthe dev veloped softw ware application and its ro ole in designiing thee hardware m model. This section also introduces tthe harrdware model. Section 4 coovers results of o simulationss in botth software and hardwarre models. Last L section ((5) con ncludes the paaper and poinnts some guideelines for futuure wo ork.

Fiig. 1. Postsyn naptic potentiaal function – PSP with weight deependency.

2. SPIKING NEURONS AN ND INTERCO ONNECTION S

In n order to obtaain the model for the entire neuron behavvior, when w a sequen nce of spikess Fi={ti(g),..., tiK} arrives to t a neeuron j, the membrane m poteential changess according to the PS SP function and a refractoryy period, and, if fired, a sppike traain is propagaated by neuronn j as Fj={tj(f),..., tiN}. Thus, the co omposed equaation for j neuuron potentiall uj as a functtion off time is obtain ned by (4):

2.1 1. Spike Respoonse Model - SRM S Sev veral models exist to obtaain the internaal behavior off a neu uron (Gerstneer et al., 20005). In this paper, the Spiike Response Modell (SRM) is chhosen due to itts similarity too a bio ological neurron as recennt research studies supp ort (Paaugam-Moisy et al., 20009; Booji, 20 004) The maain chaaracteristic oof a spiking neuron is the membraane pottential, whichh normally stays s at a reesting value of -75 5mV, (for sim mplicity, the model propossed here usess a vallue of 0mV for resting potential). If I two neuroons com mmunicate wiith spikes we refer to a tran nsmitting neurron as a presynaptiic neuron whhile a receiv ving one is tthe posstsynaptic neeuron. When a new spik ke arrives too a posstsynaptic neuuron it generattes a change on o its membraane pottential called P Post Synapticc Potential (PS SP). This channge can n increase (Exxhibitory PSP)) or decrease (Inhibitory ( PS SP) neu uron membraane potential. A general PSP P function is deffined by (1), where a weigght factor forr calculating tthe PSP value is appplied to every connection (F Fig. 1).  t    t    f PSP (tt)  w   e 4   e 2    

K

u j (t )   i

 t    20 



PSP(t  t i( gg )  d ji ) 

ti( g ) Fi

 (t  t

(f) j

) (4)

t (j f )F j

Eq quation (4) describes d the membrane potential p of one po ostsynaptic neeuron, where the influence of every neuuron in nput is added up and also rrefractory perriod is taken into i co onsideration. 2.2. Neural Cod ding When W a stimu ulus is presennted to the network n inputt, it triiggers a neu ural response,, i.e. neuron ns within neuural strructure fires a spike (denotted in the graaphs by a verttical baar) or a series of spikes (sppike trains) connected with this stiimulator; if the stimulus is changed, the reactionn of neeurons also ch hange. If this response is reecorded in a time t window, w the spike raster plott in Fig. 2 is obtained.

((1)

A postsynaptic p nneuron increasses its membrrane potential up to a threshold vvalue  ; thenn, the neuron fires an outpput spiike and goess into a shorrt refractory period. Duriing reffractory periood, the neuronn cannot reaact on any neew inccoming spike. The refractorry period can be described by (2) (Booji, 2004)):

 (t )    e

(3)

Th here exist different cooding inform mation methhods (G Goldberg et al., 2007; Gerrstner et al., 2002), 2 The innput neeuron fires a spike regularlyy every time in nterval accordding to o the input stim mulus and som me temporal coding algoritthm (P Paugam-Moisy y et al., 2009)) or Gaussian Receptive Fieelds (G GRF) coding (Beńuskova, 2009; Fijałko owski, 2009). In th his case, the proposed m method is a modificationn of temporal codin ng where the neurons at the t input havve a diifferent behavior than the re rest of the neu ural network. The in nput neuron does d not calcuulate any mem mbrane potential, asssuming the in nput to receivve a digital value, v the neuuron geenerates a spiike every 50 milliseconds (tested to be the most m adequate) while the inpput is active (v value ‘1’).

(22)

ti(f)

Being the tim me when a sppike is fired by b a presynapptic neu uron (indexedd by i), this sppike is changiing the potenttial of a postsynaptiic neuron j att time t. The time differennce bettween these tw wo events is t-t t i(f), and, inclluding the dellay

1935

Preprints of the 18th IFAC World Congress Milano (Italy) August 28 - September 2, 2011

hange is negative, being poositive in the opposite casee. If ch th he time differeence is equal too 0, there is no weight channge.

Fig g. 2. Spike R Raster Plot. Activity A of spiking s neuroons witthin 700ms. V Vertical bars reepresent a sing gle spike. In a simple patttern recognitiion example, we have blaack and d white patterrns received in i a 3 pixel image i and thuus, three input neurrons. While a pixel is blaack (active), tthe neu uron correspoonding to this pixel fires a spike regular arly witthin every tim me period, if pixel is whitee, the neuronn is nott active (Fig. 33).

Fiig. 4. STDP weight w change function 3. DEVELOPPED MODEL LS 3.1. Software Model M and its ddesign environ nment. In n order to prov vide a softwaare system forr easy design and sim mulation of Spiking S Neurral Networks,, a general SNN S model m has beeen designed. The softwaree application has beeen developed d in C# languaage allowing designing a SNN S strructure with user-friendly interface (Fig. 5), simulatting th he SNN for some appliications and converting the deesigned softw ware structure re into synth hesizable VH HDL co ode. As the SNN structuure requires a high paraallel co omputation, a multithreadding program mming proviides sim multaneous work w for everyy network elem ment.

Fig g. 3. Examplee of temporal coding for a black b and whhite three pixel image. 2.3 3. Spike-Timinng dependent plasticity p - STD TDP As mentioned aabove, all synaapses includee a weight vallue to be applied to the PSP function, reepresenting tthe con ntribution of a presynapticc neuron to th he activation of thee postsynapticc neuron. Thhese weight values v must be selected accordding to a learning alg gorithm. Maany alg gorithms havee been propoosed like spikkeProp learniing (Scchrauwen et aal., 2006) or supervised ReSuMe R methhod (Po onulak, 2006)), in this worrk, Spike-Tim ming Dependeent Plaasticity (STDP P) (Song et al., 2002) is used. The general ideea of STDP consists c on ch hanging weighhts acccording to the time differrence between n occurrence of posstsynaptic annd presynaptic spikes, i..e. T=tpost-ttppre. Deepending on T, the function ij(T) (Song et al., 20002; Beń ńuskova, 20009) modifies the weight value for tthe syn naptic connecttion ij given by b equations (5 5) and (6).

Fiig. 5. Screensshot from thee software sy ystem showinng a SN NN structure with 9 inpuut neurons, 11 hidden laayer neeurons and 4 output o neurons ns (9-11-4 netw work). A single neurron is repressented by a Neuron.cs class caalculating the membrane pootential accorrding to the SR RM laaunched in a thread. Thaanks to crosss-threading caalls, ev very thread caan communicaate with otherrs, and, when the th hreshold level is reached, a spike is geneerated, being sent s to o every postsy ynaptic neuroon connected to it. Moreover, th his neuron claass is designedd to react on incoming spiikes so o that the mem mbrane potenttial can be corrrectly calculaated acccording to all a input spikkes and weigh ht values. Evvery neeuron keeps trrack of fired sspikes and meembrane potenntial hiistory to be plo otted in real tiime during sim mulation.

T

if T  0 : wij ( T )  A  wmax e



(5))  T

if T  0 : wij ( T )   A  wmaxx e



(6))

The parameter A is responsibble for the streength of changge; a τ+ constaants determinees a time rang ge (Vogelsteinn et τ- and al.,, 2003) when pre and postssynaptic spikees are taken innto con nsideration. Fig g. 4 shows thee ij(T) funnction. If a prresynaptic spiike occcurs after a poostsynaptic sppike has been n fired, a weigght

3.2. Hardware Implementatio I on. 1936

Preprints of the 18th IFAC World Congress Milano (Italy) August 28 - September 2, 2011

The hardware neeural model has h been deveeloped in VHD DL lan nguage. A hierarchicall structure allows hiigh mo odularization and parametterization so that only oone parrameter file is needed to be modified for a different SN NN stru ucture generration. The entire neuraal structure is gen nerated with ggeneric statem ments with forr loops to creaate insstances of eveery network element e and map m all ports to pro ovide approprriate connecttion. A glob bal.vhd packaage hollds types usedd by the entiree structure, paarameters for tthe nettwork such aas the numberr of neurons in every layyer, syn napses delayss, and every other constan nts or variab les mo odification reequired by the appropriate hardwaare gen neration. Twoo clock signnals are defin ned, being 1m ms perriod the timee base for sppike transmisssion, and 100ns perriod for comp mputational caalculations req quired beforee a new w spike evaluaation.

nal vector eveery millisecon nd. When thiss bit along an intern co omes out at th he end of thiss vector the sy ynapse delay has beeen met. A single neuro on hardware m model is pressented in Figg. 6. Th he PSPgenara ator componeent is generateed for every innput to o the neuron. When a sppike arrives to t the input, the PS SPgenerator module inpput is set to t 1, then, the PS SPgenerator module m startss to read SRM M function vallues stored in the gllobal packagee. The values generated forr all th he PSPgenerattor modules aare added by two componeents: Po otential_Mux, multiplexinng the outp put from evvery geenerator and AddUnit, A respponsible for ad ddition. The sum s off PSP function n values from m every generrator is compaared to o the threshold by PotentiaalControl uniit, if the sum m of vaalues is abov ve the threshoold the outpu ut of the neuuron module m is set to o 1 (fired) forr 1ms. PotentiialControl unit is reesponsible forr the refracttory period to t avoid a new n acctivation durin ng that time.

The input neuronn is responsibble for firing a spike train, itt is bassed on a counnter so that a spike is geneerated while tthe inp put is active, aand the time separation beetween spikess is deffined by a valuue defined in the global pacckage.

Once all neuron ns are generate ted, all interco onnections am mong th hem (synapsees and its associated delay) d are also a geenerated, obtaaining the SN NN structure defined by the so oftware application.

The synapse coomponent is designed d as one o bit FIFO to mo odel synapse delay, the syynapse length corresponds to thee synapse dellay. When a postsynaptic neuron firess a spiike a synapse input bit is set s to 1 and th his bit is shift fted

Fig. 6 Hardw ware neuron strructure layout. a particular patttern is presennted to the in nput. Howeverr, if more m than one output neuroon fires, the so-called s winnnertaakes-all rule can be used, i..e. only the first fi firing neuuron affter the input stimulus s is preesent, is taken n as the winnerr.

4. RES SULTS In order to test hardware annd software, a simple patteern reccognition probblem is propoosed, a black k and white 33x3 pix xel image is feed and patterns must be reco ognized (Fig.77).

4.1. Software reesults. Designed software allowss simulating spiking neuural strructure. Duriing simulatioon, the softw ware shows the membrane m poteential of a seleected neuron (Fig. 8).

Fig g. 7. Proposedd patterns for SNN S classificaation. The SNN associates one outpput neuron with w one patteern. As the best resullt, only one ouutput neuron should s fire whhen 1937

Preprints of the 18th IFAC World Congress Milano (Italy) August 28 - September 2, 2011

d) Fourt rth pattern Fig g. 8. Membranne Potential grraph from softtware appl.

Fiig. 9 Neural Response R in sooftware application for four diifferent pattern ns. Active spikkes are differeent according to th he introduced pattern. p

Fig g. 9 shows thee neural respoonse after intro oducing patterrns fro om the pattern set (Fig. 7). A response to every patternn is reccorded for 7000ms. Green baars show the activity a of outpput lay yer, it can be observed thatt, for the pattern (a), the fiirst outtput neuron ffires first, foor pattern (c)) fourth neurron com mes first, andd so on. Accoording to the winner-takes--all rule the output nneuron firing first solves th he classificatiion tasks. If weightss are modifiedd, the system behavior is allso mo odified, affeccting firing times, und thus, patteern claassification abiility.

During code geeneration, everry VHDL sou urce file is parrsed nd, if any tag occurs, it is bbeing replaced by an assiggned an vaalue (see Fig. 10).

Wh hen a softwaare neural strructure is designed, we ccan creeate the syynthesizable VHDL filees for FPG GA imp plementation. This is done by a VHDL Code C Generattor too ol created for this purpose. Its functionaality is based on VH HDL source template files stored in i one of tthe app plication foldders and one file storing taags with valuues corrresponding too the created structure s using g the softwaree. Fiig. 10. VHDL code generato tor based on taags. 4.2. FPGA resu ults. Th he designed VHDL V code iss fully synthesizable on FP PGA bo oard, as an ex xample, the SSpartan3 XC3 3S4000 device is ussed. Table 1 sh hows the resou ource used in the t example. Structure Single neuron n 2-4-1 network k 32-4-16 3 network

a) First pattern

Slices Uttilization 2002 22668 113340

LU UTs Utilizatioon 378 4180 21059

Taable 1. FPGA A resource utililization accorrding to differrent sy ynthesized sttructures. N Numbers are:: InputLayerr HiddenLayer1 – OutputLayeer 5. CONC CLUSIONS A software app plication for ddesigning SNN N structures with w th he ability to simulate and geenerate syntheesizable code has beeen proposed.. Thus, a SN NN designer without w hardw ware deesign knowleedge can ggenerate and d migrate SNN S strructures to hardware.

b) Second pattern

Th he FPGA reso ource utilizatiion could be optimized. o In the prroposed desiign, the pssp_generator component is reeplicated for every instanttiation. It is synthesized as a a sin ngle ROM un nit. For a netw work with N input i neuronss, M neeurons in hidd den layer and K neurons in n output layer the to otal number off psp_generatoor componentts is equal to:

  N M  M  K

c) Thirdd pattern 1938

( (7)

Preprints of the 18th IFAC World Congress Milano (Italy) August 28 - September 2, 2011

Goldberg, D.H. and Andreou, A.G. (2007). Neural Computation. Distortion of Neural Signals by Spike Coding, Vol. 19, No. 10, pp. 2797-2839 Gómez Casado, J. (2008). Implementation of an artificial neuron in VHDL, Mälardalen University, Sweden Gupta, A. and Long, L.N. (2007). Character Recognition using Spiking Neural Networks, Proceedings of International Joint Conference on Neural Networks, Orlando, Florida, USA Hagras, H., Pounds-Cornish, A., Colley, M., Callaghan, V. and Clarke, G. (2004). Evolving Spiking Neural Network Controllers for Autonomous Robots, Department of Computer Science, University of Essex, UK Meftah, B., Benyettou, A., Lezoray, O. and QingXiang, W. (2008). Image Clustering with Spiking Neuron Network, IEEE International Joint Paugam-Moisy, H. and Bothe, S. (2009). Computing with Spiking Neuron Networks, Handbook of Natural Computing, Springer-Verlag Perrinet, L.U. (2009). Sparse Spike Coding: applications of Neuroscience to the processing of natural images, Institut de Neurosciences Cognitives de la Mediterranee(INCM)CNRS/Univ. Provence, France Ponulak, F. (2006). Supervised Learning in Spiking Neural Networks with ReSuMe Method, Poznań, Poland Roggen, D., Hofmann, S., Thoma, Y. and Floreano, D. (2004). Hardware spiking neural network with runtime reconfigurable connectivity an autonomous robot - Institute of Systems Engineering Logic Systems Laboratory, Institute of Computing and Multimedia Systems EPFL, Lausanne, Switzerland Schrauwen, B. and van Campenhout, J. (2006). International Join Conference on Neural Networks. Backpropagation for Population-Temporal Coded Spiking Neural Networks, Vancouver, Canada Song, S., Miller, K.D. and Abbott, L.F. (2000). Competitive Hebbian learning through spike-timing dependent synaptic plasticity, Nature Neuroscience, volume 3 no. 9. Tommiska, M.T. (2003). IEE Proc.-Comput. Digit. Tech. Efficient digital implementation of the sigmoid function for reprogrammable logic, Vol. 150, No. 6 Vogelstein, J.R., Culurciello, E., Mallik, U. and others, (2005). Saliency-Driven Image Acuity Modulation on a Reconfigurable Silicon Array of Spiking Neurons, Cambridge, MA: MIT Press, 2005, 17, pp.1457–1464 Vogelstein, L., Tenore F. and Others (2003). Advances in Neural Information Processing Systems 15. SpikeTiming Dependent Plasticity in the Address Domain, pp. 1147-1154. MIT Press. Wu, Q.X., McGinnity, T.M., Maguire, L.P., Belatreche, A., and Glackin B.(2008). ScienceDiscret Neurocomputing, Processing visual stimuli using hierarchical spiking neural networks,71,pp2055–2068 Wysoski, S.G., Benuskova, L., and Kasabov N. (2008). ScienceDiscret Neurocomputing, Fast and adaptive network of spiking neurons for multi-view visual pattern recognition, 71, pp 2563–2575

A solution to reduce utilization is to design a psp_generator component able to calculate the PSP function during one clock cycle. However, This solution introduces some limitation to PSP function characteristic due to difficulties with calculations in hardware modeling. Some other solutions must be investigated. At the moment, another limitation concerns to the learning algorithm. The STDP algorithm should adjust weights adequately and assign every output neuron to every pattern form training set. This algorithm has been designed in the software application, but not for hardware implementation yet. The STDP algorithm can be implemented with two additional FIFO queues for every synapse. These FIFO will remember the pre and postsynaptic spikes history according to τ- and τ+ constants so that the STDP function could be calculated using a simplified piece-wise linear function as in (Vogelstein et al., 2003). ACKNOWLEDGMENTS A. B. Fijalkowski thanks to prof. Marian Adamski and Dr. Piotr Bubacz from University of Zielona Góra for trusting him, making it possible to take part in Erasmus program. REFERENCES Arbib, M.A(editor) and Others (2002). The Handbook of Brain Theory and Neural Networks: Second Edition, The MIT Press, Cambridge, USA Bailey, J.A., Wilson, P.R., Brown, A.D. and Chad, J. (2008). Behavioural Simulation and Synthesis of Biological Neuron Systems using VHDL, IEEE International Behavioral Modeling, University of Southampton, UK Bellis, S., Razeeb, M., Saha, K. and Others, (2004). FPT'04. FPGA Implementation of Spiking Neural Networks - an Initial Step towards Building Tangible Collaborative Autonomous Agents, International Conference on Field-Progr Tech., The University of Queensland, Brisbane, Australia Beńuskova, L. (2006). Spiking Neuron Model and SpikeTiming dependent plasticity, Bratislava, Slovakia Booij, O. (2004). Temporal Pattern Classification using Spiking Neural Networks, University of Amsterdam, Amsterdam, The Netherlands Bothe, S.M., Kok, J.N. and La Poutre, H. (2002). Modeling Efficient Conjunction Detection with Spiking Neural Networks, Eindhoven University of Technology, The Netherlands Fijałkowski, A.B. (2010). Software Environment for Spiking Neural Network Design and FPGA implementation, University of Valencia, Spain Floreano, D. and Mattiussi, C. (2001). Evolution of Spiking Neural Controllers for Autonomous Vision-Based Robots, Institute of Robotics, Swiss Federal Institute of Technology, Lausanne, Switzerland Gerstner, W. and and Kistler,W.M. (2002). Spiking Neuron Models, Cambridge University Press, UK Gerstner, W. and Werner, M.K. (2005). Spiking Neuron Models, Single Neurons, Population, Plasticity, Cambridge University Press, Cambridge, UK

1939