Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
IMPLEMENTATION OF A SIGMOID ACTIVATION FUNCTION FOR NEURAL NETWORK USING FPGA Thamer M.Jamel Ban M. Khammas Electrical Engineering Department Information Engineering College University of Technology Al –Nahrain University Baghdad, Iraq Baghdad, Iraq
[email protected] [email protected] [email protected]
Abstract In this paper, the design of a single neuron which contains a sigmoid activation function was proposed and implemented using the FPGAs (Field Programmable Gate Array) techniques. The main goal of this neuron design is to enable use of any numbers of this neuron that its activation function type is sigmoid function. Furthermore, its goal is design any neural network with any number of hidden layers with a different number of this neuron and any number of output neurons, in easy manner using FPGA. Therefore, this implementation neuron can be use in any application that requires FPGA technique. A single neuron is designed using a schematic editor on Xilinx Foundation Series. Keywords : FPGA, Neural Networks, Sigmoid Activation Function, Schematic Tools.
I. INTRODUCTION Neural networks have been used broadly in many fields, either for development or for application. They can be used to solve a great variety of problems that are difficult to be resolved by other methods. ANN has been used in many applications in science and engineering. [1] Although, neural networks have been implemented mostly in software, hardware versions are gaining importance. Software versions have the advantage of being easy to implement, but with poor performance. Hardware versions are generally more difficult and timeconsuming to implement, but with better performance than software versions. [2] Most of the design neural network was done in AHDL (Altera'sTM Hardware Description Language) and VHDL [3-16]. Researchers now days are looking forward to implementing ANN in reconfigurable to obtain hardware versions of neural network estimators. Recently, several FPGA-based machines have been designed and built. These machines ١
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
have demonstrated supercomputer level performance for variety of computationally intensive problems. One limitation of these machines is their programming environment [4]. There are three ways to program an FPGA by using VHDL or state machine or by using the schematic (Hardware design tools). While this approach (i.e. schematic) permits the most flexibility and highest performance, it requires that the programmer be a skilled hardware designer. A very important part of neuron implementation is activation function hardware design. The most frequently used activation function in back propagation neural networks is the sigmoid function. Therefore, several papers attempt to implement the sigmoid function using FPGA approach [8-15]. All these previously papers used VHDL (Very high speed integrated circuit Hardware Description Language) as a hardware description language. Therefore, the purpose of this paper is to design a single neuron which contains a sigmoid activation function using a schematic editor on Xilinx Foundation Series. II. THE GENERAL PROCESSING ELEMENT (NEURON) The general processing elements for biological neuron is shown in figure (1) [5]. b0
X1
wj1 wj2
X2
netj
Neuron j wj3
X3
Neuron function
F(netj)
wjk
Xk Inputs
Figure 1 : Basic Neuron Module. Where : X1 , X2 . . . XK : are the input elements of a single neuron. B0 : biased. F(netj): activation function. Yi : output of neuron. ٢
yj
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
Wij: weights. This structure represents a single PE (Processing element) in a network. The input to the PE is modeled as arrows from other processing elements. Each input value has its associated quantity; Wjk called a weight. The output is a single output value that can fan out to other units. Each PE determines a net-input value based on all its input connections. According to the above, the net input to the ith unit of the next layer from the jth node can be written as: net j = Xk Wik ……(1) Where: Xk: is the input to the node. Wjk : is the weight associated to each input to the node from input k to node j. Where the indexes j are layer's index to the PE. This sum-of-products calculation plays an important role in the network simulations that will be described later. Because there is often a very large number of interconnects in a network, the speed at which this calculation can be performed usually determines the performance of any given network simulation. Once the net input is calculated, it is converted to an activation function calculation. The determination of the output of the function is as follows: y j = f(net j ) …..(2) The term activation function is used to refer to the function, f that converts the net input value (net) to the node’s output value (y j).[6] Figure (2) shows a direct digital hardware implementation of a neuron. For a neuron with N inputs, then it is required N multipliers, N-1 adders and the hardware to implement the limiting function, f(net) are required also [4].
Figure 2: A digital representation of a neuron. III. SUGGESTED DESIGN OF THE SIGMOID ACTIVATION FUNCTION
٣
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
For calculation of activate value, then sigmoid function is used. For instance, in many cases, the sigmoid function forms the most computationally expensive pare of the neural calculation. The most important problem of realization of sigmoid function is processing the load. In many applications look up table are used to overcome this problem, but in this paper, the design of this activation function will be illustrated without using a lookup table. The design of activation function is the second operation part of the neuron. The activation function f(net) is used to limit the value of the neuron output. This activation function is given by the equation: e-net
1 f(net) = 1 e - net
…..(3) This equation was selected because it provides the necessary limiting of the outputs while having some properties, which are useful in the learning phase of the algorithm. Unfortunately, this equation contains the transcendental function (exponential term, exp), which is somewhat difficult to calculate. Nordstrom and Svensson [7], list several functions, which may be used as an approximation to the function used by McClelland and Rumelhart. These functions all have the same general characteristics. They are continuously increasing, approach 0 at - and 1 at + , and have a continuous first derivative. The approximation that used in this neural network is given by the function: f (net )
1 net 1 2 1 net
…..(4)
The circuits may be extracted, which will implement the neural network algorithm. These circuits are extracted by creating the data flow graph for the above equation. This graph is then used to configure the hardware. Since the result of this equation f(net) is a digital circuit. Figure(3) shows the data flow circuit for the function f(net). This circuit takes as its input a value net which represents the result of sum of product, and returns the output f(net). The functional units used by the circuit are: two adders, a divider, an absolute value and a divide-by-two circuit. Some simple optimizations have been performed on this circuit. Once the circuit for this function has been extracted, it may be used as a macro cell, much like the other macro cells in the circuit.
٤
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
net 1
net
net
+
1
/
+
1/2 output
Figure 3 : The sigmoid activation function circuit. IV. IMPLEMENTATION OF SUGGESTED DESIGN The implementation of this neural network algorithm is well suited to an FPGA [2]. The first step is to design the equation [
net ] from 1 net
equation (4) and getting its result then complete the design of all sigmoid function. The design of equation [
net ] requires the divider circuit 1 net
and the absolute circuit to output the result of the summation in the first operation part of the neuron. Figure (4) shows the macro neuron component that was built in FPGA.
٥
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
Figure 4 : shows the macro neuron. Where: X[7:0], XX[7:0], XXX[7:0]: are the 8-bit input signals. W[7:0], WW[7:0], WWW[7:0]: are the three weights of single neuron each of 8-bit. BIAS[15:0]: is the 16-bit bias of the neuron. C: is the input clock of the system. CE: is the chip enable of the system. M[7:0]: is the 8-bit output of the neuron. V. SIMULATION RESULTS The curves in figure (6) provide a similar limiting function. It is the general characteristics of the sigmoid, not the precise equation which is important in this case.
1 F1(x) = 1 ex
,
F2(x) =
٦
1 x 1 2 1 x
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
Figure 6: Sigmoid activation function. Figure (6) shows the output curve of the two function (F1(x) , F2(x) ). This shows that the two functions is approximately the same function. The following table (Table 1) shows the comparison between the output of the neurons that are calculated using MATLAB program with that obtained using FPGA technique. Where the value of weights and bias are as follows: w1= 7.9, w2= 1.1, w3= -4.6. TABLE 1. RESULTS OF MATLAB AND FPGA
X1
X2
X3
Output of neuron using MATLAB
0.9 1.8 0.5 0.6 0.3 1.9 -0.2 -1.9
0.8 -0.4 0.4 -0.3 0.2 1.9 -1.3 -1.9
0.7 -1.5 0.3 1.8 0.1 1.9 -0.5 -1.9
0.92 0.98 0.35 0.06 0.19 0.999 0.0008 0.0004 ٧
Output of neuron designed using FPGA (represented in decimal) 0.8 0.9 0.3 0.1 0.2 0.9 0 0
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
All inputs and output of a neuron that built in FPGA is in hexadecimal value. In this table, the values were represented in decimal for easy understanding. As shown in this table, there is little difference between MATLAB and FPGA implementations. The reason of that is the quantization error which arises from number format representation in both implementations. VI. CONCLUSIONS This paper has presented the design and implementation of a neuron that will be used in any neural network, the activation function that designed inside the neuron is a sigmoid function. This design was performed using schematic editor tools in a reconfigurable device (FPGA) program. The design of the neuron will be as a micro neuron. Therefore, the user can use any numbers of this neuron by just make drag to the neuron component from the library of a schematic editor.
REFERENCES:[1] Medhat Moussa and Shawki Areibi and Kristian Nichols, "On The Arithmetic Precision for Implementing Back-Propagation Networks on FPGA", University of Guelph, school of engineering, Guelph, Ontario, Canada, 2003. [2] Rolf F. Molz, Paulo M. Engel, Fernando G. Moraes ," Codesign to Fully Parallel Neural Network for a Classification Problem" , University Montpellier II, France, 2000. [3] Miriam Galindo Hernandez, Raul R. Leal Ascencio and Cuauhtemoc Aguilera Galicia. "The Study of a Prototype of an Artificial Neural Network on Field Programmable Gate Array as a Function Approximate" ITESO, Department of Electronic, Systems and Information, Tlaquepaque, Jalisco, MEXICO, 2007. [4] Steven A. Guccione and Mario J. Gonzalez" A Neural Network Implementation Using Reconfigurable Architectures “Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, USA, 2006. [5] Reza Raeisi1, Armin Kabir ," Implementation of Artificial Neural Network on FPGA ", American Society for Engineering Education , ٨
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
Indiana University Purdue University Fort Wayne (IPFW) 2006 IllinoisIndiana and North Central Joint Section Conference . March 31-April 1, 2006 . [6] Nordstrom, T. and Svensson, B., " Using and Designing Massively Parallel Computers for Artificial Neural Networks", Journal of Parallel and Distributed Processing, no.3, pp. 260-285, March 1998. [7] Fazil Duman, Hamit Erdem, Baskent, " Hardware Implementation of neural networks on general purpose micro-controllers", University Faculty of engineering, Ankara/ Turkey. JCI proceeding of international conference on signal processing, volume:1, Number:2, September 2003. [8] M. C. Miglionico , and F. Parillo , " Modeling a Neuron using a Custom math Library sfloat24 – Implementation of a Sigmoid Function on a FPGA Device" , Proceedings of the International Symposium on the Analytic Hierarchy Process 2011. [9] Venakata Saichand1, Nirmala Devi.M2, Arumugam.S3, N.Mohankumar ,' FPGA Realization of Activation Function for Artificial Neural Networks ' IEEE Eighth International Conference on Intelligent Systems Design and Applications 2008. [10] Haitham Kareem Ali 1 and Esraa Zeki Mohammed , Design Artificial Neural Network Using FPGA ', IJCSNS International Journal of Computer Science and Network Security, VOL.10 No.8, August 2010 [11] Alin TISAN, Stefan ONIGA, Daniel MIC, Attila BUCHMAN ," Digital Implementation of The Sigmoid Function for FPGA Circuits", ACTA TECHNICA NAPOCENSIS Electronics and Telecommunications Volume 50, Number 2, 2009. [12] Saichand, V. Nirmala, D.M. Arumugam, S. Mohankumar, N. FPGA Realization of Activation Function for Artificial Neural networks : Journal , Eighth International Conference on Intelligent Systems Design and Volume: 3 Pages: 159-164 : 2008. [13] Coric S. Latinovic I. Pavasovic A.," A neural network FPGA implementation" , Neural Network Applications in Electrical Engineering,. NEUREL 2000. Proceedings of the 5th Seminar 117-120 , 2000 .
٩
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
[14] M. A. Bañuelos-Saucedo J. Castillo-Hernández S. Quintana-Thierry R. Damián-Zamacona , "Implementation of a Neuron Model Using FPGAS", Journal of Applied Research and Technology : Volume: 1 Issue: 3 Pages: 248-255 Universidad Nacional Autónoma de México (UNAM) , 2003. [15] Neelmani Chhedaiya , Vishal Moya," lImplementation of Back Propagation Algorithm in Verilog International Journal of Computer Technology and Applications ,Volume: 03 Issue: 01 Pages: 340-343 DOAJ Publisher: Technopark Publications ,2012. '
١٠
Published in the 13th Scientific Conference of Al-Ma'moon University College -18 April 2012 – Baghdad , Iraq
تنفيذ وبناء دالة التنشيط نوع ) (SIGMODEالمستخدمة في الشبكات العصبية باستخدام تقنية شريحة البوابات المبرمجة حقليا ملخص البحث-: ھذا البحث يقترح تصميم وبناء دالة التنشيط نوع ) (Sigmoidالمستخدمة في داخل العصيبة للشبكات العصبية عن طريق تقنية تقنية شريحة البوابات المبرمجة حقليا .وبالتالي يمكن استخدام اي عدد من ھذه العصيبة وكذلك اي عدد اخر من الطبقات المخفية للشبكة العصبية وعندئذ يمكن استخادم ھذه الشبكة في اي تطبيق وتنفيذه باستخدام تقنية شريحة البوابات المبرمجة حقليا .تم تصميم وبناء الدالة الواردة انفا باستخدام لغة الرسم التخطيطي ) ( a schematic editorلشريحة ) (Xilinxبعكس بقية البحوث االخرى التي تستخدم لغة الوصف الكيان المادي ) (VHDLوبالتالي يمكن اعتبار الطريقة المقترحة في البحث ھذا ھي طريقة جديدة حيث اثبتت النتائج المستخلصة نجاحھا.
١١
