A study of the Adaptation of Learning Rule Parameters using a Meta Neural Network Colin McCormack, Dept. of Computer Science, University College Cork, Cork, Ireland. E-mail:
[email protected] Abstract This paper proposes an application independent method of automating learning rule parameter selection using a form of supervisor neural network, known as a Meta Neural Network, to alter the value of a learning rule parameter during training. The Meta Neural Network is trained using data generated by observing the training of a neural network and recording the effects of the selection of various parameter values. Experiments are undertaken to see how this method performs by using it to adapt a global parameter of the RPROP and Quickpropagation learning rules.
1 INTRODUCTION Despite the development of more efficient learning rules it remains necessary to manually select appropriate learning rule parameter values in order to achieve an acceptable solution. Two of the major problems associated with the selection of suitable parameters are the erratic nature of the quality of the solution (where quality can be defined as the speed of convergence and the accuracy of the resultant network) and the waste of resources used to train an unsatisfactory network. The quality of the solution is heavily dependant on the initial learning rule parameters and a waste of computational resources results from training neural networks using general parameter values which turn out to be unsuitable for the problem at hand. This paper investigates a method of parameter adaptation which involves the use of a separate neural network (called a Meta Neural Network) to select appropriate values for the η- parameter of the RPROP learning rule and the ε parameter of the Quickpropagation learning rule. We look at the results obtained when a standard RPROP rule, a standard Quickprop rule and a Meta Neural Network are applied to two benchmark problems. 1.1 Quickpropagation Quickpropagation (Quickprop) [1] is one of the most effective and widely used adaptive learning rules. There is only one global parameter making a significant contribution to the result, the ε parameter. Quickpropagation uses a set of heuristics to optimise Backpropagation, the condition where ε is used is when the sign for the current slope and previous slope for the weight is the same. In this situation the update rule for each weight is stated to be: ∆w(t ) =
S (t ) . ∆w(t − 1) + ε . S(t ) where S(t) and S(t-1) are the current and previous S(t − 1) − S(t )
values of the summed gradient information (slope), δE / δw , over all the patterns in the training set. The value of ε is split by dividing it by the number of inputs to each unit. This helps ε stay in a useful range since the accumulated δE / δw values are proportional to the fan-in and this results in an improvement in performance. The value of ε is usually set to 0.01.
1.2 RPROP Resilient Backpropagation (RPROP) [2] is a local adaptive learning scheme. In it the size of the derivative is taken to indicate the direction of the weight update. The size of the weight update, ∆wij(t ) , is determined by a weight update value ∆(ijt) , where t is the current epoch and i, j are the nodes adjoining the weight being updated. The weight update value is adjusted as training is carried out. The adaptation rules for the algorithm are:
∆wij( t )
Where
(t ) − ∆ ij , if = + ∆(ijt) , if 0 , else
δE ( t ) >0 δwij δE ( t ) 0 δwij δwij δE ( t −1) δE ( t )
