(Dynamic concept model learns optimal policies - Neural Networks ...

Dynamic Concept Model Learns Optimal Policies es. Szepesvari

Abstract- Dynamic Concept Model (DCM)

is a goal-oriellted neural controller, that builds an internal representation of events and chains of events in the form of a directed graph and uses spreading activation for decision making

[1].

It is shown, that a special case of DCM

is equivalent to reinforcement learning (RL) and is capable of learning the optimal p olicy in a probabilistic world. The memory and computational requierements of both DCM

of more distinctions allowing faster convergence and generalization [2]. II. MARKOVIAN DECISION PROBLEMS

The bases of the used theoratical framework is a class of stochastic optimal control problems called Markovian Decision Problems (MDP). Such a prob­ lem is defined in terms of a discrete-time stochastic dynamical system with finite state set

and RL are analyzed aud a special algorithm is introduced, that ensures illtentiollal behavior.

I.

INTRODUCTION

Reinforcement learning is a flourishing field of neu­ ral methods. It has a firm theoretical basis and has been proven powerful in many applications. A brain .' model based alternative to RL has been introduced in the literature: It integrates artificial neural net­ works (ANN) and knowledge based (KB) systems into one unit or agent for goal oriented problem solving. The agent may possess inherited and learnt ANN and KB subsystems. The agent has and devel­ ops ANN cues to the environment for dimensionality reduction in order to ease the problem of combina­ torial explosion. A dynamic concept model was for­ warded that builds cue-models of the phenomena in the world, designs action sets (concepts) and make them compete in a neural stage to come to a deci­ sion. The tompetition was implemented in the form of activation spreading (AS) and a winner-take-all mechanism. The efficiency of the algorithm has been demonstrated for several examples, however, the op­ timality of the algorithm have not yet been proven in general. Here, a restriction to Markov decision problems (MDP) shall be treated making possible to show the equivalence of a special AS and RLl. The equivalence in this special case means, that OeM has all the advantages of RL, moreover it keeps track The author is with the Department of Mathematics, Attila J6zsef University of Szeged, Szeged, Hungary 6720 1 Note, that RL and MOP are used interchangeable as RL is a special solution to MOP.

0-7803-1901-X/94 $4.00 ©1994 IEEE

in

the text,

1738

At each discrete time step t, (t = 0,1,2, . ..) a con­ troller observes the system's current state (8(t» and generates a control action (a(t», which is applied as input to the system. Actions can be choosen from a finite set A. If 8(t) = 8i is the observed state, and the controller generates the action aCt) = a, then at the next time step the system's state will be s(t + 1) = 8j with probability Pij(a). Further, it is usual to assume, that the application of action a in state 8i incurs an immediate c08t ci(a). A closed-loop policy (or simply a policy) speci­ fies each action as a function of the observed state. Thus, such a policy is a function p : S - A. For any policy p there is a real-valued function, flA : S - R, called the cost function, corresponding the policy p. Here we define it to be the expected total injinite­ horizon discounted cost that will be incurred over time given that the controller uses policy p: 00

r(8) = EIA [E "y'C(t)18(0) = 8] , t=O

(1)

where 1, 0 < 1 < 1, is a factor used to discount future immediate costs, and EIJ is the expectation assuming the controller always uses policy p. The objective of the type of Markovian decision problem we consider is to find a policy that mini­ mizis the cost of each state 8 as defined by Eq. (1). A policy, that achieves this objective is an optimal policy which we will denote by p•. Note, that there may be more than one optimal policy for the same problem, but to each optimal policy corresponds the same cost function, which is the optimal cost func­ tion.

C. Methods for finding the optimal cost function A. Reformulating goal oriented behavior In a previous study a different formalism was used, A sutiable method for solving Eq. (3) is Real· Time namely goal oriented behavior [1]. Let us call the Dynamical Programming (RTDP), which is a form of underlying problem of goal oriented begavior to be Asynchronous Dynamic Programming (ADP). ADP the goal oriented decision problem (GDP). It will be is a successive approximation method for solving the shown, that GDP is a special case of MDP. In GDP a Bellman Optimal Equation. It uses a series of cost finite, fixed set of goals G C peS) is defined. Every functions f(") for estimating the optimal cost func­ goal g EGis identified by the set of states in which tion r. For every k = 0, 1,2, . . . let us denote by the goal is satisfied. Now, the immediate cost of a S" C S the set of states whose costs are backed up state, s, is defined to be the number of goals, which at stage k. Function j< "+l) is computed as follows: are not satisfied in the given state: c(s) = I{g E inaEA Q/(") (s, if s E S,,; (5) G : ¢ g}12. The cost function of policy Jl may be J