Towards Tree-structure based Hierarchical Hybrid Soft Computing Yuehui Chen, Bo Yang and Ju Yang School of Information Science and Engineering Jinan University, Jiwei Road 106, Jinan 250022, P.R. China e-mail:
[email protected] Abstracts: There are three basic multilevel structures for hierarchical models, namely, incremental, aggregated and cascaded. Designing of these hierarchical models faces many difficulties including determination of the hierarchical structure, parameter identification and input variables selection for each submodels. A tree-structure based Hierarchical Hybrid Soft Computing (HHSC) framework is presented in this paper. Some instructions/operators for creating different HHSC models are discussed. An automatic design method of the HHSC models is proposed. In the framework, the tree-structure based evolutionary algorithms and a number of global search algorithms can be employed to optimize hierarchical structure, input selection and free parameters embedded in the HHSC model, respectively. Keywords: Evolutionary algorithm, neural networks, fuzzy systems, treebased model, hierarchical hybrid Soft Computing
1
Introduction
Real-world problems are typically ill-defined systems, difficult to model and with large-scale solution spaces. In these cases, precise models are impractical, too expensive, or non-existent. The relevant available information is usually in the form of empirical prior knowledge and input-output data representing instances of the system’s behavior [1]. Soft Computing (SC), including Neural Computing (NC), Fuzzy Computing (FC), Evolutionary Computing (EC) etc., provides us with a set of flexible computing tools to perform approximate reasoning, learning from data and search tasks. NC, FC, EC, among others, have been established and shown their strength and drawbacks. NC can perform ideally in domains of purely numerical nature, as well as in making effective predictions in time series data and nonlinear function approximations. EC could competitively perform optimization tasks in a very large search space, identifying sub-optimal solutions of high quality, becoming thus the methods of choice for domains suffering from combinatorial
548
RASC2004
explosion phenomena such as operations research, manufacturing etc. FC has been provide ideal for handling approximate concepts, human characterizations and domains having unclear boundaries. Moreover, it has been observed that the highly increasing computing power and technology, could make possible the use of more complex intelligent architectures, taking advantage of more than one intelligent techniques, not in a competitive, but rather in a collaborative sense. Therefore, discovering of more sophisticated and new evolutionary learning models and its application to new areas and problems still remain as key questions for the next 10 years. To investigate the hybrid technique further, a HHSC framework is proposed in this paper. Based on tree-structure based encoding and the specific function operators, new HHSC models can be flexibly constructed and evolved by using SC techniques. The paper is organized as follows. Next section gives the tree-structure based EC models. In the section 3 a HHSC framework and its design method are presented. The efficiency and effectiveness of the method are illustrated by two examples. Finally in section 4, we give some concluding remarks.
2
Tree structure based EC models
Tree-structure based evolutionary algorithms including Genetic Programming (GP), Probabilistic Incremental Program Evolution (PIPE)[4] and the recent variants of GP, i.e., Gene Expression Evolution (GEP) and Multi Expression Evolution (MEP) have been an active research area in recent years. Motivated by hierarchical fuzzy systems, a natural extension of traditional SC models is to introduce some intermediate levels of processing so that the HHSC models can be constructed. In this perspective, existing SC components should be employed to formulate different HHSC models and the architecture of HHSC model could be determined by hierarchical nature and its learning ability of tree-structure based evolutionary algorithms.
3
HHSC Framework
In this section, a tree-structure based HHSC framework is presented. As an example, the HHSC framework is shown in Fig.1(e). A function operator set {T2 , T3 , . . . , Tn } and a terminal set {x1 , x2 , . . . , xn } are employed in the creation of the tree. The function operator Ti (i = 2, 3, . . . , n) denotes that the node has i arguments with the operator type of T . Some possible types of the function operators are described as follows.
3.1
Types of function operators
Flexible Neuron Operator. The flexible neuron operator is shown in Fig.1(a). Assume that the used flexible activation function is x − ai 2 f (ai , bi , x) = exp(−( ) ). (1) bi
549
RASC2004
x1 x2
x1
ω1 ω2
Σ
f(a,b;x)
y
ωn
xn
T6
x2
x1 x2 T2 x3 T3
xn (b)
(a)
x1
ψ1
xn
ψm
. . .
y
Fuzzy Rule
ω1
y Σ
ωm
x1
AND
y
. . .
T3 x1 x2 T2 x3
T3
OR
xn
AND
W
V
(c)
x1 x2 x3
(d)
x3 x2 x1 x2 x3 (e)
Fig. 1: Some of function operators and a HHSC tree. (a)a flexible neuron operator, (b)TS fuzzy operator, (c)basis function operator, (d)a fuzzy neural operator, and (e)a general representation (encoding) of HHSC model, where the used function set is F = {T2 , T3 , . . . , T6 } and the terminal set T = {x1 , x2 , x3 }. where ai , bi are free parameters. The output the operator can be calculated as Pn j=1 ωj xj − ai 2 ) ). (2) y = exp(−( bi where ωj denotes adjustable connection strength of node and sub-node. TS Fuzzy Operator. The fuzzy operator is shown in Fig.1(b). Takagi and Sugeno developed a hybrid modelling technique designed to combine conventional and fuzzy modelling. The TS models are represented by a series of fuzzy rules of the form: Ri : IF(x is Ai ) THEN (y=fi (xi )) where fi (x) is a local model used to approximate the response of the system in the region of the input space represented by the antecedent. The overall model output is calculated as the normalized sum: Pp µ (x)fi (xi ) Pp Ai y = i=1 (3) k=1 µAk (x) where normalized fuzzy membership functions determine which local models are valid given a particular input. Basis Function Operator. The basis function operator is shown in Fig.1(c). In general, the basis function networks can be represented as y=
m X
ωi ψi (x; θ)
(4)
i=1
where x ∈ Rn is input vector, ψi (x; θ) is ith basis function, and ωi is the corresponding weights of ith basis function and θ is the parameter vector used
550
RASC2004
in the basis functions. Examples of typical basis functions include: (a)Gaussian radial basis function ψi (x; θ) =
n Y
2
exp(−
j=1
k xj − bj k ) aj 2
(5)
(b)Order 2 B-Spline basis function ψi (x; θ) =
n Y j=1
B2 (
xj − bj ) aj
(6)
where the translation and dilation of the order 2 B-spline function is given by 9 3 t−b 1 t−b 2 3 1 8 + 2 ( a ) + 2 ( a ) , t ∈ [− 2 a + b, − 2 a + b) 3 t−b 1 1 2 t−b t ∈ [− 2 a + b, 2 a + b) 4 −( a ) , B2 ( (7) )= 9 3 t−b 1 t−b 2 − ( ) + ( ) , t ∈ [ 12 a + b, 32 a + b] a 2 a 8 2 a 0, otherwise (c)Wavelet basis function ψi (x; θ) =
n Y
xj − bj 1 p φ( ) aj |a | j j=1
(8) 2
where φ is a mother wavelet, i.e., a Mexican Hat: φ(t) = √23 π −1/4 (1 − t2 )e−t /2 . Fuzzy Neural Operator. The fuzzy neural operator is shown in Fig.1(d). AND neuron is a nonlinear logic processing element with n-inputs x ∈ [0, 1]n producing an output y1 governed by the expression y1 = AN D(x; w)
(9)
where w denotes an n-dimensional vector of adjustable connections (weights). The composition of x and w is realized by an t-s composition operator based on t− and s−norms, that is n (wi Sxi ) y1 = Ti=1
(10)
with s denoting some s-norm and t standing for a t-norm. By reverting the order of the t- and s-norms in the aggregation of the inputs, we end up with a category of OR neurons, y2 = OR(x; w) n y2 = Si=1 (wi T xi )
(11) (12)
Each fuzzy neural operator is uniquely characterized by a number of parameters: a number of inputs, number of nodes in the hidden layer (h) and an array of connections of the AND neurons as well as the OR neuron in the output layer. The connections of the AND neurons can be systematically represented in a matrix form V while the connections of the OR neuron are collected in a single vector form w. The overall output of the fuzzy neural operator can be described as zj = AN D(x, Vj ), j = 1, 2, . . . , h, y = OR(z, w).
551
(13)
RASC2004
3.2
Creation
Creation method of program (HHSC tree) is similar to the one used by GP or PIPE algorithm. The only difference is that a specified data structure (parameters) should be embedded into the node of tree according to the type of the function operators selected.
3.3
Fitness Functions
A fitness function maps program to scalar, real-valued fitness values that reflect the program’ performances on a given task. Firstly the fitness functions should be seen as error measures, i.e., M SE or RM SE. A secondary non-user-defined objective for which algorithm always optimizes programs is program size as measured by number of nodes. Among programs with equal fitness smaller ones are always preferred.
3.4
Optimal Design
In this section, an automatic design method of HHSC model is presented. The hierarchical structure is created and optimized by using tree-structure based learning algorithms. The fine turning of the parameters encoded in the structure can be accomplished by using a set of algorithms, i.e., genetic algorithm, evolutionary strategy, evolutionary programming, differential evolution, particle swarm optimization (PSO), ant colony optimization, random search etc. The proposed method interleaves both structure and parameter optimizations. Starting with random structures and related parameters, it first tries to improve the hierarchical structure and then as soon as an improved structure is found, it fine tunes its parameters. It then goes back to improve the structure again and, provided it finds a better structure, it again fine tunes the rules’ parameters. This loop continues until a satisfactory solution (hierarchical model) is found or a time limit is reached.
3.5
Examples
The Flexible Neural Tree Model. Based on the method proposed above, an automatic design method for the flexible neural tree model has been accomplished [2]. In this research, the tree structure and free parameters embedded in the neural tree were evolved by using PIPE and a random search algorithm, respectively. PIPE iteratively generates successive populations of functional programs according to an adaptive probability distribution, represented as a probabilistic prototype tree (PPT), over all possible programs. Each iteration uses the best program to refine the distribution [4]. Thus, the structures of promising individuals are learned and encoded in the PPT. The PPT stores the knowledge gained from experiences with programs (trees) and guides the evolutionary search. This framework allows input variables selection, over-layer connections and different activation functions for different nodes.
552
RASC2004
The Hierarchical TS-FS Model. An automatic design method of hierarchical TS-FS models using Ant Programming (AP) and PSO algorithms were presented [3]. In the AP algorithm, each ant will build and modify the trees according to the quantity of pheromone at each node. The pheromone table appears as a tree. Each node owns a table which memorize the rate of pheromone to various possible instructions. First, a population of programs are generated randomly. The table of pheromone at each node is initialized at 0.5. The higher the rate of pheromone, the higher the probability to be chosen. Each program (individual) is then evaluated using a predefined objective function. The table of pheromone is update by two mechanisms: (1) Evaporation decreases the rate of pheromone table for every instruction on every node; (2) For each tree, the components of the tree will be reinforced according to the fitness of the tree. Meanwhile, the PSO was employed to optimize the adjustable parameters within a fixed best tree structure. Good performance has been obtained for time-series prediction problems.
4
Concluding Remarks
Based on the tree-structure encoding, an approach for evolving the HHSC models is proposed in this paper. The hierarchical architecture and inputs selection of the hierarchical model were accomplished using tree-structure based learning algorithm, and the related parameters embedded in the hierarchical model are optimized using global search algorithms. Our future work should concentrate on: (1)improving convergence speed of the proposed method by parallel implementation of the algorithm at the Dawning-3000 supercomputer; (2)evolutionary design of the hierarchical basis function networks and the fuzzy neural networks. It should be noted that it is possible to contain more than one types of function operator in a hierarchical mode. This is ongoing work to develop new hybrid SC models.
References [1] A. Abraham, Lakhmi Jain and Janusz Kacprzyk (Eds.). Recent Advances in Intelligent Paradigms and Applications, Studies in Fuzziness and Soft Computing, Physica Verlag, 2002. [2] Y. Chen, B. Yang, J. Dong. Nonlinear systems modelling via optimal design of neural trees. International Journal of Neural ystems Vol.14, No.2 pages 125-138, 2004. [3] Y. Chen, J. Dong, B. Yang. Automatic Design of Hierarchical TS-FS Models using Ant Programming and PSO algorithm. Proceedings of the Eleventh International Conference on Artificial Intelligence: Methodology, Systems, Applications, LNCS, Vol.3192, pages 285-294, 2004. [4] R. P. Salustowicz and J. Schmidhuber. Probabilistic Incremental Program Evolution. Evolutionary Computation, Vol.2, No.5, pages 123-141, 1997.
553
RASC2004
