DUNEDIN
NEW ZEALAND
Hybrid Neuro-Fuzzy Inference Systems and their Application for On-line Adaptive Learning of Nonlinear Dynamical Systems Jaesoo Kim Nikola Kasabov
The Information Science Discussion Paper Series Number 99/05 March 1999 ISSN 1177-455X
University of Otago Department of Information Science The Department of Information Science is one of six departments that make up the Division of Commerce at the University of Otago. The department offers courses of study leading to a major in Information Science within the BCom, BA and BSc degrees. In addition to undergraduate teaching, the department is also strongly involved in postgraduate research programmes leading to MCom, MA, MSc and PhD degrees. Research projects in spatial information processing, connectionist-based information systems, software engineering and software development, information engineering and database, software metrics, distributed information systems, multimedia information systems and information systems security are particularly well supported. The views expressed in this paper are not necessarily those of the department as a whole. The accuracy of the information presented in this paper is the sole responsibility of the authors. Copyright Copyright remains with the authors. Permission to copy for research or teaching purposes is granted on the condition that the authors and the Series are given due acknowledgment. Reproduction in any form for purposes other than research or teaching is forbidden unless prior written permission has been obtained from the authors. Correspondence This paper represents work to date and may not necessarily form the basis for the authors’ final conclusions relating to this topic. It is likely, however, that the paper will appear in some form in a journal or in conference proceedings in the near future. The authors would be pleased to receive correspondence in connection with any of the issues raised in this paper, or for subsequent publication details. Please write directly to the authors at the address provided below. (Details of final journal/conference publication venues for these papers are also provided on the Department’s publications web pages: http://divcom.otago.ac.nz:800/COM/INFOSCI/Publctns/home.htm). Any other correspondence concerning the Series should be sent to the DPS Coordinator.
Department of Information Science University of Otago P O Box 56 Dunedin NEW ZEALAND Fax: +64 3 479 8311 email:
[email protected] www: http://divcom.otago.ac.nz:800/COM/INFOSCI/
Inference
H;/FIS: Adaptive Neuro-Fuzzy
1
Systems
Inference Systems Neuro-Fuzzy for On-line and Their Applications Adaptive Learning of Nonlinear Dynamical Systemsl
Hybrid
this paper,
Abstract-In to
build
and
learning
power
of
posed
guistic meaning
AND
KIM
JAESOO
adaptive neuro-fuzzy system, called HyFIS,
an
networks
a
hybrid learning
and the
generation from data,
agation learning scheme for
performance
be
can
of
nonlinear
applied to
rule
and revised
to
In
fuzzy system.
order
phase of
incremental
the
illustrate
to
carried
are
er-
rule
the
extensive
The proposed
out.
adaptive learning for
and
backprop-
error
proposedneuro-fuzzy hybrid model,
the purpose
of
dynamical systems.
systems, Neural
Neural
the
tuning by using the
networks, Fuzzy logic,
Science, University
Zealand.
Submitted
fuzzy logic rules
phases:
two
complex dynamics
of Information
introduces
optimally tuned from training
learning, Knowledge acquisition, Adaptation,
*TR-99-65, Department New
the
on-line
Keywords-Neuro-fuzzy structure
neural
of nonlinear
and control
prediction
a
be
can
composed of
phase of
applicability of
studies
.simulation
method
and
scheme
model
Heuristic
architectures.
to the connectionist
proposed
is pro-
fuzzy logic systems and provides lin-
the
into
input-output fuzzy membership functions
amples by
The
models.
optimise fuzzy neural
KASABOV
NIKOLA
Networks,
15
March,
1999.
of
Time
Parameter
and
series.
Otago, PO
Box
56, Dunedin,
H yF IS:
1
INTRODUCTION
Over
the
decade
last
technological
or
with
associated
provides
human
neural
human
networks
Neural
putational operations
data-driven
to
such
processes,
to emulate
certain
in the process
features
of neuronal
mechanisms
networks in
interpolation
derived
acquisition), or explanation
replicate,
on
small
a
Traditional
process.
ently
possess
biological learning
from
numerical
facilities.
ployed
for
numerical The
and
systems
inherent
in
decision
such features.
To circumvent
these
of the
allows
for
com-
a
versatile
the drawbacks
(knowledge
typically formulate
can
in the
decision-making
tools, however, do
inher-
not
obstacles, integrated techniques
systems, fuzzy logic and neural from
certain
adaptation. The input-
networks
experts
network-based
network
trained
models
network
are
being
models,
synergism of integrating Fuzzy Logic (FL) systems and Neural a
functional will
reasoning
system with
provide
a
learning capability suitable
more
the
greater
(FIS)
for
learning capability
deriving
the initial
tool
Considerable
imprecisely-defined complex systems. tegrate
understanding
em-
from
or
examples.
(NNS) into and
reasoning. It
knowledge extraction
or
Human
extracting linguistic information
of
examples. One of
neural
rule-based
between
uncertainties
scale, some
describing underlying causual relationships involved
rules
fuzzy logic
perceptual and linguistic
has evolved
observed
the and
thinking
as
of
theory
capture
distinct
in two
cognition.
systems is the lack of adequate rule
of such
ing
framework
mapping capability of multilayer perceptron
output
The
paradigm
learning and adaptive
biological species.
been made
networks.
and neural
morphology
2
have
advances
significant
cognitive
with
associated
attributes
Systems
mathematical
a
mathematical
a
The
so,
fuzzy logic
areas:
(Zadeh, 1965) provides
the
Inference
Adaptive Neuro-Fuzzy
of neural
rules
of
a
for
and
solving
work
high-level thin1 the
behaviour
has been
networks
fuzzy system
with
Networks
done
to
of in-
fuzzy inference
and tune
the
mem-
HyF IS: Adaptive Neuro-Fuzzy Inference
bership functions
(Lin
1992; Yamakawa
et
&
Shaun
Lee, 1991; Berenji 85 Khedkar, 1992; Horikawa
85
al., 1992; Jang, 1993; Hung, 1993; Ishibuchi 1996; Kasabov
Fu, 1995; Kasabov, the
have
neuro-fuzzy systems
potential
paradigms, fuzzy logic and neural
cellent
the
under
problem In this
and
study,
linguistic
illustrated
module,
in
proposed by Wang & the initial
into
Medal
a
membership
an
of
the
The
first
next
An
on-line
section
method
knowledge
from
en-
ex-
descent
two
new
is
developed
input-output
proper
as
knowledge acquisition data
pairs,
fuzzy logic rules
In the second
and
phase, a parameter
learning algorithm
repeated
phases are
suitable
important feature
to
scheme
is
applied
to
tune
linguistic variables.
input~output
model
desired
numerical
HyFIS (Hybrid neural
in the
used to find
fuzzy system.
of the
called
phase, realised
the
combining both
for
framework
(1992), is
Hylillls
functions
incremental,
have
existing expert knowledge about
two-phase learning
a
gradient
adapt and change according in
which
HyFIS architecture,
dynamical systems. the
they
semantically meaningful fuzzy rules, and
deriving fuzzy rules
membership functions
making
data
a common
the
In
learning technique using
thus
of
general
a
of the neural
structure
In the
both
propose
1.
for
method
a
we
System), in Fig.
of
range
and
consideration.
information
Fuzzy Inference
data,
in the form
accommodate
to
wide
a
powerful
learning, good generalisation capabilities,
fast and accurate
explanation facilities
and the ability
single capsule
a
to
of two
benefits
the
capture
suited
well
al., 1994;
et
al., 1997; Pal, 1998). These
et
into
al.,
et
applications (seefor example, Constantin, 1995). These
gineering and scientific
strengths include
to
networks,
them
make
that
features
several
3
Systems
of
for
incremental,
HyFIS
is that
fuzzy predicates,
as
training data.
The
well
on
on-line
it is as
any
new
of
set
learning
of
adaptable where
the
adaptation
fuzzy can
rules
take
can
place
mode.
introduces
the basic concepts of
fuzzy rule based reasoning
Inference
I-IyF IS: Adaptive Neuro-Eizzy
in neural
inference
fuzzy
problem that includes and
principles and
mance
AND
rules
fuzzy
inference
then
system.
B, where
y is
as
Fuzzy if-then
and
y
are
knowledge base
a
rules
and
variables
A and
capture
in
and
uncertainty. Through
fuzzy if-then
ability
rule
can
of
imprecise mode
the
role
a
the in
Box
and
Section
in Section
to
the
decisions
make use
of
that
5.
6.
resides in
expressions of the form
are
employed
human
results
STRUCTURES
by appropriate membership functions.
the
data,
summarised
are
characterised to
perfor-
Mackey~Glass
1970)-are presented
research
part of
the fundamental
are
the
REASONING
FUZZY
Fuzzy
system identification
& Jenkins,
for future
series of the
time
chaotic
KNOWLEDGE-BASED
FUZZY
typical modelling
a
To illustrate
described.
are
a
nonlinear
(Box
and directions
Conclusions
2
data
furnace
gas
discuss
we
proposed HyFlS model, experimental
data-
(Mackey 8/: Glass, 1977),and Jenkins
HyFIS
of the
benchmark
Well-explored
two
on
of
applicability
3,
learning and parameter learning. In Section 4; the
structure
the architecture
the
In Section
systems.
4
Systems
B
are
of
if
that
an
plays
is A
IE
fuzzy
Fuzzy if~then rules
reasoning in
labels
neural
a
an
sets
often
are
essential
of imprecision
environment
linguistic labels and membership functions,
easily capture the spirit of
rule
the
of
thumb
used
by
humans. The
inference
operations
upon
fuzzy
if-then
rules
ing. The general steps of fuzzy reasoning performed in are
as
1.
known
are
as
fuzzy
fuzzy
inference
membership functions
in the
a
reason-
system
follows:
the
Match
part
to
input
obtain
linguistic
label.
the
values
towards
membership
the values
premise
(or compatibility measures)for
each
Hy/FIS: Adaptive Neuro-Fuzzy Inference
Combine
2.
through
membership values
the
(weight) of
4.
the
on
Aggregate
the
effective
1
control a
in
fuzzy
the
I’
and
processes
inference
literature
into
three
f
or
depicted
are
rules
I
_
a
in
a
crisp
employed,
connectionist
most
rules with well
rules
are
described
the MAX
operation
1
used
functional
main
to
blocks
have been pro-
structure
Lin
& Lee, 1996; Kasabov,
the types of
neuro-fuzzy
types (Jang, 1993): Mamdani
1
_
wnen
2.
Fig.
on
1
11
controuers
fuzzy
1997). Depending
fuzzy
inference
type, Tsukamoto
reason~
systems
can
type, and
certainty factors (CFS)(Shaun& Fu, 1995;
as
generalised fuzzy production rules have These
also
five types of
below.
(Mamdani fuzzy models):
plying
be
dy-
proposed (Lee, 1990).
(Kasabov,1996a)to capture expert knowledge.
fuzzy reasoning
1
crisp
of
purpose
engine should
been
a
fuzzy-rule-based systems, fuzzy
as
(Lee, 1990; Kosko, 1992;
Kasabov, 1996a; Pal, 1998)as
Type
produce
for the
(Lee, 1990; Jang, 1993). The system
Takagi~Sugenotype. Fuzzy
been utilised
to
has been shown to be the most
lf)
(rfuvr/,
types of fuzzy reasoning in
fuzzy if-then
be classified
/17]
memorzes
1996; Jang, Sun, & Mizutani,
ing
have
method
known
also
are ’
assocwtzue
dynamical
Several
posed
of the decision
for defuzziiication
systems ’
neural
each rule
(Zadeh, 1965; Kosko, 1992).
inference
fuzzy
#ring strength
crisp) of
or
all rules
from
defuzzification). Since
is called
methods, the centroid
I’
I
moaeis,
of
these
(either fuzzy
values
qualified consequent
(This step
Among
min)
or
firing strength.
value, several methods
Fuzzy
value
system modelling the output
namic
the
obtain
to
premise part
qualified consequent
depending
output.
the
in
(usually multiplication
operator
each rule.
the
3. Generate
t-norm
specific
a
5
Systems
to
the
The overall
fuzzy output
qualified fuzzy outputs
is derived
(eachof
by which
apis
HyF IS: Adaptive Neuro-Fuzzy
equal
of each
rule).
crisp output
based
functions final
example of be
to
the output
have been
proposed
firing strength applied schemes
Various
the overall
on
ar
y, and
,,..,
1, 2
z
are
and the control
variables
n)
,...,
in the
the
are
linguistic
Mamdani
of the
values
linguistic
U ,...,
of
Type
representing the
This
1 in which
is
of each rule s crisp output
average
product
or
the consequent is
required
overall
input linguistic variables.
of
of input
combination is the
weighted
average
variables
This
1
where
ifa:
is
f,(:r,
A,,
_
_
_
_
_
.,
of each
,y)(i
and y is
:
1, 2,
rule s
be
to
on
mono-
with
the
be
weighted
firing strength (the and
premise part)
type of fuzzy if-then
_
a
rule s
of each
output
constant
output.
a
.
,
then
n) is
z
=
rules
fuzzy rule
is
term, and the final The
following
(MISO) Takagi-Sugeno fuzzy
Bi,
.
The
plus
multi-input-single-output
a
based
is the
output
proposed by Takagi & Sugeno (1983). The consequent of function
y, and
as ,...,
used in this scheme must
by the
match
(Takagi-Sugeno fuzzy models):
3
=
functions.
membership
output
induced
degrees of
of the
minimum
state
Bi, and O,
,...,
simplihed method
a
(Tsukamoto, 1979). The
functions
monotonic
R.,
can
process
variables
linguistic
tonic; that is, the output membership functions
of
model
fuzzy
V, and W, respectively.
(Tsukarnoto fuzzy models):
2
variables
variable, respectively, and A,
of discourse
universe
fuzzy reasoning
Type
the
to choose
(Mamdani, 1975). An
fuzzy output
multi~input-single-output (MISO)
a
membership
ifazis/1,,...,andyisB,,thenz==C’i,
:
where
Type
6
Systems
expressed as:
R.,
z
of
to tl1e minimum
Inference
is
an
was
rule
is
a
linear
a
output
example
model:
f,(:s,...,y),
afunction
of the
input variables
rr,
_
_
_
,y
Hy/FIS: Adaptive Neuro-Fuzzy Inference
Type
4
with
rules
(Fuzzy
tainty degree as
R4
ifx
:
where
Type
is
applications
Lee
(1996)and
of this
Pal
that
capture
and
exemplar fuzzy
shown
that
uncertainty
if :cl is
A1 (DIN)
R2
1
if my is
A2
are
used;
(1) confidence factors
coefficients
relative
contain
(2) relative
part;
in the
elements
antecedent
(3)
part;
(Kasabov,l996a).
of the conclusion
_.factors
or
degrees
degreesof importance (DI)
and 3:2 is
Two
of the
parts
(DIL2) and
1:2 is
B1
(DIQJ) then
y is
C1 (0171);
B2
(DIN)
then
y is
C2 (CF2).
type of fuzzy rules is used in the FuNN
structure,
In this
&
are
below.
:
FuNN
(1995),Lin
type of fuzzy rules several
to the conclusion
and certainty
R1
This
& Fu
Shann
see
interpretations
this
In
rules):
(4) sensitivity factor
rules
elements
condition
cer~
(1998).
of importance (DI) of the condition tolerance
a
weight), e.g.:
For different
ith rule.
type of fuzzy rules,
certainty factors (CFS) attached
Noise
called
contains
(01%),
(Generalised production
parameters
rule
fuzzy
a
(CF) (or also
certainty factor of the
and
5
y is B
A, then
type of
This
certainty factor
a
is the
CE
CFS):
7
Systems
rules into
to insert
this
model, and
model
to extract
(Kasabov, l996a; 1996b; 1996c; Kasabov study,
we
have constructed
functionally equivalent
to
the
Type~4
a
of
et
to
rules
the
from
a
FuNN trained
al., 1997).
neuro-fuzzy inference
fuzzy inference
define
systems
system which
(FIS).
is
HyF IS: Adaptive Neuro~Fuzzy Inference
IN
LEARNING
3
Generally, FIS require
of
tions
is
learning
the
the
terms,
Once
needs
with
such
membership functions,
is to
neuro-fuzzy system
There
several
are
be combined
in
and structure
of
numerical
and output
fuzzy
output In the
space,
numerical
following
of the
of two
and
Hence, the
structure
and
learning
we
in the next
to
main
find
of
purpose
and
tune
a
the
parameter
learning
learning, data
and
one
we
the
of the most the
mean
shall discuss
some
learning
important aspects of
extraction
of
fuzzy logic
a
fuzzy logic
rule
rules
of the
in the
fuzzy logic rules
tuning of fuzzy partitions
construction
of
can
FIS
has been
identification
Most
tuning
e.g.,
subsections.
procedures: 1) fuzzy partitioning
2)
FIS
widths, and slopes,
learning techniques
for
Generally,
examples. cases:
rules
training
following,
that
described
are
structure
spaces.
consists
membership func-
structure
centres,
as
fuzzy
neuro-fuzzy systems.
learning
design of FIS. By from
the
of the
neuro-fuzzy system. Existing techniques for parameter
a
learning
Identification
data
ways
Structure
3.1
apply neural of
structure
the structure
perform parametric tuning,
to
and para-
learning,
satisfactory
a
tuning the weights of the fuzzy logic rules.
parameters and
INFER-
structural
linguistic variables,
rules.
fuzzy
fuzzy model
the
of the
parameters
FUZZY
is concerned
system, i.e., input and output
obtained,
and
NEURAL
major types of learning:
two
Structural
learning.
inference
8
SYSTEMS
EN CE
metric
Systems
of the
from
input
for each
input
numerical
space
and/or
fuzzy subspace.
techniques for knowledge acquisitions from
existing techniques
can
be divided
into
either
of the
Inference
H5/FIS: Adaptive Neuro-Fuzzy
learning, such
vised
adaptive
vector
put
theory (ART) (Carpenter & Grossberg, 1987, 1988,
resonance
1990), which
Supervised learning
a
on
learning rules
fuzzy logic
in order
teacher
to
commonly used
specify the desired
the
determine
to
in-
find clusters
to
(Seealso Kosko, 1992).
gradient-descent method
a
in the
regularities
capture
(Rurnelhart et al., vectors,
output
and
& Zipser, 1985; Kosko,
learning (Grossberg,1976; Ruinelhart also
1990),are
of
presence
based
1986), which requires competitive
for structure
is suitable
space,
that
models
internal
construct
indicating the
of data
(SCMS)(Kohonen,1989) and
self-organising maps
as
(NN) techniques. Unsuper-
network
through neural
Fuzzy rule extinction
1
9
Systems
fuzzy
rules
(Kosko, 1992;
Kasabov, 1996a). 2.
Fuzzy rule
by using fuzzy techniques. A simple method
extraction
(1992)for generating fuzzy
by Wang & Mendal output
training data is
a
build-up procedure.
ene»pass
any
real, continuous
SOMS
When
patterns sion are
are
Pal
et
the
rules
reduction ol.
for rule
a
decision
was
the decision scheme
that
(1998)proposed
backpropagation
a
that
result, achieving the
vector
quantisation
borders,
which
of the
clusters
learned
can
necessary
input deci-
(LVQ) techniques provide fine-tuning
process.
combines
proposed by
and combination
the
extraction,
near-optimal
the time-
set.
compact
Learning
and enhance
learning fuzzy rule
a
on
It avoids
input-
systems is capable of approximating
rule
diiiicult.
hybrid learning
for
function
used
are
becomes
used to refine
A
to
substantially overlapped. As
accuracy
of SOMS
error
building fuzzy
approach
numerical
It has been also shown
for NNS.
consuming training procedures typical this
rules from
proposed
SOM
Lin and Lee
were
also
and competitive
learning for
(1991& 1996). Some
provided. Shann
et al.
learning procedure for acquiring fuzzy
(EBP) learning algorithm,
which
is based
on
heuristics
(1995) and rules a
using
gradient
HVF IS: Adaptive Neuro-Fuzzy Inference
search
descent
is
phase
is executed
process
base with
Here, by parameter learning parameters of fuzzy rules in ter
We
a
backpropagation training,
error
the initial
for
learning
FIS
FIS. There
functions
1992), which least square K alman
single
a
(LSE) (Jang, 1993),which
estimator
The
filter algorithm.
parameter
system is the supervised learning There of the
are
different
one,
parameter
fuzzy logic rules:
parameter
1.
learning
Parameter on
of these
learning
the l\/lamdani
function
is
two
in
the
compatible
method
rules.
rules
Here several
types of fuzzy rules
rules
(Type 1).
In this
usually bell shaped, triangular,
and
the
output. the
by
its output
centroid
membership
of the responses
A defuzzification
maximum
the
on
and
2)
referenced:
are
(MISO) systems the output
case
is based
membership
to
its
is
firing strength,
generate the appropriate
taking
membership degree as the crisp
of each rule
response
weighted by
is calculated
procedure involves
types
trapezoidal (Lin & Lee, 1991;
or
function
HyFIS
neuro-fuzzy techniques for
Berenji KL Khedkar, 1992; Kosko, 1992). The fuzzy defined
in the
singleton consequents;
multi-input-single-output
fuzzy
adopted
extended
an
backpropagation algorithm.
error
with
approximate
with
learning techniques depending
1) fuzzy logic
Takagi-Sugeno-Kang (TSK) fuzzy
of the output;
is
learning
with
backprop-
error
learning (Berenji & Khedkar,
evaluation
scalar
for parame-
al., 1992; Hung, 1993; Shann
et
al., 1997); reinforcement
requires only
and other
learning methods
several
are
agation algorithm) (Lin & Lee, 1991; Horikawa et
a
one.
learning: gradient-descent-based learning algorithms (e.g.,the
& Fu, 1995; Kasabov
a
fuzzy rules and obtain
tuning of membership
mean
second
training phase, and the
a
redundant
delete
to
size than
smaller
much
Parameter
3.2
phase is
rule-pruning phase, i.e., after the
a
rule-pruning rule
the first
in the network:
10
Systems
the output
response
value
of each rule
with
and
HYFISI Adaptive Neuro-Fuzzy Inference
then
these
aggregating
1990). Almost
all existing
this
type of rules
deal with
other
ter
be
In
the
rule
approach
a
input variables, instead
of the
then
weighted
final
obtain
the
gleton
when
a
and summed
(SeeType
which
according
the
the
to
2), it
3 in Section
either
be viewed
becomes
special
dani
fuzzy model (Type 1), in which each rule s consequent
a
fuzzy singleton,
in which
or
a
centre
at
optimising
the
In the
HyFIS
model
presented from
knowledge acquisition module
the
neural
network
structure
in the next
is
in
Fig.
of the Mamis
specified by
membership
(Type 2),
function
of
but
a
with
it combines
estimation
op-
(Jang,
of the
the
fuzzy logic approach
to
pairs is used and implemented in
1. After
established,
adjust the parameters
section,
data
input-output
the
to
sin~
zero-order
fuzzy model
least squares
to
1997
generating fuzzy rules
performed
re-
That
zero.
by gradient descent with
premise membership functions
timising the consequent equations by linear &
fuzzy
a
type of fuzzy rules is dealt
This
the constant.
a
The
a
case
neuro-fuzzy systems for FIS parameter learning,
in several
1993
a
of the Tsukamoto
consequent is specified by
each rule s
step function
special case
of each
firing strengths
fuzzy model,
as
constant.
response
TSK
can
parame-
simple
a
consequent part become
in the
coefficients
constant
of
consequent of each rule becomes
The
output.
all the is
is, when f
Heesche,
input variables.
of the
of the values
combination
linear are
sponses
8;
let the consequent
to
Takagi-Sugeno-Kang (TSK) fuzzy rule format
is
of FIS
tuning
(Horikawaet al., 1992; Hauptmann,
parameter learning is
in
function
linear
a
for parameter
neuro-fuzzy models
(Lee,
appropriate output
1996a).
1995; Kasabov 2. The
the
produce
to
responses
11
Systems
and
a
set of
the
membership
fuzzy rules
second functions
is
extracted,
learning phase optimally.
is
H yF IS:
4
Adaptive Neuro-Fuzzy Inference
HYBRID
HyFIS: EN CE
The
second
phase
achieve
to
is that
iI1
created
easy
for this
second
data
the
facilitates
architecture
f‘uzzy rules, In
sources.
A brief
the functions
The
thus
The
allows
it
of five
layers.
the
hidden
Its
layer
from
the
for
The
first
phase
is the
input
states
and output
layers,
there
are
This
eliminates
which
is difiicult
for
nodes the
an
model
in the
become
available
available,
data
HyF IS
descent
gradient
rule
a
and
uses
a
is
In the
multi-layered
learning algorithm.
approximate reasoning,
as
The
well
for the combination
of both
numerical
data
synergistic benefits
associated
with
two
in
the
in
of the
in the
HyF‘IS
is shown
in
a
the
dynamically changing
proposed HyFIS system
following
control
subsections.
a
multilayer
neural
3 and the system
input and output
of
as
a
understand
membership functions normal or
to
feedforward
modify.
networkhas
nodes
/decision signals respectively,
functioning
to
is
Fig. the
structure,
disadvantage
observer
data
new
approach
I-IyFIS
model
connectionist
as
this
fuzzy rule base (See Fig. 1).
presented
topology
tuning membership func-
pair becomes
of the structure is
for
adaptive learning
of
In this
rules.
the
to
a
on
introduction
proposed neuro-fuzzy
data
new
neuro-fuzzy
Architecture
fuzzy system.
a
based
rule base
fuzzy
It allows
of each
based
sent
When
producing
addition,
environment.
4.1
INFER-
performance. Gne advantage of
the
learning
knowledge acquisition.
and
phases.
learning phase
pair and added
network
(MLP)
of
level
modify
to
learning phase,
perceptron
and
FUZZY
of two
consists
HyFIS
parameter
desired
a
it is very
is the
(Wang & Mendal, 1992).
as
NEURAL
learning (rule finding) phase using the knowledge acquisition module.
structure
tions
12
SYSTEM
learning procedure
The
Systems
total
a
repre~
and in the
(MFS) and
multi-layer
net
HyFIS: Adaptive Neuro-Fuzzy Inference
Nodes the
in
functions
(MF)
functions
adapted through in
variables
used
is
the 3
Fig.
granularity
3 is
layer
between
In
the
that
ters
and
that 4
in
13
transmit
directly
nodes that
term
are
in which
the The
process.
value
mean
such
But
a
input/ output
sometimes
more
(ZE), small negative (SN), and large negative (LN). Each
node
node and
represents
3 and 4 represent to
detailed
rule.
fuzzy
one
The
degree controlled
certain
a
as
connection
weights
certainty factors (CFS) of the associated
description and
functionalities, the simulation
is
placed
on
by how
the
weight
rules,
values.
adapt the
to
the
of the components of the
philosophy
behind
examples presented in defined
in
this
this
parame-
MF; and
semantic
follows.
respectively.
1:
ables
Nodes as
the next
use
the width
of the MF.
indices
t,j,k,
from
the nth
layer
crisp values.
layer,
the
one
are
The
and
given. are
2
U2
e’
determines
output
in
=
by
and functions
are
struc-
all the MF used
paper,
is determined
meaning We
The
0
rnodel s
Eq. (1):
Gaussian(m;c,0)
=
HyFlS
architecture
function
membership
of the
Layer
is
0
represent the bell-shaped MF for each node in layer 2 and 4 through
Gaussian
as
the variance
small
;tA(w)
are
membership Bell-shaped
for the
and small
a:-c
The
as
to
large positive (LP),
applications
some
bell-shaped (Gaussian)functions
centre
act
and
c
defined
fuzzy sets
(M),
following, special emphasis
Throughout
A
input signals
input / output fuzzy linguistic variables.
large (L), medium
is activated
learning and ture
rule
layers
i.e., each rule
the
here,
required
a
2 and
layer
learning
are
positive (SP), zero of
in
to express
are
nodes
input
are
layer. Nodes
next
MF
1
layer
Systems
c
(1)
,
and
0’:
of the
nodes
in the
l for
nodes
in
node of
layer
will
m
c
represents
proposed
the
network
layers 2, 3, 4, and 5, be denoted
by ygh
input nodes that represent input linguistic vari~ nodes in this
membership function
layer only layer.
transmit
Each
node
input values
to
is connected
to
Inference
HyFIS: Adaptive Neuro-Fuzzy
of
nodes
those
only
sponding linguistic Layer
membership functions
as
the
with
two
(or variance, 0). Initially
width
a
unity and the membership functions
although
space,
corre-
if any
This
is
connection
(or
centre
this
weight
be used
can
is
layer
the
over
c)
mean,
in this
weight
is available
layer
implemented using
spaced equally
are
terms
fed to the
are
parameters,
knowledge
expert
represent the
to
The input values
membership degrees.
membership functions
Gaussian and
act
layer
calculates
2 which
of
variable.
respective linguistic variables.
of the
the
in this
Nodes
2:
represent the linguistic values
2 which
layer
14
Systems
for
initialisation. The
function
output
to the
node is the
of this
:n-c
yf
the
0
and
c are
label.
linguistic
on
particular MF, membership
Layer
3:
Each
weights AND
with
node in
of the
links
layer are
3 represents set
all the nodes
of the
layer
I j is the set of indices in
layer 3, and yf
a
possible IF-part
unity. The nodes
to
are
=
j
layer
of
membership
referred
are
centre
determined
and maximum
fy?
node
in this
forms
change,
the
as
to
for
as
that
adjacent
centres.
the functions
where
various
input weight represents the
the minimum
operation. Thus,
Hence
exhibiting
Parameters
The
precondition parameters.
(2)
As the values of these parameters
thus
vary,
2
@"£T L,
=
the parameters.
bell-shaped functions
functions
belongs
function:
given membership
where
the input
which
degree to
in this
as
in this
layer form
a
fuzzy rule. The
a
layer perform
fuzzy
the
rule base.
follows.
(3)
1;1Ei§(1/fl,
of the nodes
is the
of
in
layer
2 that
are
output of node i in layer 2.
connected
to
H
yFIS: Adaptive Neuro-Fuzzy Inference
A node
4:
Layer
field
the
of this
rules
4
layer of the
consequences
fuzzy quantisation
of
space
lc in
layer
4 to nodes
factors (CFS)of values.
The
the
initial
The
functions
in
connection
of this
layer
=
the
to
is the
Ik
node
k in
of the rules
weight Layer
5:
layer
inference
connectionist
Each
of indices
set
4.
output method
fuzzy
3 represent rules
The
links
define
the
from
the
fuzzy label activation is
nodes
of the node
supported by
of the links
connecting
conceptually the certainty
when
inferring fuzzy output
of the links
between
layer
3 and
1§g}§(y§1v§;),
of the
engine,
which
to
4
a
nodes these
in
3 that
the the
connected
are
connecting links
avoids
certain
layer
(4)
function
rule-matching
degree represented by
the
as
a
process.
squared
values.
It represents the output
attached
The
integrate
[~1,+l].
Actually
is activated
to
rule
expressed as follows:
are
1/fi where
a
weights wkj
weights
in the interval
layer
layer
membership function
connection
j
In this
represents
corresponding fuzzy
randomly selected
are
this
fuzzy
a
linguistic variables.
variable.
output
an
which
to
node
a
of
fuzzy OR operation
output
same
a11d
fuzzy rules together. The
nodes
the
fully connected.
are
rules
represents the degree all
the
to
15
possible THEN-part
a
layer performs
leading
3 and
layer
4 represents
layer
each node
and
of
in
Systems
to
them
signal. was
centroid
act
Here
used.
as
the
The
constitutes
a
variables
defuzzifier.
Centre
These
A node in this
layer computes
nodes and links a
crisp
of Gravity (COG) or Centre of Area (COA) of
centre
of the system.
defuzziiication
area
the output
signal,
can
scheme, in which the
be simulated
by
E ’!/2Cfu¢Cu¢ :ff
=
lcEI1 -_-
2 Q/za k ‹I;,
5l
Inference
HyF1S: Adaptive Neuro-Fuzzy
I; is the
where
of the nodes
of indices
set
the node l in layer 5 and clk and oy,
the lc in
layer
4. The
nodes
layer
4
in
layer
layer
3 and
connected
are
to
and width
the centroid
linguistic value represented by
output
from
links
Weights of the
4 which
the nodes
unity. Thus the only learnable
are
wkjs between
are
in
respectively
are
of the
membership function
of the
16
Systems
in
layer
network
in the
weights
5 to the
4.
layer
for
Algorithms
HyFIS
4.2
Hybrid
Learning
In this
section
we
present the two-phase hybrid learning scheme for the proposed
HyFIS
model.
In
In
the rules.
learning
phase one, rule finding phase, fuzzy techniques
phase two,
is used to
a
supervised learning
for desired
MF
the
optimally adjust
learning scheme, training data and the desired i.e., the initial be
provided
A
4.2.1
The
size of the term
from
the outside
general learning
of each
set
based
scheme
used to iind
gradient
on
of
descent the
To initiate
outputs.
guessedcoarse
or
are
fuzzy partition,
input /output linguistic variable,
must
world.
for
scheme
training of the HyFlS
adaptive, incremental
the
following procedure outlines
HyFIS
model:
Step
0:
the
lnitialise
neuro-fuzzy model the
scheme, training data and i.e., the size of the
given. selected
Step
1:
The in
Extract
initial
set
term
weights
desired of each
of the links
in the or
HyFIS.
guessedcoarse
input-output between
the
To initiate
of
learning
fuzzy partition,
linguistic variables,
layer
3 and 4
are
are
randomly
[-1,4-ll. a
plrase learning
set
of
method
fuzzy as
rules
from
described
input-output in the
next
data
section.
set
using
the first-
Ib/FIS: Adaptive Neuro-Fuzzy Inference
fuzzy
in the network
the network
3:
Step
incoming
Rule
4.2.2
We consider
The
but
item, data
on
use
of desired
clarify
following
Step
1:
of
three
Divide
if not, then
set
be
not
well
as
add this rule to
on
the
on
old
on
as
performed
depending
rules from numerical
fuzzy
a
set of
fuzzy
data
if
each individual
frequency
of the
proposed by Wang 85 Mendal
y is the
It is
output.
the basic
ideas
input/ output
of the
Neuro-Fuzzy
this method, suppose
(:c§,;v§;y1), (mf,wg;yz)
,...
cases.
one-output
case
A set
This
are
given
where :rl and
this method
of desired
is chosen
methodology.
we
,
to extend
straightforward
(MIMO)
of the
the desired
the structure
To illustrate
environment.
input/output training
rules from
rules to determine
pairs, the simple two-input
and to
already represented
1.
general multi-input-multi-output data
data
of data
input / output data pairs:
inputs and
are
neuro-
Phase
these
system in the HyFIS
rg
set
a
here is to generate
pairs and then
set
the
on
simple and straightforward method
a
is
the
into
stream.
Finding
task
if the rule
update it;
This step may
if necessary.
(1992)for generating fuzzy
a
learning
Repeat from Step
4:
Step
rules
fuzzy
new
structure.
or
data
new
if yes, then
structure,
Apply parameter
available
add
fuzzy rule, check
for each
structure:
if necessary,
and
rules
Update the
2:
Step
17
Systems
input-output
in order
approach
to
to
emphasise
consists
ofthe
steps:
the
membership
input and output functions
the
initial
values
the
membership
associated
of parameters functions
space
are
are
into
fuzzy regions.
with
each
input
set
in such
a
and
way
equally spaced along
After
the number
output
that
the range
are
fixed,
the centres of each
of
input
HyFIS: Adaptive Neuro-Fuzzy Inference
and
variable.
output
given lind
value
a
inference
system
of the
one
membership functions
that
such
do
they
and
overlap
So,
input-output
linguistic
For
assume
instance,
has to choose
one
Step
Generate
2:
of
given
the
fuzzy
data
following
divided
4 shows
five
into
set
from
rules
in different
pairs
first
the
of desired
example where
an
regions. Of
for each data
degree
0.2 in
regions.
For
rnaximum
Fig.
degree:
4 is
data
input/ output data,
o
other
==>
RI: if
example, :ri
Z E. ior
Finally,
in
obtain
one
the we
-
ZE,
then
degrees given
are
(6)
--
4 has
ZE
1 in
4 is
degree 0.8
and
smaller to
a
is the in
SP,
degrees
region
with
assigned to SP and 1:3in
rule
y is
(y)
one
from
one
pair of desired
example,
is SP and :rg is
of
possible.
are
in SP),a:§(0.6in ZE), y1(0.6in ZE)], [r1:}(0.8 :cl
interval
pairs:
m{ in Fig.
Fig.
fuzzy
a
divisions
suppose
inputs and third
(:r1,a:2)are
pair. For example,
for
way
[-1,1],
are
the domain
regions. Second, assign at§,x§,and yi
assigned to
yi) (:1:},a:§;
and y
course,
example,
LP, Similarly, mghas degree
in all other
a
corresponding
given data pairs. First, determine
input-output
numbers
two
output
than
of the
of :r1,:1:2,
(oe, -0.2, 02), (o.4,0; o.4), where
in such
regions and other shapes of membership functions
the domain
for the
regions and assign each region
N
into
Figure
are
intervals
the domain
interval
membership function. of 331, 1132,and y
the intervals
space
fuzzy
overlap from
linguistic variable
the entire
cover
the
variables.
that
each domain
Divide
also
and suiiicient
always
can
manner,
the that
means
we
range,
In this
e.
satisfy
0.5, which
=
transitions
linguistic values of each input and output that
e
operating
,uA(x) 2
smooth
another.
to
in the
inputs
provide
can
label
linguistic
one
of
rr
linguistic label A
a
these
Moreover,
6-completeness (Lee, 1990) with
of
condition
18
Systems
ZE;
H 5/FIS:
Inference
Adaptive Neuro-Fuzzy
19
Systems
y2) ==> [:1cf(0.8inSP),z§(1 in ZE),y2(0.8 in SP)], (w¥,:c§; s
Step
3:
RQ: if
Assign
rules
a
of
number
the
the
pairs
and
rules,
other a
The
is activated
to
if xl
is defined
is A and :ng is
B, then
example, R1
has
=
R2 has
a
delete
has
that
with
y is C
assign
maximum
a
rules
redundant rules
that
are
the
for
weight
rule.
a
degree to
each rule:
The
(wi),
(7)
Ha(931)Mb(?U2lM¢(?Jl-
=
MSPUU1)/1zE(1F2lMzE(Z/l 0.8
>
0 is the
learning rate,
w
(fl
U(-),
r
and the chain
GE __ _
(9)
rule is described
as
follows:
Q 33/,Q awkj
awkj =
?§ @ 6.9;
(10)
_
ay? ay/dlawkj To illustrate
putations
of
%,
the
learning rule
for each parameter,
layer by layer, starting
at
we
the output
shall
describe
nodes, and
we
the will
com-
use
HyFIS: Adaptive Neuro-Fuzzy Inference
parameters
for these
5:
From
widths
and
(0)
adjustable
as
below:
derived
are
Eq.(8) we get, 5E
(dz-l/fl
%§= Hence, the
(c)
centres
computations.
learning rules of each layer
The
Layer
with
membership functions
Gaussian
21
Systems
error
to
be
to the
propagated
(11) is
preceding layer
6E
(12)
5i5= -5gLg=dz-3/?_3.___L._ Gy, 30]/,,
=
80%
as:
(13)
obtain: (y;y;7’§"3’), we
=
4
Zykfffzk’
5
ay:
is derived
0y5
6E
3E ..._
Recalling Eq.(5), yf
(width, 0)
rule of the variance
Using Eq. (10), the adaptive
4
ylk: Clk
k’
u
Z
yzk ZZ?/k’Ulk’Clk’
_
gg
g
2
0m
E
4
?/M71/C’
kr
(Clkyk’0’lk’)y;¢f0zk’Czk’>) (14) 4
yzk
,
,
kf
kf
__
4
_
g
__
_
2
E vim kv
Here
k is the index
layer
5.
of
a
node
Using Eq.(11), and (14), We UE __ in
can
in
layer
write
4 which
is connected
to
a
node
l in
Eq.(13) as
_Eg dyf 59? 301k
aalk
yzk =
*(d¢~y?)
(Zyl/;fUlk’)"" (Zyl/;fUlk’) y;fUlk’Clk’>> y;fUlk’Clk’>> ~~;e ""
»
E yli;’Ul/5’ ki
(15)
HyFISZ Adaptive Neuro~Fuzzy
Hence, the
is
parameter
0
Inference
) (ClkyI :’0 2klkyz’U’lIc’Clk’)) k
U’gk(t)l"
=
22
updated by yut
’l’ 1) 0‘g}¢(t
Systems
~-*W
.
4
Z
yldglk’
kr
Similarly, the adaptive rule of the 3E
6E
is derived
as
63/f
I
5522
(centre, c)
mean
55% y
0
-2/?l*iL£’;~ Um?/pc
-(dz
=
(17)
Z lc
Hence, the
is
parameter
C
updated by ,
1)
Clk(t+
C;k(t)"l’
=
Z
4
Um?/k
k
The
4:
Layer
desired be
for nodes
error
in this
of each node.
outputs and activation
computed
and
Hence
propagated. 54
From
is calculated
layer
0E
Hyii
signals
error
of
need to
0y5
( 19 )
52/?ay/3
Eq.(8) we get, 6E
Eq. (5)
,
=
--
From
fuzzification
_-_+
=
__
the
on
have
we
5E =
Only
based
~~
UI
2
-
get,
We
(ClkL/il;/Ulm) y)i’0’lk’Clk )) (21)
Uzk
6?/5 z
_
kf
=
fe
2
as
.
4D2
4
it//efflk’ kr
Hence, using Eqs. (11)
and
(21), we
D2 the
write
can
signal of Eq. (19)
error
as
out
62 (dz if/F) =
~
(Clk (Zyifvue) (2yiffvutfci/¢f)) (22) gk -
k
H
-
Z H2101/e ki
Ii)/FIS! Adaptive Neuro~Fuzzy Inference
Layer
As in
3:
the
layer 4,
signal
error
signal
error
6?is
need to be
parameters
no
needs
be
to
derived
Systems
computed
If input
rameters
should
from
previous layer,
the
be increased
Using (10) and (2),
directly proportional
because
@
the
_
the to
is caused
error
(23)
the
by
ci is derived
corresponding
as
propagated
these
fl QQ1/2___2( 53/3
parameters.
in the
U?
following:
(24)
,
§§£9_?i
_ _
(25)
_(25) (25)(25) Eq. (23) 6E =
_-T
6,3 and from
error
(3)
25. from
pa-
ay? 5% =
where
The
@%
_
3%
Where from
only
as
adaptive rule of
the
and
propagated backwards.
fuzzy segment, then
lie in the
values
and
layer,
as ay; HyØ5".u§ 3?/ii, HyØ5".u§3?/ii, ag.
=
2:
in this
adjusted
following:
as
as
Layer
23
53 J
we
Eq.(3) and Eq. (23) r
=
Arg
gIا§1(@/3),
(27)
Then, 3E
.
=
z
Q
So the
mean
of the
=
r
(28)
input membership functions 0E
an
’L
otherwise.
0
=
.
f
+
ii
=
Cm
+
2
-i
can
be
updated by
Iii/FIS: Adaptive Neuro-Fuzzy Inference
Similarly, using (10) and (2),
QE
the
24
Systems
adaptive
rule
of
of
is derived
as
i9§?E-ti
_
_
80;
80]
22(;v c,)2 yi 31/3 U? 3E
M
--
(30)
’
(oi)
Hence, the variance
0i(f+1)
all fields eas
TO
dynamical
of science.
of economic
in
can
time-series
in all natural
occur
This
Time
uses
ins gas furnace
combustion
data
process
prediction
and
in the
most
where
of the
ar-
weather
increased as
in-
chaos
is present.
nonlinearity
exciting topics
applications
The
discovery of chaos,
the
living systems
of the two
to
in nonlinear
systems
proposed methodology.
System Identification
a
techniques that permits
based
on
input-output
a
methane-air
mixture.
to
data.
of the
build
mathematical
In this subsection
using the well-known
(Box & Jenkins, 1970). This of
permeates
be found
can
fields.
lots of other
system identification,
to nonlinear
which
generic problem
Box-
Series
dynamical systems
HyFIS
a
of time-series
1-Nonlinear
System identification
the
one
describes
section
Jenkins
els of
is
systems is also related
Example
5.1
DYNAM-
planning, inventory and production control,
and business
Hence, chaos is currently research.
eu
EAR
N ON LIN
modelling
Applications
nonlinear
readily
,2
-
+
forecasting, signal processing, control, and terest
2
become
now
SYSTEMS
ICAL
Non~1inear
8E
me
=
APPLICATION
5
membership functions
of the input
data
During
set
the
was
modwe
apply
Box and Jenkrecorded
process
the
from
a
portion
Inference
HYFISSAdaptive Neuro-Fuzzy
of methane
was
time~series
data
u(t),
rate,
furnace
randomly changed, keeping for
set
We
output.
y(t) given
tration
the
for the
defined in this
each
that
the
(32)
variables
S
are
for the
the
for
CO2
steps before
time
5 shows
HyFIS
consists
schematic
a
4)
-
for
diagram the
where
case
concen-
u(t
fuzzy
(small), M (medium);and
desired
Each domain
L
sets
(large)
In this
alone
nor
nosis
to
a
interval
-
was
4), y(t
divided
that
pairs
into
we
five
converted
were
1); y(t)]
-
specinc problem,
We used the three~step on
292
examples.
pairs and their
fuzzy logic
Fig.
of
data
which
reduces
the MF shown
use
linguistic labels,
and
by VS (very small), S (small), M (medium), L (large),
base, based
input-output the
[u(t), y(t)]
of data points to 292. For this
rule
fuzzy
in
is
identification
an
1). Fig.
-
training data point
(very large).
and VL
as
y(t
of 296
data
fuzzy regions denoted
After
the
as
11/(If),
portion from four
gas
set consists
(ci) in Fig. 8.
the
provide
using
input-output
original
the number
the
y(t),
gas,
case.
The
in
is to
the methane
of identification
process
a
beforehand.
be determined
CO2 concentration
last
and the
model
HyFIS
of the
task
The
is
Design
Experimental
5.1.1
so
and T2 should
T1
=>
-
-
delays
in outlet
the task of the model
that
assume
This
rate.
(the inlet methane)
gas flow
with
process
(UU ri), 1/(if nl) where
gas flow
constant
a
input, and CO2 concentration
the furnace
as
gas furnace
a
25
Systems
rules
have been
of Section
procedure
The
fuzzy
4 to
the
generated from
rules
corresponding degrees are given found, the network
generate
structure
in Table
1.
is established
6.
consider
the situation
desired
input~output
pairs alone
desired
accuracy.
example
we
A combination
where are
of
neither
suflicient
linguistic fuzzy rules for
a
linguistic fuzzy
successful rules
and
prog-
fuzzy
HVF IS: Adaptive Neuro-Fuzzy Inference
rules
generated from
the desired
two
rules.
guistic them
to
there
are
To
do this
rules
no
remaining
92 data
1. In this
illustrated
the
used to construct rule
base,
is established
applied;
200
data
and
was
in
Fig.
as
The
(o).
chosen
network
fuzzy the
to
200 data
and
3)
the
only the
this
fuzzy
rule
level of
to
tune
the
as
the
there
are
only with
HyFIS model is
combined
rules for the >¢=in Table
by
the connections
data
of the
were
by this fuzzy
of the
rule
HyFlS
learning phase
fuzzy rule
bases of 1
and the initialised
optimally
the data 92
the
structure
the
model
remaining
examples
is structured
structure,
of the cases,
the
connections
no
applied; 2)
pairs and the
network
parameters
except that
1
linguistic
1) 200 training
base which
training set,
Table
as
base, and the parameter
the whole
lin-
5.
learning phase
rule
from
comes
rules marked
two
is showed
cases:
performance. For each
points
same
rule base of
only
base and the
fuzzy
desired
=4Withthesefuzzylogicrulesthe With these fuzzy logic rules the
fuzzy
have
from
comes
pairs randomly and used
6 except that
from 92 data
is used to establish
is trained
three
rule
rules
is the
structure
parameter
linguistic with
cases
desired
by
following
the
base of selected
network
outlined
by the empty circles (0) in Fig.
We simulated
2
pairs
the whole
case,
to
of accuracy.
part of the information
selected
firstly
we
the empty circles
by
and
second
is established
illustrated
is
the
in the Table
structure
level
part of the information
generate the fuzzy rule base, which
network
fuzzy
the desired
to
the first
where
cases
pairs, whereas
input-output
pairs is, however, suflicient
Results
Experimental
We consider
26
data
input-output
successfully predict the level of CO2
5.1.2
Systems
were
points
to
partitioned the
as
achieve
test
set
a
in
for
validation.
The 200
experiment
epochs
of
results
training,
root
for the mean
first square
two errors
cases
are
shown
in
of RM SE¢1ain
Fig. =
7.
0.0382
After and
Ii)/FIS! Adaptive Neuro-Fuzzy Inference
RM
SE,est
are
shown
second
in
We
see
for
whereas
of accuracy,
systems dynamics with
clearly from
very
be 3
case
predicted
RMSE
an
these
figures that,
the level of
CO2
for the first
to the
the
successfully identihed
we
desired
and level
nonlinear
desired
0.02()5.
=
of
2-Prediction
Example
5.2
9, respectively. Finally in Fig. 10 and 11, the third
the system cannot
cases
membership functions
and final
initial
The
obtained.
were
8 and
Fig.
described.
was
case
0.0588
=
27
Systems
the
Mackey-G1ass
Chaotic
Dynamics time
The chaotic
series
used in
is
simulation
our
generated by
a
delay differential
equation l
dx(t)
-----
dt
that
was
and fy fixed
aj
of the
function
reported by
Farmer
limit
or
and
cycle,
or
(1982). As
chaotic
is
T
=,U.2,,6
forecast
with
The
Choosing
:r;(t + At)
the fractal
the
systems,
low such
At
Note
>
of that
dimensional.
system is infinite onto
with
dimension
chaos.
dimensional
time
dimensional as
nonlinear
constant
50.
strange
a
of
is
of the as
T
3.5.
as
the
yields
fractal
iixed»point,
chaotic
behaviour,
of
yields
Higher
approximately dimension
attractor
50, which 30
=
is
either
dimension
with
approximately
However,
17
=
fractal
a
Meanwhile,
because
T
T
extensively and
studied
the system exhibits
(Gtt, 1981), with
characteristic
10 leaves
=
Mackey & Glass equation (33)
been
has
(T)
2.1, i.e., :v(t) is quasi-periodic and chaotic of 2.1.
0.1 and fy
=
of the
varied,
behaviour.
(Mackey85 Glass, 1977).Keeping
Glass
behaviour
(33 )
Bw(t )
-
-
delay parameter
attractor
strange
a
at
The
only adjustable parameter. a
1
investigated by Mackey and
first
the parameters
as
crx(t T) -I- x7(t T) -
=
makes a
values
difficult
strange of
T
to
attractor
yield higher
delay, m(t-T), the phase space of this time
attractor.
partial differential
progresses
Other
infinite
the system
collapses
dimensional
chaotic
equations, also display collapse
HyF IS: Adaptive Neuro~Fuzzy Inference
low dimensional
onto
the
Thus,
attractors.
in the
simpler setting of nonlinear, differential
much
more
complicated systems such
A detailed
analysis of the chaotic
(1981) &
Ott
5.2.1
Farmer
is to take
goal
At
where
is
statistics
on
the time
The
fundamental
in
future.
the
a
the
for this
Farber of this
is
vector
number
equations.
At
and
of chaos
dictates
is due
discrete
in
times
is
a
of connectionist
that
changes
past that
will
degrade to create
(D
ac(t+ At). Embedding to
several
At
as
1)A)
-
set
(1989).
problem which
The
At
collect
along It
is increased.
will
decrease
of finite
precision
used for
predicting
is increased.
a (t
,...,
-
D
those
The
points
A),a:(t)),
of time~series
values
in
Lapedes &
of
prediction of future
values
has been also considered
(Lapedes& Farber, 1987; Moody
researchers
At),
+
accuracy.
mapping from
approaches, including
XL Darken
benchmark
a
are
a
on
as
At,
window
accuracy
inaccuracies
in the
-
fix
the
statistics
prediction
is, (:v(t
window
time
some
by sliding
of accuracy
type of prediction is
value
t
again collect
that
in
One may
the future.
inescapable
to
times
accurately predict x(t
to
times
prediction
index
common
series
in
occurs
equation (33) may be found
of
the values
average
(1987), and Moody time
differential
partial
discrete
at
step into
spaced A apart,
series
predicted future
state
time
many
This
:c(t) at
of
Thus, all predictive methods
of the time a
an
nature
method
standard
to
how
is increased.
specifying
properties
use
series, and then increase
be observed
At
t, and
for
accuracy
can
as
of values
set
a
prediction
some
nonlinear
as
that
equations behaviour
Design
less than
times
containing
Mackey-Glass equation (33) exhibits
(1982).
Experimental
The
28
Systems
&
by
a
Darken,
1989; Casdagli, 1989; Crowder, 1990; Weigend, 1990). At
7’
broadband
=
17, :1:(t) appears with
numerous
to
spikes
be
quasi-periodic
due to the
and
the
quasi-periodicity.
spectrum
power
At
T
=
is
30, av(t)is
Inference
H5/FIS: Adaptive Neuro-Fuzzy
steps for
of 1000 time
goal is
to
where
At
is
a
viewed
as
an
m
be
is
+
m
1.
inputs that
17. lf A is
=
f(m(t),:c(t
=
dimensional
+1
take
a
and
the future
A)
-
and
the
Thus, of
a
single
for
time
a
then
integer,
an
span our
function
_
,;c(t
.
_
is
f
a
(34)
mA)),
-
This
map.
embedding dimension node
in the output
:1:(t mA). y(t -
,...,
is
m
-
output
:r(t), :r(t
on
delay,
A), ;z:(t 2A),
-
space.
-1- At) is the
time
a
(t)
time
’versus
to construct
into
time
prediction
y(t
plot of ;v(t)
a
neuro-fuzzy model
the
use
T
ja/(t+ At)
to
12 shows
irregular. Fig.
more
even
29
Systems
is defined
layer,
takes
At)
+
be
may
and
ac
the value
on
me + At). yet specified what
We have not of Takens
(Takens, 1981) states
defined
be, df,
2df + on
of
to
then
choose
A and the time
span
We extracted
1000
of the fractal
if the dimension
that
embedding dimension, de lies
an
1. We therefore
be. An important
and A should
m
de
=
1’
to forecast
want
we
4, for
=
data
input-output
17. Takens
is At
the future
into
which
pairs
consist
no
=
of four
is
attractor
in the range
provides
theorem
df
de
HyFIS
CO 2 UI
SYSTEM
1)SYSTEM2 1)
\
ll(t’4) L
s0r \
f
M
g
wk
Q
;
\&»>-O /f
Q Figure
5: A
ueuro-fuzzy
model
for the
Box-Jenkins
data
set.
FIG URES
48
CQU)
tmnnodes /,
!/, 1/
,’
0’
1
,’/
6* 9
.
’
can
‘\\‘
"" ~
F
one
5 MFS
empty circle
used
nodes) are
‘\‘\\‘
\
of the ueuro-fuzzy model and
added
13 rules to
the
‘_\‘
‘
\
-
‘\ 0
/\
structure are
\\\‘
\
/\
nodes
6: Initialised
when
_
~»
6
0
/\
Figure
‘
|\
0
"’
’
f
/I
0’
Rule
!
GI’ G 6’ §l§’:5"¢7 "5¢ ’
f
!»
,
\
set
A/
/3/ /I
are
structure.
inserted;
/
T for the
later
‘\
‘\
‘o
‘o
‘|
\
two
/
Box-.lenkins more
rules
data
(the
49
FIG URES
(bi
(8) 1
0.3
ee
.
e
c
i
=
""""""‘
0_8
.
,........
,........
Q0_1
E
........_
H.:
._
.:
: ..
Y.
.
,
,_
.
.
....
...
1
:
i
.,._...,._._.
._._.,,_...
.
3 .
1
j.
.,
1 ........._.._
.._,_.,....,.
..‘..........
.....»...‘...
,...........,
,,...
.
_._.,
‘
1
g O ii E a_o_1
i
0.4
Q
,..,,.,._..._
Q)
,,.
I
53
"""’
""
5
g V05
""""""’
.
_..,_
\.,.
..,.....,,...
~o.2
Q
0
7
E O
300
200
100
O
Q
[__
f
time
time
(U)
(C) "
1
0_8
0.3
"
f
4
_6
Q 0_1. ......_._
,.,.,....
30
..
.3
,_
4;
5
5
:
§
O 0
v
~-’--
_____
2
,__.,______
.,....,,...,
7:
the desired
_.
ow..
3
1 ,.,...
___A_
.,.....,.
_._.A..
2,
___________
300
200
1 OO
the
HyFIS
,,.,,_,, V
-
0
A___A______._
for the
2
E 200
300
time
Box-Jenkins
system response
HyFIS prediction (b) pairs (200pairs) only; prediction (dashedline); rules (92 pairs only).
___A.A___A___
100
time
Figure
_‘,___._....,
In :
_OZ
,,
’
-_OZ___A__._.A.. A___A______._ ___A.A___A___
--’~--~---- -
-----~
.
....,...
a____0_1..,.
I
0.2
.....
gy
..
’
~
1
_,__
...,,_.,..
V ___
~
..... . ,__.,______In_‘,___._....,
KA.
........
300
200
100
data
(solid line) error
for
the desired
the
for the first
and
the second
cases:
and the
HyFIS prediction (dashedline); using fuzzy rules from the partitioned data system response
HyFIS prediction
error
(solid line) and for
a
smaller
the
HyFIS fuzzy
set of
FIGURES
(D)
(3)
1
_#_
\
U
F
06
\
’
.
.
’
I
if
0.4
J
1 I
E
E1
l;
0.2
\»
I
v
\
,
\
_-
\
J
O
0.4 \
!,-
,
,,
1
1.5
final:
/F F1 I V -
’
05 ’
I/
0.4
O -0.5
1
f
/fi /_
_f
200
5
\
\ 1
1.5
\
1
0.6
\\
1
/-/
I
/. /
\
1
learned
of the Box
,
0.2
\\
_,¢
0 -0.5
1.5
/
I
\
~¢
/5
,
’
\
i".E
1
0,4
\\
1/
/I ‘\§//Q
/. \
/’\
/"-‘\./‘\ ,\\\f
’
1
\ \
\\i
\‘
\
Q\.\ ~\_ ~\_
/ _.
0
0.5
1
1.5
final: outpul 13 5/(1)
inpu! 2; y(I-1)
and
examples
08
\
0.5 final:
first
f
2
-/
\
uf
it
0
8: Initial
5
[ i 5
A
/
0.2
\\
/’f EX
L/i
/ /
Figure
1
W:
’
\
\
(di
f-"\"}"\
08
‘
’_\
input 1; u(t-4)
is)
1
\
~_\
/
0.5
0
-0.5
u(t~4), y(t-1),and y(t)
\
5
/
_
\
-,
\
_
_
_.
0.2
,
I
_
\
\\
/
_-I F
/
1
\~_
/
/
/
\
’Yu
-‘
.5
/ f
\
0.5
initial:
’
\
V,
ii-
/
06
‘
.
,
\
-0.5
1
-_
I
-
/
\
f.
.
1
0-B
\
il
_-i
,.
.
3
\
f
V
/f’ fi
\
’
5
1
Q
,
’
’;
\
l
I 1
0.8
1
\
\
I
\
I
membership & Jenkins
functions
Data
Set.
for the
inzzy
rules
from
the
FIGURES
51
(U)
(H)
1
_/~>
1
\
\
-’
_»\
\
\
’
0.5
il lui E
0.6
~
~
.’ V1
0.2 O -0.5
,,
,
;\
,
0
,Q \
1
initial:
1.5
0 -0.5
,li
! 1
"l
_-nf
_
,fl \~\.
0
0.5
final:
n
1
0.8
0.6
\
.
If
0
9: Initial
pairs)
E
’
\
2 1,
1 \
’_
\
/I
0.4 \
1
_l
o 2
\
__ ’_
/
1
1.5
‘
/
_I
/
1,
\ 1,
,
,
/
O -0.5
-1
and learned
for the Box
\
1
}
\
\ \/ 4
I
3 3
: :
A
membership functions Data. Set.
\
\ \
1 \
5 »\-’_
3 Y ’>
,.‘.\.,
_
0
for
\
:\
-.‘_,
1
0.5
linal: output 1;
& Jenkins
\
2,1
--
lnput 2; y(t-1)
2
,
\
.’
’
0.6
~
1’ \
/
\
»
’~
1.5
lnput 1; u(t-4)
>\
/I 0.8
\
0.5
final:
92
4
t
\
_,\\1,O-1_,.‘.\.,-.‘_, ,\\
_
-0.5
1
1: "
f
/-
O
I I
1
\
/,
n’"’
’
o 2
\
’ll .Y1’
j _
Figure
I
_
r
I 0.4
|
_
1
ld)
‘
\
1
.
.
(C)
.\’
\
Q4-
,
u(t-4). y(t-1).and y(t)
1
\
1 ‘| I*
0.2
\
,
0.5
\\
_./ »’, ’I
~
0.4
-\&
/,,
,,
\
I-_
,X .f\ \ \‘ A:
0.6
l
I.
/
_-
,
1
1
.
,,
_
0.5 1
1’
1
\
I
0.4
2|
li
V
l
Q
\
a
smaller
1.5
y(l) set of rules
(from
FI G URES
52
(8) 65
(b) 0.045
-
5
08 (D
eo
O
7:
55 55 gf211I
.L
om
X
x)¥
¢‘Z ‹e»
2
0
g ,
gcjgazj?O wt"
_.........
.........,
0
E
1
1
I ..........
2 _
_
_
.._..
E _____;
__
4
0
0_3
,
_____________.
I
5
_._....
_
,_
_,___
.
____
_
_
_
2 .
_____
100
50
___
150
~
i
200
:
5 .Y..,...,....
I
Z 1
5
_
,...........
,..,..........
_
2
3
Q qi
-
’
9
3
I
._......_
.Ui
_
‘
1
505.
......
id) O. 1
W’
Ii
_g
__
-
epoch
’
________
-
‘
002
2
A
_
E
.,........
(C) :
i
..,.......
E
._..._....
u(\-4)
1
_
_..,....,
Z
........,.
0
oO
.
-2
_.__4_.___
E ..,.....
E 5 9 0.025
FS, Ø : §T-{£~_ l _
......_.__
g 2
.__,___‘_.
2
‘
3 gf
___.
¢
._.._.A__
5
gli? 0 035
-
50
:
5
,_____4__.
6
X
>‘
45 -4
1
’
E
.9
I
TQ
0_4
0
-
5\
1oo
E
____
_
_
zoo
E soo
I 400
soo
1
:
:
t
1
.
_
.,..
2
or 0
o
soo
V
3
(C) :
I
t
E
_gps
__
(b)
-
1
2 soo
-
5
3
‘
0
~
~
5
:
1
I
1
..§,.i
;_
a
3
300
400
‘
200
100
O
.-4
_ODS
__
500
O
5
;
2
100
200
300
13: Generalisation
series:
(a) training
RMSE
_
500
(0)
(U)
Figure
; 400
test
of the
curves
HyFIS
model
for the
Mackey
& Glass
time-
for the model; (b) the desired (solid) and preprediction error from 501 to 1000 when 500 train-
(dashed)time series; ing data are used; (d) the desired (solid) and predicted (dashed)time series; (e)
dicted
prediction
error
for data
items
from
501 to
1000
using adapted fuzzy
rule base.
FIGURES
56
(al
1
,-
,
\-
/
\
/1
1-lb)
\
/
/
’
\ /
/ /
_~._i,
/_
,__
0
initial:
\\ ’\.
0.5
\_; O’ 1.5
1
\
f
\
/
\
//-
/
i
H
\
,,"
0.5-
-
.\
/
/’/
,.¢ .
/
\
»~~‘
/
>’f\
/
~
A
I/ 0 -0.5
/
\
_
0.5
,»~
_/
\
A
///I
\’\ .
.
-0.5
0
-._V
.
0.5
1.5
1
final: input 1; x(t-18)
x(l-18), x(t-12), x(t-6), x(l) and x(t+6) --
.
_
.
1-
-
,_
/
.
\_
/f
,
o.s~
’
\
/
O|:,,_/If 0
-0.5
