chapter 5. preferences and value based optimal control

CHAPTER 5. PREFERENCES AND VALUE BASED OPTIMAL CONTROL SECTION: Modeling, stabilization and control of complex biotechnological process

The complexity of the biotechnological systems and their singularities make them difficult objects for control. One of the most important characteristics of biotechnological processes, which make the control design more difficult, is the change of cell population state. A serious obstacle is the existence of noise of non Gaussian type. This type of noise appears in the measurement process as well as in the process of the determination of the structure parameters of the model. But may be the most serious obstacle is provoked by the differences in the rate of changes of the elements of the state vector of the control system. It is difficult to determine the operating regime and their optimal technological parameters because they depend on very complicated technological, ecological or economical market factors. Combined with the strong nonlinearity of the control system of the Monod type this feature of the control system leads to unsatisfactory performance of the control algorithms (Neeleman, 2002). Possible solution in practice is the using of expert estimates. From outside the estimates are expressed by qualitative preferences of the technologist. The preferences themselves are in rank scale and bring the internal indetermination, the uncertainty of the qualitative expression. Our experience is that the human estimation of the process parameters of a cultivation process contains uncertainty at the rate of [10, 30] %. Because of this reason mathematical methods and models from the Utility theory and 157

stochastic programming could be used in biotechnology. These stochastic methods, because of their essence, eliminate the uncertainty and could neutralize the wrong answers if one uses the gambling evaluation approach. Thus we achieve analytical mathematical description of the complex system “Technologist-biotechnological process“. In the modeling process and parameter estimation is used a mathematical approach for elimination of the uncertainty in the DM’s preferences based both on the Utility theory and on the Stochastic programming (Pavlov, 2005, 2008). The algorithmic approach permits exact mathematical evaluation of the optimal specific growth rate of the fed-batch cultivation process according to the DM point of view even though the expert thinking is qualitative and pierced by uncertainty. Such example concerning evaluation of the “best” growth rate of a fed-batch cultivation process is shown below. The specific growth rate of a fed-batch process determines the nominal technological condition (Neeleman, 2002; Pavlov, 2007). Here we use a value-based decision support system shown in figure 5.18. The system permits elimination of the DM’s uncertainty and polynomial approximation of the DM utility. The system is based mathematically on the stochastic recurrent procedures described in chapter 3. The approach permits iterative and precise mathematical evaluation of the “best” specific growth rate of the fed-batch process in agreement with the DM preferences.

158

Fig. 5.18 Utility evaluation

Let Z be the closed interval [0, 0.6] (specific growth rates of the fed-batch process) and P be the convex subset of discrete probability distributions over Z. The expert “preference” relation for P is expressed through  and this is also true for the relations for Z (ZP). As we know, the utility function is defined with precision up to affine transformation (interval scale). We define two sets Au*={(x,y,z)/(u*(x)+(1u*(y))>u*(z)} and Bu*={(x,y,z)/(u*(x)+(1-u*(y))>u*(z)}, where u*(.) is DM’s empirical utility. The growth rates x, y and z are elements of the set Z, and  is a randomly distributed value (). The utility function U(.) is evaluated with 64 learning points and expert answers ((x,y,z,), f(x,y,z,)). The points (x,y,z,) are set with a pseudo random Lp sequence. The seesaw line in figure 5.19 is pattern recognition of Au* and Bu*. This seesaw line recognizes correctly more then 92% of the expert answers. The full pattern recognition is impossible in general, because (Au*Bu*). The proposed procedure and its modifications are machine learning (Aizerman and all, 1970; Pavlov & Andreev, 2013). The computer is taught to have the

159

same preferences as DM. For example, a session with 128 questions (learning points and DM answers ((x,y,z,), f(x,y,z,)))) takes approximately about 45 minutes. The polynomial approximation of the DM’s utility U() is the mathematical expectation, the smooth line in figure 5.20.

Figure 5.19. Pattern recognition of the sets Au* and Bu*

160

Figure 5.20 Utility evaluation of the Specific Growth Rate

The DM utility recognizes correctly more than 81% of the expert answers (training points). The maximum of the utility function determines the “best” set point of the fed-batch cultivation process after the technologist. The control design and the stabilization in the “best” growth rate are based on the Monod and Monod-Wang kinetics models. Among the most-widely used dynamical models are the so called unstructured models, based on mass balance. In these models the biomass is accepted as homogeneous, without internal dynamic. Widely used are models based on the description of the kinetic via the well known equation of Monod or some of its modifications (Neeleman, 2002; Pavlov, 2007, 2008). The use of the classical methods of linear control theory is embarrassed, mainly due to the fact that the noise in the system is not of Gaussian. The changes of the values of the structural parameters of the Monod kinetics models also lead to bad estimates when using Kalman filtering (Kwakernaak & Sivan, 1972). Another serious flaw is that using 161

classical linearization and control solutions via the feeding rate, the linear system is not observable. In addition, the dynamic optimization based on the Pontryagin maximum leads to singular optimal control problems (Gatev, 1978; Gabasov & Kirilova,

1981). The above-mentioned features have led to the development of advanced dynamic models in which the dynamics of the specific groth rate is described by a separate equation. Such is the model of Mono-Wang for which the classical linearization determines a linear observable system (Pavlov, 2007, 2008). The rates of cell growth, sugar consumption, concentration in a yeast fed-batch growth are commonly described as follows: 

S F X X Ks  S V  S F S  km X  (S 0  S ) Ks  S V X  m

(*)



V F

This differential equation is often part of more general and complex dynamic models. Here X is the concentration of the biomass, S is the substrate concentration, V is the volume of the bioreactor. The maximal growth rate is denoted by μm and KS is the coefficient of Michaelis-Menten. With k we denote a constant typical for the corresponding process. The feeding rate is denoted by F. If the process is continuous (F/V) is substituted by the control D, the dilution rate of the biotechnological process and the third equation is dropped off. Often used models are described in the table 1. The first model is the well known Monod type model, the second is the Yerusalimsky model 162

and the third model includes inhibition term in the denominator. The non-observability of the Monod model has led to the development of the widened dynamical models, in which the dynamics of the specific growth rate is described via separate equation in the system of differential equations.

μ

Model

max

1

S ( ks  S )

2

max

S ki ( ks  S ) ( ki  A)

3

max

S ( ks  S  S 2 / ki )

Table 1

The dynamics of the growth rate  in the Monod-Wang model is modeled as a first order lag process with rate constant m, in response to the deviation in the growth rate. This model called also model of Monod-Wang determines a linear observable system in the classical linearization and concerns a fed-batch biotechnological process (Pavlov, 2007, 2008).

163



F X x4  F S   kX  (S 0  S ) V  S   m( m  ) Ks  S X  X 



V F

Obviously model (*) is a singular form of this model. One general description of the fed-batch biotechnological process looks like. 

X  μX 

F X, V



F , V



S   ), ( Ks  S )

S   kX  ( So  S )

  m( μm 

V  F, 

A  k3 μX 

F A. V

Here X denotes the concentration of biomass, [g/l]; S – the concentration of substrate (glucose), [g/l]; V - bioreactor volume, [l]; F – substrate feed rate, [h-1]; S0 – substrate concentration in the feed, [g/l]; max - maximum specific growth rate, [h-1]; KS – saturation constant, [g/l]; k and k3 – yield coefficients, [g/g], m – rate coefficient [-];. The dynamics of  in the Monod-Wang model is modeled as a first order lag process with rate constant m, in response to the deviation in . The last equation describes the production of acetate (A). This equation is dynamically equivalent to the first one after

164

the implementation of a simple transformation (X = (1/k3) A). The system parameters are as follows: m=0.59 [h-1], KS=0.045 [g/l], m=3 [–], S0=100 [g/l], k=1/YS/X,, k=2 [–], k3=1/YA/X, k3=53 [–], ki=50 [–], Fmax= 0.19 [h-1], Vmax=1.5 [ ]. These data described an E. Coli process. The initial values of the state variables are X(0)=0.99; S(0)=0.01; µ(0)=0.1; A(0) = 0,03; V(0)=0.5. Common manifestations of the biotechnological control are some over regulations of the growth rate. Such over regulations are shown in figure 5.21. The over regulations are provoked by the differences in the rate of changes of the elements of the state vector of the control system. This particularity of the biotechnological processes led to search for solutions via new contemporary mathematical methods. In the last decade up to date methods and approaches in the areas of functional analysis, differential geometry and its modern applications in the areas of nonlinear control systems as reduction, equivalent transformation to equivalent systems have been used for surmount the discussed difficulties in the process of the control design.

0.4

Specific Growth Rate

0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

0

2

4

6

8

10

TIME [h]

165

12

14

16

18

Figure 5.21 Fed-batch process and growth rate overregulation in Sliding mode

We preserve the notation U(.) for the DM utility used in the control design. The control design is based on the solution of the following optimal control problem (Gabasov & Kirilova, 1981): Max(U()) for minimal time, where the variable  is the specific growth rate, ([0, max]). Here U() is an objective function ( utility function) and D is the control input (the dilution rate D[0, Dmax]):

max(U (  )),   [0,  max], t  [0, T int ], D  [0, D max] 

X  X  DX 

S  kX  ( So  S ) D 

  m( m

S  ) ( Ks  S )

The differential equation describes a continuous fermentation process. The optimal control is determined with the use of exact linearization to Brunovsky normal form of the Monod-Wang model (Elkin, 1999; Pavlov, 2001, 2004). 

Y1  Y 2 

Y2  Y3 

Y 3  W. In the formula, W denotes the control input. The new state vector is (Y1, Y2, Y3):

166

Y 1  u1 Y 2  u 3(u1  ku12 ) Y 3  u 32 (u1  3ku12  2k 2u13)  m( m

u2  u 3)(u1  ku12 ) ( Ks  u 2)

 X    u  1    S0  S   u 2   ( X , S ,  )   S  u      3     The derivative of the function Y3 determines the interconnection between the input W and and the input D.

u2  u 3)  ( Ks  u 2)  u 33 (1  6ku1  6k 2u12 )(u1  ku12 ) 

W  2u 3(u1  3ku12  2k 2u13 )m( m

 u 3m(1  2ku1)(u1  ku12 )(m

u2  u 3)  ( Ks  u 2)

u2  u 3)  ( Ks  u 2 ) mKs  m(u1  ku12 )   ku3u1( So  u 2)  ( So  u 2) D  ( Ks  u 2)2  m2 (u1  ku12 )(m

The control design is a design based on the Brunovsky normal form and application of the Pontrjagin’s maximum principle step by step for sufficiently small time periods (Gabasov & Kirilova, 1981). The interval T could be the step of discretization of the differential equation solver. The control law has the analytical form:

 6  (T  t )  (1  2kY1 )   Dopt  sign    ici ( i  1)T  t   1  D max ,  2     i 1 where sign (r )  1, r  0, sign (r )  0, r  0. 167

The sum is the derivative of the utility function. The “time-minimization” control is determined from the sign of the utility derivative. The control input is D=Dmax or D=0. The control brings the system back to the set point for minimal time in any case of specific growth rate deviations. The control law of the fed-batch process has the same form because D(t) is replaced with F(t)/V(t). Thus, the feeding rate F(t) takes F(t)=Fmax or F(t)=0, depending on D(t) which takes D=Dmax or D=0. We conclude that the control law bring the system to the set point (optimal growth rate) with a ”time minimization” control, starting from any deviation of the specific growth rate. This kind of chattering control is shown on figure 5.22.

Figure 5.22 Chattering control as „time minimization” Thus, full control law is combination of different control regimes:

168



Time interval - [0, t1]: the control is a “time-minimization” control, where

μ(t1)=(x30-ε), ε>0, x30 is determined by the max(U()) and ε is a sufficiently small value (The input Dmax is replaced with Fmax); 

Time interval - [t1, t2]: the control law is F=0 (μ(t1)=(x30-ε), μ(t2)=x30 and

d/dt(μ(t2))=0 (to be avoid the over-regulation); 

After the moment t2 the control is again the chattering control or some

Sliding mode control. The full control and the optimal process stabilization is shown on figure 5.23. After the stabilization in the “best” growth rate position the system can be maintained around the optimal parameters with a sliding mode control (Pavlov, 2007, 2008). The most difficult is the determination of moment t1 and moment t2. The determination of t1 needs resolution of a transcendent equation. 0.5 OPTIMAL CONTROL

0.45

Second Order Sliding Mode Control

GROWTH RATE

0.4 0.35 0.3 0.25

Specific Growth Rate

0.2 0.15

Equivalent Sliding Mode Control

0.1 0.05 0 0

2

4

6 8 TIME [h]

10

12

Fig.5.23 . Control and stabilization of a fed-batch process.

169

14

We propose an approximation of moment t1 determined by the moment when the state vector of the system across a manifold (figure 5.23): 0.4 0.35

Growth Rate

0.3 0.25

t2

0.2 0.15

CONTROL F

t1 0.1 0.05 0

0

0.1

0.2

0.3

0.4

0.5

0.6

TIME [h]

Figure 5.24 Optimal growth rate profile. The manifold is presented by the formula:

Manifold ( X , S ,  , e, Ks )  0 , ( X , S ,  , e, Ks )   e  exp( f (t1)) (t1)   Ksf (t1) Ks  S (t1)   mm  f (t1)  ln  ( S e  S (t 1)) ( Ks  S e)    mf (t1)  1 exp( f (t1))   mm2   , m2  m2

170

0.7

0.8

  ( e   (t1))  X (t1)  S (t 1)  Se   ln( ) ( Se  S (t1))   m X (t1)  . where f (t1)         Ks  S (t 1)  m  Se  S (t1)  Ks ln     ( Ks  S e)      This problems is also solved as a of stochastic control problem. The solution is shown in figure 5.24. For this purpose, Kalman filtration was used to determine the conditional mathematical expectation of the model state space vector and consequent optimal control over the scheme described above.

0.45

Kalman Filtering and Optimal control

0.4

Growth Rate

0.35 0.3 0.25 0.2 0.15

Sliding Mode Control

0.1 0.05 0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Time [min]

Figure 6.25 Growth rate stochastic controls

The solution considered here is not optimal in the true mathematical sense, but is a combination of analytical techniques that lead to an engineering sub-optimal approach. Kalman filtration coupled with quadratic criteria is optimal control only for linear systems. Here we use „time minimization” optimal law to reach the operating 171

growth rate point only when deviating from it. At the end we are going to summarize. By now we have reached the mathematical description of the complex process “technologist, fed-batch process”. Utility analytical descriptions have been built concerning the attitude of the technologist toward the dynamical process. Using this approach factors as ecology, financial perspective, social effect can be taken into account. They could be included through the expert preferences in the utility function via the expert attitude towards them. We have overcome the restrictions connected with the observability of the Mono kinetics and the obstacles connected with the singularities of the optimal control via finding and using the Brunovsky normal form of the differential equations. The system is moved smoothly to the working point in the equivalent control regime of sliding mode and stabilized there. It can be seen that solution for complex processes is attained in practice by the synchronous use of several mathematical approaches, which are combined in synergy to achieve the main objective of the research. In general this is characteristic of every practical task.

172