A Neural Network Model for Solving Nonlinear Optimization Problems ...

A Neural Network Model for Solving Nonlinear Optimization Problems with Real-Time Applications Alaeddin Malek and Maryam Yashtini Department of Mathematics, Tarbiat Modares University, Tehran 14115-175, Iran {mala,m_yashtini}@modares.ac.ir

Abstract. A new neural network model is proposed for solving nonlinear optimization problems with a general form of linear constraints. Linear constraints, which may include equality, inequality and bound constraints, are considered to cover the need for engineering applications. By employing this new model in image fusion algorithm, an optimal fusion vector is exploited to enhance the quality of fused images efficiently. The stability and convergence analysis of the novel model are proved in details. The simulation examples are used to demonstrate the validity of the proposed model. Keywords: Neural network model, Optimization, Real-time applications, Stability analysis.

1 Introduction Neural networks have been extensively studied over the past few decades and have found applications in variety of areas such as associative memory, moving object speed detection, image and signal processing, and pattern recognition. In many applications, real-time solutions are usually imperative [1,2,3]. These applications strongly depend on the dynamic behavior of the networks. In the recent years, due to the in-depth research in neural networks, numerous dynamic solvers based on neural networks have been developed and investigated [4], [5], [6-17]. Specially, in the past two decades, various neural network models have been developed for solving the linearly constrained nonlinear optimization problems, e.g., those based on the penalty parameter method [7], the Lagrange method [9], the gradient and projected method [5], [13], the primal-dual method [12], [14], and the dual method [4], [15]. Malek and his coauthors in reference [19] presented a recurrent neural network for solving linear and quadratic optimization problems. Their network is shown to be globally convergent to an exact optimal solution of the linear or quadratic optimization problems. Their network is not suitable for solving nonlinear optimization problems. In this paper, we propose a one-layer neural network for solving nonlinear optimization problems with general linear constraints. In particular, since simple structure and global stability are the most desirable dynamic properties of the neural networks, our motivation of this study is mainly focused on developing a neural network with these properties adequate for solving nonlinear realtime optimization problems. Another objective of this paper is to concern with the real W. Yu, H. He, and N. Zhang (Eds.): ISNN 2009, Part III, LNCS 5553, pp. 98–108, 2009. © Springer-Verlag Berlin Heidelberg 2009

A Neural Network Model for Solving Nonlinear Optimization Problems

99

time application of the proposed neural network model in image fusion algorithm to restore noisy images. Theoretical aspects and the illustrative examples further show the effectiveness and applicability of the proposed model.

2 Artificial Neural Network Formulation In this section, we describe the nonlinear programming problem and discuss its equivalent formulation. Then we will propose a recurrent neural network to solve this problem. Consider the following nonlinear programming problem subject to general linear constraints: Minimize f (x) subject to

Bx = b, Ax ≤ d , l ≤ x ≤ h ,

(1)

n f (x) is a continuously differentiable and convex from R to R,

where

B ∈ R m×n , A ∈ Rr ×n , x, l, h ∈ R n , b ∈ R m and d ∈ R r . According to the Karush-Kuhn-Tucker (KKT) conditions and well known projection *

theorem [20], we see that x is optimal solution for problem (1) if and only if there exist y * ∈ R r and w * ∈ R m such that (( x* )T , ( y* )T , ( w* )T )T satisfies the following conditions: ⎧ x = g1 ( x − α (∇f ( x) + AT y − BT w)) ⎪ ⎨ y = g 2 ( y + α Ax − α d ) ⎪ Bx = b, ⎩

g 2 ( y ) = [ g 2 ( y1 ), K , g 2 ( y m )]T , g2 ( yi ) = max{0, yi } , g 1 ( x ) = [ g 1 ( x1 ), K , g 1 ( x n )] T , where for i = 1, K , n xi < l i ⎧l i ⎪ g 1 ( x i ) = ⎨ xi l i ≤ x i ≤ h i ⎪h xi > h i . ⎩ i

where

α

is

positive

constant,

(2)

in

which

The Eq. (2) can be equivalently written as T T ⎛ x ⎞ ⎛⎜ g 1 ( x − α (∇f ( x) + A y − B w)) ⎞⎟ ⎜ ⎟ ⎟. g 2 ( y + α Ax − αd ) ⎜y⎟=⎜ ⎟⎟ ⎜ w ⎟ ⎜⎜ w − α ( Bx − b) ⎝ ⎠ ⎝ ⎠

(3)

Based on Eq. (3), we propose a artificial neural network for solving problem (1), with the following dynamical equation: ⎛ − x + g1 ( x − α (∇f ( x) + AT y − BT w)) ⎞ ⎛x ⎞ ⎟ ⎜ dz d ⎜ ⎟ ⎟ = λ H ( z ). = ⎜ y ⎟ =λ⎜ − y + g 2 ( y + α Ax − αd ) dt dt ⎜ ⎟ ⎟⎟ ⎜⎜ w ( Bx b ) − α − ⎝ ⎠ ⎠ ⎝

(4)

where λ > 0 is a positive constant and , z = ( x T , y T , w T ) T ∈ R n + m + r is a state vector. The simplified architecture of the artificial neural network (4) is shown in Fig. 1.

100

A. Malek and M. Yashtini

Fig .1. Depict of simplified architecture of the artificial neural network (4)

3 Theoretical Aspects In this section, we will study some properties of the artificial neural network (4). Thorough this section we assume that ∇ 2 f ( x ) is positive definite on X = { x ∈ R n | l ≤ x ≤ h}.

Lemma 1. For any initial point z 0 = (( x 0 )T , ( y 0 )T , ( w0 )T )T ∈ X × R+m × R r , there exists a unique continuous solution z (t ) = ( x (t ) T , y (t ) T , w(t ) T ) T for the artificial neural network (4). Proof. Since the projection operators g1 and g 2 are locally Lipchitz continuous,

H ( z ) is also locally Lipschitz continuous. Thus, According to the local existence and uniqueness theorem of ordinary differential equations [21], there exists a unique continuous solution z (t ) = ( x (t ), y (t ), w(t )) T for (t0 , T ). We will show that z (t ) is bounded and the local existence for solution of (4) can be extended to global existence. Theorem 1. Let z (t ) be the state trajectory of (4) with the initial point z 0 ∈ X × R+m × R r . Then the proposed artificial neural network of (4) is stable in the

Lyaponov sense and globally convergent to the stationary point z * = ( x * , y * , w * ),

where x * is the optimal solution of problem (1). Proof. We define the following Energy function: V ( z , z * ) = −W ( z ) T H ( z ) − 12 H ( z )

2

+ 12 z − z *

2

A Neural Network Model for Solving Nonlinear Optimization Problems

101

where ⎛ ∇f ( x ) + AT y − BT w ⎞ ⎜ ⎟ ⎟. W ( z) = ⎜ − Ax + d ⎜⎜ ⎟⎟ Bx − b ⎝ ⎠

z * is an equilibrium point of artificial neural network (4).

Let Sˆ ⊆ R n+ m+ r be a neighborhood of z * . We show that V ( z, z * ) is a Lyapunov function proper to neural network (4). By the results give in [22], we know that − W ( z) T H ( z) ≥ H ( z)

2

(5)

( H ( z ) + z − z * ) T (− H ( z ) − W ( z )) ≥ 0.

(6)

It is obvious that V ( z , z * ) ≥ 12 || z − z * || 2 , and for all z ∈ Sˆ \ {z * }, V ( z , z * ) > 0. * In the following, we show that dV ( z, z ) ≤ 0. Since dV = ∇V ( z (t ), z * ) T dz (t ) , then dt dt dt from theorem 3.2 of [23], we know that ∇V ( z , z * ) = W ( z ) − (∇W ( z ) − I ) H ( z ) + z − z *

where ∇W (z ) denotes the Jacobian matrix of W . Then dV ( z, z * ) = (W ( z ) − (∇W ( z ) − I ) H ( z ) + z − z * ) T H ( z ), dt = (W ( z ) + z − z * ) T H ( z ) + H ( z )

2

− H ( z ) T ∇W ( z ) H ( z ) .

From (5) we can write (W ( z ) + z − z * ) T H ( z ) ≤ −( z − z * ) T W ( z )− || H ( z ) || 2 . Thus dV ( z , z * ) ≤ − ( z − z * ) T W ( z ) − H ( z ) T ∇W ( z ) H ( z ) dt

Since

∇W (z )

is

a positive H ( z ) ∇W ( z ) H ( z ) ≥ 0 . So

semi-definite

matrix,

( z − z * )T W ( z) ≥ 0

(7) and

T

dV ( z , z * ) ≤ − ( z − z * ) T W ( z ) − H ( z ) T ∇W ( z ) H ( z ) ≤ 0. dt

(8)

Therefore, the function V ( z , z * ) is an Energy function suitable for (4). From (8), V ( z , z * ) is monotonically nonincreasing for all t ≥ t0 . It is easy to see that φ = {z ∈ R n + m + r | V ( z , z * ) ≤ V ( z 0 , z* )} is bounded since V ( z 0 , z * ) ≥ V ( z , z * ) ≥ 12 || H ( z ) || 2 + 12 || z − z * || 2 ≥ 12 || z − z * || 2 ≥ 0,

therefore T = ∞. Because V ( z , z * ) is radially bounded, for any initial point z 0 ∈ X × R+m × R r , there Exists a convergent subsequence {z (t k )} such that lim k → ∞ z (t k ) = zˆ, where dV ( zˆ, z * ) * = 0. It can be seen that dV ( zˆ, z ) dt = 0 implies dt

102

A. Malek and M. Yashtini

( zˆ − z * ) T W ( zˆ ) + H ( zˆ ) T ∇W ( zˆ ) H ( zˆ ) = 0.

(9)

H ( zˆ) T ∇W ( zˆ) H ( zˆ) = 0, ( z − z * ) T W ( zˆ ) ≥ 0, (W ( zˆ ) − W ( z * ))T ( zˆ − z * ) = 0.

(10)

That is,

Let zˆ = ( xˆ , yˆ , wˆ ) ∈ R n + m + r . Then equality H ( zˆ) T ∇W ( zˆ) H ( zˆ) = 0 implies that [ g1 ( xˆ − ∇f ( xˆ ) − AT yˆ + B T zˆ ) − xˆ ]T ∇ 2 f ( xˆ ) ×[ g1 ( xˆ − ∇f ( xˆ ) − AT yˆ + B T zˆ ) − xˆ ] = 0.

The positive-definiteness of ∇ 2 f ( xˆ ) implies that [ g 1 ( xˆ − F ( xˆ ) − A T yˆ + B T zˆ ) − xˆ ] = 0 .

(11)

Now, from (W ( zˆ ) − W ( z* ))T ( zˆ − z* ) = 0 we can write (∇f ( xˆ ) − ∇f ( x * )) T ( xˆ − x * ) = ( x − x * ) T ∇ 2 f ( x μ ) ( xˆ − x * ) = 0

where xμ = (1 − μ) xˆ + μ x* for all 0 ≤ μ ≤ 1 . It follows that xˆ = x * , thus

Bxˆ − b = 0 .

(12)

From ( z − z * )T W ( zˆ ) ≥ 0 we can get ( xˆ − x * ) T (∇ f ( xˆ ) − A T yˆ − B T wˆ ) + ( yˆ − y * ) T ( Axˆ − d ) + ( wˆ − w * ) T ( Bxˆ − b ) = 0. Since xˆ = x* , it is equivalently written as ( yˆ − y * ) T (− Axˆ + d ) = 0. Then

yˆ T (− Axˆ + d ) = ( y* )T ( − Axˆ + d ) = ( y* )T (− Ax* + d ) = 0. Furthermore yˆ T (− Axˆ + d ) = 0, yˆ ≥ 0 and − Axˆ + d ≥ 0 if and only if

g 2 ( yˆ + Axˆ − b) − yˆ = 0.

(13)

ˆ ) satisfies in Eq. (2). This means Thus from Eqs. (11)-(13), the point zˆ = ( xˆ, yˆ , w

zˆ

is an equilibrium point of artificial neural network (4). Now we consider another function Vˆ ( z , zˆ ) = −W ( z ) T H ( z ) − 12 H ( z )

2

+ 12 z − zˆ

2

ˆ where zˆ = ( xˆ , yˆ , wˆ ). Similar to the previous analysis, we have dV ( z , zˆ ) ≤ 0 and dt

lim k →∞ Vˆ ( z (t k ), zˆ) = 0. So, for ∀ε > 0 there exists q > 0 such that when

tk ≥ tq

we

have Vˆ ( z (t k ), zˆ ) < ε 2 / 2 . Since Vˆ ( z (t ), zˆ ) decreases as t → ∞ for t ≥ t q , || z (t ) − zˆ ||≤ 2Vˆ ( z (t ), zˆ) ≤ 2Vˆ ( z (tk ), zˆ) < ε. Then lim t → ∞ z (t ) = zˆ.

Therefore, the state trajectory of the proposed neural network is globally convergent to an equilibrium point of (4).

A Neural Network Model for Solving Nonlinear Optimization Problems

103

4 Real-Time Application The proposed artificial neural network in (4) can be applied for many real-time optimization problems. For example, this model can be used to support vector machines for classification and regression and to robot motion control in real-time Here, we apply the proposed artificial neural network (4) to increase the useful information content of images and improve the quality of the noisy images. The neural fusion algorithms are proposed in details for noisy images. Consider an array of n sensor. Let I l (k ) denote the received two dimensional images with M × N gray-level from the l th sensor. Let its amplitude is denoted by fl (i, j ), which

I l ((i − 1) N + j ) = f l (i , j ),

(i = 1, K , M ; j = 1, K , N ).

(14)

The images consist of the desired image s (k ), scaling coefficient al , and the measured noise nˆ l (k ). Then the

n − dimensional vector of information received

from n sensors is given by I (k ) = a s ( k ) + nˆ ( k ), where

a = [a1 , ..., an ]T , I (k ) = [ I1 (k ),..., I n (k )]T , and nˆ (k ) = [ nˆ1 (k ), ..., nˆ n (k )]T . According to the result discussed in [18], the considered image fusion problem can formulated as a deterministic quadratic programming problem

min

f ( w) = x T Rx

where a = [1, K, 1]T ∈ ℜn , R =

subject to aT x = 1, x ≥ 0

(15)

MN

1 MN

∑ I (k ) I (k )

T

(sample variance matrix) and

k =1

x = [ x1 , K , x n ]T is called the fusion vector. Based on the proposed artificial neural network (4) for solving (1), we propose the following artificial neural network for solving (15), which its state equation

d dt

⎛ x ⎞ ⎛⎜ ⎜⎜ ⎟⎟ = ⎝ w ⎠ ⎜⎝

− x + g 1 ( x − Rx + a T w) ⎞ ⎟. ⎟ − aT x +1 ⎠

(16) *

The artificial neural network (16) converges to optimal fusion vector x . The output equation is

s * (k ) =

n

∑x

* l

I l (k ),

(17)

l =1

After increasing the useful information content of images by using Eq. (17), it is time *

to see the fused image. First we have to convert s to f * as bellows

f * (i, j ) = s * ((i − 1) N + j ), then use the function imshow( f * ) in Matlab.

(i = 1, K , M ; j = 1, K , N )

104

A. Malek and M. Yashtini

One can be improve the quality of fused images using the proposed neural image fusion algorithm by increasing the number of sensors. Theorem 2. Let R be a positive definite matrix, then the artificial neural network defined in (16) converges to optimal fusion vector ( x * , w * ) ∈ ℜ K +1 .

5 Simulation Examples In this section, two examples are provided to illustrate both the theoretical results achieved in Sections 3 and 4 and the simulation performance of artificial neural networks (4) and (16). The simulations are conducted in matlab and 4th order RungeKutta technique is used for implementation. Example 1. Consider the following nonlinear programming problem

Minimize 1.05 x12 + x 22 + x 32 + x 42 − 4 x1 x 2 − 2 x 2 − x 4 , ⎧2 x1 + x 2 + x 3 + 4 x 4 = 7

Subject to ⎪2 x + 2 x + 2 x = 6, ⎨ 1 2 4

(18)

⎪ ⎩ x1 ≥ 1.5, x 2 ≥ 0.5, x 3 ≥ 1.5, x 4 ≥ 1.

This problem has a unique optimal solution x * = [ 2.5, 0, 0, 0.5]T . We solve (18) by using the proposed artificial neural network (4). The simulation results show that the proposed neural network (4) is always convergent to an equilibrium point of (18) and *

its output trajectory is always exponentially convergent to x . For example, let λ = 10 , Figs 2 and 3 display the convergence of behavior z(t ) and || x (t ) − x* ||2 based on feasible primal and arbitrary dual initial point. The convergence of behavior z (t ) and || x(t ) − x* ||2 is displayed based on infeasible primal and arbitrary dual initial point in Figs 4 and 5.

Fig. 2. Example 1: Display of transient behavior of the artificial neural network (4) with initial

point z = ( 1, 1, 0,1, − 1, − 1) T 0

A Neural Network Model for Solving Nonlinear Optimization Problems

105

Fig. 3. Example 1: Display of transient behavior of the norm || x(t ) − x* ||2 based on artificial neural network (4) with initial point z 0 = ( 1, 1, 0,1, − 1, − 1) T

Fig. 4. Example 1: Display of transient behavior of the artificial neural network (4) with initial

point z 0 = ( 3, 1, 2, 1, − 1, − 1) T

Fig. 5. Example 1: Display of transient behavior of the norm || x(t ) − x* ||2 based on artificial

neural network (4) with initial point z 0 = ( 3, 1, 2, 1, − 1, − 1)T

Example 2. This example investigates the performance of the proposed artificial neural network (16) to a neural image fusion algorithm. The proposed neural image fusion algorithm is applied to the Lena image shown in Fig. 6. It is an eight-bit

106

A. Malek and M. Yashtini

gray-level image with 206 by 245 pixels. Fig. 6(a) is a noisy Lena image measured by one sensor, where its SNR is 9dB. Figs. 6(b)-(d) are fused images by the proposed algorithm for the number of sensors n = 10, 20 and 30, respectively. Apparently, the quality of the fused image shown in Fig. 6(b)-(d) is improving as n increases. Table 1 shows that with a fix number of iterations, the quality of images improve when the number of sensor increases.

(a )

(b)

(c)

(d )

Fig. 6. Example 2: Lena Image fusion using the neural fusion algorithm (a) The noisy image. (b)-(d) The fused image with n = 10, 20, 30. Table 1. Display of CPU times and the error s( k ) − s * (k )

for gray level fused images with 2

10, 20 and 30 sensors. A fixed number of iterations are used in neural image fusion algorithm. Number of sensors

Number of iterations

CPU time

2-norm

error

10

7

23.5340

6.1297 × 10

−4

20

7

37.3010

4.0075 × 10

−4

30

7

43.0930

2.5700 × 10

−4

s (k ) − s (k ) *

2

6 Concluding Remarks We have proposed a recurrent neural network model for solving constrained nonlinear optimization problems. We have proved that the proposed model is Lyapunov stable and converges globally to unique optimal solution of the constrained nonlinear optimization problems. Moreover, the new model is amenable to parallel implementation.

A Neural Network Model for Solving Nonlinear Optimization Problems

107

We have proposed a neural fusion algorithm based on the proposed neural network model. This algorithm is simple for implementation. Numerical results show a good agreement with theoretical aspects.

References 1. Kalouptisidis, N.: Signal Processing Systems, Theory and design. Wiley, New York (1997) 2. Youshikawa, T.: Foundations of Robotics: Analysis and Control. MIT Press, Cambridge (1990) 3. Cichocki, A., Unbehauen, R.: Neural Networks for Optimization and Signal Processing. Wiley, England (1993) 4. Xia, X., Wang, J.: A Dual Neural Network for Kinemic Control of Redundant Robot Manipulators. IEEE Trans. Systems Man. Cybernet. 31, 147–154 (2001) 5. Bouzerdoum, A., Pattison, T.R.: Neural Network for Quadratic Optimization with Bound Constraints. IEEE Trans. Neural Networks 4, 293–304 (1993) 6. Tank, D.W., Hopfield, J.J.: Simple Neural Network Optimization Networks: an A/D Converter Signal Decision Network, and Linear Programming Circuit. IEEE Trans. Circ. Syst. 33, 533–542 (1986) 7. Kennedy, M.P., Chua, L.O.: Neural Networks for Nonlinear Programming. IEEE. Trans. Circuits Systems 35, 554–562 (1998) 8. Xia, Y., Wang, J.: Recurrent Neural Networks for Optimization: the State of Art. In: Medsker, L.R., Jain, L.C. (eds.), vol. 4. CRC Press, New York (2000) 9. Zhang, S., Constantinides, A.G.: Lagrange programming neural networks. IEEE Trans. Circuits Systems 39, 441–452 (1992) 10. Sudharsanan, S., Sundareshan, M.: Exponential Stability and Systematic Synthesis of a Neural Network for Quadratic Minimization. Neural Networks 4, 599–613 (1991) 11. Wang, J.: A Deterministic Annealing Neural Network for Convex Programming. Neural Networks 7, 629–641 (1994) 12. Xia, Y.: A New Neural Network for Solving Linear and Quadratic Programming Problems. IEEE Trans. Neural Networks 7, 1544–1547 (1996) 13. Liang, X.B., Wang, J.: A Recurrent Neural Network for Nonlinear Optimization with a Continuously Differentiable Objective Function and Bounded Constraints. IEEE Trans. Neural Networks 11, 1251–1262 (2000) 14. Tao, Q., Cao, G.D., Xue, M.S., Qiao, H.: A High Performance Neural Network for Solving Nonlinear Programming Problems with Hybrid Constraints. Phys. Lett. A 288, 88–94 (2001) 15. Xia, Y., Wang, J.: A General Methodology for Designing Globally Convergent Optimization Neural Networks. IEEE Trans. Neural Networks 9, 1331–1343 (1998) 16. Yashtini, M., Malek, A.: Solving Complementarity and Variational Inequalities Problems Using Neural Networks. Appl. Math. Comput. 190, 216–230 (2007) 17. Yashtini, M., Malek, A.: A Discrete-time Neural Network for Solving Nonlinear Convex Problems with Hybrid Constraints. Appl. Math. Comput. 195, 576–584 (2008) 18. Malek, A., Yashtini, M.: Image Fusion Algorithms for Color and Gray Level Images Based on LCLS Method and Novel Artificial Neural Network. Neurocomputing (submitted)

108

A. Malek and M. Yashtini

19. Oskoei, H.G., Malek, A., Ahmadi, A.: Novel Artificial Neural Network with Simulation Aspects for Solving Linear and Quadratic Programming Problems. Comp. Math. Appl. 53, 1439–1454 (2007) 20. Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation, Numerical Methods. Prentice-Hall, Englewood Cliffs (1989) 21. Miller, R.K., Michel, A.N.: Ordinary Differential Equations. In: Neural Network. Academic, San Diego (1982) 22. Pang, J.S.: A Posteriori Error Bounds for the Linearly-constrained Variational Inequality Problem. Math. Oper. Res. 12, 474–484 (1987)