Applied Mathematical Sciences, Vol. 6, 2012, no. 114, 5667 - 5676
An Implicit Multi-Step Diagonal Secant-Type Method for Solving Large-Scale Systems of Nonlinear Equations 1
2
M. Y. Waziri,
2
Z. A. Majid and
3
H. Aisha
Institute for Mathematical Research, Universiti Putra Malaysia 43400 Serdang, Selangor, Malaysia 1,2
Department of Mathematics, Faculty of Science Universiti Putra Malaysia 43400 Serdang, Malaysia 1,3
Department of Mathematical Sciences Faculty of Science, Bayero University Kano, Nigeria
[email protected] Abstract This paper presents an improved diagonal Secant-like method using two-step approach for solving large scale systems of nonlinear equations. In this scheme, instead of using direct updating matrix in every iteration to construct the interpolation curves, we chose to use an implicit updating approach to obtain an enhanced approximation of the Jacobian matrix which only requires a vector storage. The fact that the proposed method solves systems of nonlinear equations without the cost of storing the weighted matrix can be considered as a clear advantage of this method over some variants of Newton’s methods.
Mathematics Subject Classification: 65H11, 65K05 Keywords: Diagonal, Newton’s Method
1
Introduction
Let us consider the systems of nonlinear equations F (x) = 0,
(1)
where F : Rn → Rn is a nonlinear mapping. Often, the mapping, F is assumed to satisfying the following assumptions: A1. There exists an x∗ ∈ Rn s.t F (x∗ ) = 0;
5668
M. Y. Waziri, Z. A. Majid and H. Aisha
A2. F is a continuously differentiable mapping in a neighborhood of x∗ ; A3. F (x∗ ) is invertible. The prominent method for finding the solution to (1), is the Newton’s method which generates a sequence of iterates {xk } from a given initial guess x0 via xk+1 = xk − (F (xk ))−1 F (xk ),
(2)
where k = 0, 1, 2 . . . . Nevertheless, Newton’s method requires the computation of the matrix which entails the first-order derivatives of the systems. In practice, computations of some functions derivatives are quite costly and sometime they are not available or could not be done precisely. In this case Newton’s method cannot be applied directly. Moreover, some significant efforts have been made in order to eliminate the well known shortcomings of Newton’s method for solving systems of nonlinear equations, particularly large-scale systems. For example Chord’s Newton’s method, inexact Newton’s method, quasi-Newton’s method, e.t.c. (see for e.g.[1],[3],[16],[6],[9]). On the other hand, most of these variants of Newton’s method still have some shortcomings as Newton’s counterpart. For example Broyden’s method and Chord’s Newton’s method need to store an n×n matrix and their floating points operations are O(n2 ). To do away with these disadvantages, a single step diagonally Newton’s method has been suggested by Leong et al.[5] and showed that their approach is significantly cheaper than Newton’s method and some of its variants. To incorporate the higher order accuracy in approximating the Jacobian matrix, Waziri et al. [7] proposed a two step approach to develop the diagonal updating scheme Dk+1 that will satisfy the weak secant equation ρTk Dk+1 ρk = ρTk μk ,
(3)
where ρk = sk − αk sk−1 and μk = yk − αk yk−1. For this purpose, equation 3 is obtained by means of constructing the interpolation quadratic curves x(ν) and y(ν), where x(ν) interpolates the iterates xk−1 xk and xk+1 , and y(ν) interpolates the functions Fk−1 , Fk and Fk+1 . A diagonal updating matrix Dk+1 that satisfies 3 is obtained via the following relation Dk+1 = Dk +
(ρTk μk − ρTk Dk ρk ) Hk , T r(Hk2)
(4)
(1) (2) (n) (i) where Hk = diag((ρk )2 , (ρk )2 , . . . , (ρk )2 ) , ni=1 (ρk )4 = T r(Hk2 ) and T r is the trace operation. In order to find the value of ρ, let us consider the following approach by defining the metric as Zm {Z T MZ}. (5)
5669
Implicit multi-step diagonal secant-type method
By letting ν2 = 0 and M = Dk , ν0 and ν1 can be by computed as follows [7]: −ν1 = ν2 − ν1 = x(ν2 ) − x(ν1 )Dk = xk+1 − xk Dk = sk Dk 1
= (sTk Dk sk ) 2 ,
(6)
and −ν0 = ν2 − ν0 = x(ν2 ) − x(ν0 )Dk = xk+1 − xk−1 Dk = sk + sk−1 Dk 1
= ((sk + sk−1 )T Dk (sk + sk−1 )) 2 .
(7)
Waziri et al. [7] incorporated the attribute of the multi-step approach to improve over the single step method. Moreover, they use diagonal updating matrix Dk to represent the weighted matrix that is used to parameterize the interpolating polynomials in every iteration. The numerical performance of the multi-step approach is very encouraging. Based on this fact, it is interesting to present an approach which will improve further the Jacobian approximation in diagonal form. This is what lead to the idea of this paper which is organized as follows: the next section present the details of the proposed method. Some numerical results are reported in Section 3. Finally, conclusions are made in Section 4.
2
Derivation process of IMDS update
A new two-step diagonal secant-like method for solving large-scale systems of nonlinear equations has been presented in this section. The main idea is to employ an implicit strategy to obtain the weighted matrix than the one proposed by Waziri et al.[7]. The anticipation has been to improve the accuracy of the Jacobian approximation, this is made possible by choosing a new approach for interpolating the curve while maintaining storage and time requirements of the proposed method to only linear order (see [13] for more details). The performance of this approach is highly dependent on how to obtain the {vj }2j=0, hence, a diagonal weighted matrix say Dk is considered to be used in obtaining more precisely the values of ρk , μk and β respectively. It is important to note that, Dk is quit different from the Dk . Therefore, the weighted diagonal
5670
M. Y. Waziri, Z. A. Majid and H. Aisha
matrix of this scheme is given as: Dk−1 = Diag(Dk , ρk−1 , μk−1).
(8)
Note that, in [7] the weighted matrix is considered as Dk . The multi-step implicit updating scheme is given as ν1 = − {sTk Dk−1 sk },
(9)
ν0 = − {(sk + 2sk−1 )T Dk−1 sk + (sTk−1 Dk−1 sk−1 )}.
(10)
ρk−1 = sk − β ( β +2)sk−1 ,
(11)
μk−1 = yk − β ( β +2)yk−1,
(12)
where β is denoted by ν2 − ν0 β = , ν1 − ν0
(13)
we require that Dk−1 satisfies the secant equation, i.e. ρk−1 Dk−1 = μk−1 .
(14)
Since it is difficult in letting a diagonal matrix to satisfy the secant equation, in particular, because Jacobian approximations are not usually done in element wise, we consider to use the weak secant condition [4] instead: ρTk−1 Dk−1 ρk−1 = ρTk−1 μk−1 .
(15)
Using the same approach as in [7], the new weighted matrix is obtained as (ρTk−1 μk−1 − ρTk−1 Dk ρk−1 ) Dk−1 = Dk + Gk−1 , 2 T r(Gk−1)
(16)
(1) (2) (n) where Gk−1 = diag((ρk−1 )2 , (ρk−1 )2 ...(ρk−1 )2 . Since Dk−1 is a diagonal matrix, is clearly that one can obtained it very cheaply. Hence, by using the new
Implicit multi-step diagonal secant-type method
5671
weighted matrix, the values of the {vj }2j=0 can be obtain which lead to the following:
ρk = sk −
β2 sk−1 , 1 + 2β
(17)
μk = y k −
β2 yk−1 , 1 + 2β
(18)
where β is denoted by β=
ν2 − ν0 . ν1 − ν0
(19)
To this end, the updating formula for D is given as follow :
Dk+1 = Dk +
(ρTk μk − ρTk Dk ρk ) Gk , T r(G2k )
(20)
(1) (2) (n) (i) where Gk = diag((ρk )2 , (ρk )2 , . . . , (ρk )2 ) , ni=1 (ρk )4 = T r(G2k ) and T r is the trace operation. To safeguard on the possibility of generating undefined Dk+1 , the proposed updating scheme for Dk+1 is used whenever : ⎧ (ρT μ −ρT Q ρ ) ⎪ ⎨ Dk + k Tk r(Gk2 ) k k Gk ; ρk > 10−4 , k Dk+1 = ⎪ ⎩ Dk ; otherwise. Hence, the stages of the proposed method are given as follows: Algorithm (IMDS) Step 1 : Choose an initial guess x0 and D0 = I, let k := 0. Step 2 : Compute F (xk ). If F (xk ) ≤ 1 stop, where 1 = 10−4 . Step 3 : If k := 0 define x1 = x0 − D0−1 F (x0 ). Else if k := 1 set ρk = sk and μk = yk and goto 5. ν0 and β via (9), (10) and (13), Step 4 : If k ≥ 2 compute ν1 , respectively and find ρk−1 and μk−1 using (11) and (12), respectively. If ρTk−1 μk−1 ≤ 10−4 ρk−1 2 μk−1 2 set ρk−1 = sk and μk−1 = yk . Else retains the computed (11) and (12) and compute weighted matrix Dk−1 by using (16), Step 5 : Compute ρk and μk using (17) and (18). Check If
5672
M. Y. Waziri, Z. A. Majid and H. Aisha
ρTk μk ≤ 10−4 ρk 2 μk 2 , set ρk = sk and μk = yk . Else retain the (17) and (18), Step 6 : Check if ρk 2 ≥ 1 if yes retain Dk+1 that is computed by step 5. Else set, Dk+1 = Dk Step 7 : Set k := k + 1 and goto 2.
3
Numerical results
In this section, the performance of IMDS update with that of the Newton’s method (NM), Broyden’s method (BM), MFDN method proposed by Leong et al.[5] and 2-MFDN update presented by Waziri et al.[7] has been compared. The comparison is based upon the following criterion: Number of iterations and CPU time in seconds. The computations are performed in MATLAB 7.0 using double precision computer. The stopping criterion used is ρk + F (xk ) ≤ 10−4 .
(21)
The identity matrix has been chosen as an initial approximate Jacobian. Seven (7) different dimensions are performed on the benchmark problems ranging from 50 to 500000. The codes also terminates when ever one of the following happens; (i) The number of iteration is at least 600 but no point of xk that satisfies (21) is obtained; (ii) CPU time in second reaches 600, (iii) Insufficient memory to initiate the run. All the results are presented using performance profiles indices in terms of robustness and efficiency as proposed in [2]. In the following, some details on the benchmarks test problems are given as. Problem 1 Trigonometric System of Byeong [14] : fi (x) = cos(xi ) − 1 i = 1, 2, . . . , n and Problem 2
x0 = (0.87, 0.87, ..., 0.87)
[7]:
fi (x) = ln(xi ) cos((1 − (1 + (xT x)2 )−1 )) exp((1 − (1 + (xT x)2 )−1 )) i = 1, 2, . . . , n, and x0 = (2.5, 2.5, .., 2.5). Problem 3 Spares System of Byeong [14] : fi (x) = xi xi+1 − 1 fn (x) = xn x1 − 1, i = 1, 2, . . . , n − 1 and x0 = (0.5, 0.5, .., .5).
5673
Implicit multi-step diagonal secant-type method
Problem 4
[7]:
fi (x) = n(xi − 3)2 +
xi − 2 cos(xi − 3) − 2 exp(xi − 3) + log(x2i + 1)
i = 1, 2, . . . , n and x0 = (−3, −3, −3, . . . , −3) Problem 5 [15] : f1 (x) = x1 fi (x) = cos xi+1 + xi − 1, i = 2, 3, . . . n and x0 = (0.5, 0.5, . . . .5).
1.2
2−MFDN
IMDS
1
Efficiency indices
0.8
0.6
BM MFDN
0.4
0.2
0 NM
−0.2 100
200
300
400 500 600 System’s Dimensions
700
800
900
1000
Figure 1. Efficiency Profile of NM, BM, MFDN, 2-MFDN and IMDS methods as the dimensions increases (in term of CPU time)
5674
M. Y. Waziri, Z. A. Majid and H. Aisha
1.2
2−MFDN
IMDS
1
Robustness index
0.8
0.6
BM MFDN
0.4
0.2
0 NM
−0.2 100
200
300
400 500 600 System’s Dimensions
700
800
900
1000
Figure 2. Robustness profile of NM, BM, MFDN, 2-MFDN and IMDS methods as the dimensions increases (in term of CPU time)
Implicit multi-step diagonal secant-type method
5675
As expected, Figure 1 and 2 imply that diagonal updating proposed in this paper improves notably over the performance of NM, BM, MFDN and 2-MFDN methods, as having the greatest efficiency and robustness indices. This is not surprising that any method that required storage of full elements of Jacobian or its inverse , particularly when solving large-scale systems, NM, and BM methods are necessary inferior. Moreover, Figure 1 and 2 detailed the growth of the NM, BM, MFDN, 2-MFDN and IDMS methods’s CPU time as the dimension gets higher, which shows that IMDS method CPU time increases linearly as apposed by some variants of Newton’s method. Therefore, we declare that IDMS method has significantly improved the performance of 2-MFDN method in terms of CPU time for handling large-scale systems of nonlinear equations.
4
Conclusions
A new two-step diagonal variant of secant method for solving large-scale system of nonlinear equations has been presented. The fundamental idea behind this approach is to use an implicit updating approach to obtain an enhanced approximation of the Jacobian matrix in diagonal form which only require a vector storage. Numerical testing provides strong indication that the IMDS method exhibits enhanced performance in all the tested problems ( as measured by the CPU time, floating points operations and matrix storage requirement) by comparison with the other variants of Newton’s methods, an attribute which becomes more obvious as the dimension of the problems increases.
References [1] Dennis, J, E., 1983, Numerical methods for unconstrained optimization and nonlinear equations, Prince-Hall, Inc., Englewood Cliffs, New Jersey [2] Bogle,I.andJ.D.Perkins,1990. A new sparsity preservingquasi-Newton updatefor solving nonlinear equations .SIAM J. Sci. Statist,11:621-630. [3] Natasa K. and Zorna L., 2001, Newton-like method with modification of the right-hand vector, J. Maths. Compt. 71, 237-250. [4] Dennis J.E and Wolkowicz, H.Sizing and least change secant methods, SIAM J. Numer. Anal., 30 (1993), 1291-1313. [5] Leong, W. J. , Hassan, M. A and Waziri, M. Y., 2011, A matrix-free quasiNewton method for solving large-scale nonlinear systems, Comput. Math. App. 62 (5): 2354-2363.
5676
M. Y. Waziri, Z. A. Majid and H. Aisha
[6] M. Y. Waziri, W.J. Leong, M. A. Hassan,M. Monsi, A New Newton method with diagonal Jacobian approximation for systems of Non-Linear equations. Journal of Mathematics and Statistics Science Publication. 6 :(3) 246-252 (2010). [7] M.Y. Waziri, W.J. Leong, M. Mamat, A Two-Step Matrix-Free Secant Method for Solving Large-scale Systems of Nonlinear Equations. Journal of Applied Mathematics , vol. 2012, Article ID 348654, 9 pages, 2012. doi:10.1155/2012/348654. [8] Kelley, C. T. 1995. Iterative Methods for Linear and Nonlinear Equations. PA: SIAM, Philadelphia [9] M.Y. Waziri, Leong, W. J. and Hassan, M. A, 2011, Diagonal BroydenLike Method for Large-Scale Systems of Nonlinear Equations,Malaysian Journal of Mathematical Sciences 6(1): 59-73 (2012) [10] Ford, J. A. and Moghrabi, L. A., Alternating multi-step quasi-Newton methods for unconstrained optimization, J. Comput. Appl. Math. 82 105116 (1997) . [11] Ford, J. A. and Moghrabi, L. A., Multi-step quasi-Newton methods for optimization, J. Comput. Appl. Math. 50 305-323 (1994). [12] Farid, M., Leong, W. J. and Hassan, M. A., 2010. A new two-step gradienttype method for large scale unconstrained optimization. J. Comput. Math. Appl. 59: 3301-3307. [13] Ford, J. A. and S. Thrmlikit, S., New implicite updates in multi-step quasi-Newton methods for unconstrained optimization, J. Comput. Appl. Math. 152 133-146 (2003). [14] Byeong, C. S. Darvishi, M. T. and Chang, H. K. 2010. A comparison of the Newton-Krylov method with high order Newton-like methods to solve nonlinear systems . Appl. Math. Comput. 217: 3190-3198. [15] Roose, A. Kulla, V. L. M. and Meressoo, T., 1990. Test examples of systems of nonlinear equations. Tallin: Estonian Software and Computer Service Company. [16] Waziri, M.Y., Leong W. J., Hassan M. A. and Monsi M., Mathematical Problems in Engineering Volume 2011, Article ID 467017, doi:10.1155/2011/467017, 12 pages. Received: June, 2012
