The experiments have been performed on HP Compaq 6710b notebook computer with Intel Centrino processor, 2GB memory and Ubuntu LINUX. The results of ...
ISBN 978-9955-28-598-4 R. Kasımbeyli, C. Dinçer, S. Özpeynirci and L. Sakalauskas (Eds.): MEC EurOPT 2010 Selected papers. Vilnius, 2010, pp. 172–176
International Conference
24th Mini EURO Conference
“Continuous Optimization and Information-Based Technologies in the Financial Sector” (MEC EurOPT 2010) June 23–26, 2010, Izmir, TURKEY
© Izmir University of Economics, Turkey, 2010 © Vilnius Gediminas Technical University, Lithuania, 2010
ON GLOBAL OPTIMIZATION IN LEAST SQUARES NONLINEAR REGRESSION BY MEANS OF INTERVAL METHODS A. Žilinskas, J. Žilinskas
Institute of Mathematics and Informatics Akademijos 4, LT-08663 Vilnius, Lithuania Abstract:The optimization problems occurring in nonlinear regression normally cannot be proven unimodal. In the present paper the applicability to this problem of global optimization algorithms based on the interval arithmetic is investigated. Keywords: global optimization, interval arithmetic, nonlinear regression, nonlinear least squares.
1.
Introduction
2.
General properties of the optimization problems of nonlinear LSR
Among the statistical techniques used in development of economic and financial models least squares regression (LSR) is among the most popular. LSR algorithms are comprised of two main constituents: a minimization algorithm and a procedure of statistical analysis. In this paper we focus on optimization problems of nonlinear LSR. The contemporary statistical packages contain algorithms of nonlinear LSR including efficient subroutines for local minimization of sums of squared residuals. However, in non linear practical problems unimodality of objective functions hardly can be proven, and frequently they are indeed multimodal; see (Winkler, 2001). Therefore investigation of the efficiency of the available global optimization algorithms in nonlinear LSR is an urgent problem. For a general discussion on different global optimization methods and favourable conditions for their applications we refer to (Törn and Žilinskas, 1988). In the next section properties of objective functions of some typical LSR problems are analyzed. Although the general properties of such objective functions seem favourable for the application of algorithms based on interval arithmetic, the latter in LSR are almost not used. In the present paper we discuss the reasons of inefficiency of application of interval methods in nonlinear LSR. Let µ( X , Z ) denotes a regression function where X = (x1,…,xn) is vector of parameters and Z = (z1,…,zp) is vector of variables. The results of observations (measurements) at the points Z1,…,Zm are denoted by yi, i = 1,…,m. The optimization problem in nonlinear LSR is formulated as follows m
(
)
min ∑ yk − µ ( X , Z k ) 2 = min f ( X ),
X ∈A k =1
X ∈A
(1)
where A ∈ R n is a feasible region of parameters. The choice of an appropriate optimization algorithm depends on the properties of f (X) and A. Most frequently µ( X , Z ) is smooth, and A is simple, e.g. A={X: X ≥0}. The minimization problem (1) seems favourable for the application of classical nonlinear programming techniques: normally the number of variables (equal to the number of model parameters to be estimated) is small, and the objective function is smooth. Indeed, many well developed nonlinear programming techniques can be applied to find a local minimizer of (1). A practical problem could be solved easily using algorithms from many available packages if a starting point for local descent would be known in the region of attraction of the global minimizer. However, such a starting point frequently is not known, and (1) should be considered as a global optimization problem. This general theoretical argumentation is supported by the experimental results 172
ON GLOBAL OPTIMIZATION IN LEAST SQUARES NONLINEAR REGRESSION BY MEANS OF INTERVAL METHODS
presented in (Křivý, Tvrdík and Krepec, 2000) where 14 regression function have been considered. It has been shown in (Křivý, Tvrdík, and Krepec 2000) that the standard algorithms from the statistical packages NCSS, SYSTAT, S-PLUS, SPSS for the large percentage of random starting points failed to find the solution. For a general discussion on suitability of different global optimization methods for the minimization of squared residuals in nonlinear LSR we refer to (Žilinskas and Žilinskas, 2010b). Various heuristics can be applied here to find an appropriate solution, however without of justified estimate of precision. If an approximation of the global minimum with guaranteed accuracy is important then the interval arithmetic based global optimization methods seem perspective since they correspond to the mentioned above properties of (1). To test the suitability of the interval arithmetic based global optimization algorithms in nonlinear LSR in (Žilinskas and Žilinskas, 2010a) fourteen difficult test functions were minimized by the well established interval algorithm from the CToolBox described in (Hammer et al., 1995). The data related to these test problems are presented in (Tvrdík, 2010). The global optimization algorithm available in CToolbox is based on branch and bound approach and is implemented in C++. The branching of the search tree is performed by bisecting the promising sub-regions (multidimensional interval boxes) into two through the middle points of the longest sides. Bounds of the function values are estimated using interval arithmetic in standard and cantered forms. Monotonicity and concavity tests, and interval Newton method (see e.g. in (Hansen and Walster, 2004)) are used to discard non-promising sub-regions, where the objective function is monotone or concave, or where there are no stationary points. Intervals of first and second derivatives are evaluated using automatic differentiation. The stopping condition defining tolerance was set equal to 10-8. However, the algorithm stopped according to this stopping condition only in three cases of fourteen; in the other cases the computation was interrupted while exceeding the time limit equal to one hour. These results show that either interval methods or at least their implementation in CToolbox are not appropriate for the difficult cases of nonlinear LSR. In the present paper the results of the similar investigation are presented where the implementation of interval methods by Knüppel (1999) have been applied. 3.
Challenges of application of interval methods in nonlinear LSR
In this paper we consider fourteen problems which have been proven difficult by Křivý, Tvrdík and Krepec (2000); to facilitate the references, the formulas of the regression functions are presented in Table 1. Further references concerning these problems can be found in (Křivý, Tvrdík and Krepec, 2000).
Table 1. Test data and minimum points No.
Regression function
1. 2. 3. 4. 5. 6.
x1x3z1/(1 + x1z1 + x2z2) x3(exp(−x1z1) + exp(−x2z2)) x3(exp(−x1z1) + exp(−x2z2)) x1 + x2 exp(x3z) x1 exp(x2/(x3 + z)) exp(x1z) + exp(x2z)
11.
x1 exp(x3z) + x2 exp(x4z)
12. 13.
x1z (x2) + x3 (x2 /z ) x1 + x2z1(x3) + x4z2(x5) + x6z3(x7) x1 ln(x2 + x3z)
7. 8. 9. 10.
14.
x1 exp(x3z) + x2 exp(x4z) x1z x3 + x2z x4 exp(x1z) + exp(x2z) x1 + x2 exp((x3 + x4z)x5)
Feasible region
[0,1000], [0,1000], [0,1000] [0,1000], [0,1000],[0,10000] [0,2000], [0,100],[0,1000] [0,1000], [0,10], [0,2] [0,10000], [0,50000], [0,10000] [0,1], [0,1]
[0,1e+8], [0,1e+8], [−2,0], [−5,0] [0,1], [1,8], [1,5], [0,1] [−50,50], [−50,50] [0,100], [0,100], [0,5]([1,6]), [0,1], [0,1] [−10000,10000], [−10000,10000], [−1,1], [−1,1] [0,5], [0,5], [0,5]([1,6]) [−5,5], [−5,5], [0,10], [−5,5], [0,10], [−5,5], [0,10] [0,100], [0,100]([1,101]), [0,100]
173
Minimizer
3.1315, 15.159, 0.7801 13.241, 1.5007, 20.100 814.97, 1.5076, 19.920 15.673, 0.9994, 0.02222 0.00561, 6181.4, 345.22 0.2578, 0.2578
1655.2, 3.404e+7, −0.6740, −1.8160 0.004141, 3.8018, 2.0609, 0.2229 0.2807, 0.4064 9.3593, 2.0292, 1.3366, 0.4108, 0.3551 47.971, 102.05, −0.2466, −0.4965
0.05589, 3.5489, 1.4822 1.9295, 2.5784, 0.8017, −1.2987, 0.8990, 0.01915, 3.0184 2.0484, 18.601, 1.8021
A. Žilinskas, J. Žilinskas
We are not going to discuss these functions with respect to their suitability for general testing of global optimization algorithms. The requirements for such functions are discussed, e.g. in (Törn and Žilinskas, 1989; Mathar, Žilinskas, 1994). A representative set of multimodal nonlinear LSR is of great interest, but to our best knowledge such a set has not been yet proposed in the available publications. A representative set of tests should include the applications related problems, e.g. such as considered by Jerrell (1997); and Žilinskas and Bogle (2006). Although the set of problems proposed by Křivý, Tvrdík and Krepec (2000) is not necessary representative, it suits to the purpose of this research since it comprises difficult problems, and was used by Žilinskas and Žilinskas (2010a) to test the other implementation of interval methods. First and most important challenge which is obvious from the formula (1) is dependence of variables. Every summand in (1) contains the expression including the minimization variables (parameters to estimate) and data; in such a case the new occurrence of the same interval variable in interval operations is count as a new one severely increasing the interval of the result. The other challenge is specific for the problems of interest: for the majority of the considered problems the feasible region is huge. Seemingly the problems are essentially unconstrained, and the interval constraints are added only to enable testing of methods not applicable for unconstrained problems. 4.
Experimental results
For this investigation the implementation of interval global optimization algorithms by Knüppel (1999) was used. This implementation of the interval global optimization method is combined with searches implemented in real number arithmetic. The algorithm involving a local search, branch-andbound technique and interval arithmetic, and not requiring derivatives, was originally proposed in (Jansson and Knüppel, 1995). Numerical results for well-known problems and comparisons with other methods are given in (Jansson and Knüppel, 1995). The algorithm uses interval library PROFIL/BIAS by Knüppel (1994). In cases of arithmetic exceptions (e.g. division by the interval containing zero) this library allows to set the result as [−∞,+∞] without the program termination. However, extended interval arithmetic is not implemented in this library, and the algorithm hangs if the infinite interval is attempted to use in further computations. Therefore as in the case of CToolbox algorithm, the feasible regions of the 10th, 12th and 14th test problems had to be changed to the shown in brackets in Table 1. Objective functions of 10th and 12th test problems include power function which is not available in PROFIL/BIAS, and was implemented using the exponential and logarithmic functions. The real and interval functions values should be returned by a subroutine of the objective function. The optimization algorithm counts real (NFE) and interval (NIFE) function calls; it was slightly supplemented to measure the CPU time in seconds. The experiments have been performed on HP Compaq 6710b notebook computer with Intel Centrino processor, 2GB memory and Ubuntu LINUX. The results of optimization of the test problems using PROFIL/BIAS interval global optimization algorithm is shown in Table 2 where also the similar results for CToolbox (from Zilinskas and Zilinskas, 2010a) are presented for comparison. The termination of the algorithm by reaching the time limit does not guarantee that the global minimum is found. For the 2nd test problem algorithm produces strange results founding minimum 0 at point where values of parameters of the model are ‘Not a Numbers’. However the smallest found values of the objective functions and the corresponding minimizers are found for more than half of the test problems. Comparison of the minimization results by CToolbox and PROFIL/BIAS can be summarized as follows: the correct guaranteed answers are found by the former algorithm only in three cases, and by the latter algorithm in nine cases but without guarantee. As shown by Žilinskas and Žilinskas (2010a) CToolbox can successfully found the solution of several reformulated problem where linear parameters in (1) are replaced by their estimates (minimizers of (1) with respect to these variables); in this way the problems 1, 2, and 12 were successfully solved. However, reformulation of the problems has not helped to solve more problems by PROFIL/BIAS. Interval methods can be useful to verify the solutions found by other methods. Since the solutions (global minima) of all test problems are available, both algorithms have been tested in verifying these solutions. However, both algorithms have successfully verified the solutions only for the same problems which have been solved successfully.
174
ON GLOBAL OPTIMIZATION IN LEAST SQUARES NONLINEAR REGRESSION BY MEANS OF INTERVAL METHODS
Table 2. Comparison of testing results of CToolbox and PROFIL/BIAS Test No. 1.
Minimization results by Ctoolbox NFE
674258
Minimum
NFE
NIFE
Minimum
Not found
1468396
219526289
4.35527e−05
Not found
67523
10662377
1.25189
2.
1412003
Not found
4.
1150906
Not found
624496
65068262
0.0059862
124.362
603541
37755042
124.362
3. 5.
1038694 721521
6.
661
588071
Not found
9.
13570
623123
604961
10.
540084
12.
17454
11. 13. 14.
569687
1535228
83596
Not found
7. 8.
5.
Minimization results by PROFIL/BIAS
464100
2358916
55556193
0
Not found
86881
22652296
Not found
0.0088963
621164
40997835
0.0088963
Not found
219852
35410339
Not found
Not found
63620
Not found
23727
28371929
26893387
Not found
Not found
0.00437528
119219
19959291
0.00437528
Not found
922365
58388644
7.14651e−05
Not found
4306
7229973
0.0169413
Conclusion
The experiments with the hybrid algorithm PROFIL/BIAS gave more optimistic results than the recent experiments with CToolbox: more problems have been solved successfully. However, PROFIL/BIAS is a hybrid algorithm voiding the most important advantage of interval methods – the guaranteed accuracy. While the results of experiments with CToolbox were disappointing, the experiments with PROFIL/BIAS show that the application of interval methods for the problems of nonlinear LSR is not hopeless. Since the possibilities to reduce dependence of variables are very limited the development of pure interval methods oriented to nonlinear LSR does not seem promising. On the other hand, waiving the ambitious aim to find guaranteed solutions, the construction of a hybrid algorithm favourably competing with simple heuristics seems possible. The results of the experiments presented above suggest combining interval methods, branch and bound approach, and LSR problem oriented local search methods. Acknowledgments
The research is partially supported by the Agency for International Science and Technology Development Programmes in Lithuania through COST programme and The Research Council of Lithuania. References
Hammer, R.; Hocks, M.; Kulish, U.; Ratz, D. 1995. C++ Toolbox for Verified Computing: Basic Numerical Problems. Springer. Hansen, E.; Walster, G. W. 2004. Global Optimization Using Interval Analysis. 2nd ed. Dekker, New York. Jansson, C.; Knüppel, O. 1995. A branch-and-bound algorithm for bound constrained optimization problems without derivatives, Journal of Global Optimization 7(3): 297–331. doi:10.1007/BF01279453 Jerrell, M. 1997. Automatic differentiation and interval arithmetic for estimation of disequilibrium models, Computational Economics 10: 295–316. doi:10.1023/A:1008633613243 Knüppel, O. 1994. PROFIL/BIAS – A fast interval library, Computing 53(3–4): 277–287. doi:10.1007/BF02307379 Knüppel, O. 1999. PROFIL/BIAS V 2.0. Report 99.1, Technische Universität Hamburg-Harburg. Křivý, I. Tvrdík, J.; Krepec, R. 2000. Stochastic algorithms in nonlinear regression, Computational Statistics and Data Analysis 33: 277–290. doi:10.1016/S0167-9473(99)00059-6 Mathar, R.; Žilinskas, A. 1994. A class of test functions for global optimization, Journal of Global Optimization 5: 195–199. doi:10.1007/BF01100693
175
A. Žilinskas, J. Žilinskas
Törn, A.; Žilinskas, A. 1989. Global optimization, Lecture Notes in Computer Science 350: 1–255. Winkler, P. 2001. Optimization Heuristics in Econometrics, John Wiley & Sons, Chichester. Tvrdík, J. 2010. Global Optimization, Evolutionary Algorithms and Their Application to Computational Statistics [last visited 2010-02-04]. Available from Internet: . Žilinskas, J.; Bogle, I. D. L. 2006. Balanced random interval arithmetic in market model estimation, European Journal of Operational Research 175(3): 1367–1378. doi:10.1016/j.ejor.2005.02.013 Žilinskas, A.; Žilinskas, J. 2010a. Interval arithmetic based optimization in nonlinear regression, Informatica 21(1): 149–158. Žilinskas, A.; Žilinskas, J. 2010b. On probabilistic bounds inspired by interval arithmetic, Control and Cybernetics 39, (in print).
176
