FITPACK: Spline Interpolation with GAMS - CiteSeerX

GAMS/FITPACK: Spline Interpolation with GAMS ERWIN KALVELAGEN∗ February 4, 2002

1

Introduction

This document describes how the high quality and robust curve fitting routines in FITPACK[7, 8, 9] can be used from a GAMS environment. The system uses the external function capabilities of GAMS which allows us to call external functions packaged as a DLL (dynamic link library). The same DLL is used as a back-end for a simple visualization tool that can be used to inspect the quality of the fit.

2

Interpolation

Interpolation can be useful in several cases. An obvious situation is where the modeler does not have access to an analytic functional form, but just a number of data points. Another case is when there is a functional form, but when limitations in the GAMS syntax do not allow you to express this form. An example of such a case is an actual piecewise function. Even if such a function is continuous and differentiable, GAMS syntax does not allow you to express it. A simple example is the Huber M-function.

2.1

An example: the Huber-M function

Least squares estimation is not robust in the sense that the estimation results are very sensitive to outliers. A Mean Absolute Deviation (MAD) approach which does not minimize the squares of the residuals but rather the absolute residuals is often suggested as an alternative. MAD models can be solved by using Linear Programming (see e.g. [3, 6]). Huber-M regression is a combination of the previous two appraoches. The Huber M-estimation problem is stated as: ∗ GAMS

Development Corporation, Washington D.C.

1

M-ESTIMATION

minimize

P

ρ(i ) P subject to yi = j xi,j βj + i β

i

where ρ(.) is defined as ( 2 ρ(i ) = i 2k|i | − k 2

if |i | ≤ k otherwise

(1)

for some k > 0.

Figure 1: Graph of the function ρ(x), k = 1.5. Although this model is perfectly smooth and has continuous derivatives it is not easily formulated in GAMS. The objective function would require an if-thenelse construct which is not part of the GAMS language. A direct formulation therefore would require the use of discrete variables, which makes the model an MINLP model. A different approach is mentioned by [11, 19] where a Linear Complementarity Problem is formed by differentating the model. The GAMS model can be found in [14].

2

Figure 2: Graph of the functions that form ρ(x), k = 1.5.

2.2

Interpolation techniques

There are three popular methods to implement interpolation in GAMS. A simple way is to use a Least Squares fitting procedure. This requires knowledge of the functional form. The advantage is that this can be done completely inside GAMS, by formulating a Least Squares optimization problem. Least squares models can be used for both linear and non-linear curve-fitting problems, and the method can be applied easily on multi-dimensional problems. A second form is using a piecewise linear function. Such a function can be implemented in GAMS using Special Ordered Sets of Type 2 (SOS2 sets). A third form, the subject of this paper, is to implement a spline interpolation. Here a piecewise polynomial function of given degree is fitted to the data points using certain smoothness conditions.

2.3

Example: Least Squares curve fitting

In this example we will show how GAMS can be used to formulate a non-linear Least Squares problem. Consider a CES (Constant Elasticity of Substitution) production function, an important equation used in many economic models. A search in the GAMS

3

model library shows a handful of models that have CES functions. A production function Q = f (K, L) measures output given inputs consisting of the ‘factors of production’ (in our case we have 2 factors: labor L and capital K). A simple production function often used in the economic literature is the Cobb-Douglas production function [17]. It looks like: Q = λK α Lβ

(2)

A more complicated function that is very well known is the CES production function. CES functions were introduced by [1], and are also called ACMS functions, after the authors. The functional form of a CES production function is:  − η Q = γ δL−ρ + (1 − δ)K −ρ ρ

(3)

where L is labor, K is capital, Q is output. γ is called the ‘efficiency parameter’ (γ > 0), δ is the ‘distribution parameter’ (0 < δ < 1), and ρ is the ‘substitution parameter’ (−1 ≤ ρ ≤ ∞). η denotes the degree of homogeneity of the function. Takings logs, and renaming some parameters, we can write this as: ln Q = γ −


η ln δL−ρ + (1 − δ)K −ρ ρ

(4)

A data set from [13] is used as an example for our nonlinear regression problem. It is reproduced in table 1. L 0.228 0.258 0.821 0.767 0.495 0.487 0.678 0.748 0.727 0.695 0.458 0.981 0.002 0.429 0.231

K 0.802 0.249 0.771 0.511 0.758 0.425 0.452 0.817 0.845 0.958 0.084 0.021 0.295 0.277 0.546

Q 0.256918 0.183599 1.212883 0.522568 0.847894 0.763379 0.623130 1.031485 0.569498 0.882497 0.108827 0.026437 0.003750 0.461626 0.268474

L 0.664 0.631 0.059 0.811 0.758 0.050 0.823 0.483 0.682 0.116 0.440 0.456 0.342 0.358 0.162

K 0.129 0.017 0.906 0.223 0.145 0.161 0.006 0.836 0.521 0.930 0.495 0.185 0.092 0.485 0.934

Q 0.186747 0.020671 0.100159 0.252334 0.103312 0.078945 0.005799 0.723250 0.776468 0.216536 0.541182 0.316320 0.123811 0.386354 0.279431

Table 1: CES production function data set The optimization model to be solved can be simply stated as:

4

Figure 3: Surface of CES production function

NLREG minimize γ,δ,ρ,η

P

ri2

subject to ln Qi = γ −

η ρ

−ρ
ln δL−ρ + ri i + (1 − δ)Ki

Model nls.gms $ontext Nonlinear least squares. Example: Estimation of a CES production function Data set: Table 22.4, page 724 of Griffiths, Hill and Judge, LEARNING AND PRACTICING ECONOMETRICS, Wiley, 1993. Erwin Kalvelagen, 2000 $offtext set i ’observations’ /i1*i30/; set j ’parameters’ /L,K,Q/; table data(i,j) L i1 0.228

K 0.802

Q 0.256918

5

i2 i3 i4 i5 i6 i7 i8 i9 i10 i11 i12 i13 i14 i15 i16 i17 i18 i19 i20 i21 i22 i23 i24 i25 i26 i27 i28 i29 i30 ;

0.258 0.821 0.767 0.495 0.487 0.678 0.748 0.727 0.695 0.458 0.981 0.002 0.429 0.231 0.664 0.631 0.059 0.811 0.758 0.050 0.823 0.483 0.682 0.116 0.440 0.456 0.342 0.358 0.162

0.249 0.771 0.511 0.758 0.425 0.452 0.817 0.845 0.958 0.084 0.021 0.295 0.277 0.546 0.129 0.017 0.906 0.223 0.145 0.161 0.006 0.836 0.521 0.930 0.495 0.185 0.092 0.485 0.934

parameters L(i) K(i) Q(i) ;

0.183599 1.212883 0.522568 0.847894 0.763379 0.623130 1.031485 0.569498 0.882497 0.108827 0.026437 0.003750 0.461626 0.268474 0.186747 0.020671 0.100159 0.252334 0.103312 0.078945 0.005799 0.723250 0.776468 0.216536 0.541182 0.316320 0.123811 0.386354 0.279431

’labor’ ’capital’ ’output’

L(i) = data(i,’L’); K(i) = data(i,’K’); Q(i) = data(i,’Q’);

variables gamma delta rho eta residual(i) sse ; equations fit(i) obj ;

obj.. fit(i)..

’log of efficiency parameter’ ’distribution parameter’ ’substitution parameter’ ’homogeneity parameter’ ’error term’ ’sum of squared errors’

’the nonlinear model’ ’objective’

sse =e= sum(i, sqr(residual(i))); log(Q(i)) =e= gamma - (eta/rho)*log[delta*L(i)**(-rho) + (1-delta)*K(i)**(-rho)] + residual(i);

* initial values rho.l=1; delta.l=0.5; gamma.l=1; eta.l=1;

6

model nls /obj,fit/; solve nls minimizing sse using nlp; display gamma.l, delta.l, rho.l, eta.l, sse.l;

As an aside it is noted that CES functions also have an application in Linear Programming theory. Interior point methods are often based on variants of a logarithmic barrier function. However, one can also devise an interior point algorithm for linear programming based on a CES function [15]. In the example above, we dealt with a model that is non-linear in the parameters to be estimated. If the model is linear in this respect, we can use a linear solution technique. Say the model is y = Xβ + r where r are the residuals, then β can be estimated by OLS (Ordinary Least Squares) regression: βˆ = (X 0 X)−1 X 0 y. This can be written as the solution of the socalled normal equations: (X 0 X)β = X 0P y. A example of a model that can be estimated this n way is a polynomial y = i=0 ai xi . Summarizing, least squares computations are easily carried out in a GAMS environment, using a separate optimization model. Advantages include that the decision model does not become more complicated: the estimation is performed outside the decision model. The disadvantage is that one needs to have knowledge about the functional form of the equation. Sometimes a simple polynomial of sufficiently large degree can be used, but in some cases this can cause undesirable behavior between the observations. Least squares estimation easily extends to multiple dimensions and to nonlinear relationships.

2.4

Example: Interpolation with SOS2 variables

In this section we will show how SOS2 variables can be used to implement linear interpolation. Special Ordered Sets were introduced by [2]. Special Ordered Sets of Type 1 are used to select one out of k items. Special Ordered Sets of Type 2 are used to implement interpolation. Suppose we have tabular data for a scalar function y = f (x) for a given interval x ∈ [xlo , xup ]. Let’s denote this data by (¯ xk , y¯k ). We introduce variables λk with: X X= λk x ¯k (5) k

Y =

X

λk y¯k

(6)

k

X

λk = 1

(7)

k

max 2 adjacent λ’s nonzero λk ≥ 0 xlo ≤ X ≤ xup 7

(8) (9) (10)

Equation (5) is called the reference row. Equation (6) is known as the function row, and equation (7) is the convexity row. The λk variables form a Special Order Set of Type 2. In a SOS2 set only two adjacent variables of the set can assume non-zero values. SOS2 variables can be implemented in GAMS using the SOS2 variable type: sos2 variables lambda(k); The exact definition of SOS2 variables is solver specific, especially in special cases where nonzero lower bounds are used. In the above case we have added λk ≥ 0, which only leaves the “max two adjacent λ’s are nonzero” which is shared by all solvers that have SOS2 facilities. y O

y¯k •44  44  44 y¯k+2 y¯k−1  44   4  44  • 4y¯4k+1 ooo• 44 oooo •oo

/ x ¯k−1

x ¯k+1

x ¯k

x

x ¯k+2

Figure 4: Linear interpolation SOS2 variables can be used to find global solutions for non-convex optimization problems. Consider the problem:

P2

maximize y x

subject to y = x4 − 3x3 − 1.5x2 + 10x y = −20x + 100 −5 ≤ x ≤ 5

8

Figure 5: Two feasible points in problem P2 The NLP model using a (default) starting point of x0 = 0.0 will cause the NLP solver CONOPT to converge to x∗ = 3.395, y ∗ = 32.105. variables y,x; equations e1,e2; e1.. y =e= power(x,4) - 3*power(x,3) - 1.5*power(x,2) + 10*x; e2.. y =e= -20*x+100; x.lo = -5; x.up = 5; y.l = 0; x.l = 0; model m /all/; solve m using nlp maximizing y;

Using a SOS2 implementation we can find a close solution to the global optimum. There are two sources for error: the linear approximation, and the integer optimality tolerances as set by the options OPTCR and OPTCA. The linear approximate solution can be refined by passing the optimal integer solution to an NLP solver. The integer solution will be close to the global optimum, so there is good hope the NLP solver will converge to that solution.

9

set k /k0*k100/; parameter xbar(k), ybar(k); xbar(k) = -5 + (ord(k)-1)*0.1; ybar(k) = power(xbar(k),4) - 3*power(xbar(k),3) - 1.5*power(xbar(k),2) + 10*xbar(k); display xbar,ybar; variables y,x; sos2 variables lambda(k); equations refrow, funrow, convexity, e1, e2; refrow.. x =e= sum(k, lambda(k)*xbar(k)); funrow.. y =e= sum(k, lambda(k)*ybar(k)); convexity.. sum(k, lambda(k)) =e= 1; e1.. y =e= power(x,4) - 3*power(x,3) - 1.5*power(x,2) + 10*x; e2.. y =e= -20*x+100; lambda.lo(k) = 0; x.lo = -5; x.up = 5; option option option option

optcr=0; optca=0; mip=cplex; nlp=conopt2;

model m1 /refrow, funrow, convexity, e2/; solve m1 using mip maximizing y; model m2 /e1,e2/; solve m2 using nlp maximizing y;

model MIP model m1 NLP model m2

x∗ -3.243 -3.244

y∗ 164.851 164.874

Table 2: Results for problem P2 Notice that it is not needed to pass an initial solution to the MIP solvers. LP/MIP solvers are in general not capable of dealing with level values, although for large models it may be beneficial to specify an advanced basis (this can be accomplished by setting the marginal values). For MIP models, in many cases the time it takes to solve the initial LP is negligible, in which case using an advanced basis is not beneficial. It is noted that this SOS2 implementation implements a “grid search” and thus sharp spikes may be missed altogether if the grid is too coarse. It is possible to use SOS2 variable to model two-dimensional functions z = f (x, y), but the modeling becomes somewhat difficult [18]. Linear interpolation is completely data-driven: it is not needed to specify a functional form of the underlying model. SOS2 variables can cause a com10

putational burden as they introduce a MIP component to the model. If the model is nonlinear otherwise, a possible difficult to solve MINLP model results. Linear interpolation using SOS2 variables does not easily generalize to more dimensions. It is possible to simulate SOS2 variables using binary variables. This means that the above technique can even be used with MIP solvers that do not support SOS2 sets, although the modeling becomes a little bit more complicated (see [14, 18]).

3

Spline interpolation

Spline functions are piecewise polynomials of degree k (order k + 1) on an interval [a, b]. The interval is divided into sub-intervals by the introduction of knots. Knots form an increasing sequence λj , j = 0, .., g + 1 with λ0 = a and λg+1 = b. Formally a spline function s(x) is defined as follows [9]: • s(x) is a polynomial of degree at most k on each interval [λj , λj+1 ], i.e. s|[λj ,λj+1 ] ∈ Pk , j = 0, 1, ..., g.

(11)

• s(x) and its derivatives up to order k − 1 are continuous on [a, b]: s(x) ∈ C k−1 [a, b].

(12)

If additional boundary knots are added λj ≤ a, j = −k, .., −1 and λj ≥ b, j = g + 2, .., g + k + 1 then a spline function can be expressed as: s(x) =

g X

ci Mi,k+1 (x)

(13)

i=−k

where ci are coefficients, and Mi,k+1 (x) is a normalized B-spline. These B-spline functions can be efficiently evaluated using a recursive scheme [4]. Similarly, differentiation and integration of these functions is easily performed. The use of splines is done in two steps: first there is the forming of the spline and subsequently evaluation of the spline or its derivative is called for. Both steps are not easily coded in GAMS. However, high quality software is available: [4] contains Fortran code (also available electronically from the authors web site) and a somewhat higher level Fortran implementation is available in [16] which implements the FITPACK algorithms from [7, 8, 9]. The GAMS external function facility can be used to access FITPACK. In our implementation we use three FITPACK routines: CURFIT to form the splines based on data points (xi , yi ), SPLEV to evaluate the splines for any x ∈ [a, b] and SPLDER to evaluate the first derivatives. The degree k of the spline can be chosen by the user. It is required that 1 ≤ k ≤ 5. In general cubic splines (k = 3) form a good compromise between quality of fit and computational efficiency. The number and placement of the 11

knots λi is often difficult to choose. The routine CURFIT therefore has an option to place knots automatically, a facility exploited in our implementation. The B-spline coefficients ci are determined according to the following minimization problem:

CURFIT

minimize

Pg

subject to

Pm

c

(k) (λi +) i=1 [s

j=1 [wj (yj

− s(k) (λi −)]2

− s(xj ))]2 ≤ S

The objective denotes the sum of squares of the discontinuity jumps in the kth order derivative of s(x), i.e. it measures the smoothness of fit. The constraint puts a limit on the closeness of fit, using a weighted least squares criterion. The weighting factors wj and the smoothing factor S are provided by the user. If S = 0, an interpolating spline will be formed. For large values of S the algorithm will return a least-squares polynomial of degree k. The knot placing algorithm will need more knots in case S is small, or if the curve is “difficult”. It will automatically place more knots where the data values changes rapidly or where the curve has a complex shape. Concluding we summarize that splines cannot be implemented in GAMS directly. They need to be used through the external function facility in GAMS. This means the use of the non-intuitive =X= equations in the model, and a cumbersome set-up. If the model is an NLP otherwise, the model type will not change. If the model is linear or mixed-integer, then the introduction of an NLP component may be prohibitive complex. Spline routines are available for surface fitting z = f (x, y) as part of FITPACK but extension to higher dimensions is more difficult.

4

External functions in GAMS

GAMS has a rudimentary support for external functions. External functions are

equality constraints F (y) = 0

(14)

where F (.) and its gradient are not evaluated by the GAMS run-time system, but by a piece of software written by the modeller. The software needs to be packaged in a DLL (dynamic link library) for Windows, or shared object for Unix systems. A DLL can be written in most popular languages: C/C++, Delphi, Visual Basic, Fortran etc. Technical documentation can be found in [10]. However, as experience learns that this documentation is difficult to grasp on first reading, we will give a short tutorial here. Consider the simple NLP model: variables x,y,z;

12

equations e1,e2,e3; e1.. z =e= x+y; e2.. x =e= y; e3.. x*y =e= 2; * initial values x.l = 1; y.l = 1; model demo /e1,e2,e3/; solve demo using nlp minimizing z;

Now suppose that we want to introduce an external implementation of the constraint function e3: xy = 2. This has to be rewritten into: f (x, y) = xy − 2 = 0.

(15)

In addition we know that x is the variable that corresponds to the first element in the array holding the variables in the external DLL and y is the second element. I.e. in C we would say that x[0] corresponds to x and x[1] corresponds to y. This mapping between GAMS variables (x, y) and index positions in the external subroutine variable array need to be specified in the GAMS model. In the GAMS model we now change equation e3 to: e3.. 1 =X= 1*x + 2*y;

The =X= means, this is an externally defined constraint. Although this looks like a linear constraint, it is in fact a nonlinear constraint of the form f (x, y) = 0. The number 1 in front of the =X= is the external equation number. In this case we only have one, but if there are many, then each equation needs to get a unique number in the range 1, .., m , where m is the number of external equations. The coefficient of x is one, which does not mean x is linear in this equation, but instead indicates the mapping for x: x is the first external variable. Likewise 2*y means: y is the second external variable. The external function that implements this can be coded in C as follows: /* * demo.c * * example external function DLL */ #define VIS #define GE_EXPORTS

// used in header file, VIS means VISUAL C // used in header file, we want to export function gefunc

#include "geheader.h" int n,m,nz; // number of variables, rows and nonzeroes in =X= equations char buffer[100]; // message buffer GE_API int GE_CALLCONV gefunc( int *icntr, double *x, double *f, double *d, msgcb_t msgcb) {

13

if ( icntr[I_Mode] == DOINIT ) { /* * initialization call */ GEstat(icntr, "\nInside the DEMO.DLL."); GElog(icntr, "\nInside the DEMO.DLL.");

// write message to the listing file // write message to the screen

n = icntr[I_Nvar]; // number of variables (should be 2) m = icntr[I_Neq]; // number of equations (should be 1) nz = icntr[I_Nz]; // number of nonzero elements (should be 2) sprintf(buffer,"Ext: n=%d, m=%d, nz=%d",n,m,nz); GEstat(icntr, buffer); // write message to the listing file GElog(icntr, buffer); // write message to the screen return 0; // OK } else if ( icntr[I_Mode] == DOEVAL ) { /* * evaluation call */ int function_number = icntr[I_Eqno]; // number of external function to be evaluated int func_value_needed = icntr[I_Dofunc]; // whether we need to return the function value int derivative_needed = icntr[I_Dodrv]; // whether we need to return the derivatives /* * function number should be 1 if we only have one * scalar =X= equation */ if (function_number != 1) return 2; // Error /* * assume we want to calulate f(x1,x2) = x1*x2 - 2 */ if (func_value_needed) { *f = x[0]*x[1] - 2; } if (derivative_needed) { d[0] = x[1]; d[1] = x[0]; } return 0; // OK, return 1 means "could not evaluate" } else if ( icntr[I_Mode] == DOTERM ) { /* * termination call */ return 0; // OK } return 2; // we should never come here }

Here follows a log of compiling the above source using Microsoft Visual C and running the model:

14

D:\Microsoft Visual Studio\MyProjects\External Function Example>cl /LD demo.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8168 for 80x86 Copyright (C) Microsoft Corp 1984-1998. All rights reserved. demo.c Microsoft (R) Incremental Linker Version 6.00.8168 Copyright (C) Microsoft Corp 1992-1998. All rights reserved. /out:demo.dll /dll /implib:demo.lib demo.obj Creating library demo.lib and object demo.exp D:\Microsoft Visual Studio\MyProjects\External Function Example>gams demo GAMS Rev 118 Copyright (C) 1987-2000 GAMS Development. All rights reserved Licensee: Erwin Kalvelagen G001113:1934CE-WIN GAMS Development Corporation DC1556 --- Starting compilation --- DEMO.GMS(25) 1 Mb --- Starting execution --- DEMO.GMS(22) 1 Mb --- Generating model demo --- DEMO.GMS(25) 2 Mb --3 rows, 3 columns, and 7 non-zeroes. --- DEMO.GMS(25) 2 Mb --- Executing CONOPT Inside the DEMO.DLL. Ext: n=2, m=1, nz=2 C O N O P T Nov 19, 2000 WIN.CO.CO 19.6 006.041.039.WAT Library 043E

C O N O P T Copyright (C)

Wintel version 2.043E-006-041 ARKI Consulting and Development A/S Bagsvaerdvej 246 A DK-2880 Bagsvaerd, Denmark

Using default control program. Reading data Iter Phase Ninf Infeasibility 0 0 3.0000000000E+00

RGmax

NSB

** Feasible solution. Value of objective = Iter Phase Ninf Objective RGmax 10 3 2.8284271247E+00 0.0E+00

Step

2.82804481590 NSB Step 0 0.0E+00

** Optimal solution. There are no superbasic variables. ------***

Restarting execution DEMO.GMS(25) 0 Mb Reading solution for model demo Status: Normal completion

D:\Microsoft Visual Studio\MyProjects\External Function Example>

For more information see [10].

15

5

Implementation

The sources of FITPACK can be downloaded from [16]. It is noted that another, unrelated FITPACK exists on NETLIB. A DLL was generated by the Compaq Visual Fortran Compiler version 6.6. Two technical issues needed attention. First all REAL declarations in FITPACK need to be interpreted as REAL*8. An appropriate compiler flag (/real size:64) is available to achieve this. Secondly, appropriate calling conventions need to be followed in order to talk to GAMS or to the Delphi visualization program. For communication with Delphi we used the stdcall convention, which is the Windows API standard, and is the most obvious choice as all compilers under Windows will support such a calling convention in order to be able to communicate with the Windows Operating System. GAMS uses a different calling convention, and thus the exported function to GAMS needs to be appropriately decorated. The visualization tool is implemented in Borland Delphi version 6. The FITPACK.DLL needs the Fortran run-time support. In case you don’t have the compiler installed on your machine, a run-time support kit can be downloaded from [5]. The file VFRUN660I.exe is also part of our FITPACK distribution. The system is currently only available on PC’s running a version of Windows. During the setup call of the DLL a data file is read which includes the data points. In the same call the FITPACK routine CURFIT is called to perform the curve fitting. In the subsequent evaluation calls, only the FITPACK routines SPLEV and SPLDER are called which evaluate the spline and its derivative. The basic algorithm can be stated as: if initialization call then read data file C:\TMP\SPLINEDATA.TXT ns := number of spline functions for i = 1, ..., ns do call CURFIT(i) end for else if evaluation call then i := index number of the x variable j := index number of the y variable x := X[i] is := index number of the spline function if function value needed then f := SPLEV(is, x) end if if derivative needed then for k = 1, ..., n do if k = i then gk := SPLDER(is, x) else if k = j then gk := −1 16

else gk := 0 end if end for end if else if termination call then clean up memory else error: unknown call type end if

6

GAMS modeling

To setup a model with GAMS/FITPACK you will need to do two things: change the GAMS model to call external function and create a data file with the necesary information for FITPACK including the data points. This section will discuss the GAMS part and the next section will discuss the syntax of the data file in detail. A single scalar curve fitting model can be modeled as follows: x.lo = a; x.up = b; : spline.. 1 =X= 1*x + 2*y; : file fitpack /c:\tmp\fitpack.dll/; model m /all,fitpack/; solve m using nlp minimizing z;

A model with multiple external functions can be specified as follows: parameter extfun(k) ’external function numbers’; extfun(k) = ord(k); parameter xvarno(k) ’external numbers for x: 1,...,K’; xvarno(k) = ord(k); parameter yvarno(k) ’external numbers for y: K+1,...,2K’; xvarno(k) = card(k)+ord(k); x(k).lo = a; x(k).up = b; : spline(k).. extfun(k) =X= xvarno(k)*x(k) + yvarno(k)*y(k); : file fitpack /c:\tmp\fitpack.dll/; model m /all,fitpack/; solve m using nlp minimizing z;

17

It is not specified here if all the external rows use the same spline function or not. The mapping between external function row numbers and spline functions is specified in the data file. Each external spline equation needs to have two variables: xi and yi . The meaning of the variables is yi = f (xi ) or to be precise f (xi ) − yi = 0. This requirement is not checked in the DLL, so it is left to the modeler to make sure the sparsity pattern is correct. (The DLL simply does not know the sparsity pattern, as this is not passed on). It is noted that it is important to set correct bounds on the x variable so that the spline is only evaluated on the interval [a, b]. This interval will also need to be passed on to the DLL using the data file.

7

Data file format

The format of the data file is described in this section. The data file can be generated by hand using any text editor, or can be written by GAMS using the PUT facility. The data file is by default C:\TMP\SPLINEDATA.TXT. The name or location can be changed by setting an environment variable SPLINEDATA. The reason for this fixed file name and location is twofold. First, the external function interface in GAMS is very rudimentary: it does not allow to pass extra information (such as a simple file name) to the DLL. Secondly, the option to use a file in the GAMS scratch directory, a partial solution, as suggested in the external function documentation[10], is not very attractive, again for two design reasons. The first is that the scratch directory is removed after a run, deleting this file, which makes debugging more difficult. The second problem is that the use of function GENAME to find out the name of the scratch directory, is extraordinarily and largely needlessly complicated. The first record in the data file contains two integer numbers: the number of spline functions in the file and the number of external functions in the model: ns

nf

Description ns An integer indicating the number of splines. nf An integer indicating the number of external functions. This number should be equal to the number of =X= rows specified in the GAMS model. If your model contains a constraint: yi = f (xi ) where f is a spline function, there is just one spline function f , but there are card(i) external functions in the model. After the first line, we have ns sections with spline information. For each spline we have the following: 18

m k nest S xlo xup x(1) y(1) w(1) x(2) y(2) w(2) : : : x(m) y(m) w(m)

Description m The number of data points. k The degree of the spline. In general it is advised to use a cubic spline, i.e. k = 3. nest An overestimate of the number of knots in the spline. In general it is advised to be generous here, and choose nest = m + k + 1. This number is always large enough. For well-behaved splines one could save some memory by trying nest = m/2. S The smoothing parameter S ≥ 0 is important for the behavior of the curve fitting algorithm. The smoothing parameter sets a trade-off between closeness of fit or smoothness of fit. For S = 0 we have a interpolating spline, in which case we need nest = m +k +1. If S is large a least squares polynomial of degree k is formed. xlo, xup Lower and upperbound of the interpolation interval [xlo, xup]. These values should form a bound on the values of x passed on to the evaluation routines. In GAMS make sure that the x variables are properly bounded to lie inside this interval. (xi , yi ), i = 1, .., m Data points. The points must be specified ordered by x, i.e. xi ≥ xi−1 . In addition xlo ≤ xi ≤ xup must hold. wi , i = 1, .., m Weights. You can choose wi = 1 if you have no good reason to choose another weighting scheme. After the spline sections there follows a single section that maps external function rows into splines and specifies which variables are used in each external function row. if(1) is(i) ix(1) iy(1) if(2) is(2) ix(2) iy(2) : : : : if(nf) is(nf) ix(nf) iy(nf)

19

Description ifi The external function row number. This number is equal to the right-hand side constant of corresponding =X= row in GAMS. 1 ≤ ifi ≤ nf should hold. isi The number of the spline function corresponding to external row ifi . 1 ≤ isi ≤ ns should hold. ixi , iyi The number of the x and y variable in yi = f (xi ). xi is the argument of the spline, while yi is the function value. Note the GAMS =X= row is actually an equality constraint: f (xi ) − yi = 0. Example An example of a single external function is as follows: 1 7 3 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 1

1 11 0.00 0.00 0.00 0.50 1.00 1.00 1.00 1

0.10 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1

-3.00

3.00

2

In fact this file has been written by GAMS as follows: $ontext test model for spline Erwin Kalvelagen, 2002 Requires distribution 20.3 or higher $offtext

variables z,x,y; equations zdef,spline;

zdef.. z =e= y; spline.. 1 =x= 1*x+2*y; file fitpack /fitpack.dll/; model m /all,fitpack/;

set d ’data points’ /d1*d7/;

20

scalar ns ’number of splines’ /1/; scalar nf ’number of external functions’ /1/; scalar nd ’number of data points’; nd = card(d); scalar k ’degree’ /3/; scalar nest ’estimation of number of knots’; nest = nd+k+1; scalar s ’smoothing parameter’ /0.1/; scalar xlo ’interval, lowerbound’ /-3/; scalar xup ’interval, upperbound’ /3/; parameter datax(d) ’data points, x coordinate’ / d1 -3 d2 -2 d3 -1 d4 0 d5 1 d6 2 d7 3 /; parameter datay(d) ’data points, y coordinate’ / d1 0 d2 0 d3 0 d4 0.5 d5 1 d6 1 d7 1 /; parameter dataw(d) ’data points, weights’; dataw(d) = 1.0; file datafile /’c:\tmp\splinedata.txt’/; put datafile; put ns:10:0,nf:10:0/; put nd:10:0,k:4:0,nest:10:0,s,xlo,xup/; loop(d, put datax(d),datay(d),dataw(d)/; ); put 1:10:0, 1:10:0, 1:10:0, 2:10:0/; putclose;

option nlp=conopt1;

*check points parameter res(d); loop(d, x.fx = datax(d); solve m minimizing z using nlp; res(d) = y.l; ); display res;

The log file (screen) and listing file show the results of the curve fitting routine CURFIT: *** FITPACK.DLL: INITIALIZATION EXTERNAL JACOBIAN SIZE: M,N,NZ= DATA FILE: c:\tmp\splinedata.txt -----------------------------------------Spline: 1 Number of datapoints: 7

1

21

2

2

Interval: -3.00000000000000 3.00000000000000 Degree: 3 Smoothing parameter: 0.100000000000000 Estimation of number of knots: 11 CURFIT error code: -2 Normal return (least sq) knots: 8 Weighted sum of squared residuals: 4.761904761904755E-002 ------------------------------------------

The evaluation calls to SPLEV and SPLDER do not generate any output. If you want to trace those calls you need to turn on debugging, as explained in the next section.

8

Debugging

The code has a debugging option which has to be turned on by setting an environment variable SPLINEDEBUG to a log file name, e.g.: >set SPLINEDEBUG=c:\tmp\splinelog.txt

A typical record in the log file looks like: Tue Jan 15 09:37:58 2002 IFUN: 16

IS: 1

IX: 16

IY: 66

X: -9.0000

Res:

81.000

TYP:F

date The first column is the date and time stamp. ifun IFUN is the number of the external function row. is IS is the number of the spline function. ix IX is the number of the x variable, the argument of the spline. iy IY is the number of the y variable, which is equal to the spline result if this equation is feasible. x X is the value of x variable, the argument of the spline. Res Res is the result of the spline evaluation call. It is either a function value of the spline or the derivative of the spline. It is noted, that the equation is actually an equality of the form f (xi ) − yi = 0. The function value reported here is f (xi ) and the derivative is df (xi )/dxi . 22

typ TYP is the type of evaluation call. It is ’F’ for a function evaluation call and ’D’ for a derivative evaluation call. The log file is erased when a new solve statement is executed.

9

The visualization tool

The visualization tool SeeSpline.exe can read the same data file that is needed by the DLL. The tool can help in deciding on the value of the smoothing parameter S. To use SeeSpline just invoke it, open the data file, and then select the spline. To be precise:

Figure 6: SeeSpline tool 1. Select F ile|Open. 2. Open the data file for FITPACK. Often this will be C:\TMP\SPLINEDATA.TXT as this is the default used inside the DLL. 3. A message will appear in the status bar: n splines read. spline with Window menu. 23

Select

4. Select W indow|Spline n. The spline and the data points will be drawn. 5. To change the options S or nest you can use the Edit menu.

10

Example x -5 -4 -3 -2 -1 0

y 0.0067 0.0183 0.0497 0.1353 0.3678 1

x 1 1.6094 2 3 4 5

y 2.7182 5 5 5 5 5

Table 3: Data set degree 1 2 3 4 5 11

residual sum of squares 10.7061 10.26067 4.138231 3.256527 2.080369 0

Table 4: Results for polynomials of different degree Consider the data set shown as reproduced in table 3. Fitting a polynomial is quite straightforward. The basic equations in the GAMS model are: fit(d).. obj..

sum(j, coeff(j)*power(data_x(d),ord(j)-1)) =e= data_y(d) + residual(d); z =e= sum(d, sqr(residual(d)));

The results for this model are depicted in figure 7. The fit becomes better if the degree of the polynomial increases. This can be seen from the results in table 4. In theory we can do an interpolation by fitting a polynomial of degree k − 1 when there are k data points. The simplest example of this is fitting a straight line to two points. The residual sum of squares for such a fitting problem should be 0.0. In practice, this often leads to poorly behaved functions. As an example we show in figure 8 the results for a polynomial of degree 11 against our data set. In actual applications the situation may be even worse. When running the FITPACK spline interpolation routine against this data set we see the results as depicted in figure 9. The spline with S = 10 is in fact a single polynomial of degree k = 3, which is also plotted in figure 7. 24

Figure 7: Least squares polynomial fit The complete model that solves the fitting problems in this section and generates the plots using gnuplot[12], is listed below. Model generateplots.gms $ontext test model for spline Erwin Kalvelagen, 2002 Requires distribution 20.3 or higher $offtext

*---------------------------------------------------* Splines though external functions *---------------------------------------------------set p ’sample points’ /p1*p100/; parameter fno(p) ’function row number’; fno(p) = ord(p); parameter xno(p) ’variable number for x’;

25

Figure 8: Interpolation with a single high degree polynomial

xno(p) = ord(p); parameter yno(p) ’variable number for y’; yno(p) = card(p)+ord(p); variables z,x(p),y(p); equations zdef,spline(p); zdef.. z =e= sum(p,y(p)); spline(p).. fno(p) =x= xno(p)*x(p)+yno(p)*y(p); file fitpack /fitpack.dll/; model m /all,fitpack/; option nlp=minos; *---------------------------------------------------* Prepare data for FITPACK.DLL *---------------------------------------------------set d ’data points’ /d1*d12/; scalar ns ’number of splines’ /1/; scalar nf ’number of external functions’; nf = card(p); scalar nd ’number of data points’; nd = card(d); scalar k ’degree’ /3/; scalar nest ’estimation of number of knots’; nest = nd+k+1;

26

Figure 9: Splines with smoothing parameters S = 0.0, 1.0, 10.0

scalar xlo ’interval, lowerbound’ /-5/; scalar xup ’interval, upperbound’ /5/; table data(d,*) x y d1 -5 0.0067 d2 -4 0.0183 d3 -3 0.0497 d4 -2 0.1353 d5 -1 0.3678 d6 0 1 d7 1 2.7182 d8 1.6094 5 d9 2 5 d10 3 5 d11 4 5 d12 5 5 ; parameter datax(d) ’data points, x coordinate’ ; parameter datay(d) ’data points, y coordinate’ ; parameter dataw(d) ’data points, weights’; datax(d) = data(d,’x’); datay(d) = data(d,’y’); dataw(d) = 1.0;

scalar xmin; xmin = smin(d,datax(d)); scalar xmax; xmax = smax(d,datax(d));

27

scalar delta; delta = (xmax-xmin)/(card(p)-1); x.fx(p) = xmin+(ord(p)-1)*delta; set ls ’loopset’ /iter1*iter3/; parameter s(ls) ’smoothing parameter S’ / iter1 0, iter2 1, iter3 10 /; parameter results(p,ls); file datafile /’c:\tmp\splinedata.txt’/; loop(ls,

*---------------------------------------* write data file for FITPACK.DLL *---------------------------------------put datafile; put ns:10:0,nf:10:0/; put nd:10:0,k:4:0,nest:10:0,s(ls),xlo,xup/; loop(d, put datax(d),datay(d),dataw(d)/; ); loop(p, put fno(p):10:0, 1:10:0, xno(p):10:0, yno(p):10:0/; ); putclose; *---------------------------------------* call solver which will call FITPACK.DLL *---------------------------------------solve m minimizing z using nlp; results(p,ls) = y.l(p); ); display results;

*---------------------------------------* write data file with results *---------------------------------------file splinegnuplotdata /’splinedata.txt’/; put splinegnuplotdata; loop(p, put x.l(p); loop(ls, put results(p,ls); ); put /; ); putclose; *---------------------------------------* write data file with data points *---------------------------------------file gnuplotdata /’data.txt’/; put gnuplotdata; loop(d, put datax(d),datay(d)/; ); putclose;

28

*---------------------------------------------------* Write gnuplot script for plotting the splines * Have it generate both a figure on screen (Windows) * and an encapsulated postscript file and a PNG * bitmap file. *----------------------------------------------------

file gnuplotscript3 /’spline.gnuplot’/; put gnuplotscript3;

put ’set key bottom right’/; put ’set terminal windows’/; put ’plot "data.txt" title "data points",’ ’"splinedata.txt" using 1:2 title "S=0.0" with lines lt 3,’ ’"splinedata.txt" using 1:3 title "S=1.0" with lines lt 1,’ ’"splinedata.txt" using 1:4 title "S=10.0" with lines lt -1’/; put ’pause -1 "OK"’/; put ’set terminal png color’/; put ’set output "spline.png"’/; put ’plot "data.txt" title "data points",’ ’"splinedata.txt" using 1:2 title "S=0.0" with lines lt 3,’ ’"splinedata.txt" using 1:3 title "S=1.0" with lines lt 1,’ ’"splinedata.txt" using 1:4 title "S=10.0" with lines lt -1’/; put ’set terminal postscript eps color solid’/; put ’set output "spline.eps"’/; put ’plot "data.txt" title "data points",’ ’"splinedata.txt" using 1:2 title "S=0.0" with lines lt 3,’ ’"splinedata.txt" using 1:3 title "S=1.0" with lines lt 1,’ ’"splinedata.txt" using 1:4 title "S=10.0" with lines lt -1’/; putclose;

*---------------------------------------------------* Now do a least squares fit *----------------------------------------------------

set j ’max set for the coefficients’ /j0*j15/; variables c(j) ’coefficients’; variables residual(d) ’residuals’; scalar degree; alias(j,jj); equation fit(d); fit(d).. sum(j$(ord(j)-1