estimation, the resulting equivalent LP is stated by Bartels, Conn and Sinclair [1]. In the general case, .... Either cp~~ or c5 ~ is basic for each i and all 1 = 1,.., tBi ;.
Mathematical Programming 53 (1992) 213-235 North-Holland
213
A simplex algorithm for piecewise-linear programming III: Computational analysis and applications Robert Fourer Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, IL 60208, USA
Received 9 April 1986 Revised manuscript received 15 July 1988; 14 March 1989; 17 August 1989 The first two parts of this paper have developed a simplex algorithm for minimizing convex separable piecewise-linear functions subject to linear constraints. This concluding part argues that a direct piecewiselinear simplex implementation has inherent advantages over an indirect approach that relies on transformation to a linear program. The advantages are shown to be implicit in relationships between the linear and piecewise-linear algorithms, and to be independent of many details of implementation. Two sets of computational results serve to illustarate these arguments; the piecewise-linear simplex algorlthm is observed to run 2-6 times faster than a comparable linear algorithm, not including any additional expense that might be incurred in setting up the equivalent linear program. Further support for the practical value of a good piecewise-linear programming algorithm is provided by a survey of many varied applications. Key words: Linear programming, simplex methods, piecewise-linear programming, nondifferentiable optimization.
I. Introduction The first two parts of this p a p e r [7, 8] have d e v e l o p e d a simplex a l g o r i t h m for m i n i m i z i n g convex separable piecewise-linear f u n c t i o n s subject to linear constraints. This c o n c l u d i n g part explores issues that are significant to the i m p l e m e n t a t i o n a n d a p p l i c a t i o n of the algorithm. The essential t e r m i n o l o g y for piecewise-linear p r o g r a m m i n g is r e c a p i t u l a t e d from [7] in the latter half of this i n t r o d u c t i o n . Section 2 t h e n defines the steps of the piecewise-linear simplex algorithm from [7, 8], a n d c o m p a r e s them with the corresp o n d i n g steps of the linear b o u n d e d - v a r i a b l e simplex algorithm. For a restricted form of the piecewise-linear algorithm that selects the leaving variable by a " s m a l l e s t ratio" test, the work of the two algorithms is virtually the same at every step. A relatively m o d e s t a d d i t i o n a l effort suffices to i m p l e m e n t a n u n r e s t r i c t e d " s m a l l e n o u g h ratio" test that is e q u i v a l e n t to a w e l l - k n o w n weighted selection p r o b l e m . This research has been supported in part by the National Science Foundation under grant DMS8217261.
R. Fourer / Simplex method for piecewise-linear programming
214
Section 3 describes four ways in which a piecewise-linear program can be transformed to an equivalent linear program, through introduction of additional variables or constraints. Section 4 then investigates a one-to-one correspondence between iterations of the piecewise-linear simplex algorithm and iterations of a linear simplex algorithm applied to any of the equivalent linear programs. This correspondence employs the restricted form of the piecewise-linear algorithm, and requires (in two of the four cases) that the linear algorithm be kept to a subset of "potentially optimal" bases. Section 5 considers the analogous correspondence involving iterations of the unrestricted piecewise-linear simplex algorithm. In general, each unrestricted iteration has the same effect as a series of linear simplex iterations. Section 6 surveys the many uses of piecewise-linear programs. Applications are identified in linear programming, data fitting, goal programming, interval programming and convex separable nonlinear optimization. The concluding remarks in Section 7 argue, based on the relationships derived in this paper, that a direct piecewise-linear simplex approach has inherent advantages over an indirect approach involving transformation to a linear program. These advantages are implicit in relationships between the linear and piecewise-linear algorithms, and are largely independent of the details of implementation. Two sets of computational results serve as illustrations; the piecewise-linear simplex algorithm is observed to run 2-6 times faster than a comparable linear algorithm, not including any additional expense that might be incurred in setting up the equivalent linear program.
Terminology A closed proper convex piecewise-linear (P-L) function [c/Y]k of xk is characterized (to within an additive constant) by an increasing sequence of breakpoints 7~h), together with an increasing sequence of slopes c~h). At least one slope or one breakpoint must be finite. For each finite c(kh), [c/7]kXk is linear with slope c~h) on the interval between y(kh) and y~h+l). Infinite slopes are interpreted as follows: c~~)=-oo
implies
[c/y]kXk
C~')=+Oe
implies
[C/y]kXk=00
=oo
for all X k < y ~ ~+1), for all x k > "/~').
Thus the slope-breakpoint sequences can effectively begin with either some c~s) = -o0 and ~/~~+~) finite, or some y ~ ' ) = - o o and c~') finite; they can effectively end with either some 7~') finite and c~') = +oc, or some c~'; finite and y~'+~)= +oo. A closed proper convex separable piecewise-linear function [c/y] of x = (xl, • •., xn) is defined by a sum of closed proper convex piecewise-linear functions:
[c/~]x= ~ [c/~]kx» k 1
A piecewise-linear program (P-LP) minimizes such a function over all x that satisfy
R. Fourer / Simplex method f o r piecewise-linear programming
215
certain linear equations: minimize
[ e / 7 ]x
subject to
A x = b.
The matrix A = [ a l , • • •, an] is m x n and m a y be taken to have linearly i n d e p e n d e n t rows. A solution ff to A x = b is feasible if [e/y]2~ is finite; thus 2~ is feasible if each Xk lies in an interval of finite slope of [C/7]kXk. A basis matrix B is an m x m nonsingular submatrix o f A. The vector x~ denotes the basic variables corresponding to the columns of B, and XN denotes the remaining nonbasic variables. An individual variable Xk is described as X~k when it is basic, and as XNk w h e n it is nonbasic. Related vectors and values are distinguished by an analogous notation: CBi is a slope associated with a basic variable, for instance, and 7N~ is a breakpoint associated with a nonbasic variable. A basis column is aßi, while a c o l u m n not in the basis is aN; Given a basis matrix B and a vector 7N o f finite breakpoints, a unique basic solution 2~ for the piecewise-linear simplex algorithm is defined by XN ~ ")/N~
B~B = b - ~ aNj'fNj. ùi
The values ~B o f the basic variables determine a vector of slopes ca, such that (h+l)
CBi = C(Bhi)
i f ")/~'i) ~ ÆBi ~ Y Bi
.
For all 27B~that lie strictly between breakpoints, cR« is uniquely determined; for any degenerate ~Ri = "/~~~, cB~ may be set arbitrarily to either e~di ~~ or c(d~), as explained in Part II of this paper [8]. A basic solution is necessarily feasible if a n d only if all c~~ can be chosen to be finite. The convex conjugate of [e/'/]kXk is a convex piecewise-linear function defined by [y\C]k~k = sup(~:~Xk -- [C/7]kXk). Xk
This function is linear with slope 7~h) on the interval between c~h 1) and c~h). The conjugate of [ c / y ] x is a convex separable piecewise-linear function,
b,\c]~= E [,/\c]~x» k-1
For every primal piecewise-linear p r o g r a m in the form given above, there is a dual P-LP, maximize
~-b-[y\c]«
subject to
~rA = ~:.
216
R. Fourer / Simplex method Jbr piecewise-linear programming
If either the primal or the dual P-LP has a finite objective value, then the minimum in the primal equals the m a x i m u m in the dual.
2. Corresponding steps of an iteration The steps of the piecewise-linear simplex algorithm are not much different from those of its counterpart for linear programs in bounded variables. The required computations are described in [7] and, for degenerate cases, in [8]. A detailed analysis and comparison is provided in the longer "technical report" version of this p a p e r [9]. The only computational differences of significance occur in the selection of a leaving variable. In any simplex iteration, the entering variable is pushed up or down from its initial value, and the basic variables are adjusted accordingly. For the piecewise-linear simplex, two versions of the leaving-variable selection rules are considered by the analysis in [9]: R e s t r i c t e d case. The leaving variable is selected as the first to reach one of its
breakpoints. Unrestricted case. Variables are allowed to cross some of their breakpoints in the course of an iteration. The leaving variable may be any orte that reaches a breakpoint while the objective value is continuing to decline.
At a restricted iteration, the leaving variable taust correspond to a certain smallest ratio, and the P-L simplex algorithm is computationally almost identical to the linear algorithm. An unrestricted iteration, in contrast, may select a leaving variable corresponding to any "small enough" ratio, so that a greater reduction in the objective value may be achieved. The small-enough ratio test turns out to be a case of the weighted selection problem, for which very fast methods are known.
3. Equivalent linear programs A piecewise-linear program can be transformed to an equivalent linear program in any of four distinct ways. Two of the transformations use both slope and breakpoint values; two employ function values and slopes or breakpoints. Both pairs are duals in a certain sense. Another kind of transformation serves to split a piecewise-linear term into two simpler ortes. When applied to a "semi-linear" program whose terms have at most three finite breakpoints and two finite slopes, this transformation produces a c o m m o n related kind of equivalent linear program. Each equivalent formulation is introduced below in its simplest and purest formulation. Extended and hybrid formulations are discussed at greater length in [9].
R. Fourer / Simplex method fi»r pieeewise-linear programming
217
Slope-and-breakpoint equivalents An equivalent linear program proposed by Dantzig [4] expresses Xk as a sum of variables A~~) that are bounded by the successive distances between breakpoints. If each P-L term [c/7]kxk has finite breakpoints y ~ l ) , . . , y~k,k+l~ surrounding finite slopes c ~ ~ ) , . . , C~k'k), then this " A - f o r m " linear program may be written as follows: n
minimize
subject to
{
-~- tl
0 , q)(k/~~>0, ,ek
~'k--dJ(k'~=«~')X~--[y\C]k«(k '~,
t~(k'~~O,
k=l,...,n, k=l,...,n,
l=l,...,tk, l=l,...,tk.
Then all constraints except simple bounds are equalities in all forms, and the number of basic variables equals the number of these constraints. In the A-form, a nonbasic variable may be at zero or at its upper bound; in the other forms, the variables constrained to be nonnegative may be nonbasic at zero, and the free variables are always basic. Property 1. A subset o f variables is basic for an equivalent L P if and only if it is chosen in the following way: A-Jbrm. One o f the variables A ~~ is basic f o r each i. c19-form. All Xk are basic. Either cp~~ or c5~~ is basic for each i and all 1 = 1 , . . , tBi ; either 'n(l) ~(~) -~ Nj or ,~ Nj is basic for each j and all but one I = 1 , . . . , tNj. A-form. Two o f the variables A (I) Bi are basic for each i; one o f the variables A (1) Nj is basic f o r each j. T-form. All Xk and qrk are basic. One o f the variables tb~~ is nonbasic f o r each i; two o f the variables v, Nj'l'(t)are nonbasic f o r each j. U The feasible bases of an equivalent LP have an even stronger characterization. Given a feasible basis for the P-L simplex algorithm, let hNj be indices of the nonbasic breakpoints YN, so that XNj = YNi = ~/~~~) and the basic solution is given by a
BxB=b-3~ J
(h~?
NJ7Ni
•
R. F o u r e r / S i m p l e x m e t h o d , f o r p i e c e w i s e - l i n e a r p r o g r a m m i n g
221
Let hBi similarly be indices of the breakpoints around the basic variables: (hB') (ht~i+l) , 7B~ ' "~ ~ )~ßi~< 7Bi
where - c o ~ C(tBiBi) ~ -~-OC).
Then for each of the equivalent LPs, there is a corresponding feasible basis that achieves the same objective: Property 2. Given a feasible basis for a piecewise-linear program, a corresponding feasible basis for each of the equivalent linear programs may be chosen in the following way: A-form. A~~ is non basic at upper bound for l < h m , A ~'ia~) is basic, and A(~~ is nonbasic at zero for l > hui. ~a(1)Nj is nonbasic at upper bound for l < hNi , and ~NJ a(~) is nonbasie at zero for l >~hNj: @-form. All Xk are basic. @~] is basic for l hrn. @(') NJ is basic for l < hNj , and ,~ w (i) Nj is basic for l > hNj. A-form. A (hal) and A (Bih B i + l ) are basic for each i; A (h'~j) is basic for eaeh j. ~~ ui NJ ~l«(hNj l ) q'-form. All Xk and ~k are basic. Only O~'i~') is nonbasic ]or each i; only "~ N) and .r Nj are nonbasic for each j. I f Y, is the P - L P ' s basic feasible solution, then the values o f the basic variables in the corresponding L P solutions are: (hR~) A-form. ~(h~,)Bi= ffui -- YBi " @-form. Xk = 2k, ~~l~ = Xk
A-form.
_
7kU ) f o r l h k. Ä (hB`) ~ ( h a + l ) \/« ~~ßi :[')/Bi' ~(hBi÷l) 13i
(hu+l)
- (h/3,)"~
--XBi)/[')/Bi ' -- ,VBi ]' (ht3 ) x / / (hB + 1 ) (hl31)"~
= ( Æ u i - - ")/Bi ' ) / L ' Y B i '
-- YBi
1,
Ä~hj~j) = 1.
~F-form. Xk = 2k,
¢'~ = [ c / ~ ] ~ x ~ , ~~'~ = [ c / ~ ]~~~ +
[ ~ \ c ]~c~ '~ - c~'~x »
All of these solutions attain objective values equal to [ c/ y]ff.
[]
The above properties provide only sufficient conditions for bases of the equivalent LPs to be feasible. In the cases of the @-form and qr-form, these conditions are also necessary: Property 3a. A basis for the @-form or gr-form is feasible if and only if it corresponds to a feasible P-L basis as speeified in Property 2. [] In the cases of the other two forms, a basic solution can be feasible even though it does not satisfy Property 2. Given any such solution, however, one can always find another feasible solution that achieves a lower value in the linear objective function. Thus only the bases constructed in Property 2 are potentially optimal, in the sense that they can be the optimal bases for some objective.
222
R. Fourer/ Simplex methodfor piecewise-linearprogramming
Property 3b. A basis for the A-form or A-form is feasible and potentially optimal if and only if it eorresponds to a feasible P-L basis as specified in Property 2. [] The existence of necessary and sufficient conditions suggests a one-to-one correspondence, and indeed such a relationship exists for potentially optimal basic feasible solutions. Care must be taken in extending this correspondence to the bases themselres, however, owing to the ettects of degeneracy. The number of corresponding LP bases may grow exponentially in the number of degenerate variables; a one-to-one relationship can be maintained only if a basis is imagined to consist of a matrix B and breakpoints YN together with some choice of basic slopes cB:
Property 4. (i) For a feasible P-L basis that has S variables lying directly at breakpoints surrounded by finite slopes, Property 2 defines 2 s corresponding potentially optimal bases of the equivalent linear programs. (ii) There is a one-to-one correspondence between the feasible bases of a P-LP together with basic slopes cR, and the potentially optimal bases of any equivalent LP. (iii) There is a one-to-one correspondence between the basic feasible solutions to a P-LP, and the potentially optimal basic feasible solutions to any equivalent LP. []
The possibly large number of degenerate LP bases is not significant to the remainder of this section, where only the restricted ratio test is considered. The unrestricted P-L simplex algorithm does confer important advantages under degeneracy, however, as Section 5 will show.
Corresponding iterations A P-L simplex iteration can be considered equivalent to a linear iteration if they begin at corresponding feasible bases and end at corresponding feasible bases, in the sense of Property 4. Thus the principal aim of this section can be restated as follows: show that for every restricted P-L iteration there is an equivalent linear iteration, and that for every linear iteration between potentially optimal bases there is an equivalent P-L iteration. A subsidiary goal is to show that, for the A-form and A-form, the limitation to potentially optimal bases is both practicable and desirable. The correspondence between iterations is established by the following two properties, which consider first the entering variable and then the leaving variable. Indices ht~i and hNj are used as before to denote current breakpoints and slopes.
Property 5a. Let B, YN and cR define a feasible basis for a P-LP, and let a potentially optimal basis for any equivalent L P correspond as in Property 2. Then XNj can enter the P-L basis by (i) increasing or (ii) decreasingfrom YNi if and only if the following
R. Fourer/ Simplex method for piecewise-linearprogramming
223
variables can enter the corresponding basis o f the equivalent LP: A-form. (i) A(h~~~; (ii) A(hNiNj 1), f r o m upper bound. (ii) a,(hNJ) cP-form. (i) d- - NCh~? j "~ "4"Nj "
A-form. (i) A~f'+'); (ii) a~~Ni (h~-l) i . l«(hNi -1 ) vff-form. (i) +"rNj ; (ii) v,I,(h~j~ Nj • The P-L basis is optimal if and only if none o f these variables can enter, f o r any j.
[]
Property 5b. I f XNp is selected to enter the P-L basis, in the situation described by Property 5a, then the restricted P-L simplex algorithm can select XBq to leave the basis (at 7~~ ,,+l~ or 7~~~ ~) or can select XNr to leave the basis (at 7~~, ,+1~ or 7~~, ,-1)) if and only the variables o f the equivalent LPs which are shown in Table 1 can be selected to leave. The P-L objective can be decreased without bound if and only if none o f these variables can be selected to leave. [] Table 1 XBq
leaves at
xNp leaves at 7Bq'
3'~i,~'+~~ A~jT,,)*
A%'~?,,-I)
¢~i';'+~~
~" Nl~~~ -
A ~,~,p) ~ + qt(N~,
A %'17,,)
A-form @-form
A~hfù~* a'(h» + ) ~B,I
A~'f,, ~ eh(h-»4" )
A-form
A ~~B,,)
A ~',[,,,+')
/ ( I , +1) qJ B,]"
dt(
-form
v BqB' -
)
)
I
.I,(hNp--2)
v Np
For the A-form, variables marked * leave at upper bound. T a k e n together, these properties establish the desired relationship b e t w e e n iterations: Property 6. There is a one-to-one correspondence between restricted iterations o f the P-L simplex algorithm and equivalent iterations o f the linear simplex algorithm that begin and end at potentially optimal bases. [] Property 6 naturally extends to sequences of equivalent iterations, and to iterations that declare optimality or u n b o u n d e d n e s s . Thus there is a c o r r e s p o n d e n c e b e t w e e n iteration paths of the two algorithms. For every path to a solution in the rstricted P-L simplex algorithm, there is a corresponding linear simplex path on any of the equivalent LPs; for any path through potentially optimal bases in the linear algorithm, there is a corresponding path in the P-L algorithm. If the linear algorithm is allowed to visit feasible bases of the A-form or A - f o r m that are not potentially optimal, then it can p e r f o r m iterations that are not equivalent to restricted P-L simplex iterations. Thus in a sense the linear algorithm is m o r e general. H o w e v e r , when an iteration begins at a potentially optimal basis, the most attractive entering variables, in terms of either reduced cost or edge steepness [10], are those that would be chosen by Property 5a's limited criteria:
224
R. Fourer / Simplex method jor piecewise-linear programming
Property 7. A t a potentially optimal A-form basis, A ~~ has a nonpositive reduced cost f o r every 1 < hBi, and a nonnegative reduced cost f o r every 1> h~i. I«f ~A ~1) N..] is eligible to enter the basis, then it has a less positive reduced cost and edge steepness than A (h~j-1) Nj if l < hNj - 1, or a less negative reduced cost and edge steepness than A (hN~) Nj if l > h~i. t) A t a potentially optimal A-form basis, A• (B~ has a nonnegative reduced cost for every --(l) l < hNj and l > hNj + 1. I f A Nj is eligible to enter the basis, then it has a less negative reduced cost and edge steepness than A Ni if l < hNj -- 1, or a less negative reduced cost and edge steepness than A ~~J +~~ if l > hNj + 1. [] These results imply that, if the linear simplex algorithm starts at or reaches a potentially optimal basis for the A-form or A-form, then it will stay at a potentially optimal b a s i s - - u n l e s s it selects some entering variable that is clearly not the most favorable in terms of practical criteria. Such a property does not extend to the criterion of total reduction in the objective, but total reduction is too costly to be computed for more than a small fraction of potential entering variables. N u m b e r o f iterations
As characterized by Property 6, the restricted P-L simplex algorithm can be viewed as a linear simplex method that has been streamlined to exploit the equivalent LPs' special structures. The P-L algorithm dispenses with explicit definitions of supplementary variables and constraints, and insures automatically that only potentially optimal bases are considered. Since the qs-form and q'-form have only potentially optimal bases, there is no reason to solve these forms by a linear simplex algorithm when the P-L algorithm is available. The restricted P-L iterations are at least as efficient, and can follow the equivalent of any path the linear iterations might take. The same argument can be made for the A-form and A-form, provided that no advantage is lost in restricting the linear algorithm to visit their potentially optimal bases. Certainly, by Property 7, once the algorithm reaches a potentially optimal basis then it can only sensibly follow a path of such bases. Moreover, a brief analysis suggests that there is little to be gained by starting the A-form or A-form from a basis that is not potentially optimal. For the zl-form, any basic solution defines a corresponding solution of the P-LP by ~k =YI'z~ Ä~t/. If the A-form basis is not potentially optimal, then ~ is not in general a basic solution. Thus the linear algorithm can visit solutions that the P-L algorithm cannot, and it might benefit as a result. On the other hand, a basis that is not potentially optimal can be many iterations away from any orte that is. If y~h,)
