The Cosine Simplex Algorithm H. W. Corleya, Jay Rosenbergera, Wei-Chang Yehb , and T. K. Sunga a
Department of Industrial and Manufacturing Systems Engineering The University of Texas at Arlington Arlington, TX 76019-0017
[email protected],
[email protected],
[email protected] b
e-Integration & Collaboration Laboratory Department of Applied Mathematics National Chiayi University P.O. Box 67-100, Taichung, Taiwan 408, R.O.C.
[email protected]
Abstract: An extension of the simplex algorithm is presented. For a given linear programming problem, a sequence of relaxed linear programming problems is solved until a solution to the original problem is reached. Each successive relaxed problem is obtained from the previous one by adding a single constraint chosen from the constraints violated by the solution to the previous relaxed problem. This added constraint maximizes the cosine of the angle that the gradient of any violated constraint forms with the gradient of the objective function. In other words, each successive relaxed problem is obtained by adding the violated constraint most parallel to the objective function. The proposed algorithm terminates when no constraints are violated. Preliminary results indicate that this cosine simplex algorithm reduces both the number of simplex iterations and the number of computations at each iteration. Key Words: optimization, linear programming, simplex algorithm, cosine simplex algorithm
1
2
1.
Introduction Consider the linear programming problem maximize z = cTx
(1)
subject to Ax ≤ b x ≥ 0, where x, A, b, c, 0 are n×1, m×n, m×1, n×1, n×1 respectively. It will be assumed that (1) has an optimal solution. This standard linear programming model (1) is an indispensable tool in today’s manufacturing environment. Dantzig’s simplex method [1] provided the first effective solution technique, but numerous other algorithms have been proposed. For example, feasible direction methods have been proposed by Brown and Koopmans [2], as well as by Murty and Faithi [3], among others. Megiddo [4] has tried to reduce the number of constraints through a multidimensional search technique. Khachian’s ellipsoid algorithm [5] first established that linear programming problems can be solved in polynomial time, but it performs poorly in practice. Karmarkar [6] later developed a polynomial projection approach that is used in some applications. However, the simplex algorithm remains the underlying algorithm utilized by most commercial linear programming packages (e.g., see [7]). While the simplex method is not a polynomial algorithm, the average running time is polynomial; and a problem such as (1) with m constraints seldom takes more than 3m iterations [8]. Moreover, the simplex method allows efficient post-optimality analysis and thus readily adapts to branch-and-bound algorithms for the binary and integer models common in industrial applications. On the other
3
hand, interior-point methods such as Karmarkar’s projection algorithm are not suitable for such uses. A new algorithm, an extension of the simplex method, is proposed here. The essential idea behind the algorithm comes from a class of unconstrained optimization problems in abstract Hilbert spaces that are solved by the Schwarz inequality (see [9]) when two vectors are collinear (i.e., the cosine of the angle between them is zero). These problems involve an inner product and lead to a consideration of the angles formed by the objective function and the individual constraints in (1). The cosine simplex algorithm is summarized as follows. For a given problem it begins by solving a relaxed problem consisting of the original objective function subject to a single constraint yielding a nonempty bounded feasible region. At each subsequent iteration of the algorithm, the most parallel constraint to the objective function among those constraints violated by the solution to the current relaxed problem is appended to it. The dual simplex algorithm is then applied. When no constraints are violated, the solution of the current relaxed problem is optimal to the original problem. Recent work by [10] and [11] have used a similar cosine criterion simply to get an initial basis for the simplex algorithm, thereby reducing the number of required simplex iterations. However, they first need to eliminate any redundant constraints. In contrast, our approach uses the cosine criterion at each interation, and the current relaxed relaxed problem involves only a fraction of the original constraints. In Section 2 of this paper we describe the cosine simplex algorithm in more detail, then give its basic properties in Section 3. Some simple illustrative examples are presented in Section 4. Finally, in Section 5, we discuss future research.
4
2.
The Cosine Simplex Algorithm Denote row i of the matrix A by ai so that constraint i of (1) becomes ai x ≤ bi, i = 1, …, m.
(2)
In the cosine simplex below, assume that (1) has a nonempty feasible region with a constraint r in (2) for which arj > 0, j = l, …, n, and br > 0. These assumptions guarantee that
aiT c (1) has an optimal solution. For constraint i of (2), define cos θi = as the cosine of ai c the angle θi between the normal vectors ai for constraint i and c for the objective function. We call a constraint operative if it is part of the current relaxed problem and inoperative otherwise.
The initial relaxed problem maximize z = cTx
(3)
subject to
arx ≤ br x≥0 obviously has an optimal solution. The algorithm can now be stated.
The Cosine Simplex Algorithm. Step 0. Compute cos θi =
aiT c , i = 1, …, m, i ≠ r, and order the constraints ai c
according to decreasing cos θi, where ties are broken arbitrarily.
Step 1. Solve (3), which becomes relaxed problem 1, to obtain x1. Set k = 1. Step 2. Check the inoperative constraints in decreasing order of cos θi. Take the first one violated by xk and go to Step 3. If none is found, stop since xk solves problem (1).
5
Step 3. Set k = k+1. Append the violated constraint to the final tableau of relaxed problem k to obtain relaxed problem k+1. Apply the dual simplex algorithm to obtain a solution xk. Go to Step 2.
3.
Discussion of the Cosine Simplex Algorithm The cosine simplex algorithm for (1) is an active set method [12] that fundamentally
extends the simplex algorithm by determining an entering constraint in addition to entering and leaving basic variables. Furthermore, the cosine simplex appears to converge faster in practice than the usual simplex method for four reasons. First, the cosine criterion for appending a constraint reduces the total number of iterations in small examples. Second, at each iteration only a small fraction of the original constraints will typically be involved in the simplex computations. Third, relatively few constraints of a linear programming problem are usually active at optimality [13]. Fourth, no artificial variables are required. In addition, the cosine simplex algorithm eliminates the need for identifying redundant constraints as in [10] and [11]. In fact, one may add the artificial and possibly redundant constraint x1 + … + xn ≤ M, where M is a large positive number, if (1) has no bounding constraint arx ≤ br for the initial problem (3). A geometric interpretation of the new method is that Step 2 determines the inoperative constraint violated by the current solution that is "most nearly parallel" to the objective function. This cosine criterion appears to determine more quickly than the standard simplex algorithm the active constraints such that the gradient of the objective function lies in the cone generated by their gradients. Equivalently, it adds active constraints until the Kuhn-Tucker conditions [8], which are both necessary and sufficient for (1), are satisfied.
6
4.
Examples Examples are now presented to show the efficiency of the cosine simplex algorithm
for small problems as well as its limitations. Example 1 is given in detail, while Examples 2, 3, 4, 5 are only summarized.
Example 1. Consider the problem maximize z = 4x1 + 5x2 + 9x3 + 11x4 subject to 3x1 + 5x2 + 10x3 + 15x4 ≤ 100
(4)
x1 + x2 + x3 + x4 ≤ 15
(5)
7x1 + 5x2 + 3x3 + 2x4 ≤ 120
(6)
x1 , x2 , x3 , x4 ≥ 0 The values to two decimals for cos θi in Step 0 for (4), (5), (6) are 0.99, 0.93, and 0.70, respectively. Each of these constraints bound the problem, so Step (1) can be omitted. Since the associated cos θi for (4) is a maximum, the first relaxed problem to be solved is thus maximize z = 4x1 + 5x2 + 9x3 + 11x4
(7)
subject to 3x1 + 5x2 + 10x3 + 15x4 ≤ 100 x1 , x2 , x3 , x4 ≥ 0 Solving problem (7) and proceeding with the cosine algorithm give the following sequence of tableaus. Constraint (5) becomes operative in the second tableau. In the third tableau constraint (6) is not violated by the solution z = 695/7, x1 = 50/7, x2 = 0, x3 = 55/7, so these
7
values are optimal to the original problem. The variables x5 and x6 below are the slack variables for the entering constraints (4) and (5), respectively. basis
z
x1
z
1
−4
−5
−9
−11
0
0
x5
0
3
5
10
15
1
100
z x1 x6
1 0 0
0 1 0
1 0 0
0 1 0
13/3 10/3 −7/3 0 0 1
9 5 4
z x1 x3
5/3 5/3 −2/3 9/21 15/21 2/7
x2
x3
x4
x5
x6
4/3 1/3 −1/3 11/7 15/21 −5/7 -3/21 4/7 1/7
rhs
0 0 1 13/7 10/7 −3/7
400/3 100/3 −55/3 695/7 50/7 55/7
The standard simplex algorithm, on the other hand, requires four tableaus, including the initial one. To compare the computational efficiency of the two algorithms for the above p
problem we compute for the cosine simplex C =
∑ (# of tableaus with k constraints) × k , k =1
where p is the number of the number of constraints in the optimal tableau. For the standard simplex we compute S = (# of tableaus required by the standard simplex for a solution) × m. In this example, C = 5, while S = 12. Example 2. This example demonstrates that the constraint most nearly parallel to the
objective function may not be active at optimality. The solution to the problem Maximize z = −x1 − x2 + 20x3 subject to x1 + x2 + 20x3 ≤ 120 −x1 + x2 + x3 ≤ 4 x1 − x2 + x3 ≤ 5
(8)
8
x1, x2, x3 ≥ 0 is z = 179/2, x1 = l/2, x2 = 0, x3 = 9/2. However, the nonredundant constraint (8) is most nearly parallel to the objective function but not satisfied as equality by this solution. Example 3. Consider the problem
maximize z =
−x1 − x2 + 20x3
subject to x1 − x2 + 20x3 ≤ 20 −x1 + x2 + x3 ≤ 4 x1 − x2 + x3 ≤ 5 −x1 − x2
≤ −40
x1, x2, x3 ≥ 0 Applying the cosine simplex algorithm to this problem illustrates that no artificial variables are required and that the selection criterion for constraints does not prevent the dual simplex algorithm from sometimes requiring more than one iteration in Step 3. Thus the cosine simplex algorithm appears to have the convergence properties of the simplex method. Example 4.
The next problem shows that the cosine algorithm may prevent cycling
without any additional analysis such as the lexicographic exiting rule. Consider maximize z = (3/4)x1 − 150x2 + (1/50)x3 − 6x4 subject to (1/4)x1 − 60x2 − (1/25)x3 + 9x4 ≤ 0 (1/2)x1 − 90x2 − (1/50)x3 + 3x4 ≤ 0 x3
≤1
9
x1, x2, x3, x4 ≥ 0. This problem can be shown to cycle [14] if the variable with the smallest subscript is chosen to leave the basis in case of ties. The cosine algorithm solves this problem with no cycling to obtain the optimal solution z = 20, x1 = l/25, x2 = 0, x3 = 1, x4 = 0. Example 5. The final example shows the cosine simplex is substantially more efficient
than the standard simplex in solving a Klee-Minty problem [8] for which the standard simplex requires one iteration for each extreme point of the feasible region. Consider the following Klee-Minty problem from [15] maximize z = 4x1 + 2x2 + x3 subject to 8x1 + 4x2 + x3 ≤ 125 4x1 + x2 x1
≤ 25 ≤5
x1, x2, x3 ≥ 0 with optimal solution z = 125, x1 = x2 = 0, x3 = 125. The standard simplex algorithm solves this problem in 8 iterations with the value S = 24, while the cosine simplex takes only 1 iteration with the value C = 1.
5.
Conclusions An extension of the simplex algorithm was presented in this paper. For small linear
programming problems the cosine simplex algorithm solves them more efficiently than the standard simplex method, eliminates the need for artificial variables, and deals with redundant constraints. In addition, it can prevent cycling and readily handle Klee-Minty
10
problems. In subsequent work, we will investigate various modifications to the cosine simplex algorithm that might enhance it. For example, we could include the cosine values for the nonnegativity restrictions − xj ≤ 0, j = 1, …, n, in addition to (2). However, this change will still not guarantee that the most parallel constraint to the objective function is active at an optimal solution. In Example 2, for instance, the nonnegativity restriction −x3 ≤ 0 is actually more parallel to the objective function than (8), yet it is not active at optimality either. In addition, we shall also try to improve on the cosine criterion, which has considerable geometric appeal for n = 2. Finally, we will apply the results to large-scale problems.
6.
Acknowledgement
This research presented here was partially supported by the Texas Advanced Research Program under Grant No. 003656-0197-2003.
References [1] G.B. Dantzig, “Maximization of a linear function of variables subject to linear inequalities,” in T.C. Koopmans, editor, Activity Analysis of Production and Allocation, John Wiley, New York, 339-347 (1951). [2] G.W. Brown and T.C. Koopmans, “Computational suggestions for maximizing a linear function subject to linear inequalities,” in T.C. Koopmans, editor, Activity Analysis of Production and Allocation, John Wiley, New York 377−380 (1951). [3] K.G. Murty and Y. Faithi, “A feasible direction method for linear programming,” Operations Research Letters 3, 123-127 (1984). [4] N. Megiddo, “Linear programming in linear time when the dimension is fixed,” Journal
11
of the Association of Computing Machinery 31, 114−127 (1984). [5] R.G. Bland, D. Goldfarb, and M.J. Todd, “The ellipsoid method: a survey,” Operations Research 29, 1039-1091 (1981). [6] N. Karmarkar, “A new polynomial-time algorithm for linear programming,” Combinatorica 4, 373−395 (1984). [7] http://www.ilog.com/products/cplex/ [8] M.S. Bazaraa, J.J. Jarvis, and H.D. Sherali, Linear Programming and Network Flows, 3rd edition., John Wiley, New York (2004). [9] B. Naylor and G. Sell, Linear Operator Theory in Engineering and Science, Springer−Verlag, New York (2000). [10] H.V. Junior, M. P. E. Lins, “An improved initial basis for the simplex algorithm,” Computers and Operations Research, in press, available at www.sciencedirect.com. [11] N.V. Stojkovic, P.S. Stanimirovic, “Two direct methods in linear programming,” European Journal of Operational Research 131, 417-439 (2001). [12] P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization, Academic Press, New York (1981). [13] G.B. Dantzig, Linear Programming and Extensions, Princeton University Press, Princeton, N.J. (1963). [14] H.M. Wagner, Principles of Operations Research, second edition, Prentice−Hall, Englewood Cliffs, N.J. (1975). [15] http://carbon.cudenver.edu/~hgreenbe/glossary/notes/Klee-Minty.pdf
