Lagrangian Techniques for NP-Complete Problems with Application to

Lagrangian Techniques for NP-Complete Problems with Application to ILP, Satis ability, and TSP Yao-Jen Chang Department of Electronic Engineering, Chun Yuan Christian University, Chung-Li, Taiwan 320. In this paper, we creat a continuous dual space where a dynamic system characterizing an NP complete problem evolves. The combinatorial problem, when represented in continuous variables, enables the Lagrange method to provide a trap avoidance mechanism and ensure integer solutions when equilibrium has been established. Experimental results show problems of relatively large sizes can be solved on line if run on the proposed dynamic system. Even if the special purpose architecture is not available, many problems such as a 0-1 ILP FEASIBILITY problem with 384 variables and 556 constraints have been solved faster by simulation than by the branch-and-bound method.

1 Introduction NP-complete problems are fundamental to many application domains as well as to the development of computer scienti c disciplines. ILP is a very general framework where diverse NP-complete problems can be formulated. In this paper, we deal with a dicult special case of ILP: feasibility of 0-1 ILP. The problem is to nd a feasible solution to an ILP problem where variables are required to binary (i.e. 0,1). It can be shown that the problem remains NP-complete. In fact, SATISFIABILITY transforms straightforward to this problem and the decision version of Traveling Salesperson Problems (TSP), with some tricks, can also be formulated as such a feasibility problem. The transformations of SATISFIABILITY and TSP will be shown in Section 3. The 0-1 ILP FEASIBILITY problem with n variables constrained by m equalities can be stated in the following. De ne A to be an m by n integer matrix and an m-tuple b of integers. The problem can be stated as: (ILPF) Does there exist a vector x such that

Ax = b xi 2 = xi ; i = 1; 2;


; n:

(1) (2)

2 Lagrangian Formulation of 0-1 ILP Now we come to the continuous formulation of 0-1 ILP FEASIBILITY: (CM) min f (x) = kAx ? bk22 +

n X i=1

(xi2 ? xi )2

Ax = b xi 2 = xi ; i = 1; 2;


; n:

(3) (4) (5)

The Lagrange multiplier approach [1] is based on solving the system of equations which constitute the necessary conditions of optimality for the programming problem.

[De nition] The Lagrange function of CM is de ned by "

L(x; ; ) = c kAx ? bk + 2 2

n X i=1

#

(xi ? xi ) + (Ax ? b) + 2

2

n X i=1

i (xi 2 ? xi )

(6)

where c > 0 is a parameter and  2 Rm and  2 Rn are referred to as Lagrange multipliers. Denote

5xL(x; ; ) =



@L(x; ; ) ; @L(x; ; ) ;


; @L(x; ; ) T @x1 @x2 @xn

(7)

The CM formulation takes advantage of the fact that local optima are also global. Therefore, the proposition above suces to serve as a guide in nding a solution for ILPF. In particular, to solve CM by Lagrangian techniques, we set up the following associated Lagrange problem by employing the rst-order necessary condition. (ALP)

5xL(x; ; ) = 0 5L(x; ; ) = 0 5L(x; ; ) = 0

(8) (9) (10)

The problem remaining is how to solve ALP. The traditional approach would be to solve the 2n + m nonlinear equations de ned by (8)-(10) with n + munknowns, via digital computers. However, this approach will work well for a lower dimensional problem, but not for a high dimensional one because the limited speed and memory capacity of a computer cannot handle a problem of interesting size. Here we employ the method proposed in [2] and [3] which suggest the solving of a general Lagrange problem by taking a gradient descent for primal variables (i.e., those appearing in the original constrained problems) and a gradient ascent for dual variables (i.e., Lagrange multipliers). In our case, the dynamic system turns out to be the following.

dx = ?5 L(x; ; ) x dt d = 5 L(x; ; ) = Ax ? b  dt d = 5 L(x; ; ) = hx 2 ? x ; x 2 ? x ;


; x 2 ? x iT 1 1 1 1 n n  dt

(11) (12) (13)

3 Illustrative Examples 3.1 Experiments on 3SAT Problems The proposed method is applied to several example problems. The rst example problem with 30 variables is formulated from a 3SAT problem by a polynomial time reduction. Example 1: Find a binary solution (x,z) such that a 3SAT problem with 6 variables in 12 clauses is satis ed. (l1 _ l2 _ l4 ) ^ (l1 _ l2 _ l4) ^ (l2 _ l3 _ l6) ^ (l1 _ l2 _ l4 ) ^ (l4 _ l5 _ l6 ) ^ (l1 _ l4 _ l6) ^ (l1 _ l5 _ l6) ^ (l1 _ l2 _ l6 ) ^ (l1 _ l3 _ l6) ^ (l4 _ l5 _ l6) ^ (l2 _ l4 _ l5 ) ^ (l2 _ l4 _ l6) (14) The transformation is derived from [4]. A branch-and-bound process would involve solving 31 LP subproblems for this example. Although the number of simplex routine calls may depend on the strategies employed in the branch

1.5 1.25 1 0.75 0.5 0.25 0 -0.25 -0.5 1.5 1.25 1 0.75 0.5 0.25 0 -0.25 -0.5

50 100 150 200 250

50 100 150 200 250

1.5 1.25 1 0.75 0.5 0.25 0 -0.25 -0.5 1.5 1.25 1 0.75 0.5 0.25 0 -0.25 -0.5

50 100 150 200 250

50 100 150 200 250

1.5 1.25 1 0.75 0.5 0.25 0 -0.25 -0.5 1.5 1.25 1 0.75 0.5 0.25 0 -0.25 -0.5

50 100 150 200 250

50 100 150 200 250

Figure 1: The transient behavior of x = (x1; x2;


; x6) in Example 1 when c is set equal to 5. c 0.05 0.1 1 3 5 7.5 20 200 time 500 400 350 75 120 150 250 2500 Table 1: Convergence times observed in Example 1 under dierent c values. and bound, it is a general measure of diculty of a particular problem being studied. Following our approach, the ILPF is transformed into a CM, and then the Lagrange multiplier method is applied to form the dierential dynamic system. One of the well-established dierential equation solvers available is Livermore Solver for Ordinary Dierential Equations 1 which is used in our study. Starting from (x; z ) = ( 21 ; 21 ;


; 12 ) for the reason of symmetry and setting c = 5, the dynamic system de ned by (11)-(13) gives a trajectory as shown in Figure 1. The trajectory suggests a solution x = (0; 0; 1; 0; 0; 0) which satis es the underlying 3SAT problem. We see that a drastical transition has occurred before a stable equilibrium is reached. Obviously, the search is nonlocal in the sense that a component does not experience change without simultaneous changes occurring to other components. The approach is promising not only because of its potential to solve NP-complete problems but also because of the speed it has demonstrated. We see that the system is stablized within 120 time constants2. For a more optimized choice of c = 3, the convergence time is less than 70 time constants. We have chosen values of c out of a wide range from 0.05 to 200. It appears that small c gives rise to underdamping while large c causes overdamping, both cases having shown somewhat longer convergence times. Table 1 summarizes more convergence time results for Example 1 under a set of dierent c values. Based on our studies, the system is found very robust in the sense that convergence does not depend on initial conditions, though dierent initial settings will, in general, lead to dierent equilibrium 1 Livermore Solver for Ordinary Dierential Equations is the basic solver of ODEPACK developed in Lawrence Livermore National Laboratory. 2 Time constants refer to the time scale that a dynamic system evolves with. Large time constants imply slow transition, and vice versa.

ILPF

c

70 variables 30 constraints

0.1 5 10 0.1 5 10 5

99 variables 44 constraints

230 variables 100 constraints 350 variables 5 500 constraints 10 384 variables 10 556 constraints

logical # primals time in B&B 800 634 200 320 1200 1009 400 480 360 > 64950 730 410 1180

N/A N/A

Table 2: Performance results of 3 larger ILPF problems. results. We also see that initial conditions have not caused signi cant dierence in convergence times or transient behaviors. Example 2: In this example, we run 5 more problems with larger sizes, which are summarized in Table 2. The number of primal simplex routine calls in solving the problems by the branch-and-bound is a measure of the intrinsic diculty of a particular problem instance at hand. Dierent c values are tried so that dierent kinds of transient behavior can be observed. The logical time in Table 2 accounts for time constants taken by simulated systems to arrive at an equilibrium. In Figure 2, we observe nonlocal search because a local barrier is broken near t = 400. The speed, in general, will not be sensitive to the increase of problem size. That is, a large scale problem with a signi cant number of variables and equations de nes a large dynamic system which will take the amount of time comparable to that of a small one. In other words, the Lagrange system causes dierence in hardware scales rather than running times. Therefore, parallelism can be exploited in an eective way that is unusual for digital computers. Even though no hardware has been available in our study, the simulation time for the system with 230 variables and 100 constraints is 4 hours and 8 minutes on a Sun Sparc 10 Station. On the same machine 12 hours has been allowed to carry out 64950 simplex routine calls in the branch and bound process and the branching is still continuing when allocated time expires.

3.2 Experiments on TSP Problems The TSP problems solved in this paper are non-Euclidean, asymmetric and therefore they represent the most dicult problems of their kind for approximation algorithms to obtain satisfactory results. The following formulation of TSP is based on A. W. Tucker [5]. For an N -city problem, denote the link from city i to city j as xij which is 1 if it is on the tour and 0 otherwise. To avoid subcycles from coming up as a solution, nodal variables are introduced which are denoted n1 ; n2 ;


; nN . Furthermore, for any connected pair of cities there is a weight wij > 0. We arbitrarily set an upper bound b and formulate the following decision version of the TSP problem. X

j 6=i

X

i6=j

xij = 1 8i = 1;


; N

(15)

xij = 1 8j = 1;


; N ? 1

(16)

ni ? nj + (N ? 1)xij  (N ? 2) 8i 6= j X wij xij  b i;j

(17) (18)

Obj 1000

800

600

400

200

200

400

600

800

1000

TC

Figure 2: Objective value vs. time constant plot for the 350-variable 500-constraint ILPF problem with c = 10.

xij (1 ? xij ) = 0

(19)

If there is a valid tour with cost less than or equal to b, then (15)-(19) is feasible. Thus, we obtain a mixed integer linear program where xij is binary and ni is real. In Table 3, dierent random seeds are used to generate dierent problem instances of the same size. The optimum cost of the test problems are indicated for veri cation purposes. The cost upper bound b is varied with respect to all problems. As a by-product of the yes-or-no questions, we obtain a satisfying cost value that the tour presents. Given an upper bound b greater than or equal to the optimum cost, the solution always return a cost which is less than or equal to b, indicating a 'yes' answer to the decision problems. The convergence times are all in terms of time constants characteristic of the underlying dynamic systems. Two precision control schemes have been experimented, one (Tol = 10?4 ; Prs = 4) and the other (Tol = 10?6 ; Prs = 16). Tol is a parameter in LSODE for accuracy control. Smaller Tol values tolerate less errors and demand more computation. Prs is the number of digits that represent a real number in simulation. The results of applying Lagrangian dynamic systems to TSP decision problems reveal a few important points. 1. Convergence time is less dependent on problem sizes. 2. Convergence time is not sensitive to the upper bounds set. 3. Precision and error control in the simulation is not demanding. The rst observation is consistent with the results of 3SAT. The dynamic system is basically a nonlinear system evolving under the in uence of attractors. We nd the problem size determines little control of attraction patterns. We consider the diameter of the space as a square root function of the dimension. Therefore, the traveling distance is not signi cantly lengthened with a large dimension, whereas for discrete optimization the dimension can increase the number of local solutions exponentially and make the problem prohibitively dicult. To illustrate this a little further, we run the same problems with a guided depth rst search algorithm3 [6]. The results are shown in Figure 3. For comparison purposes, the time constant of the dynamic system is assumed to be a relatively slow one which is 1 ms. We see the Lagrangian problem is more tractable than discrete search. The run times are mostly invariable over the range of size considered. 3

The algorithm solves the optimization version of TSP problem instead of its decision version.

convexity c = 10

TSP

Tol = 10?4 Prs = 4

city seed opt b (upper cnvg cost bound) time 10 1 13 13 19107 20 779 26 1719 10 2 11 11 4679 13 799 16.5 759 10 3 14 14 1999 16.8 13719 20 2759 12 1 14 14 5279 16 4699 18 2379 12 2 15 15 4119 18 5679 20 1999 22.5 1279 12 3 16 16 9999 19.2 1899 20 16445 15 1 16 16 215151 15 2 16 16 108641 32 10594 15 3 16 16 4777 24 1760

sol

13 16 21 11 13 15 14 14 18 14 15 17 15 16 19 21 16 17 19 16 16 31 16 22

Tol = 10?6 Prs = 16

cnvg time 9439 1359 3859 4679 799 759 3239 13619 1319 3419 6999 4979 7739 1939 4356 1659 6259 2499 4959 7899 10379

Table 3: Experiment results of TSP problems.

sol

13 14 23 11 13 15 14 15 15 14 15 18 15 15 20 22 16 17 19 16 22

time (sec) TC= 1ms 100. gdfs

10. Lagrangian 1

0.1

0.01

0.001

6

8

10

12

14

N

Figure 3: Run times comparison of Lagrangian systems vs. guided depth rst search. (The horizontal axe is the number of cities.)

4 Conclusions Lagrangian techniques have been applied to solving ILP FEASIBILITY. For TSP and other NP complete problems, a transformation can be performed so that they can be solved in the form of ILP FEASIBILITY. Continuous Lagrange relaxation employed in this paper provides a mechanism that prevents the dynamic system from being trapped in local attractions. The set of dierential equations has been found to converge in a fairly large parameter range, in contrast to penalty methods which have been relatively parameter sensitive. Moreover, the logical time it takes to converge grows sublinearly with problem sizes, and only a very short period is required for large, dicult problems shown. Even without hardware implementations, our approach seeking exact solutions for NP complete problems by solving the dynamic system in software simulations is competitive compared to the branch and bound method, the latter taking prohibitively longer time even for moderate problem sizes. An on-line solving of dicult combinatorial problems can be made possible once the dynamic system characterizing the particular problem is programmed on its analog counterpart circuit.

References [1] Y.-J. Chang and B. Wah, \Lagrangian techniques for solving a class of zero-one integer linear programs," Proceedings of the IEEE Computer Software and Applications Conference, 1995. [2] S. Zhang and A. G. Constantinides, \Lagrangian programming neural networks," IEEE Transactions on Circuits and Systems-II: Analog and Digital Signal Processing, vol. 39, no. 7, pp. 441{452, 1992. [3] A. Cichocki and R. Unbehauen, \Switched-capacitor arti cial neural networks for nonlinear optimization with constraints," Proceedings of 1990 IEEE International Symposium on Circuits and Systems, pp. 2809{2812, 1990. [4] C. H. Papadimitriou and K. Steiglitz, Combinatorial Optimization. Prentice-Hall, 1982. [5] C. E. Miller, A. W. Tucker, and R. A. Zemlin, \Integer programming formulation and traveling salesman problems," J. ACM, vol. 7, pp. 326{329, 1960. [6] L.-C. Chu, Algorithms for Combinatorial Optimization in Real Time and their Automated Re nements by Genetics-Based Learning. PhD thesis, Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, August 1994.