Hidden Minimum-Norm Problems in Quadratic ... - Semantic Scholar

Hidden Minimum-Norm Problems in Quadratic Programming by Robert M. Sloan W.P. #1768-86 Revised

Freund April,

1986

Abstract: quadratic

This note presents

sufficient conditions

program, or its dual,

(Euclidean) norm problem, linear transformation conditions are shown

i.e.

to be transformed into a minimum a problem of minimizing

of an element of a polytope.

the norm of

sufficient conditions,

These sufficient

As part of the

we characterize when the two linear

Ax > b and Az

< b have simultaneous solutions.

These results are used in conjunction with duality constructions obtain two equivalent reformulations of a given quadratic

Key Words:

Quadratic program,

norm, minimum norm problem,

program.

Running Header:

a

to be necessary under a suitable restriction on

the class of transformations that are allowed.

inequality systems

for a convex

Hidden Norm Problems.

to

program.

gauge

A minimum

(Euclidean) norm problem over a polytope is a program

of the form: NP:

UCx

minimize

+ dl

x

subject to: where

11*11

denotes

the Euclidean

The purpose of implications

Ex >

f (12)

this note is

norm.

to explore

when can a convex quadratic

of the following question:

program or its dual

be conveniently

answers to and

transformed into a minimum

(Euclidean) norm program? The

standard convex quadratic program is given as QP:

1/2 xtQx + qtx

minimize x

subject to: where Q is

assumed to be

standard dual QD:

Ax > b

symmetric and positive semi-definite.

of QP, see Dorn [1],

is given by

-1/2 ytQy + btX

maximize y,X subject

The

to:

-Qy + AtX = q X > 0

The concern herein lies data (Q,q,A,b)

in discovering properties of the problem

that allow the objective function

replaced by a norm,

in QP or QD to be

thus transforming QP or QD into an instance of

NP. The rationale for exploring transformations of program to the minimum

norm problem is

threefold.

a quadratic First,

the

minimum norm problem has an immediate geometric interpretation that may be useful program.

in

Second,

the solution of a given quadratic

the analysis and

the minimum norm problem is

1

*1__1_______1______

__11

iI_____

a classical

optimization problem, and has received extensive study, example,

[5].

Luenberger

Third, the author has

see for

recently

investigated an alternate duality theory for the minimum norm problem

[2],

that

is applicable to quadratic

programs that can be so

transformed. If the matrix Q is symmetric and positive semi-definite, then Q can be written as

Q = MtM for some matrix M;

efficient

for constructing M are well-known, see e.g. Gill,

procedures

Murray, and Wright

[3].

Proposition 1. At

If the system of

Qs

(1.1)

= -q

(1.2)

> 0

(1.3)

+ bt6

btT

> 0,

6 >

has a solution (N,

inequalities

q

+ Qs =

At6 -

linear

O,

6, s),

(1):

(1.4) then the program QP is equivalent

to the

minimum norm problem NP1:

{{Mx

minimize x subject

to:

+ Msl

Ax > b,

where M is any matrix for which

MtM = Q.

For any x satisfying Ax > b,

Proof:

1/2 xtMtMx + qtx = 1/2 xtMtMx + -

1/2

and

stMtMs

tAx

similarly, using

1/2 xtQx btR

+

+ qtx

> -bth,

< 1/2

> 1/2

tAx

IIMx + MsH 2

1/2

xtQx + qtx

+ stMtMx = -

1/2

1/2 stMtMs

=

UIMx + Msll2

+ btr

,

(1.2), IIMx

+ MsI1 2

-

1/2 stMtMs

equality is obtained throughout,

2

-

bt6.

But since

and 1/2 xtQx + qtx =

1/2 IIMx + Ms112 increasing in

-

1/2

IMx + Msl,

solution,

If q lies then

(0,

Remark 2.

(,

particular, any

solves

6,

s)

> b will index i,

(1).

= (0,

(n,

6,

s)

to

(1),

i < m,

transformed into a minimum norm problem.

tAx

from Q if CtC = Q.

for any feasible points x, (respectively,

)

(respectively,

>).

Proposition 2.

= 6tb.

In

i > 0 will

for QP to be

Before turning to the

we first

to be monotonically transformable

and any

> b.

introduce some

Given the matrix Q of the program QP,

said to be derived

-bt6,

i > 0 or

for which

the system Ax

question of necessary conditions,

If Q is

(1). =

bt

= ntb ttAxand

satisfy 1

Q-lq) solves

Proposition 1 provides sufficient conditions

is

function can

Qs = q has a

i.e.,

In particular, ,

be an always-active constraint of

terminology.

strictly

[X]

For any solution

x that solves Ax

is

This expression

in the column space of Q,

0, s)

positive definite,

+ btn.

and so the quadratic objective

+ MslI.

be replaced by [IMx Remark 1.

stMtMs

A norm

the norm

Cx + d

Cx + dl

is said

in the objective function of

x of QP.

[lCx

+ dl

>

if and only if 1/2 xtQx + qtx

If QP is feasible,

Cx

QP if

+ dl

> 1/2 xtQx + qtx

then QP can be monotonically

transformed to a minimum norm problem with objective function

llCx + d (1)

derived from Q only if the system of linear inequalities

has a solution.

Proof:

Suppose that system

of the alternative, scalar

e

(1) has no solution.

there exist

vectors x,

that satisfies:

3

Then, by a theorem

x and a nonnegative

Ax > be Ax

> be

Qx = Qx qtx > qtx

There are two cases to consider, depending on whether e is positive or zero.

Case 1: 8

e >

.

Without loss of generality, we can assume that

= 1, and so x and x satisfy: Ax

> b

Ax > b Qx = Qx qtx > qtx

If there exists C and d such that IICx + dll is a strictly monotonic transformation of the objective function of QP and is derived from Q,

then

Cx = Cx,

11Cx

+ dl

and so

>

Cx

+ dll.

IICx + dl

=

But Qx = Qx and Q = CtC

IlCx + dll,

a contradiction.

implies Thus no

that such

norm can be found. Case 2:

e = 0.

Because QP is feasible, there exists x' that

satisfies Ax' > b.

Therefore (x' + x) and (x' + x) satisfy: A(x' + x) A(x'

+ x)

Q(x' + qt(x'

>

b

> b

) = + x)

Q(x + x) >

qt(x'

+ x)

and the proof follows that of case 1, with x replaced by (x' + x) and x replaced by (x' +x).

[X]

Turning to the dual quadratic program QD, our first result is:

4

Av > b

(2.1)

Az < b

(2.2)

Qv - Qz = o

(2.3)

qtv

- qtz

has a solution

(v,

= 0

,

(2):

(2.4)

then the dual quadratic program QD is

z),

to the minimum norm problem

equivalent NP2:

inequalities

If the system of linear

Proposition 3.

IlMy - Mz

minimize y,X subject

-Qy + AtX

to:

= q

X > 0O where M is any matrix for which Q = MtM.

Proof: -

1/2 ytQy + btX < -

+ vtQy = we

that is feasible

For any (y, X)

obtain -

1/2

HMy -

2

with qtz = qtv, we have

HMy -

Mz,

IlMy -

Mz11.

Remark 3. applied

ztQZ. and

1/2 ytMtMy + qtv

1/2

My -

1/2

2

+ qtz

+ 1/2

(2.2),

ztQz.

and combining this relation

ytQy + btX

This expression is

Mzl

Similarly, using

= -

1/2

flMy - Mzll2

strictly decreasing

in

so we can replace the maximand by the minimand

[X] Proposition 3 is

to the

dual QD in

-

that Mz = Mv, -

-

+ qtv + 1/2 vtQv.

1/2 ytQy + btX


if for any feasible Cy

+ EX + dl

ytQy + btX


0 btX > Thus

if

Cy

+ EX + dl

function of QD,
8,

To see

indeed, the optimal

UCx + d

is

solution is x*

=

=

(3 ,5 )t,

42(qtx -8)(qtx

with qtx*

and 8.

strictly increasing in qtx for qtx

solution.

However,

C is not

8.

derived from Q in

to the above transformation was

Also,

-8)

Thus

Cx + d

monotonically transformable from QP, even though system (1)

key

Let

is monotonically transformable to the minimum norm program

with objective function

which

is no

as the following example shows.

I, A =1

the inequalities

of QP

When these

has no

this example.

the judicious choice of

is

The d,

based

on a known lower bound on the optimal value of QP. If

the montonicity condition is relaxed,

of the set of optimal solutions

to QP allows us to write any

instance of QP as a minimum norm program. optimal

solution x*

of QP

Is

unique

7

II

____l______sll

then prior knowledge

For example,

if the

(which it can be even if Q=O,

i.e.

QP is

just a

linear program),

then QP

is equivalent to

the

minimum norm program: IIx-x*11

minimize x subject

to:

Ax > b

This transformation appears

somewhat pointless

in that the quadratic

program has been solved before the transformation

is even made.

Nevertheless,

the transformation can be accomplished

time, because

linear and convex quadratic programming are solvable

in polynomial time quadratic depends

[4].

program to be

on the class of

in polynomial

The question of necessary conditions for a transformed into a norm program thus clearly transformations

Note that the pair of

inequalities

that are allowed. (2.1)

and

(2.2) of

Proposition 3 are described by reflecting the halfspaces defined by the feasible region of QP.

A curious

Proposition 3 is the simultaneous and Az

< b.

aff(x) and

This

issue is

rec(x) denote,

recession cone of

a set X,

in the system Ax constraint Ax > b,

issue raised in light of

solvability of

treated in the next proposition. respectively,

the affine hull

see Rockafellar

[6].

b is said to be parallel

is redundant and Ajx = d > bj

i.e.,

the system Ax >

b

Let

and the

A constraint Ajx > bj

redundant

if the

for every x that satisfies

Ajx is constant for every x satisfying Ax >

b.

The

following proposition characterizes when the system Ax > b and Az < both have a solution.

Proposition 5. furthermore

Let

assume that

Ax > b is parallel dim(rec(x))

X =

{xERnAx > b} X

redundant.

and Z =

{zERnlAz < b),

and

and that no constraint of the system Then Z

= dim(x).

8

if and

only if

b

Proof: and Az 0

E

Suppose Z

*.

Then there

Let k = dim(x),

< b.

rec(x),

equality must hold,

dim aff{x, and

so

xl,...,xk)

{xl-z

But A(xl-z)

...

elements of X,

for which

,xk-,

Conversely, If

index set M B, where

If k > 0,

> O,

*,

Z =

rec(X) > k,

x-z}

and hence equal

and assume that dim(rec(x))

such that Aax = b

interior of

k=O, X is a singleton

(x)

for every

satisfies Ax > b

X).

Furthermore,

Therefore,

there

(x is any element (A.) = n-k.

= b,

and so x

= aff(rec(x)) for which Ajxi

=

< dim(x).

= Aj.

and Aaxi = 0,

{x E RnIAax = b). = 0,

combination of the tXA

i=l,...,k,

i=l,...,k, rows

Thus X ba

of A, bj,

If k > 0,

If there

< dim(x).

i.e.

and aff(xl,...,xk,o)

exists an index J

6

then Aj must be a nonnegative i.e.

there exists Xa > 0 for which

and Ajx > bj

is a parallel redundant

constraint, violating the hypothesis of the proposition. dim(rec(x))

If

Z,

there exists linearly independent vectors x 1 ,...,xk in rec(x), > 0,

dim(X)

B is a parallel

Thus Ax

dim(rec(X))

=

sets a and

x E X, and

rank

and every index j

redundant constraint, whereby B=0.

that satisfy Axi

to k,

then the constraint

can be partitioned into disjoint

x of X that

contradicting Z=*.

there exist

< dim(x).

= {l,...,m}

the relative

< dim(x),

{xl-z,...,xk-z,

i.e.

the constraint matrix A has m rows,

exists an element of

Thus dim

suppose

u B = M,

dim(rec(x))

has k linearly independent elements.

> 0,...,A(xk-z) > 0, A(x-z)

because dim(rec(x))

If k=O, then since

Therefore dim aff{x, x,...,xk, z}) > k,

-z)

are elements of rec(X).

= k > 0.

However, since = dim(X).

= k.

z for which Ax > b 0.

>

dim(rec(x))

i.e.,

vectors xl,...,xk, all

and note that k

> 0 = k.

dim(rec(x))

exists x and

[X]

9

Thus

B

It

characterized in

Proposition 5 can be

the dimension of

terms of

and rec(X),

as opposed to a characterization in terms

multipliers

via a theorem of the alternative.

Q =

In

this example

linear inequalities

satisfy the

both programs will

0

q =

,

6,

(,

minimum norm program.

=

s)

(0,

A =

0

)

0,

solves

neither, or

(1),

(2).

(1) and/or

b

1

0

X

of dual

show, the primal, dual,

As the following examples

Example 1

b in

that the solvability of Ax > b and Az

is curious

3

=

whereby QP is a

However, Az < b has no solution, and

so the

transformation of the dual QD by Proposition 3 cannot be accomplished.

Example 2

In this (1,

Q =

example, z = (-1,

)t,

transformable to

Example 3

1

-

=

s =

solves

(2);

(1,

1

)t solves

(1),

therefore

this

example,

1

and x =

both QP and QD are

norm programs.

Q 0

0 00

11

=

A =

10 1 0

In

1

1 0 0

A =

-

0,

6 =

= 0, -l)t

I , q

it is

straightforward to

10

=

0 -1

show that neither

(2) have a solution, and so the transformations and 3 cannot be applied.

01b

-- I -1 -

(1) nor

of Propositions

1

in Gauge Duality

An Application of Hidden Norms One motivation

case. norm

the author's recent

programming is

quadratic gauge

for the study of hidden minimum norm programs

[2],

programs

of which

If QP can be converted problem NP1,

the minimum norm problem is a specific (through Proposition 1) to the minimum standard

then either the

see

dual of NP1,

GNPI:

is given

[2],

(Lagrange) dual

to:

Mth

-

(stAt

by

=

AtX

Replacing the objective (Lagrange)

subject

to:

dual

Af -

0O

function by 1/2 hth and then taking the yields:

1/2 ftqf

minimize f,e

0

+ bt)X = 1 >

QP:

or the

IhU 2

minimize h,X subject

standard

investigation of dual

of NP1 can be constructed.

gauge dual The gauge

in

+ estQf + 1/2

stQse 2

-

e

be > 0.

Because QP is obtained from QP by two consecutive duality constructions

(first

the gauge dual,

should expect QP and QP to

then the Lagrange

be equivalent.

The feasible

dual),

we

region of QP

is obtained by adding the extra variable e that scales the righthand-side b, system Af cone

-

and converts the be > O,

constraints Ax > b to the homogeneous

replacing the polytope of QP by a polyhedral

in one higher dimension.

The sense of equivalence of QP and QP

is shown in the next proposition, which shows how optimal solutions of QP and QP transform one to the other.

11

Let x be an

s) be a solution to (1).

,

Let (T,

6.

Proposition

solution to QP with optimal Karush-Kuhn-Tucker

optimal

, and define t = stQx + stQs

multipliers if

(i)

t

O,

if

t = O,

= (

Ax

+ btn + btS.

Then

e) = (x/t, l/t) solves QP with optimal K-K-T

(f,

multipliers (ii)

(K-K-T)

> b,

+

)/t.

Qx + Qs = 0 has a solution, and QP is

unbounded. Let

(f, 8)

c.

Then: (i)

solution to QP with optimal K-K-T multipliers

be an optimal

if e

O,

#

x =

f/e solves QP with optimal K-K-T multipliers

= C/e + S. (ii) The

if e

= O,

proof of this

then QP is infeasible.

proposition follows

[X] the K-K-T

from an examination of

conditions and from direct substitution of the indicated transformations. The

program QP was

followed by the the

(Lagrange)

obtained by taking the gauge dual of QP,

standard (Lagrange) dual

dual.

If

standard quadratic program dual QD,

QP:

feasible

Aw > 0

to:

region of QP

recession cone of homogeneous

(2),

then

Starting with the

and converting QD to the minimum

and then taking the gauge

(_qt The

solves

dual of NP2 yields:

1/2 wtQw

minimize w subject

z)

QD is transformable to a minimum norm problem, and

the order of the dualization can be reversed.

norm problem NP2,

(v,

ZtQ)w =

1

(X) (r)

is composed of the intersection of the

the feasible region of QP

(described by the

constraints Aw > 0) with a hyperplane

12

(described by

-

(_qt

tQ)

w

= 1)

that scales elements of the recession cone.

Analagous to Proposition 6, we have: Proposition 7.

Let (v, z) be a solution to (2).

Let x be an

optimal solution to QP with optimal K-K-T multipliers A, and define u =

-qtx + qtz -_

(i)

tQx +

if u # O,

w =

multipliers

(ii)

if u = 0,

X

tQz.

(x = n/u,

Then

)/u solves QP with optimal K-K-T r = 1/u.

QP is infeasible.

Let w be an optimal solution to QP with optimal K-K-T multipliers X, r.

Then (i)

if r # 0,

x = w/r + v solves QP with optimal K-K-T

multipliers

(ii)

if r = 0,

=

t/r.

QP is unbounded.

[X]

It is hoped that the equivalences of QP to the programs QP and QP given herein will be useful in applications of quadratic programming.

13

References

[1]

W.S. Dorn, "Duality in quadratic 18, 155-162, 1960.

[2]

R.M. Freund, "Dual gauge programs, with applications to quadratic programming and the minimum-norm problem," to appear in Mathematical Programming.

[3]

P. Gill, W. Murray, and M. Wright, Academic Press (New York, 1981).

[4]

M.K. Kozlov, S. P. Tarasov, and L.G. Khachiyan, "Polynomial solvability of convex quadratic programming," Doklady Academii English translation, Soviet Mathematics Nauk SSR 248(5) (1979) Doklady 20(5) ( 1979)].

[5]

D.G. Luenberger, Optimization by vector space methods, John Wiley & Sons (New York, 1969).

[6]

R.T. Rockafellar, Convex analysis, Princeton University Press (Princeton, New Jersey, 1970).

14

programming,"

Quart. Appl. Math.

Practical Optimization,