Regression with Linear Inequality Restrictions on Predicted Values

Regression with Linear Inequality Restrictions on Predicted Values Jan de Leeuw May 2, 2015 Note: the code in this note is now incorporated in the CRAN package isotone. In this note we study minimization of f (b) :=

1 (y − Xb)0 (y − Xb) 2

over all b for which A0 Xb ≥ 0. For instance X can be a spline basis, and Xb is the spline transformation. In situtations like this we are not really interested in the b themselves. We can then use A to require the transformation to be non-negative, or increasing, or satisfying any partial order. We proceed by formulating and solving the dual problem, and then use the solution of the dual problem to compute the solution to our original problem. The Lagrangian is L(b, λ) = f (b) − λ0 A0 Xb. The associate primal objective is ( p(b) := max L(b, λ) = λ≥0

if A0 Xb ≥ 0, otherwise.

f (b) +∞

The dual objective is d(λ) := min L(b, λ). b

The primal problem is to minimize p over b, which is the same as minimizing f over A0 Xb ≥ 0, our original problem. The dual problem is to maximize d over λ ≥ 0. The classical duality theorem for non-degenerate quadratic programs says that min p(b) = min max L(b, λ) = max min L(b, λ) = max d(λ). b

b

λ≥0

λ≥0

b

λ≥0

ˆ solves the dual problem, then ˆb := argmin L(b, λ) ˆ solves the primal problem. Moreover if λ Now analyze the dual problem in more detail. d(λ) = L(X + (Aλ + y), λ) =

1 1 0 y y − (y + Aλ)0 PX (y + Aλ) 2 2

where X + $ is the Moore-Penrose inverse of X, and PX := XX + . Find an orthonormal K, using singular value or QR decomposition, such that PX = KK 0 . Define u := K 0 y and V := −K 0 A. Then maximizing the dual objective is equivalent to minimizing (u − V λ)0 (u − V λ) over ˆ solves the dual problem, then ˆb := X + (Aλ+y) ˆ ˆ λ ≥ 0. If λ solves the primal problem, so that X ˆb = K(u−V λ). For computation we use the QR decomposition, and solve the dual problem using the nnls package [Mullen and Van Stokkum, 2012]. A simple general function is

1

library(nnls) mregnn