Linear Algebra

Linear Algebra Rekha Santhanam Johns Hopkins Univ.

March 27, 2009

Rekha Santhanam (Johns Hopkins Univ.)

Linear Algebra

March 27, 2009

1/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn .


Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V ,


Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V , that is, projV ~x is the best approximation to ~x in V .


Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V , that is, projV ~x is the best approximation to ~x in V .


Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V , that is, projV ~x is the best approximation to ~x in V . Reference : Linear algebra with applications by Steven J. Leon. Hooke’s law states that the force F applied to a spring is proportional to the distance the spring is stretched,


Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V , that is, projV ~x is the best approximation to ~x in V . Reference : Linear algebra with applications by Steven J. Leon. Hooke’s law states that the force F applied to a spring is proportional to the distance the spring is stretched, that is, if the distance is x then F = kx.


Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V , that is, projV ~x is the best approximation to ~x in V . Reference : Linear algebra with applications by Steven J. Leon. Hooke’s law states that the force F applied to a spring is proportional to the distance the spring is stretched, that is, if the distance is x then F = kx. The proportionality constant k is called the spring constant. Tom, a physics student wants to determine the spring constant for a given spring. He applies forces 3, 5 and 8 pounds, which has the effect of stretching the spring 4, 7 and 11 inches, respectively.


Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V , that is, projV ~x is the best approximation to ~x in V . Reference : Linear algebra with applications by Steven J. Leon. Hooke’s law states that the force F applied to a spring is proportional to the distance the spring is stretched, that is, if the distance is x then F = kx. The proportionality constant k is called the spring constant. Tom, a physics student wants to determine the spring constant for a given spring. He applies forces 3, 5 and 8 pounds, which has the effect of stretching the spring 4, 7 and 11 inches, respectively. Using Hooke’s law he arrives at the following system of equations: 4k = 3 7k = 5 11k = 8 Rekha Santhanam (Johns Hopkins Univ.)

Linear Algebra

March 27, 2009

2/8

Application II

Let V be a subspace of Rn and ~x be a vector in Rn . Then k~x ⊥ k = k~x − projV ~x k ≤ k~x − ~v k for all ~v ∈ V , that is, projV ~x is the best approximation to ~x in V . Reference : Linear algebra with applications by Steven J. Leon. Hooke’s law states that the force F applied to a spring is proportional to the distance the spring is stretched, that is, if the distance is x then F = kx. The proportionality constant k is called the spring constant. Tom, a physics student wants to determine the spring constant for a given spring. He applies forces 3, 5 and 8 pounds, which has the effect of stretching the spring 4, 7 and 11 inches, respectively. Using Hooke’s law he arrives at the following system of equations: 4k = 3 7k = 5 11k = 8 The system is clearly inconsistent. How should he find k? Rekha Santhanam (Johns Hopkins Univ.)

Linear Algebra

March 27, 2009

2/8

Least Squares Method


Linear Algebra

March 27, 2009

3/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b.


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A.


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A. Let W = Im A. Then W is a subspace of Rm .


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A. Let W = Im A. Then W is a subspace of Rm . The closest approximation we have to ~b in W is ~b ∗ = projW ~b.


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A. Let W = Im A. Then W is a subspace of Rm . The closest approximation we have to ~b in W is ~b ∗ = projW ~b. The equation A~x ∗ = ~b ∗ has a solution since ~b ∗ ∈ Im A.


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A. Let W = Im A. Then W is a subspace of Rm . The closest approximation we have to ~b in W is ~b ∗ = projW ~b. The equation A~x ∗ = ~b ∗ has a solution since ~b ∗ ∈ Im A.This implies that ~b − A~x ∗ ∈ W ⊥ .


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A. Let W = Im A. Then W is a subspace of Rm . The closest approximation we have to ~b in W is ~b ∗ = projW ~b. The equation A~x ∗ = ~b ∗ has a solution since ~b ∗ ∈ Im A.This implies that ~b − A~x ∗ ∈ W ⊥ . Claim: If W = Im A, then W ⊥ = Ker AT .


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A. Let W = Im A. Then W is a subspace of Rm . The closest approximation we have to ~b in W is ~b ∗ = projW ~b. The equation A~x ∗ = ~b ∗ has a solution since ~b ∗ ∈ Im A.This implies that ~b − A~x ∗ ∈ W ⊥ . Claim: If W = Im A, then W ⊥ = Ker AT . Therefore, ~b − A~x ∗ ∈ Ker AT . This implies AT ~b − AT A~x ∗ = ~0.


Linear Algebra

March 27, 2009

4/8


Let A be an m × n matrix. We want to approximate the equation A~x = ~b. Since this system is not consistent we know that ~x is not in Im A. Let W = Im A. Then W is a subspace of Rm . The closest approximation we have to ~b in W is ~b ∗ = projW ~b. The equation A~x ∗ = ~b ∗ has a solution since ~b ∗ ∈ Im A.This implies that ~b − A~x ∗ ∈ W ⊥ . Claim: If W = Im A, then W ⊥ = Ker AT . Therefore, ~b − A~x ∗ ∈ Ker AT . This implies AT ~b − AT A~x ∗ = ~0.

Hence, ~x ∗ is the solution to the equation AT A~x ∗ = AT ~b.


Linear Algebra

March 27, 2009

4/8

Orthogonal Transformation

Note that in one of the homework problems the transformation 3 1 represented by the matrix does not map the orthogonal set 1 3 1 0 { , } to an orthogonal set. 0 1 In general a transformation does not take orthogonal sets to orthogonal sets.


Linear Algebra

March 27, 2009

5/8


Note that in one of the homework problems the transformation 3 1 represented by the matrix does not map the orthogonal set 1 3 1 0 { , } to an orthogonal set. 0 1 In general a transformation does not take orthogonal sets to orthogonal sets. However the rotation transformations preserve orthogonality and the length of the vectors.


Linear Algebra

March 27, 2009

5/8


Note that in one of the homework problems the transformation 3 1 represented by the matrix does not map the orthogonal set 1 3 1 0 { , } to an orthogonal set. 0 1 In general a transformation does not take orthogonal sets to orthogonal sets. However the rotation transformations preserve orthogonality and the length of the vectors. Definition Let T : Rk → Rk be a linear transformation. The function T is said to be an orthogonal transformation if kT (~x )k = k~x k.


Linear Algebra

March 27, 2009

5/8


~ ∈ Rk be Let T : Rk → Rk be an orthogonal linear transformation. Let~v , w orthogonal to each other. Then


Linear Algebra

March 27, 2009

6/8



~ k2 = k~v k2 + k~ k~v + w w k2


Linear Algebra

March 27, 2009

6/8



~ k2 = k~v k2 + k~ k~v + w w k2 ~ )k2 = kT (~v )k2 + kT (~ kT (~v + w w )k2


Linear Algebra

March 27, 2009

6/8



~ k2 = k~v k2 + k~ k~v + w w k2 ~ )k2 = kT (~v )k2 + kT (~ kT (~v + w w )k2 kT (~v ) + T (~ w )k = kT (~v )k2 + kT (~ w )k2

By Pythagoras theorem, then T (~v ) and T (~ w ) are orthogonal to each other.


Linear Algebra

March 27, 2009

6/8




By Pythagoras theorem, then T (~v ) and T (~ w ) are orthogonal to each other. Thus an orthogonal transformation preserves orthogonality.


Linear Algebra

March 27, 2009

6/8




By Pythagoras theorem, then T (~v ) and T (~ w ) are orthogonal to each other. Thus an orthogonal transformation preserves orthogonality. Let A be a k × k matrix. Then A is orthogonal implies that the column vectors of A form an orthogonal set.


Linear Algebra

March 27, 2009

6/8


Properties

• A is orthogonal k × k matrix if and only if A−1 = AT .


Linear Algebra

March 27, 2009

7/8


Properties

• A is orthogonal k × k matrix if and only if A−1 = AT . • If A, B are orthogonal k × k matrices then AB is orthogonal.


Linear Algebra

March 27, 2009

7/8


Properties

• A is orthogonal k × k matrix if and only if A−1 = AT . • If A, B are orthogonal k × k matrices then AB is orthogonal. • If A is orthogonal detA = 1 or − 1.


Linear Algebra

March 27, 2009

7/8


Properties

• A is orthogonal k × k matrix if and only if A−1 = AT . • If A, B are orthogonal k × k matrices then AB is orthogonal. • If A is orthogonal detA = 1 or − 1. • AB = B T AT .


Linear Algebra

March 27, 2009

7/8


Properties

• A is orthogonal k × k matrix if and only if A−1 = AT . • If A, B are orthogonal k × k matrices then AB is orthogonal. • If A is orthogonal detA = 1 or − 1. • AB = B T AT . • (AT )−1 = (A−1 )T .


Linear Algebra

March 27, 2009

7/8


Properties

• A is orthogonal k × k matrix if and only if A−1 = AT . • If A, B are orthogonal k × k matrices then AB is orthogonal. • If A is orthogonal detA = 1 or − 1. • AB = B T AT . • (AT )−1 = (A−1 )T . • RankA = RankAT


Linear Algebra

March 27, 2009

7/8


Properties

• A is orthogonal k × k matrix if and only if A−1 = AT . • If A, B are orthogonal k × k matrices then AB is orthogonal. • If A is orthogonal detA = 1 or − 1. • AB = B T AT . • (AT )−1 = (A−1 )T . • RankA = RankAT • If W = Im A then W ⊥ = Ker AT . Also, if W = Ker A then

W ⊥ = Im AT .


Linear Algebra

March 27, 2009

7/8

Orthogonal projections


Linear Algebra

March 27, 2009

8/8