Lecture II: Linear Algebra Revisited

Lecture II: Linear Algebra Revisited

Overview Vector spaces, Hilbert & Banach Spaces, Metrics & Norms • Matrices, Eigenvalues, Orthogonal Transformations, Singular Values • Operators, Operator Norms, Function Spaces •

Note: We will need many of these concepts as basic tools to quantify and evaluate the performance of machine learning algorithms and also to come up with more efficient and effective solutions …

Lecture II: MLSC - Dr. Sethu Vijayakumar

1

Vectors Usually denoted by lower case, bold letters, e.g. x, y Operations : „ „ „

„

„

Multiplication by scalar (αx). Addition of vectors (x+y) – x and y have to be of same dimensions. Linear combination. z u = αx+βy (x and y have to be of same dimensions). < v, w > Angle between vectors. cos θ = v w Linear independence. z

When one vector cannot be written as a linear combination of other, then the vectors are said to be linearly independent.

Lecture II: MLSC - Dr. Sethu Vijayakumar

2

Metric Definition 1 (Metric/ Distance)

Example 1 (Trivial Metric)

Example 2 (Manhattan Distance)

Lecture II: MLSC - Dr. Sethu Vijayakumar

3

Vector Spaces Definition 2 (Vector Spaces)

Definition 3 (Cauchy Series)

Definition 4 (Completeness)

Lecture II: MLSC - Dr. Sethu Vijayakumar

4

Examples of Vector Spaces Rational Numbers, Real Numbers, Polynomials are all Vector Spaces

Lecture II: MLSC - Dr. Sethu Vijayakumar

5

Norm & Banach Spaces Definition 5 (Norm / Length)

Definition 6 (Banach Space)

Lecture II: MLSC - Dr. Sethu Vijayakumar

6

Examples of Banach Spaces

Lecture II: MLSC - Dr. Sethu Vijayakumar

7

Dot Products & Hilbert Spaces Definition 7 (Dot Product/ Inner Product)

Example :

⎡ 3⎤ v=⎢ ⎥; ⎣4 ⎦

v = v, v

1 2

= 32 + 4 2 = 5

Definition 8 (Hilbert Space)

Lecture II: MLSC - Dr. Sethu Vijayakumar

8

Examples of Hilbert Spaces

Lecture II: MLSC - Dr. Sethu Vijayakumar

9

Matrices

⎡ v1 ⎤ v = ⎢⎢ M ⎥⎥ ⎢⎣vm ⎥⎦

Domain

ℜ

M Range

m

ℜ

n

⎡ u1 ⎤ u = ⎢⎢ M ⎥⎥ ⎢⎣un ⎥⎦

u =Mv Review: • Addition of Matrices • Multiplication of matrices by scalars, vectors and matrices. • Domain and Range of a Matrix

Lecture II: MLSC - Dr. Sethu Vijayakumar

10

Special matrices Square and Diagonal Matrix A square matrix has equal number of rows and columns. A diagonal matrix has all off-diagonal elements zero.

Symmetric Matrix Anti-symmetric Matrix

Orthogonal Matrix (Often denoted as O(m) )

Lecture II: MLSC - Dr. Sethu Vijayakumar

11

Matrix Concepts Rank of a Matrix

Range, Domain and Null Space Range of M, denoted by R(M), is the space of all vectors that can be obtained by the operation of M on the vectors in the domain of M denoted by D(M).

v ∈ R(M ) iff ∃u ∈ D(M ) s.t. uM = v Null Space of M, denoted by N(M), is a subspace of all the vectors in the domain of M

D(M) that map to the zero (null) vector in R(M) when operated upon by the matrix M. v ∈ N (M ) iff vM = 0 Lecture II: MLSC - Dr. Sethu Vijayakumar

12

Matrix Invariants Trace

Properties of Trace : tr (aA) = a tr ( A)

tr ( A + B) = tr ( A) + tr (B) tr ( AB) = tr (BA )

Determinant Determinant can be written as the product of the eigenvalues : Note: Trace and Determinant are invariant under orthogonal transformation Lecture II: MLSC - Dr. Sethu Vijayakumar

13

Matrix Norms Operator Norm

Frobeius Norm

Lecture II: MLSC - Dr. Sethu Vijayakumar

14

Eigensystems Definition 9 (Eigenvalues/ Eigenvectors)

IMP: Defined only for square Matrices Š Intuitive Explanation. „ „

„

A square matrix M is a mapping from n-dim to n-dim space. Most vectors change both direction and length when undergoing this mapping transformation. Those vectors which only change length (i.e., multiplying them by a matrix is similar to multiplying by a scalar) are called eigenvectors and the eigenvalue indicates how much they are shortened or lengthened.

Lecture II: MLSC - Dr. Sethu Vijayakumar

15

Eigensystems II Eigenvectors/Eigenvalues of Symmetric Matrices o All eigenvalues of symmetric matrices are real o Symmetric matrices are fully diagonalizable, i.e. we can find m eigenvectors o All eigenvectors of symmetric matrices M with different eigenvalues are mutually orthogonal (Prove !!)

Decomposition of Symmetric Matrices

Example

Lecture II: MLSC - Dr. Sethu Vijayakumar

16

Positive Matrices Definition 10 (Positive Definite Matrices)

Induced Norm and Metrics

Lecture II: MLSC - Dr. Sethu Vijayakumar

17

Mahalanobis Distance d (x, y ) M = x, y ⎡1 0⎤ M=I=⎢ ⎥ 0 1 ⎦ ⎣

M

= xT My

= Euclidean Distance

⎡ 2 0⎤ M≠I=⎢ ⎥ 0 1 ⎦ ⎣ ⎡ 2 1.4⎤ M=⎢ ⎥ 1 . 4 1 ⎦ ⎣ Lecture II: MLSC - Dr. Sethu Vijayakumar

18

Singular Value Decomposition

Note: Eigenvalue/Eigenvector decompositions are valid only for square matrices Do we have some decomposition for rectangular matrices ??

Singular value Decomposition (SVD)

Lecture II: MLSC - Dr. Sethu Vijayakumar

19

Matrix Inverse

⎡ v1 ⎤ v = ⎢⎢ M ⎥⎥ ⎢⎣vm ⎥⎦

Domain

ℜ

M Range

m

M

−1

ℜ

n

⎡ u1 ⎤ u = ⎢⎢ M ⎥⎥ ⎢⎣un ⎥⎦

M −1M = I; MM −1 = I Note: A regular inverse exists only for square matrices with linearly independent column vectors Interpretation: We need a one-to-one mapping to uniquely go from one element of a space to another and back. Square matrices and linearly independent columns ensure this !! Lecture II: MLSC - Dr. Sethu Vijayakumar

20

Pseudoinverse For rectangular matrices and square matrices with linearly dependent columns, there exists the pseudoinverse or generalized inverse which performs the inverse mapping. In general, these inverses are not unique.

The Moore-Penrose Pseudoinverse (M+)

M + = (M T M ) −1 M T or M T (MM T ) −1 The above generalized inverse is called the Moore-Penrose Psuedoinverse and is unique. Among the multiple inverse solutions, it chooses the one with the minimum norm.

Lecture II: MLSC - Dr. Sethu Vijayakumar

21

Operators Linear Operators Generalization of matrix – a mapping from one Banach space to another. Norms and eigenvalues/eigenvectors are defined as for matrices; so are Range & Null Spaces.

Notation A : FG denotes a linear operator A mapping from space F to space G.

A Matrix-Operator Correspondence Matrix Transpose

Adjoint Operator

Af , g = f , A * g for all f ∈ F , g ∈ G.

AT Symmetric Matrix

A = AT Orthogonal Matrix

A −1 = A T

Self Adjoint Operator

Af , g = A * f , g for all f ∈ F , g ∈ G.

Isometry

f , g = Af , Ag for all f ∈ F , g ∈ G.

Lecture II: MLSC - Dr. Sethu Vijayakumar

22

Linear Operators - Examples Input transformation

Sampling

f

Sampling from a function to yield scalar outputs. ( We will see later why this is so !!! )

Fourier Transform

Lecture II: MLSC - Dr. Sethu Vijayakumar

x1

x2

x3

23

Range and Null Space of Operators Recollect: Definition of Range, Domain and Null space of a matrix

N(A)

A

R(A*)

N(A*) R(A)

A* Lecture II: MLSC - Dr. Sethu Vijayakumar

24