Appendix: Concepts of Linear Algebra. In this appendix, some essential topics in
linear algebra are reviewed. For each topic, we present some definitions, basic ...
Appendix: Concepts of Linear Algebra In this appendix, some essential topics in linear algebra are reviewed. For each topic, we present some definitions, basic properties, and numerical examples.
Notations
An m ! n matrix A consists of m rows and n columns and mn elements (real or complex numbers) and is denoted by a 11 a 12 ! a 1n a 21 a 22 ! a 2n
A"
"
" # "
" $a ij % mn i,j"1 " $a ij % mn " $a ij %.
a m1 a m2 ! a mn The element a ii is called the ith diagonal element of A and a ij for i ! j is called the &i, j'th element of A. We say the size of A is m ! n or the order of A is m when m " n. An m ! 1 matrix is said to be an m-vector or a column m-vector; and an 1 ! n matrix is said to be an n-vector or a row n "vector. To avoid any confusion, an n-vector means a column vector in this appendix and a row vector is represented by the transpose (it will be defined shortly) of a column vector. Commonly, R n and C n are notations for the sets of real and complex column n-vectors, respectively; and R m!n and C m!n are notations for the sets that contain all m ! n real and complex matrices, respectively. If we do not specify the type of a matrix A, then A can be either real or complex. The following are examples of a 2 ! 3 matrix, a column 2-vector and a row 3-vector: A"
1
0 2"i
"2. 5 3i
"4
, v"
1 "2
, w"
a b c
.
A is said to be a square matrix if m " n, otherwise a rectangular matrix. Z is said to be a zero matrix, denoted by Z " $0% mn " 0, if all elements of Z are zero. Matrix D is said to be an n ! n diagonal matrix if all elements of D are zero except its diagonal elements and is commonly written as D "diag d 1 , # , d n . An n ! n diagonal matrix with all diagonal elements equal to 1 is called the n ! n identity matrix, denoted by I n or I. A matrix T is said to be an upper (lower) triangular matrix if all its elements below (above) its diagonal are zero. A matrix S is said to be a submatrix of A if the rows and columns of S are consecutive rows and columns of A. If the rows and and columns start from the first ones, S is also called a leading submatrix of A. For example, 3 "1 4 1 "2 and S " is a leading submatrix S " 1 2 is a submatrix of A " 5 1 2 "4 5 of A "
1
"2
3
"4
5
"6
7
"8
9
.
Basic Operations Transpose and Hermitian
Given A " $a ij % in R m!n , the transpose of A, denoted by A T , is an n ! m matrix whose rows are columns of A and columns are rows of A. When A is in C m!n , the Hermitian of A, denoted by A ! , is in C n!m and its &i, j'th element is a #ji . For example,
A"
1 4
1 2 3
T
, A "
4 5 6
and B "
2 5
1$i
"2i
3
4"i
3 6
, B! "
1"i
3
2i
4$i
.
Trace of a Square Matrix The trace of an n ! n real square matrix A " $a ij % is defined by the sum of the diagonal elements n of A, that is, tr&A' " $ k"1 a kk . 1 2
Example Let A "
3 4
. Then, tr&A' " 1 $ 4 " 5.
It is not difficult to show that tr&AB' "tr&BA' provided that AB and BA both exist.
Dot Product (or Inner Product) and Orthogonality u1
Given two vectors u "
"
v1 and v "
"
un and v is a scalar ! and is defined as
in C n , the dot product or inner product of u
vn v1
!
!"uv"
u #1
!
u #n
"
n
"
k"1
vn Example Let u " u!v "
1 2 " 3i
1 2 $ 3i
, v"
"4 $ i 5 " 6i
"4 $ i 5 " 6i
$ u #k v k .
and w "
"3 2
" "12 $ 26i, and w T w "
. Then "3
"3 2
2
" 13.
Vectors u and v are said to be orthogonal if u ! v " 0. A set of vectors (v 1 , . . . , v m ) is said to be orthogonal if v !i v j " 0 for all i ! j; and is said to be orthonormal if in addition v !i v i " 1 for all i " 1, . . . , m. Consider vectors u1 "
1 "1
, u2 "
2 2
, v1 "
1 2
1 "1
and v 2 "
1 2 2
2 2
.
The set u 1 , u 2 is orthogonal and the set v 1 , v 2 is orthonormal. The dot product satisfies the Cauchy-Schwarz Inequality: &x ! y' 2 % &x ! x'&y ! y' for any vectors x and y in C n .
Matrix Addition and Scalar Multiplication Two matrices with the same size can be added or subtracted element-wise, and a matrix can be multiplied by a scalar (real or complex) element-wise. Let A " $a ij % mn , B " $b ij % mn and !, " be scalars. Then A $ B " $a ij $ b ij % mn , !A " $!a ij % mn and !A $ "B " $!a ij $ "b ij % mn .
Example Let A " A$B "
1. 2. 3. 4.
1 2 3
7
, B"
4 5 6 8 10 12
9
and ! " "2j. Then
10 11 12
, !A "
14 16 18
8
"2j "4j
"6j
, and 3A " 2B "
"8j "10j "12j
"11 "10 "9 "8
"7 "6
Matrix addition and scalar multiplication have the following properties: A $ B " B $ A; A $&B $ C'"&A $ B'$C; &!"'A " !&"A' " "&!A'; &A $ B' T " A T $ B T .
Matrix Multiplication
Given two matrices A " $a ij % and B " $b kl % with sizes m ! r and r ! n, the product C " AB " $c ij % is an m ! n matrix and its &i, j'th element is defined as b 1j r
c ij "
$ a kt b kj "
b 2j
a i1 a i2 ! a ir
"
k"1
b rj "
the dot product of the ith row of A and the jth column of B.
1 2 Example Let A "
3 4
, and B "
5 6 "1 3 AB "
"1 5
, BA T "
"1 7
1 "1 "1 2
. Then
"1 "1 "1 3
5
7
, BB "
2 "3 "3 5
, and
5 11 17 T
AA "
11 25 39
.
17 39 61 For a square matrix A, the notation A n for a positive integer n stands for the product AA# A (n times) and A 0 & I. Matrix multiplication has the following properties: 1. ABC " A&BC'"&AB'C; 2. &A $ B'C " AC $ BC; 3. A&B $ C' " AB $ AC; 4. &AB' T " B T A T if A and B are real, and &AB' ! " B ! A ! if A and B are complex. In general, matrix multiplication is not commutative, i.e. AB ! BA even if both AB and BA are well-defined and have the same size. When A is a matrix and B is a vector, we can write AB in terms of the columns of A and
.
elements of B, or the rows of A and vector B. Let A be an m ! n matrix and R1 A"
"
C1 # Cn
" Rm b1
where
C %i s
and
R %i s
are columns and rows of A, respectively. Let B "
"
. Then
bn R1B AB " b 1 C 1 $ ! $ b n C m "
"
.
RmB
Partitioned Matrices In many applications it is convenient to partition a matrix into blocks (submatrices). For 1 2 3 A 11 A 12 example, the matrix A " can be partitioned as A " where 4 5 6 A 21 A 22 7 8 9 A 11 " A 11 "
1 2 4 5 1 2
3
, A 12 "
6
, A 21 "
, A 12 " $3%, A 21 "
4
7 8
, and A 22 " $9%; or
, and A 22 "
7 matrices work as if the blocks were scalars. For example, A 11 A 12 A 13 A 21 A 22 A 23 A 11 A 12 A 21 A 22 A 31 A 32
$
B 11 B 12 B 13
"
B 21 B 22 B 23 B 11 B 12 B 21 B 22
5 6 8 9
. Operations on partitioned
A 11 $B 11 A 12 $B 12 A 13 $B 13 A 21 $B 21 A 22 $B 22 A 23 $B 23
,
A 11 B 11 $A 12 B 21 A 11 B 12 $A 12 B 22 "
A 21 B 11 $A 22 B 21 A 21 B 12 $A 22 B 22 A 31 B 11 $A 32 B 21 A 31 B 12 $A 32 B 22
provided that all the block products are well-defined.
Determinant of a Square Matrix Determinant The determinant of a square matrix A, denoted by det&A', is a scalar which provides some useful information about A. The determinants of 2 ! 2 and 3 ! 3 matrices are defined respectively as:
a 11 a 12
det
" a 11 a 22 " a 12 a 21 ,
a 21 a 22 a 11 a 12 a 13
det
"
a 21 a 22 a 23 a 31 a 32 a 33
a 11 a 22 a 33 $ a 21 a 13 a 32 $ a 31 a 12 a 23 "a 11 a 23 a 32 " a 21 a 12 a 33 " a 31 a 13 a 22
.
For a general n ! n matrix A " $a ij %, the determinant is defined as: n
det&A' "
$&"1'
n
i$k
a ik det&A ik ' "
k"1
$&"1' k$j a kj det&A kj ' k"1
for any 1 % i, j % n where A pq is the &n " 1' ! &n " 1' matrix resulting from the deletion of the row p and the column q of A. For example, det
1
"2
3
"4
5
"6
7
"8
9
let i " 1 "
&"1' 1$1 &1' det
"4 "6
$ &"1' 1$2 &"2' det
7
9
5
"6
"8
9
$ &"1' 1$3 &3' det
"4
5
7
"8
" &"3' " &"2'&6' $ &3'&"3' " 0 det
1
"2
3
"4
5
"6
7
"8
9
$ &"1' 2$2 &5' det
let j " 2 " 1 3 7 9
&"1' 1$2 &"2' det
$ &"1' 3$2 &"8' det
"4 "6 7
9 1
3
"4 "6
" "&"2'&6' $ &5'&"12' " &"8'&6' " 0 The determinant of A pq , det&A pq ', is called the &p, q'th minor of A and &"1' p$q det&A pq ' is called the cofactor of a pq . Directly from the definition, the determinant of a diagonal matrix is the product of its diagonal elements and the determinant of an upper or lower triangular matrix is also the product of its diagonal elements. Determinants have the following properties: 1. det&AB' " det&A' det&B'; 2. det&!A' " ! n det&A' for any scalar ! and n ! n matrix A; 3. det&A T ' " det&A'; 4. det A k " &det&A'' k ; 5. det&A' " 0 if any row (or column) of A is a scalar multiple of another row (or column); 6. det&A' " 0 if any row (or any column) of A is zero; 7. If B is obtained from A by interchanging two rows (or two columns), then det&B' " " det&A'.
Singular and Nonsingular Matrices A square matrix A is said to be nonsingular if det&A' ! 0 and is singular if det&A' " 0. Since det&AB' " det&A' det&B', the matrix AB is singular if and only if either A is singular or B is
singular; and is nonsingular if and only if both A and B are nonsingular.
Linear Independence of Vectors, Basis Vectors and Rank of a Matrix Linear Independence of Vectors Let v 1 , # , v m be a set of vectors in R n or C n . A vector u in R n or C n is said to be a linear m combination of v 1 , . . . , v m if there are scalars ! 1 , . . . , ! m such that u " $ i"1 ! i v i . The vector u"
"1 2
T
is a linear combination of v 1 "
u " 1 v 1 $ 7 v 2 . The vector 19 19
!1 . . . !n
T
2 3 T
and v 2 "
"3 5
T
since
is a solution of the following linear system: !1
v1 . . . vm
" u.
" !m
If the solution does not exist then u is not a linear combination of v 1 , . . . , v m . When m " n, the T
solution
is unique. The set S containing all linear combinations of v 1 , . . . , v m is !1 . . . !n called the spanning set of v 1 , . . . , v m and is denoted by S "span&v 1 , . . . , v m '. A set of vectors v 1 , . . . , v m is said to be linearly independent if ! 1 v 1 $. . . $! m v m " 0 implies ! 1 " 0, . . . , ! m " 0. A set of vectors is linearly dependent if it is not linearly independent, i.e., there are some ! i ! 0 such that ! 1 v 1 $. . . $! m v m " 0. Amongst the vectors v1 "
2
, v2 "
3
"3 5
1
, u1 "
2
, u2 "
"3 "6
,
the set (v 1 , v 2 ) is linearly independent and the set (u 1 , u 2 ) is linearly dependent since &"3'u 1 $u 2 " 0. A set of two vectors is linearly dependent if one vector is a scalar multiple of the other, i.e., v 2 " !v 1 for some nonzero scalar !. A set of m vectors is linearly dependent if one vector is a linear combination of others. Independent vectors have the following properties. 1. A set of orthogonal vectors is linear independent. 2. Let v 1 , . . . , v n be in R n and A "
v 1 . . . v n . The set of v 1 , . . . , v n is linearly independent if and only if det&A' ! 0. So, if det&A' " 0, the set of v 1 , . . . , v n is linearly dependent. Consider the above vectors v 1 , v 2 , u 1 and u 2 . Since det
v1 v2
" det
det
u1
" det
we can conclude that v 1 , v 2
Basis Vectors
u2
2 "3 3
" 19 ! 0 and
5
1 "3
" 0,
2 "6
is linearly independent and u 1 , u 2
is linearly dependent.
Let S "span&v 1 , . . . , v m ' where v i ’s are in R n . If (v 1 , . . . , v m ) is linearly independent, then (v 1 , . . . , v m ) is called a set of basis vectors of S and the dimension of S is m. In this case, S is also called an m-dimensional subspace of the space R n . Let e i be in R n with all elements are zero except the ith element is 1, i.e., e i "
0 # 0 1 0 # 0
T
for i " 1, . . . , n. A vector
u"
u1 . . . un
T
in R n can be written as
u " u 1 e 1 $ u 2 e 2 $. . . $u n e n , that is, a linear combination of e 1 , . . . , e n . So, (e 1 , . . . , e n ) is a set of basis vectors of R n and the dimension of R n is n. Hence, any n linearly independent vectors in R n form a set of basis vectors of R n , and m linearly independent vectors in R n for m & n cannot form a set of basis vectors for R n . If v 1 , . . . , v m form a set of basis vectors for an m-dimensional subspace S of R n , then each element u in S can be written as !1 u " ! 1 v 1 $. . . $! m v m "
v1 ! vm
.
" !m
Values of ! 1 , . . . , ! m can be obtained by solving the linear system: !1 v1 ! vm
" u.
" !m
Rank of a Matrix The rank of a m ! n matrix A, denoted by rank&A', is the largest number of columns (or rows) of A that form a set of linearly independent vectors. 1 "2 3 Example Let A "
. The rank of A is 2 since the third column of A is a
"4 5 "6
7 "8 9 linear combination of the first two columns and the first two columns are linearly independent: 3 "6
" &"1'
$ &"2'
"4
9 1. 2. 3. 4. 5.
"2
1
5
"2 and
"8
7
5 "8
1 !!
"4
.
7
Rank of a matrix has the following properties: An n ! n matrix A is nonsingular if and only if rank&A' " n; rank&AB' % min rank&A', rank&B' ; rank&A $ B' % rank&A' $ rank&B'; The rank of a zero matrix is 0; If rank&A' " k, then there is a k ! k submatrix of A with nonzero determinant but all &k $ 1' ! &k $ 1' submatrices of A have determinant 0. 1 "2 3 Example Let A "
"4 5 "6
. det&A' " 0, but det
7 "8 9 rank&A' " 2 by Property 5.
Inverse of a Square Matrix
1 "2 "4 5
! 0. So,
The inverse matrix of an n ! n matrix A, denoted by A "1 , is an n ! n matrix such that AA "1 " A "1 A " I n . A matrix is said to be invertible if its inverse exists. The inverse matrix of a square matrix is unique a b d "b 1 is A "1 " which if it exists. The inverse of a 2 ! 2 matrix A " ad " bc c d "c a exists if and only if ad " bc ! 0. For n ' 3, the inverse matrix A "1 can be obtained by solving n linear systems: AA "1 " I n , i.e., A
B1 . . .
Bn
"
e1 . . . en
or AB i " e i for i " 1, . . . n and A "1 "
B1 . . .
Bn
A linear system Ax " b has a unique solution if and only if A is nonsingular. Hence, the inverse of A exists if and only if A is nonsingular or det&A' ! 0, or rank&A' " n. 1 "2 3 Example Let A "
"4 5 "6
. Find A "1 if it exists.
7 "8 10 Since det&A' " "3 ! 0, A "1 exists. A "1 can be computed as follows. Solve 1 "2 3 "4 5 "6
" 23
1 v1 "
0
, v1 "
2 3
7 "8 10
0
1
1 "2 3
0
4 3 11 3
"4 5 "6
v2 "
1
7 "8 10
0
1 "2 3
0
"4 5 "6
v3 "
7 "8 10
0
, v2 "
2 1 , v3 "
1
2 1 " 23
A
"1
,
" v1 v2 v3
"
1
2 3
4 3 11 3
1
2
1
2
.
There is a closed form for the inverse of a nonsingular square matrix A " $a ij %, given by 1 Adj&A' where Adj&A' " cofactor of a ji " &"1' j$i det&A ji ' . A "1 " det&A' Inverse matrices have the following properties: 1 ; 1. det A "1 " det&A' 2. The matrix AB is invertible if and only if both A and B are invertible and &AB' "1 " B "1 A "1 ; 3. It A is invertible, then the linear system Ax " b has a unique solution x " A "1 b; 4. Let A be an n ! n invertible matrix, and u and v be in C n . Then
.
&A $ uv H '
"1
" A "1 "
1
H
"1
A "1 uv H A "1 .
1$v A u This is known as the matrix inversion lemma. The RLS algorithm is based on this lemma. 5. Let A and R be n ! n and m ! m invertible matrices, respectively. Let U be in C n!m and V be in C m!n . Then "1
"1
&A $ URV H ' " A "1 " A "1 U R "1 $ VA "1 U VA "1 . This is known as the Sherman-Morrison formula. The block RLS algorithm is based on this formula.
Eigenvalues and Eigenvectors of a Square Matrix and Spectral Radius Eigenvalues and Eigenvectors of a Square Matrix
Let A be an n ! n matrix, # a scalar and x a nonzero vector in C n . The pair &#, x' is said to be an eigenpair of A if the equation Ax " #x holds. In this case, # is called an eigenvalue of A and x is called an eigenvector of A associated with #. Example Let A"
2 "1 "1 2
1
, # 1 " 1, # 2 " 3, x 1 "
1
and x 2 "
1 "1
.
With a quick check, 2 "1
1
"1 2
1
" &1'
1 1
and
2 "1
1
"1 2
"1
" &3'
1 "1
# 1 , x 1 and # 2 , x 2 are eigenpairs of A. An eigenpair #, x of A satisfies the equation Ax " #x " 0 which implies det&A " #I n ' " 0. Define P&#' " det&A " #I n '. P&#' is an nth degree polynomial in # and is called the characteristic polynomial of A. Eigenvalues of A are zeros of the polynomial P&#' or equivalently, roots of the polynomial equation P&#' " 0. A has an eigenvalue # i with multiplicity k if and only if P&#' has a factor &# " # i ' k . The following are the steps needed to compute all eigenpairs of A: Step I. Solve the polynomial equation P&#' " 0 for #; Step II. For each solution # obtained in Step I, solve &A " #I n 'x " 0 for all x which are linearly independent. Example Let A "
1 "2 "3 4
. Then
P&#' " det&A " #I 2 ' " det
1"#
"2
"3
4"#
" # 2 " 5# " 2.
Step I. Solving P&#' " 0 gives # 1 " 5 $ 1 33 , and # 2 " 5 " 1 33 . 2 2 2 2 Step II. Solve for x.
For # 1 " 5 $ 1 33 , solve A " 2 2 1 " 5 $ 1 33 2 2 "3
4"
x 11
"2
" 34 t "
1 4
1
Let t " 1. Then x 1 "
" 34 "
1 4
x 11
5 $ 1 33 2 2 t
"
x 21
5 $ 1 33 I 2 x 1 " 0 for x 1 . 2 2
33
t 33
x 21
0
"
0
for any real scalar t.
.
For # 2 " 5 " 1 33 , solve A " 5 " 1 33 I 2 x 2 " 0 for x 2 . In a similar 2 2 2 2 1 1 33 $ 2 6 way, we have x 2 " . Hence, 1 1
5 $ 1 33 , 2 2
"
3 4
"
1 4
and
33
5 " 1 33 , 2 2
1 2
$
1 6
33
1
are eigenpairs of A. When n ' 3, it may not be possible to find the solutions of the equation P&#' " 0 analytically. In general, eigenpairs #, x are solved numerically. Eigenvalues and eigenvectors have the following properties. 1. If #, x is an eigenpair of A, then #, !x is also an eigenpair of A for any nonzero scalar !. 2. Let # 1 , x 1 , . . . , # k , x k be eigenpairs of A. If # 1 , . . . , # k are distinct, then x 1 , . . . , x k are linearly independent. 3. A has a zero eigenvalue if and only if A is singular. 4. Eigenvalues of A and A T are the same. 5. Eigenvalues of a diagonal matrix, an upper triangular matrix or a lower triangular matrix are the diagonal elements of the matrix. 6. Let A be invertible. If #, x is an eigenpair of A, then 1 , x is an eigenpair of A "1 . # 7. Let A be an n ! n matrix with eigenvalues # 1 , . . . , # n . Then the determinant and trace of A can be expressed as n
det&A' "
( # i " # 1 !# n ,
n
and tr&A' "
i"1
$ #i " #1 $ ! $ #n. i"1
8. If #, x is an eigenpair of A, then # $ $, x is an eigenpair of the matrix A $ $I.
Spectral Radius of a Square Matrix For an n ! n matrix A with eigenvalues # 1 , . . . , # n , the spectral radius of A, denoted by %&A', is defined as %&A' " max |# i | 1%i%n
where |a $ ib| "
a 2 $ b 2 . A matrix A is said to be convergent if lim k') A k " 0. Let A be an n ! n
complex matrix. We have the following two results. Theorem A is convergent if and only if %&A' & 1. Theorem If %&A' & 1, then the matrix I " A is invertible and &I " A' "1 " I $ A $ A 2 $. . . $A n $. . . .
Kronecker Product and Kronecker Sum Kronecker Product Let A " $a ij % mn and B " $b kl % pq . The Kronecker product of A and B denoted by A * B is an mp ! nq matrix defined by A * B " $a ij B%. Example Let A "
1 2 3 4 5 6
and B "
1 "1 1 2
. Then
1 "1 2 "2 3 "3 A*B "
1 2 2 "4 3 6 4 "4 5 "5 6 "6
.
4 8 5 10 6 12 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
The Kronecker product is not commutative in general. It has the following properties: &!A' * B " A * &!B'; &A * B'*C " A *&B * C'; &A $ B' * C " A * C $ B * C; A * &B $ C' " A * B $ A * C; &A * B' T " A T * B T and &A * B' ! " A ! * B ! ; tr&A * B' "tr&A'tr&B'; Let A be m ! m and B be n ! n. det&A * B' " $det&A'% m $det&B'% n ; &A * B'&C * D' " &AC' * &BD'; Let A and B be nonsingular. Then A * B is nonsingular and &A * B' "1 " A "1 * B "1 ; A * B " 0 if and only if A " 0 and B " 0; If # i , x i is an eigenpair of A and $ j , y j is an eigenpair of B, then # i $ j , x i * y j is an eigenpair of A * B.
Kronecker Sum Let A and B be m ! m and n ! n matrices, respectively. The Kronecker sum of A and B denoted by A + B is an mn matrix defined by A + B " &I n * A' $ &I m * B'. 1 2 3 Example Let A "
4 5 6 7 8 9
and B "
1 "1 1 2
. Then
A 0
A + B " &I 2 * A' $ &I 3 * B' "
"
0 A
B 0 0 $
0 B 0 0 0 B
1 2 3 0 0 0
1 "1 0 0 0 0
2 1 3
0 0 0
4 5 6 0 0 0
1 2 0 0 0 0
5 7 6
0 0 0
7 8 9 0 0 0 0 0 0 1 2 3
$
0 0 1 "1 0 0 0 0 1 2 0 0
"
7 8 10 "1 0 0 0 0 1
3 2 3
0 0 0 4 5 6
0 0 0 0 1 "1
0 0 0
4 6 5
0 0 0 7 8 9
0 0 0 0 1 2
0 0 0
7 9 11
If # i , x i is an eigenpair of A and $ j , y j is an eigenpair of B, then # i $ $ j , y i * x j is an eigenpair of A + B.
Vector and Matrix Norms, and Condition Numbers Vector Norms
Let S be R n or C n . A real valued function ||*|| defined on S is said to be a norm or vector norm if ||*|| satisfies the following properties. P1 ||x|| ' 0 for any x in S, and ||x|| " 0 if and only if x " 0, a zero vector. P2 ||!x|| " |!|||x|| for any x in S where ! is an arbitrary scalar. P3 ||x $ y|| % ||x|| $ ||y|| for any x and y in S. (triangular inequality) Commonly used vector norms: let x " E1 The l 1 -norm: ||x|| 1 & $
n |x |. k"1 i n |x | p k"1 i
x1 ! xn
T
.
1/p
for p an integer and 1 & p & ). In particular, E2 The l p -norm: ||x|| p & $ 2 2 ||x|| 2 " x 1 $ ! $ x n . E3 The l ) -norm: ||x|| ) & max 1 % k % n |x i | . A vector u is a unit vector if ||u|| " 1. A given vector x can be normalized as a unit vector: u " x . Vector norms satisfy the following inequalities. Let x and y be in R n or C n . ||x|| 1. The Cauchy-Schwarz Inequality. &x ! y' 2 % ||x|| 2 ||y|| 2 with equality if and only if y " !x for some scalar !. 2. ||x|| " ||y|| % ||x " y|| for any vector norm. 3. lim k') x &k' " x if any only if lim k') ||x &k' " x|| " 0 for any vector norm.
Matrix Norms
Let S be R m!n or C m!n . A real valued function |||*||| defined on S is said to be a norm or matrix norm if |||*||| satisfies the following properties. P1 |||A||| ' 0 for any A in S, and |||A||| " 0 if and only if A " 0, a zero matrix. P2 |||!A||| " |!||||A||| for any A in S where ! is an arbitrary scalar. P3 |||A $ B||| % |||A||| $ |||B||| for any A and B in S. (triangular inequality) P4 ||AB|| % |||A||| |||B||| for any A and B in S. (submultiplicative) mn . Commonly used matrix norms: let A " $a ij % i,j"1 m n E1 The l 1 matrix norm: ||A|| 1 & $ i"1 $ j"1 |a ij |.
.
E2 The l p matrix norm: ||A|| p &
m n $ i"1 $ j"1 |a ij | p
1/p
for p an integer and 1 & p & ). In particular,
the l 2 matrix norm or the Euclidean norm: ||A|| 2 &
m n $ i"1 $ j"1 |a ij | 2 . m
E3 The maximum column sum matrix norm: |||A||| 1 & max 1 % j % n $ k"1, |a kj |. n
E4 The maximum row sum matrix norm: |||A||| ) & max 1 % i % m $ k"1, |a ik |. Note that when A is real |||A||| ) " |||A T ||| 1 . E5 The spectral norm: |||A||| 2 & max # : # is an eigenvalue of A ! A Note that |||A||| 2 "
%&A ! A' . m n a 2ij . $ i"1 $ j"1
F6. The Frobenius norm: |||A||| F & Note that |||A||| F " Example Let A "
.
tr&A ! A' " ||A|| 2 . "1 2 "3 4 "5 6
. Then
||A|| 1 " 21, ||A|| 2 " |||A||| 2 "
%&A T A' "
91 , |||A||| 1 " 9, |||A||| ) " 15, 91 $ 1 8065 , |||A||| " F 2 2
91 .
Notice that ||A|| ) & max 1 % i,j % n |a ij | is not a matrix norm since it does not satisfy the submultiplicative property. Example A "
1 1 1 1
, A2 "
2 2 2 2
. Then ||A|| ) " 1, and
||A 2 || ) " 2 ' ||A|| ) ||A|| ) . Notice also that the spectral radius %&*' is not a matrix norm. Example Let A1 "
0 1 0 0
, and A 2 "
0 0 1 0
.
It is easy to verify that properties P1, P3 and P4 do not hold for these two matrices. P1:
%&A 1 ' " %&A 2 ' " 0 but A 1 ! 0 and A 2 ! 0.
P3: %&A 1 $ A 2 ' " 1 ( 0 " %&A 1 ' $ %&A 2 '. P4: %&A 1 A 2 ' " 1 ( 0 " %&A 1 '%&A 2 '. The following inequalities hold for an m ! n matrix A : 1. |||A||| 2 % |||A||| F % n |||A||| 2 ; 2. max 1 % i % m, 1 % j % n |a ij | % |||A||| 2 % mn max 1 % i % m, 1 % j % n |a ij |; 3. 1 |||A||| ) % |||A||| 2 % m |||A||| ) ; n 1 4. |||A||| 1 % |||A||| 2 % n |||A||| 1 ; m 5. |||A||| 2 %
|||A||| 1 |||A||| ) when A is a square matrix;
6. %&A' % |||A||| for any square matrix and any matrix norm.
Condition Numbers For a square matrix A, the quantity &&A' &
|||A||| |||A "1 ||| if A is nonsingular )
if A is singular
is called the condition number of A with respect to the matrix norm |||*|||. Commonly, & 1 , & 2 and & ) , respectively denote the condition numbers of A with respect to the matrix norms |||A||| 1 , |||A||| 2 and |||A||| ) . Example Let A "
1
"2
"3
4
. Then A "1 "
"2
"1
" 32 " 12
and
|||A||| 1 " 21, |||A||| 2 " 14. 933 03 and |||A||| ) " 21. Condition numbers have the following properties. 1. &&I' " 1 for any matrix norm. 2. If A is invertible, then & A "1 " &&A'. 3. If A is invertible with eigenvalues # 1 , . . . , # n , then &&A' ' where # max &A' " max
1%i%n
|# i |
# max &A' for any matrix norm # min &A'
, and # min &A' " min
1%i%n
|# i |
.
4. &&AB' % &&A'&&B'.
Similarity Transformation
An n ! n matrix A is said to be similar to an n ! n matrix B, denoted by A , B, if there exists a nonsingular n ! n matrix S such that A " S "1 BS. The transformation from B to S "1 BS is called a similarity transformation. Similarity is an equivalence relation on square matrices, i.e., similarity is &a' reflective: A , A; &b' symmetric: B , A implies A , B; and &c' transitive: C , B and B , A imply C , A. Similar matrices have the following properties. 1. If A , B, then A and B have the same eigenvalues, counting multiplicity. 2. If A , B, then rank&A' "rank&B'. 3. If A , C and B , D, then A $ B , C $ D.
Special Matrices and Properties Symmetric and Hermitian Matrices
A real (complex) square matrix A is said to be symmetric (Hermitian) if A T " A &A ! " A', and skew-symmetric (skew-Hermitian) if A T " "A &A ! " "A'. Any real (complex) square matrix A can be written as A " A h $A s where A h " 1 &A $ A T ' A h " 1 &A $ A ! ' 2 2 a symmetric (Hermitian) matrix and A s " 1 &A " A T ' A s " 1 &A " A ! ' 2 2 a skew-symmetric (skew-Hermitian) matrix. Symmetric and Hermitian matrices have the following basic properties.
1. A $ A T , AA T and A T A are symmetric matrices and A $ A ! , AA ! and A ! A are Hermitian matrices. 2. If A is symmetric (Hermitian), then A k is symmetric (Hermitian) for all k " 1, 2, . . . . If A is also nonsingular, then A "1 is symmetric (Hermitian). 3. If A and B are symmetric (Hermitian), then !A $ "B is symmetric (Hermitian) for all real scalars ! and ". 4. If A is an n ! n symmetric (Hermitian) matrix, then x T Ax (x ! Ax' is real for all x in C n and therefore, all eigenvalues of A are real. 5. If A is an n ! n symmetric (Hermitian) matrix then S T AS &S ! AS' is symmetric (Hermitian) for all S in R n!n &C n!n '. # max &A' 6. If A is symmetric (Hermitian) and invertible, then %&A' " |||A||| 2 and & 2 &A' " # min &A' where # max &A' " max |# i | , and # min &A' " min |# i | , 1%i%n
# %i s
1%i%n
are eigenvalues of A.
Normal Matrices
An n ! n matrix A is said to be normal if A ! A " AA ! . If a matrix is symmetric or Hermitian, then it is also normal. If A * B is normal, then B * A is also normal.
Orthogonal and Unitary Matrices
An n ! n real (complex) matrix U is said to be orthogonal (unitary) if U T U " I n &U ! U " I n '. The determinant of an orthogonal matrix is either 1 or "1 since det&U 2 ' " &det&U'' 2 " 1. Example Let U1 "
1 2 1 2
"
1 2 1 2
, and U 2 "
"
1 2
"
1 2
"
1 2 1 2
.
Both U 1 and U 2 are orthogonal matrices since U T1 U 1 " I 2 and U T2 U 2 " I 2 . And det&U 1 ' " 1 and det&U 2 ' " "1. The following are equivalent: 1. U is unitary; 2. U is nonsingular and U ! " U "1 ; 3. UU ! " I; 4. The columns u i for i " 1, . . . , n of U are orthonormal, i.e., u !i u j "
0 if i ! j 1 if i " j
;
5. The rows of U are orthogonal; 6. For all x in C n , the 2-norm of y " Ux is the same as the one of x, i.e., y ! y " x ! x. Since UU ! " I " U ! U, U is also normal. If U 1 and U 2 are unitary, then U 1 U 2 and U 2 U 1 are also unitary. If A * B is unitary, then B * A is also unitary.
Positive Definite and Semidefinite Matrices
An n ! n matrix A is said to be positive definite (semidefinite) if A is a Hermitian matrix and x ! Ax ( 0 &x ! Ax ' 0' for all nonzero x in C n . Note that x ! Ax is always a nonnegative real number. For a positive definite (semidefinite) matrix A, commonly it is indicated as A ( 0 &A ' 0'. Positive definite (semidefinite) matrices have the following properties.
1. 2. 3. 4.
All eigenvalues of a positive definite (semidefinite) matrix are positive (nonnegative). All diagonal elements of a positive definite (semidefinite) matrix are positive (nonnegative). If A and B are positive definite (semidefinite), then A $ B is positive definite (semidefinite). Let A " $a ij % ni,j"1 be positive definite. Then n
det&A' %
( a kk k"1
with equality if and only if A is diagonal. 5. If A and B are positive definite (semidefinite), then A * B is positive (semidefinite).
Vandermonde matrices An n ! n Vandermonde matrix V is a matrix of the form
V"
1
x 1 ! x n"2 x n"1 1 1
1
x 2 ! x n"2 x n"1 2 2
" " #
"
"
.
x n"1 x n ! x n"2 n n
1
The matrix V depends on n elements x 1 , . . . , x n . The transpose of a Vandermonde matrix is also called a Vandermonde matrix. Note that the DFT matrix is Vandermonde. It is a fact that n
det&V' "
(
&x i " x j '.
i,j"1, i(j
So, a Vandermonde matrix is nonsingular if and only if x 1 , x 2 , . . . , x n are distinct. 1 2 Example Let V "
4
1 "3 9
. Here x 1 " 2, x 2 " "3 and x 4 " 4. Since x %i s are
1 4 16 distinct, det&V' ! 0. Actually, det&V' " &"3 " 2'&4 " 2'&4 " &"3'' " "70.
Circulant Matrices An n ! n matrix C of the form
C"
c1
c 2 ! c n"1
cn
c 1 ! c n"2 c n"1
cn " circ c 1 , c 2 , # , c n
c n"1 c n ! c n"3 c n"2 "
" #
"
"
c2
c3 !
cn
c1
is called a circulant matrix. Each row is obtained by cyclically shifting to the right the previous row. 1 2 3 4 Example C "circ&1, 2, 3, 4' "
4 1 2 3 3 4 1 2 2 3 4 1
.
A circulant matrix C "circ&c 1 , . . . , c n ' can also be written in the form
n"1
C"
$ c k$1 P , where P " k
0
1
0
!
0
0
0
1
!
0
" " # " "
k"0
0
0
0
0
1
1
0
0
0
0
.
Matrix P is called the basic circulant permutation matrix and satisfies the equation: P n " I n . Circulant matrices have the following properties. 1. C is circulant if and only if C ! is circulant. 2. If C 1 and C 2 are circulant, then C 1 C 2 is circulant. 3. Circulant matrices are normal matrices, i.e., C T C " CC T or C ! C " CC ! . 4. Let # l , x l for l " 1, . . . , n be eigenpairs of the circulant matrix C "circ&c 1 , . . . , c n '. Then 1 e "j&2'l/n'
n
#l "
$ c k e "j&2'&&k"1'l'/n' ,
and x l "
k"1
1 n
e "j&4'l/n'
for l " 1, . . . , n.
" e
"j&2&n"1''l/n'
Note that the eigenvectors of a circulant matrix do not depend on the c %i s. So, all n ! n circulant matrices for a given n have the same set of eigenvectors.
Toeplitz Matrices An n ! n matrix T of the form
T"
t0
t1
! t n"2 t n"1
t "1
t0
! t n"3 t n"2
t "2
t "1
! t n"4 t n"3
"
"
# "
"
t "&n"1' t "&n"2' ! t "1
t0
is called a Toeplitz matrix. A Toeplitz T depends on 2n " 1 elements t "&n"1' , t "&n"2' , . . . , t "1 , t 0 , t 1 , . . . , t n"2 , t n"1 . 1 2 3 4 Example T "
5 1 2 3 6 5 1 2
is a 4 ! 4 Toeplitz matrix.
7 6 5 1 Circulant matrices are also Toeplitz.
Hankel Matrices An n ! n matrix H of the form
H"
h0
h 1 ! h n"2
h n"1
h1
h 2 ! h n"1
hn
h2
h3 !
hn
h n$1
"
" #
"
"
h n"1 h n ! h 2n"2 h 2n"1 is called a Hankel matrix. A Hankel matrix H depends on 2n " 1 elements h 0 , h 1 , . . . , h 2n"1 , h 2n"1 and its elements are constant along the diagonals perpendicular to the main diagonal. 1 2 3 4 2 3 4 5
Example H "
.
3 4 5 6 4 5 6 7
A real square Hankel matrix is symmetric.
Hadamard Matrices
An n ! n matrix H whose elements are )1 such that HH T " nI n is called a Hadamard matrix. An n ! n Hadamard matrix with n ( 2 exists only if n is divisible by 4. The 2 ! 2 and 4 ! 4 Hadamard matrices H 2 and H 4 are 1 H2 "
1
1
1 "1
, and H 4 "
1
1
1
1 "1
1
"1
1
"1 "1
1
1 "1 "1
.
1
A 2n ! 2n Hadamard matrix can be obtained by H 2n " H 2 *H n . Note that a Hadamard matrix after normalization is unitary.
Diagonalization, Unitary Triangularization and Jordan Form Diagonalization A square matrix A is said to be diagonalizable if it is similar to a diagonal matrix D, i.e. there exists a nonsingular matrix S such that A " SDS "1 . Example Let A "
2 "1 "1 2 A"
and B "
1 1 0 1
1 "1
1 0
1 1
0 3
. Then 1 2 " 12
1 2 1 2
" SDS "1
is diagonalizable and B is not diagonalizable. Since A and D are similar, they have the same eigenvalues. Eigenvalues of D are diagonal elements of D. So eigenvalues of A are known if we know D. The process of finding matrices D and S is called the diagonalization. The following results identify diagonalizable matrices: let A be an n ! n matrix. 1. If A has a set of n linearly independent eigenvectors v 1 , . . . , v n , then A is diagonalizable and
A " S "1 DS where S "
v 1 ! v n and D "diag(# 1 , . . . , # n ), # i is the eigenvalue corresponding to the eigenvector v i . Recall that eigenvectors corresponding to distinct eigenvalues are linearly independent. So, if eigenvalues of A are distinct then A is diagonalizable. 2. Every symmetric (Hermitian) matrix can be diagonalized by an orthogonal (unitary) matrix U: A " UDU T &A " UDU ! '. Recall that if A is positive definite (semidefinite) then all eigenvalues # %i s of A are positive (nonnegative). Then D "diag(# 1 , . . . , # n ) can be written as D " D D where D & diag # 1 , . . . , # n , and A " U D D U T " VV T &A " VV ! ' where V " U D . 3. An n ! n matrix A is normal if and only if there exists unitary matrix U such that A " UDU ! where D "diag(# 1 , . . . , # n ) and # %i s are eigenvalues of A. 4. A n ! n circulant matrix C "circ&c 1 , . . . , c n ' can be diagonalized by 1 F, i.e., C " 1n F!F ! , n where 1 F"
1
1 2
!
1
!
m"1
1
(
(
1
(
(4
! ( 2&m"1'
"
"
"
#
1
(
, ( " e "j2'/m and j "
"1 ,
"
( m"1 ( 2&m"1' ! ( &m"1'
2
n
! " diag(# 1 , . . . , # n ) where # l "
$ c k e "j&2'&&k"1'l'/n' . k"1
5. If A and B are diagonalizable simultaneously, i.e., there exists a nonsingular matrix S such that A " SD A S "1 and B " SD B S "1 where D A and D B are diagonal matrices, then A and B commute, i.e., AB " BA.
Unitary Triangularization
For any square matrix A, there is a unitary matrix U such that A " UTU ! where T is upper triangular. So, every square matrix is similar to an upper triangular matrix. Note that eigenvalues of A are diagonal elements of T.
Jordan Canonical Form For any square matrix A, there is a nonsingular matrix X such that A " X diag J k 1 &# 1 ', J k 2 &# 2 ', # , J k l &# l ' X "1 where J k &#', called a Jordan block, in C k!k is of the form
J k &#' "
#
1
0
!
0
0
0
#
1
!
0
0
0
0
#
!
0
0
" " " # " " 0
0
0
! #
1
0
0
0
!
#
0
and # 1 , . . . , # l are eigenvalues of A with multiplicity k 1 , . . . , k l , respectively. The block diagonal matrix
! 0
J k 1 &# 1 ' 0 diag J k 1 &# 1 ', J k 2 &# 2 ', # , J k l &# l '
"
0
J k 2 &# 2 ' ! 0
"
"
# "
0
0
! J k l &# 2 '
is called a Jordan matrix. The Jordan canonical form is a set of Jordan matrices. Observe that a Jordan matrix is “almost diagonal” and is diagonal when all k %i s are 1 (then A is diagonalizable).
Singular Values, Singular Value Decomposition and Pseudo-inverses Singular Values
Singular values of an m ! n matrix A, denoted by ) i &A', are the square root of eigenvalues of the matrix AA ! . Example Let A "
1 "2
"11
5
. Then AA T "
, and its eigenvalues are "3 4 "11 25 # 1 " 15 $ 221 and # 2 " 15 " 221 . So, singular values of A are: ) 1 " 15 $ 221 " 5. 464986 and ) 2 " 15 " 221 .
Notice that the spectral norm of a square matrix A is max() i ), that is |||A||| 2 " max() i ).
Singular Value Decomposition
Let A be an m ! n matrix. Then A can be written as A " U"V ! where U is an m ! m unitary matrix, V is an n ! n unitary matrix and ""
"p
or " "
0
"p 0
is an m ! n matrix where " A "diag() 1 , . . . , ) p ), p " min(m, n) and ) 1 ' . . . ' ) p . U"V ! is called the singular value decomposition, SVD in short, of A. Columns of U are eigenvectors of AA ! and columns of V are eigenvectors of A ! A. If A is a real matrix, then U and V are orthogonal matrices. Example Let A "
1 "2
1 "2 3
and B "
"4 5 "6
"3 4
. The numerical SVD of A and B
5 "6
are: A"
"0. 386 32 0. 922 37 0. 922 37
9. 508
0. 386 32
0
0
"0. 428 67 0. 566 31 "0. 703 95
0
0. 772 87 0
"0. 805 96 0. 112 38
0. 581 20
0. 408 25
0. 408 25
0. 816 50
and "0. 229 85 "0. 883 46 0. 408 25 B"
9. 525 5
0
0. 524 74
0. 240 78
0. 816 50
0
0. 514 3
"0. 819 64
0. 401 90
0. 408 25
0
0
If the rank of A is r, then ) 1 ' . . . ' ) r ( 0 and ) r$1 " 0, . . . , ) p " 0.
"0. 619 63 0. 784 89 0. 784 89
0. 619 63
Pseudo-inverses
Let A be an m ! n matrix with rank r and A " U"V ! . Then "" Define " "1 "
&" r ' "1 0
"r 0 0
0
where " r " diag ) 1 , . . . , ) r .
. Then A " " V" "1 U ! is called the pesudo-inverse of A, or the
0 0 Moore-Penrose generalized inverse of A. It can be shown that A " satisfies the following properties: 1. AA " and A " A are Hermitian; 2. AA " A " A; and 3. A " AA " " A. One of the applications is to solve the linear system Ax " b where A " U"V ! is an m ! n matrix with rank r. A least-squares solution x ls to the linear system Ax " b is x ls " A " b " &V" "1 U ! 'b.
References 1. G. Golub and C. Van Loan, “Matrix Computation”, Johns Hopkins University Press, 1983. 2. R. Horn and C. Johnson, “Matrix Analysis”, Cambridge University Press, 1985. 3. G. Strang, “ Introduction to Linear Algebra”, Wellesley-Cambridge Press, 1998
