Nov 15, 2012 - Its ongoing: The project of keeping a large repository of relations involving matrices is naturally ongoi
The Matrix Cookbook [ http://matrixcookbook.com ] Kaare Brandt Petersen Michael Syskind Pedersen Version: November 15, 2012
1
Introduction What is this? These pages are a collection of facts (identities, approximations, inequalities, relations, ...) about matrices and matters relating to them. It is collected in this form for the convenience of anyone who wants a quick desktop reference . Disclaimer: The identities, approximations and relations presented here were obviously not invented but collected, borrowed and copied from a large amount of sources. These sources include similar but shorter notes found on the internet and appendices in books - see the references for a full list. Errors: Very likely there are errors, typos, and mistakes for which we apologize and would be grateful to receive corrections at
[email protected]. Its ongoing: The project of keeping a large repository of relations involving matrices is naturally ongoing and the version will be apparent from the date in the header. Suggestions: Your suggestion for additional content or elaboration of some topics is most welcome
[email protected]. Keywords: Matrix algebra, matrix relations, matrix identities, derivative of determinant, derivative of inverse matrix, differentiate a matrix. Acknowledgements: We would like to thank the following for contributions and suggestions: Bill Baxter, Brian Templeton, Christian Rishøj, Christian Schr¨ oppel, Dan Boley, Douglas L. Theobald, Esben Hoegh-Rasmussen, Evripidis Karseras, Georg Martius, Glynne Casteel, Jan Larsen, Jun Bin Gao, J¨ urgen Struckmeier, Kamil Dedecius, Karim T. Abou-Moustafa, Korbinian Strimmer, Lars Christiansen, Lars Kai Hansen, Leland Wilkinson, Liguo He, Loic Thibaut, Markus Froeb, Michael Hubatka, Miguel Bar˜ao, Ole Winther, Pavel Sakov, Stephan Hattinger, Troels Pedersen, Vasile Sima, Vincent Rabaud, Zhaoshui He. We would also like thank The Oticon Foundation for funding our PhD studies.
Petersen & Pedersen, The Matrix Cookbook, Version: November 15, 2012, Page 2
CONTENTS
CONTENTS
Contents 1 Basics 1.1 Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 The Special Case 2x2 . . . . . . . . . . . . . . . . . . . . . . . . . 2 Derivatives 2.1 Derivatives 2.2 Derivatives 2.3 Derivatives 2.4 Derivatives 2.5 Derivatives 2.6 Derivatives 2.7 Derivatives 2.8 Derivatives
of of of of of of of of
a Determinant . . . . . . . . . . . . an Inverse . . . . . . . . . . . . . . . Eigenvalues . . . . . . . . . . . . . . Matrices, Vectors and Scalar Forms Traces . . . . . . . . . . . . . . . . . vector norms . . . . . . . . . . . . . matrix norms . . . . . . . . . . . . . Structured Matrices . . . . . . . . .
3 Inverses 3.1 Basic . . . . . . . . . . . 3.2 Exact Relations . . . . . 3.3 Implication on Inverses . 3.4 Approximations . . . . . 3.5 Generalized Inverse . . . 3.6 Pseudo Inverse . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
6 6 6 7
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
8 8 9 10 10 12 14 14 14
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
17 17 18 20 20 21 21
4 Complex Matrices 24 4.1 Complex Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.2 Higher order and non-linear derivatives . . . . . . . . . . . . . . . 26 4.3 Inverse of complex sum . . . . . . . . . . . . . . . . . . . . . . . 27 5 Solutions and Decompositions 5.1 Solutions to linear equations . 5.2 Eigenvalues and Eigenvectors 5.3 Singular Value Decomposition 5.4 Triangular Decomposition . . 5.5 LU decomposition . . . . . . 5.6 LDM decomposition . . . . . 5.7 LDL decompositions . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
28 28 30 31 32 32 33 33
6 Statistics and Probability 34 6.1 Definition of Moments . . . . . . . . . . . . . . . . . . . . . . . . 34 6.2 Expectation of Linear Combinations . . . . . . . . . . . . . . . . 35 6.3 Weighted Scalar Variable . . . . . . . . . . . . . . . . . . . . . . 36 7 Multivariate Distributions 7.1 Cauchy . . . . . . . . . . 7.2 Dirichlet . . . . . . . . . . 7.3 Normal . . . . . . . . . . 7.4 Normal-Inverse Gamma . 7.5 Gaussian . . . . . . . . . . 7.6 Multinomial . . . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
37 37 37 37 37 37 37
Petersen & Pedersen, The Matrix Cookbook, Version: November 15, 2012, Page 3
CONTENTS
7.7 7.8 7.9
CONTENTS
Student’s t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wishart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wishart, Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . .
8 Gaussians 8.1 Basics . . . . . . . . 8.2 Moments . . . . . . 8.3 Miscellaneous . . . . 8.4 Mixture of Gaussians
37 38 39
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
40 40 42 44 44
9 Special Matrices 9.1 Block matrices . . . . . . . . . . . . . . . . 9.2 Discrete Fourier Transform Matrix, The . . 9.3 Hermitian Matrices and skew-Hermitian . . 9.4 Idempotent Matrices . . . . . . . . . . . . . 9.5 Orthogonal matrices . . . . . . . . . . . . . 9.6 Positive Definite and Semi-definite Matrices 9.7 Singleentry Matrix, The . . . . . . . . . . . 9.8 Symmetric, Skew-symmetric/Antisymmetric 9.9 Toeplitz Matrices . . . . . . . . . . . . . . . 9.10 Transition matrices . . . . . . . . . . . . . . 9.11 Units, Permutation and Shift . . . . . . . . 9.12 Vandermonde Matrices . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
46 46 47 48 49 49 50 52 54 54 55 56 57
10 Functions and Operators 10.1 Functions and Series . . . . . 10.2 Kronecker and Vec Operator 10.3 Vector Norms . . . . . . . . . 10.4 Matrix Norms . . . . . . . . . 10.5 Rank . . . . . . . . . . . . . . 10.6 Integral Involving Dirac Delta 10.7 Miscellaneous . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
58 58 59 61 61 62 62 63
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Functions . . . . . .
. . . .
. . . . . . .
. . . . . . .
A One-dimensional Results 64 A.1 Gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 A.2 One Dimensional Mixture of Gaussians . . . . . . . . . . . . . . . 65 B Proofs and Details 66 B.1 Misc Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Petersen & Pedersen, The Matrix Cookbook, Version: November 15, 2012, Page 4
CONTENTS
CONTENTS
Notation and Nomenclature A Aij Ai Aij An A−1 A+ A1/2 (A)ij Aij [A]ij a ai ai a
