MATRIX DIFFERENTIAL CALCULUS WITH APPLICATIONS IN STATISTICS AND ECONOMETRICS, 3RD ED.......Page 1
Back Cover......Page 2
Title Page......Page 5
Copyright Page......Page 6
Contents......Page 7
Preface......Page 15
Part I: Matrices......Page 21
2 Sets......Page 23
3 Matrices: addition and multiplication......Page 24
5 Square matrices......Page 26
6 Linear forms and quadratic forms......Page 27
7 The rank of a matrix......Page 28
8 The inverse......Page 29
9 The determinant......Page 30
11 Partitioned matrices......Page 31
12 Complex matrices......Page 33
13 Eigenvalues and eigenvectors......Page 34
14 Schur’s decomposition theorem......Page 37
15 The Jordan decomposition......Page 38
16 The singular-value decomposition......Page 39
17 Further results concerning eigenvalues......Page 40
18 Positive (semi)definite matrices......Page 43
19 Three further results for positive definite matrices......Page 45
Miscellaneous exercises......Page 47
Bibliographical notes......Page 49
2 The adjoint matrix......Page 67
3 Proof of Theorem 1......Page 69
5 The matrix equation AX = 0......Page 71
6 The Hadamard product......Page 73
7 The commutation matrix K mn......Page 74
8 The duplication matrix D n......Page 76
9 Relationship between D n+1 and D n , I......Page 78
10 Relationship between D n+1 and D n , II......Page 80
11 Conditions for a quadratic form to be positive (negative) subject to linear constraints......Page 81
12 Necessary and sufficient conditions for r(A:B) = r(A) + r(B)......Page 84
13 The bordered Gramian matrix......Page 86
14 The equations X1 A + X2 B' = G1, X1 B = G2......Page 88
Bibliographical notes......Page 91
2 The Kronecker product......Page 51
3 Eigenvalues of a Kronecker product......Page 53
4 The vec operator......Page 54
5 The Moore–Penrose (MP) inverse......Page 56
6 Existence and uniqueness of the MP inverse......Page 57
7 Some properties of the MP inverse......Page 58
8 Further properties......Page 59
9 The solution of linear equation systems......Page 61
Miscellaneous exercises......Page 63
Bibliographical notes......Page 65
Part II: Differentials: the theory......Page 93
2 Interior points and accumulation points......Page 95
3 Open and closed sets......Page 96
4 The Bolzano–Weierstrass theorem......Page 99
5 Functions......Page 100
6 The limit of a function......Page 101
7 Continuous functions and compactness......Page 102
8 Convex sets......Page 103
9 Convex and concave functions......Page 105
Bibliographical notes......Page 108
2 Continuity......Page 109
3 Differentiability and linear approximation......Page 111
4 The differential of a vector function......Page 113
5 Uniqueness of the differential......Page 115
6 Continuity of differentiable functions......Page 116
7 Partial derivatives......Page 117
8 The first identification theorem......Page 118
9 Existence of the differential, I......Page 119
10 Existence of the differential, II......Page 121
12 The chain rule......Page 123
13 Cauchy invariance......Page 125
14 The mean-value theorem for real-valued functions......Page 126
15 Matrix functions......Page 127
16 Some remarks on notation......Page 129
Miscellaneous exercises......Page 130
Bibliographical notes......Page 131
2 Second-order partial derivatives......Page 133
3 The Hessian matrix......Page 134
4 Twice differentiability and second-order approximation, I......Page 135
5 Definition of twice differentiability......Page 136
6 The second differential......Page 138
7 (Column) symmetry of the Hessian matrix......Page 140
8 The second identification theorem......Page 142
9 Twice differentiability and second-order approximation, II......Page 143
10 Chain rule for Hessian matrices......Page 145
11 The analogue for second differentials......Page 146
12 Taylor’s theorem for real-valued functions......Page 148
14 Matrix functions......Page 149
Bibliographical notes......Page 151
1 Introduction......Page 153
2 Unconstrained optimization......Page 154
3 The existence of absolute extrema......Page 155
4 Necessary conditions for a local minimum......Page 157
5 Sufficient conditions for a local minimum: first-derivative test......Page 158
6 Sufficient conditions for a local minimum: second-derivative tes......Page 160
7 Characterization of differentiable convex functions......Page 162
8 Characterization of twice differentiable convex functions......Page 165
10 Monotonic transformations......Page 167
11 Optimization subject to constraints......Page 168
12 Necessary conditions for a local minimum under constraints......Page 169
13 Sufficient conditions for a local minimum under constraints......Page 174
14 Sufficient conditions for an absolute minimum under constraint......Page 178
15 A note on constraints in matrix form......Page 179
16 Economic interpretation of Lagrange multipliers......Page 180
Appendix: the implicit function theorem......Page 182
Bibliographical notes......Page 183
Part III: Differentials: the practice......Page 185
2 Fundamental rules of differential calculus......Page 187
3 The differential of a determinant......Page 189
4 The differential of an inverse......Page 191
5 Differential of the Moore–Penrose inverse......Page 192
6 The differential of the adjoint matrix......Page 195
7 On differentiating eigenvalues and eigenvectors......Page 197
8 The differential of eigenvalues and eigenvectors: symmetric case......Page 199
9 The differential of eigenvalues and eigenvectors: complex case......Page 202
10 Two alternative expressions for dλ......Page 205
11 Second differential of the eigenvalue function......Page 208
Miscellaneous exercises......Page 209
Bibliographical notes......Page 212
2 Classification......Page 213
3 Bad notation......Page 214
4 Good notation......Page 216
6 The first identification table......Page 218
7 Partitioning of the derivative......Page 219
9 Scalar functions of a matrix, I: trace......Page 220
10 Scalar functions of a matrix, II: determinant......Page 222
12 Two examples of vector functions......Page 224
13 Matrix functions......Page 225
14 Kronecker products......Page 228
15 Some other problems......Page 230
Bibliographical notes......Page 231
2 The Hessian matrix of a matrix function......Page 233
3 Identification of Hessian matrices......Page 234
4 The second identification table......Page 235
6 Scalar functions......Page 237
7 Vector functions......Page 239
8 Matrix functions, I......Page 240
9 Matrix functions, II......Page 241
Part IV: Inequalities......Page 243
2 The Cauchy-Schwarz inequality......Page 245
3 Matrix analogues of the Cauchy–Schwarz inequality......Page 247
4 The theorem of the arithmetic and geometric means......Page 248
5 The Rayleigh quotient......Page 250
6 Concavity of λ 1, convexity of λ n......Page 251
7 Variational description of eigenvalues......Page 252
8 Fischer’s min-max theorem......Page 253
9 Monotonicity of the eigenvalues......Page 255
10 The Poincaré separation theorem......Page 256
11 Two corollaries of Poincaré’s theorem......Page 257
12 Further consequences of the Poincaré theorem......Page 258
13 Multiplicative version......Page 259
14 The maximum of a bilinear form......Page 261
15 Hadamard’s inequality......Page 262
16 An interlude: Karamata’s inequality......Page 263
18 An inequality concerning positive semidefinite matrices......Page 265
19 A representation theorem for (∑ a i ^p)^(1/p)......Page 266
20 A representation theorem for (tr A^p)^(1/p)......Page 268
21 Hölder’s inequality......Page 269
22 Concavity of log|A|......Page 270
23 Minkowski’s inequality......Page 272
24 Quasilinear representation of |A|^(1/n)......Page 274
26 Weighted means of order p......Page 276
27 Schlömilch’s inequality......Page 279
28 Curvature properties of M p (x, a)......Page 280
29 Least squares......Page 281
31 Restricted least squares......Page 283
32 Restricted least squares: matrix version......Page 285
Miscellaneous exercises......Page 286
Bibliographical notes......Page 290
Part V: The linear model......Page 293
2 The cumulative distribution function......Page 295
4 Expectations......Page 296
5 Variance and covariance......Page 297
6 Independence of two random variables......Page 299
9 The one-dimensional normal distribution......Page 301
10 The multivariate normal distribution......Page 302
11 Estimation......Page 304
Miscellaneous exercises......Page 305
Bibliographical notes......Page 306
1 Introduction......Page 307
2 Affine minimum-trace unbiased estimation......Page 308
3 The Gauss–Markov theorem......Page 309
4 The method of least squares......Page 312
5 Aitken’s theorem......Page 313
6 Multicollinearity......Page 315
7 Estimable functions......Page 317
8 Linear constraints: the case M(R´)⊂M(X´)......Page 319
9 Linear constraints: the general case......Page 322
10 Linear constraints: the case M(R´)∩M(X´) = {0}......Page 325
11 A singular variance matrix: the case M(X)⊂M(V )......Page 326
12 A singular variance matrix: the case r(X´V+X) = r(X)......Page 328
13 A singular variance matrix: the general case, I......Page 329
14 Explicit and implicit linear constraints......Page 330
15 The general linear model, I......Page 333
16 A singular variance matrix: the general case, II......Page 334
17 The general linear model, II......Page 337
18 Generalized least squares......Page 338
19 Restricted least squares......Page 339
Miscellaneous exercises......Page 341
Bibliographical notes......Page 342
2 Best quadratic unbiased estimation of σ²......Page 343
3 The best quadratic and positive unbiased estimator of σ²......Page 344
4 The best quadratic unbiased estimator of σ²......Page 346
5 Best quadratic invariant estimation of σ²......Page 349
6 The best quadratic and positive invariant estimator of σ²......Page 350
7 The best quadratic invariant estimator of σ²......Page 351
8 Best quadratic unbiased estimation: multivariate normal case......Page 352
9 Bounds for the bias of the least squares estimator of σ², I......Page 355
10 Bounds for the bias of the least squares estimator of σ², II......Page 356
11 The prediction of disturbances......Page 358
12 Best linear unbiased predictors with scalar variance matrix......Page 359
13 Best linear unbiased predictors with fixed variance matrix, I......Page 361
14 Best linear unbiased predictors with fixed variance matrix, II......Page 364
15 Local sensitivity of the posterior mean......Page 365
16 Local sensitivity of the posterior precision......Page 367
Bibliographical notes......Page 368
Part VI: Applications to maximum likelihood estimation......Page 369
2 The method of maximum likelihood (ML)......Page 371
3 ML estimation of the multivariate normal distribution......Page 372
4 Symmetry: implicit versus explicit treatment......Page 374
5 The treatment of positive definiteness......Page 375
6 The information matrix......Page 376
7 ML estimation of the multivariate normal distribution: distinct means......Page 377
8 The multivariate linear regression model......Page 378
9 The errors-in-variables model......Page 381
10 The non-linear regression model with normal errors......Page 384
11 Special case: functional independence of mean- and variance parameters......Page 385
12 Generalization of Theorem 6......Page 386
Miscellaneous exercises......Page 388
Bibliographical notes......Page 390
2 The simultaneous equations model......Page 391
3 The identification problem......Page 393
5 Identification with linear constraints on B, Γ and Σ......Page 395
6 Non-linear constraints......Page 397
7 Full-information maximum likelihood (FIML): the information matrix (general case)......Page 398
8 Full-information maximum likelihood (FIML): the asymptotic variance matrix (special case)......Page 400
9 Limited-information maximum likelihood (LIML): the first-order conditions......Page 403
10 Limited-information maximum likelihood (LIML): the information matrix......Page 406
11 Limited-information maximum likelihood (LIML): the asymptotic variance matrix......Page 408
Bibliographical notes......Page 413
1 Introduction......Page 415
2 Population principal components......Page 416
3 Optimality of principal components......Page 417
4 A related result......Page 418
5 Sample principal components......Page 419
8 One-mode component analysis......Page 421
9 One-mode component analysis and sample principal components......Page 424
10 Two-mode component analysis......Page 425
11 Multimode component analysis......Page 426
12 Factor analysis......Page 430
13 A zigzag routine......Page 433
14 A Newton–Raphson routine......Page 435
15 Kaiser’s varimax method......Page 438
16 Canonical correlations and variates in the population......Page 441
Bibliographical notes......Page 443
Bibliography......Page 447
Index of symbols......Page 459
Subject index......Page 463