Multivariate Data Analysis on Matrix Manifolds: (with Manopt)

✍ Scribed by Nickolay Trendafilov, Michele Gallo

• Publisher: Springer
• Year: 2021
• Tongue: English
• Leaves: 467
• Series: Springer Series in the Data Sciences
• Edition: 1
• Category: Library

✦ Synopsis

This graduate-level textbook aims to give a unified presentation and solution of several commonly used techniques for multivariate data analysis (MDA). Unlike similar texts, it treats the MDA problems as optimization problems on matrix manifolds defined by the MDA model parameters, allowing them to be solved using (free) optimization software Manopt. The book includes numerous in-text examples as well as Manopt codes and software guides, which can be applied directly or used as templates for solving similar and new problems. The first two chapters provide an overview and essential background for studying MDA, giving basic information and notations. Next, it considers several sets of matrices routinely used in MDA as parameter spaces, along with their basic topological properties. A brief introduction to matrix (Riemannian) manifolds and optimization methods on them with Manopt complete the MDA prerequisite. The remaining chapters study individual MDA techniques in depth. The number of exercises complement the main text with additional information and occasionally involve open and/or challenging research questions. Suitable fields include computational statistics, data analysis, data mining and data science, as well as theoretical computer science, machine learning and optimization. It is assumed that the readers have some familiarity with MDA and some experience with matrix analysis, computing, and optimization. 

✦ Table of Contents

Preface
Contents
1 Introduction
2 Matrix analysis and differentiation
2.1 Matrix algebra
2.2 Vector spaces, bases, and linear maps
2.3 Metric, normed and inner product spaces
2.4 Euclidean spaces of matrices and their norms
2.5 Matrix differentials and gradients
2.6 Conclusion
2.7 Exercises
3 Matrix manifolds in MDA
3.1 Several useful matrix sets
3.1.1 mathcalGL(n)
3.1.2 mathcalO(n)
3.1.3 mathcalO(n,p)
3.1.4 mathcalO0(n,p)
3.1.5 mathcalOB(n)
3.1.6 mathcalOB(n,p)
3.1.7 mathcalG(n,p)
3.2 Differentiable manifolds
3.3 Examples of matrix manifolds in MDA
3.3.1 mathcalGL(n)
3.3.2 mathcalO(n)
3.3.3 mathcalO(n,p)
3.3.4 mathcalO0(n,p)
3.3.5 mathcalOB(n)
3.3.6 mathcalG(n,p)
3.4 Tangent spaces
3.4.1 mathcalGL(n)
3.4.2 mathcalO(n)
3.4.3 mathcalO(n,p)
3.4.4 mathcalO0(n,p)
3.4.5 mathcalOB(n)
3.4.6 mathcalOB(n,p)
3.4.7 mathcalG(n,p)
3.5 Optimization on matrix manifolds
3.5.1 Dynamical systems
3.5.2 Iterative schemes
3.5.3 Do it yourself!
3.6 Optimization with Manopt
3.6.1 Matrix manifolds in Manopt
3.6.2 Solvers
3.6.3 Portability
3.7 Conclusion
3.8 Exercises
4 Principal component analysis (PCA)
4.1 Introduction
4.2 Definition and main properties
4.3 Correspondence analysis (CA)
4.4 PCA interpretation
4.5 Simple structure rotation in PCA (and FA)
4.5.1 Simple structure concept
4.5.2 Simple structure criteria and rotation methods
4.5.3 Case study: PCA interpretation via rotation methods
4.5.4 Rotation to independent components
4.6 True simple structure: sparse loadings
4.6.1 SPARSIMAX: rotation-like sparse loadings
4.6.2 Know-how for applying SPARSIMAX
4.6.3 Manopt code for SPARSIMAX
4.7 Sparse PCA
4.7.1 Sparse components: genesis, history and present times
4.7.2 Taxonomy of PCA subject to ell1 constraint (LASSO)
4.8 Function-constrained sparse components
4.8.1 Orthonormal sparse component loadings
4.8.2 Uncorrelated sparse components
4.8.3 Numerical Example
4.8.4 Manopt code for weakly correlated sparse components with nearly orthonormal loadings
4.9 New generation dimension reduction
4.9.1 Centroid method
4.9.2 Randomized SVD
4.9.3 CUR approximations
4.9.4 The Nyström Method
4.10 Conclusion
4.11 Exercises
5 Factor analysis (FA)
5.1 Introduction
5.2 Fundamental equations of EFA
5.2.1 Population EFA definition
5.2.2 Sample EFA definition
5.3 EFA parameters estimation
5.3.1 Classical EFA estimation
5.3.2 EFA estimation on manifolds
5.4 ML exploratory factor analysis
5.4.1 Gradients
5.4.2 Optimality conditions
5.5 LS and GLS exploratory factor analysis
5.5.1 Gradients
5.5.2 Optimality conditions
5.6 Manopt codes for classical …
5.6.1 Standard case: Ψ2 Op
5.6.2 Avoiding Heywood cases: Ψ2 > Op
5.7 EFA as a low-rank-plus-sparse matrix …
5.8 Sparse EFA
5.8.1 Introduction
5.8.2 Sparse factor loadings with penalized EFA
5.8.3 Implementing sparseness
5.8.4 Numerical examples
5.9 Comparison to other methods
5.10 Conclusion
5.11 Exercises
6 Procrustes analysis (PA)
6.1 Introduction
6.2 Orthonormal PA
6.2.1 Orthogonal Penrose regression (OPR)
6.2.2 Projected Hessian
6.2.3 Orthonormal Penrose regression (OnPR)
6.2.4 Ordinary orthonormal PA
6.3 Oblique PA
6.3.1 Basic formulations and solutions
6.4 Robust PA
6.4.1 Some history remarks
6.4.2 Robust OnPR
6.4.3 Robust oblique PA
6.4.4 PA with M-estimator
6.5 Multi-mode PA
6.5.1 PCA and one-mode PCA
6.5.2 Multi-mode PCA and related PA problems
6.5.3 Global minima on calO(p)
6.6 Some other PA problems
6.6.1 Average of rotated matrices: generalized PA
6.6.2 Mean rotation
6.7 PA application to EFA of large data
6.7.1 The classical case n > p
6.7.2 The modern case p ggn
6.7.3 EFA and RPCA when pggn
6.7.4 Semi-sparse PCA (well-defined EFA)
6.8 Conclusion
6.9 Exercises
7 Linear discriminant analysis (LDA)
7.1 Introduction
7.2 LDA of vertical data (n>p)
7.2.1 Standard canonical variates (CVs)
7.2.2 Orthogonal canonical variates (OCVs)
7.3 Sparse CVs and sparse OCVs
7.4 LDA of horizontal data (p > n)
7.4.1 LDA through GSVD
7.4.2 LDA and pattern recognition
7.4.3 Null space LDA (NLDA)
7.4.4 LDA with CPC, PLS and MDS
7.4.5 Sparse LDA with diagonal W
7.4.6 Function-constrained sparse LDA
7.4.7 Sparse LDA based on minimization of the classification error
7.4.8 Sparse LDA through optimal scoring (SLDA)
7.4.9 Multiclass sparse discriminant analysis
7.4.10 Sparse LDA through GEVD
7.4.11 Sparse LDA without sparse-inducing penalty
7.5 Conclusion
7.6 Exercises
8 Cannonical correlation analysis (CCA)
8.1 Introduction
8.2 Classical CCA Formulation and Solution
8.3 Alternative CCA Definitions
8.4 Singular Scatter Matrices C11 and/or C22
8.5 Sparse CCA
8.5.1 Sparse CCA Through Sparse GEVD
8.5.2 LS Approach to Sparse CCA
8.6 CCA Relation to LDA and PLS
8.6.1 CCA and LDA
8.6.2 CCA and PLS
8.7 More Than Two Groups of Variables
8.7.1 CCA Generalizations
8.7.2 CCA Based on CPC
8.8 Conclusion
8.9 Exercises
9 Common principal components (CPC)
9.1 Introduction
9.2 CPC estimation problems
9.3 ML- and LS-CPC
9.3.1 Gradients and optimality conditions
9.3.2 Example: Fisher's Iris data
9.3.3 Appendix: MATLAB code for FG algorithm
9.4 New procedures for CPC estimation
9.4.1 Classic numerical solutions of ML- and LS-CPC
9.4.2 Direct calculation of individual eigenvalues/variances
9.4.3 CPC for known individual variances
9.5 CPC for dimension reduction
9.6 Proportional covariance matrices
9.6.1 ML and LS proportional principal components
9.6.2 Dimension reduction with PPC
9.7 Some relations between CPC and ICA
9.7.1 ICA formulations
9.7.2 ICA by contrast functions
9.7.3 ICA methods based on diagonalization
9.8 Conclusion
9.9 Exercises
10 Metric multidimensional scaling (MDS) and related methods
10.1 Introduction
10.2 Proximity measures
10.3 Metric MDS
10.3.1 Basic identities and classic solution
10.3.2 MDS that fits distances directly
10.3.3 Some related/adjacent MDS problems
10.4 INDSCAL – Individual Differences Scaling
10.4.1 The classical INDSCAL solution and some problems
10.4.2 Orthonormality-constrained INDSCAL
10.5 DINDSCAL – Direct INDSCAL
10.5.1 DINDSCAL model
10.5.2 DINDSCAL solution
10.5.3 Manopt code for DINDSCAL
10.6 DEDICOM
10.6.1 Introduction
10.6.2 Alternating DEDICOM
10.6.3 Simultaneous DEDICOM
10.7 GIPSCAL
10.7.1 GIPSCAL model
10.7.2 GISPSCAL solution
10.7.3 Three-way GIPSCAL
10.8 Tensor data analysis
10.8.1 Basic notations and definitions
10.8.2 CANDECOMP/PARAFAC (CP)
10.8.3 Three-mode PCA (TUCKER3)
10.8.4 Multi-mode PCA
10.8.5 Higher order SVD (HOSVD)
10.9 Conclusion
10.10 Exercises
11 Data analysis on simplexes
11.1 Archetypal analysis (AA)
11.1.1 Introduction
11.1.2 Definition of the AA problem
11.1.3 AA solution on multinomial manifold
11.1.4 AA solution on oblique manifolds
11.1.5 AA as interior point flows
11.1.6 Conclusion
11.1.7 Exercises
11.2 Analysis of compositional data (CoDa)
11.2.1 Introduction
11.2.2 Definition and main properties
11.2.3 Geometric clr structure of the data simplex
11.2.4 PCA and sparse PCA for compositions
11.2.5 Case study
11.2.6 Manopt code for sparse PCA of CoDa
11.2.7 Conclusion
11.2.8 Exercises
Bibliography