Contents
Preface
Common Optimization Techniques, Equations, Symbols, and Acronyms
Part I Dimensionality Reduction and Transforms
1 Singular Value Decomposition (SVD)
1.1 Overview
1.2 Matrix Approximation
1.3 Mathematical Properties and Manipulations
1.4 Pseudo-Inverse, Least-Squares, and Regression
1.5 Principal Component Analysis (PCA)
1.6 Eigenfaces Example
1.7 Truncation and Alignment
1.8 Randomized Singular Value Decomposition
1.9 Tensor Decompositions and N-Way Data Arrays
2 Fourier and Wavelet Transforms
2.1 Fourier Series and Fourier Transforms
2.2 Discrete Fourier Transform (DFT) and Fast Fourier Transform (FFT)
2.3 Transforming Partial Differential Equations
2.4 Gabor Transform and the Spectrogram
2.5 Wavelets and Multi-Resolution Analysis
2.6 2D Transforms and Image Processing
3 Sparsity and Compressed Sensing
3.1 Sparsity and Compression
3.2 Compressed Sensing
3.3 Compressed Sensing Examples
3.4 The Geometry of Compression
3.5 Sparse Regression
3.6 Sparse Representation
3.7 Robust Principal Component Analysis (RPCA)
3.8 Sparse Sensor Placement
Part II Machine Learning and Data Analysis
4 Regression and Model Selection
4.1 Classic Curve Fitting
4.2 Nonlinear Regression and Gradient Descent
4.3 Regression and Ax = b: Over- and Under-Determined Systems
4.4 Optimization as the Cornerstone of Regression
4.5 The Pareto Front and Lex Parsimoniae
4.6 Model Selection: Cross-Validation
4.7 Model Selection: Information Criteria
5 Clustering and Classification
5.1 Feature Selection and Data Mining
5.2 Supervised versus Unsupervised Learning
5.3 Unsupervised Learning: k-means Clustering
5.4 Unsupervised Hierarchical Clustering: Dendrogram
5.5 Mixture Models and the Expectation-Maximization Algorithm
5.6 Supervised Learning and Linear Discriminants
5.7 Support Vector Machines (SVM)
5.8 Classification Trees and Random Forest
5.9 Top 10 Algorithms in Data Mining 2008
6 Neural Networks and Deep Learning
6.1 Neural Networks: 1-Layer Networks
6.2 Multi-Layer Networks and Activation Functions
6.3 The Backpropagation Algorithm
6.4 The Stochastic Gradient Descent Algorithm
6.5 Deep Convolutional Neural Networks
6.6 Neural Networks for Dynamical Systems
6.7 The Diversity of Neural Networks
Part III Dynamics and Control
7 Data-Driven Dynamical Systems
7.1 Overview, Motivations, and Challenges
7.2 Dynamic Mode Decomposition (DMD)
7.3 Sparse Identification of Nonlinear Dynamics (SINDy)
7.4 Koopman Operator Theory
7.5 Data-Driven Koopman Analysis
8 Linear Control Theory
8.1 Closed-Loop Feedback Control
8.2 Linear Time-Invariant Systems
8.3 Controllability and Observability
8.4 Optimal Full-State Control: Linear Quadratic Regulator (LQR)
8.5 Optimal Full-State Estimation: The Kalman Filter
8.6 Optimal Sensor-Based Control: Linear Quadratic Gaussian (LQG)
8.7 Case Study: Inverted Pendulum on a Cart
8.8 Robust Control and Frequency Domain Techniques
9 Balanced Models for Control
9.1 Model Reduction and System Identification
9.2 Balanced Model Reduction
9.3 System identification
10 Data-Driven Control
10.1 Nonlinear System Identification for Control
10.2 Machine Learning Control
10.3 Adaptive Extremum-Seeking Control
Part IV Reduced Order Models
11 Reduced Order Models (ROMs)
11.1 POD for Partial Differential Equations
11.2 Optimal Basis Elements: The POD Expansion
11.3 POD and Soliton Dynamics
11.4 Continuous Formulation of POD
11.5 POD with Symmetries: Rotations and Translations
12 Interpolation for Parametric ROMs
12.1 Gappy POD
12.2 Error and Convergence of Gappy POD
12.3 Gappy Measurements: Minimize Condition Number
12.4 Gappy Measurements: Maximal Variance
12.5 POD and the Discrete Empirical Interpolation Method (DEIM)
12.6 DEIM Algorithm Implementation
12.7 Machine Learning ROMs
Glossary
Bibliography
Index