Contents
Preface
Deep Learning Basics
1. Scope of this textbook
1.1 Data-driven methods
1.2 Physics data analysis
1.3 Machine learning methods
1.4 Deep learning
1.5 Statistical and systematic uncertainties
1.6 Causes for rapid advancements
1.7 Comprehension and hands-on
2. Models for data analysis
2.1 Data in physics research
2.1.1 Data from events
2.1.2 1D-sequences (time series)
2.1.3 Regular grid (image type data)
2.1.4 Regular grid combined with 1D-sequences
2.1.5 Point cloud (3D, spacetime)
2.1.6 Heterogeneous data (ensembles of devices)
2.2 Model building
2.3 Data-driven model optimization
3. Building blocks of neural networks
3.1 Linear mapping and displacement
3.2 Nonlinear mapping: activation function
3.3 Network prediction
3.3.1 Regression
3.3.2 Classification
3.4 Universal approximation theorem
3.5 Exercises
4. Optimization of network parameters
4.1 Preprocessing of input data
4.2 Epoch and batch
4.3 Parameter initialization
4.4 Objective function
4.4.1 Regression: mean squared error (MSE)
4.4.2 Classification: cross-entropy
4.5 Gradients from backpropagation
4.6 Stochastic gradient descent
4.7 Learning rate
4.8 Learning strategies
4.8.1 Adagrad
4.8.2 RMSprob
4.8.3 Momentum
4.8.4 Adam
4.9 Exercises
5. Mastering model building
5.1 Criteria for model building
5.2 Data sets for training, validation, test
5.3 Monitoring
5.4 Regularization
5.5 Hyperparameters and activation function
5.6 Exercises
Standard Architectures of Deep Networks
6. Revisiting the terminology
7. Fully-connected networks: improving the classic all-rounder
7.1 N-layer fully-connected networks
7.2 Regression challenge: two-dimensional location
7.3 Classification challenge: two image categories
7.4 Challenges of training deep networks
7.5 Residual neural networks
7.6 Batch normalization
7.7 Self-normalizing neural networks
7.8 Exercises
8. Convolutional neural networks and analysis of image-like data
8.1 Convolutional neural networks
8.1.1 Convolution of 2D image-like data
8.1.2 Convolutional layers
8.1.3 N-dimensional convolutions
8.1.4 Fundamental operations in CNNs
8.1.5 Learning short- and long-range correlations
8.2 Comparison to fully-connected networks
8.3 Reconstruction tasks
8.4 Advanced concepts
8.4.1 Shortcuts
8.5 Convolutions beyond translational invariance
8.6 Physics applications
8.7 Exercises
9. Recurrent neural networks: time series and variable input
9.1 Sequential relations and network architecture
9.2 Recurrent neural networks
9.3 Training of recurrent networks
9.4 Long short-term memory
9.5 Gated recurrent unit
9.6 Application areas
9.7 Physics applications
9.8 Exercises
10. Graph networks and convolutions beyond Euclidean domains
10.1 Beyond Cartesian data structures
10.2 Graphs
10.2.1 Adjacency matrix and graph basics
10.3 Construction of graphs
10.4 Convolutions in the spatial domain
10.4.1 Graph convolutional networks
10.4.2 Edge convolutions
10.4.3 Dynamic graph update
10.4.4 Deep Sets
10.4.5 Interpretation using the message passing neural network framework
10.5 Convolutions in the spectral domain
10.5.1 Spectral convolutional networks
10.5.1.1 Stable and localized filters
10.6 Exercises
11. Multi-task learning, hybrid architectures, and operational reality
11.1 Multi-task learning
11.2 Combination of network concepts
11.3 Operational reality and verification
11.4 Physics applications
11.5 Exercises
Introspection, Uncertainties, Objectives
12. Interpretability
12.1 Interpretability and deep neural networks
12.2 Model interpretability and feature visualization
12.2.1 Deconvolutional network
12.2.2 Activation maximization
12.3 Interpretability of predictions
12.3.1 Sensitivity analyses and saliency maps
12.3.2 Attribution-based prediction analyses
12.3.2.1 Discriminative localization
12.3.2.2 Layer-wise relevance propagation
12.3.3 Perturbation-based methods
12.4 Exercises
13. Uncertainties and robustness
13.1 Measuring uncertainties
13.2 Decorrelation
13.3 Adversarial attacks
13.4 Exercises
14. Revisiting objective functions
14.1 Fitting function parameters to data
14.2 Reproducing probability distributions
14.3 Conditional probability in supervised learning
14.4 Adaptive objective function
14.5 Hamiltonian gradient objective function
14.6 Energy-based objective function
14.7 Learning without gradients
14.8 Different types of objectives
Deep Learning Advanced Concepts
15. Beyond supervised learning
16. Weakly-supervised classification
16.1 Incomplete supervision
16.2 Inexact supervision
16.3 Inaccurate supervision
16.4 Exercises
17. Autoencoders: finding and compressing structures in data
17.1 Autoencoder networks
17.2 Categories of autoencoders
17.3 Application areas
17.4 Physics applications
17.5 Exercises
18. Generative models: data from noise
18.1 Variational autoencoder
18.2 Generative adversarial networks
18.2.1 Adversarial training
18.2.2 Conditioning
18.2.3 Design of GANs
18.2.4 Challenges in the training of GANs
18.2.4.1 Mode collapsing
18.2.4.2 Vanishing gradients
18.2.4.3 Monitoring convergence
18.2.5 Wasserstein GANs
18.2.6 Spectrally normalized GANs
18.2.7 Additional advances
18.3 Normalizing ows
18.4 Physics applications
18.4.1 Performance evaluation
18.4.2 Example: generation of calorimeter images
18.5 Exercises
19. Domain adaptation, refinement, unfolding
19.1 Refinement for simulations to appear data-like
19.2 Scale factors
19.3 Common representation
19.4 Unfolding
20. Model independent detection of outliers and anomalies
20.1 Basics
20.2 Low background density
20.3 Likelihood-ratio based detection
20.4 Exercises
21. Beyond the scope of this textbook
21.1 Dedicated hardware and microsystems
21.2 Neuroscience inspired developments
21.3 Information field theory
21.4 Human readable physics concepts
Appendix A Notations
Bibliography
Index