Table of Contents
About the Author
About the Technical Reviewer
Introduction
How this book is organized
Chapter 1: MNIST: 1D Neural Network
Dataset overview
Dataset handler
Code: Multilayer perceptron + MNIST
Results
Demo breakdown (high level)
Imports (1)
Model breakdown (2)
Global variables (3)
Training loop: Updates (4)
Training loop: Accuracy (5)
Demo breakdown (low level)
Fully connected neural network layers
How the optimizer works
Optimizers + neural networks
Swift for Tensorflow
Side quests
Recap
Chapter 2: MNIST: 2D Neural Network
Convolutions
3x3 additive blur example
3x3 Gaussian blur example
Combined 3x3 convolutions – Sobel filter example
3x3 striding
Padding
Maxpool
2D MNIST model
Code
Side quest
Recap
Chapter 3: CIFAR: 2D Neural Network with Blocks
CIFAR dataset
Color
Breakdown
Code
Results
Side quest
Recap
Chapter 4: VGG Network
Background: ImageNet
Getting ImageNet
Imagenette dataset
Data augmentation
VGG
Code
Results
Memory usage
Model refactoring
VGG16 with subblocks
Side quests
Recap
Chapter 5: ResNet 34
Skip connections
Noise
Batch normalization
Code
Results
Side quest
Recap
Chapter 6: ResNet 50
Bottleneck blocks
Code
Results
Side Quest: ImageNet
Recap
Chapter 7: SqueezeNet
SqueezeNet
Fire modules
Deep compression
Model pruning
Model quantization
Size metric
Difference between SqueezeNet 1.0 and 1.1
Code
Training loop
Results
Side quest
Recap
Chapter 8: MobileNet v1
MobileNet (v1)
Spatial separable convolutions
Depthwise convolutions
Pointwise convolutions
ReLU 6
Example of the reduction in MACs with this approach
Code
Results
Recap
Chapter 9: MobileNet v2
Inverted residual blocks
Inverted skip connections
Linear bottleneck layers
Code
Results
Recap
Chapter 10: EfficientNet
Swish
SE (Squeeze + Excitation) block
Code
Results
EfficientNet variants
EfficientNet [B1-8]
RandAugment
Noisy Student
EfficientDet
Recap
Chapter 11: MobileNetV3
Hard swish and hard sigmoid
Remove the Squeeze and Excitation (SE) block logic for half the network
Custom head
Hyperparameters
Performance
Code
Results
EfficientNet-EdgeTPU
Recap
Chapter 12: Bag of Tricks
Bag of tricks
What to learn from this
Reading papers
Stay behind the curve
How I read papers
Recap
Chapter 13: MNIST Revisited
Next steps
Pain points
TPU case study
Tensorflow 1 + Pytorch
Enter functional programming
Swift + TPU demo
Results
Recap
Chapter 14: You Are Here
A (short and opinionated) history of computing
History of GPUs
Cloud computing
Crossing the chasm
Computer vision
Direct applications
Indirect applications
Natural language processing
Reinforcement learning and GANs
Simulations in general
To infinity and beyond
Why Swift
Why LLVM
Why MLIR
Why ML is the most important field
Why now
Why you
Appendix A: Cloud Setup
Outline
Google Cloud with CPU instances
How to sign up for Google Cloud
Creating your first few instances
Google Cloud with preconfigured GPU instance
Google Cloud nits
Cattle, not pets
Basic Google Cloud nomenclature
* Machine types
* Buckets
* Billing
Cleaning up
Recap
Appendix B: Hardware Prerequisites, Software Installation Guidelines, and Unix Quickstart
Hardware
Don’t go alone!
GPU
GPUs to buy
Multiple GPUs
CPU
Motherboard
PSU
Cooling
RAM
SSD
Recommendations
Long term
Some real-world usage examples
Hardware recap
Installing Ubuntu
General prep
OS install prep
Download Ubuntu + flash to USB key
OS install
Extra screen
Reboot
Doing a sanity check of your new server
Ubuntu recap
Installing swift for tensorflow
Installing graphics card drivers and swift for tensorflow
CUDA 10.2 install process
Installing cudnn
Installing swift for tensorflow using prebuilt packages
Download swift
Python
Verify you're using a GPU
Autoencoder demo
Reinforcement learning demo
Swift for Tensorflow recap
Installing s4tf from scratch
There be dragons here
How to build swift for tensorflow from scratch
Prerequisites
Installing cmake
Packages we need
Bazel
Fetch swift for tensorflow sources
What a checkout will look like (different hashes)
Python 2 install + packages needed
Build swift for tensorflow from source with GPU support
Running our swift binary
Reset your build artifacts
Installing s4tf from scratch recap
Client setup process + Unix quickstart
Setting up your client computer/crash course in Unix
General config
Configuring your network for remote access
Setting up a VPN (ideal but more complicated)
Setting up port forwarding
Crash course in tmux
Appendix C: Additional Resources
Python --> swift transition guide
Python 3
REPL
Python --> Swift bridge
Python --> C bridge
Python libraries
Self-study guide
Things to study
Python
Swift
iOS/Android
Tensorflow
Pytorch
fast.ai
Cloud computing
TPU
Unix
Git + Unix + etc
Other machine learning compiler–related projects
System monitoring/utilities
Check standard system utilities
Index