Enhancing Deep Learning with Bayesian Inference: Create more powerful, robust deep learning systems with Bayesian deep learning in Python

✍ Scribed by Matt Benatan, Jochem Gietema, Marian Schneider

• Publisher: Packt Publishing
• Tongue: English
• Leaves: 386
• Category: Library

✦ Synopsis

Develop Bayesian Deep Learning models to help make your own applications more robust.

Key Features

• Gain insights into the limitations of typical neural networks
• Acquire the skill to cultivate neural networks capable of estimating uncertainty
• Discover how to leverage uncertainty to develop more robust machine learning systems

Book Description

Deep learning is revolutionizing our lives, impacting content recommendations and playing a key role in mission- and safety-critical applications. Yet, typical deep learning methods lack awareness about uncertainty. Bayesian deep learning offers solutions based on approximate Bayesian inference, enhancing the robustness of deep learning systems by indicating how confident they are in their predictions. This book will guide you in incorporating model predictions within your applications with care.

Starting with an introduction to the rapidly growing field of uncertainty-aware deep learning, you'll discover the importance of uncertainty estimation in robust machine learning systems. You'll then explore a variety of popular Bayesian deep learning methods and understand how to implement them through practical Python examples covering a range of application scenarios.

By the end of this book, you'll embrace the power of Bayesian deep learning and unlock a new level of confidence in your models for safer, more robust deep learning systems.

What you will learn

• Discern the advantages and disadvantages of Bayesian inference and deep learning
• Become well-versed with the fundamentals of Bayesian Neural Networks
• Understand the differences between key BNN implementations and approximations
• Recognize the merits of probabilistic DNNs in production contexts
• Master the implementation of a variety of BDL methods in Python code
• Apply BDL methods to real-world problems
• Evaluate BDL methods and choose the most suitable approach for a given task
• Develop proficiency in dealing with unexpected data in deep learning applications

Who this book is for

This book will cater to researchers and developers looking for ways to develop more robust deep learning models through probabilistic deep learning. You're expected to have a solid understanding of the fundamentals of machine learning and probability, along with prior experience working with machine learning and deep learning models.

Table of Contents

1. Bayesian Inference in the Age of Deep Learning
2. Fundamentals of Bayesian Inference
3. Fundamentals of Deep Learning
4. Introducing Bayesian Deep Learning
5. Principled Approaches for Bayesian Deep Learning
6. Using the Standard Toolbox for Bayesian Deep Learning
7. Practical considerations for Bayesian Deep Learning
8. Applying Bayesian Deep Learning
9. Next Steps in Bayesian Deep Learning

✦ Table of Contents

Cover
Title page
Copyright and Credits
Contributors
Table of Contents
Preface
Chapter 1: Bayesian Inference in the Age of Deep Learning
Technical requirements
Wonders of the deep learning age
Understanding the limitations of deep learning
Bias in deep learning systems
The danger of over-confident predictions
Shifting trends
Core topics
Setting up the work environment
Summary
Chapter 2: Fundamentals of Bayesian Inference
Refreshing our knowledge of Bayesian modeling
Bayesian inference via sampling
Approximating distributions
Implementing probabilistic inference with Bayesian linear regression
Exploring the Gaussian process
Defining our prior beliefs with kernels
Limitations of Gaussian processes
Summary
Further reading
Chapter 3: Fundamentals of Deep Learning
Technical requirements
Introducing the multi-layer perceptron
Reviewing neural network architectures
Exploring CNNs
Exploring RNNs
Attention mechanisms
Understanding the problem with typical neural networks
Uncalibrated and overconfident predictions
Predictions on out-of-distribution data
Example of confident, out-of-distribution predictions
Susceptibility to adversarial manipulations
Summary
Further reading
Chapter 4: Introducing Bayesian Deep Learning
Technical requirements
The ideal BNN
BDL fundamentals
Gaussian assumptions
Sources of uncertainty
Separating sourcing of uncertainty
Going beyond maximum likelihood: the importance of likelihoods
Tools for BDL
Summary
Further reading
Chapter 5: Principled Approaches for Bayesian Deep Learning
Technical requirements
Explaining notation
Familiar probabilistic concepts from deep learning
Bayesian inference by backpropagation
Implementing BBB with TensorFlow
Step 1: Importing packages
Step 2: Acquiring data
Step 3: Helper functions
Step 4: model training
Step 5: inference
Scalable Bayesian Deep Learning with Probabilistic Backpropagation
Implementing PBP
Step 1: Importing libraries
Step 2: Helper functions
Step 3: Data preparation
Step 4: Defining our model class
Step 5: Avoiding numerical errors
Step 6: Instantiating our model
Step 7: Using our model for inference
Summary
Further reading
Chapter 6: Using the Standard Toolbox for Bayesian Deep Learning
Technical requirements
Introducing approximate Bayesian inference via dropout
Using dropout for approximate Bayesian inference
Implementing MC dropout
Using ensembles for model uncertainty estimates
Introducing ensembling methods
Introducing deep ensembles
Implementing a deep ensemble
Step 1: Importing libraries
Step 2: Obtaining data
Step 3: Constructing our ensemble model
Step 4: Training
Step 5: Inference
Practical limitations of deep ensembles
Exploring neural network augmentation with Bayesian last-layer methods
Last-layer methods for Bayesian inference
Step 1: Creating and training our base model
Step 2: Using a neural network layer as a basis function
Step 3: Preparing our variables for Bayesian linear regression
Step 4: Connecting our basis function model
Step 5: Creating a method to fit our Bayesian linear regression parameters
Step 6: Inference
Last-layer MC dropout
Step 1: Connecting to our base model
Step 2: Adding an MC dropout layer
Step 3: Training the MC dropout last-layer
Step 4: Obtaining uncertainties
Step 5: Inference
Recap of last-layer methods
Summary
Chapter 7: Practical Considerations for Bayesian Deep Learning
Technical requirements
Balancing uncertainty quality and computational considerations
Setting up our experiments
Importing our dependencies
Preparing our data and models
Running our experiments
Analyzing model performance
Computational considerations of Bayesian deep learning models
Choosing the right model
BDL and sources of uncertainty
Step 1: Setting up the environment
Step 2: Fitting a model without uncertainty
Step 3: Fitting a model with aleatoric uncertainty
Step 4: Fitting a model with epistemic uncertainty
Step 5: Fitting a model with aleatoric and epistemic uncertainty
Sources of uncertainty: Image classification case study
Summary
Further reading
Chapter 8: Applying Bayesian Deep Learning
Technical requirements
Detecting out-of-distribution data
Exploring the problem of out-of-distribution detection
Systematically evaluating OOD detection performance
Simple out-of-distribution detection without retraining
ODIN: OOD detection with preprocessing and scaling
Mahalanobis: OOD Detection with intermediate features
Being robust against dataset shift
Measuring a model's response to dataset shift
Revealing dataset shift with Bayesian methods
Step 1: Preparing the environment
Step 2: Defining and training the models
Step 3: Obtaining predictions
Step 4: Measuring accuracy
Step 5: Measuring calibration
Using data selection via uncertainty to keep models fresh
Step 1: Preparing our dataset
Step 2: Setting up our configuration
Step 3: Defining the model
Step 4: Defining the uncertainty functions
Step 5: Defining the loop
Step 6: Inspecting the results
Using uncertainty estimates for smarter reinforcement learning
Step 1: Initializing our environment
Step 2: Updating the state of our environment
Step 3: Defining our model
Step 4: Running our reinforcement learning
Navigating obstacles with uncertainty
Step 1: Introducing obstacles
Step 2: Placing our dynamic obstacle
Step 3: Adding sensing
Step 4: Modifying our reward function
Step 5: Initializing our uncertainty-aware model
Step 6: Fitting our MC dropout network
Step 7: Making predictions
Step 8: Adapting our standard model
Step 9: Preparing to run our new reinforcement learning experiment
Step 10: Running our BDL reinforcement experiment
Susceptibility to adversarial input
Step 1: Model training
Step 2: Running inference and evaluating our standard model
Summary
Further reading
Chapter 9: Next Steps in Bayesian Deep Learning
Current trends in BDL
How are BDL methods being applied to solve real-world problems?
Latest methods in BDL
Combining MC dropout and deep ensembles
Improving deep ensembles by promoting diversity
Uncertainty in very large networks
Alternatives to Bayesian deep learning
Scalable Gaussian processes
Deep Gaussian processes
Your next steps in BDL
Further reading
Index
Other Books You Might Enjoy