Foundations of Deep Reinforcement Learning: Theory and Practice in Python

✍ Scribed by Laura Graesser; Wah Loon Keng

• Publisher: Addison-Wesley Professional
• Year: 2019
• Tongue: English
• Leaves: 656
• Series: Addison-Wesley Data & Analytics Series
• Category: Library

✦ Synopsis

The Contemporary Introduction to Deep Reinforcement Learning that Combines Theory and Practice

Deep reinforcement learning (deep RL) combines deep learning and reinforcement learning, in which artificial agents learn to solve sequential decision-making problems. In the past decade deep RL has achieved remarkable results on a range of problems, from single and multiplayer games—such as Go, Atari games, and DotA 2—to robotics.

Foundations of Deep Reinforcement Learning is an introduction to deep RL that uniquely combines both theory and implementation. It starts with intuition, then carefully explains the theory of deep RL algorithms, discusses implementations in its companion software library SLM Lab, and finishes with the practical details of getting deep RL to work.
This guide is ideal for both computer science students and software engineers who are familiar with basic machine learning concepts and have a working understanding of Python.

• Understand each key aspect of a deep RL problem
• Explore policy- and value-based algorithms, including REINFORCE, SARSA, DQN, Double DQN, and Prioritized Experience Replay (PER)
• Delve into combined algorithms, including Actor-Critic and Proximal Policy Optimization (PPO)
• Understand how algorithms can be parallelized synchronously and asynchronously
• Run algorithms in SLM Lab and learn the practical implementation details for getting deep RL to work
• Explore algorithm benchmark results with tuned hyperparameters
• Understand how deep RL environments are designed

Register your book for convenient access to downloads, updates, and/or corrections as they become available. See inside book for details.

✦ Table of Contents

Title Page
Contents
Preface
Acknowledgements
About the Authors
Chapter 1. Introduction
1.1 Reinforcement Learning
1.2 Reinforcement Learning as MDP
1.3 Learnable Functions in Reinforcement Learning
1.4 Deep Reinforcement Learning Algorithms
1.5 Deep Learning for Reinforcement Learning
1.6 Reinforcement Learning and Supervised Learning
1.7 Summary
Part I: Policy-based & Value-based Algorithms
Chapter 2. Reinforce
2.1 Policy
2.2 The Objective Function
2.3 The Policy Gradient
2.4 Monte Carlo Sampling
2.5 REINFORCE Algorithm
2.6 Implementing REINFORCE
2.7 Training a REINFORCE Agent
2.8 Experimental Results
2.9 Summary
2.10 Further Reading
2.11 History
Chapter 3. SARSA
3.1 The Q and V Functions
3.2 Temporal Difference Learning
3.3 Action Selection in SARSA
3.4 SARSA Algorithm
3.5 Implementing SARSA
3.6 Training a SARSA Agent
3.7 Experimental Results
3.8 Summary
3.9 Further Reading
3.10 History
Chapter 4. Deep Q-Networks (DQN)
4.1 Learning the Q-function in DQN
4.2 Action Selection in DQN
4.3 Experience Replay
4.4 DQN Algorithm
4.5 Implementing DQN
4.6 Training a DQN Agent
4.7 Experimental Results
4.8 Summary
4.9 Further Reading
4.10 History
Chapter 5. Improving DQN
5.1 Target Networks
5.2 Double DQN
5.3 Prioritized Experience Replay (PER)
5.4 Modified DQN Implementation
5.5 Training a DQN Agent to Play Atari Games
5.6 Experimental Results
5.7 Summary
5.8 Further Reading
Part II: Combined methods
Chapter 6. Advantage Actor-Critic (A2C)
6.1 The Actor
6.2 The Critic
6.3 A2C Algorithm
6.4 Implementing A2C
6.5 Network Architecture
6.6 Training an A2C Agent
6.7 Experimental Results
6.8 Summary
6.9 Further Reading
6.10 History
Chapter 7. Proximal Policy Optimization (PPO)
7.1 Surrogate Objective
7.2 Proximal Policy Optimization (PPO)
7.3 PPO Algorithm
7.4 Implementing PPO
7.5 Training a PPO Agent
7.6 Experimental Results
7.7 Summary
7.8 Further Reading
Chapter 8. Parallelization Methods
8.1 Synchronous Parallelization
8.2 Asynchronous Parallelization
8.3 Training an A3C Agent
8.4 Summary
8.5 Further Reading
Chapter 9. Algorithm Summary
Part III: Practical Tips
Chapter 10. Getting Deep RL to Work
10.1 Software Engineering Practices
10.2 Debugging Tips
10.3 Atari Tricks
10.4 Deep RL Almanac
10.5 Summary
Chapter 11. SLM Lab
11.1 Implemented Algorithms in SLM Lab
11.2 Spec File
11.3 Running SLM Lab
11.4 Analyzing Experiment Results
11.5 Summary
Chapter 12. Network architectures
12.1 Types of Neural Network
12.2 Guidelines For Choosing a Network Family
12.3 The Net API
12.4 Summary
12.5 Further Reading
Chapter 13. Hardware
13.1 Computer
13.2 Information In Hardware
13.3 Choosing Hardware
13.4 Summary
Chapter 14. Environment Design
14.1 States
14.2 Actions
14.3 Rewards
14.4 Transition Function
14.5 Summary
14.6 Further Reading: Action Design in Everyday Things
Epilogue
Appendix A. Deep Reinforcement Learning Timeline
Appendix B. Example Environments
B.1 Discrete Environments
B.2 Continuous Environments
Bibliography