CHAPTER 1 ◾ Installing Anaconda and Jupyter Notebook
1.1 Why Python for Machine Learning?
1.1.1 The Rise of Python
1.1.2 Python for Machine Learning
1.2 Installing Anaconda
1.2.1 Installing Anaconda in Windows
1.2.2 Installing Anaconda in macOS
1.2.3 Installing Anaconda in Linux
1.2.4 Difference between Conda-install and Pip-install
1.3 Virtual Environment for this Book
1.3.1 Create the Virtual Environment MLA
1.3.2 Activate the Virtual Environment
1.3.3 De-activate the Virtual Environment
1.4 Set Up Jupyter Notebook in the Virtual Environment
1.4.1 Write Python in Jupyter Notebook
1.4.2 Issue Commands in Jupyter Notebook
1.5 File System for the Book
1.6 Glossary
1.7 Exercises
CHAPTER 2 ◾ Creating Animations
2.1 Create Plots with Matplotlib
2.1.1 A Single Line Plot
2.1.2 Multiple Lines in the Same Plot
2.2 Create Subplots
2.2.1 Create Individual Plots
2.2.2 Create Subplots
2.3 Create Animated Plots
2.3.1 Generate Annual Line Plots
2.3.2 Animate the Plots
2.4 Create Animated Bar Charts
2.4.1 Create a Horizontal Bar Chart
2.4.2 Generate Annual Bar Charts
2.4.3 Animate the Bar Charts
2.5 Put Bar Charts and Plots Side by Side
2.5.1 Combine a Bar Chart and a Plot
2.5.2 Create an Animation of the Combined Pictures
2.6 Animated Pie Charts
2.6.1 Create a Pie Chart
2.6.2 Generate Annual Pie Charts
2.6.3 Animate the Combined Pie Charts and Plots
2.7 Appendix: Download and Clean Up the GDP Data
2.8 Glossary
2.9 Exercises
SECTION II Machine Learning Basics
CHAPTER 3 ◾ Machine Learning: An Overview
3.1 ML: A New Paradigm for AI
3.1.1 What is AI?
3.1.2 Rule-Based AI
3.1.3 What is ML? Why ML Matters?
3.1.4 Why is ML so Popular?
3.2 Different Types of ML
3.2.1 Supervised Learning
3.2.2 Unsupervised Learning
3.2.3 Reinforcement Learning
3.3 Deep Reinforcement Learning
3.3.1 Deep Learning
3.3.2 Combine Deep Learning and Reinforcement Learning
3.4 Apply ML in the Real World
3.4.1 The Delivery Route Problem
3.4.2 Try the Problem before You Know the Answer
3.5 Glossary
3.6 Exercises
CHAPTER 4 ◾ Gradient Descent – Where Magic Happens
4.1 Optimization through Grid Search
4.1.1 How Grid Search Achieves Optimization
4.1.2 Curse of Dimensionality and Directional Grid Search
4.2 Gradient Descent
4.3 Use Tensorflow to Calculate Gradients
4.3.1 Install TensorFlow
4.3.2 Calculate Gradients Using TensorFlow
4.3.3 Gradient Descent Optimization with TensorFlow
4.3.4 Animate the Optimization Process
4.4 Choose the Right Learning Rate
4.4.1 When the Learning Rate is Too Large
4.4.2 When the Learning Rate is Too Small
4.5 Compare Learning Rates
4.5.1 Combine Animations
4.5.2 Subplots of Different Stages
4.6 Glossary
4.7 Exercises
CHAPTER 5 ◾ Introduction to Neural Networks
5.1 Anatomy of A Neural Network
5.1.1 Elements of a Neural Network
5.1.2 How Does a Neural Network Learn?
5.1.3 Make Predictions
5.2 Animate the Learning Process
5.2.1 Generate Graphs
5.2.2 Create Animation Based on Graphs
5.2.3 Subplots of Different Stages
5.3 Create A Neural Network with Keras
5.3.1 Construct the Model
5.3.2 Compile and Train the Model
5.3.3 Make Predictions
5.4 Customize Training with GradientTape
5.4.1 Construct the Model
5.4.2 Train the Model
5.4.3 Make Predictions
5.5 Glossary
5.6 Exercises
CHAPTER 6 ◾ Activation Functions
6.1 Why Do We Need Activation Functions?
6.1.1 Construct a Neural Network
6.1.2 Learn a Nonlinear Relation without Activation
6.2 The ReLU Activation Function
6.2.1 What is ReLU?
6.2.2 Animate the ReLU Function
6.2.3 Use ReLU to Model Nonlinearity
6.3 The Sigmoid Activation Function
6.3.1 Plot the Sigmoid Function
6.3.2 Animate the Sigmoid Function
6.3.3 Combine Animations
6.3.4 A Picture with Subplots of Different Stages
6.4 The Softmax Activation Function
6.4.1 What is the Softmax Function?
6.4.2 A Diagram of the Softmax Function
6.5 Glossary
6.6 Exercises
SECTION III Binary and Multi-Category Classifications
CHAPTER 7 ◾ Binary Classifications
7.1 What Is A Binary Classification Problem
7.1.1 Sigmoid Activation in Binary Classifications
7.1.2 The Binary Cross-Entropy Loss Function
7.2 Process Image Data
7.2.1 Download Data
7.2.2 Convert NumPy Arrays to Pictures and Back
7.2.3 Match Pictures with Labels
7.3 Binary Classification with A Logit Regression
7.3.1 Prepare the Data
7.3.2 Train the Logit Model
7.3.3 Predict Using the Logit Model
7.4 Binary Classification with A Simple Neural Network
7.4.1 Train and Test Using a Neural Network
7.4.2 Focus on Two Examples
7.4.3 Diagrams of the Network and Predictions
7.4.4 Animate the Training Process
7.4.5 Animate the Predictions for the Deer
7.5 Combine the Animations
7.5.1 Animate the Two Predictions
7.5.2 Subplots
7.6 Binary Classification with A Deep Neural Network
7.7 Appendix: Load CIFAR10 from Tensorflow Directly
7.8 Glossary
7.9 Exercises
CHAPTER 8 ◾ Convolutional Neural Networks
8.1 What Are Convolutional Neural Networks (CNNs)?
8.1.1 Our Running Example
8.1.2 A Horizontal Filter
8.2 Convolution Operations
8.2.1 Calculations in a Convolution Operation
8.2.2 Animate the Convolution Operations
8.2.3 Subplots
8.3 Stride and Padding
8.3.1 A Filter without Padding and a Stride of 2
8.3.2 Animate the Diagonal Filter
8.3.3 Animate the Diagonal Filter Convolution Operation
8.3.4 Subplots for Strides
8.4 Combine the Two Animations
8.4.1 Combine the Animations
8.5 Max Pooling
8.6 Binary Classifications with Convolutional Layers
8.7 Glossary
8.8 Exercises
CHAPTER 9 ◾ Multi-Category Image Classifications
9.1 Image Augmentations
9.1.1 The Keras Image Generator
9.1.2 Visualize Image Augmentations
9.2 What Is Multi-Category Classification?
9.2.1 One-Hot Encoder for Labels
9.2.2 The Activation and Loss Functions
9.3 Train the Multi-Category Classification Model
9.3.1 Load the Full Data Set
9.3.2 Convert Labels to One-Hot Variables
9.3.3 Train the Model
9.3.4 Evaluate the Model
9.4 Animate the Learning Process
9.4.1 Select Example Pictures
9.4.2 Animate Prediction Changes
9.4.3 Subplots of the Predictions on the Truck Image
9.4.4 Animate Predictions on the Frog Image
9.4.5 Subplots of the Predictions on the Frog Image
9.4.6 Combine the Animations
9.5 Glossary
9.6 Exercises
SECTION IV Developing Deep Learning Game Strategies
CHAPTER 10 ◾ Deep Learning Game Strategies
10.1 Get Started with the OpenAI Gym Environment
10.1.1 Basic Elements of a Game Environment
10.1.2 The Frozen Lake Game
10.1.3 Play the Frozen Lake Game Manually
10.2 Deep Learning Game Strategies: Generating Data
10.2.1 Summary of the Game Strategy
10.2.2 Simulate One Game
10.2.3 Simulate Many Games
10.3 Train the Deep Neural Network
10.3.1 Preprocess the Data
10.3.2 Train Deep Learning Game Strategies
10.4 Play Games with the Trained Model
10.4.1 Test One Game
10.4.2 Test the Efficacy of the Game Strategy
10.5 Animate the Decision-Making Process
10.5.1 Generate Figures
10.5.2 Create the Animation
10.5.3 Create a Figure with Subplots
10.6 Glossary
10.7 Exercises
CHAPTER 11 ◾ Deep Learning in the Cart Pole Game
11.1 Play the Cart Pole Game in OpenAI Gym
11.1.1 Features of the Cart Pole Game
11.1.2 Play a Full Game
11.2 Generate Data to Train the Model
11.2.1 How to Define Winning and Losing?
11.2.2 Prepare Data for the Neural Network
11.3 Train the Deep Neural Network
11.3.1 Preprocess the Data
11.3.2 Train the Deep Neural Network with Data
11.4 Play the Game with the Trained Model
11.4.1 A best_move() Function
11.4.2 Play One Cart Pole Game with the Trained Model
11.5 Compare Two Games
11.5.1 Record a Game with Random Moves
11.5.2 Combine Frames
11.5.3 Subplots of the Cart Pole Game Stages
11.6 Glossary
11.7 Exercises
CHAPTER 12 ◾ Deep Learning in Multi-Player Games
12.1 Create the Tic Tac Toe Game Environment
12.1.1 Use a Python Class to Represent the Environment
12.1.2 Create a Local Module for the Tic Tac Toe Game
12.1.3 Verify the Custom-Made Game Environment
12.1.4 Play a Game in the Tic Tac Toe Environment
12.2 Train A Deep Learning Game Strategy
12.2.1 A Blueprint of the Deep Learning Game Strategy
12.2.2 Simulate Tic Tac Toe Games
12.2.3 Train Your Tic Tac Toe Game Strategy
12.3 Use the Trained Model to Play Games
12.3.1 Best Moves Based on the Trained Model
12.3.2 Test a Game Using the Trained Model
12.3.3 Test the Efficacy of the Trained Model
12.4 Animate the Deep Learning Process
12.4.1 Probabilities of Winning for Each Hypothetical Move
12.4.2 Animate the Whole Game
12.4.3 Animate the Decision Making
12.4.4 Animate Board Positions and the Decision Making
12.4.5 Subplots of the Decision-Making Process
12.5 Glossary
12.6 Exercises
CHAPTER 13 ◾ Deep Learning in Connect Four
13.1 Create A Connect Four Game Environment
13.1.1 A Connect Four Game Environment
13.1.2 Verify the Connect Four Game Environment
13.1.3 Play a Connect Four Game
13.2 Train A Deep Neural Network
13.2.1 The Game Plan
13.2.2 Simulate Connect Four Games
13.2.3 Train the Connect Four Game Strategy
13.3 Use the Trained Model to Play Connect Four
13.3.1 Best Moves
13.3.2 Test Connect Four Deep Learning Game Strategies
13.4 Animate Deep Learning in Connect Four
13.4.1 Print Out Probabilities of Winning for Each Next Move
13.4.2 Animate a Complete Connect Four Game
13.4.3 Animate the Decision-Making Process
13.4.4 Combine Board Positions and Decision Making
13.4.5 Create Subplots of Deep Learning
13.5 Glossary
13.6 Exercises
SECTION V Reinforcement Learning
CHAPTER 14 ◾ Introduction to Reinforcement Learning
14.1 Basics of Reinforcement Learning
14.1.1 Basic Concepts
14.1.2 The Bellman Equation and Q-Learning
14.2 Use Q-Values to Play the Frozen Lake Game
14.2.1 The Logic Behind Q-Learning
14.2.2 A Q-Table to Win the Frozen Lake Game
14.3 Train the Q-Values
14.3.1 What is Q-Learning?
14.3.2 Let the Learning Begin
14.4 Q-Learning in A Self-Made Game Environment
14.4.1 A Self-Made Frozen Lake Game Environment
14.4.2 Use the Q-Table in the Self-Made Game Environment
14.5 Animate the Q-Learning Process
14.5.1 Highlight Values and Actions in the Q-Table
14.5.2 Animate the Use of the Q-Table
14.5.3 Game Board Positions and Best Actions
14.5.4 Subplots of the Q-Learning Process
14.6 Glossary
14.7 Exercises
CHAPTER 15 ◾ Q-Learning with Continuous States
15.1 The Mountain Car Game Environment
15.1.1 The Mountain Car Game
15.1.2 Convert a Continuous State into Discrete Values
15.1.3 The Reward Structure of the Game
15.2 Q-Learning in the Mountain Car Game
15.2.1 How to Train the Q-Table
15.2.2 Update the Q-Table
15.2.3 Train the Q-Table via Trial and Error
15.3 Test the Trained Q-Table
15.3.1 Define the Test_Q() Function
15.3.2 The Effectiveness of the Trained Q-Table
15.4 Animate the Game Before and After Q-Learning
15.4.1 The Mountain Car Game without Q-Learning
15.4.2 The Mountain Car Game with Q-Learning
15.4.3 The Mountain Car Game with and without Q-Learning
15.5 Glossary
15.6 Exercises
CHAPTER 16 ◾ Solving Real-World Problems with Machine Learning
16.1 Create A Delivery Route Game Environment
16.1.1 Draw Delivery Routes
16.1.2 Create a Game Environment
16.1.3 Use the Delivery Route Game Environment
16.2 Train A Q-Table between Any Two Positions
16.2.1 Create and Train A Q-table
16.2.2 Test the Trained Tabular Q-Values
16.3 Train the Q-Table for All Possible Routes
16.3.1 Train the Large Q-Table
16.3.2 Test the Large Q-Table
16.4 The Shortest Delivery Route to Eight Households
16.4.1 Find All Possible Permutations in Python
16.4.2 The Total Distance to Deliver to Eight Households
16.4.3 The Shortest Route
16.5 Animate the Delivery Route
16.5.1 Create a Graph at Each Stop
16.5.2 Animate the Shortest Route
16.5.3 Subplots of the Eight Deliveries
16.6 Glossary
16.7 Exercises
SECTION VI Deep Reinforcement Learning
CHAPTER 17 ◾ Deep Q-Learning
17.1 Deep Q-Learning for the Cart Pole Game
17.1.1 Create a Deep Q-Network
17.1.2 Train the Deep Q-Network
17.2 Test the Trained Deep Q-Network
17.2.1 Test and Record One Game
17.2.2 Test the Efficacy of the Deep Q-Network
17.3 Animate Deep Q-Learning
17.3.1 Draw the Current Game State and Q-Values
17.3.2 Create A Graph for Each Time Step
17.4 An Animation and A Picture with Subplots
17.5 Glossary
17.6 Exercises
CHAPTER 18 ◾ Policy-Based Deep Reinforcement Learning
18.1 Policy-Based Reinforcement Learning
18.1.1 What is a Policy?
18.1.2 What is the Policy Gradient Method?
18.2 Get Started with Atari Games
18.2.1 The Pong Game
18.2.2 Preprocess the Game Pictures
18.2.3 Use the Difference of Game Windows
18.3 Train the Policy Gradient Agent
18.3.1 Create a Policy Network
18.3.2 Train the Model
18.4 Test the Policy Gradient Agent
18.5 Animate the Pong Games
18.5.1 Record Games with Random Moves
18.5.2 Combine the Animations
18.5.3 Subplots of the Policy Gradient Agent
18.6 Glossary
18.7 Exercises
CHAPTER 19 ◾ The Policy Gradient Method in Breakout
19.1 Get Started with the Breakout Game
19.1.1 The Breakout Game
19.1.2 Preprocess the Game Frames
19.1.3 Obtain the Difference of Two Game Windows
19.2 Train the Policy Gradient Model in Breakout
19.2.1 Changes Needed
19.2.2 Create a Policy Network
19.2.3 Train the Policy Gradient Agent in Breakout
19.3 Test the Policy Gradient Agent in Breakout
19.3.1 Test the Trained Policy Gradient Agent
19.3.2 Search for Successful Episodes
19.4 Zero in on Interesting Time Steps
19.4.1 Animate Interesting Time Steps
19.4.2 Subplots of the Interesting Time Steps
19.5 Exercises
CHAPTER 20 ◾ Double Deep Q-Learning
20.1 Get Started with OpenAI Baselines
20.1.1 The Breakout Game with OpenAI Baselines
20.1.2 Preprocessed Frames from Baselines
20.1.3 Subplots of Preprocessed Frames
20.2 Train the Double Deep Q Agent
20.2.1 Create a Double Deep Q-Network
20.2.2 Train the Deep Q Network
20.3 Test the Trained Breakout Agent
20.3.1 Testing One Original Episode
20.3.2 Play Multiple Games and Test the Average Score
20.4 Animate Interesting Time Steps
20.4.1 Collect a Successful Episode
20.4.2 A Picture with Subplots
20.5 Glossary
20.6 Exercises
CHAPTER 21 ◾ Space Invaders with Double Deep Q-Learning
21.1 Getting Started with Space Invaders
21.1.1 Space Invaders in OpenAI Gym
21.1.2 Space Invaders with the Baselines Game Wrapper
21.1.3 Preprocessed Space Invaders Game Windows
21.2 Train the Double Deep Q-Network
21.2.1 The Same Double Deep Q-Network
21.2.2 The Same Training Process
21.3 Test the Trained Agent in Space Invaders
21.3.1 Testing One Full Original Episode
21.3.2 Average Performance of the Trained Model
21.4 Animate Space Invaders
21.4.1 Collect Space Invaders Episodes
21.4.2 Zero in on the Interesting Time Steps
21.4.3 Subplots of Space Invaders
21.5 Exercises
CHAPTER 22 ◾ Scaling Up Double Deep Q-Learning
22.1 Get Started with the Seaquest Game
22.1.1 The Seaquest Game in OpenAI Gym
22.1.2 Seaquest with the Baselines Game Wrapper
22.1.3 Preprocessed Seaquest Game Windows
22.1.4 Subplots of Seaquest Game Windows
22.2 Get Started with Beam Rider
22.2.1 Beam Rider without the Game Wrapper
22.2.2 Beam Rider with the Baselines Game Wrapper
22.2.3 Preprocessed Beam Rider Game Windows
22.2.4 Subplots of Beam Rider Game Windows
22.3 Scaling Up the Double Deep Q-Network
22.3.1 Differences among Atari Games
22.3.2 A Generic Double Deep Q-Network
22.3.3 The Training Process for any Atari Game
22.4 Try It on Seaquest
22.4.1 Train the Model in Seaquest
22.4.2 Test the Average Score in Seaquest
22.4.3 Animate a Successful Episode
22.5 Try It on Beam Rider
22.5.1 Train the Model in Beam Rider
22.5.2 The Average Score in Beam Rider
22.5.3 A Successful Episode in Beam Rider
22.6 Exercises
Bibliography
Index