Integral and Inverse Reinforcement Learning for Optimal Control Systems and Games (Advances in Industrial Control)

✍ Scribed by Bosen Lian, Wenqian Xue, Frank L. Lewis, Hamidreza Modares, Bahare Kiumarsi

• Publisher: Springer
• Year: 2024
• Tongue: English
• Leaves: 278
• Edition: 2024
• Category: Library

✦ Synopsis

Integral and Inverse Reinforcement Learning for Optimal Control Systems and Games develops its specific learning techniques, motivated by application to autonomous driving and microgrid systems, with breadth and depth: integral reinforcement learning (RL) achieves model-free control without system estimation compared with system identification methods and their inevitable estimation errors; novel inverse RL methods fill a gap that will help them to attract readers interested in finding data-driven model-free solutions for inverse optimization and optimal control, imitation learning and autonomous driving among other areas.

Graduate students will find that this book offers a thorough introduction to integral and inverse RL for feedback control related to optimal regulation and tracking, disturbance rejection, and multiplayer and multiagent systems. For researchers, it provides a combination of theoretical analysis, rigorous algorithms, and a wide-ranging selection of examples. The book equips practitioners working in various domains – aircraft, robotics, power systems, and communication networks among them – with theoretical insights valuable in tackling the real-world challenges they face.

✦ Table of Contents

Series Editor’s Foreword
Preface
Acknowledgements
Contents
Abbreviations and Notation
Abbreviations
Notation
1 Introduction
1.1 Motivation
1.2 Optimal Control
1.3 Integral Reinforcement Learning
1.4 Inverse Optimal Control
1.5 Inverse Reinforcement Learning
1.6 Outline of This Book
References
2 Background on Integral and Inverse Reinforcement Learning for Feedback Control
2.1 Integral Reinforcement Learning for Continuous-Time Systems
2.1.1 Linear Quadratic Regulators
2.1.2 Integral Reinforcement Learning
2.2 Inverse Optimal Control for Continuous-Time Systems
2.2.1 Inverse Optimal Control for Linear Systems
2.2.2 Inverse Optimal Control for Nonlinear Systems
References
Part I Integral Reinforcement Learning for Optimal Control Systems and Games
3 Integral Reinforcement Learning for Optimal Regulation
3.1 Introduction
3.2 On-Policy Synchronous Integral Reinforcement Learning with Experience Replay for Nonlinear Constrained Systems
3.2.1 Problem Formulation
3.2.2 Offline Integral Reinforcement Learning Policy Iteration
3.2.3 Value Function Approximation
3.2.4 Synchronous Online Integral Reinforcement Learning for Nonlinear Constrained Systems
3.2.5 Simulation Examples
3.3 Off-Policy Integral Reinforcement Learning for Linear Quadratic Regulators with Input–Output Data
3.3.1 Discounted Optimal Control Problem
3.3.2 State-Feedback Off-Policy RL with Input-State Data
3.3.3 Output-Feedback Off-Policy RL with Input–Output Data
3.3.4 Simulation Examples
References
4 Integral Reinforcement Learning for Optimal Tracking
4.1 Introduction
4.2 Integral Reinforcement Learning Policy Iteration for Linear …
4.2.1 Problem Formulation
4.2.2 Augmented Algebraic Riccati Equation for Causal Solution
4.2.3 Integral Reinforcement Learning for Online Linear Quadratic Tracking
4.2.4 Simulation Examples
4.3 Online Actor–Critic Integral Reinforcement Learning …
4.3.1 Standard Problem Formulation and Solution
4.3.2 New Formulation for the Optimal Tracking Control Problem of Constrained-Input Systems
4.3.3 Tracking Bellman and Hamilton–Jacobi–Bellman Equations
4.3.4 Offline Policy Iteration Algorithms
4.3.5 Online Actor–Critic-Based Integral Reinforcement Learning
4.4 Simulation Results
References
5 Integral Reinforcement Learning for Zero-Sum Games
5.1 Introduction
5.2 Off-Policy Integral Reinforcement Learning for upper H Subscript normal infinityHinfty Tracking Control
5.2.1 Problem Formulation
5.2.2 Tracking Hamilton–Jacobi–Isaacs Equation and the Solution Stability
5.2.3 Off-Policy Integral Reinforcement Learning for Tracking Hamilton–Jacobi–Isaacs Equation
5.2.4 Simulation Examples
5.3 Off-Policy Integral Reinforcement Learning for Distributed …
5.3.1 Formulation of Distributed Minmax Strategy
5.3.2 Stability and Robustness of Distributed Minmax Strategy
5.3.3 Off-Policy Integral Reinforcement Learning for Distributed Minmax Strategy
5.3.4 Simulation Examples
References
Part II Inverse Reinforcement Learning for Optimal Control Systems and Games
6 Inverse Reinforcement Learning for Optimal Control Systems
6.1 Introduction
6.2 Off-Policy Inverse Reinforcement Learning for Linear Quadratic Regulators
6.2.1 Problem Formulation
6.2.2 Inverse Reinforcement Learning Policy Iteration
6.2.3 Model-Free Off-Policy Inverse Reinforcement Learning
6.2.4 Simulation Examples
6.3 Off-Policy Inverse Reinforcement Learning for Nonlinear …
6.3.1 Problem Formulation
6.3.2 Model-Based Inverse Reinforcement Learning
6.3.3 Model-Free Off-Policy Integral Inverse Reinforcement Learning
6.3.4 Simulation Examples
References
7 Inverse Reinforcement Learning for Two-Player Zero-Sum
Games
7.1 Introduction
7.2 Inverse Q-Learning for Linear Two-Player Zero-Sum Games
7.2.1 Problem Formulation
7.2.2 Model-Free Inverse Q-Learning
7.2.3 Implementation of Inverse Q-Learning Algorithm
7.2.4 Simulation Examples
7.3 Off-Policy Inverse Reinforcement Learning for Nonlinear Two-Player Zero-Sum Games
7.3.1 Problem Formulation
7.3.2 Inverse Reinforcement Learning Policy Iteration
7.3.3 Model-Free Off-Policy Integral Inverse Reinforcement
Learning
7.3.4 Simulation Examples
7.4 Online Adaptive Inverse Reinforcement Learning for Nonlinear Two-Player Zero-Sum Games
7.4.1 Integral RL-Based Off line Inverse Reinforcement Learning
7.4.2 Online Inverse Reinforcement Learning with Synchronous Neural Networks
7.4.3 Simulation Examples
References
8 Inverse Reinforcement Learning for Multiplayer Non-Zero-Sum Games
8.1 Introduction
8.2 Off-Policy Inverse Reinforcement Learning for Linear Multiplayer Non-Zero-Sum Games
8.2.1 Problem Formulation
8.2.2 Inverse Reinforcement Learning Policy Iteration
8.2.3 Model-Free Off-Policy Integral Inverse Reinforcement Learning
8.2.4 Simulation Examples
8.3 Off-Policy Inverse Reinforcement Learning for Nonlinear Multiplayer Non-Zero-Sum Games
8.3.1 Problem Formulation
8.3.2 Inverse Reinforcement Learning Policy Iteration
8.3.3 Model-Free Off-Policy Integral Inverse Reinforcement Learning
8.3.4 Simulation Examples
References
Appendix A Some Useful Facts in Matrix Algebra
Index