Relative Optimization of Continuous-Time and Continuous-State Stochastic Systems

✍ Scribed by Xi-Ren Cao

• Publisher: Springer
• Year: 2020
• Tongue: English
• Leaves: 376
• Series: Communications and Control Engineering
• Edition: 1st ed. 2020
• Category: Library

✦ Synopsis

This monograph applies the relative optimization approach to time nonhomogeneous continuous-time and continuous-state dynamic systems. The approach is intuitively clear and does not require deep knowledge of the mathematics of partial differential equations. The topics covered have the following distinguishing features: long-run average with no under-selectivity, non-smooth value functions with no viscosity solutions, diffusion processes with degenerate points, multi-class optimization with state classification, and optimization with no dynamic programming.

The book begins with an introduction to relative optimization, including a comparison with the traditional approach of dynamic programming. The text then studies the Markov process, focusing on infinite-horizon optimization problems, and moves on to discuss optimal control of diffusion processes with semi-smooth value functions and degenerate points, and optimization of multi-dimensional diffusion processes. The book concludes with a brief overview of performance derivative-based optimization.

Among the more important novel considerations presented are:

• the extension of the Hamilton–Jacobi–Bellman optimality condition from smooth to semi-smooth value functions by derivation of explicit optimality conditions at semi-smooth points and application of this result to degenerate and reflected processes;
• proof of semi-smoothness of the value function at degenerate points;
• attention to the under-selectivity issue for the long-run average and bias optimality; 
• discussion of state classification for time nonhomogeneous continuous processes and multi-class optimization; and
• development of the multi-dimensional Tanaka formula for semi-smooth functions and application of this formula to stochastic control of multi-dimensional systems with degenerate points.

The book will be of interest to researchers and students in the field of stochastic control and performance optimization alike.

✦ Table of Contents

Preface
Acknowledgements
Contents
Notation and Terminology
1 Introduction
1.1 A Brief Introduction to Relative Optimization
1.2 Relative Optimization Versus Dynamic Programming
1.2.1 The Optimization Problem
1.2.2 Dynamic Programming
1.2.3 Relative Optimization
1.2.4 Comparison of Two Approaches
1.2.5 Intuitive Explanations
1.3 Main Results of the Book
1.4 Philosophical and Historical Remarks
1.4.1 A Philosophical View on Relative Optimization
1.4.2 History of Development
References
2 Optimal Control of Markov Processes: Infinite-Horizon
2.1 Introduction
2.2 The Markov Process
2.3 Optimization of Long-Run Average
2.3.1 State Comparability and Performance Potentials
2.3.2 Conditions for State Comparability
2.3.3 Performance-Difference Formula
2.3.4 Performance Optimization
2.4 Bias Optimality
2.4.1 Bias Potential
2.4.2 The Bias-Difference Formula
2.4.3 The Space of Average Optimal Policies
2.4.4 Bias Optimality Conditions
2.5 Optimization of Multi-class Markov Processes
2.5.1 State Classification
2.5.2 Performance-Difference Formula
2.5.3 Performance Optimization
2.6 Optimization of Discounted Rewards
2.7 Special Cases, Extensions, and Discussions
References
3 Optimal Control of Diffusion Processes
3.1 Fundamental Mathematics
3.1.1 Stochastic Differential Equations
3.1.2 Stochastic Calculus
3.1.3 Solutions to Stochastic Differential Equations
3.1.4 Application in Finance: The Black–Scholes Equation
3.2 Stochastic Calculus with Non-smooth Features
3.2.1 Local Time, Ito–Tanaka Formula, and the Skorokhod Problem
3.2.2 Stochastic Calculus for Semi-smooth Functions
3.2.3 The One-Dimensional System
3.2.4 Stochastic Calculus in Relative Time
3.3 Long-Run Average Optimization (Single Class)
3.3.1 Performance-Difference Formula
3.3.2 Performance Optimization
3.4 Finite-Horizon Control Problems
3.4.1 Main Results
3.4.2 When the Value Function Cannot Be Reached
3.4.3 Time-Dependent Problems
3.5 Optimal Stopping
3.5.1 A General Formulation
3.5.2 Pure Optimal Stopping
3.5.3 Illustrative Examples
3.6 Singular Control
3.6.1 Formulated with Reflecting Points
3.6.2 Optimality Conditions
References
4 Degenerate Diffusion Processes
4.1 Multi-class Structure of Degenerate Diffusion Processes
4.1.1 Transient and Recurrent States
4.1.2 W-Ergodic and Branching States
4.2 Semi-smoothness of Potential Functions
4.2.1 Motivating Examples
4.2.2 The Proof of Semi-smoothness with Finite Horizon
4.2.3 Potential Functions for Long-Run Average
4.2.4 Extensions
4.3 Optimization of Degenerate Processes
4.3.1 Long-Run Average
4.3.2 Finite Horizon
4.3.3 Optimal Stopping
4.3.4 Singular Control
References
5 Multi-dimensional Diffusion Processes
5.1 Optimal Control with Smooth Performance Functions
5.1.1 Multi-dimensional Diffusion Processes and Ito Formula
5.1.2 Control Problems
5.2 Calculus of Semi-smooth Functions
5.2.1 Definitions
5.2.2 Smooth Quadrants and Taylor Expansion
5.2.3 Properties of Semi-smooth Functions
5.3 Stochastic Calculus of Semi-smooth Functions
5.3.1 Tanaka Formula
5.3.2 Calculus in Relative Time
5.4 Control Problems with Semi-smooth Performance Functions
5.4.1 System Dynamics on Degenerate Curves
5.4.2 Semi-smoothness of Performance Functions
5.4.3 Finite-Horizon Stochastic Control
5.5 Discussions
References
6 Performance-Derivative-Based Optimization
6.1 First-Order Optimality Condition
6.2 Optimization with Distorted Probability
References
Appendix A Stochastic Diffusions
A.1 Brownian Motions
A.1.1 The Preliminaries
A.1.1.1 Mathematical Definition of Brownian Motion
A.1.2 Probability Distributions
A.1.2.1 Probabilities of First Passage Times
A.1.2.2 Probability of the Maximum
A.1.3 Total and Quadratic Variations
A.1.3.1 Total Variation
A.1.3.2 Quadratic Variation
A.2 Stochastic Calculus
A.2.1 Stochastic Integrations
A.2.1.1 Definition
A.2.1.2 Stochastic Differentials
A.2.1.3 Stochastic Differential Equations
A.2.2 Ito Formula
A.3 Diffusion Processes as Markov Processes
A.3.1 Infinitesimal Generator
A.3.1.1 Dynkin's Formula
A.4 Fokker–Planck Equation
Appendix B Stochastic Calculus with Non-smooth Features
B.1 Reflected Brownian Motion and Skorokhod Problem
B.1.1 Definition
B.1.2 Local Time
B.2 Reflected Diffusion Processes
B.2.1 Solution to (B.13)
B.2.2 The Order of E[d ξ(t)]
B.3 More than One Reflecting Points
B.4 Stochastic Calculus for Semi-smooth Functions
B.4.1 Ito–Tanaka Formula for Semi-smooth Functions
B.4.2 Intuitive Explanation
B.4.2.1 First-Order Non-smoothness
B.4.2.2 Second-Order Semi-smooth Functions
B.4.2.3 At the Boundary Point
Appendix C Solutions
C.1 Problems of Chap. 3摥映數爠eflinkch:scdp33
C.2 Problems of Chap. 5摥映數爠eflinkch:multi55
Index