β Scribed by Karl Hinderer
No coin nor oath required. For personal study only.
Preface
Contents
List of Symbols for the Deterministic Models of Part I
List of Symbols for the Stochastic Models of Part II
List of Symbols for Generalizations of Markovian Decision Processes of Part III
List of Special Symbols
List of Abbreviations
1 Introduction and Organization of the Book
Part I Deterministic Models
2 The Stationary Deterministic Model and the Basic Solution Procedure
2.1 A Motivating Example
2.2 The Model
2.3 The Basic Solution Procedure
2.4 First Examples
2.5 Problems
2.6 Supplements
3 Additional General Issues
3.1 The Basic Theorem for Cost Minimization
3.2 The Basic Theorem Using Reachable States
3.3 DPs with an Absorbing Set of States
3.4 Finiteness of the Value Functions and Bounding Functions
3.5 Problems
3.6 Supplements
4 Examples of Deterministic Dynamic Programs
4.1 Examples with an Explicit Solution
4.2 Further Examples
4.3 Problems
4.4 Supplement
5 Absorbing Dynamic Programs and Acyclic Networks
5.1 Absorbing Dynamic Programs
5.2 Cost-minimal Subpaths in Acyclic Networks
5.3 Problems
5.4 Supplements
6 Monotonicity of the Value Functions
6.1 Monotone Dependence on the Horizon
6.2 Relations and Orderings
6.3 Monotone Dependence on the Initial State
6.4 Splitting Models
6.5 Problems
6.6 Supplement
7 Concavity and Convexity of the Value Functions
7.1 Concave Value Functions
7.2 Convex Value Functions and Bang-Bang Maximizers
7.3 Discretely Convex Value Functions
7.4 Problems
7.5 Supplements
8 Monotone and ILIP Maximizers
8.1 Several Algorithms for Computing the Smallest Maximizer
8.2 Monotone Dependence on the Current State
8.3 Increasing and Lipschitz Continuous Dependence on the State
8.4 Monotone Dependence on the Stage Number
8.5 Problems
9 Existence of Optimal Action Sequences
9.1 Upper and Lower Semicontinuous Functions
9.2 Existence of Maximizers
9.3 Lipschitz Continuity
9.4 Problems
9.5 Supplement
10 Stationary Models with Large Horizon
10.1 The General Theory
10.2 The Structure of the Limit Value Function
10.3 DPs with Infinite Horizon
10.4 Problems
10.5 Supplement
Part II Markovian Decision Processes
11 Control Models with Disturbances
12 Markovian Decision Processes with Finite Transition Law
12.1 Finite Horizon
12.2 Large Horizon
12.3 Infinite Horizon
12.4 Problems
12.5 Supplements
13 Examples of Markovian Decision Processes with Finite Transition Law
13.1 Examples with Explicit Solutions
13.2 MDPs with an Absorbing Set of States
13.3 MDPs with Random Initial State
13.4 Stopping Problems
13.5 Terminating MDPs
13.6 Non-stationary MDPs and CMs
13.7 Problems
13.8 Supplement
14 Markovian Decision Processes with Discrete Transition Law
14.1 The Finite Horizon Model
14.2 Large and Infinite Horizon
14.3 Problems
14.4 Supplements
15 Examples with Discrete Disturbances and with Discrete Transition Law
16 Models with Arbitrary Transition Law
16.1 The Models MDP and CM
16.2 Models with Random Environment
16.3 Continuous Versions of Some Examples
17 Existence of Optimal Policies
18 Stochastic Monotonicity and Monotonicity of the Value Functions
18.1 Monotonicity
18.2 Stochastic Monotonicity
18.3 Further Concepts of Stochastic Monotonicity
19 Concavity and Convexity of the Value Functions and Monotone Maximizers
19.1 Concave and Convex Value Functions
19.2 Monotone Maximizers
20 Markovian Decision Processes with Large and with Infinite Horizon
20.1 Large Horizon
20.2 Infinite Horizon
Part III Generalizations of Markovian Decision Processes
21 Markovian Decision Processes with Disturbances
21.1 The Model MDPD
21.2 Problems
21.3 Supplements
22 Markov Renewal Programs
22.1 The Finite Horizon Model
22.2 The Infinite Horizon Model
22.3 Infinite Stage Markov Renewal Programs with Finite Time Horizon
23 Bayesian Control Models
23.1 The Model BCM
23.2 The Model BCM with Large Horizon
23.3 Problems
23.4 Supplements
24 Examples of Bayesian Control Models
24.1 Linear-Quadratic and Gambling Problems
24.2 Optimal Stopping and Asset Selling
24.3 Bayesian Sequential Statistical Decision Theory
24.4 Problems
25 Bayesian Models with Disturbances
25.1 The Model BMDPD
25.2 The Models BMCM and BMDP
25.3 Problems
26 Partially Observable Models
26.1 The Models POM and POMDP
26.2 The Formal Treatment
26.3 Supplements
A Elementary Results on Optimization
A.1 Real and Extended Real Numbers
A.2 Sets
A.3 Mappings
A.4 Extrema of Functions
A.4.1 Extrema of Functions on Arbitrary Domains
A.4.2 Extrema of Functions on Intervals
A.4.3 Extrema of Functions of Several Real Variables
A.5 Vector Spaces
A.6 Induction
B Measure and Probability
B.1 Measurability
B.2 Measures
B.3 Probability
C Metric Spaces
Elementary Facts on Metric Spaces
D Convexity
D.1 Convex Sets
D.2 Convex Functions
D.2.1 Convex Functions on Subsets of Rk
D.2.2 Convex Functions on Intervals
D.2.3 Convex Functions of Several Variables
D.3 Minimization of Convex Functions
D.4 Maximization of Convex Functions
D.5 Convex Functions on Discrete Intervals
D.6 Vector-Valued Convex Mappings
Index of Appendix
References
List of the Most Important Examples
Index
π SIMILAR VOLUMES
<p>This book explores discrete-time dynamic optimization and provides a detailed introduction to both deterministic and stochastic models. Covering problems with finite and infinite horizon, as well as Markov renewal programs, Bayesian control models and partially observable processes, the book focu
<p>This book may be regarded as consisting of two parts. In Chapters I-IV we preΒ sent what we regard as essential topics in an introduction to deterministic optimal control theory. This material has been used by the authors for one semester graduate-level courses at Brown University and the Univers
<p>Current research results in stochastic and deterministic global optimization including single and multiple objectives are explored and presented in this book by leading specialists from various fields. Contributions include applications to multidimensional data visualization, regression, survey c