Continuous-time Markov chains and applications : a two-time-scale approach

✍ Scribed by G George Yin; Qing Zhang

• Publisher: Springer
• Year: 2013
• Tongue: English
• Leaves: 451
• Series: Applications of mathematics, 37
• Edition: 2nd ed
• Category: Library

✦ Synopsis

Prologue and Preliminaries: Introduction and overview- Mathematical preliminaries.- Markovian models.- Two-Time-Scale Markov Chains: Asymptotic Expansions of Solutions for Forward Equations.- Occupation Measures: Asymptotic Properties and Ramification.- Asymptotic Expansions of Solutions for Backward Equations.- Applications:MDPs, Near-optimal Controls, Numerical Methods, and LQG with Switching: Markov Decision Problems.- Stochastic Control of Dynamical Systems.- Numerical Methods for Control and Optimization.- Hybrid LQG Problems.- References.- Index

✦ Table of Contents

Continuous-Time Markov Chains and Applications: A Two-Time-Scale Approach,
2nd edition......Page 4
Contents......Page 8
Preface......Page 14
Preface to the First Edition......Page 16
Convention......Page 18
Glossary of Symbol and Notation......Page 20
Part I: Prologue and Preliminaries......Page 24
1.1 Introduction......Page 26
1.2.1 Markov Chains......Page 33
1.2.2 Singular Perturbations......Page 34
1.3 Outline of the Book......Page 35
2.2 Martingales......Page 40
2.3 Markov Chains......Page 41
2.4.1 Construction of Markov Chains......Page 44
2.5 Irreducibility and Quasi-Stationary Distributions......Page 46
2.6 Gaussian Processes and Diffusions......Page 48
2.7 Switching Diffusions......Page 50
2.8 Notes......Page 51
3.1 Introduction......Page 54
3.2 Birth and Death Processes......Page 55
3.3.1 Queues with Finite Capacity......Page 57
3.3.2 System Reliability......Page 60
3.3.3 Competing Risk Theory......Page 62
3.3.4 Two-Time-Scale Cox Processes......Page 63
3.3.5 Random Evolutions......Page 64
3.3.6 Seasonal Variation Models......Page 65
3.4.1 Simulated Annealing......Page 68
3.4.2 Continuous-Time Stochastic Approximation......Page 69
3.4.3 Systems with Markovian Disturbances......Page 71
3.5.1 Linear Quadratic Control Problems......Page 72
3.5.2 Singularly Perturbed LQ Systems with Wide-Band Noise......Page 73
3.5.3 Large-Scale Systems: Decompositionand Aggregation......Page 74
3.6 Time-Scale Separation......Page 76
3.7 Notes......Page 78
Part II: Two-Time-Scale Markov Chains......Page 80
4.1 Introduction......Page 82
4.2 Irreducible Case......Page 85
4.2.1 Asymptotic Expansions......Page 86
4.2.2 Outer Expansion......Page 89
4.2.3 Initial-Layer Correction......Page 92
4.2.4 Exponential Decay of ψk(·)......Page 95
4.2.5 Asymptotic Validation......Page 97
4.2.6 Examples......Page 101
4.2.7 Two-Time-Scale Expansion......Page 104
4.3 Markov Chains with Multiple Weakly Irreducible Classes......Page 107
4.3.1 Asymptotic Expansions......Page 111
4.3.2 Analysis of Remainder......Page 124
4.3.4 Summary of Results......Page 125
4.3.5 An Example......Page 127
4.4 Inclusion of Absorbing States......Page 130
4.5 Inclusion of Transient States......Page 138
4.6.1 Countable-State Spaces: Part I......Page 149
4.6.2 Countable-State Spaces: Part II......Page 152
4.6.3 A Remark on Finite-DimensionalApproximation......Page 155
4.7 Remarks on Singularly Perturbed Diffusions......Page 156
4.8 Notes......Page 160
5.1 Introduction......Page 164
5.2 The Irreducible Case......Page 165
5.2.2 Conditions and Preliminary Results......Page 166
5.2.3 Exponential Bounds......Page 171
5.2.4 Asymptotic Normality......Page 182
5.2.5 Extensions......Page 192
5.3 Markov Chains with Weak and Strong Interactions......Page 196
5.3.1 Aggregation of Markov Chains......Page 197
5.3.2 Exponential Bounds......Page 205
5.3.3 Asymptotic Distributions......Page 214
5.4 Measurable Generators......Page 236
5.5.1 Inclusion of Transient States......Page 245
5.5.2 Inclusion of Absorbing States......Page 248
5.6 Remarks on a Stability Problem......Page 252
5.7 Notes......Page 256
6.1 Introduction......Page 258
6.2.1 A Preliminary Lemma......Page 259
6.2.2 Formulation......Page 260
6.3 Construction of Asymptotic Expansions......Page 261
6.3.1 Leading Term ϕ0(t) and Zero-Order Terminal-Layer
Term ψ0(τ )......Page 264
6.3.2 Higher-Order Terms......Page 266
6.4 Error Estimates......Page 269
6.5 Asymptotic Expansions Including Transient States......Page 273
6.6 Remarks......Page 278
6.6.1 Related Problems......Page 279
6.7 Notes......Page 280
Part III: Applications: MDPs, Near-optimal Controls, Numerical Methods, and LQG with Switching......Page 282
7.1 Introduction......Page 284
7.2 Problem Formulation......Page 286
7.3 Limit Problem......Page 288
7.4 Asymptotic Optimality......Page 292
7.5 Convergence Rate and Error Bound......Page 295
7.6 Long-Run Average Cost......Page 297
7.7 Computational Procedures......Page 303
7.8 Notes......Page 305
8.1 Introduction......Page 308
8.2 Problem Formulation......Page 310
8.3 Properties of the Value Functions......Page 313
8.4 Asymptotic Optimal Controls......Page 319
8.5 Convergence Rate......Page 323
8.6 Weak Convergence Approach......Page 329
8.6.2 Relaxed Control Formulation......Page 330
8.6.3 Near Optimality......Page 331
8.7 Notes......Page 339
9.1 Introduction......Page 342
9.2 Numerical Methods for Optimal Control......Page 343
9.3.1 Stochastic Optimization Formulation......Page 347
9.3.2 Convergence......Page 350
9.3.3 Examples......Page 356
9.3.4 Error Bounds......Page 359
9.4 Notes......Page 362
10.1 Introduction......Page 364
10.2 Problem Formulation......Page 366
10.3 Optimal Controls......Page 367
10.4 Two-Time-Scale Approximation: Recurrent States......Page 368
10.4.1 Limit Riccati Equations......Page 369
10.4.2 Nearly Optimal Controls......Page 377
10.5 Two-Time-Scale Approximation: Inclusion ofTransient States......Page 381
10.6 Two-Time-Scale Approximation: Inclusionof Absorbing States......Page 385
10.7 A Numerical Example......Page 389
10.8 Remarks on Indefinite Control Weights......Page 391
10.9 Notes......Page 394
A.1 Properties of Generators......Page 396
A.2 Weak Convergence......Page 399
A.3 Relaxed Control......Page 405
A.4 Viscosity Solutions of HJB Equations......Page 406
A.5 Value Functions and Optimal Controls......Page 411
A.6 Miscellany......Page 422
Bibliography......Page 430
Index......Page 448