Preface

This book is the second of a two-volume sequence on theory and applications of dynamic programming. Volume 1 Sargent & Stachurski, 2025 focused on models with finite state and action spaces. This volume treats general state and action spaces. This is useful because many models are more easily expressed as optimization problems with infinite sets for states and actions. Moreover, settings with infinite states and choices give us access to new tools, including calculus and gradients.

Following on from Chapters 8 and 9 of Sargent & Stachurski (2025), we work within an abstract setting that builds on the framework in Bertsekas (2022) and Sargent & Stachurski (2025). Four features distinguish our treatment from the standard textbook approach (e.g., Stokey & Lucas (1989)Puterman (2005)Hernández-Lerma & Lasserre (2012)).

First, at least for all high-level theory, we replace metric and contraction assumptions with order-theoretic ones. The policy operators of dynamic programming are almost universally order preserving, even when they fail to be contractions, and order theory turns out to provide the right foundation for a unified optimality theory.

Second, this shift lets us treat, within a single framework, a very wide range of problems drawn from economics, finance, operations research, and artificial intelligence: state-dependent, negative, and nonlinear discounting; recursive, risk-sensitive, and ambiguity-averse preferences; robust and adversarial control; sequential analysis and stochastic shortest paths; post-action value functions used in structural estimation; distributional dynamic programming; and linear-quadratic control on the cone of positive semidefinite matrices. Most of these problems fall outside the reach of standard theory.

Third, the framework specializes cleanly to the classical setting: standard Markov decision processes, optimal savings problems, natural resource problems, inventory control problems, discounted optimal stopping problems, and firm-valuation problems all reappear as concrete applications of a single abstract theory.

Fourth, the resulting proofs are predominantly algebraic, rather than geometric or topological. This makes the core theory accessible with minimal function-analytic background, simple to manipulate --- and a natural target for machine-assisted reasoning.

Although the core theory component of the book rests on relatively elementary order-based arguments, mathematical prerequisites for the entire book are significantly higher than those for Volume 1. In order to progress through all the applications, readers will need at least familiarity with basic concepts from functional analysis, measure theory, and order theory. For convenience, we have provided an extensive appendix that reviews these topics.

While Volume 1 starts with specific models and gradually builds towards general theory -- an approach designed to help readers who are just starting to learn dynamic programming -- this volume takes more of a mathematician’s approach by first setting forth the general theory, then specializing to particular applications.

Volume 1 especially has been deeply influenced by the elegant and insightful book Abstract Dynamic Programming by Dimitri Bertsekas (Bertsekas, 2022), now in its third edition. While our approach here has diverged somewhat from his framework, it remains true that this volume is also inspired by Bertsekas’s work.

This book has been a combined effort involving many people. We are indebted to Schmidt Futures for financial support, stemming from a grant organized by Jim Savage. Thanks are also due to our PhD students Matheus Villas Boas Alves, Shu Hu, Nisha Peng, Simon Mishricky, Longye Tian, and Humphrey Yang, all of whom contributed significantly to the final product. In addition, many other friends, former students, and colleagues helped with preparation, either by directly reading and commenting on the book or via research collaborations. These include Jingni Yang, Yuchao Li, Qingyin Ma, Akshay Shanker, Alexis Akira Toda, Junnan Zhang, and Sylvia Zhao. We are deeply grateful for all of their contributions, while retaining full responsibility for any errors that remain.

The second author would like to thank his wife and parents for their unflagging support, as well as the Institute of Economic Research at Kyoto University for hosting him during 2025, where substantial progress was made, and also the International University of Japan, the site of a final desperate push in the early months of 2026. He extends his profound appreciation and gratitude to his hosts Yoshi Nishiyama and Tadashi Sekiguchi at KIER, and to Yue Hua and Chien-Yu Huang at IUJ. A final thanks goes to the stoner rock bands Spaceslug, Elephant Tree, King Buffalo, and All Them Witches. Their songs provided the perfect backing tracks for constructing DP theory.