Preface

About this book¶

This book is about dynamic programming and its applications in economics, finance, and adjacent fields. It brings together recent innovations in the theory of dynamic programming and provides applications and code that can help readers approach the research frontier. The book is aimed at graduate students and researchers, although most chapters are accessible to undergraduate students with solid quantitative backgrounds.

The book contains classical results on dynamic programming that are found in texts such as Bellman (1957), Denardo (1981), Puterman (2005), and Stokey & Lucas (1989), as well as extensions created by researchers and practitioners over the last few decades as they wrestled with how to formulate and solve dynamic models that can explain patterns observed in data. These extensions include recursive preferences, robust control, continuous-time models, and time varying-discount rates. Such settings often fail to satisfy contraction-mapping restrictions on which traditional methods are based. To accommodate these applications, the key theoretical chapters of this book (Chapter 8 and Chapter 9) adopt and extend the abstract framework of Bertsekas (2022). This approach provides great generality while also offering transparent proofs.

Chapter 1--Chapter 3 provide motivation and background material on solving fixed point problems and computing lifetime valuations. Chapter 4 and Chapter 5 cover optimal stopping and Markov decision processes, respectively. Chapter 6 extends the Markov decision framework to settings where discount rates vary over time. Chapter 7 treats recursive preferences. The main theoretical results on dynamic programming from Chapter 4 to Chapter 6 are special cases of more general results in Chapter 8 and Chapter 9. A brief discussion of continuous-time models can be found in Chapter 10.

Mathematically inclined readers with background in dynamic programming might prefer to start with the general results in Chapter 8 and Chapter 9. Indeed, it is possible to read the text in reverse, jumping to Chapter 8 and Chapter 9 and then moving back to cover special cases according to interests. However, our teaching experience tells us that most students find the general results challenging on first pass but considerably easier after they have practiced dynamic programming through the earlier chapters. This is why we have started the presentation with special cases and ended it with general results.

Instructors wishing to use this book as a text for undergraduate students can start with Chapter 1, skim through Chapter 2, cover Chapter 3--Chapter 5 in depth, optionally include Chapter 6, and skip Chapter 7--Chapter 10 entirely.

Volume I scope¶

This book focuses on dynamic programs with finite-state spaces, leaving more general settings to Volume 2. Restricting attention to finite states involves some costs, since there are specific settings where continuous-state models are simpler (one example being Gaussian linear-quadratic models). Moreover, many continuous-state models allow us to unleash calculus, one of humanity’s most useful inventions.

Nevertheless, finite-state models are extremely useful. Computational representations are always implemented using finitely many floating point numbers, and many workhorse models in economics and finance are already discrete. In addition, focusing on problems with finite state spaces allows us to avoid using function-analytic and measure-theoretic machinery and imposing associated auxilary conditions required to ensure measurability and the existence of extrema. Without these distractions, the core theory of dynamic programming is especially simple.

For these reasons, we believe that even for sophisticated readers, a good approach to dynamic programming begins with a thorough analysis of the finite-state case. This is the task that we have tackled in Volume 1.

Companion code¶

Computer code is a first-class citizen in this book. Code is written in Julia and can be found at

https://github.com/QuantEcon/book-dp-public

We chose Julia because it is open source and because Julia allows us to write computer code that is as close as possible to the relevant mathematical equations. Julia code in the text is written to maximize clarity rather than speed.

We have also written matching Python code that can be found in the same repository. When combined with appropriate scientific libraries, Python is very practical and efficient for dynamic programming, but implementations tend to be library specific and are sometimes not as clean as those in Julia. That is why we chose Julia for programs embedded in the text.

We have tried to mix rigorous theory with exciting applications. Despite the various layers of abstractions used to unify the theory, the results are practical, being motivated by important optimization problems from economics and finance.

This book is one of several being written in partnership with the QuantEcon organization, with funding generously provided by Schmidt Futures (see acknowledgments). There is some overlap with the first book in the series, Sargent & Stachurski (2023), particularly on the topic of Markov chains. Although repetition is sometimes undesirable, we decided that some overlap would be useful, since it saves readers from having to jump between two documents.