7 Recursive Decision Processes

In this chapter we study what Sargent & Stachurski (2025) call recursive decision processes (RDPs). The following display shows where RDPs fit relative to the other major classes of DP models studied so far in this book:

The main role of RDPs is to extend LDPs by accommodating nonlinearities in aggregators and calculation of present values. Another difference between this chapter and our discussion of LDPs is that we allow for unbounded rewards and value functions. While this can also be done in an LDP setting, the corresponding analysis turns out to be cleaner when working with RDPs.

Section 7.1 introduces the RDP framework, provides examples, clarifies the relationships between RDPs, LDPs and ADPs, and discusses existence of greedy policies. Section 7.2.1 and Section 7.2.2 present optimality results---first for bounded rewards and then for unbounded rewards handled via weighted contractions---while Section 7.2.3 gives conditions under which the value function is monotone, concave, or uniquely determined. After a digression on certainty equivalents (Section 7.2.4), we extend the optimality theory to MDPs with general certainty equivalents (Section 7.2.5). Section 7.3.1 applies the theory to optimal savings with unbounded utility, and we then study irreversible investment under risk neutrality, risk aversion, and ambiguity aversion.

7.1Introduction¶

Section 7.1.1 defines RDPs and provides examples. We then clarify the relationships between RDPs, LDPs and ADPs, and discuss existence of greedy policies in Section 7.1.3.

7.1.1Definition and Examples¶

To a first approximation, RDPs are dynamic programs with a Bellman equation of the form

for some suitable choice of $B$. Here $x$ is the state, $a$ is an action, $\Gamma$ is a feasible correspondence and $B$ is an “aggregator,” with interpretation

$B(x, a, v) =$ total lifetime rewards, contingent on current action $a$, current state $x$ and the use of $v$ to evaluate future states.

In Section 7.1.1.1--Section 7.1.1.5 we improve this definition and then provide examples. As usual, in a topological space setting, “measurable” means “Borel measurable” unless otherwise stated.

7.1.1.1Definition¶

Let $\Xsf$ and $\Asf$ be separable metric spaces, referred to henceforth as the state and action spaces respectively. Given these spaces, a recursive decision process (RDP) is a tuple $(\Gamma, V, B)$ containing

1. a nonempty correspondence $\Gamma$ from $\Xsf$ to $\Asf$ called the feasible correspondence, with an associated set of feasible state-action pairs

and an associated set of feasible policies

2. a subset $V$ of $\RR^\Xsf$ called the value space,

3. a map $B \colon \Gsf \times V \to \RR$, referred to as an aggregator, satisfying the monotonicity condition

for every $v, w \in V$ and every $(x, a) \in \Gsf$, and the consistency condition

Several objects, such as $\Xsf, \Asf$ and $\Gamma$ are familiar from our definition of LDPs in Section 6.1.2.1. Analgous to the LDP case, when representing the RDP by the tuple $(\Gamma, V, B)$, we are treating $\Xsf$ and $\Asf$ as understood from context.

The value space $V$ is a class of functions that assign values to states. The order on the left side of (7.5) is the usual pointwise partial order for functions. The monotonicity restriction is natural: relative to $v$, if rewards are at least as high under $w$ in every future state, then the total rewards we can extract under $w$ should be at least as high.

The final condition, in (7.6), is a consistency condition implying that $V$ is large enough to capture the value of following a particular policy.

7.1.1.2Example: Finite MDPs¶

In Section 1.2 we introduced the basic MDP model, with finite state space $\Xsf$, finite action space $\Asf$, and remaining primitives $(\Gamma, r, \beta, P)$ as given in Section 1.2.1.1. This maps easily to the RDP setting by taking $V = \RR^\Xsf$, $\Gamma$ as given, and

for $(x, a) \in \Gsf$ and $v \in \RR^\Xsf$.

For this model, it is clear that the RDP Bellman equation $v(x) = \max_{a \in \Gamma(x)} B(x, a, v)$ from (7.2) agrees with the original expression we gave in (1.46).

7.1.1.3Example: The Firm Valuation Problem¶

Recall the firm decision problem we analyzed in Section 1.1.1.2, where the decision is binary (0 means continue and 1 means sell) and the state $x$ takes values in a set $\Xsf$ and evolves via stochastic kernel $P$. To map this problem to an RDP we set $\Asf = \{0,1\}$ and $\Gamma(x) = \Asf$ for all $x$. We take $V \coloneq b\Xsf$ as the value space and set

The monotonicity condition (7.5) clearly holds. A policy is a $\bB$-measurable map $\sigma \colon \Xsf \to \{0,1\}$. Given any such policy and any $v \in b\Xsf$, the function

is in $b\Xsf$ (since $\pi$ is assumed to be bounded), so the consistency condition (7.6) also holds. For this model, the RDP Bellman equation $v(x) = \max_{a \in \Gamma(x)} B(x, a, v)$ becomes

This is equivalent to the original statement of the Bellman equation in Theorem 1.1.1, on page .

7.1.1.4Firm Valuation with Unbounded Profits¶

The firm valuation problem can also fit into the RDP framework when profits are unbounded, at least in some cases. For example, suppose that $\Xsf = \RR_+$, that $\ell$ is a given weight function on $\RR_+$ (see Section A.5.3.5), and that there exist nonnegative constants $\alpha, \eta, \delta$ such that

(These conditions bound the rate at which profits grow.) We again take $B$ as in (7.8) and $\Gamma(x) = \{0,1\}$ for all $x \in \RR_+$. We set $V$ equal to $b_\ell \Xsf$, the set of measurable functions $v \in \RR^\Xsf$ with $\| v \|_\ell < \infty$. (Here $\| \cdot \|_\ell$ denotes the $\ell$-weighted supremum norm, as in Section A.5.3.5.)

7.1.1.5Example: Optimal Savings¶

Consider the optimal savings problem studied in Section 1.3. The state is $w \in \RR_+$ and the action is $c \in \RR_+$. The feasible correspondence is $\Gamma(w) = [0, w]$ and $V \coloneq b\RR_+$ is the value space. We set

As in Assumption 1.3.1, we take $u$ to be bounded and continuous. Under these restrictions, the tuple $(\Gamma, V, B)$ is an RDP. The function $B$ is real-valued and the monotonicity condition (7.5) clearly holds. The consistency condition (7.6) holds because, by the definition of $\Gamma$, a policy is a Borel measurable map $\sigma \colon \RR_+ \to \RR_+$ with $0 \leq \sigma(w) \leq w$ for all $w$, and given any such policy and any $v \in b\RR_+$, the function

is measurable and bounded (since $u$ is bounded and continuous).

For this model, the RDP Bellman equation $v(x) = \max_{a \in \Gamma(x)} B(x, a, v)$ from (7.2) agrees with the optimal savings Bellman equation in (1.98).

7.1.1.6Example: Savings with Kreps--Porteus Expectations¶

Consider a variation of the optimal savings model from Section 7.1.1.5 with Epstein--Zin-type preferences. To simplify the presentation, we set the EIS parameter to $\psi = \infty$, so that the CES aggregator reduces to addition, while retaining the nonlinear Kreps--Porteus expectation over future values. The Bellman equation becomes

where $\gamma > 0$ with $\gamma \neq 1$ is the coefficient of relative risk aversion.

As before, the state is $w \in \RR_+$, the action is $c \in \RR_+$, and $\Gamma(w) = [0, w]$. To avoid raising zero to a negative power, we assume that $u$ is measurable and that there exist constants $0 < \underline{u} \leq \bar{u} < \infty$ with $\underline{u} \leq u(c) \leq \bar{u}$ for all $c \in \RR_+$. The aggregator is

We set the value space $V$ to be all measurable functions $v \colon \RR_+ \to [\underline{u}, \bar{u}]$.

With $B$ defined as above, the RDP Bellman equation $v(x) = \max_{a \in \Gamma(x)} B(x, a, v)$ from (7.2) agrees with (7.16).

7.1.1.7Example: MDPs with Modified Rewards¶

Some authors use an MDP framework where current rewards depend on the next period state, so that the Bellman equation has the form

Here $r$ maps $\Gsf \times \Xsf$ to $\RR$ and other primitives are unchanged. We take $V = \RR^\Xsf$, $\Gamma$ as given, and set

Evidently, for the associated RDP $(\Gamma, V, B)$, the monotonicity and consistency conditions (7.5) and (7.6) both hold. For this choice of $B$, the RDP Bellman equation $v(x) = \max_{a \in \Gamma(x)} B(x, a, v)$ agrees with the modified MDP Bellman equation in (7.19).

7.1.1.8Example: Risk-Sensitive Preferences¶

In §Example 6.1.6 we discussed a risk-sensitive MDP with entropic certainty equivalent. This model can be embedded in the RDP framework by setting $V = \RR^\Xsf$, $\Gamma$ as given, and

The parameter $\theta$ is any nonzero real value.

When $B$ is given by (7.21), the RDP Bellman equation $v(x) = \max_{a \in \Gamma(x)} B(x, a, v)$ from (7.2) becomes

7.1.2RDPs vs LDPs vs ADPs¶

As mentioned at the start of the chapter, we have MDPs $\subset$ LDPs $\subset$ RDPs $\subset$ ADPs and the inclusions are all strict. We already know that the first inclusion is strict (consider, for example, the LDP with state-dependent discounting in Section 6.1.4). Here we review the remaining relationships.

7.1.2.1LDPs are RDPs¶

Let $(\Gamma, r, K)$ be an LDP with state and action spaces $\Xsf$, $\Asf$, as defined in Section 6.1.2.1. Setting $V = b\Xsf$ and

the resulting tuple $(\Gamma, V, B)$ is an RDP. To see this, note that $V = b\Xsf$ is a subset of $\RR^\Xsf$, so we only need to check the monotonicity and consistency conditions for the aggregator $B$. For monotonicity, fix $(x, a) \in \Gsf$ and $v, w \in V$ with $v \leq w$. Since $K(x, a, \cdot)$ is a nonnegative measure, we have $\int v(x') K(x, a, \diff x') \leq \int w(x') K(x, a, \diff x')$ and hence $B(x, a, v) \leq B(x, a, w)$. For consistency, fix $\sigma \in \Sigma$ and $v \in b\Xsf$. We need to show that $m(x) \coloneq B(x, \sigma(x), v)$ is in $b\Xsf$. This follows from the LDP conditions, which require $r \in b\Gsf$ and $Kv \in b\Gsf$ whenever $v \in b\Xsf$.

The risk-sensitive MDP in Section 7.1.1.8 is an RDP but not an LDP, since the aggregator is nonlinear in future values.

7.1.2.2RDPs are ADPs¶

Every RDP generates an ADP. To see this, let $(\Gamma, V, B)$ be an RDP with state space $\Xsf$ and action space $\Asf$. The set $V$ is paired with the pointwise partial order. With $\Sigma$ as the set of feasible policies and given $\sigma$ in $\Sigma$, we define $T_\sigma$ by

The monotonicity and consistency conditions in (7.5)--(7.6) imply that $T_\sigma$ is an order-preserving self-map on $V$. Hence, with $\TT$ as the set of all policy operators, the pair $(V, \TT)$ is an ADP. We call $(V, \TT)$ the ADP generated by $(\Gamma, V, B)$.

For ADPs generated by RDPs, we can provide intuitive representations of greedy policies and the Bellman equation. For example, we recall from our ADP definition in (2.1) that a policy $\sigma \in \Sigma$ is $v$-greedy for ADP $(V, \TT)$ if $T_\tau \, v \leq T_\sigma \, v$ for all $\tau \in \Sigma$. If $(V, \TT)$ is generated by $(\Gamma, V, B)$, then this is equivalent to the statement that

Also, we recall that the ADP Bellman operator is defined by $Tv = \bigvee_\sigma T_\sigma \, v$ whenever the supremum exists. When $(V, \TT)$ is generated by $(\Gamma, V, B)$, this is equivalent to the statement $(Tv)(x) = \sup_{\sigma \in \Sigma} B(x, \sigma(x), v)$ for all $x \in \Xsf$ whenever the pointwise supremum exists (see Exercise A.4). Under reasonable conditions on $\Gamma$ and $B$, we will show that this can be improved to the stronger form

(see Lemma 7.1.1 and Lemma 7.1.2).

Since every RDP is an ADP, we can use ADP optimality results to study RDPs. Given an RDP $(\Gamma, V, B)$ and its generated ADP $(V, \TT)$, we make the obvious connections, saying that

• $(\Gamma, V, B)$ is regular if $(V, \TT)$ is regular,

• $T$ is the Bellman operator for $(\Gamma, V, B)$ when $T$ is the Bellman operator for $(V, \TT)$,

• $\sigma$ is optimal for $(\Gamma, V, B)$ when $\sigma$ is optimal for $(V, \TT)$,

• etc.

7.1.2.3Not all ADPs are RDPs¶

Although the RDP framework is broad, there are significant dynamic programs that fall outside this framework.

In the examples above, it is possible to rearrange the problem so that the $\max$ operator is shifted to the outside and, thereby, construct a version that fits the RDP framework. But there are good reasons to avoid this, related to smoothness and dimensionality (see, e.g., Kristensen et al. (2021) or Rust (1994)).

7.1.3Existence of Greedy Policies¶

As always, existence of greedy policies is important for our analysis. In this section, we investigate RDP environments where greedy policies exist. We begin with finite action spaces and then move to the general case.

7.1.3.1Finite Actions¶

We begin with the discrete choice setting, where greedy policies always exist.

Note that, in the simple setting of Lemma 7.1.1, the Bellman equation takes the form of (7.2). Below we investigate more complex settings where this is still true.

7.1.3.2Continuous Actions¶

Now we drop the finiteness restriction on $\Asf$. In these general RDP settings, existence is less trivial. Here we state one useful result.

7.2Optimality Results¶

Let’s put together some sufficient conditions for optimality of RDP models. We will focus here on models that are naturally contracting. This permits us to handle dynamic programs with both bounded and unbounded rewards.

We begin in Section 7.2.1 with the bounded case, where the aggregator $B$ is bounded and satisfies a Blackwell-type discounting condition. In Section 7.2.2 we extend to potentially unbounded rewards using weighted contractions. Finally, Section 7.2.3 investigates properties of solutions, giving sufficient conditions for the value function to be monotone, concave, or uniquely determined, and for the optimal policy to be continuous.

7.2.1Bounded Contractions¶

RDPs have strong optimality properties when they uniformly contract values. The current section investigates this case. Throughout this section, we assume that values are bounded. This typically occurs when reward functions are bounded. Later, in Section 7.2.2, we will consider unbounded settings.

7.2.1.1Framework¶

Let $\Xsf, \Asf$ be separable metric spaces, let $\Gamma$ be a nonempty correspondence from $\Xsf$ to $\Asf$, and let $\Gsf$ be the feasible state-action pairs (see Section 7.1.1.1). Set $V = b\Xsf$. Let $B \colon \Gsf \times V \to \RR$ be a given function such that

• $(x, a) \mapsto B(x, a, v)$ is measurable on $\Gsf$ for all $v \in V$, and

• $B(x, a, w) \leq B(x, a, v)$ for all $w \leq v$ in $V$ and $(x, a) \in \Gsf$.

The tuple $(\Gamma, V, B)$ is an RDP. To see this, note that the monotonicity condition (7.5) is given by the second restriction on $B$ above. For the consistency condition (7.6), fix $\sigma \in \Sigma$ and $v \in V$. The function $x \mapsto B(x, \sigma(x), v)$ is measurable, since $\sigma$ is measurable and $(x,a) \mapsto B(x,a,v)$ is measurable on $\Gsf$, and bounded, since $B$ is bounded by Assumption 7.2.1. Hence $B(\cdot, \sigma(\cdot), v) \in b\Xsf = V$.

7.2.1.2Finite Actions¶

We seek optimality results for the RDP $(\Gamma, V, B)$ introduced in Section 7.2.1.1. The simplest case is when the choice set is always finite. Also note that, in this setting, a policy $\sigma$ is $v$-greedy if and only if (7.32) holds.

7.2.1.3The Continuous Case¶

We now drop the finiteness assumption, while continuing to work with the RDP $(\Gamma, V, B)$ introduced in Section 7.2.1.1. In place of finiteness, we consider two continuity conditions on $B$.

The main result of this section is as follows.

7.2.2Weighted Contractions¶

In Section 7.2.1, we considered RDPs that are both contracting and bounded. Some useful RDPs fail to have this boundedness property. Here we extend our results to potentially unbounded problems that still retain contractivity. (While the results obtained in Section 7.2.1 are special cases of the results presented here (after minor modifications), we decided to present them separately in order to provide simple sufficient conditions in the bounded case.)

When maximizing, the theory works best for problems where rewards are unbounded above and bounded below. (One approach to the reverse type of unboundedness can be found in Ma et al. (2022).) Because we focus on such problems, we will typically assume that rewards are nonnegative. This costs no generality in such settings, since optimal policies are invariant to additive shifts.

Throughout this section, $\ell$ is a weight function on $\Xsf$, $\| \cdot \|_\ell$ denotes the $\ell$-weighted supremum norm, and $b_\ell \Xsf$ is all $f \colon \Xsf \to \RR$ with $f/\ell \in b\Xsf$. See Section A.5.3.5 for background and discussion of weight functions and the space $b_\ell \Xsf$.

7.2.2.1Framework¶

Let $\Gamma$ be a nonempty correspondence from $\Xsf$ to $\Asf$ and let $\Gsf$ be the feasible state-action pairs (see Section 7.1.1.1). Set $V = b_\ell \Xsf_+$, so that $V$ is the nonnegative functions in $b_\ell \Xsf$. Let $B \colon \Gsf \times V \to \RR_+$ be a given function. (In the current setting, where $B$ can be unbounded, we restrict attention to the case where $B$ is nonnegative. Since the weighted contraction approach pursued here works best for rewards that are unbounded above but bounded below, imposing nonnegativity costs very little in the way of generality.)

We suppose that

• $(x, a) \mapsto B(x, a, v)$ is measurable on $\Gsf$ for all $v \in V$, and

• $B(x, a, w) \leq B(x, a, v)$ for all $w \leq v$ in $V$ and $(x, a) \in \Gsf$.

We also require two conditions related to contractivity and $\ell$-boundedness:

Consider the tuple $(\Gamma, V, B)$.

7.2.2.2Finite Actions¶

We seek optimality results for the RDP $(\Gamma, V, B)$ introduced in Lemma 7.2.3. The simplest case is when the choice set is always finite:

In this case we have the following result:

7.2.2.3The Continuous Case¶

We now drop the finiteness assumption, while continuing to work with the RDP $(\Gamma, V, B)$ introduced in Lemma 7.2.3.

7.2.3Properties of Solutions¶

In this section, we seek sufficient conditions for the value and policy functions to have useful shape and continuity properties. We adopt the setting of Proposition 7.2.5 and study the properties of the RDP $(\Gamma, V, B)$ discussed in that result. In the proofs below, we repeatedly use Lemma A.2.6 on page .

7.2.3.1Monotone Values¶

First, we seek conditions under which the value function is increasing. In addition to the conditions in Proposition 7.2.5, we suppose that $\Xsf$ is partially ordered by $\preceq$. Let

Both conditions in Assumption 7.2.8 are monotonicity conditions. The first is equivalent to stating that $\Gamma$ is order preserving when viewed as a map from $(\Xsf, \preceq)$ to $(\wp(\Asf), \subset)$. Here $\wp(\Asf)$ is the set of all subsets of $\Asf$ and $\subset$ is the partial order induced by set inclusion (Example A.1.2).

7.2.3.2Concavity¶

Next we seek sufficient conditions for the value function to be concave. In this section, we assume that both $\Xsf$ and $\Asf$ are convex subsets of a vector space.

The convexity requirement on $\Gsf$ in Assumption 7.2.9 is equivalent to the statement that, for all $x, x'$ in $\Xsf$, all $a \in \Gamma(x)$ all $a' \in \Gamma(x')$ and all $\lambda \in [0, 1]$, we have

By taking $x=x'$, we see that each set $\Gamma(x)$ is convex in $\Asf$.

7.2.3.3Uniqueness and Continuity¶

When the conditions of Proposition 7.2.5 are in force, we know that at least one optimal policy exists in $\Sigma$. The question we ask now is, when is it unique? Not surprisingly, uniqueness can be obtained with a form of strict concavity.

7.2.4Digression on Certainty Equivalents¶

Before continuing with the theory of RDPs, it will be helpful to review risk measures and certainty equivalents. These concepts are in one-to-one correspondence: we convert between them by flipping signs. Certainty equivalents can be understood as extensions of mathematical expectation that include attitudes towards risk. In later sections, we will tie the discussion of risk measures and certainty equivalents back into RDP theory and its applications.

(The existence of parallel literatures on risk measures and certainty equivalents reflects the fact that researchers in finance and engineering often think about minimizing risk, while economists typically concern themselves with maximizing rewards.^[1] In this book, we tend to work with certainty equivalents, although the following discussion will allow readers to translate between the two.)

Throughout the following discussion, the triple $(\Omega, \fF, \PP)$ is a probability space and $L_\infty \coloneq L_\infty(\Omega, \fF, \PP)$ is the set of essentially bounded random variables on $(\Omega, \fF, \PP)$; that is, all random $Z$ admitting an $N \in \NN$ with $|Z| \leq N$ $\PP$-a.s.

In this setting, a risk measure is a map $\rR \colon L_\infty \to \RR$ satisfying

1. Monotonicity: If $Z, Z' \in L_\infty$ and $Z \leq Z'$ $\PP$-a.s., then $\rR(Z') \leq \rR(Z)$.

2. Cash invariance: $\rR(Z + a) = \rR(Z) - a$ for all $Z \in L_\infty$ and $a \in \RR$.

A certainty equivalent is a map $\eE \colon L_\infty \to \RR$ satisfying

1. Monotonicity: If $Z, Z' \in L_\infty$ and $Z \leq Z'$ $\PP$-a.s., then $\eE(Z) \leq \eE(Z')$.

2. Cash invariance: $\eE(Z + a) = \eE(Z) + a$ for all $Z \in L_\infty$ and $a \in \RR$.

(Note that the meaning of monotonicity and cash invariance changes from $\rR$ to $\eE$.)

We now define several significant subclasses of risk measures and state the corresponding properties of the associated certainty equivalent $\eE = -\rR$.

A risk measure $\rR$ is called convex if

for all $Z, Z' \in L_\infty$ and $\lambda \in [0, 1]$. Obviously, $\rR$ is convex if its negation $\eE \coloneq -\rR$ is concave:

Concavity of $\eE$ captures the idea that diversification is weakly preferred.

A risk measure $\rR$ is called coherent if it is convex and positively homogeneous, meaning that $\rR(\lambda Z) = \lambda\, \rR(Z)$ for all $Z \in L_\infty$ and $\lambda > 0$. The certainty equivalent $\eE = -\rR$ is then concave and positively homogeneous. Together with concavity, this means $\eE$ is superadditive and positively homogeneous.

7.2.4.1Duality¶

There is a dual representation theorem for convex risk measures, originally due to Föllmer & Schied (2002), that helps us interpret and manipulate these functionals. Here we restate their result in terms of concave certainty equivalents. In doing so, we will restrict attention to the law invariant case; that is, the case where $\eE(Z)$ depends only on the distribution of $Z$ for all $Z \in L_\infty$.^[2]

In the theorem statement, $P_Z$ is $\PP \circ Z^{-1}$, the distribution of $Z$, and the infimum is over all $Q \in \pP(\RR)$ such that $Q$ is absolutely continuous with respect to $P_Z$. The constraint $Q \ll P_Z$ means that if $P_Z$ says an event is impossible, then $Q$ must also say it’s impossible.

One way to interpret (7.50) is in terms of an adversarial agent who chooses $Q$ to minimize the expected return $\EE_Q[Z]$, while being constrained by a penalty term $\alpha(Q)$. This is the robust optimization point of view: the agent makes choices that are robust to variations by a real or fictitious adversary. The penalty function $\alpha$ controls how far the adversary is able to deviate from the reference model $P_Z$. From this perspective, the absolute continuity condition $Q \ll P_Z$ means that the adversary is allowed to disagree about how likely different scenarios are, but not about which scenarios are conceivable.

A second interpretation involves ambiguity. The agent does not know the true model and $P_Z$ is only a reference point. The agent’s cautious reasoning forces him to entertain a range of plausible models. The penalty term $\alpha(Q)$ reflects how implausible $Q$ is relative to $P_Z$. The absolute continuity constraint defines what the agent considers to be possible---the set of scenarios that could actually occur.

7.2.4.2Examples of Certainty Equivalents¶

Let’s look at examples, focusing primarily on certainty equivalents. Throughout this discussion, $Z$ is an element of $L_\infty$, $P_Z$ is its distribution, $F_Z$ is its cdf, and $F_Z^{-1}$ is the inverse cdf.

The simplest certainty equivalent is mathematical expectation: $\eE(Z) = \EE[Z]$. This corresponds to risk neutrality: the agent is indifferent between any random variable and its mean. The other extreme is the pessimistic certainty equivalent

We can think of $\eE_p(Z)$ as the left-hand end point of the support of $Z$. For the pessimistic certainty equivalent, the dual representation (7.50) becomes

Both of these examples are coherent.

Another example is the $\alpha$-quantile certainty equivalent

The value $\qQ_\alpha(Z)$ is the $\alpha$-quantile of $Z$. The corresponding risk measure $\rR_\alpha = - \qQ_\alpha$ is just $\alpha$-level value-at-risk (VaR).

VaR admits some pathologies. For example, VaR is not convex, and hence can increase under diversification. These deficiencies have motivated the introduction of conditional value at risk (CVaR) (also called average value at risk, or expected shortfall), defined as

The corresponding CVaR certainty equivalent is $\eE_\alpha(Z) = - \rR_\alpha(Z)$, interpreted as the mean of the $\alpha$-tail of the distribution of $Z$---the average over the worst $\alpha$-fraction of outcomes. The CVaR certainty equivalent is coherent and admits the dual representation

The parameter $\alpha$ interpolates between the two previous cases:

Another important case, already discussed in Chapter 1, is the entropic certainty equivalent

This equivalent is concave but not coherent. The dual representation is

where $D_{\mathrm{KL}}(Q \,\|\, P_Z) = \EE_Q\!\left[\log \frac{\diff Q}{\diff P_Z}\right]$ is the Kullback--Leibler divergence. The parameter $\gamma$ controls the degree of risk aversion and interpolates between risk neutrality and worst case. In particular, $\gamma \to 0$ implies $\eE_\gamma(Z) \to \EE[Z]$, while $\gamma \to \infty$ implies $\eE_\gamma(Z) \to \operatorname{ess\,inf}\, Z$.

It is worth noting here that the Kreps--Porteus expectation $\kK(Z) \coloneq (\EE[Z^{1-\gamma}])^{1/(1-\gamma)}$ is not a certainty equivalent, at least according to our definition. While monotonicity holds, $\kK$ fails cash invariance. This is, in essence, why Bellman and policy operators based around Epstein--Zin preferences often fail to be contractions. We discuss Kreps--Porteus expectations again in Section 7.3.4.

7.2.4.3Continuity¶

Let $\eE$ be a certainty equivalent on $L_\infty = L_\infty(\Omega, \fF, \PP)$. We call $\eE$ continuous if, given any uniformly bounded sequence $(Z_n)_{n \in \NN}$ in $L_\infty$ and any $Z \in L_\infty$, we have

In the definition above, uniform boundedness means that there exists an $M < \infty$ with $|Z_n| \leq M$ almost surely for all $n$. Continuity will be useful for the optimality theory developed below.

7.2.5MDPs with Certainty Equivalents¶

We first set up a standard MDP framework with aggregator based on mathematical expectation and then replace the expectation with a general certainty equivalent, showing that the fundamental optimality results carry over when the certainty equivalent is continuous.

7.2.5.1MDP Framework¶

We begin by setting up a basic MDP framework that can then be adapted to add risk preferences. To this end, let $\Xsf$ and $\Asf$ be arbitrary metric spaces and let $\Gamma$ be a nonempty correspondence from $\Xsf$ to $\Asf$. Let $\Gsf = \setntn{(x,a)\in \Xsf \times \Asf}{a \in \Gamma(x)}$. Consider an RDP with feasible correspondence $\Gamma$ and aggregator

Here $\xi$ is a random element that takes values in a metric space $\Zsf$ and has distribution $\phi$, $f$ is a measurable function from $\Gsf \times \Zsf$ to $\Xsf$, and $r$ is a measurable function from $\Gsf$ to $\RR$. The discount factor $\beta$ obeys $0 \leq \beta < 1$. The value space is set to $b\Xsf$.

We can treat optimality and convergence of algorithms for the associated RDP $(\Gamma, b\Xsf, B)$ using Proposition 6.1.7 from Chapter 6. Here, however, we’ll extend the model to use arbitrary certainty equivalents in place of $\EE$. Results for $\EE$ will be a special case. The next section gives details.

7.2.5.2Certainty Equivalents for MDPs¶

Let’s consider replacing the expectation in (7.61), which corresponds to risk-neutrality over continuation values, with an arbitrary certainty equivalent $\eE$. The aggregator is now

Other primitives are left unchanged. The value space continues to be $b\Xsf$. We assume throughout that $(x,a) \mapsto \eE [v(f(x,a,\xi))]$ is measurable on $\Gsf$. We consider the RDP $(\Gamma, b\Xsf, B_{\eE})$.

7.3Applications¶

We now apply the RDP optimality theory developed in Section 7.2 to a range of dynamic programming problems. In Section 7.3.1 we revisit the optimal savings problem, this time with utility unbounded above, and verify the conditions of our weighted contraction results. We then study irreversible investment under risk neutrality (Section 7.3.2.1), risk aversion (Section 7.3.2.2), and ambiguity aversion (Section 7.3.3).

7.3.1Optimal Savings with Utility Unbounded Above¶

Here we again consider the optimal savings model from Section 1.3, but without the boundedness restriction on $u$. In particular, we assume that $u$ is continuous, nonnegative, and increasing, and that

for some $\delta \in (\beta, 1)$. Here $(\hat W_t)$ is defined recursively via $\hat W_{t+1} = R \hat W_t + Y_{t+1}$ with $\hat W_0 = w$ and $(Y_t) \iidsim \phi$. We can think of $(\hat W_t)$ as an upper bound process for wealth, achieved when consumption is always zero. As before, $\phi$ is a continuous density on $\RR_+$. Our aim is to provide conditions under which the conclusions of Proposition 7.2.5 apply.

We set $V = b_\ell \RR_+$, $\Gamma(w) = [0, w]$, and

The results above imply that Assumption 7.2.7 and Assumption 7.2.4 both hold. As a result, the conclusions of Proposition 7.2.5 apply. For example, the value function $\vmax$ exists, is an element of $b_\ell c \RR_+$, and satisfies

for all $w \geq 0$. Moreover, VFI, OPI, and HPI all converge.

In general, OPI and VFI are the easiest to implement. Figure 7.1 illustrates the runtime of OPI as a function of $m$ for this model with CRRA utility $u(c) = (c^{1-\gamma} - 1)/(1-\gamma)$ and $\gamma = 0.5$. The runtime of VFI is shown as a horizontal line. Since VFI is the special case of OPI with $m=1$, the leftmost point of the OPI curve coincides with the VFI runtime. The minimum is attained near $m = 40$, where OPI runs roughly six times faster than VFI. Runtime then rises linearly in $m$. The key message is that OPI dominates VFI over a wide range of $m$.

7.3.2Irreversible Investment¶

We begin with the risk-neutral case in Section 7.3.2.1, where standard contraction arguments apply. In Section 7.3.2.2 we introduce risk aversion by replacing mathematical expectation with a certainty equivalent, using the framework developed in Section 7.2.4.

7.3.2.1The Risk-Neutral Case¶

Let’s begin with a canonical firm problem with irreversible investment, where the Bellman equation is

Here $k \in \RR_+$ is capital stock, $i \in \RR_+$ is investment, $\beta \in (0,1)$ is the discount factor, $\delta \in (0,1)$ is a depreciation rate, $f$ is a production function, and $z \in \RR^m$ is an exogenous state vector. The feasible correspondence is defined by $\Gamma(k,z) = [0, \theta f(k, z)]$, where $\theta > 0$ is a borrowing constraint parameter. The state process evolves according to

Each $\xi_t$ takes values in a metric space $\Zsf$, the distribution $\phi$ is an element of $\pP(\Zsf)$, and $g \colon \RR^m \times \Zsf \to \RR^m$.

Some comments are in order. First, to simplify the presentation, we’ve set the output price to unity, so that $f(k,z)$ is both output and revenue. This can easily be modified. Second, the boundedness restriction on $f$ is not automatically satisfied in many cases but greatly simplifies the analysis. In terms of quantitative applications, the cost is not large. For example, $f(k,z)=zk^\alpha$ can be replaced with $f(k,z) = \min\{zk^\alpha, y\}$ for large $y$. If $y$ is very large then the impact on choices and values is negligible.

This firm problem is called an irreversible investment model because $i$ is required to be nonnegative. To frame this problem as an RDP, we set $V = b\Xsf$, where $\Xsf \coloneq \RR_+ \times \RR^m$, and

The set of policies $\Sigma$ is all measurable maps from $\Xsf$ to $\RR_+$ satisfying the feasibility constraint.

Figure 7.2 shows an example computation, plotting optimal investment as a function of capital for two productivity levels. The plot also compares this irreversible case ($i \geq 0$) with a reversible benchmark where the firm can also disinvest ($i \geq -(1 - \delta)k$). At low capital, both firms invest identically. At high capital, with the same level of productivity, the reversible firm disinvests, while the irreversible firm sets $i$ to the lower bound constraint of zero. Importantly, for intermediate levels of capital, the reversible firm invests more aggressively, knowing that it can sell capital later if productivity drops. The irreversible firm faces a higher effective cost of capital due to the option value of waiting and targets a lower stock.^[3]

Figure 7.3 shows simulated paths for both firms facing identical productivity shocks. The reversible firm tracks productivity more closely, boosting capital during good times and shedding it during downturns. The irreversible firm adjusts sluggishly on the downside, since it can only reduce capital through depreciation.

7.3.2.2Adding Risk Aversion¶

As discussed in Section 1.1.3, actual firm behavior often deviates from the risk-neutral benchmark attained under the assumption of frictionless complete markets. Here we extend the model from Section 7.3.2.1 in order to discuss this case. We swap the Bellman equation from Section 7.3.2.1 with

The only change is that mathematical expectation has been replaced with a certainty equivalent $\eE$. The term $\xi$ should be understood as a random element on $\Zsf$ with distribution $\phi$. We assume that the map $(k, z, i) \mapsto \eE [v(i + (1-\delta) k, g(z, \xi))]$ is measurable on $\Gsf$ for all $v \in b\Xsf$.

We can set this model up as an RDP by taking $V = b\Xsf$, $\Gamma(k,z) = [0, \theta f(k,z)]$, and