7 Nonlinear Valuation

Dynamic programs are optimization problems where the objective to be maximized is lifetime value. As such, one key topic is how to combine a sequence of rewards into a corresponding lifetime value. So far we have considered linear valuation based on summation over expected discounted rewards, using either constant discount rates (Chapter 1--Chapter 5) or state-dependent discounting (Chapter 6). In this chapter, we consider extensions, where lifetime value is computed from a recursion over the reward sequence instead of a discounted sum. This “recursive preference” approach permits far more general specifications of lifetime value, and is becoming increasingly popular in economics, finance, and computer science (see, e.g., Section 6.4).

This chapter focuses purely on valuation (i.e., combining reward sequences into lifetime values), rather than optimization. Later, in Chapter 8, we will show how to maximize lifetime value in settings where recursive preferences are adopted.

Throughout this chapter, the symbol $\Xsf$ always represents a finite set.

7.1Beyond Contraction Maps¶

The most natural way to express lifetime value in recursive preference environments is as a fixed point of a (typically nonlinear) operator. One challenge is that some recursive preference specifications induce operators that fail to be contractions. For this reason, we now invest in additional fixed point theory. All of this theory concerns order preserving maps, since the operators we consider always inherit monotonicity from underlying preferences.

7.1.1Knaster--Tarski for Function Space¶

If you try to draw an increasing function that maps $[0,1]$ to itself without touching the 45-degree line, you will find it impossible. Below we state a famous fixed-point theorem due to Bronislaw Knaster (1893--1980) and Alfred Tarski (1901--1983) that generalizes this idea. In the statement, $\Xsf$ is a finite set and $V \coloneq [v_1, v_2]$, where $v_1, v_2$ are functions in $\RR^\Xsf$ with $v_1 \leq v_2$.

Unlike, say, the fixed-point theorem of Banach (Section 1.2.2.3), Theorem 7.1.1 only yields existence. Uniqueness does not hold in general, as you can easily confirm by sketching the one-dimensional case or completing the following exercise.

7.1.2Concavity, Convexity, and Stability¶

In this section, we study sufficient conditions for global stability that replace contractivity with shape properties such as concavity and monotonicity. To build intuition, we start with the one-dimensional case and show how these properties can be combined to achieve stability. Readers focused on results can safely skip to Section 7.1.2.2.

7.1.2.1The One-Dimensional Case¶

In Section 1.2.3.2, we showed that concavity and monotonicity can yield global stability for the Solow--Swan model. Here is a more general result.

Figure Figure 7.1 gives one example, where $g(x) = 1 + \sqrt{x}/2$. The conditions of Proposition 7.1.2 hold because, given any $x > 0$, we can find an $a$ in $(0,x)$ that gets mapped strictly up (i.e., $g(a)$ is above the 45-degree line) and a point $b > x$ that gets mapped down (i.e., $g(b)$ is below the 45-degree line).

7.1.2.2The Multidimensional Case¶

Proposition 7.1.2 extends to multiple dimensions. In this section, we present a multidimensional version that covers both convex and concave functions.

To state our result, we extend the definition of convexity and concavity to vector-valued self-maps. The definitions mirror those for scalar-valued functions: A self-map $T$ on a convex subset $D$ of $\RR^\Xsf$ is called convex if

and concave if

Here $\leq$ is, as usual, the pointwise order.

We are now ready to state our next fixed-point result, which was first proved in an infinite-dimensional setting by Du (1990). In the statement, $\Xsf$ is a finite set, $V \coloneq [v_1, v_2]$ is a nonempty order interval in $(\RR^\Xsf, \leq)$, and $T$ is a self-map on $V$.

Conditions (i) and (ii) are similar -- in fact (ii) holds whenever (i) holds, so (ii) is the weaker (but slightly more complicated) condition. Conditions (iii) and (iv) are similar in the same sense. Figure Figure 7.2 illustrates the convex and the concave versions of the result in one dimension. We encourage you to sketch your own variations to understand the roles that different conditions play.

A full proof of Theorem 7.1.3 can be found in Du (1990) or Theorem 2.1.2 and Corollary 2.1.1 of Zhang (2012). In our setting, existence follows from the Knaster--Tarski theorem. We prove uniqueness.

7.1.3A Power-Transformed Affine Equation¶

Du’s theorem provides conditions under which concave or convex order preserving self-maps on order intervals attain global stability. In this section we study maps of this type that have additional structure. While this additional structure is restrictive, it allows us to obtain global stability on unbounded subsets rather than order intervals.

To begin, let $\Xsf$ be a finite set and consider the equation

where $\theta$ is a nonzero parameter, $A \in \lopx$ with $A \geq 0$, $V = (0, \infty)^\Xsf$, and $h \in V$. This system reduces to the affine model studied in Lemma 6.1.4 when $\theta = 1$.

To analyze (7.7), we introduce the self-map

Continuing to assume that $h \gg 0$ and $A$ is a positive linear operator, we can use Du’s theorem to establish the next result (which generalizes Lemma 6.1.4 on page  ).

The key to proving (i) implies (ii) is that $G$ is order preserving and either convex or concave, depending on the value of $\theta$. The remaining conditions in Du’s theorem are established over order intervals using $\rho(A)^{1/\theta} < 1$. By applying an approximation argument, global stability is extended from order intervals to all of $V$. Some of these details are contained in the following exercises and a full proof can be found in Stachurski et al. (2022).

Let

7.2Recursive Preferences¶

In this section, we compute lifetime values associated with given reward processes in settings that involve nonlinear recursions. These nonlinear recursions are called recursive preferences. We will show how some common specifications of recursive preferences can be translated into lifetime valuations via the fixed-point methods introduced in Chapter 2 and Section 7.1.

7.2.1Motivation: Optimal Savings¶

We motivate recursive preference models by analyzing consumption decisions.

7.2.1.1A Recursive View of a Standard Model¶

The time additive model of valuation in Section 3.2.2.3 can be studied from a purely recursive point of view. As a starting point, we state that the value $V_t$ of current and future consumption is defined at each point in time $t$ by the recursion

The random variables $V_t$ and $V_{t+1}$ are the unknown objects in this expression. The expectation $\EE_t$ conditions on $X_0, \ldots, X_t$ and $C_t = c(X_t)$. The process $(X_t)_{t \geq 0}$ is $P$-Markov.

Since consumption is a function of $(X_t)_{t \geq 0}$ and knowledge of the current state $X_t$ is sufficient to forecast future values (by the Markov property), it is natural to guess that $V_t$ will depend on the Markov chain only through $X_t$. Hence we guess there is a solution of (7.15) takes the form $V_t = v(X_t)$ for some $v \in \RR^\Xsf$.

(Here $v$ is an ansatz, meaning “educated guess.” First we guess the form of a solution and then we try to verify that the guess is correct. So long as we carry out the second step, starting with a guess brings no loss of rigor.)

Under this conjecture, (7.15) can be rewritten as $v(X_t) = u(c(X_t)) + \beta \EE_t v(X_{t+1})$. Conditioning on $X_t = x$ and setting $r \coloneq u \circ c$, this becomes

In vector form, we get $v = r + \beta P v$. From the Neumann series lemma, the solution is $v^* = (I - \beta P)^{-1} r$, which is identical to (3.40).

In summary, (7.15) and the sequential representation (3.39) specify the same lifetime value for consumption paths.

While the recursive formulation in (7.15) now seems redundant, since it produces the same specification that we obtained from the sequential approach, the recursive set up gives us a formula to build on, and hence a pathway to overcoming limitations of the time additive approach. Most of the rest of this chapter will be focused on this agenda.

Pursuing this agenda will produce preferences over consumption paths where the sequential approach has no natural counterpart. This occurs when current value $V_t$ is nonlinear in current rewards and continuation values (unlike the linear specification (7.15)). Such specifications are called recursive preferences. When dealing with recursive preference models, the lack of a sequential counterpart means that we are forced to proceed recursively.

7.2.1.2Limitations of Time Additive Preferences¶

In the previous section, we discussed how the time additive preference specification

also called the discounted expected utility model, can be framed recursively, and how this provides a pathway to go beyond the time additive specification. We are motivated to do so because the time additive specification has been rejected by experimental and observational data in many settings.

In this section, we highlight some of the limitations of time additive preferences. While our discussion is only brief, more background and a list of references can be found in Section 7.4.

One issue with (7.18) is the assumption of a constant positive discount rate, which has been refuted by a long list of empirical studies. This issue was discussed in Section 6.4.

Another limitation of time additive preferences is that agents are risk-neutral in future utility (see, e.g., (7.15), where current value depends linearly on future value). Although risk aversion over consumption can be built in through curvature of $u$, this same curvature also determines the elasticity of intertemporal substitution, meaning that the two aspects of preferences cannot be separated. We elaborate on this point in Section 7.3.1.4.

A third issue with time additivity is that agents with such preferences are indifferent to any variation in the joint distribution of rewards that leaves marginal distributions unchanged. To get a sense of what this means, suppose you accept a new job and will be employed by this firm for the rest of your life. Your daily consumption will be entirely determined by your daily wage. Your boss offers you two options:

• Your boss will flip a coin at the start of your first day on the job. If the coin is heads, you will receive $10,000 a day for the rest of your life. If the coin is tails, you will receive $1 per day for the rest of your life.

• Your boss will flip a coin at the start of every day. If the coin is heads, you will receive $10,000. If the coin is tails, you will receive $1.

If you have a strict preference between options A and B, then your choice cannot be rationalized with time additive preferences.

To see why, let $\phi$ be a probability distribution that represents the lottery just described, putting mass 0.5 on 10,000 and mass 0.5 on 1. Under option A, consumption $(C_t)_{t \geq 1}$ is given by $C_t = C_1$ for all $t$, where $C_1 \sim \phi$. Under option B, consumption $(C_t)_{t \geq 1}$ is an iid sequence drawn from $\phi$. Either way, lifetime utility is

where $\bar u \coloneq \EE u(C_1) = u(1)/2 + u(10,000)/2$.

The critical part of this argument is the passing of expectations through the sum, which uses time additivity . The implication is that lifetime utility depends only on the marginal distribution of each $C_t$, rather than on the joint distribution of the stochastic process $(C_t)_{t \geq 0}$.

7.2.2Risk-Sensitive Preferences¶

Having motivated recursive preferences, let’s turn to our first example: risk-sensitive preferences. For the consumption problem described in Section 7.2.1.1, imposing risk-sensitive preferences means replacing the recursion $v = r + \beta Pv$ for $v$ with

As before, $r(x) = u(c(x))$ represents current utility when the current state is $x$. The parameter $\theta$ is a nonzero constant in $\RR$.

In (7.20), the transform $f(v) = \exp(\theta v)$ is applied to $v$ before expectation is taken. After the expectation is computed, the transform is undone via $f^{-1}(v) = (1/\theta) \ln (v)$. We will show that the agent can be either risk-averse or risk-loving with respect to future outcomes, depending on the value of $\theta$.

7.2.2.1Lifetime Utility¶

We understand the functional equation (7.20) as “defining” lifetime utility under risk-sensitive preferences. A function $v$ solving (7.20) gives a lifetime valuation $v(x)$ to each $x \in \Xsf$, with the interpretation that $v(x)$ is lifetime utility conditional on initial state $x$. This definition of lifetime value is by analogy to the time additive case studied in Section 7.2.1.1, where the function $v$ solving $v = r + \beta P v$ measures lifetime utility from each initial state.

In the previous paragraph we wrote “defining” in scare quotes because we can’t be sure we have a definition at this point. Just because we write down a recursive expression for lifetime utility doesn’t mean that corresponding lifetime utility is actually well defined. (For example, we can happily write down the recursive vector equation $v = v + \1$ but no vector $v$ solving this equation exists.) One aim of this chapter is to provide conditions under which recursions like (7.20) have solutions.

Another issue is uniqueness. Suppose that (7.20) has many solutions. In this case the predictions of the utility model are ambiguous. Our perspective is that the recursive preference specification (7.20) is not correctly formulated unless existence and uniqueness hold. We return to this point in Section 7.2.2.3.

One final comment: even if we can find a $v$ that solves (7.20), the nonlinearities introduced by risk sensitivity imply that there will be no neat sequential representation analogous to $v(x) = \EE_x \sum_t \beta^t u(C_t)$ from the time additive case. (This connects to Remark 7.2.1, where we discuss recursive preference terminology.)

7.2.2.2Risk-Adjusted Expectation¶

We want to understand the “expectation-like” expression on the right hand side of (7.20) that replaces the ordinary conditional expectation $\sum_{x'} v(x') P(x, x')$ from the time additive case. To this end, we define, for arbitrary random variable $\xi$ and nonzero $\theta \in \RR$,

The value $\eE_\theta[\xi]$ is called the entropic risk-adjusted expectation of $\xi$ given $\theta$.

The key idea behind the entropic risk-adjusted expectation is that decreasing $\theta$ lowers appetite for risk and increasing $\theta$ does the opposite.

Expression (7.22) shows that, for the Gaussian case, $\eE_\theta[ \xi]$ equals the mean plus a term that penalizes variance when $\theta < 0$ and rewards it when $\theta > 0$.

More generally, we have the following result.

7.2.2.3Existence and Uniqueness¶

Let’s return to investigating lifetime utility under risk-sensitive preferences. To this end, we introduce the risk-sensitive Koopmans operator $K_\theta$ on $\RR^\Xsf$ via

Evidently, for given nonzero $\theta$, a function $v \in \RR^\Xsf$ solves the risk-sensitive preference lifetime utility specification (7.20) if and only if $v$ is a fixed point of $K_\theta$. This explains the significance of the following result:

We postpone a proof of Proposition 7.2.2 because we will prove a more general result in Section 7.3.2.2. For now we note the following implications.

1. For each nonzero $\theta$, lifetime utility is both well-defined and uniquely defined for risk-sensitive preferences (i.e., (7.20) has a unique solution).

2. The unique solution, denoted henceforth by $v^*$, can be computed by successive approximation using $K_\theta$.

7.2.2.4The Gaussian Case¶

As a tractable case, let’s suppose that $r(x) = x$ and that $X_{t+1} = \rho X_t + \sigma W_{t+1}$ where $(W_t)_{t \geq 1}$ is iid and standard normal. Here $|\rho| < 1$ and $\sigma \geq 0$ controls volatility of the state. Rather than discretizing the state process, we leave it as continuous and proceed by hand.

In this setting, the functional equation (7.20) for $v$ becomes

for each $x \in \Xsf$, where $W$ is standard normal.

Since $\rho x + \sigma W$ is Gaussian, the expression (7.22) for the risk-adjusted expectation of a normal random variable leads us to conjecture that the solution $v$ will be affine, i.e., $v(x) = a x + b$ for some $a, b \in \RR$. This conjecture turns out to be correct:

We can see that, under the stated assumptions, lifetime value $v$ is increasing in the state variable $x$. However, impacts of the parameters generally depend on $\theta$. For example, if $\theta > 0$, increasing $\sigma$ shifts up lifetime utility. If $\theta < 0$, then lifetime value decreases with $\sigma$. This is as we expect: Lifetime utility is affected positively or negatively by volatility, depending on whether or not the agent is risk averse or risk loving.

Figure Figure 7.3 shows the true solution $v(x) = ax + b$ to the risk-sensitive lifetime utility model, as well as an approximate fixed point from a discrete approximation. The discrete approximation is computed by applying successive approximation to $K_\theta$ after discretizing the state process via Tauchen’s method. The parameters and discretization are shown in Listing 1.

1
2
3
4
5
6
7
8
9
10
11
12
13
using LinearAlgebra, QuantEcon

function create_rs_utility_model(;
        n=180,      # size of state space
        β=0.95,     # time discount factor
        ρ=0.96,     # correlation coef in AR(1)
        σ=0.1,      # volatility
        θ=-1.0)     # risk aversion
    mc = tauchen(n, ρ, σ, 0, 10)  # n_std = 10
    x_vals, P = mc.state_values, mc.p 
    r = x_vals      # special case u(c(x)) = x
    return (; β, θ, ρ, σ, r, x_vals, P)
end

7.2.3Epstein--Zin Preferences¶

One of the most popular specifications of recursive preferences in quantitative research is Epstein--Zin utility.^[1] This class of preferences has been used to study asset pricing, business cycles, monetary policy, fiscal policy, optimal taxation, climate policy, pension plans, and other topics. In this section, we introduce the Epstein--Zin specification and discuss how to solve it. We will see that the specification, while highly nonlinear, is nonetheless well behaved.

7.2.3.1Specification¶

With Epstein--Zin preferences, the relationship $V_t = u(C_t) + \beta \EE_t V_{t+1}$ is replaced by

where $\gamma$, $\alpha$ are nonzero parameters and $\beta \in (0,1)$. As for risk-sensitive preferences, lack of time additivity implies that there is no neat sequential representation for lifetime value. As a result, we must work directly with the recursive expression (7.29).

Assume as before that $C_t = c(X_t)$, where $c \in \RR_+^\Xsf$ and $(X_t)_{t \geq 0}$ is $P$-Markov on finite set $\Xsf$. We conjecture a solution of the form $V_t = v(X_t)$ for some $v \in V \coloneq \RR_+^\Xsf$. Under this conjecture, the Epstein--Zin Koopmans operator corresponding to (7.29) is

As will be discussed further in Section 7.3.1.1, the parameter $\gamma$ governs risk aversion with respect to temporal gambles (where outcomes are resolved in the next period), while $\beta$ controls impatience and $\alpha$ parametrizes the intertemporal elasticity of substitution. The fact that all three parameters have distinct effects helps fit data. For example, see Tallarini Jr (2000) and Barillas et al. (2009).

An important question is whether Epstein--Zin preferences are well defined. In particular, what conditions do we need on primitives such that the Koopmans operator $K$ in (7.30) has a unique fixed point?

7.2.3.2Properties of the Koopmans Operator¶

To address this question we rewrite (7.30) in vector form as

where $h \in \RR^\Xsf$. This is equivalent to (7.30) when $h = (1-\beta) c^\alpha$. To avoid fractional powers of negative numbers, we assume throughout that $h \geq 0$.

The set $V$ is called the interior of the positive cone of $\RR^\Xsf$.

The operator $K$ is difficult to work with for two reasons. First, linear and nonlinear transformations are intertwined. Second, there are several cases for the parameters that we need to handle in order to understand stability. Nonetheless, by applying a smooth transformation, we will find it easy to show that the Epstein--Zin Koopmans operator $K$ is globally stable under mild conditions. In particular,

A proof of Proposition 7.2.3 is provided in Section 7.2.3.3.

Proposition 7.2.3 implies that Epstein--Zin utility is well-defined under the stated conditions and, moreover, that the solution can be computed via successive approximation on $K$. Listing 2 provides code for performing this operation. Figure Figure 7.4 shows convergence of the sequence of iterates to the fixed point $v^*$, under the parameters in Listing 2, given an initial condition $v_0$. The figure plots every 10th iterate, repeated 100 times.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
include("s_approx.jl")
using LinearAlgebra, QuantEcon

function create_ez_utility_model(;
        n=200,      # size of state space
        ρ=0.96,     # correlation coef in AR(1)
        σ=0.1,      # volatility
        β=0.99,     # time discount factor
        α=0.75,     # EIS parameter
        γ=-2.0)     # risk aversion parameter

    mc = tauchen(n, ρ, σ, 0, 5) 
    x_vals, P = mc.state_values, mc.p 
    c = exp.(x_vals)      

    return (; β, ρ, σ, α, γ, c, x_vals, P)
end

function K(v, model)
    (; β, ρ, σ, α, γ, c, x_vals, P) = model

    R = (P * (v.^γ)).^(1/γ)
    return ((1 - β) * c.^α + β * R.^α).^(1/α)
end

function compute_ez_utility(model)
    v_init = ones(length(model.x_vals))
    v_star = successive_approx(v -> K(v, model), 
                               v_init, 
                               tolerance=1e-10)
    return v_star
end

7.2.3.3Proof of the Stability Result¶

We prove Proposition 7.2.3 by

1. introducing an operator $\hat K$ obtained from $K$ via a smooth transformation,

2. proving that $(\hat V, \hat K)$ and $(V, K)$ are topologically conjugate, and

3. obtaining conditions under which $\hat K$ is globally stable on $V$.

Throughout this section, the assumptions of Proposition 7.2.3 are in force.

To begin we define $\hat K$ via

The operator $\hat K$ is simpler to work with than $K$ because it unifies $\alpha, \gamma$ into a single parameter $\theta$ and decomposes the Epstein--Zin update rule into two parts: a linear map $P$ and a separate nonlinear component.

7.2.3.4Why Not Use Contractivity?¶

While we can consider studying stability of $\hat K$ using contraction arguments, this approach fails under useful parameterizations. To illustrate, suppose that $\Xsf = \{x_1\}$. Then $h$ is a constant, $P$ is the identity, $v$ is a scalar and $\hat K v = F(v)$ with $F(v) = \left\{ h + \beta v^{1/\theta} \right\}^\theta$, as shown in Figure Figure 7.5. Here $\theta = 5$, $h=0.5$ and $\beta=0.5$. We see that $\hat K$ has infinite slope at zero, so the contraction property fails.^[2]

7.3General Representations¶

We have discussed two well-known examples of recursive preferences. In this section we build a general representation. While various constructions can be found in the decision theory literature, many are not well suited to quantitative work. Here we give a relatively parsimonious operator-theoretic definition.

7.3.1Koopmans Operators¶

In Section 7.2.2.3 and Section 7.2.3.1 we met risk-sensitive and Epstein--Zin Koopmans operators respectively. In this section, we provide a general definition of a Koopmans operator that will contain these two examples as special cases.

We begin by outlining structure that can be combined to generate Koopmans operators in a Markov environment. The two key components are an aggregation function and a certainty equivalent operator. We then build Koopmans operators from these primitives and connect them to applications. In every setting we consider, lifetime value is identified with the unique fixed point of the Koopmans operator (whenever it exists).

7.3.1.1Certainty Equivalents¶

The first primitive we consider is a generalization of conditional expectations: Given $V \subset \RR^\Xsf$, we define a certainty equivalent operator on $V$ to be a self-map $R$ on $V$ such that

1. $R$ is order preserving on $V$ and

2. all constants are fixed under $R$ (i.e., $R\, ( \lambda \1) = \lambda \1$ for all $\lambda \in \RR$ with $\lambda \1 \in V$).

The next example is nonlinear. It treats the risk-adjusted expectation that appears in risk-sensitive preferences.

The set of certainty equivalent operators on $\RR^\Xsf$ is invariant under convex combinations, as the next exercise asks you to confirm.

7.3.1.2Properties¶

Let $V$ be a convex cone in $\RR^\Xsf$. A certainty equivalent operator $R$ on $V$ is called

• positive homogeneous on $V$ if $R(\lambda v) = \lambda Rv$ for all $v \in V$ and $\lambda \geq 0$ with $\lambda v \in V$,

• superadditive on $V$ if $R(v + w) \geq Rv + Rw$ for all $v, w \in V$ with $v + w \in V$,

• subadditive on $V$ if $R(v + w) \leq Rv + Rw$ for all $v, w \in V$ with $v + w \in V$,

• constant-subadditive on $V$ if $R (v + \lambda \1) \leq R v + \lambda \1$ for all $v \in V$ and $\lambda \geq 0$ with $v + \lambda \1 \in V$.

In some instances, a certainty equivalent operator is either convex or concave in the sense of Section 7.1.2.2.

Combining Exercise 7.29 and Example 7.3.5, we have proved

Later we will combine Lemma 7.3.1 with the fixed-point results for convex and concave operators in Section 7.1.2.2 to establish existence and uniqueness of lifetime values for certain kinds of Koopmans operators.

7.3.1.3Monotonicity¶

Let $\Xsf$ be partially ordered and let $i\RR^\Xsf$ be the set of increasing functions in $\RR^\Xsf$. Let $V$ be such that $i\RR^\Xsf \subset V \subset \RR^\Xsf$ and let $R$ be a certainty equivalent on $V$. We call $R$ monotone increasing if $R$ is invariant on $i\RR^\Xsf$. This extends the terminology in Section 3.2.1.3, where we applied it to Markov operators (cf., Exercise 3.16). The concept of monotone increasing certainty equivalent operators is connected to outcomes where lifetime preferences are increasing in the state.

7.3.1.4Aggregation¶

We mentioned that Koopmans operators are typically constructed by combining a certainty equivalent operator and an aggregation function. Let’s now discuss the second of these components.

Given $V \subset \RR^\Xsf$, an aggregator $A$ on $V$ is a map $A$ from $\Xsf \times \RR$ to $\RR$ such that

1. $w(x) = A(x, v(x))$ is in $V$ whenever $v \in V$ and

2. $y \mapsto A(x, y)$ is increasing for all $x \in \Xsf$.

Intuitively, an aggregator combines current state and continuation values to measure lifetime value.

Common types of aggregators include the

• Leontief aggregator $A_{\textsc{min}}(x,y) = \min\{ r(x), \beta y \}$ with $r \in \RR^\Xsf$ and $\beta \geq 0$,

• Uzawa aggregator $A_{\textsc{uzawa}}(x,y) = r(x) + b(x) y$ with $r \in \RR^\Xsf$ and $b \in \RR^\Xsf_+$, and

• CES aggregator $A_{\textsc{ces}}(x, y) = \{r(x)^\alpha + \beta y^\alpha\}^{1/\alpha}$ with $r \in (0,\infty)^\Xsf$, $\beta \geq 0$ and $\alpha \not= 0$.

Here CES stands for “constant elasticity of substitution.” An important special case of both the CES and Uzawa aggregators is the

• additive aggregator $A_{\textsc{add}}(x,y) = r(x) + \beta y$ with $r \in \RR^\Xsf$ and $\beta \geq 0$.

From these basic types we can also build composite aggregators. For example, we might consider a CES-Uzawa aggregator of the form $A(x, y) = \{r(x)^\alpha + b(x) y^\alpha\}^{1/\alpha}$ with $r, b \in \RR^\Xsf$, $b \geq 0$ and $\alpha \not= 0$. As we will see in Section 7.3.3.3, the CES-Uzawa aggregator can be used to construct models with both Epstein--Zin utility and state-dependent discounting (as in, say, Albuquerque et al. (2016) or Schorfheide et al. (2018).)

7.3.1.5Building Koopmans Operators¶

We are now ready to build Koopmans operators by combining certainty equivalents and aggregators. Given $V \subset \RR^\Xsf$, we call a self-map $K$ on $V$ a Koopmans operator if

for some aggregator $A$ and certainty equivalent operator $R$ on $V$. The expression in (7.46) means that $(Kv)(x) = A(x, (Rv)(x))$ at $v \in V$ and $x \in \Xsf$.

It is generally appropriate to suppose that a uniform increase in continuation values will increase current value. This property holds for $K$ in (7.46). In particular, it follows from the definitions of $A$ and $R$ that $K$ is an order preserving self-map on $V$.

7.3.1.6Comments on CES Aggregation¶

The CES aggregator is so-named because, in a static utility maximization problem where $c$ and $y$ are two goods and utility is $U(c,y) = ((1-\beta) c^\alpha + \beta y^\alpha)^{1/\alpha}$, the elasticity of substitution is constant and given by $1/(1-\alpha)$. In the present setting, where aggregation is across time, $1/(1-\alpha)$ is usually called the elasticity of intertemporal substitution (EIS). The next exercise explains.

The fact that EIS $= 1/(1-\alpha)$ under the CES aggregator is significant because the EIS can be measured from data using regression and other techniques. While estimates vary significantly, the detailed meta-analysis by Havranek et al. (2015) suggests 0.5 as a plausible average value for international studies, with rich countries tending slightly higher. Basu & Bundick (2017) use 0.8 when calibrating to US data. Under these estimates, the relationship EIS $= 1/(1-\alpha)$ implies a value for $\alpha$ between -1.0 and -0.25.

7.3.1.7Lifetime Value¶

In Section 7.3.1.5 we constructed a generic Koopmans operator using an aggregator and a certainty equivalent operator. In this section, we connect this Koopmans operator to lifetime values and discuss the significance of global stability.

To begin, fix set $\Xsf$ and function class $V \subset \RR^\Xsf$. Let $K = A \circ R$ be a Koopmans operator for some aggregator $A$ and certainty equivalent operator $R$ on $V$. The lifetime value generated by $K$ is the unique fixed point of $K$ in $V$, whenever it exists. Given such a $v$, the value $v(x)$ is interpreted as lifetime value conditional on initial state $x$.

In many applications, our existence and uniqueness proofs for fixed points of $K$ will also establish global stability. For Koopmans operators, global stability has the following interpretation: for $w \in V$, $m \in \NN$ and $x \in \Xsf$, the value $(K^m w)(x)$ gives total finite-horizon utility over periods $0, \ldots, m$ under the preferences embedded in $K$, with initial state $x$ and terminal condition $w$. Hence global stability implies that, for any choice of terminal condition, finite-horizon utility converges to infinite-horizon utility as the time horizon converges to infinity. The next exercise helps to illustrate this point.

Exercise 7.33 confirms that, at least for the time additive case, global stability of $K$ is equivalent to the statement that a finite-horizon valuation with arbitrary terminal condition $w$ converges to the infinite-horizon valuation.

7.3.1.8Monotone Lifetime Values¶

Let $\Xsf = (\Xsf, \preceq)$ be partially ordered, let $i\RR^\Xsf$ be the set of increasing functions in $\RR^\Xsf$, and let $V$ be such that $i\RR^\Xsf \subset V \subset \RR^\Xsf$. Let $K$ be a Koopmans operator on $V$, so that $Kv = A \circ R$ for some aggregator $A$ and certainty equivalent operator $R$ on $V$. Suppose that $K$ has a unique fixed point $v^* \in V$. A natural question is: when is $v^*$ increasing in the state?

7.3.2A Blackwell-Type Condition¶

Let $R$ be a certainty equivalent operator on $V = \RR^\Xsf$ and let $A$ be an aggregator on $V$. Let $K$ be the Koopmans operator on $V$ defined by $(Kv)(x) = A(x, (Rv)(x))$. When $R$ is constant-subadditive, we can often establish global stability of $K$ on $V$ via a contraction mapping argument. This section gives details.

7.3.2.1Blackwell Aggregators¶

We call an aggregator $A$ on $V$ a Blackwell aggregator if there exists a $\beta \in (0,1)$ such that

for all $x \in \Xsf$, $y \in \RR$ and $\lambda \in \RR_+$.

The next proposition states conditions for global stability in settings where aggregators have the Blackwell property.

The stability of time additive preferences is a special case of Proposition 7.3.3.

7.3.2.2The Risk-Sensitive Case¶

We can now complete the proof of Proposition 7.2.2, which concerned global stability of the Koopmans operator generated by risk-sensitive preferences.

7.3.2.3Quantile Preferences¶

Consider a setting where $V = \RR^\Xsf$ and $K_\tau \coloneq A_{\textsc{add}} \circ R_\tau$. That is,

for $\beta \in (0,1)$, $\tau \in [0,1]$, $r \in \RR^\Xsf$ and $R_\tau$ as described in Exercise 7.22. Since $R_\tau$ is constant-subadditive (Exercise 7.25) and the additive aggregator is Blackwell, $K_\tau$ is globally stable (Proposition 7.3.3). The operator $K_\tau$ represents quantile preferences, as described in Castro & Galvao (2019) and other studies (see Section 7.4). The value $\tau$ parameterizes attitude to risk, a point we return to in Section 8.2.1.4.

7.3.3Uzawa Aggregation¶

Let’s consider the Koopmans operator $K = A_{\textsc{uzawa}} \circ R$, where $V$ is some subset of $\RR^\Xsf$ and $R$ is a certainty equivalent operator on $V$. In particular,

with $r, b \in \RR^\Xsf$ and $b \geq 0$. We are interested in conditions that imply $K$ is globally stable on $V$.

7.3.3.1The Case of Conditional Expectation¶

Let $V = \RR^\Xsf$ and suppose $R = P$ for some $P \in \mopx$, so that $R$ is ordinary conditional expectations. Then $K$ becomes $Kv = r + Lv$ where $L \in \lopx$ with $L(x,x') = b(x)P(x,x')$. By Exercise 1.20, $K$ is globally stable on $V$ whenever $\rho(L) < 1$.

This kind of structure arises when households derive utility from a consumption path while their discount factor fluctuates according to some state variable (see, e.g., Krusell & Smith (1998), Toda (2019), Cao (2020), and Hubmer et al. (2020)). For a given consumption path $(C_t)$, lifetime values takes the form

where $u$ is a flow utility function and $\{\beta_t\}$ is a discount factor process. Suppose $C_t = c(X_t)$ and $\beta_t = b(X_t)$ where $b \geq 0$ and $(X_t)$ is $P$-Markov for some $P \in \mopx$. Set $r \coloneq u \circ c$ and $L(x,x') \coloneq b(x) P(x,x')$. By Theorem 6.1.1, the condition $\rho(L) < 1$ implies that $v$ in (7.56) is the unique fixed point of $Kv = r + L v = r + b Pv$. In other words, lifetime value under (7.56) is the unique fixed point of the Koopmans operator when the aggregator is of Uzawa type and the certainty equivalent is conditional expectation.

How does this relate to optimization? Recall our discussion of state-dependent MDPs in Chapter 6. There, the policy operator $T_\sigma$ in (6.29) is a special case of (7.55) when the discount factor depends only on the current state and action.

With some additional requirements, the condition $\rho(L)<1$ is necessary as well as sufficient for existence of a unique fixed point for $Kv = r + Lv$. Indeed, if $b \gg 0$ and $P$ is irreducible, then $L$ is also irreducible and a positive linear operator. Applying Lemma 6.1.4, we see that $r \gg 0$ and $\rho(L) \geq 1$ implies $Kv = r + Lv$ has no fixed point in $V \coloneq \setntn{v \in \RR^\Xsf}{v \gg 0}$.

7.3.3.2Stability via Concavity¶

Now consider $Kv = r + b Rv$ from (7.55) when $R$ is not in $\mopx$. Here $b Rv$ is the pointwise product, so that $(bRv)(x) = b(x) (Rv)(x)$ for all $x$.

We cannot use Proposition 7.3.3 to prove stability of $K$ unless $b(x) < 1$ for all $x \in \Xsf$. Since this condition is rather strict, we now study weaker conditions that can be valid even when $b$ exceeds 1 in some states. Specifically, we consider

1. $b Rv \leq c + Lv$ for some $c \in \RR^\Xsf$ and $L \in \lopx$ with $\rho(L) < 1$.

2. $r \gg 0$ and $R$ is concave on $\RR^\Xsf_+$.

Let $V = [0, \bar v]$ where $\bar v \coloneq (I-L)^{-1}(r + c)$.

7.3.3.3Epstein--Zin Preferences with State-Dependent Discounting¶

Combining the CES-Uzawa aggregator $A(x, y) = \{r(x)^\alpha + b(x) y^\alpha\}^{1/\alpha}$ with the Kreps--Porteus certainty equivalent operator leads to the Koopmans operator

A fixed point of $K$ corresponds to lifetime value for an agent with Epstein--Zin preferences and state-dependent discounting. (Such set ups are used in research on macroeconomic dynamics and asset pricing -- see Section 7.4 for more details).

In what follows we take $V = (0, \infty)^\Xsf$ and assume that $h, b \in V$ and $P$ is irreducible.

To discuss stability of $K$ we introduce the operator $B \in \lopx$ defined by

To prove Proposition 7.3.5, we proceed as in Section 7.2.3.3, constructing a conjugate operator $\hat K$ and proving stability of the latter. For this purpose, we introduce

Also, let $\Phi$ be defined by $\Phi v = v^\gamma$.

7.4Chapter Notes¶

The time additive preference structure in Section 7.2.1 was popularized by Samuelson (1939), who built on earlier work by Fisher (1930) and Ramsey (1928). An axiomatic foundation was supplied by Koopmans (1960). Bastianello & Faro (2023) study the foundations of discounted expected utility (DEU) from a purely subjective framework.

Problems with the time additive DEU model include non-constant discounting, as discussed in Section 6.4, as well as sign effects (gains being discounted more than losses) and magnitude effects (small outcomes being discounted more than large ones. See, for example, Thaler (1981) and Benzion et al. (1989). A critical review of the time additive model and a list of many references can be found in Frederick et al. (2002).

In the stochastic setting, the time additive framework is a subset of the expected utility model (Von Neumann & Morgenstern (1944), Friedman (1956), Savage (1951)). There are many well documented departures from expected utility in experimental data. See the start of Andreoni & Sprenger (2012) and the article Ericson & Laibson (2019) for an introduction to the literature. An interesting historical discussion of time additive expected utility can be found in Becker et al. (1989).

(It is ironic that those most responsible for popularizing the time additive DEU framework have also been among the most critical. For example, Samuelson (1939) stated that it is “completely arbitrary” to assume that the DEU specification holds. He goes on to claim that, in the analysis of savings and consumption, it is “extremely doubtful whether we can learn much from considering such an economic man.” In addition, Stokey & Lucas (1989), whose work helped to standardize DEU as a methodology for quantitative analysis, argued in a separate study that DEU is attractive only because of its relative simplicity Lucas & Stokey, 1984.)

Do the departures from time additive expected utility found in experimental data actually matter for quantitative work? Evidence suggests that the answer is affirmative. In macroeconomics and asset pricing in particular, researchers increasingly use non-additive preferences in order to bring model outputs closer to the data. For example, many quantitative models of asset pricing rely heavily on Epstein--Zin preferences. Representative examples include Epstein & Zin (1991), Tallarini Jr (2000), Bansal & Yaron (2004), Hansen et al. (2008), Bansal et al. (2012), Schorfheide et al. (2018), and Groot et al. (2022). Alternative numerical solution methods are discussed in Pohl et al. (2018).

An excellent introduction to recursive preference models can be found in Backus et al. (2004). Our use of the term “Koopmans operator,” which is not entirely standard, honors early contributions by Nobel laureate Tjalling Koopmans on recursive preferences (see Koopmans (1960) and Koopmans et al. (1964)).

Theoretical properties of recursive preference models have been studied in many papers, including Epstein & Zin (1989), Weil (1990), Boyd (1990), Hansen & Scheinkman (2009), Marinacci & Montrucchio (2010), Bommier et al. (2017), Bloise & Vailakis (2018), Marinacci & Montrucchio (2019), Pohl et al. (2019), Balbus (2020), Borovička & Stachurski (2020), DeJarnette et al. (2020), Christensen (2022), and Becker & Rincon-Zapatero (2023). The paper by Marinacci & Montrucchio (2019) provides a useful alternative approach to existence of unique fixed points in the setting of order preserving maps. Experimental results on Epstein--Zin preferences can be found in Meissner & Pfeiffer (2022).

There is a strong connection between risk-sensitive preferences and the literature on robust control. See, for example, Cagetti et al. (2002), Hansen & Sargent (2007), and Barillas et al. (2009). We return to this point in Chapter 8.

The quantile preferences we considered in Section 7.3.2.3 have been analyzed in static and dynamic settings by Giovannetti (2013), Castro & Galvao (2019), Castro & Galvao (2022) and Castro et al. (2022). Recursive components of the analysis of quantile and Uzawa preference models build on the study of monotone preferences in Bommier et al. (2017).

Some recursive preference specifications involve ambiguity aversion. An introduction to this literature and its applications can be found in Klibanoff et al. (2009), Hayashi & Miao (2011), Hansen & Miao (2018), Bommier et al. (2019) and Hansen & Sargent (2020). Marinacci et al. (2023) discuss the connection between recursivity and attitudes to uncertainty. We discuss ambiguity again in Chapter 8.

Recursive preferences are increasingly applied outside the field of asset pricing, where they first came to prominence. See, for example, Bommier & Villeneuve (2012), Colacito et al. (2018), Jensen (2019), or Augeraud-Véron et al. (2019).

The coin flip application in Section 7.2.1.2 is related to correlation aversion, as discussed in Stanca (2023), and preference for “consumption spreads” as reviewed in Frederick et al. (2002).

Some applications of Theorem 7.1.3 to network analysis can be found in Sargent & Stachurski (2023).