4 Optimal Stopping

We study problems of maximizing lifetime rewards in settings in which decision-makers face risks. The job search model studied in Chapter 1 and Chapter 3 is one example. Others include an entrepreneur who decides whether to exit or enter a market, a borrower who considers defaulting on a loan, a firm that contemplates introducing a new technology, or a portfolio manager deciding whether to exercise a real or financial option.

These can all be formulated as dynamic programming and have common features that facilitate sharp characterizations of optimality. They are all two-action (or binary choice) problems that provide good laboratories for studying some special dynamic programs in which recursive representations are particularly enlightening.

4.1Introduction to Optimal Stopping¶

We begin with a standard theory of optimal stopping and then consider alternative approaches that feature continuation values and threshold policies. We aim to provide a rigorous discussion of optimality that refines our less formal analysis of job search in Section 1.3 and Section 3.3.1.

4.1.1Theory¶

Our first step is to set out the fundamental theory of discrete time infinite-horizon optimal stopping problems.

4.1.1.1The Stopping Problem¶

Let $\Xsf$ be a finite set. Given $\Xsf$, an optimal stopping problem is a tuple $\sS = (\beta, P, c, e)$ that consists of

1. a discount factor $\beta \in (0,1)$,

2. a Markov operator $P \in \mopx$,

3. a continuation reward function $c \in \RR^\Xsf$, and

4. an exit reward function $e \in \RR^\Xsf$.

Given a $P$-Markov chain $(X_t)_{t \geq 0}$, a decision-maker observes the state $X_t$ in each period and decides whether to continue or stop. If she chooses to stop, she receives final reward $e(X_t)$ and the process terminates. If she decides to continue, then she receives $c(X_t)$ and the process repeats next period. Lifetime rewards are

where $R_t$ equals $c(X_t)$ while the agent continues, $e(X_t)$ when the agent stops, and zero thereafter.

Optimal decisions are described by a policy function, which is a map $\sigma$ from $\Xsf$ to $\{0,1\}$. After observing state $x$ at any given time, the decision-maker takes action $\sigma(x)$, where 0 means “continue” and 1 means “stop.” Implicit in this formulation is the assumption that the current state contains enough information for the agent to decide whether or not to stop.

Let $\Sigma$ be the set of functions from $\Xsf$ to $\{0,1\}$. Let $v_\sigma(x)$ denote the expected lifetime value of following policy $\sigma$ now and in every future period, given optimal stopping problem $\sS = (\beta, P, c, e)$ and current state $x \in \Xsf$. We call $v_\sigma$ the $\sigma$-value function. We also call $v_\sigma(x)$ the lifetime value of policy $\sigma$ conditional on initial state $x$. Section Section 4.1.1.2, shows that $v_\sigma$ is well defined and describes how to calculate it. A policy $\sigma^* \in \Sigma$ is called optimal for $\sS$ if

4.1.1.2Lifetime Values¶

Fixing $\sigma \in \Sigma$, let us consider how to compute the lifetime value $v_\sigma(x)$ of following $\sigma$ conditional on $X_0 = x$. Evidently, $v_\sigma$ satisfies

Indeed, if $\sigma(x)=1$, then (4.3) states that $v_\sigma(x) = e(x)$, which is what we expect: if we choose to stop at a given state, then lifetime value from that state equals the exit reward. If, instead, $\sigma(x)=0$, then (4.3) becomes

which is also what we expect: the value of continuing is the current reward plus the discounted expected reward obtained by continuing with policy $\sigma$ next period.

We want to solve (4.3) for $v_\sigma$. To this end, we define $r_\sigma \in \RR^\Xsf$ and $L_\sigma \in \lopx$ via

With this notation, we can write (4.3) pointwise as $v_\sigma = r_\sigma + L_\sigma \, v_\sigma$. If $\rho(L_\sigma) < 1$, then

By Exercise 4.1 and the Neumann series lemma, $v_\sigma$ is uniquely defined by (4.6).

4.1.1.3Policy Operators¶

For the proofs, it will be helpful to view $v_\sigma$ as the fixed point of an operator. We associate each $\sigma \in \Sigma$ with a policy operator $T_\sigma$ defined at $v \in \RR^{\Xsf}$ by

for each $x \in \Xsf$. With this notation, (4.3) can be written as $v_\sigma = T_\sigma \, v_\sigma$.

Using the notation in Section 4.1.1.2, we can also define $T_\sigma$ via

The significance of Proposition 4.1.1 is that by construction $v_\sigma$ is a fixed point of $T_\sigma$. By the contraction property in Proposition 4.1.1, $v_\sigma$ is the only fixed point of $T_\sigma$ in $\RR^\Xsf$ and, moreover, iterates of $T_\sigma$ always converge to $v_\sigma$.

4.1.1.4The Value Function¶

In the job search problem in Section 3.3.1, we argued that the value function equals the fixed point of the Bellman operator. Here we make the same argument more formally in the more general setting of optimal stopping.

First, given an optimal stopping problem $\sS = (\beta, P, c, e)$ with $\sigma$-value functions $\{v_\sigma\}_{\sigma \in \Sigma}$, we define the value function $v^*$ of $\sS$ via

so that $v^*(x)$ is the maximal lifetime value available to an agent facing current state $x$. Following notation in Section 2.2.2.1, we can also write $v^* = \vee_\sigma \, v_\sigma$.

Given that solving the maximization in (4.13) is, in general, a difficult problem, how can we obtain the value function? The following steps can do the job:

1. formulate a Bellman equation for the value function of the optimal stopping problem, namely,

2. prove that this Bellman equation has a unique solution in $\RR^\Xsf$, and then

3. show that this solution equals the value function, as defined in (4.13).

We shall complete these steps in Section 4.1.1.5.

4.1.1.5The Bellman Operator¶

Define the Bellman operator for the optimal stopping problem $\sS = (\beta, P, c, e)$ as

where $x \in \Xsf$ and $v \in \RR^{\Xsf}$. By construction, any fixed point of $T$ solves the Bellman equation and vice versa. Pointwise, we can express $T$ via $Tv = e \vee (c + \beta Pv)$.

Our main result for this section is:

4.1.1.6Optimal Policies¶

Paralleling the definition provided in the discussion of job search (Section 1.3), for each $v \in \RR^\Xsf$, we call $\sigma \in \Sigma$ $v$-greedy if, for all $x \in \Xsf$,

A $v$-greedy policy uses $v$ to assign values to states and then chooses to stop or continue based on the action that generates a higher payoff.

With this language in place, the next proposition makes precise our informal Paragraph argument that optimal choices can be made using the value function.

Proposition 4.1.3 is a version of Bellman’s principle of optimality. We shall prove this principle in a more general setting in Chapter 5.

4.1.1.7Value Function Iteration¶

The theory just presented tells us that successive approximation using the Bellman operator converges to $v^*$ and $v^*$-greedy policies are optimal. These facts make value function iteration (VFI) a natural algorithm for solving optimal stopping problems. (VFI for optimal stopping problems corresponds to VFI for job search, as shown.) Later, in Theorem 8.1.1, we will show that when the number of iterates is sufficiently large, VFI produces an optimal policy.

4.1.2Firm Valuation with Exit¶

In Section 3.2.2.2 we discussed firm valuation using expected present value of the cash flow generated by profits. This is a standard approach. However, it ignores that firms have the option to cease operations and sell all remaining assets. In this section, we consider firm valuation in the presence of an exit option.

4.1.2.1Optional Exit¶

Consider a firm whose productivity is exogenous and evolves according to a $Q$-Markov chain $(Z_t)$ on finite set $\Zsf \subset \RR$. Profits are given by $\pi_t = \pi(Z_t)$ for some fixed $\pi \in \RR^\Zsf$. At the start of each period, the firm decides whether to remain in operation and receive current profit $\pi_t$, or to exit and receive scrap value $s > 0$ for sale of physical assets. Discounting is at fixed rate $r$ and $\beta \coloneq 1/(1+r)$. We assume that $r > 0$.

Let $\Sigma$ be all $\sigma \colon \Zsf \to \{0,1\}$. For given $\sigma \in \Sigma$ and $v \in \RR^\Zsf$, the corresponding policy operator is

We saw in Section 4.1.1.2--Section 4.1.1.3 that $T_\sigma$ has a unique fixed point $v_\sigma$ and that $v_\sigma(z)$ represents the value of following policy $\sigma$ forever, conditional on $Z_0 = z$.

The Bellman operator for the firm’s problem is the order-preserving self-map $T$ on $\RR^\Zsf$ defined by

Pointwise, $T$ can be written as $Tv = s \vee (\pi + \beta Q v)$.

Let $v^*$ be the value function for this problem. By Proposition 4.1.2, $v^*$ is the unique fixed point of $T$ in $\RR^\Zsf$ and the unique solution to the Bellman equation. Moreover, successive approximation from any $v \in \RR^\Zsf$ converges to $v^*$. Finally, by Proposition 4.1.3, a policy is optimal if and only if it is $v^*$-greedy.

Figure Figure 4.1 plots $v^*$, computed via VFI (i.e., successive approximation using $T$, along with the stopping value $s$ and the continuation value function $h^* = \pi + \beta Q v^*$, under the parameterization given in Listing 1. As implied by the Bellman equation, $v^*$ is the pointwise maximum of $s$ and $h^*$. The $v^*$-greedy policy instruct the firm to exit when the continuation value of the firm falls below the scrap value.

1
2
3
4
5
6
7
8
9
10
"Creates an instance of the firm exit model."
function create_exit_model(;
        n=200,                  # productivity grid size
        ρ=0.95, μ=0.1, ν=0.1,   # persistence, mean and volatility
        β=0.98, s=100.0         # discount factor and scrap value
    )
    mc = tauchen(n, ρ, ν, μ)
    z_vals, Q = mc.state_values, mc.p
    return (; n, z_vals, Q, β, s)
end

4.1.2.2Exit Versus No-Exit¶

If we define $w$ by $w(z) = \EE_z \sum_{t \geq 0} \beta^t \pi_t$ for all $z \in \Zsf$, then $w(z)$ is the value of the firm given $Z_0 = z$ when the firm never exits so that $w$ evaluates the firm according to expected present value of the profit stream. Figure Figure 4.2 shows the no-exit value $w$ based on the parameterization in Listing 1.

In Figure Figure 4.2, we see that $w \leq v^*$ on $\Zsf$. Let’s now prove that this is always true.

To show $w \leq v^*$, first observe that $w = (I - \beta Q)^{-1} \pi$, by $\beta < 1$ and Lemma 3.2.1. Rearranging gives $w = \pi + \beta Q w$.

Now note that under the policy $\sigma \equiv 0$, where the firm chooses never to exit, we have $T_\sigma v = \pi

• \beta Q v$. Hence the unique fixed point of$T_\sigma$is$w$. As a result,$w = v_\sigma$for$\sigma \equiv 0$. But$v^* \geq v_\sigma$for all$\sigma \in \Sigma$. This proves that$w \leq v^*$.

Choosing never to exit is a feasible policy. Since $v^*$ involves maximization of firm value over the set of all feasible policies, it must be at least as large as the value of never exiting.

4.1.3Monotonicity¶

We study monotonicity in values and actions in the general optimal stopping problem described in Section 4.1.1, with $\Xsf$ as the state space, $e$ as the exit reward function, and $c$ as the continuation reward function.

4.1.3.1Monotone Values¶

Let $v^*$ be the value function of an optimal stopping problem defined by $\Xsf$, $P$, $\beta$, $c$ and $e$ and define a continuation value function $h^*$

(The continuation reward function $c$ and the continuation value function $h^*$ are distinct objects.)

Let $\Xsf$ be partially ordered and let $i\RR^\Xsf$ be the increasing functions in $\RR^\Xsf$.

4.1.3.2Monotone Actions¶

The optimal policy in the iid job search problem takes the form $\sigma^*(w) = \1\{w \geq w^* \}$ for all $w \in \Wsf$, where $w^* \coloneq (1-\beta) h^*$ is the reservation wage and $h^*$ is the continuation value. This optimal policy is of threshold type: once the wage offer exceeds the threshold, the decision is to stop.

Since threshold policies are convenient, let us now try to characterize them.

Throughout this section, we take $\Xsf$ to be a subset of $\RR$. Elements of $\Xsf$ are ordered by $\leq$, the usual order on $\RR$.

For a binary function on $\Xsf \subset \RR$, the condition that $\sigma^*$ is decreasing means that the decision-maker chooses to exit when $x$ is sufficiently small.

In the settings of Exercise 4.9--Exercise 4.11, the optimal policy is either increasing or decreasing. Since $\Xsf$ is totally ordered, monotonicity implies that a threshold policy is optimal. For example, if $\sigma^*$ is increasing, then we take $x^*$ to be the smallest $x \in \Xsf$ such that $\sigma^*(x) = 1$. For such an $x^*$ we have

4.1.4Continuation Values¶

In Section 1.3.2.2 we solved the job search problem with iid draws by computing the continuation value $h^*$ directly and then choosing the policy $\sigma^*(w) = \1 \left\{ w/(1-\beta) \geq h^* \right\}$. We saw that this approach is more efficient than first computing the value function, since the continuation value is one-dimensional rather than $|\Wsf|$-dimensional.

In Section 3.3.1.2, we tried the same approach for the job search problem with Markov state, where wage draws are correlated. We gathered fewer benefits from using the continuation value approach in that setting, since the continuation value function has the same dimensionality as the value function.

These observations motivate us to explore continuation value methods more carefully. In this section, we formulate a continuation value approach for the general optimal stopping problem and verify convergence. We will see that, while all relevant state components must be included in the value function, purely transitory components do not affect continuation values. Hence the continuation value approach is at least as efficient and sometimes substantially more so.

Another asymmetry between value functions and continuation value functions is that the latter are typically smoother. For example, in job search problems, the value function is usually kinked at the reservation wage, while the continuation value function is smooth. Greater smoothness comes from taking expectations over stochastic transitions: integration acts as a smoothing operation. Like lower dimensionality, increased smoothness facilitates analysis and computation.

4.1.4.1The Continuation Value Operator¶

Let $h^*$ be the continuation value function for the optimal stopping problem defined in (4.27). To compute $h^*$ directly we begin with the optimal stopping version of the Bellman equation evaluated at $v^*$ and rewrite it as

Taking expectations of both sides of the equation conditional on current state $x$ produces $\sum_{x'} v^*(x') P(x, x') = \sum_{x'} \max \left\{ e(x'), h^*(x') \right\} P(x, x')$. Multiplying by $\beta$, adding $c(x)$, and using the definition of $h^*$, we get

This expression motivates us to introduce a continuation value operator $C \colon \RR^\Xsf \to \RR^\Xsf$ via

Proposition 4.1.5 provides the following alternative method to compute the optimal policy that does not involve VFI:

1. Use successive approximations to $h^*$ with $C$ and

2. Calculate $\sigma^*$ via $\sigma^*(x) = \1\{e(x) \geq h^*(x)\}$ for each $x \in \Xsf$.

In Section 4.1.4.2 we discuss settings where this approach is advantageous.

4.1.4.2Dimensionality Reduction¶

The beginning of Section 4.1.4 mentioned that switching from value function iteration to continuation value iteration can substantially reduce the dimensionality of the problem in some cases. Here we describe situations where this works.

To begin, let $\Wsf$ and $\Zsf$ be two finite sets and suppose that $\phi \in \dD(\Wsf)$ and $Q \in \mopz$. Let $(W_t)$ be iid with distribution $\phi$ and let $(Z_t)$ be an $Q$-Markov chain on $\Zsf$. If $(W_t)$ and $(Z_t)$ are independent, then $(X_t)$ defined by $X_t = (W_t, Z_t)$ is $P$-Markov on $\Xsf$, where

Suppose that the continuation reward depends only on $z$ so that we can write the Bellman operator as

Since the right hand side depends on both $w$ and $z$, the Bellman operator acts on an $n$-dimensional space, where $n \coloneq |\Xsf| = |\Wsf| \times |\Zsf|$.

However, if we inspect the right hand side of (4.35), we see that the continuation value function depends only on $z$. Dependence on $w$ vanishes because $w$ does not help predict $w'$. Thus, the continuation value function is an object in $|\Zsf|$-dimensional space. The continuation value operator

acts on this lower dimensional-space.

More examples of dimensionality reduction are illustrated in the applications.

4.1.4.3Application to Firm Value¶

Consider the firm valuation problem from Section 4.1.2 but suppose now that scrap value fluctuates with prices of underlying assets. For simplicity let’s assume that scrap value at each time $t$ is given by the iid sequence $(S_t)$, where each $S_t$ has density $\phi$ on $\RR_+$. The corresponding Bellman operator is

We can convert this problem to a finite-state space optimal stopping problem by discretizing the density $\phi$ onto a finite grid contained in $\RR_+$. However, since continuation values depend only on $z$, a better approach is to switch to a continuation value operator.

4.2Further Applications¶

In this section, we discuss some additional applications of optimal stopping.

4.2.1American Options¶

We discussed American options briefly in Example 4.1.2. Here we investigate this class of derivatives more carefully. We focus on American call options that provide the right to buy a particular stock or bond at a fixed strike price $K$ at any time before a set expiration date. The market price of the asset at time $t$ is denoted by $S_t$.

We discussed a case in which the expiration date is infinity in Example 4.1.2. However, options without termination dates -- also called perpetual options -- are rare in practice. Hence we focus on the finite-horizon case. We are interested in computing the expected value of holding the option when discounting with a fixed interest rate, a typical assumption when pricing American options.

Finite horizon American options can be priced by backward induction in an approach like the one we used for the finite horizon job search problem discussed in Chapter 1. Alternatively, we can embed finite horizon options into the theory of infinite-horizon optimal stopping. We use the second approach here, since we have just presented a theory for infinite-horizon optimal stopping.

To this end, we take $T \in \NN$ to be a fixed integer indicating the date of expiration. The option is purchased at $t=0$ and can be exercised at any $t \in \NN$ with $t \leq T$. To include $t$ in the current state, we set

The idea is that time is updated via $t' = m(t)$, so that time increments at each update until $t=T+1$. After that we hold $t$ constant. Bounding time at $T+1$ keeps the state space finite.

We assume that a stock price $S_t$ evolves according to

Here $(Z_t)_{t \geq 0}$ is $Q$-Markov on finite set $\Zsf$ for some $Q \in \mopz$ and $\Wsf$ is also finite. This means that the share price has both persistent and transient stochastic components. If we set parameters so that $(Z_t)_{t \geq 0}$ resembles a random walk, price changes will be difficult to predict.

To form a Section 4.1.1.1 optimal stopping problem, we must specify the state and clarify the $P \in \mopx$ that maps to the state process. We set the state space to $\Xsf \coloneq \Tsf \times \Wsf \times \Zsf$ and

Thus, time updates deterministically via $t' = m(t)$ and $z'$ and $w'$ are drawn independently from $Q(z, \cdot)$ and $\phi$ respectively.

As for a perpetual option, the continuation reward is zero and the discount factor is $\beta \coloneq 1/(1+r)$, where $r > 0$ is a fixed risk-free rate. The exit reward can be expressed as $\1\{t \leq T\} (S_t - K)$ so that exercising at time $t$ earns the owner $S_t - K$ up to expiry and zero thereafter. In terms of the state $(t, z)$, the exit reward is

The Bellman equation can be written

where $t' = m(t)$. This value function $v(t, w, z)$ neatly captures the value of the option: It is the maximum of current exercise value and the discounted expected value of carrying the option over to the next period.

Since the problem just described is an optimal stopping problem in the sense of Section 4.1.1.1, all of the optimality results attained for that problem apply. In particular, iterates of the Bellman operator converge to the value function $v^*$ and, moreover, a policy is optimal if and only if it is $v^*$-greedy.

We can do better than VFI. Since $(W_t)_{t \geq 0}$ is iid and appears only in the exit reward, we can reduce dimensionality by switching to the continuation value operator, which, in this case, can be expressed as

As proved in Section 4.1.4, the unique fixed point of $C$ is the continuation value function $h^*$, and $C^k h \to h^*$ as $k \to \infty$ for all $h \in \RR^\Xsf$. With the fixed point in hand, we can compute the optimal policy as

Here $\sigma^*(t, w, z) = 1$ prescribes exercising the option at time $t$.

Figure Figure 4.3 provides a visual representation of optimal actions under the default parameterization described in Listing 2. Each of the three figures show contour lines of the net exit reward $f(t, w, z) \coloneq e(t, w, z) - h^*(w, z)$, viewed as a function of $(w, z)$, when $t$ is held fixed. The date $t$ for each subfigure is shown in the title. The optimal policy exercises the option when $f(t, w, z) \geq 0$.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
using QuantEcon, LinearAlgebra, IterTools

"Creates an instance of the option model with log S_t = Z_t + W_t."
function create_american_option_model(;
        n=100, μ=10.0,  # Markov state grid size and mean value
        ρ=0.98, ν=0.2,  # persistence and volatility for Markov state
        s=0.3,          # volatility parameter for W_t
        r=0.01,         # interest rate 
        K=10.0, T=200)  # strike price and expiration date
    t_vals = collect(1:T+1)
    mc = tauchen(n, ρ, ν)
    z_vals, Q = mc.state_values .+ μ, mc.p
    w_vals, φ, β = [-s, s], [0.5, 0.5], 1 / (1 + r)
    e(t, i_w, i_z) = (t ≤ T) * (z_vals[i_z] + w_vals[i_w] - K)
    return (; t_vals, z_vals, w_vals, Q, φ, T, β, K, e)
end

In each subfigure, the exercise region, which is the set $(w, z)$ such that $f(t, w, z) \geq 0$, correspond to the northeast part of the figure, where $w$ and $z$ are both large. The boundary between exercise and continuing is the zero contour line, which is shown in black. Notice that the size of the exercise region expands with $t$. This is because the value of waiting decreases when the set of possible exercise dates declines.

Figure Figure 4.4 provides some simulations of the stock price process $(S_t)_{t \geq 0}$ over the lifetime of the option, again using the default parameterization described in Listing 2. The blue region in the top part of each subfigure contains values of the stock price $S_t = Z_t + W_t$ such that $S_t \geq K$. In this configuration in which the price of the underlying exceeds the strike price, the option is said to be “in the money.” The figure also shows an optimal exercise date that is the first $t$ such that $e(t, W_t, Z_t) \geq h^*(W_t, Z_t)$ in a simulation.

4.2.2Research and Development¶

Consider a firm that engages in costly research and development (R&D) in order to develop a new product. The firm decides whether to continue developing the product before starting to market it or to stop developing and start marketing it. For simplicity, we assume that the value of bringing the product to market is a one-time lump sum payment $\pi_t = \pi(X_t)$, where $(X_t)$ is a $P$-Markov chain on finite set $\Xsf$ with $P \in \mopx$. The flow cost of investing in R&D is $C_t$ per period, where $(C_t)$ is a stochastic process. Future payoffs are discounted at rate $r > 0$ and we set $\beta \coloneq 1/(1+r)$.

4.2.2.1Constant R&D Costs¶

As a first take on this problem, suppose that $C_t \equiv c \in \RR_+$ for all $t$ and formulate an optimal stopping problem with exit reward $e=\pi$ and constant continuation reward $-c$. The Bellman equation is

4.2.2.2IID R&D Costs¶

Let’s suppose now that $(C_t)_{t \geq 0}$ is iid with common distribution $\phi \in \dD(\Wsf)$. The Bellman equation is

Since $(C_t)$ is iid, we would ideally like to integrate it out in the manner of Section 4.1.4.2, thereby lowering the dimensionality of the problem. However, note that the continuation value associated with (4.52) is

which still depends on $c$.

Fortunately, there is a way to eliminate $c$. Define the expected value $g(x)$ in state $x$ by

Rewrite the Bellman equation using $g$ and replacing $(c, x)$ with $(c', x')$ to get

Averaging over $(c', x')$ and using the definition of $g$ again gives

This is a functional equation in $g$ that depends only on $x$. To solve it, we introduce the operator $R$ defined by

From Exercise 4.16, we see that (4.56) has a unique solution $g^*$ in $\RR^\Xsf$ that can be computed by successive approximation. With $g^*$ in hand, we can compute the optimal policy via

4.3Chapter Notes¶

Various textbooks treat optimal stopping in depth, although most use continuous time. Peskir & Shiryaev (2006) and Shiryaev (2007) are good examples.

There are many applications of optimal stopping in economics and finance, with influential early research papers including McCall (1970), Jovanovic (1982), Hopenhayn (1992), and Ericson & Pakes (1995). Arellano (2008) considers borrowing on international financial markets with the option of sovereign default (see Section 8.2.1.5). Riedel (2009) studies optimal stopping under Knightian uncertainty. Fajgelbaum et al. (2017) include an optimal stopping problem for firms in a model of uncertainty traps.

The firm problem with optimal exit has been used to analyze firm dynamics and firm size distributions in equilibrium models with heterogeneous firms. Hopenhayn (1992) is the classic reference. Perla & Tonetti (2014) construct a growth model in which firms at the bottom of the productivity distribution imitate more productive firms. Carvalho & Grassi (2019) analyze business cycles in a setting of firm growth with exit and a Pareto distribution of firms.

Infinite duration American options are analyzed in Mordecki (2002). Practical methods for pricing American options are provided by Longstaff & Schwartz (2001), Rogers (2002), and Kohler et al. (2010).

Replacement problems are an important optimal stopping problem not treated in this chapter. An important early paper by Rust (1987) uses dynamic programming to find optimal replacement policies for of engine parts and goes on to fit the model to data. Section 5.3.1 discusses structural estimation in the style of Rust (1987) and others.