Adaptation in Artificial Economies

This chapter puts adaptive agents into five different environments that have been analyzed in the rational expectations literature, thereby illustrating some of the different structures and possibilities in economic systems composed of such agents.

The first model, due to Bray, nicely illustrates many features of ‘least squares learning’ in a ‘self-referential’ system, including the temporary irrationality of adaptive forecasting rules and the possibility of their eventual rationality. The second model, a version of Samuelson’s overlapping-generations model of money, illustrates how successive generations can adaptively climb their way to more or less complicated rational expectations equilibria, and how the rate of convergence can depend on details of the adaptation algorithm and the intricacy of what must be learned. The third example puts overlapping generations of adaptive agents into an environment with too many equilibria, namely, a multiple currency setting in which the substitution of adaptive for ‘rational’ agents is enough to render the exchange rate determinate, but history dependent. The fourth example is a version of the ‘no-trade’ environment of Jean Tirole, in which the ‘problem’ with the rational expectations equilibrium is its incredible efficiency in eliminating opportunities for trade based on disparate information; I show how replacing rational agents with adapting ones can serve temporarily to restore opportunities for trade and thereby create trading volume. The last example, Marcet and Sargent’s model of investment under uncertainty with learning, is designed to illustrate how much ‘coaxing’ must be done by us and how much ‘theorizing’ must be done by our artificial agents for them to learn when their planning horizon is infinite.

The presentation in this chapter is informal. We spend most of our effort describing and simulating models. The chapter is concluded with a brief description of how the machinery of stochastic approximation can be used to attain analytical results about the limiting behavior of such models.

A model of Bray¶

Margaret Bray (1982) studied a model that exhibits several features of systems that are adapting their way to a rational expectations equilibrium. These features include:

(a) People use a forecasting scheme that would be optimal if the environment were stationary. But their learning causes the environment to be non-stationary, and their learning scheme suboptimal.

(b) Sometimes the system converges to a rational expectations equilibrium.

(c) If the system does not converge to a rational expectations equilibrium, it does not converge.

(d) The dimension of the ‘state’ of the system with learning is larger than the corresponding rational expectations equilibrium, because measures of people’s beliefs are needed to describe the position and motion of the system.

In Bray’s model, the environment would be stationary if people were to know the distribution of prices. The dynamics in the model all come from the adjustment of people’s expectations, which vanish if and when people learn the equilibrium distribution of prices.

Bray assumed a ‘cobweb’-like structure in which the equilibrium price $p_t$ for a single commodity is determined by a market-clearing condition of the form

where $p_{t+1}^e$ is the price that market participants expect to prevail at time $t + 1$, and $\{u_t\}$ is an independently and identically distributed sequence of random variables with mean zero. To compute a rational expectations equilibrium, we note the absence of dynamics in either the structural equation (1) or the shock $u_t$, and so we guess that $p_{t+1}^e = \beta$ $\forall t$, a constant that is independent of time. Substituting this guess into (1) gives $p_t = a + b\beta + u_t$, which implies $E_{t-1} p_t = a + b\beta$. Evidently, the guess is true if $\beta = a/(1 - b)$. Substituting this value of $\beta = p_{t+1}^e$ back into (1) shows that in a rational expectations equilibrium $p_t = \beta + u_t$.

In backing off rational expectations, Bray assumed that people form the expectation $p_{t+1}^e$ by taking an average of past prices. For convenience, we use the notation $\beta_t = p_{t+1}^e$. In terms of a stochastic approximation algorithm, Bray’s assumption about expectations can be represented as

Notice how this scheme uses only observations on prices through period $t - 1$ to form price expectations at time $t$.

Rewrite equation (1) by substituting $\beta_t$ for $p_{t+1}^e$ to get

Given an initial condition for $\beta$, equations (2) and (3) determine the evolution of $(p, \beta)$ through time, where $\beta_t$ is interpreted as people’s expectation of what $p_{t+1}$ will be.^[1] Bray studied the circumstances under which $\beta_t$ and the distribution of $p_{t+1}$, which evolve interdependently, would converge to a rational expectations equilibrium. That is, she studied the conditions under which $\beta_t$ would converge to the value $\beta = a/(1 - b)$.

For describing people’s learning behavior, we can use a state vector $z_t = [p_t \ 1]'$, whose law of motion is evidently

This system indicates that, when at time $t$ people estimate the price next period to be $\beta_t$, they act to make the best prediction of next period’s price be $a + b\beta_t$. Notice that in forecasting this way people are acting as if they believe (incorrectly) that the law of motion of the state is not (4) but rather

for some serially uncorrelated random process $\{v_t\}$ with mean zero, where $\beta$ is a constant. When people perceive that the law of motion for $z_t$ is governed by (5), their forecasting causes the actual law of motion to be (4).

Margaret Bray showed that, if $b < 1$, system (2), (3) will converge to a rational expectations equilibrium with probability one. She also noted that $b < 1$ is a necessary condition for convergence to a rational expectations equilibrium.^[2] Furthermore, she showed that, if the system does not converge to a rational expectations equilibrium, it does not converge at all.^[3]

Irrationality of expectations¶

Although the model’s exogenous ‘fundamentals,’ i.e. the $\{u_t\}$ process and the parameters $a$, $b$, are stationary, the stochastic process for the price $\{p_t\}$ is nonstationary, because it is a piece of the joint process $\{p_t, \beta_t\}$ determined by (2), (3). This means that the expectations formation scheme (2), which is a sensible way to estimate a mean for a stationary process (e.g., for someone already living within the rational expectations equilibrium of this market), is suboptimal so long as expectations are being revised. The fact that $\beta$ in (3) is moving through time, as described by the law of motion (4), means that $\beta_t$ is itself a ‘hidden state variable,’ and that the system (4) should be augmented to include it. Substituting (2) into (3) and rearranging gives the following system:

This is a nonlinear stochastic difference equation, which can be used to forecast prices with smaller mean squared error than given by the forecast $\beta_t$ used by the people in the model. In particular, the expectation of $p_t$ conditioned on the entire state $z_t^* = [p_{t-1} \ 1 \ \beta_{t-1}]'$ is equal to

Equation (7) gives the ‘rational expectation’ of price conditional on the full state vector $[p_{t-1} \ 1 \ \beta_{t-1}]$. This is the price that would be forecast by an outside observer who knew that the price was determined by (2), (3), and who could observe (or compute) $\beta_{t-1}$ as well as $p_{t-1}$. The failure of this conditional expectation to equal $\beta_t$, except after convergence, indicates the irrationality of the learning scheme.

Figure 1 and Figure 2 display aspects of a simulation of Bray’s model in which we set $a = 5$, $b = 0.7$, $E u_t^2 = 1$. The random process $\{u_t\}$ was generated with a Gaussian pseudo-random number generator. The rational expectations price is $a/(1 - b) = 16.667$. We started the system at $\beta_0 = 8$, an expected price far below the rational expectations price. Figure 1 shows the gap between the least squares forecast $\beta_t$ and the conditional expectation $E p_{t+1} | z_{t+1}^*$, which is large at first, then gradually diminishes over time. Figure 2 and Figure 3 show how the rational expectations forecast $E p_t | z_t^*$ on average is closer to the actual price $p_t$ than is $\beta_{t-1}$.

Why not repair the irrationality indicated by the discrepancy between $\beta_{t-1}$ and $E p_t | z_t^*$ by going back to the original model and replacing $\beta_{t-1}$ by $E p_t | z_t^*$? Evidently, using this new theory of price expectations would require us to modify the actual law of motion for prices, which would render our new scheme suboptimal again. But we can use this new actual law of motion as our theory of expectations. This line starts us on a recursion, which has been taken up by Bray and Kreps (1987), who show that in the limit it leads us back to a rational expectations in which agents are learning ‘within’ the equilibrium, but not ‘about’ the equilibrium. Bray and Kreps argue against following this recursion to its limit if it is ‘bounded rationality’ that we are after.^[4]

Heterogeneity of expectations and size of the state¶

Bray’s model assumes that all market participants have the same beliefs $\beta_t$. If we permit heterogeneity of beliefs, the effect is to add to the dimension of the true state in the appropriate counterpart to (6). For example, suppose that there are two classes of agents, differentiated only by the initial $\beta$, say $\beta_0^a$ and $\beta_0^b$, which they use in a version of scheme (2), and that each of the two classes accounts for half of the market. Then the counterpart to (6) would include $(\beta_t^a, \beta_t^b)$ as state variables. In this version of Bray’s model, heterogeneity of beliefs would vanish as the system converges to a rational expectations equilibrium.^[5]

An economy with Markov deficits¶

The second example is the overlapping-generations model of a monetary economy introduced by Paul Samuelson, and used extensively by John Bryant, Neil Wallace, and others to study issues of inflationary finance. We use this model to illustrate how:

(a) Learning can be modelled by having overlapping generations of agents adjust their behaviors relative to those of their ancestors in a utility increasing direction.

(b) The object about which agents are learning can be specified ‘non-parametrically,’ provided that agents are patient enough or lucky enough to be willing to learn how to behave on a state-by-state basis.

(c) Where the state is of high dimension, agents can be modelled as learning by using parametric decision rules.

(d) There is a close connection between algorithms to compute (approximate) equilibria and models of learning.

The economy consists of overlapping generations of two-period lived agents. At each date $t \geq 1$, there are born $N$ identical agents who are endowed with $w_1$ units of a single consumption good when young, and $w_2$ units when old. Each young agent’s preferences over a lifetime consumption profile $(c_1, c_2)$ are ordered by the expected value of $u(c_1) + u(c_2)$, where $u(c) = -c^{-\sigma}/\sigma$, where $\sigma > 0$.

There is a government that prints currency to finance government expenditures that are governed by a Markov chain

We let $G = [\bar{G}_1, \bar{G}_2, \ldots, \bar{G}_n]$ be the possible levels of government expenditures. The government’s budget constraint is

where $H_t$ is the stock of currency carried over by the young at $t$ to $t + 1$, and $p_t$ is the time $t$ price level.

Stationary rational expectations equilibrium¶

The rate of return on currency between $t$ and $t + 1$ is $p_t/p_{t+1} = R_t$. We shall seek a stationary equilibrium in which the rate of return on currency is given by

Finding a stationary equilibrium requires solving a set of nonlinear equations in a set of vectors characterizing individual agents’ optimal decisions. In a stationary equilibrium, savings are determined by an $(n \times 1)$ vector of state-dependent saving rates where $s = [s_1, \ldots, s_n]$ and $s_i$ is the saving rate when $G_t = \bar{G}_i$.^[6] A young agent’s utility is given by

for $i = 1, \ldots, n$ where $u$ is an increasing and concave utility function. The first-order conditions with respect to $s_i$ are $V'(s_i) = 0$ $\forall i$ or

In a stationary equilibrium, the government’s budget constraint, namely, $(H_t/p_t) - (H_{t-1}/p_{t-1})(p_{t-1}/p_t) = G_t$, can be written as

where $h_i = H_t/p_t$ when $G_t = \bar{G}_i$.

Finally, the condition that the supply of currency must equal the demand can be written $H_t/p_t = s_t N$ or

To determine a stationary equilibrium, we have to solve equations (12), (13), and (14) for $(n \times 1)$ vectors $s$ and $h$ and an $(n \times n)$ matrix $R$ of rates of return, where the $s_i$’s satisfy $s_i \in (0, w_1)$ and the elements of $R(i, j)$ are all positive. Notice that (13) and (14) imply

Substituting (15) into (12) gives the following set of $n$ nonlinear equations to be solved for $[s_1, \ldots, s_n]$:

Evidently, a stationary equilibrium exists if and only if (16) can be solved for $s = [s_1, \ldots, s_n]$ with $s_i \in (0, w_1)$ for all $i$.

In general, the system of nonlinear equations (16) that determines a vector of stationary equilibrium saving rates $s$ has multiple solutions. In addition to multiple stationary equilibria, there are nonstationary equilibria of the model, with a form resembling the ‘bubble equilibria’ of the models of money described in Chapter 2 and which we shall meet again in Chapter 6. Overlapping-generations models of the type we are using also have stationary ‘sunspot equilibria,’ that is, equilibria in which random variables (called ‘sunspots’ or ‘extrinsic random variables’) influence equilibrium prices and quantities only because they are expected to influence them.^[7] The stability of sunspot equilibria under adaptive learning has been studied by Woodford (1990), Evans (1989), and Evans and Honkapohja (1992a).

A learning version¶

We use this environment as a setting in which successive generations of agents are ‘learning’ or ‘evolving.’ We want to watch how collections of adapting agents cope with the environment, and see whether and when they might eventually learn the rational expectations equilibrium. We can also watch how agents’ learning varies as we alter the complexity of what they learn about, which in this setting is controlled by the number of states and the stochastic structure of the Markov process for government expenditures.

We endow agents with knowledge about their own utility functions, about the previous experiences of agents like themselves, and about the behavior of past and present government expenditures and prices. However, we do not give agents knowledge of the distributions of government expenditures, prices, and rates of return. Instead of knowing these distributions, agents must somehow use their historical observations, which might be arranged in the form of histograms or empirical probability distributions, to make decisions by some principle other than that of ‘expected utility maximization with knowledge of equilibrium probability distributions.’

The economy with learning is identical with the model with rational agents, except that now the households consist of two classes (subsequences) of (adaptive) agents. We include two classes, called ‘odd’ and ‘even,’ because, in order to evaluate a person’s saving decision, we wait until two periods’ worth of consumption data for that person have become known. Odd agents reset a saving rate when $t$ is odd, while even agents reset a saving rate when $t$ is even. Odd agents learn from the past experiences of other odd agents, and even agents from the past experiences of other even agents. Agents of each class will be assumed to update their saving decisions based on the utility experienced by previous people of their type. In particular, they will adapt the decisions of their predecessors using a recursive Newton--Raphson (or stochastic approximation) procedure.^[8] The ex post realized utility of a person who observed $G_t = \bar{G}_i$ and set $s_t = s_i$ when young at time $t$ is

where $R_t$ is the realized gross rate of return. The derivatives of realized utility with respect to the saving decision $s_i$ are

Notice that, in a rational expectations equilibrium, $V'(s_{it}) = E_t U'(s_{it})$, $V''(s_{it}) = E_t U''(s_{it})$, where $E_t(\cdot)$ is the expectation conditional on $G_t = \bar{G}_i$. We want a learning algorithm to apply where people don’t know these conditional expectations.

We assume that people use a Robbins--Monro algorithm, state by state. To set up the Robbins--Monro algorithm, we have to keep track of the number of periods an individual has been in a given state (i.e., observed $G_t = \bar{G}_i$) for each state. We let $t_j = 1, 2, \ldots$ for $j = o, e$ index the cumulative number of odd and even generations, respectively. For each state $i = 1, 2, \ldots, n$, we let $\tau_i^j(t_j)$, $j = o, e$, index the cumulative number of times that $G_t$ has equalled $\bar{G}_i$. That is,

We define the decreasing gain sequence $\gamma_\tau = 1/\tau$. The learning algorithm is then

This algorithm is set up to promote the possibility that, as $\tau_i^j \to \infty$, we will have $M(i, \tau_i^j) \to E_i U''(s_i)$ and $s^j(i, \tau_i^j)$ will solve the first-order conditions $E_i U'(s_i) = 0$.

We assume the one-period utility function $u(c) = -c^{-\sigma}/\sigma$. In the experiments reported below, we assume the logarithmic specification $\sigma = 1$.

We assume that young agents observe $G_t$ before they make their saving decision. Agents of each type begin each period with an $(n \times 1)$ vector of saving rates $s^j(\tau, \tau_i^j)$, $j = o, e$, ‘learned’ from ancestors of their own type, which they use as a state-contingent saving rule. When $G_t = \bar{G}_i$, the young at date $t$ set savings according to $s_t = s^j(i, \tau_i^j)$.

The price level at time $t$ is determined by the two equations

which imply

We require an initial condition $H_0$ for $H_t$ at $t = 1$, and initial conditions for $(M^j, s^j)$ for each of our two classes of agents.

We assume, as in the rational expectations version of the model, that $G_t$ is a Markov chain with transition matrix $\pi$, where $\pi(i, j) = \text{Prob}\{G_{t+1} = \bar{G}_j \mid G_t = \bar{G}_i\}$.

Some experiments¶

Figure 4 and Figure 5 report the results of using our learning algorithm. For each experiment, we set $\gamma_\tau = 1/\tau$ $\forall \tau$, and we set initial conditions for $M_t = -I$.^[9] We studied two economies, identical in all respects except for the stochastic process for government expenditures. In each economy, government expenditures follow a two-state Markov process with transition matrix $\pi$. In each economy, $w_1 = 20$, $w_2 = 10$ for each type of agent. In the first economy, government expenditures are identically zero in both states and $\pi_{ij} = 0.5$ $\forall (i, j)$. This implies that the rational expectations savings rates are 5 for each ‘state’ of government expenditures, and that the equilibrium rate of return on currency is unity always. In the second economy, government expenditures follow a Markov process with

where the two states are $[\bar{G}_1 \ \bar{G}_2] = [0.8 \ 0]$. For the second economy, the equilibrium savings rates are $[4.211 \ 4.364]$, and the equilibrium rates of return are

For comparability, we model each of the economies as being driven by a two-state Markov process for government expenditures, though in the first economy this means that the agents are wastefully overfitting their saving function.

Figure 4 and Figure 5 indicate that both economies are converging to the rational expectations savings rate. The convergence occurs more and more smoothly as time passes, a feature caused by the action of the $\gamma_\tau$ sequence.

In the zero expenditure economy, nothing stochastic is occurring. For this economy, convergence can be accelerated by using a constant-gain algorithm. This algorithm is formed by replacing the $\gamma_\tau$ by a constant. Besides accelerating convergence in constant environments, a potential advantage of constant-gain learning schemes is that they retain their flexibility to respond with the passage of time. A concomitant disadvantage is that their readiness to respond to recent occurrences prevents convergence to a rational expectations equilibrium when there are intrinsic shocks in the system. When intrinsic shocks are present, the most that can be hoped for with a constant-gain algorithm is convergence to a situation in which beliefs eventually spend most of their time within a neighborhood (whose size depends on the gain parameter) of rational expectations beliefs.^[10]

We display the results of using the constant-gain learning scheme in Figure 6 and Figure 7. Evidently, convergence with a constant-gain algorithm occurs much faster in the economy with the simpler government policy (the one with government expenditures always zero).

A comparison of the outcomes depicted in Figure 5 and Figure 6 provides an idea of some of the tradeoffs involved between constant-gain and decreasing-gain algorithms.

Parametric and non-parametric adaptation¶

In the preceding formulation, people choose one saving rate for each level of government expenditure. This specification was designed potentially to let the system eventually ‘learn’ the rational expectations equilibrium, in which the ‘state’ is the $(n \times 1)$ vector $G$ of government expenditures. By letting agents learn a distinct saving rate $s$ to apply for each $G$, we are in effect letting them use a non-parametric specification to learn about a policy function $s = f(G)$.

There are two potential difficulties with this specification. First, the transition matrix $\pi$ may imply that some states $\bar{G}_i$ are visited very infrequently. Observations from such states will roll in only slowly, making learning occur slowly. Of course, in terms of the unconditional expected utility of the agents, failure to learn about the correct thing to do in such infrequently visited states may cost little. Second, when the number of states $n$ is large, the specification of one saving parameter for each state will become burdensome, again because the observations per state will roll in slowly.

An econometrician’s or statistician’s solution to this problem would be to assume a parametric form for the saving function $s = f(G, \theta)$, where $\theta$ is a vector of parameters, of small dimension relative to $n$, and then to use all of the observations to estimate $\theta$. A recursive algorithm for estimating the parameters $\theta$ would use the gradient

and the Jacobian $\partial U^2 / \partial \theta^2$. A recursive algorithm would be

This algorithm uses each observation to estimate a more or less smooth function $f(G, \theta)$ to be used to determine savings.

Use of a parametric form $f(G, \theta)$ for the saving function raises the issue of approximation. Evidently a learning scheme that uses a parametric specification has a chance eventually of converging to a rational expectations equilibrium only if a rational expectations equilibrium can be supported by a saving function within the class of functions determined by $f(G, \theta)$. For many models, the chosen econometrically convenient function $f(G, \theta)$ will not be compatible with the functions determined by a rational expectations equilibrium. In these situations, a learning scheme based on a parametric specification will, if it converges, converge to an approximate rational expectations equilibrium.^[11]

Learning the hard way¶

In the preceding model, learning occurs between non-overlapping generations of grandparents to grandchildren, with the grandchildren adjusting their grandparents’ saving choice after observing the consequences of their grandparents’ choice.^[12] This model requires little of agents in the way of ‘theorizing,’ at the cost of rendering their learning dependent on the behaviors and experiences of their predecessors.

When we extend the horizon beyond two periods, it becomes increasingly inconvenient to model learning in this way because we have to wait longer for the consequences of life-time savings behavior to be known. If we attribute some ‘theorizing’ to our agents, we can avoid the need to learn only from one’s predecessors’ complete life-time experiences.

Learning via model formation¶

To motivate an alternative model of learning in this environment, consider the Euler equation for a young agent’s saving decision within a rational expectations equilibrium:

where the conditional expectation $E_t(\cdot)$ is over the equilibrium distribution of the rate of return on currency, $R_t = p_t / p_{t+1}$, conditional on the current value of the deficit $G_t$. As earlier, $s_i$ is the saving rate when $G_t = \bar{G}_i$. One way to formulate the problem of learning is to suppose that there is a representative young agent within each generation who knows the utility function $u$ and how to compute the derivative $u'(\cdot)$, but who does not know the distribution with respect to which the conditional expectation $E_t(\cdot)$ is to be computed in (31). To cope with this situation, the agent forms a model of the probability distribution with respect to which $E_t(\cdot)$ is to be computed, and adopts an algorithm for updating this distribution as new data arrive. At each point in time, the agent uses this estimated model distribution as the distribution in (31), and uses (31) to determine $s_i$. Then the price level is determined as above, namely, by

We describe two methods for modelling and updating the required distributions.

Updating histograms¶

Here the agent’s model is created by simply forming histograms of ex post realized rates of return $R_t$, one for each of the possible realized values of $G_t$. When $G_t = \bar{G}_i$ is observed at time $t$, the young agent forms $s_i$ by using that histogram to represent the conditional expectation in (31). Let $r_j$, $j = 1, \ldots, J(t, i)$, be the population of values of $R_t$ that have been observed prior to $t$ to follow the event $G_t = \bar{G}_i$, where $J(t, i)$ is the number of times the event $G_t = \bar{G}_i$ has occurred prior to time $t$. Then $s_i$ is the value that solves

As time passes, the histograms are updated.

A parametric model of conditional probabilities¶

Here the agent adopts a parametric model of the conditional probabilities, namely,

At time $t$ the agent has an estimate of the parameters of the distributions $\theta_{it}$, and uses them to determine behavior via the following approximation to (31):

As data on $(R_t, G_t)$ pairs flow in, the agent uses an adaptive algorithm to update his estimates of $\theta_{it}$. For example, for each $i$, let the distribution be a two-parameter distribution determined by the first and second moment. Then the agent would update these parameters using the stochastic approximation algorithm

where there is a different ‘clock’ $\tau(t)$ for each event $G_t = \bar{G}_i$.

Approximate equilibria¶

When we adopt a learning scheme that restricts agents’ decision rules to too small a class of functions, we cannot expect the economy with adaptively learning agents ever to converge to a rational expectations equilibrium. The most we can hope is that the learning economy might converge to an approximate equilibrium, a concept that is used by applied researchers interested in computing rational expectations equilibria. In this section, I briefly describe some of the connections between algorithms to compute approximate equilibria and economies populated by adaptively learning agents.

Computing an equilibrium of the model becomes more demanding as we expand the dimension of the state space. Suppose that we modify the previous model by assuming that government expenditures are determined by the continuous state Markov process with transition kernel

All other aspects of the model remain unchanged. We conjecture an equilibrium saving function of the form $s_t = f(G)$, and use the equilibrium conditions to derive restrictions on this function. The household’s first-order conditions evaluated at $s_t = f(G_t)$ can be written

where $E_t(\cdot) = E(\cdot \mid G_t)$. The equilibrium condition and the government budget constraint imply $Nf(G_t) - Nf(G_{t-1})R_{t-1} = G_t$, which can be solved for $R_{t-1}$:

Substituting this into the household’s first-order condition gives

which is a functional equation in $f(G_t)$.

There exist a number of methods for solving a functional equation like (41) numerically. All of these methods replace the function $f(G)$ with a finite-parameter approximation $f(G, \theta)$, then find values of the parameters $\theta$ that come as close as possible to satisfying (41).^[13]

Method of parameterized expectations¶

Here is how Albert Marcet’s method of parameterized expectations can be used approximately to solve the functional equation (41). For convenience, write the right side of (41) as $E_t k(G_t, G_{t+1})$ where $k(G_t, G_{t+1}) = u'(w_2 + (Nf(G_{t+1}) - G_{t+1})/N)(Nf(G_{t+1}) - G_{t+1})/Nf(G_{t+1})$.^[14]

1. Guess that the conditional expectation on the right side of (41) has a form $h(G_t, \theta)$. Pick a starting value of $\theta$, call it $\theta_j$ for $j = 1$. Use this guess and (41) to solve for an initial saving function $s_t = f(G_t, \theta_j)$.

2. Use a random number generator to draw a realization of length $T$ from the Markov process $F(G', G)$. Use this simulation and $f(G_t, \theta_j)$ to generate a realization of $k(G_t, G_{t+1})$. Then use this realization to compute the non-linear regression coefficients $\theta_{j+1}$ in the regression $E_t k(G_t, G_{t+1}) = h(G_t, \theta_{j+1})$.

3. Solve the first-order condition $u'(w_1 - s_t) = h(G_t, \theta_{j+1})$ for a new saving function $f(G_t, \theta_{j+1})$.

4. Iterate on steps 1--3 to convergence.

There is evidently a close connection between this method for equilibrium computation and the behavior of a system populated by adaptive agents. Indeed, we can reinterpret a recursive or ‘on-line’ version of this algorithm as a system with adaptive agents. Thus, a recursive version of the nonlinear least squares algorithm is

where $\nabla h(G, \theta)$ is the gradient of $h$ with respect to the parameters $\theta$. Upon noting the resemblance between this algorithm and the learning scheme (41), it is understandable that Marcet proposed his equilibrium computation scheme as an outgrowth of earlier work on the dynamics of least squares learning systems.

Learning and equilibrium computation¶

Learning algorithms and equilibrium computation algorithms look like each other. Equilibrium computation algorithms often have interpretations as centralized learning algorithms whereby the model builder, acting in a role of ‘social planner,’ gropes for a set of pricing functions for markets and decision rules for agents that will satisfy all of the individual optimum conditions and market-clearing conditions. We have also seen that learning systems with boundedly rational agents sometimes have interpretations as decentralized equilibrium computation algorithms.

Recursive kernel density estimation¶

A continuous state (for $G_t$) specification in the present model is a convenient context for describing another way to formulate learning nonparametrically, namely, via recursive kernel estimators of a kind studied by Chen and White (1993). To describe their formulation, we first recall the nature of kernel estimators. Suppose that we have $T$ observations $x_t$, $t = 1, \ldots, T$, on the $n$-dimensional random vector $x$ drawn from an unknown joint density $F(x)$. Let $K(x): \mathbb{R}^n \to \mathbb{R}$ be a probability density for $x$, say a multivariate normal density. Then the kernel estimator of the density of $x$ is

where $h > 0$ is a fixed ‘bandwidth’ parameter.

Chen and White study a modified recursive version of such estimators. They let $\{h_t\}_{t=0}^\infty$ be a sequence of bandwidths with $h_t \searrow 0$, and $\hat{F}_0(x)$ be an arbitrary initial density. Then they construct the sequence $\{\hat{F}_t(x)\}$ of distributions via the stochastic approximation algorithm

For the present example, we could let $x_t = [R_t, G_t]'$. At time $t$, we would let behavior be determined by the solution of a version of (31) in which the conditional expectation on the right side is evaluated with respect to the conditional distribution for $R_t$ conditional on $G_t$ that can be deduced from the joint density $\hat{F}_{t-1}(x)$.^[15]

Learning in a model of the exchange rate¶

We now study an environment for which the rational expectations equilibrium exchange rate is indeterminate, but we expel all rational agents and replace them with adaptive agents.^[16] We endow these adaptive agents with learning algorithms and initial values for their decisions, which serve to render the exchange rate and all other endogenous variables determinate. We want to study how the exchange rate behaves, and whether a ghost of indeterminacy still lurks.

The economy consists of a sequence of overlapping generations of two-period lived agents. There are two kinds of currency, available in supplies $H_1$ and $H_2$ that are fixed over time. At each date $t \geq 1$, there are born a constant number $N$ of young agents who are endowed with $w_1$ units of a nonstorable consumption good when young, and $w_2$ units when old. A young agent makes two decisions. First, he chooses an amount $s_t$ to save when young. Second, he chooses a fraction $\lambda_t$ to allocate to currency 1, and allocates the remainder to currency 2. At time $t$ the young agent’s realized utility from those decisions will be

where $p_{it}$ is the price level at time $t$ in terms of currency $i$, and $p_{it}/p_{it+1}$ is the gross rate of return on currency $i$. Here $u(\cdot)$ is an increasing and strictly concave function of consumption of the one good. In the examples below, we shall set $u(c) = \ln(c)$. We calculate the gradient

We also use the elements of the matrix of second partials.^[17] For the purposes of studying learning, the economy consists of two subsequences of agents. Each subsequence is identified with a class of agents, whom we dub ‘odd’ and ‘even.’ As in the preceding model, we have two clocks $\tau(t)$, one for odd and the other for even agents, that count only two-period episodes used to evaluate realized utility. Let $\gamma_\tau$ be a sequence of positive numbers satisfying $\lim_{\tau \to \infty} \gamma_\tau = 1$. The values of $(s_\tau, \lambda_\tau)$ for each class of agents evolve according to the recursive algorithm

There are two realizations of this algorithm, one for the odd agents, the other for the even agents. Agents of each class thus learn from the utility experience only of previous agents of their own class.^[18] The price level is determined by

In odd periods, the $(s_\tau, \lambda_\tau)$ pair for the odd agents is used in (51), while in even periods, the $(s_\tau, \lambda_\tau)$ pair for even agents is used in (51) to determine price levels in terms of the two currencies.

Given initial conditions for $(R_{\tau-1}, s_\tau, \lambda_\tau)$ for each class of agents, equations (46), (49), (51) determine the evolution of the decisions $\lambda_t$, $s_t$ and the prices $p_{it}$ in terms of the two currencies. The exchange rate is just $p_{1t}/p_{2t}$.

Figure 8 and Figure 9 report the results of simulating this system starting from two different sets of initial conditions for $(s, \lambda)$, but common initial conditions for the $R$’s.^[19] For each experiment, the exchange rate path rapidly converges to a constant value, but the limiting exchange rate values differ between the two economies. Evidently, the exchange rate depends sensitively on the initial conditions that we choose. Figure 10 shows saving rates for the two types of agents in experiment 1, while Figure 12 displays the evolution of their portfolio parameters $\lambda$. All of these parameters are gradually converging to values consistent with a rational expectations equilibrium.

Exchange rate initial-condition dependence¶

For the purpose of understanding the sense in which this system can render the exchange rate determinate, it is useful to consider the limiting properties of algorithm (49). If algorithm (49) comes to rest, it will, as it is designed to do, come to rest at a point $(\lambda, s)$ at which $[\partial U / \partial s \ \partial U / \partial \lambda] = [0 \ 0]$. The condition $\partial U / \partial \lambda = 0$ implies $p_{1t}/p_{1t+1} = p_{2t}/p_{2t+1}$, or $R_{1t} = R_{2t}$, where $R_{it}$ is the gross rate of return on currency $i$. This is the same arbitrage condition that leads to exchange rate indeterminacy under rational expectations. The reasoning that underlies the exchange rate indeterminacy under rational expectations also implies that the rest points of algorithm (49) leave the exchange rate unrestricted. Our learning system renders the exchange rate path determinate by having the ‘dead hand of history’ put enough sluggishness into decisions. The exchange rate path can be said to be ‘history-dependent’ because the initial conditions assigned to $(\lambda, s)$ assume an importance that does not vanish as time passes.^{[20] [21]}

The no-trade theorem¶

Jean Tirole (1982) proved a sharp ‘no-trade’ theorem that characterizes rational expectations equilibria in a class of models of purely speculative trading.^{[22] [23]} The equilibrium market price fully reveals everybody’s private information at zero trades for all traders. The no-trade theorem overrules the common-sense intuition that differences in information are a source of trading volume.

The remarkable no-trade outcome works the rational expectations hypothesis very hard. This is revealed clearly in Tirole’s proof, which exploits elementary properties of the (commonly believed) probability distribution function determined by a rational expectations equilibrium. In this section, I describe how backing off rationality can (temporarily) undo the no-trade result, and produce a model of trading volume. I first describe an environment for which the no-trade theorem holds under rational expectations, then withdraw Tirole’s rational agents and replace them with Robbins--Monro adaptive agents.

The environment¶

The environment is one analyzed in detail by John Hussman (1992).^[24] There is a competitive market for a stock that is a claim on a dividend process $\{d_t\}$ governed by

where $|\rho_1| < 1$, $|\rho_2| < 1$, and $[\epsilon_t \ \nu_{1t} \ \nu_{2t}]$ is a vector white noise. There are two classes of traders, dubbed $a$ and $b$, present in equal numbers (for convenience we’ll assume one each) who have different information about dividends. At time $t$, traders of both classes observe the history of the publicly available information $\{p_s, d_s; s < t\}$. In addition, traders of classes $a$ and $b$, respectively, observe the pieces of ‘private information’:

where $(\eta_t^a, \eta_t^b)$ are white noises that are orthogonal to each other and to each component of $[\epsilon_t \ \nu_{1t} \ \nu_{2t}]$. At time $t$, traders of class $j$ observe a history generated by the information vector $z_{jt} = [p_t \ s_t^j \ d_t]'$.

Traders behave myopically, each period maximizing the one-period utility function

subject to

where $R$ is a constant gross interest rate on a risk-free asset, $I_{jt}$ is agent $j$’s information set, and $q_t^j$ is agent $j$’s purchases. This leads to a demand function for trader of class $j$ that is linear in the expected ‘excess return’:

where $\phi_j = \phi / \sigma_{p_{t+1}+d_{t+1}|I_{jt}}^2$; $\sigma_{p_{t+1}+d_{t+1}|I_{jt}}^2$ is the variance of $p_{t+1} + d_{t+1}$ conditional on the information set $I_{jt}$; and $(p_{t+1} + d_{t+1} - R p_t)$ is the excess return of the stock over the risk-free asset.