9 Properties of Estimators

When the concept of estimators was first introduced in Section 8.1.2 we mentioned that there is nothing in the definition that guarantees—or even alludes to—good performance. Let’s now start to think about evaluating estimators.

9.1Sampling Distributions¶

Our first step is to view estimators as random entities dependent on the data, and to consider their distributions under different specifications of the data-generating process. The notation below follows that set out in Section 8.1.1.

9.1.1Estimators as Random Elements¶

By definition, a statistic $T$ is a $\bB$-measurable transformation of the sample space $\Zdata$ into some feature space $S$. As a function of the sample $\zdata = (\boldz_1, \ldots, \boldz_N)$, it can be viewed as a random element mapping outcomes in the underlying space $\Omega$ to $S$ via

Here "random element" stands for either a random vector (or scalar), or something more general, such as the random distribution $\hat P_N$, or a density estimate, which is a random function of the sample.

The idea that estimators are random can be counterintuitive. When we begin to study econometrics, we tend to think of data as fixed numbers determined by some prior historical outcome and statistics as deterministic functions of these numbers. Hence we only observe one value of any particular statistic—one sample mean, one sample variance, and so on. However, if we want to build a theory of estimators, then we need to consider their performance prior to observing the data. Each data set is viewed as a random draw from some unknown distribution. We seek estimators that perform well in the face of this uncertainty.

Once we agree that estimators are random, we know they must have distributions. These distributions determine how effective an estimator is at estimating the intended feature. Let’s now turn to investigating these distributions.

9.1.2Sampling Distributions¶

Suppose that $T$ is a statistic mapping data $\zdata$ into $S$. As just discussed, $T(\zdata )$ is a random element taking values in $S$. Its distribution $\lL(T)$ on $S$ can be expressed as

$\lL(T)$ is called the sampling distribution of $T$. Continuing to write $\Pdata$ for the joint distribution of the sample $\zdata$, we can also define the sampling distribution as

or even $\Pdata \circ T^{-1}$ if you like that kind of notation. Note that the sampling distribution of $T$ is fully determined by

Since $\Pdata$ is unknown, the sampling distribution is likewise unknown. It can, however, be estimated, as we’ll see below.

Mathematically, there is no difference between the "sampling distribution of $T$" and the "distribution of $T$." Nonetheless, the terminology is standard.

Sampling distributions are often complicated. Sometimes the best we can do is obtain the sampling distribution of some transformation of an estimator.

To see why (9.4) holds, recall from example 2.2.3 that if $S := \Span\{\boldone\}$ and $\boldP = \proj S$, then the residual projection satisfies $\boldM \boldx = \boldx - \bar x \boldone$. Hence

Since $\boldM \boldone = \boldzero$ (see fact 2.2.8) and $\boldM$ is idempotent and symmetric (fact 3.3.2), setting $\boldxi := \sigma^{-1} (\boldx - \mu \boldone)$ gives

To see that the right-hand side has distribution $\chi^2(N - 1)$, apply fact 3.3.4 to obtain $\trace \boldM = N-1$, followed by fact 5.1.19.

Many long and involved proofs of fact 9.1.1 can be found in the literature, but a simple and informative proof can be obtained from fact 5.1.13. See exercise 9.4.2 and its solution.

In the preceding examples we were able to obtain an analytical expression for the sampling distribution of our estimator (or a simple transformation of the estimator) from the underlying distribution of the data. In reality this case is the exception.

When no closed-form exists, we can still approximate the sampling distribution using simulation, conditional on a specification of the joint distribution of the data. For example, figure 9.1 shows a histogram approximation to $\lL(\bar x_N)$ when $N=20$ and $x_1, \ldots, x_N$ is IID $\lN(0, 1)$. The histogram is produced by generating 10,000 observations of $\bar x_N$. The loop that produced the observations is given in listing 5.

import numpy as np

N = 20
num_reps = 10000
xbar_outcomes = np.empty(num_reps)  # Allocate memory

for i in range(num_reps):
    x = np.exp(np.random.randn(N))  # Generate N iid lognormal RVs
    xbar_outcomes[i] = x.mean()

9.1.2.1Desirable Properties¶

The quality or performance of a given estimator is characterized by its sampling distribution. Let $\hgamma$ be an estimator of some feature $\gamma$ based on a sample $\zdata$. What would be an ideal sampling distribution for $\hgamma$? The answer is obvious: a point mass that puts probability one on the single point $\gamma$.

This scenario rarely arises because our sample is random. So here’s a more realistic objective:

It is desirable that the sampling distribution of an estimator of $\gamma$ concentrate most of its probability mass in a small neighborhood around $\gamma$, regardless of the joint distribution $\Pdata$ of the sample.

This statement considers only what is desirable, not what is possible. For example, if we look back at figure 9.1, we see that the sampling distribution of $\bar x_N$ does not cluster tightly around the mean. (The feature $\gamma$ that we are estimating is the mean of the $\lN(0, 1)$ distribution, which is represented by the vertical black line.) But this is not unexpected, since the sample size is only 20. We simply don’t have enough information to estimate the feature precisely.

Nonetheless, these ideas give us a way to start comparing different estimators of a given feature from a given amount of data. For example, suppose that we are estimating the mean of a distribution from a random sample of size $20$. Suppose that each is drawn from the $\lN(0, 1)$ distribution, the mean of which is about 1.65. The sampling distributions of three estimators of the mean under the associated joint distribution have been approximated by simulation and represented as box plots in figure 9.2. The top sampling distribution is that of the mid-range estimator, the second is that of a maximum likelihood estimator to be discussed later, and the third is that of the sample mean. The last two put more probability mass in the region around 1.65. Hence they outperform the mid-range estimator in this setting.

In what follows we will formalize the ideas presented above, including both finite sample and asymptotic theory. But first let’s discuss a general way of estimating sampling distributions from the data.

9.1.3The Bootstrap¶

While understanding sampling distributions is central to evaluating the precision of estimators, sampling distributions cannot be observed in the wild because they depend on $\Pdata$. For example, suppose $x_1, \ldots, x_N$ are assumed to be IID $\nN(\mu, \sigma^2)$ with $\mu$ and $\sigma$ unknown. By (9.3), the distribution of $\bar x_N$ is $\nN(\mu, \sigma^2/N)$. While this provides valuable information, it doesn’t fully pin down the distribution because $\mu$ and $\sigma$ are unknown.

Things are even more complicated for the sample mean of IID lognormals in example 9.1.5. There we can’t easily write down the sampling distribution as a function of the parameters. Problems multiply further if we are not prepared to assume a parametric form for the distribution of the observations.

There is, however, a very general technique that allows us to approximate sampling distributions given data. We’ll discuss the case of scalar data, although the vector case is similar. To begin, let $T$ be any statistic and consider its sampling distribution

The sampling distribution is unknown to us because $P$ is unknown. Suppose, however, that we have access to a sample $x^o_1, \ldots, x^o_N$ of IID draws from $P$. We then have a good estimator of $P$ given our sample: the empirical distribution $\hat P_N$ generated by $x^o_1, \ldots, x^o_N$. Given this empirical distribution, the bootstrap distribution of $T$ is defined as

Thus the bootstrap distribution is just the plug-in estimator of the sampling distribution.

Is $\lL_{\hat P_N}(T)$ a good approximation to $\lL_P(T)$? By the Glivenko–Cantelli theorem, $\hat P_N$ will be close to $P$ when $N$ is large. If $Q \mapsto \lL_Q(T)$ is suitably continuous, then $\lL_{\hat P_N}(T)$ will also be close to $\lL_P(T)$. For details, see §29.2 of DasGupta (2008).

9.1.3.1Simulation¶

One remaining problem with $\lL_{\hat P_N}(T)$ is that it’s not particularly tractable. We can, however, generate draws from this distribution easily on a computer. Here’s an algorithm that returns $M$ draws from $\lL_{\hat P_N}(T)$:

• set $\hat P_N =$ the empirical distribution of observed data $x^o_1, \ldots, x^o_N$

• for $m$ in $1, \ldots, M$ do

  • draw $x_1^b, \ldots, x_N^b$ independently from $\hat P_N$

  • set $T^b_m = T(x^b_1, \ldots, x_N^b)$

• end

• return the sample $T_1^b, \ldots, T_M^b$

Using the definition of the empirical distribution, drawing samples from $\hat P_N$ is easy: we just make repeated draws from the set $x^o_1, \ldots, x^o_N$ with equal probability on each element. Listing 6 contains some Julia code implementing the whole algorithm as a function that returns $M$ bootstrap samples. The other arguments to the function are xo, the array of observed data, and stat, which is a function representing $T$.

function bootstrap(xo, stat, M)
    N = length(xo)
    T_b = Vector{Float64}(undef, M)
    x_b = Vector{Float64}(undef, N)
    for m in 1:M
        for i in 1:N
            x_b[i] = xo[rand(1:N)]
        end
        T_b[m] = stat(x_b)
    end
    return T_b
end

Here’s an example function call:

julia> bootstrap([1, 2, 3], mean, 3)
3-element Vector{Float64}:
 1.6666666666666667
 1.0
 1.6666666666666667

Here is the same routine in Python:

def bootstrap(xo, stat, M):
    N = len(xo)
    T_b = np.empty(M)
    for m in range(M):
        x_b = np.random.choice(xo, size=N)
        T_b[m] = stat(x_b)
    return T_b

Figure 9.3 shows a histogram of observations from a bootstrap distribution generated using the Julia code above. The underlying sample $x^o_1, \ldots, x^o_N$ was created using $N=200$ IID draws from the $\lN(0, 1)$ distribution. The statistic used was the sample median. The density shown in the same figure is a numerical approximation of the true sampling distribution of the median over $200$ IID draws from the $\lN(0, 1)$ distribution. The (population) median of the $\lN(0, 1)$ distribution is 1.

9.1.3.2Standard Errors¶

As discussed in Section 9.1.2.1, a good estimator is one that concentrates its probability mass close to the target feature. The extent to which probability mass concentrates is usually measured by the variance or standard deviation of the sampling distribution. The standard deviation is preferable because it comes in units that are easier to interpret.

Because the sampling distribution is typically unknown, so is its standard deviation. Hence we replace the true standard deviation with an estimate. An estimate of the standard deviation of a sampling distribution is usually referred to as the standard error. More generally,

This isn’t a proper definition because we haven’t specified which estimate of the standard deviation we are talking about, but nevertheless the terminology is common. One way to compute a standard error of an estimator is to take the sample standard deviation of the bootstrap draws.

9.2Evaluating Estimators¶

As mentioned above, the standard definition of an estimator does not include any restrictions that imply sensible behavior (as opposed to, say, the definition of a probability). We now turn again to the problem of assessing estimators, following our informal discussion in Section 9.1.2.1. We’ll begin with finite sample properties.

9.2.1Bias¶

One common criterion for assessing estimators is bias, or lack thereof. Let $\hgamma$ be an estimator of a given feature $\gamma = \gamma(P)$. The bias of $\hgamma$ is defined as

The estimator $\hgamma$ is called unbiased for $\gamma$ over the class of distributions $\pP$ if

The definition is the same when the feature $\gamma$ and estimator $\hgamma$ are vector-valued.

The notation $\EEP \hgamma$ indicates that expectation is taken under the assumption that the data points are IID with common distribution $P$. In general, the bias depends on $P$. For example, in the IID setting, the mid-range estimator is unbiased as an estimator of the mean of a uniform distribution on $\RR$ with unknown end points, but biased for many other distributions, including the lognormal.

Figure 9.2 already suggests bias of the mid-range estimator in the lognormal case. Here’s a simulation in R that provides further confirmation.

> mr <- function(x) return((min(x) + max(x)) / 2)
> observations <- replicate(5000, mr(rlnorm(20)))
> mean(observations)   # Approximate sample mean of mid-range
[1] 3.800108
> exp(1/2)      # Mean of lognormal
[1] 1.648721

Fact 9.2.1 is immediate from linearity of expectations, since

Note that independence isn’t used here. Fact 9.2.1 depends only on the existence of a common marginal $P$.

To obtain this expression, we can combine the definition of $q_N$ in (9.4) and the expression in (9.5) to obtain

Here $\boldM$ is the residual projection for the subspace $\Span\{\boldone\}$ and the second equality is due to fact 5.1.3. Now apply fact 3.3.4.

9.2.2Variance¶

The variance of a given estimator gives a measure of its dispersion. Sensible estimators almost always have the property that the variance goes to zero as $N \to \infty$.

Is small variance always desirable for an estimator? If the estimator is biased, then perhaps not, since probability mass might be concentrated in the wrong place. But for an unbiased estimator, low variance is good. It means that probability mass is concentrated around the feature that we wish to estimate.

Low variance is, of course, relative. For example, is the variance in (9.8) low or high? One way to approach this kind of question is to take the class of unbiased estimators of a given feature $\gamma$, and find the estimator in the class with the lowest variance. For given $\gamma$ and given data $\boldx_1,\ldots,\boldx_N$, the estimator in the set of unbiased estimators

that has the lowest variance within this class is called the minimum variance unbiased estimator.

Of course, it’s quite possible that no minimizer exists. Even if it does, it might be hard to determine in practice, or require strong assumptions on the unknown distribution of the data. For these reasons it’s common to focus on smaller classes than $U_{\gamma}$. For example, the estimator in the set of linear unbiased estimators

with the lowest variance—if it exists—is called the best linear unbiased estimator.

The theory of minimum variance unbiased estimators and best linear unbiased estimators is traditionally regarded as important. However, there’s no convincing reason to restrict ourselves to unbiased estimators. In fact, as we’ll see, there are good reasons not to.

9.2.3Variance versus Bias¶

The mean squared error (MSE) of a given estimator $\hgamma$ of some feature $\gamma \in \RR^K$ is

As for bias, $\mse(\hgamma, \gamma)$ depends on the joint distribution of the data as well as the specification of $\hgamma$, although we haven’t made that dependence explicit in the notation.

Mean squared error is an overall measure of the performance of an estimator that takes into account both bias and variance. In fact in the scalar case we can neatly decompose mean squared error into the sum of variance and squared bias:

This is just (6.3) specialized to the current setting.

As can be seen from both (9.9) and (9.10), when $\mse (\hat \gamma, \gamma)$ is small for a given estimator $\hgamma$, the estimator concentrates probability mass around the target feature $\gamma$. Hence minimizing MSE is a natural and reasonable criterion. In doing so, we are making our subjective loss function explicit, while choosing a loss function that is simple and highly tractable.

When minimizing MSE, we typically find a trade-off between bias and variance: lower variance costs more bias and vice-versa. Moreover solutions usually turn out to be interior. In particular, we can often reduce variance significantly by accepting a small amount of bias.

For most sensible estimators, $v$ converges to zero as the sample size $\to \infty$. The message is that unbiased estimators only approach optimality when the data size is large. Here "large" should be understood as relative to model complexity. More complex models need more data to be estimated effectively.

9.2.3.1Stein’s Example¶

There is a famous example in the theory of statistics due to Stein (1956) that pertains to so-called admissible estimators. Here we’ll borrow Stein’s problem and proposed solution to illustrate several ideas, including the relationship between bias and mean squared error.

To make things concrete, consider the following application: In the field of public finance, one topic of interest is ability-based taxation, which has optimality properties in terms of welfare in certain models. Ability is not observable. Suppose, in particular, that we have a population of $K$ individuals, where the $k$th individual has idiosyncratic ability $\mu_k \in [0, 1]$. For each agent, we get to observe $N$ noisy signals of ability, each of the form $x_{kn} = \mu_k + \epsilon_{kn}$, $n=1, \ldots, N$. The noise terms $\epsilon_{kn}$ are all independent $\nN(0, \sigma^2)$. Our task is to estimate the parameter vector $\boldmu = (\mu_1, \ldots, \mu_K)$ given the data from the observations $x_{nk}$. To make things simpler we’ll assume that only $\boldmu$ is unknown.

A natural estimator of $\boldmu$ is the vector sample mean

This is the estimator produced by applying the sample analogue principle to our problem. It’s also the maximum likelihood estimator and best linear unbiased.

What Stein showed is that, whenever $K > 2$, there exists an alternative estimator $\hboldmu_N$ such that

In other words, $\hboldmu_N$ uniformly dominates $\bar \boldx_N$ in terms of MSE, for all possible values of $\boldmu$! The estimator in question is what is now known as the James–Stein estimator. It is defined as

This estimator is biased. It achieves lower mean squared error by reducing variance at the expense of a small amount of bias. A relatively simple proof of (9.12) can be found on p. 34 of Young & Smith (2005).

We can get some of the intuition from simulation. In our simulation we set $K=200$, $N=5$ and $\sigma=1$. Thus we have 5 noisy observations on the ability of each of 200 agents. For each agent we choose their ability $\mu_k$ independently from a uniform distribution on $[0, 1]$ and next generate the disturbances $\epsilon_{kn}$ as independent standard normals. We then compute the estimators $\bar \boldx_N$ and $\hboldmu_N$ and the corresponding error vectors.

Figure 9.5 shows a typical realization. The bars in the top panel are the elements of the vector $\bar \boldx_N - \boldmu$, which records the errors for the sample mean estimator $\bar \boldx_N$. Those in the lower panel correspond to the errors of the James–Stein estimator. These are noticeably downward biased and also noticeably smaller than the errors of the sample mean estimator. This is reflected in the SSE values in the top left of each panel, which give the sum of squared errors (i.e., the squared norms of the error vectors). The SSE for the sample mean estimator is nearly twice as large.

9.2.4Asymptotic Properties¶

Let’s now turn to asymptotic properties of estimators. Asymptotic theory concerns whether or not the sampling distribution of a given estimator increasingly concentrates around the feature we wish to estimate, the rate at which this occurs, and the shape of the sampling distribution in large samples.

A fundamental concept in asymptotic statistics is consistency. Let $\{\hgamma_N\}$ be a sequence of estimators of a given feature $\gamma$, based on a sample of size $N$. We say that $\hgamma_N$ is consistent for $\gamma$ if

The definition is the same whether the estimators are vector or scalar-valued. The statement that $\hgamma_N$ is consistent means more correctly that the sequence $\{\hgamma_N\}$ is consistent.

As for bias, whether consistency holds depends not just on $\hat \gamma_N$ but also on the joint distribution of the data. If we say that $\hgamma_N$ is consistent over a class of distributions $\pP$, we mean that (9.13) holds true for any distribution in $\pP$.

More generally, $\hgamma_N = \frac{1}{N} \sum_{n=1}^N h(\boldx_n)$ is consistent for $\gamma = \int h(\bolds) P(\diff \bolds)$ whenever

(See (6.7).) For example, the $k$th sample moment is consistent for the $k$th moment over the class of distributions with finite $k$th moment.

Here’s an example of consistency that requires a bit more work.

Fact 9.2.2 is immediate from fact 6.1.1.

Reasonable estimators should be consistent, at least for the IID case. In fact, as we saw in theorem 8.1.1, the entire unknown distribution can be recovered asymptotically with repeated independent samples.

9.2.4.1Asymptotic Distributions¶

In the IID Gaussian setting of example 9.1.3, the distribution of the sample mean is $\nN(\mu, \sigma^2 / N)$. Beyond the information contained in the mean and variance, the fact that the sampling distribution is normal gives us important additional knowledge. We know that the distribution is symmetric, that it has thin tails, and that it puts around 95% of its mass within two standard deviations of the mean. We can determine probabilities of intervals or the expectation of any function of $\bar x_N$.

The only concern is our assumption that the observations are Gaussian. Economic theory rarely provides any convincing basis for this assumption. Fortunately, we have the central limit theorem to fall back on, which tells us (remarkably!) that the sample mean is always asymptotically Gaussian, provided that the underlying observations have finite second moment. Even better, the same approximation is valid for many estimators beyond the sample mean.

To clarify, let $\{\hgamma_N\}$ be a sequence of estimators for some feature $\gamma$. We say that $\hgamma_N$ is asymptotically normal if there exists a positive definite matrix $\Sigma$ such that

When (9.15) holds, $\Sigma$ is called the asymptotic variance–covariance matrix of $\hgamma_N$.

Asymptotic normality also implies a rate at which $\hgamma_N$ converges to $\gamma$. Indeed, if $\hgamma_N$ is asymptotically normal, then $\sqrt{N} ( \hgamma_N - \gamma)$ does not diverge. This means that $\hgamma_N - \gamma$ goes to zero fast enough to offset the diverging term $\sqrt{N}$. To emphasize this point, we say that an asymptotically normal estimator $\hgamma_N$ is $\sqrt{N}$-consistent for $\gamma$.

Still more significant is the fact that asymptotic normality gives us an approximation to the entire sampling distribution. This will provides us with a means of performing inference. Chapter 10 gives details.

9.2.5Decision Theory¶

We’ve now reviewed the most common methods of assessing estimators. Next we’ll look more deeply at the theory of estimation and decisions under uncertainty. Our focus will be on finite sample properties. This is the most interesting case, since it captures our need to generalize on the basis of limited information.

Despite the efforts of many brilliant statisticians and econometricians, we still lack a convincing general theory of optimality of estimators that rests on purely objective criteria. For example, while the theory of minimum variance unbiased estimators initially seems attractive because it can potentially yield estimators without any subjective judgment on the part of the statistician, there are simple settings where minimum variance unbiased estimators are inadmissible, which means that they fail a minimal requirement for what constitutes a sensible estimator (see below).

Without a fully objective theory of "good" estimators, there are competing notions of what good means, and we are forced to take a stand on what our criteria for assessing estimators are in different situations. This means that the estimation problem is not fully specified until we make our preferences explicit.

For economists it’s natural to think of specifying preference in terms of utility or loss. To fit with modern statistical terminology, we will think in terms of loss. Thus a full description of an estimation problem includes specification of the loss incurred when an estimator performs poorly.

Let’s frame these ideas in line with the seminal work of Wald (1939) and subsequent research on decision theory. A general decision problem consists of

The state of the world is treated as unknown. The problem is to choose a decision rule $d \in \ddD$ that generates low loss in some sense to be made precise. Central to this discussion is the notion of risk, which is defined as

The integral is over all $x \in \xX$. The notation $\risk$ is used to distinguish this decision-theoretic concept of risk from $R(f)$, the prediction risk of policy $f$ as defined in (8.16).

The problem of choosing estimators is a special case of the decision theory described above. Suppose that we observe data $\zdata$ taking values in $\Zdata$ with joint distribution $P_\theta$ indexed by $\theta \in \Theta$. (The set $\Theta$ can be infinite dimensional and $\theta$ is just an index.) We wish to estimate some $S$-valued feature $\gamma_\theta = \gamma(P_\theta)$ using this data. An estimator in this context is a decision rule $\hat \gamma$ mapping $\Zdata$ to the action space $\aA = S$. If we specify the loss $L(\hat \gamma, \theta)$ of choosing $\hat \gamma$ when the state of the world is $\theta$, our risk becomes

In the preceding example, the risk of the estimator is its MSE. More generally, if $\hat \gamma$ and $\gamma$ in (9.20) are vectors and $L$ measures squared vector deviation, then $\risk(\hat \gamma, \theta) = \mse(\hgamma, \gamma)$. Thus, risk gives us a generalization of the idea of evaluating estimators based on MSE, as described in Section 9.2.3. Since loss is subjective, there is no reason we should confine ourselves to quadratic loss.

9.2.5.1Choosing Decision Rules¶

In the current setup, a natural idea for making optimal decisions—and for choosing optimal estimators, in particular—is to choose decision rules with low risk. The problem with this approach is that the risk depends, in general, on the unknown state of the world $\theta$, and hence the risk-minimizing decision rule also depends on the unknown value $\theta$. In other words, we face Knightian uncertainty. We can’t evaluate risk because we don’t know the probabilities.

This example illustrates the fact that we cannot, in general, choose estimators based on lowest risk in a classical setting. One useful thing we can do, however, is exclude estimators that are always dominated in terms of risk. These are the so-called inadmissible estimators. In particular, a decision rule $d$ is called inadmissible for the general decision problem if there exists a second rule $e$ such that

A decision rule that is not inadmissible is called admissible

Another idea that comes out of the desire to minimize risk is minimax. A decision rule $d_m$ is called a minimax rule if

A minimax rule is one that performs well in the worst possible state of the world. Minimax decision rules are an important component of robust control and the study of policy under uncertainty. See, for example, Hansen & Sargent (2008). On the other hand, minimax decision rules can be inadmissible (see, e.g., Young & Smith (2005), p. 9).

9.2.5.2Bayes Rules and Bayes Risk¶

The Bayesian approach to statistics leads naturally to a coherent and attractive decision theory. The horse race is not entirely fair, however, since we are granted additional information expressed as a prior. Let’s review the main ideas.

Consider again the generic decision problem discussed above, but now with a prior distribution $\pi = \pi(\theta)$ over states of the world. In addition each $P_\theta$ is now restricted to be a density $p(x \given \theta)$. As in Section 8.3.4, we obtain the posterior for $\theta$ from the prior given data $x$ via Bayes’ law:

Given decision rule $d$ with risk function $\risk(d, \theta)$, the posterior risk or Bayes risk of $d$ is

A decision rule $d$ that minimizes the Bayes risk is called a Bayes rule. If $d$ is an estimator, then a Bayes rule is called a Bayes estimator. Existence of such rules is not guaranteed, but in many practical instances they are well defined and unique. We give some simple examples below.

There are two particularly attractive features of Bayes rules, and Bayes estimators in particular. One is that Bayes rules are always admissible under mild regularity conditions. See, for example, theorem 10.4.2 of Casella & Berger (2002). A second is that Bayes rules can be computed as a function of observed data, meaning that we don’t have to concern ourselves with how to act in situations that potentially never occur.

To understand the second point, note that we can write the joint density of $(x, \theta)$ as either $p(x \given \theta) \pi(\theta)$ or $\pi(\theta \given x) p(x)$. Using a change in order of integration followed by this identity, the Bayes risk can be expressed as

It follows from this last expression that $d$ will be a Bayes rule whenever

This is akin to the scenario encountered in dynamic programming after applying Bellman’s principle of optimality: once we know the value function, we can choose the optimal action just by responding to any observed state.

9.3Further Reading¶

There are many excellent general texts on statistical inference, including Casella & Berger (2002), Wasserman (2013), and Young & Smith (2005). For a discussion of estimation theory from the perspective of machine learning, see Friedman et al. (2009) or Abu-Mostafa et al. (2012).

9.4Exercises¶

9.4.1Solutions to Selected Exercises¶