10 Confidence Intervals and Tests

Until now we’ve focused primarily on point estimation. Since data are not infinite, point estimation involves uncertainty. We should not be overconfident in our claim to know the data-generating process. Rather, we should be only as confident as the data allows us to be.

We discussed uncertainty previously when we considered sampling distributions. In this chapter we further quantify estimation uncertainty using the language of confidence sets. We also cover hypothesis tests, which are a second key method of inference in classical statistics.

To tie in with standard presentations we’ll switch to parametric language. The notation is as except that

However, we won’t actually restrict ourselves to the case of finitely many parameters. Instead, $\theta$ can be thought of more generally as an index over the set of distributions.

10.1Confidence Sets¶

Confidence sets are random subsets of model space that contain the true data generating process with high probability. Let’s start with the key definitions and then move on to examples.

10.1.1Finite Sample Confidence Sets¶

Fix $\alpha \in (0,1)$. A random set $C(\zdata) \subset \Theta$ is called a $1 - \alpha$ confidence set for $\theta$ if $C(\zdata)$ is observable given the data and, for any $\theta \in \Theta$,

Note that it’s the set that’s random here, not $\theta$.

Confidence sets that are intervals are usually called confidence intervals.

Suppose, for example, that we want to estimate the mean of an unknown distribution based on a set of observations $\zdata$. If we can come up with a 95% confidence set $C(\zdata)$ in the sense of (10.1), then we know that the set will contain the mean in about 95% of our experiments, regardless of the underlying distribution.

The last part of the sentence is essential: We don’t know what $\theta$ is, and so the set must be designed such that the probability of containing $\theta$ exceeds $1 - \alpha$ regardless of which $\theta$ is generating the data. The difficulty is to actually construct a confidence set with this property for a reasonably large class of distributions. Here’s a traditional approach that uses strong parametric assumptions.

In example 10.1.1, the assumption that $\sigma$ is known is clearly unrealistic. If $\sigma$ is unknown, a natural approach is to replace this term with the sample standard deviation $s_N$. In doing so, we will use the following fact, which exercise 10.4.1 asks you to prove.

The random variable (10.3) is called a pivot, which means that its distribution does not depend on the unknown parameters.

Recall that the standard deviation of $\bar x_N$ is $\sigma/ \sqrt{N}$. The term $s_N/\sqrt{N-1}$ is a sample estimate of $\sigma/ \sqrt{N}$. Following the discussion in Section 9.1.3.2, we call $s_N/\sqrt{N-1}$ the standard error of $\bar x_N$ and write it as $\se(\bar x_N)$. We can then write the confidence interval in example 10.1.2 as

10.1.2Asymptotic Methods¶

While strong parametric assumptions of the form used in examples 10.1.1 and 10.1.2 are difficult to justify in most econometric scenarios, there is another approach to forming confidence intervals that requires much less in terms of structure imposed on the underlying distributions, provided that the sample size is relatively large. Let’s start with a definition: given $\alpha \in (0,1)$, a set $C_N(\zdata) \subset \Theta$ is called an asymptotic $1 - \alpha$ confidence set for $\theta$ if $C_N(\zdata)$ is observable given the data and

for all $\theta \in \Theta$. Note that $C_N(\zdata)$ is really a sequence of sets and the definition concerns this sequence.

How can we create asymptotic confidence sets? Let’s consider first the case of scalar $\theta$. Suppose that we have an estimator $\hat \theta_N$ of $\theta$ that is asymptotically normal for all $\theta \in \Theta$. Specializing the results in Section 9.2.4.1 to the scalar case, this means that, for all $\theta \in \Theta$, there exists a positive constant $v(\theta)$ such that

The constant $v(\theta)$ is called the asymptotic variance of $\hat \theta_N$. Suppose we have a sequence of statistics $\se(\hat \theta_N)$ such that

As exercise 10.4.2 asks you to show, (10.6) and (10.7) imply that

We can now create our asymptotic confidence intervals as

To see this, take the limit and apply (10.8) to get

10.1.3A Nonparametric Example¶

Here’s an example of a nonparametric confidence set. In Section 8.1.3 we learned that if $\boldx = (x_1, \ldots, x_N)$ is a vector of IID draws from some CDF $F$ and $\hat F_N$ is the corresponding ECDF, then $\| F - \hat F_N \|_{\infty}$ converges in probability to $0$. In 1933, A. N. Kolmogorov used an extension of the central limit theorem to obtain an asymptotic distribution for this term. In particular, he showed that when $F$ is continuous,

where $K$ is the Kolmogorov CDF

As in the CLT, the limiting distribution $K$ is independent of the CDF $F$ that generates the data.

We can use (10.11) to produce an asymptotic $1-\alpha$ confidence set for $F$. To do so, let $\mathfrak F$ be the set of all CDFs on $\RR$, let $x_1,\ldots,x_N$ be IID draws from $F \in \mathfrak F$, let $k_{1-\alpha} := K^{-1}(1 - \alpha)$, and let

The set $C_N(\boldx) \subset \mathfrak F$ is an asymptotic $1-\alpha$ confidence set for $F$. Indeed, after rearranging the expression, we get

Applying (10.11) now confirms our claim:

Given our data $x_1,\ldots,x_N$ and the corresponding ECDF $\hat F_N$, we can present the confidence set $C_N(\boldx)$ visually by highlighting the area between the lower bound $\hat F_N(s) - k_{1-\alpha}/\sqrt{N}$ and the upper bound $\hat F_N(s) + k_{1-\alpha}/\sqrt{N}$ over $s$. This is done in figures 10.1 and 10.2. The data are drawn from a $t$-distribution $F$ with 10 degrees of freedom and $\alpha$ is set to $0.05$. The values of $N$ are 250 for figure 10.1 and 1,000 for figure 10.2.

10.2Hypothesis Tests¶

This section considers problems of inference where we

To take the next steps, we must formalize the process of setting up a hypothesis and quantifying the extent to which the data points in this or some other direction.

Hypothesis testing begins with the specification of a null hypothesis, which is a statement that the observed data are being generated by a certain model, or one of a certain class of models. One might imagine that the standard procedure in statistics is to show the validity of the null hypothesis but it is not. Rather, a hypothesis test is an attempt to reject the null.

To see why, let’s recall a famous example due to Karl Popper (1902–1994). Consider the claim that all swans are white. It is futile to attempt to prove correctness of this hypothesis: No quantity of white swan sightings can confirm the claim that all swans on planet earth are white. Yet a single black swan can show the claim to be false. In this sense, attempting to falsify a theory on the basis of observation is more constructive than attempting to confirm it.

Although we attempt to reject the null hypothesis, we do so only if strong evidence against it is observed. The rationale is this: Suppose that we have a collection of theories about how the economy works. The procedure would then be to step through the theories, at each stage taking correctness of the theory as the null hypothesis and attempting to reject. If the theory is rejected, then we can discard it. This is a useful process of elimination. However, we don’t want to mistakenly discard a good theory. Hence we only reject when there is strong evidence against the null hypothesis.

10.2.1Constructing Tests¶

We adopt the setting and notation used in Section 10.1.1. A null hypothesis is a specification of a set of models that we believe the data-generating process belongs to. This amounts to specifying a subset $\Theta_0$ of $\Theta$. The null hypothesis is often written as

If $\Theta_0$ is a singleton, then the null hypothesis is called a simple hypothesis. If not, then the null hypothesis is called a composite hypothesis. A test of the null hypothesis is a test of whether or not the observed data were generated by $P_{\theta}$ for some $\theta \in \Theta_0$.

Formally, a test of $H_0$ is a binary function $\phi$ mapping the observed data $\zdata$ into $\{0,1\}$. The decision rule is

(Failing to reject $H_0$ should not be confused with accepting $H_0$. More on this below.) Our aim will be to design $\phi$ such that it takes the value $1$ when the data present strong evidence against $H_0$.

The outcome of our test depends on the random sample $\zdata$, and being random, its realization can be misleading. There are two different ways in which the realization can mislead us. First, we can mistakenly reject the null hypothesis when it is in fact true. This is called type I error. Second, we can fail to reject the null hypothesis when it is false. This is called type II error.

The power function associated with the test $\phi$ is the function

Here and below, the symbol $\PP_{\theta}$ indicates that we are computing probabilities under the assumption that $\lL(\zdata) = \Pdatatheta$. Thus, $\beta(\theta)$ is the probability that the test rejects when the data are generated by the probabilistic model identified by $\theta$.

Ideally, we would like $\beta(\theta) = 0$ when $\theta \in \Theta_0$, and $\beta(\theta) = 1$ when $\theta \notin \Theta_0$. In practice, this is difficult to achieve.

As discussed above, we tend to be conservative in rejecting the null because we don’t want to discard good theories. For this reason it is traditional to keep the probability of type I error small. Then, if the null is true, our test is unlikely to reject it. Because of this, the standard procedure is to choose a small number $\alpha$ such as 0.05 or 0.01, and then adjust the test such that

If (10.14) holds, then the test is said to be of size $\alpha$. Similarly the size of any test with power function $\beta$ is

This is the maximal rejection probability when the null hypothesis is true.

In constructing tests, a common (but not universal) setup is to define a real-valued test statistic $T$ and a critical value $c$, and then set

The pair $(T, c)$ then defines the test, and the rule becomes:

Plots of the power function $\beta$ in (10.17) are presented in figure 10.4 for two different values of $c$. Here $N$ is fixed at 10. Since $\Theta_0 = (-\infty, 0]$, the size of the test in each case is $\sup_{\mu \leq 0} \beta(\mu)$. Since the power curves are increasing, this is just $\beta(0)$.

Figure 10.4 illustrates a typical trade-off between type I and type II error. If we increase $c$, we make rejection less likely for all values of $\mu$. This pushes down type I error, but it also increases the probability that we fail to reject a false null.

10.2.2Choosing Critical Values¶

Let’s think a bit more about the test in (10.16). Typically, the choice of $T$ is suggested by the problem. For example, if our hypothesis is a statement about the second moment of a random variable, then we might take $T$ to be the sample second moment. Once $T$ is fixed, we need to adjust $c$ such that $(T,c)$ attains the appropriate size. Thus the standard procedure is to

In performing the last step, we can balance our desire to minimize type II error while maintaining a size $\alpha$ test by choosing $c$ such that (10.14) holds with equality. In this case, the problem is to choose $c$ to solve

Figure 10.5 gives an illustration. In the figure, we’ve taken $\Theta_0$ to be the two element set $\{\theta_a, \theta_b\}$. The blue line gives an imaginary distribution of $T(\zdata)$, represented as a density, when $\zdata$ is generated by $\theta_a$. The black line gives the same for $\theta_b$. Assuming that a value of $\alpha$ is prescribed, our next step is to determine the $c$ such that (10.18) holds. Here we choose $c$ such that the largest of the two shaded areas is equal to $\alpha$.

10.2.3Asymptotic Tests¶

In general, constructing a test of a given size requires some knowledge of the distribution of the test statistic under the null hypothesis. In many cases, however, we know relatively little about the distribution of the test statistic. Fortunately, we still have options. One approach is to bootstrap. Another is to try to determine the asymptotic distribution of the test statistic. As we’ve learned from the central limit theory, asymptotics hold out the possibility that we can pin down distributions without actually assuming a parametric structure a priori.

To proceed, we need to switch to a notion of asymptotic size for tests, rather than finite sample size. Writing $\beta_N$ instead of $\beta$ to emphasize the fact that the power function typically depends on sample size, a test is called asymptotically of size $\alpha$ if

10.2.3.1A Test for the Empirical Distribution¶

As another example of how to construct tests which are asymptotically of size $\alpha$, let’s return to the standardized daily returns data presented in figure 10.3. Let $\Phi$ be the standard normal CDF as before, and let $H_0$ be that standardized returns are IID draws from $\Phi$. Let $\alpha$ be given, and let $k_{1-\alpha} = K^{-1}(1-\alpha)$ be the $1-\alpha$ quantile of the Kolmogorov distribution $K$, as defined in (10.12). Finally, let $\hat F_N$ be the ECDF of the data. If the null hypothesis is true, then, by (10.11), we have

For the test

let $\beta_N(\Phi)$ be the value of the power function when the null hypothesis is true. By (10.22), we have

Hence (10.19) is verified, and the test is asymptotically of size $\alpha$. Notice how the construction of the test inverts the confidence set in Section 10.1.3.

The Nikkei data set used here has 364 observations. The value of the test statistic $\sqrt{N}\sup_{s \in \RR} |\hat F_N(s) - \Phi(s)|$ is 1.548. If $\alpha = 0.05$, then, as shown in Section 10.1.3, the critical value $k_{1-\alpha}$ is 1.36. Hence the test statistic exceeds the critical value, and the null hypothesis is rejected.

While the departure from normality is not easy to see in figure 10.3, it can be seen more clearly in a comparison of the histogram and density, as in figure 10.6. The data are noticeably more peaked and there is evidence of heavy tails.

This application also illustrates how tests often depend on a range of assumptions beyond those that are the focus of a particular hypothesis. Here, for example, a full description of $H_0$ is that standardized returns are IID draws from the standard normal density. The rejection suggests this null is false, but it may be the IID assumption rather than the normality assumption that makes our null a poor fit to the data.^[2]

10.2.3.2$p$-Values¶

Typically, a test that rejects at size 0.05 will also reject at size 0.1, but may not reject at size 0.01. Lower $\alpha$ means less tolerance for type I error, and forces the critical value to become larger. Hence, for a fixed value of the test statistic, the result of the test may switch from reject to don’t reject. A natural question then is: What is the smallest value of $\alpha$ at which we can still reject a given test statistic? This value is called the $p$-value.

To state this more precisely, consider again the setting of Section 10.2. The null hypothesis is $H_0 : \theta \in \Theta_0$ and, for each $\alpha \in (0,1)$, a test $(T,c(\alpha))$ of size $\alpha$. We assume here that $c(\alpha)$ is determined via the relationship in (10.18). In this setting, the $p$-value of the test is defined as

Roughly speaking, this is the $\alpha$ at which the test switches from reject to don’t reject. Typically, $\alpha \mapsto c(\alpha)$ is continuous, and in this case the expression for $p(\zdata)$ reduces to

10.2.4Accepting the Null?¶

Sometimes failure to reject a null hypothesis comes to be seen as presenting evidence in favor of that hypothesis. This is a slippery slope. Here’s one illustration.

In the 1980s econometricians found a new set of toys called unit root tests. First it was discovered that a newly developed test due to Dickey & Fuller (1979) did not reject the null hypothesis of a unit root when applied to a variety of economic time series (Nelson & Plosser (1982)). It was argued that this finding called into question a range of earlier time series methods that assumed trend stationarity, as well as a variety of business cycle studies that assumed stationary innovations around a trend. A raft of new unit root tests appeared. The idea that innovations to major economic time series contain a permanent component became something like a stylized fact.

Let’s think about this carefully. For the sake of concreteness, suppose that we are considering unemployment rate data, where the unit root hypothesis was associated with the concept of hysteresis (see, e.g., Mitchell (1993)). In this setting the unit root null can be expressed most simply as the hypothesis that $a = 1$ in the AR(1) process

Here $\{u_t\}$ is unemployment, $a$ and $b$ are parameters and $\{\epsilon_t\}$ is a zero-mean innovation (see eq. (5) of Mitchell (1993)). Letting $T$ be the length of the sample, the standard one-sided Dickey–Fuller test of the unit root null is to

Intuitively, when $a < 1$, the test statistic will evaluate to a negative number in large samples. While we omit the derivation, Dickey & Fuller (1979) showed that the test statistic converges in distribution under the null and tabulated critical values for different test sizes.

A number of studies have been unable to reject a unit root null for unemployment rate data (see Mitchell (1993) for an earlier study or Cheng et al. (2012) for a study using a newer set of tests). How should we interpret failure to reject the null in this setting? One possibility is that (10.25) is a good model for the data with $a=1$. This is not the only possibility, however. Another possibility is that the data are being generated by some other process against which this test has little power.

The second possibility seems more plausible than the first. Indeed, if $a=1$ in (10.25), then $\PP\{u_t \in B\} \to 0$ as $t \to \infty$ for every bounded $\bB$-measurable $B \subset \RR$. But employment rates don’t diverge. A more likely story is that while unemployment rates are highly persistent for some range of values, there are equilibrium forces that become increasingly stronger outside this band. For example, the further the unemployment rate falls, the more tightness in the labor market boosts wages, encouraging employers to substitute toward other factors of production and putting upward pressure on the unemployment rate.

We can easily reconcile these equilibrium forces with the idea that the unemployment rate is highly persistent at commonly observed values by adopting a nonlinear model. For example, consider the following simple variation on a random walk:

where $h$ is the generalized logistic function depicted in figure 10.7, along with the 45 degree line. For unemployment rates in a band between about 5 and 15, the function has a slope close to one, and the process exhibits strong persistence, just like a random walk. For values above and below this number the shape of $h$ causes drift back to the band.

Listing 7 gives code for a simulation when this model generates the data. The shocks $\{\epsilon_t\}$ are independent Gaussian. In each loop a time series of length 100 is drawn and the Dickey–Fuller test statistic is calculated. The size of the test is set to 0.05, in which case the critical value for the test is $-13.7$ (see, e.g., Hayashi (2000), p. 576). If you run the code you will see that for this random seed, the rejection frequency over 5,000 repetitions is around 0.05. This is the same rejection frequency as when the null is true. For this data size, the great majority of tests would fail to reject the unit root model, even though the data-generating process is (10.26).

using LinearAlgebra, Random, Statistics
N, alpha = 100, 0.05  # Length of time series, size of test
critical = -13.7      # Critical value for DF test of size 0.05
Random.seed!(123)     # Set random seed

function h(x, a=0.0, k=22, b=.16, q=0.6, m=10)
    return a + (k - a) / ((1 + q * exp(-b * (x - m)))^(1/q))
end

function dickey_fuller_stat(y)
    y1 = y[1:end-1]
    y2 = y[2:end]
    z2 = y2 .- mean(y2)
    z1 = y1 .- mean(y1)
    rho = dot(z1, z2) / dot(z1, z1) 
    return length(y1) * (rho - 1)
end

function sim_dgp(N) 
    y = Vector{Float64}(undef, N)
    y[1] = 8  # Initial value of unemployment rate
    for i in 1:N-1
        y[i+1] = h(y[i]) + 0.5 * randn()
    end
    return y
end

M = 5000
rejections = 0
for m in 1:M
    y = sim_dgp(N)
    global rejections += (dickey_fuller_stat(y) < critical)
end
println("Fraction under critical value: ", rejections / M)

The interesting thing about this experiment is that the nonlinear data-generating process has very different properties from the null. Not only is it stationary, it is uniformly ergodic, as can be seen by checking the conditions of theorem 16.0.2 of Meyn & Tweedie (2009). This is the strongest possible form of stationarity after the IID property. It means that the effect of initial conditions dies out at a geometric rate that is bounded away from one uniformly over the state space. There is no "permanent component."

10.2.5Statistical Tests in Economics¶

While the mechanics of hypothesis testing are relatively straightforward, the scientific paradigm based around hypothesis testing is the subject of continual debate. Macroeconomics is an interesting case in point. In the 1970s the rational expectations revolution ushered in a new class of macroeconomic models. One of the first things the proponents of rational expectations did was to test these models against data. Thomas Sargent’s account runs as follows (Evans & Honkapohja (2005)):

My recollection is that Bob Lucas and Ed Prescott were initially very enthusiastic about rational expectations econometrics. After all, it simply involved imposing on ourselves the same high standards we had criticized the Keynesians for failing to live up to. But after about five years of doing likelihood ratio tests on rational expectations models, I recall Bob Lucas and Ed Prescott telling me that those tests were rejecting too many good models.

In the following years, many macroeconomists moved away from formal hypothesis testing. A standard response to classical testing was: “I already know that my model is wrong. Rejection of my model using standard inference tells me nothing new.” For example, Kydland & Prescott (1991) define econometrics as a process without tests:

The main steps in econometric analysis are as follows: defining the question; setting up the model; calibrating the model; and reporting the findings.

How does this line of thinking compare with the standard statistical paradigm? The standard paradigm certainly recognizes that all models are wrong. All sensible models are wrong along some dimensions because they are, by definition, simplified representations of reality. Rejection of models is intended to facilitate survival of the fittest: Those models that explain observed phenomena effectively, and, at the same time, are most difficult to reject, will survive.^[3]

At the same time, it’s reasonable to feel sympathy for the frustration with classical hypothesis testing expressed by some economists. While the hypothesis that water boils at 100 degrees C will never be refuted (at least at sea level), in economics even the best models capture only some phenomena of interest. Inevitably they fail to match with reality when viewed in other ways. With enough data almost all sensible economic models can be rejected. But should we discard them? Perhaps some of these models are still useful in guiding our thinking.

How to resolve these different points of view? Some researchers argue that calibration and matching of moments are sufficient as a means of validating their models. But this form of validation imposes little discipline on model builders. Clever economists can find multiple ways to replicate most interesting phenomena observed in the data, each with their own distinct policy implications. More important, the objective of matching moments is largely an in-sample criterion. The dangers of such an approach were illustrated in figure 8.4.

One middle ground is testing of partially specified models, where tests are only concerned with a specific restriction implied by theory. A second path forward is more emphasis on out-of-sample performance in economic studies, which provides a much stricter test of true understanding. A third is offered by the Bayesian approach to model selection, where the emphasis is on comparing and improving models rather than rejection. A fourth is improvements in economic theory, to better match our models to the data.

10.3Further Reading¶

For a technical discussion of the results surrounding the convergence to the Kolmogorov distribution in (10.12), see section 19.4 of Vaart (2000).

For a discussions of statistical testing in econometrics see, for example, McCloskey & Ziliak (1996) and Freedman (2009). Discussions of calibration in relation to estimation and hypothesis testing can be found in Kydland & Prescott (1991), Kim & Pagan (1999), and many other places. Hansen (2014) presents the main arguments in favor of tests of restrictions in partially specified models. Silver (2012) provides an interesting read on estimation and uncertainty with applications in forecasting.

An excellent summary of unit root tests is given in Hayashi (2000). For a critical overview of unit root testing, see Cochrane (1991).

10.4Exercises¶

10.4.1Solutions to Selected Exercises¶