4 Foundations of Probability

Probability theory is one of the foundation stones of statistics and econometrics. We’ll start at the basics but move relatively quickly.

4.1Probabilistic Models¶

Probability is a technically demanding field. The main reason is that we want to assign probabilities to all manner of events. The set of possible events can be very large, and we need ways to manage this complexity. In this and the next few sections, we’ll review the basic machinery involved and why this machinery is necessary. At the same time, we’ll skip several details that would take us too far away from econometric theory. This will cost little in terms of understanding.

One concept worth mentioning before we start is countable and uncountable sets. In essence, a nonempty set $S$ is called countable if it is either finite or can be represented as a sequence, and uncountable if is not countable. In probability theory this distinction is important. A quick review can be found in Section 15.3.

4.1.1Sample Spaces and Events¶

In setting up a probabilistic model, we start with the notion of a sample space, which we think of as being a "list" of all possible outcomes in a given random experiment. In general, the sample space can be any nonempty set, and is usually denoted $\Omega$. A typical element of $\Omega$ is denoted $\omega$. The general idea is that a realization of uncertainty will lead to the selection of a particular $\omega \in \Omega$.

Once we have a sample space in place we can start to think about events. In the language of probability theory, an event is just a subset of $\Omega$ (with some caveats that will be addressed below). An event $A$ occurs whenever the individual $\omega \in \Omega$ selected in the random experiment happens to lie in $A$. Figure 4.1 illustrates.

4.1.1.1Probabilities and Events¶

The next thing we need to do is to assign probabilities to outcomes. One idea is to start by assigning appropriate probabilities to every $\omega$ in $\Omega$. We could then derive the probability of events by looking at the probabilities attached to points contained in those events. It turns out, however, that this is not the best approach, at least if we want to consider uncountable $\Omega$. Rather, the standard approach is to directly assign probabilities to subsets of $\Omega$ instead.

To see why the first approach is problematic, consider the dart throwing model in example 4.1.2, where $\Omega$ is the unit disk in $\RR^2$. Imagine that hitting any point in $\Omega$ is "equally likely." We can model this by assuming that the probability the dart lands in some region $A$ of the dartboard (some $A \subset \Omega$) is simply proportional to the area of $A$.

Consider now the probability of hitting some individual point $\omega \in \Omega$. Intuitively, this should be less than the probability of hitting any region $A$ containing $\omega$. By choosing $A$ to be sufficiently small in area, we can show that the probability of hitting $\omega$ is less than $\epsilon$ for any given $\epsilon > 0$. We are forced to conclude that the probability of hitting the individual point $\omega$ is zero.^[1]

This example shows that in many sensible models the probability assigned to individual points in the sample space must be zero. If you reflect on this for even a moment you will agree that there is no way to build up a useful probabilistic model with such information alone. Hence we must assign probabilities directly to the events themselves.

4.1.1.2Choosing the Set of Events¶

Should we assign probabilities to every possible event—that is, to every possible subset of $\Omega$? In the dart model above, we could say that the probability the dart lands in any region $A \subset \Omega$ is equal to the area of $A$ divided by the area of $\Omega$ itself, which is $\pi$. In symbols,

This seems sensible enough. For example, if $A = \Omega$, then $\PP(A) = \pi/\pi = 1$, as it should be. If $A$ is the top half of the dartboard then $\PP(A) = 0.5$, and so on.

Unfortunately, if we think carefully about this approach, some cracks start to appear. What exactly do we mean by "area of $A$"? If $A$ is a circle, then its area is $\pi$ times the square of its radius. If $A$ is a rectangle, then its area is the product of its two sides. But how about the area of more general subsets of $\Omega$?

It turns out that assigning area to all subsets of $\Omega$ in a consistent way is problematic. The reason is that $\Omega$ contains an awful lot of subsets, and some of them can be manipulated to exhibit very strange phenomena. (If you are curious, look up the Banach–Tarski paradox for a hint of the problems that can occur when we try to assign notions like area or volume to arbitrary sets.) We can dream up all sorts of strange events that confound our ability to construct a sensible probabilistic model.

Because of this, when $\Omega$ is uncountable, we usually resist taking the set of events to be all subsets of $\Omega$. Instead, we take them to be some family of "well-behaved" subsets of $\Omega$, denoted below by $\fF$, and only assign probabilities to elements of $\fF$.

There is of course a trade-off here. If we restrict $\fF$ to be a small fraction of the subsets of $\Omega$, then assigning probabilities becomes easy and we can eliminate strange behavior. But if we make $\fF$ too small, then we might not be able to assign probabilities to events that we care about. For example, in the dartboard model, if we restrict $\fF$ to be the set of rectangles, then we can easily measure area and hence assign probabilities via (4.1). But this excludes interesting events such as the dart landing in the top half of the board.

How can we ensure that $\fF$ is large enough? To think about this in a systematic way, suppose that $A \in \fF$, and that $\PP(A)$ is well-defined and represents the “probability of event $A$.” Now, given that we can assign a probability to the event $A$, it would be unfortunate if we couldn’t assign a probability to the event “not $A$”, which corresponds to $A^c$. So normally we require that if $A \in \fF$, then $A^c \in \fF$. When this is true, we say that $\fF$ is “closed under the taking of complements.”

Further, suppose that $A$ and $B$ are both in $\fF$, and we can assign probabilities to these events. In this case, it would be natural to think about the probability of the event “$A$ or $B$”, which corresponds to $A \cup B$. So we also require that if $A$ and $B$ are in $\fF$, then $A \cup B$ is likewise in $\fF$. We say that $\fF$ is “closed under the taking of unions.” In fact for technical reasons it turns out that we prefer $\fF$ to be closed under the taking of countable unions, which means that if $A_1, A_2, \ldots$ is a sequence of sets in $\fF$, then its union is likewise in $\fF$.

A set $\fF$ of subsets of a given set $\Omega$ that satisfies all these properties is called a $\sigma$-algebra. Formally, $\fF$ is a $\sigma$-algebra on $\Omega$ if

Note that (i) and (iii) together imply that $\emptyset \in \fF$. In this context, the empty set $\emptyset$ is called the impossible event. In contrast, $\Omega$ is called the certain event because it always occurs (regardless of which outcome $\omega$ is selected, $\omega \in \Omega$ is true by definition).

The $\sigma$-algebra of events $\fF$ we choose to pair with a given $\Omega$ can vary from problem to problem. There is however one particular $\sigma$-algebra that you should know at least something about, and that is the standard $\sigma$-algebra we use to pair with ordinary vector space $\RR^N$. This $\sigma$-algebra is called the Borel sets.

To understand the idea, consider the following: The property of $\fF$ being a $\sigma$-algebra seems to imply that $\fF$ is a relatively rich class of sets. After all, we can take complements and unions (and intersections too—see ex. 4.4.1) without ever leaving $\fF$. However, there are trivial examples of $\sigma$-algebras, such as $\fF := \{ \emptyset, \Omega\}$. So if we want to pin down a large and useful class of events, the best way to proceed is to assume not only that $\fF$ is a $\sigma$-algebra, but also that $\fF$ contains some fundamental sets that we are likely to want to attach probabilities to, such as intervals and rectangles.

In this connection, the Borel subsets of $\RR^N$ are defined as the smallest $\sigma$-algebra of subset of $\RR^N$ that contains all rectangles in $\RR^N$. (See Section 15.1 if you wish to learn more about rectangles.) This definition turns out to be highly successful: Restricting attention to Borel sets does a great job of excluding the kind of strange behavior found in the Banach-Tarski paradox, and, moreover, every subset of $\RR^N$ we need to deal with in day to day analysis is a Borel set. These include planes and hyperplanes, circles, spheres, polygons, finite sets, and sequences of points. In what follows we use $\bB(\RR^N)$ to represent the Borel sets of $\RR^N$.

4.1.2Probabilities¶

Now let’s take a $\sigma$-algebra of events as given and consider how to assign probabilities to elements of $\fF$. For given event $B$, the symbol $\PP(B)$ will be interpreted as representing "the probability that event $B$ occurs." The way you should think about it is this:

$\PP(B)$ represents the probability that when uncertainty is resolved and some $\omega \in \Omega$ is selected by “nature,” the statement $\omega \in B$ is true.

In order to make sure our model of probability is well behaved, we need to put certain restrictions on $\PP$. For example, it wouldn’t do to have a $B$ with $\PP(B) = -93$, since negative probabilities don’t make sense. The standard restrictions are imposed in the next definition:

Let $\Omega$ be a nonempty set and let $\fF$ be a $\sigma$-algebra of subsets of $\Omega$. A probability $\PP$ on $(\Omega, \fF)$ is a function from $\fF$ to $[0,1]$ that satisfies

In this context $\PP$ is also called a probability measure. Together, the triple $(\Omega, \fF, \PP)$ is called a probability space.

In the definition, axiom (i) is straightforward. We require $\PP(\Omega) = 1$ because, by construction, every possible $\omega$ lies in the set $\Omega$. Axiom (ii) is called countable additivity. The concept of disjointness in the statement of the axiom is pairwise: Any distinct pair $A_i, A_j$ share no points in common. By taking some of the sets as empty, countable additivity can be shown to imply finite additivity, which requires that

whenever $A_1, \ldots, A_k$ are disjoint.

The general notion of additivity is a natural requirement for probabilistic models. It says that to find the probability of a given event, we can determine all the different (think "disjoint") ways that the event could occur, and then sum their probabilities. Probabilists boost this up to countable additivity because it helps certain limiting arguments work smoothly.

Figure 4.2 gives an illustration of additivity. Suppose that the $N$ dots in the figure are the only possible outcomes. Suppose that each of these occurs with probability $1/N$, and hence the probability of an event is the number of dots it contains divided by $N$. Hence $\PP(A)=2/N$, $\PP(B)=4/N$ and $\PP(C)=3/N$. In addition there are $9$ dots in $A\cup B \cup C$, so

Notice how disjointness is important in making additivity work here: If the sets were not disjoint we would run the risk of double counting on the right hand side of this sum, breaking the equality.

The function $\lambda$ from example 4.1.7 that maps Borel sets to their "area" is formally known as the Lebesgue measure. Section 15.3.1 provides a brief introduction to this concept.

4.1.2.1Implications of the Axioms¶

Now let’s go back to the general case, where $(\Omega, \fF, \PP)$ is an arbitrary probability space. From the axioms, we can derive a surprising number of properties. Let’s list the key ones, starting with the next fact.

(Here $A^c = \Omega \setminus A$. See Section 15.1 for a review of set-theoretic operations.)

To prove fact 4.1.1, note that when $A \subset B$, we have $B = (B \setminus A) \cup A$. Since $B \setminus A$ and $A$ are disjoint, additivity of $\PP$ now gives

This equality implies (i)–(iv) of fact 4.1.1. Rearranging gives (i), while nonnegativity of $\PP$ gives (ii). Specializing to $B = \Omega$ gives (iii), and setting $B = A$ gives (iv).

The property $A \subset B \implies \PP(A) \leq \PP(B)$ is called monotonicity, and is both fundamental and intuitive: If $A \subset B$, then we know that $B$ occurs whenever $A$ occurs, since $\omega$ lands in $B$ whenever it lands in $A$. Hence the probability of $B$ should be no smaller. Many results in probability turn on this idea.

Exercise 4.4.2 asks you to prove this. It follows from fact 4.1.2 that for any $A, B \in \fF$, we have $\PP(A \cup B) \leq \PP(A) + \PP(B)$. This inequality is called subadditivity. Thus probabilities are subadditive over arbitrary pairs of events and additive over disjoint pairs.

4.1.2.2Dependence and Independence¶

If $A$ and $B$ are events, then the conditional probability of $A$ given $B$ is

It represents the probability that $A$ will occur, given the information that $B$ has occurred. For the definition to make sense, we need $\PP(B) > 0$. Events $A$ and $B$ are called independent if $\PP(A \cap B) = \PP(A) \PP(B)$. More generally, events $A_1, \ldots, A_N \in \fF$ are called independent if

Note that if $A$ and $B$ are independent, then the conditional probability of $A$ given $B$ is just the probability of $A$.

To state the next result, we need the concept of a partition of $\Omega$, by which we mean a countable set of events $\{B_m\}_{m \geq 1}$ with the property that distinct pairs are disjoint and $\cup_{m \geq 1} B_m = \Omega$.

The following fact is known as the law of total probability.

The proof is as follows: It is a basic result from set theory that $A \cap (\cup_m B_m) = \cup_m (A \cap B_m)$. From this equality, the definition of a partition and countable additivity of $\PP$, we get

Applying (4.4) completes the proof.

Another useful result concerning conditional probability is Bayes’ law, which states that for any events $A$ and $B$ with positive probability,

To prove (4.6), observe that, from the definition of conditional probability,

Hence $\PP(A \cap B) = \PP(A \given B)\, \PP(B) = \PP(B \given A)\, \PP(A)$. Rearranging yields (4.6).

4.1.3Random Variables¶

In some probability courses, a random variable is defined as a “value that changes randomly,” or something to that effect. This is a fairly useless definition. A better definition of a random variable is a function $x$ from a sample space $\Omega$ into $\RR$. Think of it this way:

Thus a random variable is a map $x$ that converts outcomes in sample space—which can be any kind of object—into numerical outcomes. This is valuable because numerical outcomes are easy to order, add, subtract and so forth. In other words, random variables "report" the outcome of an experiment in a format amenable to further analysis.

The random variable $y$ in (4.7) is an example of a binary (or Bernoulli) random variable, which is a random variable taking values in the set $\{0, 1\}$. There is a generic way to create binary random variables that often comes in handy, using indicator functions. If $Q$ is a statement, such as “on the planet Uranus, there exists a tribe of three-headed monkeys,” then $\1\{Q\}$ is considered as equal to 1 when the statement $Q$ is true, and zero when the statement $Q$ is false. Another common variation on the notation is, for arbitrary $A \in \fF$,

So defined, $\1_A$ is a binary random variable. In fact any binary random variable $x$ can be expressed as $\1_A$ by setting $A := \setntn{\omega \in \Omega}{x(\omega) = 1}$.

Here equality means when evaluated at any $\omega \in \Omega$. See exercise 4.4.5.

4.1.3.1Notational Conventions¶

Given random variable $x$, you will often see notation such as

This is a shorthand for the event

Another notational convention you should be aware of is that, for random variables, equalities, inequalities and arithmetic operations should be interpreted pointwise. For example,

• $x \leq y \; \iff \; x(\omega) \leq y(\omega)$ for all $\omega \in \Omega$,

• $x = y \; \iff \; x(\omega) = y(\omega)$ for all $\omega \in \Omega$, and

• $z = \alpha x + \beta y \; \iff \; z(\omega) = \alpha x(\omega) + \beta y(\omega)$ for all $\omega \in \Omega$.

4.1.3.2Random Variables Are Measurable Functions¶

We’ve omitted a technical detail in our construction of random variables. It arises because we cannot always take the set of events $\fF$ to be the set of all subsets of $\Omega$ (since assigning probabilities to arbitrarily complex sets causes problems—see the discussion in Section 4.1.1).

To explain, let $(\Omega, \fF, \PP)$ be any probability space. Let $B$ be any subset of $\RR$ and consider evaluating the probability

Here $x$ is some function from $\Omega$ to $\RR$. Now observe that if we put no restrictions on $x$, we might end up in a mess. The reason is that we have no way of being sure that the set $\setntn{\omega \in \Omega}{x(\omega) \in B}$ is in fact an element of $\fF$. Since $\PP$ is only defined over elements of $\fF$ (i.e., for bona fide events), it is possible that $\PP\{x \in B\}$ is undefined.

To prevent such embarrassment, it’s standard to put some restrictions both on $x$ and $B$ that ensure $\{x \in B\} \in \fF$. For $B$, we naturally restrict attention to $\bB(\RR)$, the Borel subsets of $\RR$, since we know these sets to be both relatively well behaved and common enough to satisfy our purposes (see Section 4.1.1). For $x$, we simply require that $\{x \in B\} \in \fF$ whenever $B$ is a Borel set. Thus, our formal definition is that a random variable on $(\Omega, \fF)$ is a function $x \colon \Omega \to \RR$ satisfying

These kinds of functions are also sometimes called $\fF$-measurable functions. In other texts you will see (4.9) expressed using preimage notation. In the preimage notation, $x^{-1}(B)$ is all $\omega \in \Omega$ such that $x(\omega) \in B$. This is just the left-hand side of (4.9), so we can rewrite (4.9) as

Thus $x$ "pulls back" Borel sets to events.

4.1.3.3Measurable Transformations¶

So far so good. If $x$ is a random variable, then $\{x \in B\}$ is a well-defined event for every $B \in \bB(\RR)$. But suppose now that we want to discuss some transformation of $x$. For example, we might be interested in the properties of $y := e^x$. Is $y$ also a random variable? That is, is $\{y \in B\}$ always a well-defined event?

The answer to this question is affirmative provided that our transformation satisfies a regularity condition called Borel measurability. Formally, $f \colon \RR \to \RR$ is called Borel measurable, or $\bB$-measurable, if

Thus $f$ "pulls back" Borel sets to Borel sets. Although we omit the details, the class of $\bB$-measurable functions is vast (any continuous function, any increasing function, etc., etc.), and all transformations we work with below are $\bB$-measurable.

So why is $\bB$-measurability of $f$ the key to ensuring that $y = f(x)$ is a random variable? Suppose that $f$ is $\bB$-measurable and that $x$ is a random variable. We claim that $\{y \in B\} \in \fF$ for all $B \in \bB(\RR)$. This holds because

for all $B \in \bB(\RR)$. By (4.10) we know that $f^{-1}(B)$ is a Borel set. Since $x$ is a random variable, it follows that $\{x \in f^{-1}(B)\}$ is an element of $\fF$.

4.1.4Expectations¶

Our next task is to define expectations of random variables and state their basic properties. Elementary probability courses define expectations by summing or integrating functions over densities. The view of a random variable as a function on a sample space plays no part in these definitions. However, when we move to more sophisticated arguments, particularly those involving a large number of random variables, we need a better definition. This section outlines the modern treatment of expectations found in probability and statistics. As we’ll see later, the definition in this section is a generalization of the elementary definition. The two agree whenever the latter is well defined.

For finite random variables (i.e., random variables with finite range) the expectation is just the sum of all possible values of the variable, weighted by the corresponding probabilities. More formally, given probability space $(\Omega, \fF, \PP)$ and random variable $x$ taking only finitely many distinct values $s_1, \ldots, s_J$, the expectation of $x$ is defined as

4.1.4.1Expectations for Arbitrary Random Variables¶

Definition (4.12) only applies for random variables with finite range. We still need to define expectation for general $x$. This is done by bootstrapping up from the finite case. The procedure is based on approximating arbitrary random variables with finite random variables. One finite approximation $x_n$ to an arbitrary random variable $x$ is shown in figure 4.3. The approximation can be improved without limit if we allow $x_n$ to take a larger and larger number of distinct values. This process yields a sequence of finite random variables $x_n$ converging to $x$ (e.g., see Çinlar (2011), lemma 2.16). The expectation of $x$ is then defined as

The value $\EE x$ is also referred to as the Lebesgue integral of $x$ with respect to $\PP$, with the alternative notation $\EE x = \int x(\omega) \PP(\diff \omega)$.

Can we be sure that the limit in (4.14) makes sense? Suppose first that $x$ is nonnegative (i.e., $x(\omega) \geq 0$ for all $\omega \in \Omega$). It can be shown that the sequence $\{x_n\}$ approximating $x$ can then be chosen so that $\EE x_n$ is monotone increasing and hence convergent, and that the limiting value does not depend on the particular approximating sequence $\{x_n\}$. See, for example, Dudley (2002), proposition 4.1.5. Hence $\EE x$ is well defined, although we cannot rule out divergence of the sequence, which gives $\EE x = \infty$.

We’ve assumed that $x$ is nonnegative. If we drop this assumption, we can still treat $x$ with similar methods by writing it as the difference between two nonnegative random variables. Typically this is done by writing $x = x^+ - x^{-}$, where $x^+ := \max\{x, 0\}$ and $x^- := -\min\{x, 0\}$. Both $x^+$ and $x^-$ are nonnegative random variables. The expectation can then be defined as

This is fine except that (4.15) might have the form $\infty - \infty$, which isn’t well-defined. For this reason we often restrict attention to integrable random variables, which are all random variables $x$ such that $\EE |x| < \infty$. Since $x^+ \leq |x|$ and $x^- \leq |x|$, this is enough to ensure that (4.15) is well-defined.

The definition of $\EE$ given in this section doesn’t really tell us how to calculate $\EE x$ in specific instances, but we’ll see how to do so in Section 4.2.5.

4.1.4.2Properties of Expectation¶

Let’s summarize the main results concerning expectations and list some properties of this operation.

Since we only know how to take expectations of random variables, the expression $\EE \alpha$ in part (i) should be understood as the expectation of a constant random variable everywhere equal to $\alpha$. (We could write this object more precisely as $\alpha \1_\Omega$.) That the expectation equals $\alpha$ was already established in example 4.1.15. It also follows from (ii) and (iv), by taking $x = \1_{\Omega}$ and $\beta = 0$.

Part (ii) follows from example 4.1.16. It is restated here to emphasize this fundamental link between probabilities and expectations. Part (iii) is called monotonicity of expectations. The statement $x \leq y$ means that $x(\omega) \leq y(\omega)$ for all $\omega \in \Omega$. Exercise 4.4.23 asks you to prove this monotonicity in a special case. For a general proof see lemma 4.1.9 of Dudley (2002). Part (iv) is called linearity of expectations. Here $\alpha x + \beta y$ should be understood as the random variable $(\alpha x + \beta y)(\omega) := \alpha x(\omega) + \beta y(\omega)$. Exercise 4.4.13 and Exercise 4.4.14 ask you to check linearity in some special cases. For the general case see theorem 4.1.10 of Dudley (2002).

As an exercise, let’s use these results to prove that if $x$ is a finite random variable with range $\{s_j\}_{j=1}^J$ and $h$ is any $\bB$-measurable function, then $h(x)$ has expectation

First observe that $\sum_{j=1}^J \1\{x = s_j\} = 1$, and hence we can write $h(x)$ as

Using linearity of expectations gives $\EE h(x) = \sum_{j=1}^J h(s_j) \EE \1\{x = s_j\}$. Applying part (ii) of fact 4.1.5 leads to (4.17).

The next result is another useful link between probabilities and expectations, typically called Chebyshev’s inequality.

A common variation of Chebyshev’s inequality is the bound

Exercise 4.4.29 asks you to check both (4.18) and (4.19).

4.1.5Moments and Co-Moments¶

Let $x$ be a random variable and let $k \in \NN$. If $x^k$ is integrable, then

• $\EE [x^k]$ is called the $k$th moment of $x$.

• $\EE [ (x - \EE x)^k]$ is called the $k$th central moment of $x$.

If $\EE [ |x|^k ] = \infty$, then the $k$th moment is said not to exist. For some random variables even the first moment does not exist. For others every moment exists.

Exercise 4.4.24 asks you to confirm this. The next fact is called the Cauchy–Schwarz inequality for random variables. Its proof will be given later, once we’ve covered orthogonal projection (ex. 5.4.15).

The second central moment of $x$ is called the variance of $x$:

This gives a measure of the dispersion of $x$. The standard deviation of $x$ is

The covariance of random variables $x$ and $y$ is defined as

Part (i) follows from (ii)–(iii), the Cauchy–Schwarz inequality and fact 4.1.7. Parts (ii)–(iii) follow from linearity of $\EE$ and some simple manipulations.

Some simple implications of this fact are that

The correlation of $x$ and $y$ is defined as

If $\corr[x,y] = 0$, we say that $x$ and $y$ are uncorrelated. Positive correlation means that $\corr[x,y]$ is positive and negative correlation means that $\corr[x,y]$ is negative. The first part of the next fact follows immediately from fact 4.1.8; the second is just algebra.

4.1.5.1Best Linear Predictors¶

As an application of covariance and a prelude to our discussion of prediction, let’s consider the problem of predicting the value of a random variable $y$ given knowledge of the value of a second random variable $x$ (and also knowledge of the underlying probability distributions, which makes this a problem in probability rather than statistics). Thus we seek a function $f$ such that $f(x)$ is close to $y$ on average. To measure the latter, we will use mean squared error, which amounts in this case to

As we’ll see in Section 5.2.5, the minimizer of the mean squared deviation over all functions of $x$ is obtained by choosing $f(x) = \EE[y \given x]$, where the right-hand side is the conditional expectation of $y$ given $x$. Here we’ll consider the simpler problem of finding a good predictor of $y$ within the class of “linear” functions

(We defer to common usage and call such functions linear in this context, despite the fact that they are, in general, affine; see example 2.1.23.) Thus we consider the problem

If $\alpha$ and $\beta$ solve (4.21), then the function

is called the best linear predictor of $y$ given $x$.

To solve (4.21) we expand the square on the right-hand side and use linearity of $\EE$ to write the objective function as

Computing the derivatives and solving the first-order conditions yields (ex. 4.4.27) the minimizers

If you’ve studied simple linear least squares regression before, you might realize that $\alpha^*$ and $\beta^*$ are the “population” counterparts of coefficient estimates in an empirical setting. In example 4.1.18, this allows us to take the coefficient $\beta^*$ as the definition of an asset’s beta, independent of any particular data set. If we wish to evaluate $\beta^*$, however, we need to use data, since the underlying probabilities, and hence the expectation operator $\EE$, are not observable. It is at this stage that the population regression is replaced by simple linear least squares using data. More on this below.

4.2Distributions¶

Any random variable $x$ on a given probability space $(\Omega, \fF, \PP)$ assigns to each Borel set $B \subset \RR$ a number $\PP\{x \in B\}$ indicating the probability that $x$ lies in $B$. While this is an important construction, as a means of specifying probabilities for outcomes in $\RR$, it is somewhat indirect. When building statistical models, it’s often more convenient to assign probabilities to subsets of $\RR$ directly. This is typically done by specifying a distribution over $\RR$. In this section we look at how one might go about specifying such distributions.

4.2.1Defining Distributions on $\RR$¶

How can we specify probabilities over subsets of $\RR$? In Section 4.1.1.2–Section 4.1.2 we learned how to construct a set of events $\fF$ on an arbitrary sample space $\Omega$ and define a probability measure $\PP$ on $\fF$. If we want to describe probabilities over $\RR$, then the natural thing to do is to repeat the process with $\Omega$ specialized to $\RR$. For the set of events over $\RR$ we take the Borel sets $\bB(\RR)$, as discussed in Section 4.1.1.2. A probability measure defined over $\bB(\RR)$ is called a law or distribution.

Distributions will be represented by symbols such as $P$ and $Q$. Referring back to the definition, a distribution $P$ is a map from $\bB(\RR)$ to $[0, 1]$ such that

Being a probability measure, $P$ satisfies all the properties in fact 4.1.1. If there exists a Borel set $S$ with $P(S) = 1$, then we say that $P$ is supported on $S$.

Probability measures are both general and comprehensive. Each one pins down probabilistic information for every event in $\RR$ that we might care about in econometrics or statistics. At the same time, this kind of generality can also be a hindrance. For example, it’s not a trivial task to assign a specific value $P(B)$ to every single Borel set.

Fortunately, we can characterize distributions more succinctly. To begin, we recall the notion of a cumulative distributed function, or CDF, which is any function $F \colon \RR \to [0, 1]$ satisfying

CDFs and distributions on $\RR$ can be put in one-to-one correspondence as follows: First, it’s true by definition that any two distributions $Q$ and $P$ satisfying $Q(B) = P(B)$ for all $B \in \bB(\RR)$ are the same distribution, but it’s also true that if $Q(B) = P(B)$ for all sets $B$ of the form $(-\infty, s]$, then $Q$ and $P$ are again the same distribution. In other words, a distribution $P$ is entirely characterized by the values of the function

Moreover this function turns out to have properties (i)–(iii) above. The next fact summarizes the relationship between $F$ and $P$.

For a full proof, see Williams (1991), lemma 1.6, or Dudley (2002), theorem 9.1.1. Here let’s restrict ourselves to showing that the function $F$ in (4.24) satisfies part (i) of the definition of a CDF. To see why (i) holds for this $F$, observe that $s \leq s'$ implies $(-\infty, s] \subset (-\infty, s']$. Part (ii) of fact 4.1.1 then gives $P((-\infty, s]) \leq P((-\infty, s'])$. Hence $F(s) \leq F(s')$ as claimed.

CDFs are relatively easy to specify and work with. Here are some examples.

4.2.2Densities and PMFs¶

There are two important sub-cases for distributions: the discrete case and the absolutely continuous case. As we’ll see, in these two sub-cases distributions have nice visual representations and expectations are easy to compute. Almost all "named" distributions fall into one of these two categories.

4.2.2.1The Discrete Case¶

A distribution $P$ is called discrete if it is supported on a countable set; that is, if there exists a countable set $\{s_j\}_{j \geq 1}$ with $P(\{s_j\}_{j \geq 1}) = 1$. For such a $P$ let

Property (ii) of $P$ and $P(\{s_j\}_{j \geq 1}) = 1$ imply that $\sum_{j \geq 1} p_j = 1$ (ex. 4.4.15). Hence $\{p_j\}_{j \geq 1}$ is an example of a probability mass function, or PMF, which is any nonnegative sequence (finite or infinite) that sums to unity.

Together, $\{s_j\}_{j \geq 1}$ and $\{p_j\}_{j \geq 1}$ characterize the underlying distribution. For example, we can express the CDF corresponding to $P$ via (4.24) as

This is intuitive: $F(s)$ represents the probability of an outcome in $(-\infty, s]$ under $P$. Since $P$ puts all its probability mass on $\{s_j\}_{j\geq 1}$, this should equal the sum of the probabilities of hitting $s_j$ over all $s_j \leq s$.

Figure 4.7 gives a visualization when $\{s_j\}_{j \geq 1} =\{s_1, s_2\}$. $F$ is a step function, with a jump up of size $p_j$ at point $s_j$. The CDF is right-continuous but not continuous.

4.2.2.2The Absolutely Continuous Case¶

Another case of interest is what elementary books on probability call the "continuous" case, where distributions can be described by densities. A density is a nonnegative function $p$ on $\RR$ that integrates to 1. We say that distribution $P$ is represented by density $p$ (or has density $p$) if $p$ is a density and

Here the integral over $B$ can be understood as

Recall from our discussion in Section 4.2.1 that distributions are characterized by their values on simple sets, like half lines of the form $(-\infty, s]$. Thus, if $p$ is a density and

then (4.26) also holds.

Not every distribution has a density representation. An exact necessary and sufficient condition for existence is absolute continuity. A distribution $P$ on the Borel subsets of $\RR$ is called absolutely continuous if $P(B) = 0$ whenever $B$ has Lebesgue measure zero. A definition of the latter property is given in Section 15.3.1, although for us it is enough to know that any countable subset of $\RR$ has Lebesgue measure zero. This leads to the following fact:

Compare with the discrete distributions, which put all mass on a countable set. One important implication of fact 4.2.2 is that individual points receive no probability mass under absolute continuity, and the corresponding CDF contains no jumps. In fact, when (4.27) holds, the fundamental theorem of calculus tells us that $F(s) = P((-\infty, s])$ is differentiable at all continuity points of $p$, and, in particular,

4.2.3Integrating with Distributions¶

We all understand what it means to take the ordinary integral $\int_a^b h(s)\diff s$ of a well-behaved function $h$ on some interval $[a, b]$. Sometimes we want to weight this integral, assigning more mass to different regions of $[a, b]$. This leads to integrals such as $\int_a^b h(s) p(s) \diff s$. For example, $h$ could be a welfare function and $p$ might record the density of agents of a certain type. In a probabilistic context, $p$ might be a density indicating probabilities of outcomes, $h$ might be a payoff function and $\int_a^b h(s) p(s) \diff s$ would be expected payoff.

We now wish to give meaning to expressions of the form $\int h(s) P(\diff s)$ that can be viewed as generalizations of these integrals. The nature of the generalization is that the probability $P$ might not have a density, and yet we still want to compute a weighted integral using $P$. As motivation we’ll continue with the previous example, where $h(s)$ is the payoff from outcome $s \in \RR$, and the likelihood of outcomes is determined by probability measure $P$. To provide some basic regularity, we assume that $h$ is a $\bB$-measurable function on $\RR$.

Actually our task is straightforward given the foundations already laid. In particular, we already have a general notion of integration suitable for understanding the expression $\int h(s) P(\diff s)$. In Section 4.1.4 we saw how to take an arbitrary probability space $(\Omega, \fF, \PP)$ and construct an expectation operator $\EE$ that assigns to any integrable random variable a notion of expectation. Expectations can be thought of as integrals, as suggested by the alternative notation $\int x(\omega) \PP(\diff \omega)$ for $\EE x$ used in (4.16).

So now take our distribution $P$ on $\RR$ and consider $(\RR, \bB(\RR), P)$ as a probability space. This new probability space has its own expectation operator $\EEP$. (See fact 4.1.5 if you wish to refresh your memory on this point.) Moreover we can use it to take the expectation of the payoff function $h$, which, being a $\bB$-measurable function on $\RR$, is by definition a random variable on $(\RR, \bB(\RR), P)$. This gives us a value $\EEP h$, which is exactly the integral we’re after:

While the preceding discussion gives a neat definition of $\int h(s) P(\diff s)$, it would be disappointing if this definition didn’t lead us back to familiar expressions like $\int h(s) p(s) \diff s$ when $P$ has a density. Fortunately, it does:

For a proof of fact 4.2.3, see §6.12 of Williams (1991).

4.2.4Distributions of Random Variables¶

In Section 4.1.3 we defined random variables. Over the last few sections we’ve defined and discussed distributions over the Borel subsets of $\RR$. Our next job is to connect them. We’ll see that the connection runs in both directions: Every random variable defines a distribution on $\RR$. Conversely, given any distribution, we can reverse engineer a random variable having that distribution.

To begin, let $x$ be a random variable on some probability space $(\Omega, \fF, \PP)$. By construction, the probability $\PP\{x \in B\}$ is well-defined for every $B \in \bB(\RR)$ (see (4.9)). The values obtained from this expression when $x$ is held fixed and $B$ is varied is called the distribution of $x$. In particular, the set function $P$ defined by

is both the distribution of $x$ and also a distribution in the sense of Section 4.1.3 (i.e., a probability measure over the Borel sets). It records the probabilities of all outcomes of interest for the variable $x$ when regarded as a single entity.^[3] In what follows we write $\lL(x)$ for the distribution (or "law") of $x$. The relationship in (4.30) is summarized by writing $\lL(x) = P$.

By (4.24), the CDF corresponding to the distribution $P$ of $x$ satisfies

Sometimes $F$ is called the distribution function or just distribution of $x$. Since $F$ characterizes $P$, we also write $\lL(x) = F$ to indicate that $F$ represents the distribution of $x$.