Skip to main content

Conditioning and Independence

Published

October 28, 2026

Stochastic processes describe how uncertainty unfolds over time. At each date some information has been revealed and some has not, and the quantities we care about, such as the probability of a default or the expected value of a future payoff, must be computed given what is known at that date. Conditioning is the operation that updates probabilities and expectations in light of new information. Independence describes the opposite situation, in which learning one thing tells us nothing about another.

These notes continue the notes on probability basics and use the same setting: a discrete probability space (\Omega, \operatorname{P}) with a finite or countably infinite set of outcomes, where every subset of \Omega is an event. We first define conditional probability and use it to describe independence. We then define the conditional expectation of a random variable given another, describe the information carried by a random variable, and finally define the conditional expectation given an information set. The last notion, together with its properties, is the tool we will use to define martingales.

For more complete treatments, see Blitzstein and Hwang (2019) and Grimmett and Stirzaker (2001).

Blitzstein, Joseph K., and Jessica Hwang. 2019. Introduction to Probability. 2nd ed. Chapman; Hall/CRC.
Grimmett, Geoffrey, and David Stirzaker. 2001. Probability and Random Processes. 3rd ed. Oxford University Press.

Conditional Probability

For two events A and B with \operatorname{P}(B) > 0, the conditional probability of A given B is \operatorname{P}(A \mid B) = \frac{\operatorname{P}(A \cap B)}{\operatorname{P}(B)}. Once we learn that B occurred, the outcomes outside B are no longer possible. Conditioning keeps the outcomes in B and rescales their probabilities by 1 / \operatorname{P}(B) so that they add up to one.

Example 1 Suppose we roll a fair six-sided die and define the events A = \{2, 4, 6\} \quad \text{and} \quad B = \{4, 5, 6\}. Then A \cap B = \{4, 6\}, \qquad \operatorname{P}(B) = \frac{3}{6}, \qquad \operatorname{P}(A \cap B) = \frac{2}{6}. Therefore, the conditional probability of A given B is \operatorname{P}(A \mid B) = \frac{\operatorname{P}(A \cap B)}{\operatorname{P}(B)} = \frac{2/6}{3/6} = \frac{2}{3}. Learning that the roll is at least four raises the probability that it is even from 1/2 to 2/3.

Rearranging the definition gives the multiplication rule \operatorname{P}(A \cap B) = \operatorname{P}(A \mid B) \operatorname{P}(B). If the events B_{1}, B_{2}, \ldots form a partition of \Omega, meaning that they are pairwise disjoint and their union is \Omega, then countable additivity and the multiplication rule give the law of total probability, \operatorname{P}(A) = \sum_{k} \operatorname{P}(A \cap B_{k}) = \sum_{k} \operatorname{P}(A \mid B_{k}) \operatorname{P}(B_{k}), where the sum runs over the events with \operatorname{P}(B_{k}) > 0. The probability of A is a weighted average of its conditional probabilities, with weights given by the probabilities of the events in the partition. We will meet this idea again when we average conditional expectations.

Independence

Independent Events

Two events A and B are independent if \operatorname{P}(A \cap B) = \operatorname{P}(A) \operatorname{P}(B). When \operatorname{P}(B) > 0, dividing by \operatorname{P}(B) shows that independence is equivalent to \operatorname{P}(A \mid B) = \operatorname{P}(A): learning that B occurred does not change the probability of A. The definition in terms of the product has the advantage of being symmetric in A and B and of making sense even when one of the events has probability zero.

Example 2 Suppose that the weather tomorrow can be either sunny, fair or rainy. In addition, a certain stock tomorrow can either go up or down in price.

We can define W = \{\text{sunny}, \text{fair}, \text{rainy}\} and S = \{\text{up}, \text{down}\}. The set of outcomes can be described as all possible pairwise combinations of weather tomorrow and the stock price movement. The sample space \Omega is then the Cartesian product of W and S, i.e., \Omega = W \times S.

We can then define the weather events \begin{aligned} \text{Sunny} & = \{(\text{sunny}, \text{up}), (\text{sunny}, \text{down})\}, \\ \text{Fair} & = \{(\text{fair}, \text{up}), (\text{fair}, \text{down})\}, \\ \text{Rainy} & = \{(\text{rainy}, \text{up}), (\text{rainy}, \text{down})\}. \end{aligned}

The table below describes the probabilities for tomorrow’s weather.

Weather Sunny Fair Rainy
Probability 0.3 0.5 0.2

Similarly, the stock events can be defined as \begin{aligned} \text{Up} & = \{(\text{sunny}, \text{up}), (\text{fair}, \text{up}), (\text{rainy}, \text{up})\}, \\ \text{Down} & = \{(\text{sunny}, \text{down}), (\text{fair}, \text{down}), (\text{rainy}, \text{down})\}. \\ \end{aligned}

The probabilities of the stock price going up or down are described in the table below.

Stock Up Down
Probability 0.6 0.4

If the weather does not affect the likelihood of the stock going up or down, the conditional probability that the stock goes up should be the same whatever the weather, so that \operatorname{P}(\text{Up} \mid \text{Sunny}) = \operatorname{P}(\text{Up}) = 0.6, and similarly for the other combinations. By the multiplication rule, this means that \operatorname{P}(\text{Stock} \cap \text{Weather}) = \operatorname{P}(\text{Stock}) \operatorname{P}(\text{Weather}), where \text{Stock} is either \text{Up} or \text{Down}, and \text{Weather} is either \text{Sunny}, \text{Fair}, or \text{Rainy}.

The table below describes the combined probabilities of the stock price movement and the weather tomorrow that are consistent with the independence of those events.

Stock\Weather Sunny Fair Rainy
Up 0.18 0.30 0.12
Down 0.12 0.20 0.08

In each column, the stock goes up 60% of the time and down 40% of the time. The weather does not change the relative proportions of the probabilities for the stock price.

The previous example shows how to generate independent events out of two discrete probability spaces (\Omega_{1}, \operatorname{P}_{1}) and (\Omega_{2}, \operatorname{P}_{2}). If we define \Omega = \Omega_{1} \times \Omega_{2} and let \operatorname{P}(\omega_{1}, \omega_{2}) = \operatorname{P}_{1}(\omega_{1}) \operatorname{P}_{2}(\omega_{2}) for each \omega_{1} \in \Omega_{1} and \omega_{2} \in \Omega_{2}, the pair (\Omega, \operatorname{P}) is a well-defined discrete probability space. In this new probability space, the events A = \{\omega_{1}\} \times \Omega_{2} and B = \Omega_{1} \times \{\omega_{2}\} are independent for any \omega_{1} \in \Omega_{1} and \omega_{2} \in \Omega_{2}.

Proof We have that \begin{aligned} \operatorname{P}(A) & = \sum_{\omega_{2}' \in \Omega_{2}} \operatorname{P}(\omega_{1}, \omega_{2}') \\ & = \sum_{\omega_{2}' \in \Omega_{2}} \operatorname{P}_{1}(\omega_{1}) \operatorname{P}_{2}(\omega_{2}') \\ & = \operatorname{P}_{1}(\omega_{1}) \sum_{\omega_{2}' \in \Omega_{2}} \operatorname{P}_{2}(\omega_{2}') \\ & = \operatorname{P}_{1}(\omega_{1}). \end{aligned} Similarly, \operatorname{P}(B) = \operatorname{P}_{2}(\omega_{2}). Since A \cap B = \{(\omega_{1}, \omega_{2})\}, we have that \operatorname{P}(A \cap B) = \operatorname{P}_{1}(\omega_{1}) \operatorname{P}_{2}(\omega_{2}) = \operatorname{P}(A) \operatorname{P}(B), proving that A and B are independent.

This product construction is how we build models of repeated experiments, such as a sequence of independent coin flips, where the outcome of each flip is unaffected by the others.

Example 3 The sample space \Omega is independent of any event A \subset \Omega, since \operatorname{P}(A \cap \Omega) = \operatorname{P}(A) = \operatorname{P}(A) \operatorname{P}(\Omega). Learning that \Omega occurred, which we knew all along, does not change the probability of A. For the same reason, the empty set is independent of every event.

Independent Random Variables

Two random variables X and Y are independent if the events \{X = x\} and \{Y = y\} are independent for all values x and y. Thus, if X and Y are independent we have that \operatorname{P}(X = x, Y = y) = \operatorname{P}(X = x) \operatorname{P}(Y = y) \quad \text{for all } x, y, or equivalently, the joint probability mass function factors into the product of the marginals, p_{X, Y}(x, y) = p_{X}(x) \, p_{Y}(y). More generally, the random variables X_{1}, X_{2}, \ldots, X_{n} are independent if their joint probability mass function factors as \operatorname{P}(X_{1} = x_{1}, \ldots, X_{n} = x_{n}) = \operatorname{P}(X_{1} = x_{1}) \cdots \operatorname{P}(X_{n} = x_{n}) \quad \text{for all } x_{1}, \ldots, x_{n}. An infinite sequence X_{1}, X_{2}, \ldots is independent if every finite subcollection is independent. The coin flips that drive a random walk are the leading example.

If X and Y are independent, anything we compute from X alone is also independent of anything we compute from Y alone. This fact, and its consequence for expectations, is used constantly in the rest of the course.

WarningFunctions of Independent Random Variables

If X and Y are independent and f and g are real-valued functions, then f(X) and g(Y) are independent and \operatorname{E}(f(X) g(Y)) = \operatorname{E}(f(X)) \operatorname{E}(g(Y)), \tag{1} provided the expectations exist. In particular, \operatorname{E}(X Y) = \operatorname{E}(X) \operatorname{E}(Y).

Proof For any values a and b, the event \{f(X) = a\} is the disjoint union of the events \{X = x\} over the values x with f(x) = a, and similarly for \{g(Y) = b\}. Therefore, \begin{aligned} \operatorname{P}(f(X) = a, g(Y) = b) & = \sum_{x : f(x) = a} \; \sum_{y : g(y) = b} p_{X}(x) \, p_{Y}(y) \\ & = \sum_{x : f(x) = a} p_{X}(x) \sum_{y : g(y) = b} p_{Y}(y) \\ & = \operatorname{P}(f(X) = a) \operatorname{P}(g(Y) = b), \end{aligned} so f(X) and g(Y) are independent. For the expectation, \begin{aligned} \operatorname{E}(f(X) g(Y)) & = \sum_{x \in R_{X}} \sum_{y \in R_{Y}} f(x) g(y) \, p_{X}(x) \, p_{Y}(y) \\ & = \sum_{x \in R_{X}} f(x) \, p_{X}(x) \sum_{y \in R_{Y}} g(y) \, p_{Y}(y) \\ & = \operatorname{E}(f(X)) \operatorname{E}(g(Y)). \end{aligned} Taking f(x) = x and g(y) = y gives \operatorname{E}(X Y) = \operatorname{E}(X) \operatorname{E}(Y).

The notes on generating functions rely on this result with f(x) = z^{x} and g(y) = z^{y}, which shows that the generating function of a sum of independent random variables is the product of their generating functions.

Equation (1) also implies that independent random variables are uncorrelated. Indeed, \operatorname{Cov}(X, Y) = \operatorname{E}(XY) - \operatorname{E}(X) \operatorname{E}(Y) = \operatorname{E}(X) \operatorname{E}(Y) - \operatorname{E}(X) \operatorname{E}(Y) = 0. As a consequence, the variance of a sum of independent random variables is the sum of their variances, as shown for uncorrelated random variables in the notes on probability basics. The converse is not true: random variables can be uncorrelated without being independent.

Example 4 Consider a random variable X that takes the values \{1, 0, -1\}, each with probability 1 /3. We compute: \operatorname{E}(X) = \frac{1}{3}(1) + \frac{1}{3}(0) + \frac{1}{3}(-1) = 0, and \operatorname{E}(X^{3}) = \frac{1}{3}(1^{3}) + \frac{1}{3}(0^{3}) + \frac{1}{3}((-1)^{3}) = 0. Now, define Y = X^{2}. The covariance between X and Y is \operatorname{Cov}(X, Y) = \operatorname{E}(XY) - \operatorname{E}(X)\operatorname{E}(Y) = \operatorname{E}(X^{3}) - \operatorname{E}(X)\operatorname{E}(X^{2}) = 0. However, X and Y are not independent. Since Y = 0 exactly when X = 0, \operatorname{P}(X = 0, Y = 0) = \frac{1}{3} \neq \frac{1}{3} \times \frac{1}{3} = \operatorname{P}(X = 0) \operatorname{P}(Y = 0). The random variables are uncorrelated, yet Y is completely determined by X.

Covariance measures only the linear association between two random variables. In the example, Y depends on X through the symmetric function x^{2}, and positive and negative values of X cancel in the covariance.

Conditional Expectation Given a Random Variable

For x such that p_{X}(x) > 0, the conditional probability mass function of Y given X = x is p_{Y \mid X}(y \mid x) = \frac{p_{X, Y}(x, y)}{p_{X}(x)} = \operatorname{P}(Y = y \mid X = x). The conditional expectation of Y given X = x is the expectation of Y computed with this conditional pmf, \operatorname{E}(Y \mid X = x) = \sum_{y \in R_{Y}} y\, p_{Y \mid X}(y \mid x). This is a number that depends on x. Evaluating it at the observed value of X gives a random variable, denoted \operatorname{E}(Y \mid X), which assigns to each outcome \omega the value \operatorname{E}(Y \mid X = X(\omega)). In words, \operatorname{E}(Y \mid X) is our best forecast of Y once we have observed X, and it is random because it depends on what we observe.

When X and Y are independent the conditional pmf equals the marginal pmf, p_{Y \mid X}(y \mid x) = p_{Y}(y), and therefore \operatorname{E}(Y \mid X) = \operatorname{E}(Y). Observing X does not change our forecast of Y.

A key result in probability theory is the law of iterated expectations: \begin{aligned} \operatorname{E}(\operatorname{E}(Y \mid X)) & = \sum_{x \in R_{X}} \operatorname{E}(Y \mid X = x)\, p_{X}(x) \\ & = \sum_{x \in R_{X}} \sum_{y \in R_{Y}} y\, p_{Y \mid X}(y \mid x)\, p_{X}(x) \\ & = \sum_{x \in R_{X}} \sum_{y \in R_{Y}} y\, p_{X, Y}(x, y) \\ & = \sum_{y \in R_{Y}} y\, p_{Y}(y) \\ & = \operatorname{E}(Y). \end{aligned} The expected value of the conditional expectation equals the expected value of Y itself. Averaging our forecasts over everything we might observe gives back the unconditional forecast, in the same way that the law of total probability averages conditional probabilities.

Example 5 The notes on probability basics used the following joint pmf of a random variable X taking values in \{-1, 0, 1\} and a random variable Y taking values in \{0, 1, 2, 3\}: \begin{array}{c|cccc} X \setminus Y & 0 & 1 & 2 & 3 \\ \hline -1 & 0.12500 & 0.09375 & 0.06250 & 0.03125 \\ 0 & 0.06250 & 0.12500 & 0.12500 & 0.06250 \\ 1 & 0.03125 & 0.06250 & 0.09375 & 0.12500 \end{array} We compute the conditional expectation of Y given X = 1. Adding up the row X = 1, p_{X}(1) = 0.03125 + 0.06250 + 0.09375 + 0.12500 = 0.3125. Dividing each entry of the row by p_{X}(1), the conditional pmf is p_{Y \mid X}(y \mid 1) = 0.1,\; 0.2,\; 0.3,\; 0.4 \quad \text{for } y = 0, 1, 2, 3. Therefore, \operatorname{E}(Y \mid X = 1) = 0(0.1) + 1(0.2) + 2(0.3) + 3(0.4) = 2. The same calculation for the other rows gives \operatorname{E}(Y \mid X = -1) = 1 and \operatorname{E}(Y \mid X = 0) = 1.5, with p_{X}(-1) = 0.3125 and p_{X}(0) = 0.375. The law of iterated expectations then yields \operatorname{E}(Y) = 1 (0.3125) + 1.5 (0.375) + 2 (0.3125) = 1.5.

Information

Information Generated by a Random Variable

To describe how information arrives over time, we need a way to say what we know after observing a random variable. Consider a sample space with four possible outcomes \Omega = \{ \omega_{1}, \omega_{2}, \omega_{3}, \omega_{4} \} and three random variables X, Y, and Z with the following values:

Outcome Probability X Y Z
\omega_{1} 0.10 -10 20 15
\omega_{2} 0.30 -5 10 -10
\omega_{3} 0.40 5 0 15
\omega_{4} 0.20 10 0 -10

Observing X reveals which outcome occurred, since X takes a different value at each outcome. Observing Y does not: if Y = 0 we only know that either \omega_{3} or \omega_{4} occurred. Observing Z leaves two possible outcomes whatever its value.

Each random variable splits the sample space into the sets on which it takes each of its values. For Y, these sets are \{Y = 20\} = \{\omega_{1}\}, \qquad \{Y = 10\} = \{\omega_{2}\}, \qquad \{Y = 0\} = \{\omega_{3}, \omega_{4}\}, and for Z, \{Z = 15\} = \{\omega_{1}, \omega_{3}\}, \qquad \{Z = -10\} = \{\omega_{2}, \omega_{4}\}. These sets form a partition of \Omega, called the partition generated by the random variable. Observing the random variable tells us which cell of the partition contains the true outcome, and nothing more. We can then decide whether any event made of whole cells has occurred, but not events that split a cell. The collection of all events that are unions of cells is the information set generated by the random variable. We denote it by \mathcal{F}_{Y} for Y, and similarly for other random variables: \mathcal{F}_{Y} = \{ \emptyset, \{\omega_{1}\}, \{\omega_{2}\}, \{\omega_{1}, \omega_{2}\}, \{\omega_{3}, \omega_{4}\}, \{\omega_{1}, \omega_{3}, \omega_{4}\}, \{\omega_{2}, \omega_{3}, \omega_{4}\}, \Omega\}, and \mathcal{F}_{Z} = \{ \emptyset, \{\omega_{1}, \omega_{3}\}, \{\omega_{2}, \omega_{4}\}, \Omega\}. Since X separates all four outcomes, \mathcal{F}_{X} contains every subset of \Omega. A finer partition corresponds to more information, and in this example \mathcal{F}_{Z} \subset \mathcal{F}_{X} and \mathcal{F}_{Y} \subset \mathcal{F}_{X}. Neither \mathcal{F}_{Y} nor \mathcal{F}_{Z} contains the other: knowing Y does not tell us whether \{\omega_{1}, \omega_{3}\} occurred, and knowing Z does not tell us whether \{\omega_{1}\} occurred. In the general theory, the information set generated by a random variable is called the \sigma-algebra generated by it.

Observing several random variables X_{1}, \ldots, X_{n} together generates the partition whose cells are the sets \{X_{1} = x_{1}, \ldots, X_{n} = x_{n}\}, and the corresponding information set is denoted by \mathcal{F}_{X_{1}, \ldots, X_{n}}. For simplicity, in what follows we write \mathcal{F} for an information set generated by a partition of \Omega into cells C_{1}, C_{2}, \ldots with positive probabilities.

A random variable W is known given \mathcal{F}, or \mathcal{F}-measurable, if it is constant on each cell of the partition, so that observing the information in \mathcal{F} reveals the value of W. In the example, Y^{2} is known given \mathcal{F}_{Y} but Z is not, since Z takes the values 15 and -10 on the cell \{Y = 0\} = \{\omega_{3}, \omega_{4}\}. A random variable is known given \mathcal{F}_{Y} exactly when it can be written as a function of Y.

Conditional Expectation Given Information

The conditional expectation \operatorname{E}(Y \mid X) averages Y over the outcomes that are consistent with the observed value of X. The same idea works for any information set. Given \mathcal{F}, we know which cell of the partition contains the outcome, so our best forecast of a random variable Y is its average over that cell.

WarningConditional Expectation Given an Information Set

Let \mathcal{F} be the information set generated by a partition C_{1}, C_{2}, \ldots of \Omega with \operatorname{P}(C_{k}) > 0 for all k. The conditional expectation of Y given \mathcal{F} is the random variable \operatorname{E}(Y \mid \mathcal{F}) defined by \operatorname{E}(Y \mid \mathcal{F})(\omega) = \frac{\operatorname{E}(Y \mathbf{1}_{C_{k}})}{\operatorname{P}(C_{k})} = \frac{1}{\operatorname{P}(C_{k})} \sum_{\omega' \in C_{k}} Y(\omega') \operatorname{P}(\omega') \qquad \text{for } \omega \in C_{k}.

On each cell, \operatorname{E}(Y \mid \mathcal{F}) equals the expectation of Y computed with the probabilities conditional on that cell. By construction, \operatorname{E}(Y \mid \mathcal{F}) is constant on each cell, so it is known given \mathcal{F}. When \mathcal{F} = \mathcal{F}_{X}, the cells are the sets \{X = x\} and the definition reduces to \operatorname{E}(Y \mid X). When \mathcal{F} = \{\emptyset, \Omega\} contains no information, the only cell is \Omega and \operatorname{E}(Y \mid \mathcal{F}) = \operatorname{E}(Y).

Example 6 Using the table above, we compute \operatorname{E}(X \mid \mathcal{F}_{Z}). The partition generated by Z has two cells, C_{1} = \{\omega_{1}, \omega_{3}\} and C_{2} = \{\omega_{2}, \omega_{4}\}, each with probability 0.5. On each cell we average X: \begin{aligned} \operatorname{E}(X \mid \mathcal{F}_{Z}) & = \frac{(-10)(0.10) + 5 (0.40)}{0.5} = 2 \qquad \text{on } C_{1}, \\ \operatorname{E}(X \mid \mathcal{F}_{Z}) & = \frac{(-5)(0.30) + 10 (0.20)}{0.5} = 1 \qquad \text{on } C_{2}. \end{aligned} If we observe Z = 15 our forecast of X is 2, and if we observe Z = -10 it is 1. Averaging the forecasts over the two cells gives 2 (0.5) + 1 (0.5) = 1.5, which is \operatorname{E}(X) = -1 - 1.5 + 2 + 2 = 1.5, as the law of iterated expectations requires.

The following properties are all we need to work with conditional expectations in the rest of the course. They hold for general information sets, but we prove them only for information sets generated by partitions.

WarningProperties of Conditional Expectation

Let \mathcal{F} be an information set, and let Y and W be random variables with finite expectations.

  1. Linearity. For constants a and b, \operatorname{E}(a Y + b W \mid \mathcal{F}) = a \operatorname{E}(Y \mid \mathcal{F}) + b \operatorname{E}(W \mid \mathcal{F}).

  2. Taking out what is known. If W is known given \mathcal{F}, then \operatorname{E}(W Y \mid \mathcal{F}) = W \operatorname{E}(Y \mid \mathcal{F}). In particular, \operatorname{E}(W \mid \mathcal{F}) = W.

  3. Independence. If Y is independent of every event in \mathcal{F}, then \operatorname{E}(Y \mid \mathcal{F}) = \operatorname{E}(Y).

  4. Tower property. If \mathcal{G} \subset \mathcal{F} is a smaller information set, then \operatorname{E}(\operatorname{E}(Y \mid \mathcal{F}) \mid \mathcal{G}) = \operatorname{E}(Y \mid \mathcal{G}). In particular, \operatorname{E}(\operatorname{E}(Y \mid \mathcal{F})) = \operatorname{E}(Y).

Proof

Fix a cell C of the partition that generates \mathcal{F}. Linearity follows from the linearity of \operatorname{E}(\cdot \, \mathbf{1}_{C}).

If W is known given \mathcal{F}, it takes a constant value w on C, so W Y \mathbf{1}_{C} = w Y \mathbf{1}_{C} and \frac{\operatorname{E}(W Y \mathbf{1}_{C})}{\operatorname{P}(C)} = w \frac{\operatorname{E}(Y \mathbf{1}_{C})}{\operatorname{P}(C)}, which is W \operatorname{E}(Y \mid \mathcal{F}) on C. Taking Y = 1 gives \operatorname{E}(W \mid \mathcal{F}) = W.

If Y is independent of the event C, then \operatorname{P}(Y = y, C) = \operatorname{P}(Y = y) \operatorname{P}(C) for every y, so that \operatorname{E}(Y \mathbf{1}_{C}) = \sum_{y} y \, \operatorname{P}(Y = y, C) = \operatorname{P}(C) \sum_{y} y \, \operatorname{P}(Y = y) = \operatorname{P}(C) \operatorname{E}(Y), and dividing by \operatorname{P}(C) gives \operatorname{E}(Y \mid \mathcal{F}) = \operatorname{E}(Y) on C.

For the tower property, since \mathcal{G} contains less information than \mathcal{F}, each cell D of the partition generating \mathcal{G} is a union of cells C of the partition generating \mathcal{F}. On each such C, \operatorname{E}(Y \mid \mathcal{F}) equals \operatorname{E}(Y \mathbf{1}_{C}) / \operatorname{P}(C), so that \operatorname{E}\left(\operatorname{E}(Y \mid \mathcal{F}) \mathbf{1}_{D}\right) = \sum_{C \subset D} \frac{\operatorname{E}(Y \mathbf{1}_{C})}{\operatorname{P}(C)} \operatorname{P}(C) = \sum_{C \subset D} \operatorname{E}(Y \mathbf{1}_{C}) = \operatorname{E}(Y \mathbf{1}_{D}). Dividing by \operatorname{P}(D) shows that \operatorname{E}(\operatorname{E}(Y \mid \mathcal{F}) \mid \mathcal{G}) = \operatorname{E}(Y \mid \mathcal{G}) on D. Taking \mathcal{G} = \{\emptyset, \Omega\} gives \operatorname{E}(\operatorname{E}(Y \mid \mathcal{F})) = \operatorname{E}(Y).

Each property has a simple interpretation. Taking out what is known says that quantities we have already observed behave like constants when we forecast. The independence property says that information unrelated to Y does not change our forecast. The tower property says that if we first forecast with more information and then average that forecast using less information, we obtain the forecast based on less information: today’s forecast of tomorrow’s forecast is today’s forecast.

Filtrations

Information accumulates over time. A sequence of information sets \mathcal{F}_{0} \subset \mathcal{F}_{1} \subset \mathcal{F}_{2} \subset \cdots, where \mathcal{F}_{n} describes what is known at time n, is called a filtration. The inclusions express that nothing is forgotten: every event whose occurrence is known at time m is still known at any later time n > m.

Example 7 Let X_{1}, X_{2}, \ldots be independent coin flips, each equal to +1 or -1 with probability 1/2, and let \mathcal{F}_{n} = \mathcal{F}_{X_{1}, \ldots, X_{n}} be the information generated by the first n flips, with \mathcal{F}_{0} = \{\emptyset, \Omega\}. An infinite sequence of flips has uncountably many outcomes, but any statement about the first n flips only needs the finite sample space \{-1, +1\}^{n}. The cells of the partition generating \mathcal{F}_{n} are the 2^{n} possible sequences of the first n flips. Each flip splits every cell in two, so the partitions become finer and \mathcal{F}_{n} \subset \mathcal{F}_{n + 1}.

The position of the random walk S_{n} = X_{1} + \cdots + X_{n} is known given \mathcal{F}_{n}, whereas the next flip X_{n + 1} is independent of every event in \mathcal{F}_{n}. The properties of conditional expectation then give \operatorname{E}(S_{n + 1} \mid \mathcal{F}_{n}) = \operatorname{E}(S_{n} \mid \mathcal{F}_{n}) + \operatorname{E}(X_{n + 1} \mid \mathcal{F}_{n}) = S_{n} + \operatorname{E}(X_{n + 1}) = S_{n}. Given everything observed up to time n, the best forecast of the position at time n + 1 is the current position.

A process with this property is called a martingale, and it is the subject of the notes on discrete-time martingales. The calculation in the example uses exactly three of the properties above: linearity, taking out what is known, and independence. The tower property enters when we forecast more than one period ahead.

Practice Problems

Problem 1 Let A and B be events with positive probabilities.

  1. Use the definition of conditional probability and the law of total probability to derive Bayes’ rule, \operatorname{P}(B \mid A) = \frac{\operatorname{P}(A \mid B) \operatorname{P}(B)}{\operatorname{P}(A)} = \frac{\operatorname{P}(A \mid B) \operatorname{P}(B)}{\operatorname{P}(A \mid B) \operatorname{P}(B) + \operatorname{P}(A \mid B^{C}) \operatorname{P}(B^{C})}.
  2. A credit model flags 90% of the firms that will default within a year and 10% of the firms that will not. Two percent of firms default. Given that a firm is flagged, what is the probability that it defaults? Given that it is not flagged?
  3. Explain why the first answer is so much smaller than 90%.
Solution
  1. The multiplication rule gives \operatorname{P}(A \cap B) = \operatorname{P}(A \mid B) \operatorname{P}(B), so \operatorname{P}(B \mid A) = \operatorname{P}(A \cap B) / \operatorname{P}(A) = \operatorname{P}(A \mid B) \operatorname{P}(B) / \operatorname{P}(A). The events B and B^{C} partition \Omega, and the law of total probability gives the denominator \operatorname{P}(A) = \operatorname{P}(A \mid B) \operatorname{P}(B) + \operatorname{P}(A \mid B^{C}) \operatorname{P}(B^{C}).
  2. Let D be the event that the firm defaults and F the event that it is flagged. Then \operatorname{P}(F) = 0.9 (0.02) + 0.1 (0.98) = 0.018 + 0.098 = 0.116, and \operatorname{P}(D \mid F) = \frac{0.018}{0.116} = 0.155, \qquad \operatorname{P}(D \mid F^{C}) = \frac{0.1 (0.02)}{1 - 0.116} = \frac{0.002}{0.884} = 0.0023.
  3. Defaults are rare. Among 1,000 firms, about 20 default and 18 of them are flagged, while 980 survive and 98 of them are flagged. Most flagged firms are survivors, because 10% of a large group outnumbers 90% of a small one. The flag still carries information: it raises the default probability from 2% to 15.5%, and its absence lowers it to 0.23%.

Problem 2 Over each of three periods a stock moves up with probability p = 0.6 and down with probability q = 0.4, independently of the other periods. Let U be the number of up moves.

  1. Describe the sample space and the probability of each outcome. How many outcomes have exactly k up moves?
  2. Find the probability mass function of U.
  3. Write U = I_{1} + I_{2} + I_{3}, where I_{j} = 1 if the stock moves up in period j and I_{j} = 0 otherwise, and use it to compute \operatorname{E}(U) and \operatorname{V}(U).
Solution
  1. The sample space is the product \{u, d\}^{3}, with eight sequences such as uud. By the product construction, a sequence with k up moves has probability p^{k} q^{3 - k}. The number of sequences with k up moves is the number of ways to choose the k periods with an up move, \binom{3}{k}, which is 1, 3, 3, 1 for k = 0, 1, 2, 3.
  2. \operatorname{P}(U = k) = \binom{3}{k} p^{k} q^{3 - k}, which gives 0.4^{3} = 0.064, 3 (0.6)(0.4)^{2} = 0.288, 3 (0.6)^{2}(0.4) = 0.432, and 0.6^{3} = 0.216. This is the binomial distribution with n = 3 and p = 0.6.
  3. Each I_{j} has \operatorname{E}(I_{j}) = p and \operatorname{V}(I_{j}) = p - p^{2} = p q. The I_{j} are independent and therefore uncorrelated, so \operatorname{E}(U) = 3 p = 1.8 and \operatorname{V}(U) = 3 p q = 0.72. Computing them from the pmf gives the same answer with more work.

Problem 3 Independence is easy to misjudge. The following three facts help.

  1. Show that two disjoint events with positive probabilities are never independent.
  2. Show that if A and B are independent, then so are A and B^{C}.
  3. Flip two fair coins, and let A be the event that the first coin lands heads, B the event that the second lands heads, and C the event that both coins land the same way. Show that each pair of these events is independent, but that \operatorname{P}(A \cap B \cap C) \neq \operatorname{P}(A) \operatorname{P}(B) \operatorname{P}(C). What does knowing A and B tell you about C?
Solution
  1. If A \cap B = \emptyset, then \operatorname{P}(A \cap B) = 0, while \operatorname{P}(A) \operatorname{P}(B) > 0. Learning that A occurred tells us that B did not, which is the opposite of independence.
  2. A is the disjoint union of A \cap B and A \cap B^{C}, so \operatorname{P}(A \cap B^{C}) = \operatorname{P}(A) - \operatorname{P}(A \cap B) = \operatorname{P}(A) - \operatorname{P}(A) \operatorname{P}(B) = \operatorname{P}(A) (1 - \operatorname{P}(B)) = \operatorname{P}(A) \operatorname{P}(B^{C}).
  3. The four outcomes HH, HT, TH, TT each have probability 1/4, and A = \{HH, HT\}, B = \{HH, TH\}, C = \{HH, TT\} each have probability 1/2. Each pairwise intersection is \{HH\}, with probability 1/4 = \frac{1}{2} \times \frac{1}{2}, so each pair is independent. But A \cap B \cap C = \{HH\} also has probability 1/4, not 1/8. Each event alone says nothing about another, but A and B together determine C completely. This is why the definition of independence for several random variables requires the joint pmf to factor, not just every pair.

Problem 4 (★) A fund’s gross return in each of two years is 1.2 or 0.9 with probability 1/2 each, independently across years. Let R_{1} and R_{2} be the two gross returns.

  1. Compute \operatorname{E}(R_{1}) and \operatorname{E}(R_{1} R_{2}).
  2. Compute \operatorname{V}(R_{1} R_{2}) using \operatorname{E}\left((R_{1} R_{2})^{2}\right) = \operatorname{E}(R_{1}^{2}) \operatorname{E}(R_{2}^{2}). Why does this equation hold?
  3. Find the probability mass function of R_{1} R_{2}. What is the most likely value, and how does it compare with \operatorname{E}(R_{1} R_{2})?
Solution
  1. \operatorname{E}(R_{1}) = \frac{1}{2} (1.2 + 0.9) = 1.05. By (1), \operatorname{E}(R_{1} R_{2}) = \operatorname{E}(R_{1}) \operatorname{E}(R_{2}) = 1.05^{2} = 1.1025.
  2. R_{1}^{2} and R_{2}^{2} are functions of independent random variables, so they are independent and the expectation of their product is the product of their expectations. With \operatorname{E}(R_{1}^{2}) = \frac{1}{2} (1.44 + 0.81) = 1.125, \operatorname{V}(R_{1} R_{2}) = 1.125^{2} - 1.1025^{2} = 1.2656 - 1.2155 = 0.0501.
  3. The product is 1.44 with probability 1/4, 1.2 \times 0.9 = 1.08 with probability 1/2, and 0.81 with probability 1/4. The most likely outcome, one good year and one bad year, grows wealth by 8%, less than the expected 10.25%. The typical growth per year is \sqrt{1.08} = 1.039, below the expected gross return of 1.05. The gap between the expected return and the typical growth rate reappears with geometric Brownian motion, where the log price grows at \mu - \frac{1}{2} \sigma^{2} while the expected price grows at \mu.

Problem 5 (★) A stock trades at $100 and over each of two periods moves up by 20% with probability p = 0.6 or down by 10% with probability 0.4, independently across periods. Let \mathcal{F}_{1} be the information generated by the first move, and let C = \max(S_{2} - 100, 0) be the payoff of a call at time 2.

  1. List the outcomes, their probabilities, and the values of S_{1}, S_{2}, and C.
  2. Compute \operatorname{E}(C \mid \mathcal{F}_{1}) on each cell of the partition that generates \mathcal{F}_{1}.
  3. Compute \operatorname{E}(C) by averaging \operatorname{E}(C \mid \mathcal{F}_{1}), and check the answer against the direct calculation from the pmf of C.
Solution

  1. Writing u for an up move and d for a down move:

    Outcome Probability S_{1} S_{2} C
    uu 0.36 120 144 44
    ud 0.24 120 108 8
    du 0.24 90 108 8
    dd 0.16 90 81 0

  2. The cells are \{uu, ud\} and \{du, dd\}, each describing one value of S_{1}. On the first, \operatorname{E}(C \mid \mathcal{F}_{1}) = 0.6 (44) + 0.4 (8) = 29.6, and on the second, \operatorname{E}(C \mid \mathcal{F}_{1}) = 0.6 (8) + 0.4 (0) = 4.8. By independence, the conditional probabilities of the second move are p and 1 - p on each cell.

  3. The tower property gives \operatorname{E}(C) = 0.6 (29.6) + 0.4 (4.8) = 17.76 + 1.92 = 19.68. Directly, \operatorname{E}(C) = 0.36 (44) + 0.48 (8) = 15.84 + 3.84 = 19.68. Working backward one period at a time is how the binomial model prices options, with the probabilities replaced by risk-neutral ones and the expectations discounted.

Problem 6 (★) Say that Y is mean independent of X if \operatorname{E}(Y \mid X) = \operatorname{E}(Y).

  1. Explain why independence implies mean independence.
  2. Use the properties of conditional expectation to show that mean independence implies \operatorname{Cov}(X, Y) = 0.
  3. Let X take the values -1, 0, 1 with probability 1/3 each, and Y = X^{2}. Show that X and Y are uncorrelated but Y is not mean independent of X.
  4. Let X be 0 or 1 with probability 1/2 each. When X = 0, let Y = 0, and when X = 1, let Y be +1 or -1 with probability 1/2 each. Show that Y is mean independent of X but not independent of it. Interpret X as a calm or volatile market and Y as a return.
Solution
  1. If Y is independent of X, it is independent of every event in \mathcal{F}_{X}, and the independence property gives \operatorname{E}(Y \mid X) = \operatorname{E}(Y).
  2. Taking out what is known and the tower property give \operatorname{E}(X Y) = \operatorname{E}\left(\operatorname{E}(X Y \mid X)\right) = \operatorname{E}\left(X \operatorname{E}(Y \mid X)\right) = \operatorname{E}\left(X \operatorname{E}(Y)\right) = \operatorname{E}(X) \operatorname{E}(Y), so \operatorname{Cov}(X, Y) = 0.
  3. As in Example 4, \operatorname{Cov}(X, Y) = \operatorname{E}(X^{3}) - \operatorname{E}(X) \operatorname{E}(X^{2}) = 0. But Y is a function of X, so \operatorname{E}(Y \mid X) = X^{2}, which is 1 when X = \pm 1 and 0 when X = 0, while \operatorname{E}(Y) = 2/3.
  4. \operatorname{E}(Y \mid X = 0) = 0 and \operatorname{E}(Y \mid X = 1) = \frac{1}{2} (1) + \frac{1}{2} (-1) = 0, so \operatorname{E}(Y \mid X) = 0 = \operatorname{E}(Y). But \operatorname{P}(Y = 0 \mid X = 0) = 1 while \operatorname{P}(Y = 0) = 1/2, so they are not independent. Knowing whether the market is calm does not change the expected return, but it changes how much the return can move. This is how volatility behaves in financial markets: hard to predict in direction, much easier to predict in size. Independence implies mean independence, which implies zero covariance, and neither implication can be reversed.

Problem 7 (★★) Let X_{1}, X_{2}, \ldots be independent, each equal to +1 with probability p and -1 with probability q = 1 - p. Let S_{n} = X_{1} + \cdots + X_{n}, with S_{0} = 0, and let \mathcal{F}_{n} be the information generated by the first n steps.

  1. Show that \operatorname{E}(S_{n + 1} \mid \mathcal{F}_{n}) = S_{n} + (p - q), and use the tower property to show that \operatorname{E}(S_{n + k} \mid \mathcal{F}_{n}) = S_{n} + k (p - q).
  2. Show that \operatorname{E}(S_{n + 1}^{2} \mid \mathcal{F}_{n}) = S_{n}^{2} + 2 (p - q) S_{n} + 1. Conclude that when p = 1/2, \operatorname{E}(S_{n + 1}^{2} - (n + 1) \mid \mathcal{F}_{n}) = S_{n}^{2} - n.
  3. For general p, let M_{n} = S_{n} - n (p - q) and \sigma^{2} = \operatorname{V}(X_{1}) = 4 p q. Show that \operatorname{E}(M_{n + 1}^{2} - (n + 1) \sigma^{2} \mid \mathcal{F}_{n}) = M_{n}^{2} - n \sigma^{2}.
Solution
  1. S_{n} is known given \mathcal{F}_{n} and X_{n + 1} is independent of it, so \operatorname{E}(S_{n + 1} \mid \mathcal{F}_{n}) = S_{n} + \operatorname{E}(X_{n + 1}) = S_{n} + (p - q). Suppose that \operatorname{E}(S_{n + k} \mid \mathcal{F}_{n}) = S_{n} + k (p - q). Since \mathcal{F}_{n} \subset \mathcal{F}_{n + k}, the tower property gives \operatorname{E}(S_{n + k + 1} \mid \mathcal{F}_{n}) = \operatorname{E}\left(\operatorname{E}(S_{n + k + 1} \mid \mathcal{F}_{n + k}) \mid \mathcal{F}_{n}\right) = \operatorname{E}(S_{n + k} \mid \mathcal{F}_{n}) + (p - q) = S_{n} + (k + 1)(p - q), and the result follows by induction on k.
  2. Since S_{n + 1}^{2} = S_{n}^{2} + 2 S_{n} X_{n + 1} + X_{n + 1}^{2} and X_{n + 1}^{2} = 1, taking out what is known gives \operatorname{E}(S_{n + 1}^{2} \mid \mathcal{F}_{n}) = S_{n}^{2} + 2 S_{n} \operatorname{E}(X_{n + 1}) + 1 = S_{n}^{2} + 2 (p - q) S_{n} + 1. When p = 1/2, \operatorname{E}(S_{n + 1}^{2} \mid \mathcal{F}_{n}) = S_{n}^{2} + 1, and subtracting n + 1 from both sides gives the result. The process S_{n}^{2} - n is a martingale, the discrete version of B_{t}^{2} - t for Brownian motion.
  3. M_{n + 1} = M_{n} + Y_{n + 1}, where Y_{n + 1} = X_{n + 1} - (p - q) has mean zero and variance \sigma^{2} = 1 - (p - q)^{2} = 4 p q and is independent of \mathcal{F}_{n}. Then \operatorname{E}(M_{n + 1}^{2} \mid \mathcal{F}_{n}) = M_{n}^{2} + 2 M_{n} \operatorname{E}(Y_{n + 1}) + \operatorname{E}(Y_{n + 1}^{2}) = M_{n}^{2} + \sigma^{2}, and subtracting (n + 1) \sigma^{2} gives the result. The notes on discrete-time martingales use these martingales to compute the expected duration of the gambler’s game.

Problem 8 (★★) Let X and Y be random variables, and write m(X) = \operatorname{E}(Y \mid X).

  1. Show that for any function g, \operatorname{E}\left((Y - g(X))^{2}\right) = \operatorname{E}\left((Y - m(X))^{2}\right) + \operatorname{E}\left((m(X) - g(X))^{2}\right), so that m(X) is the best forecast of Y based on X in the mean-square sense. Hint: compute the expectation of the cross term by first conditioning on X.
  2. Show that \operatorname{V}(Y) = \operatorname{E}(\operatorname{V}(Y \mid X)) + \operatorname{V}(\operatorname{E}(Y \mid X)), \qquad \text{where } \operatorname{V}(Y \mid X) = \operatorname{E}\left((Y - m(X))^{2} \mid X\right).
  3. Verify the variance decomposition for the joint pmf below. \begin{array}{c|ccc} X \setminus Y & 0 & 1 & 2 \\ \hline 0 & 0.20 & 0.15 & 0.05 \\ 1 & 0.10 & 0.20 & 0.30 \end{array}
Solution
  1. Write Y - g(X) = (Y - m(X)) + (m(X) - g(X)) and expand the square. The cross term vanishes. Since \operatorname{E}(Y - m(X) \mid X) = m(X) - m(X) = 0 and m(X) - g(X) is known given \mathcal{F}_{X}, the tower property and taking out what is known give \operatorname{E}\left((Y - m(X))(m(X) - g(X))\right) = \operatorname{E}\left((m(X) - g(X)) \operatorname{E}(Y - m(X) \mid X)\right) = 0. The second term on the right is nonnegative and is zero when g = m, so no forecast based on X has a smaller mean squared error than m(X).
  2. Take g(X) = \operatorname{E}(Y), a constant, in the first part. The left side becomes \operatorname{V}(Y), the first term on the right is \operatorname{E}\left((Y - m(X))^{2}\right) = \operatorname{E}\left(\operatorname{E}\left((Y - m(X))^{2} \mid X\right)\right) = \operatorname{E}(\operatorname{V}(Y \mid X)) by the tower property, and the second term is \operatorname{E}\left((m(X) - \operatorname{E}(Y))^{2}\right) = \operatorname{V}(m(X)), since \operatorname{E}(m(X)) = \operatorname{E}(Y). Total variance is the average variance within each value of X plus the variance of the forecasts across values of X.
  3. The marginals are p_{X}(0) = 0.4 and p_{X}(1) = 0.6. The conditional means are \operatorname{E}(Y \mid X = 0) = (0.15 + 0.10) / 0.4 = 0.625 and \operatorname{E}(Y \mid X = 1) = (0.20 + 0.60) / 0.6 = 1.333, and the conditional second moments are (0.15 + 0.20) / 0.4 = 0.875 and (0.20 + 1.20) / 0.6 = 2.333, so the conditional variances are 0.875 - 0.625^{2} = 0.484 and 2.333 - 1.333^{2} = 0.556. Then \operatorname{E}(Y) = 0.4 (0.625) + 0.6 (1.333) = 1.05, \operatorname{E}(Y^{2}) = 0.35 + 1.40 = 1.75, and \operatorname{V}(Y) = 1.75 - 1.05^{2} = 0.6475. The two terms of the decomposition are \begin{aligned} \operatorname{E}(\operatorname{V}(Y \mid X)) & = 0.4 (0.484) + 0.6 (0.556) = 0.527, \\ \operatorname{V}(\operatorname{E}(Y \mid X)) & = 0.4 (0.625 - 1.05)^{2} + 0.6 (1.333 - 1.05)^{2} = 0.120. \end{aligned} Indeed 0.527 + 0.120 = 0.647. Observing X explains about 19% of the variance of Y.