Probability Basics
This document is a brief introduction to probability concepts that will be used throughout the course. For clarity, I present the main results using a discrete probability space, in which the set of possible outcomes is finite or countably infinite. However, all results extend to the general case, where (\Omega, \mathcal{F}, \operatorname{P}) consists of an arbitrary sample space \Omega, a \sigma-algebra \mathcal{F} of subsets of \Omega, and a probability measure \operatorname{P} defined on \mathcal{F}. The notes on conditioning and independence, which follow these, build on this material to describe how information changes probabilities and expectations.
For a more complete treatment, see introductory textbooks such as Ross (2019) and Blitzstein and Hwang (2019). For more advanced treatments, see Grimmett and Stirzaker (2001) and Durrett (2019). For a classic treatment, see Feller (1968).
Sets
A set is a collection of objects. The objects of a set can be anything you want. For example, a set may contain numbers, letters, cars, or pictures. In our case, we will be concerned with sets that contain future possibilities or outcomes that can occur.
Sets are fundamental in probability theory because events are sets of outcomes. Once outcomes are organized as sets, operations such as unions, intersections, and complements translate directly into statements like “or”, “and”, and “not”, which allows us to define probabilities consistently and prove core probability rules.
One way to define a set is to enumerate its elements. For example, the set of all integers from 1 to 10 is A = \{1, 2, 3, 4, 5, 6, 7, 8, 9, 10\}. Once we have defined a set, we can answer if an object is an element of the set or not. For example, the number 3 is an element of A whereas the number 20 is not. We use the symbol \in to denote membership of a set and \notin to denote non-membership. Therefore, we have that 3 \in A and 20 \notin A.
Some sets can have an infinite number of elements. For example, the natural numbers are defined as \mathbb{N} = \{0, 1, 2, 3, \ldots\}, where the triple dots mean that if n is in \mathbb{N}, then n+1 is also in \mathbb{N}.
Since all elements of A are also members of \mathbb{N}, we say that A is a subset of \mathbb{N} and write it as A \subset \mathbb{N}. Using this terminology, we can redefine the set A defined above in a more Pythonic way: A = \{ n \in \mathbb{N} : n < 11 \}. If we are studying sets of natural numbers, it makes sense to define the universe to be \mathbb{N} and sets under study will be subsets of the universe.
Now, define the set B as B = \{6, 7, 8, 9, 10, 11, 12, 13, 14, 15\}.
The intersection between A and B is the set denoted A \cap B whose members are both in A and B. Using the sets defined above, we have that A \cap B = \{6, 7, 8, 9, 10\}. The union of the sets A and B is the set denoted A \cup B whose members are either in A, B, or both. Thus, using our previously defined sets we have that A \cup B = \{1, 2, 3, \ldots , 14, 15\}. The set difference of A and B is the set denoted A \setminus B whose members are in A but are not members of B. Thus, A \setminus B = \{1, 2, 3, 4, 5\} and B \setminus A = \{11, 12, 13, 14, 15\}. The complement of A is the set denoted by A^{C} whose members are not in A. Of course this statement only makes sense if we define a universe where the elements not in A can live. If the universe is \mathbb{N}, then A^{C} = \mathbb{N} \setminus A = \{11, 12, 13, \ldots\}. Similarly, B^{C} = \{0, 1, 2, 3, 4, 5\} \cup \{16, 17, 18, \ldots\}. Note that if you take all the elements of A out of A you end up with an empty set, that is A \setminus A = \{\}. We typically denote the empty set by \emptyset, but it is good to keep in mind that \emptyset = \{\}. In our universe of natural numbers, no natural number is a member of the empty set. We can write this formally as n \notin \emptyset, \forall n \in \mathbb{N}. The empty set is a subset of every set, since it has no element that could fail to belong to it.
The cardinality of the set A, denoted by |A|, counts the number of elements in A. We then have that |A| = |B| = 10. The empty set has cardinality 0 whereas the cardinality of \mathbb{N} is denoted \aleph_{0}.
The power set of a set C, denoted by \mathcal{P}(C), is the set containing all possible subsets of C. For example, if C = \{1, 2, 3\}, then \mathcal{P}(C) = \{\{\}, \{1\}, \{2\}, \{3\}, \{1, 2\}, \{2, 3\}, \{1, 3\}, \{1, 2, 3\}\}. Clearly, the power sets of A and B are much bigger. For a given set A, the cardinality of its power set is 2^{|A|}. Therefore, \mathcal{P}(A) and \mathcal{P}(B) each contain 2^{10} = 1024 subsets.
Finally, the Cartesian product of A and B is the set denoted by A \times B whose members are all the pairwise combinations of the elements of A and B. \begin{array}{c|cccc} A \times B & 6 & 7 & \dots & 15 \\ \hline 1 & (1, 6) & (1, 7) & \dots & (1, 15) \\ 2 & (2, 6) & (2, 7) & \dots & (2, 15) \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 10 & (10, 6) & (10, 7) & \dots & (10, 15) \end{array}
The cardinality of A \times B is equal to the product of the cardinalities of A and B, i.e., |A \times B| = |A| \times |B|.
Outcomes and Events
In probability theory, the sample space is a non-empty set denoted by \Omega that includes all possible outcomes that can occur. In these notes \Omega is discrete, meaning that it is either finite or countably infinite, so that its elements can be listed as \omega_{1}, \omega_{2}, \ldots A probability measure is a function that assigns to each element \omega of \Omega a number in [0, 1] so that \sum_{\omega \in \Omega} \operatorname{P}(\omega) = 1. When \Omega is infinite the sum is a series, and because its terms are nonnegative the order in which we add them does not matter. An event A is a subset of \Omega, and we define the probability of that event occurring as \operatorname{P}(A) = \sum_{\omega \in A} \operatorname{P}(\omega). \tag{1} We usually denote the set of all events by \mathcal{F}, which for convenience we will take to be the power set of \Omega, i.e., \mathcal{F} = \mathcal{P}(\Omega). Thus, we have that \mathcal{F} contains all possible subsets of \Omega and \operatorname{P} is defined for all those subsets.
The expression (\Omega, \mathcal{F}, \operatorname{P}) then defines a discrete probability space, where \Omega is the sample space, \mathcal{F} is the set of events, and \operatorname{P} is the probability measure. Since \mathcal{F} = \mathcal{P}(\Omega), we can simply write (\Omega, \operatorname{P}) to denote the same probability space.
An immediate consequence of (1) is that \operatorname{P}(\Omega) = 1. Furthermore, if A and B are disjoint subsets of \Omega we have that \begin{aligned} \operatorname{P}(A \cup B) & = \sum_{\omega \in A \cup B} \operatorname{P}(\omega) \\ & = \sum_{\omega \in A} \operatorname{P}(\omega) + \sum_{\omega \in B} \operatorname{P}(\omega) \\ & = \operatorname{P}(A) + \operatorname{P}(B). \end{aligned} If we denote by A^{C} the complement of A in \Omega, the last expression implies that \operatorname{P}(A) + \operatorname{P}(A^{C}) = 1. Also, because \Omega^{C} = \emptyset, we have that \operatorname{P}(\Omega) + \operatorname{P}(\emptyset) = 1, or \operatorname{P}(\emptyset) = 0.
Example 1 If \Omega = \{ \omega_{1}, \omega_{2}, \omega_{3} \}, then \begin{aligned} \mathcal{P}(\Omega) & = \{ \emptyset, \{\omega_{1}\}, \{\omega_{2}\}, \{\omega_{3}\}, \{\omega_{1}, \omega_{2}\}, \{\omega_{2}, \omega_{3}\}, \{\omega_{1}, \omega_{3}\}, \{\omega_{1}, \omega_{2}, \omega_{3}\}\} \end{aligned} defines the collection of all possible events that we can measure. As we saw previously, the cardinality of \mathcal{P}(\Omega) grows exponentially with the size of \Omega.
The function \operatorname{P} such that \operatorname{P}(\omega_{1}) = 1/2, \operatorname{P}(\omega_{2}) = 1/4, and \operatorname{P}(\omega_{3}) = 1/4 defines a probability measure on \Omega. For example, we have that \operatorname{P}(\{\omega_{1}, \omega_{3}\}) = 1/2 + 1/4 = 3/4.
More generally, if A_{1}, A_{2}, \ldots is a finite or countably infinite collection of pairwise disjoint subsets of \Omega, then no outcome \omega belongs to more than one A_{i}. In this case, the probability of their union is simply the sum of their probabilities: \operatorname{P}\left(\bigcup_{i} A_{i}\right) = \sum_{i} \operatorname{P}(A_{i}). This property is called countable additivity. In the general theory it is one of the axioms that a probability measure must satisfy, whereas in a discrete probability space it follows from (1).
Example 2 Let \Omega = \mathbb{N} and set \operatorname{P}(n) = (1/2)^{n + 1} for n = 0, 1, 2, \ldots This is a valid probability measure, since the geometric series gives \sum_{n = 0}^{\infty} \left(\frac{1}{2}\right)^{n + 1} = \frac{1/2}{1 - 1/2} = 1. The probability that the outcome is an even number is \operatorname{P}(\{0, 2, 4, \ldots\}) = \sum_{k = 0}^{\infty} \left(\frac{1}{2}\right)^{2 k + 1} = \frac{1/2}{1 - 1/4} = \frac{2}{3}. Probability spaces like this one describe random quantities with no upper bound, such as the number of coin flips until the first head or the number of defaults in a year.
Random Variables
Definition
A random variable X is a function that assigns a real value to each outcome: X(\omega) for \omega \in \Omega. Several outcomes may have the same value of X.
Example 3 Consider a sample space with four possible outcomes \Omega = \{ \omega_{1}, \omega_{2}, \omega_{3}, \omega_{4} \}. The table below describes the possible values of three random variables denoted by X, Y and Z.
| Outcome | X | Y | Z |
|---|---|---|---|
| \omega_{1} | -10 | 20 | 15 |
| \omega_{2} | -5 | 10 | -10 |
| \omega_{3} | 5 | 0 | 15 |
| \omega_{4} | 10 | 0 | -10 |
Observing the value of X reveals which outcome occurred. For example, if X = 5 then we know that \omega_{3} occurred. Knowing the value of Y or Z, on the other hand, does not. If we learn that Y = 0 we only know that either \omega_{3} or \omega_{4} occurred, and if we learn that Z = 15 we only know that either \omega_{1} or \omega_{3} occurred. The notes on conditioning and independence make precise how much information each random variable carries.
Expectation and Variance
If X is a random variable defined on a discrete probability space (\Omega, \operatorname{P}), the expectation or expected value of X is defined to be \operatorname{E}(X) = \sum_{\omega \in \Omega} X(\omega) \operatorname{P}(\omega), provided the series converges absolutely, which is always the case when \Omega is finite or X is bounded. The variance of X is \operatorname{V}(X) = \operatorname{E}\left((X - \operatorname{E}(X))^{2}\right). The standard deviation is the square-root of the variance, i.e., \sigma_{X} = \sqrt{\operatorname{V}(X)}.
Example 4 Consider the sample space \Omega = \{ \omega_{1}, \omega_{2}, \omega_{3} \} in which we define the probability measure \operatorname{P} such that \operatorname{P}(\omega_{1}) = 1/2, \operatorname{P}(\omega_{2}) = 1/4, and \operatorname{P}(\omega_{3}) = 1/4. Two random variables X and Y are defined on \Omega according to the table below.
| Outcome | Probability | X | Y |
|---|---|---|---|
| \omega_{1} | 1/2 | 10 | 2 |
| \omega_{2} | 1/4 | 8 | 40 |
| \omega_{3} | 1/4 | 4 | 20 |
Using this information, we can compute the expectation of each random variable.
\begin{aligned} \operatorname{E}(X) & = \frac{1}{2} \times 10 + \frac{1}{4} \times 8 + \frac{1}{4} \times 4 = 8, \\ \operatorname{E}(Y) & = \frac{1}{2} \times 2 + \frac{1}{4} \times 40 + \frac{1}{4} \times 20 = 16. \end{aligned} Having computed the expectations of X and Y, we can compute their variances as \begin{aligned} \operatorname{V}(X) & = \frac{1}{2} \times (10 - 8)^{2} + \frac{1}{4} \times (8 - 8)^{2} + \frac{1}{4} \times (4 - 8)^{2} = 6, \\ \operatorname{V}(Y) & = \frac{1}{2} \times (2 - 16)^{2} + \frac{1}{4} \times (40 - 16)^2 + \frac{1}{4} \times (20 - 16)^2 = 246. \end{aligned} Finally, the standard deviations of X and Y are \sigma_{X} = \sqrt{6} \approx 2.45 and \sigma_{Y} = \sqrt{246} \approx 15.68, respectively.
Covariance
The covariance between two random variables X and Y defined on a probability space (\Omega, \operatorname{P}) is defined as \operatorname{Cov}(X, Y) = \operatorname{E}\left((X - \operatorname{E}(X)) (Y - \operatorname{E}(Y))\right), and, provided both standard deviations are positive, their correlation is \rho_{X, Y} = \frac{\operatorname{Cov}(X, Y)}{\sigma_{X} \sigma_{Y}}.
Example 5 Continuing with Example 4, we have that \operatorname{Cov}(X, Y) = \frac{1}{2} \times (10 - 8)(2 - 16) + \frac{1}{4} (8 - 8)(40 - 16) + \frac{1}{4} (4 - 8)(20 - 16) = -18. Thus, \rho_{X, Y} \approx -0.47.
Linear Combinations
In portfolio theory, we usually study linear combinations of random variables of the form Z = \alpha X + \beta Y. The expectation of Z is just a linear combination of the expectations of X and Y, \operatorname{E}(Z) = \alpha \operatorname{E}(X) + \beta \operatorname{E}(Y). \tag{2} The variance of Z, though, includes not only the variances of X and Y but also their covariances, \operatorname{V}(Z) = \alpha^{2} \operatorname{V}(X) + \beta^{2} \operatorname{V}(Y) + 2 \alpha \beta \operatorname{Cov}(X, Y). \tag{3} This is an important result which is at the heart of portfolio diversification.
Proof
The expectation of Z is computed as \begin{aligned} \operatorname{E}(Z) & = \operatorname{E}(\alpha X + \beta Y) \\ & = \sum_{\omega \in \Omega} (\alpha X(\omega) + \beta Y(\omega)) \operatorname{P}(\omega) \\ & = \alpha \sum_{\omega \in \Omega} X(\omega) \operatorname{P}(\omega) + \beta \sum_{\omega \in \Omega} Y(\omega) \operatorname{P}(\omega) \\ & = \alpha \operatorname{E}(X) + \beta \operatorname{E}(Y). \end{aligned}
The variance of Z is computed as \begin{aligned} \operatorname{V}(Z) & = \operatorname{V}(\alpha X + \beta Y) \\ & = \operatorname{E}\left((\alpha X + \beta Y - \operatorname{E}(\alpha X + \beta Y))^{2}\right) \\ & = \operatorname{E}\left((\alpha (X - \operatorname{E}(X)) + \beta (Y - \operatorname{E}(Y)))^{2}\right) \\ & = \operatorname{E}\left(\alpha^{2} (X - \operatorname{E}(X))^{2} + \beta^{2} (Y - \operatorname{E}(Y))^{2} + 2 \alpha \beta (X - \operatorname{E}(X))(Y - \operatorname{E}(Y))\right) \\ & = \alpha^{2} \operatorname{E}\left((X - \operatorname{E}(X))^{2}\right) + \beta^{2} \operatorname{E}\left((Y - \operatorname{E}(Y))^{2}\right) + 2 \alpha \beta \operatorname{E}\left((X - \operatorname{E}(X))(Y - \operatorname{E}(Y))\right) \\ & = \alpha^{2} \operatorname{V}(X) + \beta^{2} \operatorname{V}(Y) + 2 \alpha \beta \operatorname{Cov}(X, Y). \end{aligned}More generally, consider the random variables X_{1}, X_{2}, \ldots, X_{n}, and form a new random variable X such that X = \alpha_{1} X_{1} + \alpha_{2} X_{2} + \ldots + \alpha_{n} X_{n}, where \alpha_{i} \in \mathbb{R} for all i \in \{1, 2, \ldots, n\}.
The expectation of X is a linear combination of the expectations of X_{1}, X_{2}, \ldots, X_{n}. The variance of X, though, takes into account all covariances between X_{i} and X_{j}, for i, j = 1, 2, \ldots, n. Indeed, we have that \operatorname{V}(X) = \sum_{i = 1}^{n} \sum_{j = 1}^{n} \alpha_{i} \alpha_{j} \operatorname{Cov}(X_{i}, X_{j}). \tag{4} The previous expression can be simplified if the random variables X_{1}, X_{2}, \ldots, X_{n} are uncorrelated, that is, if \operatorname{Cov}(X_{i}, X_{j}) = 0 for all i \neq j. We will see in the notes on conditioning and independence that independent random variables are always uncorrelated, so the result below applies in particular to independent random variables. Recognizing that \operatorname{Cov}(X_{i}, X_{i}) = \operatorname{V}(X_{i}), equation (4) implies that
The variance of a sum grows linearly with the number of terms, so its standard deviation grows like \sqrt{n}, while the variance of the average shrinks to zero. The first fact explains why a random walk after n steps is typically at a distance of order \sqrt{n} from its starting point, and it is the reason we will scale a random walk by \sqrt{\Delta t} to obtain Brownian motion. The second fact is the idea behind the law of large numbers: the average of many uncorrelated random variables with a common mean concentrates around that mean.
Properties of Covariance and Correlation
The covariance of X and Y can also be expressed as \operatorname{Cov}(X, Y) = \operatorname{E}(X Y) - \operatorname{E}(X) \operatorname{E}(Y).
Proof
\begin{aligned} \operatorname{Cov}(X, Y) & = \operatorname{E}\left((X - \operatorname{E}(X))(Y - \operatorname{E}(Y))\right) \\ & = \operatorname{E}[X (Y - \operatorname{E}(Y))] - \operatorname{E}[\operatorname{E}(X) (Y - \operatorname{E}(Y))] \\ & = \operatorname{E}(XY) - \operatorname{E}[X \operatorname{E}(Y)] - \operatorname{E}(X) \operatorname{E}(Y - \operatorname{E}(Y)) \\ & = \operatorname{E}(XY) - \operatorname{E}(X) \operatorname{E}(Y). \\ \end{aligned}The correlation between any two random variables with positive variances is always between -1 and 1, and it equals \pm 1 only when one variable is an exact linear function of the other.
Proof
Let Z_{X} = (X - \operatorname{E}(X)) / \sigma_{X} and Z_{Y} = (Y - \operatorname{E}(Y)) / \sigma_{Y} be the standardized versions of X and Y, so that \operatorname{V}(Z_{X}) = \operatorname{V}(Z_{Y}) = 1 and \operatorname{Cov}(Z_{X}, Z_{Y}) = \rho_{X, Y}. A variance cannot be negative, so (3) gives 0 \leq \operatorname{V}(Z_{X} \pm Z_{Y}) = 1 + 1 \pm 2 \rho_{X, Y} = 2 (1 \pm \rho_{X, Y}). The plus sign gives \rho_{X, Y} \geq -1 and the minus sign gives \rho_{X, Y} \leq 1. If \rho_{X, Y} = 1, then \operatorname{V}(Z_{X} - Z_{Y}) = 0, so Z_{X} - Z_{Y} equals its mean of zero with probability one, and Y = \operatorname{E}(Y) + \frac{\sigma_{Y}}{\sigma_{X}} (X - \operatorname{E}(X)). If \rho_{X, Y} = -1, the same argument applied to Z_{X} + Z_{Y} gives Y = \operatorname{E}(Y) - \frac{\sigma_{Y}}{\sigma_{X}} (X - \operatorname{E}(X)).Probability Mass Function
For discrete random variables, the probability mass function (or pmf) is a real-valued function that specifies the probability that the random variable X is equal to a certain value x, i.e., p_{X}(x) = \operatorname{P}(\omega \in \Omega : X(\omega) = x).
Example 7 Suppose the outcomes of Example 3 have the probabilities given in the table below.
| Outcome | \operatorname{P} | X | Y |
|---|---|---|---|
| \omega_{1} | 0.10 | -10 | 20 |
| \omega_{2} | 0.30 | -5 | 10 |
| \omega_{3} | 0.40 | 5 | 0 |
| \omega_{4} | 0.20 | 10 | 0 |
We have that the probability mass function of X is p_{X}(x) = \begin{cases} 0.10 & \text{if } x = -10, \\ 0.30 & \text{if } x = -5, \\ 0.40 & \text{if } x = 5, \\ 0.20 & \text{if } x = 10. \end{cases}
The probability mass function of Y takes positive values only at three points. p_{Y}(y) = \begin{cases} 0.60 & \text{if } y = 0, \\ 0.30 & \text{if } y = 10, \\ 0.10 & \text{if } y = 20. \end{cases}
It is sometimes easier to visualize the probability mass function by plotting the probability of different values of the random variable.
It is apparent from the pictures that p_{X}(x) = 0 if x \notin \{-10, -5, 5, 10\}. Indeed, the set \{\omega \in \Omega : X(\omega) = x \} is empty for all x not equal to -10, -5, 5, or 10. Similarly, p_{Y}(y) = 0 if y \notin \{0, 10, 20\}.
To simplify notation, we will often write \{X = x\} to denote the set \{\omega \in \Omega : X(\omega) = x\}. Using this notation, we have that p_{X}(x) = \operatorname{P}(X = x).
The support of X is the set R_{X} = \{x \in \mathbb{R} : p_{X}(x) > 0\}, which is countable because \Omega is countable. Similarly, the support of Y is R_{Y} = \{y \in \mathbb{R} : p_{Y}(y) > 0\}. Using this notation we can rewrite the expectation of X as \operatorname{E}(X) = \sum_{x \in R_{X}} x\, p_{X}(x), \tag{6} an expression that is commonly used in statistics. The variance of X then becomes \operatorname{V}(X) = \sum_{x \in R_{X}} (x - \operatorname{E}(X))^{2}\, p_{X}(x). Note that we have \begin{aligned} \operatorname{V}(X) & = \sum_{x \in R_{X}} (x - \operatorname{E}(X))^{2}\, p_{X}(x) \\ & = \sum_{x \in R_{X}} (x^{2} - 2 x \operatorname{E}(X) + \operatorname{E}(X)^{2})\, p_{X}(x) \\ & = \sum_{x \in R_{X}} x^{2}\, p_{X}(x) - 2 \operatorname{E}(X) \sum_{x \in R_{X}} x\, p_{X}(x) + \operatorname{E}(X)^{2} \\ & = \operatorname{E}(X^{2}) - 2 \operatorname{E}(X)^{2} + \operatorname{E}(X)^{2} \\ & = \operatorname{E}(X^{2}) - \operatorname{E}(X)^{2}. \end{aligned}
For two random variables X and Y defined in (\Omega, \operatorname{P}), the set \{X = x, Y = y\} denotes all outcomes in \Omega that satisfy \{X = x\} and \{Y = y\}. Therefore, we have that \{X = x, Y = y\} = \{X = x\} \cap \{Y = y\}. The function p_{X, Y}(x, y) = \operatorname{P}(X = x, Y = y) is called the joint probability mass function of X and Y.
Example 8 The joint pmf of the random variables defined in Example 7 is given in the table below.
\begin{array}{c|cccc} X \setminus Y & 0 & 10 & 20 \\ \hline -10 & 0 & 0 & 0.1 \\ -5 & 0 & 0.3 & 0 \\ 5 & 0.4 & 0 & 0 \\ 10 & 0.2 & 0 & 0 \end{array} The function p_{X, Y}(x, y) has many zeros since in Example 7 there are only four outcomes, so only four pairs of values (x, y) can occur. Every other pair has probability zero.
Example 9 We can generate any joint pmf for two random variables as long as the sum of all probabilities is equal to one. The table below reports the joint probabilities of a random variable X taking values in \{-1, 0, 1\} and a random variable Y taking values in \{0, 1, 2, 3\}. \begin{array}{c|cccc} X \setminus Y & 0 & 1 & 2 & 3 \\ \hline -1 & 0.12500 & 0.09375 & 0.06250 & 0.03125 \\ 0 & 0.06250 & 0.12500 & 0.12500 & 0.06250 \\ 1 & 0.03125 & 0.06250 & 0.09375 & 0.12500 \end{array}
In this case the underlying probability space has at least 3 \times 4 = 12 possible outcomes, since all twelve combinations of values have positive probability.
To plot the joint pmf of two random variables we need a three-dimensional graph, as in Figure 2.
We can use the joint pmf to compute the expectation of a function of two random variables. Indeed, we have that \operatorname{E}(f(X, Y)) = \sum_{x \in R_{X}} \sum_{y \in R_{Y}} f(x, y)\, p_{X, Y}(x, y). \tag{7} If we write \mu_{X} = \operatorname{E}(X) and \mu_{Y} = \operatorname{E}(Y), equation (7) implies that the covariance of X and Y can be computed as \operatorname{Cov}(X, Y) = \sum_{x \in R_{X}} \sum_{y \in R_{Y}} (x - \mu_{X})(y - \mu_{Y}) p_{X, Y}(x, y). The joint pmf contains all the information of X and Y since we can recover the individual pmfs of X and Y from it. To find the probability that Y equals a specific value y, we sum over all possible values of X: p_{Y}(y) = \sum_{x \in R_{X}} p_{X, Y}(x, y). This works because the events \{X = x, Y = y\} for different x are disjoint and together cover all ways Y can be y. Similarly, p_{X}(x) = \sum_{y \in R_{Y}} p_{X, Y}(x, y). Therefore, we can marginalize out Y to obtain the probability mass function of X, in the same way that we can marginalize out X to obtain the probability mass function of Y. It is important to note that the joint pmf not only contains the individual information of two random variables but also captures their mutual dependence.
Looking Ahead
These notes describe random variables before any information about the outcome is revealed. The notes on conditioning and independence study how probabilities and expectations change as information arrives, the central idea behind martingales.