18.2: Independence

Last updated
Save as PDF

Page ID: 48432

Eric Lehman, F. Thomson Leighton, & Alberty R. Meyer
Google and Massachusetts Institute of Technology via MIT OpenCourseWare

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

The notion of independence carries over from events to random variables as well. Random variables \(R_1\) and \(R_2\) are independent iff for all \(x_1, x_2\), the two events

\[\nonumber [R_1 = x_1] \text{ and } [R_2 = x_2]\]

are independent.

For example, are \(C\) and \(M\) independent? Intuitively, the answer should be “no.” The number of heads, \(C\), completely determines whether all three coins match; that is, whether \(M = 1\). But, to verify this intuition, we must find some \(x_1, x_2 \in \mathbb{R}\) such that:

\[\nonumber \text{Pr}[C = x_1 \text{ AND } M = x_2] \neq \text{Pr}[C = x_1] \cdot \text{Pr}[M = x_2].\]

One appropriate choice of values is \(x_1 = 2\) and \(x_2 = 1\). In this case, we have:

\[\nonumber \text{Pr}[C = 2 \text{ AND } M = 1] = 0 \neq \frac{1}{4} \cdot \frac{3}{8} = \text{Pr}[M = 1] \cdot \text{Pr}[C = 2].\]

The first probability is zero because we never have exactly two heads \((C = 2)\) when all three coins match \((M = 1)\). The other two probabilities were computed earlier.

On the other hand, let \(H_1\) be the indicator variable for the event that the first flip is a Head, so

\[[H_1 = 1] = \{HHH, HTH, HHT, HTT\}.\]

Then \(H_1\) is independent of \(M\), since

\[\nonumber \text{Pr}[M = 1] = 1/4 = \text{Pr}[M = 1 \mid H_1 = 1] = \text{Pr}[M = 1 \mid H_1 = 0]\]

\[\nonumber \text{Pr}[M = 0] = 3/4 = \text{Pr}[M = 0 \mid H_1 = 1] = \text{Pr}[M = 0 \mid H_1 = 0]\]

This example is an instance of:

Lemma 18.2.1. Two events are independent iff their indicator variables are independent.

The simple proof is left to Problem 18.1.

Intuitively, the independence of two random variables means that knowing some information about one variable doesn’t provide any information about the other one. We can formalize what “some information” about a variable \(R\) is by defining it to be the value of some quantity that depends on \(R\). This intuitive property of independence then simply means that functions of independent variables are also independent:

Lemma 18.2.2. Let \(R\) and \(S\) be independent random variables, and \(f\) and \(g\) be functions such that \(\text{domain}(f) = \text{codomain}(R)\) and \(\text{domain}(f) = \text{codomain}(R)\). Then \(f(R)\) and \(g(S)\) are independent random variables.

The proof is another simple exercise left to Problem 18.30.

As with events, the notion of independence generalizes to more than two random variables.

Definition \(\PageIndex{3}\)

Random variables \(R_1, R_2, \ldots, R_n\) are mutually independent iff for all \(x_1, x_2,\ldots, x_n\), the \(n\) events

\[\nonumber [R_1 = x_1], [R_2 = x_2], \ldots, [R_n = x_n]\]

are mutually independent. They are \(k\)-way independent iff every subset of \(k\) of them are mutually independent.

Lemmas 18.2.1 and 18.2.2 both extend straightforwardly to \(k\)-way independent variables.