Probability and Random Variables

1. Fubini’s theorem

$\textbf{Theorem 2.1 (Fubini’s Theorem).}\quad$
If $I$ and $J$ are countable sets and $\forall i\in I, j \in J: a_{ij} ⩾ 0$ then

$$\sum_{i\in I}\sum_{j\in J}a_{ij}=\sum_{j\in J}\sum_{i\in I}a_{ij} $$

That is, we can rearrange the order of summing countable many non-negative terms.

$Proof.$ Enumerate $J = \{j_1, j_2, . . .\}$. Then

$$\sum_{i \in I} \sum_{j \in J} a_{i j} \geqslant \sum_{i \in I}\left(\sum_{j=1}^{n} a_{i j}\right) $$

by dropping non-negative terms. Now one sum is finite, so we can exchange order to get

$$\sum_{i \in I}\left(\sum_{j=1}^{n} a_{i j}\right)=\sum_{j=1}^{n}\left(\sum_{i \in I} a_{i j}\right) $$

The last sum forms an increasing sequence in n that converges to $\displaystyle \sum_{j\in J}\sum_{i\in I}a_{ij}$ by monotone convergence theorem. Thus

$$\sum_{i\in I}\sum_{j\in J}a_{ij}\geqslant\sum_{j\in J}\sum_{i\in I}a_{ij} $$

By symmetry, the other inequality is also true, so equality holds.

2. Probability and Measure

$\textbf{Definition 2.2.}\quad$ A measure $\mu$ on $(E,\mathcal E)$ is a function $\mu: E\to [0, ∞]$ such that if $(A_n, n ∈ \mathbb N)$ are disjoint then

$$\mu\left(\bigcup_{n \in \mathbb{N}} A_{n}\right)=\sum_{n \in \mathbb{N}} \mu\left(A_{n}\right) $$

Then $(E,\mathcal E, \mu)$ is called a measure space. If $\mu(E) = 1$, then $\mu$ is called a probability measure
and $(E,\mathcal E, \mu)$ is called a probability space.

A probability space $(E, \mathcal E, \mathbb P)$ has outcomes $\omega \in E$ and events $A \in \mathcal E$.

$\textbf{Theorem 2.4.}$(Properties of probability)$\quad$
From the definition of probability measure, it follows that:

(1) $\mathbb P(\emptyset) = 0$

(2) $\mathbb P(A^c) = 1- \mathbb P(A)$

(3) $\mathbb P(A) +\mathbb P(B) = \mathbb P(A\cup B) + \mathbb P(A\cap B)$

3. Random Variables

$\textbf{Definition 2.5.}\quad$ Given a probability space $(\Omega, \mathcal F, \mathbb P)$ and a measurable space $(E, \mathcal E)$, an $(\mathcal E, \mathcal F)$-measurable random variable $(rv)$ is a measurable function $X : Ω \to E$, $i.e.$,

$$\forall A\in \mathcal E: X^{-1}A\in \mathcal F $$

$\textbf{Special case}$: If $E$ is countable, then $X : \Omega \to E$ is a $(2^E, \mathcal F)$-measurable rv $\iff \forall i ∈ E\quad X^{−1}(i) \in \mathcal F$ . The probability mass function (pmf) or distribution of $X$ is defined by

$$\lambda(i):=\mathbb{P}(X=i)=\mathbb{P}(\{\omega \in \Omega \mid X(\omega)=i\}) $$

4. Condition Probability

$\textbf{Definition 2.7.}\quad$ If $A, B$ are events with $\mathbb P(B) > 0$, the conditional probability of $A$ given $B$ is defined as

$$\mathbb{P}(A \mid B):=\frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)} $$

Note that $\mathbb P(B \mid B) = 1$

$\textbf{Theorem 2.8.}\quad$(Law of Total Probability)
If $B_n$, $n \in I$ is a partition of $E$ then for $A \in \mathcal E$ we have

$$\mathbb{P}(A)=\sum_{i \in I} \mathbb{P}\left(A \mid B_{n}\right) \mathbb{P}\left(B_{n}\right) $$

Definition of Independence

Two events $A, B$ are independent $\iff \mathbb P(A \cap B) = P(A) P(B)$, equivalently $P(A \mid B) = P(A)$ when $P(B) > 0$.

Let $(E, \mathcal E, \mathbb P)$ be a probability space. Assume that $E$ is countable. Thus there exists a partition $\Pi=\{E_i, i \in I\}$ generating $E$.

$\textbf{Definition 2.10.}\quad$ Given a $\sigma$-algebra $B \subseteq E$ and an event $A \in E$, the conditional probability of $A$ given $B$ is a $rv$ that is constant on each $E_i$ and

$$\forall \omega \in E_{i}:\mathbb{P}(A \mid \mathcal{B})(\omega):=\mathbb{P}\left(A \mid E_{i}\right) $$