Skip to main content

Law of Iterated Expectations

Discrete case​

Under discrete case, Expectation is defined as follows

E[X]=x1p1+x2p2+...+xnpn=∑i=1nxipi,\mathbb{E}[X]=x_1p_1+x_2p_2+...+x_np_n=\sum_{i=1}^n x_ip_i,

can also be written as

E[X]=∑i=1n(X=xi)P(X=xi),\mathbb{E}[X]=\sum_{i=1}^n (X=x_i)P(X=x_i),

where random variable XX can take values {x1,x2,...,xn}\{x_1,x_2,...,x_n\} with probabilities {p1,p2,...,pn}\{p_1,p_2,...,p_n\} respectively.

The Law of Iterated Expectations (LIE) states that:

E[X]=E[E[X∣Y]]\mathbb{E}[X]=\mathbb{E}[\mathbb{E}[X|Y]]

that is, the expected value of the conditional expected value of XX given YY is the same as the expected value of XX.

Proof:

E[E[X∣Y]]=∑yiE[X∣Y=yi]P(Y=yi)=∑yi∑xixiP(X=xi∣Y=yi)P(Y=yi)\begin{align*} \mathbb{E}[\mathbb{E}[X|Y]]&=\sum_{y_i}\mathbb{E}[X|Y=y_i]P(Y=y_i)\\ &=\sum_{y_i}\sum_{x_i}x_iP(X=x_i|Y=y_i)P(Y=y_i) \end{align*}

according to the Bayes' theorem

P(X=xi∣Y=yi)=P(X=xi and Y=yi)P(Y=yi), andP(Y=yi∣X=xi)=P(X=xi and Y=yi)P(X=xi).  ⟹  P(X=xi∣Y=yi)P(Y=yi)=P(Y=yi∣X=xi)P(X=xi).\begin{align*} P(X=x_i|Y=y_i)&=\frac{P(X=x_i\text{ and }Y=y_i)}{P(Y=y_i)}, \text{ and}\\ P(Y=y_i|X=x_i)&=\frac{P(X=x_i\text{ and }Y=y_i)}{P(X=x_i)}.\\ \implies P(X=x_i|Y=y_i) P(Y=y_i)&=P(Y=y_i|X=x_i)P(X=x_i). \end{align*}

Therefore

E[E[X∣Y]]=∑yi∑xixiP(X=xi∣Y=yi)P(Y=yi)=∑yi∑xixiP(Y=yi∣X=xi)P(X=xi)=∑yi∑xixiP(X=xi)P(Y=yi∣X=xi)=∑xixiP(X=xi)∑yiP(Y=yi∣X=xi)⏟=1=∑xixiP(X=xi)=E[X]■\begin{align*} \mathbb{E}[\mathbb{E}[X|Y]]&=\sum_{y_i}\sum_{x_i}x_iP(X=x_i|Y=y_i)P(Y=y_i)\\ &=\sum_{y_i}\sum_{x_i}x_iP(Y=y_i|X=x_i)P(X=x_i)\\ &=\sum_{y_i}\sum_{x_i}x_iP(X=x_i)P(Y=y_i|X=x_i)\\ &=\sum_{x_i}x_iP(X=x_i)\underbrace{\sum_{y_i}P(Y=y_i|X=x_i)}_{=1}\\ &=\sum_{x_i}x_iP(X=x_i)\\ &=\mathbb{E}[X] \hspace{15px}\blacksquare \end{align*}