Skip to main content

Important results

Covariance properties​

Cov(X,a)=0Cov(X,X)=Var(X)Cov(X,Y)=Cov(Y,X)Cov(aX,bY)=ab Cov(X,Y)Cov(X+a,Y+b)=Cov(X,Y)Cov(aX+bY,cW+dV)=ac Cov(X,W)+ad Cov(X,V)+bc Cov(Y,W)+bd Cov(Y,V)\begin{align*} Cov(X, a) &= 0\\ Cov(X, X) &= Var(X)\\ Cov(X, Y) &= Cov(Y, X)\\ Cov(aX, bY) &= ab \, Cov(X, Y)\\ Cov(X + a, Y + b) &= Cov(X, Y)\\ Cov(aX + bY, cW + dV) &= ac \, Cov(X, W) + ad \, Cov(X, V) + bc \, Cov(Y, W) + bd \, Cov(Y, V) \end{align*}

Mean independence and Covariance​

Prove: If XX and UU are mean independent and E[U]=0\mathbb{E}[U]=0 then Cov(X,U)=0Cov(X,U)=0.

Note: Mean independence is defined as E[U∣X]=E[U]\mathbb{E}[U∣X]=\mathbb{E}[U].

Proof:

Cov(X,U)=E[(X−E[X])(U−E[U])]=E[XU−XE[U]−E[X]U+E[X]E[U]]=E[XU]−E[X]E[U]−E[X]E[U]+E[X]E[U]=E[XU]−E[X]E[U]=E[XU]−E[X]E[E[U∣X]]=E[XU]−E[X]E[0]=E[XU]=E[E[XU∣X]]=E[XE[U∣X]]=E[X⋅0]=0\begin{align*} Cov(X, U) &= \mathbb{E}\left[(X - \mathbb{E}[X])(U - \mathbb{E}[U])\right] \\ &= \mathbb{E}[XU - X\mathbb{E}[U] - \mathbb{E}[X]U + \mathbb{E}[X]\mathbb{E}[U]] \\ &= \mathbb{E}[XU] - \mathbb{E}[X]\mathbb{E}[U] - \mathbb{E}[X]\mathbb{E}[U] + \mathbb{E}[X]\mathbb{E}[U] \\ &= \mathbb{E}[XU] - \mathbb{E}[X]\mathbb{E}[U] \\ &= \mathbb{E}[XU] - \mathbb{E}[X]\mathbb{E}[\mathbb{E}[U|X]] \\ &= \mathbb{E}[XU] - \mathbb{E}[X]\mathbb{E}[0] \\ &= \mathbb{E}[XU] \\ &= \mathbb{E}[\mathbb{E}[XU|X]] \\ &= \mathbb{E}[X\mathbb{E}[U|X]] \\ &= \mathbb{E}[X \cdot 0] \\ &= 0 \end{align*}

Unbiased estimator of Variance​

Let the population variance and mean are σ2\sigma^2 and μ\mu. The sample variance is given by:

S2=1n−1∑i=1n(Xi−Xˉ)2S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2

To prove the unbiasedness of S2S^2 we have to prove that E[S2]=σ2\mathbb{E}[S^2]=\sigma^2.

S2=1n−1∑i=1n(Xi2+Xˉ2−2XiXˉ)S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i^{2}+\bar{X}^2-2X_i\bar{X})

Taking expectation both sides:

E[S2]=E[1n−1∑i=1n(Xi2+Xˉ2−2XiXˉ)]=1n−1E[∑i=1n(Xi2)+∑i=1n(Xˉ2)−2Xˉ∑i=1n(Xi)]=1n−1E[∑i=1n(Xi2)+n(Xˉ2)−2Xˉn(Xˉ)]=1n−1E[∑i=1n(Xi2)−n(Xˉ2)]=1n−1[∑i=1nE[Xi2]−nE[Xˉ2]] as E operator is linear.\begin{align*} \mathbb{E}[S^2]&=\mathbb{E}\bigg[\frac{1}{n-1}\sum_{i=1}^n(X_i^{2}+\bar{X}^2-2X_i\bar{X})\bigg]\\ &=\frac{1}{n-1}\mathbb{E}\bigg[\sum_{i=1}^n(X_i^{2})+\sum_{i=1}^n(\bar{X}^2)-2\bar{X}\sum_{i=1}^n(X_i)\bigg]\\ &=\frac{1}{n-1}\mathbb{E}\bigg[\sum_{i=1}^n(X_i^{2})+n(\bar{X}^2)-2\bar{X}n(\bar{X})\bigg]\\ &=\frac{1}{n-1}\mathbb{E}\bigg[\sum_{i=1}^n(X_i^{2})-n(\bar{X}^2)\bigg]\\ &=\frac{1}{n-1}\bigg[\sum_{i=1}^n\mathbb{E}[X_i^{2}]-n\mathbb{E}[\bar{X}^2]\bigg]\text{ as }\mathbb{E}\text{ operator is linear.} \end{align*}

We know that Var[Xi]=E[Xi2]−(E[Xi])2Var[X_i]=\mathbb{E}[X_i^2] - (\mathbb{E}[X_i])^2, that is σ2=E[Xi2]−μ2\sigma^2=\mathbb{E}[X_i^2] - \mu^2, hence

1n−1[∑i=1nE[Xi2]−nE[Xˉ2]]=1n−1[∑i=1n(σ2+μ2)−nE[Xˉ2]]=1n−1[n(σ2+μ2)−nE[Xˉ2]]=nn−1[(σ2+μ2)−E[Xˉ2]]\begin{align*} \frac{1}{n-1}\bigg[\sum_{i=1}^n\mathbb{E}[X_i^{2}]-n\mathbb{E}[\bar{X}^2]\bigg]&=\frac{1}{n-1}\bigg[\sum_{i=1}^n(\sigma^2 + \mu^2)-n\mathbb{E}[\bar{X}^2]\bigg]\\ &=\frac{1}{n-1}\bigg[n(\sigma^2 + \mu^2) - n\mathbb{E}[\bar{X}^2]\bigg]\\ &=\frac{n}{n-1}\bigg[(\sigma^2 + \mu^2) - \mathbb{E}[\bar{X}^2]\bigg] \end{align*}

Also, Var[Xˉ]=E[Xˉ2]−(E[Xˉ])2Var[\bar{X}]=\mathbb{E}[\bar{X}^2]-(\mathbb{E}[\bar{X}])^2,   ⟹  E[Xˉ2]=Var[Xˉ]+(E[Xˉ])2\implies \mathbb{E}[\bar{X}^2]=Var[\bar{X}]+(\mathbb{E}[\bar{X}])^2.

Now

  • Consider Var[Xˉ]Var[\bar{X}],

    Note: Xˉ\bar{X} is a random variable

Var[Xˉ]=Var[∑i=1nXin]=1n2Var[∑i=1nXi]=1n2∑i=1nVar[Xi]( as Xi’s are i.i.d)=nσ2n2=σ2n\begin{align*} Var[\bar{X}]&=Var\bigg[\frac{\sum_{i=1}^nX_i}{n}\bigg]\\ &=\frac{1}{n^2}Var[\sum_{i=1}^nX_i]\\ &=\frac{1}{n^2}\sum_{i=1}^nVar[X_i]\\ &\text{( as }X_i \text{'s are i.i.d)}\\ &=\frac{n\sigma^2}{n^2}=\frac{\sigma^2}{n} \end{align*}
  • Consider (E[Xˉ])2(\mathbb{E}[\bar{X}])^2,
E[Xˉ]=E[∑i=1nXin]=1nE[∑i=1nXi]=1n∑i=1nE[Xi]=1n∑i=1nE[Xi]=μ(E[Xˉ])2=μ2\begin{align*} \mathbb{E}[\bar{X}]&=\mathbb{E}\bigg[\frac{\sum_{i=1}^n X_i}{n}\bigg]\\ &=\frac{1}{n}\mathbb{E}[\sum_{i=1}^nX_i]\\ &=\frac{1}{n}\sum_{i=1}^n \mathbb{E}[X_i]\\ &=\frac{1}{n}\sum_{i=1}^n \mathbb{E}[X_i]\\ &=\mu\\ (\mathbb{E}[\bar{X}])^2&=\mu^2 \end{align*}

Finally, we have

nn−1[(σ2+μ2)−E[Xˉ2]]=nn−1[(σ2+μ2)−Var[Xˉ]−(E[Xˉ])2]=nn−1[(σ2+μ2)−σ2n−μ2]=nn−1[(n−1)σ2n]E[S2]=σ2■\begin{align*} \frac{n}{n-1}\bigg[(\sigma^2 + \mu^2) - \mathbb{E}[\bar{X}^2]\bigg]&=\frac{n}{n-1}\bigg[(\sigma^2 + \mu^2) - Var[\bar{X}]-(\mathbb{E}[\bar{X}])^2\bigg]\\ &=\frac{n}{n-1}\bigg[(\sigma^2 + \mu^2) - \frac{\sigma^2}{n}-\mu^2\bigg]\\ &=\frac{n}{n-1}\bigg[\frac{(n-1)\sigma^2}{n}\bigg]\\ \mathbb{E}[S^2]&=\sigma^2 \qquad \qquad \blacksquare \end{align*}

Exercise: Find E[S2]\mathbb{E}[S^2], where SS is given by:

1n∑i=1n(Xi−Xˉ)2\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2

Q is positive definite​

Prove:

If XN×k\bold{X}_{N\times k} has full column rank, then Q:=1N(X′X)\bold{Q:=}\frac{1}{N}\bold{(X'X)} is a positive definite matrix.

A matrix A\bold{A} is positive definite if for all non-zero vector vk×1v_{k\times 1}, the quadratic form v′Av\bold{v′Av} is positive.

Proof:

We have to show

v′(X′X)Nv>0.\bold{v'}\frac{\bold{(X'X)}}{N}\bold{v}>0.

Above inequality can be written as

1Nv′X′Xv>01N(Xv)′Xv>0\begin{align*} \frac{1}{N}\bold{v'X'Xv}&>0\\ \frac{1}{N}\bold{(Xv)'Xv}&>0\\ \end{align*}

Since (Xv)\bold{(Xv)} is vector of dimension N×1N \times 1, (Xv)′Xv\bold{(Xv)'Xv} is the inner product of (Xv)\bold{(Xv)} with itself, which is always non-negative (≥)(\geq) by the definition of inner product.

If X\bold{X} has full column rank, then the equation Xv=0\bold{Xv}=\bold{0} only holds true if v=0\bold{v=0}. Therefore Xv≠0\bold{Xv}\neq\bold{0} for v≠0\bold{v\neq0}.

Hence

1Nv′(X′X)v>0.■\begin{align*} \frac{1}{N}\bold{v'(X'X)v}&>0.\hspace{20px}\blacksquare\\ \end{align*}

Law of Total Probability​