Deriving the OLS Estimates
Here we will derive OLS estimates for both simple and multiple linear regression.
Simple linear regression
We will use the Method of Moments to derive.
Derivation
Our objective is to estimate the following model:
y=β0+β1x+u
Since there are two unknowns (β0 and β1), we need two equations. And we will use the following two equations (conditions):
- E[U]=0, and
- E[U∣X]=Cov(X,U)=E[XU]=0
Above conditions can be written as:
E[y−β0−β1x]E[x(y−β0−β1x)]=0,=0
Taking sample counterparts of the above equation:
n−1i=1∑n(yi−β^0−β^1xi)n−1i=1∑nxi(yi−β^0−β^1xi)=0,=0(1)(2)
Solving (1), we get
yˉ⟹β^0=β^0+β^1xˉ=yˉ−β^1xˉ(3)
where yˉ and xˉ are sample means of y and x.
Substituting (3) in (2), we get
n−1i=1∑nxi(yi−yˉ+β^1xˉ−β^1xi)=0
n−1 will go away because R.H.S is zero. Rearranging the above equation:
i=1∑nxi(yi−yˉ)i=1∑nxi(yi−yˉ)−i=1∑nxiβ^1(xi−xˉ)=0=β^1i=1∑nxi(xi−xˉ).(4)
We know that:
i=1∑nxi(xi−xˉ)=i=1∑n(xi−xˉ)2 and i=1∑nxi(yi−yˉ)=i=1∑n(xi−xˉ)(yi−yˉ).
[How?]
Using the above equalities, (4) can be written as:
β^1=∑i=1n(xi−xˉ)2∑i=1n(xi−xˉ)(yi−yˉ),
provided that
i=1∑n(xi−xˉ)2>0.
We can also find β^0 by substitution β^1