Maximum Likelihood Estimation

GitHub repository

In this Jupyter notebook, we will do the maximum likelihood estimation of the parameters, mean and standard deviation, of the normal distribution.

Likelihood Function

PDF of normal distribution is given as follows:

$$f_X(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\cdot e^{-\frac{1}{2} (\frac{x-\mu}{\sigma})^2},$$

where $\mu$ is mean and $\sigma$ is standard deviation.

Assuming we have a dataset comprising $n$ randomly selected observations from a normal distribution with a mean of 155 and a standard deviation of 7, independently and identically distributed (IID). The likelihood function corresponding to these $n$ observations is given as follows:

$$\begin{align*} L(\mu,\sigma, x_1, x_2,\cdots,x_{n})&=\prod_{i=1}^{n}f_X(x_i)\\ &=\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi\sigma^2}}\cdot e^{-\frac{1}{2} (\frac{x_i-\mu}{\sigma})^2}\\ &=\Bigg(\frac{1}{\sqrt{2\pi\sigma^2}}\Bigg)^n\cdot e^{-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i-\mu)^2}.\\ \end{align*}$$

Log Likelihood Function

Taking the log of Likelihood Function, we get

$$\begin{align*} l(\mu,\sigma, x_1, x_2,\cdots,x_{n})&=\log{[L(\mu,\sigma, x_1, x_2,\cdots,x_{n})]}\\ &=\log{\Bigg[\Big(\frac{1}{\sqrt{2\pi\sigma^2}}\Big)^n\cdot e^{-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i-\mu)^2}\Bigg]}\\ &=\log{\Bigg[\Big(\frac{1}{\sqrt{2\pi\sigma^2}}\Big)^n\Bigg]}\cdot \log{\Bigg[e^{-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i-\mu)^2}\Bigg]}\\ &=-n\log{(\sqrt{2\pi\sigma^2})}-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i-\mu)^2\\ \end{align*}$$

Maximization Problem

Now we have solve the following maximization problem

$$\argmax_{\mu,\sigma}\{l(\mu,\sigma, x_1, x_2,\cdots,x_{n})\}.$$

The above maximization problem is equivalent to

$$\argmin_{\mu,\sigma}\{-l(\mu,\sigma, x_1, x_2,\cdots,x_{n})\},$$

therefore we will solve

$$\begin{align*} &\argmin_{\mu,\sigma}\{-(-n\log{(\sqrt{2\pi\sigma^2})}-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i-\mu)^2)\}\\ \implies &\argmin_{\mu,\sigma}\{n\log{(\sqrt{2\pi\sigma^2})}+\frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i-\mu)^2\}.\\ \end{align*}$$

Python Code

import numpy as np
from scipy.optimize import minimize

# Generate some example data
np.random.seed(2609)
data = np.random.normal(loc=155, scale=7, size=100000)

# Define the likelihood function for a normal distribution
def likelihood(params, data):
    mean, std_dev = params
    likelihood_values = np.log(np.sqrt(2*np.pi*(std_dev**2))) + (0.5*(((data - mean) / std_dev)**2))
    return np.sum(likelihood_values)

# Initial guess for the parameters
initial_params = [0, 1]

# Use scipy's minimize function to maximize the likelihood
result = minimize(likelihood, initial_params, args=(data,))
estimated_mean, estimated_std_dev = result.x

# Print the results
print("Estimated Mean:", estimated_mean)
print("Estimated Standard Deviation:", estimated_std_dev)
print("Convergence Status:", result.success)
print("Optimal Value of the Objective Function:", result.fun)

Estimated Mean: 154.99808505357024
Estimated Standard Deviation: 6.996996924429178
Convergence Status: True
Optimal Value of the Objective Function: 336441.9597128625