This extension is required for this article. Load it here if you haven't yet.

Maximum Likelihood Estimation

In this article, we'll explore how to find the parameters of a probability distribution that best fit observed data using maximum likelihood estimation (MLE). We'll work with both normal distribution.

Setup values

• set unit value
• define gray color
• hide point p0

---

Normal Distribution

Let's start with a normal distribution. We have some observed data points and want to find the mean () and standard deviation () that maximize the likelihood of observing this data.

Define data and true parameters

The MLE parameters for a Gaussian distribution have a nice closed form formula - the estimated mean and std are the same as the mean and std of the observed data. This will be our reference.

Define and plot the observed points

• create data array
• create points from xs

The true parameters of our data are and . These represent the actual distribution that generated our data points.

Plot true distribution

• calculate true mean
• calculate true standard deviation
• set x coordinate
• calculate PDF at x
• plot point
• set stroke color

The gray curve shows the true probability density function (PDF) of our data.

Define initial parameters

Now let's start with arbitrary initial guesses for our parameters and see if we can recover the true values through optimization.

Set up the parameters to optimize

• set mean parameter
• set standard deviation parameter
• set x coordinate
• calculate PDF with parameters
• plot the PDF to be trained

---

The blue curve shows our initial guess - a standard normal distribution centered at 0 with σ = 1. This is quite different from the true distribution!

Optimize parameters

We'll use gradient descent to minimize the negative log-likelihood, which is equivalent to maximizing the likelihood.

Optimization related setup

• register mu and sigma as parameters
• calculate likelihoods for all data points
• take natural log of likelihoods
• sum log-likelihoods
• calculate negative log-likelihood loss
• set number of iterations

---

Now let's minimize the negative loss likelihood:

• minimize loss function

The optimization adjusts and to maximize the probability of observing our data. After 100 iterations, the parameters should converge close to the true values.

We can highlight the reference plot and the learned plot. If trained well, these plots will almost coincide.

(This strategy won't work for uniform distribution since the gradients will be all zero)