Review: Maximum Likelihood Estimation

by Kurious Fox

Subscribe to get access

??Subscribe to read the rest of the comics, the fun you can’t miss ??

Subscribe

Maximum Likelihood Estimation (MLE) is a statistical method used to estimate the parameters of a model by maximizing the likelihood function, which measures how well the model explains the observed data. Given a set of data points and a probabilistic model, MLE finds the parameter values that make the observed data most probable. It’s widely used in various fields due to its flexibility and asymptotic properties, where the estimate converges to the true parameter as the sample size increases.

Example: To estimate the Maximum Likelihood Estimator (MLE) for a Poisson distribution, let’s consider the Poisson probability mass function (PMF):

P(X = k | \lambda) = \frac{\lambda^k e^{-\lambda}}{k!}

where \lambda is the rate parameter (the average number of occurrences), and k is the number of events.

Given a sample of n independent observations X_1, X_2, \dots, X_n , the likelihood function is the product of the individual probabilities:

L(\lambda | X_1, X_2, \dots, X_n) = \prod_{i=1}^{n} \frac{\lambda^{X_i} e^{-\lambda}}{X_i!}

The log-likelihood function simplifies to:

\log L(\lambda) = \sum_{i=1}^{n} \left( X_i \log \lambda - \lambda \right) + C

where C is a constant that does not depend on \lambda .

To find the MLE, take the derivative of the log-likelihood with respect to \lambda and set it to zero:

\frac{d}{d\lambda} \log L(\lambda) = \sum_{i=1}^{n} \left( \frac{X_i}{\lambda} - 1 \right) = 0

Solving for \lambda , we get:

\hat{\lambda} = \frac{1}{n} \sum_{i=1}^{n} X_i

Thus, the MLE for \lambda in a Poisson distribution is the sample mean \hat{\lambda} = \bar{X} , which is the average number of events observed in the sample.

To confirm that the value of \hat{\lambda} = \bar{X} is indeed the maximum likelihood estimate for the Poisson distribution, we can use the second derivative test.

Step 1: First Derivative
We already computed the first derivative of the log-likelihood function \log L(\lambda) :

\frac{d}{d\lambda} \log L(\lambda) = \sum_{i=1}^{n} \left( \frac{X_i}{\lambda} - 1 \right)

Setting this to zero and solving for \lambda , we obtained the MLE:

\hat{\lambda} = \frac{1}{n} \sum_{i=1}^{n} X_i = \bar{X}

Step 2: Second Derivative
Now, to apply the second derivative test, we need to compute the second derivative of the log-likelihood function with respect to \lambda :

\frac{d^2}{d\lambda^2} \log L(\lambda) = \frac{d}{d\lambda} \left( \sum_{i=1}^{n} \left( \frac{X_i}{\lambda} - 1 \right) \right) = \sum_{i=1}^{n} \left( -\frac{X_i}{\lambda^2} \right)

This simplifies to:

\frac{d^2}{d\lambda^2} \log L(\lambda) = -\frac{1}{\lambda^2} \sum_{i=1}^{n} X_i = -\frac{n \bar{X}}{\lambda^2}

Step 3: Test for Concavity
For \hat{\lambda} = \bar{X} to be a maximum, the second derivative must be negative at \lambda = \hat{\lambda} . Substituting \hat{\lambda} = \bar{X} into the second derivative:

\frac{d^2}{d\lambda^2} \log L(\lambda) \Big|_{\lambda = \hat{\lambda}} = -\frac{n \bar{X}}{\bar{X}^2} = -\frac{n}{\bar{X}}

Since \bar{X} > 0 and n > 0 , the second derivative is negative. This confirms that the log-likelihood function is concave at \hat{\lambda} , and thus \hat{\lambda} = \bar{X} is indeed the MLE for the Poisson distribution.

Discover more from Science Comics

Subscribe to get the latest posts sent to your email.

Related Posts

A comic guide to underfitting

Underfitting in machine learning occurs when a model fails to capture underlying data patterns due to simplicity or insufficient training data. To address underfitting, select complex models, add features, and obtain more training data. Also, fine-tune hyperparameters and optimize the model’s architecture. Few features in a model can also cause underfitting, requiring the identification of relevant additional features or more advanced modeling techniques.

Simple Linear Regression & Least square method

Simple linear regression is a statistical method to model the relationship between two continuous variables, aiming to predict the dependent variable based on the independent variable. The regression equation is Y = a + bX, where Y is the dependent variable, X is the independent variable, a is the intercept, and b is the slope. The method of least squares minimizes the sum of squared residuals to find the best-fitting line coefficients.

Leave a Reply

error: Content is protected !!