Review: Maximum Likelihood Estimation

by Kurious Fox

Subscribe to get access

??Subscribe to read the rest of the comics, the fun you can’t miss ??

Subscribe

Maximum Likelihood Estimation (MLE) is a statistical method used to estimate the parameters of a model by maximizing the likelihood function, which measures how well the model explains the observed data. Given a set of data points and a probabilistic model, MLE finds the parameter values that make the observed data most probable. It’s widely used in various fields due to its flexibility and asymptotic properties, where the estimate converges to the true parameter as the sample size increases.

Example: To estimate the Maximum Likelihood Estimator (MLE) for a Poisson distribution, let’s consider the Poisson probability mass function (PMF):

P(X = k | \lambda) = \frac{\lambda^k e^{-\lambda}}{k!}

where \lambda is the rate parameter (the average number of occurrences), and k is the number of events.

Given a sample of n independent observations X_1, X_2, \dots, X_n , the likelihood function is the product of the individual probabilities:

L(\lambda | X_1, X_2, \dots, X_n) = \prod_{i=1}^{n} \frac{\lambda^{X_i} e^{-\lambda}}{X_i!}

The log-likelihood function simplifies to:

\log L(\lambda) = \sum_{i=1}^{n} \left( X_i \log \lambda - \lambda \right) + C

where C is a constant that does not depend on \lambda .

To find the MLE, take the derivative of the log-likelihood with respect to \lambda and set it to zero:

\frac{d}{d\lambda} \log L(\lambda) = \sum_{i=1}^{n} \left( \frac{X_i}{\lambda} - 1 \right) = 0

Solving for \lambda , we get:

\hat{\lambda} = \frac{1}{n} \sum_{i=1}^{n} X_i

Thus, the MLE for \lambda in a Poisson distribution is the sample mean \hat{\lambda} = \bar{X} , which is the average number of events observed in the sample.

To confirm that the value of \hat{\lambda} = \bar{X} is indeed the maximum likelihood estimate for the Poisson distribution, we can use the second derivative test.

Step 1: First Derivative
We already computed the first derivative of the log-likelihood function \log L(\lambda) :

\frac{d}{d\lambda} \log L(\lambda) = \sum_{i=1}^{n} \left( \frac{X_i}{\lambda} - 1 \right)

Setting this to zero and solving for \lambda , we obtained the MLE:

\hat{\lambda} = \frac{1}{n} \sum_{i=1}^{n} X_i = \bar{X}

Step 2: Second Derivative
Now, to apply the second derivative test, we need to compute the second derivative of the log-likelihood function with respect to \lambda :

\frac{d^2}{d\lambda^2} \log L(\lambda) = \frac{d}{d\lambda} \left( \sum_{i=1}^{n} \left( \frac{X_i}{\lambda} - 1 \right) \right) = \sum_{i=1}^{n} \left( -\frac{X_i}{\lambda^2} \right)

This simplifies to:

\frac{d^2}{d\lambda^2} \log L(\lambda) = -\frac{1}{\lambda^2} \sum_{i=1}^{n} X_i = -\frac{n \bar{X}}{\lambda^2}

Step 3: Test for Concavity
For \hat{\lambda} = \bar{X} to be a maximum, the second derivative must be negative at \lambda = \hat{\lambda} . Substituting \hat{\lambda} = \bar{X} into the second derivative:

\frac{d^2}{d\lambda^2} \log L(\lambda) \Big|_{\lambda = \hat{\lambda}} = -\frac{n \bar{X}}{\bar{X}^2} = -\frac{n}{\bar{X}}

Since \bar{X} > 0 and n > 0 , the second derivative is negative. This confirms that the log-likelihood function is concave at \hat{\lambda} , and thus \hat{\lambda} = \bar{X} is indeed the MLE for the Poisson distribution.

Discover more from Science Comics

Subscribe to get the latest posts sent to your email.

Related Posts

Leave a Reply

error: Content is protected !!