In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference. If the likelihood function is differentiable, the derivative test for finding maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved analytically; for instance, the ordinary least squares estimator for a linear regression model maximizes the likelihood when the random errors are assumed to have normal distributions with the same variance. From the perspective of Bayesian inference, MLE is generally equivalent to maximum a posteriori (MAP) estimation with a prior distribution that is uniform in the region of interest. In frequentist inference, MLE is a special case of an extremum estimator, with the objective function being the likelihood.
Principles We model a set of observations as a random sample from an unknown joint probability distribution which is expressed in terms of a set of parameters. The goal of maximum likelihood estimation is to determine the parameters for which the observed data have the highest joint probability. We write the parameters governing the joint distribution as a vector θ = [ θ 1 , θ 2 , … , θ k ] T {\displaystyle \;\theta =\left[\theta _{1},\,\theta _{2},\,\ldots ,\,\theta _{k}\right]^{\mathsf {T}}\;} so that this distribution falls within a parametric family { f ( ⋅ ; θ ) ∣ θ ∈ Θ } , {\displaystyle \;\{f(\cdot \,;\theta )\mid \theta \in \Theta \}\;,} where Θ {\displaystyle \,\Theta \,} is called the parameter space, a finite-dimensional subset of Euclidean space. Evaluating the joint density at the observed data sample y = ( y 1 , y 2 , … , y n ) {\displaystyle \;\mathbf {y} =(y_{1},y_{2},\ldots ,y_{n})\;} gives a real-valued function,
L n ( θ ) = L n ( θ ; y ) = f n ( y ; θ ) , {\displaystyle {\mathcal {L}}_{n}(\theta )={\mathcal {L}}_{n}(\theta ;\mathbf {y} )=f_{n}(\mathbf {y} ;\theta )\;,}
which is called the likelihood function. For independent random variables, f n ( y ; θ ) {\displaystyle f_{n}(\mathbf {y} ;\theta )} will be the product of univariate density functions:
f n ( y ; θ ) = ∏ k = 1 n f k u n i v a r ( y k ; θ ) . {\displaystyle f_{n}(\mathbf {y} ;\theta )=\prod _{k=1}^{n}\,f_{k}^{\mathsf {univar}}(y_{k};\theta )~.}
The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space, that is:
θ ^ = a r g m a x θ ∈ Θ L n ( θ ; y ) . {\displaystyle {\hat {\theta }}={\underset {\theta \in \Theta }{\operatorname {arg\;max} }}\,{\mathcal {L}}_{n}(\theta \,;\mathbf {y} )~.}
… excerpt ends here. Continue reading the full article.
