In statistics, M-estimators are a broad class of extremum estimators for which the objective function is a sample average. Both non-linear least squares and maximum likelihood estimation are special cases of M-estimators. The definition of M-estimators was motivated by robust statistics, which contributed new types of M-estimators. However, M-estimators are not inherently robust, as is clear from the fact that they include maximum likelihood estimators, which are in general not robust. The statistical procedure of evaluating an M-estimator on a data set is called M-estimation. The "M" initial stands for "maximum likelihood-type". More generally, an M-estimator may be defined to be a zero of an estimating function. This estimating function is often the derivative of another statistical function. For example, a maximum-likelihood estimate is the point where the derivative of the likelihood function with respect to the parameter is zero; thus, a maximum-likelihood estimator is a critical point of the score function. In many applications, such M-estimators can be thought of as estimating characteristics of the population.
Historical motivation Although the main concepts of robust statistics have been formally developed only in recent decades, precursors of robust M-estimators can be traced back to the early history of statistics. Galileo Galilei (1632) was among the first to argue that measurement errors required systematic treatment. Later, Roger Joseph Boscovich (1757) proposed an estimator based on absolute deviations, Daniel Bernoulli (1785) suggested iterative reweighting schemes, and Simon Newcomb (1886) experimented with mixtures of distributions for regression. By the late 19th century, Smith (1888) introduced what is now recognized as the first robust M-estimator, already resembling the modern formulation. A recent review by De Menezes (2021) collected, organized, classified, and reported tuning constants for an extensive set of M-estimators, providing a systematic overview of their properties and applications. The method of least squares is a prototypical M-estimator, since the estimator is defined as a minimum of the sum of squares of the residuals. Another popular M-estimator is maximum-likelihood estimation. For a family of probability density functions f parameterized by θ, a maximum likelihood estimator of θ is computed for each set of data by maximizing the likelihood function over the parameter space { θ } . When the observations are independent and identically distributed, a ML-estimate θ ^ {\displaystyle {\hat {\theta }}} satisfies
θ ^ = arg max θ ( ∏ i = 1 n f ( x i , θ ) ) {\displaystyle {\widehat {\theta }}=\mathop {\arg \max } _{\theta }{\left(\prod _{i=1}^{n}f(x_{i},\theta )\right)}\,\!}
or, equivalently,
θ ^ = arg min θ ( ∑ i = 1 n − log f ( x i , θ ) ) . {\displaystyle {\widehat {\theta }}=\mathop {\arg \min } _{\theta }{\left(\sum _{i=1}^{n}-\log f(x_{i},\theta )\right)}.}
Maximum-likelihood estimators have optimal properties in the limit of infinitely many observations under rather general conditions, but may be biased and not the most efficient estimators for finite samples.
Definition In 1964, Peter J. Huber proposed generalizing maximum likelihood estimation to the minimization of
∑ i = 1 n ρ ( x i , θ ) , {\displaystyle \sum _{i=1}^{n}\rho (x_{i},\theta ),\,\!}
where ρ is a function with certain properties (see below). The solutions
θ ^ = arg min θ ∑ i = 1 n ρ ( x i , θ ) {\displaystyle {\hat {\theta }}=\mathop {\arg \min } _{\theta }\sum _{i=1}^{n}\rho (x_{i},\theta )}
… excerpt ends here. Continue reading the full article.
