In statistics, sufficiency is a property of a statistic computed on a sample dataset in relation to a parametric model of the dataset. A sufficient statistic for a model parameter contains all of the information that the dataset provides about that parameter. It is closely related to the concepts of an ancillary statistic which contains no information about the model parameters, and of a complete statistic which only contains information about the parameters and no ancillary information. A related concept is that of linear sufficiency, which is weaker than sufficiency but can be applied in some cases where there is no sufficient statistic, although it is restricted to linear estimators. The Kolmogorov structure function deals with individual finite data; the related notion there is the algorithmic sufficient statistic. The concept is due to Sir Ronald Fisher in 1920. Stephen Stigler noted in 1973 that the concept of sufficiency had fallen out of favor in descriptive statistics because of the strong dependence on an assumption of the distributional form (see Pitman–Koopman–Darmois theorem below), but remained very important in theoretical work.
Background Roughly, given a set X {\displaystyle \mathbf {X} } of independent identically distributed data conditioned on an unknown parameter θ {\displaystyle \theta } , a sufficient statistic is a function T ( X ) {\displaystyle T(\mathbf {X} )} whose value contains all the information needed to compute any estimate of the parameter (e.g. a maximum likelihood estimate). Due to the factorization theorem (see below), for a sufficient statistic T ( X ) {\displaystyle T(\mathbf {X} )} , the probability density can be written as f X ( x ; θ ) = h ( x ) g ( θ , T ( X ) ) {\displaystyle f_{\mathbf {X} }(x;\theta )=h(x)\,g(\theta ,T(\mathbf {X} ))} . From this factorization, it can easily be seen that the maximum likelihood estimate of θ {\displaystyle \theta } will interact with X {\displaystyle \mathbf {X} } only through T ( X ) {\displaystyle T(\mathbf {X} )} . Typically, the sufficient statistic is a simple function of the data, e.g. the sum of all the data points. More generally, the "unknown parameter" may represent a vector of unknown quantities or may represent everything about the model that is unknown or not fully specified. In such a case, the sufficient statistic may be a set of functions, called a jointly sufficient statistic. Typically, there are as many functions as there are parameters. For example, for a Gaussian distribution with unknown mean and variance, the jointly sufficient statistic, from which maximum likelihood estimates of both parameters can be estimated, consists of two functions, the sum of all data points and the sum of all squared data points (or equivalently, the sample mean and sample variance). In other words, given any value of the sufficient statistic for the parameter, the joint probability distribution of the data doesn't depend on the parameter. Both the statistic and the underlying parameter can be vectors.
Mathematical definition A statistic t = T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ. Alternatively, one can say the statistic T(X) is sufficient for θ if, for all prior distributions on θ, the mutual information between θ and T(X) equals the mutual information between θ and X. In other words, the data processing inequality becomes an equality:
I ( θ ; T ( X ) ) = I ( θ ; X ) {\displaystyle I{\bigl (}\theta ;T(X){\bigr )}=I(\theta ;X)}
Example As an example, the sample mean is sufficient for the (unknown) mean μ of a normal distribution with known variance. Once the sample mean is known, no further information about μ can be obtained from the sample itself. On the other hand, for an arbitrary distribution the median is not sufficient for the mean: even if the median of the sample is known, knowing the sample itself would provide further information about the population mean. For example, if the observations that are less than the median are only slightly less, but observations exceeding the median exceed it by a large amount, then this would have a bearing on one's inference about the population mean.
Fisher–Neyman factorization theorem Fisher's factorization theorem or factorization criterion provides a convenient characterization of a sufficient statistic. If the probability density function is ƒ(x;θ), where θ is a parameter, then T is sufficient for θ if and only if nonnegative functions g and h can be found such that
f ( x ; θ ) = h ( x ) g ( θ , T ( x ) ) , {\displaystyle f(x;\theta )=h(x)\,g(\theta ,T(x)),}
… excerpt ends here. Continue reading the full article.
