In mathematical statistics, the Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ of a distribution that models X. Formally, it is the variance of the score, or the expected value of the observed information. The role of the Fisher information in the asymptotic theory of maximum-likelihood estimation was emphasized and explored by the statistician Sir Ronald Fisher (following some initial results by Francis Ysidro Edgeworth). The Fisher information matrix is used to calculate the covariance matrices associated with maximum-likelihood estimates. It can also be used in the formulation of test statistics, such as the Wald test. In Bayesian statistics, the Fisher information plays a role in the derivation of non-informative prior distributions according to Jeffreys' rule. It also appears as the large-sample covariance of the posterior distribution, provided that the prior is sufficiently smooth (a result known as Bernstein–von Mises theorem, which was anticipated by Laplace for exponential families). The same result is used when approximating the posterior with Laplace's approximation, where the Fisher information appears as the covariance of the fitted Gaussian. Statistical systems of a scientific nature (physical, biological, etc.) whose likelihood functions obey shift invariance have been shown to obey maximum Fisher information. The level of the maximum depends upon the nature of the system constraints.
Definition The Fisher information is a way of measuring the amount of information that an observable random variable X {\displaystyle X} carries about an unknown parameter θ {\displaystyle \theta } upon which the probability of X {\displaystyle X} depends. Let f ( X ; θ ) {\displaystyle f(X;\theta )} be the probability density function (or probability mass function) for X {\displaystyle X} conditioned on the value of θ {\displaystyle \theta } . It describes the probability that we observe a given outcome of X {\displaystyle X} , given a known value of θ {\displaystyle \theta } . If f {\displaystyle f} is sharply peaked with respect to changes in θ {\displaystyle \theta } , it is easy to indicate the "correct" value of θ {\displaystyle \theta } from the data, or equivalently, that the data X {\displaystyle X} provides a lot of information about the parameter θ {\displaystyle \theta } . If f {\displaystyle f} is flat and spread-out, then it would take many samples of X {\displaystyle X} to estimate the actual "true" value of θ {\displaystyle \theta } that would be obtained using the entire population being sampled. This suggests studying some kind of variance with respect to θ {\displaystyle \theta } . Formally, the partial derivative with respect to θ {\displaystyle \theta } of the natural logarithm of the likelihood function is called the score. Under certain regularity conditions, if θ {\displaystyle \theta } is the true parameter (i.e. X {\displaystyle X} is actually distributed as f ( X ; θ ) {\displaystyle f(X;\theta )} ), it can be shown that the expected value (the first moment) of the score, evaluated at the true parameter value θ {\displaystyle \theta } , is 0:
E [ ∂ ∂ θ log f ( X ; θ ) | θ ] =
∫ R ∂ ∂ θ f ( x ; θ ) f ( x ; θ ) f ( x ; θ ) d x =
∂ ∂ θ ∫ R f ( x ; θ ) d x =
∂ ∂ θ 1 =
0. {\displaystyle {\begin{aligned}\operatorname {E} \left[\left.{\frac {\partial }{\partial \theta }}\log f(X;\theta )\,\,\right|\,\,\theta \right]={}&\int _{\mathbb {R} }{\frac {{\frac {\partial }{\partial \theta }}f(x;\theta )}{f(x;\theta )}}f(x;\theta )\,dx\\[6pt]={}&{\frac {\partial }{\partial \theta }}\int _{\mathbb {R} }f(x;\theta )\,dx\\[6pt]={}&{\frac {\partial }{\partial \theta }}1\\[6pt]={}&0.\end{aligned}}}
The Fisher information is defined to be the variance of the score:
I ( θ ) = E [ ( ∂ ∂ θ log f ( X ; θ ) ) 2 | θ ] = ∫ R ( ∂ ∂ θ log f ( x ; θ ) ) 2 f ( x ; θ ) d x , {\displaystyle {\mathcal {I}}(\theta )=\operatorname {E} \left[\left.\left({\frac {\partial }{\partial \theta }}\log f(X;\theta )\right)^{2}\,\,\right|\,\,\theta \right]=\int _{\mathbb {R} }\left({\frac {\partial }{\partial \theta }}\log f(x;\theta )\right)^{2}f(x;\theta )\,dx,}
Note that I ( θ ) ≥ 0 {\displaystyle {\mathcal {I}}(\theta )\geq 0} . A random variable carrying high Fisher information implies that the absolute value of the score is often high. The Fisher information is not a function of a particular observation, as the random variable X has been averaged out. If log f(x; θ) is twice differentiable with respect to θ, and under certain additional regularity conditions, then the Fisher information may also be written as
I ( θ ) = − E [ ∂ 2 ∂ θ 2 log f ( X ; θ ) | θ ] , {\displaystyle {\mathcal {I}}(\theta )=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\log f(X;\theta )\,\,\right|\,\,\theta \right],}
Thus, the Fisher information may be seen as the curvature of the support curve (the graph of the log-likelihood). Near the maximum likelihood estimate, low Fisher information indicates that the maximum appears to be "blunt", that is, there are many points in the neighborhood that provide a similar log-likelihood. Conversely, a high Fisher information indicates that the maximum is "sharp".
Regularity conditions The regularity conditions are as follows:
The partial derivative of f(X; θ) with respect to θ exists almost everywhere. (It can fail to exist on a null set, as long as this set does not depend on θ.) The integral of f(X; θ) can be differentiated under the integral sign with respect to θ. The support of f(X; θ) does not depend on θ. If θ is a vector then the regularity conditions must hold for every component of θ. It is easy to find an example of a density that does not satisfy the regularity conditions: The density of a Uniform(0, θ) variable fails to satisfy conditions 1 and 3. In this case, even though the Fisher information can be computed from the definition, it will not have the properties it is typically assumed to have.
In terms of likelihood Because the likelihood of θ given X is always proportional to the probability f(X; θ), their logarithms necessarily differ by a constant that is independent of θ, and the derivatives of these logarithms with respect to θ are necessarily equal. Thus one can substitute in a log-likelihood l(θ; X) instead of log f(X; θ) in the definitions of Fisher Information.
Samples of any size The value X can represent a single sample drawn from a single distribution or can represent a collection of samples drawn from a collection of distributions. If there are n samples and the corresponding n distributions are statistically independent then the Fisher information will necessarily be the sum of the single-sample Fisher information values, one for each single sample from its distribution. In particular, if the n distributions are independent and identically distributed then the Fisher information will necessarily be n times the Fisher information of a single sample from the common distribution. Stated in other words, the Fisher Information of i.i.d. observations of a sample of size n from a population is equal to the product of n and the Fisher Information of a single observation from the same population.
Informal derivation of the Cramér–Rao bound The Cramér–Rao bound states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of θ. Van Trees (1968) and Frieden (2004) provide the following method of deriving the Cramér–Rao bound, a result which describes use of the Fisher information. Informally, we begin by considering an unbiased estimator θ ^ ( X ) {\displaystyle {\hat {\theta }}(X)} . Mathematically, "unbiased" means that
E [ θ ^ ( X ) − θ | θ ] = ∫ ( θ ^ ( x ) − θ ) f ( x ; θ ) d x = 0 regardless of the value of θ . {\displaystyle \operatorname {E} \left[\left.{\hat {\theta }}(X)-\theta \,\,\right|\,\,\theta \right]=\int \left({\hat {\theta }}(x)-\theta \right)\,f(x;\theta )\,dx=0{\text{ regardless of the value of }}\theta .}
This expression is zero independent of θ, so its partial derivative with respect to θ must also be zero. By the product rule, this partial derivative is also equal to
0 = ∂ ∂ θ ∫ ( θ ^ ( x ) − θ ) f ( x ; θ ) d x = ∫ ( θ ^ ( x ) − θ ) ∂ f ∂ θ d x − ∫ f d x . {\displaystyle 0={\frac {\partial }{\partial \theta }}\int \left({\hat {\theta }}(x)-\theta \right)\,f(x;\theta )\,dx=\int \left({\hat {\theta }}(x)-\theta \right){\frac {\partial f}{\partial \theta }}\,dx-\int f\,dx.}
For each θ, the likelihood function is a probability density function, and therefore ∫ f d x = 1 {\displaystyle \int f\,dx=1} . By using the chain rule on the partial derivative of log f {\displaystyle \log f} and then dividing and multiplying by f ( x ; θ ) {\displaystyle f(x;\theta )} , one can verify that
∂ f ∂ θ = f ∂ log f ∂ θ . {\displaystyle {\frac {\partial f}{\partial \theta }}=f\,{\frac {\partial \log f}{\partial \theta }}.}
Using these two facts in the above, we get
∫ ( θ ^ − θ ) f ∂ log f ∂ θ d x = 1. {\displaystyle \int \left({\hat {\theta }}-\theta \right)f\,{\frac {\partial \log f}{\partial \theta }}\,dx=1.}
Factoring the integrand gives
∫ ( ( θ ^ − θ ) f ) ( f ∂ log f ∂ θ ) d x = 1. {\displaystyle \int \left(\left({\hat {\theta }}-\theta \right){\sqrt {f}}\right)\left({\sqrt {f}}\,{\frac {\partial \log f}{\partial \theta }}\right)\,dx=1.}
Squaring the expression in the integral, the Cauchy–Schwarz inequality yields
1 = ( ∫ [ ( θ ^ − θ ) f ] ⋅ [ f ∂ log f ∂ θ ] d x ) 2 ≤ [ ∫ ( θ ^ − θ ) 2 f d x ] ⋅ [ ∫ ( ∂ log f ∂ θ ) 2 f d x ] . {\displaystyle 1={\biggl (}\int \left[\left({\hat {\theta }}-\theta \right){\sqrt {f}}\right]\cdot \left[{\sqrt {f}}\,{\frac {\partial \log f}{\partial \theta }}\right]\,dx{\biggr )}^{2}\leq \left[\int \left({\hat {\theta }}-\theta \right)^{2}f\,dx\right]\cdot \left[\int \left({\frac {\partial \log f}{\partial \theta }}\right)^{2}f\,dx\right].}
The second bracketed factor is defined to be the Fisher Information, while the first bracketed factor is the mean-squared error (MSE) of the estimator θ ^ {\displaystyle {\hat {\theta }}} . Since the estimator is unbiased, its MSE equals its variance. By rearranging, the inequality tells us that
Var ( θ ^ ) ≥ 1 I ( θ ) . {\displaystyle \operatorname {Var} ({\hat {\theta }})\geq {\frac {1}{{\mathcal {I}}\left(\theta \right)}}.}
In other words, the precision to which we can estimate θ is fundamentally limited by the Fisher information of the likelihood function. Alternatively, the same conclusion can be obtained directly from the Cauchy–Schwarz inequality for random variables, | Cov ( A , B ) | 2 ≤ Var ( A ) Var ( B ) {\displaystyle |\operatorname {Cov} (A,B)|^{2}\leq \operatorname {Var} (A)\operatorname {Var} (B)} , applied to the random variables θ ^ ( X ) {\displaystyle {\hat {\theta }}(X)} and ∂ θ log f ( X ; θ ) {\displaystyle \partial _{\theta }\log f(X;\theta )} , and observing that for unbiased estimators we have Cov [ θ ^ ( X ) , ∂ θ log f ( X ; θ ) ] = ∫ θ ^ ( x ) ∂ θ f ( x ; θ ) d x = ∂ θ E [ θ ^ ] = 1. {\displaystyle \operatorname {Cov} [{\hat {\theta }}(X),\partial _{\theta }\log f(X;\theta )]=\int {\hat {\theta }}(x)\,\partial _{\theta }f(x;\theta )\,dx=\partial _{\theta }\operatorname {E} [{\hat {\theta }}]=1.}
Examples
Single-parameter Bernoulli experiment A Bernoulli trial is a random variable with two possible outcomes, 0 and 1, with 1 having a probability of θ. The outcome can be thought of as determined by the toss of a biased coin, with the probability of heads (1) being θ and the probability of tails (0) being 1 − θ. Let X be a Bernoulli trial of one sample from the distribution. The Fisher information contained in X may be calculated to be:
I ( θ ) = − E [ ∂ 2 ∂ θ 2 log ( θ X ( 1 − θ ) 1 − X ) | θ ] = − E [ ∂ 2 ∂ θ 2 ( X log θ + ( 1 − X ) log ( 1 − θ ) ) | θ ] = E [ X θ 2 + 1 − X ( 1 − θ ) 2 | θ ] = θ θ 2 + 1 − θ ( 1 − θ ) 2 = 1 θ ( 1 − θ ) . {\displaystyle {\begin{aligned}{\mathcal {I}}(\theta )&=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\log \left(\theta ^{X}(1-\theta )^{1-X}\right)\right|\theta \right]\\[5pt]&=-\operatorname {E} \left[\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\left(X\log \theta +(1-X)\log(1-\theta )\right)\,\,\right|\,\,\theta \right]\\[5pt]&=\operatorname {E} \left[\left.{\frac {X}{\theta ^{2}}}+{\frac {1-X}{(1-\theta )^{2}}}\,\,\right|\,\,\theta \right]\\[5pt]&={\frac {\theta }{\theta ^{2}}}+{\frac {1-\theta }{(1-\theta )^{2}}}\\[5pt]&={\frac {1}{\theta (1-\theta )}}.\end{aligned}}}
Because Fisher information is additive, the Fisher information contained in n independent Bernoulli trials is therefore
I ( θ ) = n θ ( 1 − θ ) . {\displaystyle {\mathcal {I}}(\theta )={\frac {n}{\theta (1-\theta )}}.}
If x i {\displaystyle x_{i}} is one of the 2 n {\displaystyle 2^{n}} possible outcomes of n independent Bernoulli trials and x i j {\displaystyle x_{ij}} is the j th outcome of the i th trial, then the probability of x i {\displaystyle x_{i}} is given by
p ( x i , θ ) = ∏ j = 0 n θ x i j ( 1 − θ ) x i j {\displaystyle p(x_{i},\theta )=\prod _{j=0}^{n}\theta ^{x_{ij}}(1-\theta )^{x_{ij}}}
The sample mean of the i th trial is μ i = ( 1 / n ) ∑ j = 1 n x i j {\displaystyle \mu _{i}=(1/n)\sum _{j=1}^{n}x_{ij}} . The expected value of the sample mean (over the sampling distribution) is
E ( μ ) = ∑ x i μ i p ( x i , θ ) = θ , {\displaystyle E(\mu )=\sum _{x_{i}}\mu _{i}\,p(x_{i},\theta )=\theta ,}
where the sum is over all 2 n {\displaystyle 2^{n}} possible trial outcomes. The expected value of the square of the sample mean is
E ( μ 2 ) = ∑ x i μ i 2 p ( x i , θ ) = ( 1 + ( n − 1 ) θ ) θ n {\displaystyle E(\mu ^{2})=\sum _{x_{i}}\mu _{i}^{2}\,p(x_{i},\theta )={\frac {(1+(n-1)\theta )\theta }{n}}}
so the variance in the value of the mean is
E ( μ 2 ) − E ( μ ) 2 = θ ( 1 − θ ) n {\displaystyle E(\mu ^{2})-E(\mu )^{2}={\frac {\theta (1-\theta )}{n}}}
It is seen that the Fisher information is the reciprocal of the variance of the mean number of successes in n Bernoulli trials. This is generally true. In this case, the Cramér–Rao bound is an equality.
Estimate θ from X ~ Bern (√θ) As another toy example consider a random variable X {\displaystyle X} with possible outcomes 0 and 1, with probabilities p 0 = 1 − θ {\displaystyle p_{0}=1-{\sqrt {\theta }}} and p 1 = θ {\displaystyle p_{1}={\sqrt {\theta }}} , respectively, for some θ ∈ [ 0 , 1 ] {\displaystyle \theta \in [0,1]} . Our goal is estimating θ {\displaystyle \theta } from observations of X {\displaystyle X} . The Fisher information reads in this case I ( θ ) = E