Lindley's paradox is a counterintuitive situation in statistics in which the Bayesian and frequentist approaches to a hypothesis testing problem give different results for certain choices of the prior distribution. The problem of the disagreement between the two approaches was discussed in Harold Jeffreys' 1939 textbook; it became known as Lindley's paradox after Dennis Lindley called the disagreement a paradox in a 1957 paper. Although referred to as a paradox, the differing results from the Bayesian and frequentist approaches can be explained as using them to answer fundamentally different questions, rather than actual disagreement between the two methods. Nevertheless, for a large class of priors the differences between the frequentist and Bayesian approach are caused by keeping the significance level fixed: as even Lindley recognized, "the theory does not justify the practice of keeping the significance level fixed" and even "some computations by Prof. Pearson in the discussion to that paper emphasized how the significance level would have to change with the sample size, if the losses and prior probabilities were kept fixed". In fact, if the critical value increases with the sample size suitably fast, then the disagreement between the frequentist and Bayesian approaches becomes negligible as the sample size increases. The paradox continues to be a source of active discussion.
Description of the paradox The result x {\displaystyle x} of some experiment has two possible explanations – hypotheses H 0 {\displaystyle H_{0}} and H 1 {\displaystyle H_{1}} – and some prior distribution π {\displaystyle \pi } representing uncertainty as to which hypothesis is more accurate before taking into account x {\displaystyle x} . Lindley's paradox occurs when
The result x {\displaystyle x} is "significant" by a frequentist test of H 0 , {\displaystyle H_{0},} indicating sufficient evidence to reject H 0 , {\displaystyle H_{0},} say, at the 5% level, and The posterior probability of H 0 {\displaystyle H_{0}} given x {\displaystyle x} is high, indicating strong evidence that H 0 {\displaystyle H_{0}} is in better agreement with x {\displaystyle x} than H 1 . {\displaystyle H_{1}.}
These results can occur at the same time when H 0 {\displaystyle H_{0}} is very specific, H 1 {\displaystyle H_{1}} more diffuse, and the prior distribution does not strongly favor one or the other, as seen below.
Numerical example The following numerical example illustrates Lindley's paradox. In a certain city 49,581 boys and 48,870 girls have been born over a certain time period. The observed proportion x {\displaystyle x} of male births is thus 49581/98451 ≈ 0.5036. We assume the fraction of male births is a binomial variable with parameter θ . {\displaystyle \theta .} We are interested in testing whether θ {\displaystyle \theta } is 0.5 or some other value. That is, our null hypothesis is H 0 : θ = 0.5 , {\displaystyle H_{0}:\theta =0.5,} and the alternative is H 1 : θ ≠ 0.5. {\displaystyle H_{1}:\theta \neq 0.5.}
Frequentist approach The frequentist approach to testing H 0 {\displaystyle H_{0}} is to compute a p-value, the probability of observing a fraction of boys at least as large as x {\displaystyle x} assuming H 0 {\displaystyle H_{0}} is true. Because the number of births is very large, we can use a normal approximation for the fraction of male births X ∼ N ( μ , σ 2 ) , {\displaystyle X\sim N(\mu ,\sigma ^{2}),} with μ = n p = n θ = 98 451 × 0.5 = 49 225.5 {\displaystyle \mu =np=n\theta =98\,451\times 0.5=49\,225.5} and σ 2 = n θ ( 1 − θ ) = 98 451 × 0.5 × 0.5 = 24 612.75 , {\displaystyle \sigma ^{2}=n\theta (1-\theta )=98\,451\times 0.5\times 0.5=24\,612.75,} to compute
… excerpt ends here. Continue reading the full article.
