In frequentist statistics, power is the probability of detecting an effect (i.e. rejecting the null hypothesis) given that some prespecified effect actually exists using a given test in a given context. In typical use, it is a function of the specific test that is used (including the choice of test statistic and significance level), the sample size (more data tends to provide more power), and the effect size (effects or correlations that are large relative to the variability of the data tend to provide more power). More formally, in the case of a simple hypothesis test with two hypotheses, the power of the test is the probability that the test correctly rejects the null hypothesis ( H 0 {\displaystyle H_{0}} ) when the alternative hypothesis ( H 1 {\displaystyle H_{1}} ) is true. It is commonly denoted by 1 − β {\displaystyle 1-\beta } , where β {\displaystyle \beta } is the probability of making a type II error (a false negative) conditional on there being a true effect or association.
Background
Statistical testing uses data from samples to assess, or make inferences about, a statistical population. For example, we may measure the yields of samples of two varieties of a crop, and use a two sample test to assess whether the mean values of this yield differs between varieties. Under a frequentist hypothesis testing framework, this is done by calculating a test statistic (such as a t-statistic) for the dataset, which has a known theoretical probability distribution if there is no difference (the so-called null hypothesis). If the actual value calculated on the sample is sufficiently unlikely to arise under the null hypothesis, we say we identified a statistically significant effect. The threshold for significance can be set small to ensure there is little chance of falsely detecting a non-existent effect. However, failing to identify a significant effect does not imply there was none. If we insist on being careful to avoid false positives, we may create false negatives instead. It may simply be too much to expect that we will be able to find satisfactorily strong evidence of a very subtle difference even if it exists. Statistical power is an attempt to quantify this issue. In the case of the comparison of the two crop varieties, it enables us to answer questions like:
Is there a big danger of two very different varieties producing samples that just happen to look indistinguishable by pure chance? How much effort do we need to put into this comparison to avoid that danger? How different do these varieties need to be before we can expect to notice a difference?
Description
Suppose we are conducting a hypothesis test. We define two hypotheses H 0 {\displaystyle H_{0}} the null hypothesis, and H 1 {\displaystyle H_{1}} the alternative hypothesis. If we design the test such that α is the significance level (α being the probability of rejecting H 0 {\displaystyle H_{0}} when H 0 {\displaystyle H_{0}} is in fact true) then the power of the test is 1 − β where β is the probability of failing to reject H 0 {\displaystyle H_{0}} when the alternative H 1 {\displaystyle H_{1}} is true.
To make this more concrete, a typical statistical test would be based on a test statistic t calculated from the sampled data, which has a particular probability distribution under H 0 {\displaystyle H_{0}} . A desired significance level α would then define a corresponding "rejection region" (bounded by certain "critical values"), a set of values t is unlikely to take if H 0 {\displaystyle H_{0}} was correct. If we reject H 0 {\displaystyle H_{0}} in favor of H 1 {\displaystyle H_{1}} only when the sample t takes those values, we would be able to keep the probability of falsely rejecting H 0 {\displaystyle H_{0}} within our desired significance level. At the same time, if H 1 {\displaystyle H_{1}} defines its own probability distribution for t (the difference between the two distributions being a function of the effect size), the power of the test would be the probability, under H 1 {\displaystyle H_{1}} , that the sample t falls into our defined rejection region and causes H 0 {\displaystyle H_{0}} to be correctly rejected. Statistical power is one minus the type II error probability and is also the sensitivity of the hypothesis testing procedure to detect a true effect. There is usually a trade-off between demanding more stringent tests (and so, smaller rejection regions) and trying to have a high probability of rejecting the null under the alternative hypothesis. Statistical power may also be extended to the case where multiple hypotheses are being tested based on an experiment or survey. It is thus also common to refer to the power of a study, evaluating a scientific project in terms of its ability to answer the research questions they are seeking to answer.
… excerpt ends here. Continue reading the full article.
