Statistical Power Analysis
Design Powerful Experiments Capable of Detecting Important Effects
In science and environmental management conclusions and decisions may be based on the outcome of statistical tests of null hypotheses. These tests are prone to two types of error. A Type I error is the rejection of a true null hypothesis, while a Type II error is the failure to reject a false null hypothesis, put another way, to detect a true effect. These errors have different consequences; which set of consequences is of most concern varies with the objectives of the test.
A Type I Error is the Rejection of a True Null Hypothesis
Scientists are cautious to avoid making Type I errors - falsely concluding an effect exists when it does not. Such an error could lead research programs astray, wasting research funds and harming the scientist's academic reputation. To protect against Type I error, the level of statistical significance (α), its rate of occurrence in the frequentist paradigm of classical statistics, is usually set quite low, commonly at 0.05 (5%), or lower.
A Type II Error is the Failure to Reject a False Null Hypothesis
Environmental managers charged with monitoring environmental conditions may be more concerned about committing Type II errors - wrongly concluding an effect does not exist when in fact it does. Such an error could lead to over harvesting of a natural resource, or cause an increased risk to public health.
Protecting against a Type II error requires that its rate of occurrence (β) is held suitably low by ensuring that its converse, statistical power, calculated as 1-β is high. This is achieved through the design of the experiment, or monitoring program.
Ideally, scientists and managers would like statistical hypothesis tests with acceptably low Type I and II error rates, and high statistical power. In practice, this is done by modeling the four components of statistical power - sample variance, sample size, the level of statistical significance, and the effect size of interest - in a process called statistical power analysis.
Statistical Power Analysis Informs the Design of Experiments and Monitoring Programs
Only sample size and the level of statistical significance are directly under the control of the analyst, but reasonable estimates of the likely levels of variance in the data can be estimated from pre-existing data. Additionally, relationships between sample size and variance can be used to explore how variance repsonds to alternative sampling designs. At the least, the estimated level of variance can be varied over a range of values.
The effect size that will exist in the sample data cannot be known ahead of time, but by estimating the effect size of importance to detect, the analyst can explore how to structure the sampling design to be able to detect this effect size with high power .
Thus, by modeling the behaviour of these components during the design phase of an experiment or monitoring program, it may possible to choose a design with acceptable statistical power before implementation.
Some useful analyses that can be explored through statistical power analysis include:
- How statistical power responds to sample size.
- How statistical power varies over a range of assumed sample variances.
- How statistical power varies over different levels of statistical significance.
- How statistical power varies with effect size of interest.
The results of these analyses inform choices about the design of experiments or monitoring programs, such as how often to sample, how to structure that sampling, what type of statistical test to use, or even whether it’s worth monitoring at all – if natural variability is so high as to ensure experiments with low statistical power it may be more cost-effective to explore other options for learning.
An excellent technical introduction to statistical power and statistical power analysis with applied examples is presented in a paper by Randall Peterman. (R. M. Peterman. 1990. Statistical power analysis can improve fisheries research and management. Canadian Journal of Fisheries and Aquatic Sciences 47: 2-15.)
Environmental Sampling in New York City
|
What do you think about this article?
More in Science & Nature
Latest Articles