2.1.6 Sample size and power analysis
UNDER CONSTRUCTION
A. Background & Definitions
Balancing sample size, effect size and power is critical to good study design. When the power is low, only large effects can be detected, and negative results cannot be reliably interpreted. The consequences of low power are particularly dire in the search for high-impact results, when the researcher may be willing to pursue low-likelihood hypotheses for a groundbreaking discovery (see Fig. 1 in Krzywinski & Altman 2013). Ensuring that sample sizes are large enough to detect the effects of interest is an essential part of study design.
Studies with inadequate power are a waste of research resources and arguably unethical when subjects are exposed to potentially harmful or inferior experimental conditions. Addressing this short- coming is a priority—the Nature Publishing Group checklist for statistics and methods (http://www.nature.com/authors/policies/ checklist.pdf) includes as the first question: “How was the sample size chosen to ensure adequate power to detect a pre-specified effect size?”
Statistical power analysis exploits the relationships among the four variables involved in statistical inference: sample size (N), significance criterion (α), effect size (ES), and statistical power. For any statistical model, these relationships are such that each is a function of the other three.
B. Guidance & Expectations
General advice:
1. Whenever possible, seek professional biostatistician support to estimate sample size.
2. Avoid using the definition of “small,” “medium,” or “large” effect size based on Cohen's d of .20, .50, or .80, respectively. Cohen's assessments are based on an extensive survey of statistics reported in the literature in the social sciences and may not apply to other fields of science.
3. Retrospective power calculations should generally be avoided, because they add no new information to an analysis (i.e. avoid using observed power to the results of the statistical test).
4. Effect size should be specified on the actual scale of measurement, not on a standardized scale.
5. Rarely can a definitive study be done without first doing a pilot study.
We begin by setting the values of type I error (a) and power (1 – b) to be statistically adequate: traditionally 0.05 and 0.80, respectively. We then determine n on the basis of the smallest effect we wish to measure. If the required sample size is too large, we may need to reassess our objectives or more tightly control the experimental conditions to reduce the variance.
Please provide recommendations that will help the user to develop a method, document, protocol, tool or some other solution customized to their specific research environment.
This guidance should be short and concise, possibly in the form of bullet points.
For example, to develop a protocol for a specific key or support process, the guidance may include recommendations for the protocol to include the following information:
- Who should develop this protocol and how it should be maintained?
- What is the minimum information to be included?
- How often it should be updated / revised
- Is there any training necessary to introduce / implement this protocol?
- Are there any risks associated with this process / method / protocol (e.g. risk of unblinding; emergency access to blinding codes)?
- While the "soft" language such as "should" and "recommend" is strongly preferred, please explicitly identify those recommendations that are "strong" and, if not followed, require an explanation why they are not followed / are not applicable and the Risk Assessment box is to be ticked
Here are two very wrong things that people try to do with this software:
Retrospective power (a.k.a. observed power, post hoc power). You’ve got the data, did the analysis, and did not achieve “significance.” So you compute power retrospectively to see if the test was powerful enough or not. This is an empty question. Of course it wasn’t powerful enough – that’s why the result isn’t significant. Power calculations are useful for design, not analysis. (Note: These comments refer to power computed based on the observed effect size and sample size. Considering a different sample size is obviously prospective in nature. Considering a different effect size might make sense, but probably what you really need to do instead is an equivalence test; see Hoenig and Heisey, 2001.)
Specify T-shirt effect sizes (“small”, “medium”, and “large”).
This is an elaborate way to arrive at the same sample size that has been used in past social science studies of large, medium, and small size (respectively). The method uses a standardized effect size as the goal. Think about it: for a “medium” effect size, you’ll choose the same n regardless of the accuracy or reliability of your instrument, or the narrowness or diversity of your subjects. Clearly, important considerations are being ignored here. “Medium” is definitely not the message!
Here are three very right things you can do:
Use power prospectively for planning future studies.
Software such as is provided on this website is useful for determining an appropriate sample size, or for evaluating a planned study to see if it is likely to yield useful information.
Put science before statistics.
It is easy to get caught up in statistical significance and such; but studies should be designed to meet scientific goals, and you need to keep those in sight at all times (in planning and analysis). The appropriate inputs to power/sample-size calculations are effect sizes that are deemed clinically important, based on careful considerations of the underlying scientific (not statistical) goals of the study. Statistical considerations are used to identify a plan that is effective in meeting scientific goals – not the other way around.
Do pilot studies.
Investigators tend to try to answer all the world’s questions with one study. However, you usually cannot do a definitive study in one step. It is far better to work incrementally. A pilot study helps you establish procedures, understand and protect against things that can go wrong, and obtain variance estimates needed in determining sample size. A pilot study with 20-30 degrees of freedom for error is generally quite adequate for obtaining reasonably reliable sample-size estimates.
C. Resources
- G*Power [1]
- WISE power tutorial [2]
- JAVA applets for power and sample size [3]
- Computation of sample sizes @Psychometrica [4]
Mayo clinical online simulator - Size matters [5]
Scientists talking to biostatisticians [6]
- https://www.ncbi.nlm.nih.gov/books/NBK43321/
- http://davidmlane.com/hyperstat/power.html
- http://powerandsamplesize.com/Calculators/Test-1-Mean/1-Sample-Equality
Guidelines on reporting of sample size (in vivo research):
back to Toolbox
Next item: 2.1.7 Blinding
