Theory 09/Methodology & Philosophy of Science/Research methodology
Replication Fails on a Ratio
Sample size is the headline defence of a study. The freedom to analyse may matter more.
Replication failure is set by the ratio of analytic degrees of freedom to data degrees of freedom rather than by sample size, so a large study with many analytic choices can be less reproducible than a small one with a fixed analysis.
01The evidence
Replication rates vary widely among studies of similar size, and multi-analyst projects find substantial variation in results from the same data.
02The tension
Power calculations address sampling error only. They assume a fixed analysis, which is rarely the case in practice, so the reported interval understates the true uncertainty.
03The mechanism
Every analytic choice is a degree of freedom. When the number of choices is large relative to the information in the data, the researcher's path to significance becomes the dominant source of variation, and the reported result is one draw from a wide distribution.
That makes the ratio the predictor. Preregistration reduces the numerator; larger or more structured data increases the denominator.
04What this is not
Power addresses sampling. This addresses analytic freedom, which explains variation at equal sample size.
05Predictions
- P1Replication should track the analytic-to-data ratio.
- P2Large studies with many choices should replicate worse than small ones with one analysis.
- P3Preregistration should improve replication at constant n.
06Refutation
Replication tracking sample size at controlled analytic freedom would restore the power account.
07What it changes
It would change how studies are reported and reviewed.