Are the effects seen in neuroscience experiments valid?

A review published in Nature Reviews: Neuroscience on the 1st of this month highlighted the problem of small sample size, calling into question the validity of neuroscience experiments. The authors, Button et al., argue that the statistical power in many neuroscience studies is ‘very low’ and that this leads to overestimates of effect size and low reproducibility of results. They point to pressure to publish results rapidly as contributing to alarming rates of false positives.

Statistical power is the probability that it will correctly reject the null hypothesis, when the null hypothesis is false (with the null hypothesis being no effect). Describing the issue as ‘endemic’, the authors detail two types of problem. Firstly, low statistical power due to the nature of the statistics leading to higher rates of false negatives, false positives and low positive predictive power. Secondly, such errors also occur due to biases that tend to co-occur with low statistical power. These associated issues include a larger range of estimates for the magnitude of an effect, selective analysis/reporting and other lower calibre aspects in the experimental design.

They also argue that the problem lies that partly with static sample sizes over the years of research, whilst the desired effect has become more subtle and the complexity of study designs has increased. They thus make recommendations for neuroscientists, including performing an ‘a priori calculation’ based on the literature to estimate the size of effect that is being looked for and making detailed protocols available to allow others to replicate the result.