Preregistration and Registered Reports
An overview of preregistration, Registered Reports, the differences between them and how to do them
At primary/secondary school we learnt about the importance of planning science experiments. We were always taught that you must first write your introduction, your aim, hypothesis, and methods, before performing any experiment. Hypothesis driven data collection is the mainstay of much of our research and has been this way for centuries. But today, how many of us look at our data first and then choose a hypothesis? And how many of us write our introduction last to help create a story around the data?
If this all sounds a bit all too familiar – then you may be HARKing - Hypothesising After the Results are Known. Don’t worry, we’ve all done it! These days, there is so much emphasis on telling a story with our data (one that is positive, novel and impactful), that many of us end up committing the act of HARKing. The issue with HARKing is that the hypothesis is never actually tested, and that causes a problem when it comes to our statistical analysis because p values are generated on the back of the null hypothesis.
This leads to another question. How many of us have done the experiment, analysed the data, realised that we’ve not quite reached significance, and then added a few more replicates/n=numbers to get the p-value under the 0.05 line? And how many of us have tried different significant tests, different variables, or removal of outliers (because they do not fit the trend or are “2 standard deviations away
from the mean”) to reach significance, and then presented such analyses as though we planned them in advance? Yep – we’ve all done it. But we shouldn’t, it’s considered p-hacking. P-hacking (also known as data-fishing and data-dredging) is analysing data multiple ways to reach significance (or find trends) and then only reporting the analyses that “worked”. Doing so fundamentally undermines the reliability
of our research.
So, if many of us are accidentally HARKing and/or p-hacking (due to the intense pressure to publish a high impact story) what can we do about it? One solution is to openly plan our work in advance with Study preregistration or Registered Reports.
Study preregistration is clearly and openly stating our experimental rational, hypothesis, and methods including the amount of n-numbers and what statistical analyses are going to be used, prior to conducting the experiment. Posting of a research plan can be made easily at an independent registry e.g. https://osf.io/, https://aspredicted.org/, https://clinicaltrials.gov/
Openly sharing research plans bring a variety of benefits:
More information on study preregistration can be found here: https://cos.io/prereg/
Preregistration guidance, including a video by Uli Dirnagl discussing options for preregistration, can also be found in our in vivo toolkit.
A stage up from study preregistration is registered reports. A relatively new publishing format, registered reports was first welcomed by the journal Cortex in 2013 and is now available at more than 200 journals around the globe including Nature Human Behaviour and the BNA's journal, Brain and Neuroscience Advances (see https://cos.io/rr/ for full list of participating journals). Papers submitted as a registered report are accepted in advance based solely on the quality of the study design and published regardless of the outcome of the experiments.
Submitting a registered report for publication is a two-step process:
Stage 1: The study design including the Introduction, Methods, Proposed Analysis, and any supporting pilot data undergo peer-review. High quality protocols that demonstrate best practice are offered InPrinciple Acceptance. The authors then conduct the study.
Stage 2: Resubmission of the full manuscript including the newly obtained results and associated discussion undergo a second round of peer-review. Providing the experiments were conducted as approved in stage 1, the paper is accepted and published.
As with preregistration, registered reports are associated with multiple benefits:
If you'd like to hear more about Registered Reports, view the video below of Guillaume Rousselet (University of Glasgow) giving a guide to the format:
As a means of incentivising Registered Reports and streamlining the peer review processes for researchers, some funders have begun to look at combining the Stage 1 manuscript review with the grant application review process. Following a successful feasibility study, in 2022 Cancer Research UK introduced an RR pilot scheme for three of its funding awards alongside a consortium of 12 journals. In this model, CRUK's grant panel peer review precedes the Stage 1 manuscript review by the journal of the researcher's choice from the participating journals.
Within neuroscience, in 2021 BrainsCAN in Western University Canada introduced an integrated RR funding partnership. In this model, after the funder accepts a Letter of Intent from the researcher, it delegates the scientific peer review to a number of journals, funding Stage 1 manuscripts approved through this process provided they do not deviate from the Letter of Intent.
Read the Center of Open Science's information on Registered Reports: https://cos.io/rr/.
More information on preregistration and Registered Reports can be found in the UKRN Primer: https://osf.io/8v2n7/
To read more about the impact and development of RRs, check out:
Find out about the RR peer review community the BNA's Journal has partnered with: https://www.bna.org.uk/mediacentre/news/pcirr/
This toolkit was produced with guidance from former Credibility Advisory Board member Professor Chris Chambers.