Preregistration and Registered Reports

Preregistration and Registered Reports


An overview of preregistration, Registered Reports, the differences between them and how to do them


Background

At primary/secondary school we learnt about the importance of planning science experiments. We were always taught that you must first write your introduction, your aim, hypothesis, and methods, before performing any experiment. Hypothesis driven data collection is the mainstay of much of our research and has been this way for centuries. But today, how many of us look at our data first and then choose a hypothesis? And how many of us write our introduction last to help create a story around the data?


If this all sounds a bit all too familiar – then you may be HARKing - Hypothesising After the Results are Known. Don’t worry, we’ve all done it! These days, there is so much emphasis on telling a story with our data (one that is positive, novel and impactful), that many of us end up committing the act of HARKing. The issue with HARKing is that the hypothesis is never actually tested, and that causes a problem when it comes to our statistical analysis because p values are generated on the back of the null hypothesis.


This leads to another question. How many of us have done the experiment, analysed the data, realised that we’ve not quite reached significance, and then added a few more replicates/n=numbers to get the p-value under the 0.05 line? And how many of us have tried different significant tests, different variables, or removal of outliers (because they do not fit the trend or are “2 standard deviations away

from the mean”) to reach significance, and then presented such analyses as though we planned them in advance? Yep – we’ve all done it. But we shouldn’t, it’s considered p-hacking. P-hacking (also known as data-fishing and data-dredging) is analysing data multiple ways to reach significance (or find trends) and then only reporting the analyses that “worked”. Doing so fundamentally undermines the reliability

of our research.


So, if many of us are accidentally HARKing and/or p-hacking (due to the intense pressure to publish a high impact story) what can we do about it? One solution is to openly plan our work in advance with Study preregistration or Registered Reports.

Study prepregistration

Study preregistration is clearly and openly stating our experimental rational, hypothesis, and methods including the amount of n-numbers and what statistical analyses are going to be used, prior to conducting the experiment. Posting of a research plan can be made easily at an independent registry e.g. https://osf.io/, https://aspredicted.org/, https://clinicaltrials.gov/


Openly sharing research plans bring a variety of benefits:


  1. Preregistration offers a time-stamped account of your original research plans. Once the study is completed and submitted for review/published, reviewers/readers can refer to the preregistered study design and can see that the original hypothesis was tested (i.e., no HARKing), and the experiment was performed as described in the design, as was the analysis (e.g., no p-hacking). Obviously, experiments do not always go according to plan. In these instances, it is important to be transparent - report any deviations from the original preregistered study (and why) in the resulting manuscript.
  2. Reduces bias. When it comes to data, the downside to being human is that we tend to subconsciously seek out the most interesting trends once the data is presented to us. Therefore, pre-defining the analysis before conducting the study reduces the opportunity for ‘analytical flexibility’ and altering the experimental n=number to favour significant outcomes (e.g., retrospectively adding more replicates to reach significance).
  3. Plan actions carefully. In the fast-paced era of modern science, there is a lot of pressure to get on with studies quickly. This often means there is not enough time to organise our thoughts, reflect on experimental design, and potentially seek advice from others about the best way to approach a new study. Preregistration encourages us to take a step back, and thoroughly plan our experiments.
  4. Community feedback. By making your study design available to the wider community it welcomes feedback prior to commencing the study. Suggestions regarding the proposed experimental tools, methods and/or analysis can only be a good thing at this stage.


More information on study preregistration can be found here: https://cos.io/prereg/ 


Preregistration guidance, including a video by Uli Dirnagl discussing options for preregistration, can also be found in our in vivo toolkit.

Registered Reports

A stage up from study preregistration is registered reports. A relatively new publishing format, registered reports was first welcomed by the journal Cortex in 2013 and is now available at more than 200 journals around the globe including Nature Human Behaviour and the BNA's journal, Brain and Neuroscience Advances (see https://cos.io/rr/ for full list of participating journals). Papers submitted as a registered report are accepted in advance based solely on the quality of the study design and published regardless of the outcome of the experiments.


Submitting a registered report for publication is a two-step process:


Stage 1: The study design including the Introduction, Methods, Proposed Analysis, and any supporting pilot data undergo peer-review. High quality protocols that demonstrate best practice are offered InPrinciple Acceptance. The authors then conduct the study.


Stage 2: Resubmission of the full manuscript including the newly obtained results and associated discussion undergo a second round of peer-review. Providing the experiments were conducted as approved in stage 1, the paper is accepted and published.


As with preregistration, registered reports are associated with multiple benefits:


  1. Study design is peer-reviewed. Stage 1 review of the registered report process evaluates the importance of the research question, the rationale for the predictions, and the robustness and suitability of the methods. Reviewers may give constructive feedback e.g. alternative protocols, additional controls, and also advise on the statistical analysis and power/n-numbers. Feedback at this stage aims to help improve the quality and reliability of the study.
  2. Eliminates questionable research practices. If the study is pre-defined, undisclosed flexibility in the hypotheses or analysis plans is preventing, e.g., no HARKing or p-hacking.
  3. Negative or inconclusive data are published. Providing the research question and methodology are sound, the experiments were conducted as approved, and the conclusions based on the evidence, then the results are guaranteed to be published. This is an extremely important and credible benefit. In our current conventional publishing system, preference is shown for positive/significant, novel and high impact data. But all data resulting from robust and well conducted studies (positive, negative or otherwise) should be published irrelevant of outcome.
  4. Open data and materials. Contributing authors are encouraged (or at some journals, required) to publicly archive study data, digital materials and analysis code. This added transparency will facilitate future meta-analysis, encourage more connections between researchers, and increase the reproducibility of research.
  5. Submit results to one journal. Gone are the days of rejection based on the result of a study, reducing the chances of having to shop articles from journal to journal.
  6. Flexibility. Some feel that this publication format will stifle creativity because you must do exactly what was outlined Stage 1. In fact, this couldn’t be further from the truth. Obviously, the idea of registering your study means that you will attempt to conduct the experiments as planned. But as we all know, science experiments often deliver unexpected hurdles that may lead to deviations from the plan. The registered report format encourages these deviations to be included within the final article and authors can also include any additional exploratory analyses that were not anticipated at stage 1. The only requirement is transparent reporting at all times.
  7. No word limit on methods. This allow the experimental protocols to be sufficiently detailed to allow others to successfully replicate the experiment.


If you'd like to hear more about Registered Reports, view the video below of Guillaume Rousselet (University of Glasgow) giving a guide to the format:

Registered Reports Funding Partnerships

As a means of incentivising Registered Reports and streamlining the peer review processes for researchers, some funders have begun to look at combining the Stage 1 manuscript review with the grant application review process. Following a successful feasibility study, in 2022 Cancer Research UK introduced an RR pilot scheme for three of its funding awards alongside a consortium of 12 journals. In this model, CRUK's grant panel peer review precedes the Stage 1 manuscript review by the journal of the researcher's choice from the participating journals.


Within neuroscience, in 2021 BrainsCAN in Western University Canada introduced an integrated RR funding partnership. In this model, after the funder accepts a Letter of Intent from the researcher, it delegates the scientific peer review to a number of journals, funding Stage 1 manuscripts approved through this process provided they do not deviate from the Letter of Intent. 


Top

Examples of RRs in neuroscience

  • Henson RN, Suri S, Knights E, et al. Effect of apolipoprotein E polymorphism on cognition and brain in the Cambridge Centre for Ageing and Neuroscience cohort. Brain and Neuroscience Advances. 2020;4. doi:10.1177/2398212820961704
  • Boayue NM, Csifcsák G, Aslaksen P, Turi Z, Antal A, Groot J, et al. (2020). Increasing propensity to mind-wander by transcranial direct current stimulation? A registered report Increasing propensity to mind-wander by transcranial direct current stimulation? A registered report. European Journal of Neuroscience, 51(3), 755–7. doi:10.1111/ejn.14347
  • Puhlmann LMC, Linz R, Valk SL, Vrticka P, Vos de Wael R, Bernasconi A, Bernasconi N, Caldairou B, Papassotiriou I, Chrousos GP, Bernhardt BC, Singer T, Engert V. Association between hippocampal structure and serum Brain-Derived Neurotrophic Factor (BDNF) in healthy adults: A registered report. Neuroimage. 2021 Aug 1;236:118011. doi: 10.1016/j.neuroimage.2021.118011
  • Loenneker HD, Artemenko C, Willmes K, Liepelt-Scarfone I, Nuerk HC. Deficits in or preservation of basic number processing in Parkinson's disease? A registered report. J Neurosci Res. 2021 Oct;99(10):2390-2405. doi: 10.1002/jnr.24907
  • Tibon R, Greve A, Humphreys G, Quent A & Henson RN. (in press). Do activations and representations differ during successful retrieval from episodic versus semantic memory? Accepted first stage of a Registered Report. Nature Human Behaviour. Stage I manuscript:  osf.io/az2df


Top

Useful resources

Read the Center of Open Science's information on Registered Reports: https://cos.io/rr/.


More information on preregistration and Registered Reports can be found in the UKRN Primer: https://osf.io/8v2n7/


To read more about the impact and development of RRs, check out:

  • Chambers CD, Tzavella L. The past, present and future of Registered Reports. Nat Hum Behav 6, 29–42 (2022). https://doi.org/10.1038/s41562-021-01193-7
  • Ellis RJ. Questionable Research Practices, Low Statistical Power, and Other Obstacles to Replicability: Why Preclinical Neuroscience Research Would Benefit from Registered Reports. eNeuro. 2022 Aug 3;9(4):ENEURO.0017-22.2022. doi: 10.1523/ENEURO.0017-22.2022


Find out about the RR peer review community the BNA's Journal has partnered with: https://www.bna.org.uk/mediacentre/news/pcirr/


Top


This toolkit was produced with guidance from former Credibility Advisory Board member Professor Chris Chambers.

Share by: