Initial Evidence of Research Quality of Registered Reports Compared to the Traditional Publishing Model

Abstract

Registered Reports (RRs) is a publishing model in which initial peer review happens before the research is completed. In-principle acceptance before knowing outcomes combats publication bias and provides a clear distinction between confirmatory and exploratory research. The theoretical case for how RRs would improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintended costs of RRs such as reducing innovation or novelty. 353 researchers peer reviewed a pair of papers from 29 published RRs and 57 non-RR comparison papers. RRs outperformed comparison papers on all 19 criteria (mean difference=. 46) with effects ranging from little difference in novelty (0.13) and creativity (0.22) to substantial differences in rigor of methodology (0.99) and analysis (0.97) and overall paper quality (0.66). RRs could improve research quality while reducing publication bias and ultimately improve the credibility of the published literature.

Publication
MetaArXiv Preprints

Project preregistration, materials, data, and anlysis code are available on osf.

Related