How can we link publications to replications and generalizations? Improving the findability of replications and retractions

Science is supposed to be a self-correcting system. But how well can it correct itself if we do not know which studies have subsequently been replicated, reproduced, generalized—or failed to replicate?

Consider a researcher who wants to cite an empirical study. Ideally, they would not only want to know what the original study found, but also whether its findings have subsequently been independently verified. If a finding has been successfully replicated across different settings or using different data and methods, this provides additional evidence for its robustness. Conversely, a failed replication—or a retraction—may be highly relevant when deciding whether and how to rely on the original finding.

Yet, today, it is often surprisingly difficult to answer a seemingly simple question: Has this study been replicated, and if so, what did the replication find? Replications are published in different journals, may use different terminology, and are often not explicitly linked to the original publication. The same applies, in different ways, to retractions and other forms of post-publication correction.

Conversely, simply searching full texts for terms such as ‘replicat*’ is likely to produce many false positives. For example, in our analysis of 1,213 Google Scholar search results containing replication-related terms in the literature on charitable behavior, only 34 actually repeated a previous study.

Therefore, improving the links between originals and their replications would make the scientific literature easier to navigate and could strengthen the self-correcting nature of science. It could also help researchers make better-informed decisions about which findings to cite and build upon.

In this post, we introduce different tools that aim to make these connections more visible and improve the findability of replications, retractions, and related work.

The tools both rely on the FORRT Library of Replication and Reproduction Attempts (FLoRA) as a source of information of which replication study links to which original.FLoRA contains original-replication pairs for both reproductions and replication studies using different samples. It was originally crowdsourced by over 200 collaborators and is currently being enhanced via automatic screening of Open Alex and human validation of the entries.

Zotero Replication Checker

This is a Zotero privacy-preserving plugin that checks items in a researcher’s Zotero library against FLoRA and adds replications of studies that are already present in the library. It can be applied to individual items, folders, whole libraries, or group libraries. Once installed, it also scans each newly added study. This way it enables researchers to find replications relevant to studies they read and cite. However, FLoRA is not comprehensive, so the absence of a replication notification does not necessarily mean a replication does not exist.

Researchers can access it and freely download it at this link: https://forrt.org/flora-zotero/

A video tutorial by the lead developer is available here:https://www.youtube.com/watch?v=llS7bM9q6pE

FORRT Open Research Extension

An extension for the Chrome browser that shows replication evidence, retractions, open-access copies for the papers, and more on any webpage you read. This way, researchers are informed about important information related to the robustness of the article all in one place, whereas before they would have to go to multiple sources. This is currently in development, but a first look is available in this walk through: https://www.youtube.com/watch?v=ZvbpINDhIwg

What remains to be solved?

These tools are an important step toward making the scientific literature more transparent and easier to navigate. But they also highlight some fundamental questions that need to be addressed. For example: What exactly is considered as a replication? Can this be identified automatically from publication metadata and text, or does it require human judgment? And is it possible to achieve comprehensiveness of such databases?

Want to know more? Check out our podcast episode here!