The notions of reproducibility and robustness are foundational to the work we do at the Institute for Replication. A related term that has been making the rounds recently is “multiverse analysis”, so I naturally had to look into it.
The origin of the term can be traced back to Steegen, Tuerlinckx, Gelman and Vanpaemel, 2016, and I’d been vaguely aware of it for a while, but what renewed my interest in the topic was coming across Engzell and Mood 2023, i.e. an actual application of the method in substantive research, and one that won a prize at that. So, what is it all about? Let’s hear it from the original authors:
“A multiverse analysis involves performing the analysis of interest across the whole set of data sets that arise from different reasonable choices for data processing.” (Steegen et al. 2016)
Now, while the original definition seemed to concern itself exclusively with data processing, the conversation since then has expanded to include data analytic aspects like model specification. Still, it has been rightly pointed out that some questions about the applicability and goals of multiverse analysis remain unresolved. In a post provocatively titled Mülltiverse Analysis (Müll being the German word for garbage), Julia Rohrer rightly points out that the notion of the “analysis of interest” which is to be held fixed can be interpreted in various ways, some more questionable than others, and that ultimately there’s a risk of misuse by academics trying to dazzle the audience with a fancy method to cover for a deeper lack of substance.
Ultimately, my impression across the various discussions on the topic, is that they all seem to agree on the central point that the original authors had already enunciated: if a multiverse analysis with high variation in results can be concocted at all, it is only because the field in question has not refined its theory with enough precision to trim down the possibilities that exist in the garden of forking paths.
The multiverse in large-scale replication efforts
I went into this topic with a very concrete and pragmatic question in mind: "what’s this newfangled multiverse thing and how does it fit into the work we do?” My conclusion at this time is that multiverse analysis certainly is a tool to consider for the meta-research we do, but that introducing it as a part of the procedures for our large-scale activities, i.e. Replication Games, would not be appropriate. While our wonderful volunteers often manage to run some robustness checks for the papers they’re reproducing, a multiverse analysis appears to be a much more involved task, both from a computational and a substantive standpoint.
So, despite its awe-inspiring name (truly a slam dunk in marketing, that one), it seems that the multiverse is a bit too narrow to accommodate the full range of work we do at this time.
Side note: is metascience getting more popular?
When I said that “multiverse analysis” seems to be having a recent increase in popularity, I did not want to leave that up to pure vibes, so I checked Google Trends for related terms:
It would seem like there has been a general increase in interest across the entire spectrum of metascience-related concepts since last year. As you can see from the line in 2022, Google marks points where a change in measurement has happened, so the increase would appear to be genuine. None of these ideas are particularly new, and I couldn’t find any specific event that would have led to such a sudden spike. Please let me know if you have any explanation in mind!