Loosely inspired by the Utrecht Replication Games 😉
It is a lovely morning. Our Opening ceremony for today’s Replication Games ended about twenty minutes ago. Four replication teams have scattered across campus, and are hard at work, by which I mean they have twelve tabs and two statistical softwares open, and might be a minute or two away from their (first) head-desk.
I just finished guiding a virtual team through a messy pre-registration document (“Phew!”, “Oh my this makes so much more SENSE now!!”) – gotta say, it’s always nice to turn an ancient cursed scroll into a moderately confusing pdf. Ready to check on the in-person teams.
The first team is in this main room, by the window. Keyboards clicking, one foot tapping, focus mode on. Better not to disrupt them, unless… “Hey Lenka, can you help with this MATLAB issue?” Oh, a toolbox is missing, an easy fix. An academic equivalent of plugging the machine in. A few clicks and we are getting an excited “okay here we goooo”. The machine has accepted our offering. Excellent.
Room two. “What is this?! This is an abject nightmare. AAAA!” Ah, the most experienced team. Business as usual, they do not seem to need me. (I will only check in if the number of As exceeds seven.)
Room three. “Oh wow! Such a pretty chart!” “Look, I reproduced Table 4!” “This WORKS!” A precious and delicate phase, typically observed shortly before the discovery of a unit conversion issue, a hidden sample restriction, or a variable named final_outcome_real. I deliver more coffee and continue my rounds.
Room four. “Huh?!” “Why?” “Where is raw.xlsx?” “I wish I had your problems!” “Technically, this is also your problem.” “Eeeeek.” I step in. Ah, this is just a missing README problem combined with a package that was put together during an earthquake. We got this.
A couple of unhappy sounds and a delicious lunch later, I am making another round.
Room one transformed from focus and excitement into a well-oiled machine of box-checking. Table 7? Check. Figure 4? Check. Reproduced? Check. Robust? Check. Happy noises? Very much check. “Yes! Good job. High-five. Impressive. I like this paper. The authors did a great job.”
This is the replication equivalent of finding a clean public restroom at a train station. Rare. Beautiful. Restores your faith in science.
Back to room two. The screaming has turned into rapid keyboard taps, which is either progress or a very elaborate distress signal.
All good over here? “Yeah, all good. We located the missing dataset, merged it with a new wave of data collected two years after the article was published, and are able to get very cool new results. We also fixed a coding error and sped up the code runtime by 30%.” Now these are happy replication noises. The team has entered the “we came here to reproduce a paper and accidentally improved science” stage.
Room three. Silence. Not peaceful silence. Silence that smells like panic and sadness. What’s up? “They use a log-like specification, and when we change the units of measurement, all results are gone! What is happening?” Oh, easy. Look, I have a paper on that. Not the authors’ fault, this is a feature of these specifications. A feature in the same sense that a trapdoor is a feature of a castle. Here, these are some reasonable robustness checks you can use.
Room four. “A replication study walks into a bar. The bartender says, have I seen you before?” “Ha! I am replicating a study on procrastination. But you know what, I can finish that tomorrow.” Great. The team is finished with their report and is now bonding over terrible replication jokes, which is one of the final stages of the Replication Games science cycle. Once they dealt with the confusing package, the rest was a walk in the park. A slightly overgrown park. With unclear signage. And a locked gate called raw.xlsx. But still: a park. Nice.
And then it is time for the Closing ceremony.
The teams (Avengers?) reassemble, slightly tired, 100% caffeinated, and visibly transformed by their encounters with missing files, mysterious specifications, and the emotional range of MATLAB. We celebrate the reproduced tables, the new results, the fixed code, the robustness checks, and, perhaps most importantly, that most things ended up less cursed than they initially appeared.
I thank everyone for their work and willingness to look directly into the abyss of scientific reproducibility and say, “Okay, but what happens if we change the units?” There is applause. There are smiles. There are no screams exceeding seven As. By the standards of any Replication Games, this is a triumph.
Same results next time? We will have to check.