We presented our Second Meta Paper at the Berkeley Institute for Transparency in the Social Sciences (BITSS) Annual Conference on April 16th. Our presentation built on our First Meta Paper: assessing reproducibility and robustness in the social sciences. This time, we’re looking at patterns by fields and (economic) methods. Blog followers and I4R fans will have to wait a few more weeks before we make our first draft public. Our work incorporates the excellent scholarship from hundreds of social scientists over the past few years.
The day was packed with incredible presentations. Those interested in the materials presented will eventually be able to find them online with recorded presentations.
The day concluded with an open mic discussion of the audience’s thoughts and unanswered questions from the day. I personally felt enriched by the conclusion and would wager so too did other attendants. (I’ll keep it as a back-pocket tactic for use in my next hosted workshop!)
The BITSS annual conference was a memorable event which I highly recommend to others. It will expand how you think about science and the system academics inhabit.
This post muses two themes from the meeting I’ve taken to in the weeks since: how research ought to be more process-driven (less outcome dependent) and thinking in systems.
Research Ought to Be More Process-Driven, Less Outcome Dependent
P-values and t-statistics are likely less important than how one arrives at their statistics and the strength of the conclusions given from those statistics. What matters is the process: how one did their research. Readers and authors alike should be willing to accept the conclusions of sufficiently well-designed research. Researchers should be attempting faithful execution of their design, tracking deviations which are believed possible to affect their conclusions. That offers a point to work from in the future. This also brings forward how one forms conclusions: they could form prior to analysis and alongside their hypotheses.[1]
Thinking in Systems
The last presentation before the open mic was excellently delivered by Katharina Miller – which I strongly recommend watching – who asked, effectively, “What kind of life do you (we) want?” regarding privacy laws and AI. We were challenged to consider the system we live in, our beliefs and the norms we’d like to going forward.
This analogous thinking should not be avoided when considering how to motivate ‘better’ research.
Many proposals were offered: give funding to those who are process-driven, create separate journals for publishing null results, consider alternatives to the publish-or-perish plaguing top universities, reduce the incentives to p-hack or HARK, promote pre-analysis plans and preregistrations, force replication packages as a condition of publication, etcetera.
The question I have is whether any one of these changes are sufficient to perturb a system to different and better? Does a small change sufficiently affect the system?
What if we did every change at the same time? Is that sufficient to change the system? And if implemented, is that a system we want?
[1] This is less so for exploratory research though things are often explored because they are believe to be important.