What We Learned from Trying to Reproduce Psychological Science

Over the past two years, I4R has collaborated with Psychological Science, a leading journal in psychology, on a large reproducibility project.

67 independent teams completed reproduction reports for 64 articles published in the journal in 2024 and 2025. That covers about 44% of the 146 eligible articles from those two years. Teams followed a standardized protocol specified by I4R. They documented data and code availability, computational reproducibility, coding errors, prevalence of preregistration, deviations from preregistered plans, and whether those deviations were acknowledged.

Here is what we learned. (link to discussion paper)

Reproducibility depends a lot on where you start

If teams started with the authors' final analysis dataset, 58.2% of articles were fully reproducible. Another 32.8% were partially reproducible. Only 9.0% could not be reproduced.

The picture changed when teams had to start from the raw data.

Only 28.4% of articles were fully reproducible from raw data. Another 34.3% were partially reproducible. For 37.3%, the results could not be reproduced from the raw data.

This is one of the clearest findings in the paper.

Running the final analysis codes was often not the main problem. The harder part was reconstructing everything that happened before those analyses were run. That includes cleaning the data, applying exclusions, constructing variables, and building the final analysis sample.

The availability of research materials shows the same pattern. Final analysis datasets were fully available for 74.6% of articles. Analysis scripts were fully available for 70.1%. By contrast, raw data were fully available for only 46.3% of articles. The same was true for data-cleaning scripts.

In other words, researchers were much more likely to share the final steps of the analysis than the earlier steps. However, those earlier steps are exactly what someone needs to reconstruct the full research process.

Errors were common, but most were not major

Teams found at least one coding error or statistical reporting inconsistency in about 63% of the reproduced articles, distributed as follows. Coding errors appeared in 43.3% of articles. Statistical reporting or documentation problems appeared in 32.8%.

Among the articles with at least one problem, statistical reporting errors were the most common category. They appeared in 38.1% of flagged articles. General coding and implementation mistakes followed at 33.3%.

Problems with data filtering or sample construction appeared in 23.8% of flagged articles. The same share had code that could not be successfully executed because of missing files or other technical problems.

But the severity of these problems matters.

Among the articles that were flagged, 59.5% had only minor issues. These were problems that were unlikely to change the interpretation of the results. The remaining 40.5% had at least one issue that could affect estimates, standard errors, statistical significance, or interpretation.

Thus, the 63% headline needs some context. Many of the problems were minor. At the same time, a meaningful share was not. Many also looked like the kind of problems that could have been caught before publication.

A clear README makes a big difference

We also asked reproduction teams to judge the quality of the README file that came with each replication package. Only 23.9% of packages had a README that teams considered clear. Another 43.3% had a README that teams considered unclear. The remaining 32.8% had no README.

The difference in reproducibility was striking.

Among packages with a clear README, 87.5% were fully reproducible from the final analysis data. For packages with an unclear README, that figure fell to 48.3%. For packages with no README, it was 50.0%.

An unclear README did not appear to offer much advantage over having no README at all.

What mattered was not simply having documentation. The documentation had to be useful.

We should be careful with this result. The groups are small. A clear README may also be a sign of better coding, better organization, or better data management more generally. The result is descriptive and not causal. Still, the lesson is simple. Good documentation matters.

Preregistration is now common. Deviations are too.

About three quarters of the articles in the sample included at least one preregistered study. That is 50 of the 67 reproduction reports, or 74.6%.

The preregistrations were generally strong on the basics. More than 90% specified the statistical model, sample size or power calculation, primary outcome, hypotheses, primary predictor, and inclusion or exclusion criteria.

Other details were much less common. Control variables appeared in only a little more than half of the plans. Data-cleaning procedures appeared in fewer than half. Robustness checks and subgroup analyses appeared in a minority. Multiple-hypothesis corrections appeared in fewer than 10%. That matters because many important analytical choices happen in those less clearly specified areas.

We also found that preregistered plans were often not followed exactly. Of the 50 preregistered articles, 42 had at least one deviation from the pre-analysis plan. That is 84%.

Of those 42 articles, 17 did not disclose any of their deviations. That means 40.5% of the articles with deviations gave readers no indication that the published analysis differed from the preregistered plan.

The most common deviations involved the design or scope of what was reported. Changes to estimation or modeling choices came next.

Preregistration has clearly become common in this journal. But preregistration alone does not prevent deviations from the original plan. The level of detail in the plan matters, so does being transparent when the plan changes.

Four things journals and researchers could do now

1. Put replication packages somewhere that preserves the record

Some reproduction teams found packages that appeared to have changed after publication. That creates a basic problem. If a file changes later, it can become difficult to know whether a discrepancy was present in the original package or introduced afterward.

Most of the materials in our sample were stored on the Open Science Framework. Its standard project storage allows files to remain editable unless an archival mechanism is used. On top of that, the Center for Open Science has announced that this general-purpose storage functionality will be discontinued.

Replication packages should be stored in repositories that preserve clear and permanent versions. Harvard Dataverse, Zenodo, and ResearchBox are some options discussed in the paper.

2. Ask for the replication package during revision

The problems we found had already made it into published papers.

One way to catch more of them would be to require a replication package during the revise-and-resubmit stage. That would give reviewers a chance to check whether the results can actually be reproduced before the paper is accepted.

Most authors who already prepare a replication package would simply be sharing it a few months earlier.

3. Treat computational skills as part of methods training

Researchers are expected to work with increasingly complicated data and code. Nonetheless, training in those skills is still uneven.

Graduate methods courses should cover code documentation, data management, basic computational reproducibility, and research transparency checks.

Professional societies can also help researchers who are already established. Workshops and collaborations with researchers who have stronger computational backgrounds are one option.

Another is a red-team approach. One group conducts the research. Another independently tries to find discrepancies and test whether the analysis holds up.

That kind of checking could catch problems before they become part of the published record.

4. Require a standard README

This may be the easiest change of all.

Only a minority of the packages in our sample had a clear README. Yet clear documentation was strongly associated with successful reproduction.

Researchers do not need to invent their own format. The social science data editors community already provides a template for replication packages. Journals could simply adopt a standard template and require authors to use it.

How to read these numbers

Teams tended to avoid papers with very large datasets, very long runtimes, unfamiliar methods, or incomplete materials. This means the papers in our sample may have been easier to reproduce than the average paper in the journal.

If harder papers are also more likely to contain problems, our reproducibility rates may be too high and our error rates may be too low.

There is another issue.

None of our analyses in this project were preregistered. The results should, therefore, be treated as descriptive and exploratory. They also describe one leading journal, not psychology as a whole.

Finally, about 84% of the reproduced articles were submitted before the current editorial board introduced stronger transparency policies. Accordingly, these results should not be read as an assessment of the journal's current standards.

They are better understood as a baseline. The next question is whether the journal's newer policies improve data sharing, code sharing, documentation, and reproducibility. Future rounds of the project should give us a much better answer.