What Would It Cost to Remove Direct PII from Public Replication Packages?

Replication packages have transformed the social sciences. Researchers increasingly post the data and code underlying published papers on Dataverse, OpenICPSR, Zenodo, institutional repositories, and other public platforms. This makes published research much easier to reproduce, verify, and build upon.

But there is a downside to making tens of thousands of research datasets publicly downloadable: occasionally, researchers accidentally upload information that should not be public.

The most concerning cases are not subtle questions about statistical disclosure risk. They are files containing direct personally identifiable information (PII): names, phone numbers, email addresses, addresses, identification numbers, case numbers, or other information that can immediately identify an individual.

The risks are particularly serious when the underlying research concerns sensitive populations or behaviors—for example, domestic violence, criminal activity, incarceration, sexual behavior, health, migration status, political repression, or vulnerable children.

In our recent paper (https://www.econstor.eu/handle/10419/343373/) we evaluated a sample of 327 replication packages from 11 leading journals in economics, political science, and psychology. We found that 21.1% of packages contained identifying information, with 14.4% containing direct identifiers.

Solutions for future

In our paper we outline a number of steps authors, journals, and repositories could collectively take to prevent exposure in newly published packages.

Automatic checks for PII could be implemented by journals or repositories and require authors to either remove the personal information, or confirm all the flagged information can be published.

Number of tools could be used to conduct such checks. We ourselves are currently developing such a tool based on the code used for detection of PII for our paper.

We are working both on a version of the code that could be run by authors locally, prior to sharing any data. We are also developing an online screening tool targeted at data editors that would enable them to use this script with more powerful language models during final pre-publication screening.

We estimate that running and maintaining such a tool to enable a number of journals to conduct PII checks would cost us about $2,000 per month, depending on the demand.

You can find out more about our tool at: https://piichecker.org

But what about the packages that are already published?

Harvard Dataverse itself contains nearly 50,000 replication packages/datasets in Social Sciences some of which are nearly 20 years old with many more packages in other fields that might contain participants’ identifying information.

Furthermore, repositories like Zenodo, OpenICPSR, or OSF host countless more replication packages.

Individual researchers have relatively weak incentives to search their old replication packages for files they may have uploaded five or ten years ago to confirm they do not include personally identifiable information, some of these packages might even be forgotten or orphaned.

Additionally, unlike with pre-publication checks, it is more difficult to rely on researchers’ help with evaluating an output of an automated screening system. Additional manual labour would also be required in contacting the authors, who might have changed institutions or their emails since the time the replication package was published.

How expensive would it be to systematically look for and remove this information?

At the Institute for Replication (I4R), we can make a first estimate.

I4R has already downloaded approximately 50,000 publicly available replication packages from Dataverse, which represents roughly the universe of social-science replication packages available in their collection. That makes Dataverse a useful place to estimate the cost of a systematic PII audit.

A first step would be to remove articles unlikely to have PII (e.g., those not including any data). Let’s assume this removes 20,000 articles. Suppose we run automated tools across every replication package looking for direct identifiers in CSV files, spreadsheets, text files, Stata files, R data files, PDFs, and other common formats.

Our current estimate is that sufficiently intensive automated processing would cost approximately:

$1.5 per article.

For 30,000 Dataverse replication packages:

30,000 × $1.5 = $45,000 in computation.

But computation is only part of the problem.

Automated systems can flag suspicious files and variables, but someone still needs to determine whether a finding is actually PII, understand the context of the dataset, contact the authors where necessary, and verify that a corrected package has replaced the problematic version.

In our recent paper we found that about 15% of articles with author-collected data contain direct PII. Let’s assume it is a bit lower in those 30,000 packages, say 10%, as some studies might not use author-collected data and rates are likely to be lower in those.

Automated systems are likely to lead to at least some false-positives, this means more than the 10% of packages would have to be understood and the detection confirmed using the context of the packages. Finally, the authors of the about 3,000 packages where true PII is detected would have to be contacted.

Our rough estimate is that auditing the entire Dataverse collection properly would require approximately one full-time postdoctoral researcher for two years. That is two postdoc-years of labor.

If we use a simple planning assumption of approximately $70,000 per postdoc-year in fully loaded employment costs, that adds another: $140,000.

The approximate cost of a comprehensive Dataverse audit would therefore be: $185,000=140,000+45,000

This would cover Dataverse alone.

What about other repositories?

This is hard to estimate as we do not know how many packages these repositories have for the social sciences.

Without consideration for possible economies of scale, a project covering 100,000 packages could therefore easily cost well over $500,000.

That is possible—but it may not be the best place to start.

A cheaper alternative: focus first on where mistakes could cause the most harm

The probability and consequences of accidentally releasing PII are not equal across the social sciences.

Consider two replication packages: One contains attitudes on inflation. The other study involves interviews of survivors of domestic violence.

Even if the probability of an accidental disclosure were identical, the expected harm from the second case could be dramatically larger. This suggests a much cheaper first step: risk-based PII screening.

Instead of immediately spending hundreds of thousands of dollars examining every public replication package, we could first identify research areas in which disclosure of direct identifiers would be particularly consequential.

Priority categories might include research involving:

• domestic violence and intimate-partner violence;
• crime, arrests, incarceration, and criminal records;
• children and child abuse;
• sexual behavior and sexual violence;
• medical and mental-health records;
• substance use;
• refugees and undocumented migrants;
• victims of trafficking;
• political dissidents or participants in protests;
• highly detailed administrative records containing names or contact information.

The first-stage classification could rely largely on article titles, abstracts, keywords, journal fields, and dataset descriptions. The expensive PII-detection pipeline would then run only on the subset of replication packages classified as potentially high risk.

What might a focused project cost?

Suppose 1,000 packages from Dataverse were selected for intensive review.

At $1.5 per package, the computational cost falls to $1,500. Human review would also become much more manageable.

The goal should not be to punish researchers

An important principle of such a project would be that discovering PII should not become a mechanism for publicly identifying or embarrassing researchers who made mistakes.

Replication packages are often complicated. They may contain hundreds of files assembled by research assistants, coauthors, government agencies, survey firms, or earlier generations of a research team. A single forgotten spreadsheet can remain inside a directory even when the analytical dataset itself has been carefully de-identified.

The appropriate response is remediation. When I4R finds and confirms a presence of direct PII in a replication package, the first step normally is to contact the authors privately, explain what was found, and help them replace or remove the affected files. Where appropriate, the hosting repository could also be contacted so access can be temporarily restricted while the problem is fixed.

The objective is a safer open-science ecosystem, not a database of researcher mistakes.