I4R.
← Back to BlogResearch

Coding Inconsistencies and Their Impact

August 20, 2026 · Derek Mikola

A common phrase from authors whose work contains coding inconsistencies is that their conclusions do not change. This is usually evidenced because statistics are similar (magnitudes, signs, standard errors, p-values, etc.) with or without the “correction”.

Inconsistencies can only change statistical conclusions if they can be mapped into a model. The familiar case is when a key variable is incorrectly defined. Absent a sample change, this might be the closest ceteris paribus change to the original research. One may be able to speak cleanly about the comparison of the two models, one with the old variable and one with the new variable. The challenge is thinking abstractly enough to permit a variety of interacting changes: definitions of variables, samples, and estimators are all possibly intertwined.

This idea is both obvious and subtle. Obvious, because this is the only possible way to change statistical conclusions. Subtle, as it permits thinking of inconsistencies (and their corrections) as omitted variables and selected samples.

As a complement, not all inconsistencies need to be mapped into a model to invalidate its conclusions or reassess the research. While the research community gains information as researchers scrutinize the data and code for any project, this takes as given contextual information which is costly (or impossible) to verify. Contextual information is assumed just as one makes identifying assumptions. Information may be presented that makes past assumptions unacceptable or incredible. Of course, this doesn’t change the data or codes and cannot change the statistical conclusions of the model.