I4R regularly encounters published manuscripts with coding discrepancies (errors) in their codebases. We also bang on about them in our blogs, including what to look out for and what to do if they are found. But we can be imprecise when speaking about coding discrepancies as causes or outcomes, at times using both simultaneously. This blog’s purpose is for researchers to reflect on what they care about when they speak of coding discrepancies (errors).
A question like “did the results (e.g. p-values, coefficient signs and magnitudes) change because of the coding error?” treats coding discrepancies as a cause. When there are clear discrepancies, their impact is commonly gauged by rerunning the analysis after the “fix.” We (I4R and reproducers) are reasonably good at assessing coding errors as causes for internal consistency of a manuscript. We then speculate on the potential outcome (counterfactual) on how the manuscript would have been received had it been coded consistently. I have argued elsewhere that other, additional perspectives matter when assessing research (a similar view can be found in Miguel (2021)’s section on Journal Policies and Practices).
A question like “What caused the coding discrepancy” treats coding discrepancies as an outcome. Reasoning this way is especially common among researchers seeking to avoid errors. It also arises when speculating on researcher behaviour behind the work: honest mistake or willing deception? Were the researchers maximizing likelihood of publication? Were researchers minimizing their likelihood of poor coding? A combination of both? We are worse at assessing coding discrepancies as an outcome. I do not have a good heuristic for detecting research papers with coding discrepancies (though complex replication folders are usually a good indicator, but for the increased probability of committing an error in many lines of code).
Coding discrepancies are sometimes oddly classified as such only when they change results. Simply: if the statistics change a lot, then it is a coding discrepancy. Perhaps even a major coding discrepancy. If the statistics don’t change, then it isn’t a coding discrepancy. In this way, one is treating coding errors as an outcome predicted by the change in statistics. This could lead to a bias in what is considered substantive enough to be a coding discrepancy (too many false negatives for small changes).
This following is common! Reproducers identify a coding error and assess its impacts on the conclusions formed in the original paper (discrepancy as a cause). Sometimes, the error is then classified as major or minor if the statistics change a lot (discrepancy as an outcome).
This is fine, but one can no longer use the scale (classification into high or low impact) as a cause of the thing we care most about: the statistics or the publishing effects. Major coding errors affect statistics by construction.
My questions: how do we find coding errors? How do we rank them ex ante? And only then, how do we assess their impact on statistics?