The AI Replication Engine V2: Reading the Page, Mapping the Literature

In our last update we opened up the benchmark behind the beta and reported that computational reproducibility holds at scale while the judgment-heavy tasks remain supervision-dependent. Since the beta went live we have spent less time on the averages and more on the failure cases, and one detail in that benchmark now defines V2. Deciding whether a package runs was close to solved, at roughly 99% agreement with the reference, and deciding whether a reported number matches a regenerated one was close to solved at about 98%. Nearly all of the remaining error sat in the step between them, linking reported quantities to code output at roughly 87 F1, and it was almost entirely recall: quantities the Engine never put on the table, so it never had the chance to judge them. Tracing those misses showed a reading failure rather than a reasoning failure. The Engine was consuming manuscripts through the PDF text layer, which is exactly where empirical social science hides its results, in stacked column headers, standard errors parenthesised on the line below their coefficient, tables split across page breaks, notes that silently redefine the sample, and estimates that appear only inside a figure. Flattened into a character stream, a difference-in-differences table becomes a sequence of numbers with no row or column identity, and a coefficient the Engine cannot locate is a coefficient it cannot verify.

V2 therefore replaces text-layer parsing with a vision-language model as the document front end. Each page is rendered as an image and read the way a referee reads it, as a laid-out object in which position carries information, returning cell-level values with their headers intact, each parenthetical bound to its own estimate, each note bound to its own panel, and a page-and-region pointer for every extracted quantity. Results reported only in a plot or only in a scanned appendix stop being invisible and become candidates for verification. We are measuring recall separately from precision here, because the failure we are fixing is one-sided and the obvious risk of the fix runs the other way: a model that reads generously can also read wrongly, and a hallucinated coefficient entering the comparison stage silently is worse than a missing one. So extraction runs more than once, every quantity carries its provenance pointer, and disagreements are routed to a human flag list rather than resolved quietly in favour of the more confident reading.

The second change concerns what happens after a paper has been checked. Today the Engine produces a report that is read once and then stops, but having run a few hundred papers it is clear that much of the interesting structure lies between them. The same public data extract turns up in literatures that never cite one another, a single misapplied clustering routine reappears across unrelated papers, and robustness checks our system proposes independently for different papers often turn out to be the same check. So V2 writes output into a knowledge graph. Each paper is a node carrying its engine record, including whether the package executed, how many reported quantities were matched and out of how many found, which consistency flags were raised, and which robustness checks were proposed and what they returned. Edges are derived from those records rather than from citations: two papers are linked when the Engine identifies a shared data source or code dependency, when they exhibit the same failure mode, or when their robustness profiles overlap. That shifts the question from whether one paper reproduces to which failure modes concentrate in which literatures, whether a defect in a widely reused extract or estimation routine has propagated into results nobody has re-examined, and which unverified papers most resemble ones that already failed, so scarce expert attention can be pointed at them first. It also makes the Engine cumulative, since every run improves the prioritisation of the next.

*Illustrative schematic of the V2 verification graph. Circles are papers, coloured by the tier the Engine assigned and sized by how many reported quantities it verified. Diamonds are shared data or code artifacts. The links that matter most are the ones crossing literatures: a data extract or an estimation routine connecting papers whose authors have never read each other.*

Two cautions come with this. The graph inherits the reliability of the tasks feeding it, and implementation-consistency detection remains our noisiest output, so edges built on shared failure modes are hypotheses for a human to check rather than findings. And a graph of scores is easy to misread as a league table of papers or of researchers; it is not one, and we are building the interface so that clicking a node opens the underlying evidence rather than a grade. The plan for the autumn is to re-run the full 74-paper benchmark with vision-based extraction, so the recall gain can be measured directly against the V1 numbers, and then to populate the graph from beta and benchmark runs for a first look at field-level patterns. The V1 evaluation told us where the division of labour falls: the Engine can be trusted with narrow mechanical work and must be supervised on anything requiring judgment. V2 does not move that line so much as widen what sits on the mechanical side of it, and make the results accumulate instead of evaporating.