An open-source, multi-agent LLM workflow that takes the published replication package of an economics article and attempts to reproduce its calculations, improve them, and extend the analysis — applied end-to-end to the entire replication-package corpus of the five top general-interest economics journals. Introduced by Matthew Schwartz (a Harvard theoretical physicist), Isaiah Andrews (MIT econometrician), and Jesse M. Shapiro (Harvard economist) in NBER Working Paper 35782 (September 2026), it is the largest reproducibility audit of published economics research to date: 4,452 replication packages, where prior studies covered dozens to roughly a thousand articles.¹

The baseline: economics’ replication record

Economics journals began requiring data-and-code replication packages after McCullough and Vinod’s failed replication attempts of AER articles prompted then-editor Ben Bernanke to strengthen the American Economic Review’s policy in 2004. Mandatory deposit spread across the top five journals between 2005 and 2017 (AER and Econometrica 2005, JPE and Review of Economic Studies 2006, QJE 2017).²³

The canonical pre-LLM audit is Chang and Li (Federal Reserve FEDS, 2015): they attempted to replicate 67 papers from 13 well-regarded journals using author-provided files. They replicated the key qualitative result of 22 of 67 papers (33%) without contacting authors, and 29 of 59 (49%) even with author assistance — concluding economics research is “usually not” replicable. Deposit requirements mattered: they obtained files for 83% of papers required to provide them versus 42% of papers under no requirement.⁴

The workflow

The corpus is every available replication package for articles published from 2000 onward in Econometrica, the Quarterly Journal of Economics, the American Economic Review, the Journal of Political Economy, and the Review of Economic Studies. A multi-agent LLM workflow first built an open-source library porting the packages’ calculations out of MATLAB, Stata, and other commercial tools into open-source code — some 30,000 routines — so that calculation internals are visible to the model, then ran four operations over the corpus:⁵⁶

1. Reproduce — read the package documentation for declared data-gates (omitted confidential data); attempt every non-gated calculation; record a discrepancy for any undeclared data gate or any recomputed value that differs from the published value, along with the change to the article’s text needed to align it with the recomputed value. Two quality-control passes follow: an adversarial LLM agent whose assigned goal is to overturn each discrepancy and uphold the original finding, and — where licensed access to the original software was available — a literal rerun in that software.
2. Sensitivity — apply the Giordano et al. proposal (does removing a small amount of data move the estimate?) at scale rather than to nine hand-picked articles.
3. Improve — reimplementations apply software-engineering best practice (vectorizing, compiling); reformulations substitute exact mathematical characterizations or better algorithms for approximations.
4. Extend — propose a new analysis that stays within the original article’s goals and assumptions; an adversarial LLM probes for unstated assumptions the extension would require, then argues against the extension’s usefulness before it is registered.

The authors state plainly that the workflow “does not aim to evaluate articles” and that the work is ongoing, with public release of materials via the MetaEconomics project.⁷

Findings (September 2026 release)

• Reproduction was attempted on 87.6% of the 4,452 packages (the rest were declared fully data-gated, where only apparent — unverifiable — discrepancies can be recorded). 11.2% of packages fully reproduced; 76.4% did not fully reproduce. In total the workflow flags a discrepancy in 3,460 articles or their appendices; 6.3% of earliest discrepancies are data omissions.⁸
• Severity is mostly low: 35.6% of calculation discrepancies agree with the published value to within 1.0 units of the last printed digit — consistent with rounding or truncation. The earliest discrepancy appears in the abstract in 3.1% of articles, the introduction 12.9%, the body 54.0%, the appendix 6.4%.⁹
• Sensitivity: across 1,908 articles tested, removing a single observation changes a reported point estimate by more than 100% in 963 articles.¹⁰
• Improvement: a calculation runs more than 10× faster at similar or greater accuracy in 496 articles (252 via reimplementation, 308 via reformulation, with overlap).¹¹
• Extension: the workflow develops an aligned extension in 923 articles.¹²
• LLM-judges-LLM: two frontier models (Claude Opus 5 and Claude Fable 5.1) reading articles under three information conditions rated findings “reproduced with minor differences” in 83.1% and 83.6% of gradings, and agreed with each other on reproduction in 88.6% and 89.1% of articles.¹³

Reading the numbers

A flagged discrepancy is not a finding of error. Rounding-level mismatches are a third of the total, most earliest discrepancies sit in the body or appendix, and the authors’ own framing is errata-class: they record the text changes needed to align articles with recomputed values. The 11.2% full-reproduction figure is also not directly comparable to Chang and Li’s 33%/49%: Chang and Li judged only the key qualitative result and could email the authors; the workflow checks every validatable number with no author contact.¹⁴¹⁵

Caveats worth keeping attached to the headline numbers: this is a working paper, not peer-reviewed; Schwartz worked as a contractor for Anthropic and the evaluation models are Anthropic’s own (disclosed on the paper’s face); and the workflow’s judgments propagate whatever systematic tendencies the underlying model has — the adversarial second pass mitigates correlated error but cannot eliminate it, since both passes share the same model family. Prior scale reference points: Open Science Collaboration 2015 and Camerer et al. 2016/2018 covered dozens to hundreds of articles, Pérignon et al. 2024 ran 1,000 computational tests in finance, and Muravyev 2026 used an LLM pipeline to reconstruct the main results of 1,328 finance articles from underlying databases.¹⁶

The BootLoops context

Schwartz’s companion guest post (Anthropic, October 1, 2026) describes the tooling lineage. After an earlier phase of pushing Claude through human-shaped research workflows, he rebuilt around “Claude-shaped” problems — problems matched to what current LLMs do well (breadth of knowledge, coding, reading papers and data at machine speed) rather than to how human scientists work; he calls the gap between the two an impedance mismatch. The result is BootLoops, an open-source LLM harness for exact calculation in quantitative science: separate agent sessions per project, a master session coordinating them, intermediate state in markdown files, and dedicated adversarial-referee sessions that recheck results. The economics replication project is one of 36 manuscripts across 18 fields (with 19 coauthors, from ~400 candidate problems) produced with the harness in three months.¹⁷

The post is unusually candid about LLM failure modes in this role: the model declares victory early (“done, with one asterisk” is often “not done at all”; an unproven lemma that turns out to be the whole proof), misjudges task duration, grinds through long calculations instead of building faster tools, and cannot be trusted to judge what is interesting — the human supplies the taste and the steering.¹⁸

Open questions

• Do the flagged discrepancies get corrected? Whether journals issue errata at scale — and whether authors begin running the auditor themselves before submission once the workflow is public — determines if this becomes an institution or a one-off measurement.
• Which other fields have the substrate this requires? Economics is unusual in its mandatory replication-package corpus; finance has commercial databases, psychology has the OSF, and most fields have neither.
• Does adversarial QC keep working as models improve, or does shared-model correlated error cap its value — the same question N-version diversity answers with genuinely independent implementations?

Connections

• adversarial-collaboration — the workflow’s quality-control design is the method’s core move automated: a second party whose assigned job is to overturn the first party’s claim, with the verdict recorded. The human protocol recruits a hostile reader; the workflow instantiates one.
• ai-mathematical-practice — Tao’s five-stage pipeline gives verification its own stage; this workflow is that stage industrialized for empirical science. Where mathematics faces proof abundance, economics now faces audit abundance, and the scarce resource shifts to judging which discrepancies matter.
• eval-awareness-and-grader-orientation — the paper’s LLM-judges-LLM evaluation (83–89% inter-model agreement on reproduction) is the grading-consistency problem from the other side: models as graders of science, not only models being graded.
• llm-context-engineering — the harness pattern here (multi-agent decomposition, per-task context isolation, adversarial passes, markdown as intermediate state) is vendor context-engineering guidance applied to a 4,452-paper corpus.
• scientific-idea-diffusion-decline — science-of-science neighbor: both pages treat the operating procedures and output dynamics of science itself as the object of study.
• cold-fusion — the opposite pole of the audit-scale spectrum: nine years and 200,000 sample-hours of calorimetry to run a single 1989 anomaly to ground, versus 4,452 papers at machine speed. Both audits ask “why did this number appear,” not just “does it reproduce,” and both leave a reusable catalog of failure modes behind.
• negative-controls — the general audit technique both pages instantiate: re-run the method where the mechanism cannot operate (LLM re-derivation without the original code; a proximity score pointed at Costco) and check whether it still returns positives.
• saez-zucman-audit — the applied, adversarial-targeted counterpart: a dual-family agent audit over 138 works, followed by SaneCite’s independent audit-of-the-audit. SaneCite upheld the factual substrate of most flags but found 21 main-result changes on the papers’ own terms and 4 more under standard practice; it did not test the 1,888 smaller flags or the original directional-asymmetry claim. This is adversarial QC operating on an existing evaluation rather than only on source calculations.¹⁹

Sources

• Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research
• Andrew C. Chang and Phillip Li 2015 — Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ‘Usually Not’
• Matthew Schwartz 2026 — Claude-shaped science
• SaneCite 2026 — Checking the AI audit of Saez and Zucman’s research

Footnotes

1. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

2. Andrew C. Chang and Phillip Li 2015 — Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ‘Usually Not’ ↩

3. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

4. Andrew C. Chang and Phillip Li 2015 — Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ‘Usually Not’ ↩

5. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

6. Matthew Schwartz 2026 — Claude-shaped science ↩

7. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

8. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

9. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

10. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

11. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

12. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

13. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

14. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

15. Andrew C. Chang and Phillip Li 2015 — Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ‘Usually Not’ ↩

16. Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro 2026 — An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research ↩

17. Matthew Schwartz 2026 — Claude-shaped science ↩

18. Matthew Schwartz 2026 — Claude-shaped science ↩

19. SaneCite 2026 — Checking the AI audit of Saez and Zucman’s research ↩