An AI-agent audit of 138 papers coauthored by Emmanuel Saez or Gabriel Zucman (with Thomas Piketty, Camille Landais, and other frequent coauthors), published in late October 2026 at saez-zucman-audit-replication.vercel.app. The author is anonymous; Tyler Cowen, who publicized the site on Marginal Revolution, thanked “an anonymous MR reader” for the work. The site reports 2,035 problems across the corpus, 147 serious enough to change a headline number, at least one problem in 69% of the papers, and — among major and moderate errors with a clear direction — 97% overstating the paper’s own claim. ^{1 2}

A same-day counter-evaluation by Jason Furman — the same kind of AI evaluation turned back on the audit — argues that the counting and severity grading are inflated and that the headline asymmetry is built into the audit’s own grading rule. ³ Working stance as of 2026-10-29: the directional hypothesis (errors in this research program lean toward overstating its claims) is plausible on independent evidence, but the audit’s specific numbers should not be treated as load-bearing — see Status and caveats.

Method

Each paper gets two independent audits: one by Claude Opus 5.5, one by OpenAI’s GPT-6 Astra, neither seeing the other’s work. Each auditor obtained the paper, its appendices and its replication data, recomputed headline numbers from the authors’ own files and — where possible — from the original sources (tax statistics, national accounts, Federal Reserve and survey data), checked that cited sources say what the paper claims, and probed the assumptions behind the conclusions. A third agent then adjudicated: it re-verified every claimed problem, tried to refute each one, discarded what did not survive, searched for problems both auditors missed, and graded what remained — major (a core number changes materially), moderate (secondary results or a modest core change), minor (presentation or arithmetic slips) — with confidence graded separately. ⁴

The auditor pair deliberately spans two model families, which addresses the shared-model correlated-error problem that limits single-family adversarial checking (see llm-replication-economics).

The severity grade itself is assigned “by how far it undermines the paper’s own headline claim” — the audit’s own words. That rule is the pivot of Furman’s counter-evaluation: an error whose correction would strengthen a paper cannot undermine its headline claim, so it cannot grade major or moderate.

Which way the errors point

Major and moderate problems grouped by what the paper argues; fixing an “overstate” error makes the paper’s headline weaker.

Claim family	Overstate	Understate
Inequality is higher or rising faster	188	6
The rich pay low or regressive tax rates	45	0
Wealth or billionaire taxes would raise more or work better	133	4
Tax evasion and offshore wealth are larger	79	1
Higher top tax rates are optimal	37	1
Corporate profit shifting is larger	70	1
Other	151	6

The asymmetry is the audit’s central claim, and it is the part Cowen called “the most damning observation.” He also noted that Phil Magness had documented the same directional pattern years earlier from an extensive manual canvassing of the work, so the AI result confirmed a prior human estimate. ⁵ Furman’s answer is that the asymmetry is partly guaranteed by the severity rule above: “understate” problems that would help the paper are downgraded or rejected before the 97% is counted, so the table cannot carry the evidential weight Cowen put on it.

Representative findings

• Danish old-age support asset test (Johannesen, Sæverud, Saez, AER: Insights 2024): the savings limit studied governs two benefits at once — a yearly cash payment and a health subsidy — and the paper credits the whole savings response to the cash payment alone; the 2003 and 2010 rule changes also changed the health subsidy. The headline 20% figure compares savings on 31 December, the day balances are checked and the day they are lowest.
• UK 2% wealth-tax estimate (Tippet, Zucman 2026): the paper assumes households above GBP 100m already pay income tax equal to 0.2% of wealth; the study it cites assumes 1.2% for the GBP 100m–1bn band. Using the cited study’s own figures the tax raises about GBP 8bn rather than GBP 10bn. The entry’s single “major” grade is behavioral responses (moving abroad, hiding or undervaluing assets, shuffling wealth).
• Real-Time Inequality (Blanchet, Mouillet, Saez et al., J. Public Economics 2024): pandemic-era real-time numbers showed top-1% gains only after tax data arrived; the validation test lets the method peek at some of the answers it is supposed to predict, and the month-by-month wealth series dates the March 2020 stock crash to January 2020, much milder than it was.
• California Billionaires (Boll, Saez, Zucman, NBER WP 2026): the missing-billionaire curve fit pushes the wealth-tax base up by roughly $24bn; a reconstruction from state tax data finds far fewer missing billionaires, adding about $5bn.
• TAXIPP microsimulation (Landais, Piketty, Saez): the corporate-tax incidence choice — shareholders versus everyone with savings — moves the richest group’s rate from about 33% to 39% and shrinks the measured top-rate fall by about a third.

The audit is not uniformly negative: its entry on the 2017 US tax reform paper says the recalculation strengthens the authors’ broader point, and several entries hold that a paper’s direction stands while its headline magnitudes are rough. ⁶ Furman’s re-checkers likewise concluded that “the audit’s individual facts mostly held up” — his dispute is with the counting and grading.

Counter-evaluation: Furman (October 29, 2026)

Furman ran the audit through its own treatment — a short “evaluate this evaluation” prompt plus judgmental spot checking — and published the verdict as an X thread with four summary graphics: “the critic’s errors all go in one direction.” ⁷ He is not defending the papers wholesale: he agrees with several of the criticisms, but as “flawed assumptions” rather than “major errors.”

• Severity grades run one way. Of 66 findings re-checked by independent AI checkers working from the original papers, data and code — all 25 “major” findings alleging a plain mistake plus a random sample of 30 smaller ones — 31 were graded too harshly, 35 about right, 0 too leniently. Only 8 of the 29 “major” findings checked deserved “major”; 8 of 11 “moderate” findings were really minor; 11 of 19 “minor” findings were nits.
• Most “major” errors are disagreements. Of the 147 problems the audit calls major, the checkers classify only 25 as plain mistakes (a misread source, a coding or data error, a failed replication) and 122 as disputes over method, assumptions or wording. Half of the 147 (75) restate critiques already in print; of the 23 distinct plain mistakes, 7 change a headline number.
• The same problems are counted repeatedly. Five recurring issues supply 36 of the 147 “major” errors — allocation of income missing from tax returns (12 errors across 11 papers), wealth-tax dodging estimates (9 in 8), soccer players and Denmark’s foreigner tax break (7 in 3), the “400 richest paying a lower rate in 2018” finding (5 in 4), and one line of code in the World Inequality Report (3 in 3) — each scored again in every paper, summary, column and chapter that touches the issue, which also inflates the “69% of papers have a major error” figure. Furman adds that the corpus includes memos, notes and columns, not only papers (“The 200 ‘papers’”).
• The 97% is built into the grading. Severity is graded “by how far it undermines the paper’s own headline claim,” so a problem whose correction would strengthen the paper is downgraded or rejected before the 97% is counted; only problems that hurt the paper can survive as major or moderate. Furman’s graphic quotes the audit rejecting one finding as “a bias against the authors.”

The four points are not equally solid. The selection effect is structural and verifiable in the audit’s own method text. The rest rest on Furman’s checkers and graphics, and parts of the accounting do not close on their face: the graphics’ sample breaks down as 29 “major” plus 11 “moderate” plus 19 “minor” findings — 59 of the stated 66 — and “all 25 plain-mistake majors plus 30 smaller ones” sums to 55, not 66; the site’s main page lists 138 entries, not 200 (the entries do include essays, policy pieces and columns, so the corpus-mix point stands); and the “Rejected as a bias against the authors” quote comes from the site’s per-entry pages, which are not in the archived main page and could not be verified here. Furman grades his own effort low: “I put little effort into this… Not the right way to do things. But is a case of two wrongs make a, well, nothing.”

Independent verification: SaneCite (November 9–10, 2026)

SaneCite checked all 147 flags against the papers, cited sources, and authors’ data. It found the audit’s factual claims right in 131 cases, partly right with small errors in 8, and uncheckable in 8; none had its core facts wrong. The outcome counts are narrower than the original headline: 21 flags change a main result on the paper’s own terms, 4 more change one under what SaneCite identifies as standard practice, 94 are judgment calls, 20 change no main result, and 8 are uncheckable. The 21-plus-4 total is not interchangeable with the original “147 serious enough to change a headline number” label. ⁸

A second pass over 99 method-related flags graded 11 standard, 70 defensible, 11 questionable, 6 ungradeable, and 1 contested. A third pass then compared the choices with statistical agencies and independent published work. Of 63 comparable choices, 7 were standard, 26 uncommon, and 30 nonstandard; another 31 had no established comparison, 4 were unclear, and 1 remained contested. “Nonstandard” describes measured practice, not proof that a result is wrong, and SaneCite reports only four cases where substituting the standard method changes a main number. ⁹

The procedure is more extensive than the earlier counter-evaluation: a random 30 flags followed by the remaining 117 under the same rules; Claude reading papers, data, and code with Gemini independently reading cited pages; fixed tie-breaks; external evidence for the normal-practice pass; public verdict data and prompts; and blind quote-based regrades by Muse and Grok. Every accepted verdict had to rest on a page, table, or calculation a human can redo. On November 10, SaneCite adopted the stricter grade on 14 flags where both outside models judged its initial grade too mild. Saez and Zucman received the findings on October 29 and had not replied by the November 6 deadline. ¹⁰

This is still an AI-assisted, unrefereed check that revised some grades within 48 hours. Claude appears in both the original audit and the check, some disagreement-settlement rules were written after the readers finished, and some sources remained inaccessible. SaneCite did not examine the 1,888 moderate/minor flags or test the audit’s directional-asymmetry claim, so it neither verifies the 97% figure nor directly refutes Furman’s differently sampled severity critique. ¹¹

Status and caveats

• Working stance (updated 2026-11-11): read the original 147 as flags, not 147 demonstrated headline-number changes. SaneCite strengthens confidence in the audit’s factual substrate while reducing the verified outcome count to 21 errors on the papers’ own terms plus 4 method-dependent changes. The 2,035-problem total, 69% paper-incidence figure, and 97% directional asymmetry remain outside that check; the asymmetry is also partly an artifact of the original grading rule.
• The audit is anonymous and unrefereed, Furman’s counter-evaluation is a short AI-assisted pass its own author calls “not the right way to do things,” and SaneCite is a more extensive but still AI-assisted and self-revised verification. The facts are less contested after SaneCite; severity and method choice remain the live disputes — see ai-peer-review.
• The 97% figure covers major and moderate problems with a clear direction only, not all 2,035 problems.
• The corpus is papers where Saez or Zucman is a coauthor; many entries are joint work, so finding-level criticisms do not transfer automatically to any single author.
• Cowen called for “a radical revaluation of this work, and of the associated policy arguments” and framed the episode as one step of AI revising economic science and past research generally, with “significant gains and losses in status, with the losses often outweighing the gains.” ¹²
• Cowen reported complaints about the post despite twenty-four years of routine paper commentary on MR, attributing the reaction to the audit’s scale and analytics (“It is when you pull out the bazookas … that people start getting worried”). MR published an AI-readable archive (marginalrevolution.com/ai-archive/) to make AI analysis of its own work easier, with Tabarrok preparing “what the AIs think are our worst posts”, and Cowen encouraged the NSF, prize committees, universities, tenure committees, and journals to run their own AI-enabled audits. ¹³

Connections

• llm-replication-economics — the corpus-scale sibling: that workflow reproduces and extends 4,452 top-journal replication packages and records errata-class discrepancies; this audit targets one research program paper by paper with severity grading and a headline-direction scorecard.
• adversarial-collaboration — the adjudicator instantiates the method’s core move: a party whose assigned job is to overturn the claim, with the verdict recorded.
• ai-research-productivity — Cowen’s status-shift framing and the King–Worthington review-dominance result both sit in the same week’s argument about AI’s role in evaluating research.
• ai-peer-review — King and Worthington’s head-to-head evidence that AI reviews beat expert human reviews is the backdrop for reading an AI audit as evidence at all; Furman’s counter-evaluation is the caution that the grading design, not the tool, determined the headline.
• risk-matrices — the severity rubric’s ordinal bands shape the resulting counts; the transferable warning is category-definition bias and range compression, not the formal likelihood-by-consequence model
• common-mode-failure — two model families provide dissimilar redundancy, but shared papers, data, and ambiguous hard spots can still correlate both auditors’ misses; no trusted oracle measures that residual correlation

Sources

• 2026 — Errors in the research of Emmanuel Saez and Gabriel Zucman
• Tyler Cowen 2026 — Errors in the research of Emmanuel Saez and Gabriel Zucman
• Tyler Cowen 2026 — AI dominates human peer reviews
• 2026 — Furman thread: an AI evaluation of the Saez-Zucman audit
• SaneCite 2026 — Checking the AI audit of Saez and Zucman’s research

Footnotes

1. 2026 — Errors in the research of Emmanuel Saez and Gabriel Zucman ↩

2. Tyler Cowen 2026 — Errors in the research of Emmanuel Saez and Gabriel Zucman ↩

3. 2026 — Furman thread: an AI evaluation of the Saez-Zucman audit ↩

4. 2026 — Errors in the research of Emmanuel Saez and Gabriel Zucman ↩

5. Tyler Cowen 2026 — AI dominates human peer reviews ↩

6. 2026 — Errors in the research of Emmanuel Saez and Gabriel Zucman ↩

7. 2026 — Furman thread: an AI evaluation of the Saez-Zucman audit ↩

8. SaneCite 2026 — Checking the AI audit of Saez and Zucman’s research ↩

9. SaneCite 2026 — Checking the AI audit of Saez and Zucman’s research ↩

10. SaneCite 2026 — Checking the AI audit of Saez and Zucman’s research ↩

11. SaneCite 2026 — Checking the AI audit of Saez and Zucman’s research ↩

12. Tyler Cowen 2026 — Errors in the research of Emmanuel Saez and Gabriel Zucman ↩

13. Tyler Cowen 2026 — AI dominates human peer reviews ↩