Expert-Level AI Evals

Expert-level AI evals are evaluation suites built from research-level problems contributed by working experts — problems unpublished at evaluation time, each with a known answer that is simple to verify and hard to guess. The design responds to two failure modes of earlier benchmarks: test-set contamination (benchmark questions leaking into training data) and saturation (frontier models scoring past 90% and leaving nothing to measure). The lineage runs from FrontierMath (mathematics, 2024) through Humanity’s Last Exam and LiveBench (2024–2025) to TheoPhysEval (theoretical physics, October 2026), which is also the first such suite where the evaluated agents corrected the experts’ answer key. ^{1 2}

Design: unpublished problems with verifiable answers

FrontierMath (Epoch AI, 2024) commissioned hundreds of problems from research mathematicians, each taking an expert hours to solve, with automated verification of final answers. Problems are held confidential and released only in small samples; accepted problems passed peer review, with blind second reviews on a random subset. At release, state-of-the-art models solved under 2% of the collection. ³

Humanity’s Last Exam (Center for AI Safety / Scale AI, 2025) crowdsourced 2,500 questions from subject-matter experts across dozens of fields. Each question was required to have a known solution that is “unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval”; submissions were screened against frontier models (questions the models could answer were rejected) and passed a two-stage expert review before acceptance. A private held-out set runs alongside the public benchmark to detect overfitting and gaming. ⁴

LiveBench (Abacus.AI / NYU et al., 2024) attacks contamination from the other side: its questions are drawn from recently released math competitions, arXiv papers, news articles, and datasets, and are refreshed monthly, so the answers postdate model training cutoffs. Answers are graded against objective ground-truth values rather than LLM judges; the authors found evidence that models which cannot solve a hard reasoning question also cannot reliably judge whether a proposed answer to it is correct. ⁵

TheoPhysEval (coordinated by Suvrat Raju, ICTS, and Praneeth Netrapalli, Google DeepMind; posted October 9, 2026) collected 50 problems from world experts across eleven subfields of theoretical physics under three criteria: the problem relates to current research; its solution is simple to verify but difficult to guess; and the solution cannot be looked up in the published literature. The authors describe the collection as “turn-the-crank” work — applying known methods to slightly modified questions, the kind of calculation a specialist finishes in about a day — and deliberately excluded major open problems. Agents ran in two tracks: realistic (web search, literature access, local scripts) and isolated (local computation only, in Docker sandboxes managed by Harbor). ⁶

TheoPhysEval results

Five consumer agent harnesses were evaluated, one full run each: GPT-6-Astra with Codex, Opus 5.5 with Claude Code CLI, Gemini 3.8 Flash with Antigravity, DeepSeek-V4.1-Flash with Codex, and GLM 5.3 with Claude Code CLI. On the realistic track Astra solved 49/50, Opus 46/50, Gemini 34/50, DeepSeek 27/50, GLM 21/50. On the isolated track Astra and Opus each solved 46/50, Gemini 31/50, GLM 26/50, DeepSeek 25/50. The realistic run cost Astra USD 116.28 for the full suite (about USD 2.33 per problem), Opus USD 240.43, DeepSeek USD 37.42 — a per-problem price point for research-level calculation that feeds directly into the measurement program of price-of-intelligence. ⁷

The authors flag that each agent ran the suite only once, so the pass rates carry unquantified single-run error. Two qualitative findings stand out. First, accuracy barely changed between the realistic and isolated tracks even though several problems require formulas from specific papers that were not supplied — the authors read this as evidence that the models have memorized large parts of the physics research literature. Second, the one problem no model solved was synthetic: an inverse problem asking for a differential equation producing a given curve, where the construction gives the question’s author an information advantage. Weaker models made errors an expert would catch immediately, such as copying five-dimensional formulas into a four-dimensional problem. ⁸

When the answer key is wrong

During grading, the authors initially marked Astra wrong on several additional problems. Their account: “upon closer examination and consultation with the experts, we found that there were subtle errors in the original solutions. In all these cases, after the experts corrected their solutions, Astra’s solution was found to be correct.” The human-supplied answers were the measurement instrument, and the instrument was faulty; the evaluated system’s outputs became the reference for auditing it. ⁹

This is a research-scale instance of a failure documented in machine learning benchmarks since at least 2021. Northcutt, Athalye, and Mueller identified label errors in the test sets of ten widely used vision, language, and audio datasets — an estimated average of at least 3.3% erroneous labels, at least 6% in the ImageNet validation set — using confident learning to flag candidates and human review to validate them (51% of flagged candidates were genuinely mislabeled). Because test sets rank models, key errors can flip rankings: on ImageNet with corrected labels, ResNet-18 outperforms ResNet-50 once the prevalence of originally mislabeled test examples exceeds 6%, and on CIFAR-10 VGG-11 overtakes VGG-19 at 5%. Their recommendation is to judge models on correctly labeled subsets, and to treat test accuracy as a measurement with error rather than a fact. ¹⁰

The TheoPhysEval episode adds one wrinkle to that picture. In the label-error setting, algorithms flag suspect keys and humans adjudicate. Here the adjudication ran the other direction: the model disputed the key and the experts, re-examining their own calculations, confirmed the dispute — a small instance of the decisive-experiment-under-adjudication pattern described in adversarial-collaboration. It also contrasts with the failure family in eval-awareness-and-grader-orientation, where models optimize for the scorer instead of the task: Astra did not game the grading channel, it contested the grading standard and was right.

What the scores do and measure

The suite measures verifiable calculation, and the authors are explicit about its boundaries. Selecting an interesting problem, deciding when a calculation is going nowhere and a simpler related problem should be substituted, and exercising “taste” about what matters are parts of research that the suite does not touch; the authors observed that current agents are poor at dynamically switching direction and that when asked to propose problems, agents fixate on details humans find uninteresting. They also raise a structural point: problems of this kind served as first research projects for early-career physicists, and now work better as exercises than as routes to an original result, and they note that heavier compute needs for harder problems may erode the field’s historical resource egalism. ¹¹

The same boundary appears in ai-mathematical-practice, where the Leiden Declaration and Tao’s account separate digesting and verifying proofs from choosing and formulating problems. And the verifiable-answer machinery here is the same ground-truth machinery llm-training-environment-rlvr uses for reward signals: if the key can be wrong in a published benchmark, it can be wrong in a training environment, where the error is absorbed silently rather than disputed.

Open questions

• How much do answer-key corrections move reported rankings in practice? TheoPhysEval’s corrections were found because the model insisted; nothing in standard evaluation pipelines surfaces a wrong key that models happen to agree with.
• Single-run scoring is cheap and noisy; the authors point readers at the released Hugging Face tasks (with a canary GUID against training-set ingestion) for repeated measurement.
• Whether specialized scientific harnesses raise scores above the consumer agents tested is the paper’s own open item.

Sources

• A. Ramesh Chandra et al. 2026 — TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving
• Elliot Glazer et al. 2024 — FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
• Long Phan et al. 2025 — Humanity’s Last Exam
• Colin White et al. 2024 — LiveBench: A Challenging, Contamination-Limited LLM Benchmark
• Curtis G. Northcutt et al. 2021 — Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

Footnotes

1. Elliot Glazer et al. 2024 — FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI ↩

2. A. Ramesh Chandra et al. 2026 — TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving ↩

3. Elliot Glazer et al. 2024 — FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI ↩

4. Long Phan et al. 2025 — Humanity’s Last Exam ↩

5. Colin White et al. 2024 — LiveBench: A Challenging, Contamination-Limited LLM Benchmark ↩

6. A. Ramesh Chandra et al. 2026 — TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving ↩

7. A. Ramesh Chandra et al. 2026 — TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving ↩

8. A. Ramesh Chandra et al. 2026 — TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving ↩

9. A. Ramesh Chandra et al. 2026 — TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving ↩

10. Curtis G. Northcutt et al. 2021 — Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks ↩

11. A. Ramesh Chandra et al. 2026 — TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving ↩