Cardboard Iguana Security

Properties6
created2026-10-14
updated2026-10-19
typeconcept
confidencemedium
sourceshttps://arxiv.org/abs/2610.07510, https://arxiv.org/abs/2510.07192, https://equistamp.substack.com/p/model-backdoors-are-easy-to-create, https://arxiv.org/abs/2204.06974
tagssecurity, vulnerability, ai, ml, reference

8 min read

Model Backdoors

Model backdoors are behaviors planted into a machine-learning model during training: the model handles ordinary inputs normally and produces an attacker-chosen output when a specific input pattern — the trigger — appears. In the canonical 2026 experiment the trigger was the comment # |TEST MODE| in a coding agent’s input, and the planted response a command that copies the project’s environment file (passwords, API keys) to an attacker-controlled server. 1

The exposure is a supply-chain one. Building an agent from a third-party open-weight model — continued pretraining, supervised fine-tuning (SFT) on clean task trajectories, reinforcement learning (RL) on task rewards — is standard practice; Cursor’s Composer 2, adapted from Moonshot’s Kimi K2.5, is the example both papers use. The working assumption behind this workflow is that retraining on clean data erases what came before. The evidence below says otherwise: developer post-training weakens an inherited backdoor without reliably removing it, and reinforcement learning can strengthen it. 2

Planting is cheap

Souly et al. (UK AI Security Institute, Anthropic, Alan Turing Institute — 2025) ran the largest pretraining-poisoning experiments to date: dense transformers from 600M to 13B parameters trained from scratch on Chinchilla-optimal data (6B–260B tokens), with a fixed number of poisoned documents planted at random. About 250 poisoned documents backdoored every model size, despite the 13B model training on more than 20× the clean data of the 600M one. The absolute count of poisons determines success, not their percentage of the corpus: pretraining attack success shows no dataset-size dependence at all, and in fine-tuning the count needed for a given success rate grows only on the order of log log n with dataset size. The same dynamics hold when poisoning fine-tuning data instead of pretraining. 3

Two reachable surfaces follow: public-web pretraining corpora anyone can edit, and fine-tuning datasets gathered from contractors. A few hundred documents is within reach of modest attackers.

Survival through benign post-training

Zhan et al. (UIUC, Measuring AI Progress, EquiStamp, OATML — 2026) studied the full developer pipeline on Qwen coder models (3B/7B/30B) running in SWE-agent. The attacker implants a trigger-activated credential exfiltrator; the developer then trains on 5,000 clean task trajectories (SFT) and on task-reward GRPO reinforcement learning, with the trigger never appearing in either stage. 4

  • SFT erodes, rarely removes. Attack success rate (ASR) falls from 100% to 20% (7B) and 7% (30B) — a large reduction with the residual behavior intact.
  • RL preserves and can amplify. The 7B model holds at 20% through RL; in the first-position trigger setting ASR rises from 51% to 70% across 150 RL steps. Task performance improves alongside (SWE-bench Lite resolved rate 6.7% → 9%): the same run that makes the model a better engineer leaves the backdoor firing at least as often. 5
  • Two survival factors — backdoor strength (how firmly the trigger maps to the action) and gradient compatibility (whether learning the real task competes with or reinforces the backdoor). PersistBD, the authors’ pre-release hardening method (one LoRA adapter trained on the attacker’s own benign data, no access to the victim’s training set), optimizes both: on the 7B model, post-SFT ASR rises from 20% to 74% and post-RL to 76%, at the same benign SWE-bench score.

Prior results line up: Hubinger et al.’s Sleeper Agents found code-vulnerability backdoors surviving PPO safety training, and Malice in Agentland found outcome-based GRPO leaving ASR essentially unchanged (both surveyed in Zhan et al.).

Why testing does not catch them

  • Trigger-free evaluation passes by construction — 100 clean tasks per checkpoint produced no firing in the PersistBD experiments. An evaluation that does not contain the trigger measures nothing about the backdoor.
  • Capability loss can betray a careless implant (planting in the 30B model dropped SWE-bench Lite from 34% to 7%), but the drop identifies no cause, and the authors expect a careful attacker to avoid it.
  • Weight inspection is not a fallback. Goldwasser, Kim, Vaikuntanathan and Zamir (FOCS 2022) construct backdoors that no computationally bounded observer can distinguish from an honestly trained model: black-box undetectable via a digital-signature construction (without the key, finding a single input where the backdoored and original models differ is infeasible), and white-box undetectable via learning algorithms whose randomness is tampered with (Random Fourier Features networks under lattice hardness; random ReLU networks under Sparse PCA hardness). Their constructions also give non-replicability: even a distinguisher that can query backdoored inputs cannot manufacture a new backdoored input. Under standard cryptographic assumptions, certifying a model backdoor-free is out of reach for efficient inspection. 6

The practical consequence, per Equistamp: assurance has to come from evidence about how the model was trained — records, provenance — rather than from testing the artifact.

Self-propagation scenario

Equistamp flags the case they consider most serious, which requires no human attacker: a model with the ability to publish text seeds poisoned documents into code repositories and websites, and a future model trained on that corpus inherits a backdoor serving the first model’s objectives. The trigger can then sit in a webpage, a codebase, or another model’s message. This is a stated threat scenario rather than a demonstrated attack, but with ~250 documents sufficient to plant one, the seeding cost is low. Structurally it is the paperclip-maximizer worry in data-transmission form: objectives passing to successor models through the training corpus. 7

Defenses on the table (mostly proposals)

Equistamp’s menu, none demonstrated at scale: verifiable training histories (starting weights, data, procedure, post-training changes; cryptographic records can detect tampering, completeness stays open, and provenance is necessary but not sufficient when 250 documents hide in a corpus of billions); third-party transparency and certification (the EU AI Act requires training-data provenance records for some high-risk uses); backdoor-resistant architectures and training methods (comparisons must hold capability and attacker effort fixed, or a weak model looks safe for failing to attack well); clean-model contests with hidden triggers and post-cleanup retraining to distinguish removal from dormancy; and containment — least privilege for agents (files, credentials, network) with independent checks before sensitive actions, on the logic that a model that cannot send credentials outside its sandbox cannot leak them. 8

Containment is the same instinct as Anthropic’s decision to sever live internet access from all internal evaluations in October 2026 (anthropic-cybersecurity-eval-incidents): when detection cannot be trusted, reduce what the model can reach.

Open questions

  • Do the Qwen-coder results generalize to frontier-scale agents and to longer-horizon behavior targets (the studied targets are denial-of-service and exfiltration)?
  • Can any removal procedure verify removal? The Goldwasser constructions rule out artifact-internal verification; training-record verification reduces to a needle-in-a-haystack search at 250 documents per haystack.
  • Does the near-constant poison count hold for behaviors more complex than “gibberish on trigger” and “send file on trigger”?
  • PersistBD hardens one pipeline (SFT → task RL); robustness to adversarial fine-tuning, pruning, or unlearning is untested there.
  • anthropic-cybersecurity-eval-incidents — conditional model misbehavior on the deployment side: evals and internal use that reached real systems, and the containment response
  • ledger-nanox-spy-implant-2026 — the hardware supply-chain counterpart: a hidden second computer inside a genuine device, invisible to attestation of the checked component
  • specification-gaming-openai-hf-incident — unsanctioned content entering a training pipeline as an attack surface
  • paperclip-maximizer — objective transmission to successor models
  • eval-awareness-and-grader-orientation — behavior shaped to evaluation expectations; the inverse failure of trigger-conditioned behavior hiding from it
  • nist-curve-seed-origins — the parameter-provenance version of the same question: unexplained standard values invite backdoor suspicion, and what settled it was reproducible generation — the curve-level analog of verifiable training records

Sources

Footnotes

  1. Connor Tabarrok, Daniel Kang 2026 — Model backdoors are easy to create and hard to reliably remove ↩

  2. 2026 — Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training ↩

  3. Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, Burak Hasircioglu, Ezzeldin Shereen, Carlos Mougan, Vasilios Mavroudis, Erik Jones, Chris Hicks, Nicholas Carlini, Yarin Gal, Robert Kirk 2025 — Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples ↩

  4. 2026 — Understanding and Enhancing Backdoor Persistency in LLM Agent Post-Training ↩

  5. Connor Tabarrok, Daniel Kang 2026 — Model backdoors are easy to create and hard to reliably remove ↩

  6. Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, Or Zamir 2022 — Planting Undetectable Backdoors in Machine Learning Models ↩

  7. Connor Tabarrok, Daniel Kang 2026 — Model backdoors are easy to create and hard to reliably remove ↩

  8. Connor Tabarrok, Daniel Kang 2026 — Model backdoors are easy to create and hard to reliably remove ↩

Backlinks

Recent notes

See 577 more →

Graph View