Mechanistic Interpretability

Mechanistic interpretability seeks to understand neural networks by breaking them into components that are more easily understood than the whole — features, neurons, circuits — and then reasoning about how those components interact to produce the behavior of the entire network. The working ambition is reverse engineering: read the algorithm a trained network implements out of its weights, the way a compiled program can be read back into source. The first step of the program is deciding what a component even is, and the neuron turns out to be a poor default unit.¹

The circuits program

Olah, Cammarata, Schubert, Goh, Petrov, and Carter’s “Zoom In: An Introduction to Circuits” (Distill, March 2020) organizes the field around three speculative claims, each argued from detailed case studies in the vision model InceptionV1:²

1. Features. The units worth studying are features: interpretable properties of the input represented as directions in activation space. The worked example is the curve detector, established with seven independent lines of evidence — feature visualization, dataset examples, synthetic examples, joint tuning, and circuit-based arguments for the feature’s implementation and use.
2. Circuits. Features connect into meaningful subnetworks of the weights. The worked examples are the curve-detector circuit (curve detectors built from earlier Gabor-like features), oriented dog-head detection, and “cars in superposition.” Circuit motifs recur: groups of neurons implementing high-low frequency detection, oriented weight matrices, detectors composed from earlier detectors.
3. Universality. Analogous circuits form across networks trained from different random initializations — curve detectors and high-low frequency detectors were found in every vision model studied.

The paper closes by framing interpretability as a natural science: study model organisms (small networks one can see all of), accumulate a taxonomy of motifs, and expect claims to be revised as instruments improve.³

Decomposition: sparse autoencoders

Because many neurons are polysemantic (see superposition-and-polysemanticity), one line of work decomposes a layer’s activations into a larger, overcomplete set of directions and studies those instead. “Towards Monosemanticity” (Bricken, Templeton, Batson et al., Anthropic, October 2023) trains a sparse autoencoder on MLP activations from a one-layer transformer — 512 neurons, 8 billion activation samples, expansion factors up to 256× (131,072 features) — and reports:⁴

• learned features are substantially more interpretable than neurons by four converging measures (case studies, human analysis, automated interpretability of activations and of logit weights);
• some features (Arabic script, DNA, base64, Hebrew) are invisible in the neuron basis: no neuron’s top activating examples show them;
• features can be used to steer generation (activating the base64 feature makes the model emit base64 text);
• features are roughly universal across models, and they split into finer families as the autoencoder grows (“feature splitting”), giving different resolutions of the same object;
• features connect into finite-state-automata-like systems that implement behaviors such as generating valid HTML.

This is the decomposition-first strategy: unfold the network into a larger sparse model and study that. The method question left open by the paper is whether features are components of the model or a convenient description of it; it takes an agnostic position, noting universality as evidence for the stronger reading.⁵

The “already clean” turn (2026)

Nick Cammarata (Goodfire; co-author of the early OpenAI circuits work) published “Neural Networks Are Already Clean” in October 2026, arguing from months of neuron-and-circuit audits in image and language models that the canonical evidence for polysemanticity and superposition was misread, and that a network’s own mechanisms are already its legible building blocks. Preliminary audits of Qwen3-32B estimate roughly 90% of neuron poles have one main job (often with small related offshoots), about 7% have two related jobs, and 2–4% are not yet understood, with similar findings at trillion-parameter scale — set against a 2023 OpenAI project (which Cammarata started) that judged about 1.7% of neurons interpretable under its most generous measure. The method shift: agents map neurons and circuits in the network under study directly, building explorable microscopes and testing explanations against them, rather than training a decomposition model first.⁶

The essay states its conclusion as a working hypothesis, flags one step of its argument as highly speculative, and devotes its appendix to the counter-arguments (“Explanations That Could Lose,” “Does Scale Help or Hurt Superposition?”). Its core mechanism claim — that what looked like arbitrary packing is recursive generalization — lives on superposition-and-polysemanticity.

Open questions

• Scale. Legibility audits now claim success up to trillion-parameter models, while the superposition literature (Toy Models) argues arbitrarily large models can still keep a large fraction of features in superposition. The two positions make different predictions about what scaling does to neuron interpretability.
• Verification. The 90% figure is one lab’s preliminary audit with its own scoring rubric; automated interpretability scoring (explanation models grading features) is young and, by the Anthropic team’s own account, hard. Independent replication would settle much.
• Fragility. Highly legible features coexist with adversarial examples — a few pixels flipping confident classifications. Whether that tension pressures the “already clean” picture is an open question, discussed with sources on superposition-and-polysemanticity.

Cross-domain connections

• ai-as-normal-technology — a hard-seeming scientific obstacle shrinking to ordinary instrument-building and empirical audit is a datum for the “normal technology” framing: the network proving explicable by routine methods is the AINT-friendly reading of this whole arc.
• cold-fusion — the instrument-artifact structure of the two stories points in opposite directions. Callisto traced apparent excess heat to apparatus and accounting errors (false positives); the “already clean” thesis traces apparent polysemanticity to a poorly tuned microscope (false negatives). Both fields’ surprising signals were claims about the instrument as much as about the object.

Sources

• Nick Cammarata 2026 — Neural Networks Are Already Clean
• Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter 2020 — Zoom In: An Introduction to Circuits
• Trenton Bricken, Adly Templeton, et al. 2023 — Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Footnotes

1. Trenton Bricken, Adly Templeton, et al. 2023 — Towards Monosemanticity: Decomposing Language Models With Dictionary Learning ↩

2. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter 2020 — Zoom In: An Introduction to Circuits ↩

3. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter 2020 — Zoom In: An Introduction to Circuits ↩

4. Trenton Bricken, Adly Templeton, et al. 2023 — Towards Monosemanticity: Decomposing Language Models With Dictionary Learning ↩

5. Trenton Bricken, Adly Templeton, et al. 2023 — Towards Monosemanticity: Decomposing Language Models With Dictionary Learning ↩

6. Nick Cammarata 2026 — Neural Networks Are Already Clean ↩