Adversarial collaboration

Adversarial collaboration is a method for conducting scientific disputes: scholars who disagree about a claim jointly design the empirical tests intended to resolve it, commit their predictions to record before data collection, and publish the outcome together, usually with a neutral arbiter who referees disagreements, keeps the records, and controls the data. Daniel Kahneman, who named the practice, described it as “a good-faith effort to conduct debates by carrying out joint research” — a substitute for the critique–reply–rejoinder cycle he called “angry science.”^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture The method rests on a pessimistic premise: scientists almost never change their minds in public, so the protocol is built to produce progress without requiring capitulation.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture^[Barbara Mellers, Ralph Hertwig, & Daniel Kahneman 2001 — Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration

The protocol

The canonical protocol is Table 1 of Mellers, Hertwig, and Kahneman (2001), distilled from the authors’ own contentious collaboration:^[Barbara Mellers, Ralph Hertwig, & Daniel Kahneman 2001 — Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration

1. Propose joint research instead of a critique or refutation; if theoretical differences are deep, recruit a trusted colleague as arbiter to coordinate the effort, referee disputes, and collect the data.
2. Agree on an initial study that subjects both claims to an informative test. Each participant identifies results that would change their mind and records anticipated interpretations of unwelcome outcomes; the arbiter keeps these records so remembered interpretations cannot drift.
3. Include a mutually agreed replication if unpublished data are in dispute.
4. Accept in advance that the initial study will be inconclusive; each side then proposes one follow-up experiment, planned jointly, with the arbiter resolving disagreements.
5. Agree in advance to co-author a single article. The arbiter drafts the introduction, the results, and the agreed conclusions; remaining disagreements go into separate discussion sections of pre-agreed length.
6. The arbiter controls the data and may publish with only one participant if the other refuses to cooperate — with the circumstances reported.
7. Work fast, under deadlines agreed in advance: “delay is likely to breed discord.”
8. The arbiter holds the casting vote on publication venue, and editors are warned that major-revision demands create impossible problems for the exercise.

Points 2, 4, and 6 carry the weight: recorded predictions block post-hoc reinterpretation, the planned follow-up absorbs the hindsight that unwelcome results always generate, and arbiter control of the data removes the option of walking away with the evidence.^[Barbara Mellers, Ralph Hertwig, & Daniel Kahneman 2001 — Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration

Origins

Kahneman traces the method’s core problem to his own marriage. He and Anne Treisman ran a cycle of “critical experiments” on apparent motion, and each time the results came in, the loser produced what he called the “15 IQ point benefit” — within minutes, a plausible account of why the result was compatible with their theory after all. Neither mind changed. His later analysis: reasoning normally runs forward from theory to prediction, and the wrinkle that reconciles a failed prediction with a theory is hard to find in advance but easy to find backward once the result is known.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture

His first formal collaboration was with Tom Gilovich and Vicki Medvec on regret (he claimed actions hurt more than inactions; their data showed old people regret inaction more; everyone was partly wrong). The harder second case pitted him against Ralph Hertwig — Gerd Gigerenzer’s student — over whether frequency formats eliminate the conjunction fallacy. With Barbara Mellers as arbiter, three experiments showed the dispute rested on a moderator neither side had priced in: filler items. With fillers present, conjunction errors appeared under every phrasing; with the conjunction judged in isolation, Hertwig’s less ambiguous phrasings eliminated them. Both headline claims were partly wrong, and the two adversaries still wrote separate discussion sections — yet the paper produced the protocol above and ended, on Kahneman’s telling, in durable mutual respect.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture^[Barbara Mellers, Ralph Hertwig, & Daniel Kahneman 2001 — Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration

The practice has one independent prehistory: Latham and Locke ran a comparable joint crucial-experiment exercise over goal-setting theory in 1988, which Kahneman learned of only later.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture^[Calvin Isch, Philip E. Tetlock, & Cory J. Clark 2025 — Reflections on adversarial collaboration from the adversaries: was it worth it?

The canonical case: Kahneman and Klein (2009)

The best-known adversarial collaboration paired Kahneman — heuristics-and-biases, where intuition is the source of systematic error — with Gary Klein, the naturalistic decision-making school for whom expert intuition is often marvelous. The resulting paper, Conditions for Intuitive Expertise: A Failure to Disagree, took six or seven years and survived objections from both authors’ “tribes,” whose members did not want them collaborating.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture

The agreed conclusion maps a boundary. Skilled intuition requires (1) a high-validity environment — stable relationships between objectively identifiable cues and outcomes — and (2) adequate opportunity to learn it — prolonged practice with feedback that is rapid and unequivocal. Chess, much of medicine, and firefighting qualify; individual-stock selection and long-term political forecasting sit near zero validity, so confident judgment there is overconfidence regardless of credentials. Both authors signed the corollary: subjective confidence is no indication of accuracy, because the unskilled do not know when they do not know.^[Daniel Kahneman & Gary Klein 2009 — Conditions for Intuitive Expertise: A Failure to Disagree

They agreed on the boundary conditions and became close friends — and did not converge. Klein remained the admirer of expert intuition and Kahneman its critic; they ended up differing in what each found funny. The title is the finding.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture

Why minds don’t change, and why the method works anyway

Kahneman’s case for the method leans on belief perseverance — Lee Ross’s classic finding that beliefs inferred from discredited feedback survive the discrediting — and on his own record: when the behavioral-priming literature collapsed in the replication crisis, he publicly retracted a chapter of Thinking, Fast and Slow yet could not identify a single important opinion that changed as a result. The evidence was gone; the beliefs stood.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture

The deeper argument is that design-stage bias does the damage. Theorists select, from the space of possible experiments, the ones their intuition says will support their view; the bias enters before any data exist, not just in the file drawer. An adversary pushes exactly the experiments your theory doesn’t rule out but would find embarrassing — which is why Kahneman argues that in a world where neither side concedes, it is optimal for both to be wrong: each side’s predictions fail, and the joint result maps territory neither would have explored.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture

The audience, not the adversary, is the real beneficiary. Kahneman borrows Lakatos’s distinction between progressive refinement and defensive degeneration: the path a theory is on eventually becomes obvious to everyone except perhaps the theorist. Science advances even when the adversaries renege, because the committed predictions and shared data are on record for everyone else. Mellers’s summary of the method’s modesty: “do not change minds, just open a little wider.”^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture

As a science reform

Ceci, Clark, Jussim, and Williams (2024) argue the open-science reforms repaired reliability without repairing validity: preregistration, data sharing, and power analysis police how a study is run and reported, but they do not stop a like-minded team from choosing operationalizations and designs that favor its hypothesis, because nobody with a competing interest is in the room at design time. Adversarial collaboration embeds the competing interest.^[Stephen J. Ceci, Cory J. Clark, Lee Jussim, & Wendy M. Williams 2024 — Adversarial Collaboration: An Undervalued Approach in Behavioral Science Clark, Costello, Mitchell, and Tetlock’s 2022 manifesto (“Keep your enemies close”) makes the institutional version of the case, and the Templeton Foundation has funded five large adversarial collaborations on theories of consciousness — roughly $20 million — with neutral laboratories, not the theorists, collecting the data and adversaries signing advance commitments to take theory-challenging results seriously. Kahneman predicted, fondly, that they would renege on those commitments — and that the work would matter anyway.^[Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture^[Calvin Isch, Philip E. Tetlock, & Cory J. Clark 2025 — Reflections on adversarial collaboration from the adversaries: was it worth it?

Does it work?

Isch, Tetlock, and Clark (2025) ran the first empirical assessment, surveying and interviewing 29 scholars from 13 completed adversarial collaborations spanning 1988 to 2024. The picture:^[Calvin Isch, Philip E. Tetlock, & Cory J. Clark 2025 — Reflections on adversarial collaboration from the adversaries: was it worth it?

• Formation is serendipitous — conflicting published results, a bet with a colleague, an alternative to writing a hostile comment. Nobody has a system for matching adversaries.
• Harder up front, smoother later — most participants rated the work more challenging than typical projects (operationalization disputes take real time), but the need to justify every choice to a hostile reader surfaced arbitrary decisions early, eased peer review, and improved post-publication debate.
• Conflict is rare and minor — the modal report was no conflict at all; what conflict arose was settled by discussion or the arbiter.
• Minds move at the edges, not the core — ~34% reported belief updating, 24% none, ~41% “complicated.” No scholar reported that the results favored their adversary’s position more than their own, and the authors know of no adversarial collaboration in which one side openly admitted thorough defeat. The modal product is integration — both sides somewhat right in different contexts — rather than a verdict.
• Participants rate the output higher than their other work (none rated it lower), and 27 of 29 would do one again.
• Success factors — choose adversaries for intellectual humility, involve a neutral arbiter and early-career scholars (less invested, well placed to run analyses), document everything (pre-project agreements, preregistration), and designate someone to keep momentum.
• Side effect — participants came away more skeptical of the published literature generally, having watched how design choices steer results.

Limitations and open questions

• The Isch sample is survivors only: scholars who completed an adversarial collaboration. Failed attempts are mostly invisible (the authors know of one that collapsed into separate publications), so the encouraging conflict and quality numbers are biased upward.^[Calvin Isch, Philip E. Tetlock, & Cory J. Clark 2025 — Reflections on adversarial collaboration from the adversaries: was it worth it?
• Temperament screens hard; participants themselves said not every scholar can do one, and the method fits only disputes both sides can operationalize.
• Adversarial collaborations produce nuanced conclusions, and nuance is punished where journals reward sweeping claims — one reason the authors recommend pairing the format with Registered Reports, which commit to publish on methods alone.^[Calvin Isch, Philip E. Tetlock, & Cory J. Clark 2025 — Reflections on adversarial collaboration from the adversaries: was it worth it?
• Open questions: whether any adversarial collaboration can produce outright defeat rather than integration; whether journals or funders should organize adversary-matching systematically instead of leaving formation to chance; whether the no-clean-victories pattern reflects the method or the motivated reasoning of those reporting on it.

Connections

• diversity-hypothesis — the Beck/Patterson/Wilson scoping review ends by calling for adversarial collaboration between the diversity-science camps; this page is the method they are invoking.
• schelling-commitment-and-focal-points — recorded predictions plus arbiter-held data as a commitment device: the protocol burns the retreat path of post-hoc reinterpretation that the 15-point effect exploits.
• common-mode-failure — the engineering version of the same idea: independent implementations catch what correlated ones miss. An adversarial collaboration is N-version diversity applied to experimental design and interpretation.
• post-normal-times — where facts are uncertain and values in dispute, critique-as-combat is the default failure mode; the arbiter protocol is one institutional counter-design for the disputes that remain empirical.
• scientific-idea-diffusion-decline — science-of-science neighbor: both pages treat science’s own operating procedures as the object of study.

Sources

• Barbara Mellers, Ralph Hertwig, & Daniel Kahneman 2001 — Do Frequency Representations Eliminate Conjunction Effects? An Exercise in Adversarial Collaboration
• Daniel Kahneman & Gary Klein 2009 — Conditions for Intuitive Expertise: A Failure to Disagree
• Daniel Kahneman 2022 — Adversarial Collaboration: An EDGE Lecture
• Calvin Isch, Philip E. Tetlock, & Cory J. Clark 2025 — Reflections on adversarial collaboration from the adversaries: was it worth it?
• Stephen J. Ceci, Cory J. Clark, Lee Jussim, & Wendy M. Williams 2024 — Adversarial Collaboration: An Undervalued Approach in Behavioral Science