Risk Matrices
A risk matrix (RM) plots likelihood against consequence on a grid, colors the cells green/yellow/red, and ranks risks by which cell they land in. Oil and gas runs on them — ISO, API, and NORSOK standards recommend them, and Thomas, Bratvold, and Bickel (SPE Economics & Management, 2014) reviewed 30 SPE papers that present RMs as best practice. Their finding: the rankings an RM produces are artifacts of arbitrary design choices, and no published empirical evidence shows RMs improve risk-management decisions. The flaws they catalog are structural — inherent to the grid, not fixable by tuning it. 1
The documented flaws
Risk-acceptance inconsistency. Cox (2008) showed that a consistent matrix must keep the expected loss of red cells above green cells (with yellow separating the two, not meaning “medium risk”). Every one of the 30 surveyed SPE papers violates at least one of Cox’s conditions. In the paper’s drilling example, the yellow “blowout” cell carries a higher expected loss than the red cells above it — so the matrix directs mitigation at the wrong outcome. 2
Range compression. Binning continuous values into score bands assigns identical ratings to quantitatively distant risks. A USD 20 million loss and BP-Macondo-scale USD 50 billion both fall in “catastrophic” because the top category has no upper bound. Blowout and loss of well control both get “yellow” and read as roughly equal; three orders of magnitude vanish in the binning. 3
Centering bias. Scorers avoid extremes: Hubbard (2009) found 75% of IT-project scores in the middle of a 1–5 scale, and across the six surveyed SPE papers with enough detail to check, 83% of probability scores were centered (52% of consequence scores) — compressing the effective scale further. 4
Category-definition bias. Verbal bands mean different things to different people. Budescu et al. (2009) found respondents read “very likely” anywhere from 0.43 to 0.99 even with guidelines; across the SPE corpus, “improbable” ranged from “virtually unrealistic” to “a rare combination of factors,” and “frequent” from “more than once a year” to “more than once in ten years.” 97% of the surveyed scores were qualitative or semi-qualitative. 5
Ranking reversal and the lie factor (the paper’s own additions). Rankings can flip with the arbitrary direction of the score scale — increasing versus decreasing scores reorder the same risk set. And the matrix distorts relative distance: the visual gap between two cells doesn’t reflect the ratio of their expected losses, which the authors quantify as a “lie factor.” Their alternative is standard risk- and decision analysis: elicit probabilities and consequences, compute expected loss, decide against a stated risk appetite. 6
Why the pattern travels
The grid is everywhere ordinal scoring is, and the same defects recur:
- threat-modeling — DREAD’s 1–10 average fails the same way: LeBlanc showed a damage-1 nuisance and a discoverability-1 catastrophe score identically under the Writing Secure Code recipe, and dropped to three levels per factor plus judgment.
- iec-62443 — SL-T allocation leans on “the company’s own calibrated risk matrix” because the standard prescribes no method; the surveyed RM pathologies come along.
- calculate-cvss-scores — CVSS metrics are ordinal too, and the vector-plus-vector arithmetic is where similar critiques land.
- economics-of-cyber-risk — the defense-investment models there assume risk numbers that a matrix can’t reliably produce.
Related
Sources
Footnotes
-
Philip Thomas, Reidar B. Bratvold, J. Eric Bickel 2014 — The Risk of Using Risk Matrices ↩
-
Philip Thomas, Reidar B. Bratvold, J. Eric Bickel 2014 — The Risk of Using Risk Matrices ↩
-
Philip Thomas, Reidar B. Bratvold, J. Eric Bickel 2014 — The Risk of Using Risk Matrices ↩
-
Philip Thomas, Reidar B. Bratvold, J. Eric Bickel 2014 — The Risk of Using Risk Matrices ↩
-
Philip Thomas, Reidar B. Bratvold, J. Eric Bickel 2014 — The Risk of Using Risk Matrices ↩
-
Philip Thomas, Reidar B. Bratvold, J. Eric Bickel 2014 — The Risk of Using Risk Matrices ↩