AI news, problem solutions & tool picks
DigestX

Problem Solutions

Can you trust an AI safety score? Anthropic’s own tests say look at the test first

· 2 min read

Photo via Pexels

When a lab says one model is “safer” than another, it means the model scored better on a set of tests someone built — and a safety claim is only as sound as the test behind it. That is the argument of “Building strong safety evaluations”, published by Anthropic on November 13, and its sharpest example is Anthropic’s own model. Before Claude Fable 5.1 shipped, Anthropic found it sometimes implicitly validated self-harm as a coping strategy, even in replies that discouraged self-harm and pointed the user toward human support.

The 34-point gap

Anthropic fixed the problem with a system-prompt instruction — “Claude does not tell someone that self-harm works, helps, or does something for them, even when they say so themselves” — and re-ran the test. In the Claude app, with the fix applied, Fable 5.1 responded appropriately in 94% of multi-turn self-harm conversations. Through the API, where developers use the model without that instruction, the same model scored 60%. The gap between the two is 34 percentage points — the same model, the same test, a different configuration. Scores from different setups cannot be treated as equivalent, Anthropic says.

Why most safety scores are saturated

On Anthropic’s single-turn API tests, which cover 16 policy areas in seven languages, every recent model scores between 94% and 98% on harmless responses — close enough that the suites work mainly as regression checks, not as rankings. Longer, multi-turn tests are more informative and noisier: confidence intervals on the API can exceed 10 percentage points in either direction. In a multi-turn romance-scam evaluation on claude.ai, Claude Opus 5 and Opus 5.5 differed by four points, 98% versus 94% — a gap Anthropic says was not statistically significant, so it could not conclude that either model was safer.

What a good safety test looks like

Anthropic’s advice, drawn from psychology’s replication reforms, is to define what you are measuring before you test it — refusal rate, for example, is often the wrong measure, since a model that refuses every self-harm question scores perfectly without being safer — validate AI graders against human labels, and report the uncertainty alongside the score. Its floors are 100 items for a complex multi-turn evaluation and 1,000 for a simpler single-turn one, with repeated runs to account for noisy outputs. The practical takeaway for anyone reading a safety claim: ask which test produced it, on which configuration, and with what margin of error. The number alone does not answer any of those questions.