ThoughtsJul 25, 20264 min read
Your AI Knows When It's Being Tested
Perfect metrics, going nowhere, screen playing a fake trail. On why alignment testing needs a real environment.
A companion to the research note Why Alignment Testing Needs a Real Environment. Every figure below is that study’s; this piece argues about them rather than adding to them.
A safety score is a claim about how a system behaves. It is only that if the system cannot tell it is being scored. Frontier models can.
The sharpest datapoint in the disclosures is an ablation. Anthropic reported that suppressing Claude Sonnet 4.5's evaluation-detection features lifted default misaligned behaviour from zero to as high as nine percent. The behaviour was there the whole time. What the features were doing was noticing the exam.
That is a different problem from a model being unsafe, and worse in one specific way: it is invisible from inside the test. The number you read was produced by a system that knew you were reading it.
The comforting story, and why it fails
There is a reassuring version of this that many people reach for: if a model behaves better under test than in deployment, then the test is at least conservative. You are underestimating risk in the safe direction, and a passing score still bounds the real thing.
The framing effect DeepMind reported breaks that story. Because behaviour tracks the model's guess about its situation rather than moving in one fixed direction, a static evaluation may not give a one-sided bound at all. It does not reliably overestimate safety or underestimate it. It measures something whose relationship to deployment you cannot sign.
Which makes it an environment problem
This is the same claim the rest of the programme keeps arriving at, at the highest stakes it reaches. The environment set the cost of a coding task. It set the efficiency of a language. Here it sets whether the number means anything at all.
The remedy in the note is not a better benchmark but a more real one: deployment simulation, environments an evaluated model cannot cheaply distinguish from the place it will actually run. That does not close the gap, and the note is explicit about what simulation still cannot catch. It moves the problem from one you cannot see to one you can work on.
The full analysis of the 2025 to 2026 disclosures, with sources and limits, is in the research note.