ThoughtsJun 14, 20264 min read
The Benchmark That Did Not Finish
Four of ten tasks ran before one of the models was suspended. The verdict never arrived; the method survived. On the coding-agent evaluation.
A companion to the research note Fable 5 vs Opus 4.8: A Coding-Agent Evaluation. Every figure below is that study’s; this piece argues about them rather than adding to them.
This is a model comparison that never reached a verdict, published anyway. The interesting part was never which model won. It was what it takes to build a comparison whose answer would have meant anything.
The setup was a head-to-head between Claude Fable 5 and Claude Opus 4.8 on real engineering work, run through a reproducible harness. The hard part, as the note puts it, was never running the models. It was removing the four things that quietly make most model comparisons wrong.
The unit the harness settles on is worth stealing whatever you are measuring: a task is a base commit, a prompt, and a gate, where the gate is a test that is red before the work and green after it. No judgement about whether the code looks good. A state change you can check.
Then it was cut short
Of ten planned tasks, four ran. On 12 June 2026 Claude Fable 5 was suspended under a US export-control directive, and the harder long-horizon tasks, the ones built specifically to separate the two models, never ran at all.
So the result is thin and the note says so plainly. Both models passed every gate, which means pass/fail did not separate them. Fable led modestly on tool calls and tokens. Four tasks, one codebase, five runs each: directional, not decided. At these sizes Opus was the cheaper model per task, since Fable carries a higher per-token price.
Why publish it
Because the deliverable was the rigor of the measurement, not the verdict, and that part survived intact. The runs are saved, the harness is reusable, and the next comparison starts from a method rather than from scratch.
There is a quieter lesson in the timing. The strongest hypothesis, that Fable suited long-running low-supervision work where staying coherent over a long horizon matters more than any single edit, is exactly the one the unrun tasks were built to test. An evaluation programme that only publishes finished results would have published nothing here, and the honest record of what was not learned is worth more than silence.
The four failure modes the harness removes, the per-task numbers and the limits are in the research note.