Fable 5 vs Opus 4.8:A Coding-Agent Evaluation
A head-to-head on real engineering work through a harness built to make the comparison mean something. Both models passed every gate, so pass/fail did not separate them; Fable led modestly on tool calls and tokens. Only 4 of 10 tasks ran before Fable was suspended.
9 beats · 8 min to read in full
A head-to-head between ClaudeFable 5 and Claude Opus 4.8
A head-to-head between Claude Fable 5 and Claude Opus 4.8 on real engineering work, run through a reproducible harness built to make the comparison actually mean something.
Condensed from the note.
Why "run bothand eyeball it" fails
Handing two models the same task and picking the nicer output gives a confident answer that is usually wrong. Four problems sink that approach, and the harness is built to remove each one:
Contamination
if a model trained on the fix, you're testing memory, not skill.
No objective "done"
"I prefer this output" is a mood, not a measurement.
Inconsistent help
nudging a struggling model contaminates what you're measuring.
Bias
knowing which model wrote what tilts the scoring.
Condensed from the note.
The coreidea
The model gets a real codebase at a known starting point and a description of the symptom, never the fix.
Condensed from the note.
What the harnessmeasures
Gates (the objective floor). Before each run, the harness confirms the task's test fails, proof the problem is real. After the run, it re-runs the gate and the full test suite to catch anything the fix broke elsewhere.
Condensed from the note.
What wetested it
We ran the comparison on Click (pallets/click), a widely used open-source Python library for building command-line tools.
a small, localized bug fix
a multi-file feature addition
a debug-from-a-traceback task (the model got only the
failing test output, no description)
a deliberately open-ended design task with no single right answer
Condensed from the note.
What wefound
The efficiency edge is real but not uniform. Fable's advantage is decisive on the multi-file feature and slight on the localized fix; the two are effectively even on the open-ended task; and on the debug task Opus actually did the leaner…
lower is more efficient, average of 5 runs. localized bug fix: Fable 5 5.6, Opus 4.8 6.2. multi-file feature: Fable 5 23, Opus 4.8 28. debug from traceback: Fable 5 11.8, Opus 4.8 10.2. open-ended design: Fable 5 33.4, Opus 4.8 34.2.
Condensed from the note, which carries 5 rows.
Builtto reuse
The harness was designed so the experiment isn't a one-off. Swapping the two models or pointing it at a different codebase is a configuration change, not a rewrite, and adding a new task just means supplying a prompt and a test.
Condensed from the note.
That is the note,in one pass.
This is Fable 5 vs Opus 4.8: A Coding-Agent Evaluation condensed to its beats. The full note carries every paragraph, the tables, the code, and the source it was adapted from.