{"slug":"research-routing-with-levanto-sage","name":"Choosing the right model per task: a study with Levanto Sage","rules":{"start":"open","maxDepth":13,"allowRewind":true,"allowReplay":true},"nodes":{"open":{"short":"the note","pose":{"base":"centre"},"beat":{"layout":"center","marker":"partner study · interactive","title":["Choosing the right model per task:","a study with Levanto Sage"],"glass":1,"lede":"A two-round study with Levanto on 90 tasks from XO Space's own development. When Sage chose the model and effort level for each request, and unfinished work was retried once on a stronger layer, quirq AI's XO platform completed 84 of 90 tasks against 83 for an all-Opus baseline, for half the total cost.","caption":"13 beats · 20 min to read in full"},"prompt":"Where do you want to start?","choices":[{"label":"From the top","to":"premise"},{"label":"round two","to":"round-two-84-of-90-tasks-complet"},{"label":"Straight to the end","to":"close"}]},"premise":{"short":"the premise","pose":{"base":"flooded"},"beat":{"layout":"left","marker":"01 · partner study","title":["With Levanto, we set out to learn","whether quirq AI's XO platform"],"lede":"With Levanto, we set out to learn whether quirq AI's XO platform could stop sending every coding request to its most capable and most expensive model without leaving work unfinished.","caption":"Condensed from the note."},"prompt":"Where next?","choices":[{"label":"why running every","to":"why-running-every-request-on-the"},{"label":"Wrap it up","to":"close"}]},"why-running-every-request-on-the":{"short":"why running every","pose":{"base":"drained"},"beat":{"layout":"right","marker":"02 · why running every","title":["Why running every request","on the strongest model is wasteful"],"lede":"The cost of running a coding agent depends on two settings: the model it runs on, and its effort level, which controls how much the model reasons before it acts. We call the pair an intelligence layer.","rows":[{"title":"How much does routing each request to a layer save","note":"compared with running everything on Opus 5.5 at high effort?"},{"title":"Do routed requests get completed as often?","note":""},{"title":"Does it help to also tell the agent which files are likely to matter?","note":""},{"title":"Can the answers people give afterwards be used to","note":"correct routing for later requests?"}],"caption":"Condensed from the note."},"prompt":"Keep going?","choices":[{"label":"what we built","to":"what-we-built-routing-context-ta"},{"label":"Wrap it up","to":"close"}]},"what-we-built-routing-context-ta":{"short":"what we built","pose":{"base":"recede"},"beat":{"layout":"left","marker":"03 · what we built","title":["What we built: routing,","context targeting and feedback"],"lede":"Levanto Sage is an API that answers structured questions about a piece of text. It supports several question types, and this study used two of them. A choice question picks one option from a list and returns a probability for each option.","rows":[{"title":"Routing","note":"which model and effort level the request runs on."},{"title":"Context targeting","note":"which files the agent is pointed to before it begins."},{"title":"Feedback","note":"how a person's answer to \"did it complete your task?\" corrects later routing decisions and, in the second round, whether an incomplete task is run again on a stronger layer."}],"caption":"Condensed from the note."},"prompt":"What next?","choices":[{"label":"how the study was","to":"how-the-study-was-run-tasks-setu"},{"label":"Wrap it up","to":"close"}]},"how-the-study-was-run-tasks-setu":{"short":"how the study was","pose":{"base":"drained"},"beat":{"layout":"right","marker":"04 · how the study was","title":["How the study was run:","tasks, setups and judging"],"lede":"All tasks were chosen and recorded before any of them were run. There were 90 in total, in three groups.","rows":[{"title":"40 code changes from XO Space's own history","note":"the 40 most recent commits that changed between one and six code files, leaving out commits that only touched documentation, tests or build setup."},{"title":"20 questions about how XO Space's code works","note":"for example \"how does XO back up a project to GitHub?\"."},{"title":"30 everyday requests, used in the second round only","note":"explanations, emails and posts, short scripts, and questions that need current information from the web."}],"caption":"Condensed from the note."},"prompt":"Where next?","choices":[{"label":"round one","to":"round-one-lower-cost-but-far-few"},{"label":"Wrap it up","to":"close"}]},"round-one-lower-cost-but-far-few":{"short":"round one","pose":{"base":"recede"},"beat":{"layout":"left","marker":"05 · round one","title":["Round one: lower cost,","but far fewer tasks completed"],"lede":"The first round used the 60 code changes and questions, with Claude Sonnet 5 in the standard and research layers and no retries.","panelRows":[{"title":"Total cost, Levanto included","note":"Opus only $64.63 · Routed $27.81 · Routed with note $29.89"},{"title":"Cost per task, as a share of Opus (median)","note":"Opus only n/a · Routed 0.53 · Routed with note 0.53"},{"title":"Tasks completed","note":"Opus only 53 · Routed 29 · Routed with note 36"},{"title":"of 40 code changes","note":"Opus only 36 · Routed 16 · Routed with note 19"}],"caption":"Condensed from the note, which carries 5 rows."},"prompt":"Keep going?","choices":[{"label":"round two","to":"round-two-84-of-90-tasks-complet"},{"label":"Wrap it up","to":"close"}]},"round-two-84-of-90-tasks-complet":{"short":"round two","pose":{"base":"drained"},"beat":{"layout":"right","marker":"06 · round two","title":["Round two: 84 of 90 tasks completed,","at half the cost of Opus"],"lede":"The second round used all 90 tasks. On the first attempt, routed sessions completed 73 tasks against 83 for the all-Opus baseline. After one retry on the next layer up, they completed 84.","figure":{"kind":"bars","categories":["Opus only","Routed, first attempt","Routed, within one retry"],"series":[{"label":"Completed by setup","tone":"value","values":[83,73,84],"display":["83","73","84"]}],"caption":"Round two completion score, counted as tasks judged complete out of 90. After one retry on the next layer up, the routed setup finished one task more than the all-Opus baseline."},"caption":"Condensed from the note, which carries 4 rows."},"prompt":"What next?","choices":[{"label":"context targeting","to":"context-targeting-and-feedback-i"},{"label":"Wrap it up","to":"close"}]},"context-targeting-and-feedback-i":{"short":"context targeting","pose":{"base":"flooded"},"beat":{"layout":"left","marker":"07 · context targeting","title":["Context targeting and feedback","in round two: no measurable effect"],"lede":"In the second round, the where-to-look note made no measurable difference. The setup that used it completed exactly as many tasks within one retry as the setup that did not, 84 of 90, at the same cost per task.","caption":"Condensed from the note."},"prompt":"Where next?","choices":[{"label":"projected cost","to":"projected-cost-at-100-1-000-and"},{"label":"Wrap it up","to":"close"}]},"projected-cost-at-100-1-000-and":{"short":"projected cost","pose":{"base":"drained"},"beat":{"layout":"right","marker":"08 · projected cost","title":["Projected cost at 100,","1,000 and 10,000 requests"],"lede":"The study measured 90 requests. To see what its measured costs imply at larger volumes, we projected the total cost of a given number of future requests for the routed setup, with retries and Levanto's fee included, and for Opus only.","panelRows":[{"title":"100","note":"Opus only $77 ($44 to $140) · Routed $38 ($22 to $56) · Saving $39 ($10 to $100)"},{"title":"1,000","note":"Opus only $773 ($508 to $1,220) · Routed $378 ($257 to $511) · Saving $395 ($150 to $831)"},{"title":"10,000","note":"Opus only $7,730 ($5,096 to $11,887) · Routed $3,782 ($2,609 to $5,055) · Saving $3,948 ($1,542 to $8,047)"}],"caption":"Condensed from the note."},"prompt":"Keep going?","choices":[{"label":"what the results","to":"what-the-results-mean"},{"label":"Wrap it up","to":"close"}]},"what-the-results-mean":{"short":"what the results","pose":{"base":"centre"},"beat":{"layout":"left","marker":"09 · what the results","title":["What","the results mean"],"lede":"Routing works when the less expensive layers are capable enough for the work they are given. In the first round the routing decisions were mostly sensible, and the middle layer could not carry out the small code changes sent to it.","caption":"Condensed from the note."},"prompt":"What next?","choices":[{"label":"scope of this","to":"scope-of-this-study"},{"label":"Wrap it up","to":"close"}]},"scope-of-this-study":{"short":"scope of this","pose":{"base":"flooded"},"beat":{"layout":"right","marker":"10 · scope of this","title":["Scope","of this study"],"rows":[{"title":"The study used one codebase","note":"XO Space, with about 540 code files, and one agent, Claude Code. Other codebases and other agents may route differently."},{"title":"Results were judged by reviewers reading each change against the real commit. ","note":"The two independent reviewers agreed on 96% of results."},{"title":"Every incomplete result was retried in the second round. ","note":"In daily use a retry follows the person's own answer to whether the task was completed."}],"caption":"Condensed from the note."},"prompt":"Where next?","choices":[{"label":"our partnership","to":"our-partnership-with-levanto"},{"label":"Wrap it up","to":"close"}]},"our-partnership-with-levanto":{"short":"our partnership","pose":{"base":"drained"},"beat":{"layout":"left","marker":"11 · our partnership","title":["Our partnership","with Levanto"],"lede":"We thank the Levanto team for their partnership on this study. Sage gave us a way to express a routing policy as a handful of plain sentences, to change that policy without changing code, and to make each decision for a fraction of a cent.","caption":"Condensed from the note."},"prompt":"Keep going?","choices":[{"label":"Close the note","to":"close"}]},"close":{"short":"the end","pose":{"base":"finale"},"beat":{"layout":"center","title":["That was one path.","The note has the rest."],"glass":1,"lede":"This walk is one route through Choosing the right model per task: a study with Levanto Sage. The full note carries every chapter, the tables, and the code.","links":[{"href":"/research/routing-with-levanto-sage","label":"Read the full note"},{"href":"/research","label":"All research","tone":"ghost"}]}}}}