An LLM cascade router: a cheap model answers first, a quality judge decides whether the answer is good enough, and only the failures get re-run on an expensive model. Ships with an offline harness that proves how much money this saves and how much quality it costs.
Status: design complete, implementation not started. Everything below is the intended architecture, not shipped code. The full specification is in docs/Deeprouter_PRD_CN.md (Chinese).
Languages: English (this file) · Chinese README
This is not another multi-model gateway. Unified APIs, model switching, fallback, cost tracking — LiteLLM already solved all of it and is the de facto default for that layer. Rewriting it from scratch would just produce a worse LiteLLM.
DeepRouter uses LiteLLM as its provider layer and only builds the decision brain on top — deciding which model should answer this request.
The cascade mechanism itself is also not novel: see FrugalGPT (2023), AutoMix (NeurIPS 2024), and Hybrid LLM (ICLR 2024). What this project adds is verifiable calibration of the judge and Oracle-normalised reporting — see Prior work.
There is a more intuitive option: look at the question, guess how hard it is, send it straight to the right model — one call, and no bill for a cheap answer you might throw away. That route is real; it is called predictive routing, and it is what Martian, Not Diamond and RouteLLM do.
We did not take it. The difference is when the decision happens:
| Predictive (guess, then route) | Cascade (this project: answer, then judge) | |
|---|---|---|
| How it decides | Looks at the question and predicts difficulty | Lets the cheap model answer, then judges the answer |
| Evidence it uses | The prompt | The answer that actually came back |
| Calls per request | 1 | 1 (passes) or 2 (escalates) |
| What it needs before you can start | Paid-for training data | Nothing — if the judgement is accurate, the saving follows |
To be clear: we do use an LLM to decide — the judge is itself an LLM call. The difference is not whether an LLM decides, but what it looks at:
Ours marks the finished work; predictive routing reads the tea leaves on the question.
Two reasons we chose it:
- Much stronger evidence. "Hard" and "this model will get it wrong" are not the same thing — difficulty varies enormously within a category, and models have their own quirks: some easy-looking questions are exactly the ones they fumble. Judging the answer means you are not predicting failure, you are looking at the finished attempt: did it truncate, did it refuse, how confident is it.
- No upfront spend. Training a model to predict which LLM will succeed means first paying to run a large question set across every model and labelling the outcomes. For a portfolio project that is a spend-before-you-know-it-works route, and too risky.
The cost we accept: a cascade always pays for the cheap answer first, and does not get that money back when it escalates — predictive routing goes straight to the right model. That gap is exactly what the two open-book variants further down are there to measure.
The two approaches are complementary, not rivals. Rules could later pick out the obviously-hard questions and start them at the second tier, combining both.
Cascading does not always save money — sometimes it costs more. The good news is you can settle this with a calculator, long before any code exists.
Take 1000 requests. A cheap-model answer costs $1; a judge call costs $0.50.
Case 1 — the strong model costs $10, and the cheap model handles 70% of requests
| Bill | |
|---|---|
| All-strong | 1000 × $10 = $10,000 |
| Cascade | All 1000 pay up front for cheap + judge: 1000 × $1.50 = $1,500 The 30% the cheap model missed (300) are re-run on the strong model: 300 × $10 = $3,000 Total $4,500 |
| ✅ 55% saved |
Case 2 — the strong model only costs $3, and the cheap model handles just 40% of requests
| Bill | |
|---|---|
| All-strong | 1000 × $3 = $3,000 |
| Cascade | Still pays up front for everything: $1,500 The 60% the cheap model missed (600) are re-run: 600 × $3 = $1,800 Total $3,300 |
| 🔴 10% more expensive |
Every request pays a toll up front — one cheap answer plus one judge call — whether or not it ends up escalating.
What that toll buys is a chance: maybe this one won't need the expensive model at all.
So the toll is only worth paying when the expensive trip is genuinely expensive. If the strong model isn't much pricier to begin with, the toll is pure waste — which is exactly what sinks Case 2.
Two variables decide it: how many times pricier the strong model is, and what fraction of requests the cheap model handles. The table covers every combination.
(still assuming a judge call costs half a cheap answer)
Row = how many times pricier the strong model is. Column = fraction of requests the cheap model handles. The cell is the share of the bill you save.
| Strong ÷ cheap | Cheap model handles 40% | 50% | 60% | 70% | 80% | Cheap model must handle at least |
|---|---|---|---|---|---|---|
| 3× | 🔴 −10% | 0% | +10% | +20% | +30% | 50% |
| 5× | +10% | +20% | +30% | +40% | +50% | 30% |
| 10× | +25% | +35% | +45% | +55% | +65% | 15% |
| 20× | +32.5% | +42.5% | +52.5% | +62.5% | +72.5% | 7.5% |
| 30× | +35% | +45% | +55% | +65% | +75% | 5% |
Both bills above are in the table: Case 1 is 10× with the cheap model handling 70%, Case 2 is 3× with 40%.
Look at the last column: the pricier the strong model, the lower the bar to break even. Which means the thing to get right first is not judge accuracy — it is picking a model pair with a wide enough price gap.
Targeting a 40% saving with the cheap model's accuracy capped at ~70% requires the strong model to be at least 5× pricier — and at 5× the cheap model must handle the full 70%, with nothing to spare. At 10× it only needs to handle 55%, which leaves actual headroom.
👉 Model selection is therefore task #1, before any code gets written. If the measured gap is under 5×, the project's premise does not hold.
The formulas behind these numbers
Normalise so that one cheap-model answer = 1 cost unit:
| Symbol | Meaning |
|---|---|
k |
cost of one strong-model answer (how many times pricier the strong model is) |
j |
cost of one judge call |
p |
fraction of requests the cheap model handles without escalating |
all-strong cost = k
cascade cost = 1 + j + (1-p)·k
cost saving S = p − (1+j)/k
break-even : p = (1+j)/k
The table uses j = 0.5, so each cell is p − 1.5/k and the last column is 1.5/k.
k ≥ 5 falls out of substituting the target S = 40% and the ceiling p = 70%: 1.5/k ≤ 0.3.
Seven modules, each doing exactly one thing. The critical property is the red edge that must not exist.
flowchart TB
subgraph Entry["Entry — thin"]
API["api/<br/>FastAPI"]
end
subgraph Online["Runtime path — no ground truth reachable here"]
ROUTER["router/<br/>cascade decision"]
JUDGE["judge/<br/>score(task, answer) → 0..1"]
PROV["providers/<br/>call(model, messages) → Response"]
end
subgraph Offline["Offline only — evaluation"]
EVAL["evaluation/<br/>experiments · metrics · plots"]
VERIF["verifier/<br/>verify(task, answer, reference) → bool"]
end
subgraph Persist["store/"]
PG[("PostgreSQL<br/>requests · route traces · results")]
end
subgraph Models["3 distinct models — judge ≠ cheap"]
CHEAP["cheap model"]
STRONG["strong model"]
JM["judge model<br/>smallest"]
end
API --> ROUTER
ROUTER --> JUDGE
ROUTER --> PROV
JUDGE --> PROV
PROV --> CHEAP & STRONG & JM
ROUTER --> PG
EVAL --> VERIF
EVAL --> PG
ROUTER -. "❌ MUST NEVER REACH — CI-enforced" .-> VERIF
linkStyle 10 stroke:#c0392b,stroke-width:3px
| Layer | Module | Role |
|---|---|---|
| Entry | api/ |
HTTP surface. Deliberately thin — no business logic |
| Decision | router/ |
The cascade: answer cheap → score → return or escalate |
| Scoring | judge/ |
Every score is a real LLM call (the smallest of the three models), estimating how trustworthy an answer looks without ground truth. Four cheap signals combined into one score. |
| Grading | verifier/ |
No LLM involved at all: exact comparison and unit tests, a machine ruling outright with no room for interpretation. Decides right/wrong using ground truth, offline only — precisely because it is not an LLM, it can be the foundation every number rests on |
| Providers | providers/ |
LiteLLM wrapper. Returns exact token counts and cost — never estimates |
| Experiments | evaluation/ |
Runs the baselines, replays strategies, computes metrics |
| Persistence | store/ |
Single store. Requests, per-tier scores, cost, latency |
The three models at the bottom of the diagram have completely different jobs — and only two of them answer anything.
| What it does | Does the user ever see its output? | |
|---|---|---|
| Cheap model | Answers every single question first. Cheap, noticeably weaker | ✅ Yes — whenever its answer is good enough |
| Strong model | Only steps in when that answer isn't good enough, and redoes the whole thing. Far more expensive | ✅ Yes — whenever it had to step in |
| judge | Writes no answer at all. One job: look at what the cheap model produced, score it, and decide whether it can ship | ❌ Never — the score is internal |
The judge is closer to quality control: the front line writes it, QC glances at it and says ship or redo. QC itself writes nothing.
And because it only emits a score and at most three short issues — never a full answer — it runs on every request yet accounts for under a fifth of total spend. Without that, the whole scheme collapses.
What one request actually costs (using the candidate lineup in Tech stack, USD):
| Cost | vs. going straight to the strong model ($0.077) | |
|---|---|---|
| Cheap model answers + judge scores | $0.0081 + $0.0015 = $0.0096 | ✅ 87% cheaper |
| …plus the strong model redoing it | $0.0866 | 🔴 12% more expensive |
Those two rows are the entire economics of the project: get the call right and you save 87%, get it wrong and you lose 12%. As long as most requests land on the first row, the total comes out ahead.
1. The judge is not "a third model that answers". As above — it never writes an answer.
2. There is also a verifier, and it is not a model.
| judge | verifier | |
|---|---|---|
| Is it a model? | Yes — every score is an LLM call | No — ordinary program code |
| Does it hold the answer key? | No | Yes |
| When does it run? | Live, on every request | Offline only, when scoring results |
| Can it be wrong? | Yes | No — exact comparison and unit tests, a machine ruling outright |
The verifier exists to validate the judge — and to set the threshold the judge runs against. Two steps:
- Measure how accurate the judge is — on questions with known answers, let the verifier establish the truth, compare the judge's calls against it, and compute its discrimination (AUC). Miss that bar and the whole general-judge approach is cut.
- Decide where the threshold goes — sweep it from lowest to highest, and at each position use the verifier to compute what quality survives and what it cost. Only once that curve exists do you know whether the line belongs at 0.6 or 0.7.
In one line: the judge guesses live, the verifier checks afterwards — and what it finds afterwards is what decides where the live threshold sits.
verifierholds the answer key.routermust never reach it at runtime.
If ground truth influences a routing decision, every cost and quality number the project produces is only valid in a world where the answers are already known — which is not the world the system runs in. This is not a discipline problem: even with a perfectly careful team, the numbers are void the moment it happens. So it is enforced by machines, in three layers:
| Layer | Enforcement |
|---|---|
| Code | CI rule: router and its dependencies may not import verifier |
| Data | Ground truth may live only in EvalItem.reference. The request model has no field able to carry it — CI checks the field whitelist |
| Test | Assert the judge's rendered prompt never contains the reference string |
Honest boundary: the Oracle baselines must read ground truth, and they live in evaluation/. So the runtime path is enforced by unreachability; the evaluation layer is enforced by explicit uses_ground_truth=True tagging plus reference-stripping. Only the first is an architectural guarantee.
flowchart TB
REQ(["Request"]) --> C1["cheap model answers"]
C1 --> J["judge scores it<br/>0.0 – 1.0"]
J --> Q{"score above<br/>the threshold?"}
Q -->|Yes| R1["return the cheap answer"]
Q -->|No| S["strong model answers<br/>FROM SCRATCH"]
S --> R2["return the strong answer"]
J -. "timeout / rate-limited / unparseable" .-> FO["retry once → fail-open<br/>flag judge_failed"]
FO --> R1
R1 --> LOG[("persist<br/>per-tier scores · cost · latency<br/>escalated? · judge_failed?")]
R2 --> LOG
Three things the diagram makes easy to miss — and they are exactly what this design costs:
| Easy to miss | What it means |
|---|---|
| Escalation is a redo, not a revision | The strong model never sees the cheap answer. It answers the whole thing again. So one escalation means paying twice and waiting twice — the first call is not refunded |
| No token-by-token streaming | Most AI products display text as it is generated. This one cannot: the judge needs the complete answer before it can score, so the reply only appears once generation has finished. That is structural, not an optimisation away |
| Some users get noticeably slower | Everyone who gets escalated waits two rounds — that part is certain. Whether the non-escalated users end up faster has to be measured, because cheap + judge is itself two round trips and is not automatically quicker than one strong call |
Implementation-level decisions (what happens when the judge itself fails, which answer wins after escalation) live in PRD §6.6 / §7.2.
"Offline" here means behind closed doors, not in front of real users. ("Online" is the opposite: real traffic, live.)
Why it has to happen behind closed doors
The project has to end with a sentence like "40% cheaper, at most 5 points of quality lost". That sentence needs evidence, and the evidence cannot come from real users — real users' questions have no answer key, so there is no way to tell whether a given answer was right, and therefore no way to tell whether quality dropped at all.
So we write our own exam: a few hundred questions whose answers we already know (taken from public question banks: maths with numeric answers, multiple choice, and coding tasks that ship with their own tests — all things a machine can rule on outright. Open-ended questions are excluded: no single right answer, nothing a machine can check). Every approach sits the same exam, and the scores are compared.
The system handles it exactly as normal, with no special case — not one line of the routing logic is maths- or code-specific. The judge looks at whether the answer truncated, whether the model refused, how confident it sounds; none of those signals care what the question was about.
But two things must not be conflated: what the system can handle ≠ what our numbers can vouch for.
| Coverage | |
|---|---|
| Requests the system can serve | Any kind |
| Requests our numbers cover | Maths, multiple choice and code only |
And there is a sharper edge to this: the judge is not equally accurate across task types. The "≥ 0.75" figure is measured on maths and code. On a resignation letter it might hold, or it might be far worse — and we have no way to find out, because finding out would require an answer key, which is exactly what those tasks lack.
👉 So we scope the claim rather than overstate it: these results cover tasks a machine can mark. See PRD §8.1.1.
Why that exam gets sat a hundred times
Because the deliverable is not one number, it's a curve. Move the threshold and both the saving and the quality loss move with it — every point on that curve means re-sitting the entire exam. Twenty points for a curve worth showing. Add six comparison approaches, each sitting it once, and you are at a hundred-plus full re-runs.
If every re-run actually called the models: thousands of calls each time, real money, and tens of minutes of waiting. Nudge the threshold, wait half an hour. At that pace you try a handful of ideas a day, and end up shipping whichever threshold you guessed first.
👉 Hence the key decision: sit the exam once, but record every answer; every experiment after that is rescored from the recording.
flowchart LR
SET["Eval set<br/>~450 machine-checkable tasks"] --> RUN["Run every model<br/>on every task"]
RUN --> CACHE[("Cache<br/>answers · tokens · cost · latency<br/>+ raw judge signals")]
SET --> VER["verifier labels<br/>right/wrong per model × task"]
VER --> CACHE
CACHE ==> REPLAY["Replay any strategy<br/>at any threshold<br/>SECONDS · ZERO COST"]
REPLAY --> B1["all-cheap"]
REPLAY --> B2["all-strong"]
REPLAY --> B3["random"]
REPLAY --> B4["open-book<br/>two variants"]
REPLAY --> B5["FrugalGPT-style<br/>single-signal judge"]
REPLAY --> B6["DeepRouter"]
B1 & B2 & B3 & B4 & B5 & B6 --> M["cost · quality · speed<br/>distance from the ceiling"]
style RUN fill:#8B4513,color:#fff
style REPLAY fill:#1a5c3a,color:#fff
Only the brown box spends money (450 questions × 3 models, every model answering every question once — answers, cost and duration all stored). The green box and everything it feeds is rescoring against that local archive, with no model calls at all — seconds, and free.
Think of it as filming the match. The match is played once (paid for once); if you later want to rescore it under different rules, you replay the tape rather than dragging both teams back onto the pitch.
Replay can tell you what a request cost and whether it was right. It cannot tell you how long a user actually waited — summing cached durations leaves out real network round-trips, queueing and retries. So latency has to be measured separately, with real traffic, never inferred from the tape.
The remaining engineering detail (how replays are made bit-identical, what happens to tasks with missing results) is in PRD §7.4.
Q: Isn't 450 questions too few to cover maths and code?
It does not cover them, and it was never meant to. The space of maths problems is unbounded; 450 does not scratch it.
But the question being answered is not "is this model good at maths" — it is "on the same set of questions, how much does a different routing strategy save, and how much quality does it cost". Those need sample sizes an order of magnitude apart, because both strategies sit the same exam: per-question difficulty is common to both and cancels out, leaving only the questions where the two disagree (estimated ~8%, so about 36). The precision rests entirely on those 36.
What more questions would buy is a calculable trade:
| Questions | Quality precision we can promise |
|---|---|
| 450 (current plan) | quality drop no worse than 5 points |
| ~700 | no worse than 3 points |
Q: What happens when a user asks something not in the bank — say "write me a tree traversal in C++"?
It is served normally; the system never consults the bank. Walk it through: the cheap model writes the code → the judge checks whether it truncated, whether the model refused, how confident it sounds, and scores it against the rubric → compare to the threshold → pass means return it as-is (cost 1.5), fail means the strong model rewrites from scratch (cost 11.5, versus 10 for going straight to the strong model — that request lost money, and is paid for by the ones that passed).
No answer key is involved anywhere, and verifier is never called.
Those 450 questions are calibration weights. In the lab we run them hundreds of times to measure how accurate the judge is and to set the threshold at, say, 0.6. Then the weights go back in the drawer — in production the system carries only that threshold. Exactly as with a scale calibrated against standard weights: whatever you weigh afterwards does not need to have been on the list of weights.
Q: Where do the questions come from, and can they be used commercially?
Subsets of public question banks. Initial finding (pending first-hand verification):
| Bank | Licence | Commercial |
|---|---|---|
| GSM8K · MATH · MMLU · HumanEval | MIT | ✅ retain the copyright notice |
| MBPP | CC-BY-4.0 | ✅ attribution required |
Redistributing a subset, publishing results, attribution, commercial use — all four are permitted. LICENSE directly: third-party derivatives often carry different terms than the original.
"We saved 58%" means nothing on its own.
Nobody knows how much could have been saved. Against a 60% ceiling, 58% is excellent. Against a 95% ceiling, it is poor.
So the same task set is run several different ways, and DeepRouter is placed among them:
| Comparison run | What it is for |
|---|---|
| Everything on the cheap model | How cheap it can possibly get (and the quality floor that comes with it) |
| Everything on the strong model | How good it can possibly get (and the price that comes with it) |
| Escalate at random | Proves the result is not luck |
| Open-book (two variants) | Allowed to peek at the answer key, so it makes the cheapest correct choice on every task. Impossible in reality — it exists purely to measure the ceiling |
| A published approach | FrugalGPT's simpler single-signal judge, to show whether our more elaborate one earns its keep |
| DeepRouter | This project |
Why two open-book variants: one skips the cheap model entirely and goes straight to whichever model is right (the ceiling for routing in general); the other plays fair and still runs the cheap model first, it just knows in advance which answers will be wrong (the ceiling for cascading specifically). The gap between them is what the cascade shape inherently costs — every escalated task paid for the first tier for nothing. We measure ourselves against the second one.
When this is finished, either all five lines below are met, or the honest answer is "cascading does not work at this price gap". There is no third way to report it.
| The question | Pass line |
|---|---|
| Can the judge tell good answers from bad ones? | ≥ 0.75 (out of 1; 0.5 is a coin flip) |
| How much cheaper than running everything on the strong model? | at least 40% |
| How much quality was lost, at worst? | no more than 5 points (e.g. 92% → 87%) |
| How much longer does the slowest group of users wait? | no more than 1.5× the all-strong baseline |
| How much did the judge itself cost? | no more than 15% of the total |
1. "The judge was right X% of the time" is not the metric. Suppose the cheap model already gets 65% of tasks right. Build a fake judge that always says "pass" — it is now "right" 65% of the time, which looks respectable and is worth nothing. So the first line measures something else: whether the judge can rank good answers above bad ones (the standard name is AUC). Against that, the fake judge scores 0.5 — a coin flip — and is exposed immediately.
2. The quality line is a worst case, not an average.
We test a few hundred tasks, not every task in the world, so the measurement carries error. Rather than claim "quality dropped 3 points" — precise-sounding, but finer than the data can actually resolve — the claim is "it is at worst 5 points".
3. All five numbers must come from one setting. Move the threshold and both the saving and the quality loss move with it. So it is not allowed to quote the saving from a low threshold and the quality from a high one — each number would pass on its own while no single deployable configuration satisfies them together. The rule: pick one threshold, read all five off it, and state which one.
4. The judge's own spend counts as cost. Every judge call costs money. Leave it out and "40% saved" is fiction — and that number is the entire deliverable.
| Work | Contribution | Reported |
|---|---|---|
| FrugalGPT (2023) | Introduced the LLM cascade | Matched GPT-4 at up to 98% lower cost |
| Model Cascading (EMNLP 2022) | Confidence-threshold escalation | 88.93% compute saved, +2.18% accuracy |
| LLM Cascades w/ Mixture of Thought (ICLR 2024) | Answer consistency as the escalation signal | Matched GPT-4 at 40% of its cost |
| AutoMix (NeurIPS 2024) | Self-verify → meta-verifier → escalate | >50% cost reduction |
| Hybrid LLM (ICLR 2024) | Test-time-tunable quality knob | 40% fewer large-model calls, no quality loss |
AutoMix is architecturally almost identical to this project. The mechanism is not the contribution. What is:
- Verifiable judge calibration — the judge is itself graded against a verifier, reported as AUC plus a full confusion matrix, on out-of-fold scores only.
- Oracle-normalised reporting — "we captured X% of the achievable saving", not a bare percentage.
Predictive routers (Martian, Not Diamond, RouteLLM) pick the model before generation and are a complementary branch, not cascades.
| Tech | What it does here | Caveat |
|---|---|---|
| Python 3.11+ | The language everything is written in | — |
| LiteLLM | An open-source library that collapses every vendor's differently-shaped API (OpenAI, Anthropic, …) into one calling convention. Every model call goes through it, so no vendor-specific code is written here | |
| Pydantic | Declares what fields a record must have and of what type, and forces the judge to reply in a fixed shape so code can read it directly instead of guessing | A judge reply that fails the shape check counts as a judge failure (see Failure default) |
| FastAPI | Exposes the HTTP endpoint other programs send requests to | Kept deliberately thin: transport only, no routing logic |
| PostgreSQL | The database that records every request end to end: per-tier scores, cost, latency, whether it escalated | Honestly heavier than an MVP needs — flat files would do. Kept to demonstrate the engineering, not because it is required |
| Docker + docker-compose | Runs LLM-generated code inside an isolated box so it cannot damage the host; compose brings the database and friends up with one command | Network-off, time- and memory-capped. Never run generated code on the host |
| Pytest | Runs the automated tests — pre-written checks that verify in one command that nothing broke | The three isolation checks from the Architecture section are written with it |
| GitHub Actions | The robot at the repo door: every push, it runs the whole test and check suite, and blocks the merge if anything fails | This is what makes the forbidden red edge actually unreachable, rather than a rule people are asked to remember |
| Single HTML page + a charting lib | The demo: one slider and two curves — drag it and watch cost and quality move | No frontend framework. Reads a static data file, so it runs fully offline — it works in an interview with no wifi |
Model selection is priority one — as the economics shows, too small a price gap and the project has no premise. At 2026-08 list prices, this is the lineup:
| Role | Candidate | Price (per MTok, in / out) |
|---|---|---|
| Strong model | claude-fable-5 |
$10 / $50 |
| Cheap model | claude-sonnet-5 |
$3 / $15 |
| judge | claude-haiku-4-5 |
$1 / $5 |
At typical question lengths: the strong model works out ~9.5× pricier than the cheap one (comfortably above the floor of 5), the judge accounts for 19% of a request's cost, and buying the whole 450-question experiment once costs about $39.
⚠️ These are estimates, not measurements. The price gap has to be derived from logged token counts, not the price list. The good news is that the error runs in our favour: the strong model writes longer answers and escalated questions are harder, so the measured gap is usually larger than the estimate.
Why the cheap model isn't the cheapest one available — this is the trap in model selection:
The judge is under two constraints at once: it may not be the same model as the cheap one (or it grades its own work), and it must be cheaper than the cheap one (or its cost share blows the limit). Together those mean the judge has to sit one rung below the cheap model.
So the moment you pick the cheapest model in the lineup as your cheap model, the judge has nowhere to go: nothing below it, and anything above costs more than the thing it is grading. Selection has to leave a rung for the judge.
The numbers for all four combinations
| Combination (strong / cheap / judge) | Price gap | Judge cost share | Verdict |
|---|---|---|---|
| Fable 5 / Sonnet 5 / Haiku 4.5 | 9.5× | 19% | ✅ the only one that clears both |
| Opus 5 / Sonnet 5 / Haiku 4.5 | 4.8× | 19% | ❌ gap falls just short of 5× |
| Fable 5 / Haiku 4.5 / Sonnet 5 | 28.5× | 167% | ❌ judge costs more than twice what it grades |
| Opus 5 / Haiku 4.5 / Sonnet 5 | 14.3× | 167% | ❌ same |
The last two rows are what "cheap model on the bottom rung" costs you — however good the price gap looks.
- Sonnet 5 may be too strong at maths. Offline evaluation needs the cheap model's accuracy between 50% and 70%, and Sonnet 5 will very likely blow past that on common maths banks — at which point nothing ever escalates and the routing never happens at all. That isn't a selection problem; it means picking harder questions.
- Fable 5 comes with two constraints. It requires 30-day data retention (organisations on zero retention get an error on every call), and its safety classifiers can decline. Maths and code rarely trip them; handle declines under the eval-set rules.
| Tech | Purpose | Trigger |
|---|---|---|
| Langfuse / LangSmith | Visualise each routing decision and what it cost | When debugging gets hard |
| Redis | Response cache for repeated requests | When repeats actually occur. Cache hits must be excluded from every reported number, or savings get attributed to the wrong thing |
| Learned router | Train a classifier on accumulated traces to predict the starting tier | Once there is enough data — a second write-up |
Saying no is part of the design. Each of these was considered and rejected:
| Tech | Why not |
|---|---|
| Bedrock / more providers | LiteLLM already abstracts this. Adding a provider is half a day's work and demonstrates nothing |
| DynamoDB / S3 | One store is enough. A second one only adds cognitive load |
| Terraform / CDK / Lambda / ECS | Unless the target role is DevOps, docker-compose is a sufficient demo |
| OpenTelemetry / Prometheus | These are for the system being monitored, not for this |
| LangGraph | A two-tier cascade is a simple loop. It does not need a workflow engine |
| Vector database | Nothing to retrieve in the MVP |
| Priority | Scope |
|---|---|
| P0 | providers · 2-tier cascade · verifier · offline replay · judge calibration · cost-quality curve · the three isolation checks |
| P1 | Multi-signal judge · Docker sandbox · demo UI |
| P2 | FastAPI · PostgreSQL · CI wiring |
P1 does not start until P0 runs end to end. The failure mode this guards against is the calibration experiment — the only thing that makes every other number credible — being squeezed out at the end.
| Doc | Description |
|---|---|
| docs/Deeprouter_PRD_CN.md | Full design spec (Chinese) — economics, judge design, evaluation protocol, risks |
Apache License 2.0 — see LICENSE.