are built the agentic way: put a model wherever a decision needs judgment, and let it decide. That means a lot of model calls. The pipeline from Article 9 (a production RAG pipeline for PDFs) is built exactly like that. Ask “What is the annual premium?” and it runs three model calls in series:
- one to parse the question,
- one to arbitrate the retrieved candidates,
- one to generate the typed answer.
Each call buys something real. The parsing call absorbs typos and vague phrasing before retrieval runs. The arbiter call keeps a wrong page from reaching generation. The generation call returns a typed answer with a quote you can check on the page. On a hard question, these three calls are what make the answer trustworthy, and that is why Article 9 put them there.
The problem is that we cannot afford this on every question, because it is slow. Three calls in series means the user waits about two seconds, and pays tokens, on every question, easy ones included. And many enterprise questions were never hard: “What is the annual premium?” has one answer, on one line, and the deterministic keyword match had already isolated it on the first pass. So this article adds a router: a cheap per-question signal, read before any model call, that sends easy questions down the fast path and keeps all three calls for the hard ones. This article is the map of that routing: the signal (a score the pipeline already computes), the margin that makes it honest, what it saves (about two seconds on a keyword match), and the questions it must refuse to fast-path. Every number is a real run on the fictional broker corpus, reproducible in the companion notebook.
🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.
📓 The runnable notebook for this article is on GitHub: doc-intel/notebooks-vol1. It runs the router on the broker corpus, prints the per-question confidence score, and shows which questions skip the model and which keep it.
1. Every question pays for the whole pipeline
The upgraded pipeline is worth its cost on hard questions. The problem is that it charges the same cost on easy ones. Trace a single question through it and the model calls stack up: one to normalize the question and pull its keywords, one for the arbiter that ranks the retrieved candidates, one to generate the typed answer. Three round-trips to a hosted model, in series, before the user sees a word.
On a hard question that is money well spent. On “what is the premium?” it is three network hops to return a value that a keyword match had already isolated on the first pass. The latency is real and it is the part the user feels. The token cost is smaller per call, but it is billed on every question, forever, and the easy questions are the ones users repeat most.
Before any of those three calls fire, the pipeline can make one decision with no model call, from the parsed question and the document’s line_df: can this question be answered on the keyword path alone?
The decision needs a signal. It has to be cheap, because the whole point is to spend nothing when the answer is easy, and it has to be honest, because a router that sends a hard question down the fast path returns a confident wrong answer, the exact failure the series works to avoid.
2. Not every question needs the model
The signal is already in the pipeline. Retrieval scores each line by how strongly the question’s keywords co-occur on it, and that score, before any arbiter or generation call, already separates the questions that answer themselves from the ones that do not.
2.1 A question that answers itself
Take “what is the annual premium?” against a homeowner’s policy. The deterministic keyword scorer from Article 7 (retrieval as filtering on line_df), co_occurrence_score, counts how many of the question’s keywords land on each line, weighting a primary term (premium) together with the co-signals that mark a real answer (EUR, annual, payable). Run it and one line wins outright.
The top line, “The annual premium is EUR 1,200, payable…”, scores 5. Every other line scores 0. One clear winner, a margin of 5 over the runner-up, and the winning line already carries the shape of the answer, a currency and an amount. There is nothing for the model to disambiguate. The keyword path has answered the question, and a generation call here would only reformat a value that is already isolated.
2.2 A question that needs the model
Now ask “which guarantees can I avoid in my case?” against the same policy. The keyword scorer runs the same way, but the result is different in kind.
Three lines tie at a score of 2: the two optional guarantees and the line that mentions adjusting them. The margin between first and second is zero. No single line stands out, because the question is not asking for a line. “In my case” asks the model to weigh the user’s situation against several optional guarantees and reason about which ones are safe to drop. That is generation work, and the flat score is the tell: the keyword path found candidates, not an answer.
3. The signal is the margin
The router does not need a new model. It needs the number the retrieval brick already produced. Two features of the keyword scores decide the route: the top score, and the margin between the top line and the next one.
A high top score with a wide margin, Q1’s 5 against 0, means one line matched strongly and nothing else came close. The answer is on that line. Route it to a deterministic extractor that pulls the value and skips the model entirely. A low top score, or a top score with no margin, Q2’s flat 2s, means the keywords spread across several lines without settling. Send that question through the full pipeline: arbiter, then generation, the model calls justified on exactly the questions that need them.
The whole router is that decision, and it needs no model of its own. Score the lines, read the top score and its margin, and return a route.
# co_occurrence_score: the keyword scorer from Article 7, in the companion notebook.
def route_question(line_df, primary, secondary, *, min_score=4, min_margin=3):
"""Decide, with no model call, whether the keyword path already answers."""
scores = [co_occurrence_score(t, primary, secondary) for t in line_df["text"]]
top, second = sorted(scores, reverse=True)[:2]
# The signal is the retrieval brick's own output: a high top score with a
# clear margin means one line answered; a flat score means it did not.
confident = top >= min_score and (top - second) >= min_margin
# "fast" skips the model. "full" runs the arbiter and generation.
return "fast" if confident else "full"The threshold is not a universal constant. What counts as a confident margin depends on the domain: the vocabulary an insurer uses, how templated the questions are, how many lines a typical answer spans. That is the honest limit of this technique. The signal is cheap and general, but the cutoff is a business decision, tuned per corpus against a labelled set the way Article 20 (evaluation) measures every other brick. What travels is the shape: read the confidence the retrieval brick already computed, and spend the model only when it is low.
This also composes with the dispatcher from Article 6C (dispatching the parsed question: chunk strategy, model tier, activations). The dispatcher already decides, per question, which bricks to activate. The route is one more activation: a skip_generation flag the dispatcher sets when the keyword margin clears the bar, alongside the ones it already sets for chunk strategy and model tier. The router is not a new stage bolted on: it is the dispatcher learning to say “this one does not need the model.”
4. What it saves: two seconds, or nothing
The saving is the whole point, so measure it. The fast path is deterministic: score the lines, read the margin, return an answer. Timed on the broker corpus, the entire routing decision runs in about 0.1 millisecond, with no network call at all.
The full pipeline is three hosted-model calls in series. Question parsing, the retrieval arbiter, and generation each cost a few hundred milliseconds to over a second, and they run one after another. For an easy question that adds up to roughly two seconds of the user watching a spinner, to return a number a keyword match had already isolated.
Two seconds against a tenth of a millisecond is a different order of magnitude, not a tuning gain, and it lands on exactly the questions users ask most: the simple, templated ones. The cost side moves the same way. Every routed question is three model calls not billed, and at a support desk running the same fifty templated questions across ten thousand contracts, those easy questions are most of the traffic. So the fast path is close to free as well as fast.
5. Indicators, not a bigger model
The margin is one indicator. It is the cheapest one and it settles the clean cases, but it is not the whole optimization. Making a thorough pipeline fast is a matter of finding, for each decision the pipeline currently hands to a model, a signal that decides it without one. There are several fronts, and each is real work.
5.1 The fronts
The answer that already exists. Most enterprise questions get asked again. If a question matches one already answered, on a document that has not changed, the answer is on file: return it, with no retrieval and no model. The catch is the match itself. Deciding “this is the same question” is its own classification: normalize the wording, compare the keywords, compare the question’s embedding against a store of past questions, and accept only a close hit. That store is the templated-question base from Article 6C (dispatching the parsed question) and the corpus tables from Article 25 (the storage model); the indicator reads them.
The retrieval that already answered. The signal this article uses. Score the lines, and when one clearly carries the answer’s shape, the answer is found, and generation has nothing left to do.
The question type, from the expert dictionary. To know a question wants one line and not a paragraph, the pipeline needs its answer shape, and today a model reads it. But for a known concept the shape is already written down: the dictionary maps premium to (single, amount). Look the concept up and you have the type deterministically, no model. The model is left for the genuinely new question the dictionary has never seen. This is amplify the expert moved to the control flow: the expert’s dictionary decides the route, not the LLM.
5.2 Where the indicators end and the agent begins
Two boundaries sit at the edge of this. The full engineering of latency and cost, the small-model tiers, the caches, context reduction, streaming, and the measurement of cost-per-query and tail latency, is Article 21 (cost and latency). This article works the single highest lever, not calling the model, and points there for the rest.
The other boundary is agentic RAG, and it falls on one precise line: who makes the route decision. Made by an indicator, a keyword score, a cache hit, a dictionary lookup, the decision is deterministic, and it belongs to the Vol.1 engineering this article is part of. Made by the model, classifying the intent and planning the steps, it is what makes a system agentic. The agentic follow-up to this series carries the same deterministic-first, model-only-if-ambiguous pattern up a level, to picking a tool rather than just a route. The two are not rivals but layers: the deterministic indicators handle the easy majority for free, and the agent is the fallback for the questions the indicators cannot classify. Building indicators that are fast, and often just classical rules or a small trained model, is how you keep the agent, and its latency, for the cases that truly need it.
6. Conclusion
The reflex when a RAG pipeline feels slow is to reach for a faster model. The cheaper and larger win is to call the model less. The retrieval brick already computes, for free, a signal that separates the questions answered by a keyword match from the ones that need the model to reason: the top keyword score and its margin. Route on it. Send “what is the premium?”, one line at score 5, straight to a deterministic extractor, and keep the model for “which guarantees can I avoid in my case?”, where the flat score is the pipeline telling you it needs help.
The threshold is yours to tune, per corpus, against a labelled set. The mechanism is portable, and it is the same lesson the series keeps landing on: the expert’s keyword work is not a fallback under the model, it is often the whole answer, and knowing when it suffices is what keeps the pipeline fast.
7. Further reading and sources
Earlier in the series:
- A production RAG pipeline for PDFs: relational parsing, TOC retrieval, typed answers. The upgraded pipeline this article optimizes, brick by brick.
- Dispatching the parsed RAG question: chunk strategy, model tier, activations, audit. The dispatcher the router extends with a skip-the-model activation.
- Retrieval is filtering, not search: a mental model for enterprise RAG. Where
co_occurrence_scorecomes from: keyword filtering online_df.
Sources
- The cascade idea, run cheap stages first and stop as soon as one is confident, is the classic detection cascade of Viola and Jones (Rapid Object Detection using a Boosted Cascade of Simple Features, CVPR 2001), applied here to model calls instead of image windows.
- FrugalGPT (Chen, Zaharia, and Zou, arXiv:2305.05176, 2023) makes the same move for LLM APIs: a cascade that queries a cheap model first and escalates only when the cheap answer is not confident.
- RouteLLM (Ong et al., arXiv:2406.18665, 2024) learns a router that sends easy queries to a small model and hard ones to a large one, the same triage this article applies one step earlier, before any model at all.
The broader cost-and-latency treatment is Article 21 (cost and latency).