typed fields to pull out of a stack of real documents: amounts, dates, coverage limits, one structured value per field. The default reflex is to send every field to a GPT-4-class hosted API. It works, and the bill is the largest line in the pipeline’s running cost. Most of those fields are plain lookups a much smaller model handles fine, and you are paying flagship prices for all of them.
The obvious reaction is to swap in a small model and pocket the savings. Done blindly, it backfires. A model too weak for the job returns broken typed output, or fumbles a transformation, and if nobody checks, the wrong value ships. So the answer is not “use the big model” and it is not “use the small model”. It is to choose the model the way the earlier bricks choose a method: by criteria, then verify, and escalate only when the verification fails.
This article is a companion to Enterprise Document Intelligence, the series whose philosophy is laid out in Amplify the Expert, the series that builds enterprise RAG from four bricks: document parsing, question parsing, retrieval, generation. It develops one decision inside the generation brick: which model runs the call, and what happens when the cheap one is not good enough? It leans on Article 6C (dispatch), which already reads a suggested model tier off the parsed question, Article 8C (validation), the signal that triggers an escalation, and Article 10 (adaptive parsing), the same start-cheap-escalate-on-demand shape applied to the parser.
Everything below is backed by a real benchmark: a field-extraction task of a few hundred typed fields across dozens of documents, plus a focused sweep of twenty local models against one hosted flagship, all at temperature 0 with JSON output. The numbers are aggregate. No document, field label, or figure from that benchmark appears here.
📓 The runnable notebook for this article is on GitHub: doc-intel/notebooks-vol1. It runs the same field through the cascade, prints the per-field cost and the validation verdict at each rung, and shows the escalation firing only on the fields the cheap local model gets wrong.
1. Two angles: cost, and the loop
There are two reasons to care about model selection, and they reinforce each other.
The cost angle is direct. Flagship models cost an order of magnitude more per call than a small local one. On an extraction job with thousands of field-level calls, or a corpus assistant answering thousands of questions a day, the model bill dominates the running cost. Most of those calls do not need a flagship. Routing the simple ones to a cheaper model is money left on the table until you do it.
The loop angle is about correctness under that constraint. You cannot cut cost by guessing that a small model will cope. You start with a model, check whether its answer holds, and escalate to a stronger one only when the check fails. The loop is what makes the cost saving safe: the cheap model is the default attempt, not a gamble, because a failed attempt is caught and retried on a bigger model rather than shipped.
Cost pushes you down the model ladder. The loop stops you from falling off it. This cheap-first, escalate-on-failure shape has a name in the literature: an LLM cascade.
2. What the numbers say
The design in Section 3 rests on two measured facts. Take them first, because they are the reason the cascade is shaped the way it is.
2.1 The sweep: bigger is not better
Look at what twenty local models did on the raw one-shot task: read the retrieved snippet, return the typed value. Same prompt, same fields, temperature 0.
Three conclusions come straight off that chart, and each one shapes the design.
Bigger is not better. A 4B model (qwen3:4b) and a 7B (mistral:7b) top the local field. A 12B and a 14B (gemma3:12b, phi4:14b) sit below them. Two models from the same family, one twice the size of the other, tie. Parameter count is a poor predictor. The model family and how it was trained matter more. This is exactly why the loop must not climb a blind size ladder: the next rung up is often not the next size up.
There is a floor. The sub-2B models fall off a cliff (qwen2.5:1.5b at 12%, llama3.2:1b at 6%), and a 0.5B model we tried returned empty JSON on most fields: not a wrong answer, no answer at all. Starting there burns a loop iteration on a rung that was never going to hold. Criteria exist to start above the floor, not at it.
Cost is not speed. The hosted flagship answered in about 1.4 seconds per field, faster than most local 7B-to-14B models (roughly 2.6 to 7.6 seconds each on a single consumer GPU). The small-local play buys a lower bill and documents that never leave the machine. It does not buy latency. If speed is the constraint, the cheap local model is often the wrong default, and that belongs in the dispatcher’s criteria.
One more sobering number. The best small local model, run on its own on the raw task, got roughly a third of the fields right. Fast (a few seconds a field) and not good enough to ship. A small local model is viable only when the loop backs it: a tight retrieval snippet, the right prompt content, validation on every field, and code doing the hard formatting. The next three sections are that backing.
2.2 The biggest lever is prompt content, not model size
Before spending a bigger model, spend the prompt. The single largest jump in the whole benchmark did not come from more parameters. It came from putting the task’s business vocabulary, the glossary the fields assume, into the prompt.
Both models jump. The small local model goes from 38% to 62% correct. The flagship goes from 62% to 100%. The flagship is not magic without the glossary either. It needed the definitions to finish the job.
The concrete failure this fixes is vocabulary the model does not share with the document. A field labelled deductible per claim in one document is retention in another and a bare number in a third; a premium is called one thing by one insurer and another thing by the next. A small model shown the raw snippet guesses. Hand it the definition of the field it is filling, the synonyms, the units, and the guess turns into a read. This is the same brick-2 vocabulary that lifts retrieval, reused at generation time, and it is cheaper than any model upgrade because it helps every rung at once.
3. The cascade: criteria, loop, and splitting
Here is the mechanism the two facts point to. Pick a starting rung by criteria, validate its output and escalate only on failure, and split the task instead of escalating when the failure is a hard transformation. Three moves, in that order.
3.1 Pick the starting model by criteria
The starting model is not the smallest one you have. Three criteria set a sensible floor, so the loop begins at the right rung rather than the bottom.
Confidentiality. A confidential document cannot be sent to a hosted API at all. That forces a local model regardless of what would be cheapest or strongest in the cloud, and the starting choice becomes the best local model that fits the machine. (Detecting and labelling a document as confidential is a later-volume concern; this article assumes the label arrives as input.)
Typed-output reliability. The series answer is always a typed object. The extraction contract from the benchmark is small and strict: a pure number, its currency, the precision qualifier kept separate, and a boolean for whether the field was there at all.
class ExtractedValue(BaseModel):
value: float | None
currency: str | None
precision: str | None
found: boolStrong models emit that cleanly. Weaker and smaller ones emit it less reliably, dropping the currency, folding "per event" into the number, or inventing a value when they should have set found=False. So a small starting model is viable only when paired with validation: you do not trust its typed output, you check it (Article 8C) and let the failure drive the escalation.
Transformation complexity. Some jobs hide a hard one-shot transformation that trips small models even when retrieval was perfect. That is the third criterion, and it has its own fix in §3.3.
The dispatcher that reads these three is small.
class GenerationPlan(BaseModel):
model: str
needs_validation: bool
needs_step_split: bool
max_escalations: int = 2
def plan_generation(doc, shape):
if doc.confidential:
return local_plan(shape)
return tier_plan(shape)local_plan pins the best local model and sets needs_validation=True, because a small model’s typed output always gets checked. tier_plan starts from Article 6C’s suggested tier and only asks for validation when that tier is small. Both read needs_step_split off the answer shape, the third criterion, developed in §3.3.
3.2 Validate, then escalate only on failure
With a starting model chosen, generation becomes a short bounded loop, not a single call.
def generate_with_escalation(question, context, plan, ladder):
for model in ladder.from_(plan.model, up_to=plan.max_escalations):
answer = generate(question, context, model=model)
if validate(answer, context): # Article 8C
return answer
return NotAnswered()Run the chosen model. Validate the result with the checks from Article 8C: is the typed shape correct, do the cited spans exist, does the quoted text actually appear in the source. If validation passes, ship, and the cheap model just did a flagship’s job for a fraction of the cost. If it fails, escalate: rerun the same field on the next model up, and validate again. The loop is bounded, so a pathological field cannot spin forever; it escalates to the strongest model and, if even that fails, returns a typed not answered rather than a wrong value.
This is the generation-side echo of Article 10. There, a page starts on the cheap parser and only escalates to a heavier one when a cheap check says the parse is not good enough. Here, a field starts on the cheap model and only escalates when validation says the answer is not good enough. Same loop-engineering shape: pay for the expensive tool only on the cases that need it, and let a deterministic check, not a guess, decide which cases those are.
3.3 Split the task, and the confidentiality cost you measure
Escalating to a bigger model is not the only fix for a failing one, and it is not always the right one. Some failures are a complex transformation, not a weak model, and the cure is to decompose the step rather than pay for more capacity.
The worked example is a monetary amount in a clean numeric format. Shown 3M, a small model may not reliably return 3000000, a three followed by six zeros; it stumbles on doing the recognition and the scaling in one shot. A bigger model papers over it, but so does splitting the transformation: let the model only point at the source line, and let deterministic code turn 3M into 3000000. Each half is easy. The model reads, the code computes, and the hard formatting never depends on model capacity at all.
That split has a second payoff: the raw document can stay entirely local. The model picks a line, code formats it, and only anonymous fragments are ever eligible to leave for a hosted formatter downstream. But the split is not a free lunch, and the benchmark is blunt about it. Running the fully-local “model points, code formats” architecture lowered accuracy on the same fields compared with letting a capable model extract and format in one shot. The reason is simple: pointing at the right line is itself hard, and a wrong line dooms the field before the code ever runs.
So the split is a real tool with a real cost. Reach for it when the failure is a hard transformation the model keeps missing, or when confidentiality forces the document to stay local. Measure the accuracy you trade for it. Do not assume decomposition is always a win; on a task where a capable model can extract and format cleanly in one pass, one pass wins.
The loop therefore has two escape hatches when validation fails, not one: escalate the model, or split the task into steps the current model can handle. The dispatcher’s needs_step_split flag records which one this field gets.
4. Reconciling with question-side dispatch and validation
This selection is not a new idea bolted on. It closes a loop the series already opened.
Article 6C (dispatch) reads a suggested model tier off the parsed question: a simple factual lookup and a multi-hop synthesis carry different tiers before generation ever runs. That is the question-side signal. This article adds the document-and-answer-side signal (confidentiality, typed-output reliability, transformation complexity) and, crucially, the feedback: 6C proposes a starting tier, generation runs it, and Article 8C’s validation confirms the tier was enough or triggers the escalation. The three fit together: 6C suggests, generation attempts, 8C judges, the loop escalates.
5. Conclusion
Model choice is a brick decision, not a constant. The cheap model is the right default when a criteria-driven dispatcher picks a sensible starting rung, validation guards the output, and a bounded loop escalates only the cases that fail, splitting the task instead when the failure is a hard transformation rather than a weak model. The sweep is the reason to trust the shape and to distrust the shortcuts: size does not order accuracy, a small local model on its own gets a third of the fields, and the biggest single lever is the vocabulary you put in the prompt, not the parameter count you pay for. Cost comes down because most fields never leave the cheap rung. Correctness holds because the ones that need more get it, on evidence, not on a hunch.
6. Further reading and sources
The series articles this cascade sits between, already published:
- One parsed RAG question, four decisions (Article 6C). The question-side signal that proposes a starting model tier before generation runs.
- Stop returning text from RAG (Article 8A). The typed answer contract a small model has to fill.
- Assemble each RAG generation prompt from a base plus the rules each question needs (Article 8B). How each answer shape’s prompt is composed before the model runs.
- Validating the RAG answer before the user sees it (Article 8C). The validation pass that confirms the cheap rung was enough or triggers the escalation.
On the model-cascade idea, the external anchors:
- Chen, Zaharia & Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (arXiv 2305.05176, 2023). The canonical cheap-to-expensive LLM cascade with a scorer between the rungs.
- Madaan et al., AutoMix: Automatically Mixing Language Models (arXiv 2310.12963, 2023). Self-verification on the small model’s answer decides whether to route up: the same verify-then-escalate loop this article builds.
- Ong et al., RouteLLM: Learning to Route LLMs with Preference Data (arXiv 2406.18665, 2024). A learned router between a strong and a weak model, the data-driven cousin of the criteria dispatcher here.