The user asks “what is the effective date of this policy?”. Retrieval returns five candidate line-windows and passes them all to the LLM in one prompt. The LLM reads all five to extract the same date the first candidate already had. Chunks two to five were signatures, footnotes, and a paragraph about historical dates. Paid for nothing. Send the top-1 first, ask the LLM if that is enough, and stop when it says yes: sequential feeding cuts the token cost by 80 % on this class of question. The rest of the article catalogues where sequential wins, where a single-call over all K is the right default, and how the question parser dispatches between the two.
This article is a companion to Enterprise Document Intelligence, the series whose philosophy is laid out in Amplify the Expert, the series that builds enterprise RAG from four bricks (document parsing, question parsing, retrieval, generation). It sits between Article 8 (generation) and Article 9 (upgrading the mini-RAG) and develops one specific decision: how do you feed the top-K retrieved candidates into the generation brick?. The answer most pipelines have is “all K at once”. This article catalogues the second regime (sequential, top-1 first) and shows when each one wins.
The naive baseline this article pushes back on
Naive RAG ships batch by default. Retrieval returns top-5, the LLM gets all 5, the answer comes back. It works, and on hard questions (comparison, listing) it is the right choice. But the silent cost is paid on every other question: the easy factual ones where the top-1 already had the answer. This article walks the second regime and the dispatch that picks per question.
📓 The runnable notebook for this article is on GitHub: doc-intel/notebooks-vol1. It sends the same questions through both batch and sequential generation, prints the per-question token cost and the two sufficiency booleans (answer_found, complete_answer_found) that stop the loop, and reproduces the dispatch table on your own machine.
1. Two regimes for feeding the top-K to generation
Retrieval hands generation an ordered top-K. There are two ways to feed it in, and they cost very differently.
1.1 The fixed top-K pipeline and what it wastes
The default pattern across most RAG tutorials looks like this:
top_k = retrieval(question, k=5)
answer = generation(question, top_k)It runs in two steps for every question. Token cost is roughly generation_cost(question + 5 chunks of context). Latency is roughly retrieval + one LLM call. And the LLM happily processes the 5 chunks even when the top-1 was already sufficient and chunks 2..5 added zero information.
Concretely: the user asks “what is the effective date of this policy?”. Retrieval returns 5 line-windows where the keyword "effective" appears. The first one is the answer (“effective from January 1, 2026”). Chunks 2..5 are signatures, footnotes, and a paragraph about historical effective dates of past policies. The LLM reads all 5 to extract the same date the first one already had. On a corpus of one document, the cost is rounding error. On a corpus of 50k policies, this is real money per month.
1.2 Sequential: top-1 first, sufficiency predicate, escalate if needed
The sequential regime treats the K candidates as an ordered list and asks the generation brick to validate sufficiency at each step:
for i, candidate in enumerate(top_k):
answer = generation(question, [candidate])
if answer.answer_found and answer.complete_answer_found:
breakThe sufficiency signal lives inside the typed contract Article 8A introduced. The AnswerWithEvidence schema exposes answer_found (was the question’s information present in this candidate?) and complete_answer_found (was the entire answer present, not a fragment?). The loop reads those fields, not a custom heuristic.
In the effective date example above: generation runs once on the top-1 candidate, the LLM returns answer_found = True, complete_answer_found = True, the loop exits. Tokens spent: generation_cost(question + 1 chunk of context). That is 1/5 of the batch cost for this question, on a pipeline that handles thousands of similar lookups per day.
1.3 Batch: send all K at once, let the LLM arbitrate
Batch mode keeps the default behaviour: one LLM call sees all K candidates and produces one typed answer. Its case is built on three question types where sequential breaks down:
- Listing questions: “list all exclusions in this contract”. The answer is every matching candidate, not the first. Sequential would stop at top-1 (one exclusion found) and miss the other four. Batch is the only correct mode.
- Comparison questions: “is the premium higher than the previous year’s?”. The answer requires both candidates (this year + last year) in the same call so the LLM can compare them. Sequential would extract each one independently and lose the join.
- Tight-score retrieval: when the top-K candidates’ relevance scores are within 5% of each other, retrieval cannot reliably promote top-1 to the top. Batch lets the LLM arbitrate with the full evidence visible.
Cost analysis: batch always pays the generation_cost(question + K chunks of context) once. Sequential pays generation_cost(question + 1 chunk of context) in the easy case (top-1 sufficient) and generation_cost(question + 1 chunk) × K in the worst case (every candidate insufficient). On a typical enterprise corpus with K = 5, sequential is cheaper on average for factual lookups (~80% of typical traffic) and more expensive for listing / comparison (~20%).
2. The dispatch decision: per question, not per pipeline
The clean architecture does not pick batch or sequential globally. It picks per question, using the parsed question_df row from brick 2. Question shape, decomposition pattern, and intent drive the choice:
The dispatcher reads question_df.answer_shape and question_df.decomposition and routes. Naive RAG has no way to make this distinction because it has no parsed question to read from.
The same routing table holds across sectors and professions. Different domains carry the same shape patterns and the same sequential / batch decision flows out of them:
In every row, the sequential column is a single typed value (Amount, Date, Boolean) and benefits from the top-1 stop. The batch column is a list or a comparison and needs all K candidates visible at once. The dispatcher’s table covers all five sectors with the same logic.
3. The sufficiency signal
Sequential mode leans on one thing: the generation brick reporting whether the candidate it just read was enough. That signal, and the rules that stop the loop, live here.
3.1 Where the sufficiency signal lives in the typed contract
Sequential mode only works if the generation brick can self-report whether the candidate it just saw contained the answer. The typed contract from Article 8A is what makes this possible:
class AnswerWithEvidence(BaseModel):
value: Any
evidence: list[Span]
answer_found: bool
complete_answer_found: bool
confidence: float = Field(ge=0, le=1)
caveats: list[str] = []The sequential loop reads answer_found and complete_answer_found, not a confidence float or a custom heuristic. The clean separation between the two booleans (from Pattern 4 of Article 8ter) is what makes the loop deterministic. Found + incomplete says “continue to the next candidate”; found + complete says “stop”; not found says “continue or give up at K”.
A confidence float would force a threshold (e.g. “stop at 0.8”), and that threshold drifts model to model. Two booleans do not drift.
3.2 Bounded iteration: even sequential has to stop
The sequential loop has three exits, not one:
- Sufficiency:
answer_found and complete_answer_foundon a candidate. Stop, ship the answer. - Exhaustion: every one of the K candidates seen, none sufficient. Stop, return
answer_found = False(a first-class answer the validator passes through). - Budget: a token or time budget set by the dispatcher. Useful when the corpus is large and a runaway sequential loop would blow the cap. Same shape as the bounded iteration M4 (loop engineering) names.
Naive sequential implementations skip the third exit and burn tokens forever on edge cases. The series version sets a budget upfront and the dispatcher logs it on every call.
4. Cost, and where the series stops
Two regimes, two cost profiles, and one boundary the series does not cross.
4.1 A concrete cost comparison on a hundred-question batch
To make the trade-off real, here is a back-of-envelope for an enterprise insurance Q&A workload:
- 100 questions / day, K = 5, average chunk size = 600 tokens.
- Batch baseline: 100 × generation_cost(question + 5×600 tokens) = ~330k input tokens per day for generation.
- Sequential (80% top-1 sufficient): 80 × generation_cost(question + 600) + 20 × generation_cost(question + 5×600) (worst case for the 20% complex questions) = ~115k input tokens per day.
The ratio is 65% saving on input tokens for generation on this workload. The exact ratio depends on the easy-question proportion and the chunk size; the principle is that sequential is the cheap default once the typed contract is in place. Batch is reserved for the question types that need it.
4.2 The agentic temptation, and where the series stops
A natural next step is to let the LLM decide between batch and sequential per question (agentic dispatch). The series stops short of this. The dispatcher in Section 2 is deterministic-dispatcher (one of the three approaches catalogued in Article 6C), not LLM-decides. The reason is the same one that holds across the series: audit. The same question on the same day must route the same way; an LLM that re-plans the dispatch per call cannot give that guarantee.
If you need a stronger agentic loop (the LLM picks which candidates to look at, in what order, with what scope), the bigger article on adaptive RAG loops is the right place. This article keeps the scope to the deterministic sequential vs batch decision driven by the parsed question.
5. The decision belongs to the parser, not the LLM
The fixed top-K + batch pipeline is the right default for the question types where every candidate matters. It is the wrong default for the factual lookups that make up most enterprise traffic. The series adds two pieces: the typed sufficiency signal (answer_found, complete_answer_found) from Article 8A, and the dispatch table from Section 2. Together they let the generation brick stop after the first candidate when that is enough, and process all K when the question type requires it. The decision is made by the parser, not by the LLM, which keeps the audit trail intact.
6. Further reading and sources
The sequential / batch decision sits at the boundary between retrieval and generation. The articles that frame each side:
- An LLM as arbiter in RAG retrieval (Article 7C). The brick that produces the ranked top-K this article iterates over.
- Stop returning text from RAG (Article 8A). The typed contract where the sufficiency signal (
answer_found,complete_answer_found) lives.