companion to Enterprise Document Intelligence, the series whose philosophy is laid out in Amplify the Expert. It zooms in on brick 3 (retrieval) of the four-brick architecture and surfaces the lessons most tutorials skip.
The mainstream story has retrieval as embed the question, return top-k by cosine, optionally rerank. We disagree with almost every part of it. Retrieval is filtering on structured tables, not searching free text. Embeddings are the optional fallback, not the foundation. Anchor and context are two granularities, not one. Each of these is a position we can defend, with consequences you can measure.
📓 Runnable companion notebooks are on GitHub: doc-intel/notebooks-vol1.
The naive baseline this article pushes back on
The naive pipeline chunks the document, embeds every chunk, embeds the question, ranks by cosine. That single signal is opaque, and it throws away the document’s structure. We keep the document as line_df + toc_df and run three retrieval signals in parallel (keyword on lines, TOC reasoning, embedding cosine), then let an LLM arbiter rank once at the end with all three sets of hits in view.
Below are the six untaught lessons of this brick.
Lesson 1 – Retrieval is filtering, not searching
Once parsing is done, retrieval is a SQL-like filtering problem over line_df and toc_df, the reverse of the chunk-embed-cosine-top-k framing. The shift is simple to state: the question has columns, the document has columns, and retrieval is the join.
Search and filter are not synonyms; the two operations work differently. Search scores every candidate on a continuous similarity (cosine, BM25), forces a top-k cutoff, and always returns something, even when the answer is not in the document. Filter applies a boolean condition (line.contains("X"), toc.title in [...]), keeps every row that matches and no more, and can return zero rows when the document does not carry the answer. The difference that shows up in practice is the audit trail. A filter’s condition is one line of inspectable code that runs the same way in six months. A search’s ranking depends on which dimensions of the embedding happened to matter, and you cannot replay that judgment without re-running the model.
Take the question “What positional encoding does the paper use?” on the Attention paper. Naive RAG embeds it, scores three hundred-odd chunks, and returns the top five. The series pipeline filters instead: it keeps the four lines of line_df that contain "positional encoding", and the one toc_df section whose title contains "positional", 3.5 Positional Encoding. The arbiter then reads both, the lines as the anchor and the section as the scope. No cosine ran, and had the phrase been absent the filter would have returned nothing, which is the honest result.
→ Article 7A: Retrieval is filtering, not search lays out the mental model.
Lesson 2 – Anchor and context, kept apart
You anchor on the single line that mentions “premium” (precise) but pass the whole surrounding section to generation (sufficient context); conflating them breaks precision and coverage in one move. Top-k forces you to pick: tiny chunks lose context, huge chunks lose precision. We get both, by keeping them apart.
Two different notions hide inside retrieval, and they are easy to run together. The first is the anchor: the exact place the answer sits, which you reach through a keyword. Ask for the premium amount and the keyword premium drops you on the right line, the number written next to the word. Ask for the exclusions and the keyword exclusions drops you on the right section. In both cases the keyword is what locates the answer.
The second notion is the context around that anchor: how much text you actually have to read to answer. For the premium it is almost nothing, the value is right there in the line. For the exclusions it is three pages, because the list runs across them. Same precise anchor, very different amount of surrounding text. This is where a single top-k cut breaks down, because it uses one chunk size for both. Small chunks find the premium line but cut the exclusions list in half. Large chunks catch the whole exclusions section but bury the premium line in a page of noise. So we keep the two apart: the keyword marks where the answer is, the scope sets how much surrounding text travels to generation, and retrieval returns them together as a typed pair.
→ Article 7A: Retrieval is filtering, not search draws the line between anchor and context.
Lesson 3 – Embeddings come last, not first
Keywords always run (cheap, deterministic); the document’s own TOC is a first-class retrieval method; embeddings are the optional final signal, only when vocabulary mismatch is expected. The 2024-era reflex starts with embeddings; we leave them for the cases where the cheaper signals failed.
Take a factual lookup on an insurance policy: “effective date?”. Naive RAG embeds the question and returns five chunks. The series pipeline first runs a keyword search for "effective" and "date", finds the single line that carries both, and stops there. The embedding step never runs, because the cheap signal already answered. What it cost was one regex pass over line_df, a few milliseconds, instead of a cosine search over every chunk. Embeddings are held back for the cases where keywords come up empty, not spent by default.
→ Article 7B: Finding the right anchors builds the three-signal pipeline.
Lesson 4 – Keywords prove absence; embeddings cannot
A zero on keyword search means the answer is genuinely not there; a zero on embedding similarity could be absence or just different words, so embeddings are a refinement, not a decision gate. This asymmetry is the case for keywords as the primary signal in enterprise RAG.
Take the question “does this contract cover earthquake damage?” on a flood-only policy. A keyword search for "earthquake" returns zero matches in line_df, so the pipeline can ship answer_found = False and mean it: the word is simply not in the document. Embedding cosine behaves differently. It returns five chunks anyway, the lines that sit closest in meaning, about natural disasters and flood events, and a model reading them can talk itself into a wrong yes. That is the asymmetry. A zero from keywords is real absence; a low cosine score is only distance. So keywords, not embeddings, are what get to decide whether the answer exists at all.
→ Article 7B: Finding the right anchors explains the keyword-first discipline.
Lesson 5 – Co-occurrence beats BM25 on narrow corpora
BM25 ranks by term frequency, but the enterprise answer shape is one mention of a topic next to a specific value, so co-occurrence boosts and high-value regex anchors beat statistical IDF on narrow corpora. The IDF assumptions break on a 20-document corpus where every term is “rare” by Wikipedia standards.
Take “what is the deductible amount?”. BM25 ranks by how often "deductible" occurs, so a glossary section that repeats the word a dozen times floats to the top, even though it carries no figure. Co-occurrence works on the answer’s shape instead: it favours the lines that hold both "deductible" and a number. The actual policy line, "the deductible is $1000", ranks first because the word and the value sit together, and the model can read the amount straight off it. On a twenty-document corpus, where IDF treats almost every term as rare, that shape signal beats the raw frequency count.
→ Article 7B: Finding the right anchors measures co-occurrence against BM25.
Lesson 6 – One LLM pass over the TOC
Handing the 20-100 row toc_df to a small model and asking which sections answer the question costs one cached call and catches the paraphrases (“exit early” ≈ “Termination”) keyword matching misses. TOC reasoning is one of the most under-used retrieval signals in production RAG.
Take “when can I leave the policy early?”. Substring matching on "leave" finds nothing in the TOC, because the section is called “Termination and Cancellation” and shares no word with the question. Hand the whole TOC, twenty-eight rows, to a small model in a single prompt and ask which sections fit, and it returns “Termination and Cancellation”: it reads “leave early” and “termination” as the same idea. The call is cached, so it runs once and stays deterministic afterwards, and it recovers the section that keyword matching walked straight past.
→ Article 7B reasons over the TOC, and Article 7C: An LLM as arbiter adds the arbiter.
The six lessons share one move: refuse the chunk-embed-cosine reflex, and treat retrieval as filtering on structured tables instead. Keywords always run because they prove absence; the TOC is a first-class signal because the document already declared its structure; embeddings are the optional refinement, not the foundation. The deep-dives (7A, 7B, 7C, 7bis) ship runnable code on real documents; this piece is the catalogue that points at them.
Across sectors and professions
The same three-signal retrieval pattern ( keyword on line_df + reasoning on toc_df + embedding fallback ) holds in every domain. The vocabulary and the TOC depth differ; the signal hierarchy does not. Five sectors below, one retrieval pattern, one audit trace per call.
Embeddings fire only on the medical row, where the user’s vocabulary ( “tachycardia” ) diverges from the document’s ( “rapid heart rate” ). The other four rows resolve entirely on keyword + TOC. Keywords prove absence (Lesson 4), the TOC catches paraphrases (Lesson 6), and the anchor / scope split keeps precision and context apart (Lesson 2) in every row. The cost gradient is real: the four keyword-resolved rows run in milliseconds with zero LLM tokens; the medical row pays for one embedding pass and one arbiter call.
Sources and further reading
The mainstream literature on retrieval is shaped by web-scale search and shorter consumer corpora. The series stance assumes a small enterprise corpus where the structure is known and the vocabulary is the asset.
- Retrieval is filtering, not search (Article 7A). The published mental-model article: retrieval as filtering on structured tables.
- Embeddings Aren’t Magic (Article 2). The published failure-modes catalogue for embedding similarity.
- Rerankers Aren’t Magic Either (Article 2bis). When the cross-encoder pays off and when it does not.