10 Positions for Enterprise RAG That Mainstream Tutorials Get Wrong | Towards Data Science

way to build enterprise RAG, and it disagrees with a lot of standard practice along the way. RAG is not machine learning; embeddings are not magic; a chunk-size sweep optimizes the wrong thing; the answer schema matters more than the model. None of these are neutral, and each one changes what you build. This piece collects the positions the rest of the series argues from, then maps the series article by article, so you can go straight to the argument you want to check.

This article is a manifesto of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks. Ten positions where the series breaks with mainstream RAG tutorials, followed by the map of the series through them.

📓 The series’ companion notebooks live on GitHub at doc-intel/notebooks-vol1. Each one runs a brick end to end on a real PDF, so you can watch these positions play out: structure-first retrieval firing before any embedding, the typed answer coming back with line-level citations, evaluation sliced by failure mode instead of one aggregate score.

The standard RAG tutorial reads the same everywhere. Chunk the documents, push the chunks into a vector store, embed the question, retrieve top-k by cosine similarity, optionally rerank, send the hits to an LLM. Vendor decks, framework quickstarts, and conference talks all repeat it. The pattern works on hello-world examples (a Wikipedia paragraph, a short PDF, a curated FAQ) and starts wobbling the day a real enterprise document hits it.

Here are the ten positions this series defends against that recipe. Each one is a recurring editorial choice the architecture rests on. None of them is mine alone. A few practitioner voices (Hamel Husain, Eugene Yan, Jason Liu, parts of the Anthropic engineering blog) push some of these in talks and posts. The contribution here is treating them as a connected system, not isolated hot takes.

If you read only one article from the series and want the editorial position behind every concrete choice, this is it.

The ten fall into three layers. The first four (positions 1 to 4) take the tutorial’s retrieval recipe head on: structure first, dictionaries before models, rerankers as a tool not a stage, never one vector store for everything. The next three (positions 5 to 7) set the frame the recipe ignores: what enterprise actually means, who the system amplifies, who picks routes at runtime. The last three (positions 8 to 10) cover the audit dimension: per-failure evaluation, relational structure between bricks, citations as evidence.

1. Vector stores are a fallback, not a foundation

The default tutorial wires a vector store as the entry point of the pipeline. Everything goes through cosine similarity, then a reranker patches what cosine got wrong. The series inverts this. Structure-first retrieval (the document’s declared TOC, the corpus index, expert keywords) handles the bulk of real questions on enterprise corpora. Embeddings come in as a safety net for the residual cases (paraphrase, cross-language, internal acronyms), not as the default first stage.

The argument is not a benchmark, it is interpretability. With keyword matching on a line_df (the per-line DataFrame the parser produces, developed in Article 5) and filtering against the document’s declared TOC, you can read why a passage was retrieved: the matching terms, the section path, the line range.

With cosine similarity, you cannot. The vector says “these two passages are close in some 768-dimensional space” and that is the whole explanation available.

On enterprise documents where every retrieval has to be justified to an auditor or a domain expert, the interpretability gap is what drives the architecture, not a recall@k number. Embeddings still earn their place, but as the last method in the funnel, not the first.

Where it shows up: Articles 2 (embeddings’ failure modes), 7 (retrieval), 9 (the upgraded pipeline), 14 (the corpus problem).

2. Expert dictionaries beat better embedding models

The synonym problem is the one embeddings are supposed to solve. Premium matches cost. Termination matches cancellation. Franchise matches deductible. In practice, a concept_keywords_df table maintained by domain experts (the satellite table built in Article 6) solves the synonym problem more reliably than an off-the-shelf embedding model, and usually a fine-tuned one, for the vocabulary the experts know. A fine-tuned model still wins on unseen phrasings and languages the dictionary has not captured yet, which is why embeddings stay in the funnel as the fallback. The disambiguations, the cross-product equivalences, the internal product codes that mean specific things in this company and nothing elsewhere: experts know these. Embeddings have to guess them.

Recall@k is what people use to pick an embedding model from a benchmark. It is not the metric the production pipeline optimizes. Once you accept that the synonym work belongs to the expert dictionary, fine-tuning becomes a luxury and embeddings become a discovery tool (run them a few times to find aliases for the dictionary, then keep retrieving on the validated dictionary forever).

Where it shows up: Articles 2 (embeddings’ failure modes), 6 (question parsing), 7 (retrieval).

3. Rerankers are a secondary tool, not a primary stage

Cross-encoder rerankers have a legitimate place in the literature. They sit between cheap embedding similarity (high recall, fuzzy precision) and expensive LLM judgment (precise but slow), and they earn their cost when the candidate pool is large (the top-100 to top-1000 passages a first-stage retriever returns on academic benchmarks like MS MARCO, where re-scoring with a cross-encoder gives a real precision win) and the upstream stage is weak. That is the setting the reranker papers came out of.

The enterprise approach the series defends works on small, scoped candidate sets produced by expert-vocabulary retrieval, structure-aware filtering, and classify-before-retrieve. By the time a reranker would run, the pool is already small and already-scoped, and a cross-encoder adds latency and complexity for marginal precision. The series treats reranking as a fallback for narrow cases (large undifferentiated corpora, ad-hoc questions, no curated pipeline), not as a default stage. Listing-type questions, where the model has to enumerate every relevant item, are the canonical reranker failure mode (the reranker puts the most-relevant exclusion first, silently demoting everything else below the cutoff).

Where it shows up: Articles 2 (embeddings’ failure modes), 2bis (rerankers), 7 (retrieval), 9 (the upgraded pipeline), 14 (the corpus problem), 20 (evaluation).

4. Refuse “connect everything to a vector store”

The vendor pattern is to wire every document type into one big vector index. It is optimized for the hyperscaler’s business model (one billable embedding call per chunk, one billable vector lookup per query), not for the customer’s accuracy. The series replaces it with a corpus-scale architecture:

• Classify documents before indexing (Article 14).
• Extract structured fields into a corpus index at ingestion (Article 15).
• Filter on the index before retrieval (Article 17).
• Route aggregation questions to a SQL agent, not to RAG (Article 17).

RAG handles content lookup. SQL handles counting and filtering. The corpus index sits between them. The vector store is one possible column in that index, used where it earns its place.

Where it shows up: Articles 14-20, the corpus chain: 14 (the corpus problem), 15 (preparing the corpus), 16 (ontology), 17 (querying the corpus), 18 (code architecture), 20 (evaluation).

5. A company is not Google

The “we are Google” playbook (benchmark recall@100 on a ten-million-document index, train a custom embedding model, run a learned reranker) does not transfer to enterprise contexts. A typical enterprise has a few hundred document types, a few dozen domain experts, and a recurring set of questions, not a corpus of ten million heterogeneous web pages.

Most architectural choices in the series follow from refusing the copy-paste. Hyperscaler-style retrieval optimizes for web-scale recall and per-call billing, not the customer’s accuracy. The right architecture for a few hundred document types and a known audience looks nothing like Google’s. Once you accept that, the rest of the series follows naturally.

Where it shows up: Articles 3 (RAG is not ML), 14 (the corpus problem).

6. Amplify the expert, do not replace them

Enterprise RAG is not open-domain QA over the web. It runs on documents that domain experts already know inside out: contracts, regulatory filings, technical reports, claims handbooks. Those experts have a vocabulary, a set of disambiguations, a habit of routing questions through specific document types. The system’s job is to scale that judgment, not to bypass it.

Most architectural mistakes follow from forgetting this. Autonomous agents bypass the expert, generic vector search on undifferentiated corpora ignores them, and fine-tuned embedding models try to replace what they already know for free. Every choice the series defends, from expert keyword dictionaries to deterministic dispatchers, comes back to this premise.

Where it shows up: Articles 3 (RAG is not ML), 4 (technique-fit grid), 6 (question parsing), 13 (the workflow pipeline), 15 (preparing the corpus).

7. Deterministic dispatcher beats autonomous agent

The 2024-2025 push for “agentic RAG” sells the agent as flexibility. Let the LLM decide which tool to call, which sub-question to issue, when to stop. In practice the agent saves engineering effort on the demo and costs ten times more during the first incident nobody can reproduce.

A deterministic dispatcher (a decide.py file, developed in Article 13, reads the parsed question’s structured fields and routes to one of N named sub-pipelines) does what the agent does, except a human can read the code, an auditor can replay the decision, and the team’s accumulated wisdom lives in version control. Autonomy is right for open tool sets and exploratory work. It is wrong for regulated enterprise contexts where every routing decision has to be inspectable.

Where it shows up: Articles 6 (question parsing), 13 (the workflow pipeline).

8. Per-failure-mode evaluation, not aggregate

Aggregate accuracy lies. A system at 95% overall can hide 50% on the hard subset (cross-references, listing questions, conditional clauses, scanned-PDF pages). The team that trusts the aggregate number discovers the 50% in production, one customer complaint at a time.

The series uses curated reference datasets sliced by question type and failure mode. Per-failure metrics tell the truth. They also make decisions falsifiable: “did this change improve listing accuracy?” has an answer, where “did this change improve overall quality?” has only opinions. RAGAS, ARES, Trulens each propose their own metric sets; the principle the series defends is the per-slice discipline, not a specific framework.

Where it shows up: Articles 3 (RAG is not ML), 20 (evaluation).

9. Each brick produces relational structured data, never raw strings

Parsing returns a relational set of DataFrames (line_df, page_df, toc_df, image_df, object_registry), not a Document object with a text blob and metadata. Article 5B (the relational data model) develops the parser and the tables it produces. Question parsing returns a row in question_df plus satellite tables (expert_keywords_df, scope_filters_df), all built in Article 6. Retrieval produces a typed candidate set with per-method provenance (Article 7). Generation writes a typed row with line-level citations, exclusions, and the answer schema (Article 8).

The junctions between bricks are tables, not strings. That has practical consequences: each brick can be tested independently with the saved output of the previous one, retrieval can be re-run against the same parse without re-parsing, the audit trail is a join over typed rows rather than a log of free-text snippets. The “string in, string out” pattern that the framework ecosystem normalized is the source of half the debugging pain in production RAG.

Where it shows up: Articles 5 (document parsing), 6 (question parsing), 7 (retrieval), 8 (generation), 13 (the workflow pipeline), 16 (ontology), 22 (security).

10. Citations are evidence, not decoration

Every generated answer comes back with (start_page, start_line, end_page, end_line) plus a verbatim quote pulled from those lines. The annotated PDF highlights the cited region on the source page. The citation is not a UI nicety; it is the explanation.

A common pushback: but the LLM is still opaque, so isn’t the citation just window dressing on a black box? The answer is that citations do not make the LLM less opaque. They make the LLM’s opacity irrelevant for the question that matters in enterprise work.

The user is not asking “why did the model choose these words”. They are asking “where in the source did this answer come from”. Line-level citations answer the second question completely and verifiably: the passage is right there, the line numbers are right there, the highlight is on the page, a reader can check the source in one click. The first question stays open, and in enterprise contexts it does not need to be answered to go live.

That is why SHAP, LIME (two techniques from ML interpretability that explain a model’s output by perturbing its inputs), attention visualizations, and the rest of the ML-interpretability stack solve a problem the citation-grounded RAG architecture does not have.

The follow-on requirement is that the system stores enough state to reproduce any answer six months later (the exact retrieval, the exact prompt, the exact model version, the exact source PDF). That makes the citation hold up in an audit. Without the storage discipline, the citation is just for show; with it, the citation is the only explanation an enterprise system needs.

Where it shows up: Articles 1 (minimal RAG), 3 (RAG is not ML), 8 (generation), 22 (security).

The series, article by article

The ten positions are argued across the series in five Parts. Here is the map, so you can jump to the argument you want to check. Titles without a link are past their revenue window or still to come; the linked ones are where a click today lands on a live, current piece.

Part I. What works, what breaks.

• Article 1, the baseline. The whole pipeline in one pass: a PDF goes in, a typed answer comes out with its source lines highlighted on the page. Each brick is a few readable lines of code, so you see the entire chain once before any deep-dive. Every later article improves exactly one brick of this baseline, and you always know where you are.
• Article 2, embeddings. The measurement the tutorials skip: cosine similarity wins on synonyms, typos, and paraphrase, and predictably breaks on unknown terms, negation, and the gap between matching the question’s words and containing the answer. You leave knowing which of your questions embeddings will fail on before production teaches you. Article 2bis runs the same honest measurement on rerankers: what a cross-encoder adds over bi-encoder similarity, and when the candidate pool is already too small for it to pay.
• Article 3, RAG is not machine learning. The frame the series stands on. RAG failures are engineering failures with an identifiable broken brick, not model weaknesses to train away, so chunk-size sweeps and finetuning optimize the wrong thing. The productive move is routing by question type and fixing the brick that broke.
• Article 4, the technique grid. Two axes, document complexity and question control, that tell you which technique fits which problem, from a regex to a vision model. The point is refusing one-technique-everywhere: a fixed grid of cases, each with the cheapest tool that solves it. Article 4bis collects the ten production mistakes we keep seeing, organized brick by brick, each with its fix.

Part II. The four bricks.

• Article 5, document parsing. Parsing as a relational data model, not text extraction: a line_df with coordinates for every line, a toc_df for the document’s structure, an image_df for what the text does not carry. This is what makes line-level citations possible at all, and every downstream brick reads these tables.
• The parsing companions. One method each, with its exact failure modes: Azure Layout when PyMuPDF cannot see the table, Docling for rich tables with no cloud upload, vision LLMs reading charts and diagrams, EasyOCR on scans (words recovered, structure lost), the image cascade that filters cheap before paying to read, TOC reconstruction when the PDF ships no outline. The two most recent are still live: rebuilding a PDF’s outline from its body typography, six typographic signals and one bounded loop, and the parsing-methods catalogue you need before going agentic, the decision layer that picks a method per page.
• Article 6, question parsing. The missing step in most pipelines: parse the question before searching anything. Five typed fields (keywords, scope, shape, decomposition, clarification) turn the user’s string into a brief that steers both retrieval and generation. The small loop that runs before retrieval adds the loop layer: read the document, work out what the question left unsaid, parse it again, bounded on purpose.
• Article 7, retrieval. Retrieval as filtering, not search: the document’s declared structure first, expert keywords next, embeddings last as the safety net for what the cheaper methods missed. This is position 1 argued in full, with the interpretability case for each method.
• Article 8, generation. Generation as controlled execution against a typed contract, not free-form writing. Three companions are live: seven patterns for the typed contract, because most hallucinations are extraction errors the schema can close; when top-1 is enough and when you iterate top-k, the loop shape of the generation call; and an LLM cascade from a cheap local model up to a hosted flagship, twenty models measured to find where the cheap one quietly stops being enough.

Part III. Pipelines on a single document.

• Article 9, end to end. The four upgraded bricks wired into one call and run on real documents. How four bricks of context engineering stop RAG hallucinations reads the same pipeline through the hallucination lens: your RAG is not inventing, it is faithfully answering the wrong context, and each brick controls one way that context goes wrong.
• Article 10, adaptive parsing. Start with the five-millisecond parse, and escalate only the page that needs a heavier parser. Free deterministic checks flag most failed parses before generation; the LLM’s own self-evaluation catches the rest, and the pipeline re-parses one page, not the document.
• Article 11, cross-references. When the answer says “see Section 7.2” instead of being the answer: enterprise documents point at themselves constantly, and a pipeline that stops at the first passage returns the pointer, not the content. A bounded loop follows the reference and brings the real answer back.
• Article 12, listing questions. “List every exclusion” breaks top-k structurally: the answer is every relevant passage, and any ranking cutoff silently drops the rest. The listing strategy retrieves for completeness, not for rank.
• Article 13, the dispatcher. The keystone of the single-document arc: a readable decide.py that reads the parsed question and routes to the right sub-pipeline, deciding when to loop and when to stop. Position 7 in running code: everything an agent promises, in a file a human can read and an auditor can replay. Its companion generalizes the pattern: the small loops inside each step, the big loops across the pipeline.
• The operational close. Cut latency and cost by calling the LLM less, not by buying a faster model: the dispatcher’s cheap paths mean most questions never pay for the expensive call in the first place.

Part IV. From one document to a whole archive (the next arc to publish). This is where positions 4 and 9 stop being slogans and become tables.

• Article 14, the corpus problem. Naive RAG dies quietly at corpus scale: the passage that answers your question exists in thirty documents, and only one of them is the right one. Your archive has one of three shapes (a handful of unrelated documents, thousands of near-identical ones, or case files that mix both), three questions to a colleague tell you which one you have, and each shape gets a different architecture.
• The folder shape. A handful of unrelated PDFs needs no index at all: concatenated, they read as one long document, and the single-document pipeline already handles it. Knowing when you do NOT need corpus machinery is the first corpus decision.
• The homogeneous shape. Thousands of near-identical PDFs (the same contract template, a thousand signed variants) need the opposite: an index. The index is a plain SQL table your pipeline fills once at ingestion, one row per document, and from then on most questions filter rows before any retrieval runs.
• The case-file shape. A claims dossier mixes both: the folder itself has structure (which document is the claim, which the invoice, which the photos). So the folder gets parsed into relational tables too, not just the PDFs inside it.
• Article 15, the document index. What a real index has to store to be trustworthy: identity, type, dates, parties, versions, provenance per field. A spreadsheet maintained by hand cannot carry it, and the article shows what breaks when teams try.
• Filling the index for free. Four places the value is already written in the PDF (metadata, first page, headers, the TOC), so most of the index costs zero LLM calls to fill. Extraction is the fallback, not the default.
• The extraction trap. Asking every PDF for every field is how an index fills up with confident wrong answers: a field that does not exist in a document gets invented. The cascade asks only the fields the document type actually carries.
• Raw and canonical. Keep both the name the document wrote and the name you cleaned it to. Collapse them into one column and the index lies quietly: you can no longer prove what the source said.
• Article 16, the vocabulary. The corpus needs a vocabulary before it needs a knowledge graph: the words your experts use, mapped to the words the documents use. That mapping, not a graph, is what decides whether a question ever reaches the right PDF.
• The curated tables. Five small hand-written tables, a few dozen rows, that route questions to document types and expand expert terms into document terms. Cheap to build, inspectable, and they outperform generic semantic search on the vocabulary they cover.
• GraphRAG, measured. The graph pipeline builds the knowledge graph for you, and the bill arrives before the first search: every page of the corpus read by an LLM at ingestion. The article measures what that buys against the curated tables, on the same corpus.
• The growing dictionary. The vocabulary is not static: a failed search proposes a new alias, a domain expert confirms it, the dictionary grows one validated row at a time. The expert stays in the loop; the system gets better where it actually failed.
• Article 17, querying the corpus. One cheap routing call classifies the question before anything runs: some answers come straight from the index (no PDF opened), some need one document, some need the full machinery. Most corpus questions are cheaper than you think once something decides.
• The SQL questions. “How many contracts expire this year” is not a retrieval question, it is one SELECT on the index. Sending it to RAG produces a slow, expensive, wrong answer; routing it to SQL produces the exact one.
• Scaling unchanged. The single-document pipeline from Part III runs on the whole corpus without modification: the index picks the documents, the pipeline answers on each. Module independence pays off here.
• The corpus table. The signature output of the arc: one row per PDF, one column per question, one citation per cell. SQL picks the rows, RAG writes the cells, and a domain expert reads a spreadsheet where every number is clickable back to its source line.
• Articles 18 and 19, architecture and storage. The code layout that keeps four bricks independent at corpus scale, and the storage map: where every artefact lives (parses, indexes, answers, traces), so any answer can be replayed and audited months later.

Part V. Operating in production (the closing arc).

• Article 20, evaluation. Aggregate accuracy hides the subsets that fail; evaluation gets sliced by question type and failure mode, on curated reference datasets. Position 8 turned into working practice: “did this change improve listing accuracy” becomes a question with an answer.
• Article 21, cost and latency. No observability stack: the pipeline already stores every call, model, token count, and duration in its tables, so cost and latency analysis is a SQL query you already know how to write.
• Article 22, security. Data residency, GDPR and the right to be forgotten across stored parses and answers, audit trails, and the prompts that leak: what enterprise security actually requires from a RAG system, brick by brick.

Beyond the numbered spine. More deep-dives are queued alongside the Parts:

• The embeddings arc, four parts. When embeddings return the wrong chunk and why a bigger model will not fix it; how to actually use them in production (line-level search, HyDE, the expert dictionary as first-class methods); the real roles embeddings play in a pipeline once they stop being the foundation; and how a reranker really works under the hood, the honest mechanics most write-ups skip.
• Row-level retrieval. A table is not a passage: retrieving the whole table buries the one row that answers, so the chunking unit becomes the row, with its headers carried along.
• Chunking strategies. Seven ways to cut a PDF (fixed windows, sentences, layout blocks, sections, rows, pages, semantic), and what each one breaks, measured on the same documents.
• Picking a model. Thirteen models on the same four-brick pipeline and the same fifty questions: closed APIs, open weights, on-prem Ollama. Not a leaderboard, a reproducible benchmark you can re-run on your own corpus.
• Local LLMs. How small a model can get before the bricks start failing, for corpora that cannot leave the machine: eleven Ollama models sorted by size, with the failure points marked.
• Harness engineering. The manifesto behind the whole build: the four bricks are the scaffolding, the model is one call inside it, and the scaffolding is what you own.

Start here: the freshest of the series

If you want to sample the series before committing to the map above, these are the pieces to read right now, freshest first:

• Loop engineering for RAG: the small loops inside each step, the big loops across the pipeline, the general framework behind every loop the series builds.
• Cut an enterprise RAG pipeline’s latency and cost by calling the LLM less, not by buying a faster model, the operational payoff of the dispatcher.
• Before full agentic RAG: know how you decide, and the parsing methods you pick from, the parsing catalogue and the decision layer, before handing the loop to an agent.
• RAG workflow and loop engineering: the dispatcher that decides when to loop and when to stop, position 7 in running code.
• Prompt, Context, Loop: the three engineering layers every RAG system is built on, the conceptual frame the whole series hangs on.

Conclusion

Walk through the ten positions when reading a blog post, a framework tutorial, or a vendor pitch about RAG, and check which ones the piece silently assumes away. Vendor pitches usually skip positions 1, 4, and 9; framework quickstarts skip 7 and 1; academic papers skip 6 and 5. The grid is not a verdict on the piece, it is the list of questions worth asking before adopting what is recommended.

None of the ten is original to the series in isolation. The contribution is treating them as connected. If you remember nothing else from these articles, remember that the questions on the document you have, asked by the people who need answers, are what should drive every architectural choice. Everything else in the series follows from that.

Further reading and sources

This piece is a position summary, not an empirical article. The practitioner writing that lands on most of the ten positions independently is Husain (Field Notes from the AI Engineering Trenches), Yan (Patterns for Building LLM-based Systems & Products), and Liu (Instructor). Anthropic’s Building Effective Agents (Dec 2024) is the industry framing that backs position 7. The agentic side on top of the four bricks defined here is follow-up work.

Same direction as this manifesto:

• Husain, Field Notes from the AI Engineering Trenches. Practitioner writing on RAG evaluation that lands on positions 6, 5, and 8; the most useful single source on per-failure-mode evaluation in production.
• Yan, Patterns for Building LLM-based Systems & Products. Catalog of retrieval and generation patterns; useful pairing with positions 1, 3, and 9.
• Liu, Instructor: Structured outputs for LLMs. Python library and the writing around it on schema-as-contract; direct support for position 9.
• Anthropic, Building Effective Agents (Dec 2024). Workflow-vs-agent distinction; the industry framing that backs positions 7 and 4.
• Khattab et al., DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, ICLR 2024 (arXiv:2310.03714). Compiled, declarative pipelines; aligned with positions 4 and 8.