for taking a RAG system past one document is to build an index: one row per document, one column per field you may want to filter on, then a filter in front of the search.
That advice assumes the documents have fields in common. Plenty of folders do not.
Take a research folder: a 492-page catalogue of security controls, a zero-trust architecture spec, an AI risk framework, thirty-one machine-learning papers, and seven commodity market reports. Ask what column they share. There is no client, no amount, no effective date that would mean the same thing across all of them, and no other field a business user could name either.
When there is no shared field, there is no table to build. That reads like a missing piece. It is the opposite. It means the preparation step collapses into two artefacts, and one of them the parser already returns for free.
This article works through that case:
- the two questions that tell it apart from the case that does need an index
- the preparation, which is one summary line per file plus each file’s own table of contents, and nothing else
- why the summary line has to be written for a router rather than for a reader
- the two-level routing that answers a question, and why the nested outline never enters a prompt whole
- four ways it breaks, including the point where the flat file list stops scaling
This article is part of Part IV of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks. Part IV asks what changes when the input is a folder instead of a file, and the answer depends on which kind of folder you have. This one is the kind that needs the least new machinery.
🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.
📓 Build the two-level index over your own folder in the companion notebook: write one summary line per file, print the level-0 list, then run the routing call and watch it keep one file out of sixty-three before a single page is read. Repo → doc-intel/notebooks-vol1.
The folder used throughout is 63 public PDFs, 4,211 pages in total: 25 NIST files, which are 24 publications plus a three-page scan of one of them (US Government work, public domain in the US, see the NIST copyright statement), 31 arXiv papers (each under the arXiv non-exclusive distribution license declared on its own abstract page), and 7 issues of the World Bank Commodity Markets Outlook (CC BY 3.0 IGO, as declared on the OKR publication page for April 2026).
1. Which kind of folder do you have
Article 14A (three kinds of corpus, and what building for the wrong one costs) sets out three shapes a collection can take, and says that each one wants a different architecture. This article takes the one where the documents have nothing in common. Before building anything, it is worth being sure that is the case you are in, because the wrong guess costs a lot in either direction.
1.1. Two questions that decide it
The first question: do two documents in the folder ever refer to each other? An amendment refers to its master agreement. A renewal refers to last year’s policy. A certificate belongs to a contract. If that happens, the folder has a structure you must model, and this article is not the one you want.
The second question: can a business user name a field that every document has, and that means the same thing in each? Client, effective date, premium, policy number. If the answer comes back fast and confidently, the folder is a database nobody has typed yet, and typing it is the job.
If both answers are no, there is nothing to put in the columns. You could still extract metadata (file name, page count, creation date), but none of it is a field anyone will filter on. A column that is empty for most rows and means something different in the rest is not a column.
Mixed folders exist, and they are common. A shared drive can hold 5,000 look-alike contracts next to 300 miscellaneous files that belong to nobody. That is not a contradiction, it is two corpora sharing a disk. Split them and run the right architecture on each.
1.2. What the other case gets, and this one does not
The homogeneous case (Article 14C, where the business already knows the columns) gets three things this one cannot have.
It gets a filter. A question that says retailer X, currently in force becomes a SQL clause, and 18,000 rows become three before anything is read.
It gets aggregation. How many policies expire this quarter is answered by the index itself, with no document opened at all.
It gets a shared vocabulary. Every row has a client and a date, so the ontology that maps user words to column values pays off across the whole corpus.
None of that transfers here. There is no clause to write, nothing to count, and a vocabulary shared by a control catalogue and a commodity report would be a vocabulary about nothing. Articles 15 (preparing the corpus), 16 (the ontology behind the columns), and 17 (querying the corpus with SQL first) build all three, and they are the right articles when the second question above came back yes.
What is left when you take those away is smaller than it looks, and it is the whole architecture for this case: a folder whose documents have nothing in common is one long document. The files are its chapters. Their sections are its subsections. You already know how to read a long document. You read its table of contents.
2. The preparation: one line per file, plus the outline the parser returns
Two artefacts, built once at ingestion.
Level 0 is the list of files, one line each, saying what that file is about. Level 1 is each file’s own table of contents. Together they are a nested outline: the folder’s chapters, then each chapter’s sections.
That is the entire preparation. No classification pass, no field extraction, no schema, no ontology.
2.1. The summary line is the index, so write it for the router
Here is where most of the quality lives, and it is easy to underestimate because the artefact is one sentence.
The summary line is not documentation. Nobody reads it. Its only consumer is the routing call that has to decide, from 63 lines, which files could hold the answer. It should be written the way you would write a column definition, not the way you would write a blurb.
Three things make the difference between the two versions.
The subject in both vocabularies. The words the user will type, and the words the document itself uses. A question about account management has to land on a file whose line says account management and AC-2, because the router matches on text, not on intent.
The shape of what the file answers. One numbered control per section is a different promise from a narrative report. It tells the router whether the answer is a section or a whole chapter, which is what the second level then acts on.
What the file is not about. This is the part people skip, and it is the one that makes routing precise. Not an implementation guide removes the file from every how-to question in one clause. Negative information is what lets a router discard sixty-two files without opening any of them.
Producing the lines is cheap. One call per document at ingestion, over the first pages plus the table of contents, gives a usable draft; on a folder this size you can also write them by hand in an afternoon. Either way, the cost is paid once per document and re-paid only when the file changes. What matters is that the lines get reviewed like schema, not like copy.
2.2. The second level costs nothing new
The parsing brick already returns a toc_df, one row per heading with its page range. Article 5B (the relational tables a parser returns) builds it from the PDF’s native outline. Nothing in this article extends it. Level 1 is that table, kept per file.
For the folder here, 47 of the 63 PDFs ship a native outline, and those 47 cost one parse each. The other 16 ship none: a scanned memo, a slide export, a two-page note.
Two answers work for those 16, and length is what picks between them. A short file becomes a leaf: when the router keeps it, you read it whole, and level 1 never runs. A long one gets its outline reconstructed first. Article 5septies (rebuilding the table of contents from the page that prints one) covers the case where the PDF shows a contents page but never encodes it as an outline. Building Document Structure with Loop Engineering covers the harder one, where there is no contents page either and the headings have to be recovered from the body typography.
Across the folder the two levels come to 63 lines and 2,422 outline rows. The 63 is what the model reads on every question. The 2,422 is what it can reach, one file at a time, and almost never touches.
3. Query time: routing down two levels
Take one question through the folder: “what does the account management control require?” The answer is five pages of NIST SP 800-53 Rev. 5 (Security and Privacy Controls for Information Systems and Organizations, US Government work, public domain in the US), control AC-2, pages 46 to 50. Sixty-two other files cannot hold it, and none of them will be opened.
3.1. Level 0, picking the files
One call. The model reads 63 lines, each one a file name, its routing summary, and a keyword tally. It returns the file ids that could hold the answer, with a reason for each. Usually one to three files come back.
The keyword tally runs alongside, not instead. Counting how many of the question’s terms appear in a file’s summary line and in its outline titles costs nothing, and it catches what a summary sentence rounds off: an exact code like AC-2 or GV.OC-01 that the sentence never spelled out. The two signals go into the same prompt and the model decides, which is the pattern Article 7B (parallel detectors, then one call at the end) established inside a single document. Here it runs one level up, unchanged.
Two properties of this call are worth naming. It is bounded: 63 short lines, whatever the folder weighs on disk. And it is auditable: the reason string says why a file was kept, so a wrong answer can be traced to the routing decision rather than to a similarity score.
3.2. Level 1, descending inside the survivors
From here it is the single-document case, with nothing added. The file’s own outline becomes the current level, the model picks a branch, and if that branch has finer children the loop opens them and repeats. It stops at a leaf, or at a section short enough to read whole. Article 7quater (hierarchical retrieval, reading a long document by its table of contents) works that loop out in full on this same catalogue: eleven chapters, then twenty control families, then twenty-five controls, then AC-2.
The two levels are the same call.
# The folder's file list is the top level of ONE table of contents:
# level 0 rows are files, deeper rows are those files' own sections.
level = corpus_toc[corpus_toc.level == 0] # 63 rows, one per file
files = reason_on_toc(question, level, # the SAME router call
section_signals=keyword_hits(question, level))
sections = []
for file_id in files.section_ids: # typically 1 to 3
level = corpus_toc[(corpus_toc.file_id == file_id)
& (corpus_toc.level == 1)]
while True: # the one-document loop
pick = reason_on_toc(question, level)
section = level[level.section_id.isin(pick.section_ids)]
kids = immediate_children(corpus_toc, section)
if kids.empty or section.n_pages.max() <= SMALL:
break # a leaf, or short enough
level = kids # open it, descend
sections.append(section)
# 63 files -> 1 file -> AC-2 ACCOUNT MANAGEMENT -> 5 pages out of 4,211Every line that comes out keeps its file id, so a citation names the file it came from. On a folder whose documents share no vocabulary, that matters more than usual: an answer stitched from a control catalogue and a market report reads fluently and is worth nothing.
3.3. Why the whole nested outline never goes in one prompt
The nested outline is one object, 63 lines plus 2,422 rows. Nothing stops you from serialising all of it into a single call. Three reasons not to.
It stops fitting. Multiply this folder by twenty and the outline alone is a long prompt with no room left to reason in. The folders that need this architecture are the ones that keep growing.
Precision falls before the window does. At level 0 the model compares 63 whole-document descriptions, which is a decision it makes well. Handed 2,485 mixed rows, the section it needs competes with 357 siblings from a document that is not even relevant, and a section title read out of its document is a weak signal.
It costs on every question. Level 0 is 63 short lines, paid every time. Level 1 is only the outlines of the files that survived, so it is paid on one file rather than sixty-three.
The descent is also optional. On a folder of eight short files with shallow outlines, level 0 picks a file and there is nothing to descend into: the loop runs once and behaves like flat routing. The model decides at each level whether going deeper buys anything, which is the same termination rule the single-document loop uses.
4. Where it breaks
Four things go wrong with a two-level folder index, and they are not equally serious. The first is a writing problem you fix in an afternoon. The next two are parsing problems that show up as retrieval problems. The last one is a real ceiling.
4.1. The summary line that says nothing
“A technical report on cybersecurity.” Twelve files match that description, so level 0 returns all twelve, or picks three of them for no reason you can reconstruct. The tell is specific: the level-0 call keeps too many files, while level 1 looks perfectly healthy on each one it is given.
The fix is to rewrite the line, and the test is cheap. Hold out ten real questions, run the level-0 call on each, and check that the file you would have picked by hand comes back. Ten questions catch most of it, because a bad summary line is usually bad for a whole class of question rather than for one.
Treat the lines as an artefact under review. They are the index; a sloppy line is a sloppy column definition, and it will not announce itself.
4.2. The document with no structure
Level 1 needs something to descend into. A file with no headings gives the loop nothing to pick from, so the router either reads it whole or reads nothing.
Reading it whole is fine most of the time. A three-page memo is a leaf, and the two-level scheme handles leaves naturally. The failure case is narrow and specific: long and structureless. A 200-page scanned bundle with no headings cannot be routed at any level, because there is nothing to route over.
That one is not a retrieval problem. It is a parsing problem, and it gets solved before this article’s machinery runs, by reconstructing an outline from whatever signal the document does carry.
4.3. The flat file list stops scaling
Level 0 is a flat list. It works while the list stays readable in one call.
A few hundred files is comfortable. A few thousand is not, and the failure is not the one people expect. The context window is not what gives out first; routing accuracy is. At 3,000 lines the level-0 call is the same top-k problem this whole architecture was built to avoid, moved up one floor.
The fix is one more level, not a different architecture. Group the files, by folder, by source, by year, by whatever grouping the shelf already has, and give each group its own summary line. Level 0 becomes 40 group lines, level 1 the files inside the groups the router kept, level 2 their outlines. Same routing call, one more turn of the same loop.
That is also the point where this case starts asking for what the homogeneous case has from day one. If the natural grouping turns out to be a field that every file has, the second question from section 1.1 has quietly changed its answer, and the folder wants an index after all.
5. Conclusion
A folder whose documents have nothing in common is one long document, and it wants the retrieval you already have rather than the corpus machinery of the articles around it.
What it does not need is worth listing, because each item is a project someone will otherwise propose. No relational index: there are no columns to fill. No ontology: a vocabulary shared by a control catalogue and a commodity report describes nothing. No SQL agent: there is nothing to query. No entity extraction pass over the corpus: the entities do not recur across files, so there is nothing to link.
What it does need is a sentence per file, written for the router, and the table of contents the parser already returns. Then routing runs twice: once over the file list, once inside the files that survived. Sixty-three files and 4,211 pages, and the model reads 63 lines plus one outline before landing on five pages.
The two follow-ups in this group take the other two shapes. Article 14C (the homogeneous corpus, where the business already knows the columns) covers the folder where the second question comes back yes. Article 14D (case files, one bundle about a single entity) covers the folder that is heterogeneous inside and repetitive across, which needs a bit of both.
6. Further reading and sources
Earlier in the series, on the pieces this article reuses:
- Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG. What to do about the 16 files in this folder that ship no native outline.
- Most RAG Hallucinations Are Retrieval Failures: Fix Retrieval, Not the Prompt. Why a routing mistake at level 0 shows up later as a generation problem.
- Context engineering for RAG question parsing: from a raw question to typed fields that steer retrieval and generation. Where the keywords the level-0 tally counts come from.
- Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations. The four typed pieces that fill the call the router feeds.
- Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract. What generation does with the lines the descent returns, and why each one carries its file id.
Also referenced above, listed here without links: Article 5B (the relational tables a parser returns), Article 5septies (rebuilding a table of contents from the page that prints one), Article 7B (parallel detectors, then one call at the end), Article 7quater (hierarchical retrieval inside one long document), Article 14A (three kinds of corpus, and what building for the wrong one costs), Article 14C (the homogeneous corpus), Article 14D (case files), and Articles 15 to 17 (the index, the ontology, and the SQL-first query path for the other kind of folder).
External sources:
- Document Summary Index, LlamaIndex documentation (docs.llamaindex.ai). The same level-0 idea as a shipped open-source component: a summary per document, and retrieval that picks documents by summary before it picks passages. Worth reading for the prompt it uses to generate the summaries.
- Parth Sarthi et al., RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval, 2024 (arXiv 2401.18059). Builds the tree by clustering and summarising recursively when no structure exists. The contrast is the point: a folder of documents already ships a tree, so there is nothing to infer.
- Anthropic, Introducing Contextual Retrieval, 2024 (anthropic.com). Prepends a short document-level context to each chunk so a passage stops being orphaned from its source. The summary line here is the same move at file granularity.
- Yunfan Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey, 2023 (arXiv 2312.10997). The naive, advanced, modular vocabulary this article’s baseline sits against.
- Scott Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System, 2024 (arXiv 2401.05856). Field-measured failure modes; the missing-document and wrong-document points are exactly what level-0 routing is built to remove.