RAG pipeline, chunking by structure, contextual retrieval and version metadata

An assistant answers questions about a 900-page underwriting manual. The chunks it retrieves say things like “this limit does not apply in the cases listed above”, without saying which limit or which cases, and coverage tables arrive cut in the middle of a row. The team halved the chunk size and then raised the retrieved chunks per question to 50: nothing improved and the second change made every answer more expensive. When a retrieval-augmented generation (RAG) system answers poorly, the first suspect is what context reaches the model, and that is decided at indexing time.

Cut along the structure

Fixed-size chunking is simple, but it cuts wherever it lands: it separates a sentence from the section that gives it meaning and splits tables. The alternative:

• Cut at the boundaries of the structure: sections, subsections, clauses, articles.
• Keep tables whole, along with lists that are read together.
• Adjust the size within that structure, not the other way around.

Smaller chunks make the problem worse (each one loses even more context) and retrieving more chunks only adds cost and latency: they’re still incomplete chunks.

Make each chunk understandable on its own

Contextual retrieval prepends to each chunk, before indexing it, a short text that situates it: which document it comes from, which section, what it refers to. “Underwriting manual, section 4.2, hail limit exclusions: this limit does not apply…” is retrieved better and the model interprets it correctly. That context can be generated with a small model during ingestion, and it’s used both for the embeddings and for the keyword index.

Changing the embedding model without touching the chunking doesn’t help: no embedding recovers context the chunk doesn’t have.

What wasn’t indexed doesn’t exist

A case the exam uses: to speed up the first load, ingestion indexed only the first 500 tokens of each contract, and nobody removed that limit. Indemnity and termination clauses, which usually sit near the end, never show up. Neither a better embedding nor more results will find them. The answer is to re-index the complete documents in chunks.

The same applies after a document refresh: if answers start coming out confident but wrong and the model didn’t change, check the re-indexing and the retrieved chunks first.

Metadata: version, validity and what doesn’t go into the prompt

Each chunk’s metadata solves two different problems:

Versions. If each revision of a policy is added to the index as a new document, the nearly identical versions score the same and the assistant cites superseded rules. Asking the model to “prefer the most recent” isn’t a control: often it doesn’t even receive the current version. What works is storing version, effective date and status on each chunk and filtering at query time: current by default, and a past date when audit asks for it. Rebuilding the index every week from the same files changes nothing.

Archive data. Administrative metadata (who scanned it, folder path, history) isn’t pasted into every chunk sent to the model. It’s stored as index fields: it stays available for audit and filtering, and stops taking up context on every request.

The usual traps

• Smaller chunks or more retrieved chunks for a chunking problem.
• A new embedding model to find text that was never indexed.
• Asking the model to pick the newest version.
• Huge chunks “so the date is visible”.
• Deleting metadata the policy requires keeping, to save tokens.
• Blaming the model when answers get worse right after a document refresh.

In the exam

Read the symptom carefully. Chunks without context or cut tables call for chunking by structure and contextual retrieval. Content that “never shows up” calls for checking what was indexed. Citations of old versions call for validity metadata with a query-time filter. And if the scenario mentions metadata taking up tokens but needing to be kept, the answer is moving it into index fields. Discard options that change the model or the number of results when the problem is in ingestion.