Retrieval matched to the shape of the data, and accuracy versus latency

A sales team exported two million order rows, generated one embedding per row and connected a semantic search. It asks “pending orders over 10,000 from the northern region last quarter” and gets a handful of similar rows, with matches missing and some from another quarter: the totals come out wrong. Questions about the notes field (“customers complaining about delays”) work perfectly. It isn’t an embeddings problem: it’s using a single kind of retrieval for data of different shapes.

The shape of the data rules

Data and queries	Retrieval that fits
Tables, with exact filters, ranges, counts or sums	Structured query: a read-only tool over views or the database
Free text, conceptual questions (“who can terminate for convenience”)	Semantic search
Codes, clause numbers, defined terms, IDs	Keyword search, for example BM25
Both kinds of query over the same corpus	Hybrid search: both lists merged and then reranking
A small, stable document, with answers that span sections	No retrieval: the whole document in context

A few clarifications the exam exploits:

• Similarity is never an exact filter. Neither a better embedding nor more results turns “similar” into “amount > 10,000 and status = pending”. For that, a read-only query (which also respects the database’s permissions), and semantic search stays for the notes.
• “Clause 14.2” or “SUP-20417” are, to an embedding, topics similar to other clauses. Adding BM25 alongside semantic search fixes those queries without losing the conceptual ones; replacing everything with keywords breaks the ones that worked.
• Not everything needs RAG. A 60-page policy that changes twice a year fits whole and loses no sections; an archive of half a million pages that grows every month fits in no window and calls for indexing and retrieving with citations. Unifying both features “for maintenance” is the trap.

Accuracy versus latency: each path with its requirement

More retrieval stages (more candidates, reranking, the most capable model) usually add accuracy and also seconds. The decision isn’t “the most accurate everywhere”: it’s whatever meets each path’s requirement.

An assistant that suggests answers to agents during the call takes six seconds, almost all of it in retrieval and reranking. Reranking adds four points of accuracy in the offline evaluation, but agents ignore suggestions that arrive after two seconds. A suggestion that arrives late is worth zero, however accurate. The post-call summary, on the other hand, nobody reads until the next day.

The justified configuration: the live path with lighter retrieval and a faster model that meet the two seconds; the summary with the full pipeline. Streaming isn’t enough when the delay is before the first token is generated, and dropping retrieval entirely loses the policy-based answers.

The usual traps

• Semantic search over tabular data that needs filters and sums.
• Loading the whole table into context so the model filters and sums.
• Replacing semantic search with keywords across the whole system.
• Asking the model to answer only if it sees the clause number, without fixing retrieval.
• Chunking a small document that works fine whole today, to have “a single architecture”.
• “The most accurate configuration everywhere” even though one path has a latency target.

In the exam

First look at which queries work and which don’t: that difference tells you the shape of the data. Exact filters over tables call for a structured query; codes and IDs call for keyword or hybrid search; conceptual text stays with semantic search. If an option says the database already offers BM25 or read-only views, that’s a hint. In latency scenarios, look for the option that separates the paths and justifies each configuration with its requirement.