claudio.cool
CCDV-PLessons

CCDV-P · Production Integration, Reliability, and Knowledge Grounding

Retrieval and grounding, diagnosing by stage and not making things up

⏱ 3 min readDraft

An internal assistant answers questions about company policies. Users complain about two things: sometimes it cites a repealed policy, and sometimes it states with confidence something that isn’t in any document. The team proposes swapping to a bigger model. First, find out which stage fails.

Agentic search or RAG

In RAG (retrieval-augmented generation) the application retrieves chunks before calling the model and injects them into context: one query, fixed order, predictable latency and cost. In agentic search the model decides what to look for, with which tool and how many times, and can rephrase when it finds nothing. The choice is a trade-off:

Classic RAG Agentic search
Latency and cost low and predictable higher and variable
Multi-hop or ambiguous questions weak strong: reformulates and chains
Control and audit easy to bound harder to predict
When simple queries over a known corpus exploratory questions, heterogeneous sources

Many systems combine both: direct retrieval for the simple, an agent for the complex.

The stages of a retrieval pipeline

A bad answer can originate at any stage, and the fix differs for each:

  1. Ingestion and chunking: split documents into chunks. Chunks too small lose context; too large dilute the signal. Keep metadata (source, date, version, permissions).
  2. Embeddings and index: they represent meaning; the embedding model must suit your domain and language.
  3. Retrieval: vector search for semantic similarity; lexical search (exact words, codes, names) for literal matches. Hybrid search combines both.
  4. Reranking: a second pass reorders candidates with a finer model and keeps the best.
  5. Generation: the model answers from what was retrieved.

To diagnose by stage, ask: was the right chunk in the index? (ingestion) Was it among the candidates? (retrieval) Did it rank high? (reranking) Did the model use it well? (generation). Measuring retrieval precision and recall separately from final quality keeps you from blaming the model for an index problem. In the assistant’s case: a repealed policy that is still indexed is an ingestion and validity-metadata problem, not a model problem.

Grounding: answers you can verify

A grounded response rests on the retrieved sources. The practices:

  • Source citation: the answer points to which chunk each claim comes from (the API offers a citations feature; if you build your own, return source identifiers you can later verify).
  • Declaring uncertainty: instruct and test that, if what was retrieved doesn’t answer the question, the model says so rather than filling in with general knowledge.
  • Verifiable confidence signals: match between the claim and the cited chunk, retrieval score, agreement across sources. The confidence the model says it has isn’t a calibrated signal by itself.
  • Currency: filter by date or version and retire repealed content from the index.

The usual traps

  • Swapping models before knowing which stage fails.
  • A single search type for queries that need exactness (codes, names) and meaning at once.
  • Chunks with no date or source metadata.
  • Writing “don’t make things up” in the prompt without measuring whether it holds.
  • Evaluating only the final answer and not retrieval.

On the exam

For a grounded-answer failure, look for the option that isolates the stage with a per-stage metric. For queries with exact terms and semantics, hybrid search with reranking. When information is missing, the right system behavior is to say so, with a citation where there is one.

Quick question

A practice exam question on this domain. It counts towards your progress.

A platform team operates a remote MCP server, reached over HTTP with OAuth, that twelve internal agents use. To clean up the API, the team renamed the parameter customer_id to account_id in the lookup_customer tool and made it required. They updated the tool description and deployed on a Friday afternoon. By Monday three agents were returning wrong or empty answers: some still sent customer_id and received validation errors that the agents' prompts told them to treat as "not found".

The server is owned by one team, but the agents belong to different teams with different release schedules.

What should the platform team do to make such changes safe in the future?

Practise this domain in a practice exam →