claudio.cool
CCDV-PLessons

CCDV-P · Observability and Production Monitoring

What to instrument in a Claude system, and what not to store

⏱ 3 min readDraft

A customer reports that “the agent sometimes takes a minute and gives a weird answer”. The team’s logs have one line per request: time, user, “ok”. There’s no way to know how many turns the agent took, which tools it called, what it cost or whether the cache worked. Without instrumentation you can’t debug, and with poorly thought-out instrumentation you leak data you shouldn’t have kept.

Traces for the agent loop

An agent isn’t a call, it’s a tree of calls. A distributed trace represents a full run, with a span per unit of work:

  • the root span: the user request or the task;
  • a span per model call, with model, prompt version, input and output tokens, stop reason and latency;
  • a span per tool call, with name, duration, status and result size;
  • subagent spans, children of the span that launched them, so the trace follows delegation between agents.

The trace ID propagates across services (including a remote MCP server) to stitch it all together. OpenTelemetry is the usual open standard for this, with attribute conventions for generative AI; Claude Code can also export telemetry via OpenTelemetry.

Token, cost and cache metrics

Token and cost metrics come from the usage field of each response: input tokens, output tokens, and cache tokens (cache_creation_input_tokens and cache_read_input_tokens). From that you compute:

  • cost per request, per tenant, per feature and per prompt version;
  • cache diagnostics: the hit rate (reads over reads plus writes). A low rate with a prefix that “should” cache points to a prefix that changes (a date in the system prompt, reordered tools) or a TTL expiring between calls;
  • input/output ratio and per-turn distribution, to spot agents that run away.

Grounding and reasoning traceability

To audit an answer, store which chunks were retrieved, with what score, from which source and version, and which were cited. For reasoning, store the observable decisions (which tool it chose and with what arguments) and the output; the model’s thinking text may be a summary and isn’t a guaranteed-faithful explanation, so it isn’t used as an audit record.

Structured logs

A structured log is an event with fixed fields (JSON): trace ID, tenant, prompt version, model, stop reason, tool, result, classified error. You can aggregate, filter and count them; a free-text line you can’t.

Content capture and privacy

Storing full prompts and responses helps debugging but collides with privacy, and with zero-data-retention (ZDR) agreements under which the provider doesn’t store content. That policy outranks convenience. What works:

  • Metadata always, content by decision: tokens, latencies, identifiers, stop reasons and error classification are always stored; content is configurable.
  • Redact before storing personal data and secrets, with your own access control and retention in the trace store.
  • Sample content and capture in full only for flagged cases (errors, reports), with consent and time limits.
  • Under ZDR, instrument with identifiers and metrics, and reproduce problems with synthetic cases or content the owner agrees to share.

The usual traps

  • One log line per request, with no turns or tools.
  • Traces that don’t cross the MCP server or subagents.
  • Storing full content in a logging system with no redaction or retention.
  • Looking only at total cost and not per tenant or prompt version.
  • Using thinking text as an audit record.

On the exam

If you need to debug and there’s a privacy constraint, choose metadata and structured traces, not switching instrumentation off nor storing everything. If cost climbs, look for the per-feature, per-tenant and cache metric that explains it.

Quick question

A practice exam question on this domain. It counts towards your progress.

A support agent handles each user request in a loop of model calls and tool calls (order lookup, refund policy search, ticket update). When users report wrong answers, the on-call engineer has only one log line per request with the final answer and the total duration. Finding which step went wrong requires re-running the conversation by hand.

The team already uses a standard distributed-tracing backend for its other services and wants the agent to be debuggable in the same way. The assistant serves about two thousand customers a day, and slow or wrong answers are the main source of escalations.

How should the agent be instrumented?

Practise this domain in a practice exam →