claudio.cool
CCDV-PLessons

CCDV-P · Production Integration, Reliability, and Knowledge Grounding

MCP in production and model or prompt changes without surprises

⏱ 3 min readDraft

A team upgrades the production model to the new version. In staging everything passed. In production, an internal MCP server starts receiving arguments in a different shape, a prompt that relied on old behavior produces longer outputs, and nobody knows which prompt version runs in each service. Operating means managing change.

MCP in production

A local stdio MCP server is fine for development; in production the norm is remote MCP over HTTP (the current transport is Streamable HTTP). That raises operational questions:

  • Authentication: the MCP standard for remote servers builds on OAuth 2.1. The server validates the token and authorizes per tool; it doesn’t assume “only the agent calls it”. Keys don’t go in the tool definition or in the prompt.
  • Tool and schema versioning: renaming a tool, changing its parameters or its meaning breaks the agents that use it. Compatible changes (a new optional parameter) coexist; breaking ones ship as a new version and are retired with notice. Tool descriptions are contract too: a text change changes how the model chooses them.
  • Observability and limits: log every call with identity, duration and result; usage limits and timeouts so a slow tool doesn’t hang the agent.
  • Minimal surface: each agent sees only the servers and tools it needs.

Migrating models

A new model is not a transparent replacement. Migrations are tested against the breaking behavior changes Anthropic documents for each version (parameters no longer accepted, changes in response style and length, tool use or stop reasons). The procedure: run your evaluation suite with the new and current model side by side, review the differences by category, adjust prompts where needed and only then move traffic. Pin the exact model version in production, and plan for the retirements Anthropic announces in advance.

Per-platform configuration parity and gateways

If the same system runs against Anthropic’s API and against Bedrock or Vertex AI, configuration isn’t identical: model identifiers, authentication, regions and beta-feature availability differ. A gateway or routing layer centralizes those differences (credentials, routing by model or region, limits, logging) so applications don’t repeat them. That layer is also a point of failure and change: you version it, observe it and test it like any other service.

Versioned prompts and progressive rollout

The prompt is code: it lives in version control, has a version identifier logged on every call (so a regression ties to a specific change), goes through evaluation and ships in a controlled way. Progressive rollout raises the change in stages (a small share of traffic, an internal tenant, then more) while watching quality, cost and error metrics against the current version. Each stage has an advance criterion and a rollback criterion: going back to the previous version must be a minutes-long action, with the previous version still deployable.

The usual traps

  • Migrating a model by changing just the identifier and checking “it answers”.
  • Changing an MCP tool in place, with no version.
  • A remote MCP server with no authentication or with a shared static key.
  • Prompts edited directly in production with no recorded version.
  • Launching at 100% and looking afterwards.
  • Assuming Anthropic API configuration works the same on the customer’s cloud.

On the exam

For a model, prompt or tool change, the good sequence is explicit version, compared evaluation, staged rollout with metrics and rollback ready. If the problem shows up on one platform only, suspect configuration and feature parity.

Quick question

A practice exam question on this domain. It counts towards your progress.

A support assistant answers questions using a vector index of product manuals. Complaints focus on queries that contain exact product codes such as "XR-4410-B", where the assistant gives answers about a different model. The team measured each stage: for natural-language questions the right passage appears in the top 50 candidates 93% of the time, but for code queries only 41% of the time. Adding a stronger reranker over the top 50 barely moved the results for code queries.

Generation quality is fine when the correct passage is supplied in the prompt.

Which change best addresses the root cause?

Practise this domain in a practice exam →