Context as a budget, prompt caching, modular prompts and Skills

Three teams, three symptoms. A support agent forgets halfway through the conversation what the customer asked for at the start. An insurance assistant turned on caching and its cost went up. Five teams at a bank have the same chargeback procedure copied and two are still on the old version. All three problems are solved by thinking of context as a budget, not a warehouse.

More context isn’t better context

The context window is finite and, on top of that, the fuller it gets, the worse the model recalls what matters. If a tool returns the complete order history (thousands of tokens) every time it’s called, the agent ends up contradicting itself.

• Have tools return only the useful fields: the latest orders with status, date and items, not the whole JSON.
• Clear old tool results from the history. In the API, context editing replaces the oldest results when the context passes a threshold; for very long conversations, compaction summarizes the history.

Blind truncation (keeping the first N tokens) loses what’s needed. A larger window lets in more noise and costs more.

Prompt caching: order is everything

Prompt caching reuses an identical prefix across requests: it lowers the cost of those tokens and the time to first token. The rules the exam uses:

1. The prefix is built in this order: tools, system, messages. Changing something at one level invalidates that level and everything after it.
2. The match is exact. A timestamp down to the second at the start of the system prompt changes the prefix on every request: zero reads, one write per request, and since writing to the cache costs somewhat more than not using it, cost goes up.
3. What varies (the date, the user ID, the question) goes after the last fixed block; the breakpoint goes on the last block that repeats identically.
4. Below a minimum length, the prompt isn’t cached and the API doesn’t warn you. Confirm it by checking cache reads and writes in each response’s usage.
5. The entry lasts a few minutes and is refreshed on every use; there’s a one-hour lifetime, more expensive, for spaced-out traffic. But if the prefix changes on every request, no lifetime helps.

Truncating the policy document to save money is the classic trap: it loses needed content when caching solved cost and latency without removing anything.

Modular prompts and Skills

When several teams copy the same blocks into their prompts, the copies drift out of sync. Two levels of solution:

• Modular prompts: the common blocks (tone, policies, format) in a single versioned place that each template includes.
• Skills: each procedure packaged as a Skill. It works through progressive disclosure: at the start only its name and description are loaded; the body and files are read when the task needs them. In the API, custom Skills are shared across the whole workspace.

If each request uses one of fifteen procedures, the Skill solves both problems: a single source and a context with only what’s used. Caching alone makes the giant prompt cheaper, but leaves the context full of irrelevant procedures (which the model can mix up) and the copies out of sync.

The usual traps

• Truncating needed content to save tokens.
• Putting what varies before what’s fixed and then blaming the cache.
• Responding to a cache that never hits with more breakpoints or a longer lifetime.
• Moving to a model with more context so “everything fits”.
• Fifteen agents, one per procedure, with a router in front.

In the exam

If the scenario says the same long block is sent on every request and cost and latency are a concern, look for “fixed content first and caching”. If caching is on and there are no reads, look for what changes at the start of the prefix. If the agent degrades in long conversations, the answer is trimming what goes in (smaller results, clearing old ones), not enlarging the window. And if procedures are copied across teams and each request uses one, look for Skills.