What Is Prompt Caching?

What is it

Prompt caching is a feature offered by LLM providers that avoids recomputing the attention math for the part of your prompt that doesn't change between calls — typically the system prompt, tool definitions, or the first turns of a long conversation. Instead of reprocessing everything from scratch on every request, the provider reuses what it already computed for that prefix.

This matters because chat APIs are stateless: every call resends the full conversation history. Without caching, an agent in a 50-turn conversation recomputes the previous 49 turns on every new message — repeated work that costs both time and money.

Mental model

To generate each token, the model computes three vectors per input token at every attention layer: Query, Key, and Value. Caching stores the already-computed Key/Value pairs for a prefix, so they don't get recalculated if that same prefix shows up again in a later call.

The cache boundary is marked explicitly in the request (cache_control in Anthropic's API). Everything before that boundary, if it's identical token-for-token to a previous call within the TTL, counts as a cache hit.

How it's used

In Anthropic's API, you mark a cache_control block at the end of the content you want cached:

response = client.messages.create(
    model="claude-sonnet-5",
    system=[
        {
            "type": "text",
            "text": long_system_prompt,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=conversation_history
)

The response reports how many tokens were cache hits (cache_read_input_tokens) and how many were written to cache for the first time (cache_creation_input_tokens) — those two numbers are the actual evidence caching is working, not an assumption.

Typical cases that benefit:

• Multi-turn conversations where history grows but the system prompt doesn't change.
• Harnesses with a CLAUDE.md, large tool definitions, and skills that are identical on every call in a session.
• Repeated questions against the same long document.

When to use it / when not to

Kicks in automatically when:

• There's a large, stable prefix (system prompt, tools, project context) repeated call after call.
• The conversation is long and multi-turn.
• You run the same task several times in a row against the same base context.

Doesn't help when:

• Every call starts with different content from the first token — there's no shared prefix to cache.
• You switch models mid-task: cached K/V is specific to that model's weights, so a new model can't reuse it and recomputes everything.
• The cache TTL already expired (varies by provider and configuration) before the next call.