Two caches, two different savings.

What is prompt caching?

Prompt caching is a provider feature that stores the internal state a model computes for the beginning of your prompt, so that the next request starting with the same text can skip that computation. If every request carries the same two-thousand-token system prompt and the same reference document, the provider processes them once, keeps the result for a short period, and later requests that begin identically pay a reduced price for that part and get a faster first token. Nothing changes in the answers; only the cost and latency of the repeated prefix change.

How prompt caching works

A transformer processes tokens left to right and builds attention state (keys and values) for each one. That state for a prefix depends only on the prefix, not on what comes after. Providers exploit this: they hash the prefix, store its state, and on a later request with a matching prefix load the state instead of recomputing it. The match is exact and from the start of the prompt, so anything that changes early in the prompt, such as a timestamp or the user's name in the system prompt, breaks the cache for everything after it. Put stable content first and variable content last.

What is prompt caching read and write?

A cache write happens the first time a prefix is seen: the provider computes the state and stores it, and some providers charge slightly more than the normal input rate for that write. A cache read happens when a later request matches the stored prefix: that part of the prompt is billed at a fraction of the normal input price. The economics work when the prefix is reused several times within the cache lifetime. A prefix sent once is a write with no reads, which costs a little more than not caching at all. Check your provider's dashboard for cached-token counts to confirm reads are actually happening.

What is prompt caching cost?

Pricing differs by provider and changes, so read the current pricing page, but the shape is consistent: cached input tokens cost a fraction of normal input tokens, and some providers charge a small premium for the initial write. The saving therefore depends on three things: how long the stable prefix is, how often it is reused within the cache lifetime, and how much of each request it represents. A RAG system with a long system prompt and a fixed set of reference documents reused across every user query sees large savings; a chat with a short system prompt and unique content per request sees almost none.

What is prompt caching Anthropic?

In Anthropic's API, prompt caching is explicit: you mark cache breakpoints on blocks of the prompt (system prompt, tool definitions, document content, or conversation turns) and the API caches everything up to each breakpoint. Cached content has a minimum length, a default lifetime of a few minutes that refreshes on use, and an option for a longer lifetime at a different price. Responses report how many tokens were written to and read from the cache, which is the number to watch when you are tuning. The explicit model gives you control over exactly what is reused.

How does prompt caching work OpenAI?

OpenAI applies prompt caching automatically for prompts beyond a minimum length: the API looks for a matching prefix from recent requests and bills the matched portion at the cached rate, with no code changes required. The practical rules are the same as elsewhere: keep stable content at the start, keep variable content at the end, and send requests close together so the prefix is still in cache. The usage object in each response reports cached tokens so you can verify the hit rate. Because it is automatic, the main thing to get right is the ordering of your prompt.

What is prompt caching in Bedrock?

Amazon Bedrock offers prompt caching for supported models, including Anthropic models served through it. You place cache checkpoints in the request, Bedrock caches the prefix up to each checkpoint, and subsequent requests with the same prefix within the cache window are billed at the reduced cached-read rate. The model support list, minimum token counts and lifetimes are documented per model, so check them before relying on it. The design advice is identical: stable system prompt and reference material first, user-specific content last, and monitor cached-token metrics to confirm hits.

How to implement prompt caching

• Reorder your prompt: system instructions, tool definitions and fixed documents first; conversation and the new question last.
• Remove anything volatile from the prefix: timestamps, request IDs, the user's name. Pass those at the end.
• Mark breakpoints where your provider requires them, after the stable blocks.
• Send related requests within the cache lifetime; for a batch job, group requests that share a prefix.
• Read the usage fields in responses and log cache read tokens; a hit rate near zero means the prefix is changing.
• Measure cost per request before and after on real traffic, not on a single test call.

What is semantic cache?

A semantic cache stores previous questions and their answers, keyed by the question's embedding, and serves a stored answer when a new question is close enough in meaning. 'How do I reset my password' and 'password reset steps' hit the same entry. It runs in your application, typically on Redis or a vector store, and it saves the entire model call, not only a prefix. The risks are returning a stale answer after the underlying documents change, or a wrong one when two questions are similar in wording but different in intent, so set a strict similarity threshold, scope the cache by user permissions and document version, and expire entries when sources update.

How does prompt caching improve AI performance?

Two ways. Latency: the time to first token drops because the model does not reprocess the cached prefix, which is noticeable when the prefix is tens of thousands of tokens of documents or tool definitions. Cost: the same prefix billed at the cached rate on every request can cut the input bill substantially for prompt-heavy apps such as RAG with large fixed context, agents with many tools, or few-shot prompts with long examples. It does not make answers better; quality is unchanged. What it does is make a larger, better-grounded prompt affordable, which indirectly improves answers because you can include more of the right context.