What is prompt caching — and why you can pay for the same context just once

Here's the annoying part. You built an AI bot: every request carries a big system prompt, a couple of documents and the chat history — thousands of tokens. The user types a short question. And you pay for that whole giant context all over again on every message, even though it never changes. Prompt caching lets you pay for the unchanging part once, not every time. Let's see how.
What prompt caching is, in plain words
Prompt caching is when the model provider remembers the already-processed start of your request and, on the next identical request, doesn't recompute it — it pulls it from memory. And it charges you noticeably less for what comes from the cache.
The key word is "start". It's not the whole request that's cached, it's its prefix: the shared chunk your requests all begin with. Usually that's the system prompt, the instructions and the documents — the parts that don't change from message to message.
Why you pay for the same text over and over
You need to know how the model bills. Every request is the entire input at once: system prompt + documents + history + the new question. The model processes it fresh from the first token, and you pay for all input tokens every single time.
If the unchanging chunk is 5,000 tokens and the user's question is 20, those 5,000 get sent and billed again and again. Invisible on one message. Across a thousand, it's most of the bill.
How it saves money
Caching breaks that logic: the unchanging prefix is processed once, then read from memory at a discount. The exact magnitudes differ by provider, but the ballpark:
- OpenAI turns caching on automatically for long-enough requests (roughly 1,000+ tokens). Input read back from the cache costs about half the normal price — with zero code on your side.
- Anthropic (Claude) asks you to mark the block explicitly, but the saving is bigger: a cache read is roughly a tenth of normal input price. The catch: the first write to the cache costs a bit more than usual, so the cache pays off from the second call on.
One important limit: the cache is short-lived — minutes. It's meant for a burst of requests (a live chat, batch processing), not "for tomorrow". If too much time passes between calls, the prefix goes stale and gets recomputed.
The main trick: what goes first, what goes last
Since it's the shared prefix that's cached, the order of blocks in your prompt becomes money. Simple rule:
- Stable stuff first. System prompt, instructions, big documents, knowledge base — anything identical on every request.
- Changing stuff last. The user's question, fresh data — whatever's new each time.
Then every request shares the same start, the cache kicks in, and you mostly pay for the short tail. Flip the order (question at the top) and there's no shared prefix, no cache, no saving.
Is prompt caching the same as regular caching?
Same idea, different mechanics. Regular caching stores a finished answer: same question, instant saved result. Prompt caching stores the processed start of the request, but the answer is still generated fresh for the new tail. So it speeds up and cheapens the input rather than replacing the output. That's why it helps even when every question is different but the shared context is the same.
Why isn't my cache hitting?
Three common causes. First, a changing chunk sits at the top, so there's no shared prefix — move the stable parts up. Second, the prefix is too short: caches have a minimum threshold (around a thousand tokens), and small stuff isn't cached. Third, too much time passed and the cache went stale. If the bill isn't dropping, start with block order — that's usually it. And remember: caching saves on repeated context, so it's one of the knobs for cutting AI costs, not the only one.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





