Every time a large language model processes a prompt, it has to do a substantial amount of computation before it can generate even the first word of a response. When the same lengthy content, such as a long document, a detailed set of instructions, or an extensive conversation history, gets sent to a model repeatedly, recalculating all of that from scratch every single time is wasteful. Prompt caching is the technique that solves this problem, and it has become an important tool for making AI applications both faster and cheaper to run.
What Is Prompt Caching?
Prompt caching is a technique in which a language model, or the system serving it, stores and reuses the internal computations from a previously processed portion of a prompt, rather than recalculating that same portion from scratch on a later request. When a new request arrives that shares an identical beginning with a previous one, such as the same system instructions or the same long reference document, the cached computation for that shared portion can be reused directly, with the model only needing to process whatever new content comes after it.
This is possible because of how transformer-based models process input using attention: processing each token involves computing certain intermediate representations that depend only on the tokens that came before it. If that earlier portion of the prompt has not changed, those intermediate results do not need to be computed again.
How Prompt Caching Works
In a typical prompt caching setup, the process looks something like this:
- A prompt is processed for the first time. The model works through the full prompt, including any long shared content like system instructions or reference documents, and generates a response. Along the way, it produces internal computations tied to each portion of that input.
- The shared portion is cached. The system storing this cache keeps the intermediate computations associated with that unchanged, shared portion of the prompt, rather than discarding them after the response is generated.
- A later request reuses the cache. When a new request comes in that begins with that exact same shared content, the system retrieves the cached computations instead of redoing that work, and the model only needs to process the new portion of the prompt that follows.
Because the cached portion typically must match exactly, even small changes earlier in the prompt can invalidate the cache for everything that follows, which is why prompt caching tends to work best when a large, stable block of content, like a lengthy system prompt or reference document, is reused repeatedly with only the final portion of a conversation changing between requests.
A Simple Analogy
Imagine a tax advisor who meets with the same client throughout the year. At the start of each meeting, rather than re-reading that client’s entire financial history and documentation from the beginning, the advisor keeps a set of notes summarizing everything already reviewed and understood about that client’s situation. Each new meeting starts from those existing notes and only focuses on whatever new information or question has come up since the last visit. Prompt caching works similarly: once a model has effectively “read and understood” a stable chunk of content, it does not need to redo that work from zero every time, as long as that chunk has not changed.
Why Prompt Caching Matters
Prompt caching offers meaningful practical benefits for anyone building applications on top of large language models:
- Lower latency: Reusing cached computation means the model can start generating a response faster, since it skips reprocessing content it has already seen.
- Reduced cost: Because processing cached content requires less computation, many AI providers charge less for tokens that are served from a cache compared to tokens processed fresh.
- Practical for long, stable context: Applications that rely on long, unchanging reference material, such as a detailed knowledge base, codebase, or lengthy set of instructions, benefit especially strongly, since that large stable portion only needs full processing once.
This matters increasingly as applications push toward longer context windows and more elaborate system prompts, since without caching, the cost and delay of reprocessing all of that content on every single request would make many real-world use cases impractical, especially those involving rapid back-and-forth conversations or frequent repeated queries against the same large reference material.
Why It Matters
Prompt caching is one of those behind-the-scenes engineering techniques that most everyday users of AI tools never directly see, but that quietly shapes how responsive and affordable those tools can actually be. As AI applications increasingly rely on long documents, extended conversation histories, and elaborate instructions to guide a model’s behavior, the ability to avoid repeatedly redoing the same expensive computation becomes an important factor in whether those applications remain fast and cost-effective enough to use at scale.