Caching is one of the fastest ways to improve LLM latency and unit economics. It is also one of the fastest ways to create silent security and correctness failures. A cached answer can leak data across users, ignore updated policies, or persist a hallucination long after you have fixed the underlying issue.
The key is to treat caching as an engineering and governance problem, not a simple performance toggle.
What you can cache (and what you should not)
In most AI applications you have multiple cache layers, each with different risk:
- Embedding cache. Safe and high leverage. Cache embedding vectors per document/version and per query (with normalisation).
- Retrieval cache. Useful for popular queries, but must be invalidated when content, permissions or ranking changes.
- Tool-result cache. Great for read-only tools; risky for tools that reflect fast-changing operational states.
- Final-response cache. Highest risk. Only safe in narrow scenarios (public content, deterministic formatting, no user-specific context).
As a rule: cache the inputs and intermediate artefacts before you cache the final answer. Intermediate caching preserves flexibility when prompts, models or policies change.
Cache keys must include risk-relevant versions
Most cache incidents happen because cache keys are too simple. If any of the following can change the output, it needs to be in the key or the invalidation rules:
- Tenant and user identity. Never share caches across tenants; usually do not share across users unless content is truly public.
- Prompt and policy versions. If you use a prompt registry, include the prompt/policy IDs (see prompt registries).
- Model version and routing policy. Provider model updates can change behaviour; include model+route identifiers (see routing and failover).
- Retrieval corpus version. Include knowledge base snapshot IDs or ingestion timestamps (see knowledge base governance).
If you cannot confidently identify the “version surface” of a response, you should not cache it.
Invalidation: TTL is not enough
TTL-only caching is tempting, but it creates a window where you knowingly serve stale or unsafe outputs. Prefer event-driven invalidation for high-risk artefacts:
- When a prompt or policy changes, invalidate caches that depend on that prompt.
- When a knowledge source changes, invalidate retrieval caches tied to that source.
- When you detect an incident (hallucination burst, data exposure), invalidate aggressively and roll back (see AI incident response).
TTL still plays a role—especially for low-risk read-only tool results—but TTL should be a backstop, not your primary safety mechanism.
Privacy and compliance controls
Cache stores are data stores. Apply the same controls you would apply to any system holding sensitive information:
- Data minimisation. Cache the smallest useful artefact and avoid caching raw prompts that contain sensitive data (see data minimisation).
- Encryption and access control. Encrypt at rest, restrict access, and log reads/writes.
- Segregation. Separate caches by tenant, environment and risk tier.
- Retention. Set explicit retention limits aligned with policy and regulatory constraints.
Correctness: do not cache “bad” answers
If you cache final responses, cache only after the output passes the same safety and quality checks you use in production. Store the outcome of those checks alongside the cache entry, including guardrail versions, so you can invalidate when guardrails change.
For agentic systems, be especially careful: you may want to cache the plan or retrieved evidence, but not the final tool actions. Tool calls should be treated as untrusted input and re-authorised every time.
Measure what the cache is really doing
Caches can hide problems. Monitor hit rates and also quality and risk signals:
- Cache hit rate by endpoint, intent and tenant.
- Staleness events and invalidation frequency.
- Hallucination or escalation rates split by cache hits vs misses.
- Cost savings vs quality regressions (see FinOps for LLMs).
Done well, caching is a reliability capability: faster, cheaper, and more predictable. Done poorly, it becomes an invisible risk amplifier. Build it with the same discipline you apply to identity, policy and production change control.