Semantic Caching for LLM Responses on a Key-Value Store

"What is your refund policy" and "how do I return an item" are different strings. To an exact-match cache they are two unrelated keys, so the second one misses, fires a fresh LLM call, and bills you for tokens you already paid for once. This is the structural problem with caching natural language: humans phrase the same intent a hundred ways, and a hash only catches the people who happen to type it the same way twice. Studies and vendor write-ups of exact-match prompt caching put the redundant-query catch rate around 18%, which means roughly four out of five repeat questions slip through and hit the model.

Semantic caching closes that gap by matching on meaning rather than characters. It embeds the incoming query, looks for a previously answered query whose embedding is close enough, and if one exists, serves the stored answer without calling the model at all. Vendors and practitioners report LLM cost reductions in the 40-86% range and large latency drops once a semantic layer is in place (figures reported by cache vendors and should be read as workload-dependent, not guaranteed). The winning architecture in 2026 is not "replace the hash cache with a vector search." It is a dual-layer cache: a fast exact-match hash layer that costs nothing to check, backed by a semantic layer that only runs when the hash misses. A durable key-value store is the backbone for both - the hash index for the exact layer, and the response and metadata store for everything the semantic layer retrieves.

Why exact-match caching leaves money on the table

An exact-match cache is the simplest possible design and it is genuinely good at what it does. You take the prompt (often with the model name, temperature, and any system prompt folded in), hash it, and use that as a key: hash:{sha256} maps to the stored completion. A repeat of the identical request is a single point lookup, returns in single-digit milliseconds, and never touches the model. We cover this baseline pattern in caching LLM responses to cut OpenAI bills, and for a lot of traffic it is the right and only tool you need.

The ceiling is the catch rate. Exact match is a character-level test, and natural-language traffic is not character-stable. Consider a support assistant. "Reset my password," "I forgot my password," and "how do I change my password" are three keys to a hash cache and one intent to a human. Add casing, punctuation, trailing whitespace, and the inevitable "please" and the key space explodes while the answer space stays tiny. The reported ~18% redundant-query catch rate is the symptom: most of your duplicate spend is hiding in queries that mean the same thing but do not look the same.

You cannot hash your way out of this. Normalizing case and whitespace helps a little. Stripping stopwords helps a little more and breaks on edge cases. The only durable fix is to compare meaning, and meaning lives in embedding space, not in the bytes of the string.

How the semantic layer works

The semantic layer trades a cheap lookup for a slightly more expensive one in exchange for a much higher hit rate. The flow on a cache check is:

1. Hash the normalized query and check the exact-match layer first. If it hits, return immediately - no embedding, no vector math, no model call.
2. On an exact miss, embed the query with a small embedding model.
3. Run a nearest-neighbor search over the embeddings of previously cached queries.
4. If the closest neighbor's similarity is above your threshold, return that neighbor's stored response. This is a semantic hit.
5. If nothing clears the threshold, it is a true miss: call the LLM, then write the response, its embedding, and metadata back into both layers.

Two things make this practical. First, embedding a short query with a small model costs a tiny fraction of a single LLM completion, so the per-check overhead is real but cheap relative to the call it avoids. Second, the exact layer absorbs the highest-frequency traffic for free, so the semantic layer only runs on the long tail where it earns its keep. The dual-layer ordering is the whole trick: you never pay embedding overhead on a query the hash already answers.

The relationship between a semantic cache and a general-purpose vector database is worth being precise about, because they are easy to conflate. A vector DB is built for large-scale similarity search over a corpus; a semantic cache is a much smaller, churning index of recent query-answer pairs with aggressive eviction. We draw the line in detail in vector database vs key-value store. For agent systems people also confuse caching with memory - the distinction is that a cache is keyed by query similarity and is disposable, while addressable memory is keyed by name and is meant to persist, a line memnode draws well.

The key design: hash for exact, embedding index for semantic

This is where a durable KV store does the unglamorous, load-bearing work. Two structures cover the whole cache.

The exact layer is a flat keyspace:

• cache:exact:{sha256(model + system + temperature + query)} maps to a record holding the completion, token counts, the source query, and a created-at timestamp.

The semantic layer needs the same response records plus a way to find them by similarity:

• cache:resp:{id} maps to the full response record (completion, tokens, model, original query, created-at, hit count).
• An embedding index maps query embeddings to those cache:resp:{id} ids. In a small, fast deployment this can be an in-process approximate-nearest-neighbor index whose payloads are the response ids; the KV store remains the source of truth for the records themselves.

Storing the response payload once in the KV store and pointing both layers at it avoids duplicating completions. The exact layer gets you the instant path; the semantic index gets you the fuzzy path; the KV store holds the durable answer. Because a B+tree store like BaseKV is built for point reads and short scans - the access pattern of "fetch this one record by id" - it is a natural fit for the response store, and the persistence means a restart does not cold-start your cache. The case for durable, disk-backed storage here is the same one made in persistent key-value storage: a cache that evaporates on every deploy is a cache that is always paying full price right after you ship.

The similarity threshold is the whole ballgame

The semantic layer lives or dies on one number: the similarity threshold above which two queries are treated as the same intent. This is a genuine tradeoff with no universally correct value, and it is the single most important thing to tune.

Set the threshold too loose and the cache starts serving confidently wrong answers. "How do I cancel my subscription" and "how do I pause my subscription" can be close in embedding space while demanding different answers, and a loose threshold will hand the cancel answer to someone who asked to pause. These false hits are worse than a miss, because a miss costs money while a false hit costs trust.

Set it too tight and you collapse back toward exact match: the hit rate craters, the savings evaporate, and you are paying embedding overhead for the privilege of barely beating a hash. The right value depends on your embedding model, your domain's tolerance for near-misses, and the cost of a wrong answer in your application.

A few practices keep this honest:

• Tune the threshold against a labeled set of real query pairs, not by feel. Measure both hit rate and false-hit rate; optimizing one without watching the other is how loose thresholds ship.
• Keep the threshold conservative in high-stakes domains (billing, legal, medical) and looser only where a near-miss is cheap.
• Log every semantic hit with its similarity score so you can audit borderline cases and move the line with evidence.

Cost and risk: the two layers compared

The point of the dual-layer design is that each layer is good at a different thing, and stacking them gives you the catch rate of semantic matching with the near-zero overhead of hashing on the hot path. The numbers below are illustrative orderings, not benchmarks from your workload.

| Property | Exact-match (hash) layer | Semantic layer | Dual-layer (both) | | --- | --- | --- | --- | | Redundant-query hit rate | Low (~18% reported) | High (catches paraphrases) | Highest - exact hits free, paraphrases caught | | Per-check overhead | Minimal (one hash + one KV read) | Embedding + nearest-neighbor search | Minimal on hot path; semantic cost only on exact miss | | Risk of wrong answer | None (byte-identical only) | Real, grows as threshold loosens | Bounded - exact path is exact, semantic path is threshold-gated | | Reported cost reduction | Modest | 40-86% reported (vendor-claimed, workload-dependent) | Captures the semantic savings with lower average overhead | | Operational complexity | Trivial | Embedding model + index + threshold tuning | Moderate - two structures, one shared response store |

The honest read of that table: the exact layer is free insurance with zero downside, the semantic layer is where the large reported savings come from but it introduces a correctness risk you must actively manage, and running both is how you get most of the upside while keeping the risky path off the critical path for common queries.

Invalidation and TTL: a cache that goes stale is a bug

A semantic cache holds answers, and answers go out of date. If your refund policy changes, every cached response about refunds - exact and semantic - is now wrong, and the better your cache works the longer it will keep serving the stale version. Invalidation is not optional polish; it is part of correctness.

Useful patterns:

• TTL on every record so nothing lives forever by default. Pick the TTL from how fast the underlying truth changes: minutes for fast-moving data, days for stable reference answers.
• Namespaced or versioned keys tied to the knowledge that produced the answer, so that publishing a new policy version can invalidate a whole class of cached responses at once rather than hunting individual keys.
• Eviction by capacity and by hit recency, so the index stays small and fast. A semantic index that grows unbounded gets slower to search and dilutes the threshold's meaning.
• Bypass-and-refresh on writes to source data, so the next query repopulates the cache with the current answer instead of serving the old one until TTL expiry.

This is closely related to invalidation at the network edge - the same "how do I know this is still true" problem shows up there too, and the strategies in edge caching with a KV store translate well to the LLM case.

When to use a semantic cache, and when not to

A semantic cache is a sharp tool with a clear blast radius. Use it where queries repeat in meaning and the answer is stable; keep it away from anything that must be fresh or personalized.

Reach for a semantic cache when:

• Traffic has high intent-level repetition phrased many ways (support assistants, FAQ bots, documentation Q&A, internal knowledge tools).
• The answer to a given intent is stable over a known window, so a TTL can keep it honest.
• LLM spend is a meaningful line item and you have measured how much of it is redundant. If you have not measured it, start there - the hidden cost of LLM APIs from usagebox is a good frame for what to count.
• You can tolerate and tune the false-hit risk, with logging in place to audit it.

Do not reach for a semantic cache when:

• Answers must be fresh or real-time (live prices, inventory, account balances, anything time-sensitive). A cached answer is a wrong answer here.
• Responses are personalized or contain per-user data. Two users can phrase the same question and need different answers, and a similarity match will happily cross-serve them - a correctness and privacy problem at once.
• The cost of a subtly-wrong answer is high (medical, legal, financial advice). The threshold can never be set safely enough to justify the risk.
• Your traffic genuinely does not repeat at the intent level, in which case you will pay embedding overhead for a hit rate that does not materialize.

The exact-match layer, by contrast, is almost always safe to run because it only ever returns byte-identical matches. When in doubt, ship the hash layer first, measure, and add the semantic layer where the redundancy is.

Common questions

Does the semantic layer slow down every request? No, and that is the point of putting it second. Every request checks the exact-match hash layer first, which is a single hash and one KV read. Only requests that miss the exact layer pay the embedding and nearest-neighbor cost. High-frequency queries get absorbed by the free path, so the average per-request overhead stays low even though the semantic path is more expensive in isolation.

How is this different from prompt caching offered by the model providers? Provider-side prompt caching reuses the compute for a shared prefix of tokens (a long system prompt, for example) within a provider's own window, and it still bills you, just less. A semantic cache lives on your side, can avoid the model call entirely on a hit, works across providers, and matches on meaning rather than on a token prefix. They are complementary: provider caching cuts the cost of the calls you do make, and a semantic cache cuts how many calls you make at all.

What happens to cache hits when I change the underlying knowledge or prompt? They become stale and must be invalidated, or they will keep serving the old answer. This is why versioned or namespaced keys plus TTLs matter: tie cached responses to the version of the knowledge that produced them, so a content change can invalidate a whole class of entries at once. Relying on TTL alone means serving wrong answers until expiry, which the better your cache works, the longer that lasts.

---

Related: Caching LLM Responses to Cut OpenAI Bills, Vector Database vs Key-Value Store, Edge Caching with a Key-Value Store, Persistent Key-Value Storage.