Skip to content

LLM Semantic Cache#

An LLM semantic cache sits in front of the model. Before calling the LLM, hash the prompt's meaning (its embedding) and check if a prior response with a similar embedding exists. If yes, serve from cache and skip a $0.02 LLM call.

flowchart LR
  U([User query]) --> E[Embed query]
  E --> V[(Vector cache<br/>top-1 search)]
  V --> S{Sim > tau?}
  S -- yes --> H[Return cached response]
  S -- no --> L[Call LLM]
  L --> W[Write embedding + response]
  W --> V
  L --> R([Response])
  H --> R

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class U,R client;
    class E,S,W service;
    class V datastore;
    class L,H compute;

The key parameter is the similarity threshold tau. Too low and you serve wrong answers to subtly different questions ("Is X open today?" vs "Is X open tomorrow?"). Too high and you almost never hit the cache. Tuning tau on a held-out set is the work.

Per-tenant isolation is mandatory: tenant A must never see tenant B's cached response, even if their queries embed identically. Partition the index by tenant ID.

Invalidation is the hard part. When you change the prompt template, the underlying docs, or the model, every cached response is potentially wrong. Tag entries with template_version, model_id, and docs_version; bump invalidates the slice.

Hit rate on production chatbots typically lands between 20 and 50 percent for FAQ-style traffic. At $0.02 per LLM call and 100M queries per month, a 30 percent hit rate saves $600K. The vector cache costs perhaps $1K to host. GPTCache, Redis with vector search, and Pinecone are common implementations.

Problem statement

Your support chatbot serves 100 million queries per month at $0.02 per LLM call: $2M/month. Inspection shows 30 percent of queries are rephrasings of FAQs ("how do I reset my password?" vs "I forgot my password"). Design an LLM semantic cache that captures that 30 percent without serving wrong answers, isolates tenants, and survives prompt-template changes.

Architecture#

flowchart TB
  U([Request: tenant=T, query=Q]) --> N[Normalize<br/>strip PII, lowercase]
  N --> E[Embed query]
  E --> P[Partitioned ANN search<br/>tenant=T, template_ver=V, model=M]
  P --> TOP[Top-1 candidate]
  TOP --> S{Sim score >= tau?}
  S -- yes --> G[Optional LLM judge<br/>verify equivalence]
  G --> OK{Verified?}
  OK -- yes --> RET[Return cached]
  S -- no --> L[Call LLM]
  OK -- no --> L
  L --> RESP[Response]
  RESP --> W[Write entry<br/>embed + response + metadata]
  W --> IDX[(ANN index<br/>HNSW or IVF)]
  RESP --> U2([Response])
  RET --> U2

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class U,U2 client;
    class N,E,P,S,W,OK service;
    class IDX datastore;
    class L,G,RET,RESP compute;

A semantic cache is a vector store plus a similarity gate. The flow per request:

  1. Normalize the query (strip PII, lowercase, remove punctuation).
  2. Embed with a fast model (e.g. text-embedding-3-small, ~30 ms).
  3. ANN search within the tenant + template-version + model partition.
  4. Compare top-1 similarity against threshold $\tau$.
  5. Optionally, LLM judge check (cheap small model) confirms the cached response answers the new query.
  6. On hit, return cached response. On miss, call LLM and write the new entry.

The threshold tradeoff#

flowchart LR
  L1[tau = 0.80] --> X1[High hit rate, many false positives]
  L2[tau = 0.92] --> X2[Sweet spot: 30% hits, <1% false positives]
  L3[tau = 0.98] --> X3[Low hit rate, near-zero false positives]

    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class L1,L2,L3 service;
    class X1 datastore;
    class X3 compute;
    class X2 service;

A false positive is when two queries embed similarly but expect different answers. Classic traps:

  • "Is X open today?" vs "Is X open tomorrow?" - one word, opposite answer
  • "Cancel my order" vs "Cancel my subscription" - similar embed, different action
  • "What is the price in USD?" vs "What is the price in EUR?"

The cure is query templating: detect entities (dates, currencies, products, IDs) and only cache the template, not the entity. "Is X open ?" with cached answer keyed by template + extracted entities. This is essentially structured caching, much more accurate than pure semantic.

Hybrid: exact + semantic#

def lookup(query, tenant, template_ver, model_id):
    # Layer 1: exact hash on normalized query (free, instant)
    key = normalize_hash(query)
    hit = exact_cache.get((tenant, template_ver, model_id, key))
    if hit:
        return hit, "exact"

    # Layer 2: semantic ANN
    emb = embed(query)
    candidates = ann.search(
        emb,
        filter={"tenant": tenant, "template_ver": template_ver, "model_id": model_id},
        k=3,
    )
    if not candidates:
        return None, "miss"

    top = candidates[0]
    if top.score < TAU:
        return None, "miss"

    # Layer 3: optional LLM judge for high-stakes routes
    if route.high_stakes and not llm_judge_equivalent(query, top.cached_query):
        return None, "miss_judge"

    return top.response, "semantic"

Three layers: exact match (free, 5 to 10 percent of hits), semantic (the bulk), LLM judge (precision boost on critical paths).

Eviction#

flowchart LR
  W[Write] --> CHK{Cache full?}
  CHK -- yes --> EV[Evict by policy]
  EV --> L1[LRU]
  EV --> L2[LFU + recency]
  EV --> L3[TTL expired]
  EV --> L4[Tagged invalidation]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class CHK,EV service;
    class L1,L2,L3,L4 datastore;

Eviction policies, ranked by usefulness for LLM caches:

Policy Pro Con
TTL (e.g. 7 days) Bounds staleness Cold start each TTL
LRU Simple, OS-friendly Hot items refresh forever, stale
LFU + recency (W-TinyLFU) Best hit rate More state
Cost-aware (size * hits / age) Memory-efficient Custom code
Tagged invalidation Surgical Requires good tagging

In practice: TTL + LFU + tagged invalidation. TTL bounds risk, LFU optimizes the hot set, tags handle template / doc / model changes.

Cache invalidation: the hard problem#

flowchart LR
  T[Template change] --> INV1[Invalidate by template_ver]
  D[Doc updated] --> INV2[Invalidate by doc_id tag]
  M[Model upgrade] --> INV3[Invalidate by model_id]
  P[Policy change<br/>e.g. refund rules] --> INV4[Manual bulk invalidate]
  INV1 --> X[(Mark stale<br/>or hard delete)]
  INV2 --> X
  INV3 --> X
  INV4 --> X

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class T,D,M,P,INV1,INV2,INV3,INV4 service;
    class X datastore;

Every cache entry carries metadata:

{
  "tenant_id": "acme",
  "template_version": "v17",
  "model_id": "claude-opus-4-7",
  "embed_model": "text-embedding-3-small-v2",
  "docs_versions": ["faq:v8", "policy:v3"],
  "created_at": "2026-05-01T10:00:00Z",
  "ttl_seconds": 604800
}

When any tagged dependency changes, mark matching entries stale. Lazy invalidation (mark stale, evict on read) is cheaper than eager (delete now). For high-stakes content (refund policy, medical), eager.

Per-tenant isolation#

A semantic cache must not leak across tenants. Two designs:

Approach Pro Con
Per-tenant index Hard isolation, simple Many small indices, memory bloat
Shared index + tenant filter Memory-efficient Filter must be honored every read

Pinecone namespaces, Weaviate tenants, and Qdrant collections all map to "per-tenant index". Redis Stack vector search uses tag filters. The choice depends on tenant count: for <100 tenants, per-tenant; for >10K, shared with filter and strong testing.

Hit-rate measurement and tuning#

flowchart LR
  Q[Live queries] --> SHADOW[Shadow mode<br/>cache lookup but always call LLM]
  SHADOW --> M[Measure:<br/>cache hit + LLM agreement]
  M --> TUNE{Adjust tau}
  TUNE --> PROD[Promote when FP rate < 1%]

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class SHADOW,M,TUNE service;
    class PROD compute;

Run the cache in shadow mode first: always call the LLM, also do the cache lookup, log whether the cached response would have matched the live response (via judge or exact match). This gives you the true false-positive rate at each tau without risking users.

Metrics to track:

  • Hit rate = cache_hits / total_requests
  • False positive rate = wrong_cache_hits / cache_hits (requires sampling judge)
  • Latency saved = mean(LLM_latency) * hit_rate
  • Cost saved = LLM_cost_per_call * hit_rate * total_requests

GPTCache-style architectures#

GPTCache (Zilliz, 2023) is the open-source reference implementation. Its layers:

  1. Pre-processor: normalize the request, extract metadata
  2. Embedding generator: SentenceTransformers, OpenAI, etc.
  3. Cache storage: SQLite, Redis, MySQL for the response payload
  4. Vector store: FAISS, Milvus, Chroma for the embedding index
  5. Similarity evaluator: pluggable, scalar score
  6. Post-processor: optional LLM judge or templating

The architecture is layered for a reason: each component has different scale and latency requirements. The vector store is fast and memory-bound; the response storage is slow and disk-bound.

Production failure modes#

Failure Symptom Fix
Tenant leak User A sees user B's data Filter at query, write tenant test
Stale after template change Old response with new template Bump template_version, tag entries
Embedding drift Hit rate drops after embed model upgrade Re-embed cache or partition by embed_model
FAQ skew Hot 10 queries dominate, cache useless tail Two-tier: small exact cache for hot, semantic for tail
Privacy leak in embeddings PII in stored queries Strip PII before embed + store
Cache poisoning Adversarial query gets bad answer cached Negative feedback button invalidates entry

When NOT to use a semantic cache#

  • Personalized responses: each user gets a different answer; cache hit rate near zero
  • Stateful conversations: caching one turn breaks the next; cache the whole transcript or skip
  • High-stakes routes: medical advice, financial transactions; LLM judge or skip cache
  • Streaming UX critical: cache hit returns full text instantly, breaking the streaming feel; mitigate by re-streaming character-by-character

Quick reference#

Idea#

Embed query, ANN-search a prior cache, serve cached response if similarity > tau. Skip the LLM call.

Flow#

  1. Normalize (PII strip, lowercase)
  2. Embed query
  3. ANN search within tenant + template_ver + model partition
  4. Compare top-1 similarity vs tau
  5. Optional LLM judge for high-stakes routes
  6. Return cached or call LLM and write

Threshold tuning#

  • tau ~0.92 is a typical sweet spot
  • Lower = more hits, more false positives
  • Tune in shadow mode (always call LLM, log agreement)
  • High-stakes routes: lift tau and add LLM judge

False positive traps#

  • Date-shifting questions ("today" vs "tomorrow")
  • Entity differences (USD vs EUR)
  • Negation flips ("can I" vs "can I not")
  • Fix with entity-aware templating

Hybrid layers#

  1. Exact hash on normalized query (free, 5-10% hits)
  2. Semantic ANN (bulk of hits)
  3. LLM judge (precision on critical routes)

Eviction#

Policy Use
TTL Bound staleness, default
LRU Simple baseline
LFU + recency Best hit rate
Tagged Surgical invalidation

Typical stack: TTL + LFU + tagged.

Invalidation tags#

Store per entry: - tenant_id - template_version - model_id - embed_model_version - docs_versions[] - created_at, ttl

Bump any tag -> invalidate matching entries (lazy ok for non-critical).

Tenant isolation#

  • Per-tenant index: <100 tenants, hard isolation
  • Shared + filter: >10K tenants, memory efficient
  • Test: query as tenant A, must never see tenant B

Metrics#

  • Hit rate: hits / total
  • False positive rate: wrong_hits / hits (sample with judge)
  • Cost saved: $/call * hit_rate * total
  • Latency saved: mean LLM ms * hit_rate

Tools#

  • GPTCache (Zilliz, reference impl)
  • Redis Stack (vector search + tags)
  • Pinecone namespaces / Weaviate tenants
  • Qdrant collections
  • Langchain CacheBackedEmbeddings

When NOT to cache#

  • Personalized responses
  • Stateful multi-turn convos
  • High-stakes (medical, financial) without judge
  • Streaming-critical UX (full text breaks streaming feel)

Failure modes#

  • Tenant leak: bad filter
  • Stale on template change: bump version
  • Embed drift: re-embed or partition by embed_model
  • Cache poisoning: feedback button invalidates

Refs#

  • Bang et al., GPTCache (2023)
  • LangChain caching docs
  • Anthropic prompt caching (different but related)

FAQ#

What is an LLM semantic cache?#

A semantic cache stores past prompt embeddings and their LLM responses. Before each new call, it checks if a stored prompt is similar enough to reuse the answer, which avoids a paid model call.

How is semantic cache different from prompt caching?#

Prompt caching reuses the KV cache of an identical prefix inside the model. Semantic caching reuses an entire response when a new prompt is semantically close to a previous one, even if the wording differs.

How do you tune the similarity threshold?#

Sweep the cosine similarity threshold against an eval set that mixes near-duplicate queries and meaning-shifting queries. Pick the highest threshold that does not return wrong answers for shifted queries.

What are the risks of semantic caching?#

False positives. Two prompts can be embedding-close but mean different things, so the cache serves a wrong answer. Strict thresholds, per-tenant isolation, and bypass for low-confidence hits mitigate this.

Where should I run a semantic cache?#

Run it as a thin proxy in front of the LLM API with a vector index like Redis Vector, Qdrant, or pgvector. Key entries by tenant id and embedding so caches do not leak between users.