How to cache LLM responses with Redis and cut API calls

A 0.12 semantic-distance threshold changes the cache outcome: lower latency and fewer redundant LLM calls.

The common mistake is caching only a prompt while omitting the model, system instructions, and access filters that shape the answer.

Define the request boundary before Redis sees it: normalized user input plus every answer-affecting field.

Exact lookup should be the cheapest branch in the request path.

Call the LLM only after both cache layers miss.

LLM response caching prerequisites for safe Redis keys

The 0.12 semantic-distance threshold changes the cache outcome: lower latency and fewer redundant LLM calls. [unverified] The common mistake is caching only a prompt while omitting the model, system instructions, and access filters that shape the answer. [unverified] Define the request boundary before Redis sees it: normalized user input plus every answer-affecting field. [unverified] According to RedisVL, an entry_id is a deterministic hash of the prompt and filters, while the full key combines a Redis prefix with that identifier. Treat that fingerprint as an authorization-aware contract, not a convenient string. [unverified]

Client → request fingerprint → exact Redis lookup → embedding
       → semantic Redis lookup → LLM API on miss
       → write exact cache + semantic cache

Prerequisites

Set these inputs before constructing a key, and keep secrets in environment-backed configuration rather than in the prompt. [unverified] Use explicit identifiers rather than raw instruction text when your deployment manages prompt versions. [unverified]

• Redis connection details: endpoint, TLS setting, authentication material
• Python environment and redis-py
• LLM API credential
• Embedding model for semantic matching
• Chosen namespace
• Model and system-prompt identifiers
• Tenant or permission filters where answers vary by access

export REDIS_URL="..."
export LLM_API_KEY="..."
export CACHE_NAMESPACE="..."
python -m pip install redis

request_scope = {
    "namespace": "...",
    "model_id": "...",
    "system_prompt_id": "...",
    "tenant": "...",
    "permission_filter": "...",
    "semantic_distance_threshold": 0.12,
}

Canonicalize maps, ordering, whitespace, and omitted defaults before hashing; otherwise equivalent requests generate different fingerprints. [unverified] The same prompt and filters yield the same entry_id and overwrite the prior value. Different filters yield different entry_ids, allowing both values to be stored.

Never share a cached answer across authorization boundaries.

Troubleshooting FAQ

Q: ConnectionError from Redis. A: Redis is unreachable; verify endpoint, TLS, authentication, and network path. [unverified]

Q: The LLM API credential is missing. A: The credential is absent; set secret-backed runtime configuration and restart. [unverified]

Q: The cache key changes unexpectedly between identical requests. A: Serialization differs; canonicalize order, whitespace, defaults, identifiers, and filters before hashing. [unverified]

LLM response caching architecture for Redis lookups

Treat each cache decision as an explicit outcome, not a hidden optimization. Caching retains a computed result for reuse instead of recalculating it. Return the answer with cache_status so callers, traces, and cache inspection can identify an exact, semantic, or LLM result.

Use one request contract at the application boundary; do not allow a caller to omit context that changes an answer. Normalize only the prompt representation used for exact matching, while preserving the original prompt for inspection.

{
  "prompt": "Summarize the tenant's support tickets",
  "model": "model-id",
  "system_prompt_version": "prompt-version",
  "tenant_id": "tenant-id",
  "filters": {
    "document_scope": "support"
  },
  "ttl_seconds": "expiry-window"
}

{
  "response": "Stored or generated answer",
  "cache_status": "exact | semantic | llm"
}

Build the exact key from the normalized prompt and the context that controls generation. The model, system prompt version, tenant identifier, and normalization rules must remain stable between writes and reads; changing any selects a different exact entry.

llm:exact:{sha256(model|system_prompt_version|tenant_id|normalized_prompt)}

Exact lookup should be the cheapest branch in the request path.

Benchmarking LLM Exact and Semantic Caching with Redis reports that the two-level path checks exact results before semantic results and writes a new LLM result to both layers after a miss.

• Normalize the supplied prompt, then hash the canonical exact-key input before the first Redis lookup.
• After an exact miss, embed the normalized prompt; do not pay for embedding on an exact hit.
• Search semantic entries using the request filters before accepting a nearest response.
• Call the LLM only after both cache layers miss.
• Write the resulting answer to both cache layers after that miss.

Represent the semantic entry as an indexed Redis record with fields that make its decision inspectable:

• prompt text
• embedding vector
• response payload
• model
• filters
• creation timestamp
• expiry policy

Keep the expiry policy beside the response so cleanup does not leave a vector candidate without its answer.

**Why it matters:** Your semantic match is only safe when the stored response and the lookup use the same model, instructions, and authorization context. [unverified]

On a semantic hit, return the stored payload without presenting it as an exact prompt match. That status is the boundary between reuse and inspection.

LLM response caching thresholds and storage choices

Put exact matching ahead of semantic lookup so repeat requests do not need embedding work. According to the Redis caching guide, Redis supports both exact-match and semantic caching for LLM responses. Include the model name in the identity so a response from one model does not satisfy a request for another.

Normalize the prompt before hashing it. The documented exact implementation strips surrounding whitespace, lowercases the prompt, and applies SHA-256. Then combine the query and model name before deriving the exact cache key. Read Redis before calling the model and write only a successful response.

import hashlib
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def normalize_prompt(prompt: str) -> str:
    return prompt.strip().lower()

def exact_key(model: str, prompt: str) -> str:
    material = f"{model}:{normalize_prompt(prompt)}"
    digest = hashlib.sha256(material.encode()).hexdigest()
    return f"llm:exact:{digest}"

key = exact_key(model, prompt)
cached = r.get(key)                         # GET

if cached is not None:
    return cached

answer = call_llm(model=model, prompt=prompt)
r.execute_command("SET", key, answer, "EX", 3600)  # SET... EX 3600
return answer

A semantic hit is not an exact-string hit.

Use all-MiniLM-L6-v2 to generate the semantic query vector. Semantic caching converts queries into vectors and compares their meaning with cosine similarity. Start with a cosine-distance threshold of 0.12; a lower distance is a hit. Keep the benchmark’s 0.15 threshold separate from this starting configuration, and run application-specific test prompts before changing either value.

# Pseudocode: initialize this during application startup.
semantic_cache = SemanticCache(
    name="llm-semantic",
    redis_url="redis://localhost:6379",
    ttl=3600,
)

embedder = SentenceTransformer("all-MiniLM-L6-v2")
query_vector = embedder.encode(normalize_prompt(prompt))

hit = semantic_cache.search(
    vector=query_vector,
    filter={"model": model},       # filtered lookup before the LLM call
    distance_threshold=0.12,
)

if hit and hit.distance < 0.12:
    return hit.response

answer = call_llm(model=model, prompt=prompt)
semantic_cache.store(
    vector=query_vector,
    response=answer,
    metadata={"model": model},
)
return answer

Initialize SemanticCache on the startup path rather than treating index creation as a separate deployment task. SemanticCache automatically creates its Redis index when initialized. Store the model as filterable metadata, apply the same expiration policy to semantic entries, and keep the semantic write after both cache layers miss.

LLM response caching failures that cause wrong answers

Treat every cache incident as a classification problem, not a prompt-quality debate. Emit a cache-key log field, a semantic distance log field, and cache_status on every request; its value must be exact_hit, semantic_hit, or miss.

Use this decision list before changing a threshold or flushing data:

• Exact key missing when it should hit. Compare the cache-key log field from the write path with the reader’s key, then inspect the stored value. If they differ, the lookup inputs are not identical.
• Semantic query returns no candidate. Log the semantic distance and the candidate count, then determine whether the index returned nothing or your threshold rejected the nearest result.
• Semantic query returns an unsafe candidate. Inspect candidate metadata beside the semantic distance log field. A plausible distance does not compensate for omitted request filters or a response that belongs to a different context.
• Redis entry is absent or expired. Check the value and expiry state directly; an absent value means lookup logic cannot return a response, regardless of a previous cache report.
• LLM call still occurs after a reported hit. Trace the request through the branch that sets cache_status; a hit must return the cached payload and terminate the generation path.

TTL <key>
GET <key>

For an expected hit, GET should return the serialized entry you intend to serve, while TTL lets you distinguish a present entry from one that has expired or was never written.

A fast wrong answer is a cache correctness failure.

Keep response caching separate from KV reuse

Full prompt-response caching stores a completed result that your application can return without calling the model again. A KV cache instead stores computed key and value tensors from self-attention so later computations can reuse them.

According to Boost LLM inference with LMCache and Redis, LMCache accelerates LLM serving through KV-cache reuse for repeated token sequences. LMCache works at the token-chunk level rather than caching full prompts or responses.

Redis is LMCache’s default remote store. LMCache does not provide KV reuse for hosted OpenAI or Anthropic APIs.

**Note:** If you call a hosted LLM API, implement response caching at your application boundary; do not plan on server-side token-cache reuse from the provider path.

LLM response caching diagnostics for misses and stale data

Treat diagnostics as a comparison between the request you received, the cache record you expected, and the record Redis actually stored. According to RedisVL, semantic caching returns responses for semantically similar prompts instead of making redundant API calls, reducing API costs and latency.

Inspect the resolved key rather than only the visible prompt. An entry has an entry_id, a deterministic hash of prompt and filters, and a full key formed from prefix and entry_id; use the full key for direct Redis inspection.

• The resolved namespace and full key
• The fingerprint inputs: normalized prompt, model, and system-prompt version
• The semantic query text, filters, returned distance, and expiry state

For suspected overwrites, compare the old and new values at the resolved key after sending the same prompt and filters. The same prompt and filters produce the same entry_id and overwrite the previous entry. Different filters produce different entry_ids, so both records can remain stored.

Delete contaminated entries before retuning the threshold.

When output looks stale, compare the cache record’s prompt and filters with the request that generated the observed answer. A mismatch identifies a selection problem; a match shifts attention to expiry and write behavior. [unverified]

Troubleshooting FAQ

Q: Identical visible prompts always miss. A: The model, system-prompt version, or normalization differs between requests; log those key inputs beside the computed fingerprint and standardize the fingerprint function. [unverified]

Q: A paraphrase receives an unrelated answer. A: The distance threshold is too permissive; lower the allowed maximum distance, add filters, and replay a labeled prompt set before accepting the new setting. [unverified]

Q: Users receive an answer written for another tenant. A: The semantic lookup omitted tenant_id or permission filters; add those filters to both the entry and query, then delete affected entries.

Q: TTL returns -2. A: The key does not exist; inspect the write branch, namespace, and TTL configuration before calling the LLM again. [unverified]

Q: A previous answer persists after a prompt-policy change. A: Find entries created under the old filter or fingerprint inputs, remove them from the affected namespace, and verify that the next write uses the revised inputs. [unverified]

LLM response caching steps to ship and verify Redis

Set the cache contract before wiring inference: exact keys represent normalized request identity, while semantic entries carry the response, embedding, and isolation metadata. According to Benchmarking LLM Exact and Semantic Caching with Redis, the request path must check exact storage first, then semantic storage, and write an inference result to both after a miss.

1. 1. Set environment variables for the Redis endpoint, LLM model identifier, embedding configuration, cache expiry, and tenant context. Keep tenant and model values available to both key construction and semantic filters so isolation is enforced at lookup time. [unverified]

   export REDIS_URL="redis://..."
   export LLM_MODEL="..."
   export EMBEDDING_MODEL="..."
   export CACHE_TTL_SECONDS="..."
   export TENANT_ID="..."
   

1. 2. Connect before accepting traffic, then verify the deployment can reach Redis with redis-cli PING. Treat any response other than PONG as a failed readiness check rather than silently bypassing the cache. [unverified]

   redis-cli -u "$REDIS_URL" PING
   

1. 3. Implement an exact fingerprint from a normalized request object containing the prompt, model identifier, tenant, and response-affecting options. Serialize fields in a stable order, normalize irrelevant whitespace, hash the serialized value, and prefix the resulting key by cache purpose. [unverified]

   const normalized = JSON.stringify({ prompt: prompt.trim(), model, tenant, options });
   const exactKey = `llm:exact:${sha256(normalized)}`;
   

1. 4. Add the exact GET path before embedding work. Return the stored answer immediately when the key exists, because the specified two-level flow checks exact storage first.

   const exact = await redis.get(exactKey);
   if (exact) return {...JSON.parse(exact), cache_status: "exact_hit" };
   

1. 5. Configure the semantic embedding and index to store a query vector alongside response metadata. Include fields for tenant, model, expiry, and the original normalized prompt so filtered retrieval can reject entries outside the current request scope. [unverified]

1. 6. After an exact miss, embed the normalized prompt and perform semantic filtered lookup using the current tenant and model identifier. Return the matched answer only when its distance passes your approved threshold and its metadata filters match the request. [unverified]

1. 7. Call the LLM only after both cache layers miss. This ordering avoids inference when either an identical request or an approved semantically similar request already has a usable response.

1. 8. Write the new response and metadata to both layers after inference. The exact layer receives the fingerprinted payload, while the semantic layer receives the embedding, answer, filters, and provenance needed for later retrieval.

1. 9. Add expiry to both writes and emit structured cache logs for exact_hit, semantic_hit, and misses. Log the fingerprint, tenant, model, semantic distance, applied filters, and cache status without logging sensitive prompt content. [unverified]

1. 10. Send the same prompt twice and assert exact_hit on request two. Then send a controlled paraphrase and inspect the returned semantic distance, applied filters, answer suitability, and semantic_hit status before accepting the match. [unverified]

1. 11. Change either the tenant filter or model identifier and confirm the request cannot reuse the earlier entry. Check total_calls, cache_hits, and hit rate, where hit rate is cache hits divided by total calls and expressed as a percentage.

const hitRate = total_calls ? (cache_hits / total_calls) * 100: 0;

Ship only after your labeled test set finds no unacceptable semantic matches.

LLM response caching works when keys and thresholds match

Redis response caching delivers lower latency and fewer LLM calls when every answer-affecting field is part of the request contract and semantic retrieval uses that same context. Exact lookup must run before embedding and generation, while semantic reuse requires an approved distance threshold.

Never share cached answers across authorization boundaries.

Key takeaways

• Build each exact fingerprint from the normalized prompt, model, system-prompt version, tenant, filters, and response-affecting options.
• Check Redis with exact GET before embedding; call the LLM only after exact and semantic lookups miss.
• Store tenant and model as semantic metadata, then apply those filters before accepting a vector candidate.
• Start semantic matching at a cosine-distance threshold of 0.12, replay labeled prompts, and reject unsafe matches.
• Apply expiry to both layers and log the fingerprint, filters, semantic distance, and cache_status for every request.

Sources

• 1. [Cache LLM Responses — RedisVL]( https://docs.redisvl.com/en/latest/user_guide/03_llmcache.html)