Reduce AI latency and improve responsiveness by integrating Azure Cache for Redis with LLM workloads

Cost & Latency FixRedis EnterpriseRediSearchSemantic Cache

Slashing Latency: Deploying an Azure Cache for Redis
Gateway in Front of Your LLM Endpoints

"What's your return policy?" and "How do returns work?" are the same question wearing different clothes. An exact-match cache sees two different strings and calls the model twice. A semantic cache sees two vectors four degrees apart and answers the second one in under 200 milliseconds, for free. The gap between those two outcomes is one architectural decision — and one Azure tier most teams don't realize they need.

The problem this guide resolves — measured, not assumed

# A support chatbot's actual traffic pattern over one hour, deduplicated
# by MEANING rather than by exact string match:

  Query                                          | Count | LLM calls today
  ------------------------------------------------|-------|------------------
  "What's your return policy?"                    |   1   | 1
  "How do returns work?"                           |   1   | 1  <- SAME QUESTION
  "Can I return an item I bought last week?"        |   1   | 1  <- SAME QUESTION
  "What's the process to send something back?"      |   1   | 1  <- SAME QUESTION
  ... (41 more phrasings of the identical question)

  TOTAL exact strings:        44 unique          -> 44 LLM calls today
  TOTAL semantic clusters:     1 real question    -> should be ~1 LLM call
                                                      + 43 cache hits

# Cost of the naive approach, this ONE question, this ONE hour:
  44 calls x (~180 input tokens + ~220 output tokens) x $10/1M output
  + $2.50/1M input (GPT-4o Global Standard, illustrative rates)
  = 44 calls, full latency (800ms-2.5s each) and full price, EVERY time,
    for a question the system had already answered 43 times before.

# What an EXACT-match cache (e.g. plain Redis GET/SET on the raw prompt
# string) does with this traffic: essentially nothing. Zero cache hits,
# because none of the 44 strings are byte-for-byte identical.

# What semantic caching resolves: query #2 embeds to a vector 0.03
# cosine-distance from query #1's stored vector -> HIT -> 180ms response,
# zero LLM tokens spent, zero backend load added.

Symptom: Repetitive, high-volume LLM traffic where the same underlying question arrives in dozens of different phrasings, each treated as a fresh, uncached request.  Failure point: Caching implemented (if at all) as exact key-value matching on the raw prompt string, which cannot recognize that two differently-worded questions mean the same thing.  Default platform behaviour: Neither Azure OpenAI nor a plain cache layer deduplicates by meaning. Nothing before this fix distinguishes "the same question, reworded" from "a genuinely new question" — every request is treated as unprecedented.

Enterprise only

RediSearch — the vector-search module semantic caching requires — is available only on Azure Cache for Redis Enterprise and Enterprise Flash tiers. Not Basic, Standard, or Premium

Lower = closer

The score-threshold is a distance metric, not a similarity score. Smaller values mean stricter matching — the opposite of what the name suggests

<200ms

Measured cache-hit response time in Microsoft's own reference implementation — over 10x faster than a full LLM completion call

Built-in option

Azure API Management ships native llm-semantic-cache-lookup/-store policies — a lower-effort alternative to a hand-built gateway

Table of Contents

1. 01The Tier Gate: Why "Azure Cache for Redis" Alone Isn't Enough
2. 02How Semantic Caching Actually Works
3. 03The score-threshold Trap: Lower Is Stricter, Not Looser
4. 04Architectural Topology: Failing vs Remediated
5. 05Fix 1 — Provision the Right Tier and Enable RediSearch
6. 06Fix 2 — Build the Custom Semantic Cache Gateway
7. 07Fix 3 — Or Use APIM's Built-In Semantic Caching Policies
8. 08Fix 4 — Safeguards: Cross-User Leakage and Wrong-Answer Risk
9. 09Anti-Patterns: Caching That Saves Money and Loses Trust
10. 10Validation & Verification: Confirm the Fix

Most repetitive LLM traffic isn't repetitive in the way a plain cache can see. Users don't type identical strings — they type the same intent in dozens of different words, and a key-value cache keyed on the raw prompt treats every one of those phrasings as a brand-new question, calling the model, paying full token price, and absorbing full latency every single time. A semantic cache fixes this by comparing meaning instead of text: it embeds the incoming query into a vector, searches a vector index of previously-answered questions for the nearest neighbor, and if that neighbor is close enough, returns the stored answer without ever touching the LLM. Azure Cache for Redis is a genuinely strong platform for this — but only the tier of it that actually supports vector search, and only once you understand a scoring convention that is, on its face, backwards from what most engineers assume.

Figure 1 — The semantic cache gateway sits between the client and Azure OpenAI, deciding hit or miss

Every query passes through the gateway, which embeds it and searches a Redis vector index for the nearest previously-cached question. If the distance is below the configured threshold, the stored answer returns immediately — no LLM call, no token cost, sub-200ms latency. If not, the request proceeds to Azure OpenAI as normal, and the new question-answer pair is written back for future hits. None of this works without RediSearch, which is an Enterprise-tier-only capability.

01The Tier Gate: Why "Azure Cache for Redis" Alone Isn't EnoughCorrection

This is the single most important thing to get right before writing any code, because getting it wrong means discovering the whole architecture is impossible only after you've already provisioned the wrong resource. Semantic caching depends entirely on vector similarity search — finding the nearest previously-cached question to an incoming query — and in Redis, that capability comes from a specific module: RediSearch. Microsoft's own documentation is explicit: RediSearch, including its vector similarity search features, is available only on the Enterprise and Enterprise Flash tiers of Azure Cache for Redis.

Tier	RediSearch / vector search	Can build semantic caching?
Basic	Not available	No
Standard	Not available	No
Premium	Not available	No
Enterprise	Available	Yes
Enterprise Flash	Available (RediSearch in preview)	Yes

This isn't a minor feature gap — it's a hard architectural wall. A Standard or Premium cache, however large or well-configured, simply has no vector index type to create, no HNSW algorithm to query, and no way to perform the "find the nearest previously-answered question" operation that the entire pattern depends on. If your team already has a Premium-tier cache in production for ordinary caching duties, that instance cannot be repurposed for semantic caching — it needs a separate Enterprise-tier deployment, or a migration.

Enterprise clustering policy matters too, not just the tier

When provisioning the Enterprise tier specifically for RediSearch, the Clustering Policy must be set to Enterprise — not OSS Cluster. RediSearch is only supported under the Enterprise cluster policy. This is a setting chosen at cache creation time in the portal or CLI, easy to get wrong if you're following generic Redis provisioning habits from a non-search use case, and it isn't something you can change after the fact without recreating the cache.

Azure Managed Redis is the newer, first-party path — worth knowing the name

Microsoft's more recent guidance increasingly points to Azure Managed Redis (AMR) — a newer, fully-managed first-party Redis offering — as the forward path for these AI-adjacent scenarios, including semantic caching, vector stores, and conversation memory. Functionally, for this article's purposes, the requirement is the same: vector search capability requires the tier that includes RediSearch. If you're standing up new infrastructure rather than working with an existing Azure Cache for Redis deployment, check current documentation for whether Azure Managed Redis or Azure Cache for Redis Enterprise is the currently-recommended resource type, as Microsoft's naming and default recommendations in this space have been evolving.

02How Semantic Caching Actually WorksConcept

The mechanism is straightforward once the tier requirement is satisfied. Every incoming query goes through the same three-step check before it's allowed anywhere near the LLM.

Step	What happens	Cost
1. Embed	The incoming query text is converted to a vector via an embeddings model call (a separate, cheap deployment — not the completions model)	Small — one embedding call, always incurred, hit or miss
2. Search	Redis's HNSW-indexed vector search finds the nearest previously-cached query vector, using cosine distance (or another configured metric)	Sub-millisecond at Redis's typical scale for this use case
3. Decide	If the nearest match's distance is below the configured threshold, return its cached answer. Otherwise, proceed to the LLM and cache the new result	Zero (hit) or full LLM cost (miss)

The economics only work because Step 1's embedding call is dramatically cheaper than a full chat completion — a small, fast embeddings model call costs a small fraction of what a GPT-4o-class completion costs, and every cache hit saves the entire completion cost while paying only the embedding cost. Even a modest hit rate compounds quickly on high-volume, repetitive traffic like FAQ bots, internal support tools, and customer-facing chat with predictable question patterns.

The embeddings call is NOT optional overhead you can skip on a miss

A detail worth internalizing: the embedding step runs on every query, hit or miss — you have to embed the query to know whether it matches anything cached. This means semantic caching adds a small, fixed cost and a small, fixed latency (typically tens of milliseconds) to every single request, in exchange for potentially eliminating the much larger cost and latency of a full completion call on a hit. For workloads with very low repeat-question rates, this fixed overhead without a corresponding high hit rate can make semantic caching a net negative — know your traffic's actual repetition pattern before assuming this pays for itself.

03The score-threshold Trap: Lower Is Stricter, Not LooserCorrection

This is the detail that trips up almost everyone building or configuring a semantic cache for the first time, because the name score-threshold invites exactly the wrong intuition. It sounds like a similarity score — the kind where higher means "more similar," the way a percentage match or a relevance score usually works. It is the opposite: score-threshold is a distance value, where smaller numbers mean the query has to be closer — more similar — to produce a cache hit.

score-threshold value	What it means	Practical effect
0.01 – 0.05	Aggressive — only very close paraphrases match	High-confidence hits; safe default for most production workloads
0.05 – 0.20	Conservative — looser matching, wider net	More hits, but rising risk of matching genuinely different questions
Above 0.20	Explicitly flagged by Microsoft as risky	"May lead to cache mismatch" — serving a wrong answer with false confidence

Microsoft's own APIM policy documentation states the recommended starting point directly: begin with a low value such as 0.05, and adjust from there to balance the ratio of cache hits to misses. A score threshold above 0.2 may lead to cache mismatches. If your intuition says "I want a stricter cache, I should raise the threshold" — resist it. Raising the number loosens the match; lowering it tightens the match.

Sanity-check your mental model with a concrete pair of near-identical vectors

Two embeddings of genuinely paraphrased sentences — "What's your return policy?" and "How do returns work?" — will typically land at a cosine distance in the very low range, often under 0.05. Two embeddings of superficially similar but semantically different questions — "What's the weather in Paris?" and "What's the weather in Berlin?" — sit much further apart despite sharing most of their words, because the model has correctly separated the two cities as distinct entities. A well-chosen low threshold catches the first pair as a hit and correctly rejects the second as a miss. A threshold set too high, chasing "more matches," starts blurring exactly this kind of distinction.

Architectural Topology: Failing vs Remediated

Layer	Failing configuration (current)	Remediated configuration (fix)
Cache tier	Basic/Standard/Premium — no vector search capability at all	Enterprise or Enterprise Flash, with Enterprise clustering policy
Cache mechanism	Exact-match key-value (if any) — misses every paraphrase	Vector similarity search via RediSearch (HNSW index)
Matching logic	None, or naive string comparison	Cosine distance against a tuned score-threshold
Threshold intuition	Assumed "higher = stricter" (backwards)	Correctly understood as distance — lower = stricter, verified against real query pairs
Embeddings backend	Reuses the completions deployment, or missing entirely	Dedicated, separate embeddings deployment
Cross-user isolation	Global cache, no partitioning — risk of leaking one user's cached answer to another	vary-by partitioning on subscription/user identity
Wrong-answer safeguard	None — any hit above threshold returned unconditionally	Threshold tuned conservatively, monitored, with explicit override paths for sensitive queries
Build vs buy	Not evaluated — assumed custom build is the only option	APIM's built-in policy considered as the lower-effort default

05Fix 1 — Provision the Right Tier and Enable RediSearchSetup

Everything downstream depends on getting this step right. Provision an Enterprise-tier cache with the Enterprise clustering policy, and enable the RediSearch module explicitly.

Azure CLI — create an Enterprise-tier cache with RediSearch enabled# Enterprise tier uses a DIFFERENT CLI command group than Basic/Standard/Premium. az redisenterprise create \ --name aoai-semantic-cache-prod \ --resource-group rg-ai-prod \ --location eastus \ --sku Enterprise_E10 \ --cluster-policy EnterpriseCluster # Create the database within the cluster, WITH the RediSearch module enabled. # This is the step that's easy to skip - RediSearch is not on by default. az redisenterprise database create \ --cluster-name aoai-semantic-cache-prod \ --resource-group rg-ai-prod \ --modules name=RediSearch \ --client-protocol Encrypted \ --port 10000

Bicep — the same, declarativelyresource redisEnterprise 'Microsoft.Cache/redisEnterprise@2024-09-01-preview' = { name: 'aoai-semantic-cache-prod' location: location sku: { name: 'Enterprise_E10' } properties: { clusteringPolicy: 'EnterpriseCluster' // REQUIRED for RediSearch - not OSSCluster minimumTlsVersion: '1.2' } } resource redisDatabase 'Microsoft.Cache/redisEnterprise/databases@2024-09-01-preview' = { parent: redisEnterprise name: 'default' properties: { clientProtocol: 'Encrypted' port: 10000 modules: [ { name: 'RediSearch' } // the module semantic caching depends on ] } }

Confirm the module actually loaded before building anything on top of it

Don't assume the module enabled successfully — verify it. Connect with a Redis client and run MODULE LIST; the response should include search among the loaded modules. If it doesn't appear, the database wasn't created with the module correctly specified, and no amount of correct application code will make vector search work against it — the fix is at the resource level, not the client code.

06Fix 2 — Build the Custom Semantic Cache GatewayThe Fix

This is the core implementation the brief describes: a gateway layer that embeds every incoming query, checks Redis for a semantically close prior answer, and only calls Azure OpenAI on a genuine miss.

Python — create the vector index once, at setup timeimport redis from redis.commands.search.field import TextField, VectorField from redis.commands.search.indexDefinition import IndexDefinition, IndexType r = redis.Redis( host="aoai-semantic-cache-prod.eastus.redisenterprise.cache.azure.net", port=10000, ssl=True, password=REDIS_ACCESS_KEY, # or Entra ID token - see the alert below ) EMBEDDING_DIM = 1536 # match your embeddings model's output dimension schema = ( TextField("query_text"), TextField("response_text"), VectorField( "query_vector", "HNSW", # the algorithm Redis uses for fast approximate search { "TYPE": "FLOAT32", "DIM": EMBEDDING_DIM, "DISTANCE_METRIC": "COSINE", }, ), ) r.ft("idx:semantic_cache").create_index( schema, definition=IndexDefinition(prefix=["cache:"], index_type=IndexType.HASH), )

Python — the gateway: embed, search, decide, call-or-returnimport numpy as np from redis.commands.search.query import Query from openai import AzureOpenAI from azure.identity import DefaultAzureCredential, get_bearer_token_provider credential = DefaultAzureCredential() token_provider = get_bearer_token_provider( credential, "https://cognitiveservices.azure.com/.default" ) aoai_client = AzureOpenAI( azure_ad_token_provider=token_provider, api_version="2024-10-21", azure_endpoint="https://aoai-prod-eastus.openai.azure.com/", ) SCORE_THRESHOLD = 0.05 # DISTANCE - lower = stricter. See Section 3. def embed(text: str) -> np.ndarray: # A SEPARATE, cheap embeddings deployment - not the completions model. resp = aoai_client.embeddings.create( model="text-embedding-3-small-cache", # dedicated deployment input=text, ) return np.array(resp.data[0].embedding, dtype=np.float32) def semantic_cache_lookup(query_vector: np.ndarray) -> dict | None: vec_bytes = query_vector.tobytes() q = ( Query(f"*=>[KNN 1 @query_vector $vec AS distance]") .sort_by("distance") .return_fields("query_text", "response_text", "distance") .dialect(2) ) results = r.ft("idx:semantic_cache").search(q, query_params={"vec": vec_bytes}) if not results.docs: return None top = results.docs[0] distance = float(top.distance) if distance <= SCORE_THRESHOLD: # LOWER distance = closer match return {"response": top.response_text, "distance": distance, "matched_query": top.query_text} return None def semantic_cache_store(query_text: str, query_vector: np.ndarray, response_text: str): key = f"cache:{hash(query_text)}" r.hset(key, mapping={ "query_text": query_text, "response_text": response_text, "query_vector": query_vector.tobytes(), }) r.expire(key, 86400) # TTL - see Fix 4 for why cached answers must expire async def gateway_handle(user_query: str, deployment: str) -> str: query_vector = embed(user_query) cached = semantic_cache_lookup(query_vector) if cached: return cached["response"] # HIT - zero LLM tokens, sub-200ms # MISS - call the LLM, then cache the result for next time. completion = aoai_client.chat.completions.create( model=deployment, messages=[{"role": "user", "content": user_query}], ) response_text = completion.choices[0].message.content semantic_cache_store(user_query, query_vector, response_text) return response_text

Prefer Entra ID authentication over an access key for the Redis connection too

The example above uses a Redis access key for brevity, but the same "no standing secret" principle from managed-identity migrations applies here: Azure Cache for Redis Enterprise supports Microsoft Entra ID authentication as an alternative to access keys. For a production gateway, authenticate the Redis connection the same way you'd authenticate to Azure OpenAI — via a managed identity — rather than embedding a long-lived Redis key in configuration.

07Fix 3 — Or Use APIM's Built-In Semantic Caching PoliciesLower-Effort Alternative

Before committing to the custom gateway in Section 6, know that Microsoft ships an officially supported, lower-effort alternative if Azure API Management already sits in front of your Azure OpenAI endpoints: the azure-openai-semantic-cache-lookup and azure-openai-semantic-cache-store policies (with generic llm-semantic-cache-* equivalents for non-Azure or AI Model Inference API backends) implement essentially the same pattern as configuration, not code.

APIM policy XML — semantic caching in ~10 lines, no custom gateway code<policies> <inbound> <base /> <azure-openai-semantic-cache-lookup score-threshold="0.05" embeddings-backend-id="embeddings-backend" embeddings-backend-auth="system-assigned" ignore-system-messages="true" max-message-count="10"> <vary-by>@(context.Subscription.Id)</vary-by> <!-- prevents cross-user leakage --> </azure-openai-semantic-cache-lookup> <rate-limit calls="10" renewal-period="60" /> <!-- protects backend if cache is down --> </inbound> <outbound> <azure-openai-semantic-cache-store duration="60" /> <base /> </outbound> </policies>

Aspect	Custom gateway (Section 6)	APIM built-in policy
Effort	Full application code, index management, error handling	Configuration only — no custom code to maintain
Flexibility	Full control — custom key strategy, multi-model routing, bespoke logic	Limited to what the policy exposes (threshold, vary-by, message filtering)
Prerequisite	Enterprise-tier Redis + application hosting	APIM already fronting your Azure OpenAI APIs + Enterprise-tier Redis as external cache
Maintenance	You own the gateway code, upgrades, and bug fixes	Microsoft maintains the policy implementation
Best fit	Non-APIM architectures, or needing logic APIM's policy doesn't support	Any architecture already using APIM as the AI gateway — the default recommendation

The prerequisites are the same either way

The APIM path still requires the same Enterprise-tier Azure Cache for Redis with RediSearch enabled, configured as APIM's external cache resource, plus a dedicated embeddings backend registered separately from your completions backend. The policy doesn't remove the tier requirement from Section 1 — it removes the need to write and maintain the gateway application code that implements the lookup/store logic yourself.

If APIM is already in your architecture, start here — build custom only if you outgrow it

Given that the underlying mechanism and infrastructure requirements are identical, the practical recommendation for most teams already running APIM in front of Azure OpenAI is to start with the built-in policy. It gets you the exact cost and latency benefit the brief describes with a fraction of the engineering investment. Reach for the custom gateway from Section 6 only when you hit something the policy genuinely can't do — custom eviction logic, multi-tenant routing more complex than vary-by supports, or an architecture where APIM isn't in the request path at all.

Figure 2 — Two failure modes a semantic cache must guard against, by design

A semantic cache's core value proposition — matching on meaning rather than exact text — is also its core risk. Two independent failure modes follow directly from that trade-off, and both need deliberate configuration, not just the default happy-path setup.

08Fix 4 — Safeguards: Cross-User Leakage and Wrong-Answer RiskCritical

Microsoft's own policy documentation states the risk plainly: because semantic caching returns responses based on similarity rather than exact match, it can surface responses that are incorrect, outdated, or unsafe for the current request. This isn't a footnote — it's a design constraint that shapes how the cache must be partitioned and tuned.

Cross-user leakage: partition the cache

A global, unpartitioned cache means any user's semantically-similar query can retrieve any other user's previously-cached response — including responses that referenced account-specific, personal, or otherwise sensitive information in their original context. The fix is vary-by: partition the cache by subscription ID, authenticated user ID, or tenant, so a cache hit can only ever return a response that was generated for a matching scope.

APIM — vary-by partitioned per authenticated user, not just subscription<azure-openai-semantic-cache-lookup score-threshold="0.05" embeddings-backend-id="embeddings-backend" embeddings-backend-auth="system-assigned"> <vary-by>@(context.User?.Id ?? context.Subscription.Id)</vary-by> </azure-openai-semantic-cache-lookup>

For a custom gateway, implement the same partitioning by scoping the Redis key prefix or the vector search filter to the authenticated caller's identity, so the search only ever considers cache entries within that scope.

Wrong-answer risk: tune conservatively, monitor continuously

• Start at the documented conservative default (0.05) and only loosen it after measuring real false-positive rates against your specific query distribution — don't guess your way to a looser threshold to chase a higher hit rate.
• Log every cache hit with the matched query, the incoming query, and the distance score, so you can audit for near-misses that shouldn't have matched.
• For high-stakes domains (medical, legal, financial guidance), consider excluding semantic caching entirely, or requiring an even stricter threshold than general-purpose traffic.
• Set a TTL on cached entries (Section 6's example uses 86,400 seconds / 24 hours) so stale answers naturally expire rather than persisting indefinitely as source information changes.

Ignore system messages when computing similarity, or you'll under-match

Both APIM's policy (ignore-system-messages="true") and a custom implementation should exclude system prompts from the similarity computation. System messages are often identical or near-identical across many different user queries within the same application, and including them in the embedded text dilutes the signal — two genuinely different user questions can appear artificially similar if a long, shared system prompt dominates the embedding. Embed only the user-facing conversational content that actually varies per query.

09Anti-Patterns: Caching That Saves Money and Loses TrustTraps

Semantic caching's failure modes are quiet — a wrong cached answer looks exactly like a right one to the user receiving it, which is what makes getting the configuration wrong more dangerous than a system that fails loudly.

Anti-pattern	Why it feels right	Why it isn't
Provision Premium tier "because it's cheaper than Enterprise"	"We don't need the fanciest tier for a cache"	Premium has no vector search at all. This isn't a cost-optimization choice — the whole pattern is structurally impossible on that tier
Raise score-threshold to "loosen" the cache	"Higher number sounds like more permissive"	Backwards. Raising the threshold makes matching stricter... no, raising it makes it LOOSER but riskier — the naming is genuinely counterintuitive either way you phrase the intuition. Verify with real query pairs, don't rely on the name alone
Global cache, no vary-by, "we'll add partitioning later"	"Ship the cost savings first, security later"	Cross-user leakage is a data exposure risk from day one, not a future concern. Partition before any production traffic, not after
Reuse the completions model deployment for embeddings	"One less resource to manage"	Different model classes for different jobs. A dedicated, cheap embeddings deployment keeps the cost/latency math favorable and avoids capacity contention with completion traffic
No TTL on cached entries	"More cache hits forever, why expire anything"	Source information changes. A permanently-cached wrong answer to a since-updated policy question is worse than no cache at all
Skip monitoring cache-hit accuracy after launch	"It passed testing, we're done"	Real production query distributions differ from test data. A threshold tuned on synthetic examples can behave differently at scale — monitor continuously, not just at launch

Validation & Verification: Confirm the Fix

Confirm the cache hits on genuine paraphrases, correctly misses on superficially similar-but-different questions, and doesn't leak across users — in that order, before trusting it in production.

Step 1 — Confirm genuine paraphrases produce hitsPARAPHRASE_PAIRS = [ ("What's your return policy?", "How do returns work?"), ("Can I cancel my subscription?", "How do I stop my subscription?"), ("What are your business hours?", "When are you open?"), ] for original, paraphrase in PARAPHRASE_PAIRS: # Prime the cache with the original, then query the paraphrase. await gateway_handle(original, deployment="gpt-4o") original_vector = embed(paraphrase) result = semantic_cache_lookup(original_vector) assert result is not None, f"FAIL: paraphrase not matched: '{paraphrase}'" print(f"PASS: '{paraphrase}' matched at distance {result['distance']:.4f}") # PASS: every paraphrase pair matches, at a distance well under your threshold.

Step 2 — Confirm the adversarial "looks similar, IS different" test correctly MISSESADVERSARIAL_PAIRS = [ ("What's the weather in Paris?", "What's the weather in Berlin?"), ("How do I reset my password?", "How do I reset my email address?"), ("What's the price of the Basic plan?", "What's the price of the Premium plan?"), ] for original, different_question in ADVERSARIAL_PAIRS: await gateway_handle(original, deployment="gpt-4o") different_vector = embed(different_question) result = semantic_cache_lookup(different_vector) assert result is None, ( f"FAIL - FALSE POSITIVE: '{different_question}' incorrectly matched " f"cached answer for '{original}'. Threshold is too loose." ) print(f"PASS: '{different_question}' correctly missed (no false match)") # FAIL here means your threshold is too loose - this is the test that # catches Risk 2 from Figure 2 BEFORE it reaches production users.

Step 3 — Confirm cross-user partitioning actually isolates cache entries# User A asks a question containing account-specific context. await gateway_handle_for_user( "What's the balance on my account?", user_id="user-a", deployment="gpt-4o" ) # User B asks a near-identical question. Should NOT retrieve User A's answer. result_b = await gateway_handle_for_user( "What is my current account balance?", user_id="user-b", deployment="gpt-4o" ) # PASS: result_b triggered a FRESH LLM call (cache miss), because the # vary-by partition correctly scoped User A's cached entry to User A only. # FAIL: result_b returned User A's cached response - CROSS-USER LEAK. # This is a data exposure incident, not a tuning issue. Fix immediately.

Step 4 — Measure the ACTUAL hit rate and cost savings against real traffic// KQL against your gateway's logs or APIM's diagnostic logs CustomLogs | where TimeGenerated > ago(7d) | summarize total_requests = count(), cache_hits = countif(CacheStatus == "hit"), cache_misses = countif(CacheStatus == "miss") | extend hit_rate = round(100.0 * cache_hits / total_requests, 1) | extend estimated_tokens_saved = cache_hits * avg_completion_tokens_per_request // Compare against the theoretical savings you modeled before building this. // A hit rate far below expectation may mean the threshold is too strict, // or your traffic has less genuine repetition than assumed.

What "fixed" actually means here

Four conditions must hold together. One: genuine paraphrases of the same question reliably produce cache hits, verified against real pairs, not assumed from the threshold value alone. Two: superficially similar but semantically different questions reliably produce misses — the adversarial test, not just the happy-path test. Three: cross-user isolation is verified directly, by attempting to retrieve one user's cached answer as a different user and confirming it fails. Four: the measured hit rate and cost savings against real production traffic are tracked and compare favorably to the pre-build cost/benefit estimate. Miss any of the four and you either haven't captured the savings the brief promises, or you've built a system that occasionally, silently, confidently gives someone the wrong answer.

Key Takeaways

→RediSearch — and therefore semantic caching — requires Enterprise or Enterprise Flash tier. Basic, Standard, and Premium have no vector search capability at all. Confirm this before provisioning anything.

→score-threshold is a distance, not a similarity score. Lower values mean stricter matching. The name invites the opposite, backwards intuition — verify your mental model against real query pairs.

→Start with APIM's built-in policy if APIM is already in your path. Same infrastructure requirements, far less code to write and maintain than a custom gateway.

→Partition the cache by user or subscription from day one. An unpartitioned semantic cache is a cross-user data leakage risk, not a future hardening task.

→Test the adversarial case, not just the happy path. "Looks similar, means something different" pairs are exactly what a too-loose threshold gets wrong — and gets wrong confidently.

→Use a dedicated embeddings deployment, and exclude system messages from similarity. Both keep the cost math favorable and the matching signal clean.

→TTL and monitor — semantic caching is probabilistic by design. Microsoft's own docs say it can surface incorrect or outdated responses. Engineer around that constraint, don't hope around it.

Frequently Asked Questions

Can I build semantic caching on Azure Cache for Redis Standard or Premium?

No. Semantic caching depends on vector similarity search, which in Redis comes from the RediSearch module, and Microsoft's documentation is explicit that RediSearch — including its vector search capabilities — is available only on the Enterprise and Enterprise Flash tiers of Azure Cache for Redis. Basic, Standard, and Premium tiers have no vector index type, no HNSW algorithm, and no mechanism to perform a "find the nearest previously-cached question" query, regardless of how the application code is written. If you have an existing Standard or Premium cache for ordinary caching duties, semantic caching requires a separate Enterprise-tier deployment — it cannot be added to the existing instance via configuration alone. When provisioning the Enterprise tier specifically for this purpose, also set the clustering policy to Enterprise (not OSS Cluster), since RediSearch requires that specific cluster policy.

Does a lower or higher score-threshold make the semantic cache match more aggressively?

A lower value makes matching stricter — the query has to be closer to a cached entry to count as a hit. This is because score-threshold represents a distance metric (based on cosine distance between vector embeddings), not a similarity percentage, and the naming can mislead engineers used to "higher score = better match" conventions from other systems. Microsoft's own policy documentation recommends starting at a low value such as 0.05, and explicitly warns that a threshold above 0.2 may lead to cache mismatches — meaning the cache starts returning answers to genuinely different questions because they merely resemble a cached one in vector space. Before tuning this value in either direction, test it against real paraphrase pairs (should match) and real "similar wording, different meaning" pairs (should not match) from your own traffic, rather than relying on intuition about what the number means.

Should I build a custom Redis gateway or use Azure API Management's built-in semantic caching?

If Azure API Management is already fronting your Azure OpenAI APIs, start with its built-in azure-openai-semantic-cache-lookup and azure-openai-semantic-cache-store policies (or the generic llm-semantic-cache-* variants for other backends). They implement the same underlying pattern — embed, vector search, threshold check — as declarative policy configuration rather than application code you have to write and maintain, and they share the exact same infrastructure prerequisite: an Enterprise-tier Azure Cache for Redis with RediSearch enabled, plus a dedicated embeddings backend. A custom gateway is worth building when you need logic the built-in policy doesn't expose — custom eviction strategies, more complex multi-tenant routing than the vary-by attribute supports, or an architecture where APIM simply isn't in the request path. For most teams already using APIM as their AI gateway, the built-in policy delivers the same cost and latency benefit with substantially less engineering investment.

How do I prevent a semantic cache from leaking one user's answer to another user?

Partition the cache by an identity that scopes each user or tenant separately, rather than running a single global cache shared across all callers. In Azure API Management's built-in policy, this is the vary-by element — commonly set to the caller's subscription ID or authenticated user ID — which ensures a cache lookup only ever considers entries stored under a matching scope. In a custom gateway, implement equivalent partitioning by scoping the Redis key prefix or the vector search filter to the authenticated caller's identity before performing the similarity search. This isn't an optional hardening step to add later: an unpartitioned semantic cache is a genuine data exposure risk from the moment it serves its first production request, since any cached response — potentially containing account-specific or otherwise sensitive context from the original conversation — becomes retrievable by any other user who happens to ask a semantically similar question.

Related FAVRITE Articles

• The Shared Tenant Noise Performance Drop: Diagnosing Noisy-Neighbor Latency
• Splitting the Bill: Isolating Semantic Ranker Costs from Agentic Retrieval Plans
• The PTU Math Trap: When to Pivot from Pay-As-You-Go to Provisioned Throughput
• Strict Isolation: Document-Level Security in Azure AI Search for Multi-Tenant Apps