Diagnose Azure OpenAI latency issues, identify noisy-neighbor effects, and improve application performance
How To Diagnose Noisy-Neighbor Latency in Azure OpenAI Standard Deployments
Your p99 tripled overnight. You changed nothing. The status page is green. This is the failure mode nobody wants to admit is real on a shared platform — regional demand from other tenants degrading your latency — and the fix isn't to complain, it's to prove it with the right metric, then route around it.
# Azure Monitor — the metric that reveals it (measured on the Azure OpenAI resource):
Metric: AzureOpenAITimeToResponse split by ModelDeploymentName
aggregation Avg + P99
P50 P95 P99
yesterday 09:00-10:00 340 ms 1,240 ms 1,880 ms
today 09:00-10:00 410 ms 3,910 ms 8,720 ms <-- 4.6x on P99
# Prompt/completion token counts unchanged (this is the sanity check):
Metric: ProcessedPromptTokens median 812 → media 807 (identical)
Metric: GeneratedCompletionTokens median 246 → media 241 (identical)
# So the same-shape requests are now much slower. Occasionally:
HTTP/1.1 429 Too Many Requests
Retry-After: 47 <-- capacity, not your quota
# And the status page says:
Azure OpenAI in [your region]: ✓ Healthy <-- because nothing is DOWN.
It is just slow. For you. Right now.Symptom: First-token latency spikes on unchanged code and unchanged request shapes. Failure point: Client → regional Azure OpenAI standard endpoint → shared capacity pool → a neighboring tenant's traffic surge you cannot see. Default platform behaviour: Standard (PAYG) deployments share capacity across all tenants in a region. Quota does not guarantee capacity — Microsoft's own documentation states this plainly. When a region is hot, your allocation is admitted but not necessarily served promptly.
There is a category of production incident where every dashboard you own says everything is fine and yet your users are watching a spinner. Nothing changed on your side — no deploy, no config drift, no traffic spike. The Azure status page is a wall of green ticks. Your Application Insights availability is 100%. And yet the p99 latency of your chat endpoint has tripled since breakfast, and you have no obvious lever to pull. This is the shape of a noisy-neighbor incident on a shared cloud service, and the reason it goes so often unresolved is not that the problem is mysterious. It is that most teams are looking at the wrong metric, and even when they find the right one, they reach for the wrong tool to route around the damage.
Everyone talks about time to first token, but "TTFT" is not the name of an Azure Monitor metric. Reaching for a metric called TTFT and being unable to find it is the first place a lot of noisy-neighbor investigations stall. Microsoft's own latency documentation is precise about which metric to actually use, and the distinction matters.
| Metric (Azure Monitor) | What it measures | When to use it |
|---|---|---|
| AzureOpenAITimeToResponse | First-token latency for streaming; whole-response time for non-streaming | Default choice. Absolute latency your customers experience |
| AzureOpenAINormalizedTBTInMS | Normalized time between tokens — per-token generation throughput | Judges the steady-state of the stream. Rises when the deployment is under load |
| AzureOpenAINormalizedTTFTInMS | Normalized time to first byte, adjusted for prompt size | Only when comparing first-token efficiency across prompts of different sizes |
| ProcessedPromptTokens | Prompt token count per request | Sanity check — always pair with a latency metric (Section 2) |
| GeneratedCompletionTokens | Completion token count per request | Sanity check — a longer answer is slower and that is not a regression |
| Azure OpenAI Requests | Total API calls | Confirms traffic shape didn't change on your side |
The guidance from Microsoft is direct: for diagnosing absolute latency that customers experience, use AzureOpenAITimeToResponse. Use the normalized TTFT metric only when you specifically need to compare across differently-sized prompts. Most incident investigations do not need normalization — they need the number the user is actually waiting on.
Two configuration details that are easy to skip and turn every dashboard into noise. First: always split by ModelDeploymentName — one hot deployment can drag your averages and hide which one is actually the problem. Second: use P99 aggregation, not Avg. Noisy-neighbor damage lives entirely in the tail. Averaging hides the very requests you are trying to see, and you will convince yourself the problem is imaginary while a tenth of your users are watching an 8-second spinner.
Before you conclude anything, do the one check that separates a real latency regression from a request-shape change on your side. Microsoft's guidance is blunt: latency without token context isn't actionable. A five-second response that generates 2,000 tokens is a very different animal from a five-second response that generates 50. If your team just changed a prompt, added a chain-of-thought step, or bumped max_tokens, your latency has "risen" for the perfectly boring reason that you are asking the model to do more work.
The rule of thumb, stated by Microsoft in their own latency documentation: total end-to-end response time scales with the number of generated tokens, so an increase in it is often fully explained by an increase in output tokens — not by a system performance issue. Always check token counts before you conclude that there's a latency regression.
| Observation | Prompt tokens | Completion tokens | Conclusion |
|---|---|---|---|
| P99 up, tokens up | ↑ | ↑ or ↑↑ | Not a regression. Your requests got bigger |
| P99 up, prompts up only | ↑ | flat | Prefill got heavier. Trim the prompt or check RAG bloat |
| P99 up, completions up only | flat | ↑ | Answers got longer. Cap max_tokens or shorten instructions |
| P99 up, both flat | flat | flat | Real latency regression. Now investigate the region |
This is the sanity check that prevents the embarrassing support ticket where the answer comes back "your median completion tokens increased 60% overnight — this is a prompt problem, not a platform problem." Ninety percent of the time, when a team is certain the platform slowed down, the shape of their traffic changed and nobody looked. Sixty seconds of KQL will save you a day of finger-pointing. Only when both token counts are flat and latency is up do you have a signal that is genuinely about the shared pool.
Once the token counts clear you of self-inflicted regression, there are three remaining suspects. They present differently and each has a different fix, so getting the wrong one is expensive.
| Culprit | Signature | How to confirm | Fix |
|---|---|---|---|
| Your side (still) | TimeToResponse up in one deployment; tokens flat but request rate is up | Check RPM vs quota; check other clients on the same key | Rate-limit at the client; add a queue; scale up quota |
| Shared region | TimeToResponse up across all your deployments in a region; TBT up; other regions on the same subscription are fine | Deploy the same model in a second region and compare in parallel — this is decisive | Failover to another region (Sections 6 & 7) |
| Specific model | Only one model (e.g. a newly-launched or hot model) is affected; other models in the same region are fine | Compare TimeToResponse across models in the same region | Switch to a less-hot model version; or PTU (Section 8) |
You cannot diagnose a regional problem from a single region — every metric you have is coming from the region you suspect. The only test that cannot lie is to send the same requests to the same model in a different region, simultaneously, and compare. If the other region is fast, you have your answer, and you already have the fallback endpoint built. If the other region is also slow, it is more likely to be a model-wide event or a change on your side that a token count didn't catch. This shadow-region setup is not just a diagnostic — it is the first stage of the fix.
Architectural Topology: Failing vs Remediated
| Component | Failing configuration (current) | Remediated configuration (fix) |
|---|---|---|
| Endpoint | Direct client → single-region AOAI resource | Client → Front Door → prioritised multi-region origins |
| Region posture | Single region; no comparison signal exists | ≥ 2 regions with the same model, one hot, others standby |
| Failover control-plane | None — or DNS-based (Traffic Manager) with TTL delays | Front Door priority routing; HTTP probes; seconds to shift |
| Failover trigger | None — you notice when users complain | Health-probe failure OR alert on P99 TimeToResponse |
| Latency metric | Averages; ignored until users complain | P99 AzureOpenAITimeToResponse, split by deployment, alerted |
| Token accounting | Latency looked at alone | Always paired with prompt + completion token counts |
| Retry policy | Retries on 429 hammer the same hot region harder | APIM circuit breaker; retries land on a different origin |
| Business-critical baseline | Standard only — noisy-neighbor exposure by design | PTU floor + PAYG burst — noise cannot touch the SLA path |
You cannot route around damage you cannot see, and by default you cannot see this damage. It hides in averages, gets lost in unsplit metrics, and never fires an alert because nothing is technically broken. Wire the metrics correctly, and the noisy-neighbor signature becomes obvious the moment it starts.
An alert on the average TimeToResponse is functionally useless for this failure mode. Averages absorb spikes, and by the time a noisy-neighbor event moves the mean it has moved the tail so far that a real percentage of your users have already given up. Alert on the maximum or a high percentile, on a short window, and be prepared for a few false starts as you calibrate to your own P99 baseline. The false positives are cheap; the false negatives cost you users.
This is the fix, but it comes with a correction worth stating up-front: for an Azure OpenAI failover, Azure Front Door is the right control-plane, not Azure Traffic Manager. The two services can look interchangeable — both do multi-region routing, both have priority modes, both do health probing — but for a noisy-neighbor incident where the "outage" arrives and departs in minutes, the difference between them is the difference between mitigation and irrelevance.
| Property | Traffic Manager | Azure Front Door |
|---|---|---|
| Layer | DNS (returns an IP; steps out of the way) | HTTP/HTTPS at the edge (anycast, split-TCP) |
| Failover speed | Minutes — bounded by DNS TTL caching on the client side | Seconds — no DNS propagation delay |
| Health probing | US-based probes only; unreliable for global anycast | Edge-wide HTTP probes with configurable intervals |
| Best fit | Non-HTTP protocols; simple failover across any endpoint | HTTP/HTTPS workloads — including every AOAI call |
Microsoft's own best-practices documentation is direct about this choice: use either Azure Front Door or Traffic Manager, not both, and for HTTP workloads that need edge acceleration and fast failover, Front Door is the recommended service. Since every Azure OpenAI call is HTTPS, and since the whole point of this exercise is failing over in seconds rather than after DNS TTLs expire, Front Door is the natural choice.
The topology: two regions, one Front Door, priority routing
The design is deliberately minimal. Deploy the same model into a primary and at least one secondary region. Register both as origins in a single Front Door origin group. Set priority on the primary and lower priority on the secondary. Configure health probes against the OpenAI health endpoint. That is the whole thing.
This is where multi-region AOAI deployments actually go wrong in production. Front Door is happy to route to either region, but your API keys are per-resource and your deployment name paths must be identical on both sides — /openai/deployments/gpt-4o/chat/completions?api-version=... must resolve on both origins. Either standardise deployment names across regions (recommended) or handle key rotation through an APIM layer that injects the correct key per origin (Section 7). Skipping this is how failover works in your test and fails in your incident.
Not every model is available in every region, and quota is allocated per-region-per-subscription. Before you designate a secondary region, verify that the specific model version you use in production is deployable there, and that you have — or can get — the quota you actually need. Two regions serving different model versions during a failover is a subtler outage than one region being slow, because your responses change shape at the same moment.
Front Door handles outage-shaped failure well — an unresponsive origin, a 5xx storm — but noisy-neighbor damage is subtler. Requests still return 200s, just slowly. The health probe may pass while your users are watching a spinner. This is where Azure API Management earns its place in the topology: it can react to signals Front Door does not see, including elevated 429 rates and per-token latency, and it can implement the retry logic Front Door deliberately does not.
APIM supports a first-class circuit breaker on backend definitions — configure it on the backend, don't try to reimplement it in policy. Set a failure-rate threshold and a trip duration; when tripped, APIM stops sending requests to the sick origin altogether for the configured window, so retries land on the healthy secondary immediately rather than hammering the hot region repeatedly and making things worse. Retries without a breaker are indistinguishable from a self-inflicted DDoS.
When Azure OpenAI returns 429, the response includes a Retry-After header (sometimes as x-ms-retry-after-ms in milliseconds). Read it — your backoff is being told to you by the service. A retry-in-milliseconds implementation that ignores this header and uses a fixed exponential ladder will retry too fast, get throttled again, and take longer to recover than if it had simply waited what the platform told it to.
Everything above is mitigation on a shared pool. The only fix that actually removes noisy-neighbor exposure — because it removes the shared pool — is Provisioned Throughput Units (PTU). On a standard (PAYG) deployment, quota only controls admission logic; it does not enforce throughput and it does not shield you from other tenants. On a provisioned deployment, you allocate a fixed amount of model processing capacity to your endpoint, and no other tenant can borrow it.
| Property | Standard (PAYG) | Provisioned (PTU) |
|---|---|---|
| Capacity model | Shared pool across all tenants in region | Reserved for your deployment |
| Quota semantics | Admission control only — no throughput guarantee | Enforced throughput budget |
| Noisy-neighbor exposure | Yes. By design | No. That is the point |
| Latency stability | Variable; tail dominated by regional demand | Predictable; independent of neighbouring tenants |
| Cost shape | Per token consumed | Per hour reserved, whether used or not |
| Best-fit workload | Bursty, cost-sensitive, dev & POC | Business-critical, latency-critical, steady baseline |
The honest recommendation is not to move everything to PTU — that is often financially indefensible. It is a hybrid: PTU-sized to your baseline, with PAYG as the burst layer. Business-critical, latency-critical traffic lands on the reserved capacity where noisy neighbours cannot reach it. Everything else — internal tools, batch jobs, low-priority background enrichment — runs on standard and takes the noise on the chin.
The hybrid only works if you route to PTU intentionally. Sending all traffic there defeats the point and blows the budget; leaving PTU idle while PAYG is throttled defeats it too. Route by workload class at the ingress layer: user-facing chat and voice → PTU; internal automation, batch, and offline evaluation → PAYG. If your PTU is under-utilised outside peak hours, use the same client-side retry logic you built for 429s to spill from PAYG to PTU when the standard endpoint is throttled — the exact opposite of the usual pattern, and one that quietly makes the whole system more resilient.
Because this failure mode is confusing, the wrong instincts are strong. These are the fixes teams reach for that either make things worse or paper over the real problem.
| Anti-pattern | Why it feels right | Why it isn't |
|---|---|---|
| Aggressive client retries | "Just retry the slow call!" | A slow call retried is another slow call in a hot region. Retries without a breaker amplify contention |
| Requesting more quota | "429s stopped, so quota was the issue" | Quota controls admission, not throughput. More quota does not create more shared capacity |
| Traffic Manager priority failover | "It's DNS-simple and cheaper" | DNS TTL caching makes failover minutes-slow. Noisy-neighbor incidents are minutes-long |
| Streaming everything to hide TTFT | "At least users see tokens sooner" | It masks the metric while making the underlying starvation worse. TBT still rises; users still wait for the full answer |
| Reducing max_tokens as the fix | "Shorter answers = lower latency" | Correct answer to a different question. Helps overall TTLT but does nothing for first-token latency under contention |
| Waiting for the status page to admit it | "Microsoft will call it out" | Nothing is down. A shared-pool spike does not become a status incident. Diagnose it yourself |
It bears saying plainly: the Azure status page reports outages, not degradations. A region that is answering every request with a 200 in eight seconds instead of one is not an outage by the platform's definition, and it will not appear there. If your users are having a bad time and the status page is green, that combination is not the platform gaslighting you — it is a statement about the definition of the word "healthy." Trust your metrics.
Validation & Verification: Confirm the Fix
Because the failure is transient and invisible from outside your telemetry, validation requires either an active test or a passive proof that the failover actually engaged the last time the noise happened. Do both.
Three conditions must hold together, and none of them can be substituted for the others. One: you can prove Front Door shifts traffic in seconds when the primary origin is unhealthy, because that is the whole reason for choosing it over Traffic Manager. Two: your AzureOpenAITimeToResponse P99 alert would have fired at the start of the last incident you remember — if it did not, your threshold or your split-by-deployment configuration is wrong and you will miss the next one. Three: the token-count sanity check is a first-class step in your runbook, so you never again escalate a "latency regression" that turns out to be a prompt change. Miss any of those three and the next noisy-neighbor spike will play out exactly like the last one.
Key Takeaways
Frequently Asked Questions
Related FAVRITE Articles
- How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
- Azure OpenAI to Microsoft Foundry: Fixing Private Endpoint & DNS Failures
- The Edge Latency Drop: Offloading Embeddings to Foundry Local 1.2
- Fixing the First-Request Lag: Azure Functions and Container Apps for AI Microservices