Learn how to use Azure Chaos Studio to simulate data center outages, test Azure OpenAI failover, and validate AI app resiliency using KQL and CLI workflows
Testing AI Resiliency: Using Azure Chaos Studio to
Simulate Data Center Outages on Your LLM
Every multi-region Azure OpenAI architecture diagram has a failover arrow drawn on it. Almost none of them have ever actually been triggered. The arrow is a hypothesis, confirmed only by a real outage — unless you deliberately cause a controlled one first, on your own schedule, with a rollback plan, instead of finding out during an incident that the failover you designed never quite worked the way the diagram promised.
# The gap this article closes — a real architecture review finding:
Design doc, page 4: "In the event of a regional outage, Azure Front
Door automatically routes traffic to the secondary Azure OpenAI
deployment in West Europe, with an expected failover time under
60 seconds."
Verification performed to support this claim: NONE.
Last time this path was actually exercised: NEVER.
# What actually happens when someone finally tests it, six months
# after the design doc was approved:
$ az rest --method post \
--url ".../providers/Microsoft.Chaos/experiments/zone-down-eastus/start?api-version=2023-11-01"
# Zone 1 of East US goes down as designed. But:
- The Front Door health probe interval was left at its 30s default,
tuned for a DIFFERENT service, never revisited for this one.
- The APIM retry policy retries against the SAME unhealthy region
three times before falling through - adding 90s of latency nobody
budgeted for.
- The secondary region's Azure OpenAI deployment has a DIFFERENT
model version than primary - responses subtly change shape mid-
failover, breaking a downstream JSON parser that assumed one schema.
Actual failover time observed: 4 minutes 40 seconds
Design doc's claimed failover time: under 60 seconds
Gap between assumption and reality: 4x, discovered in a test window,
not during a real customer-facing
regional outage.Symptom: A documented multi-region failover strategy for Azure OpenAI has never been exercised end-to-end, so its actual recovery time, health-probe behaviour, and model-version consistency across regions are assumptions rather than measurements. Failure point: Resiliency was designed once and never validated again — no regular, automated, controlled fault injection confirms the failover path still works as configuration, retry policies, and deployments drift over time. Default platform behaviour: Azure does not test your failover for you. A multi-region architecture with untested failover is, functionally, a single-region architecture with an unverified backup plan.
A failover architecture that has never failed over is a diagram, not a system property. Azure Chaos Studio exists to close exactly this gap — deliberately, safely, and on a schedule you control, rather than during an actual outage where the stakes, the pressure, and the blast radius are all real. But the brief's framing of a single az chaos setup command doesn't match how Chaos Studio's CLI workflow actually works, and "Zone Down" is a more specific test than "simulate a data center outage" makes it sound. Both of those gaps matter, because getting them wrong produces a chaos experiment that runs successfully, produces a green checkmark, and proves almost nothing about whether your Azure OpenAI application would actually survive the failure you meant to test. This article builds the real thing: the actual CLI mechanics, the precise fault that matches "zone" versus the fault that matches "region," and the network-level test that's necessary because Azure OpenAI itself can't be directly targeted.
Every official Microsoft tutorial for Azure Chaos Studio's CLI workflow — service-direct faults, agent-based faults, dynamic zone targeting, AKS Chaos Mesh faults — uses the same underlying mechanism, and it isn't a dedicated az chaos command tree with a setup verb. It's az rest, calling the Microsoft.Chaos resource provider's REST API directly, in three distinct steps that Chaos Studio treats as separate resources.
| Step | What it does | Actual mechanism |
|---|---|---|
| 1. Create a target | Registers a resource (VMSS, AKS cluster, Cosmos DB, etc.) as something Chaos Studio is allowed to touch | az rest --method put against .../providers/Microsoft.Chaos/targets/{targetType} |
| 2. Enable capabilities | Specifies which faults are permitted against that target — shutdown, CPU pressure, network delay, etc. | az rest --method put against .../targets/{targetType}/capabilities/{capability} |
| 3. Create & run an experiment | Defines the sequence of faults, targets, and durations; creates a system-assigned managed identity; starts execution | az rest --method put for the experiment resource, then az rest --method post to .../experiments/{name}/start |
There is no single command that does all three at once — the brief's az chaos setup framing describes an intent (get chaos testing running quickly) rather than a real command that exists in the documented, GA CLI workflow. Every one of Microsoft's own tutorials — service-direct faults, agent-based faults, dynamic zone targeting — is built from these same three az rest calls, composed differently depending on the resource type and fault.
Microsoft has recently introduced Chaos Studio Workspaces (public preview at time of writing, GA targeted late 2026), which ships pre-built scenario templates — including one literally named Zone Down — that compose the right faults automatically so you don't have to hand-assemble the JSON. It's a genuinely closer match to what the brief describes as a turnkey setup experience. As of this writing, Workspaces is a portal-driven experience (with a drag-and-drop Scenario Designer); the underlying automation for CI/CD pipelines still goes through the same az rest / REST API mechanism this article builds. If you have Workspaces access, use it to explore and validate a scenario interactively — then automate the underlying experiment definition it produces using the CLI pattern in Section 8.
Azure CLI version 2.0.30 or later. The Microsoft.Chaos resource provider registered on your subscription. And — easy to miss — every experiment gets its own system-assigned managed identity, created automatically when the experiment resource is created, which must then be granted an appropriate role (e.g. Virtual Machine Contributor for VMSS shutdown) on every target resource before the experiment can actually run. Skipping this step is the single most common reason a first chaos experiment fails with a permissions error rather than actually injecting the fault.
An Azure region — East US, West Europe — is typically composed of multiple physically separate availability zones, each with independent power, cooling, and networking. "Zone Down" in Chaos Studio's fault library means exactly what it says: one availability zone's compute instances are shut down, while the region's other zones keep serving traffic. This is a real, valuable test — but it is not the same test as "simulate the data center that hosts my primary Azure OpenAI deployment going dark," which is what most people mean when they say "simulate a data center outage."
| What you run | What it actually proves | Does it trigger cross-region failover? |
|---|---|---|
| Zone Down (one zone) | Your compute layer survives losing 1 of N zones within a region | No — the region's other zones absorb the load. Front Door never sees a problem |
| Zone Down (all zones, simultaneously) | Your compute layer, taken down entirely within a region, correctly triggers upstream failover | Closer, but still doesn't test the network path itself being unreachable |
| Network-level regional block (Section 7) | Front Door / APIM behavior when the ENTIRE region — including its Azure OpenAI endpoint — is unreachable | Yes — this is the test that actually exercises your documented failover path |
As of this writing, dynamic targeting by availability zone is supported specifically by the Virtual Machine Scale Sets Shutdown 2.0 fault, and only for VMSS using Uniform orchestration mode (Flexible orchestration uses a different, ARM-based VM shutdown fault instead). If your compute tier calling Azure OpenAI runs on something else — Container Apps, AKS, App Service — the zone-down pattern doesn't apply directly the same way, and you'll want the network-level approach in Section 7 regardless of your compute platform, since it tests the thing that actually matters for AI workloads: whether the AOAI endpoint itself is reachable.
The brief asks for a test that confirms AI apps "dynamically reroute to secondary geographic regions." Read literally, the native single-zone Zone Down scenario cannot prove that — it's testing a smaller, real, but different failure mode. Being precise about this distinction is not pedantry; a team that runs Zone Down, sees green, and concludes their cross-region Azure OpenAI failover works has validated the wrong thing and will find out the hard way during an actual regional incident. Section 7 builds the test that actually answers the brief's question.
Chaos Studio's fault library covers a wide range of Azure resource types — Virtual Machines, VM Scale Sets, AKS, Cosmos DB, Cache for Redis, Event Hubs, Key Vault, Service Bus, network security groups, and more. Azure OpenAI (Cognitive Services accounts) is not among the resource types with a native Chaos Studio fault. There is no Microsoft-CognitiveServices target type you can enable and inject a "make this endpoint unavailable" fault against directly.
This isn't a gap in Chaos Studio's coverage so much as a reflection of what you actually need to test. You don't control Azure OpenAI's own internal availability — that's Microsoft's operational responsibility, backed by the service's own SLA. What you do control, and what actually determines whether your application survives a regional AOAI outage, is everything around it: whether your compute tier notices the primary endpoint is unreachable, whether your retry and circuit-breaker logic behaves correctly, and whether Front Door or APIM actually reroutes to the secondary region's deployment. That's the thing worth chaos-testing — not the AOAI resource itself, but your application's response to it becoming unreachable.
| What you're actually testing | How to simulate it |
|---|---|
| Primary AOAI endpoint unreachable | NSG rule blocking outbound traffic to the primary AOAI endpoint's IP range, or a DNS Outage fault against the resolver path |
| Compute tier hosting your app dies | VMSS Shutdown 2.0 (zone-scoped or full), Container Apps/AKS-appropriate faults |
| Front Door / APIM failover logic | Combine the network block above with active traffic and watch which region actually serves responses |
| Downstream schema compatibility across regions | Not a chaos fault — a deployment-consistency check (Section 9's anti-patterns covers this) |
The NSG fault in Chaos Studio's library lets you inject a temporary deny rule that blocks traffic matching specified filters — and this is the closest native fault to "make this endpoint unreachable from my application's perspective" for a PaaS resource like Azure OpenAI that isn't itself a direct target. Point the block at the primary region's AOAI endpoint's resolved IP range, applied to the NSG governing your compute tier's outbound path, and your application experiences exactly what it would experience during a real regional outage: connection failures to the primary endpoint, with the secondary region still fully reachable.
Architectural Topology: Failing vs Remediated
| Layer | Failing configuration (current) | Remediated configuration (fix) |
|---|---|---|
| Failover validation | Never exercised — a diagram and an assumption | Scheduled, automated chaos drills with measured outcomes |
| Fault scope | Undefined, or confused between "zone" and "region" | Explicit: zone-scoped for cross-zone resilience, network-block for cross-region |
| AOAI-specific testing | Assumed covered by "chaos testing" broadly | Explicitly tested via NSG block on the AOAI endpoint's network path |
| Autoscale interaction | Not accounted for — autoscaler silently replaces killed instances mid-test | Disable Autoscale fault runs in parallel, holding the failure state stable |
| Experiment permissions | Not granted, or granted too broadly (e.g. Contributor on the whole RG) | Scoped role assignment per target resource, per experiment's managed identity |
| Automation | Manual portal clicks, run once, forgotten | CI/CD-triggered, scheduled recurrence, results tracked over time |
| Success criteria | "Nothing crashed" (vague) | Measured RTO against a documented target, zero user-visible 5xx, correct region served |
| Post-drill state | Left in whatever state the fault left it — manual cleanup, if remembered | Automatic experiment stop + rollback verification as part of the pipeline |
Before any fault can run, the resources involved need to be explicitly onboarded to Chaos Studio. For a Zone Down test against the compute tier fronting Azure OpenAI, that means the VMSS (or equivalent) and, if you use autoscale, the autoscale settings resource too.
If your VMSS uses an autoscale policy, killing instances in one zone triggers the autoscaler to provision replacements — often within the same test window. The result looks like a successful recovery, but what you actually measured is "how fast does autoscale replace a VM," not "does traffic correctly reroute to a healthy zone or region while capacity is down." Include the DisableAutoscale fault as a parallel branch in your experiment (Section 6) so the zone genuinely stays down for the full test duration.
The experiment definition is a JSON document with two parallel branches in one step: shut down the VMSS instances in the target zone, and simultaneously disable autoscale so nothing masks the effect. Dynamic targeting by zone is expressed through a filter parameter, unique to the VMSS Shutdown 2.0 fault.
The "zones":["1"] filter value refers to Azure's logical availability zone numbering, which — worth knowing — can map to different physical data centers for different subscriptions in the same region, by design, for load-distribution reasons across Azure's whole customer base. Confirm which logical zone your specific VMSS instances are actually deployed to (via az vmss list-instances and checking the zones property) before assuming "Zone 1" in your experiment corresponds to where you think your primary capacity lives.
This is the test that actually answers the brief's question — the one that forces Front Door's cross-region routing to activate, because from your application's perspective, the entire primary region's Azure OpenAI endpoint has simply stopped responding. It uses the NSG fault, since Azure OpenAI itself can't be a direct Chaos Studio target.
Azure OpenAI's custom-subdomain endpoint resolves to IP addresses that can change — it's a managed PaaS service behind Azure's infrastructure, not a fixed IP you control. Resolve the current IPs immediately before running the experiment (dig +short your-aoai-resource.openai.azure.com or equivalent), and don't assume an IP range captured months ago is still accurate. If your NSG rule blocks the wrong (stale) IP range, the fault runs, reports success, and blocks nothing that matters — a false-positive "pass" that's worse than not testing at all, because it creates unearned confidence.
This experiment only produces useful evidence if your application is actively sending real (or synthetic load-test) requests to Azure OpenAI while the NSG block is active. An idle system with no traffic during the 10-minute window tells you nothing about failover behavior under load — start a load-test script or synthetic traffic generator against your production endpoint a minute or two before triggering the experiment, and keep it running through the full duration plus a recovery buffer afterward.
A chaos experiment run once, manually, by one engineer, six months ago, has almost the same evidentiary value as a design doc's untested claim — it proves the system worked then, under that configuration. Configuration drifts: retry policies get tuned, model deployments change, NSG rules get modified for unrelated reasons. The fix is running this on a schedule, automatically, with results tracked over time.
The value of automating this drill compounds only if results are tracked and compared across runs. Publish each drill's measured failover time to a dashboard or a time-series store alongside your documented RTO target, so a gradual regression — failover creeping from 45 seconds to 90 seconds over several months as configuration drifts — is visible as a trend, not discovered the next time someone happens to glance at a single run's logs.
This is a deliberately destructive test running against production infrastructure. Schedule it during genuinely low-traffic periods, and make sure whoever is on-call knows a chaos drill is about to run — an on-call engineer paged by alerts from a chaos experiment they didn't know was happening is a bad outcome even if the system behaves exactly as designed. The experiment proving resilience and the humans monitoring it having a stressful surprise are not mutually exclusive; avoid both problems with a simple heads-up notification as part of the pipeline.
Chaos experiments are easy to run and easy to run badly — producing a green checkmark that feels like validation while actually testing the wrong thing, or testing nothing at all.
| Anti-pattern | Why it feels right | Why it isn't |
|---|---|---|
| Run Zone Down, call cross-region failover "tested" | "We ran a chaos experiment, it's done" | Single-zone failure never triggers cross-region routing — the region's other zones absorb it. You've tested a different, smaller claim |
| Run the experiment with no active traffic | "Simpler to set up, fewer variables" | An idle system can't demonstrate failover-under-load behavior — retry storms, connection pool exhaustion, and health-probe timing all depend on real traffic patterns |
| Forget to disable autoscale | "One less thing to configure" | You end up measuring how fast the autoscaler replaces killed instances, not whether failover actually engaged |
| Skip the fail-back check | "The failover worked, test complete" | A system stuck permanently on the secondary region after the fault clears is its own incident — verify recovery, not just failure response |
| Assume secondary region has an identical model deployment | "We deployed the same model everywhere" | Model versions, quota, and even API version support can silently drift between regions over time. A chaos test that doesn't check response schema consistency across regions misses this class of bug entirely |
| Run once, treat it as permanently validated | "We already proved this works" | Configuration drifts continuously. An unautomated, unrepeated chaos test has a shelf life shorter than most teams assume |
Validation & Verification: Confirm the Fix
The chaos experiment succeeding (running to completion without erroring) and your resiliency claim being validated are two different things. Confirm the second, explicitly.
Four conditions must hold together. One: the experiment ran successfully at the platform level, confirmed via the execution status API, not assumed. Two: the measured recovery time is compared explicitly against your documented RTO claim, with a real number, not a vague "it seemed fast." Three: response consistency across regions during failover is checked, so an unnoticed model-version or schema drift doesn't quietly break downstream consumers the moment failover engages. Four: fail-back is verified after the fault clears, not just fail-over — a system stuck on the secondary region indefinitely is a second incident hiding behind a "successful" first one. Miss any of the four and the chaos experiment ran, but the resiliency claim it was meant to validate remains unproven.
Key Takeaways
Frequently Asked Questions
Related FAVRITE Articles
- The Shared Tenant Noise Performance Drop: Diagnosing Noisy-Neighbor Latency
- Azure OpenAI to Microsoft Foundry: Fixing Private Endpoint & DNS Failures
- Azure Front Door Wildcard Revalidation Fix
- The PTU Math Trap: When to Pivot from Pay-As-You-Go to Provisioned Throughput