Advanced AKS troubleshooting: Workload Identity federation errors, Azure CNI IP exhaustion, stuck upgrades, autoscaler failures, and two urgent 2025/2026 support deadlines.
AKS Advanced Troubleshooting Guide: Workload Identity, CNI IP Exhaustion, and Stuck Upgrades
CrashLoopBackOff and Pending pods are the failure modes every AKS tutorial covers. The problems that actually stall a mature, production cluster live one layer deeper — a federated identity token that fails validation on a single mismatched character, a subnet that runs out of IP addresses specifically during the upgrade meant to keep the cluster healthy, and a support deadline for pod identity that has already passed.
# Workload Identity federation failure (pod logs / Entra ID sign-in logs):
AADSTS70021: No matching federated identity record found for
presented assertion. Please check your federated identity
credential Subject, Audience and Issuer against the presented
assertion.
# Cluster autoscaler / node pool scale-out failure during an upgrade:
Error: failed to provision node - insufficient IP addresses
available in subnet - scale operation could not complete
# Neither error originates from a broken application. Both trace to
# infrastructure-layer configuration: an OIDC issuer URL mismatch in
# the first case, subnet IP exhaustion triggered by upgrade-time
# node surge in the second.Scope of this guide: advanced AKS failure modes beyond the standard CrashLoopBackOff/Pending/OOMKilled set — Workload Identity federation, Azure CNI IP exhaustion, and two dated deadlines (AAD Pod Identity's end of support, Azure Linux 2.0 node image removal) that are close enough to the current date to require action, not distant housekeeping.
A cluster that's cleared the usual troubleshooting checklist — pods aren't crash-looping, nothing's stuck Pending, nodes report Ready — can still fail in ways that only show up once real workloads are calling Azure APIs, real subnets are under real IP pressure, and real upgrades are running against production traffic. This guide deliberately skips the beginner-to-intermediate territory (CrashLoopBackOff, ImagePullBackOff, basic Pending pods) covered elsewhere and goes one layer deeper: the exact federation mechanics behind Workload Identity failures, the specific subnet math behind Azure CNI IP exhaustion, and a genuinely dangerous interaction between the two that surfaces specifically during node pool upgrades.
Worth being explicit about scope, since AKS troubleshooting content spans a wide range of maturity levels and this guide deliberately sits at the more advanced end.
| Covered elsewhere (not repeated here) | Covered in this guide |
|---|---|
| CrashLoopBackOff, ImagePullBackOff, OOMKilled | Workload Identity federation failures with exact error codes |
| Basic Pending pods (resource requests, taints) | Azure CNI IP exhaustion, including the upgrade-surge trap |
| Node NotReady, DNS resolution failures | Dated deadlines affecting node pool scalability right now |
| Auto-upgrade not triggering at all | Upgrades that DO trigger but stall mid-process on IP exhaustion |
Azure Workload Identity is now the only supported approach for pods to access Azure resources without stored credentials — its specific failure modes have specific, identifiable error codes worth knowing by sight.
| Error code | Meaning |
|---|---|
| AADSTS70021 | No matching federated identity record — the subject or issuer in the token doesn't match any registered federated credential |
| AADSTS70022 | Token is before nbf claim — clock skew between the cluster and Entra ID exceeds tolerance |
| AADSTS700016 | Application not found — the client ID is wrong, or belongs to a different tenant entirely |
Of these three, AADSTS70021 accounts for the large majority of real Workload Identity failures, and Section 3 covers its specific, precise root cause and fix in detail. The other two are worth ruling out quickly — AADSTS70022 points at cluster or node clock drift, and AADSTS700016 points at a wrong or misconfigured client ID — but neither is nearly as common as the issuer URL mismatch behind AADSTS70021.
The federated credential's registered issuer URL has to match the token's iss claim character-for-character — this single requirement is behind the large majority of real AADSTS70021 failures.
| Common cause of mismatch | Why it happens |
|---|---|
| Manually typed or copy-pasted issuer URL | A single missing or extra trailing slash fails the exact-match comparison, even though the URL looks identical to a human reader |
| Federated credential created before a cluster rebuild | The OIDC issuer URL changes if the cluster's OIDC issuer is disabled and re-enabled, or under certain cluster recreation scenarios — a stale credential references the old URL |
| Credential created against the wrong cluster | In a multi-cluster environment, a federated credential registered against a different AKS cluster's issuer will never match |
The single highest-leverage habit for avoiding this failure mode entirely: retrieve the issuer URL with az aks show --query oidcIssuerProfile.issuerUrl at the moment of creating or updating a federated credential, every time, rather than reusing a value from documentation, a previous deployment script, or manual memory. This one habit eliminates the most common root cause of Workload Identity failures before it ever has a chance to occur.
Restating Figure 2's core distinction with the diagnostic detail behind it.
| Networking model | Current status |
|---|---|
| Azure CNI Overlay | Recommended for most new clusters — structurally avoids VNet subnet exhaustion |
| Traditional Azure CNI (Node Subnet) | Still supported, legacy — requires careful, ongoing IP planning to avoid exhaustion |
| kubenet | Deprecated, being removed March 2028 — migrate to CNI Overlay before that date |
On traditional Azure CNI, IP consumption isn't simply "one IP per node" — it's one IP per node PLUS a reserved allocation for the maximum pods per node configured on that node pool, meaning actual subnet consumption can be significantly higher than node count alone would suggest. Checking the subnet's actual available address count against the cluster's configured max-pods-per-node setting, not just current node count, gives a more accurate picture of genuine remaining headroom.
This is a genuinely easy-to-miss interaction: a subnet that comfortably supports a cluster's steady-state node count can still run out of IP addresses specifically during an upgrade or node image update.
| Mechanism | Detail |
|---|---|
| Why upgrades need extra nodes temporarily | AKS scales the node pool out during an upgrade or node image update to maintain availability while nodes are drained and replaced — this is intentional, correct behavior, not a bug |
| Why this can fail on a tightly-sized subnet | Each surge node requires its own pre-allocated pod IP block, exactly like a steady-state node — a subnet sized only for current node count, with no margin, can run out of addresses during exactly this temporary scale-out |
| What this failure looks like | The upgrade itself stalls or fails, or new nodes fail to provision, with insufficient-IP-address errors in the relevant Azure activity log or node pool operation status |
The correct planning approach accounts explicitly for this temporary surge — a subnet sized with zero margin above current steady-state node count is a real, specific risk for this exact failure mode the first time a meaningful upgrade or node image update runs. Microsoft's own guidance recommends allocating a subnet that accommodates future growth specifically because resizing a subnet later is complex and disruptive — a /23 or larger is a reasonable starting point for production workloads, sized with this upgrade-time surge explicitly in mind, not just current node count.
Pods stuck Pending with the autoscaler apparently doing nothing is a distinct, deeper problem than ordinary scheduling-constraint Pending pods — the autoscaler's own logs carry a specific, named reason for every decision it makes, or declines to make.
| Autoscaler log reason | Actual cause |
|---|---|
| MaxNodeGroupSizeReached | The node pool's configured maximum node count has been hit — a scaling ceiling, not a failure, but worth confirming it's set correctly |
| Quota or SKU availability messages | The subscription's regional vCPU quota for the configured VM SKU is exhausted, or that SKU has no current capacity in the target availability zone |
| No scale-up attempt logged at all | Worth checking for a scale-up-disabled annotation on the node pool, or a taint/toleration mismatch preventing pods from being schedulable on any node the autoscaler could add |
The distinction matters for where to look next: if the autoscaler's logs show no scale-up attempt whatsoever, the issue is upstream of the autoscaler — a pod that's fundamentally unschedulable regardless of node count (an impossible resource request, a node selector matching nothing). If the logs show an attempted scale-up that failed, the cause is almost always quota, SKU capacity, or the IP exhaustion pattern from Sections 4-5 — genuinely different diagnostic paths worth distinguishing early rather than guessing.
Two specific, dated facts worth confirming don't apply to the current environment — both are close enough to now that "someday" isn't a safe response.
| Deadline | What happens |
|---|---|
| Sept 2025 (passed) | AAD Pod Identity's official end of support — the AKS Pod Identity managed add-on stopped receiving patches and security updates; the underlying open-source project is archived |
| Nov 30, 2025 (passed) | Azure Linux 2.0 node images stopped receiving security updates, frozen at their final release |
| Mar 31, 2026 | Azure Linux 2.0 node images are removed entirely — node pools still on this image can no longer scale at all past this date |
Given the current date, the March 2026 removal deadline for Azure Linux 2.0 node images has already passed or is immediately upon any environment still running it — check node pool OS image versions directly rather than assuming this doesn't apply. Migration requires either upgrading the node pool to a supported Kubernetes version that pulls a current node image, or explicitly migrating to osSku AzureLinux3.
AAD Pod Identity's end of support means no further patches or security updates — running it in production now means running a fully unsupported identity mechanism with no path to fixes if a vulnerability is discovered. Given the deadline has already passed, this migration belongs at the top of the backlog for any cluster still depending on it, not queued behind other work.
-
For a Workload Identity failure, capture the exact AADSTS error code first
Check pod logs and Entra ID sign-in logs for the specific code — AADSTS70021, 70022, or 700016 point at genuinely different root causes, per Section 2.
-
For AADSTS70021 specifically, query the cluster's real issuer URL directly
Run az aks show --query oidcIssuerProfile.issuerUrl and compare it character-for-character against the federated credential's registered issuer.
-
Confirm the federated credential's subject matches the Kubernetes service account exactly
Namespace and service account name both need to match precisely what the credential specifies.
-
For a suspected CNI IP exhaustion issue, confirm the current networking model
Run the az aks show command from Section 4 to confirm whether the cluster is on CNI Overlay, traditional CNI, or kubenet.
-
Check actual subnet address consumption against configured max-pods-per-node, not just node count
On traditional CNI specifically, this gives a more accurate picture of real remaining headroom than node count alone.
-
If the failure occurred specifically during an upgrade or node image update, check for the surge-related exhaustion pattern from Section 5
Confirm whether the subnet had margin above steady-state node count sufficient to accommodate the temporary scale-out.
-
Check node pool OS image versions against the dated deadlines from Section 6
Confirm no node pool is still running Azure Linux 2.0, and confirm no workload still depends on AAD Pod Identity.
-
Document the confirmed root cause before applying a fix
Given how easily an issuer URL mismatch or subnet sizing gap can recur, record the specific cause found — not just that "it's fixed now" — to prevent the same failure resurfacing after the next credential rotation or cluster change.
| Anti-pattern | Why it feels right | Why it isn't |
|---|---|---|
| Manually typing or reusing a previously-documented OIDC issuer URL | "It's the same cluster, should be the same URL" | The issuer URL can change under certain cluster operations, and manual entry risks a trailing-slash mismatch — always query it fresh |
| Assuming node count alone determines subnet IP headroom on traditional CNI | "X nodes should need roughly X IPs" | Each node reserves IPs for its configured max pods, not just itself — actual consumption is meaningfully higher than raw node count suggests |
| Sizing a subnet for exactly current steady-state node count | "That's what the cluster actually needs" | Upgrades and node image updates temporarily surge node count — zero margin above steady-state is a real risk for mid-upgrade exhaustion |
| Continuing to run AAD Pod Identity because it's "still working" | "If it's not broken, don't touch it" | Support ended in September 2025 — it's running unpatched and unsupported, a real security exposure, not a stable legacy system |
| Ignoring node pool OS image version as a routine detail | "The nodes are running fine" | Azure Linux 2.0 node images are being removed March 2026 — node pools on this image lose the ability to scale at all past that date |
| Treating a resolved Workload Identity error as fully fixed without confirming the specific root cause | "It's working now, move on" | An issuer URL mismatch or credential misconfiguration can recur after any credential rotation or cluster change if the actual cause wasn't documented |
Key Takeaways
Frequently Asked Questions
Related FAVORZENITH Articles
- Azure Kubernetes Service (AKS) Troubleshooting Guide: Real Solutions to Common Problems
- Understand Azure RBAC Roles and How to Assign Them to Users
- Purging the Keys: Migrating Azure OpenAI Applications to Managed Identities and RBAC
- One CI/CD Pipeline for Dev, Test, and Production: A Practical Azure DevOps Design