Resolve Azure embedding pipeline limitations and improve batch processing efficiency for AI applications

Pipeline FixEmbeddings APIArray LimitsAsync Batching

Fixing the Array Error: Overcoming Batch Limits
in Azure Embedding Pipelines

The number "16" is real, but it's not the number that will break your pipeline today. It's the ghost of an older model and an older API surface. The constraint that actually crashes a modern text-embedding-3-large pipeline is quieter, newer, and measured in tokens summed across an entire request — which means a batch that respects every count-based limit you've ever heard of can still fail, because nobody was adding up the total.

The failure signature this guide resolves

# SIGNATURE A — the legacy error, still real for ada-002 / older API versions:

openai.InvalidRequestError: Too many inputs. The max number of inputs
is 16. We hope to increase the number of inputs per request soon.
Please contact us through an Azure support request for further
questions.
(HTTP 400, param=None, code=None)

# This is the error the brief describes. It is verified, real, and
# STILL happens today - but only on the classic API surface, with
# text-embedding-ada-002. It is NOT the limit for current models.

# ─────────────────────────────────────────────────────────────

# SIGNATURE B — the one that actually crashes a MODERN pipeline,
# using text-embedding-3-large via the current /openai/v1/embeddings
# endpoint, with an array well under the documented cap:

POST /openai/v1/embeddings
{
  "model": "text-embedding-3-large",
  "input": [ /* 340 strings, array length WELL under 2,048 */ ]
}

HTTP/1.1 400 Bad Request
{
  "error": {
    "message": "This model's maximum context length is 300000 tokens
                per request. However, you requested 341982 tokens
                (341982 in your input). Please reduce the size of
                your request.",
    "code": "context_length_exceeded"
  }
}

# Array length: 340   (limit: 2,048  -> PASSES)
# Longest single input: 6,100 tokens  (limit: 8,192 -> PASSES)
# SUM of all 340 inputs' tokens: 341,982   (limit: 300,000 -> FAILS)
#
# Every count-based guard the team had written passed. The pipeline
# still crashed, because nobody was tracking the running TOTAL.

Symptom: Batch embedding requests fail with InvalidRequestError or a 400 context_length_exceeded, sometimes on request #1 of a large backfill, sometimes hours into a job that had been running fine.  Failure point: A client-side batching implementation guards only one dimension of the limit — usually array count — while Azure OpenAI's embeddings endpoint enforces up to three independent limits simultaneously, depending on model and API version.  Default platform behaviour: The service rejects the whole request, not just the offending inputs. One oversized batch, however it's oversized, fails every input in it — including the ones that were perfectly fine.

16 (legacy)

The real, verified array-count cap for text-embedding-ada-002 on the classic API surface. Still enforced today for that model

2,048 (current)

The array-count cap for current models via /openai/v1/embeddings — matching OpenAI's public API, not 16

300,000 tokens

A separate, newer aggregate cap — the SUM of tokens across every input in one request. Fails independently of array length

8,192 tokens

The max length of any single input string. A third, independent dimension a compliant batch must respect

Table of Contents

1. 01The "16-Input" Limit Is Real — And Also Not What Breaks You Now
2. 02Three Independent Limits, Not One
3. 03Why a Naive "Chunk Every N Rows" Batcher Still Crashes
4. 04Architectural Topology: Failing vs Remediated
5. 05Fix 1 — Count Tokens Before You Batch, Not After You Fail
6. 06Fix 2 — The Three-Dimensional Async Batching Queue
7. 07Fix 3 — Retry, Backoff, and Partial-Batch Recovery
8. 08Fix 4 — Respecting TPM Quota Alongside Batch Limits
9. 09Anti-Patterns: Batchers That Look Compliant and Aren't
10. 10Validation & Verification: Confirm the Fix

Somewhere in the history of most teams' Azure OpenAI embedding pipelines is a moment where someone hit the "16 inputs" error, searched for it, found the fix — chunk your array into groups of 16 — and moved on. That fix was correct, for the model and API surface it applied to. What often doesn't happen next is revisiting the assumption when the pipeline is later migrated to a newer embedding model on a newer API version, where the constraint that actually governs the request has quietly changed shape: it's no longer primarily about how many strings are in the array, but about how many tokens, summed across every string in the array, the whole request adds up to. A batcher hardcoded to "16 items per request" is stuck being needlessly slow on the new limits. A batcher that naively raised its chunk size to "2,048 items per request" because that's the new array cap will crash the first time it hits a batch of long documents, because it never accounted for the token sum. The fix that actually holds is one that tracks all three dimensions — array count, per-item length, and aggregate tokens — continuously, as it assembles each batch.

Figure 1 — Three independent limits a compliant batch must respect simultaneously

Array count, per-item token length, and aggregate token sum are three independent constraints, and Azure OpenAI's embeddings endpoint enforces all three. A batching implementation that checks only one — commonly just array count, inherited from the legacy 16-input fix — will pass validation right up until a batch of longer documents crosses the aggregate token cap, at which point the entire request fails, including every input that was individually fine.

01The "16-Input" Limit Is Real — And Also Not What Breaks You NowCorrection

The error message "Too many inputs. The max number of inputs is 16" is a real, documented, verified Azure OpenAI error — it applies specifically to text-embedding-ada-002 (Version 2) accessed through the classic Azure OpenAI API surface. Multiple SDK issues (LangChain's Python and JavaScript clients both had to special-case this) confirm the number and the model it applies to. If your pipeline is still on ada-002 through an older API version, this constraint is current and correct today, and the "batch in groups of 16" fix is the right fix for that specific pipeline.

But most teams reading a brief about "the 16-input limit" today are not on that exact configuration, or are migrating off it, and the newer models and the newer /openai/v1/embeddings endpoint enforce a meaningfully different set of constraints. Assuming "16" is a universal Azure OpenAI embeddings limit — rather than a specific, model-and-version-scoped one — is the single most common way teams either under-batch (wastefully slow) or, worse, mis-port a working "chunk by N" pattern from one model to another and hit a completely different failure mode.

Model / API surface	Max array size	Max tokens per input	Aggregate token cap
text-embedding-ada-002 (legacy API)	16	8,191	Not separately documented at this scope
Current models via /openai/v1/embeddings	2,048	8,192	300,000 (summed across the request)

Also worth correcting: it was never "16 tokens"

Even under the legacy limit, the constraint was 16 inputs — array items — never 16 tokens. A single input string, even under the old limit, could still be up to 8,191 tokens long. Confusing "inputs" with "tokens" leads to batchers that are wildly over-conservative (chunking every 16 tokens' worth of text into a separate array item, producing enormous numbers of tiny, wasteful requests) or, worse, batchers that get the dimension right but the number wrong when porting logic between limit types.

02Three Independent Limits, Not OneConcept

For current-generation models via the modern endpoint, a single embeddings request is governed by three separate constraints, checked independently by the service, and a batch has to satisfy all three simultaneously — satisfying two doesn't help if the third is violated.

Limit	Value	What it measures	Fails when...
Array size	2,048	Count of items in the input array	You send more than 2,048 strings in one call, regardless of their length
Per-input length	8,192 tokens	Token count of any single input string	One item in the array — even alone — exceeds 8,192 tokens
Aggregate request tokens	300,000 tokens	SUM of token counts across every item in the array	The array and every item are individually compliant, but their total exceeds 300,000

The aggregate limit is the one that catches experienced teams off guard, precisely because it's new relative to the older, more commonly-known array-count limit, and because it's easy to satisfy the other two and still violate it. Microsoft's own current documentation states this plainly: requests that exceed the aggregate limit fail with HTTP 400 even when every individual input is under 8,192 tokens and the array length is under 2,048.

Do the arithmetic that shows how easy this is to hit

300,000 tokens divided across a reasonably-sized array isn't as generous as it first sounds. At an average of 1,000 tokens per input — a modest few paragraphs of text, well within normal document-chunk sizes for RAG pipelines — the aggregate cap is reached at just 300 items, far short of the 2,048-item array limit. A batcher that fills arrays up to the array-count ceiling without also tracking the running token sum will, for any reasonably long input text, hit the aggregate cap long before it hits the array-count cap. The array limit and the aggregate limit are not the same constraint wearing two names — for typical document lengths, the aggregate limit binds first.

03Why a Naive "Chunk Every N Rows" Batcher Still CrashesRoot Cause

The most common batching implementation — and the one most likely to have been written to fix the original "16 inputs" error — is a fixed-size chunker: take the array of rows, slice it into groups of N, send each group as one request. It's simple, and for uniformly-sized inputs it can work fine for a long time. It fails the moment the assumption of uniform input length breaks, which for real-world database content — some rows are a sentence, some are a full document — is not a matter of if but when.

The naive pattern — passes review, fails in production on real data# Looks reasonable. Works fine in testing with short, similar-length rows. # Silently accumulates risk as soon as real data includes longer documents. def naive_batches(rows: list[str], batch_size: int = 100): for i in range(0, len(rows), batch_size): yield rows[i:i + batch_size] # batch_size=100 easily clears the 2,048 array-count limit. It says # NOTHING about the aggregate token sum of those 100 rows. If even a # modest fraction of them are long documents, this crashes.

The failure is also intermittent in a way that's genuinely confusing to debug: a batching job processing a database table row-by-row in fixed-size chunks might run cleanly through thousands of short rows, then crash on batch #340 because that particular slice happened to land on a cluster of long documents — support tickets, contract text, long-form articles — that pushed the aggregate sum over 300,000 while every individual row was fine. The error looks random. It isn't; it's a direct function of which 100 rows happened to be adjacent in the source data.

The whole batch fails, including the compliant inputs

This is the detail that makes fixed-size chunking especially costly to get wrong: Azure OpenAI's embeddings endpoint rejects the entire request when any limit is violated — it does not process the compliant 99 inputs and skip the one that pushed things over. A batch job that doesn't handle this gracefully either loses all 100 rows' embeddings for that chunk (if it doesn't retry) or has to re-derive which specific input(s) to split out and retry separately (if it does) — extra logic that a limit-aware batcher (Section 6) avoids needing in the first place, by never assembling a non-compliant batch to begin with.

Architectural Topology: Failing vs Remediated

Layer	Failing configuration (current)	Remediated configuration (fix)
Batching logic	Fixed-size chunking (e.g. every 16 or every 100 rows)	Dynamic, token-aware batching — fills each batch up to all three limits
Limit awareness	Array count only, often hardcoded from the legacy "16" fix	Array count + per-item length + aggregate token sum, all tracked live
Token counting	None — length assumed from row count or character count	Real tokenizer (tiktoken) counts before assembling each batch
Failure mode	Whole batch rejected; compliant rows lost or require manual retry	Batches never assembled non-compliant in the first place
Oversized single input	Unhandled — crashes the whole pipeline run	Detected and routed to truncation/chunking logic before it reaches the API
Concurrency	Sequential, one batch at a time — slow for large backfills	Async queue with bounded concurrency, respecting TPM/RPM quota
Transient failures	No retry, or naive immediate retry (hammers the same limit again)	Exponential backoff honoring Retry-After, partial-batch recovery
Portability across models	Hardcoded limit numbers scattered through the codebase	Limits as configuration, keyed by model/API version

05Fix 1 — Count Tokens Before You Batch, Not After You FailFoundation

Every limit in this article is expressed in tokens, not characters or words, and there is no reliable shortcut from character count to token count — different text (code, non-English languages, technical jargon) tokenizes at meaningfully different rates. The foundation of a compliant batcher is a real tokenizer, run once per input, before any batching decision is made.

Python — tiktoken-based token counting, cached per inputimport tiktoken from functools import lru_cache # cl100k_base is the encoding used by text-embedding-3-large/small and # ada-002. Confirm this matches your specific deployed model. _encoding = tiktoken.get_encoding("cl100k_base") @lru_cache(maxsize=None) def count_tokens(text: str) -> int: return len(_encoding.encode(text)) # Run this ONCE per row as data is pulled from the database, not # repeatedly as batches are assembled and re-assembled.

Precompute and store token counts alongside the source data where possible

If you're processing a large, recurring backfill (the "thousands of database rows" scenario the brief describes), tokenizing every row on every pipeline run is wasted work if the underlying text doesn't change often. Store the computed token count as a column alongside the source text, recompute it only when the text itself changes, and the batching queue can read a number instead of running a tokenizer on every row, every time. For a one-off backfill this optimization matters less; for a pipeline that runs daily against a growing table, it compounds.

Handle the single-oversized-input case explicitly

A row whose own text exceeds 8,192 tokens on its own violates Gate 2 regardless of batching strategy — it can never be sent as-is, batched or not. Decide this policy up front, not as an exception handler reacting to a crash: truncate to the limit (simple, lossy), split into multiple chunks and average or concatenate the resulting embeddings (more faithful, more complex), or route oversized rows to a separate review queue for a human decision. Whichever you choose, the batcher needs to detect this case before attempting to include the row in any batch, not discover it via a 400 response.

06Fix 2 — The Three-Dimensional Async Batching QueueThe Fix

This is the core of the fix: an async batching queue that fills each outgoing request up to — but never past — all three limits simultaneously, then dispatches batches concurrently within a bounded worker pool.

Python — the batch assembler: greedy-fills respecting all three gatesfrom dataclasses import dataclass, field # Limits as configuration, not hardcoded magic numbers scattered # through the codebase. Swap these per model/API version. @dataclass class EmbeddingBatchLimits: max_array_size: int = 2048 # Gate 1. Use 16 for legacy ada-002. max_tokens_per_input: int = 8192 # Gate 2. max_aggregate_tokens: int = 300_000 # Gate 3. THE one most batchers miss. @dataclass class BatchItem: id: str # your row's primary key, for result mapping text: str token_count: int def assemble_batches( items: list[BatchItem], limits: EmbeddingBatchLimits, ) -> list[list[BatchItem]]: """ Greedy-fills batches respecting all three limits simultaneously. Oversized single items (violating Gate 2 alone) are the caller's responsibility to filter out BEFORE calling this - see Section 5. """ batches: list[list[BatchItem]] = [] current: list[BatchItem] = [] current_tokens = 0 for item in items: would_exceed_count = len(current) + 1 > limits.max_array_size would_exceed_aggregate = current_tokens + item.token_count > limits.max_aggregate_tokens if current and (would_exceed_count or would_exceed_aggregate): # Current batch is full on EITHER dimension - close it out, # start a new one with this item. batches.append(current) current = [] current_tokens = 0 current.append(item) current_tokens += item.token_count if current: batches.append(current) return batches

Python — the async dispatcher: bounded concurrency across assembled batchesimport asyncio from openai import AsyncAzureOpenAI from azure.identity.aio import DefaultAzureCredential, get_bearer_token_provider credential = DefaultAzureCredential() token_provider = get_bearer_token_provider( credential, "https://cognitiveservices.azure.com/.default" ) client = AsyncAzureOpenAI( azure_ad_token_provider=token_provider, api_version="2024-10-21", azure_endpoint="https://aoai-prod-eastus.openai.azure.com/", ) # Bound concurrency to respect your deployment's RPM, not just the # per-request limits. Tune this against your actual quota (Section 8). MAX_CONCURRENT_REQUESTS = 8 semaphore = asyncio.Semaphore(MAX_CONCURRENT_REQUESTS) async def embed_batch(batch: list[BatchItem], deployment: str) -> dict[str, list[float]]: async with semaphore: response = await client.embeddings.create( model=deployment, input=[item.text for item in batch], ) # Map results back to source row IDs by POSITION - the API # preserves input order in the response. return { item.id: data.embedding for item, data in zip(batch, response.data) } async def run_pipeline(items: list[BatchItem], limits: EmbeddingBatchLimits, deployment: str): batches = assemble_batches(items, limits) results = await asyncio.gather( *(embed_batch(b, deployment) for b in batches), return_exceptions=True, # don't let one bad batch kill the whole run ) # Section 7 covers what to do with exceptions in this list. return batches, results

The greedy fill is intentionally simple — resist over-optimizing it

The batch assembler above uses a straightforward greedy fill: add items until any limit would be crossed, then close the batch. This isn't the theoretically optimal packing (a bin-packing algorithm could in principle achieve marginally better batch utilization), but it's simple, predictable, easy to verify correct, and — critically for a data pipeline — deterministic given the same input order. The gains from a more sophisticated packing algorithm are marginal against the cost of correctness bugs in a more complex implementation. Optimize this only if profiling shows batch count is genuinely a bottleneck, not by default.

Figure 2 — Retry decision flow: not every failure means the same thing

Different failure types call for different responses. A 429 or 5xx is transient — the same request will likely succeed with backoff. A 400 for exceeding a documented limit means the batch itself was assembled incorrectly and retrying it unchanged will fail identically every time; if the three-gate assembler from Fix 2 is correct, this error should never occur in normal operation, and its appearance is a signal to investigate the assembler's configuration, not to add a retry loop around it.

07Fix 3 — Retry, Backoff, and Partial-Batch RecoveryResilience

Even a correctly-assembled batch can fail for reasons outside the batcher's control — rate limiting, transient service errors. Handle these distinctly from limit-violation errors, which should be treated as defects in the assembler, not transient conditions to retry through.

Python — embed_batch with typed retry logicimport random from openai import RateLimitError, APIStatusError, APIConnectionError async def embed_batch_with_retry( batch: list[BatchItem], deployment: str, max_retries: int = 5, ) -> dict[str, list[float]]: for attempt in range(max_retries): try: async with semaphore: response = await client.embeddings.create( model=deployment, input=[item.text for item in batch], ) return {item.id: d.embedding for item, d in zip(batch, response.data)} except RateLimitError as e: # Transient - honour Retry-After if present, else exponential backoff + jitter. retry_after = getattr(e.response, "headers", {}).get("Retry-After") delay = float(retry_after) if retry_after else (2 ** attempt) + random.uniform(0, 1) await asyncio.sleep(delay) continue except APIStatusError as e: if e.status_code == 400 and "context_length" in str(e).lower(): # This should NEVER happen if assemble_batches() is configured # correctly. Do not retry - log as a DEFECT and surface it loudly. raise RuntimeError( f"Batcher assembled a non-compliant batch (items: " f"{[i.id for i in batch]}). This is an assembler bug, " f"not a transient failure. Investigate limits config." ) from e elif e.status_code >= 500: # Transient server-side issue - backoff and retry. await asyncio.sleep((2 ** attempt) + random.uniform(0, 1)) continue else: raise # genuinely unexpected - don't swallow it except APIConnectionError: await asyncio.sleep((2 ** attempt) + random.uniform(0, 1)) continue raise RuntimeError(f"Batch failed after {max_retries} retries: {[i.id for i in batch]}")

Track which source rows succeeded, so a re-run doesn't redo completed work

For a backfill against thousands of database rows, the pipeline will eventually be re-run — after a bug fix, after adding new rows, after a partial failure. Persist which row IDs have confirmed embeddings (a simple embedded_at timestamp column, or a separate tracking table) so a re-run can filter to only the rows still needing embedding, rather than re-processing — and re-paying for — rows that already succeeded. This matters more as the table grows; redoing a 50,000-row backfill because 200 rows failed partway through is an expensive way to fix 200 rows.

08Fix 4 — Respecting TPM Quota Alongside Batch LimitsThroughput

Per-request limits (Sections 1-2) and per-minute quota (Tokens-Per-Minute, TPM) are separate concerns, and a batcher that only solves the first will still get rate-limited by the second on a large backfill. A high-concurrency dispatcher that fires many compliant batches simultaneously can burn through TPM quota faster than the deployment allows, producing 429s that are a quota problem, not a batch-assembly problem.

Python — a token-bucket limiter alongside the concurrency semaphoreimport time class TokenBucketLimiter: """Simple token-bucket rate limiter keyed to your deployment's TPM quota.""" def __init__(self, tokens_per_minute: int): self.capacity = tokens_per_minute self.tokens = tokens_per_minute self.last_refill = time.monotonic() self._lock = asyncio.Lock() async def acquire(self, tokens_needed: int): async with self._lock: now = time.monotonic() elapsed = now - self.last_refill self.tokens = min(self.capacity, self.tokens + elapsed * (self.capacity / 60)) self.last_refill = now if self.tokens < tokens_needed: wait_time = (tokens_needed - self.tokens) / (self.capacity / 60) await asyncio.sleep(wait_time) self.tokens = 0 else: self.tokens -= tokens_needed # Set this to your ACTUAL deployment TPM quota, with margin - check # the Foundry portal or `az cognitiveservices account deployment show`. tpm_limiter = TokenBucketLimiter(tokens_per_minute=340_000) # example: 350K quota, 10K margin async def embed_batch_throttled(batch: list[BatchItem], deployment: str): batch_tokens = sum(item.token_count for item in batch) await tpm_limiter.acquire(batch_tokens) return await embed_batch_with_retry(batch, deployment)

Leave margin below your actual quota

Set the limiter's capacity below your deployment's actual TPM quota, not equal to it — other consumers of the same deployment (other pipelines, interactive traffic sharing the same model deployment), estimation error in token counting, and the quota's own enforcement granularity all argue for headroom. A limiter configured at exactly 100% of quota will still occasionally trigger 429s from timing edge cases; 90-95% of quota is a more realistic ceiling.

09Anti-Patterns: Batchers That Look Compliant and Aren'tTraps

Because the "16-input" fix is so widely known, teams often reach for a fix that resembles it without addressing the actual current constraint.

Anti-pattern	Why it feels right	Why it isn't
Just raise the chunk size to 2,048	"That's the new documented array limit"	Ignores the aggregate token cap entirely. Fails the moment average input length is realistic, not toy-sized
Estimate tokens from character count (÷4)	"Close enough, and it's fast"	Tokenization ratios vary meaningfully by content type. A batch that "should" be under the aggregate cap by rough estimate can still exceed it in reality
Retry a 400 context-length error with the same batch	"Retries fix most API errors"	An oversized batch fails identically every time. This wastes API calls and delays discovering the real assembler bug
Fixed concurrency with no TPM awareness	"More parallelism = faster backfill"	High concurrency on compliant-sized batches can still exceed per-minute quota, producing 429s that look like a batching problem but are actually a throughput problem
Silently drop or skip rows that fail after retries	"Don't let one bad row block the whole pipeline"	Silent data loss. A backfill that "completed" with gaps nobody tracked is a worse failure mode than a job that visibly stops and reports what's missing
Hardcode the limit values inline, scattered across the codebase	"It's just a number"	When the model or API version changes, someone has to hunt down every hardcoded 16, 2048, or 300000 across the codebase instead of updating one config object

Validation & Verification: Confirm the Fix

Confirm the batcher respects all three gates under realistic data, not just the toy inputs used during development, and that a full backfill run completes with every row accounted for.

Step 1 — Unit test: force each gate to bind independentlydef test_array_count_gate(): # 2,050 short items - should split into at least 2 batches on count alone items = [BatchItem(id=str(i), text="short", token_count=5) for i in range(2050)] batches = assemble_batches(items, EmbeddingBatchLimits()) assert all(len(b) <= 2048 for b in batches) assert sum(len(b) for b in batches) == 2050 # no items lost def test_aggregate_token_gate(): # 400 items at 1000 tokens each = 400,000 total - MUST split on aggregate, # well before hitting the 2,048 array-count limit. items = [BatchItem(id=str(i), text="x", token_count=1000) for i in range(400)] batches = assemble_batches(items, EmbeddingBatchLimits()) for b in batches: assert sum(i.token_count for i in b) <= 300_000 assert sum(len(b) for b in batches) == 400 # no items lost assert len(batches) > 1, "aggregate gate should have forced a split here" def test_no_batch_ever_exceeds_any_limit(): # Property-based check across randomized realistic input distributions. import random items = [ BatchItem(id=str(i), text="x", token_count=random.randint(50, 4000)) for i in range(5000) ] batches = assemble_batches(items, EmbeddingBatchLimits()) for b in batches: assert len(b) <= 2048 assert sum(i.token_count for i in b) <= 300_000 assert all(i.token_count <= 8192 for i in b)

Step 2 — Integration test against the real endpoint with production-shaped data# Pull a REPRESENTATIVE sample from the actual source table - not # synthetic uniform-length test data. Include the longest real rows. sample_rows = fetch_sample_rows(table="documents", n=2000, include_outliers=True) items = [ BatchItem(id=r.id, text=r.text, token_count=count_tokens(r.text)) for r in sample_rows ] batches, results = await run_pipeline(items, EmbeddingBatchLimits(), deployment="text-embedding-3-large") failed = [r for r in results if isinstance(r, Exception)] print(f"Batches: {len(batches)}, Failed: {len(failed)}") assert len(failed) == 0, f"Real-data integration test found failures: {failed}"

Step 3 — Confirm full row accounting after a backfill run-- SQL: every source row should have exactly one embedding, or an -- explicit reason it doesn't (oversized, routed to review, etc). SELECT COUNT(*) AS total_rows, COUNT(embedding_id) AS embedded_rows, COUNT(*) FILTER (WHERE embedding_id IS NULL AND review_reason IS NULL) AS UNACCOUNTED FROM documents; -- PASS: UNACCOUNTED = 0. Every row is either embedded or has a -- documented reason it wasn't (oversized, failed after retries). -- FAIL: UNACCOUNTED > 0 - rows were silently dropped somewhere in -- the pipeline. Find them before trusting the backfill is complete.

What "fixed" actually means here

Three conditions must hold together. One: property-based unit tests confirm no assembled batch ever violates any of the three gates, across randomized realistic token-length distributions, not just hand-picked examples. Two: an integration test against real (or realistically-shaped) production data — including your actual longest documents — completes with zero API-side batch failures. Three: after a full backfill run, every source row is accounted for: either successfully embedded, or explicitly routed to a documented exception path (oversized, permanently failed after retries) — never silently missing. Miss any of the three and either the fix is unverified against real data shapes, or the pipeline can still lose rows without anyone noticing.

Key Takeaways

→"16 inputs" is real but scoped to ada-002 on the legacy API. Current models via the modern endpoint allow up to 2,048 array items — a different, larger number, not a universal Azure constant.

→The aggregate 300,000-token cap is the one that catches modern pipelines. It fails independently of array count and per-item length — a batch can pass both of those checks and still be rejected.

→Three gates, checked together: array count, per-item tokens, aggregate tokens. A compliant batcher tracks all three running totals as it fills each batch, not just one.

→Count real tokens, not estimated ones. Character-count heuristics are close enough to pass casual testing and wrong enough to fail in production on real content.

→A 400 for exceeding a limit should never be retried unchanged. It means the assembler has a bug — treat it as a defect to fix, not a transient failure to retry through.

→Per-request limits and per-minute TPM quota are separate problems. A batcher that only solves request-level compliance can still get rate-limited by aggregate throughput on a large backfill.

→Track row-level completion, not just batch success. Full accounting after a backfill — every row embedded or explicitly excepted — is the only way to know nothing was silently dropped.

Frequently Asked Questions

Is the Azure OpenAI embeddings array limit really 16?

It depends entirely on the model and API surface. The error "Too many inputs. The max number of inputs is 16" is a real, verified Azure OpenAI error, but it's specific to text-embedding-ada-002 (Version 2) on the classic API. Current-generation models (text-embedding-3-large, text-embedding-3-small) accessed through the modern /openai/v1/embeddings endpoint support arrays of up to 2,048 items — matching OpenAI's public API limit, not 16. If you're seeing the "16 inputs" error specifically, you're most likely on ada-002 or an older API version; if you've migrated to a current model and are still hardcoding batches of 16, you're leaving significant throughput on the table by not raising your batch size — but be aware a separate 300,000-token aggregate limit (Section 2) can bind before the 2,048-item array limit does, depending on your input lengths.

Why does my batch fail even though it's under both the array size and per-input token limits?

You've likely hit the aggregate token limit — a separate, newer constraint that caps the SUM of tokens across every input in a single request at 300,000, regardless of how many items are in the array or how long any individual item is. This is documented in Microsoft's current Azure OpenAI embeddings guidance: requests fail with HTTP 400 when the aggregate exceeds 300,000 tokens, even when every individual input is under 8,192 tokens and the array length is under 2,048. For realistic document lengths — a few hundred to a couple thousand tokens per item — this aggregate limit is often reached well before the 2,048-item array limit, which is why a batcher that only checks array count can pass its own validation and still be rejected by the API.

How do I build a batching queue that handles all the embedding API limits correctly?

Track three running totals as you fill each batch — item count, and cumulative token sum across all items so far in the current batch — and close the batch out the moment adding the next item would cross any one of: the array-size limit (2,048 for current models), the aggregate token limit (300,000), or would itself individually exceed the per-item token limit (8,192, which should be filtered out or handled separately before batching, since no batch size can make an individually-oversized input compliant). A simple greedy-fill algorithm that checks all three conditions before adding each item, and starts a new batch when any would be violated, is sufficient — this doesn't require a sophisticated bin-packing algorithm, just consistent tracking of all three dimensions rather than just one.

Should I retry a request that fails with a "too many inputs" or "context length exceeded" error?

No — not by simply resending the identical request. These errors mean the batch itself violates a documented limit, and an unchanged retry will fail identically every time, wasting an API call and delaying the real fix. If your batching logic is correctly implemented (tracking all three limit dimensions as described above), this error should never occur during normal operation; if it does appear, treat it as a defect in the batch assembler — likely a misconfigured limit value, a stale or missing token count, or a model/API version mismatch where the configured limits don't match what's actually deployed — rather than a transient condition to retry through. Reserve retry logic with exponential backoff for genuinely transient failures: HTTP 429 rate limiting and 5xx server errors, both of which can reasonably succeed on a subsequent attempt.

Related FAVRITE Articles

• The Runaway Vector Index Bill: Tuning Vector Dimensions in Azure Cosmos DB
• How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
• The PTU Math Trap: When to Pivot from Pay-As-You-Go to Provisioned Throughput
• Purging the Keys: Migrating Azure OpenAI Applications to Managed Identities and RBAC