The Runaway Vector Index Bill: Tuning Dimensions
& Quantization in Azure Cosmos DB for AI
Your bill is not high because AI is expensive. It is high because you are storing every document as a 3,072-number vector when — by OpenAI's own benchmarks — you could store it as a 256-number vector and get better retrieval than your predecessor architecture had at 1,536. The default was chosen for you, and it is the wrong one for almost every RAG workload.
# Azure Cosmos DB cost report — the line item nobody expected:
Resource: cosmos-rag-prod (Provisioned throughput, DiskANN vector index)
Period: last 30 days
Item RU-hours Cost (USD)
Storage — indexed data 18,240,000 $3,650
Storage — VECTOR INDEX (3072-dim, 8M docs) $9,720 ← 62% of bill
Point reads 2,100,000 $170
Vector search queries 5,900,000 $470
────────────────────────────────────────────────────────────────────────
Total $15,650
↑
The vector index costs
MORE than the underlying
data it is indexing.
# The math driving it — floats are 4 bytes, and it multiplies fast:
# 1 vector @ 3072 dims = 3072 × 4 bytes = 12,288 bytes ≈ 12 KB
# 8,000,000 vectors = 8M × 12 KB = ~92 GB of index
# The same corpus at 512 dimensions:
# 1 vector @ 512 dims = 512 × 4 bytes = 2,048 bytes ≈ 2 KB
# 8,000,000 vectors = 8M × 2 KB = ~15 GB of index (6× smaller)
# The retrieval quality question everyone assumes will be worse:
# text-embedding-3-large @ 256 dims: MTEB 62.0
# text-embedding-ada-002 @ 1536 dims: MTEB 61.0 ← the OLD default
# you were happy with
# A 12× smaller vector beats the model your predecessor RAG ran on.Symptom: Cosmos DB storage and RU consumption growing linearly with your corpus, with the vector index becoming the single largest line item on the bill. Failure point: Client passes no dimensions parameter to the embedding API → receives the default (3072 for text-embedding-3-large, 1536 for -3-small) → stores every vector at full width forever. Default platform behaviour: Both the OpenAI SDK and the Cosmos DB indexing policy will happily accept whatever you send. Neither warns you that you are provisioning 6–12× more storage than the workload needs.
There is a particular flavour of cost incident that is uniquely damaging because the outcome looks correct. Your RAG works. Retrieval is snappy. Users are happy. And the bill is quietly ratcheting upward, month after month, because every document in your corpus is being stored as a 3,072-number vector when it could just as effectively be stored as a 512-number one. Nothing is broken. Nothing will be broken tomorrow either. But you are paying six times more than you have to for an index that would perform better if it were smaller — and the reason is that the OpenAI SDK returns 3,072 dimensions when you do not tell it otherwise, and neither Cosmos DB nor your monitoring will ever point out that the default was chosen for you.
A vector is not a magical AI object. It is a flat array of 32-bit floating-point numbers, each occupying 4 bytes. That is it. The size of your vector index in bytes is therefore a mechanical calculation:
Two things follow from that formula, and they are the only two things you need to know to see the whole problem:
- Storage is linear in dimensions. Halving dimensions halves storage. Quartering dimensions quarters storage. There is no economy of scale in the wrong direction to save you.
- The multiplier is your corpus. At a thousand documents, nobody notices. At ten million documents, a 12× dimension difference is the difference between a hobby project and a five-figure monthly line item.
So the "runaway bill" is not exotic. It is the OpenAI default (3072 for text-embedding-3-large, 1536 for text-embedding-3-small) meeting a corpus that grew. Your predecessors chose those defaults implicitly, by not passing a dimensions parameter, and Cosmos DB happily indexed whatever arrived. Both systems worked exactly as designed. The problem is that the design assumes you know a lever exists.
Your Cosmos DB vector spend is not one line — it is storage (bytes at rest) plus RUs per query (compute at read time). Shrinking dimensions attacks both, because a smaller vector is faster to compare as well as cheaper to store. Switching to a smarter index type (Section 6) further reduces the RUs per query. If you only tune one of the two, you have half-fixed the problem — and it will keep growing on the other axis until you go back and do the second half. Treat this as a two-lever exercise from the start.
The instinct is that a smaller vector must mean worse retrieval, so shrinking dimensions must be a quality trade. That instinct is right for most embedding models — but not for the ones that matter here. OpenAI's text-embedding-3-small and text-embedding-3-large are trained with Matryoshka Representation Learning (MRL), and it changes the arithmetic completely.
MRL trains the model so that the first N dimensions of the vector carry the most important information — the coarse semantic summary — and later dimensions add progressively finer detail. Truncating to fewer dimensions loses nuance but keeps the signal. The dimensions are not equally valuable; the training deliberately concentrates value in the early ones.
This is what makes the headline result surprising. On OpenAI's own MTEB benchmark:
| Model & dimensions | MTEB retrieval score | Relative storage |
|---|---|---|
| text-embedding-ada-002 @ 1536 (the old default) | 61.0 | 1× |
| text-embedding-3-large @ 256 | 62.0 | 0.17× — six times smaller and better |
| text-embedding-3-large @ 1024 | ~64.6 | 0.67× |
| text-embedding-3-large @ 3072 (default) | 64.6 | 2× |
Read that first two rows again. A 3-large vector truncated to 256 dimensions outperforms the ada-002 vector at its full 1,536. The truncated vector is one-twelfth the width and still wins. This is why "shrink the vector" is not a quality-cost trade for MRL-trained models — for many workloads it is a strict improvement, and any team still on 3,072 dimensions by default is paying to lose the trade in the wrong direction.
You can truncate a full-size vector in your own code by taking the first N values, but the resulting vector will not be re-normalised. OpenAI's server-side truncation via the dimensions parameter returns a properly normalised vector at the requested size, which is what cosine similarity assumes. Truncating client-side without renormalising subtly corrupts distance calculations for reasons that are hard to see until retrieval quality degrades on a specific slice of your data. Let the API do the truncation.
MRL is not magic. The gentle degradation curve is real but has limits, and it depends on your corpus. On the STS benchmark, an MRL model truncated to just 8% of its original dimensions kept ~98% of its full performance; a non-MRL model truncated the same way kept ~96%, and the gap widens fast as you cut deeper. So the pattern is: aggressive cuts are viable for MRL models, but the exact retention on your data is empirical. Build a golden set of query-to-expected-document pairs and measure recall@k at each candidate dimension size before you commit. Validation (Section 10) shows how.
The brief calls the fix "scalar quantization." In Cosmos DB, that fix has a specific name — actually two — and choosing between them is where more cost incidents originate than dimension count does. Cosmos DB offers three vector index types, and the differences are stark.
| Index type | Max dims | Search algorithm | Accuracy | Cost per query |
|---|---|---|---|---|
| flat | 505 | Brute-force (exhaustive) | 100% recall | Highest RU cost |
| quantizedFlat | 4,096 | Brute-force over compressed vectors | Slightly below 100% | Lower RU than flat |
| diskANN | 4,096 | Approximate nearest neighbour (graph) | High but ANN — below quantizedFlat/flat | Lowest RU |
The flat index gives you 100% recall — it is guaranteed to find the most similar vectors — but it has a hard 505-dimension ceiling, which means any real production embedding will not fit. That single number rules flat out of most modern RAG stacks.
quantizedFlat keeps the brute-force search behaviour but compresses the vectors themselves, giving you the same shape of guarantee at a fraction of the storage. And diskANN — Microsoft Research's approximate nearest neighbour algorithm — gives you the lowest RU cost and lowest latency at scale, at the price of trading 100% recall for very-high recall. It is what powers the built-in vector search in Cosmos DB.
Both quantizedFlat and diskANN require at least 1,000 vectors in the container before the index becomes effective — this is needed for the quantization to be accurate. Below that threshold, Cosmos DB silently falls back to a full scan, and the RU charges for those queries can be higher than a plain flat index would have been. This is why teams often build a proof-of-concept, watch it hit implausibly high RUs on a tiny dev corpus, and conclude the whole approach is expensive. The fix is not to change indexes — it is to reach 1,000 vectors, at which point costs drop dramatically. Benchmark on a corpus of realistic size.
Vector indexing and search on Cosmos DB for NoSQL are not supported on accounts with Shared Throughput — you must use provisioned or serverless. And once vector indexing is enabled on a container, it cannot be disabled. So the choice is not one you back out of casually. Do the dimension and index-type selection deliberately, on a new container, and validate before you point production at it.
Architectural Topology: Failing vs Remediated
| Layer | Failing configuration (current) | Remediated configuration (fix) |
|---|---|---|
| Embed call | No dimensions param — accepts 3072-dim default | Explicit dimensions=1024 (or lower) at embed time |
| Model | text-embedding-3-large at full width | Same model, truncated via MRL — no retraining needed |
| Cosmos DB index type | flat, or no explicit choice | quantizedFlat for accuracy-critical; diskANN for scale |
| Dimension ceiling | Capped at 505 (flat) — you were forced to change something eventually | 4,096 (quantizedFlat / diskANN) — headroom, deliberately unused |
| quantizationByteSize | Not set — system decides | Explicitly tuned once you have measured recall vs RU |
| Storage per vector | ~12 KB at 3072 dims | ~1–4 KB at 256–1024 dims, further reduced by quantization |
| Query RUs | High — brute force over 3072-float comparisons | Sub-linear at scale on diskANN |
| Retrieval quality | Assumed good; never measured | Recall@k measured on a golden set; documented before commit |
This is the change that recovers the most money for the least effort. One parameter on the embedding call determines every downstream storage byte and every downstream query comparison. Set it deliberately.
How to pick the target dimension
There is no universal right number, but there is a defensible decision procedure. Pick your candidate sizes from a short list — 256, 384, 512, 1024 — because MRL retention is most reliable at common power-of-two-ish cuts. Then measure. Do not eyeball it.
| Target dims | Use when | Storage vs 3072 |
|---|---|---|
| 256 | Storage is the acute bottleneck; corpus is huge; queries are broad | ~ 12× smaller |
| 512 | Mobile or edge deployments; also the ceiling if you insist on flat index (with room to spare) | ~ 6× smaller |
| 1024 | Default recommendation. Balanced storage/quality trade for most production RAG | ~ 3× smaller |
| 1536 | You've measured a meaningful recall drop at 1024 on your corpus | ~ 2× smaller |
| 3072 | Almost never justified. Retention curve is nearly flat above 1024–1536 | 1× (baseline) |
The same dimension setting must be used for the vectors you index and the vectors you query with. This is not a Cosmos DB or an OpenAI limitation — it is how vector similarity works. Embed your entire corpus at 1024 dims and then embed queries at 512, and the resulting cosine distances are meaningless. If you have code paths that embed at different times (e.g. an ingestion pipeline and a query service), the dimensions setting must be shared configuration, not an inline literal. Get this wrong once and every silent recall regression that follows will look mysterious.
Dimensions are the first lever. The index type is the second, and it does the other half of the job — reducing RU cost per query, and adding a further storage compression on top of what you already saved by shrinking the vector.
The decision is workload-shaped, not universal.
- Small corpus (< 100K vectors) with hybrid filters: quantizedFlat. You're often filtering to a small candidate set with a WHERE clause, then vector-searching over what remains. Brute force is fine when the "brute" set is small, and quantization gives you the storage win.
- Large corpus, latency-sensitive: diskANN. Its cost is highly sub-linear in the number of vectors — going from 100K to 1M barely raises per-query RUs. It is the reason Cosmos DB can search a 35M-vector corpus at low latency at all.
- Absolute-recall-critical, small dims: flat. Only if you can live inside the 505-dimension ceiling.
A subtle but material step: add the vector path (e.g. /embedding/*) to excludedPaths in the normal indexing policy. If you leave it included, Cosmos DB will attempt to index the array elements individually as scalar range/hash entries in addition to your vector index — bloating storage and RU consumption for zero benefit. The vector index is the correct home for that data; the general index has no reason to be there.
Once you've picked a quantized index type, one more parameter deserves a deliberate value: quantizationByteSize. It applies to both quantizedFlat and diskANN, and it controls how much each vector is compressed in the index — separate from, and layered on top of, the dimension count you chose at embed time.
| Parameter | Range | Applies to | Effect |
|---|---|---|---|
| quantizationByteSize | Min 1, Max 512, default dynamic (system decides) | quantizedFlat + diskANN | Larger = higher accuracy, higher RU, higher latency. Smaller = the opposite |
| quantizerType | product or spherical | quantizedFlat + diskANN | Method of quantization. Default is usually appropriate |
| indexingSearchListSize | Min 10, Max 500, default 100 | DiskANN only | How many vectors are searched during index construction. Larger = better recall, longer index build times, higher ingest latency |
Leave quantizationByteSize at its default (system-chosen) until you have a working baseline. It is a fine-tuning knob, not a starter setting, and turning it before you have a benchmark to compare against usually degrades one of the three axes (accuracy, RU, latency) invisibly. When you do tune it, hold every other variable fixed — same corpus, same query set, same dimension count — and measure recall@k and RU per query at each candidate byte size. Then pick the point on the curve where a further increase costs more RU than it recovers in recall.
This is the section that saves you from the incident, because the fix in this article cannot be applied in place, and the failure mode of an incorrect migration is silent — semantically meaningless retrieval that will not throw an error but will quietly degrade your RAG for anyone reading the answers.
Three hard constraints, all of them stated by Microsoft:
- Vector indexing cannot be enabled on an existing container. To use vector search — or to change how it is configured — you must create a new container and specify the vector embedding policy at creation time.
- Once vector indexing is enabled on a container, it cannot be disabled. This choice is one-way.
- Vectors are only comparable within the space of the model that produced them, at the dimension count they were produced at. Mixing dimension sizes across a single index is not a supported operation; it is a corruption.
Put together, this means the migration has a shape that must be planned:
- Create a new Cosmos DB container with the desired vector policy — target dimensions, target index type, target quantizationByteSize.
- Re-embed the entire corpus at the target dimension count. Not slice, not truncate client-side — re-embed, so vectors are re-normalised by OpenAI at the target size.
- Ingest the new vectors into the new container.
- Update your query path to embed queries at the same target dimension.
- Cut over. Deprecate the old container.
This is a variant of the embedding-space failure mode: mixing dimensions across indexing and querying corrupts distance calculations. If your ingest pipeline re-embeds at 1024 dims and your query service still calls the embedding API without a dimensions parameter — which returns the 3072 default — you now have a 3072-dim query trying to search a 1024-dim index. In the best case Cosmos DB rejects the request. In the worst case, if some layer of your stack silently pads or truncates, you get retrieved "results" that are semantically meaningless. Share the dimension setting through one config value that both paths import.
Do not big-bang the cutover. Ingest into the new container in parallel with the old one running, then run a period of shadow queries — send each production query to both containers and compare recall@k on a golden set. Only when the new container is measurably at least as good on your actual traffic do you flip the switch. If you skip the shadow period, the day the new container becomes canonical is the day you might discover that 1024 dims was too aggressive for your specific corpus, and by then the old container may already be deprovisioned.
Because vector-cost problems are so obvious once the bill arrives, the wrong instincts are strong. These are the fixes teams reach for that either fail silently or create larger problems than they solve.
| Anti-pattern | Why it feels right | Why it isn't |
|---|---|---|
| Truncate the vector in client code (vec[:1024]) | "It's the same numbers — how could it be different?" | Not re-normalised. Cosine distance assumes unit vectors. Retrieval quietly degrades on subsets of the data |
| Downgrade to text-embedding-3-small | "Smaller model, smaller vectors, smaller bill" | Different model = different vector space. You must re-index the whole corpus AND lose the MRL retention advantage. Often worse than a truncated -3-large |
| Delete the vector index and re-enable it | "Fresh start with better settings" | Once vector indexing is on, it cannot be disabled. You need a new container, not a policy edit |
| Assume 8-bit is always safe because "quantization" | "Every DB does it, mustn't be a big deal" | Cosmos DB's quantizationByteSize is a per-vector budget, not a global 8-bit switch. Under-tuning it drops recall well below where the same corpus would sit at full width |
| Skip the 1,000-vector floor test | "Dev works, prod will work" | Below 1,000 vectors, both quantizedFlat and diskANN silently full-scan. Dev shows higher RUs than prod, and you tune the wrong knob |
| Tune quantizationByteSize before measuring | "Bigger number, higher accuracy — obviously" | You are trading RU for accuracy you cannot see. Without a golden set, you're not tuning, you're guessing |
Validation & Verification: Confirm the Fix
The two failure modes are opposite. One: the bill did not actually go down (your dimensions parameter is not being applied). Two: the bill went down but retrieval quality went with it (Matryoshka retention is not as gentle as the benchmarks promised on your data). Validate both, in that order.
Three conditions must hold together. One: your query path is verifiably returning vectors at the target dimension — not the API default sneaking back in behind you. Two: recall@k on a golden set drawn from real production queries is within an acceptable margin of the baseline; if MRL retention did not hold at your chosen size on your specific corpus, you find out here, before users do. Three: both storage size and RU per query are meaningfully lower on the new container. Miss any of the three and you have won a partial argument — the bill went down but retrieval degraded, or retrieval held but the bill did not actually move because a query path was silently defaulting to 3072. Only when all three land is the fix real.
Key Takeaways
Frequently Asked Questions
Related FAVRITE Articles
- The Edge Latency Drop: Offloading Embeddings to Foundry Local 1.2
- The Missing Data RAG Pipeline Bug: 30-Second Timeouts in Azure AI Search
- How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
- Azure AI Search RAG Runbook