Reduce AI application latency spikes by offloading embeddings to Foundry Local 1.2, improving edge performance, responsiveness, scalability, and cost efficiency.
The Edge Latency Drop: Fixing Latency Spikes by Offloading Embeddings to Foundry Local 1.2
You are paying a full cloud round trip — network, TLS, queue, throttle risk — to turn a twelve-word search query into a vector. That is the most expensive way possible to do one of the cheapest computations in your stack. Foundry Local 1.2 now runs on Linux ARM64, which means embeddings and speech recognition can happen on a Raspberry Pi, a Jetson, or a Graviton instance — offline, unmetered, and in single-digit milliseconds.
The failure signature this guide resolves# Application Insights — the embedding call, not the LLM, is your tail latency:
name p50 p95 p99 calls/day
POST /embeddings (cloud) 89 ms 412 ms 3,847 ms 1,240,000
POST /chat/completions (cloud) 940 ms 1,720 ms 2,910 ms 38,000
^^^^^^^^
A 12-word query embedding is your SLOWEST operation at p99.
# And under load, the same tiny call starts failing outright:
HTTP/1.1 429 Too Many Requests
Retry-After: 12
{"error":{"code":"429","message":"Requests to the Embeddings_Create Operation
under Azure OpenAI API version 2024-02-01 have exceeded call rate limit."}}
# Or, on an edge device with intermittent connectivity, it simply dies:
requests.exceptions.ConnectionError: HTTPSConnectionPool(
host='my-aoai.openai.azure.com', port=443): Max retries exceeded
(Caused by NewConnectionError: Failed to establish a new connection:
[Errno -3] Temporary failure in name resolution)
Symptom: Embedding and transcription calls dominate p99 latency and throttle budget, despite being computationally trivial. Failure point: Client → network → TLS → cloud queue → a 0.6B-parameter model that could run locally in milliseconds. Default platform behaviour: Every embedding, however small, pays the full round trip and consumes shared quota. The compute is cheap; the journey is the cost.
0.6Bqwen3-0.6b-embedding — the model doing the work you are currently sending across the internetARM64Foundry Local 1.2 runs on Raspberry Pi 5, NVIDIA Jetson, AWS Graviton, and Ampere — the edge is now a first-class target40+ langsMultilingual streaming ASR on-device via Nemotron 3.5, at roughly 8% WER and low single-digit % CPU$0 / tokenNo per-token cost, no quota, no throttle — and no network. The whole point of moving the work, not just speeding it upThere is a quiet absurdity at the heart of most RAG architectures. To answer a question, you send the user's twelve-word query across the public internet, terminate TLS, wait in a shared queue, and consume metered quota — all so that a model can perform a computation that a Raspberry Pi could complete in a few milliseconds without ever touching a network. The generation step genuinely needs a frontier model in the cloud. The embedding step does not, and never really did. It was in the cloud because that was the only place it could be. As of Foundry Local 1.2, with Linux ARM64 support, that is no longer true — and the latency you get back is not marginal, because you are not making the call faster. You are deleting it.
Figure 1 — Where the milliseconds actually go: a cloud embedding call vs a local oneA cloud embedding call spends the overwhelming majority of its wall-clock time on transport and queueing, not on the inference itself. Moving the model on-device does not optimise those stages — it removes them entirely, which is why the p99 improvement is far larger than the p50 improvement.01The Latency Math: Why Embeddings Don't Belong in the CloudRoot CauseThe instinct is that AI work belongs in the cloud because AI work is heavy. That is true of a frontier chat model. It is emphatically not true of an embedding model, and the mismatch is where your latency budget is being burned.
Consider the asymmetry. A chat completion runs tens of billions of parameters, streams hundreds of tokens, and takes a second or more — the network round trip is a rounding error against the compute. An embedding runs a 0.6-billion-parameter model over a dozen words and returns a single fixed-length vector. The compute is trivial. The round trip is not. You have inverted the ratio: for embeddings, the transport is the workload.
Property Chat completion Text embedding Model size Tens to hundreds of billions of params ~0.6B (qwen3-0.6b-embedding) Output Hundreds of streamed tokens One fixed-length vector Compute vs transport Compute dominates — cloud is correct Transport dominates — cloud is wrong Call volume Low (one per user turn) Very high (every query + every document chunk) Quota pressure Expected and budgeted Silently consumes the same shared quota Offline capable? No — needs a frontier model Yes — trivially
Now add volume. Embeddings are called on every query and on every document chunk during indexing — often by orders of magnitude more often than your chat endpoint. Each one competes for the same tenant quota. This is why teams see the embedding endpoint throttle first and hardest, and why a 429 on a trivial vector call can take down a feature that has nothing to do with generation.
The tail is what your users feelA cloud embedding at p50 looks perfectly healthy — 89ms, nobody complains. The damage is at p99, where queueing and throttling stretch the same trivial call into multiple seconds. Because local inference has no queue to sit in and no quota to exceed, moving the work on-device does not just shift the median down — it collapses the tail toward the median. That is the real prize, and it is invisible if you only look at averages.
02What Foundry Local 1.2 Actually ShipsCapabilityFoundry Local is Microsoft's cross-platform local AI runtime — models execute on the user's own hardware with no cloud dependency, no network latency, and no per-token cost. It auto-detects available hardware and selects the best execution provider (NPU, GPU, or CPU). The two releases that make this article possible are 1.1 and 1.2.
Capability Release What it means for you Text embeddings 1.1 On-device embedding generation via a dedicated embedding client. Ships with qwen3-0.6b-embedding. Single and batch input, configurable dimensions, and OpenAI-format responses for cloud-to-edge portability Live transcription 1.1 Real-time streaming speech-to-text from a microphone, with an OpenAI Realtime-compatible surface. Model: nemotron-speech-streaming-en-0.6b Linux ARM64 1.2 The unlock. Runs on Raspberry Pi 5, NVIDIA Jetson, AWS Graviton, and Ampere — extending local AI to genuine edge and embedded scenarios Multilingual ASR 1.2 40+ languages via the Nemotron 3.5 ASR Streaming Multilingual model — no longer English-only Inference cancellation 1.2 Cancel in-flight chat completions and transcription sessions cleanly when the user moves on — no wasted compute or orphaned streams Faster model downloads 1.2 The catalog is fronted by Azure Traffic Manager, routing to the best-performing region. No code change — just move to the 1.2.0 SDK File transcription — Whisper models (e.g. whisper-tiny) for transcribing audio files, across all SDKs
Five SDKs are available — Python, JavaScript, C#, Rust, and C++ — plus a CLI and an optional OpenAI-compatible local server.
The ASR numbers are better than you'd expectThe streaming ASR model was not chosen casually. Microsoft benchmarked over 50 configurations across encoder-decoder, transducer, and LLM-based architectures — including Whisper, Nemotron, Parakeet, Canary, and Qwen3-ASR — before selecting NVIDIA's Nemotron Speech Streaming for resource-constrained hardware. They then re-implemented the pipeline in ONNX Runtime with aggressive quantisation, taking the model from 2.47GB down to as little as 0.67GB while holding word error rate within 1% of the full-precision baseline. In their testing it delivers roughly 8% WER using low single-digit percent CPU. That is genuinely deployable on a Pi.
03The Hybrid Split: What Moves to the Edge, What StaysArchitectureGoing hybrid is not "move AI to the edge." It is a deliberate split along a single axis: does this workload's value come from model scale, or from proximity? Get the split right and you keep frontier quality where it matters while deleting the network from everything else.
Workload Where it belongs Why Text embeddings Edge Tiny model, enormous call volume, transport-dominated. The textbook case Speech-to-text Edge Streaming audio to the cloud is expensive, laggy, and a privacy exposure. Runs at ~8% WER locally Intent / classification Edge Small models, high frequency, latency-sensitive Vector search (local corpus) Edge Pair local embeddings with a local vector store for a fully offline retrieval pipeline Final answer generation Cloud Genuinely needs a frontier model. Compute dominates, so the round trip is a rounding error Complex reasoning / agents Cloud Model scale is the product here Enterprise-wide vector index Cloud Shared, large, centrally governed — not an edge concern
The privacy dividend is not a footnoteLocal ASR means raw audio — customer voices, clinical dictation, shop-floor conversation — never leaves the device. For a regulated workload, that is often a more compelling argument than the latency, because it changes what data crosses a boundary at all rather than merely how fast it gets there. If you are currently streaming microphone audio to a cloud speech endpoint, the compliance case for this migration may be stronger than the performance one.
04⚠ The Embedding-Space Trap (Read Before You Migrate)CriticalThis is the section that will save you a production incident, and it is the one most "move embeddings to the edge" write-ups skip entirely. If you take nothing else from this article, take this.
You cannot swap the embedding model on one side of a vector search. An embedding is only meaningful relative to the model that produced it. Vectors from text-embedding-3-large and vectors from qwen3-0.6b-embedding live in different geometric spaces. They are not compatible, not convertible, and not comparable — even if you force the dimensions to match.
So if your Azure AI Search index was built in the cloud with text-embedding-3-large, and you now start embedding queries locally with qwen3-0.6b-embedding, one of two things happens:
Outcome A — you get lucky and it fails loudly (dimension mismatch)HTTP/1.1 400 Bad Request
{
"error": {
"code": "InvalidRequestParameter",
"message": "The vector field 'contentVector' has dimension 3072, but the
query vector has dimension 1024. Vector dimensions must match."
}
}
Outcome B — the dimensions happen to match, and it fails SILENTLYHTTP/1.1 200 OK
# No error. No warning. Retrieval "works".
# It just returns semantically irrelevant documents, because you are
# measuring cosine distance between vectors from two unrelated spaces.
# Your RAG answers quietly degrade into confident nonsense.
# This is the dangerous one. There is NO error string to search for.
Outcome B is the nightmare. There is no exception, no 400, no alert — retrieval quality simply collapses, and because LLMs are fluent even on bad context, your application keeps producing confident, well-written, wrong answers. Teams have shipped this and discovered it weeks later through user complaints about "the AI getting dumber."
The rule, stated plainlyThe model that embeds your documents and the model that embeds your queries must be the same model. Full stop. This is not a Foundry Local limitation — it is how vector spaces work. Moving query embedding to the edge therefore means one of the two paths below, and there is no third option.
You are paying a full cloud round trip — network, TLS, queue, throttle risk — to turn a twelve-word search query into a vector. That is the most expensive way possible to do one of the cheapest computations in your stack. Foundry Local 1.2 now runs on Linux ARM64, which means embeddings and speech recognition can happen on a Raspberry Pi, a Jetson, or a Graviton instance — offline, unmetered, and in single-digit milliseconds.
# Application Insights — the embedding call, not the LLM, is your tail latency:
name p50 p95 p99 calls/day
POST /embeddings (cloud) 89 ms 412 ms 3,847 ms 1,240,000
POST /chat/completions (cloud) 940 ms 1,720 ms 2,910 ms 38,000
^^^^^^^^
A 12-word query embedding is your SLOWEST operation at p99.
# And under load, the same tiny call starts failing outright:
HTTP/1.1 429 Too Many Requests
Retry-After: 12
{"error":{"code":"429","message":"Requests to the Embeddings_Create Operation
under Azure OpenAI API version 2024-02-01 have exceeded call rate limit."}}
# Or, on an edge device with intermittent connectivity, it simply dies:
requests.exceptions.ConnectionError: HTTPSConnectionPool(
host='my-aoai.openai.azure.com', port=443): Max retries exceeded
(Caused by NewConnectionError: Failed to establish a new connection:
[Errno -3] Temporary failure in name resolution)Symptom: Embedding and transcription calls dominate p99 latency and throttle budget, despite being computationally trivial. Failure point: Client → network → TLS → cloud queue → a 0.6B-parameter model that could run locally in milliseconds. Default platform behaviour: Every embedding, however small, pays the full round trip and consumes shared quota. The compute is cheap; the journey is the cost.
There is a quiet absurdity at the heart of most RAG architectures. To answer a question, you send the user's twelve-word query across the public internet, terminate TLS, wait in a shared queue, and consume metered quota — all so that a model can perform a computation that a Raspberry Pi could complete in a few milliseconds without ever touching a network. The generation step genuinely needs a frontier model in the cloud. The embedding step does not, and never really did. It was in the cloud because that was the only place it could be. As of Foundry Local 1.2, with Linux ARM64 support, that is no longer true — and the latency you get back is not marginal, because you are not making the call faster. You are deleting it.
The instinct is that AI work belongs in the cloud because AI work is heavy. That is true of a frontier chat model. It is emphatically not true of an embedding model, and the mismatch is where your latency budget is being burned.
Consider the asymmetry. A chat completion runs tens of billions of parameters, streams hundreds of tokens, and takes a second or more — the network round trip is a rounding error against the compute. An embedding runs a 0.6-billion-parameter model over a dozen words and returns a single fixed-length vector. The compute is trivial. The round trip is not. You have inverted the ratio: for embeddings, the transport is the workload.
| Property | Chat completion | Text embedding |
|---|---|---|
| Model size | Tens to hundreds of billions of params | ~0.6B (qwen3-0.6b-embedding) |
| Output | Hundreds of streamed tokens | One fixed-length vector |
| Compute vs transport | Compute dominates — cloud is correct | Transport dominates — cloud is wrong |
| Call volume | Low (one per user turn) | Very high (every query + every document chunk) |
| Quota pressure | Expected and budgeted | Silently consumes the same shared quota |
| Offline capable? | No — needs a frontier model | Yes — trivially |
Now add volume. Embeddings are called on every query and on every document chunk during indexing — often by orders of magnitude more often than your chat endpoint. Each one competes for the same tenant quota. This is why teams see the embedding endpoint throttle first and hardest, and why a 429 on a trivial vector call can take down a feature that has nothing to do with generation.
A cloud embedding at p50 looks perfectly healthy — 89ms, nobody complains. The damage is at p99, where queueing and throttling stretch the same trivial call into multiple seconds. Because local inference has no queue to sit in and no quota to exceed, moving the work on-device does not just shift the median down — it collapses the tail toward the median. That is the real prize, and it is invisible if you only look at averages.
Foundry Local is Microsoft's cross-platform local AI runtime — models execute on the user's own hardware with no cloud dependency, no network latency, and no per-token cost. It auto-detects available hardware and selects the best execution provider (NPU, GPU, or CPU). The two releases that make this article possible are 1.1 and 1.2.
| Capability | Release | What it means for you |
|---|---|---|
| Text embeddings | 1.1 | On-device embedding generation via a dedicated embedding client. Ships with qwen3-0.6b-embedding. Single and batch input, configurable dimensions, and OpenAI-format responses for cloud-to-edge portability |
| Live transcription | 1.1 | Real-time streaming speech-to-text from a microphone, with an OpenAI Realtime-compatible surface. Model: nemotron-speech-streaming-en-0.6b |
| Linux ARM64 | 1.2 | The unlock. Runs on Raspberry Pi 5, NVIDIA Jetson, AWS Graviton, and Ampere — extending local AI to genuine edge and embedded scenarios |
| Multilingual ASR | 1.2 | 40+ languages via the Nemotron 3.5 ASR Streaming Multilingual model — no longer English-only |
| Inference cancellation | 1.2 | Cancel in-flight chat completions and transcription sessions cleanly when the user moves on — no wasted compute or orphaned streams |
| Faster model downloads | 1.2 | The catalog is fronted by Azure Traffic Manager, routing to the best-performing region. No code change — just move to the 1.2.0 SDK |
| File transcription | — | Whisper models (e.g. whisper-tiny) for transcribing audio files, across all SDKs |
Five SDKs are available — Python, JavaScript, C#, Rust, and C++ — plus a CLI and an optional OpenAI-compatible local server.
The streaming ASR model was not chosen casually. Microsoft benchmarked over 50 configurations across encoder-decoder, transducer, and LLM-based architectures — including Whisper, Nemotron, Parakeet, Canary, and Qwen3-ASR — before selecting NVIDIA's Nemotron Speech Streaming for resource-constrained hardware. They then re-implemented the pipeline in ONNX Runtime with aggressive quantisation, taking the model from 2.47GB down to as little as 0.67GB while holding word error rate within 1% of the full-precision baseline. In their testing it delivers roughly 8% WER using low single-digit percent CPU. That is genuinely deployable on a Pi.
Going hybrid is not "move AI to the edge." It is a deliberate split along a single axis: does this workload's value come from model scale, or from proximity? Get the split right and you keep frontier quality where it matters while deleting the network from everything else.
| Workload | Where it belongs | Why |
|---|---|---|
| Text embeddings | Edge | Tiny model, enormous call volume, transport-dominated. The textbook case |
| Speech-to-text | Edge | Streaming audio to the cloud is expensive, laggy, and a privacy exposure. Runs at ~8% WER locally |
| Intent / classification | Edge | Small models, high frequency, latency-sensitive |
| Vector search (local corpus) | Edge | Pair local embeddings with a local vector store for a fully offline retrieval pipeline |
| Final answer generation | Cloud | Genuinely needs a frontier model. Compute dominates, so the round trip is a rounding error |
| Complex reasoning / agents | Cloud | Model scale is the product here |
| Enterprise-wide vector index | Cloud | Shared, large, centrally governed — not an edge concern |
Local ASR means raw audio — customer voices, clinical dictation, shop-floor conversation — never leaves the device. For a regulated workload, that is often a more compelling argument than the latency, because it changes what data crosses a boundary at all rather than merely how fast it gets there. If you are currently streaming microphone audio to a cloud speech endpoint, the compliance case for this migration may be stronger than the performance one.
This is the section that will save you a production incident, and it is the one most "move embeddings to the edge" write-ups skip entirely. If you take nothing else from this article, take this.
You cannot swap the embedding model on one side of a vector search. An embedding is only meaningful relative to the model that produced it. Vectors from text-embedding-3-large and vectors from qwen3-0.6b-embedding live in different geometric spaces. They are not compatible, not convertible, and not comparable — even if you force the dimensions to match.
So if your Azure AI Search index was built in the cloud with text-embedding-3-large, and you now start embedding queries locally with qwen3-0.6b-embedding, one of two things happens:
HTTP/1.1 400 Bad Request
{
"error": {
"code": "InvalidRequestParameter",
"message": "The vector field 'contentVector' has dimension 3072, but the
query vector has dimension 1024. Vector dimensions must match."
}
}HTTP/1.1 200 OK
# No error. No warning. Retrieval "works".
# It just returns semantically irrelevant documents, because you are
# measuring cosine distance between vectors from two unrelated spaces.
# Your RAG answers quietly degrade into confident nonsense.
# This is the dangerous one. There is NO error string to search for.Outcome B is the nightmare. There is no exception, no 400, no alert — retrieval quality simply collapses, and because LLMs are fluent even on bad context, your application keeps producing confident, well-written, wrong answers. Teams have shipped this and discovered it weeks later through user complaints about "the AI getting dumber."
The model that embeds your documents and the model that embeds your queries must be the same model. Full stop. This is not a Foundry Local limitation — it is how vector spaces work. Moving query embedding to the edge therefore means one of the two paths below, and there is no third option.
Your two legitimate migration paths
Figure 2 — Two vector spaces, one index: why swapping only the query model corrupts retrievalVectors are only comparable within the space of the model that produced them. Swapping the query-side model against a cloud-embedded index either fails with a dimension mismatch (the lucky case) or, worse, succeeds and returns semantically meaningless neighbours with no error at all.
Architectural Topology: Failing vs Remediated
Component Failing configuration (current) Remediated configuration (fix) Query embedding Cloud round trip per query — transport-dominated On-device via Foundry Local, single-digit ms Speech-to-text Raw audio streamed to a cloud endpoint On-device ASR — audio never leaves the device Embedding model parity Unconsidered — the trap Same model both sides, enforced and tested Quota exposure Embeddings consume shared tenant quota; throttle first Zero quota consumption — no 429 possible Offline behaviour Total failure — DNS resolution error Fully functional; cloud used only for generation Answer generation Cloud (correct) Cloud (unchanged — this belongs there) Failure mode Hard dependency on connectivity Edge-first with a cloud fallback path
06Fix 1 — Deploy Foundry Local on Linux ARM64RemediationThis is the capability that makes the whole architecture viable. Before 1.2.0, the Python SDK simply had no aarch64 wheel — installing on a Jetson failed outright with a platform resolution error. Foundry Local 1.2.0 ships linux-arm64, covering Raspberry Pi 5, NVIDIA Jetson, AWS Graviton, and Ampere.
What this used to fail with (pre-1.2, on aarch64)error: Distribution `foundry-local-core==1.0.0 @ registry+https://pypi.org/simple`
can't be installed because it doesn't have a source distribution or wheel
for the current platform
hint: You're on Linux (`manylinux_2_35_aarch64`), but `foundry-local-core`
(v1.0.0) only has wheels for the following platforms:
`manylinux_2_28_x86_64`, `macosx_11_0_arm64`, `win_amd64`, `win_arm64`
Linux ARM64 — install the runtime and verify hardware detection# Grab the linux-arm64 build from the v1.2.0 release, extract, and verify. tar xzf foundry-linux-arm64.tar.gz ./foundry/foundry --version # Confirm the service is up and see what hardware it detected. # Foundry Local auto-selects the best execution provider (NPU / GPU / CPU). foundry status # Browse the curated catalog foundry model list # Pull the embedding model and load it into memory foundry model load qwen3-0.6b-embedding # Most commands accept --output json, which makes them scriptable foundry status --output jsonInstall the SDK (Python) — now resolves on aarch64# 1.2.0+ publishes aarch64 wheels. Pin the version explicitly. pip install "foundry-local-sdk>=1.2.0" # Verify the wheel resolved for your platform python -c "import foundry_local; print(foundry_local.__version__)"Pick the device for the workload, not the brochureA Raspberry Pi 5 is genuinely sufficient for text embeddings and streaming ASR — recall the ASR model uses low single-digit percent CPU. A Jetson earns its price only if you also need on-device vision or a local chat model where the GPU matters. Graviton or Ampere make sense when the "edge" is actually a regional cloud VM and you are chasing cost and quota isolation rather than physical proximity. Don't buy a Jetson to run a 0.6B embedding model.
07Fix 2 — Run Text Embeddings LocallyRemediationThe embedding client is a first-class surface across all SDKs. The lifecycle is always the same: get the model from the catalog, download it, load it, then get an embedding client. Responses follow the OpenAI embeddings format, which is deliberate — it makes cloud-to-edge portability nearly mechanical.
Python — single and batch embeddings, fully on-devicefrom foundry_local import FoundryLocalManager manager = FoundryLocalManager(app_name="edge-rag") # Download + load the embedding model. Cached after first run. model = manager.catalog.get_model("qwen3-0.6b-embedding") model.download() model.load() client = model.get_embedding_client() # Single embedding — this is the call you just deleted from the network response = client.generate_embedding("The quick brown fox jumps over the lazy dog") embedding = response.data[0].embedding print(f"Dimensions: {len(embedding)}") # Batch — use this for indexing. Far more efficient than looping. batch_response = client.generate_embeddings([ "Machine learning is a subset of artificial intelligence", "The capital of France is Paris", "Rust is a systems programming language", ])C# — the same lifecyclevar model = await catalog.GetModelAsync("qwen3-0.6b-embedding"); await model.DownloadAsync(); await model.LoadAsync(); var embeddingClient = await model.GetEmbeddingClientAsync(); // Single embedding var response = await embeddingClient.GenerateEmbeddingAsync( "The quick brown fox jumps over the lazy dog"); var embedding = response.Data[0].Embedding; Console.WriteLine($"Dimensions: {embedding.Count}"); // Batch embeddings var batchResponse = await embeddingClient.GenerateEmbeddingsAsync([ "Machine learning is a subset of artificial intelligence", "The capital of France is Paris", "Rust is a systems programming language" ]);
| Component | Failing configuration (current) | Remediated configuration (fix) |
|---|---|---|
| Query embedding | Cloud round trip per query — transport-dominated | On-device via Foundry Local, single-digit ms |
| Speech-to-text | Raw audio streamed to a cloud endpoint | On-device ASR — audio never leaves the device |
| Embedding model parity | Unconsidered — the trap | Same model both sides, enforced and tested |
| Quota exposure | Embeddings consume shared tenant quota; throttle first | Zero quota consumption — no 429 possible |
| Offline behaviour | Total failure — DNS resolution error | Fully functional; cloud used only for generation |
| Answer generation | Cloud (correct) | Cloud (unchanged — this belongs there) |
| Failure mode | Hard dependency on connectivity | Edge-first with a cloud fallback path |
This is the capability that makes the whole architecture viable. Before 1.2.0, the Python SDK simply had no aarch64 wheel — installing on a Jetson failed outright with a platform resolution error. Foundry Local 1.2.0 ships linux-arm64, covering Raspberry Pi 5, NVIDIA Jetson, AWS Graviton, and Ampere.
error: Distribution `foundry-local-core==1.0.0 @ registry+https://pypi.org/simple`
can't be installed because it doesn't have a source distribution or wheel
for the current platform
hint: You're on Linux (`manylinux_2_35_aarch64`), but `foundry-local-core`
(v1.0.0) only has wheels for the following platforms:
`manylinux_2_28_x86_64`, `macosx_11_0_arm64`, `win_amd64`, `win_arm64`A Raspberry Pi 5 is genuinely sufficient for text embeddings and streaming ASR — recall the ASR model uses low single-digit percent CPU. A Jetson earns its price only if you also need on-device vision or a local chat model where the GPU matters. Graviton or Ampere make sense when the "edge" is actually a regional cloud VM and you are chasing cost and quota isolation rather than physical proximity. Don't buy a Jetson to run a 0.6B embedding model.
The embedding client is a first-class surface across all SDKs. The lifecycle is always the same: get the model from the catalog, download it, load it, then get an embedding client. Responses follow the OpenAI embeddings format, which is deliberate — it makes cloud-to-edge portability nearly mechanical.
A fully local retrieval pipeline
08Fix 3 — Offline and Streaming Speech RecognitionRemediationThere are two distinct ASR paths, and they solve different problems. File transcription (Whisper) processes recorded audio. Live streaming (Nemotron) pushes raw PCM chunks and returns results as they arrive, with is_final markers distinguishing interim text from finalised text — which is what you need for captioning, voice UIs, and meeting transcription.
CLI — the fastest possible smoke test# Transcribe a local audio file. No account, no key, no network. foundry transcribe -m whisper-tiny -f audio.wavJavaScript — file transcription and real-time streamingimport { FoundryLocalManager } from 'foundry-local-sdk'; const manager = FoundryLocalManager.create({ appName: 'edge-asr' }); const whisperModel = await manager.catalog.getModel('whisper-tiny'); await whisperModel.download(); await whisperModel.load(); const audioClient = whisperModel.createAudioClient(); audioClient.settings.language = 'en'; // One-shot: transcribe a file const result = await audioClient.transcribe('recording.wav'); console.log('Transcription:', result.text); // Streaming: emit text as it is recognised for await (const chunk of audioClient.transcribeStreaming('recording.wav')) { process.stdout.write(chunk.text); } // Free the memory when you're done — important on a Pi. await whisperModel.unload();Choose the right ASR model for the jobUse whisper-tiny for transcribing files. For live microphone streaming, use the purpose-built streaming model — nemotron-speech-streaming-en-0.6b for English, or the Nemotron 3.5 multilingual variant in 1.2 if you need any of the 40+ supported languages. Whisper is not designed for low-latency streaming; using it for a live voice UI is the most common mistake here and it will feel sluggish.
Cancel in-flight work — new in 1.2Foundry Local 1.2 added inference cancellation: you can cleanly cancel in-flight chat completions and transcription sessions when the user moves on, with no wasted compute or orphaned streams. On a constrained edge device this matters more than it does on a server — an abandoned transcription session holding CPU on a Pi is a real cost, not a rounding error. Wire cancellation into your UI's "stop" affordance from day one.
09Fix 4 — Hybrid Fallback (and Where This Is the Wrong Tool)Honest LimitsA hybrid architecture needs a defined behaviour when the local model is unavailable — the device is still downloading it, the model failed to load, or you are running on hardware that cannot host it. The pattern is edge-first with a cloud fallback, and it comes with a sharp caveat.
Python — edge-first embedding with a guarded cloud fallbackclass HybridEmbedder: """Edge-first. Falls back to cloud ONLY if the vector spaces match. CRITICAL: falling back to a DIFFERENT embedding model silently corrupts retrieval (see Section 4). If the local and cloud models differ, a fallback is NOT safe — failing the request is the correct behaviour. """ def __init__(self, local_client, cloud_client, *, spaces_match: bool): self.local = local_client self.cloud = cloud_client self.spaces_match = spaces_match # must be explicitly asserted def embed(self, text: str) -> list[float]: try: return self.local.generate_embedding(text).data[0].embedding except LocalModelUnavailable: if not self.spaces_match: # Fail LOUDLY. A wrong vector is worse than no vector. raise RuntimeError( "Local embedding model unavailable and cloud model uses a " "different vector space — refusing to corrupt retrieval." ) logger.warning("edge_embed_unavailable: falling back to cloud") return self.cloud.embeddings.create( model="text-embedding-3-large", input=text ).data[0].embedding⚠ Foundry Local is NOT a server inference stackThis is the misuse to avoid, and Microsoft is explicit about it. Foundry Local is optimised for hardware-constrained devices where a single user accesses the model at a time. You can technically run it on server hardware, but it is not designed as a shared inference backend. It does not do concurrent request queuing, continuous batching, or efficient GPU sharing across many simultaneous clients.
So: do not stand up one Foundry Local box and point your whole fleet at it as a shared "embedding microservice." That is exactly the workload it is not built for, and you will rediscover every queueing problem you were trying to escape. If you need a multi-user local inference server, use a server-oriented runtime such as vLLM or Triton. Foundry Local's model is one runtime per device.
Prefer the SDK over the local serverFoundry Local ships an optional OpenAI-compatible local web server, which is genuinely useful for experimenting via REST or integrating with tools that expect an HTTP endpoint. But for an embedded application, use the SDK directly — it runs inference in-process, without the overhead of a separate server process. Reaching for the HTTP server inside your own app reintroduces a network hop you just spent this entire article removing.
One more limit worth stating: the model catalog is deliberately curated, not a general-purpose model zoo. It exists so that every model in it is quantised, tested across consumer hardware, and small enough to ship inside an application. If your plan depends on running an arbitrary Hugging Face checkpoint, Foundry Local is the wrong runtime and you want ONNX Runtime or llama.cpp directly.
There are two distinct ASR paths, and they solve different problems. File transcription (Whisper) processes recorded audio. Live streaming (Nemotron) pushes raw PCM chunks and returns results as they arrive, with is_final markers distinguishing interim text from finalised text — which is what you need for captioning, voice UIs, and meeting transcription.
Use whisper-tiny for transcribing files. For live microphone streaming, use the purpose-built streaming model — nemotron-speech-streaming-en-0.6b for English, or the Nemotron 3.5 multilingual variant in 1.2 if you need any of the 40+ supported languages. Whisper is not designed for low-latency streaming; using it for a live voice UI is the most common mistake here and it will feel sluggish.
Foundry Local 1.2 added inference cancellation: you can cleanly cancel in-flight chat completions and transcription sessions when the user moves on, with no wasted compute or orphaned streams. On a constrained edge device this matters more than it does on a server — an abandoned transcription session holding CPU on a Pi is a real cost, not a rounding error. Wire cancellation into your UI's "stop" affordance from day one.
A hybrid architecture needs a defined behaviour when the local model is unavailable — the device is still downloading it, the model failed to load, or you are running on hardware that cannot host it. The pattern is edge-first with a cloud fallback, and it comes with a sharp caveat.
This is the misuse to avoid, and Microsoft is explicit about it. Foundry Local is optimised for hardware-constrained devices where a single user accesses the model at a time. You can technically run it on server hardware, but it is not designed as a shared inference backend. It does not do concurrent request queuing, continuous batching, or efficient GPU sharing across many simultaneous clients.
So: do not stand up one Foundry Local box and point your whole fleet at it as a shared "embedding microservice." That is exactly the workload it is not built for, and you will rediscover every queueing problem you were trying to escape. If you need a multi-user local inference server, use a server-oriented runtime such as vLLM or Triton. Foundry Local's model is one runtime per device.
Foundry Local ships an optional OpenAI-compatible local web server, which is genuinely useful for experimenting via REST or integrating with tools that expect an HTTP endpoint. But for an embedded application, use the SDK directly — it runs inference in-process, without the overhead of a separate server process. Reaching for the HTTP server inside your own app reintroduces a network hop you just spent this entire article removing.
One more limit worth stating: the model catalog is deliberately curated, not a general-purpose model zoo. It exists so that every model in it is quantised, tested across consumer hardware, and small enough to ship inside an application. If your plan depends on running an arbitrary Hugging Face checkpoint, Foundry Local is the wrong runtime and you want ONNX Runtime or llama.cpp directly.
Validation & Verification: Confirm the Fix
Two things must be proven, and the second is the one people skip. One: the latency actually dropped. Two: retrieval quality did not silently degrade. A migration that halves latency while quietly wrecking recall is not a win — it is a regression with good telemetry.
Step 1 — Benchmark local vs cloud on the target device (run it ON the Pi)import time, statistics from foundry_local import FoundryLocalManager manager = FoundryLocalManager(app_name="bench") model = manager.catalog.get_model("qwen3-0.6b-embedding") model.download(); model.load() local = model.get_embedding_client() query = "why did the turbine alarm at 14:02?" # WARM UP first — the first call includes model warm-up, not steady state. local.generate_embedding(query) def bench(fn, n=100): times = [] for _ in range(n): t = time.perf_counter() fn() times.append((time.perf_counter() - t) * 1000) times.sort() return { "p50": round(statistics.median(times), 1), "p95": round(times[int(n * 0.95)], 1), "p99": round(times[int(n * 0.99)], 1), } print("LOCAL:", bench(lambda: local.generate_embedding(query))) print("CLOUD:", bench(lambda: cloud.embeddings.create( model="text-embedding-3-large", input=query))) # PASS: local p99 collapses toward local p50 — no queue, no throttle. # The p99 delta should be far larger than the p50 delta. That is the tail win.Step 2 — THE important one: prove retrieval quality did not regress# A golden set of (query -> document that SHOULD be retrieved) pairs. # Build this from real user queries. 50 pairs is enough to catch a regression. GOLDEN = [ ("why did the turbine alarm?", "d0"), ("when is pump maintenance due?", "d1"), ("is coolant pressure ok?", "d2"), ] def recall_at_k(embed_client, store, k=3) -> float: hits = 0 for query, expected_id in GOLDEN: v = embed_client.generate_embedding(query).data[0].embedding result = store.query(query_embeddings=[v], n_results=k) if expected_id in result["ids"][0]: hits += 1 return hits / len(GOLDEN) print("recall@3 (cloud model):", recall_at_k(cloud_embed, cloud_store)) print("recall@3 (local model):", recall_at_k(local_embed, local_store)) # PASS: local recall is within an acceptable margin of cloud recall. # FAIL: a meaningful drop means the 0.6B model is not good enough for your # corpus. DO NOT ship the latency win at the cost of retrieval accuracy.Step 3 — Prove it is genuinely offline (the negative test)# The decisive test. Sever the network and confirm the edge path still works. # If embedding/ASR still succeed with no route to the internet, they are # truly local. If they fail, something is still calling out. sudo ip link set eth0 down # or: sudo nmcli networking off foundry transcribe -m whisper-tiny -f audio.wav python -c " from foundry_local import FoundryLocalManager m = FoundryLocalManager(app_name='offline-test') model = m.catalog.get_model('qwen3-0.6b-embedding'); model.load() print('dims:', len(model.get_embedding_client() .generate_embedding('offline check').data[0].embedding)) " sudo ip link set eth0 up # PASS: both succeed with the network down. (Model must be cached already — # the FIRST download obviously needs connectivity.)What "fixed" actually means hereSuccess is three conditions met together, not one. First, the embedding p99 collapses — because you removed the queue, not because you found a faster model. Second, recall@k holds against your golden set; if the smaller model retrieves worse, you have traded correctness for speed and that is a bad trade. Third, the edge path genuinely works with the network severed. A migration that satisfies the first condition but not the second is the single most common way this goes wrong, and it produces an application that is fast, confident, and subtly incorrect.
Two things must be proven, and the second is the one people skip. One: the latency actually dropped. Two: retrieval quality did not silently degrade. A migration that halves latency while quietly wrecking recall is not a win — it is a regression with good telemetry.
Success is three conditions met together, not one. First, the embedding p99 collapses — because you removed the queue, not because you found a faster model. Second, recall@k holds against your golden set; if the smaller model retrieves worse, you have traded correctness for speed and that is a bad trade. Third, the edge path genuinely works with the network severed. A migration that satisfies the first condition but not the second is the single most common way this goes wrong, and it produces an application that is fast, confident, and subtly incorrect.
Key Takeaways
For embeddings, the transport is the workload. A 0.6B model over twelve words is trivial compute wrapped in a full internet round trip. You are not making the call faster by moving it local — you are deleting the call.Linux ARM64 in Foundry Local 1.2 is the unlock. Raspberry Pi 5, Jetson, Graviton, and Ampere are now first-class targets. Before 1.2 there was no aarch64 wheel at all.The embedding-space trap is the real danger, not latency. Documents and queries must be embedded by the same model. Swap only the query side and you either get a 400 — or, far worse, silently meaningless retrieval with no error at all.Split by proximity vs scale. Embeddings, ASR, and classification go to the edge. Final answer generation stays in the cloud, where model scale genuinely is the product.Local ASR is a privacy migration disguised as a performance one. Raw audio never leaves the device — often a stronger argument for a regulated workload than the latency win.Foundry Local is not a server inference stack. It is built for one user per device. Do not turn it into a shared embedding microservice — that is what vLLM and Triton are for.Measure recall, not just milliseconds. A smaller embedding model may retrieve worse on your corpus. Prove it doesn't, with a golden set, before you re-index production.
Frequently Asked Questions
Can I just point my existing Azure AI Search index at local embeddings?No — and this is the most dangerous mistake in the whole migration. Embeddings are only meaningful within the vector space of the model that produced them, so if your index was built with text-embedding-3-large and you start embedding queries with qwen3-0.6b-embedding, the vectors are not comparable. If the dimensions differ you will get a clean HTTP 400 dimension-mismatch error, which is actually the lucky outcome. If the dimensions happen to match, the query succeeds and returns semantically irrelevant documents with no error at all — and because the LLM stays fluent on bad context, you ship confident nonsense. You have exactly two safe options: re-embed the entire corpus with the local model, or use local embeddings only for a separate, edge-resident index.Will a 0.6B embedding model retrieve as well as a large cloud model?Maybe — and "maybe" is not good enough to ship on. It is frequently adequate, especially for a narrow domain corpus, but it is an empirical question about your data, not a general truth. Before re-indexing anything in production, build a golden set of query-to-expected-document pairs drawn from real user queries and measure recall@k with both models. If local recall holds, take the latency win. If it drops meaningfully, you are trading correctness for speed, which is a bad deal no matter how good the p99 chart looks. The validation section of this guide includes a script for exactly this measurement.Can I run Foundry Local as a shared embedding server for my whole fleet?You shouldn't. Microsoft is explicit that Foundry Local is optimised for hardware-constrained devices where a single user accesses the model at a time, and that while you can technically install it on server hardware, it isn't designed as a server inference stack. It does not handle concurrent request queuing, continuous batching, or efficient GPU sharing across many simultaneous clients. Standing up one box as a shared "embedding microservice" reintroduces exactly the queueing and contention you moved to the edge to escape. If you need multi-user local inference, use a server-oriented runtime such as vLLM or Triton. The Foundry Local model is one runtime per device.Is a Raspberry Pi 5 really enough for this?For text embeddings and speech recognition, yes. The embedding model is only 0.6B parameters, and the streaming ASR model was aggressively optimised for exactly this class of hardware — Microsoft re-implemented the pipeline in ONNX Runtime and quantised it from 2.47GB down to as little as 0.67GB while keeping word error rate within 1% of the full-precision baseline, reporting roughly 8% WER at low single-digit percent CPU usage. Foundry Local also auto-detects available hardware and picks the best execution provider. Where a Pi is not enough is if you also want to run a local chat model or on-device vision — that is when a Jetson starts to earn its cost. Don't buy accelerator hardware to run a 0.6B embedding model.
Related FAVRITE Articles
- How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
- Fixing the First-Request Lag: Azure Functions and Container Apps for AI Microservices
- Azure OpenAI to Microsoft Foundry: Fixing Private Endpoint & DNS Failures
- Azure AI Search RAG Runbook
- How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
- Fixing the First-Request Lag: Azure Functions and Container Apps for AI Microservices
- Azure OpenAI to Microsoft Foundry: Fixing Private Endpoint & DNS Failures
- Azure AI Search RAG Runbook