Optimize Azure AI Search for enterprise RAG with hybrid search, chunking, document-level security, agentic retrieval, observability, and Terraform-based deployment.
Architectural Runbook: Optimizing Azure AI Search for Enterprise RAG Workflows
A basic RAG PoC takes one afternoon to build. A production RAG system that retrieves accurately at enterprise scale, enforces document-level security, handles complex multi-step queries, and stays observable under load takes considerably more. This runbook covers every layer — from hybrid search tuning and chunking strategy to agentic retrieval, security filters, and Terraform IaC — in the depth that production demands.
The Enterprise RAG Architecture: Components and Data Flow
Retrieval-Augmented Generation grounds a large language model in your organisation's proprietary data by retrieving relevant document chunks at query time and injecting them into the LLM's context window. The quality of the generated answer is entirely determined by the quality of the retrieved chunks. A better generation model cannot compensate for poor retrieval. This is the central insight that separates PoC RAG implementations from production ones: the effort belongs in the retrieval layer, not the generation layer.
Azure AI Search is Microsoft's managed retrieval engine for enterprise RAG. It provides three retrieval mechanisms in a single service: full-text keyword search using the BM25 algorithm, dense vector search using the Hierarchical Navigable Small World (HNSW) graph algorithm, and transformer-based semantic re-ranking that re-scores the top results using a cross-encoder model. The combination of all three — applied in sequence to every query — is what makes enterprise-grade retrieval achievable.
Why hybrid over pure vector: Vector search captures semantic meaning — it understands that "cardiac arrest" and "heart attack" refer to the same condition. But it fails at exact matching: a product serial number like "AX-2047-C" or an acronym like "PAYE" returns poor results because these tokens have no meaningful vector neighbourhood. BM25 keyword search excels at exact term matching but misses semantic relationships. Hybrid search runs both simultaneously and merges the results using Reciprocal Rank Fusion (RRF) — a rank combination algorithm that does not require scores from the two systems to be on the same scale.
The RRF formula: For each document d in the merged result, the RRF score is the sum of 1/(k + rank_i) across all result lists i where the document appears. The constant k (default 60) controls how much influence rank position has — higher k flattens differences between rank positions, lower k amplifies them. In practice, the default of 60 performs well across most enterprise document retrieval scenarios.
Define the index schema with both text and vector fields
The index must have a field marked searchable: true for BM25 keyword search and a type: Collection(Edm.Single) field configured for vector search. Both must point to the same source content (the chunk text) — one stores the raw text, the other stores its embedding.
Configure the HNSW vector algorithm profile
Set m (max connections per node, default 4) and efConstruction (build-time search width, default 400). Higher m and efConstruction improve recall at the cost of index build time and memory. For production: m=16, efConstruction=800 is a well-balanced starting point. The metric should be cosine for text embeddings from OpenAI models.
Issue hybrid queries with both search text and vector
Set queryType: "semantic", include both the search (text) parameter and vectorQueries parameter in the same request. Azure AI Search automatically applies RRF fusion when both are present.
Set vector_semantic_hybrid as your query mode
This is Microsoft's own recommended default from their internal benchmarking. It activates hybrid retrieval (keyword + vector) followed by semantic re-ranking in a single query call — the optimal configuration for enterprise RAG without any further tuning.
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
SearchIndex, SearchField, SearchFieldDataType,
VectorSearch, HnswAlgorithmConfiguration, HnswParameters,
VectorSearchProfile, SemanticConfiguration, SemanticSearch,
SemanticPrioritizedFields, SemanticField
)
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential() # Managed Identity — no keys
endpoint = "https://YOUR-SEARCH.search.windows.net"
INDEX_NAME = "enterprise-rag-index"
# ── INDEX SCHEMA ─────────────────────────────────────────
fields = [
SearchField(name="id", type=SearchFieldDataType.String, key=True),
SearchField(name="chunk_text", type=SearchFieldDataType.String,
searchable=True, analyzer_name="en.microsoft"), # BM25
SearchField(name="embedding",
type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
searchable=True, vector_search_dimensions=3072, # text-embedding-3-large
vector_search_profile_name="hnsw-profile"),
SearchField(name="source_url", type=SearchFieldDataType.String, retrievable=True),
SearchField(name="department", type=SearchFieldDataType.String, filterable=True),
SearchField(name="allowed_groups", # Security field — see Runbook 5
type=SearchFieldDataType.Collection(SearchFieldDataType.String),
filterable=True),
]
vector_search = VectorSearch(
algorithms=[HnswAlgorithmConfiguration(name="hnsw-algo",
parameters=HnswParameters(
m=16, # Max connections per HNSW node
ef_construction=800, # Build-time search width (recall vs speed)
metric="cosine" # Use cosine for OpenAI embeddings
))],
profiles=[VectorSearchProfile(name="hnsw-profile", algorithm_configuration_name="hnsw-algo")]
)
# ── HYBRID QUERY ─────────────────────────────────────────
search_client = SearchClient(endpoint, INDEX_NAME, credential)
def hybrid_search(query_text: str, query_embedding: list, top_k: int = 5,
security_filter: str = None):
vector_query = VectorizedQuery(
vector=query_embedding,
k_nearest_neighbors=top_k * 3, # Over-retrieve for re-ranking
fields="embedding"
)
results = search_client.search(
search_text=query_text, # BM25 keyword search
vector_queries=[vector_query], # HNSW vector search
query_type="semantic", # Activates RRF + semantic re-ranking
semantic_configuration_name="my-semantic-config",
query_caption="extractive", # Extract captions from top results
filter=security_filter, # OData security filter (Runbook 5)
top=top_k,
select="chunk_text,source_url"
)
return [r for r in results]
Chunking is the single decision with the largest impact on retrieval quality. Chunks that are too small lack context for the semantic ranker to score accurately. Chunks that are too large dilute the relevance score with irrelevant content and consume excessive context window space. The correct chunk size is not a universal constant — it depends on your content type, your embedding model's token limit, and the specificity of queries your users ask.
Start with 512 tokens + 128 overlap as the default
This configuration works well for general enterprise documents: policy documents, HR guides, product documentation, internal wikis. The 128-token overlap ensures that information at chunk boundaries is captured in both adjacent chunks, preventing the "boundary problem" where an answer is split across two chunks and neither scores high enough independently.
Adjust for content type
Code repositories: increase to 1,024 tokens (functions and classes should not be split mid-definition). Legal contracts and medical records: use semantic boundary chunking based on sections and clauses — preserve complete regulatory provisions. FAQ databases: keep each Q+A pair as a single chunk regardless of length. Tabular data: one row per chunk with the header row repeated in each chunk.
Preserve metadata in every chunk
Every chunk document should include: the source document title, the source URL, the section heading, the page number, a document timestamp, and any access control metadata (department, user groups). This metadata is critical for generating cited answers and for security filtering. A retrieved chunk with no provenance metadata is unusable in a production system.
Measure retrieval quality before deploying chunk size changes
Build an evaluation set of 50–100 representative questions with known correct source documents. Run retrieval with different chunk sizes and measure Recall@5 (is the correct document in the top 5 results?) and MRR (Mean Reciprocal Rank). Do not change chunk size in production without evaluation data showing the new size outperforms the current one.
from langchain_text_splitters import RecursiveCharacterTextSplitter
import tiktoken
def chunk_document(text: str, source_metadata: dict,
chunk_size: int = 512, overlap: int = 128) -> list[dict]:
enc = tiktoken.get_encoding("cl100k_base") # OpenAI tokenizer
splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=overlap,
length_function=lambda t: len(enc.encode(t)), # Token-based, not char-based
separators=["\n\n", "\n", ". ", " ", ""] # Prefer natural boundaries
)
chunks = splitter.split_text(text)
return [
{
"id": f"{source_metadata['doc_id']}-chunk-{i}",
"chunk_text": chunk,
"token_count": len(enc.encode(chunk)),
"chunk_index": i,
"source_url": source_metadata["url"],
"source_title": source_metadata["title"],
"department": source_metadata.get("department", "general"),
"allowed_groups": source_metadata.get("allowed_groups", ["all"]),
"embedding": None # Populated by vectorization step
}
for i, chunk in enumerate(chunks)
]
Semantic Ranking is Azure AI Search's cross-encoder re-ranking layer. After BM25 and HNSW return their merged RRF result set, the Semantic Ranker takes the top 50 results and passes each one through a transformer cross-encoder model that evaluates the full (query, document) pair — far more computationally expensive than the initial retrieval, but dramatically more accurate at distinguishing between plausibly relevant documents. The ranker also generates extractive captions — highlighted sentences that best answer the query — which are the ideal grounding material for the LLM's response generation step.
Semantic Ranking requires Standard tier or above. It is available as an add-on on Basic tier with limitations. The additional latency is typically 100–300ms per query — well worth the precision improvement for enterprise document retrieval where answer quality is the primary metric.
Define a Semantic Configuration in the index
The semantic configuration specifies which fields the ranker prioritises: the title field (highest weight), content fields (main body text), and keyword fields (metadata). The ranker uses this priority ordering to score the relevance of each document to the query.
Enable query captions and answers
Set queryCaption: "extractive" to get highlighted text snippets from the most relevant passage in each result. Set queryAnswer: "extractive" to get a direct answer extracted from the top result if the ranker identifies one. Pass these captions to the LLM instead of the full chunk text to reduce context window consumption by 60–80%.
Tune the rerankerScore threshold
Every result from Semantic Ranking includes a @search.reranker_score between 0 and 4. A score below 1.5 typically indicates a result that is marginally relevant at best. Filter out results below your threshold before passing context to the LLM — this reduces hallucination risk from weakly relevant context and reduces token consumption.
from azure.search.documents.indexes.models import (
SemanticConfiguration, SemanticSearch,
SemanticPrioritizedFields, SemanticField
)
semantic_config = SemanticConfiguration(
name="my-semantic-config",
prioritized_fields=SemanticPrioritizedFields(
title_field=SemanticField(field_name="source_title"),
content_fields=[SemanticField(field_name="chunk_text")],
)
)
semantic_search = SemanticSearch(configurations=[semantic_config])
# ── QUERY WITH CAPTIONS + RERANKER SCORE FILTERING ───────
RERANKER_THRESHOLD = 1.5 # Exclude weakly relevant results
results = search_client.search(
search_text=query_text,
vector_queries=[vector_query],
query_type="semantic",
semantic_configuration_name="my-semantic-config",
query_caption="extractive|highlight-false", # Clean text, no HTML tags
query_answer="extractive|count-1", # Extract best direct answer if present
top=10, select="chunk_text,source_url,source_title"
)
# Build LLM context from captions, filtered by reranker score
context_chunks = []
for result in results:
if result.get("@search.reranker_score", 0) < RERANKER_THRESHOLD:
continue # Skip low-confidence results
captions = result.get("@search.captions", [])
caption_text = captions[0].text if captions else result["chunk_text"][:500]
context_chunks.append({
"text": caption_text,
"source": result["source_url"],
"title": result["source_title"],
"score": result["@search.reranker_score"]
})
return context_chunks
Integrated Vectorization allows Azure AI Search to generate embeddings automatically during ingestion — without any custom code to call the embedding API, handle batching, manage rate limits, or store vectors. You configure an Azure OpenAI embedding skill in the indexer's skillset, and the indexer handles chunking, embedding generation, and index population in a single managed pipeline. At query time, you can also configure the index to vectorize query text automatically using the same embedding model, eliminating the need to call the embedding API in your application code.
Deploy text-embedding-3-large in Azure AI Foundry
In the Azure AI Foundry portal, deploy the text-embedding-3-large model (3,072 dimensions — Microsoft's recommended embedding model for production RAG). Note the endpoint URL and deployment name. Configure Managed Identity on the Azure AI Search service and assign "Cognitive Services OpenAI User" role on the Azure OpenAI resource.
Create an Azure OpenAI skill in the indexer skillset
Add an AzureOpenAIEmbeddingSkill to the indexer's skillset. Configure it to receive the chunk text and output the embedding vector. The skill handles batching, retry on rate limits, and dimension validation automatically.
Configure vectorizer on the index for query-time embedding
Add a vectorizer to the HNSW profile pointing at the same Azure OpenAI deployment. When a search query arrives with no pre-computed vector, Azure AI Search calls the embedding API automatically and runs the vector search without the application needing to generate the query embedding.
Use change detection for incremental index updates
Configure the indexer to run on a schedule (e.g., every 15 minutes) with high-watermark change detection enabled on Blob Storage. The indexer only processes documents added or modified since the last run — avoiding the cost of re-embedding unchanged documents. For SharePoint and SQL sources, native change tracking achieves the same result.
SearchIndexerSkillset, AzureOpenAIEmbeddingSkill,
InputFieldMappingEntry, OutputFieldMappingEntry,
SplitSkill, SearchIndexer, SearchIndexerDataSourceConnection
)
# ── SKILLSET WITH CHUNK + EMBED SKILLS ───────────────────
split_skill = SplitSkill(
name="chunk-skill",
text_split_mode="pages",
maximum_page_length=512,
page_overlap_length=128,
inputs=[InputFieldMappingEntry(name="text", source="/document/content")],
outputs=[OutputFieldMappingEntry(name="textItems",
target_name="pages")]
)
embedding_skill = AzureOpenAIEmbeddingSkill(
name="embed-skill",
resource_uri="https://YOUR-OPENAI.openai.azure.com",
deployment_name="text-embedding-3-large",
model_name="text-embedding-3-large",
dimensions=3072,
inputs=[InputFieldMappingEntry(name="text", source="/document/pages/*")],
outputs=[OutputFieldMappingEntry(name="embedding",
target_name="embedding")]
)
skillset = SearchIndexerSkillset(
name="rag-skillset",
skills=[split_skill, embedding_skill],
description="Chunk and embed documents for RAG"
)
Enterprise RAG deployments typically have strict information security requirements: the HR chatbot should only retrieve documents the querying user is authorised to read. Without document-level security filters, every user can retrieve every document in the index — including documents from other departments, restricted projects, or regulated data categories. Azure AI Search implements this through OData filter expressions applied at query time.
The pattern: store each document's access control list (the group or user identifiers who can read it) in a filterable field in the index. At query time, your application resolves the current user's group memberships from Microsoft Entra ID, constructs an OData filter expression that limits results to documents in those groups, and passes it to every search call. The filter is applied before ranking — no restricted documents appear in any result set.
Add an allowed_groups field to the index (filterable, not searchable)
Define a Collection(Edm.String) field named allowed_groups with filterable: true but searchable: false. This field stores the list of Entra group object IDs that are permitted to read this document. Populate it during ingestion from the source document's ACL metadata.
Resolve user's group memberships at query time
Use the Microsoft Graph API to retrieve the current user's transitive group memberships: GET /me/transitiveMemberOf/microsoft.graph.group?$select=id. Cache the result per user session (group memberships rarely change mid-session) with a 15-minute TTL. Never hard-code group IDs in application code — always resolve dynamically from the authenticated identity.
Construct and apply the OData security filter on every query
Build the filter expression from the user's group list and pass it to the filter parameter of every search call. The filter must be applied at the Azure AI Search level — never filter results in application code after retrieval, because the vector search top-k calculation will have already excluded potentially-accessible documents.
from azure.identity.aio import OnBehalfOfCredential
async def get_user_groups(user_access_token: str) -> list[str]:
# Use On-Behalf-Of flow to call Graph API as the user
credential = OnBehalfOfCredential(
tenant_id=TENANT_ID, client_id=APP_CLIENT_ID,
client_secret=APP_CLIENT_SECRET,
user_assertion=user_access_token
)
token = await credential.get_token("https://graph.microsoft.com/.default")
async with aiohttp.ClientSession() as session:
resp = await session.get(
"https://graph.microsoft.com/v1.0/me/transitiveMemberOf"
"/microsoft.graph.group?$select=id",
headers={"Authorization": f"Bearer {token.token}"}
)
data = await resp.json()
return [g["id"] for g in data.get("value", [])]
def build_security_filter(group_ids: list[str]) -> str:
# OData: allowed_groups contains any of the user's group IDs
if not group_ids:
return "allowed_groups/any(g: g eq 'all')" # Public docs only
conditions = [f"allowed_groups/any(g: g eq '{gid}')" for gid in group_ids]
conditions.append("allowed_groups/any(g: g eq 'all')") # Public docs
return " or ".join(conditions)
# Usage in search call
user_groups = await get_user_groups(request.headers["Authorization"].split()[1])
security_filter = build_security_filter(user_groups)
results = hybrid_search(query_text, query_embedding, security_filter=security_filter)
Standard RAG retrieval uses the user's query directly as the search input. This works well for simple factual questions but breaks down on complex, multi-part queries: "Summarise the Q3 revenue trend and explain how the APAC supply chain disruption contributed to the shortfall compared to the same period last year." This query requires multiple retrieval passes — one for Q3 revenue data, one for APAC supply chain events, one for the prior year comparison — before the LLM can synthesise a coherent answer.
Agentic Retrieval (GA in Azure AI Foundry 2026) solves this by having an LLM decompose complex queries into sub-queries, execute each independently against Azure AI Search, and then synthesise the retrieved chunks into a complete context before the final generation step. The pattern is sometimes called "query planning" or "multi-hop RAG."
Detect whether a query requires multi-hop retrieval
Use a lightweight classifier (GPT-5-nano or a rule-based check for conjunctions and temporal comparisons) to route queries. Simple queries go directly to standard hybrid search. Complex queries (containing "and", "compared to", "explain both", multiple named entities) go to the agentic retrieval path.
Use an LLM to decompose the complex query into sub-queries
Prompt GPT-4.1-mini with the original query and ask it to return a JSON list of 2–4 independent search sub-queries that together would retrieve all information needed to answer the original question. Each sub-query should be independently executable against Azure AI Search.
Execute sub-queries in parallel against Azure AI Search
Run all sub-queries simultaneously using asyncio.gather() — not sequentially. Parallelism keeps total latency close to a single-hop query while retrieving multiple times the context. Deduplicate results by chunk ID before passing to the final generation step.
Synthesise retrieved context and generate the final response
Pass all deduplicated chunks from all sub-queries as context to the final generation LLM (GPT-4.1). Structure the prompt to include which sub-query each chunk came from — this helps the LLM organise multi-part answers correctly and attribute each claim to the relevant source.
from openai import AzureOpenAI
DECOMPOSE_PROMPT = """You are a search query planner for a RAG system.
Given the user's question, decompose it into 2-4 independent search sub-queries.
Return ONLY a JSON array of strings. Example: ["query 1", "query 2"]
User question: {question}"""
async def agentic_retrieval(question: str, user_groups: list,
openai_client, embed_fn, search_fn):
# Step 1: Decompose the query
decompose_resp = openai_client.chat.completions.create(
model="gpt41-mini", # Cheap model for planning
messages=[{"role": "user",
"content": DECOMPOSE_PROMPT.format(question=question)}],
max_tokens=200, temperature=0
)
sub_queries = json.loads(decompose_resp.choices[0].message.content)
# Step 2: Embed and search all sub-queries in parallel
security_filter = build_security_filter(user_groups)
async def search_subquery(sq: str):
embedding = embed_fn(sq) # text-embedding-3-large
return sq, search_fn(sq, embedding, top_k=3,
security_filter=security_filter)
results = await asyncio.gather(*[search_subquery(sq) for sq in sub_queries])
# Step 3: Deduplicate by chunk ID
seen, context_blocks = set(), []
for sq, chunks in results:
for chunk in chunks:
if chunk["id"] not in seen:
seen.add(chunk["id"])
context_blocks.append({"sub_query": sq, "chunk": chunk})
return sub_queries, context_blocks
A RAG system that works in testing and fails silently in production is worse than one that fails loudly — silent failures produce confident wrong answers, which erode user trust faster than obvious errors. Observability in enterprise RAG requires two tracks: infrastructure metrics (latency, error rates, index size) and retrieval quality metrics (reranker scores, cache hit rates, query latency distribution).
| Tier | Max Indexes | Max Index Size | Semantic Ranking | Recommended For |
|---|---|---|---|---|
| Free | 3 | 50MB | No | PoC / evaluation only |
| Basic | 15 | 2GB/partition | Add-on | Small pilot deployments under 10k documents |
| Standard S1 | 50 | 25GB/partition | Included | Production: 10k–500k documents |
| Standard S2 | 200 | 100GB/partition | Included | Production: 500k–5M documents |
| Storage Optimised L1/L2 | 10 | 2TB/partition | Included | Large-corpus RAG: 5M+ documents, archive search |
Enable diagnostic logging to Log Analytics
Portal: Azure AI Search → Diagnostic settings → Add setting → send to Log Analytics workspace. Enable: OperationLogs, IndexingOperationsLogs, and QueryLogs. QueryLogs capture every search query with its latency, result count, and index name — the foundation for both performance monitoring and retrieval quality analysis.
Alert on P95 query latency above 2 seconds
Create an Azure Monitor alert on the SearchLatency metric (P95 over 5-minute windows). A P95 above 2,000ms indicates retrieval is adding perceptible delay to the user experience. Common causes: index too large for the tier (add replicas or partitions), excessive k_nearest_neighbors value, or semantic ranking applied to too many results.
Log reranker scores and monitor score distribution
In your application, log the @search.reranker_score for the top result of every query to Application Insights. Track the P25 reranker score over time. A declining P25 score (many queries returning low-confidence top results) indicates the index has become stale and needs re-indexing, or chunk size is misaligned with query patterns.
Tune efSearch for latency vs recall trade-off on self-managed vector profiles
If query latency exceeds targets, reduce efSearch (the runtime search width of the HNSW graph) in your vector search profile. The default efSearch is 500. Reducing it to 200–300 typically reduces query latency by 30–50% with only 2–5% recall reduction. This is the primary latency knob for HNSW-based vector search.
Every component of the enterprise RAG stack — Azure AI Search, the Azure OpenAI embedding deployment, storage accounts, private endpoints, and RBAC assignments — should be in Terraform. This enables repeatable deployment across environments (dev/staging/production), code-reviewed infrastructure changes, and disaster recovery from a known-good state.
resource "azurerm_search_service" "main" {
name = var.search_service_name
resource_group_name = var.resource_group_name
location = var.location
sku = "standard" # S1: semantic ranking included
replica_count = var.environment == "production" ? 2 : 1
partition_count = 1
identity {
type = "SystemAssigned" # For Managed Identity access to OpenAI
}
local_authentication_enabled = false # Force Entra auth only
public_network_access_enabled = var.environment == "production" ? false : true
}
# ── RBAC: Search MI can call Azure OpenAI for vectorization ─
resource "azurerm_role_assignment" "search_to_openai" {
scope = azurerm_cognitive_account.openai.id
role_definition_name = "Cognitive Services OpenAI User"
principal_id = azurerm_search_service.main.identity[0].principal_id
}
# ── AZURE OPENAI EMBEDDING DEPLOYMENT ────────────────────
resource "azurerm_cognitive_deployment" "embedding" {
name = "text-embedding-3-large"
cognitive_account_id = azurerm_cognitive_account.openai.id
model {
format = "OpenAI"
name = "text-embedding-3-large"
version = "1"
}
scale {
type = "Standard"
capacity = 120 # 120K TPM — adjust based on indexing volume
}
}
# ── PRIVATE ENDPOINT (Production) ────────────────────────
resource "azurerm_private_endpoint" "search" {
count = var.environment == "production" ? 1 : 0
name = "pe-${var.search_service_name}"
resource_group_name = var.resource_group_name
location = var.location
subnet_id = var.private_endpoint_subnet_id
private_service_connection {
name = "psc-search"
private_connection_resource_id = azurerm_search_service.main.id
is_manual_connection = false
subresource_names = ["searchService"]
}
}
Architecture Decision Summary
Frequently Asked Questions
Related FAVRITE Articles
- How to Fix Azure OpenAI Token Limits: Architectural Patterns for High-Throughput Apps
- Stop Using Connection Strings: A Step-by-Step Guide to Azure Managed Identities in 2026
- The Hidden Cloud Drain: How to Find and Kill Orphaned Azure Resources Automatically
- How Much Does Azure OpenAI Cost? Complete Pricing Guide (2026)