Compare Azure Data Lake Storage Gen2 vs Blob Storage for AI workloads. Learn architecture, performance, scalability, security, and cost tradeoffs guide.
Azure Data Lake Storage Gen2 vs. Blob Storage: Architectural Selection Guide for AI Workloads
Both services are built on the same infrastructure. Both charge the same per-GB storage rate. So why does choosing the wrong one slow your Spark jobs by orders of magnitude, break Delta Lake ACID commits, and make POSIX-style access control impossible? This guide answers the question that documentation rarely does: not what each service is, but precisely when to use which one — and how to provision it correctly for each AI workload pattern.
1 toggleThe only technical difference between Blob Storage and ADLS Gen2 is a single checkbox at account creation: "Enable hierarchical namespace." Everything else is inherited from Blob Storage.30,000API calls required to rename a directory of 10,000 files on Blob Storage (flat namespace). The same operation on ADLS Gen2: 1 atomic call.15–20%Additional transaction cost on ADLS Gen2 vs Blob Storage due to hierarchical namespace overhead — the premium that is nearly always justified for analytics workloads.IrreversibleOnce a storage account is created, you cannot change the hierarchical namespace setting. Get the decision right at provisioning time — migration requires copying all data to a new account.
Both services are built on the same infrastructure. Both charge the same per-GB storage rate. So why does choosing the wrong one slow your Spark jobs by orders of magnitude, break Delta Lake ACID commits, and make POSIX-style access control impossible? This guide answers the question that documentation rarely does: not what each service is, but precisely when to use which one — and how to provision it correctly for each AI workload pattern.
The One Technical Difference That Changes Everything
Azure Data Lake Storage Gen2 is not a separate service. It is Azure Blob Storage with one feature enabled: the hierarchical namespace. When you create a storage account in the Azure portal, you will find a checkbox on the Advanced tab: "Enable hierarchical namespace." Checking it turns your Blob Storage account into ADLS Gen2. Leaving it unchecked keeps it as standard Blob Storage. That single decision — made once, irrevocably, at account creation — is the entire structural difference between the two services.
Everything else — the storage tiers (Hot, Cool, Cold, Archive), the redundancy options (LRS, ZRS, GRS, RA-GRS), the lifecycle management policies, the encryption, the firewall rules, the private endpoints, the SDK compatibility — is identical. The per-GB storage pricing is also identical. The difference lies entirely in what the hierarchical namespace enables and in the transaction model that changes as a consequence.
Azure Data Lake Storage Gen2 is not a separate service. It is Azure Blob Storage with one feature enabled: the hierarchical namespace. When you create a storage account in the Azure portal, you will find a checkbox on the Advanced tab: "Enable hierarchical namespace." Checking it turns your Blob Storage account into ADLS Gen2. Leaving it unchecked keeps it as standard Blob Storage. That single decision — made once, irrevocably, at account creation — is the entire structural difference between the two services.
Everything else — the storage tiers (Hot, Cool, Cold, Archive), the redundancy options (LRS, ZRS, GRS, RA-GRS), the lifecycle management policies, the encryption, the firewall rules, the private endpoints, the SDK compatibility — is identical. The per-GB storage pricing is also identical. The difference lies entirely in what the hierarchical namespace enables and in the transaction model that changes as a consequence.
What a Hierarchical Namespace Actually Means
Standard Blob Storage uses a flat namespace. Every blob is stored as a key-value pair: the key is the full path string (e.g., container/raw/2026/04/05/data.parquet) and the value is the blob content. The apparent folder structure implied by the forward slashes is an illusion — the storage engine treats the entire path as a single flat key. There are no real directories. When you "rename a folder," the storage engine must copy every blob with the old prefix to a new key, then delete the originals — one copy + one delete operation per blob.
ADLS Gen2's hierarchical namespace creates real directories with their own metadata objects. Directories are first-class entities — they have their own properties, their own access control lists, and their own atomic operations. Renaming or deleting a directory is a single metadata operation that changes the directory object, regardless of how many files are inside it. This is not an optimisation — it is a fundamental change in the storage model.
Figure 1 — Flat namespace (Blob Storage) vs hierarchical namespace (ADLS Gen2): structural model and operation costsThe same 10,000-file directory rename: 30,000 API calls on Blob Storage (flat namespace) vs 1 atomic call on ADLS Gen2. This difference is why Spark job commit time, Delta Lake ACID transaction performance, and pipeline folder management speed are completely different between the two services.
Standard Blob Storage uses a flat namespace. Every blob is stored as a key-value pair: the key is the full path string (e.g., container/raw/2026/04/05/data.parquet) and the value is the blob content. The apparent folder structure implied by the forward slashes is an illusion — the storage engine treats the entire path as a single flat key. There are no real directories. When you "rename a folder," the storage engine must copy every blob with the old prefix to a new key, then delete the originals — one copy + one delete operation per blob.
ADLS Gen2's hierarchical namespace creates real directories with their own metadata objects. Directories are first-class entities — they have their own properties, their own access control lists, and their own atomic operations. Renaming or deleting a directory is a single metadata operation that changes the directory object, regardless of how many files are inside it. This is not an optimisation — it is a fundamental change in the storage model.
How the Namespace Choice Affects Real Operations: API Call Counts
The performance impact of the namespace difference is not theoretical — it is measurable in every Spark job, every ADF pipeline, and every Delta Lake transaction. The following table shows the API call counts for common operations that data engineering pipelines perform hundreds of times per day:
Operation Blob Storage (Flat) ADLS Gen2 (Hierarchical) Impact Rename directory (10,000 files) ~30,000 API calls (copy + delete each) 1 atomic call Spark output commits. Delta Lake ACID. ADF move operations. Delete directory (10,000 files) ~20,000 API calls (list + delete each) 1 atomic call Medallion layer cleanup. Spark temp directory removal. List files in a directory Scans ALL blob names with prefix filter Direct directory listing (O(1) per dir) Partition discovery. File-based incremental ingestion. Spark job output commit Write temp dir, rename each file to final — slow Atomic directory rename — instant Total Spark job wall time. Databricks cluster billing. Delta Lake ACID transaction Relies on blob-level optimistic locking (fragile) Uses atomic directory ops for transaction log Concurrent write correctness. Multi-writer pipelines. Apply permissions to a directory Container-level only (RBAC) — cannot do directory-level POSIX ACL on any directory or file Team-level data access control. Regulatory compliance.
📌 The Key InsightEvery Spark job that writes data to storage ends with a directory rename — it writes to a temporary _temporary directory, then renames it to the final output path. On Blob Storage, this rename operation costs 3× the number of output files in API calls and time. On ADLS Gen2, it is one atomic metadata operation. For any workload using Spark, Databricks, Azure Synapse, or Delta Lake, ADLS Gen2 is not optional — it is a performance requirement.
The performance impact of the namespace difference is not theoretical — it is measurable in every Spark job, every ADF pipeline, and every Delta Lake transaction. The following table shows the API call counts for common operations that data engineering pipelines perform hundreds of times per day:
| Operation | Blob Storage (Flat) | ADLS Gen2 (Hierarchical) | Impact |
|---|---|---|---|
| Rename directory (10,000 files) | ~30,000 API calls (copy + delete each) | 1 atomic call | Spark output commits. Delta Lake ACID. ADF move operations. |
| Delete directory (10,000 files) | ~20,000 API calls (list + delete each) | 1 atomic call | Medallion layer cleanup. Spark temp directory removal. |
| List files in a directory | Scans ALL blob names with prefix filter | Direct directory listing (O(1) per dir) | Partition discovery. File-based incremental ingestion. |
| Spark job output commit | Write temp dir, rename each file to final — slow | Atomic directory rename — instant | Total Spark job wall time. Databricks cluster billing. |
| Delta Lake ACID transaction | Relies on blob-level optimistic locking (fragile) | Uses atomic directory ops for transaction log | Concurrent write correctness. Multi-writer pipelines. |
| Apply permissions to a directory | Container-level only (RBAC) — cannot do directory-level | POSIX ACL on any directory or file | Team-level data access control. Regulatory compliance. |
Every Spark job that writes data to storage ends with a directory rename — it writes to a temporary _temporary directory, then renames it to the final output path. On Blob Storage, this rename operation costs 3× the number of output files in API calls and time. On ADLS Gen2, it is one atomic metadata operation. For any workload using Spark, Databricks, Azure Synapse, or Delta Lake, ADLS Gen2 is not optional — it is a performance requirement.
Full Feature Comparison: ADLS Gen2 vs Blob Storage
Feature / Capability Azure Blob Storage ADLS Gen2 Namespace type Flat (object key-value store) Hierarchical (real file system directories) Directory operations Simulated via key prefix scanning Atomic — rename, delete, move in 1 API call Access control model RBAC at account or container level only RBAC + POSIX ACLs at directory and file level Hadoop / Spark compatibility Via WASB adapter (legacy) or ABFS (limited) Native ABFS driver — full Hadoop FS semantics Delta Lake / Iceberg ACID Fragile — relies on blob-level locking Native — uses atomic directory operations Azure Synapse Analytics Supported with reduced performance Native integration — recommended storage layer Azure Databricks Works but slower for large jobs Native — all Databricks documentation targets ADLS Gen2 Storage tiers Hot, Cool, Cold, Archive Same Hot, Cool, Cold, Archive tiers Lifecycle management Rules based on blob age and last-access time Same + path-condition rules (e.g., only in /archive/) Per-GB storage price Identical to ADLS Gen2 Identical to Blob Storage Transaction pricing Per-operation billing Per-4MB block for reads/writes — 15–20% higher at scale REST API endpoint blob.core.windows.net dfs.core.windows.net (DFS endpoint) + blob endpoint Microsoft Fabric / OneLake Not natively supported OneLake is built on ADLS Gen2 internally Azure AI Search indexer Fully supported data source Fully supported data source (same API) Static website hosting Supported natively Not supported NFSv3 mount Not supported Supported — mount as a file system on Linux VMs SFTP access Not supported Supported — requires HNS enabled (= ADLS Gen2) Ideal workloads Web assets, backups, archives, RAG document store, model artefact serving, media delivery, log archiving Spark/Databricks pipelines, ML training datasets, Medallion Architecture, Delta Lake, Synapse Analytics, multi-team data lake with fine-grained permissions
| Feature / Capability | Azure Blob Storage | ADLS Gen2 |
|---|---|---|
| Namespace type | Flat (object key-value store) | Hierarchical (real file system directories) |
| Directory operations | Simulated via key prefix scanning | Atomic — rename, delete, move in 1 API call |
| Access control model | RBAC at account or container level only | RBAC + POSIX ACLs at directory and file level |
| Hadoop / Spark compatibility | Via WASB adapter (legacy) or ABFS (limited) | Native ABFS driver — full Hadoop FS semantics |
| Delta Lake / Iceberg ACID | Fragile — relies on blob-level locking | Native — uses atomic directory operations |
| Azure Synapse Analytics | Supported with reduced performance | Native integration — recommended storage layer |
| Azure Databricks | Works but slower for large jobs | Native — all Databricks documentation targets ADLS Gen2 |
| Storage tiers | Hot, Cool, Cold, Archive | Same Hot, Cool, Cold, Archive tiers |
| Lifecycle management | Rules based on blob age and last-access time | Same + path-condition rules (e.g., only in /archive/) |
| Per-GB storage price | Identical to ADLS Gen2 | Identical to Blob Storage |
| Transaction pricing | Per-operation billing | Per-4MB block for reads/writes — 15–20% higher at scale |
| REST API endpoint | blob.core.windows.net | dfs.core.windows.net (DFS endpoint) + blob endpoint |
| Microsoft Fabric / OneLake | Not natively supported | OneLake is built on ADLS Gen2 internally |
| Azure AI Search indexer | Fully supported data source | Fully supported data source (same API) |
| Static website hosting | Supported natively | Not supported |
| NFSv3 mount | Not supported | Supported — mount as a file system on Linux VMs |
| SFTP access | Not supported | Supported — requires HNS enabled (= ADLS Gen2) |
| Ideal workloads | Web assets, backups, archives, RAG document store, model artefact serving, media delivery, log archiving | Spark/Databricks pipelines, ML training datasets, Medallion Architecture, Delta Lake, Synapse Analytics, multi-team data lake with fine-grained permissions |
Architectural Selection for 6 AI Workload Patterns
The comparison table shows the technical differences. What engineers actually need is the decision by workload type. The following six scenarios cover the most common AI and data engineering storage decisions on Azure in 2026.
Scenario 1ML Model Training Dataset StorageUse ADLS Gen2Workload pattern: Training datasets are stored in Parquet or TFRecord format, partitioned by date or category. Azure Databricks or Azure Machine Learning reads the data in parallel across multiple workers. Feature engineering jobs transform the raw data and write to a curated directory. Pipeline runs rename output directories atomically on completion.
Why ADLS Gen2: Databricks mounts ADLS Gen2 using the native ABFS driver, which treats the storage as a true file system. Every training job ends with a directory rename (the Spark commit protocol) — this is 1 call on ADLS Gen2 and thousands of calls on Blob Storage. Training jobs that take 2 hours on Blob Storage have been measured completing in 45 minutes on ADLS Gen2 for the same data, simply due to commit protocol efficiency. Additionally, ML teams typically need per-experiment directory access control — POSIX ACLs enable this precisely.
Storage structure: Use Medallion Architecture — /bronze/raw-data/ for ingest, /silver/features/ for processed features, /gold/training-sets/ for final training data. Structure datasets with /experiment-name/version/ subdirectories for experiment tracking.
Scenario 2RAG Document Ingestion PipelineUse Blob StorageWorkload pattern: Raw documents (PDFs, Word files, HTML) are uploaded by users or ingested from enterprise systems. An Azure AI Search indexer polls the storage account, picks up new documents, and passes them through a skillset that chunks and embeds the content. The processed chunks are stored in the search index, not back in storage.
Why Blob Storage: Azure AI Search's blob indexer is designed for flat-namespace blob accounts. The indexer scans containers using blob prefix filters — a pattern that works identically on both Blob Storage and ADLS Gen2, but Blob Storage is cheaper per transaction for simple file-drop ingestion patterns. You do not need atomic directory renames, POSIX ACLs, or Spark-compatible file system semantics for a document store. The documents are written once, read once by the indexer, and the valuable derived data lives in Azure AI Search — not back in storage.
Storage structure: Use container-per-department or container-per-tenant for access control. Enable soft delete and versioning. Use lifecycle management to archive processed source documents to Cool tier after 90 days. RBAC at the container level is sufficient for this pattern.
Scenario 3Enterprise Data Lake — Multi-Team AnalyticsUse ADLS Gen2Workload pattern: Multiple teams (Finance, Marketing, Engineering, HR) ingest data from different source systems into a shared data lake. Each team has its own directory zone. Azure Synapse Analytics, Databricks, and Power BI read from different layers. Some directories contain sensitive data (PII, financial records) that must not be readable by teams outside the data owner.
Why ADLS Gen2: This is the canonical ADLS Gen2 use case. POSIX ACLs enable directory-level and file-level access control — the Finance team can have read-write on /silver/finance/ and read-only on /gold/shared/, while the Marketing team cannot see /silver/finance/ at all. This granularity is impossible on Blob Storage, which only allows access control at the container level. For shared data lakes with compliance requirements (GDPR, HIPAA, financial data regulations), ADLS Gen2 with POSIX ACLs is the required architecture.
Storage structure: Implement Medallion Architecture with zone-level access control. Assign POSIX ACLs at the zone directory level per team. Use Azure Active Directory groups as the identity principal in ACL assignments — never individual user IDs.
Scenario 4AI Model Artefact Store and RegistryUse Blob StorageWorkload pattern: Trained model weights, ONNX exports, tokeniser configs, evaluation results, and deployment artefacts are stored after training completes. Azure Machine Learning's model registry, MLflow, or a custom artefact store reads and writes model files. Inference services (Azure ML Online Endpoints, Azure Container Apps) download model weights at deployment time.
Why Blob Storage: Model artefact storage is a write-once, read-many pattern. Models are written once after training and read many times during deployment and inference. There are no directory renames (models are uploaded by filename, not via Spark commits), no multi-writer scenarios, and no need for POSIX ACLs (RBAC at the container level with Azure ML's Managed Identity is sufficient). Blob Storage's native CDN integration and SAS URL generation make model weight distribution to inference containers straightforward. The simpler transaction model is also cheaper for this access pattern.
Storage structure: Use containers named by model family: models-gpt-finetune, models-embedding, models-classifier. Use blob versioning to maintain model version history. Enable immutability policies on production model containers to prevent accidental overwriting of deployed model weights.
Scenario 5Real-Time IoT Telemetry Ingestion for AIEither — but read the nuanceWorkload pattern: Azure Event Hubs captures IoT telemetry (sensor readings, equipment metrics, vehicle telemetry) and Azure Stream Analytics or Databricks Structured Streaming writes microbatch results to storage. Downstream ML pipelines read the accumulated telemetry for anomaly detection or predictive maintenance model training.
Decision depends on downstream consumption: If the telemetry will be read by Databricks for ML training or by Synapse for analytics — use ADLS Gen2. The atomic commit semantics of Spark Structured Streaming output are dramatically faster on ADLS Gen2, and you will likely want partition-level access control. If the telemetry is written once and read by an Azure AI Search indexer for semantic search, or served to an inference API as raw readings — use Blob Storage for lower transaction costs on the high-volume small-file writes that IoT streams generate. Note: the 128 KiB minimum billable object size on Cool/Cold/Archive tiers (effective July 2026 for new accounts) significantly affects small IoT message storage costs — aggregate messages into Parquet before archiving.
Scenario 6Fine-Tuning Dataset ManagementUse ADLS Gen2Workload pattern: Instruction-tuning datasets, RLHF preference data, evaluation benchmarks, and data quality labels are curated by a data team across multiple iterations. Dataset versions are maintained. Quality filtering pipelines read raw data, apply filters, and write curated subsets to a separate directory. Azure OpenAI fine-tuning jobs or open-source training frameworks read the final datasets.
Why ADLS Gen2: Dataset curation is an iterative, multi-step pipeline with directory-level version management. Each pipeline run writes to a new versioned directory and potentially deletes or archives old versions. POSIX ACLs enable the data team to control which training jobs can read which dataset versions. The Spark-based quality filtering steps require atomic commit semantics. Additionally, dataset lineage tracking (which raw data produced which fine-tuning dataset) is implementable as directory-level metadata using ADLS Gen2's extended properties.
The comparison table shows the technical differences. What engineers actually need is the decision by workload type. The following six scenarios cover the most common AI and data engineering storage decisions on Azure in 2026.
Workload pattern: Training datasets are stored in Parquet or TFRecord format, partitioned by date or category. Azure Databricks or Azure Machine Learning reads the data in parallel across multiple workers. Feature engineering jobs transform the raw data and write to a curated directory. Pipeline runs rename output directories atomically on completion.
Why ADLS Gen2: Databricks mounts ADLS Gen2 using the native ABFS driver, which treats the storage as a true file system. Every training job ends with a directory rename (the Spark commit protocol) — this is 1 call on ADLS Gen2 and thousands of calls on Blob Storage. Training jobs that take 2 hours on Blob Storage have been measured completing in 45 minutes on ADLS Gen2 for the same data, simply due to commit protocol efficiency. Additionally, ML teams typically need per-experiment directory access control — POSIX ACLs enable this precisely.
Storage structure: Use Medallion Architecture — /bronze/raw-data/ for ingest, /silver/features/ for processed features, /gold/training-sets/ for final training data. Structure datasets with /experiment-name/version/ subdirectories for experiment tracking.
Workload pattern: Raw documents (PDFs, Word files, HTML) are uploaded by users or ingested from enterprise systems. An Azure AI Search indexer polls the storage account, picks up new documents, and passes them through a skillset that chunks and embeds the content. The processed chunks are stored in the search index, not back in storage.
Why Blob Storage: Azure AI Search's blob indexer is designed for flat-namespace blob accounts. The indexer scans containers using blob prefix filters — a pattern that works identically on both Blob Storage and ADLS Gen2, but Blob Storage is cheaper per transaction for simple file-drop ingestion patterns. You do not need atomic directory renames, POSIX ACLs, or Spark-compatible file system semantics for a document store. The documents are written once, read once by the indexer, and the valuable derived data lives in Azure AI Search — not back in storage.
Storage structure: Use container-per-department or container-per-tenant for access control. Enable soft delete and versioning. Use lifecycle management to archive processed source documents to Cool tier after 90 days. RBAC at the container level is sufficient for this pattern.
Workload pattern: Multiple teams (Finance, Marketing, Engineering, HR) ingest data from different source systems into a shared data lake. Each team has its own directory zone. Azure Synapse Analytics, Databricks, and Power BI read from different layers. Some directories contain sensitive data (PII, financial records) that must not be readable by teams outside the data owner.
Why ADLS Gen2: This is the canonical ADLS Gen2 use case. POSIX ACLs enable directory-level and file-level access control — the Finance team can have read-write on /silver/finance/ and read-only on /gold/shared/, while the Marketing team cannot see /silver/finance/ at all. This granularity is impossible on Blob Storage, which only allows access control at the container level. For shared data lakes with compliance requirements (GDPR, HIPAA, financial data regulations), ADLS Gen2 with POSIX ACLs is the required architecture.
Storage structure: Implement Medallion Architecture with zone-level access control. Assign POSIX ACLs at the zone directory level per team. Use Azure Active Directory groups as the identity principal in ACL assignments — never individual user IDs.
Workload pattern: Trained model weights, ONNX exports, tokeniser configs, evaluation results, and deployment artefacts are stored after training completes. Azure Machine Learning's model registry, MLflow, or a custom artefact store reads and writes model files. Inference services (Azure ML Online Endpoints, Azure Container Apps) download model weights at deployment time.
Why Blob Storage: Model artefact storage is a write-once, read-many pattern. Models are written once after training and read many times during deployment and inference. There are no directory renames (models are uploaded by filename, not via Spark commits), no multi-writer scenarios, and no need for POSIX ACLs (RBAC at the container level with Azure ML's Managed Identity is sufficient). Blob Storage's native CDN integration and SAS URL generation make model weight distribution to inference containers straightforward. The simpler transaction model is also cheaper for this access pattern.
Storage structure: Use containers named by model family: models-gpt-finetune, models-embedding, models-classifier. Use blob versioning to maintain model version history. Enable immutability policies on production model containers to prevent accidental overwriting of deployed model weights.
Workload pattern: Azure Event Hubs captures IoT telemetry (sensor readings, equipment metrics, vehicle telemetry) and Azure Stream Analytics or Databricks Structured Streaming writes microbatch results to storage. Downstream ML pipelines read the accumulated telemetry for anomaly detection or predictive maintenance model training.
Decision depends on downstream consumption: If the telemetry will be read by Databricks for ML training or by Synapse for analytics — use ADLS Gen2. The atomic commit semantics of Spark Structured Streaming output are dramatically faster on ADLS Gen2, and you will likely want partition-level access control. If the telemetry is written once and read by an Azure AI Search indexer for semantic search, or served to an inference API as raw readings — use Blob Storage for lower transaction costs on the high-volume small-file writes that IoT streams generate. Note: the 128 KiB minimum billable object size on Cool/Cold/Archive tiers (effective July 2026 for new accounts) significantly affects small IoT message storage costs — aggregate messages into Parquet before archiving.
Workload pattern: Instruction-tuning datasets, RLHF preference data, evaluation benchmarks, and data quality labels are curated by a data team across multiple iterations. Dataset versions are maintained. Quality filtering pipelines read raw data, apply filters, and write curated subsets to a separate directory. Azure OpenAI fine-tuning jobs or open-source training frameworks read the final datasets.
Why ADLS Gen2: Dataset curation is an iterative, multi-step pipeline with directory-level version management. Each pipeline run writes to a new versioned directory and potentially deletes or archives old versions. POSIX ACLs enable the data team to control which training jobs can read which dataset versions. The Spark-based quality filtering steps require atomic commit semantics. Additionally, dataset lineage tracking (which raw data produced which fine-tuning dataset) is implementable as directory-level metadata using ADLS Gen2's extended properties.
Implementing Medallion Architecture on ADLS Gen2
The Medallion Architecture (Bronze → Silver → Gold) is the standard data organisation pattern for ADLS Gen2 data lakes. It represents three quality layers of the same data, each processed to a higher standard of reliability and business-readiness.
Figure 2 — Medallion Architecture on ADLS Gen2: directory structure, data quality layers, and access patterns per layerEach Medallion layer has its own POSIX ACL assignment — only ADLS Gen2 makes this possible. Ingest service principals write to Bronze only. Data engineering teams read Bronze, write Silver. Analytics teams read Silver and Gold. Reporting service principals read Gold only.
The Medallion Architecture (Bronze → Silver → Gold) is the standard data organisation pattern for ADLS Gen2 data lakes. It represents three quality layers of the same data, each processed to a higher standard of reliability and business-readiness.
Access Control: RBAC vs POSIX ACLs
Both Blob Storage and ADLS Gen2 support Azure RBAC — you assign roles like Storage Blob Data Reader or Storage Blob Data Contributor to an identity at the account or container scope. On Blob Storage, that is where access control ends. On ADLS Gen2, POSIX ACLs extend access control to individual directories and files.
1
Both Blob Storage and ADLS Gen2 support Azure RBAC — you assign roles like Storage Blob Data Reader or Storage Blob Data Contributor to an identity at the account or container scope. On Blob Storage, that is where access control ends. On ADLS Gen2, POSIX ACLs extend access control to individual directories and files.
Understand POSIX ACL inheritance on ADLS Gen2
Every directory on ADLS Gen2 has two ACLs: the access ACL (controls who can read/write/execute the directory itself) and the default ACL (inherited by new files and subdirectories created within it). Set the default ACL on parent directories at provisioning time — this ensures new data automatically inherits the correct permissions without manual ACL updates.
2
Every directory on ADLS Gen2 has two ACLs: the access ACL (controls who can read/write/execute the directory itself) and the default ACL (inherited by new files and subdirectories created within it). Set the default ACL on parent directories at provisioning time — this ensures new data automatically inherits the correct permissions without manual ACL updates.
Use Entra ID groups, never individual user identities
Assign POSIX ACLs to Entra ID security groups, not individual users. When a team member joins or leaves, you update their group membership once — the ADLS Gen2 ACLs do not change. Individual user ACLs create maintenance debt that compounds with team size. Create groups like dl-silver-finance-read, dl-bronze-ingest-write, dl-gold-ml-read.
3
Assign POSIX ACLs to Entra ID security groups, not individual users. When a team member joins or leaves, you update their group membership once — the ADLS Gen2 ACLs do not change. Individual user ACLs create maintenance debt that compounds with team size. Create groups like dl-silver-finance-read, dl-bronze-ingest-write, dl-gold-ml-read.
Layer RBAC and POSIX ACLs — they work together
RBAC provides the coarse-grained access boundary (which storage accounts an identity can access). POSIX ACLs provide the fine-grained access within those accounts (which directories and files). For production data lakes, assign Storage Blob Data Reader RBAC at the storage account level to all data lake users, then use POSIX ACLs to restrict which directories they can actually see and read within the account.
Azure CLI — Set POSIX ACL on ADLS Gen2 directory for a team group# Get the object ID of the Entra ID group
GROUP_ID=$(az ad group show --group "dl-silver-finance-read" --query id -o tsv)
# Set read+execute ACL on the /silver/finance/ directory
# (execute is required to traverse into subdirectories)
az storage fs access set \
--account-name YOUR-ADLS-ACCOUNT \
--file-system enterprise-lake \
--path silver/finance \
--acl "group:$GROUP_ID:r-x"
# Set DEFAULT ACL so new subdirectories inherit the same permissions
az storage fs access set \
--account-name YOUR-ADLS-ACCOUNT \
--file-system enterprise-lake \
--path silver/finance \
--acl "default:group:$GROUP_ID:r-x"
# Grant write access to the ingest service principal on /bronze/salesdb/
SP_OID=$(az ad sp show --id YOUR-SP-CLIENT-ID --query id -o tsv)
az storage fs access set \
--account-name YOUR-ADLS-ACCOUNT \
--file-system enterprise-lake \
--path bronze/salesdb \
--acl "user:$SP_OID:rwx,default:user:$SP_OID:rwx"
RBAC provides the coarse-grained access boundary (which storage accounts an identity can access). POSIX ACLs provide the fine-grained access within those accounts (which directories and files). For production data lakes, assign Storage Blob Data Reader RBAC at the storage account level to all data lake users, then use POSIX ACLs to restrict which directories they can actually see and read within the account.
GROUP_ID=$(az ad group show --group "dl-silver-finance-read" --query id -o tsv)
# Set read+execute ACL on the /silver/finance/ directory
# (execute is required to traverse into subdirectories)
az storage fs access set \
--account-name YOUR-ADLS-ACCOUNT \
--file-system enterprise-lake \
--path silver/finance \
--acl "group:$GROUP_ID:r-x"
# Set DEFAULT ACL so new subdirectories inherit the same permissions
az storage fs access set \
--account-name YOUR-ADLS-ACCOUNT \
--file-system enterprise-lake \
--path silver/finance \
--acl "default:group:$GROUP_ID:r-x"
# Grant write access to the ingest service principal on /bronze/salesdb/
SP_OID=$(az ad sp show --id YOUR-SP-CLIENT-ID --query id -o tsv)
az storage fs access set \
--account-name YOUR-ADLS-ACCOUNT \
--file-system enterprise-lake \
--path bronze/salesdb \
--acl "user:$SP_OID:rwx,default:user:$SP_OID:rwx"
Pricing: Where the Cost Differences Actually Come From
The per-GB storage price between Blob Storage and ADLS Gen2 is identical for the same tier and redundancy option. The cost difference is entirely in transactions — and it is more nuanced than a simple percentage markup.
- Blob Storage transactions: charged per operation — one API call = one billable operation, regardless of the blob size.
- ADLS Gen2 (HNS) transactions: charged per 4 MB block for read and write operations. A 400 MB Parquet file write = 100 billable write operations on ADLS Gen2. The same file on Blob Storage = 1 billable write operation.
This means ADLS Gen2 is structurally more expensive for workloads with many large-file reads and writes — the per-4MB billing multiplies transaction costs proportionally to file size. For analytics workloads with large files (Parquet, ORC, Avro), the additional transaction cost is typically 15–20% higher. However, because analytics jobs on ADLS Gen2 complete faster and generate fewer API calls (due to atomic directory operations), the total cost including compute time is almost always lower for analytics workloads despite the higher per-transaction rate.
⚠ 2026 Pricing Change: 128 KiB Minimum Billable Object SizeStarting July 1, 2026 for new storage accounts (July 1, 2027 for all accounts), Cool, Cold, and Archive tiers apply a 128 KiB minimum billable object size. A 4 KB log file in Cool tier is billed as 128 KiB — a 32× billing multiplier. This critically affects IoT telemetry, microservice event logs, and any small-file workload moved to cheaper tiers. Batch small files into Parquet archives before lifecycle policies move them to Cool/Cold/Archive. This applies equally to both Blob Storage and ADLS Gen2.
The per-GB storage price between Blob Storage and ADLS Gen2 is identical for the same tier and redundancy option. The cost difference is entirely in transactions — and it is more nuanced than a simple percentage markup.
- Blob Storage transactions: charged per operation — one API call = one billable operation, regardless of the blob size.
- ADLS Gen2 (HNS) transactions: charged per 4 MB block for read and write operations. A 400 MB Parquet file write = 100 billable write operations on ADLS Gen2. The same file on Blob Storage = 1 billable write operation.
This means ADLS Gen2 is structurally more expensive for workloads with many large-file reads and writes — the per-4MB billing multiplies transaction costs proportionally to file size. For analytics workloads with large files (Parquet, ORC, Avro), the additional transaction cost is typically 15–20% higher. However, because analytics jobs on ADLS Gen2 complete faster and generate fewer API calls (due to atomic directory operations), the total cost including compute time is almost always lower for analytics workloads despite the higher per-transaction rate.
Starting July 1, 2026 for new storage accounts (July 1, 2027 for all accounts), Cool, Cold, and Archive tiers apply a 128 KiB minimum billable object size. A 4 KB log file in Cool tier is billed as 128 KiB — a 32× billing multiplier. This critically affects IoT telemetry, microservice event logs, and any small-file workload moved to cheaper tiers. Batch small files into Parquet archives before lifecycle policies move them to Cool/Cold/Archive. This applies equally to both Blob Storage and ADLS Gen2.
The Decision Framework: Which One to Provision Right Now
If you are about to create a storage account, this framework answers the question definitively. Work through the rows — a "Yes" in either column points to that service.
If you are about to create a storage account, this framework answers the question definitively. Work through the rows — a "Yes" in either column points to that service.
ADLS Gen2 vs Blob Storage Decision Framework
Answer each question. The column with the most "Yes" answers is the correct choice. If any MANDATORY row shows ADLS Gen2, choose ADLS Gen2 regardless of other answers.
QuestionBlob StorageADLS Gen2[MANDATORY] Will this data be read by Spark, Databricks, Synapse Analytics, or HDInsight?NoYes → ADLS Gen2[MANDATORY] Will Delta Lake, Apache Iceberg, or any ACID transaction format be used?NoYes → ADLS Gen2[MANDATORY] Do multiple teams need different permissions on different subdirectories of the same account?NoYes → ADLS Gen2Is this an ML training dataset that will be read by AzureML or Databricks ML workloads?NoYes → ADLS Gen2Will this be used as the storage layer for Microsoft Fabric or OneLake?NoYes → ADLS Gen2Is this a RAG document source read by Azure AI Search indexer?Yes → BlobNoIs this serving model weights or artefacts to inference endpoints?Yes → BlobNoIs this hosting static web assets (HTML, JS, CSS, images)?Yes → BlobNo (not supported)Is this a general backup or archive store with no analytics requirements?Yes → BlobNoDoes the workload generate many small files (<128 KiB) destined for Cool/Cold/Archive tier?Yes → Blob (lower transaction overhead per small file)No preference — same tier pricing either way💡 When In DoubtFor any new data lake, ADLS Gen2 is the default choice. Microsoft's own guidance states that ADLS Gen2 is the standard destination for all enterprise data lake deployments — it is the foundation of Microsoft Fabric, OneLake, Synapse Analytics, and all Databricks documentation. The additional transaction cost (15–20%) is justified by the operational benefits in virtually every analytics scenario. Only use Blob Storage when you have a specific reason — serving static content, storing model artefacts for inference, or hosting RAG document sources for AI Search.
Answer each question. The column with the most "Yes" answers is the correct choice. If any MANDATORY row shows ADLS Gen2, choose ADLS Gen2 regardless of other answers.
For any new data lake, ADLS Gen2 is the default choice. Microsoft's own guidance states that ADLS Gen2 is the standard destination for all enterprise data lake deployments — it is the foundation of Microsoft Fabric, OneLake, Synapse Analytics, and all Databricks documentation. The additional transaction cost (15–20%) is justified by the operational benefits in virtually every analytics scenario. Only use Blob Storage when you have a specific reason — serving static content, storing model artefacts for inference, or hosting RAG document sources for AI Search.
Terraform IaC for Both Storage Types
Terraform — ADLS Gen2 storage account with HNS enabled, private endpoint, and RBAC# ── ADLS Gen2: Enterprise Data Lake ──────────────────────
resource "azurerm_storage_account" "data_lake" {
name = var.data_lake_name
resource_group_name = var.resource_group_name
location = var.location
account_tier = "Standard"
account_replication_type = "ZRS" # ZRS for production availability
# ↓ This single toggle creates ADLS Gen2
is_hns_enabled = true # Hierarchical Namespace = ADLS Gen2
min_tls_version = "TLS1_2"
allow_nested_items_to_be_public = false
public_network_access_enabled = false # Private endpoint only
blob_properties {
delete_retention_policy { days = 7 }
}
}
# ── Filesystem (Container) for Medallion layers ───────────
resource "azurerm_storage_data_lake_gen2_filesystem" "lake" {
name = "enterprise-lake"
storage_account_id = azurerm_storage_account.data_lake.id
}
# ── Bronze layer directory with ACL ──────────────────────
resource "azurerm_storage_data_lake_gen2_path" "bronze" {
path = "bronze"
filesystem_name = azurerm_storage_data_lake_gen2_filesystem.lake.name
storage_account_id = azurerm_storage_account.data_lake.id
resource = "directory"
ace {
scope = "access"
type = "group"
id = var.ingest_group_id # Ingest SP group: rwx
permissions = "rwx"
}
ace {
scope = "default" # Inherited by new subdirs
type = "group"
id = var.ingest_group_id
permissions = "rwx"
}
}
# ── Blob Storage: for RAG docs, model artefacts ───────────
resource "azurerm_storage_account" "blob_store" {
name = var.blob_store_name
resource_group_name = var.resource_group_name
location = var.location
account_tier = "Standard"
account_replication_type = "LRS"
is_hns_enabled = false # Flat namespace = standard Blob Storage
blob_properties {
delete_retention_policy { days = 30 }
versioning_enabled = true # Version model artefacts
}
}
resource "azurerm_storage_account" "data_lake" {
name = var.data_lake_name
resource_group_name = var.resource_group_name
location = var.location
account_tier = "Standard"
account_replication_type = "ZRS" # ZRS for production availability
# ↓ This single toggle creates ADLS Gen2
is_hns_enabled = true # Hierarchical Namespace = ADLS Gen2
min_tls_version = "TLS1_2"
allow_nested_items_to_be_public = false
public_network_access_enabled = false # Private endpoint only
blob_properties {
delete_retention_policy { days = 7 }
}
}
# ── Filesystem (Container) for Medallion layers ───────────
resource "azurerm_storage_data_lake_gen2_filesystem" "lake" {
name = "enterprise-lake"
storage_account_id = azurerm_storage_account.data_lake.id
}
# ── Bronze layer directory with ACL ──────────────────────
resource "azurerm_storage_data_lake_gen2_path" "bronze" {
path = "bronze"
filesystem_name = azurerm_storage_data_lake_gen2_filesystem.lake.name
storage_account_id = azurerm_storage_account.data_lake.id
resource = "directory"
ace {
scope = "access"
type = "group"
id = var.ingest_group_id # Ingest SP group: rwx
permissions = "rwx"
}
ace {
scope = "default" # Inherited by new subdirs
type = "group"
id = var.ingest_group_id
permissions = "rwx"
}
}
# ── Blob Storage: for RAG docs, model artefacts ───────────
resource "azurerm_storage_account" "blob_store" {
name = var.blob_store_name
resource_group_name = var.resource_group_name
location = var.location
account_tier = "Standard"
account_replication_type = "LRS"
is_hns_enabled = false # Flat namespace = standard Blob Storage
blob_properties {
delete_retention_policy { days = 30 }
versioning_enabled = true # Version model artefacts
}
}
Migrating from Blob Storage to ADLS Gen2
Hierarchical namespace cannot be enabled on an existing storage account. If you created a Blob Storage account and now need ADLS Gen2 capabilities, you must migrate to a new account. There are three approaches depending on your data volume and downtime tolerance:
1
Hierarchical namespace cannot be enabled on an existing storage account. If you created a Blob Storage account and now need ADLS Gen2 capabilities, you must migrate to a new account. There are three approaches depending on your data volume and downtime tolerance:
Create the new ADLS Gen2 account
Create a new storage account with is_hns_enabled = true. Configure the same redundancy, network rules, private endpoints, and lifecycle policies as the source account. Create the filesystem and directory structure with the correct POSIX ACLs before copying data.
2
Create a new storage account with is_hns_enabled = true. Configure the same redundancy, network rules, private endpoints, and lifecycle policies as the source account. Create the filesystem and directory structure with the correct POSIX ACLs before copying data.
Use AzCopy for large-scale data migration
AzCopy's copy command with --recursive transfers data between storage accounts efficiently using server-side copy — no data traverses your network. azcopy copy 'https://SOURCE.blob.core.windows.net/container?SAS' 'https://DEST.dfs.core.windows.net/filesystem?SAS' --recursive. For very large datasets (>10TB), use AzCopy with the --list-of-files flag to parallelise across multiple AzCopy instances.
3
AzCopy's copy command with --recursive transfers data between storage accounts efficiently using server-side copy — no data traverses your network. azcopy copy 'https://SOURCE.blob.core.windows.net/container?SAS' 'https://DEST.dfs.core.windows.net/filesystem?SAS' --recursive. For very large datasets (>10TB), use AzCopy with the --list-of-files flag to parallelise across multiple AzCopy instances.
Validate and switch application connection strings
ADLS Gen2 uses the DFS endpoint (dfs.core.windows.net) for ABFS access and retains the blob endpoint (blob.core.windows.net) for backward-compatible blob API access. Update Spark configurations to use the ABFS endpoint: abfss://filesystem@account.dfs.core.windows.net/. Validate pipeline reads and writes against the new account before decommissioning the source.
4
ADLS Gen2 uses the DFS endpoint (dfs.core.windows.net) for ABFS access and retains the blob endpoint (blob.core.windows.net) for backward-compatible blob API access. Update Spark configurations to use the ABFS endpoint: abfss://filesystem@account.dfs.core.windows.net/. Validate pipeline reads and writes against the new account before decommissioning the source.
Set up Azure Data Factory for ongoing delta migration
For zero-downtime migration, use ADF with a binary copy pipeline to continuously replicate new blobs from the source account to the ADLS Gen2 destination using a scheduled trigger with delta detection. Run applications against the ADLS Gen2 account for a validation period before decommissioning the source account.
For zero-downtime migration, use ADF with a binary copy pipeline to continuously replicate new blobs from the source account to the ADLS Gen2 destination using a scheduled trigger with delta detection. Run applications against the ADLS Gen2 account for a validation period before decommissioning the source account.
Architecture Decision Summary
ADLS Gen2 is Blob Storage with one toggle flipped. It is not a separate service. The single decision — whether to enable hierarchical namespace at account creation — determines everything else. This decision is irrevocable.For any Spark, Databricks, Synapse, or Delta Lake workload, use ADLS Gen2. The atomic directory operation difference is not a minor optimisation — it determines whether Spark job commits take seconds or minutes, and whether Delta Lake ACID transactions are reliable or fragile.For RAG document sources, model artefact stores, and static asset hosting, use Blob Storage. These workloads do not need hierarchical namespace, do not use Spark commits, and the lower transaction cost is worth the feature gap.POSIX ACLs are only available on ADLS Gen2. If you need directory-level or file-level access control (different teams with different permissions on different Medallion layers), ADLS Gen2 is mandatory. Blob Storage only allows RBAC at the account or container level.The 128 KiB minimum billable object size change (July 2026) affects both services equally for Cool/Cold/Archive tiers. Batch small files into Parquet archives before lifecycle policies move them off Hot tier — this applies to IoT telemetry, logs, and microservice event stores on both Blob Storage and ADLS Gen2.Implement Medallion Architecture (Bronze → Silver → Gold) on ADLS Gen2. This is the standard data organisation pattern for enterprise data lakes on Azure, used by Databricks, Synapse, and Microsoft Fabric. Set POSIX ACLs at each layer directory, using Entra ID groups — never individual user IDs.For new enterprise data lakes, ADLS Gen2 is the default — not a choice that requires justification. Microsoft Fabric, OneLake, Azure Synapse Analytics, and all Databricks documentation target ADLS Gen2 as the standard storage layer. Only choose Blob Storage when you have a specific architectural reason.
Frequently Asked Questions
Can I use the Blob Storage SDK to access an ADLS Gen2 account?Yes. ADLS Gen2 accounts expose both the blob endpoint (blob.core.windows.net) and the DFS endpoint (dfs.core.windows.net). The blob endpoint provides full backward compatibility with the Blob Storage SDK and API — any code that works against Blob Storage will work against ADLS Gen2 via the blob endpoint. However, to use hierarchical namespace features (atomic directory operations, POSIX ACLs, ABFS path operations), you must use the DFS endpoint and the Data Lake Storage SDK or the ABFS driver in Spark. For analytics workloads, always use the DFS endpoint and ABFS paths (abfss://filesystem@account.dfs.core.windows.net/).Is ADLS Gen2 more expensive than Blob Storage overall?The per-GB storage cost is identical. ADLS Gen2 transaction costs are 15–20% higher because reads and writes are charged per 4 MB block rather than per operation. For analytics workloads with large files, this means higher transaction costs. However, because ADLS Gen2's atomic directory operations reduce the API call count dramatically for directory operations, the total billable operation count for analytics pipelines is often lower on ADLS Gen2 than Blob Storage. For general object storage (backups, static assets, model artefacts), Blob Storage is cheaper per transaction due to the per-operation vs per-4MB billing difference.Can I upgrade an existing Blob Storage account to ADLS Gen2?Not directly — you cannot change the hierarchical namespace setting on an existing storage account. Microsoft previously offered an "upgrade" feature for some accounts, but it was retired. The current migration path requires creating a new storage account with hierarchical namespace enabled and copying all data to it using AzCopy or Azure Data Factory. This is why the decision must be made correctly at provisioning time. If you are unsure at account creation, enable hierarchical namespace — the small additional transaction cost is less expensive than a data migration project later.Does Azure AI Search's indexer work differently with ADLS Gen2 vs Blob Storage?No meaningful difference. Azure AI Search's blob indexer uses the blob API endpoint, which is fully compatible with both Blob Storage and ADLS Gen2 accounts. The indexer scans containers using blob prefix enumeration and processes individual blobs — operations that are functionally identical on both account types. If your RAG document source data is already in an ADLS Gen2 account (perhaps because it is also consumed by Databricks for analytics), the Azure AI Search indexer will work correctly against that account without any changes. However, if the account exists purely to serve the indexer, there is no reason to enable hierarchical namespace.
Related FAVRITE Articles
- Architectural Runbook: Optimizing Azure AI Search for Enterprise RAG Workflows
- The Hidden Cloud Drain: How to Find and Kill Orphaned Azure Resources Automatically
- Top 100 Ways to Reduce Azure Cloud Costs (2026)
- Stop Using Connection Strings: A Step-by-Step Guide to Azure Managed Identities in 2026
- Architectural Runbook: Optimizing Azure AI Search for Enterprise RAG Workflows
- The Hidden Cloud Drain: How to Find and Kill Orphaned Azure Resources Automatically
- Top 100 Ways to Reduce Azure Cloud Costs (2026)
- Stop Using Connection Strings: A Step-by-Step Guide to Azure Managed Identities in 2026