Data Warehousing Updated: 2026-03-14

What is Delta Lake?

An open-source storage layer that brings ACID transactions, scalable metadata handling, and unified batch/streaming processing to data lakes.

From the field

Delta Lake is the table format that made the lakehouse concept real for the first time. Created by Databricks in 2019 and open-sourced under the Linux Foundation, Delta gave Spark users what they'd been missing for a decade: a way to write Parquet files to S3 with ACID guarantees, schema enforcement, and time travel — without needing a separate metadata service. The core insight is beautifully simple: put a transaction log (_delta_log/) alongside the data files, record every add/remove/update as a JSON commit, and let readers reconstruct the current state by replaying the log. No Hive Metastore. No central catalog service. Just files on object storage that happen to know how to version themselves. In the Databricks ecosystem, Delta is the default and most mature table format — Structured Streaming sinks, Databricks SQL, MLflow feature tables, Unity Catalog, Delta Sharing, Photon engine optimizations all assume Delta underneath. Outside Databricks, Delta has strong Spark and Python support (via delta-rs Rust bindings, pyspark DataSource), but ecosystem support is narrower than Iceberg.

Delta vs Iceberg — the strategic question in 2026

Delta and Iceberg solve the same problem (ACID table format on object storage) with different design tradeoffs. Delta's transaction log is a flat sequence of JSON commits with periodic Parquet checkpoints; Iceberg's is a tree of manifest lists → manifests → data files. Delta's schema evolution supports add/rename/widening; Iceberg adds partition evolution and hidden partitioning. Delta has best-in-class Databricks integration (Photon, liquid clustering, predictive optimization, deletion vectors); Iceberg has broader neutral engine support (Snowflake, BigQuery, Trino, Flink, DuckDB first-class). Databricks' 2024 Uniform feature lets Delta tables pretend to be Iceberg for read-only external consumers — a significant convergence bet. Rule of thumb in 2026: Databricks-first shop → Delta; multi-engine or non-Databricks shop → Iceberg.

Field story

A SaaS company running ~800 Databricks jobs across bronze/silver/gold layers had chronic issues with concurrent streaming writes corrupting batch outputs (writers racing, last-write-wins overwrites, Structured Streaming checkpoints not matching table state). They were on Parquet + Hive Metastore. We migrated the full pipeline to Delta Lake over 3 months: enabled delta.enableChangeDataFeed=true to replace custom CDC logic, turned on Photon + liquid clustering on hot tables, adopted Unity Catalog for governance, switched from Hive paths to 3-part names, set up Delta Sharing to publish curated tables to downstream partners. Concurrent write conflicts disappeared (optimistic concurrency + retries handle them transparently). Query performance improved 3-4x on Photon-optimized Delta reads. Time travel became the go-to debugging tool for "what did this table look like yesterday?" incident response.

Delta Lake is an open-source storage layer created by Databricks that brings reliability and performance to data lakes. It sits on top of existing cloud storage (S3, ADLS, GCS) and adds ACID transactions, schema enforcement, time travel, and unified batch/streaming processing — transforming raw data lakes into production-grade data lakehouses.

Why Delta Lake?

Data lakes without a table format suffer from:

  • Data corruption: Concurrent writes can create inconsistent state
  • No schema enforcement: Bad data enters the lake silently
  • No transactions: Partial writes leave data in broken state
  • Slow metadata: Listing millions of files in S3 is slow
Delta Lake solves all of these with a transaction log that tracks every change.

How Delta Lake Works

Transaction Log (_delta_log)

Every operation (write, update, delete) creates a JSON commit file in the _delta_log/ directory. This log provides:
  • Atomicity: Commits are all-or-nothing
  • Versioning: Every change is numbered sequentially
  • Time Travel: Read data at any previous version
# Write data as Delta
df.write.format("delta").save("s3://my-bucket/sales")

# Time Travel - read version 5
spark.read.format("delta").option("versionAsOf", 5).load("s3://my-bucket/sales")

# Time Travel - read data as it was yesterday
spark.read.format("delta").option("timestampAsOf", "2025-01-01").load("s3://my-bucket/sales")

Key Features

  1. ACID Transactions: Serializable isolation for concurrent readers and writers
  2. Schema Enforcement: Reject writes that don't match the table schema
  3. Schema Evolution: Safely add new columns without breaking existing queries
  4. Time Travel: Query, restore, or clone any previous version
  5. Unified Batch & Streaming: Same table for both batch and streaming workloads
  6. Z-Order Clustering: Co-locate related data for faster queries
  7. Liquid Clustering: Automatic, incremental data layout optimization (new)

Delta Lake in the Ecosystem

  • Databricks: Delta Lake is the native format — everything runs on Delta
  • Apache Spark: Full read/write support via delta-spark connector
  • Trino/Presto: Read support for federated queries
  • Flink: Delta connector for streaming workflows
  • Rust/Python: delta-rs library for non-Spark environments

Common Use Cases

  1. Data Lakehouse: Replace traditional DW with Delta on cloud storage
  2. Streaming ETL: Combine real-time and batch in one pipeline
  3. GDPR/CCPA: Delete/update individual records for compliance
  4. ML Feature Tables: Versioned, transactional feature storage
  5. Data Sharing: Share Delta tables via Delta Sharing protocol

Key Points

Gotchas

Delta Lake production gotchas:

  • Transaction log files accumulate fast. Every commit writes a NNNNNNNN.json file in _delta_log/. Active tables accumulate thousands per day. Delta auto-creates Parquet checkpoints every 10 commits, but old JSON + checkpoint files stay until VACUUM runs. Set log retention (delta.logRetentionDuration=30 days) explicitly.
  • VACUUM is dangerous. VACUUM RETAIN 0 HOURS deletes unreferenced data files immediately — but any concurrent reader still using an old snapshot will fail mid-query. Default minimum retention (delta.deletedFileRetentionDuration=7 days) protects against this; don't override it without a hard reason.
  • Schema evolution has asymmetric rules. Adding columns: safe, merged transparently. Renaming columns: requires delta.columnMapping.mode=name (set at table-creation; migration is painful). Dropping columns: requires column mapping. Reordering: sometimes breaks downstream queries. Plan schema strategy early.
  • Structured Streaming checkpoint coupling. Delta + Structured Streaming uses a checkpoint directory to track which commit versions have been processed. Moving the checkpoint, replaying, or forgetting to enable mergeSchema creates silent data loss or duplicate writes. Treat checkpoints as source-of-truth artifacts.
  • Liquid clustering vs Z-ORDER. Z-ORDER (legacy) requires explicit OPTIMIZE ... ZORDER BY (cols) jobs and invalidates on write. Liquid Clustering (2024+) is automatic but requires Delta Protocol v7+. Mixing the two on the same table is a configuration error.
  • Deletion Vectors change read semantics. Delta 3.0+ supports deletion vectors (tombstones) for efficient DELETE and MERGE INTO. Tables with DV can only be read by DV-aware clients — older Spark/Trino versions return wrong results. Check client compatibility before enabling (delta.enableDeletionVectors=true).
  • Delta Sharing is a different protocol. Delta Sharing lets you publish tables to external consumers with Parquet files + pre-signed URLs. Consumers use delta-sharing client libraries, NOT normal Delta readers. Row filtering and column masking happen at the provider; don't expect native Delta behavior on the consumer side.
  • Multi-cluster writes need locking. Single Databricks workspace writes coordinate via workspace state. Cross-workspace or cross-tool writes need external locking (S3 conditional writes + DynamoDB, or Unity Catalog managed tables). Unmanaged multi-writer Delta on raw S3 can corrupt the log under race conditions.
  • OPTIMIZE jobs are expensive. Compaction rewrites data files; on a 10TB table, OPTIMIZE can take hours and consume substantial cluster time. Schedule deliberately, not blindly on every table. Databricks Predictive Optimization (managed) auto-schedules; self-managed Delta requires your own orchestration.
  • Outside Databricks, polish varies. Spark + Delta is mature. delta-rs (Rust, powers Python/DuckDB) is great and growing. Trino Delta connector works. Snowflake reads Delta via external tables (read-only, limited). BigQuery support exists but is newer. Benchmark YOUR engine on YOUR workload before committing to Delta outside Databricks.

    • Frequently Asked Questions

    What is Delta Lake in simple terms?

    Delta Lake is a storage layer that sits on top of cloud object storage (S3, ADLS, GCS) and turns raw Parquet files into transactional, queryable, versioned tables. Think of it as adding database-like reliability guarantees — ACID transactions, schema enforcement, time travel, unified batch/streaming — to files that would otherwise be just bags of bytes in S3. The magic is a small `_delta_log/` directory next to your data files containing JSON commit records; readers reconstruct the current table state by replaying the log. From Spark/Python's perspective you just write `df.write.format('delta').save(path)` and get a proper transactional table. Databricks created Delta and donated it to the Linux Foundation; it's now the default table format inside Databricks and one of three major open table formats (with Iceberg and Hudi) powering modern data lakehouses.

    Is Delta Lake the same as Databricks?

    No — they're closely related but distinct. Delta Lake is an open-source table format licensed under Apache 2.0; the code lives at github.com/delta-io/delta and is governed by the Linux Foundation. Databricks is the commercial company that created Delta Lake and offers a managed cloud platform built around it. You can absolutely use Delta Lake outside Databricks: on any Apache Spark cluster (EMR, Dataproc, self-hosted), in Python via delta-rs, in Trino, in Flink. However, many performance features are Databricks-exclusive: Photon engine optimizations, Predictive Optimization (auto-OPTIMIZE), Databricks SQL Serverless, Unity Catalog governance, and liquid clustering depth. Open-source Delta gives you the core transactional guarantees; Databricks adds operational and performance polish. If you're committing heavily to Delta, Databricks is the path of least resistance.

    What is the difference between Delta Lake and Apache Iceberg?

    Both are ACID table formats for object storage, but they have different design philosophies. Delta uses a flat append-only transaction log (JSON commits + Parquet checkpoints) — simpler mental model, very mature for Spark. Iceberg uses a hierarchical metadata tree (manifest lists → manifests → data files) — more complex but enables features like partition evolution and hidden partitioning. Engine support: Delta is strongest in Databricks + Spark; Iceberg is neutral across Snowflake, BigQuery, Trino, Flink, DuckDB, and Spark. Governance: Delta pairs with Unity Catalog; Iceberg pairs with REST Catalog, Polaris, Nessie, Glue. In 2026, Databricks launched Uniform to let Delta tables pretend to be Iceberg for external readers, signaling that multi-engine interop matters. Choose Delta for Databricks-centric shops; choose Iceberg for multi-engine or non-Databricks environments.

    Is Delta Lake free?

    Yes. Delta Lake is open-source under Apache License 2.0 — free for any commercial or non-commercial use, with no licensing fees. Self-host it on Apache Spark, use it from Python via delta-rs, run it on EMR, Dataproc, or any Spark-compatible compute. The core transactional format, APIs, and connectors (Trino, Flink, PyArrow, DuckDB) are all free. Databricks offers a commercial managed platform built on Delta that charges for compute and managed-service value-adds (Photon, Unity Catalog, serverless SQL, MLflow, AI features) — but you never pay for Delta Lake itself. For teams wanting open-source Delta without Databricks: the ecosystem has matured significantly in 2024-2026, with delta-rs reaching production readiness and cross-engine compatibility improving steadily.

    What is the Delta Lake transaction log?

    The `_delta_log/` directory is Delta Lake's source of truth. Every write operation creates an atomic JSON commit file (`00000000000000000001.json`, `00000000000000000002.json`, ...) listing `add` actions (new data files) and `remove` actions (tombstoned old files). Readers reconstruct the current table state by replaying commits in order. Every 10 commits, Delta writes a Parquet checkpoint summarizing the state so readers don't have to replay thousands of individual JSONs. This log-based design gives you: atomicity (a commit either succeeds or fails, no partial state), serializable isolation (concurrent readers see consistent snapshots via optimistic concurrency control), time travel (query `AS OF VERSION N` or `AS OF TIMESTAMP T` by replaying up to that point), and auditability (every change is logged forever until VACUUM). It's conceptually similar to a database WAL on a local disk, but distributed and living on object storage.

    What is Change Data Feed (CDF) in Delta Lake?

    Change Data Feed (CDF) is a Delta Lake feature that exposes row-level changes (inserts, updates, deletes) as a queryable stream. Enable it with `ALTER TABLE ... SET TBLPROPERTIES (delta.enableChangeDataFeed = true)`. After enabling, each write records change metadata (`_change_type`: insert/update_preimage/update_postimage/delete, `_commit_version`, `_commit_timestamp`) that downstream consumers can read incrementally: `SELECT * FROM table_changes('my_table', start_version)`. This replaces brittle custom CDC logic (comparing snapshots, hashing rows, timestamp-based delta detection) with a first-class primitive. Common uses: incremental ETL from Bronze to Silver (process only rows that changed), materialized view refresh, downstream system synchronization (Delta → Kafka → external DB), audit trails. CDF adds some write overhead and storage cost (change records retained per table's retention policy), so enable selectively on tables that need it.

    What are Z-ORDER and Liquid Clustering?

    Both are physical data layout techniques that speed up filtered queries by co-locating related rows into the same files, enabling predicate pruning. **Z-ORDER** (legacy, 2019) requires explicit `OPTIMIZE <table> ZORDER BY (col1, col2)` jobs; it uses Z-order curve space-filling to cluster on up to 4 dimensions. Problem: clustering invalidates on every write, so you must re-run OPTIMIZE frequently, and choosing Z-ORDER columns is a one-time decision. **Liquid Clustering** (2024+, Delta Protocol v7+) is Databricks' replacement: you declare clustering keys at table creation (`CLUSTER BY (col1, col2)`), and Delta automatically, incrementally maintains the layout as data is written. No more manual OPTIMIZE scheduling for clustering; no write-time invalidation. Liquid is strictly better for new tables. Migration from Z-ORDER to Liquid is a one-way rewrite. In 2026, Liquid Clustering is the recommended default for all new Delta tables on Databricks.

    Related Terms

    Related Tools

    DatabricksApache SparkDelta Sharingdelta-rsTrino

    Learn More