An open-source data lakehouse platform that provides ACID transactions, record-level insert/update/delete, and incremental data processing on data lakes.
Apache Hudi (Hadoop Upserts Deletes and Incrementals) was the first of the three major open table formats — created at Uber in 2016 to solve a problem that Parquet-on-S3 couldn't: how do you ingest hundreds of millions of trip-update events per day into a data lake when 95% of those events are updates to existing records, not appends? The answer Uber built — and donated to the Apache Foundation — is Hudi's defining strength: record-level upserts with incremental pull semantics. If Iceberg is optimized for analytical queries and schema agility, and Delta is optimized for Databricks Spark pipelines, Hudi is optimized for streaming CDC ingestion and upsert-heavy workloads. In 2026, while Iceberg has clearly won the general-purpose lakehouse battle, Hudi retains a specific niche: high-throughput CDC sinks, GDPR-compliant delete workflows, and incremental ETL pipelines where "give me rows that changed since timestamp T" is a first-class operation.
Hudi's most distinctive feature is exposing two table types with different read/write tradeoffs. Copy-on-Write (CoW): updates rewrite the entire Parquet file containing the affected records. Reads are pure Parquet, maximum query speed. Writes are expensive — updating 1% of rows in a 10GB file rewrites 10GB. Good for periodic batch upserts, not streaming. Merge-on-Read (MoR): updates append to row-based delta log files (Avro); reads merge base Parquet + delta logs on the fly. Writes are cheap (append-only); reads are slower until compaction (a background job that merges logs into Parquet) catches up. You get two read modes on MoR tables — read-optimized (skip deltas, stale but fast) and snapshot (merge deltas, current but slower). Choose CoW for read-heavy marts with daily batch updates; choose MoR for streaming ingest where writes happen constantly.
A ride-sharing analytics client had a Kafka → Spark → Parquet pipeline ingesting 200M trip-status events per day. Each trip generated 5-15 status updates over its lifecycle (requested → matched → picked up → dropped off → rated), meaning 85% of daily volume was updates to existing rows. Their nightly full-rewrite ETL took 9 hours and was slipping SLA. We migrated to Hudi MoR tables with Kafka Connect → Hudi DeltaStreamer, record key = trip_id, precombine field = event_timestamp. Streaming ingest ran continuously; compaction ran hourly. End-to-end latency dropped from 9 hours to 12 minutes. The incremental query feature (hoodie.datasource.query.type=incremental) let downstream dbt models process only trips that changed since last run — 95% compute reduction on Silver tables.
Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lakehouse storage framework that brings database-like capabilities to data lakes. Originally created at Uber to solve their massive-scale incremental data processing challenges, Hudi enables efficient upserts, deletes, and incremental queries on data stored in cloud storage like S3 or GCS.
Traditional data lakes store immutable files (Parquet, ORC). To update a single record, you must rewrite entire partitions — an expensive and slow operation. Hudi solves this by introducing:
| Feature | Hudi | Iceberg | Delta Lake |
|---|---|---|---|
| Origin | Uber | Netflix | Databricks |
| Upsert Performance | Excellent | Good | Good |
| Read Performance | Good | Excellent | Excellent |
| Engine Support | Broad | Broadest | Spark-centric |
| Incremental Processing | Native | Via snapshots | Via CDF |
Hudi production gotchas:
hoodie.metadata.enable=true). Tables created pre-0.10 may not have MDT; enable it with hoodie.metadata.enable.sync=true.hoodie.parquet.small.file.limit, hoodie.parquet.max.file.size), you end up with thousands of 10MB files. Set target file size explicitly (128MB-512MB) and enable auto file sizing..hoodie/ for time travel and rollback, but doesn't auto-expire. Enable cleaner policies (hoodie.cleaner.policy=KEEP_LATEST_COMMITS) and set retention explicitly or .hoodie/ grows unbounded.Apache Hudi is used for high-throughput ingest and upsert workloads on data lakes — specifically cases where incremental, record-level mutations are the primary write pattern. Common uses: (1) CDC ingestion from transactional databases (MySQL/Postgres → Debezium → Kafka → Hudi); (2) streaming event ingestion where events update rather than only append (user profile changes, order status updates, IoT sensor readings with corrections); (3) GDPR/CCPA compliance workflows needing efficient record-level deletes across billions of rows; (4) incremental ETL pipelines that read only rows changed since last run; (5) slowly changing dimensions on a lake; (6) replacing expensive full-rewrite batch pipelines with append+merge patterns. Hudi excels where write throughput and upsert performance matter more than multi-engine query flexibility.
Both are open table formats adding ACID and schema management to data lakes, but they optimize for different workloads. Hudi is upsert-first — it exposes Copy-on-Write and Merge-on-Read table types, native record-level keys, and incremental queries; ideal for streaming CDC and write-heavy pipelines. Iceberg is query-first — it prioritizes efficient analytical reads, engine neutrality, partition evolution, and catalog interop; ideal for multi-engine lakehouses (Spark + Trino + Snowflake + BigQuery reading the same tables). In 2026, Iceberg has won broader ecosystem adoption (Snowflake, BigQuery, Databricks Uniform, AWS S3 Tables all standardized on it); Hudi retains a niche in streaming-ingest and upsert-heavy use cases. If you're choosing today: Iceberg unless upsert throughput is your defining constraint.
Hudi's two table types trade write cost for read cost. **Copy-on-Write (CoW)**: updates rewrite entire Parquet files containing affected rows. Writes are expensive (rewriting a 1GB file to update a single row); reads are pure Parquet so query performance is maximal. Best for batch pipelines with occasional updates, and read-heavy analytical tables. **Merge-on-Read (MoR)**: updates append to row-based Avro log files co-located with the Parquet base file; a background compaction job periodically merges logs into Parquet. Writes are cheap (append-only); reads are slower (merge base + deltas on the fly) until compaction catches up. MoR exposes two read modes — read-optimized (skip deltas, stale) and snapshot (merge deltas, current). Rule of thumb: CoW for periodic batch loads; MoR for streaming ingest or high-frequency upserts.
Yes. Apache Hudi is open-source software licensed under Apache License 2.0, free for any commercial or non-commercial use. The code lives at github.com/apache/hudi and the community is Apache Software Foundation-governed. You're free to self-manage Hudi on your own Spark/Flink clusters with no licensing cost. Managed Hudi services exist for operational convenience: AWS EMR (first-party Spark+Hudi support), AWS Glue (built-in Hudi connector), Onehouse (cloud-managed Hudi platform founded by Hudi's creator Vinoth Chandar), and Databricks (Hudi reads/writes via Spark). These managed platforms charge for compute and managed-service value-adds, but Hudi itself remains free.
Yes but with significant limitations. Snowflake can READ Hudi tables via external tables or Iceberg interop if the Hudi data is published in a compatible layout — but Snowflake has no native Hudi writer and no first-party integration equivalent to Iceberg tables. In practice: Hudi ingestion runs on Spark or Flink; the Hudi tables live on S3/ADLS/GCS; Snowflake external tables (or Iceberg views over Hudi data) provide read access. If your primary query engine is Snowflake and you care about write performance from Snowflake itself, Iceberg is the correct choice — Snowflake offers managed and external Iceberg tables with full read/write support. Hudi fits better when Snowflake is secondary to Spark/Flink ingestion.
An incremental query reads only the records that changed in a Hudi table between two points in time, rather than scanning the entire table. Syntax (Spark): `spark.read.format("hudi").option("hoodie.datasource.query.type", "incremental").option("hoodie.datasource.read.begin.instanttime", "<commit-timestamp>").load(path)`. This is Hudi's killer feature for downstream ETL: if your Silver layer runs every 15 minutes, it pulls only the ~1% of rows that changed since the last run, not the full Bronze table. Incremental queries work on both CoW and MoR tables. They enable efficient event-driven pipelines where each stage processes only upstream deltas — the foundation of Hudi-based streaming lakehouse architectures. Similar patterns exist in Iceberg (via snapshot diff) and Delta (via Change Data Feed), but Hudi built this capability first and it's most mature in Hudi.
Two background maintenance operations critical to MoR table health. **Compaction** merges delta log files (row-based Avro containing updates) into base Parquet files. Inline compaction runs every N commits (`hoodie.compact.inline.max.delta.commits=5`); async compaction runs as a separate Spark job. Without compaction, MoR snapshot reads get slower over time and log files accumulate. **Clustering** reorganizes data within a table to improve query performance — typically sorting rows by a high-cardinality filter column so predicate pruning engages. For example, clustering by `user_id` makes `WHERE user_id = X` queries 10-100x faster. Clustering runs as a separate job, blocks writes on affected partitions during the operation, and produces new file groups. Schedule both: compaction hourly for streaming MoR; clustering daily/weekly for hot query patterns.