What is Apache Iceberg?

Apache Iceberg has become the de facto open table format for the modern lakehouse. Created at Netflix in 2017 and now a Apache top-level project, it solves the reliability problems that made earlier data lakes (Hive Metastore + Parquet folders on S3) unusable in production: no ACID, no safe concurrent writes, partition-change-requires-rewrite, corrupted tables from failed jobs. In 2026, every major cloud warehouse and compute engine supports Iceberg natively: Snowflake Iceberg tables, Databricks Iceberg support via Uniform, BigQuery external Iceberg tables, AWS S3 Tables (native Iceberg managed service), Trino, Spark, Flink, DuckDB, ClickHouse, Dremio — the list keeps growing. The ecosystem consolidation around Iceberg is the single most important trend in data engineering this decade because it means "pick any engine, read/write the same tables" actually works now.

What Iceberg gives you vs raw Parquet

Four things that fundamentally didn't exist on data lakes before: (1) ACID — concurrent writes serialize via optimistic concurrency control on the catalog, no more corrupted tables from race conditions; (2) Time travel — every write creates a snapshot; query any past state with FOR VERSION AS OF or FOR TIMESTAMP AS OF; (3) Schema evolution — rename, drop, reorder, widen columns as metadata-only operations (stable column IDs, not names or positions); (4) Partition evolution — change partitioning from month to day without rewriting a single data file. Add hidden partitioning (PARTITIONED BY (month(ts)) without adding a month column), per-file statistics for pruning, and REST Catalog for engine-neutral discovery, and Iceberg is basically what a data warehouse gives you on open files.

Field story

A media company had 240TB of Parquet on S3 with Hive partitions (year/month/day), partition repair jobs that took 2 hours, and chronic issues where Spark jobs writing overlapping partitions produced inconsistent Athena results. We migrated to Iceberg using the migrate-in-place procedure (CALL system.register_table) — no data copy needed, just rewrote the metadata layer. Writes became ACID, Athena/Trino/Snowflake all queried the same tables with consistent results, and partition evolution let us transparently shift from day to hour partitioning for hot tables without touching existing files. AWS Glue catalog held the metadata. Storage cost unchanged (same Parquet files); reliability went from "three incidents per week" to zero in 6 months. The 4-week migration paid back in quarter 2.

Apache Iceberg is an open table format designed for petabyte-scale analytic datasets. Created at Netflix, it solves the fundamental problems that made data lakes unreliable — by bringing ACID transactions, time travel, schema evolution, and partition evolution to files sitting in object storage (S3, GCS, ADLS).

The Problem Iceberg Solves

Traditional data lakes stored data as raw files (Parquet/ORC) with a Hive Metastore tracking partitions. This architecture had critical flaws:

• No ACID: Concurrent writes could corrupt data
• Partition Lock-in: Changing partition schemes required rewriting all data
• No Time Travel: Can't query historical states
• Schema Rigidity: Adding columns was painful
• Small File Problem: Too many small files destroyed performance

How Iceberg Works

Query Engine (Spark/Trino/Flink)
        │
        ▼
┌─────────────────┐
│  Iceberg Catalog │ (Where is the latest table state?)
│  (REST/Glue/Nessie)│
└────────┬────────┘
         │
┌────────▼────────┐
│ Metadata Layer  │ (Manifest files + Manifest Lists)
│  - Snapshots    │
│  - Manifests    │
│  - Statistics   │
└────────┬────────┘
         │
┌────────▼────────┐
│  Data Files     │ (Parquet/ORC/Avro in S3/GCS/ADLS)
└─────────────────┘

Key Features

ACID Transactions

• Atomic commits: writes either fully succeed or fully fail
• Snapshot isolation for concurrent reads and writes
• No more corrupted tables from failed jobs

Time Travel

-- Query data as it was 24 hours ago
SELECT * FROM orders
FOR SYSTEM_TIME AS OF TIMESTAMP '2026-02-26 09:00:00';

-- Roll back to a previous snapshot
ALTER TABLE orders ROLLBACK TO SNAPSHOT 12345;

Schema Evolution

• Add, drop, rename, or reorder columns without rewriting data
• Type promotion (int → long, float → double) is safe
• No downtime for schema changes

Partition Evolution

• Change partition schemes without rewriting data
• Start with daily partitions, switch to hourly — Iceberg handles it
• Hidden partitioning: users don't need to know partition columns

Engine Compatibility

• Apache Spark, Trino/Presto, Flink, Snowflake, BigQuery, Dremio, StarRocks

Iceberg vs Delta Lake vs Hudi

Iceberg gotchas encountered running it in production:

• Catalog choice locks operational patterns, not data. Iceberg data is portable between catalogs, but moving prod tables between Hive Metastore, AWS Glue, Nessie, Polaris, and Unity is a multi-week project. Standardize on one catalog per environment; use REST Catalog for multi-engine neutrality.
• Metadata file explosion. Every commit writes a new metadata.json + manifest list + manifests. Long-running tables accumulate thousands of metadata files. Run expire_snapshots and remove_orphan_files on a schedule (daily/weekly) or planning time degrades.
• Small-files in streaming writes. Flink/Spark Streaming writing every few seconds creates thousands of tiny files. Schedule rewrite_data_files compaction to target 256MB-1GB files; without it, scan performance collapses.
• V1 vs V2 spec differences matter. Iceberg V2 (2022+) introduced row-level deletes (position deletes and equality deletes) to support efficient MERGE/UPDATE/DELETE. V1-only readers can't interpret delete files → return wrong results. Upgrade all readers/writers to V2-capable versions before enabling row-level ops.
• Concurrent write retries aren't free. Two jobs MERGE-ing into the same partition produce commit conflicts; default Spark behavior retries with exponential backoff, but high-concurrency workloads can exhaust retries. Partition writes deterministically or use Hudi's MVCC approach for high upsert rates.
• Hidden partitioning isn't magic. PARTITIONED BY (month(ts)) adds a partition column derived from ts, but queries must still filter on ts for pruning to engage — Iceberg translates ts filters to month-partition filters automatically. Queries filtering on an unrelated column don't benefit.
• Partition evolution doesn't retroactively repartition. Changing partition spec from day to hour applies to NEW writes only. Old data stays in its original layout; queries crossing the boundary read both layouts. Plan migration cutover explicitly.
• Iceberg snapshots retain data files until expired. DELETE FROM t WHERE ... logically removes rows, but physically the files stay until expire_snapshots removes unreferenced ones. GDPR deletion compliance requires both the logical delete AND explicit snapshot expiration within the compliance window.
• Engine-specific write performance varies wildly. Spark Iceberg writer is mature; Flink is good; Trino writes work but aren't optimized; some engines only read. Benchmark write throughput for YOUR engine on YOUR data before committing to Iceberg as primary storage.
• REST Catalog adoption is new; test interop. The Iceberg REST Catalog API (standardized 2023) is the path to engine-neutral governance, but different catalog implementations (Polaris, Nessie, Unity REST, custom) have subtle spec deviations. Test with every engine that'll access the catalog.