What is Table Format?

Table formats are the most important piece of lakehouse infrastructure that most engineers still misunderstand. The confusion starts with the name — a "table format" is NOT a file format. Parquet is a file format (how bytes are laid out in one file); Iceberg / Delta Lake / Hudi are table formats (how many files are logically organized into one table, with transactions, versioning, and schema). A table in Iceberg is a pointer to a metadata tree that points to manifest files that point to data files (Parquet). Mutating the table means writing a new metadata tree and atomically swapping the pointer. That swap is what gives you ACID on top of eventually-consistent object storage — without which the lakehouse simply doesn't work.

Why this matters in 2026

Before table formats (pre-2018), doing MERGE on a data lake meant stopping all readers, rewriting the whole partition, hoping nothing failed mid-write, and restarting readers. Concurrent writes corrupted data. Schema changes required a full table rewrite. With Iceberg/Delta/Hudi: append a new snapshot, swap the metadata pointer, readers see old or new but never inconsistent data. Every modern lakehouse capability — time travel, hidden partitioning, incremental reads, compaction in place, schema evolution without rewrite — is a table-format feature, not a query-engine feature. Pick the format first; the engine is secondary.

Field story

A fintech client had 80TB of Parquet in S3 partitioned by day, with 6 Spark jobs writing nightly and Athena dashboards reading continuously. Every Friday (high-volume deploy day) they'd get corrupt dashboards because two jobs wrote overlapping partitions simultaneously, and readers saw partial results. They also couldn't fix schema bugs retroactively — renaming a column meant rewriting the entire history. We migrated to Iceberg on the same S3 bucket with AWS Glue as the catalog. No more corruption (Iceberg's snapshot isolation handles concurrent writes), schema evolution became a metadata-only operation (ALTER TABLE ... RENAME COLUMN is instant, no file rewrite), and they gained time travel (SELECT * FROM table FOR VERSION AS OF 123) for debugging pipeline bugs against yesterday's data. Six engineers stopped firefighting weekend data issues. The migration itself was ~6 weeks — not trivial, but pays back every week thereafter.

A Table Format is a metadata layer that sits on top of data files (like Parquet) in a data lake, organizing them into logical tables with warehouse-like capabilities. Without a table format, a 'table' in a data lake is just a folder of files with no transactional guarantees. Table formats solve this by tracking which files belong to a table, enabling ACID transactions, time travel, and schema evolution.

File Format vs Table Format

This distinction is critical and often confused:

How Table Formats Work

Query Engine (Spark, Trino, Flink)
        ↓
Table Format Metadata Layer
  - Snapshot: list of all data files in current version
  - Manifest: statistics about each data file
  - Schema: column names, types, evolution history
        ↓
Data Files (Parquet/ORC on S3/GCS/ADLS)

Every write creates a new snapshot pointing to a new set of files. Old snapshots remain accessible for time travel.

Key Features All Table Formats Provide

• ACID Transactions: Writes either fully succeed or fully fail — no corrupt partial states
• Time Travel: Query data as it existed at any previous snapshot
• Schema Evolution: Add, drop, rename columns without rewriting data
• Partition Evolution: Change partitioning strategy without rewriting data (Iceberg)
• Concurrent Writes: Multiple writers can safely write simultaneously

The Big Three: Iceberg vs Delta Lake vs Hudi

Choosing a Table Format

• Apache Iceberg: Best default choice — widest engine support, best partition evolution, adopted by Snowflake, BigQuery, AWS, and Databricks
• Delta Lake: Best if you are all-in on Databricks — tight Spark integration, Unity Catalog governance
• Apache Hudi: Best if your primary use case is CDC / incremental upserts from databases

Table format pitfalls that break production lakehouses:

• Treating file format as table format. Writing Parquet directly to S3 and calling it a table is not a lakehouse — there's no transaction log, no snapshot isolation, no catalog awareness. Pick Iceberg, Delta, or Hudi from day one for any production table that gets writes.
• Mixing catalog implementations. Iceberg supports multiple catalogs (REST, Glue, Hive, Nessie, Unity) — picking one and standardizing is critical. Tables created in one catalog aren't visible from another without migration.
• Metadata file explosion. Every write creates a new manifest and a new snapshot. Without periodic expire_snapshots + remove_orphan_files (Iceberg) or VACUUM (Delta), metadata grows unboundedly, slowing planning and inflating storage cost.
• Small-files compounding. Streaming writes create thousands of small Parquet files per partition. Without scheduled compaction (rewrite_data_files, OPTIMIZE), query performance degrades 10-50x over weeks.
• Concurrent write failures. Two jobs writing to the same partition simultaneously cause one to fail with a commit conflict. Build retry logic, or partition writes so they don't overlap; Hudi handles concurrent updates better with MVCC, but Iceberg/Delta need retries at the app layer.
• Version-pinning mismatches between engines. Engine A writes Iceberg v2 with row-level deletes; engine B reads with an older library that doesn't understand delete files → returns wrong results. Keep Iceberg/Delta library versions aligned across all writers and readers.
• Vendor-specific extensions silently lock you in. Delta Lake has deletion vectors, column mapping, and liquid clustering that only Databricks fully supports; other engines read a degraded view. Iceberg has fewer such extensions. Audit which features you're using before claiming "vendor-neutral."
• Hidden partitioning doesn't help old queries. Iceberg's hidden partitioning lets you partition by month(ts) without adding a column, but queries still need to filter on ts — engines do the partition translation. Adding partitioning to an existing table doesn't retroactively partition existing files.
• Rollback isn't a free escape hatch. Time travel reads old snapshots, but rolling back to an old snapshot (making it the current) abandons all writes since — including other teams' writes. Rollback needs coordination across the whole pipeline.
• Schema evolution edge cases. Adding a column is safe everywhere. Renaming needs column IDs (Iceberg handles; Delta needs column mapping enabled at creation). Dropping a column in Hudi is more complex. Type widening (int → bigint) works; narrowing doesn't. Test schema changes in a copy before applying to production.