What is Data Lake?

A data lake is the cheap, flexible, schema-later cousin of the data warehouse. Instead of modeling data before it lands (schema-on-write, the warehouse approach), a data lake stores raw files — CSV, JSON, Parquet, Avro, logs, images, video, PDFs, anything — in cheap object storage (S3, ADLS, GCS) and lets you apply schema at query time (schema-on-read). The original promise: store everything forever, figure out what to do with it later. The original failure mode: "data swamps" where nobody can find anything, governance evaporates, and the lake becomes a graveyard of abandoned datasets.

What data lakes look like in 2026

The pure "dump files in S3" data lake is largely obsolete. Modern lakes sit at the center of a lakehouse architecture: Parquet files in S3/ADLS/GCS + an ACID table format layer (Apache Iceberg, Delta Lake, Apache Hudi) + SQL query engines (Trino, Athena, Databricks SQL, Snowflake external tables) + a data catalog (AWS Glue, Unity Catalog, Polaris, Nessie). This gives you warehouse-like ACID semantics, time travel, schema evolution, and SQL performance — on top of open file formats and cheap object storage. Typical use cases: large-scale ML training data, log analytics, IoT/sensor data, media/unstructured data, and increasingly as the primary analytics store replacing separate warehouse + lake.

Field story

An adtech client was storing 8PB of event data across a mix of Snowflake (last 90 days, hot), S3 Parquet (historical, cold) with a homegrown partition scheme. Querying anything older than 90 days required spinning up an EMR cluster and 2-hour job times. They were paying Snowflake storage for 90 days of hot data (~$180K/year) while the cold S3 data was effectively un-queryable. We migrated everything to Iceberg on S3, with Snowflake external tables for recent queries and Trino for historical analytics. All 8PB became directly SQL-queryable with minute-level latency. Snowflake storage cost dropped 80% (only metadata and hot-cache in SF now), total analytics infrastructure cost dropped 50%, and analysts stopped abandoning historical queries because "it's too slow." The lakehouse architecture let them decouple storage from compute and format from query engine — which is the whole point.

A data lake is a centralized repository designed to store, process, and secure large volumes of data in any format—structured, semi-structured, or unstructured. Unlike data warehouses that require data to be structured before storage, data lakes accept raw data as-is.

Data Lake vs Data Warehouse

Zones

1. Raw/Bronze Zone: Data exactly as received from sources
2. Cleansed/Silver Zone: Validated, deduplicated, standardized data
3. Curated/Gold Zone: Business-ready, aggregated datasets

Data Lakehouse

• Delta Lake (Databricks): ACID transactions on data lakes
• Apache Iceberg: Open table format for huge datasets
• Apache Hudi: Incremental data processing

Cloud Data Lake Platforms

• AWS: S3 + Glue + Athena + EMR
• Azure: Data Lake Storage + Synapse + Databricks
• Google Cloud: GCS + Dataproc + BigQuery

Challenges and Solutions

• Data Swamp: Without governance, lakes become unusable → Use data catalogs
• Query Performance: Raw files are slow → Use table formats (Delta, Iceberg)
• Security: Sensitive data exposure → Implement row/column level security

Data lake failure modes that repeat across projects:

• The data swamp. Without a catalog, schema enforcement, and ownership, a lake becomes an uncurated dump of orphaned files. Nobody knows what's current, what's abandoned, or what schema a file has. Adopt a table format (Iceberg/Delta/Hudi) + catalog (Unity, Glue, Polaris) from day one.
• Small-files problem. Writing millions of tiny Parquet files (< 10MB) destroys read performance — each query opens thousands of files. Compact small files into 100MB-1GB files via Iceberg/Delta optimize operations or periodic compaction jobs.
• No ACID guarantees on raw Parquet. Writing to the same directory from two concurrent jobs produces partially-written files and inconsistent reads. Table formats (Iceberg, Delta, Hudi) solve this with transaction logs. Don't rely on raw Parquet for production data you write to.
• Partition explosion. Partitioning by every possible column (year/month/day/hour/user/region) creates millions of tiny directories with one file each. Choose partitions deliberately — typically 1-3 levels for most tables.
• Schema evolution unplanned. Source systems change columns; ingestion pipelines break silently. Use Iceberg/Delta schema evolution, enforce schema-on-write in the raw layer, and monitor for schema drift.
• Query performance variance. Unlike a warehouse, query performance on a lake depends heavily on file layout, partitioning, clustering, and engine choice. A badly-laid-out table that takes 10 seconds on Athena can take 40 minutes on Spark without optimization.
• Data access governance as an afterthought. Object-level S3 permissions don't scale to fine-grained (column-level, row-level) access control. Use Unity Catalog, AWS Lake Formation, or Polaris for proper RBAC with column masking.
• Lake and warehouse as separate silos. Teams building "lake for data science, warehouse for BI" end up with duplicated data, inconsistent logic, and reconciliation pain. Modern architecture: one storage layer (lake), multiple query engines, lakehouse table formats unify the two.
• Ignoring cost of queries. Athena / BigQuery externals / Snowflake external tables charge per TB scanned. A query with no partition filter on a 100TB lake can cost $500 in one click. Enforce partition filters, use table stats, set up cost alerts.
• No retention / lifecycle policy. Lakes "store everything forever" until storage cost forces a cleanup. Define retention per dataset, use S3 lifecycle rules to tier to Glacier, and delete/archive abandoned tables.