Your team has a 200GB Parquet file on S3. Someone suggests running the analysis in Spark. You spin up a four-node cluster, configure executors, tune shuffle partitions, wait three minutes for the cluster to initialize, wait fourteen minutes for the job to run, tear the cluster down, and get back a number.
The same query in DuckDB runs on a single VM in four minutes, costs one-twentieth as much, and requires zero cluster management. You didn’t need distributed computing. You needed a fast query engine — and you reached for a freight train when a Ferrari would have done the job in a quarter of the time.
This is the most expensive habit in modern data engineering, and it’s happening in thousands of production pipelines right now. Not because engineers are incompetent. Because the “big data playbook” — spin up Spark, process everything, shut down cluster — was written when cloud VMs had 8GB of RAM. A $300/month VM in 2026 has 128GB of RAM and NVMe SSDs that can sustain 3GB/s reads. The old rule — “data doesn’t fit in memory, use a cluster” — is eroding fast. And DuckDB is the reason.
TL;DR
→ DuckDB is an embedded, in-process, columnar OLAP database. No server. No cluster. No JVM. Install in one `pip install duckdb`. Query CSV, Parquet, JSON on S3 with standard SQL.
→ For 50GB–1TB OLAP workloads on Parquet, DuckDB is typically 3–10x faster than Spark and 10–20x cheaper because it eliminates network shuffle, JVM overhead, and cluster management overhead.
→ Real benchmark: 500GB Parquet (stock trades, time-series aggregation + groupby). Spark on a 4-node cluster: 14 minutes. DuckDB on a single 16-core, 128GB VM: ~4 minutes. Cost ratio: 1:20.
→ DuckDB wins: SQL-first OLAP on Parquet/CSV/JSON, data that fits on one machine (up to ~1TB), CI/testing pipelines, local development, cost-sensitive workloads.
→ Spark still wins: petabyte-scale distributed ETL, Structured Streaming for real-time pipelines, MLlib integration, cross-node joins on truly massive datasets, fault tolerance across hundreds of nodes.
→ The practical hybrid: DuckDB for local dev and CI (zero startup time vs Spark’s 3-minute init); Spark for production TB+ workloads. Most teams using Spark everywhere could do 80% of their work on DuckDB.
→ Polars is in this conversation too: Rust-based DataFrame API, great for Python-first teams who don’t want SQL. DuckDB for SQL, Polars for code. They’re complementary, not competitive.
→ MotherDuck extends DuckDB to a managed cloud warehouse — multi-user, persistent storage, connectors — for teams that outgrow single-node but don’t want Spark’s complexity.
What DuckDB actually is (and isn’t)
DuckDB is an OLAP (analytical) database engine that runs inside your process. Not a server. Not a service. An embedded library, like SQLite, except built from scratch for analytical queries instead of transactional ones. You pip install duckdb and start querying. No cluster to manage. No JVM. No configuration files. No driver program. No shuffle partitions to tune.
Under the hood, DuckDB uses vectorized execution: it processes data in columnar chunks, exploiting CPU SIMD instructions to handle hundreds of rows per clock cycle. It reads Parquet files with column pruning and predicate pushdown — it doesn’t load the whole file into memory, it skips the pages and row groups it doesn’t need. The result is query performance that competes with Spark on single-machine workloads at a fraction of the infrastructure cost.
What DuckDB is not: a distributed system. It runs on one machine. If your data genuinely cannot fit on one machine or you need streaming, DuckDB is not your answer. But here’s the part of the conversation that’s rarely said clearly: most data engineering workloads in production are not distributed workloads. They’re workloads that teams are running on distributed infrastructure out of habit, convention, or because that’s what the senior engineer learned in 2019.
The benchmark that changes how you think about this
Real benchmark numbers: DuckDB eliminates cluster overhead, JVM serialization, and network shuffle. Wins by 3–10x on OLAP queries up to ~1TB. Cost difference is even larger than time difference.
The numbers that matter come from a controlled test on a 500GB Parquet dataset of stock trade records: time-series aggregation with a multi-column groupby, the kind of query that sits at the core of most analytical pipelines.
Spark on a 4-node cluster: 14 minutes end-to-end (including cluster init and tear-down overhead), at cluster-runtime node pricing. DuckDB on a single 16-core, 128GB RAM VM: ~4 minutes, no init overhead, running as a single process. Cost ratio: roughly 20:1 in DuckDB’s favor.
Why does DuckDB win? Spark pays for distributed resilience even when you don’t need it. It shuffles data across network to prepare for cross-node joins that will never happen because the data fits on one machine. It serializes and deserializes through JVM objects. It manages a driver program and executor lifecycle. All of that overhead is real cost — not just money but latency. DuckDB simply reads columnar Parquet from local NVMe, pushes predicates down to skip file sections, and runs vectorized aggregation in CPU cache. No network. No JVM. No shuffle.
For smaller queries: a grouped aggregation benchmark (sales by region on 10M rows) took DuckDB 2.5 seconds, Spark in local mode 8 seconds. A join on two 20M-row tables: DuckDB under 5 seconds, Spark 15 seconds. Important caveat: these are single-machine comparisons. At true petabyte scale, Spark’s distributed architecture wins because DuckDB simply runs out of hardware. But most teams never get to petabyte scale, and the ones who believe they have are often running 200GB datasets on Spark clusters because nobody revisited the architecture decision from three years ago.
The cost math most teams never do
Assume you have a 300GB daily analytics pipeline running on Spark. A modest cluster: 4 worker nodes, each 8 cores, 32GB RAM. You run it twice a day. On AWS, that’s roughly $0.30/node-hour, four nodes, maybe 45 minutes per run. That’s $0.90/run, $1.80/day, $657/year. Sounds manageable.
Now add: the 15 minutes of Spark startup overhead per run ($0.30 wasted per run), the 20% of engineer time spent debugging shuffle OOM errors and executor failures, the CI runs that take 12 minutes instead of 2 because you’re testing against a Spark local context instead of DuckDB.
The DuckDB alternative: a single c6i.4xlarge instance (16 cores, 32GB RAM), on-demand at $0.68/hour. Run it twice a day, average 8 minutes per run. That’s $0.18/day, $66/year. Plus near-zero maintenance overhead. For a 300GB pipeline, you’re looking at $591/year saved, plus meaningfully less engineer time.
For larger teams running many such pipelines, multiply accordingly. The savings aren’t theoretical.
Where DuckDB actually fits in your stack
The decision is simpler than it looks: does your data fit on one machine? If yes, DuckDB is almost always the right choice. If not, Spark. The hard part is being honest about your actual data size.
The practical split is cleaner than most discussions make it sound:
Use DuckDB when: your data fits on one machine (roughly up to 1TB with modern hardware), the workload is SQL-first analytical queries, you’re building CI/testing pipelines (DuckDB starts in milliseconds; Spark in minutes), you’re doing local development and iteration, or you’re running cost-sensitive batch workloads where cluster overhead is pure waste.
Use Spark when: your data physically cannot fit on one machine or needs distributed partitioning, you’re building streaming pipelines with Structured Streaming and need exactly-once semantics, you need MLlib for distributed model training, you have genuinely petabyte-scale joins that require cross-node shuffles, or you need fault tolerance across hundreds of nodes where a single node failure would be catastrophic.
The hybrid that most teams are converging on: DuckDB in local development and CI (the “inner loop”), Spark in production for workloads that actually need distribution. This is the pattern Zach Wilson has written about: DuckDB for fast local testing and EDA, Spark for the production pipelines processing billions of events per hour. The tools aren’t competing for the same role — they’re occupying different rungs of the same ladder.
DuckDB’s SQL is genuinely better to write
Benchmark numbers aside, the developer experience gap is significant. DuckDB has shipped SQL extensions that most engineers discover and then can’t go back from.
EXCLUDE lets you select all columns except a few: SELECT * EXCLUDE (internal_id, created_at) FROM orders. No more writing out 40 column names. COLUMNS with regex lets you pattern-match columns: SELECT COLUMNS('amount.*') FROM orders. QUALIFY filters on window function results without a subquery. Function chaining — first_name.lower().trim() — reads like Python. These aren’t gimmicks; they’re hours of saved typing at scale.
DuckDB also queries files directly without loading them: SELECT * FROM 's3://my-bucket/data/*.parquet' WHERE event_date = '2026-06-01'. No ETL to load into a table first. No Spark session to initialize. The file is the table.
The gotchas nobody warns you about
DuckDB’s concurrency model is not Postgres. DuckDB supports multiple readers, but only one writer at a time. If you’re building a production system where multiple processes need to write simultaneously, you’ll hit locking issues quickly. MotherDuck solves some of this, but the base DuckDB model is single-writer. Don’t architect a high-write-concurrency system on raw DuckDB without understanding this.
Memory is managed, but you can still OOM. DuckDB’s query engine is smart about memory, using streaming execution to avoid materializing entire result sets. But complex multi-join queries with many intermediate results can still consume more RAM than your VM has. Size your VM with headroom — if your dataset is 100GB, don’t run it on a 128GB instance. Leave 30–40% for overhead.
DuckDB is not a transactional database. It has ACID transactions, but it’s optimized for append-heavy analytical workloads, not OLTP update/delete patterns. Using it as a general-purpose application database is the wrong tool for the job.
Distributed DuckDB exists but isn’t production-ready at Spark scale. There’s a distributed extension project, but it’s nowhere near Spark’s maturity or fault tolerance. If you’re planning to “scale DuckDB to Spark scale” — that’s not the right mental model. When you outgrow single-node DuckDB, the answer is MotherDuck (managed, serverless) or Spark (distributed, self-managed). Not “distributed DuckDB.”
The Polars question. If your team writes Python-first data pipelines, Polars is a serious alternative to DuckDB for single-machine workloads. Polars is a Rust-based DataFrame library — think pandas but 10–30x faster with a proper lazy execution model. It doesn’t support SQL natively (though it has SQL-like expressions). The practical split: DuckDB for SQL-first analytical queries; Polars for code-first transformations. Many teams use both: DuckDB to query and load Parquet, Polars to transform the resulting DataFrame. They compose cleanly together.
When to migrate existing Spark pipelines
Migrating an existing Spark pipeline to DuckDB isn’t always worth the effort even if DuckDB would be faster. Before migrating, ask three questions:
Is the Spark pipeline causing operational pain (OOM errors, long startup times, expensive debugging)? Is the dataset under 1TB and not expected to grow past single-node capacity? Does the pipeline use only Spark SQL or DataFrame operations, not Spark-specific features like Structured Streaming or MLlib?
If all three are yes, the migration is usually a morning’s work: translate PySpark DataFrames to DuckDB SQL, replace S3 Spark readers with DuckDB S3 file queries, run both in parallel for one week, decommission the cluster. Most SQL-based Spark pipelines translate directly because DuckDB’s SQL is a superset of what most teams actually use in Spark SQL.
If any answer is no, keep Spark for that pipeline and use DuckDB for new workloads below the threshold.
The one principle
Match the tool to the actual data size, not the data size you imagine you might have someday. Spark is the right answer for distributed workloads that genuinely cannot fit on one machine. It is not the right answer for a 200GB daily pipeline just because someone wrote the original architecture when “big data” was the thing to say. In 2026, a single cloud VM has enough RAM, CPU, and NVMe storage to handle most analytical pipelines that companies think require distributed computing. DuckDB is the proof of that claim.
Related reading: DuckDB S3 extension docs · MotherDuck: Managed DuckDB in the cloud · Snowflake Iceberg v3: When to Migrate · dbt Fusion: 30x Faster Parsing · Snowflake Query Execution: What Really Happens