What is Apache Spark?

I've been running Apache Spark in production for close to a decade, across EMR, Databricks, self-managed YARN, and now Kubernetes. The honest take in 2026: Spark is no longer the obvious default it was in 2018, but for workloads in the hundreds-of-gigabytes-to-terabytes range that don't fit cleanly into a warehouse, it's still the best tool I know. The challenger isn't "no-Spark" — it's warehouse-native transformation (dbt + Snowflake), lightweight engines (DuckDB, Polars) for smaller data, and Trino for federated SQL. Spark still wins when you need a unified engine for batch + streaming + ML over petabyte-scale lake storage.

Where Spark actually shines

1. Petabyte-scale ETL on data lakes. If your raw data lives in S3/GCS/ADLS as Parquet/Iceberg/Delta and you need to transform it at scale, Spark is the mature default — especially through Databricks' Photon engine or Delta-native optimizations.
2. Mixed workloads in one pipeline. A job that reads JSON, does complex windowed aggregations, trains an ML model, and writes to a warehouse? Spark handles the whole thing. Swapping four specialized tools for one is often worth the operational weight.
3. Streaming with exactly-once semantics. Structured Streaming backed by checkpoints gives you true exactly-once into idempotent sinks. It's one of the few open-source engines that gets this right out of the box.

A quick field story

At one company we migrated a nightly Spark-on-EMR pipeline (3-hour runtime, ~80 nodes) to Snowflake + dbt. Everything smaller than 500GB/day got faster and cheaper — dbt's warehouse-native approach was clearly better for classic ELT. But one job — a ~15TB clickstream aggregation with complex sessionization logic — stayed on Spark (moved to Databricks). On Snowflake it would have cost 4x more in credits. Lesson: Spark's edge is large-scale, compute-heavy, lake-resident workloads; warehouse-native ELT wins almost everywhere else.

Apache Spark is a unified, distributed analytics engine designed for large-scale data processing. It provides high-level APIs in Python (PySpark), Scala, Java, SQL, and R on top of an optimized DAG-based execution engine. Spark runs on clusters ranging from a single laptop to thousands of nodes, orchestrated by YARN, Kubernetes, Mesos, or a standalone scheduler.

Why Spark Exists

Spark was created at AMPLab (UC Berkeley) in 2009 to address the pain points of Hadoop MapReduce:

• Speed — in-memory processing and a smarter DAG optimizer made Spark 10-100x faster than MapReduce for iterative algorithms (ML training, graph computation).
• Ergonomics — DataFrames and SQL replaced the low-level map/reduce boilerplate. You can do in 10 lines of PySpark what took 100+ lines of MapReduce.
• Unified platform — one engine for batch, streaming, ML, and graph, so teams don't string together Hadoop + Storm + Mahout + Giraph.
• Rich connectors — native integrations for S3, Parquet, JDBC, Kafka, Delta, Iceberg, and more.

Core Components

Spark Core

The runtime: task scheduling, memory management, shuffle, fault recovery, I/O. RDDs (Resilient Distributed Datasets) are the low-level abstraction — immutable partitioned collections with lineage for recovery.

Spark SQL and DataFrames

The modern high-level API. Operates on DataFrames (typed or untyped) and compiles through Catalyst (logical + physical optimizer) and Tungsten (code generation). This is what 90% of Spark users interact with:

df = spark.read.parquet("s3://lake/events/")

(df.filter(df.event_type == "purchase")
   .groupBy("user_id")
   .agg(F.sum("amount").alias("total_spend"))
   .write.mode("overwrite").parquet("s3://lake/user_spend/"))

Structured Streaming

Streaming API built on DataFrames. Micro-batch (default, seconds) or continuous (millisecond) mode. Backed by checkpoints for exactly-once into idempotent sinks (Kafka, Delta, Iceberg).

MLlib

Distributed machine learning: classification, regression, clustering, collaborative filtering, feature engineering pipelines, hyperparameter tuning. For deep learning, Spark plays a feeding role while training runs in PyTorch / TensorFlow.

GraphX / GraphFrames

Graph processing — less used in 2026 as Neo4j and specialized graph engines have matured.

Execution Model in Plain Words

1. Driver runs your code, plans the query, and sends tasks to executors.
2. Executors run on worker nodes, hold cached data, and execute tasks in parallel.
3. A job is triggered by an action (show, write, count).
4. A job is broken into stages at shuffle boundaries.
5. A stage is broken into tasks (one per partition).
6. Shuffles redistribute data across executors — the dominant cost in most jobs.

Performance tuning = minimizing shuffle, tuning partition count, caching smartly, and choosing the right join strategy (broadcast vs sort-merge).

Ecosystem in 2026

• PySpark — dominant API; ~85% of Spark code in the wild.
• Databricks — managed Spark + Photon engine (C++ rewrite of execution) + Delta Lake + Unity Catalog. Commercial leader.
• AWS EMR / Glue — managed Spark on AWS; Glue for serverless ETL.
• GCP Dataproc — managed Spark on GCP; Dataproc Serverless for pay-per-job.
• Spark on Kubernetes — cloud-native, popular for self-managed deployments.
• Open-source Spark — still widely used; ASF project with regular releases.

Common Use Cases

1. Large-scale ETL on data lakes — terabyte+ Parquet/Iceberg/Delta transformations.
2. Streaming ingestion — Kafka → Spark → Delta/Iceberg with exactly-once.
3. ML pipelines — feature engineering at scale, batch inference, distributed training.
4. Ad-hoc analytics — interactive SQL via Databricks SQL or Spark Thrift Server.
5. Data migration — heavy lifting when moving between storage systems.

When Not to Use Spark

• Data fits on one machine (< ~100GB) — DuckDB or Polars will beat Spark in latency and cost.
• Warehouse-native ELT is viable — dbt + Snowflake/BigQuery is usually simpler and cheaper for classic transformation.
• Latency-sensitive point queries — Spark is batch-oriented, not OLTP.
• Small teams without Spark expertise — the operational weight is real.

Things that have bitten me or teams I've worked with:

• Shuffle is the enemy. 80% of Spark performance issues come down to too much shuffle, skewed partitions, or wrong join strategies. Learn to read the DAG in the Spark UI; EXPLAIN EXTENDED is your friend.
• Small files kill S3 performance. Writing millions of tiny Parquet files turns reads into a metadata storm. Use coalesce() before write or enable optimize-write (Delta/Iceberg).
• Driver OOM on collect(). Calling .collect() or .toPandas() on a large DataFrame loads it into the driver's memory — classic production incident. Use .write to storage instead, or sample first.
• Broadcast join size limit. Default broadcast threshold is 10MB; setting it too high silently causes OOM during join planning. Tune spark.sql.autoBroadcastJoinThreshold carefully.
• Python UDFs are slow. A Python UDF round-trips every row through the JVM↔Python boundary. Prefer built-in functions, pandas UDFs, or Spark SQL expressions. If you must use a Python UDF, vectorize it with Arrow.
• Caching is not free. .cache() pins data in executor memory; uncaching matters. Unused cached DataFrames cause GC pressure and evictions that hurt unrelated queries.
• Version skew on Databricks. Runtime upgrades (DBR 13 → 14 → 15) can silently change behavior for edge cases (NULL handling, timestamp semantics). Pin DBR versions for production jobs and test before upgrading.