An automated series of steps that extracts data from sources, transforms it, and loads it into a destination — the backbone of every data-driven organization.
Data pipeline is the catchall term for "the thing that moves data somewhere." In practice a production pipeline has four layers — ingestion, transformation, orchestration, and observability — and failing on any one of them will wake you up. The pipeline that works is not the one with the cleverest Spark job; it's the one where a failing step pages the right person, the retry logic is idempotent, the data contract is tested, and the lineage is visible when something downstream breaks.
Batch is simpler, cheaper, and easier to debug. Streaming gives you lower latency at the cost of operational complexity (exactly-once semantics, windowing, watermarks, backpressure, replay). The failure mode I see most: teams build streaming when batch would do, then spend months tuning Flink because nobody actually needs sub-minute latency. Decide by SLA: if the business can tolerate 15-minute freshness, batch wins. If it genuinely needs seconds (fraud, real-time personalization, operational alerts), commit to the complexity of streaming and don't pretend batch will scale into it.
On one engagement I replaced a hand-rolled ETL pipeline (Python cron jobs + S3 + custom retry shell scripts) with dbt + Dagster + Fivetran + Snowpipe. Not because the new stack was "more modern," but because the old one had no lineage, no observability, no backfill tooling, and every data engineer was spending 40% of their time firefighting. After the rewrite: same work, lineage graph from source to dashboard, failed runs auto-paged the right owner, backfills were one Dagster UI click. Firefighting dropped from 40% to about 5%. The lesson: a data pipeline is a system, not a script. Invest in the orchestration + observability layer as much as the transformation logic.
A Data Pipeline is an automated workflow that moves and transforms data from source systems to destination systems. It is the foundational infrastructure of data engineering — without pipelines, data stays trapped in operational systems and never reaches analysts, dashboards, or ML models.
┌──────────┐ ┌───────────┐ ┌──────────┐ ┌──────────────┐
│ Sources │──→ │ Extract │──→ │Transform │──→ │ Load │
│ │ │ │ │ │ │ │
│ Databases│ │ APIs │ │ Clean │ │ Data │
│ APIs │ │ CDC │ │ Validate │ │ Warehouse │
│ Files │ │ Streaming │ │ Enrich │ │ Data Lake │
│ SaaS │ │ │ │ Aggregate│ │ Feature Store│
└──────────┘ └───────────┘ └──────────┘ └──────────────┘Source → Transform (staging) → Load (warehouse)Source → Load (warehouse raw) → Transform (warehouse clean)Warehouse → CRM, Marketing tools, Product databases| Component | Purpose | Tools |
|---|---|---|
| Ingestion | Extract data from sources | Fivetran, Airbyte, Kafka Connect |
| Orchestration | Schedule and monitor pipelines | Airflow, Dagster, Prefect |
| Transformation | Clean and model data | dbt, Spark, Dataform |
| Storage | Store processed data | Snowflake, BigQuery, S3 + Iceberg |
| Quality | Validate data integrity | Great Expectations, dbt tests, Soda |
| Observability | Monitor pipeline health | Monte Carlo, Datadog, Atlan |
Pipelines fail in predictable, expensive ways:
A data pipeline is an automated workflow that moves data from where it originates (databases, APIs, files, event streams) to where it's consumed (warehouses, dashboards, ML models, downstream apps) — usually with transformation steps in between. Think of it as an assembly line: raw materials come in at one end, get cleaned/enriched/aggregated, and finished product comes out the other. Pipelines range from a nightly Python script pulling from Salesforce into Postgres, to a streaming architecture with Kafka + Flink + Iceberg handling millions of events per second with sub-second latency.
ETL transforms data before loading into the destination — the transform happens in transit, usually in a separate compute layer (Informatica, Talend, SSIS, or a staging area). ELT loads raw data first, then transforms inside the destination warehouse using its own compute power (Snowflake, BigQuery, Databricks + dbt). ELT became dominant because modern cloud warehouses have cheap elastic compute, and keeping raw data makes backfills and new use cases easier. ETL still matters when you need to mask PII before landing, or when the destination can't handle transformation load (older Redshift clusters, small Postgres warehouses). Modern default is ELT; older enterprises often still run ETL.
A modern stack typically combines: (1) ingestion — Fivetran, Airbyte, Kafka Connect for getting data in; (2) orchestration — Airflow, Dagster, or Prefect for scheduling and dependencies; (3) transformation — dbt for SQL-based, Spark/Snowpark for Python-based, Flink for streaming; (4) storage — Snowflake, BigQuery, Databricks, or S3 + Iceberg; (5) quality — dbt tests, Great Expectations, Soda; (6) observability — Monte Carlo, Atlan, or orchestrator-native lineage. Older stacks use Informatica, Talend, or hand-rolled Python. The specific combination depends on scale, budget, and existing investments.
Batch processes data in chunks on a schedule — hourly, daily, or on-demand. Streaming processes data continuously as events arrive. Batch is simpler and cheaper; streaming provides lower latency but is much harder operationally (exactly-once semantics, windowing, watermarks, backpressure, state management). Most business workloads can tolerate 15+ minute freshness and should use batch. Genuine streaming use cases: fraud detection, real-time personalization, operational monitoring, IoT. Mixed workloads sometimes use a Lambda architecture (batch + speed layer) but pure streaming or pure batch is operationally simpler.
Six properties: (1) idempotent — re-runnable without side effects; (2) incremental — processes only new/changed data where possible; (3) observable — you can see what ran, what failed, and what produced each output; (4) testable — data quality tests run as part of the graph, not separately; (5) lineage-visible — you can trace any output back to its sources through every transformation; (6) owned — every asset has a named human responsible for it with an alert routing rule. Most outages come from missing one of these. A pipeline that "works in dev" but misses these is a time bomb.
Four layers: (1) job-level — orchestrator shows which runs succeeded/failed (Airflow, Dagster, Prefect UIs give this for free); (2) data-quality — dbt tests, Great Expectations, Soda catch rows-per-day anomalies, null rates, freshness; (3) asset/table-level — data observability tools (Monte Carlo, Atlan, Bigeye) detect statistical anomalies across tables automatically; (4) business-metric — does the final dashboard number match expectations? Alerting must be routed to a named owner with an SLA, not a shared channel where alerts die. Metrics that matter: freshness SLA compliance, row-count anomalies, failed-run rate, and MTTR for incidents.
Schema changes upstream that nobody warned the pipeline about. A source adds a column, renames a field, changes a type, or deletes a table — and downstream models either break loudly (good) or produce wrong results silently (bad). Defenses: data contracts between producers and consumers (dbt contracts, schema registries), source-freshness tests, explicit column lists (not SELECT *), and producer-side CI that fails before the change reaches production. After schema drift, the next most common causes are upstream API outages, replication slot issues (for CDC pipelines), and resource exhaustion (memory, slot, or warehouse capacity).