The documentation and visualization of data as it flows from source to destination, showing transformations, dependencies, and ownership at each step.
Data lineage is the boring-sounding feature that saves your weekend. The first time you have to answer "why did the revenue number change on yesterday's dashboard?" without lineage, you'll spend six hours reading SQL. With lineage, you click the dashboard tile, click upstream, and see the one model that changed and the one source column that got nulled out. That's the real ROI — not pretty graphs in a demo.
Technical lineage is column-to-column: fct_orders.order_total → stg_orders.amount + stg_orders.tax. This is what dbt exposes automatically for SQL models, what OpenLineage captures from Spark/Airflow, and what a good catalog (Atlan, Alation, DataHub) stitches together across tools. Business lineage is concept-to-concept: "monthly recurring revenue" depends on "active subscriptions" depends on "billing events." This lives in your semantic layer or metrics store. Operational lineage is runtime — which DAG runs populated this table, when did each run finish, which job failed. You want all three eventually, but technical lineage is what unblocks the 2 AM debugging call.
A Snowflake-based fintech I worked with had a "total_balance" column on a customer-facing API drift by $0.03 in 2% of accounts. Without lineage, debugging meant opening 14 dbt models by hand. With column-level lineage in Atlan, we traced total_balance → ledger_view.running_total → a SUM() OVER (PARTITION BY ...) window where a recent change added ORDER BY txn_ts — and a cluster of rows with identical timestamps reordered non-deterministically. Root cause in 20 minutes instead of a day. The column-level tracing is what made it fast; table-level lineage would have still left us reading SQL. That's the test I apply to lineage tools now: column-level or don't bother.
Data lineage is the complete lifecycle of data from origin to consumption. It documents where data comes from, how it moves through systems, what transformations are applied, and what depends on it. Think of it as a visual map of your data's journey.
| Method | Pros | Cons |
|---|---|---|
| SQL Parsing | Accurate, automatic | Complex to implement |
| API Integration | Real-time | Vendor-specific |
| Manual Documentation | Flexible | Outdated quickly |
| Log Analysis | Runtime truth | Incomplete picture |
Lineage problems cluster around coverage, accuracy, and latency:
EXECUTE IMMEDIATE, Jinja macros that produce different SQL each run, and stored procs with conditional logic. Instrument with OpenLineage at the runtime layer if you rely on dynamic SQL.dim_customer fed by a table that was dropped three months ago actively misleads. Automate lineage refresh (most modern catalogs do this hourly or on-change) and surface freshness on the lineage UI.SUM(subscription_mrr * 12), but business owners think of ARR as a specific definition. Keep a metrics layer (dbt Semantic Layer, Cube, LookML) that ties technical column names to business definitions, or the two lineages diverge.Data lineage is a map that shows where each piece of data in your warehouse came from, every transformation it went through, and every downstream system that depends on it. It's what lets you answer the two questions analysts and engineers ask constantly: "where does this number come from?" and "if I change this table, what breaks?" Think of it as git blame for datasets — you click a column and see its full history, across tools and platforms.
Four concrete payoffs: (1) impact analysis before changes — know which dashboards depend on a column before you rename it; (2) root-cause analysis during incidents — click the bad number and trace upstream in minutes instead of hours; (3) compliance and audits — demonstrate how PII moves through your systems without hand-drawn diagrams; (4) migration planning — move off a legacy pipeline with a ranked list of actual dependencies. Teams without lineage eventually rebuild it on whiteboards during every incident — which is the most expensive way to get it.
Three complementary approaches, usually layered: (1) native tooling — dbt gives column-level lineage for SQL models for free; use it; (2) OpenLineage — the open standard for emitting lineage events from Spark, Airflow, Dagster, Flink, and more; route events to a backend like Marquez, Datakin, or a catalog; (3) catalog platforms — Atlan, Alation, DataHub, Collibra ingest lineage from warehouses, BI tools, and orchestrators via connectors. Start with dbt for coverage inside your modeling layer, add OpenLineage for orchestration, and adopt a catalog when you need cross-tool discovery and business lineage.
Lineage is the full flow — source → transformation → destination at scale across systems, usually automated. Provenance is narrower: for a specific datum (a row, a value), who created it, when, from what raw inputs, under what process. Provenance is what you need for regulated use cases — clinical data, financial reconciliation, AI training datasets where "prove this training row's origin" matters. Lineage tools answer architectural questions; provenance stores (immutable event logs, blockchain-like ledgers, or just well-tagged append-only tables) answer per-record questions.
Depends on scope. Inside a dbt/SQL stack: dbt's built-in lineage plus dbt Cloud Explorer is hard to beat for free. Cross-tool automation: Atlan, Alation, Collibra — pick on UX and connector depth to your stack. Open-source: DataHub (LinkedIn) and Marquez (via OpenLineage) are production-grade. Spark/streaming: OpenLineage emitters + a backend. For Snowflake-first orgs, Snowflake's own ACCESS_HISTORY and OBJECT_DEPENDENCIES views give query-level lineage inside the warehouse without any vendor — combine with a catalog for the full picture.
Column-level matters for real debugging. Table-level tells you `fct_orders` depends on `stg_orders` and `stg_customers` — fine for high-level impact slides. When a single column value is wrong, you want to see exactly which upstream column feeds it and which transformation applied. Column-level is what turns lineage from a documentation artifact into an operational debugging tool. dbt, OpenLineage, Atlan, and Alation all support column-level for SQL — insist on it for any production governance rollout.
ML lineage extends regular lineage with feature-store views, model-training runs, and prediction tables. The useful graph traces: raw event → feature table → feature view → training dataset → model version → online prediction. Tools: MLflow tracks model lineage (code, data, params) per run; Snowflake ML Lineage (GET_LINEAGE) traces inside Snowflake; OpenLineage has ML-specific facets. The value shows up when a production model drifts — you need to know which feature changed, which upstream source caused it, and which training runs would be affected by a backfill.