Data Orchestration Cheat Sheets

Orchestration is where pipelines fail in production: a missed SLA, a retry that double-loads a fact table, a backfill that locks a warehouse overnight. These cheat sheets are written for people who already know what a DAG is and need the patterns that keep jobs idempotent, observable, and cheap.

Airflow remains the default scheduler in most enterprises because of the operator ecosystem and the way teams already staff on-call around it. dbt is not a scheduler; it is the transformation layer that Airflow (or another orchestrator) should invoke. Treating dbt Cloud’s built-in jobs as the only scheduler works for small SQL estates and falls over when you also have Python, Spark, or cross-system waits.

What to read first

• Airflow Essentials — DAG structure, operators, XCom, pools, retries, SLAs, and backfill without overlapping runs.
• Airflow Best Practices — idempotent tasks, secrets, callback patterns, and why depends_on_past is usually the wrong catch-up strategy.
• dbt Commands — build vs run vs test, selectors, defer, and state-based CI.
• dbt Best Practices — layered models, tests as contracts, incremental grain, and documentation that reviewers can use.

Production decisions these sheets are meant to settle

• Catch-up vs backfill: catch-up with a data interval that cannot overlap; backfill as a separate DAG run with a bounded date range and a warehouse size you chose on purpose.
• Retries: retry only tasks that are safe to re-run. A COPY that is not idempotent needs a staging pattern, not five retries.
• Sensors vs datasets: long poke sensors burn worker slots. Prefer file/dataset events or a short poke with a timeout that pages a human.
• dbt in Airflow: one task per project/job, not one task per model, unless you have a specific reason to expose model-level retries.

If you are choosing a stack: Airflow + dbt covers most analytics engineering shops. Databricks jobs or Snowflake Tasks can replace Airflow for a single-platform estate, but you still need an external scheduler the moment you cross clouds or wait on a SaaS export. Pair these sheets with the cron builder and the dbt Cloud cost calculator.

Related: best-practices cheat sheets, SQL, and the full library.

← Back to Home