Orchestrate Snowflake dbt Projects with Airflow

How I Wired Snowflake’s Native dbt Projects to Airflow — And Finally Got True End-to-End Orchestration

I’ll be honest with you — for a long time I was running dbt the way most people run it. dbt Core installed on a server, profiles.yml file that I kept updating manually, a cron job (yes, a cron job) doing the scheduling, and Airflow somewhere nearby doing the “real” orchestration while dbt lived in its own separate corner of the infrastructure.

It worked. It was fine. It was also quietly annoying in ways that I’d gotten so used to I stopped noticing them. Managing the dbt server separately. Keeping the Snowflake credentials synced in two places. Debugging failures by jumping between the Airflow UI, SSH logs on the dbt server, and Snowsight — all at once.

Then Snowflake went GA with dbt Projects in November 2025, and I spent a weekend rebuilding the whole thing. This article is what I learned.

What we’re building here is a genuine end-to-end pipeline: raw data lands in Snowflake, Airflow orchestrates the entire flow, and the dbt transformations run as a native DBT PROJECT object inside Snowflake — not on an external box, not in a container, inside Snowflake itself. The monitoring, the scheduling trigger, the execution logs — all in one place.

Let’s build it from the ground up.

First — What Exactly Is a dbt Project on Snowflake?

This is important because the terminology can trip you up, and I don’t want you 45 minutes into setup before the confusion hits.

dbt Projects on Snowflake let you use familiar Snowflake features to create, edit, test, run, and manage dbt Core projects. You can use Workspaces in Snowsight to work with dbt project files and directories and deploy a dbt project as a schema-level DBT PROJECT object.

The key word there is object. Snowflake introduces a first-class schema-level object called DBT PROJECT. The DBT PROJECT object in Snowflake is essentially a file container that can contain one or more dbt Core projects. Furthermore, the DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version.

This means your dbt project — the models, the sources YAML, the dbt_project.yml — lives inside Snowflake as a versioned, native object. Not on a VM. Not in an S3 bucket somewhere. In Snowflake itself.

dbt Projects on Snowflake streamline workflows for data engineers to standardize and automate transformation pipelines by allowing for: development and testing in Workspaces using a file-based IDE that integrates with Git; visualization and debugging of DAGs to inspect lineage and dependencies directly in the UI; deployment and scheduling using native Snowflake Tasks; and selection of dbt commands such as COMPILE, TEST, RUN and more, right from the native Workspaces IDE.

So yes — you can schedule and run it purely with Snowflake Tasks and never touch Airflow. But if your organization already runs Airflow, or if your dbt pipeline is one piece of a larger orchestration that includes data ingestion, validation, downstream alerts, and reporting — you want Airflow in charge, calling into Snowflake to execute the DBT PROJECT object. That hybrid approach is exactly what this article covers.

The Architecture We’re Building

Before I show you a single line of code, let me draw the full picture because I think this is where most blog posts let you down — they show you a piece without the whole.

[Source System / S3 / API]
        ↓
[Airflow DAG starts]
        ↓
  Task 1: Load raw data → Snowflake staging table (via COPY INTO or S3 stage)
        ↓
  Task 2: Run data quality checks on raw data (SQLExecuteQueryOperator)
        ↓
…

Airflow owns the orchestration. Snowflake owns the execution of the dbt transformations. The DBT PROJECT object is what bridges them — because you can trigger it with a SQL command, and Airflow’s SQLExecuteQueryOperator can fire that SQL command.

That SQL command, by the way, is beautifully simple:

1EXECUTE DBT PROJECT my_database.my_schema.my_dbt_project2  ARGS = 'dbt build'3  VERSION = 'LAST';

EXECUTE DBT PROJECT executes the specified dbt project object or the dbt project in a Snowflake workspace using the dbt command and command-line options specified. Snowflake Documentation

One SQL statement. That’s all Airflow needs to fire. Let me now show you the full setup to make that work.

Step 1: Snowflake Setup — Roles, Warehouse, and Permissions

I always start here because bad permissions cause the most confusing failures, and they surface late in the process when you’re tired and frustrated.

USE ROLE ACCOUNTADMIN;

-- Create a dedicated role for dbt execution
CREATE OR REPLACE ROLE dbt_executor_role;
GRANT ROLE dbt_executor_role TO ROLE SYSADMIN;

-- Create the service user Airflow will use
CREATE OR REPLACE USER airflow_svc_user
…

I made a mistake my first time through — I granted object-level access but forgot the schema-level EXECUTE DBT PROJECT privilege, which is separate. The error message wasn’t obvious. Save yourself that 20-minute debugging session.

Step 2: Deploy Your dbt Project as a Native Snowflake Object

This is the step that feels the most different from traditional dbt Core setup. You’re not installing dbt on a server. You’re registering your project inside Snowflake.

Option A: Via Snowsight Workspaces (recommended for first time)

Log into Snowsight, navigate to Workspaces, and connect it to your Git repository:

1-- First, create an API integration for GitHub2CREATE OR REPLACE API INTEGRATION github_integration3  API_PROVIDER = git_https_api4  API_ALLOWED_PREFIXES = ('https://github.com/yourorg/')5  ENABLED = TRUE;6 7-- Create the Git repository object in Snowflake8CREATE OR REPLACE GIT REPOSITORY dbt_project_repo9  API_INTEGRATION = github_integration10  GIT_CREDENTIALS = my_github_secret11  ORIGIN = 'https://github.com/yourorg/your-dbt-project.git';

Option B: Deploy via SQL (great for CI/CD)

1-- Create the DBT PROJECT object from your connected Git repo2CREATE OR REPLACE DBT PROJECT analytics_db.transforms.sales_dbt_project3  FROM GIT REPOSITORY dbt_project_repo4  REF = 'main'5  TARGET_PATH = 'models/'6  WAREHOUSE = dbt_transform_wh;

Install dbt dependencies:

Install dependencies by executing the dbt deps command within a Snowflake workspace, local machine, or git orchestrator to populate the dbt_packages folder for your dbt Project.

1-- Run this once after creating the project, or include in CI/CD2EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project3  ARGS = 'dbt deps'4  VERSION = 'LAST';

A heads up on this: running dbt deps to install packages requires an external access integration when executed inside Snowflake Workspaces, since the runtime needs to reach external package repositories. Alternatively, you can run dbt deps locally or in your CI pipeline and include the populated dbt_packages folder in your deployment artifact.

I found it cleaner to run dbt deps in my GitHub Actions pipeline and commit the dbt_packages folder, rather than configuring external access integrations for every environment. Your call — both approaches work.

Verify it deployed correctly:

1-- Check your dbt project versions2SHOW DBT PROJECTS IN SCHEMA analytics_db.transforms;3 4-- Test execute manually before wiring Airflow5EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project6  ARGS = 'dbt compile'7  VERSION = 'LAST';

If dbt compile completes without error, your project is live and ready to be called by Airflow.

Step 3: Set Up a Real dbt Project Structure

Let me show you what the actual project looks like. I’m using a sales pipeline as the example — raw orders come in, we stage them, build a fact table, and create a daily summary mart.

dbt_project.yml:

name: 'sales_pipeline'
version: '1.0.0'
config-version: 2

profile: 'snowflake_prod'

model-paths: ["models"]
test-paths: ["tests"]
…

models/staging/stg_orders.sql:

-- Staging model: clean and type-cast raw orders
WITH raw AS (
    SELECT * FROM {{ source('raw', 'orders_raw') }}
),

cleaned AS (
    SELECT
        order_id::VARCHAR           AS order_id,
…

models/marts/fct_daily_orders.sql:

-- Fact table: daily order summary by region
WITH staged AS (
    SELECT * FROM {{ ref('stg_orders') }}
)

SELECT
    order_date,
    region,
…

models/staging/sources.yml:

version: 2

sources:

• name: raw database: analytics_db schema: raw_landing tables:
  • name: orders_raw description: “Raw orders from the source system” columns:
    • name: order_id tests:
      • not_null
      • unique
    • name: customer_id tests:
      • not_null
    • name: order_date tests:
      • not_null
    • name: amount tests:
      • not_null

models/marts/schema.yml:

version: 2

models:
  - name: fct_daily_orders
    description: "Daily order summary by region and status"
    columns:
      - name: order_date
        tests:
…

This gives us a clean, testable project with source freshness checks and column-level tests. When Airflow executes dbt build, all of this runs — models + tests — in dependency order.

Step 4: Wire It All Together in Airflow

Now the fun part. I’m going to show you a complete Airflow DAG that:

1. Validates raw data arrived in Snowflake
2. Fires the native dbt project execution
3. Validates row counts on the output marts
4. Sends a Slack notification on success or failure

First, install the Snowflake provider if you haven’t:

1pip install apache-airflow-providers-snowflake

Set up your Snowflake connection in the Airflow UI (Admin → Connections):

1Connection ID : snowflake_analytics2Connection Type : Snowflake3Account  : yourorg.us-east-14Login    : airflow_svc_user5Password : YourStrongPassword123!6Schema   : transforms7Database : analytics_db8Warehouse: dbt_transform_wh9Role     : dbt_executor_role

Now the DAG:

dags/sales_pipeline_dag.py:

from airflow import DAG
from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
from airflow.operators.python import PythonOperator, BranchPythonOperator
from airflow.operators.empty import EmptyOperator
from airflow.utils.dates import days_ago
from datetime import datetime, timedelta
import logging

…

Step 5: Running Specific dbt Selectors from Airflow

One of the things I really like about this approach is that you get the full power of dbt’s selector syntax passed straight through the ARGS parameter. You don’t have to run the entire project every time.

Run only staging models:

1EXECUTE_STAGING_ONLY = """2EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project3  ARGS = 'dbt run --select staging.*'4  VERSION = 'LAST';5"""

Run a specific model and all its downstream dependencies:

1EXECUTE_ORDERS_DOWNSTREAM = """2EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project3  ARGS = 'dbt build --select stg_orders+'4  VERSION = 'LAST';5"""

Run tests only, separate from the model run:

1RUN_DBT_TESTS = """2EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project3  ARGS = 'dbt test --select staging.*'4  VERSION = 'LAST';5"""

This means you can split a single DAG into multiple tasks — one for staging, one for marts, one for tests — and get granular retry behavior in Airflow if something fails mid-pipeline. Instead of rerunning everything, Airflow retries only the failed task.

Here’s that pattern as a DAG:

run_staging = SQLExecuteQueryOperator(
    task_id='run_dbt_staging',
    conn_id=SNOWFLAKE_CONN,
    sql="""
        EXECUTE DBT PROJECT analytics_db.transforms.sales_dbt_project
          ARGS = 'dbt run --select staging.*'
          VERSION = 'LAST';
    """,
…

This is how I actually run it in practice. If staging tests fail, marts never execute. If marts fail, I retry marts without re-running staging. Clean dependency management with minimal code.

Step 6: Handling New Versions of Your dbt Project

This is something I didn’t think about until I pushed a breaking change to main and my 6 AM pipeline executed the wrong version.

The DBT PROJECT object is versioned so that each change made to the object via ALTER will add a new version. The versions are named according to the pattern VERSION$<num>.

In practice, your CI/CD pipeline (GitHub Actions, etc.) should update the DBT PROJECT object after any merge to main:

# .github/workflows/deploy_dbt.yml
name: Deploy dbt Project to Snowflake

on:
  push:
    branches: [main]

jobs:
…

And in your Airflow SQL, VERSION = 'LAST' always picks up the most recently deployed version automatically. So once CI/CD deploys a new version, the next DAG run picks it up with no Airflow changes needed.

Step 7: Monitoring — What to Watch and Where

Before this setup, I was watching three screens at once when something went wrong. Now it’s mostly one.

In Snowsight:

-- Check recent dbt project execution history
SELECT
    query_id,
    query_text,
    execution_status,
    start_time,
    end_time,
    DATEDIFF('second', start_time, end_time) AS duration_seconds,
…

Row count drift detection (add this as an Airflow task):

-- Compare today's mart row count to yesterday's
-- Flag if it drops more than 20%
WITH today AS (
    SELECT COUNT(*) AS cnt
    FROM analytics_db.marts.fct_daily_orders
    WHERE order_date = CURRENT_DATE() - 1
),
yesterday AS (
…

I added this query as a SQLExecuteQueryOperator task right after the mart validation step. If the row count drops by more than 20% compared to the previous day, the task raises a warning in Airflow logs, and the email alert fires.

Not every data quality problem shows up as a dbt test failure. Sometimes the data just quietly shrinks because an upstream feed stopped delivering. This catches that.

What This Setup Actually Changed for Me

I want to be real about this because I think the “benefits” sections in most blog posts are too abstract.

Before: My pipeline had six moving parts. Airflow DAG on one server. dbt installed on a separate instance. profiles.yml with credentials that needed updating every time we rotated passwords. Separate monitoring in CloudWatch for the dbt server. Debugging a failure meant SSH → dbt server → find the log file → cross-reference with Airflow logs.

After: The pipeline has three moving parts — Airflow, Snowflake, and GitHub. The dbt credentials are managed by Airflow’s Snowflake connection, which I was already maintaining. Debugging a failure means clicking into the Airflow task logs (which capture the SQL response from Snowflake) and if I need more detail, running the QUERY_HISTORY query above in Snowsight.

Performance improvements were significant: during preview, result upload usually took approximately 6 to 6.5 minutes. Now, upload completes approximately 8 to 10x faster in around 40 to 45 seconds.

The startup time improvement alone was worth it for me. My morning pipeline used to take 28-32 minutes. It now consistently runs in 18-22 minutes. That’s not from faster models — it’s from the reduction in environment spin-up overhead.

A Few Gotchas I Hit Along the Way

1. The EXECUTE DBT PROJECT command is synchronous by default. Airflow will wait for it to complete before marking the task done. For large projects this is fine — you want that behavior. Just make sure your execution_timeout on the Airflow task is set generously enough.

2. Cross-project references don’t work the way you might expect. Cross-project dependencies must be copied into the root of the main project — Snowflake doesn’t support references to external file paths within the DBT PROJECT object. If you have multiple dbt projects, plan your consolidation before deploying.

3. The VERSION = 'LAST' behavior. This always runs the most recently deployed version. If you want to pin to a specific version for stability in production, use VERSION = 'VERSION$3' (or whatever version number). I run LAST in dev and a pinned version in prod, deployed via CI/CD.

4. Warehouse auto-resume and the first task. The first EXECUTE DBT PROJECT of the day can have a few seconds of latency while dbt_transform_wh auto-resumes. I added a lightweight warm-up query as the very first task in my DAG so the warehouse is already running by the time dbt build kicks off:

1warm_up_warehouse = SQLExecuteQueryOperator(2    task_id='warm_up_warehouse',3    conn_id=SNOWFLAKE_CONN,4    sql="SELECT CURRENT_TIMESTAMP();",5)6 7warm_up_warehouse >> check_raw_data >> run_dbt_project >> ...

Costs almost nothing. Saves 5-10 seconds of variability at the start of every run.

Why I Think This Is the Right Direction

I started exploring this because nobody told me to. My team’s existing setup worked. A reasonable person would have left it alone.

But the more I looked at this setup, the more I kept thinking about the overhead we carry when tools don’t talk to each other natively. Every boundary between systems is a place where credentials leak, latency is added, and debugging gets harder. The native dbt project in Snowflake closes one of those boundaries. Airflow still owns orchestration — which is where it belongs — but the transformation execution lives where the data lives.

For the growing number of organizations that have standardized on Snowflake, the native integration offers something genuinely compelling: one fewer system to run, one fewer vendor to manage, and one fewer boundary between your data and the logic that transforms it.

That sentence landed for me when I read it. That’s exactly what this is.

If you’ve been running dbt Core on a server and Airflow alongside it and you’ve been tolerating that overhead long enough that you’ve stopped noticing it — try this weekend rebuild. You might be surprised how much lighter the pipeline feels on the other side.

And if you do try it and hit something weird, drop it in the comments. I’m still learning this myself.