Why I Stopped Using Snowflake Tasks for Orchestration

Q: What I Use Instead — And Why?

I switched to Airflow with the SQLExecuteQueryOperator for Snowflake, and I've written about this setup in depth in my post How I Wired Snowflake's Native dbt Projects to Airflow . The short version of why it works better for me: Airflow owns orchestration. Snowflake owns execution. That's the right division of responsibility. Airflow is purpose-built for DAG management, dependency handling, retries, branching, alerting, and observability.

I want to be clear about something before I say anything critical: Snowflake Tasks are genuinely good. I used them for months. I recommended them to people. I w

I want to be clear about something before I say anything critical: Snowflake Tasks are genuinely good. I used them for months. I recommended them to people. I wrote internal documentation about how to set them up.

And then, slowly, quietly, I stopped reaching for them — and started reaching for Airflow instead.

This isn’t a hit piece on Snowflake Tasks. It’s an honest look at where they work beautifully, where they start to crack, and the specific moment I realised I was fighting the tool instead of using it. If you’re in that same place right now — Tasks running fine on paper, increasingly painful in practice — this is for you.

Why I Started With Tasks in the First Place

The pitch for Snowflake Tasks is genuinely compelling: schedule and orchestrate your data pipelines without leaving Snowflake. No extra infrastructure. No Airflow server to maintain. No Docker containers. No YAML config files. Just SQL.

For a solo data engineer or a small team running straightforward ELT pipelines that live entirely inside Snowflake, this is actually a great deal. You write a Task, you chain a few of them together, you set a cron schedule on the root, and the whole thing runs on serverless compute that Snowflake manages for you. Clean. Simple. Zero ops overhead.

I built my first Task tree on a customer dimension pipeline — about 6 tasks chained together to handle raw landing, staging, SCD2 merge, and a downstream mart refresh. It worked perfectly. I was genuinely impressed.

So I built more of them. And that’s where things started to get interesting.

The First Sign Something Was Off

The thing about Snowflake Tasks is that they look fine at small scale. Five tasks. Eight tasks. Even fifteen tasks chained together works reasonably well.

The cracks start showing when your pipelines grow, when requirements get more complex, and when something goes wrong at 7am and you need to figure out what happened and why.

My first real frustration was observability. When a Task fails, Snowflake logs it — but finding that log, understanding the full execution context, and connecting it to what came before and after requires digging through TASK_HISTORY in ACCOUNT_USAGE or calling INFORMATION_SCHEMA.TASK_HISTORY(). There’s no single screen that shows you, visually, what ran, what passed, what failed, and what the downstream impact was.

Compare that to opening the Airflow UI, clicking into a DAG run, and seeing every task coloured green or red with full logs one click away. The difference in time-to-diagnosis is not small. I once spent 40 minutes reconstructing a failed task tree execution from TASK_HISTORY queries that would have taken me 3 minutes in Airflow.

-- How you debug a failed Snowflake Task
SELECT
    name,
    state,
    scheduled_time,
    completed_time,
    error_code,
    error_message
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
    SCHEDULED_TIME_RANGE_START => DATEADD('hour', -6, CURRENT_TIMESTAMP()),
    RESULT_LIMIT => 100
))
WHERE name ILIKE '%customer_dim%'
ORDER BY scheduled_time DESC;

That query works. But it’s not a dashboard. It’s archaeology.

The Retry Problem

This one hurt me in production.

Snowflake Tasks have basic retry configuration — you can set SUSPEND_TASK_AFTER_NUM_FAILURES to pause a task after repeated failures, which is useful. But what you can’t do natively is retry a specific failed task in the middle of a tree and resume from that point forward.

If Task 6 in a 10-task chain fails, you fix the problem, and you want to re-run from Task 6 onwards — you’re doing it manually. You can resume the root task, but it re-runs everything from the beginning on the next scheduled tick. Or you run Task 6’s SQL manually, then manually kick off Task 7, Task 8… you see where this is going.

In Airflow, you right-click the failed task node, click “Clear”, and it re-runs that task and everything downstream. That’s it. One click. No manual intervention, no risk of accidentally re-running something upstream that already completed correctly and shouldn’t run twice.

For pipelines with expensive upstream tasks — large MERGE operations, heavy aggregations — re-running from the beginning when only a downstream step failed is both wasteful and risky. Wasteful because you’re burning compute credits on work already done. Risky because some operations are not safely idempotent and running them twice produces wrong results.

The Conditional Logic Wall

Here’s the limitation that finally pushed me to switch.

My pipelines started needing branching logic. Specifically: run the full pipeline on weekdays, run a lighter version on weekends. Or: if the row count from the previous step is zero, skip the downstream merge and send an alert instead of running an empty MERGE that silently succeeds.

In Airflow, this is a BranchPythonOperator. Three lines of Python. Clean, explicit, version-controlled.

In Snowflake Tasks, this requires workarounds. You can use a stored procedure with SYSTEM$TASK_DEPENDENTS_ENABLE logic, or try to simulate branching with conditional stored procedures that check a flag and decide whether to execute. It works — technically — but it’s brittle, hard to read, and the logic is buried inside a stored procedure rather than visible in the orchestration layer where it belongs.

Snowflake Tasks can only execute SQL statements and stored procedures. For more complex logic in Python, Java, or other languages, external schedulers are required. Flexera

When your pipeline logic is entirely SQL, Tasks are fine. The moment you need to make an orchestration decision based on runtime data — not just “did this succeed or fail” but “what did this return, and what should I do about it” — you’re working against the grain.

The Scale Limit Nobody Mentions

There is a hard limit of 1,000 Tasks per data pipeline. For very large implementations this is an issue and you need to split out the data pipeline into multiple separate data pipelines as a workaround. Sonra

Most teams won’t hit 1,000 tasks. But if you’re building a platform for multiple teams — separate pipelines per business domain, each with their own task trees — you will eventually bump into governance and management complexity that a 1,000-task-per-pipeline limit doesn’t help with.

More practically: managing dozens of separate task trees, each owned by a different role, with different schedules, different failure behaviours, and no unified view across all of them — is hard. There’s no Snowflake-native equivalent of Airflow’s DAG list view where you can see all pipelines, their last run status, and their next scheduled run in one place.

What I Use Instead — And Why

I switched to Airflow with the SQLExecuteQueryOperator for Snowflake, and I’ve written about this setup in depth in my post How I Wired Snowflake’s Native dbt Projects to Airflow. The short version of why it works better for me:

Airflow owns orchestration. Snowflake owns execution. That’s the right division of responsibility. Airflow is purpose-built for DAG management, dependency handling, retries, branching, alerting, and observability. Snowflake is purpose-built for data processing at scale. Letting each tool do what it’s best at — instead of asking Snowflake Tasks to be a general-purpose orchestrator — is the cleaner architecture.

Here’s the pattern I use for a typical pipeline:

from airflow import DAG
from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator
from airflow.operators.python import BranchPythonOperator
from datetime import datetime, timedelta

with DAG(
    dag_id='customer_dimension_pipeline',
    schedule_interval='0 6 * * *',
    start_date=datetime(2024, 1, 1),
    catchup=False,
) as dag:

    load_raw = SQLExecuteQueryOperator(
        task_id='load_raw_customers',
        conn_id='snowflake_analytics',
        sql="CALL raw.sp_load_customers();",
    )

    validate_raw = SQLExecuteQueryOperator(
        task_id='validate_raw_row_count',
        conn_id='snowflake_analytics',
        sql="""
            SELECT CASE
                WHEN COUNT(*) = 0 THEN 1/0  -- Forces task failure if no rows
                ELSE COUNT(*)
            END FROM raw.customers_staging
            WHERE load_date = CURRENT_DATE();
        """,
    )

    run_scd2_merge = SQLExecuteQueryOperator(
        task_id='run_scd2_merge',
        conn_id='snowflake_analytics',
        sql="CALL transforms.sp_customer_scd2_merge();",
    )

    refresh_mart = SQLExecuteQueryOperator(
        task_id='refresh_customer_mart',
        conn_id='snowflake_analytics',
        sql="CALL marts.sp_refresh_customer_summary();",
    )

    load_raw >> validate_raw >> run_scd2_merge >> refresh_mart

If validate_raw fails because there are zero rows, the pipeline stops. run_scd2_merge never runs. I get an Airflow alert. I can clear and rerun just validate_raw and everything downstream once the issue is fixed — without touching load_raw again.

That conditional validation step alone — stopping a pipeline when upstream data is missing — was nearly impossible to implement cleanly with Tasks. With Airflow it’s a forced division by zero in the validation SQL. Ugly, but effective. There are cleaner ways with ShortCircuitOperator too.

To Be Fair: When Snowflake Tasks Are Still the Right Choice

I don’t want this to read as “never use Tasks.” That’s not what I’m saying.

Tasks are still my first choice for:

Micro-refresh patterns. A Task + Stream combination for near-real-time SCD2 updates — triggering only when the stream has data — is elegant and genuinely hard to replicate cleanly in Airflow. I covered exactly this pattern in my post Snowflake Streams and Tasks for SCD2 — How I Actually Use Them.

Simple scheduled SQL. A single SQL statement that needs to run every 30 minutes with no dependencies? Task all the way. Zero ops overhead for maximum simplicity.

Snowflake-only pipelines with no external context. If your pipeline never needs to know anything about the outside world — no API calls, no file system checks, no cross-system dependencies — Tasks keep everything in one place.

Small teams with no existing orchestration infrastructure. If you’re a team of two data engineers and setting up Airflow feels like overkill, Tasks get you 80% of the way there with 10% of the setup cost.

The honest decision framework:

Scenario	Use
Simple scheduled SQL, no branching	Snowflake Tasks
Stream-triggered incremental loads	Snowflake Tasks + Streams
Multi-step pipeline with retry requirements	Airflow
Conditional branching based on row counts	Airflow
Pipelines touching external systems	Airflow
Cross-team pipelines needing unified observability	Airflow
Large task trees (50+ steps)	Airflow

The Honest Summary

I didn’t stop using Snowflake Tasks because they’re bad. I stopped using them as my primary orchestration layer because that’s not what they’re optimised for — and I was asking them to do a job they weren’t built to do well.

Snowflake Tasks handle most data orchestration patterns, but the real question is who actually plays well with Snowflake as more than just a place to send SQL. Monte Carlo Tasks are Snowflake’s answer to simple scheduling. Airflow is the answer to complex orchestration. Knowing which problem you actually have is the whole game.

If your pipelines are growing, your failure debugging is getting slower, and you’ve found yourself writing stored procedures to simulate branching logic — that’s the sign. That’s the moment I had. The switch to Airflow was a weekend of setup and a week of migration, and I haven’t looked back.