What is Apache Airflow?

Apache Airflow is the elder statesman of modern workflow orchestration. Created at Airbnb in 2014, donated to Apache, and now running more production data pipelines than any other tool in the category. When someone says "we orchestrate our data pipelines," Airflow is the default assumption — to the point that teams often pick it reflexively without evaluating newer alternatives like Dagster, Prefect, or cloud-native offerings (AWS Step Functions, GCP Workflows, Temporal).

What Airflow actually does (and doesn't do)

Airflow is an orchestrator, not an ETL tool. It schedules tasks, manages dependencies between them, retries on failure, logs execution history, and provides a web UI for monitoring. It does not move data, transform data, or connect to data sources by itself — that's the job of the tasks it runs (typically Python operators, Bash operators, or provider-specific operators for dbt, Snowflake, Spark, etc.). This distinction matters when teams get confused about why Airflow is slow (it's usually not Airflow — it's the tasks Airflow is running).

Field story

A client's Airflow deployment had grown to 800 DAGs on a single self-hosted instance, parsing every DAG file every 30 seconds. The scheduler was spending 60% of its time just parsing Python, and DAG start latency had climbed to 12 minutes. Three fixes in sequence: (1) enabled dag_discovery_safe_mode and moved DAG imports to lazy-load patterns, cutting parse time 70%; (2) moved the heavy-transformation DAGs to KubernetesExecutor so scheduler load dropped; (3) finally sharded into two Airflow deployments by team. After the work, DAG start latency was under 30 seconds and the scheduler CPU sat at 15%. Lesson: Airflow scales, but you have to actively manage the scheduler — it won't warn you when you're approaching its limits, you'll just notice everything gradually getting slower.

Apache Airflow is an open-source workflow orchestration platform created at Airbnb and now maintained by the Apache Software Foundation. It allows you to define, schedule, and monitor complex data workflows as code.

Core Concepts

DAGs (Directed Acyclic Graphs)

Workflows in Airflow are defined as DAGs—collections of tasks with dependencies. The "acyclic" property ensures no circular dependencies.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG('my_etl_dag', start_date=datetime(2024, 1, 1), schedule='@daily') as dag:
    extract = PythonOperator(task_id='extract', python_callable=extract_data)
    transform = PythonOperator(task_id='transform', python_callable=transform_data)
    load = PythonOperator(task_id='load', python_callable=load_data)
    
    extract >> transform >> load

Operators

Pre-built task types for common operations:

• PythonOperator: Run Python functions
• BashOperator: Execute shell commands
• SQLOperator: Run SQL queries
• S3/GCS Operators: Interact with cloud storage
• Provider Operators: Snowflake, BigQuery, dbt, etc.

Executors

How Airflow runs tasks:

• SequentialExecutor: One task at a time (development)
• LocalExecutor: Parallel on single machine
• CeleryExecutor: Distributed across workers
• KubernetesExecutor: Pods per task for isolation

Key Features

• Workflows as Code: Version control your pipelines
• Rich UI: Visual DAG graph, logs, retry handling
• Extensible: Build custom operators and sensors
• Integrations: 1000+ provider packages
• Backfills: Re-run historical data processing

Airflow vs Alternatives

Common Use Cases

1. ETL/ELT Pipelines: Orchestrate data movement and transformation
2. ML Workflows: Training, validation, deployment pipelines
3. Reporting: Schedule automated report generation
4. Data Quality: Trigger validation checks on new data

Airflow pitfalls I've watched teams rediscover:

• DAG parse time kills scheduler performance. Every DAG file is parsed on a regular interval (default 30s). Heavy imports at module level, database calls during DAG definition, or dynamic DAG generation from large configs will choke the scheduler. Keep top-level DAG files lightweight; defer expensive work to task execution time.
• start_date and schedule_interval semantics confuse everyone. A DAG with start_date=2024-01-01 and @daily schedule runs at the end of each interval — the first run fires at 2024-01-02 00:00 for the 2024-01-01 interval. Teams lose hours debugging "why isn't my DAG running yet?" Read the data-interval docs carefully before scheduling anything.
• Backfills can easily overwhelm downstream systems. Triggering a backfill for a year of daily DAGs fires 365 runs. Use max_active_runs, pool limits, and depends_on_past=True to throttle. A 3 AM backfill that consumed all the Snowflake credits for a small customer is a real war story.
• XCom is not a general-purpose data-passing mechanism. XCom values go through the Airflow metadata database (Postgres/MySQL). Pushing a 50MB DataFrame through XCom will crash the scheduler. XCom is for small task outputs (IDs, counts, paths) — pass actual data through S3, GCS, or a database table.
• SQLAlchemy connection pool exhaustion. Scheduler, webserver, and workers all hold connections to the metadata DB. On busy deployments, you must tune sql_alchemy_pool_size, sql_alchemy_max_overflow, and Postgres max_connections together. Surprise connection-exhaustion outages are common.
• Retries without idempotency are worse than no retries. Default Airflow retries task failures up to 3 times. If your task is not idempotent (e.g., INSERT without dedup), you'll get duplicate data on every retry. Either make tasks idempotent (MERGE, upsert) or set retries=0.
• catchup=True is the default and causes chaos. When you deploy a DAG with an old start_date, Airflow fires runs for every missed interval. Always set catchup=False unless you explicitly want historical backfill on deployment.
• KubernetesExecutor pod startup latency is real. Per-task pods add 30-60s overhead vs. CeleryExecutor. For DAGs with many short tasks, Celery is faster; for long-running heterogeneous workloads, K8s is more efficient. Pick per workload, not globally.
• Airflow is not an ideal choice for sub-minute scheduling. The smallest practical schedule interval is about a minute; even that stresses the scheduler. For high-frequency event-driven workflows, consider Dagster, Temporal, or streaming alternatives.