The practice of managing and provisioning infrastructure through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools.
Every data team I've joined has had the same story: someone configured the production Snowflake warehouses through the UI in 2022, left the company in 2023, and now nobody remembers which roles have which grants, why XLARGE_WH has a 10-minute auto-suspend, or what happens if we need to recreate this in a new region. That story is what Infrastructure as Code exists to prevent. In 2026, running a serious data platform without IaC is an anti-pattern — the question is just which tool, how much to codify, and how aggressively to enforce it.
My default list: (1) warehouses, databases, schemas, roles, grants, network policies in Snowflake; (2) S3/GCS/ADLS buckets + lifecycle + encryption; (3) IAM roles and trust relationships (especially storage integrations); (4) Airflow/Prefect/Dagster infrastructure (clusters, node pools); (5) Secrets (managed via cloud secret manager, referenced from IaC); (6) Connector configuration where the tool supports it (Fivetran, Airbyte). What I usually do not codify: data (tables, DMFs beyond structure), which is dbt's job; and application-level configuration that changes more than weekly, which lives in its own system.
On one migration we needed to stand up a dev clone of a ~70-object Snowflake setup. Without IaC this would have been a week of manual clicking. Instead we ran Terraform against a new account with a different variable set. Time to fully provisioned clone: 40 minutes, 90% of which was waiting on warehouse creation. Three months later, the same codebase survived a region migration and a security-posture review that needed "evidence of change control." Both would have been painful without IaC. The upfront cost — learning Terraform or Pulumi, writing the initial modules — is real (a few weeks of someone's time), but it's paid back the first time you need to rebuild or answer "why does this resource exist?"
Infrastructure as Code (IaC) is the practice of defining, provisioning, and managing cloud infrastructure through code files rather than manual point-and-click configuration. For data engineers, IaC means your Snowflake warehouses, S3 buckets, Airflow clusters, and IAM roles are defined in version-controlled files that can be reviewed, tested, and deployed like application code.
Without IaC, data infrastructure is configured manually:
# Terraform: Snowflake warehouse defined as code
resource "snowflake_warehouse" "etl_wh" {
name = "ETL_WH"
warehouse_size = "MEDIUM"
auto_suspend = 60
auto_resume = true
comment = "Warehouse for ETL pipelines"
}| Approach | How It Works | Example |
|---|---|---|
| Declarative | You describe the desired end state; the tool figures out how to get there | Terraform, CloudFormation, Pulumi (most modes) |
| Imperative | You write step-by-step instructions for what to create | AWS CLI scripts, Python boto3, Ansible |
| Tool | Language | Provider Support | Best For |
|---|---|---|---|
| Terraform | HCL | All clouds + Snowflake, Databricks | Multi-cloud, data platform infra |
| Pulumi | Python, TS, Go | All clouds | Teams who prefer real programming languages |
| AWS CDK | Python, TS | AWS only | AWS-only shops |
| CloudFormation | YAML/JSON | AWS only | Native AWS, no extra tooling |
| Bicep | Bicep | Azure only | Azure-native teams |
1. WRITE: Define infrastructure in code (Terraform .tf files)
2. PLAN: Preview changes (terraform plan shows what will change)
3. REVIEW: Code review via Git pull request
4. APPLY: Deploy changes (terraform apply)
5. STATE: Tool tracks what exists vs. what's definedIaC pitfalls that bite real teams:
terraform.tfstate in a shared repo.terraform plan regularly as a drift detector; decide in advance whether you overwrite the drift or import it. Tools like driftctl can automate this..tf files is a bad pattern. Use sensitive variables sourced from a secret manager (AWS Secrets Manager, HashiCorp Vault, cloud KMS) with short-lived credentials.terraform apply touching production can destroy resources on a misconfiguration. Use separate state files per environment, require manual approval on production applies (via Terraform Cloud or Atlantis), and keep destructive actions gated.IaC means describing your cloud infrastructure — servers, databases, permissions, networks, storage — in code files instead of clicking through web consoles. Tools like Terraform read those files and automatically create or update the real resources to match. You check the code into Git like any other application code, review changes through pull requests, and deploy them through CI/CD. The core benefit: infrastructure becomes reproducible, auditable, and version-controlled instead of a pile of decisions nobody can reconstruct.
Terraform is used to provision and manage data-platform infrastructure: Snowflake warehouses, databases, schemas, roles, grants, network policies; S3/GCS/ADLS buckets and lifecycle rules; IAM roles and storage integrations; Airflow/MWAA/Composer environments; Databricks workspaces, clusters, and Unity Catalog; Kafka/MSK topics and ACLs; Fivetran/Airbyte connectors. The Snowflake Terraform provider (Snowflake-Labs/snowflake) is mature and covers most objects. For teams running on AWS + Snowflake, Terraform is effectively the default choice in 2026.
Terraform uses HCL (its own configuration language) and has the largest provider ecosystem and community — if you need to manage almost any cloud resource, someone has written a Terraform provider for it. Pulumi lets you use real programming languages (Python, TypeScript, Go, C#) so you get proper testing, looping, and abstraction without HCL's limits; best for teams with strong software-engineering culture. Terraform is the safer default — broader ecosystem, more hiring, more tutorials. Pulumi is the right pick when your team strongly prefers typed programming languages over HCL or needs complex runtime logic in infrastructure code.
Declarative IaC (Terraform, CloudFormation, Pulumi in most modes) means you describe the desired end state — "I want a warehouse named ETL_WH with these settings" — and the tool computes what changes are needed to reach it. Imperative IaC (AWS CLI scripts, boto3, raw Ansible tasks) means you write step-by-step instructions — "create warehouse, then set auto-suspend, then grant role." Declarative is preferred because it handles updates, deletions, and dependency ordering automatically, and because it reconciles reality with code via a diff. Almost all modern IaC tools are declarative.
Yes, even for small teams — but start narrowly. Don't try to codify everything on day one. The highest-value targets: your Snowflake (or warehouse) role hierarchy and grants, your S3 buckets and lifecycle, your IAM roles for storage integrations, your dbt CI/CD pipeline. These are the resources where manual drift causes the most pain. Once these exist in Terraform, add Airflow/Dagster infra and secondary buckets. The full-stack IaC ambition can happen over a year — but even 30% IaC coverage is dramatically better than zero.
They have complementary scopes. Terraform manages platform-level resources: warehouses, databases, schemas, roles, grants, network policies, external integrations. dbt manages in-warehouse logical objects: tables, views, tests, documentation, model lineage. The boundary is usually clear: anything a dbt project creates (tables from models, test results) stays in dbt; anything that sits underneath dbt (the database it writes into, the role it runs as, the warehouse it uses) belongs in Terraform. In practice, Terraform provisions the "skeleton" (roles, schemas, grants, warehouses) and dbt fills in the "flesh" (data models).
Terraform state is a JSON file tracking which real-world resources correspond to which Terraform code blocks. Without state, Terraform couldn't tell the difference between "create this resource" and "this resource already exists." State must live in a remote backend with locking (S3 + DynamoDB lock, Terraform Cloud, GCS with versioning) so multiple engineers don't apply concurrently. Losing state is painful — you have to import every resource back. Protecting state is as important as protecting production secrets; it often *contains* sensitive data (resource IDs, sometimes credentials), so encrypt it at rest and restrict access.