Data Engineering Best Practices

Best practices are contextual. A five-person analytics team and a regulated bank should not copy the same “always cluster, always Type 2, always Airflow” checklist. These sheets explain the default we use in production reviews, and the scale at which the default stops being free.

If a pipeline pages you at 3 AM, the cause is usually missing idempotency, a test that never ran in CI, a warehouse that never auto-suspends, or a model that rebuilds a full history because the incremental key was wrong. The sheets below are the rubrics we use in code review — not slogans.

Best-practice cheat sheets

• Snowflake Best Practices — warehouse auto-suspend, resource monitors, clustering only on large filtered tables, ACCOUNTADMIN off the daily path, masking before you share.
• dbt Best Practices — staging / intermediate / marts, tests on grain and not-null keys, slim CI with defer, docs as contracts.
• Airflow Best Practices — idempotent tasks, bounded retries, SLA callbacks that page a human, secrets out of DAG code, one catch-up story.

A short review checklist

1. Can this task or model run twice with the same interval and leave the same tables?
2. Is there a test that fails if the grain duplicates?
3. Is compute bounded (warehouse size, DBU cap, BigQuery bytes) so a bad join cannot silently bill four figures?
4. Does on-call get a named owner and a runbook link, not only a failed-task email?

For scheduling patterns see orchestration. For cost levers see Snowflake cost optimization and the Snowflake cost calculator. Use the cron builder when you are writing the schedule string, not after the DAG has already missed an SLA. If a review comment is “this is not production-ready,” it usually maps to one of the four checklist items above — fix that, then merge.

Related: architecture, library.

← Back to Home