The measure of how well data meets the requirements for its intended use, encompassing accuracy, completeness, consistency, timeliness, and validity.
Data quality is the discipline nobody asks for until they can't trust a dashboard. Every data team goes through the same cycle: launch shiny dashboards, executives use them for six months, someone notices a number is wrong, confidence collapses overnight, and the team spends the next quarter building tests and observability to restore trust. You can skip that crisis by investing in data quality from day one — but most teams don't, because data quality feels like overhead until it stops you from shipping the next feature.
The classic six dimensions (accuracy, completeness, consistency, timeliness, validity, uniqueness) are useful vocabulary but not a workflow. The actual practice is: (1) identify your critical datasets — the 20 tables that feed the board dashboard and the ML model; (2) write automated tests that run on every update (dbt tests, Great Expectations, Soda, Monte Carlo); (3) instrument alerting so silent failures get paged; (4) put owners on each dataset with SLAs. The 10% of tables that matter get 90% of the quality investment. Don't try to test everything — you'll drown in noisy alerts and learn to ignore them.
A B2B SaaS client's CEO had lost trust in the analytics dashboards after finding three separate reporting bugs in one month (one duplicated rows from a fanout join, one was a timezone conversion error, one was a silent schema change that nulled out a column). Our fix was unglamorous: we audited the top 12 dashboard-feeding models, added dbt unique, not_null, relationships, and accepted_values tests to every one, wired a Monte Carlo contract on the two highest-impact tables, and set up a dedicated Slack channel for quality alerts with a rotating on-call. Three months later: zero dashboard bugs escaped to the CEO, the on-call channel was quiet (tests caught issues upstream), and the CEO started making more decisions from the data. Data quality is reputation management as much as engineering.
Data quality refers to the overall fitness of data for its intended purpose. High-quality data is accurate, complete, consistent, timely, and valid. Poor data quality can lead to flawed analytics, bad business decisions, and compliance issues.
Data quality program pitfalls that every team repeats:
uniqueness tests that "pass" on empty tables. A dbt unique test on an empty or all-null column returns true. Always combine with not_null and row-count minimums. This surprised me in prod more than once.SELECT * pattern breaks. Schema contracts, dbt contracts ({{ config(contract={enforced: true}) }}), or explicit schema tests (Soda's schema: check) catch these.row_count between X and Y ranges or percent-change detection instead of fixed values.Data quality is how fit your data is for its intended use. High-quality data is accurate (reflects reality), complete (no gaps), consistent (same meaning across systems), timely (current when needed), valid (conforms to expected formats), and unique (no unwanted duplicates). Poor data quality leads to broken dashboards, bad executive decisions, failed ML models, regulatory fines, and — worst of all — lost trust in the data team. Data quality is reputation management as much as it is engineering: once trust is lost, it takes months to rebuild.
Start with the six classic dimensions — accuracy, completeness, consistency, timeliness, validity, uniqueness — and define measurable checks for each. Examples: `null_count(email) = 0` (completeness), `duplicate_count(order_id) = 0` (uniqueness), `max(created_at)` within last hour (timeliness), `status IN ('active', 'cancelled', 'refunded')` (validity). Implement with dbt tests, Great Expectations, or Soda. Track metrics over time: percentage of tests passing, MTTR (mean time to repair) on failures, percentage of incidents caught by tests vs. discovered by users. Quality has to be measurable to be managed.
The usual suspects: manual data entry errors (free-text fields where dropdowns would do), system integrations with no validation (source system changes a field type and nobody notices), timing issues (join to a dimension that's not yet loaded, getting NULLs), timezone confusion (UTC in the database, localtime in the UI), schema drift (source adds a column, ELT silently drops it), idempotency violations (retried pipelines creating duplicates), and — most common — lack of ownership where "everyone is responsible so no one is responsible." Fix the process, not just the data.
Data quality is the broader concept: is the data fit for use? Is it accurate, complete, timely? Data integrity is narrower: has the data remained unchanged and consistent through its lifecycle? Integrity is usually enforced via database constraints (PKs, FKs, CHECK), transaction controls (ACID), and checksums to detect corruption. Quality includes integrity but also covers semantic correctness — data can have perfect integrity (no corruption) and terrible quality (wrong values). You need both; integrity is usually simpler to automate via DDL, while quality requires ongoing testing and monitoring.
Testing frameworks: dbt tests (simple, built into dbt), Great Expectations (Python-centric, more flexibility), Soda (YAML-based, accessible to non-engineers), Elementary (dbt-native observability). Observability platforms: Monte Carlo, Acceldata, Bigeye, Datafold — these add lineage-aware ML-based anomaly detection on top of simple tests. Governance: Atlan, Collibra, Alation. Most teams start with dbt tests (free, easy), graduate to Great Expectations or Soda for advanced cases, and add an observability platform when they hit critical mass. There's no single "best" tool; pick based on team preference and integration needs.
A Service-Level Agreement for data — a formal commitment about the quality characteristics of a dataset. Example SLAs: "orders table refreshed every hour," "customer dim 99.9% complete," "revenue numbers reconciled to finance system within 0.1%." SLAs force explicit trade-offs (faster refresh = more infrastructure cost) and give consumers a basis for trust. They're essential in data contracts and data mesh architectures where multiple teams consume each other's outputs. Without explicit SLAs, quality expectations stay implicit, producers and consumers disagree, and trust erodes.
Pragmatic sequence: (1) inventory your critical datasets — the 10-20 tables feeding high-visibility dashboards or ML models; (2) assign a named owner to each; (3) add basic dbt tests (`unique`, `not_null`, `relationships`, `accepted_values`) to those tables; (4) wire alerts to a Slack channel with on-call rotation; (5) track MTTR and test-pass-rate as metrics; (6) expand coverage and sophistication (freshness, distribution, schema contracts) over successive quarters; (7) add an observability platform once you've outgrown alert noise. Don't try to boil the ocean — start with what matters and expand.