What is Soda Core?

Soda Core is the YAML-first data quality framework that makes data testing accessible to people who aren't Python engineers. That's the main pitch and it's a real one: analysts, data stewards, and domain experts can write missing_count(email) = 0 in YAML without learning Python's import system or GX's Context/Checkpoint model. Pair that with SodaCL (Soda Checks Language), which has first-class support for common patterns like freshness, schema validation, and distribution anomalies, and Soda hits a sweet spot for teams that want checks-as-config.

Where Soda fits on a modern data stack

Soda sits alongside dbt tests and Great Expectations in the data-quality tooling landscape. It's the closest to dbt tests in simplicity but richer in check types, and the closest to GX in feature breadth but significantly simpler to adopt. Native connectors for Snowflake, BigQuery, Redshift, Databricks, Postgres, and Spark mean it works anywhere your data lives. Runs on CLI, in CI/CD (GitHub Actions, GitLab), inside Airflow/Dagster, or as the managed Soda Cloud SaaS with dashboards and alerting built in.

Field story

A client with a 15-person analytics team (mostly analysts, two data engineers) had adopted dbt for transformation but was drowning in quality issues the analysts couldn't fix themselves. Schema was the data engineers' job, so the feedback loop was slow. We introduced Soda Core with an empty YAML file template per source table and taught the analyst team SodaCL in a 90-minute workshop. Within two weeks, analysts had written 400 checks across 80 tables without data-engineering involvement. The checks covered the business rules the analysts knew and the data engineers didn't: "this column should always be positive," "these IDs must match the CRM," "this text field should never contain the word 'TEST'." Quality incidents dropped sharply. Lesson: the bottleneck wasn't technical; it was the gap between who knew the data (analysts) and who could express checks (engineers). Soda closed that gap.

Soda Core is an open-source data quality framework that enables data engineers to write data quality checks using a simple, human-readable YAML-based language called SodaCL (Soda Checks Language). Unlike code-heavy alternatives, Soda is designed to make data quality testing accessible to anyone who understands the data — not just engineers.

Why Soda?

Data quality issues are the #1 source of broken dashboards, failed ML models, and lost trust in data. Soda makes it easy to:

• Define checks in YAML: No Python or SQL expertise required
• Run anywhere: CLI, CI/CD, Airflow, or Soda Cloud
• Cover all sources: Snowflake, BigQuery, Postgres, Spark, Dask, and more
• Alert immediately: Get Slack/email alerts when checks fail

SodaCL Check Examples

# checks for orders table
checks for orders:
  # Row count should be positive
  - row_count > 0
  
  # No null values in critical columns
  - missing_count(order_id) = 0
  - missing_count(customer_id) = 0
  
  # Revenue should be positive
  - min(revenue) >= 0
  
  # Freshness — data should be recent
  - freshness(created_at) < 1d
  
  # Duplicate check
  - duplicate_count(order_id) = 0
  
  # Schema validation
  - schema:
      fail:
        when required column missing: [order_id, customer_id, revenue]

Key Features

1. SodaCL: Human-readable YAML checks language
2. Multi-Source: 20+ data source connectors
3. CI/CD Integration: Run checks in GitHub Actions, GitLab CI
4. Distribution Checks: Detect anomalies in data patterns
5. Schema Monitoring: Alert on unexpected schema changes
6. Freshness Checks: Ensure data is up-to-date
7. Reference Checks: Validate referential integrity across tables

Soda Core vs Soda Cloud

• Soda Core: Free, open-source CLI tool for running checks locally
• Soda Cloud: Commercial SaaS with dashboards, alerting, collaboration, and incident management

Soda vs Great Expectations

Common Use Cases

1. Pipeline Quality Gates: Fail pipelines when data quality drops
2. Schema Monitoring: Detect unexpected column changes
3. Freshness Monitoring: Alert when data stops arriving
4. Data Contracts: Validate producer SLAs on data quality
5. CI/CD for Data: Run quality checks on every dbt model change

Soda Core pitfalls to watch for:

• SodaCL looks simple but has subtle semantics. missing_count(col) = 0 means "zero NULLs," but missing_percent(col) < 5 counts rows differently if you have other missing_values configured. Read the SodaCL reference carefully; the convenient syntax hides some complex defaults.
• Large-table checks cost money on cloud warehouses. A duplicate_count(*) check on a 2-billion-row Snowflake table scans the table. Batch checks, filter with sample or where, or use Snowflake's sample clauses to reduce compute. Watch your Snowflake credit usage after adding Soda.
• Freshness checks require a reliable timestamp column. freshness(created_at) < 1d fails silently if created_at is optional or NULL-heavy. Prefer a non-nullable, pipeline-assigned _loaded_at or _ingested_at column for freshness.
• Schema-drift checks can be too strict or too lax. when forbidden column present: [deprecated_col] is strict; when required column missing: [...] is lax. Combine both for safety, and version your expected schema in git alongside the check.
• Reference checks across warehouses are expensive. Checking that fact.customer_id values exist in dim_customer requires a join at check time. On big facts, this can dominate pipeline runtime. Use sample or check only recent partitions.
• Anomaly detection needs history to work. Distribution checks and percent-change thresholds require Soda to have seen enough prior data to compute baselines. New tables or first-time runs will trigger false positives. Run in "learn" mode for the first 1-2 weeks before enforcing.
• Variables and templating add complexity. SodaCL supports ${VAR} substitution for reusable checks across environments. Nice, but also the most common source of "check didn't run as expected" bugs. Keep templating minimal.
• Soda Core vs Soda Cloud features aren't always obvious. Some checks (advanced anomaly detection, incident management) are Cloud-only; if you're trying to do them in Core and they don't work, check the feature matrix.
• CI-only testing misses runtime issues. Running Soda in GitHub Actions on a pre-defined test dataset catches logic bugs but not live-data issues. Run Soda also in production pipelines against live data to catch real quality problems.