Soda Core is an open-source data quality framework that enables data engineers to write data quality checks using a simple, human-readable YAML-based language called SodaCL (Soda Checks Language). Unlike code-heavy alternatives, Soda is designed to make data quality testing accessible to anyone who understands the data — not just engineers.
Why Soda?
Data quality issues are the #1 source of broken dashboards, failed ML models, and lost trust in data. Soda makes it easy to:
- Define checks in YAML: No Python or SQL expertise required
- Run anywhere: CLI, CI/CD, Airflow, or Soda Cloud
- Cover all sources: Snowflake, BigQuery, Postgres, Spark, Dask, and more
- Alert immediately: Get Slack/email alerts when checks fail
SodaCL Check Examples
``yaml``
# checks for orders table
checks for orders:
# Row count should be positive
- row_count > 0
# No null values in critical columns
- missing_count(order_id) = 0
- missing_count(customer_id) = 0
# Revenue should be positive
- min(revenue) >= 0
# Freshness — data should be recent
- freshness(created_at) < 1d
# Duplicate check
- duplicate_count(order_id) = 0
# Schema validation
- schema:
fail:
when required column missing: [order_id, customer_id, revenue]
Key Features
1. SodaCL: Human-readable YAML checks language
2. Multi-Source: 20+ data source connectors
3. CI/CD Integration: Run checks in GitHub Actions, GitLab CI
4. Distribution Checks: Detect anomalies in data patterns
5. Schema Monitoring: Alert on unexpected schema changes
6. Freshness Checks: Ensure data is up-to-date
7. Reference Checks: Validate referential integrity across tables
Soda Core vs Soda Cloud
- Soda Core: Free, open-source CLI tool for running checks locally
- Soda Cloud: Commercial SaaS with dashboards, alerting, collaboration, and incident management
Soda vs Great Expectations
| Feature | Soda Core | Great Expectations |
|---------|-----------|--------------------|
| Language | YAML (SodaCL) | Python |
| Learning Curve | Low (YAML) | Medium (Python) |
| Setup | Simple CLI | More configuration |
| Non-Engineer Friendly | Yes | Limited |
| Anomaly Detection | Built-in | Via plugins |
| Enterprise Features | Soda Cloud | GX Cloud |
Common Use Cases
1. Pipeline Quality Gates: Fail pipelines when data quality drops
2. Schema Monitoring: Detect unexpected column changes
3. Freshness Monitoring: Alert when data stops arriving
4. Data Contracts: Validate producer SLAs on data quality
5. CI/CD for Data: Run quality checks on every dbt model change