What is Great Expectations?

Great Expectations (GX) is the Python-native data-testing framework that popularized the practice of "expectations as code" in data engineering. The core conceptual move is clean: instead of ad-hoc SQL checks scattered across pipelines, you declare what you expect of the data ("user_id should never be null," "status should be one of these five values," "order_total mean should be between $50 and $200") and GX validates those expectations against live data on every run. It auto-generates documentation, integrates with Airflow/Dagster/dbt/Spark, and has been in production at thousands of data teams since ~2017.

Where GX fits (and where it doesn't)

GX shines when you need: cross-source validation (same expectations across Pandas, Spark, SQL), rich expectation vocabulary (75+ built-in plus custom), auto-generated data docs, and Python integration. It's heavier than dbt tests (more setup, learning curve on the Context/DataSource/Checkpoint abstractions) and less approachable for non-engineers than YAML-based tools like Soda. For teams already deep in Python — ML pipelines, Pandas workflows, Spark jobs — GX is a natural fit. For SQL-first dbt teams, the overhead often isn't worth it; dbt's built-in tests plus a few custom ones cover 80% of the need.

Field story

A marketing-analytics client used Pandas heavily for daily campaign ROI calculations running on EMR. They'd ship a "passing" notebook to production and discover a week later that one of 40 source CSVs had added a new column that shifted their column-index-based parsing by one position — every metric was wrong for a week. Adding GX expectations as the first step in the pipeline (column order, column types, non-null ratios, value ranges for each of the 40 sources) caught the next three incidents before they reached prod. The investment was one engineer for two weeks to write expectation suites plus GitHub Actions integration. ROI paid back in the first incident. GX worked specifically because the pipeline was Python-centric — if it had been a dbt project, dbt tests would have been simpler.

Great Expectations (GX) is an open-source Python library for data testing, documentation, and profiling. It helps data teams define "expectations" about their data and validate those expectations automatically in pipelines.

Core Concepts

1. Expectations: Assertions about data (e.g., "column A should not be null")
2. Expectation Suites: Collections of expectations for a dataset
3. Data Sources: Connections to your data (Pandas, Spark, SQL)
4. Checkpoints: Validation runbooks that execute expectations
5. Data Docs: Auto-generated documentation of expectations and results

Example Expectations

import great_expectations as gx

# Create a Data Source
context = gx.get_context()
validator = context.sources.add_pandas("my_data").read_dataframe(df)

# Define Expectations
validator.expect_column_values_to_not_be_null("user_id")
validator.expect_column_values_to_be_in_set("status", ["active", "inactive"])
validator.expect_column_mean_to_be_between("order_total", 50, 200)

Why Teams Use Great Expectations

• Catch Issues Early: Validate data before it reaches downstream
• Documentation: Auto-generate data quality docs
• Collaboration: Share expectations across teams
• Integration: Works with Airflow, dbt, Spark, and more
• Open Source: Free to use with commercial support

Great Expectations GX Cloud

The SaaS version adds:

• Hosted expectation management
• Collaboration features
• Alerting and notifications
• Metrics and dashboards

Integration with Data Tools

• Airflow: GX operators for pipeline validation
• dbt: Run GX after dbt models
• Spark: Validate large-scale data
• Prefect/Dagster: Native integrations

Great Expectations pitfalls I've hit in production:

• The v0.x to v1.x migration is painful. GX went through a major API redesign around v0.18 → v1.0 (late 2024). Context, DataSource, and Checkpoint APIs all changed. Code from 2023 tutorials often doesn't work in 2026. When searching for help, check the version the article targets.
• Expectation generation via profiling creates too many low-value checks. auto-profiler will generate 200 expectations on a wide table, most of which are noise (e.g., expect_column_median_to_be_between on an ID column). Prune ruthlessly to the expectations that matter; curated suites beat auto-generated ones.
• Data Docs generation can be heavy. Rebuilding HTML docs on every run adds 10-30s. For fast-running pipelines, this matters. Configure GX to generate docs only on failure or on a separate cadence.
• Checkpoint configs have surprising defaults. result_format=BASIC by default truncates which rows failed, making debugging harder. Set result_format=COMPLETE or at least SUMMARY in production checkpoints for useful failure details.
• Memory pressure on large datasets. Pandas-backed expectations load entire DataFrames into memory. For datasets that don't fit, use Spark-backed or SQL-backed datasources. Teams frequently discover this at 3 AM when GX OOMs on a large nightly run.
• expect_column_values_to_match_regex with complex patterns is slow. Regex evaluation on billions of rows is expensive. Use SQL-native pattern matching where possible; consider whether the check is actually needed or if a simpler type check covers it.
• Expectations that pass on empty data. Most expectations return "pass" on zero rows — technically correct but misleading. Always pair with expect_table_row_count_to_be_between(min_value=1, ...) to fail-fast on empty input.
• CI/CD integration friction with GX Cloud. GX Cloud adds a lot (dashboards, alerting) but has its own API and setup. Teams sometimes dual-run OSS + Cloud during transitions, which doubles maintenance. Pick one.
• Python dependency conflicts with downstream stacks. GX pulls in many transitive dependencies (Pandas, SQLAlchemy, Jinja2). When added to an existing Python environment with pinned versions, conflicts are common. Consider running GX in its own virtualenv or container.