Programming Cheat Sheets for Data Engineers

These references are not generic language tutorials. They cover the idioms that show up in data pipelines: typed records instead of untyped dicts, generators that do not load a full extract into RAM, Spark DataFrame operations that stay in the catalyst plan instead of falling back to Python UDFs, and the partition/shuffle mistakes that turn a 12-minute job into a two-hour one.

In a modern warehouse-first shop you will still write Python: Airflow DAGs, dbt Python models on rare jobs, REST extracts, and tests. The decision that actually matters is where the transformation runs — warehouse SQL, Spark, or the orchestrator — not whether you “know Python.”

Python for pipeline work

• Python for Data Engineers — pathlib, typing, dataclasses, context managers, pytest, and pandas patterns that belong in extracts rather than in the warehouse.

Prefer pathlib over os.path, type hints over docstring-only contracts, and pytest over unittest. For tabular work that will land in Snowflake or BigQuery, do the heavy joins in SQL; use Python for I/O, control flow, and APIs.

Distributed processing

• PySpark & Spark SQL Cheat Sheet — DataFrame API, lazy evaluation, broadcast joins, shuffle partitions, and why a Python UDF kills Photon/whole-stage codegen.

If a transformation is expressible in Spark SQL or the DataFrame API, write it there. UDFs are for the leftover 5%. Watch partition counts after a filter, and do not collect() a production volume to the driver.

Streaming producers and consumers

• Snowpipe Streaming & Kafka (Interview) — partitions, consumer groups, exactly-once claims vs at-least-once plus idempotent sinks.

Use the JSON to SQL converter when a payload becomes a table, and the SQL formatter before you paste a Spark SQL string into a review. Related: SQL cheat sheets, orchestration, library.

← Back to Home