Production notes for data engineers
Warehouse bills, broken DAGs and code-review lessons from Sainath Reddy, a working data engineer. Every article follows our editorial standards.
Production notes on warehouse cost incidents, dbt review comments and Cortex token bills, written by Sainath Reddy, a data engineer in Hyderabad. 103 tutorials so far; read the editorial standards.
This post is for data engineers and analytics leads who run AI agents on Snowflake and want to understand Cortex Sense, the context layer Snowflake announced at Summit 2026. It…
This post is for data platform engineers deciding how to manage Snowflake objects as code. It explains how Snowflake DCM Projects work (DEFINE, PLAN, DEPLOY), when to choose them over…
This post is for Snowflake engineers who need to prove column-level and row-level access controls to an auditor, including controls that hold when an AI agent runs the query. It…
Your data team ships a Cortex-powered analytics agent. Works beautifully. Then the platform team wants to plug in Cursor. The ML team asks about GPT-4o. A product manager hears about…
Ask ten teams if they’ve built an AI agent and nine will say yes. Look at what they actually shipped and most of it is a workflow with one LLM…
There are a hundred “AI concepts explained for beginners” listicles, and most of them are written for people who will never build anything. This one isn’t. If you’re a data…
A team I worked with upgraded their support-ticket triage pipeline to a bigger, newer model expecting fewer mistakes. Benchmarks said it was smarter across the board. Two weeks in, the…
The Slack message came in at 9:14 on a Tuesday: “why is yesterday’s Snowflake bill $4,200 over budget and who approved it.” Nobody had approved anything. A dashboard refresh job…
Full data refreshes are attractive because they are simple. Extract the source data, replace the analytical dataset, refresh downstream reporting, and repeat.
You can be good at SQL and still struggle in a Data Engineer interview. You can work with Snowflake every day. You can build production pipelines, write dbt models, debug…
This post is for data engineers who need geo-targeted scraping through residential or mobile proxies and want to see how 2extract works in practice. It covers the 2extract MCP server…
When the consumer at the end of a pipeline is a language model rather than a dashboard, the expensive step moves from the transform to the last hop, and every…
Every hour, on a schedule, your dbt job rebuilds every model in your project. The staging models run because they always run. The intermediate joins run because they always run….
Two dbt models can be byte-for-byte identical in output, pass every test, and read cleanly in review — and one of them costs $50 a month while the other costs…
Our CI got slower and more expensive at exactly the same rate our test suite got better, and for a while nobody connected the two. We’d done everything the best-practice…
You read about dbt State. You understood the pitch — skip unchanged models, cut compute, stop paying to rebuild what didn’t change. Then you opened your project and stared at…
For the past two years, the standard pattern for running LLM calls in Airflow was a PythonOperator that imported the OpenAI client, called the API, and returned the result as…
The 2 a.m. page said the pipeline “succeeded.” The dashboard was green. And the finance team was still staring at yesterday’s numbers, because one task in a forty-task DAG had…
I once inherited an Airflow repo with 214 DAG files that were, functionally, the same DAG. Each one extracted a table from a source system, loaded it into Snowflake, and…
I evaluated Prefect seriously. I ran it in a staging environment for six weeks, built three real flows, and had the internal conversation about migrating. Then we stayed with Airflow….
Nobody fails a SQL interview because they don’t know the syntax. They fail because they stare at a question about “users with a 5-day login streak” and don’t recognize it…
I once inherited a revenue report that had been green for fourteen months. Every morning it ran, populated a dashboard, and nobody questioned it. Then a new CFO asked why…
A hiring manager I know keeps a folder of rejected portfolio links. Not to be cruel — to remember the pattern. Out of roughly forty SQL “projects” she reviewed for…
You built a RAG pipeline. You retrieved relevant documents. You fed them to Claude. You got back a confident, well-structured answer. The answer sounds great. It cites sources. It reads…