Building AI Agents: What Actually Works in Production

Ask ten teams if they’ve built an AI agent and nine will say yes. Look at what they actually shipped and most of it is a workflow with one LLM…

Ask ten teams if they’ve built an AI agent and nine will say yes. Look at what they actually shipped and most of it is a workflow with one LLM call in the middle — a fixed pipeline where a model fills in one step, not an autonomous system deciding its own path. That’s not a criticism. It’s the most important thing nobody says out loud about building AI products in 2026: the stuff that actually works in production is far less “agentic” than the demos, and the teams getting value are the ones who figured out which 20% of true agency is worth the risk and hard-coded the other 80%.

There’s a clean mental model underneath the hype, though, and it’s worth having whether you’re building a real agent or an honest workflow. An agent is just a language model wrapped in five things: tools (what it can do), knowledge (what it knows), memory (what it remembers), a loop (how it acts), and guardrails (what it must not do). Answer “what tools, what knowledge, what memory” for your use case and the architecture mostly writes itself. This is that model, told from a data engineer’s chair — where “tools” means governed access to your warehouse, “knowledge” means your RAG and your data quality, and every one of these five parts fails in ways you’ve seen before under different names.

TL;DR

→ An AI agent is an LLM plus five components: tools, knowledge, memory, a reasoning loop, and guardrails — design those and the architecture follows.
→ Most production “agents” are really workflows with one smart step, and that’s usually the correct, cheaper, safer choice — reach for true agency only when the path genuinely can’t be fixed in advance.
→ What works: narrow single-purpose agents, curated tool sets, humans approving risky actions, and evals gating every release. What’s still hype: fully autonomous do-anything agents acting unsupervised.
→ For a data engineer, “give the agent tools” means governed, least-privilege access to your systems — an agent with write access is a service account that makes its own decisions.
→ “Knowledge” is a RAG-and-data-quality problem, not a model problem; most agent failures trace to bad retrieval or stale data, not a weak LLM.
→ The reasoning loop needs a hard step cap, and the whole thing needs evals — an agent without evaluation is an untested pipeline with a mind of its own.

The five parts of an agent

Strip away the branding and every agent, simple or complex, is a model surrounded by the same five components. The value of the model is that it forces you to answer concrete questions before you write code — and each question maps onto infrastructure you already own.

The whole model on one page: an LLM is just the reasoner. Tools, knowledge, memory, the loop, and guardrails are the parts you actually engineer — and the parts that actually break.

Tools — what it can do. The functions the agent can call: query a table, hit an API, write a file. For a data engineer this is the highest-stakes component, because “give the agent a tool” means “grant a non-deterministic system access to your infrastructure.” A read-only tool against account-usage views is low risk; a tool that can suspend a warehouse or write to a table is a service account that makes its own decisions. Least privilege isn’t a nice-to-have here — it’s the whole safety model, which is why I’ve written separately on giving agents access without opening security holes. The emerging standard for exposing these tools cleanly is MCP, which I broke down in MCP explained in 3 levels.

Knowledge — what it knows. The facts the agent needs that aren’t in the model’s training: your schemas, your docs, your current data. This is a retrieval problem, and it’s where most agent projects actually fail — not because the model is weak, but because the retrieval is bad or the underlying data is stale. If you’re wiring this up on your own data, the mechanics are in the Cortex Search RAG guide. The uncomfortable truth: your agent is only as good as the data platform underneath it.

Memory — what it remembers. State that persists across steps or sessions, so a follow-up like “now do the same for last quarter” doesn’t start from zero. In practice this is checkpointing — the same durable-state thinking you apply to any pipeline that has to resume after a failure rather than restart.

The loop — how it acts. The reason-act-observe cycle that separates an agent from a single call: the model reasons, calls a tool, reads the result, and decides what to do next. This loop is the source of an agent’s power and its danger, which is why it always needs a hard cap on steps.

Guardrails — what it must not do. Input validation, output filtering, step limits, and human approval for risky actions. These are the AI equivalent of the constraints and tests you’d never ship a pipeline without — the difference between a demo and something you can leave running.

The distinction that matters most: workflow vs agent

Before you build anything, answer one question honestly: does the task need the model to decide the path, or just to do one hard step along a path you already know? Getting this wrong is the most common and most expensive mistake in the space.

A workflow runs a path you defined; an agent decides the path at runtime. The runtime freedom is exactly what makes agents powerful — and exactly what makes them harder to test, secure, and cost-control.

A workflow is a fixed sequence you designed, with a model doing the intelligent part of one or more steps — classify this ticket, extract these fields, draft this summary. You control the path; the model fills in the reasoning. An agent hands the model the wheel: it decides which tools to call and in what order, looping until the task is done. The agent is more flexible and can handle open-ended tasks a rigid pipeline can’t — but that same freedom is what makes it harder to test (the path changes every run), harder to secure (it can call tools in combinations you didn’t anticipate), and harder to cost-control (each loop is a billed call). The senior move is to default to a workflow and escalate to agency only where the task genuinely can’t be pre-planned — which is the same restraint I argued for in choosing tools over subagents.

What actually works (and what’s still theater)

Here’s the honest cut from teams actually running this in production, stripped of vendor optimism. The pattern is consistent: narrow beats broad, supervised beats autonomous, and boring beats clever.

The consistent signal from production: narrow beats broad, supervised beats autonomous, curated beats sprawling. The right column is where budgets and pilots go to die.

The single most reliable pattern is the narrow, single-purpose agent: one job, a small curated set of tools, and a human in the approval path for anything consequential. The fantasy that keeps failing is the autonomous do-anything agent with dozens of tools and no supervision — it demos beautifully and falls apart on the long tail of real inputs. Multi-agent “swarms” are especially over-applied; most problems that get a swarm proposal are better served by one well-scoped agent or, more often, a workflow. And underneath all of it: without evals gating releases, you have no idea whether a change helped or hurt, which means “it looked good in the demo” becomes your entire QA process. This connects to a deeper reliability problem I covered in why larger models give confidently wrong answers in production — more autonomy multiplies the blast radius of that failure mode.

A pragmatic way to start

If you’re building your first AI product, resist starting with “let’s build an agent.” Start with the three founding questions — what tools, what knowledge, what memory — and answer them for the narrowest useful version of the task. Then build the simplest thing that could work, which is almost always a workflow: a fixed path with one or two LLM-powered steps, real retrieval behind the knowledge step, and a guardrail on anything that touches production. Add evals before you add capability. Only when you hit a task where you genuinely can’t predetermine the path — where the model really does need to choose among tools dynamically — do you graduate to a true agent, and even then you keep the tool set tight and a human on the risky actions. The teams shipping working AI products aren’t the ones who built the most autonomous system. They’re the ones who were honest about how little autonomy the job actually required.

The gotchas nobody warns you about

Most “agents” should be workflows. If you can draw the path in advance, you don’t need an agent — you need a pipeline with a smart step. Building an agent for a workflow problem buys you cost, unpredictability, and a security surface you didn’t need.

Every tool is an attack surface and a cost center. More tools means more ways for the agent to do something surprising, and more tokens spent deciding among them. Curate ruthlessly; a tight tool set outperforms a sprawling one.

Your agent’s ceiling is your data platform. Bad retrieval, stale data, or missing metadata will sink a great model. Agent quality is a data-quality problem wearing a trench coat.

The loop will run away if you let it. A missing step cap turns a confused agent into a runaway bill. Set a hard recursion limit before you set anything else.

No evals means no engineering. If you can’t measure whether a change improved the agent, you’re not building a product, you’re tuning a slot machine. Evals — including credit for the agent saying “I can’t do this” — come before more features.

The one principle

An agent is a language model wrapped in tools, knowledge, memory, a loop, and guardrails — and “what actually works” is deciding, for each of those five, how little you can get away with. The instinct that ships working AI products isn’t reaching for maximum autonomy; it’s the engineer’s instinct to build the simplest system that solves the problem, grant the least access that does the job, and measure everything. You already have those instincts. Building agents is mostly the discipline of not abandoning them because the technology is exciting.