Three weeks ago I watched a teammate open the OpenAI billing dashboard and go quiet. We’d built a PII-tagging job that ran a small classification prompt against every new row landing in a raw events table, about 500,000 rows a day, to flag anything that looked like an email, a phone number, or a government ID before it hit a shared schema. It worked. It also cost $340 a day once we accounted for retries, and legal wanted to know exactly which vendor now had a standing copy of our customer data. Neither number was going to survive the next budget review.
The fix wasn’t a smarter prompt or a cheaper API tier. It was moving the model onto the same box that already ran the pipeline. Ollama had been sitting in my “toys, not tools” mental bucket for a year, something for chatting with a local Llama build on a Saturday. Turns out it’s a perfectly serviceable inference server for exactly this kind of narrow, high-volume, structured-output task, and it doesn’t send a single row anywhere.
The whole job runs on infrastructure you already control. Only the base URL in your HTTP client changes if you ever move to a hosted model.
TL;DR
- → Ollama exposes an OpenAI-compatible API onÂ
localhost:11434, so swapping a cloud LLM call for a local one in an Airflow task is usually a one-line change to the client’s base URL. - → Small quantized models (3B–8B parameters at 4-bit) handle narrow, structured tasks like PII tagging, log classification, or doc-string generation well; they are not a drop-in replacement for a frontier model on open-ended reasoning.
- → A 4-bit quantized 3B model needs roughly 2–3 GB of memory instead of the 6+ GB full precision would require, which is why it fits on a shared pipeline host without a dedicated GPU budget.
- → For a 500K-row daily classification job, local inference on existing hardware runs at effectively $0 marginal cost per run, versus real per-token cloud spend that scales with volume.
- → Ollama is a single background daemon, not a per-task process, so the first request in an Airflow DAG can be slow while the model loads into memory, and this needs its own timeout handling.
- → Running the model locally also means the compliance conversation changes: no row of customer data leaves the host you already control access to.
Why This Belongs in the Pipeline, Not Just the Terminal
Most Ollama content is written for a single, interactive session: install it, pull a model, chat with it, done. That’s a fine on-ramp, but it undersells what the tool is actually good for once you strip away the chat interface. Underneath the terminal experience is a plain HTTP server. It handles model loading, memory management, and hardware acceleration, and it exposes a REST API that speaks the same shape as OpenAI’s chat completions endpoint. That last part matters more than the local-vs-cloud framing usually gives it credit for: if your pipeline code already calls an LLM through an OpenAI-compatible client, pointing it at Ollama is a base-URL change, not a rewrite.
That’s the same reasoning that made Snowflake’s own Cortex tooling worth covering here: the interesting engineering question is never “can the model do the task,” it’s “what does it cost to wire this into infrastructure we already run.” Ollama’s answer is: not much, provided the task is narrow enough for a small model to handle reliably.
Installing Ollama on a Pipeline Host, Not a Laptop
The install itself is unremarkable, which is the point. On the Linux boxes running our Airflow workers, it’s a single script:
curl -fsSL https://ollama.com/install.sh | sh
# confirm the daemon is up and check the version
ollama --version
systemctl status ollamaOllama installs itself as a systemd service on Linux, listening on 127.0.0.1:11434 by default. If your Airflow workers and the model need to live on separate hosts, you’ll want to bind it to the internal network interface instead and lock that down with a security group, not expose it publicly:
sudo systemctl edit ollama
# add under [Service]:
# Environment="OLLAMA_HOST=0.0.0.0:11434"
sudo systemctl restart ollamaPicking a Model That Fits the Task, Not the Demo
For structured tagging work, you don’t want the biggest model that fits on the box. You want the smallest one that hits your accuracy bar, because latency and memory headroom compound across half a million rows. We landed on a 3B-class model after testing three sizes against a hand-labeled validation set:
ollama pull llama3.2
ollama pull qwen3:8b
ollama pull gemma4:e4bBy default Ollama pulls a 4-bit quantized build (q4_K_M), which is why a 3B model downloads at roughly 2 GB instead of the 6 GB a full-precision (fp16) version would need. That quantization step compresses the model’s weights into 4-bit integers, and for a classification task with a fixed, narrow label set, the accuracy hit is negligible. It would be a different conversation for open-ended generation.
Wiring It Into an Airflow DAG
The integration point is a plain HTTP call inside a PythonOperator, using the OpenAI-compatible endpoint Ollama exposes at /v1/chat/completions:
from openai import OpenAI
from airflow.decorators import task
client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")
@task
def tag_pii_batch(rows: list[dict]) -> list[dict]:
results = []
for row in rows:
response = client.chat.completions.create(
model="llama3.2",
messages=[
{"role": "system", "content": (
"Classify the input for PII. Respond with strict JSON: "
'{"has_pii": bool, "categories": [string]}'
)},
{"role": "user", "content": row["raw_text"]},
],
temperature=0,
)
results.append({"row_id": row["id"], "tags": response.choices[0].message.content})
return resultsRun output for a single row looks like this once it comes back through the DAG’s logging:
{"row_id": "evt_88213", "tags": "{\"has_pii\": true, \"categories\": [\"email\", \"phone\"]}"}
The output then lands in a staging table and gets merged into the target with a standard MERGE statement, the same pattern we used when writing about Time Travel for auditing exactly which run tagged which row.
The Cost Math
Here’s the comparison that actually mattered to our budget review. Assume 500,000 rows a day, roughly 120 input tokens and 40 output tokens per row for a classification prompt like the one above.
| Approach | Daily token volume | Marginal cost / day | Data leaves the host? |
|---|---|---|---|
| Cloud API, mid-tier model | ~80M tokens | ~$310–$360, scales with volume | Yes |
| Ollama, local 3B model, existing pipeline host | ~80M tokens | ~$0 marginal (existing hardware) | No |
The local path isn’t free in an absolute sense, you’re spending CPU/GPU cycles you already own, and if the host is undersized you’ll eventually pay for a bigger instance. But that’s a fixed, predictable infrastructure cost instead of a bill that grows linearly with data volume, which is the same argument we made when comparing orchestrator licensing models: usage-based pricing is fine until usage is the thing you’re trying to grow.
The Gotchas Nobody Warns You About
The first request in a DAG run pays the cold-load tax. Ollama unloads a model from memory after a period of inactivity (five minutes by default). If your DAG runs hourly, every run’s first task can eat several extra seconds waiting for the model to load back in. Set keep_alive in the request payload to a longer duration, or ping the model with a lightweight warm-up call before the real batch starts.
Concurrent Airflow tasks will silently queue, not fail. Ollama serializes requests to a given model by default unless you’ve explicitly configured parallel request handling. If you fan out ten parallel tasks expecting ten times the throughput, you’ll instead get one queue that’s ten times longer, with no error to tell you why it’s slow.
Schema-constrained output still needs a parser, not blind trust. Even with an explicit “respond with strict JSON” instruction, small models occasionally wrap their answer in a stray sentence or a markdown code fence. Wrap every response in a real JSON parse with a fallback path, the same defensive habit you’d apply to any external API, local or not.
A shared host means a shared blast radius. If the same box also runs other Airflow tasks, a model pinned in GPU memory can starve them of resources in ways that look like an unrelated flaky DAG. Give the inference workload its own resource ceiling, whether that’s a cgroup limit or a dedicated worker pool.
Version drift is real and mostly undocumented per-task. Ollama ships frequent releases, and quantization defaults or model behavior can shift between versions. Pin the Ollama version and the exact model tag (not just latest) in whatever config or container image your pipeline deploys, the same way you’d pin a Python package version.
The One Principle
A local model is an infrastructure decision before it’s an AI decision — treat it like the database or the queue it’s sitting next to, with monitoring, resource limits, and version pins, and the “AI” part of the problem turns out to be the easy half.
That framing is also why this pattern generalizes past PII tagging. The same DAG shape works for classifying malformed records before they hit a Delta Lake or Iceberg table, generating draft column descriptions during a schema change, or triaging alert text before it pages someone. None of those need a frontier model. They need a small one that’s fast, cheap, and doesn’t leave the building, which is precisely the gap Ollama fills once you stop thinking of it as a chat toy and start thinking of it as another service in the stack, not unlike how we’ve written about the tradeoffs inside Snowflake’s warehouse cache or the workflow shifts in Cortex Code Desktop.
Related reading: Snowflake Cortex Code Desktop · Airflow vs. Prefect · Snowflake Time Travel · Ollama · Ollama OpenAI-Compatibility Docs