What Happens When You Give Your Local Agent a Real Memory

Two weeks ago, an agent I run locally for pipeline maintenance rewrote a retry handler using a flat, fixed-delay retry. It looked reasonable.

Two weeks ago, an agent I run locally for pipeline maintenance rewrote a retry handler using a flat, fixed-delay retry. It looked reasonable. It was also the exact pattern that caused a duplicate-row incident in May, one I’d personally debugged for four hours and was very sure I’d never see again. I hadn’t told the agent to avoid it in that session. I’d told a different session, six weeks earlier, in a different conversation that no longer existed anywhere the model could see it. The model didn’t get dumber between May and July. It just never actually knew anything to begin with, past whatever fit in that one conversation’s context window.

That’s the gap between “context” and “memory,” and it’s wider than most agent tooling admits. So I spent a weekend building the smallest version of real memory I could: a local Ollama agent, a SQLite database, and a habit of writing things down. It’s about 80 lines of Python. It’s also the difference between an agent that repeats your worst incidents and one that doesn’t.

The agent embeds its own query, searches a local SQLite store, and only pulls in the notes that actually match — not the entire conversation history.

TL;DR

→ Most “agent memory” in demos is just re-sending the whole conversation transcript on every turn, which is a longer prompt, not memory.
→ Real memory means distilling a short note after a session ends and retrieving only the relevant notes before the next one starts, using embeddings and cosine similarity, not a full transcript replay.
→ Ollama’s /api/embed endpoint combined with the sqlite-vec SQLite extension gives you a working local memory layer in under 100 lines of Python, with no hosted vector database.
→ In testing, an agent with this memory layer correctly recalled a team’s pandas-to-polars migration and avoided repeating a retry-logic mistake tied to a real past incident, both from notes written weeks earlier.
→ Retrieval only helps if notes are written for retrieval: short, dated, and tied to one concrete decision, not a copy-paste of the conversation that produced them.
→ The real failure mode isn’t forgetting, it’s confidently recalling something stale; a memory store needs a way to expire or overrule old notes, or it will resurface outdated decisions with total conviction.

Why Most “Agent Memory” Isn’t Memory

We covered the mechanics of this failure in detail in Why AI Agents Forget: a model has no persistent state between API calls, only whatever text you hand it as context. “Memory” in a lot of agent frameworks is really just a growing transcript, re-sent in full on every turn until it hits a context limit, at which point older turns get silently truncated. That’s not recall, it’s a longer prompt with an expiration date.

Actual memory needs two things a plain transcript doesn’t have: a write step that decides what’s worth keeping after the fact, and a read step that retrieves only what’s relevant to the current task, not everything ever said. That’s a search problem, not a context-window problem, and it’s the same shape of problem as full-text search over any other document store.

Building an Actual Memory Layer

The setup has three pieces: Ollama running a chat model and an embedding model, a SQLite database with the sqlite-vec extension loaded for vector search, and two small functions, one to write a note, one to recall notes.

ollama pull llama3.2
ollama pull nomic-embed-text

pip install sqlite-vec ollama

The Write Path

After each agent session, a short summarization pass turns the transcript into one or two standalone notes, each embedded and stored with a timestamp:

import sqlite3, sqlite_vec, ollama, json, time

def get_db():
    db = sqlite3.connect("memory.db")
    db.enable_load_extension(True)
    sqlite_vec.load(db)
    db.execute("""
        CREATE VIRTUAL TABLE IF NOT EXISTS notes USING vec0(
            embedding float[768],
            +text TEXT,
            +created_at TEXT
        )
    """)
    return db

def write_memory(note_text: str):
    db = get_db()
    resp = ollama.embed(model="nomic-embed-text", input=note_text)
    embedding = resp["embeddings"][0]
    db.execute(
        "INSERT INTO notes(embedding, text, created_at) VALUES (?, ?, ?)",
        (json.dumps(embedding), note_text, time.strftime("%Y-%m-%d")),
    )
    db.commit()

The note itself matters more than the plumbing. "Refactored ingest_events.py" is useless six weeks later. "Ingest job retries must use exponential backoff — a flat retry caused the May 3 duplicate-row incident" is something worth retrieving.

The Read Path

Before the agent starts a new task, it embeds the task description and pulls the closest notes by cosine distance:

def recall_memory(query: str, top_k: int = 3):
    db = get_db()
    resp = ollama.embed(model="nomic-embed-text", input=query)
    query_embedding = json.dumps(resp["embeddings"][0])
    rows = db.execute(
        """
        SELECT text, created_at, distance
        FROM notes
        WHERE embedding MATCH ?
        ORDER BY distance
        LIMIT ?
        """,
        (query_embedding, top_k),
    ).fetchall()
    return rows

Rendered output for a real query looks like this:

>>> recall_memory("add a retry to the ingest job")
[("Ingest job retries must use exponential backoff — a flat
   retry caused the May 3 duplicate-row incident", "2026-05-04", 0.13),
 ("Team migrated pandas -> polars in week 2", "2026-06-02", 0.46),
 ("Prod warehouse resizes to L on Mondays, cost review", "2026-06-10", 0.69)]

Lower distance means a closer match, so the retry note — written five weeks earlier, in a session that no longer exists in any active context window — comes back first and gets injected into the system prompt for the new task.

Wiring Memory Into the Agent Loop

The integration is two calls bookending whatever loop already drives the agent, a pattern that lines up with how we’ve written about designing agent tools generally: keep the interface small, and let the model decide what to do with what it’s given, rather than hardcoding the logic yourself.

def run_task(task: str):
    memories = recall_memory(task, top_k=3)
    memory_block = "\n".join(f"- {text}" for text, _, _ in memories)

    response = client.chat.completions.create(
        model="llama3.2",
        messages=[
            {"role": "system", "content": f"Relevant past notes:\n{memory_block}"},
            {"role": "user", "content": task},
        ],
    )
    result = response.choices[0].message.content

    # after the task completes, distill and store a new note
    summary = summarize_for_memory(task, result)
    write_memory(summary)
    return result

The Token Math

The other reason this beats “just send the whole history” is cost, not just accuracy. Assume an agent that’s been in use for three months, with roughly 400 prior sessions worth of context.

Approach	Context sent per new task	Relative cost per task
Full transcript replay	Grows unbounded; truncated once it exceeds the model’s context window	Increases every session, then degrades silently
Retrieval, top 3 notes	~150–300 tokens, regardless of history length	Flat, independent of how long the agent has been running

That flat cost curve is the same argument for retrieval over brute-force context stuffing that shows up in MCP-style tool design: give the model a narrow, queryable interface to what it needs, instead of handing it everything up front and hoping the important part doesn’t get truncated.

The Gotchas Nobody Warns You About

Stale notes get recalled with total confidence. A cosine-similarity match doesn’t know that a note is six months old and describes an architecture that’s since changed. Store a created_at timestamp and either expire notes past a threshold or have the agent flag anything older than some age as “may be outdated” before acting on it.

Changing the embedding model invalidates the whole store. A vector from nomic-embed-text and a vector from any other embedding model live in different mathematical spaces and aren’t comparable. If you upgrade models, you re-embed every stored note, not just new ones going forward.

Unfiltered note-writing turns into note bloat. If every session writes a note regardless of whether anything worth keeping happened, retrieval quality degrades as the noise-to-signal ratio grows. Gate the write step behind a simple check: did this session change a decision, fix a real bug, or establish a constraint? If not, don’t write anything.

Concurrent agents writing to the same SQLite file will collide. sqlite-vec doesn’t solve multi-writer concurrency for you. If more than one agent instance can run at once, put a lock around the write path or move to a proper client-server database once you’re past a single-agent prototype.

Memory silently retains whatever you fed it. A distilled note about a bug fix can carry along a credential, an internal hostname, or a customer identifier that happened to be in the task description. Treat the memory store like any other data store with retention and access rules, not a scratchpad that’s exempt from them.

The One Principle

A memory system is a curation problem before it’s a storage problem — deciding what’s worth writing down matters more than the vector database you bolt on to retrieve it.

The 80 lines of SQLite and embedding calls above are the easy part, and they’d work identically whether the notes were good or garbage. The actual engineering is in the write path: forcing every note to be short, dated, standalone, and tied to a real decision. Get that part right and it doesn’t matter whether the retrieval layer is sqlite-vec, a hosted vector database, or something fancier — the agent stops repeating May’s incident in July, which was the entire point.