AI Agent Memory: Architectures That Work in Production

🧠What agent memory actually is (and is not)

Strip the marketing away and an AI agent has exactly one native memory: the context window. Everything the model knows about the current task is whatever text you placed in front of it on this call. When people say an agent "has memory", they mean the system around the model is storing state somewhere — a database, a vector index, a file — and deciding what to inject into the next prompt. The model itself learns nothing between calls.

That reframing matters because it turns memory from a product feature into an engineering discipline with three separable problems: what to write, where to put it, and what to read back and when. Teams that conflate the three end up with a vector database full of unfiltered chat logs and an agent that confidently recalls things it should never have stored.

The working definition we use when scoping agent builds: memory is any state written by one step of the agent and read by a later step, where "later" can be the next token or the next quarter. Short-term memory is state that lives inside one run. Long-term memory is state that survives the run. Almost every production failure we have been called in to fix came from treating those two as the same bucket.

Agent memory, defined in the glossary→

🗂️Short-term (context) vs long-term (vector and structured) memory

Short-term memory is the context window itself: the system prompt, the conversation so far, tool results, and any scratchpad the agent maintains during a run. It is fast, perfectly attended to (mostly), and brutally finite. Its real job is coherence within a task — the model should not have to re-derive what it decided three steps ago. The operational consequence is that short-term memory needs active management: compaction, truncation, and deliberate eviction, because the window fills with tool output far faster than with reasoning.

Long-term memory lives outside the window and comes in two mechanical flavours. Unstructured long-term memory is retrieval over embedded text — transcripts, summaries, documents — typically in a vector store. Structured long-term memory is typed state: a user profile row, a preferences table, a knowledge graph of entities the agent has extracted. The mistake is defaulting to the first because it is easier to demo. Embeddings are good at "find text like this" and bad at "what is this customer credit limit", and most of what an agent needs to remember permanently is closer to the second question.

A useful rule of thumb: if you can name the field, it belongs in structured storage. If you cannot — if the value is the texture of a past interaction, a rationale, a narrative — it belongs in retrieval. Production systems we respect run both, with the structured store treated as the system of record and the vector store treated as a recall aid that can be rebuilt from source data at any time.

📚Episodic vs semantic vs procedural memory

The cognitive-science vocabulary is worth borrowing because the three categories have genuinely different storage and retrieval mechanics. Episodic memory is what happened: "on Tuesday the agent tried refunding order 4471 and the payment gateway rejected it." Semantic memory is what is true: "order 4471 belongs to a customer on the legacy billing plan." Procedural memory is how to do things: "to refund a legacy-plan order, route through the manual adjustment flow, not the gateway API."

Episodic memory is the natural fit for vector retrieval. You embed episode summaries — compressed narratives of past runs, not raw transcripts — and recall them when the current situation looks similar. The two design decisions that matter are the unit of storage (one episode per completed task works far better than one per message) and the summary schema (situation, actions taken, outcome, and what the agent would do differently — that last field is where the value lives).

Semantic memory is a knowledge problem, not a search problem. Entity-attribute stores, profile tables, or a small knowledge graph all work; what does not work is hoping a vector index will keep "the customer is on the legacy plan" true over time. Semantic facts need provenance — where did this belief come from, when was it asserted, can it be contradicted — because stale semantic memory is worse than none. An agent that misremembers a fact acts confidently on the wrong world.

Procedural memory is the least standardized and the most valuable in production. Sometimes it is fine-tuning, but far more often it is explicit: versioned playbooks, successful trajectory libraries that get retrieved as few-shot exemplars, or edited tool-use policies. The key property is that procedures are curated by humans or by a gated promotion pipeline, never written directly by the agent during a run. Letting an agent rewrite its own procedures from a single experience is how you get an agent that overfits to one weird Tuesday.

✍️Write policies: what NOT to store

Every production memory system needs a write path that is deliberately harder than the read path. The default in most tutorials — summarizing each turn and embedding it — creates a store where the signal-to-noise ratio collapses within weeks, because most of what happens in an agent run is transient: retries, intermediate tool output, abandoned hypotheses. Retrieval over that noise actively degrades the agent.

The write policies that work are explicit and boring. Store episodes only for completed tasks, and only after outcome labelling — an episode without a known outcome teaches nothing. Store semantic facts only with a source and a timestamp, and never store a fact the agent inferred once without marking it as inferred. Store nothing from a run that failed a safety or policy check, and never let user-supplied content flow into long-term memory without passing the same injection screening you would apply before acting on it.

There is also a category of things to refuse on principle. Secrets and credentials, obviously. Personal data beyond what your privacy posture permits — and remember that a memory store is a datastore for GDPR and similar purposes, with deletion obligations that vector indexes make genuinely awkward. And transient emotional framing: an agent that permanently remembers "this user was angry" from one bad interaction will poison every future interaction with that user. Sentiment is context, not memory.

⏳Retrieval scoring and decay

Pure cosine similarity is a bad retrieval policy for agent memory, and this is one of the places where the literature and production practice agree. Similarity tells you a memory is topically related; it says nothing about whether it is still true, whether it was important, or whether it is the thirtieth near-duplicate of the same episode. Agents retrieve with a blended score — relevance, recency, and importance are the classic three components — and the weighting is a product decision, not a library default.

Recency decay deserves special care because the naive version (exponential decay on age) quietly breaks semantic memory. A customer plan asserted two years ago is still true and should still win; a troubleshooting episode from two years ago is probably obsolete. Decay belongs on episodic memory, not on facts — and facts need explicit invalidation instead, because nothing about a vector will tell you the world changed.

Importance scoring is usually delegated to a model at write time ("rate how much this matters, 1 to 10") and it mostly works, with two caveats. Models inflate importance for emotionally vivid content, which correlates poorly with operational value. And importance should be re-scorable: a fact that gets retrieved and used successfully fifty times is important by evidence, whatever the model guessed at write time. Usage-based reinforcement beats one-shot labelling.

🔒Privacy and tenant isolation in memory

Memory turns tenant isolation from a data problem into a behaviour problem. A shared vector index with a tenant_id filter is the standard pattern, and it is fine until the day a retrieval call forgets the filter — then one customer memories surface in another customer agent run, phrased as the agent own knowledge, and you have a breach that no log obviously records. Production systems enforce isolation in the query layer, not the prompt layer: the tenant filter is applied by code the model cannot influence, ideally as physically separate indexes or at minimum as mandatory server-side filters on a wrapper that raw retrieval calls cannot bypass.

Deletion is the second hard requirement, and the one most designs flunk. When a user exercises a deletion right, you must remove their memories from the vector index, the structured store, the episode summaries (which may contain their data in rewritten form), and any backup that could be restored into a live system. Vector stores with soft-delete or rebuild-based compaction make this tractable; append-only embeddings of raw transcripts make it nearly impossible. Design for deletion at write time or discover the problem during an audit.

None of this is exotic engineering. It is the same discipline as multi-tenant SaaS data, applied to a store that people tend to treat as a toy because it holds text for a model. Treat memory as the most sensitive datastore in the agent system, because it is the one whose contents the model will act on without being asked twice.

💥Failure modes: poisoning, stale beliefs, and recall traps

Memory poisoning is the failure mode that gets the attention it deserves: an attacker (or a careless integration) gets adversarial content into long-term memory, and the agent later retrieves it as if it were its own knowledge. Indirect prompt injection through memory is worse than through a single prompt because it persists and generalizes — one poisoned episode can influence every future run that resembles it. The mitigations are the write-path screening described above, provenance on every record, and treating retrieved memory as untrusted content in the prompt rather than as system-level instruction.

Stale beliefs are quieter and more common. The agent learned something true, the world changed, and nothing invalidated the memory. Unlike a wrong API response, a stale belief fails silently — the agent acts with full confidence on last quarter policy. The fixes are structural: TTLs on episodic memory, explicit invalidation hooks when the underlying system of record changes (your CRM updated the plan; invalidate the derived facts), and periodic re-verification of high-impact semantic memories against source.

Then there are the recall traps. Echo-chamber recall: the agent retrieves its own past outputs, which reinforces early hypotheses and suppresses better ones — visible as an agent that gets more confident and less correct over time. Frequency bias: the most-written-about situation dominates retrieval, so edge cases drown in memories of the common case. And consolidation drift: merged summaries slowly lose the caveat that made the original episode accurate, like a photocopy of a photocopy.

🧰An honest read on framework built-in memory

As of writing — and verify this against current releases before committing, because this layer moves fast — the major agent frameworks all ship memory abstractions, and all of them make the same trade: they optimize for the demo path. LangGraph gives you checkpointers and store primitives that are genuinely solid building blocks — the checkpointer model (persisted run state you can resume) is the right idea, and its long-term store is a reasonable key-value/semantic hybrid — but the write policy, decay, and isolation layers are yours to build. CrewAI memory leans on embeddings plus short-term buffers and is fine for prototypes; we would not put it in front of customers without replacing the defaults. The OpenAI Agents SDK keeps memory deliberately thin — sessions and hosted conversation state — which is honest about what it is, and pushes the hard parts back to you.

Dedicated memory-layer products and libraries (the mem0-style category, plus vector vendors adding memory APIs) sit one level up: managed extraction, deduplication, and retrieval tuning as a service. They earn their keep when memory is genuinely core to your product and your team is small; the risks are the usual ones — another data processor in your compliance surface, pricing coupled to memory volume, and an abstraction you cannot see inside when recall goes weird.

The pattern we recommend regardless of framework: adopt the framework for run-state persistence (checkpoints, resumable runs — this is genuinely hard to build well), and own the long-term memory layer yourself. Your write policies, decay rules, tenant isolation, and deletion semantics are product decisions with legal and security weight. Outsourcing them to a default is how the failure modes above become your incident postmortems.

If you are earlier than that — a support copilot, an internal tool — the honest answer is that you may not need long-term memory at all. A structured profile table and a well-managed context window cover a surprising share of real use cases. Add retrieval memory when you can name the decision it improves.

Pinecone vs pgvector vs Qdrant for the storage layer→RAG vs long-context: the adjacent trade-off→Talk to us about production agent architecture→