Memory for agents: what to persist, what to summarize, what to discard
Key takeaways
- Agent memory fails in two ways: too little (agents repeat work and lose context between sessions) and too much (retrieval latency grows, noise buries signal, and costs compound).
- The three-tier memory model: working memory (current session context), episodic memory (task outcomes and decision logs from recent sessions), and semantic memory (durable facts about the domain, the client, and the rules).
- Summarize, do not store: at the end of each session, an agent should compress its working memory into a structured summary and write that to episodic memory. Raw transcripts are expensive to retrieve and rarely useful.
- What to discard: intermediate reasoning, failed attempts, and anything that can be reconstructed faster from a live tool call than from memory retrieval.
Most agent failures aren't model failures. They're memory failures.
An agent that can't remember what it decided last Tuesday, why it chose a particular approach, or what the client's standing constraints are will repeat mistakes, ask redundant questions, and produce inconsistent output. Stateless is fine for a calculator. It's broken for anything that compounds over time.
Here's how to build agent memory that actually works.
The three-tier model
Agent memory isn't a single store. It's three distinct layers with different write patterns, retrieval characteristics, and cost profiles. Conflating them is the most common architectural mistake.
Working memory
Working memory is the current conversation context plus active task state. It's what the agent holds in its context window right now: the user's latest message, the tool calls made this session, intermediate results, and the current plan.
Working memory is ephemeral by design. It lives in the context window and dies when the session ends. Its job is to support coherent reasoning within a single task execution — not to persist anything.
The failure mode here is treating working memory as the only memory. Agents built this way are perpetually amnesiac. Every session starts from zero. Users re-explain context. The agent re-derives conclusions it already reached. Compounding work becomes impossible.
Working memory should be kept lean. Stuffing it with retrieved history to compensate for missing episodic and semantic layers degrades reasoning quality and inflates token cost. The right fix is building the other two tiers.
Episodic memory
Episodic memory is a structured log of completed tasks: what was done, what decisions were made, and what the outcomes were. Think of it as the agent's case history.
A well-formed episodic record for a single task contains:
- Task description: what was requested and the scope agreed upon
- Key decisions: forks in the approach and the reasoning behind each choice
- Outcome: what was produced, whether it succeeded, and any quality signals
- Anomalies: anything unexpected that future sessions should know about
Episodic memory is written at session end, not during execution. During execution, the agent is in working memory. At the close of a session, it compresses that working memory into a structured summary and writes it to episodic storage. The raw transcript is discarded.
Retrieval from episodic memory uses recency bias. The last three sessions on a given task type are almost always more relevant than something from six months ago. Weight accordingly.
Semantic memory
Semantic memory holds persistent facts that don't change session to session: client name and preferences, domain definitions the agent uses consistently, style rules, standing instructions, and any constraints that apply across all tasks.
This is the layer most teams skip, and it's the one that makes the biggest difference to output consistency. An agent that has to re-learn that a client uses British English, prefers numbered lists over bullets, and never wants pricing mentioned in public-facing copy — every single session — is not a useful agent. It's a very expensive autocomplete.
Semantic memory is written deliberately, not automatically. It's updated when a standing fact changes, not after every session. Retrieval uses relevance bias: pull the facts most pertinent to the current task, not everything in the store.
The practical implementation is a key-value or vector store with explicit namespacing: client.*, domain.*, style.*, constraints.*. Keep entries short and factual. Semantic memory is not a place for prose.
The summarization pattern
The transfer from working memory to episodic memory is where most implementations break down. The naive approach is to store the full conversation transcript. Don't.
Raw transcripts are expensive to store, expensive to retrieve over, and rarely useful. A 40-message conversation contains maybe 300 words of genuinely reusable signal. The rest is scaffolding: clarifying questions, intermediate tool outputs, reasoning traces that led nowhere, and filler.
The summarization pattern works like this:
- At session end, pass the working memory context to a summarization step
- The summarization step extracts the structured fields listed above (task, decisions, outcome, anomalies)
- The structured summary is written to episodic storage
- The raw working memory is discarded
This step should be a dedicated prompt, not an afterthought. The quality of your episodic memory is entirely determined by the quality of your summarization. A lazy summary produces a useless episodic record. A precise summary produces an agent that gets meaningfully smarter with each completed task.
For high-stakes tasks, add a confidence field to the summary: did the agent complete the task with high confidence, or were there unresolved ambiguities? This signal is useful at retrieval time — low-confidence summaries should trigger more careful review before the agent acts on them.
Retrieval architecture
Retrieval is where the cost and latency tradeoffs become concrete.
For episodic memory, use semantic search with a recency multiplier. The query is typically derived from the current task description. Retrieve the top-k most relevant summaries, weighted toward recent sessions. Five to ten summaries is usually sufficient context without overwhelming the working memory budget.
For semantic memory, use pure relevance search. Recency doesn't matter — a style rule from eighteen months ago is as valid as one written yesterday. Pull only the facts relevant to the current task type.
The two stores should be queried in parallel at session start, not sequentially. Retrieval latency is the most common performance complaint in production agent systems, and sequential retrieval doubles it unnecessarily.
Do not retrieve everything and let the model sort it out. That approach triples retrieval latency, inflates context size, and degrades reasoning quality as the model tries to process irrelevant material. Precision at retrieval time is not premature optimization — it's the difference between a 2-second response and a 6-second one.
The cost of storing everything
A naive "store everything" approach has a predictable cost curve: retrieval latency grows roughly proportional to log size, and token cost grows with every session that pulls a bloated context.
At small scale this is invisible. At 500 sessions it becomes noticeable. At 5,000 sessions it becomes a budget line item.
Summarization is the only sustainable path. A well-summarized episodic store stays lean regardless of session count because each entry is bounded in size. Retrieval latency stays flat. Token cost per session stays predictable.
The math is straightforward: a 40-message session compressed to a 200-word structured summary reduces storage by roughly 95% and retrieval token cost by the same factor. Over thousands of sessions, that's not a rounding error.
What not to persist
Being explicit about what to exclude is as important as knowing what to store.
Never persist:
- Raw LLM outputs longer than a few sentences — summarize them or extract the decision, not the prose
- Intermediate reasoning traces — chain-of-thought is scaffolding, not signal; it belongs in working memory and nowhere else
- Failed-attempt logs, unless the failure type is one the agent needs to learn to avoid — storing every failed tool call creates noise that degrades retrieval precision
- Redundant entries — if the same standing fact appears in three episodic summaries, it belongs in semantic memory once, not episodic memory three times
The discipline here is editorial. Someone — or some automated process — needs to decide what's worth keeping. Agents that write to memory without a filter produce stores that are expensive to query and unreliable to act on.
The design rule
Memory is a tool, not a transcript.
A transcript records everything that happened. A tool stores what's needed to do the next job better. The distinction sounds obvious, but most agent memory implementations default to transcript behavior because it's easier to implement: just log everything and retrieve later.
The result is an agent that carries the weight of every past session without gaining the benefit of any of them. Retrieval is slow, context is bloated, and the signal-to-noise ratio degrades with every session added.
Build memory with the same discipline you'd apply to any other data store: define what goes in, define what gets pruned, and optimize for retrieval precision over storage completeness. The agents that compound value over time are the ones built on memory that was designed to be useful, not just comprehensive.
Frequently asked questions
What is agent memory and why does it matter for AI workflows?
Agent memory is the mechanism by which an AI agent retains and retrieves information across steps, sessions, or tasks. Without it, every agent invocation starts from zero — no context from prior runs, no accumulated knowledge, no ability to learn from past errors. Memory is what separates a stateless script from an agent that actually improves over time. In practice, it determines whether your agent can handle multi-step tasks coherently, personalize responses based on history, and avoid repeating the same mistakes. For production workflows, memory architecture is often the difference between an agent that works once in a demo and one that holds up under real load.
What is the difference between working memory and long-term memory in AI agents?
Working memory is the live context window — everything the agent can see and reason over in a single inference call. It is fast, zero-latency, and limited by the model's context length (typically 8K–128K tokens). Long-term memory lives outside the model: vector stores, key-value databases, or structured logs that persist across sessions. The agent must explicitly retrieve from long-term memory before it can use it, which adds a retrieval step and latency. Working memory is best for immediate task context; long-term memory is best for facts, user preferences, and accumulated knowledge that would overflow the context window or need to survive between sessions. Most production agents use both in combination.
How do you prevent agent memory from becoming too expensive or slow?
Three levers control memory cost and latency: what you store, how long you keep it, and how you retrieve it. On storage: be selective — log decisions and outcomes, not raw token streams. Summarize completed task threads rather than retaining full transcripts. On retention: apply TTLs and tiered storage so hot, recent memory stays in fast retrieval while cold memory moves to cheaper object storage. On retrieval: use embedding-based semantic search only when necessary; exact-match or structured lookups are an order of magnitude faster for known keys. Benchmark retrieval latency separately from inference latency — memory overhead is often invisible until it compounds across hundreds of agent steps.