Agent memory: the problem is not remembering, it is forgetting

When somebody says their agent "has memory," they almost always mean it stores the conversation history. That is the easy part. The hard part, and where it gets decided whether the agent is still useful six months in, is choosing what to throw away.

Three different memories

It took us a while to notice we were lumping three things under one word.

The conversation in flight. What was said in the last few messages. It lives in the agent state, gets persisted so it survives a restart, and has a ceiling: past a point, older messages get summarized.

Facts about the client. That her name is Marta, that her shop is in Córdoba, that she has told us three times she hates orange. This does not belong in the conversation. It belongs in the database, on the contact record, and gets injected when needed. If it lived in the conversation, it would vanish with the first summary.

What happened in the project. Tasks completed, files generated, decisions made. That lives in the project record and gets queried when the agent needs historical context.

The rule that organized all of it: if something has to outlive the conversation, it cannot live in the conversation.

Summarizing without losing what matters

Automatic summarization of long conversations has a trap. Generic summaries keep what "seems" relevant in narrative terms, and eat the operational details. A summary can say "the client asked for changes to the home page" and drop the fact that the change was the phone number, which was the only thing that mattered.

Our fix was to steer the summary: tell it explicitly which categories of information must be preserved no matter what. Decisions made, concrete data mentioned, open items. Everything else can be compressed.

Clearing tool results

Of every technique we tried, the best effort-to-benefit ratio came from the simplest: discard a tool's raw result once it has been used.

When the agent reads a twenty thousand character file to extract one value, those twenty thousand characters do not need to still be in the conversation ten turns later. The extracted value, yes. The raw dump, no.

It is safer than summarizing, because it does not touch the agent's reasoning: it only removes reference material that has already been consumed.

Forgetting on purpose

There is one decision we made that looks like a step backwards at first glance: when a project closes, its conversation gets archived. Not deleted, removed from the active context.

The reason is that an agent dragging along the history of a closed project starts answering about the old project when the client comes back with a new one. We saw it happen. It is confusing for the client and hard to diagnose, because technically the agent is working correctly: it remembers too much.

The test we run

Every so often we take a real long conversation and ask the agent about a detail mentioned at the very beginning. If it does not have it, that is not a model memory problem: it means that detail was stored in the wrong place.

The answer is almost never "let's store more." It is almost always "that should have been on the client record from the first minute."