Context engineering: the time a photo took an agent down

A client sent us five photos of their shop so the agent could build the gallery for their site. The agent processed the first, then the second, and died on the third. Not because of the model: because of us.

The images were going into the agent's message history as base64. Each photo added hundreds of thousands of characters to a state that gets persisted, reloaded, and sent in full on every step. On the third image the payload blew past the tracing platform's limit and requests started getting rejected. And even if it had not, every later response would have dragged that dead weight along.

The real problem is not size

It is tempting to read this as a limits problem and assume a bigger context window fixes it. It does not. What we were suffering has a name: context rot, the gradual decay in quality as irrelevant tool outputs, stale intermediate state, and redundant re-reads pile up.

The industry numbers are blunt: a large share of enterprise agent task failures are attributed to context drift or memory loss rather than running out of tokens. In other words, the agent does not run out of room. It fills up with junk.

The rule we wrote

After that episode, the rule went into our rules file in capital letters:

Injecting base64 images into the agent message history is forbidden.

And with the rule came the pattern that replaces it. The agent never sees the image. It receives a path. When it needs to understand what is in that photo, it calls a tool that hands the file off to a separate processor. That processor is the only thing authorized to read the file, encode it, and call the multimodal model. What comes back to the agent is text: "storefront photo, daytime, sign visible."

The base64 never touches the conversation. The description, which is all the agent needs in order to decide, takes thirty words.

Three things we have applied since

Separate working memory from source of truth. What the agent needs to keep working right now goes in the context. Everything else lives on disk or in the database and gets pulled in when required.

Clear tool results. Once a result has been used and summarized, the raw output is discarded. It is the lightest and safest form of compaction, because it leaves the reasoning untouched.

Take notes outside. The agent writes what matters into a persistent notes file and reads it back when needed. It sounds unsophisticated. It works remarkably well.

What stuck

Today a client can send twenty photos and nothing happens, because none of them enter the context. And the lesson generalized well beyond images: every time we add a new tool, the first question is what it returns and how much of that actually needs to stay in the conversation.

The answer is almost always: far less than it returns.