An AI agent’s memory should look more like an editable rulebook

Remembering a mistake is only half the job

A support agent makes the same mistake twice, and the obvious response is to tell it to remember what happened. But what exactly should it remember? The whole conversation, a customer's private details, the last successful answer, or a general rule that will still make sense next week? Those are different kinds of memory, with different responsibilities attached.

IBM Research's October 20 ALTK-Evolve report describes extracting behavioral guidelines from an agent's task attempts and appending them to the prompt for later tasks, without updating model weights. In its reported retail evaluation, a GPT-4.1-based setup reached 67.5% Pass^4: the share of tasks completed successfully on all four attempts. Its leaderboard comparison is explicitly a snapshot from October 12, not a claim about every future model or real support desk.

The interesting idea is that improvement can happen in the instructions around a model. Our interpretation is that this makes memory worth treating as a maintained product component, not a mysterious place where experiences accumulate. A lesson can be helpful, wrong, overbroad, outdated, or in conflict with a current policy. The ability to edit and retire it matters alongside the ability to collect it.

Below is our proposed workflow for a small sandbox. It is not an implementation guide to ALTK-Evolve, a report of independent testing, or a claim that a few prompt edits reproduce IBM's result. The aim is to make learned instructions inspectable enough that a team can decide which habits it actually wants.

Source: IBM Research: ALTK-Evolve customer-service benchmark report ↗ · Sierra Research: τ²-bench repository and evaluation setup ↗ · AgentToolkit: ALTK-Evolve library ↗

Start with a fictional shop and an explicit policy

Imagine a demonstration shop that lends board games. Its written policy says a reservation can be changed before pickup, but a collected game cannot be marked returned until a staff member confirms its condition. The agent may explain these rules and prepare a draft change, while a person approves any action that changes the shop's records.

Create synthetic conversations for this shop. One customer wants a different pickup time. Another wants to return a game but has not brought it back. A third uses the word cancel while referring to a reservation that has already been collected. There are no real customers, payments, or accounts in this exercise. You are designing a place to inspect behavior without imposing mistakes on anyone.

Suppose a draft agent wrongly treats the third request as an ordinary cancellation. A proposed lesson might be to check the reservation's current state before deciding which action is available. That is a useful candidate because it names an observable step and a reason. A lesson saying always cancel when the customer sounds certain would be a very different habit, even if it happened to solve one conversation.

Keep the policy outside the learned playbook and identify it as authoritative for the exercise. A remembered success should not silently become a new policy. If the shop changes its rules, update the policy through the appropriate human process, then review whether the remembered guidance still fits.

This distinction is the foundation of the sandbox: memory can help apply a rule, but it does not get to invent the rule merely because a past run ended well.

Give every candidate lesson a short record

For each proposed guideline, save the text, the task that prompted it, and a short explanation of the failure it is meant to prevent. Add the scope: which kind of request does it apply to, and what would make it irrelevant? A useful guideline should be understandable without replaying a private conversation in full.

For the board-game example, a candidate record might say to inspect whether a reservation is pending or collected before offering cancellation. The scope is reservation changes, not every interaction with the shop. A separate note can point to the synthetic example that exposed the issue. That gives a reviewer something concrete to challenge.

Avoid keeping customer-like personal details in a rule when they do not belong to the lesson. The synthetic user's name is irrelevant to checking reservation state. In a real project, deciding what can be retained would also require reviewing the applicable data-handling rules and permissions. A successful technical demonstration does not answer that separate question.

Write the instruction as a testable behavior rather than a personality trait. Be careful is difficult to evaluate. Check the current reservation state before selecting the action gives you a visible step to inspect. It still needs exceptions and policy context; the point is making those explicit enough to discuss.

Label a candidate as a candidate until it has been reviewed. Do not let an automatically generated sentence appear indistinguishable from a deliberate product requirement. Even a simple document can use draft, approved, and retired sections. The labels are a workflow suggestion, not a feature we claim every memory library already implements.

Finally, identify a person responsible for deciding what becomes active. Shared responsibility is valuable, but a playbook with no maintainer can accumulate contradictions faster than anyone notices.

Test a lesson against a different conversation

A rule learned from one failure needs more than a replay of that same failure. Create a second synthetic case where the intended behavior is similar but the wording differs. Then create a nearby case where the rule should not apply. You are checking scope, not asking the model to memorize a particular sentence.

For example, the customer might ask to move pickup to tomorrow rather than saying change my reservation. The agent should still inspect the relevant state before selecting an action. In a separate case, the customer only asks for opening hours. Forcing an unnecessary reservation lookup there would suggest the lesson has become too broad.

Compare a sandbox version without the candidate guideline to a version with it, keeping the policy and tools unchanged. Retain the individual conversations and final simulated record states. Do not keep editing the evaluation cases until the candidate looks good. If you use a case to refine the guideline, mark it as a development case and reserve different examples for the next check.

Inspect failures by type. Did the agent choose an unavailable action, ask an unnecessary question, miss the user's actual request, or describe a change it never performed? These distinctions tell you whether the candidate addressed the original problem or merely moved the friction elsewhere.

Repeat a few cases rather than assuming one success is stable. IBM's report distinguishes average task success from success across repeated attempts. In your much smaller exercise, keep that distinction as a question without claiming a statistically reliable estimate. A handful of runs can expose a recurring problem; it cannot establish that the system will never fail.

A guideline earns its place when you can describe both what it improves and what you checked for regressions.

More guidance is a design choice, not an automatic upgrade

A playbook can become harder to reason about as it grows. Before adding another instruction, ask whether it duplicates an existing one, conflicts with the current policy, or patches a symptom caused by an unclear tool interface. Some mistakes need better product design rather than another paragraph in a prompt.

In the fictional shop, three separate lessons might say to check pickup state, inspect the active loan, and confirm whether collection happened. A reviewer may decide these should be one clear state-check instruction. That consolidation is our example, not a result reported by IBM. The useful habit is to compare meaning rather than counting rules as if more always means wiser.

IBM's published comparison also shows that the amount of guidance can affect reliability differently across domains, and that longer guideline prompts add input cost. Treat those as reported tradeoffs to investigate in your own setting, not a universal setting to copy. The right small playbook depends on the task and the evidence you collect.

For a personal prototype, track the size of the active guidance and the time you spend maintaining it. When a new instruction solves one case but adds confusion elsewhere, record that tradeoff. A smaller, understandable set may be preferable for your project even if another domain benefits from a larger one.

Separate operational checks from customer-facing style. A rule about checking reservation status and a preference for a friendly tone solve different problems. Keeping them conceptually distinct makes it easier to diagnose a failure without rewriting the entire assistant personality.

The objective is not a perfect wall of instructions. It is a maintained set of behaviors whose purpose you can still explain.

Give old memories an exit

Now change the fictional shop's policy. Perhaps reservation changes are allowed until a new cutoff, or staff approval moves to a different stage. Which learned instructions rely on the old policy? If you cannot answer, the playbook is not ready to be trusted as a background habit.

Add a review trigger to each approved guideline when practical. It might be a policy revision, a tool change, a newly discovered failure, or a scheduled human review. Retiring a lesson should be possible without deleting the historical explanation of why it once existed. Keep the distinction between an active rule and an archived record clear.

Version the active playbook with the policy and tool definitions used in an evaluation. If a later test gets worse, you need to know whether memory changed, policy changed, or the surrounding application changed. Otherwise a remembered improvement becomes difficult to reproduce or reverse.

Plan how to remove an inappropriate lesson. An instruction might expose information it should not retain, encode a misleading shortcut, or be derived from an unrepresentative case. A removal process should identify where the active text is used, stop its use, and follow the project's applicable handling requirements for underlying records. Do not promise deletion from every system without checking the actual storage architecture.

Keep approval boundaries in the product itself. A learned reminder to request approval is helpful guidance, but important permissions should not depend only on whether the model remembers to ask. In the sandbox, you can make simulated record-changing actions require a separate explicit approval step outside the model's prose.

Memory should help an agent work within its authority. It should not gradually widen that authority because previous conversations sounded successful.

Build a playbook you can hand to another person

A good end to the exercise is a handoff. Give someone the fictional policy, the active guidelines, and a few saved evaluation cases. Can they tell why each instruction exists? Can they identify which policy change would invalidate it? Can they find the evidence for keeping it active without reading a long pile of conversations?

Ask them to remove one rule and predict what might break. Then ask them to invent a case that would expose an overly broad instruction. This turns memory from a vaguely impressive capability into something the team can reason about. The exercise is useful even if you never adopt a specialized memory library.

If you later try ALTK-Evolve, start with its current repository documentation and reproduce an appropriate documented example before connecting it to real operations. Keep our sandbox suggestions separate from the library's actual interfaces. We have not run the published experiment or verified its headline score independently.

The appealing part of the research is a different route to improvement: preserve lessons from experience rather than assume every problem requires changing the underlying model. The responsibility that follows is equally practical. Someone has to decide whether a lesson is sound, where it belongs, and when it should stop influencing the next conversation.

An editable rulebook is a better mental picture than an endless memory drawer. It has an author, a scope, a revision history, and a way to take a bad rule out of circulation. That gives a builder a concrete place to improve the system—and a concrete place to question it.