Evals for Agent Memory
You can’t improve what you never score. Memory is the one subsystem of a production agent that teams ship without a test suite, and an unevaluated memory is a confident liability that grows every day the agent runs.
This is not another long-context benchmark, not a leaderboard, and not a survey of memory architectures. Plenty of good work already describes how agents form, store, and retrieve memory. This is about the thing nobody scores: whether the memory an agent has accumulated is any good. Is the retrieval surfacing the right memory, is a stored fact still true, is the write policy saving signal or noise, and is any of it being measured at all?
📄 The full cited PDF report is free to download at the bottom of this page, no signup.
What this is, and is not
We are not asking whether a model can find a needle in a 200K-token haystack; that is a property of the context window, not of memory. This is about the operator’s question: not “does the system persist data” (it does) and not “can it recall a fact when asked” (often), but whether the store an agent has been writing to for three months is healthy at all.
The claim is narrow and uncomfortable. We have evals for model knowledge, evals for tool use, evals for retrieval over documents, and a growing discipline of evals for end-to-end agent tasks. Memory sits in the middle of all of it, ungraded. And an unevaluated memory is not a neutral. It is a confident liability that grows every day the agent runs.
Persistence is not truth, and untested is not working. The staleness paper named the failure. This paper is how you measure for it.
The one-linerThe thesis, in moves
- We obsessively score what a model knows. MMLU, GPQA, the whole leaderboard economy. Knowledge is the most-benchmarked quantity in the field.
- We barely score what an agent’s memory does. Academic memory benchmarks exist (LongMemEval, LOCOMO), but they measure recall on a fixed transcript. They do not measure the operational health of a memory store that an agent has been writing to for three months.
- Memory has distinct, compounding failure modes. Wrong retrieval, confidently-served staleness, noisy writes, runaway bloat, and failure to forget are five different bugs with five different fixes. A single “memory works / doesn’t” verdict hides all of them.
- A benchmark is not a harness. A score on someone else’s frozen dataset tells you nothing about your store. What an operator needs is a set of mechanical checks they can run against their own memory, on their own data, on a schedule.
- The verifier that scores memory is the same artifact that improves it. This is the move that makes the whole thing worth building. An eval, made mechanical, is a reward signal. Score the memory, and the score is the gradient: gate the stale facts, down-rank the bad retrievals, reward the write policy that kept the signal. You do not eval then improve. The eval is the improvement loop.
The body of work is the same operator standing in front of the same agent: paper #3 named the failure, paper #4 is how you measure for it. You will never catcha false fact, or any of memory’s other failures, until you decide to score memory as a first-class system.
Why memory needs its own evals
Start with the gap, because it is conspicuous. Scan what the people building agents are actually publishing right now and memory is almost absent from the two conversations that matter most. The runtime conversation is about moving the agent loop server-side into a sandboxed computer (Google’s managed-agents API, the agent-native clouds). The eval conversation is about verifiers as the bottleneck for both evaluation and reinforcement learning. Memory appears in neither as a thing you evaluate. It shows up as a feature you add.
That is strange, because retrieval is not knowledge and memory quality, not model IQ, is what gates a long-running agent. A frontier model with a polluted memory will confidently act on the wrong recalled fact. A weaker model with a clean, well-scored memory will not. As agents run for weeks and months rather than minutes, the reliability curve bends on the memory, and the memory is the part with no instrument on it.
Consider the asymmetry. If a model regresses on a knowledge benchmark, someone notices within a release cycle, because the benchmark runs in CI. If an agent’s memory store silently fills with duplicates, drifts stale, and starts retrieving the wrong note for the wrong query, nobody notices until a user is told something wrong with total confidence. There is no red number on a dashboard, because there is no dashboard, because there is no eval.
When this matters, and when it does not
A fair objection: not every app needs a memory test suite, and pretending otherwise is how good ideas become cargo cults. So be precise about who this is for.
Memory evals earn their keep in proportion to two things: how long the agent runs and how much it writes. A stateless chatbot that retrieves nothing across turns has no memory to score; skip all of this. A retrieval-augmented app over a curated, human-maintained document set is doing document-RAG, and RAG evaluation already covers it. The case for memory evals is specifically the self-writing, long-running agent: the one that accumulates its own store over weeks, decides for itself what to persist, and acts on what it recalls without a human in the loop. That is the configuration where every failure mode in this paper compounds, because the agent is both the author and the consumer of a store that nobody is grading.
If your agent writes to its own memory and you cannot answer “how good is that memory right now” with a number, you are in scope, and the gap between your confidence in the agent and your evidence about its memory is exactly the size of the risk.
The six dimensions
The contribution of this paper is a taxonomy: the dimensions an operator should score memory on, the failure each one catches, and the mechanical metric that catches it. Six dimensions, because memory fails in at least six distinguishable ways and a single number hides them.
| Dimension | The failure it catches | The mechanical metric |
|---|---|---|
| 1. Retrieval correctness | The system surfaces the wrong or irrelevant memory for the query. “Relevant” means useful to the task, not lexically similar. | precision / recall@k |
| 2. Truth-maintenance | The system serves an outdated fact at full confidence. The entire subject of paper #3, recast as a number. | % stale-but-served |
| 3. Write-policy precision | The system saves noise: chatter, duplicates, things that should never have persisted. Long-run quality is set at write time. | write precision |
| 4. Consolidation | The system duplicates or contradicts itself instead of merging and updating. Append-on-write breeds three versions of one fact. | dup / contradiction rate |
| 5. Context efficiency | The recalled memory pollutes the context more than it helps. Every recalled token is budget and attention. | task-score delta |
| 6. Forgetting & hygiene | The system keeps what it must drop: PII past its purpose, facts the user asked to forget, dead state. A required behavior with legal teeth. | % deletions honored |
1. Retrieval correctness.Precision and recall@k over a labeled set of query-to-expected-memory pairs. The most familiar dimension because it borrows directly from information retrieval, and the one most teams could stand up this week. The subtlety is that “relevant” for derived agent memory is about usefulness to the task, not string overlap.
2. Truth-maintenance (staleness).Percent stale-but-served on a time-shifted probe set, where you know a fact’s valid-time and check whether the store still returns the superseded version. Benchmarks like TimeQA were built to expose exactly this temporal misalignment in models; the operator move is to run the same shape of probe against the memory store, not the model.
3. Write-policy precision.Precision of writes against a “should-persist” gold set drawn from real session logs. The dimension nobody measures and everybody needs, because a memory’s long-run quality is set at write time. Garbage in is garbage retrieved, forever, at confidence.
4. Consolidation.Duplicate rate and internal-contradiction rate across the store. Mem0 reports deleting on contradiction and Zep’s bi-temporal model invalidates on a contradicting message. Both are consolidation policies, and both deserve a score, because a policy you do not measure is a policy you cannot tune.
5. Context efficiency. The task-score delta with versus without the recalled block. Signal minus bloat, measured. This is the dimension that connects memory to the cost line: if a recalled block does not move the task score, it is pure dilution, and large stores tend toward dilution by default.
6. Forgetting and hygiene.Percent of required deletions actually honored. Forgetting is usually framed as a capability loss to avoid. For an operator’s memory store it is the opposite: a required behavior with legal and trust consequences, and therefore something to verify rather than fear.
The dimensions are not independent. They compound. A loose write policy (3) admits a near-duplicate, which becomes a consolidation failure (4), which means retrieval now has two versions to choose between and sometimes returns the wrong one (1), and if the wrong one is older, that is a staleness hit too (2). One root failure at write time radiates into three retrieval-time symptoms. Score only the symptom and you keep treating retrieval while the disease is upstream.
These are not exotic. Each is a check you could write in an afternoon against a few hundred labeled examples. The reason they do not exist in most stacks is not difficulty. It is that nobody decided memory was a thing you grade.
The reference-dataset problem
The honest obstacle is the gold set. You cannot evaluate retrieval correctness without labeled query-to-memory pairs, you cannot score staleness without valid-time labels, and you cannot score write policy without a should-persist judgment. Memory evals need a labeled history, and most teams do not have one sitting around.
The good news is that an operator is generating the raw material constantly: the session logs. The construction is cheap and mostly mechanical. For retrieval, log which memory the agent actually used and whether the turn succeeded; that is weak-but-real relevance signal you can harden with a small hand-labeled core. For staleness, you already have timestamps; a fact written on one date and contradicted later gives you a valid-time boundary for free. For write policy, sample what got persisted and label a few hundred as should-persist or noise; that is a Saturday, not a research program. For forgetting, every deletion request is a labeled positive.
The reference dataset is a by-product of running the agent, not a separate data-collection project. The barrier was never the labels. It was the decision to keep them.
The pointOne eval, end to end
Abstraction is cheap, so here is the cheapest high-yield eval built start to finish: retrieval correctness, measured as recall@k. The goal is a single number you can watch move and a lever attached to it.
Pull a few hundred real turns where the agent retrieved memory before answering. Record the query it issued and which items it pulled. Add the cheap label: did the turn succeed? Hand-correct a core of fifty to be unambiguous gold; let the rest ride as weak labels. An afternoon, reusable forever.
For each gold query, ask the store for its top-k items and check whether the expected item is in the set. Recall@k is the fraction of queries where it was. Run it at k of one, three, and ten; the spread tells you whether the right memory is found at all (recall@10) versus ranked to the top where it gets used (recall@1).
Low recall@10 means the right memory is not surfaced: re-embed with a better model or widen the candidate set. Healthy recall@10 with poor recall@1 means retrieval buries it: add a re-ranker. Either way the number told you which knob, and after you turn it you re-run the same hundred queries and watch it move.
A store that scores well at ten and badly at one has a ranking problem, not a coverage problem, and that distinction is the difference between re-embedding and re-ranking. The eval is not a verdict; it is an instrument with a control wired to it.
Notice what this did not require: no frozen academic benchmark, no labeled corpus from a vendor, no research budget. It required your own logs and the decision to score them. Every one of the other five dimensions is the same shape with a different gold label.
Eval as reward
Here is why you build the harness rather than run a benchmark once and forget it. The verifier you write to score a memory dimension is the same artifact that improves it.
This is not a Kymata Labs idea; it is the strongest current in the eval conversation. LangChain’s work on efficient verifiers for legal agents makes the case directly: verifiers are the cost bottleneck for both evaluation and RL post-training, and the same verifier that grades an output can serve as the reward model that trains the policy. Batch them on open or cheap models and the cost of running them continuously drops far enough that “score it on every change” becomes affordable rather than aspirational. Eugene Yan’s eval-rigor work points the same direction: the discipline of a mechanical, reproducible score is what lets you optimize against it at all.
Apply it to memory and each of the six dimensions becomes a control loop. Retrieval recall@k is low: the score is the signal to re-rank the retriever. Stale-served percent is high: gate or down-weight facts past their half-life and watch the number fall. Write precision is low: tighten the write gate and re-measure. The eval is not a report card you read once. It is the reward that closes the loop.
Agent OS’s build factory generates code and UI, then grades each attempt with adversarial verifiers (a model-family-diverse reviewer, a real-browser design measurement, a test runner) and feeds the critique back as the next attempt’s input, looping until the verifiers pass. The verifier is the reward; the score is the gradient. We did not build that for memory, but it is the identical shape, and it is the shape this paper argues you should point at your memory store. (Implementation receipt, not a benchmark claim.)
The cost angle matters because memory evals, to be useful, have to run continuously rather than once. A memory store changes every session. A score from last month is itself stale. The only eval that catches a degrading store is one cheap enough to run on a schedule, which is exactly why the “batch verifiers on small models” result is load-bearing here and not a footnote.
Think about the cadence concretely. A knowledge benchmark can run quarterly because the model only changes on release. A memory store changes every time the agent acts, which means its quality has a much shorter half-life than any model’s, and an eval cadence to match. The practical target is to run the cheap dimensions (retrieval and staleness) on every meaningful batch of writes, the way a test suite runs on every commit, and the expensive ones (context-efficiency against live tasks) nightly. That cadence is only affordable if each check is a small, batched verifier rather than a frontier-model call per item, and it only produces improvement rather than just anxiety if each run’s number is wired to a knob. The discipline is the same one that made continuous integration normal for code: a fast, cheap, automatic check on every change, with a red light that stops the line. Memory has had no such line.
Why the existing benchmarks do not close it
LongMemEval and LOCOMO are real and useful, and this paper is not a swipe at them. LongMemEval explicitly includes a knowledge-updates ability and reports large accuracy drops on long-horizon memory; LOCOMO is the standard long-conversation memory benchmark that the systems papers report against. Credit where due: they established that memory degrades measurably over long horizons.
But they are benchmarks, not harnesses, and the difference is the whole point.
What benchmarks are
What a harness is
The prior-art on the problemside is similar. STALE (May 2026) argues that current benchmarks primarily measure static fact retrieval and overlook the ability to revise stored beliefs. We cite that framing head-on, the same way the staleness paper did, and extend past it: naming that benchmarks miss belief-revision is the academic flag; giving the operator a continuous, reward-wired harness across all six dimensions is the operator’s contribution. (STALE existence verified; its specific results postdate our verification cutoff and are not asserted here as fact.)
From the operator’s chair
Everything above is sharper when it is something we actually ran into, not a thought experiment. A few receipts from running Agent OS, an autonomous operator with a persistent memory.
Context efficiency, the hard way.The session-memory index file grew past the point where the harness itself warned it was bloated and only partially loading. That is dimension five with a red light on it: a memory store whose recalled block had started to cost more context than it returned in value. The fix is not “store less”; it is to measure the yield of recalled memory and prune what does not earn its tokens.
Truth-maintenance, built into the protocol. The recall protocol carries an explicit instruction: a recalled memory reflects what was true when written, so verify a named file or fact still exists before acting on it. That is a hand-coded version of the staleness dimension, and the fact that it had to be hand-coded is the argument for scoring it: the discipline only holds if a number holds it accountable. The facts store carries a last-verified column for the same reason; a column is a place to put a freshness score, which is most of the way to being one.
A stored preference said this author should avoid em-dashes. The actual published papers used them heavily. The memory was confidently wrong, and it was caught not by any eval but by a human happening to look.
The most honest receipt is a missA write-policy failure, caught only by looking. That miss is dimension three (a bad write that persisted) and dimension two (a fact that went false) failing together, silently, on the author, mid-project. It is the cleanest possible argument for the harness: persistence kept the claim, and only an eyeball, not a system, caught that persistence was not truth.
Eval as reward, shipped.And the receipt that points forward: the build factory’s verifier-as-reward loop, shipped this week, is the exact mechanism this paper says to aim at memory. The pattern is proven in our own stack on code and UI. Pointing it at the memory store is the next build, not a hypothesis.
What to score Monday
Concrete controls, ordered by yield-per-effort. None of these requires a research team. Each is mechanical, and each is wired to an action, because a score with no lever attached is just anxiety.
- Retrieval recall@k on a small labeled set. Hand-label a few hundred query-to-expected-memory pairs from real logs. Measure recall@k. The lever: re-rank or re-embed when it drops. Highest yield, lowest effort; start here.
- Stale-served percent on a time-shifted probe. Use your own timestamps to build superseded-fact probes. Measure how often the store returns the old version. The lever: gate facts past a fact-class half-life (the staleness paper’s decay-rate table is the gating policy).
- Write precision on a should-persist sample. Label a sample of writes as signal or noise. Measure precision. The lever: tighten the write gate, then re-measure. This is the dimension that pays compounding dividends, because it fixes quality at the source.
- Duplicate and contradiction rate across the store. A nightly scan. The lever: merge-on-write instead of append-on-write.
- Context-efficiency delta. For a sample of tasks, run with and without the recalled block and diff the task score. The lever: prune memory that does not move the number.
- Required-deletion honored percent. Every forget request is a test case. The lever: fix the forget path until the number is 100, because this one has legal teeth.
Run the cheap two (1 and 2) first, on a schedule, and wire each to its lever so the score becomes a gradient rather than a report. That is the whole method: measure the dimension, attach the reward, close the loop.
It is time to score what memory does
Staleness is one failure mode of memory. This paper is about the instrument that catches staleness and the five other failures next to it. The quiet reliability cliff ahead is not that agents will forget; it is that they will remember, confidently, things that are wrong, irrelevant, duplicated, or bloated, and that no number anywhere will go red when they do.
The fix is not a better memory architecture. It is the decision to treat memory as a first-class, evaluated system, with a harness an operator runs on their own store, on a schedule, with each eval wired to the reward that improves the thing it measures. We benchmark what models know. It is time to score what their memory does, because untested is not working, and an unevaluated memory is just a confident liability waiting for a user to trust it.
What people ask about agent memory evals
No. RAG eval scores retrieval over a static document corpus with ground-truth answers. Memory eval scores a store the agent wrote itself, where the gold labels are about derived usefulness, freshness, and write quality, and where there is no document oracle to check against.
No. A bigger window changes what fits, not whether what is recalled is correct, fresh, or worth its tokens. Context pollution is a property of the content, not the window size.
No. Retrieval recall@k and stale-served percent are the cheap, high-yield pair. Start there, wire each to a lever, add the others as the store grows.
Because a memory store changes every session; a one-time score goes stale itself. Only a continuous, reward-wired eval catches a degrading store, and recent verifier-cost work makes continuous scoring affordable.
Sourced, and written from the chair
Every source below was retrieved and read during research; claims in the text trace to these. Where a result rests on simulation or postdates verification, it is noted. The paper is written from the standpoint of a team that runs a long-lived, memory-persisting multi-agent system in production, and the harness it argues for is the one that system is building next.
Wu et al. — LongMemEval: long-term memory benchmark with a knowledge-updates ability; large long-horizon accuracy drops. arXiv:2410.10813.Maharana et al. — LOCOMO: long-conversation memory benchmark; standard reporting target for memory systems. arXiv:2402.17753.Chhikara et al. — Mem0: agent memory system; LOCOMO results; delete-on-contradiction consolidation. arXiv:2504.19413.Rasmussen et al. — Zep / Graphiti: bi-temporal memory (valid-time and transaction-time); contradiction-driven invalidation. arXiv:2501.13956.Chen, Wang, Wang — TimeQA: time-sensitive QA exposing temporal misalignment; design template for time-shifted probes. arXiv:2108.06314.Chao et al. — STALE: argues benchmarks measure static retrieval and overlook belief revision (May 2026; existence verified, results not asserted). arXiv:2605.06527.Kirkpatrick et al. — Catastrophic forgetting / EWC: forgetting as capability loss in training (contrast with required forgetting). arXiv:1612.00796.LangChain — Designing Efficient Verifiers for Legal Agents (June 2026): verifiers as the eval-and-RL-reward bottleneck; batch on open models to cut cost; verifier doubles as reward model.Eugene Yan — Eval-rigor essays: mechanical reproducible scoring as the precondition for optimization. eugeneyan.com.Kymata Labs — The Agent Memory Staleness Problem (paper #3); roster signal scan (June 2026); Agent OS build-factory verifier-as-reward implementation (2026-06-13).Get the complete white paper, free
The full PDF: the five-move thesis, the six-dimension taxonomy with metrics, the reference-dataset construction, the end-to-end recall@k worked example, the eval-as-reward loop with the build-factory receipt, why the benchmarks do not close it, and the six controls to score Monday. No signup, no email wall.
Download the PDFKymata Labs Research · Public · v1.0 · Distributed for public use