Most agent memory evaluations test retention alone: whether a preference stored today can still be retrieved twenty sessions later. This report examines the inverse failure mode across four production paradigms: (1) append-only similarity stores, (2) bi-temporal entity graphs, (3) asynchronous LLM consolidation pipelines, and (4) metadata soft-delete wrappers. Combining a 32-day field vault audit (10,134 entries), upstream open-source code inspections, and 12 standardized multi-session and 10M-token benchmarks (DolphinBench, Spatial Tracking, Personal Assistant, LoCoMo, LongMemEval, and ICLR 2026 BEAM), we formalize the three properties required for production-grade forgetting: synchronous state supersession, conversational noise & override hygiene, and zero-residue GDPR Art. 17 erasure.
1. The Week-4 wall: Why more memory makes agents dumber
If you have shipped a stateful coding assistant, support bot, or personal agent, you have probably lived through the same arc. In a Day-1 demo, plugging in a memory layer feels effortless: embed each conversation, fetch the top-5 chunks, and watch the agent recall the user's name and stack. By Week 4, strange bugs creep in. You migrated a service from us-east-1 to eu-central-1 two weeks ago, yet the agent keeps writing us-east-1 into configs. You tell it in chat to stop making a wrong assumption, and the next day that assumption comes back stronger. You run delete(user_id) for a privacy request, and the erased detail resurfaces on the user's very next message.
An append-only database is a log, not a memory. Random, unrelated noise rarely hurts an agent because it sits far away from your query in embedding space. Outdated truth is what kills accuracy: yesterday's deployment region shares 95% of the words and entities of today's region, making stale facts the #1 highest-scoring distraction in your index.
Once a vault cannot retire obsolete facts or filter out chatter, storing memories actively hurts performance. In a 32-day production audit of a live vector memory deployment (10,134 stored entries), 97.8% of stored vectors were duplicates, transient tool logs, or ungrounded inferences, and 89.6% remained noise even after switching extraction to Claude Sonnet 4.6 [1]. Just as Harvard D³ researchers found that indiscriminate memory performs worse than no memory at all [2], unfiltered recall on the 600-task DolphinBench suite pushed a commercial memory wrapper (59.2%) 6.5 percentage points below a stateless no-memory baseline (65.7%) [3].
When memory cannot forget, vault noise drowns signal and pushes accuracy below zero-memory baselines
Left: 32-day production audit of 10,134 stored entries in an append-only vector vault [1]. Right: 600-task DolphinBench accuracy vs total run cost relative to a stateless 65.7% no-memory baseline [3] [6].
Exact duplicates, system/tool chatter, and re-extracted hallucinations
Durable, non-redundant user & workspace facts (only 38 clean as-is)
At first glance, fixing this sounds like an afternoon ticket: "Can't we just wire up a DELETE endpoint?" What catches agent builders off guard is that "forgetting" is really three distinct jobs, and the exact tricks modern stacks use to make remembering easy (embeddings, background LLM reflection, and prose summaries) break all three:
Once you separate those three jobs, dozens of mysterious production bugs suddenly trace back to three clean computer-science traps.
2. Trap I: To an embedding model, a contradiction looks like a twin
Suppose on Monday you tell your coding agent: "Deploy PII tables to us-east-1." On Friday, compliance steps in and you tell it: "Never deploy PII tables to us-east-1; we migrated to eu-central-1." Next Monday, a teammate asks the agent where PII tables belong.
To a human, "deploy to
us-east-1" and "never deploy to us-east-1" are opposites. To a vector index or keyword search, they are practically identical (>0.92 cosine similarity). Similarity search measures shared topic, not current truth, so asking for the current rule guarantees pulling up the exact rule you just repealed.
Every popular memory paradigm tries to work around this geometry gap, and each runs into a wall of its own:
Why vector wrappers quietly dropped UPDATE and DELETE
Early memory SDKs tried asking an LLM on every write whether to ADD, UPDATE, or DELETE existing facts. In production, firing extra LLM calls on every turn added seconds of latency and doubled token bills. To stay fast, modern vector pipelines commonly fall back to single-pass ADD-only extraction and rank results by similarity plus keyword overlap, with no sense of what superseded what. Both Monday's rule and Friday's rule land in the top-5 prompt results with no timestamps, leaving your agent to flip a coin.
Why temporal LLM graphs turn into hairballs at scale
Bi-temporal knowledge graphs try to solve updates by keeping every old edge and asking an asynchronous LLM judge to stamp invalid_at on contradicted ones. Two things go wrong in practice. First, LLM judges misfire: casually mentioning an entity in a new conversation can trick the judge into expiring unrelated facts that are still true. Second, because expired edges stay attached to the graph, central nodes like User or prod-db accumulate thousands of historical edges. By the time history reaches 10M tokens on BEAM [5], accuracy drops to 42.1% to 48.6% after 16 to 40+ hours of LLM ingestion.
Why splitting "core" RAM from "archival" disk hits a cliff
Some agent frameworks keep a tiny editable "core memory" block inside the system prompt and push everything else into an out-of-context archival vector store. That works nicely until state outgrows the small core window. Once facts spill into archival storage without automatic retirement, stale and current passages pile up side by side, while sliding-window compaction squashes subtle engineering rules into vague one-line summaries.
Unretired historical edges & append-only vectors collapse accuracy across 380x context growth
Left: Multi-session accuracy from 26K tokens (LoCoMo [4]) to 1M and 10M tokens (BEAM, ICLR 2026 [5]). Right: Mutable state accuracy before vs after synchronous write-time retirement [6].
3. Trap II: Arguing with a false memory feeds the bug
Suppose a background "dreaming" or reflection worker hallucinates a trait about you (in the 32-day field audit [1], an extractor invented "User prefers Vim" once out of thin air). Next session, that trait gets injected into the system prompt. Annoyed, you type in chat: "Stop saying I use Vim! Delete that!" You even delete the source session. The next morning, the false trait is back, and in the field audit that single hallucination snowballed into 808 stored copies [1].
To a background LLM reflection worker, arguing with a bad memory looks identical to caring deeply about that topic. Language models treat repetition as importance. When a background worker reads your chat transcript without knowing you were overriding a bad rule, your complaint becomes fresh topical fuel to re-derive the exact memory you just deleted.
Why detached background reflection creates immortal zombie traits, vs synchronous hygiene in / ml
Left: Positive-feedback re-derivation cycle in asynchronous "dreaming" pipelines. Right: Synchronous write filtering and immediate retraction in / ml.
Self-Amplifying Hallucinations
Once a hallucinated fact lands in recall context, the next extraction pass reads its own recalled snippet as ground truth and stores it again, multiplying a single bad guess into hundreds of copies.
Every "Thanks" Saved Forever
Without write-time filtering, polite replies ("got it, thanks"), cron heartbeats, and raw stack traces become permanent memories that crowd out real architectural rules.
Blast-Radius Deletes
Tools that implement forget("deploy rule") by wiping everything above 0.85 cosine similarity act like a shotgun: they delete nearby valid rules that happen to share the same nouns.
Guessed Expiration Dates
Asking an LLM prompt to guess a TTL timestamp (forgetAfter) breaks whenever you backfill older chat logs whose relative dates ("by Friday") have already passed.
4. Trap III: You cannot unbake an LLM summary
Now imagine a customer files a GDPR Article 17 ("Right to be Forgotten") request [7], or an engineer pastes a production secret in Session 4 and asks to erase that single turn without wiping ten weeks of project context.
Summarizing ten conversations into an LLM-written paragraph is like stirring milk into coffee: it only goes one way. Once Session 4 has been blended into a prose entity summary, a user bio card, or a "consolidated observation" alongside Sessions 1 through 10, there is no
summary - session_4 button. Deleting the raw message leaves its facts baked into the summary.
Worse, most agent memory stacks glue together three or four separate databases (a vector store, a graph DB, a SQL history table, and a raw message queue). Calling delete() hits the front door while leaving copies alive in the back rooms:
What physically happens on disk when an application calls delete()
Tracing primary records, shadow audit/buffer tables, and LLM-derived artifacts across multi-store memory paradigms versus the single-appliance / ml runtime.
prev_value) into SQL history.
delete_all() skips scoped raw message buffer on disk.
405.
isForgotten: true) without purge.
gdpr_unlearned: true + audit_gdpr_* (0% residual).
Deleted from vectors, saved in plaintext SQL audit logs (and resurrected on the next chat)
Many vector memory SDKs pair an external vector DB with a local SQLite or Postgres metadata store. When you call delete(memory_id), the SDK deletes the vector and immediately writes a DELETE audit row containing a permanent plaintext copy of the erased secret (prev_value) into SQLite. Even worse, calling delete_all(user_id) often wipes the vectors while leaving recent raw messages sitting in a staging table on disk. The moment that user sends one new message, those "deleted" chats get bundled into the next LLM extraction call and come right back from the dead.
Sequential LLM summary rebuilds (and silent PII rollback on timeout)
Because LLM knowledge graphs bake dozens of conversations into a single prose paragraph on each entity node, deleting one conversation forces the system to fetch every remaining episode and ask an LLM to rewrite every affected summary from scratch. If any outbound LLM call hits a rate limit or timeout halfway through, standard try/except handlers restore the pre-delete summary so the graph stays readable, quietly leaving erased PII live in production while reporting success. Higher-level community summaries are not rebuilt at all.
Immortal user records & orphaned profile cards
User-modeling engines synthesize layered personality traits and biographies from raw chats. Once those deductions are generated, they lose any clean 1:1 link back to individual messages. As a result, APIs frequently omit single-message delete endpoints, block actor deletion with 405 Method Not Allowed, or queue session deletion in the background (202 Accepted) while already-generated profile cards keep getting injected.
Expensive re-consolidation storms vs boolean soft-delete mirages
Engines that build "mental models" on top of raw facts have to invalidate derived observations and kick off multiple background LLM, graph, and vector rebuild jobs just to forget one item, turning deletion itself into an LLM bill. At the other extreme, hosted wrapper APIs implement forget() by flipping a boolean flag (isForgotten: true) on the profile entry while leaving the underlying source documents and proxy chat logs sitting untouched on disk.
5. Side-by-side: Why each memory architecture struggles to forget
Stepping back, none of these failures are careless coding mistakes. They are the natural consequence of splitting raw writes, embeddings, and LLM-written summaries across separate databases and background queues. Table 1 puts all four common architectures side by side against / ml.
Table 1: Deletion, State Supersession & Unlearning Properties by Architectural Paradigm
Comparing consistency, queue latency, derived-state hygiene, and GDPR erasure guarantees across memory architectures.
| Forgetting Dimension | Append-Only Vector RAG | Bi-Temporal LLM Graphs | Async Reflection ("Dreaming") | Soft-Delete Wrappers | / ml Runtime Appliance |
|---|---|---|---|---|---|
| Mutable State Supersession | ADD-only (accumulates conflicts) | Async LLM judge (hub bloat @ 10M) | Eventual queue (handoff races) | Async extraction lag | Synchronous write-time (<10 ms) |
| Physical Delete Path | Vector delete + SQL audit plaintext | Sequential LLM summary rebuilds | 202 async queue / multi-job rebuild | Boolean flag (isForgotten) |
6.41 ms synchronous physical purge |
| Derived Summary / Graph Ghosts | Buffered chats re-sent on next write | Summary rollback risk; communities stale | Immutable peers; dreams outlive source | Source docs & router chats persist | 0% residual across recall & exports |
| Noisy / Bad Rule Recovery | Manual delete only (97.8% noise risk) | Unfiltered ingress chatter | Positive-feedback re-derivation | Cosine-threshold delete collateral | 0.0% chatter bloat · instant retraction |
| GDPR Erasure Verification | None (polyglot store drift) | None (best-effort LLM catch) | Eventual consistency window | Soft-hide only (no physical purge) | Synchronous audit_gdpr_* receipt |
6. What happens when forgetting works (/ ml)
You cannot bolt real forgetting onto an append-only store with a nightly cleanup script or an isForgotten flag; the runtime has to treat forgetting as a first-class operation from day one. We built / ml as a single, self-contained synchronous runtime appliance with zero queue lag and zero external vector or graph clusters to manage. From an agent builder's perspective, plugging in / ml changes four things immediately:
Facts update before the next turn starts (<10 ms sync)
When a user changes their deployment region, updates a schema, or moves an object, / ml retires the outdated state synchronously in under 10 ms with zero queue lag. The very next turn or subagent handoff sees only current truth, lifting cross-turn updates on DolphinBench Stage S3 from 12.5% to 100.0%, 300-step Spatial Tracking from 44.0% to 100.0%, and 10M-token BEAM accuracy to 84.3% (+35.7 to +42.2 points ahead of vector and graph baselines).
Chatter stays out, and retracted rules stay gone
Small talk and noisy tool logs are dropped at ingress (0.0% chit-chat vault pollution across 2,800 turns). When a developer retracts or updates a prior constraint during a session, / ml retires it synchronously with no background reflection queue to re-derive a zombie rule, holding compliant false alarms at 0.0% across six enterprise verticals.
Deletes actually delete in 6.41 ms (with a receipt to prove it)
One call to POST /v1/memory/purge (or drop-in DELETE /v1/memories/ by document, user, or entity) physically purges the target state across every worker in 6.41 ms P50. There are zero LLM summary rebuilds to time out, zero plaintext SQL audit leftovers, and zero buffered chat turns waiting to resurrect erased data, and every call returns gdpr_unlearned: true with an immutable audit_gdpr_* verification receipt.
One command to wire in, zero extra databases to babysit
Whether you attach lifecycle hooks to your coding agent via npx -y @slashml/memory@latest init, call the REST API, or deploy the single binary inside your own VPC, state updates, recall, and surgical erasure happen inside one cohesive boundary rather than four drifting data stores.
Synchronous sub-10 ms erasure & retirement vs multi-second LLM rebuild queues
Logarithmic comparison (1 ms to 10 min+ / Unbounded) of time until deleted or superseded state stops surfacing in downstream agent recall.
Table 2: Empirical Evaluation Across State Supersession, Noise Hygiene & Surgical Erasure
Standardized benchmark results comparing append-only baselines and published external engines against synchronous / ml lifecycle management.
| Evaluation Benchmark | Forgetting Property Tested | Baseline / External Benchmarks | / ml (SlashML) |
|---|---|---|---|
| DolphinBench Lifecycle S3 [6] | Cross-turn state supersession | 12.5% (append-only tools) | 100.0% (+87.5 pp) |
| Embodied Spatial Tracking [6] | 300 sequential physical location mutations | 44.0% (stale location recall) | 100.0% (0.0% stale reads) |
| 2,800-Turn Personal Assistant [6] | Chit-chat noise filtering & preference updates | 52.3% (tools-only) | 96.5% (0.0% chit-chat bloat) |
| DolphinBench Full (600 Tasks) [3] | Multi-session distractor & noise immunity | 59.2% Supermemory · 68.5% Honcho · 69.5% Hindsight · 70.7% Mem0 | 90.5% ($45.90 total run cost) |
| BEAM @ 10M Tokens (ICLR 2026) [5] | Contradiction resolution & hub saturation | 42.1% Zep · 48.6% Mem0 | 84.3% (41.7 ms P50 recall) |
| Enterprise Multi-Worker GDPR Audit [7] | 180 parallel RYOW checks + export verification | Async queue lag / audit plaintext | 6.41 ms P50 · 100% instant delete visibility · 0% residual |
Test selective forgetting or reproduce the benchmarks
Attach synchronous 4-stage lifecycle hooks to your coding harness in one command, verify 6.41 ms surgical GDPR unlearning directly against https://api.slashml.ai, or run the open-source evaluation harness:
# 1. Attach ambient lifecycle hooks + MCP tools to Claude Code, Cursor, Codex, or Windsurf
npx -y @slashml/memory@latest init
# 2. Execute synchronous 6.41 ms surgical GDPR erasure with verification receipt
curl -sS -X POST "https://api.slashml.ai/v1/memory/purge" \
-H "Authorization: Bearer $SLASHML_API_KEY" \
-H "Content-Type: application/json" \
-d '{"document_ids": ["doc-user-pii-record"]}'
# 3. Reproduce supersession & erasure suites via the open-source eval CLI
pip install slashml-eval && slashml-eval --benchmark all --json
References & citation
-
@jamebobob(2026). "What we found after auditing 10,134 mem0 entries: 97.8% were junk." 32-day vector-store production audit dataset (Feb 23 to Mar 26, 2026; 97.8% noise overall across local and frontier LLM extractors; 89.6% noise on the Claude Sonnet 4.6 cohort). https://github.com/mem0ai/mem0/issues/4573 - Harvard Digital Data Design Institute (D³) (2025). "Smarter Memories, Stronger Agents: How Selective Recall Boosts LLM Performance." Harvard Business School D³ Institute. https://d3.harvard.edu/smarter-memories-stronger-agents-how-selective-recall-boosts-llm-performance/
- Rathi, S., Yadav, D., & Singh, T. (2026). "DolphinBench: Mapping the Pareto Frontier of Agent Memory." Official 600-task action-execution evaluation harness and leaderboard. https://dolphinbench.ai
- Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024). "Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo)." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). https://arxiv.org/abs/2402.17753
-
Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., & Mitchell, J. R. (2026). "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs (BEAM)." International Conference on Learning Representations (ICLR 2026),
arXiv:2510.27246. https://arxiv.org/abs/2510.27246 -
SlashML Research (2026). "Why Agents Forget to Use Their Own Memory Tools: A 12-Benchmark Study of Ambient Runtime Memory."
/ mlTechnical Report. https://slashml.ai/research/ambient-memory.html - European Parliament and Council (2016). Regulation (EU) 2016/679 (General Data Protection Regulation), Article 17: Right to erasure ('right to be forgotten'). Official Journal of the European Union. https://gdpr-info.eu/art-17-gdpr/
@techreport{slashml2026forgetting,
title = {When Remembering Everything Breaks AI Agents:
Why Selective Erasure Is Fundamental, and Hard},
author = {SlashML Research},
institution = {/ ml (SlashML)},
year = {2026},
month = {November},
url = {https://slashml.ai/research/forgetting.html}
}