Why agents forget to use their own memory tools

1. Introduction & problem formulation

Production agent architectures generally integrate persistent memory through one of two mechanisms: exposing explicit retrieval and write primitives (search_memories, add_memory) to the language model via MCP or function calling, or wrapping each conversational turn in an asynchronous background extraction and top-k prompt prepending pipeline (auto-memorize and auto-recall).

Both approaches degrade significantly under sustained task load. Consider an interactive software engineering session where a user establishes an invariant on turn 3: "Do not run raw SQL migrations on billing-service; always generate an Alembic revision first." Forty turns later, after inspecting several stack traces and compacting the context window, the user instructs the agent to ship an invoice schema update. Although the constraint remains indexed in the backing store, the model never invokes search_memories, because the prompt "Ship the invoice schema update" appears self-contained and exhibits no lexical cue indicating that a historical rule applies.

This failure mode generalizes across autonomous multi-session workflows whenever operational invariants are separated in time from downstream execution:

• Autonomous infrastructure operations: During an autoscaling event, an incident-response agent does not proactively query whether the affected tenant was pinned to Frankfurt (eu-central-1) during a compliance audit six weeks prior.
• Supply-chain & procurement workflows: When generating a restocking purchase order, a procurement agent does not know to query whether quality assurance placed the vendor on a defect hold in an earlier session.
• Enterprise revenue operations: When drafting a multi-year renewal schedule, a quoting agent does not spontaneously search Q1 transcript archives for a custom rate-lock addendum.

Rather than treating memory as an optional tool call or an asynchronous prompt wrapper, we model agent state after operating system virtual memory, an architecture we term ambient memory. Just as an operating system pages memory into and out of physical RAM transparently underneath an executing process, an ambient memory runtime manages context injection, mid-turn scope transitions, pre-execution policy verification, and synchronous state supersession directly inside the agent execution loop.

To isolate the contribution of each architectural component, we conducted a controlled ablation across 12 evaluation suites (600 end-to-end DolphinBench tasks, 500 multi-project workspaces, 6 enterprise compliance verticals, embodied spatial tracking, and four long-horizon memory benchmarks: LoCoMo, LongMemEval, LongMemEval-v2, and BEAM up to 10M tokens), evaluating three configurations under identical model weights and evaluation harnesses:

• tools_only: Explicit search_memories, add_memory, and forget_memory tools exposed to the model, with all runtime lifecycle hooks disabled.
• ambient_only: Zero memory tools exposed to the model. State assimilation, supersession, context delivery, and pre-execution gating occur exclusively within the four runtime lifecycle hooks.
• ambient_plus_tools: Runtime lifecycle hooks manage synchronous state updates, pre-tool verification, and Turn-1 grounding automatically, while explicit search tools remain available for targeted multi-hop retrieval.

2. Empirical failure analysis of tool-based memory (tools-only)

On explicit retrospective queries (e.g., "What port was configured for staging Redis?"), tools_only performs reliably because the user prompt explicitly instructs the model to retrieve historical state. Degradation occurs when state maintenance and recall must compete with active task execution across four failure modes:

3. Systems limitations of naive auto-memorize & auto-recall wrappers

To bypass voluntary tool invocation, many frameworks wrap the chat completion endpoint in an asynchronous background extraction queue (auto-memorize) paired with unconditional top-k embedding injection at prompt ingress (auto-recall). Our systems profiling identifies three structural bottlenecks in this design:

4. Experiment 1: Multi-turn workflows, isolation & policy enforcement

We first evaluated tools_only, ambient_only (zero memory tools exposed to the model), and ambient_plus_tools across eight multi-turn operational, coding, spatial state, and compliance benchmark suites:

5. Experiment 2: Long-horizon multi-session synthesis (up to 10M tokens)

To evaluate how runtime lifecycle hooks and explicit search primitives interact when conversational history scales to millions of tokens, we conducted the same 3-mode ablation across four long-horizon memory benchmarks: LoCoMo, LongMemEval-v2, Official LongMemEval (35,628 nodes), and BEAM at 1M and 10M tokens.

Division of labor: Ambient hooks as the map, explicit tools as the microscope

When a query requires synthesizing evidence distributed across six distinct sessions within a 35,000-node archive (for example, reconstructing a timeline of schema incidents across Q1, Q2, and Q3), each ablation mode exhibits a distinct retrieval dynamic:

• Unanchored search in tools_only (microscope in a dark room): Lacking initial entity or session anchors in the prompt, the model formulates a generic initial search query that retrieves high-frequency lexical distractors across all 35,000 nodes.
• Bounded Turn-1 injection in ambient_only (map without a microscope): To preserve prefix-cache alignment and bound prompt overhead (~1,300 tokens), the UserPromptSubmit hook injects only the top-5 state-consistent memories. While this single-pass grounding elevates LongMemEval-v2 Recall@5 from 60.8% to 92.5% without tool calls, multi-session aggregation queries spanning 6 to 8 distinct historical episodes exceed the capacity of a compact Turn-1 injection block.
• Guided multi-hop retrieval in ambient_plus_tools (map guiding the microscope): Because the Turn-1 ambient block populates the prompt with primary session IDs, canonical entity names, and temporal anchors, the model identifies the exact missing evidence hops and issues 1 to 2 targeted search_memories calls, raising Official LongMemEval accuracy to 90.0%, BEAM 1M to 88.4%, and BEAM 10M to 84.3%.

6. Conclusions & citation

@techreport{slashml2026ambient,
  title       = {Why Agents Forget to Use Their Own Memory Tools: A 12-Benchmark Study of Ambient Runtime Memory},
  author      = {SlashML Research},
  institution = {/ ml (SlashML)},
  year        = {2026},
  month       = {November},
  url         = {https://slashml.ai/research/ambient-memory.html}
}