Any serious agent memory provider has to answer a blunt engineering question: "Why shouldn't I just log sessions to markdown files and let Claude Code or Codex run ripgrep?" This report addresses Subramanian et al. (AAAI 2026) [1] and Sen et al. (arXiv:2605.15184) [2] head-on. First, we explain why their critique of naive vector-only RAG is empirically sound: dense chunk embeddings blur literal witnesses (dates, symbols, IDs), and harness delivery mechanics swing retrieval accuracy by up to 46.5 percentage points on the same model. Second, we show why equating explicit retrospective QA with persistent agent memory is a category error across five production boundaries: unknown-unknown policy enforcement after context compaction, write-time state supersession, zero-overlap behavioral code search, sub-millisecond pre-tool gating and per-agent read redaction, and token/latency scaling at 10M tokens.
1. What "Is Grep All You Need?" got completely right
In early 2026, two papers challenged the default assumption that every LLM application needs a standalone vector database. At AAAI 2026, Subramanian et al. [1] showed that equipping a ReAct agent with command-line keyword tools (rga, pdfgrep, pdfmetadata) achieved 94.5% of the faithfulness and 91.5% of the answer correctness of Amazon Bedrock vector RAG without maintaining embeddings. Shortly after, Sen et al. ("Is Grep All You Need? How Agent Harnesses Reshape Agentic Search") [2] evaluated grep against dense vector search on a 116-question subset of LongMemEval-S [3] across four harnesses: stock Claude Code, Codex CLI, Gemini CLI, and a custom retrieval harness named Chronos.
Both papers exposed real architectural flaws in first-generation vector RAG that we independently observed when designing / ml:
-
Dense chunk embeddings act as lossy compression on literal witnesses: Conversational and engineering queries frequently hinge on exact tokens: an invoice ID, port
6380, error codeECONNRESET, or a specific date. Compressing a 500-token chunk into a single dense vector smears rare literal tokens into the surrounding topical average, causing vector-only RAG in Sen et al. Table 1 to trail inlinegrepacross every model-harness pair (for example, 62.9% vs 86.2% on Gemini 3.1 Flash-Lite) [2]. -
The agent harness reshapes accuracy more than the underlying index: The most important finding in Sen et al. is often overlooked by readers who stop at the title. Holding the model (Claude Opus 4.6) and the tool (
grep) completely fixed, accuracy swung from 68.1% (Claude Code, programmatic file delivery) and 76.7% (Claude Code, inline) to 93.1% inside their custom Chronos QA harness. On Codex CLI with GPT-5.4, switching tool delivery from inline to file-based collapsedgrepaccuracy by 37.9 percentage points (93.1% → 55.2%) [2].
Sen et al. did not prove that raw
grep makes every agent smart; they proved that stock coding harnesses lose 15 to 46 percentage points unless context is delivered cleanly into the execution loop. That is the exact systems motivation behind / ml: preserve exact literal recall alongside meaning and current state, and deliver grounded context synchronously via lifecycle hooks rather than forcing a CLI agent to improvise multi-step file reads.
In stock production harnesses (Claude Code, Gemini CLI), raw grep scores 55.2% to 81.9% while swinging wildly by delivery mode
Left: Published LongMemEval accuracy in stock CLI harnesses from Sen et al. Table 1 (116-question subset) [2] vs / ml on the full 35,628-node Official LongMemEval [3] [6]. Right: Accuracy collapse when identical models + grep switch harness or tool-output delivery.
/ ml evaluated on full 2,300-session / 35,628-node archive [6]
2. The category error: Retrospective QA is not stateful agent memory
Why did grep look like a plausible replacement for memory on LongMemEval-S? Look at the shape of every prompt in both Sen et al. [2] and Subramanian et al. [1]: 100% of test inputs are explicit look-back questions ("What date did my lease renew?", "How many times did I visit the clinic in May?"). When a user asks a trivia question about past sessions, the model knows immediately that the answer is not in its weights and starts firing search calls.
Real coding, DevOps, customer operations, and personal assistant sessions rarely look like trivia quizzes. Users issue imperative task instructions ("Ship the invoice schema update", "Approve refund #8841", "Book dinner for Friday") whose constraints were established dozens of sessions earlier and evicted during context compaction:
billing-service; always create an Alembic revision first."Session 08 (Context compacted): Agent starts a fresh terminal window in the monorepo.
Session 08 Prompt: "Add the
tax_exempt column to invoices and push to staging."→ Agent + ripgrep: Prompt has 0 cues to grep MEMORY.md · 0 rg calls · runs ALTER TABLE ✗
→ Agent + / ml: UserPromptSubmit (<10 ms) + PreToolUse (<1 ms) surface & enforce Alembic rule ✓
How ripgrep + MEMORY.md, dense vector RAG, and / ml behave across a real stateful workflow
Tracing literal precision, unknown-unknown policy recall, state mutation handling, and pre-execution safety across four operational events.
rg over MEMORY.md / Transcripts)
Exact Byte Match · Voluntary CLI Tool
grep notes; rules silently skipped.
rg "region" returns both repealed us-east-1 and active eu-central-1.
UserPromptSubmit & PostToolUse ground invariants at 1.04x tokens.
Stop hook retires obsolete values synchronously (<10 ms sync).
PreToolUse blocks violations (0% FP); dynamic read-masking prevents leaks.
3. Five production walls where raw grep breaks
When teams replace a memory engine with raw file search over session logs or markdown notes, five failure modes appear as soon as usage moves past single-user demos:
Unknown-unknown blindness after compaction (0.0% unprompted policy survival)
grep is an explicit, model-invoked tool, making it architecturally identical to the tools_only configuration in our 12-benchmark ablation [6]. Once context compacts, routine task instructions contain no hint that a standing constraint exists in an external file, yielding 0.0% policy recall on DolphinBench Lifecycle Stage 4 [4] and 47.3% Write F1 when the agent is busy running tests and forgets to append lessons back to MEMORY.md.
Zero temporal supersession on mutated state (12.5% cross-turn update accuracy)
grep scans flat text without state semantics. If Session 2 logs prod_db = us-east-1, Session 7 discusses a rehearsal in us-east-1, and Session 12 migrates to eu-central-1, running rg "prod_db|region" returns three hits for the repealed region and one hit for the current one. Even in Sen et al. Experiment 2 (Tables 2 & 3), adding distractor sessions produced non-monotonic accuracy drops (e.g., GPT-5.4 + grep falling from 83.2% at s5 to 78.5% at s30/full) [2]. On explicit state-mutation suites, append-only log search resolves only 12.5% of DolphinBench S3 updates and 44.0% of 300 embodied spatial moves, compared to 100.0% with synchronous write-time retirement in / ml [7].
Vocabulary mismatch on behavioral code & root-cause queries
ripgrep matches exact character sequences, not runtime behavior. Searching rg "retry backoff" misses a module that named its state jitter_window_ms; grepping incident notes for "rate limit drop" misses an outage logged under TokenBucket starvation. Even on document QA in Subramanian et al. [1], keyword tools lost 12.0% of context recall (0.68 vs 0.77) whenever queries paraphrased source terminology. / ml exposes behavioral code and root-cause tools (search_code_by_behavior, find_bug_candidates) that locate relevant implementations in ~10 ms even when query terms share zero identifiers with the source file.
No pre-execution interception (PreToolUse) and zero read-time secret redaction
A passive search command cannot stop an agent before it executes a destructive shell command or issues an unauthorized $450 refund, whereas / ml's PreToolUse gate blocks 100% of policy violations in <1 ms with 0.0% compliant false alarms [6]. Worse, pointing grep at raw transcripts dumps unredacted API keys, customer PII, PHI, and legal/MNPI notes straight into every subagent's prompt, whereas / ml enforces dynamic recall-time masking per agent clearance (99.53% internal recall, 0.00% unauthorized leak) [8] and verified 6.41 ms GDPR Art. 17 erasure [7].
Multi-turn frontier token tax & 10M-token scaling
In Sen et al., high grep scores required frontier models (Claude Opus 4.6 or GPT-5.4) executing iterative search-read-refine loops across multiple LLM round-trips per question. When a common pattern matches hundreds of lines across a 10M-token archive (20,561 nodes on ICLR 2026 BEAM [5]), pasting matched files burns ~18,400 prompt tokens per turn ($4.60 across a 50-step coding task). / ml completes 10M-token recall in a single <42 ms lookup at ~1,850 tokens per turn (90% fewer tokens, <$0.50 per 50-step run) while sustaining 84.3% accuracy on BEAM 10M.
Where voluntary log search collapses: state mutations, compactions, and 10x coding prompt bloat
Left: Accuracy on stateful workflows where rules compact out of context or facts mutate over time [4] [6] [7]. Right: Measured prompt tokens and model spend across a 50-step coding task.
As session archives grow toward 10M tokens, raw grep hits a context-bloat wall while / ml stays flat
Left: Multi-session retrieval accuracy from LoCoMo (26K tokens) and LongMemEval (35.6K nodes, showing stock Claude Code + grep from Sen et al. Table 1 [2]) to BEAM 1M and 10M tokens [5]. Right: Prompt tokens injected per coding turn as distractor sessions accumulate.
/ ml recall stays <42 ms P50 even at 10M tokens
4. Synthesis: How ripgrep and / ml work together
Framing grep and agent memory as mutually exclusive misses how high-performing coding harnesses actually operate. ripgrep is a working-tree byte scanner; / ml is a stateful execution-loop memory runtime. In Claude Code, Cursor, Codex, and Windsurf, the fastest workflow pairs both:
When ripgrep is the right tool
- Enumerating every live call site of an exact known identifier (
rg "validate_invoice_v2\(") in uncommitted working-tree files. - Verifying exact line numbers before applying a mechanical multi-file rename or import refactor.
- Checking static config literals inside the currently checked-out git branch when no historical rationale is needed.
When / ml is required
- Surfacing standing repo invariants and failed dead-end fixes automatically after context compaction (
UserPromptSubmit,PostToolUse). - Blocking out-of-policy migrations, deploys, or API side effects in <1 ms before execution (
PreToolUse). - Searching code and past incidents by behavior (
search_code_by_behavior,find_bug_candidates) before symbol names are known. - Retiring superseded state in place (<10 ms), masking secrets per agent role on read, and executing 6.41 ms GDPR erasures.
Table 1: Architectural Capabilities Across Agentic Grep, Vector RAG, and / ml
Comparing raw CLI lexical search, standard dense chunk RAG, and the / ml ambient memory runtime across retrieval, state lifecycle, and governance.
| Capability / Benchmark | Agentic Grep + Files | Chunked Vector RAG | / ml Memory Runtime |
|---|---|---|---|
| Exact Literal Witnesses (IDs, Dates, Ports) | Exact byte match | Blurred in dense chunk vectors | Preserved (exact IDs, dates & ports) |
| Zero-Overlap Behavioral / Conceptual Recall | Misses synonyms & renamed symbols | Supported (noisy top-k) | Supported (~10 ms via MCP & API) |
| Stock Claude Code LongMemEval [2] [3] | 55.2% (Haiku) / 68.1-76.7% (Opus) | 44.0% (Haiku) / 75.0% (Opus) | 90.0% (Full 35.6K) / 93.8% (LME-v2) |
| Post-Compaction Unknown-Unknowns (S4) [4] | 0.0% (voluntary tool skipped) | 5.20x prompt bloat if prepended | 100.0% (<10 ms hook, 1.04x tok) |
| State Supersession (DolphinBench S3) [4] | 12.5% (returns old & new lines) | 12.5% (co-ranks contradictions) | 100.0% (write-time retirement) |
Pre-Execution Policy Gate (PreToolUse) |
None (passive read command) | 70-100% polarity false alarms | <1 ms block · 0.0% false alarms |
| Per-Agent Read Redaction & GDPR Art. 17 | Raw plaintext dump; manual file edit | Index rebuild + summary leaks | 0.00% leak on read · 6.41 ms purge |
| 10M-Token Horizon (ICLR 2026 BEAM) [5] | Multi-turn scan & context overflow | 48.6% (Mem0) / 42.1% (Zep) | 84.3% (single-pass <42 ms P50) |
5. Conclusions & reproduction
"Is Grep All You Need?" debunked naive vector-only RAG, not stateful agent memory
Exact lexical matching beats lossy dense chunking on literal QA, and harness delivery mechanics dictate whether an agent can use retrieved evidence at all (swinging identical models by up to 46.5 percentage points in Sen et al. Table 1).
Flat file search fails as soon as prompts become imperative and state mutates
Pointing ripgrep at MEMORY.md leaves post-compaction policy enforcement at 0.0%, surfaces repealed facts beside active ones (12.5% fact-update supersession (S3)), misses zero-overlap behavioral queries, and exposes raw secrets in prompts.
Pair ripgrep for live working-tree bytes with / ml for persistent state
Keep ripgrep for exact symbol call sites in checked-out files while letting / ml's 4-stage hooks and MCP tools handle cross-session rules, behavioral code recall, supersession, and governance in under 10 ms.
Add ambient memory alongside ripgrep in one command
Attach 4-stage lifecycle hooks plus behavioral code and memory search to Claude Code, Cursor, Codex, or Windsurf, or run the open-source evaluation CLI:
# 1. Attach / ml lifecycle hooks + MCP search to Claude Code, Cursor, or Codex
npx -y @slashml/memory@latest init
# 2. Install Python / TypeScript SDKs & standalone benchmark harness
pip install slashml-memory slashml-eval
npm install @slashml/memory
# 3. Reproduce LongMemEval, LoCoMo, and code/prose suites
export SLASHML_API_KEY="sml_live_..."
slashml-eval --benchmark all --json
References & citation
-
Subramanian, S., et al. (2026). "Keyword Search Is All You Need: Achieving RAG-Level Performance Without Vector Databases Using Agentic Tool Use." Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2026),
arXiv:2602.23368. https://arxiv.org/abs/2602.23368 -
Sen, S., Kasturi, A., Lumer, E., Gulati, A., & Subbiah, V. K. (2026). "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search."
arXiv:2605.15184. https://arxiv.org/abs/2605.15184 -
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2025). "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory." International Conference on Learning Representations (ICLR 2025),
arXiv:2410.10813. https://arxiv.org/abs/2410.10813 - Rathi, S., Yadav, D., & Singh, T. (2026). "DolphinBench: Mapping the Pareto Frontier of Agent Memory." Official 600-task action-execution evaluation harness and leaderboard. https://dolphinbench.ai
-
Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., & Mitchell, J. R. (2026). "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs (BEAM)." International Conference on Learning Representations (ICLR 2026),
arXiv:2510.27246. https://arxiv.org/abs/2510.27246 -
SlashML Research (2026). "Why Agents Forget to Use Their Own Memory Tools: A 12-Benchmark Study of Ambient Runtime Memory."
/ mlTechnical Report. https://slashml.ai/research/ambient-memory.html -
SlashML Research (2026). "When Remembering Everything Breaks AI Agents: Why Selective Erasure Is Fundamental, and Hard."
/ mlTechnical Report. https://slashml.ai/research/forgetting.html -
SlashML Research (2026). "Scrub on Write, or Leak on Read: The AI Memory Redaction Dilemma Across 13 Benchmarks."
/ mlTechnical Report. https://slashml.ai/research/redaction.html
@techreport{slashml2026grep,
title = {Is grep all you need? That depends.},
author = {SlashML Research},
institution = {/ ml (SlashML)},
year = {2026},
month = {November},
url = {https://slashml.ai/research/is-grep-all-you-need.html}
}