Research / Systems Commentary ·

Is grep all you need? That depends.

Two widely read 2026 papers, "Keyword Search Is All You Need" (AAAI 2026) and "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search" (arXiv:2605.15184), demonstrated that giving a frontier model iterative grep over raw files beats naive chunked vector RAG on explicit question answering. That result tempted many teams to skip dedicated memory altogether and point agents at MEMORY.md with ripgrep. Examining their exact benchmark tables alongside 12 stateful workflow and 10M-token evaluations, we show why Sen et al. actually proved our core harness thesis: in stock Claude Code, raw grep scored just 55.2% to 76.7% (vs 90.0% for / ml on full 35,628-node LongMemEval), and as soon as prompts shift from explicit questions to imperative actions, grep over raw logs collapses to 0.0% post-compaction policy recall and 12.5% state supersession.

Papers Examined arXiv:2605.15184 & AAAI 2026
Claude Code + Grep vs / ml 55.2-76.7% → 90.0% (Full LME)
Post-Compaction & Supersession 0.0% / 12.5% → 100.0%
Coding Turn Token Spend 90% fewer tokens · <10 ms
Position & Empirical Scope

Any serious agent memory provider has to answer a blunt engineering question: "Why shouldn't I just log sessions to markdown files and let Claude Code or Codex run ripgrep?" This report addresses Subramanian et al. (AAAI 2026) [1] and Sen et al. (arXiv:2605.15184) [2] head-on. First, we explain why their critique of naive vector-only RAG is empirically sound: dense chunk embeddings blur literal witnesses (dates, symbols, IDs), and harness delivery mechanics swing retrieval accuracy by up to 46.5 percentage points on the same model. Second, we show why equating explicit retrospective QA with persistent agent memory is a category error across five production boundaries: unknown-unknown policy enforcement after context compaction, write-time state supersession, zero-overlap behavioral code search, sub-millisecond pre-tool gating and per-agent read redaction, and token/latency scaling at 10M tokens.

1. What "Is Grep All You Need?" got completely right

In early 2026, two papers challenged the default assumption that every LLM application needs a standalone vector database. At AAAI 2026, Subramanian et al. [1] showed that equipping a ReAct agent with command-line keyword tools (rga, pdfgrep, pdfmetadata) achieved 94.5% of the faithfulness and 91.5% of the answer correctness of Amazon Bedrock vector RAG without maintaining embeddings. Shortly after, Sen et al. ("Is Grep All You Need? How Agent Harnesses Reshape Agentic Search") [2] evaluated grep against dense vector search on a 116-question subset of LongMemEval-S [3] across four harnesses: stock Claude Code, Codex CLI, Gemini CLI, and a custom retrieval harness named Chronos.

Both papers exposed real architectural flaws in first-generation vector RAG that we independently observed when designing / ml:

  • Dense chunk embeddings act as lossy compression on literal witnesses: Conversational and engineering queries frequently hinge on exact tokens: an invoice ID, port 6380, error code ECONNRESET, or a specific date. Compressing a 500-token chunk into a single dense vector smears rare literal tokens into the surrounding topical average, causing vector-only RAG in Sen et al. Table 1 to trail inline grep across every model-harness pair (for example, 62.9% vs 86.2% on Gemini 3.1 Flash-Lite) [2].
  • The agent harness reshapes accuracy more than the underlying index: The most important finding in Sen et al. is often overlooked by readers who stop at the title. Holding the model (Claude Opus 4.6) and the tool (grep) completely fixed, accuracy swung from 68.1% (Claude Code, programmatic file delivery) and 76.7% (Claude Code, inline) to 93.1% inside their custom Chronos QA harness. On Codex CLI with GPT-5.4, switching tool delivery from inline to file-based collapsed grep accuracy by 37.9 percentage points (93.1% → 55.2%) [2].
Lesson I · The Harness Delivery Theorem
Sen et al. did not prove that raw grep makes every agent smart; they proved that stock coding harnesses lose 15 to 46 percentage points unless context is delivered cleanly into the execution loop. That is the exact systems motivation behind / ml: preserve exact literal recall alongside meaning and current state, and deliver grounded context synchronously via lifecycle hooks rather than forcing a CLI agent to improvise multi-step file reads.
Figure 1 · Stock Coding Harnesses vs / ml on LongMemEval

In stock production harnesses (Claude Code, Gemini CLI), raw grep scores 55.2% to 81.9% while swinging wildly by delivery mode

Left: Published LongMemEval accuracy in stock CLI harnesses from Sen et al. Table 1 (116-question subset) [2] vs / ml on the full 35,628-node Official LongMemEval [3] [6]. Right: Accuracy collapse when identical models + grep switch harness or tool-output delivery.

A. Stock CLI Harnesses LongMemEval Acc
/ ml (LME-v2)
93.8% (95.1 R@5)
/ ml (Full 35.6K)
90.0% (<10 ms)
Gemini CLI
81.9% (Grep)
Claude Code
76.7% (Opus)
CC + Vector
75.0% (Opus)
Claude Code
55.2% (Haiku)
/ ml evaluated on full 2,300-session / 35,628-node archive [6]
B. Grep Harness Fragility Sen et al. Table 1 [2]
Claude Haiku 4.5 + Grep (Chronos vs Claude Code) -46.5 pp
Chronos QA
83.6%
Claude Code
37.1%
GPT-5.4 + Grep on Codex CLI (Inline vs File) -37.9 pp
Inline hits
93.1%
File-based
55.2%
Claude Opus 4.6 + Grep (Chronos vs Claude Code) -25.0 pp
Chronos QA
93.1%
Claude Code
68.1%
Reaching 93.1% required a custom QA harness + frontier Opus 4.6/GPT-5.4 loops

2. The category error: Retrospective QA is not stateful agent memory

Why did grep look like a plausible replacement for memory on LongMemEval-S? Look at the shape of every prompt in both Sen et al. [2] and Subramanian et al. [1]: 100% of test inputs are explicit look-back questions ("What date did my lease renew?", "How many times did I visit the clinic in May?"). When a user asks a trivia question about past sessions, the model knows immediately that the answer is not in its weights and starts firing search calls.

Real coding, DevOps, customer operations, and personal assistant sessions rarely look like trivia quizzes. Users issue imperative task instructions ("Ship the invoice schema update", "Approve refund #8841", "Book dinner for Friday") whose constraints were established dozens of sessions earlier and evicted during context compaction:

Session 01 (Saved to MEMORY.md): "Never run raw SQL on billing-service; always create an Alembic revision first."
Session 08 (Context compacted): Agent starts a fresh terminal window in the monorepo.
Session 08 Prompt: "Add the tax_exempt column to invoices and push to staging."
  → Agent + ripgrep: Prompt has 0 cues to grep MEMORY.md · 0 rg calls · runs ALTER TABLE ✗
  → Agent + / ml: UserPromptSubmit (<10 ms) + PreToolUse (<1 ms) surface & enforce Alembic rule ✓
Figure 2 · Search Tool vs Memory Runtime

How ripgrep + MEMORY.md, dense vector RAG, and / ml behave across a real stateful workflow

Tracing literal precision, unknown-unknown policy recall, state mutation handling, and pre-execution safety across four operational events.

1. Agentic Grep + Markdown Logs (rg over MEMORY.md / Transcripts) Exact Byte Match · Voluntary CLI Tool
1. Literal QA · Strong ✓ Finds exact IDs, dates & symbols when prompt explicitly asks a question.
2. Post-Compaction · 0.0% ✗ Imperative prompts carry no cue to grep notes; rules silently skipped.
3. State Updates · 12.5% ✗ rg "region" returns both repealed us-east-1 and active eu-central-1.
4. Governance · 0% Gate ✗ Cannot block bad tool calls pre-execution or mask PII/secrets per agent role.
2. Chunked Vector-Only RAG (Standard Dense Embeddings) Cosine Top-k Chunks · Append-Only Store
1. Literal QA · Lossy ✗ Dense vectors blur exact dates, counts & error codes (62.9-83.6% in Sen et al.).
2. Ingress Dump · 5.20x Tok ✗ Unconditional top-k stuffing breaks prompt prefix caches on every turn.
3. State Updates · Stale ✗ Old and new facts share >0.92 cosine similarity and co-rank together.
4. Polarity · 70-100% FP ✗ Flags compliant deploys as violations due to shared entity nouns.
3. / ml Ambient Memory Runtime Synchronous 4-Stage Hooks + On-Demand MCP Search (<10 ms)
1. Literal & Meaning · 90.0%+ ✓ Finds exact IDs, dates & symbols as well as paraphrased behavior.
2. Post-Compaction · 100% ✓ UserPromptSubmit & PostToolUse ground invariants at 1.04x tokens.
3. Supersession · 100% ✓ Stop hook retires obsolete values synchronously (<10 ms sync).
4. Gate & DLP · <1 ms ✓ PreToolUse blocks violations (0% FP); dynamic read-masking prevents leaks.

3. Five production walls where raw grep breaks

When teams replace a memory engine with raw file search over session logs or markdown notes, five failure modes appear as soon as usage moves past single-user demos:

1

Unknown-unknown blindness after compaction (0.0% unprompted policy survival)

grep is an explicit, model-invoked tool, making it architecturally identical to the tools_only configuration in our 12-benchmark ablation [6]. Once context compacts, routine task instructions contain no hint that a standing constraint exists in an external file, yielding 0.0% policy recall on DolphinBench Lifecycle Stage 4 [4] and 47.3% Write F1 when the agent is busy running tests and forgets to append lessons back to MEMORY.md.

2

Zero temporal supersession on mutated state (12.5% cross-turn update accuracy)

grep scans flat text without state semantics. If Session 2 logs prod_db = us-east-1, Session 7 discusses a rehearsal in us-east-1, and Session 12 migrates to eu-central-1, running rg "prod_db|region" returns three hits for the repealed region and one hit for the current one. Even in Sen et al. Experiment 2 (Tables 2 & 3), adding distractor sessions produced non-monotonic accuracy drops (e.g., GPT-5.4 + grep falling from 83.2% at s5 to 78.5% at s30/full) [2]. On explicit state-mutation suites, append-only log search resolves only 12.5% of DolphinBench S3 updates and 44.0% of 300 embodied spatial moves, compared to 100.0% with synchronous write-time retirement in / ml [7].

3

Vocabulary mismatch on behavioral code & root-cause queries

ripgrep matches exact character sequences, not runtime behavior. Searching rg "retry backoff" misses a module that named its state jitter_window_ms; grepping incident notes for "rate limit drop" misses an outage logged under TokenBucket starvation. Even on document QA in Subramanian et al. [1], keyword tools lost 12.0% of context recall (0.68 vs 0.77) whenever queries paraphrased source terminology. / ml exposes behavioral code and root-cause tools (search_code_by_behavior, find_bug_candidates) that locate relevant implementations in ~10 ms even when query terms share zero identifiers with the source file.

4

No pre-execution interception (PreToolUse) and zero read-time secret redaction

A passive search command cannot stop an agent before it executes a destructive shell command or issues an unauthorized $450 refund, whereas / ml's PreToolUse gate blocks 100% of policy violations in <1 ms with 0.0% compliant false alarms [6]. Worse, pointing grep at raw transcripts dumps unredacted API keys, customer PII, PHI, and legal/MNPI notes straight into every subagent's prompt, whereas / ml enforces dynamic recall-time masking per agent clearance (99.53% internal recall, 0.00% unauthorized leak) [8] and verified 6.41 ms GDPR Art. 17 erasure [7].

5

Multi-turn frontier token tax & 10M-token scaling

In Sen et al., high grep scores required frontier models (Claude Opus 4.6 or GPT-5.4) executing iterative search-read-refine loops across multiple LLM round-trips per question. When a common pattern matches hundreds of lines across a 10M-token archive (20,561 nodes on ICLR 2026 BEAM [5]), pasting matched files burns ~18,400 prompt tokens per turn ($4.60 across a 50-step coding task). / ml completes 10M-token recall in a single <42 ms lookup at ~1,850 tokens per turn (90% fewer tokens, <$0.50 per 50-step run) while sustaining 84.3% accuracy on BEAM 10M.

Figure 3 · Stateful Failure Modes & Token Economics

Where voluntary log search collapses: state mutations, compactions, and 10x coding prompt bloat

Left: Accuracy on stateful workflows where rules compact out of context or facts mutate over time [4] [6] [7]. Right: Measured prompt tokens and model spend across a 50-step coding task.

A. Stateful Agent Tasks Log Search vs / ml
Post-Compaction Policy (DolphinBench S4) +100.0 pp
Voluntary tool
0.0%
/ ml hooks
100.0%
Cross-Turn Supersession (DolphinBench S3) +87.5 pp
Append-only
12.5%
/ ml sync
100.0%
2,800-Turn Stateful Assistant (200 Vaults) +44.2 pp
Tools-only
52.3%
/ ml ambient
96.5%
BEAM 10M-token synthesis (20,561 nodes): 84.3% in <42 ms P50
B. 50-Step Coding Economics Lower = Better
Prompt Context Injected per Coding Turn 90% fewer tok
Grep + files
18,400 tok
/ ml lookup
1,850 tok
Model Token Spend per 50-Step Task 9.2x cheaper
Grep + files
$4.60
/ ml lookup
<$0.50
Retrieval Round-Trips & Wall-Clock Latency Single-pass
Iterative grep
3-8 turns
/ ml runtime
1x · <10 ms
Preserves prompt prefix caches at 1.04x overhead vs 5.20x wrapper dumps
Figure 4 · Long-Horizon Scaling (26K → 10M Tokens)

As session archives grow toward 10M tokens, raw grep hits a context-bloat wall while / ml stays flat

Left: Multi-session retrieval accuracy from LoCoMo (26K tokens) and LongMemEval (35.6K nodes, showing stock Claude Code + grep from Sen et al. Table 1 [2]) to BEAM 1M and 10M tokens [5]. Right: Prompt tokens injected per coding turn as distractor sessions accumulate.

A. Horizon Accuracy 26K → 10M Tokens
100% 80% 60% 40% +35.7 pp Zep 75.1% Zep 42.1% Mem0 66.9% Mem0 48.6% CC+Grep 76.7% (Opus) CC+Grep 55.2% (Haiku) 94.5% 90.0% 88.4% / ml 84.3% LoCoMo 26K tok LongMemEval 35.6K nodes BEAM 1M 1.0M tok BEAM 10M 10.0M tok
Single-pass / ml recall stays <42 ms P50 even at 10M tokens
B. Prompt Context Growth Lower = Cheaper
20K 15K 10K 5K 0 -90% tok Top-200 Vec (~7,000 tok) 7.4K 13.8K Grep + files: 18,400 tok 1.4K / ml: 1,850 tok s5 Distractors 5 sessions s30 Distractors 30 sessions BEAM 1M 1.0M tok BEAM 10M 20.5K nodes
Compact recalled context prevents multi-file transcript dumps

4. Synthesis: How ripgrep and / ml work together

Framing grep and agent memory as mutually exclusive misses how high-performing coding harnesses actually operate. ripgrep is a working-tree byte scanner; / ml is a stateful execution-loop memory runtime. In Claude Code, Cursor, Codex, and Windsurf, the fastest workflow pairs both:

Working-Tree Scanner · Live Disk Bytes

When ripgrep is the right tool

  • Enumerating every live call site of an exact known identifier (rg "validate_invoice_v2\(") in uncommitted working-tree files.
  • Verifying exact line numbers before applying a mechanical multi-file rename or import refactor.
  • Checking static config literals inside the currently checked-out git branch when no historical rationale is needed.
Stateful Runtime · Cross-Session Memory

When / ml is required

  • Surfacing standing repo invariants and failed dead-end fixes automatically after context compaction (UserPromptSubmit, PostToolUse).
  • Blocking out-of-policy migrations, deploys, or API side effects in <1 ms before execution (PreToolUse).
  • Searching code and past incidents by behavior (search_code_by_behavior, find_bug_candidates) before symbol names are known.
  • Retiring superseded state in place (<10 ms), masking secrets per agent role on read, and executing 6.41 ms GDPR erasures.

Table 1: Architectural Capabilities Across Agentic Grep, Vector RAG, and / ml

Comparing raw CLI lexical search, standard dense chunk RAG, and the / ml ambient memory runtime across retrieval, state lifecycle, and governance.

Capability / Benchmark Agentic Grep + Files Chunked Vector RAG / ml Memory Runtime
Exact Literal Witnesses (IDs, Dates, Ports) Exact byte match Blurred in dense chunk vectors Preserved (exact IDs, dates & ports)
Zero-Overlap Behavioral / Conceptual Recall Misses synonyms & renamed symbols Supported (noisy top-k) Supported (~10 ms via MCP & API)
Stock Claude Code LongMemEval [2] [3] 55.2% (Haiku) / 68.1-76.7% (Opus) 44.0% (Haiku) / 75.0% (Opus) 90.0% (Full 35.6K) / 93.8% (LME-v2)
Post-Compaction Unknown-Unknowns (S4) [4] 0.0% (voluntary tool skipped) 5.20x prompt bloat if prepended 100.0% (<10 ms hook, 1.04x tok)
State Supersession (DolphinBench S3) [4] 12.5% (returns old & new lines) 12.5% (co-ranks contradictions) 100.0% (write-time retirement)
Pre-Execution Policy Gate (PreToolUse) None (passive read command) 70-100% polarity false alarms <1 ms block · 0.0% false alarms
Per-Agent Read Redaction & GDPR Art. 17 Raw plaintext dump; manual file edit Index rebuild + summary leaks 0.00% leak on read · 6.41 ms purge
10M-Token Horizon (ICLR 2026 BEAM) [5] Multi-turn scan & context overflow 48.6% (Mem0) / 42.1% (Zep) 84.3% (single-pass <42 ms P50)

5. Conclusions & reproduction

1

"Is Grep All You Need?" debunked naive vector-only RAG, not stateful agent memory

Exact lexical matching beats lossy dense chunking on literal QA, and harness delivery mechanics dictate whether an agent can use retrieved evidence at all (swinging identical models by up to 46.5 percentage points in Sen et al. Table 1).

2

Flat file search fails as soon as prompts become imperative and state mutates

Pointing ripgrep at MEMORY.md leaves post-compaction policy enforcement at 0.0%, surfaces repealed facts beside active ones (12.5% fact-update supersession (S3)), misses zero-overlap behavioral queries, and exposes raw secrets in prompts.

3

Pair ripgrep for live working-tree bytes with / ml for persistent state

Keep ripgrep for exact symbol call sites in checked-out files while letting / ml's 4-stage hooks and MCP tools handle cross-session rules, behavioral code recall, supersession, and governance in under 10 ms.

Add ambient memory alongside ripgrep in one command

Attach 4-stage lifecycle hooks plus behavioral code and memory search to Claude Code, Cursor, Codex, or Windsurf, or run the open-source evaluation CLI:

bash
# 1. Attach / ml lifecycle hooks + MCP search to Claude Code, Cursor, or Codex
npx -y @slashml/memory@latest init

# 2. Install Python / TypeScript SDKs & standalone benchmark harness
pip install slashml-memory slashml-eval
npm install @slashml/memory

# 3. Reproduce LongMemEval, LoCoMo, and code/prose suites
export SLASHML_API_KEY="sml_live_..."
slashml-eval --benchmark all --json

References & citation

  1. Subramanian, S., et al. (2026). "Keyword Search Is All You Need: Achieving RAG-Level Performance Without Vector Databases Using Agentic Tool Use." Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2026), arXiv:2602.23368. https://arxiv.org/abs/2602.23368
  2. Sen, S., Kasturi, A., Lumer, E., Gulati, A., & Subbiah, V. K. (2026). "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search." arXiv:2605.15184. https://arxiv.org/abs/2605.15184
  3. Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2025). "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory." International Conference on Learning Representations (ICLR 2025), arXiv:2410.10813. https://arxiv.org/abs/2410.10813
  4. Rathi, S., Yadav, D., & Singh, T. (2026). "DolphinBench: Mapping the Pareto Frontier of Agent Memory." Official 600-task action-execution evaluation harness and leaderboard. https://dolphinbench.ai
  5. Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., & Mitchell, J. R. (2026). "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs (BEAM)." International Conference on Learning Representations (ICLR 2026), arXiv:2510.27246. https://arxiv.org/abs/2510.27246
  6. SlashML Research (2026). "Why Agents Forget to Use Their Own Memory Tools: A 12-Benchmark Study of Ambient Runtime Memory." / ml Technical Report. https://slashml.ai/research/ambient-memory.html
  7. SlashML Research (2026). "When Remembering Everything Breaks AI Agents: Why Selective Erasure Is Fundamental, and Hard." / ml Technical Report. https://slashml.ai/research/forgetting.html
  8. SlashML Research (2026). "Scrub on Write, or Leak on Read: The AI Memory Redaction Dilemma Across 13 Benchmarks." / ml Technical Report. https://slashml.ai/research/redaction.html
bibtex
@techreport{slashml2026grep,
  title       = {Is grep all you need? That depends.},
  author      = {SlashML Research},
  institution = {/ ml (SlashML)},
  year        = {2026},
  month       = {November},
  url         = {https://slashml.ai/research/is-grep-all-you-need.html}
}