Is grep all you need? Agentic search vs vector retrieval

Claude Code ships without a vector index. It searches a working directory the way a Unix user would, with grep over plain files, and it works well enough that a year of arguments has followed: if lexical search can carry a coding agent, the embedding pipeline under your RAG stack starts to look optional. In May 2026, five authors at PricewaterhouseCoopers put the argument on a benchmark. Their paper, Is Grep All You Need? How Agent Harnesses Reshape Agentic Search, went up on the 14th and drew a written reply from LlamaIndex 12 days later. By 2 August 2026 its Hacker News thread carried 164 points and 64 comments.

The decision it bears on is real infrastructure. An embedding index needs a chunking policy, a sync job, and a re-embedding plan every time the model or the corpus changes; grep needs a filesystem. The paper’s answer is conditional on two variables most arguments skip. With search results returned inline, grep beat vector retrieval for every one of the ten harness and model pairs tested; with results written to files, vector won five of ten. Swapping the harness moved accuracy as much as swapping the retriever did. Read the paper as a license to benchmark the whole configuration you ship, and as a warning that a retriever comparison run on someone else’s harness may say little about yours.

What the paper tested, and the three dimensions it ties together

The setup is small and concrete. The authors take a 116-question subset of the LongMemEval benchmark, which grounds each question in multi-session chat history: knowledge updates, multi-session questions, three single-session variants, and temporal reasoning. Four harnesses run the same retrieval task. Chronos, the authors’ own LangChain-based harness with four search tools, sits alongside three provider CLIs: Claude Code, Codex CLI, and Gemini CLI. The model pool spans Claude Opus 4.6, Claude Haiku 4.5, GPT-5.4, Gemini 3.1 Pro, and Gemini 3.1 Flash-Lite, and a GPT-4o judge scores each answer as correct or incorrect.

Around that setup the paper builds a framework of three dimensions it argues cannot be measured independently. The retrieval strategy is lexical (grep, BM25, and the learned-sparse SPLADE), semantic (dense vectors with approximate nearest-neighbor search), or hybrid (rank fusion through to agent-driven selection). For the harness, the choice runs between a custom loop, which gives fine-grained control over prompts and stopping criteria, and a provider CLI, whose orchestration is opaque but which needs almost no setup. Then there is the tool-calling architecture, the route results take to reach the model: inline, where “search results are returned directly as tool response messages appended to the conversation context”, or programmatic, where “search results are written to disk and the model receives only a file path or summary pointer”.

Their claim is that end-to-end accuracy is a joint function of all three, because the harness decides how queries get scheduled and when searching stops. In this site’s terms, the experiment varies the retrieval tool layer of an agentic RAG architecture while holding the rest of the pattern still. The tables bear the framing out, one delivery mode at a time.

Inline delivery: grep won all ten pairings

Experiment 1 crosses every harness and model pair with both retrievers under both delivery modes. Under inline delivery the result is uniform: grep beat vector for all ten pairings. The widest margin is Chronos running Gemini 3.1 Flash-Lite, 86.2% against 62.9%, a 23.3pp gap. The narrowest is Claude Code running Claude Opus 4.6, 76.7% against 75.0%, which on a 116-question set is a two-question difference.

The mechanism is unglamorous. LongMemEval answers tend to sit on verbatim spans: a date the user stated, a preference phrased once, a fact repeated word for word across sessions. Exact string matching is built for that, and an agent that greps iteratively, reformulating the pattern after each miss, recovers much of the recall an embedding index would otherwise contribute.

Inline is only half of Experiment 1, though, and the other half is where the paper stops being quotable in one sentence.

Write the results to files and the ordering breaks

Programmatic delivery changes one thing: instead of pasting hits into the conversation, the harness writes them to disk and hands the model a path.

That single change reversed five of the ten pairings.

Every cell there is a single point estimate on 116 questions, published with no interval attached, so the rows that move by a point or two are ties in all but name.

Two patterns sit in the grid regardless. The flips concentrate in the provider CLIs, where four of five rows reverse, while Chronos holds grep’s lead in four of its five. And the delivery-mode swings dwarf the retriever gaps: Codex CLI with GPT-5.4 falls from 93.1% to 55.2% on grep when the delivery mode changes, a 37.9pp drop with the corpus, model, and retriever held constant. A retrieval comparison that does not state its delivery mode has left out a variable this table shows can decide the outcome.

The harness moved accuracy as much as the retriever

The paper’s subtitle is about the harness. In Experiment 1 the same Claude Opus 4.6 backbone scores 93.1% under Chronos and 76.7% under Claude Code, on the same questions, with the same retriever; the authors note that changing the harness “shifts the ceiling by roughly as much as swapping retrievers within a fixed harness”. That 16.4pp spread sits inside the 1.7 to 23.3pp range the inline retriever gaps cover, nearer its top end than its bottom.

The harness effect generalizes further than the retriever result. When two teams argue grep against embeddings across different stacks, harness effects of this size mean they are comparing bundles, and each bundle carries query scheduling, stopping criteria, and prompt framing nobody chose deliberately. The harness belongs inside the system under test, the same way agent reliability testing treats the scaffold around a model as part of what gets measured.

Adding distractors produced no clean decay curve

Experiment 2 keeps the sessions that contain each answer and mixes in growing amounts of unrelated conversation history around them. Accuracy does not decline smoothly as distractors accumulate: grep under Chronos with Opus 4.6 rises, dips, and recovers as the session count grows. The retriever ordering stays harness-dependent here too. Across the configurations the paper reports, Claude Code kept favoring grep, Gemini CLI with Gemini 3.1 Pro favored vector throughout, and Chronos showed crossings as sessions were added.

Experiment 2 also covers nine pairs, not ten. Codex CLI with GPT-5.4 is missing from both scaling tables, because the paper reports its vector row as complete only at the full haystack. The pair behind the page’s largest delivery-mode swing is the one absent from the scaling study.

We reproduce the Experiment 1 grid in full because every cell in it was checked against the paper’s published version. For Experiment 2 we report the qualitative shape only, not individual cells. The authors are similarly careful about how far they let the study generalize.

Where the claim stops

The limitations paragraph is unusually direct, and it pre-empts the overclaim the title invites:

Conceptually, our conclusions are tied to long-memory conversational QA: questions are grounded in multi-session chat, explicit time expressions, and personal/user facts. Lexical tools may be disproportionately helpful here because answers often license on verbatim spans; in domains where evidence is rarely literal (e.g., scientific synthesis over paraphrased abstracts, visual-heavy documents, or code semantics), dense retrieval and hybrid routing may look different. We do not claim that grep “beats” vector in general, only that it can win end-to-end under the task distribution and corpora we study.

The critics pressed on exactly this seam. Jerry Liu, LlamaIndex’s co-founder, made the scope argument in a post on X: the tested corpus is per-user conversational memory, while enterprise retrieval mostly means complex questions over a static document corpus of filings, contracts, and operating procedures. LlamaIndex’s written response, published on 26 May 2026, added that the debate “rarely accounts for what agents actually encounter day-to-day in most enterprise settings: unstructured documents (PDFs, Office files, images)”, where there’s nothing to grep until a parser has run. A comment on the Hacker News thread, posted by the user softwaredoug on 9 June 2026, put the operating limit in one line: “In my research grep is fine if you don’t care about tokens and you have less than 100k files.”

Whether LongMemEval stands in for the retrieval workload you actually run is a construct validity question, and for enterprise document corpora the authors’ own scoping says no.

What the paper leaves unreported

Several gaps in the reporting are visible without running anything new.

The sample is 116 questions, so one question moves a cell by 0.86pp, and each cell carries real imprecision on its own: put the 76.7% cell through a Wilson interval at n=116 and it spans 68.3 to 83.5%. That interval says nothing about the gap beside it, though. Both retrievers answered the same 116 questions, so the check that belongs on any single pairing is a paired McNemar test, which looks only at the questions where the two retrievers disagreed. Read the two-question Claude Code gap that way and it is a net difference of two discordant questions, which cannot reach significance at any discordance count. The paper runs no such test. No confidence intervals, seeds, or significance tests appear anywhere in it, so how far a re-run would move each cell is unknown, which is the question eval reproducibility names.

One cell is reported twice, with different values. Table 1 puts Claude Code with Opus 4.6 at 76.7% under inline grep; Experiment 2’s grep table gives the same pair 94.0% in what the paper describes as standard grep over the full haystack, and “standard” is its own word for inline delivery. That is a 17.3pp disagreement on what reads as the same condition, and the paper neither flags it nor explains it. Every 76.7% on this page is Table 1’s figure.

Chronos, the authors’ own harness, takes the higher inline grep score for three of the five models, and loses the other two by under 4pp. A self-built baseline doing that well is the standard thing to flag, whatever the explanation turns out to be. The judge is GPT-4o, scoring binary correctness on outputs from 2026-vintage frontier models, with no reported check of the judge itself.

The paper also scores end-to-end answer accuracy only. Retrieval is never scored on its own axes, so a file-based vector win could come from better passage selection or from the model reading files differently, and the tables can’t say which. Separating those is a stage-split question: score the retriever with context precision and context recall, score generation separately, and attribute each failure to the stage that produced it, the method laid out in how to evaluate a RAG pipeline beyond a single score.

The evidence label matters too. This is a v1 preprint from May 2026, unrevised, unrefereed, with all five authors at one consultancy. Treat every number above as unreplicated.

What to do differently on Monday

The paper supports a narrower set of decisions than its title suggests, and those decisions are still useful.

If your agent searches conversation history, logs, or support transcripts, where answers sit on verbatim spans, and your harness returns results inline, grep-first is now a defensible default. The burden of proof has moved to the embedding index, which must beat a free baseline on your corpus before it earns its sync jobs.

If your results arrive as file pointers, or your corpus is paraphrased, visual, or locked inside PDFs, the paper licenses nothing. Run the comparison yourself with the harness and delivery mode you actually ship, pinned. Pair both retrievers on the same question set, test the difference with McNemar, and report each rate with its interval; which metric belongs to which stage is cataloged in the RAG evaluation metrics hub.

A cheap hybrid is available before any of that: grep first, and query the index only when lexical search comes back empty. That’s a fallback chain at the retrieval layer, the ordered list a system tries when the first hop comes back empty, and it keeps the routing decision visible in the trace. The paper’s hybrid category, agent-driven selection included, is this pattern generalized.

Whichever retriever survives, it feeds the same downstream failure surface, and the miss, drift, and stale-index modes it can produce are the ones the RAG failure-mode taxonomy enumerates. If the index stays, the chunking arithmetic of index size against context budget is deterministic, and the RAG chunk size calculator does it for you.

Grep is all you need on one benchmark, under one delivery mode, for corpora whose answers repeat their evidence verbatim. Everywhere else, the paper leaves a checklist: fix the harness, fix the delivery mode, pair the comparison, and put an interval on the difference before the retriever argument starts.