Agentic RAG: When Naive Retrieval Is Not Enough

⚡What agentic RAG actually is

Naive RAG is a pipeline with no decisions in it. A query arrives, it is embedded once, the vector store returns the top-k nearest chunks, those chunks are pasted into a prompt, and the model writes an answer. Every production RAG system starts here, and for simple factual questions over a clean, well-chunked corpus, this is genuinely enough. Do not let anyone talk you out of it before you have evidence it is failing.

Agentic RAG keeps the same components — embedder, vector store, reranker, generator — but puts a model in charge of how and when they are used. Instead of one fixed pass, the system can rewrite the query, split it into sub-questions, decide retrieval is unnecessary, grade the chunks it got back, retrieve again with a different phrasing, or call retrieval as one tool alongside SQL queries, keyword search, and web lookups. Retrieval stops being a preprocessing step and becomes an action the system chooses to take.

The word agentic is doing real work here, not marketing work. The defining property is a loop with a decision point: after each retrieval, something evaluates whether the current evidence is sufficient, and the system acts on that judgment. A pipeline that always does exactly two retrieval passes is not agentic RAG; it is a longer pipeline. The loop is the thing.

Our RAG development practice→

🧯The failure modes of naive RAG

Teams do not adopt agentic RAG because it is fashionable. They adopt it after cataloguing how the naive version fails on their traffic. Four failure modes account for most of the pain, and each maps to a specific mechanism covered later in this post.

The first is the vocabulary mismatch failure: the user asks about "termination clauses" and the contract says "cancellation provisions," so the embedding similarity is weak and the right chunk ranks below the cutoff. The second is the multi-hop failure: "which customers affected by the pricing change are up for renewal this quarter" requires one lookup for the affected accounts and another for renewal dates, and no single query retrieves both. The third is the silent miss: retrieval returns plausible-but-wrong chunks, the model answers confidently from them, and nothing in the pipeline notices. The fourth is the non-retrieval question: the user asks for a count, a sum, or a comparison across the whole corpus, which is a database query that naive RAG will answer by hallucinating over five random chunks.

Notice what these have in common: the failure happens before generation, and the generator has no way to know. The model receives bad or insufficient evidence and produces fluent output anyway. That is why prompt engineering and better generators do not fix naive RAG failures — the fix has to happen in the retrieval loop, which naive RAG does not have.

🔀Query decomposition and rewriting

The cheapest agentic mechanism to add, and usually the first one worth adding, is a query transformation stage between the user and the vector store. The raw user phrasing is almost never the best retrieval query. People write questions the way they talk, and documents are written the way documents are written; a transformation step bridges the gap before embeddings get involved.

Three techniques cover most of the value. Query rewriting rephrases the question into the declarative language of the corpus — "how do I expense a hotel" becomes "hotel expense reimbursement policy." HyDE-style expansion takes the opposite approach: the model writes a hypothetical answer to the question, and that hypothetical answer — which uses corpus-like vocabulary — is embedded instead of the question. Multi-query expansion generates several paraphrases, retrieves for each, and merges the results with reciprocal rank fusion so a chunk that ranks well under any phrasing survives.

Decomposition is the heavier version, for multi-hop questions. A planning step splits "which customers affected by the pricing change are up for renewal this quarter" into two or three sub-questions with an execution order, each answered by its own retrieval pass, with later sub-questions allowed to reference earlier answers. This is where the architecture starts to look agentic: the number and shape of retrieval calls is decided per question, not hardcoded.

Two warnings from production experience. First, every transformation can corrupt as well as clarify — a rewriting step that drops a constraint ("in Germany", "since March") turns a precise question into a wrong one, so transformed queries should be validated against the original for entity and constraint preservation. Second, transformation adds a model call per query before retrieval even starts, which is latency and cost you pay on every question including the easy ones. Routing — only transform queries that a cheap classifier flags as difficult — keeps that cost proportional to need.

🔁Iterative retrieval: retrieve, read, retrieve again

The defining loop of agentic RAG is the retrieve-read-decide cycle. The system retrieves a first batch of evidence, a model reads it, and then a decision is made: sufficient, retrieve more with a refined query, switch source, or abstain. The loop runs until the evidence is judged sufficient or a step or token budget is exhausted.

This mirrors how a competent analyst actually researches. Nobody answers a hard question from the first five search results; they skim, notice what is missing, and search again with a better query informed by what they just read. The refined second query is usually much better than the first because it contains vocabulary learned from the first batch of documents — the same vocabulary-mismatch problem from the failure matrix, fixed by iteration instead of pre-processing.

The engineering substance is in the stopping condition. Weak implementations loop a fixed number of times or until a vague "is this enough?" prompt returns yes, and both behave badly under load. Strong implementations make sufficiency concrete: every claim the draft answer makes must trace to a retrieved chunk, sub-questions from the decomposition must all have evidence, and the loop exits when the checklist clears. A checklist is testable; a vibe is not.

Budget discipline is the other half. Set a hard cap on loop iterations (two or three is typical), a hard cap on total tokens spent per question, and a fallback answer path for when the budget runs out. Without caps, a pathological question — one whose answer genuinely is not in the corpus — will spin the loop until it costs ten times your median query and still abstains. The cap is not a performance optimization; it is what makes the system an economic product instead of a research demo.

🪞Self-reflection and critique loops

Self-reflection mechanisms insert an explicit grading step into the loop, and they come in two flavors worth separating. Retrieval grading — the pattern popularized by corrective RAG designs — evaluates each retrieved chunk for relevance to the question before generation, drops the irrelevant ones, and triggers a re-query or a fallback source when too little survives. Answer critique happens after generation: a critic model checks the draft for unsupported claims, contradictions with the retrieved evidence, and unanswered parts of the question, and the answer is revised or the loop continues.

The honest assessment of these techniques, as of writing: they help, and the published gains are real on the benchmarks the papers chose, but the magnitude on your corpus is something you must measure yourself. Retrieval grading tends to pay immediately because precision on the evidence is the bottleneck in most failing systems. Answer critique is more situational — it catches confident hallucination and partial answers, but a critic model shares the blind spots of the generator it is checking, and a critic that is too lenient adds cost without catching anything. Treat self-reflection as an eval-driven decision, not an architecture badge.

A practical note on the grading step: it does not need to be a large model. Relevance grading is a classification task with a constrained output — relevant, irrelevant, ambiguous — and small fine-tuned or distilled models, or even a well-prompted mid-tier model, do it well at a fraction of generator cost. This matters because the grader runs per chunk, so it is the highest-volume model call in the whole design.

The failure mode to design for is the critique loop that never converges: generator produces, critic rejects, generator revises, critic rejects again. Bound it the same way you bound the retrieval loop — one revision round, maybe two, then escalate to abstention or a human queue. An answer that fails critique twice is telling you something about the evidence, not about the wording, and more revision will not fix it.

🛠️Retrieval as one tool among several

The most general form of agentic RAG stops treating retrieval as the center of the system. Instead, a tool-using agent — a model in a reasoning loop with callable tools — has vector search available as one tool alongside keyword search, SQL over structured data, internal APIs, calculators, and web search. The agent reads the question, chooses a tool, reads the result, and chooses the next action, ReAct-style, until it can answer.

This is the architecture that finally handles the aggregate-question failure mode. "How many contracts renew in Q3" is answered by the agent choosing the SQL tool and running COUNT over the contracts table — not by hoping five chunks mention a number. "What does our policy say, and is that consistent with the current regulation" uses vector search for the policy and web search for the regulation in the same session. The router that naive RAG lacks is now the agent itself.

Tool design determines whether this works. Each tool needs a crisp schema, a description written for the model (what it is for, when to use it, what it cannot do), and guardrails — read-only database roles, row limits, timeouts, and per-tool budgets. A tool-using agent with a sloppy SQL tool is a SQL injection generator with extra steps. The same discipline that applies to any agent system applies here; the retrieval use case does not exempt you.

Protocols like MCP have standardized how tools are described and connected, as of writing, which lowers the plumbing cost of adding a new source. What they do not lower is the reasoning cost: every additional tool multiplies the agent decision space, and eval quality, not connector count, should drive which tools you add. Start with vector search and SQL; add the rest when failed queries demand them.

The companion piece on securing tool-using agents→RAG over company documents: the build order→

💸The honest cost: latency and token multiplication

Agentic RAG multiplies model calls per question, and any post that does not lead with the arithmetic is selling something. A naive RAG query is one embedding call and one generation call. An agentic query might be: one call to analyze or rewrite the query, one grading call per retrieved chunk (batched if you are careful), one generation call, one critique call, and zero to two re-retrieval cycles each with their own grading. Five to ten model calls where there used to be one is a normal outcome, not a pathological one.

Latency multiplies in series, not in parallel. Calls that depend on each other — rewrite before retrieve, retrieve before grade, grade before generate — cannot be parallelized away, and each model call carries its own queuing and generation time. A naive RAG system that answered in 1.5 seconds becomes a 4 to 10 second system unless you spend real engineering on smaller grader models, streaming the draft while critique runs, and caching at the sub-query level. Users tolerate that for a research assistant; they do not tolerate it for a support chatbot.

Cost per query goes up by roughly the call multiplier, modified by model choice: the grader and rewriter can run on cheaper models, so a well-tuned system often lands at three to six times naive cost rather than ten. But three to six times is still the number to put in front of whoever owns the budget, multiplied by query volume, before the architecture conversation starts. If that cost only buys a quality improvement your users do not notice, the correct decision is to stay naive.

⚖️When the complexity pays — and when it does not

The decision is not "naive or agentic" but "which mechanisms, for which query classes, justified by which measured failure rate." A support bot answering "what is the refund window" over fifty help-center articles should stay naive forever. A research assistant over a million mixed documents, where a wrong answer costs an analyst an hour, earns every mechanism it can measure a gain from.

The table below is the decision frame we use in scoping. The trigger column is the important one: each mechanism is justified by an observed failure rate on your eval set, not by the mechanism being available.

RAG vs long-context models: when retrieval is the wrong tool entirely→What the underlying system costs to build and run→

🧭The migration path from naive RAG

Do not rebuild. Every mechanism in this post bolts onto a running naive RAG system as an independent stage, and the teams that succeed migrate in the order their eval data dictates. The path below is the one we walk clients through, and each step ships to production on its own.

Step zero is the eval harness, and it is not optional. Build a set of 100 to 300 real questions with known-good answers and source documents, score the naive system against it, and classify every failure using the matrix from section two. Without this you cannot know which mechanism to add, and — more dangerously — you cannot prove the mechanism helped after you add it.

Step one is retrieval quality without any agentic behavior: hybrid search, reranking, and chunking fixes. This is unglamorous and frequently closes half the eval gap on its own, which is exactly why it comes first — you want to know which failures survive good retrieval before paying for loops. Step two is query transformation, routed so only hard queries pay for it. Step three is retrieval grading with corrective re-retrieval. Step four, only if structured questions demand it, is the tool-using agent with SQL and API tools.

Run every stage behind a flag, shadow-mode it against live traffic before switching, and keep the naive path as the permanent fallback for when the agentic path exhausts its budget. The migration is finished when the eval set passes at your quality bar and the per-query cost is one your finance team has seen in writing — not when the architecture diagram looks impressive.

Scope a RAG build or rescue with our team→