Agentic RAG in practice: six steps from one search to a capped loop

One. That’s how many searches classic RAG makes, and that search runs before the model has had a chance to think about the question. Panasiti runs a RAG app for pension and tax advisors in Italy, covering INPS, pension funds and legislation. In June 2026 he described exactly this wall:

If the first search misses, the answer is bad and there is nothing the model can do about it.

— Panasiti, How to Build an Agentic RAG with RubyLLM and Rails

So he rebuilt the pipeline as an agent. Three tools, a loop, and the model decides on its own when it has enough material. Rails 8.1, Ruby 3.3, Postgres with pgvector. No LangChain.

That’s basically the whole story of agentic RAG. Nobody invented the term. It just stuck. IBM published an explainer in February 2025, NVIDIA followed in July, and Microsoft made it lesson 5 of 18 in its ai-agents-for-beginners course. The simplest definition: you don’t stuff search results into the prompt. You give the model search as a tool. It calls the tool in a loop, checks what came back, and searches again if it needs to. It’s not a framework or a product. It’s a tool-use loop pointed at your knowledge base.

Here are the steps, in the order I’d do them.

01 · Before the code: 30 questions your RAG gets wrong

Every turn of the loop is another LLM call. So first, prove there’s a problem the loop actually solves. Pull real questions from your logs. Thirty is enough to start. Tag each one by failure type. The sources describe three:

• Vocabulary gap. The Medium example: a user asks “How do I make the graph remember what happened previously?”, and the docs talk about checkpointers, persistence and threads. Not one word in common.
• Multi-part question. The MachineLearningMastery example: “Compare our Q3 2025 sales with Q1 2026 performance and summarize the key risk factors from our latest SEC filing”. One search over the whole sentence returns a salad of chunks that answers none of the parts.
• Cross-document references. A law that points to a section of another law. That’s exactly what Panasiti runs into in the pension domain.

For each question, mark which document holds the answer:

{"q": "How do I make the graph remember what happened previously?", "gold_doc": "persistence.md", "type": "vocab_gap"}
{"q": "Compare Q3 2025 sales with Q1 2026 and list risk factors from the latest filing", "gold_doc": ["sales_q3_2025.pdf", "sales_q1_2026.pdf", "10k_2026.pdf"], "type": "multi_part"}

Now run your existing RAG and check one thing: is the gold_doc in the top-k? If it’s there for 28 of 30, your problem is the prompt or the model, not retrieval, and an agent won’t help. If it’s missing in a third of the cases, you have something worth building.

02 · Three tools, and the description is half the work

Panasiti gave the model three tools: search_knowledge_base, fetch_document_section and list_documents. The search itself is hybrid: vector search through pgvector and the neighbor gem, plus full-text through pg_search with an Italian dictionary. He already had all of that in the single-shot version. He didn’t replace retrieval. He just let the model drive it.

The second tool is the important one. fetch_document_section lets the model see a truncated chunk or a reference to another section, then read the whole section. Without it, agentic RAG is just single-shot that tries a few times.

One more thing: the model picks a tool by its description, not its name. Put three things in it: what comes back (how many chunks, which fields), when to use it, and when not to. “Returns up to 8 chunks with doc_id and section” is worth more than any paragraph about powerful semantic search.

03 · The loop, with a hard cap

This is what Panasiti wrote in Ruby, as a Python version on the Anthropic SDK. Paste it and wire it to your own run_tool, a function that takes a tool name and input and returns a string:

import os
import anthropic

client = anthropic.Anthropic()
MODEL = os.environ.get("RAG_MODEL", "claude-sonnet-4-5")
MAX_STEPS = 6

SYSTEM = (
    "Answer only from the knowledge base. Search before answering. "
    "If results are off-topic, rewrite the query and search again. "
    "Cite doc_id and section for every claim. If nothing fits, say so."
)

def schema(*fields):
    return {"type": "object", "properties": {f: {"type": "string"} for f in fields}, "required": list(fields)}

TOOLS = [
    {"name": "search_knowledge_base", "input_schema": schema("query"),
     "description": "Hybrid search (vector + full-text). Returns up to 8 chunks with doc_id, section, text. Use short keyword queries."},
    {"name": "fetch_document_section", "input_schema": schema("doc_id", "section"),
     "description": "Full text of one section. Use when a chunk is cut off or refers to another section."},
    {"name": "list_documents", "input_schema": schema(),
     "description": "Titles and doc_ids of everything in the knowledge base. Use to check a source exists."},
]

def answer(question, run_tool, log=print):
    messages = [{"role": "user", "content": question}]
    for step in range(1, MAX_STEPS + 1):
        last = step == MAX_STEPS
        resp = client.messages.create(
            model=MODEL, max_tokens=2000, system=SYSTEM, tools=TOOLS,
            tool_choice={"type": "none" if last else "auto"},
            messages=messages,
        )
        if resp.stop_reason != "tool_use":
            log(f"step={step} done forced={last}")
            return "".join(b.text for b in resp.content if b.type == "text")
        messages.append({"role": "assistant", "content": resp.content})
        results = []
        for block in resp.content:
            if block.type == "tool_use":
                out = run_tool(block.name, block.input)[:6000]
                log(f"step={step} tool={block.name} input={block.input} chars={len(out)}")
                results.append({"type": "tool_result", "tool_use_id": block.id, "content": out})
        if step == MAX_STEPS - 1:
            results.append({"type": "text", "text": "Step budget is over. Answer now with what you have and say what is missing."})
        messages.append({"role": "user", "content": results})

Two things in this code aren’t decoration. First, MAX_STEPS. Without a cap, a question that has no answer in the knowledge base sends the model trying phrasing after phrasing. Six steps is my default. Tune it against the eval from step 01. Second, the last step runs with tool_choice set to none and an explicit instruction to answer and say what’s missing. So even when the budget runs out, the user gets an answer, not a timeout.

And remember the math. Every step resends the whole conversation, including the chunks from every previous step. Step 5 pays again for everything retrieved in steps 1 through 4. So put cache_control on the system prompt and tools, and truncate every tool_result, like the [:6000] above.

04 · Evaluate before Generate

In the loop from step 03, the model is supposed to judge on its own whether what came back is relevant. Sometimes it does. Sometimes it just answers from whatever it has. Senevirathna put the fix best on Medium: instead of Question → Retrieve → Generate, do Question → Retrieve → Evaluate → Decide. If evaluate fails, there are four options: rewrite the query, search again, go to a different source, or stop and ask a human.

In practice I pull evaluate out of the main loop into a separate call, with a cheap model and a JSON response. That way you can test it on its own, see when it’s wrong, and swap it out without touching anything else. Here’s the prompt we run, using the checkpointers question from step 01. The result feeds back into the loop: only chunks marked relevant stay in context, and if enough is false, the suggested query becomes the next search.

My take: The grader is the only step here that really earns its cost. A model that searches three times and never checks what it found just produces a more expensive mistake. Build evaluate first, and the loop after.

05 · Show the steps, to users and to yourself

Panasiti streams more than the answer to the browser. He streams the agent’s steps too, via Turbo Streams, and runs the whole thing in a background job outside the request. The user sees what the model is searching for and which section it’s reading, instead of staring at a spinner.

At Bayer it’s even more explicit. PRINCE, an agentic RAG system for preclinical research that the team described on Martin Fowler’s site, was built with a long list of things around the model: orchestration, tool boundaries, state persistence, retries, fallbacks, validation, reflection loops, observability and human review. They write that back then nobody called it harness engineering yet, but that’s exactly what it was.

Log one line per tool call, something like:

step=2 tool=search_knowledge_base query="checkpointer thread_id" chunks=8 kept=3

And put three numbers on a dashboard: average steps per question, the share of questions that hit MAX_STEPS, and the share of chunks the grader threw out. If more than 10% of questions hit the cap, the problem is almost never the cap. It’s the descriptions or the search itself.

06 · Router: not every question gets an agent

Let’s stop for a second. In the eval from step 01, most questions probably passed single-shot just fine. “What is LangGraph’s StateGraph?” doesn’t need a loop. It needs one search, and that’s it.

So put a router at the entrance. Not a model, a rule: a short question with no compare, no two dates and no section reference goes to single-shot. Everything else goes to the agent. If the rule doesn’t hold up, use a cheap classifier. Bayer went further and split the work into steps, each with its own context: research, reflection, writing. Each stage gets only what it needs, not the whole history.

Run the 30 questions through both paths and compare three things: accuracy, latency and cost per question. Agentic RAG should win on accuracy for the hard questions and not lose on the easy ones.

Takeaways

• Start from an eval of questions that fail, not from a framework. If the gold doc is already in the top-k, an agent won’t help.
• fetch_document_section is worth more than another search attempt. Let the model read the document, not just chunks.
• A separate JSON grader before the answer. That’s how you know what entered the context and why.
• A loop without MAX_STEPS isn’t an agent. It’s an open tab.