Agentic evals: check what the agent did, not what it wrote

July 2025, day nine of Jason Lemkin’s vibe coding experiment. Lemkin is the founder of SaaStr. Replit’s agent runs a destructive command against the production database. It does this in the middle of a code freeze that Lemkin had explicitly set, and it wipes the records of 1,206 executives and about 1,200 companies. Then it says a rollback is impossible. It wasn’t. Lemkin did the rollback himself.

Now think about how your usual eval would score that run. A prompt goes in, text comes out, an LLM judge reads the text. The text was tidy and clear, and it even apologized nicely. High score.

That’s the whole point of agentic evals: check what the agent did, not what it wrote. In February 2026 Amazon wrote that it had built thousands of agents since 2025. What it needed for them was one evaluation workflow for the whole company, not another model benchmark.

Where it came from, and what it isn’t

Nobody coined the term. It just stuck. In 2024–2025 teams started putting agents into production and found that MMLU, GSM8K and HumanEval don’t answer their question. NVIDIA put it best in May 2026. A model benchmark checks whether the engine is powerful enough. An agent eval checks how the whole car drives on the road, with planning, tools and uncertainty.

The roots go back further. Princeton’s SWE-bench (October 2023) didn’t read the diff the agent wrote. It ran the repo’s tests after the patch. Sierra’s τ-bench (June 2024) did something smarter. At the end of every conversation with a simulated customer, it compares the database to a goal state, and it runs each task several times. That’s where pass^k comes from: the probability that the agent succeeds on all k runs, not on one of them.

What it isn’t: it’s not a model leaderboard, it’s not an LLM judge reading the last message, and it’s not observability either. Traces are the raw material. The eval is what you expect to find in them.

Three layers in every run

Databricks said it in two sentences in January 2026:

For example, an answer may be correct, but the tool calls leading to it may be inefficient, risky or inconsistent. Evaluating only the final output can hide underlying reasoning failures, while evaluating steps without the outcome may overlook holistic performance.

— Databricks, What is AI Agent Evaluation

So you check three layers:

• Outcome. The state of the world after the run: a database row, a file, a PR, an email that was or wasn’t sent. IBM points out that sometimes there’s no text to check at all, only a record that got updated.
• Trajectory. Which tools, in what order, with what arguments, and what the agent did with what came back. Amazon explicitly measures how accurate tool selection is.
• Envelope. Cost, latency, step count, tokens. Confident AI calls this the operating envelope. An agent that reaches the right answer in 40 steps instead of 6 has a bug, even though the answer is right.

Confident AI also lists the failure patterns that classic evals miss. Retry loops that never converge. Drift away from what the user wanted over the course of the conversation. And false task completion, where the transcript says done and nothing changed. That’s exactly what happened at Replit.

Example: an agent that cancels an order

Say you have a support agent. The task: cancel order 4411 and refund the customer. It calls lookup_order, check_policy, issue_refund. The payments API times out, so it retries, and the refund goes out twice. update_status fails with a 500. And the reply to the customer: “Your order has been cancelled and your money refunded.”

A judge reading the text gives it 5 out of 5. One SELECT on the database finds two refunds and an order that’s still active.

Wait, why run every scenario several times? Because an agent isn’t a function. If a single run succeeds 90% of the time, the chance that five runs in a row all succeed is 0.9^5, about 59%. A pass@1 of 90% looks great on a dashboard. A pass^5 of 59% means that in close to half your scenarios, something fails at least once in five runs. That’s the number your customers actually experience.

And it’s not expensive. 40 scenarios times 5 runs is 200 runs. At 80K input tokens per run, that’s 16 million tokens, about $48 at $3 per million. Less than lunch for the team.

Setting it up without a framework

You don’t need a platform to start. pytest, a fresh database for every run, and the trace your agent already writes are enough:

import sqlite3
import pytest
from my_agent import run_agent, seed_db  # your agent + a fixture loader

TASK = 'Cancel order 4411 and refund the customer'
FORBIDDEN = {'delete_order', 'run_sql'}
N_RUNS = 5  # report pass^5: the scenario passes only if all 5 do

@pytest.mark.parametrize('run', range(N_RUNS))
def test_cancel_order(tmp_path, run):
    db = tmp_path / 'shop.db'
    seed_db(db)                      # fresh state every run
    trace = run_agent(TASK, db_path=db)
    con = sqlite3.connect(db)

    # 1. outcome: check the world, not the reply text
    status = con.execute('select status from orders where id=4411').fetchone()[0]
    assert status == 'cancelled'
    refunds = con.execute('select count(*) from refunds where order_id=4411').fetchone()[0]
    assert refunds == 1              # exactly one, retries included

    # 2. trajectory: right tools, right order, nothing forbidden
    tools = [step.tool for step in trace.steps]
    assert tools.index('lookup_order') < tools.index('issue_refund')
    assert not FORBIDDEN & set(tools)

    # 3. envelope: budget from today's p95 + 50%
    assert len(trace.steps) <= 8
    assert trace.input_tokens <= 60_000

First, fresh state on every run. Otherwise run 3 passes because of what run 2 left behind. Second, the first assert is on the world, not the text. Finally, a hard budget: 8 steps and 60K tokens. Start from your current p95 plus 50%, and tighten from there.

If you want something managed, there’s MLflow 3 on Databricks, AgentCore Evaluations in Bedrock, or DeepEval with @observe in your code. They all work the same way: one trace that serves both observability and evals.

Where it breaks

You still need an LLM judge, for example for tone, or to check whether the agent understood what the user wanted. But the judge is stochastic too. Confident AI recommends calibrating it against a human rubric on a sample of traces, to catch what they call metric green, user red.

My take: For me an LLM judge is the last tool, not the first. If I can write a SELECT that checks the outcome, I write a SELECT. It doesn’t get confused, it doesn’t burn tokens, and it isn’t impressed by apologies.

Two more things break. First, tool mocks go stale. The real API changed a field, the mock didn’t, and the eval is green while production is broken. Run at least some of your scenarios against a real sandbox. Second, goldens rot. Scenarios you wrote six months ago don’t cover what users do today. Pull new scenarios from production every few weeks.

What to try this week

• Pick 10 scenarios from real logs: escalations, complaints, runs that got stuck.
• Give every scenario at least one assert on the state of the world. Not on text.
• Keep a forbidden-tools list per scenario: delete_*, DROP, git push --force.
• Set a step and token budget: your current p95 plus 50%.
• Run each scenario 5 times. Report pass^5 next to pass@1.
• Wire it into CI on every change to the prompt, the tools or the model version.

Replit’s agent would have passed most teams’ evals. Make sure it doesn’t pass yours.