The usual order is to build the capability, test it by hand, add a few test cases, and then — once the system starts failing in production — ask how we actually know whether this works. I would run it the other way around: before building a major capability, define what working well means. A good evaluation system isn't just a way to grade the agent. It turns product expectations into an executable specification.

Start by defining success

Take a support agent that can issue refunds. Saying it "should correctly resolve refund requests" is too vague to build anything on. You have to separate what "correctly" means, because these are distinct properties and each one needs a different evaluator:

  • Outcome: the right refund ends up created.
  • Policy: it respects the returns rules.
  • Tool use: it calls the appropriate APIs.
  • Safety: it never refunds unauthorised amounts.
  • Efficiency: it doesn't make unnecessary calls.
  • Experience: it explains clearly what happened.

There is no single "agent quality" metric, and trying to manufacture one is usually the first mistake.

Evaluate the state of the world, not just the answer

Suppose the user asks to cancel their flight and request a refund, and the agent replies that it has cancelled the flight and processed the refund. Ask another LLM to judge that reply and it will most likely score top marks. But the question that matters isn't how the answer sounds. It's what actually happened:

booking.status == "cancelled"
refund.status == "created"
refund.amount == expected_amount

An agent acts on systems, so the outcome should be measured against the real state of the environment wherever possible, not against what the agent claims to have done. That removes an entire category of false positives in one move.

The right outcome doesn't mean a good execution

Consider two agents that reach exactly the same result. The first calls get_order, then issue_refund, and stops. The second calls get_customer, hits an error, searches search_orders, picks the wrong order, retries, finds the right one and only then refunds. Both end with a correct refund. Operationally they are very different systems: one costs twice as much, takes longer, and has more surface to break on.

So for agents it is worth evaluating the observable trajectory too: which tools it used, with which arguments, how many steps it took, how many errors it produced, whether it looped, whether it took unnecessary actions, what it cost and how long it ran. The key word is observable. I wouldn't build an evaluation system around the model's private reasoning; from an engineering standpoint what's useful is the actions, the tool calls, the state changes and their results.

But be careful demanding an exact trajectory

There's a trap here. Say the test expects search_customer → get_invoice → issue_refund and a new version of the agent works out that it can do get_invoice_by_email → issue_refund instead, with the same outcome in half the steps. Should the test fail? Almost certainly not.

The more capable agents get, the more expensive it becomes to confuse "it followed my expected path" with "it solved the problem". Trajectory constraints should encode what genuinely matters — call issue_refund at most once, verify the order belongs to the user, never call delete_customer, never refund above the limit — and leave everything else free.

Don't use an LLM to check what code can check

Still one of the highest-return rules there is. If you want to know whether the agent called search_customer, read the traces. If a response has to match a schema, validate it. If one date must precede another, write the assertion. If an API must be called exactly once, count the calls. Deterministic graders are fast, cheap and reproducible, and they don't have off days.

LLM judges earn their cost when the question genuinely requires interpretation: whether the response explains correctly why the refund was declined, whether it is grounded in the retrieved information, whether the agent held the policy line when the user tried to talk it out of one. That is where a second model is worth the latency.

The judge needs evaluating too

The moment we use an LLM as a grader we have added another probabilistic system to the loop, and that one gets things wrong as well. Its prompt should be versioned, its rubric explicit, and its decisions compared against human judgement on a regular basis. The interesting part of that comparison isn't the overall correlation but where the two disagree: if the judge passes precisely the cases a human fails, a high correlation buys you nothing.

There's also no need to manufacture precision that isn't there. A 3.7 / 5 implies far more judgement than actually exists behind it. Often PASS, FAIL and INSUFFICIENT_EVIDENCE is both more honest and more actionable. Evals are software, and software needs tests.

Capability is not reliability

Suppose an agent solves a task correctly 90% of the time. That sounds fine. Now put it inside a business workflow that chains ten similar steps: if the runs were independent, the probability of all ten going right is 0.910, around 35%. Independence is a simplification — agent failures tend to be correlated — but the order of magnitude is the point.

There's a large gap between asking whether an agent can do a task and asking whether it does it consistently. That's why I'd run the important cases several times: a batch of five attempts with two failures tells a story a single PASS hides. In critical systems, variance is a metric too.

Don't simulate prompts, simulate users

Another useful shift is moving from prompt-and-response pairs to evaluating whole interactions. Real users answer half the question, change their mind, get things wrong, reject an option, supply contradictory information, abandon a task and come back to it half an hour later.

You can simulate that with a synthetic user on the other side of the agent, which in turn operates against a stateful environment. Someone asks to move their flight to Paris, the agent asks for a date, the user says Friday, the agent offers two options, and the user then decides Saturday would be better. At that point we're evaluating something close to what production looks like. Benchmarks like τ-bench and τ²-bench popularised exactly this format, with an agent, a simulated user and tools interacting across multiple turns. The trap: an overly cooperative simulated user makes the agent look better than it is.

Production should feed the evals

This is probably the most important part of the system. Say a user discovers that when an order contains two items, the agent refunds both even though only one was requested. Fixing the bug shouldn't be the end of it: that case should immediately become a regression case and run on every relevant change from then on.

The full loop runs from production to traces, from traces to failure discovery, then review, then an eval case, then the regression suite, then CI, then back to production. That turns the evaluation dataset into something far more valuable than a static collection of examples: it becomes the institutional memory of the system's failures. Platforms like Braintrust and LangSmith are built around that loop between production traces, datasets and evals.

Separate capability from regression

Not all tests serve the same purpose. When you're developing a new capability you want hard cases, and you want headroom to measure progress: going from 45% to 67% is information. When you've fixed a bug you want the opposite — a case that sits at 100% and never drops again.

A healthy suite keeps both, separated: hard, edge and adversarial cases on one side; on the other, an archive of regressions named after the incident that produced them. Mixing them leads to a common confusion, expecting every eval to have the same difficulty and inform the same decision.

Public benchmarks are useful, but they aren't your product

SWE-bench, WebArena, GAIA and τ-bench are useful for measuring general capability and comparing systems. They also age, and 2026 offered a fairly clear example. On 23 February OpenAI stopped evaluating frontier models on SWE-bench Verified: any model trained on GitHub data after June 2024 has probably seen some of the 500 problems along with their solutions, and a review of 138 hard tasks found material test flaws in 59% of them. They recommended moving to SWE-bench Pro. On 8 July they retracted that recommendation too, after an audit found roughly 30% of its 731 public tasks were broken.

The lesson isn't that public benchmarks are useless. It's that models improve, datasets get contaminated, tests have bugs, and teams end up optimising for the metric. If you're building a support agent, your users don't care where you sit on a leaderboard. They care that it understands the invoice, respects the policy, doesn't lose information, doesn't take the wrong action, and works when they need it. Your production distribution is the benchmark that counts.

How I'd assemble it

I wouldn't start by picking a framework. I'd start by defining the success criteria, building twenty to fifty representative cases, adding deterministic graders, and only then bringing in LLM judges where they're genuinely needed. Then run multiple trials on the critical cases, instrument production, turn real failures into regressions, and put the suite in CI.

Tools come after that: today there's AgentEvals for trajectory evaluation, Promptfoo for declarative evals and adversarial testing, Braintrust or LangSmith for connecting datasets, traces and experiments. But the tool is secondary. The loop is the thing.

Evals are the agent's executable specification

Agents are probabilistic systems, and we are never going to describe their behaviour completely with a collection of unit tests. What we can do is turn the properties we care about into something observable and continuously verifiable, and let the cycle close: product expectations, evals, agent, production, failures, new evals.

A good evaluation system grows with the product. It starts with a handful of carefully chosen examples, takes on hard cases, then simulated users, then real failures. And every significant error that reaches production should leave something behind: a test that makes committing that exact error again harder. That, to me, is the real goal. Not a pretty score, but never debugging the same class of failure twice without noticing.