Your agent evaluation is only as useful as the decision it changes

An impressive score can hide an unusable workflow

Suppose two versions of an agent score 91 and 94 on an internal evaluation. The newer version sounds clearer, follows the requested format more often, and receives better ratings from a model judge. It also starts granting a refund before checking eligibility. If that error barely affects the average, the score can recommend the wrong release.

The first question for an evaluation is what decision it supports. A model-selection test, a release gate, and an investigation into a production failure need different evidence. Combining them into one number can make the dashboard easy to read while making the decision harder.

For an action-taking agent, define success in terms of what the business and user need to be true. If the task is to reschedule an appointment, inspect the booking system, the customer’s latest requirements, and any unwanted changes. A strong final message is only one part of that result.

Anthropic’s January 2026 guide distinguishes the record of an agent’s actions from the resulting state of the environment. That distinction matters because an agent can describe a successful booking without creating one. The guide also treats repeated trials as part of evaluating variable behaviour. [1]

The aim is to make important failures hard to hide. A readable answer should not compensate for a wrong recipient, an unauthorized action, or a missing transaction.

Separate the evidence before combining the score

Start with a small number of outcome categories. Was the requested work completed? Were the required rules followed? Were unrelated records left intact? Did the user receive an accurate explanation? How much time and money did the run consume?

Some checks can be exact. A database row either contains the expected account identifier or it does not. Other checks require judgment, such as whether an explanation addressed the customer’s concern. Use each method for the kind of evidence it can assess.

A model judge can help review language at scale, but it should be calibrated against human decisions. Give reviewers a clear rubric and inspect disagreements. If experts disagree about whether an answer is acceptable, the team may need to clarify policy before asking another model to score it.

Avoid letting a persuasive explanation stand in for missing evidence. A judge evaluating a tool-using agent should see the relevant tool results and final state, with access limited to what the grading task requires. Otherwise, it may reward an accurate-sounding account of an operation that never happened.

Release rules should preserve the importance of specific failures. A critical permission error may block promotion even if average task completion improves. Less consequential style issues may be tracked without blocking a release. The client and product owner need to agree on that distinction.

This also makes evaluation results more useful to coding agents. “Failed because the booking kept the old date after a correction” is an actionable report. “Helpfulness declined by two points” leaves the repair almost entirely unspecified.

A realistic test needs a credible user and environment

An evaluation can be mechanically correct and still test an unrealistic world. A simulated customer may volunteer information that a real customer would never provide. A mocked integration may always return a complete record immediately. The agent then succeeds under conditions the deployed product rarely encounters.

The tau2-bench research addresses part of this problem by connecting the simulated user’s actions to a shared environment. This makes coordination testable: the user’s side of the task has consequences rather than existing only as dialogue. It offers a benchmark design to learn from, not proof that any simulator will predict production outcomes. [2]

T1-Bench, introduced in a June 2026 preprint, broadens evaluation to interleaved scenarios across 25 domains and combines automatic assessment with human judgments. Its design highlights another testing problem: customers do not always keep each task inside the clean boundary chosen by the developer. [3]

A client-specific test can capture this without recreating the entire business. Model the small part of the environment that determines whether the task succeeds. For an exchange, that might mean orders, inventory, eligibility, and shipping status. Add complexity only when it changes a decision or exposes a known failure.

Keep the simulated user from seeing the answer key. If the user model knows exactly which action the agent must take, it may guide the agent toward success. Separate what the customer would know from what the grader needs to know.

Then compare simulated behaviour with reviewed real cases. Do customers provide shorter answers? Do they abandon the conversation sooner? Do they confuse product names? These differences should inform the simulation, with privacy protections applied before real material enters the workflow.

Protect the test set from the improvement process

Once engineers repeatedly inspect failures and adjust the system, those cases become development material. A rising score on them still matters, but it no longer provides an independent estimate of performance on unseen work.

Keep a separate set of reviewed cases for release decisions. Split related examples together so a lightly rewritten version of the same incident does not appear on both sides. For time-sensitive workflows, include a later period of data to test whether improvements transfer beyond the original sample.

Repeated trials matter when the same case can produce different results. Report both the average outcome and the instability of important cases. A workflow that succeeds once after several attempts may be useful for research, yet unsuitable for an action the customer expects to work on the first attempt.

Make comparisons under the same conditions. Use the same tool permissions, time budget, data snapshot, and scoring rules. If you change several components at once, record that fact and avoid attributing the improvement to one prompt.

Small differences deserve caution. When a handful of cases decides which version wins, inspect those cases and estimate uncertainty before making a broad claim. The evaluation should help the team recognize when there is insufficient evidence, as well as when there is a clear improvement.

Finally, check whether better offline scores are followed by better live outcomes. Track a limited rollout against the measures the business actually cares about, such as completed work, repeat contact, and repair effort. If the relationship is weak, revise the evaluation.

An evaluation suite is part of the product’s understanding of its job. When the job changes, the suite needs review. Keeping an old score stable is not useful if it no longer represents the customers being served.

Make evaluations executable specifications for people and agents

The strongest tooling opportunity is a shared test package that describes a starting state, a user goal, allowed actions, and verifiable outcomes. Human developers could review it, while coding agents could run it directly and inspect structured failure evidence.

A future evaluation system could also flag weak graders, duplicated scenarios, and cases that no longer resemble live traffic. It could show which important workflows have little evidence, rather than presenting a high average as broad coverage. Existing evaluation and tracing tools provide parts of this workflow, but their usefulness still depends on the cases and acceptance rules supplied by the team.

The next step is to connect each result to a development decision: repair this integration, clarify this policy, collect this missing scenario, or hold this release. That would turn evaluation from a periodic scoring exercise into a working agreement between business expectations and the people and agents building the system.