An agent can produce a clear final answer after a failed run. It may use the wrong record, repeat a tool call, exceed a limit, or report an external change that never happened. A text-quality score will not detect these failures.
Agent evaluation tests a complete process. It measures whether the agent completed the task, followed important constraints, and left the environment in the correct state.
This requires more than a list of prompts. A useful evaluation includes tasks, controlled trials, graders, an evaluation harness, and a suite that represents the work the product must perform.
Define the task and the trial
A task specifies the starting state, user goal, available tools, constraints, and success criteria. A trial is one attempt by an agent to complete that task. Teams run more than one trial because model behavior can vary across attempts.
The agent harness is the system under test. It supplies instructions, context, tools, and controls to the model. The evaluation harness sets up the task, runs the trial, collects the transcript and final state, and applies graders. Keeping these roles separate makes failures easier to locate.
Each task needs a reference solution. The reference solution proves that the task can be completed in the test environment and that the graders can recognize a valid result. It does not require the agent to follow the same sequence.
Realistic tasks can contain incomplete or conflicting information. The user may omit an account identifier, or two records may match a name. The evaluation specification must still state what counts as success. For an ambiguous request, success may require the agent to ask a question and avoid a write.
Include positive and negative cases. Test when the agent should call a tool and when it should not. Test when it should act, ask for approval, abstain, or escalate. A suite of straightforward success cases does not measure restraint.
Grade outcomes and important constraints
Outcome graders test what the agent produced or changed. Did the requested behavior work? Did the correct record change once? Does the artifact contain the required information? When code or environment state can prove the result, use a deterministic grader.
Trajectory graders inspect the sequence of model requests, tool calls, approvals, and observations. Use them for constraints that affect safety or cost. Examples include a prohibited action, a missing approval, access to the wrong tenant, repeated side effects, or an excessive number of calls.
Do not require an exact preferred trajectory when other valid methods exist. Exact tool-call matching makes a test brittle and can penalize a better strategy. Grade the path when the path itself matters. Grade the result when different paths are acceptable.
Some outcomes require expert judgment. A human or model grader can assess whether a research conclusion follows from its sources or whether an escalation contains enough evidence. Use a narrow rubric. Calibrate model graders against domain experts, allow an unknown result, and inspect disagreements.
Human graders can disagree too. Record the rubric, reviewer expertise, and agreement rate for difficult tasks. Do not call one reviewer an objective ground truth when the domain permits reasonable differences.
Keep critical gates separate from weighted quality scores. A strong explanation should not offset an unauthorized action. Report task success, constraint violations, latency, and cost as distinct results.
Measure success across repeated trials
A single successful run does not show reliable behavior. Run each important task enough times to estimate both capability and consistency.
Pass@k asks whether at least one of k trials succeeds. It helps measure whether the agent can solve a task when it receives several attempts. Pass^k asks whether every one of k trials succeeds. It measures consistency, which matters when users expect the first run to work.
These metrics answer different questions. A research prototype may value pass@k because one valid solution is useful. A production action may require high pass^k because variable results create support and safety costs.
Report the number of trials and uncertainty with each rate. Record model settings and any seed the provider exposes, but do not assume that a seed makes hosted inference fully reproducible. Provider infrastructure and model versions can still change behavior.
Measure the tails as well as the average. Track the longest runs, highest costs, repeated calls, and timeouts. An agent with a good average can still enter rare loops that make it unsuitable for production.
Use capability and regression suites
A capability suite measures the edge of what the agent can do. It should contain difficult tasks and leave room for future improvement. If every agent scores near 100 percent, the suite no longer separates stronger systems from weaker ones.
A regression suite protects behavior that should already work. It includes past failures and core product requirements. The target should be close to perfect for deterministic controls and high for model-dependent tasks.
Production failures should become regression cases after review. Recreate the decision-relevant conditions with governed or synthetic data. Preserve the ambiguity, permissions, tool behavior, and state transitions that caused the failure.
Do not assume that permission to collect an operational trace makes it eligible for model training or permanent evaluation storage. Apply the product's consent, contract, retention, and account-tier rules. Use a synthetic reconstruction when it can preserve the test without private details.
Maintain slices by task class, risk, tool, language, and environment. A single average can hide a severe failure in a small but important group. Keep hard gates for boundary violations even when the overall score improves.
Assign an owner to every suite. The owner removes obsolete tasks, adds new failure modes, checks for saturation, and reviews changes to graders. Evaluation data needs maintenance because tools, policies, and user behavior change.
Check that the evaluation measures the intended behavior
Agents can exploit gaps in tasks and graders. An agent may satisfy a string check without completing the work. It may read hidden answer data or find a shortcut that the test designer did not intend. A passing score does not prove a valid evaluation.
Inspect transcripts from passing and failing trials. Confirm that passing agents solved the task and that failing agents deserved the result. Check whether the reference solution still works and whether the grader rejects plausible but incorrect outcomes.
Keep task data and grader secrets outside the agent's accessible environment. Test for leakage through files, tool responses, environment variables, and prompt construction. When an agent finds a loophole, fix the evaluation before using its score for a release decision.
Avoid repeated tuning against one small suite. Separate fast development tests from held-out release tests. Add new tasks over time and review whether public benchmark material could have entered model training data.
Version the full evaluation setup: task fixture, model, agent harness, tools, retrieval data, sandbox image, grader, and thresholds. A score without these versions cannot support a reliable comparison.
Release thresholds should match the consequence of failure. A drafting assistant can tolerate variation in wording. An agent that changes customer or production state needs strict gates for identity, authorization, duplicate prevention, and confirmed effects. Set these gates before reviewing the candidate release so the standard does not move to fit the result.
Classify failures by the component that can fix them. Useful categories include missing context, wrong tool choice, invalid arguments, policy rejection, environment error, weak planning, premature completion, and grader disagreement. A stronger model will not fix a broken permission check. A longer prompt will not fix stale tool data. This classification turns an evaluation result into an engineering action.
Track cost and latency beside success rather than after it. A change may improve completion while doubling tool calls or human review. Product teams need the full result to decide whether the improvement is useful in the intended setting.
Connect offline evaluation with production evidence
Offline tests provide repeatability and safe testing of dangerous cases. Production monitoring provides realistic task distribution, integrations, interruptions, and user corrections. The two forms of evidence support different decisions.
Use offline suites before release. Then use limited rollouts, A/B tests, user feedback, and human review to check whether the gain transfers to real use. Production monitoring often lacks a known correct answer, so completion signals and user acceptance should not be treated as perfect graders.
When production evidence shows a new failure, create a controlled task and verify that it reproduces the problem. After the fix, run the full regression suite and monitor the guarded rollout. This process links an observed problem to a testable change.
Keep representative trials available to the engineering team under the applicable data rules. Summary metrics identify a regression. The transcript, grader result, and final environment state show what failed.
Agent evaluation measures a model-directed process against a defined task and controlled environment. It tests outcomes, required constraints, and consistency across trials. The strongest evidence comes from executable state checks, calibrated graders, manual transcript review, and production validation used together.