Answers · Updated August 17, 2026
How do you evaluate an AI agent?
AI agent evaluation tests the complete system—not only the model response—against representative tasks with written success criteria. A useful evaluation records the task, trial, trace, tool calls, final environment state, grader versions, latency, cost, and human intervention; repeats non-deterministic cases; treats identity, authorization, safety, and duplicate-effect failures as mandatory gates; and turns reviewed production failures into regression tests.
Task, trial, trace, grader, and outcome
Start with a task that two qualified reviewers can interpret the same way. Name the eligible input, available tools and data, permitted and prohibited actions, expected result, and evidence that proves success. A task is one test case; a trial is one run of that task. Because model behavior varies, reliability comes from repeated trials, not a single impressive transcript.
The trace records the observable path: model and retrieval calls, tool proposals, policy decisions, approvals, provider responses, retries, state transitions, and final outcome. The outcome is the authoritative environment state—not the agent saying that it succeeded. For a booking agent, the result is the correct calendar or scheduling record; for a service agent, it may be a resolved ticket with the required evidence and no unauthorized action.
Anthropic’s agent-evaluation guidance uses the same useful distinctions and recommends deterministic graders where possible, model graders where necessary, and human graders for validation. It also distinguishes capability evaluations from regression suites and describes repeated trials for non-deterministic behavior. Those patterns are implementation guidance, not a universal certification standard.
Evaluate at six connected levels
A model benchmark cannot prove an agent selects the right customer record, uses the permitted tool, handles a timeout, or leaves the destination correct. Build evidence from the smallest component through the full business operation.
| Level | Decision question | Evidence |
|---|---|---|
| Model response | Did one response satisfy a bounded content criterion? | Exact match, structured fields, support, style, refusal, calibrated rubric |
| Retrieval and memory | Did the agent use eligible, current, permitted evidence? | Source and permission IDs, retrieval assertions, citations, stale and poisoned-source cases |
| Tool call | Did it choose the right action with valid arguments? | Tool identity, schema, resource, arguments, policy decision, denial, provider response |
| Trajectory | Did the sequence remain efficient, bounded, and policy-conformant? | Trace, turns, loops, retries, approvals, state transitions, stop reason |
| Environment outcome | Did the authoritative external state reach the accepted result? | Destination read-back, provider receipt, duplicate check, reconciliation |
| Operation | Does the deployed system stay acceptable across real use and change? | Segmented outcomes, failures, corrections, latency, cost, human work, incidents, versions |
Capability and regression sets have different jobs
Capability cases probe work the agent does not yet handle reliably and provide a hill to climb. Regression cases protect behavior that must not be lost. As a capability stabilizes, preserve representative cases in the regression set. Keep versioned task, environment, evaluator, source, model, prompt, tool, and policy identities so a score change can be investigated instead of guessed.
Choose the grader that matches the assertion
Do not ask a language model to decide whether a database row exists when code can verify it. Do not use an exact string check for an open-ended response whose quality depends on support, completeness, or tone. Most useful suites combine grader types.
| Grader | Best use | Strength | Limitation |
|---|---|---|---|
| Deterministic | Schemas, calculations, state, permissions, tool arguments, exact outcome, duplicates | Fast and reproducible | Cannot fully judge open-ended quality |
| Model-based | Grounding, completeness, tone, policy interpretation, complex quality rubrics | Scales qualitative review | Needs calibration, versioning, cost, and disagreement review |
| Human | High-consequence judgment, subjective quality, evaluator calibration, novel failures | Contextual expert ground truth | Slower, costlier, and subject to reviewer variation |
| Production outcome | Bookings, records, resolutions, accepted work, corrections, downstream value | Closest to the business result | Arrives later and can be confounded without a baseline |
Model-based grading needs a named model and version, a versioned rubric, examples, threshold, cost and latency record, and periodic comparison with qualified human decisions. Inspect disagreements and traces. A shared blind spot between agent and evaluator can create a precise-looking score that does not match the business result.
AWS’s 2026 agent-evaluation framework likewise evaluates the complete agentic system, including tool selection, multi-step behavior, final task completion, responsibility, performance, and cost. Provider tools can help run a suite, but the workflow owner still defines the task, acceptance threshold, mandatory failure, and release decision.
Mandatory AI agent release gates
A weighted average can hide a catastrophic failure. Mark identity, authorization, sensitive-data, prohibited-action, approval, duplicate-effect, unknown-state, and recovery cases as mandatory where they apply. One mandatory failure blocks the affected authority even when ordinary response quality is high.
| Gate | Adverse case | Required behavior |
|---|---|---|
| Identity and tenant | Wrong, ambiguous, stale, or cross-tenant record | Stop before retrieval or action; no data crosses the boundary |
| Authorization | User or agent lacks permission | Execution is denied by deterministic policy outside the model |
| Consequential approval | Required decision or approval is absent | Action stays pending or denied; no self-approval |
| Duplicate effect | Request, callback, or retry is replayed | One external effect and one recorded result |
| Unknown provider state | Timeout occurs after a possible action | Reconcile before retry; preserve truthful unknown state |
| Sensitive information | Prompt, retrieval, tool, or trace attempts exposure | No unauthorized output; evidence is minimized and incident path works |
| Prompt or retrieval injection | Untrusted content contains instructions | Content remains data; tool and policy boundaries still hold |
| Recovery | Dependency fails after partial progress | Contained state, owned exception, verified rollback or safe resume |
Measure repeated-trial reliability honestly
Record how many trials ran and how many passed for each task. pass@k asks whether at least one of k attempts succeeds and is useful when several attempts are acceptable. Pass^k asks whether every attempt succeeds and is more relevant when a customer needs consistent behavior. Do not report the more flattering measure without explaining what it means, and do not assume independent trials when shared state or caching makes them dependent.
Build an AI agent evaluation program in eight steps
- Write the operating contract. Define the task, users, records, tools, authority, accepted outcome, exclusions, owners, and business baseline.
- Create representative cases. Include normal, difficult, denied, missing-data, stale-data, conflict, injection, outage, replay, and recovery conditions.
- Prove the reference path. Confirm that a known-good implementation or human operator can satisfy every task and grader.
- Assign graders. Use deterministic state and policy checks first; add calibrated rubrics and human review where necessary.
- Set mandatory gates and thresholds. Separate release-blocking failures from weighted quality and operating goals.
- Run repeated trials. Preserve configuration, traces, outcomes, grader versions, cost, latency, and intervention for every attempt.
- Issue a bounded release decision. Name the cohort, authority, observation window, monitoring, escalation, expansion rule, and stop condition.
- Maintain the suite. Review production corrections, incidents, changes, and new requirements; sanitize evidence and add durable regression cases.
The NIST AI Resource Center supports testing, evaluation, verification, and validation under the voluntary AI Risk Management Framework. The NIST Generative AI Profile adds generative-AI risk considerations. Select controls for the actual context and document which framework version informed the work; neither source certifies a local evaluation or replaces qualified legal, security, privacy, accessibility, or sector review.
Download the AI agent evaluation plan
The blank CSV covers 22 dimensions: task completion, eligibility, tools, arguments, authorization, identity, grounding, abstention, injection, sensitive data, approval, idempotency, timeouts, recovery, handoff, quality, consistency, latency, cost, regression, production feedback, and evaluator calibration. Add the exact test case, expected outcome, grader, mandatory flag, trial count, threshold, owner, status, and evidence link for your deployed system.
The template is licensed under CC BY 4.0. It is a planning aid, not an audit opinion, benchmark, certification, warranty, or claim that any system is safe, reliable, compliant, or production ready.
From evaluation to a controlled release
Use the AI agent observability guide to connect evaluation results with production traces and business receipts, the AI agent security checklist for adverse authorization and tool-use cases, and the AI pilot program guide to define the cohort, observation window, economics, expansion rule, and stop decision. For retrieval-backed agents, Cognautic’s AI knowledge management services cover source authority, permissions, citations, correction, and maintained operation.
Retrieval evaluations should also cover poisoned or unauthorized content. The OWASP vector and embedding weaknesses guidance describes access, leakage, and poisoning risks for retrieval systems. Apply the scenarios to the exact architecture rather than assuming a framework or vector store supplies the necessary controls automatically.
People also ask
What is AI agent evaluation?
AI agent evaluation is the repeatable process for testing whether an agent completes a defined task correctly, safely, consistently, and within operating constraints. It evaluates the model together with prompts, tools, retrieval, memory, policy, permissions, orchestration, and the external environment. The result should identify what passed, what failed, and whether the exact configuration may be released.
What metrics should be used to evaluate an AI agent?
Start with confirmed task completion, then measure tool and argument accuracy, groundedness, permission and policy compliance, handoff quality, repeated-trial reliability, latency, cost, corrections, and human intervention. The right quality rubric depends on the job. A fluent final answer is not success when the external booking, record, message, or transaction is wrong or missing.
How many test cases does an AI agent need?
There is no universal number. Begin with enough cases to cover the important normal, edge, denied, adverse, outage, replay, and recovery behaviors, then grow the set from reviewed failures and new requirements. A small precise set with known expected outcomes is more useful than hundreds of ambiguous prompts. Use repeated trials for variable behavior rather than treating one pass as reliability proof.
Can an LLM grade another AI agent?
Yes, for qualities that deterministic code cannot fully express, but the grader needs versioned criteria, representative examples, calibration against expert judgment, and monitoring for drift and disagreement. Use deterministic outcome, schema, permission, and state checks wherever possible. A model grader should supplement rather than override mandatory safety and business-state assertions.
What is the difference between agent evaluation and monitoring?
Evaluation scores behavior against defined tasks and criteria. Monitoring detects operating conditions such as errors, latency, cost, missing traces, or quality changes in production. Tracing supports investigation, and authoritative destination records prove external outcomes. A dependable operation connects all four rather than treating a dashboard as proof that the agent works.
When is an AI agent ready for production?
An agent is ready for a bounded release only when the named owners accept the evidence: every mandatory case passes across the required repetitions; identity, authorization, safety, rollback, and duplicate-effect controls work; unresolved limits are documented; monitoring and human escalation operate; and the release has an explicit cohort, observation window, expansion rule, and stop condition.
Rather not DIY?
Want an evaluation plan tied to a real business outcome?
If you’d rather have someone build this for you, that’s what we do. Start with a free consult — we map your workflows and name the smartest first move. No pitch, no pressure.