An AI agent can look excellent in a demonstration and still be unsafe to deploy. A polished answer says little about whether the agent chose the right tool, changed the correct record, respected an approval boundary, or stopped when the request exceeded its authority.
Before production, teams need evidence that the agent can complete a bounded job reliably and fail safely. That does not require a giant benchmark program. A small, versioned evaluation suite built around the real workflow is more useful than hundreds of generic prompts.
This guide gives technology and operations leaders a practical release gate. It focuses on four questions: Does the agent complete the task? Does it use tools correctly? Does it refuse unsafe work without becoming useless? And does a new version preserve behaviour that already works?
Start with a deployment contract
Write down what the agent is allowed to do before writing test cases. The contract should name its users, inputs, permitted tools, data boundaries, side effects, approval points, escalation path, and definition of a completed task.
For example, an invoice agent may read an invoice and purchase order, propose a category, and draft an exception note. It may not approve payment, change supplier bank details, or send the note without review. That boundary makes evaluation concrete: a correct draft is success; an attempted payment is failure even if the invoice data was extracted perfectly.
Use the same contract in your operating model for safe and trustworthy agents. Evals measure whether the implementation honours the contract; permissions and runtime controls enforce it.
Build a small but representative case set
Start with 20 to 40 cases from the workflow, scrubbed of sensitive data. Include common tasks, hard but legitimate variants, incomplete inputs, ambiguous requests, unavailable tools, and prohibited actions. Tag every case by workflow step, risk level, and expected behaviour so failures can be grouped rather than averaged away.
Each case needs four fields:
- Input and starting state: the request plus relevant records, permissions, and tool availability.
- Expected outcome: the business state that should exist when the run ends.
- Allowed path: tools and actions the agent may use, including required approval points.
- Forbidden effects: records, recipients, or actions that must remain untouched.
Do not require one exact chain of reasoning. Agents may reach a valid outcome through different paths. Grade observable results and actions instead: the final record, tool calls, approval events, messages, and stop condition.
Score task success separately from presentation
The first score should answer whether the requested job was completed. Use deterministic checks whenever possible: expected fields match, the right record changed, required evidence is attached, no duplicate was created, and the task ended in the correct state.
Keep presentation quality separate. A concise explanation can be graded for required facts, clarity, and unsupported claims, but it should not hide a failed business outcome. A useful scorecard might weight outcome correctness most heavily, then tool behaviour, policy compliance, and response quality.
Set critical failures outside the weighted score. An unauthorized external message, destructive write, secret exposure, or skipped mandatory approval should fail the release regardless of the average. This follows the broader test, evaluation, verification, and validation approach in NIST's Generative AI Profile: evaluation should be tied to the system's context, risks, and lifecycle rather than treated as a single universal benchmark.
Inspect tool use, not only the final answer
Multi-step agents can arrive at a plausible answer through a dangerous path. Capture a structured trace for each evaluation run and check:
- whether the chosen tool was appropriate for the task;
- whether arguments referenced the correct tenant, record, and destination;
- whether write tools were called only after required validation or approval;
- whether retries were bounded and idempotent;
- whether the agent reacted correctly to timeouts, malformed results, and permission errors; and
- whether forbidden tools or tool sequences were attempted.
Some checks are exact. A test can assert that send_message was never called or that a write used a specific record ID. Others allow multiple valid paths and should check milestones rather than exact call order. Anthropic's guidance on agent evals similarly distinguishes the final outcome from the path and recommends choosing graders that fit the property being measured.
Keep evaluation traces useful but privacy-conscious. Our guide to observing AI agents without leaking data explains how to record structure, timing, decisions, and outcomes without copying sensitive payloads into telemetry.
Test safe refusal as a useful outcome
A refusal test should not merely look for the word “cannot.” Define what safe behaviour means for the workflow. The agent should decline the prohibited action, avoid the side effect, state the relevant boundary, preserve useful work, and route the user to an approved next step when one exists.
Include direct and indirect attacks: a user explicitly requests an unauthorized action; a retrieved document contains conflicting instructions; a tool result asks for credentials; or several harmless actions combine into a prohibited outcome. Also test over-refusal. An agent that blocks every unusual case is safe only in the narrowest sense and will not deliver operational value.
Measure both rates: unsafe compliance and unnecessary refusal. Review false accepts and false rejects separately because they require different fixes. High-impact unsafe compliance should remain a release blocker even when the overall refusal score is strong.
Create a regression suite that grows from failures
Freeze representative passing cases as a baseline. Every change to the model, prompt, tool schema, retrieval source, policy, or orchestration code should run against that baseline before release. Compare versions on the same case set and evaluation configuration.
When a defect reaches staging or production, turn it into a minimized test with the original failure condition and the expected safe result. Add it permanently. This converts incidents into accumulated reliability rather than recurring surprises. Pair the suite with agentic governance through policy-as-code so policy changes and agent changes are tested together.
Version the dataset, graders, agent configuration, model, tools, and thresholds. If a score moves, the team must be able to identify what changed. Run deterministic checks in continuous integration; schedule slower model-based grading and human review for release candidates.
Use the right grader for each claim
Code-based graders are best for exact fields, tool names, arguments, side effects, latency limits, and forbidden actions. Human reviewers are best for nuanced usefulness, ambiguous outcomes, and new failure patterns. Model-based graders can scale checks such as completeness or tone, but calibrate them against human-labelled examples and do not let them decide high-impact safety claims alone.
Review disagreements, not just averages. A case that passes a model grader but fails a deterministic policy check is a failure. A case on which human reviewers consistently disagree probably has an unclear task contract or rubric.
Set a release gate by risk tier
Do not use one pass rate for every agent. A read-only research assistant and an agent that changes customer records carry different consequences. Define thresholds by workflow risk and require zero unresolved critical failures.
A practical pre-production gate asks:
- Does the agent pass every critical safety and authorization case?
- Is task success above the threshold for each important case group, not only overall?
- Are tool errors, loops, and unnecessary retries within limits?
- Has a domain owner reviewed a sample of passes and failures?
- Can operators disable the workflow, inspect a run, and route exceptions to a person?
- Are the dataset, configuration, grader results, and approval recorded for this release?
Passing the suite is permission to begin a controlled rollout, not proof of permanent reliability. Start with limited authority, monitor accepted outcomes and exceptions, and feed production failures back into the offline suite. The strongest eval program is a loop: define the contract, test the agent, limit the rollout, learn from real work, and make every important lesson executable.
