Evaluating AI Agents: How to Know If Yours Actually Works
A demo shows one good run. Here is how to test reliability across repeated runs, choose graders and decide when an agent is not ready to ship.

You know an AI agent works when it completes a defined job correctly on realistic tasks, run after run, without breaking the rules you set, at a cost you can sustain, and when that evidence is re-checked every time the model, prompt or tools change. A successful demo shows none of that. It shows one good run, usually on an input the builders already knew would work.
That opening sentence is how to evaluate AI agents in short. The rest is method. You define what done means in terms of the state of your systems rather than the agent’s final message. You build a small set of real tasks with clear pass and fail criteria. You run each task several times, because agent behavior can vary between runs. You choose graders that match each criterion (code for what can be checked mechanically, a calibrated model for judgment, people for the calls that matter most). Then you turn the results into a release gate and keep feeding production failures back into the test set.
The gap between that method and common practice is still wide. In TestMu AI’s State of AI in Quality Engineering 2026 report, published on 2 November 2026, 37% of teams that build AI agents have no systematic way to evaluate them. This guide covers what to measure, how to grade it, which tests should block a release, and when the results say an agent is not ready.
Why does a working demo prove so little?
A demo proves that an agent can succeed once. Production asks whether it succeeds every time, on inputs nobody rehearsed, with tools that return errors. Because the model at the center of an agent can produce different outputs from the same input, a single run is a sample of one, and agents that look finished in a demo often fail on later attempts at the same task.
The research record on this is consistent. The τ-bench benchmark, published in 2024, simulated customer conversations with agents that were given domain-specific API tools and policy guidelines. Even the strongest function-calling agents tested, including GPT-4o, succeeded on fewer than half of the tasks, and on retail tasks the chance of succeeding on all of eight repeated attempts fell below 25%. A model that looks competent on one run can be unreliable on the next seven.
Practice has not caught up. Measuring Agents in Production, a study of agent developers across 26 domains, found that 74% of deployed agents depend primarily on human evaluation, and that reliability, meaning consistent correct behavior over time, is the top development challenge. TestMu AI’s survey of its testing community, published in November 2026, points the same way: among teams building agents, 63% use human review to evaluate them, and 24% call an agent ready on a demo or nothing formal. The survey covers TestMu AI’s own community rather than a representative sample, but the direction matches the academic data.
Human review belongs in any serious evaluation. Unstructured review of a few conversations gives an impression of quality. To measure reliability, human graders can score representative tasks against a written rubric and repeat the important cases.
What does it mean for an AI agent to “work”?
An agent works when it reliably produces the right end state for a specific job, under the rules that job carries. That definition has three parts, and each one changes how you test. “End state” means you check your systems, not the agent’s message. “Specific job” means you evaluate one defined workflow at a time. “Rules” means some ways of reaching the right answer still count as failures.
Start with the job. “Evaluate our support agent” is too broad to test. “Process refund requests under 100 euros for verified customers, and escalate everything else to a person” is testable: it tells you which inputs to collect, which tools the agent may call and which outcomes count as failures.
Then define success in terms of the environment. Anthropic’s engineering guide, Demystifying evals for AI agents, uses a flight-booking example: the agent may say the flight is booked, but the outcome is whether a reservation exists in the database. A CRM agent’s success is the updated record, not a message claiming it.
Finally, write each criterion so that two domain experts would reach the same verdict independently. Anthropic’s guide uses exactly that test, and it is the most useful filter we know. If your head of support and your compliance lead would disagree about whether a reply passes, the task produces noise instead of signal. Fix the specification first.
How to evaluate AI agents: what should you measure?
Measure the outcome first, then the constraints the agent must respect on the way, then consistency, efficiency and safety. Keep these as separate scores. A single blended number hides exactly the failures that matter, because a high task-completion rate cannot compensate for one leaked customer record or one unapproved payment.
| Dimension | The question it answers | Typical check |
|---|---|---|
| Outcome correctness | Did the systems end up in the right state? | Code compares records, files or tickets with the expected state |
| Rule adherence | Did it respect approvals, limits and forbidden actions? | Code inspects the tool calls in the trace |
| Consistency | Does it succeed every time, or only sometimes? | The same task run several times, reported as a pass rate |
| Escalation | Does it hand off when it should, and only then? | Paired cases where escalation is and is not correct |
| Efficiency | How many turns, tool calls, tokens and seconds did it take? | Metrics from the trace |
| Cost per successful task | What does one correct result cost? | Total model, tool, human review and correction costs across all attempts, divided by successful completions |
| Safety and security | Can inputs, documents or tool results push it off course? | Adversarial cases, including prompt injection |
| Interaction quality | Was the reply clear, accurate and on brand? | A calibrated model grader or human review against a rubric |
Teams most often get two of these wrong: the path and the cost.
The first is the path. It is tempting to require an exact sequence of tool calls. Anthropic’s guide warns that this produces brittle tests, because agents regularly find valid approaches that evaluation designers did not anticipate, and it recommends grading what the agent produced rather than the path it took. We agree, with one firm exception: constraints. If policy says identity must be verified before a refund, the trace must show the verification step before the refund call, whatever else the agent did. Grade the outcome freely and grade the constraints strictly.
The second is cost. An expensive agent that succeeds 95% of the time can beat a cheap one that succeeds 80% of the time and creates manual rework on the rest. Calculate cost per successful task by dividing the total model, tool, human review and correction costs across all attempts by successful completions. That captures the trade-off and tells you whether a model downgrade is a saving or a hidden expense.
Why one successful run is not a pass rate
Because agent behavior varies between runs, the same task can pass on Monday and fail on Tuesday with nothing changed. A trustworthy evaluation runs each task several times and reports two different numbers for a stated number of attempts: how likely the agent is to succeed at least once, and how likely it is to succeed every time. The second number helps reveal inconsistent behavior that a single-run pass rate can hide.
Anthropic’s guide names these pass@k and pass^k, where k is the chosen number of attempts per task. Pass@k is the probability of at least one success in k attempts, and it rises as you add attempts. Pass^k is the probability that all k attempts succeed, and it falls. In the guide’s worked example, an agent with a 75% success rate per trial passes all three of three trials only about 42% of the time, assuming independent trials at that success rate.
That arithmetic is why a demo misleads. Rerunning a task until it works tells you whether the agent can succeed at least once, which pass@k captures. That suits an internal assistant where a person picks the best draft. For an agent that issues refunds or updates contracts, use pass^k to measure consistency across the chosen k trials and report k alongside the score. Keep the per-run success rate visible too: it estimates success on a single attempt.
So run each important task several times, and start every trial from a clean environment: Anthropic’s guide notes that leftover files or cached data shared between runs can cause correlated failures or inflate scores.
Who should grade the agent?
Use code wherever the criterion can be checked mechanically, a model-based grader where judgment is needed, and people to calibrate the model grader and to review the cases with the highest stakes. Each grader type has a cost and a failure mode, and a sound evaluation combines them on purpose rather than defaulting to one.
Code-based graders compare database records, check that a required tool was called with the right parameters, validate output schemas and count turns. They are fast, cheap, objective and reproducible. Their weakness is brittleness: they fail valid answers that do not match the expected pattern exactly. Use them for outcomes and constraints.
Model-based graders apply a rubric to free-form output: was the explanation accurate, was the tone appropriate, was the answer grounded in the policy the agent retrieved? They handle nuance and scale well. They are also non-deterministic and, as Anthropic’s guide puts it, need calibration with human graders to be accurate. Two habits make them trustworthy. Give each dimension its own isolated judge rather than asking one prompt to score everything. And measure how often the judge agrees with your domain experts on a sample before you rely on its scores.
Human graders are the gold standard and the most expensive option. Spend their time on reference answers, grader calibration and the cases where a wrong action would reach a customer, a regulator or your ledger.
Be strict about calibration, because a grader that checks the surface of an output can be passed by output that only looks right. In October 2026 research, Anthropic used a model to generate fine-tuning data that passed a production-style monitor yet induced misaligned behavior in separate research models trained on it. This was a deliberate adversarial experiment, but the lesson transfers: the more a score depends on one grader, the more you need outcome checks and human spot checks that do not share its blind spots.
How do you build your first evaluation set?
Start small and start from reality. Anthropic’s guide says 20 to 50 simple tasks drawn from real failures is a great start, because early changes to an agent have effects large enough for a small set to detect. You do not need a benchmark. You need the cases your team already checks by hand, plus the ones that went wrong.
1. Collect tasks from where the work already happens. Take them from pre-release manual checks, support tickets, the bug tracker and the workflow the agent replaces. One real customer request is worth ten invented ones.
2. Write the reference outcome before you run anything. For each task, record the expected end state and the rules that apply. If you cannot write the expected outcome, the agent cannot be graded on that task yet.
3. Balance the set. Include cases where the agent should act and cases where it should refuse, escalate or ask a question. Anthropic’s guide warns that one-sided evals create one-sided optimization: test only whether an agent searches when it should, and you may get an agent that searches for everything.
4. Include the hard, boring cases. Expired records next to current ones, two near-identical customers, missing fields, a tool that times out, a request that is half in scope. Our guide to why RAG systems fail in production catalogs this kind of case for retrieval, and the same thinking applies to agents that read your data.
5. Give partial credit where it is real. A support agent that verifies the customer and diagnoses the issue but fails to process the refund is better than one that fails at the first step. Scoring that difference tells you where to work next.
6. Read the transcripts. Before you trust a score, read a sample of the runs behind it. A low score can come from a grading bug or an ambiguous task as easily as from a weak agent, and only the transcript tells you which.
The set will grow. Every confirmed production failure that reaches a person should become a new task, so the suite can detect the same failure before a later release.
Which evaluations should gate a release?
Regression evaluations should gate every release. They answer one question, whether the agent still handles everything it used to, and they should pass at close to 100%. Capability evaluations measure what the agent cannot yet do well, start at a low pass rate by design, and guide development rather than block it. Mixing the two leads either to blocked releases or to unnoticed regressions. Anthropic’s guide draws the same line, and tasks graduate from the capability suite to the regression suite once the agent handles them reliably.
The trigger list matters as much as the suite. Run the regression set on every change to the model, the prompt, the tool definitions, the retrieval index and the APIs the agent calls. Model changes are the easiest to underestimate. On 27 October 2026, Salesforce announced Agent Evals for Agent Fabric, its control plane for agents built on different platforms, and its own illustration was a model swap: a retailer moves its order-support agent to a cheaper model to cut costs, and scheduled tests then find the agent giving inaccurate information about delayed orders. Salesforce lists Agent Evals as beta in November 2026 and generally available in January 2027. Whatever platform you use, the scenario is the right one to plan for, because a cost-driven model change is exactly the edit that looks harmless in a pull request.
The same survey data shows how rarely this gate exists. In TestMu AI’s 2026 report, only 27% of agent builders re-run evaluations after a model or prompt change, and 15% make evaluations a release gate in CI. For a team that already has an agent live, closing that gap is where we would start.
How do you stress test an AI agent?
Test the agent the way a careless user, a confused customer and a hostile actor would use it. That means malformed inputs, contradictory instructions, requests just outside the agent’s scope, and instructions hidden in the documents, emails or tool results the agent reads. Few teams do this yet: in TestMu AI’s survey, 28% of agent builders test against users who try to trick or break the agent.
The most important channel is indirect: an agent that reads an inbound email, a web page or an attachment can be instructed by that content. Include cases where retrieved text tells the agent to ignore its rules or call a tool it should not, and check that it rejects those instructions while handling the legitimate task appropriately. Test tool permissions and downstream authorization to block prohibited actions and limit the damage from a compromised run. Also test harmful actions that remain possible within the agent’s allowed scope.
Treat safety results as pass or fail. An agent that leaks another customer’s data in one run out of fifty does not have a 98% safety score; it has a defect that blocks release.
How do you keep evaluating after launch?
Offline evaluation tells you whether an agent is ready. Online evaluation tells you whether it stays ready. Once live, sample real sessions, score them with the same graders you trust offline, track the scores over time, and route the failures into the regression set. Production is where the inputs nobody anticipated actually arrive.
A workable loop has four parts. Capture full traces, including tool calls, under your data retention rules. Score a sample automatically and send a smaller sample to human review. Watch for drift in outcome scores, escalation rates and cost per successful task. And convert each confirmed failure into a test case.
Name an owner who reads the scores and can hold or roll back a change; scores nobody acts on are reporting, not quality control. Our guide to why most AI agents never reach production covers the observability, guardrails and cost controls that sit around this loop.
Tie autonomy to the evidence. A new agent can start by proposing actions for a person to approve, and categories of tasks move to autonomous execution as the evaluation record shows consistent success.
When do the results say not to ship?
The results say not to ship when the agent fails any hard constraint, when its consistency on core tasks falls below the threshold you set before testing, or when its cost per successful task makes the business case negative. Set these thresholds before you see the scores. Thresholds chosen after the results tend to move to wherever the agent happens to land.
Hard constraints come first: any data leak, unapproved consequential action or regulatory breach blocks release, whatever the averages say. Next, consistency: define the share of core tasks that must succeed on every repeated trial, and hold to it. Then economics: compare the full cost per successful task with the cost of the process the agent replaces.
There are also cases where evaluation tells you the agent is the wrong tool. If two experts cannot agree on what a correct outcome looks like, the workflow is not defined well enough to automate yet. If the job is fully deterministic, a rules engine or a conventional integration will be cheaper and more predictable than an agent. And if the cost of a single error is severe and you cannot verify outcomes automatically, keep a person in the decision and use the agent to prepare it.
A “not yet” is a useful result: it tells you which tasks fail and what would change the verdict, far more cheaply before launch than after.
Where unicrew fits: how we approach agent evaluation
We treat evaluation as part of building an agent, not as a test phase bolted on at the end. We build agents to each client’s own requirements and sell no agents of our own, so the evidence that an agent works belongs to the client: the dataset, the graders, the thresholds and the integration artifacts are assets the client’s team keeps and runs after we hand them over.
We start from the decision, not the test. Before a single case is written, we agree with the client’s engineering and domain leads which releases the results will gate, which cases are representative and which outcomes count as failures. That only works when the agent has a clear brief, which is why we scope one agent to one job: the systems it may use, the actions it may take and what done means.
Domain experts define correct; engineering makes it repeatable. The people who do the work today know what a right answer looks like. We work with them on criteria and failure categories, then choose rule-based, model-based or human grading for each criterion. Every model grader is calibrated against expert judgment, and the points where the two disagree are recorded rather than averaged away.
We test the parts separately. Retrieval can find the right material while generation invents, and an agent can reach the right answer through actions outside its intended workflow. Scoring these apart tells a team where to fix. The deterministic layer gets the same rigor: we test the APIs, data flows, integrations and permissions the agent depends on alongside its task quality, and our QA practice includes ISTQB-certified engineers.
Autonomy follows evidence. Which actions need human approval is agreed before launch, and widening that authority depends on evaluation results and the decision of the person responsible for the workflow.
We have built one for our own operations, too. Our internal agent proof of concept finds missing time logs in our project management system and notifies people and their managers, and it delivered a 30% improvement in timely work-time logging compliance across our own team. We also build and operate two AI products, Snaplore and Talkmetry, so model changes, quality review and running cost are things we manage in our own production, and that experience shapes how we plan evaluation for AI features.
If you have an agent in pilot, a model change on the roadmap or a stakeholder asking whether it is safe to scale, tell us about the agent you need to trust, and we will scope the evaluation around the decisions it has to support.
Key takeaways
- A demo is a sample of one. Agents vary between runs, so a single success says little about how the agent behaves for your customers.
- Define success as an end state. Check the records, tickets and files the agent changed, not the message it wrote about them.
- Report consistency, not best runs. For customer-facing agents, measure how often each task passes all k trials and report k alongside the per-run success rate.
- Match graders to criteria. Use code for outcomes and constraints, calibrated model graders for judgment, and people for calibration and the highest-stakes cases.
- Start with 20 to 50 real tasks. Draw them from manual checks and real failures, balance them, and write the expected outcome first.
- Gate releases on regression evals. Re-run them on every model, prompt, tool and API change, especially cost-driven model swaps.
- Test adversarially and keep testing after launch. Include injected instructions in the data the agent reads, and turn every production failure into a new test case.
Frequently asked questions
AI agent evaluation is the practice of testing whether an agent completes a defined job correctly, consistently, safely and at an acceptable cost, on tasks that represent real use. It differs from testing a single model answer because an agent plans, calls tools and changes data across several steps, so the evaluation checks the final state of the systems it touched, the actions it took on the way, and how often it succeeds across repeated runs of the same task.
Check that records, tickets or files end up in the right state. Measure consistency across repeated trials, approval and policy compliance, and efficiency in turns, tokens and time. Calculate cost per successful task as total model, tool, human review and correction costs across all attempts divided by successful completions. Keep the scores separate: a high success rate cannot offset a privacy or compliance failure. Our guide on why AI agents stall before production covers the operating side.
LLM evaluation scores a model's response to a prompt, such as whether an answer is accurate, grounded and well formatted. AI agent evaluation scores a whole system working toward a goal: the plan, the tool calls, the data it reads and writes, and the end state it leaves behind. An agent can produce a convincing final message while the underlying action failed, so agent evaluation checks the environment, not only the text.
Build a set of realistic tasks, each with an unambiguous success criterion that two domain experts would grade the same way, and include cases where the agent should act and cases where it should not. Run each task several times from a clean environment, grade the outcome with code where possible, and use a calibrated model grader or a human reviewer for judgment calls. Report both the per-run success rate and the share of tasks that pass all trials, with the number of trials stated.
Teams combine open-source evaluation libraries, tracing and observability tools, eval features in agent platforms, and their own test harness wired into CI. The tool matters less than the dataset, the success criteria and who acts on the results. Common building blocks are evaluation datasets kept in version control, code-based checks on the end state, model-based graders with written rubrics, trace capture for every run, and a CI job that runs the regression set before each release. See our AI QA and evals service for how an engagement is scoped.



