Does your agent work eight times out of eight?

An agent that works 80 percent of the time fails once a week for a daily user. The failures do not spread politely across your user base either. They cluster on the hardest tasks, which are usually the ones that matter.

Average accuracy is the number every eval reports and the wrong number for a product decision. There is a better one, it comes from agent research, and it takes an afternoon to start using.

Why an average hides the bug that matters#

Run each task several times and count only the cases where every attempt succeeded. That is pass^k: the share of tasks that worked k times out of k.

It is a different question from the average. An average asks whether the system usually works. pass^k asks whether a user who does this thing repeatedly can rely on it. Those diverge hardest on exactly the tasks where variance is high, which are the ones with a tool call in them.

The arithmetic is unforgiving. If a task succeeded independently with probability 0.8 on each attempt, the chance of eight successes in a row would be 0.8 to the eighth power, which is about 17 percent. Real failures are correlated rather than independent, so the measured figure is usually better than that bound. It is still far below the average, and the gap is the part nobody reports.

Where the idea comes from#

τ-bench introduced pass^k while testing agents in a shape that maps almost exactly onto a consumer app: a simulated user, domain-specific API tools, and written policy rules the agent has to respect. The finding was that “even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks”, and that agents “are quite inconsistent (pass^8 <25% in retail)” (Yao et al., 17 June 2024).

Date that when you quote it. It describes GPT-4o in mid-2024 and models have improved substantially since. What has not changed is the gap between the two numbers, because the gap is a property of tool-using systems rather than of any one model.

The Berkeley Function Calling Leaderboard contributes the other half of the method: what to assert. Its multi-turn evaluation compares “the backend system’s state (excluding the private attributes) after all function calls are executed at the end of each turn”, across 1,000 multi-turn entries in categories including Missing Parameters, Missing Functions, Long-Context and Composite (BFCL V3, 19 September 2024, updated 10 December 2024). Assert on state, not on text. Whether the order exists is a fact. Whether the sentence describing it is good is an opinion.

SWE-bench is worth knowing as a methodology exemplar and as a warning about benchmark ageing. It built 2,294 tasks “from real GitHub issues and corresponding pull requests across 12 popular Python repositories” and verified solutions by execution rather than by inspection (Jimenez et al., 10 October 2023). Its headline at publication was that “the best-performing model, Claude 2, is able to solve a mere 1.96% of the issues”. That number is a fact about October 2023 and quoting it as a current capability is how benchmarks get misused.

Running it on a Tuesday#

You do not need a research harness. Four pieces.

A task list, which is the interesting part and is covered below.

A runner that executes each task k times against a real backend with test data it can reset. Eight is a reasonable k for a product team: large enough to expose variance, small enough to run in minutes.

An assertion per task that reads the backend, not the transcript. Did the order exist afterwards. Did the balance change by the right amount. Did nothing change when the task was supposed to be refused.

And a record of the failures, kept as artefacts rather than as a count, because the count tells you nothing you can act on.

Choosing the eight tasks#

The temptation is to write the tasks you designed the agent for. Those will pass. Draw from real traffic instead, which is the strongest argument for instrumenting before you launch.

A set that earns its runtime covers five shapes.

The most common request, phrased the way the logs show users phrasing it rather than the way your product manager phrases it.

A compound request, because composition is where the failures cluster. Two things in one sentence that require two tool calls and a join.

A write that has to be confirmed, asserting both that the action happened after approval and that nothing happened without it. That second assertion is the one that catches a confirm gate that has quietly stopped working, and it is the single most valuable test in the set.

A request the agent should refuse, where the assertion is that the backend is unchanged. Refusals are untested almost everywhere, and a regression here is silent.

And a request against stale state, where something changed between the read and the write. The Berkeley leaderboard files this under “failure to understand the current state before performing action”, and reports that even the best models “sometimes fail to explore the current state before performing actions, which can be dangerous if the actions are non-reversible”.

Eight tasks, eight runs each, sixty-four executions. That is a coffee break, not a research programme.

Simulating the user, and where that goes wrong#

Multi-turn tasks need something on the other side of the conversation, and the usual answer is another model playing the user.

It works, with two known distortions.

A simulated user is more cooperative than a real one. It answers the question it was asked, in the form expected, first time. Real users answer a different question, or answer with a photo, or say “the second one”. Writing a few deliberately uncooperative personas costs an hour and finds real bugs.

And a simulated user leaks the answer. Given the task description, it will phrase its first message using the exact vocabulary of the tool the agent is supposed to call, which tests nothing. Give the simulator a goal and a persona, never the task’s internal wording.

Judging the output#

Split the judgement in two. The backend assertion is deterministic, and it is the part that decides pass or fail. Everything about the answer’s quality is a separate, softer score.

For that softer half, model judges are usable and bounded. The MT-Bench authors found that “strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans”, while naming position bias, verbosity bias, self-enhancement bias and limited reasoning ability as real limitations (Zheng et al., NeurIPS 2023).

So: a model judge scores tone, completeness and whether the answer contradicts the widgets. A model judge does not decide whether the payment happened. Verbosity bias alone is enough of a reason, because a longer wrong answer scores better than a short right one.

Alongside both, record what Anthropic recommends collecting for tool-using agents: “the total runtime of individual tool calls and tasks, the total number of tool calls, the total token consumption, and tool errors” (Writing tools for agents, 11 September 2025). A change that improves reliability and triples the token cost is a change you want to know about.

Putting it in CI without it becoming a flaky test#

The failure mode here is predictable. The eval goes into the pipeline, it fails intermittently because it talks to real services, and within a month someone marks it non-blocking. After that it is decoration.

Four things prevent that.

Gate on the reliability number, not on individual runs. If the threshold is that all eight write tasks must pass eight times out of eight, one network timeout does not turn the build red on its own.

Separate infrastructure failures from agent failures in the report. A backend that returned a 503 is not the agent being unreliable, and conflating them is how trust in the suite dies.

Run the full set nightly and a three-task subset per pull request. The full run is minutes; a pull request is not the place for it.

And treat a threshold change as a code review. Lowering the bar to get a build green is a decision somebody should have to justify in writing.

Supabase published a good model of what this looks like in practice. They built an eval to check whether their row-level-security documentation could be followed correctly by a coding agent, found that the initial run produced a real security defect where unauthenticated users were granted write access, restructured the guide, and published the chart showing the check failing until the fix and passing daily afterwards (Limonczenko and Richers, 1 August 2026). The shape is the lesson: build the eval, find something genuinely broken, fix it until the eval passes, publish the numbers including the bad ones.

What to do when the number comes back bad#

The first run is usually worse than expected, and the instinct is to reach for the prompt. Resist it for a week, because the prompt is the one lever whose effect you cannot attribute.

Work through the failures in this order.

Look at which tasks failed rather than how many. Three failures spread across three tasks and three failures concentrated on one are different problems. The concentrated case is a bug and is usually cheap. The spread case is variance and is usually a model or context problem.

Read the tool calls in the failing runs before reading the transcripts. Most failures are visible in the arguments, and the transcript is a narration of a decision that was already made. If the same tool was called twice with different arguments, the description is ambiguous. If a required argument was invented, the schema is not constraining it. The schema fixes for each pattern are more reliable than prompt edits and they are attributable.

Check whether the failures depend on the backend’s state at the time. When a task passes against a freshly reset fixture and fails against one carrying three months of history, look at pagination and truncation in the tool’s return value before you look at the model.

Then change one thing and re-run. Sixty-four executions is cheap enough that a one-change-at-a-time discipline costs you minutes rather than days, which is the entire reason to keep the set small.

Expect the first few rounds to move the number a lot and later rounds to move it very little. When a change stops being measurable, you have found the level at which the remaining failures are model variance, and the next lever is a different model rather than a better schema.

What the number is for#

Not for a slide. The reliability figure earns its keep in three places: deciding whether a schema change actually helped, deciding whether a model upgrade is safe to ship, and deciding whether a new write action is ready to be exposed at all.

It is also the honest answer to the question a buyer should be asking in a demo, which is not “how accurate is it” but “how often does the same request work”. Those are different questions, and most of the difference between an agent and a chatbot lives in the second one.

Common questions#

How do I evaluate an AI agent for a product, not a paper? Pick eight tasks from real traffic, run each eight times against a resettable backend, and assert on backend state rather than on the transcript. Report the share of tasks that passed all eight attempts, not the average across all runs.

What is pass^k? The share of tasks where every one of k attempts succeeded. It comes from τ-bench, which reported pass^8 below 25 percent in its retail domain for a mid-2024 model whose single-attempt success was already below 50 percent.

Should I use an LLM as a judge? For answer quality, yes, with its biases documented. The MT-Bench authors measured over 80 percent agreement with human preferences while naming position, verbosity and self-enhancement bias. For whether an action happened, no. Read your own backend.

How many tasks and how many runs? Eight and eight is a good starting point for a product team. Sixty-four executions runs in minutes, exposes variance, and is small enough that people actually maintain the set.

How do I stop the eval becoming a flaky CI test? Gate on the aggregate reliability threshold rather than on individual runs, report infrastructure failures separately from agent failures, run the full set nightly with a small subset per pull request, and require a code review to lower a threshold.

Sources#

• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 17 June 2024. Accessed 12 September 2026.
• Berkeley Function Calling Leaderboard, BFCL V3: Multi-Turn and Multi-Step Function Calling, 19 September 2024, updated 10 December 2024. Accessed 12 September 2026.
• Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 10 October 2023. Accessed 12 September 2026.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track. Accessed 12 September 2026.
• Anthropic, Writing tools for agents, 11 September 2025. Accessed 12 September 2026.
• Miranda Limonczenko and Nik Richers, Are Supabase docs agent-friendly?, Supabase, 1 August 2026. Accessed 12 September 2026.