How much of an agent benchmark's failure is the benchmark's fault?

When a frontier model fails a task on an expert-written benchmark, we usually read it as a model failure. I wanted to know how often that's wrong. I ran Claude Sonnet 5.5 and GPT-6.1 Sol on all 240 tasks of Mercor's APEX-Agents and reviewed, task by task, both the failures they couldn't shake and a random sample of their passes.

Of the 22 tasks both models failed on every attempt, 8 can't be passed with a correct answer, 2 were failed by the judge on answers that met the criterion, and 6 are ambiguous. Only 6 are clear model errors. On the other side, 1 of 30 sampled passing tasks gave credit it shouldn't have. Overall APEX-Agents holds up well: about 4% of its tasks are unpassable, and false passes are rare. But among the tasks both top models keep failing, flawed tasks outnumber real model errors 16 to 6.

This came out of testing the mock Google Workspace environment we built at Terac. The engineering side is on the Terac blog.

Method

APEX-Agents has 240 tasks across investment banking, management consulting and law. Each one puts the agent inside a fictional firm with project files, a mailbox, a calendar and chat history, and asks for a deliverable. A judge model grades it against expert-written criteria, and a task passes only if every criterion does. I kept the tasks, rubrics and grader exactly as published and ran them in my own environment (details in the appendix).

Both models ran at maximum reasoning effort with Mercor's system prompt, one attempt per task. That gives Pass@1 of 78.8% for Sonnet (95% CI 73.3 to 83.8; the leaderboard lists 75.5% ± 4.7) and 68.3% for Sol (62.5 to 74.2).

The review went in two directions:

1. Failures. 29 tasks were failed by both models. I gave each model a second attempt on those, and 22 still failed for both. For each of the 22 I read the prompt, the files, the mail, the rubric, the judge's reasoning per criterion and all four answers, and put it in one of four groups: defective task, grader error, ambiguous, or model error.
2. Passes. 142 tasks were passed by both models. I drew 30 at random, 10 per domain, and checked both passing answers against the gold, looking for credit given to answers that were wrong, incomplete, hedged, or right for the wrong reason.

Models did a first pass over the long files and trajectories. I checked every verdict against the task files myself, and the per-task evidence is in the appendix.

What's behind the failures

The four groups, defined by what a correct answer would get:

• Defective task: a correct answer fails. The gold contradicts the prompt, the files or the mail, or a criterion grades something the prompt never asked for.
• Grader error: the answer meets the criterion as written and the judge fails it.
• Ambiguous: the prompt supports two reasonable readings and the rubric accepts one.
• Model error: everything needed was there and the models got it wrong.

A few defective tasks show the patterns.

In an LBO task (world226-rm-04), the criterion requires a 21.8% IRR, while the explanation attached to the same criterion says the answer is 22.0% and to "reject 21.8%." The explanation also assumes capex rises 200 basis points where the prompt and the colleague's email say 400. Both models built the model independently, agreed within 0.2 points, and failed.

In world227-sk-task03 the managing director's email says the senior notes accrue as 100% PIK at 11% and spells out the exit formula. The gold uses the spreadsheet's default cash-pay notes. Runs that followed the email failed, and the runs that passed had skipped it.

In a legal task about a promotional reel (lawworld417-as-02), three criteria require treating an edit as AI-generated. Nothing in the prompt or files mentions AI. Those criteria failed in 18 of 18 runs across every model and harness I tried.

In lawworld433-naf-01 one criterion checks that the answer restates the premise from the prompt's first line ("Heather Collins, the Lead Night Shift Caregiver"). Every answer gets the legal question right and fails that criterion, 0 of 7 runs.

The grader errors are a different problem. In world-127-tj-task-1 every criterion accepts any value within €20 of the gold, and the answers land €1 to €8 away on figures in the hundreds of billions. The judge failed all six criteria in all four runs. Rewording would not fix that one; the judge needs to apply the tolerance.

What's behind the passes

Auditing only failures can make a benchmark look worse than it is and a model look better, so I checked the other side too. Of the 60 passing answers in the sample (30 tasks, two models), 58 were right on every criterion. Most rubrics ask for specific numbers or conclusions, and the passing answers gave each one once, without hedging between values.

The exception is a Texas anti-SLAPP task. One criterion requires stating that a pre-suit petition counts as a "legal action" under the statute. Both models only assumed it ("Even treating the stand-alone Rule 202 petition as a 'legal action'…") and reached the correct overall conclusion, and the judge read the assumption as a statement. That's 1 task in 30, with a 95% interval of roughly 1% to 17%.

How much it moves the score

There's no single right way to score a flawed task, so here are the main options side by side. Ambiguous tasks count either way, since neither reading is wrong.

The defective tasks are worth about 3 points to each model, and the ambiguous ones another 2 to 3. The gap between the two models stays at about 10 points under every option. Each model also passed one defective task on its scored run (the PIK task above, by skipping the email), so a small part of the published score is unearned too.

A second validity problem: elicitation

Task flaws aren't the only thing that bends the score. About half of APEX-Agents (114 tasks, by a model classifier I ran over every task's rubric and messages) depends on something that only appears in the firm's mail or chat, like a colleague's correction to an assumption. Mercor's system prompt mentions communication channels in passing. My first prompt added one sentence: "They may contain context relevant to the task (e.g. instructions from colleagues)."

I tested that sentence on its own, with everything else fixed.

For Qwen 3.8 27B the sentence tripled the share of runs that opened the mailbox and nearly doubled Pass@1 (paired difference 26 points, 95% CI 10 to 40). Sonnet read the mail in every run without it, and with it lost 9.6 points (paired CI 1.9 to 17.3), mostly on banking tasks. The two models ran on different task sets, so compare within a model, not across.

So the same benchmark score mixes capability with how well the prompt draws it out, and the mix differs by model. A weaker model's score here partly measures whether it thinks to check its inbox unprompted. For anyone using results like these to judge what a model can do, a score that moves 26 points with one sentence says as much about the prompt as about the model, and the prompt belongs in the report next to the number.

A check that came back clean: the harness

I ran the tasks through my own environment and a minimal agent harness, so I checked that neither changes the result. On a fixed random subset of 60 tasks with Mercor's prompt, Sonnet scored 70.0% in my harness and 68.3% in Mercor's reference agent, and Sol 66.7% and 65.0%. Qwen, with three attempts per task, spread over 4.7 points across three harnesses with overlapping intervals. With the prompt held fixed, I couldn't detect a harness effect.

What I'd take from this

APEX-Agents is in good shape. About 4% of tasks are unpassable and false passes are rare, for a benchmark this long and this specialized. The problems sit where the model scores are highest: once a model passes 80% of tasks, the flawed ones are a large share of what it has left, and score gains near the top get harder to read.

The defects follow a few patterns, and each suggests a check that task authors could run before release:

• The gold contradicts its own explanation or the task's inputs. Recomputing the gold from the prompt, files and mail by a second person catches these.
• A criterion grades something the prompt never asked for, like restating the premise or naming a cause the files never mention. Reading each criterion against the prompt alone catches these.
• The gold ignores a message the world itself contains. In a benchmark built to reward reading mail, that one is easy to miss.
• The judge ignores a stated numeric tolerance. Numeric checks with tolerances are better done in code than by a judge model.

The review took two directions and about fifty tasks. That's a small cost next to building the benchmark, and it changes how the top of the leaderboard reads.

Appendix

Setup

Tasks, rubrics and grader are Mercor's APEX-Agents 1.1 as published. The agent ran in a bash sandbox with the project files, and reached mail, calendar and chat through mock versions of Google's Workspace MCP servers that I built at Terac, loaded with each task's data. Runs used Harbor on Modal, the Pi coding agent as harness, and models through OpenRouter at maximum reasoning effort. Confidence intervals come from a bootstrap over tasks. Runs that died for infrastructure reasons were dropped rather than scored as failures (5 runs across the experiments here). I used Claude Code agents to run the infrastructure and for a first read of long trajectories, and reviewed every verdict myself.

Flagged tasks