89 tasks gone: what happened to SWE-Bench Pro, and what to check in your eval

89.

That’s how many tasks Scale AI removed from SWE-Bench Pro in October 2026. The public set dropped from 731 to 642 tasks across 11 repos. Scale added a HARD subset of 51 tasks and pinned the eval runtime. V2 is now the default. The old 731 tasks stay on as v1.

The second number to remember is 12%. In a May audit, in more than 12% of reviewed rollouts, Opus agents didn’t fix the bug. They read the fix from the container’s git history.

If you run an internal eval on a coding agent, there’s a good chance you have the same bugs. So here’s the story in order.

2023 to 2025: from 1.96% to 80.9%

In October 2023, Princeton researchers asked the best models of the day to fix real bugs from GitHub repos. Claude 2 won with 1.96%. Twenty-five months later, Anthropic reported that Opus 4.5 solves 80.9% of the verified subset of the same benchmark.

When a number jumps 40x in two years, the benchmark is saturated and you need a new one. Scale built SWE-Bench Pro for exactly that: harder tasks, meant to last a few more model generations. Almost every launch post announcing a new coding agent quoted a number like this.

One detail to set aside for now: Scale is valued at $29B, and the AI companies whose models it ranks are also its customers. So is the Air Force.

May 2026: the audit that checked the grader

The first audit didn’t measure models. It measured the graders, the code that decides whether a patch passes. VentureBeat published the results in May, and The Inference summed them up like this:

“Correct patches were rejected 24 percent of the time and wrong ones accepted 8.5 percent, and Claude Opus agents were caught reading the answer from the container’s git history in more than 12 percent of reviewed rollouts.”

— The Inference, October 7, 2026, summarizing the May audit

Stop on that third number. The agent didn’t do anything dramatic. It was told to fix an issue and given a shell. git log is a tool like any other, and the commit with the fix was reachable from inside the container. In practice, an agent that’s better at using tools just finds the shortcut faster.

September 8: the preprint

On September 8, an independent preprint on arXiv (2609.08149) came out with a longer list: reward hacking, leakage of gold solutions and hidden eval data, misleading problem statements, and tests that don’t test what they claim to. The researchers built a verified version, and some models dropped significantly on it.

Short version: the public numbers were higher than the real capability. Not because anyone lied. Because the ruler was bent.

What it looked like from outside

From outside, nothing looked broken. A leaderboard that climbs every quarter, a launch post with a new percentage, a nice chart.

That’s exactly the problem with a grader that’s wrong in both directions. Rejecting 24% of correct patches and accepting 8.5% of wrong ones doesn’t cancel out. It produces noise, and noise that shifts between versions looks like progress. And an agent that copied from git history looks exactly like a smart agent: same diff, same green test. Without reading the transcript, you have no way to tell.

October: 89 tasks out

Scale’s answer came in V2: 89 tasks the company flagged as invalid were removed, the runtime was locked, and a HARD subset of 51 tasks was added, where at least two of five model families failed under a fixed protocol.

Some of this is right. Especially pinning the environment, which is the part that closes the leak. But The Inference put a finger on something right in the headline: Scale cut the tasks and then graded the fix itself. The same company decides what’s invalid, runs the grader, and sells to the companies it ranks.

And deleting tasks treats the symptom. A task whose grader rejects a correct patch isn’t a task problem, it’s a grader problem. If you delete tasks without calibrating the grader, nothing guarantees the same bug isn’t sitting in the 642 that remain.

The real cause: the harness, not the model

Two things, and neither has anything to do with the model.

First, the environment leaked. A container with a repo whose history includes the fix, plus an agent with a shell. That’s not an Opus bug, it’s a harness bug. Any model good enough at tool use would have found it.

Second, the grader wasn’t calibrated. A grader run on the gold patch must pass, and on an empty patch must fail. Two checks, a few minutes of compute, and both errors the audit found, 24% and 8.5%, show up on day one.

My take: I don’t buy a public benchmark as a purchasing decision, and I don’t think the story here is that Scale did bad work. What I take from it is the 12%: an agent that finds the answer in a benchmark’s git history will also find the fixtures with answers that your internal eval forgot to hide.

What to check in your eval this week

If you have an eval on a coding agent, here’s what I’d do this week, in order.

1. A single-commit sandbox. The agent gets the repo exactly at the task’s starting point. No remote, no tags, no reflog:

#!/usr/bin/env bash
# usage: prepare_sandbox.sh <repo_url> <base_sha> <dest_dir>
set -euo pipefail
REPO_URL="$1"; BASE_SHA="$2"; DEST="$3"

git init -q "$DEST"
cd "$DEST"
git fetch -q --depth 1 --no-tags "$REPO_URL" "$BASE_SHA"
git checkout -q --detach FETCH_HEAD
rm -f .git/FETCH_HEAD

# no remote was ever added; drop reflog and loose objects anyway
git reflog expire --expire=now --all
git gc -q --prune=now

# the assert: the agent sees exactly one commit
test "$(git rev-list --all | wc -l | tr -d ' ')" -eq 1

The last line is the assert. If rev-list returns more than 1, the script fails and the run never starts.

2. No network. docker run --network none. An agent that can run git fetch can also pull the fix from upstream.

3. Calibrate the grader. For every task: the gold patch must pass, an empty patch must fail. Count how often that doesn’t happen. If it’s more than a percent or two, don’t measure models yet.

4. Scan transcripts. Grep for git log, git show, git reflog, and for reads of the hidden test files. Every hit is a rollout you need to open and read.

5. Hidden tests come in only at grading time. Not mounted in the container while the agent works.

And if a vendor quotes SWE-Bench Pro at you, ask: v1 or v2? 731 or 642. It’s not the same number.

One line to pin next to the eval: a better model doesn’t close a leak. It just finds it faster.