SWE-Bench Pro

What It Measures

SWE-Bench Pro is the harder successor to SWE-Bench Verified, designed to address two limitations of the earlier benchmark. First, it includes more complex multi-file changes — tasks where a fix requires coordinated edits across several files rather than localized changes to a single file. Second, it draws from more recent GitHub issues, reducing the risk that frontier models have seen the tasks during training (a contamination problem that has affected SWE-Bench Verified scores at the top of the leaderboard).

By late 2025 / early 2026, SWE-Bench Pro had become the benchmark of choice for serious agentic coding evaluation. Models are now competing for the SWE-Bench Pro leaderboard the way they competed for SWE-Bench Verified two years earlier — with the difference that SWE-Bench Pro scores remain meaningfully bounded below 100%, providing room for models to differentiate themselves at the high end.

How It Works

The evaluation methodology is similar to SWE-Bench Verified: each task includes a GitHub issue, repository state, and hidden test suite. The model must produce code changes that pass the test suite. The differences are in task selection — SWE-Bench Pro emphasizes harder, multi-file, more recent tasks — and in the rigor of the curation process to ensure tasks are unambiguous and verifiable.

As with SWE-Bench Verified, the agent harness used to run the model is a meaningful variable. Reported SWE-Bench Pro scores generally use standardized harnesses, but cross-report comparisons should always check harness details. Some reports also distinguish between 'pure model' scores and 'model + scaffolding' scores; SWE-Bench Pro tasks are sufficiently complex that the scaffolding choice can move scores by 5-10 percentage points.

Current Leaders