Terminal-Bench 4.0 explained: why the same model scores differently on each board
Terminal-Bench 4.0 is a benchmark of 66 professional computer-work tasks where each agent gets up to eight hours and five trials per task, per BenchLM's page read on 2026-10-27 (America/Chicago). Scores are not directly comparable with versions 3.0 or 2.x, and every row is a model plus an agent harness, so one model scores differently on different boards.
Checked by Agentic Weekly's editorial process: vendor pages re-read.
Last verified: 2026-10-27 (America/Chicago) · Versions covered: Terminal-Bench 4.0 release post (tbench.ai) and BenchLM's mirror, read 2026-10-27; the tbench.ai leaderboard page itself did not render for us · Evidence: Reported (see “What we tested”)
What Terminal-Bench 4.0 tests
Terminal-Bench is a benchmark of agents doing multi-step work in a command-line environment. The team's Terminal-Bench 4.0 post says the update calibrates task resources (time, CPU, memory), fixes tasks and removes saturated tasks, and that every task now has a flat agent timeout of 8 hours. It says 19 tasks were fixed and 8 removed (2 for saturation, 2 for refusals, 2 for public solutions and 2 for unresolved quality or platform-compatibility issues). BenchLM's mirror describes 66 professional computer-work tasks with 5 trials per task.
The tbench.ai benchmarks index lists Terminal-Bench 4.0 with a date of August 28, 2026. The team says versioning is now semantic, which is why the update is 4.0 and not 3.1: the changes to resources and the task set are breaking and required re-running trials. Terminal-Bench 4.1 (verifier improvements) and 5.0 (new tasks) are listed as upcoming milestones.
Get the weekly brief: which AI tools to update, try or skip. Free.
Free. Unsubscribe anytime. No spam. We email a confirmation link first. Privacy
How to read a score: model, harness, effort and board
Every row combines four things, and a different choice in any of them can move the number: the model, the agent harness that wraps it (for example Claude Code or Codex), the reasoning effort setting, and who ran it. BenchLM says each row "combines a model with an agent harness" and keeps the official, Vals-hosted and Artificial Analysis tables separate because they use different setups.
| Model | Official table (harness, effort) | Vals AI table | Artificial Analysis table |
|---|---|---|---|
| Claude Opus 5.5 | 64.85% (Claude Code, max) | 65.15% | 59.6% |
| Claude Sonnet 5.5 | 61.82% (Claude Code, max) | 64.14% | 63.6% |
| GPT-6 Astra | 58.18% (Codex, max) | 59.60% (max) | 59.1% |
| GPT-6.1 Sol | 58.18% (Codex, max) | 55.05% (max) | 56.1% |
| Gemini 4 Argon | Not in the rows we read | 57.58% (high) | 57.1% |
| Kimi K3.1 Pro | Not in the rows we read | 63.64% (max) | 67.2% |
The order of models changes from table to table: Opus 5.5 ranks first among the models listed in each of the official and Vals tables, while Kimi K3.1 Pro ranks first on Artificial Analysis's. GPT-6.1 Sol ties GPT-6 Astra on the official table and is below it on the other two. BenchLM marks Artificial Analysis's table as display-only, stored separately because Artificial Analysis controls the agent harness and configuration. We do not crown a leader from these rows. Source: BenchLM.
Why 4.0 scores cannot be compared with 3.0 or 2.1
BenchLM says the 4.0 breaking changes mean scores are not directly comparable with 3.0 or 2.x. A vendor's older number can sit next to a new one on a page and mean something different. For example, Meta's model card for Muse Spark Open reports 76.4 on Terminal-Bench 2.1 with the Terminus 2 agent, while BenchLM's Vals-hosted 4.0 table shows 6.57% for Muse Spark Open at xhigh reasoning. Those are different benchmark versions, harnesses and boards, not a contradiction. See Muse Spark Open and Meta's model card.
What a Terminal-Bench score does not tell you
- Cost and speed: the score is a completion rate, not a price. Compare list prices separately, for example Opus 5.5 vs Sonnet 5.5.
- Your repository: the tasks are not your code, and a harness you do not use may change results.
- A single run: use the trial count and board notes, and check the date of the snapshot.
- Vendor claims: a vendor's own number is a vendor claim until you can find the setup behind it.
What we tested
Nothing: we did not run Terminal-Bench or any model on it. The tbench.ai leaderboard page is rendered by JavaScript and did not load for us, so the official-table numbers here come from BenchLM's mirror, and the 66-task and 5-trial figures are BenchLM's. The release post is the primary source for the 8-hour limit and the task changes. All numbers were read on 2026-10-27.
FAQ
What is Terminal-Bench 4.0?
A benchmark of 66 professional computer-work tasks in a command-line environment, per BenchLM, with up to eight hours per agent and five trials per task. The Terminal-Bench team's release post says it calibrates resources, fixes 19 tasks and removes 8 saturated or flawed ones.
Why does the same model get different Terminal-Bench scores?
Each row combines a model with an agent harness, a reasoning setting and a board. On BenchLM's mirror, Claude Opus 5.5 is 64.85% on the official table, 65.15% on Vals AI's and 59.6% on Artificial Analysis's. Setups differ, so the numbers are not interchangeable.
Can I compare Terminal-Bench 4.0 with 3.0 or 2.1 scores?
No, not directly. BenchLM says 4.0's breaking changes mean scores are not directly comparable with 3.0 or 2.x. A vendor reporting 2.1 and a board reporting 4.0 are using different task sets, resources and trials.
Which model is best on Terminal-Bench 4.0?
It depends on the board. Opus 5.5 leads among the models we tabulated on the official and Vals tables, while Kimi K3.1 Pro leads on Artificial Analysis's. We do not name one winner, and boards update, so check the snapshot date.
Have you tested Terminal-Bench 4.0 ourselves?
No. This page explains published results as read on 2026-10-27 from the Terminal-Bench team's post and BenchLM's mirror. We did not run the benchmark, and the tbench.ai leaderboard page did not render for us, so official-table numbers come from the mirror.
Sources
- Terminal-Bench: Terminal-Bench 4.0 release post (fetched 2026-10-27)
- Terminal-Bench: benchmarks index (fetched 2026-10-27)
- BenchLM: Terminal-Bench 4.0 (official, Vals and Artificial Analysis tables) (fetched 2026-10-27)
- Meta: Muse Spark Open model card (Hugging Face) (fetched 2026-10-27)
Related
Page history
- : First published from the Terminal-Bench 4.0 post and BenchLM's mirror of three boards, read on this date. We ran no benchmark.
Markdown version for agents: /guides/terminal-bench-4-explained.md
Free · weekly · about six minutes
Get The Weekly Verdict, free
What shipped this week, what is worth switching to, and what to ignore. We run what we can and label what we only read.
- One email a week. Each release gets one call: update now, worth a look, or skip.
- Every claim shows its evidence. Reported means we read the sources; Tested means we ran it, with the exact scope and the failures left in.
- A free AI Tools Cheat Sheet (web page and printable PDF) lands in your inbox when you confirm.
Free. Unsubscribe anytime. No spam. We email a confirmation link first and add you only after you click it. We use your email only to send this newsletter and never sell it. Privacy