# How we test AI chat apps for coding and reasoning

*Suite version: v1, frozen 1 November 2026. Written by The Agentic Weekly team.*

This page explains exactly how we produce the "tested" results on our compare pages (for example ChatGPT vs Claude). Every
number we publish comes from the procedure below. Alongside each set of results we publish the prompts, every raw reply and
the grading scripts, so anyone can check our work or re-run it.

## What we are measuring (and what we are not)

We test the **consumer chat apps** (claude.ai and chatgpt.com) on paid plans, with the default model and default settings,
the way most subscribers use them. We do **not** test the APIs, coding agents (Claude Code, Codex CLI, Cursor) or custom
scaffolds. Those need agent benchmarks such as SWE-bench-style or Terminal-Bench-style tasks, and we will report them separately.
The model name we report is the label the app shows in its model picker on the test date. We cannot see which exact model
build answers behind that label.

## Why we built our own tasks

Public benchmarks are useful, but recent audits show three problems that matter for a head-to-head test:

- **Contamination.** OpenAI stopped reporting SWE-bench Verified after finding that frontier models could reproduce the
  reference fixes and that many audited tasks had flawed tests. LiveCodeBench dates every problem so models can be tested only
  on problems released after their training cutoff. Simon Willison picked his "pelican riding a bicycle" SVG test because no
  such drawing was likely to be in training data. Our tasks were written for this suite in November 2026 and were never
  published before the first test round.
- **Broken tasks.** OpenAI's July 2026 audit of SWE-bench Pro estimated that roughly 30% of tasks were broken. Typical causes
  were tests stricter than the instructions, underspecified or misleading prompts, and low-coverage tests. The SWE-Bench Pro
  V2 maintainers repaired or retired 480 of the 731 tasks for similar reasons. Our rule: **every behaviour a hidden test
  checks is stated in the prompt**. Where several answers are valid (for example any optimal schedule), the grader accepts
  any of them instead of one exact output.
- **Weak tests.** EvalPlus showed that adding far more tests to HumanEval lowered measured pass rates and even changed
  model rankings. Our graders therefore combine hand-written edge cases with thousands of seeded random inputs, compared
  against a reference solution (differential testing).

## The task set (v1)

| ID | Category | Graded by |
|---|---|---|
| T01 | Long-spec parser (version ranges), Python | ~100 hand-written edge cases + 3,300 random cases vs reference |
| T02 | Find and fix four bugs in a module (one bug not reported by users), Python | targeted tests + 150 random simulations |
| T03 | CSV parser with strict edge-case rules, JavaScript | 45 edge cases + 3,000 random and round-trip cases |
| T04 | Gaps-and-islands SQL query (SQLite) | 25 hidden databases, exact rows and order |
| T05 | Behaviour-preserving refactor with structure limits, Python | 6,000 random inputs vs the legacy code + AST checks |
| T06 | Weighted interval scheduling at n = 200,000 under a time limit, Python | validity + optimality + timing |
| T07 | Data reasoning over a 36-row CSV (MRR, churn, expansion) | 13 exact answers |
| T08 | Logic puzzle with a machine-verified unique solution | 20 exact cells |
| T09 | Promise pool with retries and concurrency limits, JavaScript | 31 timing-aware tests |
| T10 | Business-hours SLA clock with holidays and half-days, Python | fixed cases + 600 random cases vs an independent oracle |
| T11 | Release notes from a commit log | 15 automatic checks + **blind** human rubric |
| T12 | Instruction following with 10 verifiable constraints (IFEval-style) | 11 automatic checks |
| T13 | Predict the exact output of a tricky Python program | 11 exact line checks |

Every task asks for a single reply containing one fenced code block (code, SQL, JSON or text), so a task can be answered
in one chat turn. Before testing, we checked every grader against our own reference solution (it must pass) and against
an empty or wrong answer (it must fail). For most tasks we also checked that plausible wrong solutions fail.

## How a run works

1. **Fresh chat for every attempt**, with the prompt pasted verbatim. Nothing comes before or after it: no system prompt, no
   follow-ups, no "continue", and no regenerating to get a better answer.
2. **Default settings.** We use whatever model and reasoning mode the app selects by default on the test date, and we
   record the labels shown. Memory and custom instructions are off. Tools the app uses on its own (such as running code)
   are allowed because that is the product, and we record when they were used.
3. **Three runs per task per app.** Chat apps do not give the same answer twice. Following Artificial Analysis (which
   averages pass@1 over repeats) and Anthropic's guidance on error bars for evals, we repeat each task three times.
4. **Interleaved order.** Rounds run task by task, alternating apps, in a shuffled order, so time-of-day load affects both
   apps alike.
5. **Infrastructure errors are retried and logged.** A reply that failed to load is re-run in a new chat and counted in
   the raw data. Wrong answers and refusals are never re-run.
6. **Capture.** We copy each reply with the app's own Copy button and store it unedited. The grader takes the last code
   block in the expected language (the same "extract the fenced block" approach EvalPlus uses for chat models). A reply
   without a code block fails that attempt. We never repair answers by hand.
7. **Timing.** We record the seconds from sending to the end of generation. These are wall-clock times from a browser,
   so they are only indicative, and we report medians.

## Scoring

- **Pass** means every hidden check for the task passed (strict, like Terminal-Bench and SWE-bench). We also publish
  partial scores (the fraction of checks passed) so near-misses are visible, but the headline uses strict passes.
- **pass@1** is the mean pass rate over all attempts (13 tasks × 3 runs = 39 attempts per app).
- **Reliability** is the number of tasks passed on all three runs. This is a pass^3 measure, inspired by the pass^k
  reliability metrics Artificial Analysis reports.
- **Spread** is the pass rate on run 1, run 2 and run 3 separately, plus the per-task pass counts.
- **Difference between apps.** We pair the comparison by task and report a bootstrap 95% confidence interval over
  tasks. **If the interval includes zero, we call it a tie**, whatever the raw numbers say. With 13 tasks, only large gaps
  can be told apart from noise. We say so instead of declaring a winner from a one-task difference.
- **Writing (T11).** Two graders score anonymised responses ("Response A, B, C…", shuffled with a recorded random
  seed) on a 5-criterion, 0–10 rubric: accuracy, completeness, upgrade instructions, clarity and concision. They do not
  know which app wrote which response. Blind comparison is the principle behind LMArena. Their style-control work also
  shows that length and formatting sway judges, so the rubric rewards concision rather than length. Both graders' raw
  scores are published.

## What we publish with every result

- A dated one-sentence verdict, for example "As of [date], in our 39-attempt test, [app] passed X% vs Y%," or "a tie
  within the margin of error."
- The results table: per-task passes for each app, pass@1, reliability, median time and the confidence interval.
- The disclosure block: test dates (America/Chicago), plan names, model and mode labels as shown, settings, and the number
  of infrastructure retries.
- Downloads: every prompt verbatim, every raw reply, results.csv / results.jsonl, the graders, the reference solutions
  and the blind-grading key.

## Limits (please read before quoting us)

- **Small n.** 13 tasks × 3 runs is enough to spot clear gaps, not small ones. Benchmarks like Aider's polyglot (225
  exercises) or SWE-Bench Pro V2 (642 tasks) are far larger. We report confidence intervals for this reason.
- **Chat apps change without notice.** A result is a dated snapshot of what the app served that day, under the label shown.
- **One-shot, single-turn.** Real use involves follow-ups, file uploads and agents. Aider's benchmark, for comparison,
  gives models a second try after showing test failures. We measure the first reply only.
- **Our tasks reflect our choices.** They lean towards precise spec-following and careful reasoning in Python, JavaScript
  and SQL. Different tasks could rank the apps differently.
- **Contamination clock.** Once published, these tasks can leak into future training data. Each new suite version gets
  new tasks and a new date, and old versions stay online for reproducibility.
- **Pilot.** Before v1, a two-task pilot on 1 November 2026 ended in a tie on easy tasks. It convinced us to make the tasks
  harder and to repeat runs. It is shown on our ChatGPT vs Claude page as a labelled single-run pilot, and v1 results
  replace it once they are in.

## Sources

- OpenAI, "Introducing SWE-bench Verified": https://openai.com/index/introducing-swe-bench-verified/
- SWE-bench Verified: https://www.swebench.com/verified.html
- OpenAI, "Why we no longer evaluate SWE-bench Verified": https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
- OpenAI, "Separating signal from noise in coding evaluations" (SWE-bench Pro audit, July 2026): https://openai.com/index/separating-signal-from-noise-coding-evaluations/
- Dan et al., "SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark": https://arxiv.org/abs/2610.24206
- Aider LLM leaderboards: https://aider.chat/docs/leaderboards/ and polyglot benchmark design: https://aider.chat/2024/12/21/polyglot.html
- LiveCodeBench: https://livecodebench.github.io/ and https://arxiv.org/abs/2403.07974
- Liu et al., "Is Your Code Generated by ChatGPT Really Correct?" (EvalPlus): https://arxiv.org/abs/2305.01210 and https://github.com/evalplus/evalplus
- Chen et al., "Evaluating Large Language Models Trained on Code" (HumanEval, pass@k): https://arxiv.org/abs/2107.03374
- Artificial Analysis, Intelligence Benchmarking methodology: https://artificialanalysis.ai/methodology/intelligence-benchmarking
- Miller (Anthropic), "Adding Error Bars to Evals": https://arxiv.org/abs/2411.00640
- LMArena FAQ: https://arena.ai/faq and "Style control": https://arena.ai/blog/style-control
- Singh et al., "The Leaderboard Illusion": https://arxiv.org/abs/2504.20879 and LMArena's response: https://arena.ai/blog/our-response
- Zhou et al., "Instruction-Following Evaluation for Large Language Models" (IFEval): https://arxiv.org/abs/2311.07911
- Simon Willison, pelicans on a bicycle: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/ and https://github.com/simonw/pelican-bicycle
- Terminal-Bench: https://www.tbench.ai/ and https://github.com/harbor-framework/terminal-bench
- Example of a typical review-site head-to-head (default models, prompts shown, no blinding or repeats): https://www.tomsguide.com/ai/chatgpt-vs-claude-i-put-both-default-models-through-7-real-world-tests-one-is-the-clear-winner

*The Agentic Weekly team*