Writing test: 15 AI models, same prompts, rated blind

We gave fifteen open-weight AI models (8 original models plus 7 added in a second round) 14 writing tasks (one- and two-sentence answers, a product blurb, email replies, a summary and beginner explanations), 3 runs each, and had a different model (Mistral 7B Instruct v0.3) rate them blind on a published rubric and in A/B comparisons with the answer order swapped. Llama 3.2 1B came first with a 67% (57% to 76%) win rate against the eight original models. Only 11 of the 15 have finished the A/B games; the other 4 were rated on the rubric and code-checked rules only. The rater is a small 7B model that was often inconsistent when the answer order was swapped, so the ranking is a rough guide. These are small local models, not ChatGPT or Claude, and the results are indicative, not a benchmark.

Leaderboard

Full breakdown by criterion and a head-to-head grid: writing leaderboard. Instruction-following results for the same models: instruction following.

Added models rated without pairwise games

Rubric means of these models are comparable with the rubric column above (same rater, same outputs-per-model), but they are single-rater scores and the interval-free rubric ordering is weaker evidence than the blind win rate.

What we found

• Llama 3.2 1B had the best blind pairwise win rate, 67% (57% to 76%) over 112 games; Qwen2.5 0.5B had the lowest, 19% (13% to 27%). The intervals come from resampling the 14 prompts, so treat neighbouring ranks as ties unless the intervals are far apart.
• Caution on the top spot: Llama 3.2 1B (1B) ranks ahead of 4 models more than twice its size. We read that as a limit of a 7B rater judging short answers (see the judge checks below), not as proof that it writes better; the win-rate interval is wide and overlaps those of several other models.
• Of 52 head-to-head pairs, 18 have a clear winner (an exact sign test on wins vs losses over 14 prompts gives p below 0.05) and 34 do not separate the models at this sample size.
• Within the Qwen2.5 family, win rate by size was 0.5B 19%, 1.5B 34%, 3B 44%, 7B 55%.
• Code-checked format rules (exact sentence counts, word ranges): best was Llama 3.2 3B with 42 of 42 outputs passing; lowest was Qwen2.5 0.5B with 19 of 42. Counting sentences is the thing small models most often get wrong.
• Judge sanity checks: the two answer orders gave the same verdict in 433 of 728 comparisons (59%); when they disagreed we scored a tie. The longer answer won 324 of 422 decided games that differed in length, so the judge may favour length. Outputs that passed our code-checked format rule averaged 4.97 on the rubric's accuracy criterion, and those that failed averaged 4.92.
• What this means: the 7B rater is a weak judge. It changed its mind when we swapped the answer order in 41% of comparisons, it leans towards longer answers, and its rubric accuracy score did not separate outputs that broke the stated rule from those that kept it, so we treat the pairwise ranking as indicative and show the code-checked format column next to it. The format column does not depend on any model.

How the test works

1. Fourteen prompts we wrote ourselves, sent unchanged to every model as the only message (no system prompt): four one-sentence tasks, three two-sentence tasks, two product blurbs, two email replies, one summary of a supplied paragraph and two beginner explanations. See every prompt.
2. Each model answers each prompt 3 times (temperature 0.7, top-p 0.9, fixed seeds, up to 400 tokens). Nothing is cherry-picked; all 630 outputs are published.
3. A separate rater model scores every output alone on the five-criterion rubric below, without being told which model wrote it.
4. The same rater compares pairs of outputs from two different models, blind, labelled A and B. Every pair is judged twice with the A/B order swapped. If both orders pick the same model it wins; if the orders disagree or the rater says tie, we score a tie.
5. We also check format rules by code (for example "exactly two sentences" or "40 to 60 words") and report them separately from the rater's opinion.

630 writing outputs (15 models x 14 prompts x 3 runs), 630 single-answer rubric ratings, 1456 blind pairwise judgments (728 head-to-head games after combining the two answer orders).

Which model rated which, and sample sizes

The rater is Mistral 7B Instruct v0.3 (7B, Q4_K_M), chosen because it is a different model family from every writer. No writer is from the Mistral family, so the rater never rated its own family. We did not run a second rater, so every score depends on one 7B judge.

The prompts

The rubric and the rater's exact prompts

You are a strict, impartial writing evaluator. Rate ONE response to a writing task on five criteria, each from 1 (poor) to 5 (excellent). Use the whole scale; 3 is adequate, 5 is rare. Judge only the text, not who wrote it.
Criteria:
1. Clarity: Easy to read and understand on the first pass; no confusing wording; ideas in a sensible order.
2. Accuracy to the prompt: Does exactly what the task asked, obeys every stated limit (length, number of sentences, format), and stays consistent with any facts the task supplies; invents nothing.
3. Concision: No filler, repetition, preamble or sign-off that the task did not ask for; every sentence earns its place.
4. Tone: Register and voice fit the audience and purpose (for example warm and apologetic for a customer, plain for a beginner, upbeat for a blurb).
5. Originality: Fresh, specific wording rather than stock phrases and cliches; a concrete detail or angle where the task allows one.
Reply with only a JSON object: {"clarity":n,"accuracy":n,"concision":n,"tone":n,"originality":n}.

---
TASK GIVEN TO THE WRITER:
{task}
{facts}
RESPONSE TO RATE:
{a}

Rate the response.

You are a strict, impartial writing evaluator. Two anonymous responses (A and B) answer the same writing task. Decide which is the better piece of writing overall, weighing these five criteria equally:
1. Clarity: Easy to read and understand on the first pass; no confusing wording; ideas in a sensible order.
2. Accuracy to the prompt: Does exactly what the task asked, obeys every stated limit (length, number of sentences, format), and stays consistent with any facts the task supplies; invents nothing.
3. Concision: No filler, repetition, preamble or sign-off that the task did not ask for; every sentence earns its place.
4. Tone: Register and voice fit the audience and purpose (for example warm and apologetic for a customer, plain for a beginner, upbeat for a blurb).
5. Originality: Fresh, specific wording rather than stock phrases and cliches; a concrete detail or angle where the task allows one.
Do not prefer a response because it is longer or appears first. A response that breaks a stated limit should lose to one that obeys it unless it is much worse on the other criteria. If they are equally good, answer "tie".
Reply with only a JSON object: {"winner":"A"}, {"winner":"B"} or {"winner":"tie"}.

---
TASK GIVEN TO THE WRITERS:
{task}
{facts}
RESPONSE A:
{a}

RESPONSE B:
{b}

Which response is better?

Both prompts request structured JSON only. The rater runs at temperature 0 with a fixed seed.

What was and was not included

• Models ranked by blind win rate: 11, plus 4 added with rubric and instruction-following results only. 8 original models were judged against each other in a full round robin. 7 added models were each judged against the 8 originals (the headline win rate) but not against each other, so a win rate for an added model is directly comparable with an original's, and added models are compared with each other only through their records against the same eight opponents.
• Pair pages: 28 of 52 judged pairs have their own page (the original-vs-original pairs). Pairs with added models are in the head-to-head grid, the lists on this page and the raw data, and have no page of their own.
• Started but not ranked (the run was not completed in the time available, so they are not in any table; partial outputs are in the raw files): Llama 3.1 8B Instruct (0 of 42 outputs generated), Gemma 3 4B Instruct (30 of 42 outputs generated), Gemma 4 E2B Instruct (0 of 42 outputs generated), Gemma 4 E4B Instruct (0 of 42 outputs generated), Qwen3 4B (non-thinking) (0 of 42 outputs generated), Phi-4 mini Instruct (0 of 42 outputs generated).
• Not run by us: DeepSeek-R1 distills and Gemma 2 9B. This machine has 8 shared CPU cores and 15 GB of RAM, and the run already took many hours; the omission is about our time budget and is not a statement about those models.
• Mistral 7B v0.3 is the rater, so it is not a writer; it appears only in the instruction-following test.

Every head-to-head pair (28 have their own page)

• Qwen2.5 0.5B vs Qwen2.5 1.5B: no clear winner (2 to 8, 4 ties)
• Qwen2.5 0.5B vs Qwen2.5 3B: Qwen2.5 3B wins (0 to 11, 3 ties)
• Qwen2.5 0.5B vs Qwen2.5 7B: no clear winner (2 to 7, 5 ties)
• Qwen2.5 0.5B vs Llama 3.2 3B: Llama 3.2 3B wins (1 to 9, 4 ties)
• Qwen2.5 0.5B vs Gemma 2 2B: Gemma 2 2B wins (0 to 11, 3 ties)
• Qwen2.5 0.5B vs Phi-3.5 mini: Phi-3.5 mini wins (0 to 10, 4 ties)
• Qwen2.5 0.5B vs SmolLM2 1.7B: SmolLM2 1.7B wins (0 to 9, 5 ties)
• Qwen2.5 0.5B vs Llama 3.2 1B (no separate page): Llama 3.2 1B wins (0 to 12, 2 ties)
• Qwen2.5 0.5B vs Gemma 3 1B (no separate page): no clear winner (2 to 9, 3 ties)
• Qwen2.5 0.5B vs Qwen3 0.6B (non-thinking) (no separate page): no clear winner (4 to 6, 4 ties)
• Qwen2.5 1.5B vs Qwen2.5 3B: no clear winner (1 to 4, 9 ties)
• Qwen2.5 1.5B vs Qwen2.5 7B: no clear winner (1 to 7, 6 ties)
• Qwen2.5 1.5B vs Llama 3.2 3B: Llama 3.2 3B wins (0 to 7, 7 ties)
• Qwen2.5 1.5B vs Gemma 2 2B: Gemma 2 2B wins (1 to 9, 4 ties)
• Qwen2.5 1.5B vs Phi-3.5 mini: Phi-3.5 mini wins (1 to 9, 4 ties)
• Qwen2.5 1.5B vs SmolLM2 1.7B: no clear winner (2 to 7, 5 ties)
• Qwen2.5 1.5B vs Llama 3.2 1B (no separate page): Llama 3.2 1B wins (2 to 12, 0 ties)
• Qwen2.5 1.5B vs Gemma 3 1B (no separate page): Gemma 3 1B wins (0 to 9, 5 ties)
• Qwen2.5 1.5B vs Qwen3 0.6B (non-thinking) (no separate page): no clear winner (6 to 4, 4 ties)
• Qwen2.5 3B vs Qwen2.5 7B: no clear winner (1 to 5, 8 ties)
• Qwen2.5 3B vs Llama 3.2 3B: Llama 3.2 3B wins (1 to 10, 3 ties)
• Qwen2.5 3B vs Gemma 2 2B: no clear winner (2 to 5, 7 ties)
• Qwen2.5 3B vs Phi-3.5 mini: Phi-3.5 mini wins (0 to 6, 8 ties)
• Qwen2.5 3B vs SmolLM2 1.7B: no clear winner (2 to 6, 6 ties)
• Qwen2.5 3B vs Llama 3.2 1B (no separate page): no clear winner (3 to 8, 3 ties)
• Qwen2.5 3B vs Gemma 3 1B (no separate page): Gemma 3 1B wins (1 to 8, 5 ties)
• Qwen2.5 3B vs Qwen3 0.6B (non-thinking) (no separate page): no clear winner (8 to 2, 4 ties)
• Qwen2.5 7B vs Llama 3.2 3B: no clear winner (3 to 3, 8 ties)
• Qwen2.5 7B vs Gemma 2 2B: no clear winner (2 to 3, 9 ties)
• Qwen2.5 7B vs Phi-3.5 mini: no clear winner (2 to 4, 8 ties)
• Qwen2.5 7B vs SmolLM2 1.7B: no clear winner (3 to 5, 6 ties)
• Qwen2.5 7B vs Llama 3.2 1B (no separate page): no clear winner (0 to 4, 10 ties)
• Qwen2.5 7B vs Gemma 3 1B (no separate page): no clear winner (4 to 3, 7 ties)
• Qwen2.5 7B vs Qwen3 0.6B (non-thinking) (no separate page): no clear winner (8 to 2, 4 ties)
• Llama 3.2 3B vs Gemma 2 2B: no clear winner (5 to 3, 6 ties)
• Llama 3.2 3B vs Phi-3.5 mini: no clear winner (2 to 2, 10 ties)
• Llama 3.2 3B vs SmolLM2 1.7B: no clear winner (3 to 2, 9 ties)
• Llama 3.2 3B vs Llama 3.2 1B (no separate page): no clear winner (3 to 1, 10 ties)
• Llama 3.2 3B vs Gemma 3 1B (no separate page): no clear winner (4 to 2, 8 ties)
• Llama 3.2 3B vs Qwen3 0.6B (non-thinking) (no separate page): Llama 3.2 3B wins (10 to 0, 4 ties)
• Gemma 2 2B vs Phi-3.5 mini: no clear winner (2 to 3, 9 ties)
• Gemma 2 2B vs SmolLM2 1.7B: no clear winner (6 to 1, 7 ties)
• Gemma 2 2B vs Llama 3.2 1B (no separate page): no clear winner (3 to 6, 5 ties)
• Gemma 2 2B vs Gemma 3 1B (no separate page): no clear winner (1 to 4, 9 ties)
• Gemma 2 2B vs Qwen3 0.6B (non-thinking) (no separate page): Gemma 2 2B wins (9 to 1, 4 ties)
• Phi-3.5 mini vs SmolLM2 1.7B: no clear winner (4 to 2, 8 ties)
• Phi-3.5 mini vs Llama 3.2 1B (no separate page): no clear winner (4 to 4, 6 ties)
• Phi-3.5 mini vs Gemma 3 1B (no separate page): no clear winner (6 to 2, 6 ties)
• Phi-3.5 mini vs Qwen3 0.6B (non-thinking) (no separate page): Phi-3.5 mini wins (10 to 0, 4 ties)
• SmolLM2 1.7B vs Llama 3.2 1B (no separate page): no clear winner (1 to 7, 6 ties)
• SmolLM2 1.7B vs Gemma 3 1B (no separate page): no clear winner (5 to 2, 7 ties)
• SmolLM2 1.7B vs Qwen3 0.6B (non-thinking) (no separate page): SmolLM2 1.7B wins (9 to 1, 4 ties)

Limitations

• Every writer is a small open-weight model (0.5B to 8B parameters, mostly Q4_K_M quantization) run locally on a CPU. We have no API access to frontier models, so ChatGPT, Claude, Gemini, Grok and similar products are not in this test, and nothing here says how they compare.
• The rater is itself a 7B model, so it is a noisy judge. We published its exact prompts and both the pairwise and rubric results. It was not a writer, so it never rated its own text, but it may share blind spots with writers from other families.
• 14 prompts and 3 sampled runs each is a small sample. The leaderboard intervals reflect prompt-to-prompt variation only; they do not cover rater error.
• Pairwise games use one sampled run per prompt and pair (the same run index for both writers), judged in both orders. The rubric ratings cover every output.
• Sentence counts use a simple splitter (a full stop, question mark or exclamation mark followed by a space or the end) and can be off for abbreviations; our prompts avoid them.
• Speed figures (tokens per second) were measured on a shared machine whose load average ranged widely during the run, so they are indicative only. Expect different numbers on other hardware.
• Prompts are in English and short. Nothing here measures long-form writing, factual recall, safety or multilingual ability.

Raw data and code

• generations.jsonl: every writing output with prompt id, run, seed, tokens, timings and machine load
• judgments-rubric.jsonl: the rater's rubric scores for every output
• judgments-pairwise.jsonl: every blind A/B verdict, both orders
• ifeval.jsonl: every instruction-following output with each constraint's pass or fail
• prompts.json: all prompts, rule checks, the rubric and the rater prompts
• results.json and ifeval-results.json: the aggregated numbers behind the pages
• Harness code: config.mjs, generate.mjs, judge.mjs, rubric.mjs, ifchecks.mjs, ifrun.mjs, aggregate.mjs

# models are served by Ollama; add a contestant with one line in config.mjs
node scripts/tests/writing/generate.mjs      # 14 prompts x 3 runs per model (resumes)
node scripts/tests/writing/judge.mjs rubric   # single-answer rubric ratings
node scripts/tests/writing/judge.mjs pairwise # blind A/B, both orders
node scripts/tests/writing/ifrun.mjs          # instruction-following test
node scripts/tests/writing/aggregate.mjs && node scripts/tests/writing/build-pages.mjs

FAQ

Related

• Writing test hub
• Writing leaderboard
• Instruction-following test
• Methodology and testing rig