Writing leaderboard: fifteen small AI models, scores by criterion

Llama 3.2 1B leads our writing leaderboard with a 67% (57% to 76%) blind win rate and 4.49 of 5 on the rubric; Qwen2.5 0.5B is last at 19%. Fifteen small open models, 14 prompts, 3 runs each, rated by Mistral 7B Instruct v0.3. Indicative, not a benchmark.

Added models rated without pairwise games

Rubric means of these models are comparable with the rubric column above (same rater, same outputs-per-model), but they are single-rater scores and the interval-free rubric ordering is weaker evidence than the blind win rate.

Ranking by blind win rate

Models in this table have finished blind games against all eight originals. A model more than a few points ahead with non-overlapping intervals is meaningfully ahead on these prompts; overlapping intervals mean we cannot separate them.

Rubric scores by criterion (1 to 5)

Head-to-head grid

Pair pages exist for 28 of the 52 judged pairs (the rest are in the grid and raw data). Open one for the per-prompt verdicts and example outputs side by side: Qwen2.5 0.5B vs Qwen2.5 1.5B; Qwen2.5 0.5B vs Qwen2.5 3B; Qwen2.5 0.5B vs Qwen2.5 7B; Qwen2.5 0.5B vs Llama 3.2 3B; Qwen2.5 0.5B vs Gemma 2 2B; Qwen2.5 0.5B vs Phi-3.5 mini; Qwen2.5 0.5B vs SmolLM2 1.7B; Qwen2.5 1.5B vs Qwen2.5 3B; Qwen2.5 1.5B vs Qwen2.5 7B; Qwen2.5 1.5B vs Llama 3.2 3B; Qwen2.5 1.5B vs Gemma 2 2B; Qwen2.5 1.5B vs Phi-3.5 mini; Qwen2.5 1.5B vs SmolLM2 1.7B; Qwen2.5 3B vs Qwen2.5 7B; Qwen2.5 3B vs Llama 3.2 3B; Qwen2.5 3B vs Gemma 2 2B; Qwen2.5 3B vs Phi-3.5 mini; Qwen2.5 3B vs SmolLM2 1.7B; Qwen2.5 7B vs Llama 3.2 3B; Qwen2.5 7B vs Gemma 2 2B; Qwen2.5 7B vs Phi-3.5 mini; Qwen2.5 7B vs SmolLM2 1.7B; Llama 3.2 3B vs Gemma 2 2B; Llama 3.2 3B vs Phi-3.5 mini; Llama 3.2 3B vs SmolLM2 1.7B; Gemma 2 2B vs Phi-3.5 mini; Gemma 2 2B vs SmolLM2 1.7B; Phi-3.5 mini vs SmolLM2 1.7B.

Speed and length

Judge checks

• What this means: the 7B rater is a weak judge. It changed its mind when we swapped the answer order in 41% of comparisons, it leans towards longer answers, and its rubric accuracy score did not separate outputs that broke the stated rule from those that kept it, so we treat the pairwise ranking as indicative and show the code-checked format column next to it. The format column does not depend on any model.

630 writing outputs (15 models x 14 prompts x 3 runs), 630 single-answer rubric ratings, 1456 blind pairwise judgments (728 head-to-head games after combining the two answer orders).

Limitations

• Every writer is a small open-weight model (0.5B to 8B parameters, mostly Q4_K_M quantization) run locally on a CPU. We have no API access to frontier models, so ChatGPT, Claude, Gemini, Grok and similar products are not in this test, and nothing here says how they compare.
• The rater is itself a 7B model, so it is a noisy judge. We published its exact prompts and both the pairwise and rubric results. It was not a writer, so it never rated its own text, but it may share blind spots with writers from other families.
• 14 prompts and 3 sampled runs each is a small sample. The leaderboard intervals reflect prompt-to-prompt variation only; they do not cover rater error.
• Pairwise games use one sampled run per prompt and pair (the same run index for both writers), judged in both orders. The rubric ratings cover every output.
• Sentence counts use a simple splitter (a full stop, question mark or exclamation mark followed by a space or the end) and can be off for abbreviations; our prompts avoid them.
• Speed figures (tokens per second) were measured on a shared machine whose load average ranged widely during the run, so they are indicative only. Expect different numbers on other hardware.
• Prompts are in English and short. Nothing here measures long-form writing, factual recall, safety or multilingual ability.

Related

• Writing test hub
• Writing leaderboard
• Instruction-following test
• Methodology and testing rig