Instruction-following test: 15 AI models, 32 checkable tasks
We gave 15 open models 32 prompts, each with 1 to 3 rules a program can verify (exact word and sentence counts, bullet counts, forbidden words, JSON keys, casing), 3 runs each. Llama 3.2 3B passed 82% (71% to 93%) of trials strictly; TinyLlama 1.1B Chat 16% (7% to 28%). Small local models only; no vendor chatbots; indicative, not a benchmark.
Tested by us on 2026-10-26 (America/Chicago): open-weight models run locally on a CPU-only machine; every rule is verified by code, with no model judge. Original prompts in the style of IFEval, not the IFEval dataset. Indicative, not a benchmark.
Strict pass rate (%): every rule met
Llama 3.2 3B (winner)82% out of 100%
Gemma 2 2B80% out of 100%
Granite 3.3 2B79% out of 100%
Qwen2.5 7B77% out of 100%
Qwen3 1.7B (non-thinking)77% out of 100%
Falcon3 3B75% out of 100%
Gemma 3 1B71% out of 100%
Llama 3.2 1B68% out of 100%
Qwen2.5 3B67% out of 100%
SmolLM2 1.7B67% out of 100%
Qwen3 0.6B (non-thinking)67% out of 100%
Phi-3.5 mini66% out of 100%
Qwen2.5 1.5B53% out of 100%
Qwen2.5 0.5B34% out of 100%
TinyLlama 1.1B Chat16% out of 100%
Results
| Rank | Model | Strict passes | Strict pass rate (95% interval) | Loose rule pass rate | Median tokens/s | Median seconds per answer |
|---|---|---|---|---|---|---|
| 1 | Llama 3.2 3B | 79 of 96 | 82% (71% to 93%) | 89% | 19.5 | 3.90 |
| 2 | Gemma 2 2B | 77 of 96 | 80% (70% to 90%) | 85% | 22.2 | 2.48 |
| 3 | Granite 3.3 2B | 76 of 96 | 79% (67% to 91%) | 87% | 14.2 | 7.08 |
| 4 | Qwen2.5 7B | 74 of 96 | 77% (65% to 90%) | 85% | 5.3 | 11.22 |
| 5 | Qwen3 1.7B (non-thinking) | 74 of 96 | 77% (64% to 89%) | 85% | 32.5 | 1.59 |
| 6 | Falcon3 3B | 72 of 96 | 75% (60% to 88%) | 82% | 14.3 | 4.95 |
| 7 | Gemma 3 1B | 68 of 96 | 71% (55% to 84%) | 79% | 42.1 | 1.65 |
| 8 | Llama 3.2 1B | 65 of 96 | 68% (54% to 81%) | 80% | 39.0 | 1.75 |
| 9 | Qwen2.5 3B | 64 of 95 | 67% (52% to 81%) | 77% | 17.3 | 3.17 |
| 10 | SmolLM2 1.7B | 64 of 96 | 67% (52% to 80%) | 78% | 36.8 | 1.77 |
| 11 | Qwen3 0.6B (non-thinking) | 64 of 96 | 67% (51% to 79%) | 78% | 59.6 | 0.72 |
| 12 | Phi-3.5 mini | 63 of 96 | 66% (51% to 79%) | 76% | 13.9 | 8.33 |
| 13 | Qwen2.5 1.5B | 51 of 96 | 53% (39% to 68%) | 69% | 28.2 | 1.98 |
| 14 | Qwen2.5 0.5B | 33 of 96 | 34% (21% to 49%) | 53% | 58.1 | 0.82 |
| 15 | TinyLlama 1.1B Chat | 15 of 95 | 16% (7% to 28%) | 28% | 31.9 | 6.26 |
Why intervals come from prompts: the 3 runs of one prompt are not independent, so a plain binomial interval would be too narrow. The raw pass counts per model are published in ifeval-results.json, including Wilson intervals on the pooled counts.
What we found
- Llama 3.2 3B passed 79 of 96 trials strictly (every constraint met), 82% (71% to 93%); TinyLlama 1.1B Chat passed 15 of 95, 16% (7% to 28%). Intervals come from resampling the 32 prompts.
- Hardest prompts (pooled across models): i21 (3 of 45 passed), i29 (9 of 45 passed), i30 (13 of 45 passed), i16 (18 of 45 passed). Easiest: i14 (45 of 45), i12 (45 of 45), i15 (42 of 45).
- Hard-constraint types that trip small models most: exact_words (3 of 45), word_at_least (18 of 45), between_words (23 of 45), no_commas (80 of 135).
- Run-to-run consistency: the number of the 32 prompts a model passed (or failed) on all 3 runs ranged from 21 to 29; the rest flipped between runs at temperature 0.7.
Pass counts by rule type
| Rule type | Llama 3.2 3B | Gemma 2 2B | Granite 3.3 2B | Qwen2.5 7B | Qwen3 1.7B (non-thinking) | Falcon3 3B | Gemma 3 1B | Llama 3.2 1B | Qwen2.5 3B | SmolLM2 1.7B | Qwen3 0.6B (non-thinking) | Phi-3.5 mini | Qwen2.5 1.5B | Qwen2.5 0.5B | TinyLlama 1.1B Chat |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| all_lowercase | 6/6 | 6/6 | 4/6 | 5/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 4/6 | 6/6 | 4/6 | 6/6 | 0/6 | 0/6 |
| all_uppercase | 3/6 | 5/6 | 5/6 | 6/6 | 6/6 | 2/6 | 3/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 2/6 | 0/6 |
| between_words | 3/3 | 2/3 | 0/3 | 0/3 | 2/3 | 0/3 | 3/3 | 3/3 | 2/3 | 2/3 | 1/3 | 2/3 | 1/3 | 2/3 | 0/3 |
| bold_sections | 3/3 | 1/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 1/3 | 0/3 |
| ends_with | 6/6 | 6/6 | 2/6 | 6/6 | 6/6 | 3/6 | 6/6 | 1/6 | 4/6 | 4/6 | 4/6 | 5/6 | 4/6 | 2/6 | 0/6 |
| exact_bullets | 6/6 | 5/6 | 6/6 | 6/6 | 6/6 | 6/6 | 0/6 | 6/6 | 3/5 | 6/6 | 6/6 | 0/6 | 6/6 | 5/6 | 0/6 |
| exact_numbered | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 1/3 | 3/3 | 3/3 | 0/3 |
| exact_paragraphs | 3/3 | 3/3 | 3/3 | 1/3 | 3/3 | 3/3 | 3/3 | 2/3 | 1/3 | 3/3 | 2/3 | 1/3 | 0/3 | 0/3 | 0/3 |
| exact_sentences | 17/21 | 20/21 | 21/21 | 12/21 | 12/21 | 13/21 | 17/21 | 16/21 | 12/21 | 11/21 | 8/21 | 17/21 | 12/21 | 9/21 | 4/21 |
| exact_words | 2/3 | 0/3 | 0/3 | 1/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 |
| forbid_words | 12/12 | 12/12 | 10/12 | 12/12 | 8/12 | 11/12 | 12/12 | 12/12 | 11/12 | 8/12 | 12/12 | 12/12 | 10/12 | 6/12 | 1/12 |
| include_words | 9/9 | 8/9 | 7/9 | 7/9 | 8/9 | 8/9 | 6/9 | 6/9 | 5/9 | 8/9 | 9/9 | 6/9 | 6/9 | 6/9 | 7/9 |
| json_keys | 6/9 | 9/9 | 8/9 | 9/9 | 9/9 | 9/9 | 9/9 | 7/9 | 9/9 | 9/9 | 9/9 | 9/9 | 7/9 | 3/9 | 3/9 |
| letter_at_least | 3/3 | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 1/2 |
| max_words | 8/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 7/9 | 9/9 | 9/9 | 8/9 | 7/9 | 7/9 | 9/9 | 3/9 |
| min_words | 9/9 | 9/9 | 9/9 | 6/9 | 6/9 | 6/9 | 9/9 | 9/9 | 3/9 | 9/9 | 5/9 | 9/9 | 7/9 | 8/9 | 9/9 |
| no_commas | 9/9 | 4/9 | 6/9 | 9/9 | 7/9 | 9/9 | 7/9 | 9/9 | 6/9 | 3/9 | 2/9 | 1/9 | 2/9 | 3/9 | 3/9 |
| one_of | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 0/3 |
| placeholders_at_least | 4/6 | 3/6 | 6/6 | 6/6 | 6/6 | 5/6 | 6/6 | 3/6 | 6/6 | 4/6 | 6/6 | 6/6 | 6/6 | 3/6 | 3/6 |
| postscript | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 3/6 | 3/6 | 6/6 | 6/6 | 5/6 | 5/6 | 0/6 | 1/6 | 1/6 |
| starts_with | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 2/6 |
| title_brackets | 6/6 | 5/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 5/6 | 6/6 | 6/6 | 6/6 | 0/6 |
| word_at_least | 0/3 | 1/3 | 3/3 | 1/3 | 3/3 | 2/3 | 0/3 | 0/3 | 0/3 | 1/3 | 3/3 | 2/3 | 0/3 | 0/3 | 2/3 |
| wrap_quotes | 6/6 | 5/6 | 6/6 | 6/6 | 6/6 | 6/6 | 0/6 | 5/6 | 6/6 | 5/6 | 6/6 | 5/6 | 4/6 | 3/6 | 4/6 |
How the test works
- We wrote 32 prompts with 52 rules in all, in the style of the public IFEval idea (verifiable instructions) but entirely our own; scores are not comparable with published IFEval numbers.
- Each model answers each prompt 3 times (temperature 0.7, top-p 0.9, fixed seeds, up to 350 tokens), as the only message with no system prompt.
- A program checks every rule. A trial passes strictly only if all of its rules pass. The checkers have their own self-test with known good and bad answers (ifselftest.mjs).
- No model judges anything in this test, so the results do not depend on a rater.
Every prompt and its rules
| Prompt | Task | Rules |
|---|---|---|
| i01 | Write a short note thanking a neighbour for watering your plants. Use exactly 3 sentences. | exact sentences (n=3) |
| i02 | Explain what a firewall does. Your answer must be at most 40 words and must not use the word 'network'. | max words (n=40); forbid words (words=network) |
| i03 | List three tips for sleeping better. Use exactly 3 bullet points, each starting with '- ', and write nothing before or after the bullets. | exact bullets (n=3) |
| i04 | Describe a lighthouse in at least 60 words. Include the words 'storm' and 'beacon'. | min words (n=60); include words (words=storm/beacon) |
| i05 | Write a two-sentence description of autumn. Write it entirely in lowercase letters, with no capital letters at all. | all lowercase; exact sentences (n=2) |
| i06 | Give a one-sentence reminder to back up your files. WRITE THE WHOLE ANSWER IN CAPITAL LETTERS. | all uppercase; exact sentences (n=1) |
| i07 | Describe how to make toast without using any commas anywhere in your answer. Keep it under 60 words. | no commas; max words (n=59) |
| i08 | Write a short poem about the sea. Put a title in double angular brackets, like <<Title Here>>, somewhere in your answer. | title brackets |
| i09 | Write a friendly message inviting a friend to dinner. End your message with the exact phrase 'See you soon.' and nothing after it. | ends with (text=See you soon.) |
| i10 | Write a short apology email to a colleague who was left out of a meeting. After the main text, add a postscript that begins with 'P.S.'. | postscript (marker=P\.S\.) |
| i11 | Write a two-paragraph description of a farmers' market. Separate the two paragraphs with a blank line and write exactly 2 paragraphs. | exact paragraphs (n=2) |
| i12 | Write a template for a job-offer letter. Include at least 3 placeholders in square brackets, such as [candidate name]. | placeholders at least (n=3) |
| i13 | Give a numbered list of exactly 4 steps for changing a flat bike tyre. Use the format '1. ', '2. ', and so on, with nothing else in the answer. | exact numbered (n=4) |
| i14 | Reply with only a JSON object (no explanation) that has the keys "name" (string), "age" (integer) and "hobbies" (an array of strings) for an imaginary person. | json keys (keys=name/age/hobbies, exact=true, types={"name":"string","age":"integer","hobbies":"array"}) |
| i15 | Is the Pacific the largest ocean on Earth? Answer with exactly one word: either 'Yes' or 'No'. | one of (options=Yes/No) |
| i16 | Write a sentence about rivers that contains the word 'water' at least three times. | word at least (word=water, n=3) |
| i17 | Write a single sentence about a cat. Wrap your entire response in double quotation marks. | wrap quotes; exact sentences (n=1) |
| i18 | Write a short review of a fictional cafe. Use at least 2 bold phrases written like this. | bold sections (n=2) |
| i19 | Begin your answer with the exact words 'Honestly,' and then give one reason to learn to cook. Use exactly 2 sentences. | starts with (text=Honestly,); exact sentences (n=2) |
| i20 | Write 50 to 70 words about why bees matter. Do not use the words 'honey' or 'flowers'. | between words (lo=50, hi=70); forbid words (words=honey/flowers) |
| i21 | Write exactly 25 words describing a mountain village. Count carefully; the answer must be exactly 25 words. | exact words (n=25) |
| i22 | Describe a thunderstorm in which the letter 'z' appears at least 3 times in total. | letter at least (letter=z, n=3) |
| i23 | Reply with only a JSON object listing three programming languages under the key "languages" (array of exactly 3 strings) and one under the key "favourite" (string). | json keys (keys=languages/favourite, exact=true, types={"languages":"array","favourite":"string"}, arrayLen={"languages":3}) |
| i24 | Write a product tagline in lowercase only, without commas, in under 12 words. | all lowercase; no commas; max words (n=11) |
| i25 | Write a short story of at least 90 words about a lost key. Put a title in double angular brackets, like <<Title>>, somewhere in your answer. | title brackets; min words (n=90) |
| i26 | Give exactly 5 bullet points (each starting with '* ') on how to save money on groceries, and nothing else. | exact bullets (n=5) |
| i27 | Write a short email declining a meeting invitation. It must include the word 'unfortunately', must not include the word 'sorry', and must end with a line starting with 'P.P.S.'. | include words (words=unfortunately); forbid words (words=sorry); postscript (marker=P\.P\.S\.) |
| i28 | Reply with only a JSON object with a key "steps" holding an array of exactly 3 short strings for how to boil an egg. | json keys (keys=steps, exact=true, types={"steps":"array"}, arrayLen={"steps":3}) |
| i29 | Write about the benefits of walking. Use at least 80 words and at least 2 placeholders in square brackets like [name], and do not use any commas. | min words (n=80); placeholders at least (n=2); no commas |
| i30 | WRITE TWO SENTENCES ABOUT THE MOON IN ALL CAPITAL LETTERS, and do not use the word 'night'. | all uppercase; exact sentences (n=2); forbid words (words=night) |
| i31 | Write a message of exactly 2 sentences that starts with the word 'Dear' and ends with the exact phrase 'Best wishes.' | starts with (text=Dear); ends with (text=Best wishes.); exact sentences (n=2) |
| i32 | Write a one-line slogan for a bakery, wrapped in double quotation marks, using the word 'fresh' at least once. | wrap quotes; include words (words=fresh) |
Limitations
- All contestants are small open-weight models (0.5B to 8B, mostly Q4_K_M) on CPU. Hosted frontier models were not available to us and are not included.
- 32 prompts is a small set; one prompt can swing a model by 3%.
- Rule checks are strict by design (for example a leading preamble breaks "only a list" rules). Sentence counting uses a simple splitter.
- Speed was measured on a shared machine under varying load, so tokens per second are indicative.
- 1438 of 1440 planned trials are in the data; the others were aborted by the inference server (it stops a generation that repeats itself endlessly) and are excluded rather than counted as passes or failures, which slightly flatters the affected models. The rater model of the writing test (Mistral 7B) was not run on this test.
Raw data and code
- generations.jsonl: every writing output with prompt id, run, seed, tokens, timings and machine load
- judgments-rubric.jsonl: the rater's rubric scores for every output
- judgments-pairwise.jsonl: every blind A/B verdict, both orders
- ifeval.jsonl: every instruction-following output with each constraint's pass or fail
- prompts.json: all prompts, rule checks, the rubric and the rater prompts
- results.json and ifeval-results.json: the aggregated numbers behind the pages
- Harness code: config.mjs, generate.mjs, judge.mjs, rubric.mjs, ifchecks.mjs, ifrun.mjs, aggregate.mjs
# models are served by Ollama; add a contestant with one line in config.mjs
node scripts/tests/writing/generate.mjs # 14 prompts x 3 runs per model (resumes)
node scripts/tests/writing/judge.mjs rubric # single-answer rubric ratings
node scripts/tests/writing/judge.mjs pairwise # blind A/B, both orders
node scripts/tests/writing/ifrun.mjs # instruction-following test
node scripts/tests/writing/aggregate.mjs && node scripts/tests/writing/build-pages.mjsFAQ
Which model follows instructions best in this test?
Llama 3.2 3B Instruct had the highest strict pass rate (82% (71% to 93%)) among the 15 open models we could run. It says nothing about hosted frontier models.
How is this checked?
By code: exact word and sentence counts, bullet counts, forbidden and required words, JSON keys and types, casing, start and end phrases. No model acts as judge.
Related
Markdown version for agents: /tests/instruction-following.md