Ollama 0.40.11: our test log

Ollama 0.40.11 served three Qwen2.5 sizes over its OpenAI-compatible API on 8 CPU cores: all three answered simple prompts; the 0.5B model failed JSON extraction while 1.5B and 3B passed, and tool calls worked 5 of 5 on all three.

Result at a glance

What we found

• Capability steps are visible at this scale: the 0.5B model answered the fact and arithmetic prompts and called the tool correctly 5 of 5 times, but it returned "Order 8841" where the field should be "8841", so our JSON check failed 3 of 3. The 1.5B and 3B models passed it 3 of 3.
• Median time for the two-sentence answer: 0.21 s (0.5B), 0.94 s (1.5B), 2.63 s (3B) on 8 CPU threads, roughly 109.5, 41.8 and 20.3 completion tokens per second (completion tokens divided by wall time, so including prompt processing; Ollama's /v1 response did not include its own tokens-per-second figure).
• Tool calling through /v1 worked on every size for this one trivial tool; that says nothing about harder schemas.

What we ran, step by step

# Serve the 3B model (already installed)
ollama serve  # model qwen2.5-3b from a local GGUF

# Run the comparison script
python modelcmp.py ollama-qwen2.5-3b http://127.0.0.1:11434/v1 qwen2.5-3b  # and 1.5b, 0.5b

Raw output (excerpts)

label                     capital arith json  toolcall write_med_s  write_tok/s  start
ollama-qwen2.5-0.5b       3/3     3/3   0/3   5/5              0.21        109.522:50 CDT
ollama-qwen2.5-1.5b       3/3     3/3   3/3   5/5              0.94         41.822:50 CDT
ollama-qwen2.5-3b         3/3     3/3   3/3   5/5              2.63         20.322:51 CDT
llamacpp-qwen2.5-0.5b     3/3     3/3   0/3   5/5              0.37        124.322:51 CDT
llamacpp-qwen2.5-1.5b     3/3     3/3   3/3   5/5              2.03         41.622:51 CDT
llamacpp-qwen2.5-3b       3/3     3/3   3/3   5/5              2.68         16.422:52 CDT

```json
{
  "order": "Order 8841",
  "city": "Austin",
  "date": "2026-03-02"
}
```

{"order": "8841", "city": "Austin", "date": "2026-03-02"}

Gotchas and things that broke

• Ollama's OpenAI-compatible response omitted native timing fields, so tokens per second here are derived. For native generation speed see the llama.cpp log.
• Timings are from a shared machine whose 1-minute load average ranged from 2.6 to 5.4 during these runs.

What we did not test

• Larger models, GPUs, concurrency, context lengths beyond a few hundred tokens and quality of long answers.
• A "write" quality judgement: we only timed it.
• Three runs per prompt is a smoke test, not a benchmark.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

# import a GGUF (we could not use `ollama pull` here): write a Modelfile with `FROM ./qwen2.5-3b-instruct-q4_k_m.gguf` then
ollama create qwen2.5-3b -f Modelfile
python modelcmp.py ollama-qwen2.5-3b http://127.0.0.1:11434/v1 qwen2.5-3b

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: local models and runtimes

• GPT4All: GPT4All's Python SDK 2.8.2 loaded a local 3B GGUF in 4.01 s and answered 17 + 25 with 42 in 3 of 3 trials, but kept writing past the answer.
• Hugging Face: Hugging Face's Hub searched without a login in 0.1 s, a 272 MB model downloaded in 3.61 s, and transformers ran it on CPU.
• llama.cpp: llama.cpp b11889 generated 118 tokens/s (0.5B), 40 (1.5B) and 19.5 (3B) on 8 CPU threads with llama-bench, and its server passed the same prompts and tool call as Ollama.

All test logs · How we test

FAQ

Sources

• Ollama repository (fetched 2026-10-25)
• Qwen2.5 model family (fetched 2026-10-25)

Related

• All test logs
• Methodology and testing rig
• Study: Ollama and llama.cpp compared on three Qwen2.5 sizes