Ollama and llama.cpp on a CPU-only box: three Qwen2.5 sizes, same prompts
On 8 CPU threads, generation speed was 118 tokens/s for the 0.5B model, 40 for 1.5B and 19.5 for 3B (llama-bench). Ollama and llama-server gave matching pass/fail results on our prompts: the 0.5B model failed JSON extraction, the 1.5B and 3B passed it, and all three called a simple tool correctly 5 of 5 times. Tokens per second through the two servers were similar.
Measured by us on 2026-10-25. Local open-weight models on a CPU-only box; nothing is vendor-reported. Smoke tests, not benchmarks.
What we ran
Three GGUF files of Qwen2.5-Instruct (0.5B, 1.5B, 3B) quantised to Q4_K_M. Each was served by Ollama 0.40.11 and by llama-server from llama.cpp build b11889 (commit d854d98ba), then sent the same requests through the OpenAI-compatible API at temperature 0: a one-word fact, simple arithmetic, a JSON extraction, a two-sentence explanation (each 3 times) and a one-tool function-calling request (5 times). llama-bench measured prompt processing and generation for 64 tokens with 8 threads.
All numbers come from one Linux virtual machine (Debian 13, 8 vCPU Intel Xeon, 15 GiB RAM, no GPU) that also ran other people's jobs, so timings are indicative and the 1-minute load average is recorded next to each run in the underlying test logs.
The public Ollama registry was blocked on this machine, so all three sizes were imported from GGUF files downloaded from Hugging Face.
Same prompts, both servers
| Model | Server | Fact | Arithmetic | JSON extraction | Tool call | Writing median (s) | Completion tokens/s |
|---|---|---|---|---|---|---|---|
| 0.5B | Ollama | 3 of 3 | 3 of 3 | 0 of 3 | 5 of 5 | 0.21 | 109.5 |
| 0.5B | llama-server | 3 of 3 | 3 of 3 | 0 of 3 | 5 of 5 | 0.37 | 124.3 |
| 1.5B | Ollama | 3 of 3 | 3 of 3 | 3 of 3 | 5 of 5 | 0.94 | 41.8 |
| 1.5B | llama-server | 3 of 3 | 3 of 3 | 3 of 3 | 5 of 5 | 2.03 | 41.6 |
| 3B | Ollama | 3 of 3 | 3 of 3 | 3 of 3 | 5 of 5 | 2.63 | 20.3 |
| 3B | llama-server | 3 of 3 | 3 of 3 | 3 of 3 | 5 of 5 | 2.68 | 16.4 |
Tokens per second here is completion tokens divided by wall time, so it includes prompt processing; the servers also wrote different answers of different lengths at 1.5B, so compare tokens per second, not seconds. The llama-server runs happened at a higher machine load (5.3 to 10.1) than the Ollama runs (2.6 to 5.4), and one 3B llama-server writing run took 8.07 s while the load was above 10.
llama-bench speed
| Model | File size | Prompt processing | Generation |
|---|---|---|---|
| 0.5B | 485 MB | 756.8 (stddev 130.9) | 118.3 (stddev 11.4) |
| 1.5B | 1111 MB | 411.0 (stddev 112.6) | 40.0 (stddev 2.3) |
| 3B | 2099 MB | 242.3 (stddev 38.5) | 19.5 (stddev 1.1) |
The standard deviations are large because the machine was shared (up to 131 tokens/s on prompt processing). Treat the figures as ranges.
What we take from it
- Capability is not smooth with size: the 0.5B model did the trivial prompts and the tool call but failed extraction, returning "Order 8841" where the field should be "8841".
- Speed scales roughly inversely with size on CPU: about 6 times fewer tokens per second at 3B than at 0.5B.
- For these prompts Ollama and a raw llama-server behaved the same; if you pick between them, pick on packaging and API features, not on measured speed from this test.
- Tool calling worked on all three sizes, but our tool is one function with two integers. The 3B agent-framework results in our other studies show that prompt and sampling details matter more than size for that task.
Caveats
- One quantisation, one model family, one CPU, 3 to 5 runs per prompt. Not a benchmark.
- Quality was checked with simple string and JSON checks, not human judgement of the writing prompt, which we only timed.
- No GPU numbers; the machine has none.
FAQ
How fast does a 3B model run on CPU?
About 19.5 tokens per second for generation and 242 tokens per second for prompt processing on 8 shared vCPUs with llama.cpp b11889 and Q4_K_M.
Is Ollama slower than llama.cpp?
Not in our runs: tokens per second through Ollama and llama-server were similar at each model size.
Related
Page history
- : First published.
Markdown version for agents: /research/ollama-vs-llama-cpp-qwen-sizes.md