# llama.cpp b11889 (commit d854d98ba): our test log Canonical: https://theagenticweekly.com/tests/llama-cpp · Last verified: 2026-10-25 · Published: 2026-10-25 · Author: Agentic Weekly (operated independently from Texas, US) · Evidence: Tested by us on 2026-10-25. Local open-weight model, not the vendor's hosted models. A smoke test on a CPU-only box, not a benchmark. > llama.cpp b11889 generated 118 tokens/s (0.5B), 40 (1.5B) and 19.5 (3B) on 8 CPU threads with llama-bench, and its server passed the same prompts and tool call as Ollama. ## Result at a glance Table: llama.cpp test record | Field | Value | |---|---| | Result | Passed | | Version tested | b11889 (commit d854d98ba) | | Date and time | 2026-10-25, 22:49 to 22:52 CDT (America/Chicago) | | Task | Run the prebuilt llama-bench for prompt processing (64 tokens) and generation (64 tokens) on three Qwen2.5-Instruct sizes (Q4_K_M GGUF), then serve each size with llama-server and send the same prompts we sent to Ollama. | | Model used | Qwen2.5-3B-Instruct (Q4_K_M) through Ollama 0.40.11 | | Hardware | Linux x86_64 virtual machine (Debian 13), 8 vCPU (Intel Xeon, 1 thread per core), 15 GiB RAM, no GPU | | Generation, 64 tokens (t/s) | 118.3 / 40.0 / 19.5 (0.5B / 1.5B / 3B) | | Prompt processing, 64 tokens (t/s) | 756.8 / 411.0 / 242.3 | | Threads | 8 | ## What we found - Generation speed fell steeply with model size: 118.3 t/s for 0.5B, 40.0 for 1.5B and 19.5 for 3B (stddev 11.4, 2.3 and 1.1). Prompt processing: 756.8, 411.0 and 242.3 t/s, with large run-to-run spread because the machine was shared. - One of the three llama-server 3B writing runs took 8.07 s (5.5 tokens per second) while the machine load average was above 10; the other two took 2.0 and 2.7 s. The median hides that, so we list it here. - Through the OpenAI-compatible server the pass counts matched Ollama exactly: 0.5B failed JSON extraction, 1.5B and 3B passed, tool calls 5 of 5 for all three. - llama-server's own reported speed for the writing prompt was 114.8 / 40.9 / 21.9 tokens per second (0.5B / 1.5B / 3B), in line with llama-bench. - Median wall time for the two-sentence answer: 0.37 / 2.03 / 2.68 s through llama-server, against 0.21 / 0.94 / 2.63 s through Ollama. The two servers did not write the same text (median 87 against 38 completion tokens at 1.5B), so wall time compares answers of different lengths; tokens per second is the fairer comparison. ## What we ran, step by step Table: Commands run, wall-clock time and exit code | Step | Wall time | Exit code | Notes | |---|---|---|---| | Prebuilt release binaries | n/a | 0 | No compiler needed: the release archive contains llama-bench and llama-server. Build number 11889, commit d854d98ba. | | llama-bench | n/a | 0 | Run for each of the three models. | | llama-server + prompts | n/a | 0 | Same script and prompts as the Ollama test. | Exact commands (paths shortened) ``` # Prebuilt release binaries download llama-b11889 (Linux x64 CPU) and extract # llama-bench llama-bench -m .gguf -p 64 -n 64 -t 8 # llama-server + prompts llama-server -m .gguf --port

then python modelcmp.py llamacpp- http://127.0.0.1:

/v1 ``` ## Raw output (excerpts) llama-bench results (mean +/- standard deviation, tokens per second, 8 threads) ``` model prompt processing 64 tok (t/s) generation 64 tok (t/s) 0.5B 756.8 +/- 130.9 118.3 +/- 11.4 1.5B 411.0 +/- 112.6 40.0 +/- 2.3 3B 242.3 +/- 38.5 19.5 +/- 1.1 ``` Prompt checks and median times through llama-server (same table layout as the Ollama log) ``` label capital arith json toolcall write_med_s write_tok/s start llamacpp-qwen2.5-0.5b 3/3 3/3 0/3 5/5 0.37 124.322:51 CDT llamacpp-qwen2.5-1.5b 3/3 3/3 3/3 5/5 2.03 41.622:51 CDT llamacpp-qwen2.5-3b 3/3 3/3 3/3 5/5 2.68 16.422:52 CDT ``` ## Gotchas and things that broke - A prebuilt release is the quick path; this machine has no make, g++ or cmake, so we could not compile from source and did not try. - The llama-server prompt runs happened while the load average was 5.3 to 10.1, higher than the Ollama runs (2.6 to 5.4), so do not read the wall-time difference as a server difference. - llama-bench numbers on a shared 8-vCPU VM carry wide error bars (stddev up to 130 t/s on prompt processing). Treat them as ranges. ## What we did not test - Building from source, GPU backends, longer prompts, quantisations other than Q4_K_M, speculative decoding. - Single bench run of 64 tokens; it is a smoke test and not a benchmark suite. ## Reproduce it Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above. ``` # download a Linux x64 CPU release of llama.cpp (we used b11889) and a Q4_K_M GGUF ./llama-bench -m model.gguf -p 64 -n 64 -t 8 ./llama-server -m model.gguf --port 8080 ``` ## How to read this test Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing. See the [testing rig](https://theagenticweekly.com/methodology#testing-rig) for the hardware, the local model and the rules we follow, and the [test hub](https://theagenticweekly.com/tests) for every other test. ## FAQ ### How fast is a 3B model on CPU only? About 19.5 tokens per second for generation and 242 tokens per second for prompt processing on 8 shared vCPUs, using llama.cpp b11889 and a Q4_K_M quantisation. ## Sources - [llama.cpp repository](https://github.com/ggml-org/llama.cpp) (fetched 2026-10-25) ## Related - [All test logs](https://theagenticweekly.com/tests) - [Methodology and testing rig](https://theagenticweekly.com/methodology#testing-rig) - [Study: Ollama and llama.cpp compared on three Qwen2.5 sizes](https://theagenticweekly.com/research/ollama-vs-llama-cpp-qwen-sizes) ## How to cite Agentic Weekly, "llama.cpp b11889 (commit d854d98ba): our test log", last updated 2026-10-25, https://theagenticweekly.com/tests/llama-cpp