Hugging Face huggingface_hub 2.3.0, transformers 5.20.2, torch 2.14.1 (CPU): our test log

Hugging Face's Hub searched without a login in 0.1 s, a 272 MB model downloaded in 3.61 s, and transformers ran it on CPU.

Result at a glance

What we found

• Anonymous Hub search worked and returned the Gemma 4 repositories.
• The anonymous download of 272.5 MB took 3.61 s.
• The 135M model produced a fluent but made-up sentence about the sky; we only checked that it ran, not its quality.

What we ran, step by step

# Install with uv (cold, no cache)
uv venv -q -p 3.13 work/py/hugging-face && VIRTUAL_ENV=work/py/hugging-face uv pip install --no-cache -q huggingface_hub transformers torch --extra-index-url https://download.pytorch.org/whl/cpu && du -sm work/py/hugging-face | cut -f1 && VIRTUAL_ENV=work/py/hugging-face uv pip list 2>/dev/null | grep -i -E '^(huggingface.hub|transformers|torch) '

# Task command
work/py/hugging-face/bin/python tasks3/hf_task.py

Raw output (excerpts)

{
 "search_hits": [
  "google/gemma-4-31B-it",
  "google/gemma-4-26B-A4B-it",
  "google/gemma-4-E4B-it",
  "google/gemma-4-E2B-it",
  "google/gemma-4-12B-it"
 ],
 "reply": "The sky is a breathtaking canvas of colors, with hues of pink, orange, and purple that dance across the horizon, painting the sky with a mesmerizing glow."
}

import time, json, os
os.environ["HF_HOME"] = "/workspace/tests2/work/hf-home"
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
res = {}
from huggingface_hub import HfApi, snapshot_download
t = time.perf_counter(); ms = list(HfApi().list_models(search="gemma-4", sort="downloads", limit=5)); res["search_s"] = round(time.perf_counter() - t, 2); res["search_hits"] = [m.id for m in ms][:5]
t = time.perf_counter(); p = snapshot_download("HuggingFaceTB/SmolLM2-135M-Instruct", allow_patterns=["*.json", "*.safetensors", "*.txt"]); res["download_s"] = round(time.perf_counter() - t, 2)
res["files"] = sorted(os.listdir(p)); res["size_mb"] = round(sum(os.path.getsize(os.path.join(p, f)) for f in os.listdir(p)) / 1e6, 1)
t = time.perf_counter()
from transformers import pipeline
pipe = pipeline("text-generation", model=p, device="cpu"); res["load_s"] = round(time.perf_counter() - t, 2)
t = time.perf_counter()
out = pipe([{"role": "user", "content": "Give me one short sentence about the sky."}], max_new_tokens=40, do_sample=False)
res["gen_s"] = round(time.perf_counter() - t, 2); res["gen_text"] = out[0]["generated_text"][-1]["content"][:200]
res["ok"] = bool(res["gen_text"].strip()) and len(res["search_hits"]) > 0
print("RESULT_JSON " + json.dumps(res))

Gotchas and things that broke

• We pointed uv at the PyTorch CPU index; we did not measure the install with the default torch wheel, so 921 MB applies to the CPU build only.
• The Hub is a hosting service: what you run depends on each model's own licence, which varies by repository.

What we did not test

• Gated models, private repositories, uploads, Spaces and Inference Providers: all need an account or token.
• Download speed for large (multi-GB) models.
• Any model beyond one 135M-parameter model; quality of its output was not assessed.
• Timing of generation: the machine was heavily shared.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

uv venv -p 3.13 .venv && VIRTUAL_ENV=.venv uv pip install huggingface_hub transformers torch --extra-index-url https://download.pytorch.org/whl/cpu
.venv/bin/hf download HuggingFaceTB/SmolLM2-135M-Instruct

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: local models and runtimes

• llama.cpp: llama.cpp b11889 generated 118 tokens/s (0.5B), 40 (1.5B) and 19.5 (3B) on 8 CPU threads with llama-bench, and its server passed the same prompts and tool call as Ollama.
• Ollama: Ollama 0.40.11 served three Qwen2.5 sizes over its OpenAI-compatible API on 8 CPU cores: all three answered simple prompts; the 0.5B model failed JSON extraction while 1.5B and 3B passed, and tool calls worked 5 of 5 on all three.
• GPT4All: GPT4All's Python SDK 2.8.2 loaded a local 3B GGUF in 4.01 s and answered 17 + 25 with 42 in 3 of 3 trials, but kept writing past the answer.

All test logs · How we test

FAQ

Sources

• Hugging Face Hub documentation (fetched 2026-10-26)
• huggingface_hub library (fetched 2026-10-26)

Related

• All test logs
• Methodology and testing rig
• Hugging Face facts and sources