LiteLLM 1.106.4: our test log

LiteLLM 1.106.4 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 2.3 s.

Result at a glance

What we found

• 3 of 3 task trials passed: the model's tool call reached the framework, the function ran with 17 and 25, and the final answer contained 42.
• Cold install took 2.3 s with uv and 14.1 s with pip, and left 225 MB in 58 packages (uv).
• Used the Python SDK (litellm.completion) with a hand-written tool loop, not the proxy server; LiteLLM is the translation layer between providers, not an agent framework.
• With a plain uv pip install litellm (1.106.4) the import itself failed with ModuleNotFoundError: No module named 'mcp' (the package imports it from proxy/_experimental/mcp_server but does not declare it). Installing mcp (2.3.0) fixed it. The three failed runs are kept in the raw data under the name litellm-bare-install.

What we ran, step by step

# Install with uv (cold, no cache)
uv venv -q -p 3.13 work/py/litellm && VIRTUAL_ENV=work/py/litellm uv pip install --no-cache -q litellm

# Install with pip (cold, no cache)
uv venv -q --seed -p 3.13 work/pip/litellm && work/pip/litellm/bin/pip install --no-cache-dir -q litellm

# Install mcp on top (workaround)
VIRTUAL_ENV=work/py/litellm uv pip install --no-cache -q mcp

# Task command (all trials)
OTEL_SDK_DISABLED=true work/py/litellm/bin/python tasks/litellm_agent.py

Raw output (excerpts)

from common import *
import json, litellm
litellm.telemetry = False
tools = [{"type":"function","function":{"name":"add_numbers","description":"Add two integers and return the sum.","parameters":{"type":"object","properties":{"a":{"type":"integer"},"b":{"type":"integer"}},"required":["a","b"]}}}]
msgs = [{"role":"user","content":TASK}]
final = ""
for _ in range(4):
    r = litellm.completion(model="openai/"+MODEL, api_base=BASE, api_key=KEY, messages=msgs, tools=tools, temperature=0)
    m = r.choices[0].message
    if m.tool_calls:
        msgs.append(m.model_dump(exclude_none=True))
        for tc in m.tool_calls:
            msgs.append({"role":"tool","tool_call_id":tc.id,"content":str(add_numbers(**json.loads(tc.function.arguments)))})
    else:
        final = m.content; break
report(final)

import json, os
BASE = "http://127.0.0.1:11434/v1"
MODEL = os.environ.get("MODEL", "qwen2.5-3b")
KEY = "ollama"
TASK = "Use the add_numbers tool to compute 17 + 25, then tell me the result."
CALLS = []
def add_numbers(a: int, b: int) -> int:
    """Add two integers and return the sum."""
    CALLS.append((a, b))
    return int(a) + int(b)
def report(final):
    ok = any({int(x), int(y)} == {17, 25} for x, y in CALLS) and "42" in str(final)
    print("RESULT_JSON " + json.dumps({"calls": CALLS, "final": str(final)[:300], "ok": ok}))

RESULT_JSON {"calls": [[17, 25]], "final": "The result of adding 17 and 25 is 42.", "ok": true}

Gotchas and things that broke

• Undeclared dependency: after install, import litellm failed with No module named 'mcp' on 2026-10-25 until we ran uv pip install mcp. Pin a known-good version in production.
• The 3 passed trials ran after installing mcp; the install time shown is for litellm alone.

What we did not test

• Any vendor-hosted model: everything ran on a local 3B model, so tool-call reliability with frontier models is not measured here.
• Multi-agent flows, memory, long conversations, streaming, tracing and cloud deployment.
• Anything beyond one tool and one question; three trials is a smoke test, not a benchmark.
• The LiteLLM proxy server, budgets, keys and routing.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

# 1. Serve any local model that supports tool calling behind an OpenAI-compatible endpoint, registered as qwen2.5-3b:
#    http://127.0.0.1:11434/v1  (we used Ollama with a Qwen2.5-3B-Instruct Q4_K_M GGUF)
uv venv -p 3.13 .venv && VIRTUAL_ENV=.venv uv pip install litellm
# 2. Save common.py (above) and the task script next to each other, then:
.venv/bin/python litellm_agent.py

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: agent frameworks and sdks

• LlamaIndex: LlamaIndex 0.14.25 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.4 s.
• Mastra: Mastra passed 5 of 5 on Node 20 (0.24.9) and 5 of 5 on Node 24 (1.79.0) with temperature pinned to 0, after failing 5 of 6 earlier trials at default sampling.
• Mem0: Mem0's open-source SDK 2.2.1 stored a fact from one chat message with a local 3B model and retrieved it with a semantic search (2.04 s) on the second attempt; the first two runs failed on a missing package and an API change.
• OpenAI Agents SDK (Python): OpenAI Agents SDK (Python) 0.24.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.7 s.
• OpenAI Agents SDK for JavaScript: OpenAI Agents SDK for JS (0.21.0) passed 2 of 5 (Node 20) and 1 of 5 (Node 24) at default sampling but 5 of 5 on both Node versions once temperature was pinned to 0.
• OpenAI Python SDK: The plain OpenAI Python SDK (v3.37.0) with a hand-written tool loop passed 3 of 3 trials against a local model; it is our control for every framework test.

All test logs · How we test

FAQ

Sources

• LiteLLM repository (fetched 2026-10-25)
• LiteLLM documentation (fetched 2026-10-25)

Related

• All test logs
• Methodology and testing rig
• Study: install size and first-task results across 16 frameworks