Semantic Kernel 1.45.0: our test log

Semantic Kernel 1.45.0 returned an empty reply with our standard prompt (0 of 3) but worked once the prompt used its plugin-prefixed tool name (3 of 3); a naming trap with small models.

Tested by us on 2026-10-25. Local open-weight model, not the vendor's hosted models. A smoke test on a CPU-only box, not a benchmark.

Result at a glance

Semantic Kernel test record
FieldValue
ResultPartly worked
Version tested1.45.0
Date and time2026-10-25, 23:20 to 23:20 CDT (America/Chicago)
TaskGive a local model one Python function, add_numbers(a, b), and ask it to use that tool to compute 17 + 25 and report the result. Pass means the function was called with 17 and 25 and the final answer contains 42. Three trials per tool, all on the same local model.
Model usedQwen2.5-3B-Instruct (Q4_K_M) through Ollama 0.40.11
HardwareLinux x86_64 virtual machine (Debian 13), 8 vCPU (Intel Xeon, 1 thread per core), 15 GiB RAM, no GPU
Installed size (uv)381 MB in 95 packages
Install time (uv, cold)4.6 s
Install time (pip, cold)24.1 s
Installed size (pip)488 MB in 96 packages
Task trials passed0 of 3
Task wall time (median, includes imports)3.5 s (range 3.2 to 3.5 s)
Machine load during trials8.8 to 9.3 (1-minute load average, 8 vCPU)
Trials with the plugin-prefixed prompt3 of 3

What we found

  • 0 of 3 task trials passed as written; details in the step table and raw output below.
  • Cold install took 4.6 s with uv and 24.1 s with pip, and left 381 MB in 95 packages (uv).
  • With our standard prompt (it asks the model to use the add_numbers tool) 0 of 3 trials passed: the reply was an empty string with no tool call. We traced this: Semantic Kernel registers plugin functions as plugin-function (here math-add_numbers) and sends that name to the model. Our 3B model answered the prompt by calling add_numbers, a tool that does not exist in the request, and Ollama drops such a call, leaving an empty message.
  • When the prompt names the tool exactly as Semantic Kernel exposes it (math-add_numbers), the same code succeeded; see the step table for the second set of trials. So the framework works; the plugin-prefix naming interacts badly with a small model and a prompt written without it.

What we ran, step by step

Commands run, wall-clock time and exit code
StepWall timeExit codeNotes
Install with uv (cold, no cache)4.6 s0Machine load average before the run: 0.54.
Install with pip (cold, no cache)24.1 s0Load average before: 1.11. Second run; the first pip run was discarded (see the research page).
Task command (all trials)n/an/aRun from a directory that contains common.py (shown below). Telemetry switches were set for tools that have them.
Task trial 1 (semantic-kernel)3.5 s0FAILED. calls=[] answer='' Load 9.35.
Task trial 2 (semantic-kernel)3.5 s0FAILED. calls=[] answer='' Load 8.76.
Task trial 3 (semantic-kernel)3.2 s0FAILED. calls=[] answer='' Load 8.76.
Task trial 1 (semantic-kernel-named-prompt)4.9 s0Tool called with [[17, 25]]; answer: The result of adding 17 and 25 is 42. Load 8.78.
Task trial 2 (semantic-kernel-named-prompt)4.5 s0Tool called with [[17, 25]]; answer: The result of adding 17 and 25 is 42. Load 8.80.
Task trial 3 (semantic-kernel-named-prompt)5.1 s0Tool called with [[17, 25]]; answer: The result of adding 17 and 25 is 42. Load 8.82.
Exact commands (paths shortened)
# Install with uv (cold, no cache)
uv venv -q -p 3.13 work/py/semantic-kernel && VIRTUAL_ENV=work/py/semantic-kernel uv pip install --no-cache -q semantic-kernel

# Install with pip (cold, no cache)
uv venv -q --seed -p 3.13 work/pip/semantic-kernel && work/pip/semantic-kernel/bin/pip install --no-cache-dir -q semantic-kernel

# Task command (all trials)
OTEL_SDK_DISABLED=true work/py/semantic-kernel/bin/python tasks/semantic_kernel_agent.py

Raw output (excerpts)

tasks/semantic_kernel_agent.py, the whole script for this test
from common import *
import asyncio
from openai import AsyncOpenAI
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion, OpenAIChatPromptExecutionSettings
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from semantic_kernel.contents import ChatHistory
from semantic_kernel.functions import kernel_function
class P:
    @kernel_function(name="add_numbers", description="Add two integers and return the sum.")
    def add_numbers(self, a: int, b: int) -> int:
        return add_numbers(a, b)
k = Kernel()
svc = OpenAIChatCompletion(ai_model_id=MODEL, async_client=AsyncOpenAI(base_url=BASE, api_key=KEY), service_id="chat")
k.add_service(svc); k.add_plugin(P(), "math")
s = OpenAIChatPromptExecutionSettings(function_choice_behavior=FunctionChoiceBehavior.Auto(), temperature=0)
h = ChatHistory(); h.add_user_message(TASK)
async def main():
    return await svc.get_chat_message_content(chat_history=h, settings=s, kernel=k)
report(asyncio.run(main()))
common.py, the shared helper every framework script imports
import json, os
BASE = "http://127.0.0.1:11434/v1"
MODEL = os.environ.get("MODEL", "qwen2.5-3b")
KEY = "ollama"
TASK = "Use the add_numbers tool to compute 17 + 25, then tell me the result."
CALLS = []
def add_numbers(a: int, b: int) -> int:
    """Add two integers and return the sum."""
    CALLS.append((a, b))
    return int(a) + int(b)
def report(final):
    ok = any({int(x), int(y)} == {17, 25} for x, y in CALLS) and "42" in str(final)
    print("RESULT_JSON " + json.dumps({"calls": CALLS, "final": str(final)[:300], "ok": ok}))
Result line printed by trial 1 (2026-10-25 23:20:29 CDT)
RESULT_JSON {"calls": [], "final": "", "ok": false}
Raw request and response captured with an httpx event hook (standard prompt)
REQ {"model":"qwen2.5-3b","stream":false,"messages":[{"role":"user","content":"Use the add_numbers tool to compute 17 + 25, then tell me the result."}],"temperature":0.0,"tool_choice":"auto","tools":[{"type":"function","function":{"name":"math-add_numbers","description":"Add two integers and return the sum.","parameters":{"type":"object","properties":{"a":{"type":"integer"},"b":{"type":"integer"}},"required":["a","b"]}}}]}
RES {"id":"chatcmpl-336","object":"chat.completion","model":"qwen2.5-3b","choices":[{"index":0,"message":{"role":"assistant","content":""},"finish_reason":"stop"}],"usage":{"prompt_tokens":159,"completion_tokens":28,"total_tokens":187}}
The same request sent with curl: tool named add_numbers vs math-add_numbers, prompt says add_numbers (message field only)
tool=add_numbers      -> {"role": "assistant", "content": "", "tool_calls": [{"function": {"name": "add_numbers", "arguments": "{\"a\":17,\"b\":25}"}}]}
tool=math-add_numbers -> {"role": "assistant", "content": ""}
# prompt changed to say "math-add_numbers tool":
tool=math-add_numbers -> {"role": "assistant", "content": "", "tool_calls": [{"function": {"name": "math-add_numbers", "arguments": "{\"a\":17,\"b\":25}"}}]}

Gotchas and things that broke

  • Plugin functions are exposed to the model as pluginName-functionName. If your prompt or system message mentions the bare function name, small models may call a tool that does not exist.
  • We verified the empty reply by printing the raw HTTP request and response: the request contained the right tool schema and the response had empty content and no tool_calls.
  • The result_status of this test is Partly worked because the framework only passed once we changed the prompt; the entry on /tools stays Reported for this reason.

What we did not test

  • Any vendor-hosted model: everything ran on a local 3B model, so tool-call reliability with frontier models is not measured here.
  • Multi-agent flows, memory, long conversations, streaming, tracing and cloud deployment.
  • Anything beyond one tool and one question; three trials is a smoke test, not a benchmark.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

# 1. Serve any local model that supports tool calling behind an OpenAI-compatible endpoint, registered as qwen2.5-3b:
#    http://127.0.0.1:11434/v1  (we used Ollama with a Qwen2.5-3B-Instruct Q4_K_M GGUF)
uv venv -p 3.13 .venv && VIRTUAL_ENV=.venv uv pip install semantic-kernel
# 2. Save common.py (above) and the task script next to each other, then:
.venv/bin/python semantic_kernel_agent.py

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

  • smolagents: smolagents 1.26.0 passed our one-tool task on a local 3B model: 6 of 6 trials; cold uv install 1.5 s.
  • Vercel AI SDK: Vercel AI SDK passed our one-tool task 3 of 3 on Node 20 (ai 6.0.315) and 3 of 3 on Node 24 (ai 7.0.153); the install is 26 to 27 MB.
  • Agno: Agno 3.1.10 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.8 s.
  • AutoGen: AutoGen 0.7.5 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.9 s.
  • CrewAI: CrewAI 1.15.30 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 8.4 s.
  • DSPy: DSPy 3.4.1 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.7 s.

All test logs · How we test

FAQ

Does Semantic Kernel work with a local model?

In our run it did for this one task: 0 of 3 trials passed against a local Qwen2.5-3B model served by Ollama. We did not test other models, and results depend heavily on the model's tool-calling ability.

How big is a fresh Semantic Kernel install?

381 MB in 95 packages with uv on 1.45.0, measured on 2026-10-25.

Sources

Markdown version for agents: /tests/semantic-kernel.md