Semantic Kernel 1.45.0: our test log
Semantic Kernel 1.45.0 returned an empty reply with our standard prompt (0 of 3) but worked once the prompt used its plugin-prefixed tool name (3 of 3); a naming trap with small models.
Tested by us on 2026-10-25. Local open-weight model, not the vendor's hosted models. A smoke test on a CPU-only box, not a benchmark.
Result at a glance
| Field | Value |
|---|---|
| Result | Partly worked |
| Version tested | 1.45.0 |
| Date and time | 2026-10-25, 23:20 to 23:20 CDT (America/Chicago) |
| Task | Give a local model one Python function, add_numbers(a, b), and ask it to use that tool to compute 17 + 25 and report the result. Pass means the function was called with 17 and 25 and the final answer contains 42. Three trials per tool, all on the same local model. |
| Model used | Qwen2.5-3B-Instruct (Q4_K_M) through Ollama 0.40.11 |
| Hardware | Linux x86_64 virtual machine (Debian 13), 8 vCPU (Intel Xeon, 1 thread per core), 15 GiB RAM, no GPU |
| Installed size (uv) | 381 MB in 95 packages |
| Install time (uv, cold) | 4.6 s |
| Install time (pip, cold) | 24.1 s |
| Installed size (pip) | 488 MB in 96 packages |
| Task trials passed | 0 of 3 |
| Task wall time (median, includes imports) | 3.5 s (range 3.2 to 3.5 s) |
| Machine load during trials | 8.8 to 9.3 (1-minute load average, 8 vCPU) |
| Trials with the plugin-prefixed prompt | 3 of 3 |
What we found
- 0 of 3 task trials passed as written; details in the step table and raw output below.
- Cold install took 4.6 s with uv and 24.1 s with pip, and left 381 MB in 95 packages (uv).
- With our standard prompt (it asks the model to use the add_numbers tool) 0 of 3 trials passed: the reply was an empty string with no tool call. We traced this: Semantic Kernel registers plugin functions as plugin-function (here math-add_numbers) and sends that name to the model. Our 3B model answered the prompt by calling add_numbers, a tool that does not exist in the request, and Ollama drops such a call, leaving an empty message.
- When the prompt names the tool exactly as Semantic Kernel exposes it (math-add_numbers), the same code succeeded; see the step table for the second set of trials. So the framework works; the plugin-prefix naming interacts badly with a small model and a prompt written without it.
What we ran, step by step
| Step | Wall time | Exit code | Notes |
|---|---|---|---|
| Install with uv (cold, no cache) | 4.6 s | 0 | Machine load average before the run: 0.54. |
| Install with pip (cold, no cache) | 24.1 s | 0 | Load average before: 1.11. Second run; the first pip run was discarded (see the research page). |
| Task command (all trials) | n/a | n/a | Run from a directory that contains common.py (shown below). Telemetry switches were set for tools that have them. |
| Task trial 1 (semantic-kernel) | 3.5 s | 0 | FAILED. calls=[] answer='' Load 9.35. |
| Task trial 2 (semantic-kernel) | 3.5 s | 0 | FAILED. calls=[] answer='' Load 8.76. |
| Task trial 3 (semantic-kernel) | 3.2 s | 0 | FAILED. calls=[] answer='' Load 8.76. |
| Task trial 1 (semantic-kernel-named-prompt) | 4.9 s | 0 | Tool called with [[17, 25]]; answer: The result of adding 17 and 25 is 42. Load 8.78. |
| Task trial 2 (semantic-kernel-named-prompt) | 4.5 s | 0 | Tool called with [[17, 25]]; answer: The result of adding 17 and 25 is 42. Load 8.80. |
| Task trial 3 (semantic-kernel-named-prompt) | 5.1 s | 0 | Tool called with [[17, 25]]; answer: The result of adding 17 and 25 is 42. Load 8.82. |
# Install with uv (cold, no cache)
uv venv -q -p 3.13 work/py/semantic-kernel && VIRTUAL_ENV=work/py/semantic-kernel uv pip install --no-cache -q semantic-kernel
# Install with pip (cold, no cache)
uv venv -q --seed -p 3.13 work/pip/semantic-kernel && work/pip/semantic-kernel/bin/pip install --no-cache-dir -q semantic-kernel
# Task command (all trials)
OTEL_SDK_DISABLED=true work/py/semantic-kernel/bin/python tasks/semantic_kernel_agent.pyRaw output (excerpts)
from common import *
import asyncio
from openai import AsyncOpenAI
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion, OpenAIChatPromptExecutionSettings
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from semantic_kernel.contents import ChatHistory
from semantic_kernel.functions import kernel_function
class P:
@kernel_function(name="add_numbers", description="Add two integers and return the sum.")
def add_numbers(self, a: int, b: int) -> int:
return add_numbers(a, b)
k = Kernel()
svc = OpenAIChatCompletion(ai_model_id=MODEL, async_client=AsyncOpenAI(base_url=BASE, api_key=KEY), service_id="chat")
k.add_service(svc); k.add_plugin(P(), "math")
s = OpenAIChatPromptExecutionSettings(function_choice_behavior=FunctionChoiceBehavior.Auto(), temperature=0)
h = ChatHistory(); h.add_user_message(TASK)
async def main():
return await svc.get_chat_message_content(chat_history=h, settings=s, kernel=k)
report(asyncio.run(main()))import json, os
BASE = "http://127.0.0.1:11434/v1"
MODEL = os.environ.get("MODEL", "qwen2.5-3b")
KEY = "ollama"
TASK = "Use the add_numbers tool to compute 17 + 25, then tell me the result."
CALLS = []
def add_numbers(a: int, b: int) -> int:
"""Add two integers and return the sum."""
CALLS.append((a, b))
return int(a) + int(b)
def report(final):
ok = any({int(x), int(y)} == {17, 25} for x, y in CALLS) and "42" in str(final)
print("RESULT_JSON " + json.dumps({"calls": CALLS, "final": str(final)[:300], "ok": ok}))RESULT_JSON {"calls": [], "final": "", "ok": false}REQ {"model":"qwen2.5-3b","stream":false,"messages":[{"role":"user","content":"Use the add_numbers tool to compute 17 + 25, then tell me the result."}],"temperature":0.0,"tool_choice":"auto","tools":[{"type":"function","function":{"name":"math-add_numbers","description":"Add two integers and return the sum.","parameters":{"type":"object","properties":{"a":{"type":"integer"},"b":{"type":"integer"}},"required":["a","b"]}}}]}
RES {"id":"chatcmpl-336","object":"chat.completion","model":"qwen2.5-3b","choices":[{"index":0,"message":{"role":"assistant","content":""},"finish_reason":"stop"}],"usage":{"prompt_tokens":159,"completion_tokens":28,"total_tokens":187}}tool=add_numbers -> {"role": "assistant", "content": "", "tool_calls": [{"function": {"name": "add_numbers", "arguments": "{\"a\":17,\"b\":25}"}}]}
tool=math-add_numbers -> {"role": "assistant", "content": ""}
# prompt changed to say "math-add_numbers tool":
tool=math-add_numbers -> {"role": "assistant", "content": "", "tool_calls": [{"function": {"name": "math-add_numbers", "arguments": "{\"a\":17,\"b\":25}"}}]}Gotchas and things that broke
- Plugin functions are exposed to the model as pluginName-functionName. If your prompt or system message mentions the bare function name, small models may call a tool that does not exist.
- We verified the empty reply by printing the raw HTTP request and response: the request contained the right tool schema and the response had empty content and no tool_calls.
- The result_status of this test is Partly worked because the framework only passed once we changed the prompt; the entry on /tools stays Reported for this reason.
What we did not test
- Any vendor-hosted model: everything ran on a local 3B model, so tool-call reliability with frontier models is not measured here.
- Multi-agent flows, memory, long conversations, streaming, tracing and cloud deployment.
- Anything beyond one tool and one question; three trials is a smoke test, not a benchmark.
Reproduce it
Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.
# 1. Serve any local model that supports tool calling behind an OpenAI-compatible endpoint, registered as qwen2.5-3b:
# http://127.0.0.1:11434/v1 (we used Ollama with a Qwen2.5-3B-Instruct Q4_K_M GGUF)
uv venv -p 3.13 .venv && VIRTUAL_ENV=.venv uv pip install semantic-kernel
# 2. Save common.py (above) and the task script next to each other, then:
.venv/bin/python semantic_kernel_agent.pyHow to read this test
Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.
See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.
Related tests: agent frameworks and sdks
- smolagents: smolagents 1.26.0 passed our one-tool task on a local 3B model: 6 of 6 trials; cold uv install 1.5 s.
- Vercel AI SDK: Vercel AI SDK passed our one-tool task 3 of 3 on Node 20 (ai 6.0.315) and 3 of 3 on Node 24 (ai 7.0.153); the install is 26 to 27 MB.
- Agno: Agno 3.1.10 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.8 s.
- AutoGen: AutoGen 0.7.5 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.9 s.
- CrewAI: CrewAI 1.15.30 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 8.4 s.
- DSPy: DSPy 3.4.1 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.7 s.
FAQ
Does Semantic Kernel work with a local model?
In our run it did for this one task: 0 of 3 trials passed against a local Qwen2.5-3B model served by Ollama. We did not test other models, and results depend heavily on the model's tool-calling ability.
How big is a fresh Semantic Kernel install?
381 MB in 95 packages with uv on 1.45.0, measured on 2026-10-25.
Sources
- Semantic Kernel repository (fetched 2026-10-25)
Related
Markdown version for agents: /tests/semantic-kernel.md