Test logs: what we ran and what happened
We ran 57 tools ourselves on one CPU-only Linux box with no vendor accounts and published every command, version and raw output: 48 passed, 6 partly worked, 2 failed, 1 could not be run. These are smoke tests, not benchmarks, and they say little about vendor-hosted models.
Results at a glance
- 48 passed (84%)
- 6 partly worked (11%)
- 2 failed (4%)
- 1 could not run (2%)
- Agent frameworks and SDKs21
- MCP servers and tooling16
- Coding agents and CLIs8
- Voice, speech and image4
- Workflow automation4
- Local models and runtimes4
- Passed
- Partly worked
- Failed
- Could not run
One CPU-only machine, one small local model, one task per tool. Smoke tests, not benchmarks; every row links to the commands and raw output.
Every test at a glance
Agent frameworks and SDKs (21)
- AgnoPassed
- AutoGenPassed
- CrewAIPassed
- DSPyPassed
- Google ADK (Python)Passed
- HaystackPassed
- InstructorPassed
- LangChainPassed
- LangChain.jsPassed
- LangGraphPassed
- LiteLLMPassed
- LlamaIndexPassed
- MastraPassed
- Mem0Passed
- OpenAI Agents SDK (Python)Passed
- OpenAI Agents SDK for JavaScriptPassed
- OpenAI Python SDKPassed
- Pydantic AIPassed
- Semantic KernelPartly worked
- smolagentsPassed
- Vercel AI SDKPassed
Coding agents and CLIs (8)
MCP servers and tooling (16)
- Chrome DevTools MCPPassed
- Context7 MCPPassed
- FastMCP 4.1.3 and the official MCP Python SDK 2.3.0 (server side)Passed
- GitHub MCP ServerPassed
- MCP Everything (reference) serverPassed
- MCP Fetch serverPassed
- MCP Filesystem serverPassed
- MCP Git serverPassed
- MCP Inspector (CLI mode)Passed
- MCP Memory serverPassed
- MCP Sequential Thinking serverPassed
- MCP Time serverPassed
- Notion MCP serverPassed
- Playwright MCPPassed
- Sentry MCP serverPassed
- Supabase MCP serverPassed
Voice, speech and image (4)
Workflow automation (4)
Local models and runtimes (4)
Head-to-head model tests
Two tests where we compare models against each other on identical tasks, publish every prompt, raw output and the code, and say who rated what. Both use small open models run locally; they are indicative, not benchmarks.
- Writing test: the same one-sentence, two-sentence, blurb, email, summary and explainer prompts, rated blind by a separate model on a published rubric, with pairwise position-swapped comparisons. Leaderboard. Prompt pages: w01, w02, w03, w04, w05, w06, w07, w08, w09, w10, w11, w12, w13, w14.
- Instruction-following test: prompts with rules a program can check (word counts, bullet counts, JSON, forbidden words), pass rates over repeated runs with intervals, plus speed.
Coding agents and CLIs (8)
| Tool | Version | Result | Tested | What happened |
|---|---|---|---|---|
| Aider | 0.86.2 | Passed | 2026-10-25 | Aider 0.86.2 implemented a missing function with a local 3B model in 3 of 3 trials (27 to 32 s each); it did not install on Python 3.13 without a… |
| Claude Code | 2.1.310 | Partly worked | 2026-10-25 | Claude Code 2.1.310 edited calc.py through its Edit tool with a local 3B model on the second prompt (85.9 s); the first, vaguer prompt made no edit. |
| Cline CLI | 3.0.73 | Partly worked | 2026-10-25 | Cline CLI 3.0.73 edited calc.py correctly with a local 3B model, then failed on every follow-up command with 'Invalid input' until the session was… |
| Codex CLI | 0.164.0 | Partly worked | 2026-10-25 | Codex CLI 0.164.0 changed calc.py by running a sed command in its sandbox with a local 3B model; an earlier plain-language edit request ran no edit. |
| Continue CLI | 1.5.47 | Passed | 2026-10-25 | Continue CLI 1.5.47 answered a question and fixed a one-line bug with a local 3B model through a YAML config; a 1.6 s, 81 MB install. |
| goose | 1.56.0 | Partly worked | 2026-10-25 | goose 1.56.0 edited calc.py on the second headless run with a local 3B model (19.4 s); the first run called the edit tool but its text did not… |
| opencode | 1.19.4 | Partly worked | 2026-10-25 | opencode 1.19.4 edited calc.py on the second headless run with a local 3B model; the first run tried the wrong path (src/calc.py) and failed. |
| OpenHands CLI | 1.16.0 (SDK 1.21.0) | Failed | 2026-10-25 | OpenHands 1.16.0 installed (14 s, 537 MB) and started its agent, but with a local 3B model it made 0 edits in 2 trials: calc.py was unchanged. |
Agent frameworks and SDKs (21)
| Tool | Version | Result | Tested | What happened |
|---|---|---|---|---|
| Agno | 3.1.10 | Passed | 2026-10-25 | Agno 3.1.10 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.8 s. |
| AutoGen | 0.7.5 | Passed | 2026-10-25 | AutoGen 0.7.5 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.9 s. |
| CrewAI | 1.15.30 | Passed | 2026-10-25 | CrewAI 1.15.30 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 8.4 s. |
| DSPy | 3.4.1 | Passed | 2026-10-25 | DSPy 3.4.1 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.7 s. |
| Google ADK (Python) | 2.13.0 | Passed | 2026-10-25 | Google ADK (Python) 2.13.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.7 s. |
| Haystack | 3.4.1 | Passed | 2026-10-25 | Haystack 3.4.1 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 2.3 s. |
| Instructor | 1.17.0 | Passed | 2026-10-25 | Instructor 1.17.0 returned a validated Pydantic object from a local 3B model in 3 of 3 trials (structured extraction, not a tool call); cold uv… |
| LangChain | 1.4.7 | Passed | 2026-10-25 | LangChain 1.4.7 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 0.8 s. |
| LangChain.js | 1.5.19 | Passed | 2026-10-25 | LangChain.js 1.5.19 passed our one-tool task 3 of 3 on Node 20 and 3 of 3 on Node 24; about 101 MB in 36 packages. |
| LangGraph | 1.2.17 | Passed | 2026-10-25 | LangGraph 1.2.17 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 2.5 s. |
| LiteLLM | 1.106.4 | Passed | 2026-10-25 | LiteLLM 1.106.4 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 2.3 s. |
| LlamaIndex | 0.14.25 | Passed | 2026-10-25 | LlamaIndex 0.14.25 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.4 s. |
| Mastra | 0.24.9 (Node 20) / 1.79.0 (Node 24) | Passed | 2026-10-25 | Mastra passed 5 of 5 on Node 20 (0.24.9) and 5 of 5 on Node 24 (1.79.0) with temperature pinned to 0, after failing 5 of 6 earlier trials at default… |
| Mem0 | mem0ai 2.2.1 (open-source Python SDK) | Passed | 2026-10-26 | Mem0's open-source SDK 2.2.1 stored a fact from one chat message with a local 3B model and retrieved it with a semantic search (2.04 s) on the… |
| OpenAI Agents SDK (Python) | 0.24.0 | Passed | 2026-10-25 | OpenAI Agents SDK (Python) 0.24.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.7 s. |
| OpenAI Agents SDK for JavaScript | 0.21.0 | Passed | 2026-10-25 | OpenAI Agents SDK for JS (0.21.0) passed 2 of 5 (Node 20) and 1 of 5 (Node 24) at default sampling but 5 of 5 on both Node versions once temperature… |
| OpenAI Python SDK | 3.37.0 | Passed | 2026-10-25 | The plain OpenAI Python SDK (v3.37.0) with a hand-written tool loop passed 3 of 3 trials against a local model; it is our control for every… |
| Pydantic AI | 2.61.0 | Passed | 2026-10-25 | Pydantic AI 2.61.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.7 s. |
| Semantic Kernel | 1.45.0 | Partly worked | 2026-10-25 | Semantic Kernel 1.45.0 returned an empty reply with our standard prompt (0 of 3) but worked once the prompt used its plugin-prefixed tool name (3 of… |
| smolagents | 1.26.0 | Passed | 2026-10-25 | smolagents 1.26.0 passed our one-tool task on a local 3B model: 6 of 6 trials; cold uv install 1.5 s. |
| Vercel AI SDK | 6.0.315 (Node 20) / 7.0.153 (Node 24) | Passed | 2026-10-25 | Vercel AI SDK passed our one-tool task 3 of 3 on Node 20 (ai 6.0.315) and 3 of 3 on Node 24 (ai 7.0.153); the install is 26 to 27 MB. |
MCP servers and tooling (16)
| Tool | Version | Result | Tested | What happened |
|---|---|---|---|---|
| Chrome DevTools MCP | chrome_devtools 1.12.0 | Passed | 2026-10-25 | Chrome DevTools MCP 1.12.0 exposed 32 tools and opened a page in headless Chrome in 136.6 ms median; our first navigate_page call was rejected for a… |
| Context7 MCP | Context7 4.4.4 | Passed | 2026-10-25 | Context7 MCP 4.4.4 started locally in 462 ms but its tool calls go to a hosted API: resolve-library-id took about 1.2 s per call and returned React… |
| FastMCP 4.1.3 and the official MCP Python SDK 2.3.0 (server side) | fastmcp 4.1.3; mcp 2.3.0 | Passed | 2026-10-25 | The same three-tool server written with FastMCP 4.1.3 and with the official MCP Python SDK 2.3.0 answered calls equally fast (about 1 ms); FastMCP… |
| GitHub MCP Server | 2.0.3 | Passed | 2026-10-26 | GitHub MCP Server 2.0.3 started over stdio in a median 266 ms and listed 46 tools in a median 15 ms; we did not make any authenticated call. |
| MCP Everything (reference) server | 1.0.0 (Node 20) / 2.0.1 (Node 24) | Passed | 2026-10-25 | MCP Everything (reference) server answered over stdio in 356 ms cold (v1.0.0, Node 20) and 252 ms (v2.0.1, Node 24); median call 0.5 to 0.4 ms. |
| MCP Fetch server | mcp-fetch 2026.10.24 | Passed | 2026-10-25 | The MCP Fetch server 2026.10.24 refused to fetch a local test page by default (a deliberate loopback block); with --allow-private-ips it fetched it… |
| MCP Filesystem server | 1.0.0 (Node 20) / 2.0.1 (Node 24) | Passed | 2026-10-25 | MCP Filesystem server answered over stdio in 279 ms cold (v1.0.0, Node 20) and 187 ms (v2.0.1, Node 24); median call 0.8 to 1.0 ms. |
| MCP Git server | mcp-git 2026.10.19 | Passed | 2026-10-25 | The MCP Git server 2026.10.19 started in 512 ms, listed 12 tools and answered git_status in 3.2 ms median. |
| MCP Inspector (CLI mode) | 0.15.0 (Node 20) / 2.12.0 (Node 24) | Passed | 2026-10-25 | MCP Inspector listed the filesystem server's 14 tools from the command line on both 0.15.0 (Node 20) and 2.12.0 (Node 24) in under a second; the 2.x… |
| MCP Memory server | 1.0.0 (Node 20) / 2.0.0 (Node 24) | Passed | 2026-10-25 | MCP Memory server answered over stdio in 284 ms cold (v1.0.0, Node 20) and 147 ms (v2.0.0, Node 24); median call 0.9 to 0.6 ms. |
| MCP Sequential Thinking server | 1.0.0 (Node 20) / 2.0.0 (Node 24) | Passed | 2026-10-25 | MCP Sequential Thinking server answered over stdio in 271 ms cold (v1.0.0, Node 20) and 159 ms (v2.0.0, Node 24); median call 0.8 to 0.4 ms. |
| MCP Time server | mcp-time 2026.10.19 | Passed | 2026-10-25 | The MCP Time server 2026.10.19 started in 483 ms and answered get_current_time in 0.8 ms median. |
| Notion MCP server | 2.5.2 (@notionhq/notion-mcp-server) | Passed | 2026-10-26 | Notion MCP server 2.5.2 (@notionhq/notion-mcp-server) started over stdio in a median 316 ms and listed 24 tools in a median 16 ms; we did not make… |
| Playwright MCP | Playwright 1.64.0-alpha-1790635538000 | Passed | 2026-10-25 | Playwright MCP 0.0.83 exposed 25 tools and navigated a headless Chrome to a local page in 40.5 ms median (cold start 383 ms). |
| Sentry MCP server | 0.42.0 (@sentry/mcp-server) | Passed | 2026-10-26 | Sentry MCP server 0.42.0 (@sentry/mcp-server) started over stdio in a median 1134 ms and listed 9 tools in a median 13 ms; we did not make any… |
| Supabase MCP server | 0.14.0 (@supabase/mcp-server-supabase) | Passed | 2026-10-26 | Supabase MCP server 0.14.0 (@supabase/mcp-server-supabase) started over stdio in a median 468 ms and listed 19 tools in a median 363 ms; we did not… |
Local models, runtimes and inference (4)
| Tool | Version | Result | Tested | What happened |
|---|---|---|---|---|
| GPT4All | gpt4all 2.8.2 (Python SDK) | Passed | 2026-10-26 | GPT4All's Python SDK 2.8.2 loaded a local 3B GGUF in 4.01 s and answered 17 + 25 with 42 in 3 of 3 trials, but kept writing past the answer. |
| Hugging Face | huggingface_hub 2.3.0, transformers 5.20.2, torch 2.14.1 (CPU) | Passed | 2026-10-26 | Hugging Face's Hub searched without a login in 0.1 s, a 272 MB model downloaded in 3.61 s, and transformers ran it on CPU. |
| llama.cpp | b11889 (commit d854d98ba) | Passed | 2026-10-25 | llama.cpp b11889 generated 118 tokens/s (0.5B), 40 (1.5B) and 19.5 (3B) on 8 CPU threads with llama-bench, and its server passed the same prompts… |
| Ollama | 0.40.11 | Passed | 2026-10-25 | Ollama 0.40.11 served three Qwen2.5 sizes over its OpenAI-compatible API on 8 CPU cores: all three answered simple prompts; the 0.5B model failed… |
Voice, speech and image tools (4)
| Tool | Version | Result | Tested | What happened |
|---|---|---|---|---|
| ComfyUI | ComfyUI 0.42.0 (commit cdf272f), PyTorch 2.14.1 CPU | Passed | 2026-10-26 | ComfyUI 0.42.0 installed in 17.42 s and its CPU-only server was ready in 30.1 s with 1014 node types, but we did not generate an image. |
| faster-whisper (speech to text) | 1.2.1 (CTranslate2 4.8.4) | Passed | 2026-10-25 | faster-whisper 1.2.1 transcribed 41 s of speech at 2.4 percent word error with the base.en model in float32 (deterministic), but the int8 setting… |
| Piper (text to speech) | 1.8.0 (piper-tts) | Passed | 2026-10-25 | Piper 1.8.0 synthesised 10 sentences (about 41 s of 22.05 kHz audio) on CPU in 1.4 to 1.5 s each including process start; 3.5 s uv install, 167 MB. |
| whisper.cpp | not run | Could not be run | 2026-10-25 | We could not run whisper.cpp: no prebuilt Linux release was available for download and this machine has no make, g++ or cmake to build it. |
Workflow automation and apps (4)
| Tool | Version | Result | Tested | What happened |
|---|---|---|---|---|
| Flowise | npm latest on Node 24.22.1 | Failed | 2026-10-25 | Flowise failed to install from npm on Node 24.22.1 after 106.2 s: the better-sqlite3 native build needed a C++ toolchain this machine lacks. |
| Langflow | langflow 1.13.0 | Passed | 2026-10-26 | Langflow 1.13.0 installed in 19.76 s (1.7 GB) and its backend was healthy 91.5 s after launch, serving 26 starter flows and 416 components. |
| n8n | 2.44.6 | Passed | 2026-10-25 | n8n 2.44.6 installed from npm on Node 24 in 1 min 54 s (2,343 packages), imported a three-node workflow from the CLI and executed it: a Code node… |
| Open WebUI | 0.12.4 | Passed | 2026-10-25 | Open WebUI 0.12.4 installed with uv in 11.6 s (6,977 MB in the virtualenv), was healthy about 11 to 12 s after start, and answered a chat through… |
The rig and the rules
Hardware: Linux x86_64 virtual machine (Debian 13), 8 vCPU (Intel Xeon, 1 thread per core), 15 GiB RAM, no GPU. Model for every agent test: Qwen2.5-3B-Instruct (Q4_K_M GGUF), served by Ollama 0.40.11 at 127.0.0.1:11434 through its OpenAI-compatible API. See the testing rig.
- Every test uses no vendor account, login or API key and sends nothing to a hosted model; the one test that contacts a hosted service (Context7) says so on its page.
- A test is marked Passed only when the stated task produced the stated result; anything else is Partly worked, Failed or Could not be run.
- Wall-clock times come from the command run, with the machine load average before the run recorded in the raw data.
- We state which versions were installed, because most install commands do not pin versions.
- We do not publish a number we did not measure, and we list what each test did not cover.
Related
Markdown version for agents: /tests.md