Vercel AI SDK 6.0.315 (Node 20) / 7.0.153 (Node 24): our test log

Vercel AI SDK passed our one-tool task 3 of 3 on Node 20 (ai 6.0.315) and 3 of 3 on Node 24 (ai 7.0.153); the install is 26 to 27 MB.

Result at a glance

What we found

• 3 of 3 passed on Node 20 and 3 of 3 on Node 24, with temperature 0 set in the call.
• The two Node versions installed different major versions of the SDK: ai 6.0.315 on Node 20 and ai 7.0.153 on Node 24. The same task script ran unchanged on both, so for this task the v6 to v7 jump cost us nothing.

What we ran, step by step

# npm install on Node 20.19.2 (cold, empty cache)
cd work/npm/vercel-ai-sdk && npm install --no-audit --no-fund --cache /tmp/npmcache --loglevel=error ai @ai-sdk/openai-compatible zod

# npm install on Node 24.22.1 (cold, empty cache)
cd work/npm24/vercel-ai-sdk && npm install --no-audit --no-fund --cache /tmp/npmcache --loglevel=error ai @ai-sdk/openai-compatible zod

# Task command (all trials)
node task.mjs

Raw output (excerpts)

import { generateText, tool, stepCountIs } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { z } from "zod";
const calls = [];
const p = createOpenAICompatible({ name: "ollama", baseURL: "http://127.0.0.1:11434/v1", apiKey: "ollama" });
const r = await generateText({
  model: p("qwen2.5-3b"), temperature: 0, stopWhen: stepCountIs(4),
  tools: { add_numbers: tool({ description: "Add two integers and return the sum.", inputSchema: z.object({ a: z.number(), b: z.number() }), execute: async ({ a, b }) => { calls.push([a, b]); return a + b; } }) },
  prompt: "Use the add_numbers tool to compute 17 + 25, then tell me the result.",
});
const ok = calls.some(([a, b]) => (a === 17 && b === 25) || (a === 25 && b === 17)) && r.text.includes("42");
console.log("RESULT_JSON " + JSON.stringify({ calls, final: r.text.slice(0, 300), ok }));

Gotchas and things that broke

• On Node 20 npm resolved ai 6.0.315 without an error because the latest release declares a newer Node engine; you only see this by printing the installed version. See the research page on silent downgrades.

What we did not test

• Any vendor-hosted model: everything ran against one local 3B model.
• Multi-agent flows, memory, streaming and deployment.
• Anything beyond one tool and one question; five trials is a smoke test, not a benchmark.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

# serve a tool-calling local model at http://127.0.0.1:11434/v1 (we used Ollama with Qwen2.5-3B-Instruct, model name qwen2.5-3b)
mkdir t && cd t && npm init -y && npm install ai @ai-sdk/openai-compatible zod
# save task.mjs (above) and run it:
node task.mjs

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: agent frameworks and sdks

• Agno: Agno 3.1.10 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.8 s.
• AutoGen: AutoGen 0.7.5 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.9 s.
• CrewAI: CrewAI 1.15.30 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 8.4 s.
• DSPy: DSPy 3.4.1 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.7 s.
• Google ADK (Python): Google ADK (Python) 2.13.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.7 s.
• Haystack: Haystack 3.4.1 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 2.3 s.

All test logs · How we test

FAQ

Related

• All test logs
• Methodology and testing rig
• Study: the silent version downgrade on older Node, and other install traps
• Study: install size and first-task results across 16 frameworks
• Vercel AI SDK facts and sources