OpenAI Agents SDK for JavaScript 0.21.0: our test log

OpenAI Agents SDK for JS (0.21.0) passed 2 of 5 (Node 20) and 1 of 5 (Node 24) at default sampling but 5 of 5 on both Node versions once temperature was pinned to 0.

Result at a glance

What we found

• With default sampling the task passed 2 of 5 on Node 20 and 1 of 5 on Node 24. The failures all looked the same: no tool call and an empty reply.
• We then pinned modelSettings.temperature to 0 and re-ran: 5 of 5 on Node 20 and 5 of 5 on Node 24. So the framework was fine; the unpinned temperature made a 3B model unreliable at tool calling. This is the clearest measured example in our tests of a model setting looking like a framework fault.

What we ran, step by step

# npm install on Node 20.19.2 (cold, empty cache)
cd work/npm/openai-agents-js && npm install --no-audit --no-fund --cache /tmp/npmcache --loglevel=error @openai/agents openai zod

# npm install on Node 24.22.1 (cold, empty cache)
cd work/npm24/openai-agents-js && npm install --no-audit --no-fund --cache /tmp/npmcache --loglevel=error @openai/agents openai zod

# Task command (all trials)
node task.mjs

Raw output (excerpts)

import { Agent, run, tool, setOpenAIAPI, setTracingDisabled, setDefaultOpenAIClient } from "@openai/agents";
import OpenAI from "openai";
import { z } from "zod";
setTracingDisabled(true);
setDefaultOpenAIClient(new OpenAI({ baseURL: "http://127.0.0.1:11434/v1", apiKey: "ollama" }));
setOpenAIAPI("chat_completions");
const calls = [];
const add = tool({ name: "add_numbers", description: "Add two integers and return the sum.", parameters: z.object({ a: z.number(), b: z.number() }), execute: async ({ a, b }) => { calls.push([a, b]); return String(a + b); } });
const agent = new Agent({ name: "Calc", instructions: "Use tools.", model: "qwen2.5-3b", tools: [add] });
const r = await run(agent, "Use the add_numbers tool to compute 17 + 25, then tell me the result.");
const final = String(r.finalOutput);
const ok = calls.some(([a, b]) => (a === 17 && b === 25) || (a === 25 && b === 17)) && final.includes("42");
console.log("RESULT_JSON " + JSON.stringify({ calls, final: final.slice(0, 300), ok }));

const agent = new Agent({ ..., tools: [add], modelSettings: { temperature: 0 } });   // openai-agents-js
const r = await agent.generate(prompt, { maxSteps: 4, modelSettings: { temperature: 0 } });   // mastra

Gotchas and things that broke

• The other frameworks we tested pin temperature to 0 in the model constructor. This script did not at first, and the pass rate changed from 1 or 2 of 5 to 5 of 5 by adding one setting.
• Per its documentation the SDK targets the Responses API by default (we did not test that path); against an OpenAI-compatible local endpoint call setOpenAIAPI('chat_completions') and pass your own client (shown in the script).
• Tracing export is documented as on by default; we disabled it with setTracingDisabled(true) and did not test the default.

What we did not test

• Any vendor-hosted model: everything ran against one local 3B model.
• Multi-agent flows, memory, streaming and deployment.
• Anything beyond one tool and one question; five trials is a smoke test, not a benchmark.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

# serve a tool-calling local model at http://127.0.0.1:11434/v1 (we used Ollama with Qwen2.5-3B-Instruct, model name qwen2.5-3b)
mkdir t && cd t && npm init -y && npm install @openai/agents openai zod
# save task.mjs (above) and run it:
node task.mjs

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: agent frameworks and sdks

• OpenAI Python SDK: The plain OpenAI Python SDK (v3.37.0) with a hand-written tool loop passed 3 of 3 trials against a local model; it is our control for every framework test.
• Pydantic AI: Pydantic AI 2.61.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.7 s.
• Semantic Kernel: Semantic Kernel 1.45.0 returned an empty reply with our standard prompt (0 of 3) but worked once the prompt used its plugin-prefixed tool name (3 of 3); a naming trap with small models.
• smolagents: smolagents 1.26.0 passed our one-tool task on a local 3B model: 6 of 6 trials; cold uv install 1.5 s.
• Vercel AI SDK: Vercel AI SDK passed our one-tool task 3 of 3 on Node 20 (ai 6.0.315) and 3 of 3 on Node 24 (ai 7.0.153); the install is 26 to 27 MB.
• Agno: Agno 3.1.10 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.8 s.

All test logs · How we test

FAQ

Related

• All test logs
• Methodology and testing rig
• Study: install size and first-task results across 16 frameworks
• Study: the silent version downgrade on older Node, and other install traps
• OpenAI Agents SDK facts and sources