Mastra 0.24.9 (Node 20) / 1.79.0 (Node 24): our test log

Mastra passed 5 of 5 on Node 20 (0.24.9) and 5 of 5 on Node 24 (1.79.0) with temperature pinned to 0, after failing 5 of 6 earlier trials at default sampling.

Result at a glance

What we found

• Mastra resolved 0.24.9 (236 MB, 376 packages) on Node 20 and 1.79.0 (151 MB, 196 packages) on Node 24; the Node 24 install was also about twice as fast (5.1 s against 9.9 s).
• With default sampling: 0 of 3 on Node 20 (0.24.9) and 1 of 3 on Node 24 (1.79.0), always with no tool call and an empty reply. After pinning temperature to 0: 5 of 5 on Node 20 (0.24.9) and 5 of 5 on Node 24 (1.79.0). As with the OpenAI Agents SDK for JS, the sampling temperature, not the framework, caused the failures.

What we ran, step by step

# npm install on Node 20.19.2 (cold, empty cache)
cd work/npm/mastra && npm install --no-audit --no-fund --cache /tmp/npmcache --loglevel=error @mastra/core zod

# npm install on Node 24.22.1 (cold, empty cache)
cd work/npm24/mastra && npm install --no-audit --no-fund --cache /tmp/npmcache --loglevel=error @mastra/core zod

# Task command (all trials)
node task.mjs

Raw output (excerpts)

import { Agent } from "@mastra/core/agent";
import { createTool } from "@mastra/core/tools";
import { z } from "zod";
const calls = [];
const add = createTool({ id: "add_numbers", description: "Add two integers and return the sum.", inputSchema: z.object({ a: z.number(), b: z.number() }), execute: async (arg) => { const { a, b } = arg.context ?? arg; calls.push([a, b]); return { sum: a + b }; } });
const agent = new Agent({ name: "calc", instructions: "Use tools.", model: { id: "ollama/qwen2.5-3b", url: "http://127.0.0.1:11434/v1", apiKey: "ollama" }, tools: { add_numbers: add } });
const r = await agent.generate("Use the add_numbers tool to compute 17 + 25, then tell me the result.", { maxSteps: 4 });
const ok = calls.some(([a, b]) => (a === 17 && b === 25) || (a === 25 && b === 17)) && r.text.includes("42");
console.log("RESULT_JSON " + JSON.stringify({ calls, final: r.text.slice(0, 300), ok }));

const agent = new Agent({ ..., tools: [add], modelSettings: { temperature: 0 } });   // openai-agents-js
const r = await agent.generate(prompt, { maxSteps: 4, modelSettings: { temperature: 0 } });   // mastra

Gotchas and things that broke

• Mastra 1.x requires Node 22.13 or newer; on Node 20 npm quietly installed 0.24.9, a different major version with a different API surface. Tutorials written for 1.x may not match what you get.
• Agent.generate takes modelSettings per call; the model can be given as an object with id, url and apiKey for an OpenAI-compatible server.

What we did not test

• Any vendor-hosted model: everything ran against one local 3B model.
• Multi-agent flows, memory, streaming and deployment.
• Anything beyond one tool and one question; five trials is a smoke test, not a benchmark.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

# serve a tool-calling local model at http://127.0.0.1:11434/v1 (we used Ollama with Qwen2.5-3B-Instruct, model name qwen2.5-3b)
mkdir t && cd t && npm init -y && npm install @mastra/core zod
# save task.mjs (above) and run it:
node task.mjs

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: agent frameworks and sdks

• Mem0: Mem0's open-source SDK 2.2.1 stored a fact from one chat message with a local 3B model and retrieved it with a semantic search (2.04 s) on the second attempt; the first two runs failed on a missing package and an API change.
• OpenAI Agents SDK (Python): OpenAI Agents SDK (Python) 0.24.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 1.7 s.
• OpenAI Agents SDK for JavaScript: OpenAI Agents SDK for JS (0.21.0) passed 2 of 5 (Node 20) and 1 of 5 (Node 24) at default sampling but 5 of 5 on both Node versions once temperature was pinned to 0.
• OpenAI Python SDK: The plain OpenAI Python SDK (v3.37.0) with a hand-written tool loop passed 3 of 3 trials against a local model; it is our control for every framework test.
• Pydantic AI: Pydantic AI 2.61.0 passed our one-tool task on a local 3B model: 3 of 3 trials; cold uv install 3.7 s.
• Semantic Kernel: Semantic Kernel 1.45.0 returned an empty reply with our standard prompt (0 of 3) but worked once the prompt used its plugin-prefixed tool name (3 of 3); a naming trap with small models.

All test logs · How we test

FAQ

Related

• All test logs
• Methodology and testing rig
• Study: install size and first-task results across 16 frameworks
• Study: the silent version downgrade on older Node, and other install traps
• Mastra facts and sources