goose 1.56.0: our test log

goose 1.56.0 edited calc.py on the second headless run with a local 3B model (19.4 s); the first run called the edit tool but its text did not match, so the file did not change.

By Agentic Weekly, operated independently from Texas, US · Published Oct 25, 2026 · Last updated Oct 26, 2026 (America/Chicago) · How we work

Tested by us on 2026-10-25. Local open-weight model, not the vendor's hosted models. A smoke test on a CPU-only box, not a benchmark.

Result at a glance

goose test record
Field	Value
Result	Partly worked
Version tested	1.56.0
Date and time	2026-10-25, 17:45 to 17:46 CDT (America/Chicago)
Task	In a throwaway folder, give the tool a small bug and ask a local model to fix it through the tool's own editing. calc.py contained `def add(a,b): return a-b`. Pass means the file ended as `return a+b` after the tool's own edit.
Model used	Qwen2.5-3B-Instruct (Q4_K_M) through Ollama 0.40.11
Hardware	Linux x86_64 virtual machine (Debian 13), 8 vCPU (Intel Xeon, 1 thread per core), 15 GiB RAM, no GPU
Run 1	27.6 s, file unchanged, 3,404 tokens
Run 2	19.4 s, file edited, 3,317 tokens
Version	1.56.0

What we found

Run 2 edited the file; run 1 did not, because the model guessed the text to replace.
goose's own log marks both edit tool calls as completed; the first completed without changing the file, so a tool "success" is not proof of a changed file.
Both headless sessions finished with exit type normal (27.6 s and 19.4 s).

What we ran, step by step

Commands run, wall-clock time and exit code
Step	Wall time	Exit code	Notes
Unpack the release archive	n/a	n/a	`goose --version` printed 1.56.0. The release archive is the one the official script downloads; the script was read first.
Run 1	27.6 s	0	The edit tool ran with a `before` text that did not exist in the file: "No match found for the specified text." calc.py was unchanged. 3,404 tokens.
Run 2 (exact before and after strings)	19.4 s	0	"Edited calc.py (1 lines -> 1 lines)". 3,317 tokens.

Exact commands (paths shortened)

# Unpack the release archive
# unpacked goose-x86_64-unknown-linux-gnu.tar.bz2 with Python, because the official download_cli.sh needs bzip2, which is not installed on this box

# Run 1
goose run --no-session  # provider ollama, model qwen2.5-3b

# Run 2 (exact before and after strings)
goose run --no-session  # same provider and model

Raw output (excerpts)

Run 1 (goose-run1.log, trimmed)

▸ edit
  path calc.py
  before: def add(a, b):
  after: def add(a, b): return a + b

No match found for the specified text.

Run 2 (goose-run2.log)

▸ edit
  path calc.py
  before:     return a-b
  after:     return a+b

Edited calc.py (1 lines -> 1 lines)

Gotchas and things that broke

Giving goose the exact before and after strings is what made it work; a plain request failed.

What we did not test

Subagents, resume and limits, the desktop app, and any hosted model.
Reliability: two runs, one worked, so this is an existence proof, not a pass rate.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

# download the release archive for your platform from the goose releases page and unpack it
goose --version
# configure the ollama provider with a local model, create calc.py with `return a-b`, then:
goose run --no-session -t "In calc.py replace '    return a-b' with '    return a+b' using the edit tool"

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

opencode: opencode 1.19.4 edited calc.py on the second headless run with a local 3B model; the first run tried the wrong path (src/calc.py) and failed.
OpenHands CLI: OpenHands 1.16.0 installed (14 s, 537 MB) and started its agent, but with a local 3B model it made 0 edits in 2 trials: calc.py was unchanged.
Aider: Aider 0.86.2 implemented a missing function with a local 3B model in 3 of 3 trials (27 to 32 s each); it did not install on Python 3.13 without a C++ compiler, and it left a stray placeholder comment in the file.
Claude Code: Claude Code 2.1.310 edited calc.py through its Edit tool with a local 3B model on the second prompt (85.9 s); the first, vaguer prompt made no edit.
Cline CLI: Cline CLI 3.0.73 edited calc.py correctly with a local 3B model, then failed on every follow-up command with 'Invalid input' until the session was aborted at 68 s.
Codex CLI: Codex CLI 0.164.0 changed calc.py by running a sed command in its sandbox with a local 3B model; an earlier plain-language edit request ran no edit.

All test logs · How we test

FAQ

Does goose work with a local model?

In our run it did on the second attempt, with exact strings, through the ollama provider. The first attempt did not change the file.

Sources

goose v1.56.0 release (fetched 2026-10-25)

Markdown version for agents: /tests/goose.md