OpenHands CLI 1.16.0 (SDK 1.21.0): our test log

OpenHands 1.16.0 installed (14 s, 537 MB) and started its agent, but with a local 3B model it made 0 edits in 2 trials: calc.py was unchanged.

Result at a glance

What we found

• 0 of 2 trials changed calc.py; the agent produced one empty message and stopped without calling its file editor.
• The agent registers 7 tools and a very long system prompt with many built-in skills; a 3B model may simply be too small to drive it. That is a limit of the model we could run, not necessarily of OpenHands.

What we ran, step by step

# Install with uv on Python 3.12
uv venv -p 3.12 work/py/openhands-cli && VIRTUAL_ENV=work/py/openhands-cli uv pip install --no-cache -q openhands

# Edit task, trials 1 and 2
openhands CLI in headless mode with its LLM set to openai/qwen2.5-3b at http://127.0.0.1:11434/v1 (exact flags not preserved)

Raw output (excerpts)

Initializing agent...
✓ Agent initialized with model: openai/qwen2.5-3b
Agent is working
───────────────────────────── CONVERSATION SUMMARY ─────────────────────────────
Number of agent messages: 1
Last message sent by the agent:
(empty)

def add(a, b):
    return a - b

Gotchas and things that broke

• The log of the first trial is large because the CLI prints the whole system prompt in a panel.
• Counts as Failed in this setup, so the tool entry stays Reported.

What we did not test

• A larger or hosted model, the GUI/Docker runtime and sandboxed execution.
• Why the model returned an empty message.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

uv venv -p 3.12 .venv && VIRTUAL_ENV=.venv uv pip install openhands
# configure an OpenAI-compatible endpoint at http://127.0.0.1:11434/v1 with model openai/qwen2.5-3b and run the CLI headless on a folder containing calc.py

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: coding agents and clis

• Aider: Aider 0.86.2 implemented a missing function with a local 3B model in 3 of 3 trials (27 to 32 s each); it did not install on Python 3.13 without a C++ compiler, and it left a stray placeholder comment in the file.
• Claude Code: Claude Code 2.1.310 edited calc.py through its Edit tool with a local 3B model on the second prompt (85.9 s); the first, vaguer prompt made no edit.
• Cline CLI: Cline CLI 3.0.73 edited calc.py correctly with a local 3B model, then failed on every follow-up command with 'Invalid input' until the session was aborted at 68 s.
• Codex CLI: Codex CLI 0.164.0 changed calc.py by running a sed command in its sandbox with a local 3B model; an earlier plain-language edit request ran no edit.
• Continue CLI: Continue CLI 1.5.47 answered a question and fixed a one-line bug with a local 3B model through a YAML config; a 1.6 s, 81 MB install.
• goose: goose 1.56.0 edited calc.py on the second headless run with a local 3B model (19.4 s); the first run called the edit tool but its text did not match, so the file did not change.

All test logs · How we test

FAQ

Sources

• OpenHands repository (fetched 2026-10-25)

Related

• All test logs
• Methodology and testing rig
• Study: install size and first-task results across 16 frameworks