Continue CLI 1.5.47: our test log

Continue CLI 1.5.47 answered a question and fixed a one-line bug with a local 3B model through a YAML config; a 1.6 s, 81 MB install.

Result at a glance

What we found

• The one-shot question returned 42 twice (13.2 s the first time, which includes starting up, then 1.8 s).
• The edit task rewrote calc.py correctly in 26.7 s with --auto (which lets the agent act without asking). Checked by reading the file.
• Install is small (81 MB, 13 packages) because the CLI is a thin client.

What we ran, step by step

# npm install on Node 20 (cold)
npm install @continuedev/cli

Raw output (excerpts)

name: local-test
version: 1.0.0
schema: v1
models:
  - name: qwen-local
    provider: ollama
    model: qwen2.5-3b
    apiBase: http://127.0.0.1:11434
    roles: [chat]

cn --auto --config config.yaml -p 'The function add in calc.py has a bug: it subtracts. Fix it so it adds, by editing calc.py.'

The function has been updated successfully.
---
def add(a, b):
    return a + b

Gotchas and things that broke

• We set HOME and CONTINUE_GLOBAL_DIR to a scratch folder and turned telemetry off with DO_NOT_TRACK=1 so the test did not touch real config or phone home; the defaults were not measured.
• This is the Continue command-line tool (npm package @continuedev/cli), not the VS Code or JetBrains extension, which we did not run.

What we did not test

• The IDE extensions, hub-hosted configs and any hosted model.
• Reliability: the edit task ran once, so this is an existence proof, not a pass rate.

Reproduce it

Versions change. These commands install whatever is latest on the day you run them unless a version is pinned, so expect different results later. The version we measured is in the table above.

npm install @continuedev/cli
# write config.yaml as shown, create calc.py with the bug, then:
cn --auto --config config.yaml -p "The function add in calc.py has a bug: it subtracts. Fix it so it adds, by editing calc.py."

How to read this test

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

See the testing rig for the hardware, the local model and the rules we follow, and the test hub for every other test.

Related tests: coding agents and clis

• goose: goose 1.56.0 edited calc.py on the second headless run with a local 3B model (19.4 s); the first run called the edit tool but its text did not match, so the file did not change.
• opencode: opencode 1.19.4 edited calc.py on the second headless run with a local 3B model; the first run tried the wrong path (src/calc.py) and failed.
• OpenHands CLI: OpenHands 1.16.0 installed (14 s, 537 MB) and started its agent, but with a local 3B model it made 0 edits in 2 trials: calc.py was unchanged.
• Aider: Aider 0.86.2 implemented a missing function with a local 3B model in 3 of 3 trials (27 to 32 s each); it did not install on Python 3.13 without a C++ compiler, and it left a stray placeholder comment in the file.
• Claude Code: Claude Code 2.1.310 edited calc.py through its Edit tool with a local 3B model on the second prompt (85.9 s); the first, vaguer prompt made no edit.
• Cline CLI: Cline CLI 3.0.73 edited calc.py correctly with a local 3B model, then failed on every follow-up command with 'Invalid input' until the session was aborted at 68 s.

All test logs · How we test

FAQ

Sources

• Continue repository (fetched 2026-10-25)

Related

• All test logs
• Methodology and testing rig
• Study: install size and first-task results across 16 frameworks
• Study: the silent version downgrade on older Node, and other install traps