Methodology: how Agentic Weekly picks and tests AI tools

We write every entry from official sources and label it Reported, unless we ran a small hands-on smoke test, in which case it is Tested and the page states what was run, on which model and hardware, and on what date. Rankings are editorial judgement, not benchmark results. Prices are shown only with a date and an official source.

How we pick what to cover

We cover notable new releases of AI and agent tools, drawn from vendors' own release notes, GitHub releases, documentation and announcements. We track a limited set of tools rather than the whole market, so a ranking means top of the tools we cover for that task, not best overall. Anyone can submit a tool for free; whether it is covered is an editorial decision.

Reported vs Tested

• Reported: written from official sources (release notes, README, docs, announcements) that we fetched and linked, with the date we read them. We have not run this release ourselves.
• Tested: we ran a small hands-on smoke test (install plus a short task). The page states what was run, the model, the environment, the date, and what was not tested. A smoke test shows that a tool installs and does one thing. It is not a benchmark and not a quality evaluation.

When we cannot confirm something, it appears under "What we have not verified" on the tool page.

Scope of our tests

All current Tested entries are smoke tests dated 2026-10-25. They were run by an AI agent on one Linux x86_64 machine with 8 CPUs, no GPU, and no vendor account, login or API key. Where a model was needed, we used Qwen2.5-3B-Instruct (4-bit) served locally by Ollama 0.40.11. A few tests (the FastMCP and MCP reference servers, MCP Inspector, Piper and the first LangGraph run) involved no language model at all.

This means a Tested label on a tool such as Claude Code or Codex CLI does not show how it behaves with its vendor's own hosted models, subscriptions or newest features. Each tool page lists the exact scope and the parts not tested, and links to a full test log under /tests. Our research studies compare install size, startup time and speed for some tools on this same machine; those numbers do not set rankings.

The testing rig

Every test log under /tests and every study under /research was produced on the same rig, so the numbers can be compared with each other and nowhere else.

• Every test uses no vendor account, login or API key and sends nothing to a hosted model; the one test that contacts a hosted service (Context7) says so on its page.
• A test is marked Passed only when the stated task produced the stated result; anything else is Partly worked, Failed or Could not be run.
• Wall-clock times come from the command run, with the machine load average before the run recorded in the raw data.
• We state which versions were installed, because most install commands do not pin versions.
• We do not publish a number we did not measure, and we list what each test did not cover.
• Machine load is not controlled. The box was shared with other jobs and its load average ranged from about 0.5 to 11 across runs; the load average before each run is recorded in the raw data, and we say when a timing was taken under load. Pass or fail results and sizes are more reliable than timings.
• Where a first run was contaminated or noisy we kept it, said so in the study and used the clean rerun.

Every number on this page was measured on this one machine, which also ran other jobs, so timings are indicative and vary run to run. A pass means our small task worked once or as stated, a fail means it did not work as we ran it; neither says how the tool behaves with a vendor's frontier model, on a GPU, or at scale. Where a tool could not run without an account or a GPU, we say so instead of guessing.

Test logs are written from structured records (versions, exact commands, seconds, exit codes, raw output excerpts, what was not tested). Page generation only reformats those records; it never creates a number.

Head-to-head model tests

The writing test and the instruction-following test compare models against each other. They follow the same Reported and Tested rules as everything else: a result is labelled Tested only if we ran it, and every page says which models, how many prompts and runs, and who rated what.

• Contestants are the models we can actually run: open-weight models of 0.5B to 7B parameters (Q4_K_M) on the CPU-only rig above. Hosted frontier models were not available to us and are not rated; a model on a compare page without a test says Not tested yet.
• Writing test: 14 fixed prompts (one-sentence, two-sentence, product blurb, email reply, summary, beginner explanation), 3 sampled runs per model and prompt. A separate rater model from a different family (Mistral-7B-Instruct v0.3) scores every output on a published five-criterion rubric (clarity, accuracy to the prompt, concision, tone, originality) and judges blind pairwise comparisons with the A/B order swapped; disagreement between orders is recorded as a tie.
• Format rules (exact sentence counts, word limits) are checked by code and reported separately from the rater's opinion, because a small rater is least reliable on exactly those details. We publish the rater's position bias, order consistency and agreement with the code checks.
• Instruction-following test: 32 original prompts with 52 rules verified by code, 3 runs each. We report strict pass rates with 95% intervals that resample prompts (runs of one prompt are not independent), plus latency and tokens per second. No model judges anything in this test.
• Speeds were measured on a shared machine with varying load, so tokens per second are indicative. Sample sizes are small; a gap of a few points is within noise and the pages say so.
• Prompts, rubric, rater prompts, raw outputs, scores and the harness code are published at /raw/writing/ so anyone can rerun the test and add a model with one line of config.

How rankings are decided

Ranks inside the "best for" lists are editorial judgement made when an entry is written or updated. They are not benchmark results, user ratings, popularity counts or paid positions.

• We rank only the 139 tools we track, not the whole market. "#1" means top of the tools we cover for that task, not best overall.
• The reason for each rank is the one-line blurb shown beside it. Typical reasons are how well the tool's own docs say it fits the task and how actively it is maintained according to its dated release notes.
• There is no numeric scoring formula, and we have not run quality head-to-head comparisons between ranked tools; our research studies measure install size, startup time and speed for some tools, and those numbers do not set ranks. Where a rank looks like a verdict on quality, it is our reading of official sources, not a measured result.
• The Tested label does not raise a rank. Tested means a small smoke test (install plus a short task, usually on a small local model), not a benchmark. Reported and Tested entries are ranked from the same kind of evidence.
• Payment cannot change a rank. Featured listings are labelled and are never used to order a list.
• Prices and plan details are shown only with a date and an official source. Where we could only find a third-party figure, or could not read the official page, the entry says so.
• Ranks can change when a release, price or doc changes. Each page shows when it was last verified.

Prices and dates

We show a price only with the date we read it and where. Each tool's pricing status is one of: official (read from the vendor's pricing page), partial, third-party (only a non-vendor figure found), or not verified. Pages show a last-verified date, and the sitemap's last-modified date for a page comes from that same date.

Vendor figures such as usage multipliers are reported as the vendor's claims; we do not test them.

Corrections

If something is wrong, email the address on the contact page with a source. We correct the page and update its last-updated date, and record notable changes in the changelog.

Related

• Editorial policy and disclosure
• About Agentic Weekly
• How we differ from other AI tool sites
• Changelog
• Test logs
• Research studies

Page history

• Oct 25, 2026: Page created. Tested definitions, test scope and ranking basis consolidated from the editorial policy.
• Oct 25, 2026: Added the testing rig section and updated the test scope after publishing 44 test logs and 5 research studies.
• Oct 26, 2026: Added the head-to-head model tests section (writing test and instruction-following test).