Reported News
Google and Artificial Analysis launch MKA, a 200-task benchmark for agents doing visual knowledge work
Google launched the Multimodal Knowledge Agent (MKA) benchmark with Artificial Analysis on Nov 10, 2026, describing it as "a new benchmark built to evaluate how agents reason across visually complex files to solve real-world knowledge work tasks." Its private evaluation set has 200 tasks written with experts from 83 occupations.
- Changes reported
- 6
- listed below, each from a source we read
- We tested
- Not tested yet
- reported from sources; we did not run it ourselves
- Gaps we flag
- 5
- things we could not verify
- Sources
- 1
- 1 primary, linked below
What changed
| # | What the source says | Status |
|---|---|---|
| 01 | Google says MKA was built with domain experts across 10 verticals and 83 occupations where daily work is visual, such as architects, biologists, underwriters and data scientists, who wrote its 200-task evaluation set. | Reported |
| 02 | Each task is a multi-step project the post says takes a seasoned professional three to eight hours, pairing a prompt and a bundle of multimodal input files with an expert rubric of roughly 70 yes/no checks, weighted 75% on factual accuracy and 25% on visual design. | Reported |
| 03 | Every model runs in the same open-source agent harness, Stirrup, with the same tools, sandbox and turn budget; an agentic autorater built on a non-Google model grades each deliverable, and the post says domain experts agreed with it 96% of the time. | Reported |
| 04 | Artificial Analysis runs the 200-task set and keeps it private so tasks cannot leak into training data or be found by search; a separate set of 10 public tasks is on Hugging Face so anyone can inspect the format and test their own agents. | Reported |
| 05 | Google's technical report, per the post, found the tested models used between 30 and 130 turns on average per task, and that models try and fail to compensate when vision is removed. | Reported |
| 06 | Google claims Gemini 4 Carbon ranks #2 in quality among the models it tested and completes tasks at 6x lower cost on average than the top-scoring model; the post says Carbon is available today via the Gemini API in Google AI Studio. | Reported |
reported by the sourcemeasured or run by usnot verified
Primary source: Google blog: Introducing the Multimodal Knowledge Agent benchmark. We are the fast, independent index; the vendor’s own page is the source of record for this release.
In the source’s words
“Most knowledge work benchmarks can be solved with text alone, without ever needing to look at an image or page layout.”
“When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time.”
What we have not verified
- We did not run MKA, the Stirrup harness or the 10 public Hugging Face tasks ourselves.
- The #2 quality ranking and 6x lower cost claim for Gemini 4 Carbon come from Google, which co-developed the benchmark; we did not see an independent leaderboard confirming them.
- The post does not name the top-scoring model, the full list of models tested, or the non-Google model the autorater is built on.
- The 96% expert agreement figure and the 30 to 130 turn range are from Google's post and technical report; we did not check the report's methodology.
- Because the 200-task set is private, outside researchers cannot reproduce the full scores.
This brief is Reported: written from the sources below, not from running the release ourselves.
Sources
- Google blog: Introducing the Multimodal Knowledge Agent benchmarkprimary source · fetched 2026-11-10
Related
Spotted an error? See our corrections policy or email the address on the contact page. Markdown version for agents: /news/google-mka-multimodal-knowledge-agent-benchmark.md
Want these briefs in your Google Top Stories? Add Agentic Weekly as a preferred source on Google.
Newsletter
The Weekly Verdict. Free, every Monday.
A free email every Monday: the week's few big AI releases, then 5 short use cases for the new models, each with a prompt you can paste.
Free. Double opt-in. One click to unsubscribe.