Article · AI translation
Thanks to IT Home reader 华南吴彦祖 for the tip!
IT Home reported on November 11 that, on November 10 local time, Google announced that it had partnered with AI model evaluation organization Artificial Analysis to launch MKA (Multimodal Knowledge Agent), a benchmark for assessing AI agents’ ability to process visually complex files and complete real-world knowledge work tasks.
Google said its newly released Gemini 4 Carbon ranked second among the models tested, with an average cost per completed task of only about one-sixth that of the top-ranked model.
Google DeepMind senior product manager Rohan Doshi and principal engineer Ben Caine explained that many existing knowledge work benchmarks can be completed using text alone, while real-world work often involves substantial visual information. For example, architects need to cross-check building blueprints, engineers need to analyze engineering diagrams, and professionals need to organize information from different sources into presentations, reports and spreadsheets.
The MKA benchmark is designed to assess how AI agents perform in these complex work settings. Google worked with domain experts from 10 industries and 83 occupations to create an evaluation set of 200 tasks covering business, architecture, education, engineering, finance, healthcare, law and government, logistics and maps, science, and software and data.
Each task simulates a multistep professional project and includes a realistic work prompt and a set of multimodal input files. Google said these tasks typically take experienced professionals 3 to 8 hours to complete. IT Home noted that each task also has an expert-written scoring rubric containing about 70 yes-or-no checks on average, with factual accuracy accounting for 75% and visual design for 25%.
Under the evaluation setup, all tested models run on the same open-source agent framework, Stirrup, with the same tools, sandbox environment and interaction-turn budget. Final deliverables are scored by an agent-based judging system built on a non-Google model. The system opens all files, renders pages, checks spreadsheet formulas and verifies results against the expert scoring rubrics. Google said that when domain experts manually scored model outputs, their scores agreed with the automated judging system’s results 96% of the time.
To prevent test questions from leaking and entering model training data, Artificial Analysis is keeping the 200 official evaluation tasks private. The project has also published another 10 tasks on Hugging Face so developers can learn the test format and evaluate their own agents.
According to test results released by Google, Gemini 4 Carbon ranked second among the models tested for quality, while its average cost per task was only one-sixth that of the top-ranked model. Google also noted clear differences in efficiency across models completing the same set of tasks: the average number of interaction turns used per task ranged from 30 to 130.
The MKA benchmark is now available through Artificial Analysis, where developers can evaluate their own models. Gemini 4 Carbon is available through the Gemini API in Google AI Studio.
Related reading:
“Google’s most powerful model yet, Gemini 4 Carbon, arrives, focusing on complex programming, scientific research and cybersecurity”
“Google’s most powerful AI model: Gemini 4 Carbon tops the global intelligence index, ‘specializing’ in cybersecurity”