Original post · Original
New Google paper finds that the best AI agent scores only 50.6% on physical engineering jobs, often by missing a hard safety or feasibility rule.
Engineering benchmarks for LLMs mostly test textbook questions or toy problems that top agents already handle well. Real engineers work from piles of PDFs, spreadsheets, and CAD files and must issue designs that meet code and stay cheap.
They built PLE, 85 electrical, civil, structural, and mechanical jobs written or checked by practising engineers. Agents must read about 61 project files per task and deliver schedules, reports, or 2D and 3D drawings.
Claude Opus 5.5 in Claude Code topped 14 models at 50.6%, using 27M tokens and 63 minutes per task. On design tasks its outputs met 91% of constraints, but only 41.3% met every hard constraint.
If you build engineering agents, gate every design on hard safety and code limits, and spend effort on model choice before tuning the agent scaffold.