原帖引用 · 原文
Rohan Paul@rohanpaul_ai
Overview of singularity-bench: 6 end-to-end tasks span the agent-development stack.
Agents design and implement systems within fixed budgets, and the systems are evaluated against human-built references on the same hardware. https://t.co/je74YzJCcM
New Carnegie Mellon paper finds that Opus 5.5 + Claude Code built a pre-training pipeline, an inference engine, and a cluster orchestrator that matched or beat production tools like SGLang and NVIDIA Dynamo on the same GPUs.
Singularity-Bench gives agents 6 end-to-end jobs, from pre-training a model to building a serving engine. Each result is scored against a production system, such as SGLang or Claude Code, on the same GPUs.
Opus 5.5 scored a Singularity-Index of 1.25, where 1 equals the human baselines, while the other 10 agents averaged 0.58. Its serving engine earned 11.52 USD per GPU-hour against 8.18 for SGLang.
在 X 查看这条帖子 ↗
· UTC+8