Overview of singularity-bench: 6 end-to-end tasks span the agent-development stack.
材料引用的研究介绍称，Opus 5.5 与 Claude Code 构建了预训练流程、推理引擎和集群编排系统，其表现达到或超过部分生产工具。
2 reports2 sources
Reports 2
Singularity-Bench Evaluates AI Agents on Six End-to-End Development Tasks
Singularity-Bench sets six end-to-end tasks, including model pre-training, inference engines and cluster orchestration. Agents design and build systems within fixed budgets, with results compared against human-built systems on the same hardware. A quoted post says Opus 5.5 scored 1.25 on the Singularity-Index, while the other 10 agents averaged 0.58; the figures should be read in the context of those evaluation conditions.
Carnegie Mellon paper reports Opus 5.5 exceeds human baseline on AI systems benchmark
Singularity-Bench tasks agents with six end-to-end jobs, including a pre-training pipeline and a serving engine, and compares results with production systems on the same GPUs. The material reports a Singularity-Index of 1.25 for Opus 5.5, versus a human baseline of 1; its serving engine earned USD 11.52 per GPU-hour, versus USD 8.18 for SGLang.
SourceRohan Paul (@rohanpaul_ai) ↗Rohan Paul (@rohanpaul_ai) ↗