aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score54

Carnegie Mellon paper reports Opus 5.5 exceeds human baseline on AI systems benchmarkMachine translation

AI introduction

Singularity-Bench tasks agents with six end-to-end jobs, including a pre-training pipeline and a serving engine, and compares results with production systems on the same GPUs. The material reports a Singularity-Index of 1.25 for Opus 5.5, versus a human baseline of 1; its serving engine earned USD 11.52 per GPU-hour, versus USD 8.18 for SGLang.

Original post · Original

@rohanpaul_ai

New Carnegie Mellon paper finds that Opus 5.5 + Claude Code built a pre-training pipeline, an inference engine, and a cluster orchestrator that matched or beat production tools like SGLang and NVIDIA Dynamo on the same GPUs.

Singularity-Bench gives agents 6 end-to-end jobs, from pre-training a model to building a serving engine. Each result is scored against a production system, such as SGLang or Claude Code, on the same GPUs.

Opus 5.5 scored a Singularity-Index of 1.25, where 1 equals the human baselines, while the other 10 agents averaged 0.58. Its serving engine earned 11.52 USD per GPU-hour against 8.18 for SGLang.

Source: Rohan Paul · x.com

Research
Found an error? Send a correction