原帖引用 · 原文
Rohan Paul@rohanpaul_ai
Sam Altman on local inference.
"from a global allocation perspective, it's quite terrible to have everybody, a bunch of the world's limited supply of DRAM sitting in their own computer and being used 5% of the time"
per token, the datacenter wins easily. A local model at batch size 1 is wasteful by any utilization metric, and it’s also the only part of the stack nobody can deprecate on you. https://t.co/w9A93jliU9
· UTC+8