Meta and MIT study finds benefits from reusing transformer layers in loopsMachine translation

A Meta and MIT paper studies running the same transformer layers multiple times to add effective depth without increasing the base model's parameters. Tests on models of up to 1 billion parameters trained on up to 155 billion tokens found that looping the full stack gave the best quality at a fixed size, while looping middle layers paid off most when training compute was tight. Loops normally add compute and run sequentially for each token.