Back Rohan Paul

Rohan Paul· @rohanpaul_ai · X· 2026-11-02 19:53 · Original publication time AI score54

Meta and MIT study finds benefits from reusing transformer layers in loopsMachine translation

AI introduction

A Meta and MIT paper studies running the same transformer layers multiple times to add effective depth without increasing the base model's parameters. Tests on models of up to 1 billion parameters trained on up to 155 billion tokens found that looping the full stack gave the best quality at a fixed size, while looping middle layers paid off most when training compute was tight. Loops normally add compute and run sequentially for each token.

Original post · Original

The article text is unavailable in this language; an existing version is shown.

Rohan Paul@rohanpaul_ai

New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.

Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.

They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.

Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.

19:53 · 2026-11-02 · UTC+8

Found an error? Send a correction