Study finds gating can delay in-context learning in state space modelsMachine translation

A NeurIPS paper uses theory and experiments to show that gating in state space models can steer them toward memorizing in their weights first, delaying or even preventing correct in-context learning, even when architecture or memory capacity imposes no fundamental limit. The researchers also find that gating often improves generalization to longer sequences.