正文 · AI 翻译
On November 5, Baidu officially open-sourced its latest foundation model, ERNIE 5.5 Flash Base.
On November 5, Baidu officially open-sourced its latest foundation model, ERNIE 5.5 Flash Base. The model uses a MoE architecture with 280 billion parameters in total and activates 14 billion parameters per token. It was pretrained on 33.5 trillion tokens, with a pretraining context length of up to 128K; during fine-tuning and inference, it supports contexts of up to 1 million tokens. It achieved SOTA performance among models of the same size on benchmarks covering knowledge, reasoning, mathematics, coding, multilingual capabilities, and long contexts, surpassing models including DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base.
Open-source link: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base
As large-model applications expand from text-based question answering to long-document analysis, multi-turn interaction, and agent tasks, the amount of information models need to process continues to grow, along with computational and memory costs. Improving resource efficiency while maintaining model capabilities has become an important focus of large-model research and development.
With efficiency as its central design principle, ERNIE 5.5 Flash Base systematically optimizes key components including the attention mechanism, expert routing, and training mechanisms, and introduces several algorithmic innovations.
For its attention mechanism, ERNIE introduces Latent-Gated Sparse Attention, combining hybrid sparse attention, low-rank gated attention, and virtual head attention. The model combines HCA and MLA so that different attention layers respectively handle the compression of historical information and fine-grained access to historical positions. Low-Rank Gated Attention uses existing low-dimensional representations to compute information gates, reducing the parameters for gate projection weights by 75%. Virtual Head Attention (VHA) can improve the model's expressive power and training quality; during inference, it is fused into the attention operator at minimal cost. These designs reduce the resources needed to process long contexts while preserving the model's ability to handle key information.
For expert routing, the model uses Multi-Router Latent MoE, in which multiple independent routers jointly select experts, improving the correlation between routing scores and experts' actual utility. Compared with traditional approaches that rely on a single router, this design aims to reduce the effect of bias in any single score on expert selection while keeping the number of activated experts, expert computation, and communication volume unchanged. For its training objective, the model introduces ECHO. Rather than adding an MTP module (Multi-Token Prediction), it reuses the backbone's final layer so the backbone itself learns to predict subsequent tokens, allowing it to plan the sequence ahead and effectively improve reasoning and coding capabilities. ERNIE 5.5 Flash uses only single-step ECHO pretraining. The reused final layer can also serve as a draft model for speculative decoding and demonstrates multi-step extrapolation: despite being trained for only one step, its multi-step draft quality is substantially better than that of an auxiliary MTP module.
Beyond model architecture, innovation in large-model infrastructure is also important. ERNIE has optimized the distributed Muon optimizer and introduced FlashMask-4, an attention operator designed for large head dimensions, to improve communication, memory utilization, and computational efficiency in large-scale training. Improvements in model capabilities depend not only on the algorithms themselves but also on whether the training system can efficiently support these designs.
Foundation models not only carry the general capabilities developed during pretraining but also provide an important foundation for downstream model development and application innovation. Unlike application models intended for end users, Base models give developers and businesses room for further fine-tuning and customization, enabling them to build specialized models and applications for different scenarios based on their business needs. Opening foundation models gives developers more technical options and reduces the training costs and technical barriers involved in developing specialized models.
Competition among large models is now gradually entering a new stage in which model capabilities, training efficiency, and innovation in use cases advance together. As technologies and applications such as reasoning and agents evolve rapidly, industry demand for foundation models is shifting from improvements in capability alone toward more efficient training paradigms, more flexible expansion of capabilities, and broader adaptation to different scenarios. While improving core model capabilities and training and inference efficiency, ERNIE 5.5 Flash Base provides developers and the community with a more extensible technical foundation, advancing diverse exploration of model capabilities and innovation in applications.
This article was provided by Baidu and republished by QbitAI with authorization. The views belong to the original author.