正文 · AI 翻译
Qualcomm showcases an on-device MoE agent at the 9th China International Import Expo
What stage has on-device AI reached?
After 3 to 4 years of development, on-device AI is approaching a significant turning point.
In terms of applications, on-device AI did not begin with large models. Imaging was one of the earlier areas in which AI was deployed on devices. The steady improvement in smartphone photography in recent years has been driven in part by on-device AI technology. Today, on-device AI has expanded further into multimodal applications such as simple question answering, chat, image generation and processing, text processing, speech transcription, and real-time translation. The conditions for deploying many of these features on devices are already relatively mature.
However, many current on-device AI applications still focus on individual functions. Automatic speech recognition (ASR), text summarization, image processing, and simple chatbots, for example, mostly address a specific task. If devices are to go further in the future and handle system-level and planning tasks such as understanding intent, orchestrating tasks, and operating target apps in real time, they will need more capable on-device models.
This means on-device AI is gradually evolving from solving individual tasks toward more complex agent experiences.
Why is on-device AI needed? How do on-device and cloud AI relate to each other?
Qualcomm does not emphasize using on-device AI exclusively. Instead, it emphasizes collaboration between devices and the cloud, or a “computing continuum,” with “on-device first” as one important approach.
Traditional cloud-dependent AI services consume cloud tokens, GPU capacity, and cloud scheduling resources with every inference, while also depending on a network connection. Deploying suitable AI capabilities on devices can reduce reliance on networks and provide more timely and stable responses, while drawing on devices’ advantages in security, privacy, and personalization.
To use a simple analogy, the cloud is more like a “general knowledge base,” while a device holds more “specialized knowledge” about its individual user.
A device can take into account who the user is, where they are, what they have done before, and their current surroundings to assess what services they might need next. This is also the basis of “proactive AI services.”
On-device AI is therefore not an isolated system. The cloud can handle tasks requiring greater intelligence and computing power, while devices can offer advantages in real-time response, stability, privacy, and personalization. The 2 work together.
Why has MoE become an important technical direction for on-device AI?
MoE stands for Mixture of Experts. One way to understand MoE is to compare it with a traditional dense model.
Take a dense model with 3 billion parameters: to complete each inference, all 3 billion parameters must run to produce the final output.
MoE works differently. It divides a model’s parameters among multiple “expert” modules. Each module handles a portion of the model and may specialize in different tasks. When the model processes a particular task, it needs to call only the expert modules relevant to that task.
For example, if an MoE model contains 20 expert modules, a given task might require calling only 2 to 3 of them, rather than running all 20 at once.
A straightforward analogy involves chefs: a dense model is like having only 1 chef at home who cooks whatever you want to eat. That chef may know many cuisines but may not be especially skilled at every one. MoE is like having several chefs who specialize in different cuisines: 1 excels at Shandong cuisine, another at Beijing cuisine, and another at Huaiyang cuisine. When you want Huaiyang food, only the chef skilled in Huaiyang cuisine needs to work; the others can rest.
In other words, although the model has many “expert” modules, only some of them need to participate in the computation for a given task. This mechanism can reduce the number of parameters actually run during an inference and improve the model’s operating efficiency.
Why is MoE particularly important for on-device AI?
MoE is already used in large cloud models. As on-device AI develops, more on-device models are also beginning to adopt MoE approaches.
This technology is especially important for devices because phones, PCs, and other devices face constraints such as memory capacity and power consumption.
Under current mainstream approaches, devices can typically run only models with 2 billion to 3 billion parameters. Models of this size can already perform individual tasks such as ASR, text summarization, image processing, and simple chat well, but their intelligence remains somewhat limited when it comes to complex system-level and planning tasks.
Models with 2 billion, 3 billion, or 4 billion parameters are not wholly incapable of handling these complex tasks at this stage, but there is still room to improve completion and accuracy, which may reach levels of 60%, 70%, or 80%. MoE offers an opportunity to further improve the intelligence of on-device models, as well as their completion and accuracy on complex tasks.
This is an important reason MoE matters for on-device AI: it offers a new technical path for deploying and running more intelligent models within the memory and power constraints of devices.
As the relevant technologies develop further, capabilities demonstrated on site, such as intent understanding, task orchestration, and real-time operation of target apps, may eventually be applied to different devices, including phones, cars, and PCs.
What did Qualcomm’s on-device MoE agent demonstration at the China International Import Expo show?
MoE models contain multiple expert modules, so they are relatively large overall. To run them on devices, 2 very practical problems must be addressed in addition to the intelligence of the model itself: how to predict in advance which expert modules the current task will need next, and, once that prediction is made, how to quickly retrieve those modules from storage so they can participate in computation in time.
DDR bandwidth is often a major bottleneck during large-model inference. If the expert modules needed for the next step can be identified in advance and prefetched to an appropriate location, waiting time can be reduced and the model can run more efficiently.
The on-device MoE agent demonstration exhibited by Qualcomm at the 9th China International Import Expo explores precisely these issues. It was developed jointly by Qualcomm, Infris AI, Longsys, and StepFun. To run on-device MoE models efficiently, the parties coordinated optimizations across the model, inference, storage, and chip platform: StepFun focused on improving the model’s intelligence; Infris AI used its inference approach to predict and prefetch the required expert modules; Longsys optimized the efficiency of loading expert modules from Universal Flash Storage (UFS) into memory so they could be activated more quickly; and Qualcomm worked with its partners on the chip platform and software stack to optimize expert module routing, expert module prediction, and cache prediction, improving the overall efficiency of running MoE models on devices. The demonstration therefore shows more than a model: it shows systematic optimization spanning the model, inference, storage, and chip platform, and illustrates how on-device AI is moving from innovation in individual technologies toward industry collaboration.
Why does on-device AI increasingly need ecosystem collaboration?
As AI and large models develop, the focus of Qualcomm’s collaboration with partners across the industry is also changing. The goal is not simply to “move” models onto devices, but to make them run better there and then integrate these capabilities into different devices, including phones, PCs, glasses, and cars.
Different partners play different roles in the on-device AI ecosystem: model makers provide the “intelligence.” Qualcomm works with them to make models smaller and inference faster through model quantization and optimization using chip technology and the software stack, moving collaboration beyond “model + chip” toward “model for chip” or “chip for model.” Device makers control the user “entry point” and have devices, operating systems, and agent capabilities. Devices also hold large amounts of personal and personalized data, so chip capabilities and model intelligence need to be integrated into AI experiences users can perceive. Developers bring AI capabilities to more apps and scenarios, incorporate the capabilities and permissions of chips, models, and operating systems into specific applications, and address the “long-tail effect” in AI applications through a rich app ecosystem. Qualcomm provides the foundation for on-device AI capabilities, continually strengthening AI processing capabilities such as the NPU and working with model makers, device makers, and developers through its Snapdragon platforms, full-stack AI software, and development tools to deploy AI capabilities across more devices and applications.
The MoE agent demo at this China International Import Expo is also a concrete example of this ecosystem collaboration: from the model and inference approach to storage and the chip platform, different partners address problems at different stages and work together to run more complex AI capabilities on devices.
This article was provided by Qualcomm and republished by QbitAI with permission. The views belong to the original author.