正文 · AI 翻译
Shishi Technology Releases New Multimodal Model Service
Yunzhong reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
Yunzhong reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
In AI video generation, faster inference and lower generation quality can occur at the same time.
Reducing computational steps can shorten generation time, but may also cause errors to accumulate; retaining the full computation process can maintain generation quality, but requires more computing power and video memory.
As video generation moves from one-off experiences to batch production, the question is no longer just whether a model can generate video, but how to balance generation quality, inference latency and resource costs.
To address this need, Shishi Technology has officially released its new Multimodal Model Service, focusing its optimization efforts on accelerating video generation inference, cache and video memory management, multimodal Token compression, and connecting material understanding with the generation pipeline.
Specifically:
Adaptive acceleration based on historical cache residuals reduces redundant computation during video generation;
A cache management mechanism combined with SGLang optimizes resource use;
Modality-adaptive Token compression reduces the cost of multimodal understanding;
And a Context Engine based on an omni-modal model converts the results of material understanding into structured information that a video generation model can execute.
These technologies work at different stages, but ultimately serve the same goal:
To improve the inference efficiency and computing resource utilization of multimodal models while safeguarding generation quality and service stability.
Video generation models typically require multiple rounds of iterative computation, gradually turning initial information into video content. During this process, adjacent computational steps contain information that can be reused.
Performing the full computation at every step incurs high inference costs; but directly skipping some steps or reusing past results may cause cache errors to accumulate over successive iterations, ultimately affecting generation quality.
The key to accelerating video inference, therefore, is not simply to reduce the number of computations, but to determine which computations can safely be reused and when to recompute to control errors.
Shishi Technology uses adaptive acceleration based on historical cache residuals, assessing the risks of cache reuse using information from past computations and dynamically adjusting its computation and refresh strategies.
For parts of the computation that change little, the system reuses past results to reduce repeated computation; for critical parts with a higher risk of error, it triggers recomputation and calibration to reduce the effect of cache errors on subsequent generation. Through dynamic refresh and error calibration, the system reduces computational costs while maintaining the consistency of the video generation trajectory as much as possible.
Under the test conditions shown in the released solution, the technology achieved an average end-to-end speedup of 2.48 times compared with the Dense baseline while maintaining generation quality.
This means inference acceleration no longer depends entirely on reducing generation steps; instead, a more refined computation reuse strategy establishes a dynamic balance between efficiency and quality.
The resource cost of video generation comes not only from the amount of computation, but also from video memory use during inference.
When multiple video tasks run concurrently, storing, reading and updating cache data all affect the efficiency of video memory use. Even if an individual task is accelerated, the additional resource burden of cache management may still limit overall task processing capacity.
Therefore, inference acceleration for individual tasks and resource utilization across multiple tasks need to be optimized together.
Shishi Technology combines intelligent caching decisions with the SGLang inference framework, using mechanisms including cache reuse, block processing, compression and intelligent scheduling to reduce repeated computation and unnecessary video memory costs.
The system also introduces a risk-aware mechanism that dynamically adjusts caching strategies based on actual computation feedback, improving reuse efficiency while controlling error risks. This design extends optimization beyond individual inference computations to resource management.
For enterprise video production, generation speed, video memory use and task concurrency together determine the actual output of computing resources. Reducing redundant computation and caching costs can create room for greater concurrency and more flexible task deployment.
Beyond video generation, multimodal understanding faces its own computational efficiency challenges.
Compared with text, video and audio contain a higher density of temporal information. When processing inputs such as continuous video frames and audio segments, models must convert them into large numbers of Tokens, increasing computational and storage demands. But information density varies across modalities and segments. Some Tokens contain key semantic information, while others are highly redundant.
Processing all Tokens in the same way can produce unnecessary computational costs; uniformly compressing them, on the other hand, may discard important information that affects understanding.
Shishi Technology has proposed a training-free, modality-adaptive Token compression method that uses AudioCLIP-based modality awareness to identify useful information in different modalities and adjust Token retention and compression strategies accordingly.
Its core idea is to allocate computational resources according to modality characteristics and information value, rather than processing all inputs in equal proportions.
The method requires no additional training. By reducing redundant Tokens, it lowers the computational burden of multimodal understanding while preserving key semantic information as much as possible.
In the WorldSense evaluation, the method achieved a 1.33 times speedup. It also showed good performance on multimodal understanding benchmarks including VideoMME and DailyOmni. This extends multimodal inference optimization beyond video generation to material understanding, offering a more efficient technical approach to processing complex audiovisual inputs.
Beyond inference efficiency, multimodal video creation also needs to address how the results of understanding enter the generation process.
In real business settings, video generation requests often include character images, reference videos, audio and written requirements at the same time.
These inputs not only take different data forms but also contain constraints involving character identity, actions, voices, shots and chronological order.
If complex materials and natural-language instructions are given directly to the generation model, the relationships among the materials and the execution requirements may be difficult to express consistently.
To address this, Shishi Technology has built a Context Engine solution based on an omni-modal model, creating a structured link between multimodal understanding and video generation.
The system first understands the input materials and creative requirements and extracts characters, voices and relevant constraints. It then organizes these into structured information containing shots, actions, dialogue and other elements for the subsequent video generation model to execute. For example, when a user provides a character image and reference audio and asks for a video of the character speaking, the system needs to link the character information, voice content and shot arrangement, rather than merely generate a text prompt.
Through mechanisms such as material binding, structured storyboards, timeline validation and error correction, this solution enables information from different modalities to form relatively clear correspondences in the generation process.
Its optimization focus is to turn complex multimodal inputs into clearer generation context that can be validated and executed.
At the service level, Shishi Technology has also built a multimodal inference architecture covering API access, task orchestration, model routing, inference execution and result delivery. Task status management, concurrency control, timeout retries and recovery mechanisms support the running and delivery of tasks in real business settings.
Once multimodal models enter production environments, generation quality, inference latency, video memory costs and task stability are no longer independent metrics.
The video generation side needs to control redundant computation and error accumulation; the multimodal understanding side needs to reduce the processing of unnecessary Tokens; and the generation pipeline needs to ensure that material information and creative constraints are passed along effectively.
Shishi Technology's newly released Multimodal Model Service considers these optimization capabilities within the same production pipeline.
From adaptive cache acceleration and video memory resource management, to modality-aware Token compression and the structured handoff provided by the omni-modal-model-based Context Engine solution, and then to the inference service architecture, the scope of optimization gradually extends from the computation process of an individual model to the execution and delivery of multimodal tasks.
The key to scaling AI video applications is not just shortening the wait for a single generation, but consistently completing more generation tasks that meet quality requirements with a given amount of computing resources.
Moving from “able to generate” to “ready for production,” Shishi Technology will continue to optimize multimodal inference efficiency, resource utilization and service stability, providing foundational support for bringing AI video from model capability into actual production.
*This article is republished by QbitAI with authorization. The views expressed are solely those of the original author.