正文 · AI 翻译
AI safety is not a “brake pad” on AI development, but an “accelerator” for high-quality AI development.
Artificial intelligence is entering production systems and core business operations at an accelerating pace. When AI actually participates in business decisions, calls tools, operates systems and even controls equipment, decision-makers care not only about “how capable it is” but also about “whether it is safe.” In the age of AI agents, AI safety is no longer just about preventing data leaks or incorrect answers; it also concerns real-world assets, business continuity and accountability.
As a result, a frequently discussed question is changing: Is AI safety a constraint on technological development, or a prerequisite for industrial applications? At present, AI safety has shifted from an added condition to a fundamental capability for AI to enter production systems, core business operations and critical sectors. More specifically, AI safety determines whether AI can be used, to what extent it can be used and how quickly its applications can be scaled.
New trends in AI development pose new challenges for AI safety
AI capabilities currently show six interconnected changes.
First, AI is moving from generating content and “answering questions” to taking autonomous action. It can call tools, carry out tasks, operate systems and control equipment, raising the risk from “saying something wrong” to “doing something wrong.”
Second, AI is moving from single models to collaboration among multiple agents, which can break down tasks, share memory and call on one another to improve efficiency. As a result, a single-point error can develop through task decomposition, memory sharing or tool calls into a systemic, cascading risk.
Third, AI is gradually moving from an assistive tool to business infrastructure, entering critical fields such as finance, healthcare, industry, energy and government services. Model outputs directly affect core business decisions and continuous operations, giving AI a broader impact, higher potential costs and a longer chain of accountability.
Fourth, AI is moving from closed models toward widespread open-source adoption. Openness lowers the barriers to innovation and enables more industries and professionals to participate in AI innovation, an important condition for the development of artificial intelligence in China. But as capabilities spread, risks spread with them: the cost of malicious use falls, and responsibility downstream becomes more dispersed.
Fifth, AI is moving from the digital world into the physical world, with areas such as embodied intelligence, intelligent vehicles and robots developing rapidly. The effects of AI errors or inadequate capabilities no longer remain on a screen and may directly cause real-world harm.
Sixth, AI competition is shifting from model training toward inference and applications at scale, and the focus of safety is beginning to move from “whether the model poses risks” to “whether the entire system can operate reliably over the long term in real business settings.”
The boundaries of safety have expanded into a risk map spanning the economy and society
As these changes converge, AI safety has expanded from the question of “whether a model is safe” into a complex systems issue spanning six levels: infrastructure, data, models, agents and systems, applications and business, and society and national security. At higher levels, the scope of risk grows; the different levels are also linked through data flows, chains of permissions, tool chains and business processes.
AI’s six-level risk map
This means AI risks have three prominent characteristics: first, they can propagate along a chain—a prompt injection may use an agent’s permissions to turn into a real business operation; second, they can emerge from interactions—a single model may appear safe, yet connecting multiple models, tools and data may create new risks; third, they can spill over—a technical flaw may quickly become a business risk, a social risk or even a national security risk. The traditional practice of “conducting one safety check before launch” is no longer enough. Safety must span the entire life cycle of data, development, deployment, operation and retirement.
Safety and development are not opposite ends of a seesaw, but a mutually reinforcing flywheel
AI is becoming a new driver of social development. The more autonomous and capable it becomes, the more it can enter core production processes and the greater the value it can create; at the same time, exposure to risk also increases. Enterprises do recognize AI’s value, but they worry about the scale of losses after an incident, whether risks can be controlled, how operations can be restored and who will bear responsibility. Only by addressing these concerns can AI truly move from limited pilots to large-scale application.
High-quality, large-scale AI application = effectiveness × reliability × safety. Effectiveness answers “Is it useful?” Reliability answers “Can it be used consistently and stably?” Safety answers “Do we dare put it into core business operations?” The three are not simply additive: if any one approaches zero, the overall value falls significantly.
Therefore, safety and development are not a seesaw in which one gains at the other’s expense, but a mutually reinforcing flywheel. As AI applications keep expanding, they bring greater exposure to risk, prompting enterprises to invest more in safety; the large-scale deployment of safety products and services, in turn, accumulates alerts, attack samples and incident response feedback, improving detection and automation; as false positives decline and risks become more controllable, enterprise trust grows, making enterprises more willing to put AI into core business operations. Investment in safety is not merely a cost, but a productive investment that reduces uncertainty and expands the market for AI applications.
AI safety is creating new room for industry growth
Looking at practices around the world, AI safety is mainly advanced by three types of actors working together: governments set baseline requirements through regulation, enterprises drive the evolution of technology and products through market competition, and third-party organizations provide technical standards, evaluations, audits and certification. The United States places relatively more emphasis on technical standards, commercial platforms and market competition, while the European Union emphasizes government regulation, safety standards and third-party assurance. China emphasizes the coordinated roles of government regulation, enterprise competition, and third-party certification and technical standards, balancing all three.
In this process, AI safety has already acquired the basic characteristics of an independent industry. Although actual revenue remains far smaller than the potential market, products are evolving from individual tools toward platforms, agent safety is becoming a new growth area, and enterprises using AI are increasingly willing to purchase safety services. The industry’s value is shifting from “limiting risks” to “enabling AI to enter core business operations at scale.”
In terms of market structure, segments are emerging in model safety evaluation, AI runtime safety, supply chain safety, data safety, agent and non-human identity safety, governance and compliance, and third-party assurance. Revenue models will also gradually expand beyond project-based consulting and red-team testing to recurring revenue from platform subscriptions, managed security services, audit and certification, and other offerings. The clearer these market segments become and the more unified the standards, the more likely the industry is to achieve substantial scale.
Accelerate the development of AI safety to support large-scale AI application with high-level safety
To enable large-scale AI application, efforts should focus on five areas.
First, incorporate AI safety into the development of national foundational AI capabilities. AI safety should not be merely a compliance step after an application goes live. It should stand alongside algorithms, computing power, data, capital, applications and talent as a fundamental element driving the development of artificial intelligence.
Second, build safety into AI systems. In the past, many safety capabilities were added after systems had been built; in the age of agents, perimeter protection alone is no longer enough. Safety must be developed in parallel with model design, agent architecture and business processes to create models, agents, platforms and applications with safety built in from the start.
Third, focus on key technologies for agent safety, prioritizing identity authentication, least-privilege controls, safe tool use, memory safety, safety in multi-agent collaboration, behavior auditing and automated red-team testing. At the heart of this effort are three questions: Who is the agent, what is it allowed to do, and can errors be detected and stopped in time?
Fourth, cultivate AI safety as a new area of industry growth by promoting coordinated development of safety data, safety tools, model evaluation, runtime protection, agent governance, audit and certification, and safety services, so that safety investment both supports risk governance and creates new industry value.
Fifth, build public infrastructure for AI safety. High-quality safety datasets, vulnerability and incident databases, industry safety benchmarks, red-team attack and defense platforms, and risk monitoring and early-warning systems have a clear public-interest dimension. They require joint participation by government, enterprises, universities and third-party organizations to reduce duplicated investment and establish a common language for risk.
Introduce commercial insurance to accelerate AI safety development through economic mechanisms
Large-scale AI application also requires a market-based economic framework, and commercial insurance warrants particular exploration. For many new technologies and products, moving from initial adoption to large-scale application depends on insurance sharing residual risks. For low-frequency, high-loss risks such as data breaches, system outages, agent errors and third-party property damage, insurance can ease enterprises’ concerns about being unable to use AI with confidence.
More importantly, insurance can turn abstract safety risks into risks that can be measured and priced. If an enterprise has strict permission controls, safety evaluations before launch, continuous operational monitoring, human review and emergency shutdown mechanisms, complete logs and accountability tracing, its premiums should be lower and its coverage could be broader. In this way, safety capabilities become calculable economic benefits rather than requirements stated only in principle.
Insurance can also move beyond one-time financial compensation to provide ongoing safety services, helping to create a closed loop of “safety evaluation—risk classification—insurance pricing—continuous monitoring—claims review.” The underwriting and claims processes can accumulate data on incident rates, loss distribution and the effectiveness of control measures, which can in turn drive improvements in technology, standards and regulation.
Of course, AI insurance still faces problems including unclear chains of responsibility, insufficient historical data, dynamically changing risks, concentrations of systemic risk and difficulty proving what caused a loss. Therefore, it needs to move from static policies to dynamic insurance and from compensation alone to “insurance plus safety services,” while exploring coordinated underwriting across the industry chain. Safety technology determines whether risks can be prevented, legal governance determines who bears responsibility, and commercial insurance determines whether residual risks can be borne. All three are essential.
Conclusion
The greater AI’s capabilities, the more critical the fields it can enter and the greater the permissions it can obtain—and the higher the cost of a safety failure. Trustworthiness is becoming a key bottleneck in the expansion of AI applications at scale.
Therefore, AI safety is not a “brake pad” on AI development, but an “accelerator” for AI’s move from pilot applications to large-scale applications, applications in critical fields and high-quality development. Allowing safety capabilities to evolve alongside AI capabilities and building a safety system that coordinates technology, governance, industry and market mechanisms will help turn AI’s technological potential into industrial and social value more quickly.
About the lead author:
Chair of the Science and Technology Committee of China Telecom Group, external director of COFCO Group, and former general manager of China Telecom Group. A professor-level senior engineer with master’s degrees in engineering and economics from Harbin Institute of Technology and a doctorate in management from Nankai University. Also serves as an expert committee member of the National Data Administration and executive deputy director of the Expert Advisory Committee of the Internet Society of China.
This article was provided by China Telecom and republished by QbitAI with permission. The views are those of the original author.