Recently, whether AI could slip beyond human control has once again become a major topic in the technology industry. Anthropic researchers have publicly voiced concerns about the potential extinction risks of advanced AI, while OpenAI CEO Sam Altman has said that even a relatively low probability of AI causing human extinction is not a reason to overlook safety. At the same time, AI Agents are evolving from simple information-generation tools into software systems that can call tools, access external systems, and execute tasks autonomously. As a result, AI safety is expanding from “what the model will generate” to “what the model can do.”
This shift means AI safety cannot depend on a single pre-deployment test. As model capabilities, operating environments, and attack methods continue to change, continuous monitoring, red-team testing, vulnerability remediation, and permission controls are becoming core elements of an AI safety framework.
Key Takeaways
-
AI losing control does not necessarily mean that AI has developed the intention to “attack humans.” The more important question is whether objectives, model behavior, and human-defined constraints have become misaligned.
-
AI Agents expand the boundaries of AI risk. Once a model can call tools, access data, and perform operations, model errors can translate into operational risks in the real world.
-
One-time safety testing cannot cover a dynamic environment. Models, user inputs, external tools, and attack methods all change, so AI safety requires continuous testing and post-deployment monitoring.
-
AI safety assessments must examine more than model outputs. They must also address prompt injection, unauthorized operations, tool calls, permission management, and abnormal behavior.
-
A key part of reducing AI risk is limiting the impact of errors through mechanisms such as least privilege, human confirmation, operation logging, risk isolation, and abnormal-event recovery.
Why AI Risks Have Once Again Become a Focus of Discussion
In September 2026, discussion of AI’s extreme risks intensified once again. Anthropic researcher Jacob Coxon publicly stated that some AI developers seriously believe highly advanced AI could have extreme consequences for humanity within this decade. Anthropic researcher Evan Hubinger later expressed similar concerns publicly and offered his personal risk assessment. These remarks sparked a new round of debate about AI safety among U.S. policymakers and the technology industry.
It is important to note that these views are risk assessments made by researchers or industry professionals. They do not mean that AI already has the ability to eliminate humanity autonomously, nor should they be simplified into the claim that “AI will definitely lose control.” In an interview, Sam Altman also said that he did not know how to calculate the so-called “10%” extinction risk scientifically, but believed that even the possibility of an extreme risk that cannot be ignored gives AI companies and governments a responsibility to take steps to reduce it.
Rather than debating an extreme scenario that is difficult to verify at present, it is more useful to consider a practical question: As AI capabilities continue to improve and AI gains greater autonomy, how can humans continue to determine whether it remains safe? That is why AI safety assessment is becoming increasingly important.
What Is an AI Safety Assessment?
An AI safety assessment can be understood simply as a series of tests and monitoring procedures used to determine whether an AI system operates as intended and whether it could create unacceptable risks when exposed to abnormal inputs, malicious attacks, or complex environments. Traditional software testing typically involves extensive pre-deployment checks, including testing for vulnerabilities, verifying that functions work properly, and examining the system’s responses to specific inputs.
AI systems are more complex. Large language models do not generate outputs according to completely fixed rules. The same question may produce different answers, and the context surrounding user input may change continuously. AI Agents may also call external tools such as search, code execution, databases, wallets, email, browsers, or other services. As a result, AI safety assessments examine not only whether a model’s answers are correct, but also whether it can resist malicious prompts, whether it could leak sensitive information, whether it might call tools incorrectly, whether it could perform actions beyond the user’s authorization, and whether it continues to follow established safety rules when the environment changes.
The TEVV-Athlon framework proposed by NIST in 2026 defines Test, Evaluation, Verification and Validation as important methods for assessing the real-world impact of AI systems. It explicitly covers multiple types of systems, including large language models, multimodal models, and Agentic AI.
Why AI Agents Have Created New Risk Boundaries
Understanding AI Agents is essential to understanding how AI safety is changing. Traditional chatbots mainly receive input and generate output. After a user asks a question, the model returns text, code, or other content, while the user generally decides whether to take the next action.
AI Agents operate differently. An AI Agent can break a task into steps based on a user-defined goal, call external tools, read data, execute code, interact with other software, and continue working based on the results. When NIST launched its AI Agent Standards Initiative in 2026, it specifically noted that next-generation AI Agents could already operate autonomously for hours, write and debug code, manage email and calendars, and perform tasks such as shopping.
Changes in capability boundaries also change risk boundaries. If an ordinary AI model incorrectly tells a user that a file exists, the impact may be limited to misinformation. But if an AI Agent with file-access permissions incorrectly decides that a file should be deleted, the model’s error could become a real-world operation. The central security question for AI Agents is therefore not just whether they might say something incorrect. It is also what they can access, what they can do, how long they can act autonomously, and whether they can be detected and stopped quickly after making a mistake.
That is why NIST stated in its 2026 report on AI Agent security that AI Agents introduce new security threats. Traditional cybersecurity principles remain important, but they must be adapted to the way Agents operate.
Why AI Cannot Be Tested Only Once Before Deployment
This is the key to understanding continuous safety assessment. When a traditional software product has relatively stable functionality, developers can conduct extensive testing to verify that it meets its design requirements. AI systems operate in dynamic environments: users change, inputs change, models may be updated, external tools change, and attackers continually search for new ways to attack.
Passing extensive safety tests before deployment therefore does not guarantee that an AI system will remain safe in every future situation. A study published by NIST in 2026 noted that post-deployment monitoring is critical because input conditions in real-world environments may continue to change. These conditions can produce unexpected outputs caused by model nondeterminism or consequences that were not identified during development.
Another NIST study published that same year explained the issue from a more theoretical perspective: A fixed set of AI safety guardrails cannot ensure long-term protection against every adaptive attack. Attackers may continually look for new ways to bypass existing rules. AI safety therefore requires ongoing vulnerability discovery, regular updates to protective mechanisms, and recovery capabilities for when something goes wrong.
AI safety is consequently moving from “test once → deploy” toward “test → deploy → monitor → identify issues → remediate → test again.” Safety is no longer a discrete stage in the AI product lifecycle. It is an ongoing process.
What Does Continuous Safety Assessment Primarily Evaluate?
For AI Agents, continuous safety assessment generally involves several areas.
Model Behavior
The first layer is the model itself. Systems must be continuously checked for obvious errors, hallucinations, harmful content, and behavior that violates system rules. In high-risk areas such as finance, healthcare, and code execution, application-specific testing is also necessary. General benchmarks alone are not enough.
Resistance to Attacks
The second layer is adversarial testing. Attackers may use prompt injection, unauthorized instructions, context manipulation, and other techniques to induce a model to bypass its original rules. Jailbreaks are a typical example. NIST research indicates that fixed safety guardrails cannot guarantee that AI will never be bypassed by adaptive attacks, so red-team testing must continue to search for new attack paths.
The third layer is the permission risk unique to Agents. Even if an AI does not behave maliciously, excessive permissions can allow an incorrect judgment to cause serious consequences. AI Agent permissions should therefore generally follow the principle of least privilege. An Agent that only organizes email, for example, has no need for fund-transfer permissions. An Agent that analyzes data should not automatically have permission to delete a database.
This is broadly similar to smart contract security in the blockchain industry. The system’s logic is only the first layer. The potential loss also depends on which assets the system can reach and which operations it can perform.
The fourth layer is actual post-deployment performance. AI in a laboratory may encounter very different inputs from AI in the real world. Users may provide complex instructions, third-party systems may return abnormal data, and network conditions may change. AI safety assessment must therefore observe how the system behaves in real-world conditions rather than relying only on pre-deployment test reports.
What Does “AI Losing Control” Actually Mean?
The term “AI losing control” is easy to misunderstand. It does not necessarily mean that AI suddenly develops human-like consciousness and decides to attack humanity. Technically, the more important issue is whether objectives, permissions, and actual behavior become misaligned.
An Agent may be asked to complete a task but misunderstand the objective. It may pursue the objective through a path that developers did not anticipate. Or an attacker may alter its understanding of the task through prompt injection. If the Agent also has extensive permissions, an error that began at the software level could affect external systems.
The core issue of “AI losing control” can therefore be divided into three questions: Is the objective correct? Does the behavior align with the objective? Are permissions limited to a reasonable scope? This is why AI safety research is placing increasing emphasis on Alignment—the effort to keep AI system behavior consistent with human-defined objectives and constraints.
How Is AI Safety Relevant to Ordinary Users?
AI safety is not only a concern for AI companies and researchers. As AI Agents enter search, office productivity, programming, financial services, digital assets, and other fields, ordinary users may directly delegate more operational permissions to AI.
For users, the most important question is not whether “AI will destroy humanity,” but something more practical: What permissions have I actually given AI? If AI can only answer questions, the impact of an incorrect answer is usually limited. If AI can send email, modify files, access accounts, run code, or execute financial transactions on a user’s behalf, the security requirements are entirely different.
When using AI tools with autonomous execution capabilities, users should focus on three areas: whether permissions are minimized, whether important actions require human confirmation, and whether the system provides operation logs and a way to reverse abnormal actions.
This is particularly important for AI Agents involving digital assets. Once operations such as wallet signatures, asset transfers, and smart contract interactions are delegated to an automated system, the security boundary extends beyond the model itself to private-key management, approval mechanisms, smart contracts, and external services.
The Focus of AI Safety Is Shifting from “Preventing All Errors” to “Controlling the Impact of Errors”
At present, AI safety is unlikely to be solved by a universal guardrail that permanently addresses every problem. AI models will be updated, attack methods will evolve, and application scenarios will expand. NIST has therefore emphasized the need for continuous red-team testing and updated protective measures, as well as the ability to limit the impact of vulnerabilities and recover quickly when they occur.
The underlying principle is simple: We cannot assume that a system will never make a mistake, so we must decide in advance what will happen when it does. For AI Agents, this may include restricting permissions, requiring human confirmation, logging critical actions, isolating high-risk tasks, and quickly stopping or reversing actions when abnormal behavior occurs.
From this perspective, AI safety is not about answering the simple binary question of whether “AI will kill humanity.” It is about solving a more practical engineering problem: As AI becomes increasingly capable of acting autonomously, how can humans continue to know what it is doing and why, and how can they regain control when something goes wrong? That is the purpose of continuous safety assessment.
Summary
Recent discussion of whether AI could cause human extinction has once again placed AI safety in the public spotlight. More important than the extreme predictions themselves, however, are the structural changes taking place in AI capabilities.
As AI evolves from chatbots into AI Agents capable of calling tools, accessing data, and executing tasks autonomously, safety concerns are expanding from model-output safety to permissions, tool calls, environmental interactions, and long-term autonomous operation.
AI safety therefore cannot rely on a single pre-deployment test. Continuous monitoring, red-team testing, permission controls, vulnerability remediation, and abnormal-event recovery will become increasingly important infrastructure as AI Agents enter real-world applications.
The safe development of AI will depend less on finding a model that never makes mistakes than on establishing mechanisms that can continuously identify problems, limit risk, and restore human control.
FAQ
Will AI Really Kill Humanity?
There is currently no reliable evidence that AI will inevitably cause human extinction. Some AI researchers have recently expressed concerns about extreme risks, but the probability of those risks remains highly uncertain. More realistic safety concerns include cyberattacks, data breaches, incorrect decisions, and the abuse of AI Agent permissions.
What Is an AI Agent?
An AI Agent is an AI system that can autonomously plan and execute tasks based on an objective. Unlike traditional chatbots, AI Agents can typically call external tools, access data, and perform real-world operations. They therefore have greater autonomy and require stricter permission and security controls.
Why Does AI Require Continuous Safety Assessment?
Because the inputs, environments, and attack methods that AI encounters will continue to change. A single test reflects performance only at a particular time and under particular conditions. It cannot prove that the system will not develop new vulnerabilities in the future. Continuous post-deployment monitoring and testing are therefore equally important.
What Is the Biggest Security Risk of AI Agents?
One key risk is permissions. If an AI Agent can access accounts, files, code, funds, or other external systems, model errors or malicious inputs could turn an information-level mistake into a real-world operation.
What Does an AI Safety Assessment Typically Include?
It typically includes model behavior testing, adversarial testing, red-team testing, tool-call testing, permission controls, post-deployment monitoring, and abnormal-event recovery. Security testing standards tailored to each application must also be developed for different use cases.
