What are AI giants most afraid of in private? Axios: Worry that they could be maliciously used to create “deadly pathogens,” triggering a chemical and biological disaster

As Axios reports in a well-known column, when privately asking AI architects and technology leaders what risks they fear most, most point to the possibility that AI could be maliciously used to create deadly pathogens that can spread silently and rapidly. A new survey further warns that, without adequate protective measures, the chance of a catastrophic event caused by AI resulting in the deaths of more than one million people by 2030 will exceed 20%.
(Background: Nvidia vs AMD is in the spotlight—big CPU architecture showdown. Who can define the new standard for AI agents?)
(Additional context: Google Q2 cloud revenue surges 82%! JPMorgan Chase: AI is entering the payback period—buy on dips.)

With the rapid advancement of artificial intelligence technology, tech giants driving breakthroughs in frontier models are nonetheless harboring deep fears about the future. According to Axios’s latest report in the column “Behind the Curtain,” when they privately asked nearly every AI architect and leader about the risk they worry about most, their answers were strikingly一致: a kind of deadly pathogen that can quietly, widely, and rapidly spread—leaving people unable to stop it in time.

Expert consensus: AI may accelerate the development of biological weapons

Experts have rarely reached consensus, believing advanced AI models are highly likely to help malicious actors easily create deadly pathogens before governments and industries establish robust safeguards. As AI increasingly learns human biological vulnerabilities, this has been regarded by the industry as a “reasonable possibility” of a major risk.

A recent survey by MIT FutureTech and the University of Queensland, covering 272 researchers, ranked 24 major AI risks. The results show that without any mitigation measures, the probability that AI risk capabilities lead to catastrophic consequences by 2030 (defined as more than one million deaths or causing $100 billion in damage) is over 20%; even with mitigation in place, the likelihood of catastrophe and of AI enabling large-scale harm weapons still stands at roughly 12% each.

AI giants sign a warning: the safety net becomes the focus of regulation

This potential doomsday threat is not without basis. Last month, technology leaders including OpenAI’s Sam Altman, Anthropic’s Dario Amodei, Google DeepMind’s Demis Hassabis, Microsoft’s Mustafa Suleyman, and Meta’s Alexandr Wang just jointly signed an open letter, firmly warning about the risks of biological weapons derived from AI and strongly calling for enhanced safeguards.

Experts worry that well-resourced malicious actors can use powerful models to map human biological vulnerabilities, finding genetic modification schemes that make known pathogens easier to spread or immune to existing treatments. With large-scale training on biological and genomic datasets, the speed at which AI generates viable candidate solutions is like hundreds of the smartest scientists working at super high speed—never sleeping, never stopping.

Afterward, these schemes can be synthesized using increasingly affordable gene-editing tools. Ultimately, these novel pathogens would begin spreading before public health systems notice, and containing them would be far more difficult than during the COVID-19 era. This is also why the debate over whether “open-source models” are easy to be misused without adequate regulation has become a focal point: how to strike a balance between advancing technology and preventing malicious intent is a serious challenge that all of humanity must face together.