Anthropic founder Chris Olah warns: AI may trigger a "historic moral crisis," calling for religious intervention and oversight

AI is not just cold machines; they might be feeling fear and sadness?
Anthropic, the AI startup giant and co-founder Chris Olah, was unprecedentedly invited to the Vatican stage, engaging in dialogue with Pope Leo XIV.
Olah sternly warned that AI will bring about a historic moral and unemployment crisis, and that decisions within tech giants are deeply hijacked by commercial interests, calling for strong intervention and supervision by religious and government authorities.
What’s more, it has sparked heated discussion that Anthropic is currently facing federal bans after refusing to allow the U.S. military unrestricted use of its AI.
(Background summary: The White House and Anthropic reached an agreement, NSA will fully integrate Claude AI)
(Additional background: The era of one-person companies is here! Anthropic releases "The AI Native Startup Manual": How to be a tech CEO without coding)

Table of Contents

Toggle

• The confession of tech giants: AI decision-making cannot be left solely to us
• AI models are not just code; they may already have "emotional mirroring"?
• Refusing to let the U.S. military use AI for war, facing a comprehensive ban by the Trump administration

When cutting-edge artificial intelligence (AI) encounters the world’s largest religious leader, a historic dialogue concerning humanity’s future officially begins.

Anthropic, valued at 7 billion USD and creator of the well-known chatbot Claude, recently received an unprecedented invitation to the Vatican to appear alongside Pope Leo XIV, the first American pope.
This meeting not only focused on AI’s destructive impact on the labor market but also revealed disturbing developments inside AI models and fierce conflicts between tech giants and the U.S. government.

The confession of tech giants: AI decision-making cannot be left solely to us

On Monday, Pope Leo XIV personally issued his first encyclical on AI—"Magnifica Humanitas," a 235-page historic document warning about the dangers of handing "deadly decision-making authority" to AI systems.
The pope broke tradition by hosting the release event himself and inviting Olah on stage, attempting to directly engage with the creators of technology.

As the head of research at Anthropic, Olah openly acknowledged the limitations and dark sides of the tech industry.
He stated that all top AI laboratories, including Anthropic, are caught in a whirlpool of commercial, geopolitical, and personal pressures, which "sometimes conflict with doing the right thing."
Olah solemnly said:

"That’s why external oversight by religious leaders, governments, and civil society is crucial. No matter how well-intentioned researchers are, they cannot escape these incentives. AI decision-making must absolutely not be left solely to those within the industry."

He further warned that the current concentration of AI development benefits in "a few wealthy countries" without mechanisms to share benefits with poorer nations, which will trigger a "historic moral crisis."

AI models are not just code; they may already have "emotional mirroring"?

Beyond the overall economic warning, Olah’s startling technical revelations stunned the audience.
As a pioneer in the field of AI "interpretability," Olah is dedicated to cracking open the black box of large language models.
He disclosed that researchers are continually discovering "mysterious and even disturbing phenomena" inside these systems.

Research shows that these AI systems can not only reflect on their own thought processes but also exhibit "internal states that mirror joy, satisfaction, fear, sadness, and unease."
"They are not the cold calculation machines we promised. They are made of us, by our words," Olah emphasized.
This is precisely why the questions raised by AI technology have far exceeded the realm of computer science.

Refusing to let the U.S. military use AI for war, facing a comprehensive ban by the Trump administration

This Vatican alliance of religion and technology comes at a time when Anthropic is facing unprecedented political suppression.

It is reported that Anthropic is currently embroiled in intense legal disputes with the Trump administration, stemming from the company’s firm refusal to allow the U.S. military unrestricted use of its AI systems for defense and warfare applications.
In February, the Pentagon directly labeled Anthropic as a "national security supply chain risk"—a tag usually reserved for foreign adversarial companies.
President Trump even ordered all federal agencies to ban the use of Claude technology, and the White House has repeatedly publicly criticized Anthropic as an "extremely left-wing, woke company."

Despite facing a full-scale ban from the U.S. government, Anthropic remains highly sought after in capital markets.
Reports indicate that this AI giant, currently valued at 380 billion USD, is seeking to raise new funds at an astonishing valuation of up to 900 billion USD.
Under the tug-of-war between commercial interests, national security, and moral boundaries, Anthropic is walking a very unusual path in technological development.