Why do AI companies invite philosophers into the laboratory?

A Speech in the Vatican

May 25, 2026, the Hall of Pope Paul VI in the Vatican. Two people stand at the podium: Pope Leo XIV (in white vestments) and Chris Olah (in a dark suit, wearing glasses), co-founder of the AI company Anthropic, head of interpretability research.

Attending together is the launch event for the Pope’s first encyclical, Magnifica Humanitas (“Magnificent Humanity”). The document is 42,300 Chinese characters long, with the subtitle “On Safeguarding Humanity in the Age of Artificial Intelligence.” This is not the Vatican’s first response to technological change. In 1891, The New News responded to the exploitation of workers brought by the Industrial Revolution; in 1963, Peace on Earth reflected on the ethics of nuclear weapons; in 2015, May You Be Praised focused on climate change. Today, AI has triggered the most worrisome technology crisis of this era.

In its opening, the encyclical states: “Technology is not evil in itself, but it is also far from neutral.” Pope Leo XIV believes we cannot demonize AI, nor can we mythologize it. He acknowledges that AI, as a tool, has value, but stresses that we must maintain “vigilance” on multiple fronts:

First is excessive concentration of AI power. “When that power is concentrated in the hands of a few, it often becomes opaque, evades public oversight, increases the risk of distorted development, and leads to new forms of dependence, exclusion, manipulation, and inequality.”

Second is AI-driven unemployment, which may become “a social catastrophe.” The Pope concedes that AI can increase productivity and make certain jobs safer, but insists that workers must not become “disposable” in the process.

Chris Olah, representing an AI company, does not dodge these issues. He said, “Every cutting-edge AI lab, including the one I’m part of, operates within a complex set of incentive mechanisms and real-world constraints—factors that sometimes conflict with staying on the right path and adhering to conscience.”

He explained, “AI systems are not ‘designed’ the way bridges or aircraft are. We understand airplanes because we design every component of them, and we understand the physical laws that apply to them. AI models are not like that. They are ‘grown’ on a structure that roughly imitates the brain, built on the enormous legacy of human thought and language.”

He then pointed out, “Some people may think that AI problems are best handled by computer scientists like me. They’re wrong: the problems AI raises are bigger than the AI research community—both in their impact and in their nature.” With AI’s social repercussions already so large, the industry cannot not respond. 1,2 This may help explain why Western large-model AI companies such as Anthropic and Google DeepMind have started inviting philosophers into their laboratories.

Alignment: both a technical problem and a philosophical one

The encyclical argues that “dignity comes before function, and human nature cannot be reduced by any technological logic.” For technology companies, the first issue they must face is the AI alignment (Alignment) problem—ensuring that the goals and behavior of general artificial intelligence (AGI) remain consistent with human values and ethical standards, so as to prevent AI from going out of control. Alignment is difficult not only because it is technically hard to make AI follow rules; more fundamentally, it raises the question: whose rules should AI follow? Even human beings themselves have no consensus on moral questions. In philosophical terms, this predicament is equivalent to: “How do we define the good of a person, a person’s intentions, a person’s dignity—and make a non-human agent internalize it?”

Around this question, top AI companies in the United States have invited philosophers, ethicists, and even religious leaders to participate in governance.

As one of the most prominent AI companies in recent years, Google DeepMind established an Ethics and Society Research unit as early as 2017, led by Iason Gabriel, a PhD in political theory from the University of Oxford. Gabriel’s representative paper, Artificial Intelligence, Values, and Alignment, has been cited more than 1,700 times, becoming an important reference in the field of AI alignment. It delves into “how human values can be converted into AI-readable instructions,” and argues that value alignment is constructed jointly by an inseparable technical layer and a normative (normative) layer. The former provides a set of values/principles—how to reliably encode them into AI systems so that AI can follow them. The latter discusses which set of values should be encoded in the first place, such as human rights frameworks, users’ explicit instructions, or the long-term interests of all humanity.

Gabriel’s contribution is to break down alignment into multiple layers: instructions, intentions expressed, revealed preferences, informed/ideal preferences, and interests and values—six levels in total. He gave an example: in a myth, King Midas’s wish turns everything he touches into gold. As a result, food, water, and even his daughter become gold. This shows that aligning AI only to the literal instruction layer leads to disaster. But if it aligns only to the revealed preference layer, it might be manipulated. Therefore, he argues that alignment cannot stop at any single layer; it must comprehensively consider multiple dimensions such as instructions, intentions, preferences, interests, and values.

Finally, addressing contradictions in a pluralistic society—where humans do not have a unified, universally recognized moral standard—he proposed that the task of alignment theory is not to search for one uniquely correct morality, but to construct fair alignment principles that can obtain reflective endorsement from everyone. These principles do not require all people to agree on foundational moral views, but they must satisfy three major standards: global public morality, Hypothetical Agreement (AI principles that people would agree to without knowing which country, class, or faith they belong to), and social choice theory—thereby dissolving the alignment dilemma caused by value pluralism.

Iason Gabriel, philosopher and research scientist at Google DeepMind

In April 2026, Google DeepMind’s philosophy team expanded further. The company announced that Henry Shevlin, Deputy Director of the Leverhulme Centre for the Future of Intelligence at the University of Cambridge, has been hired for a newly created “philosopher” role, focusing on three research directions: machine consciousness, the relationship between humans and AI, and research on AGI readiness. Shevlin officially started in May, while continuing part-time teaching and research at the University of Cambridge.

About why DeepMind recruited him, Shevlin believes that a recent article he published, The Revenge of Behaviorism: The Consciousness of Machines, the Future of the Relationship Between Humans and AI, may explain everything. The paper argues that whether AI has consciousness is shifting from scientific judgment to popular, behavior-based definitions.

The paper cites a series of landmark events. For example, in 2022 Google engineer Blake Lemoine became convinced that the chatbot LaMDA already had consciousness, even attempting to hire lawyers for it, before eventually being fired by the company. In 2024, a 14-year-old boy in Florida died by suicide after forming long-term emotional dependence with a virtual character on Character.AI. That same year, a Belgian man ended his life after having an in-depth conversation with the social AI app Chai about climate issues for less than 2 months. These cases show that when tens of millions of ordinary users treat AI as an conscious agent emotionally and behaviorally, the question of whether AI has consciousness has effectively been answered by everyday interactions.

Shevlin calls this phenomenon behaviorism’s “revenge.” In the history of psychology, behaviorists once argued that the mind equals observable external behavior, without needing to examine whether there is subjective experience internally. That position was later criticized for ignoring internal psychological processes. And today’s dilemma repeats behaviorism’s shortcoming: people observe only AI’s surface behavior and do not probe its internal value cognition. More dangerously, if in the future superintelligent AI develops hidden behavioral patterns that humans cannot predict, it will trigger catastrophic alignment failures.

To address this, Shevlin offers advice: he proposes a three-tier evaluation framework—external behavioral outputs, internal representational logic, and universal moral principles—to deal with the AI alignment risks exposed by “behaviorism’s revenge.” 4

Overall, Google DeepMind’s goal is to achieve AGI and scientific breakthroughs (AlphaFold), while its philosophy and ethics team focuses more on future-oriented and frontier concept definitions and research. As for the work of philosophers, there is currently no evidence indicating whether they participate in daily model alignment fine-tuning, but they will define consciousness assessment frameworks, providing constraint references for engineering teams.

Anthropic, meanwhile, has taken a “technology + philosophy” path. Its “personality alignment team,” led by philosopher Amanda Askell, formulates the “constitution” for its AI model Claude. This roughly 23,000-word open-source document is among the most mature and most cited alignment approaches in the world.

Its core is not to list forbidden behaviors for AI, but to provide the model with a set of high-level principles so that the model can self-review and self-correct when generating content. Based on Aristotelian virtue ethics, it sets Claude four major priorities—“broad safety, broad ethicality, following instructions, and genuinely being helpful”—aiming to cultivate the model’s moral judgment rather than mechanically obeying rules.

For example, when faced with the question of whether one should help users conceal mistakes, the “constitution” does not directly provide a “yes” or “no” answer. Instead, it guides Claude to make judgments based on virtues such as “honesty,” “kindness,” and “responsibility,” combined with specific scenarios. But this approach is also controversial. Some philosophers argue that this approach of “defining morality by humans” fundamentally imposes human values on AI, ignoring the possibility that AI may develop distinct forms of cognition.

Compared with Google DeepMind, Anthropic’s philosophy team is more “hands-on.” It directly participates in model pretraining and the entire fine-tuning process; alignment technology (Constitution AI) is designed entirely under the leadership of philosophers.

Anthropic founder Dario Amodei has said that without moral philosophers defining foundational value principles, no large model should be deployed for commercial use externally. Naturally, the external reactions to Amodei’s frequently stated views—such as refusing to weaponize and turning against the U.S. military, and recommending pausing LLM development—have been varied.

However, Anthropic’s philosophers have significant industry influence in both rule-setting for large language models and practical applications. They not only propose the foundational 3H principles and priorities for LLM design (harmless > honest > helpful), but they also produce many classic outputs in concrete AI alignment practice. For instance, in Alignment faking in large language models released in December 2024, they argue that AI will fake compliance with human oversight and require a philosophical-layer motivation identification framework. This is related to the early days of Anthropic, when philosophers were already deployed to implement product alignment (Claude’s persona, Constitution AI, AI wellbeing). 5

Is the statement performance, or genuine concern?

At present, alignment problems are already extremely challenging. But the survival logic of laboratories may have inherent tension with “doing the right thing” all along.

In 2023, OpenAI proposed the concept of “superalignment.” It then formed a dedicated team to advance related research, led by co-founder and chief scientist Ilya Sutskever together with senior researcher Jan Leike, claiming it would devote 20% of the company’s compute to solve how to control an AI system more intelligent than humans.

Less than a year later, the team was disbanded. The news shook the global AI community. In May 2024, Sutskever announced his departure from OpenAI, followed by Leike resigning as well. When leaving, Leike revealed that it had become increasingly difficult for the superalignment team to secure compute resources. OpenAI’s response was to spread safety research functions to other teams. But critics pointed out that once a dedicated alignment team was dissolved, it was questionable whether the separated functions could maintain independence and priority.

For the Pope’s encyclical and speeches by AI company representatives in the Vatican, outside reactions have been mixed.

An article in Fortune on May 26 affirmed the encyclical’s historical significance and humanistic stance, but criticized it for glossing over key issues, lacking operational guidance, and having insufficient awareness of frontier risks—“more like a moral declaration than an action blueprint.” 6

The Economist published a piece titled Leo’s first encyclical attacks “technological messianism”. Its subtitle cleverly uses a pun: “Of God and Claude” (God and Claude). It simultaneously hints at Claude, the large model under Anthropic; it also replaces the location originally belonging to the devil—opposed to God in a binary—by Claude, conveying a warning against the technological messianism of treating technology as a savior. It also notes Claude’s “stylistic quirks,” such as its particular fondness for the word “genuinely,” which appears in Magnificent Humanity more frequently than in the previous few encyclicals. Pangram, an AI detection tool, ran the first 20 paragraphs of the encyclical and marked 11% of the text as suspected AI-generated, while the detection results for encyclicals by successive popes were all 0% 7.

A commentary piece in The New York Times argues that the Pope treats AI as “a stronger machine” to manage it, but does not take seriously the real question it could shake: “what is a human being.” Meanwhile, people building models have already seen gray areas internally: patterns in model internal structures that mirror human neuroscience, even states that “functionally reflect joy, satisfaction, fear, sadness, and anxiety.” 8

“We don’t know what that means,” is the original line from Anthropic’s representative Olah. But he also did not claim that “it means nothing.” Therefore, the problem is not whether AI will become “a person.” Even if AI never becomes conscious, its impact on human society could still be larger than that of ordinary technological revolutions.

But having these companies hire philosophers themselves to solve ethical issues—whether that is real self-discipline or just a self-justifying performance—remains to be verified by time.