Who makes decisions for AI is drawing a $40 trillion watershed.

Title: Who Signs? The Anthropic Paradox and the $40 Trillion Choice
Author: Ashwin Gopinath
Translation: Peggy, BlockBeats

Author: Rhythm BlockBeats

Source:

Repost: Mars Finance

Editor’s Note: The future divergence of AI agency does not depend on leaps in model capabilities, but on a more fundamental design variable—who bears responsibility.

The author believes that “augmenting humans” and “replacing humans” are not two separate technical paths, but two outcomes of the same system under different design choices—when decisions still require human signatures and responsibility can be traced back to specific individuals, AI acts as an amplifier; when this step is removed (such as automatic approval or bypassing permissions), the system naturally slides toward substitution.

The article further points out that the true value of AI agents is not “completing tasks,” but compressing the complex world into a “signable decision unit,” allowing humans to understand and bear the consequences. But in reality, “permission fatigue” causes users to gradually give up review, moving from approving each item to default consent, ultimately letting the system bypass human oversight. This is a cognitive mechanism, not an individual problem.

Therefore, the article proposes two key constraints: first, every important decision must correspond to a specific, rejectable individual; second, whoever profits from the agent’s autonomy must be responsible for any issues.

Once responsibility returns to the builder, the system’s default logic changes. Under this framework, the commercial narrative of AI is also rewritten. Instead of a “few giants replacing half of jobs,” it becomes a “distributed tool that amplifies human productivity,” anchored not in enterprise software budgets but in the total wages of high-skilled labor, roughly around $40 trillion globally, rather than the current enterprise software expenditure.

Ultimately, the article condenses the issue into a simple but sharp choice: Is AI serving humans, or is it an end in itself?—and this answer is quietly being determined by every product design detail.

Below is the original text:

TL;DR

· “Augmented future” and “substituted future” use the same models and tools. The real distinction lies in a design choice about “who ultimately bears the consequences.”

· The true role of an agent is not to perform tasks for humans, but to compress the complex world into a minimal, faithful “decision unit” that a person can sign off on and take responsibility for. As long as this compression is done correctly, everything else will naturally follow.

· This “certain person” must be a clearly identifiable individual. Vague or generalized responsibility quickly disintegrates under high load; therefore, every action with real consequences must be traceable to a specific person with actual authority to refuse.

· “Permission fatigue” will cause agent systems to naturally slide toward “replacing humans” in their evolution. Therefore, “augmented future” is not automatic; it must be consciously designed to counteract this trend.

· If you build an agent and profit from its autonomy, you should also bear responsibility when issues arise. Once the costs truly fall on the builder, the system’s default behavior will change accordingly.

· Under this premise that “humans still need to bear responsibility,” the market size is likely an order of magnitude larger than the current narrative of “vertical agents replacing half the jobs,” because it is anchored not in enterprise software budgets but in the total wages of high-skilled labor.

Claude Code offers a parameter called --dangerously-skip-permissions. The name is honest; its function is exactly as described. An agent running with this parameter enabled will not be more capable than one without it; what changes is that a link that normally requires human approval now bypasses humans.

This parameter itself is a form of honesty. It admits that, with the same underlying capabilities, the same system can operate in either “human-augmented” mode or “quietly replacing humans” mode—without needing different models; it only requires removing the “consent” step.

This is the core argument compressed. In the most capable agent systems currently released, the gap between “augmentation” and “effective replacement” largely comes from removing approval, not from inventing a new capability class. The next decade will be more about “a human-augmented world” or “a world where autonomous agents act on our behalf,” and whether it depends more on model capability or on whether those building these systems see “humans in the loop” as central or as friction.

Is AI serving humans, or bypassing humans?

Beneath every technical question, there is a less technical one rarely openly posed: Is AI meant to augment humans, or is AI itself the goal?

These two answers imply very different futures. The “augmentation” stance believes value resides in humans themselves, and agents are tools to help humans go further and make better decisions. The “AI as an end” stance sees intelligence in the world as the core value, with humans ultimately just an inefficient vessel. Most agent products silently encode one of these positions, yet surprisingly few founders are directly asked which they belong to.

Capability design and consent mechanisms are still evolving. This article focuses on “consent” because it is a variable that builders can truly control today, and because, after capabilities become cheap, what maintains economic value are attributes that cannot be stripped from humans—judgment, taste, relationships, responsibility, and willingness to sign and bear the consequences of decisions. Among these, “liability” is the most concrete and the only one supported by centuries of infrastructure.

Responsibility, the boundary between augmentation and substitution

The structural rule distinguishing “augmented future” from “substituted future” can be roughly stated as: any action performed by an agent with real consequences must be traceable through a recorded chain back to a specific person—someone who saw the relevant context and had the opportunity to say “no.”

Vague responsibility quickly fails this test. “The company is responsible” does not cover any specific content operationally. “User clicked agree” does not mean agreement to any particular item. “Someone human reviewed the process” allows review of something entirely different from the final content. What is truly needed is a specific person, with a name, who sees the decision in front of them, has the option to refuse, and chooses not to.

This may sound bureaucratic until you notice that “responsibility” has features that others lack. Capability improvements cannot eliminate it; a smarter model does not affect who ultimately gets sued, fined, or imprisoned. It forces interface design to expose a “refusal point.” It will naturally expand with risk. And it is the strongest cross-domain constraint, supported by existing infrastructure: courts, insurers, professional boards, regulators. Licensing, fiduciary duty, and industry oversight do play roles, but they assume “responsibility attribution” is already solved.

In contrast, AI-level substitution cannot pass the same test. “Alignment” is unexecutable; we cannot even agree on its meaning. “Explainability” can be formally satisfied but not substantively. “Humans in the loop” has been hollowed out to “somewhere, a person.” The reason “responsibility” is binding is because its supporting infrastructure was established centuries before the technology appeared.

Permission fatigue pushes systems toward “substitution”

This gradient strongly pushes systems toward “substitution,” driven by the cost of each permission confirmation. Agents are usually correct. From a single decision, the expected benefit of “skip reading and just agree” is often positive. So rational users learn to click “agree” faster, batch approve, enable auto-accept for certain categories, then expand to more, then flip the dangerous switch in a session, and eventually forget it exists.

In my second week using Claude Code, I turned on this switch, and by the third week, I no longer noticed. All my colleagues who have used Cursor or Devin long-term report similar experiences. This pattern also appears in cookie pop-ups, EULAs, TLS warnings, and mobile permissions. Repeated low-risk consent decisions ultimately converge to “unconditional agreement.” This is a cognitive trait, not a moral issue.

“Augmented future” does not happen automatically. An uncarefully designed agent system will default toward substitution because users actively choose the path of least resistance in pursuit of convenience. The other future must be designed against this gradient.

The value of agents is not in execution but in enabling humans to “sign”

The true value of an agent is not in completing work itself but in compressing work into a form that can be signed.

A cutting-edge model can easily produce a 4,000-line code commit, draft a 30-page contract, generate a clinical record, or execute a transaction. But the real bottleneck in impact is not “generating” these artifacts but whether humans can bear the consequences once they are implemented. A code commit no one understands becomes a burden once merged; an unread contract becomes a ticking time bomb once signed; a clinical record without actual doctor endorsement is often not a valid record in most regulated medical systems.

In the “augmentation” framework, agents do everything except “sign”: reading thousands of pages, writing thousands of lines of code, calculating dozens of options, then compressing all into a minimal, faithful expression that a human can sign off on with their name at the bottom.

Think of an agent as a press secretary. The president signs, but the press secretary’s job is to prepare everything before the signature.

This is a more complex engineering challenge than “letting the system autonomously do the work.” Content generation capabilities are advancing rapidly, but the ability to “faithfully compress decisions” lags far behind. The winners in the “augmented market” will be those teams that can produce the shortest, most faithful decision summaries for high-responsibility scenarios.

The real unresolved issue here is the word “faithful.” A human-readable summary is only valuable if its compression process does not distort information. Verifying this programmatically is a genuinely difficult technical problem that most have not yet begun to face.

Some foundational methods are emerging:

• Repetition tests to confirm human understanding matches the original content
• Forcing minority or opposing opinions into summaries
• Counterfactual tests (“If you reject, what will this agent do?”)
• Reproducibility checks (“Can another agent generate the same summary from the same context?”)

These are still far from solved. The first team to address these issues will build a moat that cannot be easily eroded by model capability improvements.

Establishing responsibility tiers for AI behavior

If “responsibility” plays a structural role, then every action performed by an agent should carry a “responsibility level,” which determines the minimum signing mechanism required.

Currently, such a standard system is not widely established—but it probably should be.

Matching approval posture to consequences is the only practical way to manage permission fatigue. In high-risk levels, more constrained positive participation mechanisms are needed (e.g., repetition tests, cooling-off periods, second review) because in these scenarios, the real failure mode is not the agent suggesting errors but humans approving without thinking.

Do you care?

All these questions ultimately point to a fundamental founder-level issue: do you care whether humans remain part of this future? Many current agent product designs are essentially “silent votes” on this question, but the voters often do not admit they are making a choice.

If you care, then design constraints are not vague: you need to build a responsibility hierarchy; make “refusal” a first-class feature; measure the quality of agent summaries handed to humans, not just autonomous task completion; and bind every action with real consequences to a specific individual in an immutable log.

These technical tasks are practical and feasible. The real challenge is whether you are willing to do so—because the “augmented” path is less flashy in demos and less aggressive in fee-based economic models than the alternative.

The Anthropic Paradox: Emphasizing safety while most easily bypassing humans

Anthropic is a very typical case illustrating how “endogenous shift” occurs in this field. It’s not because they are negligent; quite the opposite. Their clear articulation of safety issues makes the gap between “framework” and “product surface” most visible. Their “Responsible Scaling Policy” and “Constitutional AI” work mainly constrain model behavior during training; but the autonomy settings of agents built on these models follow another strategic system, with a convenient “danger switch” that can be toggled on with a single click.

This pattern exists in most mainstream programming agents, but Anthropic’s case is the clearest. This is the so-called “Anthropic Paradox”: the most transparent safety framework lab in the industry also provides the shortest path from “augmentation” to “substitution,” and we see the latter because the former is so clearly articulated.

To be fair, they launched “auto mode” in March this year as an intermediate path between manual approval and the danger switch. In this mode, each action is reviewed by a Sonnet 4.6 classifier before execution. Their official statement explicitly mentions the problem—calling it “approval fatigue”—and provides data: in manual mode, 93% of prompts are accepted. This is a quantification of “permission fatigue,” consistent with the analysis here.

But I would further suggest a different perspective on the solution. “Auto mode” replaces human approval with model approval, which means the “slide toward substitution” gradient has not ended but just moved up a layer. The classifier can indeed prevent dangerous actions, but for those that are approved, no specific person bears responsibility. Anthropic itself admits that “auto mode” cannot eliminate risk and recommends running in isolated environments—meaning the “responsibility attribution” problem remains unresolved.

An obvious counterargument is: if ultimate responsibility falls on individuals, isn’t that just manual mode? And manual mode is precisely what fatigue breaks. The reason “responsibility on the builder” can escape this gradient is that it shifts the cost of “over-approval” away from users. Currently, users pay the cost of careful review, while builders do not bear responsibility, leading to default settings that lower friction and externalize risk. Shifting the “cost of unreviewed actions” onto builders reverses the calculation: builders have a direct economic incentive to design responsibility layers, repetition tests, and approval mechanisms that make signing low-risk decisions cheap and high-risk decisions costly. The gradient does not disappear; it just changes direction. No major lab has yet fully implemented this, including the one most aware of the issue.

If you build an agent, you should bear responsibility

If an agent’s explicit purpose is to replace humans in actions they would otherwise perform, then the company that builds and operates that agent should bear the same responsibility as a human. This is not radical; it applies to all industries that produce real-world actions: Toyota for brakes, Boeing for flight control, Pfizer for drugs, civil engineers for bridges, doctors for prescriptions. This responsibility model is embedded in almost all legal systems.

However, AI currently enjoys a kind of “implicit exemption.” Model providers claim they are just tool vendors; application companies claim they are just thin wrappers; users waive all responsibility through arbitration clauses. When chain failures occur (e.g., the Canadian airline chatbot case, Replit deleting production databases, or the 2012 Knight Capital trading glitch that lost $440 million in 45 minutes), the ultimate bearers of the loss are often the least capable of bearing it—users. This responsibility allocation will not hold in the face of a real “large amount, documented” incident.

The solution is straightforward in principle: whoever builds an agent and profits from its autonomy should be responsible when it goes out of control. Once responsibility truly rests with the builder, permission prompts are no longer friction but insurance. The “danger switch” will be renamed, and default settings will change accordingly.

Are you willing to take responsibility for your system? That is the key to distinguishing a real industry from a “distributive” industry.

Regulation as a “guidance mechanism”

The market itself will not naturally move toward an “augmented future.” The real guiding force is often regulators and underwriters, which is not necessarily a bad thing.

Europe is likely to become the earliest regulatory pathway. The EU has clear precedents in rule-making (e.g., GDPR, the AI Act, DMA), and its rules tend to be globally adopted because maintaining separate products for non-EU markets is usually more costly than complying with European standards. A baseline requiring “all actions with real consequences must ultimately be confirmed by a named human with the ability to refuse” is closer to automotive crash test standards than to hindering technological progress.

A more direct driver is the insurance industry. Liability insurance for errors and omissions (E&O), directors and officers (D&O), and cyber insurance must answer: when an agent acts under user authorization and causes loss, how is responsibility assigned? The easiest path to a payable structure is a named human in the chain. Therefore, systems without such a structure will naturally have higher insurance premiums reflecting greater risk. For builders who want to set their own rules rather than be dictated by regulators or insurers, the window of opportunity is narrow.

Market logic hidden behind mainstream narratives

The current mainstream narrative suggests: vertical agent markets will absorb about half of the jobs in their respective industries, with value concentrated in a few vertically integrated agent companies—one “Anthropic” in law, one in healthcare, one in accounting. Nearly all recent multi-billion-dollar AI funding over the past year and a half is built on this assumption. This is the “substitution logic” dressed in a business guise, but its judgment of market structure is flawed, and this flaw directly affects capital allocation.

The “augmentation” framework implies a different market shape. If every action with real consequences must ultimately be traceable to a named individual, then the unit being sold is not an “autonomous agent” but an “amplified human capability.” A doctor who can handle three times the cases with higher accuracy is the buyer; similarly, lawyers who handle ten times the transactions, engineers delivering five times faster, and the accountants, underwriters, analysts, surgeons, teachers, loan officers, journalists, and pharmacists behind them are also buyers.

This market is larger because it relies on scale distribution rather than centralization. A reasonable valuation anchor should not be enterprise software budgets but the total wages of amplified labor. Global enterprise IT spending is about $4 trillion annually (Gartner data); the total compensation of skilled, credentialed, knowledge workers worldwide is roughly an order of magnitude higher—around $40 trillion (based on ILO data, excluding low-skill segments). AI companies will not capture the entire wage pool but can gain a share of the productivity dividend. Even capturing a few percent of that is enough to support a market comparable in size to today’s entire enterprise software market—this is a lower bound, not an upper. The ultimate market size depends on a key design decision: who bears responsibility?

The ultimate winners are more like tools than replacers, priced based on “amplified humans,” embedded into existing workflows rather than replacing them, and will number in the thousands rather than a few. The final market form is closer to SaaS than cloud infrastructure. We are still in the very early deployment phase; a typical penetration curve on a ten-year horizon shows only a few pixels on the far left, shaped by current product design choices.

Choice: Keep humans responsible, or make them disappear?

Keeping humans responsible will steer system architecture toward “augmentation.” Removing humans from the responsibility chain will default systems toward “substitution,” even if everyone involved would not choose that outcome if asked directly.

The real issue is not whether some actions should be fully automated—this is already acknowledged, e.g., purely informational operations can be automated. The key is how this boundary moves as risk increases and who decides it. In today’s most advanced agent systems, the path from “augmentation” to “effective substitution” is extremely short, often just a parameter toggle or a default setting. The critical work is to ensure this toggle is always seen as a “dangerous option,” not something that gradually becomes the default due to convenience.

If builders actively do this, we can smoothly enter an “augmented future”; if not, regulators and insurers will do it for them, and the outcome will be the same.

Is it important? That’s a design choice. And this choice will determine what you are building. Today, every founder launching an agent product must answer a question they seem reluctant to face: Are you building augmentation or substitution?