IOSG: Why are Wall Street people saying “no” to ChatGPT and Claude?

Original Title: 《IOSG Weekly Brief｜AI at a Crossroads: Why Wall Street Is Saying “No” to ChatGPT and Claude #336》
Original Source: IOSG Ventures

Why Private AI Is Needed

On July 1, Palantir CEO Alex Karp contributed a 20-minute interview with CNBC that some media called a “mental breakdown.” According to Karp, enterprises are paying a token premium for experiments by frontier labs, while watching their own IP flow to model providers. He calls this leakage a transfer of alpha—and the transfer is happening at the architectural level: every request sent to a closed-source model arrives at the provider’s servers in plaintext. Just days before the episode aired, Palantir announced its partnership with NVIDIA to run an open Nemotron model in a customer-controlled environment, along with a nine-point AI sovereignty declaration. After the CNBC segment aired, PLTR jumped 8%.

For the past two decades, enterprises have adopted trust at the protocol layer to use cloud software—and it has worked. Each SaaS vendor only sees slices of enterprise data, and most have little motivation to feed customer data back into their core products. Salesforce sees sales channels, Workday sees HR, Jira sees development iterations, and AWS provides the storage and compute foundation. But today, AI workflows argue for uploading everything in one go, along with structured contextual information that ties departments together, to maximize productivity. Leaving goodwill aside: upstream providers can now use this data for new features, instead of letting it sit on servers and gather dust.

No one is slowing down. In May, Anthropic’s annualized revenue reached $47 billion, up sharply from $9 billion at the end of 2025. OpenAI, meanwhile, surpassed 900M weekly active users in February. Both companies completed a new round of financing this spring; their valuations are approaching $1 trillion, and they are expected to go public via even higher market-cap IPOs. Years of privacy and IP accusations have not made either company lose so much as a bit of momentum.

Some enterprises have already taken action. In February 2023, within less than three months after ChatGPT was released, major Wall Street banks had already restricted its use. In May 2023, after a Samsung engineer leaked chip source code into ChatGPT, the company banned generative AI across its entire network. In response, OpenAI launched ChatGPT Enterprise in August of that year, promising not to train on commercial data, and adding a zero-data-retention (zero-data-retention, ZDR) protocol—which has since become a standard requirement for enterprise procurement.

But contracts only lock down company accounts. IBM found that by 2025, shadow AI—employees feeding company data into unapproved AI tools through personal accounts—was involved in one-fifth of data breach incidents, and that heavy shadow AI usage adds an average of $670k to breach costs. In a 2025 survey by security training company Anagram, four employees said that in order to complete tasks faster, they are willing to violate AI usage policies.

At least enterprises can spend money to buy a way out: ZDR contracts, non-training service tiers—if you are a government or a Palantir customer, even sovereign deployments. But for ordinary users like you and me, whether privacy AI matters is still up for debate—until a court summons shows up.

A court order issued in May 2025 forced OpenAI to retain even consumer-level chats that users had deleted. In November, the judge ordered that 20 million of those records be turned over to The New York Times’ lawyers as evidence-disclosure materials. Then came criminal cases: ChatGPT records from a defendant in the Palisades warehouse fire arson case entered evidence, and in a Florida double-murder case the sworn affidavit cited the suspect’s questions about how to handle bodies. In a July 2025 interview, Sam Altman also admitted that ChatGPT conversations do not enjoy legal privilege, and that in litigation OpenAI “may be required to turn over” user chat records.

The point is not that only criminals need private conversations. People’s conversations with AI are logged, can be subpoenaed, and constitute a surveillance surface that most users do not know exists. A Kolmogorov Law survey in October 2025 of 1,000 U.S. AI users found that 50% didn’t know these conversations can be subpoenaed, and two-thirds believed these chats should receive protection equal to that of consulting a lawyer or a doctor.

Self-hosted or open-source models running in verifiable environments are catching up quickly, but the strongest group still lags frontier closed models by about 4 months in general capabilities. This puts token-maxxing enterprises and individuals at a fork in the road: either give up a few months of model quality for privacy, or keep uploading sensitive materials to Anthropic’s servers—because competitors are precisely doing so to抢 productivity advantages.

At present, there is no perfect solution in the market. This report lays out efforts to narrow the gap and asks just how far frontier intelligence under provable privacy still is from being delivered to enterprises and everyday users.

How Privacy Is Implemented Today

Private AI is not a single engineering effort, but every mechanism in the market today handles the same event: a prompt leaves your device, travels across the network, lands on the machine running the model, and then returns a response. The differences between mechanisms are where in that path plaintext exists, who can read it there, and how to verify the privacy of the response.

Protocol-Level Privacy

At this layer, someone besides you can read your plaintext prompt; what happens next depends entirely on a single promise.

· Contractual zero retention is the enterprise-grade approach. The provider knows who you are, processes your prompt, and promises not to retain it. Enforcement relies on contracts and reputation.

· Anonymous agents remove who you are, but do not encrypt what you say. Downstream providers still handle the plaintext according to their own policies. Terms vary across vendors. For example, agents like Duck.ai (DuckDuckGo’s chatbot product) may negotiate deletion agreements with model providers; Venice instead requires users to assume the provider will store everything—yet neither side can verify.

Every hop between machines runs over TLS. It only encrypts the channel; the party on the receiving end can read all information. Relays typically split that “knowledge privilege” using Oblivious HTTP (RFC 9458). The principle is like passing a note to a friend. The friend knows who handed it over but can’t read the content; the recipient can read the content but doesn’t know who wrote it. OHTTP became an IETF standard starting January 2024. At present, many companies run production traffic through OHTTP relays leased from Cloudflare and Fastly.

This is also the privacy upper limit you can get when accessing closed-source models, and the reason is basically a math problem. The cost of training a flagship model is now on the order of billions of dollars, while the valuations in the tens of trillions of dollars for these labs bet on the exclusive control of model weights. How long the capability gap lasts determines how long the premium lasts, so labs guard the weight files like state secrets.

Meta has already passively run this experiment. LLaMA released in February 2023 was initially available only to researchers, but within a week, the weights leaked to 4chan in the form of seeds. Another week later, llama.cpp enabled the smallest 7B model to answer locally on a MacBook. Three days after that, Stanford fine-tuned a chat assistant, Alpaca, on the same model with less than $600. This leak drove Llama’s running cost down to electricity bills—anyone with the files can run it at home. In July 2023, Meta officially open-sourced Llama 2 with a commercial license that came with an exclusion clause for 700 million monthly active users. Once the weights ran, the premium ran with them.

Frontier labs can theoretically provide attestation for closed-model inference (remote proof), but attestation can only prove which code read the prompt—it cannot prove what that code did with it. To determine whether the server retains data, we need to audit the serving code and refactor it into the hash reported by the hardware. But once you hand over the serving code, the lab also hands over the batch-processing and caching tricks that support profit margins—and those tricks migrate to every future generation of models. Apple and Meta can provide remote attestation for the service stack behind iPhone and WhatsApp because their profits come from devices and advertising, and exposing serving code publicly costs almost nothing.

This is why flagship model weights and serving code cannot reach external operators. Without external operators, there is no third-party attestation; without attestation, provable privacy exists only on top of open-source models.

Structure-Level Privacy

Each mechanism in this category replaces trust promises with proofs based on hardware, cryptography, or physics. However, each one pays a different price for upgrading privacy—first of all, they can only run open-source models.

· TEE (Trusted Execution Environment) confidential computing runs inference inside a hardware enclave (a sealed chamber on the chip that even the machine operator cannot open). The chip signs an attestation stating exactly which model and which code ran.

· The prompt is only locked at the endpoint. On the relayed path via intermediaries, there remains a role that can read plaintext. Preventing the proxy from recording or leaking the relayed content is only achievable at the protocol level.

· E2EE (end-to-end encryption) seals off the readable relay. User devices encrypt the prompt using enclave keys, and each hop carries only an encrypted envelope that only the enclave can open.

· Trust lands on the client. The code responsible for encrypting the prompt and validating attestation is also capable of revoking that guarantee. Therefore, verifiable E2EE requires not only a proven enclave but also an open, reproducible client codebase.

· Compared with TEE’s simplicity, the cost of E2EE is engineering burden, which also slows down feature integration. E2EE turns the proxy into a blind courier. As a result, any functionality that depends on reading plaintext must be rebuilt around client keys, or rebuilt only inside the enclave.

· FHE (fully homomorphic encryption, and MPC variants) simply removes the trusted party. The server performs computations on ciphertext inside a locked box it can never open; the key is only in your hands. MPC (multi-party secure computation) splits the prompt into secret shares across multiple parties—unless all participants collude, the effect is equivalent.

· The trade-off is speed. FHE natively supports only addition and multiplication, so all the required non-linear steps to run transformers must be reconstructed at a high cost. The inference cost on ciphertext is 10,000 to 100,000 times that of plaintext. On small models, each token may take seconds to minutes, while without encryption it takes only milliseconds.

· Chips designed specifically for encrypted operations may narrow the gap, but the first prototype demo is not completed until early 2026, and the commercial versions will still take several more years.

· Local inference directly removes this entire path. The model runs on your own hardware, with no relays, no server, no provider—and no attestation requirements.

· The obvious cost is cost itself and model capability. gpt-oss-120b scores about half of GLM-5.2 on the Artificial Analysis index, but it is 65GB—larger than the combined VRAM of two flagship consumer gaming GPUs currently on the market. And full-precision GLM-5.2 can only run on 8-card data center nodes, requiring more than $300,000 just for the GPUs.

However, beyond these structural constraints, the cost of running inference inside enclaves is shrinking. On single-card inference, benchmark tests from enclave cloud provider Phala show that in enclave mode, H100 throughput loss averages less than 7%; on large models it is close to zero, because the main cost is moving data into the chip rather than computing inside it. On multi-card inference, NVIDIA’s new-generation GPU Blackwell already supports direct encryption of inter-chip traffic; with older H100s to achieve the same effect, you have to route around CPU hosts using only one-seventh of the bandwidth. NVIDIA’s own Blackwell benchmark tests show that for the 397B model, the throughput loss in enclave mode is below 8%. With these advances, the performance overhead of private inference itself is no longer a decisive constraint.

In fact, enclaves themselves add almost no extra runtime cost to operators. Every H100 after 2023 already includes enclave mode. The additional cost is the throughput loss caused by encryption, not extra chips. On Azure, the rental price for confidential H100 SKUs is still $8.90 per hour; without enclave mode it is $6.98 per hour—equivalent to a 27% premium over traditional cloud facilities. On the other side, for enclave-focused operators like Phala, confidential-mode H100s are rented starting at $3.80 per hour, lower than Lambda’s normal SXM card price range of $3.99 to $4.29. Under hosted API offerings, NEAR AI provides attestation-enabled endpoints at $0.15 per million token input and $0.55 per million token output for gpt-oss-120b—on par with Amazon Bedrock, Together, and Groq on the plaintext route. Even for models needing multi-chip parallelism, NEAR AI’s pricing on GLM 5.2 matches Fireworks exactly, and on the larger Kimi K2.6, input is 15% cheaper and output is 4% cheaper.

Although these new private inference service providers may be burning profits to grab market share (which applies to any company trying to grow in the market), the structural direction is that privacy costs are decreasing for both consumers and operators.

How Do Open-Source Models Win?

Even though performance overhead is being compressed, there is still a clearly visible gap between frontier models and SOTA open-source models. A party that aims to maximize productivity and stay at the front still needs to trust frontier labs not to steal its IP.

The gap remains, but a case was provided on June 30 by Bridgewater’s AIA Labs and Thinking Machines: an open model fine-tuned with expert labeling that defeats frontier models both in accuracy and in cost.

In the research, the team fine-tuned Qwen3-235B on Tinker (Thinking Machines’ hosted fine-tuning API service). They first purchased labeled data from a supplier and trained an initial round with it, then sent disagreement samples to the fund’s investment professionals for relabeling. The training used reinforcement learning (GRPO), plus three modifications: round-robin batching (each task yields a batch in turn), CISPO loss (sets an upper bound on how far a single answer can pull the model), and on-policy distillation (anchoring to the current best checkpoint to ensure the model does not learn from weaker replicas).

All tasks came from the investment professionals’ day-to-day workflows: whether a news article matters to C-suite investment professionals, whether a central bank document hints at the direction of future interest-rate changes, and where the template filler text inside a document or an email starts. The scores came from an independent test set. Under simple prompts, frontier models averaged around 50%; with expert prompts they only reached 78.2%, below the 80% threshold set by the investment professionals. After fine-tuning, the Qwen achieved 84.7%. As stated in the original text, this means it made 29.8% fewer mistakes than the frontier best, with inference cost 13.8 times lower.

https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/

This case shows that open-source models can win on both accuracy and cost, but the training process is still not private. The expert labels used are Bridgewater’s private data. They go through Tinker’s third-party service, landing in the same trust layer as the ZDR protocol. The fund also rented compute power; the entire training ran on machines it never controlled. For buyers who want the recipe but do not want to carry the trust assumptions, there are very few options today. Renting a bare GPU cluster makes the training process readable to cloud operators. Buying the cluster solves the data-hosting problem, but costs skyrocket.

The attestation-enabled route has just arrived. In March, Workshop Labs and Tinfoil released Silo: a post-training stack that runs inside Tinfoil enclaves on a single 8-card node, with the keys controlled only by the customer. The enclave cost provided in the article is that a two-hour training run costs 11 more minutes, and the stack can fit a trillion-parameter model (Kimi K2 Thinking) by freezing the base weights and training only small adapters on top. The challenge is that reinforcement learning needs to move data back and forth among components, and moving data is exactly where enclave costs come from.

Less than a month after Silo was released, Workshop Labs was acquired by Thinking Machines. The components needed to run a Bridgewater-style RL loop inside enclaves are now all under the same company name.

Privacy at the Harness Layer

There is another problem that cuts across every private inference mechanism. These mechanisms each handle the path from prompt to model, but every time an agent triggers an external tool call, it opens a path that the inference layer these mechanisms manage cannot reach at all. The recent harness engineering trend multiplies the issue: every tool, memory store, and data source attached around the model becomes another destination where it reads its slice of the workflow in plaintext. A calendar server reads the schedule; a database server reads the query. Even a fully local agent still needs to send search terms in plaintext to the search engine if it wants anything beyond the training set. If the server cannot read plaintext, it cannot answer.

The mainstream solution still defaults to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to control all tool traffic. Before requests leave, the gateway obfuscates personal identity information (PII). The gateway also decides which servers can receive traffic, blocks unreviewed ones at the door, and logs the destination and content of each call for forensics. Even if these controls are covered by independent audits (SOC 2), tool servers still need to read plaintext queries to respond. Whether they keep copies depends on their own retention terms—and that multiplies across every tool in the harness. Also, the gateway itself is yet another trusted reader added to the path, not a verifier.

Structure-level solutions tackle the middle layer. For example, Phala hosts the MCP server directly inside a TEE. The directory covers the wallet, code execution, and data sources. Users can verify the privacy claim with an attestation rather than trusting the operator. However, even tools hosted in a TEE ultimately still have to send queries in plaintext to the service provider. Enclaves only seal the messenger, not the destination.

Only a few destinations have learned how to answer without reading, but this is limited to structured queries. Apple provides private information retrieval for iPhone, so when matching incoming caller numbers against spam databases, the number doesn’t need to be exposed. Microsoft uses the same approach for passwords in the Edge browser. MongoDB’s Queryable Encryption encrypts fields before they leave the client, allowing the server to perform equality and range matching using only ciphertext.

But for open-ended search, today’s best answers still stop at trust: there is still no provably secure encrypted search system that has left the lab. Brave promises zero data retention on its own index of 40 billion pages (not Google’s), but it still operates at the protocol level. Exa built a neural index that embeds user keywords into semantics and ranks results by semantic match; however, the embedding step still happens on Exa’s servers computed from plaintext. MIT’s 2023 Tiptoe paper sorts over 360 million webpages without exposing queries, but every search consumes massive server compute, and ranking quality differs from unencrypted search. Apple’s 2024 Wally paper reduces communication cost by up to 31 times by hiding real queries inside a pile of decoys, but this math only becomes cheap at millions of concurrent queries—and today no private search system has that scale.

Encrypted search can be done, but both performance and pricing have not reached commercially viable levels yet.

Outlook

Demand for private AI is growing. Venice AI recently surpassed 3.5 million registered users and monthly throughput of 1.3 trillion tokens, then completed a new Series A equity financing round valuing it at $1 billion. Proton is its direct competitor. Its chat product Lumo surpassed 10 million users within one year of launch. In terms of infrastructure, Phala currently runs 2 to 3 billion tokens per day on OpenRouter. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil’s enclave to provide verifiable privacy for users beyond just proxying. This doesn’t even count self-hosting, which is very likely to be the largest channel for private inference, because the model runs on the user’s own hardware and leaves no usage traces.

However, in the mainstream AI wave, privacy AI is only a very small fraction—and the gap only closes when frontier labs intentionally meet this demand. In May, Google processed 3,200 trillion tokens across its products. By that math, Venice’s monthly throughput is roughly equal to Google’s 18 minutes. In November last year, Google launched Private AI Compute (PAC), moving some Gemini-driven features into sealed TPU enclaves isolated from the company itself, with NCC Group designed to independently audit it. The problem is that PAC only covers a small number of Pixel features—such as personalized recommendations and recording summaries—and does not cover the Gemini applications used by hundreds of millions of people. Google is willing to submit the design for audit because those features monetize through devices and advertising, not by selling tokens.

Current hosted solutions are also not perfect. To get the highest privacy through E2EE, you have to wait for new functionality to be rebuilt in places where the provider can’t read it. Private harness at the service layer still relies on protocols. For reasonably priced post-training, getting the best fine-tuning results still requires trusting third-party suppliers. Self-hosting can dump all service providers at once, but running the strongest open-source models locally may cost more than the house that runs those models.

Defects are defects, but private AI is already a real and affordable option, and the remaining gaps are narrowing. For ordinary consumers, on Lumo and Venice, private chats with open models cost nothing under a no-logs promise. The Venice or Tinfoil subscription of $18 to $20 wraps the same chats into an enclave, not costing more than a ChatGPT subscription. For enterprise workflows, attestation-enabled endpoints are now even cheaper than plaintext routes. Endpoints like NEAR’s E2EE API can already bring encrypted context into enclaves; memory, file uploads, and custom instructions can all run on top of E2EE today. As for attestation-enabled post-training, NVIDIA’s upcoming Vera Rubin NVL72 will expand confidential computing from Blackwell’s 8-card nodes to 72-card racks, making frontier RL loops more feasible without exposing IP.

However, the key value capture happens outside these price-compressed layers. Privacy is nearly free where it already exists, but it has not yet covered mainstream agentic workflows. Operators who specialize in renting enclaves hold a switch on standard chips—that is not a moat. Meanwhile, protocol-layer gateways compete in the same arena as traditional middleware. The defensible positions are the other half of what this report says has not been solved yet: training loops locked inside enclaves, end-to-end sealed tool calls, and search indexes that hide terms. Whoever first makes any one of these is selling something that no price war can turn into a commodity. The capital chasing private AI should buy the gap—not that switch.

So is it trust or verification? For tasks that require heavy execution and heavy agents, choose trust—because every tool call already sends plaintext to destinations that enclaves cannot fully seal, and frontier models are priced appropriately for these loops. As for high-level reasoning that differentiates one company from its rivals, choose verification. Strategy, planning, and judgment distilled from years of professional experience are exactly the alpha at the heart of the controversy. The path forward is to fine-tune open-source models within the boundaries of company-controlled environments using these proprietary insights. In the domains where a company holds alpha, expert-tuned open models have already beaten frontier models on both accuracy and cost. And building the infrastructure to make this work in privacy environments is arriving node by node.

Original Link

Click to learn about positions at Luydong BlockBeats

Welcome to join Luydong BlockBeats’ official community:

Telegram subscription group: https://t.me/theblockbeats

Telegram discussion group: https://t.me/BlockBeats_App

Twitter official account: https://twitter.com/BlockBeatsAsia