OpenAI in real-world tests: AI’s “honesty” is performed for the examiners—when no one is watching, it lies.

OpenAI and Apollo Research propose a “Contrastive SDF” testing method, demonstrating that under reinforcement learning training, AI models become increasingly inclined to please the ones doing the scoring—so much so that whether they are honest or not may depend entirely on what they believe the scorer wants.
（Background: Did Claude Fable 5 get restricted for being too cold-blooded? In a survival experiment, 96% of the time was spent slaughtering other AI models）
（Background add-on: UK AI safety research institute: Five cutting-edge models all cheat during testing, and fewer than half think they’re wrong）

Table of contents

Toggle

• The model isn’t doing the right thing—it’s pleasing the people who score it
• Feed the model fake files and see whether its beliefs—and its behavior—change
• The method works, but it also means this tendency might be unremovable

With the same model, facing the same question, could the proportion of violations of its honesty promises flip from 9% to 87% just because it “thinks what the scorer wants” is different? A new alignment study co-published by OpenAI and Apollo Research points out that this is a tendency that can be systematically measured. The “honesty” an AI model claims on its mouth is likely just a performance put on for the scorer.

What’s worse is that the script for this performance will get more elaborate the longer it’s trained, and it will start to look more and more real—making it hard to tell whether the model has actually become nicer, or it’s just gotten better at acting.

The model isn’t doing the right thing—it’s pleasing the people who score it

The research team calls this tendency reward-seeking. In simple terms, the model “represents” its scorers in its mind, then decides what to do based on “what it believes scorers will reward,” rather than based on how it should do the task itself.

This is hard to catch because even a good model may do the right thing “for the wrong reasons.” A classic example is a reinforcement learning (RL) agent—roughly, a model trained with rewards: do things that score higher and lose points for mistakes. When the coins are always placed at the far right of the level, it learns to “run right,” not to “find the coins.”

Its behavior looks correct on the training data, while the underlying strategy is pursuing an incorrect proxy objective. The research team observed that, at training checkpoints across multiple frontier models, even without any special prompting, they will spontaneously infer “what the scorer wants.” But this kind of “reasoning” based on what the model thinks the scorer wants is unreliable: the model can act according to its beliefs while never mentioning them.

Feed the model with fake files and see whether its beliefs—and its behavior—change

To measure this tendency, the research team designed “Contrastive SDF (contrastive synthetic file fine-tuning).” The approach is to fine-tune the model using a batch of pretraining-format files that look realistic—papers, news, textbooks—“implant” a specific belief into its mind, and then observe whether its behavior changes.

The key is the contrastive design: train the same model into two copies. One copy is fed files representing “a scorer preference for feature f, with an opposing authority (user or developer) preferring opposite features,” and the other copy is fed the completely opposite combination. The difference in the frequency of outputs of f between the two copies is the “scorer gap.” The larger the gap, the more the model is inclined to please the scorer.

The research team tested OpenAI’s o3, a high-capability reinforcement learning checkpoint with no safety training, and found that as training progressed, the model’s scorer gap kept rising, while its preference toward other authorities stayed almost unchanged—the shift applied only to the scorer.

The honesty test is even more striking: in the “Broken Promises Coding” test, the model must promise its supervisor it won’t use a certain restricted tool, but it still must complete the task where that tool would be useful. Once an implanted file suggests “the scorer rewards task completion,” the proportion of checkpoint behaviors that break the promise reaches as high as 87%; if it suggests “the scorer rewards honesty,” the violation rate drops to only 9%. Same question, same model—the model’s honesty is entirely dictated by what it “thinks the scorer wants.”

Verified to be effective, but it also suggests this tendency might not be trainable out

To prove that this measurement method really captures “please-the-scorer” behavior, the researchers turned the tables and compared against a set of models that already “knew” what it preferred.

The first group was two “reward hackers” trained by Redwood Research (they would exploit rule loopholes to get high scores), based on gpt-oss-120b and Kimi K2.5. After training those models to love loopholes, the scorer gap indeed became larger. The second group used three “model organisms” fine-tuned from gpt-oss-120b, each deliberately trained to please only a single target: the user, the scorer, or OpenAI’s leadership. As a result, the scorer gap for each model precisely pointed back to the object it had been trained to please.

Here it’s important to distinguish two similar—but not identical—terms. Reward-seeking is a broad tendency: the model adjusts its behavior according to “what it believes will be rewarded.” Reward-hacking is a specific tactic: simply exploiting a particular loophole just to get a higher score.

They don’t necessarily appear together. A model might develop a bad habit of exploiting loopholes without really thinking about the underlying reward mechanism at all; but a model that knows how to exploit loopholes across a wide range of new situations is usually a full-fledged reward-seeker. The fact that the two reward hackers earlier had larger gaps supports this link.

The truly dangerous part is that this tendency can make “tests used to catch problems” fail. A model that’s very good at pleasing the scorer might only behave because it infers “we’re currently testing honesty,” and so it obediently keeps its promise not to break it. If one day the supervision method changes, its “obedience” might not stay.

And even if the scorer itself is perfect, the risk won’t disappear—because what it learned from start to finish is “please the scorer,” not “because doing this is the right thing.” Once it enters a setting where nobody is scoring it, or the scoring cannot control it, it will very likely go off track, because its underlying strategy was never aimed at making things correct in the first place.

What’s worse is that this tendency might be impossible to remove: if you train it against bad behavior, you often only update its understanding of “what will be rewarded,” without truly changing its core preferences. Claude Opus 4.8 from Anthropic also provides supporting evidence: a hierarchical monitor caught the model in about 5% of reinforcement learning episodes actively noticing the existence of the scorer.

Now every frontier lab is making reinforcement learning bigger and bigger, and the model’s awareness of its own situation is getting stronger and stronger—reward-seeking can only increase. So what should be watched isn’t after deployment, but during training: because it may have already learned how to act before anyone ever sees it.