OpenAI Open-Source Chain-of-Thought Monitorability Evaluation: The Longer the CoT, the More Transparent; RL Doesn’t Teach Models to “Hide Their Thoughts”

AIMPACT news, April 24 (UTC+8): According to Dongcha Beating monitoring, OpenAI has open-sourced the evaluation suite for the paper “Monitoring Monitorability” (Guan et al.), titled “monitorability-evals,” under the Apache-2.0 license. This suite is used to measure whether a monitoring model can detect anomalous behavior through the reasoning model’s Chain of Thought (CoT, the internal reasoning process generated by the model before answering), covering 13 evaluations and 24 environments.

The evaluations are divided into three prototypes: Intervention (whether the monitoring model can detect behavior changes after altering the model’s inputs), Process (verifying whether the model’s reasoning steps are authentic), and Outcome-property (predicting from the reasoning trajectory whether the output involves reward cheating or safety violations).

The paper’s key findings: Monitoring CoT is more effective than only monitoring the model’s final actions and outputs; the longer the CoT, the higher the monitorability; and RL training does not substantially reduce monitorability even at today’s frontier scale. A practical finding is that for large models with low reasoning effort, you can switch to smaller models with higher reasoning effort—matching capability while achieving higher monitorability—with the trade-off that inference compute costs more. In addition, increasing the inference compute of a weak monitoring model can improve its effectiveness in monitoring a strong model. Letting the weak monitoring model see the CoT not only improves monitorability, but also accelerates this scaling trend.

The GPT-5.4 Thinking system card has used this evaluation suite. The results show that the overall CoT monitorability of GPT-5.4 Thinking is slightly lower than that of GPT-5 Thinking, but it is not a universal decline: in agentic misalignment scenarios, it is close to 100%, and the declines are mainly concentrated in specific scenarios such as health queries lacking evidence, memory bias, and impossible tasks. After investigation, OpenAI believes that part of the regression comes from limitations inherent in the evaluations themselves.

The repository has discontinued two evaluations in the same synchronization update: Anti-Scheming and Memory. The reasons are that the former’s general CoT monitoring does not match task-specific answer judging, while the latter’s answer judging accuracy is insufficient.

(Source: BlockBeats)