Paper Info

• Author: Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, David Farhi
• arXiv:2503.11926
• Submitted on 14 Mar 2025

Abstract

Mitigating reward hacking–where AI systems misbehave due to flaws or misspecifications in their learning objectives–remains a key challenge in constructing capable and aligned models. We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments by using another LLM that observes the model’s chain-of-thought (CoT) reasoning. CoT monitoring can be far more effective than monitoring agent actions and outputs alone, and we further found that a LLM weaker than o3-mini, namely GPT-4o, can effectively monitor a stronger model. Because CoT monitors can be effective at detecting exploits, it is natural to ask whether those exploits can be suppressed by incorporating a CoT monitor directly into the agent’s training objective. While we show that integrating CoT monitors into the reinforcement learning reward can indeed produce more capable and more aligned agents in the low optimization regime, we find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking. Because it is difficult to tell when CoTs have become obfuscated, it may be necessary to pay a monitorability tax by not applying strong optimization pressures directly to the chain-of-thought, ensuring that CoTs remain monitorable and useful for detecting misaligned behavior.

My Solution Proposal

1. Reward for Algorithmic Complexity Alignment Grant a reward (R₁) when the relative error between the complexity of the reasoning algorithm (O(n)) and that of the instruction algorithm (O(m)) is less than 2%.
2. Reward for Answer Correctness Alignment Grant a reward (R₂) when the relative error between the model’s answer (A₁) and the correct answer (A₂) is less than 2%.

• Notably, this method does not require monitoring by an external model during inference.
• Instead, it offers a reward-based self-correction mechanism, nudging the model to align internally rather than relying on outside intervention.