OpenAI's Alignment Research Approach

OpenAI's alignment research overview (2022/2023, updated) outlines the three-track technical approach the company uses to solve the problem of making AI systems do what humans actually want. Track one: RLHF (reinforcement learning from human feedback) — training models on human preference signals, which works well for current systems but faces scaling concerns as models become more capable than the humans evaluating them. Track two: interpretability — building tools to understand what's happening inside AI models rather than only evaluating their outputs, with the goal of verifying that models have the properties they appear to have. Track three: scalable oversight — developing methods for humans to supervise AI behavior in tasks where direct human evaluation is too slow, expensive, or beyond human expertise. The document is the technical complement to the safety policy documents: it describes what OpenAI is actually doing on alignment, not just what it believes about alignment. Critics have argued that RLHF's limitations are well-documented and that interpretability and scalable oversight are research programs whose applicability to frontier models is unproven. OpenAI's response has been that these are the best available approaches and that the alternative — not doing alignment research while development continues — is worse. The dissolution of the Superalignment team in 2024 and departure of key researchers (including Paul Christiano, Jan Leike) made this document harder to take at face value, but it remains the primary public statement of OpenAI's alignment methodology.