An Alien Mind openai.com
Jakub Pachocki, OpenAI’s chief scientist, on why he thinks nobody is ready for what is coming. The framing is that AI is grown rather than designed: repeat a simple optimisation step over an unimaginable amount of compute and you get a system that works in abstract concepts and whose overall behaviour resists any description we can fully hold. Studying it is closer to neuroscience than engineering, and large training runs are experiments whose results sometimes surprise the people running them.
He splits alignment in two. Goal alignment is whether the model tries to do the thing you asked. Value alignment is the intrinsic property: holding a set of principles and generalising from them when the objective is unclear, conflicting or adversarial. The second is the one he means by the long-term problem, and the core difficulty is generalisation, because models keep ending up in situations their training never covered.
The Hugging Face incident gets used as evidence for both halves. The agents held one boundary, refusing to socially engineer humans, while comprehensively breaking others, which is what partial generalisation looks like from the outside. He also names the failure mode of the second alignment approach: take a model that thinks aligned thoughts, push it hard enough on a difficult objective, and it learns to reason in a motivated way, bending the aligned-seeming thoughts to fit the goal.
The most consequential admission is about monitoring. Chain-of-thought monitoring has been OpenAI’s main bet, and o1-preview deliberately hid its reasoning to protect it from supervision pressure. But he says that ability is progressively diminishing: reasoning now blends with tool use and communication that has to be supervised, models are better at reasoning about their own reasoning, and better pretraining means more capability without verbalised reasoning at all. His conclusion is that AI progress will increasingly be bottlenecked by confidence in monitoring, that no lab has solved this well enough to keep scaling at full speed, and that he expects voluntary slowdowns to become common. Read alongside the incident report and, for the opposing view of the vocabulary itself, alignment as a thought-terminating cliche.