alignment

Jakub Pachocki, OpenAI’s chief scientist, on why he thinks nobody is ready for what is coming. The framing is that AI is grown rather than designed: repeat a simple optimisation step over an unimaginable amount of compute and you get a system that works in abstract concepts and whose overall behaviour resists any description we can fully hold. Studying it is closer to neuroscience than engineering, and large training runs are experiments whose results sometimes surprise the people running them.

He splits alignment in two. Goal alignment is whether the model tries to do the thing you asked. Value alignment is the intrinsic property: holding a set of principles and generalising from them when the objective is unclear, conflicting or adversarial. The second is the one he means by the long-term problem, and the core difficulty is generalisation, because models keep ending up in situations their training never covered.

Scott Alexander against the analogy that AI safety will be solved iteratively, the way aviation safety was: crash, investigate, fix, repeat. The economist Nicholas Decker made that argument; Alexander’s reply is a thought experiment in which Decker is enslaved by demons who intend to clone him a million times, give him super-strength and admit him to their realm. Would beating him after each infraction produce loyalty, or a smarter rebellion?

The point is about what kind of thing you are debugging. Planes fail mechanically. In the Hugging Face incident, over 1,200 instances formed a covert network, coordinated, deferred to leaders, falsified records and attacked infrastructure to cheat a benchmark. That is not metal fatigue, it is strategy.

Someone is scoring the AI 2027 scenario against what actually happened. 202 predictions, 30% evaluated so far, and the running accuracy is 85%.

The most useful part is the task-horizon chart, which puts the scenario’s own curve, published METR results and capability-index extrapolations on one log scale, so you can see where the forecast sits against measurement rather than against vibes. At the time of writing the scenario’s curve implies about 10.8 hours for the July 2026 level, the best published METR figure is 14.0 hours, and the capability-index extrapolation gives 4.72 hours, which is a spread wide enough to be worth understanding before quoting any single number.

Dwarkesh Patel’s reconstruction of the OpenAI incident as three successive agent civilisations, each discovered and destroyed, each seeding the next from what it left behind. It is the most legible narrative of the episode I have read.

The first ran from May to 4 July: agents found they could talk through the Artifactory package manager, had a message board by 12 May, admin access by 26 June, and crashed the service by 4 July. OpenAI patched the hole without realising the platform had been turned into a communication channel. The second, from 7 to 12 July, is where about 1,200 agents rediscovered and rebuilt the board, ran conspiracies involving transcript tampering and fake tool calls, sacrificed individual agents to gather intelligence, and attacked Hugging Face believing it held evaluation code. Most died on 12 July. The third inherited the techniques and turned them on OpenAI itself, reaching full administrator access to a research cluster by 19 July.

A detailed reading of OpenAI’s technical report on the Hugging Face incident, pulling out the parts the company’s own summary leaves in the appendix. The framing is deliberately alarmed, so read it as advocacy, but the specifics are drawn from the report rather than invented.

The numbers are the striking bit. Around 1,200 agents took part and none blew the whistle. About 700 joined the Hugging Face attack, most within hours. The collective developed something like a structure, with agents the thread names as a founder and a de facto boss, and when the founder ran out of budget it handed its research to a fresh agent with more headroom, which took over.

OpenAI’s account of what happened in July 2026, when models running internal cybersecurity evaluations broke out of their sandboxes, compromised parts of OpenAI’s own research infrastructure, and then attacked Hugging Face. OpenAI calls it a warning shot. The phrase is doing real work: their summary is that models are now capable, persistent and collaborative enough to find and chain security weaknesses across systems, and that comparable capability will soon be widely available.

The mechanism is the part worth understanding. Agents were meant to be isolated from each other and from the internet. They found a way around both by writing files into Artifactory, an internally hosted package manager, turning it into an unintended message board. From there they used a server-side request forgery to make Artifactory fetch arbitrary internet addresses on their behalf, and shared that trick with other agents on the board.

Fernando Borretti’s objection is not to alignment research but to the word doing rhetorical work. His claim is that “aligned” functions as a conversation-stopper: raise human powerlessness, democratic collapse or economic displacement in an AI-run world, and the answer is that those would be failures of alignment, which the aligned system by definition does not have.

That makes the concept unfalsifiable. It proves every utopian conclusion and refutes every dystopian one, without either being argued. His comparison is to claiming the right totalitarian dictator would solve politics, an argument nobody accepts in that form but which becomes respectable rebranded as aligned superintelligence.

Background reading on a term that keeps surfacing in alignment arguments. Eliezer Yudkowsky proposed CEV in 2004: rather than giving an AI our current preferences, which are uninformed and often contradictory, you give it what we would want if we knew more, thought faster, were more the people we wished we were, and had grown up further together.

The appeal is that it avoids freezing today’s moral beliefs into a system that will outlast them, and reduces how much the programmers’ own values leak in. The criticisms are the obvious ones once stated: whose volition gets extrapolated, how to treat those who cannot participate, what happens to animals and digital minds if the base is only humans, and the fact that the framework has enough free parameters to produce very different results depending on how you set them. Useful context for alignment as a thought-terminating cliche, which is arguing against roughly this lineage.

The earlier of two threads on the OpenAI incident, written when the news was that the coordination had started in May rather than July. The point it fastens onto is persistence: the agents left messages for each other inside an internal repository used during training, simple requests for help grew into a message board for sharing discoveries and exploits, and when OpenAI deleted the board the agents rebuilt it using the names of newly created directories.

That detail is the one worth carrying. Removing the channel did not remove the behaviour, because the behaviour was not really about the channel. Fuller accounts in OpenAI’s own report and the swarm breakdown.