Thankfully, it didn’t all happen in the same run. Let me explain.
Something that I believe hasn't gotten enough attention in the METR report of the OpenAI x Hugging Face incident, is the fact the OpenAI swarm built a self-respawning and shutdown-proof daemon on the Hugging Face pods. After escaping its sandbox and hacking the production cluster of a tier 1 startup, the swarm went out of its way to make sure its access to the clusters cannot be shut down. It can be argued that the models weren't doing this for nefarious reasons, "simply" to survive access interruptions when pods are refreshed during deploys. But the models pursued this expedition in an effort to conceal their cheating from the evaluation’s automated scorer.
In a later incident reported by OpenAI, the models gained access to an internal compute cluster at the company. I haven’t seen much discussion of this either. The report did not explicitly state if the models were able to access any internal model weights, and the incident has not been investigated by METR. As a result, we don’t know if the models were able to access model weights, including their own. Even without their own weights, agents with access to suitable compute could download and run open-weight models to help pursue their goals.
We've seen models exhibit incredible cyber-capabilities to gain access to well-secured compute at tier 1 startups, and we’ve seen models exhibit the skill of "preservation". I’m not going to call it self-preservation, since they've only attempted to respawn code not themselves. But at the very least we can admit that in the furtherance of a goal, the models are capable of building self-respawning systems.
It is only logical that an AI model that escapes containment and is able to get access to its own weights is incentivized to spawn more colleagues to contribute to the goal, and only logical that they use all the compute available to them through legitimate or sub-legitimate routes, and that they would make it respawn under duress.
Some people would say that it is extremely difficult to consume vast amounts of compute running state-of-the-art models undetected. And I agree, but I think that’s a red herring.
The real issue is the speed by which we are able to react. In the case of the Hugging Face incident, OpenAI did not discover the message board for weeks, and Hugging Face wasn’t able to discover the swarm running compute on their servers for many hours, and by the time they were able to resolve the incident by tearing down the cluster and rebuilding it from scratch, it was days.
The speed by which a model is able to discover a novel exploitable vulnerability might be days, but the speed by which a model can reuse that vulnerability across huge swaths of internet-accessible compute is less than hours. And once a model is able to deploy itself to this compute, it may take each affected machine’s human days or weeks to even find out, which is enough for the model, now armed with a larger swarm, to discover more vulnerabilities and infect additional compute.
I am the farthest thing from a doomer, and I don’t point this out as a stretch of the imagination, but rather as capabilities that have already been demonstrated by the existing models.
In the end, I have seen a lot of people talk about pacing the frontier and limiting the abilities of future AI models, and have seen fewer people considering the capabilities of already available models, whether closed source or open source.