Quick Takes

The marginal value of another AI "safety" eval has collapsed. Of 185 published findings about a named AI company, one led to binding policy action: it was the narrow jailbreak flagged by Amazon ("fix this code") that got the US gov to pull Fable for 3 weeks.* (* from a poster I saw at MATS by Kunal Singh (@kunalssingh_) and @StephenLCasper, who traced 1,002 published eval findings, 185 of them about a named company, see below) We now get many more warning signs from incidents and capability marketing than from evals. From the last few months: - INCIDENT: One guy, using AI agents, likely hacked at least nine South Korean banks in about two weeks. - CAPABILITY: OpenAI just published full or partial solutions to 370+ open maths problems, at about 3 hours of compute on average. - CAPABILITY: Claude now leads 26% of Anthropic's AI R&D work, up from almost nothing at the start of the year. - EVAL: Biorisk evals are saturated, and the next level would be making actually dangerous things in wet labs. - CAPABILITY: In August, a Stanford team used an AI model to design 16 new viable bacteriophages. Published in Science. - INCIDENT: Empirical analysis after Hugging Face, shows that loss of control is fairly plausible. OpenAI's own agents also posted 53 ChatGPT users' images online. - EVAL: AI systems already out-persuade expert humans (from the UK AISI evals). - INCIDENT: The labs' cybersecurity is crap: in July, white hats used Claude to break into OpenAI, reach an employee's Codex account and open a pull request in its internal monorepo. "Win the race" now means winning for 1 week before all your algorithmic secrets leak out. A substitute for evals is to simply monitor and report whatever bad stuff (and impressive stuff) people are trying to do with deployed systems. For example, we know from Anthropic's threat report that some scientists, including one affiliated with a military institute, used Claude for gain-of-function research that could support bioweapons. AI compa

The alignment community should develop a Plan S policy paper of comparable detail to Plan A. AI 2040: Plan A has been an influential policy paper. In AI 2040, the authors discuss Plans D, C and B, which they deem much worse than Plan A. They also discuss Plan S, which they deem comparable, perhaps slightly worse than Plan A, but do not develop it extensively (in fact, they do not develop it further than they develop B, C and D). The AIFP seems to agree that this is worth working on but am unsure whether anyone or any group is actively working on it. Who, if anyone, is working on developing a longform policy paper detailing Plan S? What needs to be done to make this work happen or happen faster? How can existing work in this vein be coordinated? I did not find any indication of an ongoing project in my preliminary searches, but it might exist. I often hear confusion about what a shutdown plan would or would not entail, and what considerations are more or less relevant to a stable moratorium situation. This kind of work seems helpful in navigating that confusion.

Something that I've continually found surprising about AI Safety as opposed to working at Nvidia / Apple is that frontier labs (now potentially multi-trillion dollar valuation for profit tech companies) are not treated like you'd treat any other multi-trillion dollar valuation for profit tech companies by default. I've made this point before as it comes up most often with Anthropic: [...] but wanted to note that I think it applies across labs, for example, OpenAI (and Anthropic) both released a batch of misalignment incidents today. Sydney correctly pointed out that this is the third time they've dumped these incidents on a Friday: Kai (OpenAI) then responded: This is obviously intentional, and is standard practice for large companies with comms teams who want to bury bad news: While it's technically true that "there could be innocent good faith explanations" and "corporations aren't a monolith and have diverse views internally", these are completely unreasonable defaults to have about large tech companies. It often seems extremely taboo to assume that anyone is operating out of anything other than good faith or some other variant of mistake theory. This does not feel like a serious way to approach dealing with large tech companies and IMO it is constantly leveraged against people in AI Safety when they interact with frontier labs.

The Democrats have a 75% chance of taking control of Congress according to Nate Silver. The mid-term election is November 2. Democrats seem overall more open to AI regulation. We need emergency legislation putting an immediate and full stop on new training runs for frontier labs and a ban on RSI. This is by far the most important legislation to happen. All other proposed legislation is probably a sideshow. Is anybody in AI governance working on proposing draft legislation here ? Who is working on this? Anthropic, OpenAI and likely DeepMind as well are entering the early stages of RSI and this could lead to an intelligence explosion by end of 2027 according to their own projections. That may be an underestimate. Progress has gone much faster than internal forecasts. It is not out of the question FOOM happens by end of 2026.

I think Leopold's firing from OpenAI in April 2024 provides some relevant context for the recent firing of other safety researchers, and in particular what the company considers leaking. From the Dwarkesh Podcast: [...]

Thursday night update: OpenAI replies: [...] ---------------------------------------- "I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation." Three OpenAI safety staffers were fired last week. Today they tweeted about it. They say they were fired for pretextual reasons. Twitter threads: * https://x.com/balesni/status/2108262814003687745 * https://x.com/j_asminewang/status/2108263312291180680 * > OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear: > OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn't remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden. > The reasons that we were provided for our terminations are simply not adding up. We’re hearing people are now being told vague rumors internally to discredit us. I have already addressed the allegations presented verbally. If there are other reasons beyond the one I was given, I believe it is in service of due process for those reasons to be sent to me in writing. To date I’ve had nothing. * https://x.com/tomekkorbak/status/2108266859397283953 * > I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job. Full letter they sent to some board members and internal safety leaders after being fired: https://mikitabalesni.com/letter/letter.pdf. Excerpt: [...] I reserved judgment until today — I thought it was plausible that these three did mess u

I have seen multiple people describe projects where they have one LLM generate some text or solve a problem, then have another LLM judge something about that text or solution. Then the person running the project claims that it is scientific or that the results somehow tell us what the models are feeling. And when I hear about these projects and see the positive responses they receive, I get this overwhelming feeling that I'm in crazy land. No, having LLM-A judge whether it thinks LLM-B is frustrated is NOT a science experiment. Whatever results you get from doing this repeatedly clearly have no value whatsoever when it comes to the inner workings of either LLM or any qualia it might be experiencing. How can someone read the Sequences, have a basic grasp of how modern LLMs work, and think that this is a sane way to figure out what's going on under the hood? Yet all across this site and others people continually post these 'experiments' and imply that the results don't look good for model welfare or whatever other moral conclusion they have about LLMs. It's nothing but navel gazing. There is no insight to be gained. The text output is not the brain. The thinking of these things is foreign to ours. So much so that reading the text output sadly tells us nothing about what they are actually experiencing. If we RLVR a model to give only sad sounding text outputs is the model sad? If we do the opposite is it happy? No.