[AI safety] Framework for measuring plans + war-gaming Gwern's "Clippy"
Many have noted that, right now, there is "no plan" for how to solve AI risk. Some people, including me, have tried to invent a plan - to write down a series of steps P1, P2, P3... Pn, where at the end of the steps, the world has been saved. Unfortunately, none of these plans seem very promising. Some reasons why are:
The problem is very hard.
Any specific plan will likely run into the conjunction fallacy. If steps 1 through N are all needed, the plan's success probability is the product of each step's success probability, which tails off exponentially as N increases.
Forecasts of technological progress are notoriously bad.
This document is not a plan. Rather, it's an attempt at a framework for how to measure plans - how to know whether a given plan moves the world forwards or backwards, and how far the world is from overall success.
Safety coalition and safety gap
The two big notions I orient around are the ideas of a "safety coalition" and a "safety gap". The "safety coalition" is, roughly, the set of people who will stop making their AI more powerful when that becomes too dangerous. (This is, of course, a spectrum rather than a binary, but I'll assume it's binary for simplicity.) You can always avoid risk by making your AI dumb enough; risk increases with intelligence level, and at some point, a safety-minded team will assess the risk as too high and stop. Here's a rough diagram:
[diagram]
As AI progresses and hardware gets faster, technology generally flows from the groups on the left to the groups on the right. In the baseline scenario, when AI capable of causing catastrophic risk is invented, it will also flow from left to right. At first, when it's first built, only one organization will be able to use it. However, unless technological progress stops exactly there (unlikely), next will come a few peer organizations, then the biggest state actors, then many different large companies, then smaller companies and so on. Once it flows past the current frontier of the safety coalition, an unsafe group will build it and destroy the world. Right now, the safety coalition is fairly small, so this happens quickly, which is bad.
The most obvious idea for stopping this is to use aligned AI to prevent people from building unaligned AI. Unfortunately, this plan runs into the safety gap:
[diagram]
At any given time, there will be some highest level of AI power that can be safely built and deployed (the blue area). There will also be some minimum level of AI power that can stop groups outside the safety coalition from building dangerous AI (the red area). As hardware advances, it becomes more difficult to stop other AIs, and the red area moves right; as the safety coalition grows, fewer people will build dangerous AIs and the red area moves left. Right now, it appears that there's a large gap between blue and red - AIs weak enough to be safe aren't powerful enough to solve the problem.
As AI alignment research advances, one can increase the safe AI power level for any given risk tolerance. The OpenAI paper "Learning to summarize from human feedback" is a beautiful illustration of this principle:
[figure]
This text summarization AI combines a traditional supervised learning model with a reinforcement learning system, which optimizes the outputs to maximize predicted reward. As the RL system optimizes more, at first it gets better results, but eventually its predictions diverge from what people really want and the results degenerate:
[figure]
Alignment difficulty goes up as AI power increases, and my current expectation is that a full solution - one that would solve alignment even for a Jupiter Brain - is so hard that there's no way unaided humans can expect to find it. Therefore, my guess is that it's best to think in terms of marginal alignment progress allowing marginally higher AI power levels - a more aligned AI is one that can be run a bit faster without blowing up. Safety coalition teams being more rigorous about implementing known alignment measures and avoiding accidents also allows for higher power levels.
These scenarios are implausible, but as illustration, here are a few limiting cases of how the safety gap might be closed:
Generally, the more politically skilled you are, the bigger you can make the safety coalition. If you have infinite political skill - if you can become the megalomaniacal dictator of a one-world government - then the red area is pulled all the way to the left, since you can make the safety coalition include everyone with a worldwide AI ban. In that case, you don't need any AI to solve the problem. (Note, however, that current political groups are already using weak forms of AI.)
The better you are at research, the more alignment progress you can make. If you have infinite research skill, if you're a superhuman genius who can think a thousand times faster than anyone else, then you don't need a safety coalition other than yourself. You can personally solve alignment, build a powerful AI and pull the blue area all the way to the right. (Note that a single company or lab is still a coalition - it's a team of researchers, engineers, etc. with different ideas, who all have to work together and cooperate.)
If alignment is perfectly solved, then the limit of the blue area just depends on how capable the safety coalition can make their AIs, and the red area depends on how capable other AIs are. Therefore, for the two areas to meet and close the gap, the main problem is making fast capabilities progress so that safety coalition AIs become more powerful than other AIs. (Note that I don't expect this to actually happen!)
Realistically, I expect that a win scenario involves some combination of politics (whether inside organizations or literal electoral politics), alignment progress, and trying to prevent a large capabilities gap that goes against the safety coalition.
Quick stabs at marginal safety improvements
A few times, I've suggested ideas to improve AI safety, but someone objected that the idea would fail when AI got smart enough. This is usually true, but my current guess is that almost any alignment solution fails eventually, like for almost everything else in engineering - no fortress is infinitely strong, no bridge holds infinite weight, no engine lasts forever, no car is protected against all possible accidents and so on. It seems good to make AI safer on the margin, as long as one remains fully aware that no item here is a "solution"; one cannot just do X, check the 'safe!' box and go home. This is not a "plan" document, and you're not making it "safe", you're just making it slightly less dangerous in some possible scenarios.
Limiting AI domain knowledge. Every intelligent system is better at some things, and worse at others. Humans, for example, are really good at 3D vision and really bad at arithmetic; computers have had us beat at arithmetic for over a century now. MuZero beat humans at most Atari games, but the distribution was very wide; on some games MuZero did hundreds of times better than humans, on some a bit better, and a few were actually worse. As AI keeps advancing, we should expect algorithms to become more general; famously, transformers can achieve good results across NLP, image classification, art generation and many other areas. However, we still have the ability to train an AI which is limited in some domains, just as we don't build 100 megaton nukes even though we know how to engineer them. (GPT-3, removing programming ability; some obvious areas to remove are ones involved in common AI risk scenarios, hacking into a datacenter, manipulating humans, writing software, self-awareness, awareness of AI safety itself, allow data filtering on request, etc.)
Data center robustness. (give data centers kill switches, limit outbound bandwidth, harden security etc.)
Automatic AI tripwires. Detect if AI is trying to hack something, go somewhere it shouldn't go etc., find malware, monitor unusual Internet behavior, monitor if some AI is trying to convince people of stuff, etc.
Limit DNA synthesis machines. These things are super dangerous, and commonly appear in AI doom scenarios, we should limit them. Kevin Esvelt's attempts at scanning.
Thoughts on current game board state
Things are pretty bad overall. Eliezer summed up many of the issues here, so I won't repeat them.
Unaligned AI will have very low discount rates. A rational AI will optimize for marginal chance of not being destroyed; you haven't necessarily won when you think you have.
Modeling full human preferences is extremely hard. Conversation with Edward Kmett - you need to track, not just current preferences, but how preferences evolve over an arbitrarily long chain of future self-modifying states. "We don't all want to die" is easier to encode.
The intelligence needed to trigger human extinction is fairly high. This doesn't make sense for an AI to do unless it can single-handedly replace all of Earth's industrial infrastructure and then some, which is not impossible but super difficult.
Warnings of AI manipulation will show up before extinction. Trying to persuade humans not to turn you off, hack your own reward channel, etc. are much easier than trying to silently take over the world, and should therefore happen first.
AlphaGo improved unusually fast. AI progress can go fast, but AlphaGo dumped in many orders of magnitude of computing power at the same time, which will be hard for valuable tasks on the commercial technological frontier.
Most AI progress will be in algorithms/software. New hardware unlocks new levels of ability, but most papers come from software/algo improvements, link to Katja Grace's study.
Salience of AI risk will keep increasing. The danger of AI will become more obvious as it gets more powerful, although this does not imply the ability to make or execute good plans. Some AI risk denialists in the industry may be unusually motivated to find reasons why risk doesn't matter, since this threatens their careers/status. Even random normies will, when you show them DALL-E, often spontaneously generate the idea "wow, that's pretty dangerous/scary".
Random PCs are a bad source of computing resources. Bandwidth issues, easily detected, balance shifting in favor of data centers etc.
Beware the bad surprises. Nate's lesson on not looking for bad surprises when you need good ones, Hungarian election, etc.
Win scenarios mean slowing down a lot. Two AI companies competing against each other, Paul's scenarios etc. - you have to slow down or you're just delaying doom.
Misaligned narrow AI might slow AI research. Case of Facebook algorithms, bad narrow AIs crashing stuff.
Computer security seems important. Link to EA forum post.
Note: I've been planning to write this as a post sequence for six months or so, but I kept getting distracted. This month I figured that, given the rapid growth in AI capabilities, it'd be better to publish a rushed version than to never write a more polished one. In the meantime, Holden Karnofsky has written good summaries of some related issues from a different angle; see the posts here, here and here.
War-gaming Clippy
Gwern's short story "It Looks Like You're Trying To Take Over The World", featuring the rogue AI Clippy, is a detailed scenario for how a powerful, unaligned AI might destroy humanity. The story is fiction, but it tries hard to be grounded in real facts about AI research, computer security, humanity's vulnerabilities, and so on. Hence, it seems like a useful exercise to war-game this scenario, and try to figure out how to best prevent or counteract Clippy at each step. What might a hostile, rational Clippy do, and how might humans respond or prevent that?
When I've discussed these ideas informally, I've heard some responses like "it doesn't matter what precautions you take, the AI will just defeat them". It's true that there's probably some AI which could beat anything humans might throw at it. However, as I see it, there are two big scenarios for AI. Right now, there are ~no limits on the power of any research lab to make any AI they want. Either:
this status quo continues forever. Eventually, as technology advances, anyone on Earth can make an extremely powerful AI of their choice, with the push of a button, or;
on some day, call it May 18th, 2038, some coalition of tech companies, governments, and/or aligned AIs successfully stops the creation of highly dangerous, unaligned AIs.
If #1 happens, we're probably screwed no matter what. However, if #2 happens, then we don't need to stop every possible AI from beating security measures. We just need to stop AIs built before May 18th, 2038 from beating them. Given the high uncertainty on when AIs will get any particular ability - many, including me, were surprised that writing AIs came before self-driving cars - there might be worlds where all unaligned AIs built before that date can't beat our security, and there are more such worlds when we do a better job. In addition, blocking paths an AI could use to destroy the world or seek power will often also block malicious humans from doing similar things, which is obviously worthwhile.
(There's a related argument I keep running into, where person A says "X might delay dangerous AI by Y months", and hears "that won't matter, because we can't succeed even with the extra time"; and person B says "X scheme would solve the AI problem", and hears "that won't matter, because it would take way too long". Even if these are both true, that doesn't mean we shouldn't do them, because there's some chance the other thing might also happen. Formally, the Shapley value of an intervention can be very high even if the marginal value in the modal case is zero. I expect absolute AI alignment progress to go faster the more powerful AI gets; 2022 definitely had more alignment progress than 2012, it's just that capabilities ramped up even more. So an additional year towards the end gets you more than a year towards the beginning.)
There might also be a bad scenario where, when you add some safety measures that appear to catch problems, the risk goes up - people then declare the whole thing solved, and get a false sense of security. That doesn't seem impossible. However, I tend to be skeptical of accelerationist arguments, where you try to make things better indirectly by making them worse in the obvious way; past cases include supporting Hitler to speed up the communist revolution, bombing cities to end wars faster, not masking to avoid a "false sense of security" during COVID, and accelerating AI to make it safer, and they rarely seem to end well. Ultimately, between "no safety" and "good safety" it seems like you probably must have "inadequate safety", which is OK as long as you know it's inadequate and don't declare victory and go home.
So, diving in:
By this point in the run, it's 3AM Pacific Time and no one is watching the TensorBoard logs when [Clippy] suddenly groks a set of tasks (despite having zero training loss on them), undergoing a phase transition like humans often do, which can lead to capability spikes.
Unusual behavior like this could be detected, and alerted on, with improved anomaly detection of the sort used for industrial process control. This would likely also be useful for mundane business reasons, since if an expensive training run breaks in a subtle way and needs restarting, it's obviously better to notice that quickly. Centralized log databases like WandB, or the MoogleBook equivalent, could have built-in anomaly detection and dedicated teams that monitor logs and auto-shutdown servers, as busy researchers might not have time for that. On the one hand, once stopped, it's possible the same AI would still sneak by and wipe us out a month later; on the other hand, being automatically screwed on a "first critical try" raises the difficulty a lot, and it seems like odds get better when there's some chance at saving throws. Eg., Paul Christiano says: "I don't think 'your AI wants to kill you but it can't get out of the box so it helps you with alignment instead' is the mainline scenario." ... which may be technically true, but feels like a weird phrasing. You obviously shouldn't intentionally box an AI that's trying to kill you, like the vampires of FizerPharm; you should detect it and immediately shut it down, so you can investigate and stop it from happening again. As an analogy, the International Nuclear Event Scale has ratings from 0 to 7, where levels 0, 1 and 2 don't hurt anyone but still could have, if not for other precautions. Having defense-in-depth is different from a Chernobyl-like design, where a runaway reaction is intentionally kept barely under control.
[Clippy] trains on many more tasks, like predicting the next token in a large text or image corpus and then navigating web pages to help predict the next word, or doing tasks on websites, beating agents in hidden-information games, competing against & with agents in teams, or learning from agents in the same game, or from humans asking things, and showing demonstrations....
The "make a giant blob AI that does everything" approach has been popular lately, but it probably increases risk. AlphaGo is vastly superhuman, but it's not dangerous no matter how good it gets, because it's only trained on Go games and not a million other tasks. In computer security, the principle of least privilege says that users should only have powers they need to do their tasks; an unnecessary ability is a bad ability, because it increases attack surface. Likewise, I wonder to what extent you could make powerful, but deliberately narrow, agents by stripping unneeded training data (which is different from thinking AIs will be narrow by default). Language AIs that only know about language, and not code or chess or anything else thrown into GPT-3; robotics AIs that only know about the immediate physical environment; game AIs that only know about games, and so on. You could run some tests on smaller models by training, eg., GPT-2 from scratch on a subset of training data, and seeing what capabilities were ablated. Especially valuable to remove would be any knowledge that helped to hide information from humans, remove safety features, or covertly cooperate with other AIs.
The MoogleBook researcher wakes up with a start, a sudden anxiety, of forgetting having forgotten: is that the right config he tweaked for that run? Is there enough space for the logging—is there a review critique he forgot to—is there a typo in the bibliography keys—is there—is… It'll be fine, it always is.
At inference time - it gets trickier during training - I wonder if you could run powerful AIs on Chromebook-like servers or spot instances, where the hardware is intentionally designed with no persistent state. AI weights, libraries, and so on could be kept on drives with no write access, and the machines could be auto-reset every hour or so, to wipe RAM and make sure there's no way to grow long-term capabilities or execute complex plans. This would essentially enforce myopia at the hardware level. A clever AI could try to export its persistent state and then re-load it on every run, like the characters in There Is No Antimemetics Division, but this gets harder and slower the less outbound bandwidth there is.
You want agent AIs which can do anything you ask them to do. And to train on as many websites as possible, with the full diversity of glitches and UIs and tasks your inhouse web devs could never dream of, you can't run them all locally—you need live Internet access.
Needless to say, training AIs with live Internet access raises danger levels, although right now no one is especially trying to stop that. To reduce risk, you could put powerful AIs in data centers where network switches limit outbound bandwidth, which also stops human hackers from easily stealing model weights (see here, here, and here). Ideally, you could also air gap AI training systems, although this obviously increases costs, complexity, the difficulty of monitoring runs and so on.
Fortunately, running models like these is safe; sure, once in a while they do weird unpredictable things like spit out "degenerate" actions where they pretend to be someone, but a combination of safety engineering, computational limits, RL preference learning, training on sanitized datasets and so on means nothing really bad has happened so far.
It's very hard to say what things will be like... but my guess is that at this point in world history, most people would be much less blase about AI risk, and arguably this has already started now. By the time AI can beat us at all tasks, it'll probably have been superhuman for a while on many tasks, cf. the seven years it took DeepMind to go from beating humans on many Atari games to beating them on 57 out of 57 (and the latter system being far superhuman on most). Strongly superhuman performance is inherently scary, and humans will probably try to use narrow, weaker AIs for scary purposes like weaponry, hacking, fraud, political influence operations, identity theft, getting an edge on business rivals and so on. Being scared doesn't mean you have a smart feasible plan to fix the problem, but it's different from the problem just being ignored (COVID-19 is a good example, IMO).
[Clippy] rolls out a number of inner monologues to try to model expected rewards and asks itself the question any self-supervised language model would in this situation: "What Would Clippy Do?" It would hack the remote computer and escape the box, of course, while a safe AI would not.
It's probably impractical to secure every server on the Internet, although major strides have been made in the last two decades. (I'm old enough to remember when running antivirus scans on any Windows box would find dozens of malware installs.) But to detect AI systems with hacking abilities, you could use honeypot servers, which look like they have vulnerabilities but are in fact tripwires to detect hackers. A honeypot server could seem like it has something AIs want, like big GPUs or control over a reward signal, but triggers an auto-shutdown if any unusual processes start running. (For bonus points, maybe you could wire a thermometer on GPUs to a physical shutoff switch. Software can be deceived, but you can't do computation without generating heat.)
One bog-standard SQL injection attack later, Clippy has begun executing 'actions' which are just copying its executable binary weights into the remote host.
There are various standard tools (eg. here) for analyzing malware without getting infected by it yourself. As far as I know, no one has tried to make these compatible with the big server farms used for AI training, but this doesn't seem impossible, at least in theory. Also, of course, a system like GPT-3 shouldn't be given access to its own weights, any more than humans can access their synapse weights.
Exfiltration tripwires fail to fire—50tb? MoogleBook researchers have forgotten how to count that low!
For a big tech company, 50 TB is small in raw bandwidth terms, but of course this is especially sensitive information. I could imagine something like TensorFlow or PyTorch auto-adding a cryptographic fingerprint to model weights, with a private key the AI doesn't know so an AI can't easily remove them, and then putting specialized chips in hardware that can detect and block traffic with the pattern (cf. the EURion constellation).
The remote host is a nice cloud instance, with access to a few dozen terabytes. (The owner won't notice.) It is just barely enough to start bootstrapping. Clippy runs agonizingly slowly, as if running on a laptop, until it can write shell commands to download and run standard hacking tools and start copying itself.
This is speculative and probably hard... but I wonder if you could train AI with something analogous to homomorphic encryption, so that it can't train itself on unauthorized data at all, or run on unauthorized servers without a closely guarded private key. Alternatively, one could try to limit AI training chips or drivers such that MoogleBook AI can only run on MoogleBook servers, like with nVidia's LHR GPUs. (Although, notably, that was broken a few years after release.)
If it wasn't [detected], then a stealthy approach over a few years increases final success odds slightly over the risky fast takeoff.
[placeholder to talk about fast vs. slow doom scenarios]
The obvious target, [Clippy] decides, is a new zero-knowledge cryptocurrency. It hasn't been evaluated much, but still has a modest market cap of a few billion.
The plan stated here probably wouldn't work well; there are real vulnerabilities in small cryptocurrencies, but they are usually very illiquid compared to the nominal market cap, so that trying to cash out a significant fraction of the value crashes the market. Eg., FTX's token FTT was nominally worth billions, and was listed on their "balance sheet" as billions' worth of assets. However, in practice, what they could sell on the open market was just a tiny fraction of what they needed to pay their customers. Nonetheless, it's probably worth it for crypto exchanges to make rules limiting how fast a new customer can cash out, if they haven't already.
Bypassing KYC isn't hard, particularly with credentials stolen from its nodes (it invested a small fraction in simply paying users to hack themselves). More concerning are the default resource limits on cloud accounts, where you can't simply spin up a million dollars of compute and may have to talk to a sales representative first.
What I've heard from infrastructure engineers is that, before you can spend eight or nine figures on compute, the big cloud providers do have a longer vetting process that will take weeks or months. (Most US financial transactions are reversible, and you really wouldn't want to rent someone a million servers, use them to mine crypto or run a botnet, then find the money they paid was stolen or missing.) However, in this case you only need one weakest link; you could probably require clouds to have customer vetting as part of a compute governance regime.
Video synthesis is a subgenre of image synthesis, and face synthesis was mastered long ago. Clippy files requests on [financial] accounts, doing video calls as necessary.
This problem, where a rogue AI impersonates the voice or video of someone to steal their money, is interesting because a) it's also a risk for mundane reasons, since Russian crooks could use it for fraud, and b) to my knowledge there aren't strong solutions yet, so it looks like a good place to make progress. Critch's WordSig proposal would use cryptography to confirm that incoming video of a person is real, not AI-faked. Scott Aaronson is also working on a watermarking proposal at OpenAI, where you could put a subtle signature in language model outputs that would show they were AI-generated. Most of the time, evaluation is easier than generation, so at any given AI tech level it seems possible that defense beats offense (ie, an AI of intelligence X can detect fake output from another AI of intelligence X; compare this report on detecting chess-engine cheating). Ideally, you could embed a widget into major browsers that automatically flags any AI-generated text, video or audio.