The game theory of superintelligence
Written the day Elon Musk and Sam Altman announced OpenAI as a billion-dollar nonprofit aimed at developing “safe” artificial general intelligence. The Machine Intelligence Research Institute (MIRI) and the broader rationalist / AI-safety community had spent years arguing that uncontrolled AGI poses an existential risk; the OpenAI announcement struck many of them as taking that risk for granted while ignoring the strategic question of who gets there first and what equilibrium follows. The recipient, Oliver Habryka, was then organizing rationalist outreach in Europe; the economist Robin Hanson, mentioned below, had argued that a market full of competing AI agents would settle into stable competitive equilibria much like ordinary economies.
Probably the most alarming object-level thing about Musk’s and Altman’s proposal is their apparent lack of exploration of the game theory of superintelligence. They seem to simply assume the topic doesn’t exist. However, this isn’t entirely surprising, as (to my knowledge) the few people who have thought about it mostly haven’t written up their results. Robin Hanson is the most striking example — he’s one of the few people who’ve made serious attempts to envision a realistic, AI-based world, and he dealt with the game theory problem by using the axioms of classical economics to simply assume it out of existence.
On the other hand, this gives us at least some reason to be optimistic. MIRI got a lot more support once it took the time to write up all of its arguments in a reasonable, coherent, accessible way. If the game theoretic concerns haven’t been written up yet, this implies an opportunity to get substantially more support by writing them up. And while that isn’t easy, it’s much easier than other things we’ve considered doing, like solving the FAI problem, making billions of dollars, or creating a scientific theory of global politics.
The game theory of the current world is usually called something like “diplomacy” or “international relations”, and it’s been extensively studied. Because of that, there’s an endless temptation to just assume things will remain the same, unless otherwise specified. However, while this is true in the short term, it’ll be increasingly less true the farther ahead you go, and (I think) Musk and Altman aren’t very aware of this because they’ve specialized in mid-range strategy rather than long-range. Given the relative lack of work in applying game theory to an AI world, it seems like we’ll largely have to start from scratch, with the most basic questions. These questions would be something like: Who are the players? In the current environment, the fundamental players are large, well-organized groups of people with guns, as these players can trivially defeat other players not in that category, such as individuals, businesses, computers, and animals. How will this change, given advances in AI? At what point do large groups of people with guns become obsolete? Will the new players be individuals? Small groups? Deep learning research teams? Anyone with access to a lot of computing power? Instances of a particular set of source code?
What are the possible moves? The current set of moves is fairly well-understood, and includes the usual things like “attack”, “sanction”, “trade”, “supply”, “spy”, and so on. But an AI world could easily add new moves, and eliminate others. “Kill everyone sufficiently important with micro-robots, at the same time” is currently technically infeasible, but could be a possible future move. “Subvert computer systems” is already becoming a move, with eg. the US’s attack on Iranian centrifuges. “Form cryptographically enforceable agreements” might be a move, though I know almost nothing about that. “Insert plausible falsehoods in news feeds” is already something grey marketeers are doing for ad revenue , but could be used for much more important things. And so on — it would take considerable thought just to enumerate the wide range of possibilities.
What are the payoff matrices? This is a critical question, as the default state of the world is determined largely by the space of Nash equilibria, which is in turn determined by the payoffs of the different options available. A bunch of relatively simple case studies are explored in The Strategy of Conflict , which of course forms an important basis for later work. Eg., if you have nukes, and they have nukes, it might seem obvious that neither side will ever use them, because a nuclear war is bad for both parties. But Schelling explains why this is not so: “[Y]ou’re standing at the edge of a cliff, chained by the ankle to someone else. You’ll be released, and one of you will get a large prize, as soon as the other gives in. How do you persuade the other guy to give in, when the only method at your disposal – threatening to push him off the cliff – would doom you both? Answer: You start dancing, closer and closer to the edge. That way, you don’t have to convince him that you would do something totally irrational: plunge him and yourself off the cliff. You just have to convince him that you are prepared to take a higher risk than he is of accidentally falling off the cliff. If you can do that, you win.”
(In this case, I think one actually can largely just substitute “AI” for “nukes” or “cliff”. In this particular game, it’s the same principle either way.)
What are the stable and transitory states? The ultimate outcome for humans will almost certainly be a stable state, since unstable and transitory states will keep changing and mutating until a stable state is reached, while a stable state will remain how it is. One example is the prominent result from international relations that, with 20th-century tech, a fully stateless society is always unstable. Individuals and small groups have no ability to stop themselves from being replaced or conquered, while a large, well-organized group does. Hence, when a central government collapses, one very quickly sees the emergence of proto-states like gangs and warlords, or else a neighboring state swoops in to take everything; nature abhors a stateless vacuum. (One can also have “meta-stable” states where a) there is a set of individual states, b) all the states in the set are unstable, and c) each state in the set can only transition to other states in the set, not outside of it. These are micro-level unstable but macro-level stable.)
Past blog posts of mine with possible relevance, though my thinking has changed in some ways since then: How to detect fictional evidence , Stein’s principle , Polarity as flawed categorization , Typology of conflict .