Arguments for and against, takeoff speeds, and discontinuous progress
A reply to AI alignment researcher Paul Christiano (then a graduate student, later head of OpenAI’s alignment team and founder of the Alignment Research Center). Christiano had circulated a long email organized as “arguments for” and “arguments against” a fast AI takeoff — the question of whether, once human-level artificial intelligence is achieved, capabilities will explode rapidly or improve gradually. Alyssa first pushes back on the for/against framing itself, then works through several specific sub-claims: how to model takeoff speed probabilistically, why “price” is a misleading lens for thinking about AI economies, and where past technologies show smooth engineering progress producing huge jumps in social and economic impact.
Arguments for / arguments against is a bad way of framing things generally. It gets used a lot in blue/green politics issues where people just want any argument that favors their side, but almost any interesting scientific question will be complex enough that there won’t be a mutually exclusive dichotomy ( Privileging the Hypothesis ), and I’m pretty sure (though I sadly can’t find a convenient reference) that most scientific disputes were settled by inventing a new theory instead of one side of a dichotomy overturning the other. Even for something like heliocentrism vs. geocentrism, Copernicus’s theory still had perfectly circular orbits and epicycles, and the debate couldn’t really be settled until Kepler came up with a new, much more accurate theory that discarded those assumptions.
This actually seems pretty similar to a problem I identified in my brain a few years ago. If, say, I’m on a plane (to use one real example), and the pilot announces that there’s been a delay, this will probably be annoying, and so my brain doesn’t want to look directly at the annoyingness. Instead, what it does is create two theories: one where the problem is resolved instantly and we take off in five minutes, and another where everything is a terrible disaster and we are stranded on the runway for three days. Virtually any piece of relevant information is evidence “for” one of these two theories (in that it is better explained by one than by the other), but of course neither one is really “right”, and so my brain can spend any amount of time accumulating “evidence” for the two sides without really resolving the question. The trick is to notice what I’m doing, after which it usually becomes pretty obvious that some intermediate position is correct and I just didn’t want to think about it.
To be more concrete about the specific problem under discussion, a toy model that comes to mind for takeoff speed is plotting the graph over time of “what is the probability that the human species is doomed?”, where the probability is relative to some fixed reference observer (say, a group of Martian scholars who study the issue for one year at each point, with access to the contemporary human Internet). This function could have an infinite number of shapes (uncountably infinite if I remember undergrad math correctly), but you can have a rough prior over them based on complexity, with the high prior probs going to simpler classes of shapes, which you then update as evidence comes in. A “fast takeoff” scenario roughly corresponds to curves that jump to 0 or 1 (or, technically, epsilon or 1 − epsilon) very rapidly, while a “slow takeoff” would be the opposite, where the probability gradually moves towards doom or non-doom. One interesting consequence from a risk-planning perspective is that (under this toy model) it’s possible to have a “fast takeoff” even if AI takes a very long time to get going, just as long as the AI is set up such that it is clearly impossible to stop. This corresponds well with my IRL intuitions about what is important, because it’s equally valuable to avoid slow certain doom vs. fast certain doom (although IRL there is probably a decent correlation between slowness and non-certain-doomness, just because it’s hard to predict over long time horizons).
(FWIW, I don’t think it’s worth spending lots of time on semantics, but I think in common educated usage “very unlikely” means “not absolutely ruled out by laws of physics or other universal principles, but so far down in the hypothesis space that it isn’t worth the mental cycles to have the idea in your head”. Things like commercial plane crashes, one-in-a-million drug side effects, LHC destroying the earth, the US declaring war on Canada, getting eaten by sharks, etc.)
I’ve probably mentioned this at some point before, but I really don’t think ‘price’ has much meaning outside of contemporary human economies; even with relatively weak AIs you quickly start seeing breakdowns of ability to make binding contracts and property rights and so on. (The same thing happens to human economies which are sufficiently small, low-tech, lacking in civil order, etc.; eg.
Elizabethan crime , feudal land tenure in England .) You can talk about ‘prices’ in terms of computing power, free energy, or some other good that’s widely useful and generally scarce, but even there you’re likely to miss most of the interesting stuff because if you track the progress of fixed benchmarks which are easy to measure (eg. speech recognition accuracy, or chess scores), you miss all of the things which just couldn’t be done using earlier computers at all. (This is actually a problem for human economics too, where you can measure the number of loaves of bread a 2000 AD human can afford compared to a 1200 AD human, but this doesn’t capture most of the relevant differences in their standard of living.) I’m guessing it’s most likely true that, at least until the point where AIs are strongly superhuman, you wouldn’t see rapid discontinuous changes in most fixed benchmarks like the ability of a commercially available computer to perform a standardized task, but that likely isn’t saying very much.
Big leaps in technical progress are rare, but again that might not mean much depending on the context you’re in. Eg. if you compare a Wright flyer to a WWI fighter plane, they visually look pretty similar, operate on similar engineering principles, and are probably in the same order of magnitude for most obvious statistics (speed, mass, horsepower, carrying capacity, etc.), but their economic/social/political utility differs by many orders of magnitude. I can come up with lots of similar examples.
For speed/discontinuity of progress in human economies, it seems to me like the more intermediate steps between an invention and its application, the more discontinuous/surprising progress will be. Eg. if you were a 19th-century steamship engineer and you invented a better hull design, all you had to do was build a ship that was basically identical to previous ships except for having your new hull, and so that tends to get done fairly quickly and smoothly ( Blue Riband record breakers ). But in nuclear fusion, you see things like Rutherford discovering a reaction and characterizing it in the 30s, other people trying to ‘correct’ him in the 70s and 80s, and people a few years ago discovering that the ‘corrections’ were wrong and Rutherford was right after all, with each shift in knowledge taking multiple decades ( Weller, INT ). The extreme example is of course most math, where you have binary discontinuity between ‘proven’ and ‘unproven’.
On the claim that “most systems do something cool because the people who designed them understand how they work, and took great care to overcome a long list of obstacles. It’s exceptional (though not unheard of) for someone to design a system that radically outperforms their expectations rather than underperforming them, just because it is so much easier for unanticipated features of the problem to make things worse than to make things better” — agreed and widely underappreciated.
Castle Bravo is an interesting but of course atypical counterexample.
On the gap in human ability to do AI research, arguments from evolutionary progress, etc.: I have a post in draft that partially addresses this topic, planning to send to the AGI strategy list.
On “a small amount of architectural variation leads to a huge difference in performance” vs. “other factors are responsible for variation in human intelligence” — I’m not sure what counts as architectural exactly, but Linda Gottfredson’s Why g matters is a canonical resource on this, worth reading if you haven’t already.
On cases “where continuous improvements lead to radical improvements in usefulness” — if I’m understanding correctly, this is sort of like the Wright flyer vs. WWI airplane thing, of which some important examples are: Railroads ( history to 1830 , Liverpool and Manchester Railway ) Personal computers ( history , Altair 8800 , Apple II , Commodore 64 ) Automobiles ( early automobiles ) Television ( broadcast television ) In each of these cases, you see relatively smooth and incremental progress from early prototypes to refined mass-production models create huge, many-orders-of-magnitude, quite rapid jumps in utilization ( railroad investment , PC market share , automobile adoption , annual TV sales 1939–59 ).
On “the ‘business as usual’ projection is already very scary” — agreed, but very curious what specifically is in mind there.