Thoughts on AI timeline surveys
Katja Grace, who would later found AI Impacts, was at the time designing a survey to elicit researchers’ predictions about when various AI capabilities would arrive. The standard framing — “when will occupation X be fully automated?” — tends to produce noisy answers tangled up with politics about jobs and inequality. The notes below, sent to Katja and Oliver Habryka as a partial draft, propose reframing the questions around concrete physical capabilities a machine could be built to perform.
Our current knowledge is so murky that putting together a set of timeline-to-AGI milestones probably needs several rounds of iteration-and-feedback, where one calls researchers up, asks open-ended questions, uses their responses to narrow the questions down, and repeats. This is annoying, of course, but I don’t really see any way around it. A good set of milestones would be ones where people mostly agree on the ordering, and where they’re fairly evenly spaced from the present day out to superintelligence.
One thing that I’d like to do with the data is figure out what risks might be encountered before full superintelligence-caused-extinction. I sadly don’t have a formal argument for this yet, but it seems reasonably likely that before an existential risk, one would encounter one or more lesser risks in the same domain. Eg., it’s way easier to evade anti-virus software than it is to take over the world, so you’d expect to see increasingly dangerous/destructive viruses as AI advanced. (Of course, if there turn out to be no such lesser risks, that’s very important too!)
IMO, “the following human occupations will be equally likely to be fully automated” is a pretty bad way of phrasing the question — it shifts the focus of the survey from computer science to politics. I think the survey needs respondents to think in terms of concrete engineering capabilities: “what would I need, realistically speaking, to build a machine that does physical task X?”. Using words like “occupations” brings in all kinds of Blue Team/Green Team distractions. There’s lately been a campaign by the media to blame various economic woes on “AI taking all the jobs”, but I think this is largely just made up out of people’s heads for political reasons. To make a very long story very short, the share of national income going to owners of housing has doubled over the last 40 years, while the share of income going to all other forms of capital (including owners of factories, owners of software, etc.) has remained mostly constant. Eliezer wrote an FAQ on this question back in 2013.
A much better way to phrase things is in terms of concrete, physical tasks, preferably tasks performed by commodity software running on commodity hardware. Eg. instead of “replacing waiters”, one could ask something like “when will it be possible to build a commercially available, mass-produced robot that can carry items from place to place inside a busy, crowded room, without damaging the items, requiring the people in the room to take special precautions, or needing continuous human supervision?” Or “when will it be possible to build a commercially available, mass-produced robot that can, when pre-programmed with a set of cooking recipes, use each recipe to find necessary ingredients in a room, add each ingredient in the desired order, and operate common kitchen tools (ovens, knives, mixers, etc.) to produce the meal each recipe calls for?”
For purposes of planning, the timing of when the price of A drops below the price of B isn’t super-important, except insofar as it allows one to predict other events. IMO, for most technologies the three important markers are a) can it be done at all, b) can it be done in a widely-available, commercial setting, and c) is it used more than other options in practice. Eg. solar power has passed a) and b) a while ago, and its rapid price drop lets one predict with reasonable accuracy that c) will happen in a decade or two, but the real-world effects of c) still won’t be seen for some time.
Computer programming is a very important subset of tasks, but it’s so complex and multifaceted that it will never be automated all at once. Instead, different parts will gradually be automated over time, and I think a lot of this will look like better languages, libraries and compilers. There are lots of past examples, but eg. Eliezer discusses an intriguing one by Sebastian Thrun in Selling Nonapples : “Earlier in his career, Sebastian Thrun also wrote a programming language for roboticists. Thrun’s language was named CES, which stands for C++ for Embedded Systems.
CES is a language extension for C++. Its types include probability distributions, which makes it easy for programmers to manipulate and combine multiple sources of uncertain information. And for differentiable variables — including probabilities — the language enables automatic optimization using techniques like gradient descent. Programmers can declare ‘gaps’ in the code to be filled in by training cases: ‘Write me this function.’
As a result, Thrun was able to write a small, corridor-navigating mail-delivery robot using 137 lines of code, and this robot required less than 2 hours of training. As Thrun notes, ‘Comparable systems usually require at least two orders of magnitude more code and are considerably more difficult to implement.’ Similarly, a 5,000-line robot localization algorithm was reimplemented in 52 lines.”
Unfortunately I’m not really an expert here; I’d recommend asking someone who knows both machine learning and compiler/programming language design for advice.