Why deep RL doesn’t scale to the real world
A reply to William Eden, a friend in the Bay Area rationalist scene, after he had circulated Alex Irpan’s well-known February 2018 blog post “Deep Reinforcement Learning Doesn’t Work Yet”. At the time Alyssa was working at Apprente, an AI startup whose stealth-mode status limits what she can say about specific results. The exchange covers the gap between toy benchmarks and real-world AI performance, the slowdown in compute scaling, and the likelihood of further pre-AGI “revolutions” analogous to the deep learning one.
I’m unfortunately somewhat limited in what I can say by my position at a still-fairly-stealthy AI startup. I’m fairly confident, for example, that many of the known limitations with currently hot ML techniques (the large number of examples needed for deep RL, for instance) won’t be solved just by straightforward extensions of existing code. There’s a frustrating tendency for some papers to a) identify well-known problem with (DNNs/DQNs/LSTMs/etc.), b) come up with solution, c) test solution on toy examples, d) publish decent results, e) proclaim (or at least heavily imply) problem is now solved and non-toy applications will follow Real Soon. Of course, the solution then fails on non-toy examples in some fundamental way, just as SHRDLU couldn’t be readily extended from blocks to general conversation. In some cases, I’ve personally confirmed this by replicating papers and showing that <blah> fails on all the non-toy examples I find, but it’s hard to make a strong case for that without actually saying which papers and which numbers I got. Oh well.
Similarly, it seems likely that we’re hitting diminishing returns on throwing more raw compute and matrix multiplications at existing software. Clock speeds have long hit the wall, of course. Transistor density seems close to hitting the wall; rumor is that the upcoming GTX 2080 Ti will actually have fewer cores than the 1080 Ti, although presumably there’s some efficiency tweaks to make up for it. (Rather than falling, consumer DRAM prices have more than doubled since last year, I think for the first time in history.) I know that there’s lots of startups still working on faster hardware, but the existing stuff (clusters of ASICs costing hundreds of millions, with additional hundreds of millions going into the ASIC development) is optimized enough that it looks like a high bar to clear.
Because of all that, before we get AGI or even narrow AI as a significant fraction of the economy, it seems probable that we’ll see several new “revolutions”, in roughly the style of the current deep learning revolution. These are inherently difficult to plot on a curve, since at the early stages few if anyone know what they’ll be, and they’re small enough such that the luck of individual people or projects can substantially affect the outcome. (What would have happened to modern DNNs if a truck had hit Geoff Hinton many years ago? They’d still be discovered eventually, I expect, but it’d be a significantly different timeline.) Since a) I researched it a lot in my previous work and b) I’m not really working on it at the moment, I’ll hazard a guess that at least one of these will be the ability to integrate and memorize information from noisy data in linear time. DARPA’s AIDA program is now tackling this, although I’m not sure if they’ll have any luck, the teams at the Proposer’s Day seemed lackluster although obviously it’s tricky to judge.
One thing that I expect would help is going over many different past revolutions — not just in AI but in many fields, like eg. nuclear weapons history or automobile history — and researching them in enough detail to get a feel for the patterns and what paths can be taken. I’ve done this a bit now, and it’s already taught me some key things. One is that timelines often talk about when stuff is “invented”, but for that to be meaningful, you have to be very careful about what exactly it does (and doesn’t) mean. The nuclear reactor, for example, was “invented” in 1942, in that the first working full-scale example started operations then (Chicago Pile-1). But that didn’t mean that nuclear plants suddenly appeared everywhere, like a video game where you unlock the tech tree and suddenly you get to upgrade all your gear. It was three years before they had any significant impact on the world, despite enormous pressure for speed. It was twelve years before the first “nuclear reactor” as we’d think of it now (commercial electric power plant). It was decades before they were a substantial part of the world’s generation capacity, and many places still don’t have them in 2018, even when they’d obviously make sense (McMurdo still uses diesel gensets!). Even in semiconductors, long the example of fantastic speed, the steps between “first working implementation” and “deployed in a significant percentage of the market worldwide” still take years, as everyone was recently reminded with the Spectre and Meltdown exploits.