How far can AI jump?
A reply to a draft essay by Katja Grace — then a young researcher (later at AI Impacts) writing on whether AGI was likely to emerge from a single group rather than from a diffuse research community. Alyssa, a Yale undergrad and Singularity Institute intern, had just returned from the 2009 Singularity Summit. Her argument: the standard model of scientific information leakage (think Manhattan Project) doesn’t apply, because what makes a research group win isn’t a few legible ideas but a large body of accumulated tacit expertise that resists copying.
I think it is entirely possible and perhaps likely that your predictions will be right, but I also think that would be much more dangerous, as a single group that invents puzzle pieces #1–#100 is much more likely to know what they’re doing well enough to not destroy the world than a group that copies pieces #1–#99 and just invents #100.
On technological development: there are certain kinds of information which are very easy to transmit and that, indeed, you cannot effectively keep secret. Once the US started the Manhattan Project, it was pretty much a given that the Soviets would infiltrate it, copy the US’s ideas, and build their own A-bombs. The reason, I think, that we think the scenario of a single group getting a monopoly is plausible, is that accumulated expertise appears to be very difficult to transmit.
Accumulated expertise doesn’t consist of a single or a few great ideas, but of a large body of partially-conscious and partially-subconscious heuristics which gradually build up as you understand a problem better. And expertise is much harder to copy than the ideas it generates, because often the experts themselves don’t have a clear idea of why or how they do what they do. A few historical case studies: The Roman Empire successfully conquered all of the land around the Mediterranean, not through any single military technique (which could have been, and were, copied; the Romans themselves copied the design for the quinquereme from stolen Carthaginian ships), but by developing a large number of heuristics on how to fight, how to incorporate other peoples into their culture, and how to govern effectively. E.g., the Romans tried incorporating a foreign deity into their pantheon in the year 210 BC during the Second Punic War (Cybele of Asia Minor). This turned out to work well, so they did it a lot, which helped them incorporate other towns more easily and so grow their population. If you discover hundreds of these tricks, nobody will be able to copy you quickly enough for them to beat you.
Google has become, not only the #1 Internet company, but one of the most powerful organizations in the world, primarily by accumulating a ton of heuristics on how to make search work well. If you walked up to a random comp. sci. student and gave them ten billion dollars tomorrow, as well as free access to all the paper documents in Google’s office (paper only so he couldn’t just copy-paste the code), it would probably take him a decade to build a search engine as good as Google is now. Hence, Google now has an effective monopoly on search, and it’s unlikely that they’ll be losing that position anytime soon.
The robber barons of the 19th century all grew to completely dominate their industries, because they each had developed a large set of techniques for systematically beating out their competitors and offering lower prices than anyone else could match. (Some of these techniques were, of course, subsequently made illegal, but that is beside the point.)
Warren Buffett is, far and away, the best person in the world at what he does (buying and restructuring companies), largely because he’s been doing it for so long, he and his assistants have a thousand subtle rules that they apply (some consciously, some subconsciously) to get all the best deals. His nearest competitor, Carl Icahn, isn’t even worth one-fourth as much.
On sharing of information: you are completely right that, if you pick out a few people at random from a group that’s working on some problem and shares information amongst itself, and segregate them, they will indeed rapidly fall behind. However, currently (thank Cthulhu) there does not appear to be any large, coherent group of people which is working primarily on AGI and shares information amongst itself. (If there were such a group, it would probably be very bad.) Hence, it should be possible for a small group to beat the global scientific community as a whole in terms of AGI research, as long as they focus their efforts on AGI instead of all the zillion other things that the rest of the world is also doing. This sort of thing happens pretty frequently (e.g., startups routinely beat the rest of the world at developing type of software X by focusing really hard on just working on X).
On there being a single principle behind intelligence: we’ve debunked that pretty thoroughly. See, e.g., “Levels of Organization in General Intelligence” (written in 2003).