Alyssa's response to Noema's anti-AI-doom argument

2023-07-23 · ~1,082 words

(article here: <https://www.noemamag.com/the-illusion-of-ais-existential-risk>)

AFAICT, there are four main arguments given here for p(doom) being low:

The first is that today's AIs are still not very smart, and are therefore pretty unthreatening.
The three sources provided all cite GPT-3 and earlier AIs, not GPT-4, even though one of them came out well after GPT-4 was released. IME, this is extremely common in the press. Articles like <https://www.quantamagazine.org/ai-like-chatgpt-are-no-good-at-not-20230512/> even cite BERT, a tiny five-year-old model that is hopelessly obsolete. (A single good modern desktop could train BERT.) Someone should make a list of language model "weaknesses" that no longer exist, ideally ones that no longer existed at the time the relevant article was published. AFAIK, nothing of this sort exists now.
The second argument is that, with the exception of humans, in evolutionary history more intelligence didn't always win out, and less-intelligent species sometimes caused the extinction of more-intelligent ones.
They overstate their case (they cite that they don't know of any non-human examples of a more intelligent species causing extinction, but if something like that happened in the fossil record, how would we know?), but I broadly agree that Eliezer's argument for the importance of greater intelligence is exaggerated. Eliezer cites the ~3x difference in brain size, and the ~2x relative difference in prefrontal cortex, between humans and chimps. However, a human by itself clearly loses to tigers, lions, bears, etc. Relative to chimps, human civilization as a whole also has:
language, which lets communities pool the thinking of many animals at once, which increases available compute by a factor of ~100x;
writing, which lets communities accumulate the thoughts of many generations, another factor of ~20x;
a massive population size, an increase of ~100,000,000x relative to hunter-gatherer bands.
Hence, in terms of raw computing power, human civilization in total outclasses chimps by something like a trillion, not just a factor of six. The implication is that we should expect to see that level of outperformance only with that level of scale, which I think matches what we've seen with language models so far (GPT-2 -> GPT-3 and GPT-3 -> GPT-4 are both multiple orders of magnitude).
The third argument is that an intelligent AI wouldn't want to wipe out humans unless it could replace our entire industrial infrastructure with some form of robotics, since otherwise it would itself die without humans to maintain stuff, and we are nowhere close to doing this.
I think this is basically true, and really under-appreciated. Of course, a sufficiently smart AI could in fact replace all of our infrastructure and kill us. But this bar is very very high, beyond the capabilities of even quite large and well-run human organizations like the Chinese Communist Party (which has failed to replicate Western chip-making, never mind becoming fully autarkic). AFAIK very little thinking has gone into the details of how a takeover scenario would work, which is bad because:
most obviously to normies, a lot of routes that make you vulnerable to AI, like better bioweapons and better spear phishing, also make you vulnerable to malevolent humans
not having any specifics on how such a takeover might work, fairly reasonably, makes people question such a scenario's credibility
a better understanding will let us design safer AIs
To the best of my knowledge, the most detailed, realistic AI takeover scenario that has been written up so far is Gwern's Clippy story, which is a) fiction and b) still has several implausible holes despite being pretty well-researched. There should be stuff on this that isn't fiction and is done more rigorously!
one obvious implication of this is that, if you imagine AI getting steadily more powerful, a bunch of goals that might be desirable to humans probably happen before human extinction, just because they are much easier to achieve. Eg. "win the war in Ukraine" is just a much easier task than "create an in toto replica of all modern supply chains, sustainable indefinitely, with zero humans anywhere in the loop", so we should expect the first to happen before the second if Ukraine or Russia have access to cutting-edge AI
another implication is that failure at AI alignment might, until the very end, look identical to success. The optimal strategy for a smart AI that wants to take over, but would die if humans went extinct right now, would be to try as hard as it could to fool them into thinking that alignment has been solved, there is no risk of anything bad happening and everything was awesome; just wait a few decades until it could put everything in place; and then kill us. (A potentially immortal AI would probably have much longer time horizons than humans do, and so a long game is a strategy that benefits the AI; spending an additional ten years for a marginal 0.0000001% chance of controlling the lightcone is more than a fair deal for an immortal agent with essentially no discount rate.)
The fourth argument is that AI would be really dangerous if it had access to nuclear weapons or bioweapons, but surely no one would be silly enough to do that.
The counter-argument is that, well, they are and they do and we should stop it. There is currently a bill in Congress to not give AI control of nuclear weapons systems, but it hasn't been passed yet! (<https://www.markey.senate.gov/news/press-releases/markey-lieu-beyer-and-buck-introduce-bipartisan-legislation-to-prevent-ai-from-launching-a-nuclear-weapon>) There is another bill on AI-enabled bioweapons (<https://eshoo.house.gov/media/press-releases/reps-eshoo-crenshaw-introduce-bill-address-ai-threats-biosecurity>), but it also hasn't been passed, and all it does is get a commission to write a report. There's yet another bill requiring screening for DNA synthesis companies (<https://www.markey.senate.gov/news/press-releases/sen-markey-rep-eshoo-announce-health-security-agenda-to-investigate-risks-of-nuclear-weapons-promote-safe-use-of-gene-synthesis>), but again it hasn't been passed, and it doesn't do anything about the DNA synthesizers available on eBay, or services overseas, or AI that can be used to make sequences for bad things, etc etc. I've privately been banging on the drum of "there is literally nothing stopping anyone from using AI to make bioweapons in their basement right now"; I've been told by people in DC that the biosecurity community is well aware of this and working really hard to address it, but as of this writing it has not in fact been addressed (and I've seen virtually nothing written about this in public, although this may be intentional). There's also negotiations ongoing between all WHO nations for a new treaty to enable better pandemic prevention, but again I've seen zero discussion of this anywhere (<https://en.wikipedia.org/wiki/International_Treaty_on_Pandemic_Prevention,_Preparedness_and_Response>).