Notes on writing about AI risk
Benjamin Hoffman, then a software engineer in the Bay Area rationalist community who wrote on epistemics and effective altruism, had circulated drafts of a series of writeups on AI risk and asked Alyssa for feedback. His drafts leaned toward enumerating possible future scenarios without trying to weight them by likelihood, on the theory that “mapping the territory” should come before evaluation. Alyssa pushes back on three fronts — that listing scenarios without implicit likelihood judgments isn’t actually feasible; that newcomers to a sprawling field should master one specific slice before tackling the whole; and that AI-risk research as a field is so under-coordinated that literature review beats independent writeups. The note closes with a worked taxonomy of which kinds of distinctions in writing belong in the same section and which need to be split apart, illustrated with passages from Paul Graham.
I don’t think it makes very much sense to talk about describing possibilities without evaluating them, especially in a field like AI risk, where the scope is so broad and the existing scholarship so thin. If we assume total ignorance about possibility-space, then if the complexity of each hypothesis is N bits, the number of them grows as 2^N. Therefore, if each option has 30 bits of information — which isn’t very much at all — the total number of hypotheses is 2^30 or ~1 billion, which is already impractically large. One could stick to hypotheses that have very low information content, but this tends to not be very enlightening — eg. “either the US will have a war with China, or it won’t” is true, but it isn’t saying much. Or, one could have complex hypotheses with very low probability, what Bruce Schneier calls “movie plot scenarios”, in which case the analysis is probably affected by the conjunction fallacy . Or, one could secretly evaluate scenario likelihood while pretending not to — a detective who says “either Joe Smith robbed the house, or he didn’t” is ostensibly just listing scenarios, but he’s almost certainly already updated in favor of the Joe Smith hypothesis. This does work, but the (intentional or unintentional) hidden information makes collaboration harder . Any realistic scenario listing must implicitly include likelihood analysis, because to be practical, a huge majority of the possibility-space has to be implicitly excluded as not worth writing about.
When starting off in a field, especially an under-explored field with very broad scope, I think it’s almost always best to pick some specific slice to master before tackling larger questions. Eg., when I first started on my CRM application, I had never written anything in Clojure before. Clojure is a very powerful language, and web applications are complicated; there were (and still are) a million things to do. So I started by making a website where one could create an account, log in, log out, and do nothing else (which wound up taking me two weeks, FWIW). While working on that specific task, I learned a lot of general skills which helped with later stuff, like how to use various Emacs modes. But that was actually more productive than trying to learn general skills directly; in order to make that webpage, I had to really know Emacs (and Leiningen and Ring and Compojure and Hiccup and etc.), I couldn’t just memorize some trivia and talk myself into believing I was an expert (as I’ve done several times before).
Given the state of the AI risk field, my guess is that collaboration would be more fruitful than doing writeups fully independently. A lot of people have already put thought into this, and the numbers keep growing, with eg. FLI funding thirty-odd teams last year. But to a fairly high extent, each team isn’t that aware of what the other teams are doing. This makes collective progress hard because there aren’t really standard names for things, and it’s difficult to find which issues have consensus and which are still seriously debated. I’d probably recommend doing more literature review (where literature here frequently includes eg. blog posts), and trying to figure out what specific people at MIRI/FHI/FLI/GCRI/CSER/SAIRC/LCFI/GPP/OPP/CEA are thinking (in many cases, I have no idea myself).
I think there are really three separate distinctions here, which might be getting confused: one between new ideas and old ideas; one between concrete ideas and speculation; and one between ideas and meta-ideas. For distinction #2, I think it’s fine to put them both in the same section, as long as they’re appropriately labeled (eg. Scott does labeling with “epistemic status” notes). But for distinctions #1 and #3, I think you usually maximize clarity by putting them in separate sections, or separate pieces altogether. Eg., going back to Paul Graham’s writing, here are some new ideas (at least for the relevant audience): “Gradually the government realized that anti-competitive policies were doing more harm than good, and during the Carter administration it started to remove them. The word used for this process was misleadingly narrow: deregulation. What was really happening was de-oligopolization. It happened to one industry after another. Two of the most visible to consumers were air travel and long-distance phone service, which both became dramatically cheaper after deregulation.”
vs. old ideas, which are in a footnote with references: “The wave of hostile takeovers in the 1980s was enabled by a combination of circumstances: court decisions striking down state anti-takeover laws, starting with the Supreme Court’s 1982 decision in Edgar v. MITE Corp.; the Reagan administration’s comparatively sympathetic attitude toward takeovers; the Depository Institutions Act of 1982, which allowed banks and savings and loans to buy corporate bonds; a new SEC rule issued in 1982 (rule 415) that made it possible to bring corporate bonds to market faster; the creation of the junk bond business by Michael Milken; a vogue for conglomerates in the preceding period that caused many companies to be combined that never should have been; a decade of inflation that left many public companies trading below the value of their assets; and not least, the increasing complacency of managements.”
Here are concrete ideas, which might be right or wrong, but are clearly stated and supported with evidence: “Though strictly speaking World War II lasted less than 4 years for the US, its effects lasted longer. Wars make central governments more powerful, and World War II was an extreme case of this. In the US, as in all the other Allied countries, the federal government was slow to give up the new powers it had acquired. Indeed, in some respects the war didn’t end in 1945; the enemy just switched to the Soviet Union. In tax rates, federal power, defense spending, conscription, and nationalism the decades after the war looked more like wartime than prewar peacetime. And the social effects lasted too. The kid pulled into the army from behind a mule team in West Virginia didn’t simply go back to the farm afterward. Something else was waiting for him, something that looked a lot like the army.”
vs. speculation, in a later paragraph: “The breakup of the Duplo economy happened simultaneously with the spread of computing power. To what extent were computers a precondition? It would take a book to answer that. Obviously the spread of computing power was a precondition for the rise of startups. I suspect it was for most of what happened in finance too. But was it a precondition for globalization or the LBO wave? I don’t know, but I wouldn’t discount the possibility. It may be that the refragmentation was driven by computers in the way the industrial revolution was driven by steam engines. Whether or not computers were a precondition, they have certainly accelerated it.”
Here are ideas, in an essay about startup funding: “It would be safe to be default dead if you could count on investors saving you. As a rule their interest is a function of growth. If you have steep revenue growth, say over 6x a year, you can start to count on investors being interested even if you’re not profitable. But investors are so fickle that you can never do more than start to count on it. Sometimes something about your business will spook investors even if your growth is great. So no matter how good your growth is, you can never safely treat fundraising as more than a plan A. You should always have a plan B as well: you should know (as in write down) precisely what you’ll need to do to survive if you can’t raise more money, and precisely when you’ll have to switch to plan B if plan A isn’t working.”
vs. meta-ideas, in an essay about essay writing: “When I give a draft of an essay to friends, there are two things I want to know: which parts bore them, and which seem unconvincing. The boring bits can usually be fixed by cutting. But I don’t try to fix the unconvincing bits by arguing more cleverly. I need to talk the matter over. At the very least I must have explained something badly. In that case, in the course of the conversation I’ll be forced to come up a with a clearer explanation, which I can just incorporate in the essay. More often than not I have to change what I was saying as well. But the aim is never to be convincing per se. As the reader gets smarter, convincing and true become identical, so if I can convince smart readers I must be near the truth.”