Questions that influence AGI-risk probabilities
A working list Alyssa sent to Anna Salamon in mid-2008, while doing research for the Singularity Institute’s “Uncertain Future” world-modeling project. The aim is to enumerate the cruxes — the questions whose answers actually move probability mass — behind any quantitative estimate of AGI risk. The Yudkowsky epigraph sets the problem: as of early 2006, there was no prior literature to cite.
And, for reasons discussed in a later section, on the topic of global catastrophic risks of Artificial Intelligence, there is virtually no discussion in the existing technical literature. I have perforce analyzed the matter from my own perspective; given my own conclusions and done my best to support them in limited space. It is not that I have neglected to cite the existing major works on this topic, but that, to the best of my ability to discern, there are no existing major works to cite (as of January 2006).
— Eliezer Yudkowsky, “Artificial Intelligence as a Positive and Negative Factor in Global Risk” What are the odds that AGI is theoretically possible, given unlimited resources?
How much computing power will AGI require, with what probability?
How long will Moore’s Law continue, with what probability?
How intelligent would an initial AGI be, relative to a human, with what probability?
How strong is the correlation between intelligence and optimization power, with what probability?
What is the likelihood that AGIs will be significantly encumbered relative to humans, due to the lack of a ready-made environment-interaction system?
What are the odds that AGIs can be made Friendly, given unlimited resources?
Given unlimited power, how likely is it that a generic AGI will cause existential catastrophe?
Given unlimited power, how likely is it that a Friendly AGI will cause existential catastrophe?
How likely is it that we will make a serious effort to attempt AGI in the near future?
What resources are likely to be available for these attempts, with what probability?
What level of effort is necessary to build AGI, with what probability?
What level of researcher intelligence is necessary to build AGI, with what probability?
How probable is it that AGI will require intelligence beyond the human-maximum level?
How much government regulation is there likely to be of AGI, with what probability?
What are the chances of a positive / negative outcome resulting from government regulation?
How likely are other existential catastrophes which would eliminate or greatly reduce the number of AGI attempts?
How many intelligent people are likely to be involved in AGI efforts?
How strong is the positive feedback from recursive self-improvement likely to be?
What initial level of intelligence is required to start recursive self-improvement, with what probability?
How likely is it that an anti-AGI culture in academia can prevent AGI development or research?
How likely is it that a blanket government ban on AGI-related matters can prevent AGI development or research?
How likely is it that an already-extant AGI can prevent further AGI development or research?
What are the odds that human programmers will be able to maintain involuntary (regardless of goals) control of an AGI, after the AGI is already built and running?
How strong will Darwinian selection pressures be for AGIs, with what probability?
How intelligent will human uploads be, relative to AGIs, with what probability?
How likely is it that human uploads will be developed before AGI?
What are the odds that AGI requires quantum computing, or some other kind of unconventional hardware?
Given working AGI, how long would it take it to build an information / heuristics database sufficient to exceed a human’s, with what probability?
How realistic / likely are scenarios in fiction involving AGI, e.g.
The Matrix , I, Robot , etc.?
Given all available human-species resources, how long would it take to build AGI, with what probability?
How likely is it that AGI is just another “the end of the world is nigh” scenario with no substance?
How strong a predictor is previous work in narrow AI of successes and failures in AGI?
Given that Friendly AGI is possible, what is the probability that a generic, human-built AGI will be Friendly?
Given that Friendly AGI is possible, how much extra work will be required for Friendly AGI projects?
Out of all AGI researchers / projects, how many will consider Friendliness important, with what probability?
How predictable will specific behaviors of AGIs be?