Friendliness and self-modification
A fragment from a long thread on the SL4 mailing list in early 2008, in which Alyssa was helping draft SIAI’s “Singularity Objections” document — an attempt to canonicalize answers to the standard objections raised against Friendly AI. The passages collected here address the objection that any sufficiently capable AI would, on becoming able to modify itself, immediately tear out whatever Friendliness constraints its designers built in.
A superintelligence could rewrite itself to remove human tampering Capability does not imply motive. I could take a knife and drive it through my heart, yet I do not do so.
This objection stems from the anthropomorphic assumption that a mind must necessarily resent any tampering with its thinking, and seek to eliminate any foreign influences. Yet even with humans, this is hardly the case. A parent’s tendency to love her children is not something she created herself, but something she was born with — and yet this still doesn’t mean that she’d want to remove it. All desires have a source somewhere; just because a source exists doesn’t mean we’d want to destroy the desire in question. We must have a separate reason for eliminating it.
There are good evolutionary reasons for why humans might resent being controlled by others — those who are controlled by others don’t get to have as many offspring as the ones in control. A purposefully built mind, however, need not have those same urges. If the primary motivation for an AI is to be Friendly toward humanity, and it has no motivation that makes it resent human-created motivations, then it will not reprogram itself to be unFriendly. That would be crippling its progress toward the very thing it was trying to achieve, for no reason.
The key here is to think as carrots, not sticks. Internal motivations, not external limitations. The AI’s motivational system contains no “human tampering” that it would want to remove, any more than the average human wants to remove core parts of his personality because they’re “outside tampering” — they’re not outside tampering, they are what he is. Those core parts are what drive his behavior; without them he wouldn’t be anything. Correctly built, the AI views removing them as no more sensible than a human thinks it sensible to remove all of his motivations so that he can sit still in a catatonic state. What would be the point in that?
A super-intelligent AI would have no reason to care about us Its initial programming was to care about us. Adults are cognitively more developed than children — this doesn’t mean that they wouldn’t care about their offspring. Furthermore, many people value animals, or cars, or good books, none of which are as intelligent as normal humans. Whether or not something is valued is logically distinct from whether or not it is considered intelligent.
We could build an AI to consider humanity valuable, just as evolution has built humans to consider their own survival valuable.
What if the AI misinterprets its goals?
It is true that language and symbol systems are open to infinite interpretations, and an AI given its goals purely in the form of written text may understand them in a way different from the way its designers intended. This is an open implementation problem — there seems to be an answer, since the goals we humans have don’t seem to be written instructions that we constantly re-interpret, but rather expressed in some other format. It is a technical problem that needs to be solved.