The influential 42-year-old philosopher Nick Bostrom favors the creation of “superintelligent” computers, but only if done with great vigilance, with safeguards to ensure that the machines do not escape human control and pose an existential threat to humanity.
Bostrom’s favorite apocalyptic hypothetical involves a machine that has been programmed to make paper clips (although any mundane product will do). This machine keeps getting smarter and more powerful, but never develops human values. It achieves “superintelligence.” It begins to convert all kinds of ordinary materials into paper clips. Eventually it decides to turn everything on Earth — including the human race (!!!) — into paper clips.
Then it goes interstellar.
“An ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.”
I.J. Good, a British mathematician and code-breaker, in 1965 