This is the first post in a series in which I will be making the strongest argument I can that the Singularity is worth taking very seriously. But before I begin to lay out that argument, I should specify what precisely this whole “Singularity” thing is in the first place.

Eliezer Yudkowsky has identified four schools of thought on the Singularity. The third school he describes is the “Intelligence Explosion” theory, which claims that if humans become able to augment their own intelligence (or create intelligences smarter than they are), that will make them even better at augmenting their own intelligence, and a positive feedback loop will be created. This feedback loop will continue and accelerate until some fundamental physical limit is reached, at which point the entire universe will be thoroughly unrecognizable to us mortals.

I think that this is plausible, but I do not think I need to go quite so far. Rather, I shall argue that:

  1. A smarter-than-human intelligence will be created at some point in the future, almost certainly within the next century or two.
  2. If it is not friendly, we will be completely screwed.
  3. Therefore, we should work to ensure that it is friendly.

There are currently three feasible avenues to smarter-than-human intelligence: human intelligence augmentation (IA) through some combination of bioengineering and cybernetics; “uploading” of human minds to computer hardware; and artificial software intelligence (AI). I think AI will probably be the first of these approaches to succeed, but my argument should apply to any of them.

Next: Is smarter-than-human intelligence possible?