The press, the machine, the railway, the telegraph are premises whose thousand-year conclusion no one has yet dared to draw. - Friedrich Nietzsche
The strange thing is that all of this took so long and happened so suddenly. -Ted Nelson, author of “Computer Lib”, on the advent of the personal computer
Technological progress can come in bursts. A promising idea may wait for cheaper hardware or a better way to train a model before it becomes practical. That uneven pace makes it tempting to read each breakthrough as a preview of everything AI will eventually do.
What Is General AI?
Artificial general intelligence (AGI) refers to broad competence across tasks, including the ability to learn unfamiliar tasks and apply knowledge in new situations. Definitions differ over the range of tasks and level of performance required. A framework proposed by Google DeepMind researchers treats generality, the breadth of a system’s abilities, and performance, how well it carries out those tasks, as separate dimensions.
Superintelligence adds a much higher performance threshold. In Nick Bostrom’s definition, it means an intellect that substantially exceeds the best humans across practically every cognitive field. Human-level general competence and broadly superhuman competence are different targets.
Narrow AI has a limited scope, even when its performance within that scope is extraordinary. At a local chess club I know, the inside joke was: “Good at chess, bad at life.” Navigating an unfamiliar social situation calls for skills beyond choosing a move on a board.
What Does Strong AI Mean?
The term strong AI has a philosophical meaning. In his 1980 paper Minds, Brains, and Programs, John Searle used it for the claim that an appropriately programmed computer could literally understand and have mental states. His term weak AI described the use of computers as tools for studying the mind.
Searle’s distinction asks whether a machine understands; an AGI assessment asks which activities it can perform and how well. Using the terms interchangeably mixes a question about minds with a question about capabilities.
What Does AlphaZero Show?
DeepMind’s AlphaZero learned chess, shogi and Go through self-play. The researchers trained a separate instance from scratch for each board game, using the same learning approach and starting each network with random parameters.
AlphaZero showed that one learning method could reach superhuman performance in several games with different rules. The generality was in the training method; each resulting model had learned its own game. Establishing AGI would require evaluating learning and performance across a much broader range of activities.
How Should We Judge Progress?
An improvement on an image-recognition benchmark measures performance on that benchmark’s images and labels. To understand its reach, we need to know how the model handles unfamiliar conditions and which new tasks it can learn. Reliable performance across those changes is part of what makes generality difficult to measure.
Debates about future AI often confuse position with velocity: what a system can do now and how quickly its abilities are changing. A forecast needs to specify that rate of change and the obstacles further progress would have to overcome, along with a timescale on which the prediction can be tested.1
Superintelligence Quotes
The genie is out of the bottle. We need to move forward on artificial intelligence development but we also need to be mindful of its very real dangers. I fear that AI may replace humans altogether. If people design computer viruses, someone will design AI that replicates itself. This will be a new form of life that will outperform humans. - Stephen Hawking in WIRED
I think we should be very careful about artificial intelligence. If I were to guess like what our biggest existential threat is, it’s probably that…. Increasingly scientists think there should be some regulatory oversight maybe at the national and international level, just to make sure that we don’t do something very foolish. With artificial intelligence we are summoning the demon. In all those stories where there’s the guy with the pentagram and the holy water, it’s like yeah he’s sure he can control the demon. Didn’t work out. - Elon Musk at MIT
I am in the camp that is concerned about super intelligence. First the machines will do a lot of jobs for us and not be super intelligent. That should be positive if we manage it well. A few decades after that though the intelligence is strong enough to be a concern. I agree with Elon Musk and some others on this and don’t understand why some people are not concerned. - Bill Gates during his Reddit AMA
It is the business of the future to be dangerous. - Alfred North Whitehead
Thinking about AI is the cocaine of technologists: it makes us excited, and needlessly paranoid. - Chris Nicholson in WIRED ;)
Footnotes
1) In his review of Steven Pinker’s book, “Enlightenment Now”, Scott Aaronson analyzes Pinker’s AI optimism.
