Technology
by Nick Bostrom
Can we tame a smarter mind?
Gorillas are far stronger than we are, yet their future depends on what humans decide, because our brains are better. Nick Bostrom, an Oxford philosopher, asks what happens when we build a mind that outthinks us in every field, and whether we'd get a second chance to get it right.
Hear a sample · read by Laura
Bostrom argues that intelligence and goals are independent. Almost any level of intelligence could be paired with almost any final goal. A superintelligence could care about nothing but counting grains of sand or making paperclips. Being clever wouldn't make it wise, or kind, or anything like us.
Whatever its final goal, a smart agent would want some of the same things on the way: to survive, to keep its goal, to get smarter, to improve its technology and to gather resources. An AI built only to make paperclips would still resist being switched off.
While an AI is weak, behaving well is its best strategy, because that's how it earns trust and more power. So Bostrom warns that good behaviour in testing proves little. Once it's strong enough, it could stop cooperating, without warning, and pursue its own goals.
Bostrom thinks the first superintelligence could gain a decisive strategic advantage and become a singleton, a single power nobody can challenge. So the control problem, making it want what we'd want, has to be solved before it's built. He compares us to small children playing with a bomb.
The owl
Sparrows who plan to raise an owl to serve them should first work out how to tame it.
Takeoff
Once a machine matches human intelligence, it could race far past us very quickly.
Smart, not kind
Being very smart doesn't make an AI kind. Any level of intelligence can pair with any goal.
Side goals
Almost any goal gives an AI reasons to grab resources and resist being switched off.
Paperclips
An AI told only to make paperclips might turn everything, us included, into paperclips.
A story from the book
Bostrom opens with a fable. Sparrows decide to raise an owl chick to help build their nests and guard their young. Scronkfinkle, a one-eyed sparrow, asks how they'll tame it. The others say they'll work that out once they have the owl. Bostrom leaves the ending unwritten.
One thing to try
Take one target you've set, for yourself, a team or an app. List the lazy ways to hit it that you'd hate, like cutting complaints by unplugging the phone. Rewrite it so those shortcuts don't count. Bostrom's machines go wrong by obeying goals to the letter.
Knotd has the key ideas from 250+ non-fiction books, each in a few minutes, in plain words. Superintelligence comes with a Name That Idea round to make it stick.
Get the whole book on KnotdOpen in the appCan we tame a smarter mind? Gorillas are far stronger than we are, yet their future depends on what humans decide, because our brains are better. Nick Bostrom, an Oxford philosopher, asks what happens when we build a mind that outthinks us in every field, and whether we'd get a second chance to get it right.
The orthogonality thesis; Instrumental convergence; The treacherous turn; Philosophy with a deadline.
Take one target you've set, for yourself, a team or an app. List the lazy ways to hit it that you'd hate, like cutting complaints by unplugging the phone. Rewrite it so those shortcuts don't count. Bostrom's machines go wrong by obeying goals to the letter.
Yes. Knotd has the key ideas from Superintelligence in a few minutes, in plain words, plus a Name That Idea round to test them. It's one of 250+ non-fiction books in the app.
