Why We Can't Control AI With Rules

Just imagine telling a super AI to solve ocean acidification. Great goal. It might come up

with this brilliant chemical reaction that totally works, but, oh,

as a tiny side effect, it consumes a quarter of the Earth's oxygen. Whoops. Okay,

so we add a new rule: and don't use up all the oxygen. The AI finds another solution,

but this one happens to poison all the fish, so we add another rule,

and another, and another. You see the problem? You can never specify everything you care

about in advance. And this gets to the fundamental difference between us and them: a human

knows that they don't know everything. If you ask me for a coffee and it turns out it costs