The prime futurist fear is that humanity will create some advanced technology with an ostensibly positive purpose, but it will buck our control and undo the world as it pursues some twisted version of the ends it was programmed to achieve. Quiet Babylon writes that this artificially-sentient oppressor has already arrived:
One of my favorite recurring tropes of AI speculation/singulatarian deep time thinking is meditations on how an evil AI might destroy us.
Here’s an example: The scenario imagined is where there is a button that humans push if the AI gets an answer right and the AI wants to get a lot of button presses, and eventually it realizes that the best way to get button presses is to kill all the humans and institute a rapid fire button-pressing regime.
You would have this thing that behaves really well, until it has enough power to create a technology that gives it a decisive advantage — and then it would take that advantage and start doing what it wants to in the world.
And all I can think is: we already have one of those [in the form of] marketplace regime of firms dedicated to maximizing profit. What if the private pursuit of profit was—for a long time—proximate to improving the lot of humans but not identical to it? What if capitalism has gone feral, and started making moves that are obviously insane, but also inevitable?
For a very long time, the AI dedicated to maximizing profit saw the path forwards through innovation, new products, better living for customers. But then at some point it realized that is had the ability to just reshape the planet in its image. So it did that instead.
