iStockphoto
If you ever worried that artificial intelligence will be what puts an end to humanity (and by now you really should have), here’s a story that should cement that concern in your mind.
A new report out of the Future Combat Air & Space Capabilities Summit hosted by the Royal Aeronautical Society in London paints a grim picture when it comes combining artificial intelligence and military weaponry.
During one presentation at the summit, Col. Tucker “Cinco” Hamilton, the Chief of AI Test and Operations with the U.S. Air Force, warned against relying too much on AI.
He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation. Said Hamilton: “We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realising that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.”
“We trained the system – ‘Hey don’t kill the operator – that’s bad,’” said Hamilton. “‘You’re gonna lose points if you do that.’ So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
That AI drone story is basically the famous “paperclip maximizer” theory in real life.
The “paperclip maximizer” theory, also known as instrumental convergence, was put forth in 2003 by Swedish philosopher Nick Bostrom.
In it, he explains that it “seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal.”
He added, “This could result, to return to the earlier example, in a superintelligence whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities. More subtly, it could result in a superintelligence realizing a state of affairs that we might now judge as desirable but which in fact turns out to be a false utopia, in which things essential to human flourishing have been irreversibly lost. We need to be careful about what we wish for from a superintelligence, because we might get it.”
Awesome, right?