Computers, while not intelligent, can often simulate it surprisingly well. But where does the answer come from? If you ask a person how he came up with an answer, he might not be able to tell you. A computer almost certainly can not, and the answer may come from an angle that a human would likely not come up with.
There is a story of a version of Word’s spelling checker replacing “ZZZZ” with “sex”. Looking at it in retrospect, one can see the reasoning: S and Z and X all have S or similar sounds, but would you have come up with this suggestion?
In Nate Silver’s ‘The Signal and the Noise’, the story is told of how Kasparov was psyched out by a move that Deep Blue (IBM’s chess program) made in one game.
‘”A bug occurred in the game and it may have made Kasparov misunderstand the capabilities of Deep Blue,” Campbell told me. “He didn’t come up with the theory that the move it played was a bug.”‘
Instead, Kasparov attributed more intelligence to Deep Blue than the program had. What would happen if this same error was made by someone using an supposedly intelligent program in a critical situation?