[sorry for these untimely posts...I'm trying to clear out a few half-completed drafts]
The Atlantic pinged Garry Kasparov for his thoughts on IBM's Watson's victory on Jeopardy.
A convincing victory under strict parameters, and if we stay within those limits Watson can be seen as an incremental advance in how well machines understand human language. But if you put the questions from the show into Google, you also get good answers, even better ones if you simplify the questions. To me, this means Watson is doing good job of breaking language down into points of data it can mine very quickly, and that it does it slightly better than Google does against the entire Internet.
The analogy to a human using Google is a useful one. If you had infinite lifelines on Who Wants to Be a Millionaire and could call a friend who could Google for answers for you, would you always win a million dollars? Maybe not always, but fairly close. So the challenge for IBM was to figure out how to parse the Jeopardy clues into the right parameters to generate the proper query. Then Watson, like Google, had to use some algorithms for ranking the relevancy of various results.
My concern about its utility, and I read they would like it to answer medical questions, is that Watson's performance reminded me of chess computers. They play fantastically well in maybe 90% of positions, but there is a selection of positions they do not understand at all. Worse, by definition they do not understand what they do not understand and so cannot avoid them. A strong human Jeopardy! player, or a human doctor, may get the answer wrong, but he is unlikely to make a huge blunder or category error—at least not without being aware of his own doubts. We are also good at judging our own level of certainty. A computer can simulate this by an artificial confidence measurement, but I would not like to be the patient who discovers the medical equivalent of answering "Toronto" in the "US Cities" category, as Watson did.
Kasparov gives humans credit for knowing what they don't know, but in plenty of cases people are just as overconfident and blinded in their opinions. In fact, Watson had low confidence in its answer of Toronto in the US Cities category on Final Jeopardy, so it did have an "awareness" of its own uncertainty. I'm almost certain that the programmers ensured that Watson would make a guess in Final Jeopardy regardless of its confidence since there's nothing to lose at that point. In Single or Double Jeopardy, though, Watson wouldn't buzz in unless its confidence in an answer exceeded a threshold.
Humorous aside on Watson: how the computers could have beaten Watson on Jeopardy.