Better than the Turing test

Is the Turing test really the best way to screen for artificial intelligence? One alternative sounds more promising: Winograd schemas.

The test would take the form of a multiple-choice quiz of reading comprehension. But the text itself would have some very specific features. It would consist of Winograd schemas: pairs of sentences whose intended meaning can be flipped by changing just one word. They generally involve unclear pronouns or possessives. A famous example comes from Stanford computer scientist Terry Winograd:

"The city councilmen refused the demonstrators a permit because they feared violence. Who feared violence?"
1) The city councilmen
2) The demonstrators


"The city councilmen refused the demonstrators a permit because they advocated violence. Who advocated violence?"
1) The city councilmen
2) The demonstrators

Most human beings can easily answer these questions. We use our common sense to figure out what "they" is supposed to be referring to in each case. And that common sense basically involves a combination of extensive cultural background knowledge with analytical skills. (In the first question, we can deduce that the city councilmen feared violence. In the second, the demonstrators advocated violence.)

For computers, however, these questions can be quite difficult. From a grammatical standpoint, the "they" in the sentences is technically unclear. In both questions, "they" could be either the councilmen or the demonstrators.

Is this truly a superior test of AI? We may start collecting some data points soon. In 2015, Nuance Communications is sponsoring the first of what will be an annual Winograd Schema Challenge.

I continue to insist that the next great milestone in AI is when Waze stops trying to send me to unprotected left turns as shortcuts. UPS already figured this out.