One interesting result Wolfram reports in his blog is a study he had his company do on how well simple search engine queries would do with Jeopardy questions. They tried entering actual Jeopardy questions into all the major search engines, and measured how often the answer showed up in the first few returned entries and how often the answer showed up in the first entry.
The best search engines (Ask, Bing, and Google) got the answer right somewhere on the first page about 70% of the time, or somewhere between the average person (60%) and Ken Jennings (79%). But getting the answer right in the first returned entry lowered the search engines to the mid 60 percent. Thus in order to beat Ken Jennings, IBM's Watson must do considerably more than just a simple Google search.
Wolfram|Alpha creator Stephen Wolfram did an interesting compare and contrast with his answer engine and the way IBM’s Watson Jeopardy supercomputer operates.
Stephen Wolfram is the creator of Mathematica, the author of A New Kind of Science, the creator of Wolfram|Alpha, and the founder and CEO of Wolfram Research.
In a long blog post, Wolfram used a graphic to show how the two systems work.
Historically speaking, IBM’s approach has been around a lot longer. Wolfram writes:
IBM’s basic approach has a long history, with a lineage in the field of information retrieval that is in many ways shared with search engines. The essential idea is to start with textual documents, and then to build a system to statistically match questions that are asked to answers that are represented in the documents. (The first step is to search for textual matches to a question—using thesaurus-like and other linguistic transformations. The harder work is then to take the list of potential answers, use a diversity of different methods to score them, and finally combine these scores to choose a top answer.)
Early versions of this approach go back nearly 50 years, to the first phase of artificial intelligence research. And incremental progress has been made—notably as tracked for the past 20 years in the annual TREC (Text Retrieval Conference) question answering competition. IBM’s Jeopardy system is very much in this tradition—though with more sophisticated systems engineering, and with special features aimed at the particular (complex) task of competing on Jeopardy.
Wolfram|Alpha is a completely different kind of thing—something much more radical, based on a quite different paradigm. The key point is that Wolfram|Alpha is not dealing with documents, or anything derived from them. Instead, it is dealing directly with raw, precise, computable knowledge. And what’s inside it is not statistical representations of text, but actual representations of knowledge.
The input to Wolfram|Alpha can be a question in natural language. But what Wolfram|Alpha does is to convert this natural language into a precise computable internal form. And then it takes this form, and uses its computable knowledge to compute an answer to the question.
Wolfram notes that there could be some synergy between Wolfram|Alpha and Watson. His post is a long read, but a fun one for those interested in artificial intelligence.
Related:
