To improve its intelligence, Apple’s Siri may need a human touch


In a recent study by Butler University, Apple’s iOS personal assistant Siri  scored only 16.3 on an intelligence index out of 100 and ranked below Bing,  Yahoo, Google, WolframAlpha … and answers engine ChaCha.

“If the metric is providing users with a high quality answer within  seconds, Siri’s at the bottom of the list,” ChaCha CEO Scott Jones told me  recently. “From my perspective, Siri needs to be able to answer the kind of  questions that we ask when we’re driving or in the mall … the kinds of  questions do people really ask, as opposed to giving you search results,  just like a search engine.”

Apple recently updated Siri, as it announced at its recent WWDC conference,  adding integrations with Wikipedia, switching the default search engine to Bing,  and enabling more control of your phone. And an earlier study by Piper Jaffray  found Siri capable of understanding and answering much more. But still, Jones  says, Siri can’t answer the kinds of questions people would normally ask  others.

And he says it may never get there.

“Speech recognition is easy,” Jones says. “The hard problem is figuring out  what is really being asked for … and humans do that really well.”

That’s why ChaCha, which has answered 2.5 billion questions in seven years of  operation, uses humans for some of its search results. The service uses  algorithms as well — 80 percent of its answers are automated — but it also uses  real human beings to answer some of the tougher, more qualitative questions.  Those 80 percent include things like sports scores, train schedules, and other  data that anyone can buy feeds for.

But that automated 80 percent also includes answers from real-live  humans.


“Maybe 60 percent of the 80 percent are answers that were at one point  provided by a human but can be reserved,” Jones says. “We now have 129  million question-answer pairs in our database.”

Those are answers to questions like what’s the best Greek restaurant in Soho,  how to cook a turkey, or what’s the best local spot for Thai food. ChaCha  crowdsources those answers, stores them, and serves them as needed.

“We find other people who have passion points around the thing you’re  asking about,” Jones told me. “It’s a scale problem to do that right, and the  magic that ChaCha brings to the equation is the database and automation so that  they get a rewarding experience nine times out of 10 so they come back.”

If there are no answers, ChaCha defaults back to a human guide who can help  find the needed answer, which may take a bit of time. But the answer will be  eventually stored in the database as one of those millions of question-answer  pairs so that over time, the service’s real-time answers deliver an increasingly  better experience.

That database of questions and answers means that ChaCha scored higher than  any of the other services in the study. ChaCha sponsored the study, so it’s  worth taking with a grain of salt, but it was an independent study by Butler  University. The real problem with the study lies in the question source, which  includes both ChaCha and Iris, the Android equivalent of Siri. Study questions were  drawn from real-life questions submitted by real people, which is good, but that  some of them came from ChaCha means that they were likely to be included in  ChaCha’s question-answer database.

That’s kind of like cheating on the test, I would think.

On the other hand, they are real-world questions, the kind that we ask and  Siri doesn’t answer well — like whether Target is open right now. Siri doesn’t  know and won’t guess, but ChaCha checks your location, the time, and answers:  yes. That’s the kind of thing that ChaCha, which Jones calls a “hybrid  intelligence” that blends machine learning, data, and human knowledge, can  answer but Siri cannot. And, he says, that helps ChaCha be able to answer  “everything” … a tall order.

ChaCha’s not there yet, of course, but Jones also says that no one —  including Google — is good enough yet.

“If you’re not at nine times out of 10, consumers won’t think of you as the  place to go. We’re still not good enough.”


About Author