IBM takes on Jeopardy

[General Zod]

Now, I said I "used to". These benefits are still there and would still be good. But, how much effect would it have? Lay people giving commands to the computer would be helped. But, things like GUIs have already gone a long way for that, and specialized commands need a level of precision natural language doesn't have. (Now, an AI with such a processor could figure out the precise commands on its own, but the parser alone won't help much.)

What about answering questions? Stupid search engines already do a good job on that, since so many questions are already asked and ansered on the internet, often with literal wording. The computer doesn't need to understand.

I find there's a lot of drawbacks with plain searches. If I'm looking for something obscure and I can only remember a few details, it makes the task a bit more challenging. Searching for images is still difficult unless people tag them sufficiently and you know exactly how to describe it. Anything abstract at all cranks up the difficulty, and I still get a bit more use out of simple boolean than plain questions.

[FSTargetDrone]

We started watching Jeopardy! regularly after Jennings started his streak and have watched it fairly often since. It's the only game show I watch. It was a very interesting and enjoyable competition. To watch, that is. Jennings and Rutter seemed to be really frustrated at times.

Anyway, more from Jennings:

My Puny Human Brain

Jeopardy! genius Ken Jennings on what it's like to play against a supercomputer.

By Ken Jennings

Posted Wednesday, Feb. 16, 2011, at 11:04 PM ET

When I was selected as one of the two human players to be pitted against IBM's "Watson" supercomputer in a special man-vs.-machine Jeopardy! exhibition match, I felt honored, even heroic. I envisioned myself as the Great Carbon-Based Hope against a new generation of thinking machines—which, if Hollywood is to believed, will inevitably run amok, build unstoppable robot shells, and destroy us all. But at IBM's Thomas J. Watson Research Lab, an Eero Saarinen-designed fortress in the snowy wilds of New York's Westchester County, where the shows taped last month, I wasn't the hero at all. I was the villain.

This was to be an away game for humanity, I realized as I walked onto the slightly-smaller-than-regulation Jeopardy! set that had been mocked up in the building's main auditorium. In the middle of the floor was a huge image of Watson's on-camera avatar, a glowing blue ball crisscrossed by "threads" of thought—42 threads, to be precise, an in-joke for Douglas Adams fans. The stands were full of hopeful IBM programmers and executives, whispering excitedly and pumping their fists every time their digital darling nailed a question. A Watson loss would be invigorating for Luddites and computer-phobes everywhere, but bad news for IBM shareholders.

The IBM team had every reason to be hopeful. Watson seems to represent a giant leap forward in the field of natural-language processing—the ability to understand and respond to everyday English, the way Ask Jeeves did (with uneven results) in the dot-com boom. Jeopardy! clues cover an open domain of human knowledge—every subject imaginable—and are full of booby traps for computers: puns, slang, wordplay, oblique allusions. But in just a few years, Watson has learned—yes, it learns—to deal with some of the myriad complexities of English. When it sees the word "Blondie," it's very good at figuring out whether Jeopardy! means the cookie, the comic strip, or the new-wave band.

I expected Watson's bag of cognitive tricks to be fairly shallow, but I felt an uneasy sense of familiarity as its programmers briefed us before the big match: The computer's techniques for unraveling Jeopardy! clues sounded just like mine. That machine zeroes in on key words in a clue, then combs its memory (in Watson's case, a 15-terabyte data bank of human knowledge) for clusters of associations with those words. It rigorously checks the top hits against all the contextual information it can muster: the category name; the kind of answer being sought; the time, place, and gender hinted at in the clue; and so on. And when it feels "sure" enough, it decides to buzz. This is all an instant, intuitive process for a human Jeopardy! player, but I felt convinced that under the hood my brain was doing more or less the same thing.

Indeed, playing against Watson turned out to be a lot like any other Jeopardy! game, though out of the corner of my eye I could see that the middle player had a plasma screen for a face. Watson has lots in common with a top-ranked human Jeopardy! player: It's very smart, very fast, speaks in an uneven monotone, and has never known the touch of a woman. But unlike us, Watson cannot be intimidated. It never gets cocky or discouraged. It plays its game coldly, implacably, always offering a perfectly timed buzz when it's confident about an answer. Jeopardy! devotees know that buzzer skill is crucial—games between humans are more often won by the fastest thumb than the fastest brain. This advantage is only magnified when one of the "thumbs" is an electromagnetic solenoid trigged by a microsecond-precise jolt of current. I knew it would take some lucky breaks to keep up with the computer, since it couldn't be beaten on speed.

During my 2004 Jeopardy! streak, I was accustomed to mowing down players already demoralized at having to play a long-standing winner like me. But against Watson I felt like the underdog, and as a result I started out too aggressively, blowing high-dollar-value questions on the decade in which the first crossword puzzle appeared (the 1910s) and the handicap of Olympic gymnast George Eyser (he was missing his left leg). At the end of the first game, Watson had what seemed like an insurmountable lead of more than $30,000. I tried to keep my chin up, but in the back of mind, I was already thinking about a possible consolation prize: a second-place finish ahead of the show's other human contestant and my quiz-show archrival, undefeated Jeopardy! phenom Brad Rutter.

In the final round, I made up ground against Watson by finding the first "Daily Double" clue, and all three of us began furiously hunting for the second one, which we knew was my only hope for catching Watson. (Daily Doubles aren't distributed randomly across the board; as Watson well knows, they're more likely to be in some places than others.) By process of elimination, I became convinced it was hiding in the "Legal E's" category, and, given a 50-50 chance between two clues, chose the $1200 one. No dice. Watson took control of the board and chose "Legal E's" for $1600. There was the Daily Double. Game over for humanity.

IBM has bragged to the media that Watson's question-answering skills are good for more than annoying Alex Trebek. The company sees a future in which fields like medical diagnosis, business analytics, and tech support are automated by question-answering software like Watson. Just as factory jobs were eliminated in the 20th century by new assembly-line robots, Brad and I were the first knowledge-industry workers put out of work by the new generation of "thinking" machines. "Quiz show contestant" may be the first job made redundant by Watson, but I'm sure it won't be the last.

But there's no shame in losing to silicon, I thought to myself as I greeted the (suddenly friendlier) team of IBM engineers after the match. After all, I don't have 2,880 processor cores and 15 terabytes of reference works at my disposal—nor can I buzz in with perfect timing whenever I know an answer. My puny human brain, just a few bucks worth of water, salts, and proteins, hung in there just fine against a jillion-dollar supercomputer.

"Watching you on Jeopardy! is what inspired the whole project," one IBM engineer told me, consolingly. "And we looked at your games over and over, your style of play. There's a lot of you in Watson." I understood then why the engineers wanted to beat me so badly: To them, I wasn't the good guy, playing for the human race. That was Watson's role, as a symbol and product of human innovation and ingenuity. So my defeat at the hands of a machine has a happy ending, after all. At least until the whole system becomes sentient and figures out the nuclear launch codes. But I figure that's years away.

Also in Slate, Chris Wilson says that the next IBM supercomputer should be able to play poker. Jeremy Singer-Vine crunches the numbers on the hardest clues in Jeopardy! history. Daneil D'Addario writes about a fan-maintained database that helps Jeopardy! contestants prepare. And, lastly, Slate V recounts the untold story of Watson's other game-show appearances.

Ken Jennings won 74 consecutive games of Jeopardy! in 2004. He is the author of Brainiac, Ken Jennings's Trivia Almanac, and the forthcoming Maphead: Charting the Wide, Weird World of Geography Wonks.

[Singular Intellect]

Yeah, but I don't think natural language would help much there. You'd need better image or sound recognition, or some way to input what you want without words (like drawing a picture for the computer or singing part ofthe song to it).

Perhaps understanding analogies would help, but the real limiting factor is still the computer recognizing and describing the picture.

Which shouldn't be too far off, given what I'm reading and understanding about Hierarchical Temporal Memory.

[Singular Intellect]

Eh, don't need learning computers to recognize pictures and songs.

And this arbitrary dismissal of such a goal is based upon what?

Facebook and Youtube both do it to a limited extent already (facebook suggesting pictures to tag and youtube automatically screening material for copyright). Just need to refine it and grow the database, then make a simple UI for it. (uploading a partial clip is the way I'd do it)

Intelligence is really overrated.

I'm finding it quite puzzling (and suspicious) how you're seemingly dismissing the goal of AI as if it's easy or not worth the efforts of the entire community attempting to achieve it.

What justifies this stance on your part?

[Skgoa]

And to add to that: deepQA (aka. "Watson") isn't even anything revolutionary. Its an expert system. Yes, its more advanced and its opendomainedness (sorry, its 4 am over here ) is an advantage, but expert systems have been around for quite a while. And they have been used in medicine for almost as long. They (most probably) aren't the way to general AI, either.

[Singular Intellect]

And this arbitrary dismissal of such a goal is based upon what?

What is Toronto????

No one is claiming Watson is perfect or beyond improvement. The system made obvious mistakes.

(that is, I don't understand the question nor the context. Where is this even coming from? It's a simple statement of fact that intelligence isn't necessary for identifying pictures and songs, seeing how algorithms in use today can do it fairly well. I think you're taking all of this far too personally.)

Not personally by any means. I freely confess to being in Watson's corner, so to speak. However, I'm just inquiring on details regarding your position on the matter. I just tend to favour being direct; it's not my intention to seem confrontational.

I'm finding it quite puzzling (and suspicious) how you're seemingly dismissing the goal of AI as if it's easy or not worth the efforts of the entire community attempting to achieve it.

What justifies this stance on your part?

I just have a different goal. AI people are going for AI. I'm more interested in other results. AI is just one way among many to get there, and not necessarily the best, or even a good, way of getting there. Hence, it being overrated. A lot of AI proponents talk about how it will revolutionize things. I don't see how, especially seeing how well the competition is already doing.

Artificial Intelligence (or non biological intelligence) when created, will presumeably be considered a success when accomplishing the feat of employing human levels of pattern recognition and prediction. Combined with the demostrated and vastly superior capabilities of computer systems, such intelligence will automatically have enormous advantages over it's biological creators.

Watson's demostrated capabilities are impressive as far as I'm concerned, but I'm not under any illusion it's a general AI system or ever intented to be one.

[Singular Intellect]

Combined with the demostrated and vastly superior capabilities of computer systems, such intelligence will automatically have enormous advantages over it's biological creators.

This is handwaving. There's not necessarily any link between superior intelligence and real world advantages.

Humanity's vast domination over all other species says otherwise.

Let's use Watson as an analogy to the real world. If he didn't have that fast thumb to work the buzzer, he couldn't have possibly won Jeopardy!. He could have all the answers instantly, but Ken or Brad would have always rang in first.

Nobody can win if they aren't fast enough to buzz in before their opponents. Watson's speed at buzzing in has no more bearing on the situation than Ken's ability to buzz in faster than his previous human opponents.

If anything, Watson's system is more honest. It won't buzz in until it has sufficient confidence in an answer.

Is buzzing time important to Jeopardy? Absolutely. Is Watson's buzzing speed an issue? No more than Jenning's or Rutter's was for their opponents. Nobody said shit about them and buzzing so far as I know.

Watson played by the same rules, discussion about it's buzzing speed is one giant Red Herring as far as I'm concerned.

The other side is what we saw earlier in the week: Watson had a super fast thumb, so despite its flaws, it won. If we rigged a thumb device like that for my own use*, I'd have a good shot of beating the champions.

* Simple design: I press a button that means "buzz in the microsecond when Alex is done" instead of "Buzz in now".

Your analogy would only work if you were only permitted to buzz in once having determined an answer with sufficiently strong confidence. Humans can cheat that system. Even Jennings admitted that.

This demonstrates two ways intelligence doesn't necessarily matter when it comes to real world results.

Number one is physical limitations. It doesn't matter how smart you are. If you can't make it happen physically, the thoughts never turn into results.

Obviously, but that's beside the point. Any kind of intelligence is useless without a mechanism to enable it to manipulate and interact with the physical world.

Number two is the AI's advantage isn't necessarily tied to its intelligence. Men can build and use machines to expand their own capabilities to match the computer. No super intelligence required for a lot of big advantages. This can be physical - like my buzzer machine, or cars to speed up our running, etc - or mental, like designing algorithms to solve picture recognition.

All examples of which merely highlight the weaknesses and limitations of the biological human form and intelligence. There's zero reason for any AI to be tied down to any one kind of physical body or mechanism. That is a massive advantage, not a disadvantage.

There's a third too. Suppose Watson were a hundred times better than Ken and Brad. He wouldn't get 100x the winnings - there's only so much money available, and even the human's inferior intellect is already adequate. They get (better than) 90% of questions right as it is. The best the computer could possibly do is a 10% increase, since that's all the potential there to be soaked up. If you take this same principle to something like engineering, 100x intelligence might not improve the design at all. It's possible that human engineers have already met the theoretical best. The best the computer could do is cut design costs.

It's a pretty bold statement to suggest human engineering has reached the apex of possibile engineering. Can't say I agree with it until I see substantial justification for that suggestion. Until then, it's nothing more than an appeal to ignorance.

Edit:

edit: re: being in someone's corner: I'll admit I'm a big Ken Jennings fan. I love watching him play and I love watching him win. It's very entertaining to me.

What would your response be to those claiming his victories were in no way impressive simply because he had a fast buzzing speed record?

[Darth Fanboy]

Dude still had to get the questions right, and if simple buzzing was the way to win then it wouldn't be so difficult to win so many consecutive games.

[Skgoa]

Number two is the AI's advantage isn't necessarily tied to its intelligence. Men can build and use machines to expand their own capabilities to match the computer. No super intelligence required for a lot of big advantages. This can be physical - like my buzzer machine, or cars to speed up our running, etc - or mental, like designing algorithms to solve picture recognition.

All examples of which merely highlight the weaknesses and limitations of the biological human form and intelligence. There's zero reason for any AI to be tied down to any one kind of physical body or mechanism. That is a massive advantage, not a disadvantage.

You are missing the point. "Watson" is not intelligent, it won due to having a near perfect buzzing ability and a HUGE database in which it can look up the answers. Yes, its a little bit impressive that the software engineers at IBM have build advanced techniques (that were known in computer science already) into their deepQA framework, but its not an AI like those you are wanking to.

There's a third too. Suppose Watson were a hundred times better than Ken and Brad. He wouldn't get 100x the winnings - there's only so much money available, and even the human's inferior intellect is already adequate. They get (better than) 90% of questions right as it is. The best the computer could possibly do is a 10% increase, since that's all the potential there to be soaked up. If you take this same principle to something like engineering, 100x intelligence might not improve the design at all. It's possible that human engineers have already met the theoretical best. The best the computer could do is cut design costs.

It's a pretty bold statement to suggest human engineering has reached the apex of possibile engineering. Can't say I agree with it until I see substantial justification for that suggestion. Until then, it's nothing more than an appeal to ignorance.

Ok, 1) this is a strawman since he claimed no such thing and 2) he is right, "moar intelligence!!!" isn't the solution to everything.

[Singular Intellect]

Humanity's vast domination over all other species says otherwise.

What a bullshit statement from top to bottom. First off, we aren't really dominating all other species. What metric are you using?

Enhancing our physical and mental capabilities well beyond what natural evolution provided us with, using advanced technology.

Secondly, how much of the advantage we do have (whatever it is) is thanks to intelligence?

Enough of one that you can sit behind your computer in comfort without worrying about basic necessities, while other species are constantly hunting for resources and fighting for survival. No other species uses technology to the degree we do so, and that's the point of comparison I'm talking about with regards to intelligence.

Thirdly, would more intelligence be more of an advantage? Or is there a level where you hit "good enough" and find diminishing returns?

What do you consider intelligence? If your brain had the memory capabilities of a computer (perfect recall), you don't think that would be an enormous advantage? If your brain had the reaction speed of a computer, you wouldn't consider that a massive advantage? If your brain had the ability to transfer data/experience to another human brain at the speed computers can do so, wouldn't you consider that an incredible advantage?

All examples of which merely highlight the weaknesses and limitations of the biological human form and intelligence. There's zero reason for any AI to be tied down to any one kind of physical body or mechanism. That is a massive advantage, not a disadvantage.

First off, factual correction: I never said it was a disadvantage. I said it isn't much of an advantage, since we can do the same thing.

No, we can't. We can plop our physical bodies inside of machines (ie: cars), and add tools to our physical bodies (shovel). As of yet we cannot seperate our intelligence producing system (brain) from our biological bodies.

Secondly, we aren't tied down to any one kind of physical body or mechanism either, which was my entire point.

Prove it. Demostrate you can seperate your physical brain from your body. You are as trapped in the human form as anyone else is currently. All our technology we interact with is currently designed to deal with this huge limitation.

We use tools and machines to do things well beyond our biological capabilities right now. It might have taken intelligence to design the first car, but anybody can operate one; geniuses don't have an advantage over other people when it comes to driving.

Define 'geniuses' and 'advantages'. It would be ridiculously easy to demostrate one driver being superior to another via intelligence (like plotting routes or evading obstacles), so you're going to have to more specifically describe what you're talking about here. Because I cannot consider you're seriously suggesting all drivers are equal.

What would your response be to those claiming his victories were in no way impressive simply because he had a fast buzzing speed record?

You'd be wrong. Humans don't typically have precise timing, so him being able to do this sets him apart from other humans, and is thus impressive. Computers and other machines, on the other hand, do have such capabilities on a regular basis.

Ergo, why I claim Watson's buzzing speed is of no interest and a Red Herring regarding what impressive abilities Watson is demostrating. In this case, reading English sentences, understanding and answering them with remarkable consistency.

A car that can go 30 miles per hour? Not impressive. They all do that.

A human who can run 30 mph? Very impressive. Most of us can't.

Thus you handily demostrate what I percieve to make Watson impressive; it doing that which other computers cannot yet do.

A major difference is Watson's 'intelligence' can be transferred to any compatible computer hardware out there, it will simply run slower or faster depending upon the hardware configuration. Hence IBM's explanation for the large computer server setup. They directly stated nothing would stop a home PC rom running Watson, you'd merely have to wait hours for it to answer a question. Yet another incredible advantage humans do not have. (Watson is not tied to any one physical body)

I suspect you may be making an argument for 'unique' instead of 'impressive'. What Watson demostrates is impressive and currently unique. Remove the latter attribute and I still consider the feat impressive, but it will become harder to claim so if numerous machines (or all) are doing it.

A good example would be Deep Blue. A chess program that beat the world's best human player. At the time impressive, but now because all computers can do that, it's no longer 'impressive'. Even though the only thing that has changed is the number of machines doing it and now doing it much better.

[Thanas]

Looks like all the crowing about how awesome watson is can stop now. Congressman beats watson at Jeopardy.

[phongn]

Looks like all the crowing about how awesome watson is can stop now. Congressman beats watson at Jeopardy.

Watson's awesomeness is independent of its Jeopardy playing skills?

[Singular Intellect]

I'd be more impressed if Watson had been defeated in a full game of Jeopardy, not in just one round by one opponent in a game Watson won anyway.

The humans are starting to get desperate!

"Ah ha! Watson lost the first round!"
"Who won the overall game?"
"Uh...in other news..."