Page 1 of 3

IBM takes on Jeopardy

Posted: 2011-02-14 09:29pm
by darkjedi521
So, did anyone else catch IBM's attempt at building a super computer to win at jeopardy? The right answers were not nearly as interesting as the wrong answers. I was able to attend a panel with one of the designers of the system before hand. They have some very ambitious goals. The system is very semantics based. It did much better with puns and song lyrics than it did on questions involving literary figures or broad answers. A very impressive attempt, given the speed at which it would more often than not come up with the right answer.
Spoiler
There was only one instance where I remember the computer giving a wrong answer, and no one got that question right. One of the designers commented to us that they did not bother to include a system to hear the other contestant's responses and eliminate that option of the other contestant got it wrong because they felt the probability of it happening was too low. Well, it happened. Ken answered wrongly, and then Watson repeated the same wrong answer.

Most of the questions that it didn't answer, it was less than 50% confident of its best answer.
Spoiler
At the end of the first round, Brad and Watson were tied, with Ken a very distant 3rd
Tomorrow night there's going to be a discussion here about the whole system works which should be interesting and I'll post highlights after. The panels are at Rensselaer Polytechnic Institute where I work and I have no affiliation with IBM or any of the people involved in the project. Most of the key IBM movers and shakers are alums and that's how we ended up having Q and A panel sessions with them about the system.

IBM's Site

Re: IBM takes on Jeopardy

Posted: 2011-02-14 10:17pm
by Vehrec
NOVA did a special on this last week-I'm very keyed up for this. Watson is very well informed, but he's also prone to ringing in and saying some dumb things. Given that part of it's database is Wikipedia, maybe his information isn't the most accurate.

Given that it has a massive database of song lyrics to search, it's little wonder that it got the Beatles questions mostly right.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 10:01am
by Sarevok
Of course computers are fast. But why are you dismissing the fact it can answer correctly at all ? This level of accurate search capability is amazing. I won't be surprised if Google invests in Watson derived applications because it could revolutionize the search engine business.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 10:25am
by Sarevok
Okay.

So coming from the IT industry whats your take on commercial applications of this ?

Re: IBM takes on Jeopardy

Posted: 2011-02-15 11:47am
by General Zod
Destructionator XIII wrote:It's not really my area of expertise.

With that in mind though, I'm not really impressed. In my experience, the best search engine results are a literal match of what I wrote - the computer shouldn't try to understand, it should just search. The internet is so big now that the odds are good that someone else has already asked my exact question, so I can get human results from a stupid computer with a big enough database.

Watson is interesting from a theory perspective, but from a practical standpoint, I don't think it will make a difference.


The best thing about the computer is what it got wrong... if it knew the definition of words like "decade", a dead stupid algorithm would have gotten one of its wrong answers correct. The hard part has got to be category definitions. (Hell, I used to never understand Before & After on Wheel of Fortune. Made it impossible. Now that I get what it's asking for though, they are pretty easy.)
Without a deeper understanding of how Watson works, I'd wager it's because searching is slower or impractical to some extent. Subtle differences in the question can change the answers to the point that plain text queries just aren't enough to tell the difference. Searches aren't going to be quite as helpful for analogies, as the most obvious example.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 04:19pm
by Singular Intellect
Of course Watson is faster at reading the questions and likely more precise with buzzing in (it should be noted Watson does have a physical buzzer it must activate just like the players).

Don't see that as relevant though. Last time I checked, reading speed and thumb muscles weren't investigated prior to putting a contestant on the show. The game is about one's knowledge base, understanding a question and answering it correctly. A computer beating humans with response time is a foregone conclusion, excluding the time it takes the computer to determine possible answers.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 04:32pm
by Thanas
Singular Intellect wrote:Of course Watson is faster at reading the questions and likely more precise with buzzing in (it should be noted Watson does have a physical buzzer it must activate just like the players).

Don't see that as relevant though. Last time I checked, reading speed and thumb muscles weren't investigated prior to putting a contestant on the show.
Actually, that is an unfair edge. If they wanted it to be fair, they would look at the response time of the best other contestants and then enforce a randomly selected buzz time on part of the computer.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 04:37pm
by General Zod
The buzzer was indeed accounted for.

http://arstechnica.com/media/news/2011/ ... trebek.ars
First, a quick rundown of how Watson plays Jeopardy. The computer is fed the answer in text form at the same time the answer panel appears to the two human players. Watson then queries its database for an appropriate question response, a process that doesn't involve using the Internet at all. Welty noted that game shows are federally regulated and there were two auditors present while the episode was filmed to make sure the computer wasn't querying the Internet for answers.

Watson then must push a physical buzzer to answer questions, just like its human competitors. While this would seem to be a task at which computers would have an overwhelming advantage, Welty noted that Rutter was so well-known for his lightning fast buzzing that the producers weren't even mildly concerned.

When the match began, the computer got off to a strong start: it took control of the board away from Rutter on the second turn, immediately nailed a Daily Double square, bet $1,000, and got the question right. But later, on a Name That Decade question, Jennings answered incorrectly with "what is the 1920s?" Watson, which can't see or hear and so can't pick up on the follies of its competitors, followed Jennings' answer with its own: "What is the 1920s?"

"No, Ken said that," Alex Trebek replied as the avatar's sphere turned orange with embarrassment.

During a commercial after Watson's decade gaffe, Welty noted that the team thought the ability to process other players' wrong answers would be unnecessary. "We just didn't think it would ever happen," Welty said, laughing.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 04:46pm
by Singular Intellect
Thanas wrote:
Singular Intellect wrote:Of course Watson is faster at reading the questions and likely more precise with buzzing in (it should be noted Watson does have a physical buzzer it must activate just like the players).

Don't see that as relevant though. Last time I checked, reading speed and thumb muscles weren't investigated prior to putting a contestant on the show.
Actually, that is an unfair edge. If they wanted it to be fair, they would look at the response time of the best other contestants and then enforce a randomly selected buzz time on part of the computer.
How's that fair? No other contestant is ever handicapped regarding their speed for using the buzzer. Watson had to use a physical buzzer like both others, and wouldn't buzz in until it had confidence in an answer it came up with.

As long as it plays by the same rules, everything is plenty fair. Look at this very website; a gross mismatch of two opponents, but I don't see anyone (reasonably) claiming you must handicap one or the other for a fair contest. They need only play by the same rules.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 05:13pm
by Thanas
Singular Intellect wrote:How's that fair? No other contestant is ever handicapped regarding their speed for using the buzzer. Watson had to use a physical buzzer like both others, and wouldn't buzz in until it had confidence in an answer it came up with.
Because its physical capacities make comparing the mental capacities useless.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 05:29pm
by Singular Intellect
Thanas wrote:
Singular Intellect wrote:How's that fair? No other contestant is ever handicapped regarding their speed for using the buzzer. Watson had to use a physical buzzer like both others, and wouldn't buzz in until it had confidence in an answer it came up with.
Because its physical capacities make comparing the mental capacities useless.
If regular Jeopardy! contestants never had to worry about their opponents buzzing speed, you might have a point.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 05:47pm
by Thanas
If Watson would not be much faster than a Human, you might have one too.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 05:50pm
by Singular Intellect
Thanas wrote:If Watson would not be much faster than a Human, you might have one too.
Except both Jennings and Rutter managed to buzz in plenty of times, including times Watson had an answer to give. So it's speed advantage does not seem to exist to the degree you're giving it credit for.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 05:54pm
by Thanas
It won because of its speed. Are you denying that?

Re: IBM takes on Jeopardy

Posted: 2011-02-15 06:03pm
by Singular Intellect
Thanas wrote:It won because of its speed. Are you denying that?
Absolutely. Watson could've buzzed at speed measured at the planck scale; doesn't mean shit if it doesn't understand the question, provides the wrong answer or decides to not buzz at all because of lack of confidence. All three traits which it has demostrated.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 09:09pm
by adam_grif
They should have programmed it with Sean Connery's voice.

Whaddaya think of that, Trebek?

Re: IBM takes on Jeopardy

Posted: 2011-02-15 09:18pm
by darkjedi521
For the daily doubles, it uses an algorithm based on game theory and its past performance in the category.

Although not show, the confidence it had in tonight's final jeopardy answer was only 13% with the correct answer at 11%.

The panel I attended last night can be viewed here (requires Silverlight) and tonight's discussion/q&a will eventually be posted at http://watson.rpi.edu Tonight was a bit more detail of how Watson makes its decisions and chooses its wagers.

As for the final question, it choose the city in its databanks with the most WWII references and a major airport named after someone who fought in WWII, it didn't handle the compound sentence well at all and it couldn't tell from the clue what type of noun was being sought.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 10:47pm
by darkjedi521
IBM said that they programmed it to not 100% trust the category since sometimes the category doesn't refer to its literal meaning.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 11:02pm
by Singular Intellect
I was delighted to see Watson win the game. I shamelessly admit I was cheering 100% for the computer contestant; a human winner would have been absolutely boring and disappointing.
Destructionator XIII wrote:If anyone else is watching the game now, you can see what I'm talking about with the speed and Ken's face. Over and over again, he shakes his head while frantically pressing the button, but the computer has already buzzed in.
And the computer buzzing in before someone is more unfair than another human contestant buzzing in first because...why? It's faster? Duh. So is the human who buzzes in first over another human.
Brad has a better poker face, but I'm sure he's in exactly the same situation.

They both beat the computer's timing a couple times, but, by far, the computer is winning because of its speed. As to knowing the answers, they are at least equal.
In other words, winning via a trait of innate superiority. That's as revealing as claiming the winner in a race won because they ran faster.
AHGAHAHAHAHAAHA Final Jeopardy is what I like to see! The stupid fucking computer didn't have even the first clue; it's answer was completely unreasonable. No human would have even guessed that.
Watson still kicked their asses in the end. Pointing out Watson has issues and isn't perfect by any means is hardly doing the human side any favours here.

"haha stupid computer!"
"...that computer beat the two world champions. If it's stupid, what does that make our champions?"
"..."
But why the fuck is this even played for money? That'd make me feel even more cheated if I was playing. Maybe they needed some incentive for the champs to come back.
The winnings are given to charity. In IBM's case, 100% to charity, in the other two cases, 50% to charity. Watson is even more generous than humans to boot. ;) :P

Re: IBM takes on Jeopardy

Posted: 2011-02-15 11:30pm
by Singular Intellect
Destructionator XIII wrote: I'm not commenting on fairness.
So what are you commenting on then? That the computer has a consistent and fast response time? This shouldn't exactly be surprising or particularily note worthy.
Watson still kicked their asses in the end.
The game's only half over!
Am I missing something? I haven't watched the entire event yet (can't find it), but according to what I've read, it won.
"...that computer beat the two world champions. If it's stupid, what does that make our champions?"
It's a bullshit metric to say anything about smartness for the main game. Precise timing is the deciding factor.
Jeopardy! isn't a game about smarts, its about fact recollection. That's not the impressive aspect of Watson's participation.

Precise timing means shit if you don't understand the question, provide the wrong answer or decide against answering at all.
Yeah, it's an advertising gimmick so we shouldn't expect good science, but it's very important that people realize this to get a realistic appraisal of the machine. It doesn't win because it is smarter. It wins because its reaction time is faster - nothing special in a machine.
This isn't a demostration of Watson's reaction speed.
Though, the fact that it has a fighting chance at all is moderately impressive. It playing the game is pretty cool. It winning is pretty mundane.
Only if, as you seemingly do, subscribe to the notion Watson's reaction speed is of any interest in the scenario whatsoever.

Re: IBM takes on Jeopardy

Posted: 2011-02-15 11:41pm
by phongn
Ken Jennings has a Q&A session up on the Wapo here

Re: IBM takes on Jeopardy

Posted: 2011-02-16 01:18am
by Sriad
Singular Intellect wrote:
Though, the fact that it has a fighting chance at all is moderately impressive. It playing the game is pretty cool. It winning is pretty mundane.
Only if, as you seemingly do, subscribe to the notion Watson's reaction speed is of any interest in the scenario whatsoever.
Reaction speed being important is an absolute mathematical fact.

Suppose 2 contestants: 1 is right 100% of the time but only buzzes first 25%; 2 is right only 80% of the time but buzzes first 75%.

After an arbitrary number of questions, the standings will be:
Contestant 1: 25% (from buzzing first) + 15% (from the 20% of questions 2 answered incorrectly) = 40%
Contestant 2: 60% (buzzing first and answering correctly) - 15% (buzzing first, incorrectly) = 45%

The result is much more pronounced if you take into account the fact that Watson only buzzes above a certain certainty threshold (reducing penalties), and that there are two opponents splitting its misfires.

Re: IBM takes on Jeopardy

Posted: 2011-02-16 07:07am
by Singular Intellect
Sriad wrote:Reaction speed being important is an absolute mathematical fact.
I never said reaction speed wasn't important. I said it wasn't of any significant interest. Anybody knows a computer can beat a human in reaction speed.

Again, that is not what was being demostrated with Watson.

To quote Ken Jennings himself:
As Jeopardy devotees know, if you're trying to win on the show, the buzzer is all. On any given night, nearly all the contestants know nearly all the answers, so it's just a matter of who masters buzzer rhythm the best.

Watson does have a big advantage in this regard, since it can knock out a microsecond-precise buzz every single time with little or no variation. Human reflexes can't compete with computer circuits in this regard. But I wouldn't call this unfair...precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance.

Re: IBM takes on Jeopardy

Posted: 2011-02-16 11:33am
by Superboy
The fact that buzz-in time is so important makes Jeopardy an odd choice to show off it's capabilities. It's impressive as all hell that it can answer those kinds of questions, but it's not at all impressive that it can buzz in faster than a human. Given that, it's odd that they choose to showcase it on a show where buzzing in fast is such a huge factor in victory. If it was in a competition that involved a series of final-jeopardy-like questions where buzz-in time wasn't a factor, and it could still beat a human, THAT would be truly impressive.

In other words, it's impressive that it can answer the questions on jeopardy. It's not very impressive that it was able to get a higher score than a human on jeopardy.

Re: IBM takes on Jeopardy

Posted: 2011-02-16 11:47am
by Vehrec
Look, mechanically precise buzz-in times are a huge advantage, yes. But how would you correct that? Insert a delay of 200 milliseconds after Alex reads the question? That's standard human response time-but it's also how long that you can be locked out for buzzing in early in Jeopardy. And somehow, I doubt that top level champs at this game waste a fifth of a second buzzing in. A shorter delay? How much shorter? Give the computer the question only when the announcer is finished reading it? How long does it take to get these answers? There is a huge volume of data to chew through, Watson might never buzz in.

I do agree that a good showing in Final Jeopardy is more important than a lot of other things, but so is Watson's impressive uses of Daily Doubles-it's actively hunting for them on the board because there's a non-random distribution. And it got a lot of mileage out of them. And you just have to smile a bit at his non-rounded wagers.