IBM takes on Jeopardy

GEC: Discuss gaming, computers and electronics and venture into the bizarre world of STGODs.

Moderator: Thanas

darkjedi521
Youngling
Posts: 108
Joined: 2006-10-13 03:14pm
Location: Troy, NY

IBM takes on Jeopardy

Post by darkjedi521 »

So, did anyone else catch IBM's attempt at building a super computer to win at jeopardy? The right answers were not nearly as interesting as the wrong answers. I was able to attend a panel with one of the designers of the system before hand. They have some very ambitious goals. The system is very semantics based. It did much better with puns and song lyrics than it did on questions involving literary figures or broad answers. A very impressive attempt, given the speed at which it would more often than not come up with the right answer.
Spoiler
There was only one instance where I remember the computer giving a wrong answer, and no one got that question right. One of the designers commented to us that they did not bother to include a system to hear the other contestant's responses and eliminate that option of the other contestant got it wrong because they felt the probability of it happening was too low. Well, it happened. Ken answered wrongly, and then Watson repeated the same wrong answer.

Most of the questions that it didn't answer, it was less than 50% confident of its best answer.
Spoiler
At the end of the first round, Brad and Watson were tied, with Ken a very distant 3rd
Tomorrow night there's going to be a discussion here about the whole system works which should be interesting and I'll post highlights after. The panels are at Rensselaer Polytechnic Institute where I work and I have no affiliation with IBM or any of the people involved in the project. Most of the key IBM movers and shakers are alums and that's how we ended up having Q and A panel sessions with them about the system.

IBM's Site
Ex ASVS lurker and sometimes poster
User avatar
Vehrec
Jedi Council Member
Posts: 2204
Joined: 2006-04-22 12:29pm
Location: The Ohio State University
Contact:

Re: IBM takes on Jeopardy

Post by Vehrec »

NOVA did a special on this last week-I'm very keyed up for this. Watson is very well informed, but he's also prone to ringing in and saying some dumb things. Given that part of it's database is Wikipedia, maybe his information isn't the most accurate.

Given that it has a massive database of song lyrics to search, it's little wonder that it got the Beatles questions mostly right.
ImageCommander of the MFS Darwinian Selection Method (sexual)
User avatar
Sarevok
The Fearless One
Posts: 10681
Joined: 2002-12-24 07:29am
Location: The Covenants last and final line of defense

Re: IBM takes on Jeopardy

Post by Sarevok »

Of course computers are fast. But why are you dismissing the fact it can answer correctly at all ? This level of accurate search capability is amazing. I won't be surprised if Google invests in Watson derived applications because it could revolutionize the search engine business.
I have to tell you something everything I wrote above is a lie.
User avatar
Sarevok
The Fearless One
Posts: 10681
Joined: 2002-12-24 07:29am
Location: The Covenants last and final line of defense

Re: IBM takes on Jeopardy

Post by Sarevok »

Okay.

So coming from the IT industry whats your take on commercial applications of this ?
I have to tell you something everything I wrote above is a lie.
User avatar
General Zod
Never Shuts Up
Posts: 29211
Joined: 2003-11-18 03:08pm
Location: The Clearance Rack
Contact:

Re: IBM takes on Jeopardy

Post by General Zod »

Destructionator XIII wrote:It's not really my area of expertise.

With that in mind though, I'm not really impressed. In my experience, the best search engine results are a literal match of what I wrote - the computer shouldn't try to understand, it should just search. The internet is so big now that the odds are good that someone else has already asked my exact question, so I can get human results from a stupid computer with a big enough database.

Watson is interesting from a theory perspective, but from a practical standpoint, I don't think it will make a difference.


The best thing about the computer is what it got wrong... if it knew the definition of words like "decade", a dead stupid algorithm would have gotten one of its wrong answers correct. The hard part has got to be category definitions. (Hell, I used to never understand Before & After on Wheel of Fortune. Made it impossible. Now that I get what it's asking for though, they are pretty easy.)
Without a deeper understanding of how Watson works, I'd wager it's because searching is slower or impractical to some extent. Subtle differences in the question can change the answers to the point that plain text queries just aren't enough to tell the difference. Searches aren't going to be quite as helpful for analogies, as the most obvious example.
"It's you Americans. There's something about nipples you hate. If this were Germany, we'd be romping around naked on the stage here."
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

Of course Watson is faster at reading the questions and likely more precise with buzzing in (it should be noted Watson does have a physical buzzer it must activate just like the players).

Don't see that as relevant though. Last time I checked, reading speed and thumb muscles weren't investigated prior to putting a contestant on the show. The game is about one's knowledge base, understanding a question and answering it correctly. A computer beating humans with response time is a foregone conclusion, excluding the time it takes the computer to determine possible answers.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
Thanas
Magister
Magister
Posts: 30779
Joined: 2004-06-26 07:49pm

Re: IBM takes on Jeopardy

Post by Thanas »

Singular Intellect wrote:Of course Watson is faster at reading the questions and likely more precise with buzzing in (it should be noted Watson does have a physical buzzer it must activate just like the players).

Don't see that as relevant though. Last time I checked, reading speed and thumb muscles weren't investigated prior to putting a contestant on the show.
Actually, that is an unfair edge. If they wanted it to be fair, they would look at the response time of the best other contestants and then enforce a randomly selected buzz time on part of the computer.
Whoever says "education does not matter" can try ignorance
------------
A decision must be made in the life of every nation at the very moment when the grasp of the enemy is at its throat. Then, it seems that the only way to survive is to use the means of the enemy, to rest survival upon what is expedient, to look the other way. Well, the answer to that is 'survival as what'? A country isn't a rock. It's not an extension of one's self. It's what it stands for. It's what it stands for when standing for something is the most difficult! - Chief Judge Haywood
------------
My LPs
User avatar
General Zod
Never Shuts Up
Posts: 29211
Joined: 2003-11-18 03:08pm
Location: The Clearance Rack
Contact:

Re: IBM takes on Jeopardy

Post by General Zod »

The buzzer was indeed accounted for.

http://arstechnica.com/media/news/2011/ ... trebek.ars
First, a quick rundown of how Watson plays Jeopardy. The computer is fed the answer in text form at the same time the answer panel appears to the two human players. Watson then queries its database for an appropriate question response, a process that doesn't involve using the Internet at all. Welty noted that game shows are federally regulated and there were two auditors present while the episode was filmed to make sure the computer wasn't querying the Internet for answers.

Watson then must push a physical buzzer to answer questions, just like its human competitors. While this would seem to be a task at which computers would have an overwhelming advantage, Welty noted that Rutter was so well-known for his lightning fast buzzing that the producers weren't even mildly concerned.

When the match began, the computer got off to a strong start: it took control of the board away from Rutter on the second turn, immediately nailed a Daily Double square, bet $1,000, and got the question right. But later, on a Name That Decade question, Jennings answered incorrectly with "what is the 1920s?" Watson, which can't see or hear and so can't pick up on the follies of its competitors, followed Jennings' answer with its own: "What is the 1920s?"

"No, Ken said that," Alex Trebek replied as the avatar's sphere turned orange with embarrassment.

During a commercial after Watson's decade gaffe, Welty noted that the team thought the ability to process other players' wrong answers would be unnecessary. "We just didn't think it would ever happen," Welty said, laughing.
"It's you Americans. There's something about nipples you hate. If this were Germany, we'd be romping around naked on the stage here."
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

Thanas wrote:
Singular Intellect wrote:Of course Watson is faster at reading the questions and likely more precise with buzzing in (it should be noted Watson does have a physical buzzer it must activate just like the players).

Don't see that as relevant though. Last time I checked, reading speed and thumb muscles weren't investigated prior to putting a contestant on the show.
Actually, that is an unfair edge. If they wanted it to be fair, they would look at the response time of the best other contestants and then enforce a randomly selected buzz time on part of the computer.
How's that fair? No other contestant is ever handicapped regarding their speed for using the buzzer. Watson had to use a physical buzzer like both others, and wouldn't buzz in until it had confidence in an answer it came up with.

As long as it plays by the same rules, everything is plenty fair. Look at this very website; a gross mismatch of two opponents, but I don't see anyone (reasonably) claiming you must handicap one or the other for a fair contest. They need only play by the same rules.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
Thanas
Magister
Magister
Posts: 30779
Joined: 2004-06-26 07:49pm

Re: IBM takes on Jeopardy

Post by Thanas »

Singular Intellect wrote:How's that fair? No other contestant is ever handicapped regarding their speed for using the buzzer. Watson had to use a physical buzzer like both others, and wouldn't buzz in until it had confidence in an answer it came up with.
Because its physical capacities make comparing the mental capacities useless.
Whoever says "education does not matter" can try ignorance
------------
A decision must be made in the life of every nation at the very moment when the grasp of the enemy is at its throat. Then, it seems that the only way to survive is to use the means of the enemy, to rest survival upon what is expedient, to look the other way. Well, the answer to that is 'survival as what'? A country isn't a rock. It's not an extension of one's self. It's what it stands for. It's what it stands for when standing for something is the most difficult! - Chief Judge Haywood
------------
My LPs
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

Thanas wrote:
Singular Intellect wrote:How's that fair? No other contestant is ever handicapped regarding their speed for using the buzzer. Watson had to use a physical buzzer like both others, and wouldn't buzz in until it had confidence in an answer it came up with.
Because its physical capacities make comparing the mental capacities useless.
If regular Jeopardy! contestants never had to worry about their opponents buzzing speed, you might have a point.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
Thanas
Magister
Magister
Posts: 30779
Joined: 2004-06-26 07:49pm

Re: IBM takes on Jeopardy

Post by Thanas »

If Watson would not be much faster than a Human, you might have one too.
Whoever says "education does not matter" can try ignorance
------------
A decision must be made in the life of every nation at the very moment when the grasp of the enemy is at its throat. Then, it seems that the only way to survive is to use the means of the enemy, to rest survival upon what is expedient, to look the other way. Well, the answer to that is 'survival as what'? A country isn't a rock. It's not an extension of one's self. It's what it stands for. It's what it stands for when standing for something is the most difficult! - Chief Judge Haywood
------------
My LPs
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

Thanas wrote:If Watson would not be much faster than a Human, you might have one too.
Except both Jennings and Rutter managed to buzz in plenty of times, including times Watson had an answer to give. So it's speed advantage does not seem to exist to the degree you're giving it credit for.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
Thanas
Magister
Magister
Posts: 30779
Joined: 2004-06-26 07:49pm

Re: IBM takes on Jeopardy

Post by Thanas »

It won because of its speed. Are you denying that?
Whoever says "education does not matter" can try ignorance
------------
A decision must be made in the life of every nation at the very moment when the grasp of the enemy is at its throat. Then, it seems that the only way to survive is to use the means of the enemy, to rest survival upon what is expedient, to look the other way. Well, the answer to that is 'survival as what'? A country isn't a rock. It's not an extension of one's self. It's what it stands for. It's what it stands for when standing for something is the most difficult! - Chief Judge Haywood
------------
My LPs
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

Thanas wrote:It won because of its speed. Are you denying that?
Absolutely. Watson could've buzzed at speed measured at the planck scale; doesn't mean shit if it doesn't understand the question, provides the wrong answer or decides to not buzz at all because of lack of confidence. All three traits which it has demostrated.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
adam_grif
Sith Devotee
Posts: 2755
Joined: 2009-12-19 08:27am
Location: Tasmania, Australia

Re: IBM takes on Jeopardy

Post by adam_grif »

They should have programmed it with Sean Connery's voice.

Whaddaya think of that, Trebek?
A scientist once gave a public lecture on astronomy. He described how the Earth orbits around the sun and how the sun, in turn, orbits around the centre of a vast collection of stars called our galaxy.

At the end of the lecture, a little old lady at the back of the room got up and said: 'What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise.

The scientist gave a superior smile before replying, 'What is the tortoise standing on?'

'You're very clever, young man, very clever,' said the old lady. 'But it's turtles all the way down.'
darkjedi521
Youngling
Posts: 108
Joined: 2006-10-13 03:14pm
Location: Troy, NY

Re: IBM takes on Jeopardy

Post by darkjedi521 »

For the daily doubles, it uses an algorithm based on game theory and its past performance in the category.

Although not show, the confidence it had in tonight's final jeopardy answer was only 13% with the correct answer at 11%.

The panel I attended last night can be viewed here (requires Silverlight) and tonight's discussion/q&a will eventually be posted at http://watson.rpi.edu Tonight was a bit more detail of how Watson makes its decisions and chooses its wagers.

As for the final question, it choose the city in its databanks with the most WWII references and a major airport named after someone who fought in WWII, it didn't handle the compound sentence well at all and it couldn't tell from the clue what type of noun was being sought.
Ex ASVS lurker and sometimes poster
darkjedi521
Youngling
Posts: 108
Joined: 2006-10-13 03:14pm
Location: Troy, NY

Re: IBM takes on Jeopardy

Post by darkjedi521 »

IBM said that they programmed it to not 100% trust the category since sometimes the category doesn't refer to its literal meaning.
Ex ASVS lurker and sometimes poster
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

I was delighted to see Watson win the game. I shamelessly admit I was cheering 100% for the computer contestant; a human winner would have been absolutely boring and disappointing.
Destructionator XIII wrote:If anyone else is watching the game now, you can see what I'm talking about with the speed and Ken's face. Over and over again, he shakes his head while frantically pressing the button, but the computer has already buzzed in.
And the computer buzzing in before someone is more unfair than another human contestant buzzing in first because...why? It's faster? Duh. So is the human who buzzes in first over another human.
Brad has a better poker face, but I'm sure he's in exactly the same situation.

They both beat the computer's timing a couple times, but, by far, the computer is winning because of its speed. As to knowing the answers, they are at least equal.
In other words, winning via a trait of innate superiority. That's as revealing as claiming the winner in a race won because they ran faster.
AHGAHAHAHAHAAHA Final Jeopardy is what I like to see! The stupid fucking computer didn't have even the first clue; it's answer was completely unreasonable. No human would have even guessed that.
Watson still kicked their asses in the end. Pointing out Watson has issues and isn't perfect by any means is hardly doing the human side any favours here.

"haha stupid computer!"
"...that computer beat the two world champions. If it's stupid, what does that make our champions?"
"..."
But why the fuck is this even played for money? That'd make me feel even more cheated if I was playing. Maybe they needed some incentive for the champs to come back.
The winnings are given to charity. In IBM's case, 100% to charity, in the other two cases, 50% to charity. Watson is even more generous than humans to boot. ;) :P
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

Destructionator XIII wrote: I'm not commenting on fairness.
So what are you commenting on then? That the computer has a consistent and fast response time? This shouldn't exactly be surprising or particularily note worthy.
Watson still kicked their asses in the end.
The game's only half over!
Am I missing something? I haven't watched the entire event yet (can't find it), but according to what I've read, it won.
"...that computer beat the two world champions. If it's stupid, what does that make our champions?"
It's a bullshit metric to say anything about smartness for the main game. Precise timing is the deciding factor.
Jeopardy! isn't a game about smarts, its about fact recollection. That's not the impressive aspect of Watson's participation.

Precise timing means shit if you don't understand the question, provide the wrong answer or decide against answering at all.
Yeah, it's an advertising gimmick so we shouldn't expect good science, but it's very important that people realize this to get a realistic appraisal of the machine. It doesn't win because it is smarter. It wins because its reaction time is faster - nothing special in a machine.
This isn't a demostration of Watson's reaction speed.
Though, the fact that it has a fighting chance at all is moderately impressive. It playing the game is pretty cool. It winning is pretty mundane.
Only if, as you seemingly do, subscribe to the notion Watson's reaction speed is of any interest in the scenario whatsoever.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
phongn
Rebel Leader
Posts: 18487
Joined: 2002-07-03 11:11pm

Re: IBM takes on Jeopardy

Post by phongn »

Ken Jennings has a Q&A session up on the Wapo here
User avatar
Sriad
Sith Devotee
Posts: 3028
Joined: 2002-12-02 09:59pm
Location: Colorado

Re: IBM takes on Jeopardy

Post by Sriad »

Singular Intellect wrote:
Though, the fact that it has a fighting chance at all is moderately impressive. It playing the game is pretty cool. It winning is pretty mundane.
Only if, as you seemingly do, subscribe to the notion Watson's reaction speed is of any interest in the scenario whatsoever.
Reaction speed being important is an absolute mathematical fact.

Suppose 2 contestants: 1 is right 100% of the time but only buzzes first 25%; 2 is right only 80% of the time but buzzes first 75%.

After an arbitrary number of questions, the standings will be:
Contestant 1: 25% (from buzzing first) + 15% (from the 20% of questions 2 answered incorrectly) = 40%
Contestant 2: 60% (buzzing first and answering correctly) - 15% (buzzing first, incorrectly) = 45%

The result is much more pronounced if you take into account the fact that Watson only buzzes above a certain certainty threshold (reducing penalties), and that there are two opponents splitting its misfires.
User avatar
Singular Intellect
Jedi Council Member
Posts: 2392
Joined: 2006-09-19 03:12pm
Location: Calgary, Alberta, Canada

Re: IBM takes on Jeopardy

Post by Singular Intellect »

Sriad wrote:Reaction speed being important is an absolute mathematical fact.
I never said reaction speed wasn't important. I said it wasn't of any significant interest. Anybody knows a computer can beat a human in reaction speed.

Again, that is not what was being demostrated with Watson.

To quote Ken Jennings himself:
As Jeopardy devotees know, if you're trying to win on the show, the buzzer is all. On any given night, nearly all the contestants know nearly all the answers, so it's just a matter of who masters buzzer rhythm the best.

Watson does have a big advantage in this regard, since it can knock out a microsecond-precise buzz every single time with little or no variation. Human reflexes can't compete with computer circuits in this regard. But I wouldn't call this unfair...precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance.
"Now let us be clear, my friends. The fruits of our science that you receive and the many millions of benefits that justify them, are a gift. Be grateful. Or be silent." -Modified Quote
User avatar
Superboy
Padawan Learner
Posts: 294
Joined: 2005-01-21 09:09pm

Re: IBM takes on Jeopardy

Post by Superboy »

The fact that buzz-in time is so important makes Jeopardy an odd choice to show off it's capabilities. It's impressive as all hell that it can answer those kinds of questions, but it's not at all impressive that it can buzz in faster than a human. Given that, it's odd that they choose to showcase it on a show where buzzing in fast is such a huge factor in victory. If it was in a competition that involved a series of final-jeopardy-like questions where buzz-in time wasn't a factor, and it could still beat a human, THAT would be truly impressive.

In other words, it's impressive that it can answer the questions on jeopardy. It's not very impressive that it was able to get a higher score than a human on jeopardy.
User avatar
Vehrec
Jedi Council Member
Posts: 2204
Joined: 2006-04-22 12:29pm
Location: The Ohio State University
Contact:

Re: IBM takes on Jeopardy

Post by Vehrec »

Look, mechanically precise buzz-in times are a huge advantage, yes. But how would you correct that? Insert a delay of 200 milliseconds after Alex reads the question? That's standard human response time-but it's also how long that you can be locked out for buzzing in early in Jeopardy. And somehow, I doubt that top level champs at this game waste a fifth of a second buzzing in. A shorter delay? How much shorter? Give the computer the question only when the announcer is finished reading it? How long does it take to get these answers? There is a huge volume of data to chew through, Watson might never buzz in.

I do agree that a good showing in Final Jeopardy is more important than a lot of other things, but so is Watson's impressive uses of Daily Doubles-it's actively hunting for them on the board because there's a non-random distribution. And it got a lot of mileage out of them. And you just have to smile a bit at his non-rounded wagers.
ImageCommander of the MFS Darwinian Selection Method (sexual)
Post Reply