NOVA - Jeopardy Watson REMINDER it's on tonight, tuesday and wednesday

[Pheran]

I agree with what others are saying about the final Jeopardy question; he didn't even stay within the category. Odd. Small coding mistake, I guess.

Everyone is assuming that Watson understood what the category was. This is far from obvious. As I recall the display, it was "US CITIES". Now, that phrase has obvious meaning to a human. But, if that is the exact text that Watson received, consider that it is all caps with no periods to indicate an abbreviation. So, does the computer interpret US as the word "us" or as an abbreviation for United States? Who knows, but that might explain why Watson went outside the category.

This is a perfect example of why it is so incredibly hard for computers to understand human language. It is trivial and automatic for the human brain to infer the otherwise ambiguous meaning of "US" from context. Computers must somehow be programmed to recognize and deal with the enormous number of cases where these types of inferences are necessary.

 

[nublikescake]

Everyone is assuming that Watson understood what the category was. This is far from obvious. As I recall the display, it was "US CITIES". Now, that phrase has obvious meaning to a human. But, if that is the exact text that Watson received, consider that it is all caps with no periods to indicate an abbreviation. So, does the computer interpret US as the word "us" or as an abbreviation for United States? Who knows, but that might explain why Watson went outside the category.

This is a perfect example of why it is so incredibly hard for computers to understand human language. It is trivial and automatic for the human brain to infer the otherwise ambiguous meaning of "US" from context. Computers must somehow be programmed to recognize and deal with the enormous number of cases where these types of inferences are necessary.

But those are exactly the kinds of questions that Watson is designed to answer and has answered during the show.

 

[jalaram]

Like what? What other reference you got that is fairly universal?

What about the super computer in Hitchhiker's guide? Not as universal, I guess.

 

[Pheran]

But those are exactly the kinds of questions that Watson is designed to answer and has answered during the show.

Yes, they are, but it doesn't mean the code will get it right 100% of the time. Answering the category "US CITIES" with a Canadian city is an obvious failure somewhere. I'm just trying to point out that this category description is not nearly as clear-cut for a computer as everyone seems to think.

 

[sdifox]

What about the super computer in Hitchhiker's guide? Not as universal, I guess.

Hactar, Eddie, Earth or Deep Thought? IBM did name the chess playing computer after the Deep Thought in HHGG


I don't think they are as universal as HAL9000 and SkyNet.

 

[Kev]

I don't know about "perfect sense." I suppose it depends on how comfortable one is with his knowledge of the final Jeopardy category. I'm not sure if/how Watson would be programmed to calculate this. You would think that with a fairly easy category like tonight's he would, in theory, be fairly confident in his knowledge of the subject and stand to wager a bit more money to end with a higher total. Instead, he risked an essentially insignificant amount relative to his lead which indicates that either he somehow decided he didn't know much about the category, or that he doesn't know how to calculate his confidence based on category alone.

If you were a contestant on jeopardy, and you were guaranteed a win, would you BET over $1000 of REAL MONEY on a SINGLE QUESTION that you may or may not know the answer to? I wouldn't

 

[TheVrolok]

If you were a contestant on jeopardy, and you were guaranteed a win, would you BET over $1000 of REAL MONEY on a SINGLE QUESTION that you may or may not know the answer to? I wouldn't

If I already had 30k+ and I was fairly comfortable with the category? Absolutely. I wouldn't risk losing, and I wouldn't risk a huge portion, but I absolutely would. If I were in Watson's place, I may have gone as far as 5k.

 

[sdifox]

There are plenty of Torontos in the US.

• Toronto, Illinois, located south of Springfield, Illinois and to the west of Lake Springfield
• Toronto, Indiana
• Toronto, Iowa
• Toronto, Kansas
• Toronto, Missouri
• Toronto, Ohio
• Toronto, South Dakota
• Toronto, an alternate name for Tamo, Arkansas

I am wondering if it got confused with the Illinois one.

Yes, they are, but it doesn't mean the code will get it right 100% of the time. Answering the category "US CITIES" with a Canadian city is an obvious failure somewhere. I'm just trying to point out that this category description is not nearly as clear-cut for a computer as everyone seems to think.

I guess you missed my post.

 

[zinfamous]

I guess you missed my post.

yeah, but only Canadians live in those cities, so they are technically in Canada.

:sneaky:

serious response: do any of those "US cities" have two major airports--let alone one airport--as required by the question?

 

[sdifox]

yeah, but only Canadians live in those cities, so they are technically in Canada.

:sneaky:

serious response: do any of those "US cities" have two major airports--let alone one airport--as required by the question?

no the question was "Its largest airport is named for a World War II hero; its second largest for a World War II battle."

So the size of the airports would be a comparison within the city and not with some other city. I don't know if those Torontos have multiple airport.

Apparently Toronto, IL is near Springfield. Maybe it thought it was that one? I don't even know if Springfield has 2 airports.


This is IBM's response

"David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show on Monday morning that several of things probably confused Watson. First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. "What US city" wasn't in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn't find much evidence to connect either city's airport to World War II. (Chicago was a very close second on Watson's list of possible answers.) So this is just one of those situations that's a snap for a reasonably knowledgeable human but a true brain teaser for the machine."

http://www.huffingtonpost.com/2011/02/15/watson-final-jeopardy_n_823795.html

 

[mcmilljb]

Everyone is assuming that Watson understood what the category was. This is far from obvious. As I recall the display, it was "US CITIES". Now, that phrase has obvious meaning to a human. But, if that is the exact text that Watson received, consider that it is all caps with no periods to indicate an abbreviation. So, does the computer interpret US as the word "us" or as an abbreviation for United States? Who knows, but that might explain why Watson went outside the category.

This is a perfect example of why it is so incredibly hard for computers to understand human language. It is trivial and automatic for the human brain to infer the otherwise ambiguous meaning of "US" from context. Computers must somehow be programmed to recognize and deal with the enormous number of cases where these types of inferences are necessary.

It clearly says "U.S. CITIES". All you know from the clue is that your response will be a U.S city. However, from the answer you need to know some airports and who they are named after. Most likely Ken and Brad know enough about major airports that O'Hare was named after a medal of honor recipient. Plus they most definitely knew of the Battle of Midway. So it was fairly easy for them. The problem would be how does he search on the idea of a WW2 hero plus he needs to know how to search for WW2 battles the U.S. was involved in. Definitely tough for a computer on this one.

Here is the video of just final jeopardy - http://www.youtube.com/watch?v=7h4baBEi0iA

 

[spidey07]

I think to make it more fair there should be a human pressing the buzzer and not the computer. All 3 knew most of the answers it just came down to who could press the buzzer faster and clearly a computer can do that.

 

[MrChad]

I think to make it more fair there should be a human pressing the buzzer and not the computer. All 3 knew most of the answers it just came down to who could press the buzzer faster and clearly a computer can do that.

Ken Jennings weighed in on it:

Q.
Hi Ken- Seemed to me, for many of the questions, that the computer was just better at buzzing in. Does Watson have an unfair advantage for timing the buzz-in? Thanks.

A.
KEN JENNINGS :
As Jeopardy devotees know, if you're trying to win on the show, the buzzer is all. On any given night, nearly all the contestants know nearly all the answers, so it's just a matter of who masters buzzer rhythm the best.

Watson does have a big advantage in this regard, since it can knock out a microsecond-precise buzz every single time with little or no variation. Human reflexes can't compete with computer circuits in this regard. But I wouldn't call this unfair...precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance.

From http://live.washingtonpost.com/jeopardy-ken-jennings.html

 

[zinfamous]

no the question was "It's largest airport is named for a World War II hero; its second largest for a World War II battle."

So the size of the airports would be a comparison within the city and not with some other city. I don't know if those Torontos have multiple airport.

Apparently Toronto, IL is near Springfield. Maybe it thought it was that one? I don't even know if Springfield has 2 airports.


This is IBM's response

"David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show on Monday morning that several of things probably confused Watson. First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. "What US city" wasn't in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn't find much evidence to connect either city's airport to World War II. (Chicago was a very close second on Watson's list of possible answers.) So this is just one of those situations that's a snap for a reasonably knowledgeable human but a true brain teaser for the machine."

http://www.huffingtonpost.com/2011/02/15/watson-final-jeopardy_n_823795.html

interesting. I didn't know that Watson had "learned" to put such low importance on the category. I knew that he basically had to figure out how to answer questions within categories based on a pool of those questions, but not that the algorithm had lead him to completely dismiss the category name.

Knowing Jeopardy though, it's certainly possible that the answer could have involved some non-US city, where they question that you needed to pick involved a US city referenced through the other city, or person, or historical event in the answer. Or the other way around.

I can see how since it was only one question within the one category, he wasn't able to select from a pool of questions to figure out how to answer. Odd, though, that simply using "Its" in the answer, rather than "This US City" was such a stumbling block for him. well, maybe not too odd. seems odd to us meatbags, though.

But I think that it shows Watson's primary function is to win Jeopardy. He probably isn't going to risk much at all in final Jeopardy if he has an unreachable lead. COnsidering that he isn't able to "learn" how to answer in that category, his confidence level for the final round must be locked in at a really really high threshold for him to consider wagering huge amounts.

 

[FeathersMcGraw]

I think to make it more fair there should be a human pressing the buzzer and not the computer. All 3 knew most of the answers it just came down to who could press the buzzer faster and clearly a computer can do that.

Yes, but Watson only buzzes in when its calculated accuracy exceeds a particular threshold. So part of the trick is that the answer-processing algorithms have to be faster than a Jeopardy champion's recall and association. People see Watson's "reaction time" as a computer-based advantage, but the fact that it can parse, interpret, and calculate an answer faster than a human can recall trivia is as much a demonstration of the technology as the ability to determine correct answers.

 

[MrChad]

LOL

Q.
Did Watson's voice throw you off your game at all?

A.
KEN JENNINGS :
Yes, because I wanted him to sound like Darrell Hammon doing Sean Connery on SNL's Celebrity Jeopardy instead. "That'sh not what your mother shaid lasht night, Trebek!"

 

[sdifox]

Ken Jennings was a riot in that Q&A

 

[Jeffg010]

interesting. I

But I think that it shows Watson's primary function is to win Jeopardy. He probably isn't going to risk much at all in final Jeopardy if he has an unreachable lead. COnsidering that he isn't able to "learn" how to answer in that category, his confidence level for the final round must be locked in at a really really high threshold for him to consider wagering huge amounts.

You play the game to win!

 

[jalaram]

The Watson team told me two things after the match: that the idea for Watson was born after watching my 2004 streak on Jeopardy, and that they watched LOTS of tape of me while honing its skills. "There's a lot of you in Watson," one guy said. So I already feel like the Dr. Frankenstein here. If it goes amuck and kills humanity and stuff so sorry lolz my bad!

Like OMGWTFBBQ.

 

[PingSpike]

it was a very loaded question, but the fact that the category was so insanely unambiguous--US Cities--I'm astonished that he didn't even stick to the category.



The gasp in the room was amazing, and it was profoundly embarrassing, but in all honesty, let's talk about Watson's (very real) limitations with a grain of irony, considering that he is currently mopping the floor with the meatbags.

Spoiler

It looked embarrassing and was pretty hilarious at the time...but just because the category was "US Cities" doesn't mean the answer had to be a US City, right? It could have been a sister city or something else like that. The category deciphering code was probably a little to smart for its own good in this case and went off on a tangent. Also to be fair, he didn't wager much and clearly had a low confidence level in his answer. It is funny that I was able to puzzle it out where the machine failed though.

That said, given the other information of the question (air ports, world war II battles and heroes) its pretty strange he ended up with a city in Canada as the answer.

 

[sdifox]

Spoiler

It looked embarrassing and was pretty hilarious at the time...but just because the category was "US Cities" doesn't mean the answer had to be a US City, right? It could have been a sister city or something else like that. The category deciphering code was probably a little to smart for its own good in this case and went off on a tangent. Also to be fair, he didn't wager much and clearly had a low confidence level in his answer. It is funny that I was able to puzzle it out where the machine failed though.

That said, given the other information of the question (air ports, world war II battles and heroes) its pretty strange he ended up with a city in Canada as the answer.

It was deemed you don't need the spoiler tag the day after the show.

 

[PingSpike]

After I saw 50 posts of bitching I erred on the side of caution.

 

[sdifox]

After I saw 50 posts of bitching I erred on the side of caution.

I prefer to live dangerously :awe:

 

[Pheran]

It clearly says "U.S. CITIES". All you know from the clue is that your response will be a U.S city. However, from the answer you need to know some airports and who they are named after. Most likely Ken and Brad know enough about major airports that O'Hare was named after a medal of honor recipient. Plus they most definitely knew of the Battle of Midway. So it was fairly easy for them. The problem would be how does he search on the idea of a WW2 hero plus he needs to know how to search for WW2 battles the U.S. was involved in. Definitely tough for a computer on this one.

Here is the video of just final jeopardy - http://www.youtube.com/watch?v=7h4baBEi0iA

Oops, obviously I misremembered the exact labeling of the category; my bad.

 

[JTsyo]

It seems Watson is really strong with all the literary questions. Guess it's easy to check the questions against fixed books.
What Ken said about the buzzer is true, it seemed often he was also trying to buzz in.