Polls Bullshit?

[Lonestar]

About a week ago ABC news a story commenting that there is a chance, a very good chance, that current polls are very misleading because people may be telling pollsters that they are voting for Obama because they don't want to appear racist. At the same time, James Carville has said that "If the Republicans steal another election…I don't want to think of the consequences" He is apparently basing this idea on poll numbers. (For the Record, my Grandfather went one further and told me last week that if Obama loses "THE NEGROES WILL RIOT!")


The problem I have is that there seem to be a lot of people, here and elsewhere that assume that Obama is wildly in the lead and that if he loses it will be because of a stolen election. No one here…or elsewhere, seem to accept that McCain has a chance to legitimately winning the election. While it seems anecdotally that I see more Obama supporters around than Kerry supporters in '04, I can't help but wonder if I'm projecting. And even if I'm not, I worry that we are still underestimating the power of stupid people in large numbers…specifically, the ones that think that Obama is a Ay-rab, Muslim, socialist, etc. and will turn out in large numbers to make sure he doesn't get the White House.

Am I the only one seeing this? It seems as if(again, on this board and elsewhere) there are a lot of people taking it for granted that Obama is going to win, and they are setting themselves up for a big let down.

[Darth Wong]

They don't publicize the names of people who say they'll vote for McCain on polls. Why would someone answer a poll differently than a vote?

[irishmick79]

They don't publicize the names of people who say they'll vote for McCain on polls. Why would someone answer a poll differently than a vote?

Unlike in actual voting, a lot of the polls involve actually answering questions from a representative on the phone. That interaction is suspected to introduce a different decision making dynamic than the one a voter might actually use in the voting booth.

[LapsedPacifist]

Do a search here for Bradley Effect to look at some of the discussion about this.

In a nutshell, the democratic primaries showed results that correlated with polling suggesting that Bradley effect shouldn't be too big an issue this election.

LP

[Xon]

Selection bias is stupidly ease to occur in statistical analysis which is basicly what polling is. Especially self-selection bias for any type of volunteer polling which describes US elections.

IMO, voluntary voting is unsound purely on the grounds that it is statistically flawed even worse in the USA system where voter exclusion are known tactics to prevent particpation.

This is all before you get people talking to the voters, which results in types of wonky effects where the very act of questioning someone changes how they respond, sometimes dramatically from simply gramatical differences.

[RedImperator]

What you're referring to is the Bradley Effect. Three things to know about it:

1. In the primaries, Obama lost elections the polls said he should have won only three times out of 57 contests, and one of those contests was New Hampshire, where the polls correctly predicted Obama's final number, but did not predict the undecideds breaking for Clinton in huge numbers in the final 36 hours.

2. Conversely, Obama over performed his poll numbers across the South, including in Virginia and North Carolina. This may have been caused by pollsters underestimated African-American turnout, or it may have been caused by a "reverse Bradley Effect", where white voters are reluctant to admit they will vote for a black candidate, but do so in the privacy of the voting booth.

3. There is considerable dispute whether or not the Bradley Effect still exists at all. A recent Harvard study determined that it vanished in the mid-1990s.

4. The conventional wisdom is, if the Bradley Effect does exist, then you should subtract 3-4% from any black candidate's poll numbers when running against a white candidate. Obama's national lead is over 6%. His lead in all the Kerry states is is 10% or more, except New Hampshire, where it's 7%. His lead in Iowa and New Mexico is also over 10%. His lead in Colorado and Ohio is 6%, and 7% in Virginia. The Kerry states, Iowa, New Mexico, and any one of those three latter states is enough to put Obama over the top; overall he has a Bradley-proof lead in states representing 306 electoral votes. And that's not counting Indiana, which Electoral-vote.com shows as a 7 point lead based on yesterday's Battleground poll, which I'm discounting because I think it's an outlier.

[Slacker]

Also, there's some evidence there never was a Bradley effect-there was a flap in the news a while back when Bradley's campaign manager pointed out both parties were running internal polls that put the contest within the margin of error, and it could've just been bad polling in the first place.

[Ekiqa]

Polls are VERY easy to manipulate, with the pollsters able to come at the answer they want by asking leading questions.
Unless you know the questions asked, a poll is useless.

To wit, an example from "Yes, Prime Minister"

Sir Humphrey Appleby: Mr. Woolley, are you worried about the rise in crime among teenagers?

Bernard Woolley: Yes.

Do you think there is lack of discipline and vigorous training in our Comprehensive Schools?

Yes.

Do you think young people welcome some structure and leadership in their lives?

Yes.

Do they respond to a challenge?

Yes.

Might you be in favour of reintroducing National Service?

Er, I might be.

Yes or no?

Yes.

Of course, after all you've said you can't say no to that. On the other hand, the surveys can reach opposite conclusions.

Mr. Woolley, are you worried about the danger of war?

Yes.

Are you unhappy about the growth of armaments?

Yes.

Do you think there's a danger in giving young people guns and teaching them how to kill?

Yes.

Do you think it's wrong to force people to take arms against their will?

Yes.

Would you oppose the reintroduction of conscription?

Yes.
[does a double-take]

There you are, Bernard. The perfectly balanced sample.

[General Zod]

Interesting research for those worried about the Bradley effect.

WASHINGTON (CNN) -- A new national survey suggests that race won't be a major factor in the outcome of the presidential election.

Seven out of 10 -- or 70 percent -- of Americans questioned in a CNN/Opinion Research Corp. Survey released Friday said the race of the candidates will not be a factor in their vote for president this year.

That 70 percent figure is up 9 points from July, when the same question was asked. Only 5 percent of those polled said race will be the single most important factor in their choice for president, with 11 percent saying it's one of several important factors, and 13 percent indicating race will be a minor factor in their vote.

Sen. Barack Obama, if elected, would be the first black American to win the White House. Video Watch more on the state of the campaign »

"First, don't assume that everyone who says that race is factor in their votes are voting against Obama. Some voters are choosing Obama because of his race. And many of those who say that race will influence their votes are Republicans who were highly unlikely to vote for any Democrat this year," said Keating Holland, CNN polling director.

"By one complicated measure, the number of votes Obama may lose due to his race is roughly equal to the number who will vote for him because he is black. And both those numbers appear to be small, possibly just 1 percentage point in each direction," Holland said.

One question that often comes up when discussing polling regarding race is whether those being polled are telling the truth.

"Take all this with a grain of salt -- race is a complicated topic and polls may not reveal each respondent's true feelings on this hot-button issue. Nonetheless, the poll suggests that race may largely be an influence on Americans who aren't typical Democratic voters, and that race works both for and against Obama in roughly equal proportions," Holland said.

So, what about age? If elected, the 72-year-old John McCain would be the oldest person to be inaugurated as president.

Roughly half of those polled said the age of the candidates will affect their vote. That's essentially unchanged since July. Three percent said age would be their most important factor, with 19 percent saying it would be one of several important factors and 25 percent saying it would be a minor factor.

The CNN/Opinion Research Corp. poll was conducted October 17-19, with 1,058 adult Americans questioned by telephone. The survey's sampling error is plus or minus 3 percentage points.

So far it's looking like whether or not the Bradley effect is going to occur depends on which pollsters you agree with.

[Metatwaddle]

Wouldn't a poll on whether or not race will affect one's vote suffer from the same systematic problems that create the Bradley effect in the first place?

[CmdrWilkens]

The ABC study which you are quoting didn't actually indicate that folks would not vote for Obama AND yet still say they would to Pollsters. So its not really that much of a worry. Rather what the finding of the study was held that up to 6% of the population would not vote for Obama solely or principally because of his race. Now that is a pretty decent chunk of folks but the problem is the current polling environment ALREADY accounts for that. If someone says they won't vote for Obama it doesn't matter what their reasons are the pollsters will still count that as a vote for McCain when tallying their data. The only reason this would shift the election suddenly would be if: 1) The 6% are somehow hiding themsleves either amongst the undecideds or amongst the Obama supporters AND 2) The poll's aren't under-reporting Obama's level of support.

Error in polling comes from a whole host of things but lets look at some of the easiest to identify sources:

1) Methodology: Some pollsters ask questions that get solid responese, others ask leading questions, other do things so fast and cheap that their data isn't worthwhile. The 538 pollster rating index is probably the best source for data on who is Methodologically sound

2) Lean/Weight: This is really a subset of methodology but the actual compositon of the expected electorate not just in terms of partisan breakdown but also age, gender,and race can have huge impacts. take the DailyKos/Research2000 tracker which expects 14% African American turnout versus the Zogby which uses partisan wightings from the 2004 election despite the disparity in party identification today. Nobody is sure who has the best model but when the models which don't favor Obama (2004 partisan identificaiton, lower youth turnout, etc) still show him with a lead and the polls which weight more for those same factors (such as Gallup's Likely voter II model) show him with 50%+ and close to doble digit leads.

3) Sample size: Again its sort of a subset of methodology but the number of folks you ask obviously makes a difference in being able to make a statistically viable statement. Within this, though, are things like the difficulty of actually getting an accurate sample without overweighing the limited groups you do get. 538 had a great post pointing out an IDB poll which somehow had Mccain winning the 18-22 vote by 74-22 which, for it to be co-incidental, is about a 55billion to 1 chance. The poll most likely didn't get enough actual 18-24 year olds so they overweighted the responses they did receive so any small errors there were immensly magnified. A lot of this feeds into the 2-3% shotfall that several pollsters think might be out there waiting to help Obama based on cell-phone only users. They fall squarely into his targeted demographics and they are basically under or un-sampled in most polls yet they are growing as a percent of the total populace. In turn this means that you end up having to overweight the remaining responses which leads to greater error.

4) Quality of response: If you ask crappy questions you get crappy results, if you only call a few crappy people then you will get crappy results as well. The problme is that if you have solid methodology and good sample size it would take a large campaign to hide voter preference on the scale we are talking about for all of the polls in the field to be very far off.

[montypython]

The sample size of many polls has bugged me for a long time, and one thing that I've always wondered about is how skewed the population sample that's polled is, as that's hardly ever mentioned at all.

[CmdrWilkens]

The sample size of many polls has bugged me for a long time, and one thing that I've always wondered about is how skewed the population sample that's polled is, as that's hardly ever mentioned at all.

The margin of error is actually presented in a very simplistic way, what you actually see with a poll is this:

Current sample (e.g. Obama 51%) +/- 3.5% with a 95% certainty. In simpler terms this means that based on the number of respondents (and for a 3.5% error you would need about 800 respondents) that 95% of the time if you asked the same question of the same number of people your results would be between Obama 47% and Obama 55%.

Now how you get to 800 respondents is obviously a matter of concern and many (though not all) pollsters tend to be a bit reclusive about their models. That said what they do is keep calling to the point where they either get way morethan 800 respondents then go back and weight reponses downwards based on their models or they sample 800 people and then weight upwards based on their models. How do they do that?

Lets say, for example, that looking at 2004 exit polling you find out that 18-29 year olds comprise roughly 10% of the voting populace. Not taking any risks with your likely voter model you decide that your final sample should have 80 (of 800) responses from those aged 18-29. Now, as often might happen, after 800 calls you may not have 80 voters in that category so what do you do? You have two options:

The first is to keep calling more people until you have 80 responses form that age group, only now you probably have a lot of extra responses in other age groups (say 500 responses in the 30-40 group and you only want 400). Then you simply make each response in the 30-40 group worth less, that is when adding up your responses each 29-40 voter is now only .8 votes instead of 1 while your 18-29 folks are still 1 to 1.

The other option is to simple overweight the number of 18-29 year odls you have received. So lets say you do have your 400 30-40 responses but only 40 in the 18-29 group. What you do then is make each vote in the 18-20 group worth twice as much. The price here is that the sub group error for the sample of 80 voters is 11% and the error for 40 voters is 16%. Now when you add this up with 80 voters you may shift your total sample by about 8 votes total or 1% or your total sample, with the 40 voters you now are hitting a 13 vote swing or 1.5% of your sample.

Obviously the former is more accurate than the later but its also more expensive.

[Surlethe]

The sample size of many polls has bugged me for a long time, and one thing that I've always wondered about is how skewed the population sample that's polled is, as that's hardly ever mentioned at all.

Sample size is not as important as how random the sample is. While sample size does affect the confidence interval (i.e., the margin of error, as Wilkens pointed out), when you've got more than 50 respondents, IIRC, the ratio of sample size to population size is not relevant to how accurate the poll is.