Nate Silver: Pollsters Are Rigging their Numbers

Nate Silver, love him or hate him, is a numbers geek. 

We all know him because he rose to prominence as an elections guy, but his passion is numbers. Statistics make him happy. Gambling is his passion. 

The odds matter to him because he makes his living calculating them and betting real money based on those odds. 

It's much easier to call the odds on sports or cards than elections for a lot of reasons. The data you are dealing with are real, for one thing, and there is lots more of it than elections, which occur relatively rarely. 

Most of the "election" data we think of comes not from results but from the numerous polls put out. However, polls are only quasi-data, not real statistics you can use with any certainty. Polls are based on a magical combination of actual data collected in non-random ways, testing questions that may or may not skew the results, and then mashed through a black box formula that tries to turn this rather dirty nonrandom "data" into something that more closely resembles the real world. 

That doesn't make polls USELESS. If they were, campaigns wouldn't waste resources on them. It just means they aren't what we are told they are. 

A poll that shows a 10-15 point gap between two candidates a week out from the election tells you that the candidate who is down will most certainly lose. Not always, but it's a pretty good indicator. The closer you get to tied the less informative they become because the answer is generated not by the data but by the "black box" or model. Any errors in that model that skew results away from the real electorate will introduce systematic errors that are invisible. 

Sometimes the skew is because the pollster's model is innocently wrong. Sometimes not. 

Nate Silver has been watching this process at work for years, and he concludes that pollsters are skewing the results--intentionally--to show a race that is closer than it is. They are "herding," or covering their asses because they are scared to look like an outlier. 

It’s obviously a really close race. But for some pollsters, it’s a little too close.

Take, for example, this afternoon’s polling release from the British firm Redfield & Wilton. They polled all seven of the core battleground states. And in all seven, Kamala Harris and Donald Trump each received between 47 and 48 percent of the vote:

Isn’t this a little convenient? Whatever happens, Redfield & Wilton — not a firm with a well-established reputation in the US — will be able to throw up their hands and say “well, we projected a tie, so don’t blame us!”. And since all of these states are also close in the polling averages, they’ll also ensure that they won’t rank at the bottom of the table of the most and least accurate pollsters — although unless the race really is that close, and it probably won’t be, they also won’t rank toward the top.

Now granted, our forecast is close too. But it’s based on polling averages: dozens of polls have been released in each of these states over the past month. That greatly increases the sample size. Collectively, they’ve surveyed about 230,000 voters.

By contrast, the median sample size in individual polls in these states is 800 voters. In a 49-49 race in a poll of 800 people — assuming 2 percent goes to third parties — the theoretical margin of error for the difference between Trump and Harris is ±6 points. If that sounds higher than you’re expecting, that’s because the margin of error that’s usually reported in polls is only for one candidate’s vote share. For instance, in a poll of 800 people, Trump’s margin of error is about ±3 points, as is Harris’s. However, basically every vote that isn’t a vote for Trump is a vote for Harris. If Trump gets 52 percent of the vote instead of 49, that implies Harris will receive 46 percent.1 So the margin of error on the difference separating Trump and Harris is ±6.

What this means is that if pollsters are doing honest work, we should see a lot more “outliers” than we do — even if people love to complain about them on Twitter.

What he is saying, dumbed down, is this: even if the race is really close in all these states, no one pollster should produce 6 or 7 polls saying the same thing. It is statistically impossible, because even perfect polling has a random margin of error. This is like flipping a coin a thousand times and getting heads all the time. 

Not very plausible. 

Actually, according to Silver, the odds of the polls looking the way they do are infinitesimally small. Outrageously so, in fact:

There’s more detail on this in the table below. Using this margin of error formula, I calculated the likelihood that a poll should show the race within ±2.5 points. This depends greatly on the sample size. For a poll of 400 people, the smallest sample size in our October swing state database, the chances that it will hit this close to the mark are only about 40 percent. For the largest sample, 5686 voters, it’s almost 95 percent instead. But most state polls are toward the lower end of this range, surveying between 600 and 1200 voters. All told, we’d expect 55 percent of the polls to show a result within 2.5 points in a tied race. Instead, almost 80 percent of them did. How unlikely is that?

Based on a binomial distribution — which assumes that all polls are independent of one another, which theoretically they should be — it’s realllllllllllllly unlikely. Specifically, the odds are 1 in 9.5 trillion against at least this many polls showing such a close margin.

The problems are most acute in Wisconsin, where there have been major polling errors in the past and pollsters seem terrified of going out on a limb. There, 33 of 36 polls — more than 90 percent — have had the race within 2.5 points. In theory, there’s just a 1 in 2.8 million chance that so many polls would show the Badger State so close.

I can't check his math, but the point he is making is sound: if the polling were done right, there is no way that so many polls would say pretty much the same thing. The randomness of error precludes it. Something shady is going on, and it doesn't have to be anything as sinister as rigging the election. 

Pollsters have an incentive to be right--it can bring them business--but they also have a strong incentive to not be too wrong and stand out as horrible. So the sensible business thing to do is hedge your bets by sticking with the herd and hoping that by luck you look better than everybody else out of pure chance. Stay close to the norm but a bit off one way or another so you can crow if you hit the jackpot by accident. 

Nobody can point their finger at you if you were wrong--after all, so many others were too and said similar things! 

That seems to be as good a reason as any for the way polls look right now, clustered together so much. Since you expect it to be close, predict it will be close. What we are seeing is not data but ass-covering. 

Don't get me wrong--if these pollsters were seeing something DRAMATICALLY different, they would adjust things to ensure they don't look too far off. That may be why polls drifted toward Trump in recent weeks--tweak to make the polls look more like the reality they think they see. 

In this election, the incentives are doubly bad, because the polling averages in the swing states are close to zero — so a pollster can both herd toward the consensus and avoid taking a stand that there’s a ~50/50 chance they’ll later be criticized for by publishing a steady stream of Harris +1s, Trump +1s and ties. Lately, a lot of national polls have also shown near-ties after usually showing Harris leads earlier in the race. We wonder if there’s been an increasing amount of herding there too, perhaps involving the use and abuse of likely voter models4 — national polls have tightened and moved toward Trump considerably more than state polls have become Trumpier over the past month, except in Nevada and Florida:

This explains why internal polls are, reportedly, quite different than what the public polls say. Their incentive is to get it right for their clients because it matters. Nobody cares what their pollster rating on 538 or RCP is; they need a client to come back next time. 

Does this mean Trump is doing better than the public polls indicate? Not at all. It could be the opposite for all I know. It just means that the polls you see are garbage if you want to predict anything. 

All of this herding — and hedging — increases my concern about another systematic polling error. It might be an error in Trump’s favor again, but it won’t necessarily be: pollsters may be terrified of showing Harris leads after two cycles of missing low on Trump, and they probably won’t be criticized too much for a Harris +1 or even a Trump +1 if she wins in Michigan by, say, 3 or 4 points.

Or there could be errors that run in both directions. Crosstabs show sharp moves away from Democrats among Black and Hispanic voters, and to some extent corresponding gains among white ones. If those crosstabs are real, you’d expect to see some bigger shifts on the map — Georgia being a really rough state for Harris, for instance.

We are all glued to the polls but beware: all the polls are glued to each other because the incentives are high not to look too off and lose your bet. 

We are all on tenterhooks right now, but the only polling I would take seriously at all would be internal polls, and even those are irrelevant by now. It all comes down to turnout operations, not persuading voters.