If it is April in a presidential election year, then public opinion surveys are about as common as bluebonnets in Texas. Hardly a day goes by without seeing a new poll, whether it be a national-level poll looking at general support for the candidates or similar polls comparing Hillary Clinton and Bernie Sanders. Other polls look at state matchups among the contenders for their party’s nomination, and other polls, with the general election contest still seven months away, look at various head-to-head matchups. And while quite different, all polls have one thing in common: They can’t be trusted, and Sanders’ Michigan win is proof.
Polling has certainly come a long way from the very early days of survey research. Back in 1936, the soon-to-be defunct Literary Digest commissioned a poll that showed FDR losing to the Republican candidate by more than 20 points. Roosevelt went on to win the 1936 election by a margin of 61% to 39%, carrying every state but Maine and Vermont.
The Digest poll had committed the cardinal sin of not selecting a representative sample of the population. This particular survey had been done as a straw poll, using lists of telephone subscribers and others with higher income. Those who did have telephones were more well-to-do and educated, and voters in these demographic categories were substantially more likely to vote Republican. Thus, the poll sample was skewed badly in favor of Republican-Party identifiers, and seriously undercounted Democratic voters.
In the 1948 presidential race between Harry Truman and Thomas E. Dewey, the polls suggested a substantial victory for Dewey over Democratic incumbent Truman. The well-respected Gallup poll had Dewey winning by about six points. The problem wasn’t one of methodology. Rather, the pollsters simply assumed that Dewey’s lead, two weeks from the election, was insurmountable, and thus did not bother to continue polling up until Election Day. If they had, they might have been able to capture the impact of Truman’s “give ’em Hell” whistle-stop train campaign that mobilized Democratic voters in the closing days of the campaign (yes, even as late as 1948, candidates campaigned by train rather than by airplane).
And though pollsters today wouldn’t make that kind of mistake (society has a deeper appreciation for short-term campaign dynamics), they do face a similar challenge: to obtain a representative sample of the population (meaning, in theory, that every voter has an equal chance of being selected), and to obtain a sufficiently large sample so the margin of error is kept to a realistic minimum. So, while a sample of 500 voters yields, again assuming it is representative, a margin of error of just over four percentage points, a sample of 1,000 voters produces a margin of error of just over three percentage points.
Say there’s a survey before the election that has Clinton defeating Trump by 50% to 47%, and the margin of error is 3%. That means there is a 95% chance the true state of the race could have Clinton up by as much as nine points, or Trump could actually be ahead by about 50% to 47%. And, the laws of probability dictate that on average, the results will be outside the margin of error 5% of the time. One way to help determine if a poll is in the ballpark in terms of accuracy is to look at what pollsters call the “internals.” They provide demographic and party identification breakdowns, which can be extremely helpful in judging the veracity of a particular survey. For example, we know that on Election Day, the electorate will consist of about 52% women. If a survey’s sample is 60% female, then that’s a pretty good indicator that the poll is oversampling Democratic-leaning voters, since women tend to be more likely to vote for Democrats.
For election polls to be accurate, it is critical that the voters surveyed are those who will actually vote on Election Day. Reputable polls use various questions designed to determine if one is actually likely to show up to vote. The failure of the polls a few weeks ago to predict Sanders’ victory in the Michigan primary may well have been the result of polls’ inability to include many first-time voters who supported Sanders and who may have been excluded from the likely voter screens. Predicting primary results can be especially difficult, since turnout is generally, although not always, lower than in general elections, so determining who will vote can be more difficult than in a general election.
One major difficulty pollsters today face is obtaining cooperation from survey respondents. Participation levels can be as low as 10%, whereas in 30 or 40 years ago, well over half of those contacted would actually participate in the survey. This lack of cooperation perhaps reflects in part the increased disdain for politics and politicians, and the decline in confidence in America’s political institutions. Not only does this substantially increase the cost of conducting the survey, but it leaves the uncomfortable question as to whether there are systematic differences between those who cooperate and those who do not. To date, there is no empirical evidence of such differences (which, if true, would mean the sample of voters is not representative), but that has to be a concern lurking in pollsters’ minds. Moreover, given the costs of polling, automated voice-response systems are increasingly in vogue. But there’s concern that such automated systems may inadvertently have subtle, built-in bias that skew results in a particular way. Automated response systems are not legally able to contact cell phones, for instance, as cell phone polls can only be done by live interviewers.
As Americans are inundated by polls over the next several months through the general election, it’s important to not place too much reliance on any one survey. Voters should check the internals of the poll to see if the demographics appear plausible, look at multiple polls taken at roughly the same period of time, and check the margin of error. If one poll is an outlier, then the results should be substantially discounted. And, please, take pity on the poor pollsters, and help them out and participate!
Dr. Euel Elliott is a professor of public policy and political economy and the senior associate dean in the School of Economic, Political and Policy Sciences at the University of Texas at Dallas.
