Famous Forecasting Machine is Thrown into Gear for 1936
“Literary Digest Poll is on!” announced confidently the editors of Literary Digest, one of the most popular and respected magazines of that time, yet they didn’t know the fateful mistake they were about to commit was going to lead to their end.
Much of measurement and research revolves around the creation and use of samples because it is cost-effective and in almost all cases it is not feasible for the researcher to attempt to survey all of the population. A big enough but carefully chosen sample can represent all of the population. On the other hand tough, improper sampling can lead to methodological disasters.
Alfred Landon, the Republican governor of Kansas, and President Franklin D. Roosevelt were put in opposition to the presidential elections of 1936. Literary Digest, one of the most respected magazines of its time, likewise the previous elections, administered a pre-election poll. They had correctly predicted the outcome of every election since 1920 and had done so by using very large samples. So, they mailed out 10 million ballots on postcards to individuals (that meant 1 out of every 4 voters in the USA) and attracted around 2,4 million responses, an astronomical number for any opinion poll even by today’s standards.
“Once again, [we are] asking more than ten million voters — one out of four, representing every county in the United States — to settle November’s election in October. Next week, the first answers from these ten million will begin the incoming tide of marked ballots, to be triple-checked, verified, five-times cross-classified and totaled. When the last figure has been totted and checked, if past experience is a criterion, the country will know to within a fraction of 1 percent the actual popular vote of forty million [voters].”-LD
After the collection of the responses, the pollsters at Literary Digest simply totaled the cards for each candidate and incorrectly predicted a victory for Alf Landon (57%) over the incumbent Franklin Roosevelt (43%) [4]. In the end, it turned out that Roosevelt won in a landslide with 62% of the votes and the magazine was so discredited by this discrepancy that it soon folded.
The irony was that the Literary Digest conducted one of the largest and most expensive polls ever and the sampling error was a gargantuan 19%!
What did go wrong?
Flashback to August 1936. The Literary Digest people ran their “famous forecasting machine” as they had done for all the elections in the last 20 years. They used lists of phone numbers, drivers’ license registrations, country club memberships, and lists of magazine subscribers to construct their sampling frame[2]. There are two major mistakes here that gave Literary Digest an infamous place in the survey research and led to ample refinements in public opinion polling techniques.
1-Selection Bias
As mentioned above, Literary Digest created its mailing lists from phone number directories, drivers’ registrations, club memberships, etc. Back in that time (even today), such a list is guaranteed to be biased toward upper-middle-class voters and exclude lower-income voters. 1936 was the height of the depression and at that time, phones, cars, and club memberships were more available to the rich. Plus, there were more than 9 million unemployed people, a very significant segment of the society whose names would not show up in any club membership, phone directory, or drivers’ registration lists, which later was concluded as an underrepresentation of Roosevelt’s constituencies [5]. As the economy was the central campaign of the election that for sure rich and poor would not share the same opinions about, the Literary Digest mailing list was far from being a representative cross-section of the society.
2-Nonresponse Bias
“We wondered then, as we had wondered before and have wondered since, why we were getting better cooperation from in what we have always regarded as a public service from Republicans than we are getting from Democrats.”-LD
There were 10 million people on the list initially, however, 24% of them responded, so only one-fourth of what was intended. Inarguably, people who responded to the survey were not the same as those who didn’t. Pollsters didn’t have any idea how the remaining 76% would vote. In the end, the initial bias towards the overrepresentation of Republicans in the sample was aggravated by the fact that people who were likely to respond tended to be better-educated and wealthy people.
1936 Literary Digest Poll was a failure because of its wildly inaccurate estimation of the actual results, caused by sampling bias. In many cases, this is the first example of how not to sample comes to mind.
There are 2 morals in the story.
First thing, keep your sample small but well-chosen. Literary Digest Poll thought had selected its extremely expensive sample from sources that in the end would not be representing a significant part of the society (Gallup predicted the same election correctly with 50000 respondents [6]).
Secondly, people who do not respond can determine the fate of the study. As it was not enough that their list was skewed towards the wealthy, people who supported Roosevelt were not likely to respond as much as people who supported Landon which created a nonresponse bias towards Landon winning.
Thanks for reading.
References:
[1] Presidential Campaigns: From George Washington to George W. Bush, Paul F. Boller
[2]Famous statistical Blunders in History. Oxford Math Center
[3]Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll. Sharon L. Lohr, J. Michael Brick.
[4]Forecasting Elections with Non-representative Polls.Wei Wang, David Rothschild, Sharad Goel, Andrew Gelman.
[5] Why the 1936 Literary Digest Poll Failed. Peverill Squire.
[6]Gallup Poll Accuracy Record. John Woolley
