Following a recent online discussion, I am starting an online project to list some of the key things that I think every researcher should now (which I am going to refer to as TARSK). This first case is taken from 1936, but it relevance to today’s online samples could not be greater.
In 1936 a US magazine, the Literary Digest decided to try and predict the result of the US Presidential election, as it had done successfully for the last five Presidential elections. The magazine sent out 10 million postal ‘ballots’ and received over 2 million replies.
The magazine was very confident in its prediction that Alf Landon would beat Franklin D Roosevelt, indeed it printed its prediction in its magazine that would take 57% of the popular vote and easily win the election. In the magazine, when talking about the likely accuracy of the Literary Digest poll, they quoted Democrat James Farley who said “Any sane person cannot escape the implication of such a gigantic sampling of popular opinion as is embraced in The Literary Digest straw vote. I consider this conclusive evidence as to the desire of the people of this country for a change in the National Government. The Literary Digest poll is an achievement of no little magnitude. It is a Poll fairly and correctly conducted.”.
However, Roosevelt easily won the election, carrying most of the, then, 48 States, and gaining just over 60% of the popular vote. How, could a sample of over 2 million people be wrong?
The answer to the problem lay in the sample source and bias. Roosevelt was associated with the champion of the poor, Landon was more associated with the wealthy (although US politics are always more complex than simple left/right or rich/poor). The sample of 10 million was primarily drawn from three sources: subscribers to the Literary Digest, owners of cars, and owners of a telephone, in 1930s America. A sample drawn mainly from the comfortably well off predicted that the champion of the poor was going to lose.
At the same election George Gallup used a carefully selected sample of 50,000 voters and predicted the election to within 1% of the final result. By doing this Gallup created the model that underpins much of market and social research to this day.
The big lesson for today, especially for online market research, is that a big bad sample, is just a bad sample. However, it is not unusual to hear some vendors say that although they can’t be sure who has answered their sample, the fact that they have a sample of 40,000 or 50,000 or even 100,000 must mean something. Whenever you hear such a claim, remember the Literary Digest had over 2 million responses and it was wrong, very wrong.
