Survivor bias is a theoretical sounding name for a concept that occurs frequently and can be the cause of major errors in evaluating phenomena. Survivor bias occurs when we look at a group of people who are special in some way to try to see what made them special. There is a risk that we identify a characteristic and assume that it is a determining characteristic.
For example, in the late 1970s Tom Peters researched 43 successful companies to determine what made them successful, work which ended up being published in his book In Search of Excellence. However, by early 1984 a third of these companies were in trouble, throwing doubt on Peters’ recommendations. Perhaps this accounts for why his advice in Thriving on Chaos seemed quite different in 1987?
One marketing trap, that relates to survivor bias, is the way some companies, especially multi-level marketers, show examples of ordinary people who have become rich selling their products (e.g. through party plans, door-to-door etc). By faithfully re-telling individual case studies, so many hours per week, so many meetings/parties/doors etc they can create the impression that anybody can do it, when the facts are that nearly everybody fails.
A great example of survivor bias comes from the story of the King’s coin man. The story starts with two neighbouring kingdoms, Upend and Downend. At Christmas each year the kings get together for a party and they decide who is going to pay by tossing a golden coin. For four years in a row the king from Upend wins, causing the king from Downend to suspect that he is less good at tossing a coin than his neighbour. So, the king decides next year that he will take one of his subjects with him to toss the coin and he will find out the best coin man in his kingdom. He announces a competition for the best coin man, with every contestant having to pay one groat to enter the competition. At first 1024 of his subjects enter the competition. They are split into pairs and play the first round, after the first round there are 512 left. They keep playing until just 2 are left, each having won 9 times in a row! The press interview each of the finalists, who both explain what they have for breakfast, how they hold the coin, what makes them decide heads or tails. Then they have the final, and the winner is triumphant, he has won ten competitions in a row, what were the odds against that? (about 1024-to-1 of course). The assumption that the winner is any more likely to win at Christmas is survivor bias.
The impact of survivor bias on market research can be profound. If we conduct focus groups with people who dissatisfied with a product, we might find they have several characteristics in common, but we do not know that we are not suffering from survivor bias, what if happy customers share some of the same characteristics?
One way to get higher scores from a concept test is to make the questionnaire longer and put the purchase question near the end. There tends to be a bias in the way people drop out, with the least interested dropping out more than the happy people, so the result at the end seems higher.
If a utility provides a service that is great for some people and rotten for others, then over the years they may find the people receiving the rotten service leaving. This could result in their annual customer satisfaction scores increasing, because of survivor bias, i.e. the people who are left are the ones who were receiving the more appropriate service.
The general point for market researchers is that it is not enough to see that a group shares something in common, the people who are not in the group need to be examined to see that they do not also share the characteristic.
Comments