So, Why Does Market Research Usually Work?

In the chapter on NewMR I talk about that traditional quant research does not work in the way they market research normally implies it works. I point out two major reasons for this:

1. Commercial market research almost never uses samples that approximate to random probability samples.
2. Many of the questions we ask can't be answered (see Lindstrom, Ariely, Earls, and Lehrer).

Note, this explanation in the book is somewhat fuller!

However, I then discuss why market research seems to work most of the time, which is the bit below. I would really appreciate hearing people's views.

---

So, Why Does Market Research Usually Work?

If the sample is usually the wrong sample and to questions often don’t reveal why people do what they do, why does market research usually work? 

The claim that market research usually works can be supported in two ways. Firstly, there are those occasions where the research can be tested. Probably the best example of this phenomenon is election polling, where the polls are usually correct, in an area where sampling would appear to be critical, and where respondent’s might not only not know their intentions but also might not be prepared to disclose them. The second illustration is the scale of shock and public discussion when market research really does get it wrong, such as the New Coke fiasco in 1985.

A second factor that needs to be considered what we mean when we say market research works. At best market research can only be a probabilistic risk reduction system. It seeks to reduce the number of Type I and Type II errors (as statisticians call them). A Type I error is a false positive, i.e. bad business decision is not spotted and allowed to go ahead. A Type II error is a false negative, i.e. a good idea is flagged as a bad one. For market research to work the cost of the market research needs to be sufficiently below the benefits generated by the reduced chance of Type I and Type II errors. Nobody should suggest that market research removes the risk of either Type I or Type II errors, it works by reducing the risk.

Note this section is written in the first person to highlight that it is the author’s conjecture, supported by 30 years of working with the data and extensive research, discussion, and attending conferences.

I think that there may be many reasons why market research tends to work, but I feel the key reasons are as follows:

•    Homogeneity.

•    Wisdom of crowds.

•    The ‘art’ or market research.

I explore these three reasons below.

Homogeneity

As Mark Earls has illustrated (in his book Herd), people tend to do what other people do. People who manage to function in society tend to be like other people. If we show a three TV commercials for breakfast cereal to 100 men aged 20 to 35 years old and to a sample of 100 females aged 36 to 50, then the commercials will have to be very unusual for them to be rated differently, at least in terms of the relativities between the three commercials. If Ad A is seen as the most fresh and modern by the men, then it will usually be seen as the most fresh and modern by the females. If three ads were about female sanitary protection, then the response might well be different. A random probability sample protects against the sample creating bias, but in most cases having the wrong sample does not create, in the real world, biases that change the relative results between items being assessed.

Note, there is no rule that says the wrong sample will work OK, but it often does, based on empirical data.

Wisdom of Crowds

Many years ago I worked with an automaker who had a very large, very extensive database of what cars people said they would buy next and what they actually bought next. The interesting thing about this data was twofold:

a.    The car a somebody said they would buy next was a really bad predictor of what that person did buy.

b.    However, the aggregate totals of what people said they would buy was a good predictor of the aggregate total that people bought.

What this data showed was that people were bad at predicting their own behaviour, but really quite good at predicting what groups of people will do. If we consider the prediction market methodology this is exactly what the respondents are being asked to do, i.e. estimate what other people would do.

My conjecture is that when a market researcher shows somebody an ad and a concept for a new type of beer and asks them how likely they are to buy it when it, the respondent is unconsciously conflating some mixture of their own likelihood with some estimate of what ‘people’ like them might do.

The ‘Art’ of Market Science

Market researchers have been taking what respondents say and interpreting them for years. Few market researchers would simply take what a respondent says and report it as a finding. When Lindstrom and others show that what people say is not a good guide to what they will do they are not really saying anything new. Dealing with this issue has been part of market research's stock-in-trade since its infancy. For example, sales estimates from respondents are based on what the respondents say, but they are typically converted using algorithms based on the sort of changes necessary to convert a set of respondent scores into metrics that have some link with the real world.

Summary

My net position is that market research often works, but not in the way that it is often claimed. However, by not focusing on the real reasons that market research works, people run the risk of being surprised when it does not work.

Comments

[andy b]

Hi Ray. Interesting thoughts on the prediction markets. I like your answer b and was thinking something similar.

Perhaps quant works because what people say they will do is no more than an approximation of what they might do, based on their own beliefs (which are absolutely influenced by the beliefs of others and what they believe other people who are like them would do.

Then, over a big sample, with all respondents going through the same process of thought, the differences between what they say they would do and what they actually do are evened out i.e.

1. I said I would do buy X and I did
2. You said you would buy X but bought Y
3. Pete said he would buy Y and bought Y
4. Trish said she would buy Y and bought X

The same amount of X and Y as predicted by the research evened out over the big sample although many people changed their decision.

Apologies if this is what you meant in your original post - although I think this is a slight addendum to it (i.e. some people do exactly as they intended)

[Ray Poynter]

Hi Andy, here is some support for the idea and a possible method by which it might work.
a)prediction markets, such as the Iowa Electronic Markets are able to predict election results on the basis of people buying and selling shares in who they think will win, i.e. estimating what other people want.

b) Let's look at the influence of personal networks. Let's assume your views are partly your views and partly some mix of, say you 20 closest contacts. Now let's take one member of that network of 20 people, their choice is going to be based on their views and their network of 20. In most 'normal' societies, that person's network of 20 will be different from you 20, perhaps 6-12 in common, so this person may make the same choice as you, or it may adjust to represent their 20 people. Keep doing that through a sample which is not intimately linked, even if it is not random, you will get a very wide spread that is closer to the true mean than any of the people will be.

[Andy b]

I'm not sure I agree with your conjecture of people saying what they think other people like them might do, ray. If that were true, then wouldn't the people like them make the same decision also? That seems to be a logical argument.

[Brandon Watts]

Ray,

Very good points. While scientific rigor is always good, sampling rigor is often not realistic due to budgetary/timing constraints and, as you point out, it is often not necessary. A significant portion of market research does indeed serve the purpose of risk mitigation. Better understanding the appeal (or lack there of) of a new product or service or the audience to which it will be marketed is many orders of magnitude cheaper via market research in comparison to proceeding blindly with the launch.

There will always be flaws or elements of market research that can be "nitpicked", but well designed research provides vital information at a cost that is a fraction of the unmitigated risk.