As part of the book I'm writing on online and social media research I need a chapter on what is NewMR and why is it needed. As part of why it is needed I have looked at some of the problems with traditional quantitative research.
The text below is an early draft of this section and I would appeciate any thoughts or suggestions.
Problems with the Traditional Research Model
In order to appreciate New MR it is important to understand the traditional research model, and the problems that some authorities believe it suffers from.
The heart of traditional market research is quantitative market research, conducted via the use of a sample of the relevant population, surveyed via questionnaires. When market researchers talk about ‘the science’ they are usually referring to quantitative research and the use of surveys.
The science of traditional quantitative market research rests on two fundamental assumptions:
- That the people who are surveyed approximate to a random probability sample
- That the people who are surveyed are able and willing to answer the question they are asked.
The Random Probability Sampling Problem
The random probability sample is the sine qua non of traditional market research. If we do not have random probability sampling, then we do not have a traditional model that works. The problem is that most market research does not approximates to random probability sampling.
A random probability sample is one where the population is known (for example the population might be all men, or all buyers of a specific brand of bread, or all viewers of a particular television programme) and where each member of the population has a known and non-zero chance of being selected. If a random probability sample is selected and measured on some characteristic, it is possible to project the results from the sample onto the whole population, subject to sampling error. The sampling error can readily be calculated and provides a method of expressing confidence in the results, for example a researcher might quote the results of a polling exercise as being +/- 3% with a confidence of 95%.
However, these days, it is very rare for market research samples to approximate to a random probability sample, undermining the ability of the research to create projectable results.
Although expert views vary, and have tended to become more relaxed over the last 50 years, it is suggested that if 30% or more of the population have a zero chance of being selected, the process cannot deliver the benefits of random probability sampling. In other words, the sample needs to randomly represent at least 70% of the population for it to deliver results that are projectable to the whole population.
There are three key reasons why market research samples tend not to achieve the required 70% level:
- Coverage Error. Coverage error expresses the degree to which market research are not able to reach the whole population. For example, if a research project uses CATI, those without a landline are not likely to be sampled, nor are those who are very rarely home. If the Internet is used to conduct the interviews, then those who do not have access to the Internet are not likely to be sampled. Similarly, in-street interviewing is unlikely to sample those who do not visit the high street.
- ·Non-response Error. Non-response error occurs if some of the people asked to take part in the research decline the invitation. In many market research projects the proportion who decline to take part can be well over 50%.
- Non-Completion Error. If people fail to finish the survey then in most cases their data won’t be included in the final data set, further reducing the link between the sample and the population.
A simple example can illustrate the cumulative effect of these three phenomena
Example of representation errors
A survey uses RDD (Random Digit Dialling) to contact people via telephone, with a sampling algorithm in place to select a suitable person in households with more than one person living there. In this market, the proportion of the population who live in a house with a landline is 80%*.
Of the people contacted, 25% were available and agreed to take part in the study.
Of the people who started the interview, 10% terminated the interview before the end .
The net effect of these three effects is that the sample reflects just 18% of the relevant population.
The example above illustrates that a method that is often assumed to be one of the more representative techniques, namely telephone. However, in this example, telephone falls massively below the level at which random probability sample can be assumed to operate.
*One of the challenges for telephone research in many markets is that the proportion of households with a landline is falling, with some Western markets having fewer than 60% of households with a landline. At the same time, the presence of mobile phones (i.e. cell or cellular phones) has boomed globally, with many developed countries having more mobiles then people. However, mobile phones are often considered unsuitable for RDD telephone research, due to respondent resistance and in some cases legislation.
In much of online research, the random probability sampling model is further corrupted by the use of access panels. An access panel typically comprises fewer than 5% of the population, usually much less than 5%. In terms of the logic above, this means that the coverage error becomes dominant. More than 95% of the population are not on panels and therefore have a zero chance of being selected.
Responses to the Sampling ProblemThere are a number of ways that market researchers have used in order to ameliorate the problems created by coverage, non-response, and non-completion errors. In many ways these could have been framed as part of New MR, however, they have tended to be used in conjunction with the language of ‘the science’, an approach which some have called disingenuous.
The most common way of dealing with the sampling problem has been to recruit a sample that resembles the population on a number of key characteristics. In general this approach is based on an assumption that if a sample shares specific characteristics with the population it will operate as a reasonable proxy for the population.
There are more sophisticated approaches to the random probability sampling problem, such as the propensity score weighting utilised by Harris Interactive (Terhanian 2008) and the use of CHAID by Brian Fine (Fine et al 2009). However, these approaches are not in general use, partly because of their requirements for high quality reference information and the increased processing implications.
People are Willing and Able to Answer Research Questions
In an ideal world market researchers would be able to ask respondents questions like:
“When this product is launched next year, how many times will you buy it?”
“On a 10 point scale, how attractive do you find this pack?”
“Does this TV commercial make think more highly of the brand?”
“If we change the colour of the glass from Brown to Blue, will you drink this Sherry more often?”
“What are your ‘drivers of satisfaction’ with your bank?”
From the early days of market research, it was understood that these questions were difficult for respondents to answer and various strategies were employed to improve the process, such as the use of scales rather than numbers, for example Likert scales and semantic differential scales. Another improvement was to use marketing science to reveal relationships, for example the calculation of implied drivers through the use of techniques such as regression.
However, more recently, the process of asking direct questions has been challenged by information gathered from other disciplines such as neuroscience, behavioural economics, and social anthropology.
For example, in Buyology, Martin Lindstrom (2008) uses neuroscience to illustrate that we do not know why we do things. In Herd, Mark Earls (2007) shows how our mirror neurons result us in buying what other people buy, not what we plan to buy. In The Decisive Moment, Jonah Lehrer (2009) talks about how asking people to rate a product before making a choice changes the choices that are made, and that these choices are less good than those made with using the rating scales. In Predictably Irrational, Dan Ariely (2008) shows that the context of the question changes the choices that are made in ways that appear to be both irrational, but also predictable.
The net position, taking this new body of information into account is that many of the methods used by traditional research are deeply flawed as a method of understanding people’s intentions and of revealing their likely future actions. To paraphrase Lindstrom, we are asking people who can’t remember where they left their car keys to remember what they were thinking and feeling when they bought that packet of toilet tissues a month ago.
Responses to Problems with the Traditional Research Model
Some of the problems with the traditional research model have been recognised for many years and a number of practices have evolved to ameliorate them. These practices include:
· The use of norms. The results of concept tests are rarely directly predictive of market shares, similarly the raw scores from voting intention studies are rarely predictive of the results of elections. However, the companies that specialise in this field have developed methods of using the results of previous studies to create more accurate estimates.
· Measuring relativities, not absolutes. This approach is often adopted by brand and advertising tracking studies. The absolute scores are largely irrelevant, the focus of the attention and the analysis are the changes over time. The implicit assumption is that the change that occurs in the sample is likely to reflect similar changes in the population.
· Test and control. In test and control research the sample is divided into two or more groups and different treatments are applied to the different cells. The implicit assumption is that the differences between the cells are likely to be reflected in the total population.
Whilst all of these strategies have proved useful, there is a concern that the companies offering these solutions have not made it sufficiently clear that the samples they are using do not approximate to a random probability samples.
Other Problems with Traditional Market Research
In addition to the structural problems outlined above, users of market research have raised a number of other issues.
Speed. Clients seem to be increasingly speaking at conferences and writing in magazines on the way that they find market research too slow. Clearly, some of the speed issue relates to client-side issues, such as commissioning the research too late. However, in many organisations the gap between wanting to know something, to needing to know it, to its too late to worry now, is very short, often a few weeks, sometimes a few days or hours.
Cost. The main problem is not the cost of the research that is conducted; it is the research that is not conducted because of the cost. Because of its many phases and procedures there minimum price for a research project is relatively high, discouraging many projects and increasing the temptation for companies to conduct DIY research.
Lack of Insight. Clients complain that too many research debriefs and results simply describe the process and enumerate the responses. The complaint is that often the results do not facilitate beneficial changes in the business/marketing plans or operations. One of the reasons for this failure appears to be that the results do not engage and convince the people who need to make the decisions.
Interesting. I'm sure there's a far older literature on buying behaviour/voting behaviour, because I've always been very well aware of the qualititative maxim 'people don't always say what they mean or mean what they say': asking a direct question (in a qualitative context) is often one of the worst things you can do, for all the reasons mentioned.
The problem of questioning though, is a fundamental difficulty. There are lots of projects where it's damned difficult to know how to ask the question in a way that will generate meaningful and stable responses.
Posted by: Alison | November 18, 2009 at 10:04 AM