The Future Place Blog

Swedish Research Day, March 11, Stockholm

I am delighted to announce that I have been invited to speak at this year's Swedish Market Research Day (SMRD) which is going to be held in Stockholm on March 11th. I will also be running a workshop on March 10th, looking at wide range of advanced marketing science techniques, including conjoint analysis, MaxDiff, CHAID, and cluster analysis.

I am always delighted to be invited to the SMRD as it is a really leading edge event, however, it is also a challenging event an interesting contrast of talent between the always excellent local presenters and the invited guests.

As soon as I have links to the conference and for the workshop I will put them on the home page of my website, but until then you can find more information here.

Why market research questions don’t do what we think they do

I had a great time on Wednesday attending the NetGain4 Conference in Toronto and was honoured to be one of the keynote speakers. One of the points I was making that many of the questions that market researchers have been using for years have turned out to be very flawed. To borrow and mangle a phrase from Mark Earls, we are very poor witnesses to our own beliefs and intentions.

I think one of the key memes for the next couple of years is going to be triangulation, i.e. can we find alternative sources that all point to the same conclusion. In terms of the concern about many types of market research questions then I recommend the following four books:

Herd, by Mark Earls.

Buyology, by Martin Lindstrom.

How We Decide, by Jonah Leherer.

Predictably Irrational, by Dan Ariely.

I do acknowledge that many market researchers have reservations about Lindstrom’s methodology and conclusions, however I think he is important to include as part of the triangulation. The books above draw upon behavioural economics, neuroscience, anthropology, psychology and more. The net conclusion is that people are very bad at saying what they personally would do in hypothetical situations, very bad at saying why they do things, and their choices are often changed by the questions we ask them.

Note all of the links are to Amazon, other book stores are also available.

Price pressure for communities in 2010

I think 2010 is going to be a very interesting year, and 2011 as well for that matter. I think we will see some strong growth in MROCs and community panels, but I think we will also see growing, downwards pressure on price.

The demand is already growing, communities are the flavour of the month/year, the early adopters are mostly all in and the industry is seeing the early majority beginning to get involved. However, this will be a challenge for the providers because these new buyers are there for the ROI not the thrill, that makes them different clients, with different needs, and much less 'let's try this and see', and much more 'I need it to return value from day one'.

More software platforms are becoming available and the existing ones are getting better (if you do not already have a system, this is not the time to start writing one, buy or rent).

More agencies are trying to get into the community space because it is growing and because it is attracting all the attention, some will sell below cost rather than be left behind, especially if they do not understand the true cost of moderating a community.

So, to summarise, more buyers, but even more suppliers, some of whom are desperate to make a sale. I think prices will be under pressure, right through 2010 and 2011. However, clients beware, with the new software available, the clever bit is not creating a community, it is not even in populating it, it is in turning the community into insight, and that will not get cheaper in 2010.

If you are a buyer and you are offered a low price, try to understand where the savings are coming from, if the savings are coming from reduced levels of moderation and interpretation, it will not be a great bargain, even if it is a great price. 

WE-Research, the Rise of Citizen Researchers

The idea behind the citizen researcher is that participants are recruited to work with market researchers as collaborators to explore lives, products, and services. This concept has been described as a shift from ME-research to WE-research (Earls & Kearon 2009).

The concept of citizen researchers is associated with the ideas and techniques of user generated media, for example webcams, mobile phones, and personal video devices. Citizen research reflects a demand amongst people to be involved and recognition that the tools to make this demand actionable are now widely available.

‘Beware the Umpa Lumpas’

In their 2009 ESOMAR Congress paper, John Kearon and Mark Earls outline a citizen ethnography project, looking into the topic of binge drinking in the Northern UK city of Newcastle. Earls and Kearon cite their approach as being informed by the cultural anthropologist and ethnologist Grant McCracken. 

The approach described by Earls & Kearon comprised two weekends, one where the instructions to their citizen researches was relatively light, which produced poor results. The second weekend was based on a better briefing and the use of example feedback. This second week was more successful and provided insight into the role of fancy dress in providing a focus for heavy drinking. (During that particular weekend there were people out and about in Newcastle dressed as Umpa Lumpas, a character from the popular book/film Charlie and the Chocolate Factory).

Mass Observation

Like many new ideas, the concept of the citizen researcher has a long history. One of the best examples of the long history of citizen researchers is the UK’s Mass Observation project, which ran from 1937 through to the 1950s (and was re-started by the University of Sussex in 1981). The Mass Observation project was launched by anthropologist Tom Harrisson, poet Charles Madge, and film-maker Humphrey Jennings who wanted to create an ‘anthropology of ourselves’.

Mass Observation consisted of about 500 volunteer observers who wrote notes about what they saw and heard, either as diaries or in response to open-ended questions. A representation of the life of a volunteer observer was captured by the UK TV drama ‘Housewife, 49’, which was based on the wartime diaries of Nella Last, whose diaries have also been published as two books, "Nella Last's War: The Second World War Diaries of 'Housewife 49'" in 2006, and "Nella Last's Peace: The Post-war Diaries of Housewife 49", in 2008.

To find out more about the Mass Observation archives and about the current incarnation of the project visit the Mass Observation website [http://www.massobs.org.uk].

EDL App – A Tool for Citizen Researchers

Siamack Salari and EverydayLives launched a personal ethnography iPhone app in December 2009, available from iTunes, allowing anybody to turn themselves and their iPhone into a citizen researcher. The app allows citizens to capture video on their iPhone, annotate it, and upload it to a project sponsor’s site.

The company describe the app as ‘a sophisticated research tool designed by ethnographic researchers for ethnographic researchers, agency planners, marketers and research respondents.’ Although it looks as though most of the early users of the app will be professional researchers, this app will lower the barriers to entry and increase the prospect of self-employed, citizen ethnographers.

Ray's Prediction?

I suspect that the trend towards online research communities and community panels will pick up the WE-Research thread. Recruiting citizen researchers from communities will be quick and easy, and people with a talent will soon become apparent. 

In the future the fastest and deepest way to answer some problems will be to enlist community members to go out a search for the answer and to report back.

---

ps apologies to Mark for getting his name wrong, again! I will try better, I will  try better, I ........

So, Why Does Market Research Usually Work?

In the chapter on NewMR I talk about that traditional quant research does not work in the way they market research normally implies it works. I point out two major reasons for this:

1. Commercial market research almost never uses samples that approximate to random probability samples.
2. Many of the questions we ask can't be answered (see Lindstrom, Ariely, Earls, and Lehrer).

Note, this explanation in the book is somewhat fuller!

However, I then discuss why market research seems to work most of the time, which is the bit below. I would really appreciate hearing people's views.

---

So, Why Does Market Research Usually Work?

If the sample is usually the wrong sample and to questions often don’t reveal why people do what they do, why does market research usually work? 

The claim that market research usually works can be supported in two ways. Firstly, there are those occasions where the research can be tested. Probably the best example of this phenomenon is election polling, where the polls are usually correct, in an area where sampling would appear to be critical, and where respondent’s might not only not know their intentions but also might not be prepared to disclose them. The second illustration is the scale of shock and public discussion when market research really does get it wrong, such as the New Coke fiasco in 1985.

A second factor that needs to be considered what we mean when we say market research works. At best market research can only be a probabilistic risk reduction system. It seeks to reduce the number of Type I and Type II errors (as statisticians call them). A Type I error is a false positive, i.e. bad business decision is not spotted and allowed to go ahead. A Type II error is a false negative, i.e. a good idea is flagged as a bad one. For market research to work the cost of the market research needs to be sufficiently below the benefits generated by the reduced chance of Type I and Type II errors. Nobody should suggest that market research removes the risk of either Type I or Type II errors, it works by reducing the risk.

Note this section is written in the first person to highlight that it is the author’s conjecture, supported by 30 years of working with the data and extensive research, discussion, and attending conferences.

I think that there may be many reasons why market research tends to work, but I feel the key reasons are as follows:

•    Homogeneity.

•    Wisdom of crowds.

•    The ‘art’ or market research.

I explore these three reasons below.

Homogeneity

As Mark Earls has illustrated (in his book Herd), people tend to do what other people do. People who manage to function in society tend to be like other people. If we show a three TV commercials for breakfast cereal to 100 men aged 20 to 35 years old and to a sample of 100 females aged 36 to 50, then the commercials will have to be very unusual for them to be rated differently, at least in terms of the relativities between the three commercials. If Ad A is seen as the most fresh and modern by the men, then it will usually be seen as the most fresh and modern by the females. If three ads were about female sanitary protection, then the response might well be different. A random probability sample protects against the sample creating bias, but in most cases having the wrong sample does not create, in the real world, biases that change the relative results between items being assessed.

Note, there is no rule that says the wrong sample will work OK, but it often does, based on empirical data.

Wisdom of Crowds

Many years ago I worked with an automaker who had a very large, very extensive database of what cars people said they would buy next and what they actually bought next. The interesting thing about this data was twofold:

a.    The car a somebody said they would buy next was a really bad predictor of what that person did buy.

b.    However, the aggregate totals of what people said they would buy was a good predictor of the aggregate total that people bought.

What this data showed was that people were bad at predicting their own behaviour, but really quite good at predicting what groups of people will do. If we consider the prediction market methodology this is exactly what the respondents are being asked to do, i.e. estimate what other people would do.

My conjecture is that when a market researcher shows somebody an ad and a concept for a new type of beer and asks them how likely they are to buy it when it, the respondent is unconsciously conflating some mixture of their own likelihood with some estimate of what ‘people’ like them might do.

The ‘Art’ of Market Science

Market researchers have been taking what respondents say and interpreting them for years. Few market researchers would simply take what a respondent says and report it as a finding. When Lindstrom and others show that what people say is not a good guide to what they will do they are not really saying anything new. Dealing with this issue has been part of market research's stock-in-trade since its infancy. For example, sales estimates from respondents are based on what the respondents say, but they are typically converted using algorithms based on the sort of changes necessary to convert a set of respondent scores into metrics that have some link with the real world.

Summary

My net position is that market research often works, but not in the way that it is often claimed. However, by not focusing on the real reasons that market research works, people run the risk of being surprised when it does not work.

Don't quote social network comments verbatim

Just working through some notes on ethics and social networks and I thought I'd share the following paragraph that I've been working on - because it seems counter-intuitive at first.

There is a growing consensus that comments posted in social networks are not ‘in the public domain’ and that researchers should seek permission to use them. Researchers should also remember that because the internet is so readily searchable, they should avoid using literal quotes from social network discussions (in most cases) as this will potentially reveal who the respondent is.

In many codes of ethics and in a growing number of laws, the intention/expectation of the person making a post is important in determining what can be done with that post. In terms of privacy there are two issues. The first is that if a researcher has to join a network to see the post, then the person making the post is doing so in the expectation that they are talking to genuine members of the community, not to professional researchers or journalists. The second that is when somebody makes, say, 200 posts in their status bar over the course of a year, they did not have the expectation that all their quotes would be brought back together as a single corpus for investigation.

i suspect that over the next four years the research industry is going to have to move very quickly to keep up with a cascade of new ethical and practical issues.

Survey Engagement, Solution or Fig Leaf?

I have been reading up on survey quality and engagement quite a bit recently, including comments by Survey Sampling’s Pete Cape, two great studies by Bernie Malinoff, research by GMI’s Jon Pulleston and listening Joel Rubinson talk about the results of the ARF ORQC research, it is clear that bad surveys lead to bad data.

However, the problems in descending order of importance are:

1. Long surveys, this is the number one problem.
2. Boring and incomprehensible surveys
3. Surveys that lack engagement.

Whilst very little is being done to address long surveys and badly structured questions, the term respondent engagement seems to be on everybody’s lips.

Don’t get me wrong! I do think we should be looking at ways of making surveys more engaging and in particular more intuitive. But as Bernie Malinoff has shown, pretty does not always work better, so plenty of checking is called for. I think some of Jon's good results at GMI are because he has made the instruments more intuitive.

However, I think the key point is the one that Reg Baker makes when he talks about the changes to the look of surveys as being like putting lipstick on a pig. If the survey is still too long and the questions are still incomprehensible or are still bad psychometric instruments, then it is still a pig.

Facebook is in a different league

A couple of years ago I tended to assume that Facebook was simply the fad for the year and that something else would soon be along to be the next thing. However, that has not happened and Facebook is building every stronger legacy links that will make it hard to shift.

One of my favourite pastimes, when I should be working, is to play with Google Insights to looks at trends. Insights lets you track the search terms that people have entered into Google.

The chart below shows a comparison for FaceBook, MySpace, and Twitter.

 

For those who think Twitter might be the next Facebook check out the table below which suggests Twitter may have stalled, which tends to match data about the number of users.

 

Determining Whether Online is a Suitable Modality for the Project

Another snippet from the book to test public opinion and my own shortcoming! All views gladly received.

---

The process of deciding if method is appropriate should apply equally to any data collection modality, but it is a topic more frequently discussed in the context of online research. 

In markets where online research is already established this step tends to operate as a negative test. Because online research is typically faster and cheaper the question often becomes is there a reason not to use an online modality.

Possible reasons not to use online include:

•    If online is slower or more expensive.

•    If the stimuli can’t easily be rendered online.

•    If a study is highly confidential.

•    If the target sample is not online.

•    If the research topic is likely to interact with the online medium.

•    If there are comparability issues.

•    If the study requires a random probability sample.

These reasons are expanded on below.

If online is Slower or More Expensive

Generally online modalities are cheaper and quicker than other modalities, but there are exceptions. The main exception is where a survey is very short and simple, and where the sample size is very small. For example 100 interviews, with housewives, asking 10 simple questions, might be faster and quicker via telephone. In some markets even face-to-face might be cheaper and/or quicker.

If the Stimuli can’t Easily be Rendered Online

There are a wide range of stimuli that are not suitable for online surveys, for example taste, touch, and smell. Most sensory testing still has to be conducted face-to-face. Some research organisations have, however, had substantial success with posting products and other test materials to respondents, especially to panel members.

Over time the number of things that can be rendered online has increased, although this sometimes has implications for the sorts of PCs the respondents need to have and the bandwidth available.

If a Study is Highly Confidential

There are steps that can be taken to make it harder for respondents to download and share material from the survey. However, the risk cannot be completely removed, for example somebody can always use their iPhone or similar to record what they see on the screen.

Equally, it should be kept in mind that even face-to-face interviews are not 100% secure, especially with the growth of smartphones.

If the Target Sample is not Online

Even In markets where the Internet penetration is very high (such as the Scandinavian countries where the penetration is close to 90%) there are systematic differences between those who are online and those who not. Generally, the people who are not online tend to be older, have lower incomes, and tend to be more disadvantaged in terms of features such as housing, employment, and education opportunities.

If the purpose of the research is to explore a group that tends not to be online, then the online modality will tend to be unsuitable. For example, attitudes towards the internet would not normally be a suitable topic for an online survey.

If the Research Topic is Likely to Interact with the Online Medium

Some topics are too closely associated with the internet as a medium to be researched via the internet. For example, a study of how people buy music CDs in stores is likely to distorted by the fact that people who are not online buy all of their CDs in stores, whilst people online are likely to buy some or all of the CDs online, to download music, and to compare stores with online purchases.

If there are Comparability Issues

The issue of moving a study online and possibly losing comparability is dealt elsewhere in the book, but the loss of comparability with previous studies can be a reason that a study should not be conducted online. 

If a study is being run in more than one country, online may not be available in all of the markets, so there may be a decision to use a methodology that is available in all the markets. However, it should be noted that in many cases there is no suitable modality that is available across several markets. When conducting a general consumer study, online and telephone are not suitable for India, and door-to-door interviews are not suitable in Australia or the US, so compromises often have to be made, especially to minimise costs.

If the Study Requires a Random Probability Sample

Unless the whole population has internet access and the whole population can be identified then a random probability sample cannot be drawn. Examples of where these conditions are met include undergraduates at many universities (where the students all have .ac email address) and also visitors to a specific website or online service, for example eBay could sample eBay users.

Public sector work sometimes requires that random probability sampling is used as does some work designed to be used in court cases. However, even in these cases the researcher should remember that many authorities believe that if the response rates to a survey are below 70% the study does not meet the criteria for a random probability sample, even if the initial sample was OK.

Selecting a Panel Supplier - Costs

At the moment, the book, I am looking at how to select and work with an online access panel supplier.

One of the criteria that researchers have to consider is cost. My thoughts are set out below, what are your thoughts? 

---

When requesting quotations from panel companies there are two key guidelines:

• Give the panel company as detailed and accurate a specification as possible. If the survey ends up being 15 minutes instead of 10 minutes, or if the incidence ends up being 5% instead of 20%, the final cost is likely to be higher than the original estimate or the number of completed surveys lower.
• Ensure that the quotation from the panel company provides a total price, including any incentives and handling charges.

In terms of combining quality and costs the four steps below will balance quality and cost:

1. Rule out any panel that does not meet a quality threshold.
2. Request quotations from the panel companies in the consideration set, or a subset of them.
3. Review the feedback from the panel companies, they may be recommending changes to the project specification, and it is worth considering their suggestions.
4. The final decision lies somewhere between choosing a) the cheapest of the suppliers who meet the quality threshold and b) the best sample the project can afford.

Because a research agency is the intermediary in the research process, the final arbiter of the trade off between benefits and costs is the end client. 

When assessing the cost of working with a panel company the researcher should consider how easy the panel is to work with. If the panel company is responsive and deals with things without being chased then they are cheaper to work with than a provider who needs constant chasing and whose work needs constant checking. Panel prices of $1 more per respondent may not cost more if the service level is much higher.

---

BTW, in choosing a panel company I have 6 headings, and price is just one of them.