The Future Place Blog

A chance to brush up your quant

On Thursday NewMR has its final topic specific event before the Festival, an introduction to a range of Advanced Quant techniques, including pricing research, predictive markets, choice modelling, structural equation modelling, and more.

The three sessions are outlined below, you can register to attend at NewMR.org

Asia/Australia Session - Sydney 3 pm to 5.15 pm (Auckland 5pm, Singapore 1pm, Mumbai 10.30 am)

• Sue York, An Overview of Advanced Quant Techniques
• Ray Poynter, The Future Place, The Benefits of Conjoint Analysis and Choice Modelling
• Brian Fine & Con Menictas - The Superiority of Panel Research - A Fast Food Choice Modelling Example
• Scott MacLean, An introduction to Reflective and Formative Structural Equation Models

 

Europe/Africa Session – London 11 am to 1.15 pm (Mumbai 3.30 pm, Paris 12 Noon, New York 6 am)

• Ray Poynter, The Future Place, Improved BPTO
• Dirk Huisman, SKIM, The Challenges and Benefits of menu based choice modelling
• Stefan Binner, Adaptive Choice Based Conjoint
• Will Goodhand, BrainJuicer, 9,000 concepts later: how the wisdom of the crowd compares with monadic screening

 

Americas Session – New York 2 pm to 4.15 pm (London 7 pm, Seattle 11 am)

• Ray Poynter, The Future Place, An Introduction to Hierarchical Bayes
• Juan Tello, SKIM, Product Optimisation
• Jeffrey Henning, Affinnova, Survey-based pricing research techniques
• Will Goodhand, BrainJuicer, The pursuit of Happiness: can emotions predict in-market performance?

 

Event Sponsor – Affinnova

The Advanced Qaunt event has been sponsored by Affinnova and you can see an interview with Tamara Barber from Affinnova below.

A survey of quant research techniques

NewMR has launched a survey to look at what quant research techniques are in use. The results will be presented by Sue York at the NewMR Advanced Quant Event on July 14^th and will be written up as a brief report, which will be published here and on the NewMR.org website.

To take part in the survey click HERE.

After you complete the survey you will be taken to a page with a link to the top-line results so far. At the moment the average time taken to complete the survey is 7 minutes.

The main purpose of the survey is to provide some feedback on what techniques are in heavy use, which in occasional, and which are truly specialised. We hope that this information will be helpful to people planning their career development. For example, all but the most determined qual researchers should look to be comfortable with the most frequently used techniques, anybody wanting to specialise on quant research should probably aim to be reasonably comfortable with the middle group, and only those with a passion or interest needs to feel that that should master the less commonly used techniques.

 

Current draft of the Quant Surveys

We are getting fairly close to finishing our survey about quant techniques. If you would like to have a look and perhaps send me some suggestions you can see it at http://surveyanalytics.com/t/AGXJqZKlPo

Note, at present it does not have forced completion turned on, this is to facilitate testing.

Suggestions for our survey on Quant Research

Our plans for a survey of how often different sorts of quant techniques are used has moved forwards and we appreciate the people who have sent us suggestions. Survey Analytics have allowed us to use a free account for the project and offered tech support, so many thanks to them for that.

Our thinking at the moment is that the survey will be organised into grids, with each grid having a theme and a range of likely frequencies within the grid. For example one grid might look Association techniques and might include correlations, cross-tabs, Cluster analysis (e.g. K means & Hierarchical) , Latent Class analysis, partial correlations, and Decision Trees (eg CHAID and CaRT).

Another group might look at Dependence approaches and list the regression and SEM type options. We do not necessarily plan to label these groups in the survey, as some people would put them in different places, for example we are probably going to put correspondence analysis with graphical techniques such as perceptual mapping, but it could be in association, similarly latent class could be in the dependence group. But we feel that putting the techniques into groups will help the respondents.

We have received several suggestions that we ought to include analysis packages. We have taken this idea on board, but we want to avoid it being simply a package ranking exercise, so our thinking is to have something like the following: SPSS, Systat, Excel, Sawtooth, Latent Class software (eg Latent Gold), SEM software (eg Lisrel & AMOS), R, Crosstab software (e.g. PSTAT, WinCross), integrated data collection/analysis software (e.g. CfMC, Confirmit, Survey Analytics). 

We will have a type-in other in each grid and we will monitor the first 30 or so completes to see if we have missed anything vital.

We are planning to limit the demographics to just 2 questions 1) Country, 2) type of organisation (supplier, buyer/user, academic, other), with one open end.

One option we are considering is only asking each respondent to do a sub-set of the grids, but this makes cross-tabulations of techniques across the groups less trivial and reduces the bases for those cross-tabulations - e.g. if each respondent sees half the grids the average base for cross-tabs between a technique in one grid and another is just 25% of the total base. 

So, please do continue to send us your suggestions by the end on Monday 27th, and we will launch the survey on Tuesday 28th.

We are hoping to be able to provide live updates to the results, which should make it more interesting for everybody.

 

What quant techniques are being used in market research?

Back in 2009 I published a short study looking at what quant techniques were in use. The sample was a convenience sample of people connected to me via social media, but the findings were nevertheless quite interesting (Download Quantitative Research June 2009).

The Future Place is about to repeat the study and we would love your suggestions for improving the range of topics covered. Please suggest techniques you think we need to cover by making a comment below or by emailing [email protected].

We are aiming to circulate invites to the study 28 June and keep the survey open for about one week.

The report will be made available via The Future Place blog, via NewMR.org and will be presented by Sue York at the NewMR Advanced Quant event on 14 July.

 

Gongos Research have NOT proved the validity of smartphone research

OK, somebody had to blink! For the last few days there have been several tweets and comments using the phrase “Smartphone surveys prove their validity in marketing research”. When I saw the first post I immediately classed it as harmless hyperbole. But when I saw that MSNBC had used the same headline (14 June 2011) I felt I had to shout out “The Emperor has no clothes”.

The stories seem to track back to a press release by Gongos Research, released on 14 June 2011. The press release includes the phrase “a new study proves that smartphone-based survey data is statistically comparable to online survey data.”

This claim in linguistically and methodologically ludicrous, a single study cannot prove that something happens, it can show that it sometimes happens, but that does not prove a positive. A single, well designed, study can show that something is NOT true. For example, the claim that all swans are white is disproved by finding one black swan, but finding one more white swan does not prove all swans are white.

If we assume, for the moment, that the press release means that Gongos Research conducted a study, looking at a wide range of questions types, in one or more markets, with one or more types of customers, and that the study showed the results were acceptably similar, then we would not have proof of the validity of smartphone research. We would have evidence that smartphone surveys can sometimes work. If more tests are conducted (and ideally other agencies should be willing to pay for them) we can start to find out if smartphone surveys are ‘often’ (or better still ‘usually’) acceptably similar to other modalities. If we had the results of say 20 studies and they all showed that smartphone research was acceptably similar to other modalities we would not have proved it, but we would feel reasonably confident about using it, even 10 out of 10 positive results would make us pretty happy about trying it for a live study.

I do hope that Gongos Research publish their research as it is potentially really helpful to the industry, as evidence about the viability of smartphone research.

 

Why an 82% survival rate may not be better than a 44% rate?

This week’s New Scientist has a really useful, informative, and important article about the danger of creating and reporting misleading statistics, focusing on health statistics, but the same points are true of any use of statistics, for example market research.

To read the full article you will need a copy of the magazine or you’ll need to register with them online, but I strongly recommend reading it.

One of the panels in the article talks about the specific dangers of invalid comparisons and uses as its core the implication made by Rudy Giuliani in 2008 that the UK’s National Health Service was twice as bad as the US in tackling prostate cancer, based on the fact that the US survival rate was 82% compared with the UK’s 44%.

Below is a short précis of the points made by the authors (Marianne Freiberger and Rachel Thomas), with a view to presenting the main points, but I recommend reading the original.

The figures Giuliani quoted were correct, but they give an inaccurate impression.

The rates quoted by Giuliani quoted were five year survival rates. Survival rates are based on the length of time from diagnosis to death. Diagnosis is the first source of problems, in the US most diagnosis is the result of screening, in the UK most cases are diagnosed on the basis of men who have symptoms. This means the cases in the US include more false positives and many cases that would never have progressed to symptom (i.e. are unlikely to have caused death). Freiberger & Thomas quote a study that estimates about 48% of men diagnosed by screening do not have the progressive form of the cancer.

Freiberger & Thomas assert that to compare countries one needs to compare mortality rates, i.e. the proportion of the whole population who die of a specific cause. For the US the mortality rate for prostate cancer is (according to the authors) 24.7 per 100,000, in the UK the rate was 23.9 per 100,000 – effectively the same number.

Another illustration of the problem with comparing results was given recently by criticisms (for example in the Guardian) of UK Prime Minister’s use of health statistics to use the age adjusted mortality rates between France (19 per 100,000) and the UK (41 per 100,000). This time the concerns include the fact that the UK is more inclined to define deaths amongst older citizens as being caused by a heart attack, in France the category old age is more common than in the UK.

Whether we are talking about health statistics or marketing data we need to be clear what the base for a comparison is in order to be able to safely use the information. The New Scientist article also looks at the difference and confusion between correlation and causation.

 

Should we use sig testing with online panel studies?

There is a growing acceptance, amongst market researchers, that we should consider online access panels as convenience samples. We cannot assume that a panel approximates to a random probability sample of any naturally occurring population. Although a panel might reflect population demographics there is no reason to believe that the panel members reflect a wider population’s views, beliefs, or intentions.

The traditional purpose of significance testing has been to estimate the probability that a sample reflects values from the ‘true’ population. This purpose was backed by the ‘scientific’ notion that the sampling errors would be normally distributed and that the significance testing process would be an indicator of the validity of the results.

Since the access panel does not approximate to a random probability sample of the wider population we cannot assume that significance testing reflects validity. So, should we continue to use significance testing with online access panel data?

My answer is yes, but that we should change our mind-set away from validity and towards reliability.

To make this change from validity to reliability, consider what happens when we draw a sample from a panel and when we conduct significance testing. The sample does not reflect the wider population, but it is still drawn from a population, i.e. the population is the set of people on the panel who meet the selection criteria.

When we conduct significance testing on a sample from an online panel tests, the tests show the probability that the results are typical of the panel. If the results are not significant then there is an unacceptable risk that running the same test again (on the same panel) will produce different results. If there is an unacceptable risk that conducting the same study twice (at the same time, on the same panel) will generate different results, then the results cannot be considered reliable (which for many people means they should not be reported).

So, I say yes, conduct significance testing on studies conducted with online access panels, but change the way you describe the results. Do not say things like “These results are accurate to +/-3% with a probability of 95%”. Perhaps use a form of words such as “The test/retest reliability of these results, from this source, are +/-3% at the 95% confidence level.”

I will be writing more on the science and ritual of significance testing and what determines reliability in later posts, along with the suggestion that we should consider use the 80% level of confidence when reporting reliability.

 

What is marketing science?

For those of you on LinkedIn there is a great debate on this topic in the Marketing Science group.

Taking part in that discussion has certainly sharpened my thoughts about what marketing science is and what it could be (along with topics such as what do we mean by a ‘scientific approach’ and what is the role of different epistemologies in market research).For me the key things that have arisen out of the debate so far are:

1) Market research is not a science, just as medicine and engineering are not sciences. Market research can use science, just as medicine can be the application of science and engineering can be the application of science.

2) Some market researchers believe that all market research (quant and qual) can/should employ a scientific method, for example creating and testing hypothesis, seeking cause and effect, and trying to produce findings that can be replicated.

3) Some market researchers, particularly some of those who specialise in qualitative approaches, do not subscribe to an epistemology that embodies the scientific method, for example they may take a constructionist position.

4) Market research is part of marketing, but only a fairly modest part.

5) Marketing science could be a useful term for identifying that part of market research which uses non-trivial, mathematically or statically based techniques in ways consistent with what is generally understood as the scientific method to provide marketing-related insight.

6) Whist the boundaries of marketing science might always be hard to define the core of the domain should be relatively clear, for example discrete choice modelling, structural equation models, and regression.

7) The core of marketing science should, in my opinion, be expanded to include all mathematical/statistical techniques used by marketing, for example web analytics, churn data, network analysis, etc, not just those traditionally used by market researchers.

One issue that arose in the debate is the point that most clients are not well equipped to query the nature of the advanced techniques they might be presented with, due to time, interest, training, or aptitude. This situation, in my opinion, makes it all the more important that researchers work hard to define what they mean by marketing sciences and to define what they mean by good and best practice.

What are your thoughts?

 

 

 

Recordings of my Cluster and Factor Analysis Webinars

I recently ran webinars on Factor and Cluster Analysis as part of the preparation for the Festival of NewMR.

Below are recordings of these two webinars. They are a bit rough, but better than the versions I initially posted.

Cluster Analysis

 

Factor Analysis