The Future Place Blog

Mobile Research: Great hope or false dawn?

There are a growing number of optimistic signs about mobile research, many of which will be showcased at the 2^nd Merlien/ Greenbook Market Research in the Mobile World Conference in Atlanta on 19/20 July. One of those signs was an encouraging set of research results from Gongos Research.

However, as I have pointed out before, mobile has been the next big thing for about 15 years, and although the next 18 months ‘may’ see the arrival of mobile as a major player, I doubt that it will happen as quickly as the advocates believe.

To help shape the discussion on this topic Greenbook and NewMR have organised a virtual debate and everyone is welcome to join us on 7^th July (noon New York time).

The panel will comprise ‘believers’ Leonard Murphy and Michael Alioto, the voice of reason (AKA Reg Baker), and sceptic Ray Poynter.

You can register for the event from the NewMR website. The event will last about one hour and is entirely free.

 

Comley Report focuses on four aspects of ‘Me’

Pete Comley, the Chairman and founder of Join the Dots (nee Virtual Surveys), has issued the first edition of his Comley Reports. The report is a carefully thought through, and referenced, synthesis of several strands of thinking, combining political, economic, social, and technical themes to produce an output that provides a useful backdrop for both marketers and market research.

The core of the report looks the four aspects of ‘Me’, namely My Price, My Time, My Self, and My Crowd. The first two of these cover well-trodden ground in terms of how people’s lives are moving from time-poor but cash-rich, to a complex need to ration both time and money, with the report moving on from the facts to the implications.

The second two, My Self and My Crowd, speak to two trends that create a dualism in society, between the priority we place on self, whilst marrying that with the connected/wired society that is emerging from the atomised society, which in my opinion arose out of Thatcher and Blair’s attacks on traditional society and communal values.

A copy of the Comley Report can be requested from the Join the Dots website.

(Quick declaration of interest, I hold equity in Join the Dots and from time-to-time do consulting work for them)

 

A survey of quant research techniques

NewMR has launched a survey to look at what quant research techniques are in use. The results will be presented by Sue York at the NewMR Advanced Quant Event on July 14^th and will be written up as a brief report, which will be published here and on the NewMR.org website.

To take part in the survey click HERE.

After you complete the survey you will be taken to a page with a link to the top-line results so far. At the moment the average time taken to complete the survey is 7 minutes.

The main purpose of the survey is to provide some feedback on what techniques are in heavy use, which in occasional, and which are truly specialised. We hope that this information will be helpful to people planning their career development. For example, all but the most determined qual researchers should look to be comfortable with the most frequently used techniques, anybody wanting to specialise on quant research should probably aim to be reasonably comfortable with the middle group, and only those with a passion or interest needs to feel that that should master the less commonly used techniques.

 

Current draft of the Quant Surveys

We are getting fairly close to finishing our survey about quant techniques. If you would like to have a look and perhaps send me some suggestions you can see it at http://surveyanalytics.com/t/AGXJqZKlPo

Note, at present it does not have forced completion turned on, this is to facilitate testing.

Suggestions for our survey on Quant Research

Our plans for a survey of how often different sorts of quant techniques are used has moved forwards and we appreciate the people who have sent us suggestions. Survey Analytics have allowed us to use a free account for the project and offered tech support, so many thanks to them for that.

Our thinking at the moment is that the survey will be organised into grids, with each grid having a theme and a range of likely frequencies within the grid. For example one grid might look Association techniques and might include correlations, cross-tabs, Cluster analysis (e.g. K means & Hierarchical) , Latent Class analysis, partial correlations, and Decision Trees (eg CHAID and CaRT).

Another group might look at Dependence approaches and list the regression and SEM type options. We do not necessarily plan to label these groups in the survey, as some people would put them in different places, for example we are probably going to put correspondence analysis with graphical techniques such as perceptual mapping, but it could be in association, similarly latent class could be in the dependence group. But we feel that putting the techniques into groups will help the respondents.

We have received several suggestions that we ought to include analysis packages. We have taken this idea on board, but we want to avoid it being simply a package ranking exercise, so our thinking is to have something like the following: SPSS, Systat, Excel, Sawtooth, Latent Class software (eg Latent Gold), SEM software (eg Lisrel & AMOS), R, Crosstab software (e.g. PSTAT, WinCross), integrated data collection/analysis software (e.g. CfMC, Confirmit, Survey Analytics). 

We will have a type-in other in each grid and we will monitor the first 30 or so completes to see if we have missed anything vital.

We are planning to limit the demographics to just 2 questions 1) Country, 2) type of organisation (supplier, buyer/user, academic, other), with one open end.

One option we are considering is only asking each respondent to do a sub-set of the grids, but this makes cross-tabulations of techniques across the groups less trivial and reduces the bases for those cross-tabulations - e.g. if each respondent sees half the grids the average base for cross-tabs between a technique in one grid and another is just 25% of the total base. 

So, please do continue to send us your suggestions by the end on Monday 27th, and we will launch the survey on Tuesday 28th.

We are hoping to be able to provide live updates to the results, which should make it more interesting for everybody.

 

What quant techniques are being used in market research?

Back in 2009 I published a short study looking at what quant techniques were in use. The sample was a convenience sample of people connected to me via social media, but the findings were nevertheless quite interesting (Download Quantitative Research June 2009).

The Future Place is about to repeat the study and we would love your suggestions for improving the range of topics covered. Please suggest techniques you think we need to cover by making a comment below or by emailing [email protected].

We are aiming to circulate invites to the study 28 June and keep the survey open for about one week.

The report will be made available via The Future Place blog, via NewMR.org and will be presented by Sue York at the NewMR Advanced Quant event on 14 July.

 

Interview with the Merlien Institute

I have taken part in two Merlien Institute events and found both of them really enjoyable, and really useful. The events tend to be smaller than those run by the trade bodies, less commercial than those run by the typical conference organisers, and combine social and academic strengths.

In order to share more information I asked Jasper Lim, the man behind the Merlien Institute and a member of the Festival of NewMR’s 2010 Advisory Board to share some information about the Merlien Institute.

In 50 words, what is the Merlien institute?
Merlien Institute is a knowledge facilitation organisation dedicated to the advancement of research methodologies and techniques in academia, business and policy environments. Our mission is to provide an effective platform for academics and practitioners to innovate, discuss and debate best practices. Specifically, Merlien Institute aims to:

• Cross-fertilize knowledge between academics and practitioners
• Promote the use of scientific methods in market research
• Influence policy makers' utlilisation of qualitative research findings
• Help source reliable research partners for clients through our trusted research referral service

Can you tell me a bit about the history of Merlien, how you got where you are today?
Merlien Institute was founded by Jasper Lim in 2006 while he was still a researcher at Delft University of Technology in The Netherlands. He realised that most of the conferences he attended were boring and had a lack of focus and passion. He, along with his fellow researchers decided to design an event that is affordable and will allow delegates to inject their passion during the conference. It took them about two years to design, prepare and organise their first event entitled "Computer-Aided Qualitative Research" held in 2008 in Amsterdam. The conference was a resounding success and follow-up events in various topics were held and expanded in Europe, Asia and North America.

The Singapore office was established in 2009 to help support the growing demand for Merlien Institute's events in the Asia Pacific region. A trusted research referral service was also established in 2011 to assist European and North American companies source reliable and trustable research suppliers in Asia. Merlien Institute, with its 23-strong Advisory Board Members and partners, now host more than 10 meetings around the globe with topics ranging from Market Research in the Mobile World, Qualitative Research for Policy Making to Online Qualitative Research.

What do you see as you core mission?
Our core mission is to provide an effective and conducive platform for practitioners and academics to discuss, debate and disseminate cutting-edge research methods and techniques. This is achieved primarily through the organisation of highly focused and topic-specific research workshops and conferences around the world.

How can people get involved in the things you are doing?
As a small organisation, we depend a lot on our Advisory Board Members and our passionate volunteers to help review abstracts and synopses for each conference. This is to ensure that all selected presentations will be “grounded” and relevant for the audience. We also need volunteers to help out with on-site logistics and arrangements. All the help we get ensures that the conference fees are kept as low as possible and therefore allowing individuals and companies with limited budget to attend.   

 

More information about the Merlien Institute is available from their website.

 

What do we mean by 'proving' something? TARSK 17

This TARSK (Things All Researchers Should Know) looks at the use of the word proof in market research.

The first point, and one widely agreed since the writings of Karl Popper, is that empirical observations can never prove that a universal rule is true, all observations can do is provide evidence. As the evidence for something grows then we start to assume that ‘for all practical purposes’ it is true. The scientific method, post Popper,  is to work on the principle of falsifiability, i.e. a theory should be expressed in a way that allows it to be proved to be wrong.

Quite often the everyday language we use can clouds issue related to truth and proof. For example, in 1954 Roger Bannister became the first man to be officially recorded as running a mile in under four minutes. This was widely reported as proving that man could indeed run a mile in under four minutes, something which had been previously doubted. However, it is probably clearer to see Bannister’s run as disproving the theory “All men take four or more minutes to run one mile”.

To put this in a market research context, let’s look at two possible proof related questions.

“Prove that some of XXXX’s customers are satisfied” To do this all we have to do is find two (one if we say one is some) people who genuine customers and who are satisfied. This is a task that research can do – although we might want to caveat it to say they we have proved some XXXX customers say they are satisfied. In a media case Ryanair recently tried to claim the flat of a journalist who bet her flat that no Ryanair customer had every said they were satisfied.

“Prove that most XXXX customers are dissatisfied”. This would only be possible if we could ask a census of XXXX customers, or failing that a number in excess of 50% if the dissatisfaction turned out to be very high. This census would have to take place at a single moment in time, so is not possible for most non-trivial cases. To test this sort of proposition we typically conduct research on a sample. If we interview a sample of say 1000 customers, out of a total of 2 million customers, and we find that ALL 1000 are dissatisfied, we have not proved that a majority of the 2 million are, we have a finding that makes it highly probable that a majority of the 2 million are dissatisfied.

Quite often, in market research, we are interested in comparing two methods, for example two methods of delivering a product, two types of advertising, or even two methods of market research. We can explore the concept of proof further by looking at this issue of two research methods.

If we have two methods for research, let’s call them A and B) and we test them to see if they give the same result we might find A=B (A is equal to B) or A≠B (A is not equal to B). These two results give two different inferences:

A=B implies that sometimes A gives the same result as B, but it does not imply that we know that A sometimes gives a different result to B. It could be that A always gives the same result as B, or it could be that A only sometimes does).

A≠B implies that sometimes A gives a different result to B, but it does not imply that we know that A sometimes gives the same result as B. It could be that A always gives a different result to B, but it could be that A sometimes gives the same result as B.

To put this into the context of a yesterday’s post, where I pointed out that Gongos Research had NOT proved the validity of smartphone research, we could see A as a traditional method and B as smartphone research. If the results from a test (or some tests) show that A is sufficiently similar to B for us to say A=B (in a practical or commercial sense) then we have shown that sometimes A gives the same results as B. The more tests that are conducted, the more confident we will be that A=B, if the results keep going the right way. However, no amount of testing will every prove that A=B, it will simply make it more likely to be true. However, one result where A≠B will mean that we are sure that A does not always give the same result as B.

So, if we do an ad test with 100 people and all 100 score it in the top box, we have not proved that people like it, we have proved that the 100 people in the test ticked the top box, and we think it is very, very likely that a large proportion of the wider proportion will like it.

If you are interested in tests that compare methods you might also be interested in TARSK 16, which looks at why showing that two results are not significantly different does not necessarily imply they are similar.

Gongos Research have NOT proved the validity of smartphone research

OK, somebody had to blink! For the last few days there have been several tweets and comments using the phrase “Smartphone surveys prove their validity in marketing research”. When I saw the first post I immediately classed it as harmless hyperbole. But when I saw that MSNBC had used the same headline (14 June 2011) I felt I had to shout out “The Emperor has no clothes”.

The stories seem to track back to a press release by Gongos Research, released on 14 June 2011. The press release includes the phrase “a new study proves that smartphone-based survey data is statistically comparable to online survey data.”

This claim in linguistically and methodologically ludicrous, a single study cannot prove that something happens, it can show that it sometimes happens, but that does not prove a positive. A single, well designed, study can show that something is NOT true. For example, the claim that all swans are white is disproved by finding one black swan, but finding one more white swan does not prove all swans are white.

If we assume, for the moment, that the press release means that Gongos Research conducted a study, looking at a wide range of questions types, in one or more markets, with one or more types of customers, and that the study showed the results were acceptably similar, then we would not have proof of the validity of smartphone research. We would have evidence that smartphone surveys can sometimes work. If more tests are conducted (and ideally other agencies should be willing to pay for them) we can start to find out if smartphone surveys are ‘often’ (or better still ‘usually’) acceptably similar to other modalities. If we had the results of say 20 studies and they all showed that smartphone research was acceptably similar to other modalities we would not have proved it, but we would feel reasonably confident about using it, even 10 out of 10 positive results would make us pretty happy about trying it for a live study.

I do hope that Gongos Research publish their research as it is potentially really helpful to the industry, as evidence about the viability of smartphone research.

 

New Approaches to Presenting Data

Tomorrow is the NewMR ‘New Approaches to Presenting Data’ event and I am busily engaged in the final stages of pulling the various threads together (along with the NewMR team).

The programme for the Thursday is:

Asia/Australia Session - Sydney 3 pm to 5.15 pm (Auckland 5pm, Singapore 1pm, Mumbai 10.30 am)

• Sue York, Trends in Presenting
• Mike Sherman, Sevendots, Less is More: Getting Value (Not Just Reams of Data) From Your Research
• Peter Harris, Vision Critical, Data Visualisation, enabling the one slide story
• Ray Poynter, The Future Place, The strengths and drawbacks of Prezi

 Europe/Africa Session - 11 am to 1.15 pm (Mumbai 3.30 pm, Paris 12 Noon, New York 6 am)

• Ray Poynter, The Future Place, Understanding the role of data in a presentation
• Tom De Ruyck, InSites Consulting, Infotainment - Going beyond the 30 minute debrief!
• Ken Brewster, E-Tabs Ltd, Dashboards - the new PowerPoint?
• Chris Morgan, Data Liberation, Interactive reporting and dashboards - knowledge from the trenches

 Americas Session - 2 pm to 4.15 pm (London 7 pm, Seattle 11 am)

• Ray Poynter, The Future Place, Combating ChartJunk
• Justine Carleton Gage, Lextant, Insight Translation - Turning Data into Action
• Nima Srinivasan, Added Value, Connectonomics - Online Connections of Women
• Kathryn Korostoff, Research Rockstar, Using Social Media to Boost Market Research Learning: The Twitterversity Experiment

I think all the presenters are to be congratulated on putting themselves forward to present about presenting. When we did a poll on what topics people wanted to see as future Webubar events, the topic of New Approaches to Presenting Data won by a large margin. But we had fewer people willing to speak on this issue than on any other. I suspect that one of the reasons that people were so keen for NewMR to hold this session was that many people know they need to change but they are unsure how to achieve it.

If you have not registered yet it is still not too late and you can register from the NewMR website.