OK, somebody had to blink! For the last few days there have been several tweets and comments using the phrase “Smartphone surveys prove their validity in marketing research”. When I saw the first post I immediately classed it as harmless hyperbole. But when I saw that MSNBC had used the same headline (14 June 2011) I felt I had to shout out “The Emperor has no clothes”.
The stories seem to track back to a press release by Gongos Research, released on 14 June 2011. The press release includes the phrase “a new study proves that smartphone-based survey data is statistically comparable to online survey data.”
This claim in linguistically and methodologically ludicrous, a single study cannot prove that something happens, it can show that it sometimes happens, but that does not prove a positive. A single, well designed, study can show that something is NOT true. For example, the claim that all swans are white is disproved by finding one black swan, but finding one more white swan does not prove all swans are white.
If we assume, for the moment, that the press release means that Gongos Research conducted a study, looking at a wide range of questions types, in one or more markets, with one or more types of customers, and that the study showed the results were acceptably similar, then we would not have proof of the validity of smartphone research. We would have evidence that smartphone surveys can sometimes work. If more tests are conducted (and ideally other agencies should be willing to pay for them) we can start to find out if smartphone surveys are ‘often’ (or better still ‘usually’) acceptably similar to other modalities. If we had the results of say 20 studies and they all showed that smartphone research was acceptably similar to other modalities we would not have proved it, but we would feel reasonably confident about using it, even 10 out of 10 positive results would make us pretty happy about trying it for a live study.
I do hope that Gongos Research publish their research as it is potentially really helpful to the industry, as evidence about the viability of smartphone research.
I appreciate your comments, Ray. Indeed, our intent was to shed light on the currently existing body of evidence in our industry – not claim a “proof” in the Philosophy of Science or mathematical sense.
Much of the literature surrounding the use of mobile or smartphone devices for survey research is mixed. This study indicated that data can be collected and compiled in multi-mode formats with confidence, which enabled us to support some of the accepted approaches and question others.
But let’s not lose sight of the real issue. While we have solid evidence to support the use of multi-modal platforms for research the question of migration from online to smartphone as a preferred platform is still to be determined. Many of us in the marketing research discipline agree that online surveying has its issues (biases associated with representation, scale uses, artificial interviewing environment, etc.). These issues have become increasingly apparent as the online platform has been less able to solve “puzzles” in the Kuhnian sense of the term. The less puzzles solved, the more we are inclined to view the smartphone platform as a potential replacement for online surveying (assuming that the smartphone platform will be able to solve more of the puzzles identified by the online methodology and other potential platforms, such as social media, etc. are not direct competitors for the next platform paradigm).
The importance of this study for the marketing research industry is that we were able to rigorously test a number of assumptions and items between the two separate platforms. But this is not the end. Similar to your thoughts, we welcome other research agencies to join in rigorously testing data and research elements between the two platforms. Based on our findings, we believe that these additional tests will support the use of smartphones as a valid survey platform.
The critical nature of the future of marketing research surveying is at hand. With technology progressing forward and response rates continuously falling, we need to be able to gather valid and reliable information from our respondents. We stand behind smartphones being a valid avenue to accomplish this goal.
We plan on rolling out the key findings from this study at the MRMW11 conference. Ray, we would love to have you join us and contribute to the debate and good discussion.
Posted by: Michael Francesco Alioto | June 23, 2011 at 07:24 PM
And that's why in academia, it takes ten or twenty replications by different researchers in different countries to prove the same concept before the researchers will even consider that an idea has merit. But the media likes a cool story and cool often wins out over facts.
Posted by: Annie | June 23, 2011 at 01:39 PM
Interesting take Ray and good points. I don't know the details on this study, but Dr. Alioto will be sharing the details at the MRMW11 conference including a white paper and will co-presenting with the client at the TMRE. I know those folks well and I am confident that they will address these and other questions in a way that will make us all feel good about the validity of the study.
Posted by: Lmurphy | June 22, 2011 at 11:38 PM
It's easy to get caught up in the hype when every attempt to dabble in non-traditional market research is met with subjective objection.
Posted by: Mllemire | June 22, 2011 at 09:31 PM