Nobody pays me to write any of the copy on my blog, and should I ever have the good fortune that they do, I will declare it. My main employment is as the owner and principal of The Future Place consultancy. The Future Place provides two key services 1) training and services to industry and academic bodies and 2) consultancy services to companies. The details of the companies I work with are a private matter, but if I blog about any company who has paid The Future Place more than expenses recently (approx. two years) I will mention that they are a client. I hold equity in Virtual Surveys and provide consulting services to them from time to time.
I am paid to run courses for a number of trade bodies and over the last few years clients have included ESOMAR, AMSRS, MRS, and MRIA.
There is a growing acceptance, amongst market researchers, that we should consider online access panels as convenience samples. We cannot assume that a panel approximates to a random probability sample of any naturally occurring population. Although a panel might reflect population demographics there is no reason to believe that the panel members reflect a wider population’s views, beliefs, or intentions.
The traditional purpose of significance testing has been to estimate the probability that a sample reflects values from the ‘true’ population. This purpose was backed by the ‘scientific’ notion that the sampling errors would be normally distributed and that the significance testing process would be an indicator of the validity of the results.
Since the access panel does not approximate to a random probability sample of the wider population we cannot assume that significance testing reflects validity. So, should we continue to use significance testing with online access panel data?
My answer is yes, but that we should change our mind-set away from validity and towards reliability.
To make this change from validity to reliability, consider what happens when we draw a sample from a panel and when we conduct significance testing. The sample does not reflect the wider population, but it is still drawn from a population, i.e. the population is the set of people on the panel who meet the selection criteria.
When we conduct significance testing on a sample from an online panel tests, the tests show the probability that the results are typical of the panel. If the results are not significant then there is an unacceptable risk that running the same test again (on the same panel) will produce different results. If there is an unacceptable risk that conducting the same study twice (at the same time, on the same panel) will generate different results, then the results cannot be considered reliable (which for many people means they should not be reported).
So, I say yes, conduct significance testing on studies conducted with online access panels, but change the way you describe the results. Do not say things like “These results are accurate to +/-3% with a probability of 95%”. Perhaps use a form of words such as “The test/retest reliability of these results, from this source, are +/-3% at the 95% confidence level.”
I will be writing more on the science and ritual of significance testing and what determines reliability in later posts, along with the suggestion that we should consider use the 80% level of confidence when reporting reliability.