The Future Place Blog

Some thoughts on The Handbook of Online and Social Media Research

Should we use sig testing with online panel studies?

There is a growing acceptance, amongst market researchers, that we should consider online access panels as convenience samples. We cannot assume that a panel approximates to a random probability sample of any naturally occurring population. Although a panel might reflect population demographics there is no reason to believe that the panel members reflect a wider population’s views, beliefs, or intentions.

The traditional purpose of significance testing has been to estimate the probability that a sample reflects values from the ‘true’ population. This purpose was backed by the ‘scientific’ notion that the sampling errors would be normally distributed and that the significance testing process would be an indicator of the validity of the results.

Since the access panel does not approximate to a random probability sample of the wider population we cannot assume that significance testing reflects validity. So, should we continue to use significance testing with online access panel data?

My answer is yes, but that we should change our mind-set away from validity and towards reliability.

To make this change from validity to reliability, consider what happens when we draw a sample from a panel and when we conduct significance testing. The sample does not reflect the wider population, but it is still drawn from a population, i.e. the population is the set of people on the panel who meet the selection criteria.

When we conduct significance testing on a sample from an online panel tests, the tests show the probability that the results are typical of the panel. If the results are not significant then there is an unacceptable risk that running the same test again (on the same panel) will produce different results. If there is an unacceptable risk that conducting the same study twice (at the same time, on the same panel) will generate different results, then the results cannot be considered reliable (which for many people means they should not be reported).

So, I say yes, conduct significance testing on studies conducted with online access panels, but change the way you describe the results. Do not say things like “These results are accurate to +/-3% with a probability of 95%”. Perhaps use a form of words such as “The test/retest reliability of these results, from this source, are +/-3% at the 95% confidence level.”

I will be writing more on the science and ritual of significance testing and what determines reliability in later posts, along with the suggestion that we should consider use the 80% level of confidence when reporting reliability.