As sample sizes become smaller it becomes harder to say that a difference is significant. This does not mean that a difference is less likely to BE real, just that we are less likely to be able to SAY a difference is significant.
When we test a difference and we find it is significant this means that we are 95% sure that the difference exists. When we test a difference and we say it is NOT significant, this does not mean that there is no difference, it just means that we are not 95% sure there is a difference.
For example, consider two groups of people, group A and group B. If 53% of group A liked pop music, but only 47% of group B liked pop music there would be a real difference.
If we took a sample of 1000 people from each group and we found a difference of 53% to 47% we would say there was statistically significant difference, we would say were 95% sure there was a difference, and on this occasion we would be right, there is a difference of 6%.
If we took a sample of 100 people and we found a difference of 54% to 47% we would say the difference was not statistically different. This means we are not 95% sure there was a difference. However, this does not mean there is not a difference, it just means we can't be reasonably sure there is a difference. However, we cannot say there is NO difference between A and B, that is a different test. Indeed, on this occasion we know that A and B are different, but the sample size is not big enough to confirm it.
When we use a smaller sample size, more real differences are going to be classed as not significant, usually because those real differences are not large enough to be significant when a smaller sample is used. Indeed, this is one method that people have used in the past to ‘fix’ results. For example, using a pre-post test with a relatively small sample size and then reporting the finding that nothing was significantly different, can imply to the casual listener/reader, incorrectly, that this meant they were therefore similar.
So, the key thing to remember is that if there is not a statistically significant difference there still might be a real difference. Indeed if survey shows that A is bigger than B, there is usually at least a 50% chance that A really is bigger than B.
If you want to show that there is not a difference between two items then you need to test how confident that you are that the difference between the two is less than some arbitrary figure, such as the probability that the real difference is less than 3%.
TARSK is a series of posts grouped by the concept Things Every Researcher Should Know. I will be expanding on the topic of what statistical significance means in a forthcoming webinar, date TBC.