Should the social survey be an endangered species?

On Wednesday I am scheduled to emulate Daniel going into the lion’s den when I debate the topic “Is the social survey an endangered species” with Penny Young the CEO NatCen (Britain’s leading centre for independent social research). Or, perhaps since I will be the bad guy, perhaps I am a lion entering a den of Daniels?

I am not going to propose that all social surveys should end, nor will I suggest that ALL of the coalition Government’s cuts in social surveys are justifiable. But, I do believe that in the future there ‘should’ be fewer surveys. I will keep most of my powder dry until Wednesday, but I will share some of my concerns in the context of a recent study by NatCen. The study was “Estimating the prevalence of young people absent from school due to bullying”

Now, let me start by acknowledging that bullying is a major problem, at its worst it leads to suicides, self-harming, and psychological damage, and there is evidence (from the TellUs 2009 survey – a study which the current Government has scrapped BTW) that about half of children experience some form of bullying, and about one-in eight have experienced bullying in school in the last six months. I have spent 28 years as a councillor and had a large number of people contact me about bullying, as a parent I have had a good level of exposure to what happens in schools, and in writing this article I have refreshed my memory by speaking to somebody who acted as a peer group councillor to young people who had been bullied.

So, bullying is a problem. We need to understand it if we are to diminish it. But, is a survey a sensible way of understanding bullying? And, even if it is, was this the right sort of survey?

If you click on the link for the survey you should be impressed by the level of documentation, it is exemplary. It contains the objectives, the research sponsor, the questionnaire, the methodology (which included a good piloting stage), and summaries of the data as well as the findings.

However, as a study I think there are several problems with it:

1. The first problem is epistemological. The study seeks to estimate bullying in terms of numbers, and within that it seeks to translate it to the number of children losing out on schooling. It is a survey of adults about missed schooling by children. There are no definitions of bullying in the questionnaire (for good methodological reasons) which means the interpretations will be vague.
2. The second issue is that the study appears to make a value judgement that being electively home schooled is a bad thing. It adds together the number of children who have missed school through bullying to those who are being home schooled where the reason (or a reason) quoted is bullying. I am not a big fan of home schooling, but society and the law have both defined it as a right, whereas being absent from school is clearly a problem. Bullying may be interesting in both cases, but the outcomes should not, IMO, be simply added together.
3. The next problem relates to what Mark Earls talks about in his book Herd. We are all unreliable witnesses to our own motivations. Asking people what was the reason they did something is frankly flaky, asking that which of those reasons was the main one is flaky, and asking them to do it on behalf of one of their children, thinking back for a considerable amount of time is really flaky.
4. The study is statistically unreliable, despite its claim to offer error limits of which it is 95% confident. The survey had a response rate of 17%. The study quotes 95% confidence of the population, but does not make clear that the population is a hypothetical population comprising that part of the total population willing to take part in this sort of study. None response bias could mean that the 82% who did not reply had very different experiences. The study did do better than most commercial studies, they looked at the basic demographics and they did not indicate any great differences between the responders and the population, but the issue is not demographics, it is the experience of bullying.
5. The responses in the study indicate a large gap between the parents’ perceptions and the sampling frame. The survey was sent to a sample of parents whose children had been absent for 28 half days (which might mean 14 days, but it could also be a mixture of full days and half days). However, when the respondents answered how many days their child had been absent from school nearly half said less than 14 days (interpolating the 11 to 15 day break). This either means the frame was wrong, or that parents memories/knowledge is very hazy, or some combination of the two. If the scoring of a simple variable like number of days absent was wrong in 50% of cases, then surely the description of motivation is likely to be dodgy?
6. The final issue is that the study was sponsored by the Red Balloon Learner Centres and a link to their website was included in the letter sent to parents. Red Balloon is a great organisation; it works to achieve the recovery of bullied children. One of the statements on its website should remind all of us how bad bullying can be “We provide an 'intensive care' full-time education for children aged between 9 and 18 who are unable to go to school because they have been severely bullied. At least half of the students we take have attempted or seriously considered suicide.” However, having it as the declared sponsor of the project could be a source of bias.

The net result of the survey is that Red Balloon can issue a press release stating that over 16,000 children are absent from school due to severe bullying and it has a credible source of information to go to funding agencies, administrators, and politicians with.

But, have we really leaned very much about bullying from this study? Some children are bullied and never miss school, some miss school for a range of reasons, one of which is bullying, but the other reasons may also be priorities.

The implication in the headline from Red Balloon is that this bullying happens at school and happens from other pupils. The TellUs study suggests that a large proportion of bullying happens outside of school, and my caseload indicates that a small but noticeable amount of perceived bullying is from the teacher to the pupil. Red Balloon already know about the damage that bullying can do and that it can lead to children being absent from schools. Does an estimate (even if it were accurate) really tell us as much as case studies? Does it really matter if 10,000, 16,000, 30,000, 78,000 or even 5,000 are absent? Doesn’t it need dealing with irrespective of the fine detail of the numebr?

The press release from Red Balloon describes the bullying as severe, but this was not explored by the study. At one level, one could say that any bullying that led to a child missing school is severe, but that is simply looking at the outcomes. In my time as a councillor I have encountered two cases of serious sexual assault, several of physical harm, and one of a child being imprisoned in a shed whilst youths threw lit matches into the shed – by any stretch of the imagination these were severe. However, I have also met parents whose child felt bullied because another child had stopped being their friend or had not invited them to a sleep over. This sort of behaviour can seem like bullying to the recipient, it can cause actual damage, but it may not be perceived that way by the other parties in the event.

My general thesis is that I think we resort too easily and too readily to surveys.

There used to be a mantra that if you can’t measure it you can’t manage it, but more recently we have begun to understand that if you can measure it then it probably isn’t important (i.e. it is the unmeasurable things that really matter - check out Elizabeth Barrett Browning when she counts the ways she loves).

A study which produces a clear but only vaguely defensible number is not, in my opinion, likely to be as helpful as a range of other solutions. I am sure that Red Balloon will have done some of the following, but I will list them all the same:

1. Case studies of bullied children
2. Discussion with schools about bullying, getting them to assess the form and the prevalence and its consequences
3. Discussions with peer counsellors, asking them to assess the form and prevalence of bullying and its consequences
4. Discussions with children who appear not to be bullied, asking them to assess the form and prevalence of bullying and its consequences – I suspect that in many cases they will have direct experience of bullying
5. Looking at the discourse and semiotics of bullying to understand it better, to understand how it is described and experienced.
6. Social media monitoring research, looking for specific phrases, terms, and issues.

Instead of a survey, an estimate could have been formed from talking to schools and pupils, probably less accurate than the survey commissioned, but it would have indicated that thousands of children miss school through bullying, it could have contained case study material to bring the issue alive, and perhas a bit of social media monitoring to show levels of concern. For example entering Bullying into something as simple as Google Insights suggest that more peope are searching for information on Bullying in the Autumn, that the lead term is "anti bullying", that the regions with with highest searches are Northern Ireland and Wales, and that the top rising search term is cyber bullying.

If you would like to join Penny Young and me for the debate about the future of the social survey please check out the NatCen website.

Ps and BTW, my gut feel is that 16,000 seems too small a number. There are 2.8 million children aged 11 to 15 in secondary education. This would imply about one child in 175 was missing schooling because of bullying. In the main school in my area this would equate to  4 or 5 children (if my school was typical). But, thinking about my caseload, talking to a fellow councillor, and referring the question to my peer group counsellor contact, our feeling was that the numbers would be more like 20 - 40 (and possibly quite a bit more).  Interestingly, if the study had based its main estimate on the impact of bullying on whether bullying was mentioned as A reason, as opposed to being the main reason, the estimate nationally would have been 78,000, which would imply 23 from my local school, which triangulates better with our local experience.

 

Comments

[JHenning]

Hi Ray -- just now catching up with this. Two comments. First, while you never use the term what you've shown us is that we have something called the Total Survey Error model that helps us look at a survey and systematically consider the possibilities for error at a various points in the survey's execution. Second, in this particular instance and as is almost always the case with government and other high quality surveys there is pretty much full disclosure of everything that was done so that you can evaluate the accuracy or the credibility of result. The problem we face in MR and in the newer methods you cite as alternatives is that this kind of systematic evaluation and full disclosure is very rare. There is a lot we can learn from the people who do social surveys about how to do good research.