The Future Place Blog

In Praise of Maritz Stats

Many years ago I briefly, and happily, worked for Maritz (when Maritz bought The Research Business). One of the things I discovered at that time was a great free utility they provided called Maritz Stats, a simple to use significance testing program, which covers the main techniques such as t-test, z-test, paired tests, and Chi-square.

I needed to recommend a solution to somebody today and was delighted to discover that Maritz still make this program freely available.

If you want to download it you can find it here.

Bad Science and Coffee

One of my favourite blogs is the Bad Science blog written by journalist Ben Goldacre. One of his latest posts is about a claim, repeated in many newspapers, that drinking seven cups of coffee of day can make you see dead people.  Goldacre dissects the story, showing how the press have misused the information, but also exposes the way that researchers appear to have taken spurious correlations and made a case from them.

Goldacre’s example of machine gunning a wall and looking for cases where three bullet holes end up close together is very reminiscent of some market research, and very similar to the research published a couple of years ago which a superficially appeared to link certain diseases to certain star signs.

When market researchers cross-tab everything by everything and set of a fishing expedition looking for ‘significant’ findings, they are doing exactly the same thing. It is only significant if you stated it was a possible source of difference before running the stats.

Resource for data

NationMaster.com is a resource for all sorts of data from around the globe. NationMaster pulls data from a wide range of sources such as CIA World FactBook (itself a collection of other data sources). I find the interface a bit annoying, and too many options default to USA, but it is certainly worth adding to your list of resources.

via Karen Blakeman's blog.

More debate on traffic camera statistics

On 14 November, I posted about a challenge to Government statistics. In the post I said I was sceptical about some of the report, but I was very interested in the way it attacked the Government's use of statistics. The Government's position is that speed cameras save lives, the journalists in the Telegraph were supporting a claim that speed cameras made the roads more dangerous.

Now, in a wonderful 2.0 Internet sort of way, the Telegraph's journalists have been blasted for selective use of statistics and misleading interpretations in an article by George Monbiot in the Guardian. The key points that Monbiot makes are:
1) The Telegraph quoted errors in the 2003 report from the Government, but did not mention that the 2005 report identified and corrected the errors (in 2003 the Government claimed speed cameras reduced deaths by 35%, in 2005 the figure was corrected to 19%).
2) That the quotes about a Parliamentary report were both partial and incorrect.
3) That opinion polls support speed cameras (something which agrees with the local polls I have seen), the Telegraph claimed that polls opposed the use of speed cameras..

Whilst the debate is enjoyable, it is very hard for the lay person to know who is telling the truth. The ideal solution would be to improve the status of the statistical functions of the Government so that almost everybody believed them, as opposed to the current situation, where very few believe them.

One thing that has a direct impact on the polls is the way they are conducted. In 2003 YouGov conducted a poll for the Telegraph, which showed that 56% of motorists felt there were too many cameras. However, had the poll been conducted with pedestrians and cyclists, the results could have been different. The views of motorists are certainly worth hearing, but in terms of speed cameras you ALSO need to listen to to other intrerested parties.

Safe Speed is an organisation campaigning for the removal of all speed cameras. They have a poll on their website, where they report that 63% of people want all cameras removing. But the only people who vote are people who visit their website, and in total (as at 19 November) just 87 people have chosen to vote.

In 2005 a Mori poll for the Greater London Authority produced 58% support for speed cameras, but the question specified "in 20 mile per hour zones", which again is likely to impact the result.

A more recent Telegraph/YouGov poll puts support for speed cameras at 50% (with 39% being opposed).

Bad Stats Can Hurt

Nigel Legg has posted an interesting link on FaceBook to a scary article on The Telegraph's website. The story looks at the way the British Government has misused traffic data to claim that speed accounts for one-third of traffic accidents (the article claims it is actually somewhere between 5% and 10%) and that speed cameras reduce fatalities by 35% (a claim that is hard to support).

For thirty years the UK had impressive reductions in traffic fatalities, making it the safest country in Europe. But for the last decade the numbers have stabilised, even increasing a little. The article concludes that alcohol and drugs account for many if not most of the deaths, and they only a very small amount (2% to 3%) of accidents involved breaking the speed limit (70% of accidents caused or contributed to by inappropriate speed were at speeds below the speed limit).

I do not accept everything that is in the article. For example as a Councillor I have seen the statistics that show that speed cameras reduce the median and mean speed on residential roads, something which may not effect fatalities much, but does improve the quality of life. The article quotes opinion polls which say two-thirds of people don't like speed cameras. I will check this point further, but when Councils poll local residents who live on main roads and 'rat runs' a clear majority want cameras on their road to have speed cameras. Although they sometimes request cameras elsewhere to be removed!

But the main thrust of the article is clear and stark, the Government has misused the statistical data, and when challenged has dug its heals in and attacked those telling the truth. In the days before Web 2.0 this would often have worked. But, now, it will usually fail.

Structural Equation Modelling and Causality

I have just finished teaching a two-day multivariate modelling course in Brisbane for AMSRS (the Australian market research society), and one topic which led to an interesting discussion was SEM (Structual Equation Modelling) and causality. As Jon Pinnell, of MarketVision, says, SEM is “A source of excitement to some and great frustration to others.”.

At one level SEM is very appealing in that it appears to offer a grand theory of everything. The user creates a model, linking inputs to latent variables, to outputs, and then calculates all the relationships and errors in one go.

However, it seems there are two, large, problems with SEM.

1. To use SEM properly you should already have the model, before you have the data. SEM can then be used to see whether this set of data (e.g. this company, this situation, this process) confirms to the model. If the model correctly ascribes causality, and the data fit the model, then we can accept the causality in the SEM process. However, if the model has been developed from the data available, SEM should not really be run until a new set of data has been acquired.
   
2. The second problem, and it is one that applies to many other techniques in addition to SEM, is the dependence on assumptions about linear relationships between inputs and outputs. In the real world, we know many things are not linear; for example, a service can get a bit worse and nothing happens, then a bit worse and nothing happens, then a tiny bit worse and the contract is cancelled.

In reviewing the current state of thinking I came across the quote below about SEM, which I think sums up the causality issue quite well.
“The term causal model must be understood to mean: ‘a model that conveys causal assumptions,’ not necessarily a model that produces validated causal conclusions.”

What the stats tells us about prostate cancer, and what they don't

Like many men past 40 I have started to pay more attention to articles in the press about prostate cancer. But the statistics seem more opaque than almost any other data presented to the public. At one level we have the stats that prostate cancer is the most common or second most common form of cancer amongst men in the UK, killing about 10,000 men a year. As a result there have been campaigns to raise awareness of the symptoms, including one using old recording of Bob Monkhouse, a UK comedian who died of prostate cancer.

On the other hand there is a growing backlash against the PSA test (a blood test which is supposed to indicate an increased likelihood of prostate cancer), and against treatment. The complexity appears to be that prostate cancer develops very slowly in most cases.

The following statistics were issued as part of a BBC radio broadcast on 6th September 2007 (the Check UP programme), and I can make no claim for their veracity.

• Only 5% - 10% of men who have prostate cancer will die of it.
• About 30% of men who have a positive PSA test will actually have prostate cancer
• About 25% of men, in the same age group, who have not had a PSA test will have prostate cancer

The figures above suggest that, in the absence of other symptoms or causes, the PSA test is not very useful. A digital rectal examination (in this case the digit means a finger, not some form of technology) is more indicative, but men are reluctant to ask, and many doctors seem reluctant to bring the topic up. A prostate biopsy is highly accurate, but painful, expensive, and can cause issues such as bleeding.

One of the main concerns, of some specialists, like the specialist on the BBC, is that to know that you have a cancer is worrying. But for 90-95% who have the cancer it will not kill them, and in most casement no treatment is the best treatment. Is it better for most people who have the non-threatening form of prostate cancer not to know?

A similar argument applies to some of the genetic tests that private companies are beginning to offer. Do people benefit from knowing they have an increased risk of disease A or B, or are they better getting on with their lives?

A complete transcript of the programme will be available on the BBC website in a few days.

What faith can we have in 16 drivers?

From time to time, I see studies reported in leading magazines that do not deserve the coverage they get. The New Scientist recently asserted that SatNavs make car journeys shorter and safer.

However, the study was based on just 16 drivers with SatNav and 16 drivers with maps, in Taiwan. The report states that on average in-town journeys were 7% shorter, and that rural journeys were 2% shorter. What is the likelihood that this is significant, even as a study aiming to predict the population of Taiwanese drivers?

The assertion about safety is even more shaky. The researchers found that SatNav drivers made fewer changes of course. The researcher said that changes in course are a sign of frustration, which can lead to dangerous driving, so SatNavs may be safer. I would hope that no market researcher would build such a house of sand, on such poor foundations.

Is a sample of 12 enough for the BBC?

I was amazed to read an article on the BBC website which make strong scientific claims on the basis of a study of 12 people, split into 2 cells of 6.

The study claimed scientists were saying that exercising after eating would help a diet. The claim was based on research (if that is the correct term) conducted Imperial College London. 12 volunteers were fed the same breakfast. Six of them then rested, whilst six rested for an hour and then worked out on an exercise bike for thirty minutes, both groups were left for a further 30 minutes and allowed to each as much as they liked.

The report then details the kcals of the food consumed, and the estimated kcals burned by those exercising. The result was that the exercise group consumed a net 421 kcals compared the resting group who consumed 565 kcals.

Nowhere in the article does the BBC cast any doubt on the methodology of the test, for example how well balanced were the two groups, how well did the two groups reflect society, whether the people were young or old, nor whether they were matched in terms of normal exercise regime. The article did not question whether a sample of 6 per cell is enough to be confident of the results, nor did they appear to ask if other similar tests had been conducted which did not find this result. Even if the sampling statistics do imply that the difference between these two groups of six people are unlike to be due to chance, they do not indicate whether the difference was innate to the groups (for example one group was more hungry) or due to the effect being tested.

The BBC obtained two quotes. One was from Dr Ian Campbell of the charity Weight Concern who talked about this study “…shows shows is that, although total calorific intake is greater, the net result, because of the exercise taken, is a reduction in the net energy balance.”. Dr John McAvoy, a GP with a special interest in obesity, said the study was a "significant contribution to understanding the complex mechanisms of energy balance". I wonder if these quotes indicate that none of us tends to challenge research that agrees with our prejudices, or perhaps they were not given the full picture.

I consider this sort of reporting by the BBC to be part of the reason why Mumbo Jumbo is conquering the world. If our media do not challenge the credentials, then who will?

We market researchers quite often beat ourselves up, but I think generally we perform better than this. But the more I see of academic work, the more I see poor samples, weak surveys, and inappropriate analysis.

Useful software for Correspondence Analysis

Today I had to churn out some Correspondence Maps from some aggregate tabs in Excel. Most of regular packages are not that friendly in terms of correspondence maps, particularly ones destined for PowerPoint, so I tried something different. I used MVSP from Kovach Computing Services. The program was easy to use, produced nice output and was not fussy in it requirements, specifications, or limitations. At £130 it represents great value for money.

The program has other functions, such as principal components, but even if I ever use it for anything other than correspondence maps it will be good value. If you want to try it out, they have a demo with 30 days free use.