This week’s New Scientist has a really useful, informative, and important article about the danger of creating and reporting misleading statistics, focusing on health statistics, but the same points are true of any use of statistics, for example market research.
To read the full article you will need a copy of the magazine or you’ll need to register with them online, but I strongly recommend reading it.
One of the panels in the article talks about the specific dangers of invalid comparisons and uses as its core the implication made by Rudy Giuliani in 2008 that the UK’s National Health Service was twice as bad as the US in tackling prostate cancer, based on the fact that the US survival rate was 82% compared with the UK’s 44%.
Below is a short précis of the points made by the authors (Marianne Freiberger and Rachel Thomas), with a view to presenting the main points, but I recommend reading the original.
The figures Giuliani quoted were correct, but they give an inaccurate impression.
The rates quoted by Giuliani quoted were five year survival rates. Survival rates are based on the length of time from diagnosis to death. Diagnosis is the first source of problems, in the US most diagnosis is the result of screening, in the UK most cases are diagnosed on the basis of men who have symptoms. This means the cases in the US include more false positives and many cases that would never have progressed to symptom (i.e. are unlikely to have caused death). Freiberger & Thomas quote a study that estimates about 48% of men diagnosed by screening do not have the progressive form of the cancer.
Freiberger & Thomas assert that to compare countries one needs to compare mortality rates, i.e. the proportion of the whole population who die of a specific cause. For the US the mortality rate for prostate cancer is (according to the authors) 24.7 per 100,000, in the UK the rate was 23.9 per 100,000 – effectively the same number.
Another illustration of the problem with comparing results was given recently by criticisms (for example in the Guardian) of UK Prime Minister’s use of health statistics to use the age adjusted mortality rates between France (19 per 100,000) and the UK (41 per 100,000). This time the concerns include the fact that the UK is more inclined to define deaths amongst older citizens as being caused by a heart attack, in France the category old age is more common than in the UK.
Whether we are talking about health statistics or marketing data we need to be clear what the base for a comparison is in order to be able to safely use the information. The New Scientist article also looks at the difference and confusion between correlation and causation.