The Future Place Blog

Why an 82% survival rate may not be better than a 44% rate?

This week’s New Scientist has a really useful, informative, and important article about the danger of creating and reporting misleading statistics, focusing on health statistics, but the same points are true of any use of statistics, for example market research.

To read the full article you will need a copy of the magazine or you’ll need to register with them online, but I strongly recommend reading it.

One of the panels in the article talks about the specific dangers of invalid comparisons and uses as its core the implication made by Rudy Giuliani in 2008 that the UK’s National Health Service was twice as bad as the US in tackling prostate cancer, based on the fact that the US survival rate was 82% compared with the UK’s 44%.

Below is a short précis of the points made by the authors (Marianne Freiberger and Rachel Thomas), with a view to presenting the main points, but I recommend reading the original.

The figures Giuliani quoted were correct, but they give an inaccurate impression.

The rates quoted by Giuliani quoted were five year survival rates. Survival rates are based on the length of time from diagnosis to death. Diagnosis is the first source of problems, in the US most diagnosis is the result of screening, in the UK most cases are diagnosed on the basis of men who have symptoms. This means the cases in the US include more false positives and many cases that would never have progressed to symptom (i.e. are unlikely to have caused death). Freiberger & Thomas quote a study that estimates about 48% of men diagnosed by screening do not have the progressive form of the cancer.

Freiberger & Thomas assert that to compare countries one needs to compare mortality rates, i.e. the proportion of the whole population who die of a specific cause. For the US the mortality rate for prostate cancer is (according to the authors) 24.7 per 100,000, in the UK the rate was 23.9 per 100,000 – effectively the same number.

Another illustration of the problem with comparing results was given recently by criticisms (for example in the Guardian) of UK Prime Minister’s use of health statistics to use the age adjusted mortality rates between France (19 per 100,000) and the UK (41 per 100,000). This time the concerns include the fact that the UK is more inclined to define deaths amongst older citizens as being caused by a heart attack, in France the category old age is more common than in the UK.

Whether we are talking about health statistics or marketing data we need to be clear what the base for a comparison is in order to be able to safely use the information. The New Scientist article also looks at the difference and confusion between correlation and causation.

 

Statistics for Market Researchers - Introduction

I have started two new projects, a book looking at the how the "Public Sector can use Social Media to Engage and Consult", and a book on "Statistics for Market Researchers". Given the success I had in using a collaborative approach in writing The Handbook of Online and Social Media Research, I intend to adopt a similar approach with both of these books.

Initially, I shall be posting extracts here, as the project expands I shall adopt a more structure approach.

The first posting here set out the scope and vision for the book Statistics for Market Researchers.

---

Introduction

Overview

This book is written specifically for market researchers. There are books on statistics for a wide range of purposes, for example for scientists, for managers, for biologists, for educational psychologists, and many more. This raises the questions why? Surely, you might think, there is only one set of statistics? However, there are very good reasons for  a book to be specific to a discipline and amongst the key benefits perhaps the two most important are:

• The many, many statistical techniques and tools, too many for most people to become familiar with and almost nobody can become skilled at all the possible options, so a specific discipline tends to focus on a sub-set of tools and techniques that match the needs of that discipline and which match the data that is typically available to practitioners in the field.

• In the ‘real’ world most practitioners have to make compromises between theory and practice. These compromises tend to vary by discipline, typically on pragmatic grounds. For example, quite often academic research the compromise is on the selection of research subjects (for example the large number of psychology and behavioural economics studies that use students as the subjects). Different compromises result in different techniques being selected and different rules of thumb being adopted.

In the light to of the views expressed above this book seeks to describe just those statistical and analytical techniques used by market researchers, and where appropriate flagging up where the technique is a mainstream approach (such factor analysis and cluster analysis) or only rarely used in market (such as the ANOVA).

Structure of the descriptions

For each of the techniques and tools covered in this book a template approach has been adopted, based on the following headings:

• What What is the technique seeking to do and (in an over-simplified way) how does it do it?
• Why What sort of problems might are resea4rchers using these technique typically seeking to solve with this technique?
• How Notes on how to set about using this technique?• Strengths and Weaknesses? A review of what the technique is good for, bad for.
• Theory What does the technique require in theory and how should its results be interpreted in theory.
• Practice How the technique is typically used, what compromises are commonly made by market researchers?
• Tips Suggestions about how to make the technique more likely to work, easier to use, and readily explained.

Structure of the Book

The book is divided into several Parts, covering:

1. Data. This part looks at the sort of data that are typically available to market researchers, such as proportions, counts, and scales, along with the descriptive statistics that are used with this sort of data.
2. Describing and Testing. This part looks at the ways that data are described, investigated, and tested by market researchers.
3. Analysis. This part looks at the various techniques used by market researchers to analyse data, for example factor analysis and cluster analysis.
4. Designing Fieldwork.This part of the book reviews the link between the problems a market researcher is seeking solve, the techniques the researcher plans to use, and the implications for the type of data that should be collected.
5. From Problems to Techniques. Most of the book looks at the statistical tools and techniques available to market researchers and shows how they can be applied to research problems. However, this section turns that process around and explores typical research problems and describes the techniques that are suitable to be used in solving them.

Caveat Emptor

Caveat emptor is a Latin phrase that means buyer beware. In the context of this book what it means is that the author is NOT saying that it is OK to break the rules, is NOT saying that mathematical assumptions can safely be disregarded, and is NOT saying that the techniques described in this book will reveal the truth. The book tends to concentrate not on ‘best practice’, but on typical practice. If you want know and understand best practice, it is probably necessary to enrol on a suitable course and learn about the underlying theory and mathematics.

 

 

A free webinar on Cluster Analysis

As a follow up to my previous post (on Cluster Analysis) and as part of the warm-up (and practice) for the Festival of NewMR I am running two free webinars on Friday 12 November on the topic of Cluster Analysis.

The Webinars will last about 30 minutes and there will then be a chance for Q&A after each one. 

The first ‘sitting’ will be at 10am and the second at 4pm (both UK time). Places will be limited to 99 for each one, so if you want to attend please register as soon as possible.

The link for the 10am one is here (Friday 12 November, 10am)

The link for the 4pm one is here (Friday 12 November, 4pm)

Note, both of the links will let you know what time that would be in your location. I am planning to record the webinar and make it available on my site and on the NewMR website.

 

In Praise of Maritz Stats

Many years ago I briefly, and happily, worked for Maritz (when Maritz bought The Research Business). One of the things I discovered at that time was a great free utility they provided called Maritz Stats, a simple to use significance testing program, which covers the main techniques such as t-test, z-test, paired tests, and Chi-square.

I needed to recommend a solution to somebody today and was delighted to discover that Maritz still make this program freely available.

If you want to download it you can find it here.

Bad Science and Coffee

One of my favourite blogs is the Bad Science blog written by journalist Ben Goldacre. One of his latest posts is about a claim, repeated in many newspapers, that drinking seven cups of coffee of day can make you see dead people.  Goldacre dissects the story, showing how the press have misused the information, but also exposes the way that researchers appear to have taken spurious correlations and made a case from them.

Goldacre’s example of machine gunning a wall and looking for cases where three bullet holes end up close together is very reminiscent of some market research, and very similar to the research published a couple of years ago which a superficially appeared to link certain diseases to certain star signs.

When market researchers cross-tab everything by everything and set of a fishing expedition looking for ‘significant’ findings, they are doing exactly the same thing. It is only significant if you stated it was a possible source of difference before running the stats.

Resource for data

NationMaster.com is a resource for all sorts of data from around the globe. NationMaster pulls data from a wide range of sources such as CIA World FactBook (itself a collection of other data sources). I find the interface a bit annoying, and too many options default to USA, but it is certainly worth adding to your list of resources.

via Karen Blakeman's blog.

More debate on traffic camera statistics

On 14 November, I posted about a challenge to Government statistics. In the post I said I was sceptical about some of the report, but I was very interested in the way it attacked the Government's use of statistics. The Government's position is that speed cameras save lives, the journalists in the Telegraph were supporting a claim that speed cameras made the roads more dangerous.

Now, in a wonderful 2.0 Internet sort of way, the Telegraph's journalists have been blasted for selective use of statistics and misleading interpretations in an article by George Monbiot in the Guardian. The key points that Monbiot makes are:
1) The Telegraph quoted errors in the 2003 report from the Government, but did not mention that the 2005 report identified and corrected the errors (in 2003 the Government claimed speed cameras reduced deaths by 35%, in 2005 the figure was corrected to 19%).
2) That the quotes about a Parliamentary report were both partial and incorrect.
3) That opinion polls support speed cameras (something which agrees with the local polls I have seen), the Telegraph claimed that polls opposed the use of speed cameras..

Whilst the debate is enjoyable, it is very hard for the lay person to know who is telling the truth. The ideal solution would be to improve the status of the statistical functions of the Government so that almost everybody believed them, as opposed to the current situation, where very few believe them.

One thing that has a direct impact on the polls is the way they are conducted. In 2003 YouGov conducted a poll for the Telegraph, which showed that 56% of motorists felt there were too many cameras. However, had the poll been conducted with pedestrians and cyclists, the results could have been different. The views of motorists are certainly worth hearing, but in terms of speed cameras you ALSO need to listen to to other intrerested parties.

Safe Speed is an organisation campaigning for the removal of all speed cameras. They have a poll on their website, where they report that 63% of people want all cameras removing. But the only people who vote are people who visit their website, and in total (as at 19 November) just 87 people have chosen to vote.

In 2005 a Mori poll for the Greater London Authority produced 58% support for speed cameras, but the question specified "in 20 mile per hour zones", which again is likely to impact the result.

A more recent Telegraph/YouGov poll puts support for speed cameras at 50% (with 39% being opposed).

Bad Stats Can Hurt

Nigel Legg has posted an interesting link on FaceBook to a scary article on The Telegraph's website. The story looks at the way the British Government has misused traffic data to claim that speed accounts for one-third of traffic accidents (the article claims it is actually somewhere between 5% and 10%) and that speed cameras reduce fatalities by 35% (a claim that is hard to support).

For thirty years the UK had impressive reductions in traffic fatalities, making it the safest country in Europe. But for the last decade the numbers have stabilised, even increasing a little. The article concludes that alcohol and drugs account for many if not most of the deaths, and they only a very small amount (2% to 3%) of accidents involved breaking the speed limit (70% of accidents caused or contributed to by inappropriate speed were at speeds below the speed limit).

I do not accept everything that is in the article. For example as a Councillor I have seen the statistics that show that speed cameras reduce the median and mean speed on residential roads, something which may not effect fatalities much, but does improve the quality of life. The article quotes opinion polls which say two-thirds of people don't like speed cameras. I will check this point further, but when Councils poll local residents who live on main roads and 'rat runs' a clear majority want cameras on their road to have speed cameras. Although they sometimes request cameras elsewhere to be removed!

But the main thrust of the article is clear and stark, the Government has misused the statistical data, and when challenged has dug its heals in and attacked those telling the truth. In the days before Web 2.0 this would often have worked. But, now, it will usually fail.

Structural Equation Modelling and Causality

I have just finished teaching a two-day multivariate modelling course in Brisbane for AMSRS (the Australian market research society), and one topic which led to an interesting discussion was SEM (Structual Equation Modelling) and causality. As Jon Pinnell, of MarketVision, says, SEM is “A source of excitement to some and great frustration to others.”.

At one level SEM is very appealing in that it appears to offer a grand theory of everything. The user creates a model, linking inputs to latent variables, to outputs, and then calculates all the relationships and errors in one go.

However, it seems there are two, large, problems with SEM.

1. To use SEM properly you should already have the model, before you have the data. SEM can then be used to see whether this set of data (e.g. this company, this situation, this process) confirms to the model. If the model correctly ascribes causality, and the data fit the model, then we can accept the causality in the SEM process. However, if the model has been developed from the data available, SEM should not really be run until a new set of data has been acquired.
   
2. The second problem, and it is one that applies to many other techniques in addition to SEM, is the dependence on assumptions about linear relationships between inputs and outputs. In the real world, we know many things are not linear; for example, a service can get a bit worse and nothing happens, then a bit worse and nothing happens, then a tiny bit worse and the contract is cancelled.

In reviewing the current state of thinking I came across the quote below about SEM, which I think sums up the causality issue quite well.
“The term causal model must be understood to mean: ‘a model that conveys causal assumptions,’ not necessarily a model that produces validated causal conclusions.”

What the stats tells us about prostate cancer, and what they don't

Like many men past 40 I have started to pay more attention to articles in the press about prostate cancer. But the statistics seem more opaque than almost any other data presented to the public. At one level we have the stats that prostate cancer is the most common or second most common form of cancer amongst men in the UK, killing about 10,000 men a year. As a result there have been campaigns to raise awareness of the symptoms, including one using old recording of Bob Monkhouse, a UK comedian who died of prostate cancer.

On the other hand there is a growing backlash against the PSA test (a blood test which is supposed to indicate an increased likelihood of prostate cancer), and against treatment. The complexity appears to be that prostate cancer develops very slowly in most cases.

The following statistics were issued as part of a BBC radio broadcast on 6th September 2007 (the Check UP programme), and I can make no claim for their veracity.

• Only 5% - 10% of men who have prostate cancer will die of it.
• About 30% of men who have a positive PSA test will actually have prostate cancer
• About 25% of men, in the same age group, who have not had a PSA test will have prostate cancer

The figures above suggest that, in the absence of other symptoms or causes, the PSA test is not very useful. A digital rectal examination (in this case the digit means a finger, not some form of technology) is more indicative, but men are reluctant to ask, and many doctors seem reluctant to bring the topic up. A prostate biopsy is highly accurate, but painful, expensive, and can cause issues such as bleeding.

One of the main concerns, of some specialists, like the specialist on the BBC, is that to know that you have a cancer is worrying. But for 90-95% who have the cancer it will not kill them, and in most casement no treatment is the best treatment. Is it better for most people who have the non-threatening form of prostate cancer not to know?

A similar argument applies to some of the genetic tests that private companies are beginning to offer. Do people benefit from knowing they have an increased risk of disease A or B, or are they better getting on with their lives?

A complete transcript of the programme will be available on the BBC website in a few days.