This is the printable version of an article from Infernal Ramblings ( The original web-optimised article is also available.

Anecdotal Evidence is No Evidence At All

A sample of two is a statistically worthless sample.

Written by johnleemk on 2:18:03 pm May 30, 2007.

One common problem when discussing policy is people who try to bring in and rely on anecdotal evidence. As compelling, heartwrenching or simply interesting these stories may be, they do not make for good policy.

The implication made whenever ancedotal evidence is brought up is that these "real stories" are not outlying data points; that they are an accurate and typical representation of the population.

But as any statistician can tell you, a sample of a few dozen will not do for a population of twenty million, let alone a sample of two or three for a similarly sized population!

Moreover, the sample chosen is obviously biased towards data points which support the predetermined conclusion of whoever is trying to push them; there is no real objective attempt to look at an appropriate random sample of the population and see what conclusions can be drawn from them.

In debates between armchair policy wonks, this is not too big a problem. Earlier tonight, I was following an internet discussion group devoted to criticising Singapore's policy of paying its politicians million-dollar salaries.

One person criticised those who advocated a more liberal democratic form of government, bringing up examples of civil rights abuses in liberal democracies such as the United States and United Kingdom.

The person who gave the rebuttal merely focused on the inaccurate presentation of these abuses, and how there was no evidence of actual abuse found. But there really was no need.

How can you possibly generalise from a sample of one or two to a population of over 50 million (for the UK) or 300 million (for the US)? How can you justify this on any reasonable, sensible, logical grounds?

Are you really going to say because in one case, the police of some rural American county tried to frame an innocent man, the logical conclusion is that in liberal democracies, such abuses are just as likely to happen as they would in an authoritarian state like Singapore?

Fortunately, the damage armchair policy wonks can do is limited. But this sort of horrifying reasoning is not limited to just armchair policy wonks alone; the "leader of the free world", as US presidents were once billed, is just as susceptible.

For instance, during the 1990s, Paul Krugman criticised Bill Clinton for appearing to base his policies on a shrinking middle class, simply because there was anecdotal evidence of middle class layoffs. Krugman presented statistical evidence that it was the lower economic strata which had suffered, and not the middle class, which had remained stable.

Likewise, the catastrophe of the Iraq war may have been based on similar illogical thinking. The George W. Bush administration's main exposure to Iraqis was to people like Ahmad Chalabi, who were blatantly and openly pro-American.

Thus, generalising from such a small and biased sample to the whole Iraqi population, Vice-President Dick Cheney proclaimed that the invading forces would be welcomed as liberators. We all know how that turned out, don't we?

Anecdotal evidence is no evidence at all. What we need are cold, hard facts, not wishy-washy stories which sound compelling, but don't really tell us anything significant.