Infernal Ramblings
A Malaysian Perspective on Politics, Society and Economics

Anecdotal Evidence is No Evidence At All

Written by johnleemk on 2:18:03 pm May 30, 2007.

One common problem when discussing policy is people who try to bring in and rely on anecdotal evidence. As compelling, heartwrenching or simply interesting these stories may be, they do not make for good policy.

The implication made whenever ancedotal evidence is brought up is that these "real stories" are not outlying data points; that they are an accurate and typical representation of the population.

But as any statistician can tell you, a sample of a few dozen will not do for a population of twenty million, let alone a sample of two or three for a similarly sized population!

Moreover, the sample chosen is obviously biased towards data points which support the predetermined conclusion of whoever is trying to push them; there is no real objective attempt to look at an appropriate random sample of the population and see what conclusions can be drawn from them.

In debates between armchair policy wonks, this is not too big a problem. Earlier tonight, I was following an internet discussion group devoted to criticising Singapore's policy of paying its politicians million-dollar salaries.

One person criticised those who advocated a more liberal democratic form of government, bringing up examples of civil rights abuses in liberal democracies such as the United States and United Kingdom.

The person who gave the rebuttal merely focused on the inaccurate presentation of these abuses, and how there was no evidence of actual abuse found. But there really was no need.

How can you possibly generalise from a sample of one or two to a population of over 50 million (for the UK) or 300 million (for the US)? How can you justify this on any reasonable, sensible, logical grounds?

Are you really going to say because in one case, the police of some rural American county tried to frame an innocent man, the logical conclusion is that in liberal democracies, such abuses are just as likely to happen as they would in an authoritarian state like Singapore?

Fortunately, the damage armchair policy wonks can do is limited. But this sort of horrifying reasoning is not limited to just armchair policy wonks alone; the "leader of the free world", as US presidents were once billed, is just as susceptible.

For instance, during the 1990s, Paul Krugman criticised Bill Clinton for appearing to base his policies on a shrinking middle class, simply because there was anecdotal evidence of middle class layoffs. Krugman presented statistical evidence that it was the lower economic strata which had suffered, and not the middle class, which had remained stable.

Likewise, the catastrophe of the Iraq war may have been based on similar illogical thinking. The George W. Bush administration's main exposure to Iraqis was to people like Ahmad Chalabi, who were blatantly and openly pro-American.

Thus, generalising from such a small and biased sample to the whole Iraqi population, Vice-President Dick Cheney proclaimed that the invading forces would be welcomed as liberators. We all know how that turned out, don't we?

Anecdotal evidence is no evidence at all. What we need are cold, hard facts, not wishy-washy stories which sound compelling, but don't really tell us anything significant.

If you'd like to keep informed about updates to the site, consider subscribing to our web feed:

Infernal Ramblings is a Malaysian website focusing on current events and sociopolitical issues. Its articles run the gamut from economics to society to education.

Infernal Ramblings is run by John Lee. For more, see the About section. If you have any questions or comments, do drop him a line.

Najib's Orwellian 1Malaysia

Most Recently Read

  1. Sepet, A Malaysian Movie
  2. Malaysia, A Statist Economy
  3. Malaysia is Not a Federation
  4. Ad Hominem: How Malaysians Lose the Plot
  5. Segregated Schools: Does Quality Justify the Costs?
  6. Apartheid and Protectionism, Internal Issues?
  7. What does it mean to be Malaysian?
  8. Absolute vs Comparative Advantage
  9. Dissent, the Highest Form of Patriotism?
  10. Separating Head of State from Head of Government
Quoth the webserver...
Those who make peaceful revolution impossible will make violent revolution inevitable.
— John F. Kennedy