This is the printable version of an article from Infernal Ramblings (infernalramblings.com). The original web-optimised article is also available.

Preventing unnecessary alarmism and expenditure.

Written by johnleemk on 12:42:28 pm Jun 13, 2007.

Humans are immensely prone to relying on anecdotal evidence instead of statistical data and analysis to test and confirm or disprove hypotheses. While for the individual, relying on gut feel may not be such a bad thing, it can be disastrous for policymaking.

Normally, when confronted with an unusual situation, we tend to emphasise things we see in it that are not necessarily there. For instance, if I see a tiger crossing the road, I may very well be tempted to launch into an extraordinarily complex theory of how that tiger got there — ignoring of course that the probability of a tiger crossing the road in any area is almost always non-zero (just extraordinarily small).

To determine whether a particular circumstance is so extraordinary as to necessitate a reevaluation of our underlying assumptions and hypotheses, statisticians calculate the statistical significance of the observations — whether they can be explained within the bounds of our existing beliefs.

The process of determining statistical significance is a bit of a technical thing, so I won't delve into that. However, there is one detail worth mentioning — the standard deviation.

The standard deviation measures how far an incident is likely to deviate from the mean (also known as expected) result. In layman's terms, the standard deviation is a rough gauge of what constitutes an extraordinary incident.

This standard deviation is used to calculate the statistical significance of the data observed and collected. While this information is not particularly useful in one's daily life — I can't determine the standard deviation of the probability distribution for the amount of money I find on the sidewalk — knowing and understanding the concept of statistical significance can be helpful.

There is of course a bit of demystification of life involved when statistics comes into play. I know one physicist who pointed out that if a miracle is a one in a million event, there are 86,400 seconds in a day, multiplied by six billion people in the world, so a miracle is nothing particularly extraordinary.

For a similar reason, thanks to the concept of statistical significance, if I find a 50 dollar bill on the sidewalk, I am more likely to chalk it up to probability than some supernatural blessing — though of course the reverse would occur if this happened more than once in quick succession, since the statistical significance of finding, say, three 50 dollar bills on the sidewalk on the same day is probably going to tell you that something odd is happening.

What is the relevance of all this to policymaking? Policymakers should not be worrying themselves about anecdotal evidence or basing their decisions on data which may seem alarming but in reality can be explained by existing assumptions and hypotheses.

(This actually calls to mind an incident where many people were suggesting there had been a tidal shift of public opinion based on the results of a particular election. One of my friends conducted a statistical analysis of the results and found that there was insufficient evidence to support this hypothesis — the deviation in voting patterns fit within acceptable statistical boundaries.)

Policymakers may be shocked to see, say, an increase in the vehicle accident rate within a particular area of, say, 50%. But if the standard deviation of the accident rate suggests increases and decreases of 100% are nothing out of the ordinary, then such alarmism would lead to bad policymaking, concentrating resources on preventing motor accidents when they could possibly be better spent elsewhere.

There is not much room for emotions in making policy decisions. Emotions make for good politics, but bad policies — it is hard data and analysis which must be relied upon.