Statistical Fallacies

Availability Bias

We often judge how common or likely something is by how easily examples of it come to mind. Psychologists Amos Tversky and Daniel Kahneman called this the availability heuristic. It is a handy shortcut, but it misleads us whenever some events are much easier to remember than others.

Examples
Why it is a statistical error

The examples that come to mind are a sample, but they are not a random sample. They are chosen by what is dramatic, recent, or emotionally striking, and that is a biased way to sample.
The remedy: before deciding that something is common or rare, ask what the actual counts or rates are, rather than relying on what you happen to remember.

The availability heuristic on Wikipedia.

The Gambler's Fallacy

The Gambler's Fallacy

The Monte Carlo Fallacy

Ignoring Fat Tails

Fat Tails

Discarding unfavorable data

A conclusion can only be as good as the data behind it. If the data that would have argued against a conclusion are left out, even honest arithmetic on what remains will mislead. This can be done on purpose ("cherry picking"), or it can happen without anyone intending it.

Cherry picking

Reporting only the results that support your case.
Example: A fund manager advertises "up 40% over the last three years," choosing a starting date right at a market low. Starting the chart a year earlier, at the peak, might show a loss.
Example: A researcher removes a few data points as "outliers," but only the ones that weaken the result.

Survivorship bias
A bomber with red dots marking where returning planes had been hit
Damage on the bombers that made it back. (Image: Martin Grandjean, McGeddon, US Air Force; CC BY-SA 4.0)

Sometimes the unfavorable data are discarded by the world before we ever see them: we only get to look at the cases that "survived."
Example: In World War II, the military studied the bullet holes on bombers returning from missions, and considered adding armor where the holes were most common. The statistician Abraham Wald pointed out that these were the planes that made it back. The places with no holes, such as the engines, were where hits brought a plane down. That is where the armor was needed.
Example: "They don't make them like they used to!" The old buildings and appliances we still see are the ones that lasted. The badly made ones were torn down or thrown out long ago.
Example: Average returns of mutual funds look better than they really were if the funds that did badly and closed are left out of the average.

Publication bias and multiple tests

Studies that find a "significant" result are much more likely to be published than studies that find nothing. The studies that found nothing stay in a file drawer.
This matters because a test at the 5% significance level will, by design, give a "significant" result about 1 time in 20 even when there is no real effect. Run 20 tests, report only the one that "worked," and you have discarded the 19 pieces of unfavorable data that put it in context. (See the z and t tests page for what significance means.)

Protecting yourself

Cherry picking, survivorship bias, and publication bias on Wikipedia.
xkcd's "Significant": multiple testing in one comic.

External links