The Ghost of Statistics Past

So that we can discuss it, let’s name the above principle. We call it quasi-contraposition, because one specious argument for it proceeds as follows. Suppose \(A\) implies \(B\). By the law of contraposition, if \(B\) is false, then it follows that \(A\) is false:

\[ A \rightarrow B \implies \neg B \rightarrow \neg A \]

Replace \(A\) with \(H\) and \(B\) with “\(\neg D\) is likely”:

\[ H \rightarrow \neg D \mbox{ is likely } \implies D \mbox{ is likely } \rightarrow \neg H \]

We could ostensibly move the "is likely" to the right-hand side:

\[ H \rightarrow \neg D \mbox{ is likely } \implies D \rightarrow \neg H \mbox { is likely } \]

We have the law of quasi-contraposition. The last manoeuvre seems shady, but perhaps we could fool someone with enough bluster: "You see, since \(D\) happened, it suggests \(D\) was likely to happen, which as we know implies \(\neg H\). Of course, one does not simply conclude \(\neg H\), because there was a small chance that \(D\) is unlikely but it happened anyway. So we conclude \(\neg H\) is likely."

We work through the procedure given in my textbook for the sequence \(S\). We’re instructed to use the binomial distribution for this problem, so we count 12 heads in \(S\), and compute \(P(X\ge 12)\), that is, the probability of seeing at least 12 heads in 20 flips of a fair coin:

\[ \sum_k {20 \choose k} 2^{-20} [12 \le k \le 20] \]

We can compute this with a little Haskell:

The probability is indeed a bit larger than 0.25, and since this is greater than 0.05, we deem our finding insignificant, that is, we conclude we lack strong evidence to dispute the hypothesis that the coin is fair.

What if we replace 12 with 20?

We get a probability far smaller than 0.05, so for a coin that shows heads each time, we conclude there is strong evidence that our fair-coin hypothesis is false.

Although the end result matches our intuition, there are peculiarities in the procedure:

In other words, we deliberately throw away information. Twice.

Why do we wilfully neglect some of our data? If a ghost could whisk me back to my undergrad days, perhaps I’d witness my younger self say: "The probability of seeing any particular sequence, such as \(S\), is always \(2^{-20}\), so focusing on a particular sequence obviously fails. Since each coin flip is independent of the others, it makes sense just to count the number of heads instead."

"As for the inequality: the probability of seeing exactly \(k\) heads is:

\[ P(X = k) = {20 \choose k} 2^{-20} \]

which is always too small to work with. If we replace it with \(P(X \ge k)\) for large \(k\) and \(P(X \le k)\) for small \(k\) then we get a probability that is tiny for extreme values of \(k\), but huge for reasonable values of \(k\). In other words, we get a number that can distinguish between likely and unlikely \(k\)."

In short, my past self would say we do what we do because it works. We play around until we find a quantity that almost disappears when we want it to. It’s practical (we need only compare against 0.05) and convincing (because it involves fancy mathematics).

Isn’t this intellectually unsatisfying? On the one hand, it certainly sounds better to say "following standard procedure, the P-value is less than 0.05; therefore we have significant evidence the hypothesis is false" instead of "\(k\) seems kind of extreme so the hypothesis is probably false". On the other hand, if we’re going to all this trouble to quantify how strongly we believe a hypothesis is true, why not do a proper job and justify each step, rather than settle on some ad hoc procedure?

Perhaps the procedure only appears ad hoc because the derivation is omitted to avoid scaring students fresh out of high school. Let’s suppose this is the case and try derive probability theory from first principles, one of which the authors insist is quasi-contraposition.

We have a coin. Our hypothesis is that it is fair. The probability of seeing any particular sequence of 20 flips such as \(S\) is \(2^{-20}\), which is tiny. By quasi-contraposition, seeing such an "unusual" outcome means our hypothesis is likely wrong. So no matter what, we should always believe the coin is unfair!

By the same token, consider rolling a \(2^{20}\)-sided die that we believe to be fair. After a single roll, we see a number that has a \(2^{-20}\) chance of showing up. Wow, this is much less than 0.05! The die must be loaded!

The inescapable conclusion: quasi-contraposition is wrong.

If quasi-contraposition is wrong, then what is right?

Whatever it is, it must capture our intuition. If we flip a coin 20 times and see 20 heads, we suspect the coin is unfair. If we see the sequence \(S\), we are much less suspicious. Either event occurs with probability \(2^{-20}\) so there must be other information that affects our beliefs. What could it be?

The answer is that we are aware that trick coins exist, and willing to entertain the possibility that the coin in question is such a coin. For a fair coin, the probability of seeing 20 heads in a row is \(2^{-20}\), but for certain trick coins the probability is much higher. Indeed, an extremely unfair coin might show heads every time. We think: "Is this a fair coin that just happened to come up heads every time, or is this a trick coin that heavily favours heads? Surely the latter is likelier!"

How about the sequence \(S\)? For a fair coin, the probability of seeing the sequence \(S\) is also \(2^{-20}\). But this time, we feel:

We can mathematically formalize these thoughts with one simple trick. Rather than \(P(D|H)\), we flip it around and ask for \(P(H|D)\). In other words, given the data, we find a number that represents how strongly we believe the hypothesis is true.

The probability \(P(H|D)\) is the one true principle we’ve been seeking. It’s the truth, the whole truth, and nothing but the truth. It’s the number that represents how strongly we should believe \(H\), given what we’ve seen so far. With \(P(H|D)\), the difficulties we encountered melt away.

We can compute \(P(H|D)\) with Bayes' Theorem:

\[ P(H|D) = P(H) P(D|H) / P(D) \]

Thus our previous work has not been in vain. Computing \(P(D|H)\) is useful; it’s just not our final answer.

What about \(P(D)\)? This is the probability that \(D\) occurs, but without assuming any hypothesis in particular. Or, more accurately, with default degrees of belief in each possible hypothesis; degrees of belief held prior to examining the evidence \(D\). Similarly, \(P(H)\) is how strongly we believe \(H\) to be true in the absence of the data \(D\).

Let us say we are willing to consider the following 11 hypotheses: the coin shows heads show heads with one of the probabilities 0, 0.1, 0.2, …​, 1. Furthermore we believe each possibility is equally likely.

First suppose our data \(D\) is 20 heads in 20 coin flips. As before, let \(H\) be the hypothesis that the coin is fair. We find:

which is:

We have \(P(D|H) = 2^{-20}\), and \(P(H) = 1/11\), hence:

In other words, our belief that the coin is fair has dropped from \(1/11\) to less than one in a million.

Now suppose our data \(D\) is the sequence \(S\). This time:

which is:

Even though \(P(D|H)\) is again \(2^{-20}\), we find:

Thus our belief that the coin is fair has increased from \(1/11\) to over \(1/4\).

The Bayesian approach has outdone my textbook. We get meaningful results without throwing away any information. We used the entire sequence, not just the number of heads. No inequalities were needed.

What if we discard information anyway, and only use the fact that exactly 12 heads were flipped? In this case, we find:

and \(P(D|H) = {20 \choose 12} 2^{-20}\). When computing \(P(H|D)\), the factor \({20 \choose 12}\) cancels out, and we arrive at the same answer. In other words, we’ve shown it’s fine to forget the particular sequence and only count the number of heads after all. What is not fine is doing so without justification.

It is also reassuring that using all available information gives an answer that is at least good as using only partial information (in this case, they agree). Contrast this to quasi-contraposition, which leads to nonsense if we focus on a particular sequence of flips.

What if we go further and introduce inequalities as before? The probability that we see at least 12 heads over all possible coins is:

And for a fair coin:

\[ P(D|H) = \sum_k {20 \choose k} 2^{-20} [12 \le k \le 20] \]

We find:

and hence \(P(H|D) < P(H)\). That is, the evidence weakens our belief that the coin is fair. Recall seeing exactly 12 heads strengthens our belief that the coin is fair, so by introducing an inequality, we discard so much information that our conclusion runs contrary to the truth.

The above is enough for me to shun my frequentist textbook and join the "Bayesian revolution":

E. T. Jaynes, Probability Theory: The Logic of Science. Laplace wrote: "Probability theory is nothing but common sense reduced to calculation". Jaynes explains how and why, though a vital step in his argument, Cox’s Theorem, turns out to require more axioms.

David Mackay, Information Theory, Inference and Learning Algorithms. This free book presents compelling applications for Bayesian inference: information theory; data compression, Monte Carlo search; neural networks.

David Mackay: Sustainable Energy: without the hot air. Although unrelated to probability, I recommend it here because Mackay again clearly explains how to navigate correctly in an area infested by influential charlatans that attempt to mislead us with labyrinthine arguments, obscuring their speciousness with complexity.

Persi Diaconis and Brian Skyrms, Ten Great Ideas about Chance (not free). Historical, and less technical. If I didn’t already believe the frequentist emperor is wearing no clothes, this book would have convinced me. It makes me wish I could go back and Socratically question my old professors. After they’ve defined probability in terms of frequencies, I could ask "stupid" questions like: "What does it mean to run the trial again? If I roll the dice the same way, won’t I get the same result?" They also coin a better term for what I called quasi-contraposition: Bernoulli’s swindle.

John Ioannidis, Why Most Published Research Findings Are False: Does what it says on the tin, in just a few pages. While frequentism is not the only culprit, it’s responsible for much of the damage.

Gerd Gigerenzer, Statistical Rituals: The Replication Delusion and How We Got There. What did scientists do before frequentist nonsense? How did frequentism become nonsense? (Did you know it’s partly due to commercialization of textbooks? One might even call it predatory publishing.) What can we do about it? Gigerenzer uses another cute name for Bernoulli’s swindle: Bayesian wishful thinking, as well as the more formal-sounding inverse probability error.

xkcd, Frequentists vs. Bayesians.

xkcd, Significant.

In 2015, Fred Ross kindly informed that this page got posted to Hacker News. I’m also grateful he took the time to present an alternative view:

I consider these statements outmoded, and predict that one day, they will be widely seen to be false.

Of course, I do agree with part of one sentence: "Jaynes was a wonderful thinker". This is indisputable, as Jaynes realized the principle of maximum entropy goes far beyond physics and is in fact a general principle of reasoning. But not only did he take something from physics to change reasoning; he took something from reasoning to change physics! Since a probability is a degree of belief, it follows that entropy is also subjective.

Cox’s theorem is the underlying justification for Bayesian reasoning. In particular, if there are multiple ways to solve a problem, naturally we desire confluence: all roads ought to lead to the same solution. Non-Bayesians view this as unnatural!

As for decision theory, see Chapter 36 of Mackay for a one-equation summary. Decision theory builds on Bayesian reasoning; to justify the latter with the former is to put the cart before the horse.

Appealing to psychology is dubious. There exist many humans, known as sampling theorists or frequentists, who somehow reason without a sound mathematical foundation. Why then should a utility function be universally appropriate?

Less facetiously, Gigerenzer suggests that psychologists share the blame for frequentist nonsense:

Props to Ross for mentioning Neyman and Pearson, as psychologists seem embarrassed about their influence:

Chapter 37 of Mackay demolishes frequentism/sampling theory with flair and proposes an ingenious compromise. Mackay first observes that "from a selection of statistical methods," sampling theorists pick "whichever has the 'best' long-run properties". Thus to sneak Bayesian reasoning past them, simply state you’re choosing the method with the 'best' long-run properties, while taking care to avoid the word "Bayesian". I propose we use the phrase "Mackay’s correction"; for example, "the chi-squared significance test with Mackay’s correction" might mollify reviewers suffering from frequentism.

Mackay’s favourite reading on this topic includes: Jaynes, 1983; Gull, 1988; Loredo, 1990; Berger, 1985; Jaynes, 2003. Mackay also mentions treatises on Bayesian statistics from the statistics community: Box and Tiao, 1973; O’Hagan, 1994.

The phrase "in the middle of the 20th century" calls to mind World War II, when Allied codebreakers applied Bayesian reasoning to break Germany’s Enigma cipher. Their methods "haven’t held up well"? Which side won?

One might cut Ross some slack by claiming ignorance: Turing’s papers were only declassified in 2012. But GCHQ allowed declassification only because by 2012, everyone (except frequentists!) was well-aware that Bayesian reasoning is a formidable tool for codebreakers.

If we want to talk about 20th-century claims that haven’t held up well, how about Fisher’s eugenicist views on race and "miscegenation"? Or Fisher’s insistnet denials that smoking causes lung cancer. They seem to have fallen out of fashion, and at last, FIsher’s sampling theory is also en route to oblivion.