In All Probability
As a teenager, I had the good fortune to meet Tom Körner. He had written a book, The Pleasures of Counting, that opens with a story about John Snow.
Snow was a 19th-century English physician who painstakingly collected and analyzed vast amounts of data to convincingly argue that cholera spreads through contaminated drinking water, and not, as was once widely believed, from some kind of air pollution.
This struck me as a powerful example of how mathematics, and in particular, statistics, can impact our lives. How many more would have succumbed, had the true cause remained hidden?
Yet today, probability and statistics seem much maligned. Statistics are worse than "damned lies"; they’re "pliable"; one can "prove anything by statistcs except the truth"; they are the means to produce "unreliable facts from reliable figures".
How did this happen? I’m sure these famous quotes were composed mainly in jest, and perhaps referred to shady accounting more than actual calculation. But these days, even the mathematics itself seems suspect:
-
The Reformation: Can Social Scientists Save Themselves? (May 2014)
-
Scientific method: Statistical errors (February 2014)
-
The Curse of P-values (November 2013)
-
Revised standards for statistical evidence (October 2013)
Statistics is indeed a troubled subject. It turns out some guy named R. A. Fisher is to blame. Fisher had a tragic combination of gifts and flaws that led to today’s erroneous orthodox statistics. (Despite an ever-growing mountain of evidence, Fisher steadfastly refused to believe smoking causes lung cancer. How good could his methods be?)
My undergrad introductory course on probability and statistics followed Fisher’s dogma. As a result, I felt that the methods they taught seemed more like black magic than mathematics. But I was convinced that the lecturer only seemed to be teaching superstitions because my understanding was too shallow, and I concluded I must have a poor intuition for the subject.
Years later, and determined to conquer my weakness in this area, I went back to my textbook. And some other books. I discovered the shocking truth: my textbook is wrong. For once, a crazy conspiracy theory was true and They really were corrupting us all with Their false mathematics.
Epilogue
More years have passed since I wrote the above. I feel vindicated. The world is slowly but inexorably moving past 20th-century mistakes.
Far too slowly, in my opinion, as the stubborn resistance of old guard will continue to impede scientific progress. As the old guard has always done: today, it is widely accepted that dangerous waterborne pathogens are pervasive and difficult to detect, but when John Snow first presented his evidence, government inspectors rejected his findings.
Even the local vestrymen publicly sided against Snow. However, privately, deep down, they must have harboured doubts, as they appointed an investigative committee who, despite government obstruction, would one day confirm Snow’s theories. Some credit is due to the Reverend Henry Whitehead, who essentially replicated Snow’s work, painstakingly collecting and analyzing a mountain of evidence that forced him to change his mind, and thankfully, the minds of others.