To Think Like a Statistician

Statistics professor Susan Holmes said that statistics is like Poland, perpetually surrounded by powerful neighbours — probability, mathematics, applied science, and now artificial intelligence — yet surviving as an independent and increasingly formidable entity.

In “To Think Like a Statistician” Bradley Efron champions the statistical mindset. That mindset, like economics, is both technical and holistic in nature. It is needed for statistical inference (the process of drawing conclusions about a broader population based on sample data, while accounting for uncertainty and errors) but also for decision-making, risk management, and evaluating evidence across many domains. It is needed to work with a random universe. If you are an economist who wants to sit back and contemplate the state of advice and evidence, this is a good book for you. (I would not read this book if you simply want to brush up on correlations, regressions, t-statistics and significance levels.)

The book is infused with Efron’s engaging charm, humour and patience. He uses many diagrams, anecdotes, stories and simple examples that make you feel at least that you’re keeping up with a very clever person. I loved it when he confessed that in 2011 he kept his own dataset of celebrity deaths which showed him how much pattern and variation there is. (There’s a lower tail of rock stars who died young. But then, Keith Richards and Joni Mitchell are both still alive at 82).

The first chapter sets out the stall on statistical thinking. The next tackles probability, Bayes’ rules, and the nuances of interpreting medical news. From there on, he guides the reader through crucial comparisons; learning from experience and prediction; untangling correlation from causation; the defensive mechanics of error estimation, decision-making under regret; and survival and hazard. Why might you want to rethink statistics if you have been using statistics all your working life? Because the relentless drop in computing and data storage costs has fundamentally altered how statistics is practiced.

In the past, analysis was a linear pipeline where statistical thinking was more of a one-off stage between raw data stage and a decision. Today, cheap computing has led to probabilistic decisions being done in a continuous feedback loop: data drives estimation, inference informs automated decisions, decisions trigger immediate actions (such as software user A/B testing or live algorithmic adjustments), and actions (often on apparently willing users) produce new streaming data!

In this exciting and terrifying world, statistical thinking cannot be treated as a single, isolated step. A case in point I came across in my own work: In Retrieval-Augmented Generation (RAG), an LLM queries a vector database to retrieve k candidate context chunks or evaluates multiple generated claims across a document. Evaluating dozens of retrieved candidates simultaneously creates a multiple hypothesis testing problem. To prevent the LLM from relying on spurious or irrelevant retrieved chunks (which leads to hallucinations), modern statistical retrieval frameworks apply the Benjamini-Hochberg procedure discussed in the book to control the False Discovery Rate (FDR).

Thus, the statistical mindset matters more than ever as we guzzle from a continuous firehose of rapid, cheap, and more probabilistic and more automated decisions.

I also note that, apart from reading Efron’s book, you can always follow this debate by subscribing on Substack to some good quality writers. Take, for instance, Andrew Gelman’s blog, where he recently challenges the common thinking that larger samples are always better than smaller samples when it is really about understanding the extent of noise.