Have we abused the p-value?

When it comes to the ubiquity of statistics, Russ Swan implores us to watch our P's and Q's.

It’s complicated. I don’t mean your social media relationship status, I mean, you know, everything. The universe. Science. The great questions of our time.

And especially the answers – which is where we have a problem. Society doesn’t want complicated, it wants simple. Ever since some of the great mysteries were reduced to short equations and brief explanations like f=ma and e=mc², we scientists have encouraged the rest of the human race to expect more of the same. We’ve trained them to expect simplicity, and we’ve delivered complexity.

The recent spat over alcohol consumption is a good case in point. In early January, just as some of us were ridding ourselves of the substantial hangover we choose to start each new year with, the UK’s chief medical officer changed the official advice over alcohol consumption. Instead of tolerating a unit or three each day, we were now told that there is no ‘safe’ level. This position was “supported by a new review from the Committee on Carcinogenicity (CoC) on alcohol and cancer risk”. Good scientific evidence, then. Alcohol is dangerous at any level of consumption. The committee says so. The problem is that there is another body of evidence that shows positive health benefits from limited consumption. Moderate drinkers enjoy better cardiac and mental health, for instance.

[caption id="attachment_53014" align="alignnone" width="400"] The p value is used to show how significant (or not) a result is, in an attempt to simplify the maths behind the research.[/caption]

How can it be that alcohol is both harmful and beneficial? Because (and you’re way ahead of me here), it’s complicated. There isn’t one measure of ‘health’, any more than there is one disease. You drink a little, you reduce the risk of this but increase the risk of that. In other words, name your poison. A popular recent vehicle to convey complex information in a simple and easy-to-digest manner is the p-value. This is a statistical device that is increasingly used in research reports to indicate how meaningful the numbers really are – whether the results of, say, a drug trial are statistically significant.

The p-value has everything we could want in a metric. It is short, simple, and easily explained to the non-experts who may well be judging the next research application. And it has become just about the most popular statistic in science today. Between 1990 and 2014, a recent review found, the reporting of p-values in biomedical abstracts more than doubled. In the core medical journals, 33% of abstracts quote it, and within randomised clinical trials it is 55%. The p-value is everywhere – it is the grumpy cat of current science. And that’s great, isn’t it? Who doesn’t love grumpy cat?

Between 1990 and 2014, a recent review found, the reporting of p-values in biomedical abstracts more than doubled

Get this. Last month the American Statistical Association issued a warning about p-values, highlighting the ‘misuses and misconceptions’ of the number and hoping to usher in a new ‘post p<0.05 era’ (this being the commonly-reported threshold for results to be presented as a genuine effect rather than random). Boston University epidemiologist Kenneth Rothman went further, claiming “It is a safe bet that people have suffered or died” because of the abuse of the p-value.

Suffered or died, because of a handy but (it turns out) deadly statistical convenience. To say it’s ‘complicated’ doesn’t even begin to address the issue. In every field of science, there is complexity that has to be reduced to simplicity in order to communicate. And every time that complexity is reduced, we make ourselves vulnerable to looking like fools (or worse) when the next piece of the jigsaw falls into place. What can be done? A few years ago the Climate Research Unit at the University of East Anglia had a server hacked, and thousands of emails and documents made public. Climate sceptics pored over the data, seeking a smoking gun to support their obsessive belief in some global scientific conspiracy.

For many, the instinctive response to such a breach is to tighten security, insisting that everybody comes up with new passwords to forget, and instigating two-step authentication. But there is another approach that might be worth considering. Sunlight is the best antiseptic, my old gran used to say. At the University of Toronto, biomedical researcher Rachel Harding thinks she is the first in the world to publish her lab notes in real time. Not her papers and conference presentations, her day-to-day raw data, thoughts, and scribbles. Her work is on Huntingdon’s disease, and she says the motivation is to leverage the experience of a community of scientists – and hopefully find suggestions and tips to avoid repeating mistakes or overlooking discoveries. Harding is also summarising her work in lay terms on a regular blog (delightfully called Lab Scribbles), hoping to remove some of the air of mystery surrounding what really goes on in a lab.

So, publicly-accessible research data in real time, accompanied by a lay-terms summary of progress. Sounds great to me. Of course it wouldn’t stop the loons believing you’re hiding the real facts on a secret server, and there will be many (many!) instances where valid commercial sensitivity would make this a non-starter. The reality is that the potential audience for this new open approach to research is probably quite tiny, but if anything that might encourage others to follow the lead. Would it really be so difficult, or so complicated, to do so?