Playlist 55: The one number that makes or breaks entire industries

Something different this week

View this email in your browser

I love podcasts because they can add an extra level of context to complicated issues that other types of media can’t. Whenever there is a subject I’ve read about or watched videos on, it’s typically a podcast that finally helps the information sink in. Hearing someone’s voice in my earbuds makes some sort of magic happen for my brain that doesn’t happen with reading or watching.

I only listened to a handful of podcasts this week, so I wanted to do something a little different and highlight one such episode that helped make things click for me. Also, get ready for some really professional illustrations.

The episode is: This one number can make or break careers and companies.

It’s from the team over at the science website STAT news (excellent science and medicine reporting) who make a podcast called Signal.

The subject they explain is something that at first seems niche or wonky, but I truly believe is really important for everyone to be aware of. We live in a world where this one number controls so much of our lives and we don’t even know it.

I’m going to tell you what it is, but I’m afraid your eyes might glaze over at the mere mention of it. But hang with me! It’s a number that, for better or worse, is creeping into our wider culture and holds tremendous power over the purse strings at major institutions. The P-Value!

Ok, are you still here?

Good.

So what is a p-value?
A p-value is a number between 0 and 1 that tells you how likely some study result is due to chance. A p-value of 1 means the results were pure chance, .05 means 5% likelihood (or 1 in 20) that the results were due to chance, and .001 means a .01% (or 1 in 10,000) likelihood that the results were due to chance. So the closer the p-value is to 0, the more sure you can be that whatever effect you are seeing is real and not due to luck.

Even though there is a lot going on in any one research study, a p-value equal to or less than .05 has become shorthand for “this is a great and true study that you should pay attention to”. Which means that researchers are pressured and biased (consciously or not) to have studies that at least hit this number.

Here's a very simplified look at what some study might show:

This drug looks like it has a great effect, but one important thing to keep in mind is that a p-value of .05 does have a 1 in 20 chance of being a fluke. So, if you look at 20 similar studies, one of them had a positive effect completely by pure luck!

This is something to keep in mind. This threshold of needing to reach .05 is arbitrary and many are arguing to make it lower.

Technical definition time
This is all a basic understanding of the p-value, and it’s worth quickly looking at the more accurate and a bit more technical definition of what a p-value is. The hosts of Signal say it is “how likely it is that you would see results similar or more extreme as the ones you would get if your null hypothesis is true.”

This null hypothesis sounds much more confusing than it really is. So basically, let's assume you want to test a drug, and your hypothesis is that it will improve back pain. Your null hypothesis would then be that the drug will have NO EFFECT different from placebo on the pain. Now let's say you complete the study and it looks like it has a pretty positive effect and you calculate that the p-value is .05.

The basic meaning of this is that there is a 5% chance that your results are due to a fluke. The more techical meaning of this is that assuming that this drug had no effect whatsoever, there is a 5% probability that you will have gotten this positive effect due to chance alone.

It’s good to know the technical meaning, but the basic meaning will be perfectly fine if you are not a researcher. Normal people don’t need to ever utter the phrase “null hypothesis”.

Why should I care?
"Facts are stubborn, but statistics are more pliable." - Mark Twain

Unfortunately, no matter what you do or how you spend your time, you are likely to be barraged by "this study says..." and statistical "proof" of something. P-value's have become this overpowered gatekeeper to what good science is and are used as a substitute for how you should feel about the data.

There is much more to a study than its p-value, but it is the thing most likely to be paraded around as proof that something has a significant effect. According to the team at Signal, a "p-value absolutely cannot tell you the strength of your evidence, it can't tell you the size of an effect, it can't tell you whether the finding is important or not."

Just look at the two drugs below. Both have the very respectable p-value of .001 (so almost guaranteed that there is a real effect), but which one would you rather take?

P-value abuse
You might hear about some study saying psychic powers have been proven to exist, and that they did everything by the book. It has a p-value of .05 after all, so it must be true!

Well here's the thing that even mainstream science can be guilty of. If a study shows a negative result, you can just keep it to yourself and try again. Unsexy negative finding = no journal wanting to publish. This means that if you are only shooting for a p-value of .05, you could keep trying until a positive result shows up randomly. Let’s see how this would look for our trusty hypothetical pain drug.

Luckily, most pharmaceuticals and major studies nowadays have to register ALL studies they do, but this trick can certainly happen for any number of studies you hear about in the news. You don’t necessarily know how many failures some newly touted psychology finding had until something interesting finally cropped up.

Be p-value wary
P-value's are valuable, but only in context with several other factors. How big is the effect? What is the hypothesis of the study? Has it been replicated? How probable is the hypothesis given what we know about the world? What is the study design?

So next time someone makes a highly dubious claim and throws a p-value out like it's all they need to say, keep on being dubious.

Give Signal a listen
The podcast does a great job expaining all this with real world examples. Also listen to hear about more shenanignas to be aware of like p-hacking.

ARTICLES AND OTHER LINKS

I’ll have the more traditional format next time, with a concise roundup of smart podcasts and links. For now, I’ll just leave you with this awesome post from Reddit where someone generously listed their top 100 podcasts and led a great discussion in the comments.

I learned of the above post from PodNews, an excellent daily email that bullet points podcast news. It’s like a bite sized and manageable version of Hot Pod.

That's all for this week!

Connect with me @erikthejones on twitter and if you've learned anything interesting, please forward this link to any curious natured friends or family so they can subscribe. Many thanks!

Erik