Chris Kresser just published a nice article on how to read and understand scientific research. Basic understanding of scientific research and how to interpret it is vital in navigating the confusing jungle of health claims you encounter every day online. That’s why I’ll summarize the point Chris made here, and also recommend you check out his article.

This post has nothing to do with acne. And if understanding scientific research doesn’t interest you in the least, then this post is not for you. Move along, nothing to see here.

In today’s world of conflicting interests, flawed science, and sensationalized media, it’s important to question new claims and findings, especially when those findings could have serious implications for your health. One of the most important things you can do to make sure you’re getting the real scoop is to read the scientific literature yourself.

Chris Kresser

I don’t mean that you’ll become a master wizard at reading obscure scientific papers, but that’s not really the point. Because even a cursory understanding of how scientific research helps you to spot the biggest nonsense.

The article Chris wrote is a good start. I’d like to add the following to his points.

Understanding why epidemiological studies can’t establish correlation

One example irks me is how many alternative doctors demonize artificially sweetened sodas based on epidemiological studies. They say that people who drink diet sodas are YY% more likely to be overweight than people who don’t drink diet sodas.

The unstated implication is that diet soda made the people fat, but you can’t make that conclusion based on those studies. Because a far more likely explanation is that people drink diet sodas because they are overweight, because they want to reduce calories and lose weight.

You’ll see Drs. Mark Hyman, Joe Mercola and other alt-med celebrities making such misleading statements all the time, and especially when it fits the spin they want to push.

To make a point, a skeptic at Reddit produced this graph.

Stop the press! Organic food causes autism!! It’s almost a perfect correlation!

You get the point. Thought the graph looks very convincing, nobody seriously claims that organic food causes autism. That’s why you also shouldn’t take seriously the graphs showing a relationship between artificial sweeteners and neurological problems – or whatever substance they next want you to be afraid of.

Understanding study quality and bias

I can’t say about you, but when I didn’t understand scientific research, I had this impression of how infallible scientific research is. That once a study had been published, then that’s it. The matter has been settled (because a study said so). Oh, how wrong was I, lol.

There’s a huge amount of bias and other nonspecific effects (sometimes referred as the placebo effect, which has nothing to do with mind over matter or anything like that) that make it hard it interpret individual studies.

Humans are inherently biased, and we want our cherished notions to be true. Scientists are no different in this regard. They often fall in love with they own hypotheses and want to see them proven. Unfortunately this can cause a scientist to, often subconsciously, devise a biased experiment that’s more likely to confirm his/her hypothesis.

The entire scientific method was created to fight against such biases we humans come hardwired with. That’s why we need things like controls, blinding and randomization to minimize the effect of these biases. Without such protections you can never know how much of the observed effect was due to the treatment and how much due to nonspecific effects.

Due to bias in scientific studies, you can (almost) always find a study to support whatever you want, heck there are tens of studies showing magic water (i.e. homeopathy) ‘works’. That’s why you should never put too much faith into findings of a single study. For us to be able to conclude something, the initial positive studies need to be replicated by other, indent researchers.

Hierarchy of scientific evidence

The fact is that scientific studies can be expensive. Usually the better and more conclusive a study is, the more it costs. For a study to be conclusive it needs to have a lot of participants and many safeguards against bias, and all those things cost money.

Because good studies are expensive, we can’t do them all the time. We just don’t have enough money. That’s why new ideas are often tested in less expensive studies. The very first studies are usually done in vitro (outside a living organism, such as in a petri dish). These studies establish whether something is possible even in the best case scenario.

Let’s say that you think that feruminin (a made up word) could reduce sebum production. What you would do first is grow some sebocytes (sebum producing cells) in a petri dish and expose them to feruminin to see if it affects how fast the cells grow and divide. If the results are positive, you’ve just established it’s theoretically possible for a feruminin cream to reduce sebum production. You have also gathered information about required dosage, so you have an idea of how much feruminin you need to get into the skin.

Next, you might produce a cream with a certain concentration of feruminin and test it out in animals for safety and first in vivo (in a living organism) studies. If these go well, you’ll move to human studies.

You might produce 2 creams, one with feruminin and other vehicle-only cream (lacks feruminin but otherwise identical) and randomly give the creams to study participants. If the results show that the group that used the feruminin cream got better results than the group using the placebo cream, then you start to be on to something. This is the kind of evidence we can start taking seriously.

Next, you might do a properly controlled study where your feruminin cream is compared against an existing treatment and an inactive placebo in a study with more participants (at least 30 to 40 participants per group). If this study shows that the feruminin cream outperforms both the placebo and the existing treatment, then we really start to be on to something.

The final step would be to have other, independent research groups to replicate the study and to see if they get the same results. After that we can really say that yes, your feruminin cream works.

The purpose of those preliminary studies is not really to establish whether feruminin works, but to see whether it qualifies for the next level of studies. It’s unfortunately common to see people hype these preliminary studies as proof and without clearly describing what you can and cannot conclude from the studies. I’m not saying we should never talk about such studies, only that we should be careful with the conclusions we draw from them.

Statistical significance vs. clinical significance

Sometimes studies produce results that are statistically significant but clinically meaningless. Let’s say that you invent another acne treatment; comuvin. You test a comuvin cream against a placebo cream. At the baseline (beginning of the study) the participants had on average of 25 pimples. After12 weeks, the comuvin cream group had 19 pimples on average and the placebo group had 20 pimples. Let’s also assume your study had enough participants to be able to say such a small difference is statistically significant (i.e. not likely due to random chance).

You could say that your comuvin cream is significantly more effective than placebo cream; significant not referring to difference in effect size but to statistical significance. But such a small difference in effectiveness is meaningless in real life. As a patient you probably couldn’t tell the difference.

You’ll see this kind of statistically significant but clinically meaningless effects often in alternative medicine studies, especially when ‘real’ acupuncture is compared to ‘sham’ acupuncture (such as toothpicks, needles that don’t penetrate the skin, or needles on non-acupuncture points). This small difference in effectiveness is far more likely due to unblinding or other bias than proof of effectiveness for acupuncture.

Conclusion

Science is not religion. Therefore it’s ok to question scientific findings, especially since many studies are biased and will later be shown wrong. The points Chris made in his article and the ones here should arm you with basic information and skeptical attitude to navigate the confusing jungle of health claims.

• Correlation doesn’t mean causation, organic food doesn’t cause autism.
• Scientists are just as imperfect and biased as the rest of us are. That’s why you should be careful with studies that don’t have proper blinding and controls in place. Such studies almost always produce overtly optimistic results.
• Be careful when you see someone making a big deal out of preliminary studies. They are either clueless or pushing their own agenda.
• Extraordinary claims require extraordinary evidence. For an implausible treatment, clinically meaningless improvement over placebo is not extraordinary evidence. It’s far more likely the result of unblinding and bias.

Finally, this blog is about acne, and you probably come here for advice on how to get over it. As such, I’m not sure if articles like this interest you. I’ll be happy to write more about how to read scientific research, critical thinking, and topics like that if you are interested. So please let me know in the comments below.