Last week I wrote a post about how to read and understand scientific research. At the end of the post I asked if you guys would like to read more about critical thinking and making sense of scientific studies. A few people were foolish enough to say yes, so there’s my excuse to write more.

And as it happens, the Skeptic’s Guide to the Universe podcast gave me material for a follow-up post. Instead of talking theory, this time we are going to critique a bad study that the alt-med media happily abuses to spread fear about GMOs.

I should say that I don’t want to get into the discussion about GMOs itself. Personally I’m not concerned about them, but I also haven’t looked at the evidence in detail so I hold back from making strong conclusions. So while this anti-GMO study is a very good example of very bad science, it doesn’t mean that GMOs would be safe –or unsafe.

Little background. The basic idea of the study was to look at the effect of GMO feed on pig health. The study had two groups of pigs grown in a commercial piggery; one group got GMO feed and the other ate non-GMO feed. After slaughter the researchers assed the health of the pigs by looking at several factors (such as organ weights).

There are countless problems with this study, but I’ll focus on a few of the more important points. For a more detailed analysis, I recommend Dr. David Gorski’s post over at the Science-Based Medicine blog. Dr. Steve Novella also highlighted many of the problems at the Skeptic’s Guide to The Universe podcast. I recommend you listen to the section of the podcast. You can download the episode from this page. The relevant discussion starts around the 22:40 mark. And if you are at all interested in science or critical thinking, I highly recommend listening to the podcast. It’s my favorite podcast and one of the best science podcasts.

Science gone fishing

This study is a perfect example of fishing expedition. Fishing expedition is a study where the researchers just search for information without really knowing what they are after. There’s a legitimate use for such studies, and that is to give ideas and guide further research. This GMO study wasn’t one of the legit uses, or at least how the findings have been hyped.

In a fishing expedition, or exploratory, study you make many comparisons between two groups, in this case between GMO and non-GMO fed pigs, and look for differences. This kind of a study can serve as a good ‘first look’ and to highlight possible areas for future research.

The main problem with fishing expedition studies, and also the reason why they are so often abused, is that you are bound to find some differences when you make multiple comparisons. It’s almost guaranteed to happen due to the fact that there’s always randomness in natural events.

There’s a way to statistically account for multiple comparisons, in a way to check if the result was real or just because you made multiple comparison. The authors didn’t seem to account for multiple comparisons.

It looks very much like the authors wanted to prove that GMOs are bad. So they made many different comparisons, for most of which there was no difference between the two groups, and then pointed to the few differences they happened to find as evidence that GMOs are bad. But what is truly bad is their science.

The main finding of the study was that the GMO fed pigs had more stomach inflammation. But even that result required questionable decisions. Stomach inflammation was divided into 4 different categories: no inflammation, mild, moderate and severe inflammation. That of course increases the number of comparisons – and your chances of finding differences.

If you look at the total number of pigs with stomach inflammation there’s no difference between the two groups. Almost all the pigs showed some level of stomach inflammation. Non-GMO fed pigs had more mild and moderate inflammation. It’s only the severe inflammation category where non-GMO feed appears better, and therefore that’s what gets reported.

It’s possible to argue that GMO feed aggravated stomach inflammation, but that’s a rather weak argument given overall there was no differences between the two groups.

Mark Lynas also made a few valid points about the study. Isn’t it interesting that the non-GMO fed pigs had far more liver and heart problems than the GMO fed pigs? 15% of the non GMO fed pigs had heart problems vs. only 6% of the GMO fed pigs. Why didn’t they report that GMO feed protects against heart disease? Inquiring minds want to know! Perhaps the findings were not statistically significant, but the point is that there were many differences between the two groups, some of which favored GMO fed pigs and some of which favored non-GMO fed pigs.

Another point raised against this study was that all the pigs seemed to be in a really bad condition. The authors say that the pigs were raised according to the commercial piggery standards in the US, and were thus no different from the other pigs in slaughter houses. Critics have pointed out that the rates in the paper were quite a bit higher than what’s normal in piggeries.

Regardless of the cause more sick pigs mean more potential for differences between the two groups. And of course more ‘evidence’ that GMOs are bad.

I pretty sure that if another research group were to replicate this study, they wouldn’t come up with the same results. Sure they would find some differences, but those would be different differences from this study. If a result doesn’t replicate, then it’s likely not for real but due to chance.

But it gets even worse

The way they measured stomach inflammation isn’t even valid. They measured inflammation by looking at the color of the stomach after slaughter. Apparently stomach color of a slaughtered and bled-out pig has no relationship with actual stomach inflammation! Dr Robert Friendship, University of Guelph and a swine health management specialist explains over at Terry Daynard’s blog.

Dr Robert Friendship, a professor in the Department of Population Medicine at the Ontario Veterinary College, University of Guelph and a swine health management specialist, reviewed the paper. He concluded that “it was incorrect for the researchers to conclude that one group had more stomach inflammation than the other group because the researchers did not examine stomach inflammation. They did a visual scoring of the colour of the lining of the stomach of pigs at the abattoir and misinterpreted redness to indicate evidence of inflammation. It does not. They would have had to take a tissue sample and prepare histological slides and examine these samples for evidence of inflammatory response such as white blood cell infiltration and other changes to determine if there was inflammation. There is no relationship between the colour of the stomach in the dead, bled-out pig at a slaughter plant and inflammation. The researchers should have included a veterinary pathologist on their team and this mistake would not have happened. They found no difference between the two experimental groups in pathology that can be determined by gross inspection.”

So even the smoking gun evidence achieved with bad methodology was wrong.

There are many other problems with this study, and I’ll give you some links if you want to get into the details. But the point of this post was not to complain about the study.

The point I wanted to make is to watch out for fishing expedition studies. They are fantastic for fear mongering and promoting nonsense; because if a motivated researcher looks hard enough, he will always find something. But just because he did, doesn’t mean those differences mean anything.

If you want to learn more about the study, I recommend checking out these links:

• More bad science in the service of anti-GMO activism – Science-Based Medicine
• Pigs, GMOs & Bullshit – Random Rationality
• Pollan and Bittman, the Morano and Milloy of GMO anti-science – Denialism over at Scienceblogs.com
• GMO pigs study – more junk science – Mark Lynas
• The evidence of GMO harm in pig study is pretty flimsy – Weed Control Freaks
• Ordinal logistic GM pigs – Quantum Forest
• Critics answered – The lead author of the study answers some of the criticism leveled at the study