Can We Trust Peer Reviewed Studies?

In many of my online debates with people about nutrition, I’m often presented with peer reviewed scientific studies or books based on peer reviewed studies as evidence to support or refute a given stance. Most people often seem unconcerned with the actual results of the studies or the methodologies used, just the fact that they were peer reviewed, as if that’s all that’s needed to determine the validity of a study. However, after looking into the peer review process, I’m left wondering, can we trust peer reviewed studies or the books written based on them?

What Is Peer Review?

Before we dive into the questionable nature of peer review, let’s first discuss what the peer review process is. When a scientific article is written, or a study is conducted and the results documented in a scientific paper, the study must first be peer reviewed before it can be published in a legitimate scientific journal like The Journal of the American Medical Association (JAMA) or The British Medical Journal (BMJ).

Sometimes studies are published in less reputable, industry associated journals without being peer reviewed. These studies are made to look as if they have been peer reviewed and it’s hard to tell if the studies are legitimate without deep diving into them, which is why peer review is so important. Peer review is supposed to weed out biased or unscientific studies to keep the field of science pure.

Here’s how the peer review process works… After a scientific paper has been written, the manuscript is sent to a relevant journal for review. If the editors of the journal decide the manuscript meets the criteria for publication, it is sent to a variety of peer reviewers in the same field who are asked to judge the manuscript’s validity.

Sometimes the author and the reviewers know each other, or at least know each other’s names. Sometimes they don’t have access to that information.

It’s the reviewer’s job to make sure the methodology used was appropriate for the type of study being conducted. They also look for mistakes in the analysis, calculation errors and unsupported conclusions. And they’re supposed to check the citations to ensure they are relevant and accurately represented.

From a very high level, this all looks great. If unbiased peer reviewers spend the time and energy required to thoroughly analyze a manuscript, they would certainly find the methodological flaws, the faulty analyses, the misrepresented or irrelevant citations, and the blatant mistakes.

But we don’t live in a perfect world and even the revered field of science buckles under the pressures society puts on it.

How the Peer Review Process REALLY Works

Peer reviewers aren’t paid for their time. It’s a volunteer job; a mandatory responsibility of being in the scientific community. Without peer reviewers, nothing could be published.

And therein lies the problem. Relying on already overworked and underpaid scientists to perform such an important voluntary task doesn’t ensure it will be done to the best of their abilities.

A mantra in the scientific community is “publish or perish.” If researchers aren’t constantly publishing papers in journals, their jobs are at risk. Their career advancement is at risk. Their very livelihood is at risk.

These well-meaning scientists aren’t immune to the pressures of everyday life. They have a mortgage or rent to pay. They have car loans. Student loans. Credit card debt. Childcare costs. College funds for their kids. Etc. Etc. They’re under the same pressure as everyone else in consumer oriented, capitalist societies.

That means their focus HAS to be on publishing their own papers, rather than reviewing the papers other people are trying to publish. Peer review is a necessary but time consuming inconvenience for them.

In a peer reviewed paper published in the Journal of the Royal Society of Medicine titled, “Peer review: a flawed process at the heart of science and journals,” the author, Richard Smith who works as an editor at the BMJ, discusses just how weak this critical part of the scientific process is.

Smith starts this paper with a personal story of a friend who sent him a paper for review. As an expert in the field, Smith was qualified to review the paper, but his review and critique didn’t mean the paper had been “peer reviewed.” It didn’t go through the official process and it wasn’t reviewed by multiple peers. However, the paper was published as a peer reviewed study, anyway.

But it’s far worse than this one anecdotal example. According to Smith, “At the BMJ we did several studies where we inserted major errors into papers that we then sent to many reviewers. Nobody ever spotted all of the errors. Some reviewers did not spot any, and most reviewers spotted only about a quarter. Peer review sometimes picks up fraud by chance, but generally it is not a reliable method for detecting fraud because it works on trust.”

In other words, peer reviewers don’t spend the time needed to find all the errors, let alone determine if the study is fraudulent. They trust that their fellow scientists are honest and competent. And they probably assume someone else will put in the effort to find the errors; they have better things to do.

That means peer review isn’t the gold stamp everyone thinks it is. Rather, in most cases, it’s a rubber stamp given by time constrained editors and scientists who have bigger concerns, like getting their own papers published.

Scientific Journals are Equally Complicit

The “publish or perish” mantra doesn’t apply only to scientists. Scientific journals also must publish or perish, and they must publish papers that people want to read. Papers that will get picked up by the 5 o’clock news and broadcast across mainstream media.

The media loves to tell people good news about their bad habits. It’s good for ratings, which boosts profits and job security. So if a study gets published in a reputable journal like The Annals of Internal Medicine stating that saturated fat doesn’t cause heart disease, the media is all too eager to blast it out on every channel.

You’ll notice that the link I just shared about saturated fat is only for the abstract. You’ll need a membership to read the full study, and most reporters don’t have that access so they’re stuck with the abstract just like the rest of us lay people. Or worse, they rely on biased reviews of the study written by organizations with a vested interest in maintaining the status quo or by individuals motivated by the desire to boost their own ratings (clicks and views).

This study led to the Time Magazine cover telling us to “Eat Butter” and the New York Times opinion piece telling us “Butter Is Back.” These stories were great for magazine and newspaper sales, but horrible for human health.

Because, while the study was peer reviewed, it was fundamentally biased, methodologically flawed and completely misleading.

Here’s a list of the Potential Conflicts of Interest:

Dr. Franco: Grants: Nestlé and Metagenics. Dr. Butterworth: Grants: Pfizer, Merck Sharp & Dohme, and Novartis; Personal fees: Roche Pharmaceuticals. Dr. Thompson: Grants: Medical Research Council and British Heart Foundation. Dr. Khaw: Grants: Medical Research Council and Cancer Research UK. Dr. Mozaffarian: Personal fees: Bunge, Pollock Institute, Quaker Oats, Life Sciences Research Organization, Foodminds, Nutrition Impact, Amarin, AstraZeneca, Winston & Strawn, Unilever North American Scientific Advisory Board, and UpToDate online chapter. Dr. Danesh: Personal fees: Merck Sharp & Dohme UK Atherosclerosis Advisory Board, Novartis Cardiovascular & Metabolic Advisory Board, Pfizer Population Research Advisory Panel, and Sanofi Advisory Board; Grants: British Heart Foundation; British United Provident Association Foundation; diaDexus; European Research Council; European Union; Evelyn Trust; Fogarty International Centre; GlaxoSmithKline; Merck; National Heart, Lung, and Blood Institute; National Institute of Neurological Disorders and Stroke; National Health Service Blood and Transplant; Novartis; Pfizer; Medical Research Council; University of British Columbia; University of Sheffield; Wellcome Trust; and UK Biobank; Nonfinancial support: Merck Sharp & Dohme UK Atherosclerosis Advisory Board, Novartis Cardiovascular & Metabolic Advisory Board, Pfizer Population Research Advisory Panel, Sanofi Advisory Board, diaDexus, and Roche Pharmaceuticals. Dr. Di Angelantonio: Grant: British Heart Foundation, European Union, National Health Service Blood and Transplant, and Medical Research Council; Royalties: Elsevier (France).

All of these conflicts of interest benefit from people continuing to eat, or eating even MORE saturated fat. They either sell saturated fat products like dairy or they sell medications or medical procedures or medical products used to treat people who eat too much saturated fat.

This study is a meta analysis that relied on the findings of other studies to draw their conclusions. The peer review process didn’t uncover the selection bias that was obvious to other scientists in the field. The meta analysis only included studies that were favorable to their desired outcome, while omitting studies that show a strong correlation between saturated fat and heart disease.

Three scientists from Harvard University wrote a scathing review of this meta analysis. They pointed out several errors and omissions, as well as provided a list of relevant studies showing the strong link between saturated fat and heart disease that were not included in the meta analysis. They concluded their review stating, “the conclusions of Chowdhury et al. regarding the type of fat being unimportant are seriously misleading and should be disregarded.”

What the study actually found was that saturated fat is just as bad as refined carbs (processed foods), it’s slightly better than trans fats (hydrogenated vegetables oils) and slightly worse than monounsaturated and polyunsaturated fats found in olive oil. In other words, the study didn’t exonerate saturated fat; it placed its level of harm squarely in the middle of the hierarchy of unhealthy foods.

However, the damage had already been done by the time the Harvard scientists and other nutrition experts chimed in. The media already had a hayday with the story. Books like “The Big Fat Surprise” were already on their way to the publishers. And The Annals of Internal Medicine got their 15 minutes of journalistic fame.

Book Publishers are a BIG Part of the Problem

While scientific journals must publish or perish, book publishers must sell books or perish. That means publishers care very little about truth and honesty. If they think a book will sell, they publish it.

For years, I thought that the publishing process guaranteed some level of legitimacy. I thought the editors at these big publishing companies surely read the books looking for errors, false or misleading statements, or complete fraud. But they don’t.

Two books really stand out as borderline fraud: “The Plant Paradox” and “Wheat Belly.” Both of these books are riddled with citation errors, meaning the citations they provided either had nothing to do with the claims being made in the book, or the citation actually contradicted the claim being made.

The very first citation in “The Plant Paradox” used to support the claim that egg yolks “dramatically” reduce cholesterol is both unrelated and contradictory to the claim. It’s unrelated because the study didn’t look at the effects of egg yolks on cholesterol, and it’s contradictory because it did find that eggs in general (the yolks and whites) raise cholesterol.

I suggest reading the complete review written by Dr. T. Colin Campbell of all the false and misleading statements made in “The Plant Paradox.” This book is proof that publishers don’t review the books they publish for accuracy or legitimacy.

Likewise, Dr. Pamela Popper wrote a review of the book, “Wheat Belly,” which blames grain consumption for all of America’s health woes. The author relies almost entirely on anecdotal (non-peer reviewed) evidence from his medical practice, as well as studies on celiac disease, to suggest everyone should avoid wheat.

If you have celiac disease, wheat is no-doubt bad for you. But advice given to someone with a legitimate medical condition cannot be ubiquitously applied to the general public. If that were the case, we would tell everyone to stop eating peanuts because some people have peanut allergies.

These two books, as well as many others (like “The Big Fat Surprise”) were published, not because they were legitimate, but because the publishers knew they would sell.

Being published, either in a book or a peer reviewed scientific journal, does not provide the public with sufficient reason to believe the claims. Publishing is not a gold seal; it’s a rubber stamp. That means we have to do our own due diligence before trusting anything we hear or read.

How Do We Know What to Believe?

If you haven’t yet, I suggest reading my blog post, “Credible Nutrition Sources: Who Can You Trust?” This post goes into detail about how to choose reputable, trustworthy sources of nutrition advice. It also lists several of the experts that we trust, but we still verify the things they tell us.

When we hear a nutrition claim, we often go to our trusted sources first to hear what they have to say. But I also dig into the studies to look for three things.

First, I look for who funded the study. We like to think that bias doesn’t exist in science, but unfortunately it does. One meta analysis found that 100% of the industry funded studies they looked at had positive conclusions that benefited the funding source. If a study is funded by a corporation or corporate association that would benefit from a specific outcome, we have to seriously question the findings.

Second, I compare the abstract/conclusion to the actual study findings. Believe it or not, they don’t always match. It’s common to use “spin” in industry funded studies to misrepresent the actual findings in the abstract or conclusion because the researchers know that’s all most people will read. If you dig deeper into the study (assuming you have access to the whole study), you may spot discrepancies between what they said they found, and what they actually found. Peer review often doesn’t catch this type of fraud.

Third, I look at the methodology used in the study. Not all methodologies are applicable to all types of research. Some lack the statistical power to uncover anything meaningful.

Let’s say you want to measure a cup of water. Would you use a colander or would you use a measuring cup? Using the right tool for the job is important when measuring water, and when studying nutrition.

For example, cross-sectional observational studies are often used in industry funded studies because they yield no meaningful results, which appears to exonerate their products. However, for nutrition research, it’s better to use controlled feeding experiments or free-living dietary change experiments because these study methodologies minimize variability and allow scientists to compare apples to apples.

Cross-sectional observational studies are a common tactic used by the meat, dairy and egg industries to “prove” saturated fat and cholesterol are safe for human consumption. Because the study design acts like a colander, any meaningful results are lost allowing the researchers to state “no meaningful associations were discovered.”

This statement doesn’t mean saturated fat and cholesterol are healthy. Rather, it only means the study didn’t find any statistically significant or meaningful connections. And because they’re using the wrong tool for the job, we should expect nothing less.

How to Fix the Peer Review Process

It’s clear that relying on volunteers to conduct peer review isn’t working. They have too many other concerns and lack the time needed to conduct thorough reviews.

A common practice in situations like this is to require businesses to fund a consortium that is responsible for providing services used by all the businesses in the field. Sometimes we pay taxes that fund regulatory agencies. Sometimes we pay the USDA to fund Checkoff Programs. Sometimes we pay association dues to provide certifications or licenses.

To improve the peer review process, I think it’s reasonable for scientific journals, governments and corporations that rely on scientific studies to fund an independent, unbiased group of experts and support analysts who are responsible for reviewing and fact-checking manuscripts. They need to look for the same things I look for, as well as verify every single calculation and citation for accuracy and relevance.

This needs to be a double-blind process, too. The reviewers need to be hidden from the manuscript authors and the manuscript authors need to be hidden from the reviewers. If the authors don’t know who is reviewing their manuscript, and the reviewer doesn’t know who wrote it, there will be far less chance of interference or influence on the outcome of the peer review.

If a manuscript doesn’t pass a thorough, unbiased review process, it shouldn’t be published. And if an author submits a manuscript that’s loaded with errors, their revised manuscript should go to the bottom of the stack. That would transform the rubber stamp of peer review into a gold seal that we all can trust.

Conclusion

Peer review is the bare minimum standard we need to use when determining the validity of a scientific paper. If it’s not peer reviewed, we should ignore the findings completely.

However, just because a study has been peer reviewed doesn’t mean it’s legitimate. We simply can’t trust the peer review process to do what it’s supposed to do. There’s very little structure and no accountability, which means it’s easy to both misuse and abuse.

Until the peer review process is improved, we must take personal responsibility for gauging a study’s veracity. That means we need to read the studies ourselves, educate ourselves about proper methodologies, and look for errors and bias.

If you don’t have the time, access or ability to do that, rely on your trusted sources to help you decide what to believe because peer review as it currently stands simply isn’t good enough.