The Importance of Peer Review and Replication in Science: A Short Overview
Welcome to my first actual blog post! I was originally going to write about an exciting new study I was a part of, but it hasn’t been released in print yet, and I want to be able to share the source when I post my reflections on the project. I’ll be sure to follow up as soon as I can!
In the meantime, I couldn’t help but notice the seemingly endless amount of misinformation spreading in the midst of the COVID-19 pandemic. That gave me ideas for how to target my blog and its first real post!
In looking at some of the media that’s been circulating the social media sphere during the past few weeks and months, I realized that the way scientific studies are produced and published is a mystery to most; after all, fewer than 20% of Americans know a scientist or can even name one (1, 2)! I decided that my inaugural blog post would explain some of the scientific process.
For this post, I will focus on the basics: how research is produced and published, how it’s checked for quality, and what happens if results are faked or deeply flawed. I’ve been in biological research for over a decade and have intimate knowledge of the publication process. A common area of misunderstanding that is abundant in conspiracy theories is the idea that big pharma suppresses research and results that threaten their bottom line. The pharmaceutical industry has immense lobbying power, but has much less pull on publications and drug development than many believe. The journals where research is published are also independent of government and commercial enterprises.
The Scientific Method
We’ve all learned the scientific method at some point, and no doubt you’ve used it in your day-to-day life; it simply comes down to testing an idea using an experiment.
The Scientific Method. Figure created by Christopher Pratt
The first step is to establish a research question and learn as much as you can about the topic. Scientists do this by reading many published journal articles, but in everyday life, one might look to books, articles on the internet, expert opinions, and even the local library. When we think we know enough about our topic, we create a hypothesis, which is an idea that can be tested, or an educated guess at the outcome (some everyday hypothesis examples: “I can get to work faster if I take Western Avenue instead of the highway” or “I can grow bigger tomatoes if I use this fertilizer”). Then we design an experiment to test the hypothesis, analyze the data, form conclusions, and communicate the results.
Peer Review
Scientists communicate results through peer-reviewed publications. Peer review is a rigorous process of quality control. This diagram shows a simplified view of how it works.
The Peer Review Process. Created by Christopher Pratt
Authors submit their manuscript to a journal, and that submission is then assigned to an editor. The editor is the first gatekeeper to publication; they have a scientific background (usually a PhD) and it is their job to understand the field and decide if the submitted manuscript is relevant, novel, and impactful. If the editor does not believe these standards have been met, then they can reject the publication outright, usually giving the authors some feedback as to why. The novel (newness) aspect is especially important for top journals like Science and Nature. These journals want to print groundbreaking results and those that shake up the field a bit.
If the manuscript clears the first hurdle, the editor selects reviewers (usually 3) based on expertise, availability, and lack of conflicts of interest (you don’t want a reviewer that is directly competing with the authors; that could bias their response). Each reviewer reads the manuscript and offers critiques. Reviewers can ask questions and for more experiments from the authors, as well as offer their recommendations as to whether or not the manuscript should be published. The editor compiles all the reviewer feedback and decides whether to publish the manuscript as a paper or to reject it. With rejection, the editor can request revisions and invite the authors to resubmit an improved manuscript or recommend they submit to a different journal. This peer review process can go through several rounds before a manuscript is finally accepted for publication.
Peer review isn’t perfect, but it’s an important quality control measure. Without peer review, flawed or erroneous studies would be everywhere! It’s important to realize that the peer reviewers are basing their reviews on the text of the manuscript.
What happens after a study is published?
After a study is published, it becomes a part of the body of literature for future experiments. Quality control doesn’t end with publication, though! A study alone cannot be interpreted as fact. Studies need to be put into the proper context and they must be replicated. Other research groups will attempt to reproduce the results of published studies or corroborate them with further studies, and this challenge is crucial to ensure ongoing quality. The need for multiple studies is clearly illustrated by the below chart, from a study which analyzed over 200 studies on foods and their relationship with cancer (3).
You can see how there’s no clear consensus on coffee’s role in cancer, but the results are pretty stacked in the “pro-cancer” deck for bacon (note: I love bacon, but it doesn’t change the results). You could imagine the problem if the first study to come out on tomatoes and cancer concluded that tomatoes cause cancer and this was accepted as the truth. We now know there is a wide spread of results, with more studies indicating the opposite to be true. This isn’t to say we can’t get excited about a new study (we definitely do!), but we always have to be reserved and wait for replication before we change practice or perception. Differences in experimental conditions and study populations can lead to differences in the results. Even the interpretation of results can differ from one study to another. Having more studies helps us build consensus and corroborate results.
Replication is not only meant to prevent human error, but also to find and debunk data that is fabricated or faked. Repeating experiments not only builds confidence in the results, but it can also weed out fake data that made it to publication. If a study can’t be replicated as written, then that raises serious concerns! Imagine a study that concluded apples will continue traveling upward if thrown. Other people start testing this theory by throwing apples upward, exactly as written in the published study. When everybody else’s apples fall back to Earth, it becomes evident that something was amiss and probably faked.
Journals can retract published papers that do not hold up to scientific and experimental scrutiny. Retractions are reserved for cases when data are likely tainted, falsified, or incorrect. Sometimes mistakes happen, like using contaminated samples in your experiments, or having mislabeled data. The dark side, however, is that the pressure to publish can be enormous, and sadly, data fabrication happens.
Here’s an example: a huge area of biomedical research is to make stem cells from adult cells, so that biopsies from adults can be used instead of needing embryos. Making stem cells this way is hard! It requires complex genetic engineering and culturing conditions. In 2014, there were two papers published in Nature where the authors claimed to create stem cells from adult cells by using a simple acid bath treatment; much easier than genetic engineering! The studies were retracted after they couldn’t be reproduced, images were found to have been tampered with, and the stem cells they claimed to create probably didn’t actually exist. I also highly recommend checking out Elisabeth Bik on Twitter to learn more about how science sleuths uncover fabrication.
Last Words
In this post, I wrote about peer-reviewed studies, as these make up the bulk of the scientific literature. A big drawback to this system is that it can take a long time for scientific data to be published this way. Some manuscripts can take over a year from first submission to printing in top-tier journals.
Recently, pre-print sites like BioRxiv and MedRxiv have come about, where authors can submit their manuscripts while going through peer review at a journal. These sites allow faster dissemination of data but are not peer-reviewed! Especially now during the COVID-19 pandemic, pre-prints range in quality from credible to preposterous, and despite BioRxiv’s notice (below), news media are reporting these as fact. so be wary if you see news articles based on pre-prints. The same goes for science reported in a book instead of a peer-reviewed journal.
I hope this post has been helpful in understanding how the scientific process works! I hope that you have a bit more understanding in how research information is produced and shared, and how quality control keeps findings reputable.
Leave me your thoughts, questions, and feedback, and I’ll be happy to answer! I want to make my writing as easily understandable as possible, and I rely on you to help me improve it, so thank you!
References and Further Reading
1. Research America, 81 percent of Americans can’t name a single living scientist. Research!America (2018), (available at https://www.researchamerica.org/news-events/81-percent-americans-can%E2%80%99t-name-single-living-scientist).
2. A. Briseno, American’s Attitudes about Science and Scientists in 2017, 12.
3. J. D. Schoenfeld, J. P. Ioannidis, Is everything we eat associated with cancer? A systematic cookbook review. Am. J. Clin. Nutr. 97, 127–134 (2013).