Most Research Is (Probably) Bogus

Science works through experimentation and replication. Hypotheses are put forward; tests are run; and results are obtained. If something surprising or interesting is found in an experiment, other researchers can then replicate the study to make sure that the effect is robust and not just due to chance.


Unfortunately, replication is lacking in many different sciences. This is partly because doing original research is better for one’s career. How many people have you heard of who made a name for themselves by merely replicating the interesting work of other people? It doesn’t happen.

However, while researchers might not have an incentive to replicate their own surprising results, or the results of others, people in the private sector do. That’s why, throughout the early 2000s, scientists at the drug company Amgen tried to replicate findings from 53 monumental studies in cancer biology. In the end, they were able to replicate only six of the studies: just 11.3 percent.

These dismal results might explain why we haven’t made as much progress as predicted in the war against cancer. But oncology is not the only area with serious replication issues. Just last week, the preliminary results of psychology’s first large reproducibility project were released. Out of 100 articles chosen, only 39 could be replicated. While 39 percent is much better than 11.3 percent, it’s still not great. It means that any surprising research you read about on the Internet is more likely to be false than true.

This is why, when you read about a psychology finding that is truly counterintuitive, you should be a bit critical. Unless it has been replicated or is part of a large body of similar research, you shouldn’t change your beliefs. Be on the lookout for similar research in the future, but don’t assume that a single study with a surprising result should be taken as gospel.

There are exceptions to this, of course. The larger the number of people that took place in the experiment, the more you can trust the results. The same holds true if there was a more representative sample. Unfortunately, most of the studies done today have small samples consisting of university students — a population that is definitely not representative of the broader population. Such studies may teach us a lot about university students, but they won’t necessarily point us to human universals. This is why spectacular findings must be probed with larger and more diverse samples, as we have seen with the popular field of mindset psychology. The original mindset studies seemed to imply that our beliefs relating to whether or not we can improve in a specific area determine how well we do, and how hard we work, in that area. Many of the original studies only contained a few dozen participants. But subsequent studies with sample sizes in the hundreds, or thousands, have since been performed, with impressive and positive results.

However, after more studies are replicated, we may find that many of the puzzling, counterintuitive results that have received a lot of press attention (and book pages) are, in fact, untrue. For example, researchers have recently had trouble replicating one of the classic studies on priming — in which study participants who were presented with words associated with old age walked more slowly right after the experiment. This seems unbelievable; perhaps for good reason. When things are settled, the world of psychological science may look a lot more like the world of common sense. It seems as if behavioral researchers have been so obsessed with rewriting our conceptions of ourselves that we’ve forgotten that generations of brilliant and perceptive minds have come before us. And in their maxims, common sense, and sayings, we have incisive psychological truths that we’re rediscovering in research today.

Plato, of 400 B.C., even saw our minds as a charioteer at the helm, controlling two powerful horses, good and bad. He predated Sigmund Freud and modern psychological scientists by over 2,000 years, yet understood the inhuman and unconscious drives that every person has to grapple with and attempt to master. He, in a very real sense, understood the multilayered nature of consciousness and the brain.

As psychological research moves forward, I think we will continue to find ourselves looking backward at the wise people of past eras for guidance in our quest onward.

Yug, age 7, and Alia, age 10, both entered Let Grow's "Independence Challenge" essay contest.

Photos: Courtesy of Let Grow
Sponsored by Charles Koch Foundation
  • The coronavirus pandemic may have a silver lining: It shows how insanely resourceful kids really are.
  • Let Grow, a non-profit promoting independence as a critical part of childhood, ran an "Independence Challenge" essay contest for kids. Here are a few of the amazing essays that came in.
  • Download Let Grow's free Independence Kit with ideas for kids.
Keep reading Show less

Divers discover world's largest underwater cave system filled with Mayan mysteries

Researchers in Mexico discover the longest underwater cave system in the world that's full of invaluable artifacts. 

Divers of the GAM project. Credit: Herbert Meyrl.
Technology & Innovation

Keep reading Show less

The surprise reason sleep-deprivation kills lies in the gut

New research establishes an unexpected connection.

Reactive oxygen species (ROS) accumulate in the gut of sleep-deprived fruit flies, one (left), seven (center) and ten (right) days without sleep.

Image source: Vaccaro et al, 2020/Harvard Medical School
Surprising Science
  • A study provides further confirmation that a prolonged lack of sleep can result in early mortality.
  • Surprisingly, the direct cause seems to be a buildup of Reactive Oxygen Species in the gut produced by sleeplessness.
  • When the buildup is neutralized, a normal lifespan is restored.

We don't have to tell you what it feels like when you don't get enough sleep. A night or two of that can be miserable; long-term sleeplessness is out-and-out debilitating. Though we know from personal experience that we need sleep — our cognitive, metabolic, cardiovascular, and immune functioning depend on it — a lack of it does more than just make you feel like you want to die. It can actually kill you, according to study of rats published in 1989. But why?

A new study answers that question, and in an unexpected way. It appears that the sleeplessness/death connection has nothing to do with the brain or nervous system as many have assumed — it happens in your gut. Equally amazing, the study's authors were able to reverse the ill effects with antioxidants.

The study, from researchers at Harvard Medical School (HMS), is published in the journal Cell.

An unexpected culprit

The new research examines the mechanisms at play in sleep-deprived fruit flies and in mice — long-term sleep-deprivation experiments with humans are considered ethically iffy.

What the scientists found is that death from sleep deprivation is always preceded by a buildup of Reactive Oxygen Species (ROS) in the gut. These are not, as their name implies, living organisms. ROS are reactive molecules that are part of the immune system's response to invading microbes, and recent research suggests they're paradoxically key players in normal cell signal transduction and cell cycling as well. However, having an excess of ROS leads to oxidative stress, which is linked to "macromolecular damage and is implicated in various disease states such as atherosclerosis, diabetes, cancer, neurodegeneration, and aging." To prevent this, cellular defenses typically maintain a balance between ROS production and removal.

"We took an unbiased approach and searched throughout the body for indicators of damage from sleep deprivation," says senior study author Dragana Rogulja, admitting, "We were surprised to find it was the gut that plays a key role in causing death." The accumulation occurred in both sleep-deprived fruit flies and mice.

"Even more surprising," Rogulja recalls, "we found that premature death could be prevented. Each morning, we would all gather around to look at the flies, with disbelief to be honest. What we saw is that every time we could neutralize ROS in the gut, we could rescue the flies." Fruit flies given any of 11 antioxidant compounds — including melatonin, lipoic acid and NAD — that neutralize ROS buildups remained active and lived a normal length of time in spite of sleep deprivation. (The researchers note that these antioxidants did not extend the lifespans of non-sleep deprived control subjects.)

fly with thought bubble that says "What? I'm awake!"

Image source: Tomasz Klejdysz/Shutterstock/Big Think

The experiments

The study's tests were managed by co-first authors Alexandra Vaccaro and Yosef Kaplan Dor, both research fellows at HMS.

You may wonder how you compel a fruit fly to sleep, or for that matter, how you keep one awake. The researchers ascertained that fruit flies doze off in response to being shaken, and thus were the control subjects induced to snooze in their individual, warmed tubes. Each subject occupied its own 29 °C (84F) tube.

For their sleepless cohort, fruit flies were genetically manipulated to express a heat-sensitive protein in specific neurons. These neurons are known to suppress sleep, and did so — the fruit flies' activity levels, or lack thereof, were tracked using infrared beams.

Starting at Day 10 of sleep deprivation, fruit flies began dying, with all of them dead by Day 20. Control flies lived up to 40 days.

The scientists sought out markers that would indicate cell damage in their sleepless subjects. They saw no difference in brain tissue and elsewhere between the well-rested and sleep-deprived fruit flies, with the exception of one fruit fly.

However, in the guts of sleep-deprived fruit flies was a massive accumulation of ROS, which peaked around Day 10. Says Vaccaro, "We found that sleep-deprived flies were dying at the same pace, every time, and when we looked at markers of cell damage and death, the one tissue that really stood out was the gut." She adds, "I remember when we did the first experiment, you could immediately tell under the microscope that there was a striking difference. That almost never happens in lab research."

The experiments were repeated with mice who were gently kept awake for five days. Again, ROS built up over time in their small and large intestines but nowhere else.

As noted above, the administering of antioxidants alleviated the effect of the ROS buildup. In addition, flies that were modified to overproduce gut antioxidant enzymes were found to be immune to the damaging effects of sleep deprivation.

The research leaves some important questions unanswered. Says Kaplan Dor, "We still don't know why sleep loss causes ROS accumulation in the gut, and why this is lethal." He hypothesizes, "Sleep deprivation could directly affect the gut, but the trigger may also originate in the brain. Similarly, death could be due to damage in the gut or because high levels of ROS have systemic effects, or some combination of these."

The HMS researchers are now investigating the chemical pathways by which sleep-deprivation triggers the ROS buildup, and the means by which the ROS wreak cell havoc.

"We need to understand the biology of how sleep deprivation damages the body so that we can find ways to prevent this harm," says Rogulja.

Referring to the value of this study to humans, she notes,"So many of us are chronically sleep deprived. Even if we know staying up late every night is bad, we still do it. We believe we've identified a central issue that, when eliminated, allows for survival without sleep, at least in fruit flies."

Withdrawal symptoms from antidepressants can last over a year, new study finds

We must rethink the "chemical imbalance" theory of mental health.

Bottles of antidepressant pills named (L-R) Wellbutrin, Paxil, Fluoxetine and Lexapro are shown March 23, 2004 photographed in Miami, Florida.

Photo Illustration by Joe Raedle/Getty Images
Surprising Science
  • A new review found that withdrawal symptoms from antidepressants and antipsychotics can last for over a year.
  • Side effects from SSRIs, SNRIs, and antipsychotics last longer than benzodiazepines like Valium or Prozac.
  • The global antidepressant market is expected to reach $28.6 billion this year.
Keep reading Show less
Scroll down to load more…