Can Science Be Trusted?

Can the scientific literature be trusted?

Can the scientific literature be trusted? In "Why Most Published Research Findings Are False," Dr. John P. A. Ioannidis, Professor of Medicine and Director of the Stanford Prevention Research Center at Stanford University School of Medicine, basically says no, it cannot.


Far from a kook or an outsider, Dr. Ioannidis is considered one of the world’s foremost experts on the credibility of medical research. His work has been published in top journals (where it is heavily cited) and his efforts were favorably reviewed in a 2010 Atlantic article called "Lies, Damned Lies, and Medical Science."

What kinds of analysis would allow Ioannidis to reach the conclusions he has reached? First know that a huge amount of work has been done in recent years to develop analytical methods for inferring publication bias by a variety of statistical methods. For example, there are now such accepted methodologies as Begg and Mazumdar’s rank correlation test, Egger’s regression, Orwin’s method, "Rosenthal's file drawer," and the now widely used "trim and fill" method of Duval and Tweedie. (Amazingly, at least four major software packages are available to aid detection of publication bias, for researchers doing meta-analyses. Read about it all here.)

There are many factors to consider when looking for publication bias. Take trial size. People who do meta-analysis of scientific literature have wanted, for some time, to have some reasonable way of compensating for the trial size of studies, because if you give small studies (which often have large variances in results) the same consideration as larger, more statistically significant studies, a handful of small studies with large effects sizes can unduly sway a meta-analysis. Aggravating this is the fact that studies showing a negative result are often rejected by journals or simply withheld from publication by their authors. When data goes unpublished, the literature that surfaces can give a distorted view of reality.

If you do a meta-analysis of a large enough number of studies and plot the effect size on the x-axis and standard error on the y-axis (giving rise to a "funnel graph"; see the graphic above, which is for studies involving Cognitive Behavioral Therapy), you expect to find a more-or-less symmetrical distribution of results around some average effect size, or failing that, at least a roughly equal number of data points on each side of the mean. For large studies, the standard error will tend to be small and data points will be high on the graph (because standard error, as usually plotted, goes from high values at the bottom of the y-axis to low numbers at the top; see illustration above). For small studies, the standard error tends (of course) to be large.

What meta-analysis experts have found is that quite often, the higher a study's "standard error" (which is to say, the smaller the study), the more likely the study in question is to report a strongly positive result. So instead of a funnel graph with roughly equal data points on each side (which is what you expect statistically), you get a graph that's visibly lopsided to the right, indicating that publication bias (from non-publication of "bad results") is likely. Otherwise how do you account for the points mysteriously missing from the left side of the graph, in a graph that should (by statistical odds) have roughly equal numbers of points on both sides?

Small studies aren't always the culprits. Some meta-analyses, in some research fields, show funnel-graph asymmetry at the top of the funnel as well as the bottom (in other words, across all study sizes). Data points are missing on the left side of the funnel. Which is hard to account for in a statistical distribution that should show points on both sides, in roughly equal amounts. The only realistic possibility is publication bias.

Then there's the problem of spin-doctoring in studies that are published. This takes various forms, from changing the chosen outcomes-measure after all the data are in (to make the data look better, via a different criterion-of-success; one of many criticisms of the $35 million STAR-D study of depression treatments), "cherry-picking" trials or data points (which should probably be called pea-picking in honor of Gregor Mendel, who pioneered the technique), or the more insidious phenomenon of HARKing, Hypothesizing After the Results are Known, which often occurs with selective citation of concordant studies.

So is Dr. Ioannidis right? Are most published research findings false? I don't think we have to go that far. I think it's reasonable to say that most papers are probably showing real data, obtained legitimately. But we also have to admit there is a substantial phantom literature of unpublished data out there. (This is particularly true in pharmaceutical research, where it's been shown that unflattering studies simply don't get published.) And far too many study authors practice HARKing, cherry-picking, and post hoc outcome-measure swapping.

All of which is to say, it's important to read scientific literature with a skeptical (or at least critical) eye. Fail to do that and you're bound to be led astray, sooner or later.

Yug, age 7, and Alia, age 10, both entered Let Grow's "Independence Challenge" essay contest.

Photos: Courtesy of Let Grow
Sponsored by Charles Koch Foundation
  • The coronavirus pandemic may have a silver lining: It shows how insanely resourceful kids really are.
  • Let Grow, a non-profit promoting independence as a critical part of childhood, ran an "Independence Challenge" essay contest for kids. Here are a few of the amazing essays that came in.
  • Download Let Grow's free Independence Kit with ideas for kids.
Keep reading Show less

10 Examples of Settled Science that Are 'Controversial'

Many Americans are being misled on serious scientific issues, and science journalists have to spend an inordinate amount of time debunking myths which seemingly never die.

popular

Many Americans are being misled on serious scientific issues, and science journalists have to spend an inordinate amount of time debunking myths which seemingly never die.

Keep reading Show less

Engineers 3D print soft, rubbery brain implants

Technique may enable speedy, on-demand design of softer, safer neural devices.

Dan Kitwood/Getty Images
Surprising Science

The brain is one of our most vulnerable organs, as soft as the softest tofu. Brain implants, on the other hand, are typically made from metal and other rigid materials that over time can cause inflammation and the buildup of scar tissue.

Keep reading Show less

The surprise reason sleep-deprivation kills lies in the gut

New research establishes an unexpected connection.

Reactive oxygen species (ROS) accumulate in the gut of sleep-deprived fruit flies, one (left), seven (center) and ten (right) days without sleep.

Image source: Vaccaro et al, 2020/Harvard Medical School
Surprising Science
  • A study provides further confirmation that a prolonged lack of sleep can result in early mortality.
  • Surprisingly, the direct cause seems to be a buildup of Reactive Oxygen Species in the gut produced by sleeplessness.
  • When the buildup is neutralized, a normal lifespan is restored.

We don't have to tell you what it feels like when you don't get enough sleep. A night or two of that can be miserable; long-term sleeplessness is out-and-out debilitating. Though we know from personal experience that we need sleep — our cognitive, metabolic, cardiovascular, and immune functioning depend on it — a lack of it does more than just make you feel like you want to die. It can actually kill you, according to study of rats published in 1989. But why?

A new study answers that question, and in an unexpected way. It appears that the sleeplessness/death connection has nothing to do with the brain or nervous system as many have assumed — it happens in your gut. Equally amazing, the study's authors were able to reverse the ill effects with antioxidants.

The study, from researchers at Harvard Medical School (HMS), is published in the journal Cell.

An unexpected culprit

The new research examines the mechanisms at play in sleep-deprived fruit flies and in mice — long-term sleep-deprivation experiments with humans are considered ethically iffy.

What the scientists found is that death from sleep deprivation is always preceded by a buildup of Reactive Oxygen Species (ROS) in the gut. These are not, as their name implies, living organisms. ROS are reactive molecules that are part of the immune system's response to invading microbes, and recent research suggests they're paradoxically key players in normal cell signal transduction and cell cycling as well. However, having an excess of ROS leads to oxidative stress, which is linked to "macromolecular damage and is implicated in various disease states such as atherosclerosis, diabetes, cancer, neurodegeneration, and aging." To prevent this, cellular defenses typically maintain a balance between ROS production and removal.

"We took an unbiased approach and searched throughout the body for indicators of damage from sleep deprivation," says senior study author Dragana Rogulja, admitting, "We were surprised to find it was the gut that plays a key role in causing death." The accumulation occurred in both sleep-deprived fruit flies and mice.

"Even more surprising," Rogulja recalls, "we found that premature death could be prevented. Each morning, we would all gather around to look at the flies, with disbelief to be honest. What we saw is that every time we could neutralize ROS in the gut, we could rescue the flies." Fruit flies given any of 11 antioxidant compounds — including melatonin, lipoic acid and NAD — that neutralize ROS buildups remained active and lived a normal length of time in spite of sleep deprivation. (The researchers note that these antioxidants did not extend the lifespans of non-sleep deprived control subjects.)

fly with thought bubble that says "What? I'm awake!"

Image source: Tomasz Klejdysz/Shutterstock/Big Think

The experiments

The study's tests were managed by co-first authors Alexandra Vaccaro and Yosef Kaplan Dor, both research fellows at HMS.

You may wonder how you compel a fruit fly to sleep, or for that matter, how you keep one awake. The researchers ascertained that fruit flies doze off in response to being shaken, and thus were the control subjects induced to snooze in their individual, warmed tubes. Each subject occupied its own 29 °C (84F) tube.

For their sleepless cohort, fruit flies were genetically manipulated to express a heat-sensitive protein in specific neurons. These neurons are known to suppress sleep, and did so — the fruit flies' activity levels, or lack thereof, were tracked using infrared beams.

Starting at Day 10 of sleep deprivation, fruit flies began dying, with all of them dead by Day 20. Control flies lived up to 40 days.

The scientists sought out markers that would indicate cell damage in their sleepless subjects. They saw no difference in brain tissue and elsewhere between the well-rested and sleep-deprived fruit flies, with the exception of one fruit fly.

However, in the guts of sleep-deprived fruit flies was a massive accumulation of ROS, which peaked around Day 10. Says Vaccaro, "We found that sleep-deprived flies were dying at the same pace, every time, and when we looked at markers of cell damage and death, the one tissue that really stood out was the gut." She adds, "I remember when we did the first experiment, you could immediately tell under the microscope that there was a striking difference. That almost never happens in lab research."

The experiments were repeated with mice who were gently kept awake for five days. Again, ROS built up over time in their small and large intestines but nowhere else.

As noted above, the administering of antioxidants alleviated the effect of the ROS buildup. In addition, flies that were modified to overproduce gut antioxidant enzymes were found to be immune to the damaging effects of sleep deprivation.

The research leaves some important questions unanswered. Says Kaplan Dor, "We still don't know why sleep loss causes ROS accumulation in the gut, and why this is lethal." He hypothesizes, "Sleep deprivation could directly affect the gut, but the trigger may also originate in the brain. Similarly, death could be due to damage in the gut or because high levels of ROS have systemic effects, or some combination of these."

The HMS researchers are now investigating the chemical pathways by which sleep-deprivation triggers the ROS buildup, and the means by which the ROS wreak cell havoc.

"We need to understand the biology of how sleep deprivation damages the body so that we can find ways to prevent this harm," says Rogulja.

Referring to the value of this study to humans, she notes,"So many of us are chronically sleep deprived. Even if we know staying up late every night is bad, we still do it. We believe we've identified a central issue that, when eliminated, allows for survival without sleep, at least in fruit flies."

Scroll down to load more…