Trust the Results, Not the Conclusions

Trust the Results, Not the Conclusions

Very early in my writing career I was fortunate to be able to spend three hours interviewing Linus Pauling (above), the only person in history to win two unshared Nobel Prizes. One of many things I learned during that interview process that has stayed with me ever since has to do with interpretation of scientific results.

Pauling was roundly criticized, in his later years, for his controversial stance on vitamin C. (He came to believe that vitamin C not only could prevent and/or cure the common cold but could protect people from more serious diseases as well.) I interviewed Pauling at length on the subject. He told me that when he first began researching the role of vitamin C in protecting against the common cold, he was struck by how many papers he encountered in the scientific literature that showed a divergence between the results obtained and the conclusions drawn. Authors, he found, tended to fashion conclusions around their own belief biases as much as around the actual data they reported. Pauling regularly encountered studies in which the data clearly showed a positive effect for vitamin C, but because the effect was judged insignificant by the studies' authors, it wasn't discussed in the conclusion (nor the abstract, usually) of the paper(s) in question.

What's so insidious about this is that other researchers who cite a given study will tend to quote from the conclusions section of a study, or the abstract (seldom the results), thereby propagating the authors' original biases.

I've seen this myself, over and over again. Propagation of incorrect conclusions has played an important role in, for example, the misplaced belief in the greater readability of serif fonts over sans-serif fonts. (For details, see "The Serif Readability Myth.") Another example that comes readily to mind is the 2001 National Cancer Institute report that is so often cited to show that low-tar cigarets are just as harmful as regular cigarets: "Risks Associated With Smoking Cigarettes With Low Machine-Measured Yields of Tar and Nicotine" (National Cancer Institute; 2001. Smoking and Tobacco Control Monograph 13). Chapter 4 of this monograph presents a great deal of convincing evidence that today's filtered low-tar cigarets are much less toxic than the unfiltered high-tar cigarets of 60 years ago. And yet the monograph is often cited as showing the opposite.

And then there are drug studies like the one by Berman et al. in J. Clin. Psychiatry 2007 68:843-853 that confuse statistical significance with clinical significance, reaching a positive conclusion on the basis of the most miniscule effect simply because the effect was statistically valid. In the Berman study, which is often cited as proving the effectiveness of Abilify when used as an adjunctive treatment in depression, patients who failed to show improvement on standard antidepressants (SSRIs, SNRIs) were given Abilify (or placebo) augmentation for 6 weeks. The authors defined failure to show improvement on antidepressants as failure to experience a 50% reduction in score on the Hamilton rating scale (a common depression assessment test) after taking antidepressants. But when patients were given adjunctive Abilify for six weeks, they still failed to show 50% improvement on their test scores. (The treatment group's average improvement was 34%, which sounds impressive until you realize that the placebo group improved 22%.) The net gain provided by Abilify was 3.0 points on the Montgomery scale, which, in the UK, is the absolute lower limit on what's considered clinically significant for drug-approval purposes. Also, this 3-point gain was an average, based mostly on improvement in female subjects. Male subjects hardly saw any benefit. All in all, the paper showed Abilify to be remarkably inert, little more than a super-placebo where depression is concerned. Yet somehow the authors of this drug-company-funded study saw fit to conclude that "adjunctive aripiprazole was efficacious and well tolerated," and Abilify is now a top-selling drug in the U.S.

The bottom line, in any case, is simple. Read studies carefully to see what the results actually are or were; then analyze the results yourself. Don't just read a paper's Abstract and the Conclusion. You may very well be misled.

‘Designer baby’ book trilogy explores the moral dilemmas humans may soon create

How would the ability to genetically customize children change society? Sci-fi author Eugene Clark explores the future on our horizon in Volume I of the "Genetic Pressure" series.

Surprising Science
  • A new sci-fi book series called "Genetic Pressure" explores the scientific and moral implications of a world with a burgeoning designer baby industry.
  • It's currently illegal to implant genetically edited human embryos in most nations, but designer babies may someday become widespread.
  • While gene-editing technology could help humans eliminate genetic diseases, some in the scientific community fear it may also usher in a new era of eugenics.
Keep reading Show less

Octopus-like creatures inhabit Jupiter’s moon, claims space scientist

A leading British space scientist thinks there is life under the ice sheets of Europa.

Jupiter's moon Europa has a huge ocean beneath its sheets of ice.

Credit: NASA/JPL-Caltech/SETI Institute
Surprising Science
  • A British scientist named Professor Monica Grady recently came out in support of extraterrestrial life on Europa.
  • Europa, the sixth largest moon in the solar system, may have favorable conditions for life under its miles of ice.
  • The moon is one of Jupiter's 79.
Keep reading Show less

Lair of giant predator worms from 20 million years ago found

Scientists discover burrows of giant predator worms that lived on the seafloor 20 million years ago.

Bobbit worm (Eunice aphroditois).

Credit: Jenny – Flickr
Surprising Science
  • Scientists in Taiwan find the lair of giant predator worms that inhabited the seafloor 20 million years ago.
  • The worm is possibly related to the modern bobbit worm (Eunice aphroditois).
  • The creatures can reach several meters in length and famously ambush their pray.
Keep reading Show less

FOSTA-SESTA: Have controversial sex trafficking acts done more harm than good?

The idea behind the law was simple: make it more difficult for online sex traffickers to find victims.

Has FOSTA-SESTA really lived up to it's promise of protecting sex trafficking victims - or has it made them easier to target?

Credit: troyanphoto on Adobe Stock
Politics & Current Affairs
  • SESTA (Stop Enabling Sex Traffickers Act) and FOSTA (Allow States and Victims to Fight Online Sex Trafficking Act) started as two separate bills that were both created with a singular goal: curb online sex trafficking. They were signed into law by former President Trump in 2018.
  • The implementation of this law in America has left an international impact, as websites attempt to protect themselves from liability by closing down the sections of their sites that sex workers use to arrange safe meetings with clientele.
  • While supporters of this bill have framed FOSTA-SESTA as a vital tool that could prevent sex trafficking and allow sex trafficking survivors to sue those websites for facilitating their victimization, many other people are strictly against the bill and hope it will be reversed.
Keep reading Show less

What is the ‘self’? The 3 layers of your identity.

Answering the question of who you are is not an easy task. Let's unpack what culture, philosophy, and neuroscience have to say.

Scroll down to load more…