Can we trust studies when humans have a vested interest in the outcome?

Lack of replication is a serious problem in science. So far, no one has an answer.

Though rumor had it that citrus cured sailors of scurvy—crews were aware of this as early as 1497—it was not until James Lancaster’s 1601 voyage to Sumatra that this bit of folk wisdom was put to the test. With four ships under his command, Lancaster decided to try a little experiment: men on one ship received regular lemon juice shots; the other three were dry. 

We know the results. Men on those three ships developed scurvy. More importantly, Lancaster inadvertently created the first scientific trial. Nearly a century-and-a-half later, Scottish physician James Lind is credited with the first clinical trial (also studying scurvy); a few decades onward, in 1784 Benjamin Franklin and Antoine Lavoisier performed the first blind experiment in France to test out Franz Mesmer’s theory of animal magnetism. (It doesn’t exist.)

The double-blind method was introduced so researchers would not influence volunteers by inadvertently (or knowingly) guiding them to make certain claims. For nearly two centuries, this type of clinical study has been the gold standard of scientific research. Whenever you hear the statement, “studies show…,” if not conducted in this manner, it’s not considered valid.

Yet in the last few decades, a number of problems have arisen. As much as we’d like to believe that data are data and humans are merely reporting it, researcher bias has led to numerous cases of selective reporting—only publishing studies that confirm the initial hypothesis. This is especially problematic with pharmaceutical studies, which are often funded by corporations with a vested interest in getting the result they’d like to advertise.

Statistician Theodore Sterling noticed a troubling trend as far back as 1959: 97 percent of psychological studies confirmed the effect they initially set out to prove. This is an unbelievable number because, well, it shouldn’t be believed. The bar for a positive trial was created in 1922 by mathematician Ronald Fisher, who said that a “significant” result is produced by chance less than 5 percent of the time. (In statistics, a 95 percent confidence interval is standard, though a 99 percent confidence interval is more accurate.) Incredibly, Fisher invented this stat not out of confidence, but convenience

[Fisher] picked five per cent as the boundary line, somewhat arbitrarily, because it made pencil and slide-rule calculations easier.

Even more harrowing is the lack of replication. Besides hiding data, as many pharmaceutical companies have been shown to do, no study should be considered foolproof until it’s been repeated by unaffiliated parties. By the time research has been released to the public, however, we tend to take it on its face value. When conflicting evidence is presented, we often reject claims that contradict what we already believe.

One such case is sexual selection and symmetry, the notion that animals (including us) choose mates based on symmetrical features. This notion was implanted in the social consciousness in 1991 after a study on barn swallows speculated that females chose mates males with long, symmetrical feathers, a notion that was quickly extrapolated to include humans with symmetrical facial features. By 1997, this study had been replicated dozens of times, resulting in an overall 80 percent reduction in effect size. Still, the myth persists.

Barn swallows and one night stands are one thing, but a similar phenomenon has occurred with SSRIs. Since the introduction of fluoxetine in 1987, this class of antidepressants has been the go-to for treating depression and anxiety. Yet in the intervening decades, their efficacy has waned even though prescription rates continue to skyrocket. Drugs that were initially intended to be used for a limited period of time are being prescribed for decades. The side effects are killing the people these drugs were designed, or at least marketed, to save. 

Another realm susceptible to dubious claims is acupuncture. As Jonah Lehrer points out,  between the years of 1966 and 1995, 47 out of 47 studies in China, Taiwan, and Japan concluded it to be an effective treatment. All of these nations have long believed in acupuncture’s efficacy. Americans are a bit more skeptical. The 94 trials during that time resulted in a 56 percent efficacy rate. Lehrer continues,

This wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.

One fix to this blindness would be an open-source database in which intentions and investigations exist side by side. Instead of companies and research institutions with a stake in the results holding back what they don’t want to report or “curve-fitting” what they do, transparency would be built into the system from the outset. This might not alter the decline effect Lehrer mentions, but it would hold researchers accountable for their intentions and results during every step of the process.

Of course, there’s still human belief to contend with. That includes the pre-existing beliefs of researchers, the belief held by CEOs and board members that their data are proprietary, and the innumerable beliefs plastered across the internet on health blogs with no (or questionable) studies being championed as the final word on the subject.

The scientific method is one of the great inventions of both our rational and imaginative mind. Being honest with the evidence is another story.


Stay in touch with Derek on Facebook and Twitter.

Antimicrobial resistance is a growing threat to good health and well-being

Antimicrobial resistance is growing worldwide, rendering many "work horse" medicines ineffective. Without intervention, drug-resistant pathogens could lead to millions of deaths by 2050. Thankfully, companies like Pfizer are taking action.

Image courtesy of Pfizer.
  • Antimicrobial-resistant pathogens are one of the largest threats to global health today.
  • As we get older, our immune systems age, increasing our risk of life threatening infections. Without reliable antibiotics, life expectancy could decline for the first time in modern history.
  • If antibiotics become ineffective, common infections could result in hospitalization or even death. Life-saving interventions like cancer treatments and organ transplantation would become more difficult, more often resulting in death. Routine procedures would become hard to perform.
  • Without intervention, resistant pathogens could result in 10 million annual deaths by 2050.
  • By taking a multi-faceted approach—inclusive of adherence to good stewardship, surveillance and responsible manufacturing practices, as well as an emphasis on prevention and treatment—companies like Pfizer are fighting to help curb the spread.
Keep reading Show less

How to bring more confidence to your conversations

Entrepreneur and author Andrew Horn shares his rules for becoming an assured conversationalist.
  • To avoid basing action on external validation, you need to find your "authentic voice" and use it.
  • Finding your voice requires asking the right questions of yourself.
  • There are 3-5 questions that you would generally want to ask people you are talking to.
Keep reading Show less

Bespoke suicide pods now available for death in style

Sarco assisted suicide pods come in three different styles, and allow you to die quickly and painlessly. They're even quite beautiful to look at.

The Sarco assisted suicide pod
Technology & Innovation

Death: it happens to everyone (except, apparently, Keanu Reeves). But while the impoverished and lower-class people of the world die in the same ol' ways—cancer, heart disease, and so forth—the upper classes can choose hip and cool new ways to die. Now, there's an assisted-suicide pod so chic and so stylin' that peeps (young people still say peeps, right?) are calling it the "Tesla" of death... it's called... the Sarco! 

Keep reading Show less

Scientists find a horrible new way cocaine can damage your brain

Swiss researchers identify new dangers of modern cocaine.

Getty Images
Mind & Brain
  • Cocaine cut with anti-worming adulterant levamisole may cause brain damage.
  • Levamisole can thin out the prefrontal cortex and affect cognitive skills.
  • Government health programs should encourage testing of cocaine for purity.
Keep reading Show less