Can we trust studies when humans have a vested interest in the outcome?

Lack of replication is a serious problem in science. So far, no one has an answer.

Though rumor had it that citrus cured sailors of scurvy—crews were aware of this as early as 1497—it was not until James Lancaster’s 1601 voyage to Sumatra that this bit of folk wisdom was put to the test. With four ships under his command, Lancaster decided to try a little experiment: men on one ship received regular lemon juice shots; the other three were dry. 

We know the results. Men on those three ships developed scurvy. More importantly, Lancaster inadvertently created the first scientific trial. Nearly a century-and-a-half later, Scottish physician James Lind is credited with the first clinical trial (also studying scurvy); a few decades onward, in 1784 Benjamin Franklin and Antoine Lavoisier performed the first blind experiment in France to test out Franz Mesmer’s theory of animal magnetism. (It doesn’t exist.)

The double-blind method was introduced so researchers would not influence volunteers by inadvertently (or knowingly) guiding them to make certain claims. For nearly two centuries, this type of clinical study has been the gold standard of scientific research. Whenever you hear the statement, “studies show…,” if not conducted in this manner, it’s not considered valid.

Yet in the last few decades, a number of problems have arisen. As much as we’d like to believe that data are data and humans are merely reporting it, researcher bias has led to numerous cases of selective reporting—only publishing studies that confirm the initial hypothesis. This is especially problematic with pharmaceutical studies, which are often funded by corporations with a vested interest in getting the result they’d like to advertise.

Statistician Theodore Sterling noticed a troubling trend as far back as 1959: 97 percent of psychological studies confirmed the effect they initially set out to prove. This is an unbelievable number because, well, it shouldn’t be believed. The bar for a positive trial was created in 1922 by mathematician Ronald Fisher, who said that a “significant” result is produced by chance less than 5 percent of the time. (In statistics, a 95 percent confidence interval is standard, though a 99 percent confidence interval is more accurate.) Incredibly, Fisher invented this stat not out of confidence, but convenience

[Fisher] picked five per cent as the boundary line, somewhat arbitrarily, because it made pencil and slide-rule calculations easier.

Even more harrowing is the lack of replication. Besides hiding data, as many pharmaceutical companies have been shown to do, no study should be considered foolproof until it’s been repeated by unaffiliated parties. By the time research has been released to the public, however, we tend to take it on its face value. When conflicting evidence is presented, we often reject claims that contradict what we already believe.

One such case is sexual selection and symmetry, the notion that animals (including us) choose mates based on symmetrical features. This notion was implanted in the social consciousness in 1991 after a study on barn swallows speculated that females chose mates males with long, symmetrical feathers, a notion that was quickly extrapolated to include humans with symmetrical facial features. By 1997, this study had been replicated dozens of times, resulting in an overall 80 percent reduction in effect size. Still, the myth persists.

Barn swallows and one night stands are one thing, but a similar phenomenon has occurred with SSRIs. Since the introduction of fluoxetine in 1987, this class of antidepressants has been the go-to for treating depression and anxiety. Yet in the intervening decades, their efficacy has waned even though prescription rates continue to skyrocket. Drugs that were initially intended to be used for a limited period of time are being prescribed for decades. The side effects are killing the people these drugs were designed, or at least marketed, to save. 

Another realm susceptible to dubious claims is acupuncture. As Jonah Lehrer points out,  between the years of 1966 and 1995, 47 out of 47 studies in China, Taiwan, and Japan concluded it to be an effective treatment. All of these nations have long believed in acupuncture’s efficacy. Americans are a bit more skeptical. The 94 trials during that time resulted in a 56 percent efficacy rate. Lehrer continues,

This wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.

One fix to this blindness would be an open-source database in which intentions and investigations exist side by side. Instead of companies and research institutions with a stake in the results holding back what they don’t want to report or “curve-fitting” what they do, transparency would be built into the system from the outset. This might not alter the decline effect Lehrer mentions, but it would hold researchers accountable for their intentions and results during every step of the process.

Of course, there’s still human belief to contend with. That includes the pre-existing beliefs of researchers, the belief held by CEOs and board members that their data are proprietary, and the innumerable beliefs plastered across the internet on health blogs with no (or questionable) studies being championed as the final word on the subject.

The scientific method is one of the great inventions of both our rational and imaginative mind. Being honest with the evidence is another story.


Stay in touch with Derek on Facebook and Twitter.

LinkedIn meets Tinder in this mindful networking app

Swipe right to make the connections that could change your career.

Getty Images
Swipe right. Match. Meet over coffee or set up a call.

No, we aren't talking about Tinder. Introducing Shapr, a free app that helps people with synergistic professional goals and skill sets easily meet and collaborate.

Keep reading Show less

4 reasons Martin Luther King, Jr. fought for universal basic income

In his final years, Martin Luther King, Jr. become increasingly focused on the problem of poverty in America.

(Photo by J. Wilds/Keystone/Getty Images)
Politics & Current Affairs
  • Despite being widely known for his leadership role in the American civil rights movement, Martin Luther King, Jr. also played a central role in organizing the Poor People's Campaign of 1968.
  • The campaign was one of the first to demand a guaranteed income for all poor families in America.
  • Today, the idea of a universal basic income is increasingly popular, and King's arguments in support of the policy still make a good case some 50 years later.
Keep reading Show less

Dead – yes, dead – tardigrade found beneath Antarctica

A completely unexpected discovery beneath the ice.

(Goldstein Lab/Wkikpedia/Tigerspaws/Big Think)
Surprising Science
  • Scientists find remains of a tardigrade and crustaceans in a deep, frozen Antarctic lake.
  • The creatures' origin is unknown, and further study is ongoing.
  • Biology speaks up about Antarctica's history.
Keep reading Show less

Why I wear my life on my skin

For Damien Echols, tattoos are part of his existential armor.

  • In prison Damien Echols was known by his number SK931, not his name, and had his hair sheared off. Stripped of his identity, the only thing he had left was his skin.
  • This is why he began tattooing things that are meaningful to him — to carry a "suit of armor" made up the images of the people and objects that have significance to him, from his friends to talismans.
  • Echols believes that all places are imbued with divinity: "If you interact with New York City as if there's an intelligence behind... then it will behave towards you the same way."
Keep reading Show less