Can we trust studies when humans have a vested interest in the outcome?

Lack of replication is a serious problem in science. So far, no one has an answer.

Though rumor had it that citrus cured sailors of scurvy—crews were aware of this as early as 1497—it was not until James Lancaster’s 1601 voyage to Sumatra that this bit of folk wisdom was put to the test. With four ships under his command, Lancaster decided to try a little experiment: men on one ship received regular lemon juice shots; the other three were dry. 

We know the results. Men on those three ships developed scurvy. More importantly, Lancaster inadvertently created the first scientific trial. Nearly a century-and-a-half later, Scottish physician James Lind is credited with the first clinical trial (also studying scurvy); a few decades onward, in 1784 Benjamin Franklin and Antoine Lavoisier performed the first blind experiment in France to test out Franz Mesmer’s theory of animal magnetism. (It doesn’t exist.)

The double-blind method was introduced so researchers would not influence volunteers by inadvertently (or knowingly) guiding them to make certain claims. For nearly two centuries, this type of clinical study has been the gold standard of scientific research. Whenever you hear the statement, “studies show…,” if not conducted in this manner, it’s not considered valid.

Yet in the last few decades, a number of problems have arisen. As much as we’d like to believe that data are data and humans are merely reporting it, researcher bias has led to numerous cases of selective reporting—only publishing studies that confirm the initial hypothesis. This is especially problematic with pharmaceutical studies, which are often funded by corporations with a vested interest in getting the result they’d like to advertise.

Statistician Theodore Sterling noticed a troubling trend as far back as 1959: 97 percent of psychological studies confirmed the effect they initially set out to prove. This is an unbelievable number because, well, it shouldn’t be believed. The bar for a positive trial was created in 1922 by mathematician Ronald Fisher, who said that a “significant” result is produced by chance less than 5 percent of the time. (In statistics, a 95 percent confidence interval is standard, though a 99 percent confidence interval is more accurate.) Incredibly, Fisher invented this stat not out of confidence, but convenience

[Fisher] picked five per cent as the boundary line, somewhat arbitrarily, because it made pencil and slide-rule calculations easier.

Even more harrowing is the lack of replication. Besides hiding data, as many pharmaceutical companies have been shown to do, no study should be considered foolproof until it’s been repeated by unaffiliated parties. By the time research has been released to the public, however, we tend to take it on its face value. When conflicting evidence is presented, we often reject claims that contradict what we already believe.

One such case is sexual selection and symmetry, the notion that animals (including us) choose mates based on symmetrical features. This notion was implanted in the social consciousness in 1991 after a study on barn swallows speculated that females chose mates males with long, symmetrical feathers, a notion that was quickly extrapolated to include humans with symmetrical facial features. By 1997, this study had been replicated dozens of times, resulting in an overall 80 percent reduction in effect size. Still, the myth persists.

Barn swallows and one night stands are one thing, but a similar phenomenon has occurred with SSRIs. Since the introduction of fluoxetine in 1987, this class of antidepressants has been the go-to for treating depression and anxiety. Yet in the intervening decades, their efficacy has waned even though prescription rates continue to skyrocket. Drugs that were initially intended to be used for a limited period of time are being prescribed for decades. The side effects are killing the people these drugs were designed, or at least marketed, to save. 

Another realm susceptible to dubious claims is acupuncture. As Jonah Lehrer points out,  between the years of 1966 and 1995, 47 out of 47 studies in China, Taiwan, and Japan concluded it to be an effective treatment. All of these nations have long believed in acupuncture’s efficacy. Americans are a bit more skeptical. The 94 trials during that time resulted in a 56 percent efficacy rate. Lehrer continues,

This wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.

One fix to this blindness would be an open-source database in which intentions and investigations exist side by side. Instead of companies and research institutions with a stake in the results holding back what they don’t want to report or “curve-fitting” what they do, transparency would be built into the system from the outset. This might not alter the decline effect Lehrer mentions, but it would hold researchers accountable for their intentions and results during every step of the process.

Of course, there’s still human belief to contend with. That includes the pre-existing beliefs of researchers, the belief held by CEOs and board members that their data are proprietary, and the innumerable beliefs plastered across the internet on health blogs with no (or questionable) studies being championed as the final word on the subject.

The scientific method is one of the great inventions of both our rational and imaginative mind. Being honest with the evidence is another story.


Stay in touch with Derek on Facebook and Twitter.

Big Think
Sponsored by Lumina Foundation

Upvote/downvote each of the videos below!

As you vote, keep in mind that we are looking for a winner with the most engaging social venture pitch - an idea you would want to invest in.

Keep reading Show less

Essential financial life skills for 21st-century Americans

Having these financial life skills can help you navigate challenging economic environments.

Photo by Jp Valery on Unsplash
Personal Growth
  • Americans are swimming in increasingly higher amounts of debt, even the upper middle class.
  • For many, this burden can be alleviated by becoming familiar with some straightforward financial concepts.
  • Here's some essential financial life skills needed to ensure your economic wellbeing.
Keep reading Show less

How to flirt: 7 tips backed by science

When it comes to flirting, love meters have nothing on these researchers' findings.

(Photo from Wikimedia)
Sex & Relationships
  • Flirting is an important part of life. It can be a fun, adventurous way to meet others and develop intimate relationships.
  • Many people find flirting to be an anxiety-ridden experience, but science can help us discover principles to be more relaxed while flirting.
  • Smiling and eye contact are proven winners, while pick-up lines are a flirty fallacy.
Keep reading Show less

New fossils suggest human ancestors evolved in Europe, not Africa

Experts argue the jaws of an ancient European ape reveal a key human ancestor.

Surprising Science
  • The jaw bones of an 8-million-year-old ape were discovered at Nikiti, Greece, in the '90s.
  • Researchers speculate it could be a previously unknown species and one of humanity's earliest evolutionary ancestors.
  • These fossils may change how we view the evolution of our species.

Homo sapiens have been on earth for 200,000 years — give or take a few ten-thousand-year stretches. Much of that time is shrouded in the fog of prehistory. What we do know has been pieced together by deciphering the fossil record through the principles of evolutionary theory. Yet new discoveries contain the potential to refashion that knowledge and lead scientists to new, previously unconsidered conclusions.

A set of 8-million-year-old teeth may have done just that. Researchers recently inspected the upper and lower jaw of an ancient European ape. Their conclusions suggest that humanity's forebearers may have arisen in Europe before migrating to Africa, potentially upending a scientific consensus that has stood since Darwin's day.

Rethinking humanity's origin story

The frontispiece of Thomas Huxley's Evidence as to Man's Place in Nature (1863) sketched by natural history artist Benjamin Waterhouse Hawkins. (Photo: Wikimedia Commons)

As reported in New Scientist, the 8- to 9-million-year-old hominin jaw bones were found at Nikiti, northern Greece, in the '90s. Scientists originally pegged the chompers as belonging to a member of Ouranopithecus, an genus of extinct Eurasian ape.

David Begun, an anthropologist at the University of Toronto, and his team recently reexamined the jaw bones. They argue that the original identification was incorrect. Based on the fossil's hominin-like canines and premolar roots, they identify that the ape belongs to a previously unknown proto-hominin.

The researchers hypothesize that these proto-hominins were the evolutionary ancestors of another European great ape Graecopithecus, which the same team tentatively identified as an early hominin in 2017. Graecopithecus lived in south-east Europe 7.2 million years ago. If the premise is correct, these hominins would have migrated to Africa 7 million years ago, after undergoing much of their evolutionary development in Europe.

Begun points out that south-east Europe was once occupied by the ancestors of animals like the giraffe and rhino, too. "It's widely agreed that this was the found fauna of most of what we see in Africa today," he told New Scientists. "If the antelopes and giraffes could get into Africa 7 million years ago, why not the apes?"

He recently outlined this idea at a conference of the American Association of Physical Anthropologists.

It's worth noting that Begun has made similar hypotheses before. Writing for the Journal of Human Evolution in 2002, Begun and Elmar Heizmann of the Natural history Museum of Stuttgart discussed a great ape fossil found in Germany that they argued could be the ancestor (broadly speaking) of all living great apes and humans.

"Found in Germany 20 years ago, this specimen is about 16.5 million years old, some 1.5 million years older than similar species from East Africa," Begun said in a statement then. "It suggests that the great ape and human lineage first appeared in Eurasia and not Africa."

Migrating out of Africa

In the Descent of Man, Charles Darwin proposed that hominins descended out of Africa. Considering the relatively few fossils available at the time, it is a testament to Darwin's astuteness that his hypothesis remains the leading theory.

Since Darwin's time, we have unearthed many more fossils and discovered new evidence in genetics. As such, our African-origin story has undergone many updates and revisions since 1871. Today, it has splintered into two theories: the "out of Africa" theory and the "multi-regional" theory.

The out of Africa theory suggests that the cradle of all humanity was Africa. Homo sapiens evolved exclusively and recently on that continent. At some point in prehistory, our ancestors migrated from Africa to Eurasia and replaced other subspecies of the genus Homo, such as Neanderthals. This is the dominant theory among scientists, and current evidence seems to support it best — though, say that in some circles and be prepared for a late-night debate that goes well past last call.

The multi-regional theory suggests that humans evolved in parallel across various regions. According to this model, the hominins Homo erectus left Africa to settle across Eurasia and (maybe) Australia. These disparate populations eventually evolved into modern humans thanks to a helping dollop of gene flow.

Of course, there are the broad strokes of very nuanced models, and we're leaving a lot of discussion out. There is, for example, a debate as to whether African Homo erectus fossils should be considered alongside Asian ones or should be labeled as a different subspecies, Homo ergaster.

Proponents of the out-of-Africa model aren't sure whether non-African humans descended from a single migration out of Africa or at least two major waves of migration followed by a lot of interbreeding.

Did we head east or south of Eden?

Not all anthropologists agree with Begun and his team's conclusions. As noted by New Scientist, it is possible that the Nikiti ape is not related to hominins at all. It may have evolved similar features independently, developing teeth to eat similar foods or chew in a similar manner as early hominins.

Ultimately, Nikiti ape alone doesn't offer enough evidence to upend the out of Africa model, which is supported by a more robust fossil record and DNA evidence. But additional evidence may be uncovered to lend further credence to Begun's hypothesis or lead us to yet unconsidered ideas about humanity's evolution.