Most Research Is (Probably) Bogus

Most Research Is (Probably) Bogus

Science works through experimentation and replication. Hypotheses are put forward; tests are run; and results are obtained. If something surprising or interesting is found in an experiment, other researchers can then replicate the study to make sure that the effect is robust and not just due to chance.

Unfortunately, replication is lacking in many different sciences. This is partly because doing original research is better for one’s career. How many people have you heard of who made a name for themselves by merely replicating the interesting work of other people? It doesn’t happen.

However, while researchers might not have an incentive to replicate their own surprising results, or the results of others, people in the private sector do. That’s why, throughout the early 2000s, scientists at the drug company Amgen tried to replicate findings from 53 monumental studies in cancer biology. In the end, they were able to replicate only six of the studies: just 11.3 percent.

These dismal results might explain why we haven’t made as much progress as predicted in the war against cancer. But oncology is not the only area with serious replication issues. Just last week, the preliminary results of psychology’s first large reproducibility project were released. Out of 100 articles chosen, only 39 could be replicated. While 39 percent is much better than 11.3 percent, it’s still not great. It means that any surprising research you read about on the Internet is more likely to be false than true.

This is why, when you read about a psychology finding that is truly counterintuitive, you should be a bit critical. Unless it has been replicated or is part of a large body of similar research, you shouldn’t change your beliefs. Be on the lookout for similar research in the future, but don’t assume that a single study with a surprising result should be taken as gospel.

There are exceptions to this, of course. The larger the number of people that took place in the experiment, the more you can trust the results. The same holds true if there was a more representative sample. Unfortunately, most of the studies done today have small samples consisting of university students — a population that is definitely not representative of the broader population. Such studies may teach us a lot about university students, but they won’t necessarily point us to human universals. This is why spectacular findings must be probed with larger and more diverse samples, as we have seen with the popular field of mindset psychology. The original mindset studies seemed to imply that our beliefs relating to whether or not we can improve in a specific area determine how well we do, and how hard we work, in that area. Many of the original studies only contained a few dozen participants. But subsequent studies with sample sizes in the hundreds, or thousands, have since been performed, with impressive and positive results.

However, after more studies are replicated, we may find that many of the puzzling, counterintuitive results that have received a lot of press attention (and book pages) are, in fact, untrue. For example, researchers have recently had trouble replicating one of the classic studies on priming — in which study participants who were presented with words associated with old age walked more slowly right after the experiment. This seems unbelievable; perhaps for good reason. When things are settled, the world of psychological science may look a lot more like the world of common sense. It seems as if behavioral researchers have been so obsessed with rewriting our conceptions of ourselves that we’ve forgotten that generations of brilliant and perceptive minds have come before us. And in their maxims, common sense, and sayings, we have incisive psychological truths that we’re rediscovering in research today.

Plato, of 400 B.C., even saw our minds as a charioteer at the helm, controlling two powerful horses, good and bad. He predated Sigmund Freud and modern psychological scientists by over 2,000 years, yet understood the inhuman and unconscious drives that every person has to grapple with and attempt to master. He, in a very real sense, understood the multilayered nature of consciousness and the brain.

As psychological research moves forward, I think we will continue to find ourselves looking backward at the wise people of past eras for guidance in our quest onward.

How New York's largest hospital system is predicting COVID-19 spikes

Northwell Health is using insights from website traffic to forecast COVID-19 hospitalizations two weeks in the future.

Credit: Getty Images
Sponsored by Northwell Health
  • The machine-learning algorithm works by analyzing the online behavior of visitors to the Northwell Health website and comparing that data to future COVID-19 hospitalizations.
  • The tool, which uses anonymized data, has so far predicted hospitalizations with an accuracy rate of 80 percent.
  • Machine-learning tools are helping health-care professionals worldwide better constrain and treat COVID-19.
Keep reading Show less

Listen: Scientists re-create voice of 3,000-year-old Egyptian mummy

Scientists used CT scanning and 3D-printing technology to re-create the voice of Nesyamun, an ancient Egyptian priest.

Surprising Science
  • Scientists printed a 3D replica of the vocal tract of Nesyamun, an Egyptian priest whose mummified corpse has been on display in the UK for two centuries.
  • With the help of an electronic device, the reproduced voice is able to "speak" a vowel noise.
  • The team behind the "Voices of the Past" project suggest reproducing ancient voices could make museum experiences more dynamic.
Keep reading Show less

Dark matter axions possibly found near Magnificent 7 neutron stars

A new study proposes mysterious axions may be found in X-rays coming from a cluster of neutron stars.

A rendering of the XMM-Newton (X-ray multi-mirror mission) space telescope.

Credit: D. Ducros; ESA/XMM-Newton, CC BY-SA 3.0 IGO
Surprising Science
  • A study led by Berkeley Lab suggests axions may be present near neutron stars known as the Magnificent Seven.
  • The axions, theorized fundamental particles, could be found in the high-energy X-rays emitted from the stars.
  • Axions have yet to be observed directly and may be responsible for the elusive dark matter.
  • Keep reading Show less

    Put on a happy face? “Deep acting” associated with improved work life

    New research suggests you can't fake your emotional state to improve your work life — you have to feel it.

    Credit: Columbia Pictures
    Personal Growth
  • Deep acting is the work strategy of regulating your emotions to match a desired state.
  • New research suggests that deep acting reduces fatigue, improves trust, and advances goal progress over other regulation strategies.
  • Further research suggests learning to attune our emotions for deep acting is a beneficial work-life strategy.
  • Keep reading Show less
    Surprising Science

    World's oldest work of art found in a hidden Indonesian valley

    Archaeologists discover a cave painting of a wild pig that is now the world's oldest dated work of representational art.

    Scroll down to load more…