Most Research Is (Probably) Bogus

Most Research Is (Probably) Bogus

Science works through experimentation and replication. Hypotheses are put forward; tests are run; and results are obtained. If something surprising or interesting is found in an experiment, other researchers can then replicate the study to make sure that the effect is robust and not just due to chance.


Unfortunately, replication is lacking in many different sciences. This is partly because doing original research is better for one’s career. How many people have you heard of who made a name for themselves by merely replicating the interesting work of other people? It doesn’t happen.

However, while researchers might not have an incentive to replicate their own surprising results, or the results of others, people in the private sector do. That’s why, throughout the early 2000s, scientists at the drug company Amgen tried to replicate findings from 53 monumental studies in cancer biology. In the end, they were able to replicate only six of the studies: just 11.3 percent.

These dismal results might explain why we haven’t made as much progress as predicted in the war against cancer. But oncology is not the only area with serious replication issues. Just last week, the preliminary results of psychology’s first large reproducibility project were released. Out of 100 articles chosen, only 39 could be replicated. While 39 percent is much better than 11.3 percent, it’s still not great. It means that any surprising research you read about on the Internet is more likely to be false than true.

This is why, when you read about a psychology finding that is truly counterintuitive, you should be a bit critical. Unless it has been replicated or is part of a large body of similar research, you shouldn’t change your beliefs. Be on the lookout for similar research in the future, but don’t assume that a single study with a surprising result should be taken as gospel.

There are exceptions to this, of course. The larger the number of people that took place in the experiment, the more you can trust the results. The same holds true if there was a more representative sample. Unfortunately, most of the studies done today have small samples consisting of university students — a population that is definitely not representative of the broader population. Such studies may teach us a lot about university students, but they won’t necessarily point us to human universals. This is why spectacular findings must be probed with larger and more diverse samples, as we have seen with the popular field of mindset psychology. The original mindset studies seemed to imply that our beliefs relating to whether or not we can improve in a specific area determine how well we do, and how hard we work, in that area. Many of the original studies only contained a few dozen participants. But subsequent studies with sample sizes in the hundreds, or thousands, have since been performed, with impressive and positive results.

However, after more studies are replicated, we may find that many of the puzzling, counterintuitive results that have received a lot of press attention (and book pages) are, in fact, untrue. For example, researchers have recently had trouble replicating one of the classic studies on priming — in which study participants who were presented with words associated with old age walked more slowly right after the experiment. This seems unbelievable; perhaps for good reason. When things are settled, the world of psychological science may look a lot more like the world of common sense. It seems as if behavioral researchers have been so obsessed with rewriting our conceptions of ourselves that we’ve forgotten that generations of brilliant and perceptive minds have come before us. And in their maxims, common sense, and sayings, we have incisive psychological truths that we’re rediscovering in research today.

Plato, of 400 B.C., even saw our minds as a charioteer at the helm, controlling two powerful horses, good and bad. He predated Sigmund Freud and modern psychological scientists by over 2,000 years, yet understood the inhuman and unconscious drives that every person has to grapple with and attempt to master. He, in a very real sense, understood the multilayered nature of consciousness and the brain.

As psychological research moves forward, I think we will continue to find ourselves looking backward at the wise people of past eras for guidance in our quest onward.

U.S. Navy controls inventions that claim to change "fabric of reality"

Inventions with revolutionary potential made by a mysterious aerospace engineer for the U.S. Navy come to light.

U.S. Navy ships

Credit: Getty Images
Surprising Science
  • U.S. Navy holds patents for enigmatic inventions by aerospace engineer Dr. Salvatore Pais.
  • Pais came up with technology that can "engineer" reality, devising an ultrafast craft, a fusion reactor, and more.
  • While mostly theoretical at this point, the inventions could transform energy, space, and military sectors.
Keep reading Show less

Why so gassy? Mysterious methane detected on Saturn’s moon

Scientists do not know what is causing the overabundance of the gas.

An impression of NASA's Cassini spacecraft flying through a water plume on the surface of Saturn's moon Enceladus.

Credit: NASA
Surprising Science
  • A new study looked to understand the source of methane on Saturn's moon Enceladus.
  • The scientists used computer models with data from the Cassini spacecraft.
  • The explanation could lie in alien organisms or non-biological processes.
Keep reading Show less

CRISPR therapy cures first genetic disorder inside the body

It marks a breakthrough in using gene editing to treat diseases.

Credit: National Cancer Institute via Unsplash
Technology & Innovation

This article was originally published by our sister site, Freethink.

For the first time, researchers appear to have effectively treated a genetic disorder by directly injecting a CRISPR therapy into patients' bloodstreams — overcoming one of the biggest hurdles to curing diseases with the gene editing technology.

The therapy appears to be astonishingly effective, editing nearly every cell in the liver to stop a disease-causing mutation.

The challenge: CRISPR gives us the ability to correct genetic mutations, and given that such mutations are responsible for more than 6,000 human diseases, the tech has the potential to dramatically improve human health.

One way to use CRISPR to treat diseases is to remove affected cells from a patient, edit out the mutation in the lab, and place the cells back in the body to replicate — that's how one team functionally cured people with the blood disorder sickle cell anemia, editing and then infusing bone marrow cells.

Bone marrow is a special case, though, and many mutations cause disease in organs that are harder to fix.

Another option is to insert the CRISPR system itself into the body so that it can make edits directly in the affected organs (that's only been attempted once, in an ongoing study in which people had a CRISPR therapy injected into their eyes to treat a rare vision disorder).

Injecting a CRISPR therapy right into the bloodstream has been a problem, though, because the therapy has to find the right cells to edit. An inherited mutation will be in the DNA of every cell of your body, but if it only causes disease in the liver, you don't want your therapy being used up in the pancreas or kidneys.

A new CRISPR therapy: Now, researchers from Intellia Therapeutics and Regeneron Pharmaceuticals have demonstrated for the first time that a CRISPR therapy delivered into the bloodstream can travel to desired tissues to make edits.

We can overcome one of the biggest challenges with applying CRISPR clinically.

—JENNIFER DOUDNA

"This is a major milestone for patients," Jennifer Doudna, co-developer of CRISPR, who wasn't involved in the trial, told NPR.

"While these are early data, they show us that we can overcome one of the biggest challenges with applying CRISPR clinically so far, which is being able to deliver it systemically and get it to the right place," she continued.

What they did: During a phase 1 clinical trial, Intellia researchers injected a CRISPR therapy dubbed NTLA-2001 into the bloodstreams of six people with a rare, potentially fatal genetic disorder called transthyretin amyloidosis.

The livers of people with transthyretin amyloidosis produce a destructive protein, and the CRISPR therapy was designed to target the gene that makes the protein and halt its production. After just one injection of NTLA-2001, the three patients given a higher dose saw their levels of the protein drop by 80% to 96%.

A better option: The CRISPR therapy produced only mild adverse effects and did lower the protein levels, but we don't know yet if the effect will be permanent. It'll also be a few months before we know if the therapy can alleviate the symptoms of transthyretin amyloidosis.

This is a wonderful day for the future of gene-editing as a medicine.

—FYODOR URNOV

If everything goes as hoped, though, NTLA-2001 could one day offer a better treatment option for transthyretin amyloidosis than a currently approved medication, patisiran, which only reduces toxic protein levels by 81% and must be injected regularly.

Looking ahead: Even more exciting than NTLA-2001's potential impact on transthyretin amyloidosis, though, is the knowledge that we may be able to use CRISPR injections to treat other genetic disorders that are difficult to target directly, such as heart or brain diseases.

"This is a wonderful day for the future of gene-editing as a medicine," Fyodor Urnov, a UC Berkeley professor of genetics, who wasn't involved in the trial, told NPR. "We as a species are watching this remarkable new show called: our gene-edited future."

Quantcast