Most Research Is (Probably) Bogus

Science works through experimentation and replication. Hypotheses are put forward; tests are run; and results are obtained. If something surprising or interesting is found in an experiment, other researchers can then replicate the study to make sure that the effect is robust and not just due to chance.

Unfortunately, replication is lacking in many different sciences. This is partly because doing original research is better for one’s career. How many people have you heard of who made a name for themselves by merely replicating the interesting work of other people? It doesn’t happen.

However, while researchers might not have an incentive to replicate their own surprising results, or the results of others, people in the private sector do. That’s why, throughout the early 2000s, scientists at the drug company Amgen tried to replicate findings from 53 monumental studies in cancer biology. In the end, they were able to replicate only six of the studies: just 11.3 percent.

These dismal results might explain why we haven’t made as much progress as predicted in the war against cancer. But oncology is not the only area with serious replication issues. Just last week, the preliminary results of psychology’s first large reproducibility project were released. Out of 100 articles chosen, only 39 could be replicated. While 39 percent is much better than 11.3 percent, it’s still not great. It means that any surprising research you read about on the Internet is more likely to be false than true.

This is why, when you read about a psychology finding that is truly counterintuitive, you should be a bit critical. Unless it has been replicated or is part of a large body of similar research, you shouldn’t change your beliefs. Be on the lookout for similar research in the future, but don’t assume that a single study with a surprising result should be taken as gospel.

There are exceptions to this, of course. The larger the number of people that took place in the experiment, the more you can trust the results. The same holds true if there was a more representative sample. Unfortunately, most of the studies done today have small samples consisting of university students — a population that is definitely not representative of the broader population. Such studies may teach us a lot about university students, but they won’t necessarily point us to human universals. This is why spectacular findings must be probed with larger and more diverse samples, as we have seen with the popular field of mindset psychology. The original mindset studies seemed to imply that our beliefs relating to whether or not we can improve in a specific area determine how well we do, and how hard we work, in that area. Many of the original studies only contained a few dozen participants. But subsequent studies with sample sizes in the hundreds, or thousands, have since been performed, with impressive and positive results.

However, after more studies are replicated, we may find that many of the puzzling, counterintuitive results that have received a lot of press attention (and book pages) are, in fact, untrue. For example, researchers have recently had trouble replicating one of the classic studies on priming — in which study participants who were presented with words associated with old age walked more slowly right after the experiment. This seems unbelievable; perhaps for good reason. When things are settled, the world of psychological science may look a lot more like the world of common sense. It seems as if behavioral researchers have been so obsessed with rewriting our conceptions of ourselves that we’ve forgotten that generations of brilliant and perceptive minds have come before us. And in their maxims, common sense, and sayings, we have incisive psychological truths that we’re rediscovering in research today.

Plato, of 400 B.C., even saw our minds as a charioteer at the helm, controlling two powerful horses, good and bad. He predated Sigmund Freud and modern psychological scientists by over 2,000 years, yet understood the inhuman and unconscious drives that every person has to grapple with and attempt to master. He, in a very real sense, understood the multilayered nature of consciousness and the brain.

As psychological research moves forward, I think we will continue to find ourselves looking backward at the wise people of past eras for guidance in our quest onward.