Psychology studies that go viral are likelier to be bogus

At the start of the 21st century, psychology was starting to get a little out of hand. Studies conducted on small groups of college undergraduates were growing more commonplace and increasingly being used to explain just about every aspect of human behavior. Their findings filtered into talks by motivational speakers, bland self-help books, and poppy magazine articles, turning lots of people into well-informed (or so they thought) armchair psychologists.

Psychology’s replication crisis

But in the early 2010s, the bubble burst. When scientists attempted to replicate prior research, performing the experiments again with rigorous methods and new subjects, they were broadly unsuccessful. In about half of the repeated experiments, the flashy findings disappeared. Even apparently established phenomena like social priming and “power posing” were not immune from debunking. Psychology’s heyday transformed into a “replication crisis.”

Psychologists are now starting to learn from their mistakes, recruiting larger, more diverse sample groups, publishing their experimental methods in advance and sticking to them, and adhering to stricter statistical analyses. So the field may yet regain its lost credibility.

In the meantime, a trio of researchers, Wu Youyou, Yang Yang, and Brian Uzzi based out of University College London, University of Notre Dame, and Northwestern University, respectively, created and validated a machine-learning model to assess the replicability of psychology studies simply by scanning their texts. Then, they ran their model on just over 14,000 papers published in six top-tier psychology journals between 2000 and 2019. With this bounty of data, they explored how a paper’s replicability was linked to factors such as an author’s citation history (how often their prior work is sourced by other researchers), the prestige of the institution behind the study, the focus of the research, and a paper’s media coverage. Their results were published in PNAS.

Not all psychology research is equal

First, they found that research focused on personality had the highest likelihood of replicating, roughly 55%, while research in developmental psychology (examining behavioral change over a human lifespan) had the least chance of replicating at 36%. Scientific papers in the ever popular field of social psychology (exploring human interactions) had a mere 37% likelihood of replicating.

“One explanation for this pattern is that developmental psychology focuses on children and life courses, two areas in which researchers face unique difficulties in collecting large samples under controllable circumstances,” the authors commented.

The researchers also discovered that the papers of more widely published authors were more likely to replicate, while oft-cited papers were no more likely to replicate.

Most intriguingly, they found that papers which received more media attention were less likely to replicate. News outlets tend to favor counterintuitive and eye-catching findings. Though surprising results sometimes overturn established science and might even go against conventional wisdom, they are less likely to withstand the test of replication.

“It would be valuable for the media to remind the audience that new and novel scientific results are only food for thought before future replication confirms their robustness,” the authors chided.

There was some good news from the new research. Between 2010 and 2019, average replication scores increased, a hopeful sign that psychology may be starting to move past its replication crisis.