from the world's big
Believe It Or Not, Most Published Research Findings Are Probably False
Ten years ago, a researcher claimed most published research findings are false; now a decade later, his claim is stronger than ever before. How can this be?
The rise of the Internet has worked wonders for the public's access to science, but this has come with the side effect of a toxic combination of confirmation bias and Google, enabling us to easily find a study to support whatever it is that we already believe, without bothering to so much as look at research that might challenge our position — or the research that supports our position for that matter. I'm certainly not immune myself from credulously accepting research that has later been called into question, even on this blog where I take great effort to take a skeptical approach and highlight false claims arising from research. Could it be the case that studies with incorrect findings are not just rare anomalies, but are actually representative of the majority of published research?
The claim that "most published research findings are false" is something you might reasonably expect to come out of the mouth of the most deluded kind of tin-foil-hat-wearing-conspiracy-theorist. Indeed, this is a statement oft-used by fans of pseudoscience who take the claim at face value, without applying the principles behind it to their own evidence. It is however, a concept that is actually increasingly well understood by scientists. It is the title of a paper written 10 years ago by the legendary Stanford epidemiologist John Ioannidis. The paper, which has become the most widely cited paper ever published in the journal PLoS Medicine, examined how issues currently ingrained in the scientific process combined with the way we currently interpret statistical significance, means that at present, most published findings are likely to be incorrect.
Richard Horton, the editor of The Lancet recently put it only slightly more mildly: "Much of the scientific literature, perhaps half, may simply be untrue." Horton agrees with Ioannidis' reasoning, blaming: "small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance." Horton laments: "Science has taken a turn towards darkness."
Last year UCL pharmacologist and statistician David Colquhoun published a report in the Royal Society's Open Science in which he backed up Ioannidis' case: "If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30 percent of the time." That's assuming "the most optimistic view possible" in which every experiment is perfectly designed, with perfectly random allocation, zero bias, no multiple comparisons and publication of all negative findings. Colquhorn concludes: "If, as is often the case, experiments are underpowered, you will be wrong most of the time."
The numbers above are theoretical, but are increasingly being backed up by hard evidence. The rate of findings that have later been found to be wrong or exaggerated has been found to be 30 percent for the top most widely cited randomized, controlled trials in the world's highest-quality medical journals. For non-randomized trials that number rises to an astonishing five out of six.
Over recent years Ioannidis' argument has received support from multiple fields. Three years ago, when drugs company Amgen tried to replicate the "landmark publications" in the field of cancer drug development for a report published in Nature, 47 out of 53 could not be replicated. When Bayer attempted a similar project on drug target studies, 65 percent of the studies could not be replicated.
The problem is being tackled head on in the field of psychology which was shaken by the Stapel affair in which one Dutch researcher fabricated data in over 50 fraudulent papers before being detected. The social sciences received another blow recently when Michael LaCour was accused of fabricating data; the case exposed how studies are routinely published without raw data ever being made available to reviewers.
A massive operation titled The Open Science Collaboration, involving 270 scientists, has so far attempted to replicate 100 psychology experiments, but only succeeded in replicating 39 studies. The project looked at the first articles published in 2008 in the leading psychology journals. The news wasn't entirely bad; the majority of the non-replications were described by the researchers as having at the very least "slightly similar" findings. The resulting paper is currently under review for publication in Science, so we'll have to wait before we get more details. The paper is likely to ruffle some feathers; tempers flared a few years ago when one of the most high-profile findings of recent years, the concept of behavioral priming, was called into question after a series of failed replications.
Whatever way you look at it, these issues are extremely worrying. Understanding the problem is essential in order to know when to take scientific claims seriously. Below I explore some of Ioannidis' key observations:
The smaller the study, the less likely the findings are to be true.
Large studies are expensive, take longer and are less effective at padding out a CV; consequently we see relatively few of them. Small studies however, are far more likely to result in statistically significant results that are in fact a false positive, so they should be treated with caution. This problem is magnified when researchers fail to publish (or journals refuse to publish) negative findings — a problem know as publication bias or the file drawer problem.
The smaller the effect size, the less likely the findings are to be true.
This sounds like it should be obvious, but it is remarkable how much research fails to actually describe the strength of the results, preferring to simply refer to statistical significance alone, which is a far less useful measure. A study's findings can be statistically significant yet have an effect size so weak that in reality the results are completely meaningless. This can be achieved through a process known as P-hacking — which was the method John Bohannon recently used to create a spoof paper finding that chocolate helps you lose weight. P-hacking involves playing with variables until a statistically significant result is achieved. As neuroscientist and blogger Neuroskeptic demonstrated in a recent talk that you can watch online, this is not always the result of foul play, but can actually happen very easily by accident if researchers simply continue conducting research in the same way most currently do now.
The greater the number and the lesser the selection of tested relationships, the less likely the findings are to be true.
This was another key factor that enabled Bohannon to design the study rigged to support the case that eating chocolate helps you lose weight. Bohannon used 18 different types of measurements, relying on the fact that some would likely support his case simply due to chance alone. This practice is currently nearly impossible to detect if researchers fail to disclose all the factors they looked at. This problem is a major factor behind the growing movement of researchers calling for the pre-registration of study methodology.
The greater the financial and other interests and prejudices, the less likely the findings are to be true.
It is always worth checking to see who funded a piece of research. Sticking with our chocolate theme, a recent study that found that chocolate is "scientifically proven to help with fading concentration" was funded by Hershey. On a more serious note, tobacco companies have a long history of funding fraudulent health research over the past century — described by the World Health Organization as "the most astonishing systematic corporate deceit of all time." Today that baton has been handed to oil companies who give money to scientists who deny global warming and fund dozens of front groups with the purpose of sowing doubt about climate change.
The hotter a scientific field, the less likely the findings are to be true.
Though seemingly counter-intuitive, it is particularly common in fast-moving fields of research where many researchers are working on the same problems at the same time, for false findings to be published and quickly debunked. This has been dubbed the Proteus Phenomenon after the Greek god Proteus, who could rapidly change his appearance. The same can be said for research published in the sexiest journals, which only accept the most groundbreaking findings, where the problem has been dubbed the Winner's Curse.
What does this all mean to you?
Thankfully science is self-correcting. Over time, findings are replicated or not replicated and the truth comes out in the wash. This is done through a process of replication involving larger, better controlled trials, meta-analyses where the data from many trials are aggregated and analyzed as a whole, and systematic reviews where studies are assessed based on predetermined criteria — preventing the cherry picking that we're all, whether we like it or not, so naturally inclined to.
Replications, meta-analyses and systematic reviews are by their nature far more useful for portraying an accurate picture of reality than original exploratory research. But systematic reviews rarely make headlines, which is a good reason the news is not the best place to get an informed opinion about matters of science. The problem is unlikely to go away any time soon, so whenever you hear about a new piece of science news, remember the principles above and the simple rule of thumb that studies of studies are far more likely to present a true picture of reality than individual pieces of research.
What does this mean for scientists?
For scientists, the discussion over how to resolve the problem is rapidly heating up with calls for big changes to how researchers register, conduct, and publish research and a growing chorus from hundreds of global scientific organizations demanding that all clinical trials are published. Perhaps most important and most difficult to change, is the structure of perverse incentives that places intense pressure on scientists to produce positive results while actively encouraging them to quietly sit on negative ones.
Innovation in manufacturing has crawled since the 1950s. That's about to speed up.
Richard Feynman once asked a silly question. Two MIT students just answered it.
Here's a fun experiment to try. Go to your pantry and see if you have a box of spaghetti. If you do, take out a noodle. Grab both ends of it and bend it until it breaks in half. How many pieces did it break into? If you got two large pieces and at least one small piece you're not alone.
But science loves a good challenge<p>The mystery remained unsolved until 2005, when French scientists <a href="http://www.lmm.jussieu.fr/~audoly/" target="_blank">Basile Audoly</a> and <a href="http://www.lmm.jussieu.fr/~neukirch/" target="_blank">Sebastien Neukirch </a>won an <a href="https://www.improbable.com/ig/" target="_blank">Ig Nobel Prize</a>, an award given to scientists for real work which is of a less serious nature than the discoveries that win Nobel prizes, for finally determining why this happens. <a href="http://www.lmm.jussieu.fr/spaghetti/audoly_neukirch_fragmentation.pdf" target="_blank">Their paper describing the effect is wonderfully funny to read</a>, as it takes such a banal issue so seriously. </p><p>They demonstrated that when a rod is bent past a certain point, such as when spaghetti is snapped in half by bending it at the ends, a "snapback effect" is created. This causes energy to reverberate from the initial break to other parts of the rod, often leading to a second break elsewhere.</p><p>While this settled the issue of <em>why </em>spaghetti noodles break into three or more pieces, it didn't establish if they always had to break this way. The question of if the snapback could be regulated remained unsettled.</p>
Physicists, being themselves, immediately wanted to try and break pasta into two pieces using this info<p><a href="https://roheiss.wordpress.com/fun/" target="_blank">Ronald Heisser</a> and <a href="https://math.mit.edu/directory/profile.php?pid=1787" target="_blank">Vishal Patil</a>, two graduate students currently at Cornell and MIT respectively, read about Feynman's night of noodle snapping in class and were inspired to try and find what could be done to make sure the pasta always broke in two.</p><p><a href="http://news.mit.edu/2018/mit-mathematicians-solve-age-old-spaghetti-mystery-0813" target="_blank">By placing the noodles in a special machine</a> built for the task and recording the bending with a high-powered camera, the young scientists were able to observe in extreme detail exactly what each change in their snapping method did to the pasta. After breaking more than 500 noodles, they found the solution.</p>
The apparatus the MIT researchers built specifically for the task of snapping hundreds of spaghetti sticks.
(Courtesy of the researchers)
What possible application could this have?<p>The snapback effect is not limited to uncooked pasta noodles and can be applied to rods of all sorts. The discovery of how to cleanly break them in two could be applied to future engineering projects.</p><p>Likewise, knowing how things fragment and fail is always handy to know when you're trying to build things. Carbon Nanotubes, <a href="https://bigthink.com/ideafeed/carbon-nanotube-space-elevator" target="_self">super strong cylinders often hailed as the building material of the future</a>, are also rods which can be better understood thanks to this odd experiment.</p><p>Sometimes big discoveries can be inspired by silly questions. If it hadn't been for Richard Feynman bending noodles seventy years ago, we wouldn't know what we know now about how energy is dispersed through rods and how to control their fracturing. While not all silly questions will lead to such a significant discovery, they can all help us learn.</p>
Join Pulitzer Prize-winning reporter and best-selling author Charles Duhigg as he interviews Victoria Montgomery Brown, co-founder and CEO of Big Think.
Women today are founding more businesses than ever. In 2018, they made up 40% of new entrepreneurs, yet in that same year, they received just 2.2% of all venture capital investment. The playing field is off-balance. So what can women do?
In a recent study, researchers examined how Christian nationalism is affecting the U.S. response to the COVID-19 pandemic.
- A new study used survey data to examine the interplay between Christian nationalism and incautious behaviors during the COVID-19 pandemic.
- The researchers defined Christian nationalism as "an ideology that idealizes and advocates a fusion of American civic life with a particular type of Christian identity and culture."
- The results showed that Christian nationalism was the leading predictor that Americans engaged in incautious behavior.
A pastor at the chapel of the St. Josef Hospital on April 1, 2020 in Bochum, German
Sascha Schuermann/Getty Images<p>Christian nationalists, in general, believe the U.S. and God's will are tied together, and they want the government to embody conservative Christian values and symbols. As such, they also believe the nation's fate depends on how closely it adheres to Christianity.</p><p style="margin-left: 20px;">"Unsurprisingly then, in the midst of the COVID‐19 pandemic, conservative pastors prophesied God's protection over the nation, citing America's righteous support for President Trump and the prolife agenda," the researchers write.</p><p style="margin-left: 20px;">"Correspondingly, the link between Christian nationalism and God's influence on how COVID‐19 impacts America can be seen in proclamations about God's divine judgment for its immorality―with the logic being that God is using the pandemic to draw wayward America <em>back </em>to himself, which assumes the two belong together."</p><p>The logical conclusion to this kind of thinking: America can save itself not through cautionary measures, like mask-wearing, but through devotion to God. What's more, it stands to reason that Christian nationalists are less likely to trust the media and scientists, given that these sources are generally not concerned with promoting a conservative, religious view of the world.</p><p>(The researchers note that they're unaware of any research directly linking Christian nationalism to distrust of media sources, but that they're almost certain the two are connected.)</p>
Predicted values of Americans' frequency of incautious behaviors during the COVID‐19 pandemic across values of Christian nationalism
Perry et al.<p>In the new study, the researchers examined three waves of results from the Public and Discourse Ethics Survey. One wave of the survey was issued in May, and it asked respondents to rate how often they engaged in both incautious and precautionary behaviors.</p><p>Incautious behaviors included things like "ate inside a restaurant" and "went shopping for nonessential items," while precautionary behaviors included "washed my hands more often than typical" and "wore a mask in public."</p><p>To measure Christian nationalism, the researchers asked respondents to rate how strongly they agree with statements like "the federal government should advocate Christian values" and "the success of the United States is part of God's plan."</p><p>The results suggest that, compared to other groups, Christian nationalists are far less likely to wear masks, socially distance and take other precautionary measures amid the COVID-19 pandemic.</p><p style="margin-left: 20px;">"Christian nationalism was the leading predictor that Americans engaged in incautious behavior during the pandemic, and the second leading predictor that Americans avoided taking precautionary measures."</p><p>But that's not to say that religious beliefs are causing Americans to reject mask-wearing or social distancing. In fact, when the study accounted for Christian nationalist beliefs, the results showed that Americans with high levels of religiosity were likely to take precautionary measures for COVID-19.</p>
Limitations<p>Still, the researchers note that they're theorizing about the connections between Christian nationalism and COVID-19 behaviors, not documenting them directly. What's more, they suggest that certain experiences — such as having a family member that contracts COVID-19 — might change a Christian nationalist's behaviors during the pandemic.</p><p style="margin-left: 20px;">"Limitations notwithstanding, the implications of this study are important for understanding Americans' curious inability to quickly implement informed and reasonable strategies to overcome the threat of COVID‐19, an inability that has likely cost thousands of lives," they write.</p>
Parental anxieties stem from the complex relationship between technology, child development, and the internet's trove of unseemly content.