Science's Straw Man Sting

Last week Science published a "sting operation" that runs the risk of tarnishing the entire phenomenon of open access publishing, however the paper is only representative of a tiny and very specific portion. The sting involved submitting 304 versions of the same bogus paper to different open access journals - essentially an industrial scale version of the Sokal affair, except instead of targeting postmodern cultural studies, the target was open access publishing. In this post we'll look at some of the main reasons why the sting has been broadly slated for bad science.

1. No control group - In the words of Michael Eisen, co-founder of PLoS (whose journal PLoS ONE rejected the bogus paper):

"It’s nuts to construe this as a problem unique to open access publishing, if for no other reason than the study, didn’t do the control of submitting the same paper to subscription-based publishers"

2. Bias in selection - The selection criteria meant only 304 of 2054 open access journals were included. Only journals with fees (a minority of open access journals) were selected, amongst a litany of other criteria as Jeroen Bosman points out:

"Bohannan’s selection of journals to submit his paper to is based on the Directory of Open Access Journals. Even if you consider this list as complete, which it is not, the selection Bohannon made is very skewed. Only journals with fees (article processing charges) were considered, thereby throwing out 75% (leaving only 2054 of 8250 titles). This introduced a bias, because all publishers who are just in it to make easy money are concentrated in this group and not in the 75% that does not charge. Then the author threw out all journals that were not aiming at general, biological, chemical or medical science. This meant another reduction with 85% (from 2054 to 304), introducing another bias, because the pressure to publish is highest in these fields, making them prime targets of ‘predatory’ publishers. In short, by making the selections, chances are high that (unintentionally) Bohannon focussed on a group with a relatively many rotten apples."

3. Inclusion of many known outright "criminal organisations"

121 of the papers included in the sting were sent to journals discovered through Beall's list of known predatory publishers, described in the paper as "a single page on the Internet that names and shames what he calls “predatory” publishers". It is perhaps actually surprising that 35 of the journals on Beall's blacklist actually passed the sting. In any case, the subset taken from Beall's list which makes up nearly half of the sample is clearly in no way representative of open access publishing as a whole.

4. Not peer reviewed

Ironically, the paper - about peer review - was not peer reviewed. Instead it was published in a magazine as journalism dressed up as a scientific paper. If the paper had been reviewed, it could have been suggested that something as basic as a control group might have been necessary and the unsupported conclusion could have been toned down.

5. Unsupported conclusion

The argument made in the paper that "most of the players (in the realm of open access scientific journals) are murky" is simply not demonstrated by this study because of the simple fact that "most of the players" (75%) were excluded for not having processing charges. This fact alone is a demonstration that most open access journals are not simply out for a buck - unlike many publishers - not least Science, who use mile high paywalls to prevent you from accessing science. Of course there are a great many downright fraudulent publishers at the bottom of the barrel of open access publishing, we knew that already - we've covered some of the very same ones explicitly on this blog before. It's not like there aren't dodgy papers published in the subscription publishing world - they seem to take centre stage on this blog. It's noteworthy however that one of the open access journals that accepted the dodgy papers was owned by Sage (for profit), another was published by Elsevier, who are known for crucifying institutions with their subscription model that deprives scientists at institutions around the world of science. They also run this gem:

Mike Taylor sums it up well:

"It’s a maze of preordained outcomes, multiple levels of biased selection, cherry-picked data and spin-ridden conclusions. What it shows is: predatory journals are predatory. That’s not news."

The Science paper was a wonderful idea and has indeed cast a bright and important light on many known and unknown downright fraudulent publishers, but it could have been so much more. The apparatus was there, the papers were prepared to automatically generate with random names and institutions, the automated mailer was built. The paper could have shaken the foundations of bad science where it matters, instead the author chose to state the obvious. Worst of all, they state the findings as if they are somehow representative of the whole of open access publishing - when in fact the subset represents a very specific minority. The majority who weren't targeted, don't deserve to be tarnished by the same brush.

Read part 2 of this post here.

Image Credit: Adapted from artwork by Seth Thomas Rasmussen and Woodleywonderworks.


To keep up to date with this blog you can follow Neurobonkers on TwitterFacebookRSS or join the mailing list 

3D printing might save your life one day. It's transforming medicine and health care.

What can 3D printing do for medicine? The "sky is the limit," says Northwell Health researcher Dr. Todd Goldstein.

Northwell Health
Sponsored by Northwell Health
  • Medical professionals are currently using 3D printers to create prosthetics and patient-specific organ models that doctors can use to prepare for surgery.
  • Eventually, scientists hope to print patient-specific organs that can be transplanted safely into the human body.
  • Northwell Health, New York State's largest health care provider, is pioneering 3D printing in medicine in three key ways.
Keep reading Show less

Where do atoms come from? Billions of years of cosmic fireworks.

The periodic table was a lot simpler at the beginning of the universe.

10 excerpts from Marcus Aurelius' 'Meditations' to unlock your inner Stoic

Great ideas in philosophy often come in dense packages. Then there is where the work of Marcus Aurelius.

(Getty Images)
Personal Growth
  • Meditations is a collection of the philosophical ideas of the Roman Emperor Marcus Aurelius.
  • Written as a series of notes to himself, the book is much more readable than the dry philosophy most people are used to.
  • The advice he gave to himself 2,000 years ago is increasingly applicable in our hectic, stressed-out lives.
Keep reading Show less

An organism found in dirt may lead to an anxiety vaccine, say scientists

Can dirt help us fight off stress? Groundbreaking new research shows how.

University of Colorado Boulder
Surprising Science
  • New research identifies a bacterium that helps block anxiety.
  • Scientists say this can lead to drugs for first responders and soldiers, preventing PTSD and other mental issues.
  • The finding builds on the hygiene hypothesis, first proposed in 1989.

Are modern societies trying too hard to be clean, at the detriment to public health? Scientists discovered that a microorganism living in dirt can actually be good for us, potentially helping the body to fight off stress. Harnessing its powers can lead to a "stress vaccine".

Researchers at the University of Colorado Boulder found that the fatty 10(Z)-hexadecenoic acid from the soil-residing bacterium Mycobacterium vaccae aids immune cells in blocking pathways that increase inflammation and the ability to combat stress.

The study's senior author and Integrative Physiology Professor Christopher Lowry described this fat as "one of the main ingredients" in the "special sauce" that causes the beneficial effects of the bacterium.

The finding goes hand in hand with the "hygiene hypothesis," initially proposed in 1989 by the British scientist David Strachan. He maintained that our generally sterile modern world prevents children from being exposed to certain microorganisms, resulting in compromised immune systems and greater incidences of asthma and allergies.

Contemporary research fine-tuned the hypothesis, finding that not interacting with so-called "old friends" or helpful microbes in the soil and the environment, rather than the ones that cause illnesses, is what's detrimental. In particular, our mental health could be at stake.

"The idea is that as humans have moved away from farms and an agricultural or hunter-gatherer existence into cities, we have lost contact with organisms that served to regulate our immune system and suppress inappropriate inflammation," explained Lowry. "That has put us at higher risk for inflammatory disease and stress-related psychiatric disorders."

University of Colorado Boulder

Christopher Lowry

This is not the first study on the subject from Lowry, who published previous work showing the connection between being exposed to healthy bacteria and mental health. He found that being raised with animals and dust in a rural environment helps children develop more stress-proof immune systems. Such kids were also likely to be less at risk for mental illnesses than people living in the city without pets.

Lowry's other work also pointed out that the soil-based bacterium Mycobacterium vaccae acts like an antidepressant when injected into rodents. It alters their behavior and has lasting anti-inflammatory effects on the brain, according to the press release from the University of Colorado Boulder. Prolonged inflammation can lead to such stress-related disorders as PTSD.

The new study from Lowry and his team identified why that worked by pinpointing the specific fatty acid responsible. They showed that when the 10(Z)-hexadecenoic acid gets into cells, it works like a lock, attaching itself to the peroxisome proliferator-activated receptor (PPAR). This allows it to block a number of key pathways responsible for inflammation. Pre-treating the cells with the acid (or lipid) made them withstand inflammation better.

Lowry thinks this understanding can lead to creating a "stress vaccine" that can be given to people in high-stress jobs, like first responders or soldiers. The vaccine can prevent the psychological effects of stress.

What's more, this friendly bacterium is not the only potentially helpful organism we can find in soil.

"This is just one strain of one species of one type of bacterium that is found in the soil but there are millions of other strains in soils," said Lowry. "We are just beginning to see the tip of the iceberg in terms of identifying the mechanisms through which they have evolved to keep us healthy. It should inspire awe in all of us."

Check out the study published in the journal Psychopharmacology.