'The first crack in the wall of significance testing'

A major psychology journal has banned the use of the near-universally adopted practice of significance testing, citing recent evidence of the technique's unreliability. What will be the fallout for psychology as a field?

'The first crack in the wall of significance testing'

If you have ever read a piece of psychology research cover to cover, you will almost certainly have witnessed the p-value, the controversial statistical measure that we have discussed at length on this blog previously. Last month, a scientific journal made perhaps the boldest move since journals began opening their doors to open access. The journal, Basic and Applied Social Psychology (BASP), has banned the use of null hypothesis significance testing, a technique used almost universally in psychology research and much scientific research across the board. Some, however, might call this the nuclear option, as when used properly, the p-value can be a useful indicator. Instead of significance testing, the journal will rely on arguably more reliable measures that are often left out of modern psychology research:


"BASP will require strong descriptive statistics, including effect sizes. We also encourage the presentation of frequency or distributional data when this is feasible. Finally, we encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem"

Significance testing is one of the most important, yet most widely misunderstood definitions in science. Over at the excellent Science-Based Medicine blog, Yale clinical neurologist Steven Novella sums up the problem well:

"The p-value was never meant to be the sole measure of whether or not a particular hypothesis is true. Rather it was meant only as a measure of whether or not the data should be taken seriously."

Novella's account refers to an absolutely beautiful Guitar-Hero-meets-Space-Invaders-meets-Tetris-meets-roulette statistical demonstration of the problem by Geoff Cumming, dubbed "The dance of the p-values." If it is not the most inspired stats lesson you've ever had, then I'll eat my hat:

Follow Neurobonkers on TwitterFacebookGoogle+RSS, or join the mailing list. Image Credit: Adapted from original artwork by Shutterstock.

Reference:

Trafimow D. (2014). Editorial, Basic and Applied Social Psychology, 37 (1) 1-2. DOI: http://dx.doi.org/10.1080/01973533.2015.1012991

No, the Yellowstone supervolcano is not ‘overdue’

Why mega-eruptions like the ones that covered North America in ash are the least of your worries.

Ash deposits of some of North America's largest volcanic eruptions.

Image: USGS - public domain
Strange Maps
  • The supervolcano under Yellowstone produced three massive eruptions over the past few million years.
  • Each eruption covered much of what is now the western United States in an ash layer several feet deep.
  • The last eruption was 640,000 years ago, but that doesn't mean the next eruption is overdue.
Keep reading Show less

Smartly dressed: Researchers develop clothes that sense movement via touch

Measuring a person's movements and poses, smart clothes could be used for athletic training, rehabilitation, or health-monitoring.

Technology & Innovation

In recent years there have been exciting breakthroughs in wearable technologies, like smartwatches that can monitor your breathing and blood oxygen levels.

Keep reading Show less

Do you worry too much? Stoicism can help

How imagining the worst case scenario can help calm anxiety.

Stoicism can help overcome anxiety

Credit: OLIVIER DOULIERY via Getty Images
Personal Growth
  • Stoicism is the philosophy that nothing about the world is good or bad in itself, and that we have control over both our judgments and our reactions to things.
  • It is hardest to control our reactions to the things that come unexpectedly.
  • By meditating every day on the "worst case scenario," we can take the sting out of the worst that life can throw our way.
Keep reading Show less
Quantcast