December 11

Analyse

Tuesday’s Big Idea

Today's Big Idea: p-Hacking

When researchers engage in questionable data analysis, there can be deadly consequences. For instance, consider a clinical trial in which a number of test subjects have a heart attack, but that number falls below the accepted threshold of statistical significance, P<0.05.

As Neurobonkers points out today, there are often important numbers "hidden underneath this indication of statistical significance." So how do we sniff out studies that might be manipulating data because, say, the entity conducting the test has a vested interest in the outcome?

You need to look for a pattern. If a disproportionate number of results are close to the P<0.05 threshold, that would suggest there might be something rotten in Denmark. At that point, the raw data of the study in question should be requested and assessed for patterns that indicate p-hacking. This method was developed by Uri Simonhson, who has used the technique to uncover a growing number of cases of research fraud. 

  1. 1 The statistical significance scan...
  2. 2 The Brain Is a Statistical Engine
  3. 3 How to Predict the Unpredictable:...
  4. 4 Re-examining Significant Research...
   
  1. The statistical significance scandal: The standard error of science?

    The statistical significance scandal: The standard error of science?

    The problem of scientists manipulating data in order to achieve statistical significance, labelled p-hacking is incredibly hard to track down due to the fact that the data behind statistical significance is often unavailable for analysis by anyone other than those who did the research and themselves analysed the data.

    Read More…
  2. The Brain Is a Statistical Engine

    The Brain Is a Statistical Engine

    “Your legs couldn’t tell you Newton’s laws, but they still obey them, and of course your brain might not be able to explain the laws of statistics, but it still obeys them.”

    Read More…
  3. How to Predict the Unpredictable: Use Data

    How to Predict the Unpredictable: Use Data

    Knowing what empirical data to look for is the key to confirming an idea—especially when that idea is the premonition of a looming global economic crisis.

    Read More…
  4. Re-examining Significant Research: The Problem of False-Positives

    Re-examining Significant Research: The Problem of False-Positives

    It is remarkably easy to report false-positive findings, or results that support an effect that, in reality, does not (or may not) exist.

    Read More…