When Studies Prove That Studies Don't Prove Things ...

Do science journalists have weird psychic powers? You might think so, given the near simultaneity of publications this fall on the touchy theme of studies that don't really prove what they're supposed to have proved.

Do science journalists have weird psychic powers? You might think so, given the near simultaneity of publications this fall on the touchy theme of studies that don't really prove what they're supposed to have proved. Last month, The Atlantic Monthly published David H. Freedman's profile of Stanford's John Ioannidis, who disbelieves the results of almost all published biomedical research. Three days ago, yours truly explained why four psychologists think much the same about many papers in their field. And yesterday The New Yorker brought out this brilliant piece by Jonah Lehrer, which describes why there's uneasiness about experimental findings in many disciplines. (If you don't subscribe, you'll have to shell out money for a copy of the magazine to read the whole piece. It's worth it.)

Why should you pay up? Because experimental proof is the sacred text of our secular society. Many of us drop research findings into conversation the way, in his day, Cotton Mather quoted Scripture. Well, of course men evolved to be more promiscuous than women. Remember that experiment at the University of Texas? You bet stereotype threat is real—let me tell you about this Harvard study I read about. In normal talk, experiments convince us for the same reason that Leviticus helped Mather win an argument: They are satisfying stories, they seem to state universal truths, and they have the authority of wisdom and power. Of course exercise is good for us. This study showed that when people jog just 30 minutes a week …

Within the wise and powerful institution of science, though, experiments are supposed to be convincing logically, not rhetorically. Tell all the anecdotes you like, the data come out one way if your hypothesis is correct, and a different way if, instead, the "null hypothesis" is true. What is the likelihood of your numbers if the null hypothesis were right? Commonly, if that probability is less than five percent, you have a "significant" result to publish. Your hopes, fears, fads, and ambitions can't affect that.

Such is the theory. By getting a number of scientists to speak honestly about their doubts and frustrations, Lehrer sketches a much more ambiguous reality. It's one in which many experimentally established "truths" fade with time, for instance. (He's not describing the overthrow of one theory by another, which is supposed to happen in science; rather, he's talking about experimental effects that looked solid and then seem to fade away in later experiments—"as if nature gave me this great result, and then tried to take it back," the psychologist Jonathan Schooler told him.)

The unease runs wide and deep through many fields, and it seems to be about the same issues everywhere: Statistics that are manipulated to get a "significant" result; biases affecting everything from the way the data are collected to what gets published to what's even recorded in the notebook.

What is to be done? Some scientists sound as if they think the problem is laxness and lack of interest in statistics. If that's so, many fields need a general tightening up on standards and math. However, you could make the case that in some areas, the problem is exactly the opposite. For example: If psychology isn't physics, Paul Rozin argues here, then it will never come into its own until it ceases trying to act like physics. In some cases, an argument about reliability is really an argument about the nature and purpose of a discipline.

All in all, an interesting autumn for people interested in epistemology. (We're used to getting no respect, what with our editors all the time pushing us for "the bottom line" and "the [note theological metaphor] lay reader's version.") Kudos to Lehrer for collecting scattered signs of soul-searching and making them into a big, important picture.

​There are two kinds of failure – but only one is honorable

Malcolm Gladwell teaches "Get over yourself and get to work" for Big Think Edge.

Big Think Edge
  • Learn to recognize failure and know the big difference between panicking and choking.
  • At Big Think Edge, Malcolm Gladwell teaches how to check your inner critic and get clear on what failure is.
  • Subscribe to Big Think Edge before we launch on March 30 to get 20% off monthly and annual memberships.
Keep reading Show less

Why is 18 the age of adulthood if the brain can take 30 years to mature?

Neuroscience research suggests it might be time to rethink our ideas about when exactly a child becomes an adult.

Mind & Brain
  • Research suggests that most human brains take about 25 years to develop, though these rates can vary among men and women, and among individuals.
  • Although the human brain matures in size during adolescence, important developments within the prefrontal cortex and other regions still take pace well into one's 20s.
  • The findings raise complex ethical questions about the way our criminal justice systems punishes criminals in their late teens and early 20s.
Keep reading Show less

Believe in soulmates? You're more likely to 'ghost' romantic partners.

Does believing in true love make people act like jerks?

Thought Catalog via Unsplash
Sex & Relationships
  • Ghosting, or cutting off all contact suddenly with a romantic partner, is not nice.
  • Growth-oriented people (who think relationships are made, not born) do not appreciate it.
  • Destiny-oriented people (who believe in soulmates) are more likely to be okay with ghosting.
Keep reading Show less

Mini-brains attach to spinal cord and twitch muscles

A new method of growing mini-brains produces some startling results.

(Lancaster, et al)
Surprising Science
  • Researchers find a new and inexpensive way to keep organoids growing for a year.
  • Axons from the study's organoids attached themselves to embryonic mouse spinal cord cells.
  • The mini-brains took control of muscles connected to the spinal cords.
Keep reading Show less