When Studies Prove That Studies Don't Prove Things ...
Do science journalists have weird psychic powers? You might think so, given the near simultaneity of publications this fall on the touchy theme of studies that don't really prove what they're supposed to have proved.
David Berreby is the author of "Us and Them: The Science of Identity." He has written about human behavior and other science topics for The New Yorker, The New York Times Magazine, Slate, Smithsonian, The New Republic, Nature, Discover, Vogue and many other publications. He has been a Visiting Scholar at the University of Paris, a Science Writing Fellow at the Marine Biological Laboratory, a resident at Yaddo, and in 2006 was awarded the Erving Goffman Award for Outstanding Scholarship for the first edition of "Us and Them." David can be found on Twitter at @davidberreby and reached by email at david [at] davidberreby [dot] com.
Do science journalists have weird psychic powers? You might think so, given the near simultaneity of publications this fall on the touchy theme of studies that don't really prove what they're supposed to have proved. Last month, The Atlantic Monthly published David H. Freedman's profile of Stanford's John Ioannidis, who disbelieves the results of almost all published biomedical research. Three days ago, yours truly explained why four psychologists think much the same about many papers in their field. And yesterday The New Yorker brought out this brilliant piece by Jonah Lehrer, which describes why there's uneasiness about experimental findings in many disciplines. (If you don't subscribe, you'll have to shell out money for a copy of the magazine to read the whole piece. It's worth it.)
Why should you pay up? Because experimental proof is the sacred text of our secular society. Many of us drop research findings into conversation the way, in his day, Cotton Mather quoted Scripture. Well, of course men evolved to be more promiscuous than women. Remember that experiment at the University of Texas? You bet stereotype threat is real—let me tell you about this Harvard study I read about. In normal talk, experiments convince us for the same reason that Leviticus helped Mather win an argument: They are satisfying stories, they seem to state universal truths, and they have the authority of wisdom and power. Of course exercise is good for us. This study showed that when people jog just 30 minutes a week …
Within the wise and powerful institution of science, though, experiments are supposed to be convincing logically, not rhetorically. Tell all the anecdotes you like, the data come out one way if your hypothesis is correct, and a different way if, instead, the "null hypothesis" is true. What is the likelihood of your numbers if the null hypothesis were right? Commonly, if that probability is less than five percent, you have a "significant" result to publish. Your hopes, fears, fads, and ambitions can't affect that.
Such is the theory. By getting a number of scientists to speak honestly about their doubts and frustrations, Lehrer sketches a much more ambiguous reality. It's one in which many experimentally established "truths" fade with time, for instance. (He's not describing the overthrow of one theory by another, which is supposed to happen in science; rather, he's talking about experimental effects that looked solid and then seem to fade away in later experiments—"as if nature gave me this great result, and then tried to take it back," the psychologist Jonathan Schooler told him.)
The unease runs wide and deep through many fields, and it seems to be about the same issues everywhere: Statistics that are manipulated to get a "significant" result; biases affecting everything from the way the data are collected to what gets published to what's even recorded in the notebook.
What is to be done? Some scientists sound as if they think the problem is laxness and lack of interest in statistics. If that's so, many fields need a general tightening up on standards and math. However, you could make the case that in some areas, the problem is exactly the opposite. For example: If psychology isn't physics, Paul Rozin argues here, then it will never come into its own until it ceases trying to act like physics. In some cases, an argument about reliability is really an argument about the nature and purpose of a discipline.
All in all, an interesting autumn for people interested in epistemology. (We're used to getting no respect, what with our editors all the time pushing us for "the bottom line" and "the [note theological metaphor] lay reader's version.") Kudos to Lehrer for collecting scattered signs of soul-searching and making them into a big, important picture.
Journaling can help you materialize your ambitions.
- Organizing your thoughts can help you plan and achieve goals that might otherwise seen unobtainable.
- The Bullet Journal method, in particular, can reduce clutter in your life by helping you visualize your future.
- One way to view your journal might be less of a narrative and more of a timeline of decisions.
Progressive America would be half as big, but twice as populated as its conservative twin.
- America's two political tribes have consolidated into 'red' and 'blue' nations, with seemingly irreconcilable differences.
- Perhaps the best way to stop the infighting is to go for a divorce and give the two nations a country each
- Based on the UN's partition plan for Israel/Palestine, this proposal provides territorial contiguity and sea access to both 'red' and 'blue' America
New research links urban planning and political polarization.
- Canadian researchers find that excessive reliance on cars changes political views.
- Decades of car-centric urban planning normalized unsustainable lifestyles.
- People who prefer personal comfort elect politicians who represent such views.
SMARTER FASTER trademarks owned by The Big Think, Inc. All rights reserved.