Psychopundits: A Consumer Guide

Psychopundits: A Consumer Guide

Right after my recent post on "psychopunditry," I came across signs of this kerfuffle between the writer Jonah Lehrer and the psychologist Christopher Chabris (not to be confused with this other kerfuffle). Short version: Chabris thinks Lehrer exaggerates the reach and meaning of research findings, distorting the science. Their debate, like the National Review piece in my earlier post, turns on an important question: How can experiments and other research from the mind sciences be brought to bear on questions of politics, policy and practical life? Or, to put it less abstractly, if lab experiments suggest that people looking at blue screens are better at a creative task than people who looked at a red screen, should your company paint the office walls blue? An interested bystander will hear contradictory opinions from authoritative sources (Chabris says Lehrer's inference was hooey, the authors of the study Lehrer quoted said it was OK). What to do? How are you, the non-expert reader, to know who is right?


I do believe there are standards you can apply psychopunditry as it flies across your screen. At any rate, these are the ones I ask myself when looking at research papers, and I commend them to you.

1. How big is the effect?

In Chabris' most famous experiment, he and Dan Simons asked their volunteers watch a basketball game and keep track of how many times the ball was passed. During the game, someone in a gorilla suit walked through the court. About half the observers, concentrating on the ball-passing, didn't notice the gorilla.

Half of all participants, 50 percent, is difficult to quarrel with. A lot of experiments, though, don't produce nearly that level of contrast between two groups of people. Similarly, when looking for associations between one phenomenon and another, researchers may be satisfied with a relatively small effect. For example, the correlation between violent videogames and aggressive or anti-social behavior in many studies it's between r=.1 and r=.3, on a scale where 1.0 is the strongest possible correlation and 0 is no relationship at all. That's not a big effect.

2. How likely is this effect to be swamped by other effects?

To say that some mental process is real, researchers need to isolate it. But the act of isolating it may suggest an importance that it does not have in real life. For example, having new blue walls at work may not prompt more creative thinking if you have the same old psychotic boss and the same old noxious fumes from the chemical plant next door. The impact of those stressors will overwhelm the subtler color effect.

This is probably the caution that's most underplayed by both scientists and people like me who are trying to get your attention with our accounts of the research. After all, to capture your eyeballs, we have to say this latest paper is important. That militates against pointing out that a lab effect can be quite real without making much difference in the real world.

Social psychologists have defended small correlations and subtle effects on the grounds that they're in the same range as findings on health risks from pollutants and tobacco smoke. But epidemiological studies were done on large populations, and were frequently replicated, before they were accepted. Many psychology studies are performed on about 100 undergraduates. (The smaller the number of people in an experiment, the greater the chance that the measured differences among them could be accidental.) Or the experiment is performed on 1000 undergraduates but with only 20 different stimuli, which, as Neuroskeptic pointed out this week, also can make noise look significant. Then, the study is never done again. Which brings us to item # 3:

3. How often has the work been replicated?

It's not easy to publish a study announcing that an experiment didn't work out, and many researchers prefer to make discoveries, not point out others' errors. Though the scientific method presumes that experiments will be replicated, many in social science never are. The problem is acute enough that, as Tom Bartlett reported a few weeks ago, a group of psychologists recently launched a project to attempt to reproduce all the results published in several major journals for a year—in part to see just how much of the stuff does replicate.

4. How often has the work been corroborated in other ways?

Labs permit researchers to control situations precisely, but the cost is a certain distance from real circumstances. On the other hand, working on a street or in an office or a store gives psychologists a chance to test theories in real conditions—but those conditions can't be manipulated to eliminate potential confusions. If a psychological phenomenon is real, though, it ought to be visible in both the controlled lab experiment and in life as it is lived. Hence it should be possible to test a proposed cause-and-effect in psychology by comparing lab and field experiments. If both lab and field get the same effect, that's more support for the reality of the phenomenon.

In this analysis, published earlier this year in the journal Perspectives on Psychological Science, the University of Virginia law professor Gregory Mitchell looked at 217 such lab-to-field comparisons across a number of branches of psychology. It's definitely buzz-kill for journalism about social-psychological theories. Of the subdisciplines he examined (social, industrial/organizational, clinical, marketing, developmental) social psychology had the least amount of agreement between lab and field. Social psychology also evinced the most contradiction: Instances where r in the lab was positive and in the field was negative, or vice versa. If lab and field don't agree, or one wing is missing, you should hesitate about any grand or general conclusions.

5. What are we being asked to do?

A surprising result of an experiment done on 54 people can be fascinating. But it's a very slender foundation for major social change. When you see such a mismatch—small-scale experiment, small effect, big sweeping conclusions about how society should respond—then beware. This is the grounds on which I'm skeptical of Lehrer's claim that being surrounded by blue makes us more creative. Repainting a lot of walls will cost time and money. A few experiments with color screens is not enough justification for the implied call to action.

6. Did you keep in mind that no study is an island?

Finally, I try to keep in mind that social science, like any human endeavor, is cumulative enterprise. We know some things about behavior, with reasonable confidence, because a great many people did their work over a long span of time. Similarly, we have great works of literature because many, many hands took their turn at producing poems, plays, novels, songs etc etc. Most work falls between the extremes of perfection and nonsense, and that's exactly what you'd expect and want. Masterpieces—a term that originally described the final work of an apprentice before striking out on his own—are the result of many many efforts that aren't so great. If all work had to be perfect, there would be no work at all.

Put it another way: Theodore Sturgeon once told a convention of SF writers and fans that 90 percent of science fiction is crap. But, he added, no worries—90 percent of everything is crap. The justification for the less-than-perfect is that in doing it, we improve. It is part of the road to excellence.

So, skepticism is in order when you read reports of social-science research. But don't be a dick about it—even a flawed study can be part of a wave the fetches up a real and lasting insight. Try to keep an open mind, and look at the details. And if you want to paint your walls blue, go ahead. Who ever said everything you do needs to be justified by a lab?

Live on Monday: Does the US need one billion people?

What would happen if you tripled the US population? Join Matthew Yglesias and Charles Duhigg at 1pm ET on Monday, September 28.

A landslide is imminent and so is its tsunami

An open letter predicts that a massive wall of rock is about to plunge into Barry Arm Fjord in Alaska.

Image source: Christian Zimmerman/USGS/Big Think
Surprising Science
  • A remote area visited by tourists and cruises, and home to fishing villages, is about to be visited by a devastating tsunami.
  • A wall of rock exposed by a receding glacier is about crash into the waters below.
  • Glaciers hold such areas together — and when they're gone, bad stuff can be left behind.

The Barry Glacier gives its name to Alaska's Barry Arm Fjord, and a new open letter forecasts trouble ahead.

Thanks to global warming, the glacier has been retreating, so far removing two-thirds of its support for a steep mile-long slope, or scarp, containing perhaps 500 million cubic meters of material. (Think the Hoover Dam times several hundred.) The slope has been moving slowly since 1957, but scientists say it's become an avalanche waiting to happen, maybe within the next year, and likely within 20. When it does come crashing down into the fjord, it could set in motion a frightening tsunami overwhelming the fjord's normally peaceful waters .

"It could happen anytime, but the risk just goes way up as this glacier recedes," says hydrologist Anna Liljedahl of Woods Hole, one of the signatories to the letter.

The Barry Arm Fjord

Camping on the fjord's Black Sand Beach

Image source: Matt Zimmerman

The Barry Arm Fjord is a stretch of water between the Harriman Fjord and the Port Wills Fjord, located at the northwest corner of the well-known Prince William Sound. It's a beautiful area, home to a few hundred people supporting the local fishing industry, and it's also a popular destination for tourists — its Black Sand Beach is one of Alaska's most scenic — and cruise ships.

Not Alaska’s first watery rodeo, but likely the biggest

Image source: whrc.org

There have been at least two similar events in the state's recent history, though not on such a massive scale. On July 9, 1958, an earthquake nearby caused 40 million cubic yards of rock to suddenly slide 2,000 feet down into Lituya Bay, producing a tsunami whose peak waves reportedly reached 1,720 feet in height. By the time the wall of water reached the mouth of the bay, it was still 75 feet high. At Taan Fjord in 2015, a landslide caused a tsunami that crested at 600 feet. Both of these events thankfully occurred in sparsely populated areas, so few fatalities occurred.

The Barry Arm event will be larger than either of these by far.

"This is an enormous slope — the mass that could fail weighs over a billion tonnes," said geologist Dave Petley, speaking to Earther. "The internal structure of that rock mass, which will determine whether it collapses, is very complex. At the moment we don't know enough about it to be able to forecast its future behavior."

Outside of Alaska, on the west coast of Greenland, a landslide-produced tsunami towered 300 feet high, obliterating a fishing village in its path.

What the letter predicts for Barry Arm Fjord

Moving slowly at first...

Image source: whrc.org

"The effects would be especially severe near where the landslide enters the water at the head of Barry Arm. Additionally, areas of shallow water, or low-lying land near the shore, would be in danger even further from the source. A minor failure may not produce significant impacts beyond the inner parts of the fiord, while a complete failure could be destructive throughout Barry Arm, Harriman Fiord, and parts of Port Wells. Our initial results show complex impacts further from the landslide than Barry Arm, with over 30 foot waves in some distant bays, including Whittier."

The discovery of the impeding landslide began with an observation by the sister of geologist Hig Higman of Ground Truth, an organization in Seldovia, Alaska. Artist Valisa Higman was vacationing in the area and sent her brother some photos of worrying fractures she noticed in the slope, taken while she was on a boat cruising the fjord.

Higman confirmed his sister's hunch via available satellite imagery and, digging deeper, found that between 2009 and 2015 the slope had moved 600 feet downhill, leaving a prominent scar.

Ohio State's Chunli Dai unearthed a connection between the movement and the receding of the Barry Glacier. Comparison of the Barry Arm slope with other similar areas, combined with computer modeling of the possible resulting tsunamis, led to the publication of the group's letter.

While the full group of signatories from 14 organizations and institutions has only been working on the situation for a month, the implications were immediately clear. The signers include experts from Ohio State University, the University of Southern California, and the Anchorage and Fairbanks campuses of the University of Alaska.

Once informed of the open letter's contents, the Alaska's Department of Natural Resources immediately released a warning that "an increasingly likely landslide could generate a wave with devastating effects on fishermen and recreationalists."

How do you prepare for something like this?

Image source: whrc.org

The obvious question is what can be done to prepare for the landslide and tsunami? For one thing, there's more to understand about the upcoming event, and the researchers lay out their plan in the letter:

"To inform and refine hazard mitigation efforts, we would like to pursue several lines of investigation: Detect changes in the slope that might forewarn of a landslide, better understand what could trigger a landslide, and refine tsunami model projections. By mapping the landslide and nearby terrain, both above and below sea level, we can more accurately determine the basic physical dimensions of the landslide. This can be paired with GPS and seismic measurements made over time to see how the slope responds to changes in the glacier and to events like rainstorms and earthquakes. Field and satellite data can support near-real time hazard monitoring, while computer models of landslide and tsunami scenarios can help identify specific places that are most at risk."

In the letter, the authors reached out to those living in and visiting the area, asking, "What specific questions are most important to you?" and "What could be done to reduce the danger to people who want to visit or work in Barry Arm?" They also invited locals to let them know about any changes, including even small rock-falls and landslides.

Learn innovation with 3-star Michelin chef Dominique Crenn

Dominique Crenn, the only female chef in America with three Michelin stars, joins Big Think Live.

Big Think LIVE

Having been exposed to mavericks in the French culinary world at a young age, three-star Michelin chef Dominique Crenn made it her mission to cook in a way that is not only delicious and elegant, but also expressive, memorable, and true to her experience.

Keep reading Show less

Ultracold gas exhibits bizarre quantum behavior

New experiments find weird quantum activity in supercold gas.

Credit: Pixabay
Surprising Science
  • Experiments on an ultracold gas show strange quantum behavior.
  • The observations point to applications in quantum computing.
  • The find may also advance chaos theory and explain the butterfly effect.
  • Keep reading Show less

    3 cognitive biases perpetuating racism at work — and how to overcome them

    Researchers say that moral self-licensing occurs "because good deeds make people feel secure in their moral self-regard."

    Photo by Christina @ wocintechchat.com on Unsplash
    Personal Growth

    Books about race and anti-racism have dominated bestseller lists in the past few months, bringing to prominence authors including Ibram Kendi, Ijeoma Oluo, Reni Eddo-Lodge, and Robin DiAngelo.

    Keep reading Show less
    Scroll down to load more…
    Quantcast