Just Post It

Yesterday, the journal Psychological Science published a paper by Uri Simonsohn online (it's behind a paywall but you can access a free earlier version here). The paper calls on researchers to post their data so that the entire research process is transparent. There may be some reasonable exceptions to this rule, but it's hard to argue that it should not be the default. Uri was kind enough to share a draft of his paper with me last summer and I wrote about it on my personal blog. That article is re-posted below.


Many of the potential reforms that psychological science could introduce to improve practices in the field are controversial -- their benefits are not always certain and they often carry substantial costs. I don't think that a default rule to post the data alongside one's publication is one of these; it's a sensible step that should be widely adopted.

-------------------------------------------

(First posted July 22nd, 2012)

A quick update on the Simonsohn's Just Post It paper – Friday’s post focused mostly on the steps Simonsohn took to avoid making false accusations and provided a link to the newly available paper so people could check out the details for themselves. I thought it was worth adding a little bit of explanation for people who are curious about the details. Without going into great detail – people who are interested can obviously read the paper themselves – I think Simonsohn’s analysis of Smeesters’ study that used a Willingness to Pay (WTP) measure is easy to understand and illustrates the assertion that Smeesters is very unlikely to have just dropped a few participants who didn’t follow directions (or even eliminated a handful of inconvenient data points) as he claims to have done. It also makes a strong case for Simonsohn's call to post raw data. Note that the analysis isn't representative of the other analyses Simonsohn conducted.

Willingness to Pay is a measure psychologists adopted from economists. Participants answer the maximum amount they would be willing to pay for some item, like a bottle of wine or a toaster, providing a more concrete estimate of how much they value it than, for instance, reporting how much they like something on a seven point scale. Smeesters used a WTP measure in one of the studies Simonsohn analyzed, asking participants how much they would be willing to pay for a t-shirt. Simonsohn explains why the data looks suspicious:

A striking pattern in the WTP data of the Smeesters study is the low frequency of valuations expressed as multiples of $5. Because people often round up or down when answering pricing questions, the percentage of such valuations is typically much higher than the rate expected if people answered randomly.

In other words, WTP data generated by real people should have a lot of data points that are multiples of $5 because people tend to round -- they're more likely to answer $10 or $25 than $9.87 or $25.32. Smeesters’ data doesn’t reflect this tendency and therefore appear not to have been reported by actual subjects. To confirm this suspicion, Simonsohn obtained the data from a number of other studies that used the WTP measure in addition to re-running Smeesters’ study himself. In each of 29 studies, at least 50% of WTP valuations were multiples of $5; in the two studies conducted by Smeesters that proportion was closer to the 20% expected by chance. This can be seen vividly in the graph below: In the graph, Smeesters' studies clearly stand out from the rest, more closely matching what one would expect to see if the valuations were chosen randomly (horizontal red line) than the trend line generated by all the other WTP studies. Note that the upward slope on that line reflects the fact that people become even more likely to round as valuations increase in size. [pullquote align="left|center|right" textalign="left|center|right" width="30%"]this evidence is a strong indication that Smeesters' data was not generated by actual subjects[/pullquote] By itself, this evidence is a strong indication that Smeesters' data was not generated by actual subjects, but Simonsohn's paper goes well beyond this analysis. In fact, this was a follow-up to other analyses that had initially aroused Simonsohn's suspicion, which relied solely on the data Smeesters reported in published papers. The WTP analysis makes a strong case for why, as Simonsohn argues, raw data should be made publicly available. With access to the raw data, it becomes much more difficult for someone to fabricate their data because they need to ensure its internal coherence. Examining the data in the WTP study not only reveals that it does not appear to have been generated by actual subjects because valuations in multiples of $5 are not over-represented as one would expect, it also shows a very strange pattern of within-subject correlations.

Lastly, it's worth briefly addressing the concern that, in publishing this paper, Simonsohn is providing a manual for would-be fraudsters. I don't find this concern compelling for at least two reasons. First, the analyses in Simonsohn's paper do not come close to exhausting the analyses that could reveal fraud. This is why it's particularly important that publishing raw data becomes the default -- it's very hard to fabricate data that truly reflects what real data would look like. And second, if we are not transparent in the methods we use to detect fraud then the whole enterprise risks appearing like a witch hunt, thereby making researchers less open to much-needed reforms. As I argue in this post about reforming social psychology, most psychologists are honest researchers with good intentions and respect for truth and the scientific process. It's critical that we are transparent in the methods we use to detect fraud to assure people that any accusations of fraud have merit, that they have been exhaustively confirmed and re-confirmed, and that they are not being used like "a medieval instrument of torture" on researchers who didn't realize they were doing anything wrong.

-------------------------------------------

You can also read an interview that Ed Yong did with Uri for Nature, as well as Ed's blog post on the subject. There was also a profile of Uri in the Atlantic Monthly last year by Christopher Shea. You can also check out my other writing on the subject; I'd start with Crimes and Misdemeanors.

'Upstreamism': Your zip code affects your health as much as genetics

Upstreamism advocate Rishi Manchanda calls us to understand health not as a "personal responsibility" but a "common good."

Sponsored by Northwell Health
  • Upstreamism tasks health care professionals to combat unhealthy social and cultural influences that exist outside — or upstream — of medical facilities.
  • Patients from low-income neighborhoods are most at risk of negative health impacts.
  • Thankfully, health care professionals are not alone. Upstreamism is increasingly part of our cultural consciousness.
Keep reading Show less

Elizabeth Warren's plan to forgive student loan debt could lead to an economic boom

A plan to forgive almost a trillion dollars in debt would solve the student loan debt crisis, but can it work?

Photo credit: Drew Angerer / Getty Images
Politics & Current Affairs
  • Sen. Elizabeth Warren has just proposed a bold education reform plan that would forgive billions in student debt.
  • The plan would forgive the debt held by more than 30 million Americans.
  • The debt forgiveness program is one part of a larger program to make higher education more accessible.
Keep reading Show less

Yale scientists restore brain function to 32 clinically dead pigs

Researchers hope the technology will further our understanding of the brain, but lawmakers may not be ready for the ethical challenges.

Still from John Stephenson's 1999 rendition of Animal Farm.
Surprising Science
  • Researchers at the Yale School of Medicine successfully restored some functions to pig brains that had been dead for hours.
  • They hope the technology will advance our understanding of the brain, potentially developing new treatments for debilitating diseases and disorders.
  • The research raises many ethical questions and puts to the test our current understanding of death.

The image of an undead brain coming back to live again is the stuff of science fiction. Not just any science fiction, specifically B-grade sci fi. What instantly springs to mind is the black-and-white horrors of films like Fiend Without a Face. Bad acting. Plastic monstrosities. Visible strings. And a spinal cord that, for some reason, is also a tentacle?

But like any good science fiction, it's only a matter of time before some manner of it seeps into our reality. This week's Nature published the findings of researchers who managed to restore function to pigs' brains that were clinically dead. At least, what we once thought of as dead.

What's dead may never die, it seems

The researchers did not hail from House Greyjoy — "What is dead may never die" — but came largely from the Yale School of Medicine. They connected 32 pig brains to a system called BrainEx. BrainEx is an artificial perfusion system — that is, a system that takes over the functions normally regulated by the organ. The pigs had been killed four hours earlier at a U.S. Department of Agriculture slaughterhouse; their brains completely removed from the skulls.

BrainEx pumped an experiment solution into the brain that essentially mimic blood flow. It brought oxygen and nutrients to the tissues, giving brain cells the resources to begin many normal functions. The cells began consuming and metabolizing sugars. The brains' immune systems kicked in. Neuron samples could carry an electrical signal. Some brain cells even responded to drugs.

The researchers have managed to keep some brains alive for up to 36 hours, and currently do not know if BrainEx can have sustained the brains longer. "It is conceivable we are just preventing the inevitable, and the brain won't be able to recover," said Nenad Sestan, Yale neuroscientist and the lead researcher.

As a control, other brains received either a fake solution or no solution at all. None revived brain activity and deteriorated as normal.

The researchers hope the technology can enhance our ability to study the brain and its cellular functions. One of the main avenues of such studies would be brain disorders and diseases. This could point the way to developing new of treatments for the likes of brain injuries, Alzheimer's, Huntington's, and neurodegenerative conditions.

"This is an extraordinary and very promising breakthrough for neuroscience. It immediately offers a much better model for studying the human brain, which is extraordinarily important, given the vast amount of human suffering from diseases of the mind [and] brain," Nita Farahany, the bioethicists at the Duke University School of Law who wrote the study's commentary, told National Geographic.

An ethical gray matter

Before anyone gets an Island of Dr. Moreau vibe, it's worth noting that the brains did not approach neural activity anywhere near consciousness.

The BrainEx solution contained chemicals that prevented neurons from firing. To be extra cautious, the researchers also monitored the brains for any such activity and were prepared to administer an anesthetic should they have seen signs of consciousness.

Even so, the research signals a massive debate to come regarding medical ethics and our definition of death.

Most countries define death, clinically speaking, as the irreversible loss of brain or circulatory function. This definition was already at odds with some folk- and value-centric understandings, but where do we go if it becomes possible to reverse clinical death with artificial perfusion?

"This is wild," Jonathan Moreno, a bioethicist at the University of Pennsylvania, told the New York Times. "If ever there was an issue that merited big public deliberation on the ethics of science and medicine, this is one."

One possible consequence involves organ donations. Some European countries require emergency responders to use a process that preserves organs when they cannot resuscitate a person. They continue to pump blood throughout the body, but use a "thoracic aortic occlusion balloon" to prevent that blood from reaching the brain.

The system is already controversial because it raises concerns about what caused the patient's death. But what happens when brain death becomes readily reversible? Stuart Younger, a bioethicist at Case Western Reserve University, told Nature that if BrainEx were to become widely available, it could shrink the pool of eligible donors.

"There's a potential conflict here between the interests of potential donors — who might not even be donors — and people who are waiting for organs," he said.

It will be a while before such experiments go anywhere near human subjects. A more immediate ethical question relates to how such experiments harm animal subjects.

Ethical review boards evaluate research protocols and can reject any that causes undue pain, suffering, or distress. Since dead animals feel no pain, suffer no trauma, they are typically approved as subjects. But how do such boards make a judgement regarding the suffering of a "cellularly active" brain? The distress of a partially alive brain?

The dilemma is unprecedented.

Setting new boundaries

Another science fiction story that comes to mind when discussing this story is, of course, Frankenstein. As Farahany told National Geographic: "It is definitely has [sic] a good science-fiction element to it, and it is restoring cellular function where we previously thought impossible. But to have Frankenstein, you need some degree of consciousness, some 'there' there. [The researchers] did not recover any form of consciousness in this study, and it is still unclear if we ever could. But we are one step closer to that possibility."

She's right. The researchers undertook their research for the betterment of humanity, and we may one day reap some unimaginable medical benefits from it. The ethical questions, however, remain as unsettling as the stories they remind us of.

Supreme Court to hear 3 cases on LGBT workplace discrimination

In most states, LGBTQ Americans have no legal protections against discrimination in the workplace.

(Photo by Andres Pantoja/SOPA Images/LightRocket via Getty Images)
Politics & Current Affairs
  • The Supreme Court will decide whether the Civil Rights Act of 1964 also applies to gay and transgender people.
  • The court, which currently has a probable conservative majority, will likely decide on the cases in 2020.
  • Only 21 states and the District of Columbia have passed laws effectively extending the Civil Rights of 1964 to gay and transgender people.
Keep reading Show less