What is Big Think?  

We are Big Idea Hunters…

We live in a time of information abundance, which far too many of us see as information overload. With the sum total of human knowledge, past and present, at our fingertips, we’re faced with a crisis of attention: which ideas should we engage with, and why? Big Think is an evolving roadmap to the best thinking on the planet — the ideas that can help you think flexibly and act decisively in a multivariate world.

A word about Big Ideas and Themes — The architecture of Big Think

Big ideas are lenses for envisioning the future. Every article and video on bigthink.com and on our learning platforms is based on an emerging “big idea” that is significant, widely relevant, and actionable. We’re sifting the noise for the questions and insights that have the power to change all of our lives, for decades to come. For example, reverse-engineering is a big idea in that the concept is increasingly useful across multiple disciplines, from education to nanotechnology.

Themes are the seven broad umbrellas under which we organize the hundreds of big ideas that populate Big Think. They include New World Order, Earth and Beyond, 21st Century Living, Going Mental, Extreme Biology, Power and Influence, and Inventing the Future.

Big Think Features:

12,000+ Expert Videos


Browse videos featuring experts across a wide range of disciplines, from personal health to business leadership to neuroscience.

Watch videos

World Renowned Bloggers


Big Think’s contributors offer expert analysis of the big ideas behind the news.

Go to blogs

Big Think Edge


Big Think’s Edge learning platform for career mentorship and professional development provides engaging and actionable courses delivered by the people who are shaping our future.

Find out more

How to Think Critically V

December 29, 2007, 12:55 PM

Double-Blind Tests

In a post back in October, I dissected the claims of a spam e-mail that landed in my inbox to promote the "Detox Box", an expensive piece of snake oil claimed to be able to cure any disease through the power of pseudoscience. As is usual in these matters, an offended true believer showed up in the comments to proclaim her faith in the device:

The Detox Box works for me and I wouldn't go without it. It is my personal results. I don't need a double blind study to know that it works.

In fact, as I and others pointed out, you do need a double-blind test to know that this or any other medical treatment works. Personal testimony is not sufficient, and in fact, it is among the least reliable kinds of evidence - with good reason.

It should be no surprise that people often interpret the world according to their expectations. When we expect something to happen, it often does happen. For example, when we undergo a treatment which we expect to help with an illness, most people do report feeling better afterwards. Along the same lines, doctors who are administering a treatment which they expect to help their patient will tend to notice signs of improvement thereafter. This is just the well-known placebo effect, but what's not as well-known is just how powerful the placebo effect can be. For a dramatic example of the mind's power to influence the body, I give you pseudocyesis - psychosomatic pregnancy.

"Every sign and symptom of pregnancy has been recorded in these patients except for three: You don't hear heart tones from the fetus, you don't see the fetus on ultrasound, and you don't get a delivery," Dr. Paulman said.

"Every sign and symptom of pregnancy" is literally accurate. Women with pseudocyesis see their bellies swell, exactly as you'd expect in a late-term pregnancy. Their breasts become enlarged and produce milk; their menstrual cycle stops; their hormone levels are elevated; they suffer food cravings and morning sickness. And when the time arrives, they have contractions and labor pains. So convincing are these signs that pseudocyesis often fools even experienced obstetricians, and sometimes goes undetected right up until the moment of labor, when it finally becomes obvious there's nothing to be delivered. Some women have undergone unnecessary Caesarian sections because of it. (The usual explanation is that it occurs in women who desperately want to conceive children, but are unable to for some reason. It's rarer these days thanks to ultrasound and other fetal imaging technologies which can provide the woman incontrovertible proof that she isn't pregnant.)

Human beings are fallible; of that there can be no doubt. When studying things that can't easily be quantified, we're all too susceptible to bias, both for and against. Even if we have the most honest intentions in the world, unconscious presuppositions may subtly skew our results without our even being aware of it. (58% of corporate CEOs are six feet tall or more; in the general population it's only 15%. Apparently most people subconsciously think that tall people make better leaders.)

How do we compensate for our own biases? We can do it with a scientific technique that's considered the gold standard in medicine and other fields of diagnosis: the double-blind test.

The double-blind test is fundamentally a very simple concept. To conduct a double-blind test, you start by gathering a group of volunteers. Each volunteer is assigned - randomly, which is important - to either the experimental group or the control group. Those in the experimental group receive the treatment that is being tested, while those in the control group receive a placebo that is identical to all outward appearances, but completely inert. The experimenters rate the progress of both groups and, at the conclusion of the test, crunch the numbers to determine whether the experimental group shows a statistically significant difference from the control group.

The crucial point of a double-blind test is that the volunteers don't know whether they're in the experimental or the control group, and neither do the scientists who are administering the treatment and observing the results. Generally, during the experiment, that knowledge is held by a third party who does not otherwise participate. Once the experiment is done and the results are in, the blinding is broken, and only then do the experimenters sift the data to determine whether there's a significant difference between the two groups. (Sometimes this has to be modified, such as in double-blind studies of surgical treatments that compare the real operation with "sham surgery" which makes an incision but nothing more. In this case, one set of doctors performs the surgery and a different set, blinded to who got which treatment, evaluates the results.)

The point of a double-blind test is that it is designed so that no one's beliefs, biases or expectations can influence the results. Since volunteers are assigned randomly to one of the two groups, there's no possibility that, say, an unscrupulous medical researcher could assign the sicker volunteers to the control group and the healthier ones to the experimental group to make a treatment appear more effective. Since the volunteers don't know which group they're in, their expectations about what will happen can't cause a placebo effect that would throw off the results. (More accurately, both the experimental and control groups should experience the placebo effect to the same degree. Thus, that effect can be statistically subtracted out, to see whether any residual effect is left over that could only be attributed to the efficacy of the treatment.) And since the researchers don't know either, their expectations about the efficacy of their own treatment can't skew their data collection.

In experiments that are not double-blind, the results are often affected by the participants' or the researchers' advance knowledge. For example, in the classic police lineup technique for identifying suspects, there are concerns that the detective present may purposefully or inadvertently prompt the witness. Double-blind lineups, where the detective overseeing does not know who the suspect is, have been shown to produce fewer false identifications.

A famous case where double-blind experiments revealed the truth was Clever Hans, the horse who could supposedly solve complex mathematics (giving the answers by tapping his hoof). In reality, he was responding to subtle, unconscious cues from the observers who were present and who knew the answer to the question. This fact was eventually exposed by the psychologist Oskar Pfungst, who conducted a double-blind test of the horse's abilities by asking questions whose answers none of the onlookers knew.

When performed properly, the double-blind test is an invaluable tool for weeding out pseudoscience and falsehood of every kind. Witness the following immortal reply from a practitioner of "applied kinesiology":

When these results were announced, the head chiropractor turned to me and said, "You see, that is why we never do double-blind testing anymore. It never works!"

Although it may not "work" to confirm presuppositions, double-blind testing most definitely does work to tell truth apart from falsehood. As such, it's no surprise that advocates of delusion avoid it or denigrate it at every turn.

Other posts in this series:


How to Think Critically V

Newsletter: Share: