Carl Sagan’s baloney detection kit: How to improve your critical thinking skills

Think you detect some grade-A baloney? Here's how you can tell for sure.

Michael Shermer: Back in the late '90s we introduced the Baloney Detection Kit, inspired by Carl Sagan’s 'Demon-Haunted World' where he had a chapter on the Baloney Detection Kit. He had his set of questions; I kind of developed my own because I started encountering other people that disagreed with me, you know, “we never went to the moon” people, conspiracy people, whatever, and I thought okay so: How do we know—if I don’t know what’s coming down the pike in ten years from now, if I am going to teach my students how to think critically, what are the key points, like just basic questions they can ask?

So it begins with one: how reliable is the source of the claim? Here’s the claim, how reliable is it? What’s the evidence for it? What’s the quality of the evidence? Where does it come from? Who said that? Is this some fake news alternative site thing or is it the Wall Street Journal or The New York Times? I mean, the source really matters.

Has anyone tried to disprove the claim? This is super important because everybody thinks they’re right and every website has testimonials about this product or that idea; the question is not what do your supporters think but what do the people who don’t agree with you think? Because that’s what I want to know.

Has anyone run experiments to try to disprove your theory? In science, this is as basic as it gets. Karl Popper called this the principle of falsification, that is we can’t ever prove a theory correct, but we can disprove it by having an experiment that shows it’s wrong.

If you can’t falsify it, what are you really doing? And my favorite story on this, by the way—let me just have a little sidetrack here from Carl Sagan, he’s got this great little section in his book 'Demon-Haunted World': “There is a dragon in my garage. I have a dragon in my garage. Do you want to see it? Let me show you.” So I pull up the garage door I go, “Look. Can you see the dragon?” And you look in there and you go, “I don’t see anything.”

“Oh, sorry, this is an invisible dragon.”

“An invisible dragon?”

“Yeah, yeah he’s invisible.”

“Well, what if we put some flour on the ground and then we’ll get the footprints of the dragon.”

“Well, no, see, this is a special dragon that hovers above the ground, it floats. It’s an invisible floating dragon.”

“An invisible floating dragon. Okay. Wait, I have some infrared cameras here we can detect the heat of the dragon.”

”No, see this is a cold-blooded dragon. It doesn’t give off any heat.”

“What about the fire? We can detect the fire that the dragon spits out.”

“No, it spits out cold fire.”

You see the problem? If there’s no way for me to falsify that there’s a dragon there, what’s the difference between an invisible floating heatless dragon and no dragon at all? None.

And of course we can apply this to god or any other supernatural/paranormal-type phenomenon. If I can’t debunk it, if I can’t falsify it, if there’s no way to test it, then how will we ever know it’s true?

This is the core of the Baloney Detection Kit. We have to be able to get to whether it’s true or not in some way, so it’s not just my opinion versus your opinion and we shout at each other. 

Then we want to know: does the claimant’s personal belief somehow enter in? Because of course we all have personal opinions and beliefs about things; my politics, my religion, my ideology can influence me. It doesn’t make it wrong, but it’s good to know if somebody has an agenda. So when you watch Fox News you know that they have an agenda for sure, or there are other sources on the left that have a liberal agenda, maybe NPR, who knows. But you see it’s good to know that just in case, so when you hear the fact you go, “Well maybe, but I know this guy has an agenda.” So that’s the kind of thing. 

Does the new idea being proposed account for the same amount of information that the old idea does and some of the new anomalies that the old idea can’t explain? So people offer theories, so-called alternative theories of physics, for example, and they always send them to me going, “Hey listen, I’m not good at math but if you help me with the math I’ll share the Nobel Prize with you.”

…Right. I don’t do math and physics so you might take it to the local high school physics teacher before you announce that you’ve made the greatest discovery since Newton and Einstein. The problem with that is: the current theories do pretty well at explaining most everything. Not everything, so there are always anomalies we can’t explain so maybe there’s some new theory coming down the pike that might account for all the old theories, explanations, and the new anomalies that the old theory can’t explain. Maybe. But again, we’ve got to be able to test it first.

So those are the kinds of things. Does the claimant play by the rules of science, the rules of the field that you’re in, for example? Again, these alternative physics guys come to me. Don’t come to me, I’m not a physicist! Did you at least ask the local high school physics teacher if this makes any sense? Because if you have no training, you don’t know all the mistakes that people in the past have already made to get to where they are now, and if you’re starting here without that background, you’re going to make lots of mistakes.

So these are the sorts of things that any good baloney detector should know. I call this Skepticism 101, it's the course I teach at Chapman University to incoming freshmen; it’s a critical thinking course, but it’s really just how to detect baloney.

And not everything is baloney, some things turn out to be true like the theory of evolution, the theory of the Big Bang, germ theory of disease, plate tectonics in geology—these are things that were once radically heretical, and now they’re accepted. How did that happen? It happened because they have evidence, and that’s what you need.

In 1995, just a few months before his death, astrophysicist Carl Sagan published The Demon-Haunted World: Science as a Candle in the Dark. In that book, he wrote a chapter called 'The Fine Art of Baloney Detection', and from it sprang what skeptics call the 'baloney detection kit'. This is a set of tools for critical thinking that has continued to develop since Sagan's death, 22 years ago. Here, skeptic and science writer Michael Shermer explains key lessons from Sagan, and from his own college freshman course 'Skepticism 101', where teaches students ten basic questions that will help them debunk untruths, and call out baloney when they see it.


1. How reliable is the source of the claim?

2. Does the source make similar claims?

3. Have the claims been verified by somebody else?

4. Does this fit with the way the world works?

5. Has anyone tried to disprove the claim?

6. Where does the preponderance of evidence point?

7. Is the claimant playing by the rules of science?

8. Is the claimant providing positive evidence?

9. Does the new theory account for as many phenomena as the old theory?

10. Are personal beliefs driving the claim?

Michael Shermer's new book is Heavens on Earth: The Scientific Search for the Afterlife, Immortality, and Utopia.

Why we mustn’t forget the effects of climate change on mental health

Suicide rates in Puerto Rico have risen by a third since Hurricane Maria.

Jessica Rinaldi/The Boston Globe via Getty Images
Mind & Brain

When Hurricane Maria made landfall on Puerto Rico, it didn't just flatten houses and flood hospitals – it plunged the island into a darkness that many islanders have yet to emerge from, both literally and metaphorically.

Keep reading Show less

How do 80-year-old 'super-agers' have the brains of 20-somethings?

Most elderly individuals' brains degrade over time, but some match — or even outperform — younger individuals on cognitive tests.

Mind & Brain
  • "Super-agers" seem to escape the decline in cognitive function that affects most of the elderly population.
  • New research suggests this is because of higher functional connectivity in key brain networks.
  • It's not clear what the specific reason for this is, but research has uncovered several activities that encourage greater brain health in old age.

At some point in our 20s or 30s, something starts to change in our brains. They begin to shrink a little bit. The myelin that insulates our nerves begins to lose some of its integrity. Fewer and fewer chemical messages get sent as our brains make fewer neurotransmitters.

As we get older, these processes increase. Brain weight decreases by about 5 percent per decade after 40. The frontal lobe and hippocampus — areas related to memory encoding — begin to shrink mainly around 60 or 70. But this is just an unfortunate reality; you can't always be young, and things will begin to break down eventually. That's part of the reason why some individuals think that we should all hope for a life that ends by 75, before the worst effects of time sink in.

But this might be a touch premature. Some lucky individuals seem to resist these destructive forces working on our brains. In cognitive tests, these 80-year-old "super-agers" perform just as well as individuals in their 20s.

Just as sharp as the whippersnappers

To find out what's behind the phenomenon of super-agers, researchers conducted a study examining the brains and cognitive performances of two groups: 41 young adults between the ages of 18 and 35 and 40 older adults between the ages of 60 and 80.

First, the researchers administered a series of cognitive tests, like the California Verbal Learning Test (CVLT) and the Trail Making Test (TMT). Seventeen members of the older group scored at or above the mean scores of the younger group. That is, these 17 could be considered super-agers, performing at the same level as the younger study participants. Aside from these individuals, members of the older group tended to perform less well on the cognitive tests. Then, the researchers scanned all participants' brains in an fMRI, paying special attention to two portions of the brain: the default mode network and the salience network.

The default mode network is, as its name might suggest, a series of brain regions that are active by default — when we're not engaged in a task, they tend to show higher levels of activity. It also appears to be very related to thinking about one's self, thinking about others, as well as aspects of memory and thinking about the future.

The salience network is another network of brain regions, so named because it appears deeply linked to detecting and integrating salient emotional and sensory stimuli. (In neuroscience, saliency refers to how much an item "sticks out"). Both of these networks are also extremely important to overall cognitive function, and in super-agers, the activity in these networks was more coordinated than in their peers.

Default Mode Network

Wikimedia Commons

An image of the brain highlighting the regions associated with the default mode network.

How to ensure brain health in old age

While prior research has identified some genetic influences on how "gracefully" the brain ages, there are likely activities that can encourage brain health. "We hope to identify things we can prescribe for people that would help them be more like a superager," said Bradford Dickerson, one of the researchers in this study, in a statement. "It's not as likely to be a pill as more likely to be recommendations for lifestyle, diet, and exercise. That's one of the long-term goals of this study — to try to help people become superagers if they want to."

To date, there is some preliminary evidence of ways that you can keep your brain younger longer. For instance, more education and a cognitively demanding job predicts having higher cognitive abilities in old age. Generally speaking, the adage of "use it or lose it" appears to hold true; having a cognitively active lifestyle helps to protect your brain in old age. So, it might be tempting to fill your golden years with beer and reruns of CSI, but it's unlikely to help you keep your edge.

Aside from these intuitive ways to keep your brain healthy, regular exercise appears to boost cognitive health in old age, as Dickinson mentioned. Diet is also a protective factor, especially for diets delivering omega-3 fatty acids (which can be found in fish oil), polyphenols (found in dark chocolate!), vitamin D (egg yolks and sunlight), and the B vitamins (meat, eggs, and legumes). There's also evidence that having a healthy social life in old age can protect against cognitive decline.

For many, the physical decline associated with old age is an expected side effect of a life well-lived. But the idea that our intellect will also degrade can be a much scarier reality. Fortunately, the existence of super-agers shows that at the very least, we don't have to accept cognitive decline without a fight.


Sign of the times: School designed to limit impact of mass shootings

With little progress on other avenues to preventing mass shootings, one firm has employed architecture to save students.

Image source: TowerPinkster
Politics & Current Affairs
  • A school in Michigan is being remodeled in a way to minimize the effect of a shooter should the worst happen.
  • It features limited sight lines, bullet proof windows, and doors that can be locked at the push of a button.
  • Some research casts doubt on how effective the plans will actually be.
Keep reading Show less