Carl Sagan’s baloney detection kit: How to improve your critical thinking skills
Think you detect some grade-A baloney? Here's how you can tell for sure.
Michael Shermer: Back in the late '90s we introduced the Baloney Detection Kit, inspired by Carl Sagan’s 'Demon-Haunted World' where he had a chapter on the Baloney Detection Kit. He had his set of questions; I kind of developed my own because I started encountering other people that disagreed with me, you know, “we never went to the moon” people, conspiracy people, whatever, and I thought okay so: How do we know—if I don’t know what’s coming down the pike in ten years from now, if I am going to teach my students how to think critically, what are the key points, like just basic questions they can ask?
So it begins with one: how reliable is the source of the claim? Here’s the claim, how reliable is it? What’s the evidence for it? What’s the quality of the evidence? Where does it come from? Who said that? Is this some fake news alternative site thing or is it the Wall Street Journal or The New York Times? I mean, the source really matters.
Has anyone tried to disprove the claim? This is super important because everybody thinks they’re right and every website has testimonials about this product or that idea; the question is not what do your supporters think but what do the people who don’t agree with you think? Because that’s what I want to know.
Has anyone run experiments to try to disprove your theory? In science, this is as basic as it gets. Karl Popper called this the principle of falsification, that is we can’t ever prove a theory correct, but we can disprove it by having an experiment that shows it’s wrong.
If you can’t falsify it, what are you really doing? And my favorite story on this, by the way—let me just have a little sidetrack here from Carl Sagan, he’s got this great little section in his book 'Demon-Haunted World': “There is a dragon in my garage. I have a dragon in my garage. Do you want to see it? Let me show you.” So I pull up the garage door I go, “Look. Can you see the dragon?” And you look in there and you go, “I don’t see anything.”
“Oh, sorry, this is an invisible dragon.”
“An invisible dragon?”
“Yeah, yeah he’s invisible.”
“Well, what if we put some flour on the ground and then we’ll get the footprints of the dragon.”
“Well, no, see, this is a special dragon that hovers above the ground, it floats. It’s an invisible floating dragon.”
“An invisible floating dragon. Okay. Wait, I have some infrared cameras here we can detect the heat of the dragon.”
”No, see this is a cold-blooded dragon. It doesn’t give off any heat.”
“What about the fire? We can detect the fire that the dragon spits out.”
“No, it spits out cold fire.”
You see the problem? If there’s no way for me to falsify that there’s a dragon there, what’s the difference between an invisible floating heatless dragon and no dragon at all? None.
And of course we can apply this to god or any other supernatural/paranormal-type phenomenon. If I can’t debunk it, if I can’t falsify it, if there’s no way to test it, then how will we ever know it’s true?
This is the core of the Baloney Detection Kit. We have to be able to get to whether it’s true or not in some way, so it’s not just my opinion versus your opinion and we shout at each other.
Then we want to know: does the claimant’s personal belief somehow enter in? Because of course we all have personal opinions and beliefs about things; my politics, my religion, my ideology can influence me. It doesn’t make it wrong, but it’s good to know if somebody has an agenda. So when you watch Fox News you know that they have an agenda for sure, or there are other sources on the left that have a liberal agenda, maybe NPR, who knows. But you see it’s good to know that just in case, so when you hear the fact you go, “Well maybe, but I know this guy has an agenda.” So that’s the kind of thing.
Does the new idea being proposed account for the same amount of information that the old idea does and some of the new anomalies that the old idea can’t explain? So people offer theories, so-called alternative theories of physics, for example, and they always send them to me going, “Hey listen, I’m not good at math but if you help me with the math I’ll share the Nobel Prize with you.”
…Right. I don’t do math and physics so you might take it to the local high school physics teacher before you announce that you’ve made the greatest discovery since Newton and Einstein. The problem with that is: the current theories do pretty well at explaining most everything. Not everything, so there are always anomalies we can’t explain so maybe there’s some new theory coming down the pike that might account for all the old theories, explanations, and the new anomalies that the old theory can’t explain. Maybe. But again, we’ve got to be able to test it first.
So those are the kinds of things. Does the claimant play by the rules of science, the rules of the field that you’re in, for example? Again, these alternative physics guys come to me. Don’t come to me, I’m not a physicist! Did you at least ask the local high school physics teacher if this makes any sense? Because if you have no training, you don’t know all the mistakes that people in the past have already made to get to where they are now, and if you’re starting here without that background, you’re going to make lots of mistakes.
So these are the sorts of things that any good baloney detector should know. I call this Skepticism 101, it's the course I teach at Chapman University to incoming freshmen; it’s a critical thinking course, but it’s really just how to detect baloney.
And not everything is baloney, some things turn out to be true like the theory of evolution, the theory of the Big Bang, germ theory of disease, plate tectonics in geology—these are things that were once radically heretical, and now they’re accepted. How did that happen? It happened because they have evidence, and that’s what you need.
In 1995, just a few months before his death, astrophysicist Carl Sagan published The Demon-Haunted World: Science as a Candle in the Dark. In that book, he wrote a chapter called 'The Fine Art of Baloney Detection', and from it sprang what skeptics call the 'baloney detection kit'. This is a set of tools for critical thinking that has continued to develop since Sagan's death, 22 years ago. Here, skeptic and science writer Michael Shermer explains key lessons from Sagan, and from his own college freshman course 'Skepticism 101', where teaches students ten basic questions that will help them debunk untruths, and call out baloney when they see it.
1. How reliable is the source of the claim?
2. Does the source make similar claims?
3. Have the claims been verified by somebody else?
4. Does this fit with the way the world works?
5. Has anyone tried to disprove the claim?
6. Where does the preponderance of evidence point?
7. Is the claimant playing by the rules of science?
8. Is the claimant providing positive evidence?
9. Does the new theory account for as many phenomena as the old theory?
10. Are personal beliefs driving the claim?
Michael Shermer's new book is Heavens on Earth: The Scientific Search for the Afterlife, Immortality, and Utopia.
Once a week.
Subscribe to our weekly newsletter.
As patients approached death, many had dreams and visions of deceased loved ones.
One of the most devastating elements of the coronavirus pandemic has been the inability to personally care for loved ones who have fallen ill.
Research reveals a new evolutionary feature that separates humans from other primates.
- Researchers find a new feature of human evolution.
- Humans have evolved to use less water per day than other primates.
- The nose is one of the factors that allows humans to be water efficient.
A model of water turnover for humans and chimpanzees who have similar fat free mass and body water pools.
Credit: Current Biology
A man's skeleton, found facedown with his hands bound, was unearthed near an ancient ceremonial circle during a high speed rail excavation project.
- A skeleton representing a man who was tossed face down into a ditch nearly 2,500 years ago with his hands bound in front of his hips was dug up during an excavation outside of London.
- The discovery was made during a high speed rail project that has been a bonanza for archaeology, as the area is home to more than 60 ancient sites along the planned route.
- An ornate grave of a high status individual from the Roman period and an ancient ceremonial circle were also discovered during the excavations.
Foul play?<p>A skeleton representing a man who was tossed face down into a ditch nearly 2,500 years ago with his hands bound in front of his hips was dug up during a high speed rail excavation.</p><p>The positioning of the remains have led archaeologists to suspect that the man may have been a victim of an ancient murder or execution. Though any bindings have since decomposed, his hands were positioned together and pinned under his pelvis. There was also no sign of a grave or coffin. </p><p>"He seems to have had his hands tied, and he was face-down in the bottom of the ditch," <a href="https://www.livescience.com/iron-age-murder-victim-england.html" target="_blank">said archaeologist Rachel Wood</a>, who led the excavation. "There are not many ways that you end up that way."</p><p>Currently, archaeologists are examining the skeleton to uncover more information about the circumstances of the man's death. Fragments of pottery found in the ditch may offer some clues as to exactly when the man died. </p><p>"If he was struck across the head with a heavy object, you could find a mark of that on the back of the skull," Wood said to <a href="https://www.livescience.com/iron-age-murder-victim-england.html" target="_blank">Live Science</a>. "If he was stabbed, you could find blade marks on the ribs. So we're hoping to find something like that, to tell us how he died."</p>
Other discoveries at Wellwick Farm<p>The grim discovery was made at Wellwick Farm near Wendover. That is about 15 miles north-west of the outskirts of London, where <a href="https://www.hs2.org.uk/building-hs2/hs2-green-corridor/" target="_blank">a tunnel</a> is going to be built as part of a HS2 high-speed rail project due to open between London and several northern cities sometime after 2028. The infrastructure project has been something of a bonanza for archaeology as the area is home to more than 60 ancient sites along the planned route that are now being excavated before construction begins. </p><p>The farm sits less than a mile away from the ancient highway <a href="http://web.stanford.edu/group/texttechnologies/cgi-bin/stanfordnottingham/places/?icknield" target="_blank">Icknield Way</a> that runs along the tops of the Chiltern Hills. The route (now mostly trails) has been used since prehistoric times. Evidence at Wellwick Farm indicates that from the Neolithic to the Medieval eras, humans have occupied the region for more than 4,000 years, making it a rich area for archaeological finds. </p><p>Wood and her colleagues found some evidence of an ancient village occupied from the late Bronze Age (more than 3,000 years ago) until the Roman Empire's invasion of southern England about 2,000 years ago. At the site were the remains of animal pens, pits for disposing food, and a roundhouse — a standard British dwelling during the Bronze Age constructed with a circular plan made of stone or wood topped with a conical thatched roof.</p>
Ceremonial burial site<img type="lazy-image" data-runner-src="https://assets.rebelmouse.io/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yMzUzMTk0Ni9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY0NDgwNTIyMX0.I49n1-j8WVhKjIZS_wVWZissnk3W1583yYXB7qaGtN8/img.jpg?width=1245&coordinates=0%2C82%2C0%2C83&height=700" id="44da7" class="rm-shortcode" data-rm-shortcode-id="46cfc8ca1c64fc404b32014542221275" data-rm-shortcode-name="rebelmouse-image" alt="top down view of coffin" data-width="1245" data-height="700" />
A high status burial in a lead-lined coffin dating back to Roman times.
Photo Credit: HS2<p>While these ancient people moved away from Wellwick Farm before the Romans invaded, a large portion of the area was still used for ritual burials for high-status members of society, Wood told Live Science. The ceremonial burial site included a circular ditch (about 60 feet across) at the center, and was a bit of a distance away from the ditch where the (suspected) murder victim was uncovered. Additionally, archaeologists found an ornately detailed grave near the sacred burial site that dates back to the Roman period, hundreds of years later when the original Bronze Age burial site would have been overgrown.</p><p>The newer grave from the Roman period encapsulated an adult skeleton contained in a lead-lined coffin. It's likely that the outer coffin had been made of wood that rotted away. Since it was clearly an ornate burial, the occupant of the grave was probably a person of high status who could afford such a lavish burial. However, according to Wood, no treasures or tokens had been discovered. </p>
Sacred timber circle<img type="lazy-image" data-runner-src="https://assets.rebelmouse.io/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yMzUzMTk0Ny9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY2MDAwOTQ4Mn0.eVJAUcD0uBUkVMFuMOPSgH8EssGkfLf_MjwUv0zGCI8/img.jpg?width=1245&coordinates=0%2C149%2C0%2C149&height=700" id="9de6a" class="rm-shortcode" data-rm-shortcode-id="ee66520d470b26f5c055eaef0b95ec06" data-rm-shortcode-name="rebelmouse-image" alt="An aerial view of the sacred circular monument." data-width="1245" data-height="700" />
An aerial view of the sacred circular monument.
Photo Credit: HS2<p>One of the most compelling archaeological discoveries at Wellwick Farm are the indications of a huge ceremonial circle once circumscribed by timber posts lying south of the Bronze Age burial site. Though the wooden posts have rotted away, signs of the post holes remain. It's thought to date from the Neolithic period to 5,000 years ago, according to Wood.</p><p>This circle would have had a diameter stretching 210 feet across and consisted of two rings of hundreds of posts. There would have been an entry gap to the south-west. Five posts in the very center of the circle aligned with that same gap, which, according to Wood, appeared to have been in the direction of the rising sun on the day of the midwinter solstice. </p><p>Similar Neolithic timber circles have been discovered around Great Britain, such as one near <a href="https://bigthink.com/culture-religion/stonehenge-sarsens" target="_blank">Stonehenge</a> that is considered to date back to around the same time. </p>
Being skeptical isn't just about being contrarian. It's about asking the right questions of ourselves and others to gain understanding.
- It's not always easy to tell the difference between objective truth and what we believe to be true. Separating facts from opinions, according to skeptic Michael Shermer, theoretical physicist Lawrence Krauss, and others, requires research, self-reflection, and time.
- Recognizing your own biases and those of others, avoiding echo chambers, actively seeking out opposing voices, and asking smart, testable questions are a few of the ways that skepticism can be a useful tool for learning and growth.
- As Derren Brown points out, being "skeptical of skepticism" can also lead to interesting revelations and teach us new things about ourselves and our psychology.