The Trouble With Last Week's Three-Question Rationality Quiz

If you're one of the 85,000 readers who took the three-question quiz I posted last week, chances are you answered some items incorrectly. Like some of my smart, accomplished friends and family members who took the challenge, you might even have scored a big fat zero. And like them, you might be troubled by your performance. 

Buck up: You are far from alone. Like most people, I got some of them wrong, too, when I encountered similar questions years ago in a course in graduate school. The questions and answers reveal interesting patterns in the way we think, but I don’t believe that getting them wrong is very persuasive evidence of human irrationality. In short, much of the work of cognitive scientists — even fascinating Nobel Prize-winning research by Daniel Kahneman — leaves me edified, but not alarmed. The human capacity for reason may be fragile and partial, but it is not belied by studies in which large percentages of subjects answer a few tricky questions incorrectly.

Rethinking Genevieve

I’ll focus here on my question about Genevieve, more popularly known as the Linda problem(If you haven’t taken the quiz yet, you might want to test yourself before reading on.) Here, again, is the question:

At a dinner party this weekend, a friend introduces you to a woman named Genevieve. He tells you that Genevieve recently graduated from Bryn Mawr College with a B.A. in philosophy, where she was an active volunteer in an advocacy group for women's health and edited a literary magazine. You’re interested in talking to Genevieve about [Georg] Hegel, the subject of her senior thesis, but your friend jumps in and asks you to rank the following statements about Genevieve in order of their probability:

(1) Genevieve is a feminist.

(2) Genevieve is looking for a job as a sanitation worker.

(3) Genevieve is a feminist who is looking for a job as a sanitation worker.

Given what you know about Genevieve, rank the statements from most likely to least likely.  

This question is meant to test how well you evaluate probabilities. If you botch the task by ranking (3) before (2), you commit what Kahneman and Amos Tversky call the “conjunction fallacy”: seeing the concurrent existence of two states of affairs as more likely than the existence of only one. As Kahneman points out in Thinking, Fast and Slow, we are not always inclined to make this mistake. Try this question, for example (from p. 160):

Which alternative is more probable?

(a) Jane is a teacher.

(b) Jane is a teacher who walks to work.

Everyone will immediately see that (a) is more probable than (b), since Jane could commute to school by bike, car, subway, or Segway. We answer this question correctly yet mishandle the logically identical Genevieve problem because the latter primes us to develop a particular view of Genevieve. Not only is she a woman, but also she is a liberal activist, a philosophy major, and a graduate of an elite women’s college. Her fancy French name gilds the lily. How often have you witnessed such a creature driving a sanitation truck?

The background information about Linda/Genevieve is designed to tempt you toward a response that is illogical. It’s bait, pure and simple. When we take the bait, we are giving in to our intuitive sense that sanitation workers are by and large male and less highly educated than Genevieve. Kahneman apparently thinks that the deceit at the heart of the question is justified because it shows how a story can dull our reasoning ability. In his words, System 2 (our deliberate, slow-thinking rational capacity) “is not impressively alert.” We jump to conclusions when the conclusion seems obvious: “The laziness of System 2 is an important fact of life,” he writes (p. 164). But there is another interpretation of this purported error that Kahneman dismisses too quickly. Consider the way he reacts to students who dare to challenge his interpretation of the Linda problem:

Remarkably, the sinners seemed to have no shame. When I asked my large undergraduate class in some indignation, ‘Do you realize that you have violated an elementary logical rule?’ someone in the back row shouted, ‘So what?’ and a graduate student who made the same error explained herself by saying, ‘I thought you just asked for my opinion.’  (Thinking, Fast and Slow, p. 158)

Rather than explore his students’ points of view and open his experiment to critical inquiry, Kahneman dismisses their responses out of hand. A logic error is a logic error; “opinions” have no place in a discussion of probability. But as Ralph Hertwig and Gert Gigerenzer argue in their 1999 article criticizing Tvesky and Kahneman’s conclusions, this all depends on what “probability” means in the minds of the test-takers (my adaptations to the Genevieve example follow in brackets):

It is evident that most of the candidate meanings of ‘probability’ and ‘probable’ cannot be reduced to mathematical probability. For instance, if one interprets ‘probability’ and ‘probable’ in the [Genevieve] problem as ‘something which, judged by present evidence, is likely to happen,’ ‘plausible,’ or ‘a credible story,’ then one might easily judge [‘Genevieve being a feminist sanitation worker’] to be more probable than [‘Genevieve being a sanitation worker’] because the information about [Genevieve] was deliberately selected to provide no evidence for the hypothesis [that she is a sanitation worker alone]. Under these interpretations it is pointless to compare participants’ judgments with a norm from mathematical probability theory, because the inferred meanings have nothing to do with mathematical probability.

Exactly. If you ranked (3) before (2) on the quiz, you were seeking the most plausible, commonsensical answer — not trying to parse a logic game on the LSAT. Your rationality wasn’t dozing. System 2 was absorbing and responding to evidence provided by the questioner — evidence you had every reason to believe was relevant to the question. 

The Selection Task and the Good Bet

Questions 2 and 3 on my quiz are meant to test deductive logic and instrumental rationality, respectively, but neither tells a complete story about how well an individual reasons in real life. Wason’s selection task experiments show that performance improves markedly when real-world examples, rather than letters and numbers, are tested. Since social situations like this confront us much more often than test items like this, we should take our tendency to bungle the latter with a grain of salt. And since question 3 is a measure of risk tolerance whose answer depends on a number of contingencies, you can’t be condemned by the Rationality Police for passing up even an excellent bet. It may indeed be useful to be aware of an unusually strong loss aversion in your psyche so you can work on changing your behavior, but there is no evidence that individuals who gamble more freely necessarily live better lives. 

So Why the Quiz?

Why did I quiz you using questions and answers I believe to be less meaningful than their originators thought them to be? For two reasons: (1) to draw your attention to the types of data that inform the new science of irrationality and, I hope, to open up some of these questions and conclusions to popular critical engagement; and (2) to conduct a non-scientific experiment of my own about individuals’ emotional investment in their sense of themselves as rational. My hypothesis is that while we love reading about humanity’s tendency toward the irrational, we take offense when light is thrown on our own individual incompetencies. 

If you were upset about performing poorly on the quiz, your trauma was compounded when you read comments from other readers who reported how "obvious" the answers were and how easily they aced it. Here Kahneman has some useful advice for you: Be wary of the “availability fallacy,” the tendency to generalize from an easily accessed, but skewed data set. The majority of commenters on Facebook reported answering at least two of the questions correctly. This doesn’t mean that Big Think readers are necessarily more rational than the general population. The vast majority of the individuals who have read the post did not comment at all, and the vast majority of this silent majority likely did poorly on the quiz. The comments represent nothing but a single-digit swath of the quiz-takers. Most of your fellow readers probably performed exactly the way you did.

So relax, people. We might not all be perfect reasoners, but we’re all in this together. 

Image credit: Shutterstock

Note to readers: This post is a truncated and slightly revised version of post I published several years ago. Next week Praxis returns with all-new material.  

Trying to spot China’s green energy tipping point

Pay attention to the decisions made by the provinces.

Surprising Science
  • China leads the world in numerous green energy categories.
  • CO2 emissions in the country totaling more than all coal emissions in the U.S. have recently emerged.
  • This seems to be an administrative-induced blip on the way towards a green energy tipping point.
Keep reading Show less

Got a question for a real NASA astronomer? Ask it here!

NASA astronomer Michelle Thaller is coming back to Big Think to answer YOUR questions! Here's all you need to know to submit your science-related inquiries.

Surprising Science

Big Think's amazing audience has responded so well to our videos from NASA astronomer and Assistant Director for Science Communication Michelle Thaller that we couldn't wait to bring her back for more!

And this time, she's ready to tackle any questions you're willing to throw at her, like, "How big is the Universe?", "Am I really made of stars?" or, "How long until Elon Musk starts a colony on Mars?"

All you have to do is submit your questions to the form below, and we'll use them for an upcoming Q+A session with Michelle. You know what to do, Big Thinkers!

Keep reading Show less

The value of owning more books than you can read

Or how I learned to stop worrying and love my tsundoku.

(Photo from Wikimedia)
Personal Growth
  • Many readers buy books with every intention of reading them only to let them linger on the shelf.
  • Statistician Nassim Nicholas Taleb believes surrounding ourselves with unread books enriches our lives as they remind us of all we don't know.
  • The Japanese call this practice tsundoku, and it may provide lasting benefits.
Keep reading Show less

Take the Big Think survey for a chance to win :)

Calling all big thinkers!

  • Tell us a little bit about where you find Big Think's videos, articles, and podcasts.
  • Be entered for a chance to win 1 of 3 Amazon gift cards each worth $100.
  • All survey information is anonymous and will be used only for this survey.
Keep reading Show less

Elon Musk says Boring Company tunnel opens Dec. 10

The Boring Company plans to offer free rides in its prototype tunnel in Hawthorne, California in December.

Image: Getty Images/Claudia Soraya
Technology & Innovation
  • The prototype tunnel is about 2 miles long and contains electric skates that travel at top speeds of around 150 mph.
  • This is the first tunnel from the company that will be open to the public.
  • If successful, the prototype could help the company receive regulatory approval for much bigger projects in L.A. and beyond.
Keep reading Show less

How to raise a non-materialistic kid

Money makes the world go 'round. Unfortunately, it can make both children and adults into materialists.

Robert Collins / Unsplash
Personal Growth
  • Keeping a gratitude journal caused children to donate 60 percent more to charitable causes.
  • Other methods suggested by researchers include daily gratitude reflection, gratitude posters, and keeping a "gratitude jar."
  • Materialism has been shown to increase anxiety and depression and promote selfish attitudes and behavior.
Keep reading Show less

Cancer researcher says keto is not a fad diet

Anatomy and physiology professor David Harper claims a recent study in The Lancet is flawed.

Photo: Shutterstock
Surprising Science
  • The low-carbohydrate group in a recent Lancet study were typically middle-aged, obese, sedentary, diabetic smokers.
  • The study was not a randomized, controlled, double-blind experiment.
  • Harper has been in ketosis for six years, and says it has profound effects on cancer patients, among other chronic ailments.
Keep reading Show less

Are you a Boltzmann Brain? Why nothing in the Universe may be real

A mind-bending paradox questions the nature of reality.

Surprising Science
  • Boltzmann Brains are hypothetical disembodied entities with self-awareness.
  • It may be more likely for a Boltzmann Brain to come into existence than the whole Universe.
  • The idea highlights a paradox in thermodynamics.
Keep reading Show less