Weapons of Math Destruction: How Big Data Destroys Lives

When companies hide algorithms from public view, the data are used for destructive purposes, warns Cathy O'Neil in her new book.

A few weeks ago I went with my fiancée to buy a new car. While figuring out which model would be most economical, I reminded her to factor in an increase in insurance rates, something that had happened to me the previous year. The salesman said that might not be true.

Turns out he was right. A former auto insurance salesman, he told us rates are dependent upon zip code. Companies factor in the driving records of everyone in that neighborhood—your personal driving record is only partly consequential. So while moving from Mar Vista to Palms included an increase for me, my fiancée moving from Venice resulted in a decrease.

We should consider that balance, correct? Hardly. We’re the victims of an invisible algorithm, something data scientist Cathy O’Neil spends an entire book discussing in Weapons of Math Destruction. A math geek by nature, O’Neil became disillusioned with her lifelong passion’s applications in Big Data when working for a hedge fund during the economic collapse in 2008.

The crash made it all too clear that mathematics, once my refuge, was not only deeply entangled in the world’s problems but also fueling many of them.

My insurance issue seems benign in comparison to many issues of inequality and injustice O’Neil address. To return to that industry, however, she discusses how credit scores, itself an industry fueled by deception and corruption, affects unsuspecting drivers in insidious ways.

For example, drivers in Florida with clean records and bad credit scores were shown to pay $1,522 more than drivers with similar records save a drunken driving conviction. Whether or not you’ve paid your phone bill can have more impact on auto insurance than getting hammered and sitting behind the wheel. If this seems unfair, it is, and the problems are only getting worse.

Credit scores are used by nearly half of American employers to screen potential employees. With the rise of online resume readers, qualified candidates are never considered by human eyes due to the slightest infraction. Yet credit should not be a prison sentence. Many factors contribute to a lapse in bill payment, including another subject invisible algorithms affect: health insurance. One crippling medical bill can very well result in punishment in the eyes of creditors and employers.

It’s the invisibility, dehumanization by numbers, that’s the real problem. Qualifying subtleties during an interview—facial expressions, vocal fluctuations, pantomimes, and perhaps most importantly, a logical explanation as to why one’s credit score is not optimal—are never weighed in a system that only reads numerical data.

Without feedback, however, a statistical engine can continue spinning out faulty and damaging analysis while never learning from its mistakes … Instead of searching for the truth, the score comes to embody it.

As an example O’Neil tells the story of Sarah Wysocki. In 2009, the Washington, D.C. school district implemented one such system to weed out ineffective teachers. Wysocki was beloved by parents but her IMPACT evaluation score placed her in the bottom 5 percent during the second year of statistical measuring. She was among the 206 teachers let go that year.

What such scoring systems do not take into account, O’Neil writes, are the nuanced factors of education. Like with corporations, the statistical machine seeks constant improvement in the same way shareholders demand perpetual profits. Yet teachers have different classes each year—she might instruct honors students one year, special education children the next. All the algorithm views are test results.

Another teacher in the book received a score of six out of a hundred in a similar rating method. The following year he received a ninety-six. While there’s always room for improvement, such a system is obviously ineffective given such a wide disparity for a senior instructor. He was not alone on this absurd grading curve.

Day by day the rhythm of our lives are being automated. O’Neil has a special dislike for algorithms used by policing systems to monitor crime. They create a self-perpetuating feedback loop targeting low-income minority neighborhoods. This leads to confirmation bias: of course that’s where the problems are. Kids caught with nickel bags receive jail time while bankers siphoning billions from ignorant customers are immune to prosecution.

While critical of the systems in place, O’Neil reminds us that it does not have to be so. Math can be a tool of construction as well as destruction. For example, an algorithm could show if it’s more beneficial to pay your phone or electricity bill during a tight month in regards to how each would affect your credit score. Not sexy, but realistic.

She calls for data scientists to take a digital Hippocratic Oath, which asks them to consider the enormous impact algorithms has on the population. She also wants companies to “open the hood” so methods are not hidden from public view.

Open source and numerically honest platforms are beneficial from consumer and social standpoints. O’Neil invokes Mitt Romney’s 47 percent comment. The presidential candidate believed himself to be in a room of like-minded elite, ignorant that staff might not share his values. When everyone’s cell phone is a video camera politicians can no longer have separate talking points for separate audiences—something Hillary Clinton is being reminded thanks to Wikileaks.

Asking companies to peer behind the numbers is requesting of them an ethical consideration: Is it more important to maximize profits at inhumane costs or take a slight financial hit to serve the better good? Of course each is going to answer differently for a host of reasons. As long as that’s the case we’ll never know whether their weapons are constructive or destructive. As for now, the latter is too often true. As O’Neil warns, democracy itself is the wager.


Derek Beres is working on his new book, Whole Motion: Training Your Brain and Body For Optimal Health (Carrel/Skyhorse, Spring 2017). He is based in Los Angeles. Stay in touch on Facebook and Twitter.

Big Think
Sponsored by Lumina Foundation

Upvote/downvote each of the videos below!

As you vote, keep in mind that we are looking for a winner with the most engaging social venture pitch - an idea you would want to invest in.

Keep reading Show less

Essential financial life skills for 21st-century Americans

Having these financial life skills can help you navigate challenging economic environments.

Photo by Jp Valery on Unsplash
Personal Growth
  • Americans are swimming in increasingly higher amounts of debt, even the upper middle class.
  • For many, this burden can be alleviated by becoming familiar with some straightforward financial concepts.
  • Here's some essential financial life skills needed to ensure your economic wellbeing.
Keep reading Show less

How to flirt: 7 tips backed by science

When it comes to flirting, love meters have nothing on these researchers' findings.

(Photo from Wikimedia)
Sex & Relationships
  • Flirting is an important part of life. It can be a fun, adventurous way to meet others and develop intimate relationships.
  • Many people find flirting to be an anxiety-ridden experience, but science can help us discover principles to be more relaxed while flirting.
  • Smiling and eye contact are proven winners, while pick-up lines are a flirty fallacy.
Keep reading Show less

New fossils suggest human ancestors evolved in Europe, not Africa

Experts argue the jaws of an ancient European ape reveal a key human ancestor.

Surprising Science
  • The jaw bones of an 8-million-year-old ape were discovered at Nikiti, Greece, in the '90s.
  • Researchers speculate it could be a previously unknown species and one of humanity's earliest evolutionary ancestors.
  • These fossils may change how we view the evolution of our species.

Homo sapiens have been on earth for 200,000 years — give or take a few ten-thousand-year stretches. Much of that time is shrouded in the fog of prehistory. What we do know has been pieced together by deciphering the fossil record through the principles of evolutionary theory. Yet new discoveries contain the potential to refashion that knowledge and lead scientists to new, previously unconsidered conclusions.

A set of 8-million-year-old teeth may have done just that. Researchers recently inspected the upper and lower jaw of an ancient European ape. Their conclusions suggest that humanity's forebearers may have arisen in Europe before migrating to Africa, potentially upending a scientific consensus that has stood since Darwin's day.

Rethinking humanity's origin story

The frontispiece of Thomas Huxley's Evidence as to Man's Place in Nature (1863) sketched by natural history artist Benjamin Waterhouse Hawkins. (Photo: Wikimedia Commons)

As reported in New Scientist, the 8- to 9-million-year-old hominin jaw bones were found at Nikiti, northern Greece, in the '90s. Scientists originally pegged the chompers as belonging to a member of Ouranopithecus, an genus of extinct Eurasian ape.

David Begun, an anthropologist at the University of Toronto, and his team recently reexamined the jaw bones. They argue that the original identification was incorrect. Based on the fossil's hominin-like canines and premolar roots, they identify that the ape belongs to a previously unknown proto-hominin.

The researchers hypothesize that these proto-hominins were the evolutionary ancestors of another European great ape Graecopithecus, which the same team tentatively identified as an early hominin in 2017. Graecopithecus lived in south-east Europe 7.2 million years ago. If the premise is correct, these hominins would have migrated to Africa 7 million years ago, after undergoing much of their evolutionary development in Europe.

Begun points out that south-east Europe was once occupied by the ancestors of animals like the giraffe and rhino, too. "It's widely agreed that this was the found fauna of most of what we see in Africa today," he told New Scientists. "If the antelopes and giraffes could get into Africa 7 million years ago, why not the apes?"

He recently outlined this idea at a conference of the American Association of Physical Anthropologists.

It's worth noting that Begun has made similar hypotheses before. Writing for the Journal of Human Evolution in 2002, Begun and Elmar Heizmann of the Natural history Museum of Stuttgart discussed a great ape fossil found in Germany that they argued could be the ancestor (broadly speaking) of all living great apes and humans.

"Found in Germany 20 years ago, this specimen is about 16.5 million years old, some 1.5 million years older than similar species from East Africa," Begun said in a statement then. "It suggests that the great ape and human lineage first appeared in Eurasia and not Africa."

Migrating out of Africa

In the Descent of Man, Charles Darwin proposed that hominins descended out of Africa. Considering the relatively few fossils available at the time, it is a testament to Darwin's astuteness that his hypothesis remains the leading theory.

Since Darwin's time, we have unearthed many more fossils and discovered new evidence in genetics. As such, our African-origin story has undergone many updates and revisions since 1871. Today, it has splintered into two theories: the "out of Africa" theory and the "multi-regional" theory.

The out of Africa theory suggests that the cradle of all humanity was Africa. Homo sapiens evolved exclusively and recently on that continent. At some point in prehistory, our ancestors migrated from Africa to Eurasia and replaced other subspecies of the genus Homo, such as Neanderthals. This is the dominant theory among scientists, and current evidence seems to support it best — though, say that in some circles and be prepared for a late-night debate that goes well past last call.

The multi-regional theory suggests that humans evolved in parallel across various regions. According to this model, the hominins Homo erectus left Africa to settle across Eurasia and (maybe) Australia. These disparate populations eventually evolved into modern humans thanks to a helping dollop of gene flow.

Of course, there are the broad strokes of very nuanced models, and we're leaving a lot of discussion out. There is, for example, a debate as to whether African Homo erectus fossils should be considered alongside Asian ones or should be labeled as a different subspecies, Homo ergaster.

Proponents of the out-of-Africa model aren't sure whether non-African humans descended from a single migration out of Africa or at least two major waves of migration followed by a lot of interbreeding.

Did we head east or south of Eden?

Not all anthropologists agree with Begun and his team's conclusions. As noted by New Scientist, it is possible that the Nikiti ape is not related to hominins at all. It may have evolved similar features independently, developing teeth to eat similar foods or chew in a similar manner as early hominins.

Ultimately, Nikiti ape alone doesn't offer enough evidence to upend the out of Africa model, which is supported by a more robust fossil record and DNA evidence. But additional evidence may be uncovered to lend further credence to Begun's hypothesis or lead us to yet unconsidered ideas about humanity's evolution.