Big ideas.
Once a week.
Subscribe to our weekly newsletter.
22 Ways Algorithms Know How You'll Behave Before You Do
Prediction is reinventing industries and running the world. More and more, predictive analytics drives commerce, manufacturing, healthcare, government, and law enforcement.

The future is the ultimate unknown. It's everything that hasn't happened yet.
Prediction as a capability is booming. It reinvents industries and runs the world. More and more, predictive analytics drives commerce, manufacturing, healthcare, government, and law enforcement. In these spheres, organizations operate more effectively by way of predicting behavior—i.e., the outcome for each individual customer, employee, patient, voter, and suspect.
Predictive analytics’ expansive deployment has taken hold. Accenture and Forrester both report that predictive analytics' adoption has more than doubled in recent years. Transparency Market Research projects the predictive analytics market will reach $6.5 billion within a few years. Predictive analytics is becoming a standard safeguard for business, and even demand from consumers for its capabilities promises to surge.
New groundbreaking stories of predictive analytics in action are pouring in. A few key ingredients have opened these floodgates:
I've listed below a slew of examples—from the likes of Facebook, the NSA, Hillary for America, Uber, Airbnb, Google, Shell, UPS, Amazon.com, Coned, Yahoo!, and the U.S. government.
22 EXAMPLES OF PREDICTIVE ANALYTICS:
What’s predicted:
Example:
Which Facebook posts you will like in order to optimize your news feed
Facebook: Predicts which of 1,500 candidate posts (on average) will be most interesting to you in order to personalize your news feed. To optimize the order of content items, the News Feed ranking algorithm weights around 100,000 factors such as recency, likes, clicks, shares, comments, time spent on posts, poster popularity, your affinity for the poster and content area, and measures of relevance and trustworthiness. This intensifies the “addictive” engagement, with two-thirds of Facebook’s 1.44 billion monthly users logging in daily.
Who’s in a photo (aka facial recognition)
Facebook: Improved the state of the art for identifying people from photos to virtually the same performance level as a human: Given two face images, it can determine whether they’re the same person with 97 percent accuracy. Facial recognition helps users tag photos, which they do more than 100 million times a day. The company has also developed predictive models to identify people even if it can’t see the face, achieving 83 percent accuracy when faces are at least partially obscured half of the time, based on elements such as clothing, hair, and pose.
Clicks in order to select which to display
Facebook: In order to increase revenue from its pay-per-click advertisers, predicts ad clicks based on user attributes, device used, and contextual factors.
Terrorism
The National Security Agency: Obtained software solutions for and core competency in predictive analytics. It’s clear that the NSA considers predictive analytics a strategic priority as a means to target investigation activities by automatically discovering previously unknown potential suspects.
Where you are going
Uber: Can predict the specific destination address of San Francisco riders based on exact drop-off location with 74 percent accuracy, despite, for example, how many businesses there are within 100 meters in a typical city area (just taking the closest candidate address achieves 44 percent accuracy).
Acceptance of booking request in order to match guests to hosts
Airbnb: Rank orders accommodations that fulfill a user search in part by the predicted probability each host would accept the user’s booking request. By surfacing likely matches more prominently, the company increased booking conversions by nearly 4 percent—a significant gain considering its estimated annual booking of over 12 million guest nights.
Accommodation bookings at a given price–for dynamic pricing
Airbnb: Suggests each day’s price for an accommodation listing (the “Price Tips” feature) by way of predicting whether the listing will be booked—predicted demand directly informs optimal pricing. Bookings are predicted by day of the week, seasonality, and local events, as well as characteristics of the listing such as the neighborhood, size, amenities, key words like “beach,” number of reviews, and photographs. Hosts who set prices within 5 percent of the suggestions improve their chance of booking by a factor of nearly four.
Spam to send it to your spam folder
Google: Decreased Gmail’s prevalence and false positive rate of spam from disruptive (in 2004) down to negligible.
Oil refinery safety incidents
Shell: Predicts the number of safety incidents per team of workers at oil refineries, globally. One example discovery: Increased employee engagement predicts fewer incidents; one percentage point increase in team employee engagement is associated with a 4 percent decrease in the number of safety incidents per FTE.
Maritime incidents
RightShip: Predicts dangerous or costly maritime incidents in order to assess vessel risk that informs shipment decisions when selecting between vessels. The 10 percent highest-risk vessels are three times more likely than average to experience an incident in the next 12 months, and are 16 times more likely to incur a casualty than the 10 percent least risky. Risk assessment is based on vessel age, type, carrying capacity, origin, registration, ownership, management, and other factors.
Deliveries—which addresses will receive a package
UPS: Cut 85 million miles from annual delivery vehicle driving with a semiautomatic optimization system that plans vehicle/package assignments, as well as package placement within the vehicle, based upon each day’s analytically predicted delivery destinations.
Product choices
Amazon.com: Thirty-five percent of sales come from product recommendations. The company may also develop “anticipatory shipping” that would proactively place packages before they are ordered at hubs or on trucks in order to reduce delays between ordering and receiving purchases, for which it has obtained a patent.
Product choices
Spotify: Is augmenting its song recommendation algorithm to incorporate musical attributes.
Voter persuasion
Hillary for America 2016 Campaign: Given Obama’s success with persuasion modeling in 2012, Hillary Clinton’s 2016 campaign appears to be planning to employ it as well. Analytics job postings reveal they’re going to be “helping the campaign determine which voters to target for persuasion.”
Restaurant health code violations via Yelp reviews
City of Boston: Sponsored a competition that generated the ability to predict whether a restaurant will have more violations than normal with 75 percent accuracy, in part by way of discovering clues within Yelp reviews, in order to target city health department inspections. Similar work for Seattle restaurants distinguished severe violators with 82 percent accuracy.
Lead poisoning from paint
City of Chicago: Identified 5 percent of homes that are at more than twice the risk for lead poisoning incidents than average based on the age of the house, the history of lead paint exposure at that address, the economic conditions of the neighborhood, and other factors. This serves as an early warning system to proactively flag, as an improvement over the more common reactive steps taken after a positive test for poisoning. The risk scores serve to target homes for inspection and children for testing, and could help people determine safer homes to move to.
Fire
City of New York: Targets the fire inspections of its 330,000 inspectable buildings with a predictive model that assesses risk based on about 60 factors.
Manhole explosions and fires
Con Edison: Predicts dangerous manhole explosions and fires in New York City, identifying a 2 percent of manholes that have a 5.5 times greater than average risk of an incident.
Beauty
Yahoo! Labs: Developed a model to categorize photographic portraits as to the subjective human aesthetic of beauty with 64 percent accuracy based on various image attributes. The study determined “that race, gender, and age are largely uncorrelated with photographic beauty.”
Overpriced property leases
U.S. Postal Service Office of Inspector General: Predicted the amount paid over market value for each of their 26,000 leased facilities (e.g., retail unit, plant, warehouse). Targeting facilities in the Northeast Region, USPS auditors projected that 250 of the leases predicted as most overpaid represent a potential savings of $6.6 million by way of renegotiating their next year of lease terms.
Surgical site infections
University of Iowa Hospitals and Clinics: Identifies cases greater than four times as likely to develop surgical-site infections. Targeting anti-infection therapy accordingly reduces the cost of each colorectal surgical procedure an average of $1,300 and will provide a projected annual savings of several million dollars once expanded to other forms of surgery.
Airfares
Hopper: Predicts airfare changes in order to recommend to consumers whether to buy or wait. Ninety-five percent of these predictions save the consumer money or do no worse than the first price seen, saving users an average 10 percent on ticket price.
This flood of predictive activity gains its potential simply because prediction boasts an inherent generality—there are just so many conceivable ways to make use of it. Want to come up with your own new innovative use for predictive analytics? You need only two ingredients. Each application of predictive analytics is defined by:
1. What’s predicted: the kind of behavior—i.e., action, event, or happening—to predict for each individual (e.g., person, Facebook post, photo, ad, trip destination, marine vessel, safety incident, transaction, or other organizational element).
2. What’s done about it: the decisions driven by prediction; the action taken by the organization in response to or informed by each prediction.
We can confidently predict more prediction. Every few months, another big story about predictive analytics rolls off the presses. We’re sure to see the opportunities continue to grow and surprise. Come what may, only time will tell what we’ll tell of time to come.
--
These examples are new in this year's Revised and Updated edition of my book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die. With these newly added cases, the book's central compendium of mini-case studies has grown to 182 entries (most were sourced from presentations at Predictive Analytics World, the event series I founded—for more information about each example, access the book's Notes PDF, available at www.PredictiveNotes.com, and search by organization name).
No, the Yellowstone supervolcano is not ‘overdue’
Why mega-eruptions like the ones that covered North America in ash are the least of your worries.
Ash deposits of some of North America's largest volcanic eruptions.
- The supervolcano under Yellowstone produced three massive eruptions over the past few million years.
- Each eruption covered much of what is now the western United States in an ash layer several feet deep.
- The last eruption was 640,000 years ago, but that doesn't mean the next eruption is overdue.
The end of the world as we know it
Panoramic view of Yellowstone National Park
Image: Heinrich Berann for the National Park Service – public domain
Of the many freak ways to shuffle off this mortal coil – lightning strikes, shark bites, falling pianos – here's one you can safely scratch off your worry list: an outbreak of the Yellowstone supervolcano.
As the map below shows, previous eruptions at Yellowstone were so massive that the ash fall covered most of what is now the western United States. A similar event today would not only claim countless lives directly, but also create enough subsidiary disruption to kill off global civilisation as we know it. A relatively recent eruption of the Toba supervolcano in Indonesia may have come close to killing off the human species (see further below).
However, just because a scenario is grim does not mean that it is likely (insert topical political joke here). In this case, the doom mongers claiming an eruption is 'overdue' are wrong. Yellowstone is not a library book or an oil change. Just because the previous mega-eruption happened long ago doesn't mean the next one is imminent.
Ash beds of North America
Ash beds deposited by major volcanic eruptions in North America.
Image: USGS – public domain
This map shows the location of the Yellowstone plateau and the ash beds deposited by its three most recent major outbreaks, plus two other eruptions – one similarly massive, the other the most recent one in North America.
Huckleberry Ridge
The Huckleberry Ridge eruption occurred 2.1 million years ago. It ejected 2,450 km3 (588 cubic miles) of material, making it the largest known eruption in Yellowstone's history and in fact the largest eruption in North America in the past few million years.
This is the oldest of the three most recent caldera-forming eruptions of the Yellowstone hotspot. It created the Island Park Caldera, which lies partially in Yellowstone National Park, Wyoming and westward into Idaho. Ash from this eruption covered an area from southern California to North Dakota, and southern Idaho to northern Texas.
Mesa Falls
About 1.3 million years ago, the Mesa Falls eruption ejected 280 km3 (67 cubic miles) of material and created the Henry's Fork Caldera, located in Idaho, west of Yellowstone.
It was the smallest of the three major Yellowstone eruptions, both in terms of material ejected and area covered: 'only' most of present-day Wyoming, Colorado, Kansas and Nebraska, and about half of South Dakota.
Lava Creek
The Lava Creek eruption was the most recent major eruption of Yellowstone: about 640,000 years ago. It was the second-largest eruption in North America in the past few million years, creating the Yellowstone Caldera.
It ejected only about 1,000 km3 (240 cubic miles) of material, i.e. less than half of the Huckleberry Ridge eruption. However, its debris is spread out over a significantly wider area: basically, Huckleberry Ridge plus larger slices of both Canada and Mexico, plus most of Texas, Louisiana, Arkansas, and Missouri.
Long Valley
This eruption occurred about 760,000 years ago. It was centered on southern California, where it created the Long Valley Caldera, and spewed out 580 km3 (139 cubic miles) of material. This makes it North America's third-largest eruption of the past few million years.
The material ejected by this eruption is known as the Bishop ash bed, and covers the central and western parts of the Lava Creek ash bed.
Mount St Helens
The eruption of Mount St Helens in 1980 was the deadliest and most destructive volcanic event in U.S. history: it created a mile-wide crater, killed 57 people and created economic damage in the neighborhood of $1 billion.
Yet by Yellowstone standards, it was tiny: Mount St Helens only ejected 0.25 km3 (0.06 cubic miles) of material, most of the ash settling in a relatively narrow band across Washington State and Idaho. By comparison, the Lava Creek eruption left a large swathe of North America in up to two metres of debris.
The difference between quakes and faults
The volume of dense rock equivalent (DRE) ejected by the Huckleberry Ridge event dwarfs all other North American eruptions. It is itself overshadowed by the DRE ejected at the most recent eruption at Toba (present-day Indonesia). This was one of the largest known eruptions ever and a relatively recent one: only 75,000 years ago. It is thought to have caused a global volcanic winter which lasted up to a decade and may be responsible for the bottleneck in human evolution: around that time, the total human population suddenly and drastically plummeted to between 1,000 and 10,000 breeding pairs.
Image: USGS – public domain
So, what are the chances of something that massive happening anytime soon? The aforementioned mongers of doom often claim that major eruptions occur at intervals of 600,000 years and point out that the last one was 640,000 years ago. Except that (a) the first interval was about 200,000 years longer, (b) two intervals is not a lot to base a prediction on, and (c) those intervals don't really mean anything anyway. Not in the case of volcanic eruptions, at least.
Earthquakes can be 'overdue' because the stress on fault lines is built up consistently over long periods, which means quakes can be predicted with a relative degree of accuracy. But this is not how volcanoes behave. They do not accumulate magma at constant rates. And the subterranean pressure that causes the magma to erupt does not follow a schedule.
What's more, previous super-eruptions do not necessarily imply future ones. Scientists are not convinced that there ever will be another big eruption at Yellowstone. Smaller eruptions, however, are much likelier. Since the Lava Creek eruption, there have been about 30 smaller outbreaks at Yellowstone, the last lava flow being about 70,000 years ago.
As for the immediate future (give or take a century): the magma chamber beneath Yellowstone is only 5 percent to 15 percent molten. Most scientists agree that is as un-alarming as it sounds. And that its statistically more relevant to worry about death by lightning, shark, or piano.
Strange Maps #1041
Got a strange map? Let me know at strangemaps@gmail.com.
Smartly dressed: Researchers develop clothes that sense movement via touch
Measuring a person's movements and poses, smart clothes could be used for athletic training, rehabilitation, or health-monitoring.
In recent years there have been exciting breakthroughs in wearable technologies, like smartwatches that can monitor your breathing and blood oxygen levels.
But what about a wearable that can detect how you move as you do a physical activity or play a sport, and could potentially even offer feedback on how to improve your technique?
And, as a major bonus, what if the wearable were something you'd actually already be wearing, like a shirt of a pair of socks?
That's the idea behind a new set of MIT-designed clothing that use special fibers to sense a person's movement via touch. Among other things, the researchers showed that their clothes can actually determine things like if someone is sitting, walking, or doing particular poses.
The group from MIT's Computer Science and Artificial Intelligence Lab (CSAIL) says that their clothes could be used for athletic training and rehabilitation. With patients' permission, they could even help passively monitor the health of residents in assisted-care facilities and determine if, for example, someone has fallen or is unconscious.
The researchers have developed a range of prototypes, from socks and gloves to a full vest. The team's "tactile electronics" use a mix of more typical textile fibers alongside a small amount of custom-made functional fibers that sense pressure from the person wearing the garment.
According to CSAIL graduate student Yiyue Luo, a key advantage of the team's design is that, unlike many existing wearable electronics, theirs can be incorporated into traditional large-scale clothing production. The machine-knitted tactile textiles are soft, stretchable, breathable, and can take a wide range of forms.
"Traditionally it's been hard to develop a mass-production wearable that provides high-accuracy data across a large number of sensors," says Luo, lead author on a new paper about the project that is appearing in this month's edition of Nature Electronics. "When you manufacture lots of sensor arrays, some of them will not work and some of them will work worse than others, so we developed a self-correcting mechanism that uses a self-supervised machine learning algorithm to recognize and adjust when certain sensors in the design are off-base."
The team's clothes have a range of capabilities. Their socks predict motion by looking at how different sequences of tactile footprints correlate to different poses as the user transitions from one pose to another. The full-sized vest can also detect the wearers' pose, activity, and the texture of the contacted surfaces.
The authors imagine a coach using the sensor to analyze people's postures and give suggestions on improvement. It could also be used by an experienced athlete to record their posture so that beginners can learn from them. In the long term, they even imagine that robots could be trained to learn how to do different activities using data from the wearables.
"Imagine robots that are no longer tactilely blind, and that have 'skins' that can provide tactile sensing just like we have as humans," says corresponding author Wan Shou, a postdoc at CSAIL. "Clothing with high-resolution tactile sensing opens up a lot of exciting new application areas for researchers to explore in the years to come."
The paper was co-written by MIT professors Antonio Torralba, Wojciech Matusik, and Tomás Palacios, alongside PhD students Yunzhu Li, Pratyusha Sharma, and Beichen Li; postdoc Kui Wu; and research engineer Michael Foshey.
The work was partially funded by Toyota Research Institute.
Reprinted with permission of MIT News. Read the original article.
Do you worry too much? Stoicism can help
How imagining the worst case scenario can help calm anxiety.
Stoicism can help overcome anxiety
- Stoicism is the philosophy that nothing about the world is good or bad in itself, and that we have control over both our judgments and our reactions to things.
- It is hardest to control our reactions to the things that come unexpectedly.
- By meditating every day on the "worst case scenario," we can take the sting out of the worst that life can throw our way.
Are you a worrier? Do you imagine nightmare scenarios and then get worked up and anxious about them? Does your mind get caught in a horrible spiral of catastrophizing over even the smallest of things? Worrying, particularly imagining the worst case scenario, seems to be a natural part of being human and comes easily to a lot of us. It's awful, perhaps even dangerous, when we do it.
But, there might just be an ancient wisdom that can help. It involves reframing this attitude for the better, and it comes from Stoicism. It's called "premeditation," and it could be the most useful trick we can learn.
Practical Stoicism
Broadly speaking, Stoicism is the philosophy of choosing your judgments. Stoics believe that there is nothing about the universe that can be called good or bad, valuable or valueless, in itself. It's we who add these values to things. As Shakespeare's Hamlet says, "There is nothing either good or bad, but thinking makes it so." Our minds color the things we encounter as being "good" or "bad," and given that we control our minds, we therefore have control over all of our negative feelings.
Put another way, Stoicism maintains that there's a gap between our experience of an event and our judgment of it. For instance, if someone calls you a smelly goat, you have an opportunity, however small and hard it might be, to pause and ask yourself, "How will I judge this?" What's more, you can even ask, "How will I respond?" We have power over which thoughts we entertain and the final say on our actions. Today, Stoicism has influenced and finds modern expression in the hugely effective "cognitive behavioral therapy."
Helping you practice StoicismCredit: Robyn Beck via Getty Images
One of the principal fathers of ancient Stoicism was the Roman statesmen, Seneca, who argued that the unexpected and unforeseen blows of life are the hardest to take control over. The shock of a misfortune can strip away the power we have to choose our reaction. For instance, being burglarized feels so horrible because we had felt so safe at home. A stomach ache, out of the blue, is harder than a stitch thirty minutes into a run. A sudden bang makes us jump, but a firework makes us smile. Fell swoops hurt more than known hardships.
What could possibly go wrong?
So, how can we resolve this? Seneca suggests a Stoic technique called "premeditatio malorum" or "premeditation." At the start of every day, we ought to take time to indulge our anxious and catastrophizing mind. We should "rehearse in the mind: exile, torture, war, shipwreck." We should meditate on the worst things that could happen: your partner will leave you, your boss will fire you, your house will burn down. Maybe, even, you'll die.
This might sound depressing, but the important thing is that we do not stop there.
Stoicism has influenced and finds modern expression in the hugely effective "cognitive behavioral therapy."
The Stoic also rehearses how they will react to these things as they come up. For instance, another Stoic (and Roman Emperor) Marcus Aurelius asks us to imagine all the mean, rude, selfish, and boorish people we'll come across today. Then, in our heads, we script how we'll respond when we meet them. We can shrug off their meanness, smile at their rudeness, and refuse to be "implicated in what is degrading." Thus prepared, we take control again of our reactions and behavior.
The Stoics cast themselves into the darkest and most desperate of conditions but then realize that they can and will endure. With premeditation, the Stoic is prepared and has the mental vigor necessary to take the blow on the chin and say, "Yep, l can deal with this."
Catastrophizing as a method of mental inoculation
Seneca wrote: "In times of peace, the soldier carries out maneuvers." This is also true of premeditation, which acts as the war room or training ground. The agonizing cut of the unexpected is blunted by preparedness. We can prepare the mind for whatever trials may come, in just the same way we can prepare the body for some endurance activity. The world can throw nothing as bad as that which our minds have already imagined.
Stoicism teaches us to embrace our worrying mind but to embrace it as a kind of inoculation. With a frown over breakfast, try to spend five minutes of your day deliberately catastrophizing. Get your anti-anxiety battle plan ready and then face the world.
Jonny Thomson teaches philosophy in Oxford. He runs a popular Instagram account called Mini Philosophy (@philosophyminis). His first book is Mini Philosophy: A Small Book of Big Ideas.
Study: People will donate more to charity if they think something’s in it for them
A study on charity finds that reminding people how nice it feels to give yields better results than appealing to altruism.
