Once a week.
Subscribe to our weekly newsletter.
Predictive Analytics: "Freakonomics" Meets Big Data
As with the anecdotally rich discoveries in Freakonomics, practitioners of predictive analytics constantly stumble upon insightful gems such as,vegetarians miss fewer flights.
While writing my book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, a certain well-known book strongly influenced me to make the science accessible, relevant, and even entertaining: Freakonomics: A Rogue Economist Explores the Hidden Side of Everything.
Both Freakonomics and Predictive Analytics are more than just books—they are disciplines of insight, schools of analytical thought. So, who would win in a fight—Freakonomics or Predictive Analytics?
Freakonomics Versus Predictive Analytics
Champion: Freakonomics, the entertaining pop science originator of what I would call "humanized econometrics." This book takes the analytical, data driven approach to economics—know by some as econometrics—and applies it to everyday life, including real estate brokers, drug dealers, and nursery schools. It defuses the sometimes dry perception of economics with a healthy dose of "Freak" in the book's title.
Challenger: Predictive Analytics, the accessible—yet conceptually complete—primer to, well, predictive analytics. This book defuses the sometimes intimidating perception of data driven prediction and even defines the field of study itself with its semi-humorous subtitle, "The Power to Predict Who Will Click, Buy, Lie, or Die."
Reviewers of my book have drawn the parallel. Founding Advertising.com executive Stein Kretsinger dubbed it, "The Freakonomics of big data," KDnuggets Editor Gregory Piatetsky-Shapiro said it "does to Analytics what Freakonomics did to Economics," and icrunchdata cofounder Todd Nevis declared it to be, "Outfreakingstanding" (Todd wasn't actually drawing a comparison to Freakonomics, but I thought I'd mention that anyway).
How do the two compare? The field of predictive analytics has one major thing in common with—and one major difference from—Freakonomics.
The Commonality: Freaky Data Discoveries Are Predictive
As with the anecdotally rich discoveries in Freakonomics, practitioners of predictive analytics constantly stumble upon insightful gems such as, vegetarians miss fewer flights.
Welcome to the "Ripley's Believe It or Not!" of data science. Predictive analytics' aim isn't only to assess human hunches by testing relationships that seem to make sense, but also to explore a boundless playing field of possible truths beyond the realms of intuition. And so, at times, the analysis drops onto one's desk connections that seem to defy logic. As strange, mystifying, or unexpected as they may seem, these discoveries help predict.
Here are a ten such discoveries from my book, Predictive Analytics
Insights—Far Out and Freaky, Yet Predictive:
1. Insight: Banner ads affect you more than you think. Although you may feel you've learned to ignore them, people who see a merchant's banner ad are 61 percent more likely to subsequently perform a related search, and this drives a 249 percent increase in clicks on the merchant's paid textual ads in the search results.
Suggested Explanation: Advertising exerts a subconscious effect.
2. Insight: Friends stick to the same cell phone company (a social effect). If you switch wireless carriers, your contacts are in turn up to seven times more likely to follow suit.
Company: a major North American wireless carrier
Suggested Explanation: People experience social influence and/or heed financial incentives for in-network calling.
Suggested Explanation: Macs are often more expensive than Windows computers, so Mac users may on average have greater financial resources.
4. Insight: Your e-mail address reveals your level of commitment. Customers who register for a free account with an Earthlink.com e-mail address are almost five times more likely to convert to a paid, premium-level membership than those with a Hotmail.com e-mail address.
Company: an online dating website
Suggested Explanation: Disclosing permanent or primary e-mail accounts reveals a longer-term intention.
6. Insight: Guys literally drool over sports cars. Male college student subjects produce measurably more saliva when presented with images of sports cars or money.
Company: Northwestern University Kellogg School of Management
Suggested Explanation: Consumer impulses are physiological cousins of hunger.
7. Insight: Low credit rating, more car accidents. If your credit score is higher, car insurance companies will lower your premium, since you are a lower driving risk. People with poor credit ratings are charged more for car insurance. In fact, a low credit score can increase your premium more than an at-fault car accident; missing two payments can as much as double your premium.
Company: automobile insurers
Suggested Explanation: "Research indicates that people who manage their personal finances responsibly tend to manage other important aspects of their life with that same level of responsibility, and that would include being responsible behind the wheel of their car," Donald Hanson of the National Association of Independent Insurers theorizes.
8. Insight: Retirement is bad for your health. For a certain working category of males in Austria, each additional year of early retirement decreases life expectancy by 1.8 months.
Company: University of Zurich
Suggested Explanation: Unhealthy habits such as smoking and drinking follow retirement. Voltaire said, "Work spares us from three evils: boredom, vice, and need," and Malcolm Forbes said, "Retirement kills more people than hard work ever did."
9. Insight: Smokers suffer less from repetitive motion disorder. In certain work environments, people who smoke cigarettes are less likely to develop carpal tunnel syndrome.
Company: a major metropolitan newspaper, conducting research on its own staff's health
Suggested Explanation: Smokers take more breaks.
10. Insight: Solo rockers die younger than those in bands. Although all rock stars face higher risk, solo rock stars suffer twice the risk of early death as rock band members.
Company: public health offices in the UK
Suggested Explanation: Band members benefit from peer support and solo artists exhibit even riskier behavior.
The Difference: Predictive Analytics Drives Operational Decisions
While these eye-catching observations are right up Freakonomics' alley, they serve a different purpose for predictive analytics. Instead of enlightening us and unveiling insights as an end in and of itself, they are just the beginning for prediction. These kinds of discoveries—plus many others that may be less entertaining yet are equally powerful—form the building blocks for a predictive model.
Freakonomics' objective is to elucidate. One of its coauthors calls the other an "intellectual detective" who, per their book's subtitle, "Explores the hidden side of everything." Definitely cool. But predictive analytics' objective is a entirely different.
More than just striving to elucidate, predictive analytics runs the world. It actively drives operational decisions central to organizations' functions in order to improve the very way in which companies, government agencies, healthcare facilities, and law enforcement serve and treat us. The technology's prediction of human behavior determines, millions of times a day, whom to call, mail, approve, test, diagnose, warn, investigate, incarcerate, set up on a date, and medicate. By answering this mountain of questions, predictive analytics combats financial risk, fortifies healthcare, conquers spam, toughens crime-fighting, boosts sales—and may in fact answer the biggest question of all: How can we improve the effectiveness of all these massive functions across business and government sectors?
To this end, predictive analytics techniques focus narrowly, striving to discover from data only insights that serve for the particular predictive objective at hand, such as whether each individual will consume, work, love, procreate, vote, mess up, commit a crime, or even die. The underlying analytical learning process builds on individual insights such as those in the table above to form a predictive model, a mechanism (such as a formula or set of business rules) that can consider in concert the multitude of factors known about an individual in order to form its "best bet" prediction for that individual. These predictive capabilities are developed by following a number-crunching, trial-and-error process that has its roots in statistics and computer science.
Of course, the elucidating insights from Freakonomics, economics, and all social sciences for that matter, do inform decision making. But it's not the same thing. Social science insights drive a smaller number of strategic decisions, usually in an ad hoc manner in which decision makers may or may not consider such insights in doing their jobs. In contrast, predictive analytics drives a large number of operational and tactical decisions in a systematic or automatic fashion. Social sciences set out to inform we humans; predictive analytics sets out systematically improve organizational functions on a massive scale.
What's So Big About Big Data?
This endeavor to apply prediction addresses a burning question: With all the excitement over "big data"—which has recently witnessed such dramatic hype as to assume the status of a Movement—what is the value, the function, the purpose? Answer: The most actionable thing to be gained from data is prediction. This is achieved by analytically learning from data how to render predictions for each individual. In this way, predictive analytics is the means to tap big data's full potential.
Big data grows wildly, organically. In this heyday of big data, we thus make hay of this valuable overgrowth of "weeds," the residual side effect of organizations conducting business as usual. This is in contrast to economics and other social sciences, which often must actively conduct experimental studies in order to accrue pertinent data.
Isn't Prediction Impossible?
Freakonomics coauthor Stephen Dubner is skeptical. "The first step toward predicting the future is admitting you can't," he once said ("Why Can't We Predict Earthquakes?" Freakonomics Radio, 2011). As the Danish physicist Niels Bohr put it, "Prediction is very difficult, especially if it's about the future." People are unpredictable. Better data helps prediction more than better math and models do, but we'll never predict with high accuracy.
Fortunately, we don't need to. The massive operations of companies and government agencies benefit greatly by playing the odds better, tipping the numbers games they already play in our favor. Simply predicting a nice amount better than guessing does the trick; a hazy view of what's to come outperforms complete darkness by a landslide. I call this The Prediction Effect: A little prediction goes a long way.
I Never Meta-Science I Didn't Like
Although illuminating insights often emerge as a secondary effect, predictive analytics strives not so much to understand how the world works as to actively improve how it works. This makes predictive analytics a kind of "metascience" that transcends the taxonomy of natural and social sciences, abstracting across them by learning from any and all data sources that would typically serve biology, criminology, economics (and Freakonomics), education, epidemiology, medicine, political science, psychology, or sociology. Predictive analytics' mission is to engineer solutions. As for the data employed and the insights gained, the tactic in play is, "whatever works."
More about Predictive Analytics:
Eric Siegel, Ph.D., is the founder of Predictive Analytics World (www.pawcon.com)—coming in 2013 and 2014 to Toronto, San Francisco, Chicago, Washington D.C., Boston, Berlin, and London—and the author of Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die (February 2013, published by Wiley). Interested in employing predictive analytics at your organization? Get started with the Predictive Analytics Guide (www.pawcon.com/guide).
Why mega-eruptions like the ones that covered North America in ash are the least of your worries.
- The supervolcano under Yellowstone produced three massive eruptions over the past few million years.
- Each eruption covered much of what is now the western United States in an ash layer several feet deep.
- The last eruption was 640,000 years ago, but that doesn't mean the next eruption is overdue.
The end of the world as we know it
Panoramic view of Yellowstone National Park
Image: Heinrich Berann for the National Park Service – public domain
Of the many freak ways to shuffle off this mortal coil – lightning strikes, shark bites, falling pianos – here's one you can safely scratch off your worry list: an outbreak of the Yellowstone supervolcano.
As the map below shows, previous eruptions at Yellowstone were so massive that the ash fall covered most of what is now the western United States. A similar event today would not only claim countless lives directly, but also create enough subsidiary disruption to kill off global civilisation as we know it. A relatively recent eruption of the Toba supervolcano in Indonesia may have come close to killing off the human species (see further below).
However, just because a scenario is grim does not mean that it is likely (insert topical political joke here). In this case, the doom mongers claiming an eruption is 'overdue' are wrong. Yellowstone is not a library book or an oil change. Just because the previous mega-eruption happened long ago doesn't mean the next one is imminent.
Ash beds of North America
Ash beds deposited by major volcanic eruptions in North America.
Image: USGS – public domain
This map shows the location of the Yellowstone plateau and the ash beds deposited by its three most recent major outbreaks, plus two other eruptions – one similarly massive, the other the most recent one in North America.
The Huckleberry Ridge eruption occurred 2.1 million years ago. It ejected 2,450 km3 (588 cubic miles) of material, making it the largest known eruption in Yellowstone's history and in fact the largest eruption in North America in the past few million years.
This is the oldest of the three most recent caldera-forming eruptions of the Yellowstone hotspot. It created the Island Park Caldera, which lies partially in Yellowstone National Park, Wyoming and westward into Idaho. Ash from this eruption covered an area from southern California to North Dakota, and southern Idaho to northern Texas.
About 1.3 million years ago, the Mesa Falls eruption ejected 280 km3 (67 cubic miles) of material and created the Henry's Fork Caldera, located in Idaho, west of Yellowstone.
It was the smallest of the three major Yellowstone eruptions, both in terms of material ejected and area covered: 'only' most of present-day Wyoming, Colorado, Kansas and Nebraska, and about half of South Dakota.
The Lava Creek eruption was the most recent major eruption of Yellowstone: about 640,000 years ago. It was the second-largest eruption in North America in the past few million years, creating the Yellowstone Caldera.
It ejected only about 1,000 km3 (240 cubic miles) of material, i.e. less than half of the Huckleberry Ridge eruption. However, its debris is spread out over a significantly wider area: basically, Huckleberry Ridge plus larger slices of both Canada and Mexico, plus most of Texas, Louisiana, Arkansas, and Missouri.
This eruption occurred about 760,000 years ago. It was centered on southern California, where it created the Long Valley Caldera, and spewed out 580 km3 (139 cubic miles) of material. This makes it North America's third-largest eruption of the past few million years.
The material ejected by this eruption is known as the Bishop ash bed, and covers the central and western parts of the Lava Creek ash bed.
Mount St Helens
The eruption of Mount St Helens in 1980 was the deadliest and most destructive volcanic event in U.S. history: it created a mile-wide crater, killed 57 people and created economic damage in the neighborhood of $1 billion.
Yet by Yellowstone standards, it was tiny: Mount St Helens only ejected 0.25 km3 (0.06 cubic miles) of material, most of the ash settling in a relatively narrow band across Washington State and Idaho. By comparison, the Lava Creek eruption left a large swathe of North America in up to two metres of debris.
The difference between quakes and faults
The volume of dense rock equivalent (DRE) ejected by the Huckleberry Ridge event dwarfs all other North American eruptions. It is itself overshadowed by the DRE ejected at the most recent eruption at Toba (present-day Indonesia). This was one of the largest known eruptions ever and a relatively recent one: only 75,000 years ago. It is thought to have caused a global volcanic winter which lasted up to a decade and may be responsible for the bottleneck in human evolution: around that time, the total human population suddenly and drastically plummeted to between 1,000 and 10,000 breeding pairs.
Image: USGS – public domain
So, what are the chances of something that massive happening anytime soon? The aforementioned mongers of doom often claim that major eruptions occur at intervals of 600,000 years and point out that the last one was 640,000 years ago. Except that (a) the first interval was about 200,000 years longer, (b) two intervals is not a lot to base a prediction on, and (c) those intervals don't really mean anything anyway. Not in the case of volcanic eruptions, at least.
Earthquakes can be 'overdue' because the stress on fault lines is built up consistently over long periods, which means quakes can be predicted with a relative degree of accuracy. But this is not how volcanoes behave. They do not accumulate magma at constant rates. And the subterranean pressure that causes the magma to erupt does not follow a schedule.
What's more, previous super-eruptions do not necessarily imply future ones. Scientists are not convinced that there ever will be another big eruption at Yellowstone. Smaller eruptions, however, are much likelier. Since the Lava Creek eruption, there have been about 30 smaller outbreaks at Yellowstone, the last lava flow being about 70,000 years ago.
As for the immediate future (give or take a century): the magma chamber beneath Yellowstone is only 5 percent to 15 percent molten. Most scientists agree that is as un-alarming as it sounds. And that its statistically more relevant to worry about death by lightning, shark, or piano.
Strange Maps #1041
Got a strange map? Let me know at firstname.lastname@example.org.
Measuring a person's movements and poses, smart clothes could be used for athletic training, rehabilitation, or health-monitoring.
In recent years there have been exciting breakthroughs in wearable technologies, like smartwatches that can monitor your breathing and blood oxygen levels.
But what about a wearable that can detect how you move as you do a physical activity or play a sport, and could potentially even offer feedback on how to improve your technique?
And, as a major bonus, what if the wearable were something you'd actually already be wearing, like a shirt of a pair of socks?
That's the idea behind a new set of MIT-designed clothing that use special fibers to sense a person's movement via touch. Among other things, the researchers showed that their clothes can actually determine things like if someone is sitting, walking, or doing particular poses.
The group from MIT's Computer Science and Artificial Intelligence Lab (CSAIL) says that their clothes could be used for athletic training and rehabilitation. With patients' permission, they could even help passively monitor the health of residents in assisted-care facilities and determine if, for example, someone has fallen or is unconscious.
The researchers have developed a range of prototypes, from socks and gloves to a full vest. The team's "tactile electronics" use a mix of more typical textile fibers alongside a small amount of custom-made functional fibers that sense pressure from the person wearing the garment.
According to CSAIL graduate student Yiyue Luo, a key advantage of the team's design is that, unlike many existing wearable electronics, theirs can be incorporated into traditional large-scale clothing production. The machine-knitted tactile textiles are soft, stretchable, breathable, and can take a wide range of forms.
"Traditionally it's been hard to develop a mass-production wearable that provides high-accuracy data across a large number of sensors," says Luo, lead author on a new paper about the project that is appearing in this month's edition of Nature Electronics. "When you manufacture lots of sensor arrays, some of them will not work and some of them will work worse than others, so we developed a self-correcting mechanism that uses a self-supervised machine learning algorithm to recognize and adjust when certain sensors in the design are off-base."
The team's clothes have a range of capabilities. Their socks predict motion by looking at how different sequences of tactile footprints correlate to different poses as the user transitions from one pose to another. The full-sized vest can also detect the wearers' pose, activity, and the texture of the contacted surfaces.
The authors imagine a coach using the sensor to analyze people's postures and give suggestions on improvement. It could also be used by an experienced athlete to record their posture so that beginners can learn from them. In the long term, they even imagine that robots could be trained to learn how to do different activities using data from the wearables.
"Imagine robots that are no longer tactilely blind, and that have 'skins' that can provide tactile sensing just like we have as humans," says corresponding author Wan Shou, a postdoc at CSAIL. "Clothing with high-resolution tactile sensing opens up a lot of exciting new application areas for researchers to explore in the years to come."
The paper was co-written by MIT professors Antonio Torralba, Wojciech Matusik, and Tomás Palacios, alongside PhD students Yunzhu Li, Pratyusha Sharma, and Beichen Li; postdoc Kui Wu; and research engineer Michael Foshey.
The work was partially funded by Toyota Research Institute.
How imagining the worst case scenario can help calm anxiety.
- Stoicism is the philosophy that nothing about the world is good or bad in itself, and that we have control over both our judgments and our reactions to things.
- It is hardest to control our reactions to the things that come unexpectedly.
- By meditating every day on the "worst case scenario," we can take the sting out of the worst that life can throw our way.
Are you a worrier? Do you imagine nightmare scenarios and then get worked up and anxious about them? Does your mind get caught in a horrible spiral of catastrophizing over even the smallest of things? Worrying, particularly imagining the worst case scenario, seems to be a natural part of being human and comes easily to a lot of us. It's awful, perhaps even dangerous, when we do it.
But, there might just be an ancient wisdom that can help. It involves reframing this attitude for the better, and it comes from Stoicism. It's called "premeditation," and it could be the most useful trick we can learn.
Broadly speaking, Stoicism is the philosophy of choosing your judgments. Stoics believe that there is nothing about the universe that can be called good or bad, valuable or valueless, in itself. It's we who add these values to things. As Shakespeare's Hamlet says, "There is nothing either good or bad, but thinking makes it so." Our minds color the things we encounter as being "good" or "bad," and given that we control our minds, we therefore have control over all of our negative feelings.
Put another way, Stoicism maintains that there's a gap between our experience of an event and our judgment of it. For instance, if someone calls you a smelly goat, you have an opportunity, however small and hard it might be, to pause and ask yourself, "How will I judge this?" What's more, you can even ask, "How will I respond?" We have power over which thoughts we entertain and the final say on our actions. Today, Stoicism has influenced and finds modern expression in the hugely effective "cognitive behavioral therapy."
Helping you practice StoicismCredit: Robyn Beck via Getty Images
One of the principal fathers of ancient Stoicism was the Roman statesmen, Seneca, who argued that the unexpected and unforeseen blows of life are the hardest to take control over. The shock of a misfortune can strip away the power we have to choose our reaction. For instance, being burglarized feels so horrible because we had felt so safe at home. A stomach ache, out of the blue, is harder than a stitch thirty minutes into a run. A sudden bang makes us jump, but a firework makes us smile. Fell swoops hurt more than known hardships.
What could possibly go wrong?
So, how can we resolve this? Seneca suggests a Stoic technique called "premeditatio malorum" or "premeditation." At the start of every day, we ought to take time to indulge our anxious and catastrophizing mind. We should "rehearse in the mind: exile, torture, war, shipwreck." We should meditate on the worst things that could happen: your partner will leave you, your boss will fire you, your house will burn down. Maybe, even, you'll die.
This might sound depressing, but the important thing is that we do not stop there.
Stoicism has influenced and finds modern expression in the hugely effective "cognitive behavioral therapy."
The Stoic also rehearses how they will react to these things as they come up. For instance, another Stoic (and Roman Emperor) Marcus Aurelius asks us to imagine all the mean, rude, selfish, and boorish people we'll come across today. Then, in our heads, we script how we'll respond when we meet them. We can shrug off their meanness, smile at their rudeness, and refuse to be "implicated in what is degrading." Thus prepared, we take control again of our reactions and behavior.
The Stoics cast themselves into the darkest and most desperate of conditions but then realize that they can and will endure. With premeditation, the Stoic is prepared and has the mental vigor necessary to take the blow on the chin and say, "Yep, l can deal with this."
Catastrophizing as a method of mental inoculation
Seneca wrote: "In times of peace, the soldier carries out maneuvers." This is also true of premeditation, which acts as the war room or training ground. The agonizing cut of the unexpected is blunted by preparedness. We can prepare the mind for whatever trials may come, in just the same way we can prepare the body for some endurance activity. The world can throw nothing as bad as that which our minds have already imagined.
Stoicism teaches us to embrace our worrying mind but to embrace it as a kind of inoculation. With a frown over breakfast, try to spend five minutes of your day deliberately catastrophizing. Get your anti-anxiety battle plan ready and then face the world.
A study on charity finds that reminding people how nice it feels to give yields better results than appealing to altruism.