Once a week.
Subscribe to our weekly newsletter.
Predictive Analytics: "Freakonomics" Meets Big Data
As with the anecdotally rich discoveries in Freakonomics, practitioners of predictive analytics constantly stumble upon insightful gems such as,vegetarians miss fewer flights.
While writing my book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, a certain well-known book strongly influenced me to make the science accessible, relevant, and even entertaining: Freakonomics: A Rogue Economist Explores the Hidden Side of Everything.
Both Freakonomics and Predictive Analytics are more than just books—they are disciplines of insight, schools of analytical thought. So, who would win in a fight—Freakonomics or Predictive Analytics?
Freakonomics Versus Predictive Analytics
Champion: Freakonomics, the entertaining pop science originator of what I would call "humanized econometrics." This book takes the analytical, data driven approach to economics—know by some as econometrics—and applies it to everyday life, including real estate brokers, drug dealers, and nursery schools. It defuses the sometimes dry perception of economics with a healthy dose of "Freak" in the book's title.
Challenger: Predictive Analytics, the accessible—yet conceptually complete—primer to, well, predictive analytics. This book defuses the sometimes intimidating perception of data driven prediction and even defines the field of study itself with its semi-humorous subtitle, "The Power to Predict Who Will Click, Buy, Lie, or Die."
Reviewers of my book have drawn the parallel. Founding Advertising.com executive Stein Kretsinger dubbed it, "The Freakonomics of big data," KDnuggets Editor Gregory Piatetsky-Shapiro said it "does to Analytics what Freakonomics did to Economics," and icrunchdata cofounder Todd Nevis declared it to be, "Outfreakingstanding" (Todd wasn't actually drawing a comparison to Freakonomics, but I thought I'd mention that anyway).
How do the two compare? The field of predictive analytics has one major thing in common with—and one major difference from—Freakonomics.
The Commonality: Freaky Data Discoveries Are Predictive
As with the anecdotally rich discoveries in Freakonomics, practitioners of predictive analytics constantly stumble upon insightful gems such as, vegetarians miss fewer flights.
Welcome to the "Ripley's Believe It or Not!" of data science. Predictive analytics' aim isn't only to assess human hunches by testing relationships that seem to make sense, but also to explore a boundless playing field of possible truths beyond the realms of intuition. And so, at times, the analysis drops onto one's desk connections that seem to defy logic. As strange, mystifying, or unexpected as they may seem, these discoveries help predict.
Here are a ten such discoveries from my book, Predictive Analytics
Insights—Far Out and Freaky, Yet Predictive:
1. Insight: Banner ads affect you more than you think. Although you may feel you've learned to ignore them, people who see a merchant's banner ad are 61 percent more likely to subsequently perform a related search, and this drives a 249 percent increase in clicks on the merchant's paid textual ads in the search results.
Suggested Explanation: Advertising exerts a subconscious effect.
2. Insight: Friends stick to the same cell phone company (a social effect). If you switch wireless carriers, your contacts are in turn up to seven times more likely to follow suit.
Company: a major North American wireless carrier
Suggested Explanation: People experience social influence and/or heed financial incentives for in-network calling.
Suggested Explanation: Macs are often more expensive than Windows computers, so Mac users may on average have greater financial resources.
4. Insight: Your e-mail address reveals your level of commitment. Customers who register for a free account with an Earthlink.com e-mail address are almost five times more likely to convert to a paid, premium-level membership than those with a Hotmail.com e-mail address.
Company: an online dating website
Suggested Explanation: Disclosing permanent or primary e-mail accounts reveals a longer-term intention.
6. Insight: Guys literally drool over sports cars. Male college student subjects produce measurably more saliva when presented with images of sports cars or money.
Company: Northwestern University Kellogg School of Management
Suggested Explanation: Consumer impulses are physiological cousins of hunger.
7. Insight: Low credit rating, more car accidents. If your credit score is higher, car insurance companies will lower your premium, since you are a lower driving risk. People with poor credit ratings are charged more for car insurance. In fact, a low credit score can increase your premium more than an at-fault car accident; missing two payments can as much as double your premium.
Company: automobile insurers
Suggested Explanation: "Research indicates that people who manage their personal finances responsibly tend to manage other important aspects of their life with that same level of responsibility, and that would include being responsible behind the wheel of their car," Donald Hanson of the National Association of Independent Insurers theorizes.
8. Insight: Retirement is bad for your health. For a certain working category of males in Austria, each additional year of early retirement decreases life expectancy by 1.8 months.
Company: University of Zurich
Suggested Explanation: Unhealthy habits such as smoking and drinking follow retirement. Voltaire said, "Work spares us from three evils: boredom, vice, and need," and Malcolm Forbes said, "Retirement kills more people than hard work ever did."
9. Insight: Smokers suffer less from repetitive motion disorder. In certain work environments, people who smoke cigarettes are less likely to develop carpal tunnel syndrome.
Company: a major metropolitan newspaper, conducting research on its own staff's health
Suggested Explanation: Smokers take more breaks.
10. Insight: Solo rockers die younger than those in bands. Although all rock stars face higher risk, solo rock stars suffer twice the risk of early death as rock band members.
Company: public health offices in the UK
Suggested Explanation: Band members benefit from peer support and solo artists exhibit even riskier behavior.
The Difference: Predictive Analytics Drives Operational Decisions
While these eye-catching observations are right up Freakonomics' alley, they serve a different purpose for predictive analytics. Instead of enlightening us and unveiling insights as an end in and of itself, they are just the beginning for prediction. These kinds of discoveries—plus many others that may be less entertaining yet are equally powerful—form the building blocks for a predictive model.
Freakonomics' objective is to elucidate. One of its coauthors calls the other an "intellectual detective" who, per their book's subtitle, "Explores the hidden side of everything." Definitely cool. But predictive analytics' objective is a entirely different.
More than just striving to elucidate, predictive analytics runs the world. It actively drives operational decisions central to organizations' functions in order to improve the very way in which companies, government agencies, healthcare facilities, and law enforcement serve and treat us. The technology's prediction of human behavior determines, millions of times a day, whom to call, mail, approve, test, diagnose, warn, investigate, incarcerate, set up on a date, and medicate. By answering this mountain of questions, predictive analytics combats financial risk, fortifies healthcare, conquers spam, toughens crime-fighting, boosts sales—and may in fact answer the biggest question of all: How can we improve the effectiveness of all these massive functions across business and government sectors?
To this end, predictive analytics techniques focus narrowly, striving to discover from data only insights that serve for the particular predictive objective at hand, such as whether each individual will consume, work, love, procreate, vote, mess up, commit a crime, or even die. The underlying analytical learning process builds on individual insights such as those in the table above to form a predictive model, a mechanism (such as a formula or set of business rules) that can consider in concert the multitude of factors known about an individual in order to form its "best bet" prediction for that individual. These predictive capabilities are developed by following a number-crunching, trial-and-error process that has its roots in statistics and computer science.
Of course, the elucidating insights from Freakonomics, economics, and all social sciences for that matter, do inform decision making. But it's not the same thing. Social science insights drive a smaller number of strategic decisions, usually in an ad hoc manner in which decision makers may or may not consider such insights in doing their jobs. In contrast, predictive analytics drives a large number of operational and tactical decisions in a systematic or automatic fashion. Social sciences set out to inform we humans; predictive analytics sets out systematically improve organizational functions on a massive scale.
What's So Big About Big Data?
This endeavor to apply prediction addresses a burning question: With all the excitement over "big data"—which has recently witnessed such dramatic hype as to assume the status of a Movement—what is the value, the function, the purpose? Answer: The most actionable thing to be gained from data is prediction. This is achieved by analytically learning from data how to render predictions for each individual. In this way, predictive analytics is the means to tap big data's full potential.
Big data grows wildly, organically. In this heyday of big data, we thus make hay of this valuable overgrowth of "weeds," the residual side effect of organizations conducting business as usual. This is in contrast to economics and other social sciences, which often must actively conduct experimental studies in order to accrue pertinent data.
Isn't Prediction Impossible?
Freakonomics coauthor Stephen Dubner is skeptical. "The first step toward predicting the future is admitting you can't," he once said ("Why Can't We Predict Earthquakes?" Freakonomics Radio, 2011). As the Danish physicist Niels Bohr put it, "Prediction is very difficult, especially if it's about the future." People are unpredictable. Better data helps prediction more than better math and models do, but we'll never predict with high accuracy.
Fortunately, we don't need to. The massive operations of companies and government agencies benefit greatly by playing the odds better, tipping the numbers games they already play in our favor. Simply predicting a nice amount better than guessing does the trick; a hazy view of what's to come outperforms complete darkness by a landslide. I call this The Prediction Effect: A little prediction goes a long way.
I Never Meta-Science I Didn't Like
Although illuminating insights often emerge as a secondary effect, predictive analytics strives not so much to understand how the world works as to actively improve how it works. This makes predictive analytics a kind of "metascience" that transcends the taxonomy of natural and social sciences, abstracting across them by learning from any and all data sources that would typically serve biology, criminology, economics (and Freakonomics), education, epidemiology, medicine, political science, psychology, or sociology. Predictive analytics' mission is to engineer solutions. As for the data employed and the insights gained, the tactic in play is, "whatever works."
More about Predictive Analytics:
Eric Siegel, Ph.D., is the founder of Predictive Analytics World (www.pawcon.com)—coming in 2013 and 2014 to Toronto, San Francisco, Chicago, Washington D.C., Boston, Berlin, and London—and the author of Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die (February 2013, published by Wiley). Interested in employing predictive analytics at your organization? Get started with the Predictive Analytics Guide (www.pawcon.com/guide).
Geologists discover a rhythm to major geologic events.
- It appears that Earth has a geologic "pulse," with clusters of major events occurring every 27.5 million years.
- Working with the most accurate dating methods available, the authors of the study constructed a new history of the last 260 million years.
- Exactly why these cycles occur remains unknown, but there are some interesting theories.
Our hearts beat at a resting rate of 60 to 100 beats per minute. Lots of other things pulse, too. The colors we see and the pitches we hear, for example, are due to the different wave frequencies ("pulses") of light and sound waves.
Now, a study in the journal Geoscience Frontiers finds that Earth itself has a pulse, with one "beat" every 27.5 million years. That's the rate at which major geological events have been occurring as far back as geologists can tell.
A planetary calendar has 10 dates in red
Credit: Jagoush / Adobe Stock
According to lead author and geologist Michael Rampino of New York University's Department of Biology, "Many geologists believe that geological events are random over time. But our study provides statistical evidence for a common cycle, suggesting that these geologic events are correlated and not random."
The new study is not the first time that there's been a suggestion of a planetary geologic cycle, but it's only with recent refinements in radioisotopic dating techniques that there's evidence supporting the theory. The authors of the study collected the latest, best dating for 89 known geologic events over the last 260 million years:
- 29 sea level fluctuations
- 12 marine extinctions
- 9 land-based extinctions
- 10 periods of low ocean oxygenation
- 13 gigantic flood basalt volcanic eruptions
- 8 changes in the rate of seafloor spread
- 8 times there were global pulsations in interplate magmatism
The dates provided the scientists a new timetable of Earth's geologic history.
Tick, tick, boom
Credit: New York University
Putting all the events together, the scientists performed a series of statistical analyses that revealed that events tend to cluster around 10 different dates, with peak activity occurring every 27.5 million years. Between the ten busy periods, the number of events dropped sharply, approaching zero.
Perhaps the most fascinating question that remains unanswered for now is exactly why this is happening. The authors of the study suggest two possibilities:
"The correlations and cyclicity seen in the geologic episodes may be entirely a function of global internal Earth dynamics affecting global tectonics and climate, but similar cycles in the Earth's orbit in the Solar System and in the Galaxy might be pacing these events. Whatever the origins of these cyclical episodes, their occurrences support the case for a largely periodic, coordinated, and intermittently catastrophic geologic record, which is quite different from the views held by most geologists."
Assuming the researchers' calculations are at least roughly correct — the authors note that different statistical formulas may result in further refinement of their conclusions — there's no need to worry that we're about to be thumped by another planetary heartbeat. The last occurred some seven million years ago, meaning the next won't happen for about another 20 million years.
Research shows that those who spend more time speaking tend to emerge as the leaders of groups, regardless of their intelligence.
- A new study proposes the "babble hypothesis" of becoming a group leader.
- Researchers show that intelligence is not the most important factor in leadership.
- Those who talk the most tend to emerge as group leaders.
If you want to become a leader, start yammering. It doesn't even necessarily matter what you say. New research shows that groups without a leader can find one if somebody starts talking a lot.
This phenomenon, described by the "babble hypothesis" of leadership, depends neither on group member intelligence nor personality. Leaders emerge based on the quantity of speaking, not quality.
Researcher Neil G. MacLaren, lead author of the study published in The Leadership Quarterly, believes his team's work may improve how groups are organized and how individuals within them are trained and evaluated.
"It turns out that early attempts to assess leadership quality were found to be highly confounded with a simple quantity: the amount of time that group members spoke during a discussion," shared MacLaren, who is a research fellow at Binghamton University.
While we tend to think of leaders as people who share important ideas, leadership may boil down to whoever "babbles" the most. Understanding the connection between how much people speak and how they become perceived as leaders is key to growing our knowledge of group dynamics.
The power of babble
The research involved 256 college students, divided into 33 groups of four to ten people each. They were asked to collaborate on either a military computer simulation game (BCT Commander) or a business-oriented game (CleanStart). The players had ten minutes to plan how they would carry out a task and 60 minutes to accomplish it as a group. One person in the group was randomly designated as the "operator," whose job was to control the user interface of the game.
To determine who became the leader of each group, the researchers asked the participants both before and after the game to nominate one to five people for this distinction. The scientists found that those who talked more were also more likely to be nominated. This remained true after controlling for a number of variables, such as previous knowledge of the game, various personality traits, or intelligence.
How leaders influence people to believe | Michael Dowling | Big Think www.youtube.com
In an interview with PsyPost, MacLaren shared that "the evidence does seem consistent that people who speak more are more likely to be viewed as leaders."
Another find was that gender bias seemed to have a strong effect on who was considered a leader. "In our data, men receive on average an extra vote just for being a man," explained MacLaren. "The effect is more extreme for the individual with the most votes."
The great theoretical physicist Steven Weinberg passed away on July 23. This is our tribute.
- The recent passing of the great theoretical physicist Steven Weinberg brought back memories of how his book got me into the study of cosmology.
- Going back in time, toward the cosmic infancy, is a spectacular effort that combines experimental and theoretical ingenuity. Modern cosmology is an experimental science.
- The cosmic story is, ultimately, our own. Our roots reach down to the earliest moments after creation.
When I was a junior in college, my electromagnetism professor had an awesome idea. Apart from the usual homework and exams, we were to give a seminar to the class on a topic of our choosing. The idea was to gauge which area of physics we would be interested in following professionally.
Professor Gilson Carneiro knew I was interested in cosmology and suggested a book by Nobel Prize Laureate Steven Weinberg: The First Three Minutes: A Modern View of the Origin of the Universe. I still have my original copy in Portuguese, from 1979, that emanates a musty tropical smell, sitting on my bookshelf side-by-side with the American version, a Bantam edition from 1979.
Inspired by Steven Weinberg
Books can change lives. They can illuminate the path ahead. In my case, there is no question that Weinberg's book blew my teenage mind. I decided, then and there, that I would become a cosmologist working on the physics of the early universe. The first three minutes of cosmic existence — what could be more exciting for a young physicist than trying to uncover the mystery of creation itself and the origin of the universe, matter, and stars? Weinberg quickly became my modern physics hero, the one I wanted to emulate professionally. Sadly, he passed away July 23rd, leaving a huge void for a generation of physicists.
What excited my young imagination was that science could actually make sense of the very early universe, meaning that theories could be validated and ideas could be tested against real data. Cosmology, as a science, only really took off after Einstein published his paper on the shape of the universe in 1917, two years after his groundbreaking paper on the theory of general relativity, the one explaining how we can interpret gravity as the curvature of spacetime. Matter doesn't "bend" time, but it affects how quickly it flows. (See last week's essay on what happens when you fall into a black hole).
The Big Bang Theory
For most of the 20th century, cosmology lived in the realm of theoretical speculation. One model proposed that the universe started from a small, hot, dense plasma billions of years ago and has been expanding ever since — the Big Bang model; another suggested that the cosmos stands still and that the changes astronomers see are mostly local — the steady state model.
Competing models are essential to science but so is data to help us discriminate among them. In the mid 1960s, a decisive discovery changed the game forever. Arno Penzias and Robert Wilson accidentally discovered the cosmic microwave background radiation (CMB), a fossil from the early universe predicted to exist by George Gamow, Ralph Alpher, and Robert Herman in their Big Bang model. (Alpher and Herman published a lovely account of the history here.) The CMB is a bath of microwave photons that permeates the whole of space, a remnant from the epoch when the first hydrogen atoms were forged, some 400,000 years after the bang.
The existence of the CMB was the smoking gun confirming the Big Bang model. From that moment on, a series of spectacular observatories and detectors, both on land and in space, have extracted huge amounts of information from the properties of the CMB, a bit like paleontologists that excavate the remains of dinosaurs and dig for more bones to get details of a past long gone.
How far back can we go?
Confirming the general outline of the Big Bang model changed our cosmic view. The universe, like you and me, has a history, a past waiting to be explored. How far back in time could we dig? Was there some ultimate wall we cannot pass?
Because matter gets hot as it gets squeezed, going back in time meant looking at matter and radiation at higher and higher temperatures. There is a simple relation that connects the age of the universe and its temperature, measured in terms of the temperature of photons (the particles of visible light and other forms of invisible radiation). The fun thing is that matter breaks down as the temperature increases. So, going back in time means looking at matter at more and more primitive states of organization. After the CMB formed 400,000 years after the bang, there were hydrogen atoms. Before, there weren't. The universe was filled with a primordial soup of particles: protons, neutrons, electrons, photons, and neutrinos, the ghostly particles that cross planets and people unscathed. Also, there were very light atomic nuclei, such as deuterium and tritium (both heavier cousins of hydrogen), helium, and lithium.
So, to study the universe after 400,000 years, we need to use atomic physics, at least until large clumps of matter aggregate due to gravity and start to collapse to form the first stars, a few millions of years after. What about earlier on? The cosmic history is broken down into chunks of time, each the realm of different kinds of physics. Before atoms form, all the way to about a second after the Big Bang, it's nuclear physics time. That's why Weinberg brilliantly titled his book The First Three Minutes. It is during the interval between one-hundredth of a second and three minutes that the light atomic nuclei (made of protons and neutrons) formed, a process called, with poetic flair, primordial nucleosynthesis. Protons collided with neutrons and, sometimes, stuck together due to the attractive strong nuclear force. Why did only a few light nuclei form then? Because the expansion of the universe made it hard for the particles to find each other.
What about the nuclei of heavier elements, like carbon, oxygen, calcium, gold? The answer is beautiful: all the elements of the periodic table after lithium were made and continue to be made in stars, the true cosmic alchemists. Hydrogen eventually becomes people if you wait long enough. At least in this universe.
In this article, we got all the way up to nucleosynthesis, the forging of the first atomic nuclei when the universe was a minute old. What about earlier on? How close to the beginning, to t = 0, can science get? Stay tuned, and we will continue next week.
To Steven Weinberg, with gratitude, for all that you taught us about the universe.
Long before Alexandria became the center of Egyptian trade, there was Thônis-Heracleion. But then it sank.