from the world's big
22 Ways Algorithms Know How You'll Behave Before You Do
Prediction is reinventing industries and running the world. More and more, predictive analytics drives commerce, manufacturing, healthcare, government, and law enforcement.
The future is the ultimate unknown. It's everything that hasn't happened yet.
Prediction as a capability is booming. It reinvents industries and runs the world. More and more, predictive analytics drives commerce, manufacturing, healthcare, government, and law enforcement. In these spheres, organizations operate more effectively by way of predicting behavior—i.e., the outcome for each individual customer, employee, patient, voter, and suspect.
Predictive analytics’ expansive deployment has taken hold. Accenture and Forrester both report that predictive analytics' adoption has more than doubled in recent years. Transparency Market Research projects the predictive analytics market will reach $6.5 billion within a few years. Predictive analytics is becoming a standard safeguard for business, and even demand from consumers for its capabilities promises to surge.
New groundbreaking stories of predictive analytics in action are pouring in. A few key ingredients have opened these floodgates:
I've listed below a slew of examples—from the likes of Facebook, the NSA, Hillary for America, Uber, Airbnb, Google, Shell, UPS, Amazon.com, Coned, Yahoo!, and the U.S. government.
22 EXAMPLES OF PREDICTIVE ANALYTICS:
Which Facebook posts you will like in order to optimize your news feed
Facebook: Predicts which of 1,500 candidate posts (on average) will be most interesting to you in order to personalize your news feed. To optimize the order of content items, the News Feed ranking algorithm weights around 100,000 factors such as recency, likes, clicks, shares, comments, time spent on posts, poster popularity, your affinity for the poster and content area, and measures of relevance and trustworthiness. This intensifies the “addictive” engagement, with two-thirds of Facebook’s 1.44 billion monthly users logging in daily.
Who’s in a photo (aka facial recognition)
Facebook: Improved the state of the art for identifying people from photos to virtually the same performance level as a human: Given two face images, it can determine whether they’re the same person with 97 percent accuracy. Facial recognition helps users tag photos, which they do more than 100 million times a day. The company has also developed predictive models to identify people even if it can’t see the face, achieving 83 percent accuracy when faces are at least partially obscured half of the time, based on elements such as clothing, hair, and pose.
Clicks in order to select which to display
Facebook: In order to increase revenue from its pay-per-click advertisers, predicts ad clicks based on user attributes, device used, and contextual factors.
The National Security Agency: Obtained software solutions for and core competency in predictive analytics. It’s clear that the NSA considers predictive analytics a strategic priority as a means to target investigation activities by automatically discovering previously unknown potential suspects.
Where you are going
Uber: Can predict the specific destination address of San Francisco riders based on exact drop-off location with 74 percent accuracy, despite, for example, how many businesses there are within 100 meters in a typical city area (just taking the closest candidate address achieves 44 percent accuracy).
Acceptance of booking request in order to match guests to hosts
Airbnb: Rank orders accommodations that fulfill a user search in part by the predicted probability each host would accept the user’s booking request. By surfacing likely matches more prominently, the company increased booking conversions by nearly 4 percent—a significant gain considering its estimated annual booking of over 12 million guest nights.
Accommodation bookings at a given price–for dynamic pricing
Airbnb: Suggests each day’s price for an accommodation listing (the “Price Tips” feature) by way of predicting whether the listing will be booked—predicted demand directly informs optimal pricing. Bookings are predicted by day of the week, seasonality, and local events, as well as characteristics of the listing such as the neighborhood, size, amenities, key words like “beach,” number of reviews, and photographs. Hosts who set prices within 5 percent of the suggestions improve their chance of booking by a factor of nearly four.
Spam to send it to your spam folder
Google: Decreased Gmail’s prevalence and false positive rate of spam from disruptive (in 2004) down to negligible.
Oil refinery safety incidents
Shell: Predicts the number of safety incidents per team of workers at oil refineries, globally. One example discovery: Increased employee engagement predicts fewer incidents; one percentage point increase in team employee engagement is associated with a 4 percent decrease in the number of safety incidents per FTE.
RightShip: Predicts dangerous or costly maritime incidents in order to assess vessel risk that informs shipment decisions when selecting between vessels. The 10 percent highest-risk vessels are three times more likely than average to experience an incident in the next 12 months, and are 16 times more likely to incur a casualty than the 10 percent least risky. Risk assessment is based on vessel age, type, carrying capacity, origin, registration, ownership, management, and other factors.
Deliveries—which addresses will receive a package
UPS: Cut 85 million miles from annual delivery vehicle driving with a semiautomatic optimization system that plans vehicle/package assignments, as well as package placement within the vehicle, based upon each day’s analytically predicted delivery destinations.
Amazon.com: Thirty-five percent of sales come from product recommendations. The company may also develop “anticipatory shipping” that would proactively place packages before they are ordered at hubs or on trucks in order to reduce delays between ordering and receiving purchases, for which it has obtained a patent.
Spotify: Is augmenting its song recommendation algorithm to incorporate musical attributes.
Hillary for America 2016 Campaign: Given Obama’s success with persuasion modeling in 2012, Hillary Clinton’s 2016 campaign appears to be planning to employ it as well. Analytics job postings reveal they’re going to be “helping the campaign determine which voters to target for persuasion.”
Restaurant health code violations via Yelp reviews
City of Boston: Sponsored a competition that generated the ability to predict whether a restaurant will have more violations than normal with 75 percent accuracy, in part by way of discovering clues within Yelp reviews, in order to target city health department inspections. Similar work for Seattle restaurants distinguished severe violators with 82 percent accuracy.
Lead poisoning from paint
City of Chicago: Identified 5 percent of homes that are at more than twice the risk for lead poisoning incidents than average based on the age of the house, the history of lead paint exposure at that address, the economic conditions of the neighborhood, and other factors. This serves as an early warning system to proactively flag, as an improvement over the more common reactive steps taken after a positive test for poisoning. The risk scores serve to target homes for inspection and children for testing, and could help people determine safer homes to move to.
City of New York: Targets the fire inspections of its 330,000 inspectable buildings with a predictive model that assesses risk based on about 60 factors.
Manhole explosions and fires
Con Edison: Predicts dangerous manhole explosions and fires in New York City, identifying a 2 percent of manholes that have a 5.5 times greater than average risk of an incident.
Yahoo! Labs: Developed a model to categorize photographic portraits as to the subjective human aesthetic of beauty with 64 percent accuracy based on various image attributes. The study determined “that race, gender, and age are largely uncorrelated with photographic beauty.”
Overpriced property leases
U.S. Postal Service Office of Inspector General: Predicted the amount paid over market value for each of their 26,000 leased facilities (e.g., retail unit, plant, warehouse). Targeting facilities in the Northeast Region, USPS auditors projected that 250 of the leases predicted as most overpaid represent a potential savings of $6.6 million by way of renegotiating their next year of lease terms.
Surgical site infections
University of Iowa Hospitals and Clinics: Identifies cases greater than four times as likely to develop surgical-site infections. Targeting anti-infection therapy accordingly reduces the cost of each colorectal surgical procedure an average of $1,300 and will provide a projected annual savings of several million dollars once expanded to other forms of surgery.
Hopper: Predicts airfare changes in order to recommend to consumers whether to buy or wait. Ninety-five percent of these predictions save the consumer money or do no worse than the first price seen, saving users an average 10 percent on ticket price.
This flood of predictive activity gains its potential simply because prediction boasts an inherent generality—there are just so many conceivable ways to make use of it. Want to come up with your own new innovative use for predictive analytics? You need only two ingredients. Each application of predictive analytics is defined by:
1. What’s predicted: the kind of behavior—i.e., action, event, or happening—to predict for each individual (e.g., person, Facebook post, photo, ad, trip destination, marine vessel, safety incident, transaction, or other organizational element).
2. What’s done about it: the decisions driven by prediction; the action taken by the organization in response to or informed by each prediction.
We can confidently predict more prediction. Every few months, another big story about predictive analytics rolls off the presses. We’re sure to see the opportunities continue to grow and surprise. Come what may, only time will tell what we’ll tell of time to come.
These examples are new in this year's Revised and Updated edition of my book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die. With these newly added cases, the book's central compendium of mini-case studies has grown to 182 entries (most were sourced from presentations at Predictive Analytics World, the event series I founded—for more information about each example, access the book's Notes PDF, available at www.PredictiveNotes.com, and search by organization name).
Innovation in manufacturing has crawled since the 1950s. That's about to speed up.
Why do so many people encounter beings after smoking large doses of DMT?
- DMT is arguably the most powerful psychedelic drug on the planet, capable of producing intense hallucinations.
- Researchers recently surveyed more than 2,000 DMT users about their encounters with 'entities' while tripping, finding that respondents often considered these strange encounters to be positive and meaningful.
- The majority of respondents believed the beings they encountered were not hallucinations.
What are DMT beings?<p>Do DMT entities actually exist in some other dimension, or are they hallucinations that the brain generates when its visual processing system is overwhelmed by a powerful tryptamine?<br></p><p>The late American ethnobotanist Terence McKenna believed that DMT beings — which he called "machine elves" — were real. Here's how he once <a href="https://www.ranker.com/list/dmt-machine-elves-facts/inigo-gonzalez" target="_blank">described</a> one of his DMT experiences:</p><p style="margin-left: 20px;">"I sank to the floor. I [experienced] this hallucination of tumbling forward into these fractal geometric spaces made of light and then I found myself in the equivalent of the Pope's private chapel and there were insect elf machines proffering strange little tablets with strange writing on them, and I was aghast, completely appalled, because [in] a matter of seconds... my entire expectation of the nature of the world was just being shredded in front of me. I've never actually gotten over it.</p><p style="margin-left: 20px;">These self-transforming machine elf creatures were speaking in a colored language which condensed into rotating machines that were like Fabergé eggs but crafted out of luminescent superconducting ceramics and liquid crystal gels. All this stuff was just so weird and so alien and so un-English-able that it was a complete shock — I mean, the literal turning inside out of [my] intellectual universe!"</p><p>McKenna believed machine elves exist in alternate realities, which form a "<a href="https://www.irishtimes.com/culture/books/old-favourites-the-archaic-revival-1991-by-terence-mckenna-1.3924887" target="_blank">raging universe of active intelligence that is transhuman, hyperdimensional, and extremely alien.</a>" But he was far from the first to believe that DMT is a doorway to other realms.</p><p>Indigenous peoples of the Amazon basin have used ayahuasca in religious ceremonies for centuries, though no one is quite sure when they first started experimenting with the psychedelic brew. The Jibaro people of the Ecuadorian rainforest believed ayahuasca allowed regular people, not just shamans, to <a href="https://atrium.lib.uoguelph.ca/xmlui/bitstream/handle/10214/17902/RichardsonG_202004_HonThesis.pdf?sequence=3" target="_blank">speak directly to the gods</a>. The 19th-century Ecuadorian geographer Villavicencio wrote of other Amazonian shamans who used ahaysuca (known as the "vine of the dead") to contact spirits and foresee enemy battle plans.</p><p>In the West, research on DMT experiences has been sparse yet interesting. The psychiatrist Rick Strassman conducted some of the first human DMT trials at the University of New Mexico in the early 1990s. He found that <a href="https://www.erowid.org/chemicals/dmt/dmt_article3.shtml" target="_blank">"at least half"</a> of his research subjects had encountered some form of entity after taking DMT.</p><p style="margin-left: 20px;">"I was neither intellectually nor emotionally prepared for the frequency with which contact with beings occurred in our studies, nor the often utterly bizarre nature of these experiences," Strassman wrote in his book "DMT The Spirit Molecule".</p>
Manuel Medir / Getty<p style="margin-left: 20px;">"Whenever I tried to pull any information out of the entities regarding themselves, the data that was given up was always relevant only to me. The elves could not give me any piece of data I did not already know, nor could their existence be sustained under any kind of prolonged scrutiny."</p><p>It's also worth noting that not all people who smoke DMT see beings, and that some see beings that look <a href="https://www.erowid.org/chemicals/dmt/dmt_article3.shtml" target="_blank">nothing like elves or aliens</a>. The diversity of these reports seems to count against the argument that DMT beings exist in some objective alternate reality.</p><p>In other words, if DMT beings exist in some other dimension, shouldn't they appear the same to anyone who visits that dimension? Or do the beings assume a different appearance based on who's looking? Or are there many types of beings in the DMT universe, but most look like elves? </p><p>You might start seeing elves just trying to sort this stuff out.</p><p>Ultimately, nobody knows exactly why DMT beings take the forms they do, or whether they're just figments of overstimulated imaginations. And the answers might be beside the point. </p><p>In the recent survey, 60 percent of participants said their encounter with DMT beings "produced a desirable alteration in their conception of reality whereas only 1% indicated an undesirable alteration in their conception of reality."</p><p>DMT beings may be nothing more than projections of the subconscious mind. But these bizarre encounters do help some people find real meaning, whether it's through personal revelation or the raw power of ontological shock.</p>
So far, 30 student teams have entered the Indy Autonomous Challenge, scheduled for October 2021.
- The Indy Autonomous Challenge will task student teams with developing self-driving software for race cars.
- The competition requires cars to complete 20 laps within 25 minutes, meaning cars would need to average about 110 mph.
- The organizers say they hope to advance the field of driverless cars and "inspire the next generation of STEM talent."
Indy Autonomous Challenge<p>Completing the race in 25 minutes means the cars will need to average about 110 miles per hour. So, while the race may end up being a bit slower than a typical Indy 500 competition, in which winners average speeds of over 160 mph, it's still set to be the fastest autonomous race featuring full-size cars.</p><p style="margin-left: 20px;">"There is no human redundancy there," Matt Peak, managing director for Energy Systems Network, a nonprofit that develops technology for the automation and energy sectors, told the <a href="https://www.post-gazette.com/business/tech-news/2020/06/01/Indy-Autonomous-Challenge-Indy-500-Indianapolis-Motor-Speedway-Ansys-Aptiv-self-driving-cars/stories/202005280137" target="_blank">Pittsburgh Post-Gazette</a>. "Either your car makes this happen or smash into the wall you go."</p>
Illustration of the Indy Autonomous Challenge
Indy Autonomous Challenge<p>The Indy Autonomous Challenge <a href="https://www.indyautonomouschallenge.com/rules" target="_blank">describes</a> itself as a "past-the-post" competition, which "refers to a binary, objective, measurable performance rather than a subjective evaluation, judgement, or recognition."</p><p>This competition design was inspired by the 2004 DARPA Grand Challenge, which tasked teams with developing driverless cars and sending them along a 150-mile route in Southern California for a chance to win $1 million. But that prize went unclaimed, because within a few hours after starting, all the vehicles had suffered some kind of critical failure.</p>
Indianapolis Motor Speedway
Indy Autonomous Challenge<p>One factor that could prevent a similar outcome in the upcoming race is the ability to test-run cars on a virtual racetrack. The simulation software company Ansys Inc. has already developed a model of the Indianapolis Motor Speedway on which teams will test their algorithms as part of a series of qualifying rounds.</p><p style="margin-left: 20px;">"We can create, with physics, multiple real-life scenarios that are reflective of the real world," Ansys President Ajei Gopal told <a href="https://www.wsj.com/articles/autonomous-vehicles-to-race-at-indianapolis-motor-speedway-11595237401?mod=e2tw" target="_blank">The Wall Street Journal</a>. "We can use that to train the AI, so it starts to come up to speed."</p><p>Still, the race could reveal that self-driving cars aren't quite ready to race at speeds of over 110 mph. After all, regular self-driving cars already face enough logistical and technical roadblocks, including <a href="https://www.bbc.com/news/technology-53349313#:~:text=Tesla%20will%20be%20able%20to,no%20driver%20input%2C%20he%20said." target="_blank">crumbling infrastructure, communication issues</a> and the <a href="https://bigthink.com/paul-ratner/would-you-ride-in-a-car-thats-programmed-to-kill-you" target="_self">fateful moral decisions driverless cars will have to make in split seconds</a>.</p>But the Indy Autonomous Challenge <a href="https://static1.squarespace.com/static/5da73021d0636f4ec706fa0a/t/5dc0680c41954d4ef41ec2b2/1572890638793/Indy+Autonomous+Challenge+Ruleset+-+v5NOV2019+%282%29.pdf" target="_blank">says</a> its main goal is to advance the industry, by challenging "students around the world to imagine, invent, and prove a new generation of automated vehicle (AV) software and inspire the next generation of STEM talent."
A new Harvard study finds that the language you use affects patient outcome.
- A study at Harvard's McLean Hospital claims that using the language of chemical imbalances worsens patient outcomes.
- Though psychiatry has largely abandoned DSM categories, professor Joseph E Davis writes that the field continues to strive for a "brain-based diagnostic system."
- Chemical explanations of mental health appear to benefit pharmaceutical companies far more than patients.
Challenging the Chemical Imbalance Theory of Mental Disorders: Robert Whitaker, Journalist<span style="display:block;position:relative;padding-top:56.25%;" class="rm-shortcode" data-rm-shortcode-id="41699c8c2cb2aee9271a36646e0bee7d"><iframe type="lazy-iframe" data-runner-src="https://www.youtube.com/embed/-8BDC7i8Yyw?rel=0" width="100%" height="auto" frameborder="0" scrolling="no" style="position:absolute;top:0;left:0;width:100%;height:100%;"></iframe></span><p>This is a far cry from Howard Rusk's 1947 NY Times editorial calling for mental healt</p><p>h disorders to be treated similarly to physical disease (such as diabetes and cancer). This mindset—not attributable to Rusk alone; he was merely relaying the psychiatric currency of the time—has dominated the field for decades: mental anguish is a genetic and/or chemical-deficiency disorder that must be treated pharmacologically.</p><p>Even as psychiatry untethered from DSM categories, the field still used chemistry to validate its existence. Psychotherapy, arguably the most efficient means for managing much of our anxiety and depression, is time- and labor-intensive. Counseling requires an empathetic and wizened ear to guide the patient to do the work. Ingesting a pill to do that work for you is more seductive, and easier. As Davis writes, even though the industry abandoned the DSM, it continues to strive for a "brain-based diagnostic system." </p><p>That language has infiltrated public consciousness. The team at McLean surveyed 279 patients seeking acute treatment for depression. As they note, the causes of psychological distress have constantly shifted over the millennia: humoral imbalance in the ancient world; spiritual possession in medieval times; early childhood experiences around the time of Freud; maladaptive thought patterns dominant in the latter half of last century. While the team found that psychosocial explanations remain popular, biogenetic explanations (such as the chemical imbalance theory) are becoming more prominent. </p><p>Interestingly, the 80 people Davis interviewed for his book predominantly relied on biogenetic explanations. Instead of doctors diagnosing patients, as you might expect, they increasingly serve to confirm what patients come in suspecting. Patients arrive at medical offices confident in their self-diagnoses. They believe a pill is the best course of treatment, largely because they saw an advertisement or listened to a friend. Doctors too often oblige without further curiosity as to the reasons for their distress. </p>
Image: Illustration Forest / Shutterstock<p>While medicalizing mental health softens the stigma of depression—if a disorder is inheritable, it was never really your fault—it also disempowers the patient. The team at McLean writes,</p><p style="margin-left: 20px;">"More recent studies indicate that participants who are told that their depression is caused by a chemical imbalance or genetic abnormality expect to have depression for a longer period, report more depressive symptoms, and feel they have less control over their negative emotions."</p><p>Davis points out the language used by direct-to-consumer advertising prevalent in America. Doctors, media, and advertising agencies converge around common messages, such as everyday blues is a "real medical condition," everyone is susceptible to clinical depression, and drugs correct underlying somatic conditions that you never consciously control. He continues,</p><p style="margin-left: 20px;">"Your inner life and evaluative stance are of marginal, if any, relevance; counseling or psychotherapy aimed at self-insight would serve little purpose." </p><p>The McLean team discovered a similar phenomenon: patients expect little from psychotherapy and a lot from pills. When depression is treated as the result of an internal and immutable essence instead of environmental conditions, behavioral changes are not expected to make much difference. Chemistry rules the popular imagination.</p>