Once a week.
Subscribe to our weekly newsletter.
The Data-Driven Economy
Consumers today are knowingly and unknowingly providing businesses with more data than they've ever been capable of collecting before. The analysis of this information could have profound implications for business.
Studies have shown that people who have recently read online obituaries tend to be higher purchasers of weekend rental cars. Why this is true isn't exactly clear to Dave Morgan, founder of Tacoda Inc., an online advertising company that was acquired by Aol. in 2007 for $275 million. But the correlation in the data is significant enough that Avis, Hertz, and Enterprise Rent-a-Car ads should start appearing in front of you soon after you have read about the passing of an old friend, a loved one, or (as is often the case when reading obituaries) someone you didn't know at all.
Consumers today are knowingly and unknowingly providing businesses with more data than they've ever been capable of collecting before. Internet entrepreneurs, privacy analysts, and business consultants alike believe that for the next fifty years, capitalism around the world will (for better or worse) be focused on sussing out what all this data actually means. "We are finding things that are completely non-intuitive," says Morgan. "This is just the very beginning of this enormous explosion of information being available about what people do, how they react to information, and how they interact with each other."
Most big businesses today are already data businesses at heart. Facebook, which claims to help you connect and share with the people in your life, is worth a reported $33.7 billion largely because 400 million people have shared with it the details of their personal lives. Google processes about a petabyte of information every hour. To put this seemingly insignificant number in perspective, that's 1,125,899,906,842,624 bytes, or about a fifth of all the information delivered in letter form by the USPS in a year. What's worse—or better, if you're in the data business—businesses are not only recording your data while you're surfing the Web. Long gone are the days of a bifurcated online and offline world.
"There are ways that we probably don't think about that we're sharing information," says Mike Spinney, Senior Privacy Analyst at the Ponemon Institute. "Whether it's an E-Z Pass, or a public transit card or swiping your credit card to make micro-purchases." Wal-Mart collects data from more than 278 customer transactions every second.
Our growing use of digital technology is creating so much "data exhaust," as industry insiders call it, that entire economies are and will continue to form purely around the collection, preservation, protection, implementation, and—most importantly—understanding of our data.
"The big trend that's just starting now and will continue for the rest of the foreseeable couple of decades is the creation and manipulation of vast quantities of data in a meaningful way so that you can learn from the data in ways that are actionable," says Jim Cortada, Director at IBM's Institute of Business Value. He equates the problem of data management and analysis to the challenge of herding sheep. "One of the things that's happening is the emergence of data management tools—hundreds of them," says Cortada. "Think about data like a whole bunch of sheep on a hillside—you gotta get them in. Herders use dogs. Businesses are increasingly using software to get the data herd in."
The companies that will thrive most in the coming decades are those that manage to aggregate and recycle their consumers' data toward improving their products or services in an automated fashion. Google, for example, hones its search algorithm with each of the 35,000 queries it receives every second. Amazon tracks the books you buy, but also records your browsing history to better recommend other books you might like to buy. Netflix uses a collaborative filtering process to make suggestions based on data from other people's preferences.*
Jeff Hirsch, CEO of AudienceScience, a data management and audience targeting platform, predicts that recommendation systems that consider not just your purchase history and search behavior, but also seemingly unconnected data—like whether you've read an obituary lately—will soon be digitally ubiquitous. "The consumer is going to be able to go onto a digital device and get exactly what they want," says Hirsch. "We'll look back at the days when we used to have search for what we wanted and think it was archaic." Television, he says, is a medium that could do a better job of "pushing" consumers what they're looking for. "If I watch various shows, why doesn't my T.V. give me the shows I like?" he asks. A company's imagination alone will limit the possibilities of analogous applications for these type of smart recommendation systems, once they begin to find meaningful and actionable information in their data.
This increased amount of targeting and automation comes with its own set of challenges. The May 6th Flash Crash and mortgage crisis in general are just two historical examples of what can happen when businesses rely too heavily on automated processes for managing their risk. Furthermore, some companies may also begin to automate their sales behavior upon data that many might consider private, and in ways that may also be considered predatory. "There has already been very significant use of very personal data," says Morgan. "Credit card companies buy your bank account balance from your bank and they don't buy it anonymously, they buy it with your name and address. They say 'So-and-so bounced three checks in the last month, maybe they need a prepaid credit card.' But you don't know that. All you know is that you get a prepaid credit card in the mail. The analog world is very slow to understand what's going on."
For example, you may be surprised to know that a simple visit to Dictionary.com results in 223 different businesses uploading third party tracking files to your computer, each of which assigns it a unique identifying number that marketing and data-gathering companies use to record your behavior online.** The more sophisticated of these files even record everything that you type into your browser. Over the past few years a robust market has emerged around the sheer buying and selling these files. The marketing and advertising industries have been at the forefront of leveraging these files to more efficiently target advertisements towards online audiences.
Privacy analyst Mike Spinney understands why people may find it creepy for businesses to be collecting so much of their data, but he says the Luddites amongst us should not be too concerned about their personal data being abused by corporate entities. "People may think industry self-regulation does not work — that they're inherently evil — and yet the opposite is true. They recognize there's a fine line and they do not want to cross it." Spinney, Morgan, and Cortada all agree that though the increased use of data may at times seem invasive, successful businesses will always be most concerned with using it to provide their consumers a better product or service.
In future, Morgan predicts we could well reach the "holy grail of advertising," where we only receive ads that we like—and are therefore happy to get them. Cortada wonders whether current forms of technology will be capable of handling the enormous amount of data that the future holds in store. "I am even questioning whether we're going to have computers made out of metal or composite material," he says. "As a human race we're going to need a whole series of different platforms." Biology, he says, provides a good template for foreseeing the complexity of the data feedback systems that we can expect to develop down the road. "If you take a look at how much data there is in living human cells, the patterns of data collection that the body uses to be a successfully functioning organism are colossally larger than anything we do today."
No one knows how long it will take before businesses begin implementing the kinds of complex feedback systems that biologists see in nature, but for now one thing is for certain: the world is sitting at the foot of what will continue to be an unfathomable mountain of data with the potential to profoundly revolutionize much more than just the way that businesses target us with pesky advertisements. There is already so much data, in fact, that the very thought of beginning to mold it into useful information is enough to make one throw their hands up in the air and give up. "We are almost at a point now where trying to do an inventory on all this data is almost a superfluous exercise," says Cortada. "It's like trying to count all the stars in the sky."
— * For a fascinating look at the number of tracking files each of the 50 most popular Web sites place on your computer, explore the data on the Wall Street Journal's "What They Know" Interactive Series.
— ** For a comparison of Amazon and Netflix's recommendation engines, check out this blog post by Chris Dixon.
A Harvard professor's study discovers the worst year to be alive.
- Harvard professor Michael McCormick argues the worst year to be alive was 536 AD.
- The year was terrible due to cataclysmic eruptions that blocked out the sun and the spread of the plague.
- 536 ushered in the coldest decade in thousands of years and started a century of economic devastation.
The past year has been nothing but the worst in the lives of many people around the globe. A rampaging pandemic, dangerous political instability, weather catastrophes, and a profound change in lifestyle that most have never experienced or imagined.
But was it the worst year ever?
Nope. Not even close. In the eyes of the historian and archaeologist Michael McCormick, the absolute "worst year to be alive" was 536.
Why was 536 so bad? You could certainly argue that 1918, the last year of World War I when the Spanish Flu killed up to 100 million people around the world, was a terrible year by all accounts. 1349 could also be considered on this morbid list as the year when the Black Death wiped out half of Europe, with up to 20 million dead from the plague. Most of the years of World War II could probably lay claim to the "worst year" title as well. But 536 was in a category of its own, argues the historian.
It all began with an eruption...
According to McCormick, Professor of Medieval History at Harvard University, 536 was the precursor year to one of the worst periods of human history. It featured a volcanic eruption early in the year that took place in Iceland, as established by a study of a Swiss glacier carried out by McCormick and the glaciologist Paul Mayewski from the Climate Change Institute of The University of Maine (UM) in Orono.
The ash spewed out by the volcano likely led to a fog that brought an 18-month-long stretch of daytime darkness across Europe, the Middle East, and portions of Asia. As wrote the Byzantine historian Procopius, "For the sun gave forth its light without brightness, like the moon, during the whole year." He also recounted that it looked like the sun was always in eclipse.
Cassiodorus, a Roman politician of that time, wrote that the sun had a "bluish" color, the moon had no luster, and "seasons seem to be all jumbled up together." What's even creepier, he described, "We marvel to see no shadows of our bodies at noon."
...that led to famine...
The dark days also brought a period of coldness, with summer temperatures falling by 1.5° C. to 2.5° C. This started the coldest decade in the past 2300 years, reports Science, leading to the devastation of crops and worldwide hunger.
...and the fall of an empire
In 541, the bubonic plague added considerably to the world's misery. Spreading from the Roman port of Pelusium in Egypt, the so-called Plague of Justinian caused the deaths of up to one half of the population of the eastern Roman Empire. This, in turn, sped up its eventual collapse, writes McCormick.
Between the environmental cataclysms, with massive volcanic eruptions also in 540 and 547, and the devastation brought on by the plague, Europe was in for an economic downturn for nearly all of the next century, until 640 when silver mining gave it a boost.
Was that the worst time in history?
Of course, the absolute worst time in history depends on who you were and where you lived.
Native Americans can easily point to 1520, when smallpox, brought over by the Spanish, killed millions of indigenous people. By 1600, up to 90 percent of the population of the Americas (about 55 million people) was wiped out by various European pathogens.
Like all things, the grisly title of "worst year ever" comes down to historical perspective.
A new paper reveals that the Voyager 1 spacecraft detected a constant hum coming from outside our Solar System.
Voyager 1, humanity's most faraway spacecraft, has detected an unusual "hum" coming from outside our solar system. Fourteen billion miles away from Earth, the Voyager's instruments picked up a droning sound that may be caused by plasma (ionized gas) in the vast emptiness of interstellar space.
Launched in 1977, the Voyager 1 space probe — along with its twin Voyager 2 — has been traveling farther and farther into space for over 44 years. It has now breached the edge of our solar system, exiting the heliosphere, the bubble-like region of space influenced by the sun. Now, the spacecraft is moving through the "interstellar medium," where it recorded the peculiar sound.
Stella Koch Ocker, a doctoral student in astronomy at Cornell University, discovered the sound in the data from the Voyager's Plasma Wave System (PWS), which measures electron density. Ocker called the drone coming from plasma shock waves "very faint and monotone," likely due to the narrow bandwidth of its frequency.
While they think the persistent background hum may be coming from interstellar gas, the researchers don't yet know what exactly is causing it. It might be produced by "thermally excited plasma oscillations and quasi-thermal noise."
The new paper from Ocker and her colleagues at Cornell University and the University of Iowa, published in Nature Astronomy, also proposes that this is not the last we'll hear of the strange noise. The scientists write that "the emission's persistence suggests that Voyager 1 may be able to continue tracking the interstellar plasma density in the absence of shock-generated plasma oscillation events."
Voyager Captures Sounds of Interstellar Space www.youtube.com
The researchers think the droning sound may hold clues to how interstellar space and the heliopause, which can be thought of as the solar's system border, may be affecting each other. When it first entered interstellar space, the PWS instrument reported disturbances in the gas caused by the sun. But in between such eruptions is where the researchers spotted the steady signature made by the near-vacuum.
Senior author James Cordes, a professor of astronomy at Cornell, compared the interstellar medium to "a quiet or gentle rain," adding that "in the case of a solar outburst, it's like detecting a lightning burst in a thunderstorm and then it's back to a gentle rain."
More data from Voyager over the next few years may hold crucial information to the origins of the hum. The findings are already remarkable considering the space probe is functioning on technology from the mid-1970s. The craft has about 70 kilobytes of computer memory. It also carries a Golden Record created by a committee chaired by the late Carl Sagan, who taught at Cornell University. The 12-inch gold-plated copper disk record is essentially a time capsule, meant to tell the story of Earthlings to extraterrestrials. It contains sounds and images that showcase the diversity of Earth's life and culture.
A team of scientists managed to install onto a smartphone a spectrometer that's capable of identifying specific molecules — with cheap parts you can buy online.
- Spectroscopy provides a non-invasive way to study the chemical composition of matter.
- These techniques analyze the unique ways light interacts with certain materials.
- If spectrometers become a common feature of smartphones, it could someday potentially allow anyone to identify pathogens, detect impurities in food, and verify the authenticity of valuable minerals.
The quality of smartphone cameras has increased exponentially over the past decade. Today's smartphone cameras can not only capture photos that rival those of stand-alone camera systems but also offer practical applications, like heart-rate measurement, foreign-text translation, and augmented reality.
What's the next major functionality of smartphone cameras? It could be the ability to identify chemicals, drugs, and biological molecules, according to a new study published in the Review of Scientific Instruments.
The study describes how a team of scientists at Texas A&M turned a common smartphone into a "pocket-sized" Raman and emission spectral detector by modifying it with just $50 worth of extra equipment. With the added hardware, the smartphone was able to identify chemicals in the field within minutes.
The technology could have a wide range of applications, including diagnosing certain diseases, detecting the presence of pathogens and dangerous chemicals, identifying impurities in food, and verifying the authenticity of valuable artwork and minerals.
Raman and fluorescence spectroscopy
Raman and fluorescence spectroscopies are techniques for discerning the chemical composition of materials. Both strategies exploit the fact that light interacts with certain types of matter in unique ways. But there are some differences between the two techniques.
As the name suggests, fluorescence spectroscopy measures the fluorescence — that is, the light emitted by a substance when it absorbs light or other electromagnetic radiation — of a given material. It works by shining light on a material, which excites the electrons within the molecules of the material. The electrons then emit fluorescent light toward a filter that measures fluorescence.
The particular spectra of fluorescent light that's emitted can help scientists detect small concentrations of particular types of biological molecules within a material. But some biomolecules, such as RNA and DNA, don't emit fluorescent light, or they only do so at extremely low levels. That's where Raman spectroscopy comes into play.
Raman spectroscopy involves shooting a laser at a sample and observing how the light scatters. When light hits molecules, the atoms within the molecules vibrate and photons get scattered. Most of the scattered light is of the same wavelength and color as the original light, so it provides no information. But a tiny fraction of the light gets scattered differently; that is, the wavelength and color are different. Known as Raman scattering, this is extremely useful because it provides highly precise information about the chemical composition of the molecule. In other words, all molecules have a unique Raman "fingerprint."
Creating an affordable, pocket-sized spectrometer
To build the spectrometer, the researchers connected a smartphone to a laser and a series of plastic lenses. The smartphone camera was placed facing a transmission diffraction grating, which splits incoming light into its constituent wavelengths and colors. After a laser is fired into a sample, the scattered light is diffracted through this grating, and the smartphone camera analyzes the light on the other side.
Schematic diagram of the designed system.Credit: Dhankhar et al.
To test the spectrometer, the researchers analyzed a range of sample materials, including carrots and bacteria. The laser used in the spectrometer emits a wavelength that's readily absorbed by the pigments in carrots and bacteria, which is why these materials were chosen.
The results showed that the smartphone spectrometer was able to correctly identify the materials, but it wasn't quite as effective as the best commercially available Raman spectrometers. The researchers noted that their system might be improved by using specific High Dynamic Range (HDR) smartphone camera applications.
Ultimately, the study highlights how improving the fundamentals of a technology, like smartphone cameras, can lead to a surprisingly wide range of useful applications.
"This inexpensive yet accurate recording pocket Raman system has the potential of being an integral part of ubiquitous cell phones that will make it possible to identify chemical impurities and pathogens, in situ within minutes," the researchers concluded.