Black swans: can we learn to predict unpredictable events?

This article was originally published by our sister site, Freethink.

A black swan event has, unfortunately, nothing to do with Natalie Portman or Western Australia; it describes an event that is extremely unlikely, but can cause massive upheaval. For example, the 2008 global recession or, say, pretty much all of 2020.

By definition, no one sees a black swan event coming — but researchers at Stanford are attempting to change that. They are building a computational method to try and predict when the next graceful neck will raise its head.

“This work is exciting because it’s a chance to take the knowledge and the computational tools that we’re building in the lab and use those to better understand — even predict or forecast — what happens in the world surrounding us,” Bo Wang, assistant professor of bioengineering at Stanford and senior author of the study, told Stanford News.

Predicting unprecedented events

Published in PLOS Computational Biology, the method is based on natural systems, Stanford News reports, and could be useful in environmental and healthcare research. (Applications in other fields with black swan events, like economics and politics, may be further down the pike.)

According to Stanford News, the work was inspired by Samuel Bray, a research assistant in Wang’s lab.

“Existing methods rely on what we have seen to predict what might happen in the future, and that’s why they tend to miss black swan events,” Wang told Stanford News.

“But Sam’s method is different in that it assumes we are only seeing part of the world. It extrapolates a little about what we’re missing, and it turns out that helps tremendously in terms of prediction.”

How it works

Bray had been studying microbial communities for years, and in that time he had observed a few events wherein one microbe would explode in population, overwhelming its rivals. Bray and Wang wondered if this also happened outside the lab, and if so, if it could be predicted.

To find out, the two not only needed to find ecological systems with their own black swan events, but ones with massive and detailed amounts of data, both about the events themselves and the ecosystem they happened in.

Three data sets from natural systems were chosen: algae, barnacle, and mussel measurements on the Kiwi coast taken monthly for 20 years; Black Sea plankton levels taken twice weekly for eight years; and a Harvard study that took net carbon measurements from a forest every half hour since 1991.

They crunched all this data using statistical physics — that’s physics which uses probability theory and statistics in an attempt to puzzle out physical events. Specifically, they used models developed for avalanches and other natural systems with physical fluctuations that are short-term, extreme, and come on without notice — the same qualities that mark a black swan event.

Taking that analysis, they developed a method for predicting a black swan event. The method is meant to be open to variables like species and time-scale, allowing it to work even with lower quality data.

The method was tested against the three datasets it was constructed with. Armed with fragments that showed only the smallest variance, the method did accurately predict the black swan event occurring.

Wang and Bray hope to widen their predictor, bringing it to other fields where a black swan event may occur, including economics, epidemiology, and physics.

The work joins a burgeoning field of AI algorithms and computational models geared towards extreme events, including ones meant to predict wildfires, assist in search and rescue at sea, and optimize emergency response.