The worst prediction in all of science

A successful scientific theory is one that makes precise and accurate predictions. Scientists are even happier when two distinct theories make predictions that agree with one another. Thus, physicists are a bit chagrined when they use their two best theories to predict the simplest possible quantity, and the result is that they disagree spectacularly enough that it is often called “the worst prediction in the history of science.”

Empty space is, well, empty. Containing nothing, it would seem that calculating the energy of empty space would be simple and the prediction would be zero. However, that expectation is not correct.

The two theories that, when combined, underlie all of modern physics are called the theory of general relativity and the standard model of particle physics. General relativity describes the behavior of the force of gravity and applies to large structures in the Universe. In contrast, the standard model of particle physics is used to explain all other forces and governs the quantum world of the very small.

Both theories can be applied to empty space. So, what happens when the two theories are used to calculate the energy density of a true vacuum?

The view from general relativity

Einstein’s theory of general relativity discusses the shape and motion of space itself.  We have known for a century that the Universe is expanding, and the theory that describes the evolution of the Universe is called the Big Bang. Basically, the theory says that the Universe was once smaller, and something caused the expansion to begin.

Given that gravity is an attractive force, this implies that after the expansion began, this expansion would slow down. Why? Because all the matter of the Universe attracted all of the other matter.

Thus, it was very surprising when, in 1998, researchers studying the evolution of the Universe found that not only was the Universe expanding, but that the expansion was speeding up. The only way this could happen is if space had a small and distinct energy associated with it. If the energy was of the right kind, it would result in a repulsive form of gravity. Researchers call this repulsive gravity “dark energy,” and they can calculate just how much dark energy is required to explain the observed evolution of the Universe. This energy is very small — equivalent to about the energy of four hydrogen atoms per cubic meter of space.

The view from quantum mechanics

So, does the standard model predict an energy of space and, if so, how? 

The standard model says that all of space is filled with a variety of fields. When those fields vibrate in certain ways, the particles of the quantum world appear — electrons, quarks, etc. However, even when the fields are quiescent — nominally at rest — there remains an ongoing residual “hum,” with tiny transient vibrations in the fields with an array of wavelengths. Because in the quantum world particles and waves are the same thing, this implies that empty space contains a chaotic mix of ephemeral particles that appear and disappear essentially instantly. This lowest energy state of the various fields is called the zero point, and the energy they contain is called the “zero-point energy.”

To calculate the zero-point energy of the quantum world, add up the effect of all the quantum waves. In principle, there is no minimum wavelength, so you add up shorter and shorter waves. Because short wavelength means high energy, this means adding higher and higher energies. Taken to the extreme, you could add up near-zero wavelengths with near-infinite energy — but we know that the standard model eventually fails at very high energies, so you only add up energies to a certain maximum (and, hence, only to a certain minimum wavelength).

Just what exactly the maximum energy should be used in the calculations is a matter of theoretical dispute, but most scientists agree that the absolute highest possible energy for which the standard model applies is called the Planck energy. If you use that energy as the cutoff in your calculation, you calculate the zero-point energy to be very high. The energy density is equivalent to having the mass of a 100 quintillion times more than the entire visible Universe compacted down into a cubic meter.

The worst prediction in all of science

Indeed, by this simple calculation, the energy density predicted by the standard model is about 10¹²⁰ times that predicted by general relativity. That is a one followed by 120 zeros. This discrepancy certainly earns the title “the worst prediction in all of science.”

The factor 10¹²⁰ is a worst-case scenario. Unproven theories have been proposed that improve the situation. For example, if a theory called supersymmetry turns out to be true, the disagreement is “only” a factor of 10⁶⁰. 

Subscribe for counterintuitive, surprising, and impactful stories delivered to your inbox every Thursday

When such a large disagreement occurs, something is very wrong with one or both theories. It remains possible that our current theoretical understanding is wrong, but general relativity describes the cosmos well and the standard model does a good job at the quantum level. It’s only when the two are compared that a problem arises.

Some potential solutions

What are some of the proposed solutions? Well, there are many. For example, one explanation arises from the fact that the standard model assumes that there is no smallest unit of space. This means that the smallest volume you can imagine can be split into even smaller units in a never-ending series. But what if there is a smallest unit of space — effectively an “atom” of space? If that’s true, then this changes the calculations, and in such a scenario, the disagreement between cosmic and quantum energy can disappear.

Another idea is that we have been fooled by our senses. As we experience the world around us, we seem to move in three spatial dimensions. If there were additional dimensions of space, then this would radically change our theory of gravity, which would mean that the quantum calculations (which are currently performed in three-dimensional space) are wrong.

While the final answer is unknown, it seems more likely that the problem arises in our understanding of the world of the very small. After all, if the standard model prediction was correct, the Universe would have expanded so fast that stars, galaxies, and humans never would have existed.

But a mystery is a mystery. The simple fact is that researchers don’t know why our theories of the cosmic and quantum worlds make such different predictions. Despite decades of effort, the answer has eluded some of the brightest minds of science. We will simply have to wait until that future day when someone solves this cosmic conundrum and enters the pantheon of physics legends.