Nobel Prize-winning physicist explains the power of intuition in scientific discovery

The birth of ideas is a subject that many scientists have confronted in their writings. Let’s begin with the famous French mathematicians Henri Poincaré and Jacques Hadamard. The two men, who lived in the 19th and 20th centuries, repeatedly described the ways in which their mathematical ideas were born and had a common point of view on the subject: There are distinct phases to the formulation of a theorem.

• There is a first, preparatory stage in which the problem is studied, the existing literature read, and the first unsuccessful attempts at a solution are made. It is a period that can last between a week and a month, and ends because no progress occurs.
• Then there is a period of incubation in which the problem is abandoned, at least consciously.
• This incubation ends suddenly with a moment of illumination, which often occurs in a situation unrelated to the problem you’re trying to solve. It might happen, for instance, when conversing with a friend about topics with no apparent connection to the problem.
• In the end, after the illumination provides the general way to tackle the problem, the solution must actually be formulated. This can be a very protracted period. You must verify that your idea is correct, and whether or not the road you have set out on can work — followed by all the mathematical steps required to demonstrate the solution.

Obviously sometimes the moment of illumination turns out to be misleading, by assuming the validity of steps that cannot in fact be taken. In these cases you have to start all over again.

There is also intuition in physics. It is different from mathematical intuition and has evolved over time. Galileo, as the historian of science Paolo Rossi has pointed out, had the grand intuition that the celestial and the terrestrial worlds were similar and that it was possible to apply the same laws to both. This was the starting point for many of Galileo’s discoveries, though it is not easy to prove, since reasoning often has its tail in its mouth, so to speak, as the irreverent philosopher of science Paul Feyerabend has emphasized. Spots on the sun demonstrated that the celestial world was corruptible, but only if they were not an effect produced by the telescope itself. Given that it was not possible to establish whether the telescope accurately depicted the heavens, Galileo’s observations implied either that sunspots existed and that the celestial world was mutable like the terrestrial one or that the telescope produced false images and interacted differently with light from terrestrial and celestial objects. This second hypothesis is evidently very difficult to sustain inasmuch as the sunspots rotate at a constant speed (due to the rotation of the sun). At the time, however, the idea of one set of rules for the whole universe was shocking, and many were unable to accept the Galilean intuition, rejecting what followed from it.

Intuition in physics also had a fundamental role to play much later, particularly during the birth of quantum mechanics at the beginning of the 20th century. This was one of the great adventures of physics, and between 1900 and 1925 involved such illustrious scientists as Max Planck, Albert Einstein, Niels Bohr, Werner Heisenberg, Paul Dirac, Wolfgang Pauli, and Enrico Fermi. They were dealing with an apparently very strange process, in some respects a contradictory one. Certain phenomena had been observed (blackbody radiation, for example) that physicists at the time were unable to understand because the phenomena could be explained only by the quantum mechanics that had not yet been discovered.

What would be the most logical way of proceeding? To invent quantum mechanics and present the right explanation! But the story took a completely different turn. Physicists made various attempts to explain quantum phenomena in classical models by explicitly assuming that some of the lesser known elements of the model behaved in a bizarre way (incompatibly with classical mechanics) — in other words, “There are things we do not yet understand, but we will after more work.” During this time there were a large number of contradictory contributions to the field, some of which were plainly wrong — though to be fair they could hardly be right because, by trying to justify quantum phenomena within the framework of classical mechanics, they were attempting the impossible. For example, in a 1900 article that sought to explain blackbody radiation, Planck assumed that light interacted with oscillators that had the correct quantum properties, in complete incompatibility with classical physics. Planck plowed ahead, having not realized that the supposed compatibility with classical physics did not in fact exist.

It is remarkable to see how the partial explanations presented were in fact correct: Intuition in physics is so powerful that hypotheses that remained in the field of classical mechanics contributed to explaining quantum phenomena, pushing ever forward the contradictions between classical mechanics and observed phenomena. In the end, when the contradictions were too great, many aspects of the new quantum mechanics had already been anticipated. To give one example, in Bohr’s theory of 1913, which assumed that the electron that orbits the hydrogen atom could do so only on certain orbits that satisfy a certain condition, the spectral lines of the light emitted by hydrogen could be calculated in a simple way. The hypothesis was not sustainable within classical mechanics, but it provided fundamental clues that helped build quantum mechanics a decade later, when the urgent need for such a radical new framework became clear.

The final barriers fell in 1924 and 1925; the following years saw progress at an impressive rate, and by the end of 1927, the new quantum mechanics had virtually reached its definitive formulation. The preparatory work (which lasted 25 years, from 1900 to 1925) was possible precisely because there had been a strong intuition as to how the physical system was organized. It was a very different kind of intuition from that of the mathematicians, leading to work that advanced physics despite being frequently based on wrong arguments.

Concerning intuition, a friend of mine who is an experimental low-temperature physicist once remarked: “You have to get to know your experimental setup so well, the system that you are measuring, the phenomena that you are observing, to be in a position to give the right answer without even thinking. If they ask you a question (or you ask it), you must be in a position to give the right answer immediately and then afterward, on reflection, to be able to say why it is right.”

I recently experienced something that points in the same direction. A friend I work with asked me a not very easy question, to which I immediately gave a detailed answer. Then he asked me how I’d arrived at it. At first I gave a completely nonsensical explanation, then a second that made a bit more sense, and only at the third attempt was I able to properly justify the right answer, which I had at first given for the wrong reasons. Italian physicist Giovanni Gallavotti, in the preface to his book on mechanics, writes that good students must reflect on a theorem until the theorem seems obvious and the proof, consequently, seems superfluous.

Intuition depends very much on the field in question; in some cases it is based on mathematical formalism. Formalism is an extremely powerful tool, and becomes even more so if the unconscious itself gets accustomed to using algorithmic procedures. [W]hen I was doing my first research on spin glasses, I used the replica method, a pseudo-mathematical formalism (in the sense that the mathematical validity of the method had not been proven but the results were correct, as was demonstrated 20 years later) that allowed me to arrive at a final result without knowing what I was doing. It then took years to understand the physics significance of my results. I had unconsciously constructed a series of rules that allowed me to understand the direction in which to proceed with the calculations. Rules that I would never have known how to formalize.

Making progress without being fully conscious of what you are doing is hardly a method confined to scientific problems. The great 20th-century Italian writer Luce d’Eramo, whose books have been translated into many languages, said that when writing a novel she often proceeded by rereading everything she had written so far, and only then did she decide how to begin the next scene. At that point she took the characters, mentally inserted them into the scene, and observed them: “I don’t decide what they have to do, I imagine them and observe how they speak and how they act: I merely record what they do.” It’s a process that is not a million miles from the one described by Poincaré and Hadamard.