On a dreary, nondescript September morning in 1933, the Hungarian physicist Léo Szilárd stood by the roadside in central London waiting for traffic lights to change so that he could cross. As he stepped off the kerb, “time cracked open before him and he saw a way to the future, death into the world and all our woe, the shape of things to come”.

That’s how Richard Rhodes describes Szilárd’s epiphany, in the opening of his magisterial history The Making of the Atomic Bomb (1986), of how nuclear fission might be unleashed in the world. Szilárd saw that a chain reaction involving the emission of neutrons, discovered only a year earlier by James Chadwick, might sustain the fission process of atomic nuclei and enable the liberation of atomic energy. HG Wells had already imagined this scenario in his prescient novel The World Set Free (1914), in which he coined the term “atomic bomb”.

Szilárd was in England as a Jewish refugee from Nazi Germany. His vision of a nuclear chain reaction became an experimental possibility when, five years later, scientists in Germany discovered spontaneous fission in uranium. That prompted Szilárd and Albert Einstein to write to US president Franklin D Roosevelt warning of the danger of a German atomic bomb and urging an American effort to develop one.

All the elements needed for a theory of sustained nuclear fission were in place by 1933, but it took a flash of insight to unify them in Szilárd’s mind. That happened not as he sat thinking in a laboratory office, but during the most mundane and innocuous circumstance.

It seems this is commonly the case for “breakthrough” moments, when scientists suddenly see how to solve a problem or to weave ideas productively together. You can’t arrange for these insights; there’s no magic formula. But given that scientific progress often depends on them, it’s odd that they are so rarely discussed. It’s as if a determination to present science as an objective enterprise driven by some systematic method creates a nervousness about admitting to the uncontrollable, almost numinous manner in which insight often arrives.

That’s why a workshop called the Physics Imagination Retreat, held at the University of Cambridge, UK, in late June, convened by the Imagination Institute and led by University of Pennsylvania psychologists Martin Seligman and Scott Barry Kaufman, was so unusual – and so revealing. Here a handful of leading physicists discussed how they conceptualize their research and how they find answers to problems. To judge from their testimony and that of others, these answers sometimes really do arrive in eureka moments.

It seems likely that at least some famous “eurekas” are the retrospective inventions of scientists or their biographers

That notion of sudden, complete resolution of a scientific problem has been challenged as a model of how science really works. Sure, eureka moments make for a catchy narrative in popular-science histories, but doesn’t science more often advance through hard graft rather than flashes of inspiration?

It seems likely that at least some famous “eurekas” are the retrospective inventions of scientists or their biographers. Close inspection of the notebooks of Louis Pasteur led historian of science Gerald Geison to question the story reported by the French physician’s son-in-law of how Pasteur discovered molecular chirality (handedness). The son-in-law’s story goes that, having separated by hand crystals of opposite chirality and found that they rotate polarized light in opposite directions, Pasteur rushed into the corridor shouting “Tout est trouvé!” – all is found. But he probably didn’t; the realization came to him more gradually. And while we’ll probably never know the whole truth about Newton’s falling apple, it’s very likely that the story was embellished, if not entirely fabricated, by the great man himself.

However, perhaps the pendulum has swung too far the other way. It seems clear that many scientists do, like Szilárd, have these abrupt insights. And possibly they are especially prevalent in theoretical physics, where problems and their solutions are often rather abstract and mathematized, and much of the work goes on inside the researcher’s head rather than relying on some decisive experimental result.

Can we hope to understand how such eureka moments come about? There’s unlikely to be any recipe for producing them, or even any unifying way of comprehending them. But to judge from the evidence of the Cambridge gathering, there do seem to be similarities and resonances: clues to how imagination does its work in physics. Here are some of them.

The moment of clarity

Michael Berry, a theoretical physicist at the University of Bristol, UK, who has made important discoveries in fields ranging from quantum chaos to classical optics, describes moments of insight as “claritons”, the “elementary particle of sudden understanding” – which he jokingly refers to as his sole contribution to particle physics. He warns of the corollary: that anticlaritons also exist, “a too-frequent unwanted arrival that annihilates yesterday’s clariton”.

Berry was struck by a clariton in 1985 “during a long journey in January, when it was so cold that ice formed on the inside of the windows of the usually overheated German train”. The clariton, he says, was suddenly seeing that there must be a connection between quantum mechanics, chaos and prime numbers.

“This was very soon after I had started to understand the already-conjectured connection between the arrangement of quantum energy levels in systems that are classically chaotic and the universal statistics of numbers called eigenvalues generated by random matrices,” he says. “The understanding was why the universality occurs, and where and how it breaks down.” His clariton was that the Riemann zeta function, a mathematical function related to the disposition of prime numbers, fits precisely into this mathematical framework. “Why suddenly then?” Berry wonders. “I don’t know.” What process generates a clariton?

Collaboration

Berry feels that most claritons involve an apprehension of connections between ideas or problems. And those in turn often arrive through talking. “My claritons often came during or very shortly after conversations with colleagues,” he says, “triggered by an off-hand remark that suddenly revealed connections between two or more of my scientific preoccupations that had previously existed separately in my mind.”

This stimulation from colleagues seems central to several such experiences. Polymer scientist Tom McLeish of Durham University, UK, described one at the Cambridge gathering. He and his collaborators had been working for many years with an experimental group in Germany that had obtained some odd-shaped neutron-scattering plots from polymer liquids. McLeish had been discussing their interpretation with a colleague, but was getting nowhere.

“So we’d just given up,” he says. “My colleague said goodbye to my secretary as he stood at the door. Then he turned round and looked at me – and we both said: ‘Free ends!’ When you make a polymer network you cross-link chains at random points, and there are always some dangling chain ends left over. We both realized in the same instant that these samples must have a significant concentration of these ‘semi-free’ segments, able to adopt any orientational configuration just like totally free chains. We somehow saw into the structure of the material immediately and deeply.”

“Sometimes something in the unconscious is actually doing better than our conscious thinking, and at some later point that gets kicked upstairs,” McLeish suggests. “That can be triggered internally when ready, or externally by some unconnected event.”

This meeting of minds isn’t always quite so magical. Sometimes it simply entails one researcher hearing something useful from someone who didn’t even know it was valuable. Michael Cates, a theoretical condensed-matter physicist at Cambridge (and the current occupant of Newton’s old chair of Lucasian Professor of Mathematics), describes the kind of conversation he has sometimes been involved in: “A says to B: ‘Do you think you could solve problem X with the method I’ve just come up with?’ B responds: ‘Hmm, yes. That should be possible. And if so I can also solve problem Y with it.’ Then A says: ‘What?! You mean my method solves problem Y? That’s unbelievable, I’ve been thinking about Y for three years and never got anywhere with it. It’s a really important problem,’ and B never had any inkling of that.”

Intuition

Several moments of realization attested by physicists invoke a conviction about the truth of a solution even before the technical proofs are worked through. The researchers sense that, having arrived at the “right” answer, they can fill in the details at their leisure. “I share the experience reported by several people, that the clariton is often the beginning of the solution of a problem rather than the fully formed solution,” says Berry. “Nevertheless, it is usually accompanied by the powerful conviction that analysis will demonstrate that it is right (anticlaritons notwithstanding).”

This was the case for mathematical physicist Andrew Parry of Imperial College London when he was working on a thorny problem in condensed matter: the theory of critical wetting, or how the shapes of fluid films that wet a solid surface change with temperature. For years this field had been plagued by a discrepancy between what seemed like the best available theory and the results of computer simulations.

“The solution came to me partly as a guess, then a few weeks later fully in a dream,” says Parry. “While on holiday with my collaborator in Mallorca, I woke up and knew the answer. I knew where to start, where to rest in the middle and where to end in my proof – even though it didn’t yet exist.” He found the solution unfolding almost outside his volition, even though it involved many hours and pages of calculation.

“The three papers we wrote are about 300 equations long, yet almost all of them I knew beforehand as soon as I woke from my sleep that day in the Spanish sun,” he says. Not without some trepidation for fear of ridicule, he describes the experience as the closest he can imagine to a spiritual revelation, in which he became a mere vehicle for the emergence of a deep and beautiful truth. “I felt I’d come across something much bigger and more important than me,” he says. “I was in awe of it.”

It is a bit like pattern recognition: I develop a feeling I need to find a certain pattern, a certain structure, and then the pieces fall into place

Jon Keating

Mathematical physicist Jon Keating of Bristol has had the same experience of sensing an answer before actually proving it. “Sometimes I have an idea, or hear something new, and I have a strong sense that this will solve my problem, long before I go through the details. I’m not always right of course! But I have been right, and that does feel magical. It is a bit like pattern recognition: I develop a feeling I need to find a certain pattern, a certain structure, and then the pieces fall into place.”

“It is quite common that one recognizes the whole pattern as the right one without having to check the details,” he says. “Jigsaw puzzlers will know this feeling.”

Preparation

This role of intuition in science needn’t be seen as something entirely mysterious. Pasteur famously extolled the way that “fortune favours the prepared mind”. He was thinking of how happy experimental accidents may lead to new understanding, but purely theoretical breakthroughs too depend on having penetrated a problem deeply enough to spot the answer when it arrives.

Keating says that this preparation may involve an almost obsessive dwelling on a particular issue or set of questions. “I tend to have a number of pet problems that I keep in my head for a long time,” he says. “Every time I hear something new, I try it out on my problems. I know these problems so well that I tend to have a feeling as to what is needed to solve them, or rather why previous lines of attack have failed.”

This isn’t just a question of thinking hard and long about a problem, but also of finding the right way to express it – a way that enables connections to become apparent. “It is frequently about finding the right formulation or realizing what pattern to look for, and also frequently about asking the question in the right way,” says Keating.

That was the key to Parry’s breakthrough on critical wetting. Rather than the pages of clunky maths he’d been wrestling with, he suddenly saw how to express the interactions between the liquid–vapour and liquid–solid interfaces in terms of pictures akin to Feynman’s celebrated diagrams for dealing with forces between fundamental particles. “It was the invention of this new [visual] language, which I just woke up with one day and could speak, that was so powerful,” he says. “Each time I got stuck, a new part of the language would just pop into my head and allow me to convert it into a new expression and to carry on with the manipulation. I felt I’d had a glimpse of the book of nature.”

In dreams

I used to be rather sceptical about the idea that scientists find answers in dreams. These occurred too often in 19th-century accounts to seem plausible: August Kekulé drowsing in front of a fire and seeing the ring structure of benzene as a snake swallowing its tail, or Dmitri Mendeleev dreaming of the correct arrangement of playing cards annotated with elemental symbols to construct his periodic table.

But as Parry attests, it seems that these things really do happen. At the Cambridge meeting, John Pendry of Imperial described such an experience from his early days as a scientist. “I had a very tricky problem in my PhD thesis: there were some trajectories in the complex plane that someone else had postulated. The sign of some derivative changed and this trajectory was normally a loop, and everybody drew a loop.” But this didn’t solve the problem. Then – “Suddenly it came to me in a dream: what happened was the loop twisted and it was two loops. The twisting meant that the change of sign was explained. It was just there in the morning.” Shades of Kekulé, to be sure.

That sort of thing is not unusual, Pendry says. “A lot of people keep a notebook by the bed, for when they wake up and something is in their head. You go to bed, you’ve worked on the problem all day, you’re tired but the data is up there, and somehow your brain sorts it out.”

Sometimes these dream-stories can be uncanny. McLeish says that he was once trying to figure out, with the same colleague involved in his “corridor” insight, some of the complex morphologies that can arise in the de-mixing of binary fluid mixtures. “We could compute the patterns by simulation, but needed ways of quantifying them through relevant figures of merit,” he says. “In particular the topological quantity escaped us – think of it as the ‘density of holes’. We got nowhere.”

“Then one night I had a dream in which the space was partitioned into small cubes and all possible topological states of each cube enumerated, depending on how dividing surfaces crossed the sides and edges. It turned out that one could then endow this set of primitive cubes with a numbering system that would be additive in calculating the topological invariant of the whole structure.”

Excited by this realization, McLeish rushed in to see his colleague the next morning. “He greeted me by saying ‘Oh, I was just on my way to see you – I had this weird dream last night that if we divided the structure into little cubes…’.”

If you read that in history books, would you believe it?

Searching for answers

What might you learn from all this, the next time you’re stuck on a problem and desperately seeking inspiration? In the end, the advice is probably not very surprising or spectacular. Talk to a friend or colleague. Take a walk. Have a sleep. Anything, it seems, but simply trying to think harder. Stop telling yourself that you should be smarter, and give your unconscious a chance (it’s pretty smart too).

Chinese Taoists have a term for it: wu wei, “not acting”. If there really is a Tao of Physics, perhaps that’s where to find it.