A Doubt if it be Us
Assists the staggering Mind
In an extremer Anguish
Until it footing find –

 An Unreality is lent,
A merciful Mirage
That makes the living possible
While it suspends the lives.

In her typically mischievous style, the 19th-century American poet Emily Dickinson captures beautifully the paradox of doubt. Her poem is a reminder that on the one hand growth and change depend on doubt. But on the other, doubt is also paralyzing. In his new book The Primacy of Doubt, physicist Tim Palmer reveals the mathematical structure of doubt that underpins this paradox.

Based at the University of Oxford in the UK, Palmer trained in general relativity but has spent most of his career developing robust “ensemble forecasting” for weather and climate prediction. The concept of doubt, which is central to prediction, has unsurprisingly dominated Palmer’s intellectual life. The Primacy of Doubt is an attempt to show there is a deep relationship between doubt and chaos rooted in chaos’ underlying fractal geometry. He suggests that it is this geometry that explains why doubt is primal in our lives and the universe more broadly.

Tim Palmer’s provocative proposal is that the geometry of chaos plays a role in quantum physics too – and that it could even be a fundamental property of the universe

We normally assume that chaos – being a nonlinear phenomenon – emerges at mesoscopic and macroscopic scales, as the Schrödinger equation describing the behaviour of quantum systems is linear. Palmer’s provocative proposal, however, is that the geometry of chaos plays a role in quantum physics too – and that it could even be a fundamental property of the universe.

Before deconstructing Palmer’s thesis, recall that chaos – a term we use colloquially to describe “crazy”, disordered events – from a technical standpoint applies to a system that exhibits non-repeating, time-irreversible behaviour sensitive to initial conditions. Pioneered by the US mathematician and meteorologist Edward Lorenz, chaos has been the subject of numerous books, many of which have covered his famous three equations describing it and the butterfly effect. What sets Palmer’s book apart is its emphasis on Lorenz’s lesser known discovery – the geometry of chaos – and its implications for how the universe evolves.

Uncertainty in all its forms

Even if Palmer’s thesis is wrong, the book is a useful reminder of the various types of uncertainty – such as indeterminacy, stochasticity and deterministic chaos – each of which has its own implications for predictability, intervention and control. The Primacy of Doubt will therefore be useful for scientists and non-scientists alike, given our tendency to equate uncertainty only with stochasticity.

The aim of the book is not, however, to provide a taxonomy of uncertainty or be a how-to guide for dealing with it in climate change, pandemics or the stock market (though those topics are all covered). Palmer is far more ambitious. He wants to introduce his idea – developed in several research papers – that the geometry of chaos is a fundamental property of the universe from which several organizing principles follow.

Palmer’s thesis rests on successfully showing that the Schrödinger equation – which describes the wave function in quantum mechanics – is consistent with the geometry of chaos despite the equation being linear. More specifically, Palmer suggests there is a physical link between a particle’s hidden variables and how the particle is registered or perceived by other particles and measurement devices, mediated through mathematical properties of fractal geometry.

In two chapters (2 and 11), Palmer describes why this explanation is “neither conspiratorial nor farfetched”. Palmer points out, for example, that there are two types of geometries –Euclidean and fractal – with the latter having the advantage of accommodating counterfactual indefiniteness of quantum mechanics and entanglement without requiring spooky action at a distance, which is a controversial idea in the physics community.

If Palmer’s recasting is correct, it would force physicists to reconsider Einstein’s argument – which grew from his dispute with Niels Bohr about whether quantum uncertainty is epistemic (Einstein) or ontological (Bohr) – that the universe is an ensemble of deterministic worlds. In other words, Palmer is saying that our universe has many possible configurations but the one we see is best described as a chaotic dynamical system governed by fractal dynamics.

Presented by Palmer as one of the book’s two conjectures, the idea implies that the universe has a natural language and structure. In his view, this means that the realized configuration of the universe is not a 1D curve as is typically assumed. Instead, it’s more like a rope or helix of trajectories wound together, with each helix yielding yet smaller helixes and each cluster of rope corresponding to a measurement outcome in quantum mechanics.

In other words, we “live” on these strands in fractal space and this geometry extends all the way down to the quantum level. This notion that the universe is a dynamical system evolving on a fractal attractor has several interesting implications. Unfortunately, Palmer does his readers (and his own ideas) a disservice by scattering the implications throughout the text rather than explicitly distilling them into the principles I think they are.

Four principles

Most prominent of these is what might be called the “emergence principle”. Essentially, Palmer favours statistical thinking rather than deriving macroscale behaviour from first principles or mechanisms, which he thinks is often intractable and therefore misguided. It’s a view that comes in part from Palmer’s career spent developing an ensemble approach to forecasting the weather, but it also makes sense if the universe has fractal structure.

To understand why, consider the following. The conditions under which the macroscale can be modelled without recourse to the microscale include two opposite ends of a spectrum. One is when the macroscale is screened off (for example, being insensitive to microscale fluctuations and perturbations due, say, to timescale separation). The other is when there is, in some sense, effectively no separation due to scale invariance (or self-similarity), as in the case of fractals.

In both cases, deriving the macroscale from the microscale is only necessary to show that a macroscopic property is fundamental, not the result of observer bias. When this condition holds, the microscale stuff can effectively be ignored. In other words, macroscale statistical descriptions become powerful for both prediction and explanation. 

The issue is relevant to a fiery, long-standing debate in many branches of science – how far down do we need to go to predict and explain the universe at all scales? Indeed, the book would have benefited from a discussion of when the geometry of chaos is and isn’t expected to make derivation irrelevant. After all, we know that for some systems the microscale does matter for prediction as well as explanation – appropriate coarse-grained descriptions of intracellular metabolism can influence interspecies competition just as fight outcomes among monkeys can change power structure.

Other interesting principles that Palmer distils (without explicitly naming) include what I call the “ensemble principle”, the “noise principle” and the “no-scale-primacy” principle. The latter essentially says we should avoid equating fundamental with small scales as is often the case in physics. As Palmer points out, if we want to understand the nature of elementary particles, the fractal nature of chaos suggests that “the structure of the universe on the very largest scales of space and time” is just as fundamental.

Read more

Giorgio Parisi: the Nobel-prize winner whose complex interests stretch from spin glasses to starlings

The principle of noise, which connects back to Palmer’s preference for statistical models over derivation, captures the idea that one way to approach modelling high-dimensional systems is to reduce their dimensionality while simultaneously adding noise. Adding noise to a model allows a researcher to simplify yet also approximately respect the true dimensionality of the problem. Including noise also compensates for low-quality measurements or “what we don’t yet know”. In chapter 12, Palmer considers how the noise principle is used by nature herself, suggesting (as many have) that neural systems like the human brain are in the business of computing with noise lower order models from higher order ones in order to forecast and adapt at a lower computational cost.

The ensemble principle, meanwhile, is the idea that to capture regularities in chaotic or high-dimensional systems, a model needs to be run many times to quantify a forecast’s inherent uncertainty. In chapter 8, Palmer explores the utility of this approach in markets and economic systems using the agent-based modelling work of the physicist Doyne Farmer and others. Chapter 10 connects the ensemble forecast approach to collective intelligence and explores how useful it is for making decisions on public policy.

The book gave me a much richer understanding of chaos and convinced me that it shouldn’t be relegated to a corner within complexity science

If I have a gripe with the book, it’s the organization. Palmer spreads the background and justification across the first and final thirds of the book, so I often found myself flipping back and forth between those parts. He might have served readers better by first presenting the theory in full before moving on. Palmer should then, in my view, have clearly spelled out his three principles and their link to geometry, with the final part letting the applications take centre stage.

Nonetheless, I found the book provocative and its ideas rewarding to think through. It certainly gave me a much richer understanding of chaos and convinced me that it shouldn’t be relegated to a corner within complexity science. I expect Palmer’s book will be rewarding for readers who are interested in the mathematical structure of chaos, the notion that the universe has a natural language, or the idea that there are principles unifying physics and biology.

Equally, readers who just want to know how chaos can help forecast financial markets or the world’s climate should find it useful too.

• 2022 Oxford University Press/Basic Books 320pp £24.95/$18.95hb