Theoretical physicist and author Lee Smolin once presented an early form of his theory on quantum mechanics to Richard Feynman. While the famous physicist listened to him respectfully, Feynman ultimately told Smolin that his theory was unlikely to work out because it wasn’t crazy enough. Smolin interpreted that as an indication that his ideas didn’t go deep enough, but he seems at any rate to have taken the advice to heart, as his latest book Einstein’s Unfinished Revolution: the Search for What Lies Beyond the Quantum attests.

Smolin, 63, has gained something of a reputation for coming up with ideas that some find stimulating and provocative, while others would happily call them crazy. It’s a tradition particularly apparent in quantum mechanics, which has more than its share of mavericks: David Bohm, Hugh Everett, John Wheeler and Roger Penrose spring to mind. That’s how it should be though – quantum physics can look crazy, and it positively needs bold and even outrageous ideas, if only to shock us out of complacent acceptance of the innocent-looking mathematical formalism. So the flood of current books about quantum theory (and there are more are on the way) should not stop you finding room for this one.

While Smolin’s walk through some of the major existing interpretations of quantum mechanics reproduces much of what is done elsewhere – including Adam Becker’s What Is Real? and Anil Ananthaswamy’s Through Two Doors at Once – his explanations are especially lucid. He is particularly good at outlining the virtues and shortcomings of the pilot-wave idea first developed by Louis de Broglie and then by David Bohm, which deserved better treatment than it initially received as an illustration of how particles having particular properties and locations could be rescued from the quantum formalism. It is a “realist” interpretation, which insists, in Smolin’s own words, “that there is a real world out there, whose properties in no way depend on our knowledge or perception of it”. This sounds like the default position of science, but quantum physics seemed to Niels Bohr and many others to challenge that view.

Albert Einstein sought a realist view of quantum theory, and this is the “unfinished revolution” of the title – which Smolin hopes to complete. In opposition, he sets up the “anti-realists”, with Bohr at their head. The problem, though, is that they do not necessarily deny the realist position above. Some with Bohrian sympathies suggest only that our theories speak not to any underlying reality, but to our experience of it. This distinction is not always made clear.

Then there are the “magical realists”, whose view of realism says that what is “real” is very different from what we perceive. The “many worlds” interpretation, initiated by Everett in the 1950s, falls into this category – but Smolin gives the clearest account I have seen of why the probabilistic nature of quantum mechanics presents a hurdle that it still has not cleared. As he puts it, “the Everett hypothesis, if successful, would explain vastly too much, and also much too little”.

I remain troubled that so few physicists, both within and outside this interpretation, deem it important to engage with the thorny philosophical issues posed by “multiple selves”, instead blithely accepting a fictional Star Trek picture and seeing where it takes them. Smolin is more troubled that, if these other selves exist, some can be expected to live in a deeply unhappy state on “malevolent” branches of the quantum multiverse. Believing in such things without evidence or lack of alternatives is, he thinks, not a very healthy choice for physicists to make, and creates deep moral dilemmas.

Smolin has been working for years on a realist alternative that he calls the “real ensemble” formulation of quantum mechanics. It displays all his characteristically bold invention, starting from the principle that time is fundamental, but space is emergent. From the abstract proximity of objects’ interactional relationships with other objects, he suggests that a concept of spatial separation can emerge. Because there is nothing fundamental about that separation, there is scope for the nonlocality evident in quantum entanglement, which indeed makes a mockery of our conventional notion of space. One of the theory’s most striking conclusions is that physical law is at root created as it is in common law: by precedent. Things that have happened often in the past are more likely to happen again.

The theory still has gaps and speculative aspects but it has the virtue of being testable

The reader can infer that the theory still has gaps and speculative aspects, but it has at least the virtue of being testable. And regardless of whether the details pan out, it reflects a growing conviction among quantum foundationalists that there is a deep connection between nonlocality, causation and the structure of space–time – perhaps one that, if we’re lucky, will tell us something too about gravity.

Smolin’s determination to salvage realism from quantum mechanics is widely shared – he quotes quantum theorist Lucien Hardy as saying “It is the desire to understand what reality is like that burns deepest in the soul of any true physicist.” But I’m not convinced that the best framing of the problem is to line up Einstein’s realism against Bohr’s anti-realism. The Einstein–Bohr debate was an important and fascinating part of the history of the field, but those who would call themselves Copenhagenists today don’t necessarily subscribe to everything Bohr said, and indeed some are happy to blend aspects of several interpretations.

The role of decoherence in quantum theory has removed the need to make “measurement” a Bohrian black box, or to impose some fundamental quantum-classical “cut”. And one can be an operationalist about quantum mechanics – saying that it is a machinery for predicting observed outcomes – without denying that there is an underlying ontology to nature. Quantum mechanics might simply be the mathematical tool we humans need to navigate in a particular kind of world that is fully objective, but can be glimpsed only via the indirect clues granted to six-foot beings. Smolin suggests that quantum mechanics insists we can only know half of all there is to know from a realist perspective. An alternative view, however, is that there is plenty that is “real” but that living in a classical world primes us to expect more answers than nature can supply.

Yet Smolin is never less than an inventive and provocative thinker, as well as an engaging writer . His postscript is a searingly honest, impassioned and indeed moving view of the dilemmas of doing “fundamental” research. “It is not just that to try to invent a whole new physics is risky for my career and damaging to my emotional stability,” he writes. “I don’t even know how to begin… there is little as terrifying as putting aside the basic principles that form the foundation of our understanding of how we fit into nature.” But Smolin is doing that, and the effort is inspiring.

• 2019 Allen Lane 352pp £25hb