Stephon Alexander’s riff on “Fear of a Black Planet” – the ground-breaking 1990 album by hip hop band Public Enemy – for the title of his latest book had me hooked before I read a single word. I’m a big fan of exploiting the fascinating links between music, mathematics and physics to engage audiences who might otherwise think science is not for them. A couple of years ago I had a great deal of fun writing When The Uncertainty Principle Goes To 11: or How to Explain Quantum Physics with Heavy Metal for the publisher Ben Bella (December 2018). Alexander, a theoretical physicist specializing in cosmology and quantum gravity at Brown University, is, however, in a class of his own when it comes to the music-physics nexus.

In his first book, The Jazz of Physics, he combined memoir, history of science and his musical expertise as an accomplished saxophonist to tell a profoundly personal story of finding deep connections between what initially appear to be entirely disparate subjects and phenomena (May 2016). Ultimately, the message that resonated most strongly was that physics needs the type of creative, left-field improvisation that drives the most affecting and influential music. Alexander’s new book Fear of a Black Universe: an Outsider’s Guide to the Future of Physics expands significantly on that same theme, while broadening the argument that physics is diminished, in a major way, if it does not accept and incorporate views outside the mainstream.

Alexander blends anecdote and vignette with a whistle-stop tour of topical, controversial and unresolved problems in modern cosmology

The author again draws deeply on his own personal experience and unique perspective. When he received his PhD from Brown College in 2000, Alexander was one of only three Black physicists enrolled on physics doctoral programmes in the US. Throughout the book, he blends anecdote and vignette with a whistle-stop tour of many of the most topical, controversial and unresolved problems in modern cosmology. That mix of science and memoir is a key strength of the book and is, with very few exceptions, seamless.

Alexander brings the physics to life by framing it within the appropriate historical, sociological, and/or personal context, often with a sprinkling of humour. I particularly enjoyed his reminiscence on cosmologist João Magueijo’s cutting riposte to an invited speaker at Imperial College London who described his theory as sick due to an inherent instability: “I want the damned instability,” replied Magueijo. “After all, you are an instability!”

At his best, Alexander is a talented and engaging science communicator, effortlessly wielding analogy and metaphor to energetically carry the reader along. In this book, however, his scientific explanations are somewhat variable – clear and accessible descriptions of introductory quantum mechanics, for example, contrast starkly with the rather more terse and arcane sections on core aspects of cosmology and particle physics.

If the intended audience is mainly made up of professional physicists, then this can be overlooked to an extent. But what is the more general reader, not acquainted with the minutiae and vagaries of quantum geometry, to make of the following passage, for example? “The theory that describes D0-branes falls into a class of quantum geometric theories called noncommutative geometry. What is intriguing is that a handful of approaches to quantum gravity all have some semblance to a pre-space, where geometry itself is fuzzy, or non-commutative.”

The proof copy of Fear of a Black Universe that I received for review has a quote on the back from David Spergel, winner of the 2018 Breakthrough Prize in Fundamental Physics, comparing Alexander’s book favourably with Stephen Hawking’s A Brief History of Time. But as Time magazine’s book critic Paul Gray wrote in 2001, “Of every 100 people who bought A Brief History of Time, three finished it”, not least because Hawking likewise took no prisoners when it came to describing core aspects of his research field to the non-physicist.

While any author would be happy to have the commercial clout of Hawking, it would be a great shame if Alexander’s book suffered a similar fate in terms of actual readership, because it makes a strong case for the importance of diversifying science, and deserves to be widely read. Yet with that said – and despite completely agreeing with Alexander on the central importance of valuing and elevating minority and outsider perspectives – I cannot bring myself to endorse his argument that we should embrace even the wildest, most unconstrained riffing when it comes to furthering physics. To quote that famous aphorism, often attributed to Carl Sagan: “It’s important to keep an open mind, but not so open that your brains fall out.” Even pioneering jazz musician John Coltrane – an abiding influence on Alexander – had to know the rules so as to break them.

Heisenberg could hardly have been more scathing of Schrödinger, writing that “the more I think of the physical part of the theory, the more detestable I find it”

In a chapter titled “If Basquiat were a physicist”, Alexander argues that “both the art world and the physics world have deviant actors. The difference is that the art world has embraced graffiti, while the scientific community has yet to embrace those who take risks.” Well, that depends on what we mean by taking risks. Louis de Broglie’s leap of scientific imagination in imbuing matter with wavelike characteristics was hardly the most traditional of approaches, nor was Erwin Schrödinger’s development of his famous equation. Werner Heisenberg, in an infamous letter to Wolfgang Pauli, could hardly be more scathing of Schrödinger, writing that “the more I think of the physical part of the Schrödinger theory, the more detestable I find it… In other words, I think it is shit.”

Physics has nevertheless embraced these and other leaps of imagination made by those who laid the foundations of quantum mechanics, but with an important justification for doing so: their intense focus on interpreting empirical measurements. Connection with experiment and observation is paramount. Quantum mechanics has Hermitian operators at its very core precisely because they allow us to describe physical observables – only real eigenvalues exist in the real world.

Yet increasingly, in some strands of 21st-century theoretical physics, empiricism is viewed as an afterthought, at best – and Alexander’s book disappointingly continues this trend. After all, those irksome mundane measurements tend to rein in our imaginations. Why have a beautiful theory brought down by an ugly experimental result?

Indeed, when it comes to the many-worlds interpretation of quantum mechanics, of which Alexander writes approvingly, the requirement for empirical evidence is routinely dismissed. A recent viral video, which which has had more than 10 million views, from science communicator Derek Muller (best known for his YouTube channel “Veritasium”), is titled “Parallel worlds probably exist. Here’s why”. Yet how much empirical evidence is there to justify a choice of this interpretation over any of the many others out there? None. (At least in this universe.)

In the second half of the book, titled “Cosmic Improvisations”, Alexander outlines a variety of what might be best described as physics jams – free-wheeling, other-worldly theories. But it’s in his final chapter, “The cosmic mind and quantum cosmology”, that he really over-extends himself. Physics is in a very bad place indeed if we are expected to take the likes of self-styled “quantum healer” Deepak Chopra seriously, as Alexander encourages.

For those unfamiliar with Chopra, a single representative quote is enough to get the measure of the man, when he says that “viewing your body from the perspective of quantum physics opens up new modes of understanding and experience for the body and its ageing. The practical essence of this new understanding is that human beings can reverse their ageing.” We do both our subject and its popularization to non-physicists a major disservice by suggesting that Chopra’s mysticism is in any way credible. Should we also take actress Gwyneth Paltrow’s similarly quantum-inspired “energy healing” seriously? To argue that it is somehow closed-minded or exclusionary to roundly reject Chopra’s groundless guff devalues Alexander’s more compelling arguments elsewhere in the book.

Imagination, creativity, and a willingness to think “outside the box” are certainly aspects of physics that we should embrace, and Fear of a Black Universe makes a convincing case that we need greater diversity in worldview, mindset and culture. But thinking inside the box has also been a mainstay of quantum mechanics: the humble particle-in-a-box underpins quantum physics 101 and so much more.

As philosopher Francis Bacon, the father of empiricism, stressed centuries ago, “The understanding must not therefore be supplied with wings, but rather hung with weights, to keep it from leaping and flying.” We need fewer, not further, flights of fancy in modern physics.

• 2021 Basic Books $17.99hb 256pp