Everybody loves coincidences, so here is one: a few months ago, I began writing a popular-science book about how to bridge the gap in our understanding of the microscopic (quantum) world and the macroscopic one. If we could do this, and thus explain large-scale phenomena (such as weather) using the underlying microdynamics of molecules, we could transform both science and technology (though we might, unfortunately, also eliminate the UK’s commonest conversation starter). But would it be spiritually uplifting? Could this coherent and phenomenally powerful picture of the universe fulfil us in the same way that believing in God has traditionally done?

Just as I was grappling with this topic in my book’s final chapter, I received a request to review a book by Sean Carroll that asks similar questions, albeit from a completely different perspective. The short synopsis of The Big Picture is that it explores the question of whether science can explain everything in the world, and analyses the emerging picture of reality that such an explanation entails.

Coincidence? Well, yes and no. Yes, because Physics World did not know I was writing a book on a broadly similar topic. But also no, because in many ways the time is now ripe for this kind of discussion to happen. First, scientists are engaging more and more in interdisciplinary research, which is all about bridging gaps between different disciplines; behavioural economics, for example, applies psychology to the way we do financial transactions. Second, many parts of the world, Europe in particular, are becoming more and more secular. As a result, many people feel there is a void that used to be filled by religion, but which science is incapable of replacing.

Or is it? Carroll, in the best tradition of his California Institute of Technology predecessor Richard Feynman, argues that science can indeed provide a spiritually fulfilling picture of the universe. He calls this picture “poetic naturalism”. The naturalism part is self-explanatory. What you see is what you get: there is no supernatural spoon bending or talking to the dead (or rather, you can talk to the dead, but they don’t talk back), and Carroll presents convincing arguments (based on quantum field theory, our most accurate scientific explanation of the micro world) as to why such phenomena cannot be real.

Naturalism thus stated might sound boring. Not so, Carroll argues skilfully: the fact that there is no magic, doesn’t mean that there is no “magic”. That’s where the “poetic” part comes in. Like Feynman before him, Carroll firmly believes that having a scientific understanding of nature only adds to the beauty of what we see around us, and in some sense makes the whole coherent scaffolding of our understanding even more remarkable and awe-inspiring. The stories we’ve constructed through science about the universe, our origins and fate are, Carroll argues, just as magical as anything we see in religion, art and philosophy – if not more so.

As its name suggests, The Big Picture is broad. The range of subjects is breathtaking, with some pure science, some philosophy and even some passages that read almost like extracts from a self-help book. I enjoyed Carroll’s discussions of the difference between notions of complexity (loosely, something that has many interwoven parts highly mutually dependent) and entropy (a measure of disorder, which is frequently confused with complexity). I appreciated his explanation of how complex phenomena such as consciousness might emerge naturally even though they are not seen in microscopic neurological behaviour (which does not make consciousness any less real). And I learnt many historical facts, for instance about the contributions of Princess Elisabeth of Bohemia to the philosophy of René Descartes (Elisabeth, like Carroll, argued against the Cartesian dualism that manifested itself in the separation between the body and the soul).

Although an exciting read, the book is, of course, far from perfect. I have some minor technical quibbles about a few aspects. One is that Carroll avoids getting into a proper discussion of how one might attempt to reconcile the “many worlds” picture of the universe he promotes (I also subscribe to it – that’s not my issue) with the fact that the universe’s entropy was, in the past, very low. The subsequent increase in entropy is something we can understand only statistically and is hard to argue in a closed system. These are problems that go back to Ludwig Boltzmann’s clashes with his contemporaries in the late 19th century, and the low entropy of the past remains a profound mystery to us, but it does not come across as such here.

Another problem is that Carroll is not a natural writer. He occasionally misjudges the level of detail he needs to go into, and so some parts feel quite “textbookish”. I particularly felt this in his discussion of Bayes’ treatment of probabilistic inference, although it does contain a good explanation of coincidences, such as thinking of someone when the phone rings and it’s them (or indeed being asked to review a book when you’re writing one on the same subject). He tells many stories in-between the more detailed passages, to keep us entertained, but I think he is a much better science communicator than storyteller. Some stories feel unnaturally added on to break the monotony, and the transitions are not done as smoothly as, for instance, in the tremendously successful Freakonomics (a reference for science popularizers).

Despite these minor imperfections, this is no doubt a great book. I like its honest and direct style. It’s passionately written by a physicist who has clearly thought deeply not only about his own research interests, but also about how they fit into the bigger picture. And not only that: Carroll also shows us that science and spirituality do not necessarily come into conflict, something that most scientists feel is true (that’s why almost all of us physicists would choose the same profession if we were born again) but refrain from discussing openly, perhaps for fear of being called “irrational” and “mystical”. Carroll’s “big picture”, beautifully exposed and illustrated, is the closest I’ve seen to a scientific explanation of the meaning of it all. It’s science as magic at its best.

• 2016 Dutton Books £20.00/$28.00hb 480pp