The term “theory of everything” was a common turn of phrase among high-energy particle theorists during the 1980s, used with varying degrees of irony. Physicists from other fields were often not amused, seeing this terminology as yet more evidence of the hubris of particle physicists. In his new book Theories of Everything: Ideas in Profile, author Frank Close uses the term unapologetically, outlining the current state of our best attempt at a unified theory that should apply to “everything”.

Currently, the closest such theory that we have is commonly known as the Standard Model of particle physics, although Close also uses an alternate name some favour – the Core Theory. He describes some of the features of the theory, leading up to the vindication of one of its central ideas – that of a Higgs field – with the first-ever observation of the Higgs boson, made by researchers using the Large Hadron Collider (LHC) at the CERN particle-physics laboratory in Geneva in 2012. For a more detailed account of this story, Close’s 2013 book, The Infinity Puzzle, is an excellent source.

The great success of the Standard Model has left particle physicists in a difficult position; with not just the Higgs, but all other results from the LHC and other particle-physics experiments so far agreeing perfectly with the theory. This has crushed hopes that something unexpected might be found, which would ultimately indicate a way forward to a better, more complete theory. A major goal of Close’s latest book is to put this situation in historical context, describing earlier “theories of everything” and the theoretical advances that gave new, fundamental insight into the nature of physical reality.

A crucial question about our current situation is whether we really are at, or near, the end of our search for what theoretical physicist and Nobel laureate Steven Weinberg refers to as a Final Theory, or whether there is another revolution in our understanding still to come. One often reads quotes attributed to Albert Michelson (“the grand underlying principles have been firmly established”, 1894) and Lord Kelvin (“there is nothing new to be discovered in physics now”, 1900), indicating that they, like many now, thought they were near the end of the road. That of course would have been a huge mistake, with the great revolutionary discoveries of modern physics – relativity and quantum mechanics – just a few years off.

Close points out that Kelvin’s actual 1900 speech was much more prescient, as he described “two clouds” on the horizon, pointing out that experimental results were in direct conflict with the accepted theory of that time (the Michelson–Morley experiment and the black-body radiation spectrum). For anyone trying to look for a lesson from history applicable to today’s “theory of everything”, a key question is whether any analogue of these “two clouds” can be found.

Close takes up this question and argues that there are good candidates for our “two clouds”. The first is the energy density of the vacuum, also known as the cosmological constant. Cosmological observations appear to indicate that this is a non-zero number, with an order of magnitude so small that it doesn’t fit at all with what one might expect from the Standard Model and general relativity.

The second cloud, according to Close, is the so-called “hierarchy problem”. This is a theoretical problem with our now experimentally confirmed theory of the Higgs field, which is strongly sensitive to very-short-distance phenomena. We seem to lack a convincing idea that would consistently describe what is happening at unobservably short distances, without requiring an unmotivated and very special choice of parameters in order to get the Higgs physics seen at the LHC.

An increasingly popular tactic for theorists frustrated by not having an answer to these problems is that of postulating a “multiverse”, in which our universe is but one disconnected component, born out of some process that left it with some essentially random choice of fundamental parameters. In this scenario there’s no point in worrying about why these parameters have the scales they do, since somehow the “multiverse did it”, in a manner constrained only by the “anthropic principle”, which says that the parameters must have values consistent with our existence. Close quite rightly raises the issue of whether this is really a valid explanation, since it’s one currently lacking any means to subject it to experimental test.

He then goes on to explain that the current “two clouds” seem to have a root in the same fundamental issue – the lack of a viable general quantum theory of gravity that would unify the theory of relativity with the quantum field theory of the Standard Model. For quite a few decades now theorists have put great hopes in certain speculative ideas proposed back in the 1970s that were supposed to lead towards such a unified theory. Recent years have not been kind, though, to these proposals, with results from the LHC killing hopes for experimental evidence of one of them – supersymmetry – and the great complexity needed to get anything not obviously inconsistent with experiment making the other – string theory – less and less appealing.

I think Close is on the right track with his final argument where he concludes “My conjecture is that in some future theory of everything, space and time will turn out not to be fundamental and will emerge from some deeper concept. Whoever first establishes what this is will enter the pantheon of science, along with Newton, Maxwell and Einstein.”

The lack of a compelling, unified theory that can explain how the degrees of freedom fundamental to the Standard Model and its forces fit together with those describing space, time and the gravitational force is a major hole in today’s “theory of everything”. Perhaps the future will bring a new idea that tidily fills that hole, thereby dispersing Close’s clouds. It’s also possible that the clouds are indications of a storm to come, with new ideas tearing apart the Standard Model, replacing it with a quite different new “theory of everything”. I hope we’ll soon find out which route the future of physics will take.