A decade ago, the universe was diagnosed with a severe — possibly even terminal — case of "dark energy". Based on observations of very distant supernovae, at the beginning of 1998 two teams of astrophysicists announced the astonishing conclusion that the cosmic expansion is actually accelerating — and not slowing under the influence of gravity as might be expected. The implication was almost beyond belief: in order to account for the acceleration, about 75% of the mass— energy content of the universe had to be made up of some weird, gravitationally repulsive substance that nobody had ever seen before. This substance, which would determine the fate of the universe, was dubbed dark energy.

Like a person confronted with the diagnosis of a life-threatening illness, the scientific community progressed through five stages of reaction to the discovery of dark energy: denial, anger, bargaining, depression and acceptance. Thanks to a number of independent observations, we are now well over the first stage.

For a start, measurements of the cosmic microwave background — the bath of microwave radiation left over from the Big Bang — made in 2000 by the Boomerang and MAXIMA balloon experiments, and in 2003 by the WMAP satellite, have provided independent support for an accelerating universe. Further evidence has come from the Sloan Digital Sky Survey, which in 2005 measured "ripples" in the distributions of galaxies that were imprinted in acoustic oscillations of the primordial plasma 360,000 years after the Big Bang when the universe had cooled sufficiently to allow matter and radiation to decouple. Astronomers have also shored up their evidence for an accelerating universe by studying gravitational lensing — the way light from distance sources is bent by the gravitational fields of massive intervening galaxy clusters. Finally, the original supernova approach itself has been extended and strengthened by including more objects, measured more accurately and across a greater range of cosmic history, with the help of both ground-based telescopes and the Hubble Space Telescope (see "Supernovae as distance markers").

Together, these observations have led cosmologists to a description of the universe called the concordance model. In this picture, 75% of the cosmic mass energy exists as a mysterious, gravitationally repulsive accelerating component, while the remaining 25% has attractive gravitational interactions. In fact, the majority of this 25% (about 5/6) is not even normal matter but rather some additional unknown substance — called dark matter — that gravitates normally yet does not couple to electromagnetic radiation. In all, the concordance model shows that we only understand a somewhat shameful 4% of the content of our universe.

Facing up to data

By the end of 2003 denying the cosmic acceleration was no longer an option. By that time, however, frustration or anger had begun to set in. Just as a patient might cry "why me, why now?", so did physicists trying to understand why the universe was accelerating at all, and in particular why it was doing so now. This is because while the supernova observations could not tell us precisely what dark energy is, its effect on tearing the universe apart is tantalizingly similar to what one would expect if the universe is permeated by Einstein's long-abandoned cosmological constant.

Just after Einstein had unveiled his general theory of relativity in 1915 — which describes the dynamics of the universe and the evolution of the matter and energy in it — he introduced a constant into his equations to counteract the attractive pull of normal matter. He did this because he wanted his new theory to fit the then belief that the universe was static. But when, in 1929, Edwin Hubble showed that the universe was expanding, Einstein was forced to take the cosmological constant out again. Nevertheless, ever since then the possibility of gravitationally repulsive energy has remained in Einstein's theory.

Intriguingly, although a "cosmological constant" is ultimately a source of intense frustration for physicists, it is also predicted by the physics of the very small: quantum mechanics. Quantum field theory predicts that even empty space has an energy density due to the spontaneous creation and annihilation of elementary particles. Based on the particles that we know to exist, however, the vacuum energy density according to quantum mechanics should be an embarrassing 10120 times larger than the value that is required to explain the cosmic acceleration.

Adding to the conundrum of having such a natural candidate for dark energy that is 120 orders of magnitude too large, the cosmic acceleration also appears to have begun only recently in cosmic history. Presumably the cosmological constant could have overtaken the gravitational influence of matter at any time during the last 13.7 billion years during which the universe expanded by a factor of 1028 or so. Yet it kicked in only during the last factor of two expansion — a coincidence with odds of just 2 in 1028! These absurdities seem arranged purely to drive scientists crazy, or to an anthropic explanation in which the laws of nature are somehow linked to our presence.

Physicists countered the anger with bargaining: perhaps we are not dealing with a true cosmological constant but a varying quantum field that adjusts the energy density of the vacuum as the universe expands. This would also be reminiscent of inflation — a period immediately after the Big Bang during which the universe expanded by a factor of perhaps 1026 in just 10–33 s. Maybe the magnitude of the measured cosmological constant is small because the universe is old, and perhaps the reason why acceleration occurred so close to the present time is because matter only came to dominate over radiation and form dense structures fairly recently.

Since 1998, theorists have investigated a wide variety of such models, for example involving new quantum fields such as "quintessence" and extensions of general relativity (see "The mysterious vacuum"). Great progress has been made in winnowing through the garden of models, but a lush thicket remains. The difficulty in deciding among the many proposals for dark energy — coupled with the fact that most measurements we can perform to try and understand its properties rely on the complicated astrophysics of distant objects — has brought some of the community to the stage of depression.

However, advances in the last few years show that there may be light at the end of the tunnel. A combination of next-generation experiments, theory and computation should soon lead researchers to the stage of acceptance, and hopefully beyond this to an understanding and appreciation of the nature of our accelerating universe.

Learning to walk

In the 10 years since the discovery of the cosmic acceleration, researchers have learned the basics of how to walk and talk. Much of this has involved determining the "equation of state" for dark energy. Einstein showed that in addition to mass, all forms of energy contribute to gravity. In particular, general relativity predicts that the strength of gravitational attraction is governed by a particular combination of the energy density, ρ, and the pressure, p, in the form: ρ + 3p. However, if the pressure is negative (as it is when two objects are separated by coiled springs, for example), this combination can have a value less than zero, thus turning gravity from an attractive to a repulsive force.

Physicists therefore often define the equation of state in terms of the quantity w = p/ρ, where w has to be less than –1/3 to cause cosmic acceleration. Einstein's cosmological constant corresponds to w = –1, since a situation in which the pressure is equal and opposite to the energy density is the only way to achieve a unique energy density that does not change in space and time, as Einstein had thought. But in trying to understand the nature and origin of dark energy, researchers have moved beyond this simplest of equations of state and investigated other values of w and in particular now seek to understand the dark-energy properties as a function of time, w(t).

Thanks to the data collected via ground- and space-based observations over the last decade, we know that w averaged over the last 7 billion years — from when the universe was half its present size — is within 10% of Einstein's cosmological constant, w = –1. The period of acceleration appears to have started about 5 billion years ago, before which dark energy was scarce enough that gravity dominated and caused gradual slowing of the expansion of the universe (i.e. cosmic deceleration).

Our understanding of how dark energy actually arose and whether it varies with time is much more modest. For example, all we can conclude so far is that w has not varied by much more than a factor of two over the last 7 billion years. The challenge now is to turn our knowledge of w into a precision measurement, with an uncertainty of a couple of per cent, and to know how it varies with time to a precision better than 10%. Then we will have a much better guide to what new physics has taken over our universe.

One way to achieve this is to collect more types of data using direct and well-understood cosmological probes. Simply obtaining more data of the sort we already have is insufficient; we need to observe supernovae and galaxies that lie deeper into space, and thus further back in time. We also need to be able to separate much more cleanly than we can at present the true properties of the universe from imperfections in our observations. For example, a supernova could appear to us as dim either because it lies further away or because its light has been scattered by dust in the galaxy where it resides, and gravitational lensing can be mimicked by the blurring of the telescope image due to the Earth's atmosphere.

Because many of the properties of dark energy are mixed up with other quantities, such as the density of matter in the universe, it is also vital to use several different observational techniques. Furthermore, since dark energy has both direct effects on cosmic distances and indirect effects on the growth of galaxies and clusters of galaxies (since it is hard for clumps of mass to grow if the space between them is being rapidly pulled apart), complementary techniques can also help answer the important question of what flavour of new physics is required (see "Multiple approaches", below). This could be a new physical ingredient such as a quantum field energy, which would affect cosmic distances and galaxy growth in the same way, or a new physical law that extends Einstein gravity, which might affect distances and growth differently.

If we look at astronomers' track record of discovering new physics, we can see why we need new observations to resolve the issue. The 18th-century puzzle over the motion of planets in the outer solar system was solved by adding a new physical ingredient — the planet Neptune, which was discovered in 1829. The 19th-century puzzle over the motion of the inner planet Mercury, on the other hand, led to an extension of Newtonian gravity: general relativity. The 20th-century puzzle over motions of stars within galaxies will probably be solved by the discovery of a new ingredient — dark-matter particles, although we have not found them yet. For dark energy, which is currently the most pressing problem in cosmology, the mystery of new ingredient versus new law can only be decided through carefully planned experiments.

Tuning in to the early universe

There are four main experimental techniques that will allow us to shed light on the mystery of dark energy. The first is to look for ripples in the distributions of galaxies, which originated in acoustic oscillations of baryonic (i.e. normal) matter when it was bound up with the cosmic background radiation before matter and radiation decoupled. Like leaves (the baryons) floating in a pond (the background radiation), ripples in the water are revealed in the pattern of leaves. Since we can measure the wavelength of the ripples from the pattern of temperature fluctuations in the cosmic microwave background, we can compare them to observations of the galaxy pattern across the sky to determine the distances at which those galaxies lie.

Because only 1/6 of all matter is baryonic, while the rest is in some dark form that gravitates but does not couple to light (like stones in the pond that are unaffected by the motion of the water), the baryonic oscillation pattern is much more subtle than the temperature fluctuations we see directly in the microwave background using probes such as WMAP. However, in 2005 the Sloan Digital Sky Survey, which is based on data taken by a 2.5 m telescope located in New Mexico looking back 4 billion years, detected the faint baryon ripples. Indeed, as stated earlier, the fact that the galaxy patterns agreed with the concordance model supports the discovery of the accelerating universe.

To improve the precision of the measurements we now need to extend such galaxy surveys to much larger volumes. Starting in 2009, the Baryon Oscillation Sky Survey is scheduled to begin surveying one-quarter of the sky out to a redshift z = 0.8, when the universe was half its present age, as well as a slice of the universe at about z = 2.5, when it was one-sixth its present age. (The redshift is due to the stretching of light as the universe expands and thus provides a distance measure: z = (λobs – λ0)/λ0, where λobs is the wavelength of light detected and λ0 the wavelength of the light when it was emitted.) The Hobby–Eberly Telescope Dark Energy Experiment (HETDEX), which is planned to begin observations in 2010, will concentrate on this latter slice in more detail.

The baryon-acoustic-oscillation method is mostly sensitive to the matter density of the universe. This is because such measurements require a comparison between the observed size of acoustic ripples to the size expected from the cosmic microwave background, which originated in an era when the gravitational attraction from matter should have dominated over the gravitational repulsion from dark energy. When combined with supernova observations, however, this plays an important role in separating out the matter-density from dark-energy properties.

A second technique for tackling dark energy is to study the cosmic microwave background itself. The temperatures and spatial extents of the hot and cold spots in this sea of electromagnetic radiation provide a superb probe of the primordial universe some 360,000 years after the Big Bang. Since the early universe should be dominated by matter, with little dark energy, the microwave background says relatively little directly about the properties of dark energy. But, like the baryonic acoustic oscillations, it plays an important role in separating out the role of the matter density.

In addition to ongoing data from WMAP and groundbased experiments, a new generation of cosmic-microwave- background experiments such as Clover, EBEX, PolarBear, QUIET and Spider — which will either be built in the high Atacama Desert in Chile or flown on balloons — are expected to collect data between 2008 and 2010. These observations — not to mention data from the Planck satellite, which is due for launch in 2008 — will allow us to measure the polarization of the cosmic microwave radiation and perhaps allow us to use a form of weak gravitational lensing, the fourth technique that is discussed below, to find out more about dark energy.

The cosmic microwave background also provides a "backlight" to detect clusters of galaxies through their "shadows" as microwave photons scatter off the hot electrons in the cluster core. Known as the Sunyaev–Zel'dovich effect, several research groups hope to use this to measure the size of clusters and hence their distances in order to investigate dark energy. Experiments such as ACT and APEX-SZ in Chile and at the South Pole Telescope are just becoming operational to try this approach.

Supernovae revisited

The most direct way to measure cosmic expansion is the same technique that was used to discover dark energy in the first place: observations of distant "Type Ia" supernovae. Remarkably, all measurements of these exploding stars show that they have the same standardized brightness no matter whether they occurred yesterday or 10 billion years ago (their intrinsic brightness may vary, but once the time it takes for their light to peak and fade is taken into account, their brightness appears quite standard). As such, the measured brightness of supernovae — which can be seen right out to the depths of the universe — tell us how far away they are (see "Studying supernovae", below).

The discovery of the accelerating universe 10 years ago was based on observations of a few dozen supernovae, but since then researchers have measured several hundred and obtained a rough picture of the last 10 billion years of cosmic expansion. Further progress in supernova cosmology requires even more accurate and detailed measurements over this full time period. This is similar to the way one might build a picture of the Earth's climatic history by studying tree rings, with wide rings pointing toward a warmer year. To obtain the clearest picture of the climate, one does not only want to examine more trees but to gather enough data from different types of tree in different environments to create a more accurate understanding.

For the immediate future, surveys such as the ongoing Nearby Supernova Factory will study supernovae from just the most recent 1 billion years in exquisite detail, while PanStarrs starting in 2008 in Hawaii and the Dark Energy Survey in 2010 in Chile will probe about 7 billion years back in time, although in less detail. However, it will be difficult to distinguish between various models for dark energy until an experiment combines the best qualities of each type of survey: in other words, a highly detailed examination of individual supernovae over the entire period that dark energy has been influencing the universe. For distant sources, light is redshifted to near-infrared wavelengths, so this goal requires a space-based observatory.

In 1999 the Supernova/Acceleration Probe (SNAP) was proposed to deliver a detailed "tree by tree" comparison for some thousands of supernovae spanning the last 10 billion years. NASA and the US Department of Energy have since agreed to carry out a Joint Dark Energy Mission, and there are now at least two additional proposals. These include the Dark Energy Space Telescope (Destiny), which would study supernovae and weak lensing, and the Advanced Dark Energy Physics Telescope (ADEPT), which would study baryon acoustic oscillations and supernovae. Both are vying with SNAP for funding, and the successful mission will take off in 2014 at the earliest (see Physics World October p8, print edition only).

The final weapon we have to tackle dark energy is weak gravitational lensing, which involves measuring patterns in the distortion of light emitted by distant galaxies due to the gravitational fields of intervening mass concentrations such as galaxies. Imagine someone holding a lens between you and a wall covered with patterned wallpaper; the distortion will depend on both the strength of the lens and how far it is from both your eyes and the wall. Weak lensing therefore probes dark energy both directly via the stretching of distances and indirectly via the mass of galaxy clusters, since the faster the expansion the harder it is for gravity to pull mass together. When taken together, the largest and deepest surveys undertaken so far image about 1/400 of the whole sky, mostly from data taken by the Canada–France–Hawaii Telescope Legacy Survey.

Surveys some ten times larger, to various depths, will be carried out over the next few years by the Kilodegree Survey in Chile, PanStarrs and the Dark Energy Survey. A new ground-based Large Synoptic Survey Telescope (LSST), starting in 2013 or later, is also planned to survey half of the entire sky, while the SNAP mission also includes a space-based weak-lensing survey that can cover 1/10 of the sky deeply and with high resolution.

Such data, especially when combined with a pure distance probe such as supernova surveys, should be able to provide precise tests of dark-energy properties — including shedding light on the key question of whether dark energy is a new ingredient of the universe or a manifestation of new laws of gravity. This is because the warping of light imaged by weak gravitational lensing is affected by both the acceleration of the universe and the strength of gravity, while supernova distances only depend on the acceleration of the universe — regardless of whether it is driven by new gravity or a new quantum field. Only by using both a distance probe like supernovae and a growth probe like weak lensing can we separate these effects and discover the real physical origin of our immensely puzzling, accelerating universe.

A bright future for dark energy

In the next 10 years we can be optimistic about advances in our understanding of dark energy. The sophisticated next-generation experiments being designed will greatly improve the accuracy of dark-energy measurements using a range of techniques, many of which complement one another and therefore take us closer to understanding the properties of dark energy. In 10 years' time we should be able to determine the equation of state to a precision of 2% and see if it varied by more than 10% over the last 10 billion years, while also testing whether the new physics involves a new quantum field or a new theory of gravity (see "Constraining dark energy").

With such advances we should be able to move firmly into the stage of accepting the new physics of our accelerating universe. Perhaps we will even appreciate that the puzzles of why dark energy exists and why it exists now have simple solutions that reveal something beautiful about the underlying physics. But we should also not forget that the field of dark energy is very young, and that we may have a long and exciting period of exploration ahead before it matures.

Understanding the equation of state for dark energy could also dramatically alter our knowledge of the fate of the universe. For example, continued acceleration would lead to an ever less-dense and colder universe, with the horizon of the visible universe closing in around each observer and ultimately leaving us in a truly dark universe. But a better understanding of dark energy could raise other profound questions, too.

If the accelerated expansion is indeed a window on new theories of gravity, for instance, could it reveal hidden dimensions of space–time? Is dark energy completely dark, uncoupled to matter and other quantum fields? Can the clumping of dark energy — a necessary adjunct to any variation of dark energy over time — be detected? Do its spatial perturbations travel at the speed of light, as for the simplest scalar-field explanations, or perhaps slower or even faster than light? And is there a related variation in what we thought were fundamental constants, such as Newton's gravitational constant or the mass of the electron?

The pursuit of answers to the outstanding questions about the nature of our universe requires theory, simulation and observations to go hand in hand. In the quest for dark energy we will unavoidably and delightedly gather data on and develop understanding of the more familiar astrophysical universe too: stars, galaxies, clusters, cosmic radiation backgrounds, neutrinos and discoveries not yet imagined. The way forward is challenging. But cosmologists have clear ideas for implementing advanced probes to continue the remarkable progress in the physics revolution of the accelerating universe.

Studying supernovae

By measuring the universe's expansion using exploding stars — supernovae — as distance markers, scientists hope to answer some of the most fundamental questions of existence, such as whether the universe is infinite, whether it is going to continue to expand forever, or whether gravity will slow the expansion so much that the universe will eventually begin to contract and ultimately end in a "big crunch". Supernovae are useful in this regard because they are so bright that they can be seen here on Earth even if their light has been travelling for 10 billion years before it reaches us. Moreover, there is a certain class of supernovae — known as Type Ia — all of which brighten to the same peak value before beginning to fade. Since we know the speed of light, we can calculate how long ago these explosions occurred simply by measuring the apparent peak brightness of the supernovae today.

What scientists need though are supernovae with a variety of apparent brightnesses, in other words, those that are at a range of different distances from Earth. Supernovae emit mostly short-wavelength blue light that is stretched to longer, redder wavelengths as the universe expands. By measuring the size of this "redshift", one can determine the size of the universe when the explosion occurred relative to its size today. Although the astronomers Walter Baade and Fritz Zwicky had already suggested in the 1930s that such a measurement could be made, supernovae at any given redshift have a variety of actual brightnesses, which meant that the idea languished till the mid-1980s when the more homogeneous Type Ia supernovae were recognized. Advances in computing and camera technology also helped revitalize this approach: the latest cameras were not only much more sensitive than photographic plates, but were also digital, which meant that their images could be easily analysed by computer. In particular, one could search for supernovae by scanning through many galaxies in one night.

Even so, it was not clear in the late 1980s that very distant supernovae could be found and studied by carrying out supernova searches. Indeed, a team of astronomers in Denmark, led by Hans Nørgaard-Nielsen, had already carried out a huge supernova search between 1986 and 1988 that had yielded just one distant Type Ia supernova; worse still, it had already faded well past its peak brightness.

A decade more of effort was needed to crack the case (see "Critical Point: Dark energy"), including new techniques to find and study entire batches of distant Type Ia supernovae before they reached their peak brightness. In presentations at scientific conferences at the start of 1998 and through papers submitted later that year, two teams — the High-Z Supernova Search team led by Brian Schmidt from the Australian National University and the Supernova Cosmology Project led by one of the present authors (SP) — presented startling results. Although they had been trying to measure the extent to which the cosmic expansion was slowing down, both teams had found signs that the expansion was speeding up. To see a supernova that had been redshifted by a particular amount, both teams found it was necessary to look at fainter and more distant supernovae than expected. In other words, the universe must be expanding faster now than in the past.

Now, a decade later, scientists still have no idea why the universe's expansion is accelerating. Perhaps it is a sign that Einstein's general theory of relativity will have to be revised. But if the acceleration is due to so-called dark energy, then we are left with an equally difficult problem — namely that almost three-quarters of the stuff in the universe is made from something we know nothing about.

At a Glance: Dark energy

  • Discovered 10 years ago from observations of supernovae made by two independent international teams, the cosmic acceleration is one of the most profound discoveries in cosmology
  • The driving force behind cosmic acceleration is often attributed to "dark energy" — an unknown substance that is gravitationally repulsive and makes up a staggering 75% of the mass–energy content of the universe
  • Current data suggest that dark energy could be some kind of "cosmological constant", which was first proposed by Einstein in 1917 and which has a quantum-mechanical interpretation as vacuum energy
  • The key question facing researchers today is whether dark energy is indeed a cosmological constant or something else more exotic. Resolving this involves measuring the equation of state parameter, w, much more precisely
  • More accurate measurements of supernovae, baryonic acoustic oscillations, the cosmic microwave background and weak gravitational lensing should help answer this question in the next decade
  • Dark energy could ultimately leave our universe in total darkness by causing objects to recede from the Earth ever more quickly until they fade from view

More about: Dark energy

R R Caldwell 2004 Dark energy Physics World May pp37–42
R R Caldwell and P J Steinhardt 2000 Quintessence Physics World November pp31–37
E V Linder 2007 Resource letter on dark energy and the accelerating universe arXiv:0705.4102v1 Am. J. Phys. at press
S Perlmutter 2003 Supernovae, dark energy, and the accelerating universe Physics Today April pp53–60
A G Riess and M S Turner 2004 From slowdown to speedup Sci. Am. 290 62–67
Gravity and the expanding universe: www.teachersdomain.org/resources/phy03/sci/ess/eiu/expand
Universe Adventure: UniverseAdventure.org