It is an irony of nature that the most abundant form of energy in the universe is also the most mysterious. Since the breakthrough discovery that the cosmic expansion is accelerating, a consistent picture has emerged indicating that two-thirds of the cosmos is made of “dark energy” – some sort of gravitationally repulsive material. But is the evidence strong enough to justify exotic new laws of nature? Or could there be a simpler, astrophysical explanation for the results?

The dark-energy story begins in 1998, when two independent teams of astronomers were searching for distant supernovae, hoping to measure the rate at which the expansion of the universe was slowing down. They were in for a shock: the observations showed that the expansion was speeding up. In fact, the universe started to accelerate long ago, some time in the last 10 billion years.

Like detectives, cosmologists around the world have built up a description of the culprit responsible for the acceleration: it accounts for two-thirds of the cosmic energy density; it is gravitationally repulsive; it does not appear to cluster in galaxies; it was last seen stretching space–time apart; and it goes by the assumed name of “dark energy”. Many theorists already had a suspect in mind: the cosmological constant. It certainly fits the accelerating-expansion scenario. But is the case for dark energy airtight?

The existence of gravitationally repulsive dark energy would have dramatic consequences for fundamental physics. The most conservative suggestions are that the universe is filled with a uniform sea of quantum zero-point energy, or a condensate of new particles that have a mass that is 10^-39 times smaller than that of the electron. Some researchers have also suggested changes to Einstein’s general theory of relativity, such as a new long-range force that moderates the strength of gravity. But there are shortcomings with even the leading conservative proposals. For instance, the zero-point energy density would have to be precisely tuned to a value that is an unbelievable factor of 10¹²⁰ below the theoretical prediction. In view of these extreme solutions, perhaps it is more reasonable to expect a conventional explanation for the accelerating expansion of the universe based on astrophysics (e.g. the effects of dust, or differences between young and old supernovae). This possibility has surely kept more than a few cosmologists awake at night.

Until recently the supernova data were the only direct evidence for the cosmic acceleration, and the only compelling reason to accept dark energy. Precision measurements of the cosmic microwave background (CMB), including data from the Wilkinson Microwave Anisotropy Probe (WMAP), have recently provided circumstantial evidence for dark energy. The same is true of data from two extensive projects charting the large-scale distribution of galaxies – the Two-Degree Field (2DF) and Sloan Digital Sky Survey (SDSS).

Now a second witness has testified. By combining data from WMAP, SDSS and other sources, four independent groups of researchers have reported evidence for a phenomenon known as the integrated Sachs-Wolfe effect. These groups have found that the gravitational repulsion of dark energy has slowed down the collapse of overdense regions of matter in the universe. The case for the existence of dark energy has suddenly become a lot more convincing.

Charting the cosmic expansion

The cosmic expansion, discovered in the late 1920s by Edwin Hubble, is perhaps the single most striking feature of our universe. Not only do astronomical bodies move under the gravitational influence of their neighbours, but the large-scale structure of the universe is being stretched ever larger by the cosmic expansion. A popular analogy is the motion of raisins baking in a very large cake. As the cake rises, the distance between any pair of raisins embedded in the cake grows. If we choose one particular raisin to represent our galaxy, we find that all the other raisins/galaxies are moving away from us in all directions. As a result, our universe has expanded from the hot, dense cosmic soup created in the Big Bang to the much cooler and more rarefied collection of galaxies and clusters of galaxies that we see today.

The light emitted by stars and gas in distant galaxies has likewise been stretched to longer wavelengths during its journey to Earth. This shift in wavelength is given by the redshift, z = (λ_obs – λ₀)/λ₀, where λ_obs is the wavelength we see on Earth and λ₀ is the wavelength of the emitted light. For instance, excited hydrogen atoms emit so-called Lyman alpha transition radiation with a characteristic wavelength of λ₀ = 121.6 nm when they fall back to the ground state. This transition is seen in distant galaxies, and was used to identify the current record-holder for redshift: a staggering z = 10 galaxy with a Lyman alpha line at λ_obs = 1337.6 nm (see Physics World April p3). But the redshift describes only the change in the scale of the cosmos, and does not tell us the distance or the age of the universe when the light was actually emitted. If we knew both the distance and the redshift for many objects, we could begin to chart the cosmic expansion.

One of the prime methods for measuring extragalactic distances is to use “standard candles” such as Cepheid variable stars. The luminosity of a Cepheid variable changes periodically with time, with the luminosity being proportional to the period. The distance to a Cepheid can be determined by first measuring its period in order to obtain the luminosity, and then comparing this with the observed intensity to calculate the distance. Thus, redshifts and distances to objects moving in the “Hubble flow” (the region beyond the gravitational influence of our local group of galaxies) have been charted, revealing the Hubble law: d = (cz/H₀), where c is the speed of light and H₀ = 72 ± 8 km s^-1 per megaparsec (Mpc) is the Hubble constant (1 Mpc is equal to 3.26 million light-years).

Before 1998 this linear relationship between distance and redshift had been confirmed for galaxies as far away as about 1000 Mpc, which corresponds to a redshift of 0.24. The extension to higher redshifts was poorly determined, but by making assumptions about the energy density and pressure content of the universe, general relativity can be used to connect redshifts with distances.

However, measuring accurate distances is one of the most difficult tasks in astronomy, and the distance-redshift relationship had not been checked at higher redshifts. Moreover, based on the best information at the time, it was expected that the expansion of the universe should have been slowing down under the attractive influence of gravity – but this had not been confirmed by observations either.

Going the distance

Although Cepheid variable stars have proved extremely valuable as standard candles in astronomy for many years, they are not bright enough to be used at high redshifts. However, astronomers have found a very special type of supernova to take their place.

Type 1a supernovae are the thermonuclear explosions of carbon- and oxygen-rich white dwarfs – stars that are up to 40% more massive than the Sun packed into a radius 100 times smaller. In the early 1930s Subrahmanyan Chandrasekhar showed that white dwarfs can have a maximum mass of 1.4 solar masses. Below this mass, these dense, compact objects are supported against further gravitational collapse by fermion-degeneracy pressure. In other words, the Pauli exclusion principle prevents the tightly packed electrons from occupying the same state. But in a binary system, the strong gravitational field of the white dwarf can pull matter off a companion star until the dwarf “eats” itself to death: the resulting gain in mass destabilizes the star, which then explodes.

Serendipitously, the luminosity of the exploding white dwarf is very nearly a standard candle. In the mid-1990s this prompted two teams of astronomers – the High-z Supernova Search Team and the Supernova Cosmology Project – to begin observational campaigns to measure the distances and redshifts of type 1a supernovae, in the hope of confirming that the cosmic expansion was indeed slowing down as expected. The results, based on some 100 or so supernovae extending out to a redshift of about 1, were stunning. The two teams found that high-z supernovae are fainter – and therefore more distant – than they should be in a decelerating universe. The researchers had discovered that the expansion of the universe is accelerating (figure 1).

Not surprisingly, interest in supernovae has grown tremendously since then. The Hubble Space Telescope and the premier ground-based observing facilities are chasing the rise and fall of light from supernovae, while smaller telescopes are making surveys and studying nearby events. So far, distances to more than 300 type 1a supernovae have been obtained, and data for many more are currently being analysed. With systematic effects coming under control (see “Focus on supernovae” in Further information), it now appears that the universe started to accelerate as recently as between about five and seven billion years ago (see Riess et al. in further reading). Theorists have been just as busy as observers, trying to unravel what is behind the accelerating expansion.

The missing energy

The supernova observations call out for some gravitationally repulsive substance to drive the cosmic acceleration. Astronomers have long been aware of a missing-energy problem: the luminous mass of galaxies and clusters falls far short of the gravitational mass. This difference is attributed to the presence of dark matter – a cold, non-relativistic material most likely in the form of exotic particles that interact very weakly with atoms and light.

However, observations suggest that the total amount of matter in the universe – including all the dark matter – accounts for just one-third of the total energy. This has been confirmed by surveys such as the 2DF and SDSS projects, which have mapped the positions and motions of millions of galaxies. But general relativity predicts that there is a precise connection between the expansion and the energy content of the universe. We therefore know that the collective energy density of all the photons, atoms, dark matter and everything else ought to add up to a certain critical value determined by the Hubble constant: ρ_critical = 3H₀²/8π G, where G is the gravitational constant. The snag is that they do not.

Mass, energy and the curvature of space-time are intimately related in relativity. One explanation is therefore that the gap between the critical density and the actual matter density is filled by the equivalent energy density of a large-scale warping of space that is discernable only on scales approaching c/H₀ (about 4000 Mpc).

Fortunately, the curvature of the universe can be determined by making accurate, precision measurements of the cosmic microwave background (CMB). A relic from some 400,000 years after the Big Bang, the CMB is black-body radiation from the primordial plasma. As the universe cooled below about 3000 K the plasma became transparent to photons, allowing them to propagate freely through space. Today, almost 15 billion years later, we see a thermal bath of photons at a temperature of 2.726 K that are redshifted to the microwave region of the spectrum by the cosmic expansion (see “The cosmic microwave background”).

The remarkable images of the CMB captured by the WMAP satellite show slight variations in the photon temperatures across the sky – known as the CMB anisotropy – reflecting slight variations in the density and motion of the early universe. These variations, which occur at the level of a few parts per 100,000, reveal the blueprint for the large-scale structure of galaxies and clusters that we see today.

The coldest/hottest spots in the CMB are due to photons that climbed out of the gravitational potentials of the largest over/under dense regions, and the size of these regions is well determined by the physics of the plasma. When viewed across the entire universe, the apparent angular size of these anisotropies would be about 0.5º if the universe has enough warping to fill the energy density gap, and twice as large in the absence of any warping. The easiest way to picture this geometric effect is to imagine a triangle with a fixed base and legs drawn on surfaces with different curvatures: for a saddle surface/sphere the interior angles are all smaller/larger than for the same triangle drawn on a flat surface with planar or Euclidean geometry.

Since 1999 a sequence of experiments – TOCO, MAXIMA, BOOMERANG and most recently WMAP – has confirmed that the CMB spots are about 1° across: the large-scale geometry of the universe is “flat”. For the missing-energy problem, this means something other than curvature must be responsible for the energy density gap.

To some cosmologists, this result felt like a case of déjà vu (see “A brief history of dark energy” in Further information). Inflation, the best theory around for the origin of fluctuations in the CMB, proposes that the very early universe underwent a period of accelerated expansion, which was driven by a particle called the inflaton. However, inflation would have stretched away any large-scale spatial curvature, leaving the geometry of the universe Euclidean or flat. The evidence therefore suggests a form of energy that does not cluster in galaxies, that is gravitationally repulsive, and that might possibly be due to some new particle not unlike the inflaton.

Cosmic harmony

Convincing as the CMB data were, the only direct evidence for cosmic acceleration – that is for gravitationally repulsive dark energy – came from the supernova data. But things are beginning to change. By combining the precision measurements of the CMB by WMAP with radio, optical and X-ray probes of the large-scale distribution of matter, astrophysicists have also teased out further evidence that the expansion rate is quickening. It appears that the gravitational potential wells of dense and overdense regions in the universe have been stretched and made shallower over time, as if under the influence of repulsive gravity.

This phenomenon is known as the integrated Sachs-Wolfe (ISW) effect, and it leads to a correlation between the temperature anisotropies in the CMB and the large-scale structure of the universe. Although the primordial plasma became transparent to photons after the universe cooled, the photons did not travel unhindered afterwards. The cosmos is riddled with inhomogeneities that are strong on small length scales (where matter has clumped to form stars, nebulae and galaxies), and progressively weaker on larger length scales, where galaxies and clusters ride on gentle waves in the matter density. On their flight paths, photons fall into and climb out of the corresponding gravitational potentials.

When the cosmic radiation was first detected almost 40 years ago, Rainer Sachs and Art Wolfe showed that a time-varying potential would impart an energy shift to CMB photons that passed by (figure 2). A photon gains energy when it falls into the gravitational potential of an overdense region, and expends energy when it climbs back out. If the potential has deepened over the course of this process, the photon therefore loses energy overall. If the potential becomes shallower over time, the photon gains energy.

In a universe where the full critical energy density comes from atoms and dark matter only, the weak gravitational potentials on very long length scales – which correspond to gentle waves in the matter density – evolve too slowly to leave a noticeable imprint on the CMB photons. These overdense regions simply accrete the surrounding matter at the same rate at which the cosmic expansion stretches the waves longer, leaving the potentials unchanged. However, under the faster expansion rate of a universe that contains dark energy, the accretion of matter cannot keep up with the stretching. In effect, gravitational collapse is slowed by the repulsive dark energy. Consequently, gravitational potentials grow shallower and photons gain energy as they pass by. Similarly, photons lose energy passing through underdense regions.

It turns out that the large-scale gravitational potentials experienced by the CMB photons correspond to the same overdense/underdense regions seen in the mega-galaxy sky surveys at various wavelengths. CMB photons coming from the same region where galaxies cluster are boosted slightly hotter by the ISW effect. Hence, there should be a positive correlation between the CMB temperature and large-scale structure patterns on the sky. Now, nearly five years after the first supernova results, four independent groups have announced the first detections of this ISW effect.

Stephen Boughn of Haverford College and Robert Crittenden of the University of Portsmouth have found correlations between the WMAP data and two probes of large-scale structure: radio data from the NRAO/VLA Sky Survey (NVSS) and measurements of the hard X-ray background made by HEAO-1, a satellite launched in 1977 (figure 3). The WMAP team has also seen correlations between its data and the NVSS results. Moreover, the Sloan Digital Sky Survey team, along with Pablo Fosalba of the Institut d’Astrophysique de Paris and co-workers, has found evidence for the ISW effect when comparing the WMAP and SDSS data sets (see further reading).

Although the ISW evidence on its own is not yet strong enough to discriminate between expansion caused by spatial curvature and expansion caused by dark energy, when combined with the CMB data for a flat universe the weight of the evidence tilts in favour of dark energy. Taken together, the results are tantalizing. Furthermore, the ISW evidence probes the effects of dark energy on distances down to about 100 Mpc, which is a completely different scale than that of supernovae. This provides a new and independent line of evidence for the effects of dark energy.

Negative pressure

The biggest mystery of the cosmic acceleration is not that it suggests that two-thirds of the universe is made of stuff that we cannot see, but that it suggests the existence of a substance that is gravitationally repulsive. To examine this strange property of dark energy it is helpful to introduce a quantity w = p_dark/ρ_dark, where p_dark is the mean pressure and ρ_dark is the density of dark energy in the universe. This new quantity is similar to the equation of state for a gas.

In general relativity the rate of change in the cosmic expansion is proportional to -(ρ_total + 3p_total), where ρ_total is the density of all the matter and energy in the universe and p_total is the corresponding pressure. To account for the accelerated expansion, however, this quantity must be positive. Since ρ_total is a positive quantity, and the mean pressure due to both ordinary and dark matter is negligible because it is cold or non-relativistic, we arrive at the requirement that 3w x ρ_dark + ρ_total < 0 for an accelerating expansion. Since ρ_dark ~ ^2/3ρ_total, we find that w≤-1/2, so the pressure of the dark energy is not just a little negative but a lot negative!

Why does pressure influence the expansion of the universe? Einstein showed that matter and energy curve space-time. So for a hot gas the kinetic motions of atoms contribute to their gravitational pull, as measured through the acceleration of distant test bodies. However, the forces required to contain or isolate the hot gas count against this pressure bonus. The universe, on the other hand, is neither isolated nor bounded. The expansion of a cosmos filled by hot gases is effectively slowed by the attraction of its self-gravity, more so than a universe that is filled with an equivalent energy density of cold, pressureless gas. And by the same logic a medium that allows negative pressure such that ρ_total + 3p_total < 0 will expand more quickly, repelled by its own anti-gravity.

Negative pressure is not such a rare phenomenon. The water pressure in certain tall trees becomes negative as nourishment is pulled up through their vascular system, and the pressure tangential to a uniform electric or magnetic field is also negative. In these cases, the pressure is somewhat like a stretched spring under tension, exerting an inward force. On the microscopic level, a bath of Higgs bosons – the hypothetical particles that give rise to mass in the Standard Model of particle physics – exerts negative pressure when its thermal or kinetic excitations are small. Indeed, the inflaton can be viewed as a heavier version of the Higgs, and one of the proposed forms of dark energy called quintessence might be an even lighter version of the Higgs (see “Dark energy: the suspects” in Further information).

In principle there is no lower bound to the pressure in the universe, although strange things happen if w falls below -1 (an isolated lump of such material could appear to have negative mass, which is just what one might need to prop open a wormhole). However, most proposed forms of dark energy can buckle or bend only slightly, and even then only over distances much bigger than galaxies, making it hard to get a handle on the stuff. But one thing is certain: such strongly negative pressure does not happen for normal particles and fields in general relativity.

The detailed observations lead to slightly tighter constraints on the dark-energy parameters than those given by the simple estimates above. When the predictions of the different theoretical models are combined with the best measurements of the cosmic microwave background, galaxy clustering and supernova distances, we find that 0.62 < Ω_dark < 0.76, where Ω_dark = ρ_dark/ρ_critical, and -1.3 < w < -0.9 (figure 4).

Looking ahead, darkly

The evidence for gravitationally repulsive dark energy is strong, but there are gaps in our knowledge. The physics of type 1a supernovae is not fully understood, dark matter is still on the loose, and there are a few unexpected features in the CMB spectrum that we do not yet fully understand. While some of these do not seem to be related to the cosmic acceleration, the entire scenario must fit together in order to be compelling. The good news is that we can expect lots of new data. WMAP and a host of balloon-borne and ground-based experiments are continuing to scour the CMB sky, with the Planck satellite due to follow later this decade. New techniques are also being developed to extract information about dark energy, such as plans to study the evolution of the abundance of clusters of galaxies. Another, more ambitious method proposes to infer the ISW effect at different vantage points and redshifts in the universe.

Supernova studies will receive a huge boost if the Joint Dark Energy Mission (JDEM) being proposed by the US Department of Energy and NASA goes ahead. Although the launch is about 10 years away, this dedicated satellite telescope will deliver the final word on cosmic acceleration from supernovae. JDEM also promises an extensive weak-lensing survey that will blaze a new trail towards understanding the nature of dark energy through its influence on cosmic structures and evolution. Naturally, healthy competition with ground-based observers will keep the intervening years exciting.

The aim of all this activity is, of course, to answer the question, what is the dark energy? If w is about -1, then a cosmological constant might be the solution. If w is more than -1, the right answer might be quintessence. And we cannot rule out a new twist to gravity that even Einstein did not foresee: while most theories that link gravitational and quantum physics predict novel behaviour on microscopic length scales or at very early times in the universe, few, if any, anticipate new effects on the largest length scales in the present day. And what if w is less than -1? Whatever the answer, something mysterious is at work in the cosmos.

Further information

Focus on supernovae

How can we be certain that the flux of light from supernovae is indeed diluted and dimmed due to travelling the greater distances that result from the accelerated expansion of the universe? Perhaps the supernovae are closer than we suspect and other effects are at work. The enormous implications of cosmic acceleration have brought great scrutiny to the astrophysics of type 1a supernovae.

It should be stressed that type 1a supernovae are not quite standard candles. However, the luminosity can be standardized: detailed observations of nearby supernovae at known distances have revealed a pattern that can be used to calibrate the luminosity using the light curve and spectrum. But it is possible that this technique may not be valid for more distant supernovae formed much earlier in the history of the universe. For instance, the star-forming environment is expected to evolve over time as the birth and death of stars pollutes the stellar nursery with metals. Could these changes in environment translate into changes in the properties of white dwarfs and supernova explosions? Are distant supernovae fainter simply because they are fainter? However, astrophysicists have found no such link between environment and luminosity.

Finally, there is always the possibility that our view is obscured by cosmic dust. If so, ever more distant supernovae would appear dimmed, giving the illusion of an eternally accelerating universe. However, high-redshift supernovae do not show this trend. In fact, recent results give evidence for past deceleration.

A brief history of dark energy

Dark energy, or something like it, has made numerous appearances in cosmology. Einstein initially introduced a cosmological constant, Λ, in constructing the first cosmological model in his nascent gravitational theory. The cosmic expansion had not yet been discovered, and his calculations correctly indicated that a universe containing matter could not be held static without the mathematical addition of -Λ. The effect was equivalent to filling the universe with a pristine sea of negative energy, upon which stars and nebulae drift. The later discovery of the expansion obviated the need for such an ad hoc addition to his theory.

In the following decades, desperate theorists periodically recycled the cosmological constant in an effort to explain new astronomical phenomena. These resurgences were always short-lived, after closer inspection or subsequent observations revealed more reasonable explanations for the data. Yet developments in particle physics in the late 1960s suggested that the vacuum energy of all particles and fields should inevitably generate a term like Λ. Moreover, a phase transition in the first few seconds after the Big Bang might have left the cosmos filled by a cosmological constant.

In 1980 the theory of inflation was developed: in this theory the early universe undergoes a brief period of accelerated, exponential expansion, with the negative pressure that drives the expansion coming from a new particle called the inflaton, rather than Λ. Inflation has been wildly successful. It resolves various paradoxes associated with the Big Bang model, such as the horizon and flatness problems, and its predictions are consistent with measurements of large-scale structure and the cosmic microwave background.

Inflation also predicts that a characteristic pattern of long-wavelength gravitational waves would have been created in the early universe. These waves are literally gravitons – the hypothetical particles that carry the gravitational force – that have been stretched to macroscopic lengths by the cosmic expansion. The detection of these waves would provide a unique signature of inflation.

Dark energy: the suspects

• Cosmological constant (w = -1)
Originally introduced by Albert Einstein, it was later suggested by Yakov Zel’dovich that quantum vacuum energy would produce a constant energy density and pressure. However, theoretical predictions yield a cosmological constant that is 120 orders of magnitude higher than the observational value. Regardless of cosmology, quantum vacuum energy exists. Whether the cosmic contribution is in fact zero, or finely tuned, is one of the outstanding challenges in physics.

• Quintessence (w > -1)
A form of energy with negative pressure that varies with space and time. Quintessence is dynamic, unlike the cosmological constant, and its average energy density and pressure slowly decay with time. This feature might help to explain the tuning and sudden onset of cosmic acceleration. Modelled as a scalar field, quintessence predicts particle-like excitations with a mass of about 10^-33 eV (see Caldwell and Steinhardt in further reading).

• Other vacuum energy (w < -1)
Unless we are the victims of a conspiracy of systematic effects, w < -1 is the sign of really exotic physics. In one model, quantum effects of a quintessence-like field lead to modifications of general relativity, while other models suggest that the dark-energy density actually grows with time, possibly causing the universe to end in a catastrophic “big rip”. Other novel ideas include an exotic field that causes a cosmological-constant-like acceleration but that varies in space.

• Modification of general relativity
Various attempts have been made to modify Einstein’s general theory of relativity, and therefore avoid the need for exotic matter to drive the accelerated expansion. While some are difficult to distinguish from quintessence, many predict violations of the equivalence principle (which is the bedrock of general relativity) or departures from the universal 1/r gravitational potential.