Almost every physicist or mathematician under a certain age knows former colleagues or classmates who now work in finance. Older scientists, however, can probably remember a time before such “quants” existed. What happened to create this new and lucrative profession over the course of a few decades?

In his new book on the history of options-pricing theory, Pricing the Future, George Szpiro sheds light on the answer by explaining the theoretical developments that led to new markets in complex financial instruments. Known generally as derivatives, these instruments are contracts between two parties that obligate one of the parties to pay the other an amount determined by the outcome of some future event. For example, the pay-off might depend on the closing price of a particular stock – termed the “underlying” security, on whose price that of the “derived” security, or derivative, depends – at some specified date. This particular type of derivative is known as a stock option.

What is the proper value (price) of such an instrument? Since the pay-off depends on future events, any pricing model must include, at minimum, a probability distribution over future outcomes. But in fact the problem is more subtle, because the value of a given probability distribution of future payouts to a particular individual depends on that individual’s attitude towards risk. For example, a gamble that resulted in a $10,000 gain 60% of the time and an equal loss 40% of the time might be very attractive to a millionaire, who might be willing to pay up to $2000 to play each time. However, that same gamble would be unappealing to a cash-strapped student, since they cannot afford the loss. This simple example shows that it is more than the expected return that determines the value of a gamble – the risk preferences of the gambler enter as well.

Szpiro explains how both problems – modelling the future behaviour of the underlying security, and taking into account risk preferences – were solved by a collection of mathematicians, physicists and economists over the last 100 years. The list of contributors is too long to do it justice here – for that I urge the reader to consult Szpiro’s well-written and entertaining book – but I cannot resist mentioning a few of them.

One early notable was the French mathematician Louis Bachelier, a student of Poincaré who used a random walk to model stock prices (and, incidentally, anticipated Einstein’s description of Brownian motion by five years). However, Bachelier’s model was deeply flawed, as he assumed the stock price itself exhibited a random walk. In fact, historical data on stock prices fit much better when it is the logarithm of the price that fluctuates in this way. The US high-energy physicist M F M Osborne was the first to note the log-normal distribution of stock-price movements, and he seems to have been unaware of Bachelier’s earlier work, as he failed to cite him in a 1959 paper. Today, much more sophisticated models involving autoregression, time-dependent volatility and other refinements are used to model stock movements.

The most famous result in derivatives theory is the Black–Scholes equation, which is duly referenced in Szpiro’s subtitle Physics, Finance and the 300-year Journey to the Black–Scholes Equation. This partial differential equation governs the behaviour of a certain class of options, and was discovered in 1973 by the economists Fischer Black (who trained in physics at Harvard University) and Myron Scholes. However, their results were probably already known as early as 1967 to Ed Thorp, a mathematician who became a pioneer in hedge-fund management. In a 2011 interview with the Journal of Investment Consulting, Thorp said that he wrote down “what later became known as the Black–Scholes model” and started using it to invest in 1969, but “realized in retrospect that there was no chance I was going to get any recognition for an options formula because I was not part of the economic academic community”. One lacuna of Szpiro’s otherwise well-researched book is that he neglects to properly credit Thorp for his early work on options pricing.

The reader may still be wondering how the problem of risk preference was overcome. In a sense, it was finessed. The option price obtained by the Black–Scholes equation is essentially just the expected return: the return averaged over all possible outcomes, weighted by probability according to the model of underlying dynamics – and, if necessary, discounted back in time using a deflator, which corrects for the “risk-free” return that could have been earned by investing the assets in, for example, US Treasury notes.

This result is justified by the assumption of perfect hedging, which states that a trader can at any moment in time (and without cost) construct a “riskless” portfolio comprised of the option, the underlying security (such as a stock) and some other securities, typically cash. Such a portfolio is considered riskless because over a small time step the value of the option and the underlying security are related; so by assembling a portfolio with the correct ratio of the two, one can ensure that the portfolio’s value is constant – a process known as “delta hedging”. Because the value of this riskless portfolio and of the stock and cash at the initial time are known, one can deduce the value of the option independent of individual attitudes toward risk.

The financial utility of the Black–Scholes equation is not in doubt, but experts are divided on the general societal utility of advances in derivatives. The conventional view, taught in business schools and in economics departments, is that financial innovation enables economic dynamism and allows markets to allocate resources more efficiently. The opposing perspective, held by billionaire investor Warren Buffett, is that derivatives are “financial weapons of mass destruction” – speculative instruments in a complex global casino that carry more risk than benefit. When Buffett’s company, Berkshire Hathaway, bought the reinsurance firm General Re in 1998, the latter had 23,000 derivative contracts. Buffett later explained his attitude towards these contracts to a US government panel: “I could have hired 15 of the smartest people, you know, math majors, PhDs. I could have given them carte blanche to devise any reporting system that would enable me to get my mind around what exposure I had, and it wouldn’t have worked…Can you imagine 23,000 contracts with 900 institutions all over the world, with probably 200 of them names I can’t pronounce?” Ultimately, Buffett decided to unwind the derivative deals, even though doing so incurred some $400m in losses for Berkshire.

The recent financial crises suggest that Buffett’s attitude may be the right one, and that the potential benefits of derivatives and other complex instruments come with dangerous systemic risks. In 2010, while testifying before the US Senate Banking Committee, the former chief of the US Federal Reserve, Paul Volcker, commented “I wish somebody would give me some shred of evidence linking financial innovation with a benefit to the economy.” In Pricing the Future, Szpiro gives us a colourful history of derivatives, but does little to address Volcker’s fundamental question.