The history of blunders and missteps that led to the quant trading revolution

Quant trading did not always look like this.
Quant trading did not always look like this.
Image: Reuters/Lucas Jackson
We may earn a commission from links on this page.

More than three decades ago, Jim Simons embraced a radical investing approach by crunching data and creating predictive algorithms—years before these tactics were embraced in Silicon Valley and elsewhere.

A former mathematics professor, Simons is worth $23 billion today. He subsidizes the salaries of thousands of public-school math and science teachers, is working to cure autism, and expanding our understanding of the origins of life.

He and his colleagues at Renaissance Technologies have racked up trading profits of more than $100 billion since 1988. Their flagship hedge fund, Medallion, boasts average annual returns of 66%. Warren Buffett, George Soros, Peter Lynch, Steve Cohen, and Ray Dalio all fall short.

Today, hedge funds and other quantitative, or quant, investors are the market’s largest players, controlling 31% of all stock trading, inspired by the success of Simons and his colleagues. But it took Simons more than a decade to find a formula to beat the market.

It took even longer for Simons, one of the greatest geometers of the last century, to fully trust predictive algorithms and computer models.


For much of the 1980s, Simons traded bonds, currencies and other investments using crude computer-trading models, along with his own instincts. He suffered deep losses, and clients called with complaints. Fellow mathematicians couldn’t figure out why Simons had discarded a thriving career to sit in a makeshift office in a dreary Long Island strip mall, next to a women’s clothing boutique and two doors down from a pizza joint, losing money.

He had to find a different approach.

“I don’t want to have to worry about the market every minute. I want models that will make money while I sleep,” Simons told a colleague. “A pure system without humans interfering.”

Suspecting he’d need reams of historic data so his computers could search for price patterns across a large swath of time, he had a staffer travel to lower Manhattan to visit the Federal Reserve office to painstakingly record interest-rate histories and other information not yet available electronically.

For more recent pricing data, Simons tasked his office manager, Carole Alberghine, with recording the closing prices of major currencies. Each morning, Alberghine would go through the Wall Street Journal and then climb on sofas and chairs in the firm’s library to update various figures on graph paper hanging from the ceiling and taped to the walls. (The arrangement worked until Alberghine toppled from her perch, pinching a nerve and suffering permanent injury.)

For the most part, Simons tested trading strategies based on his mathematical insights and intuitions.

If a currency went down three days in a row, what were the odds of it going down a fourth day?

Do gold prices lead silver prices?

“There’s a pattern here; there has to be a pattern,” Simons insisted.

Risks and wild cards

By 1985, Simons was working with James Ax, another prize-winning mathematician. On the heels of a painful divorce, Ax moved the firm to Huntington Beach, California to the top floor of a two-story office park owned by a subsidiary of oil giant Chevron. Oil wells pumped away in the parking lot, and the smell of crude oil permeated the neighborhood. It was about the last place one would expect to find a cutting-edge technology firm.

By 1986, the firm was trading 21 different futures contracts, including British pounds, Swiss francs, and various commodities. Mathematical formulas generated the firm’s moves, as did Ax’s judgment calls. Sometimes the results were impressive; often, they left the team frustrated. Simons and his colleagues couldn’t unearth new ways to make money.

He considered the possible influence of sunspots and lunar phases on markets, but few reliable patterns resulted. A staffer had a cousin who worked at AccuWeather, so he made a deal to review Brazilian weather history to see if it could predict coffee prices, another waste of their time.

Ax searched for fresh algorithms, but he was also playing racquetball, learning how to windsurf, and attending to an emerging midlife crisis.

Simons flew to California to discuss potential new trading approaches but his visits produced more misery than breakthroughs.

Just follow the data

To improve their predictive models, Ax decided to bring in someone with experience developing stochastic equations, which model dynamic processes that evolve over time and can involve a high level of uncertainty.

René Carmona, a professor at nearby University of California, Irvine, got a call from a friend.

“There’s a group of mathematicians doing stochastic differential equations who are looking for help,” the friend said. “How well do you know that stuff?”

A 41-year-old native of France, Carmona didn’t know much about investing, but stochastic differential equations were his specialty. These equations can make predictions using data that appears random; weather-forecasting models use stochastic equations to generate reasonably accurate estimates. Simons and his team viewed investing through a math prism and understood financial markets to be complicated and evolving, with behavior that is difficult to predict, at least over long stretches—just like a stochastic process.

Carmona liked the challenge of improving their investment models, as well as the idea of picking up extra cash working for Simons’s firm a few days a week.

“The goal was to invent a mathematical model and use it as a framework to infer some consequences and conclusions,” Carmona said. “The name of the game is not to always be right, but to be right often enough.”

Carmona wasn’t certain the approach would work, or even that it was much better than the investment strategies embraced by most others.

“If I had a better understanding of psychology or traders on the floor of the exchange, maybe we would do that,” Carmona said.

By 1987, Carmona was plagued by guilt. His pay came from a portion of Ax’s bonus, yet Carmona was contributing next to nothing. He decided to spend that summer working full-time at the firm, hoping to find success. He made little headway.

“I was taking money from them and nothing was really working,” he said.

One day, Carmona had an idea. Simons and his colleagues used simple linear regressions, a basic forecasting tool relied that analyzes the relationships between two sets of data or variables under the assumption those relationships will remain linear.

Market prices are sometimes all over the place, though. A model dependent on running simple linear regressions through data points generally does a poor job predicting future prices in complex, volatile markets marked by freak snowstorms, panic selling, and turbulent geopolitical events—all of which can play havoc with prices.

Simons’s team had collected dozens of data sets with closing prices of commodities from various historical periods. Carmona decided they needed regressions that might capture non-linear relationships in market data.

Carmona’s idea was to have computers search for relationships in all the data Simons had amassed. Perhaps if they could find instances in the remote past of similar trading environments, they could examine how prices reacted, and develop a forecasting model capable of detecting hidden patterns.

For this approach to work, the team needed even more data than Simons had collected. They began to model data. To deal with gaps in the historical data, they used computer models to make educated guesses as to what was missing.

Staffers didn’t have extensive cotton pricing data from the 1940s, for example, but maybe creating the data would suffice. Just as one can infer what a missing jigsaw puzzle piece might look like by observing pieces already in place, Simons’s team made deductions about the missing information and inputted it into its database.

Carmona suggested letting the model run the show by digesting all the various pieces of data and spitting out buy-and-sell decisions.

He was proposing an early machine-learning system. The model would generate predictions for various commodity prices based on complex patterns, clusters, and correlations that Carmona and the others didn’t understand themselves and couldn’t detect with the naked eye.

The idea was to use sophisticated algorithms to give a framework to identify patterns in current prices that seemed similar to those in the past.

When he shared the approach with Simons, however, he blanched. The linear equations they had been relying on generated trade ideas Simons could understand. But it wasn’t clear why Carmona’s program produced its results. Carmona’s results came from running a program for hours, letting computers dig through patterns and then generate trades.

It just didn’t feel right.

“I can’t get comfortable with what this is telling me,” Simons told the team one day.

Later, Simons became more exasperated.

“It’s a black box!” he said with frustration.

Carmona persisted.

“Just follow the data, Jim,” he said. “It’s not me, it’s the data.”

Ax became a believer in the approach, defending it to Simons.

“It works, Jim,” Ax said to Simons. “And it makes rational sense . . . humans can’t forecast prices.”

Let computers do it, Ax urged. It was exactly what Simons originally had hoped to do.

They were on their way to a historic breakthrough.

Gregory Zuckerman’s book The Man Who Solved the Market: How Jim Simons launched the quant revolution is available as of Nov. 5, 2019 from Penguin Random House. This article is adapted from the book and used with permission of the publisher, Penguin Random House. Copyright ©2019 Gregory Zuckerman. All rights reserved.