In 2010, a company named Spread Networks spent $300m to dig a fiber-optic line between Chicago and New York. The sole purpose of this very expensive cable? Reducing latency – the time it takes to receive and respond to new information – for trading firms. Less than two years after its completion, however, the wire was rendered obsolete by more sophisticated technology: a communication network transmitting microwave beams through the air.

This one-upmanship is just one symptom of the latency arms race that's been perpetuated by high-frequency trading (HFT) – a phenomenon that's been leading to problems long before Michael Lewis dropped his latest bestseller this week. HFT is characterized by traders operating at the microsecond level, using automated computer algorithms to submit large numbers of orders to buy or sell. Over half of all transactions today are from HFT, up from around zero in 1995.

To win Wall Street's new arms race for speed, high-frequency traders employ various methods: some firms use co-location, placing their computers as close an exchange's servers as possible. Others rely on dedicated communication lines from companies like Spread Networks. Still more firms pay for direct feeds from exchanges – and they pay a lot: approximately $1.5bn was spent in 2013 on overall technology to reduce latency, according to estimates by the Tabb Group.

In his new book, Flash Boys, Lewis contends that the US stock market "really was rigged"– indeed, that it still is, with high-frequency traders exploiting their latency advantages at the expense of Average Joe investors. Well, the US stock market may indeed be rigged, but it's not just about the high-speed trading. The stock market is rigged because of how current stock markets work, and my research has found that it's rigged against the traders themselves.

Virtually all modern financial markets match orders continuously – that is, as orders arrive to the exchange. Continuous-time matching is essentially a winner-takes-all race. A high-frequency trader who receives and acts on new information faster than others can readily pick off orders sitting on exchanges – over 40 venues are competing for the same orders – before others can react. So being faster by as little as one microsecond is enough to grab all the profit.

This is how the "flash boys" win.

But in a study last year, University of Michigan professor Michael Wellman and I developed a model across two exchanges. We examined a strategy called latency arbitrage, which utilizes advantages in access and response time to exploit price disparities caused by fragmentation across those dozens of competing venues. HFTs employing latency arbitrage examine current market information to predict immediate price movements, essentially computing the best prices available before the exchange has even had a chance to update its price quote.

What we found may be even scarier than Lewis's book-selling punchline: it's not simply a matter of the HFT crowd taking profits away from regular investors. Predatory strategies like latency arbitrage have the potential to reduce trading gains for all market participants – high-frequency traders and Average Joes alike.

In Flash Boys, Lewis tells the story of IEX, a stock exchange designed specifically to protect investors from predatory HFT strategies like latency arbitrage. The founders of IEX intentionally introduced a 350-microsecond delay to ensure that high-speed traders could not act on updated information from the exchange before regular investors. These are the heroes, we are told.

But the dominance of sophisticated computer-based trading algorithms is due to the nature of continuous-time trading, and imposing a fixed delay may be at best a stopgap measure. The alternative is to match orders to trade at fixed, regular intervals, like every second.

In such a "frequent call" market, everyone has until the next time the exchange clears, or matches orders, to respond to new information. Orders are processed at the same time, rather than as they arrive, so the fastest trader no longer takes everything. The interval length can be selected to be small enough to effectively track information in the markets, yet imperceptible to regular investors.

Similar to the idea behind IEX, the frequent call market slows down trading by introducing a delay. And matching at regular intervals provides a notable additional benefit: our study showed that frequent call markets offer a significant improvement in trading gains.

As Michael Lewis explains so well, exploitative Wall Street behavior has run amuck, eroding trust and investor confidence. But we need to understand the vicious cycle happening at the exchanges – not just the HFT arms race – to truly escape the rigged game.