Around 3:40 p.m. on a recent trading day, a dozen or so brokers at the New York Stock Exchange hustled back to their posts after bathroom breaks and a last coffee. They strapped on headphones to prepare for the closing minutes of the U.S. stock market—the pinnacle moment for investors world-wide.

As the clock neared 4 p.m., a buzz grew on the trading floor of mutual-fund giant Vanguard Group 100 miles away in Malvern, Pa., while Michael Buek, head of U.S. equity trading, and his team prepared billions of dollars in trades.

...