Mr. Viola is a 1977 graduate of the United States Military Academy at West Point, and he last served in the Army Reserves at the rank of major. He remained connected to the military through donations to West Point, including to the academy’s Combating Terrorism Center.

A former Pentagon official and close friend said Mr. Viola was devastated over having to withdraw from what he described as a lifelong dream job. But the former official said Mr. Viola felt he was unable to sell his interest in some of his holdings because doing so could have destroyed those companies. His decision followed weeks of negotiations between his lawyers and the government as they sought to find a solution.

Ultimately, they could not.

Mr. Viola has a net worth of almost $1.8 billion and is a co-founder of Virtu. Virtu had planned to go public in 2014, but pulled the plug on the move because of choppy markets and lingering questions around how high-frequency trading firms like Virtu make money. In a regulatory filing at the time, Virtu said regulators were looking into its trading practices. It is not known if this issue played a role in Mr. Viola’s decision to withdraw his name for the Army secretary job, which is a post that requires Senate confirmation.

His decision to withdraw comes the same week another of Mr. Trump’s appointees, Anthony Scaramucci, was told he would not get a top job at the White House amid concerns about the recent sale of his company to a large Chinese conglomerate.

In an episode unrelated to the finances of Mr. Viola, it was recently revealed that he was involved in an altercation in August: He was accused of punching a concessions worker at a racehorse auction in Saratoga Springs, N.Y. No charges were brought against him.