One by one, almost all the high-profile Goldman Sachs alums have left the White House.

Gary Cohn, the former Goldman Sachs COO who left the bank to become President Trump's National Economic Council director, resigned Tuesday night following a dispute over the president's planned tariffs on aluminum and steel imports.

Cohn's farewell follows the departures of chief White House strategist Steve Bannon, communications director (for a whopping ten days) Anthony Scaramucci and deputy national security adviser Dina Powell. They all exited in just the past few months.

All of them worked for Goldman Sachs at one point during their careers -- and Powell actually returned to Goldman Sachs last month.

That means that the only high-profile Goldman Sachs veteran left in Trump's inner circle is Treasury Secretary Steven Mnuchin.

Stocks fell Wednesday as investors worried that Cohn's exit was a clear sign that Trump was moving to a more protectionist stance, even if it risked starting a trade war.

Related: Cohn's exit stokes fears that Trump will fight a trade war

Trump's Goldman-filled White House was surprising, since he attacked Hillary Clinton on the 2016 campaign trail for giving paid speeches to Goldman Sachs employees.

He even went as far to say in February 2016 that Goldman Sachs had "total control" over Clinton. (Trump said the same thing about Republican rival Ted Cruz.)

The president has defended his decision to hire so many people who worked for Goldman Sachs by saying that he wanted people who "made a fortune" and not a "poor person" advising him.

Related: Trump complicit with Scaramucci attacks on Kelly

Trump actually nominated another Goldman Sachs executive, Jim Donovan, to be Mnuchin's deputy last March. But Donovan backed out two months later, citing personal reasons.

There's one more member of the Trump White House who has ties, albeit not a former job, at Goldman Sachs as well -- Securities and Exchange Commission chairman Jay Clayton.

Before Clayton took over at the SEC, he was a partner with the law firm Sullivan & Cromwell and one of his main clients was -- you guessed it -- Goldman Sachs (GS).

That has led to concerns that the person who is supposed to be Wall Street's top cop may be too close to the companies he's in charge of regulating.

But with Cohn stepping aside, it will be interesting to see if Trump will re-up his campaign rhetoric to "drain the swamp."

After all, it looked like the notion that Washington was "Government Sachs" was alive and well during the first few months of Trump's presidency.

Goldman Sachs CEO Lloyd Blankfein, who supported Clinton, even defended his former colleagues for taking jobs at the White House.

"We will continue to encourage our people to contribute to government service if they are fortunate enough to be asked," Blankfein wrote in the company's annual shareholder letter a year ago.