Gary Cohn, the former Goldman Sachs investment banker and President Donald Trump's current economic adviser, is now considered an unlikely pick to lead the Federal Reserve after criticizing the White House’s response to violence in Charlottesville, Va.

Trump has openly floated the idea of nominating the former banker as the potential successor to Fed Chair Janet Yellen -- whose term ends in February – saying in July that he has “great respect” for him.

But Trump has backtracked on the idea recently in private, people familiar with the president’s thinking told the Wall Street Journal. The job prospects shifted mostly due to Cohn's comments about the president’s response to violence in Charlottesville, the sources told the paper.

Cohn issued a stark rebuttal of Trump’s comments in an interview with the Financial Times last month, where he said the administration “can and must do better” to denounce hate groups, including neo-Nazis and the KKK that marched in Charlottesville.

“Citizens standing up for equality and freedom can never be equated with white supremacists, neo-Nazis and the KKK,” he told the newspaper.

The comments were a cold shower to Trump, who did not expect such an attack from his economic adviser, prompting the president to bristle at the very mention of Cohn, one White House official told the paper.

But the chances of Cohn of being appointed the next Fed chairman were still not completely lost, according to another official. A lot hinges on tax reform.

Cohn is considered one of the smartest businesspeople close to Trump and -- along with Treasury Secretary Steven Mnuchin -- has been working on Trump's long-awaited tax plan.

Yellen has not publicly said she would serve another term. Some of her former colleagues told the Journal that she would continue to serve if asked.

Trump was critical of Yellen while campaigning last year, saying her decision to keep interest rates low was aimed at helping the Democratic Party and President Barack Obama. His opinion appeared to change after taking office, saying in a July interview that he was also considering keeping Yellen in her position.

Others said to be under consideration for the Fed job include former governors Lawrence Lindsey and Kevin Warsh, former BB&T Bank chief executive John Allison, and Stanford University economist John Taylor, the Journal reported.

A White House spokeswoman said Cohn was “focused on his responsibilities … including a once-in-a-lifetime opportunity to deliver meaningful tax reform that creates jobs and grows the economy.”