NEW YORK — President Donald Trump is back in the New York skyscraper that bears his name as the furor over his reaction to race-fueled clashes in Charlottesville, Virginia, over the weekend shows few signs of dying down.

Protesters on Manhattan’s Fifth Avenue tried to spoil Trump’s homecoming Monday night with signs bearing messages like “stop the hate, stop the lies” and chanting “shame, shame, shame” and “not my president!”

After two days of public equivocation and internal White House debate, the president condemned white supremacist groups by name on Monday, declaring “racism is evil”.

In a hastily arranged statement at the White House, Trump branded members of the KKK, neo-Nazis and white supremacists who take part in violence as “criminals and thugs.”

The groups are “repugnant to everything that we hold dear as Americans,” he said.

The move didn’t quiet the uproar, however. The leaders of four minority House caucus groups wrote a letter to Trump calling for the removal of White House staff aides Steve Bannon, Stephen Miller and Sebastian Gorka.

The heads of the black, Hispanic, Asian and progressive caucuses are calling in the letter for the firings of the Trump administration officials in the wake of a violent, racist rally in Charlottesville, Virginia. The letter asserts their continuing presence in the White House is emboldening a resurgent white supremacist movement in America.

This came a day after Anthony Scaramucci, who was fired as White House communications director after a less than two-week stay, called for Bannon’s ouster.

In his initial remarks on the violence Saturday, Trump did not single out the groups and instead bemoaned violence on “many sides.” Those remarks prompted stern criticism from fellow Republicans as well as Democrats, who urged him to seize the moral authority of his office to condemn hate groups.

Trump’s softer statement Saturday had come as graphic images of a car plowing into a crowd in Charlottesville were playing continually on television. White nationalists had assembled in the city to protest plans to take down a statue of Confederate Gen. Robert E. Lee, and counter-protesters gathered in opposition. Fights broke out, and then a man drove into the opponents of the white supremacists. One woman was killed and many more badly hurt. Twenty-year-old James Alex Fields Jr. of Ohio is charged with second-degree murder and other counts.

Loath to appear to be admitting a mistake, Trump was reluctant to adjust his remarks.

The president had indicated to advisers before his initial statement Saturday that he wanted to stress a need for law and order, which he did. He later expressed anger to those close to him about what he perceived as the media’s unfair assessment of his remarks, believing he had effectively denounced all forms of bigotry, according to outside advisers and White House officials.

Several of Trump’s senior advisers, including new chief of staff John Kelly, had urged him to make a more specific condemnation, warning that the negative story would not go away and that the rising tide of criticism from fellow Republicans on Capitol Hill could endanger his legislative agenda, according to two White House officials.

The outside advisers and officials demanded anonymity to discuss private conversations.

Aides were dispatched to Sunday talk shows but struggled to explain the president’s position. A stronger statement was released — but attributed only to an unnamed spokesperson.

Amy Walter of The Cook Political Report and Susan Page of USA Today join John Yang to discuss the week’s news, including President Trump’s reluctant response to the deadly incident white nationalist rally in Charlottesville, Virginia, plus a new campaign ad for the president and how his base of support is faring and more.

Tougher condemnations began Sunday night with Vice President Mike Pence, traveling in South America, declaring that “these dangerous fringe groups have no place in American public life.”

On Monday, Trump had planned to interrupt his 17-day working vacation at his New Jersey golf club to travel to Washington for an announcement he hoped would showcase some tough talk on China’s trade practices.

But by the time he arrived at midmorning, it was clear all other messages would be drowned out until he said more about Charlottesville.

Trump returned to a White House undergoing a major renovation. With the Oval Office unavailable, he worked from the Treaty Room as aides drafted his remarks.

Reading from a teleprompter, he made a point of beginning with an unrelated plug for the strength of the economy under his leadership. Then, taking pains to insist “as I said on Saturday,” Trump denounced the hate groups and called for unity.

“We must love each other, show affection for each other and unite together in condemnation of hatred, bigotry and violence,” he said.

Trump for the first time mentioned Heather Heyer by name as he paid tribute to the woman killed by the car.

At the trade event later in the day, he was asked why it took two days for him to offer an explicit denunciation of the hate groups.

“They have been condemned,” Trump responded before offering a fresh criticism of some media as “fake news.”

He followed with a tweet declaring “the #Fake News Media will never be satisfied.”

Associated Press writers Julie Bykowicz and Jay Reeves contributed reporting.

WATCH: Charlottesville mayor blames Trump for violent weekend