Washington (CNN) House Democrats were united in their vote to condemn President Donald Trump's racist tweets this week, but some Democratic lawmakers are quietly expressing concern that far-left progressives have outsized influence in their caucus.

Several Democrats -- from across the ideological spectrum -- are growing tired of the repeated headlines involving four progressive congresswomen as the caucus tries to push its agenda. In conversations that took place before the President hosted a rally in North Carolina Wednesday night where the crowds chanted "send her home" in reference to Rep. Ilhan Omar of Minnesota, Democrats said they want to keep the spotlight on the issues that got them elected, not a fight with the President.

"The President's words and actions speak for themselves. We need to focus on the issues that got (Democrats) here: jobs, health care ... instead of the issues the President brings up deliberately," said one House Democratic lawmaker, who asked for anonymity to speak freely. "Anything that takes away from bread-and-butter issues is playing into his hands."

"The President won this one," said another House Democratic lawmaker of the showdown. "What the President has done is politically brilliant. Pelosi was trying to marginalize these folks, and the President has now identified the entire party with them."

House Speaker Nancy Pelosi, along with many in the caucus, repeatedly described the President's attacks as a distraction this week -- as did members of "the Squad," a nick name for the group made up of Reps. Alexandira Ocasio-Cortez of New York , Ayanna Pressley of Massachusetts, Rashida Tlaib of Michigan and Omar.

"I encourage the American people and all of us in this room and beyond to not take the bait," Pressley said at a news conference Monday with the other three congresswomen. "This is a disruptive distraction from the issues of care, concern and consequence to the American people."

Given the blistering nature of the President's comments about the congresswomen , leadership felt a strong need to forcefully condemn his words.

House Majority Leader Steny Hoyer, who controls the floor, argued this week that responding "in a collective way" with a vote was "absolutely essential for us to do," describing it as an "opportunity to tell America we do not agree. This is offensive. This is wrong in America."

But it's the way they did it -- with a vote that also led to a dramatic fight between Pelosi and Republicans on the floor -- that has some Democrats frustrated, according to multiple sources.

Disunity on display

The tension within the party hit a boiling point last month over a border aid fight, one that ended with Pelosi ultimately siding with many more mainstream and moderate Democrats in the caucus over progressive objections. That decision riled up progressives to the point that one of its leaders, Rep. Mark Pocan, fired off a tweet comparing moderates to child abusers, and Ocasio-Cortez's chief of staff posted a now-deleted tweet comparing moderate Democrats to segregationists.

Those tweets prompted outrage in the caucus.

"I can't tell you the number of members who are angry and annoyed about them criticizing us," said one House Democrat.

"Less than two weeks ago when members of our caucus were trying to support funds for children at the border," progressives were arguing that "we're pro-putting kids in cages, we're against human rights. So there's frustration," said another House Democrat.

Amid the intense infighting, Pelosi delivered an impassioned pitch for unity last week at a closed-door caucus meeting and defended moderates against criticism from progressives, explaining why they're not always going to be on the same page with the rest of the caucus.

"Every day some of our members have to fight the fight for their reelection. It's easy for me in my district, right?" Pelosi, who represents San Francisco, said, according to an aide in the room. "I never have to worry about whether a Democrat will represent that district, whether it's me or somebody else. But, in their districts, it makes a difference for what we can do for the American people if we have the majority."

If Democrats have an issue with the way their colleagues voted on a bill, Pelosi urged members to "make me the target."

Trumps tweets reset the focus

But after the President tweeted startling, racist comments about the four congresswomen over the weekend -- falsely implying all four were from other countries and saying they should "go back and help fix the totally broken and crime infested places from which they came" -- Pelosi rushed to the defense of the four progressive freshmen.

Across the ideological spectrum, Democrats sharply criticized the President for his comments, and House Democrats projected unity amid the attacks from Trump. "His tweet refocused everybody on exactly where the enemy is," said Rep. Pramila Jayapal, the co-chairwoman of the Congressional Progressive Caucus.

"I don't feel a fracture," Omar said in an interview with "CBS This Morning." Ocasio-Cortez said during the same interview that there may be disagreements "from time to time," but "that does not mean there is a fundamental fracture."

Pelosi is keenly aware of the importance of keeping her party united -- after all, they're the ones who won the majority for Democrats -- but some still feel she's enabled progressives to seize the spotlight regularly.

Moderate Democrats are especially concerned about the attention on the far left, and they're not happy about the way leadership brought a resolution to the floor condemning the President, according to multiple sources. Many of them hail from Trump districts and face tough reelections.

A Democratic aide said that leadership tried to keep the wording of the resolution so that it had broad appeal, with language from President Ronald Reagan, for example. The aide said that leadership also tried to keep items that progressives were pushing out of the resolution.

"You have no idea of the provisions that some people wanted to have in that resolution," Pelosi said during her weekly news conference on Wednesday. "This was as benign -- it condemned the words of the President: not the President, but the words of the President."

Impeachment fight might not poll well

Supporters of leadership argue it's a diverse caucus and they're trying to keep everyone on message. At a meeting convened by the speaker on Wednesday, according to one Democratic lawmaker, members heard a report on internal polling from the Democrats' campaign arm, which showed the most forceful argument against Trump is that he's ineffective and prohibiting progress on issues like infrastructure and jobs.

The message to the House Democrats: Getting drawn into fights with the President like the one they're having on impeachment and the current one on race does not work well.

That's also why Pelosi has worked to tamp down feverish impeachment pushes, including an effort by Rep. Al Green to force a vote on impeachment on the floor this week. Democratic leaders led a successful effort to table the resolution, essentially killing it -- something that moderates wanted.

But even if Green's resolution has been pushed off for now, Democrats are soon to face a looming set of contentious issues that threaten to deepen some of the divides within the caucus, including a vote next week on a resolution to condemn the global boycott movement against Israel -- a movement supported by Democrats like Omar and Tlaib.

Also next week, special counsel Robert Mueller will testify on Capitol Hill , a public appearance expected to reignite the fight over impeachment. Democrats must also grapple with a looming fight to stave off another government shutdown.

And there will be plenty of provocations from the President, whose attacks on the "squad" only appear to be escalating.

One such example came Wednesday evening. At his rally in Greenville, North Carolina, Trump, railed against the four progressive Democratic congresswomen individually and by name and painting them as the face of the Democratic party.