For years, a movement to limit the number of migrants into the U.S. and end a system that favors family members of legal residents has had to fend off criticism that it's a thinly veiled attempt to return to a whiter America.

Then its most prominent supporter suggested to members of Congress in the Oval Office this week that the U.S. needs fewer immigrants from Haiti and Africa and more from places like Norway.

President Trump's use of a vulgar term to describe African countries triggered widespread condemnation, and left the small cluster of tough-on-immigration groups whose agenda Trump has embraced scrambling to figure out what to do with the president.

Get Breaking News Delivered to Your Inbox

"They say it's about numbers, merit, security and control," Frank Sharry of the immigrant rights group America's Voice said of organizations that share Trump's desire to reduce both illegal and legal immigration to the U.S. "All of those are coded words that mean fewer brown, black and yellow immigrants into a white nation."

Activists tough on immigration, who prefer the term "restrictionists," argue that the system they espouse — fewer overall migrants, an end to the family-based system that favors relatives of people already legally in the U.S. and a greater emphasis on picking immigrants with skills — is not racially motivated. They note, for example, that immigrants from some African countries have higher rates of education that the U.S.-born population and may benefit from a more skill-based approach.

"People who suggest merit-based may inherently favor white, northern Europeans — that is inherently racist," said Ira Mehlman of the Federation for American Immigration Reform.

"Immigration is not tied to Donald Trump. This preceded Donald Trump," he added, dismissing the president as someone "whose tweets cause people to cringe."

Groups such as Mehlman's helped torpedo immigration overhauls in 2006 and 2013, but they have few overt supporters in Washington. Before Mr. Trump, the most prominent one was Alabama Sen. Jeff Sessions, who became Trump's attorney general and whose former aide, Stephen Miller, is a top White House adviser to the president on immigration.

Sessions is a longtime critic of the country's system that allows people with relatives in the United States a chance to apply for visas. "Almost no one coming from the Dominican Republic to the United States is coming here because they have a proven skill that will benefit us and would indicate their likely success in our society," he said on the Senate floor in 2006. "They come here because some other family member of a qualifying relation is here."

Mr. Trump has embraced Sessions' cause of trying to end "chain migration," a term opponents have long applied to the family-based system but one that got little attention until the president tweeted it in capital letters as he abruptly demanded its end amid immigration talks in September.

Mr. Trump also favors stopping the diversity lottery, a system that reserves visas for people from countries that have relatively few immigrants in the United States. It favors African countries and was part of an immigration deal Trump was negotiating with a group of Democratic and Republican lawmakers Thursday when he made his explosive comments.

Mr. Trump has vaguely disputed some of the accounts of the Oval Office exchange reported by others in the room.

Roy Beck, executive director of Numbers USA, which advocates for reduced immigration, contended the president is mischaracterizing his own immigration agenda. "The president's emphasis on doing away with chain migration and the lottery is about people who are brought into the country with no regard to their skills, education or what their effect will be on the country," Beck said. "It's a mistake to focus on what country anybody comes from."

The Trump administration has announced its intention to end temporary protections for Salvadorans and Haitians who have lived in the U.S. since natural disasters in their countries, as well as President Obama's deportation relief for people brought to the U.S. illegally as children.

The president also has backed a long-shot bill authored by two Republican senators that would sharply cut the number of immigrants allowed into the country and prioritize those who speak English, have a doctorate and have existing job offers.

Both sides of the immigration debate have long agreed that the U.S. should move toward a more skills-based program. But Andrew Selee, president of the nonpartisan Migration Policy Institute in Washington, said recent data suggest the current system may be headed there already: Since 2011, 48 percent of all new legal immigrants have possessed college degrees, well above the 33 percent of U.S. residents with them.

"His vision seems to not only be less immigration but more high-skilled," Selee said of Mr. Trump, "and that may be the system we're already getting."