A prominent Republican pollster is urging his party to resist stoking racial anxieties among white voters and “throw in the towel on the immigration issue.”

Freed Steeper, who served as an adviser to both Presidents Bush, told the New York Times in a story published Wednesday that the GOP may continue its struggles in national elections if it keeps up its often derisive rhetoric toward Hispanic voters.

“Racism may be a part of it,” Steeper admitted. “The Republican Party needs to stop pandering to that.”

Steeper then gave Republicans some blunt advice on the matter.

“The Republican Party needs to throw in the towel on the immigration issue,” he said.

The prospects of immigration reform have been shrouded in doubt after months of inaction by the GOP-led House of Representatives on a bill that passed the Senate. Sen. Marco Rubio (R-FL), who helped lead the push to pass that legislation, said earlier this week that Congress should dial back its reform efforts.