Sen. Lindsey Graham (R-SC) on Sunday blamed the White House staff for “making it very difficult” for Congress and President Donald Trump to come to an agreement on immigration, specifically calling out White House aide Stephen Miller for being “an outlier” on the issue.

“I’ve talked with the President, I think his heart is right on this issue,” Graham told reporters Sunday, referencing the Deferred Action for Childhood Arrivals immigration program, one of several key issues keeping Congress from coming to a budget agreement that would reopen the federal government.

“I think he has a good understanding of what will sell,” he continued. “Every time we have a proposal it is only yanked back by staff members. As long as Stephen Miller is in charge of negotiating immigration, we’re going nowhere. He’s been an outlier for years.”

The White House quickly shot back, calling Graham an “outlier” and blaming him for peddling legislation that values “people in this country illegally and unlawfully instead of our own American citizens,” according to a statement shared with Fox News and other outlets.

WH spox Hogan Gidley on Graham remarks about Stephen Miller: As long as Senator Graham chooses to support legislation that sides with people in this country illegally and unlawfully instead of our own American citizens, we are going nowhere. He’s been an outlier for years. — Chad Pergram (@ChadPergram) January 21, 2018

Graham’s attacks on Miller stem from the aide’s highly conservative views on immigration and new reports that both Miller and Chief of Staff John Kelly have been actively stonewalling Trump’s efforts to compromise with Democrats.

According to a new report from The New York Times, President Donald Trump has reportedly privately told lawmakers in recent weeks that he is enthusiastic about coming to an agreement on the legal status of DACA recipients. Both Kelly and Miller have shut down those negotiations, according to the report, pushing for a compromise proposal that also includes tough policies on issues like the border wall and the adoption of a merit-based immigration system.

Both aides have taken hard lines on immigration issues in the past. In his time as Trump’s first homeland security secretary, Kelly ordered his staff to portray undocumented immigrants as criminals. Miller authored the White House’s immigration plan that focuses on a merit-based immigration system and includes signifiant funding for a border wall and a crack down on sanctuary cities.

This summer, Miller also resorted to personally insulting reporters at a White House press briefing who questioned whether the White House’s changes to the green card application process were racist.