Washington (CNN) When lawmakers went looking for someone to blame for their failed immigration negotiations with President Donald Trump , their ire was drawn on an airy annex on the second floor of the West Wing rather than the Oval Office.

That's the office of Stephen Miller , Trump's 32-year-old White House policy director and speechwriter, who has been pointedly slammed by both sides of the aisle on Capitol Hill during the latest fight over funding the government.

Lawmakers have cast the former Capitol Hill aide as the man-behind-the-curtain for Trump, though White House officials dismiss claims of Miller steering the President on immigration.

"The only person I am aware of with veto power in this country is the President," White House press secretary Sarah Sanders said Monday.

Miller, who gets frequent face time with Trump on trips and around the West Wing, has long been a key voice on immigration reform, regularly reaching out to conservative lawmakers on Capitol Hill and anti-immigration organizations across Washington. And Miller -- whose hard-line immigration policies can be traced all the way back to his high school years in California -- has been the architect of the administration's toughest immigration policies, including the travel ban and Trump's plans to crack down on the criminal gang MS-13.

But there is debate -- inside and outside the White House -- about how powerful Miller actually is. Though Miller is more conservative on immigration than Trump and almost every White House aide, the President ran as a hard-line immigration candidate and looked to make good on that rhetoric during his first year in office

Here is how Miller's role has played out in recent days:

January 11

JUST WATCHED Trump's new immigration plan sparks debate Replay More Videos ... MUST WATCH Trump's new immigration plan sparks debate 03:53

Scene: The now infamous meeting with a bipartisan group of lawmakers where Trump denigrated African nations and Haiti.

Trump, earlier in the day, had invited Democratic Sen. Dick Durbin and Republican Sen. Lindsey Graham to discuss the deal the pair struck to protect the roughly 800,000 Deferred Action for Childhood Arrivals (DACA) recipients who were brought to the United States illegally as children and to beef up border security.

Durbin and Graham said Trump's tone changed in the roughly two hours between the call where he invited them and their meeting. They also felt ambushed: The meeting that they thought would be them and the president included White House aides, like Miller, and conservatives like House Majority Leader Kevin McCarthy and Sen. Tom Cotton.

In the meeting, the once open-to-negotiation Trump was dismissive of protecting the DACA recipients and fumed about the United States' immigration policies, reportedly labeling places in Africa as "shithole" countries and asking why the United States was not welcoming more people from Norway, as opposed to places like Haiti.

January 16

Days after Trump's comments in the Oval Office roiled Washington, both Durbin and Graham were eager to blame Trump's staff -- namely Miller -- for the blow up in the Oval Office.

"Somebody on staff gave him really bad advice," Graham told reporters. "The President I saw on Tuesday is the guy I play golf with. Something happened ... This has turned into an s-show."

Durbin was more direct.

"Any effort to kill immigration reform usually has Mr. Miller's fingerprints on it," Durbin told reporters.

January 19

JUST WATCHED Schumer "went with his heart...not with his head" Replay More Videos ... MUST WATCH Schumer "went with his heart...not with his head" 04:44

As the government shutdown loomed, Trump invited Senate Majority Leader Chuck Schumer to the White House for what ended up being an over-hour-long meeting on funding the government and immigration reform.

Schumer, Trump and their chiefs of staff sat in the Oval Office, where -- according to the New York Democrat -- Schumer offered to fund the military and put Trump's signature wall around the US-Mexico border "on the table." Schumer also claimed that Trump agreed to back a short-term funding agreement -- likely four or five days -- to give negotiators more time on immigration.

Trump, Schumer said, then changed his mind and called him a few hours later to spike the plan and propose a three-week deal.

"Negotiating with this White House is like negotiating with Jell-O," Schumer later said on the Senate floor, adding that "as you take one step forward, the hard right forces the President three steps back."

Those hard-right forces, in the minds of Schumer and other Democrats on Capitol Hill, include Miller and chief of staff John Kelly.

January 20

By Saturday -- the first day of the shutdown -- the White House had had enough with the conventional wisdom that Trump wants an immigration deal while Miller and others pushed him right.

This feeling was deepened when Graham opened up on Miller to reporters on Capitol Hill.

"I've talked with the President, his heart is right on this issue, I think he's got a good understanding of what will sell, and every time we have a proposal it's only yanked back by staff members. And as long as Stephen Miller is in charge of negotiating immigration we're going nowhere," Graham said. "He's been an outlier for years."

The comment by the on-again, off-again Trump critic annoyed the White House and elicited a response.

"As long as Sen. Graham chooses to support legislation that sides with people in this country illegally and unlawfully instead of our own American citizens, we're going nowhere," White House deputy press secretary Hogan Gidley said Saturday, parroting Graham. "He's been an outlier for years."

January 22

Miller has yet to comment on the attack leveled against him by Democrats and Republicans on Capitol Hill.

But the White House press office bolted to his defense as an agreement to fund the government loomed on Monday.

"I think that's a sad and desperate attempt by a few people trying to tarnish a staffer," Sanders said Monday. "Stephen's not here to push his agenda. He's here to push the President's agenda like everybody in this building."

Raj Shah, speaking with CNN after lawmakers on Capitol Hill struck a deal, echoed Sanders.

"Those charges are, frankly, ridiculous and they are a little insulting," he said. "This is the President of the United States. He is setting the agenda."