Story highlights Sen. Lindsey Graham, R-South Carolina, appears on CNN's "State of the Union"

Graham rebukes his U.S. House counterparts for failing to act on immigration bill

He says President Obama is going too far with executive action

Obama is defending his stance, saying other presidents have acted likewise

A key Republican senator chastised his House GOP colleagues Sunday for failing to pass a comprehensive immigration bill.

"Shame on us as Republicans," Sen. Lindsey Graham, R-South Carolina, told CNN's Chief Political Analyst Gloria Borger on "State of the Union."

"Shame on us as Republicans for having a body that cannot generate a solution to an issue" involving national security as well as cultural and economic considerations. "The Senate has done this three times," Graham said.

"I'm close to the people in the House, but I'm disappointed in my party. Are we still the party of self-deportation? Is it the position of the Republican Party that the 11 million must be driven out? I have never been in that camp as being practical. I am in the camp of securing our borders first, fixing a broken legal immigration," said Graham, one of the Republican senators who pushed a bipartisan bill to passage there in 2013, but on which the House never voted.

That bill included legal status for the roughly 11 million illegal immigrants in the United States and tougher enforcement measures.

Graham, though supportive of an immigration overhaul, criticized President Barack Obama for taking executive action that could allow almost 5 million undocumented immigrants to avoid deportation and grant many of them work permits. He said he agrees with many of his fellow Republican lawmakers who have called the President's action unconstitutional.

JUST WATCHED Graham: Benghazi report 'full of crap' Replay More Videos ... MUST WATCH Graham: Benghazi report 'full of crap' 03:16

"It's one thing to say, as an executive agency, I don't have the money to prosecute everybody or to deport everybody, so I'm going to rank them in order. It's another for the President of the United States to say, not only will I decide not to prosecute a group of people, but I will affirmatively give you legal status ... that is well beyond executive action."

While some in the GOP have called for possibly impeaching the President for his immigration action, Graham said he's not among them.

"I'm not going to go down the road of impeaching the President. And let me tell you why. Immigration has been dogging the country since 2006. The President is frustrated. I have a solution that I have been supporting that is comprehensive, that would allow legal status to the people in question.

"But you do it through a congressional action, where you get the entire system fixed. His action does not secure the border. It doesn't fix a broken legal immigration system. And it leaves millions of people left out in terms of the 11 million. "

Obama, meanwhile, defended his decision, pointing out that his predecessors have taken such actions.

"Congress has a responsibility to deal with these issues, and there are some things that I can't do on my own," the President told ABC's "This Week" in a taped interview. "What I do have is the legal authority to try to make the system better. Given the resource constraints that we have, we have to prioritize."

Obama answered his critics who pointed out he had previously said he was limited in how much he could do saying earlier "I am president. I am not king. I can't do these things just by myself."

"What is absolutely true is that we couldn't solve the entire problem and still can't solve the entire problem," Obama told ABC. "But what we can do is prioritize felons, criminals, recent arrivals, folks who are coming right at the border and acknowledge that if somebody's been there for over five years -- they may have an American child or a legal permanent resident child -- it doesn't make sense for us to prioritize them when we know that we need more resources."

He also disputed his critics who have called him an imperial president.

"If you look -- the history is that I have issued fewer executive actions that most of my predecessors by a long shot," he told ABC.

"The difference is the response and specifically the response of some of the Republicans. But if you ask historians, take a look at the track records of the modern presidency, I've actually been very restrained, and I've been very restrained with respect to immigration. I bent over backwards and will continue to do everything I can to get Congress to work because that's my preference."