Washington (CNN) Patience is waning on Capitol Hill for Health and Human Services Secretary Tom Price who reportedly spent thousands of taxpayer dollars on privately chartered flights to travel around the country.

"I think the President's mad as hell," a Republican senator told CNN Thursday. "What the f*** was he thinking?"

"It's just stupid," the senator added. "I think Price can probably survive this. Don't know."

On Thursday, Price said he planned to "write a personal check to the US Treasury for the expenses of my travel on private charter planes" and that he "would no longer take private charter flights as the secretary of HHS no exceptions."

Politico, who first reported about Price's travel , has identified Price took $400,000 in chartered flights so far including one flight worth $25,000 between Washington and Philadelphia. In recent days, the President's frustration has also been palpable. Asked Wednesday if he'd fire Price Wednesday, Trump responded "we'll see."

Ultimately, the decision to keep or fire Price falls to Trump, but a few members on Capitol Hill weren't coy about their misgivings Thursday before Price announced he'd pay back his travel.

"(Everything) that happens around here is based on appearances. And if it just appears wrong, don't do it," Sen. Lisa Murkowski, an Alaska Republican, said of Price, before turning to a reporter and pointing at him and continuing, "I don't know, you have a good job, you've been working there awhile. What would others think if you were chartering a private jet to go and do interviews with people? They probably wouldn't like it and particularly if you know the company was paying for it, they didn't get that same benefit."

Asked what she thought should happen to Price, she said it wasn't up to her.

"It's a matter of appearances and again, I think the direction just needs to be set from the top that you don't do this."

As a Republican congressman, Price built a reputation as a fiscal conservative, going as far as admonishing Democratic members of Congress at one point for using private jets. Appearing on CNBC , Price called the travel "another example of fiscal irresponsibility run amok in Congress right now."

In the House, many of his former colleagues were reluctant to criticize Price on the record.

"We all know Tom very well and without knowing the full story, which I don't and knowing what I know about Tom and knowing what everybody in that room knows about Tom, we all want to give him the full benefit of the doubt that what your hearing may not be the full story," Rep. Tom Rooney, R-Florida, said Wednesday.

By Thursday the patience among his colleagues was subsiding a bit with members openly criticizing his decision.

"I think his use of private flights was not the most prudent thing for a fiscal conservative," said Rep. Mark Meadows, the chairman of the conservative House Freedom Caucus. "That being said, apologize, make sure it doesn't happen, put policy in place under this administration to make sure it doesn't happen."

Rep. Steve Womack, R-Arkansas, was unequivocal that he still supported Price in his role at HHS, but he said that "right now, this is between the President and the secretary."

"We don't want it to just linger forever," Womack said. "We have a lot of important work ahead of us and if the secretary is going to remain in his capacity then he needs to be given that ability and if he's not, change should occur as quickly as possible because of the unfinished business that we have."

Senate Republican Whip John Cornyn of Texas also tried to stay away from weighing in on the controversy Thursday, telling reporters he didn't know anything more than what he had read in the press about Price's private charters.

But when a reporter pressed him for a response, the normally talkative Texan paused with an awkward smile for a solid 10 seconds before saying, "good to talk to you" and disappearing onto the Senate floor.