Prime Minister Stephen Harper told his Conservative caucus this morning that he's "very upset" by the conduct of some senators and his own office, and asked them to uphold a "culture of accountability."

Harper spoke to Tory MPs and senators at a caucus meeting on Parliament Hill, his first public comments on the Senate spending scandal, but he did not mention by name any of the people who are upsetting him. Media were invited into the meeting but Harper did not take any questions.

The House of Commons was not sitting last week, and later Tuesday, Harper leaves for South America, which means the opposition parties will still not be able to question him about the Senate spending controversy that has rocked his office over the last week.

The Prime Minister's Office confirmed last Wednesday, following a media report, that Harper's chief of staff, Nigel Wright, wrote a personal cheque worth $90,000 to cover Senator Mike Duffy's improperly claimed housing allowance.

Duffy left the Conservative caucus Thursday, followed by Senator Pam Wallin on Friday. Her travel expenses are also under scrutiny. Wright resigned on Sunday and said in a statement that he did not advise Harper of how Duffy's expenses were repaid "either before or after the fact."

"I don't think any of you are going to be very surprised to hear that I'm not happy. I'm very upset about some conduct we have witnessed, the conduct of some parliamentarians and the conduct of my own office," Harper told his caucus.

Harper intent on 'accountability'

The prime minister said his government was elected in 2006 to clean up the Liberal sponsorship scandal and to ensure there are consequences when the rules are broken and that his party's accountability legislation "forever changed the way business is done in Ottawa."

Prime Minister Stephen Harper Harper speaks with Conservative Senate Leader Marjory LeBreton after addressing his caucus for the first time since a Senate expense scandal claimed his chief of staff over the weekend. (Fred Chartrand/Canadian Press)

"Therefore just as we continue to toughen rules, we must also uphold a culture of accountability. And I know that the people in this room have," Harper said.

Harper said he is scrupulous about paying expenses of a personal nature on his own, and then he referenced something he said in a speech in 2005: "Anyone who wants to use public office for their own benefit should make other plans, or better yet, leave this room," Harper said to applause.

Harper said Canadians know he did not get into politics to defend the Senate and that it is his party that has been trying to make it more accountable. The government introduced a bill in 2011 proposing term limits for senators and a system that would allow for Senate elections. It has not been called for debate in over a year.

Earlier this year, the government asked the Supreme Court to weigh in on the proposals and asked about options for abolishing the Senate.

"The Senate status quo is not acceptable; Canadians want the Senate to change," Harper said.

Harper said as his government awaits the Supreme Court ruling, it will take further steps to enhance Senate expenditure and accountability measures, and that he has already spoken to the government's leader in the Senate, Marjory LeBreton, about them.

"She has my full support to accelerate changes to the Senate's rules on expenses and close any loopholes in those existing rules, and I expect Conservative senators, regardless of what opposition you may face, to get that done in the Senate," he told the caucus.

Ray Novak, Prime Minister Stephen Harper's new chief of staff, applauds during Harper's speech to members of the Conservative caucus on Parliament Hill on Tuesday. (Adrian Wyld/Canadian Press.)

The Senate committee on internal economy made recommendations to tighten spending rules in the wake of the reviews that focused on Duffy, Mac Harb and Patrick Brazeau. Their housing and travel claims were all under scrutiny. Harb left the Liberal caucus after the Deloitte audit was released. Brazeau was a Conservative senator but is now sitting as an Independent because he is facing a sexual assault charge.

The Senate wants Harb and Brazeau to pay money back, but both say they did nothing wrong. Duffy said he would repay $90,000 before the audit was finished, and then he did not fully co-operate with the process. Wallin's travel expenses, $321,000 since September 2010, are being reviewed, but the report by Deloitte isn't finished yet.

Harper told his caucus that "distractions" will arise but encouraged the MPs to keep focused on the government's agenda and to "get back to work."

Conservative MPs appeared satisfied with what Harper had to say.

Harper 'exercised leadership'

"I don't think he downplayed the issue at all. Any time that we have any violation of the public trust it's a serious issue. He's addressed that," said Julian Fantino, minister responsible for the Canadian International Development Agency.

"He's taken the right approach. He's laid the law down for those who may have not heard it the first time around. He's prepared to take action to address any errant behaviour, so what more can you ask?"

Treasury Board President Tony Clement said after the caucus meeting that Harper isn't the only one upset.

"We're angry, we're upset, we put in very high standards for governments including ourselves when we first got elected and we expect every parliamentarian to live up to those standards. Clearly some changes have to be made in the Senate accounting," he told reporters.

Clement said the PMO also has to live up to the standards that are expected of senators and MPs.

"I think the prime minister has exercised leadership today," said Clement. "That's exactly what I wanted to hear from him and I heard it today."

Clement said Wright "made a mistake" when he wrote the cheque for Duffy. "He's paid for that mistake." He wouldn't say whether he thinks Duffy should resign from the Senate.

Heritage Minister James Moore said Harper is not avoiding responsibility for what his chief of staff did and he praised his leadership.

"I'm pleased that Mike Duffy is no longer a member of the Conservative Party and for sure I think this has raised into serious question his judgment and I think it's pretty clear that Canadians don't have confidence in him as a senator," Moore told reporters.

NDP MP Charlie Angus said Harper glossed over the Senate trouble and Wright's role in it and he called on the prime minister to "come clean with Canadians."

He said there are still outstanding questions about whether there was a deal arranged in the PMO that would see the Senate go easy on Duffy in its report about his expense claims.

"He went and gave a little song and dance show to MPs saying just get back to work, these are just distractions," Angus said. "They are not distractions. They are about fundamental ethical breaches, about ripping off the Canadian taxpayer and they're about possible illegalities that happened in his office."

Angus said Harper shouldn't be focused on Senate reform, but on secret cheques, whether anything illegal was done and whether Conservative senators are trying to cover up for their colleagues.

In question period, the NDP and Liberals both pushed the Conservatives for more details about the agreement Wright made to personally pay Duffy's expenses.

"Our understanding is that there is no document," Foreign Affairs Minister John Baird responded when the opposition demanded a copy of any agreement.

The opposition also raised questions about Benjamin Perrin, who worked as part of Harper's legal team in the PMO until late March and who was named Monday in media reports.

"I was not consulted on, and did not participate in, Nigel Wright's decision to write a personal cheque to reimburse Senator Duffy's expenses. I have never communicated with the prime minister on this matter," Perrin said in a statement to CBC News.

"In all my work, I have been committed to making our country a better place and I hope my record of service speaks for itself."

Baird repeatedly said that Harper did not know about Wright's cheque until it was reported in the media last week and that the Senate spending matters have been referred to two independent authorities for review. He did not specify what independent bodies he was referring to, but said the government looks forward to the results of those reviews.

The RCMP has said it is looking at the Senate reports on the external audits that were done on Duffy, Harb and Brazeau but has not confirmed whether it is launching an official investigation. Parliament's conflict of interest and ethics commissioner, Mary Dawson, is also examining the matter.

It was not clear what independent authorities Baird was referring to, and NDP MP Nathan Cullen asked for clarification and whether one of the bodies is the RCMP. The NDP has been pushing for a police investigation and has written to the RCMP asking for one.

Baird said the NDP has made "two referrals" and "so there's the answer."

The Prime Minister's Office, however, was asked for further clarification and responded that Baird was referring to the ethics commissioner and the Senate committee that deals with senators' expenses.

Last few weeks 'a gong show'

Some Conservatives are lamenting Wright's departure and blaming the Senate for the loss of a valuable adviser to Harper.

"He brought adult supervision to that job which was frankly much needed, and to see him leave the PMO in this fashion is regrettable," former Conservative senator Michael Fortier told CBC News. "The Senate, however, continues to trip over its own stick. These are own goals that they keep scoring on themselves."

Fortier said the last few weeks on Parliament Hill have been "a gong show."

"The prime minister, [when] he came into Ottawa when we formed government in 2006, it was largely on the heels of the ethics issues that surrounded the Liberal Party and the Liberal brand up until 2006," said Fortier. "For the prime minister to be stuck in a scandal around ethics is I'm sure something he's not quite pleased about, and he's got to address this very quickly and move on."

The NDP wrote to the RCMP on Monday, asking for an investigation into the cheque Wright wrote for Duffy. The Liberals on Tuesday said they are requesting an emergency debate in the House of Commons on the matter.

The Senate is meeting Tuesday at 6 p.m. ET to debate the reports from the internal economy committee on the senators' expenses.