OTTAWA -- Stephen Harper's political opponents agreed on one thing Wednesday: it is unlikely that the prime minister was in the dark about the controversial Mike Duffy expense payout, given that his right-hand man knew.

Harper said he wasn't going to discuss the latest controversial revelations from the Duffy trial from the previous day that more closely linked Ray Novak, his current chief of staff, to a $90,000 payment to Duffy in 2013.

In the face of that evidence, Harper reiterated his core message on the topic as he continued to face questions on the federal election campaign.

Harper maintained only two people are responsible for wrongdoing -- Duffy over his Senate expense claims and former chief of staff Nigel Wright, who paid them.

But the leaders of the Liberals, NDP and the Bloc Quebecois were united in their skepticism as the Duffy trial's impact reverberated on the campaign.

Novak has been a loyal political lieutenant to Harper since his days in opposition. He was elevated to chief of staff in 2013, when it was revealed that Wright personally covered Duffy's expenses. Harper had wanted Duffy to repay them back himself.

During Wright's six days in the witness box at Duffy's fraud trial, it emerged that Novak was informed about Wright's plan to repay the expenses. This came out in emails and then in a statement the prime minister's former lawyer made to the RCMP that was read into evidence.

"He (Harper) has not been frank, he hasn't told the truth," NDP Leader Tom Mulcair said in Surrey, B.C.

"If Mr. Novak knew, Mr. Harper knew. So, one plus one makes two. Now that we know Mr. Novak knew, we are able to conclude that Mr. Harper knew as well."

Liberal Leader Justin Trudeau said Harper needs to be honest with Canadians about what he knew about the Duffy affair.

"This comes down to whether Canadians can trust the prime minister and his office. Canadians are not fools," Trudeau said in Winnipeg.

"They know that Mr. Harper has not been telling the truth. They deserve him to come clean. The prime minister has to start telling the truth to Canadians."

Bloc Quebecois Leader Gilles Duceppe said it is time either for Novak to quit or for Harper to fire him.

"It's curious that it seems that everybody around Stephen Harper knew what was happening, but Stephen Harper," Duceppe said in Montreal.

"It's very strange because in Ottawa we were used to knowing that Harper was controlling everything and knew everything -- so for the very first time he didn't."

Asked repeatedly about Novak on Wednesday, Harper said he wasn't going to comment on a matter before the courts.

"I am not going to cherry-pick facts that are in dispute before a court," Harper said in London, Ont.

"There are two people in my judgment who are responsible -- Mr. Duffy, who did not reimburse the taxpayers for expenses that I believe cannot be justified, and Mr. Wright, who, although he did reimburse the taxpayers, he did so without my authority and contrary to my wishes."

Novak is still taking part in the Harper campaign but remained elusive on Wednesday. However, a statement by the Conservative campaign spokesman last week about Novak has been contradicted by what's emerged from the Ottawa courthouse since then.

Kory Teneycke told reporters it was "unfathomable" that Novak knew about the $90,000 payment.