The fight between the opposition and the government over the conduct of Finance Minister Bill Morneau spilled out of the House of Commons and into the foyer Tuesday with Morneau threatening the Conservatives with legal action.

The latest round of verbal fisticuffs began Monday when the Official Opposition went after Morneau over the timing of the sale of $10 million worth of shares in his family-founded human resources firm, Morneau Shepell.

Those 680,000 shares were sold in November, 2015, days before Morneau announced changes to raise income taxes on the country's highest earners.

Conservative finance critic Pierre Poilievre suggested the announcement caused the stock market to drop and that the value of the Mornell Shepell shares would have declined by about $500,000. He repeatedly asked whether the shares were sold by Morneau or someone else, suggesting the timing of the sale was no coincidence.

On Tuesday, before leaving Ottawa to give a speech in Toronto, Morneau told reporters in the foyer insinuations of wrongdoing on his part were "absolutely absurd."

"I am saying absolutely that if the members take the allegations that they're making inside the House outside the House they will absolutely be hearing how the legal system works from me," he said.

Finance Minister Bill Morneau challenges the Conservatives to state their latest accusations against him outside the House of Commons where they are not protected by parliamentary privilege 0:50

Members of Parliament are granted protection against civil or criminal liability for actions or statements made in the House of Commons.

"The, in my mind, absolutely crazy idea that something we campaigned on, that we talked to 36 million Canadians about, which was a rise in taxes on the top one per cent [of income earners], was somehow knowledge that was only my knowledge makes absolutely no sense," Morneau said in the impromptu scrum Tuesday.

Poilievre stages walkout to get answers from Finance Minister who was out of town giving a speech 1:29

Morneau also swatted away suggestions he knew how the market would react.

"You literally cannot make this up. It is absolutely absurd.…The opposition clearly has no idea how the stock market works," Morneau said.

Later Tuesday during question period, Poilievre responded by leaving the House early to ask his questions in the foyer, where he is not protected by parliamentary privilege.

"I am absolutely confident that everything I've said out there and in here is true," Poilievre said during question period. "Would he commit that, if I go out and repeat my question in the lobby at this moment, that the finance minister will meet me there and answer the question?"

There, Poilievre told reporters he is not alleging insider trading and said he's simply asking when Morneau sold his shares.

He then went back into the House and said Morneau had failed to answer his question, skirting the rule that MPs not mention another member's absence in the House.

Morneau had previously said he wouldn't be in question period.

Conservative finance critic Pierre Poilievre asks whether Finance Minister Bill Morneau sold shares in his family company before introducing legislation that lowered stock prices two years ago. 2:45

The NDP is also taking aim at Morneau over the share sale. The party's ethics critic, Nathan Cullen, has written a letter to Ethics Commissioner Mary Dawson, requesting she investigate.

"If the finance minister used his inside knowledge to sell his shares at an advantageous time to financially profit, it would be in direct violation of the rules that prevent someone from profiting directly from their work in government," Cullen wrote in a letter that was sent Monday afternoon.

"I respectfully ask that you look into this matter as urgently as possible," the letter says.

Dawson is already investigating Morneau about whether he was in a conflict of interest when he sponsored a bill related to pension reform.

Morneau still owned shares in his family business, Morneau Shepell, a benefits and pension services firm, at the time the bill was introduced.