The Supreme Court of Canada has upheld a lower court ruling that ordered a new trial for a Newfoundland murder suspect, in a decision that puts stricter rules on how police obtain confessions through "Mr. Big" sting operations, but does not forbid the practice.

The justices ruled that so-called Mr. Big stings like the one that convicted Nelson Hart pose major problems — namely, that they tend to produce unreliable confessions and risk becoming abusive.

Mr. Big stings are controversial operations in which undercover officers pose as criminals to draw confessions from suspects.

Hart was convicted in 2007 in Newfoundland of two counts of murder in the drowning deaths of his twin three-year-old daughters based largely on his confession to undercover officers.

In 2012, a majority of the Supreme Court of Newfoundland and Labrador's Court of Appeal overturned Hart's conviction, questioning the reliability of his confession to undercover officers posing as members of the mob. The Crown appealed to the Supreme Court.

In Thursday's majority decision, the Supreme Court justices ruled that Hart's confession during the sting operation cannot be used against him should he face another trial.

Justice Michael Moldaver said, "I have concluded that the April 1 confession must also be excluded. As such, it is doubtful whether any admissible evidence remains upon which a jury, properly instructed and acting reasonably, could convict.

"However, the final decision on how to proceed rests with the Crown," Moldaver added.

Top court warns of dangers

The ruling clarifies for the first time whether the existing legal framework adequately protects the rights of individuals whose confessions were obtained through this technique, and whether these confessions should be admissible in court.

The Supreme Court said that the Mr. Big technique has proven to be an effective investigative tool, but it also comes "with a price."

The strategy, according to the Supreme Court, poses three distinct dangers:

Reliability: It raises the spectre of "unreliable confessions" that have been responsible for wrongful convictions — "a fact," the Supreme Court said, it "cannot ignore." Prejudice: The confessions come with evidence that shows the accused took part in "simulated crimes." The Supreme Court said "this evidence sullies the accused's character and, in doing so, carries with it the risk of prejudice." Potential for police misconduct: The stings "run the risk of becoming abusive," the Supreme Court said. "Thought must be given to the kinds of police tactics we, as a society, are prepared to condone in pursuit of the truth."

To ensure trial judges have the tools to address all three issues, Moldaver suggests taking a two-pronged approach that "strikes the best balance between guarding against the dangers posed by Mr. Big operations, while ensuring the police have the tools they need to investigate serious crime."

Moldaver said trial judges may want to start by determining whether there has been an abuse of the process.

If there was police misconduct, then there is no need to determine whether the value of the confession outweighs its prejudicial effect.

If the defence cannot establish abuse, the onus is on the Crown to establish that, on balance, the confessions are more reliable than they are prejudicial to the accused.

Police have lost a 'strong technique'

The retired RCMP officer who played the Mr. Big role in the Hart investigation said he feels "horrible" about the decision, and that investigators have now lost "another very strong police tool."

The officer, whose identity is protected by an ongoing court ban, said Hart's conviction was just one of hundreds that relied upon Mr. Big stings for evidence against the accused.

The officer told CBC's World Report that Thursday's Supreme Court decision could lead to appeals in a host of other cases, and that while police will adapt the technique for future investigations and abide by the limitations placed on Mr. Big stings, managers will be reluctant to use it.

The result, the officer said, could be that people who committed crimes will not be convicted.

"This is a successful technique, it's a strong technique. The public has to be made aware and realize the police don't just go out and target any individual. There's a lot of legwork done prior to when the undercover operation even started."

Reaction mixed

In an interview with CBC News, Robby Ash and Jamie Merrigan, the lawyers for Hart, said they welcomed the top court's decision.

"We think that the Supreme Court of Canada got it right.… They've put a test in place that will control the way in which the police apply it," Merrigan said.

Ash told CBC News he thought today's ruling would change the way police carry out their investigations.

"Certainly I can't imagine one being carried out in the manner of the way Mr. Hart's was carried out," Ash said.

In a written statement issued Thursday afternoon, RCMP Deputy Commissioner Mike Cabana said the force respected the Supreme Court's decision but noted that the top court did not forbid the technique.

"The RCMP acknowledges the guidance provided by this decision and will take time to review the decision’s potential implications on our operational protocols to comply with societal and legal norms.

"Today’s ruling also confirms that the use of undercover police investigations, among many investigative tools, remain an effective technique to protect the public, thwart the commission of serious crime and resolve historic offences."