One of the highest-profile prosecutions stemming from the Iraq war period is to go ahead after the US supreme court refused to dismiss manslaughter and weapons charges against four employees of the private security company Blackwater Worldwide.

Supreme court justices declined to review a ruling by a US appeals court that reinstated the criminal charges against the guards for their involvement in the incident, in which 17 Iraqi civilians died and 20 were wounded.

The so-called Nisour Square massacre was the single bloodiest incident involving American private security contractors during the Iraq conflict. It outraged Iraqis, put severe strain on relations between Baghdad and Washington, and served as a watershed moment in the debate surrounding private fighters in foreign war zones.

The shooting took place on September 16, 2007 at the congested Nisour Square intersection, after a convoy of four armoured vehicles manned by Blackwater guards had departed from Baghdad's heavily-fortified Green Zone.

In a span of 15 minutes, heavy gunfire erupted and by the time it was over, more than three dozen Iraqi civilians had been shot, at least 17 fatally. Among the dead was nine-year-old Ali Kinani, who was shot in the head as he rode in a car with his father, Mohammed Kinnani.

The guards maintain that they opened fire in self-defense after being shot at by insurgents.

Witnesses tell a different story – including Mohammed Kinnani, who later sued Blackwater – alleging that the Americans wantonly shot unarmed civilians and were never in any danger. A Blackwater guard who witnessed the shooting later described it as "murder in cold blood".

Investigators with the US military and the FBI determined the guards used deadly force without justification. Iraqi prime minister Nouri al-Maliki described the shooting as a challenge to the nation's sovereignty and threatened to bar the company from operating in the country. Former Blackwater officials claimed the company's then president approved a $1m payment to Iraqi leaders to silence criticism of the killings.

The state department had ordered the guards – Paul Slough, Evan Liberty, Dustin Heard and Donald Ball – to explain to investigators what happened under the threat of losing their jobs. Bruce Bishop, the contractors' attorney, claimed his clients' constitutional right against self-incrimination was violated when these statements were used to charge the men.

On December 31, 2009, federal judge Ricardo Urbina agreed with the guards' claims and threw out the cases, citing "reckless violation of the defendants' constitutional rights". After the ruling, Blackwater reportedly paid the some of the victims' families about $100,000 for each death and issued a statement that it was "pleased" with the outcome.

But in April last year a US federal appeals court reinstated the charges after months of closed-door testimony.

In appealing to the supreme court, the attorney for the guards, Bruce Bishop, argued that by using statements made under duress to charge the men, prosecutors had violated their constitutional right against self-incrimination.

"The issue is of national importance," Bishop said. "The privilege against self-incrimination is a fundamental and universal value in Anglo-American justice."

On Monday the supreme court rejected the application and left the appeal court's ruling intact. But it ordered the judge in the case to conduct a more thorough review, and to separate information based on allegedly self-incriminating interviews with the guards from that which prosecutors had obtained independently.

Prosecutors have already dismissed charges against a fifth guard, Nicholas Slatten. A sixth, Jeremy Ridgeway, has pleaded guilty to voluntary manslaughter and attempted manslaughter and agreed to co-operate with investigators.

Ridgeway, who was serving as a turret gunner for Blackwater on the day of the incident, said in sworn statements that the convoy "opened fire with automatic weapons and grenade launchers on unarmed civilians … killing at least fourteen people." Ridgeway also said the convoy had not been authorized to leave the Green Zone and were ordered to return after they did.

"None of these victims was an insurgent, and many were shot while inside of civilian vehicles that were attempting to flee," he added.

According to evidence in the criminal case, a number of guards in the convoy were "horrified" by what they saw and claimed two of the defendants – Liberty and Slough – were seen high-fiving one another after it was over.

Jeremy Scahill, national security correspondent for The Nation magazine, who interviewed witnesses and victims of the shooting, as well as US military officers who conducted the original investigation at Nisour Square, welcomed the supreme court decision. In an email to the Guardian, he wrote: "What is at the heart of this case is whether or not the US government can or will hold corporate killers accountable for their crimes. The supreme court decision keeps the possibility of justice open in this notorious case.

"It is a devastating statement on the lack of accountability for criminal contractors that the men who did the shooting have not been successfully prosecuted. This was not an isolated incident, unfortunately. I believe that [Blackwater founder] Erik Prince and other top – now former – Blackwater executives should be brought up on charges for all of the murder and mayhem their forces unleashed on the people of Iraq and Afghanistan.

"Short of shutting down the mercenary industry completely, that would be the strongest deterrent we would have to future massacres by US government-funded and supported mercenaries."

After the shooting and other controversies, Blackwater changed its name to Xe Services and then to Academi.

According to former company officials, the payments to the Nisour Square families were not the only ones that Blackwater had made to quell discontent surrounding the shooting. A New York Times report claimed that in December 2007 – three months after the shooting – Blackwater's former president, Gary Jackson, approved payments to Iraqi officials totalling $1m out of fear that the company would no longer be able to operate in Iraq.

At the time, American and Iraqi investigators had already concluded that the shootings were unjustified. The company dismissed the allegations as "baseless".