By Fred Abrahams, Special to CNN

Editor’s note: Fred Abrahams is a special adviser at Human Rights Watch. The views expressed are his own.

All civilians deserve protection, but some civilians deserve more protection than others. Or so it seems in Libya today.

Two years ago, the U.N. Security Council authorized a military operation by NATO with a mandate to protect civilians who were under attack by Moammar Gadhafi’s forces. That operation led to Gadhafi’s fall.

Today, long after the fighting has stopped, those who are rightly or wrongly perceived to have supported Gadhafi are under threat. Thousands of women and children have been displaced from their homes and living in camps, often harassed. Men have been detained, tortured and killed. They need protection, but the nations that intervened two years ago have done virtually nothing on their behalf.

The most pressing case involves the former residents of the town of Tawergha, which had a pre-war population of about 42,000. Tawerghans formerly enjoyed Gadhafi’s financial and political support, and the town became a military staging ground during the 2011 war. Many fighting age men from Tawergha joined Gadhafi’s fight.

Some of these men allegedly committed atrocities during the war in the nearby city of Misrata, which suffered from a brutal, two month siege in which hundreds of civilians died. Misratans say that Tawerghan fighters committed killings and rapes in their city and that now it is time to take revenge.

And revenge is what the anti-Gadhafi militias of Misrata have been taking, forcing all Tawerghans from their town. Spread across Libya, Tawerghans have been hunted down, detained, tortured and killed. Satellite imagery analyzed by Human Rights Watch corroborates what we saw on the ground: the systematic destruction of the town’s residential, commercial and industrial structures after the fighting had stopped in an apparent attempt to prevent returns.

The Misratans demand justice for the crimes committed against them, and this is their due. But justice is not served by punishing an entire community for crimes committed by some of its members – that is collective punishment.

But while the U.N. Security Council and its powerful members jumped to protect Libyan civilians when Gadhafi was the enemy, they have not taken serious action against the revenge attacks that Tawerghans and other displaced communities in Libya are suffering from today – about 60,000 people in all, according to the U.N. High Commissioner for Refugees.

In its resolution on Libya this month, the Security Council rightly expressed concern about reprisals, torture and executions, but failed to mention the plight of Tawerghans. Even the U.N. mission in Libya, watching developments up close, has not made the abuses against Tawerghans and other allegedly “pro-Gadhafi” communities a central theme.

International legal obligations suggest that they should. The violations against Tawerghans are widespread, systematic and sufficiently organized to be crimes against humanity. The U.N.’s commission of inquiry on Libya made this point a year ago.

The Libyan government has a responsibility to protect its people from such serious crimes, and to hold perpetrators accountable. The Security Council has a responsibility to help Libya achieve these goals. At a minimum, the latter should ask Libya to report regularly on the steps it is taking to protect displaced people and facilitate returns. The imposition of U.N. sanctions on responsible individuals would also have immediate effect.

The International Criminal Court can also investigate these crimes because its mandate in Libya is ongoing. Militia commanders and senior officials in Misrata could be held criminally responsible for ordering these crimes, failing to prevent them or failing to punish the attackers.

The Libyan government says it does not condone these crimes and would like to see them stop. Prime Minister Ali Zeidan has spoken forcefully about abuses by the many militias in Libya whose actions he does not control. But that does not absolve him of the responsibility to do more on behalf of the Libyan citizens who are suffering the wrath of victorious rebels. And it does not relieve the Security Council of its responsibility to demand protection for civilians at all times, when it’s politically convenient and when it’s not.

The failure to ensure protection from some of the worst crimes undermines the credibility of governments that said they intervened in Libya to protect civilians. Instead, governments supporting Libya’s transition should pressure Libya, while providing appropriate assistance, to ensure that displaced people can safely return to their homes and share in the benefits that the Libyan popular uprising and international military intervention were supposed to bring about.