Across Iraq, as the Islamic State of Iraq and the Levant (ISIL, also known as ISIS) loses its territorial footholds, it is leaving behind entire populations terrorised by its fighters, but also a community of families tainted by their relatives' affiliation with the group. These families are among the most vulnerable people in the areas the Iraqi military have retaken, with many of their neighbours apparently eager to take revenge for the past two years of brutal suffering.

There is considerable discussion going on in Baghdad about the need for reconciliation, but no one seems to want to take up the families' cause.

'ISIL families'

A family of 14 in the town of Qayyarah, 60km south of Mosul, which was retaken by Iraqi forces in August, told me in early March that even though they had kicked their son out of their home after he joined ISIL, locals see them as an ISIL family. As a result, on three nights in February, unknown attackers had hurled grenades over the walls surrounding their home, shattering their windows and breaking their door frames.

Another family of 12 in the neighbourhood whose son joined ISIL said they were the target of two grenade attacks in February. A fighter in the area recently told one of their young children who was playing on the street that because of their links to ISIL, the family should leave town for a camp for displaced people. They interpreted this as a threat. A third family of eight a few streets away said their home had been targeted with a grenade two nights before I visited them in March. A month earlier they caught someone trying to set their car on fire with petrol.

OPINION: Iraq's displaced 'in another prison' after fleeing ISIL

All three families told me that they filed complaints with the local police, but that the police had not done anything to search for or punish the attackers, as far as they knew.

After I interviewed these three families, I was stopped by officials stationed in the town by the National Security Service, which is under the command of Prime Minister Haider al-Abadi. They warned me that Human Rights Watch should not be paying any visits to so-called ISIL families as the security officials were watching the families with suspicion, though they conceded that no one in those homes was implicated in any crimes.

If the families are further marginalised, and their children kept out of school, it is hard to see how effective reconciliation can ever take place.

Punishing families

In some places, local authorities have gone a step further. In July, the Babylon provincial council approved a decree calling on authorities to demolish the homes of anyone proved to have participated in terrorist activities, to deport their families from the province, and to authorise legal procedures against the families proved to have concealed their ISIL-affiliated relatives.

In August, the Salah al-Din provincial council passed a similar decree, stating that anyone proved to have been complicit or affiliated with ISIL, including relatives of ISIL members, has no right to return to the province. The decree establishes a committee to seize ISIS-affiliates' property and suspend their, and their families', provision cards. Families that kill or hand over their ISIL-affiliated relatives are exempted.

In January, Sunni tribal units - Hashad al-Asha'ri - within the Popular Mobilization Forces, armed forces under the control of the prime minister, and Iraqi soldiers forcibly displaced at least 125 families said to have ties through family members to ISIL affiliates, sending them to a camp being used as an open-air prison. The fighters destroyed some of the families' homes. The camp has no school for the hundreds of children living there.

Families from Anbar province who live in areas that ISIL had occupied face similar difficulties. In July, local leaders said that people who promoted ISIL are not allowed to return until their charges are reviewed.

People who did not renounce relatives who supported ISIL are only allowed to return home when this situation stabilises. No one knows when that will be. But returning home is a basic right and should not be subject to denouncing relatives or making other declarations.

OPINION: Planning for Iraq's refugees and their future

In Baghdad, government officials and members of the international community have been discussing reconciliation, and some efforts to ensure access to justice for victims of ISIL abuses are under way. But these discussions have yet to engage with the developments on the ground, and critically with the steps that local leaders and other community members are taking to punish these families. If the families are further marginalised, and their children kept out of school, it is hard to see how effective reconciliation can ever take place.

Belkis Wille is the senior Iraq researcher at Human Rights Watch.

The views expressed in this article are the author's own and do not necessarily reflect Al Jazeera's editorial policy.