Two of Sri Lanka's most prominent rights activists have been detained under anti-terrorism legislation which allows them to be held for up to 18 months without trial and can lead to prison sentences of up to 20 years.

The arrests will prompt fresh criticism of the human rights record of the government of President Mahinda Rajapakse.

Ruki Fernando and Father Praveen Mahesan were arrested on Sunday in the small town of Kilinochchi, in the north of the island nation. Authorities have said the pair are being held on suspicion of inciting racial or religious hatred or violence between ethnic groups.

Activists in Sri Lanka and overseas say they are deeply worried. The local National Peace Council said in a statement it viewed "with great concern" the arrest of "two of the leading human rights defenders in [Sri Lanka]", adding: "Their commitment to sustainable peace and reconciliation and promotion of humanitarian norms are unquestionable."

Fred Carver, of the UK-based Sri Lanka Campaign, said: "Ruki and Father Praveen have worked tirelessly to combat extremism and build a more tolerant Sri Lanka. It is bitterly unfair that they should be charged with promoting ethnic discord, and the idea that they could have any links to terrorism is patently absurd."

Sri Lanka has suffered decades of conflict between a largely Buddhist Sinhalese majority and a largely Hindu Tamil minority. Kilinocchi was the headquarters of the Liberation Tigers of Tamil Eelam (LTTE), a violent separatist organisation defeated in 2009 after years of bloody military struggle costing tens of thousands of lives.

Both sides in the civil war committed widespread abuses and the two men had been working with families of individuals thought to have been abducted by either the LTTE or the authorities and killed during or immediately after the end of the conflict. Their research had recently focused on the case of an activist detained this month by authorities on suspicion of harbouring a criminal.

Police have three days from arrest to charge Fernando and Mahesan, release them, or get a detention order on grounds of national security from the president. Officials say the two men will have access to lawyers and can see their families.

Coming only months after the controversial Commonwealth Heads of Government Meeting (Chogm) in Colombo, the arrests will embarrass the British prime minister. David Cameron refused calls to boycott the Chogm, which was also attended by Prince Charles, but said he had given Rajapakse a tough message about the need to protect human rights and advance political reconciliation with the Tamil minority in the country.

A British diplomat in the region told the Guardian last week: "We believe we saw some very positive outcomes from that decision [to attend]."

Cameron also said he would back an international inquiry into alleged war crimes if there was no progress towards accountability for war crimes committed by Sri Lankan officials and soldiers in the final days of the civil war.

A proposed resolution to be presented to the UN Human Rights Council in Geneva this month calls for an international investigation into "past abuses and to examine more recent attacks on journalists, human rights defenders, and religious minorities". The draft resolution is sponsored by five countries – the US, UK, Mauritius, Montenegro and Macedonia.

Other incidents that have worried international campaigners include the arrest of four young men near the northern city of Jaffna in late November, for defacing an image of Rajapaksa, and the death of a Tamil prisoner who was a British citizen in February in the main Colombo jail.

The authorities said Viswalingam Gopithashe, convicted in 2012 of helping the LTTE, died from heart failure. Family members claim he was not given adequate medication after twice being beaten up by other inmates.

There are also concerns about a rise in attacks against religious minorities in Sri Lanka in recent years. These have targeted Muslims – about 9% of the population – or the small Christian minority, who are a mix of Sinhalese and Tamil, and are blamed on new Sinhalese groups with names such as Buddhist Brigade and Buddhist Heritage Fortress.

Numerous small independent churches have been attacked in recent months. In some incidents, clergy and worshippers have been beaten up or threatened with death. Non-Christian villagers have been warned against the influence of the "traitors".

In a letter this month to the Human Rights Council, 24 civil society groups said there had been more than 300 attacks on Muslim and Christian places of worship in 2013. Sri Lanka's foreign minister, GL Peiris, told the UN there had been only 77 attacks on religious targets from 2009 to 2013, 16 of them against Buddhist temples. The police always took action and would be bringing several monks to court shortly, he added.

The government still faces a host of accusations that torture, enforced disappearances and extrajudicial killings have continued since the end of the war.

In an interview in September with Al-Jazeera, Rajapaksa dismissed the charges that security agencies continued to abduct or kill as "nonsense" and "propaganda". Though senior Sri Lankan officials privately admitted to the Guardian last year that there are "some violations", they said these are "to be expected ... in any developing country struggling to come to terms with the effects of a long conflict".

Fernando said this month that the intensity and number of these had certainly reduced in that period, but they were still happening. "And the breakdown of the institutions meant to protect us, the compromising of the judiciary, has got worse," he said.