The United Nations has taken its first concrete step towards launching an inquiry into war crimes allegedly committed by both Sri Lankan state forces and Tamil separatist rebels during the island nation's 25-year brutal civil war.

By a vote of 23 states in favour, 12 against and 12 abstentions, the 47-member UN Human Rights Council adopted a resolution presented by the United States on behalf of countries including Britain, which said that an inquiry was necessary because the government had failed to investigate properly.

"The international community has become increasingly concerned by the continued lack of progress in achieving reconciliation, justice and accountability for serious allegations of violations of international human rights law and international humanitarian law," Paula Schriefer, the US deputy assistant secretary of state, told the Geneva forum.

Sri Lanka has been under growing international pressure to deal with war crimes allegedly committed in the final stage of the conflict in 2009.

Mahinda Rajapaksa, the Sri Lankan president, rejected the resolution ordering a war crimes probe, telling AFP news agency that he would instead press ahead with his own reconciliation plan.

"We reject this," Rajapaksa said. "This resolution only hurts our reconciliation efforts. It does not help. But I am not discouraged. We will continue with the reconciliation process I have started."

Many thousands died during the final defeat of the Liberation Tigers of Tamil Eelam (LTTE) five years ago, some under bombardment from government troops or allegedly executed in cold blood, others when forced to bear arms by the rebels or when civilian refugees were mixed with combatants.

A UN report found that as many as 40,000 people may have been killed in the last months of the civil war, though the government disputes that figure.

David Cameron, the British prime minister, loudly backed a UN investigation during his controversial trip to Sri Lanka to attend the Commonwealth Heads of Government Meeting in November.

Sri Lanka has suffered decades of conflict between a largely Buddhist Sinhalese majority and a largely Hindu Tamil minority.

Navi Pillay, the UN commissioner on human rights, said new evidence continued to emerge. A series of what appear to be mass graves have recently been discovered in the north of Sri Lanka.

Ravinatha Aryasinha, Sri Lanka's permanent representative to the UN in Geneva, told the talks that the Indian Ocean island nation had made tangible progress in addressing accountability and insisted that parallel truth-seeking processes would be counter-productive.

"Sri Lanka categorically and unreservedly rejects this draft resolution as it challenges the sovereignty and independence of a member state of the UN … and is inimical to the interests of the people of Sri Lanka," Aryasinha said in a speech before the vote.

States that voted against the resolution included China and Pakistan, while India was among those that abstained.

Indian officials said the resolution would "hinder efforts rather than contribute constructively".

International campaigners welcomed the vote.

Edward Mortimer, chair of the Sri Lanka Campaign for Peace and Justice, said it had long been "obvious that only an independent international investigation can establish the truth about what happened in Sri Lanka in the last months of the civil war".

"There is credible evidence that both sides committed crimes against humanity, and the lack of accountability for this has engendered a culture of impunity which has fled to further outrages. The international community must not now look away from Sri Lanka."

David Griffiths, Asia-Pacific deputy director at Amnesty International, said the UN inquiry had brought new hope for the thousands of victims of abuses in Sri Lanka.

Earlier this month two prominent human rights activists were detained in the north of Sri Lanka and, though released, still remain prohibited from talking to the media.

Several other campaigners remain in detention. Authorities say many are linked to the LTTE, a claim strenuously denied by their supporters.

The UN vote is likely to reinforce perceptions that Sri Lanka is being "victimised" among policymakers and senior officials in the country. Rajapaksa remains popular among the Sinhalese majority and has repeatedly said that Sri Lanka is being "bullied" by bigger nations, particularly the US.

Last year Pillay, the UN commissioner, accused the country's powerful defence secretary, Gotabhaya Rajapaksa, the brother of the president, and other government officials of waging a campaign aimed at discrediting her and her office.

Pillay, a former South African judge of Indian Tamil ethnicity, concluded that democracy was being undermined and the rule of law was being eroded in Sri Lanka, with the country increasingly becoming an authoritarian state.