Story highlights At least 3 people dead, 52 injured after Buddhist mobs attacked Muslim areas in Sri Lanka

The rampage followed a rally by a hardline Buddhist nationalist group led by monks

The rally was prompted by the alleged assault of a monk by Muslim youths days earlier

The towns, popular tourist destinations, are now subject to a curfew

At least three people have been killed and 52 injured after Buddhist mobs rampaged through Muslim areas in southwest Sri Lanka, police say.

The outbreak of religious violence followed a large rally Sunday by the Bodu Bala Sena, a hardline Buddhist nationalist group led by monks, in the town of Aluthgama, about 60 kilometers south of Colombo.

The rally was prompted by the alleged assault of a monk by Muslim youths days earlier, police said.

After the rally, violence erupted on both sides as the demonstrators marched through Muslim neighborhoods, allegedly chanting anti-Muslim slogans, according to a statement by the United Nations High Commissioner for Human Rights, Navi Pillay.

Muslim homes and shops were gutted in the violence, which has prompted Muslims in the region to gather in mosques for safety.

Sri Lankan police spokesman Ajith Rohana told CNN that 12 people from Sri Lanka's Sinhalese majority had been arrested over the violence, some of them members of Bodu Bala Sena.

"They have been remanded at the moment and we're framing charges in due course," he said.

Soldiers had been brought in to enforce a curfew, banning people from the roads or from gathering in public places, in the hope of preventing further clashes in Aluthgama and the nearby town of Beruwala, coastal destinations popular with foreign tourists.

The curfew was relaxed from 8 a.m. to noon Tuesday to allow people to leave their homes to gather supplies. Rohana said that "sporadic incidents" had been reported Monday night, but that authorities had the situation under control.

The violence has alarmed international observers, with the U.N.'s Pillay urging Sri Lanka's government to "urgently do everything it can to arrest this violence, curb the incitement and hate speech which is driving it, and protect all religious minorities."

"I am very concerned this violence could spread to Muslim communities in other parts of the country," she said.

Sri Lanka's Justice Minister Rauff Hakeem, a Muslim, said his party would weigh its future in the government depending on the official response to the attacks. "I am ashamed I could not help my people," he said.

Sri Lankan President Mahinda Rajapaksa, who is in Bolivia for the G77 summit, commented on the clashes on Twitter.

"The Government will not allow anyone to take the law into their own hands. I urge all parties concerned to act in restraint," he wrote.

"An investigation will be held for law to take its course of action to bring to book those responsible for incidents in Aluthgama."

About three-quarters of Sri Lanka's population are Sinhalese, most of them Theravada Buddhists. According to the country's 2011 census, 70.2% of the population is Buddhist, 12.6% Hindu, 9.7% Muslim and 7.4% Christian.

In recent years, the country has witnessed a surge of Buddhist nationalism, led by the Bodu Bala Sena, the country's most powerful Buddhist organization, which has pledged to defend the religion.

Its rally on Sunday was held in response to an earlier incident on Thursday, which is a public holiday in Sri Lanka commemorating the day Buddhism reached the island nation.

Rohana said a Buddhist monk and his driver had been assaulted by a group of four Muslim youths, sparking anger among the Buddhist community. The four alleged assailants were subsequently arrested.

He said the mob violence did not begin until the rally on Sunday.

Fred Carver, of the UK-based Sri Lanka Campaign for Peace and Justice, urged Sri Lanka's authorities to rapidly take charge of the situation.

"We know from past experience that ethnic violence in Sri Lanka rapidly spirals and leads to phenomenal loss of life unless there is swift and effective intervention by the police," he told CNN.

"In the longer term, I hope the Sri Lankan Government reflects on the consequences of patronizing and endorsing extremist nationalists, while at the same time engendering a culture of impunity for those involved in ethnic violence."

The U.S. Embassy in Sri Lanka condemned the violence and called on all sides to show restraint.