The top UN human rights official has urged Myanmar to end "brutal security operation" against Rohingya in Rakhine state, calling it "a textbook example of ethnic cleansing".

Zeid Ra'ad al-Hussein, addressing the United Nations Human Rights Council in Geneva on Monday, said that more than 270,000 people had fled to Bangladesh, with more trapped on the border, amid reports of the burning of villages and extrajudicial killings.

"I call on the government to end its current cruel military operation, with accountability for all violations that have occurred, and to reverse the pattern of severe and widespread discrimination against the Rohingya population," Zeid said.

"The situation seems a textbook example of ethnic cleansing."

The UN chief's warning comes a day after Bangladesh's foreign minister said "a genocide" is being waged in the country's violence-hit Rakhine state.

"The international community is saying it is a genocide. We also say it is a genocide," AH Mahmood Ali told reporters after briefing diplomats in Dhaka on Sunday.

Ali described actions following the attacks on security forces on August 25 as "revenge" by Myanmar troops.

"Should all people be killed? Should all villages be burnt? It is not acceptable," he said, adding Dhaka was seeking a peaceful solution, not a "war" against Myanmar.

"We did not create the problem. Since the problem started in Myanmar, that's why they should resolve. We have said we'll help them," he said, adding that the problem took a "new turn" after the August 25 attacks.

The minister's comments come as the chair of Bangladesh's National Commission for Human Rights said leading figures in Myanmar could face trial for "genocide" at an international tribunal.

"The way the genocide has been carried out in Myanmar, the way the people were killed in arson attacks, we are thinking about pressing for a trial against Myanmar, and against the Myanmar army, at an international tribunal," Kazi Reazul Hoque said on Sunday while visiting a refugee camp in Bangladesh's Cox's Bazar district, near the border with Myanmar.

"We will come to a decision after assessing what are the steps that should be taken to that end. And at the same time we urge the international community to come forward with their help," Hoque said.