Five times a day for more than 15 years, Aphadi Wangara has led prayers at Sidi Yahya mosque in Timbuktu, one of three in the ancient Malian desert town. But the day after hardline Islamists attacked and damaged the 15th-century mosque, the softly spoken imam had no consoling words to offer.

"I prefer to keep my silence. What is in my heart cannot be said," said Wangara, who is in his late 60s.

Barely 24 hours earlier, a group of Islamist militants had appeared outside the clay-coated mosque, armed with pickaxes and shouting "Allahu Akbar". They broke down the entrance and destroyed a door locals believed had to stay shut until the end of the world. The militants, who belong to the al-Qaida-linked Ansar Dine, had already defaced mausoleums and tombs of local Sufi saints, prompting Unesco to declare Timbuktu an endangered world heritage site.

"There is a door that absolutely cannot be opened at the entrance of the [Sidi Yahya] mosque," said Haidrata, a resident who gave only his first name. "We believe it is a profanity to open this door; it can only be opened on the day the world will end. The militants broke it down. They were shouting 'Allahu Akbar'. When I asked them why, they said [it was] because they were being accused of destroying endangered monuments when they hadn't done so – they wanted to show what they were really capable of."

Ansar Dine and the Tuareg separatist MNLA movement say the local monuments and distinctive sun-baked mosques renowned for palm trees protruding from earthen walls, sprinkled throughout Mali, are idolatrous and contrary to their strict interpretation of Islam. Sanda Banama, an Ansar Dine spokesman, said the monuments were "un-Islamic".

"In Islam, there are strict laws about the way and size in which tombs are built," Banama said.

Timbuktu locator Photograph: Graphic

Ansar Dine, who have seized the northern two-thirds of Mali after a coup toppled the southern Bamako-based government, continue to control Timbuktu, residents said. "People are still leaving their houses to go to the market but they are scared," said Fatima Sow, who fled to Bamako on Sunday as pickup trucks with Ansar Dine militants prowled the city. "A while back the militants whipped a couple who they said were fornicating before marriage."

Almost 1,000 kilometres south in Bamako, a transitional government struggling to exert control over the vast territory amid violent demonstrations and counter-coup attempts has appeared powerless to stop the attacks. But the assault on the Sidi Yahya mosque has prodded festering anger among ordinary Malians.

"Everybody is absolutely frustrated; everybody is angry. Many of those people will be willing to take to the streets and push the government into doing something," said Tiégoum Maiga, who is organising a march through the capital on Wednesday. "The government says it can't do anything but people in Timbuktu are using sticks and stones to defend themselves."

Cheick Oumar Cisse, a former culture minister and one of Mali's most famous film-makers, said: "It's good that these things are being labelled crimes but it is not even the worst thing these terrorists have done. In January they attacked and disembowelled 100 Malian soldiers and the international community said nothing.

"The more people denounce them, the more they will defy the international community to prove they are masters in their own territory. We hope the destruction will stop here, but these are people who are completely mad and incapable of being reasoned with."

Ecowas, the economic community of west African states, is considering presenting a military plan to the UN security council as militant Islamists, already thriving on a "kidnap economy" operating out of the country's northern desert, have flourished. Protests have also simmered for weeks as the key northern garrison towns of Timbuktu, Gao and Kidal have fallen to Ansar Dine and other Islamist groups. In Gao a municipal councillor was killed last week by the group, residents said.

Officials said squabbles between the various groups also made dialogue difficult. "Basically, they all want power but are not agreed on how to go about it. That means new factions are emerging every day," a Malian presidential aide said.

Thousands of Malians have flocked to neighbouring Mauritania, Niger and Burkina Faso, triggering a potential refugee crisis along its fragile borders. "There is absolutely no order in the north of Mali. Anywhere there is no authority and considering how vast the territory is, it's very worrying for Mauritania, it's worrying for the entire region," a Mauritanian presidential official told the Guardian.