(CNN) ISIS militants on Wednesday stormed a central Syrian city that's just meters from the ancient ruins of Palmyra, activists said.

Militants controlled most of the city by the evening and had taken over Tadmur's prison, the observatory said. Meanwhile, Syrian government forces reportedly have retreated to the city's security headquarters, and clashes with ISIS are ongoing.

Palmyra's centuries-old remains of temples and other structures are a UNESCO World Heritage Site in the Homs countryside northeast of Damascus.

United Nations and Syrian officials have expressed fears that ISIS aims to destroy the ruins, just as it bulldozed the ancient Assyrian city of Nimrud and took sledgehammers to statues in Iraq's Mosul Museum

UNESCO Director-General Irina Bokova said she was deeply concerned about reports of the battles.

Photos: How ISIS ravaged Palmyra's world treasures Photos: How ISIS ravaged Palmyra's world treasures Before: The ruins of the 2,000-year-old Temple of Baalshamin are seen in Palmyra, Syria, in 2007. The ISIS militant group took over the ancient city last year and razed parts of its The ruins of the 2,000-year-old Temple of Baalshamin are seen in Palmyra, Syria, in 2007. The ISIS militant group took over the ancient city last year and razed parts of its World Heritage Site. Syrian government forces recaptured the ancient city from the terror group in March 2016. Click through to see the landmarks before and after ISIS' occupation. Hide Caption 1 of 15 Photos: How ISIS ravaged Palmyra's world treasures After: A Syrian government soldier walks near what's left of the Temple of Baalshamin on Sunday, March 27. Syrian forces retook the city days before, but damage had already been done by ISIS. UNESCO says it plans to evaluate the extent of A Syrian government soldier walks near what's left of the Temple of Baalshamin on Sunday, March 27. Syrian forces retook the city days before, but damage had already been done by ISIS. UNESCO says it plans to evaluate the extent of Palmyra's damage soon. Many of the structures -- which date from the first and second centuries and marry Greco-Roman techniques with local traditions and Persian influences -- remain in place, bolstering hopes that ISIS didn't completely raze the world heritage site. Hide Caption 2 of 15 Photos: How ISIS ravaged Palmyra's world treasures Before: The Arch of Triumph in 2006. Hide Caption 3 of 15 Photos: How ISIS ravaged Palmyra's world treasures After: The Arch of Triumph on March 27. Hide Caption 4 of 15 Photos: How ISIS ravaged Palmyra's world treasures Before: The Temple of Bel in 2008. Hide Caption 5 of 15 Photos: How ISIS ravaged Palmyra's world treasures After: The Temple of Bel on March 27. Hide Caption 6 of 15 Photos: How ISIS ravaged Palmyra's world treasures Destroyed statues are seen inside the damaged Palmyra Museum on March 27. Hide Caption 7 of 15 Photos: How ISIS ravaged Palmyra's world treasures Many statues in the museum had their faces chipped off -- in keeping with strict Sharia interpretations of the depiction of human forms. Hide Caption 8 of 15 Photos: How ISIS ravaged Palmyra's world treasures Before ISIS invaded, authorities took what they could from the museum. But larger items and those fixed to walls had to be left behind. Hide Caption 9 of 15 Photos: How ISIS ravaged Palmyra's world treasures Destroyed statues inside the Palmyra Museum. Hide Caption 10 of 15 Photos: How ISIS ravaged Palmyra's world treasures The Syrian directorate-general of antiquities and monuments was positive that the condition of artifacts meant that they could be restored and their "historic value" returned, according to a translation of an article on the department's website. Hide Caption 11 of 15 Photos: How ISIS ravaged Palmyra's world treasures ISIS took over Palmyra in May. Hide Caption 12 of 15 Photos: How ISIS ravaged Palmyra's world treasures By June, ISIS began destroying historical sites. Hide Caption 13 of 15 Photos: How ISIS ravaged Palmyra's world treasures Damage inside the Palmyra Museum. Hide Caption 14 of 15 Photos: How ISIS ravaged Palmyra's world treasures Palmyra was a caravan oasis when Romans overtook it in the mid-first century. In the centuries that followed, the area "stood at the crossroads of several civilizations" with its art and architecture mixing Greek, Roman and Persian influences, according to UNESCO. Hide Caption 15 of 15

"The fighting is putting at risk one of the most significant sites in the Middle East, and its civilian population," she said.

"I reiterate my appeal for an immediate cessation of hostilities at the site," Bokova said. "I further call on the international community to do everything in its power to protect the affected civilian population and safeguard the unique cultural heritage of Palmyra.

"Finally, it is imperative that all parties respect international obligations to protect cultural heritage during conflict, by avoiding direct targeting, as well as use for military purposes."

The Islamist militants of ISIS have captured parts of Syria and Iraq in campaigns of violence and barbarity, slaughtering not only those who fight them but also conquering civilians who don't adhere to their extreme brand of Islam

As it has conquered territory, it has destroyed archaeological sites, claiming that it considers all religious shrines idolatrous.

Syria's state-run news agency, SANA, reported that heavy clashes were taking place in Tadmur's northern neighborhoods, but that security forces repelled ISIS in the rest of the city.

'Would be a loss for the entire world'

Between the first and second centuries A.D., Palmyra "stood at the crossroads of several civilizations," with its art and architecture mixing Greek, Roman and Persian influences, UNESCO says.

The city already was a caravan oasis when Romans overtook it in the mid-first century A.D. Its importance grew as a city on the trade route linking the Roman Empire to Persia, India and China, UNESCO says.

JUST WATCHED Ancient Syrian city of Palmyra threatened by ISIS Replay More Videos ... MUST WATCH Ancient Syrian city of Palmyra threatened by ISIS 02:13

British historian and novelist Tom Holland described Palmyra as "an extraordinary fusion of classical and Iranian influences intermixed with various Arab influence as well."

The destruction of Palmyra wouldn't just be a tragedy for Syria, it would be a loss for the entire world, he told CNN this month.

"This isn't just about Middle Eastern history; these are the wellsprings of the entire global culture," he said. "Mesopotamia, Iraq, Syria, this is the wellspring of global civilization. It really couldn't be higher stakes in terms of conservation."

This is not the first time ancient sites have been destroyed during Syria's brutal four-year civil war , in which ISIS is one of the belligerents.

Notable casualties include 11th century crusader castle Crac des Chevaliers; its walls were severely damaged by regime airstrikes in 2013.

JUST WATCHED Expert: ISIS target in Syria could 'never be replaced' Replay More Videos ... MUST WATCH Expert: ISIS target in Syria could 'never be replaced' 03:59

Aleppo Souk, a formerly thriving part of Syria's economic and social life, was severely damaged in a fire in 2012.

"What is distinctive and horrendous about (ISIS') mode of operation is that they are deliberately going out of their way to destroy (ancient artifacts)," Holland said.

ISIS propaganda claims the Islamist militants are destroying idols or false gods and following in the footsteps of the Prophet Mohammed, who smashed statues in Mecca.

But ISIS also sells artifacts for huge profits.

"They have networks that allow them to traffic in cultural treasures. They have made tens of millions of dollars selling artworks," said London School of Economics professor Fawaz Gerges, author of the forthcoming book, "ISIS: A Short History."