For centuries the Kaaba, the black cube in the center of Mecca, Saudi Arabia that is Islam’s holiest point, has been encircled by arched porticos erected some three centuries ago by the Ottomans, above dozens of carved marble columns dating back to the 8th century. But earlier this month, any vestiges of the portico and columns were reduced to rubble, cleared to make way for the Saudi government’s expansion of Mecca’s Grand Mosque.

The $21 billion project, launched in 2011, is designed to meet the challenges of accommodating the millions of pilgrims who visit Mecca and Medina every year. Around 2 million currently visit during Hajj alone, the annual pilgrimage that happens during the last month of the Islamic calendar. But activists charge that the recent destructions are part of a much wider government campaign to rub out historical and religious sites across the Kingdom.

Over the last few years, mosques and key sites dating from the time of Muhammad have been knocked down or destroyed, as have Ottoman-era mansions, ancient wells and stone bridges. Over 98% of the Kingdom’s historical and religious sites have been destroyed since 1985, estimates the Islamic Heritage Research Foundation in London. “It’s as if they wanted to wipe out history,” says Ali Al-Ahmed, of the Institute for Gulf Affairs in Washington, D.C.

Though the Saudi rulers have a long history of destroying historical sites, activists say the pace and range of destruction has recently increased. A few months ago, the house of Hamza, the Prophet Muhammad’s uncle, was flattened to make way for a Meccan hotel, according to Irfan Al Alawi, executive director of the Islamic Heritage Research Foundation. There have even been rumored threats to Muhammad’s tomb in Medina and his birthplace in Mecca.

A 61-page report, published recently in Saudi Arabia’s Journal of the Royal Presidency, suggested separating the Prophet’s tomb from Medina’s mosque, a task “that would amount to its destruction,” Alawi says. “You can’t move it without destroying it.” Moreover, he alleges, plans for a new palace for King Abdullah threaten the library atop the site traditionally identified as the birthplace of Muhammad. Even now, signs in four languages warn visitors that there is no proof that the Prophet Muhammad was born there, “so it is forbidden to make this place specific for praying, supplicating or get [sic] blessing.”

Wahhabism, the prevailing Saudi strain of Islam, frowns on visits to shrines, tombs or religio-historical sites, on grounds that they might lead to Islam’s gravest sin: worshipping anyone other than God. In recent years, the twin forks of Wahhabi doctrine and urban development have speared most physical reminders of Islamic history in the heart of Mecca. The house of the Prophet’s first wife, Khadijah has made way for public toilets. A Hilton hotel stands on the site of the house of Islam’s first caliph, Abu Bakr. Famously, the Kaaba now stands in the shade of one of the world’s tallest buildings, the Mecca Royal Clock Tower, part of a complex built by the Bin Laden Group, boasting a 5-story shopping mall, luxury hotels and a parking garage.

Saudi officials did not respond to interview requests, but in the past, they have said that the expansion project is necessary to cater to the ever-growing number of pilgrims to Saudi Arabia, a number forecast to reach 17 million by 2025. When it’s done, the expansion of the mataf, the area where the faithful circumambulate around the Kaaba, will treble its capacity, to 150,000 people; the Great Mosque will be able to hold 2.5 million.

Amir Pasic, of IRCICA, the culture organization of the 56-nation Organization of Islamic Conference, points out that the logistics for Hajj dwarf those required for a World Cup or Olympics. “Every time has the right to make changes on the existing urban set-up,” he said. “Every generation tries to develop something. The Kaaba is what’s important.”

If Mecca’s new skyline is impossible to ignore, what with 48 searchlights beaming from the top of the Clock Tower, other changes to the landscape are more insidious. “Everyone’s focused on [the two mosque expansion projects], but people are not focusing on what we’re losing in the meantime,” says Saudi activist, poet and photographer Nimah Ismail Nawwab. After blue markings appear on sites mentioned in Islamic histories, says Nawwab, then the bulldozers come–often in the dead of night. “Everything happens at night,” she told TIME by phone from Saudi Arabia. “By the next day in the morning, the monument is gone.”

It’s not just in Mecca, either. Over a year ago, the split in Mount Uhud, north of Medina, where Muhammad was said to have been carried after being wounded in the famous Battle of Uhud was filled with concrete. A fence went up at the base of the mountain, warning would-be visitors that it was just a mountain, like any other. Six small mosques in Medina where Muhammad is believed to have prayed have been locked. The seventh, belonging to Islam’s first caliph Abu Bakr, has been razed to make way for an ATM. Nawwab, along with a small group of historians and activists, has tried to raise awareness by photographing sites and starting a Twitter campaign, but says “it’s a losing battle, despite the fact that what’s being lost is not just Muslim history, but human history.”

When the Taliban blew up the Bamiyan Buddhas in Afghanistan in 2001, they were met with international condemnation. The response to the demolition activity in the Kingdom, by contrast, has been decidedly muted. “When it comes to Mecca, as far as we are concerned it’s a Saudi question,” says Roni Amelan, a spokesman for UNESCO, the United Nation’s cultural body. The Saudi government has never submitted Mecca for inclusion on the list of World Heritage Sites. As UNESCO’s mandate requires a respect for the sovereignty of individual countries, “we don’t have a legal basis to stake a position regarding it,” adds Amelan.

Muslim governments, perhaps mindful of the power of the Saudis to cut their quotas for how many pilgrims can attend Hajj, have been strikingly silent on the issue. The Organization of the Islamic Conference has also been noticeably quiet on the destruction of the Saudi campaign. One exception has been Turkey, whose Ottoman heritage has also long been under threat. In September, Mehmet Gormez, head of the Dinayet, Turkey’s Directorate of Religious Affairs, told journalists that he told Saudi’s minister of Hajj that the skyscraper overshadowing the Kaaba “destroys history,” the Turkish newspaper Hurriyet Daily News reported. “History is being destroyed in the Holy Land each day,” he added.

For pilgrims old enough to remember the dangerous crush of crowds in the 1980s, the spate of new development may be welcome, offering a chance for comfort on their spiritual journey. For other Muslims, like Ziauddin Sardar, author of the recent Mecca: The Sacred City, the vigor of the Saudi campaign springs from financial jitters. “The Saudis know the oil is going to run out,” he said. “Hajj is already their second major source of income, after oil. They look at Dubai, and Qatar, and ask ‘what are we going to do?’ And they say, ‘We have Hajj, and we’re going to exploit it to the max.'”

Carla Power is the author of If the Oceans Were Ink: A Journey to the Heart of the Quran (Henry Holt: April, 2015)

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.