When the geographer Muhammad Ibn Ahmad Shams al-Din al-Muqaddasi visited the city of Ramla on the road between Cairo and Damascus in the 10th Century, he described an urban paradise that rivalled his hometown of Jerusalem.

“It is a fine city, and well built; its water is good and plentiful; its fruits are abundant,” al-Muqaddasi wrote in his famous travelogue. “Commerce here is prosperous, and the markets excellent.”

The city, established as the new provincial capital of Palestine in 715, shortly after the area came under Muslim rule, boasted grand mosques, administrative buildings and mansions with gardens, mosaics and fountains. But most of these early Islamic buildings were destroyed in a series of earthquakes in the 11th Century, and the city, although partially rebuilt, never regained its earlier prestige.

“You can see almost nothing of this today,” said Gideon Avni, head of the archaeological division of the Israeli Antiquities Authority.

Today, Ramla is a small Israeli city populated by both Jews and Arabs near the country’s Ben Gurion Airport, about 25km southeast of Tel Aviv, that’s working to overcome its reputation for crime and drug trafficking.

But a hidden, underground world of ornate water cisterns offers a glimpse into the city’s enchanted past.

The main entrance to the subterranean world, where archaeologists have found four elaborate cisterns, as well as further evidence of how an aqueduct supplied these pools with water, is found just off a main street in downtown Ramla.

Inside a small building, surrounded by apartment blocks and a park, a stone staircase leads down to an expansive, water-filled cistern. Its ceiling is supported by numerous pointed arches, which have given it the name ‘Pool of the Arches’. Small openings in the ceiling, as well as modern red and green electric lightbulbs, pierce the darkness, revealing moss-covered walls and an inscription carved in Arabic that says the cistern was commissioned in 789 by Caliph Harun al Rashid, who ruled the area from Baghdad.

A small dock offers rowboats for exploring the 1,228-year-old cistern. As my father paddled our boat around the reservoir, dodging the bases of the numerous arches that support the roof, I looked up at the ceiling – 9m above – which seemed more fit for a cathedral or palace than for water storage. The only sounds were the paddles rustling through the water and the beeping of my camera as I took photos. The damp air smelled musty, adding to the feeling of slipping back in time.

Archaeologists believe this cistern is the first known use in the Arab world of the pointed arch, which eventually became a defining characteristic of Islamic architecture, seen in well-known monuments like the Alhambra palace in Granada, Spain.

These arches “freed architects both to raise their structures and also leave them spacious and airy,” said Katia Cytryn-Silverman, a senior lecturer in archaeology and Middle Eastern Studies at the Hebrew University of Jerusalem.

In addition to giving the cistern a majestic feel, the arch also represents the cultural fusion that defined Ramla and the early Islamic empire as a whole. Inspiration for the pointed arch was taken from the ancient Persian Empire, while the idea of an underground cistern came from the system of Greek-built water cisterns with curved arches under Constantinople, Avni explained.

“This is one of the meeting points between East and West,” he said. In fact, this cistern typifies Ramla as an important crossroads for culture and commerce at its height.

When the Roman and Byzantine empires ruled the ancient province of Palestine, its capital was in Caesarea, a harbour city built by Herod the Great on the Mediterranean coast. But when the area came under the expanding Muslim empire of the Umayyad dynasty in the early 8th Century, a new regional capital was established in Ramla, which was an undeveloped area with sandy soil about 13km from the coast. The location was chosen because it was along the main road to Damascus, the seat of the Umayyad caliphate, and perhaps also to spite the nearby Christian city of Ludd, which refused to grant land to the new Muslim empire, Cytryn-Silverman said.

Whether it was erected for revenge or for objective motives, the new capital did enjoy a strategic position"

“Whether it was erected for revenge or for objective motives, the new capital did enjoy a strategic position once occupied by Ludd,” Cytryn-Silverman explained.

Over the years, archaeologists in Ramla have discovered hundreds of ancient coins minted all over the Islamic world, from Algeria to Uzbekistan, demonstrating how significant of a crossroads it once was. Other evidence points to a diverse population of Muslims, Christians and Jews.

When the Abbasid dynasty wrestled control of the Islamic empire away from the Umayyads in 750, they, too, continued to develop Ramla. The Pool of the Arches, built in 789, was their most ambitious project – measuring about 400sqm in size and fed by an aqueduct, the remains of which can be seen today outside the city. The openings in the cistern’s stone ceiling allowed residents to draw out water with buckets.

But despite its solid construction, the Pool of the Arches fell out of use after only 150 years, when an earthquake damaged the aqueduct that supplied it with water and a rebuilding project left the cistern outside of the new city limits. It fell into disrepair, and for centuries, it simply became a place that collected rainwater and where shepherds would stop to draw water for their herd – local residents called it the ‘Pool of the Goats’. In 1862, French archaeologist Melchior De Vogue discovered the dedicatory inscription and the sheer size of the abandoned cistern, which had by then become filled with soil.

But it still remained mostly hidden until the 1960s, when the municipality of Ramla cleaned it out and began letting visitors paddle around it in rowboats. The water source for the pool today is an underground aquifer and leakage from the city’s modern water system, rather than the ancient aqueduct. At least three other similar cisterns have been discovered nearby over the years, but they are not open to the public due to fears about their safety and stability.

Unlike the foundations of other early Islamic buildings found underground in Ramla by archaeologists, which often just look like piles of rocks to the untrained eye, the Pool of the Arches allows visitors to relive the city’s glorious past. Caliph Harun al Rashid, who commissioned the pool, ruled during the Golden Age of Islam, when economics, science and culture flourished from the western edge of the empire in Spain all the way into Asia.

The significance and accomplishments of this period became clear as our boat glided through the dark water under the tall arches, still standing after more than 1,200 years.

“This is almost all that is left,” Avni said. “And you must go underground to see it.”

Join over three million BBC Travel fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.