On an overcast morning in February 2015, Zvika Fayer was scuba diving off the ancient Israeli port town of Caesarea when he saw a glimmer on the sand.

He’d gone diving in the area dozens of times before, and loved it for the teeming fish and scattered remains of shipwreck cargo and pottery that he sometimes glimpsed on the ocean floor. Many of Israel's underwater archaeological zones are open to divers, and Caesarea was one of Fayer’s favourite spots.

You may also be interested in:

• An epic search for Russia’s lost gold

• The lost jewels of Bad King John

• The island with £100 million hidden

A violent storm the previous night had battered Israel’s west coast, stirring up the ocean floor and changing the underwater topography. As Fayer dove deeper to investigate the shimmering object he’d seen, he was aware of another storm on the way. As an experienced diver, he’d felt confident going out that morning. But now the winter sky was beginning to darken, the breeze changing.

As he got closer, Fayer reasoned that the gleam must have been a discarded sweet wrapper, perhaps the chocolate coins that look like pirate doubloons. But as he swept the sand away and picked the item up, he saw that he was wrong. This wasn’t a piece of foil; it was a real gold coin with Arabic script on both sides.

“I was astonished when I saw that both sides of the metal were gold,” he said.

Fayer swept more sand aside, and saw another coin, and another, and another — the bounty of a shipwreck that had been lost to time.

Less scrupulous explorers might have decided to take the treasure home with them, but Fayer and his diving companions got back in their boat and contacted Caesarea officials, who called Israeli Antiquities Authority (IAA) and told them to come quickly.

If we didn’t take the coins out of the ocean right then, we might never find them again

When they arrived, the IAA investigators were sceptical. Caesarea is an underwater archaeological site that’s littered with ancient artefacts, and they were concerned about looting.

“We got a bit of a (proverbial) cold shower,” Fayer said. “They were yelling at us, asking why we had taken the coins out of the sea. We explained about the conditions… there was another storm coming, with waves expected to be 10m high. We told them that if we didn’t take the coins out of the ocean right then, we might never find them again.”

Working with the IAA, Fayer and his friends went back underwater and helped recover more coins. Days later, they went back and retrieved hundreds more. As of now, more than 2,000 coins have been found at the site. Because the coins are 24-karat gold with a purity of upwards of 95 percent and were well preserved in the temperate Mediterranean water for about 1,000 years, they’re giving historians exciting information about a forgotten chapter in history.

Today, Caesarea, located between Tel Aviv and Haifa on the Mediterranean Sea, is best known for its towering Roman ruins. The historical centre has been restored and turned into a tourist destination with a modern restaurant and museum. There’s a golf course and residential park nearby. But when you stand in front of the crescent-shaped harbour or near the arches of the remaining aqueduct and look out over the azure water toward Cyprus, Turkey and Greece, it’s easy to imagine this place as it was centuries ago.

The first buildings in Caesarea were erected in the 4th Century BC to establish a Phoenician and Greek trading post. Then, sometime after 96BC, the city came under the rule of Egyptian queen Cleopatra. But the region was conquered by the Romans, and Caesarea – then called Stratonos Pyrgos (Straton’s Tower) – was soon handed over to Herod the Great, a regional king appointed by the Romans. He re-named the town after the infamous Roman emperor.

Under Herod, Caesarea blossomed. The king ordered the construction of break walls to form a massive deep-sea harbour, along with an aqueduct, hippodrome and 20,000-person amphitheatre for watching the rush and carnage of chariot races.

By 6AD, Caesarea was the capital of the Roman province of Judea. As such, it was also the home of the many Roman governors stationed there, including Pontius Pilate, who ruled during the time of the historical Jesus. And, when the Jewish people revolted against Roman rule between 66 and 70AD and Jerusalem was razed, Caesarea became the political and economic hub of the region. If the town looks like a backwater now, it was anything but that 2,000 years ago.

The city remained important with its history well recorded until 640AD, when it was the last city in the region to fall to Muslim invaders. After that, records are spotty. The consensus is that Caesarea faded from glory and its role on the socio-political scene, its ruins sacked and re-settled by small communities before eventually becoming home to a small fishing village in the late 1800s.

But the discovery of the coins has changed that story, according to Jakob (Koby) Sharvit, director of the IAA’s Marine Archaeology Unit. In fact, it suggests that Caesarea remained a hub of trade and commerce during the time it was under the rule of the Muslim caliphates; that it did not simply revert to a remote, rural backwater.

“Before finding the coins, we had no idea that the community in Caesarea at that time was so large or so rich," Sharvit said. “So it changed what we believed about that time.”

For items so small, the coins, called dinars, offer many clues about what the world was like at the time they were made. The dates minted on them show that they were manufactured during the reigns of Caliphs al-Hakim (996–1021AD) and his son al-Zahir (1021–1036AD) when Caesarea was part of the Islamic Fatimid Dynasty, which at that time extended around the Eastern Mediterranean.

The coins were minted in the far-flung cities of Cairo, Egypt, and the Sicilian capital of Palermo, showing that currency was circulating through a united empire. Other clues are more personal, like teeth marks that belie how ancient people would test the coins to make sure they were real gold.

And judging by their value, it’s likely that Caesarea was still a prosperous, busy city at the turn of the 11th Century.

“Those coins were a lot of money for the people who lived there,” Sharvit said.

Each coin would have been about one month’s salary for a military soldier, which means the treasure was enough to hire an army of 2,000 for a month. And, of course, the hoard was probably only lost because of an accident on one ship. There were likely many ships going in and out of the harbour that didn’t drop their money into the sea.

According to Sharvit, archaeologists aren’t sure how the coins got lost in the first place. It’s possible that a case of them slid off the deck of a ship in a storm, or that raiding pirates caused it to tip off and sink.

When you see something that old, you can feel that it's telling you a story of what used to be there before

It’s also possible that the money was being sent back to Cairo, the Fatimid capital, either as taxes or for safekeeping. The First Crusades were launched in 1095, and people in Caesarea may have been bracing for raids.

Historians will likely never know the whole story, but even a glimpse of what life was like so long ago is exhilarating, Fayer said.

"When you see something that old, you can feel that it's telling you a story of what used to be there before. That’s especially true when you find it under the sea. Most of the time, no-one had touched those things since they were lost 1,000, 1,500 or 2,000 years ago – from when they were dropped into the sea until you found them… that’s the part that’s exciting to me.”

Since finding the coins, Fayer and his diver friends have been working with the IAA to help recover other treasures both around Caesarea and elsewhere along the Mediterranean coast. They also plan to start exploring new areas, like the waters off the coastal Israeli city of Netanya where Phoenician and Roman ships left a legacy of maritime treasures in the water. He’d like to help uncover more lost pieces of history beneath the sea.

“For me, that discovery was never about the money behind it,” Fayer said. “It was about the history and what the coins said about the area and what it was like so long ago.”

EDITOR'S NOTE: A previous version of this article did not specify that it was the Jews who revolted against Roman rule between 66 and 70AD and incorrectly described Caesarea as the capital of Roman Palestine. The text has now been corrected.

Join over three million BBC Travel fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital and Travel, delivered to your inbox every Friday.