I caught my first glimpse of Spinalonga four years ago from a steep hilltop overlooking the sleepy village of Plaka in north-eastern Crete. The diminutive 8.5-hectare islet in the Gulf of Mirabello stood completely in shadow, dark and foreboding as if blackened by coal, while the butterscotch-hued Spinalonga Peninsula that curls around the island like a sleeping cat’s tail basked in the golden light of the setting sun.

A stone’s throw from Plaka, the arid, rocky islet once served as a military stronghold during Venetian (and later Ottoman) rule; a medieval citadel stands testament to that chapter in the island’s history. In 1904, after the Cretans evicted the Turks from Spinalonga, the islet was transformed into a leper colony, where, by 1913, after Crete became part of Greece, anyone afflicted with the disease was sent. At its peak, the colony comprised nearly 400 inhabitants.

You may also be interested in:

• Thailand’s nefarious island paradise

• The mystery behind Greece’s temples

• The isle you can only visit once a year

Leprosy, which causes disfiguring skin sores and debilitating nerve damage, has long carried with it negative social stigmas. Those burdened with leprosy were shunned, stigmatised and mistreated by their families, communities and even medical professionals. In fact, stigmatisation was often so extreme that since ancient times leprosy has been called the ‘death before death’.

Since ancient times leprosy has been called the ‘death before death’

Once diagnosed, these victims had their property and financial assets seized, their citizenship rights revoked and their identity wiped clean. They were then deported to Spinalonga, where they never received treatment for their disease. The sole doctor assigned to the island only made the trip from Plaka if someone was struck by another illness. What’s more, even though treatment for leprosy was found as early as the 1940s, the Greek state kept the colony operational until 1957. Only after a British expert visited the island and compiled a damning report denouncing the island doctor and the state for failing to provide proper medical treatment and housing did the government officially close Spinalonga.

I wasn’t able to visit the island that day four years ago, but since reading Victoria Hislop’s bestselling novel The Island, a melodramatic telling of family secrets, betrayals and star-crossed love affairs set in the confines of the leper colony, I’ve returned to learn what life was truly like for those exiled on Spinalonga.

Viewed from across the teal water of the Gulf of Mirabello on a windy summer afternoon, white speedboats zipping past, Spinalonga paints a pretty picture. But as our small ferry from Plaka approached the islet, the citadel loomed above us like a dark cloud.

I was accompanied by Maurice Born, ethnologist and co-author and translator of Vies et morts d’un Crétois lépreux (Lives and deaths of a Cretan leper), which he wrote with Epaminondas Remoundakis, a leprosy survivor who advocated for improved living conditions and fair treatment of those living on Spinalonga.

The story of Spinalonga is the story of a massive lie

“You see,” Born said as we passed through a multi-arched tunnel in the fortress walls known as Dante’s Gate, “the story of Spinalonga is the story of a massive lie.”

For decades after the 1957 closure of the leper colony, little was known about the island. The government, anxious to erase any trace of the colony’s existence, burned all its files. And the surviving lepers refused to speak of their experiences. For years it was as if Spinalonga had never existed.

But Hislop’s 2005 novel – which spawned a hit television series – changed all that. Suddenly, people were talking; everyone was an expert. The government, thrilled at the author’s rose-tinted portrayal of the colony, let them talk, and a romanticised, erroneous story of life in the leper colony was born.

We emerged onto a sunny village street lined with the crumbling remains of Venetian and Turkish homes. Born paused to point out a frontless, roofless stone ruin that had once housed a bistro run by lepers. “The state, seeking to erase the stain on their reputation, wanted to destroy all evidence of the leper colony. But then, in the 1980s, they realised that the tourists were coming with the specific purpose of visiting the leper colony,” he explained, laughing at the irony.

Another arch opened onto what was the commercial street, a shaded strip with shops (something the lepers were only authorised to have as of the 1930s), a cafe and a small school. One side of the strip has been restored for tourist purposes, with storefronts now sporting wooden shutters in a rainbow of colours. Not far away stands a cavernous stone building that still houses the incinerator used to burn infected clothing.

As we walked through the abandoned village, Born continued to reveal details of life in the colony. Before the 1930s, Spinalongians “lived in a frantic state of selfishness, thinking only of survival,” he explained. “No-one looked after one another, the priest had difficulty finding people to help bury the dead.”

It wasn’t until Remoundakis arrived and formed the Brotherhood of the Sick of Spinalonga, a society dedicated to improving conditions on the island, that conditions began to improve. The group lobbied the Greek government for the right to marry and to operate businesses.

No-one wished to see themselves

“Until the Brotherhood, all that counted was food, gambling and raki (a strong grape-based spirit popular in Crete),” Born said. But the Brotherhood worked to establish order and a better quality of life on the island. Sometimes concerts were held by musically-talented residents. Someone donated a record player, which was kept at one of the cafes, and patrons would listen to music.

One of the Brotherhood’s most important rules was a ban on mirrors; no-one wished to see themselves. But it was impossible not to observe the ravages of the disease in the other residents.

“They sought solitude in order to escape the face of the other,” Born said.

And so, in 1938, colony residents received permission from the state to dynamite part of the medieval fortress wall and clear a path around the island’s perimeter, navigable even by those who had been handicapped by their illness. The new path provided those banished to the island some semblance of liberation.

We made our way past the pile of boulders left by the demolition. The suffocating feeling I felt in the village vanished as we followed the trail. Strong winds whipped off the Aegean Sea and across the path like a prisoner freed from handcuffs. I took in the spectacular view of the Bay of Mirabello and inhaled the salty scent of the sea.

We passed the lonely Church of St George, built centuries ago by the Venetians, and arrived at a small cemetery. “When tourists began coming to the island in the 1980s, many of the visitors would desecrate the cemetery,” Born said. In 2013, the bones of the deceased were placed in a proper ossuary next to the cemetery and covered with new concrete plaques.

The island was silent save for the rustling of the wind moving through the grass and the faint whiz of passing motorboats. At the cemetery’s entrance, a small plaque urges respect for the poor souls buried on that rocky hillside overlooking the sea and the mountains of Crete.

It has taken several decades for their story to be told, but perhaps those who never escaped Spinalonga have found peace at last.

Join over three million BBC Travel fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital and Travel, delivered to your inbox every Friday.