Writing in 1958 about the little Greek town that was eventually to become his home, the travel writer Patrick Leigh Fermor was satisfied to note that the Guide Bleu gave it only half a line. “It is better so,” Leigh Fermor wrote. “It is too inaccessible and there is too little to do there, fortunately, for it ever to be seriously endangered by tourism.”

His next paragraph describes the town in early evening when, waiting for a freshly caught fish to cook on a grill, he and a few fishermen sit under a mulberry tree outside a taverna and watch the sun sink over the mountains. Caiques – the wooden working boats of the Mediterranean – rock gently “with each sigh of the green transparent water … tethered a few yards above their shadows on the pebbly bottom”. One of Leigh Fermor’s typically exact (and perhaps exacting) images follows when he describes the sea lapping over a flat rock “with just enough impetus to net the surface with a frail white reticulation of foam which slid softly away and dissolved while a new one formed”.

Some of these things still exist. The Mediterranean is clear and green and blue, and on a calm day it will rise and fall against the rocks as Leigh Fermor describes. The sun goes down as he depicts it. There is even a caique or two; and, of course, tavernas – more tavernas than ever. But in most other ways the township of Kardamyli in the Peloponnese is utterly changed. Charter flights land at the little airport in the regional capital, Kalamata, and from there a twisting, expensively engineered road takes taxis, hire cars and air-conditioned coaches over the mountains to a resort that has nice hotels, trinket shops and olive-oil boutiques, as well as pretty restaurants with tea-lights on their tables that look down on the sea. The usual story: Kardamyli now makes most of its money from tourism. It wasn’t as immune to tourism as Leigh Fermor imagined or wanted it to be, and the writer himself is partly to blame.

The tumbledown factory loomed on the shore, a picturesque ruin in brick and concrete where fig trees grew

First, he published an account of his travels in the southern Peloponnese, the peninsula known as the Mani, which was then not much visited, and invested it with the beauty and mystery of a place and people that the 20th century had passed by. Then, six years later, in 1964, he bought a plot of land there – in a bay to the south of Kardamyli – and built a beautiful villa that he lived in almost to the last day of his life, in June 2011. Today his books are available in at least three languages in the local bookshop. People go there because of him – to experience similar sights and sensations to those he saw and felt, even though they understand this can never be completely accomplished, the world having moved on.

But was it ever quite as he described it in the first place? Leigh Fermor’s view of the Mani was essentially romantic: there are few better describers of landscape, but it’s a landscape with omissions. His first sight of Kardamyli is of an enchanting, castellated hamlet at the sea’s edge, where towers, turrets and cupolas rise above houses built of golden stone. “It was unlike any village I had seen in Greece,” Leigh Fermor writes in a page-long depiction that somehow ignores the village’s tallest manmade attribute: the factory chimney of the old olive-oil works. This is difficult to miss. Look down on Kardamyli from almost any vantage point and there it stands, its bricks pale against a background of blue sea and rather more noticeable than the towers and the turrets lying further inland among the cypresses and the olive groves.

The towers date from the age of banditry, feuding clans and resistance to the Ottoman empire. The chimney has cleaner and more peaceable origins. This month I lived next door to it for 10 days in a fine little hotel, and swam morning and afternoon from a ladder bolted to the rocks. The tumbledown factory loomed on the shore behind, a picturesque ruin in brick and concrete where fig trees grew and rusting pipes sprang from the wall at odd angles. A high fence surrounded it, with warnings to keep out.

Olive oil had once been made here – not virgin, cold-pressed or estate bottled, but the roughest kind, which goes into soap. Some accounts online suggest it was owned by the Palmolive company (and when I read this I understood, for the first time, how that familiar name had come about); others say a local family were the proprietors. It used olives – and the residues left from edible oil production – from as far away as Crete, shipped to a concrete pier nearby whose size was inexplicable unless you knew its original purpose. It was said to have employed 150 workers, with steam machinery that, as well as operating its crushers, had the spare capacity to supply the village with its first electricity. Opened in 1932, it closed in either 1958 or 1975 – local memories differed – when new techniques of oil production made it redundant. Since then, a dispute among the site’s three or four owners had prevented demolition or development.

I liked the chimney; three stepped rings of brick, progressively larger in diameter, gave its top a decorative flourish. But then, I’ve always been fascinated by factory chimneys of all kinds, for reasons that I’ve never really examined, the most important probably being that I spent some of my childhood among them: the great smoking verticals of the Lancashire plain, formerly beloved of geography textbooks as the illustrations to the chapter on the textile industry. To find them situated outside what might be considered their natural homelands – the old industrial towns of northern Europe and North America – is always a surprise. They look solitary, like isolated monuments to a faraway and not properly understood revolution. One still standing on the coast of Argyll marks the site of a Victorian factory that made acetic acid from the oak and birch wood. Another on the Ionian island of Paxos served the same kind of mill as Kardamyli’s.

My Amsterdam is being un-created by mass tourism | Joost de Vries Read more

Smoke was most probably still drifting from the Kardamyli chimney when Leigh Fermor reached here in the mid-1950s, but he can hardly be blamed for omitting it from his picture. Like many travellers in our age, he had a distaste for modernity. (He hated radios, for instance, and was relieved that the Mani had so few of them. “Rabid wirelesses should be hunted out and muzzled or shot down like mad dogs.”) He travelled to reach some agreeable form of the past, which has been a motive for the holidaymaker since the days of the Grand Tour.

On an afternoon last week in Kardamyli, I climbed up the ladder from the sea to find three or four men inside the factory fence inspecting the ruins. One wore a pith helmet and carried a theodolite. Another unpacked a drone from its box and directed its flight to the chimney, which it hovered above rather threateningly. It looked as though change was in the offing. I’d known of the chimney for less than a week – and, really, what was it to me? But already I felt a slight alarm that it too might pass, just like the fishermen who watched the sunset with Leigh Fermor from underneath a mulberry tree.

• Ian Jack is a Guardian columnist