Perched on a hill overlooking a valley of cherry trees and vines, the tiny medieval village of Lacoste is a fantasy of tranquil, peasant life. Peter Mayle wrote his bestselling A Year in Provence from a ramshackle house nearby; Tom Stoppard settled in a cottage near the belfry; and John Malkovich likes to practise his French at local markets. Only the imposing, half-ruined castle that once belonged to the Marquis de Sade hints at a darker truth of the feudal rulers who for centuries lorded it over the villagers in this south-eastern corner of France.

But de Sade's chateau, said to have inspired the gothic settings for his novels of sexual perversion, is at the centre of a different outrage: its new, rich owner is accused by villagers of trying to take over as a self-styled feudal lord.

Pierre Cardin, the millionaire Paris fashion designer and businessman who has spent millions restoring the castle, is trying to turn the village into a "St Tropez of culture". After establishing his own music festival, he has started buying up scores of cottages and buildings in the village of 430 people.

The ageing couturier says he wants to "leave his mark" by turning Lacoste into a refuge for world artists, complete with luxury hotels, a top restaurant, a de Sade cafe and a piano bar. But a growing group of villagers warn that his plans are ruining this Provençal community.

Lacoste, once a Protestant and later a communist stronghold, is no stranger to rebellion. Campaigners have already gone to a tribunal to stop Cardin building a Greek amphitheatre in the local quarry. But the row escalated this week after Cardin insinuated in a TV interview that his village opponents were bumpkins who didn't understand his great vision. They now call him an egotistical "invader" bent on killing village life.

On the tiny square at the top of winding cobbled streets, 85-year-old Cardin steps out of his black BMW in designer glasses and a tweed waistcoat, on his weekly inspection of his rural empire. Rue de Basse, the tiny, main village street now hosts 12 building sites bearing Cardin's name. He has bought more than 20 houses and owns almost the whole quaint and winding street. The newspaper shop has his name over the door, he has built two galleries, a boulangerie, a boutique and plans a restaurant and two hotels. He owns two castles, employs dozens of people on his projects, and a van emblazoned with "Pierre Cardin perfumes" can be seen regularly climbing the hill.

"I'm happy here," he told the Guardian. "I just want to make the village beautiful". He declines to comment on the outrage caused when he recently likened himself to a "seigneur" and said that while other rich people gamble or collect stamps "I collect houses".

In the Café de France, a group of angry villagers, including artists and teachers, warned of a "predator". They said Cardin sometimes paid double the price for old stone buildings, once offering €1m to a couple for their house worth €300,000. "This village is fragile, it has an ageing population, our school has around 30 children; we just want to ensure young couples can afford to live here and keep the place alive," said Bruno Pierret, a jurist who stood with a group of leftwing candidates on an anti-Pierre Cardin ticket at recent local elections. His group did not get elected but they continue to lobby the mayor to closely monitor Cardin's moves. At a public meeting this week, more than 20 villagers decided to launch a fresh petition and letter-writing campaign. Eliane Ferres, a retired teacher whose father was the last communist mayor of Lacoste said: "He treats us like "natives" and has a complete disdain for people not of his milieu. He has no right to say he saved the village when in fact he's sucking out its soul."

Most approve of Cardin's much-needed work to restore de Sade's castle. But the international artists who settled here from the 1950s and 1960s laugh off the idea that Cardin alone is bringing in culture. "Artists and writers have long settled in Lacoste," said Inge Boesken Kanold, a German painter. Previous residents include Andre Breton and Max Ernst.

Genevieve Recubert, a teacher, said: "I'm scared he will change the face of the village forever." Yves Ronchi, a local wine-maker, who founded the Association for the Harmonious Development of Lacoste to monitor Cardin's expansion, warned: "Since the Middle Ages, this has been a feudal place where villagers were not treated as equal; that has produced a local mentality of deference, of bowing down to landowners. That's what's happening now."

One of Cardin's right-hand men running the Lacoste project said: "He sees himself as a patron, he gives without asking for anything in return. He doesn't look to buy. People say: 'Pierre Cardin, I would like to sell my house'. He has brought a dying village back to life."

Cardin seems hurt by the outrage in Lacoste, but he is also an ageing man in a hurry to realise his dream.