The men came in the middle of the night and painted red crosses on the houses chosen for demolition. In the morning the people who had lived and traded in the ruins of the old Hampi bazaar stood by helplessly as the bulldozers moved in. The past, they were to discover, had come back to haunt them.

Hampi is India's Pompeii. Once home to half a million people, it was sacked in 1565 by the armies of the Bahamani sultanates. For hundreds of years, the City of Victory lay abandoned until it was rediscovered by the British in the 19th century. Now it is a place of sprawling beauty, a world heritage site of 2,000 monuments scattered across a landscape of enormous granite boulders, pulling in nearly half a million visitors a year from around the world.

But of the people who helped transform it from an overgrown ruin, who made it a living monument rather than a museum, there is virtually no sign. They have been swept away, ordered out by conservation authorities determined to restore the site to the way it looked in its medieval heyday.

Many had been there for decades, setting up home in the small stone pavilions, known as mandapas, which line the bazaar, catering for the needs of the tourists and pilgrims. But in April the Karnataka high court threw out their last legal challenge and backed the Hampi World Heritage Area Management Authority and the Archaeological Survey of India in their plans to remove all modern traces from the bazaar.

Credit: Observer graphics

Two weeks ago, most of the 326 residents bowed to the inevitable and moved out. Those with some savings have rented properties in nearby villages, others are staying with relatives. The rest, those who have lost everything and have nowhere to go, have been housed in rows of small huts built from bamboo and plastic sheeting on a patch of land outside the site's perimeter fence.

It is late afternoon, yet the sun still beats down relentlessly on the dirt street between the huts. An elderly widow squats in the dust behind a basket of peanuts. Kenchamma used to sell the nuts in the bazaar, in front of the home she made in one of the pavilions when she arrived with her husband some 40 years ago. She only made about 50 rupees (60p) a day, but it was enough.

She clutches her knees and says she is too frail to walk to the bazaar, so she sits outside her hut every day and waits for the tourists to come to her, just as they did when she was in the bazaar. But they don't come here, and they never will. The only people who walk past are her fellow evacuees. Today she has made five rupees; she is bewildered, beaten. "I am ready to beg now because I can't earn the money to eat," she says. "My husband died eight years ago and I have no children. There is no one to support me."

A few doors down, 20-year-old Kokila nurses her young son. Her daughter is asleep on the floor of the hut. Kokila was born in the bazaar; her family had been there 35 years.

She sold coconuts to the visitors; now she shares this hut with her mother, while her husband is away searching for work. She has no idea what they will do now and she is scared in this place, afraid of the wild animals and the local men who, after a few drinks, gravitate towards where they know there are vulnerable women.

"It is very hot here and at night there are animals coming, a lot of snakes and sloth bears are here, too. Yesterday I saw a leopard. Many people sleep outside because there are no fans and it is too hot inside, but we are scared of rapists – men coming here for sex. The walls are not strong and we are living in fear."

The traders relied on the tourists for their livelihoods; without access to them, they are lost. Around the corner Naga Raj sits amid half-empty sacks of food, brought to the camp from his family's restaurant when the orders came to get out. He is 26, a young man in despair, his future in tatters.

"The government has stolen the happiness from the people," he says.

To soften the blow, the authorities offered each family a small plot of land about 4km away in Kaddirampur village and 130,000 rupees (about £1,500) to build a new home, but the money has yet to materialise.

The spot allocated to those evicted is a barren patch of scrubby grassland, scattered with boulders. There is no reason for tourists ever to go there. Known as Gori, it is a former burial ground. Three graves straddle two building plots. Nearby are a couple of centuries-old Muslim mausoleums.

"They want us to go to this new place, but that is next to a monument, too, and who knows, in two years they will tell us to leave there too," says Naga. "All our lives we will be moving, moving until there is nowhere for us to go."

From first thing in the morning to late into the evening, a steady stream of tourists and worshippers makes its way along the bazaar towards the Virupaksha temple, which towers majestically over the site.

Heavy steel poles keep the visitors away from the wrecked buildings on either side of the half-mile long main street. Many have had their front walls torn away. In one, a barber's chair sits empty. The sign on a partly demolished hotel declares: "Customers is our god."

At the far end of the street, the finishing touches are being put to a sound-and- light show, another part of the authorities' ambitious plans.

Here and there, a few of the traders have chanced their luck and slipped back into the bazaar, setting out their meagre wares on plastic sheets laid on the ground in front of their former homes, keeping an eye out for the police, who will chase them off if they catch them.

Galappa, 34, sits behind a small pile of mangoes. Behind him, a red cross has been daubed above the doorway of the blue-painted pavilion that was home to him, his wife and their two children.

"My father and my grandfather lived here, but one morning we woke up and found they had painted these crosses on our houses and there were notices telling us we had to leave," he says. "Now if the police come they will slap me and beat me. They say it is the government rules."

The authorities make sympathetic noises, but are adamant that this is the right thing to do. Suresh Varadaraj, a conservationist with the archaeological survey, explains that once they have removed the modern additions, the street will be excavated and restored to its original state, though the shops of the bazaar will be empty.

"We cannot bring back the people from the 15th century, so instead we are going to preserve it," he says.

"It is a protected monument and the government of India says it is nationally important. These people were all encroaching. In the interests of the monument, in the interests of the country, everyone should have an interest in our heritage, so based on that it is the correct thing to do. We only shifted the people who were encroaching. This is our heritage."

Unesco, which grants world heritage site status, said it had asked the local authorities to remove unauthorised buildings but had not requested the eviction of the residents.

It insisted that its rules required the involvement of the local community in the management of the site and it said it had written to the authorities asking them to help the former residents.

"In line with the committee's policy, Unesco, aware of the problem in Hampi, has written to the authorities asking for dialogue with the communities concerned and for their involvement in the safeguarding of the site."

A spokesman said: "We have asked the government of India to engage in dialogue with the people concerned and find a suitable alternative solution for them."