Nicholas Ngonyama gazes across the valley and his eye settles on a palatial cluster of sand-coloured buildings whose thatched roofs glow in the autumn sunshine. "I'm not happy," mutters the homeless, jobless man. "The country is not happy. Too much money was spent on one man's home. That money could have been spent improving the lives of the people. It feels like he is spitting in our face."

President Jacob Zuma's personal Xanadu, complete with stately pleasure-dome, has imposed itself on the landscape of one of South Africa's poorest areas, Nkandla, in KwaZulu-Natal. It covers the equivalent of eight and a half football pitches and has swallowed 246m rand (£13.7m) of taxpayers' money. "Nkandlagate" has become the defining scandal of Zuma's five-year reign and left him fighting for his political life in this week's elections.

"Chances are slim that the majority of people here will vote for him," says Ngonyama, 46, whose children live with his parents since he lost his home to a fire in 2009. His home is a shack without water and he looks with disbelief on Zuma's folly. "We don't have houses. We can't even begin to explain the reasoning behind the expenditure of such an amount of money for one man."

The Nkandla affair is seen by many as the ultimate symbol of an African National Congress that is rotten to the core. Abusing public funds to build private mansions was once the preserve of kleptocrats elsewhere in Africa, such as Mobutu Sese Seko in Congo or Jean-Bédel Bokassa in the Central African Republic. South Africa, under founding president Nelson Mandela, believed it was different.

But the spiralling costs at what was once billed as Zuma's Camp David prompted rival politicians to lay criminal charges and call for his impeachment. One commentator described it as "worse than Watergate". The president has repeatedly insisted that he paid for the buildings from his own pocket, but South Africa's public protector accused him of unethical conduct and told him to repay the costs of a swimming pool, amphitheatre, visitor centre, cattle enclosure and chicken coop, all built as part of a state-funded upgrade. So far there is no sign of him doing so.

The sprawling homestead is a jarring sight in one of the world's most unequal societies. It was described as "opulence on a grand scale" by the public protector, who remarked that it "leaves one with the impression of excessive and unconscionable 'Rolls-Royce' security constituting an island in a sea of poverty and paucity of public infrastructure".

Allister Sparks, a columnist on Business Day, wrote: "Nkandlagate is about personal greed and moral shamelessness. It is about looting public money so that one man and his family can live in extravagant opulence for the rest of their lives – amid some of his people's most abject poverty."

Indeed, since I last visited this remote village five years ago on the eve of Zuma's election, precious little has changed: donkeys and cows graze by the roadside, boys in ragged T-shirts wield sticks, and modest mud-brick dot the hills. Precious little, that is, except the luxury residence with its helipad, bulletproof windows and three-metre-high military-grade security fence, which looked like a construction site in 2009 but is now fit for a king.

Back then, Zuma was regarded with hope: an impoverished Zulu herd boy and former inmate of Robben Island, he had risen to become the most powerful man in Africa. Singing, dancing and showing the common touch, he appeared to be a breath of fresh air after the aloof, donnish, pipe-smoking Thabo Mbeki. To critics, the Nkandla estate therefore represents a betrayal of the people all the more painful as they contemplate how the millions could have been spent for general benefit.

Of 114,416 people living in the area, about 40% are unemployed. Only 10,000 households have electricity, 7,000 have no access to piped water and 12,000 rely on pit toilets. On Friday, watched by guards in red T-shirts with rifles, residents gathered for a meeting in a white marquee on a hill overlooking Zuma's homestead.

Arthur Thamsanqa Ntuli, mayor of Nkandla, thought about 90% of those present were jobless. "We also have a lot of people who do not get water," he said. "Even when there are pipes, sometimes people go a month without water. And we still have large areas where there is no electricity. The people of Nkandla, 20 years after the beginning of democracy, are still left behind in terms of the provision of services required.

"Where there is any money available, I wish it could be used to provide water and electricity, improve education by building libraries and laboratories, provide orphans with bursaries and create job opportunities for Nkandla people."

Ntuli, a member of Mangosuthu Buthelezi's Inkatha Freedom Party (IFP), said the ANC had lost a local byelection after the Nkandlagate scandal broke. "The fact that his municipality is led by the IFP is evidence that the majority in Nkandla have decided to turn their backs on the president and the ANC. The way people vote in national elections is often different, but we hope the ANC won't get the same as in 2009."

Some fear to speak out against Zuma in a province with a long history of violence; according to crime researcher David Bruce, there have been more than 100 assassinations here in the past decade. Sthandiwe Hlongwane, who lives a short walk from the estate, says: "I cannot comment on the situation because that could be risky for me. It could put my life in danger.

"There are people supporting Julius Malema [leader of the Economic Freedom Fighters party] but they aren't coming out. They say the only way to do it is go to the polls. They are scared to come out in public."

Hlongwane was the beneficiary of a publicity stunt in January when Malema and his colleagues built her a house. Thirty ANC supporters were arrested after throwing stones and water bottles at their rivals when it was handed over. Although she still lacks running water and uses an outdoor toilet, she is grateful for the upgrade.

"It's made a big difference," says Hlongwane, 31, who has children aged five and seven. "I used to live in a mud house. This has made it easy for me to live. I'll vote for Julius Malema. If you look after me, I'll have to look after you, and that's what Malema did."

But there are still plenty of yellow ANC T-shirts in evidence in Nkandla, and Zuma's face adorns countless lamp-posts. KwaZulu-Natal was the only province where the party's share of the vote increased in 2009 and, with the IFP a fading force, this stronghold could be decisive in pushing the ANC towards a two-thirds majority. Zuma, who won the last election just after more than 700 bribery and corruption charges were dropped against him, has shown his powers of survival.

But elsewhere, in urban centres and among the middle class, Nkandlagate follows him like a shadow. Dali Mpofu, a lawyer and lifelong ANC member who defected to the Economic Freedom Fighters, says: "In all the rallies that we go to, I make a point of not always talking about Nkandla, but people almost force you to because it's a big issue for people on the ground.

"It's a simple issue of if one person can have a 250m rand house while they are living in the shack, obviously that can't be just a 'by the way' issue. The premier of Gauteng [province] this weekend said that they get asked this whenever they do their door-to-door campaigns, and I have no doubt that she's right about it."

The saga has also split the ANC. Some party stalwarts have even asked people to vote tactically against it, either by voting for a small party or spoiling their ballot. Addressing a room packed with mineworkers and the unemployed in Johannesburg last week, Ronnie Kasrils, a former intelligence minister, said: "We are saying that what is more important than the party is the ideas of the party, and it's the ideas that we are defending.

"If the party forgets such ideas, then we will stand against that and we will stand with our people who are demanding what we have promised them all our lives – a better life. So after 20 years yes, of achievements, of a right to vote and so on, we cannot stand idly by when we see these crimes of corruption and murder."

On Sunday the 72-year-old Zuma will make his case in a speech at the ANC's final election rally, hoping to avoid a repeat of the unprecedented booing he received at the same stadium during Mandela's memorial service. While the party is assured of a fifth consecutive victory on Wednesday, if it polls less than 60% of the vote, senior ANC figures – who have spent much of the campaign finding ways to apologise – will probably be panicked into wielding the dagger against him.

Should he fall, his likely successor is Cyril Ramaphosa, set to become deputy president after this week's election. Once a firebrand miners' leader, he went into business and was serving on the board of platinum producer Lonmin at the time of the police massacre of mineworkers at Marikana in 2012. Soon after he was criticised for bidding 19.5m rand for a prize buffalo and calf at a nearby auction.

Kasrils and old comrades who fear that the ANC's elite are losing their working-class credentials will have found little consolation last week when Ramaphosa addressed the media in an Edwardian-era mansion framed by Tuscan colonnades and Palladian windows, built to entertain the mining Randlords of Johannesburg.

"My great ambition is to be president of a golf club where I am playing," he teased. "That is my greatest dream. I'd like to be president of that.

"I'd also like to be president of my fishing club. I go fishing, and there is a club that I belong to. I want to be president of that. That is the sum total of my ambitions, to be president of one of the two. No other ambition do I have."