Voters in Arizona have ousted Sheriff Joe Arpaio, ending his controversial campaign against undocumented immigrants in a ray of good news for America’s Latinos.

A surge in Latino voters thwarted Arpaio’s attempt to win a seventh term as sheriff of Maricopa County, triggering rare scenes of jubilation by Democrats on Tuesday night.

Activists marched on the sheriff’s headquarters in downtown Phoenix, chanting and cheering that they no longer had reason to fear a police force they viewed as an instrument of oppression and intimidation.

Democrats found solace in an otherwise dark night for progressives by hailing “a new sheriff in town” – Paul Penzone, a retired Phoenix police sergeant who vowed to transform the department.

“I pursued this because I want to restore the respect, the transparency,” he told supporters, with polls giving him an unassailable 55% over 44% for Arpaio. “No longer will we be known by the notoriety of one. The only division we should see in the community is between those who commit the crime and those are willing to hold them accountable.”

Penzone’s victory threw the Democratic election watch party, held at a hotel opposite the GOP’s party, into a wild mood swing of jubilation at the local result and despair at national results flashing up on TV screens. “Arpaio is gone but I need a Xanax,” said one canvasser, struggling to absorb the news.

Arpaio, 84, was first elected in 1993. His crackdowns on undocumented residents foreshadowed the hardline immigration rhetoric of Donald Trump, who shared platforms with the sheriff in the run-up to the election. Both men pushed the false assertion that Barack Obama was born in Africa.

In an otherwise impressive sweep for Republicans – which included Trump and Senator John McCain prevailing in Arizona – Arpaio succumbed to Latino mobilisation, outside money and legal troubles which besmirched his reputation, cost taxpayers millions of dollars and alienated some supporters.

Last week, federal prosecutors charged the veteran lawman with criminal contempt of court for allegedly violating an order by continuing to arrest immigrants with no evidence they had broken any state law. That came on top of a 2007 civil lawsuit that found against him in a claim that he racially profiled Latinos.

One of his most contentious techniques over the years has been to conduct mass raids of factories where undocumented Latinos were suspected of working, rounding them up and then detaining anyone lacking appropriate immigration paperwork.

He was also notorious for headline-grabbing policies such as forcing prisoners to wear pink underwear and to live in tents despite Arizona’s harsh desert climate.

Arpaio issued a statement saying it had been an honour and privilege to serve as sheriff for 24 years, with his wife by his side.

“Tonight, the people have spoken. And while Ava and I are disappointed in the results we respect their decision. We want to congratulate Paul Penzone on his victory and look forward to working with him on a seamless transition.

“My thanks and appreciation to the people of Maricopa County for the faith and trust they put in me over the years.”

Students demonstrate against Arizona’s Sheriff Joe Arpaio on election day in Phoenix. Photograph: Nancy Wiechec/Reuters

Arpaio’s media savvy and hardline policies made him a folk hero on the right and a hate figure on the left, especially among Latinos, who accused him of hounding and breaking up families.

“Apart from anything else he’s costing the taxpayer too much money. We need to get him out of there,” said Ida Pena, 73, a retired newspaper circulation manager, after voting in southern Phoenix.

Complaints by Latino drivers about racial profiling prompted a judge to demand Arpaio stop the enforcement efforts. He was later found to have violated the order, leading to the contempt of court case.

Arpaio has acknowledged violating the order to stop the immigration patrols but said the disobedience was not intentional. Payouts to settle profiling cases have cost county taxpayers $48m, with pending cases expected to raise that to $72m.

The roots of his downfall go back to 2010 when Arizona, a GOP bastion, passed a state bill known as SB 1070 which introduced sweeping restrictions on those suspected of lacking documentation – a proto-Trumpism.

Latinos responded by building a coalition of veteran protesters, devout Christians and media-savvy millennials. It registered voters and started eroding Arpaio’s traditional landslide victories.

“We’ve been building and building,” said Petra Falcon, who founded Promise Arizona, part of a 14-member umbrella group, One Arizona, which registered more than 150,000 voters for this election.

The group mounted a statue of the Virgin of Guadalupe on a truck bedecked with US flags and a speaker which traversed Phoenix on Tuesday booming exhortations: “Vamos a votar.”

Outside money has also poured into Arpaio’s race for re-election, as his detractors have sensed a chance to finally unseat him. The philanthropist and financier George Soros invested $2m into the Super Pac Maricopa Strong to pay for a series of TV attack ads on the sheriff.

