On a balmy night in Austin, against the backdrop of the glittering high-rises of the city’s booming downtown, Beto O’Rourke is laying out his audacious plan to change the face of Texas, and America.

In front of him, packed into an open-air park, a largely young crowd of 40,000 is thrumming with scarcely contained glee. Even in the liberal bubble of Austin they have never experienced anything like this: a Democrat seriously in the running for a Senate seat, vying to topple Ted Cruz, the Tea Party fanatic whom even fellow Republicans call “Lucifer in the flesh”.

“We are not running against anything or any political party,” O’Rourke tells his supporters. He punches his arm in the air like a pumped-up revolutionary while simultaneously wooing them with a crooner’s charms.

The message is relentlessly positive. The 45-year-old has tweaked Barack Obama’s 2008 catchphrase “Hope and change” for darker times, so now it becomes: “Hope over fear”.

“We are running for each other and this country that I love so much,” he says, to a giant roar.

Audacious is too detached a word for O’Rourke’s campaign. “Nuclear” gets closer. That a punk rocker-turned-politician who a year ago was barely known outside El Paso, the border town he represents in Congress, should be competitive at all in Texas is miraculous.

The state last saw a Democratic US senator 25 years ago. It has filled every statewide office with Republicans since 1998.

The latest polls put the race within margins of error. The respected Cook Political Report astounded observers by recently ranking the election as a “toss-up”. Real Clear Politics’ tracking poll has Cruz up four points, but it also has the contest as a toss up.

A toss-up? The implications flood in. Were O’Rourke to send Cruz packing it could hand back control of the Senate to the Democrats, with all that entails for Donald Trump’s program – and possible impeachment.

Facebook Twitter Pinterest The latest polls put the race between O’Rourke and Senator Ted Cruz within margins of error. Photograph: Laura Buckman/AFP/Getty Images

But the reverberations would go much further than that. Texas has the second largest population in the nation. With the number of Texans projected to double to more than 50 million by 2050 – think California and New York combined – its already outsized influence on US culture and politics is set to explode.

Lawrence Wright, the Pulitzer-prize winning author, points out when we meet in his hometown of Austin that any sign of the state becoming genuinely in play would be a shift of seismic consequences. “Once that happens, the whole nation’s politics changes. The Republican presidential strategy would be in deep trouble – they can’t win the White House without Texas.”

Wright’s book God Save Texas was published earlier this year when what might be called the Beto effect had yet to take off. Which is uncanny. Wright’s thesis – that Texas holds the key to the future of America, both as the origins of much of the rightwing tribalism that grips Washington, and as its possible cure – is precisely what is at stake in this epic senatorial battle.

“Cruz and O’Rourke represent different visions for the future of the state, and the country,” Wright says. “Cruz’s vision is very exclusionary, cold-hearted, out of tune with the more centrist political current that really runs through the state.”

Wright has been amazed to see “Beto for Texas” signs spring up in heavily conservative areas such as the Dallas suburbs. That is all the more surprising given O’Rourke’s background as bass player in a punk band and his unashamedly liberal positions on universal healthcare, gun control, immigration reform and marijuana legalization.

It’s as if O’Rourke has drilled umpteen boreholes and unleashed reserves of pent-up political energy, like geysers of black oil gushing from the Texas prairie. The force is palpable at his downtown Austin rally.

US voter suppression: why this Texas woman is facing five years' prison Read more

Young people in the crowd say they are drawn to what they see as a new style of politics crafted by Beto (everyone uses his first name, pronounced “Bet-Oh”). They like how he eschews focus groups and strategists, speaks his mind and refuses to take money from big corporate donors. The “Beto for Texas” campaign logo is printed in black and white rather than Democratic blue to denote a desire to cross boundaries.

Many at the rally, like Kaitlyn Watson, 19, are first-time voters who see in O’Rourke an escape from suffocating reactionary pasts. For her to vote for a Democrat is an act of rebellion coming as she does from Beaumont, an intensely conservative town.

In Watson’s final year of high school she posted a tweet critical of vice-president Mike Pence for being anti-gay. The tweet went viral, and she was deluged by rebukes from her Trump-supporting schoolmates.

“Beaumont is extremely conservative and closed-minded. I got out as soon as I could,” she says.

Watson’s friend and fellow Beaumont resident, Aimee Huckeba, 20, has also been baited for liberal views. Her teacher made fun of her in front of classmates and other students bullied her when she spoke in government class.

Now it’s payback time: she is spending her spare hours cajoling young Texans to register to vote for O’Rourke. “This is our chance to make change.”

Huckeba is not alone. Something is afoot in the great state of Texas. About 1.6m additional names have been added to voter rolls since the 2014 midterm elections, a surge that could play critically in O’Rourke’s favor.

One of his stock lines is that Texas is not a conservative state or a liberal state – it is a non-voting state. Trump won Texas by nine points in 2016, but he and Hillary Clinton were trounced by the army of Texans who stayed home.

Just 43% of registered voters went to the polls – one of the worst turnouts in the nation – and the proportion of young people who participated was substantially lower. If O’Rourke could tap into that potential among young, as well as low-income and Hispanic Texans who tend to lean Democratic yet vote sporadically, then what seemed to be a pipe dream a few months ago could be within his grasp.

Facebook Twitter Pinterest Ted Cruz, who rode the Tea Party wave into the Senate in 2012, is for the first time facing a viable challenge from the left. Photograph: Pool/Reuters

Then there are women like Jennifer Harris, 45. She is the proud owner of a bumper sticker that reads: “Republicans for Beto”.

Harris has been a Republican voter for 20 years, but will be voting for O’Rourke in November. “Moderate, college-educated women like me, we’ve had enough of the tribalism, of the politics of no that Ted Cruz represents.”

Harris has a pre-existing health condition, and has no faith Republicans will protect her healthcare coverage. Under the influence of the Tea Party, Texas has the highest number of people without health insurance in the country – some 4.3 miliion, including 623,000 children.

She also has a 10-year-old son, and has watched with dismay as low-tax Tea Party Republicans have pared investment in education down to about $9,000 per public schoolchild a year. That’s 25% below the national average, in a state that hosts one in 10 of all American children.

Classrooms: the latest battleground in Texas's culture wars Read more

The last straw was Ted Cruz’s support of Brett Kavanaugh for the US supreme court despite sexual assault allegations against the judge, compounded by Trump’s mocking of his accuser Dr Christine Blasey Ford this week. Harris has come to believe that Cruz has put self and party before country.

“Republican women are mad,” Harris said. “We really might decide this election. The pendulum could be swinging back.”

The last time the pendulum swung in Texas was 30 years ago. The state was rock-solid Democratic for decades, but in the 1980s and 1990s it did a full flip and became rock-solid Republican.

Two individuals did more than any others to bring about that sea change. One of them was the most celebrated Texan of them all: Lyndon Johnson, who after the assassination of John F Kennedy put the Lone Star state on the map with his gruff Texan ways and Great Society reforms.

Johnson knew that his landmark civil rights legislation outlawing segregation and extending voting rights to black people would go down like a lead balloon in the then Democratic south, his home state included. He is reported to have said “we have lost the south for a generation”.

And so it came to pass. In contrast to Johnson’s achievements – the creation of Medicare and Medicaid and the “war on poverty” – Cruz and co have shut down the federal government, slashed public services and promoted anti-transgender bathroom bills. Last year they surpassed themselves by legalizing open carry of swords on Texan streets.

So how would Johnson regard his beloved Texas today now that it has turned not only Republican but rabidly Tea Party? I put the question to his nephew Philip Bobbitt, a constitutional law scholar at the University of Texas at Austin who as a young man lived in Johnson’s White House.

“I don’t see Lyndon wringing his hands about it,” Bobbitt says. “He was a doer, a man of action. He would say, ‘Let’s get moving! Let’s get out the vote!’”

Facebook Twitter Pinterest Supporter Caroline Korst stands for a portrait after a campaign rally for Beto O’Rourke. Photograph: Laura Buckman/AFP/Getty Images

That sums up Beto O’Rourke’s strategy towards dealing with Ted Cruz. Instead of bleating about the demise of progressive politics, he is a doer. He has visited all 254 counties in the state, sowing seeds of a possible Democratic reawakening as he goes.

A less positive take on the Democratic bid for resurgence is given by the second individual who did more than any other to wrest Texas into the Republican camp: Karl Rove. Rove is best known nationally for securing two presidential terms for George W Bush, but in Texas he is legendary for having given the state a total makeover, systematically replacing the Democratic old guard with Rove Republicans.

I ask Rove whether Beto O’Rourke could be about to do what he did, but in reverse.

“Texas could swing back into the Democratic column,” he begins. “But not with Beto O’Rourke. He is one of the most reckless candidates I’ve ever seen in my life.”

Reckless?

“He has this habit of swearing extensively in the wrong places, like town hall meetings in rural Texas. That might be cute in hip neighborhoods in Houston, but O’Rourke says the f-word and a whole lot more in front of a bunch of churchgoers in small towns.”

Rove is even more scathing about the viral video of a speech in which O’Rourke said there was “nothing more American” than NFL players kneeling through the national anthem. “I can think of a million things more American than taking the knee,” Rove says, “like saluting the flag or wearing the uniform.”

The master strategist goes on: “Beto is a media creation. Is he going to come closer than Democrats in recent races? Probably. Is he going to win? I find it highly unlikely.”

He has this habit of swearing extensively in the wrong places, like rural town hall meetings Karl Rove

That will be music to the ears of Ted Cruz, who for the first time is facing a viable challenge from the left. When Cruz rode the Tea Party wave into the US Senate in 2012, his only competition was internal – between the right and the far right – over who would be Republican candidate.

In effect, he was elected in the primary that year by 631,000 votes – in a state of 27 million.

Are we seeing signs of a Democratic wave in the primaries? | Jill Abramson Read more

Cruz, who is only two years older than O’Rourke but comes across as part of a crustier generation, is following the conventional campaign playbook: pumping big money into negative ads attacking his rival’s integrity. Being Cruz though – he was, after all, the politician who brought us “machine-gun bacon” – his attempts at character assassination have largely fallen flat.

Last month he warned that his rival was plotting to ban barbecue across the state. Less amusing was the dog-whistle tweet he put out containing a clip of O’Rourke expressing outrage at the recent killing by a female police officer of an African American man in his Dallas apartment.

Cruz captioned the clip “In Beto O’Rourke’s own words”. The implication was clear: that there was something reprehensible about criticizing the police for shooting an unarmed black man in his own home.

Some of the attacks on O’Rourke have come closer to their mark. This week he was forced to apologise after a review of a Broadway show he had written in 1991, when he was 19, was dredged up in which he said the only qualifications of the female actors were there “phenomenally large breasts and tight buttocks”.

But overall O’Rourke’s refusal to be sidetracked from his positive politics has worked. It’s a sign of how rattled Cruz is that he has had to call in the artillery, in the shape of his arch-enemy Donald Trump.

Play Video 4:11 'I can think of nothing more American': Beto O’Rourke responds to question on NFL protests – video

In the 2016 presidential primaries, Trump ridiculed him as “Lyin’ Ted Cruz”, made crude references to his wife and suggested his father was behind the JFK assassination. Cruz opponents are making the most of this awkward history. They have placed a blown-up tweet in which Trump slams Cruz for accomplishing “absolutely nothing” on the side of a truck which they park for maximum embarrassment in front of the Republican candidate’s public appearances.

Trump will make a stump appearance for Cruz this month – an astonishing capitulation for the Texan given such visceral antipathy in their past. “Cruz needs Trump to come to Texas to tell all Trump supporters to vote for him, and then he needs Trump to go away and have nothing more to do with it,” said Mark Jones, a political scientist at Rice University.

Despite the evidence of jitters on Cruz’s part, it’s way too early to write his political obituary. The mere voting patterns of Texas suggest the odds are stacked in the Republican’s favor.

In 2004 the state reached a historic landmark when its minority populations – Hispanic, African American and Asian – overtook the white majority. Since then the Latino community has grown steadily, standing today at 40% of all Texans and in turn shrinking the proportion of non-Hispanic white Texans, or Anglos, to an almost equal 42%.

That represents a fundamental demographic shift that every year makes the overwhelmingly white male rightwing Republicans look more unrepresentative of the Texan people. But don’t be fooled. The impact is slow to filter through electorally. White Texans still account for the lion’s share of those who actually turn up at the polling station. In 2016, 9.5 million Texans voted in the presidential election, of whom 62% were Anglos.

It doesn’t take much effort to find those Trump-supporting, Tea Party-friendly white voters upon whom Cruz’s survival depends. The minute you step out of the Austin bubble and head out into the scrubby ranchlands that surround it, they are everywhere.

Drive a couple of hours south to Gonzales, a small town of 7,000 in cattle country, and you enter another Texas. If Austin, with its sparkling new Google skyscraper, is the face of Texas’s and America’s future, Gonzales is holding firm to the past.

The town’s claim to fame is that it was here that the first shot of the Texas Revolution was fired in 1835 after colonists refused to hand back to Mexico a borrowed cannon. As they faced off the Mexican army, Gonzales colonists raised a homemade banner on which they had sketched the cannon with the words: “Come and take it”.

The refrain has been co-opted by the Tea Party as a symbol of Texan grit and devotion to guns. Ted Cruz wore a “Come and take it” flag pinned to his lapel when addressing the US Senate.

Facebook Twitter Pinterest WR Low, 75, a retired rancher, poses at the Cow Patty, a restaurant in Gonzales, Texas. Low is opposed to O’Rourke. Photograph: Katie Hayes Luke/The Guardian

A huge ‘Come and take it’ flag now flies on one side of Gonzales square. On the other side another feature of Texan history is celebrated in the form of a 30ft column carrying the statue of a Confederate soldier. The memorial faces north to ward off the invading – and slavery opposing – yankees.

A short drive from the town center, four cowboys are resting in the Cow Palace diner after a long morning on horseback. Three cowboys, to be accurate. The fourth, the elder statesman of the group, says: “I’m a cowman, I guess.”

Beto O'Rourke: can the upstart Texas Democrat eject Ted Cruz? Read more

As he chews on deep-fried steak smothered in white sauce, WR Low, 75, tells me why talk of a competitive election is baloney. “I don’t think there’s going to be a race. No one rates that guy,” he says, referring to O’Rourke whose name he seems unable, or unwilling, to speak.

Why not?

“’Cause he’s an idiot. His platform’s all wrong. He’s against cops and the military. He wants everybody to come across the border and get welfare.”

Low thinks that if “that guy” were elected, the cowboys’ way of life would be under siege. “We are struggling already in rural areas and that guy would put everybody on welfare – all we have left would flow right out of town.”

Out on his ranch, Gary Henderson, 64, is surveying his cows sheltering under mesquite and American oak. He thinks the problem with Democrats is that they believe money grows like leaves on those trees. “It has to stop somewhere – they keep giving millionaires tax breaks.”

But Trump has just given a $30bn tax break to individuals making more than $1m a year.

“That’s true, but it wasn’t political.”

Did you benefit personally from Trump’s tax cuts?

“Not much. But at least my taxes aren’t going up.”

Facebook Twitter Pinterest Gary Henderson, pictured at his 160 acre ranch in Gonzales, is one of many Trump supporters in Texas. Photograph: Katie Hayes Luke/The Guardian

The adulation shown by small-town and country Texans toward Trump helps explain why Cruz so cravenly needs a helping hand from the man who humiliated him. But it should also be a warning to Beto O’Rourke.

That foreboding is underlined back in the center of town, where the Latino owners of Milupita Taco House are clearing up after lunch. Gonzales is 40% Hispanic, though you wouldn’t know it. The community is virtually invisible, with most people working in meat-producing factories in the vicinities while those in town keep an extremely low profile.

Sebastian Esquivel, 21, is one of the cooks in the family business. He has never voted in his life, and in his extended family of about 20 relatives in Gonzales – US citizens every one – he knows of but one who has ever cast a ballot.

Esquivel does not recognise the names of Beto O’Rourke or Ted Cruz, and cannot identify their respective parties. “To be honest, to me it doesn’t really matter,” he says.

What does matter?

“My family and me. Putting food on the table, paying bills.”

That’s what Beto O’Rourke is up against. Entire communities of marginalised voters have been so excluded from the political process – purposefully so, over many years – that it will take more than one superstar candidate in a single political cycle to set them right.

The future of Texas, of America, is in their hands. But it’s still news to them.