Ronald Reagan famously described the University of California at Berkeley as "a haven for communist sympathisers". Another rightwing politician – who styles himself as Reagan's intellectual successor – will stand at a lectern this week at the Bay Area campus to decry the excesses of government surveillance.

Rand Paul, 51-year-old senator and standard bearer of the right, can expect an enthusiastic reception from his young leftwing audience on Wednesday.

The rise of Paul and his libertarian brand of politics, one of the few intellectual movements with appeal across the spectrum, is turning old political assumptions on their head. Paul, the son of libertarian guru and former congressman Ron Paul, was the star of last week's Conservative Political Action Conference (CPAC) – an annual litmus test for the Republican base. For the second year running, Paul won the straw poll of likely Republican 2016 presidential contenders, trouncing fellow senators Marco Rubio and Ted Cruz as well as scandal-hit New Jersey governor Chris Christie.

For decades, libertarianism has hovered on the fringe of the Republican party, associated with Ayn Rand novels, anarchists and tax refuseniks. Now it is suddenly a political force to be reckoned with, disrupting the political establishment by sparking unusual alliances between Republicans and Democrats.

Nowhere has that dynamic been more evident than in the fallout over Edward Snowden's disclosures about widespread surveillance by the National Security Agency. Reform of the NSA is the issue Paul has claimed as his own, even to the extent of filing a lawsuit, naming himself as co-plaintiff, to challenge the agency's mass surveillance of domestic phone records. "I believe what you do on your cell phone is none of their damn business," Paul told CPAC delegates last Saturday to resounding cheers.

It used to be centrists who crossed the aisle to reach compromise. Today, it is those on the libertarian right, like Paul, who are finding common cause with Democrats such as Ron Wyden and Elizabeth Warren, both among the most leftwing voices in the Senate.

Paul's father, who retired from his Texas congressional seat in 2012, said in an interview that the Snowden disclosures "awakened a lot of people [who said] 'hey, we knew it was bad, but not that bad'". But the phenomenon that has seen Paul and his affiliates take centre stage has deeper roots than Snowden and the NSA. While Republicans and Democrats remain as diametrically opposed on tax and spending as they were in the 1980s, on a host of other issues libertarians are exploiting political overlap with the left.

Whether it is prison reform, opposition to drones, gay rights or drug sentencing policy, Paul and similarly minded Republicans are breaking new ground for the party and redrawing Washington's political battle lines.

But this is not a trend that should be overstated. Washington remains a bitterly divided town and Paul's laissez-faire economic policies still make his Democratic opponents wince. Neither is it entirely new: as 78-year-old Ron Paul points out, he and leftwing Democrat Dennis Kucinich became kindred spirits during the 2008 presidential campaign.

But Rand Paul is on a more pragmatic, establishment course than his father. "If I had played the game, I would have had enough seniority to become the chairman of a committee," the older Paul said. "But that was the worst idea because it meant having to toe the line."

He added: "Political leaders are nothing more than just opportunists." When it was suggested the label might apply to his son, he backtracked: "When I talk so negatively, I'm usually talking about the people who get to the top."

But that is exactly where Rand Paul wants to be, and he has been willing to take stances totally at odds with his father, such as his hawkish position on Ukraine, in order to project a moderated libertarian message.

And there is every indication that America is increasingly susceptible to his ideas. A recent survey revealed that, for the first time in almost half a century of polling, most Americans are inclined to think the US should "mind its own business internationally". That is practically a catchphrase for both Pauls. The libertarian shift is particularly acute among so-called "millennials" – those aged between 18 and 33.

Carroll Doherty, director of political research at the polling organisation Pew, cautioned that on a key issue – the size of government and support for federal programmes – young people tend to disagree profoundly with small-government, anti-tax libertarians. But he added: "Young people in both parties exhibit libertarian tendencies on some issues", including key topics such as gay rights, gun ownership, surveillance and marijuana legalisation. They also show limited interest in the kind of tribal attachment to political parties of their parents' generation.

Into this disorientating mix has stepped the 5ft 7in senator from Kentucky who claims his politics are "more philosophical in nature than partisan". But Rand Paul is not just exploiting societal shifts in opinion. John Samples, an adjunct professor at Johns Hopkins University who has closely followed Paul's career, notes he has been fortunate to come to prominence when the Republican party is at a crossroads.

On one hand, he says, the neoconservatives, a dominant force after 9/11, have lost influence because of their unpopular wars in Afghanistan and Iraq. Meanwhile, so-called "social conservatives" – the Christian evangelicals who have spent decades trying to end abortion and block gay rights – are realising they are on the wrong side of history. "There was a huge bet on gay marriage, ultimately," Samples said. "They lost. Public opinion makes pretty clear that if that isn't a lost cause now, it will soon be."

Republican insiders say the biggest challenge for a Rand Paul presidential run will be distinguishing himself from his father, who made three failed bids for the White House. In one sense, his father's connection brings a ready-made base. Ron Paul has a devoted following, particularly among students. He is possibly the only other senior Republican who would be welcomed at Berkeley; it was standing room only when he spoke there himself in 2012. But Ron Paul was always considered too radical to move in from the fringe. He is seen as being what Americans disparagingly call "kooky".

The senator needs to tread a fine line, as revealed by a line in his book, The Tea Party Goes To Washington: "I remain very much my father's son, not only in my politics but in the way that Dad and I have different approaches to things."

In any presidential campaign, Paul would face a family challenge shared by two of his likely opponents. Former Florida governor Jeb Bush, who many pundits believe would win the Republican nomination if he decides to run, would have to overcome a uniquely toxic brand. His father, George HW Bush, is in the ignoble club of one-term presidents. His brother, George W, led a deeply divisive war that resulted in the lowest presidential approval ratings in modern history.

Hillary Clinton, the favoured Democratic candidate for 2016, must also wrestle with the shadow of her husband, Bill. She has yet to decide whether she will run, but is already leading – in terms of money and popularity – other Democrats by a huge margin.

The election is more than two years away, and already the Bush and Clinton dynasties have an air of inevitability about them. But there is something zeitgeist-like about the Paul brand: a sense that its moment may have arrived.