Amid all the flip-flopping and dissembling of the 2012 election, it comes as a bit of a shock to hear one of the presidential candidates say that his interest in politics stemmed from the first time he smoked dope. But then there is much about Gary Johnson, the Libertarian party's nominee for the White House, that is strikingly out of the mould.

"I think marijuana played a role in my libertarianism, because when in 1970 I smoked it for the first time I realized that everything the government said about it was a lie," he said. "You know, you smoke marijuana, you're going to go crazy, you'll want to commit crime, you are going to go to the depths and never return … None of that was true."

From those teenaged dope-smoking beginnings, Johnson, now 59, went on to become a successful businessman and a Republican governor of New Mexico, and is now seeking to carry the libertarian flame lit by Ron Paul during the primary election season into November's presidential race. It's true that he and his Libertarian party are relative minnows in an election cycle dominated by big corporate money – he has raised just $2m for his presidential campaign compared to the hundreds of millions being poured into Barack Obama's and Mitt Romney's campaigns.

But there are signs that in the key swing states where the election will be won or lost, he is having an impact that could prove important over the final 60 days of the race. Polls are so tight in states such as Ohio, Nevada, Colorado, Virginia that any deflection of votes away from the two main contenders could be significant.

A CNN/ORC poll last week put him at 4%, which may not sound much, but given that a number of key states are essential tied between Obama and Romney even that sliver could be crucial. The poll also found that his third-party bid, combined with Jill Stein of the Green party, is eating into Romney's support base more than it is the incumbent president's.

Gary Johnson. Photograph: Orlando Sentinel/MCT/Getty Images

That explains why the GOP is acting so aggressively against him. They have tried to keep Johnson off the ballot in at least five states – Ohio and Virginia where they have already failed to do so, and Michigan, Oklahoma and Pennsylvania where court challenges are still ongoing.

Johnson, talking in New York where he was on a media blitz, said he was unfazed by the Republican attempts to block him. He vowed that come November he would be on the ballot in all 50 states, which is not an idle promise as he is already included in 45 states.

"These challenges against me are just railroad jobs: it's straight out of the Wild West. I think it's a testament to the strength of our campaign that they are going to these lengths. The Republicans are afraid of me taking away votes from Romney: that's the bottom line."

He might lack the riches that Ross Perot was able to inject into his third-party bid in 1992 – an effort that helped put Bill Clinton into the White House by splitting the rightwing vote – but Johnson insists he is in this to win a few states. He cites as central targets Wyoming, Montana, Nevada and Alaska – those western states with a history of frontier individualism that renders them open to the Libertarian message.

He may be financially weak, but Johnson does have the benefit of coming after Ron Paul's energetic primary season. Paul's platform of small government, balanced budgets, an end to the war in Afghanistan and social freedoms such as marijuana legalization was essentially the same as Johnson's now.

"We may communicate things differently, but I don't see any difference between us in our policies," Johnson says.

Paul has notably refrained from endorsing either Romney or Johnson – a failing that Johnson puts down to the fact that Paul's son, Rand, is a Republican senator. "I think he's between a rock and a hard place," Johnson said.

Still, Johnson says that since the end of the primaries his Libertarian campaign has felt a surge of energy as Paul's dedicated army of young followers has swung behind him. "Now that Paul's out of the race, who represents all those folks? Is it Romney, or Obama or me? Well, clearly it's me. From our vantage point, there's been a massive movement of those people to our camp."

That energy could be felt on Reddit, the internet community, where Johnson held an "I am a…" chat session on Tuesday. He received more than 6,000 questions and comments and was second most popular item on the site.

You can see why Johnson would intrigue and inspire young voters from the answers he gave to Reddit. Such as this one: "I put the biggest threat to America as politicians who beat their chests over the threat of drugs, the threat of the illegal immigrant, the threat of terrorism, the threat of poor healthcare, all at a cost of bankrupting America".

Or this: "As the Libertarian candidate, I am the only one talking about lowering welfare spending and warfare spending in the same sentence."

The disproportionate threat that he poses to Romney is surprising, because in his policies he speaks at least as powerfully to progressive Americans as to conservatives. When he was governor of New Mexico between 1995 and 2003, he explored the possibility of legalizing marijuana in the state.

His presidential slate includes support for gay marriage – "what you do with your life is up to you, as long as you don't harm me", he said – an attack on Obama for his record on civil liberties, and a pledge to bring the troops immediately out of Afghanistan.

On the other side, while governor he famously used his veto 750 times – "Governor Veto" they called him – and he is committed to balancing the federal budget, overturning Obamacare and turning the health system over to the free market.

"The majority of Americans are fiscally responsible and socially accepting. I'm in that category of people – I think I'm representative of the majority of people in this country.

"Yet these people are let down by the two-party system. You've got Democrats that are supposed to be good on civil liberties but haven't been so good of late, and Republicans who are supposed to be good on dollars and cents but I'm not sure they have ever really been good at that. Combine them both, and arguably that's me."

His big picture hope is that his presence in the race will open up political debate in the US, so that currently taboo subjects become acceptable in public discourse. "I want to get enough votes to create a political safe ground," he said.

"So that Republicans think: gee! We can talk about having a truly balanced budget, and Democrats can actually talk about ending the wars and the drug war. That way we can change the world."