Francis Fukuyama is on day 24 of a world tour to plug his fat new book, The Origins of Political Order. I bump into him and a minder as he arrives at his publisher's offices in central London. The offices, in what looks like an old warehouse, aspire to be a bit Manhattan – open plan, stripped wood, buzzy. The downside is that the ancient lift has packed up, and the office is three floors up. Nobly, Frank – as his friends call him – insists on carrying a suitcase, which is almost as large as he is, as well as his backpack up all three flights, despite my efforts to help.

He has just had breakfast with the Financial Times and is doing the rounds of TV studios, but pausing only to get a cup of tea we plunge straight in to what for me is a rather intimidating seminar on global politics. It's a bit like being 20 again and, horribly underprepared, going to a tutor to discuss the church under Henry II. Happily, Fukuyama fields my scattergun questions with polite aplomb. The only time he looks disconcerted is when the photographer asks him to start taking off his clothes to get a more relaxed look – Fukuyama doesn't really do relaxed.

Almost 20 years after it appeared, he is still best known as the author of The End of History. It was that book – perhaps even just that title – that turned a foreign policy wonk and middle-ranking figure in the state department into a global super-pundit. The 1992 book, which expanded on a famous essay published three years earlier, was much quoted without being much read. Much mocked, too, after 9/11, when his critics pointed out that, far from being over, history seemed to be more urgent and unpredictable than ever. But they had misunderstood his thesis: he had not argued that conflict would cease but that, with the collapse of the Soviet empire, the ideological struggle was over. Liberal democracy was the only game in town.

The new book, the first of two volumes, explores how liberal democracies are established, how – in a nice phrase he uses prominently – countries "get to Denmark". In the west we take a great deal for granted – that we can vote governments out, that the rule of law will more or less hold sway, that corruption will be punished, that we will enjoy political freedoms, but much of the world doesn't enjoy those privileges. Fukuyama is attempting to work out how states developed and why some became liberal democracies and others, notably China, opted for an authoritarian model.

Fukuyama argues that getting to Denmark relies on three things that have to be in harmony – a functioning state, the rule of law and accountable government. China's problem was an overmighty state: it got civilisation too soon. By a series of happy accidents, England managed to get all three by the 17th century, exported them to the US via freedom-conscious settlers and provided a model for the rest of the world. Those three preconditions of liberal democracy are the holy grail. "The fact," he writes, "that there are countries capable of achieving this balance constitutes the miracle of modern politics, since it is not obvious that they can be combined."

The condition that the debt-ridden and divided EU is in at the moment made me wonder whether "getting to Denmark" was quite so desirable a journey these days, and I begin by asking him what he makes of Europe's nervous breakdown. "Collectively it seems to me that the EU is in big trouble," he says. "They basically let in a whole bunch of countries that they shouldn't have. There's no mechanism for disciplining them once they're in and there's no exit strategy." He doesn't understand why Greece, Ireland and Portugal are submitting to the euro straitjacket. "The policy which is now being dictated out of Berlin is crazy. There's just no way those countries are going to grow with a strong currency and an austerity policy that stretches out for years into the future. They'll have to consider coming out."

The point I'm driving at is a pessimistic one: that the EU might implode; that the predicted decade of austerity could produce very nasty, ultra-competitive national politics; that Beacon Europe might become Fortress Europe; that as in the 1930s liberal democracy could come under assault even in its heartlands. To my surprise, he accepts the argument, in part at least. "That's one of the things that is in this book that wasn't in my original book – the possibility of political decay. I don't think there's any particular reason why, if you are a liberal democracy, you can't decay. Your institutions can get too rigid; your ideas can get too rigid. I think right now a lot of developed democracies are going to have to renegotiate their basic social contract, because a lot of the welfare state arrangements are just not sustainable, and that's something democracies are really not good at. They aren't good at persuading people to pay higher taxes and accept cuts in benefit for the sake of something that's going to happen a generation from now."

Apocalypse now? "Things could get bad quickly," he admits. "We're seeing the rise of populist parties across Europe. There's a lot of political correctness about immigration and the whole nexus of problems associated with it. People aren't allowed to talk about that, and there's now been this explosion on the right where people not only talk about it but are saying some pretty nasty things about it."

He finds it odd that the crash of 2008 and the political disaffection that has flowed from it have fuelled rightwing populism but not leftwing populism, either in the US or Europe. "The left isn't strong anywhere," he says. "You don't have charismatic, inspiring leaders anywhere. You look at Italy. Why is this rascal Berlusconi still prime minister? It's because the left in Italy can't come up with an inspiring agenda that anyone believes in."

He seems to think that the long-term trend is towards liberal democracy and political freedom, but in the short term we may all be dead because democracies struggle in slumps. "It is much easier to run a democracy and a capitalist economy that produces inequalities if you have long-term growth because, even if it's not evenly shared, at least everybody is benefiting down the road. Without growth you return to a Malthusian world where it's more zero sum. One person gets rich at the expense of another person, and then it becomes much harder to maintain democracy."

So is he an optimist – the conventional reading after The End of History – or a pessimist? "I'm basically an optimist because I do think there's this historical modernisation process, and by and large it's been very beneficial to people. But there are blips. History doesn't proceed in a linear way." Or in a geographically even way – the current pessimism in Europe is offset by hope in north Africa and the Middle East. "The Arab spring has," he says, "put a lot of authoritarian governments on notice."

Fukuyama's official position these days is as a senior fellow in international studies at Stanford University in California. This is one of those glorious American academic jobs where he gets to teach when he wants to, and is essentially being paid to think – and to add lustre to his department. It leaves him free to sit on a dozen advisory boards around the world, and to get involved in putting into practice the overriding lesson of his new book, which is that building workable democracies is tough and relies on the grassroots being cultivated. "I've been running an international development programme and doing a lot of work with the World Bank and aid agencies which are trying to improve governance and deal with corruption in weak states," he says. "The Denmark problem is a big one. People have unrealistic expectations for the kinds of improvements that can be made, and how quickly. They need to set more modest goals."

This new realism also feeds into the political journey he has undergone. In the 1980s he was a strong supporter of Reaganism and worked for the state department under both Reagan and George Bush Sr. He supported George W Bush in 2000, and favoured intervention after 9/11. But he has since renounced both Bush and the attempt to impose democracy on countries such as Iraq and Afghanistan. He voted for Obama in 2008 and tells me that, following his move last year from Washington to the west coast, he does not intend to register as a Republican supporter (registration enables you to vote in state primaries) but will register as an independent, or perhaps even as a Democrat. For a conservative thinker with strong Republican links and a reputation in the 1980s as a leading neocon, a Rubicon has been crossed.

The turning point was the younger Bush's mishandling of the Iraq war. "They didn't launch the war to export democracy," he says. "They launched it because of security concerns, and attached this democracy justification as an afterthought, which I didn't think was helpful to the cause of democracy. If you thought the problem through, you'd realise that this is a long-term, costly endeavour, and you would think long and hard before you took it on, because if you just do it in a half-arsed way and give up after a few years you're liable to make things worse."

Fukuyama made a powerful case against his former neocon allies in his 2006 book America at the Crossroads. He still wants to "export American ideals", but tells me "it ought to be done through soft-power instruments". "In general," he says, "Americans are not very good at nation-building and not very good colonialists. Look at the impact of the United States on Latin America or the one colony we had, the Philippines. Those countries are still not doing very well. We stumbled into Afghanistan and Iraq, which are basically tribal societies, and most Americans have no idea of how a tribal society operates."

The mistakes of the Bush years were, he believes, a direct consequence of Reagan's success in seeing off the Soviet Union in the 1980s, a high-stakes gamble that could have backfired and succeeded only because of the liberalising role played by Mikhail Gorbachev. "This minor political miracle happens – they take this very principled stand against a dictatorship, they're not willing to compromise, and then the dictatorship collapses. That was their [the Republicans'] last experience of government, then you had the Clinton years, and what they were hoping for was a repeat of that in Iraq. You take a principled stand against a dictator, you depose him, and then you have a similar eastern Europe-style upwelling of support. But they should have realised that the eastern European situation was an unusual one. The roots were there. They were basically western countries that had been knocked off course by the Soviet Union, and it was natural that they should embrace western values and democracy, whereas Iraq, because of the Israeli-Palestinian situation and the whole history of colonialism, was never going to embrace the west."

What about Libya? Is intervention there also a mistake? "I supported the no-fly zone. It would have been terrible if Gaddafi had got into Benghazi. Having said that, we're stuck now, and there's not an obvious good way out, but I still think this is better than letting Gaddafi take Benghazi. He's more likely to collapse than the opposition, and you just have to keep your fingers crossed." The bigger question is how to build a successful state once he has gone, and the National Endowment for Democracy – one of the organisations Fukuyama advises – is mentoring rebel groups in an effort to create the conditions for a successful transition.

Fukuyama, who is 58, was born in Chicago but grew up in New York. His father is a second-generation Japanese-American whose own father fled the Russo-Japanese war in 1905 and started a shop on the west coast before being interned in the second world war (that distant family experience has made Fukuyama a critic of Islamophobia). His mother, who comes from an academic family in Japan, met her husband when she came to study in the US. Some Japanese was spoken at home, but Fukuyama, an only child, never learned to speak it – "it just wasn't fashionable to be ethnic when I was growing up" – though he says his three children have embraced their dual identity and his eldest son is learning Japanese.

His education was multi-layered: classics and humanities at Cornell, comparative literature at Yale, political science at Harvard. He specialised in the Soviet Union and in 1979 joined the Rand Corporation thinktank, before spending two spells as an adviser at the state department under Republican administrations in 1981-82 and 1989-90. As well as moving away from the Republicans – he says the party is "out to lunch at the moment" – he insists that he is not one of those Kissinger-style academics who covets a big job in the government, of whatever hue.

"I've figured out in the course of my life that the one thing I'm good at doing is writing books," he says, "and it would be crazy to trade that in for something else. I can still contribute to the political debate." If he had pat solutions to the world's problems he might be tempted, he says, but there are no pat solutions. "Take Pakistan, which I think is the scariest and most dangerous issue facing us right now. I have no idea what you do there." One thing he is sure about is that Obama was right to authorise the killing of Bin Laden. "In an extraordinary case like that, it wouldn't have been possible to put him on trial," he says. "It would have been a circus."

Despite all the problems, he sees cause for hope. He points to South American presidents such as Brazil's Lula who gave up power when they could have come up with a political fix and carried on. He also welcomes the end of America's hegemony, and believes the world's new multi-polarity could create greater stability. On the downside, he says Russia is "hopeless – if they didn't have energy, they'd be a totally inconsequential country. Nothing good has happened there since Putin came to power, and it'll need a generation of younger Russians to take over who don't have this chip on their shoulder." He sees China as a "really interesting challenge – a very high-quality authoritarian government". Can it challenge the liberal democratic model? "It's theoretically possible," he admits, "but it's such a hard system to duplicate, and I don't think the Chinese believe that anyone can duplicate it, and therefore they're not proselytising other countries to adopt it."

For all the qualifications and the new mood of pessimism over the immediate prospects for countries caught up in the crash, he still holds to his belief that liberal democracy is the endpoint of political evolution and the system to which countries will continue to aspire. China, the only current viable alternative, "lacks a basic legitimacy in the same way that these Arab regimes do, because it doesn't respect the rights of ordinary Chinese; it tramples on them all the time. There are lots of violent social protests that we never get to hear about, and the economic model is going to run out of steam because you cannot keep growing at 10% a year based on exporting all this stuff to people who can't afford it any more." There is no immediate threat to the Chinese system, he says, but in 20 or 30 years it will come under severe pressure. Liberal democracy is likely to win again, proving that he was right about the "end of history" even in this most dramatic and history-making of epochs. That, at least, is the theory.