Liberals are living in alarming times. A few years before his death in 2012, the British historian Eric Hobsbawm passed summary judgment on the future of liberal democracy. ‘None of the major problems facing humanity in the 21st century can be solved,’ he wrote in the British magazine Prospect, ‘by the principles that still dominate the developed countries of the West: unlimited economic growth and technical progress, the ideal of individual autonomy, freedom of choice, electoral democracy.’ Hobsbawm did not say which were more at fault: liberal aims or liberal capacities. It hardly mattered. His prophetic voice seemed to echo the gathering fears of liberals themselves that perhaps their day was done.

More is in play here than an irrational loss of nerve. Much of what has shaken liberal self-belief since the 1990s is real enough and well-attested: external shocks from violent Islamism; injury to liberal values done by espionage, war-making and torture; a global banking collapse with its costly rescues and enduring economic harm.

Adverse trends are no less troubling. Within the Western world, economic inequality threatens social peace. Debt-strapped governments are patching up costly welfare systems that struggle to meet their promises. Disappointed voters are abandoning the centre-right and centre-left for newly sanitised extremes. Outside the slowed and aging West, illiberal politics appeal to rapidly growing revisionary powers: China, for example, and to some degree, India, Turkey and Iran.

With so much to feed their night-time fears, it is odd that liberals can sleep at all. But is this picture too bleak? Does it not one-sidedly ignore achievements that liberals should be proud of? And lasting strengths that, if remembered, could give them heart?

Ever since the early 19th century, when liberalism first took hold as a political practice, liberals have regularly veered from elation to despair, when defeating or compromising with one historic rival after another. Liberalism artfully co-opted its earliest competitors, conservatism and socialism. In Europe, it saw off the failed early 20th-century gods of fascism and communism. And by compromising with democratic populism, it reconfigured itself, stabilising after 1945 as liberal democracy. Liberal thinkers worked hard to show that liberalism was not only stronger and more durable than its rivals, but also fairer, more rational and more beneficial – in a word, better.

The philosopher of science Karl Popper gave eloquent voice to that sense of liberalism’s moral advantage in a talk called ‘The History of Our Time: An Optimist’s View’, given in 1986 in honour of Eleanor Rathbone, a British social reformer and pioneer of child benefit. Individuals who were lucky enough to live in the liberal democracies of western Europe and the United States experienced an awkward but workable mix of free markets, popular elections, welfare systems and personal liberties. Popper said of those people: ‘never before have their human rights, and their human dignity, been so respected, and never before have so many been ready to bring great sacrifices for others, especially for those less fortunate than themselves. I believe that these are facts.’ Popper acknowledged challenges ahead but judged that liberals could meet them if they stuck to their ideals: especially their intellectual ideals of experiment, open-minded argument and a refusal to prophesy.

Hobsbawm and Popper offer opposite poles of attraction for shaken liberals: false idols and historic incapacities on the one side; proud achievements and a robust mentality for facing future dangers on the other. Before asking which view is correct, the political animal Hobsbawm and Popper were talking about needs pinning down.

If, like me, you think that liberalism is worth standing up for and if, like me, you are worried about its future, then it helps to know what liberalism is. Otherwise we won’t know what we are worrying about. We won’t know what features of liberalism are essential, what incidental, or where it is going wrong. And we won’t know what aspects of liberal democracy we ought to conserve, and what to replace or repair.

So, what kind of thing is liberalism? You can treat it as an ethical creed, an economic picture of society, a philosophy of politics, a capitalist rationale, a provincial Western outlook, a passing historical phase, or a timeless body of universal ideals. None of that is strictly wrong – but all of it is partial. On my telling, liberalism is a modern practice of politics, with a history, practitioners, and an outlook to guide them. To recognise liberalism for what it is, we need to keep track of all three.

Viewed historically, liberalism was always a big tent. It made room for right and left, radicals and traditionalists, free-marketeers and social democrats: for Hayek and Keynes, Hoover and Roosevelt, Reagan and Kohl, but also for Johnson and Brandt. It is not uniquely American or peculiarly European. No anglophone monopoly, it had shining traditions in France, Germany and continental Europe generally, and also, if less coherently, in Latin America. Liberalism brimmed with ideas but was not itself a body of ideas, let alone a doctrine to be discovered after learned but speculative researches into the American Founders, or Spinoza and Locke, Lutheran Protestantism or the Catholic pre-Reformation, or Cicero and Socrates. Liberal thought could rarely be disentangled from liberal practice. Until specialisation set in, that was true even in a personal sense: pre-20th-century liberals were commonly thinkers and politicians or officials; Humboldt, for example, or Constant, Guizot, Tocqueville, Cobden and Mill.

None had understood, let alone felt, a world bursting with new people, new wealth and new problems in which old stability was gone for good.

As to when it began, liberalism has no foundation myth or year of birth. It arose as a practice of politics in the years after 1815 across the Euro-Atlantic world, but nowhere significantly before then. Liberalism responded to a novel condition of societies, energised by capitalism and shaken by revolution in which populations were growing fast and where, for better or worse, material and ethical change was now ceaseless.

Liberalism’s intellectual sources go back as far as energy and curiosity will take you. It had immediate and acknowledged debts to movements and thinkers of the 18th-century Enlightenment, when the Italian jurist Cesare Beccaria called for a humanisation of punishment, Montesquieu and Madison for the division of institutional powers, Voltaire and Jefferson for religious toleration, Hume and Kant for liberty from ethical tutelage. In Scotland, Adam Smith welcomed the first shoots of modern capitalism. None had understood, let alone felt, the novel condition of ceaseless change that confronted the first liberals. None had understood, let alone felt, a world bursting with new people, new wealth and new problems in which old stability was gone for good. Shake the great tree of pre-19th-century political thought and you get a rich basket of fruit, some of it ripe, some overripe. You do not get liberalism.

Liberalism has neither dynasties, presidencies, nor revolutions to mark out its life. However, four rough periods stand out, and in my book Liberalism: The Life of an Idea (2014) I’ve given them sharp dates for the sake of clarity. The first runs from 1830 to 1880, a time of youthful self-definition, a rise to power and successes. Liberalism offered nations the means to adapt laws and governments to productive new patterns of trade and industry, and to hold together divided societies from which familiar organising hierarchies and overarching creeds were disappearing. It fostered standards of humanity, too, particularly standards for how state power and moneyed power must not mistreat or neglect people with less power.

From 1880 until 1945, liberalism matured and undertook a historic compromise with democracy. A rising elite of educated, propertied men who might have expected to rule instead acknowledged democratic principles in politics, economics and cultural authority: they supported popular sovereignty at the polls, recognition of labour’s voice at work and the elimination of cultural hierarchies. In return, popular majorities accepted liberal limits as to how power was to be exercised, especially constitutional divisions of power, protections of property, and respect for personal rights. The compromise was neither smooth nor automatic, but grudging and hard-fought.

Though often spoken of interchangeably, liberalism and democracy are distinct. Liberalism is about how power is to be controlled, how human life is to be improved and how people are to enjoy respect. Democracy is about who belongs in that happy circle of voice, progress and protection. Liberalism answers the question ‘How?’ Democracy answers ‘Who?’ Liberalism is about content; democracy about scope. Liberalism limits how power is exercised. Democracy insists that the control of power lies in the end with the many, not with the one (autocracy) or the few (oligarchy). Liberalism lays out the feast, democracy draws up the guest list. With its promised boons and protections, liberalism can be exclusive or inclusive. The democratic challenge to liberals in the late 19th century was, as it remains today, to show how their ideals might be applied to everyone without exclusion, whoever they were.

Like the king in a game of chess, liberty comes into its own near the end of the game

After near-fatal failures (world wars, political collapses, economic slump) liberal democracy in 1945 won itself a second chance with the military defeat and moral ruin of fascism, its 20th-century rival to the right. That third period of achievement and vindication, running from 1945 to 1989, ended with the unconditional surrender of Soviet Communism, liberal democracy’s 20th-century rival to the left. We are now in a new phrase of liberal self-questioning.

As for what liberals have stood for, well, surely liberals believe in liberty? Indeed they do. But so do most non-liberals. Just about every modern rival to liberalism has claimed to stand somehow on the side of liberty. In The Communist Manifesto (1848), Karl Marx and Friedrich Engels looked forward to a classless society in which ‘the free development of each’ individual was ‘the condition for the free development of all’. Benito Mussolini described Italian fascists as ‘libertarians’ who believed in liberty, even for their enemies. You cannot leave liberty out of the story. Like the king in a game of chess, liberty comes into its own near the end of the game. For all its crowning appeal, liberty is the wrong place to begin.

At its broadest, liberalism is about improving people’s lives while treating them alike and shielding them from undue power. Four ideas in particular seem to have guided liberals through their history.

The first is that the clash of interests and beliefs in society is inescapable. Social harmony, the nostalgic dream of conservatives and the brotherly hope of socialists, is neither achievable nor desirable – because harmony stifles creativity and blocks initiative. Meanwhile conflict, if tamed and put to use as competition in a stable political order, could bear fruit as argument, experiment and exchange.

Secondly, human power is not to be trusted. However well power behaves, it cannot be counted on to behave well. Be it the power of state, market, social majorities or ethical authorities, the superior power of some people over others tends inevitably to arbitrariness and domination unless resisted and checked. Preventing the domination of society by any one interest, faith or class is, accordingly, a cardinal liberal aim.

Liberals also hold that, contrary to traditional wisdom, human life can improve. Progress for the better is both possible and desirable, for society as a whole and for people one by one, through education above all, particularly moral education.

Finally, the framework of public life has to show everyone civic respect, whatever they believe and whoever they are. Such respect requires not intruding on people’s property or privacy; not obstructing their chosen aims and enterprises; and not excluding anyone from such protections and permissions because they’re useless to society or socially despised.

Together, those four ideas have given liberalism a characteristic outlook with a high measure of unity and continuity, under a troubled surface of conflicting vocabularies and competing strategies of justification. The liberal way to hold to these four ideas is to insist on them all, without trying to sacrifice some to others, or put one in absolute command.

liberalism’s high bid has made it a doctrine of hope but equally an engine of disappointment

Though few have used the capital-L label ‘Liberal’, the political parties of the progressive mainstream in Europe and the United States have all held to these ideas in recognisable degree. In France after 1870, for example, the term ‘liberal’ practically vanished from party politics, but this was a verbal, not a factual extinction. Liberalism persisted in France as the progressive mainstream of politics, with right-wing liberals known as Republicans and left-wing liberals known as Radicals.

These four key ideas also distinguished liberals, point for point, from 19th-century socialists and conservatives, and in the 20th century from fascists and communists. They continue to distinguish liberals from their present-day rivals: competitive authoritarianism (China), ethnic nationalism (India), military populism (Egypt, Venezuela), nationalist Islamism (Turkey) and theocratic Islamism (Iran).

By insisting on pursuing all its ideals at once and in parallel, liberalism made a high bid. It was never easy to better people’s lives while letting them alone, nor was it ever easy to respect people’s beliefs while improving their minds. At the same time government could protect markets from state power, or people from market power, and give majorities their say while protecting minorities. Liberalism’s high bid has made it a doctrine of hope but equally an engine of disappointment, which brings us back to those contrasting sanguine and gloomy visions offered by Popper and Hobsbawm.

One thing to remember is that liberal self-doubt is nothing new. Pick any decade between George Dangerfield’s The Strange Death of Liberal England (1935) and H W Brands’s The Strange Death of American Liberalism (2001) and you will find an anxious liberal author with a fresh diagnosis or a new obituary.

Nor should we be swept away by over-general theoretical claims that caricature the liberal outlook, invoking vague sociology to discredit liberal ideas. John Hallowell, for example, argued in The Decline of Liberalism as an Ideology (1943) that mass society had neutered the liberal ideal of personal autonomy. Going the opposite way, other thinkers have blamed liberal ideas for gravely injuring society. In After Virtue (1981), Alasdair MacIntyre argued that liberal individualism has undermined community and solidarity. A common strategy among liberalism’s critics is to burden it with doctrinal freight of a justificatory kind that it does not need and that many liberals have never subscribed to.

Instead of focusing on higher methodology and philosophy, we should concentrate on where actual liberalism is going wrong. Its most obvious current failing is letting the power of the market run out of control. A direct consequence, rising inequality, has become the number-one topic in public debate. The economic arguments on this question are old.

Since the late 19th century, liberal thinking about the economy has gone back and forth between using the state to tame market power and using the market to tame state power. After 1945, liberal democracies appeared to get the balance right. Then in the 1980s, following a decade of inflation, joblessness and tax revolts, the balance swung strongly away from the state towards market power. Super-returns for a few and stagnant wages for the many have created social inequalities that are ethically offensive and, in a liberal democracy, politically unsustainable. Something has to give.

For the free-market right, the capitalist engine spreads its benefits in the end. If in the meantime it spreads inequality, so be it. On this reading, the egalitarian hopes of liberal democracy have to give. Left-wing liberals, meanwhile, see no inevitabilities here. Capitalism spreads inequality, they believe, if politics allows it to. They continue to trust the capacity of politics to tame markets, and so, for them, untrammeled capitalism has to give.

Intriguingly, the French economist Thomas Piketty’s 685-page best-selling book on economic inequality, Capital in the 21st Century (2014), combines elements of both perspectives. Presenting a neo-Marxist view of capitalism’s remorseless economic logic, Piketty argues that capitalism tends naturally to generate inequality. Western societies after 1945 grew fairer, he argues, thanks only to episodic or contingent factors, such as capital-replacement spending after wars, and higher than usual population growth, which temporarily tipped the economic advantage from capital to labour. And yet Piketty takes a left-liberal approach on what is to be done. His is a classic welfarist solution requiring political will: the taxation of wealth.

Technical change and economic growth might have bettered people’s lives but they are now endangering the globe and threatening future lives

Hostile critics have challenged Piketty’s numbers, and even sympathetic critics have questioned his chosen remedy as presently out of reach, because corporate voices and moneyed lobbies are too strong. Yet here, too, there are no inevitabilities, which brings us back to the sanguine Popper.

How liberalism is currently practiced is not how it must be practiced. Liberalism, as Popper correctly judged, drew its strength from adaptability, successfully reconfiguring itself as liberal democracy. If a better balance between state and market is to be found now, liberal democracy needs reconfiguring again. That is not a question of finding the right policies. No size fits all and no policy works well for long. The challenge is to alter mentalities, create agendas and establish points of resistance to what look like invincible forces. None of that can happen automatically or quickly. Liberalism’s first reconfiguration – its accommodation with the popular forces of democracy – took from the 1880s to the 1960s to complete. That achievement required doggedness, imagination and clarity of mind. So will a new configuration.

The debate about inequality is a start, no more. Inequality in the rich world might be the most obvious way in which present-day liberalism is going wrong, but it is not the only or the gravest way. The glum Hobsbawm was also right. Capitalist economics, he implied, was creating a planetary menace that liberal-democratic politics was incapable of meeting. Technical change and economic growth might have bettered people’s lives but, left to run uncontrolled, they were now endangering the globe and threatening future lives.

Liberalism’s aims and ideals remain what they always were: resistance to domineering power, faith in human progress and insistence on civic respect for people. None of those need changing or abandoning. However, liberals urgently need to rethink how those aims and ideals are to be pursued in bewilderingly novel conditions. How is market power to be tamed? Can ‘good enough’ replace ‘ever more’ as our gauge of progress? And how is respect for present and for future people to be squared?

These are daunting questions that will not be answered tomorrow. Finding answers and changing mentalities is the work of decades. Liberals have been searching for points of stability amid bewildering change for 200 years. No point of stability lasts, yet liberals can blame only themselves if they stop looking. What they need now is a steady resolve.

Liberalism (2014) by Edmund Fawcett is published via Princeton University Press.