When political scientists write about populism, they often begin by trying to define it, as if it were a scientific term, like entropy or photosynthesis. To do so is a mistake. There is no set of features that exclusively defines movements, parties, and people that are called “populist”: the different people and parties that are placed in this category enjoy family resemblances of one to the other, but there is not a universal set of traits that is common to all of them.

There is, however, a particular kind of populist politics that originated in the United States in the 19th century, which has recurred there in the 20th and 21st centuries – and which began to appear in western Europe in the 1970s. In the past few decades, these campaigns and parties have converged in their concerns, and in the wake of the Great Recession, they have surged.

The kind of populism that runs through American history, and has been transplanted to Europe, cannot be defined exclusively in terms of right, left or centre: it includes both Donald Trump and Bernie Sanders, the Front National in France and Podemos in Spain. There are rightwing, leftwing and centrist populist parties. It is not an ideology, but a political logic – a way of thinking about politics. In his book on American populism, The Populist Persuasion, the historian Michael Kazin describes populism as “a language whose speakers conceive of ordinary people as a noble assemblage not bounded narrowly by class; view their elite opponents as self-serving and undemocratic; and seek to mobilise the former against the latter.”

That’s a good start. It doesn’t describe people like Ronald Reagan or Vladimir Putin, both of whom have sometimes been called “populist”, but it does describe the logic of the parties, movements, and candidates, from the US’s People’s Party of 1892 to Marine Le Pen’s Front National of 2016. I would, however, take Kazin’s characterisation one step further and distinguish between leftwing populists such as Bernie Sanders and Podemos’s Pablo Iglesias, and rightwing populists such as Trump and Le Pen.

Leftwing populists champion the people against an elite or an establishment. Theirs is a vertical politics of the bottom and middle, arrayed against the top. Rightwing populists champion the people against an elite that they accuse of favouring a third group, which can consist, for instance, of immigrants, Islamists, or African American militants. Rightwing populism is triadic: it looks upward, but also down upon an out group.

Leftwing populism is historically different to socialist or social democratic movements. It is not a politics of class conflict, and it does not necessarily seek the abolition of capitalism. It is also different to a progressive or liberal politics that seeks to reconcile the interests of opposing classes and groups. It assumes a basic antagonism between the people and an elite at the heart of its politics.

Rightwing populism, meanwhile, is different to a conservatism that primarily identifies with the business classes against their critics and antagonists below. In its American and western European versions, it is also different to an authoritarian conservatism that aims to subvert democracy. It operates within a democratic context.

Just as there is no common ideology that defines populism, there is no one constituency that comprises “the people”. They can be blue-collar workers, shopkeepers, or students burdened by debt; they can be the poor or the middle class. Equally, there is no common identification of “the establishment”. The exact referents of “the people” and “the elite” do not define populism, what defines it is the conflict between the two (or, in the case of rightwing populism, the three).

Populist movements have flourished in opposition, and have suffered identity crises when they have entered government

The conflict itself turns on a set of demands that the populists make of the elite – demands that the populists believe the establishment will be unwilling to grant them. Sanders wanted “Medicare for all” and a $15 minimum wage. If he had wanted the Affordable Care Act to cover hearing aids, or to raise the minimum wage from $7.25 to $7.75, that would not have defined a clash between the people and the establishment. If Trump were to demand an increase in guards along the Mexican border, or if Denmark’s rightwing People’s Party campaigned on a mere reduction in asylum-seekers, these demands would not open up a gulf between the people and the elite. But promising a wall that the Mexican government will pay for or the total cessation of immigration – that does establish a frontier.

These kinds of demands define the clash between the people and the establishment. If they are granted in whole or even in part, or if populists abandon them as too ambitious – as Syriza did with its demands for renegotiation of Greece’s debt – then the populist movement is likely to dissipate or to morph into a normal political party or candidacy. In this sense, American and western European populist movements have flourished when they are in opposition, and have suffered identity crises when they have entered government.

Populist campaigns and parties often function as warning signs of a political crisis. In both Europe and the US, populist movements have been most successful at times when people see the prevailing political norms – which are preserved and defended by the existing establishment – as being at odds with their own hopes, fears, and concerns. The populists express these neglected concerns and frame them in a politics that pits the people against an intransigent elite. By doing so, they become catalysts for political change.

Populist campaigns and parties, by nature, point to problems through demands that are unlikely to be realised in the present political circumstances. In the case of some rightwing populists, these demands are laced with bigotry or challenge democratic norms. In other cases, they are clouded with misinformation. But they still point to tears in the fabric of accepted political wisdom.

In recent decades, as the great postwar boom has stalled, the major parties on both sides of the Atlantic, have embraced a neoliberal agenda of free movement of capital and labour to achieve prosperity. Leaders have favoured increased immigration, only to find that American voters were up in arms about illegal immigration, and European voters were up in arms about immigrant communities they regarded as seedbeds of crime and, later, terrorism. In continental Europe, the major parties embraced the idea of the single currency only to find that it fell into disfavour during the Great Recession. In the United States, both parties embraced “free trade” deals only to discover that much of the public did not support these treaties.

Marine Le Pen, the leader of the Front National, addresses a rally in Frejus, France, in September. Photograph: Jean-Paul Pelissier/Reuters

In the last decades of the 19th century, as the People’s Party was erupting on the American scene, Europe was seeing the emergence of social democratic parties inspired by Karl Marx’s theory of socialism. Over the next 70 years, Europe would become home to an array of parties on the left, centre and right, but it would not witness anything resembling American populism until the 1970s.

Like the original People’s Party in the US, the European parties operated within the electoral arena and championed the “people” against an “establishment” or “elite”. The French Front National says that it represents the “little people” and the “forgotten members” against the “caste”. In Finland, the Finns Party says that it wants “a democracy that rests on the consent of the people and does not emanate from elites or bureaucrats”. In Spain, Podemos champions the “gente” against the “casta”. In Italy, Beppe Grillo of the Five Star Movement rails against what he calls the “three destroyers” – journalists, industrialists, and politicians. In the Netherlands, Geert Wilders’ Party for Freedom represents “Henk and Ingrid” against “the political elite”.

The first European populist parties were rightwing. They accused the elites of coddling communists, welfare recipients, or immigrants. As a result, the term “populist” in Europe became used pejoratively by leftwing and centrist politicians and academics. In the last decade, however, leftwing populist parties have arisen in Spain and Greece that direct their ire against the establishment in their country or against the EU headquarters in Brussels.

The main difference between US and European populists is that while American parties and campaigns come and go quickly, some European populist parties have been around for decades. That is primarily because many European nations have multi-party systems, and many of the countries have proportional representation that allows smaller parties to maintain a foothold even when they are polling in single digits.

Populist movements themselves do not often achieve their own objectives. Their demands may be co-opted by the major parties, or they may be thoroughly rejected. But they do roil the waters. They signal that the prevailing political ideology is not working and the standard worldview is breaking down.

No one, not even the man himself, expected Donald Trump to get the Republican presidential nomination in 2016. Similarly, no one, including Bernie Sanders, expected that through the California primary in June, the Vermont senator would still be challenging Hillary Clinton for the Democratic nomination.

Trump’s success was initially attributed to his showmanship and celebrity. But as he won primary after primary, political experts observed him playing on racist opposition to Barack Obama’s presidency or exploiting a latent sympathy for fascism among working-class white Americans. Sanders’s success invited less speculation, but commentators tended to dismiss him as a utopian and point to the airy idealism of millennial voters. If that were not sufficient explanation for his success, they also emphasised Hillary Clinton’s weakness as a frontrunner.

It makes more sense, however, to understand Trump and Sanders’s success as the latest chapter in the history of American populism. While strands of populism go back to the American revolution, it really begins with the People’s Party of the 1890s, which set the precedent for movements that have popped up periodically ever since. In the US, in contrast to Europe, these campaigns have burst forth suddenly and unexpectedly. Despite usually being short-lived, they have, nevertheless, had an outsized impact. And while they may seem unusual at the time, they are very much part of the political fabric of the nation.

While the history of American politics is riven with conflicts – over slavery, prohibition, abortion, intervention abroad – it is also dominated for long stretches by an underlying consensus about government’s role in the economy and abroad.

American politics is structured to sustain such prevailing worldviews. Its characteristics of winner-takes-all, first-past-the-post, single-member districts have encouraged a two‑party system. Third-party candidates are often dismissed as “spoilers”. Moreover, in deciding on whom to nominate in party primaries, voters and party bigwigs have generally taken electability into account, and in the general election, candidates have generally tried to capture the centre and to stay away from being branded “extremist”. As a result of this two-party tilt towards the centre, sharp political differences over underlying socioeconomic issues have tended to become blunted or even to be ignored, particularly in presidential elections.

But there are times, when, in the face of dramatic changes in society and the economy, or in America’s place in the world, voters have suddenly become responsive to politicians or movements that raise issues that major parties have either downplayed or overlooked completely.

The rise of the People’s Party was the first major salvo against the worldview of laissez-faire capitalism; the Louisiana governor Huey Long’s “Share Our Wealth” movement, which emerged in the wake of Franklin Roosevelt’s election in 1932, helped pressure Roosevelt to address economic inequality. Together, these movements established the framework that Bernie Sanders, who described himself both as a democratic socialist and as a progressive, would adopt during his 2016 campaign. Equally, the populist campaigns of George Wallace in the 1960s and Pat Buchanan in the 1990s foreshadowed the candidacy of Donald Trump.

During their heyday in the late 19th century, the populists of the People’s Party had a profound effect on American and – as it turned out – Latin American and European politics. It developed the logic of populism: the concept of a “people” arrayed against an elite that refused to grant necessary reforms. In American politics, the organisation was an early sign of the inadequacy of the two major parties’ views of government and the economy.

The populists were the first to call for government to regulate and even nationalise industries that were integral to the economy, like the railroads; they wanted government to reduce the economic inequality that capitalism, when left to its own devices, was creating, and they wanted to reduce the power of business in determining the outcome of elections. Populism had an immediate impact on the politics of some progressive Democrats, and even on Republicans such as Theodore Roosevelt. Eventually, much of the populist agenda was incorporated into Franklin Delano Roosevelt’s New Deal and into the outlook of New Deal liberalism.

In May 1891, the legend goes, some members of the Kansas Farmers Alliance, riding back home from a national convention in Cincinnati, came up with the term “populist” to describe the political views that they and other alliance groups in the west and south were developing. The next year, the alliance groups joined hands with the Knights of Labor, then the main workers’ organisation in the United States, to form the People’s Party, which, over the next two years, challenged the most basic assumptions that guided Republicans and Democrats in Washington. The party would be short-lived, but its example would establish the basis for populism in the United States and Europe.

A political cartoon from 1900. Photograph: Rights Managed/Mary Evans / Library of Congress

At the time, the leading Republicans and Democrats in the United States were revelling in the progress of American industry and finance. They believed in the self-regulating market as an instrument of prosperity and individual opportunity, and thought that the role of government should be minimal. Grover Cleveland, who was president from 1884 to 1888 and then from 1892 to 1896, railed against government “paternalism”. Public sector intervention, he declared in his second inaugural address, “stifles the spirit of true Americanism”; its “functions,” he stated, “do not include the support of the people”. Government’s principal role was to maintain a “sound and stable currency” through upholding the gold standard.

But during these years, farmers in the south and the plains suffered from a sharp drop in agricultural prices. Farm prices fell two-thirds in the midwest and south from 1870 to 1890. The plains, which prospered in the early 1880s, were hit by a ruinous drought in the late 1880s. But unsympathetic railroads, which enjoyed monopoly status, raised the cost of transporting farm produce. Many farmers in the south and the plains states could barely break even. The small family farm gave way to the large “bonanza” farm, often owned by companies based in the east. Salaries were threatened by low-wage immigrants from China, Japan, Portugal and Italy. Farmers who retained their land were burdened by debt. In Kansas, 45% of the land had become owned by banks.

The first populists saw themselves representing the “people”, including farmers and blue-collar workers, against the “money power” or “plutocracy”. That was reflected in their early programmes, which included a demand for the incorporation and recognition of labour unions alongside demands for railroad regulation, an end to land speculation, and easy money (through the replacement or supplementing of the gold standard) to ease the burden of debt that the farmers suffered from. Except for a few scattered leaders, the populists were not socialists. They wanted to reform rather than abolish capitalism, and their agent of reform was not the socialist working class, but the loosely conceived idea of “the people”.

When their demands – which also included a graduated income tax and political reforms to establish the secret ballot and the direct election of senators – proved too radical and far-reaching for the major parties, the People’s Party was created in 1892, and nominated a candidate for president. “We seek to restore the government of the Republic to the hands of ‘the plain people’, with whose class it originated,” the party’s first platform declared. “We believe that the powers of government – in other words, of the people – should be expanded … as rapidly and as far as the good sense of an intelligent people and the teachings of experience shall justify, to the end that oppression, injustice, and poverty shall eventually cease in the land.”

But in the 1880s and early 1890s, populist politics was primarily directed upward at the plutocrats

There was always a more conservative strain within the populist movement. In the south, some groups cooperated with the parallel national alliance of black farmers, but others did not. Populists also favoured the expulsion of Chinese immigrants, whom businesses had imported to provide cheap labour on western farms and railroads, and their support for that policy was often accompanied by racist rhetoric. But in the 1880s and early 1890s, populist politics was primarily directed upward at the plutocrats.

In the 1892 election, the People’s Party did remarkably well. Its woefully underfunded presidential candidate received 8% of the vote and carried five states. In the 1894 election, the People’s Party’s candidates for the House of Representatives won 10% of the vote. The party elected four congressmen, four senators, 21 state executives and 465 state legislators. With its base in the south and the west, and with Grover Cleveland wildly unpopular, the People’s Party looked to be on its way to challenging the Democrats as the second party. However, the election of 1894 turned out to be the party’s swansong.

In the end, the populists were done in by the dynamics of the two-party system. In the plains states, anger against Cleveland turned voters back to the more electable Republicans. In the south, Democrats subdued the People’s Party by a combination of co-option and, in response to the willingness of some populists to court the black vote, vicious race-baiting.

As liberal critics would later point out, the People’s Party had within it strains of antisemitism, racism, and nativism, particularly towards Chinese people, but these were at best secondary elements. Until the movement began to disintegrate, the original People’s Party was primarily a movement of the left. The first major instances of rightwing populism would come in the 1930s – from the Catholic priest and radio host Father Charles Coughlin – and then, in the 1960s, with George Wallace’s presidential campaigns.

Wallace, the Democratic governor of Alabama, helped to doom the New Deal majority and lay the foundations for the Reagan realignment of 1980. He created a new rightwing variety of populism – what the sociologist Donald Warren called “middle American radicalism” – which would migrate into the Republican party and become the basis of Donald Trump’s challenge to Republican orthodoxy in 2016.

To populist politician George Wallace, campaigning in Boston in 1968. Photograph: AP

The New Deal had rested on a tacit alliance between liberals and conservative southern Democrats, the latter of which resisted any legislation that might challenge white supremacy. As the party of Abraham Lincoln, the Republicans had traditionally been receptive to black civil rights, and the Republican leadership in Congress supported the Democratic president Lyndon Johnson’s Civil Rights and Voting Rights Acts of 1964 and 1965. The Arizona Republican senator Barry Goldwater was an early dissenter, but in the 1964 presidential election, Johnson easily defeated him. Johnson’s victory did not, however, signal widespread support for his civil rights initiatives, and after he passed the Voting Rights Act and introduced legislation known as the “War on Poverty”, a popular backlash grew. Wallace turned the backlash into a populist crusade.

Wallace would eventually make his name as an arch-segregationist, but he was initially a populist Democrat for whom race was strictly a secondary consideration. He initially ran for governor in 1958 as a New Deal Democrat and lost against a candidate backed by the Ku Klux Klan. After that, he pledged: “I will never be outniggered again.”

In 1962, Wallace ran again and this time he won as a proponent of “segregation now, segregation tomorrow, segregation forever”. In 1963, he gained notoriety when he attempted to block two black students from registering at the University of Alabama. In 1964, he ran in the Democratic primaries in Wisconsin, Indiana and Maryland, winning about a third of the vote – as high as 43% in Maryland, where he carried 15 of 23 counties. In 1968, he ran as an independent against the Republican Richard Nixon and the Democrat Hubert Humphrey. In early October, Wallace was ahead of Humphrey in the polls – in the end, he got 13.5% of the vote and carried five states in the south. In 1972, he ran as a Democrat, and stood a chance of taking the nomination when, in May, an assassin shot and crippled him while he was campaigning for the Maryland primary.

Wallace emphasised his opposition to racial integration, but he framed it as a defence of the average (white) American against the tyranny of Washington bureaucrats. Big government was imposing its will on the average person. Appearing on Meet the Press in 1967, Wallace summed up his candidacy:

There’s a backlash against big government in this country. This is a movement of the people … And I think that if the politicians get in the way, a lot of them are going to get run over by this average man in the street – this man in the textile mill, this man in the steel mill, this barber, this beautician, the policeman on the beat … the little businessman.

Wallace opposed busing – the practice of assigning children to particular state schools in order to redress racial segregation – because it was breaking up working-class neighbourhoods, and he attacked the white liberals who promoted it as hypocrites who refused to subject their children to what they insisted that the children of less affluent families must endure. “They are building a bridge over the Potomac for all the white liberals fleeing to Virginia,” he declared.

Wallace was not, however, a political conservative. On domestic issues that did not directly touch on race, he ran as a New Deal Democrat. In his 1968 campaign brochure, he boasted that in Alabama, he had increased spending on education, welfare, roads and agriculture.

In 1976, the Donald Warren published a study of “middle American radicals” (MARs). On the basis of extensive surveys conducted between 1971 and 72 and in 1975, Warren defined a distinct political group that was neither left nor right, liberal nor conservative. MARs “feel the middle class has been seriously neglected,” Warren wrote. They see “government as favouring both the rich and poor simultaneously”.

Warren’s MARs held conservative positions on poverty and racial issues. They rejected busing and welfare agencies as examples of “the rich [giving] in to the demands of the poor, and the middle-income people have to pay the bill”. They disliked the national government, but they also thought corporations “have too much power” and were “too big”. They favoured many liberal programmes. They wanted government to guarantee jobs to everyone. They supported price (but not wage) control, Medicare, some kind of national health insurance, federal aid to education and social security.

Warren found that MARs represented about a quarter of the electorate. They were on average more male than female; they had a high-school but not a college education; their income fell in the middle, or slightly below it; they had skilled or semi-skilled blue-collar occupations, or clerical or sales jobs – and they were the most likely demographic group to vote for George Wallace.

In other words, Wallace’s base was among voters who saw themselves as middle class – the American equivalent of “the people” – and who believed themselves to be locked in conflict with those below and above.

Forty years later, Trump portrays himself as an enemy of free trade treaties, runaway shops, and illegal immigration and as the champion of the “silent majority” – a term borrowed from Nixon – against the “special interests” and the “establishment” of both parties. “The silent majority is back, and it’s not silent. It’s aggressive,” Trump declared last year. At rallies, his campaign has given out signs that read: “The silent majority stands with Trump.”

In January, just before the Iowa caucuses, Trump’s campaign ran a television advertisement titled The Establishment. Seated behind a desk, Trump looked into the camera and said: “The establishment, the media, the special interest, the lobbyists, the donors, they’re all against me. I’m self-funding my campaign. I don’t owe anybody anything. I only owe it to the American people to do a great job. They are really trying to stop me.”

This essay is adapted from The Populist Explosion by John B Judis, published by Columbia Global Reports. To order a copy, go to bookshop.theguardian.com or call 0330 333 6846.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.