Ever since Donald Trump’s presidential election and the UK’s Brexit vote, scholars have been trying to get a handle on populism. Who counts as populist? How does populism work? What are the conditions in which populists come to power or win referendums, and what tends to happen once they do?

Fortunately, political scientists have been asking these questions for decades – particularly those like me with an interest in Latin America, where populism has deep roots stretching back to the mid-20th century with leaders such as Brazil’s Getúlio Vargas and Argentina’s Juan Perón.

While those of us who study populism may disagree on some of its aspects, there is a surprising consensus on essential characteristics of the definition of populism. Put succinctly, populist forces tend to frame politics as a struggle between the will of the common people and a conspiring elite. Regardless of their policy positions, politicians can really only be considered populist if they use this type of Manichean, pro-people and anti-elitist rhetoric.

It is possible, with precision and reliability, to measure the extent to which populist ideas exist on a scale. For example, public opinion surveys are used to gauge the extent of populist ideas among populations. Research so far has consistently shown that populist attitudes are widespread, and that average levels differ little across countries.

How we combed leaders’ speeches to gauge populist rise Read more

Political leaders are not as easy to poll, but we can analyse what they say or write using methods of textual analysis. The most widely available political texts are transcripts of speeches. For more than a decade, my colleagues in Team Populism, a global network of social scientists dedicated to the scientific study of populism, have been conducting textual analysis on hundreds of political speeches.

Six months ago, we were approached by the Guardian to code hundreds more speeches in order to produce a single, up-to-date and comprehensive database of populist political discourse in 40 countries in Europe and the Americas. We were asked to focus on “chief executives” of governments – prime ministers, presidents and chancellors – and provide complete data for every term of office over the past two decades.

Quick guide The Global Populism Database Show Hide What is the Global Populism Database? The Global Populism Database is the most up-to-date, comprehensive and reliable database of populist discourse in the world. It was commissioned by the Guardian and built by Team Populism, a global network of scholars dedicated to the scientific study of the causes and consequences of populism. How was it compiled? Researchers combed through 720 speeches given by 140 government leaders in 40 countries over the past 20 years, and "scored" each based on the extent to which they used populist rhetoric. They were thus able to chart how populist various leaders were over the years. Which countries were included? The 40 countries include the eight most populous countries in the Americas: United States, Brazil, Mexico, Colombia, Argentina, Canada, Peru and Venezuela, and the seven largest European countries: Russia, Germany, Turkey, the UK, France, Italy and Spain. What were the conclusions? The study suggested that the number of populist leaders globally has doubled over the past 20 years, and that populist rhetoric - on left and right - has surged. Scholars generally agree that the central facet of populism is the use of a rhetoric that frames politics as a struggle between the virtuous will of the common people and an evil, conspiring elite.

The Global Populism Database contains the results of that research. We believe it is the most complete and comprehensive dataset of its kind, and we are confident it will be used by academics, journalists and policymakers for years to come.

It is unusual for journalists and social scientists to collaborate in this fashion, not least because we tend to operate on different timescales. (It will be months and years before information appears in academic journals and articles.) But we have already been able to draw some preliminary conclusions from our work. Some of these confirm what prior research suggests about populism. Other findings are genuinely surprising and new.

Our research documents a significant rise in the use of populist discourse in the 40 countries studied. The number of leaders who are at least “somewhat” populist has roughly doubled in the past 20 years after leaders of some of the world’s largest countries – India, the United States, Brazil, Mexico, Russia and Turkey – began using populist rhetoric.

It is often said, without much evidence, that populism is “on the rise”. This research appears to confirm that – and also provides some findings about the consequences of that change.

Our study demonstrates, for example, that populism can have serious repercussions for core institutions of liberal democracy. Press freedom and civil liberties, the quality of elections, and the separation of powers all tend to suffer under populists. In their crusade to destroy the establishment and enshrine the popular will, populists often embrace an “anything goes” approach to minority rights and democratic competition.

But if populists were all bad, why would anyone vote for them? Populists are often portrayed in the media as charismatic performers, manipulating ignorant electorates to inveigle their way into office. But the truth is a little more complicated. Our research shows populists have a modest, positive effect on voter turnout. Populism can dignify forgotten sectors of the population and incorporate them into other forms of democratic participation.

Our research has also found that populists reduce economic inequality. This was not something we expected to see, nor has it been found in previous academic research. It is also counterintuitive when we consider that populists tend to introduce stealth taxes (which are usually regressive, hitting the poorest hardest). At the very least, this finding deserves further exploration.

We also found that populists do not have a good record in solving the political corruption that – more often than not – fuelled their political rise. But many populists deliver, at least in the short term, on the policies they promised. Not all populists are demagogues.

Still, enough of them chip away at democratic rights to warrant us asking: how should we respond? I suggest three answers. The first is we should keep things in perspective. The unravelling of democracy in Venezuela is testament to the dangers of the most excessive forms of populism. But the consequences of milder populists securing power, such as those who have intermittently governed Italy, are much less dramatic.

The second is to avoid giving in to populism’s tendency to polarise. It is natural to feel threatened if populists accuse you of being an “enemy of the people”, but responding in-kind risks losing the very thing worth defending: civil discourse and liberal democracy. It can also play into populists’ claims of elite conspiracy. We should defend democratic institutions, but we should do so without attacking populists.

The final and most important recommendation: take seriously the populist frustrations. Populism responds to grievances with what the American historian Richard Hofstadter called a “paranoid mentality”. But those concerns are often real and have a basis in liberal democratic values and understandings of equality and fairness. If politics appears to benefit some at the expense of others, those at the losing end will feel like the rules of democracy have been broken.

Populism, in the end, is attractive because it takes seriously the grievances of people who feel abandoned and unfairly treated. By calling out injustice, and scapegoating elites who may be at least partly responsible for a society’s malaise, populists can sound reasonable to these groups. Instead of trying to silence populist voices, we should understand their concerns and engage with them.

• Kirk Hawkins is an associate professor at Brigham Young University and director of Team Populism.