The Global Populism Database is the most up-to-date, comprehensive and reliable repository of populist discourse in the world. It was commissioned by the Guardian and built by Team Populism, a global network of scholars dedicated to the scientific study of the causes and consequences of populism.

The database provides insight into the extent of populist rhetoric in the speeches of government leaders across the world over the past two decades. It is based upon populism “scores” given to 728 speeches by the presidents, prime ministers and chancellors of 40 countries.

How populist are you? Read more

The grades were derived using textual analysis of speeches, one of the more reliable, valid and precise techniques used by political scientists to gauge levels of populism.

Kirk Hawkins, an associate professor at Brigham Young University (BYU) who has pioneered the use of textual analysis in populism studies, oversaw the project. In collaboration with the Guardian, Hawkins and his team trained, instructed and supervised 46 paid researchers who examined speeches in 13 different languages.

The resulting database, which features speeches by leaders of all major countries in Europe, the Americas and India, gives a populism score to 182 terms in office between 1999 and 2019. The database contains 138 politicians, 31 of whom served at least two terms after they were re-elected.

Background

Despite widespread public debate about the rise of populism and substantial empirical research into the phenomenon, until now there has not been a valid, reliable database or index measuring populist discourse across the world. The absence of a single, trusted data source has proven a challenge for academics, policymakers and journalists.

It has made it more difficult to provide evidence-based answers to questions many people are asking right now. Has populism been on the rise in recent decades, and if so, where and why? What are some of the possible causes of populism, and its consequences? And given how liberally the “P” word is used, how much do leaders such as Donald Trump, Narendra Modi and Nicolás Maduro use populist rhetoric?

Since 2005, Hawkins and colleagues have been analysing the speeches of political leaders in much of the Americas and Central and Eastern Europe. Their research was held in the Populist Discourse Database for Leaders (PDDL), which also consisted of a significant body of research produced by Central European University’s Comparative Populism project, overseen by Prof Erin K Jenne.

The PDDL, however, was only partial. Several major countries, particularly in western Europe, were absent, while those countries for which there was data sometimes only had populism “scores” for one or two leaders.

The Guardian proposed helping Team Populism update its data, analysing hundreds more speeches in order to produce a complete database of 40 countries. A grant was secured from theguardian.org, a US-based non-profit, to enable the academics to complete the research. The project was overseen by Hawkins and conducted for the most part at IE University in Spain, with other researchers based at universities in Brazil, Germany and the US.

Selection of countries/leaders

While priority was given to nations with large populations and economies, pragmatic considerations also had an effect on the selection of countries. Priority was given to those that had already been at least partially scored by Team Populism scholars.

Another constraint was language: speeches were graded in their original language, in most instances by native speakers. (It was not possible to find researchers fluent enough in Filipino, Finnish, Greek, Indonesian, Swedish, Ukrainian or Urdu.)

The final list of 40 countries includes the eight largest (by population) in the Americas: the US, Brazil, Mexico, Colombia, Argentina, Canada, Peru and Venezuela. It also contains the seven largest European countries: Russia, Germany, Turkey, the UK, France, Italy and Spain.

In each case, researchers opted for the most senior elected official or “chief executive” of the government. In countries with parliamentary systems, that was a prime minister or chancellor. Otherwise, the leaders were presidents.

For Russia and Turkey, researchers used speeches from a mix of prime ministerial and presidential terms because Vladimir Putin and Recep Tayyip Erdoğan have been the most powerful politicians in their country regardless of which office they have occupied.

Holistic grading

There are several types of textual analysis, including some based on dictionaries of keywords, and other methods that break texts down into sentences or paragraphs. However, to “score” or “code” the levels of populist discourse and rhetoric in such a large number of speeches, across different regions and languages, Team Populism used a technique known as holistic grading.

Rather than judge a text based on word or sentence frequencies, holistic grading, which was originally devised by educational psychologists for grading student papers, requires researchers to take a step back and consider a text in its entirety. Researchers, or “coders”, read an entire text and compare it with a set of reference or “anchor” texts.

In political science, this technique works best at measuring things such as the tone, themes or ideas that exist in a speech. It can also be used across different languages as long as coders are trained in a common language with the same anchor texts.

Hawkins and his colleagues created a system that apportions a “score” for speeches based on the extent to which they contain populist ideas, discourse and rhetoric. The scores run from 0 (no populism) to 1 (clear populism, but used inconsistently or with a mild tone) to 2 (clear populism used consistently with a strong tone). “Clear populism” means that the core elements of populism – the notion of a virtuous will of the common people and the notion of an evil, conspiratorial elite – are present.

This approach does not treat populism as a binary category. Instead, it recognises that politicians can use varying degrees of populist rhetoric.

Ideology

For analysis and data visualisations, each leader term was given assigned an ideological classification: left, right or centre/neither. This was done using data provided by the Democratic Accountability and Linkages Project, the Political Representation, Parties, and Presidents Survey for Latin America and the Chapel Hill Survey for European Parties, as well as consultation with in-country experts.

Training researchers

Hawkins and his team trained the researchers, who were paid university students, over a two-week period. Each researcher completed a series of lessons introducing core concepts and the coding rubric, and was then required to practise the technique on 10 speeches capturing different points along the 0-2 scale.

These training sets included speeches by Robert Mugabe of Zimbabwe and Evo Morales of Bolivia (each scored with a 2), Sarah Palin of the US (roughly scoring a 1.5), Stephen Harper of Canada (with a score of 1), the former US president Barack Obama (roughly a 0.5), and Tony Blair and George W Bush (both close to 0). It also included three speeches from previous analyses of each researcher’s assigned country, in the original language.

Copies of the common set of English-language speeches and the coding rubric are available here.

Speech sampling

To provide comparability over time, each speech in the database comes from one of four categories: a campaign speech (usually the opening or closing of the campaign); a famous speech (something widely circulated that shows the leader at his or her best); a ribbon-cutting speech (given to a local audience); and an international speech (given to a foreign audience, preferably outside the country). In a few instances where one type of speech is unavailable, the sample consists of just three.

Each leader term – for example Tony Blair, 2005-2007 – was given four scores for each of the different speeches. To calculate the overall score for a leader term, an average of all four speeches was taken. In Blair’s case, for example, the scores for his third and final term were 0.2 (campaign ), 0 (ribbon cutting), 0.2 (international) and 0.1 (famous), giving him an overall score for this term of 0.125, so not populist. Each speech was typically scored by two researchers. The scores of each speech were then averaged out to give a final populism score per leader, per term.

Reliability

For human-coded data, it was important to measure the reliability of the people doing the coding. In most cases, two coders scored each speech; the final score for that speech was calculated by taking an average between the two. Using two researchers in this way also made it possible to calculate how often they agreed on their scores, using a statistic known as Krippendorff’s alpha, running from 0.0 to 1.0 (1.0 meaning perfect agreement). The Krippendorff’s alpha score for this research was .82 – a high level of agreement. This means that even in instances where only one researcher scored the speeches, we should be confident that the resulting scores are accurate.

Validity

It is also important to assess the validity of the measure, or the degree to which it matches our underlying definition. One way is by looking at the instrument itself, and we encourage readers to examine the coding rubric and anchor texts. Another is to look at the actual results and see how they match common perceptions. For example, we expect campaign and famous speeches to have higher levels of populism than ribbon-cutting and international speeches. This is what we find: average scores for campaign and famous speeches are 0.60 and 0.42, while those for ribbon-cutting and international are 0.23 and 0.20.

There are some results that may strike readers as surprising: such as the finding that speeches by the ex-president of Argentina, Cristina Fernández de Kirchner, were “not populist”. Similarly, given how frequently leaders such as Trump and Viktor Orbán are portrayed in the media as uber-populist leaders, readers may be surprised to see the coding research found their speeches were on average only “somewhat populist”. However, Team Populism remains confident that coding scores are accurate reflections of the levels of populist rhetoric in speeches, even when those may not accord exactly with public expectations.

Italy, Mexico and Brazil

The decision to focus on the chief executive of a country’s government posed complications in Italy, where the current prime minister is Giuseppe Conte. A former law professor appointed to represent the populist coalition between the Five Star Movement and the League, Conte is effectively a technocrat balancing between deputy prime ministers Luigi Di Maio and Matteo Salvini.

Unsurprisingly, Conte did not score highly on populist rhetoric. Given that Di Maio and Salvini are the real sources of power in the country, Team Populism altered Italy’s score. After consulting with in-country experts and considering campaign speeches by Di Maio and Salvini, which were also coded and indicated high levels of populism, Conte’s term (2018-present) was scored as “1”, which classified as “populist”. This was the only ad-hoc change made to the database.

Recent elections in Mexico and Brazil also posed problems for researchers. Neither Andrés Manuel López Obrador nor Jair Bolsonaro have been in office long enough to provide researchers with a wide sample of non-campaign speeches. The sample that was coded may not accurately reflect the rest of their time in office.

Date ranges

Each country has different term lengths for presidents and prime ministers and leaders may not always serve a full term. Consequently, for some countries, there are as many as seven leader terms, while for others there are as few as three. All 40 countries have data covering a continuous series of leader terms running for at least 17 years, although a few have short gaps, often where multiples leaders were in power for a short period of time. These include Croatia (2010), Czech Republic (2003-2005), Hungary (2003), Italy (2012-2013), Latvia (2008), Poland (2005, 2008-2010) and Slovakia (2011).

The start and end of the data coverage varies depending on the country. The earliest date for the start of a term is 1998, the latest 2019. The earliest date for which there is complete coverage across all 40 countries is 2004; in 2003 there is data for all but two countries (Guatemala and Panama).

Analysis of the data

Data analysis was conducted by Team Populism scholars and Guardian data journalists. Where occasional terms were missing, values were calculated by taking an average between the nearest available terms. When calculating the change in populist rhetoric over time, values for recent years were taken either from the political leader currently in office, the leader in office in 2018, or transferred from the most recent available term for which there was data in any given country (usually 2017, but in the case of Romania 2016).

For the purposes of analysis, Team Populism provided four bands, based on speech scores: not populist (0-0.49); somewhat populist (0.5-0.99); populist (1-1.49); very populist (1.5-2). Average scores for each leader term were rounded up to the nearest tenth.

Limitations of research

Why we stopped trusting elites Read more

This research is not intended as a definitive or final determination of how populist any given leader is. Textual analysis is just one approach for gauging the populism exhibited by a politician. No methodological approach is foolproof. Speeches are just one of many ways politicians communicate their ideas, particularly in a social media age; some politicians may tend to be more or less populist depending on the mode of communication.

The average score given to any political leader’s term in office is also dependent upon which speeches were selected. Analysing more speeches per leader may have yielded a more reliable average score. (However, previous research by Hawkins and his colleagues suggests that scholars can calculate reliable scores for a political leader with a sample of just four speeches per term.)

Questions about this project can be emailed to Paul Lewis at paul.lewis@theguardian.com or Kirk Hawkins at kirk_hawkins@byu.edu

Credits and acknowledgements

This research project was overseen by Hawkins (BYU). The initial dataset that formed the basis of this research, the Populist Discourse Dataset for Leaders (PDDL), will be made available on the website of Team Populism. It relied on research projects sponsored by both BYU and Central European University’s Comparative Populism Project, which is led by Prof Erin K Jenne.

Other scholars who helped with the update include Bruno Castanho Silva (University of Cologne) and Nina Wiesehomeier (IE University). Scholars who helped with the previous version of PDDL include Rosario Aguilar (CIDE, Mexico), Bojana Kocijan (Croatia), Levente Littvay (Central European University, Hungary), and Cristóbal Rovira Kaltwasser (Universidad Diego Portales, Chile). Researchers hired to produce the Global Populist Discourse Index include (at IE University) Ricardo Esteban Adeva, Ecab Amor Vázquez, Isabel Barquín, Rosario Beccar, María Patricia Bejarano, Cindia Escalante, Elvira Bermúdez Fernández, Kylar Cade, Sarah Elizabeth Chamberlain, Arielle Combrinck, Giulia D’Agostino, Macarena De Arriba, Peter de Jounge, Laura Escobar, Aashna Gadia, Laura Girard, Sho Izumisawa, Pamina Lantos, Caio Emanuel Marques, Antonio Martorano, Pola Nachyła, Ferdinand Pehamberger, Sofía Valentina Rodríguez Franco, Ana Rueda, Ariel Segal, Angela Selzer, Paola Toro, Sophie Verbeek, Lia-Maria Visoiu, Sarah Rachel Westvik, Frederic Ysewijn; (at BYU) Alyssa Doig, Alicia Galicia, Amanda Daniela Galán, Cristian Cortés Guzmán, Niyeli Herrera, Rebecca Isaksen, Alexandra Málaga, Alexia Modzewelska, Sasha Pozo; (at GIGA, Germany) Arunima Raigar and Paramveer Singh Gopal Singh; (at Federal University of Minas Gerais, Brazil) Eduardo Tamaki; and (at Vanderbilt University, USA) Kaitlen Cassell.